Software Engineer III

Responsibilities

• Build automated testing frameworks, validation checks, and CI/CD pipeline integrations with reproducible diagnostics.
• Develop methods to explore complex software behavior, identify coverage gaps, and detect defects.
• Build automated checks that validate software behavior and data against requirements.
• Develop evaluation frameworks for generative AI and LLM-based systems using LLM-as-a-judge patterns, rubric-based scoring, and automated calibration.
• Apply generative AI, NLP, machine learning, heuristic, deterministic, and model-based approaches to software testing problems.
• Integrate testing tools into development pipelines, support delivered software, and produce reproducible diagnostics.
• Prototype machine learning and LLM capabilities and collaborate with engineering, design, and quality partners on requirements, code reviews, and blockers.

Requirements

• 5+ years of professional software engineering experience, including 2+ years of hands-on experience building, integrating, or evaluating AI/ML and LLM-based systems.
• Master of Science in Computer Science, Mathematics, or a related field, or equivalent professional engineering experience.
• In-depth experience with LLMs, generative AI, and NLP concepts, architectures, and failure modes.
• Experience developing generative-system evaluation frameworks, including dataset curation, model-as-a-judge design, validation criteria, error analysis, and quantitative and qualitative metrics.
• Proficiency in Python, algorithms, data structures, modular design, automated testing, debugging, and version control.
• Experience preparing datasets, defining validation criteria, and performing error analysis using quantitative ML metrics.
• Experience with NLP or large language models, including prompting, fine-tuning, or retrieval-augmented generation.

Benefits

• British Columbia compensation range of 122,300–170,700 CAD, depending on location.
• Three weeks of vacation per year to start and 10 days of sick time annually.
• Paid top-up to EI/QPIP benefits up to 100% of base salary for eligible maternity, parental, and adoption leave.
• Extended health, dental, and vision coverage; life insurance; disability insurance; and a retirement plan for regular full-time employees.
• Certain roles may be eligible for bonus and other incentive programs.
• Additional benefits may include healthcare coverage, mental well-being support, retirement savings, paid time off, family leaves, and complimentary games.