AI Evaluation Engineer (QA)

Responsibilities

• Run large-scale evaluations across question sets, from human UAT batches to hundreds of thousands or millions of automated evaluations.
• Statistically measure factual grounding and before-and-after accuracy improvements.
• Build a metrics framework demonstrating response-quality improvement over time.
• Design load and quality tests as the content corpus scales.
• Define and maintain test plans, test cases, and quality gates.
• Automate regression and evaluation suites and integrate them into CI/CD pipelines.
• Communicate quality metrics to technical and non-technical stakeholders.
• Collaborate with engineering to reproduce, triage, and verify fixes.
• Continuously improve QA processes and test coverage.

Requirements

• Bachelor’s degree in a technical field or equivalent experience.
• At least 4 years of QA or test-engineering experience with exposure to data and machine-learning systems.
• Strong Python skills and data-science techniques for measuring factual grounding and answer quality.
• Experience with LLM evaluation frameworks and statistical analysis.
• Ability to build large-scale automated evaluation harnesses and lighter human-in-the-loop A/B tests.
• Comfort working with outputs from multiple LLM providers.
• Experience with test-automation frameworks and scripting.
• Strong analytical, communication, problem-solving, decision-making, and collaboration skills.
• Prior consulting experience.
• Life Sciences industry experience and connections are preferred.

Benefits

• Equal Opportunity Employer committed to diversity, equity, and inclusion.
• Accommodations are available upon request throughout the recruitment process.