Senior Data Scientist AI Evaluation

This position is listed on behalf of a partner company, which manages all applications and next steps. Our partner is looking for a Senior Data Scientist AI Evaluation based in Canada. We are seeking a senior-level data scientist to establish and advance the evaluation practices that measure the quality, reliability, and correctness of AI models and intelligent agents. In this role, you will transform complex quality questions into measurable standards, trusted ground truth, and rigorous evaluation frameworks. You will design repeatable testing processes that identify performance regressions and support safer, more confident AI releases. Working closely with Product, Engineering, Analytics Engineering, and business stakeholders, you will turn evaluation results into practical improvements. This is a high-impact individual contributor position offering ownership of a developing AI evaluation practice within a fast-growing financial technology environment. You will help shape how AI quality is measured, validated, documented, and continuously improved across the organization. Accountabilities - AI Evaluation Design: Define ground truth, quality metrics, scoring methodologies, and evaluation frameworks to assess the accuracy, consistency, and reliability of AI models and agents. - Evaluation Pipeline Development: Build repeatable evaluation loops that monitor model quality over time, identify performance regressions, and support reliable release decisions. - Statistical Measurement and Validation: Apply rigorous statistical methods to evaluation design, including sample sizing, confidence intervals, significance testing, and techniques for handling non-deterministic model outputs. - Automated Grader Validation: Compare automated evaluation methods with human assessments to establish reliability, identify limitations, and improve scoring accuracy.

- Evaluation Infrastructure Collaboration: Partner with Engineering and Analytics Engineering to operationalize evaluation frameworks, integrate testing into development workflows, and support scalable evaluation infrastructure. - Performance Analysis and Improvement: Interpret evaluation results, identify weaknesses and quality gaps, and provide actionable recommendations to improve model and agent performance. - Quality Standards and Documentation: Establish evaluation guidelines, documentation practices, review processes, and consistent quality standards across AI development initiatives. - Cross-Functional Partnership: Collaborate with Product, Engineering, Analytics Engineering, and business stakeholders to define success criteria and align evaluation priorities with business needs. - Continuous Improvement and Mentorship: Promote evaluation best practices, share technical insights, and help foster a culture of measurable AI quality, accountability, and evidence-based decision-making. - Safe AI Deployment: Support teams in making informed release decisions through independent, reliable assessments of model performance and readiness. Requirements: - Approximately 6–10 years of experience in quantitative data science, machine learning, or a related technical discipline, with focused experience in measurement, evaluation, experimentation, or model validation. - Strong quantitative and statistical foundations, including experience designing rigorous experiments and interpreting results with appropriate uncertainty measures. - Demonstrated experience defining meaningful evaluation metrics, establishing ground truth, and assessing the quality of complex or ambiguous model outputs. - Proficiency in Python and SQL, with experience applying these skills to data analysis, model assessment, and evaluation workflows. - Experience evaluating machine learning models in production environments and translating findings into practical improvements.