Staff Data Scientist, Ai Evaluations Platform
ODAIA
Job DescriptionRBC’s AI Group is building trusted AI capabilities for the enterprise, and evaluation is one of the core controls that makes that possible. As Staff Data Scientist, AI Evaluations, you will be a senior technical leader in the data science function responsible for how RBC measures model and agent quality, safety, risk, and performance. You will design, build, and continuously improve evaluation datasets, sourcing methods, LLM judge design, deterministic scorers, human evaluation protocols, quality benchmarks, and measurement frameworks that help AI systems move from experimentation to production with evidence and control, while providing technical guidance and mentorship to other data scientists on the team.
What will you do?
Design and drive advanced model and agent evaluation methodologies, including evaluation datasets, rubrics, LLM-as-judge methods, deterministic scorers, human evaluation, and measurement frameworks, providing technical guidance to junior team members.
Define evaluation science standards that translate model risk, responsible AI, product quality, safety, and business expectations into measurable criteria, repeatable methods, and clear evidence.
Own the end-to-end lifecycle for evaluation datasets and scorecards, including sourcing, curation, validation, quality checks, versioning, lineage, reuse, and ongoing improvement.
Design scalable evaluation approaches for generative AI and agentic systems, including task-level, workflow-level, trajectory-level, and runtime evaluation methods.
Partner with AI research, platform engineering, product, risk, governance, and business teams to embed evaluations into build, release, certification, monitoring, and recertification workflows.
Establish human evaluation and review protocols that produce reliable labels, reviewer guidance, adjudication processes, quality controls, and audit-ready evidence.
Measure and improve scorer accuracy, calibration, robustness, failure-mode coverage, and explainability across automated and human evaluation approaches.
Provide clear technical leadership, executive-ready communication, and mentorship to junior data scientists to help RBC scale trusted AI with speed, rigor, and control.
In this role, you will communicate and interact frequently with RBC partners and/or employees located across Canada and/or worldwide
What do you need to succeed?Must Have
8+ years of experience in data science, applied machine learning, AI evaluation, ML quality, or a related technical field, including experience providing technical leadership and mentorship within a team.
Strong experience designing evaluation frameworks for ML, generative AI, or agentic AI systems, including metrics, datasets, benchmarks, rubrics, and quality measurement.
Practical experience with LLM evaluation methods such as LLM-as-judge, deterministic scoring, human evaluation, hallucination assessment, factuality assessment, safety evaluation, or model quality benchmarking.
Strong technical foundation in data science, statistics, machine learning, experimentation, data curation, Python, SQL, and modern AI/ML development practices.
Proven ability to translate governance, model risk, responsible AI, and business requirements into measurable controls, repeatable evaluation processes, and decision-ready evidence.
Strong communication and stakeholder management skills, with the ability to influence senior leaders across research, engineering, product, governance, risk, and business teams.
Nice to Have
Experience evaluating agentic AI systems, tool‑calling workflows, multi‑step reasoning, runtime traces, trajectory scoring, or workflow‑level performance.
Experience in financial services, regulated AI, model risk management, responsible AI, enterprise governance, or audit‑ready evidence processes.
Familiarity with tools and platforms such as MLflow, Langfuse, LangSmith, OpenTelemetry, Grafana, CI/CD pipelines, or comparable evaluation and observability tooling.
Publications, patents, open‑source contributions, or industry work related to AI evaluation, ML quality, AI safety, applied research, or responsible AI.
Ph.D. or Master’s degree in a quantitative or technical field, such as Applied Mathematics, Engineering, Computer Science or a related computational discipline.
What’s in it for you?We thrive on the challenge to be our best, progressive thinking to keep growing, and working together to deliver trusted advice to help our clients thrive and communities prosper. We care about each other, reaching our potential, making a difference to our communities, and achieving success that is mutual.
A comprehensive Total Rewards Program including bonuses and flexible benefits, competitive compensation, commissions, and stock where applicable
Leaders who support your development through coaching and managing opportunities
Ability to make a difference and lasting impact
Work in a dynamic, collaborative, progressive, and high‑performing team
A world‑class training program in financial services
Opportunities to do challenging work.
Job SkillsBig Data Analytics, Critical Thinking, Decision Making, Industry Knowledge, Machine Learning (ML), Results‑Oriented, Software Engineering, Software Product Design
Additional Job DetailsAddress: 6666 RUE ST URBAIN, FLOOR 3:MONTRÉAL
City: Montréal
Country: Canada
Work hours/week: 37.5
Employment Type: Full time
Platform: TECHNOLOGY AND OPERATIONS
Job Type: Regular
Pay Type: Salaried
Posted Date: 2026-10-09
Application Deadline: 2026-10-31
Note: Applications will be accepted until 11:59 PM on the day prior to the application deadline date above
Our Employment Opportunities
At RBC, we are guided by living shared values of Client First, Integrity, Collaboration, Respect and Excellence and winning together as One RBC. We believe an inclusive workplace that has diverse perspectives is core to our continued growth as one of the largest and most successful banks in the world. Maintaining a workplace where our employees feel supported to perform at their best, effectively collaborate, drive innovation, and grow professionally helps to bring our Purpose to life and create value for our clients and communities. RBC strives to deliver this through policies and programs intended to foster a workplace based on respect, belonging and opportunity for all.
#J-18808-Ljbffr
What will you do?
Design and drive advanced model and agent evaluation methodologies, including evaluation datasets, rubrics, LLM-as-judge methods, deterministic scorers, human evaluation, and measurement frameworks, providing technical guidance to junior team members.
Define evaluation science standards that translate model risk, responsible AI, product quality, safety, and business expectations into measurable criteria, repeatable methods, and clear evidence.
Own the end-to-end lifecycle for evaluation datasets and scorecards, including sourcing, curation, validation, quality checks, versioning, lineage, reuse, and ongoing improvement.
Design scalable evaluation approaches for generative AI and agentic systems, including task-level, workflow-level, trajectory-level, and runtime evaluation methods.
Partner with AI research, platform engineering, product, risk, governance, and business teams to embed evaluations into build, release, certification, monitoring, and recertification workflows.
Establish human evaluation and review protocols that produce reliable labels, reviewer guidance, adjudication processes, quality controls, and audit-ready evidence.
Measure and improve scorer accuracy, calibration, robustness, failure-mode coverage, and explainability across automated and human evaluation approaches.
Provide clear technical leadership, executive-ready communication, and mentorship to junior data scientists to help RBC scale trusted AI with speed, rigor, and control.
In this role, you will communicate and interact frequently with RBC partners and/or employees located across Canada and/or worldwide
What do you need to succeed?Must Have
8+ years of experience in data science, applied machine learning, AI evaluation, ML quality, or a related technical field, including experience providing technical leadership and mentorship within a team.
Strong experience designing evaluation frameworks for ML, generative AI, or agentic AI systems, including metrics, datasets, benchmarks, rubrics, and quality measurement.
Practical experience with LLM evaluation methods such as LLM-as-judge, deterministic scoring, human evaluation, hallucination assessment, factuality assessment, safety evaluation, or model quality benchmarking.
Strong technical foundation in data science, statistics, machine learning, experimentation, data curation, Python, SQL, and modern AI/ML development practices.
Proven ability to translate governance, model risk, responsible AI, and business requirements into measurable controls, repeatable evaluation processes, and decision-ready evidence.
Strong communication and stakeholder management skills, with the ability to influence senior leaders across research, engineering, product, governance, risk, and business teams.
Nice to Have
Experience evaluating agentic AI systems, tool‑calling workflows, multi‑step reasoning, runtime traces, trajectory scoring, or workflow‑level performance.
Experience in financial services, regulated AI, model risk management, responsible AI, enterprise governance, or audit‑ready evidence processes.
Familiarity with tools and platforms such as MLflow, Langfuse, LangSmith, OpenTelemetry, Grafana, CI/CD pipelines, or comparable evaluation and observability tooling.
Publications, patents, open‑source contributions, or industry work related to AI evaluation, ML quality, AI safety, applied research, or responsible AI.
Ph.D. or Master’s degree in a quantitative or technical field, such as Applied Mathematics, Engineering, Computer Science or a related computational discipline.
What’s in it for you?We thrive on the challenge to be our best, progressive thinking to keep growing, and working together to deliver trusted advice to help our clients thrive and communities prosper. We care about each other, reaching our potential, making a difference to our communities, and achieving success that is mutual.
A comprehensive Total Rewards Program including bonuses and flexible benefits, competitive compensation, commissions, and stock where applicable
Leaders who support your development through coaching and managing opportunities
Ability to make a difference and lasting impact
Work in a dynamic, collaborative, progressive, and high‑performing team
A world‑class training program in financial services
Opportunities to do challenging work.
Job SkillsBig Data Analytics, Critical Thinking, Decision Making, Industry Knowledge, Machine Learning (ML), Results‑Oriented, Software Engineering, Software Product Design
Additional Job DetailsAddress: 6666 RUE ST URBAIN, FLOOR 3:MONTRÉAL
City: Montréal
Country: Canada
Work hours/week: 37.5
Employment Type: Full time
Platform: TECHNOLOGY AND OPERATIONS
Job Type: Regular
Pay Type: Salaried
Posted Date: 2026-10-09
Application Deadline: 2026-10-31
Note: Applications will be accepted until 11:59 PM on the day prior to the application deadline date above
Our Employment Opportunities
At RBC, we are guided by living shared values of Client First, Integrity, Collaboration, Respect and Excellence and winning together as One RBC. We believe an inclusive workplace that has diverse perspectives is core to our continued growth as one of the largest and most successful banks in the world. Maintaining a workplace where our employees feel supported to perform at their best, effectively collaborate, drive innovation, and grow professionally helps to bring our Purpose to life and create value for our clients and communities. RBC strives to deliver this through policies and programs intended to foster a workplace based on respect, belonging and opportunity for all.
#J-18808-Ljbffr
Vacancy posted 1 hour ago
Similar jobs that could be interesting for youBased on the Staff Data Scientist, Ai Evaluations Platform in Winnipeg, MB vacancy
- Meet Fetch AI & DataAI & Data at Fetch sit at the center of how we understand our business, make decisions... ...learning, experimentation, and data platforms, all working together to turn data into... ...financial planning. We are seeking a Staff Data Scientist to serve as the company‑...SuggestedFull timeRemote work
- ...developing a diversified product platform aimed at addressing... ...it.You will be one of the early data science hires shaping how Purpose... ...years — and increasingly, how our AI systems understand and serve the... ...You're an engineer as much as a scientist — you own the work from raw event...SuggestedInternshipWork at officeWork from homeFlexible hours
- Build Meeru AI’s data foundation with expert SQL and Python for production pipelines, connectors, and curated models across enterprise finance systems.As a Staff Data Engineer, you will own major data-platform subsystems end to end. You will transform warehouse data, integrate...Suggested
$80k - $175k per year
Staff Data Scientist Application Deadline: 11/14/2026Address: 33 Dundas Street WestJob Family Group: Data Analytics & ReportingThis is a hybrid... ...at scale.Experience developing machine learning models, and evaluating emerging technologies and tools to deliver innovative data-...SuggestedContract workPart time- Lenovo is seeking an AI Data Intelligence Staff Researcher to develop intelligent, efficient, and reliable AI solutions for Hybrid AI computing and data infrastructure. In this role, you will use machine learning, deep learning, data science, and software engineering to improve...Suggested
- Cohere is seeking a Senior Research Scientist focused on AI evaluation methods to support our innovative foundation models. This remote or on-site role... ...in building AI prototypes and evaluations • Proficiency in data review for quality assurance • Commitment to rigorous...Remote work
- Apply data science and Python expertise to evaluate coding agents through reproducible environments, testing, and... ...verification.This Competitive Data Scientist role focuses on building reliable... ...judgment, practical programming, and evaluation expertise to strengthen coding-...
- ...Pinterest is seeking a Senior Data Scientist to shape the future of people-facing and business-facing products. You will apply quantitative modeling... .... You will build and prototype analysis pipelines, leverage AI to accelerate execution, and communicate insights to cross-...
- About the Ads Data Science TeamThe Ads Data Science team at Reddit... ...in the areas of ads and platform measurement, campaign and creative... ...collaborative team of passionate data scientists and engineers who push the... ...for a highly experienced Staff Data Scientist to advance the...RemplacementFlexible hours
$200k - $250k per year
# Staff Data Scientist, Growth* 21 October, 2026* by adminTags: Web3 Jobs • Cryptocurrency Data Jobs • Blockchain Senior Jobs • Blockchain Full... ...environmentsExperience with experimentation and product analytics platforms such as Posthog, Statsig, AmplitudeExperience working with...Long term contractFull timeWork at officeRemote work$145.9k per year
...If so, this might be the role for you! We're looking for a Staff Data Scientist, Applied Machine Learning (ML) to join our growing Data Science... ...and systems for automated decisioning. You will train and evaluate models using techniques such as graph representation learning...Long term contractInternshipWork at officeLocal area- ...improve customer experience and trust for buyers & sellers on the platform.Define & Monitor Metrics: Build and develop KPIs to track the... ...new processes and deploying new technologiesHas strong analytical/data science skills to dive into data and identify root causesBuilds models...Long term contractImmediate start
- ...the world have invested in for years. We're looking for a Staff Data Scientist who's an expert in building and shipping production pricing... ...and measurable outcomes Set the standard for model evaluation, validation, and monitoring — including knowing when CV metrics...Local area
$137.3k - $171.6k per year
The Data and AI Platform team at Rivian supports developing and deploying Generative AI systems. We operate highly scalable infrastructure for serving and orchestrating LLMs, Vector Stores, and agents. As a staff engineer, you will be crucial in designing, implementing, and...Hourly payLocal area- Build reliable data platforms with Python and SQL at Function Health, working remotely from Canada on infrastructure for AI-powered healthcare.As a Staff Software Engineer, you will shape Function Health’s data platform infrastructure. Your work will support reliable contracts...Local areaRemote work
$180k - $200k per year
...Job Title: Staff Data Scientist (AI & Machine Learning) Location: Vancouver, BC – 3 days a week onsite Job ID: 87597 Overview As a Staff Data Scientist, you are a technical visionary and a strategic leader within our AI organization. While a Senior Data Scientist excels...3 days per week- ...signals, identity, and privacy.Reddit seeks a highly experienced Staff Data Scientist for its Ads Data Science team. You will own a critical... ...identity resolutionDesign privacy-conscious links between on-platform and off-platform actionsAdvance Brand and Conversion Lift experimentation...Remote work
- Data Science is at the heart of Lyft’s products and decision-making. As a member of the Rider... ...the world’s best transportation. Data Scientists take on a variety of problems ranging from... ...and impactful problems in ridesharing.As a Staff Data Scientist in the Rider team, you will...
$110.5k - $143k per year
...Develop and execute comprehensive data governance strategy to ensure... ...and educate data services staff while promoting growth and excellence... ...with the line of business to evaluate KPI measurement needs. Communicate... ...vacancy. Hydro One uses AI tools to assist in the...- A career as a Senior AI Data scientist in theHyper Automation – Cross-sector Solutions, Acceleration and Data team atNational Bank means serving... ...impact on the performance andevolution of an enterprise platform thanks to your expertise in applied datascience, model evaluation...Flexible hours
$170k per year
Databricks is on a mission to simplify and democratize data and AI — from making the next mode of transportation a reality to accelerating... ...building and running the world's best data and AI infrastructure platform so our customers can use deep data insights to improve their...Summer workWorldwide- Become an integral part of a machine learning squad as a Data Scientist focusing on innovative AI solutions, including ML and Generative AI. Utilize your... ...ensuring continuous improvement through monitoring and evaluation strategies. Key Responsibilities: • Develop and...
- ...Shape the data foundation behind Coveo’s next generation of innovation... ...of developers more effective? As a Staff Software Developer on Coveo’s Data Platform team, you’ll tackle complex, high-impact... ...the ability to navigate ambiguity, evaluate trade-offs and independently drive...Long term contractImmediate start
- ...are we? Cohere is the leading security-first enterprise AI company, building cutting-edge foundation AI models and end... .... Join us! Why this role? As a Member of Technical Staff in Data Analysis and Evaluation, you will play a pivotal role in ensuring the quality, reliability...Full timeWork at officeLocal areaRemote workHome officeFlexible hours
$104k per year
...Company OverviewRobots & Pencils is an applied AI engineering firm building the next... ...Position OverviewWe’re looking for a Staff Platform Engineer to define and lead platform engineering... ...and platform challenges Evaluate emerging tools and technologies, recommending...Local areaRemote work- ...Overview Function Health is building an AI operating system for health, offering... ...lab testing, imaging, longitudinal data, and guidance from AI and clinical... ...and is scaling its integrated health platform.Role Function is hiring a Staff Software Engineer for its data...Full timeContract workLocal areaRemote workFlexible hours
- Shape Instacart’s Ads data platform remotely with Spark, SQL, and distributed systems expertise... ...for critical advertising workflows.As a Staff Software Engineer, you will advance a... ...testing. The work serves advertisers, data scientists, machine learning engineers, and sales...Remote work
- Shape Jobber’s data platform through scalable pipelines, AWS expertise, and technical leadership across teams. This is a strategic, hands-on engineering role.As a Staff Data Engineer on Jobber’s Data Platform Team, you will shape data integration architecture and delivery....
$88k - $132k per year
...$132,000 / yearJob Summary EY Canada is seeking a Senior Data Scientist for its AI Centre of Excellence within the Office of the Chief Technology... ...and research applications.Implement, fine-tune, and evaluate large language model solutions, including generative AI, natural...Full timeWork at office- Shape Jobber’s data platform through scalable pipelines, technical leadership, and hands-on engineering across batch and real-time systems.As a Staff Data Engineer on the Data Platform Team, you will shape foundational data components and guide engineering excellence. You will...
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Staff Data Scientist, Ai Evaluations Platform. Be the first to apply!
Related searches
- senior data scientist Winnipeg, MB
- junior data scientist Winnipeg, MB
- data scientist no experience Winnipeg, MB
- scientifique de données Winnipeg, MB
- data scientist Winnipeg, MB
- environmental data scientist Winnipeg, MB
- data governance Winnipeg, MB
- data reviewer Winnipeg, MB
- ntt data Winnipeg, MB
- clinical data coordinator remote Winnipeg, MB

