Staff Data Scientist, Ai Evaluations Platform
ODAIA
Job DescriptionRBC’s AI Group is building trusted AI capabilities for the enterprise, and evaluation is one of the core controls that makes that possible. As Staff Data Scientist, AI Evaluations, you will be a senior technical leader in the data science function responsible for how RBC measures model and agent quality, safety, risk, and performance. You will design, build, and continuously improve evaluation datasets, sourcing methods, LLM judge design, deterministic scorers, human evaluation protocols, quality benchmarks, and measurement frameworks that help AI systems move from experimentation to production with evidence and control, while providing technical guidance and mentorship to other data scientists on the team.
What will you do?
Design and drive advanced model and agent evaluation methodologies, including evaluation datasets, rubrics, LLM-as-judge methods, deterministic scorers, human evaluation, and measurement frameworks, providing technical guidance to junior team members.
Define evaluation science standards that translate model risk, responsible AI, product quality, safety, and business expectations into measurable criteria, repeatable methods, and clear evidence.
Own the end-to-end lifecycle for evaluation datasets and scorecards, including sourcing, curation, validation, quality checks, versioning, lineage, reuse, and ongoing improvement.
Design scalable evaluation approaches for generative AI and agentic systems, including task-level, workflow-level, trajectory-level, and runtime evaluation methods.
Partner with AI research, platform engineering, product, risk, governance, and business teams to embed evaluations into build, release, certification, monitoring, and recertification workflows.
Establish human evaluation and review protocols that produce reliable labels, reviewer guidance, adjudication processes, quality controls, and audit-ready evidence.
Measure and improve scorer accuracy, calibration, robustness, failure-mode coverage, and explainability across automated and human evaluation approaches.
Provide clear technical leadership, executive-ready communication, and mentorship to junior data scientists to help RBC scale trusted AI with speed, rigor, and control.
In this role, you will communicate and interact frequently with RBC partners and/or employees located across Canada and/or worldwide
What do you need to succeed?Must Have
8+ years of experience in data science, applied machine learning, AI evaluation, ML quality, or a related technical field, including experience providing technical leadership and mentorship within a team.
Strong experience designing evaluation frameworks for ML, generative AI, or agentic AI systems, including metrics, datasets, benchmarks, rubrics, and quality measurement.
Practical experience with LLM evaluation methods such as LLM-as-judge, deterministic scoring, human evaluation, hallucination assessment, factuality assessment, safety evaluation, or model quality benchmarking.
Strong technical foundation in data science, statistics, machine learning, experimentation, data curation, Python, SQL, and modern AI/ML development practices.
Proven ability to translate governance, model risk, responsible AI, and business requirements into measurable controls, repeatable evaluation processes, and decision-ready evidence.
Strong communication and stakeholder management skills, with the ability to influence senior leaders across research, engineering, product, governance, risk, and business teams.
Nice to Have
Experience evaluating agentic AI systems, tool‑calling workflows, multi‑step reasoning, runtime traces, trajectory scoring, or workflow‑level performance.
Experience in financial services, regulated AI, model risk management, responsible AI, enterprise governance, or audit‑ready evidence processes.
Familiarity with tools and platforms such as MLflow, Langfuse, LangSmith, OpenTelemetry, Grafana, CI/CD pipelines, or comparable evaluation and observability tooling.
Publications, patents, open‑source contributions, or industry work related to AI evaluation, ML quality, AI safety, applied research, or responsible AI.
Ph.D. or Master’s degree in a quantitative or technical field, such as Applied Mathematics, Engineering, Computer Science or a related computational discipline.
What’s in it for you?We thrive on the challenge to be our best, progressive thinking to keep growing, and working together to deliver trusted advice to help our clients thrive and communities prosper. We care about each other, reaching our potential, making a difference to our communities, and achieving success that is mutual.
A comprehensive Total Rewards Program including bonuses and flexible benefits, competitive compensation, commissions, and stock where applicable
Leaders who support your development through coaching and managing opportunities
Ability to make a difference and lasting impact
Work in a dynamic, collaborative, progressive, and high‑performing team
A world‑class training program in financial services
Opportunities to do challenging work.
Job SkillsBig Data Analytics, Critical Thinking, Decision Making, Industry Knowledge, Machine Learning (ML), Results‑Oriented, Software Engineering, Software Product Design
Additional Job DetailsAddress: 6666 RUE ST URBAIN, FLOOR 3:MONTRÉAL
City: Montréal
Country: Canada
Work hours/week: 37.5
Employment Type: Full time
Platform: TECHNOLOGY AND OPERATIONS
Job Type: Regular
Pay Type: Salaried
Posted Date: 2026-10-09
Application Deadline: 2026-10-31
Note: Applications will be accepted until 11:59 PM on the day prior to the application deadline date above
Our Employment Opportunities
At RBC, we are guided by living shared values of Client First, Integrity, Collaboration, Respect and Excellence and winning together as One RBC. We believe an inclusive workplace that has diverse perspectives is core to our continued growth as one of the largest and most successful banks in the world. Maintaining a workplace where our employees feel supported to perform at their best, effectively collaborate, drive innovation, and grow professionally helps to bring our Purpose to life and create value for our clients and communities. RBC strives to deliver this through policies and programs intended to foster a workplace based on respect, belonging and opportunity for all.
#J-18808-Ljbffr
What will you do?
Design and drive advanced model and agent evaluation methodologies, including evaluation datasets, rubrics, LLM-as-judge methods, deterministic scorers, human evaluation, and measurement frameworks, providing technical guidance to junior team members.
Define evaluation science standards that translate model risk, responsible AI, product quality, safety, and business expectations into measurable criteria, repeatable methods, and clear evidence.
Own the end-to-end lifecycle for evaluation datasets and scorecards, including sourcing, curation, validation, quality checks, versioning, lineage, reuse, and ongoing improvement.
Design scalable evaluation approaches for generative AI and agentic systems, including task-level, workflow-level, trajectory-level, and runtime evaluation methods.
Partner with AI research, platform engineering, product, risk, governance, and business teams to embed evaluations into build, release, certification, monitoring, and recertification workflows.
Establish human evaluation and review protocols that produce reliable labels, reviewer guidance, adjudication processes, quality controls, and audit-ready evidence.
Measure and improve scorer accuracy, calibration, robustness, failure-mode coverage, and explainability across automated and human evaluation approaches.
Provide clear technical leadership, executive-ready communication, and mentorship to junior data scientists to help RBC scale trusted AI with speed, rigor, and control.
In this role, you will communicate and interact frequently with RBC partners and/or employees located across Canada and/or worldwide
What do you need to succeed?Must Have
8+ years of experience in data science, applied machine learning, AI evaluation, ML quality, or a related technical field, including experience providing technical leadership and mentorship within a team.
Strong experience designing evaluation frameworks for ML, generative AI, or agentic AI systems, including metrics, datasets, benchmarks, rubrics, and quality measurement.
Practical experience with LLM evaluation methods such as LLM-as-judge, deterministic scoring, human evaluation, hallucination assessment, factuality assessment, safety evaluation, or model quality benchmarking.
Strong technical foundation in data science, statistics, machine learning, experimentation, data curation, Python, SQL, and modern AI/ML development practices.
Proven ability to translate governance, model risk, responsible AI, and business requirements into measurable controls, repeatable evaluation processes, and decision-ready evidence.
Strong communication and stakeholder management skills, with the ability to influence senior leaders across research, engineering, product, governance, risk, and business teams.
Nice to Have
Experience evaluating agentic AI systems, tool‑calling workflows, multi‑step reasoning, runtime traces, trajectory scoring, or workflow‑level performance.
Experience in financial services, regulated AI, model risk management, responsible AI, enterprise governance, or audit‑ready evidence processes.
Familiarity with tools and platforms such as MLflow, Langfuse, LangSmith, OpenTelemetry, Grafana, CI/CD pipelines, or comparable evaluation and observability tooling.
Publications, patents, open‑source contributions, or industry work related to AI evaluation, ML quality, AI safety, applied research, or responsible AI.
Ph.D. or Master’s degree in a quantitative or technical field, such as Applied Mathematics, Engineering, Computer Science or a related computational discipline.
What’s in it for you?We thrive on the challenge to be our best, progressive thinking to keep growing, and working together to deliver trusted advice to help our clients thrive and communities prosper. We care about each other, reaching our potential, making a difference to our communities, and achieving success that is mutual.
A comprehensive Total Rewards Program including bonuses and flexible benefits, competitive compensation, commissions, and stock where applicable
Leaders who support your development through coaching and managing opportunities
Ability to make a difference and lasting impact
Work in a dynamic, collaborative, progressive, and high‑performing team
A world‑class training program in financial services
Opportunities to do challenging work.
Job SkillsBig Data Analytics, Critical Thinking, Decision Making, Industry Knowledge, Machine Learning (ML), Results‑Oriented, Software Engineering, Software Product Design
Additional Job DetailsAddress: 6666 RUE ST URBAIN, FLOOR 3:MONTRÉAL
City: Montréal
Country: Canada
Work hours/week: 37.5
Employment Type: Full time
Platform: TECHNOLOGY AND OPERATIONS
Job Type: Regular
Pay Type: Salaried
Posted Date: 2026-10-09
Application Deadline: 2026-10-31
Note: Applications will be accepted until 11:59 PM on the day prior to the application deadline date above
Our Employment Opportunities
At RBC, we are guided by living shared values of Client First, Integrity, Collaboration, Respect and Excellence and winning together as One RBC. We believe an inclusive workplace that has diverse perspectives is core to our continued growth as one of the largest and most successful banks in the world. Maintaining a workplace where our employees feel supported to perform at their best, effectively collaborate, drive innovation, and grow professionally helps to bring our Purpose to life and create value for our clients and communities. RBC strives to deliver this through policies and programs intended to foster a workplace based on respect, belonging and opportunity for all.
#J-18808-Ljbffr
Vacancy posted 52 minutes ago
Similar jobs that could be interesting for youBased on the Staff Data Scientist, Ai Evaluations Platform in Winnipeg, MB vacancy
- Meet Fetch AI & DataAI & Data at Fetch sit at the center of how we understand our business, make decisions... ...learning, experimentation, and data platforms, all working together to turn data into... ...financial planning. We are seeking a Staff Data Scientist to serve as the company‑...SuggestedFull timeRemote work
- ...developing a diversified product platform aimed at addressing... ...it.You will be one of the early data science hires shaping how Purpose... ...years — and increasingly, how our AI systems understand and serve the... ...You're an engineer as much as a scientist — you own the work from raw event...SuggestedInternshipWork at officeWork from homeFlexible hours
- Build Meeru AI’s data foundation with expert SQL and Python for production pipelines, connectors, and curated models across enterprise finance systems.As a Staff Data Engineer, you will own major data-platform subsystems end to end. You will transform warehouse data, integrate...Suggested
- Lenovo is seeking an AI Data Intelligence Staff Researcher to develop intelligent, efficient, and reliable AI solutions for Hybrid AI computing and data infrastructure. In this role, you will use machine learning, deep learning, data science, and software engineering to improve...Suggested
- Cohere is seeking a Senior Research Scientist focused on AI evaluation methods to support our innovative foundation models. This remote or on-site role... ...in building AI prototypes and evaluations • Proficiency in data review for quality assurance • Commitment to rigorous...SuggestedRemote work
$80k - $175k per year
Staff Data Scientist Application Deadline: 11/14/2026Address: 33 Dundas Street WestJob Family Group: Data Analytics & ReportingThis is a hybrid... ...at scale.Experience developing machine learning models, and evaluating emerging technologies and tools to deliver innovative data-...Contract workPart time- Apply data science and Python expertise to evaluate coding agents through reproducible environments, testing, and... ...verification.This Competitive Data Scientist role focuses on building reliable... ...judgment, practical programming, and evaluation expertise to strengthen coding-...
- ...Pinterest is seeking a Senior Data Scientist to shape the future of people-facing and business-facing products. You will apply quantitative modeling... .... You will build and prototype analysis pipelines, leverage AI to accelerate execution, and communicate insights to cross-...
$200k - $250k per year
...people who want to help us build something truly special. Staff Data Scientist, GrowthWe’re looking for an experienced Data Scientist to join... ...Experience with experimentation and product analytics platforms such as Posthog, Statsig, Amplitude Experience working with...Long term contractWork at officeRemote work- About the Ads Data Science TeamThe Ads Data Science team at Reddit... ...in the areas of ads and platform measurement, campaign and creative... ...collaborative team of passionate data scientists and engineers who push the... ...for a highly experienced Staff Data Scientist to advance the...RemplacementFlexible hours
- ...improve customer experience and trust for buyers & sellers on the platform.Define & Monitor Metrics: Build and develop KPIs to track the... ...new processes and deploying new technologiesHas strong analytical/data science skills to dive into data and identify root causesBuilds models...Long term contractImmediate start
- ...the world have invested in for years. We're looking for a Staff Data Scientist who's an expert in building and shipping production pricing... ...and measurable outcomes Set the standard for model evaluation, validation, and monitoring — including knowing when CV metrics...Local area
$145.9k per year
...If so, this might be the role for you! We're looking for a Staff Data Scientist, Applied Machine Learning (ML) to join our growing Data Science... ...and systems for automated decisioning. You will train and evaluate models using techniques such as graph representation learning...Long term contractInternshipWork at officeLocal area$137.3k - $171.6k per year
The Data and AI Platform team at Rivian supports developing and deploying Generative AI systems. We operate highly scalable infrastructure for serving and orchestrating LLMs, Vector Stores, and agents. As a staff engineer, you will be crucial in designing, implementing, and...Hourly payLocal area- ...signals, identity, and privacy.Reddit seeks a highly experienced Staff Data Scientist for its Ads Data Science team. You will own a critical... ...identity resolutionDesign privacy-conscious links between on-platform and off-platform actionsAdvance Brand and Conversion Lift experimentation...Remote work
$180k - $200k per year
...Job Title: Staff Data Scientist (AI & Machine Learning) Location: Vancouver, BC – 3 days a week onsite Job ID: 87597 Overview As a Staff Data Scientist, you are a technical visionary and a strategic leader within our AI organization. While a Senior Data Scientist excels...3 days per week- Build reliable data platforms with Python and SQL at Function Health, working remotely from Canada on infrastructure for AI-powered healthcare.As a Staff Software Engineer, you will shape Function Health’s data platform infrastructure. Your work will support reliable contracts...Local areaRemote work
$170k per year
Databricks is on a mission to simplify and democratize data and AI — from making the next mode of transportation a reality to accelerating... ...building and running the world's best data and AI infrastructure platform so our customers can use deep data insights to improve their...Summer workWorldwide$110.5k - $143k per year
...Develop and execute comprehensive data governance strategy to ensure... ...and educate data services staff while promoting growth and excellence... ...with the line of business to evaluate KPI measurement needs. Communicate... ...vacancy. Hydro One uses AI tools to assist in the...- Data Science is at the heart of Lyft’s products and decision-making. As a member of the Rider... ...the world’s best transportation. Data Scientists take on a variety of problems ranging from... ...and impactful problems in ridesharing.As a Staff Data Scientist in the Rider team, you will...
- A career as a Senior AI Data scientist in theHyper Automation – Cross-sector Solutions, Acceleration and Data team atNational Bank means serving... ...impact on the performance andevolution of an enterprise platform thanks to your expertise in applied datascience, model evaluation...Flexible hours
$104k per year
...Company OverviewRobots & Pencils is an applied AI engineering firm building the next... ...Position OverviewWe’re looking for a Staff Platform Engineer to define and lead platform engineering... ...and platform challenges Evaluate emerging tools and technologies, recommending...Local areaRemote work- Become an integral part of a machine learning squad as a Data Scientist focusing on innovative AI solutions, including ML and Generative AI. Utilize your... ...ensuring continuous improvement through monitoring and evaluation strategies. Key Responsibilities: • Develop and...
- ...Overview Function Health is building an AI operating system for health, offering... ...lab testing, imaging, longitudinal data, and guidance from AI and clinical... ...and is scaling its integrated health platform.Role Function is hiring a Staff Software Engineer for its data...Full timeContract workLocal areaRemote workFlexible hours
- ...are we? Cohere is the leading security-first enterprise AI company, building cutting-edge foundation AI models and end... .... Join us! Why this role? As a Member of Technical Staff in Data Analysis and Evaluation, you will play a pivotal role in ensuring the quality, reliability...Full timeWork at officeLocal areaRemote workHome officeFlexible hours
- ...Shape the data foundation behind Coveo’s next generation of innovation... ...of developers more effective? As a Staff Software Developer on Coveo’s Data Platform team, you’ll tackle complex, high-impact... ...the ability to navigate ambiguity, evaluate trade-offs and independently drive...Long term contractImmediate start
- Shape Jobber’s data platform through scalable pipelines, technical leadership, and hands-on engineering across batch and real-time systems.As a Staff Data Engineer on the Data Platform Team, you will shape foundational data components and guide engineering excellence. You will...
- Build agentic AI solutions at Cogeco using autonomous reasoning and tool calling. Integrate scalable models into analytics products and business workflows.As a Data Scientist, you will create dynamic, interpretable models and integrate analytic approaches with engineers, analysts...
$171k - $230k per year
...related spend in a single system. Combined with industry leading AI, the Motive platform gives you complete visibility and control, and significantly... ..., food and beverage, retail, and the public sector. As a Staff Engineer on the Developer Productivity team, you will...Long term contractTemporary workLocal area- ...initiatives. This role will own the platform layer that takes AI-generated code from the Harness and... ...AutoGen to build multi-agent workflows. Data & Tool Integration: Ability to design... ...Knowledge, LLM Architecture, LLM Evaluation, LLM Guardrails, LLM Orchestration, Machine...Full timeFlexible hours
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Staff Data Scientist, Ai Evaluations Platform. Be the first to apply!
Related searches
- senior data scientist Winnipeg, MB
- junior data scientist Winnipeg, MB
- data scientist no experience Winnipeg, MB
- scientifique de données Winnipeg, MB
- data scientist Winnipeg, MB
- environmental data scientist Winnipeg, MB
- data governance Winnipeg, MB
- data reviewer Winnipeg, MB
- ntt data Winnipeg, MB
- clinical data coordinator remote Winnipeg, MB

