Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Staff Data Scientist, Ai Evaluations Platform

ODAIA

Job DescriptionRBC’s AI Group is building trusted AI capabilities for the enterprise, and evaluation is one of the core controls that makes that possible. As Staff Data Scientist, AI Evaluations, you will be a senior technical leader in the data science function responsible for how RBC measures model and agent quality, safety, risk, and performance. You will design, build, and continuously improve evaluation datasets, sourcing methods, LLM judge design, deterministic scorers, human evaluation protocols, quality benchmarks, and measurement frameworks that help AI systems move from experimentation to production with evidence and control, while providing technical guidance and mentorship to other data scientists on the team.
What will you do?
Design and drive advanced model and agent evaluation methodologies, including evaluation datasets, rubrics, LLM-as-judge methods, deterministic scorers, human evaluation, and measurement frameworks, providing technical guidance to junior team members.
Define evaluation science standards that translate model risk, responsible AI, product quality, safety, and business expectations into measurable criteria, repeatable methods, and clear evidence.
Own the end-to-end lifecycle for evaluation datasets and scorecards, including sourcing, curation, validation, quality checks, versioning, lineage, reuse, and ongoing improvement.
Design scalable evaluation approaches for generative AI and agentic systems, including task-level, workflow-level, trajectory-level, and runtime evaluation methods.
Partner with AI research, platform engineering, product, risk, governance, and business teams to embed evaluations into build, release, certification, monitoring, and recertification workflows.
Establish human evaluation and review protocols that produce reliable labels, reviewer guidance, adjudication processes, quality controls, and audit-ready evidence.
Measure and improve scorer accuracy, calibration, robustness, failure-mode coverage, and explainability across automated and human evaluation approaches.
Provide clear technical leadership, executive-ready communication, and mentorship to junior data scientists to help RBC scale trusted AI with speed, rigor, and control.
In this role, you will communicate and interact frequently with RBC partners and/or employees located across Canada and/or worldwide
What do you need to succeed?Must Have
8+ years of experience in data science, applied machine learning, AI evaluation, ML quality, or a related technical field, including experience providing technical leadership and mentorship within a team.
Strong experience designing evaluation frameworks for ML, generative AI, or agentic AI systems, including metrics, datasets, benchmarks, rubrics, and quality measurement.
Practical experience with LLM evaluation methods such as LLM-as-judge, deterministic scoring, human evaluation, hallucination assessment, factuality assessment, safety evaluation, or model quality benchmarking.
Strong technical foundation in data science, statistics, machine learning, experimentation, data curation, Python, SQL, and modern AI/ML development practices.
Proven ability to translate governance, model risk, responsible AI, and business requirements into measurable controls, repeatable evaluation processes, and decision-ready evidence.
Strong communication and stakeholder management skills, with the ability to influence senior leaders across research, engineering, product, governance, risk, and business teams.
Nice to Have
Experience evaluating agentic AI systems, tool‑calling workflows, multi‑step reasoning, runtime traces, trajectory scoring, or workflow‑level performance.
Experience in financial services, regulated AI, model risk management, responsible AI, enterprise governance, or audit‑ready evidence processes.
Familiarity with tools and platforms such as MLflow, Langfuse, LangSmith, OpenTelemetry, Grafana, CI/CD pipelines, or comparable evaluation and observability tooling.
Publications, patents, open‑source contributions, or industry work related to AI evaluation, ML quality, AI safety, applied research, or responsible AI.
Ph.D. or Master’s degree in a quantitative or technical field, such as Applied Mathematics, Engineering, Computer Science or a related computational discipline.
What’s in it for you?We thrive on the challenge to be our best, progressive thinking to keep growing, and working together to deliver trusted advice to help our clients thrive and communities prosper. We care about each other, reaching our potential, making a difference to our communities, and achieving success that is mutual.
A comprehensive Total Rewards Program including bonuses and flexible benefits, competitive compensation, commissions, and stock where applicable
Leaders who support your development through coaching and managing opportunities
Ability to make a difference and lasting impact
Work in a dynamic, collaborative, progressive, and high‑performing team
A world‑class training program in financial services
Opportunities to do challenging work.
Job SkillsBig Data Analytics, Critical Thinking, Decision Making, Industry Knowledge, Machine Learning (ML), Results‑Oriented, Software Engineering, Software Product Design
Additional Job DetailsAddress: 6666 RUE ST URBAIN, FLOOR 3:MONTRÉAL
City: Montréal
Country: Canada
Work hours/week: 37.5
Employment Type: Full time
Platform: TECHNOLOGY AND OPERATIONS
Job Type: Regular
Pay Type: Salaried
Posted Date: 2026-10-09
Application Deadline: 2026-10-31
Note: Applications will be accepted until 11:59 PM on the day prior to the application deadline date above
Our Employment Opportunities
At RBC, we are guided by living shared values of Client First, Integrity, Collaboration, Respect and Excellence and winning together as One RBC. We believe an inclusive workplace that has diverse perspectives is core to our continued growth as one of the largest and most successful banks in the world. Maintaining a workplace where our employees feel supported to perform at their best, effectively collaborate, drive innovation, and grow professionally helps to bring our Purpose to life and create value for our clients and communities. RBC strives to deliver this through policies and programs intended to foster a workplace based on respect, belonging and opportunity for all.
#J-18808-Ljbffr
Vacancy posted 1 hour ago
Similar jobs that could be interesting for youBased on the Staff Data Scientist, Ai Evaluations Platform in Winnipeg, MB vacancy
  • Meet Fetch AI & DataAI & Data at Fetch sit at the center of how we understand our business, make decisions...  ...learning, experimentation, and data platforms, all working together to turn data into...  ...financial planning. We are seeking a Staff Data Scientist to serve as the company‑... 
    Suggested
    Full time
    Remote work

    Fetch

    Winnipeg, MB
    1 hour ago
  •  ...developing a diversified product platform aimed at addressing...  ...it.You will be one of the early data science hires shaping how Purpose...  ...years — and increasingly, how our AI systems understand and serve the...  ...You're an engineer as much as a scientist — you own the work from raw event... 
    Suggested
    Internship
    Work at office
    Work from home
    Flexible hours

    Purpose Unlimited

    Winnipeg, MB
    1 hour ago
  • Build Meeru AI’s data foundation with expert SQL and Python for production pipelines, connectors, and curated models across enterprise finance systems.As a Staff Data Engineer, you will own major data-platform subsystems end to end. You will transform warehouse data, integrate... 
    Suggested

    MeeruAI

    Winnipeg, MB
    1 hour ago
  • Cohere is seeking a Senior Research Scientist focused on AI evaluation methods to support our innovative foundation models. This remote or on-site role...  ...in building AI prototypes and evaluations • Proficiency in data review for quality assurance • Commitment to rigorous... 
    Suggested
    Remote work

    Cohere Inc.

    Winnipeg, MB
    1 hour ago
  • Lenovo is seeking an AI Data Intelligence Staff Researcher to develop intelligent, efficient, and reliable AI solutions for Hybrid AI computing and data infrastructure. In this role, you will use machine learning, deep learning, data science, and software engineering to improve... 
    Suggested

    Lenovo Group

    Winnipeg, MB
    1 hour ago
  • $80k - $175k per year

    Staff Data Scientist Application Deadline: 11/14/2026Address: 33 Dundas Street WestJob Family Group: Data Analytics & ReportingThis is a hybrid...  ...at scale.Experience developing machine learning models, and evaluating emerging technologies and tools to deliver innovative data-... 
    Contract work
    Part time

    Bank of Montreal

    Winnipeg, MB
    1 hour ago
  • Apply data science and Python expertise to evaluate coding agents through reproducible environments, testing, and...  ...verification.This Competitive Data Scientist role focuses on building reliable...  ...judgment, practical programming, and evaluation expertise to strengthen coding-... 

    Great Value Hiring

    Winnipeg, MB
    1 hour ago
  •  ...Pinterest is seeking a Senior Data Scientist to shape the future of people-facing and business-facing products. You will apply quantitative modeling...  .... You will build and prototype analysis pipelines, leverage AI to accelerate execution, and communicate insights to cross-... 

    Pinterest

    Winnipeg, MB
    7 days ago
  • $200k - $250k per year

     ...people who want to help us build something truly special. Staff Data Scientist, GrowthWe’re looking for an experienced Data Scientist to join...  ...Experience with experimentation and product analytics platforms such as Posthog, Statsig, Amplitude Experience working with... 
    Long term contract
    Work at office
    Remote work

    Shakepay

    Winnipeg, MB
    1 hour ago
  • About the Ads Data Science TeamThe Ads Data Science team at Reddit...  ...in the areas of ads and platform measurement, campaign and creative...  ...collaborative team of passionate data scientists and engineers who push the...  ...for a highly experienced Staff Data Scientist to advance the... 
    Remplacement
    Flexible hours

    Reddit, Inc.

    Winnipeg, MB
    1 hour ago
  • $145.9k per year

     ...If so, this might be the role for you! We're looking for a Staff Data Scientist, Applied Machine Learning (ML) to join our growing Data Science...  ...and systems for automated decisioning. You will train and evaluate models using techniques such as graph representation learning... 
    Long term contract
    Internship
    Work at office
    Local area

    Jobber

    Winnipeg, MB
    1 hour ago
  • $137.3k - $171.6k per year

    The Data and AI Platform team at Rivian supports developing and deploying Generative AI systems. We operate highly scalable infrastructure for serving and orchestrating LLMs, Vector Stores, and agents. As a staff engineer, you will be crucial in designing, implementing, and... 
    Hourly pay
    Local area

    Rivian Automotive, Inc.

    Winnipeg, MB
    1 hour ago
  •  ...improve customer experience and trust for buyers & sellers on the platform.Define & Monitor Metrics: Build and develop KPIs to track the...  ...new processes and deploying new technologiesHas strong analytical/data science skills to dive into data and identify root causesBuilds models... 
    Long term contract
    Immediate start

    eBay

    Winnipeg, MB
    1 hour ago
  •  ...the world have invested in for years. We're looking for a Staff Data Scientist who's an expert in building and shipping production pricing...  ...and measurable outcomes Set the standard for model evaluation, validation, and monitoring — including knowing when CV metrics... 
    Local area

    CSC Generation

    Winnipeg, MB
    1 hour ago
  • $110.5k - $143k per year

     ...Develop and execute comprehensive data governance strategy to ensure...  ...and educate data services staff while promoting growth and excellence...  ...with the line of business to evaluate KPI measurement needs. Communicate...  ...vacancy. Hydro One uses AI tools to assist in the... 

    Hydro One Networks Inc

    Winnipeg, MB
    1 hour ago
  • Build reliable data platforms with Python and SQL at Function Health, working remotely from Canada on infrastructure for AI-powered healthcare.As a Staff Software Engineer, you will shape Function Health’s data platform infrastructure. Your work will support reliable contracts... 
    Local area
    Remote work

    Skillhub Solutions Inc.

    Winnipeg, MB
    1 hour ago
  • $170k per year

    Databricks is on a mission to simplify and democratize data and AI — from making the next mode of transportation a reality to accelerating...  ...building and running the world's best data and AI infrastructure platform so our customers can use deep data insights to improve their... 
    Summer work
    Worldwide

    Menlo Ventures

    Winnipeg, MB
    1 hour ago
  •  ...signals, identity, and privacy.Reddit seeks a highly experienced Staff Data Scientist for its Ads Data Science team. You will own a critical...  ...identity resolutionDesign privacy-conscious links between on-platform and off-platform actionsAdvance Brand and Conversion Lift experimentation... 
    Remote work

    Reddit

    Winnipeg, MB
    1 hour ago
  • $180k - $200k per year

     ...Job Title: Staff Data Scientist (AI & Machine Learning) Location: Vancouver, BC – 3 days a week onsite Job ID: 87597 Overview As a Staff Data Scientist, you are a technical visionary and a strategic leader within our AI organization. While a Senior Data Scientist excels... 
    3 days per week

    TEEMA Group

    Winnipeg, MB
    7 days ago
  • A career as a Senior AI Data scientist in theHyper Automation – Cross-sector Solutions, Acceleration and Data team atNational Bank means serving...  ...impact on the performance andevolution of an enterprise platform thanks to your expertise in applied datascience, model evaluation... 
    Flexible hours

    National Bank of Canada

    Winnipeg, MB
    1 hour ago
  • Become an integral part of a machine learning squad as a Data Scientist focusing on innovative AI solutions, including ML and Generative AI. Utilize your...  ...ensuring continuous improvement through monitoring and evaluation strategies. Key Responsibilities: • Develop and... 

    Compunnel Inc.

    Winnipeg, MB
    1 hour ago
  •  ...are we? Cohere is the leading security-first enterprise AI company, building cutting-edge foundation AI models and end...  .... Join us! Why this role? As a Member of Technical Staff in Data Analysis and Evaluation, you will play a pivotal role in ensuring the quality, reliability... 
    Full time
    Work at office
    Local area
    Remote work
    Home office
    Flexible hours

    Cohere

    Winnipeg, MB
    1 hour ago
  • Shape Jobber’s data platform through scalable pipelines, technical leadership, and hands-on engineering across batch and real-time systems.As a Staff Data Engineer on the Data Platform Team, you will shape foundational data components and guide engineering excellence. You will... 

    Devconnectplatform

    Winnipeg, MB
    1 hour ago
  •  ...Overview Function Health is building an AI operating system for health, offering...  ...lab testing, imaging, longitudinal data, and guidance from AI and clinical...  ...and is scaling its integrated health platform.Role Function is hiring a Staff Software Engineer for its data... 
    Full time
    Contract work
    Local area
    Remote work
    Flexible hours

    Skillhub Solutions Inc.

    Winnipeg, MB
    1 hour ago
  •  ...Shape the data foundation behind Coveo’s next generation of innovation...  ...of developers more effective? As a Staff Software Developer on Coveo’s Data Platform team, you’ll tackle complex, high-impact...  ...the ability to navigate ambiguity, evaluate trade-offs and independently drive... 
    Long term contract
    Immediate start

    Coveo

    Winnipeg, MB
    1 hour ago
  • Shape Jobber’s data platform through scalable pipelines, AWS expertise, and technical leadership across teams. This is a strategic, hands-on engineering role.As a Staff Data Engineer on Jobber’s Data Platform Team, you will shape data integration architecture and delivery.... 

    Avenue.io Inc.

    Winnipeg, MB
    1 hour ago
  • Lead agentic AI architecture and delivery as a Principal Data Scientist with Robots and Pencils, working remotely across Canada. Shape enterprise-scale systems.This...  ...technical vision, reference architecture, and evaluation standards while guiding large teams through complex... 
    Remote work

    F5 Remote Jobs

    Winnipeg, MB
    1 hour ago
  • Shape Instacart’s Ads data platform remotely with Spark, SQL, and distributed systems expertise...  ...for critical advertising workflows.As a Staff Software Engineer, you will advance a...  ...testing. The work serves advertisers, data scientists, machine learning engineers, and sales... 
    Remote work

    ContentBuffer

    Winnipeg, MB
    1 hour ago
  • $88k - $132k per year

     ...$132,000 / yearJob Summary EY Canada is seeking a Senior Data Scientist for its AI Centre of Excellence within the Office of the Chief Technology...  ...and research applications.Implement, fine-tune, and evaluate large language model solutions, including generative AI, natural... 
    Full time
    Work at office

    Skillhub Solutions Inc.

    Winnipeg, MB
    1 hour ago
  • Build agentic AI solutions at Cogeco using autonomous reasoning and tool calling. Integrate scalable models into analytics products and business workflows.As a Data Scientist, you will create dynamic, interpretable models and integrate analytic approaches with engineers, analysts... 

    Y-Axis

    Winnipeg, MB
    1 hour ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Staff Data Scientist, Ai Evaluations Platform. Be the first to apply!