Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Agentic Ai Evaluation Engineer

Equifax Canada

Build reliable autonomous workflows at Equifax in Toronto using Python, LLM evaluation, and observability for production AI systems.This newly created full-time role focuses on Agentic AI Evaluation and Tuning Engineering. You will test multi-agent architectures, audit decision-making logic, and optimize tool use. The work connects automated testing, prompt tuning, orchestration frameworks, and production guardrails.Key Responsibilities:
Curate Golden Sets of documents, queries, and agent trajectoriesAutomate continuous evaluation pipelines for accuracy, latency, and token spendAudit multi-step reasoning and identify deviations from intended logicOptimize prompts, context windows, and few-shot examplesMonitor deployed agents for drift, injections, loops, and hallucinationsRequirements:
Three-plus years in quality engineering, test automation, or data engineeringRecent focus on LLM testing, prompt tuning, or orchestrationHands-on LangGraph, ADKs, or specialized internal SDK experiencePython-heavy automated testing and tracing experienceUnderstanding of JSON schemas, function calling, and structured outputsApply Agentic AI evaluation, Python automation, and observability expertise to strengthen Equifax autonomous systems in Toronto.
#J-18808-Ljbffr
Vacancy posted 25 minutes ago
Similar jobs that could be interesting for youBased on the Agentic Ai Evaluation Engineer in Winnipeg, MB vacancy
  •  ...reliable autonomous workflows at Equifax through LLM evaluation, Python automation, and multi-agent system tuning. This...  ...newly created position focuses on enterprise-grade AI.As an Agentic AI Evaluation and Tuning Engineer, you will test and optimize intelligent systems navigating... 
    Suggested

    Equifax

    Winnipeg, MB
    23 minutes ago
  •  ...will join a world-class team of engineers and data scientists from Facebook...  ...building production-grade AI/ML systems. Experience architecting complex agentic systems, including orchestration...  ...experience designing, training, evaluating, and improving ML models. Experience... 
    Suggested

    Ample Insight Inc.

    Winnipeg, MB
    21 minutes ago
  •  ...is a next-generation data and AI consulting firm dedicated to accelerating...  ..., AI technologies, data engineering, and digital transformation....  ...is looking for a Senior AI & Agentic Engineer: a full-stack...  ...pipeline feeding it, and the evaluations proving it works. This role... 
    Suggested

    FDE Times

    Winnipeg, MB
    22 minutes ago
  •  ...all applications and next steps. Our partner is looking for an AI Engineer (Agentic Systems) based in Canada. Join a high-performing team of...  ...and machine learning systems using Python. Design, train, evaluate, and continuously improve machine learning and deep learning... 
    Suggested

    Jobgether

    Winnipeg, MB
    29 minutes ago
  • Lead agentic AI architecture at Robots & Pencils, using Python, AWS, and production-focused engineering judgment across end-to-end systems.As a Staff AI Builder, you will lead AI/ML...  ...spans agentic workflows, RAG pipelines, evaluation, observability, and cloud-native services... 
    Suggested

    robotsandpencils

    Winnipeg, MB
    21 minutes ago
  • $100k - $150k per year

    Position Title: AI Engineer (Agentic Workflows) Department: Research & Development Company: Protocase Inc. Location: Waterloo, Ontario...  ...retrieval, tool integration, prompt and context design, and evaluation. You'll think in terms of workflows and outcomes—identifying... 
    Full time
    Seasonal work
    Work at office
    Worldwide
    Shift work
    Day shift

    Protocase Inc.

    Winnipeg, MB
    22 minutes ago
  • We are seeking a Senior Agentic AI Engineer to design, develop, and deploy production-grade AI applications using Large Language Models (LLMs)...  ...solutions, and the ability to troubleshoot complex RAG pipelines, evaluate LLM application quality, and design scalable cloud-based... 

    enableIT

    Winnipeg, MB
    23 minutes ago
  •  ...role. Degree in Computer Science or Engineering. Strong Python skills and experience building production-grade AI/ML systems. Experience architecting complex agentic systems, including orchestration,...  ...experience designing, training, evaluating, and improving ML models.... 

    Ampleinsightinc

    Winnipeg, MB
    23 minutes ago
  • $200k per year

     ...specializing in data, analytics & AI consulting, dedicated to...  ...analysts, data scientists, data engineers, and digital experts are all dedicated...  ...models, LLM applications,and agentic systems - ensuring scalability...  ...- orchestration layers, evaluation frameworks, guardrails, and... 
    Manual labor

    Welcome to the Jungle

    Winnipeg, MB
    24 minutes ago
  • $98k - $138k per year

     ...stronger financial future for the healthcare community. As a Lead Agentic AI Engineer, you will work directly with engineers, architects, security,...  ...secure, scalable production solutions through monitoring, evaluation, optimization, automated delivery practices, and continuous... 
    Permanent employment
    Immediate start

    Urban Ridge Supplies

    Winnipeg, MB
    28 minutes ago
  • Build agentic workflows and personalized AI features as an AI Engineer supporting Superhuman Mail from a remote-flexible Canada arrangement. Apply Python or TypeScript...  ...experiences. You will develop AI agents, evaluation frameworks, and reusable tools. The work includes... 
    Remote work
    Flexible hours

    Jobgether

    Winnipeg, MB
    22 minutes ago
  • AI Engineer / Data Engineer - Agentic AI (Security)AI Engineer / Data Engineer - Agentic AIVille de Québec, QC (Hybride) Temps plein Location: Quebec City, CanadaDans des marchés en rapide évolution, les clients à travers le monde font confiance à Thales. Thales est une... 
    Full time
    Local area
    Worldwide

    Canadian Inclusion

    Winnipeg, MB
    29 minutes ago
  • We're hiring a contract AI engineer who builds, deploys and operates AI agents in production, using leading-edge agentic development practice to ship quickly. The AI Platform team builds AI agents and products for the trucking and fleet compliance space, used by both our... 
    Contract work
    Internship

    Aplin

    Winnipeg, MB
    28 minutes ago
  • Elevate AI advancement at DataAnnotation as an Electrical Engineer, focusing on real-world circuit and systems challenges. Apply your design and debugging skills...  ...realistic engineering problems from practical experience • Evaluate AI model outputs against professional benchmarks •... 

    DataAnnotation

    Winnipeg, MB
    21 minutes ago
  • Lead RAG troubleshooting and LLM evaluation with embeddings, vector search, retrieval optimization, and measurable quality testing.As a Senior Agentic AI Engineer, you will improve how enterprise AI applications retrieve information and generate grounded responses. The role... 

    enableIT

    Winnipeg, MB
    25 minutes ago
  • Build GenAI applications with Python, LLMs, and agentic frameworks at Infosys in Mississauga, Ontario. Design production-ready AI solutions through hands-on engineering.Infosys seeks a hands-on GenAI and Agentic AI Engineer for next-generation AI solutions. You will work with... 

    Infosys Limited

    Winnipeg, MB
    29 minutes ago
  • AI is changing how software gets built. We're looking for a Staff AI Engineer to work out which changes matter for clearer.Io, prove them...  ...spot which developments in agentic AI, commerce protocols and the...  ...approved actions, failures and evaluation. Build working prototypes and... 
    Long term contract
    Work at office
    Remote work
    Worldwide

    CoinGecko

    Winnipeg, MB
    29 minutes ago
  • SST is seeking an experienced Senior Agentic AI – Validation & Automation Engineer to join our engineering team in Markham, Ontario. This position...  ...Familiarity with human-in-the-loop AI systems, agent evaluation, observability, and workflow recovery mechanisms. Experience... 

    Sapphire Stream Technology

    Winnipeg, MB
    20 minutes ago
  • Engineer reliable AI solutions for EY clients in Toronto, focusing on agent orchestration, multi-agent coordination, evaluation, and responsible production delivery. Collaborate across engagements...  ...and maintaining production-grade agentic AI systems in enterprise environments... 

    EY

    Winnipeg, MB
    21 minutes ago
  •  ...hybrid Toronto position. Shape reliable AI for go-to-market teams.As a Senior AI Engineer, you will design, develop, and deploy agentic applications end to end. You will create...  ...recommend next stepsOperate guardrails, evaluations, drift detection, and monitoringIntegrate... 

    Thomson Reuters

    Winnipeg, MB
    28 minutes ago
  • Build agentic systems for Superhuman Mail using backend engineering and applied AI in a remote-flexible model, with Vancouver hybrid availability.As an AI Engineer, you...  ...Mail. You will work with MCP, personalization, evaluation systems, model selection, and production LLM features... 
    Remote work
    Flexible hours

    Coda

    Winnipeg, MB
    25 minutes ago
  • Lead agentic AI engineering for Clearsulting’s Office of the CFO service line, using Python, TypeScript, and secure cloud architecture remotely...  ...infrastructure as codeImplement guardrails, approvals, validation, and evaluation frameworksEstablish reusable architectures, standards, and... 
    Work at office
    Remote work

    Clearsulting

    Winnipeg, MB
    25 minutes ago
  • $99.1k - $132.1k per year

     ...to hear from you.About the Role: The Senior Specialist, Agentic AI Engineering designs, builds, deploys, and supports enterprise-scale AI agents...  ...production-ready AI capabilities.Implement observability, evaluation, and monitoring so agent quality, reliability, cost, and... 
    Long term contract

    Mckesson Corporation

    Winnipeg, MB
    20 minutes ago
  • Build the next generation of AI-enabled insurance workflows, combining...  ...automation, enterprise engineering and human decision-making. The...  ...AI Engineer – Generative AI & Agentic Systems to support an exciting...  ...engineering controls: Build evaluation datasets, automated regression... 
    Internship
    Immediate start

    Capco

    Winnipeg, MB
    23 minutes ago
  •  ...enterprise-grade safeguards.We are AI-first in both product and operations. Generative AI, agentic AI, agent-assisted development,...  ...we ship.This is an early engineering role at the foundation-building...  ...knowledge into reusable specs, evaluations, examples, and context for people... 
    Full time
    Internship

    DataBraid Limited

    Winnipeg, MB
    28 minutes ago
  • Advance Wagepoint’s agentic software factory remotely with AI governance, quality engineering, and platform leadership for payroll technology.As Principal Software Development...  ...improve the development pipelineDefine CI/CD evaluation gates, drift monitoring, and release... 
    Remote work

    Wagepoint

    Winnipeg, MB
    23 minutes ago
  •  ...York, Seoul and Tokyo, is focused exclusively on sourcing, evaluating and managing private markets investments. Northleaf...  ...for insurance and wealth management clients. Position: AI Solutions Engineer, Agentic Automation Reporting to Northleaf's AI Agent Lead, the... 
    Long term contract
    Remote work
    Flexible hours
    2 days per week

    Urban Ridge Supplies

    Winnipeg, MB
    27 minutes ago
  • Hiring in Canada(Remote) | Senior Full Stack Engineer – Agentic AI (Backend-Leaning) What “Full Stack” Means Here Strong backend focus...  ...Collaborate with product and engineering teams Read specs, evaluate evolving AI platforms, and adapt fast ✅ What We’re Looking... 
    Remote work

    BayOne

    Winnipeg, MB
    28 minutes ago
  • First West Credit Union is seeking an AI Engineer to design, build, and deliver high-quality AI solutions across the organization. The role blends software engineering with modern AI practices to improve user experience, efficiency, and decision-making. You will design AI components... 

    Tru Cooperative Bank

    Winnipeg, MB
    29 minutes ago
  •  ...digital product studio composed of 180+ senior engineers and designers with backgrounds from...  ...within Lazer focused on building robust, AI-powered full-stack applications. We partner...  ...products to life, leveraging the latest in AI, agentic workflows, and scalable infrastructure.... 
    Internship
    Remote work
    Worldwide

    Lazer Technologies

    Winnipeg, MB
    26 minutes ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Agentic Ai Evaluation Engineer. Be the first to apply!