Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Agentic Ai Evaluation Engineer

Equifax Inc.

Build reliable autonomous AI at Equifax through LLM evaluation, Python automation, and multi-agent workflow testing. This newly created position focuses on production reliability.As an Agentic AI Evaluation & Tuning Engineer, you will connect LLM capabilities with dependable autonomous execution. You will curate Golden Sets, automate evaluation pipelines, audit reasoning trajectories, and tune prompts. Your work will also address tool calling, RAG pipelines, hallucinations, and guardrails. Collaboration with AI Solution Leads and AI Agent Developers supports robust agent components.Key Responsibilities:
Curate documents, queries, and expected agent trajectoriesAutomate evaluations for accuracy, latency, token spend, and reliabilityAudit multi-step reasoning and identify logic deviationsOptimize prompts, context windows, and few-shot examplesMonitor drift, injections, loops, and hallucinationsRequirements:
Three or more years in quality, automation, data, or ML engineeringRecent LLM testing, prompt tuning, or orchestration experienceHands-on LangGraph, ADKs, or internal SDK experienceJSON schema and function-calling expertisePython-heavy automation and asynchronous debugging experienceApply analytical evaluation, Python automation, and LLM orchestration expertise to strengthen Equifax autonomous AI reliability.
#J-18808-Ljbffr
Vacancy posted 51 minutes ago
Similar jobs that could be interesting for youBased on the Agentic Ai Evaluation Engineer in Winnipeg, MB vacancy
  • $200k per year

     ...specializing in data, analytics & AI consulting, dedicated to...  ...analysts, data scientists, data engineers, and digital experts are all dedicated...  ...models, LLM applications,and agentic systems - ensuring scalability...  ...- orchestration layers, evaluation frameworks, guardrails, and... 
    Suggested
    Manual labor

    Welcome to the Jungle

    Winnipeg, MB
    47 minutes ago
  •  ...all applications and next steps. Our partner is looking for an AI Engineer (Agentic Systems) based in Canada. Join a high-performing team of...  ...and machine learning systems using Python. Design, train, evaluate, and continuously improve machine learning and deep learning... 
    Suggested

    Jobgether

    Winnipeg, MB
    51 minutes ago
  •  ...will join a world-class team of engineers and data scientists from Facebook...  ...experience building production‑grade AI/ML systems. Experience architecting complex agentic systems, including orchestration,...  ...experience designing, training, evaluating, and improving ML models.... 
    Suggested

    Ample Insight

    Winnipeg, MB
    54 minutes ago
  • Lead Agentic AI and Machine Learning engineering at Loblaw Companies Limited, guiding teams that build scalable retail technology. Apply Python and LLM expertise...  ..., deployment, and ongoing operationsEstablish evaluation, testing, observability, and continuous improvement practicesRequirements... 
    Suggested

    Loblaw Digital

    Winnipeg, MB
    43 minutes ago
  •  ...is a next-generation data and AI consulting firm dedicated to accelerating...  ..., AI technologies, data engineering, and digital transformation....  ...is looking for a Senior AI & Agentic Engineer: a full-stack...  ...pipeline feeding it, and the evaluations proving it works. This role... 
    Suggested

    Artefact

    Winnipeg, MB
    48 minutes ago
  • Build agentic workflows and personalized AI features as an AI Engineer supporting Superhuman Mail from a remote-flexible Canada arrangement. Apply Python or TypeScript...  ...experiences. You will develop AI agents, evaluation frameworks, and reusable tools. The work includes... 
    Remote work
    Flexible hours

    Jobgether

    Winnipeg, MB
    50 minutes ago
  •  ...applications and next steps. Our partner is looking for a Senior Agentic AI Engineer based remotely in Canada. We are seeking an experienced...  ...enterprise platforms, and third-party services. Implement evaluation frameworks, monitoring, observability, and guardrails to... 
    Full time
    Manual labor
    Remote work

    Jobgether

    Winnipeg, MB
    52 minutes ago
  • We are seeking a Senior Agentic AI Engineer to design, develop, and deploy production-grade AI applications using Large Language Models (LLMs)...  ...solutions, and the ability to troubleshoot complex RAG pipelines, evaluate LLM application quality, and design scalable cloud-based... 

    enableIT

    Winnipeg, MB
    51 minutes ago
  • $98k - $138k per year

     ...stronger financial future for the healthcare community. As a Lead Agentic AI Engineer, you will work directly with engineers, architects, security,...  ...secure, scalable production solutions through monitoring, evaluation, optimization, automated delivery practices, and continuous... 
    Permanent employment
    Immediate start

    Urban Ridge Supplies

    Winnipeg, MB
    48 minutes ago
  •  ...will join a world-class team of engineers and data scientists from Facebook...  ...building production-grade AI/ML systems. Experience architecting complex agentic systems, including orchestration...  ...experience designing, training, evaluating, and improving ML models. Experience... 

    Ample Insight Inc.

    Winnipeg, MB
    54 minutes ago
  • Build GenAI applications with Python, LLMs, and agentic frameworks at Infosys in Mississauga, Ontario. Design production-ready AI solutions through hands-on engineering.Infosys seeks a hands-on GenAI and Agentic AI Engineer for next-generation AI solutions. You will work with... 

    Infosys Limited

    Winnipeg, MB
    50 minutes ago
  • We're hiring a contract AI engineer who builds, deploys and operates AI agents in production, using leading-edge agentic development practice to ship quickly. The AI Platform team builds AI agents and products for the trucking and fleet compliance space, used by both our... 
    Contract work
    Internship

    Aplin

    Winnipeg, MB
    48 minutes ago
  • Shape enterprise Agentic AI applications by deploying North securely across private cloud and on-premises infrastructure. Support customers...  ...through complex technical integrations.This Forward Deployed Engineer role connects North’s core product with client engineering teams... 

    Cohere

    Winnipeg, MB
    48 minutes ago
  • AI Engineer / Data Engineer - Agentic AI (Security)AI Engineer / Data Engineer - Agentic AIVille de Québec, QC (Hybride) Temps plein Location: Quebec City, CanadaDans des marchés en rapide évolution, les clients à travers le monde font confiance à Thales. Thales est une... 
    Full time
    Local area
    Worldwide

    Canadian Inclusion

    Winnipeg, MB
    51 minutes ago
  •  ...York, Seoul and Tokyo, is focused exclusively on sourcing, evaluating and managing private markets investments. Northleaf...  ...for insurance and wealth management clients. Position: AI Solutions Engineer, Agentic Automation Reporting to Northleaf's AI Agent Lead, the... 
    Long term contract
    Remote work
    Flexible hours
    2 days per week

    Urban Ridge Supplies

    Winnipeg, MB
    54 minutes ago
  •  ....Join a remote-first digital product engineering team building AI-powered applications for startups, established...  ..., and Go.Design AI applications and agentic workflows, including integrations...  ....Build testing, experimentation, and evaluation frameworks to validate model outputs... 
    Full time
    Remote work

    Skillhub Solutions Inc.

    Winnipeg, MB
    50 minutes ago
  • Elevate AI advancement at DataAnnotation as an Electrical Engineer, focusing on real-world circuit and systems challenges. Apply your design and debugging skills...  ...realistic engineering problems from practical experience • Evaluate AI model outputs against professional benchmarks •... 

    DataAnnotation

    Winnipeg, MB
    50 minutes ago
  • Lead AI engineering at SandboxAQ by building agentic systems with Python, machine learning, and production software expertise. This role is fully remote.As...  ...agentic problemsCreate monitoring, guardrailing, and agent evaluation frameworksPerform exploratory data analysis and model... 
    Remote work

    SandboxAQ

    Winnipeg, MB
    48 minutes ago
  • SST is seeking an experienced Senior Agentic AI – Validation & Automation Engineer to join our engineering team in Markham, Ontario. This position...  ...Familiarity with human-in-the-loop AI systems, agent evaluation, observability, and workflow recovery mechanisms. Experience... 

    Sapphire Stream Technology

    Winnipeg, MB
    54 minutes ago
  • Define conversational AI quality remotely from Canada or the United States using Python, LLM APIs, and rigorous evaluation systems.As Cresta’s mid-level AI Agent Evaluation Engineer, you will own quality measurement for production LLM-based conversational agents. You will combine... 
    Remote work

    JoinStartup

    Winnipeg, MB
    44 minutes ago
  •  ...enterprise-grade safeguards.We are AI-first in both product and operations. Generative AI, agentic AI, agent-assisted development,...  ...we ship.This is an early engineering role at the foundation-building...  ...knowledge into reusable specs, evaluations, examples, and context for people... 
    Full time
    Internship

    DataBraid Limited

    Winnipeg, MB
    48 minutes ago
  •  ...hybrid Toronto position. Shape reliable AI for go-to-market teams.As a Senior AI Engineer, you will design, develop, and deploy agentic applications end to end. You will create...  ...recommend next stepsOperate guardrails, evaluations, drift detection, and monitoringIntegrate... 

    Thomson Reuters

    Winnipeg, MB
    49 minutes ago
  • Lead agentic AI engineering for Clearsulting’s Office of the CFO service line, using Python, TypeScript, and secure cloud architecture remotely...  ...infrastructure as codeImplement guardrails, approvals, validation, and evaluation frameworksEstablish reusable architectures, standards, and... 
    Work at office
    Remote work

    Clearsulting

    Winnipeg, MB
    48 minutes ago
  • Hiring in Canada(Remote) | Senior Full Stack Engineer – Agentic AI (Backend-Leaning) What “Full Stack” Means Here Strong backend focus...  ...Collaborate with product and engineering teams Read specs, evaluate evolving AI platforms, and adapt fast ✅ What We’re Looking... 
    Remote work

    BayOne Solutions

    Winnipeg, MB
    52 minutes ago
  • Lead production AI architecture and agentic systems development remotely across the Americas as TraceGains’ Staff AI Engineer. Use Python or C# to build reliable capabilities.Reporting to...  ...work connects enterprise APIs, data, evaluation, and responsible AI controls.Key Responsibilities... 
    Remote work

    TraceGains

    Winnipeg, MB
    50 minutes ago
  • Build production-grade agentic AI systems at EY in Toronto, using LangGraph, CrewAI, AI/ML...  ...across complex client engagements.As an AI Engineer, you will design and implement end-to-...  ...agentic solutions with LLMs and SLMsCreate evaluation frameworks, telemetry, and production... 

    EY

    Winnipeg, MB
    44 minutes ago
  • $99.1k - $132.1k per year

     ...to hear from you.About the Role: The Senior Specialist, Agentic AI Engineering designs, builds, deploys, and supports enterprise-scale AI agents...  ...production-ready AI capabilities.Implement observability, evaluation, and monitoring so agent quality, reliability, cost, and... 
    Long term contract

    Mckesson Corporation

    Winnipeg, MB
    48 minutes ago
  • Build agentic systems for Superhuman Mail using backend engineering and applied AI in a remote-flexible model, with Vancouver hybrid availability.As an AI Engineer, you...  ...Mail. You will work with MCP, personalization, evaluation systems, model selection, and production LLM features... 
    Remote work
    Flexible hours

    Coda

    Winnipeg, MB
    48 minutes ago
  • First West Credit Union is seeking an AI Engineer to design, build, and deliver high-quality AI solutions across the organization. The role blends software engineering with modern AI practices to improve user experience, efficiency, and decision-making. You will design AI components... 

    Tru Cooperative Bank

    Winnipeg, MB
    43 minutes ago
  •  ...Cohere is seeking Software Engineers with Applied AI experience to design, build, and deploy agentic workflows powered by LLMs. You will work with customers on real-world problems to create first-of-kind agent workflows integrating LLMs with tools, APIs, and data sources.... 

    Urban Ridge Supplies

    Winnipeg, MB
    44 minutes ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Agentic Ai Evaluation Engineer. Be the first to apply!