Agentic Ai Evaluation Engineer
Equifax Inc.
Build reliable autonomous AI at Equifax through LLM evaluation, Python automation, and multi-agent workflow testing. This newly created position focuses on production reliability.As an Agentic AI Evaluation & Tuning Engineer, you will connect LLM capabilities with dependable autonomous execution. You will curate Golden Sets, automate evaluation pipelines, audit reasoning trajectories, and tune prompts. Your work will also address tool calling, RAG pipelines, hallucinations, and guardrails. Collaboration with AI Solution Leads and AI Agent Developers supports robust agent components.Key Responsibilities:
Curate documents, queries, and expected agent trajectoriesAutomate evaluations for accuracy, latency, token spend, and reliabilityAudit multi-step reasoning and identify logic deviationsOptimize prompts, context windows, and few-shot examplesMonitor drift, injections, loops, and hallucinationsRequirements:
Three or more years in quality, automation, data, or ML engineeringRecent LLM testing, prompt tuning, or orchestration experienceHands-on LangGraph, ADKs, or internal SDK experienceJSON schema and function-calling expertisePython-heavy automation and asynchronous debugging experienceApply analytical evaluation, Python automation, and LLM orchestration expertise to strengthen Equifax autonomous AI reliability.
#J-18808-Ljbffr
Curate documents, queries, and expected agent trajectoriesAutomate evaluations for accuracy, latency, token spend, and reliabilityAudit multi-step reasoning and identify logic deviationsOptimize prompts, context windows, and few-shot examplesMonitor drift, injections, loops, and hallucinationsRequirements:
Three or more years in quality, automation, data, or ML engineeringRecent LLM testing, prompt tuning, or orchestration experienceHands-on LangGraph, ADKs, or internal SDK experienceJSON schema and function-calling expertisePython-heavy automation and asynchronous debugging experienceApply analytical evaluation, Python automation, and LLM orchestration expertise to strengthen Equifax autonomous AI reliability.
#J-18808-Ljbffr
Vacancy posted 1 hour ago
Similar jobs that could be interesting for youBased on the Agentic Ai Evaluation Engineer in Winnipeg, MB vacancy
- ...will join a world-class team of engineers and data scientists from Facebook... ...experience building production‑grade AI/ML systems. Experience architecting complex agentic systems, including orchestration,... ...experience designing, training, evaluating, and improving ML models....Suggested
- ...applications and next steps. Our partner is looking for a Senior Agentic AI Engineer based remotely in Canada. We are seeking an experienced... ...enterprise platforms, and third-party services. Implement evaluation frameworks, monitoring, observability, and guardrails to...SuggestedFull timeManual laborRemote work
- We are seeking a Senior Agentic AI Engineer to design, develop, and deploy production-grade AI applications using Large Language Models (LLMs)... ...solutions, and the ability to troubleshoot complex RAG pipelines, evaluate LLM application quality, and design scalable cloud-based...Suggested
- ...will join a world-class team of engineers and data scientists from Facebook... ...building production-grade AI/ML systems. Experience architecting complex agentic systems, including orchestration... ...experience designing, training, evaluating, and improving ML models. Experience...Suggested
- Lead Agentic AI and Machine Learning engineering at Loblaw Companies Limited, guiding teams that build scalable retail technology. Apply Python and LLM expertise... ..., deployment, and ongoing operationsEstablish evaluation, testing, observability, and continuous improvement practicesRequirements...Suggested
$200k per year
...specializing in data, analytics & AI consulting, dedicated to... ...analysts, data scientists, data engineers, and digital experts are all dedicated... ...models, LLM applications,and agentic systems - ensuring scalability... ...- orchestration layers, evaluation frameworks, guardrails, and...Manual labor- ...is a next-generation data and AI consulting firm dedicated to accelerating... ..., AI technologies, data engineering, and digital transformation.... ...is looking for a Senior AI & Agentic Engineer: a full-stack... ...pipeline feeding it, and the evaluations proving it works. This role...
- ...all applications and next steps. Our partner is looking for an AI Engineer (Agentic Systems) based in Canada. Join a high-performing team of... ...and machine learning systems using Python. Design, train, evaluate, and continuously improve machine learning and deep learning...
- Build agentic workflows and personalized AI features as an AI Engineer supporting Superhuman Mail from a remote-flexible Canada arrangement. Apply Python or TypeScript... ...experiences. You will develop AI agents, evaluation frameworks, and reusable tools. The work includes...Remote workFlexible hours
$98k - $138k per year
...stronger financial future for the healthcare community. As a Lead Agentic AI Engineer, you will work directly with engineers, architects, security,... ...secure, scalable production solutions through monitoring, evaluation, optimization, automated delivery practices, and continuous...Permanent employmentImmediate start- Shape enterprise Agentic AI applications by deploying North securely across private cloud and on-premises infrastructure. Support customers... ...through complex technical integrations.This Forward Deployed Engineer role connects North’s core product with client engineering teams...
- Build GenAI applications with Python, LLMs, and agentic frameworks at Infosys in Mississauga, Ontario. Design production-ready AI solutions through hands-on engineering.Infosys seeks a hands-on GenAI and Agentic AI Engineer for next-generation AI solutions. You will work with...
- AI Engineer / Data Engineer - Agentic AI (Security)AI Engineer / Data Engineer - Agentic AIVille de Québec, QC (Hybride) Temps plein Location: Quebec City, CanadaDans des marchés en rapide évolution, les clients à travers le monde font confiance à Thales. Thales est une...Full timeLocal areaWorldwide
- Hiring in Canada(Remote) | Senior Full Stack Engineer – Agentic AI (Backend-Leaning) What “Full Stack” Means Here Strong backend focus... ...Collaborate with product and engineering teams Read specs, evaluate evolving AI platforms, and adapt fast ✅ What We’re Looking...Remote work
- SST is seeking an experienced Senior Agentic AI – Validation & Automation Engineer to join our engineering team in Markham, Ontario. This position... ...Familiarity with human-in-the-loop AI systems, agent evaluation, observability, and workflow recovery mechanisms. Experience...
$99.1k - $132.1k per year
...to hear from you.About the Role: The Senior Specialist, Agentic AI Engineering designs, builds, deploys, and supports enterprise-scale AI agents... ...production-ready AI capabilities.Implement observability, evaluation, and monitoring so agent quality, reliability, cost, and...Long term contract- Build agentic systems for Superhuman Mail using backend engineering and applied AI in a remote-flexible model, with Vancouver hybrid availability.As an AI Engineer, you... ...Mail. You will work with MCP, personalization, evaluation systems, model selection, and production LLM features...Remote workFlexible hours
- Define conversational AI quality remotely from Canada or the United States using Python, LLM APIs, and rigorous evaluation systems.As Cresta’s mid-level AI Agent Evaluation Engineer, you will own quality measurement for production LLM-based conversational agents. You will combine...Remote work
- We're hiring a contract AI engineer who builds, deploys and operates AI agents in production, using leading-edge agentic development practice to ship quickly. The AI Platform team builds AI agents and products for the trucking and fleet compliance space, used by both our...Contract workInternship
- ....Join a remote-first digital product engineering team building AI-powered applications for startups, established... ..., and Go.Design AI applications and agentic workflows, including integrations... ....Build testing, experimentation, and evaluation frameworks to validate model outputs...Full timeRemote work
- Lead agentic AI engineering for Clearsulting’s Office of the CFO service line, using Python, TypeScript, and secure cloud architecture remotely... ...infrastructure as codeImplement guardrails, approvals, validation, and evaluation frameworksEstablish reusable architectures, standards, and...Work at officeRemote work
- Elevate AI advancement at DataAnnotation as an Electrical Engineer, focusing on real-world circuit and systems challenges. Apply your design and debugging skills... ...realistic engineering problems from practical experience • Evaluate AI model outputs against professional benchmarks •...
- Lead AI engineering at SandboxAQ by building agentic systems with Python, machine learning, and production software expertise. This role is fully remote.As... ...agentic problemsCreate monitoring, guardrailing, and agent evaluation frameworksPerform exploratory data analysis and model...Remote work
- Lead production AI architecture and agentic systems development remotely across the Americas as TraceGains’ Staff AI Engineer. Use Python or C# to build reliable capabilities.Reporting to... ...work connects enterprise APIs, data, evaluation, and responsible AI controls.Key Responsibilities...
- Build production-grade agentic AI systems at EY in Toronto, using LangGraph, CrewAI, AI/ML... ...across complex client engagements.As an AI Engineer, you will design and implement end-to-... ...agentic solutions with LLMs and SLMsCreate evaluation frameworks, telemetry, and production...
- ...enterprise-grade safeguards.We are AI-first in both product and operations. Generative AI, agentic AI, agent-assisted development,... ...we ship.This is an early engineering role at the foundation-building... ...knowledge into reusable specs, evaluations, examples, and context for people...Full timeInternship
- ...York, Seoul and Tokyo, is focused exclusively on sourcing, evaluating and managing private markets investments. Northleaf... ...for insurance and wealth management clients. Position: AI Solutions Engineer, Agentic Automation Reporting to Northleaf's AI Agent Lead, the...Long term contractRemote workFlexible hours2 days per week
- ...hybrid Toronto position. Shape reliable AI for go-to-market teams.As a Senior AI Engineer, you will design, develop, and deploy agentic applications end to end. You will create... ...recommend next stepsOperate guardrails, evaluations, drift detection, and monitoringIntegrate...
- ...digital product studio composed of 180+ senior engineers and designers with backgrounds from... ...within Lazer focused on building robust, AI-powered full-stack applications. We partner... ...products to life, leveraging the latest in AI, agentic workflows, and scalable infrastructure....InternshipRemote workWorldwide
- First West Credit Union is seeking an AI Engineer to design, build, and deliver high-quality AI solutions across the organization. The role blends software engineering with modern AI practices to improve user experience, efficiency, and decision-making. You will design AI components...
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Agentic Ai Evaluation Engineer. Be the first to apply!

