Agentic Ai Evaluation Engineer
Equifax Inc.
Build reliable autonomous AI at Equifax through LLM evaluation, Python automation, and multi-agent workflow testing. This newly created position focuses on production reliability.As an Agentic AI Evaluation & Tuning Engineer, you will connect LLM capabilities with dependable autonomous execution. You will curate Golden Sets, automate evaluation pipelines, audit reasoning trajectories, and tune prompts. Your work will also address tool calling, RAG pipelines, hallucinations, and guardrails. Collaboration with AI Solution Leads and AI Agent Developers supports robust agent components.Key Responsibilities:
Curate documents, queries, and expected agent trajectoriesAutomate evaluations for accuracy, latency, token spend, and reliabilityAudit multi-step reasoning and identify logic deviationsOptimize prompts, context windows, and few-shot examplesMonitor drift, injections, loops, and hallucinationsRequirements:
Three or more years in quality, automation, data, or ML engineeringRecent LLM testing, prompt tuning, or orchestration experienceHands-on LangGraph, ADKs, or internal SDK experienceJSON schema and function-calling expertisePython-heavy automation and asynchronous debugging experienceApply analytical evaluation, Python automation, and LLM orchestration expertise to strengthen Equifax autonomous AI reliability.
#J-18808-Ljbffr
Curate documents, queries, and expected agent trajectoriesAutomate evaluations for accuracy, latency, token spend, and reliabilityAudit multi-step reasoning and identify logic deviationsOptimize prompts, context windows, and few-shot examplesMonitor drift, injections, loops, and hallucinationsRequirements:
Three or more years in quality, automation, data, or ML engineeringRecent LLM testing, prompt tuning, or orchestration experienceHands-on LangGraph, ADKs, or internal SDK experienceJSON schema and function-calling expertisePython-heavy automation and asynchronous debugging experienceApply analytical evaluation, Python automation, and LLM orchestration expertise to strengthen Equifax autonomous AI reliability.
#J-18808-Ljbffr
Vacancy posted 1 hour ago
Similar jobs that could be interesting for youBased on the Agentic Ai Evaluation Engineer in Winnipeg, MB vacancy
- Lead Agentic AI and Machine Learning engineering at Loblaw Companies Limited, guiding teams that build scalable retail technology. Apply Python and LLM expertise... ..., deployment, and ongoing operationsEstablish evaluation, testing, observability, and continuous improvement practicesRequirements...Suggested
- ...applications and next steps. Our partner is looking for a Senior Agentic AI Engineer based remotely in Canada. We are seeking an experienced... ...enterprise platforms, and third-party services. Implement evaluation frameworks, monitoring, observability, and guardrails to...SuggestedFull timeManual laborRemote work
- ...will join a world-class team of engineers and data scientists from Facebook... ...building production-grade AI/ML systems. Experience architecting complex agentic systems, including orchestration... ...experience designing, training, evaluating, and improving ML models. Experience...Suggested
- ...is a next-generation data and AI consulting firm dedicated to accelerating... ..., AI technologies, data engineering, and digital transformation.... ...is looking for a Senior AI & Agentic Engineer: a full-stack... ...pipeline feeding it, and the evaluations proving it works. This role...Suggested
- Build agentic workflows and personalized AI features as an AI Engineer supporting Superhuman Mail from a remote-flexible Canada arrangement. Apply Python or TypeScript... ...experiences. You will develop AI agents, evaluation frameworks, and reusable tools. The work includes...SuggestedRemote workFlexible hours
- ...all applications and next steps. Our partner is looking for an AI Engineer (Agentic Systems) based in Canada. Join a high-performing team of... ...and machine learning systems using Python. Design, train, evaluate, and continuously improve machine learning and deep learning...
- We are seeking a Senior Agentic AI Engineer to design, develop, and deploy production-grade AI applications using Large Language Models (LLMs)... ...solutions, and the ability to troubleshoot complex RAG pipelines, evaluate LLM application quality, and design scalable cloud-based...
- ...will join a world-class team of engineers and data scientists from Facebook... ...experience building production‑grade AI/ML systems. Experience architecting complex agentic systems, including orchestration,... ...experience designing, training, evaluating, and improving ML models....
- We're hiring a contract AI engineer who builds, deploys and operates AI agents in production, using leading-edge agentic development practice to ship quickly. The AI Platform team builds AI agents and products for the trucking and fleet compliance space, used by both our...Contract workInternship
- Shape enterprise Agentic AI applications by deploying North securely across private cloud and on-premises infrastructure. Support customers... ...through complex technical integrations.This Forward Deployed Engineer role connects North’s core product with client engineering teams...
- Build GenAI applications with Python, LLMs, and agentic frameworks at Infosys in Mississauga, Ontario. Design production-ready AI solutions through hands-on engineering.Infosys seeks a hands-on GenAI and Agentic AI Engineer for next-generation AI solutions. You will work with...
- AI Engineer / Data Engineer - Agentic AI (Security)AI Engineer / Data Engineer - Agentic AIVille de Québec, QC (Hybride) Temps plein Location: Quebec City, CanadaDans des marchés en rapide évolution, les clients à travers le monde font confiance à Thales. Thales est une...Full timeLocal areaWorldwide
- Build agentic systems for Superhuman Mail using backend engineering and applied AI in a remote-flexible model, with Vancouver hybrid availability.As an AI Engineer, you... ...Mail. You will work with MCP, personalization, evaluation systems, model selection, and production LLM features...Remote workFlexible hours
- Lead production AI architecture and agentic systems development remotely across the Americas as TraceGains’ Staff AI Engineer. Use Python or C# to build reliable capabilities.Reporting to... ...work connects enterprise APIs, data, evaluation, and responsible AI controls.Key Responsibilities...Remote work
- ...hybrid Toronto position. Shape reliable AI for go-to-market teams.As a Senior AI Engineer, you will design, develop, and deploy agentic applications end to end. You will create... ...recommend next stepsOperate guardrails, evaluations, drift detection, and monitoringIntegrate...
- ...enterprise-grade safeguards.We are AI-first in both product and operations. Generative AI, agentic AI, agent-assisted development,... ...we ship.This is an early engineering role at the foundation-building... ...knowledge into reusable specs, evaluations, examples, and context for people...Full timeInternship
- ....Join a remote-first digital product engineering team building AI-powered applications for startups, established... ..., and Go.Design AI applications and agentic workflows, including integrations... ....Build testing, experimentation, and evaluation frameworks to validate model outputs...Full timeRemote work
- Lead agentic AI engineering for Clearsulting’s Office of the CFO service line, using Python, TypeScript, and secure cloud architecture remotely... ...infrastructure as codeImplement guardrails, approvals, validation, and evaluation frameworksEstablish reusable architectures, standards, and...Work at officeRemote work
- SST is seeking an experienced Senior Agentic AI – Validation & Automation Engineer to join our engineering team in Markham, Ontario. This position... ...Familiarity with human-in-the-loop AI systems, agent evaluation, observability, and workflow recovery mechanisms. Experience...
- Lead AI engineering at SandboxAQ by building agentic systems with Python, machine learning, and production software expertise. This role is fully remote.As... ...agentic problemsCreate monitoring, guardrailing, and agent evaluation frameworksPerform exploratory data analysis and model...Remote work
- Build production-grade agentic AI systems at EY in Toronto, using LangGraph, CrewAI, AI/ML... ...across complex client engagements.As an AI Engineer, you will design and implement end-to-... ...agentic solutions with LLMs and SLMsCreate evaluation frameworks, telemetry, and production...
- Elevate AI advancement at DataAnnotation as an Electrical Engineer, focusing on real-world circuit and systems challenges. Apply your design and debugging skills... ...realistic engineering problems from practical experience • Evaluate AI model outputs against professional benchmarks •...
- Build production AI agents in Markham with Python, test automation, and CI/CD expertise. SST offers an onsite engineering role focused on validation.As a Senior Agentic AI – Validation & Automation Engineer, you will develop intelligent workflows for engineering validation...
- ...ContactMonkey Inc. is seeking an AI engineering specialist to design, prototype, and ship production-grade agentic workflows. You’ll own the eval harness, datasets, and automated grading, while collaborating with SRE to deploy AI workloads on EKS. This hands-on role blends...
- ...digital product studio composed of 180+ senior engineers and designers with backgrounds from... ...within Lazer focused on building robust, AI-powered full-stack applications. We partner... ...products to life, leveraging the latest in AI, agentic workflows, and scalable infrastructure....InternshipRemote workWorldwide
- Empower your career as a Senior Java Engineer at Agentic AI in Toronto with a hybrid work model. This role demands innovation in a technology-driven... ...: • Architect and implement Java solutions from scratch • Evaluate technologies for project feasibility and compliance •...Internship
- ...where you belong. Our teams are applying Agentic AI, Generative AI, and Machine Learning to... ...at scale. We're looking for an Engineering Manager, Agentic AI & Machine Learning to... ...engineering practices: Champion robust evaluation frameworks, automated testing, observability...Long term contractPermanent employmentTemporary workInternship
- ...Coffee. Position Summary Join our AI team as an Intermediate AI Engineer, where you'll build and deploy AI... ...into technical solutions. • Evaluate, monitor, and optimize models and applications... ...experience in AI, machine learning, agentic AI, and workflow automation. •...
- Evaluate AI-generated code remotely for Alignerr while applying software engineering, debugging, and code review expertise. Work flexibly on an hourly contract from Calgary.... ...experienceApply software engineering, code evaluation, and precise technical communication to help...Hourly payContract workRemote work
$160k - $180k per year
Senior Software Engineer, Agentic AI Products Who we are:Leo Berwick is an innovative mergers and acquisitions consulting firm. We are comprised... ...team capability compounds over time.Take part in technical evaluations, including case‑study style discussions, and help hire other...Permanent employmentFull timeInternshipWork at office2 days per week
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Agentic Ai Evaluation Engineer. Be the first to apply!

