Agentic Ai Evaluation Engineer
Equifax
Build reliable autonomous workflows at Equifax through LLM evaluation, Python automation, and multi-agent system tuning. This newly created position focuses on enterprise-grade AI.As an Agentic AI Evaluation and Tuning Engineer, you will test and optimize intelligent systems navigating complex software environments. Your work combines Golden Set curation, automated evaluation pipelines, prompt refinement, and tool-calling optimization. You will also investigate agent trajectories, asynchronous failures, and production behavior while collaborating with AI Solution Leads and AI Agent Developers. The position supports reliable, fact-based autonomous decision-making across deployed workflows.Key Responsibilities:
Curate and augment Golden Sets for agent testingDesign continuous evaluations for accuracy, latency, and token spendAudit multi-step reasoning trajectories and deviationsOptimize prompts, context windows, and few-shot examplesMonitor drift, prompt injections, loops, and hallucinationsRequirements:
Three-plus years in quality, automation, data, or ML engineeringRecent focus on LLM testing, prompt tuning, or orchestrationHands-on LangGraph, ADKs, or internal SDK experienceJSON schema and function-calling expertisePython-heavy automation and asynchronous debugging experienceApply agent evaluation, observability, Python automation, and structured tool calling to strengthen Equifax autonomous systems.
#J-18808-Ljbffr
Curate and augment Golden Sets for agent testingDesign continuous evaluations for accuracy, latency, and token spendAudit multi-step reasoning trajectories and deviationsOptimize prompts, context windows, and few-shot examplesMonitor drift, prompt injections, loops, and hallucinationsRequirements:
Three-plus years in quality, automation, data, or ML engineeringRecent focus on LLM testing, prompt tuning, or orchestrationHands-on LangGraph, ADKs, or internal SDK experienceJSON schema and function-calling expertisePython-heavy automation and asynchronous debugging experienceApply agent evaluation, observability, Python automation, and structured tool calling to strengthen Equifax autonomous systems.
#J-18808-Ljbffr
Vacancy posted 23 minutes ago
Similar jobs that could be interesting for youBased on the Agentic Ai Evaluation Engineer in Winnipeg, MB vacancy
- ...reliable autonomous workflows at Equifax in Toronto using Python, LLM evaluation, and observability for production AI systems.This newly created full-time role focuses on Agentic AI Evaluation and Tuning Engineering. You will test multi-agent architectures, audit decision-making...SuggestedFull time
- ...will join a world-class team of engineers and data scientists from Facebook... ...building production-grade AI/ML systems. Experience architecting complex agentic systems, including orchestration... ...experience designing, training, evaluating, and improving ML models. Experience...Suggested
- ...is a next-generation data and AI consulting firm dedicated to accelerating... ..., AI technologies, data engineering, and digital transformation.... ...is looking for a Senior AI & Agentic Engineer: a full-stack... ...pipeline feeding it, and the evaluations proving it works. This role...Suggested
- ...all applications and next steps. Our partner is looking for an AI Engineer (Agentic Systems) based in Canada. Join a high-performing team of... ...and machine learning systems using Python. Design, train, evaluate, and continuously improve machine learning and deep learning...Suggested
- Lead agentic AI architecture at Robots & Pencils, using Python, AWS, and production-focused engineering judgment across end-to-end systems.As a Staff AI Builder, you will lead AI/ML... ...spans agentic workflows, RAG pipelines, evaluation, observability, and cloud-native services...Suggested
$100k - $150k per year
Position Title: AI Engineer (Agentic Workflows) Department: Research & Development Company: Protocase Inc. Location: Waterloo, Ontario... ...retrieval, tool integration, prompt and context design, and evaluation. You'll think in terms of workflows and outcomes—identifying...Full timeSeasonal workWork at officeWorldwideShift workDay shift- We are seeking a Senior Agentic AI Engineer to design, develop, and deploy production-grade AI applications using Large Language Models (LLMs)... ...solutions, and the ability to troubleshoot complex RAG pipelines, evaluate LLM application quality, and design scalable cloud-based...
- ...role. Degree in Computer Science or Engineering. Strong Python skills and experience building production-grade AI/ML systems. Experience architecting complex agentic systems, including orchestration,... ...experience designing, training, evaluating, and improving ML models....
$200k per year
...specializing in data, analytics & AI consulting, dedicated to... ...analysts, data scientists, data engineers, and digital experts are all dedicated... ...models, LLM applications,and agentic systems - ensuring scalability... ...- orchestration layers, evaluation frameworks, guardrails, and...Manual labor$98k - $138k per year
...stronger financial future for the healthcare community. As a Lead Agentic AI Engineer, you will work directly with engineers, architects, security,... ...secure, scalable production solutions through monitoring, evaluation, optimization, automated delivery practices, and continuous...Permanent employmentImmediate start- Build agentic workflows and personalized AI features as an AI Engineer supporting Superhuman Mail from a remote-flexible Canada arrangement. Apply Python or TypeScript... ...experiences. You will develop AI agents, evaluation frameworks, and reusable tools. The work includes...Remote workFlexible hours
- AI Engineer / Data Engineer - Agentic AI (Security)AI Engineer / Data Engineer - Agentic AIVille de Québec, QC (Hybride) Temps plein Location: Quebec City, CanadaDans des marchés en rapide évolution, les clients à travers le monde font confiance à Thales. Thales est une...Full timeLocal areaWorldwide
- We're hiring a contract AI engineer who builds, deploys and operates AI agents in production, using leading-edge agentic development practice to ship quickly. The AI Platform team builds AI agents and products for the trucking and fleet compliance space, used by both our...Contract workInternship
- Elevate AI advancement at DataAnnotation as an Electrical Engineer, focusing on real-world circuit and systems challenges. Apply your design and debugging skills... ...realistic engineering problems from practical experience • Evaluate AI model outputs against professional benchmarks •...
- Lead RAG troubleshooting and LLM evaluation with embeddings, vector search, retrieval optimization, and measurable quality testing.As a Senior Agentic AI Engineer, you will improve how enterprise AI applications retrieve information and generate grounded responses. The role...
- Build GenAI applications with Python, LLMs, and agentic frameworks at Infosys in Mississauga, Ontario. Design production-ready AI solutions through hands-on engineering.Infosys seeks a hands-on GenAI and Agentic AI Engineer for next-generation AI solutions. You will work with...
- AI is changing how software gets built. We're looking for a Staff AI Engineer to work out which changes matter for clearer.Io, prove them... ...spot which developments in agentic AI, commerce protocols and the... ...approved actions, failures and evaluation. Build working prototypes and...Long term contractWork at officeRemote workWorldwide
- SST is seeking an experienced Senior Agentic AI – Validation & Automation Engineer to join our engineering team in Markham, Ontario. This position... ...Familiarity with human-in-the-loop AI systems, agent evaluation, observability, and workflow recovery mechanisms. Experience...
- Engineer reliable AI solutions for EY clients in Toronto, focusing on agent orchestration, multi-agent coordination, evaluation, and responsible production delivery. Collaborate across engagements... ...and maintaining production-grade agentic AI systems in enterprise environments...
- ...hybrid Toronto position. Shape reliable AI for go-to-market teams.As a Senior AI Engineer, you will design, develop, and deploy agentic applications end to end. You will create... ...recommend next stepsOperate guardrails, evaluations, drift detection, and monitoringIntegrate...
- Build agentic systems for Superhuman Mail using backend engineering and applied AI in a remote-flexible model, with Vancouver hybrid availability.As an AI Engineer, you... ...Mail. You will work with MCP, personalization, evaluation systems, model selection, and production LLM features...Remote workFlexible hours
- Lead agentic AI engineering for Clearsulting’s Office of the CFO service line, using Python, TypeScript, and secure cloud architecture remotely... ...infrastructure as codeImplement guardrails, approvals, validation, and evaluation frameworksEstablish reusable architectures, standards, and...Work at officeRemote work
$99.1k - $132.1k per year
...to hear from you.About the Role: The Senior Specialist, Agentic AI Engineering designs, builds, deploys, and supports enterprise-scale AI agents... ...production-ready AI capabilities.Implement observability, evaluation, and monitoring so agent quality, reliability, cost, and...Long term contract- Build the next generation of AI-enabled insurance workflows, combining... ...automation, enterprise engineering and human decision-making. The... ...AI Engineer – Generative AI & Agentic Systems to support an exciting... ...engineering controls: Build evaluation datasets, automated regression...InternshipImmediate start
- ...enterprise-grade safeguards.We are AI-first in both product and operations. Generative AI, agentic AI, agent-assisted development,... ...we ship.This is an early engineering role at the foundation-building... ...knowledge into reusable specs, evaluations, examples, and context for people...Full timeInternship
- Advance Wagepoint’s agentic software factory remotely with AI governance, quality engineering, and platform leadership for payroll technology.As Principal Software Development... ...improve the development pipelineDefine CI/CD evaluation gates, drift monitoring, and release...Remote work
- ...York, Seoul and Tokyo, is focused exclusively on sourcing, evaluating and managing private markets investments. Northleaf... ...for insurance and wealth management clients. Position: AI Solutions Engineer, Agentic Automation Reporting to Northleaf's AI Agent Lead, the...Long term contractRemote workFlexible hours2 days per week
- Hiring in Canada(Remote) | Senior Full Stack Engineer – Agentic AI (Backend-Leaning) What “Full Stack” Means Here Strong backend focus... ...Collaborate with product and engineering teams Read specs, evaluate evolving AI platforms, and adapt fast ✅ What We’re Looking...Remote work
- First West Credit Union is seeking an AI Engineer to design, build, and deliver high-quality AI solutions across the organization. The role blends software engineering with modern AI practices to improve user experience, efficiency, and decision-making. You will design AI components...
- ...digital product studio composed of 180+ senior engineers and designers with backgrounds from... ...within Lazer focused on building robust, AI-powered full-stack applications. We partner... ...products to life, leveraging the latest in AI, agentic workflows, and scalable infrastructure....InternshipRemote workWorldwide
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Agentic Ai Evaluation Engineer. Be the first to apply!

