Ai Agent Evaluation Engineer
JoinStartup
Define conversational AI quality remotely from Canada or the United States using Python, LLM APIs, and rigorous evaluation systems.As Cresta’s mid-level AI Agent Evaluation Engineer, you will own quality measurement for production LLM-based conversational agents. You will combine conversation analysis, annotation, rubric design, and statistical sampling to create an accurate picture of agent performance. Your work will support Cresta’s AI Receptionist, which handles millions of conversations monthly across hundreds of customers.Key Responsibilities:
Define rubrics across adherence, outcome, and conversation experienceReview production conversations and classify failure modes weeklyBuild automated judges, golden sets, and regression suitesCalibrate judges against human labels and measure agreementReport prioritized quality findings to engineering and product ownersRequirements:
Three or more years in software engineering, applied ML, or researchOne to two years working with production LLM systemsPython programming skills and experience with LLM APIsExperience building end-to-end evaluation systemsDegree in a related field or equivalent practical experienceBring Python, annotation, rubric design, and LLM evaluation expertise to measure Cresta agent quality accurately.
#J-18808-Ljbffr
Define rubrics across adherence, outcome, and conversation experienceReview production conversations and classify failure modes weeklyBuild automated judges, golden sets, and regression suitesCalibrate judges against human labels and measure agreementReport prioritized quality findings to engineering and product ownersRequirements:
Three or more years in software engineering, applied ML, or researchOne to two years working with production LLM systemsPython programming skills and experience with LLM APIsExperience building end-to-end evaluation systemsDegree in a related field or equivalent practical experienceBring Python, annotation, rubric design, and LLM evaluation expertise to measure Cresta agent quality accurately.
#J-18808-Ljbffr
Vacancy posted 2 hours ago
Similar jobs that could be interesting for youBased on the Ai Agent Evaluation Engineer in Winnipeg, MB vacancy
- Evaluate AI-generated code remotely from Canada using software expertise, detailed rubrics, and strong written communication... ...work offers 20+ hours weekly.As a Senior Software Engineer AI Data contributor, you will assess coding agents through codebases, tickets, source materials, and...SuggestedOngoing contractContract workFor contractorsRemote workFlexible hours
- Build reliable autonomous AI at Equifax through LLM evaluation, Python automation, and multi-agent workflow testing. This newly created position focuses on production reliability.As an Agentic AI Evaluation & Tuning Engineer, you will connect LLM capabilities with dependable...Suggested
- Shape trusted AI evaluation at CrowdStrike using agent frameworks, LLM APIs, and verification-first test automation. Work on cloud-based cybersecurity systems.As a Software Development Engineer in Test, you will help determine how CrowdStrike AI agents are tested, evaluated...Suggested
- Elevate AI advancement at DataAnnotation as an Electrical Engineer, focusing on real-world circuit and systems challenges. Apply your design and debugging skills... ...realistic engineering problems from practical experience • Evaluate AI model outputs against professional benchmarks •...Suggested
- Evaluate AI-generated code remotely for Alignerr while applying software engineering, debugging, and code review expertise. Work flexibly on an hourly contract from Calgary.... ...experienceApply software engineering, code evaluation, and precise technical communication to help...SuggestedHourly payContract workRemote work
- Evaluate AI-generated mechanical designs and calculations with strong engineering fundamentals and technical writing skills. Work fully remotely on flexible, project-based... ...or Python experienceBring careful engineering evaluation, failure analysis, and technical communication...For contractorsRemote workFlexible hours
- Leverage your engineering expertise as an Electrical Engineer AI Evaluator with DataAnnotation, focusing on evaluating frontier AI models in real-world electronics tasks. Engage in challenging problem design and model assessment remotely. At DataAnnotation, we seek an experienced...Remote work
$85k - $225k per year
...Toronto is seeking an experienced professional to validate and ensure the reliability of Veeva AI Agents. The ideal candidate should have expertise in data quality, prompt engineering, and proficiency in Python. This full-time role offers a competitive salary range of $85,000...Full timeRemote workFlexible hours- Evaluate frontier AI models with electrical engineering judgment, circuit analysis expertise, and flexible remote work from your own computer.DataAnnotation seeks an experienced electrical engineer for independent contract work evaluating real engineering tasks. Apply circuit...Contract workFor contractorsRemote workFlexible hours
- ...Rootly is hiring an AI Quality Engineer to own the evaluation and optimization of its agentic AI features. You will design test scenarios, probe how AI agents reason, and work with engineering to close the loop on performance across incident workflows. You will partner with...
- Improve AI agent reliability at TribalScale with evaluation pipelines, quality metrics, and disciplined testing for agentic systems. The location remains to be confirmed.This Agentic QA Engineer role focuses on validating AI-driven workflows in real-world environments. You will...
- Lead AI agent engineering at BioRender remotely, focusing on multi-agent orchestration, model routing, and dependable scientific figure generation... .... The work combines production-grade AI leadership with evaluation-driven improvements to output quality.Key Responsibilities:...Remote work
$130k - $155k per year
...Our vision is to build an ML/AI first network of advertisers and... ...hiring its first dedicated AI Engineer. Over the past year, AI at... ...to production: a service desk agent that closes tickets without a... ...demo and a production system: evaluation, monitoring, cost, and a clear...Work at officeRemote work- Build enterprise AI agents at Decagon in an in-office role requiring technical delivery, integrations... ...facing execution.As an Agent Deployment Engineer, you will own strategic customer... ...traces, logs, and model behaviorDefine evaluations and regression testing against ground...Work at office
- Shape AI agent platforms in Canada through tool calling, model routing, and production operations. Apply strong engineering judgment to practical systems.This Applied AI Engineer role centers... ...impactEstablish engineering, evaluation, and safety practicesRequirements:...
- Build AI agents for ClickStack using TypeScript, Node.Js, Python, and MCP in a product-focused engineering environment. Improve how developers investigate incidents.As an AI Product... ...responseOwn context engineering, tools, evaluations, tracing, and costBuild MCP servers, SDKs...
- ...Novara is seeking a QA Engineer to own the quality bar for AI-powered features, including non-deterministic systems and agent-based workflows. You will define what getting it right means, build tooling, and partner with product and engineering to bring rigor to testing LLM-...
- ...and Coffee. Position Summary Join our AI team as an Intermediate AI Engineer, where you'll build and deploy AI applications, agents, and automation solutions that solve real... ...requirements into technical solutions. • Evaluate, monitor, and optimize models and applications...
- Cresta# AI Agent Evaluation Engineer Remote-friendly Canada (Remote); United States (Remote)AT A GLANCEBuild and operate evaluation systems to measure quality of production LLM-based conversational AI agents through annotation, automated judges, and rigorous testing.LEVELmid...Full timeRemote work
- Secure AI agents and LLMs in Toronto through cloud security architecture, guardrails, and protected non-human identities. This is a hybrid, full-time opportunity.As an AI security specialist, you will design protections across enterprise finance, ERP, and aerospace and defense...Full time
- Design AI-agent workflows with Claude Code while building reliable engineering harnesses for software-development workflows in Toronto.This Agentic Engineer role focuses on introducing modern agentic engineering practices into a complex existing engineering organization. You...
- Build AI agents for renewals and expansion at ShareGate, using low-code orchestration, Python or TypeScript, and SQL remotely across Canada... ..., marketing operations, customer success operations, or GTM engineering backgroundHands-on n8n, Make, Zapier, or similar experiencePython...Remote work
$140k - $175k per year
Job Description Thomson Reuters Labs is seeking a Lead Research Engineer to lead the design and delivery of production-ready software for artificial intelligence (AI) model evaluation and benchmarking. You will build scalable evaluation infrastructure that enables research...InternshipWork at officeLocal areaFlexible hours- Apple’s Creativity Apps team seeks a Machine Learning Engineer focused on evaluating ML models and AI features in our creativity tools. You will define evaluation metrics, build evaluation datasets, and develop scalable infrastructure to ensure quality from model development...
- ...Will Do Lead, hire, develop, and scale AI engineering teams and technical leaders supporting... ...engineering quality for LLM-based systems, agents, and integrations, from architecture and... ...of the CoE's shared AI platform: evaluation pipelines, behavioral and quality testing...For contractors
$132.6k - $174k per year
...through automation, agentic automation, and AI. We partner with business and technology... ...As a Senior Software Engineer – AI Agents, you will lead the design and implementation... ...state management, error handling, and agent evaluation in AWSStrong experience with AWS Bedrock...Long term contractPermanent employmentPart timeManual labor- The Frontier AI Deployed Engineer is a member of Future Electronics' AI & Automation Center of Excellence and works directly with... ...to a solution architecture or technology. Rapidly evaluate emerging AI tools, models, agent capabilities, development methods, and integration...Work at officeRemote work
- Use AI coding agents to deliver reliable applications at Remitly, with measurement, prompt iteration... ...from requirements onward.This Software Engineer II role focuses on directing AI agents... ...build and ship complete solutions. You will evaluate generated output against security,...
- ...to life! We're building BioRender's AI figure agent, which can create any scientific figure... ...What you'll be doing Lead a small engineering team responsible for the core agent platform... ...improve figure quality through evaluations, experimentation, and real user feedback...Remote work
- Build production AI agents and order management systems at NewAge Products, working remotely... ...automation through hands-on software engineering.As an AI Engineer, you will design, test... ...databases, cloud services, and production evaluation.Key Responsibilities: Design order...Remote work
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Ai Agent Evaluation Engineer. Be the first to apply!
Related searches
- représentant senior Winnipeg, MB
- représentant alimentaire Winnipeg, MB
- agent d entretien Winnipeg, MB
- quality control representative Winnipeg, MB
- agent nettoyage Winnipeg, MB
- agent aux communications Winnipeg, MB
- agent de liaison Winnipeg, MB
- representant cosmetique Winnipeg, MB
- représentant industriel Winnipeg, MB
- représentant pharmaceutique Winnipeg, MB

