C Open Role · Via A16z Speedrun 1d Ago Ai Agent Evaluation Engineer Cresta Ai Express Interest →
JoinStartup
Cresta# AI Agent Evaluation Engineer Remote-friendly Canada (Remote);
United States (Remote)AT A GLANCEBuild and operate evaluation systems to measure quality of production LLM-based conversational AI agents through annotation, automated judges, and rigorous testing.LEVELmid · 3+ yrsKEY SKILLSLLM evaluationPythonLLM APIsPrompt engineeringAnnotation & labelingStatistical analysisRubric designData analysisSQLConversational AIQuality assuranceCalibration methodologyGolden setsRegression testingWHAT YOU'LL DO* Define evaluation rubrics and quality metrics across adherence, outcome, and conversation experience* Write annotation guidelines and failure taxonomies;
run calibration sessions and measure inter-rater agreement* Manually review production conversations weekly to identify, classify, and prioritize failure modes* Trace failures to root causes and route findings to appropriate engineering/product owners* Build and operate scalable evaluation pipelines combining automated checks, LLM-based judges, and human review* Calibrate automated judges against human labels and manage judge error as primary measurement risk* Design stratified sampling strategies to ensure quality measurements are statistically valid* Build and maintain golden sets and regression suites for validating changes before releaseTHE ROLECresta unlocks the true potential of the customer experience, turning every conversation into a competitive advantage. Cresta’s unified AI platform combines conversational AI agents, real-time human agent augmentation, and comprehensive conversation intelligence to drive revenue and efficiency gains across every channel. The world’s leading companies, including United Airlines, Cox Communications, and Marriott, use Cresta to power world-class customer experiences every day.Born from the Stanford AI Lab, Cresta has raised more than $270 million from the world’s leading investors, including a16z, Greylock, and Sequoia. Cresta’s leadership includes some of the leading minds in AI today. Our CEO, Ping Wu, founded and led Google's Contact Center AI and Vertex AI platforms before joining Cresta to build the future of AI-driven customer experiences.Over the next few years, AI is going to redefine how people all over the world interact with businesses every day. Come build that future at Cresta.About the role:Cresta’s AI Receptionist answers the phone for small businesses that would otherwise miss the call. It handles millions of conversations a month today, making sure no call goes unanswered and helping owners complete the front-office tasks that keep their business running, and it does so across hundreds of customers. Whether those conversations are actually good is the single most important question in the business, and it deserves a full-time owner. That’s you.As our first AI Agent Evaluation Engineer, you define what quality means for every agent we run, you build the systems that measure it across every conversation, and you are the person the company trusts when the question is “how good is it, really?”The work combines hands-on conversation analysis, rigorous evaluation design, and engineering execution: reading and annotating real conversations to learn how the agent actually fails, building the automated evaluation systems (LLM-based judges, golden sets, regression suites) that measure those failures at scale, and keeping those systems honest by calibrating them against human judgment. Fixing what you find is welcome but not the core of the job;
an accurate, prioritized picture of agent quality is.What You'll Do :Evaluation Design & Quality Definition* Define and maintain the evaluation rubric for the agent across three axes: adherence (did it follow its design, call the right tools, respect guardrails), outcome (did the conversation achieve its goal), and conversation experience (latency, turn-taking, tone, repetition).* Write the annotation guidelines and failure taxonomy the team uses to talk about failures precisely;
run calibration sessions and measure inter-rater agreement.* Classify and prioritize failure modes by severity and frequency so engineering and product work on what matters most.Conversation Analysis & Annotation* Review production conversations by hand every week, leverage QA partners;
open-code failures, cluster them, and turn the clusters into the evaluations worth automating.* Trace each failure to its likely cause (prompt, knowledge gap, tool or flow bug, speech recognition) and route it to the right owner with enough evidence to act on.* Evaluate across languages and report quality by segment (language, accent, audio quality, intent), not just in aggregate.Scalable Evaluation Systems* Build and operate the evaluation pipeline that scores every production conversation, combining automated checks, LLM-based judges, and human review.* Calibrate every automated judge against human labels before trusting it;
treat judge error as the primary measurement risk.* Design sampling strategies (stratified by intent, language, caller type, audio conditions) so that reported changes in quality are real.* Build and maintain the golden set and regression suite that every prompt, model, knowledge, and flow change is validated against before release.Coverage & Edge Cases* Maintain the failure-mode coverage map: which behaviors are tested, partially tested, or unmeasured;
close the gaps.* Designevaluations for behaviors no standard harness covers: high-stakes conversation types such as handoffs to a human, misuse and unwanted-caller handling, and resilience under upstream failures.* Define the quality and latency thresholds the agent is held to.Reporting & Cross-Functional Collaboration* Report quality trends to internal stakeholders and customers clearly, with uncertainty stated and without burying bad news.* Partner with forward-deployed engineers, backend engineers, product managers, and the voice/language engineering team to turn findings into prioritized recommendations.* Document methodology so results are reproducible and the evaluation practice outlives any one person.What We're Looking For* 3+ years of professional experience in software engineering, applied ML, or data-intensive research roles, including at least 1–2 years working hands-on with LLM-based systems in production.* Bachelor’s or Master’s degree in Computer Science, Computational Linguistics, Cognitive Science, Statistics, or a related field, or equivalent practical experience.* Built and operated an end-to-end evaluation system for a production LLM product (agent or otherwise): rubric, human labels, automated judge, regression suite.* Deep understanding of evaluation methodology: what golden sets, judges, regression suites, and calibration are for, and when each fails. Tool-agnostic;
you will work in our in-house tooling.* Fluent in how LLM agents work: prompts, tool calls, context handling, multi-turn state. You can read a trace and tell what the agent was doing and why.* Strong programming skills in Python and experience with LLM APIs;
able to build and ship evaluation tooling independently. Fine-tuning and ML frameworks are not required.* Able to write a rubric that independent reviewers would score the same way, and to measure wheth#J-18808-Ljbffr
United States (Remote)AT A GLANCEBuild and operate evaluation systems to measure quality of production LLM-based conversational AI agents through annotation, automated judges, and rigorous testing.LEVELmid · 3+ yrsKEY SKILLSLLM evaluationPythonLLM APIsPrompt engineeringAnnotation & labelingStatistical analysisRubric designData analysisSQLConversational AIQuality assuranceCalibration methodologyGolden setsRegression testingWHAT YOU'LL DO* Define evaluation rubrics and quality metrics across adherence, outcome, and conversation experience* Write annotation guidelines and failure taxonomies;
run calibration sessions and measure inter-rater agreement* Manually review production conversations weekly to identify, classify, and prioritize failure modes* Trace failures to root causes and route findings to appropriate engineering/product owners* Build and operate scalable evaluation pipelines combining automated checks, LLM-based judges, and human review* Calibrate automated judges against human labels and manage judge error as primary measurement risk* Design stratified sampling strategies to ensure quality measurements are statistically valid* Build and maintain golden sets and regression suites for validating changes before releaseTHE ROLECresta unlocks the true potential of the customer experience, turning every conversation into a competitive advantage. Cresta’s unified AI platform combines conversational AI agents, real-time human agent augmentation, and comprehensive conversation intelligence to drive revenue and efficiency gains across every channel. The world’s leading companies, including United Airlines, Cox Communications, and Marriott, use Cresta to power world-class customer experiences every day.Born from the Stanford AI Lab, Cresta has raised more than $270 million from the world’s leading investors, including a16z, Greylock, and Sequoia. Cresta’s leadership includes some of the leading minds in AI today. Our CEO, Ping Wu, founded and led Google's Contact Center AI and Vertex AI platforms before joining Cresta to build the future of AI-driven customer experiences.Over the next few years, AI is going to redefine how people all over the world interact with businesses every day. Come build that future at Cresta.About the role:Cresta’s AI Receptionist answers the phone for small businesses that would otherwise miss the call. It handles millions of conversations a month today, making sure no call goes unanswered and helping owners complete the front-office tasks that keep their business running, and it does so across hundreds of customers. Whether those conversations are actually good is the single most important question in the business, and it deserves a full-time owner. That’s you.As our first AI Agent Evaluation Engineer, you define what quality means for every agent we run, you build the systems that measure it across every conversation, and you are the person the company trusts when the question is “how good is it, really?”The work combines hands-on conversation analysis, rigorous evaluation design, and engineering execution: reading and annotating real conversations to learn how the agent actually fails, building the automated evaluation systems (LLM-based judges, golden sets, regression suites) that measure those failures at scale, and keeping those systems honest by calibrating them against human judgment. Fixing what you find is welcome but not the core of the job;
an accurate, prioritized picture of agent quality is.What You'll Do :Evaluation Design & Quality Definition* Define and maintain the evaluation rubric for the agent across three axes: adherence (did it follow its design, call the right tools, respect guardrails), outcome (did the conversation achieve its goal), and conversation experience (latency, turn-taking, tone, repetition).* Write the annotation guidelines and failure taxonomy the team uses to talk about failures precisely;
run calibration sessions and measure inter-rater agreement.* Classify and prioritize failure modes by severity and frequency so engineering and product work on what matters most.Conversation Analysis & Annotation* Review production conversations by hand every week, leverage QA partners;
open-code failures, cluster them, and turn the clusters into the evaluations worth automating.* Trace each failure to its likely cause (prompt, knowledge gap, tool or flow bug, speech recognition) and route it to the right owner with enough evidence to act on.* Evaluate across languages and report quality by segment (language, accent, audio quality, intent), not just in aggregate.Scalable Evaluation Systems* Build and operate the evaluation pipeline that scores every production conversation, combining automated checks, LLM-based judges, and human review.* Calibrate every automated judge against human labels before trusting it;
treat judge error as the primary measurement risk.* Design sampling strategies (stratified by intent, language, caller type, audio conditions) so that reported changes in quality are real.* Build and maintain the golden set and regression suite that every prompt, model, knowledge, and flow change is validated against before release.Coverage & Edge Cases* Maintain the failure-mode coverage map: which behaviors are tested, partially tested, or unmeasured;
close the gaps.* Designevaluations for behaviors no standard harness covers: high-stakes conversation types such as handoffs to a human, misuse and unwanted-caller handling, and resilience under upstream failures.* Define the quality and latency thresholds the agent is held to.Reporting & Cross-Functional Collaboration* Report quality trends to internal stakeholders and customers clearly, with uncertainty stated and without burying bad news.* Partner with forward-deployed engineers, backend engineers, product managers, and the voice/language engineering team to turn findings into prioritized recommendations.* Document methodology so results are reproducible and the evaluation practice outlives any one person.What We're Looking For* 3+ years of professional experience in software engineering, applied ML, or data-intensive research roles, including at least 1–2 years working hands-on with LLM-based systems in production.* Bachelor’s or Master’s degree in Computer Science, Computational Linguistics, Cognitive Science, Statistics, or a related field, or equivalent practical experience.* Built and operated an end-to-end evaluation system for a production LLM product (agent or otherwise): rubric, human labels, automated judge, regression suite.* Deep understanding of evaluation methodology: what golden sets, judges, regression suites, and calibration are for, and when each fails. Tool-agnostic;
you will work in our in-house tooling.* Fluent in how LLM agents work: prompts, tool calls, context handling, multi-turn state. You can read a trace and tell what the agent was doing and why.* Strong programming skills in Python and experience with LLM APIs;
able to build and ship evaluation tooling independently. Fine-tuning and ML frameworks are not required.* Able to write a rubric that independent reviewers would score the same way, and to measure wheth#J-18808-Ljbffr
Vacancy posted 56 minutes ago
Similar jobs that could be interesting for youBased on the C Open Role · Via A16z Speedrun 1d Ago Ai Agent Evaluation Engineer Cresta Ai Express Interest → in Winnipeg, MB vacancy
- ...passionate individuals to fill a variety of roles with the families I work with. You can... ...but not required -Serious knowledge of or interest in infant development in the first year (ish... ...please email me at sunupchildcare at gmail dot c o m with some info about yourself, any qualifications...SuggestedHourly payLong term contractFull timeContract workFor subcontractorInternshipMonday to fridayNight shiftDay shiftAfternoon shift
- ...Miratech is looking for a skilled Senior AI Engineer with Python, LangGraph and LangChain to join... ...healthcare client. You will play a key role in developing and enhancing an enterprise... ...Develop and enhance AI agents and orchestration capabilities using Python...SuggestedFull timeInternshipRemote workRelocation
- ...and Coffee. Position Summary Join our AI team as an Intermediate AI Engineer, where you'll build and deploy AI applications, agents, and automation solutions that solve real... ...requirements into technical solutions. • Evaluate, monitor, and optimize models and applications...Suggested
$95k - $110k per year
...Aircraft Mechanic AME M2 (C-17) Company: Boeing Canada Operations... ...(TCCA) Aircraft Maintenance Engineer (AME) licensed M2 rating or Military... ...Globemaster III Type Course via Air Educational and Training Command... ...systems. Conflict of Interest: Successful candidates for this...SuggestedRemplacementFull timeContract workRelocationRelocation packageShift work- ...Senior Evaluation and Performance Analyst Department: Executive... ...Indigenous Affairs This Job Opening is limited to employees with the... ...Analyst, you'll play a key role in improving the sustainability... ...receipt of your application, please follow up via telephone....SuggestedLong term contract
$95k - $110k per year
...Aircraft Structures Technician (C-17) Company: Boeing Canada... ...Aviation (TCCA) Aircraft Maintenance Engineer (AME) licensed S/M2 rating,... ...CC177 Globemaster III Type Course via Air Educational and Training... ...ADAM) system. Conflict of Interest: Successful candidates for this...Contract workWork at officeRemote workRelocationRelocation packageShift work- ...dans la rémunération qu’un candidat reçoit pour ce poste. Cette offre d’emploi concerne un poste vacant existant. L’entreprise utilise l’intelligence artificielle dans le but de filtrer, d’évaluer et/ou de sélectionner les candidats à ce poste. #EN #STORE #CRETAIL #MB...Permanent employmentTemporary workInternshipImmediate startWeekend workAfternoon shift
- ...Join Workada as a Remote Video Production Evaluator dedicated to improving AI technology through precise evaluations. This full-time role focuses on quality assessment of multimedia outputs. At Workada, we seek an expert who can critically analyze video, image, and text...Full timeRemote work
- ...Drive AI innovation as a Senior AI Engineer in Mississauga, Canada. Craft Python-based AI applications using FastAPI or Flask while optimizing performance... ...collaborate with stakeholders to deliver impactful solutions. Your role involves mentoring teams and providing architectural...
- Define conversational AI quality remotely from Canada or the United States using Python, LLM APIs, and rigorous evaluation systems.As Cresta’s mid-level AI Agent Evaluation Engineer, you will own quality measurement for production LLM-based conversational agents. You will combine...Remote work
- ...think so too! The Guest Service Agent is the face of our hotels. You're... ...Group, we recognize this vital role with a competitive compensation... ...Applications will not be accepted via email or in person. We thank all applicants for their interest in Career opportunities with The...Permanent employmentFull timeApprenticeshipLocal areaFlexible hoursShift work
$2000 - $2600 per week
Expression of Interest - Future OpportunitiesKickstart Entertainment is building a network of talented... ...productions. While this position is not currently open and no project or start date has been... ..., business requirements, updated role requirements, and Kickstart's staffing needs...Full timeInternshipRemote workWork from home$75k - $95k per year
Project Manager - Expression of InterestProjects Full Time Burnaby, BC... ...stakeholders while ensuring engineering standards and technical feasibility... ...planning and performance evaluations for direct reports Team... ...In compliance with the B.C. Pay Transparency Act, the expected...Full timeImmediate start$60k - $68k per year
Proposal Coordinator - Expression of Interest Sales and Marketing Full Time Burnaby, BC $60,000 -... ...excellence. You will also play a vital role in maintaining a library of proposal content... ...for this position.Bachelor's degree in Engineering, Marketing, Communications, or...Full timeWork at officeImmediate start- ...an experienced Aircraft Maintenance Engineer (AME) looking for your next opportunity... ...aviation team? We are inviting expressions of interest from skilled, safety‑driven professionals... ...continued to grow, expanding our fleet and our role in connecting communities throughout...RemplacementFlexible hoursShift workNight shift
- ...from day 1, premiums 100% covered by Image Engine, up to 8 paid Health and Wellness days per... ...in optimizing shots for efficiency Evaluate artist workflow with an eye on improving visual... ...co-workers and assuming leadership role SKILLS AND ABILITIESTechnical skills...Contract work
- ...Benefits:Extended healthcare from day 1, premiums 100% covered by Image Engine, up to 8 paid Health and Wellness days per year.*Candidates are... ...is upheld across the shots and sequences they are assignedBe open and enthusiastic about learning new things to further develop...Contract workTemporary workInternship
$52k - $65k per year
...We are currently seeking a Purchasing Agent who has a passion for construction and hands-... ...building materials. This is a junior-level role ideal for someone with a background in construction... ..., orders, and basic documentation Interest in learning and growing within a...Full timeWork at office- ...Travel Agent – Full-Time Location: Winnipeg, MB Employment Type: Full-Time Schedule: Monday–Friday, 9:00 AM–5:00 PM CT Work... ...a related field, or equivalent training/coursework. Strong interest in the travel industry and willingness to learn GDS systems....Long term contractPermanent employmentFull timeWork at officeRemote workMonday to friday
- Position: VFX Producer (Expression of Interest)Hiring Range: $140,000 to $170,000/year for a 40 hour work weekProject Contract: TBCCompany Benefits... ...healthcare from day 1, premiums 100% covered by Image Engine, up to 8 paid Health and Wellness days per year.*Candidates are...Contract workWork at officeShift work
- ...Maintain documentation as required. Plans, implements, and evaluates client treatment plans. Physical Requirements Not Applicable... ...of your family, friends and neighbours, please apply today. Interested candidates should select the "Apply" icon below to upload their...Full timeImmediate startShift work
- ...accomplish the task. They are responsible for models, textures, surfaces, look development and rigging of characters, sets and props. Your Role Take an asset or shot to a final quality, hitting all notes Model, texture, lookdev, and render assets Fluent with node-based...Contract workFixed term contractSummer workCasual workWork at officeRemote workFlexible hours
$59.8k - $78.5k per year
Expression of Interest | Executive Assistant | VancouverBusiness Unit: Store Support Centre (SSC)... ...evergreen posting for Executive Assistant I roles for future consideration.By applying to... ...and request to be considered if an opening arises. Please note this is not a formal...Permanent employmentFull timePart timeWork at officeLocal areaImmediate start- ...understanding of the production pipeline and experience in creating final shots that are in alignment with the team's artistic direction. Your Role Finalizes the overall look, colour balance, and quality for shots and sequences Creates uniformity and seamless transitions...Contract workSummer workCasual workWork at officeRemote workFlexible hours
- ...cancel and change room reservations. Provide information on hotel facilities and services. Provide general information about points of interest in the area. Process guests' departures, calculate charges and receive payments. Follow emergency and safety procedures. Clerical...Hourly payPermanent employmentImmediate startShift workNight shiftWeekend workDay shiftAfternoon shiftEarly shift
- Business Development Manager - Expression of InterestJoin to apply for the Business Development Manager - Expression of Interest role at REV Founded in 2008, REV’s sole focus... .... Qualifications Electrical Engineering Technologist diploma or Bachelor’s degree...For contractorsWork at office
- Shape the future of work with Cresta as a Senior Software Engineer specializing in AI technology. Be pivotal in creating impactful Virtual Agents and improve productivity metrics remotely. In this senior role, you’ll work within the Cresta Virtual Agents engineering team...Remote work
- Join Cresta as a Senior Product Manager, at the forefront of developing next-gen AI contact center products. Experience in B2B SaaS and virtual agent technologies will be key to your success. This senior... ...’s in Computer Science, Engineering, or Business • 4+ years' experience...
- ...General Manager at Lev Senior Living, we would encourage you to express your interest here!By applying here, you are expressing interest in future... ...Executive Leadership of Lev Senior Living, this senior leadership role, is responsible for the overall success and operation of the...Local area
- ...cancel and change room reservations. Provide information on hotel facilities and services. Provide general information about points of interest in the area. Process guests' departures, calculate charges and receive payments. Follow emergency and safety procedures. Clerical...Hourly payPermanent employmentImmediate startShift workNight shiftWeekend workDay shiftAfternoon shiftEarly shift
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to C Open Role · Via A16z Speedrun 1d Ago Ai Agent Evaluation Engineer Cresta Ai Express Interest →. Be the first to apply!
Related searches
- représentant senior Winnipeg, MB
- représentant alimentaire Winnipeg, MB
- agent d entretien Winnipeg, MB
- quality control representative Winnipeg, MB
- agent nettoyage Winnipeg, MB
- agent aux communications Winnipeg, MB
- agent de liaison Winnipeg, MB
- representant cosmetique Winnipeg, MB
- représentant industriel Winnipeg, MB
- représentant pharmaceutique Winnipeg, MB


