C Open Role · Via A16z Speedrun 1d Ago Ai Agent Evaluation Engineer Cresta Ai Express Interest →
JoinStartup
Cresta# AI Agent Evaluation Engineer Remote-friendly Canada (Remote);
United States (Remote)AT A GLANCEBuild and operate evaluation systems to measure quality of production LLM-based conversational AI agents through annotation, automated judges, and rigorous testing.LEVELmid · 3+ yrsKEY SKILLSLLM evaluationPythonLLM APIsPrompt engineeringAnnotation & labelingStatistical analysisRubric designData analysisSQLConversational AIQuality assuranceCalibration methodologyGolden setsRegression testingWHAT YOU'LL DO* Define evaluation rubrics and quality metrics across adherence, outcome, and conversation experience* Write annotation guidelines and failure taxonomies;
run calibration sessions and measure inter-rater agreement* Manually review production conversations weekly to identify, classify, and prioritize failure modes* Trace failures to root causes and route findings to appropriate engineering/product owners* Build and operate scalable evaluation pipelines combining automated checks, LLM-based judges, and human review* Calibrate automated judges against human labels and manage judge error as primary measurement risk* Design stratified sampling strategies to ensure quality measurements are statistically valid* Build and maintain golden sets and regression suites for validating changes before releaseTHE ROLECresta unlocks the true potential of the customer experience, turning every conversation into a competitive advantage. Cresta’s unified AI platform combines conversational AI agents, real-time human agent augmentation, and comprehensive conversation intelligence to drive revenue and efficiency gains across every channel. The world’s leading companies, including United Airlines, Cox Communications, and Marriott, use Cresta to power world-class customer experiences every day.Born from the Stanford AI Lab, Cresta has raised more than $270 million from the world’s leading investors, including a16z, Greylock, and Sequoia. Cresta’s leadership includes some of the leading minds in AI today. Our CEO, Ping Wu, founded and led Google's Contact Center AI and Vertex AI platforms before joining Cresta to build the future of AI-driven customer experiences.Over the next few years, AI is going to redefine how people all over the world interact with businesses every day. Come build that future at Cresta.About the role:Cresta’s AI Receptionist answers the phone for small businesses that would otherwise miss the call. It handles millions of conversations a month today, making sure no call goes unanswered and helping owners complete the front-office tasks that keep their business running, and it does so across hundreds of customers. Whether those conversations are actually good is the single most important question in the business, and it deserves a full-time owner. That’s you.As our first AI Agent Evaluation Engineer, you define what quality means for every agent we run, you build the systems that measure it across every conversation, and you are the person the company trusts when the question is “how good is it, really?”The work combines hands-on conversation analysis, rigorous evaluation design, and engineering execution: reading and annotating real conversations to learn how the agent actually fails, building the automated evaluation systems (LLM-based judges, golden sets, regression suites) that measure those failures at scale, and keeping those systems honest by calibrating them against human judgment. Fixing what you find is welcome but not the core of the job;
an accurate, prioritized picture of agent quality is.What You'll Do :Evaluation Design & Quality Definition* Define and maintain the evaluation rubric for the agent across three axes: adherence (did it follow its design, call the right tools, respect guardrails), outcome (did the conversation achieve its goal), and conversation experience (latency, turn-taking, tone, repetition).* Write the annotation guidelines and failure taxonomy the team uses to talk about failures precisely;
run calibration sessions and measure inter-rater agreement.* Classify and prioritize failure modes by severity and frequency so engineering and product work on what matters most.Conversation Analysis & Annotation* Review production conversations by hand every week, leverage QA partners;
open-code failures, cluster them, and turn the clusters into the evaluations worth automating.* Trace each failure to its likely cause (prompt, knowledge gap, tool or flow bug, speech recognition) and route it to the right owner with enough evidence to act on.* Evaluate across languages and report quality by segment (language, accent, audio quality, intent), not just in aggregate.Scalable Evaluation Systems* Build and operate the evaluation pipeline that scores every production conversation, combining automated checks, LLM-based judges, and human review.* Calibrate every automated judge against human labels before trusting it;
treat judge error as the primary measurement risk.* Design sampling strategies (stratified by intent, language, caller type, audio conditions) so that reported changes in quality are real.* Build and maintain the golden set and regression suite that every prompt, model, knowledge, and flow change is validated against before release.Coverage & Edge Cases* Maintain the failure-mode coverage map: which behaviors are tested, partially tested, or unmeasured;
close the gaps.* Designevaluations for behaviors no standard harness covers: high-stakes conversation types such as handoffs to a human, misuse and unwanted-caller handling, and resilience under upstream failures.* Define the quality and latency thresholds the agent is held to.Reporting & Cross-Functional Collaboration* Report quality trends to internal stakeholders and customers clearly, with uncertainty stated and without burying bad news.* Partner with forward-deployed engineers, backend engineers, product managers, and the voice/language engineering team to turn findings into prioritized recommendations.* Document methodology so results are reproducible and the evaluation practice outlives any one person.What We're Looking For* 3+ years of professional experience in software engineering, applied ML, or data-intensive research roles, including at least 1–2 years working hands-on with LLM-based systems in production.* Bachelor’s or Master’s degree in Computer Science, Computational Linguistics, Cognitive Science, Statistics, or a related field, or equivalent practical experience.* Built and operated an end-to-end evaluation system for a production LLM product (agent or otherwise): rubric, human labels, automated judge, regression suite.* Deep understanding of evaluation methodology: what golden sets, judges, regression suites, and calibration are for, and when each fails. Tool-agnostic;
you will work in our in-house tooling.* Fluent in how LLM agents work: prompts, tool calls, context handling, multi-turn state. You can read a trace and tell what the agent was doing and why.* Strong programming skills in Python and experience with LLM APIs;
able to build and ship evaluation tooling independently. Fine-tuning and ML frameworks are not required.* Able to write a rubric that independent reviewers would score the same way, and to measure wheth#J-18808-Ljbffr
United States (Remote)AT A GLANCEBuild and operate evaluation systems to measure quality of production LLM-based conversational AI agents through annotation, automated judges, and rigorous testing.LEVELmid · 3+ yrsKEY SKILLSLLM evaluationPythonLLM APIsPrompt engineeringAnnotation & labelingStatistical analysisRubric designData analysisSQLConversational AIQuality assuranceCalibration methodologyGolden setsRegression testingWHAT YOU'LL DO* Define evaluation rubrics and quality metrics across adherence, outcome, and conversation experience* Write annotation guidelines and failure taxonomies;
run calibration sessions and measure inter-rater agreement* Manually review production conversations weekly to identify, classify, and prioritize failure modes* Trace failures to root causes and route findings to appropriate engineering/product owners* Build and operate scalable evaluation pipelines combining automated checks, LLM-based judges, and human review* Calibrate automated judges against human labels and manage judge error as primary measurement risk* Design stratified sampling strategies to ensure quality measurements are statistically valid* Build and maintain golden sets and regression suites for validating changes before releaseTHE ROLECresta unlocks the true potential of the customer experience, turning every conversation into a competitive advantage. Cresta’s unified AI platform combines conversational AI agents, real-time human agent augmentation, and comprehensive conversation intelligence to drive revenue and efficiency gains across every channel. The world’s leading companies, including United Airlines, Cox Communications, and Marriott, use Cresta to power world-class customer experiences every day.Born from the Stanford AI Lab, Cresta has raised more than $270 million from the world’s leading investors, including a16z, Greylock, and Sequoia. Cresta’s leadership includes some of the leading minds in AI today. Our CEO, Ping Wu, founded and led Google's Contact Center AI and Vertex AI platforms before joining Cresta to build the future of AI-driven customer experiences.Over the next few years, AI is going to redefine how people all over the world interact with businesses every day. Come build that future at Cresta.About the role:Cresta’s AI Receptionist answers the phone for small businesses that would otherwise miss the call. It handles millions of conversations a month today, making sure no call goes unanswered and helping owners complete the front-office tasks that keep their business running, and it does so across hundreds of customers. Whether those conversations are actually good is the single most important question in the business, and it deserves a full-time owner. That’s you.As our first AI Agent Evaluation Engineer, you define what quality means for every agent we run, you build the systems that measure it across every conversation, and you are the person the company trusts when the question is “how good is it, really?”The work combines hands-on conversation analysis, rigorous evaluation design, and engineering execution: reading and annotating real conversations to learn how the agent actually fails, building the automated evaluation systems (LLM-based judges, golden sets, regression suites) that measure those failures at scale, and keeping those systems honest by calibrating them against human judgment. Fixing what you find is welcome but not the core of the job;
an accurate, prioritized picture of agent quality is.What You'll Do :Evaluation Design & Quality Definition* Define and maintain the evaluation rubric for the agent across three axes: adherence (did it follow its design, call the right tools, respect guardrails), outcome (did the conversation achieve its goal), and conversation experience (latency, turn-taking, tone, repetition).* Write the annotation guidelines and failure taxonomy the team uses to talk about failures precisely;
run calibration sessions and measure inter-rater agreement.* Classify and prioritize failure modes by severity and frequency so engineering and product work on what matters most.Conversation Analysis & Annotation* Review production conversations by hand every week, leverage QA partners;
open-code failures, cluster them, and turn the clusters into the evaluations worth automating.* Trace each failure to its likely cause (prompt, knowledge gap, tool or flow bug, speech recognition) and route it to the right owner with enough evidence to act on.* Evaluate across languages and report quality by segment (language, accent, audio quality, intent), not just in aggregate.Scalable Evaluation Systems* Build and operate the evaluation pipeline that scores every production conversation, combining automated checks, LLM-based judges, and human review.* Calibrate every automated judge against human labels before trusting it;
treat judge error as the primary measurement risk.* Design sampling strategies (stratified by intent, language, caller type, audio conditions) so that reported changes in quality are real.* Build and maintain the golden set and regression suite that every prompt, model, knowledge, and flow change is validated against before release.Coverage & Edge Cases* Maintain the failure-mode coverage map: which behaviors are tested, partially tested, or unmeasured;
close the gaps.* Designevaluations for behaviors no standard harness covers: high-stakes conversation types such as handoffs to a human, misuse and unwanted-caller handling, and resilience under upstream failures.* Define the quality and latency thresholds the agent is held to.Reporting & Cross-Functional Collaboration* Report quality trends to internal stakeholders and customers clearly, with uncertainty stated and without burying bad news.* Partner with forward-deployed engineers, backend engineers, product managers, and the voice/language engineering team to turn findings into prioritized recommendations.* Document methodology so results are reproducible and the evaluation practice outlives any one person.What We're Looking For* 3+ years of professional experience in software engineering, applied ML, or data-intensive research roles, including at least 1–2 years working hands-on with LLM-based systems in production.* Bachelor’s or Master’s degree in Computer Science, Computational Linguistics, Cognitive Science, Statistics, or a related field, or equivalent practical experience.* Built and operated an end-to-end evaluation system for a production LLM product (agent or otherwise): rubric, human labels, automated judge, regression suite.* Deep understanding of evaluation methodology: what golden sets, judges, regression suites, and calibration are for, and when each fails. Tool-agnostic;
you will work in our in-house tooling.* Fluent in how LLM agents work: prompts, tool calls, context handling, multi-turn state. You can read a trace and tell what the agent was doing and why.* Strong programming skills in Python and experience with LLM APIs;
able to build and ship evaluation tooling independently. Fine-tuning and ML frameworks are not required.* Able to write a rubric that independent reviewers would score the same way, and to measure wheth#J-18808-Ljbffr
Vacancy posted 2 hours ago
Similar jobs that could be interesting for youBased on the C Open Role · Via A16z Speedrun 1d Ago Ai Agent Evaluation Engineer Cresta Ai Express Interest → in Winnipeg, MB vacancy
- Define conversational AI quality remotely from Canada or the United States using Python, LLM APIs, and rigorous evaluation systems.As Cresta’s mid-level AI Agent Evaluation Engineer, you will own quality measurement for production LLM-based conversational agents. You will combine...SuggestedRemote work
$75k - $95k per year
Project Manager - Expression of InterestProjects Full Time Burnaby, BC... ...stakeholders while ensuring engineering standards and technical feasibility... ...planning and performance evaluations for direct reports Team... ...In compliance with the B.C. Pay Transparency Act, the expected...SuggestedFull timeImmediate start$60k - $68k per year
Proposal Coordinator - Expression of Interest Sales and Marketing Full Time Burnaby, BC $60,000 -... ...excellence. You will also play a vital role in maintaining a library of proposal content... ...for this position.Bachelor's degree in Engineering, Marketing, Communications, or...SuggestedFull timeWork at officeImmediate start$2000 - $2600 per week
Expression of Interest - Future OpportunitiesKickstart Entertainment is building a network of talented... ...productions. While this position is not currently open and no project or start date has been... ..., business requirements, updated role requirements, and Kickstart's staffing needs...SuggestedFull timeInternshipRemote workWork from home- ...an experienced Aircraft Maintenance Engineer (AME) looking for your next opportunity... ...aviation team? We are inviting expressions of interest from skilled, safety‑driven professionals... ...continued to grow, expanding our fleet and our role in connecting communities throughout...SuggestedRemplacementFlexible hoursShift workNight shift
- ...from day 1, premiums 100% covered by Image Engine, up to 8 paid Health and Wellness days per... ...in optimizing shots for efficiency Evaluate artist workflow with an eye on improving visual... ...co-workers and assuming leadership role SKILLS AND ABILITIESTechnical skills...Contract work
- Join Cresta as a Machine Learning Engineer to innovate in AI-driven call center solutions. Work remotely with a world-class team to advance ML technologies and improve... ...balance of teamwork and autonomy in this dynamic role. Key Responsibilities: • Define product...Remote work
$59.8k - $78.5k per year
Expression of Interest | Executive Assistant | VancouverBusiness Unit: Store Support Centre (SSC)... ...evergreen posting for Executive Assistant I roles for future consideration.By applying to... ...and request to be considered if an opening arises. Please note this is not a formal...Permanent employmentFull timePart timeWork at officeLocal areaImmediate start- Position: VFX Producer (Expression of Interest)Hiring Range: $140,000 to $170,000/year for a 40 hour work weekProject Contract: TBCCompany Benefits... ...healthcare from day 1, premiums 100% covered by Image Engine, up to 8 paid Health and Wellness days per year.*Candidates are...Contract workWork at officeShift work
- ...Benefits:Extended healthcare from day 1, premiums 100% covered by Image Engine, up to 8 paid Health and Wellness days per year.*Candidates are... ...is upheld across the shots and sequences they are assignedBe open and enthusiastic about learning new things to further develop...Contract workTemporary workInternship
- ...accomplish the task. They are responsible for models, textures, surfaces, look development and rigging of characters, sets and props. Your Role Take an asset or shot to a final quality, hitting all notes Model, texture, lookdev, and render assets Fluent with node-based...Contract workFixed term contractSummer workCasual workWork at officeRemote workFlexible hours
- ...General Manager at Lev Senior Living, we would encourage you to express your interest here!By applying here, you are expressing interest in future... ...Executive Leadership of Lev Senior Living, this senior leadership role, is responsible for the overall success and operation of the...Local area
- ...understanding of the production pipeline and experience in creating final shots that are in alignment with the team's artistic direction. Your Role Finalizes the overall look, colour balance, and quality for shots and sequences Creates uniformity and seamless transitions...Contract workSummer workCasual workWork at officeRemote workFlexible hours
- ...holding a significant minority interest. Turner & Townsend and CBRE... ...equity, and inclusion, we welcome expressions of interest from Indigenous... ...and data analytics Engineering, design, planning or construction... ...bring. QualificationsWe are open to expressions of interest...InternshipFlexible hours
- Join Cresta as a Senior Product Manager, at the forefront of developing next-gen AI contact center products. Experience in B2B SaaS and virtual agent technologies will be key to your success. This senior... ...’s in Computer Science, Engineering, or Business • 4+ years' experience...
- Shape the future of work with Cresta as a Senior Software Engineer specializing in AI technology. Be pivotal in creating impactful Virtual Agents and improve productivity metrics remotely. In this senior role, you’ll work within the Cresta Virtual Agents engineering team...Remote work
$110k - $130k per year
...culture built on respect, approachability and an open-door philosophy. Joining Grewal Guyatt... ...personal and professional well-being. Role OverviewThe Accounting Manager will oversee... ...needs. Use of Artificial Intelligence (AI)At Grewal Guyatt LLP, we use AI to support...Long term contractPermanent employmentFull timeInternshipWork at officeRemote work1 day per week- Business Development Manager - Expression of InterestJoin to apply for the Business Development Manager - Expression of Interest role at REV Founded in 2008, REV’s sole focus... .... Qualifications Electrical Engineering Technologist diploma or Bachelor’s degree...For contractorsWork at office
- Transform the future of work as a Frontend Senior Software Engineer at Cresta, specializing in AI technologies. In this remote role, you'll enhance the platform that enables knowledge workers to achieve breakthrough productivity. Cresta is in search of engineers who can drive...Remote work
- Expressions of Interest Blind Bored Ventilation Shaft Project Join to apply for the Expressions of... ...Blind Bored Ventilation Shaft Project role at Abergeldie Complex... ...including those based internationally who are open to relocating to Australia.Why Relocate...Permanent employmentRelocation
$2500 per week
Calling All Production Safety Advisors (Canada)! – Expression of Interest Are you passionate about creating a safe and thriving environment for... ...bring their visions to life. Please note that these roles are contracted directly by the productions and are not employed...For contractorsFreelance- Evaluate AI-generated code remotely from Canada using software expertise, detailed rubrics, and strong written communication... ...work offers 20+ hours weekly.As a Senior Software Engineer AI Data contributor, you will assess coding agents through codebases, tickets, source materials, and...Ongoing contractContract workFor contractorsRemote workFlexible hours
$2000 - $2400 per week
...the desired look and visual quality of an animation project. This role ensures continuity of the artistic vision throughout the... ...tous les plans du film. Collaborer avec le Superviseur CG pour évaluer la faisabilité technique des designs, en suggérant des solutions...Flexible hours- Build reliable autonomous AI at Equifax through LLM evaluation, Python automation, and multi-agent workflow testing. This newly created position focuses on production reliability.As an Agentic AI Evaluation & Tuning Engineer, you will connect LLM capabilities with dependable...
- Elevate AI advancement at DataAnnotation as an Electrical Engineer, focusing on real-world circuit and systems challenges. Apply... ...or electronics engineering. Your role involves creating realistic... ...problems from practical experience • Evaluate AI model outputs against...
- Join to apply for the Senior Principal Engineer - AI Framework Programming Expression Technology role at Huawei Canada1 month ago Be among the first 25 applicantsJoin to apply for the Senior Principal Engineer - AI Framework Programming Expression Technology...Permanent employmentImmediate startRemote workRelocation
- ...multi-commodity expertise across both open pit and underground mining. With offices... ...we continue to grow, we are seeking Expressions of Interest from talented professionals interested... ...is preferredPrincipal Geotechnical Engineer 10+ years of Geotechnical experience...InternshipWork at officeWorldwide
- ...pipelinesAn ability to interpret supervisor and client notesAbout This Role Pay Range: The posted pay range is aligned with what we... ...all types are encouraged to apply. We thank everyone for their interest and application, however only those selected for an interview will...Local area
- ...notes, tasks, calendar and they’re not AI-native. Our client’s mission is to build... ...time on valuable and meaningful things.Role As a Backend Engineer, AI, you own the inference and... ...PythonNodeJsPytorchOpenAI / Anthropic / open-source LLMsSQL & noSQLDockerHow We Work...
- Shape trusted AI evaluation at CrowdStrike using agent frameworks, LLM APIs, and verification-first test automation. Work on cloud-based cybersecurity systems.As a Software Development Engineer in Test, you will help determine how CrowdStrike AI agents are tested, evaluated...
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to C Open Role · Via A16z Speedrun 1d Ago Ai Agent Evaluation Engineer Cresta Ai Express Interest →. Be the first to apply!
Related searches
- représentant senior Winnipeg, MB
- représentant alimentaire Winnipeg, MB
- agent d entretien Winnipeg, MB
- quality control representative Winnipeg, MB
- agent nettoyage Winnipeg, MB
- agent aux communications Winnipeg, MB
- agent de liaison Winnipeg, MB
- representant cosmetique Winnipeg, MB
- représentant industriel Winnipeg, MB
- représentant pharmaceutique Winnipeg, MB

