Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Ai Agent Evaluation Engineer

JoinStartup

Define conversational AI quality remotely from Canada or the United States using Python, LLM APIs, and rigorous evaluation systems.As Cresta’s mid-level AI Agent Evaluation Engineer, you will own quality measurement for production LLM-based conversational agents. You will combine conversation analysis, annotation, rubric design, and statistical sampling to create an accurate picture of agent performance. Your work will support Cresta’s AI Receptionist, which handles millions of conversations monthly across hundreds of customers.Key Responsibilities:
Define rubrics across adherence, outcome, and conversation experienceReview production conversations and classify failure modes weeklyBuild automated judges, golden sets, and regression suitesCalibrate judges against human labels and measure agreementReport prioritized quality findings to engineering and product ownersRequirements:
Three or more years in software engineering, applied ML, or researchOne to two years working with production LLM systemsPython programming skills and experience with LLM APIsExperience building end-to-end evaluation systemsDegree in a related field or equivalent practical experienceBring Python, annotation, rubric design, and LLM evaluation expertise to measure Cresta agent quality accurately.
#J-18808-Ljbffr
Vacancy posted 2 hours ago
Similar jobs that could be interesting for youBased on the Ai Agent Evaluation Engineer in Winnipeg, MB vacancy
  • Evaluate AI-generated code remotely from Canada using software expertise, detailed rubrics, and strong written communication...  ...work offers 20+ hours weekly.As a Senior Software Engineer AI Data contributor, you will assess coding agents through codebases, tickets, source materials, and... 
    Suggested
    Ongoing contract
    Contract work
    For contractors
    Remote work
    Flexible hours

    OpenTrain AI, Inc.

    Winnipeg, MB
    2 hours ago
  • Build reliable autonomous AI at Equifax through LLM evaluation, Python automation, and multi-agent workflow testing. This newly created position focuses on production reliability.As an Agentic AI Evaluation & Tuning Engineer, you will connect LLM capabilities with dependable... 
    Suggested

    Equifax Inc.

    Winnipeg, MB
    2 hours ago
  • Elevate AI advancement at DataAnnotation as an Electrical Engineer, focusing on real-world circuit and systems challenges. Apply your design and debugging skills...  ...realistic engineering problems from practical experience • Evaluate AI model outputs against professional benchmarks •... 
    Suggested

    DataAnnotation

    Winnipeg, MB
    2 hours ago
  • Shape trusted AI evaluation at CrowdStrike using agent frameworks, LLM APIs, and verification-first test automation. Work on cloud-based cybersecurity systems.As a Software Development Engineer in Test, you will help determine how CrowdStrike AI agents are tested, evaluated... 
    Suggested

    CrowdStrike Inc.

    Winnipeg, MB
    2 hours ago
  • Evaluate AI-generated code remotely for Alignerr while applying software engineering, debugging, and code review expertise. Work flexibly on an hourly contract from Calgary....  ...experienceApply software engineering, code evaluation, and precise technical communication to help... 
    Suggested
    Hourly pay
    Contract work
    Remote work

    Alignerr Corp.

    Winnipeg, MB
    2 hours ago
  • Evaluate AI-generated mechanical designs and calculations with strong engineering fundamentals and technical writing skills. Work fully remotely on flexible, project-based...  ...or Python experienceBring careful engineering evaluation, failure analysis, and technical communication... 
    For contractors
    Remote work
    Flexible hours

    Brunel

    Winnipeg, MB
    2 hours ago
  •  ...Novara is seeking a QA Engineer to own the quality bar for AI-powered features, including non-deterministic systems and agent-based workflows. You will define what getting it right means, build tooling, and partner with product and engineering to bring rigor to testing LLM-... 

    Novara

    Winnipeg, MB
    2 hours ago
  • $85k - $225k per year

     ...Toronto is seeking an experienced professional to validate and ensure the reliability of Veeva AI Agents. The ideal candidate should have expertise in data quality, prompt engineering, and proficiency in Python. This full-time role offers a competitive salary range of $85,000... 
    Full time
    Remote work
    Flexible hours

    Veeva Systems

    Winnipeg, MB
    2 hours ago
  •  ...Rootly is hiring an AI Quality Engineer to own the evaluation and optimization of its agentic AI features. You will design test scenarios, probe how AI agents reason, and work with engineering to close the loop on performance across incident workflows. You will partner with... 

    Rootly

    Winnipeg, MB
    2 hours ago
  • Leverage your engineering expertise as an Electrical Engineer AI Evaluator with DataAnnotation, focusing on evaluating frontier AI models in real-world electronics tasks. Engage in challenging problem design and model assessment remotely. At DataAnnotation, we seek an experienced... 
    Remote work

    DataAnnotation

    Winnipeg, MB
    2 hours ago
  • Evaluate frontier AI models with electrical engineering judgment, circuit analysis expertise, and flexible remote work from your own computer.DataAnnotation seeks an experienced electrical engineer for independent contract work evaluating real engineering tasks. Apply circuit... 
    Contract work
    For contractors
    Remote work
    Flexible hours

    DataAnnotation

    Winnipeg, MB
    2 hours ago
  • Build enterprise AI agents at Decagon in an in-office role requiring technical delivery, integrations...  ...facing execution.As an Agent Deployment Engineer, you will own strategic customer...  ...traces, logs, and model behaviorDefine evaluations and regression testing against ground... 
    Work at office

    FDE Times

    Winnipeg, MB
    2 hours ago
  • Lead AI agent engineering at BioRender remotely, focusing on multi-agent orchestration, model routing, and dependable scientific figure generation...  .... The work combines production-grade AI leadership with evaluation-driven improvements to output quality.Key Responsibilities:... 
    Remote work

    BioRender

    Winnipeg, MB
    2 hours ago
  • Build AI agents for ClickStack using TypeScript, Node.Js, Python, and MCP in a product-focused engineering environment. Improve how developers investigate incidents.As an AI Product...  ...responseOwn context engineering, tools, evaluations, tracing, and costBuild MCP servers, SDKs... 

    ClickHouse

    Winnipeg, MB
    2 hours ago
  • Improve AI agent reliability at TribalScale with evaluation pipelines, quality metrics, and disciplined testing for agentic systems. The location remains to be confirmed.This Agentic QA Engineer role focuses on validating AI-driven workflows in real-world environments. You will... 

    TribalScale

    Winnipeg, MB
    2 hours ago
  • $130k - $155k per year

     ...Our vision is to build an ML/AI first network of advertisers and...  ...hiring its first dedicated AI Engineer. Over the past year, AI at...  ...to production: a service desk agent that closes tickets without a...  ...demo and a production system: evaluation, monitoring, cost, and a clear... 
    Work at office
    Remote work

    Fluent, LLC

    Winnipeg, MB
    2 hours ago
  • Shape AI agent platforms in Canada through tool calling, model routing, and production operations. Apply strong engineering judgment to practical systems.This Applied AI Engineer role centers...  ...impactEstablish engineering, evaluation, and safety practicesRequirements:... 

    Jobgether

    Winnipeg, MB
    2 hours ago
  •  ...and Coffee. Position Summary Join our AI team as an Intermediate AI Engineer, where you'll build and deploy AI applications, agents, and automation solutions that solve real...  ...requirements into technical solutions. • Evaluate, monitor, and optimize models and applications... 

    Price Industries Limited

    Winnipeg, MB
    7 days ago
  • $140k - $175k per year

    Job Description Thomson Reuters Labs is seeking a Lead Research Engineer to lead the design and delivery of production-ready software for artificial intelligence (AI) model evaluation and benchmarking. You will build scalable evaluation infrastructure that enables research... 
    Internship
    Work at office
    Local area
    Flexible hours

    Thomson Reuters

    Winnipeg, MB
    2 hours ago
  • Cresta# AI Agent Evaluation Engineer Remote-friendly Canada (Remote); United States (Remote)AT A GLANCEBuild and operate evaluation systems to measure quality of production LLM-based conversational AI agents through annotation, automated judges, and rigorous testing.LEVELmid... 
    Full time
    Remote work

    JoinStartup

    Winnipeg, MB
    2 hours ago
  • Secure AI agents and LLMs in Toronto through cloud security architecture, guardrails, and protected non-human identities. This is a hybrid, full-time opportunity.As an AI security specialist, you will design protections across enterprise finance, ERP, and aerospace and defense... 
    Full time

    Periscope Search Group

    Winnipeg, MB
    2 hours ago
  • Apple’s Creativity Apps team seeks a Machine Learning Engineer focused on evaluating ML models and AI features in our creativity tools. You will define evaluation metrics, build evaluation datasets, and develop scalable infrastructure to ensure quality from model development... 

    Lex

    Winnipeg, MB
    2 hours ago
  • Build AI agents for renewals and expansion at ShareGate, using low-code orchestration, Python or TypeScript, and SQL remotely across Canada...  ..., marketing operations, customer success operations, or GTM engineering backgroundHands-on n8n, Make, Zapier, or similar experiencePython... 
    Remote work

    Sharegate

    Winnipeg, MB
    2 hours ago
  • Design AI-agent workflows with Claude Code while building reliable engineering harnesses for software-development workflows in Toronto.This Agentic Engineer role focuses on introducing modern agentic engineering practices into a complex existing engineering organization. You... 

    TEEMA Group

    Winnipeg, MB
    2 hours ago
  • Build production AI agents and order management systems at NewAge Products, working remotely...  ...automation through hands-on software engineering.As an AI Engineer, you will design, test...  ...databases, cloud services, and production evaluation.Key Responsibilities: Design order... 
    Remote work

    NewAge Products Inc.

    Winnipeg, MB
    2 hours ago
  •  ...vision to life!We're building BioRender's AI figure agent, which can create any scientific figure...  ....What you'll be doing Lead a small engineering team responsible for the core agent...  ...Continuously improve figure quality through evaluations, experimentation, and real user... 
    Remote work

    BioRender

    Winnipeg, MB
    2 hours ago
  • The Frontier AI Deployed Engineer is a member of Future Electronics' AI & Automation Center of Excellence and works directly with...  ...to a solution architecture or technology. Rapidly evaluate emerging AI tools, models, agent capabilities, development methods, and integration... 
    Work at office
    Remote work

    Future Electronics

    Winnipeg, MB
    2 hours ago
  •  ...Will Do Lead, hire, develop, and scale AI engineering teams and technical leaders supporting...  ...engineering quality for LLM-based systems, agents, and integrations, from architecture and...  ...of the CoE's shared AI platform: evaluation pipelines, behavioral and quality testing... 
    For contractors

    GoodRx, Inc.

    Winnipeg, MB
    2 hours ago
  • Lead AI figure agent engineering at BioRender, focusing on multi-agent orchestration and dependable scientific figure generation. This remote-first...  ...reliability. The work combines frontier LLM and VLM evaluation, model routing, infrastructure, monitoring, and user feedback... 
    Remote work

    BioRender

    Winnipeg, MB
    2 hours ago
  • $250k per year

     ...leading security-first enterprise AI company. We build cutting-edge...  ...is a team of researchers, engineers, designers, and more, who are...  ...Technical Staff in the Safety for Agents team, you will make a...  ...post-training algorithms, and evaluation methods to ensure Safety in the... 
    Full time
    Work at office
    Local area
    Remote work
    Home office

    Cohere

    Winnipeg, MB
    2 hours ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Ai Agent Evaluation Engineer. Be the first to apply!