AI Agent Evaluation Engineer
Outlier
Evaluate AI agents remotely using debugging, stress testing, and technical analysis for Outlier’s large language model projects.
Outlier collaborates with AI organizations to improve autonomous systems through human feedback. This freelance work focuses on teaching agents to design, coordinate, and optimize complex architectural workflows. Candidates need software experience with live environments and multi-turn interactions.
Key Responsibilities:
- Develop evaluation rubrics for agent performance
- Inspect and debug detailed agent traces
- Stress test autonomous agents across workflows
- Assess complex system behaviors with precise feedback
Requirements:
- Two or more years of relevant engineering experience
- Production-grade software development with modular separation
- Experience with live, non-mocked environments
- Comfort handling multi-turn system interactions
- Strong attention to technical detail and failure modes
Use AI evaluation, trace debugging, and systems expertise to improve autonomous agent quality through Outlier projects.
#J-18808-LjbffrVacancy posted 5 hours ago
Similar jobs that could be interesting for youBased on the AI Agent Evaluation Engineer in Ottawa, ON vacancy
- ...Evaluate AI agents remotely through rubric writing, trace debugging, and stress testing. Outlier offers freelance work for experienced software engineers. This AI Agent Evaluation Engineer opportunity supports Outlier projects training Large Language Models as proactive,...SuggestedFreelanceRemote work
Outlier
Ottawa, ON5 hours ago - ...Evaluate autonomous AI agents remotely using technical feedback, trace debugging, and stress testing for Outlier’s generative AI projects. Outlier... ...with Supabase, Gmail, or APIs welcomed Use agent evaluation, systems integration, and failure analysis expertise to improve...SuggestedFreelanceRemote work
Outlier
Ottawa, ON5 hours ago - ...Evaluate AI agents for Outlier through rubric writing, trace debugging, and stress testing. Work remotely and freelance on a schedule you choose... ...coordinate real-world, multi-step workflows through detailed evaluation. Key Responsibilities: Create rubrics that assess agent...SuggestedFreelanceRemote work
Outlier
Ottawa, ON5 hours ago - ...Evaluate AI agents through rubrics, trace debugging, and stress testing while working remotely for Outlier. Use GPT, Claude, and other leading... ...AI agents with detailed human feedback. This freelance engineering work supports systems designed as proactive, multi-step agents...SuggestedFreelanceRemote work
Outlier
Ottawa, ON5 hours ago - ...Evaluate AI agents remotely for Outlier using technical feedback, trace debugging, and stress testing across real-world interactions. Outlier... ...from multi-step system interactions Write clear, precise evaluation rubrics Identify subtle failures in agent behavior Requirements...SuggestedFreelanceWork at officeRemote work
Outlier
Ottawa, ON5 hours ago - ...Evaluate autonomous AI agents through rubric writing, trace debugging, and stress testing while working remotely as a freelance contributor for Outlier... ...AI systems Requirements: Experience with backend engineering, AI automation, or systems integration Ability to...FreelanceRemote work
Outlier
Ottawa, ON5 hours ago - ...Evaluate advanced AI agents remotely with Outlier, applying technical feedback, debugging, and complex systems integration expertise. This freelance... ...-world architectural workflows. The work includes writing evaluation rubrics, debugging agent traces, and stress testing agent...FreelanceRemote work
Outlier
Ottawa, ON5 hours ago - ...Evaluate autonomous agents through rubric writing and trace debugging with Outlier, working remotely on flexible freelance projects... .... Outlier helps innovative companies improve AI agents through human feedback. This engineering work supports large language models designed for...FreelanceRemote workFlexible hours
Outlier
Ottawa, ON5 hours ago - ...Help Outlier improve autonomous agents using agent debugging, stress... ...opportunity supports AI organizations training large language... ...proactive, multi-step tasks. You will evaluate complex architectural... ...Two or more years in backend engineering or AI automation Experience...FreelanceRemote work
Outlier
Ottawa, ON5 hours ago - ...Apply AI automation and backend engineering skills to evaluate OpenClaw agents remotely through Outlier’s freelance contributor platform. Weekly payments are available. Outlier collaborates with leading AI organizations to improve large language models through human feedback...Weekly payFreelanceRemote work
Outlier
Ottawa, ON5 hours ago - ...Evaluate advanced AI agents remotely with Outlier, focusing on technical feedback, agent traces, and real-world system behavior. Work freelance... ...environments Requirements: Two-plus years in backend engineering, AI automation, or systems integration Experience building...Contract workFreelanceRemote work
Outlier
Ottawa, ON5 hours ago - ...Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. This is a task-based... ...clinical EHR vignettes for a medical AI evaluation project. General and internal medicine are the...Part timeFor contractorsRemote workFlexible hours
- ...Miratech is looking for a skilled Senior AI Engineer with Python, LangGraph and LangChain to... ...Responsibilities Develop and enhance AI agents and orchestration capabilities using... ..., monitoring, telemetry, and evaluation platforms. ~ Strong knowledge of software...Full timeInternshipRemote workRelocation
$104k - $154k per year
...Huawei Canada has an immediate 12-month contract opening for an Engineer. About the team: Welcome to the Advanced Wireless... ...solutions. About the job: Drive the evolution of multi-agent communication in AI networks through advanced solution development and system-level...Permanent employmentFull timeContract workInternshipImmediate startusd15 per hour
...online by using just a few words? Do you want to influence the future of global AI search? If so, we want you on board! Working on behalf of RWS, you will test and evaluate how our clients’ search engine responds to everyday searches from everyday people. Using your search...Weekly payFull timePart timeFreelanceSecond jobRemote workWork from homeHome officeFlexible hours$142.05k - $292k per year
...delivery role. It is not a research position. A Forward Deployed AI Engineer is a production engineer who works embedded inside a client's... ...prototyping, iteration, and deployment of AI-enabled workflows, agents, copilots, and automations into real operational environments...Full timeInternshipWork at officeLocal area- ...service à la clientèle sont essentiels - De bonnes aptitudes à la communication et aux relations interpersonnelles sont nécessaires pour évaluer les besoins des clients et formuler des recommandations - Une certaine expérience dans un rôle axé sur la clientèle est requise - La...
$75k - $125k per year
...: We are seeking Intermediate Agentic AI Developers / AI Engineers to support an AI modernization initiative... ...to design, develop, and enhance AI agents and intelligent automation solutions using... ...orchestration capabilities. Evaluate, troubleshoot, and optimize AI agent accuracy...Full time- About Fullscript We’re an industry-leading health technology company on a mission to help people get better. We started in 2011 with one simple idea. Make it easier for practitioners to access the products they trust so they can deliver better care. That simple idea grew into...Full time
- ...Full-time | Part-time | Commission-based Experience Level: New Agents, Experienced Agents, or Those Planning to Become Licensed Grow your... ...a business. Innovative Tech Ecosystem Work smarter with AI-powered rlpSPHERE’s built-in ecosystem with website, CRM, nurturing...Full timePart timeLocal areaShift work
$85k - $111.3k per year
...tested, and maintained through Generative AI and agentic software engineering. We are seeking an Applied AI... ...use Large Language Models (LLMs), AI agents, knowledge systems, and software-analysis... ...use cases, prototype solutions, evaluate their impact on development efficiency...Long term contractFull timeTemporary workManual labor- ...over 55 years. Backed by a legacy of engineering excellence, reliability and industry-leading... ...operations and end-to-end performance evaluation of the Telesat Lightspeed satellite... ...(IDEs). Familiarity with agentic AI, including AI agents and their use in engineering, software...Long term contractWorldwide
$69.45k - $119.45k per year
...business with Data & Advanced AI — backed by a $3B investment and... ...spanning LLM applications, agent orchestration, enterprise knowledge... ..., workflow automation, evaluation, observability and AI Ops. You... ...selection, prompt and context engineering, advanced RAG, tool/function calling...Full timeManual laborWork at officeLocal area- .... From advancing healthcare and scientific discovery to powering AI and the technologies people rely on every day, innovation at AMD... ...career. THE ROLE: We are looking for a talented ASIC RTL Design Engineer to join our IP design team developing high-performance networking...Full timeInternship
$69k - $86k per year
...software test procedure through Agentic AI-driven approach, such as autonomous agents, intelligent automation,... ...Sc in Telecommunication/Electrical Engineering /Computer Science or equivalent experience... ...AI engineering practice: agent evaluation, guardrails, observability, cost...Long term contractFull timeTemporary work$74.3k - $99.06k per year
...Nous recherchons un(e) Associé(e) principal(e), Impact ou d’un(e) agent(e), Climat et impact ! Rejoignez FinDev Canada, l’institution... ...L’équipe de l’Impact du développement est responsable de l’évaluation, de la mesure et du suivi des retombées sur le développement des...Permanent employmentFull timeContract workApprenticeshipWork at officeWorldwideDay shift- ...We are seeking a senior Professional Engineer who has built the majority of their career... ...and operational issues. ~ Develop and evaluate future growth, climate, operational, infrastructure... ..., scripting, automation tools, or AI agents to improve individual productivity and...Permanent employmentFull timeWork at office
$100k - $150k per year
...Category Engineering Hire Type Employee Job ID 19424 Base Salary Range... ...customers to rapidly innovate AI-powered products. We deliver... ...to drive new features, evaluate technologies, and integrate enhancements... ...with AI technologies, LLMs, agent-based systems, or machine...Remote workWorldwide$18.5 per hour
...abstract Benefits: Paid training Flexible Schedule Career advancement opportunities Position Summary: A Signal Patrol Agent will conduct patrols, (foot and vehicle) for courtyards, offices, pools, and other high value areas to ensure all required access...Full timeTrial periodFlexible hoursShift workNight shift- ...About FullStack FullStack is your AI-native engineering partner, built to turn AI capability into certainty. Most companies can demo AI; few... ...mentoring senior engineers, driving technical best practices, and evaluating trade-offs for org-wide initiatives. ~ Experience with...Full timeRemote workFlexible hours
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Agent Evaluation Engineer. Be the first to apply!
Related searches
- représentant développement des affaires Ottawa, ON
- enrolled agent Ottawa, ON
- représentant pharmaceutique Ottawa, ON
- de représentant Ottawa, ON
- représentant médical Ottawa, ON
- agent aux communications Ottawa, ON
- représentant interne Ottawa, ON
- security agent Ottawa, ON
- quality control representative Ottawa, ON
- vendeur représentant Ottawa, ON