Autonomous Agent Evaluation Engineer
Outlier
Evaluate autonomous agents with backend engineering expertise, debugging traces and stress testing systems through Outlier’s remote freelance platform.
Outlier collaborates with leading AI organizations to improve large language models through human feedback. This freelance work centers on teaching proactive, multi-step agents to coordinate complex architectural workflows. Candidates need at least two years in backend engineering, AI automation, or complex systems integration, plus production software experience.
Key Responsibilities:
- Develop rubrics for evaluating agent responses
- Investigate failures within agent traces
- Stress test autonomous agent workflows
- Assess behavior across live system interactions
- Deliver high-density technical feedback
Requirements:
- Two or more years of relevant engineering experience
- Production-grade software development and maintenance experience
- Modular separation across parsing, logic, and reporting
- Practical experience with non-mocked environments
- Ability to identify subtle system behavior failures
Use evaluation engineering, backend development, and detailed feedback to help improve autonomous agents and advanced generative systems.
#J-18808-LjbffrVacancy posted 5 hours ago
Similar jobs that could be interesting for youBased on the Autonomous Agent Evaluation Engineer in Toronto, ON vacancy
$130k - $155k per year
...Fluent is hiring its first dedicated AI Engineer. Over the past year, AI at Fluent has moved... ...pilots to production: a service desk agent that closes tickets without a human touch... ...between a demo and a production system: evaluation, monitoring, cost, and a clear owner. You...SuggestedFull timeWork at officeRemote work$160k - $260k per year
...customers. Cohere is a team of researchers, engineers, designers, and more, who are all... ...experience, creating the platform that powers autonomous AI agents at scale. You’ll work on hard,... ...and state management Prototype and evaluate new technologies, then help decide what...SuggestedFull timeManual laborWork at officeLocal areaRemote workHome office$136k - $187k per year
...mission. If you are too, let's talk. The Agent Gateway Team The Agent Gateway team... ...virtual MCP servers. The Senior Software Engineer Opportunity Okta is looking for a... ...routing, credential resolution, and policy evaluation as agent identity specifications evolve....SuggestedLocal areaWorldwide- ...here, and it's at Cresta. About the role: At Cresta, the AI Agent team is on a mission to create state-of-the-art AI Agents that solve... ..., secure, and reliable. We are looking for a Forward Deployed Engineer (FDE) who is passionate about building and deploying AI agent...SuggestedWork at officeLocal areaRemote workWork from homeHome office
- ...AGENT DEVOPS PLATFORM ENGINEER ROLE DESCRIPTIONS Design and operate AI agents that automate tasks across the software development lifecycle... ...prompt management, tool use, agent architectures, evaluation, observability, and reliability. • Experience with Git...SuggestedPermanent employment
$140k - $175k per year
...Job Description Thomson Reuters Labs is seeking a Lead Research Engineer to lead the design and delivery of production-ready software for artificial intelligence (AI) model evaluation and benchmarking. You will build scalable evaluation infrastructure that enables research...Full timeInternshipWork at officeLocal areaFlexible hours$100 per hour
...Angelo , Larry Summers , and Jack Dorsey . Position: Agent Engineer Type: Contract Compensation: $100–$500/hour... ...in production environments. Ensure agent reliability and evaluate performance improvements. Manage long-running processes...Remote jobContract workSummer work$100 - $150 per hour
...Larry Summers , and Jack Dorsey . Position: Software Engineer Expert Type: Contract Compensation: $100–$150... ...hour Location: Remote Role Responsibilities Evaluate AI-generated artifacts for usability in a software engineering...Remote jobContract workSummer workWork at office$85 per hour
..., and Jack Dorsey . Position: DevOps / SRE / Cloud Engineer (Coding Agent Experience) Type: Contract Compensation: $... ...Responsibilities Use frontier AI coding agents to complete and evaluate complex infrastructure engineering tasks. Review model-...Remote jobContract workSummer work$80 per hour
...Larry Summers , and Jack Dorsey . Position: Data Engineer (Coding Agent Experience) Type: Contract Compensation: $... ...Responsibilities Use frontier AI coding agents to complete and evaluate complex data engineering tasks. Review model-...Remote jobContract workSummer work$100 - $150 per hour
...and Jack Dorsey . Position: Sales Engineering Expert Type: Contract... ...objections, supporting proofs of concept, and evaluating product fit and technical risk. ~ Hands... ...~ Prior experience with AI training , evaluation, or human-data projects....Remote jobContract workSummer work$80 - $120 per hour
...General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey . Position: Engineering / manufacturing / technical operations Evaluator Type: Contract Compensation: $80–$120/hour Location: Remote Role...Remote jobContract workSummer workWork at office- ...projects across multiple manufacturing facilities. As a Project Engineer supporting the Press and Automation team, you will provide on-... ..., or make hiring decisions. All applications are reviewed and evaluated by human recruiters and hiring managers. If our hiring...Long term contractRelocation
Great Lakes Group
Toronto, ON8 days ago $74.3k - $99.06k per year
...Associé(e) principal(e), Impact ou d’un(e) agent(e), Climat et impact ! Rejoignez... ...Impact du développement est responsable de l’évaluation, de la mesure et du suivi des retombées sur... ...une capacité démontrée à mener de façon autonome des évaluations d’impact et à gérer des...Permanent employmentFull timeContract workApprenticeshipWork at officeWorldwideDay shift$30 - $50 per hour
...advantage. Cresta’s unified AI platform combines conversational AI agents, real-time human agent augmentation, and comprehensive... ...cost-effective, secure, and reliable. As a Forward Deployed Engineering Intern on the AI Agent team, you’ll support real-world AI deployments...InternshipWork at office- ...content generation, semantic search, RAG, and agents. We believe that our work is instrumental... .... Cohere is a team of researchers, engineers, designers, and more, who are passionate... ...itself into a corner Own North's agentic evaluation framework, ensuring evals are compatible...Full timeWork at officeRemote workFlexible hours
$69.45k - $119.45k per year
...changing space. You Are: A hands-on CCaaS Engineer with deep technical expertise in Amazon... ...flows, IVR logic, routing strategies, and agent workspace configurations on both Amazon... ...Implement agentic AI capabilities such as autonomous virtual agents and AI-assisted agent copilots...Full timeWork at officeLocal area- ...Magazine and is currently managing over 18 million square feet with a value of over $3.5B. We are currently seeking a Leasing Agent to join our team in Toronto. The Leasing Agent is responsible for the leasing administration of a portfolio of properties for our client...Full timeContract workWork at officeWeekend workAfternoon shiftWeekday work
$100k per year
Some of the best sales conversations Ive had didnt feel like selling. Someone told me what they wanted their life to look like and I asked what was stopping them. About us Your Liberating Journey is an international growing business dedicated to leadership development personal...Long term contractFull timeFor contractorsRemote work10 hours per weekFlexible hoursDay shift$60k - $63k per year
...possibility. We are searching for an experienced, Bilingual Agent Case Manager candidate to join our team and contribute to the building... ...consistency, and support fair decision-making. All AI-assisted evaluations are conducted in accordance with applicable privacy and...Full timeWork at officeLocal areaRemote workShift work$26 - $43 per hour
...Methods Change Management to join our team in Calgary. Reporting to the Lead, Methods, the Methods Change Management Agent will manage and execute complex Engineering Change Notices (ECNs) in our ERP System for DHC to support various aircraft programs and developmental...Hourly payPermanent employmentMonday to friday- ...Description Job Description Senior Full-Stack Engineer (Knowledge Graph Platform) Location: [... ..., structured outputs, guardrails and evaluation hooks. • Implement SSO... ...data modelling). • MCP servers, agent frameworks (LangGraph or similar) or tool...Long term contractPermanent employmentFull timeTemporary workDirect hireImmediate start
NationMind LLC
Toronto, ON4 days ago - ...Job Type: Full-time | Part-time | Commission-based Experience Level: New Agents, Experienced Agents, or Those Planning to Become Licensed Grow your real estate career with powerful tools, proven support, and a trusted brand recognized nationwide. Whether you're newly...Full timePart timeLocal areaShift work
- ...industry leader in pricing sophistication, telematics, and, more recently, device and identity protection. Become an Allstate Sales Agent: Your Next Big Opportunity Are you a driven, self-motivated sales professional who thrives on building your own pipeline and...Full timeImmediate startFlexible hours
- ...Time VACANCY STATUS: We are actively hiring for this position. First impressions count, right? We think so too! The Guest Service Agent is the face of our hotels. You're a people-pleaser who loves to go above and beyond to exceed the needs of our guests. You'll wear...Permanent employmentFull timePart timeApprenticeshipInternshipLocal areaFlexible hoursShift work
- ...Virtual Sales Agent Task Buddie is growing quickly, and we’re looking for driven Sales Agents to join our team. In this role, you’ll play a key part in fueling our growth in developing new business opportunities, connecting with potential clients, and setting up qualified...Full timeRemote work
$172k - $236k per year
...Secures AI Team The Okta for AI Agents Team is building the future of... ...scale. We are a diverse team of engineers, product managers, and designers... ...threat inspection, agent intent evaluation, and behavioral analysis—ensuring autonomous AI agents operate safely within specified...Local areaWorldwideShift work- ...intelligence. Leading infrastructure vendors, engineering firms, and construction companies use... ...Citylitics combines a network of AI agents, fine-tuned models, a proprietary infrastructure... ..., cost, and latency. You'll also own evaluation datasets, LLM-as-judge and rule-based QA...Full timeWork at officeImmediate start3 days per week
- ...content generation, semantic search, RAG, and agents. We believe that our work is instrumental... .... Cohere is a team of researchers, engineers, designers, and more, who are passionate... ...automate and scale security processes, i.e: evaluate open-source vs vendor solutions Gather...Full timeWork at officeRemote workFlexible hours
$140k - $175k per year
...research with AI? Join our Global Westlaw Engineering organization and help shape the next... ...graph–based architectures, and autonomous AI agents with multi-step reasoning; stay hands-... ...Drive innovation and technical strategy – Evaluate and drive adoption of emerging AI techniques...Full timeWork at officeLocal areaWorldwideFlexible hours
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Autonomous Agent Evaluation Engineer. Be the first to apply!
Related searches
- représentant développement des affaires Toronto, ON
- traffic agent Toronto, ON
- enrolled agent Toronto, ON
- représentant pharmaceutique Toronto, ON
- de représentant Toronto, ON
- representant cosmetique Toronto, ON
- représentant médical Toronto, ON
- agent aux communications Toronto, ON
- représentant interne Toronto, ON
- security agent Toronto, ON