AI Agent Evaluation Engineer
Outlier
Evaluate AI agents through rubrics, trace debugging, and stress testing while working remotely for Outlier. Use GPT, Claude, and other leading LLMs.
Outlier helps innovative companies improve AI agents with detailed human feedback. This freelance engineering work supports systems designed as proactive, multi-step agents coordinating complex real-world workflows. The work suits professionals with backend engineering, AI automation, or complex systems integration experience, especially those comfortable reviewing technical behavior.
Key Responsibilities:
- Create evaluation rubrics for agent performance
- Investigate and debug agent traces
- Stress-test agents in live environments
- Assess complex system behaviors with technical precision
Requirements:
- Two or more years of relevant engineering experience
- Production-grade software development with modular separation
- Practical experience with non-mocked systems
- Strong attention to detail and technical feedback skills
- Experience handling multi-turn interactions
Bring backend engineering and AI evaluation expertise to Outlier projects improving autonomous agent reliability and real-world performance.
#J-18808-LjbffrVacancy posted 7 hours ago
Similar jobs that could be interesting for youBased on the AI Agent Evaluation Engineer in Montréal, QC vacancy
$60 per hour
...Agent(e) de relations humaines (ARH) Conditions d’emploi : Adaptation et soutien en déficience (DI-DP), notamment... ...principales : Vos tâches constituent entre autres : Évaluer les besoins psychosociaux, adaptatifs et fonctionnels des jeunes...Suggested- ...Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. This is a task-based... ...clinical EHR vignettes for a medical AI evaluation project. General and internal medicine are the...SuggestedPart timeFor contractorsRemote workFlexible hours
- ...Miratech is looking for a skilled Senior AI Engineer with Python, LangGraph and LangChain to... ...Responsibilities Develop and enhance AI agents and orchestration capabilities using... ..., monitoring, telemetry, and evaluation platforms. ~ Strong knowledge of software...SuggestedFull timeInternshipRemote workRelocation
- ...Agent(e) de relations humaines Conditions d'emploi : Disposé(e) à travailler dans le Nord-du-Québec et idéalement détenteur... ...Québec. Le ou la titulaire du poste exerce des activités d’évaluation, d’orientation, de consultation, d’analyse, d’intervention, d’...SuggestedFull timePart timeNight shiftAfternoon shift
- ...innovation and a willingness to challenge consensus. The Team: As AI tools drive immense value across the firm, our organization faces... ...formed AI Platform Team owns this unified mission. While the engineering side of our team builds the platform rails, the AI Platform...Suggested
- ...that deliver healthcare. Our AI sensing platform enables... ...'re looking for an Applied AI Engineer to take our growing collection... ...model landscape cold, treats evaluation as a first-class engineering problem... ...-party APIs Build RAG and agent-style orchestration for clinical...
$80k - $155k per year
...modernise des applications avec l’appui d’agents de code IA À propos de notre pratique... ...IA peuvent être utilisés pour aider à l’évaluation des candidatures, sans jamais remplacer... ...et en management au monde. Software Engineer – AI-Assisted Development Position Description...- ...challenge consensus. The Team: As AI tools drive immense value across the firm,... ...The Role: We are looking for a Platform Engineer to help build and operationalize our AI... ...Responsibilities: Configure and manage multi-engine API gateways, implementing dynamic routing...
- ...ASICC-MED placement médical et soins infirmiers recrute actuellement un Agent de placement. La personne devra être disponible pour l’horaire de 3 soirs flexibles en semaine terminant à 21h00 et incluant les fins de semaine. Nous recherchons du personnel à l’aise avec...Full timeWork at officeFlexible hoursWeekend workAfternoon shiftWeekday work
- ...ASICC-MED placement médical et soins infirmiers recrute actuellement un Agent de télémarketing. La personne devra être disponible pour l’horaire de jour terminant à 16h30 du lundi au vendredi. Nous recherchons du personnel à l’aise avec les nouvelles technologies, soucieux...Full timeWork at officeMonday to fridayDay shift
- ...breakthrough outcomes for their toughest AI challenges. We blend the power... .... As a Senior Lead AI Engineer , you are primarily... ...Building and deploying LLM-based agent systems in production Keeping... ...Experience designing and operating evaluation frameworks and observability...Full timeWork at officeLocal areaRemote workFlexible hours3 days per week1 day per week
- ...Votre rôle : Vous souhaitez jouer un rôle clé dans le bien-être et la sécurité des employés ? À titre d’agent(e) de gestion du personnel - au service de santé, vous serez au cœur des enjeux liés à la présence au travail, à la prévention et à la gestion des invalidités....Permanent employmentFull time
$150k - $195k per year
...About You We are looking for a Principal Engineer to take architectural ownership of the agent-readable Shop Builder platform. This is the most senior individual contributor role on the DevX and AI team, and the decisions you make here — which surfaces to expose, which formats...Local areaWorldwide$80k - $155k per year
...assistée par l’IA Description de poste Senior · Évalue les risques et corrige les vulnérabilités avec l’appui d’agents de code IA À propos de notre pratique Au... ...management au monde. Application Security Engineer – AI-Assisted Remediation Position Description...- ...Agent(e) Service à la clientèle Vous êtes à la recherche d'opportunité de carrière dans le domaine de l’administration ? Nous recherchons présentement, une personne passionnée pour combler le poste d’agent (e) Service à la clientèle au sein au sein de notre équipe...Full timeWork at officeMonday to friday
- ...communication technology. The Role We are seeking a highly versatile, technically advanced, and commercially astute Forward Deployed AI Value Engineer to join our centralized AI Value Realization team. In this elite, high-impact role, you will act as a technical and strategic...Long term contractFull timeContract workInternshipManual labor
usd218k per year
...technical leadership to team for design of multiple products Evaluates and recommends adoption of open source Evaluates third party... ...internal technical teams Software Reliability & Support Engineering Manages internal/external customer relationships and communication...Full timeWork at office- ...We are hiring a Senior Machine Learning Engineer Scientist to lead the development of scalable... ...for experimentation, reproducibility, evaluation, and deployment. Define and execute the... ...Passion for advancing structured-data AI and building platforms that deliver real-world...Long term contractPermanent employmentFull timeLocal areaWorldwideFlexible hours
- ...is a next-generation data and AI consulting firm dedicated to accelerating... ..., AI technologies, data engineering, and digital transformation.... ...enterprise data, multi-step agents that automate business... ...pipeline feeding it, and the evaluations proving it works. This role...Full time
- ...works and what it costs us to run. That means building the evaluation infrastructure that tells us whether a model change made... ...without degrading quality. This is a measurement-first engineering role. Today we ship AI features and form opinions about quality from anecdotes....Full timeLive InRemote work
- ...growth. The Mission We’re building AI startups inside an established portfolio of vertical software companies—and we need engineers who can make ideas real, fast. Our mandate... ...building agentic workflows: multi-step AI agents using function calling, tool use, and...Long term contractInternshipWork at officeImmediate startRemote work
$19 per hour
...social de Ville Saint-Laurent - Du mercredi au vendredi : à distance via Microsoft Teams - Des questionnaires de connaissances et une évaluation finale devront être réussis avec une note minimale de 85 % - Une révision supplémentaire du matériel de formation à l’extérieur des...Contract workTemporary workWork at officeRemote workMonday to friday2 days per week1 day per weekWeekday work$50 - $150 per hour
...Mechanical Engineering AI Expert Remote | Contractor | $50–$150 per hour equivalent We are recruiting experienced Mechanical Engineering... ...AI Experts to support a large-scale project helping train and evaluate next-generation AI systems. No previous AI experience is...Remote jobHourly payContract workFor contractors- ...What You'll Be Doing Shape how AI agents and assistants work in our SDLC: where they... ...in platform, DevOps, or infrastructure engineering, with real software development along the... ...de leurs clients, vérifier les revenus, évaluer le risque de crédit et accélérer les paiements...Full timeLive InLocal area
- ...enterprises who are building AI systems to power magical experiences... ..., semantic search, RAG, and agents. We believe that our work is... ...is a team of researchers, engineers, designers, and more, who are... ...post-training algorithms, and evaluation methods to ensure Safety in the...Full timeWork at officeRemote workFlexible hours
- ...Job Description We are seeking a Software Safety Engineer with expertise in Artificial Intelligence (AI), Large Language Models (LLMs), and software safety... ...and ALM tools, including Jira, Jama, and Polarion. Evaluate the accuracy and coverage of automated analyses, targeting...Permanent employmentFull timeContract workFlexible hours
- Nous sommes présentement à la recherche d’un(e) Agent(e) à la conformité dynamique afin de joindre notre bureau de Montréal. Se rapportant au Directeur général, Opérations de Montréal, le titulaire du poste d’Agent (e) à la conformité coordonne et contrôle toute la documentation...Permanent employmentWork at office
- L’agent(e) à la réception est responsable de fournir un service hors pair à nos invités, en effectuant les arrivées et les départs, et en offrant des renseignements sur l’hôtel et la ville. L’agent(e) à la réception est appelé(e) à écouter et résoudre des problèmes, tout en...Daily paidRemplacementPermanent employmentFull timeNight shift
- Agent(e) à la réception Le Courtyard Marriott est situé au cœur de Montréal, à proximité du Palais des congrès, à quelques minutes de la plus célèbre ville souterraine au monde, de la Place des Arts, des magasins et boutiques sur l’effervescente rue Ste-Catherine et du...Daily paidPermanent employmentWeekend workAfternoon shift
$66.22k - $110.91k per year
...organisation, vous développez une offre qui réponde aux attentes des citoyens. Plus spécifiquement, vous : identifiez, analysez et évaluez les besoins des clientèles à l'intérieur de votre secteur ainsi que les créations et les productions culturelles existantes afin de...Permanent employmentFull timeRemote workWeekend workAfternoon shiftVille de Montréal
Montréal, QC7 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Agent Evaluation Engineer. Be the first to apply!
Related searches
- agent numerisation Montréal, QC
- représentant développement des affaires Montréal, QC
- agent télémarketing Montréal, QC
- agent nettoyage Montréal, QC
- enrolled agent Montréal, QC
- agent de cinema Montréal, QC
- représentant pharmaceutique Montréal, QC
- de représentant Montréal, QC
- representant cosmetique Montréal, QC
- représentant médical Montréal, QC