AI Agent Evaluation Engineer - Remote
YO AI Labs
Vancouver, BC
- Remote job
Senior Software Engineer
Job Type: Contractor (~15 hours/week)
Location: Remote
Job Summary
We are seeking experienced Senior Software Engineers to support an AI training project by creating reinforcement learning environments that evaluate AI models on complex software engineering tasks using Model Context Protocol (MCP) tools.
You will design reproducible environments, deterministic verification, and reference solutions for tasks such as bug fixing, feature implementation, codebase refactoring, and performance optimization . No prior AI experience is required.
Key Responsibilities
- Create reinforcement learning environments for software engineering tasks.
- Design tasks involving bug fixing, feature development, refactoring, and performance optimization .
- Build deterministic verification systems and golden reference solutions.
- Evaluate AI agents' ability to reason through complex codebases and use MCP tools effectively.
- Develop realistic, reproducible software engineering scenarios.
- Ensure tasks accurately measure coding ability, problem-solving, and tool usage.
- Document solutions and provide clear technical feedback.
Required Skills
- Strong proficiency in Python 3, Java, Rust, C++, or TypeScript .
- Strong understanding of algorithms and data structures .
- Experience with bug fixing and debugging complex software issues .
- Proven experience in feature implementation and codebase refactoring .
- Strong knowledge of performance optimization and tuning .
- Excellent written and verbal communication.
- Strong attention to detail.
Preferred Qualifications
- Experience working with large-scale or distributed codebases .
- Familiarity with AI/ML systems is a plus but not required.
- Experience with rigorous code reviews and software engineering best practices .
- Experience working effectively in remote or cross-functional teams.
Hiring Process
- Submit an application and screening questions.
- Complete an AI interview (~30 minutes).
- Complete a technical assessment, if required.
- Hiring Manager review.
Compensation
Compensation is output-based , with payment provided per task that meets project specifications. Minimum weekly submission requirements may apply.
Availability
Selected experts should be prepared to begin their first tasks within 24–48 hours of completing onboarding .
Vacancy posted 4 days ago
Similar jobs that could be interesting for youBased on the AI Agent Evaluation Engineer - Remote in Vancouver, BC vacancy
$115.5k - $132.75k per year
...About Dialpad Dialpad is the AI platform for customer experience, built to resolve... ...real time across voice and digital. Our AI agents learn from your best human agents and... ...Empathetic . Your role As an AI Evaluation Engineer, you'll be an integral part of our AI Evaluation...SuggestedInternshipWork at office$238k - $270k per year
...We're building AI Teammates: agents that work like actual users in Asana and integrated apps. They... ...work done. We're looking for an Engineering Manager to lead the Agent Orchestration... ...Experience with production observability and evaluation frameworks for AI systems At Asana,...SuggestedLong term contractWork at officeLocal areaWork from homeWorldwideShift work$138k - $208k per year
...Fortune 500 company and a leading AI platform for managing people money and agents were shaping the future of work so... ...product leaders machine learning engineers and full-stack builders to create... ...pipelines workflow orchestration evaluation and feedback loops ensuring solutions...Remote workFull timeWork at officeHome officeFlexible hours$102.55k - $146.5k per year
...excited about turning emerging AI technologies into real... ...to make your mark. As our AI Engineering Manager , you'll lead a talented... ..., challenge assumptions, and evaluate solution approaches, your greatest... .... Assist with vendor evaluations, partner performance reviews,...Suggested- ...software teams have about their code, so engineers can spend less time in meetings and... ...behind them. You live in modern AI developer tools: Claude Code, Cursor,... ...across prompting, context engineering, agent architectures, and evaluation frameworks Work across frontier models...SuggestedLong term contractFull timeLive InWork at officeFlexible hours
$16.2 - $19.2 per hour
...Automatic Collision Notification, Hands-free Destination Assist, Stolen Vehicle Locator, and more. In this role, you’ll listen carefully and evaluate each caller’s situation to provide the right solutions, from simple navigation questions to emergency situations. Fluency in both...Remote workContract workFor contractorsWork from homeFlexible hoursNight shiftWeekend work- ...Insurance Company’s distribution network, is expanding and seeking motivated individuals to join our team! About the Role: As a Remote Insurance Benefits Specialist , you’ll be the first point of contact for families seeking life insurance coverage. No prior insurance...Remote jobWeekly payFull timeContract workWork from homeLong distanceFlexible hours
$180k - $225k per year
...This role sits at the intersection of AI engineering and application development. You’ll build customer... ...RAG, tool calling, memory, or multi-agent patterns to real users and you know where... ...candidates need to be based in Canada. Remote is fine - you can expect travel to Vancouver...Remote workWork at office$120k - $150k per year
...Overview Diligent's Commercial AI Transformation function builds AI agents that automate real workflows across... ...at volume, and genuine software engineering discipline (version control, testing... ...notes. All final screening, evaluation, and hiring decisions are made by...InternshipWork at officeLocal areaVisa sponsorshipFlexible hours$100k - $150k per year
...to work from Vancouver . Software Engineer Platform AI Service Levels: Software Engineer II... ...not just building the future - youre an agent of positive change joining a global... ...summarizing interview notes. All final screening evaluation and hiring decisions are made by humans...Full timeWork at officeLocal areaVisa sponsorshipFlexible hours$114.24k - $140k per year
...We're hiring a dedicated data engineer to own the production data platform... ...automated reporting, AI-assisted workflows, and predictive... ...status. ~ Connect approved AI agents to governed data with... ...interview notes. All final screening, evaluation, and hiring decisions are made...Work at officeLocal areaVisa sponsorshipFlexible hours- ...Toptal is the world’s largest fully remote workforce. We take the best... ...(MSP) at the forefront of AI-driven IT service transformation... ...for clients: Now Assist, Virtual Agent, Predictive Intelligence, ITOM... ...and co-sell opportunities. Evaluate and manage relationships with complementary...Remote workShift work
$110k - $160k per year
...entering a new phase of growth. Smart Suite, our AI-powered agentic platform, is redefining... ...will join TrustFlight as an AI Software Engineer and serve as a core member of the AI Team.... ...through production deployment, evaluation, and monitoring. Design and implement scalable...Permanent employmentFull timeWork at office$250k - $310k per year
...We are seeking a hands-on Director of AI Software Engineering to lead and scale AI engineering efforts... ...and validate new approaches (LLMs, agents, retrieval systems, classification pipelines... ...interview notes. All final screening, evaluation, and hiring decisions are made by...Work at officeLocal areaVisa sponsorshipFlexible hours$171.1k - $227.5k per year
...building and running the world's best data and AI infrastructure platform so our customers... ...to improve their business. Founded by engineers — and customer obsessed — we leap at every... ...allow users to define business metrics and agent metadata in centrally governed systems for...Long term contractWorldwide$161.5k - $191.5k per year
...About Dialpad Dialpad is the AI platform for customer experience... ...voice and digital. Our AI agents learn from your best human agents... .... Your Role As an AI Engineer on our Speech Team, you'll own... ...with versioning and a rigorous evaluation harness. Conversational turn...Manual laborWork at office$60k - $70k per year
...phase of growth. Smart Suite, our AI-powered agentic platform, is... ...not hiring for a traditional engineering role either. We're hiring an... ...leadership. You'll be working on AI agents, marketing automation, data... ..., you'll be building the GTM engine for a platform serving 1,600+...Permanent employmentFull timeInternshipWork at office$161.5k - $191.5k per year
...About Dialpad Dialpad is the AI platform for customer experience... ...voice and digital. Our AI agents learn from your best human agents... .... Your Role As an AI Engineer on our Speech Team, you'll own... ...with versioning and a rigorous evaluation harness. Conversational turn...Manual laborWork at office$131k - $180k per year
...: About Us Diligent is the AI leader in governance, risk and... ...and move fast. Whether you're an engineer working with large language... ...building the futureyou're an agent of positive change, joining a global... ...implementations Monitor and evaluate the performance and...Full timeWork at officeLocal areaFlexible hours$4000 per month
...investments. Built on a native, AI-powered platform and more than... ...looking for an AI Automation Engineer Co-op to embed with teams... ...grows. Wire together LLMs, agents, and prompt design patterns with... ...location, and scope. We evaluate compensation bi-annually, including...Remote workHourly payLong term contractPermanent employmentFor contractorsFreelanceLocal area$135k - $165k per year
...We're looking for a Principal, AI Transformation - Sales to join... ..., low-code automation , and agent configs . Be the single... ...monitor emerging AI developments, evaluate their potential, and adapt them... ...enough to tell a data engineer what's actually needed to fix it...Full timeFlexible hours$150k - $177k per year
...hardware, software, and increasingly AI, coupled with the patient-... ...and semi-autonomous AI agent systems utilizing multi-step reasoning... .... Comprehensive NLP & LLM Evaluation: Establish robust, automated,... ...clinical, annotation, and ML Engineering teams, raising the code-...Immediate startFlexible hours$42k per year
...Opportunity Kubera Payments, a growing and fast-paced small business in the payments technology space, needs a Sales Support Agent for a full-time remote position. Our ideal candidate will have excellent organizational skills, attention to detail and computer skills along with...Remote workPermanent employmentFull timeCasual workWork from homeMonday to fridayDay shift- ...Treasure AI: Treasure AI is the agentic experience platform built to acquire, retain, and grow your most valuable customers. Powered... ..., openness, and humility. Your Role: Staff Software Engineers at Treasure AI are pod leads — player-coaches who remain active contributors...Contract workWork at officeNight shift
- ...Full-time | Part-time | Commission-based Experience Level: New Agents, Experienced Agents, or Those Planning to Become Licensed Grow your... ...a business. Innovative Tech Ecosystem Work smarter with AI-powered rlpSPHERE’s built-in ecosystem with website, CRM, nurturing...Full timePart timeLocal areaShift work
$131k - $164k per year
...business problems into working AI solutions — and you’re not... ...Finance, HR, Legal, and Product & Engineering. You’ll set the AI vision... ...(such as RAG, copilots, agents, summarization, and classification... ...workflows, prompt engineering, and evaluation, with enough technical depth...Work at officeLocal areaVisa sponsorshipFlexible hours$134k - $160k per year
...revenue company with 7000+ employees on all continents. Our leading AI technology is the backbone of our award-winning enterprise... ...Copperleaf’s portfolio. Program Management is a function within Engineering and R&D. This role will be assigned to priority initiatives based...Full timeWorldwideFlexible hours- ...construire. Nous pensons que les processus d’admission aux études supérieures ont un mandat d’aller au-delà des notes académiques, et d’évaluer le potentiel du candidat en considérant la personne dans son entièreté : son expérience personnelle, ses forces et ses qualités...Contract workApprenticeshipCasual workWork at officeFlexible hoursDay shift
$55k - $65k per year
...le site . L’occasion: Nous sommes à la recherche d’un(e) agent(e) de comptes bilingue pour notre équipe Services de correspondance... ...renseignements personnels que vous avez fournis dans le but d’évaluer vos qualifications et votre aptitude à occuper un emploi à Aviso....Daily paidFull timeApprenticeshipSummer workWork at office$100k per year
...Tenstorrent is leading the industry on cutting-edge AI technology, revolutionizing performance expectations, ease of use, and cost efficiency... .... Tenstorrent is seeking an AI Processor IP Product Engineer to be the technical bridge between our cutting-edge AI processor...Permanent employment
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Agent Evaluation Engineer - Remote. Be the first to apply!
Related searches
- agent de liaison Vancouver, BC
- signing agent Vancouver, BC
- parts representative Vancouver, BC
- work from home chat agent Vancouver, BC
- security agent Vancouver, BC
- agent aux communications Vancouver, BC
- agent de recherche Vancouver, BC
- de représentant Vancouver, BC
- représentant développement des affaires Vancouver, BC
- agent méthode Vancouver, BC