AI Agent Evaluation Engineer - Remote

YO AI Labs

Remote job

Senior Software Engineer

Job Type: Contractor (~15 hours/week)
Location: Remote

Job Summary

We are seeking experienced Senior Software Engineers to support an AI training project by creating reinforcement learning environments that evaluate AI models on complex software engineering tasks using Model Context Protocol (MCP) tools.

You will design reproducible environments, deterministic verification, and reference solutions for tasks such as bug fixing, feature implementation, codebase refactoring, and performance optimization . No prior AI experience is required.

Key Responsibilities

Create reinforcement learning environments for software engineering tasks.
Design tasks involving bug fixing, feature development, refactoring, and performance optimization .
Build deterministic verification systems and golden reference solutions.
Evaluate AI agents' ability to reason through complex codebases and use MCP tools effectively.
Develop realistic, reproducible software engineering scenarios.
Ensure tasks accurately measure coding ability, problem-solving, and tool usage.
Document solutions and provide clear technical feedback.

Required Skills

Strong proficiency in Python 3, Java, Rust, C++, or TypeScript .
Strong understanding of algorithms and data structures .
Experience with bug fixing and debugging complex software issues .
Proven experience in feature implementation and codebase refactoring .
Strong knowledge of performance optimization and tuning .
Excellent written and verbal communication.
Strong attention to detail.

Preferred Qualifications

Experience working with large-scale or distributed codebases .
Familiarity with AI/ML systems is a plus but not required.
Experience with rigorous code reviews and software engineering best practices .
Experience working effectively in remote or cross-functional teams.

Hiring Process

Submit an application and screening questions.
Complete an AI interview (~30 minutes).
Complete a technical assessment, if required.
Hiring Manager review.

Compensation

Compensation is output-based , with payment provided per task that meets project specifications. Minimum weekly submission requirements may apply.

Availability

Selected experts should be prepared to begin their first tasks within 24–48 hours of completing onboarding .

Vacancy posted a month ago

Similar jobs that could be interesting for youBased on the AI Agent Evaluation Engineer - Remote in Edmonton, AB vacancy

AI and Data Engineer
...Company Description Blend is a premier AI services provider, committed to co-creating... ...Description We are building a team of engineers in Edmonton, AB, Canada to develop a first... ...on AI/ML components — including LLM-based agents, embedding models, and behavioral simulation...
Suggested
Full time
Internship
Work at office
Weekend work
Blend360
Edmonton, AB
more than 2 months ago
AI/ML Engineer - Remote
...AI/ML Engineer Job Type: Full-Time Location: Remote Job Summary We are seeking an experienced AI/ML Engineer to build and deploy secure, scalable... ...work with LLMs, RAG, prompt engineering, multi-agent systems, and cloud AI platforms to develop production...
Remote job
Full time
YO AI Labs
Edmonton, AB
a month ago
Solution Architect (Senior AI / Technical Lead)
...Bits In Glass (BIG) is a high-growth AI and automation consulting firm with offices... ...making the decisions a single embedded engineer should not make alone: model hosting,... ...Set the standards every pod inherits for agent design, evaluation, observability, security, and cost...
Suggested
Permanent employment
Full time
Internship
Bits In Glass
Edmonton, AB
a month ago
Principal Full Stack Engineer (React & AI-Driven) - Remote - Canada
...About FullStack FullStack is your AI-native engineering partner, built to turn AI capability into... ...driving technical best practices, and evaluating trade-offs for org-wide initiatives.... ...We Offer ~ Competitive pay. ~100% remote work. ~ The ability to work with leading...
Remote work
Full time
Flexible hours
FullStack
Edmonton, AB
4 hours ago
LPN - Edmonton (AIS)
...We use technology and automation to support administrative aspects of our recruitment process. Artificial Intelligence is not used to evaluate, rank, or select candidates. All hiring decisions are made by people. CBI Home Health is committed to a diverse, equitable, and...
Suggested
Full time
Part time
Casual work
Flexible hours
Weekend work
Cbi Home Health
Edmonton, AB
a month ago
Software Development Manager - AI Solutions and Business Intelligence
$140k - $175k per year
...LawDepot’s next generation of AI-powered experiences, reporting... ...Edmonton, Alberta office or fully remote across Canada. What You’ll... ...impact Research, evaluate, and recommend AI tools, reporting... ...in Computing Science, Software Engineering, Artificial Intelligence, Data...
Remote work
Long term contract
Permanent employment
Full time
Bank staff
Work at office
Work from home
Worldwide
Flexible hours
Lawdepot Inc
Edmonton, AB
more than 2 months ago
Platform Engineer
...shaping the future of fleet readiness with AI-enabled, connected fleet technology that... ...Fleetworthy Inc. is hiring a Platform Engineer to own and evolve the systems that connect... ...VPN behavior, and TCP socket data flows in remote environments where recoverability matters...
Remote work
Full time
Fleetworthy
Edmonton, AB
a month ago
Engineer, Planning
...a Full-Time, Temporary (up to 18 months) Engineer, Planning position working out of EPCOR... ...infractions for unsafe driving behaviours will be evaluated and considered for non-selection... ...Employee Mileage Reimbursement Policy. Use of AI: Our recruitment process may use...
Long term contract
Full time
Temporary work
Work at office
Flexible hours
Epcor
Edmonton, AB
a month ago
Engineer, Structural
...a f ull Time, p ermanent Structural Engineer position working out of the Rossdale Water... ...with applicable codes and standards. Evaluate and approve consultant submissions,... ...Employee Mileage Reimbursement Policy. Use of AI: Our recruitment process may use...
Long term contract
Full time
Contract work
Temporary work
For contractors
Work at office
Epcor
Edmonton, AB
a month ago
Application Engineering Technologist
...subsidiaries. At this moment, we need an Applications Engineer to cover the province of Alberta . You will be... ...and training. Mazak is proud to read and evaluate every single profile without any AI technology. We are moved with people, by people and to...
Hourly pay
Long term contract
Full time
Local area
Night shift
Mazak Corp
Edmonton, AB
a month ago
Custom Order Engineer
$69k - $87k per year
...sector is currently seeking a Custom Order Engineer to join our team. This position is based... ...the hiring process. All candidates will be evaluated based on job-related competencies, and all... .... Our Talent Acquisition teams also uses AI to identify applicants whose skills are a...
Full time
Relocation package
Eaton Corporation
Edmonton, AB
28 days ago
Engineer, Asset Management
...are hiring two (2) Full-Time, Permanent Engineer, Asset Management working out of our... ...for unsafe driving behaviours will be evaluated and considered for non-selection regardless... ...Reimbursement Policy. ~ Use of AI: Our recruitment process may use technology...
Remplacement
Permanent employment
Full time
Epcor
Edmonton, AB
a month ago
Design Engineer
...Bradken is a global manufacturer of custom-engineered products and solutions for the resources sector. Our products operate in some of... ...be built, supplied and supported effectively. · Support field evaluations and product reviews through site and manufacturing visits when...
Full time
Work at office
Immediate start
Visa sponsorship
Work visa
Flexible hours
Bradken
Edmonton, AB
a month ago
Guest Service Agent - Matrix Hotel
...proud of, where you are valued, trusted and supported by the team Here is what you will be doing each day: As our Guest Service Agent, you will be working with our Guest Services team to provide excellent customer service to our guests. You will be checking guests...
Full time
Flexible hours
Shift work
Afternoon shift
Crescent Careers
Edmonton, AB
more than 2 months ago
Sr. Data Engineer
...Join a company that’s leading the way in AI and automation consulting. Our portfolio spans 10+ top technologies across business applications... ...could be the place for you. We’re seeking a Senior Data Engineer to join our Delivery Team. In this role, you’ll design and...
Full time
Bits In Glass
Edmonton, AB
a month ago
Project Engineer/Technologist
...position functions include: Work closely with the Project Engineering Team providing technical and design support to determine the best... ...requirements and develop the project outline. Assess, evaluate, and incorporate technical requirements for the project, including...
Full time
Stream-Flo Industries Career
Edmonton, AB
more than 2 months ago
Staff Software Engineer
$145.9k per year
...service providers grow their business. The role of Staff Software Engineer at Jobber: As a technical champion and force multiplier, you... ..., Cursor, and Copilot in your workflow, refining how you balance AI-driven efficiency with the craft of software development....
Long term contract
Full time
Internship
Work at office
Local area
Flexible hours
Jobber
Edmonton, AB
more than 2 months ago
Search Engine Marketing Manager (Remote, Edmonton)
...multifamily, B2B and technical verticals. As Search Engine Marketing Manager at Blacksmith, you own... ...Agency Marketing Full time Remote (USA, Canada or South Africa) About The Role... ...SEO, link building, programmatic pages and AI-driven content. Own KPI tracking,...
Remote work
Full time
Blacksmith Agency LLC
Edmonton, AB
4 days ago
Data Engineer - Senior (REMOTE) JP1030
...The analyst works closely with business stakeholders, data engineers, solution architects, platform administrators, and application teams... ...stakeholders to resolve platform and integration issues. Utilize AI and automation tools to streamline data engineering workflows, including...
Remote job
P@thlion Staffing Careers
Edmonton, AB
23 days ago
Senior Software Engineer — Remote and Equity
...Affirm is seeking a software engineer for a remote Canada role. You will ship web apps using React or Vue, mobile apps with React Native, and design APIs at scale with Kotlin or Python. You’ll work with product, design, and analytics to turn ideas into robust software....
Remote work
Affirm
Edmonton, AB
4 days ago
Senior Site Reliability Engineer
$145.9k per year
...Are you a seasoned production engineering professional who wants to work on a team that goes beyond the status quo? Then Jobber might be the... ...and stability. In this role, you will be expected to leverage AI tools to augment your capabilities while maintaining engineering...
Long term contract
Full time
Internship
Work at office
Local area
Worldwide
Jobber
Edmonton, AB
a month ago
Senior Transmission & Substation Engineer
...one full-time, permanent Senior Transmission & Substation Engineer working out of Edmonton, AB . The Senior Lead Transmission... ...including developing technical specifications, supporting bid evaluations, coordinating engineering reviews, reviewing supplier...
Permanent employment
Full time
For contractors
Epcor
Edmonton, AB
more than 2 months ago
Senior Data Engineer, AWS
...Position Summary We are seeking a Senior or Principal Data Engineer to design, build, and optimize data platforms on Amazon Web Services... ...orchestration and serverless data workloads. Familiarity with AI-assisted development tools and workflows. AWS certifications such...
Full time
Relocation package
Lantern
Edmonton, AB
more than 2 months ago
Bilingual French Inbound Sales Agent - Remote ON - October 23rd
$19.7 per hour
...Agent Commercial Bilingue Française - Ontario L'opportunité Êtes‑vous une personne passionnée, sympathique et axée sur vos clients? Souhaitez‑vous développer vos compétences commerciales? Si oui, nous vous invitons à rejoindre notre équipe et à nous aider à épater nos...
Remote work
Full time
Apprenticeship
Work from home
Monday to friday
10 hours per week
Day shift
O2E Brands Inc.
Edmonton, AB
5 days ago
AI Product Manager => Relocate to China
$140k - $200k per year
...Job Description: Position: AI Product Manager Location: Remote / On-site Employment Type: Full-time... ...collaborate with data scientists, ML engineers, and software teams to deliver product... ...product roadmap. Conduct A/B testing, evaluate product performance, and drive...
Remote work
Full time
Relocation
MS Systems
Edmonton, AB
10 days ago
Environmental Engineer
...demand. We invite you to bring your bold ideas and big dreams and become part of a global team of over 50,000 planners, designers, engineers, scientists, digital innovators, program and construction managers and other professionals delivering projects that create a positive...
Full time
Contract work
For subcontractor
Work at office
Local area
Worldwide
Flexible hours
Aecom
Edmonton, AB
more than 2 months ago
AI Analyst
...PCL opportunities. We are currently building enterprise-wide AI capability, creating exciting opportunities to apply your skills... ...informed on emerging AI technologies and contributing to the evaluation and recommendation of tools and platforms. Qualifications ~...
PCL Construction
Edmonton, AB
1 day ago
Specialist, Engineering
...development in the country, and supporting Alberta as a key pillar in Canada’s life sciences sector. We are seeking a hands-on Engineer to support the installation, commissioning, qualification, validation, improvement, and ongoing operation of GMP manufacturing equipment...
Full time
For contractors
Work at office
Work from home
Worldwide
Relocation
Lp-api
Edmonton, AB
more than 2 months ago
Remote Web Engineer II, MarTech & Privacy
...Affirm is a remote‑first company hiring a Software Engineer II for Marketing & Privacy Engineering. You will build high‑traffic web experiences using React and TypeScript, collaborating with product, design, analytics, and data teams to ship reliable measurement and privacy...
Remote work
Affirm
Edmonton, AB
5 days ago
Senior Data Engineer - Remote - Edmonton, AB - SDAB
$60 - $85 per hour
...Job Title: Data Engineer – Intermediate / Senior Location: Edmonton, AB Job Type:... ...workflows. ~1 year of experience using AI for code generation, automation and productivity... ...~3 years of experience supporting Talend Remote Engines, Job Servers and runtime infrastructure....
Remote work
Hourly pay
Contract work
Compu-Vision Consulting Inc.
Edmonton, AB
2 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to AI Agent Evaluation Engineer - Remote. Be the first to apply!