AI Agent Evaluation Engineer - Remote

YO AI Labs

Remote job

Senior Software Engineer

Job Type: Contractor (~15 hours/week)
Location: Remote

Job Summary

We are seeking experienced Senior Software Engineers to support an AI training project by creating reinforcement learning environments that evaluate AI models on complex software engineering tasks using Model Context Protocol (MCP) tools.

You will design reproducible environments, deterministic verification, and reference solutions for tasks such as bug fixing, feature implementation, codebase refactoring, and performance optimization . No prior AI experience is required.

Key Responsibilities

Create reinforcement learning environments for software engineering tasks.
Design tasks involving bug fixing, feature development, refactoring, and performance optimization .
Build deterministic verification systems and golden reference solutions.
Evaluate AI agents' ability to reason through complex codebases and use MCP tools effectively.
Develop realistic, reproducible software engineering scenarios.
Ensure tasks accurately measure coding ability, problem-solving, and tool usage.
Document solutions and provide clear technical feedback.

Required Skills

Strong proficiency in Python 3, Java, Rust, C++, or TypeScript .
Strong understanding of algorithms and data structures .
Experience with bug fixing and debugging complex software issues .
Proven experience in feature implementation and codebase refactoring .
Strong knowledge of performance optimization and tuning .
Excellent written and verbal communication.
Strong attention to detail.

Preferred Qualifications

Experience working with large-scale or distributed codebases .
Familiarity with AI/ML systems is a plus but not required.
Experience with rigorous code reviews and software engineering best practices .
Experience working effectively in remote or cross-functional teams.

Hiring Process

Submit an application and screening questions.
Complete an AI interview (~30 minutes).
Complete a technical assessment, if required.
Hiring Manager review.

Compensation

Compensation is output-based , with payment provided per task that meets project specifications. Minimum weekly submission requirements may apply.

Availability

Selected experts should be prepared to begin their first tasks within 24–48 hours of completing onboarding .

Vacancy posted 28 days ago

Similar jobs that could be interesting for youBased on the AI Agent Evaluation Engineer - Remote in Edmonton, AB vacancy

AI and Data Engineer
...Company Description Blend is a premier AI services provider, committed to co-creating... ...Description We are building a team of engineers in Edmonton, AB, Canada to develop a first... ...on AI/ML components — including LLM-based agents, embedding models, and behavioral simulation...
Suggested
Full time
Internship
Work at office
Weekend work
Blend360
Edmonton, AB
more than 2 months ago
AI/ML Engineer - Remote
...AI/ML Engineer Job Type: Full-Time Location: Remote Job Summary We are seeking an experienced AI/ML Engineer to build and deploy secure, scalable... ...work with LLMs, RAG, prompt engineering, multi-agent systems, and cloud AI platforms to develop production...
Remote job
Full time
YO AI Labs
Edmonton, AB
28 days ago
Solution Architect (Senior AI / Technical Lead)
...Bits In Glass (BIG) is a high-growth AI and automation consulting firm with offices... ...making the decisions a single embedded engineer should not make alone: model hosting,... ...Set the standards every pod inherits for agent design, evaluation, observability, security, and cost...
Suggested
Permanent employment
Full time
Internship
Bits In Glass
Edmonton, AB
19 days ago
LPN - Edmonton (AIS)
...We use technology and automation to support administrative aspects of our recruitment process. Artificial Intelligence is not used to evaluate, rank, or select candidates. All hiring decisions are made by people. CBI Home Health is committed to a diverse, equitable, and...
Suggested
Full time
Part time
Casual work
Flexible hours
Weekend work
Cbi Home Health
Edmonton, AB
25 days ago
Software Development Manager - AI Solutions and Business Intelligence
$140k - $175k per year
...LawDepot’s next generation of AI-powered experiences, reporting... ...Edmonton, Alberta office or fully remote across Canada. What You’ll... ...impact Research, evaluate, and recommend AI tools, reporting... ...in Computing Science, Software Engineering, Artificial Intelligence, Data...
Remote work
Long term contract
Permanent employment
Full time
Bank staff
Work at office
Work from home
Worldwide
Flexible hours
Lawdepot Inc
Edmonton, AB
more than 2 months ago
Engineer, Planning
...a Full-Time, Temporary (up to 18 months) Engineer, Planning position working out of EPCOR... ...infractions for unsafe driving behaviours will be evaluated and considered for non-selection... ...Employee Mileage Reimbursement Policy. Use of AI: Our recruitment process may use...
Long term contract
Full time
Temporary work
Work at office
Flexible hours
Epcor
Edmonton, AB
18 days ago
Engineer, Structural
...a f ull Time, p ermanent Structural Engineer position working out of the Rossdale Water... ...with applicable codes and standards. Evaluate and approve consultant submissions,... ...Employee Mileage Reimbursement Policy. Use of AI: Our recruitment process may use...
Long term contract
Full time
Contract work
Temporary work
For contractors
Work at office
Epcor
Edmonton, AB
a month ago
Junior Control Systems Engineer
...proud to be a leading provider of Engineering, Procurement, Construction and... ..., studies, reports, technical evaluations, and project proposals.... ...on annual calendar ) Hybrid remote work program ( up to 52 days per... ...unsolicited agency resumes please. AI technology is utilized by...
Remote work
Long term contract
Full time
For contractors
Summer work
Internship
Work at office
Local area
Immediate start
Flexible hours
Bantrel Co.
Edmonton, AB
26 days ago
Platform Engineer
...shaping the future of fleet readiness with AI-enabled, connected fleet technology that... ...Fleetworthy Inc. is hiring a Platform Engineer to own and evolve the systems that connect... ...VPN behavior, and TCP socket data flows in remote environments where recoverability matters...
Remote work
Full time
Fleetworthy
Edmonton, AB
a month ago
Remote Life Insurance Agent
...Insurance Company’s distribution network, is expanding and seeking motivated individuals to join our team! About the Role: As a Remote Insurance Benefits Specialist , you’ll be the first point of contact for families seeking life insurance coverage. No prior insurance...
Remote job
Weekly pay
Full time
Contract work
Work from home
Long distance
Flexible hours
AO Globe Life
Edmonton, AB
3 days ago
Application Engineering Technologist
...subsidiaries. At this moment, we need an Applications Engineer to cover the province of Alberta . You will be... ...and training. Mazak is proud to read and evaluate every single profile without any AI technology. We are moved with people, by people and to...
Hourly pay
Long term contract
Full time
Local area
Night shift
Mazak Corp
Edmonton, AB
a month ago
Custom Order Engineer
$69k - $87k per year
...sector is currently seeking a Custom Order Engineer to join our team. This position is based... ...the hiring process. All candidates will be evaluated based on job-related competencies, and all... .... Our Talent Acquisition teams also uses AI to identify applicants whose skills are a...
Full time
Relocation package
Eaton Corporation
Edmonton, AB
6 days ago
Product and technology development engineer (m/w/d)
...Job Description – Product Development Engineer (71021) July 2026 Level II About G2V Optics... ...reviews against requirements. Manage AI-assisted engineering in the same frame: specifying... ...relevant to product development. Evaluate, adopt, and integrate new capabilities as...
Full time
Internship
G2V Optics Inc.
Edmonton, AB
more than 2 months ago
Engineer, Asset Management
...are hiring two (2) Full-Time, Permanent Engineer, Asset Management working out of our... ...for unsafe driving behaviours will be evaluated and considered for non-selection regardless... ...Reimbursement Policy. ~ Use of AI: Our recruitment process may use technology...
Remplacement
Permanent employment
Full time
Epcor
Edmonton, AB
18 days ago
AI/Data Associate - Upper Bound
...Role Overview We are seeking highly curious, driven individuals to join our team as Early Career Consultants specializing in Data & AI, Azure, and modern development practices. This role is ideal for individuals with foundational skills or hands-on experience in Data,...
Full time
Lantern
Edmonton, AB
more than 2 months ago
Customer Service Call Agent
...Job Summary We are seeking a friendly and professional Customer Service Call Agent to assist customers by answering inquiries, resolving complaints, and providing product or service information. The ideal candidate has excellent communication skills, patience, and a...
Full time
Monday to friday
Orä Partenaires
Edmonton, AB
12 days ago
Design Analysis Engineer
...Bradken is a global manufacturer of custom-engineered products and solutions for the resources sector. Our products operate in some of... ...be built, supplied and supported effectively. · Support field evaluations and product reviews through site and manufacturing visits when...
Full time
Work at office
Immediate start
Visa sponsorship
Work visa
Flexible hours
Bradken
Edmonton, AB
27 days ago
Data Engineer - Senior (REMOTE) JP1030
...The analyst works closely with business stakeholders, data engineers, solution architects, platform administrators, and application teams... ...stakeholders to resolve platform and integration issues. Utilize AI and automation tools to streamline data engineering workflows, including...
Remote job
P@thlion Staffing Careers
Edmonton, AB
1 day ago
Sr. Data Engineer
...Join a company that’s leading the way in AI and automation consulting. Our portfolio spans 10+ top technologies across business applications... ...could be the place for you. We’re seeking a Senior Data Engineer to join our Delivery Team. In this role, you’ll design and...
Full time
Bits In Glass
Edmonton, AB
19 days ago
Project Engineering Technologist
...position functions include: Work closely with the Project Engineering Team providing technical and design support to determine the best... ...requirements and develop the project outline. Assess, evaluate, and incorporate technical requirements for the project, including...
Full time
Stream-Flo Industries Career
Edmonton, AB
more than 2 months ago
Guest Service Agent - Matrix Hotel
...proud of, where you are valued, trusted and supported by the team Here is what you will be doing each day: As our Guest Service Agent, you will be working with our Guest Services team to provide excellent customer service to our guests. You will be checking guests...
Full time
Flexible hours
Shift work
Afternoon shift
Crescent Careers
Edmonton, AB
a month ago
Staff Software Engineer
$145.9k per year
...service providers grow their business. The role of Staff Software Engineer at Jobber: As a technical champion and force multiplier, you... ..., Cursor, and Copilot in your workflow, refining how you balance AI-driven efficiency with the craft of software development....
Long term contract
Full time
Internship
Work at office
Local area
Flexible hours
Jobber
Edmonton, AB
more than 2 months ago
Senior Site Reliability Engineer - Services
$145.9k per year
...Are you a seasoned production engineering professional who wants to work on a team that goes beyond the status quo? Then Jobber might be the... ...and stability. In this role, you will be expected to leverage AI tools to augment your capabilities while maintaining engineering...
Long term contract
Full time
Internship
Work at office
Local area
Worldwide
Jobber
Edmonton, AB
a month ago
Structural Engineer Technologist
...results. Candidate Profile The Structural Restoration Engineer or Technologist will work collaboratively with project teams to... ...Duties include, but are not limited to: Site investigation and evaluation of existing structures Preparation of reports and studies...
Contract work
Internship
RJC Engineers Careers
Edmonton, AB
26 days ago
Senior Data Engineer, AWS
...Position Summary We are seeking a Senior or Principal Data Engineer to design, build, and optimize data platforms on Amazon Web Services... ...orchestration and serverless data workloads. Familiarity with AI-assisted development tools and workflows. AWS certifications such...
Full time
Relocation package
Lantern
Edmonton, AB
more than 2 months ago
Senior Transmission & Substation Engineer
...one full-time, permanent Senior Transmission & Substation Engineer working out of Edmonton, AB . The Senior Lead Transmission... ...including developing technical specifications, supporting bid evaluations, coordinating engineering reviews, reviewing supplier...
Permanent employment
Full time
For contractors
Epcor
Edmonton, AB
a month ago
Aircraft Maintenance Engineer (Permanent)
...Airbus H125 aircraft preferred Familiarity with NVIS‑equipped aircraft considered an asset Field and line maintenance experience in remote environments Ability to work a 2‑and‑2 rotational schedule Strong troubleshooting ability and commitment to safety Ability to...
Remote work
Remplacement
Permanent employment
Contract work
Night shift
Canadian Helicopters Limited
Edmonton, AB
a month ago
Project Manager - Remote Location Camp Operations Nunavut
$115k - $130k per year
...People Who Care. Job Description WHAT'S THE JOB? The Project Manager is a strategic builder and change agent responsible for leading multiple remote hospitality sites. This role designs and implements scalable systems, drives performance improvement, and ensures...
Remote work
Full time
Part time
Local area
Shift work
Night shift
Weekend work
Weekday work
Kitikmeot Camp Solutions
Edmonton, AB
more than 2 months ago
Specialist, Engineering
...development in the country, and supporting Alberta as a key pillar in Canada’s life sciences sector. We are seeking a hands-on Engineer to support the installation, commissioning, qualification, validation, improvement, and ongoing operation of GMP manufacturing equipment...
Full time
For contractors
Work at office
Work from home
Worldwide
Relocation
Lp-api
Edmonton, AB
a month ago
Environmental Engineer
...demand. We invite you to bring your bold ideas and big dreams and become part of a global team of over 50,000 planners, designers, engineers, scientists, digital innovators, program and construction managers and other professionals delivering projects that create a positive...
Full time
Contract work
For subcontractor
Work at office
Local area
Worldwide
Flexible hours
Aecom
Edmonton, AB
more than 2 months ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to AI Agent Evaluation Engineer - Remote. Be the first to apply!