AI Agent Evaluation Engineer - Remote

YO AI Labs

Remote job

Senior Software Engineer

Job Type: Contractor (~15 hours/week)
Location: Remote

Job Summary

We are seeking experienced Senior Software Engineers to support an AI training project by creating reinforcement learning environments that evaluate AI models on complex software engineering tasks using Model Context Protocol (MCP) tools.

You will design reproducible environments, deterministic verification, and reference solutions for tasks such as bug fixing, feature implementation, codebase refactoring, and performance optimization . No prior AI experience is required.

Key Responsibilities

Create reinforcement learning environments for software engineering tasks.
Design tasks involving bug fixing, feature development, refactoring, and performance optimization .
Build deterministic verification systems and golden reference solutions.
Evaluate AI agents' ability to reason through complex codebases and use MCP tools effectively.
Develop realistic, reproducible software engineering scenarios.
Ensure tasks accurately measure coding ability, problem-solving, and tool usage.
Document solutions and provide clear technical feedback.

Required Skills

Strong proficiency in Python 3, Java, Rust, C++, or TypeScript .
Strong understanding of algorithms and data structures .
Experience with bug fixing and debugging complex software issues .
Proven experience in feature implementation and codebase refactoring .
Strong knowledge of performance optimization and tuning .
Excellent written and verbal communication.
Strong attention to detail.

Preferred Qualifications

Experience working with large-scale or distributed codebases .
Familiarity with AI/ML systems is a plus but not required.
Experience with rigorous code reviews and software engineering best practices .
Experience working effectively in remote or cross-functional teams.

Hiring Process

Submit an application and screening questions.
Complete an AI interview (~30 minutes).
Complete a technical assessment, if required.
Hiring Manager review.

Compensation

Compensation is output-based , with payment provided per task that meets project specifications. Minimum weekly submission requirements may apply.

Availability

Selected experts should be prepared to begin their first tasks within 24–48 hours of completing onboarding .

Vacancy posted 11 days ago

Similar jobs that could be interesting for youBased on the AI Agent Evaluation Engineer - Remote in Calgary, AB vacancy

AI Engineer
(Previously Posted as Applied AI Platform Engineer) Be Boring. Make Money.™ is our investment approach... ...capabilities available to our internal agents Build the integration layer... ...reliability, security, observability, evaluation, and governance Evaluate emerging frameworks...
Remote work
Long term contract
Full time
Flexible hours
Shift work
Mawer Brand
Calgary, AB
20 hours ago
Senior Software Engineer, AI Platform (US - Remote or Calgary)
...About the Role: Syndi is Syndio's AI platform — one shared agent experience (in-product chat, Slack,... ...experience. We're hiring a staff-level engineer to take ownership of major runtime... ...Disability. Syndio covers the full premium. Remote-first in US + Alberta; in Calgary, non...
Remote job
Full time
Contract work
Internship
Live In
Work at office
Flexible hours
Syndio
Calgary, AB
20 hours ago
AI/ML Engineer - Remote
...AI/ML Engineer Job Type: Full-Time Location: Remote Job Summary We are seeking an experienced AI/ML Engineer to build and deploy secure, scalable... ...work with LLMs, RAG, prompt engineering, multi-agent systems, and cloud AI platforms to develop production...
Remote job
Full time
YO AI Labs
Calgary, AB
11 days ago
AI Engineer - Canada
...seeking a highly skilled and experienced AI/ML Developer to join our team. The ideal candidate... ...sophisticated, multi-step and multi-agent workflows using frameworks like LangChain... ...strategies for and efficiency. Prompt Engineering : Develop and refine advanced prompt engineering...
Suggested
Full time
Work at office
Local area
Visa sponsorship
Flexible hours
Pulsora
Calgary, AB
20 hours ago
Applied AI Engineer - Canada
...highly skilled and experienced engineer who can independently build... ...reliable LLM-powered systems and agent workflows in production.... ...What you will do Build & Ship AI-Powered Products Design, develop... ...Define and implement evaluation frameworks for AI outputs (quality...
Suggested
Full time
Work at office
Visa sponsorship
Flexible hours
Pulsora
Calgary, AB
20 hours ago
AI Platform & Machine Learning Engineer
...Help build the AI platform powering the future of construction... ...ve created the world’s first engineering-grade Augmented Reality... ...through to training models, evaluating their performance and ultimately... ...preparing data through to training, evaluation, deployment and monitoring....
Full time
Internship
Xyz Reality
Calgary, AB
20 hours ago
Full Stack Software Engineer - AI and Internal Tools
...team: Behind the Scenes with our Software Engineers The Role Push is looking for an... ...building and testing Push's internal tools and AI integrations. In this role, you will have... ...powered features Experience with multi-agent frameworks (e.g., CrewAI) Previous startup...
Full time
Push Operations
Calgary, AB
20 hours ago
Bilingual Customer Service Agent (REMOTE)
$16.2 - $19.2 per hour
...Automatic Collision Notification, Hands-free Destination Assist, Stolen Vehicle Locator, and more. In this role, you’ll listen carefully and evaluate each caller’s situation to provide the right solutions, from simple navigation questions to emergency situations. Fluency in both...
Remote work
Contract work
For contractors
Work from home
Flexible hours
Night shift
Weekend work
Working Solutions
Calgary, AB
21 days ago
Platform Engineer (Database Reliability) - Remote Canada
...Platform Engineer (Database Reliability) About Bold Commerce Bold Commerce is a leading ecommerce technology company helping merchants... ...anytime Employee Options to help you grow with us! Working remotely - anywhere in Canada Flexible work hours Competitive paid...
Remote work
Full time
Flexible hours
Bold Commerce
Calgary, AB
20 hours ago
Senior Platform Engineer - AI Platforms & Deployment - Landmark
...reliability of next-generation AI-enabled software platforms.... ...coordinate multiple specialized agents and tools to automate complex... ...the intersection of software engineering, infrastructure engineering, AI... ...tools and techniques on your own, evaluate them critically, and bring the...
Full time
Part time
Second job
Shift work
Halliburton
Calgary, AB
13 days ago
Applied AI Platform Engineer
...Opportunity We are seeking an Applied AI Platform Engineer who will be focused on using AI tools... ...augmented generation agentic workflows evaluation observability human-in-the-loop review... ...learn LLM application patterns such as RAG agents tool calling embeddings model...
Remote work
Long term contract
Full time
Internship
Flexible hours
Shift work
Weekend work
Mawer Brand
Calgary, AB
more than 2 months ago
Software Architect - AI Accelerated Engineering Lead
$123k - $163k per year
...or Sr. Architect - Accelerated Engineering (For Vacant Role) Mission... ...pioneering the next generation of AI-empowered software development... ...environments where AI agents and engineers work side by side... ...repeatable practices such as evaluation harnesses, code-review agents,...
Internship
Work at office
Slalom
Calgary, AB
17 days ago
Phone Support and Customer Service Agent
...Overview As a Phone Support & Customer Service Agent at Reach, you’ll help deliver exceptional... ...initiatives and adoption of new tools and AI workflows Contribute to a positive,... ...cover letter outlining your relevant experience and interest in joining Reach. #LI-Remote...
Remote work
Full time
Reach Inc.
Calgary, AB
20 hours ago
Senior Manager, Marketing AI & Analytics
...can do right now is build with AI — not just adopt AI tools, but... ...intel, reporting) Evaluate emerging AI tools and lead build... ...not necessarily as a trained engineer, but as someone who can ship working... ...Absorber? What we offer: Fully remote-first work with flexible work...
Remote work
Full time
Immediate start
Flexible hours
Absorb.
Calgary, AB
20 hours ago
Rental Agent
$20 per hour
ABOUT US Indie Campers, the go-to campervan Marketplace, is on a mission of making road trips available to everyone. Thanks to its digital approach and customer-centric orientation, Indie Campers has developed a strong booking experience and high-quality road trips at affordable...
Hourly pay
Long term contract
Full time
Temporary work
Local area
Night shift
Weekend work
Indie Campers
Calgary, AB
20 hours ago
Solution Architect (Senior AI / Technical Lead)
...Bits In Glass (BIG) is a high-growth AI and automation consulting firm with offices... ...making the decisions a single embedded engineer should not make alone: model hosting,... ...Set the standards every pod inherits for agent design, evaluation, observability, security, and cost...
Permanent employment
Internship
Bits In Glass
Calgary, AB
20 hours ago
Senior Remote Machine Learning Engineer
$130k - $150k per year
...’re hiring a Senior Machine Learning Engineer to join our AI & Analytics Engineering team. This team... ..., practitioner-facing conversational agents, and tools that help clinicians move... ...early experimentation through deployment, evaluation, and iteration. You’ll work closely...
Remote work
Full time
Flexible hours
Fullscript
Calgary, AB
20 hours ago
Support & Cleaning Agent
...for the next customer, directly influencing customer satisfaction and operational efficiency. THE ROLE The Support & Cleaning Agent focuses exclusively on maintaining the cleanliness, hygiene, and presentation of the rental fleet. This is a hands-on, physical,...
Full time
Seasonal work
Local area
Flexible hours
Shift work
Indie Campers
Calgary, AB
20 hours ago
DevOps Engineer / Platform Engineer, Senior
...Senior DevOps Engineer Location: Calgary, AB — Hybrid (Yes, our... ...growing cloud infrastructure and an AI-driven development workflow... ...beyond access controls, evaluates AI-assisted DevOps tooling critically... ...AI development or operations agents, and you bring the engineering...
Internship
Sovra
Calgary, AB
20 hours ago
Principal Software Development Engineer - Applied AI
...running a business. We're remote-first, growing, and looking for... ...Development Engineer - Applied AI , who enjoys tackling hard... ...the agentic pipeline of specs, agent implementation, agent review,... ...décisions de plateforme quantifiées (évaluations, observabilité, persistance) à...
Summer work
Internship
Day shift
Wagepoint
Calgary, AB
a month ago
Customer Service Agent, Charter Operations
...Indigenous cultures, and bring vital services to some of the most remote and remarkable places in Canada. Why You’ll Love Working... ...Canadian airline About the position The Customer Service Agent, Charter Operations (Charter Specialist) provides day to day assistance...
Remote work
Long term contract
Full time
Temporary work
For subcontractor
Internship
Worldwide
Trial period
Shift work
Night shift
Canadian North
Calgary, AB
20 hours ago
Machine Learning Software Developer - Foundational & Agentic AI
...build and deploy production-grade AI systems for our flagship... ...based platform. Establish evaluation, observability, and monitoring... ..., maintainability, and strong engineering practices. Required Skills/... ...environments. Familiarity with multi-agent systems, workflow engines,...
Full time
Local area
Flexible hours
Circle Cardiovascular Imaging
Calgary, AB
20 hours ago
Bilingual Evaluation Insights & Data Systems Analyst (1-year contract)
$55.35k - $70.11k per year
** This position can be fully remote or hybrid, if located near a Mitacs office in Montreal, Toronto, or Vancouver. Please ensure that... ...to apply. Position Summary Reporting to the Manager, Evaluations, the Bilingual Evaluation Insights & Data Systems Analyst supports...
Remote work
Long term contract
Full time
Contract work
Work at office
Flexible hours
Mitacs
Calgary, AB
20 hours ago
canvassing agent
$17 per hour
...Experience No experience (will train) On site Work must be completed at the physical location. There is no option to work remotely. Responsibilities Tasks ~ Deliver goods to customers Additional information Work conditions and physical capabilities...
Remote work
Permanent employment
Full time
IH2O LTD.
Calgary, AB
3 days ago
VP, Marketing Operations and Agentic AI Systems
...We measure it in decisions our agents make autonomously, and we want... ...Operations & Agentic AI Systems, you will own the management... ...libraries, tool definitions, and evaluation suites that govern agent... ...and retain a team of marketing engineers and operations specialists....
Remote work
Work at office
Flexible hours
Sidetrade
Calgary, AB
20 days ago
Automation Engineer - Controls & Automation
...’LL DO: CoolIT Systems is looking for an experienced Automation Engineer, Manufacturing to join our Calgary Manufacturing team. In this role... ...is capturing global attention. As demand for generative AI, high performance computing and cloud solutions rapidly expands,...
Long term contract
Casual work
Ecolab Inc.
Calgary, AB
6 days ago
Sales Agent
$100 per day
Job Description Job Description The Acquisition Group is hiring sales representatives to represent our client TELUS through direct residential sales in Calgary. If you want a role that builds real career momentum, this is a place to grow from entry-level rep into ...
The Acquisition Group
Calgary, AB
3 days ago
Weekend Property Experience & Showing Agent
$22 - $26 per hour
...delivers Four Seasons-level service in-person and genuinely loves connecting with people. As our Weekend Property Experience & Showing Agent, you are the face of Ripple when it matters most. You are the structural bridge between our weekday operations and prospective...
Hourly pay
Long term contract
Full time
Casual work
Internship
Immediate start
Work from home
Weekend work
Weekday work
Ripple Property Management
Calgary, AB
20 hours ago
Assistant Shift Engineer
...no need to be in any of them! We’re fully remote and hire across Canada and the U.S. About... ...microservices where appropriate Use AI to streamline processes, reduce friction,... ...focus toward meaningful work ~ Strong data engineering skills: complex SQL queries, optimization,...
Remote work
Full time
Immediate start
Flexible hours
Shift work
Zayzoon
Calgary, AB
20 hours ago
Go-to-Market Engineer - Calgary, Canada
...setting. These include frontend and backend engineers, AI research scientists, and others from... ...their own companies. About SIMBA Voice Agents by Speechify SIMBA lets companies... ...keep data clean and systems in sync Evaluate and adopt new AI and GTM tooling rapidly...
Full time
Speechify
Calgary, AB
20 hours ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to AI Agent Evaluation Engineer - Remote. Be the first to apply!