Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

DevOps Engineer - AI Model Evaluator

$85 per hour

Mercor

Toronto, ON
  • Remote job

About the job

Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .

Position: DevOps / SRE / Cloud Engineer (Coding Agent Experience)

Type: Contract

Compensation: $85/hour

Location: Remote

Role Responsibilities

  • Use frontier AI coding agents to complete and evaluate complex infrastructure engineering tasks.
  • Review model-generated implementations involving cloud platforms , Kubernetes , CI/CD systems , and infrastructure automation .
  • Identify bugs, edge cases, reliability issues, and failure modes in model outputs.
  • Compare outputs from multiple frontier models to assess strengths and weaknesses.
  • Apply professional engineering judgment to realistic infrastructure engineering scenarios.

Qualifications

Must-Have

  • 2+ years of professional DevOps , SRE , or Cloud Engineering experience.
  • Experience with AWS , Azure , GCP , Kubernetes , Terraform , CI/CD pipelines , or observability tooling.
  • Regular use of AI coding agents like Cursor , Claude Code , Codex , Windsurf , Gemini CLI , or similar tools.
  • Ability to evaluate model-generated infrastructure and reliability engineering solutions.

Preferred

  • Experience supporting production-scale systems .

Compensation & Legal

  • $400 per accepted task .
  • Compensation is tied to accepted work.

Application Process (Takes 20–30 mins to complete)

  • Upload resume
  • AI interview based on your resume
  • Submit form

Resources & Support

  • For details about the interview process and platform information, please check:
  • For any help or support, reach out to:

PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.

Vacancy posted 28 days ago
Similar jobs that could be interesting for youBased on the DevOps Engineer - AI Model Evaluator in Toronto, ON vacancy
  • $85 per hour

     ...and technical talent with leading AI research labs. Headquartered in San...  ...Dorsey . Position: iOS Engineer (Coding Agent Experience) Type...  ...AI coding agents to complete and evaluate complex engineering tasks. Review model-generated mobile application code... 
    Suggested
    Remote job
    Contract work
    Summer work

    Mercor

    Toronto, ON
    a month ago
  • $80k - $180k per year

     ...Zafin is an AI platform company helping regulated institutions modernize how critical...  ...scale. What’s the Opportunity?  The AI Evaluation Engineer ensures AI agent solutions are accurate,...  ...of AI evaluation, large language model behaviour, reasoning quality, hallucination... 
    Suggested
    Full time

    Zafin

    Toronto, ON
    4 days ago
  • $130k - $145k per year

     ...role: We are seeking a highly skilled and passionate Senior DevOps Engineer, AI infrastructure to join our team! You will play a key role in...  ...and refine advanced prompting strategies for LLMs. Test, evaluate, and analyze the performance of LLM and other GenAI models.... 
    Suggested
    Full time
    Internship
    Work at office
    Flexible hours

    Financeit

    Toronto, ON
    20 hours ago
  • $90k - $115k per year

     ...greener and smarter mobility worldwide, connecting cities as we reduce carbon and replace cars.     Could you be the full-time DevOps Engineer in Toronto, ON we’re looking for?   Your future role We’re looking for a motivated DevOps Engineer to help modernize and... 
    Suggested
    Long term contract
    Full time
    Internship
    Worldwide

    Alstom

    Toronto, ON
    14 hours ago
  • $56k - $84k per year

     ...Job Type:  Permanent   Work Model:  Hybrid   Reference code: 133514 Primary Location...  ...someone with: Relevant experience in AI/ML domain Knowledge of NLP, LLMs (extractive...  ...AI offerings, including testing, evaluating and benchmarking different LLMs and toolchains... 
    Suggested
    Permanent employment
    Contract work
    Flexible hours

    Deloitte

    Toronto, ON
    14 hours ago
  •  ...Our team is looking for a DevOps engineer with at least 6 years cloud and Kubernetes experience including specialization in ArgoCD. Our project involves building automations for a greenfield managed service product in AWS and we are looking for candidates who are interested... 
    Full time

    Sapsol Technologies Inc

    Toronto, ON
    7 days ago
  •  ...Tiger Analytics is a global leader in AI and analytics, helping Fortune 1000...  ...looking for a highly skilled  AI Engineer with 7+ years of experience in software...  ...AI/LLM APIs, AWS Bedrock, and other model providers. Infrastructure & DevOps: Experience with DevOps, CI/CD... 
    Long term contract
    Local area

    Tiger Analytics Inc.

    Toronto, ON
    20 hours ago
  • $80k - $120k per year

     ...Zafin is an AI platform company helping regulated institutions modernize how critical work is designed, governed, and delivered. Our technology enables organizations to move faster while maintaining the governance, accountability, and control required in highly regulated environments... 
    Full time
    Internship
    Local area

    Zafin

    Toronto, ON
    4 days ago
  •  ...Our team is looking for a DevOps engineer with at least 7 years cloud experience including specialization with Identity Our project involves building automations for a greenfield managed service product in AWS and we are looking for candidates who are interested in designing... 
    Full time

    Sapsol Technologies Inc

    Toronto, ON
    7 days ago
  •  ...inclusive world. Job Description We are seeking to hire a Senior DevOps Engineer for the Client's Branch DevOps team. The Branch DevOps Team is...  ...) to troubleshoot and remediate pipeline issues. · Leverage AI-assisted development tools (GitHub Copilot) to accelerate... 
    Permanent employment
    Full time
    Local area
    Shift work

    Capgemini

    Toronto, ON
    7 days ago
  •  ...organizations to solve complex business challenges through cloud, AI, data engineering, and modern software solutions. We foster a collaborative,...  ...outcomes. The Opportunity We're looking for a Senior DevOps Engineer who is deeply hands-on and ready to lead from the front... 
    Full time

    shyftlabs

    Toronto, ON
    5 days ago
  • $80 - $120 per hour

     ...creative and technical talent with leading AI research labs. Headquartered in San...  ...Position: Software / AI / IT / data Evaluator Type: Contract Compensation...  ...structured written feedback to improve AI model outputs . Apply deep subject-matter... 
    Remote job
    Contract work
    Summer work
    Work at office

    Mercor

    Toronto, ON
    a month ago
  • $92k - $118k per year

     ...building secure, automated, and scalable DevOps solutions. Join Capco to help transform enterprise...  ...data and analytics through cloud-native engineering and continuous delivery. The Role We...  ...At Capco, we use artificial intelligence (AI) tools to support and enhance several... 
    Internship
    Immediate start

    Capco

    Toronto, ON
    18 days ago
  • $135k - $170k per year

     ...General Information: Job Title: AI Engineer Location: Toronto, ON (Onsite/Hybrid)...  ...critical AI features Define and implement evaluation frameworks Optimize systems for...  ...Thrive We support remote and hybrid work models by offering multiple office locations to... 
    Long term contract
    Full time
    Internship
    Work at office
    Immediate start
    Remote work
    Flexible hours

    Fulfillment IQ

    Toronto, ON
    7 days ago
  • $60 - $62 per hour

     ...Platform / DevOps Engineer Client: a leading global eCommerce company Location: Toronto...  ...records. Use of Artificial Intelligence (AI): We may use Artificial Intelligence (...  ..., including sourcing, screening, and evaluating candidates. AI helps assess applications... 
    Contract work
    Toronto, ON
    21 days ago
  • $70 per hour

     ...creative and technical talent with leading AI research labs. Headquartered in San...  ...Remote Role Responsibilities Evaluate AI-generated responses to ensure accuracy...  ...structured written feedback to improve AI model outputs . Identify nuances, implicit... 
    Remote job
    Contract work
    Summer work

    Mercor

    Toronto, ON
    a month ago
  • $80 - $120 per hour

     ...creative and technical talent with leading AI research labs. Headquartered in San...  ...regulatory response with financial-services AI Evaluator Type: Contract Compensation...  ...to meet deadlines while improving AI model performance . Qualifications Must... 
    Remote job
    Contract work
    Summer work
    Work at office

    Mercor

    Toronto, ON
    a month ago
  • $80k - $138k per year

     ...Job Type:  Permanent   Work Model:  Hybrid   Reference code: 133554 Primary Location...  ...on the job coaching -- As a Data & AI Platform Engineer, you operate at the foundation of modern...  ..., and upgrades using tools like Azure DevOps, GitHub Actions, or Jenkins. • Deploy,... 
    Permanent employment
    Flexible hours

    Deloitte

    Toronto, ON
    14 hours ago
  • $92k - $118k per year

     ...Capco is seeking a DevOps/DevSecOps Engineer to join our Technology & Engineering practice in Toronto. This role is ideal for someone with a strong...  ...Talent Acquisition At Capco, we use artificial intelligence (AI) tools to support and enhance several parts of talent... 
    Internship
    Immediate start

    Capco

    Toronto, ON
    10 days ago
  •  ...Generative AI Engineer About Apertera Apertera is leading the evolution of language solutions...  ...Generative AI (GenAI) and Large Language Model (LLM) technologies. You will report to...  ...translation, cloud infrastructure, and evaluation. You'll play a pivotal role in pushing the... 
    Full time
    Shift work

    Apertera

    Toronto, ON
    6 days ago
  • $135k - $150k per year

     ...at  . The Opportunity: We’re looking for an AI Engineer, AI Platform & ML Engineering to join our Data & AI...  ...on the platform patterns, MLOps practices, model lifecycle, deployment standards, monitoring, evaluation, and governance integration needed to move AI and... 
    Full time
    Internship

    Aviso Wealth

    Toronto, ON
    20 hours ago
  •  ...Sensor Modeling & Simulation Engineer (M&S) Preferred Location: Ottawa or Toronto Reports to: Head of...  ...owns the sensor and electronic-warfare models end to end — RF/radar, EW, multi-sensor...  ...equity granted based on impact We use AI tools to support parts of the hiring... 
    Full time

    dominion%20dynamics

    Toronto, ON
    22 hours ago
  •  ...Senior DevOps Engineer Location: Toronto Role Description We are looking for a driven, top-notch Senior DevOps Engineer who is...  ...certification (AWS, Kubernetes, or similar) Experience using AI-assisted development tools Knowledge of: JWT OAuth 2.0... 

    Astra North Infoteck Inc.

    Toronto, ON
    a month ago
  •  ...Place to Work ®, five years in a row! As a  Senior Modeling & Simulation (M&S) Engineer , you will develop models, tools, and simulations of next-generation space...  ...prototype, and use new tools and technology (including AI) to learn, grow, and improve. Success in this role requires... 
    Full time
    Work at office
    Relocation package

    kepler

    Toronto, ON
    21 days ago
  •  ...Threat Modeler / Security Architect – AI Security, Cloud & DevSecOps Must Have Technical/Functional Skills...  ...programs. Secure Design & Engineering Enablement • Develop, maintain...  ...Develop innovative attack techniques to evaluate protective designs and existing... 
    Permanent employment

    Astra North Infoteck Inc.

    Toronto, ON
    19 days ago
  • $90.37 - $108.45 per hour

     ...RQ11264 - DevOPS/Cloud Engineer - Senior 11 months Contract (235 business days)  ONSITE 5 days...  ...pull request and practicing GitOps AI Disclaimer: Source Code may use...  ...to-have qualifications. The client will evaluate candidates based on both rate expectations... 
    Contract work

    Source Code

    Toronto, ON
    13 days ago
  •  ...Waabi, founded by AI visionary Raquel Urtasun, is the leader in...  ...way. To learn more visit: The Evaluation Algorithms team is responsible...  ...realistic closed-loop simulation engine built with the latest in...  ...provide a holistic understanding of model performance and enable the discovery... 
    Full time
    Internship

    waabi

    Toronto, ON
    a month ago
  • $95k - $115k per year

     ...focus on stronger infrastructure, better automation, practical AI adoption, improved reliability, stronger security practices, and...  ...level. About the Role Big Viking Games is hiring a Senior DevOps Engineer to help design, maintain, secure, and modernize the infrastructure... 
    Long term contract
    Full time
    Work at office
    3 days per week

    Big Viking Games

    Toronto, ON
    20 hours ago
  •  ...Senior AI Engineer Location: Ontario (5 Days Onsite) Duration: Long-term Contract...  ...Bedrock Amazon Lex Large Language Models (LLMs) Retrieval-Augmented Generation (RAG...  ...Responsible AI practices. ~ Experience evaluating AI models optimizing prompts monitoring... 
    Long term contract
    Full time

    Upstaff

    Toronto, ON
    20 days ago
  • $135k - $210k per year

     ...Overview:   Guidepoint seeks an experienced AI Engineer as an integral member of the Toronto-...  ...systems, including large language model (LLM) applications and research agents, while...  ...practices for LLMOps, including automated evaluation using frameworks like LLM Judges or MLflow... 
    Full time
    Worldwide

    Guidepoint

    Toronto, ON
    10 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to DevOps Engineer - AI Model Evaluator. Be the first to apply!