DevOps Engineer - AI Model Evaluator
$85 per hourMercor
Toronto, ON
- Remote job
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .
Position: DevOps / SRE / Cloud Engineer (Coding Agent Experience)
Type: Contract
Compensation: $85/hour
Location: Remote
Role Responsibilities
- Use frontier AI coding agents to complete and evaluate complex infrastructure engineering tasks.
- Review model-generated implementations involving cloud platforms , Kubernetes , CI/CD systems , and infrastructure automation .
- Identify bugs, edge cases, reliability issues, and failure modes in model outputs.
- Compare outputs from multiple frontier models to assess strengths and weaknesses.
- Apply professional engineering judgment to realistic infrastructure engineering scenarios.
Qualifications
Must-Have
- 2+ years of professional DevOps , SRE , or Cloud Engineering experience.
- Experience with AWS , Azure , GCP , Kubernetes , Terraform , CI/CD pipelines , or observability tooling.
- Regular use of AI coding agents like Cursor , Claude Code , Codex , Windsurf , Gemini CLI , or similar tools.
- Ability to evaluate model-generated infrastructure and reliability engineering solutions.
Preferred
- Experience supporting production-scale systems .
Compensation & Legal
- $400 per accepted task .
- Compensation is tied to accepted work.
Application Process (Takes 20–30 mins to complete)
- Upload resume
- AI interview based on your resume
- Submit form
Resources & Support
- For details about the interview process and platform information, please check:
- For any help or support, reach out to: Show email
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
Vacancy posted 8 days ago
Similar jobs that could be interesting for youBased on the DevOps Engineer - AI Model Evaluator in Toronto, ON vacancy
$85 per hour
...and technical talent with leading AI research labs. Headquartered in San... ...Dorsey . Position: iOS Engineer (Coding Agent Experience) Type... ...AI coding agents to complete and evaluate complex engineering tasks. Review model-generated mobile application code...SuggestedRemote jobContract workSummer work- ...training and deploying frontier models for developers and enterprises who are building AI systems to power magical experiences... ...is a team of researchers, engineers, designers, and more, who are passionate... ...the future! Why this role? Evaluation is critical to making progress...SuggestedFull timeWork at officeRemote workFlexible hours
$101k - $169k per year
...Job Type: Permanent Work Model: Hybrid Reference code: 131871 Primary Location... ...complex, high-impact engagements in the AI Model Risk Space, ensuring alignment with... ...Generative AI offerings, including testing, evaluating and benchmarking different LLMs and toolchains...SuggestedPermanent employmentContract workFlexible hours$80 - $120 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...Position: Software / AI / IT / data Evaluator Type: Contract Compensation... ...structured written feedback to improve AI model outputs . Apply deep subject-matter...SuggestedRemote jobContract workSummer workWork at office$70 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...Remote Role Responsibilities Evaluate AI-generated responses to ensure accuracy... ...structured written feedback to improve AI model outputs . Identify nuances, implicit...SuggestedRemote jobContract workSummer work$80 - $120 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...regulatory response with financial-services AI Evaluator Type: Contract Compensation... ...to meet deadlines while improving AI model performance . Qualifications Must...Remote jobContract workSummer workWork at office- ...Waabi, founded by AI visionary Raquel Urtasun, is the leader in... ...way. To learn more visit: The Evaluation Algorithms team is responsible... ...realistic closed-loop simulation engine built with the latest in... ...provide a holistic understanding of model performance and enable the discovery...Full timeInternship
$90k - $115k per year
...greener and smarter mobility worldwide, connecting cities as we reduce carbon and replace cars. Could you be the full-time DevOps Engineer in Toronto, ON we’re looking for? Your future role We’re looking for a motivated DevOps Engineer to help modernize and...Long term contractFull timeInternshipWorldwide$60 - $62 per hour
...Platform / DevOps Engineer Client: a leading global eCommerce company Location: Toronto... ...records. Use of Artificial Intelligence (AI): We may use Artificial Intelligence (... ..., including sourcing, screening, and evaluating candidates. AI helps assess applications...Contract work$70k - $100k per year
...systems designed for the Construction and Engineering industries. About CMiC CMiC is the... ...for an existing vacancy We do not use AI to screen, assess, or select applicants Job Overview We are looking for a DevOps Engineer with 5+ years of experience to support...Full timeSeasonal workFlexible hoursNight shift- ...Senior DevOps Engineer Location: Toronto Role Description We are looking for a driven, top-notch Senior DevOps Engineer who is... ...certification (AWS, Kubernetes, or similar) Experience using AI-assisted development tools Knowledge of: JWT OAuth 2.0...
- ...Place to Work ®, five years in a row! As a Senior Modeling & Simulation (M&S) Engineer , you will develop models, tools, and simulations of next-generation space... ...prototype, and use new tools and technology (including AI) to learn, grow, and improve. Success in this role requires...Full timeWork at officeRelocation package
- ...Senior AI Engineer Location: Ontario (5 Days Onsite) Duration: Long-term Contract... ...Bedrock Amazon Lex Large Language Models (LLMs) Retrieval-Augmented Generation (RAG... ...Responsible AI practices. ~ Experience evaluating AI models optimizing prompts monitoring...Long term contractFull time
- ...Waabi, founded by AI visionary Raquel Urtasun, is the leader in Physical AI. With a world-class team, we're unlocking the... ...scalable, controllable, and efficient simulation. As a Research Engineer in the World Models team, you will develop algorithms and productionize the next...Full time
- ...are seeking a highly accomplished Senior AI Engineer to spearhead the hands-on development, configuration... ...Agentic AI frameworks, implement precise Model Context Protocol (MCP) server... ...Prompt Engineering & Orchestration: Author, evaluate, and continuously refine prompt...Contract work
$90.37 - $108.45 per hour
...RQ00684 - Sr. DevOPS/Cloud Engineer 6-month contract (126 business days) - possible extension... ...Strong understanding of agile delivery models with deep experience building and managing... ...deployment progress, system performance, and evaluating the overall quality of cloud...Contract work- ...Waabi, founded by AI visionary Raquel Urtasun, is the leader in... ...leader to build a new centralized Evaluation team. This team will be... ...realistic closed-loop simulation engine built with the latest in generative... ...AI or machine learning models, ideally in self-driving or related...Full time
$80k - $138k per year
...Job Type: Permanent Work Model: Hybrid Reference code: 133554 Primary Location... ...on the job coaching -- As a Data & AI Platform Engineer, you operate at the foundation of modern... ..., and upgrades using tools like Azure DevOps, GitHub Actions, or Jenkins. • Deploy,...Permanent employmentFlexible hours- ...build, and solve meaningful problems with technology. The Opportunity We are looking for a passionate and detail-oriented DevOps Engineer to partner with our development teams and accelerate the deployment, testing, and release of applications. This role involves automating...Full time
- ...Generative AI Engineer About Apertera Apertera is leading the evolution of language solutions... ...Generative AI (GenAI) and Large Language Model (LLM) technologies. You will report to... ...translation, cloud infrastructure, and evaluation. You'll play a pivotal role in pushing the...Full timeShift work
$86k - $118k per year
As a DevOps Cloud Engineer you'll play a critical role in designing, automating, and operating the cloud infrastructure that powers Nasdaq's global solutions and enables seamless software delivery at scale. You'll thrive in this position if you're a collaborative problem...Long term contractFull timeTemporary work$65k - $85k per year
...SYSTRA group, an international consulting and engineering group, a world leader in the design of... ...Engineer Job Purpose The BIM Modeller – Structural Engineering is responsible... ...purposes and includes the feminine. We use AI-enabled tools in our recruitment platform...Internship- ...support of Artificial Intelligence (AI) Generative AI and... ...Agents leveraging Large Language Models (LLMs) Retrieval Augmented Generation... ...Generation (RAG) prompt engineering vector databases and semantic... ...technologies. ~ Experience evaluating and selecting AI technologies...Full time
$90.57k - $135.95k per year
...to develop software that provides functional, timing and power models of Intel's Field Programmable Gate Arrays (FPGAs). The significant... .... Education Requirements ~ Bachelor's degree in electrical engineering, computer engineering, Computer Science related field or master'...Full timeInternshipLocal areaFlexible hours$56k - $84k per year
...Job Type: Permanent Work Model: Hybrid Reference code: 133442 Primary Location... ...the Insurance practice in our Financial Engineering & Modeling team, you will: Conduct independent... ...utilize human screening combined with AI technology to help identify the skills and...Permanent employmentFlexible hours- ...and details. (It will be great help if you can refer someone for the same in case you are not currently available) Role: AEM DevOps Engineer Location: Toronto,ON Duration- Long Term Roles and Responsibility: AEM DevOps Skills Required: Understanding of...Long term contractFull timeInternship
- ...to create the best possible fit. Purpose of Job EQ Bank’s IT team is starting to evolve and Needs a talented Senior DevOps Engineer with a solid passion in this space. Experienced and knowledgeable about the end to end software build chain, operations and automated...Long term contract
$140.6k - $190.6k per year
...The Lead AI Forward Engineer designs and guides the delivery of AI-powered solutions that reduce operational... ...maintainable, and extensible. You will evaluate emerging AI technologies, define... ...flexible architectures that can evolve with model/provider changes and emerging AI...Full timeManual laborWork at officeLocal areaFlexible hours2 days per week3 days per week- ...that makes this real: a unified AI agent that sells, supports,... ...capabilities, we need to improve how we evaluate, iterate, and operate these... ...and tooling used by AI and engineering teams Reduce manual... ...and Evaluation loops around models to make them safe for production...Long term contractRemplacementFull timeInternshipWork at officeWork from homeShift work
$130k - $165k per year
...NetBrain’s platform. What We Need We’re looking for a Senior AI Engineer to design and build production-grade agent and RAG systems that... ...system-level thinking—owning everything from architecture and evaluation to scalability, observability, and reliability in production....Work at office1 day per week
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to DevOps Engineer - AI Model Evaluator. Be the first to apply!
Related searches