DevOps Engineer - AI Model Evaluator
$85 per hourMercor
Toronto, ON
- Remote job
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .
Position: DevOps / SRE / Cloud Engineer (Coding Agent Experience)
Type: Contract
Compensation: $85/hour
Location: Remote
Role Responsibilities
- Use frontier AI coding agents to complete and evaluate complex infrastructure engineering tasks.
- Review model-generated implementations involving cloud platforms , Kubernetes , CI/CD systems , and infrastructure automation .
- Identify bugs, edge cases, reliability issues, and failure modes in model outputs.
- Compare outputs from multiple frontier models to assess strengths and weaknesses.
- Apply professional engineering judgment to realistic infrastructure engineering scenarios.
Qualifications
Must-Have
- 2+ years of professional DevOps , SRE , or Cloud Engineering experience.
- Experience with AWS , Azure , GCP , Kubernetes , Terraform , CI/CD pipelines , or observability tooling.
- Regular use of AI coding agents like Cursor , Claude Code , Codex , Windsurf , Gemini CLI , or similar tools.
- Ability to evaluate model-generated infrastructure and reliability engineering solutions.
Preferred
- Experience supporting production-scale systems .
Compensation & Legal
- $400 per accepted task .
- Compensation is tied to accepted work.
Application Process (Takes 20–30 mins to complete)
- Upload resume
- AI interview based on your resume
- Submit form
Resources & Support
- For details about the interview process and platform information, please check:
- For any help or support, reach out to: Show email
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
Vacancy posted 7 days ago
Similar jobs that could be interesting for youBased on the DevOps Engineer - AI Model Evaluator in Toronto, ON vacancy
- ...currently seeking experienced software engineers to contribute to improving advanced AI systems through human feedback.... ...organizations in training large language models to better understand software... ...key role in shaping how AI models evaluate performance, detect issues, and generate...SuggestedContract workInternshipRemote workFlexible hours
$85 per hour
...and technical talent with leading AI research labs. Headquartered in San... ...Dorsey . Position: iOS Engineer (Coding Agent Experience) Type... ...AI coding agents to complete and evaluate complex engineering tasks. Review model-generated mobile application code...SuggestedRemote jobContract workSummer work- An innovative company is seeking a skilled Software Engineer to help train AI chatbots. This exciting remote role allows you to work on your own... ...coding challenges, writing high-quality code snippets, and evaluating AI-generated code for accuracy and performance. If you're passionate...SuggestedRemote work
- ...powered portfolio of business, technology, and engineering services and solutions is delivered... ...unique Location Independent Agile™ delivery model, recognized as a benchmark of excellence... ...CI/CD pipelines (GitHub Actions, Azure DevOps, Circles, Jenkins). Build, deploy, and...SuggestedFull timeFlexible hours
- Working knowledge of SSO, Entra ID/Azure AD, SAML, and TLS certificate renewals. Understanding of Database and reporting platforms like SQL/Snowflake/Postgres Familiarity with Kubernetes, Linux/Windows administration. Solid understanding of application monitoring and...SuggestedFull time
- Senior DevOps Engineer – Toronto, ON We’re looking for a Senior DevOps Engineer to support our team in Toronto, ON. This role offers the opportunity... ...in Hong Kong. Leveraging advanced technologies such as AI, cloud computing, and big data, we deliver innovative services across...Full time
- DevOps is part of a global organization that leverages the latest technology to communicate with our colleagues across the globe. We organize... ...belonging, collaboration, and accomplishment. Being a DevOps Engineer at iManage Means… You are an engineer, a builder, and a systems...Work at officeLocal areaRemote workWorldwideMonday to fridayFlexible hours
- Elevate your career as a Data Scientist dedicated to AI model validation at RBC. Engage with cutting‑edge technologies to mitigate model risks and improve banking services. As part of RBC's AI validation team, you will oversee critical risk assessments and create validation...
- About the Role We are looking for a DevOps Engineer to design, build, and enhance the cloud and automation platforms that power Broadridge products and services. In this role you will work closely with development teams to understand their needs, deliver cutting‑edge solutions...
- Elevate your career as a DevOps Engineer with a focus on Site Reliability Engineering. Utilize your cloud services and AI expertise to enhance our deployment platform's operational efficiency. We are seeking a skilled DevOps Engineer with over five years of experience, dedicated...
- VXG is seeking an experienced DevOps Engineer to join our team and help build reliable, automated, and scalable infrastructure for our cutting‑edge video software platform. You will play a key role in streamlining deployments, monitoring, and troubleshooting across AWS...Full timeWork at office
- ...compliance, and operational excellence, while enabling rapid onboarding and modernization of data workloads. The Role As a Senior DevOps Engineer on the GBM Data Platform, you will provide technical leadership and hands‑on expertise to design, operate, and continuously...InternshipFlexible hours
$140.6k - $190.6k per year
The Lead AI Forward Engineer designs and guides the delivery of AI‑powered solutions that reduce operational... ...Design and Technology Strategy Evaluate and recommend AI/ML technologies and... ...flexible architectures that can evolve with model/provider changes and emerging AI...Manual laborWork at officeLocal areaFlexible hours2 days per week3 days per week- Take your DevOps expertise to the next level with ZoomInfo as a Senior DevOps Engineer. Focus on enhancing data processing, cloud infrastructure, and automation in a collaborative environment. In this key role, you will leverage your 7+ years of experience in DevOps or Site...
- Our client is seeking a skilled DevOps Engineer for a full time position in Toronto, ON. This role involves onboarding applications and ensuring efficient deployment through automation. This DevOps Engineer position requires at least 8 years of experience in the field. You...Full time
$86k - $118k per year
...solutions. Build and maintain robust CI/CD pipelines that enable engineering teams to deliver high‑quality software efficiently. Develop... ...or equivalent practical experience. 3–5 years of experience in DevOps, cloud engineering, or infrastructure automation. Solid hands‑on...Long term contractTemporary workWork at office3 days per week$118k - $176k per year
# SRE / DevOps Engineer - Dynatrace & ObservabilityK&K Global Talent Solutions INC.Full TimemidToronto, Ontario, CAPosted 2 days ago## Role OverviewK... ...* Deploy and manage Dynatrace including APM, Davis AI, RUM (Real User Monitoring), and infrastructure monitoring across...Permanent employmentFull time- Take on a pivotal role as a Senior DevOps Engineer specializing in our innovative Web3.0 platform. Collaborate with product and engineering teams to enhance deployment and performance strategies. In this key position, you will support marketplace deployments, troubleshoot...
- About the job RQ08359 - DevOPS/Cloud Engineer - Senior Task 1 Monthly Performance Status Report (where applicable) All deliverables that consultant worked on during past month are included Progress and specific accomplishment against the deliverable are described...
- The bet This is the most senior engineering hire Bold AI will make, this week. The person in this role... ...engagements. Define and defend vendor, model, and platform decisions with full consideration... ..., and human-in-the-loop checkpoints. Evaluate and integrate orchestration frameworks...InternshipRemote work
- Advance your career with a focus on AWS as a Senior DevOps Engineer specializing in Terraform implementation. Design and manage Infrastructure as Code, automating cloud-native application delivery. In this role, you will take charge of implementing Infrastructure as Code...
- The DevOps Solutions Architect is a member of the IT Enterprise Applications... ...- focusing on DevOps, SRE and AI Ops. DevOps focuses on the end... ...utilization, and foundational model error rates. Set up cost-... ...technology, software engineering, computer science, or related...Local area
$90k - $120k per year
MQ DataPower DevOps Engineering Specialist Position Description Location: This role can be situated anywhere across Canada within proximity to a CGI office to accommodate a hybrid work environment. Are you a seasoned DevOps engineering specialist with a passion for...Work at office- Job Title: Platform DevOps Engineer Location: Toronto, ON Note: Prior experience in BFSI, Public Sector, or Telecom is non‑negotiable. Position Overview Seeking an experienced Platform DevOps Engineer with a strong background in BFSI, Public Sector, or Telecom environments....
- Advance your career at Scotiabank as a PeopleSoft Platform Engineer based in Toronto, Ontario. This role merges administration with cutting-edge DevOps practices. Scotiabank is looking for a highly skilled Platform Engineer with substantial expertise in PeopleSoft applications...
- ...training and deploying frontier models for developers and enterprises who are building AI systems to power magical experiences... ...is a team of researchers, engineers, designers, and more, who are passionate... ...the future! Why this role? Evaluation is critical to making progress...Full timeWork at officeRemote workFlexible hours
- ...Stacktics Inc is seeking a highly skilled and hands-on AI Engineer with proven experience in deploying AI/ML models on Google Cloud Platform (GCP), particularly using... ...techniques for improved accuracy. Prototype and evaluate AI solutions using proofs of concept and pilot projects...Remote workFlexible hours
- This role is eligible for our hybrid work model: 2 days in-office This job posting is... ...existing, currently vacant position. Senior DevOps Engineer Our Technology team is the backbone of... ...foundations for Data Engineering, ML/AI, and Data Science teams Design, implement...Summer workWork at officeRemote workFlexible hours
$123k - $180.4k per year
...Autodesk is evolving its productivity engineering ecosystem by embedding AI‑powered engineering into its... ...optimization using intelligent agents Evaluate and integrate emerging AI frameworks... ...making Implement feedback loops, evaluation pipelines, and observability for reliable...Local area- ...Cognichip, we're not just building AI; we're enabling our customers... .... As a Chip Design Engineer , you'll be the crucial link... ...into actionable insights for model training and agentic design... ...ecosystem of benchmarks and evaluation infrastructure to measure and...InternshipWork at officeRemote workWork from home1 day per week
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to DevOps Engineer - AI Model Evaluator. Be the first to apply!
Related searches