DevOps Engineer - AI Model Evaluator
$85 per hourMercor
Toronto, ON
- Remote job
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .
Position: DevOps / SRE / Cloud Engineer (Coding Agent Experience)
Type: Contract
Compensation: $85/hour
Location: Remote
Role Responsibilities
- Use frontier AI coding agents to complete and evaluate complex infrastructure engineering tasks.
- Review model-generated implementations involving cloud platforms , Kubernetes , CI/CD systems , observability , and infrastructure automation .
- Identify bugs, edge cases, reliability issues, and failure modes in model outputs.
- Compare outputs from multiple frontier models to assess their strengths and weaknesses.
- Apply professional engineering judgment to realistic infrastructure engineering scenarios.
Qualifications
Must-Have
- 2+ years of professional DevOps , SRE , or Cloud Engineering experience.
- Experience with AWS , Azure , GCP , Kubernetes , Terraform , CI/CD pipelines , or observability tooling.
- Regular use of AI coding agents such as Cursor, Claude Code, Codex, Windsurf, Gemini CLI, or similar tools.
- Ability to evaluate model-generated infrastructure and reliability engineering solutions.
Preferred
- Experience supporting production-scale systems.
Compensation & Legal
- $400 per accepted task
- Compensation tied to accepted work.
Application Process (Takes 20–30 mins to complete)
- Upload resume
- AI interview based on your resume
- Submit form
Resources & Support
- For details about the interview process and platform information, please check:
- For any help or support, reach out to: Show email
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
Vacancy posted more than 2 months ago
Similar jobs that could be interesting for youBased on the DevOps Engineer - AI Model Evaluator in Toronto, ON vacancy
$85 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ..., and Jack Dorsey . Position: DevOps / SRE / Cloud Engineer (Coding Agent Experience) Type:... ...AI coding agents to complete and evaluate complex infrastructure engineering tasks...SuggestedRemote jobContract workSummer work$80 per hour
...and technical talent with leading AI research labs. Headquartered in... ...Jack Dorsey . Position: Data Engineer (Coding Agent Experience)... ...AI coding agents to complete and evaluate complex data engineering tasks. Review model-generated implementations involving...SuggestedRemote jobContract workSummer work- ...Palona’s AI agents operate in real restaurant environments: noisy phone lines... ...more than selecting the newest model. It requires disciplined evaluation, high-quality data, modeling judgment... ...looking for an applied AI Modeling Engineer to improve the intelligence, accuracy...SuggestedLong term contractFull timeTemporary workInternshipImmediate start
$105k - $175k per year
...Job Type: Permanent Work Model: Hybrid Reference code: 135131 Primary Location... ...Summary We are hiring a Senior DevOps Engineer to own and evolve our cloud platform on AWS... ...like? You will lead how we adopt AI for infrastructure and platform work—not...SuggestedLong term contractPermanent employmentFlexible hours- ...looking for mission-driven talent to help accelerate our growth. The Opportunity SWTCH is looking for a highly motivated DevOps Engineer to join our growing tech team. Reporting to the Director of Engineering, you will design, build, and optimize our AWS container platform...SuggestedFull time
$92k - $118k per year
...building secure, automated, and scalable DevOps solutions. Join Capco to help transform enterprise... ...data and analytics through cloud-native engineering and continuous delivery. The Role We... ...At Capco, we use artificial intelligence (AI) tools to support and enhance several...Full timeInternshipImmediate start$70k - $130k per year
...Job Responsibility: DevOps Engineer Hybrid Position - Minneapolis or Toronto only! The DevOps... ...with a willingness to explore and learn AI technologies, applying them to enhance DevOps... ...for up to 8 weeks per year, and hybrid model, empowering employees to achieve a better...Full timeWork at officeLocal areaFlexible hours2 days per week3 days per week$80k - $130k per year
Senior DevOps Engineering Consultant - Azure & AI Position Description The Senior DevOps Engineering consultant... ...and Kubernetes based deployment models. .Partner with engineering,... ...peuvent être utilisés pour aider à l’évaluation des candidatures, sans jamais remplacer...$140k - $175k per year
...Job Responsibility: About the team We are looking for brilliant staff DevOps engineers who have a passion for consumer products and want to join an energetic team that is changing the way people think about travel. We ship code daily, A/B test extensively, and have a primary...Full timeWork at officeLocal areaRemote work- ...AGENT DEVOPS PLATFORM ENGINEER ROLE DESCRIPTIONS Design and operate AI agents that automate tasks across the software development lifecycle. Build and maintain... ...identity governance, and least-privilege access models. • Embed security, compliance, vulnerability...Permanent employment
$72.06k - $138.94k per year
...At Capgemini Engineering, the world leader in engineering services, we... ...seeking a highly skilled Senior DevOps Engineer to help build, operate... ...through automation and AI-assisted operations. Your role... ...Understanding of ML lifecycle management, model deployment, and production...Permanent employmentFull timeLocal areaRemote work- ...platforms such as Amazon EKS and Google Kubernetes Engine (GKE). ~ Provision and manage cloud... ...and tooling using Python to support DevOps processes. ~ Monitor system performance,... ...ZoomInfo (NASDAQ: GTM), the all-in-one AI GTM platform, enables sales, marketing, and...
$100 - $150 per hour
...technical talent with leading AI research labs. Headquartered in... ...data analyses , statistical modeling work , machine learning... ..., and technical reports. Evaluate AI-generated or human-created... ...detailed written justifications for evaluations and scores to maintain...Remote jobContract workSummer work- ...Job Responsibility: We are looking for "Senior AWS DevOps Engineer" in Canada. Full-time Permanent/T4 (hourly) - 100% Remote Responsibilities:- By utilizing Python, NodeJs, CI/CD, GitOps Methodologies, etc., create and extend Full Stack infrastructure automation for enterprise...Hourly payPermanent employmentFull timeRemote work
- ...Director, Engineering, AI Job Description Are you passionate about transforming how legal professionals... ...to a mission-based operating model. Replace project-by-project handoffs with... ...retrieval and model integration through evaluation, deployment, and monitoring. Set the...Long term contractFull timeContract workWork at officeLocal areaFlexible hoursShift work
$140.6k - $190.6k per year
...Job Description: Engineering Manager, Are you passionate about transforming... ...future of legal research with AI? Join our Global Westlaw... ...(RAG), and large language models (LLMs). You will also champion... ...review processes. Champion a DevOps-first mindset in which...Full timeInternshipWork at officeLocal areaWorldwideFlexible hoursShift work$109k - $173k per year
...City: Toronto Project Objectives & Role Summary The AI DevOps Specialist is primarily responsible for the technical... ...Celestica’s global HPS Artificial Intelligence (AI) sandbox, modeling, and software engineering toolchains. This specialist will bridge the gap...Temporary workLocal areaRemote work- ...CoreFactor is searching for a DevOps Engineer on a permanent/full-time basis. This position is hybrid and will require the successful incumbent... ...four (4) times per week. We are seeking an experienced AI/ML DevOps Engineer to join our clients Data, AI & Analytics team...Permanent employmentFull timeWork at office
- ...Our team is looking for a DevOps engineer with at least 6 years cloud and Kubernetes experience including specialization in ArgoCD. Our project involves building automations for a greenfield managed service product in AWS and we are looking for candidates who are interested...Full time
- ...Number : R2868511 Position title : AI Engineer Department: Commercial Data Science... ...enterprise data foundation strategies Monitor, evaluate, and continuously improve AI application... ...emerging AI technologies, foundational models, frameworks, and engineering practices,...Work at officeWork from homeHome officeFlexible hours
- ...DevOps Engineer About us TitanFile's mission is to democratize secure client communication. This is a challenge worthy of pursuit! The costs... ...problems. We expect our engineers to work fluently with AI tooling and to apply the same engineering judgment to what it produces...Full timeWork at officeFlexible hours
- ...staging, and production. Working with developers, architects, testers, cybersecurity specialists, and operational teams, the Senior DevOps Engineer improves deployment consistency, reduces manual effort, strengthens traceability, and supports secure and repeatable releases....Long term contractContract workInternship
$100 per hour
...Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is... ...evaluate clinical EHR vignettes for a medical AI evaluation program. General and internal medicine are our...Hourly payPermanent employmentPart time10 hours per week- ...We are seeking a Lead Azure DevOps Engineer with deep expertise in Azure Networking to drive the design, implementation, and optimization of... ...conform to best practices for Terraform IaC creation; a generic AI-developed Terraform solution is not enough. Ability to set the right...Remote job
- ...Our team is looking for a DevOps engineer with at least 7 years cloud experience including specialization with Identity Our project involves building automations for a greenfield managed service product in AWS and we are looking for candidates who are interested in designing...Full time
$95 per hour
...Experience 5 years or more Hybrid Work must be completed both in person and remotely. Experience and specialization Computer and technology knowledge Cloud Development and operations (DevOps) Jira Amazon Web Services (AWS) Atlassian Confluence API...Permanent employmentFull timeRemote work- ...Senior AI Engineer Toronto Downtown, ON - Hybrid (4 Days WFO) In Person Interview Required... ...LLM pipelines, Agentic workflows, Model Context Protocol (MCP), Graph/RAG architectures... ...using Apache Airflow/DAGs. • Cloud & DevOps (AWS): Comprehensive knowledge of AWS infrastructure...Long term contractPermanent employment
$90k - $115k per year
...greener and smarter mobility worldwide, connecting cities as we reduce carbon and replace cars. Could you be the full-time DevOps Engineer in Toronto, ON we’re looking for? Your future role We’re looking for a motivated DevOps Engineer to help modernize and...Long term contractFull timeInternshipWorldwide$84k - $126k per year
...Job Type: Permanent Work Model: Hybrid Reference code: 133157 Primary Location: Toronto, ON All Available Locations: Toronto... ...and hiring teams will utilize human screening combined with AI technology to help identify the skills and qualities that matter most...Permanent employmentFlexible hours- ...Cloud Platform Engineer – AWS, Python, DevOps & AI/LLM Required Skillsets: • 3–5+ years of hands-on CI/CD, cloud, and platform engineering... ...: prompt management, tool use, agent architectures, evaluation, observability, and reliability. • Experience with Git...Permanent employment
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to DevOps Engineer - AI Model Evaluator. Be the first to apply!
Related searches