DevOps Engineer - AI Model Evaluator
$85 per hourMercor
Toronto, ON
- Remote job
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .
Position: DevOps / SRE / Cloud Engineer (Coding Agent Experience)
Type: Contract
Compensation: $85/hour
Location: Remote
Role Responsibilities
- Use frontier AI coding agents to complete and evaluate complex infrastructure engineering tasks.
- Review model-generated implementations involving cloud platforms , Kubernetes , CI/CD systems , observability , and infrastructure automation .
- Identify bugs, edge cases, reliability issues, and failure modes.
- Compare outputs from multiple frontier models and assess their strengths and weaknesses.
- Apply professional engineering judgment to realistic infrastructure engineering scenarios.
- Work independently and asynchronously to meet sprint-based project deadlines.
Qualifications
Must-Have
- 2+ years of professional DevOps , SRE , or Cloud Engineering experience.
- Experience with AWS , Azure , GCP , Kubernetes , Terraform , CI/CD pipelines , or observability tooling .
- Regular use of AI coding agents such as Cursor, Claude Code, Codex, Windsurf, Gemini CLI, or similar tools.
- Ability to evaluate model-generated infrastructure and reliability engineering solutions.
Preferred
- Experience supporting production-scale systems.
Compensation & Legal
- $400 per accepted task .
- Compensation is tied to accepted work.
Application Process (Takes 20–30 mins to complete)
- Upload resume
- AI interview based on your resume
- Submit form
Resources & Support
- For details about the interview process and platform information, please check:
- For any help or support, reach out to: Show email
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
Vacancy posted 4 days ago
Similar jobs that could be interesting for youBased on the DevOps Engineer - AI Model Evaluator in Toronto, ON vacancy
$80 per hour
...and technical talent with leading AI research labs. Headquartered in... ...Jack Dorsey . Position: Data Engineer (Coding Agent Experience)... ...AI coding agents to complete and evaluate complex data engineering tasks. Review model-generated implementations involving...SuggestedFull timeContract workSummer workRemote work- ...Palona’s AI agents operate in real restaurant environments: noisy phone lines... ...more than selecting the newest model. It requires disciplined evaluation, high-quality data, modeling judgment... ...looking for an applied AI Modeling Engineer to improve the intelligence, accuracy...SuggestedLong term contractFull timeTemporary workInternshipImmediate start
- ...Senior Software Engineer Job Type: Contractor (~15 hours/week) Location: Remote... ...Senior Software Engineers to support an AI training project by creating reinforcement learning environments that evaluate AI models on complex software engineering tasks using...SuggestedRemote jobFor contractors
$80k - $180k per year
...Zafin is an AI platform company helping regulated institutions modernize how critical... ...scale. What’s the Opportunity? The AI Evaluation Engineer ensures AI agent solutions are accurate,... ...of AI evaluation, large language model behaviour, reasoning quality, hallucination...SuggestedFull time$80k - $130k per year
Senior DevOps Engineering Consultant - Azure & AI Position Description The Senior DevOps Engineering consultant... ...and Kubernetes based deployment models. .Partner with engineering,... ...peuvent être utilisés pour aider à l’évaluation des candidatures, sans jamais remplacer...Suggested$39 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... .../week Role Responsibilities Evaluate AI model output lyrics , voice generation, and other... ...experience as a music producer, audio engineer, and/or sound mixer. ~ College degree...Remote jobContract workSummer workImmediate start$70 per hour
...elite creative and technical talent with leading AI research labs. Headquartered in San Francisco,... ...Location: Remote Role Responsibilities Evaluate AI-generated responses to strengthen reasoning and rigor in model outputs . Provide structured written...Remote jobContract workSummer work$80 - $120 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...regulatory response with financial-services AI Evaluator Type: Contract Compensation... ...structured written feedback to improve AI model outputs. Apply deep subject-matter...Remote jobContract workSummer workWork at office$70 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...answers. Calibrate tasks against frontier models ensuring tasks ship only when strong models... ...Prior scientific software or research engineering experience. Start Date ~ Immediate...Full timeContract workSummer workImmediate startRemote work$100 - $150 per hour
...technical talent with leading AI research labs. Headquartered in... ...data analyses , statistical modeling work , machine learning... ..., and technical reports. Evaluate AI-generated or human-created... ...detailed written justifications for evaluations and scores to maintain...Remote jobContract workSummer work$70 per hour
...technical talent with leading AI research labs. Headquartered in... ...Role Responsibilities Evaluate AI-generated responses to... ...strengthen reasoning and rigor in model outputs . Provide... ...accurate application of structured evaluation guidelines. Work...Remote jobContract workSummer work$100 - $150 per hour
...technical talent with leading AI research labs. Headquartered in... ...data analyses statistical modeling work machine learning pipelines... ...A/B test write-ups . Evaluate AI-generated or human-... ...detailed written justifications for evaluations and scores to maintain...Full timeContract workSummer workRemote work$90k - $130k per year
...space to grow. About the Role We are currently looking for DevOps Specialists for various large-scale projects. Why Join Levio?... ...on the Use of Artificial Intelligence in Recruitment We use AI enabled tools to help sort and review applications based on job related...Permanent employmentFull timeFlexible hours$80 - $120 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...regulatory response with financial-services AI Evaluator Type: Contract Compensation... ...structured written feedback to improve AI model outputs. Apply deep subject-matter...Remote jobContract workSummer workWork at office$80 - $120 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ..., and Jack Dorsey . Position: Software / AI / IT / data Evaluator Type: Contract Compensation: $80–$120/hour...Remote jobContract workSummer workWork at office- ...Number : R2868511 Position title : AI Engineer Department: Commercial Data Science... ...enterprise data foundation strategies Monitor, evaluate, and continuously improve AI application... ...emerging AI technologies, foundational models, frameworks, and engineering practices,...Work at officeWork from homeHome officeFlexible hours
$100 - $150 per hour
...technical talent with leading AI research labs. Headquartered... ...Dorsey . Position: Sales Engineering Expert Type:... ...supporting proofs of concept, and evaluating product fit and technical risk... ...experience with AI training , evaluation, or human-data projects....Remote jobContract workSummer work$120 - $170 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...data science deliverables such as analyses, models, dashboards, and experiment readouts.... ...~ Prior experience with AI training , evaluation, or human-data projects. Application...Remote jobContract workSummer work- ...We need two Mid to senior level (8 years) DevOps engineers with good experience in CI/CD (AWS AZURE or GCP is fine) pipeline creation. Industry experience does not matter but banking is a plus. As a solution engineer you will be responsible for supporting application teams...Full timeInternshipLocal areaRelocation
$80k - $138k per year
...Job Type: Permanent Work Model: Hybrid Reference code: 133554 Primary Location... ...on the job coaching -- As a Data & AI Platform Engineer, you operate at the foundation of modern... ..., and upgrades using tools like Azure DevOps, GitHub Actions, or Jenkins. • Deploy,...Permanent employmentFlexible hours$140k - $175k per year
...Job Responsibility: About the team We are looking for brilliant staff DevOps engineers who have a passion for consumer products and want to join an energetic team that is changing the way people think about travel. We ship code daily, A/B test extensively, and have a primary...Full timeWork at officeLocal areaRemote work- ...Sensor Modeling & Simulation Engineer (M&S) Preferred Location: Ottawa or Toronto Reports to: Head of... ...owns the sensor and electronic-warfare models end to end — RF/radar, EW, multi-sensor... ...equity granted based on impact* We use AI tools to support parts of the hiring...Full time
- ...Tiger Analytics is a global leader in AI and analytics, helping Fortune 1000... ...looking for a highly skilled AI Engineer with 7+ years of experience in software... ...AI/LLM APIs, AWS Bedrock, and other model providers. Infrastructure & DevOps: Experience with DevOps, CI/CD...Long term contractLocal area
- ...Our team is looking for a DevOps engineer with at least 6 years cloud and Kubernetes experience including specialization in ArgoCD. Our project involves building automations for a greenfield managed service product in AWS and we are looking for candidates who are interested...Full time
$83k - $92.4k per year
...for a new tool comes up. Whenever we can, the code we produce is made public for all to use. Overview: As an Intermediate DevOps Engineer, you will join our collaborative team to support, operate, and enhance our global digital infrastructure. You will work closely with...Permanent employmentFull timeWork at officeLocal area2 days per week1 day per week$135k - $170k per year
...General Information: Job Title: AI Engineer Location: Toronto, ON (Onsite/Hybrid)... ...critical AI features Define and implement evaluation frameworks Optimize systems for... ...Thrive We support remote and hybrid work models by offering multiple office locations to...Long term contractFull timeInternshipWork at officeImmediate startRemote workFlexible hours- ...to Turkish Software Localization MT/LLM Evaluator Remote | US, Canada & Latin America Time... ...experts who want to work on cutting-edge AI and machine translation initiatives while... ...Responsibilities ~ MT/LLM Evaluation: Review machine-translated and AI-generated...Long term contractFor contractorsFreelanceRemote workWorldwideFlexible hours
$165k - $210k per year
...technology. Job Description AI Software Engineer Location(s): Toronto - 30 Adelaide... ...authenticity and offer a hybrid working model to ensure employees can manage their day... ...~ Context management strategies ~ AI evaluation and validation techniques Strong proficiency...Ongoing contractFull timeCasual workInternshipFlexible hours$93.6 - $112.32 per hour
...RQ00768 - DevOPS/Cloud Engineer - Senior Location: 3 days ONSITE at Metro Hall, 55 John Street,... ...protect against unauthorized access. AI Disclaimer: Source Code may use artificial... ...to-have qualifications. The client will evaluate candidates based on both rate...$90.37 - $108.45 per hour
...RQ11363 - DevOPS/Cloud Engineer - Senior Duration: 1 Year (252 Business Days) Location: ONSITE at... ...various versions of cellular and tablets AI Disclaimer: Source Code may use... ...to-have qualifications. The client will evaluate candidates based on both rate expectations...
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to DevOps Engineer - AI Model Evaluator. Be the first to apply!
Related searches