DevOps Engineer - AI Model Evaluator
$85 per hourMercor
Toronto, ON
- Remote job
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .
Position: DevOps / SRE / Cloud Engineer (Coding Agent Experience)
Type: Contract
Compensation: $85/hour
Location: Remote
Role Responsibilities
- Use frontier AI coding agents to complete and evaluate complex infrastructure engineering tasks.
- Review model-generated implementations involving cloud platforms , Kubernetes , CI/CD systems , observability , and infrastructure automation .
- Identify bugs, edge cases, reliability issues, and failure modes in model outputs.
- Compare outputs from multiple frontier models to assess their strengths and weaknesses.
- Apply professional engineering judgment to realistic infrastructure engineering scenarios.
Qualifications
Must-Have
- 2+ years of professional DevOps , SRE , or Cloud Engineering experience.
- Experience with AWS , Azure , GCP , Kubernetes , Terraform , CI/CD pipelines , or observability tooling.
- Regular use of AI coding agents such as Cursor, Claude Code, Codex, Windsurf, Gemini CLI, or similar tools.
- Ability to evaluate model-generated infrastructure and reliability engineering solutions.
Preferred
- Experience supporting production-scale systems.
Compensation & Legal
- $400 per accepted task
- Compensation tied to accepted work.
Application Process (Takes 20–30 mins to complete)
- Upload resume
- AI interview based on your resume
- Submit form
Resources & Support
- For details about the interview process and platform information, please check:
- For any help or support, reach out to: Show email
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
Vacancy posted 8 hours ago
Similar jobs that could be interesting for youBased on the DevOps Engineer - AI Model Evaluator in Toronto, ON vacancy
$80 per hour
...and technical talent with leading AI research labs. Headquartered in... ...Jack Dorsey . Position: Data Engineer (Coding Agent Experience)... ...AI coding agents to complete and evaluate complex data engineering tasks. Review model-generated implementations involving...SuggestedRemote jobContract workSummer work$85 per hour
...and technical talent with leading AI research labs. Headquartered in San... ...Dorsey . Position: iOS Engineer (Coding Agent Experience) Type... ...AI coding agents to complete and evaluate complex engineering tasks. Review model-generated mobile application code...SuggestedRemote jobContract workSummer work$80k - $180k per year
...Zafin is an AI platform company helping regulated institutions modernize how critical... ...scale. What’s the Opportunity? The AI Evaluation Engineer ensures AI agent solutions are accurate,... ...of AI evaluation, large language model behaviour, reasoning quality, hallucination...SuggestedFull time$130k - $145k per year
...role We are seeking a highly skilled and passionate Senior DevOps Engineer, AI infrastructure to join our team! You will play a key role in... ...and refine advanced prompting strategies for LLMs. Test, evaluate, and analyze the performance of LLM and other GenAI models....SuggestedFull timeInternshipWork at officeFlexible hours$39 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... .../week Role Responsibilities Evaluate AI model output lyrics , voice generation, and other... ...experience as a music producer, audio engineer, and/or sound mixer. ~ College degree...SuggestedRemote jobContract workSummer workImmediate start$80 - $120 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...Position: Software / AI / IT / data Evaluator Type: Contract Compensation... ...structured written feedback to improve AI model outputs . Apply deep subject-matter...Remote jobContract workSummer workWork at office$100 - $150 per hour
...technical talent with leading AI research labs. Headquartered in... ...data analyses , statistical modeling work , machine learning... ...and A/B test write-ups . Evaluate AI-generated or human-created... ...detailed written justifications for evaluations and scores to maintain...Remote jobContract workSummer work$80 - $120 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...regulatory response with financial-services AI Evaluator Type: Contract Compensation... ...structured written feedback to improve AI model outputs. Apply deep subject-matter...Remote jobContract workSummer workWork at office$120 - $170 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...data science deliverables such as analyses, models, dashboards, and experiment readouts.... ...~ Prior experience with AI training , evaluation, or human-data projects. Application...Remote jobContract workSummer work$56k - $84k per year
...Job Type: Permanent Work Model: Hybrid Reference code: 133514 Primary Location... ...someone with: Relevant experience in AI/ML domain Knowledge of NLP, LLMs (extractive... ...AI offerings, including testing, evaluating and benchmarking different LLMs and toolchains...Permanent employmentContract workFlexible hours$90k - $130k per year
...space to grow. About the Role We are currently looking for DevOps Specialists for various large-scale projects. Why Join Levio?... ...on the Use of Artificial Intelligence in Recruitment We use AI enabled tools to help sort and review applications based on job related...Permanent employmentFull timeFlexible hours$72k - $138k per year
...Job Type: Permanent Work Model: Hybrid Reference code: 130069 Primary... ...-- We are looking for a passionate AI Research Engineer to join our team. You will work at the... ...research and product engineering—designing, evaluating, and deploying generative AI (GenAI) systems...Permanent employmentFlexible hours$100 - $150 per hour
...technical talent with leading AI research labs. Headquartered... ...Dorsey . Position: Sales Engineering Expert Type:... ...supporting proofs of concept, and evaluating product fit and technical risk... ...experience with AI training , evaluation, or human-data projects....Remote jobContract workSummer work$150k - $200k per year
...communities. The Role As a Principal DevOps Engineer for a new product within Veeva, you will... ...member of a team building our next major AI-driven platform—one that will transform... ...Nice to Have ~ Experience with SQL, data modeling, and design Interviewing with Veeva...InternshipWork at officeLocal areaRemote workWork from home- ...please check Business France website: French International Young Talent Program: what is it? Could you be the full-time DevOps Engineer in Toronto, ON we’re looking for? Your future role We’re looking for a motivated DevOps Engineer to help modernize and automate...Full timeContract workInternshipWorldwide
- ...Tiger Analytics is a global leader in AI and analytics, helping Fortune 1000... ...looking for a highly skilled AI Engineer with 7+ years of experience in software... ...AI/LLM APIs, AWS Bedrock, and other model providers. Infrastructure & DevOps: Experience with DevOps, CI/CD...Long term contractLocal area
- ...Our team is looking for a DevOps engineer with at least 6 years cloud and Kubernetes experience including specialization in ArgoCD. Our project involves building automations for a greenfield managed service product in AWS and we are looking for candidates who are interested...Full time
$135k - $170k per year
...General Information: Job Title: AI Engineer Location: Toronto, ON (Onsite/Hybrid)... ...critical AI features Define and implement evaluation frameworks Optimize systems for... ...Thrive We support remote and hybrid work models by offering multiple office locations to...Long term contractFull timeInternshipWork at officeImmediate startRemote workFlexible hours$80k - $130k per year
Senior DevOps Engineering Consultant Position Description The Senior DevOps Engineering consultant... ...teams. This role is part of an AI first software engineering organization where... ...platforms and Kubernetes based deployment models. .Partner with engineering,...$80k - $120k per year
...Zafin is an AI platform company helping regulated institutions modernize how critical work is designed, governed, and delivered. Our technology enables organizations to move faster while maintaining the governance, accountability, and control required in highly regulated environments...Full timeInternshipLocal area$70k - $130k per year
...Job Responsibility: DevOps Engineer Hybrid Position - Minneapolis or Toronto only! The DevOps... ...with a willingness to explore and learn AI technologies, applying them to enhance DevOps... ...for up to 8 weeks per year, and hybrid model, empowering employees to achieve a better...Full timeWork at officeLocal areaFlexible hours2 days per week3 days per week- ...Our team is looking for a DevOps engineer with at least 7 years cloud experience including specialization with Identity Our project involves building automations for a greenfield managed service product in AWS and we are looking for candidates who are interested in designing...Full time
- ...organizations to solve complex business challenges through cloud, AI, data engineering, and modern software solutions. We foster a collaborative,... ...outcomes. The Opportunity We're looking for a Senior DevOps Engineer who is deeply hands-on and ready to lead from the front...Full time
$101k - $169k per year
...Job Type: Permanent Work Model: Hybrid Reference code: 133157 Primary Location: Toronto, ON All Available Locations: Toronto... ...required. Hands-on experience in utilizing data analytics and AI-ML (artificial intelligence-machine learning) methodologies to...Permanent employmentManual laborFlexible hours- ...Sensor Modeling & Simulation Engineer (M&S) Preferred Location: Ottawa or Toronto Reports to: Head of... ...owns the sensor and electronic-warfare models end to end — RF/radar, EW, multi-sensor... ...equity granted based on impact We use AI tools to support parts of the hiring...Full time
$140k - $180k per year
...infrastructure, better automation, practical AI adoption, improved reliability, stronger... ...Big Viking Games is hiring a Senior DevOps Engineer to modernize the infrastructure behind... ...infrastructure governance. Experience evaluating where automation should not be applied...Long term contractFull timeWork at office3 days per week$92k - $118k per year
...Capco is seeking a DevOps/DevSecOps Engineer to join our Technology & Engineering practice in Toronto. This role is ideal for someone with a strong... ...Talent Acquisition At Capco, we use artificial intelligence (AI) tools to support and enhance several parts of talent...InternshipImmediate start$84k - $175k per year
...Job Type: Permanent Work Model: Hybrid Reference code: 133810... ...design and implementation of AI-powered applications and... ...solutions including: o Prompt engineering o Retrieval-Augmented... ...mitigation o Monitoring and evaluation of model outputs • Integrate...Permanent employmentFlexible hours- ...Generative AI Engineer About Apertera Apertera is leading the evolution of language solutions... ...Generative AI (GenAI) and Large Language Model (LLM) technologies. You will report to... ...translation, cloud infrastructure, and evaluation. You'll play a pivotal role in pushing the...Full timeShift work
$92k - $118k per year
...building secure, automated, and scalable DevOps solutions. Join Capco to help transform enterprise... ...data and analytics through cloud-native engineering and continuous delivery. The Role We... ...At Capco, we use artificial intelligence (AI) tools to support and enhance several...InternshipImmediate start
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to DevOps Engineer - AI Model Evaluator. Be the first to apply!
Related searches