DevOps Engineer - AI Model Evaluator
$85 per hourMercor
Toronto, ON
- Remote job
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .
Position: DevOps / SRE / Cloud Engineer (Coding Agent Experience)
Type: Contract
Compensation: $85/hour
Location: Remote
Role Responsibilities
- Use frontier AI coding agents to complete and evaluate complex infrastructure engineering tasks.
- Review model-generated implementations involving cloud platforms , Kubernetes , CI/CD systems , and infrastructure automation .
- Identify bugs, edge cases, reliability issues, and failure modes in model outputs.
- Compare outputs from multiple frontier models to assess strengths and weaknesses.
- Apply professional engineering judgment to realistic infrastructure engineering scenarios.
Qualifications
Must-Have
- 2+ years of professional DevOps , SRE , or Cloud Engineering experience.
- Experience with AWS , Azure , GCP , Kubernetes , Terraform , CI/CD pipelines , or observability tooling.
- Regular use of AI coding agents like Cursor , Claude Code , Codex , Windsurf , Gemini CLI , or similar tools.
- Ability to evaluate model-generated infrastructure and reliability engineering solutions.
Preferred
- Experience supporting production-scale systems .
Compensation & Legal
- $400 per accepted task .
- Compensation is tied to accepted work.
Application Process (Takes 20–30 mins to complete)
- Upload resume
- AI interview based on your resume
- Submit form
Resources & Support
- For details about the interview process and platform information, please check:
- For any help or support, reach out to: Show email
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
Vacancy posted 28 days ago
Similar jobs that could be interesting for youBased on the DevOps Engineer - AI Model Evaluator in Toronto, ON vacancy
$85 per hour
...and technical talent with leading AI research labs. Headquartered in San... ...Dorsey . Position: iOS Engineer (Coding Agent Experience) Type... ...AI coding agents to complete and evaluate complex engineering tasks. Review model-generated mobile application code...SuggestedRemote jobContract workSummer work$80k - $180k per year
...Zafin is an AI platform company helping regulated institutions modernize how critical... ...scale. What’s the Opportunity? The AI Evaluation Engineer ensures AI agent solutions are accurate,... ...of AI evaluation, large language model behaviour, reasoning quality, hallucination...SuggestedFull time$130k - $145k per year
...role: We are seeking a highly skilled and passionate Senior DevOps Engineer, AI infrastructure to join our team! You will play a key role in... ...and refine advanced prompting strategies for LLMs. Test, evaluate, and analyze the performance of LLM and other GenAI models....SuggestedFull timeInternshipWork at officeFlexible hours$90k - $115k per year
...greener and smarter mobility worldwide, connecting cities as we reduce carbon and replace cars. Could you be the full-time DevOps Engineer in Toronto, ON we’re looking for? Your future role We’re looking for a motivated DevOps Engineer to help modernize and...SuggestedLong term contractFull timeInternshipWorldwide$56k - $84k per year
...Job Type: Permanent Work Model: Hybrid Reference code: 133514 Primary Location... ...someone with: Relevant experience in AI/ML domain Knowledge of NLP, LLMs (extractive... ...AI offerings, including testing, evaluating and benchmarking different LLMs and toolchains...SuggestedPermanent employmentContract workFlexible hours- ...Our team is looking for a DevOps engineer with at least 6 years cloud and Kubernetes experience including specialization in ArgoCD. Our project involves building automations for a greenfield managed service product in AWS and we are looking for candidates who are interested...Full time
- ...Tiger Analytics is a global leader in AI and analytics, helping Fortune 1000... ...looking for a highly skilled AI Engineer with 7+ years of experience in software... ...AI/LLM APIs, AWS Bedrock, and other model providers. Infrastructure & DevOps: Experience with DevOps, CI/CD...Long term contractLocal area
$80k - $120k per year
...Zafin is an AI platform company helping regulated institutions modernize how critical work is designed, governed, and delivered. Our technology enables organizations to move faster while maintaining the governance, accountability, and control required in highly regulated environments...Full timeInternshipLocal area- ...Our team is looking for a DevOps engineer with at least 7 years cloud experience including specialization with Identity Our project involves building automations for a greenfield managed service product in AWS and we are looking for candidates who are interested in designing...Full time
- ...inclusive world. Job Description We are seeking to hire a Senior DevOps Engineer for the Client's Branch DevOps team. The Branch DevOps Team is... ...) to troubleshoot and remediate pipeline issues. · Leverage AI-assisted development tools (GitHub Copilot) to accelerate...Permanent employmentFull timeLocal areaShift work
- ...organizations to solve complex business challenges through cloud, AI, data engineering, and modern software solutions. We foster a collaborative,... ...outcomes. The Opportunity We're looking for a Senior DevOps Engineer who is deeply hands-on and ready to lead from the front...Full time
$80 - $120 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...Position: Software / AI / IT / data Evaluator Type: Contract Compensation... ...structured written feedback to improve AI model outputs . Apply deep subject-matter...Remote jobContract workSummer workWork at office$92k - $118k per year
...building secure, automated, and scalable DevOps solutions. Join Capco to help transform enterprise... ...data and analytics through cloud-native engineering and continuous delivery. The Role We... ...At Capco, we use artificial intelligence (AI) tools to support and enhance several...InternshipImmediate start$135k - $170k per year
...General Information: Job Title: AI Engineer Location: Toronto, ON (Onsite/Hybrid)... ...critical AI features Define and implement evaluation frameworks Optimize systems for... ...Thrive We support remote and hybrid work models by offering multiple office locations to...Long term contractFull timeInternshipWork at officeImmediate startRemote workFlexible hours$60 - $62 per hour
...Platform / DevOps Engineer Client: a leading global eCommerce company Location: Toronto... ...records. Use of Artificial Intelligence (AI): We may use Artificial Intelligence (... ..., including sourcing, screening, and evaluating candidates. AI helps assess applications...Contract work$70 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...Remote Role Responsibilities Evaluate AI-generated responses to ensure accuracy... ...structured written feedback to improve AI model outputs . Identify nuances, implicit...Remote jobContract workSummer work$80 - $120 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...regulatory response with financial-services AI Evaluator Type: Contract Compensation... ...to meet deadlines while improving AI model performance . Qualifications Must...Remote jobContract workSummer workWork at office$80k - $138k per year
...Job Type: Permanent Work Model: Hybrid Reference code: 133554 Primary Location... ...on the job coaching -- As a Data & AI Platform Engineer, you operate at the foundation of modern... ..., and upgrades using tools like Azure DevOps, GitHub Actions, or Jenkins. • Deploy,...Permanent employmentFlexible hours$92k - $118k per year
...Capco is seeking a DevOps/DevSecOps Engineer to join our Technology & Engineering practice in Toronto. This role is ideal for someone with a strong... ...Talent Acquisition At Capco, we use artificial intelligence (AI) tools to support and enhance several parts of talent...InternshipImmediate start- ...Generative AI Engineer About Apertera Apertera is leading the evolution of language solutions... ...Generative AI (GenAI) and Large Language Model (LLM) technologies. You will report to... ...translation, cloud infrastructure, and evaluation. You'll play a pivotal role in pushing the...Full timeShift work
$135k - $150k per year
...at . The Opportunity: We’re looking for an AI Engineer, AI Platform & ML Engineering to join our Data & AI... ...on the platform patterns, MLOps practices, model lifecycle, deployment standards, monitoring, evaluation, and governance integration needed to move AI and...Full timeInternship- ...Sensor Modeling & Simulation Engineer (M&S) Preferred Location: Ottawa or Toronto Reports to: Head of... ...owns the sensor and electronic-warfare models end to end — RF/radar, EW, multi-sensor... ...equity granted based on impact We use AI tools to support parts of the hiring...Full time
- ...Senior DevOps Engineer Location: Toronto Role Description We are looking for a driven, top-notch Senior DevOps Engineer who is... ...certification (AWS, Kubernetes, or similar) Experience using AI-assisted development tools Knowledge of: JWT OAuth 2.0...
- ...Place to Work ®, five years in a row! As a Senior Modeling & Simulation (M&S) Engineer , you will develop models, tools, and simulations of next-generation space... ...prototype, and use new tools and technology (including AI) to learn, grow, and improve. Success in this role requires...Full timeWork at officeRelocation package
- ...Threat Modeler / Security Architect – AI Security, Cloud & DevSecOps Must Have Technical/Functional Skills... ...programs. Secure Design & Engineering Enablement • Develop, maintain... ...Develop innovative attack techniques to evaluate protective designs and existing...Permanent employment
$90.37 - $108.45 per hour
...RQ11264 - DevOPS/Cloud Engineer - Senior 11 months Contract (235 business days) ONSITE 5 days... ...pull request and practicing GitOps AI Disclaimer: Source Code may use... ...to-have qualifications. The client will evaluate candidates based on both rate expectations...Contract work- ...Waabi, founded by AI visionary Raquel Urtasun, is the leader in... ...way. To learn more visit: The Evaluation Algorithms team is responsible... ...realistic closed-loop simulation engine built with the latest in... ...provide a holistic understanding of model performance and enable the discovery...Full timeInternship
$95k - $115k per year
...focus on stronger infrastructure, better automation, practical AI adoption, improved reliability, stronger security practices, and... ...level. About the Role Big Viking Games is hiring a Senior DevOps Engineer to help design, maintain, secure, and modernize the infrastructure...Long term contractFull timeWork at office3 days per week- ...Senior AI Engineer Location: Ontario (5 Days Onsite) Duration: Long-term Contract... ...Bedrock Amazon Lex Large Language Models (LLMs) Retrieval-Augmented Generation (RAG... ...Responsible AI practices. ~ Experience evaluating AI models optimizing prompts monitoring...Long term contractFull time
$135k - $210k per year
...Overview: Guidepoint seeks an experienced AI Engineer as an integral member of the Toronto-... ...systems, including large language model (LLM) applications and research agents, while... ...practices for LLMOps, including automated evaluation using frameworks like LLM Judges or MLflow...Full timeWorldwide
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to DevOps Engineer - AI Model Evaluator. Be the first to apply!
Related searches