DevOps Engineer - AI Model Evaluator
$85 per hourMercor
Toronto, ON
- Remote job
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .
Position: DevOps / SRE / Cloud Engineer (Coding Agent Experience)
Type: Contract
Compensation: $85/hour
Location: Remote
Role Responsibilities
- Use frontier AI coding agents to complete and evaluate complex infrastructure engineering tasks.
- Review model-generated implementations involving cloud platforms , Kubernetes , CI/CD systems , and infrastructure automation .
- Identify bugs, edge cases, reliability issues, and failure modes in model outputs.
- Compare outputs from multiple frontier models to assess strengths and weaknesses.
- Apply professional engineering judgment to realistic infrastructure engineering scenarios.
Qualifications
Must-Have
- 2+ years of professional DevOps , SRE , or Cloud Engineering experience.
- Experience with AWS , Azure , GCP , Kubernetes , Terraform , CI/CD pipelines , or observability tooling.
- Regular use of AI coding agents like Cursor , Claude Code , Codex , Windsurf , Gemini CLI , or similar tools.
- Ability to evaluate model-generated infrastructure and reliability engineering solutions.
Preferred
- Experience supporting production-scale systems .
Compensation & Legal
- $400 per accepted task .
- Compensation is tied to accepted work.
Application Process (Takes 20–30 mins to complete)
- Upload resume
- AI interview based on your resume
- Submit form
Resources & Support
- For details about the interview process and platform information, please check:
- For any help or support, reach out to: Show email
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
Vacancy posted 28 days ago
Similar jobs that could be interesting for youBased on the DevOps Engineer - AI Model Evaluator in Toronto, ON vacancy
$85 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ..., and Jack Dorsey . Position: DevOps / SRE / Cloud Engineer (Coding Agent Experience) Type:... ...AI coding agents to complete and evaluate complex infrastructure engineering tasks...SuggestedRemote jobContract workSummer work$85 per hour
...and technical talent with leading AI research labs. Headquartered in San... ...Dorsey . Position: iOS Engineer (Coding Agent Experience) Type... ...AI coding agents to complete and evaluate complex engineering tasks. Review model-generated mobile application code...SuggestedRemote jobContract workSummer work$80k - $180k per year
...Zafin is an AI platform company helping regulated institutions modernize how critical... ...scale. What’s the Opportunity? The AI Evaluation Engineer ensures AI agent solutions are accurate,... ...of AI evaluation, large language model behaviour, reasoning quality, hallucination...SuggestedFull time$130k - $145k per year
...role We are seeking a highly skilled and passionate Senior DevOps Engineer, AI infrastructure to join our team! You will play a key role in... ...and refine advanced prompting strategies for LLMs. Test, evaluate, and analyze the performance of LLM and other GenAI models....SuggestedFull timeInternshipWork at officeFlexible hours$101k - $169k per year
...Job Type: Permanent Work Model: Hybrid Reference code: 131871 Primary Location... ...complex, high-impact engagements in the AI Model Risk Space, ensuring alignment with... ...Generative AI offerings, including testing, evaluating and benchmarking different LLMs and toolchains...SuggestedPermanent employmentContract workFlexible hours- ...Our team is looking for a DevOps engineer with at least 6 years cloud and Kubernetes experience including specialization in ArgoCD. Our project involves building automations for a greenfield managed service product in AWS and we are looking for candidates who are interested...Full time
$90k - $130k per year
...space to grow. About the Role We are currently looking for DevOps Specialists for various large-scale projects. Why Join Levio?... ...on the Use of Artificial Intelligence in Recruitment We use AI enabled tools to help sort and review applications based on job related...Permanent employmentFull timeFlexible hours$72k - $138k per year
...Job Type: Permanent Work Model: Hybrid Reference code: 130069 Primary... ...-- We are looking for a passionate AI Research Engineer to join our team. You will work at the... ...research and product engineering—designing, evaluating, and deploying generative AI (GenAI) systems...Permanent employmentFlexible hours- ...Tiger Analytics is a global leader in AI and analytics, helping Fortune 1000... ...looking for a highly skilled AI Engineer with 7+ years of experience in software... ...AI/LLM APIs, AWS Bedrock, and other model providers. Infrastructure & DevOps: Experience with DevOps, CI/CD...Long term contractLocal area
$80k - $120k per year
...Zafin is an AI platform company helping regulated institutions modernize how critical work is designed, governed, and delivered. Our technology enables organizations to move faster while maintaining the governance, accountability, and control required in highly regulated environments...Full timeInternshipLocal area- ...Our team is looking for a DevOps engineer with at least 7 years cloud experience including specialization with Identity Our project involves building automations for a greenfield managed service product in AWS and we are looking for candidates who are interested in designing...Full time
- ...inclusive world. Job Description We are seeking to hire a Senior DevOps Engineer for the Client's Branch DevOps team. The Branch DevOps Team is... ...) to troubleshoot and remediate pipeline issues. · Leverage AI-assisted development tools (GitHub Copilot) to accelerate...Permanent employmentFull timeLocal areaShift work
- ...platforms such as Amazon EKS and Google Kubernetes Engine (GKE). ~ Provision and manage cloud... ...and tooling using Python to support DevOps processes. ~ Monitor system performance,... ...that empowers businesses to grow faster with AI-ready insights, trusted data, and advanced...Worldwide
- ...organizations to solve complex business challenges through cloud, AI, data engineering, and modern software solutions. We foster a collaborative,... ...outcomes. The Opportunity We're looking for a Senior DevOps Engineer who is deeply hands-on and ready to lead from the front...Full time
$92k - $118k per year
...building secure, automated, and scalable DevOps solutions. Join Capco to help transform enterprise... ...data and analytics through cloud-native engineering and continuous delivery. The Role We... ...At Capco, we use artificial intelligence (AI) tools to support and enhance several...InternshipImmediate start$135k - $170k per year
...General Information: Job Title: AI Engineer Location: Toronto, ON (Onsite/Hybrid)... ...critical AI features Define and implement evaluation frameworks Optimize systems for... ...Thrive We support remote and hybrid work models by offering multiple office locations to...Long term contractFull timeInternshipWork at officeImmediate startRemote workFlexible hours- ...Job Responsibility: Sr DevOps Engineer At Cognizant, our global community sets us apartan energetic, collaborative, and inclusive workplace... ...for software development and delivery. Research and evaluate new technologies and tools to improve software development and...Full timeWork at office2 days per week
$80 - $120 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...Position: Software / AI / IT / data Evaluator Type: Contract Compensation... ...structured written feedback to improve AI model outputs . Apply deep subject-matter...Remote jobContract workSummer workWork at office$80 - $120 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...regulatory response with financial-services AI Evaluator Type: Contract Compensation... ...to meet deadlines while improving AI model performance . Qualifications Must...Remote jobContract workSummer workWork at office$80k - $138k per year
...Job Type: Permanent Work Model: Hybrid Reference code: 133554 Primary Location... ...on the job coaching -- As a Data & AI Platform Engineer, you operate at the foundation of modern... ..., and upgrades using tools like Azure DevOps, GitHub Actions, or Jenkins. • Deploy,...Permanent employmentFlexible hours$92k - $118k per year
...Capco is seeking a DevOps/DevSecOps Engineer to join our Technology & Engineering practice in Toronto. This role is ideal for someone with a strong... ...Talent Acquisition At Capco, we use artificial intelligence (AI) tools to support and enhance several parts of talent...InternshipImmediate start- ...Generative AI Engineer About Apertera Apertera is leading the evolution of language solutions... ...Generative AI (GenAI) and Large Language Model (LLM) technologies. You will report to... ...translation, cloud infrastructure, and evaluation. You'll play a pivotal role in pushing the...Full timeShift work
$70 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...Remote Role Responsibilities Evaluate AI-generated responses to ensure accuracy... ...structured written feedback to improve AI model outputs . Identify nuances, implicit...Remote jobContract workSummer work$101k - $169k per year
...Job Type: Permanent Work Model: Hybrid Reference code: 133157 Primary Location: Toronto, ON All Available Locations: Toronto... ...required. Hands-on experience in utilizing data analytics and AI-ML (artificial intelligence-machine learning) methodologies to...Permanent employmentManual laborFlexible hours$135k - $150k per year
...at . The Opportunity: We’re looking for an AI Engineer, AI Platform & ML Engineering to join our Data & AI... ...on the platform patterns, MLOps practices, model lifecycle, deployment standards, monitoring, evaluation, and governance integration needed to move AI and...Full timeInternship- ...Senior DevOps Engineer Location: Toronto Role Description We are looking for a driven, top-notch Senior DevOps Engineer who is... ...certification (AWS, Kubernetes, or similar) Experience using AI-assisted development tools Knowledge of: JWT OAuth 2.0...
$55 - $65 per hour
...Job Responsibility: Role: Salesforce DevOps Engineer Location: Toronto/Calgary- Hybrid Hire Type: Contract Job Summary: We are seeking a skilled Salesforce DevOps Engineer to streamline and optimize our Salesforce development and release processes. You will be...Hourly payFull timeContract workFixed term contractMonday to friday$110k - $155k per year
Cybersecurity AI Engineering Specialist Position Description The Cybersecurity AI Engineering... ..., automation, and large language models (LLMs) to improve threat detection, accelerate... ...analysts. The successful candidate will evaluate emerging AI technologies, build secure...$90.37 - $108.45 per hour
...RQ11264 - DevOPS/Cloud Engineer - Senior 11 months Contract (235 business days) ONSITE 5 days... ...pull request and practicing GitOps AI Disclaimer: Source Code may use... ...to-have qualifications. The client will evaluate candidates based on both rate expectations...Contract work$84k - $175k per year
...Job Type: Permanent Work Model: Hybrid Reference code: 133810... ...design and implementation of AI-powered applications and... ...solutions including: o Prompt engineering o Retrieval-Augmented... ...mitigation o Monitoring and evaluation of model outputs • Integrate...Permanent employmentFlexible hours
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to DevOps Engineer - AI Model Evaluator. Be the first to apply!
Related searches