DevOps Engineer - AI Model Evaluator
$85 per hourMercor
Toronto, ON
- Remote job
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .
Position: DevOps / SRE / Cloud Engineer (Coding Agent Experience)
Type: Contract
Compensation: $85/hour
Location: Remote
Role Responsibilities
- Use frontier AI coding agents to complete and evaluate complex infrastructure engineering tasks.
- Review model-generated implementations involving cloud platforms , Kubernetes , CI/CD systems , observability , and infrastructure automation .
- Identify bugs, edge cases, reliability issues, and failure modes in model outputs.
- Compare outputs from multiple frontier models to assess their strengths and weaknesses.
- Apply professional engineering judgment to realistic infrastructure engineering scenarios.
Qualifications
Must-Have
- 2+ years of professional DevOps , SRE , or Cloud Engineering experience.
- Experience with AWS , Azure , GCP , Kubernetes , Terraform , CI/CD pipelines , or observability tooling.
- Regular use of AI coding agents such as Cursor, Claude Code, Codex, Windsurf, Gemini CLI, or similar tools.
- Ability to evaluate model-generated infrastructure and reliability engineering solutions.
Preferred
- Experience supporting production-scale systems.
Compensation & Legal
- $400 per accepted task
- Compensation tied to accepted work.
Application Process (Takes 20–30 mins to complete)
- Upload resume
- AI interview based on your resume
- Submit form
Resources & Support
- For details about the interview process and platform information, please check:
- For any help or support, reach out to: Show email
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
Vacancy posted 1 day ago
Similar jobs that could be interesting for youBased on the DevOps Engineer - AI Model Evaluator in Toronto, ON vacancy
$85 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ..., and Jack Dorsey . Position: DevOps / SRE / Cloud Engineer (Coding Agent Experience) Type:... ...AI coding agents to complete and evaluate complex infrastructure engineering tasks...SuggestedRemote jobContract workSummer work$80 per hour
...and technical talent with leading AI research labs. Headquartered in... ...Jack Dorsey . Position: Data Engineer (Coding Agent Experience)... ...AI coding agents to complete and evaluate complex data engineering tasks. Review model-generated implementations involving...SuggestedRemote jobContract workSummer work- ...Palona’s AI agents operate in real restaurant environments: noisy phone lines... ...more than selecting the newest model. It requires disciplined evaluation, high-quality data, modeling judgment... ...looking for an applied AI Modeling Engineer to improve the intelligence, accuracy...SuggestedLong term contractFull timeTemporary workInternshipImmediate start
$85 per hour
...and technical talent with leading AI research labs. Headquartered in San... ...Dorsey . Position: iOS Engineer (Coding Agent Experience) Type... ...AI coding agents to complete and evaluate complex engineering tasks. Review model-generated mobile application code...SuggestedRemote jobContract workSummer work$80k - $180k per year
...Zafin is an AI platform company helping regulated institutions modernize how critical... ...scale. What’s the Opportunity? The AI Evaluation Engineer ensures AI agent solutions are accurate,... ...of AI evaluation, large language model behaviour, reasoning quality, hallucination...SuggestedFull time$130k - $145k per year
...role We are seeking a highly skilled and passionate Senior DevOps Engineer, AI infrastructure to join our team! You will play a key role in... ...and refine advanced prompting strategies for LLMs. Test, evaluate, and analyze the performance of LLM and other GenAI models....Full timeInternshipWork at officeFlexible hours$39 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... .../week Role Responsibilities Evaluate AI model output lyrics , voice generation, and other... ...experience as a music producer, audio engineer, and/or sound mixer. ~ College degree...Remote jobContract workSummer workImmediate start$101k - $169k per year
...Job Type: Permanent Work Model: Hybrid Reference code: 131871 Primary Location... ...complex, high-impact engagements in the AI Model Risk Space, ensuring alignment with... ...Generative AI offerings, including testing, evaluating and benchmarking different LLMs and toolchains...Permanent employmentContract workFlexible hours$80 - $120 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...regulatory response with financial-services AI Evaluator Type: Contract Compensation... ...structured written feedback to improve AI model outputs. Apply deep subject-matter...Remote jobContract workSummer workWork at office$100 - $150 per hour
...technical talent with leading AI research labs. Headquartered in... ...data analyses statistical modeling work machine learning pipelines... ...A/B test write-ups . Evaluate AI-generated or human-... ...detailed written justifications for evaluations and scores to maintain...Full timeContract workSummer workRemote work$70 per hour
...technical talent with leading AI research labs. Headquartered in... ...Role Responsibilities Evaluate AI-generated responses to... ...strengthen reasoning and rigor in model outputs . Provide... ...accurate application of structured evaluation guidelines. Work...Remote jobContract workSummer work$80 - $120 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...regulatory response with financial-services AI Evaluator Type: Contract Compensation... ...structured written feedback to improve AI model outputs. Apply deep subject-matter...Remote jobContract workSummer workWork at office$56k - $84k per year
...Job Type: Permanent Work Model: Hybrid Reference code: 133514 Primary Location... ...someone with: Relevant experience in AI/ML domain Knowledge of NLP, LLMs (extractive... ...AI offerings, including testing, evaluating and benchmarking different LLMs and toolchains...Permanent employmentContract workFlexible hours$90k - $115k per year
...greener and smarter mobility worldwide, connecting cities as we reduce carbon and replace cars. Could you be the full-time DevOps Engineer in Toronto, ON we’re looking for? Your future role We’re looking for a motivated DevOps Engineer to help modernize and...Long term contractFull timeInternshipWorldwide$120 - $170 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...data science deliverables such as analyses, models, dashboards, and experiment readouts.... ...~ Prior experience with AI training , evaluation, or human-data projects. Application...Remote jobContract workSummer work$90k - $130k per year
...space to grow. About the Role We are currently looking for DevOps Specialists for various large-scale projects. Why Join Levio?... ...on the Use of Artificial Intelligence in Recruitment We use AI enabled tools to help sort and review applications based on job related...Permanent employmentFull timeFlexible hours$72k - $138k per year
...Job Type: Permanent Work Model: Hybrid Reference code: 130069 Primary... ...-- We are looking for a passionate AI Research Engineer to join our team. You will work at the... ...research and product engineering—designing, evaluating, and deploying generative AI (GenAI) systems...Permanent employmentFlexible hours- ...Aqua Scan, Snyk, etc. Experience working with DevOps pipelines (GitHub Actions, Maven, Angular) Experience... ...leveraging strengths from strategy and design to engineering, all fueled by its market leading capabilities in AI, generative AI, cloud and data, combined with its...Permanent employmentFull timeLocal area
$150k - $200k per year
...communities. The Role As a Principal DevOps Engineer for a new product within Veeva, you will... ...member of a team building our next major AI-driven platform—one that will transform... ...Nice to Have ~ Experience with SQL, data modeling, and design Interviewing with Veeva...InternshipWork at officeLocal areaRemote workWork from home$100 - $150 per hour
...technical talent with leading AI research labs. Headquartered... ...Dorsey . Position: Sales Engineering Expert Type:... ...supporting proofs of concept, and evaluating product fit and technical risk... ...experience with AI training , evaluation, or human-data projects....Remote jobContract workSummer work- ...Tiger Analytics is a global leader in AI and analytics, helping Fortune 1000... ...looking for a highly skilled AI Engineer with 7+ years of experience in software... ...AI/LLM APIs, AWS Bedrock, and other model providers. Infrastructure & DevOps: Experience with DevOps, CI/CD...Long term contractLocal area
- ...platforms such as Amazon EKS and Google Kubernetes Engine (GKE). ~ Provision and manage cloud... ...and tooling using Python to support DevOps processes. ~ Monitor system performance,... ...that empowers businesses to grow faster with AI-ready insights, trusted data, and advanced...Worldwide
- ...Our team is looking for a DevOps engineer with at least 6 years cloud and Kubernetes experience including specialization in ArgoCD. Our project involves building automations for a greenfield managed service product in AWS and we are looking for candidates who are interested...Full time
$80k - $120k per year
...Zafin is an AI platform company helping regulated institutions modernize how critical work is designed, governed, and delivered. Our technology enables organizations to move faster while maintaining the governance, accountability, and control required in highly regulated environments...Full timeInternshipLocal area$80k - $130k per year
Senior DevOps Engineering Consultant Position Description The Senior DevOps Engineering consultant... ...teams. This role is part of an AI first software engineering organization where... ...platforms and Kubernetes based deployment models. .Partner with engineering,...$80k - $138k per year
...Job Type: Permanent Work Model: Hybrid Reference code: 133554 Primary Location... ...on the job coaching -- As a Data & AI Platform Engineer, you operate at the foundation of modern... ..., and upgrades using tools like Azure DevOps, GitHub Actions, or Jenkins. • Deploy,...Permanent employmentFlexible hours$70k - $130k per year
...Job Responsibility: DevOps Engineer Hybrid Position - Minneapolis or Toronto only! The DevOps... ...with a willingness to explore and learn AI technologies, applying them to enhance DevOps... ...for up to 8 weeks per year, and hybrid model, empowering employees to achieve a better...Full timeWork at officeLocal areaFlexible hours2 days per week3 days per week$165k - $210k per year
...technology. Job Description AI Software Engineer Location(s): Toronto - 30 Adelaide... ...authenticity and offer a hybrid working model to ensure employees can manage their day... ...~ Context management strategies ~ AI evaluation and validation techniques Strong proficiency...Ongoing contractFull timeCasual workInternshipFlexible hours$101k - $169k per year
...Job Type: Permanent Work Model: Hybrid Reference code: 132112 Primary Location... ...our parallel streams within the Financial Engineering & Modeling group, including market risk,... ...will utilize human screening combined with AI technology to help identify the skills and...Permanent employmentFlexible hours- ...Our team is looking for a DevOps engineer with at least 7 years cloud experience including specialization with Identity Our project involves building automations for a greenfield managed service product in AWS and we are looking for candidates who are interested in designing...Full time
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to DevOps Engineer - AI Model Evaluator. Be the first to apply!
