Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Software Engineer - AI Agent Evaluation

Temporary

Workada

What You'll Do

As a contributor, you'll take problems from your own work and turn each one into a self-contained Task that an AI agent has to solve in a Linux terminal. For every Task, you'll write a clear brief, set up the environment with the files and tools the agent needs, build a reference solution, and write automated checks on the final result. The strongest Tasks are the ones today's best agents still get wrong. Your firsthand knowledge of where real work in your field goes wrong is what makes them valuable. That might be a race condition that only shows up under load, a migration that silently drops rows, a compiler pass that miscompiles an edge case, a consensus protocol that splits on a network partition, or a CI build that flakes because of test ordering.

Specialties We're Looking For

We're building depth across: Algorithms and data structures Distributed and operating systems , including concurrency and consensus Databases and SQL , including migrations and query planning Data engineering : pipelines, ETL, and streaming Compilers and programming languages Frontend and web DevOps, CI/CD, and build systems

You're a Great Fit If

You hold, or are completing, a PhD in computer science or software engineering or a closely related field, or you have 5+ years of professional experience in it. You've done technical or computational work in your field and are comfortable in a Linux terminal: shell commands, scripting in Python or Bash, and installing packages. You've used Docker, or can pick it up quickly. We share templates and worked examples. You can draw on your own research, projects, code, or files as the source material for your Tasks. You have a precise, adversarial eye; you enjoy finding the exact point where a workflow breaks and explaining it cleanly. You're honest about uncertainty and write instructions that leave no room for misreading. You're based in the US or Canada. Academics and practitioners are equally welcome. No AI or machine-learning background is needed. Your expertise and your comfort working in a terminal are what we're after.

Materials You Can Use

Almost any piece of your work can seed a Task, for example: Repositories, schemas, and failing test suites CI configs and build scripts Data pipelines and the datasets behind them Profiler traces, logs, and incident write-ups Pull requests and bug fixes from your own work Bugs, edge cases, and failure modes you've had to debug yourself

Vacancy posted 19 hours ago
Similar jobs that could be interesting for youBased on the Software Engineer - AI Agent Evaluation in Remote vacancy
  •  ...conversation into a competitive advantage. Cresta’s unified AI platform combines conversational AI agents, real-time human agent augmentation, and...  ...time owner. That’s you.   As our first AI Agent Evaluation Engineer, you define what quality means for every agent we run... 
    Suggested
    Full time
    Work at office
    Remote work
    Home office
    Flexible hours

    Cresta

    Remote
    8 days ago
  •  ...conversation into a competitive advantage. Cresta’s unified AI platform combines conversational AI agents, real-time human agent augmentation, and...  ...full-time owner. That’s you. As our first AI Agent Evaluation Engineer, you define what quality means for every agent we run... 
    Suggested
    Full time

    Cresta

    Remote
    8 days ago
  •  ...Our vision is to build an ML/AI first network of advertisers and...  ...hiring its first dedicated AI Engineer. Over the past year, AI at...  ...to production: a service desk agent that closes tickets without a...  ...demo and a production system: evaluation, monitoring, cost, and a clear... 
    Suggested
    Full time
    Remote work

    Fluent, LLC

    Remote
    16 days ago
  •  ...conversation into a competitive advantage. Cresta’s unified AI platform combines conversational AI agents, real-time human agent augmentation, and comprehensive...  ...after transfer We are looking for strong backend engineers who want to work at the intersection of distributed... 
    Suggested
    Work at office
    Local area
    Remote work
    Work from home
    Home office

    Cresta

    Remote
    11 days ago
  •  ...competitive advantage. Cresta’s unified AI platform combines conversational AI agents, real-time human agent augmentation...  ...test execution frameworks that let engineers iterate quickly and ship with...  ...What we’re looking for ~5+ years of software engineering experience, with... 
    Suggested
    Work at office
    Local area
    Remote work
    Work from home
    Home office

    Cresta

    Remote
    11 days ago
  • $225k - $285k per year

     ...builds the best-in-class Voice AI models powering the next...  ...through our dashboard. Their coding agent does. Claude Code, Cursor,...  ...others are now how a big share of engineers discover, integrate and debug...  ...: ~2+ years of professional software engineering experience, with solid... 
    Full time

    AssemblyAI

    Remote
    2 days ago
  • $90 per hour

     ...Pay: Up to $90+/hr Experienced software engineers review and score AI-written code for frontier AI research teams. Remote and flexible, up to 40 hours a week. Pay is up to $90+ per hour for the most experienced engineers. Requirements: several years of professional... 
    Hourly pay
    Contract work
    For contractors
    Live In
    Remote work
    Flexible hours

    rustLabs

    Remote
    22 days ago
  •  ...services are deeply integrated and designed for growth. About the Role We’re looking for a Software Engineer to join our AI Tooling team and help shape how developers and agents build with Supabase. You’ll build and own our AI tooling, including agents, the MCP server,... 
    Full time
    Remote work
    Flexible hours

    Supabase

    Remote
    8 days ago
  •  ...all applications and next steps. Our partner is looking for a Software Engineer - AI Tooling based in Canada. As a Software Engineer on the AI...  ...and tools. You will design, build, and maintain intelligent agents, Model Context Protocol (MCP) servers, agent skills, and AI-powered... 
    Full time

    Jobgether

    Remote
    9 days ago
  •  ...all applications and next steps. Our partner is looking for a Software Engineer, AI Integrations based in Canada. The Software Engineer, AI...  ...maintain integrations connecting APIs and data services with coding agents, developer frameworks, cloud platforms, and automation tools.... 
    Long term contract

    Jobgether

    Remote
    10 days ago
  •  ...applications and next steps. Our partner is looking for a Senior Software Engineer – AI based in Canada. The Senior Software Engineer will join a...  ...AI Platform team responsible for building production-ready AI agents, intelligent products, and shared infrastructure across a... 
    Long term contract
    Full time

    Jobgether

    Remote
    10 days ago
  •  ...Fleetworthy is shaping the future of fleet readiness with AI-enabled, connected fleet technology that keeps...  ....com. About the role: We’re hiring a Senior Software Engineer for the AI Platform team. The team builds AI agents and products across Fleetworthy’s fleet readiness... 
    Long term contract
    Full time

    Bestpass by Fleetworthy

    Remote
    9 days ago
  • $128.5k - $160.6k per year

     ...daily at Coalition. About the role We are hiring a Software Engineer for the Automation team to use AI to solve and automate manual work across Coalition....  ...hands-on AI engineering role: authoring skills, wiring agents to our systems of record through MCP, building evals,... 
    Full time
    Fixed term contract
    Live In
    Work at office
    Remote work
    Home office
    Flexible hours

    Coalition, Inc.

    Remote
    14 days ago
  •  ...solution database and continuous calculation engine, driving efficiency, productivity and...  ...the opportunity We are looking for a Senior Software Developer to join the team that is building search, AI Assistant, and AI agents capabilities into our Dayforce product. The... 
    Full time
    Remote work

    Dayforce

    Remote
    2 days ago
  •  ...trust GitLab to ship better, more secure software faster. The same principles built...  ...in how our team works: we embrace AI as a core productivity multiplier, with...  ...role As an Intermediate Backend Engineer (Python) in GitLab's Agent Foundations stage, you'll go beyond using... 
    Full time
    Internship
    Remote work

    GitLab

    Remote
    11 days ago
  •  ...processes evidence, there's a good chance our software is working behind the scenes. That's been true for over four decades. AI is central to where we're going. Not as a...  .... The Role The AI Native Senior Software Engineer is a change agent and a driver of value within our public... 
    Full time
    Work from home

    Versaterm

    Remote
    11 days ago
  •  ...About the role AI assistants are becoming part of everyday life, helping people with...  ...and select your preferred experience. Evaluate conversational quality, helpfulness,...  ...simple testing guidelines and document your evaluations accurately. Who we're looking for Active... 
    Hourly pay
    Temporary work
    Part time
    Remote work

    micro1

    Remote
    8 days ago
  •  ...competitive advantage. Cresta’s unified AI platform combines conversational AI agents, real-time human agent augmentation...  ..., and reliable.  As an AI Agent Engineer , you'll be at the forefront of...  ...with customers as well as our software and machine learning engineers, ensuring... 
    Full time
    Work at office
    Local area
    Remote work
    Work from home
    Home office

    Cresta

    Remote
    11 days ago
  •  ...leading security-first enterprise AI company. We build cutting-edge...  ...is a team of researchers, engineers, designers, and more, who are...  ...platform that powers autonomous AI agents at scale. You’ll work on hard,...  ...management - Prototype and evaluate new technologies, then help... 
    Full time
    Manual labor
    Local area
    Remote work

    Cohere

    Remote
    23 days ago
  • $168k - $177.5k per year

     ...and shoppers to thrive. The Enterprise AI team develops intelligent systems and capabilities...  ...the company. We are seeking a Senior Software Engineer to join the Enterprise AI team and help...  ...AI team by identifying opportunities, evaluating new approaches, and sharing knowledge... 
    Long term contract
    Permanent employment
    Work at office
    Remote work
    Work from home
    Flexible hours

    Instacart

    Remote
    11 days ago
  •  ...infrastructure, and developer workflows that make AI-powered engineering more useful, reliable, and easy for Vanta...  ...the foundations for a more autonomous software development experience at Vanta, including background agents, agent evaluation systems, and tooling that helps... 
    Full time
    Remote work
    Flexible hours

    Vanta

    Remote
    11 days ago
  •  ...applications and next steps. Our partner is looking for a Full Stack Software Engineer – AI-First based remotely in Canada. In this role, you will help...  ...manual coding, you will leverage advanced AI coding agents to accelerate development, guide implementation, and deliver... 
    Remote work

    Jobgether

    Remote
    9 days ago
  • $100 per hour

     ...Check out this Loom video for more details: We're looking for highly experienced software engineer (SR+) to help evaluate the quality of interactions with modern coding agents such as OpenAI Codex and Claude Code. This is not a traditional engineering role. You... 
    Contract work
    Remote work
    Worldwide
    Flexible hours

    G2i

    Remote
    26 days ago
  •  ...Alpheva AI is looking for experienced software engineers to help train and evaluate AI coding systems through realistic repositories and engineering workflows. Projects capture the practical judgment behind reliable software, including understanding unfamiliar systems,... 
    Contract work

    Alpheva

    Remote
    a month ago
  • $153k - $213k per year

     ...what matters most. Affirm's engineering team is building a large-scale...  ...changes to systems. The AI Platform team owns Affirm's paved...  ...path for building, running, evaluating, and governing AI systems. Our...  ...interfaces to models, tools, and agents, with identity, cost... 
    Work at office
    Remote work
    Flexible hours

    Affirm

    Remote
    11 days ago
  •  ...position is listed on behalf of a partner company, which manages all applications and next steps. Our partner is looking for an AI Voice Evaluation Specialist based in Canada. Join an innovative AI-focused environment as an AI Voice Evaluation Specialist, where you will... 

    Jobgether

    Remote
    4 days ago
  •  ...applications and next steps. Our partner is looking for a Senior Software Engineer – AI Oversight based in Canada. As a Senior Software Engineer...  ...infrastructure that enables secure, privacy-preserving evaluations of advanced AI models. You will design and develop core systems... 
    Long term contract
    Full time
    Remote work

    Jobgether

    Remote
    17 days ago
  •  ...all applications and next steps. Our partner is looking for a Software Engineer – AI Tooling Infrastructure based in Canada. We are looking for an...  ...production services, including infrastructure supporting AI agents, developer assistants, and Model Context Protocol (MCP) integrations... 
    Long term contract
    Full time

    Jobgether

    Remote
    19 days ago
  •  ...About the role Samsara builds software that helps organizations run safer...  ...who use it. The Operational AI team is turning Samsara's...  ...their teams operate. As a senior engineer, you will build agentic solutions...  ...to connect AI detections and agents with operational workflows. -... 
    Long term contract
    Full time

    Samsara

    Remote
    7 days ago
  • $154.8k - $212.85k per year

     ...long term. About the role AI is changing how our R&D organization builds and runs software. This is a new role: one leader...  ...and whatever comes next. Dev Agents builds in-house and integrates...  ...the first-line work of an on-call engineer. It already runs on real incidents... 
    Long term contract
    Full time
    Internship
    Remote work
    Work from home
    Flexible hours
    Shift work

    Samsara

    Remote
    5 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Software Engineer - AI Agent Evaluation. Be the first to apply!