Software Engineer - AI Agent Evaluation
Workada
What You'll Do
As a contributor, you'll take problems from your own work and turn each one into a self-contained Task that an AI agent has to solve in a Linux terminal. For every Task, you'll write a clear brief, set up the environment with the files and tools the agent needs, build a reference solution, and write automated checks on the final result. The strongest Tasks are the ones today's best agents still get wrong. Your firsthand knowledge of where real work in your field goes wrong is what makes them valuable. That might be a race condition that only shows up under load, a migration that silently drops rows, a compiler pass that miscompiles an edge case, a consensus protocol that splits on a network partition, or a CI build that flakes because of test ordering.
Specialties We're Looking For
We're building depth across: Algorithms and data structures Distributed and operating systems , including concurrency and consensus Databases and SQL , including migrations and query planning Data engineering : pipelines, ETL, and streaming Compilers and programming languages Frontend and web DevOps, CI/CD, and build systems
You're a Great Fit If
You hold, or are completing, a PhD in computer science or software engineering or a closely related field, or you have 5+ years of professional experience in it. You've done technical or computational work in your field and are comfortable in a Linux terminal: shell commands, scripting in Python or Bash, and installing packages. You've used Docker, or can pick it up quickly. We share templates and worked examples. You can draw on your own research, projects, code, or files as the source material for your Tasks. You have a precise, adversarial eye; you enjoy finding the exact point where a workflow breaks and explaining it cleanly. You're honest about uncertainty and write instructions that leave no room for misreading. You're based in the US or Canada. Academics and practitioners are equally welcome. No AI or machine-learning background is needed. Your expertise and your comfort working in a terminal are what we're after.
Materials You Can Use
Almost any piece of your work can seed a Task, for example: Repositories, schemas, and failing test suites CI configs and build scripts Data pipelines and the datasets behind them Profiler traces, logs, and incident write-ups Pull requests and bug fixes from your own work Bugs, edge cases, and failure modes you've had to debug yourself
- ...conversation into a competitive advantage. Cresta’s unified AI platform combines conversational AI agents, real-time human agent augmentation, and... ...time owner. That’s you. As our first AI Agent Evaluation Engineer, you define what quality means for every agent we run...SuggestedFull timeWork at officeRemote workHome officeFlexible hours
- ...conversation into a competitive advantage. Cresta’s unified AI platform combines conversational AI agents, real-time human agent augmentation, and... ...full-time owner. That’s you. As our first AI Agent Evaluation Engineer, you define what quality means for every agent we run...SuggestedFull time
- ...Our vision is to build an ML/AI first network of advertisers and... ...hiring its first dedicated AI Engineer. Over the past year, AI at... ...to production: a service desk agent that closes tickets without a... ...demo and a production system: evaluation, monitoring, cost, and a clear...SuggestedFull timeRemote work
- ...conversation into a competitive advantage. Cresta’s unified AI platform combines conversational AI agents, real-time human agent augmentation, and comprehensive... ...after transfer We are looking for strong backend engineers who want to work at the intersection of distributed...SuggestedWork at officeLocal areaRemote workWork from homeHome office
- ...competitive advantage. Cresta’s unified AI platform combines conversational AI agents, real-time human agent augmentation... ...test execution frameworks that let engineers iterate quickly and ship with... ...What we’re looking for ~5+ years of software engineering experience, with...SuggestedWork at officeLocal areaRemote workWork from homeHome office
$225k - $285k per year
...builds the best-in-class Voice AI models powering the next... ...through our dashboard. Their coding agent does. Claude Code, Cursor,... ...others are now how a big share of engineers discover, integrate and debug... ...: ~2+ years of professional software engineering experience, with solid...Full time$90 per hour
...Pay: Up to $90+/hr Experienced software engineers review and score AI-written code for frontier AI research teams. Remote and flexible, up to 40 hours a week. Pay is up to $90+ per hour for the most experienced engineers. Requirements: several years of professional...Hourly payContract workFor contractorsLive InRemote workFlexible hours- ...services are deeply integrated and designed for growth. About the Role We’re looking for a Software Engineer to join our AI Tooling team and help shape how developers and agents build with Supabase. You’ll build and own our AI tooling, including agents, the MCP server,...Full timeRemote workFlexible hours
- ...all applications and next steps. Our partner is looking for a Software Engineer - AI Tooling based in Canada. As a Software Engineer on the AI... ...and tools. You will design, build, and maintain intelligent agents, Model Context Protocol (MCP) servers, agent skills, and AI-powered...Full time
- ...all applications and next steps. Our partner is looking for a Software Engineer, AI Integrations based in Canada. The Software Engineer, AI... ...maintain integrations connecting APIs and data services with coding agents, developer frameworks, cloud platforms, and automation tools....Long term contract
- ...applications and next steps. Our partner is looking for a Senior Software Engineer – AI based in Canada. The Senior Software Engineer will join a... ...AI Platform team responsible for building production-ready AI agents, intelligent products, and shared infrastructure across a...Long term contractFull time
- ...Fleetworthy is shaping the future of fleet readiness with AI-enabled, connected fleet technology that keeps... ....com. About the role: We’re hiring a Senior Software Engineer for the AI Platform team. The team builds AI agents and products across Fleetworthy’s fleet readiness...Long term contractFull time
$128.5k - $160.6k per year
...daily at Coalition. About the role We are hiring a Software Engineer for the Automation team to use AI to solve and automate manual work across Coalition.... ...hands-on AI engineering role: authoring skills, wiring agents to our systems of record through MCP, building evals,...Full timeFixed term contractLive InWork at officeRemote workHome officeFlexible hours- ...solution database and continuous calculation engine, driving efficiency, productivity and... ...the opportunity We are looking for a Senior Software Developer to join the team that is building search, AI Assistant, and AI agents capabilities into our Dayforce product. The...Full timeRemote work
- ...trust GitLab to ship better, more secure software faster. The same principles built... ...in how our team works: we embrace AI as a core productivity multiplier, with... ...role As an Intermediate Backend Engineer (Python) in GitLab's Agent Foundations stage, you'll go beyond using...Full timeInternshipRemote work
- ...processes evidence, there's a good chance our software is working behind the scenes. That's been true for over four decades. AI is central to where we're going. Not as a... .... The Role The AI Native Senior Software Engineer is a change agent and a driver of value within our public...Full timeWork from home
- ...About the role AI assistants are becoming part of everyday life, helping people with... ...and select your preferred experience. Evaluate conversational quality, helpfulness,... ...simple testing guidelines and document your evaluations accurately. Who we're looking for Active...Hourly payTemporary workPart timeRemote work
- ...competitive advantage. Cresta’s unified AI platform combines conversational AI agents, real-time human agent augmentation... ..., and reliable. As an AI Agent Engineer , you'll be at the forefront of... ...with customers as well as our software and machine learning engineers, ensuring...Full timeWork at officeLocal areaRemote workWork from homeHome office
- ...leading security-first enterprise AI company. We build cutting-edge... ...is a team of researchers, engineers, designers, and more, who are... ...platform that powers autonomous AI agents at scale. You’ll work on hard,... ...management - Prototype and evaluate new technologies, then help...Full timeManual laborLocal areaRemote work
$168k - $177.5k per year
...and shoppers to thrive. The Enterprise AI team develops intelligent systems and capabilities... ...the company. We are seeking a Senior Software Engineer to join the Enterprise AI team and help... ...AI team by identifying opportunities, evaluating new approaches, and sharing knowledge...Long term contractPermanent employmentWork at officeRemote workWork from homeFlexible hours- ...infrastructure, and developer workflows that make AI-powered engineering more useful, reliable, and easy for Vanta... ...the foundations for a more autonomous software development experience at Vanta, including background agents, agent evaluation systems, and tooling that helps...Full timeRemote workFlexible hours
- ...applications and next steps. Our partner is looking for a Full Stack Software Engineer – AI-First based remotely in Canada. In this role, you will help... ...manual coding, you will leverage advanced AI coding agents to accelerate development, guide implementation, and deliver...Remote work
$100 per hour
...Check out this Loom video for more details: We're looking for highly experienced software engineer (SR+) to help evaluate the quality of interactions with modern coding agents such as OpenAI Codex and Claude Code. This is not a traditional engineering role. You...Contract workRemote workWorldwideFlexible hours- ...Alpheva AI is looking for experienced software engineers to help train and evaluate AI coding systems through realistic repositories and engineering workflows. Projects capture the practical judgment behind reliable software, including understanding unfamiliar systems,...Contract work
$153k - $213k per year
...what matters most. Affirm's engineering team is building a large-scale... ...changes to systems. The AI Platform team owns Affirm's paved... ...path for building, running, evaluating, and governing AI systems. Our... ...interfaces to models, tools, and agents, with identity, cost...Work at officeRemote workFlexible hours- ...position is listed on behalf of a partner company, which manages all applications and next steps. Our partner is looking for an AI Voice Evaluation Specialist based in Canada. Join an innovative AI-focused environment as an AI Voice Evaluation Specialist, where you will...
- ...applications and next steps. Our partner is looking for a Senior Software Engineer – AI Oversight based in Canada. As a Senior Software Engineer... ...infrastructure that enables secure, privacy-preserving evaluations of advanced AI models. You will design and develop core systems...Long term contractFull timeRemote work
- ...all applications and next steps. Our partner is looking for a Software Engineer – AI Tooling Infrastructure based in Canada. We are looking for an... ...production services, including infrastructure supporting AI agents, developer assistants, and Model Context Protocol (MCP) integrations...Long term contractFull time
- ...About the role Samsara builds software that helps organizations run safer... ...who use it. The Operational AI team is turning Samsara's... ...their teams operate. As a senior engineer, you will build agentic solutions... ...to connect AI detections and agents with operational workflows. -...Long term contractFull time
$154.8k - $212.85k per year
...long term. About the role AI is changing how our R&D organization builds and runs software. This is a new role: one leader... ...and whatever comes next. Dev Agents builds in-house and integrates... ...the first-line work of an on-call engineer. It already runs on real incidents...Long term contractFull timeInternshipRemote workWork from homeFlexible hoursShift work
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Software Engineer - AI Agent Evaluation. Be the first to apply!
- software developer internship no experience Remote
- software engineer intern Remote
- software developer Remote
- software developer remote Remote
- software engineer - ai machine learning Remote
- part time software developer Remote
- junior software engineer Remote
- software engineer part time Remote
- software developer intern Remote
- senior software developer Remote
