AI Quality Engineer: Prompt Testing & Evaluation
Rootly
This position requires local presence. Please view similar jobs below.
Rootly is hiring an AI Quality Engineer to own the evaluation and optimization of its agentic AI features. You will design test scenarios, probe how AI agents reason, and work with engineering to close the loop on performance across incident workflows.
You will partner with product and engineering for new AI releases, define quality metrics, and stay current on evaluation techniques. This role offers a fast-paced, high-impact environment with growth opportunities.
#J-18808-LjbffrVacancy posted 18 hours ago
Similar jobs that could be interesting for youBased on the AI Quality Engineer: Prompt Testing & Evaluation in Canada vacancy
usd170k - usd200k per year
...client is a seed-stage enterprise AI company building autonomous... ...agents that execute complex engineering workflows inside customer-controlled... ...the company's first dedicated quality-infrastructure hire. It is a... ...through agent-run test harnesses, LLM graders, regression...SuggestedPermanent employmentFull timeInternshipWork at officeLocal areaRemote workFlexible hoursusd5 000 per month
...We are looking for experienced Senior Software Engineers to support a technical AI evaluation project involving data pipelines, evaluation workflows, and preference-data quality. What You'll Do Design, refine, and maintain evaluation rubrics and criteria for preference...Suggested$134.4k - $184.8k per year
...automated optimizations per second, the AI-powered StackAdapt Marketing Platform seamlessly... ...StackAdapt is seeking a talented Staff Quality Engineer to join the Automation and Intelligence... ...set the technical direction for how we test, evaluate, and ship intelligent systems,...SuggestedFull timeInternshipLocal areaRemote workWork from homeHome office- ...extraordinary. THE ROLE You'll make sure our AI agents and critical workflows actually work by building smart automated testing and evaluation, then using AI to test AI. WHAT YOU... ...AI agents on task completion, response quality, tool use and permissions, and measure how...SuggestedFull timeLive InWorldwide
- ...team is a diverse group of experienced engineers, traders, and brokerage professionals who... ...re looking for a Senior Data Scientist, AI Evaluation to design how Alpaca measures whether our... ...contributor who turns ambiguous quality questions into ground truth, scoring methods...SuggestedFull timeHome office
- ...At Replicant, we believe AI should work for people, starting with customer service. We’re looking for a Software Development Engineer in Test (SDET) to help raise the bar for quality across our voice AI platform as we continue to scale. This person will play a key role in...Full timeRemote workFlexible hours
$84k - $115.5k per year
...465 billion automated optimizations per second, the AI-powered StackAdapt Marketing Platform seamlessly... ...marketing channels. StackAdapt is seeking a talented Quality Engineer who is passionate about software quality and test automation to be part of a highly impactful team...Full timeLocal areaRemote workWork from homeHome officeusd170k - usd270k per year
...Cadence is the clinical AI company automating the treatment of... ...Role We're hiring an AI Engineer to design and ship production... ...retrieval, reasoning, tool use, evaluation, and safety guardrails. If you... ...frameworks: offline benchmarks, safety tests, regression suites, and LLM-as...Full timeLocal areaRemote workHome officeusd160k - usd200k per year
...Ravenna, we are looking for experienced engineers with a passion for AI and a strong track record of building... ..., and iterating quickly based on evaluation and real user feedback. We care deeply... ...real product workflows. This includes prompt pipelines, tool integration,...Full timeInternshipFlexible hours- ...TEST ENGINEER, FEATURES ABOUT ENT Ent is the intent-aware workspace security platform for securing human and AI-driven work. Built to protect productivity, the new attack surface, Ent understands... ...ROLE This role owns end-to-end quality for new features as they ship:...Full timeContract workInternshipWork at officeRemote workHome officeFlexible hours
- ...CSCI Consulting is looking for a Cyber Test Engineer to support Information System Security... ...regression testing Integrate SonarQube quality gates and fortify static application... ...requirements for the System Testing and Evaluation Specialist work role Possession of...Full timeLocal area
- ...to a remote first culture. Senior Engineer - Penetration Testing Department: Services Location... ...'s supply chain. Leverage AI-powered security tooling and LLM-assisted... ...relevant to embedded security research. Evaluate customer products for compliance with...Full timeRemote workRelocation
usd80k - usd105k per year
.... The Role Hone is looking for a Quality Engineer II to join our Engineering team. This is... ...role for someone who has a year or two of testing under their belt and is ready to own the... ...both test and automate, and you will use AI to do both faster. You will handle small...Full timeTemporary workPart timeFor contractorsRemote workFlexible hoursusd120k - usd130k per year
...Circle is building the world’s leading AI-powered, all-in-one platform for digital... ...ABOUT THE ROLE We’re looking for a Senior Quality Engineer to help shape how quality is built into... ...combine strong product thinking, exploratory testing, automation and technical investigation...Full timeRemote work$169k - $200k per year
...to help over two million people receive quality mental healthcare? Come join our mission... ...We are looking for an experienced Senior AI Engineer to join our team. The Senior AI Engineer... ...simulation environments for pre-training and testing agent policies, ensuring system stability...Long term contractFull timeLive InLocal areaRemote workWorldwideFlexible hours$275k - $300k per year
...Description Cobalt runs AI agents in production that test real customer applications. We’re looking for a Staff AI Engineer who has built and operated production agent systems... ...the technical direction for how we design, evaluate, ship and operate agents, and stay hands-...Full timeInternshipLocal areaWork from homeFlexible hours$121k - $214k per year
...high-growth company delivering AI solutions that address some of the... ...physics, mathematics, medicine, engineering, and other specialties. The... ...Security: Discovery of agents, evaluation, red teaming, monitoring, guardrails, etc. - Quality: Measuring ROI of agents and suggesting...Permanent employmentFull timeSeasonal workFlexible hours$184k - $205k per year
...ENGINEERING MANAGER -AI About Us Procurify is the Intelligent Spend Management company. We're on a mission to give... ...- Agentic workflows and orchestration - Prompt and context engineering frameworks - AI evaluation systems - AI observability and monitoring -...Long term contractFull timeImmediate startRemote workFlexible hoursusd136k - usd185.5k per year
.... We are not hiring you to test our software. We are hiring you... ...our scale that tension is an engineering problem rather than a process... ...the Staff Software Engineer, Quality Engineering, you will own and... ...GridUnity engineering runs on an AI-native, artifact-first model....Long term contractPermanent employmentFull timeContract workTemporary workInternshipLive InRemote workFlexible hoursNight shiftusd80k - usd90k per year
...Nasdaq: INOD) is a global data engineering company. We believe that data and Artificial Intelligence (AI) are inextricably linked. Our... ...intelligence by providing the data, evaluation frameworks, and human... ...legacy delivering the highest quality data and outstanding outcomes...Full timeFixed term contract$147k - $167k per year
...Safe Software is looking for a Staff AI Engineer to design, build, and deploy production-... ...preparation and model development through evaluation, deployment, monitoring, and continuous... ...governed tools. - Establish automated testing, evaluation, and observability practices...Full timeSeasonal workRemote workFlexible hours- ...are and what we do. Curotec is hiring AI Platform Engineers to support a large enterprise client in... ...configured, appropriately scoped, thoroughly tested, and well-documented. What You Will Do:... ...controls. Conduct security and quality reviews of third-party and externally authored...Full timeInternshipWorldwide
- ...portfolios to net-zero with an AI-driven sustainability, asset,... ...ROLE We're looking for an AI Engineer to build AI-powered features,... ...and improve model/feature quality - Build and maintain data +... ...- Experiment with new models, prompting, and retrieval approaches to improve...Full timeFlexible hours
- ...For more information, visit . Roadie is seeking a Software Engineer in Test to join our talented team of developers. In this role, you’ll... ...requirements to ensure alignment with established design and quality guidelines Partner with software engineers and fellow...Full timeInternshipLocal areaRemote workWork from homeFlexible hours
- ...science protocol that helps launch and grow AI-driven biotech research. It enables... .... You’ll work closely with full-stack engineers and scientist-evaluators to create agents that can plan, use... ...robust schemas and safeguards. - Develop quality and evaluation systems, including unit...Full time
- ...AI ENGINEER ABOUT ENT Ent is the intent-aware workspace security platform for securing... ...grade, with reproducible training, gated evaluation and efficient inference on the edge and... ...- Build production monitoring for model quality, including drift detection, retrain triggers...Full timeInternshipWork at officeRemote workHome officeFlexible hoursShift work
$143k - $252k per year
...high-growth company delivering AI solutions that address some of the... ...physics, mathematics, medicine, engineering, and other specialties. The... ...Security: Discovery of agents, evaluation, red teaming, monitoring, guardrails, etc. - Quality: Measuring ROI of agents and suggesting...Permanent employmentFull timeSeasonal workRemote workFlexible hours- ...motivated and technically talented Software AI Engineer to join our team. In this role, you will... ...Responsibilities Design, develop, test, and maintain software applications that... ...of model training, testing, evaluation, and basic machine learning workflows....Remote jobInternship
- ...portable device for tech enthusiasts and engineers. It’s a gadget with an awesome community that... ...We're looking for a Head of Production Test Engineering to lead the team responsible for... ...Chinese factories - Confident use of AI tools for rapid prototyping What we offer...Full timeWork at officeRemote workFlexible hours
- ...done. A key part of that vision is Reflow AI, agentic, automation-focused... ...workflow intelligence. - Experiment with prompt engineering, context tuning, and model orchestration... ...Comfortable building fast prototypes and testing model behavior in varied, complex contexts...Full time
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Quality Engineer: Prompt Testing & Evaluation. Be the first to apply!
Related searches
- ingénieur qualité Canada
- stage ingenieur qualite production Canada
- quality engineer manufacturing Canada
- quality engineer- qms Canada
- quality engineer Canada
- ingénieur qualité agroalimentaire Canada
- entry level quality engineer Canada
- technicien qualité Canada
- agent d'assurance de la qualité Canada
- quality inspection Canada