Ai Quality Engineer
Rootly
About RootlyAt Rootly, we are on a mission to be the go-to way companies respond when things go wrong, helping every organization be more reliable. We do this by building an industry-leading incident management platform that allows companies around the world to consistently and quickly resolve incidents. We are not simply transforming an industry, we are carving out a new multi-billion-dollar category ourselves and need incredible talent to achieve this ambitious goal together.
Customers love Rootly. Some of the fastest growing companies around the world such as NVIDIA, Figma, Canva, Tripadvisor, Squarespace and more rely on Rootly to power their critical incident management process. They obsess over our delightful enterprise-ready platform and unique partnership model. See why our customers have reviewed us 5 stars on G2.
Investors love Rootly. We are backed by some of the most respected funds in the world from Y Combinator to operators like the CTO of Dropbox and GitHub. We'd be happy to disclose our entire funding and profitability picture live during the interview. As a culture we relentlessly put transparency first. We conduct monthly financial reviews as a team so everyone has a pulse on the health of the business and publish what we are building in our weekly changelog.
Rootly is building the AI-native future of incident management, and we need someone who can push our AI to its limits before our customers do. As our AI Quality Engineer, you'll own the evaluation and optimization of Rootly's agentic AI features -- designing test scenarios, running adversarial prompts, interpreting outputs, and working directly with engineering and product to close the loop on performance.
This isn't traditional QA. You'll spend your days thinking like an attacker, a confused user, and a power user all at once -- probing how our AI agents reason, make decisions, and handle edge cases across complex incident workflows.
What you'll do
Design and execute prompt-based test scenarios that cover happy paths, edge cases, and adversarial inputs across Rootly's agentic AI features
Evaluate AI outputs for accuracy, relevance, consistency, and alignment with expected workflow behaviour
Build and maintain an evaluation framework;
structured test libraries, scoring rubrics, and regression suites to track AI performance over time
Identify failure modes, hallucinations, reasoning gaps, and unexpected agent behaviours;
document findings and work with engineers to resolve them
Partner with Product and Engineering on new AI feature releases, contributing to acceptance criteria and quality gates before launch
Define and track quality metrics (accuracy rates, failure frequency, regression trends) and report findings to stakeholders
Stay current on LLM evaluation techniques, prompt engineering best practices, and agentic testing methodologies
What we're looking for
+5 years in QA, product operations, AI/ML evaluation, or a closely related role
Hands-on experience testing or evaluating LLM-powered or agentic AI products
Strong prompt engineering instincts -- you understand how wording, context, and structure affect model behaviour
Comfortable writing scripts or working with evaluation tools(Python a plus;
not required to be a full-stack engineer)
Sharp analytical thinking;
you can spot a subtle reasoning failure and articulate exactly why it's a problem
Clear written communicator;
able to translate AI behaviour findings for both technical and non-technical audiences
Nice to have
Familiarity with incident management, DevOps, or IT operations workflows is a strong asset
Experience with evaluation frameworks (e.G. LangSmith, PromptFlow, Braintrust, or similar)
Exposure to red‑teaming or adversarial testing of AI systems
Comfortable writing E2E tests with Playwright
Background working at a B2B SaaS or developer‑tools company
Familiar with mobile app testing (iOS/Android)
Why Rootly?We're not just another startup. We're building something category-defining and want teammates who crave ownership, love solving hard problems, and thrive in a high‑bar, high-impact environment.
Here's what you can expect when you join Rootly:
Competitive compensation and early equity in a fast‑growing, venture‑backed company.
Comprehensive medical, dental, and vision coverage.
3 weeks of vacation, plus unlimited sick and mental health days, and a company‑wide end‑of‑year shutdown to recharge.
$500 stipend for home office setup.
Unlimited token usage and access to AI tools
A fast‑moving, high‑impact environment where your leadership and ideas directly shape the future of the company.
This is a newly created vacancy.
As we build out the team, we may use AI to support parts of our interview process, like reviewing assessments, but it's a tool that helps our people, not a substitute for their judgement.
Accommodations are available on request at any stage of our recruitment process. Contact View email address on ca.hrm.tc and we will work with you to meet your needs confidentially.
Rootly is an equal opportunity employer. We do not discriminate on the basis of race, ancestry, place of origin, colour, ethnic origin, citizenship, creed, sex, sexual orientation, gender identity, gender expression, age, record of offences, marital status, family status, disability, or any other ground protected by the Ontario Human Rights Code or applicable law.
#J-18808-Ljbffr
Customers love Rootly. Some of the fastest growing companies around the world such as NVIDIA, Figma, Canva, Tripadvisor, Squarespace and more rely on Rootly to power their critical incident management process. They obsess over our delightful enterprise-ready platform and unique partnership model. See why our customers have reviewed us 5 stars on G2.
Investors love Rootly. We are backed by some of the most respected funds in the world from Y Combinator to operators like the CTO of Dropbox and GitHub. We'd be happy to disclose our entire funding and profitability picture live during the interview. As a culture we relentlessly put transparency first. We conduct monthly financial reviews as a team so everyone has a pulse on the health of the business and publish what we are building in our weekly changelog.
Rootly is building the AI-native future of incident management, and we need someone who can push our AI to its limits before our customers do. As our AI Quality Engineer, you'll own the evaluation and optimization of Rootly's agentic AI features -- designing test scenarios, running adversarial prompts, interpreting outputs, and working directly with engineering and product to close the loop on performance.
This isn't traditional QA. You'll spend your days thinking like an attacker, a confused user, and a power user all at once -- probing how our AI agents reason, make decisions, and handle edge cases across complex incident workflows.
What you'll do
Design and execute prompt-based test scenarios that cover happy paths, edge cases, and adversarial inputs across Rootly's agentic AI features
Evaluate AI outputs for accuracy, relevance, consistency, and alignment with expected workflow behaviour
Build and maintain an evaluation framework;
structured test libraries, scoring rubrics, and regression suites to track AI performance over time
Identify failure modes, hallucinations, reasoning gaps, and unexpected agent behaviours;
document findings and work with engineers to resolve them
Partner with Product and Engineering on new AI feature releases, contributing to acceptance criteria and quality gates before launch
Define and track quality metrics (accuracy rates, failure frequency, regression trends) and report findings to stakeholders
Stay current on LLM evaluation techniques, prompt engineering best practices, and agentic testing methodologies
What we're looking for
+5 years in QA, product operations, AI/ML evaluation, or a closely related role
Hands-on experience testing or evaluating LLM-powered or agentic AI products
Strong prompt engineering instincts -- you understand how wording, context, and structure affect model behaviour
Comfortable writing scripts or working with evaluation tools(Python a plus;
not required to be a full-stack engineer)
Sharp analytical thinking;
you can spot a subtle reasoning failure and articulate exactly why it's a problem
Clear written communicator;
able to translate AI behaviour findings for both technical and non-technical audiences
Nice to have
Familiarity with incident management, DevOps, or IT operations workflows is a strong asset
Experience with evaluation frameworks (e.G. LangSmith, PromptFlow, Braintrust, or similar)
Exposure to red‑teaming or adversarial testing of AI systems
Comfortable writing E2E tests with Playwright
Background working at a B2B SaaS or developer‑tools company
Familiar with mobile app testing (iOS/Android)
Why Rootly?We're not just another startup. We're building something category-defining and want teammates who crave ownership, love solving hard problems, and thrive in a high‑bar, high-impact environment.
Here's what you can expect when you join Rootly:
Competitive compensation and early equity in a fast‑growing, venture‑backed company.
Comprehensive medical, dental, and vision coverage.
3 weeks of vacation, plus unlimited sick and mental health days, and a company‑wide end‑of‑year shutdown to recharge.
$500 stipend for home office setup.
Unlimited token usage and access to AI tools
A fast‑moving, high‑impact environment where your leadership and ideas directly shape the future of the company.
This is a newly created vacancy.
As we build out the team, we may use AI to support parts of our interview process, like reviewing assessments, but it's a tool that helps our people, not a substitute for their judgement.
Accommodations are available on request at any stage of our recruitment process. Contact View email address on ca.hrm.tc and we will work with you to meet your needs confidentially.
Rootly is an equal opportunity employer. We do not discriminate on the basis of race, ancestry, place of origin, colour, ethnic origin, citizenship, creed, sex, sexual orientation, gender identity, gender expression, age, record of offences, marital status, family status, disability, or any other ground protected by the Ontario Human Rights Code or applicable law.
#J-18808-Ljbffr
Vacancy posted 1 hour ago
Similar jobs that could be interesting for youBased on the Ai Quality Engineer in Winnipeg, MB vacancy
- ...Build and maintain automated tests for AI agent workflows, APIs, tools, telemetry backed... ..., logs, and resource impact. Debug quality issues directly by reading logs, tracing requests... ...small code/test changes without heavy engineering hand-holding. Partner with engineering...SuggestedLocal area
- ...We are seeking a highly experienced and hands-on Supplier Quality Engineer (SQE) with strong expertise in casting and machining processes, particularly aluminum castings. The role requires close collaboration with suppliers, frequent onsite visits, and active involvement in...Suggested
- ...and grow their careers in an equitable environment. Learn about SKF at. Position Overview We are seeking a skilled Quality Electrical Engineer to support customer and manufacturing quality for electrical and electronic products. The position is responsible for...SuggestedPermanent employment
- Take on a pivotal role as an AI Quality Assurance and Testing Engineer, focusing on the assessment of machine learning and generative AI models. Ensure high standards of quality, bias assessment, and responsible AI practices across applications. In this short-term contract...SuggestedTemporary work
- ...We are seeking a Senior Software Quality Engineer – AI Focused to drive quality, reliability, and automation for AI-enabled software products. The role will collaborate with Development, DevOps, Product, and QE teams to design robust testing strategies and scalable automation...SuggestedFull time
- Evaluate generative AI systems as a Quality Engineer in Toronto, using Python and test automation within a hybrid, full-time role.Epsilon Solutions LTD seeks an AI evaluator and quality engineer for generative-AI solutions. The position defines evaluation approaches for models...Full time
- Lead the AI Quality Engineering initiatives at Replicant, enhancing our focus on top-notch customer service solutions. This managerial role involves redefining QA standards using innovative strategies. At Replicant, you will head the quality engineering efforts by mentoring...
- Build trustworthy AI personalization quality at Autodesk in Toronto using TypeScript, Python, and automated testing expertise. Support reliable Autodesk... ..., and AI behavior evaluation. The role requires software engineering, quality engineering, or test automation experience, with a...
- ...backgrounds and experiences who are passionate about quality, teamwork, and making an impact.The Opportunity We’... ...Automation Specialists who are passionate about modern engineering practices and excited about how AI is changing the way software is tested and delivered ....Internship
- ...StackAdapt is hiring a Staff Quality Engineer to lead the quality strategy for AI/agentic systems. You will design evaluation frameworks for non-deterministic AI, build scalable automated test suites, and set quality benchmarks across CI/CD pipelines. You’ll mentor QE teams...Remote work
- Apple’s Creativity Apps team seeks a Machine Learning Engineer focused on evaluating ML models and AI features in our creativity tools. You will define evaluation... ..., and develop scalable infrastructure to ensure quality from model development to customer delivery. You will...
- ...Rootly is hiring an AI Quality Engineer to own the evaluation and optimization of its agentic AI features. You will design test scenarios, probe how AI agents reason, and work with engineering to close the loop on performance across incident workflows. You will partner...
- Become a key part of our team as an AI Quality Engineer, focusing on automated test development and quality assurance. This role combines Python expertise with a deep understanding of AI processes. With a minimum of 4 years in QA or test automation, you'll build and maintain...
$85k - $225k per year
...Toronto is seeking an experienced professional to validate and ensure the reliability of Veeva AI Agents. The ideal candidate should have expertise in data quality, prompt engineering, and proficiency in Python. This full-time role offers a competitive salary range of $85,000...Full timeRemote workFlexible hours- ...service, innovation and a commitment to people. Our dependable quality, reliable delivery, and knowledgeable follow-up service... ...free Tea and Coffee. Position Summary Join our AI team as an Intermediate AI Engineer, where you'll build and deploy AI applications, agents, and...
- Mosaic Clinical Technologies (Canada), Inc.™ is seeking a Head of Quality Engineering to own the software quality and test automation strategy across the portfolio. You will lead architecture and maintenance of scalable automation frameworks, embedding gates into CI/CD and...
- ...TCS is aiming to become the world's largest AI-led technology services company and is... ...chatbot conversation analysis and recommend quality improvements. Develop and maintain test... ...possible. Work closely with developers, AI engineers, and content teams to resolve defects....Long term contract
- ...and aspirations with us, and we'll ensure that your efforts open doors to new opportunities and growth. THE POSITION The Quality Engineer leads the development, implementation, and continuous improvement of quality standards and inspection criteria to ensure product...Full timeRemote work
- Myant is hiring a Senior Software Quality Engineer to lead testing across health software platforms, emphasizing patient safety and privacy. You... ...quality strategy, stay hands-on with automation, and leverage AI to accelerate testing without compromising standards. In this role...
- ...Vancity is seeking an Engineering Lead, Quality to steer quality engineering across member‑facing platforms. You will lead hands‑on engineering, build scalable automation, and drive API‑first testing with AI‑driven approaches. This full‑time, permanent role is based at the...Permanent employmentFull timeWork at office
- Quality Engineer (QE) / SDET – AI Chatbot ValidationQuality Engineer (QE) / SDET – AI Chatbot Validation Job Description: We are looking for a seasoned AI first Senior Quality Engineer who is empowered to help the DevOps enablement team deliver highest level quality and...
- UnitedHealth Group in Canada is seeking a Principal Quality Engineer to lead integration and end-to-end quality for Enterprise Imaging across on-premise and cloud deployments. You will set the strategy, establish automation standards, and mentor engineers to ensure evidence-...Remote work
$100k - $140k per year
...an experienced Material Review Board (MRB) Engineer to join our dynamic team. The MRB Engineer... ...with engineering, manufacturing, and quality teams to ensure compliance with design, regulatory... ...add insights directly into each article, started with the help of AI. #J-18808-LjbffrFull timeRelocation$74k - $99k per year
...comprehensive range of body, chassis and engineering solutions to global customers. Through our... ...products to market.Job Responsibilities:The Quality Engineer will provide technical support,... ...requirements and your local Data Privacy Law.AI-Assisted Screening DisclosureAs part of...Temporary workPart timeLocal area- ...Drive AI innovation as a Senior AI Engineer in Mississauga, Canada. Craft Python-based AI applications using FastAPI or Flask while optimizing performance and scalability for enterprise solutions. We are seeking an experienced Senior AI Engineer with over 8 years in software...
- ...full-time professionals, and our annual growth rate exceeds 25%. Job Description Miratech is looking for a skilled Senior AI Engineer with Python, LangGraph and LangChain to join our US healthcare client. You will play a key role in developing and enhancing an...Full timeInternshipRemote workRelocation
$107.63k - $157.85k per year
...Joignez-vous à nous et contribuez à ce qui fait avancer le monde.The Quality Engineer will serve as both a technical design authority and a customer-... ...in MS Office, MS Teams, and ability to leverage AI tools in job functions.Ability to manage multiple projects and...$115.5k per year
...precision. With 465 billion automated optimizations per second, the AI-powered StackAdapt Marketing Platform seamlessly connects brand... ...and marketing channels.StackAdapt is seeking a talented Quality Engineer who is passionate about software quality and test automation...Local areaRemote workHome office$90k - $110k per year
Arrow Machine and Fabrication Group is an engineering-focused solution provider of machined castings... ...focuses on providing innovative design, quality craftsmanship, and exceptional customer... ...Act.Candidates please note, we do not use AI for application screening.Disclaimer: Only...Work at office$86.1k - $136.1k per year
We are seeking a curious and collaborative Quality Engineer who is motivated by emerging technologies and creative problem-solving. You will contribute... ...and analytics technologies, such as artificial intelligence (AI), and automated processing tools, to analyze and process the...Long term contractTemporary workLocal areaFlexible hours
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Ai Quality Engineer. Be the first to apply!
Related searches
- quality engineer Winnipeg, MB
- ingénieur qualité Winnipeg, MB
- technicien qualité Winnipeg, MB
- quality engineer manufacturing Winnipeg, MB
- quality engineer- qms Winnipeg, MB
- entry level quality engineer Winnipeg, MB
- directeur amelioration continue audit qualite Winnipeg, MB
- stage qualité Winnipeg, MB
- qualité agroalimentaire Winnipeg, MB
- qualité santé Winnipeg, MB

