Ai Quality Engineer
Rootly
About RootlyAt Rootly, we are on a mission to be the go-to way companies respond when things go wrong, helping every organization be more reliable. We do this by building an industry-leading incident management platform that allows companies around the world to consistently and quickly resolve incidents. We are not simply transforming an industry, we are carving out a new multi-billion-dollar category ourselves and need incredible talent to achieve this ambitious goal together.
Customers love Rootly. Some of the fastest growing companies around the world such as NVIDIA, Figma, Canva, Tripadvisor, Squarespace and more rely on Rootly to power their critical incident management process. They obsess over our delightful enterprise-ready platform and unique partnership model. See why our customers have reviewed us 5 stars on G2.
Investors love Rootly. We are backed by some of the most respected funds in the world from Y Combinator to operators like the CTO of Dropbox and GitHub. We'd be happy to disclose our entire funding and profitability picture live during the interview. As a culture we relentlessly put transparency first. We conduct monthly financial reviews as a team so everyone has a pulse on the health of the business and publish what we are building in our weekly changelog.
Rootly is building the AI-native future of incident management, and we need someone who can push our AI to its limits before our customers do. As our AI Quality Engineer, you'll own the evaluation and optimization of Rootly's agentic AI features -- designing test scenarios, running adversarial prompts, interpreting outputs, and working directly with engineering and product to close the loop on performance.
This isn't traditional QA. You'll spend your days thinking like an attacker, a confused user, and a power user all at once -- probing how our AI agents reason, make decisions, and handle edge cases across complex incident workflows.
What you'll do
Design and execute prompt-based test scenarios that cover happy paths, edge cases, and adversarial inputs across Rootly's agentic AI features
Evaluate AI outputs for accuracy, relevance, consistency, and alignment with expected workflow behaviour
Build and maintain an evaluation framework;
structured test libraries, scoring rubrics, and regression suites to track AI performance over time
Identify failure modes, hallucinations, reasoning gaps, and unexpected agent behaviours;
document findings and work with engineers to resolve them
Partner with Product and Engineering on new AI feature releases, contributing to acceptance criteria and quality gates before launch
Define and track quality metrics (accuracy rates, failure frequency, regression trends) and report findings to stakeholders
Stay current on LLM evaluation techniques, prompt engineering best practices, and agentic testing methodologies
What we're looking for
+5 years in QA, product operations, AI/ML evaluation, or a closely related role
Hands-on experience testing or evaluating LLM-powered or agentic AI products
Strong prompt engineering instincts -- you understand how wording, context, and structure affect model behaviour
Comfortable writing scripts or working with evaluation tools(Python a plus;
not required to be a full-stack engineer)
Sharp analytical thinking;
you can spot a subtle reasoning failure and articulate exactly why it's a problem
Clear written communicator;
able to translate AI behaviour findings for both technical and non-technical audiences
Nice to have
Familiarity with incident management, DevOps, or IT operations workflows is a strong asset
Experience with evaluation frameworks (e.G. LangSmith, PromptFlow, Braintrust, or similar)
Exposure to red‑teaming or adversarial testing of AI systems
Comfortable writing E2E tests with Playwright
Background working at a B2B SaaS or developer‑tools company
Familiar with mobile app testing (iOS/Android)
Why Rootly?We're not just another startup. We're building something category-defining and want teammates who crave ownership, love solving hard problems, and thrive in a high‑bar, high-impact environment.
Here's what you can expect when you join Rootly:
Competitive compensation and early equity in a fast‑growing, venture‑backed company.
Comprehensive medical, dental, and vision coverage.
3 weeks of vacation, plus unlimited sick and mental health days, and a company‑wide end‑of‑year shutdown to recharge.
$500 stipend for home office setup.
Unlimited token usage and access to AI tools
A fast‑moving, high‑impact environment where your leadership and ideas directly shape the future of the company.
This is a newly created vacancy.
As we build out the team, we may use AI to support parts of our interview process, like reviewing assessments, but it's a tool that helps our people, not a substitute for their judgement.
Accommodations are available on request at any stage of our recruitment process. Contact View email address on ca.hrm.tc and we will work with you to meet your needs confidentially.
Rootly is an equal opportunity employer. We do not discriminate on the basis of race, ancestry, place of origin, colour, ethnic origin, citizenship, creed, sex, sexual orientation, gender identity, gender expression, age, record of offences, marital status, family status, disability, or any other ground protected by the Ontario Human Rights Code or applicable law.
#J-18808-Ljbffr
Customers love Rootly. Some of the fastest growing companies around the world such as NVIDIA, Figma, Canva, Tripadvisor, Squarespace and more rely on Rootly to power their critical incident management process. They obsess over our delightful enterprise-ready platform and unique partnership model. See why our customers have reviewed us 5 stars on G2.
Investors love Rootly. We are backed by some of the most respected funds in the world from Y Combinator to operators like the CTO of Dropbox and GitHub. We'd be happy to disclose our entire funding and profitability picture live during the interview. As a culture we relentlessly put transparency first. We conduct monthly financial reviews as a team so everyone has a pulse on the health of the business and publish what we are building in our weekly changelog.
Rootly is building the AI-native future of incident management, and we need someone who can push our AI to its limits before our customers do. As our AI Quality Engineer, you'll own the evaluation and optimization of Rootly's agentic AI features -- designing test scenarios, running adversarial prompts, interpreting outputs, and working directly with engineering and product to close the loop on performance.
This isn't traditional QA. You'll spend your days thinking like an attacker, a confused user, and a power user all at once -- probing how our AI agents reason, make decisions, and handle edge cases across complex incident workflows.
What you'll do
Design and execute prompt-based test scenarios that cover happy paths, edge cases, and adversarial inputs across Rootly's agentic AI features
Evaluate AI outputs for accuracy, relevance, consistency, and alignment with expected workflow behaviour
Build and maintain an evaluation framework;
structured test libraries, scoring rubrics, and regression suites to track AI performance over time
Identify failure modes, hallucinations, reasoning gaps, and unexpected agent behaviours;
document findings and work with engineers to resolve them
Partner with Product and Engineering on new AI feature releases, contributing to acceptance criteria and quality gates before launch
Define and track quality metrics (accuracy rates, failure frequency, regression trends) and report findings to stakeholders
Stay current on LLM evaluation techniques, prompt engineering best practices, and agentic testing methodologies
What we're looking for
+5 years in QA, product operations, AI/ML evaluation, or a closely related role
Hands-on experience testing or evaluating LLM-powered or agentic AI products
Strong prompt engineering instincts -- you understand how wording, context, and structure affect model behaviour
Comfortable writing scripts or working with evaluation tools(Python a plus;
not required to be a full-stack engineer)
Sharp analytical thinking;
you can spot a subtle reasoning failure and articulate exactly why it's a problem
Clear written communicator;
able to translate AI behaviour findings for both technical and non-technical audiences
Nice to have
Familiarity with incident management, DevOps, or IT operations workflows is a strong asset
Experience with evaluation frameworks (e.G. LangSmith, PromptFlow, Braintrust, or similar)
Exposure to red‑teaming or adversarial testing of AI systems
Comfortable writing E2E tests with Playwright
Background working at a B2B SaaS or developer‑tools company
Familiar with mobile app testing (iOS/Android)
Why Rootly?We're not just another startup. We're building something category-defining and want teammates who crave ownership, love solving hard problems, and thrive in a high‑bar, high-impact environment.
Here's what you can expect when you join Rootly:
Competitive compensation and early equity in a fast‑growing, venture‑backed company.
Comprehensive medical, dental, and vision coverage.
3 weeks of vacation, plus unlimited sick and mental health days, and a company‑wide end‑of‑year shutdown to recharge.
$500 stipend for home office setup.
Unlimited token usage and access to AI tools
A fast‑moving, high‑impact environment where your leadership and ideas directly shape the future of the company.
This is a newly created vacancy.
As we build out the team, we may use AI to support parts of our interview process, like reviewing assessments, but it's a tool that helps our people, not a substitute for their judgement.
Accommodations are available on request at any stage of our recruitment process. Contact View email address on ca.hrm.tc and we will work with you to meet your needs confidentially.
Rootly is an equal opportunity employer. We do not discriminate on the basis of race, ancestry, place of origin, colour, ethnic origin, citizenship, creed, sex, sexual orientation, gender identity, gender expression, age, record of offences, marital status, family status, disability, or any other ground protected by the Ontario Human Rights Code or applicable law.
#J-18808-Ljbffr
Vacancy posted 1 hour ago
Similar jobs that could be interesting for youBased on the Ai Quality Engineer in Winnipeg, MB vacancy
- ...and grow their careers in an equitable environment. Learn about SKF at. Position Overview We are seeking a skilled Quality Electrical Engineer to support customer and manufacturing quality for electrical and electronic products. The position is responsible for...SuggestedPermanent employment
- ...We are seeking a highly experienced and hands-on Supplier Quality Engineer (SQE) with strong expertise in casting and machining processes, particularly aluminum castings. The role requires close collaboration with suppliers, frequent onsite visits, and active involvement in...Suggested
- ...service, innovation and a commitment to people. Our dependable quality, reliable delivery, and knowledgeable follow-up service... ...free Tea and Coffee. Position Summary Join our AI team as an Intermediate AI Engineer, where you'll build and deploy AI applications, agents, and...Suggested
- ...and aspirations with us, and we'll ensure that your efforts open doors to new opportunities and growth. THE POSITION The Quality Engineer leads the development, implementation, and continuous improvement of quality standards and inspection criteria to ensure product...SuggestedFull timeRemote work
- ...full-time professionals, and our annual growth rate exceeds 25%. Job Description Miratech is looking for a skilled Senior AI Engineer with Python, LangGraph and LangChain to join our US healthcare client. You will play a key role in developing and enhancing an...SuggestedFull timeInternshipRemote workRelocation
- ...Drive AI innovation as a Senior AI Engineer in Mississauga, Canada. Craft Python-based AI applications using FastAPI or Flask while optimizing performance and scalability for enterprise solutions. We are seeking an experienced Senior AI Engineer with over 8 years in software...
- Quality Engineer Job Summary Reporting directly to the Quality Assurance Manager, Quality Engineer performs execution of Quality Engineering systems including; handling of nonconforming materials, Corrective/Preventive Actions, failure reporting, analysis and corrective action...Full time
- ...saving lives. Job Summary Reporting to the Sr. Director, Quality, the Director, Quality Assurance leads QA Operations, Quality... ...& Skills • Bachelor’s degree in Science, Pharmacy, Engineering, or a related field. An advanced degree is considered an asset....Flexible hours
- ...bowl, or grabbing some groceries, our tech platform connects tens of millions of customers with hundreds of thousands of restaurant, grocery and convenience partners across the globe.About this role As a Senior Quality Engineer, you will work with an elite team of develop...Full timeNight shift
- ...service, innovation and a commitment to people. Our dependable quality, reliable delivery, and knowledgeable follow-up service have made... ...problem-solving skills. • The ability to read and interpret engineering drawings. • High proficiency in Microsoft office including,...Permanent employmentFull timeWork at office
- ...Director of Quality Assurance Winnipeg, MB Full-Time, Onsite POSITION SUMMARY: The Director of Quality Assurance - Transit... ...from recurring. Partner closely with Manufacturing, Engineering, Supply Chain, Service, and Operations teams to solve complex challenges...Full timeInternshipImmediate start
- ...processes and practices across our organization to help ensure flight and product efficiency. As a Quality Inspector , you will ensure that all aircraft components, engines, and systems are maintained, repaired, or overhauled in full compliance with regulatory standards,...Long term contractTemporary workTraineeshipInternshipImmediate startFlexible hoursShift workWeekend work
- ...The Quality Control Inspector will provide Quality Process Audit support to the Assembly Line. This position will assist in reducing process variation through the use of statistical data analysis, process audits, and department metrics. The position emphasizes improving internal...Full timeTemporary work
- ...professionals where our culture is informed by our values and commitment to protecting and saving lives. Job Summary The Specialist I, Quality Compliance is responsible for supporting and representing Emergent during regulatory inspections, customer audits, and internal...Flexible hours
- ...service, innovation and a commitment to people. Our dependable quality, reliable delivery, and knowledgeable follow-up service have made... ...shipping processes. • Work collaboratively with Production, Engineering, Shipping, Logistics, Supply Chain, Quality, and other cross-...
- ...passionate professionals where our culture is informed by our values and commitment to protecting and saving lives. Job Summary The Quality Assurance Deviations and CAPA Specialist provides support by managing deviations and corrective/preventative actions. Additionally,...Flexible hours
- ...perform to your expectations. We are proud of our long-standing partnerships with family-owned Canadian chicken farmers to produce high quality products. Role: Responsible for monitoring employee GMP compliance and product quality specification compliance of the raw...Full timeAfternoon shift
- Education: College, CEGEP or other non-university certificate or diploma from a program of 1 year to 2 years. Security and safety: Criminal record check. Screening questions: Do you have experience working in this field?. Employment terms options: Evening. Experience: 1 year...Hourly payLong term contractPermanent employmentImmediate startDay shiftAfternoon shift
- ...Manager, Quality AssuranceApply: Remote, ON: Full time: Posted Yesterday: R2616835Our team members are at the heart of everything we do. At Cencora, we are united in our responsibility to create healthier futures, and every person here is essential to us being...Full timeRemote work
- Engineering Student - QualityBuild an Aviation Career You’re Proud OfAt StandardAero, we use our ingenuity and know-how to find solutions for the simple to the most complex challenges in Aviation. Together, we get the job done and done well. Bring your mechanical aptitude,...Part timeInternship
$95k - $105k per year
...Quality Assurance Manager Winnipeg, MB | Full-Time | Permanent | $95,000 – $105,000 per year About the Opportunity Staffmax is excited to partner with a well-established Winnipeg organization to find an experienced Quality Assurance Manager . This is a confidential...Permanent employmentFull timeCasual workWork at officeNight shift- ...As our project portfolio continues to grow, we are seeking a Quality Coordinator (Site) to lead project quality activities and ensure... ...backgrounds or equivalent: Power Line Technician (PLT) Electrical Engineering Technologist Construction Inspector Utility Inspector...Full timeContract work
- ..., single-source solutions provider delivering consistent, high-quality aluminum permanent mold castings to our customers. We are currently... ...to ensure product and process conformance. Support Engineering and Sales staff in New Job and Continuous Improvement activities...Permanent employmentFull timeFlexible hoursShift work
- ...sustainable, and reliable charging and mobility solutions. Further information is available at . POSITION SUMMARY: Ensure the New Flyer Quality system is fully understood, executed, and maintained across the Winnipeg facility. The Quality Manager provides leadership to the...Full timeInternshipImmediate startFlexible hoursShift work
- Take on a pivotal role as an AI Quality Assurance and Testing Engineer, focusing on the assessment of machine learning and generative AI models. Ensure high standards of quality, bias assessment, and responsible AI practices across applications. In this short-term contract...Temporary work
- Build trustworthy AI personalization quality at Autodesk in Toronto using TypeScript, Python, and automated testing expertise. Support reliable Autodesk... ..., and AI behavior evaluation. The role requires software engineering, quality engineering, or test automation experience, with a...
- Evaluate generative AI systems as a Quality Engineer in Toronto, using Python and test automation within a hybrid, full-time role.Epsilon Solutions LTD seeks an AI evaluator and quality engineer for generative-AI solutions. The position defines evaluation approaches for models...Full time
- ...insights, products, and services that make quality care more accessible and affordable. Here,... ...from you. Position Summary The People AI Solutions Designer is the builder. Where the... ...science, information systems, design, engineering, or a related field — or equivalent demonstrated...
- ...Grounded in safety, quality, and ethics, our experts lead their fields with dedication, a creative spirit, and a vision for growth. We draw from more than 20 technical specialties worldwide and are devoted to fostering a community of diverse talents, backgrounds, and expertise...Work at officeWorldwideShift work
- ...perform to your expectations. We are proud of our long-standing partnerships with family-owned Canadian chicken farmers to produce high quality products. Role: Responsible for monitoring employee GMP compliance and product quality specification compliance of the raw...Contract work
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Ai Quality Engineer. Be the first to apply!
Related searches
- quality engineer Winnipeg, MB
- ingénieur qualité Winnipeg, MB
- technicien qualité Winnipeg, MB
- quality engineer manufacturing Winnipeg, MB
- quality engineer- qms Winnipeg, MB
- entry level quality engineer Winnipeg, MB
- directeur amelioration continue audit qualite Winnipeg, MB
- stage qualité Winnipeg, MB
- qualité agroalimentaire Winnipeg, MB
- qualité santé Winnipeg, MB


