Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Ai Quality Engineer

Rootly

About RootlyAt Rootly, we are on a mission to be the go-to way companies respond when things go wrong, helping every organization be more reliable. We do this by building an industry-leading incident management platform that allows companies around the world to consistently and quickly resolve incidents. We are not simply transforming an industry, we are carving out a new multi-billion-dollar category ourselves and need incredible talent to achieve this ambitious goal together.
Customers love Rootly. Some of the fastest growing companies around the world such as NVIDIA, Figma, Canva, Tripadvisor, Squarespace and more rely on Rootly to power their critical incident management process. They obsess over our delightful enterprise-ready platform and unique partnership model. See why our customers have reviewed us 5 stars on G2.
Investors love Rootly. We are backed by some of the most respected funds in the world from Y Combinator to operators like the CTO of Dropbox and GitHub. We'd be happy to disclose our entire funding and profitability picture live during the interview. As a culture we relentlessly put transparency first. We conduct monthly financial reviews as a team so everyone has a pulse on the health of the business and publish what we are building in our weekly changelog.
Rootly is building the AI-native future of incident management, and we need someone who can push our AI to its limits before our customers do. As our AI Quality Engineer, you'll own the evaluation and optimization of Rootly's agentic AI features -- designing test scenarios, running adversarial prompts, interpreting outputs, and working directly with engineering and product to close the loop on performance.
This isn't traditional QA. You'll spend your days thinking like an attacker, a confused user, and a power user all at once -- probing how our AI agents reason, make decisions, and handle edge cases across complex incident workflows.
What you'll do
Design and execute prompt-based test scenarios that cover happy paths, edge cases, and adversarial inputs across Rootly's agentic AI features
Evaluate AI outputs for accuracy, relevance, consistency, and alignment with expected workflow behaviour
Build and maintain an evaluation framework;
structured test libraries, scoring rubrics, and regression suites to track AI performance over time
Identify failure modes, hallucinations, reasoning gaps, and unexpected agent behaviours;
document findings and work with engineers to resolve them
Partner with Product and Engineering on new AI feature releases, contributing to acceptance criteria and quality gates before launch
Define and track quality metrics (accuracy rates, failure frequency, regression trends) and report findings to stakeholders
Stay current on LLM evaluation techniques, prompt engineering best practices, and agentic testing methodologies
What we're looking for
+5 years in QA, product operations, AI/ML evaluation, or a closely related role
Hands-on experience testing or evaluating LLM-powered or agentic AI products
Strong prompt engineering instincts -- you understand how wording, context, and structure affect model behaviour
Comfortable writing scripts or working with evaluation tools(Python a plus;
not required to be a full-stack engineer)
Sharp analytical thinking;
you can spot a subtle reasoning failure and articulate exactly why it's a problem
Clear written communicator;
able to translate AI behaviour findings for both technical and non-technical audiences
Nice to have
Familiarity with incident management, DevOps, or IT operations workflows is a strong asset
Experience with evaluation frameworks (e.G. LangSmith, PromptFlow, Braintrust, or similar)
Exposure to red‑teaming or adversarial testing of AI systems
Comfortable writing E2E tests with Playwright
Background working at a B2B SaaS or developer‑tools company
Familiar with mobile app testing (iOS/Android)
Why Rootly?We're not just another startup. We're building something category-defining and want teammates who crave ownership, love solving hard problems, and thrive in a high‑bar, high-impact environment.
Here's what you can expect when you join Rootly:
Competitive compensation and early equity in a fast‑growing, venture‑backed company.
Comprehensive medical, dental, and vision coverage.
3 weeks of vacation, plus unlimited sick and mental health days, and a company‑wide end‑of‑year shutdown to recharge.
$500 stipend for home office setup.
Unlimited token usage and access to AI tools
A fast‑moving, high‑impact environment where your leadership and ideas directly shape the future of the company.
This is a newly created vacancy.
As we build out the team, we may use AI to support parts of our interview process, like reviewing assessments, but it's a tool that helps our people, not a substitute for their judgement.
Accommodations are available on request at any stage of our recruitment process. Contact View email address on ca.hrm.tc and we will work with you to meet your needs confidentially.
Rootly is an equal opportunity employer. We do not discriminate on the basis of race, ancestry, place of origin, colour, ethnic origin, citizenship, creed, sex, sexual orientation, gender identity, gender expression, age, record of offences, marital status, family status, disability, or any other ground protected by the Ontario Human Rights Code or applicable law.
#J-18808-Ljbffr
Vacancy posted 1 hour ago
Similar jobs that could be interesting for youBased on the Ai Quality Engineer in Winnipeg, MB vacancy
  •  ...and grow their careers in an equitable environment. Learn about SKF at. Position Overview We are seeking a skilled Quality Electrical Engineer to support customer and manufacturing quality for electrical and electronic products. The position is responsible for... 
    Suggested
    Permanent employment

    SKF Group

    Winnipeg, MB
    6 days ago
  •  ...We are seeking a highly experienced and hands-on Supplier Quality Engineer (SQE) with strong expertise in casting and machining processes, particularly aluminum castings. The role requires close collaboration with suppliers, frequent onsite visits, and active involvement in... 
    Suggested

    Global Connect Technologies

    Winnipeg, MB
    6 days ago
  •  ...service, innovation and a commitment to people. Our dependable quality, reliable delivery, and knowledgeable follow-up service...  ...free Tea and Coffee. Position Summary Join our AI team as an Intermediate AI Engineer, where you'll build and deploy AI applications, agents, and... 
    Suggested

    Price Industries Limited

    Winnipeg, MB
    6 days ago
  •  ...and aspirations with us, and we'll ensure that your efforts open doors to new opportunities and growth.  THE POSITION   The  Quality Engineer  leads the development, implementation, and continuous improvement of quality standards and inspection criteria to ensure product... 
    Suggested
    Full time
    Remote work

    Smartrend Manufacturing Group

    Winnipeg, MB
    15 days ago
  •  ...full-time professionals, and our annual growth rate exceeds 25%. Job Description Miratech is looking for a skilled Senior AI Engineer with Python, LangGraph and LangChain to join our US healthcare client. You will play a key role in developing and enhancing an... 
    Suggested
    Full time
    Internship
    Remote work
    Relocation

    Miratech

    Winnipeg, MB
    5 days ago
  •  ...Drive AI innovation as a Senior AI Engineer in Mississauga, Canada. Craft Python-based AI applications using FastAPI or Flask while optimizing performance and scalability for enterprise solutions. We are seeking an experienced Senior AI Engineer with over 8 years in software... 

    Talentola

    Winnipeg, MB
    6 days ago
  • Quality Engineer Job Summary Reporting directly to the Quality Assurance Manager, Quality Engineer performs execution of Quality Engineering systems including; handling of nonconforming materials, Corrective/Preventive Actions, failure reporting, analysis and corrective action... 
    Full time

    Linamar Corporation

    Winnipeg, MB
    14 days ago
  •  ...saving lives. Job Summary   Reporting to the Sr. Director, Quality, the Director, Quality Assurance leads QA Operations, Quality...  ...& Skills • Bachelor’s degree in Science, Pharmacy, Engineering, or a related field. An advanced degree is considered an asset.... 
    Flexible hours

    Emergent Biosolutions

    Winnipeg, MB
    5 days ago
  •  ...bowl, or grabbing some groceries, our tech platform connects tens of millions of customers with hundreds of thousands of restaurant, grocery and convenience partners across the globe.About this role As a Senior Quality Engineer, you will work with an elite team of develop...
    Full time
    Night shift

    Just Eat Takeaway.com

    Winnipeg, MB
    14 days ago
  •  ...service, innovation and a commitment to people. Our dependable quality, reliable delivery, and knowledgeable follow-up service have made...  ...problem-solving skills. • The ability to read and interpret engineering drawings. • High proficiency in Microsoft office including,... 
    Permanent employment
    Full time
    Work at office

    Price Industries Limited

    Winnipeg, MB
    6 days ago
  •  ...Director of Quality Assurance Winnipeg, MB Full-Time, Onsite POSITION SUMMARY: The Director of Quality Assurance - Transit...  ...from recurring. Partner closely with Manufacturing, Engineering, Supply Chain, Service, and Operations teams to solve complex challenges... 
    Full time
    Internship
    Immediate start

    New Flyer

    Winnipeg, MB
    19 days ago
  •  ...processes and practices across our organization to help ensure flight and product efficiency. As a Quality Inspector , you will ensure that all aircraft components, engines, and systems are maintained, repaired, or overhauled in full compliance with regulatory standards,... 
    Long term contract
    Temporary work
    Traineeship
    Internship
    Immediate start
    Flexible hours
    Shift work
    Weekend work

    StandardAero

    Winnipeg, MB
    11 days ago
  •  ...The Quality Control Inspector will provide Quality Process Audit support to the Assembly Line. This position will assist in reducing process variation through the use of statistical data analysis, process audits, and department metrics. The position emphasizes improving internal... 
    Full time
    Temporary work

    Motor Coach Industries

    Winnipeg, MB
    10 days ago
  •  ...professionals where our culture is informed by our values and commitment to protecting and saving lives. Job Summary  The Specialist I, Quality Compliance is responsible for supporting and representing Emergent during regulatory inspections, customer audits, and internal... 
    Flexible hours

    Emergent Biosolutions

    Winnipeg, MB
    5 days ago
  •  ...service, innovation and a commitment to people. Our dependable quality, reliable delivery, and knowledgeable follow-up service have made...  ...shipping processes. • Work collaboratively with Production, Engineering, Shipping, Logistics, Supply Chain, Quality, and other cross-... 

    Price Industries Limited

    Winnipeg, MB
    6 days ago
  •  ...passionate professionals where our culture is informed by our values and commitment to protecting and saving lives. Job Summary  The Quality Assurance Deviations and CAPA Specialist provides support by managing deviations and corrective/preventative actions. Additionally,... 
    Flexible hours

    Emergent Biosolutions

    Winnipeg, MB
    5 days ago
  •  ...perform to your expectations. We are proud of our long-standing partnerships with family-owned Canadian chicken farmers to produce high quality products. Role:   Responsible for monitoring employee GMP compliance and product quality specification compliance of the raw... 
    Full time
    Afternoon shift

    sunrisefarms

    Winnipeg, MB
    5 days ago
  • Education: College, CEGEP or other non-university certificate or diploma from a program of 1 year to 2 years. Security and safety: Criminal record check. Screening questions: Do you have experience working in this field?. Employment terms options: Evening. Experience: 1 year...
    Hourly pay
    Long term contract
    Permanent employment
    Immediate start
    Day shift
    Afternoon shift

    Smart Box Packaging Inc.

    Winnipeg, MB
    6 days ago
  •  ...Manager, Quality AssuranceApply: Remote, ON: Full time: Posted Yesterday: R2616835Our team members are at the heart of everything we do. At Cencora, we are united in our responsibility to create healthier futures, and every person here is essential to us being... 
    Full time
    Remote work

    Micro Technologies

    Winnipeg, MB
    6 days ago
  • Engineering Student - QualityBuild an Aviation Career You’re Proud OfAt StandardAero, we use our ingenuity and know-how to find solutions for the simple to the most complex challenges in Aviation. Together, we get the job done and done well. Bring your mechanical aptitude,... 
    Part time
    Internship

    StandardAero

    Winnipeg, MB
    19 days ago
  • $95k - $105k per year

     ...Quality Assurance Manager Winnipeg, MB  |  Full-Time  |  Permanent  |  $95,000 – $105,000 per year About the Opportunity Staffmax is excited to partner with a well-established Winnipeg organization to find an experienced Quality Assurance Manager . This is a confidential... 
    Permanent employment
    Full time
    Casual work
    Work at office
    Night shift

    Staffmax Staffing & Recruiting

    Winnipeg, MB
    12 days ago
  •  ...As our project portfolio continues to grow, we are seeking a Quality Coordinator (Site) to lead project quality activities and ensure...  ...backgrounds or equivalent: Power Line Technician (PLT) Electrical Engineering Technologist Construction Inspector Utility Inspector... 
    Full time
    Contract work

    Victor Energy

    Winnipeg, MB
    more than 2 months ago
  •  ..., single-source solutions provider delivering consistent, high-quality aluminum permanent mold castings to our customers. We are currently...  ...to ensure product and process conformance. Support Engineering and Sales staff in New Job and Continuous Improvement activities... 
    Permanent employment
    Full time
    Flexible hours
    Shift work

    Nordstern Group of Companies

    Winnipeg, MB
    13 days ago
  •  ...sustainable, and reliable charging and mobility solutions. Further information is available at . POSITION SUMMARY: Ensure the New Flyer Quality system is fully understood, executed, and maintained across the Winnipeg facility. The Quality Manager provides leadership to the... 
    Full time
    Internship
    Immediate start
    Flexible hours
    Shift work

    New Flyer

    Winnipeg, MB
    more than 2 months ago
  • Take on a pivotal role as an AI Quality Assurance and Testing Engineer, focusing on the assessment of machine learning and generative AI models. Ensure high standards of quality, bias assessment, and responsible AI practices across applications. In this short-term contract... 
    Temporary work

    Infotek Consulting Inc.

    Winnipeg, MB
    1 hour ago
  • Build trustworthy AI personalization quality at Autodesk in Toronto using TypeScript, Python, and automated testing expertise. Support reliable Autodesk...  ..., and AI behavior evaluation. The role requires software engineering, quality engineering, or test automation experience, with a... 

    Skillhub Solutions Inc.

    Winnipeg, MB
    1 hour ago
  • Evaluate generative AI systems as a Quality Engineer in Toronto, using Python and test automation within a hybrid, full-time role.Epsilon Solutions LTD seeks an AI evaluator and quality engineer for generative-AI solutions. The position defines evaluation approaches for models... 
    Full time

    Best Your Buy

    Winnipeg, MB
    1 hour ago
  •  ...insights, products, and services that make quality care more accessible and affordable. Here,...  ...from you. Position Summary The People AI Solutions Designer is the builder. Where the...  ...science, information systems, design, engineering, or a related field — or equivalent demonstrated... 

    McKesson

    Winnipeg, MB
    6 days ago
  •  ...Grounded in safety, quality, and ethics, our experts lead their fields with dedication, a creative spirit, and a vision for growth. We draw from more than 20 technical specialties worldwide and are devoted to fostering a community of diverse talents, backgrounds, and expertise... 
    Work at office
    Worldwide
    Shift work

    Stantec

    Winnipeg, MB
    6 days ago
  •  ...perform to your expectations. We are proud of our long-standing partnerships with family-owned Canadian chicken farmers to produce high quality products. Role:   Responsible for monitoring employee GMP compliance and product quality specification compliance of the raw... 
    Contract work

    sunrisefarms

    Winnipeg, MB
    13 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Ai Quality Engineer. Be the first to apply!