Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Ai Quality Engineer

Rootly

About RootlyAt Rootly, we are on a mission to be the go-to way companies respond when things go wrong, helping every organization be more reliable. We do this by building an industry-leading incident management platform that allows companies around the world to consistently and quickly resolve incidents. We are not simply transforming an industry, we are carving out a new multi-billion-dollar category ourselves and need incredible talent to achieve this ambitious goal together.
Customers love Rootly. Some of the fastest growing companies around the world such as NVIDIA, Figma, Canva, Tripadvisor, Squarespace and more rely on Rootly to power their critical incident management process. They obsess over our delightful enterprise-ready platform and unique partnership model. See why our customers have reviewed us 5 stars on G2.
Investors love Rootly. We are backed by some of the most respected funds in the world from Y Combinator to operators like the CTO of Dropbox and GitHub. We'd be happy to disclose our entire funding and profitability picture live during the interview. As a culture we relentlessly put transparency first. We conduct monthly financial reviews as a team so everyone has a pulse on the health of the business and publish what we are building in our weekly changelog.
Rootly is building the AI-native future of incident management, and we need someone who can push our AI to its limits before our customers do. As our AI Quality Engineer, you'll own the evaluation and optimization of Rootly's agentic AI features -- designing test scenarios, running adversarial prompts, interpreting outputs, and working directly with engineering and product to close the loop on performance.
This isn't traditional QA. You'll spend your days thinking like an attacker, a confused user, and a power user all at once -- probing how our AI agents reason, make decisions, and handle edge cases across complex incident workflows.
What you'll do
Design and execute prompt-based test scenarios that cover happy paths, edge cases, and adversarial inputs across Rootly's agentic AI features
Evaluate AI outputs for accuracy, relevance, consistency, and alignment with expected workflow behaviour
Build and maintain an evaluation framework;
structured test libraries, scoring rubrics, and regression suites to track AI performance over time
Identify failure modes, hallucinations, reasoning gaps, and unexpected agent behaviours;
document findings and work with engineers to resolve them
Partner with Product and Engineering on new AI feature releases, contributing to acceptance criteria and quality gates before launch
Define and track quality metrics (accuracy rates, failure frequency, regression trends) and report findings to stakeholders
Stay current on LLM evaluation techniques, prompt engineering best practices, and agentic testing methodologies
What we're looking for
+5 years in QA, product operations, AI/ML evaluation, or a closely related role
Hands-on experience testing or evaluating LLM-powered or agentic AI products
Strong prompt engineering instincts -- you understand how wording, context, and structure affect model behaviour
Comfortable writing scripts or working with evaluation tools(Python a plus;
not required to be a full-stack engineer)
Sharp analytical thinking;
you can spot a subtle reasoning failure and articulate exactly why it's a problem
Clear written communicator;
able to translate AI behaviour findings for both technical and non-technical audiences
Nice to have
Familiarity with incident management, DevOps, or IT operations workflows is a strong asset
Experience with evaluation frameworks (e.G. LangSmith, PromptFlow, Braintrust, or similar)
Exposure to red‑teaming or adversarial testing of AI systems
Comfortable writing E2E tests with Playwright
Background working at a B2B SaaS or developer‑tools company
Familiar with mobile app testing (iOS/Android)
Why Rootly?We're not just another startup. We're building something category-defining and want teammates who crave ownership, love solving hard problems, and thrive in a high‑bar, high-impact environment.
Here's what you can expect when you join Rootly:
Competitive compensation and early equity in a fast‑growing, venture‑backed company.
Comprehensive medical, dental, and vision coverage.
3 weeks of vacation, plus unlimited sick and mental health days, and a company‑wide end‑of‑year shutdown to recharge.
$500 stipend for home office setup.
Unlimited token usage and access to AI tools
A fast‑moving, high‑impact environment where your leadership and ideas directly shape the future of the company.
This is a newly created vacancy.
As we build out the team, we may use AI to support parts of our interview process, like reviewing assessments, but it's a tool that helps our people, not a substitute for their judgement.
Accommodations are available on request at any stage of our recruitment process. Contact View email address on ca.hrm.tc and we will work with you to meet your needs confidentially.
Rootly is an equal opportunity employer. We do not discriminate on the basis of race, ancestry, place of origin, colour, ethnic origin, citizenship, creed, sex, sexual orientation, gender identity, gender expression, age, record of offences, marital status, family status, disability, or any other ground protected by the Ontario Human Rights Code or applicable law.
#J-18808-Ljbffr
Vacancy posted 1 hour ago
Similar jobs that could be interesting for youBased on the Ai Quality Engineer in Winnipeg, MB vacancy
  •  ...Build and maintain automated tests for AI agent workflows, APIs, tools, telemetry backed...  ..., logs, and resource impact. Debug quality issues directly by reading logs, tracing requests...  ...small code/test changes without heavy engineering hand-holding. Partner with engineering... 
    Suggested
    Local area

    Centraprise

    Winnipeg, MB
    1 hour ago
  •  ...We are seeking a highly experienced and hands-on Supplier Quality Engineer (SQE) with strong expertise in casting and machining processes, particularly aluminum castings. The role requires close collaboration with suppliers, frequent onsite visits, and active involvement in... 
    Suggested

    Global Connect Technologies

    Winnipeg, MB
    6 days ago
  •  ...and grow their careers in an equitable environment. Learn about SKF at. Position Overview We are seeking a skilled Quality Electrical Engineer to support customer and manufacturing quality for electrical and electronic products. The position is responsible for... 
    Suggested
    Permanent employment

    SKF Group

    Winnipeg, MB
    6 days ago
  • Take on a pivotal role as an AI Quality Assurance and Testing Engineer, focusing on the assessment of machine learning and generative AI models. Ensure high standards of quality, bias assessment, and responsible AI practices across applications. In this short-term contract... 
    Suggested
    Temporary work

    Infotek Consulting Inc.

    Winnipeg, MB
    1 hour ago
  •  ...We are seeking a Senior Software Quality Engineer – AI Focused to drive quality, reliability, and automation for AI-enabled software products. The role will collaborate with Development, DevOps, Product, and QE teams to design robust testing strategies and scalable automation... 
    Suggested
    Full time

    PeoplePilot

    Winnipeg, MB
    1 hour ago
  • Evaluate generative AI systems as a Quality Engineer in Toronto, using Python and test automation within a hybrid, full-time role.Epsilon Solutions LTD seeks an AI evaluator and quality engineer for generative-AI solutions. The position defines evaluation approaches for models... 
    Full time

    Best Your Buy

    Winnipeg, MB
    1 hour ago
  • Lead the AI Quality Engineering initiatives at Replicant, enhancing our focus on top-notch customer service solutions. This managerial role involves redefining QA standards using innovative strategies. At Replicant, you will head the quality engineering efforts by mentoring... 

    Replicant

    Winnipeg, MB
    1 hour ago
  • Build trustworthy AI personalization quality at Autodesk in Toronto using TypeScript, Python, and automated testing expertise. Support reliable Autodesk...  ..., and AI behavior evaluation. The role requires software engineering, quality engineering, or test automation experience, with a... 

    Skillhub Solutions Inc.

    Winnipeg, MB
    1 hour ago
  •  ...backgrounds and experiences who are passionate about quality, teamwork, and making an impact.The Opportunity We’...  ...Automation Specialists who are passionate about modern engineering practices and excited about how AI is changing the way software is tested and delivered .... 
    Internship

    Plato

    Winnipeg, MB
    1 hour ago
  •  ...StackAdapt is hiring a Staff Quality Engineer to lead the quality strategy for AI/agentic systems. You will design evaluation frameworks for non-deterministic AI, build scalable automated test suites, and set quality benchmarks across CI/CD pipelines. You’ll mentor QE teams... 
    Remote work

    Devconnectplatform

    Winnipeg, MB
    1 hour ago
  • Apple’s Creativity Apps team seeks a Machine Learning Engineer focused on evaluating ML models and AI features in our creativity tools. You will define evaluation...  ..., and develop scalable infrastructure to ensure quality from model development to customer delivery. You will... 

    Lex

    Winnipeg, MB
    1 hour ago
  •  ...Rootly is hiring an AI Quality Engineer to own the evaluation and optimization of its agentic AI features. You will design test scenarios, probe how AI agents reason, and work with engineering to close the loop on performance across incident workflows. You will partner... 

    Rootly

    Winnipeg, MB
    1 hour ago
  • Become a key part of our team as an AI Quality Engineer, focusing on automated test development and quality assurance. This role combines Python expertise with a deep understanding of AI processes. With a minimum of 4 years in QA or test automation, you'll build and maintain... 

    Centraprise

    Winnipeg, MB
    1 hour ago
  • $85k - $225k per year

     ...Toronto is seeking an experienced professional to validate and ensure the reliability of Veeva AI Agents. The ideal candidate should have expertise in data quality, prompt engineering, and proficiency in Python. This full-time role offers a competitive salary range of $85,000... 
    Full time
    Remote work
    Flexible hours

    Veeva Systems

    Winnipeg, MB
    1 hour ago
  •  ...service, innovation and a commitment to people. Our dependable quality, reliable delivery, and knowledgeable follow-up service...  ...free Tea and Coffee. Position Summary Join our AI team as an Intermediate AI Engineer, where you'll build and deploy AI applications, agents, and... 

    Price Industries Limited

    Winnipeg, MB
    6 days ago
  • Mosaic Clinical Technologies (Canada), Inc.™ is seeking a Head of Quality Engineering to own the software quality and test automation strategy across the portfolio. You will lead architecture and maintenance of scalable automation frameworks, embedding gates into CI/CD and... 

    Radiology Partners, Inc.

    Winnipeg, MB
    1 hour ago
  •  ...TCS is aiming to become the world's largest AI-led technology services company and is...  ...chatbot conversation analysis and recommend quality improvements. Develop and maintain test...  ...possible. Work closely with developers, AI engineers, and content teams to resolve defects.... 
    Long term contract

    Tata Consultancy Services Limited

    Winnipeg, MB
    1 hour ago
  •  ...and aspirations with us, and we'll ensure that your efforts open doors to new opportunities and growth.  THE POSITION   The  Quality Engineer  leads the development, implementation, and continuous improvement of quality standards and inspection criteria to ensure product... 
    Full time
    Remote work

    Smartrend Manufacturing Group

    Winnipeg, MB
    15 days ago
  • Myant is hiring a Senior Software Quality Engineer to lead testing across health software platforms, emphasizing patient safety and privacy. You...  ...quality strategy, stay hands-on with automation, and leverage AI to accelerate testing without compromising standards. In this role... 

    Myant Inc.

    Winnipeg, MB
    1 hour ago
  •  ...Vancity is seeking an Engineering Lead, Quality to steer quality engineering across member‑facing platforms. You will lead hands‑on engineering, build scalable automation, and drive API‑first testing with AI‑driven approaches. This full‑time, permanent role is based at the... 
    Permanent employment
    Full time
    Work at office

    Vancity

    Winnipeg, MB
    1 hour ago
  • Quality Engineer (QE) / SDET – AI Chatbot ValidationQuality Engineer (QE) / SDET – AI Chatbot Validation Job Description: We are looking for a seasoned AI first Senior Quality Engineer who is empowered to help the DevOps enablement team deliver highest level quality and... 

    Astra-North Infoteck Inc.

    Winnipeg, MB
    1 hour ago
  • UnitedHealth Group in Canada is seeking a Principal Quality Engineer to lead integration and end-to-end quality for Enterprise Imaging across on-premise and cloud deployments. You will set the strategy, establish automation standards, and mentor engineers to ensure evidence-... 
    Remote work

    UnitedHealth Group

    Winnipeg, MB
    1 hour ago
  • $100k - $140k per year

     ...an experienced Material Review Board (MRB) Engineer to join our dynamic team. The MRB Engineer...  ...with engineering, manufacturing, and quality teams to ensure compliance with design, regulatory...  ...add insights directly into each article, started with the help of AI. #J-18808-Ljbffr
    Full time
    Relocation

    TechEra Global Inc

    Winnipeg, MB
    1 hour ago
  • $74k - $99k per year

     ...comprehensive range of body, chassis and engineering solutions to global customers. Through our...  ...products to market.Job Responsibilities:The Quality Engineer will provide technical support,...  ...requirements and your local Data Privacy Law.AI-Assisted Screening DisclosureAs part of... 
    Temporary work
    Part time
    Local area

    Magna International

    Winnipeg, MB
    1 hour ago
  •  ...Drive AI innovation as a Senior AI Engineer in Mississauga, Canada. Craft Python-based AI applications using FastAPI or Flask while optimizing performance and scalability for enterprise solutions. We are seeking an experienced Senior AI Engineer with over 8 years in software... 

    Talentola

    Winnipeg, MB
    6 days ago
  •  ...full-time professionals, and our annual growth rate exceeds 25%. Job Description Miratech is looking for a skilled Senior AI Engineer with Python, LangGraph and LangChain to join our US healthcare client. You will play a key role in developing and enhancing an... 
    Full time
    Internship
    Remote work
    Relocation

    Miratech

    Winnipeg, MB
    6 days ago
  • $107.63k - $157.85k per year

     ...Joignez-vous à nous et contribuez à ce qui fait avancer le monde.The Quality Engineer will serve as both a technical design authority and a customer-...  ...in MS Office, MS Teams, and ability to leverage AI tools in job functions.Ability to manage multiple projects and... 

    ABB Inc

    Winnipeg, MB
    1 hour ago
  • $115.5k per year

     ...precision. With 465 billion automated optimizations per second, the AI-powered StackAdapt Marketing Platform seamlessly connects brand...  ...and marketing channels.StackAdapt is seeking a talented Quality Engineer who is passionate about software quality and test automation... 
    Local area
    Remote work
    Home office

    StackAdapt Inc.

    Winnipeg, MB
    1 hour ago
  • $90k - $110k per year

    Arrow Machine and Fabrication Group is an engineering-focused solution provider of machined castings...  ...focuses on providing innovative design, quality craftsmanship, and exceptional customer...  ...Act.Candidates please note, we do not use AI for application screening.Disclaimer: Only... 
    Work at office

    Arrow Machine And Fabrication Group

    Winnipeg, MB
    1 hour ago
  • $86.1k - $136.1k per year

    We are seeking a curious and collaborative Quality Engineer who is motivated by emerging technologies and creative problem-solving. You will contribute...  ...and analytics technologies, such as artificial intelligence (AI), and automated processing tools, to analyze and process the... 
    Long term contract
    Temporary work
    Local area
    Flexible hours

    Manulife

    Winnipeg, MB
    1 hour ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Ai Quality Engineer. Be the first to apply!