Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Ai Quality Engineer: Prompt Testing & Evaluation

Rootly

Rootly is hiring an AI Quality Engineer to own the evaluation and optimization of its agentic AI features. You will design test scenarios, probe how AI agents reason, and work with engineering to close the loop on performance across incident workflows.You will partner with product and engineering for new AI releases, define quality metrics, and stay current on evaluation techniques. This role offers a fast-paced, high-impact environment with growth opportunities.
#J-18808-Ljbffr
Vacancy posted 1 hour ago
Similar jobs that could be interesting for youBased on the Ai Quality Engineer: Prompt Testing & Evaluation in Winnipeg, MB vacancy
  • Shape trusted AI evaluation at CrowdStrike using agent frameworks, LLM APIs, and verification-first test automation. Work on cloud-based cybersecurity systems.As a Software Development Engineer in Test, you will help determine how CrowdStrike AI agents are tested, evaluated... 
    Suggested

    CrowdStrike Inc.

    Winnipeg, MB
    1 hour ago
  • Take on a pivotal role as an AI Quality Assurance and Testing Engineer, focusing on the assessment of machine learning and generative AI models. Ensure high standards of quality, bias assessment, and responsible AI practices across applications. In this short-term contract... 
    Suggested
    Temporary work

    Infotek Consulting Inc.

    Winnipeg, MB
    1 hour ago
  • $85k - $225k per year

     ...Toronto is seeking an experienced professional to validate and ensure the reliability of Veeva AI Agents. The ideal candidate should have expertise in data quality, prompt engineering, and proficiency in Python. This full-time role offers a competitive salary range of $85,000... 
    Suggested
    Full time
    Remote work
    Flexible hours

    Veeva Systems

    Winnipeg, MB
    1 hour ago
  • Apple’s Creativity Apps team seeks a Machine Learning Engineer focused on evaluating ML models and AI features in our creativity tools. You will define evaluation...  ..., and develop scalable infrastructure to ensure quality from model development to customer delivery. You will... 
    Suggested

    Lex

    Winnipeg, MB
    1 hour ago
  • $118k - $152k per year

     ...experienced Software Developer with a strong understanding of AI-assisted development and prompt engineering. Ideally this person will have hands-on development...  .... Familiarity with prompt design, optimization, and testing techniques. Understanding of how AI technologies can... 
    Suggested
    Internship
    Immediate start

    Capco Group

    Winnipeg, MB
    1 hour ago
  • Shape trusted AI evaluation with performance testing and CI/CD expertise as a Software Development Engineer in Test at CrowdStrike. Work on scalable cybersecurity services.As an SDET, you will help define how AI agents are tested, evaluated, and trusted in production. CrowdStrike... 

    CrowdStrike Canada ULC

    Winnipeg, MB
    1 hour ago
  •  ...largest Indigenous-owned software testing and technology services company. For...  ...who are passionate about quality, teamwork, and making an impact....  ...who are passionate about modern engineering practices and excited about how AI is changing the way software is tested... 
    Internship

    PLATO

    Winnipeg, MB
    1 hour ago
  • $120k - $150k per year

     ...We’re also serious about technology. Make an impact as a Quality Engineering & Testing Manager to lead and strengthen quality engineering across the...  ...practical testing standards, advance automation and AI-enabled testing, promote test-driven development and other shift... 
    Work at office
    Remote work
    Flexible hours
    Shift work

    MNP

    Winnipeg, MB
    1 hour ago
  • Become a key part of our team as an AI Quality Engineer, focusing on automated test development and quality assurance. This role combines Python expertise with a deep understanding of AI processes. With a minimum of 4 years in QA or test automation, you'll build and maintain... 

    Centraprise

    Winnipeg, MB
    1 hour ago
  • Shape CoCounsel Legal with prompt engineering and legal expertise at Thomson Reuters in a hybrid work model. Evaluate artificial intelligence outputs trusted by legal professionals.As a Senior Legal AI Prompter and Evaluator, you will write prompts, skills, and agent configurations... 

    Thomson Reuters

    Winnipeg, MB
    1 hour ago
  • $100k - $140k per year

     ...value expertise, curiosity, and continuous improvement — and we give you the space to grow. About the RoleWe are seeking AI Developer /Prompt Engineer to contribute to major projects. The AI Developer lead the design, development, and deployment of advanced AI-enabled... 
    Permanent employment
    Full time
    Remote work
    Flexible hours

    Levio Inc.

    Winnipeg, MB
    1 hour ago
  • Join Capco as an AI Developer focusing on prompt engineering to transform business processes. Utilize your software development skills to leverage AI for innovative solutions. As a Software Developer at Capco, you will play a key role in integrating AI technologies into... 

    Capco Group

    Winnipeg, MB
    1 hour ago
  •  ...Responsibilities Build and maintain automated tests for AI agent workflows, APIs, tools, telemetry...  ..., and ticketing behavior. Design evaluation suites for LLM and agentic behavior,...  ...handoff, logs, and resource impact. Debug quality issues directly by reading logs, tracing... 
    Local area

    Centraprise

    Winnipeg, MB
    1 hour ago
  •  ...building in our weekly changelog. Rootly is building the AI-native future of incident management, and we need someone...  ...our AI to its limits before our customers do. As our AI Quality Engineer, you'll own the evaluation and optimization of Rootly's agentic AI features --... 
    Internship
    Home office

    Rootly

    Winnipeg, MB
    1 hour ago
  • Senior Quality Engineer – Test Automation, AI Testing & Java/PythonQuality Engineer– Test Automation, Java/Python & GenAI Roles and Responsibilities: Partner with DevOps,Development, Product, andQE teams todesign and executeexploratory, automated,and AI-focusedtests that... 

    Astra-North Infoteck Inc.

    Winnipeg, MB
    1 hour ago
  • Lead enterprise Test Automation and DevOps initiatives as a Senior Test Manager in a hybrid...  ...or Kitchener office arrangement. Shape quality engineering across complex insurance and retirement...  ...automation frameworks while applying Generative AI, Agentic AI, and modern QE practices. The... 
    Work at office
    Shift work

    Cognizant Technology Solutions Corporation

    Winnipeg, MB
    1 hour ago
  • $95k - $145k per year

     ....Job Responsibilities:This role involves testing AI models and software for diverse use cases...  ...robust testing framework to ensure high quality standards. Key tasks include engaging with...  ..., the role supports the R&D project team engineers and researchers by providing technical evaluation... 
    Local area

    Magna International

    Winnipeg, MB
    1 hour ago
  • Define conversational AI quality remotely from Canada or the United States using Python, LLM APIs, and rigorous evaluation systems.As Cresta’s mid-level AI Agent Evaluation Engineer, you will own quality measurement for production LLM-based conversational agents. You will combine... 
    Remote work

    JoinStartup

    Winnipeg, MB
    1 hour ago
  •  ...intelligence artificielle. Titre du poste : Ingénieur Prompt (Prompt Engineer)Lieu : Montréal, QuébecResponsabilités principalesLe développeur...  ...de grande taille (LLM), ainsi qu’avec les activités de tests et d’évaluation des performances, est requise. Le candidat idéal devra... 
    Apprenticeship

    Genpact Ltd.

    Winnipeg, MB
    1 hour ago
  • GenAI Prompt EngineerJOB SUMMARY The developer will be working within...  ...including ML and Gen AI. The candidate should be familiar...  ...python development and prompt engineering. The candidate should be able...  ...different LLM models, conduct testing and performance evaluation is... 

    Moonfire

    Winnipeg, MB
    1 hour ago
  • A tech company in Toronto is seeking a skilled Test Lead to oversee a team of testers and ensure application quality throughout the development lifecycle. Key responsibilities include managing test strategies, optimizing resource allocation, and providing insights for quality... 
    Work at office

    Rubicon Path

    Winnipeg, MB
    1 hour ago
  • Elevate AI advancement at DataAnnotation as an Electrical Engineer, focusing on real-world circuit and systems challenges. Apply your design and debugging skills...  ...realistic engineering problems from practical experience • Evaluate AI model outputs against professional benchmarks •... 

    DataAnnotation

    Winnipeg, MB
    1 hour ago
  • $140k - $175k per year

     ...Reuters Labs is seeking a Lead Research Engineer to lead the design and delivery of...  ...software for artificial intelligence (AI) model evaluation and benchmarking. You will build scalable...  ...large language models (LLMs), measure quality, cost, and latency, and make informed decisions... 
    Internship
    Work at office
    Local area
    Flexible hours

    Thomson Reuters

    Winnipeg, MB
    1 hour ago
  • Test GenAI and chatbot responses for accuracy, relevance, and consistency at Tata Consultancy Services. Use Jira, Confluence...  ...and Agile delivery practices.This SDET position combines quality assurance with conversational AI validation. You will identify hallucinations, incorrect... 

    Tata Consultancy Services Limited

    Winnipeg, MB
    1 hour ago
  • Lead and define qualityengineering and test strategy across Agile delivery teams Design...  ...,Developers, and Scrum teams to ensure quality delivery Ensure quality across...  ...of experience insoftware testing/quality engineering Proven experience leading QA orQuality... 

    Astra-North Infoteck Inc.

    Winnipeg, MB
    1 hour ago
  •  ...software stability through performance testing, automation, and careful analysis at...  ...the parking mobility industry.This Quality Assurance Automation Engineer validates product functionality, reliability...  ...deliver high-quality resultsExecute tests within defined estimates and... 
    For contractors

    T2 Systems

    Winnipeg, MB
    1 hour ago
  •  ...Entertainment Marketing firm in Metro Vancouver is seeking a Data Engineer in Test to enhance and ensure the reliability of analytics systems. In...  ...role, you will develop monitoring frameworks, execute data quality checks, and collaborate across teams to ensure accurate data... 

    Two Circles

    Winnipeg, MB
    1 hour ago
  • $119.1k - $230.5k per year

     ...there's no telling what you could accomplish.The Creativity Apps team is looking for a Machine Learning Engineer focused on the quality and evaluation of ML models and AI features in our creativity applications. You will define how we measure whether the product and AI features... 
    Local area
    Relocation

    Lex

    Winnipeg, MB
    1 hour ago
  • Evaluate AI-generated mechanical designs and calculations with strong engineering fundamentals and technical writing skills. Work fully remotely on flexible, project-based assignments...  ...work focuses on identifying technical errors, testing real-world applicability, and comparing... 
    For contractors
    Remote work
    Flexible hours

    Brunel

    Winnipeg, MB
    1 hour ago
  • Workada seeks an AI Evaluation Specialist to assess outputs from AI models across text, images, video and documents, applying detailed criteria...  .... You will follow precise guidelines, contribute to higher-quality systems, and collaborate with a mission-driven team focused on... 
    Full time
    Remote work

    Brock University

    Winnipeg, MB
    1 hour ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Ai Quality Engineer: Prompt Testing & Evaluation. Be the first to apply!