Ai Quality Engineer
Centraprise
Key Responsibilities
Build and maintain automated tests for AI agent workflows, APIs, tools, telemetry backed analysis, remediation flows, and ticketing behavior.
Design evaluation suites for LLM and agentic behavior, including expected-answer checks, rubric-based grading, regression datasets, tool-call validation, and safety/approval checks.
Use or help implement evaluation frameworks such as Pydantic Evals / Pydantic AI, Strands Evals, LangSmith, DeepEval, Ragas, promptfoo, or similar tools.
Validate multi-turn support scenarios, clarification flows, knowledge retrieval, script/remediation recommendations, escalation paths, and failure handling.
Test on-device agent behavior where needed, including Windows service/tray behavior, telemetry collection, anomaly detection, local remediation handoff, logs, and resource impact.
Debug quality issues directly by reading logs, tracing requests, reproducing failures, and making small code/test changes without heavy engineering hand-holding.
Partner with engineering and product to define release gates, quality metrics, evaluation rubrics, and confidence thresholds for pilot readiness.
Contribute to CI quality checks, test fixtures, mocked integrations, regression suites, and test data management.
Identify risks in AI behavior, including hallucinated diagnosis, unsafe remediation suggestions, missing consent, weak ticket summaries, brittle tool use, and poor escalation behavior.
Responsibilities
4+ years of experience in QA engineering, SDET, test automation, software engineering, or similar hands‑on quality roles.
Strong Python experience, including writing production‑quality tests and debugging application code.
Experience testing backend services, APIs, async workflows, integrations, logs, and distributed systems.
Demonstrated ability to work independently in a codebase, identify root causes, and make targeted fixes or test improvements.
Experience with modern test frameworks such as pytest, unittest, Playwright, xUnit, or similar.
Comfortable testing ambiguous AI/non‑deterministic behavior using datasets, rubrics, assertions, metrics, and regression baselines.
Ability to distinguish product bugs, prompt/model behavior issues, data issues, integration failures, and test harness problems.
Strong written communication for documenting repro steps, risks, test plans, and release readiness.
Qualifications
Direct experience with Pydantic, Pydantic AI, or Pydantic Evals.
Experience testing LLM applications, agentic systems, RAG, tool calling, MCP servers, or AI assistants.
Experience building eval datasets, LLM‑as‑judge flows, deterministic evaluators, and regression dashboards.
Windows endpoint, device telemetry, PowerShell, .NET, MQTT, or on‑device agent testing experience.
Experience with IT support workflows, ServiceNow, ticketing, endpoint management, or enterprise support tooling.
Ability to contribute small application changes, not only test changes.
#J-18808-Ljbffr
Build and maintain automated tests for AI agent workflows, APIs, tools, telemetry backed analysis, remediation flows, and ticketing behavior.
Design evaluation suites for LLM and agentic behavior, including expected-answer checks, rubric-based grading, regression datasets, tool-call validation, and safety/approval checks.
Use or help implement evaluation frameworks such as Pydantic Evals / Pydantic AI, Strands Evals, LangSmith, DeepEval, Ragas, promptfoo, or similar tools.
Validate multi-turn support scenarios, clarification flows, knowledge retrieval, script/remediation recommendations, escalation paths, and failure handling.
Test on-device agent behavior where needed, including Windows service/tray behavior, telemetry collection, anomaly detection, local remediation handoff, logs, and resource impact.
Debug quality issues directly by reading logs, tracing requests, reproducing failures, and making small code/test changes without heavy engineering hand-holding.
Partner with engineering and product to define release gates, quality metrics, evaluation rubrics, and confidence thresholds for pilot readiness.
Contribute to CI quality checks, test fixtures, mocked integrations, regression suites, and test data management.
Identify risks in AI behavior, including hallucinated diagnosis, unsafe remediation suggestions, missing consent, weak ticket summaries, brittle tool use, and poor escalation behavior.
Responsibilities
4+ years of experience in QA engineering, SDET, test automation, software engineering, or similar hands‑on quality roles.
Strong Python experience, including writing production‑quality tests and debugging application code.
Experience testing backend services, APIs, async workflows, integrations, logs, and distributed systems.
Demonstrated ability to work independently in a codebase, identify root causes, and make targeted fixes or test improvements.
Experience with modern test frameworks such as pytest, unittest, Playwright, xUnit, or similar.
Comfortable testing ambiguous AI/non‑deterministic behavior using datasets, rubrics, assertions, metrics, and regression baselines.
Ability to distinguish product bugs, prompt/model behavior issues, data issues, integration failures, and test harness problems.
Strong written communication for documenting repro steps, risks, test plans, and release readiness.
Qualifications
Direct experience with Pydantic, Pydantic AI, or Pydantic Evals.
Experience testing LLM applications, agentic systems, RAG, tool calling, MCP servers, or AI assistants.
Experience building eval datasets, LLM‑as‑judge flows, deterministic evaluators, and regression dashboards.
Windows endpoint, device telemetry, PowerShell, .NET, MQTT, or on‑device agent testing experience.
Experience with IT support workflows, ServiceNow, ticketing, endpoint management, or enterprise support tooling.
Ability to contribute small application changes, not only test changes.
#J-18808-Ljbffr
Vacancy posted 1 hour ago
Similar jobs that could be interesting for youBased on the Ai Quality Engineer in Winnipeg, MB vacancy
- ...are building in our weekly changelog. Rootly is building the AI-native future of incident management, and we need someone who... ...can push our AI to its limits before our customers do. As our AI Quality Engineer, you'll own the evaluation and optimization of Rootly's agentic...SuggestedInternshipHome office
- ...We are seeking a highly experienced and hands-on Supplier Quality Engineer (SQE) with strong expertise in casting and machining processes, particularly aluminum castings. The role requires close collaboration with suppliers, frequent onsite visits, and active involvement in...Suggested
- ...and grow their careers in an equitable environment. Learn about SKF at. Position Overview We are seeking a skilled Quality Electrical Engineer to support customer and manufacturing quality for electrical and electronic products. The position is responsible for...SuggestedPermanent employment
- Lead the AI Quality Engineering initiatives at Replicant, enhancing our focus on top-notch customer service solutions. This managerial role involves redefining QA standards using innovative strategies. At Replicant, you will head the quality engineering efforts by mentoring...Suggested
- Take on a pivotal role as an AI Quality Assurance and Testing Engineer, focusing on the assessment of machine learning and generative AI models. Ensure high standards of quality, bias assessment, and responsible AI practices across applications. In this short-term contract...SuggestedTemporary work
- Evaluate generative AI systems as a Quality Engineer in Toronto, using Python and test automation within a hybrid, full-time role.Epsilon Solutions LTD seeks an AI evaluator and quality engineer for generative-AI solutions. The position defines evaluation approaches for models...Full time
- Build trustworthy AI personalization quality at Autodesk in Toronto using TypeScript, Python, and automated testing expertise. Support reliable Autodesk... ..., and AI behavior evaluation. The role requires software engineering, quality engineering, or test automation experience, with a...
- ...We are seeking a Senior Software Quality Engineer – AI Focused to drive quality, reliability, and automation for AI-enabled software products. The role will collaborate with Development, DevOps, Product, and QE teams to design robust testing strategies and scalable automation...Full time
- ...service, innovation and a commitment to people. Our dependable quality, reliable delivery, and knowledgeable follow-up service... ...free Tea and Coffee. Position Summary Join our AI team as an Intermediate AI Engineer, where you'll build and deploy AI applications, agents, and...
- ...backgrounds and experiences who are passionate about quality, teamwork, and making an impact. The OpportunityWe’... ...Automation Specialists who are passionate about modern engineering practices and excited about how AI is changing the way software is tested and delivered ....Internship
- ...StackAdapt is hiring a Staff Quality Engineer to lead the quality strategy for AI/agentic systems. You will design evaluation frameworks for non-deterministic AI, build scalable automated test suites, and set quality benchmarks across CI/CD pipelines. You’ll mentor QE teams...Remote work
- Become a key part of our team as an AI Quality Engineer, focusing on automated test development and quality assurance. This role combines Python expertise with a deep understanding of AI processes. With a minimum of 4 years in QA or test automation, you'll build and maintain...
$85k - $225k per year
...Toronto is seeking an experienced professional to validate and ensure the reliability of Veeva AI Agents. The ideal candidate should have expertise in data quality, prompt engineering, and proficiency in Python. This full-time role offers a competitive salary range of $85,000...Full timeRemote workFlexible hours- ...Rootly is hiring an AI Quality Engineer to own the evaluation and optimization of its agentic AI features. You will design test scenarios, probe how AI agents reason, and work with engineering to close the loop on performance across incident workflows. You will partner...
- ...TCS is aiming to become the world's largest AI-led technology services company and is... ...chatbot conversation analysis and recommend quality improvements. Develop and maintain test... ...possible. Work closely with developers, AI engineers, and content teams to resolve defects....Long term contract
- Apple’s Creativity Apps team seeks a Machine Learning Engineer focused on evaluating ML models and AI features in our creativity tools. You will define evaluation... ..., and develop scalable infrastructure to ensure quality from model development to customer delivery. You will...
- ...and aspirations with us, and we'll ensure that your efforts open doors to new opportunities and growth. THE POSITION The Quality Engineer leads the development, implementation, and continuous improvement of quality standards and inspection criteria to ensure product...Full timeRemote work
- UnitedHealth Group in Canada is seeking a Principal Quality Engineer to lead integration and end-to-end quality for Enterprise Imaging across on-premise and cloud deployments. You will set the strategy, establish automation standards, and mentor engineers to ensure evidence-...Remote work
- ...Vancity is seeking an Engineering Lead, Quality to steer quality engineering across member‑facing platforms. You will lead hands‑on engineering, build scalable automation, and drive API‑first testing with AI‑driven approaches. This full‑time, permanent role is based at the...Permanent employmentFull timeWork at office
- Myant is hiring a Senior Software Quality Engineer to lead testing across health software platforms, emphasizing patient safety and privacy. You... ...quality strategy, stay hands-on with automation, and leverage AI to accelerate testing without compromising standards. In this role...
- Quality Engineer (QE) / SDET - AI Chatbot ValidationJob Description: We are looking for a seasoned AI first Senior Quality Engineer who is empowered to help the DevOps enablement team deliver highest level quality and is responsible for ensuring the team live quality values...
- ...Drive AI innovation as a Senior AI Engineer in Mississauga, Canada. Craft Python-based AI applications using FastAPI or Flask while optimizing performance and scalability for enterprise solutions. We are seeking an experienced Senior AI Engineer with over 8 years in software...
- ...full-time professionals, and our annual growth rate exceeds 25%. Job Description Miratech is looking for a skilled Senior AI Engineer with Python, LangGraph and LangChain to join our US healthcare client. You will play a key role in developing and enhancing an...Full timeInternshipRemote workRelocation
- ...automotive customers, and will operate to IATF 16949 requirements.As Quality Engineer, you lead the quality side of new product development and... ...you to meet your needs.This posting is for an existing vacancy. AI in hiring: we use artificial intelligence to screen and assess...For subcontractorInternshipWork at office
$80k - $110k per year
..., and enthusiastic individual to join our Quality team to support the day-to-day operations... ...projects through to execution. As a Quality Engineer, you will act as the project Quality Lead... ...to applicable law.Artificial Intelligence (AI) is not used to screen, assess, or select...Work at office$107.63k - $157.85k per year
...Joignez-vous à nous et contribuez à ce qui fait avancer le monde.The Quality Engineer will serve as both a technical design authority and a customer-... ...in MS Office, MS Teams, and ability to leverage AI tools in job functions.Ability to manage multiple projects and...$90k - $110k per year
Arrow Machine and Fabrication Group is an engineering-focused solution provider of machined castings... ...focuses on providing innovative design, quality craftsmanship, and exceptional customer... ...Act.Candidates please note, we do not use AI for application screening.Disclaimer: Only...Work at office$100k - $140k per year
...an experienced Material Review Board (MRB) Engineer to join our dynamic team. The MRB Engineer... ...with engineering, manufacturing, and quality teams to ensure compliance with design, regulatory... ...add insights directly into each article, started with the help of AI. #J-18808-LjbffrFull timeRelocation$115.5k per year
...precision. With 465 billion automated optimizations per second, the AI-powered StackAdapt Marketing Platform seamlessly connects brand... ...and marketing channels.StackAdapt is seeking a talented Quality Engineer who is passionate about software quality and test automation...$74k - $99k per year
...comprehensive range of body, chassis and engineering solutions to global customers. Through our... ...products to market.Job Responsibilities:The Quality Engineer will provide technical support,... ...requirements and your local Data Privacy Law.AI-Assisted Screening DisclosureAs part of...Temporary workPart timeLocal area
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Ai Quality Engineer. Be the first to apply!
Related searches
- quality engineer Winnipeg, MB
- ingénieur qualité Winnipeg, MB
- technicien qualité Winnipeg, MB
- quality engineer manufacturing Winnipeg, MB
- quality engineer- qms Winnipeg, MB
- entry level quality engineer Winnipeg, MB
- directeur amelioration continue audit qualite Winnipeg, MB
- stage qualité Winnipeg, MB
- qualité agroalimentaire Winnipeg, MB
- qualité santé Winnipeg, MB

