Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Ai Evaluation Test Automation Engineer

CrowdStrike Inc.

Shape trusted AI evaluation at CrowdStrike using agent frameworks, LLM APIs, and verification-first test automation. Work on cloud-based cybersecurity systems.As a Software Development Engineer in Test, you will help determine how CrowdStrike AI agents are tested, evaluated, and trusted in production. The position combines test automation with eval-driven development across prompts, models, tools, and orchestration. You will support functional, integration, performance, and end-to-end validation.Key Responsibilities:
Maintain golden datasets and scoring pipelines for AI agentsMeasure correctness, tool use, safety, latency, and costFeed production failures into evaluation setsBuild AI-powered workflows for test data and triageReview agent designs for testability and evaluabilityRequirements:
Computer Science degree or equivalent experienceKnowledge of object-oriented programming and distributed systemsPython preferred for evaluation and agent workExperience testing REST, gRPC, and JSON servicesVerification-first judgment about probabilistic AI outputUse AI evaluation, test automation, Python, and distributed-systems skills to improve trusted production testing at CrowdStrike.
#J-18808-Ljbffr
Vacancy posted 3 hours ago
Similar jobs that could be interesting for youBased on the Ai Evaluation Test Automation Engineer in Winnipeg, MB vacancy
  •  ...About the role We are seeking a highly skilled and self-motivated Senior AI Test Automation Engineer to join our Quality Engineering team. The ideal candidate will have extensive experience in test automation, database validation, AI-assisted testing, and modern quality... 
    Suggested

    PureFacts Financial Solutions

    Winnipeg, MB
    3 hours ago
  • Build AI test automation at CrowdStrike as a Software Development Engineer in Test, working remotely with Python and distributed systems.This hands-on role supports the Detonations and Threat Analysis Platform Engineering team. You will develop functional, integration, and performance... 
    Suggested
    Remote work

    CrowdStrike Inc.

    Winnipeg, MB
    3 hours ago
  • Build AI test automation with Java or Python as a Senior Quality Engineer focused on prompt behavior, accuracy, and reliable releases.This senior position requires five-plus years in quality engineering, quality assurance, test automation, or software development. You will... 
    Suggested

    Astra-North Infoteck Inc.

    Winnipeg, MB
    3 hours ago
  •  ...Rootly is hiring an AI Quality Engineer to own the evaluation and optimization of its agentic AI features. You will design test scenarios, probe how AI agents reason, and work with engineering to close the loop on performance across incident workflows. You will partner... 
    Suggested

    Rootly

    Winnipeg, MB
    3 hours ago
  • Build smarter Playwright tests with GitHub Copilot, CodeGen, and MCP tools in Veriday Inc.'s remote Senior Automation Engineer contract.This six-month Canadian engagement, potentially...  ...CI/CD pipelines. The role also evaluates AI tooling for test plans and work items... 
    Suggested
    Contract work
    Remote work

    Veriday

    Winnipeg, MB
    3 hours ago
  • Lead AI chatbot validation through Java, Python, and scalable automation frameworks while partnering with DevOps teams on CI/CD quality...  ...checks.This Senior Quality Engineer / SDET role supports DevOps...  ...exploratory, automated, and AI-focused testing. You will define test... 

    Astra-North Infoteck Inc.

    Winnipeg, MB
    3 hours ago
  • Shape trusted AI evaluation with performance testing and CI/CD expertise as a Software Development Engineer in Test at CrowdStrike. Work on scalable cybersecurity services.As an SDET, you will help define how AI agents are tested, evaluated, and trusted in production. CrowdStrike... 

    CrowdStrike Canada ULC

    Winnipeg, MB
    3 hours ago
  • $90k - $117k per year

    As an Automation Test Lead, you will lead AI-powered Quality Engineering and test automation solutions, including developing testing strategies and frameworks, integrating testing into CI/CD pipelines, validating AI behavior and integrations, and promoting responsible AI practices... 
    Work at office
    3 days per week

    Cognizant Technology Solutions Corporation

    Winnipeg, MB
    3 hours ago
  •  ...Canada’s largest Indigenous-owned software testing and technology services company. For...  ...We’re looking for experienced Test Automation Specialists who are passionate about modern engineering practices and excited about how AI is changing the way software is tested... 
    Internship

    Plato

    Winnipeg, MB
    3 hours ago
  • Become a key part of our team as an AI Quality Engineer, focusing on automated test development and quality assurance. This role combines Python expertise with a deep understanding of AI processes. With a minimum of 4 years in QA or test automation, you'll build and maintain... 

    Centraprise

    Winnipeg, MB
    3 hours ago
  • Excel as a Senior Test Automation Engineer at PureFacts, specializing in AI-driven testing methodologies in Toronto. Transform software quality and efficiency with AI technologies. In this role, you will join the Quality Engineering team to innovate and optimize testing strategies... 

    PureFacts Financial Solutions

    Winnipeg, MB
    3 hours ago
  •  ...Jonas Software in Canada is seeking an Automation Engineer to advance CORA Loyalty’s quality engineering and software delivery through scalable test automation, AI-enabled quality practices, and robust testing frameworks. You will collaborate with Software Engineering, Product... 
    Remote work

    Jonas Software

    Winnipeg, MB
    3 hours ago
  •  ...lead the development and maintenance of test automation frameworks and scripts for Guidewire and...  ...and quality status to stakeholders. Evaluate and integrate new testing tools and technologies...  .... Mentor and guide junior automation engineers. Conduct code reviews for automated... 
    Full time
    Work at office

    RED RIVER MUTUAL

    Winnipeg, MB
    3 hours ago
  • AI-driven test automation defines this Test Automation Specialist role at PLATO, focused on scalable frameworks, modern engineering, and practical AI applications.PLATO seeks experienced Test Automation Specialists with five-plus years in testing, automation, development, or... 

    PLATO

    Winnipeg, MB
    3 hours ago
  •  ...• Our convenient Price Café offers free Tea and Coffee. Test Automation Lead Enterprise Systems • IFS ERP • Multi-site Manufacturing...  ...we need • Bachelor's degree in Computer Science, Software Engineering, Information Technology, Computer Engineering, or a related technical... 

    Price Industries Limited

    Winnipeg, MB
    7 days ago
  • Join Cognizant as an AI Testing Automation Lead and play a crucial role in developing quality engineering solutions for AI-powered applications in a hybrid setting. This position focuses on CI/CD integrations and robust testing strategies. You'll be responsible for leading test... 

    Cognizant Technology Solutions Corporation

    Winnipeg, MB
    3 hours ago
  •  ...focuses on defining quality strategies and providing mentorship in QA practices. Candidates should possess strong technical skills in testing methodologies, experience in CI/CD, and excellent communication abilities. The position offers a full-time employment contract with... 
    Full time
    Contract work

    TMC Canada

    Winnipeg, MB
    3 hours ago
  •  ...Ontinue, a leading AI-powered MXDR provider, is hiring a QA Automation Engineer in Canada to scale quality across a fast-moving engineering team. You’ll design automated tests for frontend, backend, and API flows using Playwright in TypeScript or C#, and coordinate release... 

    Ontinue

    Winnipeg, MB
    3 hours ago
  • Upgrade is seeking a Senior QA Automation Developer to own the end-to-end quality of a new product. You will build automated tests and perform functional testing to ensure a seamless banking...  ...using Java, Selenium/Playwright, and AI-assisted workflows. You will also deploy disposable... 

    Upgrade

    Winnipeg, MB
    3 hours ago
  • 360training is seeking a QA Automation Engineer to develop and maintain automated testing across our software development and deployment processes. You will work with...  ...improving release speed and quality. You will leverage AI-assisted testing tools to enhance coverage and... 
    Remote work

    360training.com

    Winnipeg, MB
    3 hours ago
  • A leading technology service provider is seeking a QA Automation Engineer in Calgary, Alberta, responsible for ensuring high-quality software delivery...  ...through designing, executing, and maintaining effective test strategies and automation frameworks. The ideal candidate will... 

    Compunnel, Inc.

    Winnipeg, MB
    3 hours ago
  •  ...analytics software company in Toronto is seeking a Quality Assurance Automation Engineer to ensure the delivery of high-quality SaaS products. As part...  ...the Product Development team, you will develop and maintain test automation frameworks, conduct various tests, and collaborate... 
    Full time

    Askuity

    Winnipeg, MB
    3 hours ago
  • Build reliable AI product testing as a Senior Software Developer in Test with Vena, working remotely from Canada on automated quality solutions.This full-time senior role supports Vena’s...  ...testing, automation framework, and quality engineering expertise to Vena’s remote Canadian... 
    Full time
    Remote work

    Vena Solutions Inc.

    Winnipeg, MB
    3 hours ago
  • Build reliable autonomous AI at Equifax through LLM evaluation, Python automation, and multi-agent workflow testing. This newly created position focuses on production reliability.As an Agentic AI Evaluation & Tuning Engineer, you will connect LLM capabilities with dependable... 

    Equifax Inc.

    Winnipeg, MB
    3 hours ago
  • AI test automation and software quality define this hands-on specialist role at PLATO, supporting modern engineering practices across Canada’s 100% Canada-based team.PLATO seeks experienced Test Automation Specialists with five or more years in testing, automation, development... 

    Plato

    Winnipeg, MB
    3 hours ago
  • Engineer Python automation for EA-SPORTS FC quality verification at Electronic Arts, integrating CI/CD diagnostics...  ...III, you will build tools that expand test coverage and detect defects earlier. You...  .... The role also uses NLP, generative AI, and LLM capabilities.Key... 

    Electronic Arts

    Winnipeg, MB
    3 hours ago
  • Senior Quality Engineer – Test Automation, AI Testing & Java/PythonQuality Engineer– Test Automation, Java/Python & GenAI Roles and Responsibilities: Partner with DevOps,Development, Product, andQE teams todesign and executeexploratory, automated,and AI-focusedtests that... 

    Astra-North Infoteck Inc.

    Winnipeg, MB
    3 hours ago
  • $86k - $143k per year

    Role SummaryThe Senior 1 AI Engineer independently designs, builds, and operates complex AI automation for Security Services. You will own workflows from requirements through...  ...APIs and event-driven interfaces. Define evaluation criteria; test factual accuracy, missed... 
    Internship
    Remote work
    Flexible hours

    Sophos

    Winnipeg, MB
    3 hours ago
  • $95k - $125k per year

    AI Automation EngineerOsmow's | Digital & Technology Team Reports to: Director of DigitalLocation...  .... Monitoring, reconciliation, testing and appropriate controls to support reliable...  ...of Digital. Build, prompt, test and evaluate LLM-powered workflows and agents.... 
    Full time
    Work at office
    Remote work
    3 days per week

    Osmow's

    Winnipeg, MB
    3 hours ago
  • Improve automated test feedback through Continuous Integration, repeatable tests, and diagnostics...  ...in hybrid Burnaby, BC.As a Software Engineer II, you will integrate mobile and web automation...  ...failures, assess reliability, and evaluate AI-assisted approaches across the Companion... 

    Electronic Arts

    Winnipeg, MB
    3 hours ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Ai Evaluation Test Automation Engineer. Be the first to apply!