Ai Evaluation Test Automation Engineer
CrowdStrike Inc.
Shape trusted AI evaluation at CrowdStrike using agent frameworks, LLM APIs, and verification-first test automation. Work on cloud-based cybersecurity systems.As a Software Development Engineer in Test, you will help determine how CrowdStrike AI agents are tested, evaluated, and trusted in production. The position combines test automation with eval-driven development across prompts, models, tools, and orchestration. You will support functional, integration, performance, and end-to-end validation.Key Responsibilities:
Maintain golden datasets and scoring pipelines for AI agentsMeasure correctness, tool use, safety, latency, and costFeed production failures into evaluation setsBuild AI-powered workflows for test data and triageReview agent designs for testability and evaluabilityRequirements:
Computer Science degree or equivalent experienceKnowledge of object-oriented programming and distributed systemsPython preferred for evaluation and agent workExperience testing REST, gRPC, and JSON servicesVerification-first judgment about probabilistic AI outputUse AI evaluation, test automation, Python, and distributed-systems skills to improve trusted production testing at CrowdStrike.
#J-18808-Ljbffr
Maintain golden datasets and scoring pipelines for AI agentsMeasure correctness, tool use, safety, latency, and costFeed production failures into evaluation setsBuild AI-powered workflows for test data and triageReview agent designs for testability and evaluabilityRequirements:
Computer Science degree or equivalent experienceKnowledge of object-oriented programming and distributed systemsPython preferred for evaluation and agent workExperience testing REST, gRPC, and JSON servicesVerification-first judgment about probabilistic AI outputUse AI evaluation, test automation, Python, and distributed-systems skills to improve trusted production testing at CrowdStrike.
#J-18808-Ljbffr
Vacancy posted 39 minutes ago
Similar jobs that could be interesting for youBased on the Ai Evaluation Test Automation Engineer in Winnipeg, MB vacancy
- Build AI test automation at CrowdStrike as a Software Development Engineer in Test, working remotely with Python and distributed systems.This hands-on role supports the Detonations and Threat Analysis Platform Engineering team. You will develop functional, integration, and performance...SuggestedRemote work
- ...About the role We are seeking a highly skilled and self-motivated Senior AI Test Automation Engineer to join our Quality Engineering team. The ideal candidate will have extensive experience in test automation, database validation, AI-assisted testing, and modern quality...Suggested
- ...Rootly is hiring an AI Quality Engineer to own the evaluation and optimization of its agentic AI features. You will design test scenarios, probe how AI agents reason, and work with engineering to close the loop on performance across incident workflows. You will partner...Suggested
- Build AI test automation with Java or Python as a Senior Quality Engineer focused on prompt behavior, accuracy, and reliable releases.This senior position requires five-plus years in quality engineering, quality assurance, test automation, or software development. You will...Suggested
- Lead AI chatbot validation through Java, Python, and scalable automation frameworks while partnering with DevOps teams on CI/CD quality... ...checks.This Senior Quality Engineer / SDET role supports DevOps... ...exploratory, automated, and AI-focused testing. You will define test...Suggested
- Build smarter Playwright tests with GitHub Copilot, CodeGen, and MCP tools in Veriday Inc.'s remote Senior Automation Engineer contract.This six-month Canadian engagement, potentially... ...CI/CD pipelines. The role also evaluates AI tooling for test plans and work items...Contract workRemote work
- Shape trusted AI evaluation with performance testing and CI/CD expertise as a Software Development Engineer in Test at CrowdStrike. Work on scalable cybersecurity services.As an SDET, you will help define how AI agents are tested, evaluated, and trusted in production. CrowdStrike...
- ...Jonas Software in Canada is seeking an Automation Engineer to advance CORA Loyalty’s quality engineering and software delivery through scalable test automation, AI-enabled quality practices, and robust testing frameworks. You will collaborate with Software Engineering, Product...Remote work
$90k - $117k per year
As an Automation Test Lead, you will lead AI-powered Quality Engineering and test automation solutions, including developing testing strategies and frameworks, integrating testing into CI/CD pipelines, validating AI behavior and integrations, and promoting responsible AI practices...Work at office3 days per week- ...Canada's largest Indigenous-owned software testing and technology services company. For... ...'re looking for experienced Test Automation Specialists who are passionate about modern engineering practices and excited about how AI is changing the way software is tested...Internship
- Become a key part of our team as an AI Quality Engineer, focusing on automated test development and quality assurance. This role combines Python expertise with a deep understanding of AI processes. With a minimum of 4 years in QA or test automation, you'll build and maintain...
- Excel as a Senior Test Automation Engineer at PureFacts, specializing in AI-driven testing methodologies in Toronto. Transform software quality and efficiency with AI technologies. In this role, you will join the Quality Engineering team to innovate and optimize testing strategies...
- ...lead the development and maintenance of test automation frameworks and scripts for Guidewire and... ...and quality status to stakeholders. Evaluate and integrate new testing tools and technologies... .... Mentor and guide junior automation engineers. Conduct code reviews for automated...Full timeWork at office
- AI-driven test automation defines this Test Automation Specialist role at PLATO, focused on scalable frameworks, modern engineering, and practical AI applications.PLATO seeks experienced Test Automation Specialists with five-plus years in testing, automation, development, or...
- ...• Our convenient Price Café offers free Tea and Coffee. Test Automation Lead Enterprise Systems • IFS ERP • Multi-site Manufacturing... ...we need • Bachelor's degree in Computer Science, Software Engineering, Information Technology, Computer Engineering, or a related technical...
- ...analytics software company in Toronto is seeking a Quality Assurance Automation Engineer to ensure the delivery of high-quality SaaS products. As part... ...the Product Development team, you will develop and maintain test automation frameworks, conduct various tests, and collaborate...Full time
- A leading technology service provider is seeking a QA Automation Engineer in Calgary, Alberta, responsible for ensuring high-quality software delivery... ...through designing, executing, and maintaining effective test strategies and automation frameworks. The ideal candidate will...
- ...Ontinue, a leading AI-powered MXDR provider, is hiring a QA Automation Engineer in Canada to scale quality across a fast-moving engineering team. You’ll design automated tests for frontend, backend, and API flows using Playwright in TypeScript or C#, and coordinate release...
- Join Cognizant as an AI Testing Automation Lead and play a crucial role in developing quality engineering solutions for AI-powered applications in a hybrid setting. This position focuses on CI/CD integrations and robust testing strategies. You'll be responsible for leading test...
- 360training is seeking a QA Automation Engineer to develop and maintain automated testing across our software development and deployment processes. You will work with... ...improving release speed and quality. You will leverage AI-assisted testing tools to enhance coverage and...Remote work
- ...focuses on defining quality strategies and providing mentorship in QA practices. Candidates should possess strong technical skills in testing methodologies, experience in CI/CD, and excellent communication abilities. The position offers a full-time employment contract with...Full timeContract work
- Upgrade is seeking a Senior QA Automation Developer to own the end-to-end quality of a new product. You will build automated tests and perform functional testing to ensure a seamless banking... ...using Java, Selenium/Playwright, and AI-assisted workflows. You will also deploy disposable...
- Build reliable AI product testing as a Senior Software Developer in Test with Vena, working remotely from Canada on automated quality solutions.This full-time senior role supports Vena’s... ...testing, automation framework, and quality engineering expertise to Vena’s remote Canadian...Full timeRemote work
- Build reliable autonomous AI at Equifax through LLM evaluation, Python automation, and multi-agent workflow testing. This newly created position focuses on production reliability.As an Agentic AI Evaluation & Tuning Engineer, you will connect LLM capabilities with dependable...
- AI test automation and software quality define this hands-on specialist role at PLATO, supporting modern engineering practices across Canada’s 100% Canada-based team.PLATO seeks experienced Test Automation Specialists with five or more years in testing, automation, development...
- Engineer Python automation for EA-SPORTS FC quality verification at Electronic Arts, integrating CI/CD diagnostics... ...III, you will build tools that expand test coverage and detect defects earlier. You... .... The role also uses NLP, generative AI, and LLM capabilities.Key...
$86k - $143k per year
Role SummaryThe Senior 1 AI Engineer independently designs, builds, and operates complex AI automation for Security Services. You will own workflows from requirements through... ...APIs and event-driven interfaces. Define evaluation criteria; test factual accuracy, missed...InternshipRemote workFlexible hours- Build network test setups in Calgary with Python, PowerShell, and traffic-generation skills... ...produce accurate reports.This Network Engineer position focuses on datacenter lab... ...experiencePowerShell basics and network automation knowledgeFour-plus years deploying network...
- Build automated testing with Selenium, Parasoft, and Robot Framework while delivering in-sprint quality within Agile development teams.This Test Automation Engineer develops, maintains, and executes automated tests across platforms and technologies. The work covers database...
$95k - $125k per year
AI Automation EngineerOsmow's | Digital & Technology Team Reports to: Director of DigitalLocation... .... Monitoring, reconciliation, testing and appropriate controls to support reliable... ...of Digital. Build, prompt, test and evaluate LLM-powered workflows and agents....Full timeWork at officeRemote work3 days per week
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Ai Evaluation Test Automation Engineer. Be the first to apply!
Related searches
- test engineer Winnipeg, MB
- rf test engineer Winnipeg, MB
- test automation developer Winnipeg, MB
- test automation engineer Winnipeg, MB
- drive test engineer Winnipeg, MB
- field test engineer Winnipeg, MB
- test automation specialist Winnipeg, MB
- automation application engineer Winnipeg, MB
- control and automation engineer Winnipeg, MB
- electrical automation engineer Winnipeg, MB

