Ai Evaluation Test Automation Engineer
CrowdStrike Inc.
Shape trusted AI evaluation at CrowdStrike using agent frameworks, LLM APIs, and verification-first test automation. Work on cloud-based cybersecurity systems.As a Software Development Engineer in Test, you will help determine how CrowdStrike AI agents are tested, evaluated, and trusted in production. The position combines test automation with eval-driven development across prompts, models, tools, and orchestration. You will support functional, integration, performance, and end-to-end validation.Key Responsibilities:
Maintain golden datasets and scoring pipelines for AI agentsMeasure correctness, tool use, safety, latency, and costFeed production failures into evaluation setsBuild AI-powered workflows for test data and triageReview agent designs for testability and evaluabilityRequirements:
Computer Science degree or equivalent experienceKnowledge of object-oriented programming and distributed systemsPython preferred for evaluation and agent workExperience testing REST, gRPC, and JSON servicesVerification-first judgment about probabilistic AI outputUse AI evaluation, test automation, Python, and distributed-systems skills to improve trusted production testing at CrowdStrike.
#J-18808-Ljbffr
Maintain golden datasets and scoring pipelines for AI agentsMeasure correctness, tool use, safety, latency, and costFeed production failures into evaluation setsBuild AI-powered workflows for test data and triageReview agent designs for testability and evaluabilityRequirements:
Computer Science degree or equivalent experienceKnowledge of object-oriented programming and distributed systemsPython preferred for evaluation and agent workExperience testing REST, gRPC, and JSON servicesVerification-first judgment about probabilistic AI outputUse AI evaluation, test automation, Python, and distributed-systems skills to improve trusted production testing at CrowdStrike.
#J-18808-Ljbffr
Vacancy posted 46 minutes ago
Similar jobs that could be interesting for youBased on the Ai Evaluation Test Automation Engineer in Winnipeg, MB vacancy
- Build AI test automation at CrowdStrike as a Software Development Engineer in Test, working remotely with Python and distributed systems.This hands-on role supports the Detonations and Threat Analysis Platform Engineering team. You will develop functional, integration, and performance...SuggestedRemote work
- ...About the role We are seeking a highly skilled and self-motivated Senior AI Test Automation Engineer to join our Quality Engineering team. The ideal candidate will have extensive experience in test automation, database validation, AI-assisted testing, and modern quality...Suggested
- ...Rootly is hiring an AI Quality Engineer to own the evaluation and optimization of its agentic AI features. You will design test scenarios, probe how AI agents reason, and work with engineering to close the loop on performance across incident workflows. You will partner...Suggested
- Build AI test automation with Java or Python as a Senior Quality Engineer focused on prompt behavior, accuracy, and reliable releases.This senior position requires five-plus years in quality engineering, quality assurance, test automation, or software development. You will...Suggested
- Build smarter Playwright tests with GitHub Copilot, CodeGen, and MCP tools in Veriday Inc.'s remote Senior Automation Engineer contract.This six-month Canadian engagement, potentially... ...CI/CD pipelines. The role also evaluates AI tooling for test plans and work items...SuggestedContract workRemote work
- Lead AI chatbot validation through Java, Python, and scalable automation frameworks while partnering with DevOps teams on CI/CD quality... ...checks.This Senior Quality Engineer / SDET role supports DevOps... ...exploratory, automated, and AI-focused testing. You will define test...
- Shape trusted AI evaluation with performance testing and CI/CD expertise as a Software Development Engineer in Test at CrowdStrike. Work on scalable cybersecurity services.As an SDET, you will help define how AI agents are tested, evaluated, and trusted in production. CrowdStrike...
- ...Canada’s largest Indigenous-owned software testing and technology services company. For... ...We’re looking for experienced Test Automation Specialists who are passionate about modern engineering practices and excited about how AI is changing the way software is tested...Internship
$90k - $117k per year
As an Automation Test Lead, you will lead AI-powered Quality Engineering and test automation solutions, including developing testing strategies and frameworks, integrating testing into CI/CD pipelines, validating AI behavior and integrations, and promoting responsible AI practices...Work at office3 days per week- Excel as a Senior Test Automation Engineer at PureFacts, specializing in AI-driven testing methodologies in Toronto. Transform software quality and efficiency with AI technologies. In this role, you will join the Quality Engineering team to innovate and optimize testing strategies...
- Become a key part of our team as an AI Quality Engineer, focusing on automated test development and quality assurance. This role combines Python expertise with a deep understanding of AI processes. With a minimum of 4 years in QA or test automation, you'll build and maintain...
- ...Jonas Software in Canada is seeking an Automation Engineer to advance CORA Loyalty’s quality engineering and software delivery through scalable test automation, AI-enabled quality practices, and robust testing frameworks. You will collaborate with Software Engineering, Product...Remote work
- AI-driven test automation defines this Test Automation Specialist role at PLATO, focused on scalable frameworks, modern engineering, and practical AI applications.PLATO seeks experienced Test Automation Specialists with five-plus years in testing, automation, development, or...
- ...lead the development and maintenance of test automation frameworks and scripts for Guidewire and... ...and quality status to stakeholders. Evaluate and integrate new testing tools and technologies... .... Mentor and guide junior automation engineers. Conduct code reviews for automated...Full timeWork at office
- ...• Our convenient Price Café offers free Tea and Coffee. Test Automation Lead Enterprise Systems • IFS ERP • Multi-site Manufacturing... ...we need • Bachelor's degree in Computer Science, Software Engineering, Information Technology, Computer Engineering, or a related technical...
- A leading technology service provider is seeking a QA Automation Engineer in Calgary, Alberta, responsible for ensuring high-quality software delivery... ...through designing, executing, and maintaining effective test strategies and automation frameworks. The ideal candidate will...
- Upgrade is seeking a Senior QA Automation Developer to own the end-to-end quality of a new product. You will build automated tests and perform functional testing to ensure a seamless banking... ...using Java, Selenium/Playwright, and AI-assisted workflows. You will also deploy disposable...
- Join Cognizant as an AI Testing Automation Lead and play a crucial role in developing quality engineering solutions for AI-powered applications in a hybrid setting. This position focuses on CI/CD integrations and robust testing strategies. You'll be responsible for leading test...
- ...analytics software company in Toronto is seeking a Quality Assurance Automation Engineer to ensure the delivery of high-quality SaaS products. As part... ...the Product Development team, you will develop and maintain test automation frameworks, conduct various tests, and collaborate...Full time
- ...Ontinue, a leading AI-powered MXDR provider, is hiring a QA Automation Engineer in Canada to scale quality across a fast-moving engineering team. You’ll design automated tests for frontend, backend, and API flows using Playwright in TypeScript or C#, and coordinate release...
- ...focuses on defining quality strategies and providing mentorship in QA practices. Candidates should possess strong technical skills in testing methodologies, experience in CI/CD, and excellent communication abilities. The position offers a full-time employment contract with...Full timeContract work
- 360training is seeking a QA Automation Engineer to develop and maintain automated testing across our software development and deployment processes. You will work with... ...improving release speed and quality. You will leverage AI-assisted testing tools to enhance coverage and...Remote work
- Build reliable autonomous AI at Equifax through LLM evaluation, Python automation, and multi-agent workflow testing. This newly created position focuses on production reliability.As an Agentic AI Evaluation & Tuning Engineer, you will connect LLM capabilities with dependable...
- Build reliable AI product testing as a Senior Software Developer in Test with Vena, working remotely from Canada on automated quality solutions.This full-time senior role supports Vena’s... ...testing, automation framework, and quality engineering expertise to Vena’s remote Canadian...Full timeRemote work
- AI test automation and software quality define this hands-on specialist role at PLATO, supporting modern engineering practices across Canada’s 100% Canada-based team.PLATO seeks experienced Test Automation Specialists with five or more years in testing, automation, development...
- Engineer Python automation for EA-SPORTS FC quality verification at Electronic Arts, integrating CI/CD diagnostics... ...III, you will build tools that expand test coverage and detect defects earlier. You... .... The role also uses NLP, generative AI, and LLM capabilities.Key...
- O2E Brands in Vancouver is seeking a Test Automation Architect to design, scale, and maintain automated testing frameworks, incorporating AI to optimize test generation and self-healing scripts. This role integrates intelligent pipelines into CI/CD to accelerate time-to-market...
- Senior Quality Engineer – Test Automation, AI Testing & Java/PythonQuality Engineer– Test Automation, Java/Python & GenAI Roles and Responsibilities: Partner with DevOps,Development, Product, andQE teams todesign and executeexploratory, automated,and AI-focusedtests that...
$86k - $143k per year
...yearRole Summary Design, build, and operate AI automation for security services, taking workflows... ...secure APIs and event-driven interfaces.Evaluate factual accuracy, missed threats, false... ...approval for high-impact actions; test resistance to prompt injection.Build and...Full timeRemote work- Improve automated test feedback through Continuous Integration, repeatable tests, and diagnostics... ...in hybrid Burnaby, BC.As a Software Engineer II, you will integrate mobile and web automation... ...failures, assess reliability, and evaluate AI-assisted approaches across the Companion...
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Ai Evaluation Test Automation Engineer. Be the first to apply!
Related searches
- test engineer Winnipeg, MB
- rf test engineer Winnipeg, MB
- test automation developer Winnipeg, MB
- test automation engineer Winnipeg, MB
- drive test engineer Winnipeg, MB
- field test engineer Winnipeg, MB
- test automation specialist Winnipeg, MB
- automation application engineer Winnipeg, MB
- control and automation engineer Winnipeg, MB
- electrical automation engineer Winnipeg, MB

