Senior AI Agent Test Automation Engineer
Nexxa
Own quality for Nexxa AI agents through Python automation and exploratory testing across industrial systems and operational scenarios.
As a Senior or Staff SDET, you will build production-grade evaluation harnesses, simulators, regression suites, and CI/CD quality gates. You will test LLM-based agents across planning, tool selection, execution, recovery, and final outputs. The work combines API, contract, integration, end-to-end, performance, and reliability testing with manual trace review, red-teaming, and release verification.
Key Responsibilities:
- Build evaluation harnesses and regression suites for LLM-based agents
- Integrate automated evaluations into CI/CD quality gates
- Develop simulators, mocks, and sandbox environments for industrial tools
- Review traces, outputs, and agent behavior manually
- Turn failures and attacks into reproducible automated tests
Requirements:
- Five-plus years in SDET, QA, or software engineering
- Strong Python programming and software engineering practices
- Manual and automated testing experience for distributed systems
- Experience with CI/CD, Docker, and Kubernetes
- Experience testing APIs, microservices, and LLM-based products
Apply your automation, evaluation, and exploratory testing expertise to Nexxa’s high-stakes industrial AI systems.
#J-18808-LjbffrVacancy posted 13 hours ago
Similar jobs that could be interesting for youBased on the Senior AI Agent Test Automation Engineer in Toronto, ON vacancy
$118k - $162k per year
...Secure Every Identity, from AI to Human Identity is the key to unlocking the potential of AI. Okta secures AI by... ...revoked, or hand an auditor evidence that isn't true. The Senior Software Development Engineer in Test Opportunity As a Senior Software Development Engineer...SeniorFull timeLocal areaWorldwideOkta
Toronto, ON22 days ago$110k - $140k per year
...business challenges through intelligent automation, agentic AI, and custom technology solutions. Our... ...deep consulting expertise with hands-on engineering to bridge the gap between strategy and... ...software in two-week sprints. As AI agent-based automation grows alongside our RPA...SeniorFull timeWork at officeGreen Light Consulting
Toronto, ONmore than 2 months ago- ...About Nexxa Nexxa is building the best AI systems for heavy industries, enabling machines, systems... .... Role Overview We’re looking for a Senior / Staff SDET to own quality for Nexxa’s AI agent systems, combining deep test automation with hands-on manual and...SeniorFull timeContract workInternshipShift work
Nexxa
Toronto, ON2 days ago $150k - $200k per year
...Stikeman Elliott is building its own agentic AI platform, with production agents working alongside lawyers on the firm’s matters. Built on the Claude... ...in daily use and continues to expand. This is the senior engineering role responsible for advancing its development. You...SeniorFull timeStikeman Elliott Llp
Toronto, ON8 days ago- ...digital partner that combines Strategy, Experience & Design, Engineering and Managed Services. We build digital solutions that... ...ambition. Practical action. Endless possibilities. As a Senior Data Engineer, AI & Agents, you will prepare enterprise domain data for consumption...SeniorFull timeContract work
Appnovation Technologies
Toronto, ONa month ago $85k - $95k per year
...experience in mobile functional testing across both iOS and Android... ..., along with hands -on automation expertise using Appium, XCUITest... ...degree in Computer Science, Engineering, or related field (or equivalent... ...is removed. We do not use AI technology; all applications are...Permanent employmentFull timeWork at officeLocal areaInfoya
Toronto, ONmore than 2 months ago- ...Actual Impact. At Docebo, we’re using AI to change how people learn at work—... ...driver's seat of the high-octane engine that powers Docebo’s enterprise automation platform! In this role, you will... ...Observability Guru: Deep understanding of testing strategies (unit, integration, E2E)...SeniorLong term contractFull timeFor contractorsWork at officeWorldwide3 days per week
Docebo
Toronto, ON22 days ago - ...CoreFactor is searching for a Senior AI Engineer on a permanent/full-time basis. This position is hybrid and will require the successful... ...'ll take technical ownership of intelligent applications and automation tools that use LLMs to solve business problems, set best practices...SeniorPermanent employmentFull timeWork at office
CoreFactor
Toronto, ON23 days ago $118k - $162k per year
...Secure Every Identity, from AI to Human Identity is the key to unlocking the potential of AI. Okta secures AI by building... ...you are too, let's talk. Okta is seeking an experienced Senior Software Engineer in Test for our UI Engineering team. As a senior member of product...SeniorLocal areaWorldwideOkta
Toronto, ON20 days ago$110k - $130k per year
...design, build, ship, and scale the AI-enabled workflows and agents that turn AlayaCare’s GTM motion into... ...Own a prioritized backlog of AI and automation opportunities: define requirements,... ...configuration. Working knowledge of prompt engineering, agent-based workflows, AI...SeniorFull timeInternshipWork at officeWork from homeFlexible hoursAlayacare
Toronto, ONmore than 2 months ago- ...Position: Senior AI Engineer Location: Toronto, ON Job ID#: RQ11252 Duration: 8 Months Role Overview We are seeking... ...including conversational IVR systems, virtual assistants, and AI agents. The ideal candidate will bring strong experience in cloud-...SeniorFull time
Symbiotic Digital
Toronto, ONmore than 2 months ago $137.6k - $172k per year
...customers build the skills they need for a world being reshaped by AI. Read more about the combined company by visiting our blog... ...make a difference at global scale. Job Overview: As a Senior Software Test Engineer (Test Infrastructure and Tooling) at Coursera, you’ll play a...Coursera
Toronto, ONa month ago$85k - $90k per year
...About the Job: The QA Engineer is responsible for end-to-end test coverage across manual and AI-driven automated testing. This role requires the ability to design and execute traditional manual test cases as well as build and prompt AI agents to autonomously generate and execute...Permanent employmentFull timeWork at officeLocal areaNight shiftWeekend workInfoya
Toronto, ONmore than 2 months ago$140k - $160k per year
...you to join us! About the job - Applied AI Engineer Join the Engineering Team, where you'... ...agentic future. You'll work alongside senior engineers, Product, and our Chief Product... ...orchestration patterns (dynamic routing, tool-using agents, feedback loops) - contributing to the...SeniorFull timeManual laborWork at officeWorldwide1 day per weekContactmonkey
Toronto, ON14 days ago- ...There’s a wide gap between an agent that works in a demo and one that... ...and customers who depend on AI — risk, fraud, collections, payments... ...for agentic systems; mentor engineers and partner with ML, product, and... .../ edge inference. Testing and verification for non-deterministic...SeniorFull time
Paytm
Toronto, ONmore than 2 months ago - ...Waabi, founded by AI visionary Raquel Urtasun, is the leader in Physical AI. With a world-class team, we're... ...maximizing productivity. We are looking for hands-on senior software engineers to build the AI tooling, agents, and workflows that accelerate how we work. As a Software...SeniorFull timeWork at officeWork from homeFlexible hours
Waabi
Toronto, ON16 days ago $167.5k - $235k per year
...autonomous cross-functional team to build AI agents that fundamentally transform how... ...which currently consists of fewer than five engineers. The work spans critical areas including... ...knowledge infrastructure. We are seeking a Senior Software Engineer with a strong passion for...SeniorLong term contractFull timeCasual workInternshipWork at officeLocal areaImmediate startJustworks
Toronto, ONmore than 2 months ago$135k - $210k per year
...Guidepoint seeks an experienced Data/AI Engineer as an integral member of the Toronto-based... ...language model (LLM) applications and research agents, while ensuring the integration of responsible AI and best-in-class MLOps. The Senior AI/ML Engineer will be a primary contributor...SeniorFull timeWorldwideGuidepoint
Toronto, ONmore than 2 months ago- ...Future of Competitive Intelligence Klue Engineering is hiring! We're looking for a Senior Software Engineer to join our team in Toronto... ...and optimizing state-of-the-art LLM-powered agents that can reason, plan and automate workflows for users. You will be leading the...SeniorFull timeWork at office
Klue Inc.
Toronto, ONa month ago - ...branches across Canada. They are seeking a senior D365 developer. The opportunity offers a... ...\\ We seek a skilled and experienced QA Automation Tester. This role requires an understanding of... ...\\\\ \ \\ § \\ Develop automated test scripts using automated tools such as...SeniorFull time
Yoush Consulting
Toronto, ONmore than 2 months ago $120k - $160k per year
...real outcomes. The Opportunity ShyftLabs is looking for a Senior AI Engineer to sit at the intersection of AI/ML engineering and backend... ...iteration Integrate LLM APIs, ML models, and intelligent automation pipelines into scalable, production-grade backend systems...SeniorFull timeWork at office3 days per weekShyftlabs
Toronto, ONmore than 2 months ago- ...Position: Automation Testing Specialist Location: Toronto, ON (Onsite) Job ID#: RQ11340 Duration: 12 Months Role Overview... ...Responsibilities Provide guidance on testing strategies, test plans, testing tool selection, and resource requirements. Plan...Full time
Symbiotic Digital
Toronto, ONmore than 2 months ago - ...Job Description Title:Test Automation Developer Location: Yonge Street Toronto,ON Duration: 6 Months Job Description: Automation developer with 5 years of HP UFT/QTP development experience Project methodology: Scrum Agile with 3 week sprints Location...Full timeContract work
Computech Cv, Llc
Toronto, ONmore than 2 months ago $75k - $95k per year
...We're hiring Associate Software Engineers in Test to work at Veeva, working on various products... ...you'll be responsible for functional and automation testing of software, ensuring customer... ...A final conversation with our group's Senior Leader. ~ Once all interviews are...SeniorFull timeInternshipWork at officeLocal areaWork from homeRelocationVeeva Systems
Toronto, ON19 days ago$100k - $130k per year
...results. That’s why we've integrated engineering and construction, seeing our projects... ...Stadium. Why This Role Matters As Senior Building Automation Specialist, you'll play a key technical... ...We may use artificial intelligence (AI) tools to support parts of the hiring...SeniorLong term contractFor contractorsFor subcontractorEcosystem Corporation
Toronto, ON2 days ago- ...are a growing fintech organization looking to add a Senior Software Development Engineer in Test! What you’ll be responsible for in this role: Leading... ...procedures Designing, developing, and executing automation code. Identifying, recording, documenting, and...SeniorFull time
Episode Six
Toronto, ONmore than 2 months ago - ...usher in this new era, we seek AI-native thinkers across every... ...experimental mindset — who rapidly test emerging capabilities to... ...part of that mission. Our AI/ML engineering teams are delivering the intelligence... ...(RAG), and AI-driven automation Write clean, performant, and...SeniorFull timeWork at officeLocal areaWorldwide
Snowflake
Toronto, ON9 days ago $150k - $190k per year
...you’ve found your next home. ROLE OVERVIEW The engineering team is looking for an AI Engineer who can build the LLM and agentic... ...ownership of these features from API design and prompt/agent design through testing, deployment, and monitoring in production Collaborate...Long term contractFull timeFor contractorsInternshipWork at officeLocal areaSleeping nightsQuartermaster
Toronto, ON14 days ago$70.2 - $84.24 per hour
...RQ00822 - Systems Testing/QA Specialist - Senior 6 Months contract (282 Business Days) Hybrid - Metro Hall... ...of 3 years of solid experience in AI-assisted testing with hands-on experience in developing AI-powered QA agents using agentic AI, LLM technologies and chatbot...SeniorFull timeContract workSource Code
Toronto, ON11 days ago$140k - $180k per year
...Description Senior DevOps Engineer, Agentic Automation & Live Games Toronto, ON Hybrid, 3 days per week in office Full-time Department: Engineering... ...on stronger infrastructure, better automation, practical AI adoption, improved reliability, stronger security...SeniorLong term contractFull timeWork at officeLocal area3 days per weekBig Viking Games
Toronto, ONa month ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Senior AI Agent Test Automation Engineer. Be the first to apply!
Related searches
- security agent Toronto, ON
- quality control representative Toronto, ON
- de représentant Toronto, ON
- agent méthode Toronto, ON
- représentant représentante en assurances Toronto, ON
- agent de développement Toronto, ON
- vendeur représentant Toronto, ON
- agent de recherche Toronto, ON
- représentant développement des affaires Toronto, ON
- representant cosmetique Toronto, ON