Ai Agent Reliability Engineer
TribalScale
Improve AI agent reliability at TribalScale with evaluation pipelines, quality metrics, and disciplined testing for agentic systems. The location remains to be confirmed.This Agentic QA Engineer role focuses on validating AI-driven workflows in real-world environments. You will design evaluation datasets, test scenarios, and benchmarks for dynamic application outputs. You will also collaborate with AI engineers, developers, product teams, and client-facing engineering teams.Key Responsibilities:
Develop pipelines measuring agent accuracy and task completionEstablish quality metrics, datasets, scenarios, and benchmarksEvaluate edge cases, recovery, security, and resilienceIntegrate quality checks into CI/CD pipelinesInvestigate production issues and improve test coverageRequirements:
Strong analytical thinking for complex real-world test casesExperience creating automated tests and testing frameworksStrong API, integration, regression, and end-to-end testing knowledgeExperience debugging applications and technical issuesFamiliarity with modern software development practicesBring quality engineering expertise to AI agent evaluation, production reliability, and measurable software performance at TribalScale.
#J-18808-Ljbffr
Develop pipelines measuring agent accuracy and task completionEstablish quality metrics, datasets, scenarios, and benchmarksEvaluate edge cases, recovery, security, and resilienceIntegrate quality checks into CI/CD pipelinesInvestigate production issues and improve test coverageRequirements:
Strong analytical thinking for complex real-world test casesExperience creating automated tests and testing frameworksStrong API, integration, regression, and end-to-end testing knowledgeExperience debugging applications and technical issuesFamiliarity with modern software development practicesBring quality engineering expertise to AI agent evaluation, production reliability, and measurable software performance at TribalScale.
#J-18808-Ljbffr
Vacancy posted 2 hours ago
Similar jobs that could be interesting for youBased on the Ai Agent Reliability Engineer in Winnipeg, MB vacancy
- Join Rootly as an AI Engineer specializing in developing reliable AI-driven solutions for incident management. Use your LLM experience to shape the future of business reliability. As a Senior AI Engineer at Rootly, you will lead the development of a transformative AI reliability...Suggested
- As a Site Reliability Engineer II on the Serving Platforms team within Infrastructure Engineering, you will design, automate, and manage the core... ...RabbitMQ) in productionExperience operating, scaling, and monitoring AI/ML or LLM-powered services and workloads in high-concurrency...SuggestedWorldwide
- Design AI-agent workflows with Claude Code while building reliable engineering harnesses for software-development workflows in Toronto.This Agentic Engineer role focuses on introducing modern agentic engineering practices into a complex existing engineering organization. You...Suggested
- Lead AI agent engineering at BioRender remotely, focusing on multi-agent orchestration, model routing, and dependable scientific figure generation... ...platform with engineering leadership, model evaluation, orchestration, and reliable scientific figure generation. #J-18808-LjbffrSuggestedRemote work
- Shape AI agent platforms in Canada through tool calling, model routing, and production operations. Apply strong engineering judgment to practical systems.This Applied AI Engineer role centers... ...runtime architecture and orchestration for reliable customer workflows. You will design...Suggested
- Build AI agents for ClickStack using TypeScript, Node.Js, Python, and MCP in a product-focused engineering environment. Improve how developers investigate incidents.As an AI Product... ...experience, reusable skills, production reliability, and integrations. You will help agents...
$130k - $155k per year
.... Our vision is to build an ML/AI first network of advertisers and... ...hiring its first dedicated AI Engineer. Over the past year, AI at Fluent... ...to production: a service desk agent that closes tickets without a... ...It is getting trusted data and reliable actions in front of it.You are...Work at officeRemote work- Build enterprise AI agents at Decagon in an in-office role requiring technical delivery, integrations... ...-facing execution.As an Agent Deployment Engineer, you will own strategic customer... ...role connects customer requirements with reliable AI agent delivery.Key Responsibilities:...Work at office
- ...commitment to people. Our dependable quality, reliable delivery, and knowledgeable follow-up... ...and Coffee. Position Summary Join our AI team as an Intermediate AI Engineer, where you'll build and deploy AI applications, agents, and automation solutions that solve real business...
- Build AI agents for renewals and expansion at ShareGate, using low-code orchestration, Python or TypeScript, and SQL remotely across Canada... ..., marketing operations, customer success operations, or GTM engineering backgroundHands-on n8n, Make, Zapier, or similar experiencePython...Remote work
- Secure AI agents and LLMs in Toronto through cloud security architecture, guardrails, and protected non-human identities. This is a hybrid, full-time opportunity.As an AI security specialist, you will design protections across enterprise finance, ERP, and aerospace and defense...Full time
- ...MedMe Health in Toronto is seeking a senior engineer to own high-impact work end-to-end, turning ambiguous goals into shippable milestones... ..., enhance LLM/voice workflows, and raise the bar on reliability and security while collaborating across backend, frontend, and...
- ...Miratech is looking for a skilled Senior AI Engineer with Python, LangGraph and LangChain to... ...Responsibilities Develop and enhance AI agents and orchestration capabilities using... ...technical challenges, improve solution reliability, and drive continuous platform...Full timeInternshipRemote workRelocation
- Build production AI agents and order management systems at NewAge Products, working remotely... ...automation through hands-on software engineering.As an AI Engineer, you will design, test... ...integration, and evaluation expertise to reliable automation at NewAge Products. #J-18808...Remote work
$132.6k - $174k per year
...through automation, agentic automation, and AI. We partner with business and... ...Core responsibilitiesAs a Senior Software Engineer – AI Agents, you will lead the design and implementation... ...ensuring solutions meet scalability, reliability, and performance requirements. You will...Long term contractPermanent employmentPart timeManual laborLocal areaImmediate start- ...to life! We're building BioRender's AI figure agent, which can create any scientific figure... ...What you'll be doing Lead a small engineering team responsible for the core agent platform... ...to balance quality, cost, speed, and reliability. Build shared AI infrastructure and...Remote work
- Baseten is seeking a Site Reliability Engineer to define day‑2 operations standards for our ML infrastructure platform. You will build robust systems... ..., and develop observability tooling to keep services reliable at scale while enabling the broader team to operate confidently...
- ..., notes, tasks, calendar and they’re not AI-native. Our client’s mission is to build... ....Their product focuses on achieving high reliability for long-running workflows, persistent context... ...meaningful things.Role As a Backend Engineer, AI, you own the inference and...
- The Frontier AI Deployed Engineer is a member of Future Electronics' AI & Automation Center of Excellence... ...evaluate emerging AI tools, models, agent capabilities, development methods, and... ...AI solutions for grounding, accuracy, reliability, security, performance, latency,...Work at officeRemote work
- Build browser engine capabilities remotely from Canada, using profiling and AI tools to improve reliable programmatic web interaction. Investigate performance with evidence.As a Staff... ...choices with information available to AI agents.Key Responsibilities: Prototype engine...Remote work
- Use AI coding agents to deliver reliable applications at Remitly, with measurement, prompt iteration, and product ownership from requirements onward.This Software Engineer II role focuses on directing AI agents to build and ship complete solutions. You will evaluate generated...
- Workday, Inc. in Vancouver, Canada is seeking a Senior AI Engineer for the Agent Factory team to drive end-to-end system design and product integration... ..., and integrating models, balancing latency, cost, and reliability, while implementing governance and data privacy guardrails....
- Tenstorrent in Toronto, Canada, is seeking a Staff Reliability Engineer to define the reliability strategy for our next-generation AI computing systems. You’ll drive MTBF modeling, root-cause investigations, and cross-functional collaboration across mechanical, electrical, thermal...
- Design AI agents and complex software systems at lululemon, using AWS Bedrock, Python, and enterprise... ...four days weekly.As a Senior Software Engineer, you will lead AI agent implementation... ...standards while building scalable, reliable solutions. The Enterprise Automation team...
- Build production AI systems with Python and multi-agent workflows at Guidepoint’s hybrid Toronto office. Shape reliable research technology for global expert networks.As a hands-on Senior AI Engineer, you will move prototypes into production across agent design, evaluation,...Work at office
- ...Will Do Lead, hire, develop, and scale AI engineering teams and technical leaders supporting... ...engineering quality for LLM-based systems, agents, and integrations, from architecture and... ...operation, cost, latency, and reliability, ensuring teams make sound technical tradeoffs...For contractors
- Build AI-powered communication experiences with React and TypeScript at EvenUp, working hybrid from Toronto or San Francisco three days weekly.EvenUp seeks Senior and Mid Level Frontend Engineers for its Agents engineering pod within Growth & Agents. You will create user experiences...
- Decagon is building an Agent Deployment Engineering org focused on end-to-end AI agent builds for strategic customers. As an Agent Deployment Engineer you will own the implementation work to launch and scale enterprise-grade agents, partnering with customers and internal teams...
- ...Extreme Networks, Inc. is seeking an AI Agent Developer & Marketo Administrator to join our Marketing Operations team. You will focus... ...integrations across US time zones. This hands-on role blends AI engineering with marketing operations, requiring strong technical skills,...
$120k - $220k per year
Senior Founding Engineer - AI Agents Salary: $120k – $220k CAD 0.25% – 1.5% Job Location: Toronto Experience: 3 years of exp Job Type: Full Time Job LocationToronto Remote Work PolicyIn office - WFH flexibility Visa SponsorshipNot Available RelocationAllowed...Full timeFor contractorsInternshipWork at officeRemote workWork from home
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Ai Agent Reliability Engineer. Be the first to apply!
Related searches
- représentant senior Winnipeg, MB
- représentant alimentaire Winnipeg, MB
- agent d entretien Winnipeg, MB
- quality control representative Winnipeg, MB
- agent nettoyage Winnipeg, MB
- agent aux communications Winnipeg, MB
- agent de liaison Winnipeg, MB
- representant cosmetique Winnipeg, MB
- représentant industriel Winnipeg, MB
- représentant pharmaceutique Winnipeg, MB

