Senior AI Engineer for Observability
$80k - $175k per yearBank of Montreal
Application Deadline:
11/29/2026
Address:
33 Dundas Street West
Job Family Group:
Data Analytics & Reporting
Hybrid role
Designs, builds, and maintains systems that track, evaluate, and debug non-deterministic AI models and agent workflows. Trace LLM calls, monitor token usage, detect drift, set up automated evaluations, and govern autonomous agent behaviors in production.
Designs and develops machine learning (ML) and deep learning systems.
Runs machine learning tests and experiments. Trains and retrain systems to prevent drift and optimize results.
Solves complex problems with multi-layered data sets, extends existing ML frameworks and optimizes existing machine learning libraries.
Develops Machine Learning apps, implements algorithms, and builds tools to apply ML frameworks.
Turns unstructured data into useful information by auto-tagging images and text-to-speech conversions.
Develops ML algorithms to analyze huge volumes of historical data to make predictions.
Runs tests, performs statistical analysis, and interprets test results.
Focus is primarily on business/group within BMO; may have broader, enterprise-wide focus.
Provides specialized consulting, analytical and technical support.
Exercises judgment to identify, diagnose, and solve problems within given rules.
Works independently and regularly handles non-routine situations.
Broader work or accountabilities may be assigned as needed.
Take measured risks while protecting the bank by applying our Risk Management Framework in the execution of your role, in line with our Risk Culture and within our approved Risk Appetite, making sound and risk informed decisions that align to business strategy, protect assets, and adhere to applicable policy documents (Frameworks, Policies, Standards, Procedures and Supporting documents), laws and regulations.
Qualifications:
Bachelor's Degree in Computer Science, Engineering or other relevant domain
7-10 years experience in Site Reliability Engineering
Experience in Artificial Intelligence Observability Engineering, facilitating the ability to Automate Detect, Investigate, and Remediate
Exposure to Platform Engineering, building AI capabilities
Highly proficient in automation, monitoring, observability, buildouts, AI engineering, understand model architectures, evaluation metrics, token usage, and how to spot concept or data drift
Experience using Python for custom evaluations and SQL or LogQL/PromQL for querying telemetry, logs, and metrics data.
Provision, configure, and standardize observability pipelines, logging sidecars, and centralized dashboards using Prometheus, Grafana, Datadog) to track SLIs/SLOs, handle alert fatigue, and run root cause analysis.
Exposure to heavily regulated industry organizations
Agent & LLM Tracing: Implement OpenTelemetry-based frameworks to capture step-by-step agent trajectories, spans, prompts, tool calls, and intermediate outputs.
Evaluation & Quality Assurance: Design automated evaluation pipelines for relevance, groundedness, hallucination detection, and prompt adherence using LLM-as-a-judge or custom code.
Drift & Performance Monitoring: Track model drift, data quality degradation, latency spikes, and error rates across production pipelines.
Cost & Token Optimization: Build telemetry and sampling strategies to control high data and inference costs, managing log retention and cardinality.
Agent Governance & Guardrails: Define operational boundaries for autonomous AI agents, setting rules for when they can self-remediate.
Alert Hygiene & Incident Response: Reduce alert fatigue by replacing static thresholds with dynamic, intelligent anomaly detection and building runbooks for AI failures.
Salary:
$80,000.00 - $175,000.00
Pay Type:
Salaried
The above represents BMO Financial Group’s pay range and type.
Salaries will vary based on factors such as location, skills, experience, education, and qualifications for the role, and may include a commission structure. Salaries for part-time roles will be pro-rated based on number of hours regularly worked. For commission roles, the salary listed above represents BMO Financial Group’s expected target for the first year in this position.
BMO Financial Group’s total compensation package will vary based on the pay type of the position and may include performance-based incentives, discretionary bonuses, as well as other perks and rewards. BMO also offers health insurance, tuition reimbursement, accident and life insurance, and retirement savings plans. To view more details of our benefits, please visit:
About Us
At BMO we are driven by a shared Purpose: Boldly Grow the Good in business and life. It calls on us to create lasting, positive change for our customers, our communities and our people. By working together, innovating and pushing boundaries, we transform lives and businesses, and power economic growth around the world.
As a member of the BMO team you are valued, respected and heard, and you have more ways to grow and make an impact. We strive to help you make an impact from day one – for yourself and our customers. We’ll support you with the tools and resources you need to reach new milestones, as you help our customers reach theirs. From in-depth training and coaching, to manager support and network-building opportunities, we’ll help you gain valuable experience, and broaden your skillset.
To find out more visit us at
BMO is committed to an inclusive, equitable and accessible workplace. By learning from each other’s differences, we gain strength through our people and our perspectives. Accommodations are available on request for candidates taking part in all aspects of the selection process. To request accommodation, please contact your recruiter.
Note to Recruiters: BMO does not accept unsolicited resumes from any source other than directly from a candidate. Any unsolicited resumes sent to BMO, directly or indirectly, will be considered BMO property. BMO will not pay a fee for any placement resulting from the receipt of an unsolicited resume. A recruiting agency must first have a valid, written and fully executed agency agreement contract for service to submit resumes.
Employment type: Full-time
$140k - $192.5k per year
...Secure Every Identity, from AI to Human Identity is the key... ...let's talk. The Auth0 Platform Observability team owns the observability... ...looking for an Observability Engineer to help ensure our Product and... ...development in these areas. As a Senior Engineer on this team, you...SeniorLocal areaWorldwide$175k - $210k per year
...Guidepoint is investing in applied AI across its research platform. We are building... .... We are looking for a hands-on Senior AI Engineer to take working prototypes into production... ...patterns for code review, testing, observability, incident response, and production readiness...SeniorFull timeInternshipWork at office$175k - $210k per year
...Overview: Guidepoint is investing in applied AI across its research platform. We're... ...datasets. You'll be an early engineer on a new initiative: taking working prototypes... ...patterns for a growing team: code review, observability, incident habits. What You Have: ~6+...SeniorInternshipWork at office- ...our weekly changelog . About the Role We’re hiring a Senior AI Engineer to lead the development of a new AI-driven reliability... ...that retrieves relevant context from disparate data sources (observability tools, GitHub, deployment histories, past incidents, playbooks...SeniorFull timeInternshipHome office
$137k - $165k per year
...system. Combined with industry leading AI, the Motive platform gives you complete... ...com to learn more. About the Role Senior engineer who ships production software and AI systems... ...CD for apps and model pipelines. Own observability and incident response; partner on SOC 2/...SeniorLong term contractTemporary work$135k - $210k per year
...Guidepoint seeks an experienced Data/AI Engineer as an integral member of the Toronto-based... ...responsible AI and best-in-class MLOps. The Senior AI/ML Engineer will be a primary... ...frameworks like LLM Judges or MLflow, AI observability, and system monitoring. Evaluate and...SeniorFull timeInternship- ...Competitive Intelligence Klue Engineering is hiring! We're looking for a Senior Software Engineer to join our... ...infrastructure. You care about reliability, observability, and CI/CD. Familiarity with... ...and high availability. You use AI coding tools to accelerate your...SeniorFull timeWork at office
- ...Miratech is looking for a skilled Senior AI Engineer with Python, LangGraph and LangChain to join our US healthcare client. You will play... ...frameworks and agent communication patterns. Implement AI observability, monitoring, telemetry, and performance tracking capabilities...SeniorInternshipRemote workRelocation
$140.6k - $190.6k per year
...statute has been treated over time. We are looking for a Senior Manager, AI Engineering to build and lead a team that is evolving Legal Content... .... Set standards for architecture, reliability, observability, experimentation, testing, and AI evaluation. Take ideas...SeniorRemplacementFull timeWork at officeLocal areaFlexible hours$100k - $145k per year
...our evolving business needs. Thomson Reuters is seeking a Senior Inference Engineer, AI. This person will collaborate with platform teams to... ...compute utilization across heterogeneous hardware Implement observability and health monitoring for inference pipelines, ensuring...SeniorFull timeWork at officeLocal areaFlexible hours2 days per week3 days per week$130k - $150k per year
...our platform and expanding our AI capabilities, we're looking for experienced engineers who thrive in fast-moving environments... ...looking for a highly motivated Senior Software Engineer with deep... ...application performance, scalability, observability, and reliability. ~...SeniorFull timeWork at officeRemote work$136k - $170k per year
...to thrive. Lyft is looking for software engineers from a scope of disciplines. We are... ...our passengers and drivers. The Applied AI team is looking for a Backend Engineer to... ...your future In addition to provincial observed holidays, salaried team members are covered...SeniorHourly payWork at officeFlexible hours3 days per week$135k - $155k per year
...Overview: We are hiring a Senior Software Engineer to build and run the search systems that connect... ...services in front of it all, and the AI features that turn a plain-language question... ...Kubernetes (ideally AKS), CI/CD, and observability tooling; you've been on call, run a...SeniorInternshipWork at office$140.6k - $190.6k per year
...Job Description Lead the engineering team delivering production AI solutions for public accounting firms and corporate... ...Reuters. You will hire and develop senior technical talent, build the right... ...(MCP), evaluation frameworks and observability. ~ Daily use of AI coding tools...SeniorFull timeWork at officeLocal areaFlexible hours$100k - $145k per year
...Job Description Senior Software Engineer, AI Are you passionate about transforming the future of legal research with AI? Join our Global Westlaw Engineering organization and help shape the next generation of intelligent legal research solutions. The Global Westlaw...SeniorFull timeInternshipWork at officeLocal areaWorldwideFlexible hoursShift work$64k - $106k per year
...sequential architectures. Build and optimize large-scale feature engineering in PySpark over hundreds of millions of transaction records, with... ...routine rather than a project. Design and implement agentic AI solutions that automate model improvement — extending our in-...SeniorInternshipWork at office$80k - $138k per year
...passionate about cloud technologies, automation, infrastructure engineering, and the rapidly evolving world of Artificial... ...innovation at enterprise scale? As a Cloud, Automation & AI Infrastructure Engineer, Senior Consultant, you will play a key role in designing, implementing...SeniorLong term contractPermanent employmentFlexible hours$130k - $165k per year
...NetBrain’s platform. What We Need We’re looking for a Senior AI Engineer to design and build production-grade agent and RAG systems that... ...everything from architecture and evaluation to scalability, observability, and reliability in production. The ideal candidate thrives...Senior- ...The Technology Services Applied AI team is a forward-deployed engineering team that embeds across Bell to build... ...existing enterprise systems. As a Senior Forward-Deployed AI Engineer, you... ...Experience setting up evaluation and observability for agentic systems. Prior...SeniorFull timeWork at office3 days per week
$113k - $163k per year
...As a Senior AI Engineer, you will be responsible for Full-Stack Software Development i.e. building modern, cloud-based, and AI-enabled applications that help Manulife make decisions easier and lives better. You will contribute to the design, development, and delivery of...SeniorLong term contractFull timeTemporary workLocal areaFlexible hours$100k - $145k per year
...Job Description Are you a hands-on AI engineer who wants to build the next generation of legal... ...the work you do here will matter. As a Senior Software Engineer – AI, you will work... ...deployment of AI solutions in production. Observability, Quality & Trust Implement monitoring...SeniorFull timeContract workInternshipWork at officeLocal areaFlexible hours- ...through world-class networks, AI-powered solutions, and digital... ...are looking for a hands-on AI Engineer (CP2) to design, build, and deploy... ...closely with data scientists, senior engineers, and business squads... ...unit/integration tests, and observability to monitor model performance,...SeniorFull timeWork at office3 days per week
- ...breakthrough outcomes for their toughest AI challenges. We blend the power of AI... ...unprecedented products across industries. As a Senior Lead AI Engineer , you are primarily responsible for... ...operating evaluation frameworks and observability tooling to measure AI system quality,...SeniorFull timeWork at officeLocal areaRemote workFlexible hours3 days per week1 day per week
- ...Senior AI Engineer Toronto Downtown, ON - Hybrid (4 Days WFO) In Person Interview Required Key Responsibilities • AI Product Delivery: Lead and actively contribute to the development of AI products, pilots, and enterprise solutions with a focus on writing clean...SeniorLong term contractPermanent employment
$250k per year
...Role: Observability Engineer – Trading Client: Elite FinTech Compensation: $120,000 - $250,000 CAD + Bonus Location: Toronto Overview My client are seeking an Engineer with strong Linux experience and expertise within the Observability space. The organisation...Permanent employmentImmediate start$100k - $145k per year
...We're building the next generation of AI‑powered platforms for tax and accounting... ..., and we’re looking for a Senior Software Engineer who can architect high‑performance backend... ...handling, guardrails, telemetry, and deep observability for performance and reliability. Lead...SeniorFull timeWork at officeLocal areaFlexible hours$140k - $160k per year
...Join our Engineering team, where you'll help build our agentic future. Our Applied AI Engineers don't just call an LLM and hope it works... ...day. You'll report to our Senior AI Engineering Manager and team... ...containerized workloads get deployed and observed. Kubernetes familiarity is a...SeniorManual laborWork at office3 days per week1 day per week$120 - $180 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ..., Larry Summers , and Jack Dorsey . Position: Senior Software Engineering Experts Type: Contract Compensation: $120–...SeniorRemote jobContract workSummer work$137k - $172k per year
...they need for a world being reshaped by AI. Read more about the combined company... ...global scale. Job Overview As a Senior Software Engineer, you will join a fast-paced innovation... ...is made, not after Own CI/CD, observability, and production support for systems living...SeniorLive InRemote work$120k - $215k per year
...The Global Product Owner, Observability is accountable for defining,... ...(APM), and Site Reliability Engineering (SRE)—ensuring their priorities... ...role regularly engages with senior and executive leadership, translating... ...capabilities extend to AI-enabled systems, providing visibility...SeniorLong term contractContract workPart time
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Senior AI Engineer for Observability. Be the first to apply!
- senior disability case manager Toronto, ON
- senior hardware engineer Toronto, ON
- senior cook Toronto, ON
- amica senior lifestyles Toronto, ON
- senior building operator Toronto, ON
- senior project manager Toronto, ON
- senior communications Toronto, ON
- sr. IT support analyst Toronto, ON
- senior marketing Toronto, ON
- senior manager investigations Toronto, ON
