Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Senior AI Engineer for Observability

$80k - $175k per year

Bank of Montreal

Application Deadline:

11/29/2026

Address:
33 Dundas Street West

Job Family Group:

Data Analytics & Reporting

Hybrid role

Designs, builds, and maintains systems that track, evaluate, and debug non-deterministic AI models and agent workflows. Trace LLM calls, monitor token usage, detect drift, set up automated evaluations, and govern autonomous agent behaviors in production.
Designs and develops machine learning (ML) and deep learning systems.
Runs machine learning tests and experiments. Trains and retrain systems to prevent drift and optimize results.
Solves complex problems with multi-layered data sets, extends existing ML frameworks and optimizes existing machine learning libraries.
Develops Machine Learning apps, implements algorithms, and builds tools to apply ML frameworks.
Turns unstructured data into useful information by auto-tagging images and text-to-speech conversions.
Develops ML algorithms to analyze huge volumes of historical data to make predictions.
Runs tests, performs statistical analysis, and interprets test results.
Focus is primarily on business/group within BMO; may have broader, enterprise-wide focus.
Provides specialized consulting, analytical and technical support.
Exercises judgment to identify, diagnose, and solve problems within given rules.
Works independently and regularly handles non-routine situations.
Broader work or accountabilities may be assigned as needed.
Take measured risks while protecting the bank by applying our Risk Management Framework in the execution of your role, in line with our Risk Culture and within our approved Risk Appetite, making sound and risk informed decisions that align to business strategy, protect assets, and adhere to applicable policy documents (Frameworks, Policies, Standards, Procedures and Supporting documents), laws and regulations.

Qualifications:

Bachelor's Degree in Computer Science, Engineering or other relevant domain
7-10 years experience in Site Reliability Engineering
Experience in Artificial Intelligence Observability Engineering, facilitating the ability to Automate Detect, Investigate, and Remediate
Exposure to Platform Engineering, building AI capabilities
Highly proficient in automation, monitoring, observability, buildouts, AI engineering, understand model architectures, evaluation metrics, token usage, and how to spot concept or data drift
Experience using Python for custom evaluations and SQL or LogQL/PromQL for querying telemetry, logs, and metrics data.
Provision, configure, and standardize observability pipelines, logging sidecars, and centralized dashboards using Prometheus, Grafana, Datadog) to track SLIs/SLOs, handle alert fatigue, and run root cause analysis.
Exposure to heavily regulated industry organizations

Agent & LLM Tracing: Implement OpenTelemetry-based frameworks to capture step-by-step agent trajectories, spans, prompts, tool calls, and intermediate outputs.
Evaluation & Quality Assurance: Design automated evaluation pipelines for relevance, groundedness, hallucination detection, and prompt adherence using LLM-as-a-judge or custom code.
Drift & Performance Monitoring: Track model drift, data quality degradation, latency spikes, and error rates across production pipelines.
Cost & Token Optimization: Build telemetry and sampling strategies to control high data and inference costs, managing log retention and cardinality.
Agent Governance & Guardrails: Define operational boundaries for autonomous AI agents, setting rules for when they can self-remediate.
Alert Hygiene & Incident Response: Reduce alert fatigue by replacing static thresholds with dynamic, intelligent anomaly detection and building runbooks for AI failures.

Salary:

$80,000.00 - $175,000.00
Pay Type:

Salaried
The above represents BMO Financial Group’s pay range and type.

Salaries will vary based on factors such as location, skills, experience, education, and qualifications for the role, and may include a commission structure. Salaries for part-time roles will be pro-rated based on number of hours regularly worked. For commission roles, the salary listed above represents BMO Financial Group’s expected target for the first year in this position.

BMO Financial Group’s total compensation package will vary based on the pay type of the position and may include performance-based incentives, discretionary bonuses, as well as other perks and rewards. BMO also offers health insurance, tuition reimbursement, accident and life insurance, and retirement savings plans. To view more details of our benefits, please visit: 

About Us

At BMO we are driven by a shared Purpose: Boldly Grow the Good in business and life. It calls on us to create lasting, positive change for our customers, our communities and our people. By working together, innovating and pushing boundaries, we transform lives and businesses, and power economic growth around the world.

As a member of the BMO team you are valued, respected and heard, and you have more ways to grow and make an impact. We strive to help you make an impact from day one – for yourself and our customers. We’ll support you with the tools and resources you need to reach new milestones, as you help our customers reach theirs. From in-depth training and coaching, to manager support and network-building opportunities, we’ll help you gain valuable experience, and broaden your skillset.

To find out more visit us at

BMO is committed to an inclusive, equitable and accessible workplace. By learning from each other’s differences, we gain strength through our people and our perspectives. Accommodations are available on request for candidates taking part in all aspects of the selection process. To request accommodation, please contact your recruiter.

Note to Recruiters: BMO does not accept unsolicited resumes from any source other than directly from a candidate. Any unsolicited resumes sent to BMO, directly or indirectly, will be considered BMO property. BMO will not pay a fee for any placement resulting from the receipt of an unsolicited resume. A recruiting agency must first have a valid, written and fully executed agency agreement contract for service to submit resumes.

Employment type: Full-time

Vacancy posted 1 day ago
Similar jobs that could be interesting for youBased on the Senior AI Engineer for Observability in Toronto, ON vacancy
  • $140k - $192.5k per year

     ...Secure Every Identity, from AI to Human Identity is the key...  ...let's talk. The Auth0 Platform Observability team owns the observability...  ...looking for an Observability Engineer to help ensure our Product and...  ...development in these areas. As a Senior Engineer on this team, you... 
    Senior
    Local area
    Worldwide

    Okta

    Toronto, ON
    2 hours ago
  • $175k - $210k per year

     ...Guidepoint is investing in applied AI across its research platform. We are building...  .... We are looking for a hands-on Senior AI Engineer to take working prototypes into production...  ...patterns for code review, testing, observability, incident response, and production readiness... 
    Senior
    Full time
    Internship
    Work at office

    Guidepoint

    Toronto, ON
    2 hours ago
  • $175k - $210k per year

     ...Overview: Guidepoint is investing in applied AI across its research platform. We're...  ...datasets. You'll be an early engineer on a new initiative: taking working prototypes...  ...patterns for a growing team: code review, observability, incident habits. What You Have: ~6+... 
    Senior
    Internship
    Work at office

    Guidepoint

    Toronto, ON
    2 hours ago
  •  ...our  weekly changelog . About the Role We’re hiring a Senior AI Engineer to lead the development of a new AI-driven reliability...  ...that retrieves relevant context from disparate data sources (observability tools, GitHub, deployment histories, past incidents, playbooks... 
    Senior
    Full time
    Internship
    Home office

    Rootly

    Toronto, ON
    6 days ago
  • $137k - $165k per year

     ...system. Combined with industry leading AI, the Motive platform gives you complete...  ...com to learn more. About the Role Senior engineer who ships production software and AI systems...  ...CD for apps and model pipelines. Own observability and incident response; partner on SOC 2/... 
    Senior
    Long term contract
    Temporary work

    Motive

    Toronto, ON
    2 hours ago
  • $135k - $210k per year

     ...Guidepoint seeks an experienced Data/AI Engineer as an integral member of the Toronto-based...  ...responsible AI and best-in-class MLOps. The Senior AI/ML Engineer will be a primary...  ...frameworks like LLM Judges or MLflow, AI observability, and system monitoring. Evaluate and... 
    Senior
    Full time
    Internship

    Guidepoint

    Toronto, ON
    2 hours ago
  •  ...Competitive Intelligence Klue Engineering is hiring! We're looking for a Senior Software Engineer to join our...  ...infrastructure. You care about reliability, observability, and CI/CD. Familiarity with...  ...and high availability. You use AI coding tools to accelerate your... 
    Senior
    Full time
    Work at office

    klue

    Toronto, ON
    6 days ago
  •  ...Miratech is looking for a skilled Senior AI Engineer with Python, LangGraph and LangChain to join our US healthcare client. You will play...  ...frameworks and agent communication patterns. Implement AI observability, monitoring, telemetry, and performance tracking capabilities... 
    Senior
    Internship
    Remote work
    Relocation

    Miratech

    Toronto, ON
    6 days ago
  • $140.6k - $190.6k per year

     ...statute has been treated over time. We are looking for a Senior Manager, AI Engineering to build and lead a team that is evolving Legal Content...  .... Set standards for architecture, reliability, observability, experimentation, testing, and AI evaluation. Take ideas... 
    Senior
    Remplacement
    Full time
    Work at office
    Local area
    Flexible hours

    Thomson Reuters

    Toronto, ON
    1 day ago
  • $100k - $145k per year

     ...our evolving business needs. Thomson Reuters is seeking a Senior Inference Engineer, AI. This person will collaborate with platform teams to...  ...compute utilization across heterogeneous hardware Implement observability and health monitoring for inference pipelines, ensuring... 
    Senior
    Full time
    Work at office
    Local area
    Flexible hours
    2 days per week
    3 days per week

    Thomson Reuters

    Toronto, ON
    9 days ago
  • $130k - $150k per year

     ...our platform and expanding our AI capabilities, we're looking for experienced engineers who thrive in fast-moving environments...  ...looking for a highly motivated Senior Software Engineer with deep...  ...application performance, scalability, observability, and reliability.  ~... 
    Senior
    Full time
    Work at office
    Remote work

    Fable

    Toronto, ON
    12 days ago
  • $136k - $170k per year

     ...to thrive. Lyft is looking for software engineers from a scope of disciplines. We are...  ...our passengers and drivers. The Applied AI team is looking for a Backend Engineer to...  ...your future In addition to provincial observed holidays, salaried team members are covered... 
    Senior
    Hourly pay
    Work at office
    Flexible hours
    3 days per week

    Lyft

    Toronto, ON
    2 hours ago
  • $135k - $155k per year

     ...Overview:  We are hiring a Senior Software Engineer to build and run the search systems that connect...  ...services in front of it all, and the AI features that turn a plain-language question...  ...Kubernetes (ideally AKS), CI/CD, and observability tooling; you've been on call, run a... 
    Senior
    Internship
    Work at office

    Guidepoint

    Toronto, ON
    2 hours ago
  • $140.6k - $190.6k per year

     ...Job Description Lead the engineering team delivering production AI solutions for public accounting firms and corporate...  ...Reuters. You will hire and develop senior technical talent, build the right...  ...(MCP), evaluation frameworks and observability. ~ Daily use of AI coding tools... 
    Senior
    Full time
    Work at office
    Local area
    Flexible hours

    Thomson Reuters

    Toronto, ON
    1 day ago
  • $100k - $145k per year

     ...Job Description Senior Software Engineer, AI Are you passionate about transforming the future of legal research with AI? Join our Global Westlaw Engineering organization and help shape the next generation of intelligent legal research solutions.   The Global Westlaw... 
    Senior
    Full time
    Internship
    Work at office
    Local area
    Worldwide
    Flexible hours
    Shift work

    Thomson Reuters

    Toronto, ON
    1 day ago
  • $64k - $106k per year

     ...sequential architectures. Build and optimize large-scale feature engineering in PySpark over hundreds of millions of transaction records, with...  ...routine rather than a project. Design and implement agentic AI solutions that automate model improvement — extending our in-... 
    Senior
    Internship
    Work at office

    Canadian Tire Corporation

    Toronto, ON
    6 days ago
  • $80k - $138k per year

     ...passionate about cloud technologies, automation, infrastructure engineering, and the rapidly evolving world of Artificial...  ...innovation at enterprise scale? As a Cloud, Automation & AI Infrastructure Engineer, Senior Consultant, you will play a key role in designing, implementing... 
    Senior
    Long term contract
    Permanent employment
    Flexible hours

    Deloitte

    Toronto, ON
    5 days ago
  • $130k - $165k per year

     ...NetBrain’s platform. What We Need We’re looking for a Senior AI Engineer to design and build production-grade agent and RAG systems that...  ...everything from architecture and evaluation to scalability, observability, and reliability in production. The ideal candidate thrives... 
    Senior

    NetBrain

    Toronto, ON
    2 hours ago
  •  ...The Technology Services Applied AI team is a forward-deployed engineering team that embeds across Bell to build...  ...existing enterprise systems. As a Senior Forward-Deployed AI Engineer, you...  ...Experience setting up evaluation and observability for agentic systems. Prior... 
    Senior
    Full time
    Work at office
    3 days per week

    Bell

    Toronto, ON
    6 days ago
  • $113k - $163k per year

     ...As a Senior AI Engineer, you will be responsible for Full-Stack Software Development i.e. building modern, cloud-based, and AI-enabled applications that help Manulife make decisions easier and lives better. You will contribute to the design, development, and delivery of... 
    Senior
    Long term contract
    Full time
    Temporary work
    Local area
    Flexible hours

    Manulife

    Toronto, ON
    6 days ago
  • $100k - $145k per year

     ...Job Description Are you a hands-on AI engineer who wants to build the next generation of legal...  ...the work you do here will matter. As a Senior Software Engineer – AI, you will work...  ...deployment of AI solutions in production. Observability, Quality & Trust Implement monitoring... 
    Senior
    Full time
    Contract work
    Internship
    Work at office
    Local area
    Flexible hours

    Thomson Reuters

    Toronto, ON
    8 days ago
  •  ...through world-class networks, AI-powered solutions, and digital...  ...are looking for a hands-on AI Engineer (CP2) to design, build, and deploy...  ...closely with data scientists, senior engineers, and business squads...  ...unit/integration tests, and observability to monitor model performance,... 
    Senior
    Full time
    Work at office
    3 days per week

    Bell

    Toronto, ON
    6 days ago
  •  ...breakthrough outcomes for their toughest AI challenges. We blend the power of AI...  ...unprecedented products across industries. As a Senior Lead AI Engineer , you are primarily responsible for...  ...operating evaluation frameworks and observability tooling to measure AI system quality,... 
    Senior
    Full time
    Work at office
    Local area
    Remote work
    Flexible hours
    3 days per week
    1 day per week

    Marsh McLennan

    Toronto, ON
    10 days ago
  •  ...Senior AI Engineer Toronto Downtown, ON - Hybrid (4 Days WFO) In Person Interview Required  Key Responsibilities • AI Product Delivery: Lead and actively contribute to the development of AI products, pilots, and enterprise solutions with a focus on writing clean... 
    Senior
    Long term contract
    Permanent employment

    Astra North Infoteck Inc.

    Toronto, ON
    a month ago
  • $250k per year

     ...Role: Observability Engineer – Trading Client:  Elite FinTech Compensation: $120,000 - $250,000 CAD + Bonus Location:  Toronto Overview My client are seeking an Engineer with strong Linux experience and expertise within the Observability space. The organisation... 
    Permanent employment
    Immediate start

    Hunter Bond

    Toronto, ON
    more than 2 months ago
  • $100k - $145k per year

     ...We're building the next generation of AI‑powered platforms for tax and accounting...  ..., and we’re looking for a Senior Software Engineer who can architect high‑performance backend...  ...handling, guardrails, telemetry, and deep observability for performance and reliability. Lead... 
    Senior
    Full time
    Work at office
    Local area
    Flexible hours

    Thomson Reuters

    Toronto, ON
    1 day ago
  • $140k - $160k per year

     ...Join our Engineering team, where you'll help build our agentic future. Our Applied AI Engineers don't just call an LLM and hope it works...  ...day. You'll report to our Senior AI Engineering Manager and team...  ...containerized workloads get deployed and observed. Kubernetes familiarity is a... 
    Senior
    Manual labor
    Work at office
    3 days per week
    1 day per week

    ContactMonkey

    Toronto, ON
    2 hours ago
  • $120 - $180 per hour

     ...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors...  ..., Larry Summers , and Jack Dorsey . Position: Senior Software Engineering Experts Type: Contract Compensation: $120–... 
    Senior
    Remote job
    Contract work
    Summer work

    Mercor

    Toronto, ON
    19 days ago
  • $137k - $172k per year

     ...they need for a world being reshaped by AI. Read more about the combined company...  ...global scale. Job Overview As a Senior Software Engineer, you will join a fast-paced innovation...  ...is made, not after Own CI/CD, observability, and production support for systems living... 
    Senior
    Live In
    Remote work

    Coursera

    Toronto, ON
    2 hours ago
  • $120k - $215k per year

     ...The Global Product Owner, Observability is accountable for defining,...  ...(APM), and Site Reliability Engineering (SRE)—ensuring their priorities...  ...role regularly engages with senior and executive leadership, translating...  ...capabilities extend to AI-enabled systems, providing visibility... 
    Senior
    Long term contract
    Contract work
    Part time

    Bank of Montreal

    Toronto, ON
    6 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Senior AI Engineer for Observability. Be the first to apply!