Senior Ai Observability Engineer

Build AI observability systems at BMO in a hybrid role, using Python, telemetry, and evaluation methods for production models and agents.As a Senior AI Engineer, you will design systems that trace LLM calls, monitor token usage, detect drift, and evaluate autonomous workflows. The work includes machine learning, deep learning, statistical analysis, and specialized technical support. You will help govern agent behavior and diagnose non-routine production problems.Key Responsibilities:
Design and maintain AI model observability systemsTrace agent trajectories, prompts, tools, and outputsBuild automated evaluations for relevance and hallucinationsMonitor drift, latency, errors, and data qualityDefine guardrails and self-remediation boundariesRequirements:
Bachelor's degree in Computer Science or EngineeringSeven to ten years of Site Reliability Engineering experienceExperience in Artificial Intelligence Observability EngineeringPython, SQL, LogQL, or PromQL proficiencyExperience with Prometheus, Grafana, or DatadogApply AI engineering, observability, and evaluation expertise to support reliable agent workflows at BMO.
#J-18808-Ljbffr