Senior Llm Platform Observability Engineer

Shape LLM platform observability at BMO, using OpenTelemetry, Prometheus, and Grafana in a hybrid engineering environment.The Senior AI Engineer builds capabilities for tracing agent workflows and managing production AI behavior. Responsibilities include capturing spans, prompts, tool calls, and intermediate outputs, while controlling token costs and log cardinality. You will also standardize pipelines, configure dashboards, and support automated anomaly detection and incident response.Key Responsibilities:
Implement OpenTelemetry-based agent tracing frameworksCapture prompts, spans, tools, and outputsBuild telemetry strategies for token optimizationConfigure logging sidecars and centralized dashboardsReplace static thresholds with anomaly detectionRequirements:
Bachelor's degree in Computer Science or EngineeringSeven to ten years of Site Reliability Engineering experienceExperience automating detection, investigation, and remediationPython experience for custom evaluationsSQL or LogQL/PromQL querying experienceUse LLM tracing, platform engineering, and intelligent monitoring to govern BMO’s autonomous AI workflows.
#J-18808-Ljbffr