Applied AI Engineer — Evaluation & Reliability

Applied AI Engineer — Evaluation & Reliability

15Rock is a financial intelligence firm. Our research goes in front of institutional investors and gets read line by line — by people looking for the one number that doesn't hold up. Every figure has to reconcile across every table and sentence; every claim has to trace to a live source.

We build that research with a multi-agent pipeline on large language models. LLMs write well and count badly: left alone, they produce clean prose that contradicts the table three paragraphs up. Our answer isn't "prompt harder." It's to route every number through deterministic computation and gate every report behind checks that pass or fail — never a model grading its own work. You'd build that layer.

What you'll build

• Deterministic evaluation gates that block a report when a defect is present — measured against thresholds set before the run, not a gut feel.
• A fact-reconciliation layer that verifies a figure agrees with itself everywhere it appears.
• Citation-integrity checks: every reference resolves, every quote is real, every source is live.
• Golden-set and regression harnesses that catch quality drift before it ships.
• The path from research to production: hardening a validated approach into the pipeline, behind CI, with the traces to prove it works.

What we're looking for

• You've shipped LLM systems to production — not demos — where being wrong had a cost.
• You've built the evaluations that kept them honest: you think in precision and recall, and you define "good" before you measure it.
• Strong Python; you can own backend services and data pipelines yourself.
• You reason about the numbers, not just the tokens — enough quantitative or financial literacy to know when a figure is wrong, or the appetite to learn it fast.
• Self-directing. You'd rather write a one-page memo than wait for a ticket.

Bonus: model-risk or evaluation-infrastructure work in a regulated setting; LangGraph, DuckDB, CI you owned; a finance or quantitative background.

This role is not

Prompt engineering, a pure research seat, or a place where "the model said so" is an answer.

How we work

A small, senior team with direct access to the founder and research lead. Async by default — we meet on blockers, not to read status aloud. We run on hypotheses and measurable outcomes.

Details

Downtown Toronto, on-site / hybrid. Must be authorized to work in Canada. Early-stage financial intelligence firm, recognized by the Globe and Mail Changemaker Award and selected for Google for Startups.

15Rock is an equal-opportunity employer.