Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Director, Evaluations

LawZero

LawZero is a non-profit building safe-by-design AI systems. We’re building the Scientist AI, an advanced AI system designed from the ground up to be both highly capable and safe. As we develop both general‑purpose Scientist AI models and safety guardrails for frontier LLMs, we need rigorous, independent evaluation of every capability and safety claim we make. We are looking for a Director of Evaluations to build, lead, and grow LawZero’s Evaluations Team.

This is a foundational hire. You will define what world‑class evaluation looks like at LawZero, build the team and infrastructure to deliver it, and ensure that evaluations remain independent of the main research stream so that capability and safety claims can be trusted both internally and externally by the wider AI and AI safety community.

 

Key responsibilities

  • Define LawZero’s evaluations strategy and roadmap, prioritising what needs to be measured and when, in close coordination with both research and product teams.
  • Build up the Evaluations Team during your first 3–6 months, scaling to roughly 8–10 people across research, engineering, dataset and benchmark design, and red‑teaming.
  • Operate the team independently of the main research and product streams in order to avoid conflicts of interest, including designing novel benchmarks that can be applied apples‑to‑apples to evaluate both the Scientist AI and frontier LLMs.
  • Oversee the design and construction of new datasets, tasks, and virtual or interactive environments to measure performance of the Scientist AI across  capabilities, safety (including honesty and goal-directedness), explainability, causal mechanisms and detecting adversarial attacks.
  • Lead evaluation of the Scientist AI when deployed as a guardrail around frontier models, including its ability to comply with harm specifications, detect and block harmful responses, explain its decisions, and resist adversarial attacks such as jailbreaks, prompt injection, and data poisoning.
  • Establish and lead our automated and manual red‑teaming programmes, both in‑house and in partnership with external providers, to stress test the Scientist AI as a general‑purpose model and as a guardrail.
  • Lead the construction of internal tooling and infrastructure needed to run evaluations at scale, automating and standardizing the pipeline wherever possible.
  • As needed and where possible, directly support research and product streams with their own internal requirements w.r.t. evaluations and benchmarking to unblock and speed up.
  • Own LawZero’s public communication of evaluation results, including model and system cards, technical reports, peer‑reviewed publications and blog posts, to build trust with the wider AI safety community.
  • Represent LawZero externally on evaluations and AI safety measurement, including engagements with AI safety institutes, research collaborators, and grant funders.

 

Skills and qualifications

  • An advanced degree (MSc or higher) in machine learning, computer science, or a closely related field.
  • 10+ years of experience in machine learning, with at least 5 years in a leadership role building or scaling technical teams working on real-world ML products.
  • Hands‑on expertise in designing and running large‑scale evaluations of LLMs or other frontier ML systems across capabilities, safety, and adversarial robustness.
  • A track record of building evaluation datasets, benchmarks, or interactive environments from scratch, including for safety‑relevant properties such as honesty, sycophancy, refusal behaviour, and adversarial robustness.
  • Strong written and verbal communication skills, including the ability to translate technical results for non‑technical audiences such as executives, funders, and policymakers.
  • Comfortable operating in a research‑driven, fast‑moving environment with significant ambiguity, and able to bring structure to it without slowing it down.

 

Nice to have: 

  • Experience leading red‑teaming exercises (automated, manual, or both) and working with third‑party evaluation or red‑teaming partners is a bonus.
  • Experience working with third-party partners for benchmark and dataset creation 
  • Experience releasing open‑source datasets, benchmarks, or evaluation tooling is a bonus.
  • Familiarity with current AI safety policy and standards work (UK AISI, CAISI, NIST, EU AI Act, etc.) is a bonus.
  • Experience contributing to or coordinating with external safety institutes, grant funders, or government bodies is a bonus.

 

What we offer

  • The opportunity to contribute to a unique mission with a major impact
  • Comprehensive health benefits
  • A minimum of 20 days vacation per year upon start
  • A minimum retirement savings employer contribution of 4%
  • Generous flexible benefits designed to contribute to your well-being
  • A team of passionate experts in their field
  • A collaborative and inclusive work environment with offices in the heart of Little Italy, in the trendy Mile-Ex district, close to public transportation.

About LawZero

LawZero is a non-profit organization committed to advancing research and creating technical solutions that enable safe-by-design AI systems. Its scientific direction is based on new research and methods proposed by Professor Yoshua Bengio, the most cited AI researcher in the world. Based in Montreal, LawZero’s research aims to build AI that learns primarily to understand the world rather than to act in it, giving truthful answers to questions based on transparent and externalized probabilistic reasoning. Such AI systems could be used to accelerate scientific discovery, to provide oversight for agentic AI systems, and to advance the understanding of AI risks and how to avoid them. LawZero believes that AI should be cultivated as a global public good—developed and used safely towards human flourishing. For more information, visit

You belong here

At LawZero, diversity is important to us. We value a work environment that is fair, open and respectful of differences. We welcome applications from highly qualified individuals interested in working towards our mission in a respectful, inclusive and collaborative setting.

Your personal information will be collected and processed by LawZero to evaluate your application for employment in compliance with our  Privacy Policy . Under privacy laws in force in your country of residence, you may have several privacy rights, such as to request access to your personal information or to request that your personal information be rectified or erased. Details on how you can exercise your rights can be found in our Privacy Policy.

Vacancy posted 4 hours ago
Similar jobs that could be interesting for youBased on the Director, Evaluations in Montréal, QC vacancy
  •  ...de nos modèles généraux Scientist AI et de garde-fous de sécurité pour les modèles de langage de pointe (LLM frontier ), nous devons évaluer de manière rigoureuse et indépendante chaque affirmation concernant leurs capacités et leur sécurité. Nous recherchons un.e Directeur... 
    Suggested
    Apprenticeship
    Work at office
    Flexible hours
    Day shift

    LoiZéro

    Montréal, QC
    4 hours ago
  •  ...Titre du poste : Évaluateur(trice) immobilier / Appraiser Lieu : Montréal Salaire : selon expérience À propos du poste Joindre cette opportunité, c'est intégrer une organisation multidisciplinaire bien établie dans le secteur de l'évaluation et des services immobiliers... 
    Suggested
    Work at office
    Flexible hours

    Totem recruteur de talent

    Montréal, QC
    13 days ago
  •  ...développement de projets majeurs en énergie, infrastructures et ressources naturelles. Nous recherchons actuellement un(e) leader senior en évaluation environnementale souhaitant jouer un rôle clé dans la réalisation et le positionnement de projets structurants au Québec et à l’... 
    Suggested
    Local area
    Flexible hours
    Day shift

    Hatch

    Montréal, QC
    6 days ago
  • $72k - $108k per year

     ...coentreprises et de financement de projet; Planification des dépenses en immobilisations et analyses de sensibilité; Soutien à l'évaluation d'entreprise et aux évaluations liées à la présentation des états financiers; Revue et validation de modèles financiers.... 
    Suggested
    Permanent employment
    Apprenticeship
    Work at office
    Local area
    Flexible hours
    Day shift

    Deloitte

    Montréal, QC
    4 days ago
  •  ...opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. This is a task-based, independent contractor...  ...license to evaluate clinical EHR vignettes for a medical AI evaluation project. General and internal medicine are the main focus. Each... 
    Suggested
    Part time
    For contractors
    Remote work
    Flexible hours

    Mindrift

    Montréal, QC
    2 days ago
  •  .... Atouts Une expérience en réadaptation ambulatoire auprès dune clientèle ayant une déficience physique; Une expérience en évaluation des capacités fonctionnelles. Vous êtes nouvellement diplômé(e) Votre candidature nous intéresse également. Vous pourrez développer... 
    Full time

    CIUSSS du Centre-Ouest-de-l'Île-de-Montréal

    Montréal, QC
    2 days ago
  • Votre opportunité Axés sur la santé, la sécurité, la qualité et l’éthique, nos experts dirigent la réalisation de projets avec de la détermination, un esprit créatif et une vision de croissance. Présents partout dans le monde et comptant des professionnels dans plus de 20...
    Full time
    Contract work
    Temporary work
    Part time
    Work at office

    Stantec

    Montréal, QC
    6 days ago
  •  ...permettre d’atteindre leur plein potentiel, et leur servir de modèle. Renseignements sur l’équipe   Les Services-conseils en évaluation d’entreprises et modélisation se spécialisent dans la résolution de problèmes complexes liés aux affaires et aux transactions, y compris... 
    Permanent employment
    Apprenticeship
    Work at office
    Local area
    Flexible hours
    Day shift

    Deloitte

    Montréal, QC
    6 days ago
  •  ...Évaluateur(trice) en santé et sécurité – Bilingue Department: Operations Employment Type: Permanent Location: Greater Montreal Metropolitan Area, Canada Reporting To: Chef de l'accréditation en santé et sécurité Description Créer des solutions innovantes... 
    Permanent employment
    Apprenticeship
    Work at office

    Veriforce

    Montréal, QC
    17 days ago
  •  ...to apply your relevant experience to a new industry, join our team as we help shape a brighter way forward. Conseiller principal, Évaluation - JLL En quoi consiste cet emploi : Joignez-vous à l'équipe d’Évaluation chez JLL à titre de Conseiller(e) principal(e) et devenez... 
    Daily paid
    Full time
    Temporary work
    Work at office
    Remote work

    JLL

    Montréal, QC
    9 days ago
  •  ...Conseiller en gestion de programmes au sein du Décanat des études de l’ÉTS? Vous soutiendrez le doyen des études dans la création, l’évaluation et l’amélioration continue des programmes d’études, en accompagnant les directions des départements, les directions de programmes,... 
    Daily paid
    Full time
    Temporary work

    École de technologie supérieure (ÉTS)

    Montréal, QC
    12 days ago
  •  ...Localisation: Partout au Québec Nous recherchons un(e) spécialiste en milieux humides et végétation, possédant une expertise en évaluation environnementale. La personne retenue apportera un soutien technique pour les études environnementales, les évaluations d'impact et... 
    Full time
    Flexible hours

    AtkinsRéalis

    Montréal, QC
    17 days ago
  •  ...quotidien sera de gérer et de coordonner la production tant des études d’impact sur l’environnement (ÉIE), que des autres types d’évaluations environnementales et de demandes d’autorisation et de permis. Vous devrez être familier avec les législations, réglementations et... 
    Daily paid
    Work at office

    Stantec

    Montréal, QC
    6 days ago
  •  ...consiste principalement à prendre des décisions judicieuses en tirant parti d’informations exactes. En tant que membre de l’équipe Évaluation, voilà ce que vous contribuerez à rendre possible. Vous serez un membre important de notre équipe diversifiée et hautement expérimentée... 
    Daily paid
    Weekend work

    Ernst & Young

    Montréal, QC
    12 days ago
  •  ...alongside iconic brand names. About the Role: As a luxury brand evaluator, you will step into the world of luxury to discreetly assess...  ...in the luxury market. Compensation: • Non-Purchase Evaluations: Earn a fee based on mission complexity. • Purchase-Based Evaluations... 
    Contract work
    Worldwide
    Flexible hours

    CXG

    Montréal, QC
    more than 2 months ago
  • $36 per hour

     ...such as sauna, steam room, and jacuzzi. We are looking for a Spa Director to ensure the planning, organization, and supervision of the...  ...of the establishment. Plan and organize daily operations. Evaluate daily operations and propose improvements. Develop and implement... 
    Hourly pay
    Permanent employment
    Full time
    Shift work
    Night shift
    Weekend work
    Day shift
    Afternoon shift

    GESTION OASIS

    Montréal, QC
    4 days ago
  • Vos missions au quotidien L’équipe Appétit au risque et évaluation des risques fait partie de la fonction Gestion intégrée des risques de SG Americas et joue un rôle central dans le soutien des activités d’identification, d’évaluation, de surveillance, de reddition... 
    Daily paid

    Société Générale

    Montréal, QC
    more than 2 months ago
  •  ...transaction discipline and an ownership mindset to move opportunities from initial evaluation through closing, integration and value realization. What you will lead as the Managing Director, Corporate Development Define and advance the M&A strategy in alignment with... 
    Long term contract
    Local area

    GardaWorld

    Montréal, QC
    4 days ago
  •  ...material, with the touch of a button. About The Role   As Director of Marketing, you will play a key role in leading AON3D’s marketing...  ...the hiring process to promote fair, consistent, and objective evaluation. Certain initial screening steps may be automated to help... 
    Permanent employment
    Full time
    Internship
    Worldwide

    AON3D Careers

    Montréal, QC
    4 hours ago
  • $155k - $200k per year

     ...the buzz is about! Role Overview We’re looking for a Growth Director to own how Clipwire’s games find and keep their players. You’ll...  ...to screen applications or make hiring decisions. All candidate evaluations and hiring decisions are made by people. This opening is for... 
    Long term contract
    Casual work
    Work at office
    Flexible hours
    Afternoon shift

    Tripledot Studios

    Montréal, QC
    4 hours ago
  •  ...pour nous propulser vers les meilleurs résultats pour la population du Québec. Description d'emploi Au sein de l’équipe ’Évaluation, la personne titulaire de ce rôle évalue des placements directs des portefeuilles infrastructures et placements privés pour les états... 
    Daily paid
    Permanent employment
    Full time
    Work at office

    La Caisse

    Montréal, QC
    3 days ago
  • Nous sommes � la recherche d�un directeur de la comptabilit� chevronn� pour diriger les activit�s d�information financi�re d�une organisation bancaire situ�e � Montr�al. Dans ce r�le, vous superviserez l�exactitude et la production en temps opportun des rapports financiers internes...
    Permanent employment
    Contract work
    Temporary work
    Work at office

    Robert Half

    Montréal, QC
    6 days ago
  •  ...portfolio of vertical market software businesses, we are seeking a dynamic and experienced professional to join our team as Managing Director. Reporting directly to one of our Investment Partners, the Managing Director will be responsible for managing and growing their businesses... 
    Long term contract
    Permanent employment
    Full time
    Immediate start

    Valsoft Corporation

    Montréal, QC
    9 days ago
  •  ...the touch of a button. About The Role   AON3D is seeking a Director, Customer Success to own the end-to-end post-sale customer...  ...the hiring process to promote fair, consistent, and objective evaluation. Certain initial screening steps may be automated to help identify... 
    Permanent employment
    Full time
    Internship
    Worldwide

    AON3D Careers

    Montréal, QC
    4 hours ago
  •  ...the rest of Americas and occasionally in Europe. As Practice Director, Shopify Commerce, you will own the technical vision, delivery excellence...  ..., Segment, mParticle), PIM, OMS, and payment gateways. Evaluate and stay ahead of emerging Shopify platform capabilities (e.g.,... 
    Long term contract
    Full time
    Temporary work
    Work at office
    Immediate start
    Trial period
    Flexible hours

    Valtech

    Montréal, QC
    4 hours ago
  • $140k - $160k per year

     ...Your new role   Reporting directly to the President, the Director of Investments will play a key role in the deployment of capital...  ...will oversee the investment process from sourcing and initial evaluation through financial analysis, due diligence, financing, negotiation... 
    Long term contract
    Permanent employment
    Work at office

    Hays

    Montréal, QC
    19 days ago
  • Notre client est � la recherche d'un directeur financier exp�riment� pour superviser la gestion financi�re et soutenir la prise de d�cision strat�gique de l�organisation au Qu�bec. Ce poste pilotera les principales op�rations comptables, veillera � l�int�grit� de l�information...
    Permanent employment
    Temporary work
    Work at office

    Robert Half

    Montréal, QC
    8 days ago
  • Position Description: À la tête d'une équipe de professionnels des technologies de l'information, vous veillez à la qualité de la livraison de CGI, à la satisfaction de nos clients et à l'épanouissement de votre équipe. Vous travaillez en étroite collaboration avec l'équipe...
    Full time

    CGI

    Montréal, QC
    9 days ago
  • Director, Sales Corporate Emploi - Directeur siège social - Saint-Laurent Québec H4R 1V1 Dès que possibleHoraire: Aucune information Description: Reporting to the President, Mitchel Lincoln/SVP Quebec Atlantic Packaging, the Director, Sales Operations (Corporate... 
    Permanent employment
    Contract work
    Internship
    Work at office

    Emballages Mitchel-Lincoln Ltée

    Montréal, QC
    5 days ago
  •  ...Superviser la comptabilité de projets et s’assurer que le suivi financier des mandats est exact, complet. We are seeking a FINANCE DIRECTOR to join an organization in downtown Montreal, Quebec, on a contract basis. This mandate, with a minimum duration of 4 months, follows... 
    Permanent employment
    Contract work
    Temporary work
    Work at office
    Day shift
    1 day per week

    Robert Half

    Montréal, QC
    8 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Director, Evaluations. Be the first to apply!