Director, Evaluations
LawZero
LawZero is a non-profit building safe-by-design AI systems. We’re building the Scientist AI, an advanced AI system designed from the ground up to be both highly capable and safe. As we develop both general‑purpose Scientist AI models and safety guardrails for frontier LLMs, we need rigorous, independent evaluation of every capability and safety claim we make. We are looking for a Director of Evaluations to build, lead, and grow LawZero’s Evaluations Team.
This is a foundational hire. You will define what world‑class evaluation looks like at LawZero, build the team and infrastructure to deliver it, and ensure that evaluations remain independent of the main research stream so that capability and safety claims can be trusted both internally and externally by the wider AI and AI safety community.
Key responsibilities
- Define LawZero’s evaluations strategy and roadmap, prioritising what needs to be measured and when, in close coordination with both research and product teams.
- Build up the Evaluations Team during your first 3–6 months, scaling to roughly 8–10 people across research, engineering, dataset and benchmark design, and red‑teaming.
- Operate the team independently of the main research and product streams in order to avoid conflicts of interest, including designing novel benchmarks that can be applied apples‑to‑apples to evaluate both the Scientist AI and frontier LLMs.
- Oversee the design and construction of new datasets, tasks, and virtual or interactive environments to measure performance of the Scientist AI across capabilities, safety (including honesty and goal-directedness), explainability, causal mechanisms and detecting adversarial attacks.
- Lead evaluation of the Scientist AI when deployed as a guardrail around frontier models, including its ability to comply with harm specifications, detect and block harmful responses, explain its decisions, and resist adversarial attacks such as jailbreaks, prompt injection, and data poisoning.
- Establish and lead our automated and manual red‑teaming programmes, both in‑house and in partnership with external providers, to stress test the Scientist AI as a general‑purpose model and as a guardrail.
- Lead the construction of internal tooling and infrastructure needed to run evaluations at scale, automating and standardizing the pipeline wherever possible.
- As needed and where possible, directly support research and product streams with their own internal requirements w.r.t. evaluations and benchmarking to unblock and speed up.
- Own LawZero’s public communication of evaluation results, including model and system cards, technical reports, peer‑reviewed publications and blog posts, to build trust with the wider AI safety community.
- Represent LawZero externally on evaluations and AI safety measurement, including engagements with AI safety institutes, research collaborators, and grant funders.
Skills and qualifications
- An advanced degree (MSc or higher) in machine learning, computer science, or a closely related field.
- 10+ years of experience in machine learning, with at least 5 years in a leadership role building or scaling technical teams working on real-world ML products.
- Hands‑on expertise in designing and running large‑scale evaluations of LLMs or other frontier ML systems across capabilities, safety, and adversarial robustness.
- A track record of building evaluation datasets, benchmarks, or interactive environments from scratch, including for safety‑relevant properties such as honesty, sycophancy, refusal behaviour, and adversarial robustness.
- Strong written and verbal communication skills, including the ability to translate technical results for non‑technical audiences such as executives, funders, and policymakers.
- Comfortable operating in a research‑driven, fast‑moving environment with significant ambiguity, and able to bring structure to it without slowing it down.
Nice to have:
- Experience leading red‑teaming exercises (automated, manual, or both) and working with third‑party evaluation or red‑teaming partners is a bonus.
- Experience working with third-party partners for benchmark and dataset creation
- Experience releasing open‑source datasets, benchmarks, or evaluation tooling is a bonus.
- Familiarity with current AI safety policy and standards work (UK AISI, CAISI, NIST, EU AI Act, etc.) is a bonus.
- Experience contributing to or coordinating with external safety institutes, grant funders, or government bodies is a bonus.
What we offer
- The opportunity to contribute to a unique mission with a major impact
- Comprehensive health benefits
- A minimum of 20 days vacation per year upon start
- A minimum retirement savings employer contribution of 4%
- Generous flexible benefits designed to contribute to your well-being
- A team of passionate experts in their field
- A collaborative and inclusive work environment with offices in the heart of Little Italy, in the trendy Mile-Ex district, close to public transportation.
About LawZero
LawZero is a non-profit organization committed to advancing research and creating technical solutions that enable safe-by-design AI systems. Its scientific direction is based on new research and methods proposed by Professor Yoshua Bengio, the most cited AI researcher in the world. Based in Montreal, LawZero’s research aims to build AI that learns primarily to understand the world rather than to act in it, giving truthful answers to questions based on transparent and externalized probabilistic reasoning. Such AI systems could be used to accelerate scientific discovery, to provide oversight for agentic AI systems, and to advance the understanding of AI risks and how to avoid them. LawZero believes that AI should be cultivated as a global public good—developed and used safely towards human flourishing. For more information, visit
You belong here
At LawZero, diversity is important to us. We value a work environment that is fair, open and respectful of differences. We welcome applications from highly qualified individuals interested in working towards our mission in a respectful, inclusive and collaborative setting.
Your personal information will be collected and processed by LawZero to evaluate your application for employment in compliance with our Privacy Policy . Under privacy laws in force in your country of residence, you may have several privacy rights, such as to request access to your personal information or to request that your personal information be rectified or erased. Details on how you can exercise your rights can be found in our Privacy Policy.
- ...de nos modèles généraux Scientist AI et de garde-fous de sécurité pour les modèles de langage de pointe (LLM frontier ), nous devons évaluer de manière rigoureuse et indépendante chaque affirmation concernant leurs capacités et leur sécurité. Nous recherchons un.e Directeur...SuggestedApprenticeshipWork at officeFlexible hoursDay shift
- ...Titre du poste : Évaluateur(trice) immobilier / Appraiser Lieu : Montréal Salaire : selon expérience À propos du poste Joindre cette opportunité, c'est intégrer une organisation multidisciplinaire bien établie dans le secteur de l'évaluation et des services immobiliers...SuggestedWork at officeFlexible hours
- ...développement de projets majeurs en énergie, infrastructures et ressources naturelles. Nous recherchons actuellement un(e) leader senior en évaluation environnementale souhaitant jouer un rôle clé dans la réalisation et le positionnement de projets structurants au Québec et à l’...SuggestedLocal areaFlexible hoursDay shift
$72k - $108k per year
...coentreprises et de financement de projet; Planification des dépenses en immobilisations et analyses de sensibilité; Soutien à l'évaluation d'entreprise et aux évaluations liées à la présentation des états financiers; Revue et validation de modèles financiers....SuggestedPermanent employmentApprenticeshipWork at officeLocal areaFlexible hoursDay shift- ...opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. This is a task-based, independent contractor... ...license to evaluate clinical EHR vignettes for a medical AI evaluation project. General and internal medicine are the main focus. Each...SuggestedPart timeFor contractorsRemote workFlexible hours
- .... Atouts Une expérience en réadaptation ambulatoire auprès dune clientèle ayant une déficience physique; Une expérience en évaluation des capacités fonctionnelles. Vous êtes nouvellement diplômé(e) Votre candidature nous intéresse également. Vous pourrez développer...Full time
- Votre opportunité Axés sur la santé, la sécurité, la qualité et l’éthique, nos experts dirigent la réalisation de projets avec de la détermination, un esprit créatif et une vision de croissance. Présents partout dans le monde et comptant des professionnels dans plus de 20...Full timeContract workTemporary workPart timeWork at office
- ...permettre d’atteindre leur plein potentiel, et leur servir de modèle. Renseignements sur l’équipe Les Services-conseils en évaluation d’entreprises et modélisation se spécialisent dans la résolution de problèmes complexes liés aux affaires et aux transactions, y compris...Permanent employmentApprenticeshipWork at officeLocal areaFlexible hoursDay shift
- ...Évaluateur(trice) en santé et sécurité – Bilingue Department: Operations Employment Type: Permanent Location: Greater Montreal Metropolitan Area, Canada Reporting To: Chef de l'accréditation en santé et sécurité Description Créer des solutions innovantes...Permanent employmentApprenticeshipWork at office
- ...to apply your relevant experience to a new industry, join our team as we help shape a brighter way forward. Conseiller principal, Évaluation - JLL En quoi consiste cet emploi : Joignez-vous à l'équipe d’Évaluation chez JLL à titre de Conseiller(e) principal(e) et devenez...Daily paidFull timeTemporary workWork at officeRemote work
- ...Conseiller en gestion de programmes au sein du Décanat des études de l’ÉTS? Vous soutiendrez le doyen des études dans la création, l’évaluation et l’amélioration continue des programmes d’études, en accompagnant les directions des départements, les directions de programmes,...Daily paidFull timeTemporary work
- ...Localisation: Partout au Québec Nous recherchons un(e) spécialiste en milieux humides et végétation, possédant une expertise en évaluation environnementale. La personne retenue apportera un soutien technique pour les études environnementales, les évaluations d'impact et...Full timeFlexible hours
- ...quotidien sera de gérer et de coordonner la production tant des études d’impact sur l’environnement (ÉIE), que des autres types d’évaluations environnementales et de demandes d’autorisation et de permis. Vous devrez être familier avec les législations, réglementations et...Daily paidWork at office
- ...consiste principalement à prendre des décisions judicieuses en tirant parti d’informations exactes. En tant que membre de l’équipe Évaluation, voilà ce que vous contribuerez à rendre possible. Vous serez un membre important de notre équipe diversifiée et hautement expérimentée...Daily paidWeekend work
- ...alongside iconic brand names. About the Role: As a luxury brand evaluator, you will step into the world of luxury to discreetly assess... ...in the luxury market. Compensation: • Non-Purchase Evaluations: Earn a fee based on mission complexity. • Purchase-Based Evaluations...Contract workWorldwideFlexible hours
$36 per hour
...such as sauna, steam room, and jacuzzi. We are looking for a Spa Director to ensure the planning, organization, and supervision of the... ...of the establishment. Plan and organize daily operations. Evaluate daily operations and propose improvements. Develop and implement...Hourly payPermanent employmentFull timeShift workNight shiftWeekend workDay shiftAfternoon shift- Vos missions au quotidien L’équipe Appétit au risque et évaluation des risques fait partie de la fonction Gestion intégrée des risques de SG Americas et joue un rôle central dans le soutien des activités d’identification, d’évaluation, de surveillance, de reddition...Daily paid
- ...transaction discipline and an ownership mindset to move opportunities from initial evaluation through closing, integration and value realization. What you will lead as the Managing Director, Corporate Development Define and advance the M&A strategy in alignment with...Long term contractLocal area
- ...material, with the touch of a button. About The Role As Director of Marketing, you will play a key role in leading AON3D’s marketing... ...the hiring process to promote fair, consistent, and objective evaluation. Certain initial screening steps may be automated to help...Permanent employmentFull timeInternshipWorldwide
$155k - $200k per year
...the buzz is about! Role Overview We’re looking for a Growth Director to own how Clipwire’s games find and keep their players. You’ll... ...to screen applications or make hiring decisions. All candidate evaluations and hiring decisions are made by people. This opening is for...Long term contractCasual workWork at officeFlexible hoursAfternoon shift- ...pour nous propulser vers les meilleurs résultats pour la population du Québec. Description d'emploi Au sein de l’équipe ’Évaluation, la personne titulaire de ce rôle évalue des placements directs des portefeuilles infrastructures et placements privés pour les états...Daily paidPermanent employmentFull timeWork at office
- Nous sommes � la recherche d�un directeur de la comptabilit� chevronn� pour diriger les activit�s d�information financi�re d�une organisation bancaire situ�e � Montr�al. Dans ce r�le, vous superviserez l�exactitude et la production en temps opportun des rapports financiers internes...Permanent employmentContract workTemporary workWork at office
- ...portfolio of vertical market software businesses, we are seeking a dynamic and experienced professional to join our team as Managing Director. Reporting directly to one of our Investment Partners, the Managing Director will be responsible for managing and growing their businesses...Long term contractPermanent employmentFull timeImmediate start
- ...the touch of a button. About The Role AON3D is seeking a Director, Customer Success to own the end-to-end post-sale customer... ...the hiring process to promote fair, consistent, and objective evaluation. Certain initial screening steps may be automated to help identify...Permanent employmentFull timeInternshipWorldwide
- ...the rest of Americas and occasionally in Europe. As Practice Director, Shopify Commerce, you will own the technical vision, delivery excellence... ..., Segment, mParticle), PIM, OMS, and payment gateways. Evaluate and stay ahead of emerging Shopify platform capabilities (e.g.,...Long term contractFull timeTemporary workWork at officeImmediate startTrial periodFlexible hours
$140k - $160k per year
...Your new role Reporting directly to the President, the Director of Investments will play a key role in the deployment of capital... ...will oversee the investment process from sourcing and initial evaluation through financial analysis, due diligence, financing, negotiation...Long term contractPermanent employmentWork at office- Notre client est � la recherche d'un directeur financier exp�riment� pour superviser la gestion financi�re et soutenir la prise de d�cision strat�gique de l�organisation au Qu�bec. Ce poste pilotera les principales op�rations comptables, veillera � l�int�grit� de l�information...Permanent employmentTemporary workWork at office
- Position Description: À la tête d'une équipe de professionnels des technologies de l'information, vous veillez à la qualité de la livraison de CGI, à la satisfaction de nos clients et à l'épanouissement de votre équipe. Vous travaillez en étroite collaboration avec l'équipe...Full time
- Director, Sales Corporate Emploi - Directeur siège social - Saint-Laurent Québec H4R 1V1 Dès que possibleHoraire: Aucune information Description: Reporting to the President, Mitchel Lincoln/SVP Quebec Atlantic Packaging, the Director, Sales Operations (Corporate...Permanent employmentContract workInternshipWork at office
- ...Superviser la comptabilité de projets et s’assurer que le suivi financier des mandats est exact, complet. We are seeking a FINANCE DIRECTOR to join an organization in downtown Montreal, Quebec, on a contract basis. This mandate, with a minimum duration of 4 months, follows...Permanent employmentContract workTemporary workWork at officeDay shift1 day per week
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Director, Evaluations. Be the first to apply!
- directeur fondation Montréal, QC
- directeur directrice Montréal, QC
- directeur recouvrement Montréal, QC
- director of learning and development Montréal, QC
- directeur de succursale Montréal, QC
- youth director Montréal, QC
- board of directors positions Montréal, QC
- directeur amelioration continue audit qualite Montréal, QC
- directeur architecture Montréal, QC
- director government relations Montréal, QC

