Applied Ai Evaluation & Product Researcher (Contract)

Applied AI Evaluation & Product Researcher (Contract)Applied Research FT Contract Toronto, Toronto, ON, CA
Salary Range: $95,000.00 To $110,000.00 Annually
Job title
Applied AI Evaluation & Product Researcher (12 Month Contract)
Department
Reports to
Director, Applied Research
Salary range
$95,000 - $110,000
Job type
Full-Time, Contract, Hybrid working 1-2 days per week
Location
The opportunity
Are you motivated by the chance to help ensure all young people in Canada can access mental health support and Feel Out Loud? Kids Help Phone (KHP) is Canada’s Unlock The Hope Youth Charity. We’re the first and Canada's only national 24/7, free, confidential, and multilingual e-mental health service for youth and a global leader in innovation for youth and their well-being.
KHP is looking for an Applied AI Evaluation & Product Researcher to lead product evaluation and user‑centred research for its youth-facing generative AI product. This first‑of‑its‑kind experience will offer low‑intensity support through AI, incorporating human oversight informed by KHP’s 37+ years of supervising quality mental health support at scale.
Working with Data and Technology, Innovation, Applied Research, Clinical Services, and other internal teams, the researcher will develop and implement an evaluation strategy that defines success, examines user experience and risk, and translates findings into practical recommendations. They will also support coordination across teams and help develop clinical governance mechanisms for the initiative. It’ll take courage, collaboration, innovation, resilience and trust. If you’re interested in transforming e‑mental health solutions with KHP, you can explore this opportunity today!
The responsibilities
Evaluation Strategy and Measurement
Develop and implement a comprehensive evaluation strategy for the system, from early testing through launch and post-launch monitoring
Create and maintain an evaluation matrix linking objectives, user scenarios, outcomes, indicators, data sources, timing, and decision criteria
Define measures of success for service users including user experience, adoption, trust, safety, equity, operational performance, and impact
Design evaluation activities and coordinate the collection, analysis, and interpretation of appropriate data
Establish processes for ongoing monitoring, learning, and continuous improvement, and communicate findings to project teams and leadership
Coordinate evaluation activities across internal workstreams and ensure that key evaluation decisions, findings, risks, dependencies, and resulting actions are documented and communicated to the appropriate teams
User Experience Research
Support user research to understand user needs, behaviours, expectations, barriers, and experiences across the end-to-end service journey
Design and conduct post-interaction surveys, usability testing, journey mapping, co-design activities, and other appropriate research methods
Ensure research activities are inclusive, trauma-informed, accessible, and appropriate for the intended users and service context
Translate findings into clear recommendations for product design, user flows, service pathways, content, and implementation
Safety, Risk and Responsible AI
Co‑develop evaluation criteria, testing approaches, monitoring requirements, and escalation pathways for identified risks(e.G., user, clinical, operational, ethical, privacy, accessibility, and equity‑related risks)
Assess unintended outcomes, inappropriate or unclear recommendations, potential bias, and other issues that may affect user safety or access to support
Contribute research evidence to responsible AI, privacy, and data governance processes and decisions
Collaborate with designated clinical leaders and subject‑matter experts to ensure evaluation criteria reflect relevant clinical standards, service practices, and safety considerations
Implementation and Launch Readiness
Work with internal teams to define launch‑readiness criteria, decision points, evidence requirements, and timelines, and provide evaluation findings to support decisions by accountable product, operational, and clinical leaders
Identify user‑experience issues that may affect implementation and recommend strategies for improvement
Support iterative testing throughout design, development and implementation and evaluate whether changes address identified user needs and risks
Reporting and Knowledge Translation
Synthesize complex research and evaluation findings into practical recommendations, reports, presentations, briefings, and decision materials
Conduct literature reviews, environmental scans, and evidence syntheses relevant to Generative AI, digital services, user experience, safety, and implementation
Communicate research limitations, emerging risks, and evidence gaps clearly to technical and non‑technical audiences
The qualifications
PhD degree (or Master’s degree with equivalent experience) in Human-Computer Interaction, User Experience Research, Psychology, Social Sciences, Evaluation, Implementation Science, Health Informatics, or a related field
Minimum 5 years of experience in generative AI systems evaluation, UX research, product research and evaluation, applied research, evaluation, digital service design, or a related area
Strong knowledge of responsible AI, AI safety, risk assessment, algorithmic evaluation, or technology governance
Strong experience with qualitative and quantitative research methods, including surveys, usability testing, and analysis
Experience developing evaluation frameworks, measurement strategies, success indicators, logic models, or evaluation matrices
Experience translating research findings into actionable product, service, or implementation recommendations
Experience collaborating with multidisciplinary teams that may include product, design, technology, data, research, and clinical partners
Strong project coordination, facilitation, stakeholder engagement, and written and verbal communication skills
Ability to manage multiple priorities, work through ambiguity, and communicate risks, trade‑offs, and recommendations clearly
Experience working in mental health, youth services, healthcare, crisis support, or another safety‑sensitive environment
Familiarity with clinical standards, privacy, accessibility, equity, and data governance considerations
Experience supporting product pilots, implementation readiness, launch activities, and post‑launch monitoring
Familiarity with research, analytics, prototyping, or product collaboration tools
Don’t meet every qualification listed? Studies often report equity‑deserving people are less likely to apply for jobs unless they meet every requirement. KHP is dedicated to building a more diverse, inclusive and authentic workplace, so if you’re excited about this role, but your experience doesn’t align with every qualification, we welcome you to apply anyway. You may have the qualifications / experience for this role or other opportunities at KHP!
About KHP, Canada’s Unlock The Hope Youth Charity
Kids Help Phone (KHP) is Canada’s only free, national, 24/7, multilingual and confidential e-mental health service for youth to get help and Feel Out Loud. Since 1989, KHP has been the empowering, trusted changemaker giving young people’s feelings, big and small, a non‑judgmental place to go through personalized well‑being solutions. A made‑in‑Canada, global leader, KHP continually evolves by blending data, innovation and technology with real human support and clinical knowledge to unlock the hope young people need to thrive in their world. Explore help for all the feelings at KidsHelpPhone.Ca .
Inclusion, Diversity, Equity and Accessibility (IDEA)
KHP has a role to play in effecting social change and advancing equity. This is why the practice of IDEA is critical to us. KHP has a long history of celebrating and advancing IDEA through our tailored programs, services, policies, initiatives and task forces with contributions and leadership from teams across the organization. We will not waver on these commitments and will continue leading the way in making the workplace, and the world young people live in, more inclusive, diverse, equitable and accessible.
We believe it’s critical that our services empower everyone who interacts with us to bring their authentic selves to an environment of mutual respect. An open and diverse community can foster the inclusion and amplification of voices that have been, and continue to be, underrepresented.
As such, we welcome applicants from equity‑deserving communities (e.G. Indigenous Peoples, racialized persons, persons with disabilities, women, 2SLGBTQ+ individuals, etc.). We strongly believe that alleviating the underrepresentation of equity‑deserving groups will help create a stronger KHP and help us better support the needs of youth from coast to coast to coast.
KHP welcomes applications from people living with disabilities. Reasonable accommodations are available upon request during any phase of the application / hiring process. If you’re contacted about an opportunity, you can reply and let us know if you require accommodations at any time.
Compensation and benefits
Salaries at KHP are determined by candidate experience, qualifications and specific skill sets, as well as market and operational considerations. Potential additional benefits at KHP include:
Emergency Travel Assistance
Wellness programs
Recognition awards
Employee and Family Assistance Program (EFAP)
Candidate verification
The successful candidate for this opportunity will be asked to complete a background screen, which may include criminal, credit, employment and educational checks. All offers of employment are conditional on the receipt of satisfactory results of any applicable check.
Privacy
KHP cares about your privacy! We’re required to save select information about opportunities, applications and interviews for set periods of time. To learn more, you can explore KHP’s Privacy Policy or contact us using our general form .
#J-18808-Ljbffr