Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Health AI Conversation Evaluator

$20 - $160 per hour

Mercor

Toronto, ON
  • Remote job

About the job

Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .

Position: Generalist Annotator — Health AI Conversation Quality Evaluation

Type: Contract

Compensation: $20–$160/hour

Location: Remote

Role Responsibilities

  • Evaluate the relevance and clarity of AI responses . Ensure the AI answers the question in understandable language for non-experts.
  • Detect sycophancy in AI interactions. Identify when the AI tells users what they want to hear instead of what they need.
  • Assess the tone and outcome of conversations. Determine if the user leaves informed and calm or anxious and uninformed.
  • Provide written justification for ratings below the top option. Explain what went wrong and suggest improvements.
  • Work independently to review conversations ranging from 1 to 7 turns. Spend 2–13 minutes per conversation based on length.

Qualifications

Must-Have

  • Strong written English skills.
  • Ability to explain judgments in two to four specific sentences.
  • Careful reading to notice omissions in responses.
  • Comfort with applying a detailed rubric consistently.
  • Reliable availability for time-boxed batches.

Preferred

  • Experience in annotation, human-feedback, evaluation, or content review.
  • Background in explaining technical concepts to non-experts.
  • Experience with rubric-based grading or quality assurance.

Important

  • Using an AI tool to write comments is prohibited. This work captures human judgment that AI lacks. AI-written feedback corrupts the dataset and is checked for.

Application Process (Takes 20–30 mins to complete)

  • Upload resume
  • AI interview based on your resume
  • Submit form

Resources & Support

  • For details about the interview process and platform information, please check:
  • For any help or support, reach out to:

PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.

Vacancy posted a month ago
Similar jobs that could be interesting for youBased on the Health AI Conversation Evaluator in Toronto, ON vacancy
  •  ...through world-class networks, AI-powered solutions, and digital...  ...Summary Bell is seeking a Conversational AI Developer to join an innovative...  ...., ADK, MCP, A2A) as the team evaluates them. Test and troubleshoot...  ..., dental, vision and mental health benefits that you can tailor... 
    Suggested
    Full time
    Internship
    Work at office
    3 days per week

    Bell

    Toronto, ON
    7 days ago
  • $100 per hour

     ...connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our...  ...Summers , and Jack Dorsey . Position: Clinical Mental Health Expert — AI Conversation Evaluation Type: Contract Compensation: $100/hour... 
    Suggested
    Remote job
    Contract work
    Summer work

    Mercor

    Toronto, ON
    a month ago
  •  ...Engineering Manager, AI Conversation Platform Who we are About Stripe Stripe is a financial infrastructure platform for businesses. Millions of companies—from the world’s largest enterprises to the most ambitious startups—use Stripe to accept payments, grow their... 
    Suggested

    Stripe

    Toronto, ON
    1 day ago
  • $126k - $176k per year

     ...Sr. Legal AI Prompter / Evaluator Job Description Sr. Legal AI Prompters / Evaluators are responsible...  ...datasets with SMEs, coordinate evaluations, and design evaluation schemes, including...  ...flexible vacation, two company-wide Mental Health Days off, access to the Headspace app,... 
    Suggested
    Remplacement
    Full time
    Work at office
    Local area
    Flexible hours

    Thomson Reuters

    Toronto, ON
    6 days ago
  •  ...Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. This is a task-based...  ...clinical EHR vignettes for a medical AI evaluation project. General and internal medicine are the... 
    Suggested
    Part time
    For contractors
    Remote work
    Flexible hours

    Mindrift

    Toronto, ON
    3 days ago
  • $30 per hour

     ...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco our investors...  ...tasks guiding it answering questions and correcting errors. Evaluate the experience and deliverables providing concise feedback on performance... 
    Hourly pay
    Weekly pay
    Full time
    Contract work
    For contractors
    Summer work
    Remote work

    Mercor

    Toronto, ON
    6 days ago
  • $80 - $120 per hour

     ...connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors...  ...Larry Summers , and Jack Dorsey . Position: Public health communications Evaluator Type: Contract Compensation: $80–$120/hour... 
    Remote job
    Contract work
    Summer work
    Work at office

    Mercor

    Toronto, ON
    more than 2 months ago
  • $100 - $150 per hour

     ...creative and technical talent with leading AI research labs. Headquartered in San...  ...pipelines , and technical reports. Evaluate AI-generated or human-created work against...  ...Provide detailed written justifications for evaluations and scores to maintain transparency.... 
    Remote job
    Contract work
    Summer work

    Mercor

    Toronto, ON
    26 days ago
  • $140k - $175k per year

     ...delivery of production-ready software for artificial intelligence (AI) model evaluation and benchmarking. You will build scalable evaluation...  ...plans to include flexible vacation, two company-wide Mental Health Days off, access to the Headspace app, retirement savings, tuition... 
    Full time
    Internship
    Work at office
    Local area
    Flexible hours

    Thomson Reuters

    Toronto, ON
    10 days ago
  • $39 per hour

     ...creative and technical talent with leading AI research labs. Headquartered in San...  ...hours/week Role Responsibilities Evaluate AI model output lyrics , voice generation...  ...form Complete a 25-minute conversational interview on your background, experience,... 
    Remote job
    Contract work
    Summer work
    Immediate start

    Mercor

    Toronto, ON
    more than 2 months ago
  • $70 per hour

     ...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors...  ...hour Location: Remote Role Responsibilities Evaluate AI-generated responses to strengthen reasoning and rigor in model... 
    Remote job
    Contract work
    Summer work

    Mercor

    Toronto, ON
    more than 2 months ago
  • $80 - $120 per hour

     ...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors...  ..., and Jack Dorsey . Position: Software / AI / IT / data Evaluator Type: Contract Compensation: $80–$120/hour... 
    Remote job
    Contract work
    Summer work
    Work at office

    Mercor

    Toronto, ON
    more than 2 months ago
  • $70 per hour

     ...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors...  ...Upload your resume and application form. A 25-minute conversational interview covering your background, experience, and motivations.... 
    Remote job
    Contract work
    Summer work
    Immediate start

    Mercor

    Toronto, ON
    a month ago
  • $80 - $120 per hour

     ...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors...  ...Position: Compliance / regulatory response with financial-services AI Evaluator Type: Contract Compensation: $80–$120/hour... 
    Remote job
    Contract work
    Summer work
    Work at office

    Mercor

    Toronto, ON
    more than 2 months ago
  • $100 - $150 per hour

     ...creative and technical talent with leading AI research labs. Headquartered in San...  ...pipelines , and A/B test write-ups . Evaluate AI-generated or human-created work against...  ...detailed written justifications for evaluations and scores to maintain transparency and accountability... 
    Remote job
    Contract work
    Summer work

    Mercor

    Toronto, ON
    more than 2 months ago
  • $85 per hour

     ...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors...  ...Responsibilities Use frontier AI coding agents to complete and evaluate complex infrastructure engineering tasks. Review model-... 
    Remote job
    Contract work
    Summer work

    Mercor

    Toronto, ON
    more than 2 months ago
  • $80 per hour

     ...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors...  ...Responsibilities Use frontier AI coding agents to complete and evaluate complex data engineering tasks. Review model-generated... 
    Remote job
    Contract work
    Summer work

    Mercor

    Toronto, ON
    more than 2 months ago
  • $120 - $170 per hour

     ...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors...  ...Preferred ~ Prior experience with AI training , evaluation, or human-data projects. Application Process (Takes 20–30... 
    Remote job
    Contract work
    Summer work

    Mercor

    Toronto, ON
    more than 2 months ago
  • $52.38 - $57.39 per hour

     ...Branch of Parks, Forestry & Recreation, the Evaluator, Programs will help design, plan, and lead the implementation of small and large evaluations, performance measurement frameworks,...  ...anthropology, social sciences, ecology or public health or social services research or an... 
    Hourly pay
    Permanent employment
    Full time
    Part time
    Internship
    Monday to friday
    Shift work

    City of Toronto

    Toronto, ON
    7 days ago
  • $113k - $194k per year

     ...actuarial professionals working in 43 countries. AIS is part of Deloitte's Assurance & Advisory...  ...include: $4,000 per year for mental health support benefits, a $1,300 flexible benefit...  ...will utilize human screening combined with AI technology to help identify the skills and... 
    Permanent employment
    Flexible hours

    Deloitte

    Toronto, ON
    5 days ago
  • $250k per year

     ...is the leading security-first enterprise AI company. We build cutting-edge foundation...  ...and Seoul. Join us! Role Overview: Evaluation is critical to making progress in scaling...  ...in your local currency for lunch. ~ Full health and dental benefits, including a separate... 
    Full time
    Work at office
    Local area
    Remote work
    Home office

    Cohere

    Toronto, ON
    7 days ago
  • $39 per hour

     ...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors...  ...experience and deliverables, providing concise notes on performance. Evaluate AI model interactions to ensure alignment with professional... 
    Remote job

    Mercor

    Toronto, ON
    10 days ago
  • $100 per hour

     ...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors...  ...grading rubrics with pass/fail criteria and judgment calls. Evaluate AI model attempts and confirm failures for fair, defensible reasons... 
    Remote job
    Hourly pay
    Weekly pay
    Contract work
    For contractors
    Summer work

    Mercor

    Toronto, ON
    2 days ago
  • $39 per hour

     ...About the job Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco our investors include Benchmark General Catalyst Peter Thiel Adam DAngelo Larry Summers and Jack Dorsey . Position: Healthcare... 
    Full time
    Contract work
    Summer work
    Remote work

    Mercor

    Toronto, ON
    6 days ago
  • $30 - $40 per hour

     ...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco our investors...  ...outputs . Complete assigned queries each day to ensure timely evaluation. Provide short notes and annotations to enhance training data... 
    Daily paid
    Full time
    Contract work
    Summer work
    Remote work
    Flexible hours

    Mercor

    Toronto, ON
    6 days ago
  • $50 per hour

     ...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors...  ...hour Location: Remote Role Responsibilities Evaluate the quality and accuracy of LLM-generated French text across various... 
    Remote job
    Contract work
    Summer work

    Mercor

    Toronto, ON
    20 days ago
  • $42 per hour

     ...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors...  ...sessions with experienced tutors. Engage in natural, substantive conversations in Spanish based on assigned topics and lesson structures.... 
    Remote job
    Contract work
    Summer work

    Mercor

    Toronto, ON
    more than 2 months ago
  • $150 per hour

     ...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors...  ...Commitment: 10–20 hours/week Role Responsibilities Evaluate and compare AI-generated responses on challenging medical problems... 
    Remote job
    Contract work
    Summer work
    Immediate start

    Mercor

    Toronto, ON
    21 days ago
  • $57 per hour

     ...About the job Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey . Position... 
    Remote job
    Hourly pay
    Contract work
    For contractors
    Summer work

    Mercor

    Toronto, ON
    16 days ago
  •  ...Culture Align with the YNSE Project Health, Safety and Wellbeing (HSW) strategies and...  ...Collect and maintain records required for evaluating the effectiveness of safety programs. Compile...  ...and includes the feminine. We use AI-enabled tools in our recruitment platform... 
    For contractors
    Toronto, ON
    3 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Health AI Conversation Evaluator. Be the first to apply!