Health AI Conversation Evaluator

$20 - $160 per hour

Mercor

Remote job

About the job

Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .

Position: Generalist Annotator — Health AI Conversation Quality Evaluation

Type: Contract

Compensation: $20–$160/hour

Location: Remote

Role Responsibilities

Evaluate the relevance and clarity of AI responses . Ensure the AI answers the question in understandable language for non-experts.
Detect sycophancy in AI interactions. Identify when the AI tells users what they want to hear instead of what they need.
Assess the tone and outcome of conversations. Determine if the user leaves informed and calm or anxious and uninformed.
Provide written justification for ratings below the top option. Explain what went wrong and suggest improvements.
Work independently to review conversations ranging from 1 to 7 turns. Spend 2–13 minutes per conversation based on length.

Qualifications

Must-Have

Strong written English skills.
Ability to explain judgments in two to four specific sentences.
Careful reading to notice omissions in responses.
Comfort with applying a detailed rubric consistently.
Reliable availability for time-boxed batches.

Preferred

Experience in annotation, human-feedback, evaluation, or content review.
Background in explaining technical concepts to non-experts.
Experience with rubric-based grading or quality assurance.

Important

Using an AI tool to write comments is prohibited. This work captures human judgment that AI lacks. AI-written feedback corrupts the dataset and is checked for.

Application Process (Takes 20–30 mins to complete)

Upload resume
AI interview based on your resume
Submit form

Resources & Support

For details about the interview process and platform information, please check:
For any help or support, reach out to: Show email

PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.

Vacancy posted 22 days ago

Similar jobs that could be interesting for youBased on the Health AI Conversation Evaluator in Toronto, ON vacancy

Engineering Manager, AI Conversation Platform
...Engineering Manager, AI Conversation Platform Who we are About Stripe Stripe is a financial infrastructure platform for businesses. Millions of companies—from the world’s largest enterprises to the most ambitious startups—use Stripe to accept payments, grow their...
Suggested
Full time
Stripe, Inc.
Toronto, ON
more than 2 months ago
Product Manager Intern - Alfie (Conversational AI)
...at your fingertips. With our AI-powered Global Work Platform™... ...Alfie is Pebl's proprietary conversational AI — built in-house and live... ...contexts? What does a great AI conversation actually look like for our users... ...for growing families. Health and Dental Insurance – Where...
Suggested
Summer work
Internship
Local area
Flexible hours
Pebl
Toronto, ON
more than 2 months ago
Conversational AI Chat Bot Developer, Cognigy
...the same, a place where employees know they can thrive. The AI Developer is responsible for the development, testing and delivery... .... What You'll Be Doing Developing and enhancing conversational chat bot solutions in relation to Cognigy and NICE Proactive AI...
Suggested
Remote job
Full time
Ttec Digital
Toronto, ON
more than 2 months ago
AI Evaluation Engineer (QA)
...collaboration with other disciplines - all foundational ingredients in successful digital experiences and client partnerships. As a QA / AI Evaluation Engineer, you will join a highly motivated and experienced team in a forward-leaning role that proves the platform actually...
Suggested
Full time
Immediate start
Appnovation Technologies
Toronto, ON
a month ago
Financial Compliance AI Evaluator
$80 - $120 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...Position: Compliance / regulatory response with financial-services AI Evaluator Type: Contract Compensation: $80–$120/hour...
Suggested
Full time
Contract work
Summer work
Work at office
Remote work
Mercor
Toronto, ON
more than 2 months ago
Compliance AI Evaluator - Finance
$80 - $120 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...Position: Compliance / regulatory response with financial-services AI Evaluator Type: Contract Compensation: $80–$120/hour...
Full time
Contract work
Summer work
Work at office
Remote work
Mercor
Toronto, ON
more than 2 months ago
Data Scientist - AI Evaluation
$100 - $150 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...pipelines , and A/B test write-ups . Evaluate AI-generated or human-created work against... ...detailed written justifications for evaluations and scores to maintain transparency and accountability...
Full time
Contract work
Summer work
Remote work
Mercor
Toronto, ON
more than 2 months ago
Clinical Psychologist - Conversation Evaluator
$100 per hour
...connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our... ...Summers , and Jack Dorsey . Position: Clinical Mental Health Expert — AI Conversation Evaluation Type: Contract Compensation: $100/hour...
Remote job
Contract work
Summer work
Mercor
Toronto, ON
11 days ago
DevOps Engineer - AI Model Evaluator
$85 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...Responsibilities Use frontier AI coding agents to complete and evaluate complex infrastructure engineering tasks. Review model-...
Full time
Contract work
Summer work
Remote work
Mercor
Toronto, ON
more than 2 months ago
Data Engineer - AI Model Evaluator
$80 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...Responsibilities Use frontier AI coding agents to complete and evaluate complex data engineering tasks. Review model-generated...
Full time
Contract work
Summer work
Remote work
Mercor
Toronto, ON
a month ago
Document Conversion Specialist - Fully Remote
$60 - $150 per hour
...About the job Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey . Position...
Full time
Contract work
Summer work
Remote work
Flexible hours
Mercor
Toronto, ON
more than 2 months ago
Director of AI
..., CPG, Sports, and Media with AI-native delivery capability, designing... ...of approaches. Establish evaluation-driven development practices,... ...comfortably between executive conversations, product decisions,... ...value your time, safety, and health, and strive to build a work community...
Full time
Work at office
Remote work
Flexible hours
E2x Is Now Part Of Apply Digital
Toronto, ON
28 days ago
Software Engineering Evaluator - Expert
$100 - $150 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...hour Location: Remote Role Responsibilities Evaluate AI-generated artifacts for usability in a software engineering...
Contract work
Summer work
Work at office
Remote work
Mercor
Toronto, ON
18 days ago
Senior Customer Experience Consultant - Conversational Design
...they can thrive. TTEC Digital seeks an innovative and creative Conversational UI Dialog Designer to assist in the design, implementation, and... ...known as virtual agents that leverage Artificial Intelligence (AI). As a Conversational Designer , you will be responsible for...
Full time
Remote work
Ttec Digital
Toronto, ON
more than 2 months ago
AI Product Lead
...Description Role Overview The AI Lead is responsible for... ...prototypes to drive customer conversations · Validate solutions through... ...opportunities and revenue potential · Evaluate ideas based on impact,... ...structure based on performance. · Health insurance, 401(k) with company...
Full time
Valsoft
Toronto, ON
a month ago
AI Linguistic Evaluator - Telugu
usd16 per hour
...We are looking for AI Linguistic Evaluators to assess how effectively an AI application performs in Telugu for a short collaboration. Job Type... ...naturalness, and cultural appropriateness. - Providing structured evaluations and feedback following the project instructions and quality...
Hourly pay
Daily paid
Full time
Freelance
Immediate start
Remote work
Flexible hours
Rws
Toronto, ON
5 days ago
Rubric Evaluation Specialist - Writing
$25 - $45 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...Role Responsibilities Evaluate AI-generated written conversations using predefined rubrics to enhance... ...performance. Draft and refine evaluation rubrics for generalist writing tasks...
Full time
Contract work
Summer work
Remote work
Mercor
Toronto, ON
more than 2 months ago
Bilingual Evaluator - LLM Specialist
$15 - $20 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...tools . Generate high-quality human evaluation data by identifying response strengths, areas... ...model responses align with expected conversational behavior and system guidelines. Work...
Full time
Contract work
Summer work
Remote work
Mercor
Toronto, ON
more than 2 months ago
Physician (MD/DO) - Medical AI Evaluation
$130 per hour
...Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is... ...evaluate clinical EHR vignettes for a medical AI evaluation program. General and internal medicine are our...
Hourly pay
Permanent employment
Part time
10 hours per week
Mindrift
Toronto, ON
1 day ago
Mathematics PhD - AI Evaluation Expert
$70 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...Upload your resume and application form. A 25-minute conversational interview covering your background, experience, and motivations....
Remote job
Contract work
Summer work
Immediate start
Mercor
Toronto, ON
17 days ago
Video Evaluation Specialist - Annotator
$15 - $20 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...Angelo , Larry Summers , and Jack Dorsey . Position: Video Evaluation Generalist Type: Contract Compensation: $15–$...
Full time
Contract work
Summer work
Remote work
Mercor
Toronto, ON
a month ago
Evaluation Specialist - Fully Remote
$80 - $160 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...Utilize strong analytical skills to contribute to high-impact evaluation projects. Collaborate with teams to improve document standards...
Full time
Contract work
Summer work
Remote work
Mercor
Toronto, ON
a month ago
Business Operations Expert - Evaluator
$60 - $80 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...Jack Dorsey . Position: Business Domain Expert — AI Model Evaluation Pilot (Admin, Marketing, HR, Accounting) Type: Contract...
Hourly pay
Full time
Contract work
For contractors
Summer work
Remote work
Mercor
Toronto, ON
a month ago
Data Scientist - AI Evaluation
$100 - $150 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...pipelines , and technical reports. Evaluate AI-generated or human-created work against... ...Provide detailed written justifications for evaluations and scores to maintain transparency....
Remote job
Contract work
Summer work
Mercor
Toronto, ON
26 days ago
Legal Compliance Expert - Evaluator
$80 - $120 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...Summers , and Jack Dorsey . Position: Legal / compliance Evaluator Type: Contract Compensation: $80–$120/hour...
Full time
Contract work
Summer work
Work at office
Remote work
Mercor
Toronto, ON
more than 2 months ago
Sales Evaluator - GTM Specialist
$80 - $120 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...Summers , and Jack Dorsey . Position: General Sales / GTM Evaluator Type: Contract Compensation: $80–$120/hour...
Full time
Contract work
Summer work
Work at office
Remote work
Flexible hours
Mercor
Toronto, ON
more than 2 months ago
Lead Product Manager - AI/ML
$130k - $155k per year
...strong technical understanding of AI, machine learning, and data... ..., business, and customer conversations Comfortable working in a growing... ...work with technical teams to evaluate model performance and customer... ...Based Bonus & Paid Time Off Health & Wellness Benefits: Medical,...
Long term contract
Full time
Shift work
3 days per week
Stack It Recruitment
Toronto, ON
a month ago
Hindi Music Specialist - Evaluator
$14 - $42 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...Commitment: 20+ hours/week Role Responsibilities Evaluate AI-generated music across various genres. Rate against detailed quality...
Full time
Contract work
Summer work
Immediate start
Remote work
Mercor
Toronto, ON
a month ago
Office Document Evaluation Specialist
$70 - $100 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...Commitment: 20–40 hours/week Role Responsibilities Evaluate AI-generated office artifacts in Excel , Word , and PowerPoint...
Full time
Contract work
Summer work
Work at office
Immediate start
Remote work
Mercor
Toronto, ON
more than 2 months ago
Excel Specialist - Document Evaluator
$80 - $150 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...Position: MS Excel / Google Sheets Evaluator Type: Contract Compensation... ...structured feedback to enhance document evaluation processes. Utilize MS Excel and Google...
Full time
Contract work
Summer work
Remote work
Mercor
Toronto, ON
more than 2 months ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Health AI Conversation Evaluator. Be the first to apply!