Health AI Conversation Evaluator
$20 - $160 per hourMercor
Toronto, ON
- Remote job
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .
Position: Generalist Annotator — Health AI Conversation Quality Evaluation
Type: Contract
Compensation: $20–$160/hour
Location: Remote
Role Responsibilities
- Evaluate the relevance and clarity of AI responses . Ensure the AI answers the question in understandable language for non-experts.
- Detect sycophancy in AI interactions. Identify when the AI tells users what they want to hear instead of what they need.
- Assess the tone and outcome of conversations. Determine if the user leaves informed and calm or anxious and uninformed.
- Provide written justification for ratings below the top option. Explain what went wrong and suggest improvements.
- Work independently to review conversations ranging from 1 to 7 turns. Spend 2–13 minutes per conversation based on length.
Qualifications
Must-Have
- Strong written English skills.
- Ability to explain judgments in two to four specific sentences.
- Careful reading to notice omissions in responses.
- Comfort with applying a detailed rubric consistently.
- Reliable availability for time-boxed batches.
Preferred
- Experience in annotation, human-feedback, evaluation, or content review.
- Background in explaining technical concepts to non-experts.
- Experience with rubric-based grading or quality assurance.
Important
- Using an AI tool to write comments is prohibited. This work captures human judgment that AI lacks. AI-written feedback corrupts the dataset and is checked for.
Application Process (Takes 20–30 mins to complete)
- Upload resume
- AI interview based on your resume
- Submit form
Resources & Support
- For details about the interview process and platform information, please check:
- For any help or support, reach out to: Show email
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
Vacancy posted 1 day ago
Similar jobs that could be interesting for youBased on the Health AI Conversation Evaluator in Toronto, ON vacancy
- ...Engineering Manager, AI Conversation Platform Who we are About Stripe Stripe is a financial infrastructure platform for businesses. Millions of companies—from the world’s largest enterprises to the most ambitious startups—use Stripe to accept payments, grow their...SuggestedFull time
- ...the same, a place where employees know they can thrive. The AI Developer is responsible for the development, testing and delivery... .... What You'll Be Doing Developing and enhancing conversational chat bot solutions in relation to Cognigy and NICE Proactive AI...SuggestedRemote jobFull time
$80k - $180k per year
...Zafin is an AI platform company helping regulated institutions modernize how critical work is designed, governed, and delivered. Our... ...AI responsibly and at scale. What’s the Opportunity? The AI Evaluation Engineer ensures AI agent solutions are accurate, reliable, safe,...Suggested- ...at your fingertips. With our AI-powered Global Work Platform™... ...Alfie is Pebl's proprietary conversational AI — built in-house and live... ...contexts? What does a great AI conversation actually look like for our users... ...for growing families. Health and Dental Insurance – Where...SuggestedSummer workInternshipLocal areaFlexible hours
$80 - $120 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...Position: Compliance / regulatory response with financial-services AI Evaluator Type: Contract Compensation: $80–$120/hour...SuggestedFull timeContract workSummer workWork at officeRemote work$100 - $150 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...pipelines , and A/B test write-ups . Evaluate AI-generated or human-created work against... ...detailed written justifications for evaluations and scores to maintain transparency and accountability...Full timeContract workSummer workRemote work- ...collaboration with other disciplines - all foundational ingredients in successful digital experiences and client partnerships. As a QA / AI Evaluation Engineer, you will join a highly motivated and experienced team in a forward-leaning role that proves the platform actually...Full timeImmediate start
$80 - $120 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...Position: Compliance / regulatory response with financial-services AI Evaluator Type: Contract Compensation: $80–$120/hour...Full timeContract workSummer workWork at officeRemote work$70 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...Upload your resume and application form. A 25-minute conversational interview covering your background, experience, and motivations....Remote jobContract workSummer workImmediate start$120 - $170 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...Preferred ~ Prior experience with AI training , evaluation, or human-data projects. Application Process (Takes 20–30...Full timeContract workSummer workRemote work$100 - $150 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...pipelines , and technical reports. Evaluate AI-generated or human-created work against... ...Provide detailed written justifications for evaluations and scores to maintain transparency....Remote jobContract workSummer work$85 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...Responsibilities Use frontier AI coding agents to complete and evaluate complex infrastructure engineering tasks. Review model-...Full timeContract workSummer workRemote work$80 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...Responsibilities Use frontier AI coding agents to complete and evaluate complex data engineering tasks. Review model-generated...Full timeContract workSummer workRemote work$70 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...hour Location: Remote Role Responsibilities Evaluate AI-generated responses to strengthen reasoning and rigor in model...Remote jobContract workSummer work$80 - $120 per hour
...connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...Larry Summers , and Jack Dorsey . Position: Public health communications Evaluator Type: Contract Compensation: $80–$120/hour...Remote jobContract workSummer workWork at office$80 - $120 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ..., and Jack Dorsey . Position: Software / AI / IT / data Evaluator Type: Contract Compensation: $80–$120/hour...Remote jobContract workSummer workWork at office$80 - $120 per hour
...connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco our investors... ...Larry Summers and Jack Dorsey . Position: Public health communications Evaluator Type: Contract Compensation: $80$120/hour...Full timeContract workSummer workWork at officeRemote work$39 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...hours/week Role Responsibilities Evaluate AI model output lyrics , voice generation... ...form Complete a 25-minute conversational interview on your background, experience,...Remote jobContract workSummer workImmediate start- ...Remote Job Summary We are seeking experienced Senior Software Engineers to support an AI training project by creating reinforcement learning environments that evaluate AI models on complex software engineering tasks using Model Context Protocol (MCP) tools....Remote jobFor contractors
$100k - $115k per year
...About the Job : We are seeking a skilled Conversational AI Engineer to be a core contributor in designing building testing and deploying conversational experiences using Google Contact Center AI (CCAI) technologies. In this role you will develop and enhance enterprise...Permanent employmentFull timeWork at officeLocal area$70 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...Remote Role Responsibilities Evaluate AI-generated responses to strengthen reasoning... ...detail and accurate application of structured evaluation guidelines. Work independently and...Remote jobContract workSummer work$42 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...sessions with experienced tutors. Engage in natural, substantive conversations in Spanish based on assigned topics and lesson structures....Remote jobContract workSummer work- ...Freelance Software Localization and AI Language Evaluator | English to Turkish Remote | US, Canada & Latin America Time Zones Acclaro is... ...software localization, IT translation, machine translation evaluation, post-editing, or AI-generated content review. You will evaluate...Long term contractContract workFor contractorsFreelanceRemote workWorldwideFlexible hours
$60 - $150 per hour
...About the job Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey . Position...Full timeContract workSummer workRemote workFlexible hours- ...they can thrive. TTEC Digital seeks an innovative and creative Conversational UI Dialog Designer to assist in the design, implementation, and... ...known as virtual agents that leverage Artificial Intelligence (AI). As a Conversational Designer , you will be responsible for...Full timeRemote work
- ..., CPG, Sports, and Media with AI-native delivery capability, designing... ...of approaches. Establish evaluation-driven development practices,... ...comfortably between executive conversations, product decisions,... ...value your time, safety, and health, and strive to build a work community...Full timeWork at officeRemote workFlexible hours
$25 - $45 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...Role Responsibilities Evaluate AI-generated written conversations using predefined rubrics to enhance... ...performance. Draft and refine evaluation rubrics for generalist writing tasks...Full timeContract workSummer workRemote work$15 - $20 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...tools . Generate high-quality human evaluation data by identifying response strengths, areas... ...model responses align with expected conversational behavior and system guidelines. Work...Full timeContract workSummer workRemote work- ...Description Role Overview The AI Lead is responsible for... ...prototypes to drive customer conversations · Validate solutions through... ...opportunities and revenue potential · Evaluate ideas based on impact,... ...structure based on performance. · Health insurance, 401(k) with company...Full time
$80 - $160 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...Utilize strong analytical skills to contribute to high-impact evaluation projects. Collaborate with teams to improve document standards...Full timeContract workSummer workRemote work
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Health AI Conversation Evaluator. Be the first to apply!
Related searches
- estimateur en bâtiment Toronto, ON
- chargé de projet estimateur Toronto, ON
- admissions evaluator Toronto, ON
- évaluateur-estimateur Toronto, ON
- estimateur automobile Toronto, ON
- estimateur structure Toronto, ON
- remote social media evaluator Toronto, ON
- test evaluator Toronto, ON
- medical sante Toronto, ON
- health communications specialist Toronto, ON