Health AI Conversation Evaluator
$20 - $160 per hourMercor
- Remote job
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .
Position: Generalist Annotator — Health AI Conversation Quality Evaluation
Type: Contract
Compensation: $20–$160/hour
Location: Remote
Role Responsibilities
- Evaluate the relevance and clarity of AI responses . Ensure the AI answers the question in understandable language for non-experts.
- Detect sycophancy in AI interactions. Identify when the AI tells users what they want to hear instead of what they need.
- Assess the tone and outcome of conversations. Determine if the user leaves informed and calm or anxious and uninformed.
- Provide written justification for ratings below the top option. Explain what went wrong and suggest improvements.
- Work independently to review conversations ranging from 1 to 7 turns. Spend 2–13 minutes per conversation based on length.
Qualifications
Must-Have
- Strong written English skills.
- Ability to explain judgments in two to four specific sentences.
- Careful reading to notice omissions in responses.
- Comfort with applying a detailed rubric consistently.
- Reliable availability for time-boxed batches.
Preferred
- Experience in annotation, human-feedback, evaluation, or content review.
- Background in explaining technical concepts to non-experts.
- Experience with rubric-based grading or quality assurance.
Important
- Using an AI tool to write comments is prohibited. This work captures human judgment that AI lacks. AI-written feedback corrupts the dataset and is checked for.
Application Process (Takes 20–30 mins to complete)
- Upload resume
- AI interview based on your resume
- Submit form
Resources & Support
- For details about the interview process and platform information, please check:
- For any help or support, reach out to: Show email
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
- ...through world-class networks, AI-powered solutions, and digital... ...Summary Bell is seeking a Conversational AI Developer to join an innovative... ...., ADK, MCP, A2A) as the team evaluates them. Test and troubleshoot... ..., dental, vision and mental health benefits that you can tailor...SuggestedFull timeInternshipWork at office3 days per week
$100 per hour
...connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our... ...Summers , and Jack Dorsey . Position: Clinical Mental Health Expert — AI Conversation Evaluation Type: Contract Compensation: $100/hour...SuggestedRemote jobContract workSummer work- ...Engineering Manager, AI Conversation Platform Who we are About Stripe Stripe is a financial infrastructure platform for businesses. Millions of companies—from the world’s largest enterprises to the most ambitious startups—use Stripe to accept payments, grow their...Suggested
$126k - $176k per year
...Sr. Legal AI Prompter / Evaluator Job Description Sr. Legal AI Prompters / Evaluators are responsible... ...datasets with SMEs, coordinate evaluations, and design evaluation schemes, including... ...flexible vacation, two company-wide Mental Health Days off, access to the Headspace app,...SuggestedRemplacementFull timeWork at officeLocal areaFlexible hours- ...Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. This is a task-based... ...clinical EHR vignettes for a medical AI evaluation project. General and internal medicine are the...SuggestedPart timeFor contractorsRemote workFlexible hours
$30 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco our investors... ...tasks guiding it answering questions and correcting errors. Evaluate the experience and deliverables providing concise feedback on performance...Hourly payWeekly payFull timeContract workFor contractorsSummer workRemote work$80 - $120 per hour
...connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...Larry Summers , and Jack Dorsey . Position: Public health communications Evaluator Type: Contract Compensation: $80–$120/hour...Remote jobContract workSummer workWork at office$100 - $150 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...pipelines , and technical reports. Evaluate AI-generated or human-created work against... ...Provide detailed written justifications for evaluations and scores to maintain transparency....Remote jobContract workSummer work$140k - $175k per year
...delivery of production-ready software for artificial intelligence (AI) model evaluation and benchmarking. You will build scalable evaluation... ...plans to include flexible vacation, two company-wide Mental Health Days off, access to the Headspace app, retirement savings, tuition...Full timeInternshipWork at officeLocal areaFlexible hours$39 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...hours/week Role Responsibilities Evaluate AI model output lyrics , voice generation... ...form Complete a 25-minute conversational interview on your background, experience,...Remote jobContract workSummer workImmediate start$70 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...hour Location: Remote Role Responsibilities Evaluate AI-generated responses to strengthen reasoning and rigor in model...Remote jobContract workSummer work$80 - $120 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ..., and Jack Dorsey . Position: Software / AI / IT / data Evaluator Type: Contract Compensation: $80–$120/hour...Remote jobContract workSummer workWork at office$70 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...Upload your resume and application form. A 25-minute conversational interview covering your background, experience, and motivations....Remote jobContract workSummer workImmediate start$80 - $120 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...Position: Compliance / regulatory response with financial-services AI Evaluator Type: Contract Compensation: $80–$120/hour...Remote jobContract workSummer workWork at office$100 - $150 per hour
...creative and technical talent with leading AI research labs. Headquartered in San... ...pipelines , and A/B test write-ups . Evaluate AI-generated or human-created work against... ...detailed written justifications for evaluations and scores to maintain transparency and accountability...Remote jobContract workSummer work$85 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...Responsibilities Use frontier AI coding agents to complete and evaluate complex infrastructure engineering tasks. Review model-...Remote jobContract workSummer work$80 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...Responsibilities Use frontier AI coding agents to complete and evaluate complex data engineering tasks. Review model-generated...Remote jobContract workSummer work$120 - $170 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...Preferred ~ Prior experience with AI training , evaluation, or human-data projects. Application Process (Takes 20–30...Remote jobContract workSummer work$52.38 - $57.39 per hour
...Branch of Parks, Forestry & Recreation, the Evaluator, Programs will help design, plan, and lead the implementation of small and large evaluations, performance measurement frameworks,... ...anthropology, social sciences, ecology or public health or social services research or an...Hourly payPermanent employmentFull timePart timeInternshipMonday to fridayShift work$113k - $194k per year
...actuarial professionals working in 43 countries. AIS is part of Deloitte's Assurance & Advisory... ...include: $4,000 per year for mental health support benefits, a $1,300 flexible benefit... ...will utilize human screening combined with AI technology to help identify the skills and...Permanent employmentFlexible hours$250k per year
...is the leading security-first enterprise AI company. We build cutting-edge foundation... ...and Seoul. Join us! Role Overview: Evaluation is critical to making progress in scaling... ...in your local currency for lunch. ~ Full health and dental benefits, including a separate...Full timeWork at officeLocal areaRemote workHome office$39 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...experience and deliverables, providing concise notes on performance. Evaluate AI model interactions to ensure alignment with professional...Remote job$100 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...grading rubrics with pass/fail criteria and judgment calls. Evaluate AI model attempts and confirm failures for fair, defensible reasons...Remote jobHourly payWeekly payContract workFor contractorsSummer work$39 per hour
...About the job Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco our investors include Benchmark General Catalyst Peter Thiel Adam DAngelo Larry Summers and Jack Dorsey . Position: Healthcare...Full timeContract workSummer workRemote work$30 - $40 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco our investors... ...outputs . Complete assigned queries each day to ensure timely evaluation. Provide short notes and annotations to enhance training data...Daily paidFull timeContract workSummer workRemote workFlexible hours$50 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...hour Location: Remote Role Responsibilities Evaluate the quality and accuracy of LLM-generated French text across various...Remote jobContract workSummer work$42 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...sessions with experienced tutors. Engage in natural, substantive conversations in Spanish based on assigned topics and lesson structures....Remote jobContract workSummer work$150 per hour
...Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors... ...Commitment: 10–20 hours/week Role Responsibilities Evaluate and compare AI-generated responses on challenging medical problems...Remote jobContract workSummer workImmediate start$57 per hour
...About the job Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey . Position...Remote jobHourly payContract workFor contractorsSummer work- ...Culture Align with the YNSE Project Health, Safety and Wellbeing (HSW) strategies and... ...Collect and maintain records required for evaluating the effectiveness of safety programs. Compile... ...and includes the feminine. We use AI-enabled tools in our recruitment platform...For contractors
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Health AI Conversation Evaluator. Be the first to apply!
- admissions evaluator Toronto, ON
- évaluateur-estimateur Toronto, ON
- remote social media evaluator Toronto, ON
- estimateur - dessinateur Toronto, ON
- test evaluator Toronto, ON
- estimateur automobile Toronto, ON
- estimateur en bâtiment Toronto, ON
- estimateur structure Toronto, ON
- estimateur industriel Toronto, ON
- chargé de projet estimateur Toronto, ON
