Language Model Evaluator - Fully Remote | Upto $20/hr Part-time
$20 per hourMercor
Toronto, ON
- Remote job
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .
Position: Generalist - English & Punjabi
Type: Contract
Compensation: $15–$20/hour
Location: Remote
Role Responsibilities
- Conduct fact-checking using trusted public sources and external tools .
- Generate high-quality human evaluation data by identifying response strengths, areas for improvement, and factual inaccuracies.
- Assess reasoning quality, clarity, tone, and completeness of responses.
- Ensure model responses align with expected conversational behavior and system guidelines.
- Work independently and asynchronously to meet deadlines while improving AI model performance .
Qualifications
Must-Have
- Bachelor's degree
- Native speaker in Punjabi
- Significant experience using large language models (LLMs)
- Excellent writing skills in English
- Strong attention to detail
- Background or experience in domains requiring structured analytical thinking
Preferred
- Experience with RLHF, model evaluation, or data annotation work
- Experience writing or editing high-quality written content
- Experience comparing multiple outputs and making fine-grained qualitative judgments
Application Process (Takes 20–30 mins to complete)
- Upload resume
- AI interview based on your resume
- Submit form
Resources & Support
- For details about the interview process and platform information, please check:
- For any help or support, reach out to: Show email
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
Vacancy posted a month ago
Similar jobs that could be interesting for youBased on the Language Model Evaluator - Fully Remote | Upto $20/hr Part-time in Toronto, ON vacancy
$120 per hour
...Spreadsheet QA / workbook maintenance Evaluator Type: Contract... ...–$120/hour Location: Remote Role Responsibilities... ...with AI research teams to enhance model performance and training data quality... ...Application Process (Takes 20–30 mins to complete) Upload...Remote jobContract workSummer workWork at office$120 per hour
...estate / hospitality / events Evaluator Type: Contract... ...–$120/hour Location: Remote Role Responsibilities... ...deadlines while improving AI model performance . Qualifications... ...Application Process (Takes 20–30 mins to complete) Upload...Remote jobContract workSummer workWork at office$50 per hour
...Compensation: $50/hour Location: Remote Commitment: 15–20 hours/week Role Responsibilities Evaluate the accuracy and depth of AI-generated content... ...domain to strengthen reasoning and rigor in model outputs . Author complex, grounded...Part timeRemote jobContract workSummer work$120 per hour
.... Position: FP&A / corporate finance Evaluator Type: Contract Compensation: $80–$120/hour Location: Remote Role Responsibilities Evaluate... ...institution. Application Process (Takes 20–30 mins to complete) Upload resume...Remote jobContract workSummer workWork at office$50 per hour
...Type: Contract Compensation: $50/hour Location: Remote Commitment: 15–20 hours/week Role Responsibilities Evaluate the accuracy and depth of AI models on visual document understanding in the Telecom/Network domain. Author...Part timeRemote jobContract workSummer work$120 per hour
...Data quality / CRM operations Evaluator Type: Contract... ...–$120/hour Location: Remote Role Responsibilities... ...written feedback to improve AI model outputs . Apply deep subject... ...Application Process (Takes 20–30 mins to complete) Upload...Remote jobContract workSummer workWork at office$50 per hour
...Compensation: $50/hour Location: Remote Commitment: 15–20 hours/week Role Responsibilities Evaluate the accuracy and depth of AI-generated content... ...to strengthen reasoning and rigor in model outputs . Author complex, grounded tasks...Part timeRemote jobContract workSummer work$60 per hour
...: Contract Compensation: $60/hour Location: Remote Role Responsibilities Shoot 10 input images of people... ...experimenting with new visual ideas. Application Process (Takes 20–30 mins to complete) Interview: 20 minutes Form: 5...Remote jobHourly payContract workSummer work$90 per hour
...Contract Compensation: $70–$90/hour Location: Remote Role Responsibilities Develop challenging... ...with AI labs to create high-quality data for large language models. Evaluate and improve AI model outputs using domain expertise....Remote jobLanguageContract workSummer work$180 per hour
...$70–$180/hour Location: Remote Role Responsibilities Solve and evaluate challenging physics problems across... ..., or unsupported reasoning in model outputs. Apply relevant citations... ...Application Process (Takes 20–30 mins to complete) Submit...Remote jobContract workSummer workImmediate start$120 per hour
...Position: Healthcare / clinical Evaluator Type: Contract... ...–$120/hour Location: Remote Role Responsibilities... ...written feedback to improve AI model outputs . Apply deep subject... ...Application Process (Takes 20–30 mins to complete) Upload...Remote jobContract workSummer workWork at office$180 per hour
...Compensation: $130–$180/hour Location: Remote Duration: Ongoing, reviewed monthly Commitment: 20+ hours/week Role Responsibilities... ...written feedback to the research team to improve model behavior . Participate in onboarding office...Remote jobWeekly payContract workSummer workWork at office$180 per hour
...$400/hour Location: Remote Duration: Ongoing, reviewed... ...monthly Commitment: 20+ hours/week Role... ...ACS management , arrhythmia evaluation , and heart failure optimization... ...attending-level quality to guide AI model training . Grade AI-...Remote jobWeekly payContract workSummer workWork at office$85 per hour
...Compensation: $85/hour Location: Remote Role Responsibilities... ...AI coding agents to complete and evaluate complex engineering tasks. Review model-generated code for correctness, quality... ...Application Process (Takes 20–30 mins to complete) Upload...Remote jobContract workSummer work$85 per hour
...: $85/hour Location: Remote Role Responsibilities... ...AI coding agents to complete and evaluate complex security engineering tasks . Review model-generated code and architectures... ...Application Process (Takes 20–30 mins to complete) Upload...Remote jobContract workSummer work$50 per hour
...Understanding Type: Contract Compensation: $50/hour Location: Remote Commitment: 15–20 hours/week Role Responsibilities Evaluate AI models on visual document understanding in the Pharma & Clinical Trials domain. Author...Part timeRemote jobContract workSummer work$50 per hour
...Compensation: $50/hour Location: Remote Commitment: 15–20 hours/week Role Responsibilities Evaluate the accuracy and depth of AI-generated content... ...to strengthen reasoning and rigor in model outputs . Review complex scientific documents...Part timeRemote jobContract workSummer work$70 per hour
...Compensation: $50–$70/hour Location: Remote Role Responsibilities Evaluate and optimize Shopify store performance to enhance user... ...Shopify's agent . Application Process (Takes 20–30 mins to complete) Upload resume AI interview...Remote jobContract workSummer work$90 per hour
...Compensation: $70–$90/hour Location: Remote Role Responsibilities Capture... ...consistently. Write clear, natural-language task instructions that reflect expert... ...screenshots. Application Process (Takes 20–30 mins to complete) Upload resume...Remote jobLanguageContract workSummer work$85 per hour
...Compensation: $85/hour Location: Remote Role Responsibilities... ...AI coding agents to complete and evaluate complex engineering tasks. Review model-generated mobile application code for... ...Application Process (Takes 20–30 mins to complete) Upload...Remote jobContract workSummer work$80 per hour
...Compensation: $80/hour Location: Remote Role Responsibilities... ...AI coding agents to complete and evaluate complex data engineering tasks. Review model-generated implementations... ...Application Process (Takes 20–30 mins to complete) Upload...Remote jobContract workSummer work$85 per hour
...Compensation: $85/hour Location: Remote Commitment: 4–6 tasks/week... ...project involving state-of-the-art large language models . Create high-quality data to inform... ...basis. Application Process (Takes 20–30 mins to complete) Upload resume...Remote jobLanguageContract workSummer work$85 per hour
...Compensation: $85/hour Location: Remote Role Responsibilities... ...AI coding agents to complete and evaluate complex engineering tasks. Review model-generated frontend code for correctness... ...Application Process (Takes 20–30 mins to complete) Upload...Remote jobContract workSummer work$100 per hour
...Computational Scientific Software & Evaluation Design - Astrophysics & Cosmology... ...$70–$100/hour Location: Remote Commitment: 15–20 hours/week Role Responsibilities... ...Calibrate tasks against state-of-the-art AI models, refining problem designs to achieve...Remote jobContract workSummer work$50 per hour
...Contract Compensation: $40–$50/hour Location: Remote Role Responsibilities Build MCP servers in Python... ..., and containerized environments. Application Process (Takes 20–30 mins to complete) Upload resume AI interview based...Remote jobContract workSummer workLocal area$150 per hour
...Contract Compensation: $50–$150/hour Location: Remote Commitment: 5–10 hours/week Role Responsibilities... ...deliver multiple vocal styles . Application Process (Takes 20–30 mins to complete) Upload resume AI interview based...Remote jobContract workSummer work10 hours per week$120 per hour
...management / implementation planning Evaluator Type: Contract... ...–$120/hour Location: Remote Role Responsibilities... ...written feedback to enhance AI model outputs. Collaborate with... ...Application Process (Takes 20–30 mins to complete) Upload...Remote jobContract workSummer workWork at office$120 per hour
...response with financial-services AI Evaluator Type: Contract... ...–$120/hour Location: Remote Role Responsibilities... ...deadlines while improving AI model performance . Qualifications... ...Application Process (Takes 20–30 mins to complete) Upload...Remote jobContract workSummer workWork at office$180 per hour
...Compensation: $130–$180/hour Location: Remote Duration: Ongoing, reviewed monthly Commitment: 20+ hours/week Role Responsibilities... ...written feedback to the research team to improve model behavior and outputs. Participate in onboarding...Remote jobWeekly payContract workSummer workWork at office$100 per hour
...Compensation: $60–$100/hour Location: Remote Role Responsibilities Design... ...Must-Have ~4+ years full-time work experience as a Registered Nurse.... ...location. Application Process (Takes 20–30 mins to complete) Upload resume...Remote jobHourly payFull timeContract workFor contractorsSummer work
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Language Model Evaluator - Fully Remote | Upto $20/hr Part-time. Be the first to apply!
