Language Model Evaluator - Fully Remote | Upto $20/hr Part-time
$20 per hourMercor
Toronto, ON
- Remote job
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .
Position: Generalist - English & Bengali
Type: Contract
Compensation: $15–$20/hour
Location: Remote
Role Responsibilities
- Conduct fact-checking using trusted public sources and external tools .
- Generate high-quality human evaluation data by identifying response strengths, areas for improvement, and factual inaccuracies.
- Assess reasoning quality, clarity, tone, and completeness of responses.
- Ensure model responses align with expected conversational behavior and system guidelines.
- Work independently and asynchronously to meet deadlines while improving AI model performance .
Qualifications
Must-Have
- Bachelor's degree
- Native speaker in Bengali
- Significant experience using large language models (LLMs)
- Excellent writing skills in English
- Strong attention to detail
- Background or experience in domains requiring structured analytical thinking (e.g., research, policy, analytics, linguistics, engineering)
Preferred
- Prior experience with RLHF, model evaluation, or data annotation work
- Experience writing or editing high-quality written content
- Experience comparing multiple outputs and making fine-grained qualitative judgments
Application Process (Takes 20–30 mins to complete)
- Upload resume
- AI interview based on your resume
- Submit form
Resources & Support
- For details about the interview process and platform information, please check:
- For any help or support, reach out to: Show email
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
Vacancy posted 10 hours ago
Similar jobs that could be interesting for youBased on the Language Model Evaluator - Fully Remote | Upto $20/hr Part-time in Toronto, ON vacancy
$37 per hour
...Position: Bilingual French Generalist Evaluator Expert Type: Contract... ...25–$37/hour Location: Remote Duration: 2–4 months Commitment: 20+ hours/week Role... ...conventions. Run prompts through models. Assess outputs for accuracy, fluency...Remote jobContract workSummer work$70 per hour
...$60–$70/hour Location: Remote Role Responsibilities Evaluate the accuracy of AI performance on... ...structured feedback to improve AI model outputs and enhance training data... ...Strong domain knowledge in policy language interpretation. Expertise in risk...Remote jobLanguageContract workSummer work$90 per hour
...Position: Sales & Marketing Experts Type: Contract Compensation: $65–$90/hour Location: Remote Commitment: 20+ hours/week Role Responsibilities Construct enterprise sales scenarios spanning $1M+ ACV deals , multi-stakeholder...Part timeRemote jobHourly payWeekly payContract workFor contractorsSummer work$95 per hour
...Dorsey . Position: Accounting Experts Type: Contract Compensation: $70–$95/hour Location: Remote Commitment: 20+ hours/week Role Responsibilities Construct scenarios across F500 corporate accounting and controllership,...Part timeRemote jobHourly payWeekly payContract workFor contractorsSummer work$80 per hour
...$68–$80/hour Location: Remote Role Responsibilities Evaluate AI performance on blueprints , structural... ...structured feedback to enhance AI model training and output quality.... .... Application Process (Takes 20–30 mins to complete) Upload...Remote jobContract workSummer work$90 per hour
...–$90/hour Location: Remote Duration: ~2 months... ...security concepts to enhance AI model threat detection and... ...in programming with low-level languages such as C , C++ , or Java... ...for up to 1 hour of onboarding time, including screening and onboarding...Remote jobLanguageContract workSummer work$150 per hour
...Contract Compensation: $50–$150/hour Location: Remote Commitment: 5–10 hours/week Role Responsibilities... ..., corporate, energetic, and calm. Application Process (Takes 20–30 mins to complete) Upload resume AI interview based...Remote jobContract workSummer work10 hours per week$100 per hour
...–$100/hour Location: Remote Role Responsibilities... ...financial documents, including pricing models, investment memoranda, and... ...earnings reports. Annotate and evaluate complex financial documents to... ...Application Process (Takes 20–30 mins to complete) Upload...Remote jobContract workSummer work$120 per hour
...Contract Compensation: $85–$120/hour Location: Remote Commitment: 20+ hours/week Role Responsibilities Construct FP&... ..., and earnings preparation. Build three-statement models , driver-based forecasts, and variance analyses at a $1B+...Part timeRemote jobHourly payWeekly payContract workFor contractorsSummer work$120 per hour
...Legal Expert — Transactional / Corporate (Remote, Hourly) Type: Contract Compensation... ...Location: Remote Commitment: 20–40 hours/week Role Responsibilities... .... Provide written feedback to improve model behavior. Participate in onboarding...Remote jobHourly payContract workSummer workWork at officeImmediate start$130 per hour
...Dorsey . Position: Law Experts Type: Contract Compensation: $100–$130/hour Location: Remote Commitment: 20+ hours/week Role Responsibilities Construct enterprise legal scenarios across in-house counsel, regulatory, transactional...Part timeRemote jobHourly payWeekly payContract workFor contractorsSummer work$120 per hour
...Compensation: $120/hour Location: Remote Duration: Minimum four weeks... ...public SEC filings to support AI model training . Evaluate the accuracy of AI-generated equity... ...Application Process (Takes 20–30 mins to complete) Upload resume...Remote jobContract workSummer work- ...1000 per completed task Location: Remote Duration: ~2 weeks... ...asynchronously to meet deadlines and enhance AI model performance . Contribute expertise to... ...task rate. Application Process (Takes 20–30 mins to complete) Upload resume...Part timeRemote jobContract workSummer workImmediate start
$150 per hour
...–$150/hour Location: Remote Duration: 2–3 week engagement Commitment: 20+ hours/week Role Responsibilities... ...worms , and exploits . Evaluate POC exploit development to... ...deadlines while improving AI model performance . Collaborate...Remote jobHourly payWeekly payContract workFor contractorsSummer work$120 per hour
...Contract Compensation: $85–$120/hour Location: Remote Commitment: 20–40 hours/week Role Responsibilities Design... ...rubrics. Provide written feedback to improve model behavior. Participate in onboarding office hours and specialty...Remote jobHourly payContract workFor contractorsSummer workWork at officeImmediate start$80 per hour
...Compensation: $55–$80/hour Location: Remote Role Responsibilities... ...project involving state-of-the-art large language models. Create high-quality data to inform... ...Complete a short interview and task (20–30 mins ). Paid onboarding time includes...Remote jobLanguageContract workSummer work$110 per hour
...Compensation: $90–$110/hour Location: Remote Duration: 3–4 weeks... ...asynchronously to meet deadlines and improve AI model performance . Contribute expertise to... ...Immediately Application Process (Takes 20–30 mins to complete) Upload resume...Remote jobContract workSummer workImmediate startFlexible hours$175 per hour
...100–$175/hour Location: Remote Duration: 3+ months... ...Role Responsibilities Evaluate digital chip design workflows to enhance AI model training and evaluation .... ...Application Process (Takes 20–30 mins to complete) Upload...Remote jobHourly payWeekly payContract workFor contractorsSummer work$130 per hour
...Compensation: $130–$300/hour Location: Remote Duration: Ongoing, reviewed monthly Commitment: 20+ hours/week Role Responsibilities... ...written feedback to the research team to improve model behavior . Participate in onboarding office...Remote jobWeekly payContract workSummer workWork at office$90 per hour
...Contract Compensation: $60–$90/hour Location: Remote Role Responsibilities Convert legacy decks into modern... ...reliable on a flexible schedule. Application Process (Takes 20–30 mins to complete) Upload resume AI interview based...Remote jobContract workSummer workFlexible hours- ...completed task Location: Remote Role Responsibilities Review and evaluate AI-generated outputs related to... ...with research teams to refine evaluation frameworks for K-12 education AI... ...Application Process (Takes 20–30 mins to complete) Upload...Part timeRemote jobHourly payContract workSummer work
$53 per hour
...38–$53/hour Location: Remote Duration: 3–6 months Commitment: 20+ hours/week Role Responsibilities... .... Define and document evaluation standards, creating rubrics that... ...linguistic clarity. Conduct model testing and grading, assessing outputs...Remote jobContract workSummer work- ...completed task Location: Remote Role Responsibilities Review and evaluate AI-generated outputs related to... ...transition to an hourly compensation model based on sustained quality and... .... Application Process (Takes 20–30 mins to complete) Upload...Part timeRemote jobHourly payContract workSummer work
$150 per hour
...Contract Compensation: $60–$150/hour Location: Remote Role Responsibilities Convert legacy documents into modern... ...reliable on a flexible schedule. Application Process (Takes 20–30 mins to complete) Upload resume AI interview based...Remote jobContract workSummer workFlexible hours$80 per hour
...Compensation: $30–$80/hour Location: Remote Role Responsibilities Build... .... Run the agent against scenarios, evaluate performance, and guide it to success... ...Python . Application Process (Takes 20–30 mins to complete) Upload resume...Remote jobContract workSummer work- ...per completed task Location: Remote Role Responsibilities Review and evaluate AI-generated outputs related to IEP... ...transition to an hourly compensation model for top performers. Application Process (Takes 20–30 mins to complete) Upload resume...Part timeRemote jobHourly payContract workSummer work
$60 per hour
...Contract Compensation: $50–$60/hour Location: Remote Role Responsibilities Design and refine professional... ...technical, or data-heavy content . Application Process (Takes 20–30 mins to complete) Upload resume AI interview based...Remote jobContract workSummer work- ...Compensation: $1500–$1700 Location: Remote Duration: ~2 weeks (with the... ...for project expansion) Commitment: ~20 hours/week required Role Responsibilities... ...quiz enter a work trial period with a one-time $1,500 payment upon successful task...Part timeRemote jobHourly payContract workSummer workImmediate startTrial periodFlexible hoursShift work
$90 per hour
...Compensation: $70–$90/hour Location: Remote Role Responsibilities... ...Astrophysics , and Cosmology . Evaluate the impact of AI models on future innovations in the field.... ...basis. Application Process (Takes 20–30 mins to complete) Upload...Remote jobContract workSummer work- ...Detectives and Investigators Type: Contract Compensation: $1500–$1700 Location: Remote Duration: 3–4 weeks Commitment: 20–40 hours/week Role Responsibilities Create deliverables for common requests within your...Remote jobHourly payContract workSummer workImmediate startTrial periodFlexible hoursShift work
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Language Model Evaluator - Fully Remote | Upto $20/hr Part-time. Be the first to apply!
Related searches