Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

AI Evaluation Rubric Reviewer

$40 - $75 per hour
Temporary

Mercor

Pay: $40 – $75/hour
Commitment: Hourly, ~15 h/week
Field: Data Analysis

The work

We are testing AI agents on realistic shopping and delivery tasks. You'll review AI-drafted tasks (prompt, grading rubric) along with model attempts, and decide whether each failure is fair:

\- Was the information available?

\- Does the rubric accurately grade what the prompt asked for?

\- Would a valid answer have been marked wrong?

\- Is the rubric structure correct per the guidelines?

Fix the criteria that aren't fair, flag the tasks that can't be saved, and write a short note on each call.

Must-haves

\- Substantial experience writing or reviewing rubrics for LLM evals. Please list the projects in your application.

\- Can read a long agent transcript and find where it went wrong.

\- Precise writing, and you check the data instead of trusting the reference answer.

Vacancy posted 23 hours ago
Similar jobs that could be interesting for youBased on the AI Evaluation Rubric Reviewer in Remote vacancy
  • $26 - $99 per hour

     ...live online classes on writing the rubrics and criteria that other people grade AI responses with, from a first draft...  ...material, running workshops or reviewing guidelines written by others. Patience...  ...or guidelines that others used to evaluate AI output. ~ Precise written... 
    Suggested
    Contract work
    Local area
    Remote work

    YPAI

    Remote
    6 days ago
  •  ...specialists with project-based AI opportunities for leading tech...  ..., focused on testing, evaluating, and improving AI systems. This...  ...no malpractice exposure: you review written clinical cases ~ All...  ...EHR vignettes for a medical AI evaluation project. General and internal... 
    Suggested
    Part time
    For contractors
    Remote work
    Flexible hours

    Mindrift (Toloka)

    Remote
    3 days ago
  •  ...AI Fact-Checking & Response Evaluation Reviewer (German) Mission of the Role Apply independent judgment to identify AI blind spots, reasoning failures, factual inaccuracies, contradictory evidence, and quality gaps. Successful reviewers do not simply validate AI-generated... 
    Suggested
    Long term contract
    Contract work
    Freelance
    Remote work
    Flexible hours

    e2f

    Remote
    6 days ago
  •  ...provides independent, multidisciplinary evaluations of healthcare AI technologies seeking deployment in...  ...science professionals to serve as technical reviewers on expert evaluation panels. In...  ...This is an evaluation and scientific review role, not a model development position... 
    Suggested
    Contract work
    Work at office
    Shift work

    mpathic

    Remote
    29 days ago
  •  ...competitive advantage. Cresta’s unified AI platform combines...  ...’s you. As our first AI Agent Evaluation Engineer, you define what quality...  ...Define and maintain the evaluation rubric for the agent across three...  ...Analysis & Annotation - Review production conversations by hand... 
    Suggested
    Full time

    Cresta

    Remote
    4 days ago
  •  ...competitive advantage. Cresta’s unified AI platform combines...  ....   As our first AI Agent Evaluation Engineer, you define what quality...  ...Define and maintain the evaluation rubric for the agent across three...  ...Analysis & Annotation Review production conversations by hand... 
    Full time
    Work at office
    Remote work
    Home office
    Flexible hours

    Cresta

    Remote
    4 days ago
  •  ...About the role AI assistants are becoming part of everyday...  ...your preferred experience. Evaluate conversational quality, helpfulness...  ...guidelines and document your evaluations accurately. Who we're...  ...your eligibility. Profile review: If everything looks good,... 
    Hourly pay
    Temporary work
    Part time
    Remote work

    micro1

    Remote
    4 days ago
  •  ...requirements Help Shape the Future of AI in Healthcare What if the...  ...healthcare information, evaluates medical scenarios, and delivers...  ...healthcare practices. Medical history review and interpretation of clinical...  ...project-specific scoring rubrics, annotation instructions, and... 
    Contract work
    For contractors
    Traineeship
    Remote work

    Volga Partners

    Remote
    12 days ago
  •  ...Teach live online classes on grading one AI response against written criteria: accuracy...  ...for YPAI Learn trains people for AI evaluation work in live classes. Your classes take learners...  ...teaching, coaching new evaluators, reviewing other people's ratings or running... 
    Contract work
    Remote work

    YPAI

    Remote
    4 days ago
  •  ...-world primary care knowledge directly ensures AI medical responses are safe, accurate, and effective. You will review patient-facing health advice generated by AI models...  ...scenarios. Your core responsibility is to evaluate whether the guidance aligns with established clinical... 
    Contract work
    Freelance
    Flexible hours

    Revelo

    Remote
    10 days ago
  •  ...your own work and turn each one into a self-contained Task that an AI agent has to solve in a Linux terminal. For every Task, you'll...  ...programming languages - **Machine learning:** training, inference, evaluation, and kernels - **Security:** cryptography, reverse engineering,... 
    Shift work

    Workada

    Remote
    11 days ago
  • $55 - $110 per hour

     ...RealTrainAI is recruiting experts for its AI coding & SWE reviewer AI training project. Experts help train and evaluate AI models by reviewing, rating and writing responses in their field of expertise. The work is remote and paid hourly. Pay: $55–$110/hr (industry-reported... 
    Hourly pay
    Contract work
    Remote work

    RealTrainAI

    Remote
    a month ago
  •  ...French (FR) and interested in working on AI-related projects? We're currently looking...  ...talent pool for an upcoming AI Response Evaluation and Fact-Checking project . This freelance...  ...misleading, or unsupported information Review user queries and corresponding AI-... 
    Contract work
    Freelance
    Remote work
    Flexible hours

    e2f

    Remote
    15 days ago
  • $150 per hour

     ...long hiring takes Across the AI training platforms we refer...  ..., this Software Engineer - AI Reviewer Expert position compensates Software...  ...& Robotics RLHF & AI Evaluation Expert Related jobs AI...  ...following a one-size-fits-all rubric. If you don't have hands-on Software... 
    Hourly pay
    Remote work

    Ethos

    Remote
    a month ago
  • $150 per hour

     ...long hiring takes Across the AI training platforms we refer...  ...role This Data Scientist - AI Reviewer Expert role pays $150/hr for...  ...STEM & Science RLHF & AI Evaluation Expert Related jobs Research...  ...following a one-size-fits-all rubric. If you don't have hands-on... 
    Hourly pay
    Remote work

    Ethos

    Remote
    a month ago
  • $10 - $30 per hour

     ...feedback digital delivery platforms ai-driven tools familiarity independent work...  ...models and agents. We provide frontier evaluations and reinforcement learning environments to...  ...effective customer service standards. Review context-specific scenarios, such as order... 
    Part time
    For contractors
    Remote work
    Shift work

    micro1

    Remote
    21 days ago
  • usd16 per hour

     ...We are looking for AI Linguistic Evaluators to assess how effectively an AI application performs in...  ...writing, journalism, proofreading, content review, localization, or other language-...  ...appropriateness. - Providing structured evaluations and feedback following the project instructions... 
    Hourly pay
    Daily paid
    Full time
    Freelance
    Immediate start
    Remote work
    Flexible hours

    RWS

    Remote
    a month ago
  •  ...Curate and evaluate code in Rust to enhance AI training datasets. About Us Based in San Francisco, California...  ..., debugging, and code quality/review assessment. Excellent oral and...  ...communication skills for clear, structured evaluation rationales. Engagement Details... 
    Part time
    For contractors
    Flexible hours

    Turing

    Remote
    10 days ago
  • $55 - $100 per hour

     ...Pay: $55–100/hour You'll evaluate AI-generated responses to legal questions — covering case...  ...work involves reading a legal prompt, reviewing the model's response, and scoring it across...  ...arguments. Feedback is written in structured rubric fields, with an optional free-text... 
    Contract work

    Claru

    Remote
    a month ago
  •  ...What You'll Do Review real user interaction traces with an AI shopping assistant Identify logical failures, inaccuracies...  ...in the text Create structured rubrics and verifiers to judge response...  ...Commit to 20+ hours per week of evaluation work on our internal platform Who... 
    Contract work
    For contractors
    Remote work

    Terac

    Remote
    a month ago
  •  ...interview Walk us through how you currently evaluate new artificial intelligence platforms...  ...recent challenges you faced while vetting AI tools Who Should Apply Active role in...  ...Comfortable discussing technical and business evaluation criteria in a 1:1 setting Contract &... 
    Contract work
    For contractors

    Terac

    Remote
    12 days ago
  • $40 per hour

     ...of Use. To learn more about our privacy practices review our Privacy Policy. Please review your choices. Accept...  ...All Employers Job seekers Career centers Handshake AI Enterprise AI Research Log in Sign up FRENCH AI EVALUATION SPECIALIST (CANADA) Sign up now Up to $40/hr... 
    Part time
    Remote work
    Flexible hours

    Handshake

    Remote
    a month ago
  •  ...the intersection of human creativity and AI. We partner with leading AI labs and product...  .... To Join the Network # Portfolio review - We'll assess your Contra profile and portfolio...  ...through. What You May Work On Evaluate AI-generated visual outputs across a wide... 
    Full time
    Contract work

    Contra Labs

    Remote
    26 days ago
  •  ...required unless specified. To Join the Network # Portfolio review - We’ll assess your Contra profile and portfolio for strength in...  ...photography, composition, lighting, and visual quality. # Skills evaluation - You’ll complete a structured photography submission designed... 
    Full time
    Contract work

    Contra Labs

    Remote
    26 days ago
  •  ...the intersection of human creativity and AI. We partner with leading AI labs and product...  .... To Join the Network Portfolio review: We'll assess your Contra profile and portfolio...  ...and mobile applications. Skills evaluation: You'll rate AI-generated outputs across... 
    Full time
    Contract work

    Contra Labs

    Remote
    7 days ago
  •  ...Analyze GitHub issues and evaluate Python code quality for open-source...  ..., working with frontier AI labs to accelerate model development...  ...high-quality training data, evaluations, and engineering talent....  ...approximately 75 mins) : ~ Assessment based review Skills: Python... 
    Contract work
    Part time
    For contractors
    Freelance
    Manual labor
    Remote work

    Turing

    Remote
    7 days ago
  • $80 - $100 per hour

     ...seeking an experienced legal professional to support a short-term AI content evaluation project. You’ll use your legal judgment to assess written...  ...feedback that improves its quality. What you’ll do Review legal content for accuracy, consistency, and appropriate context... 
    Contract work
    Temporary work
    Immediate start
    Remote work

    Braintrust

    Remote
    a month ago
  •  ...thresholds. Familiarity or comfort with AI / generative models, visual content, or...  ...Lionbridge . Background in content evaluation, annotation, or quality assurance for AI/ML...  .... Ability to adapt quickly to changing rubrics and instructions. Why Join Rise Data Labs... 
    Contract work
    Remote work
    Flexible hours

    Rise Data Labs

    Remote
    a month ago
  •  ...This is a fully remote, hourly contractor role supporting AI data and language projects on a project-based, flexible hour basis...  ...tables, and other content to support AI training datasets. LLM evaluation: reviewing AI-generated responses for accuracy, reasoning quality,... 
    Hourly pay
    Contract work
    For contractors
    Remote work
    Flexible hours

    CNTXT AI

    Remote
    26 days ago
  •  ...Engineering · Reactor Systems · AI Safety · Nuclear Physics...  ...About the Role AI Safety Evaluation Project | Remote | Contract Locations...  ...content. The work involves reviewing AI-generated prompts and responses...  ...research, technical review, or quality assurance is a plus... 
    Hourly pay
    Contract work
    For contractors
    Remote work

    Pasiflora AI

    Remote
    a month ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to AI Evaluation Rubric Reviewer. Be the first to apply!