AI Evaluation Rubric Reviewer
$40 - $75 per hourMercor
Pay: $40 – $75/hour
Commitment: Hourly, ~15 h/week
Field: Data Analysis
The work
We are testing AI agents on realistic shopping and delivery tasks. You'll review AI-drafted tasks (prompt, grading rubric) along with model attempts, and decide whether each failure is fair:
\- Was the information available?
\- Does the rubric accurately grade what the prompt asked for?
\- Would a valid answer have been marked wrong?
\- Is the rubric structure correct per the guidelines?
Fix the criteria that aren't fair, flag the tasks that can't be saved, and write a short note on each call.
Must-haves
\- Substantial experience writing or reviewing rubrics for LLM evals. Please list the projects in your application.
\- Can read a long agent transcript and find where it went wrong.
\- Precise writing, and you check the data instead of trusting the reference answer.
$26 - $99 per hour
...live online classes on writing the rubrics and criteria that other people grade AI responses with, from a first draft... ...material, running workshops or reviewing guidelines written by others. Patience... ...or guidelines that others used to evaluate AI output. ~ Precise written...SuggestedContract workLocal areaRemote work- ...specialists with project-based AI opportunities for leading tech... ..., focused on testing, evaluating, and improving AI systems. This... ...no malpractice exposure: you review written clinical cases ~ All... ...EHR vignettes for a medical AI evaluation project. General and internal...SuggestedPart timeFor contractorsRemote workFlexible hours
- ...AI Fact-Checking & Response Evaluation Reviewer (German) Mission of the Role Apply independent judgment to identify AI blind spots, reasoning failures, factual inaccuracies, contradictory evidence, and quality gaps. Successful reviewers do not simply validate AI-generated...SuggestedLong term contractContract workFreelanceRemote workFlexible hours
- ...provides independent, multidisciplinary evaluations of healthcare AI technologies seeking deployment in... ...science professionals to serve as technical reviewers on expert evaluation panels. In... ...This is an evaluation and scientific review role, not a model development position...SuggestedContract workWork at officeShift work
- ...competitive advantage. Cresta’s unified AI platform combines... ...’s you. As our first AI Agent Evaluation Engineer, you define what quality... ...Define and maintain the evaluation rubric for the agent across three... ...Analysis & Annotation - Review production conversations by hand...SuggestedFull time
- ...competitive advantage. Cresta’s unified AI platform combines... .... As our first AI Agent Evaluation Engineer, you define what quality... ...Define and maintain the evaluation rubric for the agent across three... ...Analysis & Annotation Review production conversations by hand...Full timeWork at officeRemote workHome officeFlexible hours
- ...About the role AI assistants are becoming part of everyday... ...your preferred experience. Evaluate conversational quality, helpfulness... ...guidelines and document your evaluations accurately. Who we're... ...your eligibility. Profile review: If everything looks good,...Hourly payTemporary workPart timeRemote work
- ...requirements Help Shape the Future of AI in Healthcare What if the... ...healthcare information, evaluates medical scenarios, and delivers... ...healthcare practices. Medical history review and interpretation of clinical... ...project-specific scoring rubrics, annotation instructions, and...Contract workFor contractorsTraineeshipRemote work
- ...Teach live online classes on grading one AI response against written criteria: accuracy... ...for YPAI Learn trains people for AI evaluation work in live classes. Your classes take learners... ...teaching, coaching new evaluators, reviewing other people's ratings or running...Contract workRemote work
- ...-world primary care knowledge directly ensures AI medical responses are safe, accurate, and effective. You will review patient-facing health advice generated by AI models... ...scenarios. Your core responsibility is to evaluate whether the guidance aligns with established clinical...Contract workFreelanceFlexible hours
- ...your own work and turn each one into a self-contained Task that an AI agent has to solve in a Linux terminal. For every Task, you'll... ...programming languages - **Machine learning:** training, inference, evaluation, and kernels - **Security:** cryptography, reverse engineering,...Shift work
$55 - $110 per hour
...RealTrainAI is recruiting experts for its AI coding & SWE reviewer AI training project. Experts help train and evaluate AI models by reviewing, rating and writing responses in their field of expertise. The work is remote and paid hourly. Pay: $55–$110/hr (industry-reported...Hourly payContract workRemote work- ...French (FR) and interested in working on AI-related projects? We're currently looking... ...talent pool for an upcoming AI Response Evaluation and Fact-Checking project . This freelance... ...misleading, or unsupported information Review user queries and corresponding AI-...Contract workFreelanceRemote workFlexible hours
$150 per hour
...long hiring takes Across the AI training platforms we refer... ..., this Software Engineer - AI Reviewer Expert position compensates Software... ...& Robotics RLHF & AI Evaluation Expert Related jobs AI... ...following a one-size-fits-all rubric. If you don't have hands-on Software...Hourly payRemote work$150 per hour
...long hiring takes Across the AI training platforms we refer... ...role This Data Scientist - AI Reviewer Expert role pays $150/hr for... ...STEM & Science RLHF & AI Evaluation Expert Related jobs Research... ...following a one-size-fits-all rubric. If you don't have hands-on...Hourly payRemote work$10 - $30 per hour
...feedback digital delivery platforms ai-driven tools familiarity independent work... ...models and agents. We provide frontier evaluations and reinforcement learning environments to... ...effective customer service standards. Review context-specific scenarios, such as order...Part timeFor contractorsRemote workShift workusd16 per hour
...We are looking for AI Linguistic Evaluators to assess how effectively an AI application performs in... ...writing, journalism, proofreading, content review, localization, or other language-... ...appropriateness. - Providing structured evaluations and feedback following the project instructions...Hourly payDaily paidFull timeFreelanceImmediate startRemote workFlexible hours- ...Curate and evaluate code in Rust to enhance AI training datasets. About Us Based in San Francisco, California... ..., debugging, and code quality/review assessment. Excellent oral and... ...communication skills for clear, structured evaluation rationales. Engagement Details...Part timeFor contractorsFlexible hours
$55 - $100 per hour
...Pay: $55–100/hour You'll evaluate AI-generated responses to legal questions — covering case... ...work involves reading a legal prompt, reviewing the model's response, and scoring it across... ...arguments. Feedback is written in structured rubric fields, with an optional free-text...Contract work- ...What You'll Do Review real user interaction traces with an AI shopping assistant Identify logical failures, inaccuracies... ...in the text Create structured rubrics and verifiers to judge response... ...Commit to 20+ hours per week of evaluation work on our internal platform Who...Contract workFor contractorsRemote work
- ...interview Walk us through how you currently evaluate new artificial intelligence platforms... ...recent challenges you faced while vetting AI tools Who Should Apply Active role in... ...Comfortable discussing technical and business evaluation criteria in a 1:1 setting Contract &...Contract workFor contractors
$40 per hour
...of Use. To learn more about our privacy practices review our Privacy Policy. Please review your choices. Accept... ...All Employers Job seekers Career centers Handshake AI Enterprise AI Research Log in Sign up FRENCH AI EVALUATION SPECIALIST (CANADA) Sign up now Up to $40/hr...Part timeRemote workFlexible hours- ...the intersection of human creativity and AI. We partner with leading AI labs and product... .... To Join the Network # Portfolio review - We'll assess your Contra profile and portfolio... ...through. What You May Work On Evaluate AI-generated visual outputs across a wide...Full timeContract work
- ...required unless specified. To Join the Network # Portfolio review - We’ll assess your Contra profile and portfolio for strength in... ...photography, composition, lighting, and visual quality. # Skills evaluation - You’ll complete a structured photography submission designed...Full timeContract work
- ...the intersection of human creativity and AI. We partner with leading AI labs and product... .... To Join the Network Portfolio review: We'll assess your Contra profile and portfolio... ...and mobile applications. Skills evaluation: You'll rate AI-generated outputs across...Full timeContract work
- ...Analyze GitHub issues and evaluate Python code quality for open-source... ..., working with frontier AI labs to accelerate model development... ...high-quality training data, evaluations, and engineering talent.... ...approximately 75 mins) : ~ Assessment based review Skills: Python...Contract workPart timeFor contractorsFreelanceManual laborRemote work
$80 - $100 per hour
...seeking an experienced legal professional to support a short-term AI content evaluation project. You’ll use your legal judgment to assess written... ...feedback that improves its quality. What you’ll do Review legal content for accuracy, consistency, and appropriate context...Contract workTemporary workImmediate startRemote work- ...thresholds. Familiarity or comfort with AI / generative models, visual content, or... ...Lionbridge . Background in content evaluation, annotation, or quality assurance for AI/ML... .... Ability to adapt quickly to changing rubrics and instructions. Why Join Rise Data Labs...Contract workRemote workFlexible hours
- ...This is a fully remote, hourly contractor role supporting AI data and language projects on a project-based, flexible hour basis... ...tables, and other content to support AI training datasets. LLM evaluation: reviewing AI-generated responses for accuracy, reasoning quality,...Hourly payContract workFor contractorsRemote workFlexible hours
- ...Engineering · Reactor Systems · AI Safety · Nuclear Physics... ...About the Role AI Safety Evaluation Project | Remote | Contract Locations... ...content. The work involves reviewing AI-generated prompts and responses... ...research, technical review, or quality assurance is a plus...Hourly payContract workFor contractorsRemote work
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Evaluation Rubric Reviewer. Be the first to apply!
