Legal Reasoning AI Response Evaluator
$55 - $100 per hourTemporary
Claru
Pay: $55–100/hour
You'll evaluate AI-generated responses to legal questions — covering case analysis, statutory interpretation, procedural questions, and litigation strategy. Your primary task is assessing whether a model's answer reflects sound legal reasoning, accurate doctrinal knowledge, and appropriate caveats. This isn't about copy-editing; you're judging whether the legal analysis would hold up in front of a practitioner, and flagging anything that could mislead a non-expert relying on the output.
Day-to-day work involves reading a legal prompt, reviewing the model's response, and scoring it across dimensions like analytical rigor, jurisdictional accuracy, and practical usefulness. For litigation-focused tasks, you'll also evaluate whether the model correctly identifies procedural posture, burden of proof, and viable arguments. Feedback is written in structured rubric fields, with an optional free-text section for nuanced critique. Tasks are asynchronous — you work batches on your own schedule through a browser-based annotation platform.
A minimum of 3 years of litigation or substantive legal practice is required. Familiarity with U.S. federal and state court procedure is expected; experience in additional jurisdictions is a bonus. Law professors and clerks are also strong candidates. You do not need to be currently practicing, but you must be able to credibly evaluate whether legal reasoning is sound, not just grammatically correct.
Skills & requirements
- Legal reasoning and case analysis
- Litigation and civil procedure
- Statutory and regulatory interpretation
- Analytical writing
- Jurisdictional knowledge (US federal/state)
Category: RLHF
Vacancy posted 8 days ago
Similar jobs that could be interesting for youBased on the Legal Reasoning AI Response Evaluator in Remote vacancy
$50 - $150 per hour
...hiring PhD-level experts to help train and evaluate advanced AI models. This is remote, flexible... ...directly shapes how cutting-edge models reason, answer, and improve. You'll review AI... ...depending on project needs. Key Responsibilities You may contribute your expertise by...SuggestedContract workTemporary workInternshipRemote workFlexible hours- ...Create and evaluate board game tasks to enhance AI reasoning skills. About Turing Turing is one of the world’s fastest-growing AI companies, accelerating... ..., and rule-based systems. Evaluate AI-generated responses for correctness, consistency, and reasoning quality....SuggestedContract workPart timeFor contractorsFreelanceRemote work
- ...editing skills in your native language. Strong attention to detail and ability to follow written guidelines. Comfortable working with AI-generated content. Reliable internet connection. Ability to work remotely using a client-provided laptop. French (France)...SuggestedRemote work
- ...Evaluate AI chatbot responses for small business scenarios in Spanish. About Turing: Turing is one of the world’s fastest-growing AI companies... ...labs to advance frontier model capabilities in thinking, reasoning, coding, agentic behavior, multimodality, multilinguality,...SuggestedPart timeFreelanceRemote workFlexible hours
- ...Specialist in our Core Engineering & AI Development Community , you are the... ...Engineer. What You’ll Do (Key Responsibilities): Algorithmic Reasoning Validation: Audit and refine "Chain... ...Code Quality & Security Auditing: Evaluate AI-generated code for security vulnerabilities...SuggestedPart time
$40 per hour
...seekers Career centers Handshake AI Enterprise AI Research Log in Sign up FRENCH AI EVALUATION SPECIALIST (CANADA) Sign up now... ...based in Canada to help make AI responses more personalized. This is not a... ...sharing your honest preferences and reasoning Provide clear written feedback...LegalPart timeRemote workFlexible hours- ...Design tests to evaluate AI safety and identify model failures. About Turing: Based in... ...AI researchers who specialize in coding, reasoning, STEM, multilinguality, multimodality,... ...to provide services to others, and are responsible for their own taxes, insurance, and expenses...Ongoing contractPart timeFor contractors
- ...a global community of experts to evaluate, challenge and improve advanced AI systems . We are looking for... ...concepts and technical outputs Model reasoning and performance Data and... ...Hallucinations and factual accuracy LLM responses and instruction following...Part time
- ...About the Role As a Legal & Compliance AI Specialist, you will directly improve the accuracy, safety... ...professional judgment will teach AI to reason about statutes, case law, and the... ...Reviewing AI-Generated Legal Summaries: Evaluating AI’s interpretation of statutes, case opinions...LegalHourly payContract work
- ...Job Application for Geospatial Reasoning Specialist (Senior) - Freelance AI Trainer Project at Meridial Back to... ...reasoning agent focused on Crisis Response—helping AI systems understand, assess... ...expertise will guide model development and evaluation, laying the foundation for future...Hourly payContract workPart timeFor contractorsFreelanceRemote work
- ...About the Role: Do you have a background in Physics or Mechanical Engineering ? AI models are getting smarter, but they often struggle with "common sense" physics—like how gravity works, how objects move, or how much force is needed to lift something. At TELUS Digital...Part time
usd16 per hour
...We are looking for AI Linguistic Evaluators to assess how effectively an AI application performs in... ...Evaluating the quality of AI-generated responses according to project guidelines. - Assessing... .... - Providing structured evaluations and feedback following the project instructions...Hourly payDaily paidFull timeFreelanceImmediate startRemote workFlexible hours$13 per hour
...Argentina | Belize | Bolivia + 15 more Help evaluate how an AI assistant uses information from... ...Canada, outside Quebec, who can review AI responses carefully and explain what works and... ...guidelines for consistent, confidential evaluation work. Important account-access...Hourly payContract workFor contractorsRemote work- ...Key Responsibilities • Design and implement evaluation metrics for AI models • Conduct thorough testing of model performance across different scenarios • Analyze model outputs for bias, fairness, and accuracy • Collaborate with ML engineers to implement improvements...Contract work
- © 2026 Smart Ecosystems. All rights reserved.Full time
- ...What You'll Do Apply your subject-matter expertise to advanced AI training and evaluation work Develop or evaluate challenging problems within your area of expertise Assess AI-generated reasoning for technical accuracy and identify subtle errors or failure modes...Contract workImmediate startRemote workFlexible hours
- ...Join a leading AI lab's cutting-edge GenAI team to be at the... ...improving how frontier AI models reason about real financial work.... ...assistance is not provided. # Key Responsibilities Data QA and reviews: Vet... ...finance tasks and evaluation sets, and help build finance-...LegalFull timeContract workPart timeFreelanceLive InManual laborRelocationRelocation package
$80 - $100 per hour
...| Canada + 1 more We’re seeking an experienced legal professional to support a short-term AI content evaluation project. You’ll use your legal judgment to assess... ...consistency, and appropriate context. Identify gaps in reasoning, ambiguous language, or conclusions that need...LegalContract workTemporary workImmediate startRemote work- ...Radiological Science · Health Physics · AI Safety · CBRN What this pays Rates... ...stand behind. About the Role AI Safety Evaluation Project | Remote | Contract Locations: US... ...reviewing AI-generated prompts and responses, evaluating safety risks, identifying dual...Hourly payContract workFor contractorsRemote work
- ...ll Do Review real user interaction traces with an AI shopping assistant Identify logical failures, inaccuracies... ...Create structured rubrics and verifiers to judge response quality Commit to 20+ hours per week of evaluation work on our internal platform Who Should Apply...Contract workFor contractorsRemote work
$70 - $140 per hour
...RealTrainAI is recruiting experts for its Medical knowledge & clinical reasoning AI training project. Experts help train and evaluate AI models by reviewing, rating and writing responses in their field of expertise. The work is remote and paid hourly. Pay: $70–$140/hr...Hourly payContract workRemote work- ...on behalf of a partner company, which manages all applications and next steps. Our partner is looking for a Senior Data Scientist AI Evaluation based in Canada. We are seeking a senior-level data scientist to establish and advance the evaluation practices that measure the quality...Full time
$130 per hour
...specialists with project-based AI opportunities for leading tech... ..., focused on testing, evaluating, and improving AI systems. Participation... ...vignettes for a medical AI evaluation program. General and internal... ...systems; Score the clinical reasoning quality of benchmark items:...Hourly payPermanent employmentPart time10 hours per week- ...About micro1 micro1 is the leading AI data lab for training frontier models and evaluating AI agents. Experts contribute... ...that improve how AI systems learn, reason, and perform. Our platform identifies... ...frontier AI. Role Title: AI Legal Fellow — Litigation & Motion...LegalPart timeFor contractorsRemote work
$30 - $90 per hour
...backgrounds to support quality control on AI reasoning datasets. This role requires... ...Do Review and validate AI-generated responses in English for accuracy and reasoning quality... ...Nice to Have Experience with AI/ML evaluation or data annotation Background in AI,...Contract workRemote work10 hours per weekFlexible hours$75 - $90 per hour
...States | Canada + 21 more Help a top AI lab evaluate and improve large language models... ...document model failures, edge cases and reasoning gaps. Compare private language models... ...Build or configure coding environments for evaluation and reinforcement learning. Follow...Long term contractContract workRemote work- ...You'll use your finance expertise to help evaluate and improve advanced AI systems on complex, realistic financial... ...AI-generated work for accuracy, reasoning, completeness, and professional quality. Develop clear evaluation rubrics and scoring criteria, identifying...Contract work
- ...Responsibilities • Review and categorize legal documents • Identify key legal concepts and entities • Annotate contracts and legal texts • Ensure accuracy and legal compliance • Provide feedback on AI model outputs Qualifications • JD from accredited law school...LegalContract work
- ...About the Role: AI is great at writing, but it often makes legal documents too confusing or misses simple rules. As a Legal Content Expert , your... ...the "Common Sense" filter for legal AI. Key Responsibilities: Simple Fact-Checking: Make sure the AI is using...LegalContract work
- ...Clinical Mental Health Experts to support an AI research initiative focused on evaluating the realism and quality of simulated... ...We consider all qualified applicants without regard to legally protected characteristics and provide reasonable accommodations upon request....Temporary workPart timeRemote work
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Legal Reasoning AI Response Evaluator. Be the first to apply!
