Rubric Evaluation Specialist - Writing
$25 - $45 per hourFull-time
Mercor
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .
Position: Generalist Writer
Type: Contract
Compensation: $25–$45/hour
Location: Remote
Role Responsibilities
- Evaluate AI-generated written conversations using predefined rubrics to enhance performance.
- Draft and refine evaluation rubrics for generalist writing tasks.
- Apply structured scoring criteria to assess clarity, coherence, accuracy, and style.
- Review edge cases and ambiguous outputs to ensure consistent grading standards.
- Work independently and asynchronously to meet deadlines and improve AI model performance .
Qualifications
Must-Have
- Experience with rubric-based grading , editing, or content evaluation at Mercor .
- Strong generalist writing skills across multiple domains (e.g., business, academic, creative).
- Excellent attention to detail and consistency in applying guidelines.
- Ability to interpret nuanced criteria and make sound judgment calls.
Preferred
- Background in English , journalism, communications, or a related field.
Compensation & Legal
- Opportunities for expanded reviewer/lead responsibilities.
Application Process (Takes 20–30 mins to complete)
- Submit a resume or writing sample to get started.
- Selected applicants will be contacted within a few days regarding next steps.
Resources & Support
- For details about the interview process and platform information, please check:
- For any help or support, reach out to: View email address on jobs.jobcopilot.com
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
Vacancy posted 12 hours ago
Similar jobs that could be interesting for youBased on the Rubric Evaluation Specialist - Writing in Toronto, ON vacancy
$35 per hour
...Angelo , Larry Summers , and Jack Dorsey . Position: Writing Expert Type: Freelance Compensation: $35/hour... .... Collaborate in QA review processes to ensure prompt tasks and rubrics meet rigor, maintaining consistency and reliability before integration...SuggestedFull timeSummer workFreelanceRemote workMercor
Toronto, ON12 hours ago$15 - $20 per hour
...and Jack Dorsey . Position: Video Evaluation Generalist Type: Contract... ...two short video clips against a provided rubric for prompt adherence, motion quality, visual... ...institution completed or in progress. Strong writing and critical thinking skills....SuggestedFull timeContract workSummer workRemote workMercor
Toronto, ON12 hours ago$80 - $160 per hour
...structured feedback to improve document standards and outcomes. Analyze and evaluate grant writing and ESG strategy documents. Collaborate with teams to enhance document evaluation processes. Work independently and asynchronously to meet deadlines and project...SuggestedFull timeContract workSummer workRemote workMercor
Toronto, ON12 hours ago$80 - $120 per hour
...Summers , and Jack Dorsey . Position: General Sales / GTM Evaluator Type: Contract Compensation: $80–$120/hour... ...Evaluate AI-generated artifacts against domain-specific quality rubrics. Identify factual, aesthetic, and presentation errors in documents...SuggestedFull timeContract workSummer workWork at officeRemote workFlexible hoursMercor
Toronto, ON12 hours ago$80 - $120 per hour
.... Position: Nonprofit / philanthropy / community programs Evaluator Type: Contract Compensation: $80–$120/hour... ...Evaluate AI-generated artifacts against domain-specific quality rubrics. Identify factual, aesthetic, and presentation errors in outputs...SuggestedFull timeContract workSummer workWork at officeRemote workMercor
Toronto, ON12 hours ago$80 - $120 per hour
...and Jack Dorsey . Position: Process improvement / SOPs Evaluator Type: Contract Compensation: $80–$120/hour... ...Evaluate AI-generated artifacts against domain-specific quality rubrics. Identify factual, aesthetic, and presentation errors....Full timeContract workSummer workWork at officeRemote workMercor
Toronto, ON12 hours ago$70 - $100 per hour
...Commitment: 20–40 hours/week Role Responsibilities Evaluate AI-generated office artifacts in Excel , Word , and... ...structure, and formulas. Grade outputs against structured rubrics to improve model behavior. Provide written feedback to the...Full timeContract workSummer workWork at officeImmediate startRemote workMercor
Toronto, ON12 hours ago$15 - $20 per hour
...sources and external tools . Generate high-quality human evaluation data by identifying response strengths, areas for improvement, and... ...experience using large language models (LLMs) Excellent writing skills in English Strong attention to detail Background...Full timeContract workSummer workRemote workMercor
Toronto, ON12 hours ago$90 per hour
...nav bar that looks right at 1920px but collapses at 1400px. Write specific, evidence-cited justifications for every preference.... ...Preferred Prior RLHF , preference-labeling, model-evaluation, or structured code-review work. Pixel-perfect design-to-code...Full timeContract workSummer workLocal areaRemote workMercor
Toronto, ON12 hours ago$80 - $160 per hour
...Provide structured feedback to enhance document relevance and impact. Utilize strong analytical skills to contribute to high-impact evaluation projects. Collaborate with teams to improve document standards and guidelines. Work independently and asynchronously to...Full timeContract workSummer workRemote workMercor
Toronto, ON12 hours ago$70k - $73.85k per year
...Posting will be closed on July 29, 2026 at 5:00 pm and this posting is for an existing vacancy. Objective: The Research and Evaluation Specialist will lead the analysis and development of sector-based program and policy research and evaluation that will enhance Friendship...Full timeInternshipOfifc
Toronto, ON12 hours ago$80 - $150 per hour
..., and Jack Dorsey . Position: MS Excel / Google Sheets Evaluator Type: Contract Compensation: $80–$150/hour... ...relevance. Provide structured feedback to enhance document evaluation processes. Utilize MS Excel and Google Slides to perform...Full timeContract workSummer workRemote workMercor
Toronto, ON12 hours ago$14 - $42 per hour
...Location: Remote Duration: Up to 6 months Commitment: 20+ hours/week Role Responsibilities Evaluate AI-generated music across various genres. Rate against detailed quality standards. Compare AI-generated lyrics with published songs...Full timeContract workSummer workImmediate startRemote workMercor
Toronto, ON12 hours ago$120 - $170 per hour
...comfortable having your judgment reviewed and calibrated against peers. Preferred ~ Prior experience with AI training , evaluation, or human-data projects. Application Process (Takes 20–30 mins to complete) Submit your resume or relevant technical...Full timeContract workSummer workRemote workMercor
Toronto, ON12 hours ago$80 - $120 per hour
...and Jack Dorsey . Position: Data quality / CRM operations Evaluator Type: Contract Compensation: $80–$120/hour... ...Evaluate AI-generated artifacts against domain-specific quality rubrics. Identify factual, aesthetic, and presentation errors in...Full timeContract workSummer workWork at officeRemote workMercor
Toronto, ON12 hours ago$80 - $120 per hour
...Position: Compliance / regulatory response with financial-services AI Evaluator Type: Contract Compensation: $80–$120/hour... ...AI-generated artifacts against domain-specific quality rubrics. Identify factual, aesthetic, and presentation errors....Full timeContract workSummer workWork at officeRemote workMercor
Toronto, ON12 hours ago$80 - $120 per hour
...Position: Procurement / vendor management Evaluator Type: Contract... ...artifacts against domain-specific quality rubrics. Identify factual, aesthetic, and presentation... ...deadlines while ensuring high-quality evaluations. Qualifications Must-Have...Full timeContract workSummer workWork at officeRemote workMercor
Toronto, ON12 hours ago$80 - $120 per hour
..., and Jack Dorsey . Position: Software / AI / IT / data Evaluator Type: Contract Compensation: $80–$120/hour... ...Evaluate AI-generated artifacts against domain-specific quality rubrics. Identify factual, aesthetic, and presentation errors in outputs...Full timeContract workSummer workWork at officeRemote workMercor
Toronto, ON12 hours ago$80 - $120 per hour
...Summers , and Jack Dorsey . Position: Legal / compliance Evaluator Type: Contract Compensation: $80–$120/hour... ...Evaluate AI-generated artifacts against domain-specific quality rubrics. Identify factual, aesthetic, and presentation errors....Full timeContract workSummer workWork at officeRemote workMercor
Toronto, ON12 hours ago$100 - $150 per hour
...modeling work , machine learning pipelines , and A/B test write-ups . Evaluate AI-generated or human-created work against established... ...consistency. Provide detailed written justifications for evaluations and scores to maintain transparency and accountability....Full timeContract workSummer workRemote workMercor
Toronto, ON12 hours ago$60 - $80 per hour
...Jack Dorsey . Position: Business Domain Expert — AI Model Evaluation Pilot (Admin, Marketing, HR, Accounting) Type: Contract... ...that task. Rank them 1–5 with no ties, score each 0–100, and write evidence-based rationales citing specific parts of the output....Hourly payFull timeContract workFor contractorsSummer workRemote workMercor
Toronto, ON12 hours ago- ...collaboration with other disciplines - all foundational ingredients in successful digital experiences and client partnerships. As a QA / AI Evaluation Engineer, you will join a highly motivated and experienced team in a forward-leaning role that proves the platform actually improves...Full timeImmediate start
Appnovation Technologies
Toronto, ON12 hours ago $100 - $150 per hour
...handling technical objections, supporting proofs of concept, and evaluating product fit and technical risk. ~ Hands-on experience with... ...Preferred ~ Prior experience with AI training , evaluation, or human-data projects. Application Process (Takes 20–30...Full timeContract workSummer workRemote workMercor
Toronto, ON12 hours ago$35 - $40 per hour
...Jack Dorsey . Position: Writing Expert Type:... ...instructions. Define and document evaluation standards for correct responses... ...contexts. Develop comprehensive rubrics. Conduct model testing... ...-minute written assessment on rubric writing. If selected, you...Full timeSummer workFreelanceRemote workMercor
Toronto, ON12 hours ago$80 - $160 per hour
...structured feedback to improve document standards and outcomes. Analyze and evaluate grant writing and ESG strategy documents. Collaborate with teams to enhance document evaluation processes. Work independently and asynchronously to meet deadlines and project...Full timeContract workSummer workRemote work$50k - $60k per year
...SUMMARY The Monitoring, Evaluations & Learning Officer (MELO) provides technical expertise and support to strengthen the application of Catalyste+’s MEL system. Reporting to the MEL Manager, the individual will support the MEL team in ensuring the effective application...Full timeApprenticeshipSummer workWork at officeRemote workCatalyste
Toronto, ON12 hours ago$40 - $50 per hour
...week Role Responsibilities Evaluate full shopper-agent conversations,... ...and a specific secondary tag. Write concise comments explaining the shopper... ...annotation, labelling, or model-evaluation work with a defined rubric. Experience with the Shop app or...Full timeContract workSummer workRemote workMercor
Toronto, ON12 hours ago$80 per hour
...Compensation: $80/hour Location: Remote Role Responsibilities Use frontier AI coding agents to complete and evaluate complex data engineering tasks. Review model-generated implementations involving ETL pipelines , data warehouses ,...Full timeContract workSummer workRemote workMercor
Toronto, ON12 hours ago$70 - $100 per hour
...Commitment: 10–20+ hours/week Role Responsibilities Evaluate AI-generated office artifacts across Excel , Word , and... ...formatting, and visual structure. Grade outputs using structured rubrics and provide feedback to improve model behavior. Participate...Remote jobContract workSummer workWork at officeImmediate start$180 per hour
...Internal Medicine , Emergency Medicine , and Cardiology . Write "golden" reference responses at attending-level quality for... ...treatment planning. Grade AI-generated responses using structured rubrics to ensure adherence to evidence-based standards. Provide...Weekly payFull timeContract workSummer workWork at officeRemote workFlexible hoursMercor
Toronto, ON12 hours ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Rubric Evaluation Specialist - Writing. Be the first to apply!
Related searches
- crisis intervention specialist Toronto, ON
- leasing specialist Toronto, ON
- visual specialist Toronto, ON
- child life specialist Toronto, ON
- lead generation specialist Toronto, ON
- spécialiste en amélioration continue Toronto, ON
- spécialiste paie Toronto, ON
- literacy specialist Toronto, ON
- building performance specialist Toronto, ON
- employment specialist Toronto, ON