Evaluation Scenario Writer - QA
$38 per hourMindrift
Overview
Be among the first 25 applicants. At Mindrift, innovation meets opportunity. We believe in using the power of collective human intelligence to ethically shape the future of AI. The Mindrift platform, launched and powered by Toloka, connects domain experts with cutting-edge AI projects from innovative tech clients. Our mission is to unlock the potential of GenAI by tapping into real-world expertise from across the globe.
Who we’re looking for: We’re looking for curious and intellectually proactive contributors who never miss an error and can think outside of the box when brainstorming solutions. Are you comfortable with ambiguity and complexity? Does an async, remote, flexible opportunity sound exciting? Would you like to learn how modern AI systems are tested and evaluated?
This is a flexible, project-based opportunity well-suited for:
- Analysts, researchers, or consultants with strong critical thinking skills
- Students (senior undergrads / grad students) looking for an intellectually interesting gig
- People open to a part-time and non-permanent opportunity
About the project
We’re on the hunt for an Evaluation Scenario Writer - QA for a new project focused on ensuring the quality and correctness of evaluation scenarios created for LLM agents. This project opportunity blends manual scenario validation, automated test thinking, and collaboration with writers and engineers. You will verify test logic, flag inconsistencies, and help maintain a high bar for evaluation coverage and clarity.
What you’ll be doing
- Reviewing and validating test scenarios from Evaluation Writers
- Spotting logical inconsistencies, ambiguities, or missing checks
- Sugesting improvements to structure, edge cases, or scoring logic
- Collaborating with infrastructure and tool developers to automate parts of the review
- Creating clean and testable examples for others to follow
Although we’re only looking for experts for this current project, contributors with consistent high-quality submissions may receive an invitation for ongoing collaboration across future projects.
How To Get Started
Apply to this post, qualify, and get the chance to contribute to a project aligned with your skills, on your own schedule. Shape the future of AI while building tools that benefit everyone.
Requirements
The ideal contributor will have:
- Strong QA background (manual or automation), preferably in complex testing environments
- Understanding of test design, regression testing, and edge case detection
- Ability to evaluate logic and structure of test scenarios (even if written by others)
- Experience reviewing and debugging structured test case formats (JSON, YAML)
- Familiarity with Python and JS scripting for test automation or validation
- Clear communication and documentation skills
- Willingness to occasionally write or refactor test scenarios
We also value applicants who have:
- Experience testing AI-based systems or NLP applications
- Familiarity with scoring systems and behavioral evaluation
- Git/GitHub workflow familiarity (PR review, versioning of test cases)
- Experience using test management systems or tracking tools
Benefits
Contribute on your own schedule, from anywhere in the world. This opportunity allows you to:
- Get paid for your expertise, with rates that can go up to $38/hour depending on your skills, experience, and project needs
- Take part in a flexible, remote, freelance project that fits around your primary professional or academic commitments
- Participate in an advanced AI project and gain valuable experience to enhance your portfolio
- Influence how future AI models understand and communicate in your field of expertise
Referrals increase your chances of interviewing at Mindrift.
Get notified about new Writer jobs in Canada .
Greater Montreal Metropolitan Area $45.00-$45.00 2 days ago
Alberta, Canada CA$25.00-CA$29.00 2 weeks ago
Toronto, Ontario, Canada $55.00-$60.00 2 weeks ago
Toronto, Ontario, Canada CA$45.00-CA$55.00 2 weeks ago
Seniority level
- Internship
Employment type
- Part-time
Job function
- Other
Industries
- IT Services and IT Consulting
Toronto, Ontario, Canada
#J-18808-LjbffrVacancy posted 22 hours ago
Similar jobs that could be interesting for youBased on the Evaluation Scenario Writer - QA in Ottawa, ON vacancy
$38 per hour
A technology consulting firm in Canada is seeking an Evaluation Scenario Writer for a remote, part-time role. This position requires a strong QA background in complex testing environments and offers flexible scheduling. Ideal candidates will evaluate logical structures, collaborate...SuggestedPart timeRemote workFlexible hoursMindrift
Ottawa, ON22 hours ago$45 per hour
...who can design realistic and structured evaluation scenarios for LLM-based agents. You'll create test... ...actions against. You'll work to ensure each scenario is clearly defined, well-scored, and... ...or other related fields Background in QA, software testing, data analysis, or NLP...SuggestedFreelanceRemote workFlexible hoursMindrift
Ottawa, ON22 hours ago$45 per hour
...leading AI solutions company is looking for an expert to design and evaluate test scenarios for AI agents. The ideal candidate will have a Bachelor's or Master's degree in a relevant field, experience with QA and data analysis, and strong communication skills in English. This...SuggestedFreelanceRemote workFlexible hoursMindrift
Ottawa, ON22 hours ago$80k - $94k per year
...DESCRIPTION : We are seeking a Proposal Writer to support and grow Levio's presence... ...resources Managing, writing, and conducting QA on proposals Facilitating collaboration... ...selection process. No AI is being used to evaluate applications for this role. Position...SuggestedPermanent employmentFull timeContract work$70k - $80k per year
...Position: Monitoring and Evaluation Officer Location: Remote with Hybrid Option (Ottawa) Salary Range: $70,000 - $80,000 Summary CODE is seeking a collaborative and learning-focused Monitoring and Evaluation (M&E) Officer to join our Programs Team. Reporting to...SuggestedTemporary workLocal areaRemote workFlexible hoursCodengo
Ottawa, ONa month ago- ...Synthires seeks experienced Journalists and Writers to evaluate and improve AI-generated written content. You’ll apply journalism, writing, editing, and fact-checking expertise to assess accuracy, clarity, tone, and effectiveness. No prior AI experience is required—your professional...Remote work
Synthires
Ottawa, ON22 hours ago - ...Novara is seeking a QA Engineer to own the quality bar for AI-powered features, including non-deterministic systems and agent-based workflows. You will define what getting it right means, build tooling, and partner with product and engineering to bring rigor to testing LLM-...
Novara
Ottawa, ON22 hours ago $85k - $100k per year
...About the Role We are looking for a QA Engineer II to join our Banking Platform... ...configuration details to define effective test scenarios and test cases. Perform functional,... ...test evidence, defect updates, and scenario clarification when required. Design, develop...Full time- ...Description ADGA is seeking an experienced Senior Technical Writer to support a Department of National Defence initiative within... ...intelligence–enabled tools; however, AI is not used to screen, evaluate, or make decisions about applicants for this role. *Unless...Full timeContract workTemporary workPart timeInternshipFlexible hours
- Company Description Established in 1991, Collabera has been a leader in IT staffing for over 22 years and is one of the largest diversity IT staffing firms in the industry. As a half a billion dollar IT company, with more than 9,000 professionals across 30+ offices, Collabera...Permanent employmentFull time
Collabera
Ottawa, ONmore than 2 months ago $35 - $60 per hour
A recruitment firm is seeking a Chemistry Expert to review and verify AI-generated content and develop chemistry problems. This remote role requires an advanced degree and strong analytical skills. The compensation ranges from $35 to $60 per hour, tailored to skills and experience...Hourly payRemote workCrossing Hurdles
Ottawa, ON22 hours ago$45 per hour
...completeness. This opportunity allows for flexible, remote work fitting around your academic or professional schedule, paying up to $45 per hour based on experience. Ideal candidates will have a curiosity for complex systems and a knack for evaluating decision points. #J-18808-LjbffrHourly payPart timeRemote workFlexible hoursMindrift
Ottawa, ON22 hours ago- ...WORK WITHOUT VISA SPONSORSHIP FROM THIS AREA* **EXPERIENCE WRITING FOR A MEDICAL COMMUNICATIONS COMPANY IS REQUIRED** Our Medical Writers work closely with members of the scientific team to develop high-quality medical communications materials across a range of formats,...Full timeWork at officeRemote work
- ...Technical Writer / Editor Role Type: Contractor Location:... ...seeking experienced Technical Writers / Editors to contribute to a customer project focused on evaluating and improving advanced AI systems... ...skills throughout the evaluation process. Preferred Qualifications...Remote jobFor contractors
$70 per hour
...leading tech companies, focused on testing, evaluating, and improving AI systems. Participation... ...clinical EHR vignettes for a medical AI evaluation program. General and internal medicine... ...exam-question review, or medical annotation/QA (a plus); Strong written English (C1+)...Hourly payPermanent employmentPart time10 hours per week$56 per hour
...Takers come from a variety of different backgrounds: we work with corporate professionals, executive assistants, journalists, lawyers, writers, adventurers, retirees, stay-at-home parents, and incredible people from all walks of life and at every stage of their careers. If...Permanent employmentFull timePart timeInternshipLocal area$45 per hour
...learn how modern AI systems are tested and evaluated? Who You'll Be Ideal For Analysts,... ...holistically and thinking through scenarios, implications, and edge cases. You do... ...tested properly Working closely with QA, writers, or developers to suggest refinements or...Hourly payPermanent employmentPart timeFreelanceRemote workFlexible hoursMindrift
Ottawa, ON22 hours ago- ...WITHOUT VISA SPONSORSHIP FROM THIS AREA* **EXPERIENCE WRITING FOR A MEDICAL COMMUNICATIONS COMPANY IS REQUIRED** Our Senior Medical Writers have an important role in the development and delivery of scientifically accurate content for our clients’ projects, helping to...Full timeRemote work
- ...Coproration requires to Develop, plan, Analyze, Evaluate requirements with team and project... ...planning. Prepare research, evaluation and statistical reports for client management... ...You can evaluate incident management scenarios, operational policy requirements, identify...
TRM Technologies Inc.
Ottawa, ON-70 - Reporting to the Director, Service Management, the successful candidate will provide technical and team leadership for the testing, quality assurance, release management, and production operations of Lightspeed business and technical applications and platforms. The candidate...Full timeRelocationRelocation package
$75k - $90k per year
...shape the future of care. The Opportunity The Senior Technical Writer owns documentation for one or more product areas of the... ...enough for someone to make a call, and grow the team's scoping until writers handle their own domains unaided. Standards, quality, and governance...Full timeRemote workFlexible hours- Reporting to the Director, Service Management, the successful candidate will provide technical and team leadership for the testing, quality assurance, release management, and production operations of Lightspeed business and technical applications and platforms. The candidate...Full timeRelocationRelocation package
$57.51k - $75.48k per year
...Header Technical Writer (6-month contract) Ottawa, ON / Victoria, BC / Halifax, NS Expected Salary: $57,508 to $75,479 To determine final salary, Babcock considers a variety of factors including the successful Candidate’s skills and experience and internal equity...Contract workFixed term contractFor subcontractorInternshipusd15 per hour
...words? Do you want to influence the future of global AI search? If so, we want you on board! Working on behalf of RWS, you will test and evaluate how our clients’ search engine responds to everyday searches from everyday people. Using your search expertise, you will work through...Weekly payFull timePart timeFreelanceSecond jobRemote workWork from homeHome officeFlexible hours- ...QA Analyst About Fellow Fellow is the AI meeting assistant that helps teams turn conversations into decisions and action, built... ...coverage, add tests for new risks and escaped defects, update outdated scenarios, and remove tests that no longer provide value. Investigate...Remote work
Jaide Health
Ottawa, ON22 hours ago - ...payload on ViaSat-1 and one LEO 3 demonstration satellite. For more information, follow Telesat on X and LinkedIn or visit The Technical Writer will support the User Terminal (UT) Engineering team by developing, maintaining, and improving technical documentation throughout the...Full timeWork at officeWorldwide
$19.75 per hour
...interfere with the authenticity of the assessment process. Role Information: EN Studio: Keywords Studios Location: Americas, Canada, Ottawa Area of Work: QA Testing Services Service: Globalize Employment Type: Full Time Working Pattern: In-Office, Remote...Hourly payPermanent employmentFull timeLive InWork at officeRemote workWork from homeTrial periodWork visa$45 - $60 per hour
...assistance. Our client is currently seeking a qualified Technical Writer to join their team onsite in Ottawa, ON. Duties And... ...que l'intelligence artificielle (IA) peut être utilisée pour filtrer, évaluer ou sélectionner les candidats pour ce poste. #J-18808-LjbffrMinimum wageWork at officeLocal areaCompucom Staffing
Ottawa, ON-70- ...Experience: 6-8 Years Role Summary Seeking a highly skilled QA Tester with hands-on experience in Netcracker BSS... ...as well as middleware and API layers. Develop and execute test scenarios, test cases, and test data for complex business workflows. Conduct...Work at officeRemote work
HCL Technologies Limited
Ottawa, ON-70 - ...SecOps. That’s Ontinue. Role Overview We’re looking for a QA Automation Engineer to help us scale quality across a fast-moving... ...Azure-based and event-driven workflows, including message-based scenarios such as Service Bus topics and related integrations Combine automation...
Ontinue
Ottawa, ON22 hours ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Evaluation Scenario Writer - QA. Be the first to apply!
