AI Benchmark Researcher
$1 350 per unitTemporary
Braintrust
- Rate: $1,350/task
- Hours: 20 hours / week
- Experience: 10+ years
- Location: United States | Canada + 23 more
Bring your expertise to the problems today’s most advanced AI agents still struggle to solve.
We’re looking for experienced professionals to develop challenging tasks for an AI benchmark. You’ll draw on problems from your own field and turn them into clearly defined challenges that an AI agent must solve from scratch in a Linux terminal.
What you’ll do
- Identify difficult, meaningful problems in your area of expertise.
- Translate those problems into tasks that can be solved in a Linux terminal.
- Design challenges that expose limitations in advanced AI agents. Each task is tested against three leading models, with three attempts per model, and must prove difficult enough that the models mostly fail.
Who we’re looking for
- A completed master’s degree or higher. Candidates with a bachelor’s degree, at least 10 years of relevant professional experience and a publication may also be considered.
- At least 10 years of professional experience in a technical domain.
- Expertise in coding, machine learning, systems, cybersecurity or hardware.
- At least one academic or professional publication.
- Practical comfort working in a Linux terminal.
- Currently based in the United States.
Project details
- Task-based project expected to run for at least seven weeks.
- Onboarding includes two courses and a graded assessment, taking approximately 90 minutes.
- A live onboarding webinar is also required.
Compensation
- Earn up to $1,350 per task , with opportunities for higher rates based on task quality and submission volume.
- Tasks take six to seven hours on average, although initial tasks may take longer as you get familiar with the process.
Company: Outlier
Vacancy posted 12 days ago
Similar jobs that could be interesting for youBased on the AI Benchmark Researcher in Remote vacancy
- ...Paid, remote work authoring and validating research-grade ML benchmark tasks. For Master's or PhD-level ML researchers with at least one first-author paper. Domain: Machine Learning / AI Research Requirements Experience: 1+ years of hands-on ML research; no upper limit Education...SuggestedContract workRemote work
- ...About the Role Join our global AI Community as a Bioinformatics AI Expert specializing in advanced benchmark dataset creation and AI model evaluation. In this role, you will design complex, reasoning-intensive, "information-reversal" evaluation tasks to measure the...SuggestedPart time
- ...geo-diverse facilities. Tether Data: Fueling breakthroughs in AI and peer-to-peer technology, we reduce infrastructure costs and... ...be part of the future? About the job As a member of our AI research team, you will drive innovation in model compression and efficient...SuggestedFull timeImmediate startRemote workWorldwide
- ...Investigate and develop systems to enhance frontier AI capabilities. About Turing: Based in... ...California, Turing is the world’s leading research accelerator for frontier AI labs and a... ..., post-training, model understanding, benchmarks, and evaluation. What You’ll Do...SuggestedContract workPart timeFor contractors
$30 - $90 per hour
...Canada + 23 more About the Role We're looking for detail-oriented reviewers with strong STEM backgrounds to support quality control on AI reasoning datasets. This role requires professional fluency in English, as you'll be reviewing and validating English-language technical...SuggestedContract workRemote work10 hours per weekFlexible hours- ...speaking software engineers to design, build, and validate these benchmarks. You will create high-signal, high-quality tasks that genuinely test... .... - Prompting & Translation: finding failure points where AI does not work, in your native language - Implementation & Verification...Full timeFor contractorsFreelanceRemote work
- ...About The Role We’re hiring AI Data Trainers – Biomedical Research & Genomics to help shape the future of AI in biomedical sciences. In this role, you’ll evaluate and provide expert feedback on AI-generated content related to genomic data, molecular biology, and biomedical...Contract workWorldwide
$50 - $150 per hour
...quantitative finance experts to help train and evaluate advanced AI models on financial and quantitative reasoning. This is remote, flexible... ...ability to communicate complex ideas clearly Nice to Have Research experience (academic or industry, including buy-side or sell-side...Contract workTemporary workInternshipRemote workFlexible hours- Contract & Payment Terms You will be engaged as an independent contractor. This is a fully remote opportunity that can be completed on your own schedule. Opportunities can be extended, shortened, or concluded early depending on needs and performance. Your participation will...Contract workFor contractorsRemote work
- ...We're building a research cohort of senior domain experts across seven specialist tracks — software/systems engineering, formal methods and... ...design, and creative technology — to help evaluate how frontier AI models reason about hard, real-world expert-level problems. This...Contract work
- ...Design challenging research questions that AI cannot answer online. About Turing: Turing is one of the world’s fastest-growing AI companies... ...companies. About the Role We are building an evaluation benchmark for frontier AI browsing agents. Your job is to design...Part timeRemote work
$150k - $190k per year
...About the role: We are seeking an AI/ML Engineer to join BPM’s Enterprise Technology... ..., and presentations. Adept at researching new AI tools, frameworks, and industry trends... ...with credential requirements and market benchmarks, are incorporated into compensation ranges...Local areaImmediate startRemote workFlexible hoursShift work- ...pay Required Skills equity research financial modeling earnings analysis peer benchmarking sector expertise investment thesis development written... ...analysis About micro1 micro1 is the leading AI data lab for training frontier models and evaluating...Part timeFor contractorsRemote work
$39 per hour
...About the role Help train and evaluate AI agents on expert-level fluid simulation tasks... ...Several years of hands-on industry or research CFD with OpenFOAM, ANSYS Fluent/CFX or STAR... ...studies, and validation against experiment or benchmarks. Comfortable with Python and GitHub....Hourly payFreelanceRemote work10 hours per week- ...Circle is building the world’s leading AI-powered, all-in-one platform for digital... ...improve production AI systems, rather than ML research or model training. We’re looking for a... ...& BENEFITS Circle offers U.S.-benchmarked compensation globally, equity in the company...Full timeRemote workShift work
$70 per hour
...Pay: up to $70/hour In this hourly, remote contractor role, you will work as a Clinical Researcher Subject Matter Expert (SME) to review AI-generated clinical research and biomedical content and/or create expert research content, evaluating reasoning quality and step-by...Hourly payContract workFor contractorsImmediate startRemote work$300 per unit
...Design terminal-based engineering tasks for AI training and evaluation. Role Overview:... .... Experience developing technical benchmarks, programming tasks, simulation-based evaluations... ...coding or terminal agents or working in research software engineering. Publications,...Contract workPart timeFor contractorsManual labor- ...Tether Data: Fueling breakthroughs in AI and peer-to-peer technology, we reduce infrastructure... ...and vision, applying state-of-the-art research to improve model quality, efficiency, and... ...and implement evaluation frameworks and benchmarks to measure model performance, robustness,...Permanent employmentFull timeImmediate startRemote workWorldwide
- ...Tether Data: Fueling breakthroughs in AI and peer-to-peer technology, we reduce infrastructure... ...interactions. You will adopt a hands-on, research driven approach to developing, testing,... ...workflows, evaluation frameworks, and benchmark infrastructure. Develop rigorous evaluation...Permanent employmentFull timeImmediate startRemote workWorldwide
- ...work focuses on diverse areas including AI, social media, security and more. And we’... ...Performance: Implement rigorous frameworks for benchmarking, prompt engineering, and evaluating agent... ...and acting as a primary bridge between research and product. What You’ll Bring:...Long term contractFull timeInternshipImmediate startHome office
- ...your subject-matter expertise to advanced AI training and evaluation work Develop... ...scientific discipline Advanced academic, research, laboratory, or industry expertise... ...scientific work Prior AI training, evaluation, benchmarking, or LLM experience is a plus Flexible,...Contract workImmediate startRemote workFlexible hours
$40 - $125 per hour
...quarters actually happened. Someone has to make the numbers and the narrative agree, and that someone is you. As an Equity Research Analyst you'll evaluate AI-built company models and research notes, check the estimates and the write-up, and write the judgments that teach...Weekly payFull timeContract workFor contractorsRemote workFlexible hours- ...treasury futures statistical rigor factor modeling alpha research transaction cost analysis liquidity assessment model... ...communication About micro1 micro1 is a data lab that helps AI labs & Enterprises train foundational models and agents. We provide...Contract workPart timeFor contractorsRemote work
- ...and next steps. Our partner is looking for a Blockchain Security Researcher based in Canada. Join a globally distributed security research team... ...independently and alongside other researchers, you will leverage AI-powered tools to expand audit coverage, streamline workflows, and...Full timeContract workRemote workWorldwide
- ...builds the expert evaluation data that frontier AI labs use to measure and improve their models. We're hiring quantitative researchers and quant analysts as contractors to author... ...run on, drawn from your own work: signal research, backtesting, derivatives pricing, risk models...Contract workFor contractorsRemote work
$35 - $45 per hour
...SpaceXAI’s mission is to create AI systems that can accurately understand the universe and aid humanity in its pursuit of knowledge.... ...experience in voice, linguistics, speech data, or speech evaluation and research are especially encouraged to apply. LOCATION AND OTHER...Full timePart timeFor contractorsRemote workWorldwide10 hours per week$65 per hour
...Pay: up to $65/hour In this hourly, remote contractor role, you will work as a Market Research Subject Matter Expert (SME) to review AI-generated market research outputs and/or create expert research deliverables, evaluating reasoning quality and step-by-step analytical...Hourly payContract workFor contractorsImmediate startRemote work- ...panel of chemistry and chemical safety experts to red-team frontier AI models. The goal is to test whether a model can correctly judge... ...(ideally with red-teaming experience): Chemical defence research at USAMRICD, DoD, CDC or an allied national institute Toxicology...Part time
- ...Psychology researchers to build the behavioral science scenarios frontier AI labs use to test model judgment. AfterQuery builds the expert evaluation data that frontier... ...through interpretation, the kind a working researcher actually reasons through. Write reference answers...Weekly payOngoing contractContract workFor contractorsRemote work10 hours per week
- ...Us We’re a fast-growing product company integrating cutting-edge AI capabilities into our core offering to stay competitive and... ...autonomy. Work on practical, real-world AI applications — not just research. Opportunity to shape our AI architecture and best practices from...
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to AI Benchmark Researcher. Be the first to apply!
