Engineering Manager - LLM Evaluation
Part-time
Turing
Lead teams to create and evaluate realistic LLM conversational data.
About Turing
Turing is a leading AI company accelerating the advancement and deployment of frontier AI systems. We work with the world’s top AI labs and enterprises to improve capabilities in reasoning, coding, agentic behavior, multimodality, and other advanced AI domains.
Role Overview
We are looking for an Engineering Manager to lead teams creating high-quality conversational data for training and evaluating advanced Large Language Models (LLMs). The role involves managing the design of realistic, multi-turn interactions between users and AI assistants, including tool-based workflows involving calendars, email, maps, cloud storage, and other applications.
The manager will oversee project execution, maintain quality standards, guide technical decision-making, and collaborate with engineering, operations, and review teams to deliver reliable training and evaluation datasets.
What You’ll Do
- Lead and manage teams responsible for creating and evaluating multi-turn LLM conversations.
- Define workflows, priorities, quality standards, and delivery goals for conversation data projects.
- Guide the design of realistic assistant interactions involving APIs, JSON, function calls, and external tools.
- Review complex scenarios involving decision-making, clarification, error handling, constraints, and task completion.
- Ensure conversations accurately reflect realistic user expectations and assistant capabilities.
- Establish quality-control processes and monitor team performance, consistency, and deliverable quality.
- Coach team members on technical reasoning, conversation design, tool usage, and adherence to project guidelines.
- Collaborate with engineering, product, operations, and quality teams to resolve issues and improve workflows.
- Analyze reviewer feedback and implement process improvements to increase realism, accuracy, and training value.
- Manage project risks, timelines, escalations, and stakeholder communication.
What We’re Looking For
- 5+ years of professional experience in software engineering, technical operations, data, AI, or a related technical field.
- Prior experience managing, mentoring, or leading technical teams.
- Strong understanding of APIs, JSON, data formats, logical reasoning, and structured workflows.
- Ability to guide teams in breaking down complex tasks into realistic conversational and tool-use flows.
- Strong understanding of assistant behavior, including tool selection, clarification, limitations, and error handling.
- Excellent written and verbal English communication skills.
- Strong attention to detail and ability to establish and enforce quality standards.
- Proven ability to manage multiple priorities, provide actionable feedback, and deliver projects within deadlines.
- Ability to collaborate effectively with cross-functional teams and external stakeholders.
Perks of Freelancing With Turing
- Work in a fully remote environment.
- Opportunity to work on cutting-edge AI projects with leading LLM companies.
Offer Details
- Commitments Required: At least 4 hours per day and minimum 40 hours per week with overlap of 4 hours with PST.
- Duration of Contract: 3 months (adjustable based on engagement)
Skills: Software Development, API Integrations , JSON
Vacancy posted 12 days ago
Similar jobs that could be interesting for youBased on the Engineering Manager - LLM Evaluation in Remote vacancy
- ...Evaluate and refine AI-generated code for efficiency and reliability. About Turing... ...P&L Role Overview As a Software Engineering evaluator, you will create cutting-edge... ...communication skills for clear, structured evaluation rationales. Offer Details Commitment...SuggestedFull timePart timeFor contractorsFlexible hours
- ...We are looking for software engineers to validate real-world repositories used to evaluate large language models. You will... ...projects. Prepare repositories for evaluation, including containerisation and... ...Preferred Qualifications Prior LLM research or evaluation projects....SuggestedHourly payFor contractorsRemote work
- ...accelerate model development through high-quality training data, evaluations, and engineering talent. About the Role We are looking for senior... ...and engineering teams to design coding benchmarks and improve LLM evaluation methodologies. Required Qualifications: ~7+...SuggestedContract workPart timeFor contractors
- ...Analyze GitHub issues and evaluate LLM performance on real code. Job Title: Senior Software Engineer – LLM Evaluation & Repository Validation About the projects: we are building LLM evaluation and training datasets to train LLM to work on realistic software engineering...SuggestedContract workPart timeFor contractorsFreelanceManual laborRemote work
$70 - $85 per hour
...candidates We are building and evaluating state-of-the-art large language models... ...are looking for experienced software engineers to join our evaluation and annotation team. This role sits at... ...-Bench-like problems). Evaluate LLM outputs for code generation, refactoring...SuggestedLong term contractContract workPart timeRemote work$75 - $90 per hour
...21 more Help a top AI lab evaluate and improve large language models... ...tasks. Bring your software-engineering judgment and hands-on... ...configure coding environments for evaluation and reinforcement learning.... ...code review, coding annotation, LLM/code evaluation or benchmark...Long term contractContract workRemote work- ...Evaluate AI responses for accuracy and improve model performance.... ...seeking highly motivated Team Management to support the evaluation and... ...challenging prompts that evaluate an LLM's ability to retrieve,... ...Familiarity with prompt engineering, AI evaluation, data annotation...Contract workPart timeFor contractorsRemote workFlexible hours
- ...Create and evaluate prompts for AI models in Vietnamese. About... ...experienced professionals with strong LLM expertise, analytical skills,... ...support the development and evaluation of advanced AI systems.... ...Models (LLMs) Prompt Engineering AI Training or Evaluation...Part timeRemote workFlexible hours
$150 per hour
...Evaluate security tools for vulnerabilities in GenAI applications.... ...$150/hr depending on internal evaluation and location. About the... ...looking for experienced security engineers to evaluate how effectively static... ...Hands-on experience building LLM agents or RAG systems using...Part timeFor contractors- ...dialogues, is looking for Go develoeprs. The engineer will be working together on the definition... ...the development cycle, effectively manage project priorities, and specify goals and... ...have used Github in the past. Train LLM models with high-quality, stable, and scalable...Contract workPart timeFlexible hours
- ...seeking talented developers proficient Java Engineer. The engineer will be working together on... ...and optimize AI models. Conduct evaluations (Evals) to benchmark model performance and... ...processes. Required Skills: Train LLM models with high-quality, stable, and scalable...Contract workPart timeFor contractorsFreelanceRemote work
- ...from the ground up. We're seeking MLOps Engineers with hands-on experience in large language... ...This role involves AI model training and evaluation work, including writing and assessing MLOps... ...models at scale (vLLM, SGLang, TensorRT-LLM, Ray Serve, KV cache, paged attention,...Full timeContract workPart timeWeekday work
usd15 per hour
...to influence the future of global AI search? If so, we want you on board! Working on behalf of RWS, you will test and evaluate how our clients’ search engine responds to everyday searches from everyday people. Using your search expertise, you will work through a variety of...Weekly payFull timePart timeFreelanceSecond jobRemote workWork from homeHome officeFlexible hours- ...Please state your availability clearly in your resume (Start date, End date). Responsibilities Research and develop state-of-the-art LLM and/or video or multimodality models to accelerate open-source progress in the domains. Implement the application of large models in...Permanent employmentFull timeInternshipImmediate startRemote workWorldwide
- ...to deliver high-quality datasets for LLM training. About Turing: Based... ...foundational LLMs. This position requires managing large teams of software engineers and data scientists dedicated to... ...: India & LATAM countries Evaluation Process (approximately 120 mins) :...Contract workPart timeFor contractorsManual labor
- ...We are hiring skilled software engineers with strong full-stack experience. The ideal candidate is comfortable architecting and working across multi-file codebases, structuring realistic development environments, and building reproducible setups supported by clear documentation...Contract work
- ...data lab for training frontier models and evaluating AI agents. Experts contribute their... ...domains such as finance, healthcare, STEM engineering, and more. micro1 transforms that real-world... ...into high-quality training data, evaluations, and feedback loops that improve how AI...Part timeFor contractorsRemote work
$55 - $100 per hour
Pay: $55–100/hour You'll systematically probe large language models to find inputs that cause them to bypass safety guidelines and produce harmful, disallowed, or policy-violating outputs. Your work involves crafting adversarial prompts across defined harm categories — ...Contract work$60 - $100 per hour
...$100/hour Commitment: Hourly, ~20 h/week Field: Other Engineering About the Role Mercor is partnering with a leading AI lab... ...Graphics, or a related field Prior AI training, benchmark, or evaluation work, or heavy daily use of AI coding agents Role Highlights...Hourly payContract workFreelanceImmediate start- ...performance Who Should Apply Current or former experience as a management consultant or business analyst Familiarity with standard... ...communication skills Comfortable working independently on text evaluation tasks Contract & Payment Terms You will be engaged as an...Contract workFor contractorsRemote work
- ...labs and enterprises. Our environments evaluate and improve AI models on complex, long-range... .... Our work includes: Software engineering environments for coding agents UI environments... ...innovations in data, algorithms, evaluation, and training. This role sits at the...Contract workPart timeFor contractors
- ...This position is within a project with one of the foundational LLM companies. The goal is to assist these foundational LLM companies... ...first, as a basis for fine-tuning their models, and second, as an evaluation set to benchmark the performance of their models or competitor models...Contract workPart timeFor contractorsFreelanceRemote work
- ...high-quality outputs in AI data evaluation projects. About Turing:... ...are looking for a practicing engineer with strong technical judgment... ...edge AI projects with leading LLM companies. Offer Details:... ...of interview (60 min technical ) Skills: QA, Quality Management...Contract workPart timeFor contractorsFreelanceRemote work
- ...global community of experts to evaluate, challenge and improve... ...LLMs, Mathematics, Science, Engineering, Linguistics, Cybersecurity,... ...and computational outputs LLM & NLP AI-generated text and... ...behaviour Domain-Specific AI Evaluation Healthcare & Medicine Law...Part time
- ...training frontier models and evaluating AI agents. Experts contribute... ...as finance, healthcare, STEM engineering, and more. micro1 transforms... ...high-quality training data, evaluations, and feedback loops that improve... ...to work independently, manage multiple assignments, and meet...Part timeFor contractorsRemote work
- ...We are seeking a Dataset Enablement Manager to bridge the technical gap between scientific... ...in technical enablement, pre-sales engineering, technical product marketing, or applied... ...benchmarks, algorithmic problem-solving, and LLM evaluation and reasoning mechanics....Part timeFor contractors
- ...programming tasks for accuracy and difficulty Verify the logic and test cases within provided evaluation harnesses Assess whether coding environments effectively measure software engineering skills Walk us through your thought process while analyzing complex code structures...Contract workFor contractorsRemote work
- ...Provide technical feedback on the structure and difficulty of evaluation harnesses Walk us through your approach to verifying complex code... ...more robust Who Should Apply Professional software engineer based in South Asia Experience building or reviewing realistic...Contract workFor contractorsRemote work
- ...Engineering Manager, DevEx Ilir Sela started Slice with the belief that local pizzerias deserve every advantage available to major franchises—without... ...applications, including agentic applications/services using LLM APIs Build a highly effective team Give platform engineers...Full timeLocal area
- ...Apply your subject-matter expertise to advanced AI training and evaluation work Develop or evaluate challenging problems within your area... ...scientific work Prior AI training, evaluation, benchmarking, or LLM experience is a plus Flexible, remote contract work starting...Contract workImmediate startRemote workFlexible hours
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Engineering Manager - LLM Evaluation. Be the first to apply!
