Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Senior Software Engineer - Ruby (LLM Evaluation & Repository Validation)

Part-time

Turing

Analyze GitHub issues and evaluate LLM performance in software projects.

Job Title: Senior Software Engineer – LLM Evaluation & Repository Validation

About the projects: we are building LLM evaluation and training datasets to train LLM to work on realistic software engineering problems. One of our approaches, in this project, is to build verifiable SWE tasks based on public repository histories in a synthetic approach with human-in-the-loop ; while expanding the dataset coverage to different types of tasks in terms of programming language, difficulty level, and etc.

About the Role: We are looking for experienced software engineers (tech lead level) who are familiar with high-quality public GitHub repositories and can contribute to this project. This role involves hands-on software engineering work, including development environment automation, issue triaging, and evaluating test coverage and quality

Why Join Us? Turing is one of the world’s fastest-growing AI companies accelerating the advancement and deployment of powerful AI systems. You’ll be at the forefront of evaluating how LLMs interact with real code, influencing the future of AI-assisted software development. This is a unique opportunity to blend practical software engineering with AI research.

What does day-to-day look like :

  • Analyze and triage GitHub issues across trending open-source libraries.
  • Set up and configure code repositories, including Dockerization and environment setup.
  • Evaluating unit test coverage and quality.
  • Modify and run codebases locally to assess LLM performance in bug-fixing scenarios.
  • Collaborate with researchers to design and identify repositories and issues that are challenging for LLMs.
  • Opportunities to lead a team of junior engineers to collaborate on projects.

Required Skills:

  • Minimum 3+ years of overall experience
  • Strong experience with at least one of the following languages: Ruby
  • Proficiency with Git, Docker, and basic software pipeline setup.
  • Ability to understand and navigate complex codebases.
  • Comfortable running, modifying, and testing real-world projects locally.
  • Experience contributing to or evaluating open-source projects is a plus.

Nice to Have:

  • Previous participation in LLM research or evaluation projects.
  • Experience building or testing developer tools or automation agents.

Perks of Freelancing With Turing:

  • Work in a fully remote environment.
  • Opportunity to work on cutting-edge AI projects with leading LLM companies.

Offer Details:

  • Commitments Required : At least 4 hours per day and minimum 20 hours per week with overlap of 4 hours with PST. (We have 3 options of time commitment: 20 hrs/week, 30 hrs/week or 40 hrs/week)
  • Employment type : Contractor assignment (no medical/paid leave)
  • Location : India, Pakistan, Nigeria, Kenya, Egypt, Ghana, Bangladesh, Turkey, Mexico

Evaluation Process (approximately 75 mins) :

  • Two rounds of interviews (60 min technical + 30 min technical & cultural discussion)

Skills: Ruby

Vacancy posted a month ago
Similar jobs that could be interesting for youBased on the Senior Software Engineer - Ruby (LLM Evaluation & Repository Validation) in Remote vacancy
  •  ...About the Role We are looking for software engineers to validate real-world repositories used to evaluate large language models. You will triage issues, set up reproducible environments, assess test quality, and check how well models fix real bugs. Key Responsibilities... 
    Ruby
    Hourly pay
    For contractors
    Remote work

    RemoExperts

    Remote
    a month ago
  •  ...Evaluate and refine AI-generated code for efficiency and reliability. About Turing...  ...on the P&L Role Overview As a Software Engineering evaluator, you will create cutting-edge...  ...communication skills for clear, structured evaluation rationales. Offer Details... 
    Senior
    Full time
    Part time
    For contractors
    Flexible hours

    Turing

    Remote
    a month ago
  •  ...GitLab to ship better, more secure software faster. The same principles...  ...As an Intermediate Backend Engineer, you will take backend...  ...minimal guidance, working in Ruby/Rails, Go, or any other modern...  ...AI productively in daily work, validate what it produces, and understand... 
    Ruby
    Full time
    Contract work
    Remote work

    GitLab

    Remote
    7 days ago
  • $90k - $120k per year

     ...steps. Our partner is looking for an Experienced Intermediate Software Engineer, Ruby on Rails based in Canada . This role offers the opportunity...  ..., and reliability. You will collaborate closely with another senior engineer and the product lead in a small, hands-on environment... 
    Ruby
    Senior
    Full time
    Work at office
    Remote work

    Jobgether

    Remote
    13 days ago
  • $250k - $280k per year

     ...conversations so they can debug them. We're bringing LLM and agent observability together, and we...  ...in a small, cross-functional group of engineers with a dedicated product manager and...  ...What you’ll bring to the role: - Significant software engineering experience, with a record of... 
    Senior
    Full time
    Remote work
    Shift work

    Honeycomb

    Remote
    a month ago
  •  ...budgeting to ensure clarity and real-world relevance. Response Evaluation: Assessing AI-generated travel guides, recommendations, and...  ...Technical Writing Duration Continuous / Fast-Moving New to LLM Training?These are projects where you use your technical expertise... 
    Contract work
    Freelance
    Flexible hours

    Revelo

    Remote
    10 days ago
  •  ...Activities As an Entertainment & Pop Culture Evaluator, you will help refine AI models to ensure...  ...domain relevance. Content & Response Evaluation: Assessing AI-generated summaries, trivia...  ...Continuous / Fast-Moving New to LLM Training?These are projects where you use... 
    Contract work
    Freelance
    Flexible hours

    Revelo

    Remote
    10 days ago
  •  ...Main Activities As a Health & Fitness Specialist in LLM Training, you will evaluate AI-generated health, wellness, and exercise content to ensure...  ...health to confirm clarity and domain alignment. Response Evaluation: Assessing AI-generated fitness advice, workout programs,... 
    Contract work
    Freelance
    Flexible hours

    Revelo

    Remote
    10 days ago
  •  ...Activities As a Learning Specialist in LLM Training, you will help refine AI models...  ...study guides, and educational scenarios to evaluate clarity, target audience alignment, and...  ...instructional relevance. Response & Content Evaluation: Assessing AI-generated explanations,... 
    Contract work
    Freelance
    Flexible hours

    Revelo

    Remote
    10 days ago
  • $55 per hour

     ...hour As an hourly paid, fully remote Ruby Engineer for AI Data Training, you will review AI...  ...Rails code or generate your own solutions, evaluate the reasoning quality and step-by-step...  ...for potential inaccuracies or biases, validating their reliability across use cases. Requirements... 
    Ruby
    Hourly pay
    Contract work
    Immediate start
    Remote work

    SuperAnnotate

    Remote
    a month ago
  • $148k - $180k per year

     ...About the Team Netlify’s engineering teams build the products, platforms...  ...We’re hiring engineers at the senior and staff levels across...  ...who care about building great software, sharing knowledge, and helping...  ...TypeScript/JavaScript, React, Ruby, and Go) to ship reliable, scalable... 
    Ruby
    Senior
    Full time
    Internship
    Remote work
    Worldwide

    Netlify

    Remote
    3 hours ago
  • $80 - $120 per hour

     ...foundational models and agents. We provide frontier evaluations and reinforcement learning environments to improve LLM capabilities, as well as contextual evaluations...  ...both quality and velocity. Job Title: Senior Software Engineer Job Type: Contractor (~15 hrs a week... 
    Senior
    Part time
    For contractors
    Remote work

    micro1

    Remote
    8 days ago
  • $90 per hour

     ...Pay: Up to $90+/hr Experienced software engineers review and score AI-written code for frontier AI research teams. Remote and flexible, up to 40 hours a week. Pay is up to $90+ per hour for the most experienced engineers. Requirements: several years of professional... 
    Senior
    Hourly pay
    Contract work
    For contractors
    Live In
    Remote work
    Flexible hours

    rustLabs

    Remote
    17 days ago
  •  ...Cloudbeds , we're not just building software, we're transforming hospitality. Our platform...  ...How You'll Make an Impact: As a Senior Software Engineer on the Core AI Squad, you'll design,...  ...AI systems, agentic workflows, evaluation, and observability - not an internal tooling... 
    Senior
    Full time
    Work at office
    Local area
    Remote work
    Work from home
    Home office
    Flexible hours

    Cloudbeds

    Remote
    10 days ago
  • $101k - $132.5k per year

     ...Summary of the Role and Team As a Senior Software Developer in Test (Senior SDET), you will...  ...within a cross-functional team of Software Engineers, QA professionals, and Product Managers...  ...unit, integration, API, UI, and data-validation testing layers ~ Design, build, and... 
    Senior
    Long term contract
    Full time
    Local area
    Remote work
    Worldwide

    Varicent

    Remote
    11 days ago
  • $50 - $100 per hour

     ...and agents. We provide frontier evaluations and reinforcement learning environments to improve LLM capabilities, as well as...  ...challenging coding problems from real repositories, with reproducible environments and verifiers Role Title: Software Engineer Role Type:... 
    Part time
    For contractors
    Remote work

    micro1

    Remote
    5 days ago
  •  ...You’ll be at the forefront of evaluating how LLMs interact with real...  ...influencing the future of AI-assisted software development. This is a unique...  ...to blend practical software engineering with AI research. About...  ...AI projects with leading LLM companies. Offer Details:... 
    Senior
    Contract work
    Part time
    For contractors
    Freelance
    Remote work

    Turing

    Remote
    5 days ago
  • $142.8k - $184.8k per year

     ...$2B ARR is that we need great engineers everywhere, and we have the room...  ...at the center of hardware, software, AI, and the physical world. The...  ...flexible: we're hiring from Senior through Staff+, across platform...  ...in production, whether that's LLM-based products, agentic workflows... 
    Senior
    Long term contract
    Full time
    Internship
    Remote work
    Relocation package
    Flexible hours

    Samsara

    Remote
    7 days ago
  •  ...all applications and next steps. Our partner is looking for a Senior Software Engineer – AI based in Canada. The Senior Software Engineer will join...  ...organization develop and deploy AI agents effectively. - Evaluate whether a problem is best addressed through an AI agent or a... 
    Senior
    Long term contract
    Full time

    Jobgether

    Remote
    5 days ago
  •  ...Lead a team to create and review software security training tasks. About Turing...  ...development through high-quality training data, evaluations, and engineering talent. Role Overview We are...  ..., Java, C/C++, C#, Go, Rust, Ruby, PHP, Kotlin, or Swift. ~ Strong understanding... 
    Ruby
    Senior
    Contract work
    Part time
    For contractors
    Freelance
    Remote work

    Turing

    Remote
    3 days ago
  •  ...That’s you.   As our first AI Agent Evaluation Engineer, you define what quality means for every...  ...building the automated evaluation systems (LLM-based judges, golden sets, regression...  ...prompt, model, knowledge, and flow change is validated against before release. Coverage &... 
    Full time
    Work at office
    Remote work
    Home office
    Flexible hours

    Cresta

    Remote
    3 days ago
  • $168k - $177.5k per year

     ...Overview Instacart is seeking a backend engineer with demonstrated success building highly reliable, performant and scalable software to shape the platform powering real-world, real...  ...in at least one backend language (we use Ruby, Python and Go; Ruby experience not required... 
    Ruby
    Senior
    Permanent employment
    Work at office
    Remote work
    Work from home
    Flexible hours

    Instacart

    Remote
    7 days ago
  • usd139.2k - usd235.2k per year

     ...trust GitLab to ship better, more secure software faster. The same principles built...  .... An overview of this role As a Senior Backend Engineer on GitLab's Security Factory: Code...  ...and apply the tooling that develops, evaluates, and ships those analyzers, and you measure... 
    Ruby
    Senior
    Full time
    Live In
    Remote work

    GitLab

    Remote
    7 days ago
  •  ...production-ready builds of all the open source software engineers and AI agents rely on, Chainguard helps...  ...product, and this team builds it. As a Senior Software Engineer in Container Product...  ...than stopping at the edge of your own repository. Keep the systems you touch healthy:... 
    Senior
    Full time
    Local area
    Remote work
    Relocation
    Flexible hours
    Shift work

    Chainguard

    Remote
    9 days ago
  •  ...more at fleetworthy.com. About the role: We’re hiring a Senior Software Engineer for the AI Platform team. The team builds AI agents and products...  ...with Python and/or TypeScript Hands-on experience with LLM APIs, MCP and agent frameworks, including when not to use one... 
    Senior
    Long term contract
    Full time

    Bestpass by Fleetworthy

    Remote
    5 days ago
  • $142.8k - $184.8k per year

     ...deployed in vehicles (think: GPS, Engine RPM, Fuel Levels and hundreds...  ...that run our world: The software, firmware, and hardware you build...  ...high quality releases. Evaluates how decisions at each layer of...  ..., a fraud detection tool, to validate the authenticity of applications... 
    Senior
    Long term contract
    Full time
    Internship
    Remote work
    Relocation package
    Flexible hours

    Samsara

    Remote
    7 days ago
  • $170k - $230k per year

     ...all applications and next steps. Our partner is looking for a Senior Software Engineer, Database based in Canada. Join a fully remote, globally...  ...tools as part of everyday development, with the ability to evaluate, validate, critique, and improve AI-generated output. - Comfortable... 
    Senior
    Full time
    Remote work

    Jobgether

    Remote
    7 days ago
  •  ...applications and next steps. Our partner is looking for a Senior Staff Software Engineer – Builder Platform based in Canada. As a Senior Staff Software...  ...shared Agent Platform, enabling teams to deploy, operate, evaluate, govern, and continuously improve AI agents and automated... 
    Senior
    Long term contract
    Full time

    Jobgether

    Remote
    12 days ago
  • $149.6k - $199.6k per year

     ...daily at Coalition. About the role We are looking for a Senior Software Engineer to provide technical leadership on Coalition’s Scanning...  ...architecture and roadmap decisions for the scanning platform, evaluating tradeoffs and proposing solutions that balance speed,... 
    Senior
    Full time
    Fixed term contract
    Work at office
    Remote work
    Home office
    Flexible hours

    Coalition, Inc.

    Remote
    13 days ago
  •  ...A little about us…Fleetio is a modern software platform that helps thousands of organizations...  ...Fleetio is seeking an experienced Senior Software Engineer to join our Developer Enablement team....  ...prototypes and spikes as necessary to validate assumptions and technical feasibility.... 
    Ruby
    Senior
    Long term contract
    Full time
    Contract work
    Temporary work
    Remote work
    Worldwide

    Fleetio

    Remote
    11 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Senior Software Engineer - Ruby (LLM Evaluation & Repository Validation). Be the first to apply!