Senior Software Engineer - Ruby (LLM Evaluation & Repository Validation)
Turing
Analyze GitHub issues and evaluate LLM performance in software projects.
Job Title: Senior Software Engineer – LLM Evaluation & Repository Validation
About the projects: we are building LLM evaluation and training datasets to train LLM to work on realistic software engineering problems. One of our approaches, in this project, is to build verifiable SWE tasks based on public repository histories in a synthetic approach with human-in-the-loop ; while expanding the dataset coverage to different types of tasks in terms of programming language, difficulty level, and etc.
About the Role: We are looking for experienced software engineers (tech lead level) who are familiar with high-quality public GitHub repositories and can contribute to this project. This role involves hands-on software engineering work, including development environment automation, issue triaging, and evaluating test coverage and quality
Why Join Us? Turing is one of the world’s fastest-growing AI companies accelerating the advancement and deployment of powerful AI systems. You’ll be at the forefront of evaluating how LLMs interact with real code, influencing the future of AI-assisted software development. This is a unique opportunity to blend practical software engineering with AI research.
What does day-to-day look like :
- Analyze and triage GitHub issues across trending open-source libraries.
- Set up and configure code repositories, including Dockerization and environment setup.
- Evaluating unit test coverage and quality.
- Modify and run codebases locally to assess LLM performance in bug-fixing scenarios.
- Collaborate with researchers to design and identify repositories and issues that are challenging for LLMs.
- Opportunities to lead a team of junior engineers to collaborate on projects.
Required Skills:
- Minimum 3+ years of overall experience
- Strong experience with at least one of the following languages: Ruby
- Proficiency with Git, Docker, and basic software pipeline setup.
- Ability to understand and navigate complex codebases.
- Comfortable running, modifying, and testing real-world projects locally.
- Experience contributing to or evaluating open-source projects is a plus.
Nice to Have:
- Previous participation in LLM research or evaluation projects.
- Experience building or testing developer tools or automation agents.
Perks of Freelancing With Turing:
- Work in a fully remote environment.
- Opportunity to work on cutting-edge AI projects with leading LLM companies.
Offer Details:
- Commitments Required : At least 4 hours per day and minimum 20 hours per week with overlap of 4 hours with PST. (We have 3 options of time commitment: 20 hrs/week, 30 hrs/week or 40 hrs/week)
- Employment type : Contractor assignment (no medical/paid leave)
- Location : India, Pakistan, Nigeria, Kenya, Egypt, Ghana, Bangladesh, Turkey, Mexico
Evaluation Process (approximately 75 mins) :
- Two rounds of interviews (60 min technical + 30 min technical & cultural discussion)
Skills: Ruby
- ...About the Role We are looking for software engineers to validate real-world repositories used to evaluate large language models. You will triage issues, set up reproducible environments, assess test quality, and check how well models fix real bugs. Key Responsibilities...RubyHourly payFor contractorsRemote work
- ...Evaluate and refine AI-generated code for efficiency and reliability. About Turing... ...on the P&L Role Overview As a Software Engineering evaluator, you will create cutting-edge... ...communication skills for clear, structured evaluation rationales. Offer Details...SeniorFull timePart timeFor contractorsFlexible hours
- ...GitLab to ship better, more secure software faster. The same principles... ...As an Intermediate Backend Engineer, you will take backend... ...minimal guidance, working in Ruby/Rails, Go, or any other modern... ...AI productively in daily work, validate what it produces, and understand...RubyFull timeContract workRemote work
$90k - $120k per year
...steps. Our partner is looking for an Experienced Intermediate Software Engineer, Ruby on Rails based in Canada . This role offers the opportunity... ..., and reliability. You will collaborate closely with another senior engineer and the product lead in a small, hands-on environment...RubySeniorFull timeWork at officeRemote work$250k - $280k per year
...conversations so they can debug them. We're bringing LLM and agent observability together, and we... ...in a small, cross-functional group of engineers with a dedicated product manager and... ...What you’ll bring to the role: - Significant software engineering experience, with a record of...SeniorFull timeRemote workShift work- ...budgeting to ensure clarity and real-world relevance. Response Evaluation: Assessing AI-generated travel guides, recommendations, and... ...Technical Writing Duration Continuous / Fast-Moving New to LLM Training?These are projects where you use your technical expertise...Contract workFreelanceFlexible hours
- ...Activities As an Entertainment & Pop Culture Evaluator, you will help refine AI models to ensure... ...domain relevance. Content & Response Evaluation: Assessing AI-generated summaries, trivia... ...Continuous / Fast-Moving New to LLM Training?These are projects where you use...Contract workFreelanceFlexible hours
- ...Main Activities As a Health & Fitness Specialist in LLM Training, you will evaluate AI-generated health, wellness, and exercise content to ensure... ...health to confirm clarity and domain alignment. Response Evaluation: Assessing AI-generated fitness advice, workout programs,...Contract workFreelanceFlexible hours
- ...Activities As a Learning Specialist in LLM Training, you will help refine AI models... ...study guides, and educational scenarios to evaluate clarity, target audience alignment, and... ...instructional relevance. Response & Content Evaluation: Assessing AI-generated explanations,...Contract workFreelanceFlexible hours
$55 per hour
...hour As an hourly paid, fully remote Ruby Engineer for AI Data Training, you will review AI... ...Rails code or generate your own solutions, evaluate the reasoning quality and step-by-step... ...for potential inaccuracies or biases, validating their reliability across use cases. Requirements...RubyHourly payContract workImmediate startRemote work$148k - $180k per year
...About the Team Netlify’s engineering teams build the products, platforms... ...We’re hiring engineers at the senior and staff levels across... ...who care about building great software, sharing knowledge, and helping... ...TypeScript/JavaScript, React, Ruby, and Go) to ship reliable, scalable...RubySeniorFull timeInternshipRemote workWorldwide$80 - $120 per hour
...foundational models and agents. We provide frontier evaluations and reinforcement learning environments to improve LLM capabilities, as well as contextual evaluations... ...both quality and velocity. Job Title: Senior Software Engineer Job Type: Contractor (~15 hrs a week...SeniorPart timeFor contractorsRemote work$90 per hour
...Pay: Up to $90+/hr Experienced software engineers review and score AI-written code for frontier AI research teams. Remote and flexible, up to 40 hours a week. Pay is up to $90+ per hour for the most experienced engineers. Requirements: several years of professional...SeniorHourly payContract workFor contractorsLive InRemote workFlexible hours- ...Cloudbeds , we're not just building software, we're transforming hospitality. Our platform... ...How You'll Make an Impact: As a Senior Software Engineer on the Core AI Squad, you'll design,... ...AI systems, agentic workflows, evaluation, and observability - not an internal tooling...SeniorFull timeWork at officeLocal areaRemote workWork from homeHome officeFlexible hours
$101k - $132.5k per year
...Summary of the Role and Team As a Senior Software Developer in Test (Senior SDET), you will... ...within a cross-functional team of Software Engineers, QA professionals, and Product Managers... ...unit, integration, API, UI, and data-validation testing layers ~ Design, build, and...SeniorLong term contractFull timeLocal areaRemote workWorldwide$50 - $100 per hour
...and agents. We provide frontier evaluations and reinforcement learning environments to improve LLM capabilities, as well as... ...challenging coding problems from real repositories, with reproducible environments and verifiers Role Title: Software Engineer Role Type:...Part timeFor contractorsRemote work- ...You’ll be at the forefront of evaluating how LLMs interact with real... ...influencing the future of AI-assisted software development. This is a unique... ...to blend practical software engineering with AI research. About... ...AI projects with leading LLM companies. Offer Details:...SeniorContract workPart timeFor contractorsFreelanceRemote work
$142.8k - $184.8k per year
...$2B ARR is that we need great engineers everywhere, and we have the room... ...at the center of hardware, software, AI, and the physical world. The... ...flexible: we're hiring from Senior through Staff+, across platform... ...in production, whether that's LLM-based products, agentic workflows...SeniorLong term contractFull timeInternshipRemote workRelocation packageFlexible hours- ...all applications and next steps. Our partner is looking for a Senior Software Engineer – AI based in Canada. The Senior Software Engineer will join... ...organization develop and deploy AI agents effectively. - Evaluate whether a problem is best addressed through an AI agent or a...SeniorLong term contractFull time
- ...Lead a team to create and review software security training tasks. About Turing... ...development through high-quality training data, evaluations, and engineering talent. Role Overview We are... ..., Java, C/C++, C#, Go, Rust, Ruby, PHP, Kotlin, or Swift. ~ Strong understanding...RubySeniorContract workPart timeFor contractorsFreelanceRemote work
- ...That’s you. As our first AI Agent Evaluation Engineer, you define what quality means for every... ...building the automated evaluation systems (LLM-based judges, golden sets, regression... ...prompt, model, knowledge, and flow change is validated against before release. Coverage &...Full timeWork at officeRemote workHome officeFlexible hours
$168k - $177.5k per year
...Overview Instacart is seeking a backend engineer with demonstrated success building highly reliable, performant and scalable software to shape the platform powering real-world, real... ...in at least one backend language (we use Ruby, Python and Go; Ruby experience not required...RubySeniorPermanent employmentWork at officeRemote workWork from homeFlexible hoursusd139.2k - usd235.2k per year
...trust GitLab to ship better, more secure software faster. The same principles built... .... An overview of this role As a Senior Backend Engineer on GitLab's Security Factory: Code... ...and apply the tooling that develops, evaluates, and ships those analyzers, and you measure...RubySeniorFull timeLive InRemote work- ...production-ready builds of all the open source software engineers and AI agents rely on, Chainguard helps... ...product, and this team builds it. As a Senior Software Engineer in Container Product... ...than stopping at the edge of your own repository. Keep the systems you touch healthy:...SeniorFull timeLocal areaRemote workRelocationFlexible hoursShift work
- ...more at fleetworthy.com. About the role: We’re hiring a Senior Software Engineer for the AI Platform team. The team builds AI agents and products... ...with Python and/or TypeScript Hands-on experience with LLM APIs, MCP and agent frameworks, including when not to use one...SeniorLong term contractFull time
$142.8k - $184.8k per year
...deployed in vehicles (think: GPS, Engine RPM, Fuel Levels and hundreds... ...that run our world: The software, firmware, and hardware you build... ...high quality releases. Evaluates how decisions at each layer of... ..., a fraud detection tool, to validate the authenticity of applications...SeniorLong term contractFull timeInternshipRemote workRelocation packageFlexible hours$170k - $230k per year
...all applications and next steps. Our partner is looking for a Senior Software Engineer, Database based in Canada. Join a fully remote, globally... ...tools as part of everyday development, with the ability to evaluate, validate, critique, and improve AI-generated output. - Comfortable...SeniorFull timeRemote work- ...applications and next steps. Our partner is looking for a Senior Staff Software Engineer – Builder Platform based in Canada. As a Senior Staff Software... ...shared Agent Platform, enabling teams to deploy, operate, evaluate, govern, and continuously improve AI agents and automated...SeniorLong term contractFull time
$149.6k - $199.6k per year
...daily at Coalition. About the role We are looking for a Senior Software Engineer to provide technical leadership on Coalition’s Scanning... ...architecture and roadmap decisions for the scanning platform, evaluating tradeoffs and proposing solutions that balance speed,...SeniorFull timeFixed term contractWork at officeRemote workHome officeFlexible hours- ...A little about us…Fleetio is a modern software platform that helps thousands of organizations... ...Fleetio is seeking an experienced Senior Software Engineer to join our Developer Enablement team.... ...prototypes and spikes as necessary to validate assumptions and technical feasibility....RubySeniorLong term contractFull timeContract workTemporary workRemote workWorldwide
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Senior Software Engineer - Ruby (LLM Evaluation & Repository Validation). Be the first to apply!
- software engineer part time Remote
- software engineer Remote
- software design engineer Remote
- software engineer - computer Remote
- software developer internship no experience Remote
- software engineer co-op Remote
- software development engineer Remote
- software developer Remote
- software engineer - ai machine learning Remote
- director software engineering Remote
