Software Engineer- Inference Performance
$180k - $360k per yearBaseten
About Baseten
Baseten powers mission-critical inference for the world's most dynamic AI companies, like Cursor, Notion, OpenEvidence, Abridge, Clay, Gamma, and Writer. By uniting applied AI research, flexible infrastructure, and seamless developer tooling, we enable companies operating at the frontier of AI to bring cutting-edge models into production. We're growing quickly and recently raised our $1.5B Series F, led by Altimeter Capital, Conviction Partners, and Spark Capital. Join us and help build the platform engineers turn to ship AI products.About Baseten
Baseten powers mission-critical inference for the world's most dynamic AI companies, like Cursor, Notion, OpenEvidence, Abridge, Clay, Gamma, and Writer. By uniting applied AI research, flexible infrastructure, and seamless developer tooling, we enable companies operating at the frontier of AI to bring cutting-edge models into production. We're growing quickly and recently raised our $1.5B Series F, led by Altimeter Capital, Conviction Partners, and Spark Capital. Join us and help build the platform engineers turn to ship AI products.THE ROLE
We're looking for inference performance engineers who want to make the world's most demanding AI workloads run faster and more efficiently. You'll work across the stack, from the inference engine and runtime through scheduling, serving, and routing. Along the way you'll apply techniques like prefill/decode disaggregation, speculative decoding, and KV-cache management. You'll reason from first principles about where time and memory go, find what's holding performance back, and close the gap. Your work directly impacts how fast our customers' models run and how efficiently we serve them. This role is ideal for someone who thrives in a fast-paced startup environment and is eager to make significant contributions to the exciting field of LLM inference.EXAMPLE INITIATIVES
Agentic Kernels in ProductionHow we built the new fastest API for GLM-5.2Live draft model training for speculative decodingMaking Kimi K3 Tokenization 18x fasterThe Baseten Inference StackDriving model performance optimizationResponsibilities
Implement and productionize cutting-edge inference techniques, working deep in runtime internals. That includes quantization, speculative decoding, KV-cache reuse, chunked prefill, LoRA, guided generation for structured outputs, and custom scheduling and routing algorithms.Profile and optimize inference end to end, from kernel launch overhead and memory layout up to request scheduling, prefill/decode disaggregation, and cache-aware routing. Run cross-layer investigations, such as tracing a tail-latency regression from request timing through routing and batching down to a kernel.Turn performance into cost savings. Improve tokens per GPU-hour, raise utilization, and give customers and internal teams clear latency/throughput/cost tradeoffs.Bring up and tune new model architectures on new hardware quickly, often in the same week they're released.Build benchmarking frameworks that measure real-world performance across model architectures, batch sizes, sequence lengths, and hardware configurations.Contribute upstream to open-source inference engines (vLLM, SGLang, TensorRT-LLM), and partner closely with model, infrastructure, and customer-facing teams to ship wins.Requirements
Bachelor's, Master's, or Ph.D. degree in Computer Science, Engineering, Mathematics, or related field.Experience with one or more general-purpose programming languages, such as Python or C++.Familiarity with LLM optimization techniques (e.G., quantization, speculative decoding, continuous batching).Strong familiarity with ML libraries, especially PyTorch, TensorRT, or TensorRT-LLM.Demonstrated interest and experience in LLMs.Deep understanding of GPU architecture.NICE TO HAVE
Proficiency in enhancing the performance of software systems, particularly in the context of large language models (LLMs)Contributed to vLLM, SGLang, TensorRT-LLM, or another inference engine.Worked on large-scale distributed serving: autoscaling, load balancing, multi-region or multi-cloud capacity.Written or optimized GPU kernels (CUDA, Triton, CUTLASS, or similar)Worked on quantization (FP8/FP4, AWQ, GPTQ) or speculative decoding in production.Deep understanding of software engineering principles and a proven track record of developing and deploying AI/ML inference solutions.Benefits
Competitive compensation, including meaningful equity(U.S. only) 100% coverage of medical, dental, and vision insurance for employee and dependentsFlexible PTO policy including company wide Winter Break (our offices are closed from Christmas Eve to New Year's Day!)Paid parental leaveFertility and family-building stipend through Carrot(U.S. only) Company-facilitated 401(k)Exposure to a variety of ML startups, offering unparalleled learning and networking opportunities.At Baseten, we are committed to fostering a diverse and inclusive workplace. We provide equal employment opportunities to all employees and applicants without regard to race, color, religion, gender, sexual orientation, gender identity or expression, national origin, age, genetic information, disability, or veteran status.We are an Equal Opportunity Employer and will consider qualified applicants with criminal histories in a manner consistent with applicable law (by example, the requirements of the San Francisco Fair Chance Ordinance, where applicable).Compensation Range: $180K - $360K
#J-18808-Ljbffr
Baseten powers mission-critical inference for the world's most dynamic AI companies, like Cursor, Notion, OpenEvidence, Abridge, Clay, Gamma, and Writer. By uniting applied AI research, flexible infrastructure, and seamless developer tooling, we enable companies operating at the frontier of AI to bring cutting-edge models into production. We're growing quickly and recently raised our $1.5B Series F, led by Altimeter Capital, Conviction Partners, and Spark Capital. Join us and help build the platform engineers turn to ship AI products.About Baseten
Baseten powers mission-critical inference for the world's most dynamic AI companies, like Cursor, Notion, OpenEvidence, Abridge, Clay, Gamma, and Writer. By uniting applied AI research, flexible infrastructure, and seamless developer tooling, we enable companies operating at the frontier of AI to bring cutting-edge models into production. We're growing quickly and recently raised our $1.5B Series F, led by Altimeter Capital, Conviction Partners, and Spark Capital. Join us and help build the platform engineers turn to ship AI products.THE ROLE
We're looking for inference performance engineers who want to make the world's most demanding AI workloads run faster and more efficiently. You'll work across the stack, from the inference engine and runtime through scheduling, serving, and routing. Along the way you'll apply techniques like prefill/decode disaggregation, speculative decoding, and KV-cache management. You'll reason from first principles about where time and memory go, find what's holding performance back, and close the gap. Your work directly impacts how fast our customers' models run and how efficiently we serve them. This role is ideal for someone who thrives in a fast-paced startup environment and is eager to make significant contributions to the exciting field of LLM inference.EXAMPLE INITIATIVES
Agentic Kernels in ProductionHow we built the new fastest API for GLM-5.2Live draft model training for speculative decodingMaking Kimi K3 Tokenization 18x fasterThe Baseten Inference StackDriving model performance optimizationResponsibilities
Implement and productionize cutting-edge inference techniques, working deep in runtime internals. That includes quantization, speculative decoding, KV-cache reuse, chunked prefill, LoRA, guided generation for structured outputs, and custom scheduling and routing algorithms.Profile and optimize inference end to end, from kernel launch overhead and memory layout up to request scheduling, prefill/decode disaggregation, and cache-aware routing. Run cross-layer investigations, such as tracing a tail-latency regression from request timing through routing and batching down to a kernel.Turn performance into cost savings. Improve tokens per GPU-hour, raise utilization, and give customers and internal teams clear latency/throughput/cost tradeoffs.Bring up and tune new model architectures on new hardware quickly, often in the same week they're released.Build benchmarking frameworks that measure real-world performance across model architectures, batch sizes, sequence lengths, and hardware configurations.Contribute upstream to open-source inference engines (vLLM, SGLang, TensorRT-LLM), and partner closely with model, infrastructure, and customer-facing teams to ship wins.Requirements
Bachelor's, Master's, or Ph.D. degree in Computer Science, Engineering, Mathematics, or related field.Experience with one or more general-purpose programming languages, such as Python or C++.Familiarity with LLM optimization techniques (e.G., quantization, speculative decoding, continuous batching).Strong familiarity with ML libraries, especially PyTorch, TensorRT, or TensorRT-LLM.Demonstrated interest and experience in LLMs.Deep understanding of GPU architecture.NICE TO HAVE
Proficiency in enhancing the performance of software systems, particularly in the context of large language models (LLMs)Contributed to vLLM, SGLang, TensorRT-LLM, or another inference engine.Worked on large-scale distributed serving: autoscaling, load balancing, multi-region or multi-cloud capacity.Written or optimized GPU kernels (CUDA, Triton, CUTLASS, or similar)Worked on quantization (FP8/FP4, AWQ, GPTQ) or speculative decoding in production.Deep understanding of software engineering principles and a proven track record of developing and deploying AI/ML inference solutions.Benefits
Competitive compensation, including meaningful equity(U.S. only) 100% coverage of medical, dental, and vision insurance for employee and dependentsFlexible PTO policy including company wide Winter Break (our offices are closed from Christmas Eve to New Year's Day!)Paid parental leaveFertility and family-building stipend through Carrot(U.S. only) Company-facilitated 401(k)Exposure to a variety of ML startups, offering unparalleled learning and networking opportunities.At Baseten, we are committed to fostering a diverse and inclusive workplace. We provide equal employment opportunities to all employees and applicants without regard to race, color, religion, gender, sexual orientation, gender identity or expression, national origin, age, genetic information, disability, or veteran status.We are an Equal Opportunity Employer and will consider qualified applicants with criminal histories in a manner consistent with applicable law (by example, the requirements of the San Francisco Fair Chance Ordinance, where applicable).Compensation Range: $180K - $360K
#J-18808-Ljbffr
Vacancy posted 1 hour ago
Similar jobs that could be interesting for youBased on the Software Engineer- Inference Performance in Winnipeg, MB vacancy
- Elevate AI workloads with Baseten as an Inference Performance Engineer. Focus on optimizing cutting-edge inference techniques while working in a dynamic, fast-paced environment. At Baseten, we're seeking an Inference Performance Engineer committed to improving AI model performance...Performance
$180k - $360k per year
...Baseten powers mission-critical inference for the world's most dynamic... ...and help build the platform engineers turn to ship AI products.... ...models with industry-leading performance, scalability, and reliability... ...contributions to LLM inference engines and frameworks such as vLLM,...PerformanceFlexible hours- ...deliver industry-leading training and inference speeds; over 10 times faster than... ...decode on the Cerebras Wafer-Scale Engine. We are hiring a Software Engineer to productionize and... ...correct, observable, and exceptionally performant. You will write production code,...Performance
$165k - $330k per year
About BasetenBaseten powers mission-critical inference for the world's most dynamic AI companies, like Cursor, Notion, OpenEvidence, Abridge... ..., and Spark Capital. Join us and help build the platform engineers turn to ship AI products. THE ROLEAs an Engineer on the Dedicated...SuggestedFlexible hours- ...customers. Cohere is a team of researchers, engineers, designers, and more, who are passionate... ...? Are you energized by building high-performance, scalable and reliable machine learning... ...influence latency and throughput of inference. ~ Strong understanding or working experience...PerformanceFull timeWork at officeRemote workFlexible hours
- ...group of committed researchers, engineers, policy experts, and business... .... About the roleOur Inference team is responsible for building... ...giving our scientists the high-performance inference infrastructure they... ...Significant software engineering experience, particularly...PerformanceLong term contractWork at officeWorldwideVisa sponsorshipFlexible hours
- ...deliver industry-leading training and inference speeds; over 10 times faster than... ...decode on the Cerebras Wafer-Scale Engine. We are hiring a Software Engineer to build and evolve the ML... ...lifecycle management. Improve serving performance. Optimize streaming behavior, time...Performance
- Transform the AI experience at Cerebras Systems as a Software Engineer on our Inference Platform team. Shape next-gen architecture while deploying ultra-fast machine learning services to boost performance and scalability. Cerebras Systems seeks a Software Engineer with 3+...Performance
- ...industry-leading training and inference speeds; over 10 times... ...the RoleWe're hiring a Staff Engineer to help lead, drive, and contribute... ..., and maintain production software, with responsibilities spanning... ...areas. Reliability & Performance. Architect active-active systems...Performance
- Cerebras Systems is hiring a Software Engineer to productionize and optimize our GPU serving stack. You will work across custom inference APIs, vLLM, PyTorch, ROCm, and rack-scale AMD GPU... ...correctness, observability, and high performance. You will write production-grade code,...Performance
- ...role is for a highly experienced Back-End Software Engineer who wants to work on core systems that... ...platform where reliability, security, performance, and simplicity all matter, and where... ...shelf solutionsModel Vault is Cohere’s inference platform for deploying and operating models...PerformanceRemote work
$165k - $330k per year
...Baseten Baseten powers mission-critical inference for the world's most dynamic AI... ...Capital. Join us and help build the platform engineers turn to ship AI products. Baseten powers... ...with Forward Deployed Engineers, Model Performance Engineers, and sister engineering teams...PerformanceLong term contractFlexible hours- ...Cerebras Systems builds the world's largest AI chip and is hiring a Software Engineer to build and evolve the ML API layer for a disaggregated inference system. You will work across inference APIs, model integration, request-routing services, and Cerebras runtimes to deliver...
$165k - $330k per year
...BasetenBaseten powers mission-critical inference for the world's most dynamic... ...and help build the platform engineers turn to ship AI products.... ...THE ROLEWe are looking for Software Engineers to join our team.... ...at the intersection of high-performance computing (HPC) and Large Language...PerformanceFlexible hours- ...support onboarding of large customers and high-traffic eventsAdvise engineering teams on scalable, efficient feature design; review pull... ...plan analysis, and schema design5+ years of experience as a software developer with strong knowledge of Java or a similar language,...PerformanceLong term contractWork from home
- ...deliver industry-leading training and inference speeds; over 10 times faster than... ...decode on the Cerebras Wafer-Scale Engine. We are hiring a Senior Software Engineer to build and evolve the ML... ..., integration tests, and performance benchmarks that catch regressions in...Performance
- Xanadu is seeking a developer to join their team in Toronto, focused on quantum computing. The role involves building a high-performance software stack and benchmarking tools for quantum systems. Candidates should have a BSc in a related field and experience with high-performance...Performance
$165k - $330k per year
...Baseten powers mission-critical inference for the world's most dynamic AI companies... ...Join us and help build the platform engineers turn to ship AI products.THE ROLE... ...and who enjoy working across product, software development, performance engineering, and customer-facing implementations...PerformanceInternshipFlexible hours- ...hybrid work model in Burnaby, BC, is seeking a Systems Software Engineer to push core systems and the shared engine forward. The role involves prototyping, planning,... ...and implement high-quality software, ensuring performance and memory budgets are respected while advancing EA...Performance
- Drive AI innovation with the MTIA Software team at Meta as a Software Engineer specializing in performance optimization for deep learning. This role focuses on frameworks and compiler development. Become part of the MTIA team's mission to develop advanced software for deep...Performance
- ...bringing them the tools and insight they need to deliver faster, more performant, more dependable, and less expensive Java services. Come join... ...on LLVM. We are looking for a Senior Compiler Performance Engineer to own major performance areas of Falcon. This is not a...PerformanceRemote workWorldwideFlexible hours
- ...industry-leading training and inference speeds; over 10 times... ...at massive scale. As an Engineering Manager, Inference Router, you... ...teams to optimize end‑to‑end performance. Team Leadership Build,... ...of experience in large‑scale software engineering, distributed systems...Performance
- ...Cohere is a team of researchers, engineers, designers, and more, who are... ...you want to architect high-performance, resilient machine learning... ...ambiguous, customer-specific inference challenges—ranging from multi... ...systems, or deploying enterprise software in complex environments....PerformanceFull timeWork at officeLocal areaRemote workHome office
$100k - $145k per year
Thomson Reuters is seeking a Senior Inference Engineer, AI. This person will collaborate with platform... ...Optimize LLMs and ML models for high-performance inference using techniques such as... ...Our products combine highly specialized software and insights to empower professionals with...PerformanceWork at officeLocal areaFlexible hours2 days per week3 days per week- Motive Studio in Montreal is seeking a Senior Core Software Engineer to craft high-quality C++ code for Battlefield titles. You will focus on performance, memory efficiency, and low-level platform work, collaborating with the Frostbite team and DICE to push technology forward...Performance
- Drive performance advancements in quantum computing at Xanadu as a Quantum Software Engineer. Your focus will be on optimizing software frameworks for quantum capabilities. In the PennyLane Performance Team, you will play an essential role in building a high-performance software...Performance
- ...Build AI Inference Networking software at Equinix using SDN, proxying, and open-source engineering expertise. Work within a globally distributed development team. As a Senior SDN Network Software Engineer, you will help develop open-source software for Equinix’s AI Inference...
- Cohere is looking for a Backend Engineer to specialize in optimizing model inference systems. In this remote-first role, you will create low-latency serving... ...enterprise API, focusing on batching strategies and performance optimization. Key Responsibilities: • Establish low...PerformanceRemote work
- Build AI inference systems with Python and PyTorch as a Backend Engineer, AI, powering production interactions through reliable, observable APIs.You will own the... ...boundaries around models. You will optimize real-world performance while supporting mobile and desktop clients.Key...Performance
- Senior AI Engineer, AI Inference Systems Join to apply for the Senior AI Engineer, AI Inference... ...the LLM inference engine to ensure performance, scalability, and cost-efficiency.Collaborate... ...3+ years of experience in software engineering.Familiar with vLLM, quantization...Performance
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Software Engineer- Inference Performance. Be the first to apply!
Related searches
- software developer internship no experience Winnipeg, MB
- software engineer intern Winnipeg, MB
- software developer Winnipeg, MB
- software developer remote Winnipeg, MB
- software engineer - ai machine learning Winnipeg, MB
- part time software developer Winnipeg, MB
- junior software engineer Winnipeg, MB
- software engineer part time Winnipeg, MB
- software developer intern Winnipeg, MB
- senior software developer Winnipeg, MB

