Senior Ai System Engineer - Llm Training System
$127k - $225k per yearHuawei Canada
Huawei Canada has an immediate permanent opening for a Senior Engineer.About the team:
The Computing Data Application Acceleration Lab aims to create a leading global data analytics platform organized into three specialized teams using innovative programming technologies. This team focuses on full-stack innovations, including software-hardware co-design and optimizing data efficiency at both the storage and runtime layers. This team also develops next-generation GPU architecture for gaming, cloud rendering, VR/AR, and Metaverse applications. One of the goals of this lab are to enhance algorithm performance and training efficiency across industries, fostering long-term competitiveness.About the job:
As a Senior System Engineer, you will drive core compute operator development and distributed scaling performance across foundation model pre-training and post-training workloads on Ascend NPUs. You will identify computational and communication bottlenecks, implement high-performance kernels using AscendC, Triton, and TileLang, and optimize distributed acceleration suites (MindSpeed / open-source training frameworks).Ascend High-Performance Kernel Development: Design, implement, and tune core operators (FlashAttention variants, FP8/MXFP8/W4A8 GEMM, Fused RMSNorm, dynamic MoE routing/dispatch) using AscendC, with explorations in Triton (Triton-Ascend backend) and TileLang.Pre-training & Post-training System Optimization: Architect, scale, and optimize distributed pre-training and RL post-training pipelines using MindSpeed (MindSpeed-LLM / MindSpeed-MM) alongside upstream frameworks (Megatron-LM / Megatron-Core, PyTorch FSDP, veRL, vLLM, Ray).Distributed Parallelism & Training Efficiency: Tune multi-dimensional hybrid parallelism (TP, PP, CP, EP, ZeRO/FSDP) and RL rollout/actor pipelining on Ascend NPU clusters, achieving optimal MFU and compute-communication overlap.Low-Precision & Numerical Parity: Implement low-precision training/inference schemes (FP8, MXFP8, mixed-precision), guaranteeing numerical stability, batch-invariance, and strict precision parity between training and serving runtimes.Profiling & Hardware-Aware Tuning: Conduct deep performance profiling on Ascend hardware using Ascend Profiler (msprof), msprobe, and PyTorch Profiler to eliminate pipeline bubbles, AICore/AIVector stalls, and HCCL communication overhead.Open Source & Ecosystem Enablement: Contribute high-performance operators, bug fixes, and acceleration features to open-source Ascend communities and upstream AI acceleration projects.The total target annual compensation ranges from $127,000 to $225,000 depending on education, experience, and demonstrated expertise.Job requirements
About the ideal candidate:
Bachelor’s, Master’s, or PhD in Computer Science, Computer Engineering, Electrical Engineering, or related technical disciplines, 3+ years of experience in high-performance computing (HPC), AI accelerator operator development, or distributed large model training systems.Hands-on Operator Programming: Solid development experience in AscendC / CANN, or extensive experience in CUDA C/C++ with a proven ability to transition to AscendC. Experience or strong interest in Triton or TileLang for accelerator backends.Distributed Frameworks: Deep practical experience with distributed pre-training and/or post-training frameworks, such as MindSpeed, Megatron-LM / Megatron-Core, PyTorch FSDP, veRL, or vLLM / DeepSpeed. Systems Programming: Strong programming proficiency in C/C++ and modern Python, with hands-on experience building PyTorch custom C++/CANN extensions.Performance Analysis: Proficiency in performance profiling and debugging tools on AI hardware (e.G., Ascend Profiler/msprof, PyTorch Profiler, or Nsight toolchains).Preferred / Standout Qualifications:
Direct experience porting or optimizing Triton or TileLang frontends/backends for non-CUDA accelerators (especially Ascend NPU).Deep understanding of Ascend NPU microarchitecture (Cube & Vector compute units, local memory hierarchy/UB, memory movement engines, and HCCL collective communication). Hands-on experience optimizing end-to-end MFU for massive MoE or dense foundation models on multi-node NPU/GPU clusters.Experience in low-precision numerical alignment (FP8, MXFP8, W4A8/W8A8) and eliminating training-inference discrepancy. Active open-source contributions to Ascend repos (MindSpeed, CANN ops, Ascend-vLLM) or upstream projects (Megatron-LM, Triton, TileLang, veRL).Additional Information:
Huawei Canada is committed to a fair, inclusive, and accessible recruitment process. If you require accommodation during any stage of the hiring process, please let us know and we will work with you to meet your needs.All applications for this position are reviewed directly by our hiring team, we do not use artificial intelligence tools to screen or select candidates.
#J-18808-Ljbffr
The Computing Data Application Acceleration Lab aims to create a leading global data analytics platform organized into three specialized teams using innovative programming technologies. This team focuses on full-stack innovations, including software-hardware co-design and optimizing data efficiency at both the storage and runtime layers. This team also develops next-generation GPU architecture for gaming, cloud rendering, VR/AR, and Metaverse applications. One of the goals of this lab are to enhance algorithm performance and training efficiency across industries, fostering long-term competitiveness.About the job:
As a Senior System Engineer, you will drive core compute operator development and distributed scaling performance across foundation model pre-training and post-training workloads on Ascend NPUs. You will identify computational and communication bottlenecks, implement high-performance kernels using AscendC, Triton, and TileLang, and optimize distributed acceleration suites (MindSpeed / open-source training frameworks).Ascend High-Performance Kernel Development: Design, implement, and tune core operators (FlashAttention variants, FP8/MXFP8/W4A8 GEMM, Fused RMSNorm, dynamic MoE routing/dispatch) using AscendC, with explorations in Triton (Triton-Ascend backend) and TileLang.Pre-training & Post-training System Optimization: Architect, scale, and optimize distributed pre-training and RL post-training pipelines using MindSpeed (MindSpeed-LLM / MindSpeed-MM) alongside upstream frameworks (Megatron-LM / Megatron-Core, PyTorch FSDP, veRL, vLLM, Ray).Distributed Parallelism & Training Efficiency: Tune multi-dimensional hybrid parallelism (TP, PP, CP, EP, ZeRO/FSDP) and RL rollout/actor pipelining on Ascend NPU clusters, achieving optimal MFU and compute-communication overlap.Low-Precision & Numerical Parity: Implement low-precision training/inference schemes (FP8, MXFP8, mixed-precision), guaranteeing numerical stability, batch-invariance, and strict precision parity between training and serving runtimes.Profiling & Hardware-Aware Tuning: Conduct deep performance profiling on Ascend hardware using Ascend Profiler (msprof), msprobe, and PyTorch Profiler to eliminate pipeline bubbles, AICore/AIVector stalls, and HCCL communication overhead.Open Source & Ecosystem Enablement: Contribute high-performance operators, bug fixes, and acceleration features to open-source Ascend communities and upstream AI acceleration projects.The total target annual compensation ranges from $127,000 to $225,000 depending on education, experience, and demonstrated expertise.Job requirements
About the ideal candidate:
Bachelor’s, Master’s, or PhD in Computer Science, Computer Engineering, Electrical Engineering, or related technical disciplines, 3+ years of experience in high-performance computing (HPC), AI accelerator operator development, or distributed large model training systems.Hands-on Operator Programming: Solid development experience in AscendC / CANN, or extensive experience in CUDA C/C++ with a proven ability to transition to AscendC. Experience or strong interest in Triton or TileLang for accelerator backends.Distributed Frameworks: Deep practical experience with distributed pre-training and/or post-training frameworks, such as MindSpeed, Megatron-LM / Megatron-Core, PyTorch FSDP, veRL, or vLLM / DeepSpeed. Systems Programming: Strong programming proficiency in C/C++ and modern Python, with hands-on experience building PyTorch custom C++/CANN extensions.Performance Analysis: Proficiency in performance profiling and debugging tools on AI hardware (e.G., Ascend Profiler/msprof, PyTorch Profiler, or Nsight toolchains).Preferred / Standout Qualifications:
Direct experience porting or optimizing Triton or TileLang frontends/backends for non-CUDA accelerators (especially Ascend NPU).Deep understanding of Ascend NPU microarchitecture (Cube & Vector compute units, local memory hierarchy/UB, memory movement engines, and HCCL collective communication). Hands-on experience optimizing end-to-end MFU for massive MoE or dense foundation models on multi-node NPU/GPU clusters.Experience in low-precision numerical alignment (FP8, MXFP8, W4A8/W8A8) and eliminating training-inference discrepancy. Active open-source contributions to Ascend repos (MindSpeed, CANN ops, Ascend-vLLM) or upstream projects (Megatron-LM, Triton, TileLang, veRL).Additional Information:
Huawei Canada is committed to a fair, inclusive, and accessible recruitment process. If you require accommodation during any stage of the hiring process, please let us know and we will work with you to meet your needs.All applications for this position are reviewed directly by our hiring team, we do not use artificial intelligence tools to screen or select candidates.
#J-18808-Ljbffr
Vacancy posted 1 hour ago
Similar jobs that could be interesting for youBased on the Senior Ai System Engineer - Llm Training System in Winnipeg, MB vacancy
- ...Job Description Miratech is looking for a skilled Senior AI Engineer with Python, LangGraph and LangChain to join our US healthcare... ...capabilities. Integrate AI agents and applications with enterprise systems, services, and APIs. Support modern AI integration...SeniorFull timeInternshipRemote workRelocation
- ...Drive AI innovation as a Senior AI Engineer in Mississauga, Canada. Craft Python-based AI applications using FastAPI or Flask while optimizing performance... ...in software development. You will design and deploy AI and LLM applications, develop REST APIs, and collaborate with...Senior
- ...An opportunity exists for a Bridge Engineer-in-Training to become part of our dynamic Transportation... ..., specifying, and engineering bridge systems and products for a variety of transportation... .... Support intermediate and senior team members on transportation projects...SeniorTrainingFull timeContract workFor contractorsSummer workWork at officeLocal area
- ...Are you a passionate and driven young engineer looking to make a real impact on your community... ...under the guidance and mentorship of senior engineers. Hands-On Experience: Conduct... ...already registered) as an Engineering-In-Training with EGM. Valid Canadian driver’s license...SeniorTrainingFor contractorsLocal areaFlexible hours
- ...helping you reach your potential through training and education. Community... ...Coffee. Position Summary Join our AI team as an Intermediate AI Engineer, where you'll build and deploy AI applications... ...AI solutions with enterprise systems, applications, APIs, and data...Training
$89.91k - $113.73k per year
The Engineering Service Business Unit focuses on power system software development, power system and substation consulting studies... ...also be considered. * At the senior level, at least 8 years of... ...core competencies. Internationally-trained engineers may be considered if eligible...SeniorFull timeImmediate startRemote work- ...We are committed to helping you reach your potential through training and education. • Community Involvement - Generously supporting... ...Price Café offers free Tea and Coffee. Finishing Systems Engineering Assistant (8 Month Term – Winter 2027) The successful candidate...TrainingFor contractors
- Lead LLM systems at Axya, working hands-on with Python, retrieval, agent workflows, and evaluation in a 100% remote environment.As Lead AI Engineer, you will own Axya’s AI architecture and technical direction. You will report to the CTO while partnering with Product, Customer...Remote work
- ...learning platforms at PointClickCare using Python, Java, and MLOps expertise. This role supports AI products across healthcare. As a Senior Machine Learning Systems Engineer, you will design, build, and operate platform capabilities for model development and deployment....SeniorTraining
- ...Morningstar is seeking a Senior Software Engineer to build applied systems for AI capabilities in production, including data pipelines, LLM integrations, agentic workflows, and evaluation frameworks. This role focuses on applied engineering, delivering durable architectures...SeniorWork at office
- ...Aerospace Engine Mechanic Training Program (GTRO) Winnipeg, Manitoba Build an Aviation Career You’re Proud Of At StandardAero, we use our ingenuity and know-how to find solutions for the simple to the most complex challenges in aviation. Together, we get the job done...TrainingLong term contractTemporary workTraineeshipSecond job
- Design RAG systems and LLM features for Wiser Solutions remotely, using embeddings and retrieval to solve real customer problems.This Senior Machine Learning Engineer builds AI applications that reach production across product matching, classification, and extraction. You will...SeniorRemote work
- Get notified about new Director Of Software Engineering jobs in United States . 7,000+ Director Of Software Engineering Jobs in United States... ...Engineering – Acute Devices Director, Software Engineering - AI Platform and Full Stack Engineering - Office-based in Minneapolis...Work at officeRemote work
- ZS is seeking an LLM Engineer to design and build scalable software systems leveraging Python, LLM patterns, and cloud platforms. You will translate business requirements... ...pipelines, and implement guardrails for responsible AI, joining a collaborative, client-focused team. #J-...
$195k - $285k per year
...breakthrough outcomes for their toughest AI challenges. We blend the... ...across industries. As a Senior Lead AI Developer, you are... ...teams design and build AI/ML systems and pipelines that consistently... ...that leverage our AI and engineering capabilitiesBuilding and deploying...SeniorTrainingWork at officeLocal areaRemote workFlexible hours3 days per week1 day per week$75.5k - $125k per year
...Permanent Full Time - We are hiring a Senior IT Business Systems Analyst. This is a hybrid role located in London, Toronto or Winnipeg. Reporting... ...Information Systems, etc.) or equivalent combination of training and experience. Strong analytical and problem-solving...SeniorTrainingPermanent employmentFull timePart timeInternshipFlexible hours- ...the role247 Labs is looking for a hands-on AI Engineer to design, develop, deploy, and maintain... ...LLMs, machine learning models, retrieval systems, and agentic workflows.Evaluate business... ...including meaningful hands-on production LLM development.Strong Python programming skills...
- ...BrainyYack is seeking a Senior Python Developer to design and build scalable backend systems and AI-powered solutions. You will own backend development, APIs, and integrations... ...software. The role emphasizes hands-on engineering, system design, and mentoring teammates while...SeniorRemote work
- ...seeking a Member of Technical Staff (Post training team) to help advance model training... ...Ideal candidates possess strong software engineering skills and should be proficient in Python... ...a diverse team pushing the boundaries of AI research while enjoying a supportive work...Training
- AMD is seeking a Senior Software Development Engineer to own the end-to-end model execution stack on AMD Instinct GPUs, covering training infrastructure at scale and high-performance inference serving. You will ship LLMs on real hardware, optimize GPU kernels, and ensure frontier...SeniorTraining
$114.8k - $157.85k per year
...Job Description Senior Electrical Engineer Are you looking for a company that works on major engineering... ...expertise in electrical building systems including power utilities & distribution... ...apart. Project experience should include training centers, vehicle and fleet maintenance...SeniorTrainingFull timeWork at officeWorldwideFlexible hours$114.8k - $157.85k per year
...Job Description Senior Mechanical Engineer Are you looking for a company that works on major engineering... ...expertise in mechanical building systems including HVAC, plumbing, fire protection... .... Project experience should include training centers, vehicle and fleet maintenance...SeniorTrainingFull timeWork at officeWorldwideFlexible hours- ...Senior Automation Engineer (Project Manager) Saguenay-Lac-Saint-Jean, QC, Canada Type of position... ...operating and maintaining industrial power systems. By joining our team, you’ll... ...engineer or an equivalent combination of training and experience Experience in automation...SeniorTrainingLong term contractTemporary workWork at officeLocal area
- ...high-performance computing, cloud, and AI. Whether you’re designing next-gen... ...RoleThe AMD AI Group is looking for a Senior Software Development Engineer to own the end-to-end model execution... ...stack on AMD Instinct GPUs - spanning training infrastructure at scale and high-...SeniorTrainingInternshipManual labor
- ...Salesforce, the leading AI CRM, seeks a Senior Technical Consultant to drive enterprise deployments and AI-driven delivery. You'll partner with clients and internal teams to define architectures, implement solutions, and manage release cycles across desktop, cloud, and mobile...Senior
- ...Chez Ubisoft, programmez des systèmes ML en C++ avec une expertise en moteurs d’inférence, tests, performance et architecture logicielle. Ce poste de programmeur principal C++ contribue à la feuille de route des applications d’apprentissage automatique. Vous développerez...SeniorApprenticeship
- ...designed for both desktop and mobile web rendering. Our back-end systems (some Java SpringBoot, some PHP) manage data processing,... ...is driven by our core value of “we-are-we” and as a Senior AI Systems Engineer, you will work in an outcome-driven environment where collaboration...SeniorTrainingWork at officeFlexible hours
- ...has large-scale international engineering projects to offer. If you... ...Structural Engineer (Intermediate to Senior) to support the continued... ...of diverse opportunities for training and professional development... ...'s infrastructure and energy systems. Together, with our industry...SeniorTrainingInternshipWork at officeFlexible hours
- Are you a passionate and driven young engineer looking to make a real impact on your community? Stantec is a global engineering consulting firm dedicated to creating sustainable and innovative solutions. We are looking for enthusiastic individuals to join our team and embark...TrainingFull timeLocal area
- ...a remote-first company and a people-first global community of senior engineers. Kake engineers are behind some of the world’s most innovative... ...people stay, grow, and thrive (and love eating kake!). Senior AI/LLM Engineers with strong Python experience, skilled in designing...SeniorRemote workFlexible hours
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Senior Ai System Engineer - Llm Training System. Be the first to apply!
Related searches
- system development engineer Winnipeg, MB
- computer system validation engineer Winnipeg, MB
- space systems engineer Winnipeg, MB
- senior financial accountant Winnipeg, MB
- senior sales consultant Winnipeg, MB
- senior communications specialist Winnipeg, MB
- senior bookkeeper Winnipeg, MB
- senior system administrator Winnipeg, MB
- senior technical recruiter Winnipeg, MB
- senior director Winnipeg, MB


