Senior Ai System Engineer - Llm Training System
$127k - $225k per yearHuawei Technologies Co.
Huawei Canada has an immediate permanent opening for a Senior Engineer.
About the team:The Computing Data Application Acceleration Lab aims to create a leading global data analytics platform organized into three specialized teams using innovative programming technologies. This team focuses on full-stack innovations, including software-hardware co-design and optimizing data efficiency at both the storage and runtime layers. This team also develops next-generation GPU architecture for gaming, cloud rendering, VR/AR, and Metaverse applications. One of the goals of this lab are to enhance algorithm performance and training efficiency across industries, fostering long-term competitiveness.
About the job:As a Senior System Engineer, you will drive core compute operator development and distributed scaling performance across foundation model pre-training and post-training workloads on Ascend NPUs. You will identify computational and communication bottlenecks, implement high-performance kernels using AscendC, Triton, and TileLang, and optimize distributed acceleration suites (MindSpeed / open-source training frameworks).
Ascend High-Performance Kernel Development: Design, implement, and tune core operators (FlashAttention variants, FP8/MXFP8/W4A8 GEMM, Fused RMSNorm, dynamic MoE routing/dispatch) using AscendC, with explorations in Triton (Triton-Ascend backend) and TileLang.
Pre-training & Post-training System Optimization: Architect, scale, and optimize distributed pre-training and RL post-training pipelines using MindSpeed (MindSpeed-LLM / MindSpeed-MM) alongside upstream frameworks (Megatron-LM / Megatron-Core, PyTorch FSDP, veRL, vLLM, Ray).
Distributed Parallelism & Training Efficiency: Tune multi-dimensional hybrid parallelism (TP, PP, CP, EP, ZeRO/FSDP) and RL rollout/actor pipelining on Ascend NPU clusters, achieving optimal MFU and compute-communication overlap.
Low-Precision & Numerical Parity: Implement low-precision training/inference schemes (FP8, MXFP8, mixed-precision), guaranteeing numerical stability, batch-invariance, and strict precision parity between training and serving runtimes.
Profiling & Hardware-Aware Tuning: Conduct deep performance profiling on Ascend hardware using Ascend Profiler (msprof), msprobe, and PyTorch Profiler to eliminate pipeline bubbles, AICore/AIVector stalls, and HCCL communication overhead.
Open Source & Ecosystem Enablement: Contribute high-performance operators, bug fixes, and acceleration features to open-source Ascend communities and upstream AI acceleration projects.
The total target annual compensation ranges from $127,000 to $225,000 depending on education, experience, and demonstrated expertise.
About the ideal candidate:
Bachelor’s, Master’s, or PhD in Computer Science, Computer Engineering, Electrical Engineering, or related technical disciplines, 3+ years of experience in high-performance computing (HPC), AI accelerator operator development, or distributed large model training systems.
Hands-on Operator Programming: Solid development experience in AscendC / CANN, or extensive experience in CUDA C/C++ with a proven ability to transition to AscendC. Experience or strong interest in Triton or TileLang for accelerator backends.
Distributed Frameworks: Deep practical experience with distributed pre-training and/or post-training frameworks, such as MindSpeed, Megatron-LM / Megatron-Core, PyTorch FSDP, veRL, or vLLM / DeepSpeed. Systems Programming: Strong programming proficiency in C/C++ and modern Python, with hands-on experience building PyTorch custom C++/CANN extensions.
Performance Analysis: Proficiency in performance profiling and debugging tools on AI hardware (e.G., Ascend Profiler/msprof, PyTorch Profiler, or Nsight toolchains).
Preferred / Standout Qualifications:
Direct experience porting or optimizing Triton or TileLang frontends/backends for non-CUDA accelerators (especially Ascend NPU).
Deep understanding of Ascend NPU microarchitecture (Cube & Vector compute units, local memory hierarchy/UB, memory movement engines, and HCCL collective communication). Hands-on experience optimizing end-to-end MFU for massive MoE or dense foundation models on multi-node NPU/GPU clusters.
Experience in low-precision numerical alignment (FP8, MXFP8, W4A8/W8A8) and eliminating training-inference discrepancy. Active open-source contributions to Ascend repos (MindSpeed, CANN ops, Ascend-vLLM) or upstream projects (Megatron-LM, Triton, TileLang, veRL).
Additional Information:Huawei Canada is committed to a fair, inclusive, and accessible recruitment process. If you require accommodation during any stage of the hiring process, please let us know and we will work with you to meet your needs.
All applications for this position are reviewed directly by our hiring team,
we do not use artificial intelligence tools
to screen or select candidates.
#J-18808-Ljbffr
About the team:The Computing Data Application Acceleration Lab aims to create a leading global data analytics platform organized into three specialized teams using innovative programming technologies. This team focuses on full-stack innovations, including software-hardware co-design and optimizing data efficiency at both the storage and runtime layers. This team also develops next-generation GPU architecture for gaming, cloud rendering, VR/AR, and Metaverse applications. One of the goals of this lab are to enhance algorithm performance and training efficiency across industries, fostering long-term competitiveness.
About the job:As a Senior System Engineer, you will drive core compute operator development and distributed scaling performance across foundation model pre-training and post-training workloads on Ascend NPUs. You will identify computational and communication bottlenecks, implement high-performance kernels using AscendC, Triton, and TileLang, and optimize distributed acceleration suites (MindSpeed / open-source training frameworks).
Ascend High-Performance Kernel Development: Design, implement, and tune core operators (FlashAttention variants, FP8/MXFP8/W4A8 GEMM, Fused RMSNorm, dynamic MoE routing/dispatch) using AscendC, with explorations in Triton (Triton-Ascend backend) and TileLang.
Pre-training & Post-training System Optimization: Architect, scale, and optimize distributed pre-training and RL post-training pipelines using MindSpeed (MindSpeed-LLM / MindSpeed-MM) alongside upstream frameworks (Megatron-LM / Megatron-Core, PyTorch FSDP, veRL, vLLM, Ray).
Distributed Parallelism & Training Efficiency: Tune multi-dimensional hybrid parallelism (TP, PP, CP, EP, ZeRO/FSDP) and RL rollout/actor pipelining on Ascend NPU clusters, achieving optimal MFU and compute-communication overlap.
Low-Precision & Numerical Parity: Implement low-precision training/inference schemes (FP8, MXFP8, mixed-precision), guaranteeing numerical stability, batch-invariance, and strict precision parity between training and serving runtimes.
Profiling & Hardware-Aware Tuning: Conduct deep performance profiling on Ascend hardware using Ascend Profiler (msprof), msprobe, and PyTorch Profiler to eliminate pipeline bubbles, AICore/AIVector stalls, and HCCL communication overhead.
Open Source & Ecosystem Enablement: Contribute high-performance operators, bug fixes, and acceleration features to open-source Ascend communities and upstream AI acceleration projects.
The total target annual compensation ranges from $127,000 to $225,000 depending on education, experience, and demonstrated expertise.
About the ideal candidate:
Bachelor’s, Master’s, or PhD in Computer Science, Computer Engineering, Electrical Engineering, or related technical disciplines, 3+ years of experience in high-performance computing (HPC), AI accelerator operator development, or distributed large model training systems.
Hands-on Operator Programming: Solid development experience in AscendC / CANN, or extensive experience in CUDA C/C++ with a proven ability to transition to AscendC. Experience or strong interest in Triton or TileLang for accelerator backends.
Distributed Frameworks: Deep practical experience with distributed pre-training and/or post-training frameworks, such as MindSpeed, Megatron-LM / Megatron-Core, PyTorch FSDP, veRL, or vLLM / DeepSpeed. Systems Programming: Strong programming proficiency in C/C++ and modern Python, with hands-on experience building PyTorch custom C++/CANN extensions.
Performance Analysis: Proficiency in performance profiling and debugging tools on AI hardware (e.G., Ascend Profiler/msprof, PyTorch Profiler, or Nsight toolchains).
Preferred / Standout Qualifications:
Direct experience porting or optimizing Triton or TileLang frontends/backends for non-CUDA accelerators (especially Ascend NPU).
Deep understanding of Ascend NPU microarchitecture (Cube & Vector compute units, local memory hierarchy/UB, memory movement engines, and HCCL collective communication). Hands-on experience optimizing end-to-end MFU for massive MoE or dense foundation models on multi-node NPU/GPU clusters.
Experience in low-precision numerical alignment (FP8, MXFP8, W4A8/W8A8) and eliminating training-inference discrepancy. Active open-source contributions to Ascend repos (MindSpeed, CANN ops, Ascend-vLLM) or upstream projects (Megatron-LM, Triton, TileLang, veRL).
Additional Information:Huawei Canada is committed to a fair, inclusive, and accessible recruitment process. If you require accommodation during any stage of the hiring process, please let us know and we will work with you to meet your needs.
All applications for this position are reviewed directly by our hiring team,
we do not use artificial intelligence tools
to screen or select candidates.
#J-18808-Ljbffr
Vacancy posted 53 minutes ago
Similar jobs that could be interesting for youBased on the Senior Ai System Engineer - Llm Training System in Winnipeg, MB vacancy
- Lead LLM systems at Axya, working hands-on with Python, retrieval, agent workflows, and evaluation in a 100% remote environment.As Lead AI Engineer, you will own Axya’s AI architecture and technical direction. You will report to the CTO while partnering with Product, Customer...SuggestedRemote work
- ...Morningstar is seeking a Senior Software Engineer to build applied systems for AI capabilities in production, including data pipelines, LLM integrations, agentic workflows, and evaluation frameworks. This role focuses on applied engineering, delivering durable architectures...SeniorWork at office
- Design RAG systems and LLM features for Wiser Solutions remotely, using embeddings and retrieval to solve real customer problems.This Senior Machine Learning Engineer builds AI applications that reach production across product matching, classification, and extraction. You will...SeniorRemote work
- Get notified about new Director Of Software Engineering jobs in United States . 7,000+ Director Of Software Engineering Jobs in United States... ...Engineering – Acute Devices Director, Software Engineering - AI Platform and Full Stack Engineering - Office-based in Minneapolis...SuggestedWork at officeRemote work
- ZS is seeking an LLM Engineer to design and build scalable software systems leveraging Python, LLM patterns, and cloud platforms. You will translate business requirements... ...pipelines, and implement guardrails for responsible AI, joining a collaborative, client-focused team. #J-...Suggested
- ...the role247 Labs is looking for a hands-on AI Engineer to design, develop, deploy, and maintain... ...LLMs, machine learning models, retrieval systems, and agentic workflows.Evaluate business... ...including meaningful hands-on production LLM development.Strong Python programming skills...
- ...BrainyYack is seeking a Senior Python Developer to design and build scalable backend systems and AI-powered solutions. You will own backend development, APIs, and integrations... ...software. The role emphasizes hands-on engineering, system design, and mentoring teammates while...SeniorRemote work
$195k - $285k per year
...breakthrough outcomes for their toughest AI challenges. We blend the... ...across industries. As a Senior Lead AI Developer, you are... ...teams design and build AI/ML systems and pipelines that consistently... ...that leverage our AI and engineering capabilitiesBuilding and deploying...SeniorTrainingWork at officeLocal areaRemote workFlexible hours3 days per week1 day per week- AMD is seeking a Senior Software Development Engineer to own the end-to-end model execution stack on AMD Instinct GPUs, covering training infrastructure at scale and high-performance inference serving. You will ship LLMs on real hardware, optimize GPU kernels, and ensure frontier...SeniorTraining
- ...seeking a Member of Technical Staff (Post training team) to help advance model training... ...Ideal candidates possess strong software engineering skills and should be proficient in Python... ...a diverse team pushing the boundaries of AI research while enjoying a supportive work...Training
- ...designed for both desktop and mobile web rendering. Our back-end systems (some Java SpringBoot, some PHP) manage data processing,... ...is driven by our core value of “we-are-we” and as a Senior AI Systems Engineer, you will work in an outcome-driven environment where collaboration...SeniorTrainingWork at officeFlexible hours
- ...high-performance computing, cloud, and AI. Whether you’re designing next-gen... ...RoleThe AMD AI Group is looking for a Senior Software Development Engineer to own the end-to-end model execution... ...stack on AMD Instinct GPUs - spanning training infrastructure at scale and high-...SeniorTrainingInternshipManual labor
- ...Job Description Miratech is looking for a skilled Senior AI Engineer with Python, LangGraph and LangChain to join our US healthcare... ...capabilities. Integrate AI agents and applications with enterprise systems, services, and APIs. Support modern AI integration...SeniorFull timeInternshipRemote workRelocation
- ...a remote-first company and a people-first global community of senior engineers. Kake engineers are behind some of the world’s most innovative... ...people stay, grow, and thrive (and love eating kake!). Senior AI/LLM Engineers with strong Python experience, skilled in designing...SeniorRemote workFlexible hours
- Le Développeur Java Senior chez DELAN - IT Talent Solutions conçoit et développe des fonctionnalités IA générative (LLM) dans une plateforme SaaS en Java. Vous bâtissez des API REST robustes... ...d expérience, vous maîtrisez MCP, Vertex AI, GCP et Kubernetes, et vous collaborez au...Senior
- Engage in cutting-edge AI language model development as a Senior Software Engineer. With Ethos, work remotely and leverage your expertise for impactful software engineering... ...outputs across essential software workflows. Your system design and algorithm skills will be pivotal in...SeniorTrainingRemote workFlexible hours
- Cohere is a leading security-first enterprise AI company building foundation AI models and end-to-end products. We are hiring for a role that builds and owns pipelines for post-training data used to train large language models. You will collaborate with researchers and annotation...TrainingRemote workWorldwide
- ...Drive AI innovation as a Senior AI Engineer in Mississauga, Canada. Craft Python-based AI applications using FastAPI or Flask while optimizing performance... ...in software development. You will design and deploy AI and LLM applications, develop REST APIs, and collaborate with...Senior
- A high-growth SaaS company based in Vancouver is looking for a Senior AI Engineer to design and scale core AI systems. The role involves building scalable data pipelines and implementing advanced AI workflows. Ideal candidates have experience with LLM applications, expert software...Senior
- ...Affirm is seeking a seasoned security engineer to own and refine our enterprise AI security review process. You will evaluate architecture, data flows,... ...design choices. You will lead threat modeling of AI/LLM systems, review MCP server definitions, and help owners build...Senior
- Oliver Wyman Quotient is hiring a Senior Lead AI Developer to manage technical projects and guide teams in designing and building AI/ML systems and production-ready pipelines. You will work... ...exploring data, shaping proposals, deploying LLM-based agents, and advancing state-of-the...Senior
- Build advanced language systems as a Senior AI Engineer in Kimberley, applying NLP and LLM expertise within a hybrid work model. Develop scalable solutions that understand, generate, and process human language at scale.This senior position focuses on developing and deploying...SeniorTraining
- Senior AI EngineerLocation: Toronto, ON | Hybrid (3 days onsite: Tuesday-Thursday, 8:30 AM to... ...and highly motivated Senior AI Engineer to join the Technology Strategy team supporting... ...designing, building, and operating advanced AI systems that enable end-to-end customer...SeniorInternship
- RAPS Consulting Inc in Mississauga seeks an experienced software engineer to design and deploy Python-based AI/LLM applications, using FastAPI/Flask within scalable enterprise systems. You will build REST APIs, analytics platforms, and optimized data pipelines, delivering AI...Senior
- Drive AI innovation at Unblocked as a Senior AI Engineer based in Vancouver. Focus on designing robust AI systems that deliver exceptional user experiences through data-driven solutions. You'll lead the design and scaling of AI infrastructures, employing advanced machine...SeniorWork at office
$158.4k - $237.6k per year
...Technologies, Inc. Job AreaEngineering Group, Engineering Group Machine Learning Engineering LLM Serving Engineer (Cloud AI Engineering)Qualcomm is utilizing its... ...accelerators, in‑memory processing and distributed systems. Strong Python development skills for...SeniorWork from home- Extend Tempo for AI and agent workloads through structured APIs, observability systems, and scalable engineering. Work remotely with modern coding assistants.As a Senior Engineer on Tempo, you will build capabilities for Grafana’s AI assistant, external integrators, and agent...SeniorRemote work
- Smart IT Frame LLC is seeking an experienced software engineer to design, build, and deploy Python-based AI/LLM applications using FastAPI or Flask. The role includes developing REST APIs, scalable enterprise solutions, and AI-powered data analytics platforms. You will optimize...Senior
- Develop AI-assisted backend features for Jeeves from a full-time remote setting, combining LLM integrations with distributed financial systems. Collaborate across global engineering and data teams.In this Senior Backend Engineer position, you will connect experimental AI work...SeniorFull timeRemote work
- Take on a challenging role at Jonas Software as a Senior Software Engineer focusing on AI systems integration and modernization. Help innovate software solutions that impact real-world operations. As a senior engineer, you will lead the design and delivery of comprehensive...Senior
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Senior Ai System Engineer - Llm Training System. Be the first to apply!
Related searches
- system development engineer Winnipeg, MB
- space systems engineer Winnipeg, MB
- computer system validation engineer Winnipeg, MB
- senior disability case manager Winnipeg, MB
- senior hardware engineer Winnipeg, MB
- senior cook Winnipeg, MB
- amica senior lifestyles Winnipeg, MB
- senior building operator Winnipeg, MB
- senior project manager Winnipeg, MB
- senior communications Winnipeg, MB

