Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Senior Ai System Engineer - Llm Training System

$127k - $225k per year

Huawei Canada

Huawei Canada has an immediate permanent opening for a Senior Engineer.About the team:
The Computing Data Application Acceleration Lab aims to create a leading global data analytics platform organized into three specialized teams using innovative programming technologies. This team focuses on full-stack innovations, including software-hardware co-design and optimizing data efficiency at both the storage and runtime layers. This team also develops next-generation GPU architecture for gaming, cloud rendering, VR/AR, and Metaverse applications. One of the goals of this lab are to enhance algorithm performance and training efficiency across industries, fostering long-term competitiveness.About the job:
As a Senior System Engineer, you will drive core compute operator development and distributed scaling performance across foundation model pre-training and post-training workloads on Ascend NPUs. You will identify computational and communication bottlenecks, implement high-performance kernels using AscendC, Triton, and TileLang, and optimize distributed acceleration suites (MindSpeed / open-source training frameworks).Ascend High-Performance Kernel Development: Design, implement, and tune core operators (FlashAttention variants, FP8/MXFP8/W4A8 GEMM, Fused RMSNorm, dynamic MoE routing/dispatch) using AscendC, with explorations in Triton (Triton-Ascend backend) and TileLang.Pre-training & Post-training System Optimization: Architect, scale, and optimize distributed pre-training and RL post-training pipelines using MindSpeed (MindSpeed-LLM / MindSpeed-MM) alongside upstream frameworks (Megatron-LM / Megatron-Core, PyTorch FSDP, veRL, vLLM, Ray).Distributed Parallelism & Training Efficiency: Tune multi-dimensional hybrid parallelism (TP, PP, CP, EP, ZeRO/FSDP) and RL rollout/actor pipelining on Ascend NPU clusters, achieving optimal MFU and compute-communication overlap.Low-Precision & Numerical Parity: Implement low-precision training/inference schemes (FP8, MXFP8, mixed-precision), guaranteeing numerical stability, batch-invariance, and strict precision parity between training and serving runtimes.Profiling & Hardware-Aware Tuning: Conduct deep performance profiling on Ascend hardware using Ascend Profiler (msprof), msprobe, and PyTorch Profiler to eliminate pipeline bubbles, AICore/AIVector stalls, and HCCL communication overhead.Open Source & Ecosystem Enablement: Contribute high-performance operators, bug fixes, and acceleration features to open-source Ascend communities and upstream AI acceleration projects.The total target annual compensation ranges from $127,000 to $225,000 depending on education, experience, and demonstrated expertise.Job requirements
About the ideal candidate:
Bachelor’s, Master’s, or PhD in Computer Science, Computer Engineering, Electrical Engineering, or related technical disciplines, 3+ years of experience in high-performance computing (HPC), AI accelerator operator development, or distributed large model training systems.Hands-on Operator Programming: Solid development experience in AscendC / CANN, or extensive experience in CUDA C/C++ with a proven ability to transition to AscendC. Experience or strong interest in Triton or TileLang for accelerator backends.Distributed Frameworks: Deep practical experience with distributed pre-training and/or post-training frameworks, such as MindSpeed, Megatron-LM / Megatron-Core, PyTorch FSDP, veRL, or vLLM / DeepSpeed. Systems Programming: Strong programming proficiency in C/C++ and modern Python, with hands-on experience building PyTorch custom C++/CANN extensions.Performance Analysis: Proficiency in performance profiling and debugging tools on AI hardware (e.G., Ascend Profiler/msprof, PyTorch Profiler, or Nsight toolchains).Preferred / Standout Qualifications:
Direct experience porting or optimizing Triton or TileLang frontends/backends for non-CUDA accelerators (especially Ascend NPU).Deep understanding of Ascend NPU microarchitecture (Cube & Vector compute units, local memory hierarchy/UB, memory movement engines, and HCCL collective communication). Hands-on experience optimizing end-to-end MFU for massive MoE or dense foundation models on multi-node NPU/GPU clusters.Experience in low-precision numerical alignment (FP8, MXFP8, W4A8/W8A8) and eliminating training-inference discrepancy. Active open-source contributions to Ascend repos (MindSpeed, CANN ops, Ascend-vLLM) or upstream projects (Megatron-LM, Triton, TileLang, veRL).Additional Information:
Huawei Canada is committed to a fair, inclusive, and accessible recruitment process. If you require accommodation during any stage of the hiring process, please let us know and we will work with you to meet your needs.All applications for this position are reviewed directly by our hiring team, we do not use artificial intelligence tools to screen or select candidates.
#J-18808-Ljbffr
Vacancy posted 1 hour ago
Similar jobs that could be interesting for youBased on the Senior Ai System Engineer - Llm Training System in Winnipeg, MB vacancy
  •  ...Job Description Miratech is looking for a skilled Senior AI Engineer with Python, LangGraph and LangChain to join our US healthcare...  ...capabilities. Integrate AI agents and applications with enterprise systems, services, and APIs. Support modern AI integration... 
    Senior
    Full time
    Internship
    Remote work
    Relocation

    Miratech

    Winnipeg, MB
    8 days ago
  •  ...Drive AI innovation as a Senior AI Engineer in Mississauga, Canada. Craft Python-based AI applications using FastAPI or Flask while optimizing performance...  ...in software development. You will design and deploy AI and LLM applications, develop REST APIs, and collaborate with... 
    Senior

    Talentola

    Winnipeg, MB
    9 days ago
  •  ...An opportunity exists for a Bridge Engineer-in-Training to become part of our dynamic Transportation...  ..., specifying, and engineering bridge systems and products for a variety of transportation...  .... Support intermediate and senior team members on transportation projects... 
    Senior
    Training
    Full time
    Contract work
    For contractors
    Summer work
    Work at office
    Local area

    Stantec

    Winnipeg, MB
    9 days ago
  •  ...Are you a passionate and driven young engineer looking to make a real impact on your community...  ...under the guidance and mentorship of senior engineers.  Hands-On Experience: Conduct...  ...already registered) as an Engineering-In-Training with EGM.  Valid Canadian driver’s license... 
    Senior
    Training
    For contractors
    Local area
    Flexible hours

    Stantec

    Winnipeg, MB
    9 days ago
  •  ...helping you reach your potential through training and education. Community...  ...Coffee. Position Summary Join our AI team as an Intermediate AI Engineer, where you'll build and deploy AI applications...  ...AI solutions with enterprise systems, applications, APIs, and data... 
    Training

    Price Industries Limited

    Winnipeg, MB
    9 days ago
  • $89.91k - $113.73k per year

    The Engineering Service Business Unit focuses on power system software development, power system and substation consulting studies...  ...also be considered. * At the senior level,  at least 8 years of...  ...core competencies. Internationally-trained engineers may be considered if eligible... 
    Senior
    Full time
    Immediate start
    Remote work

    Powertech Labs

    Winnipeg, MB
    6 days ago
  •  ...We are committed to helping you reach your potential through training and education. • Community Involvement - Generously supporting...  ...Price Café offers free Tea and Coffee. Finishing Systems Engineering Assistant (8 Month Term – Winter 2027) The successful candidate... 
    Training
    For contractors

    Price Industries Limited

    Winnipeg, MB
    9 days ago
  • Lead LLM systems at Axya, working hands-on with Python, retrieval, agent workflows, and evaluation in a 100% remote environment.As Lead AI Engineer, you will own Axya’s AI architecture and technical direction. You will report to the CTO while partnering with Product, Customer... 
    Remote work

    Axya

    Winnipeg, MB
    1 hour ago
  •  ...learning platforms at PointClickCare using Python, Java, and MLOps expertise. This role supports AI products across healthcare. As a Senior Machine Learning Systems Engineer, you will design, build, and operate platform capabilities for model development and deployment.... 
    Senior
    Training

    The Institute for Performance and Learning

    Winnipeg, MB
    9 days ago
  •  ...Morningstar is seeking a Senior Software Engineer to build applied systems for AI capabilities in production, including data pipelines, LLM integrations, agentic workflows, and evaluation frameworks. This role focuses on applied engineering, delivering durable architectures... 
    Senior
    Work at office

    Urban Ridge Supplies

    Winnipeg, MB
    1 hour ago
  •  ...Aerospace Engine Mechanic Training Program (GTRO) Winnipeg, Manitoba  Build an Aviation Career You’re Proud Of At StandardAero, we use our ingenuity and know-how to find solutions for the simple to the most complex challenges in aviation. Together, we get the job done... 
    Training
    Long term contract
    Temporary work
    Traineeship
    Second job

    StandardAero

    Winnipeg, MB
    21 days ago
  • Design RAG systems and LLM features for Wiser Solutions remotely, using embeddings and retrieval to solve real customer problems.This Senior Machine Learning Engineer builds AI applications that reach production across product matching, classification, and extraction. You will... 
    Senior
    Remote work

    Wiser Solutions

    Winnipeg, MB
    1 hour ago
  • Get notified about new Director Of Software Engineering jobs in United States . 7,000+ Director Of Software Engineering Jobs in United States...  ...Engineering – Acute Devices Director, Software Engineering - AI Platform and Full Stack Engineering - Office-based in Minneapolis... 
    Work at office
    Remote work

    Talent Seed

    Winnipeg, MB
    1 hour ago
  • ZS is seeking an LLM Engineer to design and build scalable software systems leveraging Python, LLM patterns, and cloud platforms. You will translate business requirements...  ...pipelines, and implement guardrails for responsible AI, joining a collaborative, client-focused team. #J-... 

    ZS

    Winnipeg, MB
    1 hour ago
  • $195k - $285k per year

     ...breakthrough outcomes for their toughest AI challenges. We blend the...  ...across industries. As a Senior Lead AI Developer, you are...  ...teams design and build AI/ML systems and pipelines that consistently...  ...that leverage our AI and engineering capabilitiesBuilding and deploying... 
    Senior
    Training
    Work at office
    Local area
    Remote work
    Flexible hours
    3 days per week
    1 day per week

    Oliver Wyman

    Winnipeg, MB
    1 hour ago
  • $75.5k - $125k per year

     ...Permanent Full Time  - We are hiring a Senior IT Business Systems Analyst. This is a hybrid role located in London, Toronto or Winnipeg. Reporting...  ...Information Systems, etc.) or equivalent combination of training and experience. Strong analytical and problem-solving... 
    Senior
    Training
    Permanent employment
    Full time
    Part time
    Internship
    Flexible hours

    Canada Life

    Winnipeg, MB
    9 days ago
  •  ...the role247 Labs is looking for a hands-on AI Engineer to design, develop, deploy, and maintain...  ...LLMs, machine learning models, retrieval systems, and agentic workflows.Evaluate business...  ...including meaningful hands-on production LLM development.Strong Python programming skills... 

    247 Labs

    Winnipeg, MB
    1 hour ago
  •  ...BrainyYack is seeking a Senior Python Developer to design and build scalable backend systems and AI-powered solutions. You will own backend development, APIs, and integrations...  ...software. The role emphasizes hands-on engineering, system design, and mentoring teammates while... 
    Senior
    Remote work

    Brainyyack

    Winnipeg, MB
    1 hour ago
  •  ...seeking a Member of Technical Staff (Post training team) to help advance model training...  ...Ideal candidates possess strong software engineering skills and should be proficient in Python...  ...a diverse team pushing the boundaries of AI research while enjoying a supportive work... 
    Training

    Cohere

    Winnipeg, MB
    1 hour ago
  • AMD is seeking a Senior Software Development Engineer to own the end-to-end model execution stack on AMD Instinct GPUs, covering training infrastructure at scale and high-performance inference serving. You will ship LLMs on real hardware, optimize GPU kernels, and ensure frontier... 
    Senior
    Training

    Advanced Micro Devices, Inc.

    Winnipeg, MB
    1 hour ago
  • $114.8k - $157.85k per year

     ...Job Description Senior Electrical Engineer Are you looking for a company that works on major engineering...  ...expertise in electrical building systems including power utilities & distribution...  ...apart. Project experience should include training centers, vehicle and fleet maintenance... 
    Senior
    Training
    Full time
    Work at office
    Worldwide
    Flexible hours

    AtkinsRéalis

    Winnipeg, MB
    2 days ago
  • $114.8k - $157.85k per year

     ...Job Description Senior Mechanical Engineer Are you looking for a company that works on major engineering...  ...expertise in mechanical building systems including HVAC, plumbing, fire protection...  .... Project experience should include training centers, vehicle and fleet maintenance... 
    Senior
    Training
    Full time
    Work at office
    Worldwide
    Flexible hours

    AtkinsRéalis

    Winnipeg, MB
    3 days ago
  •  ...Senior Automation Engineer (Project Manager) Saguenay-Lac-Saint-Jean, QC, Canada Type of position...  ...operating and maintaining industrial power systems. By joining our team, you’ll...  ...engineer or an equivalent combination of training and experience Experience in automation... 
    Senior
    Training
    Long term contract
    Temporary work
    Work at office
    Local area

    BBA Consultants

    Winnipeg, MB
    9 days ago
  •  ...high-performance computing, cloud, and AI. Whether you’re designing next-gen...  ...RoleThe AMD AI Group is looking for a Senior Software Development Engineer to own the end-to-end model execution...  ...stack on AMD Instinct GPUs - spanning training infrastructure at scale and high-... 
    Senior
    Training
    Internship
    Manual labor

    Advanced Micro Devices, Inc.

    Winnipeg, MB
    1 hour ago
  •  ...Salesforce, the leading AI CRM, seeks a Senior Technical Consultant to drive enterprise deployments and AI-driven delivery. You'll partner with clients and internal teams to define architectures, implement solutions, and manage release cycles across desktop, cloud, and mobile... 
    Senior

    Leadout Capital

    Winnipeg, MB
    9 days ago
  •  ...Chez Ubisoft, programmez des systèmes ML en C++ avec une expertise en moteurs d’inférence, tests, performance et architecture logicielle. Ce poste de programmeur principal C++ contribue à la feuille de route des applications d’apprentissage automatique. Vous développerez... 
    Senior
    Apprenticeship

    Ubisoft Entertainment Inc.

    Winnipeg, MB
    9 days ago
  •  ...designed for both desktop and mobile web rendering. Our back-end systems (some Java SpringBoot, some PHP) manage data processing,...  ...is driven by our core value of “we-are-we” and as a Senior AI Systems Engineer, you will work in an outcome-driven environment where collaboration... 
    Senior
    Training
    Work at office
    Flexible hours

    Benevity

    Winnipeg, MB
    1 hour ago
  •  ...has large-scale international engineering projects to offer. If you...  ...Structural Engineer (Intermediate to Senior) to support the continued...  ...of diverse opportunities for training and professional development...  ...'s infrastructure and energy systems. Together, with our industry... 
    Senior
    Training
    Internship
    Work at office
    Flexible hours

    AtkinsRéalis

    Winnipeg, MB
    7 hours ago
  • Are you a passionate and driven young engineer looking to make a real impact on your community? Stantec is a global engineering consulting firm dedicated to creating sustainable and innovative solutions. We are looking for enthusiastic individuals to join our team and embark... 
    Training
    Full time
    Local area

    Stantec

    Winnipeg, MB
    22 days ago
  •  ...a remote-first company and a people-first global community of senior engineers. Kake engineers are behind some of the world’s most innovative...  ...people stay, grow, and thrive (and love eating kake!). Senior AI/LLM Engineers with strong Python experience, skilled in designing... 
    Senior
    Remote work
    Flexible hours

    Kake

    Winnipeg, MB
    1 hour ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Senior Ai System Engineer - Llm Training System. Be the first to apply!