Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Senior Ai System Engineer - Llm Training System

$127k - $225k per year

Huawei Canada

Huawei Canada has an immediate permanent opening for a Senior Engineer.About the team:
The Computing Data Application Acceleration Lab aims to create a leading global data analytics platform organized into three specialized teams using innovative programming technologies. This team focuses on full-stack innovations, including software-hardware co-design and optimizing data efficiency at both the storage and runtime layers. This team also develops next-generation GPU architecture for gaming, cloud rendering, VR/AR, and Metaverse applications. One of the goals of this lab are to enhance algorithm performance and training efficiency across industries, fostering long-term competitiveness.About the job:
As a Senior System Engineer, you will drive core compute operator development and distributed scaling performance across foundation model pre-training and post-training workloads on Ascend NPUs. You will identify computational and communication bottlenecks, implement high-performance kernels using AscendC, Triton, and TileLang, and optimize distributed acceleration suites (MindSpeed / open-source training frameworks).Ascend High-Performance Kernel Development: Design, implement, and tune core operators (FlashAttention variants, FP8/MXFP8/W4A8 GEMM, Fused RMSNorm, dynamic MoE routing/dispatch) using AscendC, with explorations in Triton (Triton-Ascend backend) and TileLang.Pre-training & Post-training System Optimization: Architect, scale, and optimize distributed pre-training and RL post-training pipelines using MindSpeed (MindSpeed-LLM / MindSpeed-MM) alongside upstream frameworks (Megatron-LM / Megatron-Core, PyTorch FSDP, veRL, vLLM, Ray).Distributed Parallelism & Training Efficiency: Tune multi-dimensional hybrid parallelism (TP, PP, CP, EP, ZeRO/FSDP) and RL rollout/actor pipelining on Ascend NPU clusters, achieving optimal MFU and compute-communication overlap.Low-Precision & Numerical Parity: Implement low-precision training/inference schemes (FP8, MXFP8, mixed-precision), guaranteeing numerical stability, batch-invariance, and strict precision parity between training and serving runtimes.Profiling & Hardware-Aware Tuning: Conduct deep performance profiling on Ascend hardware using Ascend Profiler (msprof), msprobe, and PyTorch Profiler to eliminate pipeline bubbles, AICore/AIVector stalls, and HCCL communication overhead.Open Source & Ecosystem Enablement: Contribute high-performance operators, bug fixes, and acceleration features to open-source Ascend communities and upstream AI acceleration projects.The total target annual compensation ranges from $127,000 to $225,000 depending on education, experience, and demonstrated expertise.Job requirements
About the ideal candidate:
Bachelor’s, Master’s, or PhD in Computer Science, Computer Engineering, Electrical Engineering, or related technical disciplines, 3+ years of experience in high-performance computing (HPC), AI accelerator operator development, or distributed large model training systems.Hands-on Operator Programming: Solid development experience in AscendC / CANN, or extensive experience in CUDA C/C++ with a proven ability to transition to AscendC. Experience or strong interest in Triton or TileLang for accelerator backends.Distributed Frameworks: Deep practical experience with distributed pre-training and/or post-training frameworks, such as MindSpeed, Megatron-LM / Megatron-Core, PyTorch FSDP, veRL, or vLLM / DeepSpeed. Systems Programming: Strong programming proficiency in C/C++ and modern Python, with hands-on experience building PyTorch custom C++/CANN extensions.Performance Analysis: Proficiency in performance profiling and debugging tools on AI hardware (e.G., Ascend Profiler/msprof, PyTorch Profiler, or Nsight toolchains).Preferred / Standout Qualifications:
Direct experience porting or optimizing Triton or TileLang frontends/backends for non-CUDA accelerators (especially Ascend NPU).Deep understanding of Ascend NPU microarchitecture (Cube & Vector compute units, local memory hierarchy/UB, memory movement engines, and HCCL collective communication). Hands-on experience optimizing end-to-end MFU for massive MoE or dense foundation models on multi-node NPU/GPU clusters.Experience in low-precision numerical alignment (FP8, MXFP8, W4A8/W8A8) and eliminating training-inference discrepancy. Active open-source contributions to Ascend repos (MindSpeed, CANN ops, Ascend-vLLM) or upstream projects (Megatron-LM, Triton, TileLang, veRL).Additional Information:
Huawei Canada is committed to a fair, inclusive, and accessible recruitment process. If you require accommodation during any stage of the hiring process, please let us know and we will work with you to meet your needs.All applications for this position are reviewed directly by our hiring team, we do not use artificial intelligence tools to screen or select candidates.
#J-18808-Ljbffr
Vacancy posted 2 hours ago
Similar jobs that could be interesting for youBased on the Senior Ai System Engineer - Llm Training System in Winnipeg, MB vacancy
  •  ...Job Description Miratech is looking for a skilled Senior AI Engineer with Python, LangGraph and LangChain to join our US healthcare...  ...capabilities. Integrate AI agents and applications with enterprise systems, services, and APIs. Support modern AI integration... 
    Senior
    Full time
    Internship
    Remote work
    Relocation

    Miratech

    Winnipeg, MB
    10 days ago
  •  ...Drive AI innovation as a Senior AI Engineer in Mississauga, Canada. Craft Python-based AI applications using FastAPI or Flask while optimizing performance...  ...in software development. You will design and deploy AI and LLM applications, develop REST APIs, and collaborate with... 
    Senior

    Talentola

    Winnipeg, MB
    10 days ago
  •  ...An opportunity exists for a Bridge Engineer-in-Training to become part of our dynamic Transportation...  ..., specifying, and engineering bridge systems and products for a variety of transportation...  .... Support intermediate and senior team members on transportation projects... 
    Senior
    Training
    Full time
    Contract work
    For contractors
    Summer work
    Work at office
    Local area

    Stantec

    Winnipeg, MB
    10 days ago
  •  ...Are you a passionate and driven young engineer looking to make a real impact on your community...  ...under the guidance and mentorship of senior engineers.  Hands-On Experience: Conduct...  ...already registered) as an Engineering-In-Training with EGM.  Valid Canadian driver’s license... 
    Senior
    Training
    For contractors
    Local area
    Flexible hours

    Stantec

    Winnipeg, MB
    10 days ago
  •  ...helping you reach your potential through training and education. Community...  ...Coffee. Position Summary Join our AI team as an Intermediate AI Engineer, where you'll build and deploy AI applications...  ...AI solutions with enterprise systems, applications, APIs, and data... 
    Training

    Price Industries Limited

    Winnipeg, MB
    10 days ago
  •  ...Mining and Industrial Skills Training in Tanzania! Are you a skilled...  ...machining operations, pump systems, dewatering infrastructure, hydraulics...  ...so they can confidently train future technicians and tradespeople...  ...qualification) in Mechanical Engineering, Manufacturing Engineering,... 
    Training
    Permanent employment
    Apprenticeship
    Traineeship
    Local area
    Free visa

    Cuso International

    Winnipeg, MB
    2 hours ago
  • $89.91k - $113.73k per year

    The Engineering Service Business Unit focuses on power system software development, power system and substation consulting studies...  ...also be considered. * At the senior level,  at least 8 years of...  ...core competencies. Internationally-trained engineers may be considered if eligible... 
    Senior
    Full time
    Immediate start
    Remote work

    Powertech Labs

    Winnipeg, MB
    8 days ago
  •  ...We are committed to helping you reach your potential through training and education. • Community Involvement - Generously supporting...  ...Price Café offers free Tea and Coffee. Finishing Systems Engineering Assistant (8 Month Term – Winter 2027) The successful candidate... 
    Training
    For contractors

    Price Industries Limited

    Winnipeg, MB
    10 days ago
  •  ...Aerospace Engine Mechanic Training Program (GTRO) Winnipeg, Manitoba  Build an Aviation Career You’re Proud Of At StandardAero, we use our ingenuity and know-how to find solutions for the simple to the most complex challenges in aviation. Together, we get the job done... 
    Training
    Long term contract
    Temporary work
    Traineeship
    Second job

    StandardAero

    Winnipeg, MB
    23 days ago
  •  ...learning platforms at PointClickCare using Python, Java, and MLOps expertise. This role supports AI products across healthcare. As a Senior Machine Learning Systems Engineer, you will design, build, and operate platform capabilities for model development and deployment.... 
    Senior
    Training

    The Institute for Performance and Learning

    Winnipeg, MB
    10 days ago
  • $75.5k - $125k per year

     ...Permanent Full Time  - We are hiring a Senior IT Business Systems Analyst. This is a hybrid role located in London, Toronto or Winnipeg. Reporting...  ...Information Systems, etc.) or equivalent combination of training and experience. Strong analytical and problem-solving... 
    Senior
    Training
    Permanent employment
    Full time
    Part time
    Internship
    Flexible hours

    Canada Life

    Winnipeg, MB
    10 days ago
  • $114.8k - $157.85k per year

     ...Job Description Senior Electrical Engineer Are you looking for a company that works on major engineering...  ...expertise in electrical building systems including power utilities & distribution...  ...apart. Project experience should include training centers, vehicle and fleet maintenance... 
    Senior
    Training
    Full time
    Work at office
    Worldwide
    Flexible hours

    AtkinsRéalis

    Winnipeg, MB
    3 days ago
  • $114.8k - $157.85k per year

     ...Job Description Senior Mechanical Engineer Are you looking for a company that works on major engineering...  ...expertise in mechanical building systems including HVAC, plumbing, fire protection...  .... Project experience should include training centers, vehicle and fleet maintenance... 
    Senior
    Training
    Full time
    Work at office
    Worldwide
    Flexible hours

    AtkinsRéalis

    Winnipeg, MB
    4 days ago
  •  ...Senior Automation Engineer (Project Manager) Saguenay-Lac-Saint-Jean, QC, Canada Type of position...  ...operating and maintaining industrial power systems. By joining our team, you’ll...  ...engineer or an equivalent combination of training and experience Experience in automation... 
    Senior
    Training
    Long term contract
    Temporary work
    Work at office
    Local area

    BBA Consultants

    Winnipeg, MB
    10 days ago
  • $127k - $225k per year

     ...has an immediate permanent opening for a Senior Engineer. About the team: The Computing...  ...are to enhance algorithm performance and training efficiency across industries, fostering...  ...competitiveness. About the job: As a Senior System Engineer, you will drive core compute... 
    Senior
    Training
    Long term contract
    Permanent employment
    Internship
    Local area
    Immediate start

    Huawei Technologies Co.

    Winnipeg, MB
    2 hours ago
  •  ...Salesforce, the leading AI CRM, seeks a Senior Technical Consultant to drive enterprise deployments and AI-driven delivery. You'll partner with clients and internal teams to define architectures, implement solutions, and manage release cycles across desktop, cloud, and mobile... 
    Senior

    Leadout Capital

    Winnipeg, MB
    10 days ago
  •  ...Chez Ubisoft, programmez des systèmes ML en C++ avec une expertise en moteurs d’inférence, tests, performance et architecture logicielle. Ce poste de programmeur principal C++ contribue à la feuille de route des applications d’apprentissage automatique. Vous développerez... 
    Senior
    Apprenticeship

    Ubisoft Entertainment Inc.

    Winnipeg, MB
    10 days ago
  •  ...has large-scale international engineering projects to offer. If you...  ...Structural Engineer (Intermediate to Senior) to support the continued...  ...of diverse opportunities for training and professional development...  ...'s infrastructure and energy systems. Together, with our industry... 
    Senior
    Training
    Full time
    Internship
    Work at office
    Flexible hours

    AtkinsRéalis

    Winnipeg, MB
    1 day ago
  • $18 - $22 per hour

     ...in ETC/EAC forecasts. Apply AI and automation tools to controls...  ...cost reconciliation to source systems. Review and validate contractor...  ...’s degree or diploma in Engineering, Construction Management, Quantity...  ...Power BI/Power Query, Python, or LLM‑based tools), with specific... 
    Senior
    Hourly pay
    Permanent employment
    Full time
    Contract work
    Temporary work
    Part time
    For contractors
    Work at office
    Flexible hours
    3 days per week

    Trans Mountain

    Winnipeg, MB
    10 days ago
  •  ...looking for an Operations Research Engineer to join the Workforce...  ...implement the schedule optimization engine using OR-Tools/CP-SAT.  Own the...  ...and decision-support systems.  Experience with cloud platforms...  ....  Practical experience with AI-assisted software development or... 
    Senior
    Internship
    Remote work
    Relocation

    Miratech

    Winnipeg, MB
    8 days ago
  •  ...changes of the IT industry. Summary: The Consultant, ICT Systems Engineering at Epic is responsible for timely and effective resolutions to...  ...point for all Managed service issues Provide mentorship/training to the rest of the team Qualifications: Minimum of 5+ years... 
    Training
    Full time
    Contract work
    Work at office
    Remote work
    Flexible hours
    3 days per week

    Bell Canada

    Winnipeg, MB
    9 days ago
  •  ...helping you reach your potential through training and education. Community...  ...Café offers free Tea and Coffee. Senior Manufacturing Engineering Specialist - Critical Environments...  ...capital investments, ensuring production systems remain reliable and aligned with business... 
    Senior
    Training

    Price Industries Limited

    Winnipeg, MB
    10 days ago
  •  ...Canada to recruit experienced mid-senior to senior-level engineering professionals . These opportunities...  ...relevant engineering software, tools, and systems Benefits Extended Healthcare...  ...& Vision) RPP - Group RRSP Paid Time Off Benefits Training & Development... 
    Senior
    Training
    Full time
    For contractors

    Hire Resolve.com

    Winnipeg, MB
    more than 2 months ago
  •  ...potential in an environment that’s engineered for people. At CIMA+, we care...  ...career by joining us as a  Senior Overhead Transmission Lines...  ...design /analysis software and CAD systems, along with familiarity of...  ...relevant experience ~ Tailored training to improve your existing... 
    Senior
    Training
    Permanent employment
    Full time
    Internship
    Work at office
    Flexible hours
    2 days per week

    CIMA+

    Winnipeg, MB
    7 days ago
  • $110k per year

     ...retail. Interconnected Commerce is an AI-native telecom commerce platform that acts as a system of intelligence. It replaces...  ...quality standards, and creating engineering patterns that let analytics, product...  .... About the Role: The Senior Data Engineer will design, build,... 
    Senior
    Training
    Full time
    Work at office
    Remote work
    Flexible hours

    iQmetrix

    Winnipeg, MB
    a month ago
  • $46.5k per year

     ...Position: Temporary Manager in Training Hours: 40 Weekly Deadline to apply: Tuesday,...  ...service models Ability to develop and train staff through positive coaching and feedback...  ...excel, Microsoft suit, and adaptable to other systems as required Basic HR skills in hiring,... 
    Training
    Temporary work
    Flexible hours
    Afternoon shift

    Internal Job Board

    Winnipeg, MB
    4 days ago
  •  ...Pinterest is seeking a Senior Data Scientist to shape the future of people-facing and business...  ...and algorithms to solve complex engineering challenges at scale, collaborating with Product...  ...and prototype analysis pipelines, leverage AI to accelerate execution, and communicate insights... 
    Senior

    Pinterest

    Winnipeg, MB
    10 days ago
  •  ...We are looking for a Senior Routing Engine Developer to design and build the next generation interaction...  ...distributed architectures, event-driven systems, and cloud-native technologies,...  ...architecture. Practical experience with AI-assisted software development or AI-enabled... 
    Senior
    Internship
    Remote work
    Relocation

    Miratech

    Winnipeg, MB
    10 days ago
  •  ...experience is not required. Comprehensive training modules, manufacturer resources, and collaborative...  ..., complex third-party reimbursement systems, and unfamiliar medication categories....  ...within our database. Although Randstad uses AI as a tool, candidate selection and hiring... 
    Training
    Long term contract
    Permanent employment
    Full time
    Temporary work
    Direct hire
    Immediate start
    Monday to friday
    Shift work
    Afternoon shift
    Weekday work

    Randstad Canada

    Winnipeg, MB
    10 days ago
  • $130k - $165k per year

     ...of Quanta Services, we provide engineering and design services across...  ...re looking for an experienced Senior Transmission Line Design Engineer...  ...and underground transmission systems ~ Strong knowledge of applicable...  ...may use technology, including AI-enabled tools, to support... 
    Senior
    Long term contract
    Full time
    For contractors
    Internship

    Phasor Engineering Inc

    Winnipeg, MB
    more than 2 months ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Senior Ai System Engineer - Llm Training System. Be the first to apply!