Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Senior Ai System Engineer - Llm Training System

$127k - $225k per year

Huawei Technologies Co.

Huawei Canada has an immediate permanent opening for a Senior Engineer.
About the team:The Computing Data Application Acceleration Lab aims to create a leading global data analytics platform organized into three specialized teams using innovative programming technologies. This team focuses on full-stack innovations, including software-hardware co-design and optimizing data efficiency at both the storage and runtime layers. This team also develops next-generation GPU architecture for gaming, cloud rendering, VR/AR, and Metaverse applications. One of the goals of this lab are to enhance algorithm performance and training efficiency across industries, fostering long-term competitiveness.
About the job:As a Senior System Engineer, you will drive core compute operator development and distributed scaling performance across foundation model pre-training and post-training workloads on Ascend NPUs. You will identify computational and communication bottlenecks, implement high-performance kernels using AscendC, Triton, and TileLang, and optimize distributed acceleration suites (MindSpeed / open-source training frameworks).
Ascend High-Performance Kernel Development: Design, implement, and tune core operators (FlashAttention variants, FP8/MXFP8/W4A8 GEMM, Fused RMSNorm, dynamic MoE routing/dispatch) using AscendC, with explorations in Triton (Triton-Ascend backend) and TileLang.
Pre-training & Post-training System Optimization: Architect, scale, and optimize distributed pre-training and RL post-training pipelines using MindSpeed (MindSpeed-LLM / MindSpeed-MM) alongside upstream frameworks (Megatron-LM / Megatron-Core, PyTorch FSDP, veRL, vLLM, Ray).
Distributed Parallelism & Training Efficiency: Tune multi-dimensional hybrid parallelism (TP, PP, CP, EP, ZeRO/FSDP) and RL rollout/actor pipelining on Ascend NPU clusters, achieving optimal MFU and compute-communication overlap.
Low-Precision & Numerical Parity: Implement low-precision training/inference schemes (FP8, MXFP8, mixed-precision), guaranteeing numerical stability, batch-invariance, and strict precision parity between training and serving runtimes.
Profiling & Hardware-Aware Tuning: Conduct deep performance profiling on Ascend hardware using Ascend Profiler (msprof), msprobe, and PyTorch Profiler to eliminate pipeline bubbles, AICore/AIVector stalls, and HCCL communication overhead.
Open Source & Ecosystem Enablement: Contribute high-performance operators, bug fixes, and acceleration features to open-source Ascend communities and upstream AI acceleration projects.
The total target annual compensation ranges from $127,000 to $225,000 depending on education, experience, and demonstrated expertise.
About the ideal candidate:
Bachelor’s, Master’s, or PhD in Computer Science, Computer Engineering, Electrical Engineering, or related technical disciplines, 3+ years of experience in high-performance computing (HPC), AI accelerator operator development, or distributed large model training systems.
Hands-on Operator Programming: Solid development experience in AscendC / CANN, or extensive experience in CUDA C/C++ with a proven ability to transition to AscendC. Experience or strong interest in Triton or TileLang for accelerator backends.
Distributed Frameworks: Deep practical experience with distributed pre-training and/or post-training frameworks, such as MindSpeed, Megatron-LM / Megatron-Core, PyTorch FSDP, veRL, or vLLM / DeepSpeed. Systems Programming: Strong programming proficiency in C/C++ and modern Python, with hands-on experience building PyTorch custom C++/CANN extensions.
Performance Analysis: Proficiency in performance profiling and debugging tools on AI hardware (e.G., Ascend Profiler/msprof, PyTorch Profiler, or Nsight toolchains).
Preferred / Standout Qualifications:
Direct experience porting or optimizing Triton or TileLang frontends/backends for non-CUDA accelerators (especially Ascend NPU).
Deep understanding of Ascend NPU microarchitecture (Cube & Vector compute units, local memory hierarchy/UB, memory movement engines, and HCCL collective communication). Hands-on experience optimizing end-to-end MFU for massive MoE or dense foundation models on multi-node NPU/GPU clusters.
Experience in low-precision numerical alignment (FP8, MXFP8, W4A8/W8A8) and eliminating training-inference discrepancy. Active open-source contributions to Ascend repos (MindSpeed, CANN ops, Ascend-vLLM) or upstream projects (Megatron-LM, Triton, TileLang, veRL).
Additional Information:Huawei Canada is committed to a fair, inclusive, and accessible recruitment process. If you require accommodation during any stage of the hiring process, please let us know and we will work with you to meet your needs.
All applications for this position are reviewed directly by our hiring team,
we do not use artificial intelligence tools
to screen or select candidates.
#J-18808-Ljbffr
Vacancy posted 52 minutes ago
Similar jobs that could be interesting for youBased on the Senior Ai System Engineer - Llm Training System in Winnipeg, MB vacancy
  • Lead LLM systems at Axya, working hands-on with Python, retrieval, agent workflows, and evaluation in a 100% remote environment.As Lead AI Engineer, you will own Axya’s AI architecture and technical direction. You will report to the CTO while partnering with Product, Customer... 
    Suggested
    Remote work

    Axya

    Winnipeg, MB
    47 minutes ago
  • Design RAG systems and LLM features for Wiser Solutions remotely, using embeddings and retrieval to solve real customer problems.This Senior Machine Learning Engineer builds AI applications that reach production across product matching, classification, and extraction. You will... 
    Senior
    Remote work

    Wiser Solutions

    Winnipeg, MB
    48 minutes ago
  • $195k - $285k per year

     ...breakthrough outcomes for their toughest AI challenges. We blend the...  ...across industries. As a Senior Lead AI Developer, you are...  ...teams design and build AI/ML systems and pipelines that consistently...  ...that leverage our AI and engineering capabilitiesBuilding and deploying... 
    Senior
    Training
    Work at office
    Local area
    Remote work
    Flexible hours
    3 days per week
    1 day per week

    Oliver Wyman

    Winnipeg, MB
    54 minutes ago
  • Get notified about new Director Of Software Engineering jobs in United States . 7,000+ Director Of Software Engineering Jobs in United States...  ...Engineering – Acute Devices Director, Software Engineering - AI Platform and Full Stack Engineering - Office-based in Minneapolis... 
    Suggested
    Work at office
    Remote work

    Talent Seed

    Winnipeg, MB
    53 minutes ago
  •  ...the role247 Labs is looking for a hands-on AI Engineer to design, develop, deploy, and maintain...  ...LLMs, machine learning models, retrieval systems, and agentic workflows.Evaluate business...  ...including meaningful hands-on production LLM development.Strong Python programming skills... 
    Suggested

    247 Labs

    Winnipeg, MB
    49 minutes ago
  • ZS is seeking an LLM Engineer to design and build scalable software systems leveraging Python, LLM patterns, and cloud platforms. You will translate business requirements...  ...pipelines, and implement guardrails for responsible AI, joining a collaborative, client-focused team. #J-... 

    ZS

    Winnipeg, MB
    53 minutes ago
  •  ...Job Description Miratech is looking for a skilled Senior AI Engineer with Python, LangGraph and LangChain to join our US healthcare...  ...capabilities. Integrate AI agents and applications with enterprise systems, services, and APIs. Support modern AI integration... 
    Senior
    Full time
    Internship
    Remote work
    Relocation

    Miratech

    Winnipeg, MB
    5 days ago
  •  ...designed for both desktop and mobile web rendering. Our back-end systems (some Java SpringBoot, some PHP) manage data processing,...  ...is driven by our core value of “we-are-we” and as a Senior AI Systems Engineer, you will work in an outcome-driven environment where collaboration... 
    Senior
    Training
    Work at office
    Flexible hours

    Benevity

    Winnipeg, MB
    55 minutes ago
  •  ...a remote-first company and a people-first global community of senior engineers. Kake engineers are behind some of the world’s most innovative...  ...people stay, grow, and thrive (and love eating kake!). Senior AI/LLM Engineers with strong Python experience, skilled in designing... 
    Senior
    Remote work
    Flexible hours

    Kake

    Winnipeg, MB
    53 minutes ago
  •  ...Drive AI innovation as a Senior AI Engineer in Mississauga, Canada. Craft Python-based AI applications using FastAPI or Flask while optimizing performance...  ...in software development. You will design and deploy AI and LLM applications, develop REST APIs, and collaborate with... 
    Senior

    Talentola

    Winnipeg, MB
    6 days ago
  • Le Développeur Java Senior chez DELAN - IT Talent Solutions conçoit et développe des fonctionnalités IA générative (LLM) dans une plateforme SaaS en Java. Vous bâtissez des API REST robustes...  ...d expérience, vous maîtrisez MCP, Vertex AI, GCP et Kubernetes, et vous collaborez au... 
    Senior

    DELAN - IT Talent Solutions

    Winnipeg, MB
    50 minutes ago
  • Engage in cutting-edge AI language model development as a Senior Software Engineer. With Ethos, work remotely and leverage your expertise for impactful software engineering...  ...outputs across essential software workflows. Your system design and algorithm skills will be pivotal in... 
    Senior
    Training
    Remote work
    Flexible hours

    Ethos

    Winnipeg, MB
    48 minutes ago
  • Cohere is a leading security-first enterprise AI company building foundation AI models and end-to-end products. We are hiring for a role that builds and owns pipelines for post-training data used to train large language models. You will collaborate with researchers and annotation... 
    Training
    Remote work
    Worldwide

    Cohere

    Winnipeg, MB
    48 minutes ago
  • Spellbook is the most comprehensive AI copilot for transactional...  ...The RoleYou’ll build the backend systems that power Spellbook’s product...  ...on, high-ownership role for an engineer who cares about product outcomes...  ..., MongoDB, Docker, AWS, CDK, LLM providers (OpenAI, Anthropic, and... 
    Senior
    Contract work
    Flexible hours

    Spellbook

    Winnipeg, MB
    50 minutes ago
  • Oliver Wyman Quotient is hiring a Senior Lead AI Developer to manage technical projects and guide teams in designing and building AI/ML systems and production-ready pipelines. You will work...  ...exploring data, shaping proposals, deploying LLM-based agents, and advancing state-of-the... 
    Senior

    Cim

    Winnipeg, MB
    52 minutes ago
  • Extend Tempo for AI and agent workloads through structured APIs, observability systems, and scalable engineering. Work remotely with modern coding assistants.As a Senior Engineer on Tempo, you will build capabilities for Grafana’s AI assistant, external integrators, and agent... 
    Senior
    Remote work

    Grafana Labs

    Winnipeg, MB
    50 minutes ago
  • Shape AI retrieval systems at lululemon as a Senior Software Engineer, using RAG, GraphRAG, and semantic search in onsite work. Influence enterprise technical direction.Within the Enterprise Data & AI team, this senior engineer builds context-rich systems for LLM-powered applications... 
    Senior

    Lululemon Athletica Inc.

    Winnipeg, MB
    53 minutes ago
  •  ...An opportunity exists for a Bridge Engineer-in-Training to become part of our dynamic Transportation...  ..., specifying, and engineering bridge systems and products for a variety of transportation...  .... Support intermediate and senior team members on transportation projects... 
    Senior
    Training
    Full time
    Contract work
    For contractors
    Summer work
    Work at office
    Local area

    Stantec

    Winnipeg, MB
    6 days ago
  • Build advanced language systems as a Senior AI Engineer in Kimberley, applying NLP and LLM expertise within a hybrid work model. Develop scalable solutions that understand, generate, and process human language at scale.This senior position focuses on developing and deploying... 
    Senior
    Training

    Placements24

    Winnipeg, MB
    53 minutes ago
  • FullStack is seeking a Senior AI Engineer to join our talent network and work with U.S. clients on flexible, project-based development. You will...  ...degree, and hands-on NVIDIA stack experience (NIM, NeMo, TensorRT-LLM). Hybrid work with international collaboration is expected.... 
    Senior
    Flexible hours

    FullStack

    Winnipeg, MB
    54 minutes ago
  • Python and LLM engineering define this remote Senior AI/LLM Engineer position with Kake, building production-ready applications and systems at scale.Kake seeks senior engineers who design, build, and productionize AI systems. You will work across Python, FastAPI, LAG pipelines... 
    Senior
    Remote work

    Kake

    Winnipeg, MB
    55 minutes ago
  • Develop AI-assisted backend features for Jeeves from a full-time remote setting, combining LLM integrations with distributed financial systems. Collaborate across global engineering and data teams.In this Senior Backend Engineer position, you will connect experimental AI work... 
    Senior
    Full time
    Remote work

    Jeeves

    Winnipeg, MB
    47 minutes ago
  •  ...WSP seeks an AI Developer (LLM & Agentic Systems) to design and build AI-enabled applications across mining, resources, infrastructure, and environmental domains in Canada. You will integrate LLMs, data pipelines, and knowledge graphs to enable advanced reasoning and decision... 
    Remote work

    WSP Global Inc.

    Winnipeg, MB
    48 minutes ago
  • Remote anywhere in Canada, this machine learning systems engineer benchmarks and tunes production LLM serving across AWS Trainium2 and Trainium3. Python and PyTorch skills are central.Capgemini Engineering seeks a hands-on engineer focused on accelerator performance and customer... 
    Remote work

    Capgemini Engineering

    Winnipeg, MB
    51 minutes ago
  • Smart IT Frame LLC is seeking an experienced software engineer to design, build, and deploy Python-based AI/LLM applications using FastAPI or Flask. The role includes developing REST APIs, scalable enterprise solutions, and AI-powered data analytics platforms. You will optimize... 
    Senior

    Smart IT Frame LLC

    Winnipeg, MB
    48 minutes ago
  • Drive model integration projects as a Senior Engineer focused on AI LLMs. Engage in end-to-end processes from architecture translation to runtime optimizations...  ...the integration of advanced AI technologies on specialized systems. You will combine your expertise in deep learning... 
    Senior

    Cerebras

    Winnipeg, MB
    50 minutes ago
  • Build reliable LLM applications as a Senior AI Scientist with CIBC’s artificial intelligence centre of excellence in Toronto.This hybrid opportunity...  ..., production-ready capabilities. You will apply prompt engineering, retrieval-augmented generation, and agentic workflows alongside... 
    Senior

    CIBC

    Winnipeg, MB
    54 minutes ago
  •  ...Canada is hiring a Principal Associate, AI Engineer to join the IFX team in Toronto. You will...  ...how associates interact with Capital One's systems. You will design, develop, test, deploy,...  ...components, including foundation model training, LLM inference, and observability, using... 
    Senior
    Training

    Capital One

    Winnipeg, MB
    54 minutes ago
  •  ...Are you a passionate and driven young engineer looking to make a real impact on your community...  ...under the guidance and mentorship of senior engineers.  Hands-On Experience: Conduct...  ...already registered) as an Engineering-In-Training with EGM.  Valid Canadian driver’s license... 
    Senior
    Training
    For contractors
    Local area
    Flexible hours

    Stantec

    Winnipeg, MB
    6 days ago
  • Shape Cloudbeds hospitality products through customer-facing AI systems, evaluation, and observability as part of a completely remote global team.On the Core AI Squad, this Senior Software Engineer builds production capabilities for intelligent experiences across Cloudbeds.... 
    Senior
    Remote work
    Flexible hours

    Cloudbeds

    Winnipeg, MB
    55 minutes ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Senior Ai System Engineer - Llm Training System. Be the first to apply!