Senior GPU Infrastructure Engineer

Senior GPU Infrastructure Engineer About Civo: Civo is revolutionising the cloud industry by providing developers and businesses with cutting-edge, developer-friendly cloud solutions. With a focus on simplicity, performance, and reliability, Civo enables its clients to unleash the full potential of their projects and scale with ease. As a company committed to excellence and innovation, Civo values diversity, collaboration, and continuous improvement. About the Role: As a Senior GPU Infrastructure Engineer, you will serve as the technical authority owning the reliability, performance, and automated orchestration of Civo’s AI and GPU platform layer. You will bridge the boundary between deep server hardware/firmware telemetry, high-bandwidth interconnect fabrics, and cloud-native orchestrators. In this role, you will lead complex escalation responses, build platform observability, drive blameless post-mortems, and engineer automated tooling to eliminate hardware entropy at scale. You will collaborate closely with hardware OEMs, NVIDIA engineering, and Civo’s core software teams to ensure maximum cluster uptime for large-scale training and inference workloads. Responsibilities: GPU Fleet Telemetry & Hardware SRE Deep Root-Cause Analysis: Act as the highest internal technical escalation point for persistent or intermittent hardware, PCIe bus, GPU memory (ECC double-bit errors), and XID driver faults across the fleet. Automated Diagnostics: Build and maintain automated pre-job and post-job health check frameworks utilizing NVIDIA DCGM, NCCL test suites, and custom telemetry agents. Firmware & Driver Lifecycle: Manage automated rollout pipelines for GPU VBIOS, system BIOS, BMC firmware, and NVIDIA CUDA driver stacks across live production clusters. High-Performance Networking & Storage Integration Fabric Engineering Support: Diagnose performance bottlenecks, packet drops, and link degradation across InfiniBand (Quantum-2/X800) and RoCEv2 (Spectrum-X) interconnects. Parallel Storage SRE: Optimize data paths for GPUDirect Storage (GDS) and parallel file systems (VAST Data) to prevent GPU starvation during multi-node training runs. Orchestration & Platform Automation Kubernetes & Slurm GPU Stack: Maintain and tune the NVIDIA GPU Operator, Network Operator, KubeFlow, and Slurm plugins across bare-metal and containerized environments. Automated Remediations: Design automated node-cordoning, pod eviction, and repair workflows triggered by real-time hardware telemetry alerts. Infrastructure-as-Code: Drive full-stack automation using Terraform, Ansible, and Python to streamline fleet expansion and region bring-ups. Incident Management & Operational Excellence Major Incident Response: Lead war-room technical resolution during critical platform outages and manage complex vendor escalations with NVIDIA and OEM support teams. Blameless Post-Mortems: Conduct deep-dive post-incident reviews, author detailed root-cause documents, and implement preventative engineering fixes. Key Results/Objectives: Fleet Availability: Maintain >= 99.9% platform availability across production GPU compute nodes. Automated Node Recovery: Automate detection and cordoning of degraded hardware to reduce Mean Time to Repair (MTTR) by 40%. Zero-Downtime Upgrades: Execute driver, firmware, and orchestration updates with zero unannounced impact to customer workloads. Requirements: Experience & Core Qualifications 5+ years of experience in SRE, Systems Engineering, or Infrastructure Engineering supporting high-density, production-scale Linux environments. 3+ years of hands-on experience managing and troubleshooting data center class NVIDIA GPUs (H100, B200, or newer) in distributed environments. Technical Expertise NVIDIA Stack: Deep expertise in CUDA runtime drivers, DCGM metrics, NVLink/NVSwitch topologies, and troubleshooting XID errors. Orchestration: Advanced proficiency in Kubernetes administration (CKA level) including CNI, Device Plugins, and Slurm workload manager. High-Performance Fabrics: In-depth knowledge of InfiniBand subnet management, RoCEv2 lossless Ethernet mechanisms (PFC, ECN), and NCCL communication tuning. Automation & Scripting: Strong programming skills in Python, Go, or Bash, along with hands-on experience using Terraform and Ansible. Observability: Experience deploying and scaling Prometheus, Grafana, and vector-based telemetry stacks for hardware monitoring. Soft Skills Strong technical leadership during high-pressure incidents. Excellent technical writing skills for post-mortems, architectural runbooks, and cross-team documentation. Nice to Have: NVIDIA Certified Professional certifications in AI Infrastructure or InfiniBand (NCP-AII, NCP-IB). Certified Kubernetes Administrator (CKA) or Security Specialist (CKS). Experience with liquid-cooled rack architectures (Direct-to-Chip, CDU setups). Why Join Civo? Competitive compensation and benefits package. 4-day work week company (unless attending an event). Uncapped holiday. Remote work environment with flexibility and autonomy. Collaborative and inclusive culture that values diversity and creativity. Opportunity to work with a dynamic and innovative team in the fast-growing cloud industry.