Cluster Site Reliability Engineer

Cluster & SRE

Cluster Site Reliability Engineer

Location: On-site

Type: Full-time

Team: Cluster & SRE

Preferred region: Cologix region of choice (Toronto, Montréal, Columbus, Vancouver, Ashburn)

Stack

• Linux
• InfiniBand (NDR / XDR)
• NCCL / RCCL
• Kubernetes (host‑level)
• Terraform
• Prometheus / Grafana
• Go or Python

Responsibilities

• Bring up new B200 / B300 / MI300X racks: cabling, ToR config, NCCL/RCCL all‑reduce validation, MFU baseline tests.
• Drive InfiniBand fabric to spec — NDR / XDR depending on the rack — and chase residual bit‑error budget down to zero.
• Run capacity planning across seven regions: forecast demand, model power and thermal headroom, work with procurement on lead times.
• Own the regional incident response. P1 incidents page within fifteen minutes; resolution target is four hours.
• Build and maintain the bring‑up runbook so the second hire after you can do their first rack solo.
• Carry the global pager about one week per six, alongside runtime and customer engineering.

Qualifications

• Five or more years operating large compute clusters — supercomputing centers, hyperscaler infrastructure, or HPC at a national lab counts.
• Deep InfiniBand and Ethernet RoCE experience: subnet manager tuning, fabric debugging, lossless networking.
• Strong Linux fundamentals: kernel parameters, NUMA topology, kernel cgroup limits.
• Comfort writing Go or Python for tooling. Not strict which language.
• Calm under load; named person on a $50M‑ARR account when something goes wrong.

Nice to Have

• DGX H110 / H200 / B200 bring‑up history.
• Experience with NVIDIA Bright / Base Command Manager.
• Bare‑metal provisioning systems: Tinkerbell, MAAS, Razor, or in‑house equivalents.
• Procurement / vendor management experience.

Compensation

• Base: $210,000 – $340,000 USD (US Cologix regions) / equivalent in CA.
• Equity: Meaningful early‑stage equity, refreshed on tenure milestones.
• On‑site pay differential at Cologix regions outside SF / NYC / Bay Area is +5–10% to compensate for travel.

Equal Opportunity

We hire on the work. Race, gender, age, nationality, religion, sexual orientation, disability, and veteran status do not factor into our decisions. We sponsor visas for senior roles in the US, UK, and EU — bring it up on the manager call. Need an accommodation for the interview process? Mention it in your application or write View email address on ca.talent.com.