Senior Software Engineer, GPU Cluster Infrastructure

• In frontier AI research, working out the infrastructure is often part of the science. You'd work directly with researchers and other engineers to keep our large-scale experiments performant and fault-tolerant.
• Operate the Kubernetes GPU fleet day to day. You handle node lifecycle, upgrades, driver and image rollouts, staged changes with safe rollback, and capacity planning.
• Own batch scheduling and multi-tenancy, including queues, quotas, priorities, preemption, gang scheduling, and fair share across research teams.
• Design and run the storage under the fleet, from high-performance shared filesystems for datasets and checkpoints to object storage tiers, quotas, and backups.
• Keep multi-node training runs fault-tolerant. You own node health and automated draining, debug NCCL and fabric problems, track down stragglers and flaky GPUs, and build the checkpoint and restart patterns.
• Harden the platform, covering identity and access, network policy, secrets, workload isolation, and sandboxing for the AI agents that run on the cluster.
• Bring new capacity online. You acceptance-test providers on fabric, NCCL, and storage throughput, hold them to their SLAs, and integrate new clusters into the platform with infrastructure as code.
• Work directly with research teams on their infrastructure problems and turn the recurring ones into platform fixes. Share the on-call rotation, runbooks, and postmortems.
• You have 3+ years in systems or infrastructure engineering on production Linux, running GPU, HPC, or large-scale batch platforms, and you've owned at least one system from design through operation.
• You've run production Kubernetes for GPU workloads with a batch layer on top (Slurm, Kueue, Volcano, or similar), including quotas, priority and preemption, and node health.
• You've owned infrastructure as code and observability for a production fleet, provisioning with Terraform or Ansible, deploying with Helm and ArgoCD, and monitoring with Prometheus, or their equivalents.
• You're a strong programmer in at least one language that infrastructure is commonly written in, such as Python, Go, Rust, or C++, and your automation and services are maintained as shared code.
• You write clearly for engineers, researchers, and providers, whether it's a design doc, an incident summary, or an escalation.
• If you meet most of this and not all of it, we encourage you to apply anyway.
• Real depth in one or more of these makes you a compelling candidate:
• Distributed training infrastructure: multi-node PyTorch and NCCL debugging, the NVIDIA node stack (drivers, GPU Operator, DCGM), InfiniBand or RoCE fabrics, topology-aware placement.
• Distributed storage: VAST, Weka, Lustre, Ceph, or object storage at scale; checkpoint I/O.
• Cluster security: admission control, RBAC, node and container hardening, sandboxed runtimes (gVisor, Kata, Firecracker), and isolating autonomous agents on shared infrastructure.
• Scheduler internals: Kubernetes scheduler plugins or custom controllers, gang scheduling, fair-share and quota, and the utilization, fairness, and latency tradeoffs between them.
• Multi-provider platforms: scheduling and storage across clusters at different providers so users see one system, including clusters with no shared network and uneven data locality.
• Paid Leave - Paid Bereavement, Family, Medical and Pregnancy Disability Leave
• WFH Stipend & Equipment - Work computer and stipend provided for eligible employees
• Catered Meals (Berkeley Office Only) - Catered lunches and dinners on workdays at our office
• (These benefits are currently only offered to full-time employees based in the US, we use an EOR for our international employees who have their own set of benefits for Health, Paid Leave, etc.)

None flagged.