Job description

Infrastructure Engineer – Managed Inference Location: San Francisco, CA (Onsite, 5 days/week) Employment Type: Full-Time Compensation: 350,000–450,000 Base Salary + Competitive Equity About the Opportunity We are partnering with an innovative AI company building high-performance infrastructure for large-scale generative AI applications. The team develops and operates advanced inference systems designed to deliver exceptionally fast, reliable model serving across multiple cloud providers and hardware architectures. We are seeking a Staff-level or higher Infrastructure Engineer with deep, hands-on experience operating production LLM inference infrastructure. This role focuses on scaling highly available Kubernetes environments, optimizing inference serving systems, and ensuring reliable performance across GPU and accelerator platforms. The ideal candidate is a highly technical individual contributor who takes direct ownership of production infrastructure, troubleshoots complex systems, and works closely with engineering teams and customers to solve challenging infrastructure problems. What You'll Do • Design, operate, and scale highly available Kubernetes clusters across AWS, GCP, and specialized GPU cloud providers to support inference workloads exceeding 1,000 tokens per second. • Own the production inference serving layer, including hands-on debugging of NVIDIA drivers, CUDA, NCCL, node health, and network fabric. • Collaborate with model optimization engineers to ensure performance improvements translate reliably to production environments. • Extend deployment tooling to support consistent model deployment across NVIDIA GPUs, AWS Trainium, and Google TPUs. • Build and maintain observability systems tracking time-to-first-token, inter-token latency, throughput, and availability across models, hardware, and regions. • Lead reliability engineering efforts, including monitoring, alerting, automated failover, self-healing infrastructure, and intelligent traffic routing. • Troubleshoot production infrastructure issues directly, working with technical teams and customers to identify and resolve complex challenges. Required Qualifications • 5+ years of infrastructure engineering experience , currently operating at Staff Engineer level or above in a hands-on individual contributor role. • Direct, day-to-day experience with production LLM inference infrastructure , including building, deploying, operating, or troubleshooting inference serving systems. • Extensive experience operating production Kubernetes at scale across multiple cloud providers , particularly AWS and GCP. • Hands-on expertise debugging NVIDIA GPU and accelerator systems , including drivers, CUDA, NCCL, node health, and network-related issues. • Practical experience with at least one inference serving framework, such as vLLM, SGLang, TensorRT-LLM, or NVIDIA Dynamo . • Experience with high-performance networking technologies such as InfiniBand, RDMA/EFA, or MPI . • Strong background in observability and reliability engineering for latency-sensitive production services, including metrics, alerting, and automated failover. • Ability to independently diagnose and resolve complex production infrastructure issues. • Strong communication and problem-solving skills, including the ability to collaborate directly with customers and technical stakeholders. • Ability to work onsite in San Francisco five days per week. Preferred Qualifications • Experience building or operating production-scale inference platforms at AI infrastructure companies, model-serving providers, or AI research organizations. • Experience deploying inference workloads across multiple accelerator architectures, including NVIDIA GPUs, AWS Trainium, and Google TPUs. • Contributions to open-source inference frameworks such as vLLM or SGLang. • Experience supporting high-throughput, low-latency AI applications in production environments. Technical Environment Kubernetes, AWS, GCP, NVIDIA GPUs, CUDA, NCCL, InfiniBand, RDMA/EFA, MPI, vLLM, SGLang, TensorRT-LLM, NVIDIA Dynamo, AWS Trainium, Google TPUs, PyTorch, JAX, Docker, Terraform, Prometheus, and Grafana. Compensation & Location • Base Salary: 350,000–450,000, with flexibility above this range for exceptionally qualified Staff+ candidates. • Equity: Competitive equity package. • Location: San Francisco, California. • Work Arrangement: Full-time, onsite five days per week. • Relocation: May be considered for highly qualified candidates. • Work Authorization: Open to eligible visa transfers, including H-1B and OPT. Important Considerations This position requires substantial, direct experience with production LLM inference infrastructure. Candidates whose backgrounds are primarily in general-purpose infrastructure, data engineering, recommendation systems, or inference-adjacent technologies without hands-on inference serving experience are unlikely to meet the requirements. The team is specifically seeking an experienced Staff-level or higher individual contributor who is already operating at this level and can take immediate ownership of complex production systems.

Requirements

Must-haves

Infrastructure Engineer – Managed Inference
Location: San Francisco, CA (Onsite, 5 days/week)
Employment Type: Full-Time
Compensation: 350,000–450,000 Base Salary + Competitive Equity
Design, operate, and scale highly available Kubernetes clusters across AWS, GCP, and specialized GPU cloud providers to support inference workloads exceeding 1,000 tokens per second.
Own the production inference serving layer, including hands-on debugging of NVIDIA drivers, CUDA, NCCL, node health, and network fabric.
Collaborate with model optimization engineers to ensure performance improvements translate reliably to production environments.
Extend deployment tooling to support consistent model deployment across NVIDIA GPUs, AWS Trainium, and Google TPUs.
Build and maintain observability systems tracking time-to-first-token, inter-token latency, throughput, and availability across models, hardware, and regions.
Lead reliability engineering efforts, including monitoring, alerting, automated failover, self-healing infrastructure, and intelligent traffic routing.
Troubleshoot production infrastructure issues directly, working with technical teams and customers to identify and resolve complex challenges.
5+ years of infrastructure engineering experience , currently operating at Staff Engineer level or above in a hands-on individual contributor role.

Nice-to-haves

Experience building or operating production-scale inference platforms at AI infrastructure companies, model-serving providers, or AI research organizations.
Experience deploying inference workloads across multiple accelerator architectures, including NVIDIA GPUs, AWS Trainium, and Google TPUs.
Contributions to open-source inference frameworks such as vLLM or SGLang.
Experience supporting high-throughput, low-latency AI applications in production environments.
Technical Environment
Kubernetes, AWS, GCP, NVIDIA GPUs, CUDA, NCCL, InfiniBand, RDMA/EFA, MPI, vLLM, SGLang, TensorRT-LLM, NVIDIA Dynamo, AWS Trainium, Google TPUs, PyTorch, JAX, Docker, Terraform, Prometheus, and Grafana.
Compensation & Location
Base Salary: 350,000–450,000, with flexibility above this range for exceptionally qualified Staff+ candidates.
Location: San Francisco, California.
Work Arrangement: Full-time, onsite five days per week.
Relocation: May be considered for highly qualified candidates.
Work Authorization: Open to eligible visa transfers, including H-1B and OPT.

Application prep

Use the role’s language and application details while tailoring your materials.

Keywords to mirror

aillmkubernetesawsgcpdocker

Platform: manatal
Effort: medium