Senior Principal Software Engineer – LLM Inference Optimization at People In AI

Job description

Senior Principal Software Engineer (LLM Inference, CUDA & GPU Performance Optimization) Compensation: $230,000–$280,000 base + annual bonus opportunity + potential equity Location: Remote – United States The Company An established global AI technology company building intelligent, real-time conversational experiences for the automotive industry. Its technology is deployed at significant commercial scale through partnerships with major vehicle manufacturers worldwide, with continued investment in advanced generative AI and high-performance inference infrastructure. The Opportunity We are looking for a Senior Principal Software Engineer who understands what it takes to make large language models run faster, more efficiently, and reliably in production. This is a hands-on engineering position focused on pushing the boundaries of LLM inference performance through low-level GPU optimization, inference runtime development, and advanced memory management. Your work will directly influence product responsiveness, infrastructure economics, and the performance of real-world AI systems operating at scale. The Role You will take ownership of identifying and removing performance bottlenecks in production LLM inference systems, with a particular focus on cloud-hosted workloads and real-time response requirements. Working alongside an established AI infrastructure team, you will optimize inference engines, develop CUDA kernels, and improve GPU efficiency to deliver measurable reductions in latency and improvements in throughput. This is not a conventional MLOps, infrastructure maintenance, or LLM application development role. We are looking for someone who can get deep into inference engine internals and make the underlying systems materially faster. What You'll Do • Optimize LLM inference performance across production systems, improving latency, tokens/sec, throughput, and GPU utilization. • Develop and tune CUDA kernels , including attention implementations and other performance-critical inference operations. • Extend and optimize inference engines such as vLLM and other high-performance runtimes, identifying opportunities for low-level code improvements. • Improve GPU memory efficiency through KV cache optimization, memory layout design, paging, and cache-aware execution strategies. • Optimize prefill and decode performance , particularly for workloads involving substantial contextual input and short, latency-sensitive responses. • Evaluate advanced optimization techniques , including quantization, continuous batching, speculative decoding, and distributed inference strategies. • Benchmark and deploy production improvements , delivering measurable gains in inference performance, stability, and cost efficiency. What You'll Bring • Significant hands-on experience with production ML or LLM inference systems , including direct responsibility for performance optimization. • Strong C++ programming experience , ideally five or more years, with the ability to understand, modify, and optimize complex low-level code. • Practical expertise in CUDA development, GPU profiling, and kernel-level optimization . • Deep understanding of GPU architecture, memory bandwidth, memory hierarchies, concurrency, and the trade-offs involved in high-performance inference. • Experience with inference frameworks such as vLLM, TensorRT-LLM, SGLang, or comparable runtimes , ideally including modifications beyond standard configuration. • A track record of diagnosing performance bottlenecks and producing measurable improvements in latency, throughput, or infrastructure efficiency. • An independent engineering mindset, strong technical judgment, and the ability to operate effectively in a fast-moving, globally distributed team. What This Role Requires • Real production inference experience: You must have worked directly on deploying or optimizing inference workloads beyond proof-of-concept or research environments. • Low-level engineering capability: Strong C++ and hands-on CUDA experience, with the ability to work directly with performance-critical code. • GPU performance expertise: A practical understanding of memory bandwidth, compute utilization, caching, and inference bottlenecks. • Demonstrable technical ownership: The ability to explain what you personally built or optimized, why you made specific engineering decisions, and what measurable results you achieved. • Hands-on problem-solving: Comfort investigating complex systems, validating assumptions, and implementing improvements rather than relying exclusively on existing frameworks or third-party services. Tech Stack • Languages: C++, CUDA; Rust exposure is a plus. • Inference Engines: vLLM, TensorRT-LLM, llama.cpp, QAIRT; SGLang and other emerging inference runtimes. • GPU Performance: Custom CUDA kernels, GPU profiling, attention optimization, memory bandwidth tuning. • Quantization: INT8, INT4, FP4, FP8, mixed precision, AWQ, GPTQ. • Inference Optimization: KV cache optimization, prefix caching, continuous batching, speculative decoding, prefill/decode optimization. • Infrastructure: Cloud-hosted LLM inference, Azure and other cloud environments, distributed inference architectures. Why Join • Solve technically challenging problems with direct impact. Work deep inside LLM inference systems, optimizing the actual execution of models rather than building wrappers around them. • See your engineering work translate into measurable results. Improvements in latency, throughput, and GPU efficiency directly affect real-time AI product performance and commercial economics. • Combine established scale with meaningful technical ownership. Join a commercially proven AI business investing in advanced inference capabilities, with a remote US position and the opportunity to shape critical technology. About People in AI People in AI is a specialist recruitment partner connecting exceptional AI, machine learning, data, and engineering talent with some of the most ambitious technology companies in the world. We work closely with founders, technical leaders, and hiring teams to represent opportunities accurately and help candidates assess genuine fit.

Requirements

Must-haves

Senior Principal Software Engineer
(LLM Inference, CUDA & GPU Performance Optimization)
Location: Remote – United States
We are looking for a Senior Principal Software Engineer who understands what it takes to make large language models run faster, more efficiently, and reliably in production.
You will take ownership of identifying and removing performance bottlenecks in production LLM inference systems, with a particular focus on cloud-hosted workloads and real-time response requirements.
Working alongside an established AI infrastructure team, you will optimize inference engines, develop CUDA kernels, and improve GPU efficiency to deliver measurable reductions in latency and improvements in throughput.
Optimize LLM inference performance across production systems, improving latency, tokens/sec, throughput, and GPU utilization.
Develop and tune CUDA kernels , including attention implementations and other performance-critical inference operations.
Extend and optimize inference engines such as vLLM and other high-performance runtimes, identifying opportunities for low-level code improvements.
Improve GPU memory efficiency through KV cache optimization, memory layout design, paging, and cache-aware execution strategies.
Optimize prefill and decode performance , particularly for workloads involving substantial contextual input and short, latency-sensitive responses.
Evaluate advanced optimization techniques , including quantization, continuous batching, speculative decoding, and distributed inference strategies.
Benchmark and deploy production improvements , delivering measurable gains in inference performance, stability, and cost efficiency.
Significant hands-on experience with production ML or LLM inference systems , including direct responsibility for performance optimization.
Strong C++ programming experience , ideally five or more years, with the ability to understand, modify, and optimize complex low-level code.
Practical expertise in CUDA development, GPU profiling, and kernel-level optimization .
Deep understanding of GPU architecture, memory bandwidth, memory hierarchies, concurrency, and the trade-offs involved in high-performance inference.
Experience with inference frameworks such as vLLM, TensorRT-LLM, SGLang, or comparable runtimes , ideally including modifications beyond standard configuration.
A track record of diagnosing performance bottlenecks and producing measurable improvements in latency, throughput, or infrastructure efficiency.
An independent engineering mindset, strong technical judgment, and the ability to operate effectively in a fast-moving, globally distributed team.
What This Role Requires
Real production inference experience: You must have worked directly on deploying or optimizing inference workloads beyond proof-of-concept or research environments.

Nice-to-haves

Compensation: $230,000–$280,000 base + annual bonus opportunity + potential equity
Languages: C++, CUDA; Rust exposure is a plus.

Application prep

Use the role’s language and application details while tailoring your materials.

Keywords to mirror

llmaimlrustazuremachine learning

Platform: careers-page
Effort: medium