Job description

We're transforming the grocery industry At Instacart, we invite the world to share love through food because we believe everyone should have access to the food they love and more time to enjoy it together. Where others see a simple need for grocery delivery, we see exciting complexity and endless opportunity to serve the varied needs of our community. We work to deliver an essential service that customers rely on to get their groceries and household goods, while also offering safe and flexible earnings opportunities to Instacart Personal Shoppers. Instacart has become a lifeline for millions of people, and we’re building the team to help push our shopping cart forward. If you’re ready to do the best work of your life, come join our table. Instacart is a Flex First team There’s no one-size fits all approach to how we do our best work. Our employees have the flexibility to choose where they do their best work—whether it’s from home, an office, or your favorite coffee shop—while staying connected and building community through regular in-person events. Learn more about our flexible approach to where we work. Overview As a Senior Software Engineer II in the Ads Quality team specializing in ML infrastructure, you'll work with a team of software and machine learning engineers to build state-of-the-art systems that optimize Ads performance throughout the ads serving funnel, from retrieval to pricing. You'll own the technical direction of our data, feature processing and model training/serving solutions, enabling next-generation modelling opportunities—from utilizing out of the box LLMs to building bespoke Foundation Models that powers all the Ads use cases. You will eliminate bottlenecks across the training and serving system to optimize stability, latency and cost. Building on Instacart's unique data and market position, these systems create value for advertising partners, improve consumer experience, and directly contribute to Instacart's overall revenue / business success. You'll have the unique opportunity to collaborate with a cross-functional team, turning diverse insights into powerful solutions that elevate advertising effectiveness and operational stability. Your expertise will steer the development of scalable training and serving infrastructure, driving performance and efficiency gains across our advertising platform. Together, we'll build an ads ecosystem that not only serves millions but also excels in delivering value to users and advertisers alike. Instacart Ads is transforming the online grocery shopping experience by seamlessly integrating advanced advertising solutions into our platform. The Ads Quality team is dedicated to delivering outstanding advertising outcomes by driving innovation and precision in response prediction, advertiser optimization, and ads auction processes. Our mission is to ensure relevance, fairness, and performance for all participants in the ad ecosystem. By leveraging terabytes of data and sophisticated machine learning algorithms, we build advanced predictive models that enhance user experience, drive advertiser success, and sustain a thriving ad ecosystem. We're also at the forefront of integrating Large Language Models (LLMs), unlocking powerful new capabilities that are redefining how brands connect with consumers in the digital grocery space. Our next major bet is a real-time sequential foundation model for ads—and building the training and serving infrastructure to make it possible is central to this role. About the Job Design, develop, and deploy machine learning infrastructure to tackle practical challenges in our complex marketplace while advancing the training and serving platform. Work with the company wide ML Foundation team to develop solutions to meet Ads unique needs. Optimize training and serving funnel bottlenecks—profiling real systems and driving meaningful wins in throughput, cost, and latency, from GPU/accelerator utilization to tail latency at serving time. Identify instabilities and shortcomings of our systems and operating procedures to deliver best in class availability. Build real-time feature processing infrastructure—low-latency pipelines that compute, join, and serve fresh features to models on the critical path of every ad request. Lead the infrastructure foundation for our real-time sequential foundation model, including online sequence feature stores, streaming update paths, efficient long-context serving, and training pipelines that keep pace with rapidly evolving data. Collaborate closely with product managers, data scientists, and MLEs to deeply understand business needs and co-design systems where product, model architecture and infrastructure constraints are solved together. Actively engage with diverse stakeholders to ensure that solutions are well-integrated and aligned with business goals. Mentor engineers, set technical direction, and conduct thorough code reviews to ensure high code quality and maintainability. Lead by example in fostering a culture of continuous improvement and technical excellence. About You Minimum Qualifications Have a graduate degree (masters or PhD) in computer science, artificial intelligence, machine learning, operations research or equivalent self study and experience. Have 3+ years of industry experience building ML infrastructure or using machine learning to solve real-world problems with large datasets. Deep expertise building, scaling and managing ML infrastructure for training and/or serving in production, with a strong understanding of modern machine learning systems and MLOps. Strong programming skills in software engineering (e.g., Python, Go), along with expertise in data manipulation using tools (SQL, Spark, and Pandas), and ML frameworks (Torch, Tensorflow, XGBoost). Strong analytical and problem-solving skills. Excellent communication skills (both verbal and written) and can collaborate with a diverse group of stakeholders across all levels. Eager to jump into domains, languages, and problem areas that might be new and unfamiliar. Preferred Qualifications Have 5+ years of industry experience tech leading a team to build ML infrastructure or using machine learning to solve real-world problems with large datasets. Demonstrated experience solving complex model training and serving efficiency challenges, including optimizing training pipelines, reducing model training times, and improving serving architectures to enhance operational efficiency and scalability. Hands-on experience with real-time / online feature processing and serving—streaming systems (e.g., Kafka, Flink), online feature stores, and low-latency retrieval on the request path. Experience with sequential modeling, transformer architecture, and generative retrieval or building large scale online recommendation systems. Experience serving large sequential or foundation models in low-latency, high-QPS production settings. Experience in digital advertising platforms. Familiarity with LLM integrations, prompt engineering, and productivity tooling. #LI-Remote Instacart provides highly market-competitive compensation and benefits in each location where our employees work. This role is remote and the base pay range for a successful candidate is dependent on their permanent work location. Please review our Flex First remote work policy here . Offers may vary based on many factors, such as candidate experience and skills required for the role. Additionally, this role is eligible for a new hire equity grant as well as annual refresh grants. Please rea d more about our benefits offerings here . For US based candidates, the base pay ranges for a successful candidate are listed below. CA, NY, CT, NJ $230,000 — $242,500 USD WA $220,000 — $232,000 USD OR, DE, ME, MA, MD, NH, RI, VT, DC, PA, VA, CO, TX, IL, HI $211,000 — $222,500 USD All other states $192,000 — $202,500 USD

Requirements

Must-haves

We're transforming the grocery industry
Instacart has become a lifeline for millions of people, and we’re building the team to help push our shopping cart forward. If you’re ready to do the best work of your life, come join our table.
Instacart is a Flex First team
Optimize training and serving funnel bottlenecks—profiling real systems and driving meaningful wins in throughput, cost, and latency, from GPU/accelerator utilization to tail latency at serving time.
Identify instabilities and shortcomings of our systems and operating procedures to deliver best in class availability.
Build real-time feature processing infrastructure—low-latency pipelines that compute, join, and serve fresh features to models on the critical path of every ad request.
Collaborate closely with product managers, data scientists, and MLEs to deeply understand business needs and co-design systems where product, model architecture and infrastructure constraints are solved together.
Actively engage with diverse stakeholders to ensure that solutions are well-integrated and aligned with business goals.
Mentor engineers, set technical direction, and conduct thorough code reviews to ensure high code quality and maintainability. Lead by example in fostering a culture of continuous improvement and technical excellence.
Have a graduate degree (masters or PhD) in computer science, artificial intelligence, machine learning, operations research or equivalent self study and experience.
Have 3+ years of industry experience building ML infrastructure or using machine learning to solve real-world problems with large datasets.
Deep expertise building, scaling and managing ML infrastructure for training and/or serving in production, with a strong understanding of modern machine learning systems and MLOps.
Strong programming skills in software engineering (e.g., Python, Go), along with expertise in data manipulation using tools (SQL, Spark, and Pandas), and ML frameworks (Torch, Tensorflow, XGBoost).
Strong analytical and problem-solving skills.
Excellent communication skills (both verbal and written) and can collaborate with a diverse group of stakeholders across all levels.
Eager to jump into domains, languages, and problem areas that might be new and unfamiliar.
Have 5+ years of industry experience tech leading a team to build ML infrastructure or using machine learning to solve real-world problems with large datasets.

Nice-to-haves

Hands-on experience with real-time / online feature processing and serving—streaming systems (e.g., Kafka, Flink), online feature stores, and low-latency retrieval on the request path.
Experience with sequential modeling, transformer architecture, and generative retrieval or building large scale online recommendation systems.
Experience serving large sequential or foundation models in low-latency, high-QPS production settings.
Experience in digital advertising platforms.
Familiarity with LLM integrations, prompt engineering, and productivity tooling.

Application prep

Use the role’s language and application details while tailoring your materials.

Keywords to mirror

mlmachine learningdesign systemspythongosqlllm

Platform: greenhouse
Effort: quick