Job description

At Klaviyo, we value the unique backgrounds, experiences and perspectives each Klaviyo (we call ourselves Klaviyos) brings to our workplace each and every day. We believe everyone deserves a fair shot at success and appreciate the experiences each person brings beyond the traditional job requirements. If you’re a close but not exact match with the description, we hope you’ll still consider applying. Want to learn more about life at Klaviyo? Visit klaviyo.com/careers to see how we empower creators to own their own destiny. Senior Software Engineer, Agent Runtime & Customer Workflows Customer Agent About the Role At Klaviyo, we believe the future of software lies not only in tools that help people work more efficiently, but in intelligent systems that can take action, learn from outcomes, and improve customer experiences over time. Klaviyo serves more than 167,000 customers and processes billions of consumer profiles, messages, interactions, and conversion signals. This creates a unique opportunity to build state-of-the-art AI systems that help businesses create and execute better customer experiences at scale. We’re looking for a Senior Software Engineer, Agent Runtime & Customer Workflows to join the Customer Agent team, Klaviyo’s AI-native conversational platform. You’ll design and build scalable backend systems and AI-powered product experiences that enable agents to reason, retrieve context, use tools, and take reliable action on behalf of customers. You’ll be expected to independently own meaningful technical problems, make pragmatic engineering tradeoffs, and help evolve the architecture and quality of the systems you work on. How You Will Make a Difference Design and build reliable, scalable backend systems and APIs that power AI-driven customer experiences. Build and evolve production agentic systems involving tool use, orchestration, context management, retrieval, structured outputs, and multi-step workflows. Solve hard applied AI problems across retrieval and RAG, grounding, model selection, hallucination mitigation, and deciding when deterministic or hybrid approaches are better than LLMs. Develop evaluation strategies using representative datasets, automated and human evaluation, regression testing, qualitative error analysis, and production signals. Improve production reliability through guardrails, fallbacks, retries, observability, monitoring, and thoughtful failure handling. Build asynchronous and distributed processing workflows that support high-volume AI workloads. Participate in an on-call rotation and help diagnose, mitigate, and learn from production incidents. Work directly with external customers and cross-functional partners to understand workflows, identify pain points, and translate feedback into product and technical improvements. Monitor shipped experiences and use customer behavior, quality metrics, and system performance to guide iteration. Who You Are You have 5-7 years of professional software engineering experience , with a strong foundation in backend systems and distributed applications. You have hands-on experience building and deploying generative AI or agentic AI applications into production for real users. You are proficient in Python and modern backend frameworks such as FastAPI or Django. You have experience with asynchronous processing and distributed task or event systems such as Celery, Kafka, SQS, RabbitMQ, or Redis. You have strong knowledge of databases, data modeling, APIs, and persistence patterns used in production systems. You are comfortable working in cloud-native environments with technologies such as AWS, containers, Kubernetes, infrastructure automation, or CI/CD systems. You can reason clearly about tradeoffs among quality, latency, cost, reliability, and implementation complexity. You are able to operate independently in ambiguous environments, break down loosely defined problems, and make pragmatic decisions about what to build first. You have experience with and genuine interest in working directly with external customers to understand problems and improve products. You care about shipping useful, reliable products and bring curiosity and sound engineering judgment to a rapidly evolving technical space. Nice to Have Experience training, fine-tuning, distilling, or otherwise adapting machine learning or language models for production use. Experience with reinforcement learning or feedback-driven optimization. Experience building evaluation infrastructure such as LLM-as-judge systems, evaluator calibration frameworks, benchmark suites, or AI quality platforms. Experience operating AI systems at substantial scale, including optimizing inference cost, latency, or throughput. #indeedRD Massachusetts Applicants: It is unlawful in Massachusetts to require or administer a lie detector test as a condition of employment or continued employment. An employer who violates this law shall be subject to criminal penalties and civil liability. Our salary range reflects the cost of labor across various U.S. geographic markets. The range displayed below reflects the minimum and maximum target salaries for the position across all our US locations. The base salary offered for this position is determined by several factors, including the applicant’s job-related skills, relevant experience, education or training, and work location. In addition to base salary, our total compensation package may include participation in the company’s annual cash bonus plan, variable compensation (OTE) for sales and customer success roles, equity, sign-on payments, and a comprehensive range of health, welfare, and wellbeing benefits based on eligibility. Your recruiter can provide more details about the specific salary/OTE range for your preferred location during the hiring process. Base Pay Range For US Locations: $148,000 — $222,000 USD This role may require up to 10% travel for purposes such as new hire onboarding, client or partner work if applicable, team meetings, and industry events. Travel is coordinated in advance. Get to Know Klaviyo We’re Klaviyo (pronounced clay-vee-oh). We empower creators to own their destiny by making first-party data accessible and actionable like never before. We see limitless potential for the technology we’re developing to nurture personalized experiences in ecommerce and beyond. To reach our goals, we need our own crew of remarkable creators—ambitious and collaborative teammates who stay focused on our north star: delighting our customers. If you’re ready to do the best work of your career, where you’ll be welcomed as your whole self from day one and supported with generous benefits, we hope you’ll join us. AI fluency at Klaviyo includes responsible use of AI (including privacy, security, bias awareness, and human-in-the-loop). We provide accommodations as needed. By participating in Klaviyo’s interview process, you acknowledge that you have read, understood, and will adhere to our Guidelines for using AI in the Klaviyo interview Process . For more information about how we process your personal data, see our Job Applicant Privacy Notice . Klaviyo is committed to a policy of equal opportunity and non-discrimination. We do not discriminate on the basis of race, ethnicity, citizenship, national origin, color, religion or religious creed, age, sex (including pregnancy), gender identity, sexual orientation, physical or mental disability, veteran or active military status, marital status, criminal record, genetics, retaliation, sexual harassment or any other characteristic protected by applicable law. IMPORTANT NOTICE: Our company takes the security and privacy of job applicants very seriously. We will never ask for payment, bank details, or personal financial information as part of the application process. All our legitimate job postings can be found on our official career site. Please be cautious of job offers that come from non-company email addresses (@klaviyo.com), instant messaging platforms, or unsolicited calls. By clicking "Submit Application" you consent to Klaviyo processing your Personal Data in accordance with our Job Applicant Privacy Notice. If you do not wish for Klaviyo to process your Personal Data, please do not submit an application. You can find our Job Applicant Privacy Notice here and here (FR).

Requirements

Must-haves

You’ll be expected to independently own meaningful technical problems, make pragmatic engineering tradeoffs, and help evolve the architecture and quality of the systems you work on.
How You Will Make a Difference
Design and build reliable, scalable backend systems and APIs that power AI-driven customer experiences.
Build and evolve production agentic systems involving tool use, orchestration, context management, retrieval, structured outputs, and multi-step workflows.
Solve hard applied AI problems across retrieval and RAG, grounding, model selection, hallucination mitigation, and deciding when deterministic or hybrid approaches are better than LLMs.
Develop evaluation strategies using representative datasets, automated and human evaluation, regression testing, qualitative error analysis, and production signals.
Improve production reliability through guardrails, fallbacks, retries, observability, monitoring, and thoughtful failure handling.
Build asynchronous and distributed processing workflows that support high-volume AI workloads.
Participate in an on-call rotation and help diagnose, mitigate, and learn from production incidents.
Work directly with external customers and cross-functional partners to understand workflows, identify pain points, and translate feedback into product and technical improvements.
Monitor shipped experiences and use customer behavior, quality metrics, and system performance to guide iteration.
You have 5-7 years of professional software engineering experience , with a strong foundation in backend systems and distributed applications.
You have hands-on experience building and deploying generative AI or agentic AI applications into production for real users.
You are proficient in Python and modern backend frameworks such as FastAPI or Django.
You have experience with asynchronous processing and distributed task or event systems such as Celery, Kafka, SQS, RabbitMQ, or Redis.
You have strong knowledge of databases, data modeling, APIs, and persistence patterns used in production systems.
You are comfortable working in cloud-native environments with technologies such as AWS, containers, Kubernetes, infrastructure automation, or CI/CD systems.
You can reason clearly about tradeoffs among quality, latency, cost, reliability, and implementation complexity.

Nice-to-haves

Experience training, fine-tuning, distilling, or otherwise adapting machine learning or language models for production use.
Experience with reinforcement learning or feedback-driven optimization.
Experience building evaluation infrastructure such as LLM-as-judge systems, evaluator calibration frameworks, benchmark suites, or AI quality platforms.
Experience operating AI systems at substantial scale, including optimizing inference cost, latency, or throughput.
Get to Know Klaviyo
AI fluency at Klaviyo includes responsible use of AI (including privacy, security, bias awareness, and human-in-the-loop). We provide accommodations as needed.

Application prep

Use the role’s language and application details while tailoring your materials.

Keywords to mirror

aipythonawskubernetesci/cdmachine learningllm

Platform: greenhouse
Effort: quick