Job description

About Alaffia & Our Mission Every year, U.S. health plans lose billions to improper payments and administrative waste. That wasted spending ultimately trickles down across the healthcare ecosystem, driving up costs for plans, providers, and patients alike. We’re here to change that paradigm. Alaffia is a new kind of claims operations partner for health plans. Using expert clinicians and transparent AI, we deliver deeper insights, smarter automation, and consistently better outcomes across the entire lifecycle of claims. With Alaffia, health plans can cut wasted spending more effectively than ever — and provide their members the most affordable care. We’re a high-growth, venture-backed Series B healthtech startup based in NYC and are actively scaling our company. Join us in helping to build a healthcare system that works better for everyone. *This position requires current authorization to work in the United States. Unfortunately, we are not in a position to sponsor work visas at this time. About the Role & What You’ll Be Doing Alaffia Health is a healthcare AI startup revolutionizing medical billing and data automation. Our AI-driven platform leverages state-of-the-art generative AI and machine learning technologies to enhance accuracy, efficiency, and compliance in medical billing and auditing • As we scale, we're looking for an MLOps Engineer to build the operational backbone behind Alaffia's AI agents. Our agentic systems are already in production reviewing real claims but as both our library of agents and our roster of health-plan customers grow, we need someone to make that growth reliable, measurable, and cost-efficient. • This is a hybrid role. The core is MLOps: standing up evaluation frameworks, wiring evals into CI, managing model access across cloud providers, tracking experiments, and defining how we operate a growing fleet of agents across many customers. But we also want someone who can go deeper on the models themselves; developing, fine-tuning, and improving the LLMs and ML models that power our agents. • If you like owning systems end to end from a model experiment in a dev environment all the way to a monitored, evaluated agent running in production, and want a direct hand in shaping how a Series B healthtech builds, ships, and measures AI, this role is for you. You'll work closely with our AI engineers and clinical subject-matter experts. Your Responsibilities • Design, build, and own our agent evaluation framework covering output quality, throughput, and cost, so we can measure every agent objectively and catch regressions early. • Integrate automated evals into our CI/CD pipeline so changes to our agents are tested and quality-gated before they ship. • Manage model access and enablement across cloud providers (AWS Bedrock, Azure OpenAI, and others), including onboarding new models, managing quotas, and benchmarking options for performance and reliability. • Build experiment tracking, metrics, and observability for the experiments our team runs in dev, so results are reproducible and comparable. • Develop the systems and processes to manage our agents across a growing roster of customers including versioning, configuration, and deployment as both the number of agents and customers scale. • Monitor agents in production for quality, throughput, cost, and drift, and build the dashboards and alerting to keep them healthy. • Develop, fine-tune, and evaluate LLMs and machine learning models to improve the accuracy and efficiency of our claim-review agents. • Build and maintain versioned pipelines for models, prompts, and agent configurations. • Track and optimize inference cost and unit economics as we scale. • Partner with AI engineers and clinical experts to turn domain expertise into measurable, deployable model behavior. What We’re Looking For • 5+ years of experience in MLOps, ML platforms, or ML infrastructure engineering, ideally with experience supporting LLM-based or agentic systems. • Strong software engineering fundamentals and proficiency in Python. • Hands-on experience developing, training, and fine-tuning machine learning models and/or LLMs. • Experience building evaluation or benchmarking frameworks for ML or LLM outputs. • Experience with CI/CD pipelines and integrating automated testing into them. • Working knowledge of cloud AI platforms such as AWS (Bedrock, SageMaker) and/or Azure (Azure OpenAI, Azure ML). • Familiarity with experiment-tracking and ML observability tooling (e.g., MLflow, Weights & Biases). • A solid grasp of the operational challenges specific to LLMs — non-determinism, latency, token cost, prompt and model versioning, and why evaluating them is genuinely hard. • A bias toward ownership and the ability to carry a system from experiment to production. Nice to have • Experience operating multi-tenant production systems or ML services at scale. • Experience in a regulated or healthcare environment, including handling PHI and HIPAA considerations. • Experience with agent frameworks or orchestration. • Comfort collaborating with non-engineering domain experts, such as clinicians, to translate their expertise into model behavior. Our Culture Alaffia was born out of our founders’ personal connection to the inefficiency of the U.S. healthcare system. We are deeply mission-driven, with an abiding belief that technology can help create a better future for everyone — and we’re looking for others who share our passion for change to join the team. What Else Do You Get Working With Us? • Competitive Compensation • Medical, Dental, and Vision benefits • Flexible, paid vacation policy • Work in a flat organizational structure — direct access to Leadership

Requirements

Must-haves

About Alaffia & Our Mission
We’re a high-growth, venture-backed Series B healthtech startup based in NYC and are actively scaling our company. Join us in helping to build a healthcare system that works better for everyone.
This position requires current authorization to work in the United States. Unfortunately, we are not in a position to sponsor work visas at this time.
About the Role & What You’ll Be Doing
Design, build, and own our agent evaluation framework covering output quality, throughput, and cost, so we can measure every agent objectively and catch regressions early.
Integrate automated evals into our CI/CD pipeline so changes to our agents are tested and quality-gated before they ship.
Manage model access and enablement across cloud providers (AWS Bedrock, Azure OpenAI, and others), including onboarding new models, managing quotas, and benchmarking options for performance and reliability.
Build experiment tracking, metrics, and observability for the experiments our team runs in dev, so results are reproducible and comparable.
Develop the systems and processes to manage our agents across a growing roster of customers including versioning, configuration, and deployment as both the number of agents and customers scale.
Monitor agents in production for quality, throughput, cost, and drift, and build the dashboards and alerting to keep them healthy.
Develop, fine-tune, and evaluate LLMs and machine learning models to improve the accuracy and efficiency of our claim-review agents.
Build and maintain versioned pipelines for models, prompts, and agent configurations.
Track and optimize inference cost and unit economics as we scale.
Partner with AI engineers and clinical experts to turn domain expertise into measurable, deployable model behavior.
What We’re Looking For
5+ years of experience in MLOps, ML platforms, or ML infrastructure engineering, ideally with experience supporting LLM-based or agentic systems.
Strong software engineering fundamentals and proficiency in Python.
Hands-on experience developing, training, and fine-tuning machine learning models and/or LLMs.
Experience building evaluation or benchmarking frameworks for ML or LLM outputs.
Experience with CI/CD pipelines and integrating automated testing into them.

Nice-to-haves

Experience operating multi-tenant production systems or ML services at scale.
Experience in a regulated or healthcare environment, including handling PHI and HIPAA considerations.
Experience with agent frameworks or orchestration.
Comfort collaborating with non-engineering domain experts, such as clinicians, to translate their expertise into model behavior.

Application prep

Use the role’s language and application details while tailoring your materials.

Keywords to mirror

aimachine learninggomlci/cdawsazurellmpython

Platform: careers-html
Effort: medium

Benefits

Benefits detected in the listing.

Dental

Vision