Job description

Build and ship: Take ownership of features and services within the evaluation platform: APIs, SDKs, orchestration components, evaluation runners. You'll have the room to make calls on your own work and the support to deliver it well. Productionize ML research: Partner with research engineers to take their prototype code and turn it into reliable services. You'll learn their world quickly and translate research patterns into clean Python that holds up under real load. Move fast, responsibly: You'll get scoped problems with room to figure out the how. We trust you to balance speed with care, to know when something needs a quick prototype and when it needs a design doc, tests, and a careful rollout. Improve as you go: Notice the rough edges and pick them up. The flaky test, the slow build, the confusing API, the runbook that's out of date. We want someone who leaves the codebase a little better every week. Developer experience: Help build the SDKs and abstractions that other Apple teams use to evaluate their models and agents. You'll feel the friction of bad ergonomics directly, which puts you in a great position to fix it. Operational ownership: Your code runs in production. You write the tests, set up the CI, add the metrics, and stay close when something breaks. You don't need to be an SRE, but you take care of what you ship. 4-8 years of software engineering experience building and shipping production services. Strong Python. You're fluent with FastAPI, Pydantic, and the modern Python ecosystem. You write code that's clean, tested, and easy for the next person to pick up. Builder's mindset. You enjoy shipping. You're comfortable iterating quickly on scoped problems and knowing when to slow down for the parts that need it. Fluency with AI coding tools. You actively use tools like Claude Code (or equivalents) in your day-to-day workflow, including features like skills, slash commands, and agent-style workflows. You have a good intuition for when to lean on them, when to steer them, and how to get high-quality output. Familiarity with the agentic LLM landscape. You stay current on how modern LLM systems work in production — tool use, MCP servers, agent frameworks, context management, multi-step reasoning. You can hold a real conversation about the tradeoffs. Hands-on evaluation experience. You've built evaluations for your own agents or LLM systems, or you've worked with evaluation orchestration frameworks like Inspect, Braintrust, LangSmith, Promptfoo, or equivalents (including internal tooling). You understand what makes an evaluation trustworthy vs. theatrical. Real working knowledge of LLMs in production. You're comfortable with prompt iteration, dataset curation, judge models, and statistical reasoning about non-deterministic outputs. You understand the lifecycle around models even if you haven't trained them yourself. Solid engineering fundamentals. You understand testing, CI/CD, containerization (Docker), and basic observability. You've shipped services that others depend on and stayed close when they broke. Clear communicator. You write clear PRs, ask sharp questions, and flag blockers early. You're comfortable disagreeing thoughtfully and changing your mind when the argument is good. Ownership. When something is broken or unclear, you tend to pick it up rather than wait. You either move it forward or surface it clearly. Experience working on developer platforms, internal tools, or SDKs Production experience with LLM/agent systems — building, evaluating, or operating them Familiarity with job orchestration frameworks (Temporal.io, Airflow, or similar) Distributed compute experience (Ray, Dask, or Kubernetes-based job systems) Experience with experiment tracking or ML lifecycle tooling (Weights & Biases, MLflow, etc.) Startup or early-stage experience where you wore multiple hats and shipped under constraint

Requirements

Must-haves

Improve as you go: Notice the rough edges and pick them up. The flaky test, the slow build, the confusing API, the runbook that's out of date. We want someone who leaves the codebase a little better every week.
Operational ownership: Your code runs in production. You write the tests, set up the CI, add the metrics, and stay close when something breaks. You don't need to be an SRE, but you take care of what you ship.
4-8 years of software engineering experience building and shipping production services.
Strong Python. You're fluent with FastAPI, Pydantic, and the modern Python ecosystem. You write code that's clean, tested, and easy for the next person to pick up.
Builder's mindset. You enjoy shipping. You're comfortable iterating quickly on scoped problems and knowing when to slow down for the parts that need it.
Solid engineering fundamentals. You understand testing, CI/CD, containerization (Docker), and basic observability. You've shipped services that others depend on and stayed close when they broke.
Clear communicator. You write clear PRs, ask sharp questions, and flag blockers early. You're comfortable disagreeing thoughtfully and changing your mind when the argument is good.
Ownership. When something is broken or unclear, you tend to pick it up rather than wait. You either move it forward or surface it clearly.
Experience working on developer platforms, internal tools, or SDKs
Production experience with LLM/agent systems — building, evaluating, or operating them
Familiarity with job orchestration frameworks (Temporal.io, Airflow, or similar)
Distributed compute experience (Ray, Dask, or Kubernetes-based job systems)
Experience with experiment tracking or ML lifecycle tooling (Weights & Biases, MLflow, etc.)
Startup or early-stage experience where you wore multiple hats and shipped under constraint

Nice-to-haves

None flagged.

Application prep

Use the role’s language and application details while tailoring your materials.

Keywords to mirror

mlpythongoapiaillmci/cddockerkubernetes

Platform: apple
Effort: medium