Job description

Our team builds the benchmarks, environments, and tooling that power model and agent refinement, and turns observations into actionable opportunities for the next model and agent iteration. We work across the full spectrum of evaluation: offline benchmarks, device-in-the-loop simulation, and on-device observation in production. We develop LLM-as-judge evaluators, train reward models calibrated against human feedback, optimize prompts and context for agents, and contribute targeted datasets and reward signals to foundation model post-training. In this role, you will play a crucial role in designing and developing evaluation and refinement infrastructure that supports a broad range of AI products at Apple. You will work on agent and model evaluation across offline, device-in-the-loop, and on-device settings; build automated prompt and context optimization pipelines; and partner with product and research teams to translate failure analysis into measurable model and agent improvements. You will also have the opportunity to engage with product teams across Apple and contribute to advancements in large language models and agentic systems that will reach millions of users. To succeed in this role, you should have a strong background in machine learning systems, distributed infrastructure, and a proven track record of building and maintaining ML evaluation or training infrastructure. You should be a proactive problem solver with excellent communication skills and the ability to work effectively across multiple codebases, teams, and organizations. Experience with LLM evaluation, reward modeling, prompt optimization, or agentic systems is highly desirable. Strong background in machine learning and distributed systems Experience building and maintaining ML infrastructure for evaluation, training, or deployment Ability to work effectively across multiple codebases, teams, and organizations 8+ years of professional experience as a software engineer, preferably in machine learning or a related field Bachelor's or Master's degree in Computer Science or a related field Experience with LLM evaluation, LLM-as-judge, or reward modeling Experience with prompt optimization, agent harness development, or post-training (SFT, DPO, RLHF) Proficiency in Python and ML frameworks such as PyTorch Experience with agentic systems, simulation environments, or trajectory-based data generation Familiarity with on-device or privacy-preserving ML Proactive and determined problem-solving skills Excellent communication skills

Requirements

Must-haves

In this role, you will play a crucial role in designing and developing evaluation and refinement infrastructure that supports a broad range of AI products at Apple.
You will also have the opportunity to engage with product teams across Apple and contribute to advancements in large language models and agentic systems that will reach millions of users.
To succeed in this role, you should have a strong background in machine learning systems, distributed infrastructure, and a proven track record of building and maintaining ML evaluation or training infrastructure.
Strong background in machine learning and distributed systems
Experience building and maintaining ML infrastructure for evaluation, training, or deployment
Ability to work effectively across multiple codebases, teams, and organizations
8+ years of professional experience as a software engineer, preferably in machine learning or a related field
Bachelor's or Master's degree in Computer Science or a related field
Experience with LLM evaluation, LLM-as-judge, or reward modeling
Experience with prompt optimization, agent harness development, or post-training (SFT, DPO, RLHF)
Proficiency in Python and ML frameworks such as PyTorch
Experience with agentic systems, simulation environments, or trajectory-based data generation
Familiarity with on-device or privacy-preserving ML
Proactive and determined problem-solving skills
Excellent communication skills

Nice-to-haves

None flagged.

Application prep

Use the role’s language and application details while tailoring your materials.

Keywords to mirror

llmaimachine learningmlpython

Platform: apple
Effort: medium