Job description

As a Senior Machine Learning Engineering Manager in AIML Evaluation, you will lead the technical strategy and execution for agent evaluation and automatic optimization. You will own systems that evaluate foundation models and agents, diagnose failure modes, and use those signals to drive automated prompt, context, tool, rubric, and agent-harness improvements. You will also help establish the interfaces between evaluation and post-training so that high-value failures can be converted into targeted data, environments, reward signals, and measurable model improvements. This is a hands-on leadership role. You will prototype new approaches, participate in architecture and code reviews, design experiments, and help your team translate emerging research into scalable evaluation and optimization pipelines. You will partner closely with Apple Foundation Models, product engineering teams, and other AIML groups to build an evaluation flywheel that connects real product behavior with model and agent refinement. You will also work across the organization to advance synthetic data generation for both evaluation and post-training, with strong attention to data quality, representativeness, privacy, and reproducibility. 8+ years of professional experience in machine learning, applied research, or software engineering, including experience building production ML systems or large-scale experimentation platforms. 3+ years of technical leadership experience, including direct people management of machine learning or software engineers and a demonstrated ability to mentor and grow strong technical talent. Master’s or PhD in Computer Science, Machine Learning, Artificial Intelligence, or a related technical field. Strong hands-on programming and software engineering skills, particularly in Python, with experience building reliable ML pipelines using modern machine learning or deep learning frameworks. Deep experience with large language models or agentic systems, including evaluation of multi-turn behavior, tool use, planning, reasoning, or other action-taking workflows. Experience building automated evaluation methods such as LLM-based judges, rubrics, reward models, simulation-based evaluation, or scalable benchmark infrastructure. Experience with at least one model or agent refinement area such as automatic prompt or context optimization, post-training, preference optimization, reinforcement learning, or agent-harness optimization. Excellent communication and collaboration skills, with demonstrated ability to align research, engineering, and product teams around ambiguous technical problems. Track record of applying recent machine learning research to production systems or high-impact product development. Experience with automatic prompt or context optimization, agent-search methods, evaluator optimization, or multi-objective optimization for agentic systems. Experience generating and evaluating synthetic datasets, tool-use trajectories, or multi-turn agent interactions, including methods for filtering, deduplication, diversity, and quality control. Experience designing evaluation systems that combine offline benchmarks, simulation, human evaluation, and product- or usage-derived signals. Experience with privacy-preserving or on-device machine learning and evaluation. Demonstrated ability to influence technical strategy across organizational boundaries and communicate complex model-quality tradeoffs to senior technical leaders.

Requirements

Must-haves

8+ years of professional experience in machine learning, applied research, or software engineering, including experience building production ML systems or large-scale experimentation platforms.
3+ years of technical leadership experience, including direct people management of machine learning or software engineers and a demonstrated ability to mentor and grow strong technical talent.
Master’s or PhD in Computer Science, Machine Learning, Artificial Intelligence, or a related technical field.
Strong hands-on programming and software engineering skills, particularly in Python, with experience building reliable ML pipelines using modern machine learning or deep learning frameworks.
Deep experience with large language models or agentic systems, including evaluation of multi-turn behavior, tool use, planning, reasoning, or other action-taking workflows.
Experience building automated evaluation methods such as LLM-based judges, rubrics, reward models, simulation-based evaluation, or scalable benchmark infrastructure.
Experience with at least one model or agent refinement area such as automatic prompt or context optimization, post-training, preference optimization, reinforcement learning, or agent-harness optimization.
Excellent communication and collaboration skills, with demonstrated ability to align research, engineering, and product teams around ambiguous technical problems.
Track record of applying recent machine learning research to production systems or high-impact product development.
Experience with automatic prompt or context optimization, agent-search methods, evaluator optimization, or multi-objective optimization for agentic systems.
Experience generating and evaluating synthetic datasets, tool-use trajectories, or multi-turn agent interactions, including methods for filtering, deduplication, diversity, and quality control.
Experience designing evaluation systems that combine offline benchmarks, simulation, human evaluation, and product- or usage-derived signals.
Experience with privacy-preserving or on-device machine learning and evaluation.
Demonstrated ability to influence technical strategy across organizational boundaries and communicate complex model-quality tradeoffs to senior technical leaders.

Nice-to-haves

None flagged.

Application prep

Use the role’s language and application details while tailoring your materials.

Keywords to mirror

machine learningmlpythonllm

Platform: apple
Effort: medium