Job description

In this pivotal role, you will build and lead the team that designs state-of-the-art autograder systems evaluating AI product quality at scale. * Lead and grow a high-performing team of applied AI scientists and machine learning engineers, including hiring, talent development, performance management, and building a strong technical leadership layer. * Set the technical vision and roadmap for autograder development, evaluation methodology, and toolings. * Deliver reliable, scalable LLM/VLM-based evaluation systems that can be trusted for product-quality decisions. * Navigate complex technical and organizational challenges, resolve cross-functional dependencies, and sustain alignment across organizations. * Identify opportunities to standardize and automate the end-to-end evaluation workflow, enabling the team to scale beyond bespoke autograder development. MS/PhD degree in Computer Science, Machine Learning, AI, or a related field. 1+ years of industry experience building LLM/VLM-based evaluators, autograders, or automated evaluation systems. 5+ years of hands-on experience in machine learning, model alignment, or benchmark creation. Track record of managing teams of 3+ machine learning engineers and/or AI scientists. Deep understanding of GenAI models and their evaluation methodologies, including rubric design, evaluation set development, validation, calibration, and human-model alignment. * Strong technical judgment and a track record of leading complex, ambiguous AI/ML projects from definition through delivery. Experience translating emerging GenAI techniques and research into reliable, production-ready systems. Experience scaling AI/ML solutions into reusable platforms, frameworks, or standardized workflows. Demonstrated track record of developing senior machine learning engineers and/or technical leads. Demonstrated ability to align diverse stakeholders around strategy, navigate complex situations, and drive effective outcomes across organizations.

Requirements

Must-haves

In this pivotal role, you will build and lead the team that designs state-of-the-art autograder systems evaluating AI product quality at scale.
Lead and grow a high-performing team of applied AI scientists and machine learning engineers, including hiring, talent development, performance management, and building a strong technical leadership layer.
Set the technical vision and roadmap for autograder development, evaluation methodology, and toolings.
Deliver reliable, scalable LLM/VLM-based evaluation systems that can be trusted for product-quality decisions.
Navigate complex technical and organizational challenges, resolve cross-functional dependencies, and sustain alignment across organizations.
Identify opportunities to standardize and automate the end-to-end evaluation workflow, enabling the team to scale beyond bespoke autograder development.
MS/PhD degree in Computer Science, Machine Learning, AI, or a related field.
1+ years of industry experience building LLM/VLM-based evaluators, autograders, or automated evaluation systems.
5+ years of hands-on experience in machine learning, model alignment, or benchmark creation.
Track record of managing teams of 3+ machine learning engineers and/or AI scientists.
Deep understanding of GenAI models and their evaluation methodologies, including rubric design, evaluation set development, validation, calibration, and human-model alignment.
Strong technical judgment and a track record of leading complex, ambiguous AI/ML projects from definition through delivery.
Experience translating emerging GenAI techniques and research into reliable, production-ready systems.
Experience scaling AI/ML solutions into reusable platforms, frameworks, or standardized workflows.
Demonstrated track record of developing senior machine learning engineers and/or technical leads.
Demonstrated ability to align diverse stakeholders around strategy, navigate complex situations, and drive effective outcomes across organizations.

Nice-to-haves

None flagged.

Application prep

Use the role’s language and application details while tailoring your materials.

Keywords to mirror

aimachine learningllmml

Platform: apple
Effort: medium

Benefits

Benefits detected in the listing.

Vision