AIML - Applied AI Scientist, Image Autograder Systems, Evaluation

• In this role you will focus on Autograder research, training and adoption.
• Collaborate with AI feature teams, eval design teams, and annotation teams to refine feature requirements, grading rubrics and gold annotation sets.
• Develop, evaluate, and iterate on grading prompts to align autograder behavior with grading rubrics and the gold sets.
• Identify when prompt tuning reaches its limits and apply other advanced techniques such as fine-tuning to close remaining grading accuracy gaps.
• Design insightful analysis to measure and explain Autograder quality.
• Collaborate with MLEs and feature teams on autograder deployment and adoption.
• Develop scalable system to speed up autograder training processes.
• Master's or PhD in Computer Science, Machine Learning, Artificial Intelligence, or a related field.
• Deep understanding of visual-language models.
• Familiarity with image quality assessment - perceptual quality dimensions, evaluation metrics, and rubric design.
• Proficiency in Python; capable of writing well-structured, production-ready model code.
• Strong communication skills in explaining autograder quality and driving autograder adoption with partner teams.
• 1+ years of industry experience in building VLM-based products.
• Familiarity with autograder and evaluator-specific concepts: grading accuracy, agreement with human raters, calibration, and rubric design.
• Strong expertise in prompt tuning and fine tuning for VLMs.
• Demonstrated ability to read AI literature and translate it into applied autograder experiments.
• Prior experience in building agentic system to scale autograder training/validation.

None flagged.