Job description

Overview Outlook Science is building the intelligence layer behind the next generation of AI-powered communication and productivity experiences. Our work spans quality, evaluation, post-training, personalization, and agentic AI, with a focus on turning frontier-model capabilities into reliable product experiences used on a global scale. We are looking for a  Principal Applied Science Manager  to lead a team responsible for improving the quality of some of Outlook’s most important AI experiences. This role sits at the intersection of frontier models, product science, and large-scale AI systems, with the opportunity to shape how AI quality is measured and improved across Outlook. The successful candidate will help answer some of the most important questions in applied AI today: How do we know an agent is actually getting better? How do we connect offline evaluation improvements to real user impact? How do we make AI systems understand user context, identify what matters, take the right actions, and earn user trust? Microsoft’s mission is to empower every person and every organization on the planet to achieve more. As employees we come together with a growth mindset, innovate to empower others, and collaborate to realize our shared goals. Each day we build on our values of respect, integrity, and accountability to create a culture of inclusion where everyone can thrive at work and beyond. Responsibilities • Lead and grow a team of applied scientists focused on AI quality, evaluation, post-training, and product improvement. • Define what “good” looks like for AI-powered communication, summarization, search, recommendation, and agentic experiences. • Build scalable evaluation systems combining human labels, LLM graders, telemetry, shadow experiments, online A/B testing, and root-cause analysis. • Drive improvements across critical quality dimensions, including recall, precision, grounding, freshness, ranking, personalization, actionability, hallucination reduction, and end-to-end task completion. • Diagnose complex production failures involving user context, long-running conversations, tool use, memory, retrieval, ranking, and evolving product state. • Translate user feedback, dissatisfaction signals, and production telemetry into actionable science and product insights. • Partner closely with PM and engineering leaders to turn evaluation findings into model, prompt, tool, data, and system improvements. • Develop post-training and adaptation strategies for frontier and open-source models where they can materially improve product quality. • Establish deep connections between offline quality metrics and online outcomes such as user satisfaction, engagement, retention, and successful task completion. • Provide technical leadership across teams and mentor senior scientists while maintaining a high bar for scientific rigor, product judgment, and execution. Qualifications Required Qualifications: • Bachelor's Degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 8+ years related experience (e.g., statistics, predictive analytics, research) • OR Master's Degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 6+ years related experience (e.g., statistics, predictive analytics, research) • OR Doctorate in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 5+ years related experience (e.g., statistics, predictive analytics, research) • OR equivalent experience. • 3+ years of people management experience. Other Requirements: Ability to meet Microsoft, customer and/or government security screening requirements are required for this role. These requirements include but are not limited to the following specialized security screenings: Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter. Preferred Qualifications: • 10+ years of experience in applied science, machine learning, artificial intelligence, data science, or large-scale production AI systems. • Experience leading applied science teams or major cross-functional science initiatives. • Demonstrated experience building, evaluating, or improving production AI or machine-learning systems. • Background in experimentation, metrics, telemetry analysis, evaluation, or systematic product-quality improvement. • Experience partnering closely with engineering and product teams to translate scientific insights into production impact. • Ability to explain complex technical tradeoffs and quality issues to senior stakeholders. • PhD in Computer Science, Artificial Intelligence, Machine Learning, NLP, Information Retrieval, Data Mining, or a related technical field. • Deep experience with LLMs, agentic systems, post-training, tool use, AI evaluation, or GenAI product development. • Experience in one or more of NLP, information retrieval, ranking, recommendation, personalization, summarization, search, or productivity AI. • Experience building large-scale evaluation systems using human judgment, LLM graders, telemetry, shadow A/B experiments, or online experimentation. • Track record of improving production AI quality across dimensions such as recall, precision, grounding, actionability, personalization, and user trust. • Experience connecting offline evaluation improvements to measurable online product outcomes. • Experience leading high-visibility AI initiatives in fast-moving product environments. • Publication, patent, or research record is valued, but demonstrated product and scientific impact is equally important. #Outlook; #OPG Applied Sciences M6 - The typical base pay range for this role across the U.S. is USD $165,600 - $296,400 per year. There is a different range applicable to specific work locations, within the San Francisco Bay area and New York City metropolitan area, and the base pay range for this role in those locations is USD $220,800 - $331,200 per year. Certain roles may be eligible for benefits and other compensation. Find additional benefits and pay information here: careers.microsoft.com This position will be open for a minimum of 5 days, with applications accepted on an ongoing basis until the position is filled. Microsoft is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to age, ancestry, citizenship, color, family or medical care leave, gender identity or expression, genetic information, immigration status, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran or military status, race, ethnicity, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable local laws, regulations and ordinances. If you need assistance with religious accommodations and/or a reasonable accommodation due to a disability during the application process, read more about requesting accommodations.

Requirements

Must-haves

Lead and grow a team of applied scientists focused on AI quality, evaluation, post-training, and product improvement.
Define what “good” looks like for AI-powered communication, summarization, search, recommendation, and agentic experiences.
Build scalable evaluation systems combining human labels, LLM graders, telemetry, shadow experiments, online A/B testing, and root-cause analysis.
Drive improvements across critical quality dimensions, including recall, precision, grounding, freshness, ranking, personalization, actionability, hallucination reduction, and end-to-end task completion.
Diagnose complex production failures involving user context, long-running conversations, tool use, memory, retrieval, ranking, and evolving product state.
Translate user feedback, dissatisfaction signals, and production telemetry into actionable science and product insights.
Partner closely with PM and engineering leaders to turn evaluation findings into model, prompt, tool, data, and system improvements.
Develop post-training and adaptation strategies for frontier and open-source models where they can materially improve product quality.
Establish deep connections between offline quality metrics and online outcomes such as user satisfaction, engagement, retention, and successful task completion.
Provide technical leadership across teams and mentor senior scientists while maintaining a high bar for scientific rigor, product judgment, and execution.
OR Doctorate in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 5+ years related experience (e.g., statistics, predictive analytics, research)
OR equivalent experience.

Nice-to-haves

Experience leading applied science teams or major cross-functional science initiatives.
Demonstrated experience building, evaluating, or improving production AI or machine-learning systems.
Background in experimentation, metrics, telemetry analysis, evaluation, or systematic product-quality improvement.
Experience partnering closely with engineering and product teams to translate scientific insights into production impact.
Ability to explain complex technical tradeoffs and quality issues to senior stakeholders.
PhD in Computer Science, Artificial Intelligence, Machine Learning, NLP, Information Retrieval, Data Mining, or a related technical field.
Deep experience with LLMs, agentic systems, post-training, tool use, AI evaluation, or GenAI product development.
Experience in one or more of NLP, information retrieval, ranking, recommendation, personalization, summarization, search, or productivity AI.
Experience building large-scale evaluation systems using human judgment, LLM graders, telemetry, shadow A/B experiments, or online experimentation.
Track record of improving production AI quality across dimensions such as recall, precision, grounding, actionability, personalization, and user trust.
Experience connecting offline evaluation improvements to measurable online product outcomes.
Experience leading high-visibility AI initiatives in fast-moving product environments.

Application prep

Use the role’s language and application details while tailoring your materials.

Keywords to mirror

aillmmachine learning

Platform: eightfold
Effort: medium