Job description

Overview Microsoft 365 is seeking a Principal Product Manager to shape how builders prove, understand, and continuously improve the quality and value of Copilot plugins. You will own the strategy and roadmap for customer-facing plugin evaluations and quality measurement, giving builders and owners clear evidence that a plugin works, meets the intended quality bar, and delivers value and actionable guidance on what to do next. A plugin is a unit of Copilot extensibility that packages skills, connectors or MCP capabilities, and agents into a capability that can be discovered, invoked, and used consistently across products. As plugins become central to connecting AI experiences with enterprise data, tools, and business processes, builders and owners need a clear view of whether each plugin works, is being used, and delivers value. You will define an adaptive strategy for evaluation experiences and quality measurement across the plugin lifecycle, from pre-production evaluation through continuous improvement. In partnership with the plugin DevX architect and the Observability and Analytics product leader, you will connect evaluation findings with production signals to give builders and owners an end-to-end view of plugin value and a shared flywheel to improve, scale, consolidate, or retire plugins. As a Principal Product Manager , you will turn ambiguous developer and platform needs into durable product direction, lead complex evaluation and quality-measurement decisions, and align engineering, applied science, design, security, compliance, and partner teams around shared outcomes. You will define the evaluation contracts and feedback loops that connect pre-production evidence with the end-to-end developer journey and production signals, moving the ecosystem from isolated testing to continuous, evidence-driven improvement. This is an opportunity to define a foundational capability for Microsoft’s copilot ecosystem, influence how plugin quality is measured across Microsoft experiences, and make it dramatically easier for developers to ship reliable, secure, high-quality AI extensions with confidence. Microsoft’s mission is to empower every person and every organization on the planet to achieve more. As employees, we come together with a growth mindset, innovate to empower others, and collaborate to realize our shared goals. We build on our values of respect, integrity, and accountability to create a culture of inclusion where everyone can thrive at work and beyond. Responsibilities • • Own the product vision, strategy, investment priorities, and roadmap for Copilot plugin evaluation and quality measurement. • Establish quality and Responsible AI models, regression protections, and release criteria for groundedness, prompt injection, harmful content, reliability, and performance. • Build evaluation into CLI and IDE workflows, CI/CD, and publishing, enabling developers to create evaluation sets, interpret results, identify gaps, and improve plugins. • Connect evaluation findings with production telemetry, adoption, support, customer, and business-value signals through a shared scorecard that guides investments. • Validate product concepts through AI-assisted prototypes and translate them into architecture, APIs, reusable components, integration contracts, and clear engineering handoffs. • Represent the strategy and lead cross-organizational decisions, balancing developer value, platform reuse, responsible delivery, cost, and speed. • Define evaluation methods for nondeterministic outputs, multi-step agent and tool-use flows, LLM-as-judge scoring calibrated against human ratings, evaluation datasets, and offline-vs-online gaps. Qualifications Required Qualifications: • Bachelor's Degree AND 8+ years experience in product/service/program management or software development • OR equivalent experience. Preferred Qualifications: • Bachelor’s Degree AND 12+ years of experience in product management, service management, program management, software development, or a related field, OR equivalent experience. • Experience with AI or LLM evaluation, including nondeterministic outputs, LLM-as-judge methods, human-rating calibration, multi-step agent or tool-use evaluation, dataset development, or offline-to-online measurement. • Experience defining product strategy and leading developer platforms, extensibility systems, AI products, or evaluation capabilities from concept through launch and sustained adoption. • Demonstrated data-driven product management using evaluation findings, telemetry, customer feedback, adoption, or business-value signals to prioritize investments and measure outcomes. • Experience partnering with engineering on platform architecture, APIs, SDKs, extensibility models, data systems, or production services. • Hands-on software development or equivalent experience building prototypes and integrations using APIs, SDKs, or AI-assisted development tools. • Experience building products for technical audiences and translating developer needs into clear requirements, roadmaps, and measurable outcomes. • Experience influencing senior product and engineering stakeholders and leading complex technical initiatives across organizations without direct authority. Product Management IC5 - The typical base pay range for this role across the U.S. is USD $142,800 - $274,800 per year. There is a different range applicable to specific work locations, within the San Francisco Bay area and New York City metropolitan area, and the base pay range for this role in those locations is USD $188,000 - $304,200 per year. Certain roles may be eligible for benefits and other compensation. Find additional benefits and pay information here: careers.microsoft.com This position will be open for a minimum of 5 days, with applications accepted on an ongoing basis until the position is filled. Microsoft is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to age, ancestry, citizenship, color, family or medical care leave, gender identity or expression, genetic information, immigration status, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran or military status, race, ethnicity, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable local laws, regulations and ordinances. If you need assistance with religious accommodations and/or a reasonable accommodation due to a disability during the application process, read more about requesting accommodations.

Requirements

Must-haves

Own the product vision, strategy, investment priorities, and roadmap for Copilot plugin evaluation and quality measurement.
Establish quality and Responsible AI models, regression protections, and release criteria for groundedness, prompt injection, harmful content, reliability, and performance.
Build evaluation into CLI and IDE workflows, CI/CD, and publishing, enabling developers to create evaluation sets, interpret results, identify gaps, and improve plugins.
Connect evaluation findings with production telemetry, adoption, support, customer, and business-value signals through a shared scorecard that guides investments.
Validate product concepts through AI-assisted prototypes and translate them into architecture, APIs, reusable components, integration contracts, and clear engineering handoffs.
Represent the strategy and lead cross-organizational decisions, balancing developer value, platform reuse, responsible delivery, cost, and speed.
Define evaluation methods for nondeterministic outputs, multi-step agent and tool-use flows, LLM-as-judge scoring calibrated against human ratings, evaluation datasets, and offline-vs-online gaps.
Bachelor's Degree AND 8+ years experience in product/service/program management or software development • OR equivalent experience.
Bachelor’s Degree AND 12+ years of experience in product management, service management, program management, software development, or a related field, OR equivalent experience.
This position will be open for a minimum of 5 days, with applications accepted on an ongoing basis until the position is filled.

Nice-to-haves

Experience with AI or LLM evaluation, including nondeterministic outputs, LLM-as-judge methods, human-rating calibration, multi-step agent or tool-use evaluation, dataset development, or offline-to-online measurement.
Experience defining product strategy and leading developer platforms, extensibility systems, AI products, or evaluation capabilities from concept through launch and sustained adoption.
Demonstrated data-driven product management using evaluation findings, telemetry, customer feedback, adoption, or business-value signals to prioritize investments and measure outcomes.
Experience partnering with engineering on platform architecture, APIs, SDKs, extensibility models, data systems, or production services.
Hands-on software development or equivalent experience building prototypes and integrations using APIs, SDKs, or AI-assisted development tools.
Experience building products for technical audiences and translating developer needs into clear requirements, roadmaps, and measurable outcomes.
Experience influencing senior product and engineering stakeholders and leading complex technical initiatives across organizations without direct authority.
Certain roles may be eligible for benefits and other compensation. Find additional benefits and pay information here:

Application prep

Use the role’s language and application details while tailoring your materials.

Keywords to mirror

aici/cdllm

Platform: eightfold
Effort: medium

Benefits

Benefits detected in the listing.

Vision