Job description

NVIDIA is seeking a highly analytical Product Evaluations Lead to own the evaluation, measurement, and go-forward analysis of our Generative AI Software including the Nemotron family as they are adopted by our most strategic enterprise ISV partners. As the evaluation's driver for our most strategic partners and products, you will help drive the experiments, benchmarks, and translate results into crisp insights and prioritized next steps that guide Product, Research, and leadership. This role operates at the intersection of rigorous data science, GenAI product strategy, and executive communication, staying ahead of the rapidly advancing reasoning and agentic AI landscape. You will directly influence model release criteria, product direction, and the roadmap for how NVIDIA's GenAI software is measured and improved! What You Will Be Doing: • Drive end-to-end evaluation and analysis of NVIDIA's GenAI libraries, primarily Nemotron models for a focused set of strategic ISV partners, translating complex model behavior into clear go/no-go recommendations and prioritized next steps. • Develop and own rigorous evaluation frameworks, partner experimentation, and benchmarks that measure model quality, reasoning, and agentic capabilities against partner requirements and real-world enterprise use cases. • Share learnings and impact. Deliver regular readouts, dashboards, and progress tracking on evaluation results, adoption status, and recommended next steps to Product, Engineering, Research, and leadership stakeholders, consistent with NVIDIA's culture. • Provide analytical judgment under high uncertainty, balancing model quality, risk, and impact from incomplete or noisy signals to inform high-stakes release and integration decisions. • Partner with research scientists and product teams to translate emerging model capabilities into measurable release criteria, and identify the data and signals that feed the model-improvement flywheel. • Represent partner and product evaluation needs to internal teams. Contribute to the product roadmap by synthesizing cross-partner and cross-industry patterns captured from strategic engagements. What we need to see: • 12+ years of experience in data science, model evaluation, experimentation, or analytics roles, with a focus on measuring AI/ML or GenAI systems. • Master's or PhD in a quantitative field (e.g., Data Science, Statistics, Computer Science, Economics) or equivalent experience. • Proven track record leading model evaluation and experimentation that directly informed high-stakes, go/no-go product or model-release decisions. • Strong hands-on skills in Python and statistical/experimental methods, including A/B testing, causal measurement (e.g., org-level holdouts), and metric design and failure analysis. • Direct experience evaluating and benchmarking LLMs or GenAI systems including reasoning, agentic, and/or multimodal capabilities, and building high-quality evaluation datasets and human-evaluation strategies. • Ability to translate complex, ambiguous model behavior into clear narratives, reporting, and recommendations for research, product, and executive audiences. • Proven ownership operating cross-functionally across research, engineering, product, and leadership to drive alignment and decisions in a high-velocity environment. Ways to Stand Out from the Crowd: • Founding or early-team experience standing up a GenAI evaluation or analytics function defining north-star metrics and rapid-experimentation frameworks that accelerated and de-risked release decisions. • Track record of influencing complex model and product decisions through positive relationships, showing empathy for partner needs and an instinct for translating them into measurable improvements. • Hands-on experience with reward models, data flywheels, or evaluation signals that directly improved model performance. • Published or applied work in LLM capability evaluation or benchmarking (e.g., peer-reviewed research), paired with agility in a high-velocity GenAI landscape. • Highly collaborative standout colleague, able to build deep trust with engineers, researchers, executives, and multi-functional teams at both NVIDIA and partner organizations. We're entering a defining chapter in NVIDIA's history! This role offers an outstanding opportunity to build how our Enterprise Generative AI Software is measured, trusted, and adopted by the world's leading partners. You'll have the support of Engineering, NVIDIA Research, Solutions Architects, and Product teams, and your analysis will directly influence which models ship and how our Enterprise AI business grows. This is an unparalleled opportunity for professional growth and impact in a high visibility team that’s at the forefront of innovation. NVIDIA is widely considered one of the technology world’s most desirable employers. We have some of the world's most forward-thinking and hardworking people on our team. If you're creative and autonomous, we want to hear from you! NVIDIA benefits is available online at Benefits and Support Programs | NVIDIA Benefits #LI-Hybrid Your base salary will be determined based on your location, experience, and the pay of employees in similar positions. The base salary range is 224,000 USD - 356,500 USD. You will also be eligible for equity and benefits . Applications for this job will be accepted at least until September 1, 2026. This posting is for an existing vacancy. NVIDIA uses AI tools in its recruiting processes. NVIDIA is committed to fostering an inclusive work environment and proud to be an equal opportunity employer. As we highly value diversity in our current and future employees, we do not discriminate (including in our hiring and promotion practices) on the basis of race, religion, color, national origin, gender, gender expression, sexual orientation, age, marital status, veteran status, disability status or any other characteristic protected by law.

Requirements

Must-haves

Develop and own rigorous evaluation frameworks, partner experimentation, and benchmarks that measure model quality, reasoning, and agentic capabilities against partner requirements and real-world enterprise use cases.
Provide analytical judgment under high uncertainty, balancing model quality, risk, and impact from incomplete or noisy signals to inform high-stakes release and integration decisions.
Partner with research scientists and product teams to translate emerging model capabilities into measurable release criteria, and identify the data and signals that feed the model-improvement flywheel.
Represent partner and product evaluation needs to internal teams. Contribute to the product roadmap by synthesizing cross-partner and cross-industry patterns captured from strategic engagements.
12+ years of experience in data science, model evaluation, experimentation, or analytics roles, with a focus on measuring AI/ML or GenAI systems.
Master's or PhD in a quantitative field (e.g., Data Science, Statistics, Computer Science, Economics) or equivalent experience.
Proven track record leading model evaluation and experimentation that directly informed high-stakes, go/no-go product or model-release decisions.
Strong hands-on skills in Python and statistical/experimental methods, including A/B testing, causal measurement (e.g., org-level holdouts), and metric design and failure analysis.
Direct experience evaluating and benchmarking LLMs or GenAI systems including reasoning, agentic, and/or multimodal capabilities, and building high-quality evaluation datasets and human-evaluation strategies.
Ability to translate complex, ambiguous model behavior into clear narratives, reporting, and recommendations for research, product, and executive audiences.
Proven ownership operating cross-functionally across research, engineering, product, and leadership to drive alignment and decisions in a high-velocity environment.
Founding or early-team experience standing up a GenAI evaluation or analytics function defining north-star metrics and rapid-experimentation frameworks that accelerated and de-risked release decisions.
Track record of influencing complex model and product decisions through positive relationships, showing empathy for partner needs and an instinct for translating them into measurable improvements.
Hands-on experience with reward models, data flywheels, or evaluation signals that directly improved model performance.
Published or applied work in LLM capability evaluation or benchmarking (e.g., peer-reviewed research), paired with agility in a high-velocity GenAI landscape.
Highly collaborative standout colleague, able to build deep trust with engineers, researchers, executives, and multi-functional teams at both NVIDIA and partner organizations.
You will also be eligible for equity and benefits .
Applications for this job will be accepted at least until September 1, 2026.
This posting is for an existing vacancy.
NVIDIA uses AI tools in its recruiting processes.

Nice-to-haves

None flagged.

Application prep

Use the role’s language and application details while tailoring your materials.

Keywords to mirror

goaimlpythonllm

Platform: eightfold
Effort: medium