Job description

ABOUT THE ROLE Build evaluation environments and benchmarks that measure whether AI agents can perform real company work and match or exceed human performance. This founding engineering role focuses on agent capability assessment and production readiness, helping establish reliable evidence of agent competence. WHAT YOU'LL DO - Design synthetic company environments with realistic emails, records, and files generated from scratch. - Build graders and evaluation systems that can assess agents even when their performance may exceed human reviewers' expertise. - Develop benchmarks whose rankings correlate with real-world production performance. - Find and address evaluation failure modes, including reward hacking and benchmark gaming. - Simulate tools such as inboxes and CRMs with enough fidelity to support agent interactions at scale. WHAT WE'RE LOOKING FOR - Hands-on research experience with synthetic data pipelines, benchmarks, evaluations, or post-training. - At least 1 to 2 years of experience at a fast-growing or AI-focused startup. - Experience building environments or benchmarks for frontier AI training or evaluation is valuable. - A track record of finding flaws in established benchmarks, improving them, and addressing reward hacking in evaluation systems. - Ability to simulate complex software systems that can withstand agent tool use. - Startup experience, strong performance in competitive mathematics, programming, debate, or games, and a collaborative approach under pressure are valued. COMPENSATION & BENEFITS Base salary is $200,000 to $300,000 annually, with up to $350,000 for exceptional candidates. A commission-style bonus of up to $200,000 or more is tied to the value of delivered environments. Meaningful founding equity is available. Visa sponsorship is not available. LOCATION On-site five days a week in San Francisco, California, United States.

Requirements

Must-haves

Design synthetic company environments with realistic emails, records, and files generated from scratch.
Build graders and evaluation systems that can assess agents even when their performance may exceed human reviewers' expertise.
Develop benchmarks whose rankings correlate with real-world production performance.
Find and address evaluation failure modes, including reward hacking and benchmark gaming.
Simulate tools such as inboxes and CRMs with enough fidelity to support agent interactions at scale.
Hands-on research experience with synthetic data pipelines, benchmarks, evaluations, or post-training.
At least 1 to 2 years of experience at a fast-growing or AI-focused startup.
Experience building environments or benchmarks for frontier AI training or evaluation is valuable.
A track record of finding flaws in established benchmarks, improving them, and addressing reward hacking in evaluation systems.
Ability to simulate complex software systems that can withstand agent tool use.
Startup experience, strong performance in competitive mathematics, programming, debate, or games, and a collaborative approach under pressure are valued.
On-site five days a week in San Francisco, California, United States.

Nice-to-haves

None flagged.

Application prep

Use the role’s language and application details while tailoring your materials.

Keywords to mirror

Platform: ashby
Effort: quick