Research Engineer, Benchmarks

• Own the design, implementation, and quality of HUD’s internal agent benchmarks
• Work with subject-matter experts to define tasks and create domain-specific benchmarks that evaluate agents on realistic workflows
• Build infrastructure to reliably run models and agents against benchmark tasks
• Develop metrics and analyses to understand benchmark difficulty, reliability, and failure modes
• Validate whether benchmark performance correlates with real-world evals, customer needs, and lab expectations
• Write clear documentation and benchmark reports that make results legible and credible to technical audiences
• You may be a good fit if you have:
• Proficiency in Python, Docker, and Linux environments
• Published papers or written technical blogs on relevant topics such as public benchmarks and their limitations, model failure modes, etc. - please link in your application
• Strong understanding of what a “good benchmark” means and what makes one realistic, reliable, and useful
• Experience working on environments and evals
• Curiosity and ability to truly understand how workflows in various domains work
• Be detail-oriented and able to spot subtle inconsistencies or edge cases in tasks
• Be able to reason from first principles about task design, scoring, and failure modes
• Thrive in unstructured problem spaces
• Early-stage startup experience with ability to work independently in fast-paced environments
• Strong communication skills for remote collaboration across time zones
• We prioritize technical aptitude and learning potential over years of experience. Motivated candidates are encouraged to apply even if they don't meet all criteria.
• Team Size: ~25 people currently, mostly full-time in-person, but some remote.
• Our team: Our team includes 4 International Olympiad medalists (IOI, ILO, IPhO), the former #1 ranked Catan player in the world, serial AI startup founders, and researchers with publications at ICLR, NeurIPS, etc.
• Company stage: We have 8 figures in funding and are scaling profitably and quickly to meet very strong demand.
• Employment: Full-time.
• Location: We have offices in San Francisco or Singapore but are open to remote candidates who can work hours that 70-80% overlap with either San Francisco or Singapore time zones.
• Visa Sponsorship: We provide support for relocation and visas for strong full-time candidates to the US or Singapore.

None flagged.