Job description

Machine Learning Engineer - Text and Evals 📍 Location: Bay Area / San Mateo, CA 💼 Employment Type: Full-Time 🏢 Department: Engineering About Us Deccan AI is a model training and evaluation startup headquartered in the Bay Area, with a delivery office in Hyderabad. We are founded by IIT Bombay, IIM Ahmedabad, and ex-Google alumni, and we work with leading AI labs and enterprise teams on high-quality human data, evaluations, and AI-first scaled operations. We believe frontier AI systems are only as good as the data, evaluations, and human judgment behind them. Our work sits at the intersection of customer needs, research ambition, and operational execution. That makes the work high-touch, technical, ambiguous, and execution-heavy. We are looking for people who can operate with urgency, write clearly, build trust, and turn ambiguity into outcomes. About the Role We are hiring a Machine Learning Engineer - Text and Evals to build Deccan’s fast-evals and text-evaluation capability. Frontier models still fail in important text workflows, especially in languages, domains, and cultural contexts that are underrepresented in training data. Deccan is building a fast eval motion: short, high-signal benchmark studies that identify where models break, publish useful findings, and create a path into deeper training-data and model-improvement work. What You’ll Do • Build and maintain evaluation pipelines for text, language, and domain-specific model failures. • Design fast benchmark workflows that can produce high-signal eval reports on a weekly or near-weekly cadence. • Create prompt sets, rubrics, failure taxonomies, judge / verifier logic, and benchmark charts that are technically defensible. • Run evals across frontier proprietary models and open-source models, then analyze failure patterns with enough nuance to be useful to model teams. • Work with language experts, SMEs, delivery, and operations to collect prompts, validate responses, and surface culturally or linguistically meaningful failures. • Build tooling that makes eval production faster: model-run scripts, scoring pipelines, data validation, report templates, and reproducible analysis notebooks or scripts. • Help define Deccan’s point of view on languages, evals, localized reasoning, code-mixing, domain-specific text workflows, and model failure modes. • Partner with GTM and Engagement Managers so eval findings can become credible customer conversations without turning the work into shallow marketing. • Train operators and analysts to run repeatable eval workflows without lowering the technical quality bar. What You Will Own • Technical systems for Deccan’s text and language evaluation practice. • Prompt-set design, rubric structure, failure taxonomy, and scoring methodology. • Model-run infrastructure and reproducible benchmark analysis. • Judge, verifier, and quality-control logic for text evals. • Technical review of eval outputs before they are shared externally. • Playbooks that let Deccan repeat fast evals across languages, domains, and customer contexts. • The bridge between SME-generated content and model-facing evaluation rigor. What We’re Looking For: • able build model-evaluation pipelines, deterministic and non-deterministic scoring systems, llm-judges, verifiers etc • able to identify subtle model failures in language, reasoning, instruction-following, domain knowledge, and cultural context; • produce fast studies and research like reports on relevant and hot topics • comfortable working with SMEs and language experts without outsourcing your judgment to them; • explain methodology, limitations, and findings to both technical and non-technical audiences. Preferred Qualifications • 3+ years of experience in ML engineering, data science, model evaluation, NLP, applied AI, LLM tooling, or related work; exceptional earlier-career candidates with strong evidence of ability will also be considered. • Hands-on experience with LLM evaluation, benchmark design, prompt engineering, automated grading, rubric design, data quality, or model-failure analysis. • Strong programming ability in Python and modern ML / data tooling. Sharp at thinking algorithimically. • Experience with multilingual NLP, text classification, information extraction, instruction-following evals, safety evals, or domain-specific benchmarks is a plus. • Familiarity with frontier and open-source model APIs, eval harnesses, data pipelines, notebooks, and reproducible analysis workflows. • Strong written communication; you can describe methodology and failure examples without hiding behind vague metrics. • Experience working with language experts, SMEs, annotators, or human-in-the-loop data workflows. Why Join Deccan AI At Deccan AI, you will work close to the frontier of AI model training, evaluation, and scaled human-data operations. The best evals do more than rank models: they reveal where systems break, why they break, and what data or workflow could improve them. This is a high-agency role for someone who wants to build that machine. You will help Deccan move quickly, publish useful findings, and turn text and language evals into a durable technical capability. If you are excited by LLM evals, multilingual NLP, model failure analysis, and the chance to build a fast-evals practice from zero to one, we would like to meet you.

Requirements

Must-haves

Machine Learning Engineer - Text and Evals
Location: Bay Area / San Mateo, CA
Employment Type: Full-Time
Department: Engineering
We are hiring a Machine Learning Engineer - Text and Evals to build Deccan’s fast-evals and text-evaluation capability.
What You’ll Do
Build and maintain evaluation pipelines for text, language, and domain-specific model failures.
Design fast benchmark workflows that can produce high-signal eval reports on a weekly or near-weekly cadence.
Create prompt sets, rubrics, failure taxonomies, judge / verifier logic, and benchmark charts that are technically defensible.
Run evals across frontier proprietary models and open-source models, then analyze failure patterns with enough nuance to be useful to model teams.
Work with language experts, SMEs, delivery, and operations to collect prompts, validate responses, and surface culturally or linguistically meaningful failures.
Build tooling that makes eval production faster: model-run scripts, scoring pipelines, data validation, report templates, and reproducible analysis notebooks or scripts.
Help define Deccan’s point of view on languages, evals, localized reasoning, code-mixing, domain-specific text workflows, and model failure modes.
Partner with GTM and Engagement Managers so eval findings can become credible customer conversations without turning the work into shallow marketing.
Train operators and analysts to run repeatable eval workflows without lowering the technical quality bar.

Nice-to-haves

Hands-on experience with LLM evaluation, benchmark design, prompt engineering, automated grading, rubric design, data quality, or model-failure analysis.
Strong programming ability in Python and modern ML / data tooling. Sharp at thinking algorithimically.
Experience with multilingual NLP, text classification, information extraction, instruction-following evals, safety evals, or domain-specific benchmarks is a plus.
Familiarity with frontier and open-source model APIs, eval harnesses, data pipelines, notebooks, and reproducible analysis workflows.
Strong written communication; you can describe methodology and failure examples without hiding behind vague metrics.
Experience working with language experts, SMEs, annotators, or human-in-the-loop data workflows.
Why Join Deccan AI
This is a high-agency role for someone who wants to build that machine. You will help Deccan move quickly, publish useful findings, and turn text and language evals into a durable technical capability.
If you are excited by LLM evals, multilingual NLP, model failure analysis, and the chance to build a fast-evals practice from zero to one, we would like to meet you.

Application prep

Use the role’s language and application details while tailoring your materials.

Keywords to mirror

machine learningaillmmlpython

Platform: dover
Effort: medium