Job description

Software Engineer (AI Quality & Reliability) Location: New York City | In-office THE MISSION “Leading healthcare into the agentic era.” Every day, patients call their clinic to schedule an appointment, ask a question, or request a prescription refill. Behind each conversation is a busy healthcare team coordinating people, information, and systems. Taxo builds Voice AI agents that take on this administrative work. Our agents connect with EHR and practice management systems to help teams handle patient intake, complex scheduling, and refill requests, with reliable workflows and clear handoffs to people. THE OPPORTUNITY: MAKE AI QUALITY MEASURABLE As our Software Engineer (AI Quality & Reliability), you’ll build the automated evaluation and regression systems that show whether Taxo is ready to release. You’ll turn product acceptance criteria and real failure modes into repeatable tests across voice agents, application workflows, and integrations. This is a software engineering role focused on test infrastructure, independent quality evidence, and reliable releases. WHAT YOU’LL DO - Build the Evaluation System: Own shared evaluation frameworks, test harnesses, and versioned datasets using synthetic or appropriately authorized, privacy-safe data. Make runs repeatable and results easy to investigate. - Test the Whole Workflow: Build automated end-to-end regression coverage across conversations, tool calls, application behavior, and integration boundaries, including failures, recovery, and human escalation. - Measure AI Behavior: Evaluate task completion, tool-use correctness, latency, and conversation quality. Combine deterministic checks with calibrated model-assisted scoring and appropriate human review. - Make Releases Reviewable: Produce release-readiness evidence, maintain quality dashboards, and coordinate quality gates with product and engineering. Surface regressions, coverage gaps, and unresolved risks before launch. - Turn Failures into Tests: Reproduce defects, isolate failure patterns, and add durable regression cases. Engineering service owners retain component tests, production instrumentation, on-call response, and fixes; you independently verify the results. - Improve the Test System: Integrate automated checks into CI/CD, reduce flaky tests, and keep test environments and fixtures useful as the product evolves. Partner with domain reviewers on healthcare-specific cases and escalate questions that require clinical judgment. WHO YOU ARE - The Test Engineer: Strong software engineering skills in Python or TypeScript and experience building maintainable test automation, reusable fixtures, and CI pipelines. - The AI Evaluator: Practical experience evaluating LLM-powered applications, agents, or voice systems, including nondeterministic behavior and the limits of automated scoring. - The Systems Thinker: Experience testing APIs, asynchronous workflows, and distributed systems, with the ability to trace a failure across service boundaries. - The Investigator: A habit of turning ambiguous failures into reproducible cases, actionable evidence, and reliable regression coverage. - The Quality Partner: Clear communication about risk and release readiness, with sound judgment around sensitive data and human review. - Bonus: Experience with streaming audio, telephony, healthcare workflows, EHR integrations, or reliability testing in privacy-sensitive products. WHY TAXO Healthcare teams should be able to spend more time on patients and less time on administration. Help us build the products and company that make that possible. - 🧠 Practical AI Work: Build evaluation systems for real-world AI behavior across healthcare workflows. - 🏥 Impact That Matters: Help catch failures before they disrupt patients or clinic teams. - 🛠️ Technical Ownership: Shape shared evaluation and test infrastructure that improves every release. COMPENSATION Base salary: $130,000–$170,000 USD annually Equity: 0.10%–0.25% Benefits: Health coverage HOW WE WORK Taxo is a small, fast, high-conviction team. We care deeply about the problem, move with urgency, and take ownership of the work. We collaborate closely, make practical decisions, and keep improving through what we learn from customers. If you want to build the engineering systems that make healthcare AI more dependable, we’d love to hear from you.

Requirements

Must-haves

Software Engineer (AI Quality & Reliability)
Location: New York City | In-office
“Leading healthcare into the agentic era.”
Every day, patients call their clinic to schedule an appointment, ask a question, or request a prescription refill. Behind each conversation is a busy healthcare team coordinating people, information, and systems.
Build the Evaluation System: Own shared evaluation frameworks, test harnesses, and versioned datasets using synthetic or appropriately authorized, privacy-safe data. Make runs repeatable and results easy to investigate.
Test the Whole Workflow: Build automated end-to-end regression coverage across conversations, tool calls, application behavior, and integration boundaries, including failures, recovery, and human escalation.
Measure AI Behavior: Evaluate task completion, tool-use correctness, latency, and conversation quality. Combine deterministic checks with calibrated model-assisted scoring and appropriate human review.
The Test Engineer: Strong software engineering skills in Python or TypeScript and experience building maintainable test automation, reusable fixtures, and CI pipelines.
The AI Evaluator: Practical experience evaluating LLM-powered applications, agents, or voice systems, including nondeterministic behavior and the limits of automated scoring.
The Systems Thinker: Experience testing APIs, asynchronous workflows, and distributed systems, with the ability to trace a failure across service boundaries.
The Investigator: A habit of turning ambiguous failures into reproducible cases, actionable evidence, and reliable regression coverage.
The Quality Partner: Clear communication about risk and release readiness, with sound judgment around sensitive data and human review.
Healthcare teams should be able to spend more time on patients and less time on administration. Help us build the products and company that make that possible.
Practical AI Work: Build evaluation systems for real-world AI behavior across healthcare workflows.
Impact That Matters: Help catch failures before they disrupt patients or clinic teams.
Technical Ownership: Shape shared evaluation and test infrastructure that improves every release.
If you want to build the engineering systems that make healthcare AI more dependable, we’d love to hear from you.

Nice-to-haves

Bonus: Experience with streaming audio, telephony, healthcare workflows, EHR integrations, or reliability testing in privacy-sensitive products.

Application prep

Use the role’s language and application details while tailoring your materials.

Keywords to mirror

aici/cdpythontypescriptllm

Platform: ashby
Effort: quick