Job description

At Qualia, we've built the leading B2B real estate technology that transforms the home buying and selling experience into a simple, secure, and enjoyable process. Our SMB and Enterprise products bring together users from across the real estate ecosystem—homebuyers and sellers, lenders, title and escrow agents, and real estate agents—onto a single shared digital closing platform, providing greater clarity and transparency to real estate transactions. Today, through our business customers across the country, millions of consumers use Qualia to close on homes every year. WHAT YOU'LL WORK ON We are hiring pioneering and collaborative engineers to build our next generation of intelligent products. Our quality engineers are conscientious hackers who thrive on collaboration and are adept at driving large-scale projects in the fast-changing domain of artificial intelligence. You will help build the automation, tooling, observability, and quality systems that let us ship complex, AI-driven products with confidence, bringing rigor and trust to a data-rich industry. As a Senior AI/Agentic Quality Infrastructure SDET, you're a software engineer first. You build test infrastructure, agentic validation systems, AI/ML evaluation harnesses, observability platforms, and engineering tooling—not manual test cases or collections of automation scripts. You will help evolve our quality architecture from primarily deterministic automation toward a combination of traditional automation and agentic validation, building systems where AI agents can understand application state, execute workflows, use engineering tools, validate outcomes, investigate failures, and provide evidence for their conclusions. You can take a hard, loosely defined problem and solve it. You're trusted to make technical decisions on frameworks, architecture, evaluation strategy, observability, and validation across every layer of the testing pyramid. You work closely with Engineering and Quality leadership, understand the systems you're testing at a deep technical level, and mentor engineers on building reliable tests, tooling, infrastructure, and AI-assisted validation systems. RESPONSIBILITIES Build and maintain test infrastructure, AI/ML evaluation systems, agentic validation harnesses, release orchestration tooling, observability, and performance testing platforms as production-grade software rather than scripts Design agentic test harnesses that let AI agents interact with applications, browsers, APIs, databases, logs, telemetry, and CI/CD systems to execute and validate complex workflows, reasoning about application state rather than relying solely on predefined test scripts Build AI/ML evaluation frameworks for validating non-deterministic systems, including deterministic assertions, semantic and model-based evaluation, regression datasets, and human-in-the-loop review, defining measurable quality signals such as correctness, reliability, latency, cost, and hallucination/error rates Build observability and telemetry into our automation and agentic infrastructure—structured logs, metrics, distributed tracing, and execution timelines—so engineers can understand why an automated or agentic workflow succeeded or failed Develop systems that automatically distinguish product, test, infrastructure, and AI/model reasoning failures, correlating them with logs, traces, deployments, and commits to accelerate root-cause analysis Design and build our test automation frameworks, including the migration from Cypress to Playwright, and own core infrastructure such as test environment provisioning, parallelization, and CI/CD integration at scale Apply AI to engineering-quality problems including test generation, flaky-test triage, failure clustering, and release-risk analysis, while building guardrails so agent conclusions are backed by observable evidence Partner with Software Engineers, Quality Engineers, SRE, and Platform teams to make these capabilities reusable platforms, and mentor engineers on test architecture, observability, and AI-assisted engineering YOUR BACKGROUND THAT LIKELY MAKES YOU A MATCH 5+ years of experience as an SDET or software engineer building test infrastructure, developer tooling, platform engineering, observability, or release engineering systems—not primarily writing test cases or automation scripts Strong programming ability in JavaScript/TypeScript, including designing backend services, libraries, frameworks, and production-quality infrastructure from scratch Strong software engineering fundamentals including system design, distributed systems, APIs, asynchronous systems, and production operations Experience building or scaling CI/CD systems, release orchestration, test infrastructure, or developer platforms, and designing observability/telemetry systems using structured logging, metrics, and distributed tracing Hands-on experience with LLMs, AI-assisted engineering, or agentic systems beyond simply using an AI coding assistant, including tool/function calling, agent workflows, and multi-step AI workflows Experience or familiarity with LLM and agent evaluation, including evaluation datasets, deterministic and model-based evaluators, and the unique challenges of testing probabilistic/non-deterministic systems Strong understanding of the testing pyramid and experience determining when validation belongs at the unit, integration, API, UI, performance, or agentic layer Comfortable mentoring engineers and raising the technical quality bar across teams THE TECH STACK YOU'LL USE JavaScript / Node.js / TypeScript Playwright (browser & test automation) LLM APIs, tool/function calling, agent frameworks, structured outputs, RAG/context retrieval Jest (unit/integration testing) Postman / REST Assured (API testing) Locust / Gatling / JMeter / k6 (performance testing) Jenkins / GitHub Actions / ArgoCD / Spinnaker / Harness (CI/CD & release orchestration) Datadog, OpenTelemetry (observability & telemetry) MongoDB Microservice architecture / Kubernetes AWS / Google Cloud We are a remote-first development organization that values flexibility in how our teams work. Because of that, while the role is based in Austin, TX, we are open to remote candidates for this position. We have other physical office locations in San Francisco, California, and Concord, New Hampshire. WHY QUALIA Qualia is made up of incredibly bright, mission-driven coworkers who are passionate about using technology to solve real-world problems—and we're growing quickly. In order to continue building an engaging and dynamic organization, we're committed to giving everyone the support they need to do great work. Our benefits package is designed to allow our team members to be their best selves, both in and out of the workplace. In addition to comprehensive health plans, a 401k program, and commuter benefits, we prioritize family and personal well-being through professional development, parental leave, and a flexible time off policy. Qualia offers a robust online onboarding program to train new hires, biweekly all hands meetings, and a variety of internal virtual events to keep employees connected. We believe diverse perspectives and backgrounds are critical to building great technology, and our goal is to cultivate an environment where people feel equally valued and respected. Qualia is proud to be an equal-opportunity workplace, and we welcome applicants from all backgrounds regardless of race, color, ancestry, religion, gender identity or expression, sexual orientation, marital status, age, citizenship, socioeconomic status, disability, or veteran status. By submitting your application, you acknowledge and agree to the collection, processing, and use of your personal information as described in our Employee Data Privacy Notice . #LI-Remote

Requirements

Must-haves

Develop systems that automatically distinguish product, test, infrastructure, and AI/model reasoning failures, correlating them with logs, traces, deployments, and commits to accelerate root-cause analysis
Design and build our test automation frameworks, including the migration from Cypress to Playwright, and own core infrastructure such as test environment provisioning, parallelization, and CI/CD integration at scale
Apply AI to engineering-quality problems including test generation, flaky-test triage, failure clustering, and release-risk analysis, while building guardrails so agent conclusions are backed by observable evidence
Partner with Software Engineers, Quality Engineers, SRE, and Platform teams to make these capabilities reusable platforms, and mentor engineers on test architecture, observability, and AI-assisted engineering
YOUR BACKGROUND THAT LIKELY MAKES YOU A MATCH
Strong programming ability in JavaScript/TypeScript, including designing backend services, libraries, frameworks, and production-quality infrastructure from scratch
Strong software engineering fundamentals including system design, distributed systems, APIs, asynchronous systems, and production operations
Experience building or scaling CI/CD systems, release orchestration, test infrastructure, or developer platforms, and designing observability/telemetry systems using structured logging, metrics, and distributed tracing
Hands-on experience with LLMs, AI-assisted engineering, or agentic systems beyond simply using an AI coding assistant, including tool/function calling, agent workflows, and multi-step AI workflows
Experience or familiarity with LLM and agent evaluation, including evaluation datasets, deterministic and model-based evaluators, and the unique challenges of testing probabilistic/non-deterministic systems
Strong understanding of the testing pyramid and experience determining when validation belongs at the unit, integration, API, UI, performance, or agentic layer
Comfortable mentoring engineers and raising the technical quality bar across teams
JavaScript / Node.js / TypeScript
Playwright (browser & test automation)
LLM APIs, tool/function calling, agent frameworks, structured outputs, RAG/context retrieval
Jest (unit/integration testing)
Postman / REST Assured (API testing)
Locust / Gatling / JMeter / k6 (performance testing)
Jenkins / GitHub Actions / ArgoCD / Spinnaker / Harness (CI/CD & release orchestration)
Datadog, OpenTelemetry (observability & telemetry)
Microservice architecture / Kubernetes
AWS / Google Cloud
By submitting your application, you acknowledge and agree to the collection, processing, and use of your personal information as described in our Employee Data Privacy Notice .

Nice-to-haves

None flagged.

Application prep

Use the role’s language and application details while tailoring your materials.

Keywords to mirror

aimlci/cdjavascripttypescriptllmapinode.jsrestkubernetesaws

Platform: greenhouse
Effort: quick

Benefits

Benefits detected in the listing.

Vision

401k

Parental Leave

Professional Development