PLATFORM RELIABILITY
Level: Staff (MTS 4) · Reports to: Chief AI Officer, later the Senior Engineering Manager
The role. Make our own platform dependable and scalable, working with the Principal Engineer. This is reliability engineering for Horizon AI’s software rather than for the GPU fleet, though the two meet when our agents are the ones being trusted with fleet actions.
You will
- Build the caching layer that stops every agent tool call from hitting external systems directly
- Implement event streaming so services stop depending on synchronous HTTP chains
- Harden agent dispatch: health checks, circuit breakers, retry semantics, and observability that distinguishes failure from degradation
- Build observability of our own platform: service-level metrics, tracing across services, and alerting we act on
- Contribute to the orchestration migration and to infrastructure-as-code
- Improve deployment practice: canary, traffic shifting, and real rollback
You have
- Production experience making distributed systems reliable: caching, queuing, circuit breaking, backpressure, timeouts, idempotency
- Comfort in Python and containerized environments, with real exposure to Kubernetes or equivalent orchestration
- Experience with event streaming and with the failure modes each option brings - Observability depth: metrics, distributed tracing, and the judgment to alert on symptoms rather than causes
- A record of shipping reliability improvements with measurable before and after
Nice to have: durable workflow orchestration, blue/green and canary deployment at scale, performance profiling of Python services