Staff/Senior Distributed Systems Engineer
New York City | Full-time | On-site in SoHo, five days per week
Reports to the CTO and works directly with the founding team.
ABOUT HELIOS
Helios is building a new kind of company to solve America’s hardest problems, starting with the government interaction layer.
Government shapes every consequential market, but the infrastructure connecting public institutions and private organizations remains fragmented, manual, and difficult to navigate. Helios is rebuilding that layer.
Our core platform, Proxi, gives organizations the intelligence they need to understand what government is doing, why it matters, and what to do next. From that foundation, we design and deploy secure, mission-specific systems for government agencies, enterprises, and institutions operating in complex and highly regulated environments.
We bring together frontier AI, deep public-sector expertise, and forward-deployed execution. Our team includes leaders and builders from the White House, U.S. Department of State, Datadog, and Microsoft. We are backed by leading institutional investors and trusted by organizations working on high-stakes problems across government and industry.
MISSION
As a distributed systems engineer at Helios you will expand and operate the distributed execution, storage, scheduling, and reliability primitives that support data ingestion, document processing, search indexing, model inference, agents, and continuously running research workflows.
The primary responsibility centers around the orchestration of critical infrastructure to meet mission-critical performance and reliability guarantees. Our platform runs a multitude of network crawling, compute grids, memory-heavy document processing, OCR, inference, and latency-sensitive jobs, all of which you should be familiar with. Preferred areas of expertise include:
- Queue-, log-, workflow-, and actor-based distributed execution systems, including Kafka, Redpanda, Pulsar, SQS, Pub/Sub, RabbitMQ, Temporal, and equivalent technologies.
- Establishment of at-least-once delivery with effectively once-only business outcomes through transactional outbox and inbox patterns, sagas, reconciliation, dead-letter handling, replay and historical backfill procedures.
- CPU-, memory-, disk-, network-, and GPU-aware scheduling, including workload classification, priority allocation, starvation prevention, tenant and dependency concurrency limits, placement constraints, resource quotas, and noisy-neighbor isolation.
- PostgreSQL, object storage, Redis or equivalent caches, search indices (Elasticsearch or Typesense), vector stores, change-data-capture, and graph-storage systems.
- Application of distributed coordination and concurrency controls, including leader election, locks, leases, fencing tokens, optimistic concurrency, conflict resolution, and partition recovery.
- Provisioning and operation of AWS, GCP, or Azure infrastructure through terraform or similar IaC, vulnerability scanning, CI/CD and deployment methods.
- Definition and administration of SLO/Is, error budgets, release controls, metrics, OpenTelemetry, Datadog, fault injection and load/failure testing.
- Evaluation of system performance and cost via e2e latency decomposition, resource and dependency profiling.
- Enforcement of platform security and tenant isolation policies.
The key objective of this work is to support the real-time access and availability of our entire data corpus as well as supporting the continued construction of the Helios Rapid Ontology System (H.R.O.S.), our long horizon memory data plane.
KEY RESPONSIBILITIES
- Own the operation and architecture of Proxi’s distributed execution platform, covering ingestion, document processing, search index performance and long-running research jobs.
- Manage the deployment of our GovCloud and Air-Gapped resources for sensitive environments.
- Manage the orchestration of long-running agent research tasks including scheduling, lease management and retention.
- Queue optimization and cross-cloud information pipeline scalability.
- Build out dedicated resource-aware autoscaling architecture for fast search and document processing resources.
- Manage CI/CD and compliance operations across the entire Helios platform.
- Support global forward embedded customer infrastructure efforts.
WORKING AT HELIOS
This is a full-time, in-person role based in our SoHo office in New York City. Team members are expected to work from the office five days per week.
Certain customer engagements may require background checks, security reviews, access approvals, or eligibility for a U.S. government security clearance. Some projects may be subject to U.S. citizenship or other customer-specific access requirements.
We're not looking for passengers; we want driven innovators with a hunger to build from the ground up - obsessed with pushing the boundaries of natural language understanding, document processing, and personalized relevance.
Helios is a fast-moving startup with ambitious goals. This is not a conventional 9-to-5 role. We expect flexibility during critical deployment periods, customer incidents, product launches, and other company-critical work. In return, this role offers unusual ownership, direct access to consequential institutions, and the opportunity to build systems that affect how major decisions are made.
HOW WE WORK
- Own outcomes, not just assigned tasks. Identify what needs to happen and drive it through completion.
- Move quickly without lowering the standard. Speed and rigor are complementary.
- Stay close to the mission and the user. The best decisions begin with the real problem.
- Work across boundaries. Everyone contributes beyond the narrow limits of a job title.
- Communicate directly. We value clear thinking, honest feedback, and low-ego collaboration.
- Build for the real world. Our systems must perform in complex, regulated, and high-stakes environments.
EQUAL OPPORTUNITY
Helios is an equal opportunity employer. We evaluate candidates based on their abilities, experience, and potential to contribute to our mission. We do not discriminate on the basis of race, color, religion, sex, gender identity or expression, sexual orientation, national origin, age, disability, veteran status, or any other status protected by applicable law.