Job description

As the Principal Data Architect and Manager on our team, you will serve as both the senior technical authority and the people leader for our data platform. You'll define and own the end-to-end architecture of a real-time, petabyte-scale data backbone: from ingestion through a multi-layered lakehouse to normalized serving layers that power downstream search, ranking, and on-device experiences. You'll also build, grow, and lead the team of data engineers who bring that architecture to life. This is a hands-on principal role with multiple facets: you set the technical vision, personally shape the hardest architectural decisions, drive the roadmap through to production, and manage, mentor, and grow the engineers executing against it. Your leverage comes equally from what you design and from the team you build. MS Degree in Computer Science or related degree and 12+ years of experience in Data Architecture, Data Engineering, or Platform Engineering, with at least 5 years operating in a Principal, Staff, or Lead Manager capacity. Proven experience leading and managing engineers including hiring, performance management, and technical mentorship of senior ICs and managers. Track record of shipping petabyte-scale, low-latency data platforms in production and operating them under real-world load. Deep cloud expertise: expert-level proficiency with cloud object storage (e.g., AWS S3) and its architectural nuances for massive data lakes and lake-houses. Experience architecting systems for entity resolution, conflation, or knowledge-graph construction at scale — ideally involving billions of frequently updated entities. Experience designing pipelines that process multimodal data (structured, text, image) and integrate ML model inference including LLMs and embedding models: for enrichment and transformation. Familiarity with LLM/model-serving infrastructure trade-offs (inference runtimes, GPU-backed serving) to inform architectural decisions Streaming expertise: deep, hands-on knowledge of Apache Kafka (or comparable brokers like Kinesis) and complex stream processing (Spark Structured Streaming, Flink, or similar). Data modeling: exceptional ability to design logical and physical data models for large-scale ingest, retrieval, and analytical consumption — including dimensional modeling and lakehouse patterns. Experience defining SLAs, quality metrics, and observability standards for large-scale data platforms, with hands-on use of monitoring/alerting tooling (e.g., Prometheus/Grafana, Datadog, or OpenTelemetry-based tracing). Programming: command of at least one modern data-pipeline language (Scala, Java, or Python) and strong software engineering fundamentals. Cloud services integration: proven experience wiring together event notifications, queuing, orchestration, and compute services into resilient production pipelines. Experience with vector search technologies (e.g., Pinecone, Milvus) and storing/serving embeddings (e.g., pgvector, Milvus, FAISS) Excellent written and verbal communication; proven ability to align engineers, partner teams, and senior leadership from multiple lines of business around a shared technical direction, with experience bringing a consumer-oriented product from inception to production. Experience with embedding storage and retrieval (e.g., pgvector, Milvus, FAISS) and with graph databases (e.g., TigerGraph, Neo4j). Experience deploying, serving, and optimizing LLMs or ML models directly in the production, inference runtimes/compilers (ONNX Runtime, TensorRT/TensorRT-LLM), and serving frameworks (Triton, vLLM, TorchServe or similar). Experience tuning batching, KV-cache, and GPU utilization for low-latency, high-throughput real-time inference in a data pipeline Experience with data governance tools (e.g., Apache Atlas, AWS Glue Catalog, DataHub). Familiarity with Infrastructure as Code (Terraform, Pulumi) and modern CI/CD practice. Experience designing systems that handle petabytes of unstructured media data. Working knowledge of data privacy regulations and best practices for incorporating safety and compliance, and a demonstrated instinct for building privacy-preserving systems.

Requirements

Must-haves

Proven experience leading and managing engineers including hiring, performance management, and technical mentorship of senior ICs and managers.
Track record of shipping petabyte-scale, low-latency data platforms in production and operating them under real-world load.
Deep cloud expertise: expert-level proficiency with cloud object storage (e.g., AWS S3) and its architectural nuances for massive data lakes and lake-houses.
Experience architecting systems for entity resolution, conflation, or knowledge-graph construction at scale — ideally involving billions of frequently updated entities.
Experience designing pipelines that process multimodal data (structured, text, image) and integrate ML model inference including LLMs and embedding models: for enrichment and transformation.
Familiarity with LLM/model-serving infrastructure trade-offs (inference runtimes, GPU-backed serving) to inform architectural decisions
Streaming expertise: deep, hands-on knowledge of Apache Kafka (or comparable brokers like Kinesis) and complex stream processing (Spark Structured Streaming, Flink, or similar).
Data modeling: exceptional ability to design logical and physical data models for large-scale ingest, retrieval, and analytical consumption — including dimensional modeling and lakehouse patterns.
Experience defining SLAs, quality metrics, and observability standards for large-scale data platforms, with hands-on use of monitoring/alerting tooling (e.g., Prometheus/Grafana, Datadog, or OpenTelemetry-based tracing).
Programming: command of at least one modern data-pipeline language (Scala, Java, or Python) and strong software engineering fundamentals.
Cloud services integration: proven experience wiring together event notifications, queuing, orchestration, and compute services into resilient production pipelines.
Experience with vector search technologies (e.g., Pinecone, Milvus) and storing/serving embeddings (e.g., pgvector, Milvus, FAISS)
Experience with embedding storage and retrieval (e.g., pgvector, Milvus, FAISS) and with graph databases (e.g., TigerGraph, Neo4j).
Experience tuning batching, KV-cache, and GPU utilization for low-latency, high-throughput real-time inference in a data pipeline
Experience with data governance tools (e.g., Apache Atlas, AWS Glue Catalog, DataHub).
Familiarity with Infrastructure as Code (Terraform, Pulumi) and modern CI/CD practice.
Experience designing systems that handle petabytes of unstructured media data.
Working knowledge of data privacy regulations and best practices for incorporating safety and compliance, and a demonstrated instinct for building privacy-preserving systems.

Nice-to-haves

None flagged.

Application prep

Use the role’s language and application details while tailoring your materials.

Keywords to mirror

awsmlllmjavapythonci/cd

Platform: apple
Effort: medium

Benefits

Benefits detected in the listing.

Vision