Job description

Job Title: Senior Data Engineer Location: Remote Security Clearance: Active DoD Secret Clearance We question. We listen. We adapt. Be honest. Be pragmatic. Omni Federal , founded in 2017 and headquartered in Washington, DC, is a highly specialized software solutions provider with a robust presence in key locations across the United States, including Boston, MA, Colorado Springs, CO, San Antonio, TX, and St. Louis, MO. Born out of the Department of Defense's software factory ecosystem, Omni has rapidly distinguished itself by delivering both mission-critical and enterprise solutions that enhance the technological capabilities of the federal government. With a focus on areas such as Command and Control, Cybersecurity, Space, Geospatial, and Modeling & Simulation, Omni leverages cutting-edge commercial technology tailored to government objectives, improving mission performance and delivering transformative outcomes for the Department of Defense (DoD), Intelligence Community (IC), and their end-users. The company's innovative approach is backed by its Omni Labs and SBIR Innovation centers, where they develop advanced platforms and tools in data mesh, secure connectivity, and intelligent automation. Why Omni? • Environment of Autonomy • Innovative Commercial Approach • People over process We are seeking an experienced Senior Data Engineer to support the Air Force to help in developing and operating modernized applications that are used daily by the warfighter. This is an exciting new initiative where the Air Force is mirroring Silicon Valley processes and capabilities to quickly deploy new capabilities. Candidates must be passionate, energized, and excited to work on modern architectures and solve challenging problems for our clients. Position Description (what you will do) • Design, build, and operate streaming (Debezium change-data-capture into Amazon MSK) and batch (Apache Spark via AWS EMR) data pipelines that rapidly onboard new data sources into the program’s Apache Iceberg-based data lakehouse, from initial ingestion through validation and production release. • Corroborate newly onboarded data against existing data lakehouse holdings to validate quality, consistency, and lineage before it is exposed to downstream consumers. • Develop and maintain the metadata, cataloging (AWS Glue as the Iceberg data catalog), and API Gateway integrations that make program data easily discoverable and consumable via Amazon Athena, Spark, and other Iceberg-compatible tools by downstream mission partners and vendors through the enterprise data mesh. • Own assigned technical tasking independently — translate well-articulated requirements into a working solution. • Engage directly with data providers and data consumers to understand the operational context behind each data source and translate that context accurately for both technical and non-technical audiences. • Participate in an agile (Kanban/Scrum) delivery team, contributing to sprint planning, technical design discussions, and pipeline/code reviews. • Maintain pipeline and infrastructure code as Infrastructure-as-Code (Terraform) and follow established DevSecOps build and deployment pipeline processes. • Support data governance, quality, and auditability practices in accordance with Government-approved data standards for the platform. Requirements: • 8+ years of professional data engineering experience, including hands-on design and operation of production data pipelines. • Demonstrated experience with streaming and batch data pipeline technologies, particularly Debezium (change-data-capture), Amazon MSK (Kafka), Apache Spark (AWS EMR), and Apache Airflow. • Strong proficiency in Python and SQL, with experience integrating and validating data from multiple heterogeneous sources. • Experience with AWS data services (e.g., S3, Glue, Aurora) and containerized environments (Kubernetes), delivered through DevSecOps/CI-CD pipelines. • Demonstrated ability to work independently against well-defined tasking with minimal oversight, and to communicate effectively with both technical and non-technical stakeholders, including data providers and consumers. Nice to Have: • Experience supporting DoD or Intelligence Community programs, particularly Platform One or other DoD software factory environments. • Familiarity with container security and software supply chain concepts (SBOMs, vulnerability scoring and attestation). • Experience with the Apache Iceberg table format or similar open table formats (e.g., Delta Lake, Hudi) for lakehouse architectures. • Experience with event-driven architecture (e.g., AWS EventBridge) and Infrastructure-as-Code (Terraform). • Experience designing or consuming API Gateways in an enterprise data mesh architecture. • Prior experience in a vendor- or customer-facing data steward or data liaison capacity. Our team is always hungry for more – more knowledge, more problem solving, more growth, more innovation, and, ultimately, customer success. This hunger and hustle fuel our determination to excel in everything we do. It’s not just about meeting established goals; it’s about exceeding them. We take immense pride in applying our expertise in cutting-edge technologies and our ability to adapt to emerging trends directly to the users in the field, as our nation’s protectors deserve nothing less. Omni Federal is an equal opportunity and affirmative action employer. Omni Federal is committed to administering all employment and personnel actions on the basis of merit and free of discrimination based on race, color, religion, sex, sexual orientation, gender identity, national origin, protected veteran status, or status as an individual with a disability. Consistent with this commitment, we are dedicated to the employment and advancement of qualified minorities, women, individuals with disabilities, protected veterans, persons of all ethnic backgrounds and religions according to their abilities.

Requirements

Must-haves

Job Title: Senior Data Engineer
Location: Remote
Security Clearance: Active DoD Secret Clearance
We question. We listen. We adapt.
Be honest. Be pragmatic.
Environment of Autonomy
Innovative Commercial Approach
People over process
Position Description (what you will do)
Corroborate newly onboarded data against existing data lakehouse holdings to validate quality, consistency, and lineage before it is exposed to downstream consumers.
Own assigned technical tasking independently — translate well-articulated requirements into a working solution.
Engage directly with data providers and data consumers to understand the operational context behind each data source and translate that context accurately for both technical and non-technical audiences.
Participate in an agile (Kanban/Scrum) delivery team, contributing to sprint planning, technical design discussions, and pipeline/code reviews.
Maintain pipeline and infrastructure code as Infrastructure-as-Code (Terraform) and follow established DevSecOps build and deployment pipeline processes.
Support data governance, quality, and auditability practices in accordance with Government-approved data standards for the platform.
8+ years of professional data engineering experience, including hands-on design and operation of production data pipelines.
Demonstrated experience with streaming and batch data pipeline technologies, particularly Debezium (change-data-capture), Amazon MSK (Kafka), Apache Spark (AWS EMR), and Apache Airflow.
Strong proficiency in Python and SQL, with experience integrating and validating data from multiple heterogeneous sources.

Nice-to-haves

Experience supporting DoD or Intelligence Community programs, particularly Platform One or other DoD software factory environments.
Familiarity with container security and software supply chain concepts (SBOMs, vulnerability scoring and attestation).
Experience with the Apache Iceberg table format or similar open table formats (e.g., Delta Lake, Hudi) for lakehouse architectures.
Experience with event-driven architecture (e.g., AWS EventBridge) and Infrastructure-as-Code (Terraform).
Experience designing or consuming API Gateways in an enterprise data mesh architecture.
Prior experience in a vendor- or customer-facing data steward or data liaison capacity.

Application prep

Use the role’s language and application details while tailoring your materials.

Keywords to mirror

awsapiagilescrumpythonsqlkubernetes

Platform: agile-ats
Effort: medium