Job description

Roles & Responsibilities Job Title: Senior / Lead Engineer Devops Job Description: We are looking for a versatile and experienced Senior Data Engineer with broad, end-to-end expertise spanning Data Architecture, Pipeline Development, and Cloud DevOps (DataOps). In this hybrid role, you will be instrumental in designing, building, and deploying our next-generation Distributed Big Data Lakehouse platform. The ideal candidate will combine deep hands-on expertise in open-source Apache big data ecosystems with modern cloud-agnostic architectures across Databricks and Snowflake. You will lead the engineering of scalable ETL/ELT pipelines, manage orchestrations across AWS (including MWAA, SQS, EKS) and Google Cloud Platform (GCS, BigQuery), deploy containerized data workloads to Red Hat OpenShift and Amazon EKS, and implement enterprise-grade Secured Cross-Cloud Data Sharing solutions. Key Responsibilities: Lakehouse & Data Architecture: • Distributed Lakehouse Design: Architect and evolve open, reliable, and scalable Lakehouse foundations using open table formats (Apache Iceberg, Delta Lake, Apache Hudi) to eliminate data silos and enable both BI reporting and advanced analytics/AI. • Data Modeling & Governance: Design optimized dimensional, Data Vault, and relational data models; implement unified enterprise data catalogs, schema enforcement, data lineage, and metadata management across multi-cloud environments. • Secured Cross-Cloud Data Sharing: Architect secure, zero-trust cross-cloud data sharing patterns leveraging protocols like Delta Sharing, Snowflake Secure Data Sharing / Clean Rooms, and private cloud interconnects (AWS PrivateLink / GCP Cloud Interconnect) to share governed data without physical data duplication. Data Engineering, ETL & Distributed Pipelines • Pipeline Development: Build, optimize, and maintain high-throughput batch and low-latency real-time/streaming data pipelines using Apache Spark (PySpark / Scala), Apache Flink, and Apache Kafka. • Orchestration & Workflow Scheduling: Design and automate complex DAGs and scheduling workflows utilizing Apache Airflow and AWS Managed Workflows for Apache Airflow (MWAA). • Cloud Data Integration: Integrate multi-source streaming and batch data from cloud storage (AWS S3, Google Cloud Storage / GCS) and message queues (AWS SQS, Kafka) into analytics warehouses (Snowflake) and lakehouse compute engines (Databricks Unity Catalog / Spark). • Data Quality & Observability: Implement automated data testing, anomaly detection, validation rules, and SLA tracking across all ETL stages. DevOps, Containerization & Infrastructure as Code (DataOps) • Kubernetes & Container Orchestration: Package, deploy, scale, and monitor distributed data applications and microservices on Red Hat OpenShift and Amazon EKS / Kubernetes. • CI/CD Automation: Define and maintain robust CI/CD deployment pipelines (e.g., GitLab CI, GitHub Actions, Jenkins, Tekton, ArgoCD, Harness) enforcing automated unit/integration testing, container vulnerability scanning, and automated rollouts. • Infrastructure a s Code (IaC): Provision, maintain, and version infrastructure using Terraform or CloudFormation, ensuring reproducible environments across Dev, UAT, and Production. SDLC, Security & Enterprise Standards Qualifications Education & Experience • Bachelor’s or Master’s Degree in Computer Science, Data Engineering, Software Engineering, or related technical discipline. • 7+ years of progressive experience in Data Engineering, Distributed Systems Architecture, and DataOps/DevOps. • Experience migrating legacy on-premises Hadoop/HDFS workloads to cloud-native Lakehouse platforms (OpenShift/EKS, Databricks, Snowflake). • Active certifications in relevant domains: o Databricks: Certified Data Engineer Professional o Snowflake: SnowPro Core / Advanced Architect o AWS: AWS Certified Data Engineer – Associate / Solutions Architect – Professional o Kubernetes: CKA (Certified Kubernetes Administrator) / Red Hat Certified Specialist in OpenShift • Familiarity with Data Mesh concepts, data contracts, and domain-driven data modeling. Salary Range: $125,000 to $140,000 per year </div

Requirements

Must-haves

Roles & Responsibilities
Job Title: Senior / Lead Engineer Devops
Data Engineering, ETL & Distributed Pipelines
Pipeline Development: Build, optimize, and maintain high-throughput batch and low-latency real-time/streaming data pipelines using Apache Spark (PySpark / Scala), Apache Flink, and Apache Kafka.
Orchestration & Workflow Scheduling: Design and automate complex DAGs and scheduling workflows utilizing Apache Airflow and AWS Managed Workflows for Apache Airflow (MWAA).
Data Quality & Observability: Implement automated data testing, anomaly detection, validation rules, and SLA tracking across all ETL stages.
DevOps, Containerization & Infrastructure as Code (DataOps)
Kubernetes & Container Orchestration: Package, deploy, scale, and monitor distributed data applications and microservices on Red Hat OpenShift and Amazon EKS / Kubernetes.
Infrastructure a s Code (IaC): Provision, maintain, and version infrastructure using Terraform or CloudFormation, ensuring reproducible environments across Dev, UAT, and Production.
SDLC, Security & Enterprise Standards
Education & Experience
Bachelor’s or Master’s Degree in Computer Science, Data Engineering, Software Engineering, or related technical discipline.
7+ years of progressive experience in Data Engineering, Distributed Systems Architecture, and DataOps/DevOps.
Experience migrating legacy on-premises Hadoop/HDFS workloads to cloud-native Lakehouse platforms (OpenShift/EKS, Databricks, Snowflake).
o Databricks: Certified Data Engineer Professional
o Snowflake: SnowPro Core / Advanced Architect
o AWS: AWS Certified Data Engineer – Associate / Solutions Architect – Professional
o Kubernetes: CKA (Certified Kubernetes Administrator) / Red Hat Certified Specialist in OpenShift
Familiarity with Data Mesh concepts, data contracts, and domain-driven data modeling.

Nice-to-haves

None flagged.

Application prep

Use the role’s language and application details while tailoring your materials.

Keywords to mirror

awsaigcpkubernetesci/cd

Platform: careers-page
Effort: medium

Benefits

Benefits detected in the listing.

Vision