38803 Senior Data Engineer
Cephas Consultancy Services Private Limited· Human Resources· Oct 25, 2026 · 2d ago
Glendale, California
Hybrid
Full-time
Senior
$140K–$198K
9+ years exp
Job description
Positions: 1 Full Time
Experience
9 – 13 Years
Role: Sr Data Engineer
Location: Glendale, CA (Hybrid – 2- 4 Days Onsite)
Visa: USC/GC/GCEAD/H4EAD/L2
Mandatory skills –
• Databricks experience (primary requirement)
• Apache Airflow
• Advanced SQL skills
• Python
• Spark / PySpark
• Scala
• Experience building and maintaining data pipelines and workflows
Interview Process
• Expected to follow the team’s recent hiring process for similar data engineering positions.
• Typically consists of:
? Initial screening/vetting round
? Technical interview
? Final interview (potentially onsite)
• Total process generally spans 2-3 interview rounds.
Key Responsibilities:
? Design, write, test, and deploy data pipelines using PySpark, Scala, SQL, Python
? Meet with stakeholders to gather requirements and translate them into scalable data platform solutions
? Understanding of Databricks platform and developer tooling to diagnose errors, audit platform activity, and automate updates across pipelines, objects, and integrations
? Ability to explain Spark architecture and pipeline behavior to stakeholders to diagnose root causes and recommend solutions
? Provide solution architecture across AWS, Databricks, Kubernetes, and Airflow (MWAA), including cross-platform integrations
? Manage Databricks platform governance, including Unity Catalog, ACLs, lineage, and data discovery and privacy tooling
? Build and maintain Kubernetes containers and containerized utilities supporting deployed data platform services
? Apply networking knowledge to troubleshoot connectivity and integration errors across platform components
? Perform platform administration: provision and remove access, assess resource utilization, monitor platform health and cost, and evaluate stakeholder requests
? Collaborate with engineers, architects, and product managers to drive Core Data platform success; participate in agile/scrum ceremonies
? Maintain documentation of platform changes, standards, and pipeline configurations to support data quality and governance
Qualifications:
? 5+ years of data engineering experience developing and operating large-scale data pipelines
? Deep hands-on experience with Databricks and Apache Spark (batch and streaming), including pipeline development in PySpark and/or Scala
? Strong understanding of Spark architecture—executors, stages, partitioning, shuffle, and performance tuning—with ability to explain tradeoffs to technical and non-technical stakeholders
? Proficiency with Databricks platform tooling (API, SDK, CLI) for automation, auditing, governance, and operational troubleshooting
? Proficient in SQL with advanced performance tuning capabilities
? Hands-on production experience with Airflow (MWAA) for orchestrating data pipelines
? Experience managing Databricks platform governance: ACLs, Unity Catalog, lineage, and access provisioning
? Proficiency in Python and at least one additional language (Scala, Kotlin, or SQL-driven pipeline tooling)
? Experience designing and optimizing scalable ETL/ELT pipelines integrating diverse structured and unstructured data sources
? AWS-primary experience (compute, storage, networking, IAM); experience with other cloud providers is transferable
? Proficiency with Docker and Kubernetes for building and maintaining containerized data platform services
? Working knowledge of networking concepts to diagnose cross-platform integration and connectivity issues
? Familiarity with Snowflake and comparable tooling relative to the Databricks ecosystem
? Experience designing and implementing CI/CD and DevOps practices (Git-based workflows)
? Experience implementing data quality checks, monitoring, and logging for pipeline reliability
? Self-starting problem solver with strong analytical and communication skills; willingness to learn new tooling and trends
? Familiar with Scrum and Agile methodologies
? Experience with Snowflake is a plus
? Bachelor’s Degree in Computer Science, Information Systems, or a related field, or equivalent work experience, Master’s Degree is a plus.
Application prep
Use the role’s language and application details while tailoring your materials.
Keywords to mirror
sqlpythonawskubernetesagilescrumapikotlindockerci/cd
- Platform
- wp-job-manager
- Effort
- medium
Benefits
Benefits detected in the listing.
Vision