Job description

Required Qualifications & Experience • Bachelor's degree in computer science, Information Technology, Engineering, or a related discipline, with 6-10 years of experience in Data Engineering, Application Development, System Integration, or Enterprise Data Warehouse projects. • Strong hands-on experience designing and implementing modern data platforms on Azure Databricks, leveraging Delta Lake, Delta Live Tables (DLT), and Medallion Architecture (Bronze, Silver, Gold) to build scalable and reliable data pipelines. • Proven expertise in developing and optimizing batch and streaming data pipelines using PySpark, Spark SQL, and Python, with a strong focus on performance tuning, scalability, and cost optimization. • Extensive experience in Databricks job performance optimization, including query tuning, partitioning strategies, data skew mitigation, caching, cluster sizing, workload optimization, and Spark execution plan analysis. • Strong proficiency in SQL programming and data modeling concepts, including relational, dimensional, and lakehouse data architectures. • Deep understanding of Delta Lake features such as ACID transactions, schema evolution, time travel, change data capture (CDC), and data quality enforcement. • Experience implementing data quality, monitoring, and orchestration frameworks using Delta Live Tables expectations, Databricks Workflows, and Azure-native services. • Strong exposure to the full Software Development Life Cycle (SDLC), including design, development, testing, CI/CD, deployment, and production support. • Solid understanding of Lakehouse architecture, cloud-based analytics platforms, data governance, and data engineering best practices. • Knowledge of real-time data processing, data integration patterns, and enterprise-scale data migration initiatives is a strong advantage. • Demonstrated ability to stay current with evolving cloud and big data technologies and contribute effectively to architecture discussions, technology strategy, and platform modernization initiatives. • Self-motivated, proactive, and capable of working independently in a fast-paced environment while collaborating with globally distributed teams. • Strong analytical, problem-solving, communication, and stakeholder management skills.

Requirements

Must-haves

Required Qualifications & Experience
• Proven expertise in developing and optimizing batch and streaming data pipelines using PySpark, Spark SQL, and Python, with a strong focus on performance tuning, scalability, and cost optimization.
• Strong proficiency in SQL programming and data modeling concepts, including relational, dimensional, and lakehouse data architectures.
• Deep understanding of Delta Lake features such as ACID transactions, schema evolution, time travel, change data capture (CDC), and data quality enforcement.
• Experience implementing data quality, monitoring, and orchestration frameworks using Delta Live Tables expectations, Databricks Workflows, and Azure-native services.
• Strong exposure to the full Software Development Life Cycle (SDLC), including design, development, testing, CI/CD, deployment, and production support.
• Solid understanding of Lakehouse architecture, cloud-based analytics platforms, data governance, and data engineering best practices.
• Knowledge of real-time data processing, data integration patterns, and enterprise-scale data migration initiatives is a strong advantage.
• Demonstrated ability to stay current with evolving cloud and big data technologies and contribute effectively to architecture discussions, technology strategy, and platform modernization initiatives.
• Self-motivated, proactive, and capable of working independently in a fast-paced environment while collaborating with globally distributed teams.
• Strong analytical, problem-solving, communication, and stakeholder management skills.

Nice-to-haves

None flagged.

Application prep

Use the role’s language and application details while tailoring your materials.

Keywords to mirror

azuresqlpythonci/cd

Platform: jobdiva
Effort: medium