Job description

We are seeking a Lead Data Engineer with expertise in Databricks and Data Warehousing to drive data architecture, pipeline development, and optimization efforts. The ideal candidate will play a key role in designing scalable solutions, implementing best practices, and leading data initiatives within a dynamic and collaborative environment. Key Responsibilities: Design, build, and optimize scalable data pipelines using Databricks, Apache Spark, and Azure technologies. Architect data warehousing solutions, ensuring seamless integration with cloud platforms and structured/unstructured data sources. Collaborate with business stakeholders to understand data needs and develop high-performance analytical solutions. Implement ETL/ELT processes leveraging cloud-based technologies such as Azure Data Factory, Snowflake, and Delta Lake. Ensure data quality, governance, and security compliance while managing large datasets efficiently. Drive performance tuning and optimization for data pipelines, ensuring efficiency across systems. Work closely with cross-functional teams to support machine learning and advanced analytics initiatives. Provide technical leadership and mentorship to junior data engineers, fostering a culture of innovation and continuous improvement. Stay updated on emerging data technologies and recommend strategies to enhance existing architectures. Qualifications: 8+ years of experience in data engineering, big data processing, and cloud-based solutions. Strong expertise in Databricks, Spark (PySpark/SQL), and Delta Lake architecture. Proven experience in designing and managing data warehouses using Snowflake, Azure Synapse, or equivalent technologies. Deep understanding of data modeling, SQL, and performance optimization. Hands-on experience with Azure Data Factory, Event Hubs, and cloud-based ETL processes. Solid knowledge of real-time streaming technologies (Kafka, Azure Stream Analytics, or similar). Familiarity with ML/AI data pipelines and feature engineering best practices. Strong communication and collaboration skills, with experience working in fast-paced, enterprise environments

Requirements

Must-haves

Design, build, and optimize scalable data pipelines using Databricks, Apache Spark, and Azure technologies.
Architect data warehousing solutions, ensuring seamless integration with cloud platforms and structured/unstructured data sources.
Collaborate with business stakeholders to understand data needs and develop high-performance analytical solutions.
Implement ETL/ELT processes leveraging cloud-based technologies such as Azure Data Factory, Snowflake, and Delta Lake.
Ensure data quality, governance, and security compliance while managing large datasets efficiently.
Drive performance tuning and optimization for data pipelines, ensuring efficiency across systems.
Work closely with cross-functional teams to support machine learning and advanced analytics initiatives.
Provide technical leadership and mentorship to junior data engineers, fostering a culture of innovation and continuous improvement.
Stay updated on emerging data technologies and recommend strategies to enhance existing architectures.
8+ years of experience in data engineering, big data processing, and cloud-based solutions.
Strong expertise in Databricks, Spark (PySpark/SQL), and Delta Lake architecture.
Proven experience in designing and managing data warehouses using Snowflake, Azure Synapse, or equivalent technologies.
Deep understanding of data modeling, SQL, and performance optimization.
Hands-on experience with Azure Data Factory, Event Hubs, and cloud-based ETL processes.
Solid knowledge of real-time streaming technologies (Kafka, Azure Stream Analytics, or similar).
Familiarity with ML/AI data pipelines and feature engineering best practices.
Strong communication and collaboration skills, with experience working in fast-paced, enterprise environments

Nice-to-haves

None flagged.

Application prep

Use the role’s language and application details while tailoring your materials.

Keywords to mirror

azuremachine learningsqlmlai

Platform: careers-browser
Effort: medium