Job Description:
•Review and translate requirements to end-end technical design with a goal of building data products
• Develop pipelines in Databricks using PySpark and DBT to onboard various sources
• Working knowledge in Kafka to stream the data into databricks platform
• Identify Common approaches and best practices to data integration such as metadata drive ingestion, data validations, complex data type handling
• Drive standards for reusability, while developing pipelines
• Create or update documentation associated to key decisions on requirements, design and development
• Liaise with architects, data owners and users for reviews on day-day basis to ensure the quality of deliverables are high.
Skill Proficiency Requirements:
• 8+ years of overall IT experience with 6 years as a Data Engineer
• Experience in providing data engineering solutions
• Working knowledge of below technology toolsets:
o Cloud (Azure/AWS)
o Databricks, Data Mesh, Data as a Product
o Python
o Data streaming (Kafka, EventHub, Kinesis, SNS)
o Cloud-based API management – good to have
o Immuta – good to have
o Multi-cloud and on-prem application integration – good to have