Job description

Key Responsibilities Platform Engineering & Architecture • Design, implement, and maintain Databricks platform architecture to support enterprise-scale data workloads • Establish best practices for workspace setup, cluster management, and job orchestration • Optimize platform performance, scalability, and cost efficiency • Define and enforce standards for data engineering and analytics workloads within Databricks Developer Enablement • Support development teams by providing guidance, documentation, and reusable frameworks • Enable developers with CI/CD pipelines, automation, and deployment standards • Troubleshoot and resolve platform-related issues impacting development teams • Provide technical mentorship and onboarding support to engineers using Databricks Governance, Security & Compliance • Implement and manage data governance, access controls, and security models (Unity Catalog, RBAC, etc.) • Ensure compliance with enterprise data policies and regulatory requirements • Monitor and audit platform usage, access patterns, and data lineage Integration & Ecosystem Management • Integrate Databricks with cloud platforms (AWS/Azure/GCP), data lakes, and enterprise systems • Manage connectivity with tools such as Power BI, APIs, data ingestion frameworks, and orchestration tools • Support ingestion and processing frameworks (e.g., batch, streaming, Delta Lake) Operational Excellence • Monitor platform health, performance, and usage metrics • Implement automation for provisioning, monitoring, and scaling • Drive continuous improvements in platform reliability and developer experience Required Qualifications • Bachelor's degree in Computer Science, Engineering, or related field (or equivalent experience) • 5+ years of experience in data engineering or platform engineering roles • 3+ years of hands-on experience with Databricks (architecture, administration, or platform engineering) • Strong understanding of distributed data processing (Spark) • Experience with cloud platforms (Azure preferred, or AWS/GCP) • Knowledge of data lake architecture, Delta Lake, and modern data platforms • Experience implementing security, governance, and access control frameworks • Familiarity with CI/CD pipelines, DevOps practices, and infrastructure as code (Terraform, etc.)

Requirements

Must-haves

Platform Engineering & Architecture
Design, implement, and maintain Databricks platform architecture to support enterprise-scale data workloads
Establish best practices for workspace setup, cluster management, and job orchestration
Optimize platform performance, scalability, and cost efficiency
Define and enforce standards for data engineering and analytics workloads within Databricks
Developer Enablement
Support development teams by providing guidance, documentation, and reusable frameworks
Enable developers with CI/CD pipelines, automation, and deployment standards
Troubleshoot and resolve platform-related issues impacting development teams
Provide technical mentorship and onboarding support to engineers using Databricks
Governance, Security & Compliance
Implement and manage data governance, access controls, and security models (Unity Catalog, RBAC, etc.)
Ensure compliance with enterprise data policies and regulatory requirements
Monitor and audit platform usage, access patterns, and data lineage
Integration & Ecosystem Management
Integrate Databricks with cloud platforms (AWS/Azure/GCP), data lakes, and enterprise systems
Manage connectivity with tools such as Power BI, APIs, data ingestion frameworks, and orchestration tools
Support ingestion and processing frameworks (e.g., batch, streaming, Delta Lake)
Operational Excellence
Monitor platform health, performance, and usage metrics
Implement automation for provisioning, monitoring, and scaling
Drive continuous improvements in platform reliability and developer experience
Bachelor's degree in Computer Science, Engineering, or related field (or equivalent experience)

Nice-to-haves

Experience with cloud platforms (Azure preferred, or AWS/GCP)

Application prep

Use the role’s language and application details while tailoring your materials.

Keywords to mirror

ci/cdawsazuregcp

Platform: jobdiva
Effort: medium

Benefits

Benefits detected in the listing.

Vision