Job description

#### About the Role We partner with global enterprises to design and build large-scale data platforms that power the products and operations at the center of their business. This engagement is a modernization program that moves production data workloads off legacy Dataiku and Cloudera/Hadoop environments and onto Databricks on Azure. You will join a joint team of client and Lumenalta engineers, embedded alongside the client's data, cloud and infrastructure teams. The client owns the migration and sets its priorities and timeline. The team needs one hands-on technical lead who guides the solution design, sets the Databricks patterns everyone builds on, and keeps work moving across workstreams. That is this role. You will guide design across a batch pipeline migration and a real-time streaming ingestion effort. You will stay hands-on, step in wherever a blocker comes up, and help the client's engineers grow into owning the platform long term. #### What You'll Do - Guide the overall solution design for the platform, working with the client's project lead to align on priorities as they evolve. - Define the Databricks patterns and standards the joint team follows, including medallion architecture (bronze, silver, gold) and pipeline conventions. - Lead the migration of a production pipeline from Dataiku and Cloudera/Hadoop to Databricks across the silver and gold layers, following a lift-and-shift approach first and optimization later. - Keep production outputs running throughout the transition through parallel runs, data reconciliation and safe cutover, since business-critical operations depend on this pipeline. - Guide the evaluation and design of real-time event ingestion from Azure Event Hubs, Event Grid and Azure Data Explorer into Databricks, comparing approaches such as Spark Structured Streaming and Databricks ZeroBus. - Support the transition of Power BI dashboards to Databricks dashboards. - Work hands-on across workstreams, review designs and code, and step in to unblock engineers whenever an issue comes up. - Pair with the client's engineers and share Databricks practices so their team can own and evolve the platform over time. - Write clean, efficient, and maintainable Python, PySpark, and advanced SQL. #### What We're Looking For - 7+ years of experience as a Data Engineer, including time as a technical lead or data architect on a data platform team. - Production experience with Databricks on Azure. - Strong, hands-on experience with Python, PySpark, and advanced SQL. - Proven experience designing and implementing medallion architectures in production. - Experience migrating production workloads to a modern lakehouse from Dataiku or from Hadoop-based platforms such as Cloudera, including keeping outputs stable through cutover. - Experience with streaming or event-driven ingestion on Azure, such as Event Hubs, Event Grid, or Azure Data Explorer, and with Spark Structured Streaming. - Comfortable leading through influence inside a client's team: setting technical direction while the client owns priorities and timeline. - A track record of mentoring and enabling other engineers, not just directing them. - Strong communication skills, with the ability to explain design decisions to engineers and stakeholders. - Fluent English, both written and spoken. #### Nice to Have - Experience migrating workflows specifically from Dataiku to Databricks. - Experience with Databricks ZeroBus or other direct-to-lakehouse streaming ingestion. - Experience with Databricks dashboards, or migrating reporting off Power BI. - Working knowledge of Unity Catalog. - Experience with Delta Live Tables or Databricks declarative pipelines, Databricks Workflows, and Databricks Asset Bundles. - Experience with high-volume, real-time event or sensor data. - Databricks Data Engineer Professional certification. #### Why Lumenalta is an amazing place to work at At Lumenalta, you can expect that you will: - Be 100% dedicated to one project at a time so that you can innovate and grow. - Be a part of a team of talented and friendly senior-level developers. - Work on projects that allow you to use leading tech. For Databricks practitioners specifically, we are a Databricks partner with Champions and MVPs on our team, and that carries three concrete things: - **Delivery Partner Program qualification**. Databricks co-delivers its Professional Services engagements through a short list of approved partner organizations, and practitioners have to be individually qualified before they can work on them. The qualification is granted once at the individual level and does not expire, so it stays with you. Partner organizations are the route in, so the qualified pool stays small. That scarcity is most of the value, and we sponsor it. - **Champion path**. Databricks Champion is an individual recognition with its own badge, a dedicated Databricks solutions architect as your mentor, access to product teams, confidential roadmap briefings, and an invitation to the internal Tech Summit. Nomination has to come from a partner organization, which puts it out of reach for independents and most contractors. We nominate actively and regularly, and the Champions already on our team mentor candidates through it. - **Certifications covered.** We pay the exam fee for any Databricks certification you want to take. #### Our Process A screening call, a technical interview, and a HackerRank assessment. The technical interview is a conversation rather than a whiteboard exam. We will ask you to walk through a Databricks platform you designed and led and how you set patterns for the team. We will also ask how you migrated a production workload off a legacy platform without breaking downstream consumers, and what you would change. Relevant project experience matters more to us than certifications. #### Location This is a fully remote position. Must have availability to work U.S. Eastern time zone business hours. #### Application Deadline Applications will be accepted until **October 25th, 2026**. Candidates can expect feedback by **November 2nd, 2026**.

Requirements

Must-haves

#### About the Role
#### What You'll Do
Guide the overall solution design for the platform, working with the client's project lead to align on priorities as they evolve.
Define the Databricks patterns and standards the joint team follows, including medallion architecture (bronze, silver, gold) and pipeline conventions.
Lead the migration of a production pipeline from Dataiku and Cloudera/Hadoop to Databricks across the silver and gold layers, following a lift-and-shift approach first and optimization later.
Keep production outputs running throughout the transition through parallel runs, data reconciliation and safe cutover, since business-critical operations depend on this pipeline.
Guide the evaluation and design of real-time event ingestion from Azure Event Hubs, Event Grid and Azure Data Explorer into Databricks, comparing approaches such as Spark Structured Streaming and Databricks ZeroBus.
Support the transition of Power BI dashboards to Databricks dashboards.
Work hands-on across workstreams, review designs and code, and step in to unblock engineers whenever an issue comes up.
Pair with the client's engineers and share Databricks practices so their team can own and evolve the platform over time.
Write clean, efficient, and maintainable Python, PySpark, and advanced SQL.
#### What We're Looking For
7+ years of experience as a Data Engineer, including time as a technical lead or data architect on a data platform team.
Production experience with Databricks on Azure.
Strong, hands-on experience with Python, PySpark, and advanced SQL.
Proven experience designing and implementing medallion architectures in production.
Experience migrating production workloads to a modern lakehouse from Dataiku or from Hadoop-based platforms such as Cloudera, including keeping outputs stable through cutover.
Experience with streaming or event-driven ingestion on Azure, such as Event Hubs, Event Grid, or Azure Data Explorer, and with Spark Structured Streaming.
Comfortable leading through influence inside a client's team: setting technical direction while the client owns priorities and timeline.
A track record of mentoring and enabling other engineers, not just directing them.
Strong communication skills, with the ability to explain design decisions to engineers and stakeholders.
Fluent English, both written and spoken.
Experience migrating workflows specifically from Dataiku to Databricks.

Nice-to-haves

#### Nice to Have

Application prep

Use the role’s language and application details while tailoring your materials.

Keywords to mirror

azurepythonsql

Platform: careers-page
Effort: medium