Principal Data Architect at Berkshire Hathaway Specialty Insurance

Job description

Who are we? A strategic and trusted insurance partner, Berkshire Hathaway Specialty Insurance (BHSI), provides a broad range of commercial property, casualty and specialty insurance coverages and outstanding service to customers and brokers around the world. Part of Berkshire Hathaway’s insurance operations, we bring our solutions to market with our stellar brand name, top-rated balance sheet, and the expertise of our global team of professionals, who exude excellent capabilities and strong character. We are a values-based organization where respect, integrity, excellence, collaboration, and passion define who we are and how we do business. We value diversity of backgrounds, experience, and perspectives and strive to foster an inclusive environment that enables all our team members to bring their best selves to work. We are one team committed to building a culture where every teammate has the opportunity to contribute and be recognized. Want to be part of the team building the finest property, casualty, and specialty lines insurance company in the world? Learn more about our unique culture and history . Job Opportunity: BHSI is modernizing its data ecosystem, moving from a monolithic, reporting-centric infrastructure to a Lakehouse platform that supports insight-driven decisions and AI-enabled outcomes at the speed of business. We are seeking a Principal Data Architect to set the technical direction for that platform. You will own the architecture of our Databricks-based Lakehouse across data engineering, analytics, and AI/ML, ensuring it is secure, performant, cost-efficient, and ready for Generative AI. This is a hands-on, senior individual-contributor role for a technologist and strategic thinker, not a people-management position. You will report to the Global Head of Data Analytics Solutions and partner with Enterprise Technology portfolio leaders and business stakeholders. Responsibilities: • Platform Architecture & Strategy:  Define and evolve the enterprise Lakehouse reference architecture on Databricks, supporting batch, streaming, and AI workloads at scale across AWS or Azure. • AI & Generative AI Enablement:  Lead the design and rollout of Databricks AI capabilities, including vector search, model serving, and LLM orchestration, so teams can build, fine-tune, and deploy models securely against enterprise data. • Governance & Security by Design:  Implement lineage, access controls, and auditing with Unity Catalog to meet regulatory requirements and ensure data quality across domains. • MLOps & Engineering Excellence:  Establish CI/CD and infrastructure-as-code standards for data pipelines and ML models. Optimize Spark performance, cluster configuration, and compute spend. • Technical Leadership & Influence:  Set architecture standards, run design reviews, and mentor engineers and data scientists. Translate business strategy into a phased technical roadmap and build alignment with engineering and business leaders. Qualifications, Skills and Experience: • 15+ years in data engineering, data architecture, or software architecture, including 5+ years architecting on the Databricks Lakehouse Platform • Deep expertise in Apache Spark, Delta Lake, Unity Catalog, and MLflow • Proven experience designing infrastructure for both traditional ML and Generative AI workloads (RAG, fine-tuning, vector search, agentic frameworks) • Strong proficiency in Python and SQL, infrastructure-as-code (e.g., Terraform), and software design patterns • Hands-on architecture experience on AWS or Microsoft Azure • Demonstrated ability to influence senior technical and business stakeholders without direct authority Preferred Qualifications: • Databricks Professional or Expert-level certification • Experience with advanced agentic architectures, DSPy, or graph databases • Background in insurance, financial services, fintech, or other regulated environments • Track record of optimizing cloud compute costs while improving pipeline throughput BHSI Offers: • A competitive package and exciting growth opportunities for career-oriented teammates • A dynamic, action-oriented, and thoughtful environment centered on always doing the right thing for our customers, teammates and our other stakeholders • A purposely non-bureaucratic organization that embraces simplicity over complexity and emphasizes individual excellence in a team framework • Benefits that support your life and well-being, which include: • Comprehensive Health, Dental and Vision benefits • Disability Insurance (both short-term and long-term) • Life Insurance (for you and your family) • Accidental Death & Dismemberment Insurance (for you and your family) • Flexible Spending Accounts • Health Reimbursement Account • Employee Assistance Program • Retirement Savings 401(k) Plan with Company Match • Generous holiday and Paid Time Off • Tuition Reimbursement • Paid Parental Leave The base salary range for this position in Boston is $200,000 to $240,000, along with annual bonus eligibility; a candidate’s salary is determined by their relevant skills and experience. We value our teammates – both their capabilities and character – as demonstrated by our amazing culture. NOTE: Compensation will be commensurate with experience. This job description is not intended to be all-inclusive. Team Member may perform other related duties as negotiated to meet the ongoing needs of the organization.

Requirements

Must-haves

Learn more about our unique culture and history .
Platform Architecture & Strategy:  Define and evolve the enterprise Lakehouse reference architecture on Databricks, supporting batch, streaming, and AI workloads at scale across AWS or Azure.
Governance & Security by Design:  Implement lineage, access controls, and auditing with Unity Catalog to meet regulatory requirements and ensure data quality across domains.
MLOps & Engineering Excellence:  Establish CI/CD and infrastructure-as-code standards for data pipelines and ML models. Optimize Spark performance, cluster configuration, and compute spend.
15+ years in data engineering, data architecture, or software architecture, including 5+ years architecting on the Databricks Lakehouse Platform
Deep expertise in Apache Spark, Delta Lake, Unity Catalog, and MLflow
Proven experience designing infrastructure for both traditional ML and Generative AI workloads (RAG, fine-tuning, vector search, agentic frameworks)
Strong proficiency in Python and SQL, infrastructure-as-code (e.g., Terraform), and software design patterns
Hands-on architecture experience on AWS or Microsoft Azure
Demonstrated ability to influence senior technical and business stakeholders without direct authority
A competitive package and exciting growth opportunities for career-oriented teammates
A dynamic, action-oriented, and thoughtful environment centered on always doing the right thing for our customers, teammates and our other stakeholders
A purposely non-bureaucratic organization that embraces simplicity over complexity and emphasizes individual excellence in a team framework
Benefits that support your life and well-being, which include:
Disability Insurance (both short-term and long-term)
Accidental Death & Dismemberment Insurance (for you and your family)
Flexible Spending Accounts
Health Reimbursement Account
Tuition Reimbursement
NOTE: Compensation will be commensurate with experience. This job description is not intended to be all-inclusive. Team Member may perform other related duties as negotiated to meet the ongoing needs of the organization.

Nice-to-haves

Databricks Professional or Expert-level certification
Experience with advanced agentic architectures, DSPy, or graph databases
Background in insurance, financial services, fintech, or other regulated environments
Track record of optimizing cloud compute costs while improving pipeline throughput

Application prep

Use the role’s language and application details while tailoring your materials.

Keywords to mirror

aimlawsazurellmci/cdpythonsql

Platform: adp-workforce-now
Effort: medium

Benefits

Benefits detected in the listing.

Dental

Vision

401(K)

Paid Time Off

Parental Leave