Job Summary
The Senior Data Engineer plays a key role in advancing the company's AI strategy by designing, developing, and optimizing scalable data platforms. This role focuses on data engineering as part of the AI team, contributing to technical development, data architecture, pipeline optimization, and actionable intelligence. The ideal candidate will have expert-level experience with Databricks, strong software engineering capabilities, and a proven ability to build robust, secure, and scalable data systems. The Senior Data Engineer will work closely with cross-functional teams to deliver high-quality data solutions, optimize production pipelines, and support AI and machine learning capabilities.
Responsibilities
• Design, develop, refine, and implement high-quality data architecture solutions for large-scale, complex data platforms.
• Establish best practices for writing modular, reusable, maintainable, and scalable data engineering code.
• Develop and maintain greenfield data engineering projects and production-level data pipelines.
• Design data-driven solutions to address complex cybersecurity challenges.
• Architect end-to-end data platforms by evaluating technical approaches, trade-offs, risks, and performance requirements.
• Work hands-on with engineers to implement data architecture and pipeline solutions.
• Collaborate with DevOps and MLOps teams to monitor and optimize data pipelines and machine learning workflows.
• Improve model and pipeline performance through monitoring, troubleshooting, and tuning to meet latency and cost requirements.
• Evaluate third-party AI tools against internally developed solutions to maximize business value and return on investment.
• Establish data governance, data quality, and security standards to ensure the ethical and compliant use of sensitive information in AI models.
• Develop and implement incremental development strategies and production release plans aligned with the long-term product vision.
• Serve as the subject matter expert in data engineering for the AI team.
• Ensure data pipelines are reliable, scalable, secure, and optimized for production workloads.
Qualifications
• Bachelor’s degree in Engineering, Computer Science, or a related technical field.
• Advanced degree in Engineering, Computer Science, Machine Learning, or Artificial Intelligence is preferred.
• Strong experience working as an individual contributor with a hands-on engineering mindset and a focus on delivery.
• At least 4 years of experience working with data platform solutions across data and analytics throughout the software development life cycle (SDLC).
• At least 2 years of experience designing and mapping requirements into Databricks data pipeline capabilities on Google Cloud Platform (GCP).
• Hands-on experience with Databricks Lakehouse architecture, including Databricks Datalake and Data Catalog capabilities.
• Experience designing Databricks Medallion Architecture, including data structure, data access, and data quality decisions.
• Strong experience with batch, mini-batch, and real-time data processing, including trade-offs and best practices.
• Hands-on experience with ELT/ETL processes, data pipelines, and orchestration tools.
• Experience processing large volumes of unlabeled data with complex schemas.
• Strong understanding of data modeling techniques and modern data platform trends, including AI applications in data engineering.
• Proficiency in Python, Pandas, PySpark, and related data processing frameworks.
• Experience with AI and machine learning tools such as TensorFlow, PyTorch, Keras, LangChain, LangSmith, and Infrastructure as Code (IaC).
• Experience working with structured and unstructured data formats, including Parquet, PostgreSQL, DuckDB, JSON, and PDF.
• Experience with infrastructure sizing and estimation for data platform solutions.
Preferred Qualifications
• Experience in cybersecurity, automotive, energy, or healthcare industries.
• Experience working in both large enterprises (more than 10,000 employees) and small companies (fewer than 200 employees).
• Familiarity with supervised and unsupervised machine learning models.
• Experience with AI-driven data engineering solutions and integration of AI capabilities into data platforms.
• Knowledge of data governance, security, and compliance best practices.
Technical Skills
• Data Engineering: Data architecture, ETL/ELT, pipeline development, orchestration, data modeling, and data quality.
• Databricks: Lakehouse architecture, Delta Lake concepts, data cataloging, Medallion Architecture, and production pipeline optimization.
• Cloud Platform: Google Cloud Platform (GCP) and cloud-based data platform design.
• Programming: Python, PySpark, Pandas, and SQL.
• Data Processing: Batch, mini-batch, and real-time processing.
• Data Formats and Databases: Parquet, PostgreSQL, DuckDB, JSON, and PDF.
• AI and Machine Learning: TensorFlow, PyTorch, Keras, LangChain, and LangSmith.
• Infrastructure: Infrastructure as Code, infrastructure sizing, performance optimization, and cost management.
• Production Engineering: Monitoring, troubleshooting, release planning, scalability, and reliability.
Soft Skills & Core Competencies
• Excellent verbal and written communication skills.
• Strong collaboration skills and ability to work effectively with Product, Engineering, Field, and other cross-functional teams.
• Strong analytical thinking and problem-solving abilities, with the capacity to analyze complex issues and develop innovative solutions.
• Strong self-management skills and ability to prioritize tasks and manage time effectively.
• A proactive approach to work, with the ability to take ownership and initiative.
• Ability to work independently while contributing effectively to team objectives.