Job description

We are seeking a mid-level Data Platform Engineer to build a new greenfield data platform integrating internal systems (GSCM, SEASI, SMART, and BOSS) using a Medallion Architecture. This role owns the end-to-end data pipeline—from ingestion through Bronze/Silver/Gold transformation—while migrating existing Alteryx workflows to code-based dbt and Python implementations. The engineer will also develop an MCP server enabling AI agents to query business-specific data marts, directly supporting the organization's AI-driven analytics initiatives. Success in this role will deliver a scalable, governed data foundation that accelerates business insights and AI adoption. Responsibilities • Ingest data from sources such as Microsoft SQL Server (BOSS) and Oracle into the Bronze layer, implementing CDC-based incremental ingestion • Develop dbt models for the Silver layer including cleansing, standardization, deduplication, and data quality testing • Develop Gold layer business-specific data marts and migrate existing Alteryx transformation logic • Configure Apache Iceberg tables and a REST Catalog, including schema evolution and time travel support • Develop and operate Apache Airflow DAGs, managing dependencies, scheduling, and failure notifications • Develop an MCP server enabling AI agents to query Gold-layer data marts, including catalog/metadata access, ontology integration, and query execution • Manage metadata and data lineage using OpenMetadata • Automate Excel reporting using Python and openpyxl • 3+ years of hands-on data engineering experience in production environments • Advanced SQL skills, including window functions, CTEs, and query tuning • Hands-on dbt experience, including model design, incremental strategies, macros, and tests • Python proficiency for data processing and automation, including libraries such as pandas • Experience developing Apache Airflow DAGs • Experience with at least one analytical engine: Trino/Presto, Apache Spark, or DuckDB • Experience with at least one lakehouse table format: Apache Iceberg, Delta Lake, or Apache Hudi • Experience with Medallion Architecture and star schema/dimensional modeling • Experience integrating relational database sources, such as Microsoft SQL Server and Oracle, using CDC or batch processing Preferred Qualifications • Experience with EL tools such as Airbyte or Fivetran, and Iceberg REST Catalog implementations such as Apache Polaris • Experience with governance tools such as OpenMetadata or DataHub, and semantic layers such as Cube.dev • Ability to interpret Alteryx workflows (.yxmd) and migrate their logic to SQL/dbt • Experience developing MCP servers or building LLM, RAG, or AI agent pipelines • Experience developing applications, APIs, or backend services using Java and/or Python • Experience building Generative AI (GenAI) applications, including LLM integration, retrieval-augmented generation (RAG), and AI agent workflows • Hands-on experience with LangChain and/or LangGraph for developing tool-enabled AI agents and orchestrating multi-step workflows • Understanding of retail and home appliance business domains, including PSI (Production, Sales, and Inventory) and sales analytics • Understanding of retail and home appliance business domains, including PSI (Production, Sales, and Inventory) and sales analytics Required Technical Skills SQL dbt Python pandas Apache Airflow Trino/Presto or Apache Spark or DuckDB Apache Iceberg or Delta Lake or Apache Hudi Microsoft SQL Server Oracle Kubernetes MinIO (S3-compatible storage) Preferred Technical Skills Java or Python Generative AI (GenAI) and LLM integration LangChain LangGraph RAG and AI agent pipelines MCP server development Airbyte Fivetran Apache Polaris OpenMetadata DataHub Cube.dev Alteryx openpyxl All your information will be kept confidential according to EEO guidelines.

Requirements

Must-haves

Ingest data from sources such as Microsoft SQL Server (BOSS) and Oracle into the Bronze layer, implementing CDC-based incremental ingestion
Develop dbt models for the Silver layer including cleansing, standardization, deduplication, and data quality testing
Develop Gold layer business-specific data marts and migrate existing Alteryx transformation logic
Configure Apache Iceberg tables and a REST Catalog, including schema evolution and time travel support
Develop and operate Apache Airflow DAGs, managing dependencies, scheduling, and failure notifications
Develop an MCP server enabling AI agents to query Gold-layer data marts, including catalog/metadata access, ontology integration, and query execution
Manage metadata and data lineage using OpenMetadata
Automate Excel reporting using Python and openpyxl
3+ years of hands-on data engineering experience in production environments
Advanced SQL skills, including window functions, CTEs, and query tuning
Hands-on dbt experience, including model design, incremental strategies, macros, and tests
Python proficiency for data processing and automation, including libraries such as pandas
Experience developing Apache Airflow DAGs
Experience with at least one analytical engine: Trino/Presto, Apache Spark, or DuckDB

Nice-to-haves

Experience with EL tools such as Airbyte or Fivetran, and Iceberg REST Catalog implementations such as Apache Polaris
Experience with governance tools such as OpenMetadata or DataHub, and semantic layers such as Cube.dev
Ability to interpret Alteryx workflows (.yxmd) and migrate their logic to SQL/dbt
Experience developing MCP servers or building LLM, RAG, or AI agent pipelines
Experience developing applications, APIs, or backend services using Java and/or Python
Experience building Generative AI (GenAI) applications, including LLM integration, retrieval-augmented generation (RAG), and AI agent workflows
Hands-on experience with LangChain and/or LangGraph for developing tool-enabled AI agents and orchestrating multi-step workflows
Understanding of retail and home appliance business domains, including PSI (Production, Sales, and Inventory) and sales analytics
Understanding of retail and home appliance business domains, including PSI (Production, Sales, and Inventory) and sales analytics
Preferred Technical Skills

Application prep

Use the role’s language and application details while tailoring your materials.

Keywords to mirror

pythonaisqlrestllmjavakubernetes

Platform: smartrecruiters
Effort: medium