Job description

Primary Skills: AWS Data Engineering (Expert), Data Mesh Architecture (Expert), Databricks & Modern Data Platforms (Expert), DevSecOps & Infrastructure as Code (Advanced), AI-Augmented Data Engineering (Advanced) Contract Type: W2 Only Duration: 18+ Months Location: San Francisco, CA (Hybrid) Pay Range: $90 - $100 on W2 Job Summary: We are seeking a Lead Cloud Data Engineer – Data Mesh & AI to lead the design, development, and modernization of enterprise-scale data platforms supporting large data warehouse and data hub initiatives. The ideal candidate will have deep expertise in AWS data engineering, Data Mesh architecture, Databricks, DevSecOps, and AI-driven engineering practices. This role will drive the implementation of scalable data products, federated governance, and AI-augmented solutions while providing technical leadership across cross-functional engineering teams. Key Responsibilities: Design, develop, and maintain scalable ETL/ELT pipelines using AWS services including Glue, EMR, Lambda, Kinesis, Step Functions, Redshift, and S3. Architect and implement enterprise Data Mesh solutions with domain-oriented ownership, federated governance, and reusable self-service data product frameworks. Develop modern data platforms leveraging Databricks, Delta Lake, Unity Catalog, Starburst/Trino, Collibra, and Immuta. Implement enterprise data governance, metadata management, data lineage, access controls, and automated data quality monitoring frameworks. Build and maintain CI/CD pipelines, Infrastructure as Code (Terraform/CloudFormation), containerized deployments, and DevSecOps automation. Design and implement AI-powered solutions including RAG pipelines, agentic AI workflows, Amazon Bedrock, LLM integrations, and AI-assisted SDLC automation. Optimize cloud infrastructure for scalability, security, performance, and cost efficiency across hybrid cloud environments. Provide L3 production support, mentor engineering teams, and establish best practices for modern data engineering and Data Mesh adoption. Lead technical architecture reviews, collaborate with business stakeholders, and translate business requirements into scalable technical solutions. Create and maintain technical architecture documentation, implementation standards, and engineering best practices. Must-have Skills: 8+ years of experience in Data Engineering with at least 5+ years designing enterprise AWS Data Platforms. Strong hands-on experience with AWS S3, Glue, EMR, Redshift, Lambda, Kinesis, Lake Formation, and Step Functions. Proven experience implementing Data Mesh architecture, domain-driven data products, and federated governance. Expertise with Databricks (Delta Lake, Unity Catalog), Starburst/Trino, Collibra, and Immuta. Strong knowledge of ETL/ELT development, data modeling, metadata management, data lineage, and data quality frameworks. Experience with Terraform, CloudFormation, Docker, ECS, GitHub Actions/GitLab CI, Jenkins, and DevSecOps best practices. Experience implementing AI-enabled engineering solutions using Amazon Bedrock, LLMs, RAG architectures, Copilot, Claude Code, or similar AI frameworks. Strong SQL, Python, and cloud-native data engineering expertise. Experience supporting secure, compliant cloud environments, including AWS GovCloud or regulated environments. Excellent leadership, mentoring, architecture design, communication, and stakeholder management skills. Nice-to-have Skills: Experience supporting enterprise data warehouse modernization initiatives. Knowledge of hybrid cloud and on-premises data platform integration. Experience implementing self-service analytics and reusable data product frameworks. Familiarity with cost optimization strategies for large-scale cloud data platforms. Experience working within highly regulated industries such as Financial Services or Government. Preferred Qualifications: Bachelor's or Master's degree in Computer Science, Data Engineering, Information Systems, or a related technical field. AWS Certifications (Solutions Architect, Data Analytics, or Developer) preferred. Experience leading enterprise-scale cloud modernization and AI transformation initiatives. Strong background in enterprise architecture, technical leadership, and cross-functional collaboration. ABOUT AKRAYA Akraya is an award-winning IT staffing firm consistently recognized for our commitment to excellence and a thriving work environment. Most recently, we were recognized Stevie Employer of the Year 2025, SIA Best Staffing Firm to work for 2025, Inc 5000 Best Workspaces in US (2025 & 2024) and Glassdoor's Best Places to Work (2023 & 2022)! Industry Leaders in Tech Staffing As Talent solutions provider for Fortune 100 Organizations, Akraya's industry recognitions solidify our leadership position in the IT staffing space. We don't just connect you with great jobs, we connect you with a workplace that inspires! Join Akraya Today! Let us lead you to your dream career and experience the Akraya difference. Browse our open positions and join our team!

Requirements

Requirements not extracted — see the original description below.

Application prep

Use the role’s language and application details while tailoring your materials.

Keywords to mirror

awsaici/cdllmdockersqlpython

Platform: careers-page
Effort: medium