Job description

Design, build, and maintain scalable AI data platforms, services, and APIs that support and enable AI model development and production. Develop data ingestion, transformation, and publishing pipelines for structured, unstructured, and multimodal data. Build AI-ready datasets through ground truth creation, data curation, annotation workflows, dataset versioning, and metadata management. Develop data quality frameworks, validation pipelines, observability, and evaluation metrics to ensure trusted AI datasets. Design and implement Retrieval-Augmented Generation (RAG) pipelines, embedding workflows, vector database integrations, and metadata services for enterprise AI applications. Build scalable platform capabilities for managing the end-to-end AI data lifecycle, including ground truth dataset creation, dataset versioning, metadata and lineage management, automated data quality validation, governance, and secure publishing of AI-ready datasets. Collaborate with AI/ML engineers, software engineers, product teams, and domain experts to define AI data requirements and deliver production-ready data solutions. Optimize platform scalability, reliability, performance, security, and cost across cloud-native environments. Drive engineering best practices for AI data architecture, platform design, automation, testing, monitoring, and operational excellence. Evaluate emerging AI technologies and continuously improve platform capabilities that enable GenAI, agentic AI, and embodied AI solutions. Bachelor's or Master's degree in Computer Science, Software Engineering, Data Engineering, or a related field. 5+ Experience designing and building scalable data platforms and distributed systems. Strong programming skills in Python and SQL, with proficiency in Java or Scala preferred. Experience with Airflow, Kubeflow, or MLflow to build and orchestrate scalable AI data pipelines. Experience building scalable batch and streaming data pipelines using Spark (PySpark), Kafka, Airflow, and Ray, with proficiency in Pandas and modern data lake/lakehouse architectures (e.g., Iceberg, Delta Lake). Hands-on experience with AI data engineering, including ground truth dataset creation, data curation, annotation pipelines, dataset versioning, and metadata management. Experience implementing data validation, quality frameworks, observability, and AI dataset evaluation. Knowledge of RAG architectures, embedding generation, vector databases, and AI data preparation for LLMs and agentic AI. Experience with cloud platforms (AWS, Azure, or GCP), Kubernetes, Docker, CI/CD, and Infrastructure as Code. Strong understanding of distributed systems, APIs, microservices, and enterprise integration patterns. Excellent communication, collaboration, and technical leadership skills. Experience building platforms supporting GenAI, Agentic AI, or Embodied AI applications. Experience with multimodal datasets, knowledge graphs, AI evaluation frameworks, or vector search technologies. Familiarity with enterprise data governance, lineage, metadata management, and AI compliance. Experience working with manufacturing, operational, IoT, or industrial data platforms. Demonstrated ability to lead technical initiatives and mentor engineers.

Requirements

Must-haves

Design, build, and maintain scalable AI data platforms, services, and APIs that support and enable AI model development and production.
Develop data ingestion, transformation, and publishing pipelines for structured, unstructured, and multimodal data.
Build AI-ready datasets through ground truth creation, data curation, annotation workflows, dataset versioning, and metadata management.
Develop data quality frameworks, validation pipelines, observability, and evaluation metrics to ensure trusted AI datasets.
Design and implement Retrieval-Augmented Generation (RAG) pipelines, embedding workflows, vector database integrations, and metadata services for enterprise AI applications.
Collaborate with AI/ML engineers, software engineers, product teams, and domain experts to define AI data requirements and deliver production-ready data solutions.
Optimize platform scalability, reliability, performance, security, and cost across cloud-native environments.
Drive engineering best practices for AI data architecture, platform design, automation, testing, monitoring, and operational excellence.
Evaluate emerging AI technologies and continuously improve platform capabilities that enable GenAI, agentic AI, and embodied AI solutions.
Bachelor's or Master's degree in Computer Science, Software Engineering, Data Engineering, or a related field.
5+ Experience designing and building scalable data platforms and distributed systems.
Experience with Airflow, Kubeflow, or MLflow to build and orchestrate scalable AI data pipelines.
Experience building scalable batch and streaming data pipelines using Spark (PySpark), Kafka, Airflow, and Ray, with proficiency in Pandas and modern data lake/lakehouse architectures (e.g., Iceberg, Delta Lake).
Hands-on experience with AI data engineering, including ground truth dataset creation, data curation, annotation pipelines, dataset versioning, and metadata management.
Experience implementing data validation, quality frameworks, observability, and AI dataset evaluation.
Knowledge of RAG architectures, embedding generation, vector databases, and AI data preparation for LLMs and agentic AI.
Experience with cloud platforms (AWS, Azure, or GCP), Kubernetes, Docker, CI/CD, and Infrastructure as Code.
Strong understanding of distributed systems, APIs, microservices, and enterprise integration patterns.
Excellent communication, collaboration, and technical leadership skills.
Experience building platforms supporting GenAI, Agentic AI, or Embodied AI applications.
Experience with multimodal datasets, knowledge graphs, AI evaluation frameworks, or vector search technologies.
Familiarity with enterprise data governance, lineage, metadata management, and AI compliance.
Experience working with manufacturing, operational, IoT, or industrial data platforms.

Nice-to-haves

Strong programming skills in Python and SQL, with proficiency in Java or Scala preferred.

Application prep

Use the role’s language and application details while tailoring your materials.

Keywords to mirror

aimlpythonsqljavaawsazuregcpkubernetesdockerci/cd

Platform: apple
Effort: medium