Colombia, Europe, Ukraine
Work type:
Office/Remote
Technical Level:
Senior
Job Category:
Software Development
Project:
Global stock photography provider
Our client is looking for a Senior Data Engineer to design, scale, and maintain high-performance ingestion, Change Data Capture (CDC), and delivery pipelines. In this role, you will architect enterprise-grade data platforms to process large-scale metadata, algorithmic feature extraction, and licensable asset datasets.
This team sits at the foundation of the data processing and feature enrichment pipelines. Driving consistency and performance across an ever-growing data lake, we provide guardrails and sound system design patterns to inform the creation and fine-tuning of the industry's cutting-edge foundational large language models.
You will leverage Google Cloud Platform (GCP) as the primary compute foundation of a multi-cloud data processing ecosystem, alongside hybrid data warehouses and query engines including Snowflake and ClickHouse to provide low-latency data exposure, incremental delta tracking, and seamless integration endpoints for downstream AI/ML consumption.
Primary Tech Stack: GCP, ClickHouse, Snowflake, dbt Core, Apache Airflow, Python, SQL.
Responsibilities
1. Ingestion Architecture & GCP Foundation:
- Architect, build, and optimize scalable job creation, batching, and high-throughput ingestion pipelines hosted on GCP.
- Implement serverless and containerized data pipelines using GCP infrastructure (BigQuery, Dataflow, Pub/Sub, Cloud Storage, GKE).
2. Enterprise ETL, CDC & Incremental Processing:
- Design robust Change Data Capture (CDC) and incremental update workflows to enable fast delta synchronization across platforms.
- Build dbt transformation models and Airflow orchestrations following Data Vault / dimensional modeling patterns.
- Implement delta tracking methods (e.g., hash keys, hash diffs) to track row-level asset modifications and metadata events.
3. Multi-Engine Analytics & Query Optimization:
- Maintain and optimize Snowflake integration layers, handling deduplication logic, canonical content filtering, and policy restrictions.
- Deploy and manage ClickHouse as a high-concurrency analytical engine to deliver sub-second metadata retrieval and indexing.
- Construct dedicated integration views and API query layers allowing internal and external clients to consume real-time delta updates.
4. Data Quality, Governance & Scalability:
- Enforce metadata schema standardization, canonical asset mapping, and automated validation layers across diverse asset types (images, video, 3D models, audio).
- Collaborate closely with cross-functional teams to integrate metadata pipelines with feature extraction tools, search indexes, and licensing platforms.
Requirements
- Google Cloud Platform (GCP): 4+ years of production experience with core GCP data technologies (BigQuery, Dataflow, Cloud Pub/Sub, GKE, Google Cloud Storage).
- Database & Query Engines: Deep technical proficiency with both Snowflake (warehousing, semi-structured queries, integration layers) and ClickHouse (real-time aggregation, low-latency columnar storage, indexing).
- Enterprise ETL / CDC: Proven experience architecting CDC pipelines and incremental delta loading for multi-terabyte dataset scale.
- Data Transformation & Orchestration: Hands-on mastery of dbt Core/Cloud and Apache Airflow in an enterprise CI/CD setting (e.g., GitHub Actions).
- Programming Skills: Advanced skills in Python and SQL; familiarity with Go, Java, or Scala is a strong plus.
Nice-to-Have Qualifications:
- Experience with Data Vault 2.0 architecture, hash-key generation, and raw/staging/integration layer design.
- Background handling AI/ML metadata, computer vision feature vectors, C2PA provenance headers, or semantic search configurations.
- Familiarity with media metadata schemas (images, 3D model stems, video derivatives).
- Experience with other major cloud providers such as AWS, Azure, or Oracle Cloud.
Key Success Metrics (First 90 Days)
- GCP Pipeline Optimization: Standardize ingestion batching and pipeline compute infrastructure on GCP.
- ClickHouse & Snowflake Integration: Establish streamlined integration views exposing low-latency metadata delta changes.
- Automated CDC Workflows: Finalize automated dbt delta transformations with hash tracking across primary warehouse layers.
We offer*:
• Flexible working format - remote, office-based or flexible
• A competitive salary and good compensation package
• Personalized career growth
• Professional development tools (mentorship program, tech talks and trainings, centers of excellence, and more)
• Active tech communities with regular knowledge sharing
• Education reimbursement
• Memorable anniversary presents
• Corporate events and team buildings
• Other location-specific benefits
*not applicable for freelancers
You may also be interested in:
Middle Data Engineer (#5808)
Type:
Office/Remote
Category: Software Development
Ukraine
Senior Data Engineer (Snowflake) (#5815)
Type:
Office/Remote
Category: Software Development
Project: Micro-financing fintech company
LATAM, Portugal, Spain
Senior Applied ML Engineer (#5882)
Type:
Office/Remote
Category: Software Development
Project: Global stock photography provider
Europe, Ukraine
×
Easy apply
Easy Apply
or
Refer a friend
Share
Save
×
Subscribe to your search result
We’ll drop you a note when there are new jobs that match your search.
We value your privacy
We use cookies on our website to give you the most relevant experience by remembering your preferences and repeat visits. By clicking “Accept”, you consent to the use of ALL the cookies. However you may visit Cookie Settings to provide a controlled consent.
Please review our Cookie Policy and Privacy Notice
(updated 12.03.2025)
Customize Reject All Accept All
Consent Preferences
Close
Privacy Overview
This website uses cookies to improve your experience while you navigate through the website. Out of these cookies, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may have an effect on your browsing experience.
Necessary
Necessary
Always Enabled
Necessary cookies are absolutely essential for the website to function properly. These cookies ensure basic functionalities and security features of the website, anonymously.
Cookie Duration Description
cookielawinfo-checkbox-analytics 11 months This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".
cookielawinfo-checkbox-functional 11 months The cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".
cookielawinfo-checkbox-necessary 11 months This cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".
cookielawinfo-checkbox-others 11 months This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.
cookielawinfo-checkbox-performance 11 months This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".
viewed_cookie_policy 11 months The cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.
Functional
functional
Functional cookies help to perform certain functionalities like sharing the content of the website on social media platforms, collect feedbacks, and other third-party features.
Performance
performance
Performance cookies are used to understand and analyze the key performance indexes of the website which helps in delivering a better user experience for the visitors.
Analytics
analytics
Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics the number of visitors, bounce rate, traffic source, etc.
Advertisement
advertisement
Advertisement cookies are used to provide visitors with relevant ads and marketing campaigns. These cookies track visitors across websites and collect information to provide customized ads.
Others
others
Other uncategorized cookies are those that are being analyzed and have not been classified into a category as yet.
Reject All Save My Preferences Accept All