Job description

Role Summary We are looking for a talented and driven engineer to join the Ascend Sandbox team — Experian's cloud-based analytical platform that empowers clients and data scientists to explore, model, and derive insights from data in a secure, governed environment. You will play a key role in building, maintaining, and scaling the platform that serves both internal teams and external clients across the APAC region. What you’ll be doing • Design, build, test, deploy and operate scalable batch and, where applicable, streaming data pipelines for the Ascend Sandbox analytical platform. • Develop high-quality Python, PySpark and SQL solutions to ingest, transform and process large-scale datasets with accuracy, reliability and efficiency. • Design data models and reusable, trusted datasets/data products for analytics, data science and downstream applications. • Engineer and optimise AWS-based data solutions using relevant services such as Amazon S3, AWS Glue, Amazon Athena, Amazon Redshift and Amazon EMR, according to the platform architecture. • Apply data lake and lakehouse engineering practices, including efficient storage formats, partitioning, schema evolution and storage/query optimisation. • Design robust ETL/ELT patterns, including incremental processing, dependency management, idempotency, retries, backfills and failure recovery. • Implement workflow orchestration and scheduling for data pipelines using the platform's approved tooling, such as Airflow/MWAA, AWS Step Functions or AWS Glue Workflows. • Build automated data quality controls and monitoring for completeness, validity, reconciliation, schema compliance, freshness, pipeline health and processing SLAs/SLOs. • Optimise Spark workloads and distributed data processing for performance, scalability and cost, including partitioning, joins, shuffles, skew and resource utilisation. • Implement practical data governance, metadata, lineage, cataloguing, security and access-control requirements across data pipelines and datasets. • Collaborate with clients, business users, data scientists and technical teams to translate business and data requirements into scalable data-processing solutions and integrations. • Investigate and resolve production data and pipeline issues, perform root-cause analysis, and implement preventive improvements to reliability and operational efficiency. • Ensure data engineering solutions comply with established architecture, coding, testing, security and quality standards, and contribute to code reviews, CI/CD and automated deployment practices. • Must possess a Degree / Advanced Diploma in Computer Science, Data Engineering, Software Engineering or equivalent. Seniority will be commensurate with relevant experience and qualifications. • At least 6 years of experience in data engineering or developing data-intensive systems, including the design, build and production support of data pipelines. • Strong hands-on experience with Python and PySpark for large-scale and distributed data processing. • Strong SQL skills, including complex transformations, joins, window functions, query optimisation and performance tuning. • Hands-on experience designing and operating scalable ETL/ELT data pipelines, including incremental loads, retries, backfills, dependency management and failure recovery. • Experience with data modelling for analytical use cases, including dimensional modelling, facts/dimensions, normalization/denormalization and handling historical data. • Hands-on experience with AWS data services relevant to data engineering, such as Amazon S3, AWS Glue, Amazon Athena, Amazon Redshift and Amazon EMR; exposure to Lambda, Kinesis/MSK, DMS or Lake Formation is an advantage where relevant. • Experience with data lake/lakehouse concepts, storage layout, partitioning, schema evolution and efficient data formats such as Parquet; familiarity with JSON, CSV and Avro is useful. • Strong understanding of Spark performance engineering, including partitioning, shuffles, joins, data skew, parallelism, memory/resource optimisation and troubleshooting. • Experience with pipeline orchestration and workflow scheduling using tools such as Airflow/MWAA, AWS Step Functions, AWS Glue Workflows or equivalent. • Experience implementing automated data quality checks and data observability, including monitoring pipeline health, data freshness, volumes, failures and quality anomalies. • Practical knowledge of data governance, metadata management, data cataloguing, lineage and schema management. • Practical understanding of AWS data security, including IAM/least privilege, encryption/KMS, secrets management, S3 security and data-access controls. • Familiarity with Infrastructure as Code such as Terraform, AWS CloudFormation or AWS CDK is an advantage. • Understanding of cloud cost and performance optimisation for storage, compute, Spark workloads and query patterns. • Experience with Git-based workflows, branching strategies, code reviews, CI/CD pipelines and automated deployment. Familiarity with Jira, Bitbucket, Jenkins and Checkmarx is an advantage. • Experience writing automated unit, integration and data-pipeline tests for Python/PySpark transformations and data quality. • Strong testing, debugging, troubleshooting and root-cause analysis skills in production data environments. • Good understanding of Agile, SCRUM, SDLC and software engineering practices. • Strong communication and stakeholder management skills with the ability to engage clients, business users, data scientists and technical teams. • Strong analytical thinking, problem-solving, teamwork and willingness to learn, with the ability to work independently when required and deliver under tight deadlines. • Familiarity with AI tools for coding assistance, documentation or automation is a plus. Recruitment Fraud Awareness - Experian's recruitment process is conducted only through authorised channels. Recruitment communications will only be sent from an @ experian.com email address. Experian will never ask candidates to make any payment as part of an application, interview, assessment, onboarding, or recruitment process. To apply for roles or verify opportunities, please visit experian.com/careers . Experian Careers - Creating a better tomorrow together Find out what its like to work for Experian by clicking here Experian Careers - Creating a better tomorrow together Find out what its like to work for Experian by clicking here

Requirements

Must-haves

What you’ll be doing
Design, build, test, deploy and operate scalable batch and, where applicable, streaming data pipelines for the Ascend Sandbox analytical platform.
Develop high-quality Python, PySpark and SQL solutions to ingest, transform and process large-scale datasets with accuracy, reliability and efficiency.
Design data models and reusable, trusted datasets/data products for analytics, data science and downstream applications.
Engineer and optimise AWS-based data solutions using relevant services such as Amazon S3, AWS Glue, Amazon Athena, Amazon Redshift and Amazon EMR, according to the platform architecture.
Apply data lake and lakehouse engineering practices, including efficient storage formats, partitioning, schema evolution and storage/query optimisation.
Design robust ETL/ELT patterns, including incremental processing, dependency management, idempotency, retries, backfills and failure recovery.
Implement workflow orchestration and scheduling for data pipelines using the platform's approved tooling, such as Airflow/MWAA, AWS Step Functions or AWS Glue Workflows.
Build automated data quality controls and monitoring for completeness, validity, reconciliation, schema compliance, freshness, pipeline health and processing SLAs/SLOs.
Optimise Spark workloads and distributed data processing for performance, scalability and cost, including partitioning, joins, shuffles, skew and resource utilisation.
Implement practical data governance, metadata, lineage, cataloguing, security and access-control requirements across data pipelines and datasets.
Collaborate with clients, business users, data scientists and technical teams to translate business and data requirements into scalable data-processing solutions and integrations.
Investigate and resolve production data and pipeline issues, perform root-cause analysis, and implement preventive improvements to reliability and operational efficiency.
Ensure data engineering solutions comply with established architecture, coding, testing, security and quality standards, and contribute to code reviews, CI/CD and automated deployment practices.
Must possess a Degree / Advanced Diploma in Computer Science, Data Engineering, Software Engineering or equivalent. Seniority will be commensurate with relevant experience and qualifications.
At least 6 years of experience in data engineering or developing data-intensive systems, including the design, build and production support of data pipelines.
Strong hands-on experience with Python and PySpark for large-scale and distributed data processing.
Strong SQL skills, including complex transformations, joins, window functions, query optimisation and performance tuning.
Hands-on experience designing and operating scalable ETL/ELT data pipelines, including incremental loads, retries, backfills, dependency management and failure recovery.
Experience with data modelling for analytical use cases, including dimensional modelling, facts/dimensions, normalization/denormalization and handling historical data.
Experience with data lake/lakehouse concepts, storage layout, partitioning, schema evolution and efficient data formats such as Parquet; familiarity with JSON, CSV and Avro is useful.
Strong understanding of Spark performance engineering, including partitioning, shuffles, joins, data skew, parallelism, memory/resource optimisation and troubleshooting.
Experience with pipeline orchestration and workflow scheduling using tools such as Airflow/MWAA, AWS Step Functions, AWS Glue Workflows or equivalent.

Nice-to-haves

Familiarity with AI tools for coding assistance, documentation or automation is a plus.

Application prep

Use the role’s language and application details while tailoring your materials.

Keywords to mirror

pythonsqlawsci/cdjiraagilescrumai

Platform: smartrecruiters
Effort: medium