Senior Analytics Engineer at Imagine Worldwide

Job description

About Imagine Worldwide All children have immense potential, but hundreds of millions don’t have access to the learning they need. Imagine Worldwide believes that every child should be empowered with the basic right to literacy and numeracy. We provide an innovative education technology solution and implementation model to the global literacy/numeracy learning crisis using the onebillion application and various toolkits and systems to support implementation. Our model is massively scalable, evidence-based, and can deliver foundational learning to millions of the most marginalized children. Imagine Worldwide partners with governments, organizations, and communities to provide child-directed, tablet-based learning that is accessible, effective, and affordable. We are incorporated in California (United States) as a nonprofit organization, but operate in Sub-Saharan Africa through locally-registered entities in Malawi, Sierra Leone, and Tanzania. Learn more on our website . Role Overview The Senior Analytics Engineer builds, runs and improves Imagine’s analytics platform: the infrastructure, pipelines and core data models that the organisation’s reporting and analysis depend on. The central task is to scale the platform as data volumes grow, keeping costs under control without giving up performance. In a small team, that means covering the infrastructure underneath the data as well as the data engineering itself. Imagine’s data comes from many sources. The largest is tablets in more than 1,500 schools across Malawi, Tanzania and Sierra Leone, which record learning activity as individual events, adding up to billions of records. These only become meaningful when joined with programme context, such as active schools per term, device allocations and enrolment, and with operational data from inventory and field data collection. Turning all of this into reliable data on a non-profit budget calls for careful decisions about where processing happens and what each workload costs. The role reports to the Director of Analytics, who sets the platform’s architecture and technical standards, and works alongside the rest of the Analytics Team. We are looking for someone senior who takes initiative, turns architectural direction into working systems, and brings forward evidence-based proposals for improving them. As the engineer builds context, they take on increasing ownership of the platform’s day-to-day running and technical decisions. Key Responsibilities The Senior Analytics Engineer’s responsibilities fall into four pillars. The first two cover the platform itself: the infrastructure it runs on, and the pipelines that bring data in from many sources and process it at scale. The third covers orchestration and the core data models that the rest of the Analytics Team builds on. The fourth covers the practices that keep the platform reliable, secure and affordable to run. Data Platform and Infrastructure Imagine’s analytics platform runs on AWS and ClickHouse Cloud, with infrastructure managed as code. In a small team, this role carries the infrastructure work that a larger organisation would split across dedicated platform or DevOps staff, limited to the scope of the analytics platform. Build and maintain the platform’s AWS infrastructure, including storage, compute, networking, the data catalogue, secrets and access management (S3, EC2, Glue Data Catalog, Lake Formation, IAM, Secrets Manager). Manage infrastructure as code in Terraform, keeping environments reproducible, version-controlled and reviewable. Administer the ClickHouse Cloud warehouse, including service configuration, user and role management, and monitoring of performance and capacity. Host and maintain the team’s self-hosted tools, including Apache Superset for BI and Apache Airflow for orchestration. This covers deployment, upgrades, authentication and availability. The Senior BI & Data Analyst manages the dashboards and content within Superset. Contribute to architectural decisions on how the platform evolves, setting out options and trade-offs for review with the Director of Analytics. Large-Scale Data Processing and Pipelines Data reaches the platform from application databases, operational systems and field data tools, and lands in an Apache Iceberg data lake on S3. Spark-based processing on AWS Glue does the heavy work before the data moves into ClickHouse for analysis. The aim is to do expensive processing once, as far upstream as practical, so that downstream models and queries stay fast and affordable. Build and maintain ingestion pipelines from the platform’s sources, including application databases, APIs and operational systems, using tools such as AWS Glue, Airbyte and AWS Lambda. Implement incremental loading patterns that stay efficient and recoverable as volumes grow, landing raw data in the lake in a form that preserves its full history. Develop and optimise Spark jobs that process billions of device events into learning sessions and higher-level aggregates, paying close attention to partitioning, skew and compute cost. Bring in and model the programme context that gives event data its meaning, such as schools, terms, device allocations and enrolment, and handle how that context changes over time. Design and maintain Iceberg tables for performance and cost, including partitioning strategy, file sizing, compaction and snapshot management. Validate data at ingestion and surface quality issues early, working with the Analytics Team to trace them back to their source. Contribute to evaluating new processing approaches as volumes grow, including moving more work upstream and assessing where streaming would add value. Orchestration and Transformation Apache Airflow orchestrates the platform’s pipelines, and dbt transforms data in ClickHouse into the models that reporting and analysis rely on. This role builds and maintains the foundational layers of those models. Build and maintain Airflow workflows that coordinate ingestion, processing and transformation, with clear dependencies, retries and alerting. Develop and maintain dbt staging and core models in ClickHouse, following consistent conventions for naming, structure and materialisation. Implement incremental models for high-volume tables, choosing materialisation strategies and table designs that balance freshness, correctness and cost. Write and maintain dbt tests and model contracts for core datasets, so that changes that would break downstream reporting are caught before they reach production. Document core models such as learning sessions as agreed definitions, including their lineage, and make clear where a model applies an approximation rather than a direct transformation. Work with the Senior BI & Data Analyst on the boundary between core and reporting models, and with the Data Scientist on datasets for research and evaluation. Reliability, Cost and Engineering Practice A small team needs a platform that runs predictably and that more than one person understands. This pillar covers the engineering practices that make that possible. Build and maintain CI/CD pipelines in GitHub Actions for infrastructure, pipeline and dbt code, including automated checks that stop unsafe changes from reaching production. Implement monitoring and alerting across pipelines, the warehouse and hosted tools, so that failures and delays are visible quickly and reach the right people. Respond to platform incidents and data issues, carry out root cause analysis, and contribute to the team’s data quality incident process. Apply and maintain access controls and security practices across the data lake, warehouse and BI layer, in line with Imagine’s data governance policies. Monitor platform costs across AWS and ClickHouse, track the cost and performance of individual workloads over time, and use that evidence to identify savings and prioritise optimisation work. Maintain and periodically test backup and recovery procedures for the platform’s core components. Take part in code review and write clear runbooks and technical documentation, following the team’s documentation standards, so that knowledge of the platform is shared across the team. Requirements The qualifications below are grouped by importance. Core requirements are needed to do the role from the start. Strongly preferred skills shorten the time it takes to become effective on Imagine’s platform. Advantageous experience adds value but is not expected. We do not expect any one candidate to bring everything listed here. Education A bachelor’s degree in Computer Science, Software Engineering or a related field, or equivalent practical experience. Demonstrated experience is weighted equally to formal credentials. Experience At least 7 years in data engineering, software engineering or platform engineering roles, including at least 3 years building and running distributed data processing (Spark or equivalent) in production. A track record of taking responsibility for a production data platform, or major parts of one, including its reliability, performance and day-to-day operation. Experience working with datasets in the hundreds of millions to billions of records, and designing processing that stays efficient as volumes grow. Experience running data workloads within real budget constraints: making deliberate trade-offs between compute, storage and performance, and being able to show the effect of those decisions. Experience managing the cloud infrastructure that data workloads run on, not only using managed services. Experience in small teams where engineers cover a broad scope and work with limited supervision. Experience working with remote teams across multiple countries and time zones. Experience in the non-profit, international development or social sector is an advantage but not a requirement. Technical Skills: Core Strong Python, written to production standards, with testing and modular design. Advanced SQL, including writing and optimising queries against analytical databases. Spark (PySpark), including performance tuning for partitioning, shuffles and data skew. A solid understanding of data lake design, including columnar formats such as Parquet, partitioning strategy, and when to process in the lake rather than the warehouse. Hands-on AWS experience with data services and the infrastructure around them, such as S3, Glue or EMR, IAM and EC2. Infrastructure as code, using Terraform, CloudFormation, CDK or similar. Linux administration and Docker, to the level needed to deploy and maintain self-hosted services. dbt in production, including incremental models and testing. Workflow orchestration with Airflow, Dagster, Prefect or similar. Git, CI/CD and code review practice. Technical Skills: Strongly Preferred Apache Iceberg, or another open table format such as Delta Lake or Hudi. ClickHouse, or a comparable columnar warehouse such as Snowflake, BigQuery or Redshift. Terraform specifically. Apache Airflow specifically. AWS Glue and Lake Formation. Data modelling for analytics, including handling reference data that changes over time. Technical Skills: Advantageous Managed Spark platforms such as Databricks. Streaming technologies such as Kafka, Flink, Kinesis or Spark Structured Streaming. Event, telemetry or IoT data, particularly from devices that sync intermittently. Ingestion tools such as Airbyte. Administering BI tools such as Apache Superset. Software development beyond data pipelines, such as building APIs or system integrations. Compensation & Benefits Salary is competitive and commensurate with experience. Location The Senior Analytics Engineer must be based in Africa. The role will frequently collaborate and coordinate with operations teams across Africa, the US, and the UK. This is a full-time remote position; the role can be based anywhere in Africa with reliable internet access and alignment with core working hours across time zones. The role may involve occasional international travel for in-person team meetings, field support or training, as needed. Candidates should be comfortable working independently in a virtual environment and coordinating with colleagues across global teams. Applications are reviewed on a rolling basis * * * Imagine Worldwide is proud to be an equal opportunity employer. All applicants will be considered for employment without attention to race, color, religion, sex, sexual orientation, gender identity, national origin, veteran or disability status.

Requirements

Must-haves

Data Platform and Infrastructure
Build and maintain the platform’s AWS infrastructure, including storage, compute, networking, the data catalogue, secrets and access management (S3, EC2, Glue Data Catalog, Lake Formation, IAM, Secrets Manager).
Manage infrastructure as code in Terraform, keeping environments reproducible, version-controlled and reviewable.
Administer the ClickHouse Cloud warehouse, including service configuration, user and role management, and monitoring of performance and capacity.
Contribute to architectural decisions on how the platform evolves, setting out options and trade-offs for review with the Director of Analytics.
Large-Scale Data Processing and Pipelines
Build and maintain ingestion pipelines from the platform’s sources, including application databases, APIs and operational systems, using tools such as AWS Glue, Airbyte and AWS Lambda.
Implement incremental loading patterns that stay efficient and recoverable as volumes grow, landing raw data in the lake in a form that preserves its full history.
Develop and optimise Spark jobs that process billions of device events into learning sessions and higher-level aggregates, paying close attention to partitioning, skew and compute cost.
Bring in and model the programme context that gives event data its meaning, such as schools, terms, device allocations and enrolment, and handle how that context changes over time.
Design and maintain Iceberg tables for performance and cost, including partitioning strategy, file sizing, compaction and snapshot management.
Validate data at ingestion and surface quality issues early, working with the Analytics Team to trace them back to their source.
Contribute to evaluating new processing approaches as volumes grow, including moving more work upstream and assessing where streaming would add value.
Orchestration and Transformation
Apache Airflow orchestrates the platform’s pipelines, and dbt transforms data in ClickHouse into the models that reporting and analysis rely on. This role builds and maintains the foundational layers of those models.
Build and maintain Airflow workflows that coordinate ingestion, processing and transformation, with clear dependencies, retries and alerting.
Develop and maintain dbt staging and core models in ClickHouse, following consistent conventions for naming, structure and materialisation.
Implement incremental models for high-volume tables, choosing materialisation strategies and table designs that balance freshness, correctness and cost.
Write and maintain dbt tests and model contracts for core datasets, so that changes that would break downstream reporting are caught before they reach production.
Document core models such as learning sessions as agreed definitions, including their lineage, and make clear where a model applies an approximation rather than a direct transformation.
Work with the Senior BI & Data Analyst on the boundary between core and reporting models, and with the Data Scientist on datasets for research and evaluation.
Reliability, Cost and Engineering Practice
A small team needs a platform that runs predictably and that more than one person understands. This pillar covers the engineering practices that make that possible.

Nice-to-haves

Technical Skills: Strongly Preferred

Application prep

Use the role’s language and application details while tailoring your materials.

Keywords to mirror

restawsci/cdpythonsqldocker

Platform: greenhouse
Effort: quick

Benefits

Benefits detected in the listing.

Vision