Job description

10+ Profile Data Engineer IV designs, builds, and maintains the ETL pipelines and data infrastructure that feed the IIA Data Lake, anomaly detection models, and AI agents. This role focuses on constructing robust, scalable data pipelines using Spark/Scala, ensuring data quality and availability across a growing portfolio of network data sources, and enabling downstream consumers (data scientists, agents, dashboards) to access reliable, well-structured data. MAJOR DUTIES AND RESPONSIBILITIES • Design, develop, and maintain scalable ETL pipelines using Apache Spark (Scala) to ingest, transform, and load network data into the IIA Data Lake. • Onboard new data sources (network telemetry, syslogs, SNMP traps, device configuration data, ticketing systems) by building ingestion pipelines from raw source to query-ready format. • Implement monitoring and alerting solutions to ensure data pipeline reliability and performance. • Develop and manage deployment pipelines to facilitate continuous integration and delivery of data engineering solutions. • Manage and optimize data storage solutions, including distributed file systems, relational databases, flat files, and external sources accessed via API. • Implement data quality checks, validation rules, and automated testing to ensure pipeline reliability and data integrity. • Optimize pipeline performance for large-scale data processing (billions of events per day) across batch and mini-batch processing patterns. • Manage and evolve data schemas, partitioning strategies, and storage formats to support efficient querying and downstream consumption. • Support data backfills and recovery when upstream issues or schema changes require reprocessing. • Collaborate across teams to ensure data solutions align with existing production architectures and business requirements. • Work with data scientists and agent developers to understand data requirements and deliver datasets that support anomaly detection models and AI agent workflows. • Provide technical guidance on data engineering best practices and methodologies. • Document and communicate data engineering processes and standards to business-intelligence, data, and analytics professionals with varied backgrounds. • Continuously evaluate and improve data engineering tools and approaches to enhance performance and efficiency. • Perform other duties as required. REQUIRED QUALIFICATIONS Skills/Abilities and Knowledge • Proficiency in Python with experience in distributed data processing (Spark preferred, willingness to learn Scala acceptable) • Strong experience with Apache Spark for distributed data processing • Proficiency in building and maintaining ETL pipelines at scale • Experience with AWS services: S3, Glue, Athena, EMR • Strong understanding of relational databases and SQL • Knowledge of data architecture, data warehousing, partitioning strategies, and columnar storage formats (e.g., Parquet) • Experience implementing data quality checks and validation frameworks • Experience with workflow orchestration tools (Airflow preferred) • Proficiency with Linux-based operating systems and shell scripting • Experience with Git-based version control and collaborative development workflows • Demonstrated ability and desire to continually expand skill set, and learn from and teach others PREFERRED QUALIFICATIONS Skills/Abilities and Knowledge • Experience with streaming or mini-batch data processing (Spark Streaming, structured streaming, or similar) • Experience with Apache Kafka or similar messaging/streaming platforms • Experience with NoSQL databases • Experience in the telecommunications industry or other large-scale network operations environments • Familiarity with network data sources: telemetry, syslogs, SNMP traps, device configuration data • Experience with data integration via REST APIs and cloud SDKs (e.g., boto3) • Experience writing automated tests for data pipelines • Knowledge of text analysis or log parsing technique

Requirements

Must-haves

Design, develop, and maintain scalable ETL pipelines using Apache Spark (Scala) to ingest, transform, and load network data into the IIA Data Lake.
Onboard new data sources (network telemetry, syslogs, SNMP traps, device configuration data, ticketing systems) by building ingestion pipelines from raw source to query-ready format.
Implement monitoring and alerting solutions to ensure data pipeline reliability and performance.
Develop and manage deployment pipelines to facilitate continuous integration and delivery of data engineering solutions.
Manage and optimize data storage solutions, including distributed file systems, relational databases, flat files, and external sources accessed via API.
Implement data quality checks, validation rules, and automated testing to ensure pipeline reliability and data integrity.
Optimize pipeline performance for large-scale data processing (billions of events per day) across batch and mini-batch processing patterns.
Manage and evolve data schemas, partitioning strategies, and storage formats to support efficient querying and downstream consumption.
Support data backfills and recovery when upstream issues or schema changes require reprocessing.
Collaborate across teams to ensure data solutions align with existing production architectures and business requirements.
Work with data scientists and agent developers to understand data requirements and deliver datasets that support anomaly detection models and AI agent workflows.
Provide technical guidance on data engineering best practices and methodologies.
Document and communicate data engineering processes and standards to business-intelligence, data, and analytics professionals with varied backgrounds.

Nice-to-haves

Proficiency in Python with experience in distributed data processing (Spark preferred, willingness to learn Scala acceptable)
Experience with workflow orchestration tools (Airflow preferred)
Skills/Abilities and Knowledge
Experience with streaming or mini-batch data processing (Spark Streaming, structured streaming, or similar)
Experience with Apache Kafka or similar messaging/streaming platforms
Experience with NoSQL databases
Experience in the telecommunications industry or other large-scale network operations environments
Familiarity with network data sources: telemetry, syslogs, SNMP traps, device configuration data
Experience with data integration via REST APIs and cloud SDKs (e.g., boto3)
Experience writing automated tests for data pipelines
Knowledge of text analysis or log parsing technique

Application prep

Use the role’s language and application details while tailoring your materials.

Keywords to mirror

aiapipythonawssqlrest

Platform: careers-browser
Effort: medium