Job description

Overview We are seeking an experienced Data Engineer to design, develop, and maintain scalable data processing solutions supporting complex data integration, transformation, and analytics requirements. The ideal candidate will have strong experience with Python, PySpark, Apache Spark, SQL databases, and ETL processes, along with the ability to transform large volumes of structured and unstructured data into meaningful, actionable information. This role requires strong analytical and problem-solving skills, hands-on experience with data pipelines and data quality analysis, and the ability to collaborate with customers, developers, and integration partners in an Agile development environment. What will you do? • Design, develop, and maintain scalable data processing and ETL pipelines using Python, PySpark, and Apache Spark. • Extract, transform, and load data from multiple structured and unstructured sources into optimized, query-ready datasets. • Develop and maintain data mapping, transformation, and integration processes to support business and technical requirements. • Process and convert large volumes of operating system logs, application logs, and other data sources into structured reports, metrics, and analytical dashboards. • Transform disparate data formats into enriched, indexed, and query-friendly datasets. • Develop and optimize data processing workflows to improve performance, efficiency, and reliability. • Develop and maintain data integration solutions using SQL, MySQL, and PostgreSQL. • Work with structured and semi-structured data formats, including XML and JSON. • Perform data extraction, cleansing, transformation, and validation to ensure accuracy and consistency. • Develop and maintain data models, database queries, and data mapping specifications. • Utilize Regular Expressions (RegEx) to parse, extract, validate, and transform complex data. • Support integration of data from multiple systems, applications, and external sources. • Develop and maintain ETL design documentation, including source-to-target mappings, transformation rules, and data dictionaries. • Utilize development environments and tools such as Jupyter Notebooks and Visual Studio to support data modeling, analysis, and development. • Implement data validation and quality assurance processes to ensure data integrity, accuracy, and completeness. • Troubleshoot and resolve data processing, integration, and performance issues. • Maintain technical documentation covering data workflows, processing logic, and integration requirements. Do you have what it takes? • Active TS/SCI with Polygraph • Bachelor’s degree in Computer Science, Information Technology, or a related technical field. • Demonstrated experience developing data processing solutions using Python, PySpark, and Apache Spark. • Hands-on experience designing, developing, and maintaining ETL pipelines and data transformation workflows. • Strong experience with data extraction, mapping, transformation, integration, and loading processes. • Proficiency in SQL and experience working with relational databases such as MySQL and PostgreSQL. • Experience processing structured, semi-structured, and unstructured datasets. • Experience working with XML, JSON, and other common data formats. • Strong knowledge of Regular Expressions (RegEx) for data parsing, extraction, and manipulation. • Experience analyzing operating system and application logs and converting them into actionable reports and dashboards. • Experience developing reports and visualizations using AWS CloudWatch, Kibana, or comparable tools. • Experience performing data profiling, data quality analysis, validation, and troubleshooting. • Ability to develop detailed ETL documentation, including source-to-target mappings and data dictionaries. • Experience using Jupyter Notebooks, Visual Studio, or similar development environments. • Strong analytical, troubleshooting, and problem-solving skills. • Experience collaborating with customers and integration partners to define and clarify technical requirements. • Familiarity with Agile software development methodologies and practices.

Requirements

Must-haves

What will you do?
Design, develop, and maintain scalable data processing and ETL pipelines using Python, PySpark, and Apache Spark.
Extract, transform, and load data from multiple structured and unstructured sources into optimized, query-ready datasets.
Develop and maintain data mapping, transformation, and integration processes to support business and technical requirements.
Process and convert large volumes of operating system logs, application logs, and other data sources into structured reports, metrics, and analytical dashboards.
Transform disparate data formats into enriched, indexed, and query-friendly datasets.
Develop and optimize data processing workflows to improve performance, efficiency, and reliability.
Develop and maintain data integration solutions using SQL, MySQL, and PostgreSQL.
Work with structured and semi-structured data formats, including XML and JSON.
Perform data extraction, cleansing, transformation, and validation to ensure accuracy and consistency.
Develop and maintain data models, database queries, and data mapping specifications.
Utilize Regular Expressions (RegEx) to parse, extract, validate, and transform complex data.
Support integration of data from multiple systems, applications, and external sources.
Develop and maintain ETL design documentation, including source-to-target mappings, transformation rules, and data dictionaries.
Utilize development environments and tools such as Jupyter Notebooks and Visual Studio to support data modeling, analysis, and development.
Implement data validation and quality assurance processes to ensure data integrity, accuracy, and completeness.
Troubleshoot and resolve data processing, integration, and performance issues.
Maintain technical documentation covering data workflows, processing logic, and integration requirements.
Do you have what it takes?
Active TS/SCI with Polygraph
Bachelor’s degree in Computer Science, Information Technology, or a related technical field.
Demonstrated experience developing data processing solutions using Python, PySpark, and Apache Spark.
Hands-on experience designing, developing, and maintaining ETL pipelines and data transformation workflows.
Strong experience with data extraction, mapping, transformation, integration, and loading processes.

Nice-to-haves

None flagged.

Application prep

Use the role’s language and application details while tailoring your materials.

Keywords to mirror

pythonsqlagileaws

Platform: jibe
Effort: medium