Data Engineering Manager
Location: Toronto, Ontario, Canada (Hybrid/Onsite as required)
Employment Type: Full-Time
About the Role
We are seeking a highly skilled Data Engineering Manager to lead the design, development, and optimization of our Databricks and AWS-based data ecosystem. In this role, you will oversee a team of data engineers, drive engineering best practices, and ensure the performance, reliability, and scalability of the organization's data platforms. This position is ideal for a hands-on technical leader with deep expertise in Databricks, Spark, and modern cloud data frameworks.
What You'll Do
•
Lead and mentor a team of data engineers in architecting and maintaining the Databricks Lakehouse Platform, including Delta Lake versioning and large-scale data processing.
•
Manage and optimize Databricks infrastructure, including cluster lifecycle management, cost control, and integration with AWS services (S3, Glue, Lambda, and more).
•
Architect and implement scalable ETL/ELT pipelines using Spark (Python/Scala), including streaming workloads when necessary.
•
Drive technical excellence through advanced Spark performance tuning, cluster optimization, and I/O efficiency improvements.
•
Implement strong data security and governance using Unity Catalog, ensuring alignment with industry and internal compliance standards.
•
Champion Agile development practices, conduct code reviews, and serve as a technical advisor across cross-functional teams.
•
Build and maintain monitoring and alerting solutions for data pipeline reliability, including SLAs, KPIs, and operational health checks.
•
Develop and manage end-to-end CI/CD workflows using Git, automated testing, and continuous deployment pipelines.
What You Bring
•
Bachelor's degree in Computer Science, Engineering, or a related field.
•
5+ years of hands-on experience with Databricks and Apache Spark, including building production-grade data pipelines.
•
Proven experience leading and mentoring data engineering teams in a fast-paced environment.
•
Deep expertise with AWS services: S3, EC2, Glue, EMR, Lambda, Step Functions.
•
Strong programming proficiency in Python (PySpark) or Scala, with advanced SQL and data modeling skills.
•
Experience with Infrastructure-as-Code tools such as Terraform or AWS CloudFormation.
•
Strong understanding of data warehousing concepts, dimensional modeling, and RDBMS platforms (e.g., Postgres, Redshift).
•
Proficiency with Git and CI/CD tools, including automated testing and deployment workflows.
•
Excellent communication and stakeholder engagement capabilities.
•
Demonstrated experience leveraging AI tools or techniques throughout the engineering lifecycle.
•
Knowledge of the financial industry is preferred.
Jeanhel Acuba
Senior Technical Recruiter
PRI Technology
P:973.528.1947
Jeanhel.Acuba@pritechnology.com