Job Summary
Lead and manage data engineering teams to design and deliver scalable enterprise data platforms and pipelines, driving technical execution and collaboration with stakeholders to meet business requirements.
Responsibilities
- Lead data engineering teams to design, develop, and deliver scalable enterprise data platforms and pipelines that meet business needs
- Maintain hands-on expertise in Big Data technologies including Hadoop, Spark, PySpark, Hive, Kafka, and distributed data processing to solve complex engineering challenges
- Design and implement robust ETL/ELT pipelines, data ingestion frameworks, data transformation, and integration solutions for large-scale data processing
- Drive data engineering architecture, perform technical design, and optimize performance of data workloads to ensure scalability and efficiency
- Manage implementations of data lakes, data warehouses, and modern cloud-based data platforms to support enterprise data needs
- Utilize cloud platforms such as AWS, Azure, or GCP and technologies like Databricks to build and maintain data engineering solutions
- Lead initiatives on data quality, data governance, metadata management, security, and data reliability to uphold enterprise standards
- Collaborate with solution architects, data architects, business stakeholders, and project managers to translate business requirements into technical solutions
- Manage project delivery including resource allocation, technical reviews, risk mitigation, and adherence to delivery timelines
- Mentor engineers by establishing best practices and ensuring delivery of high-quality, maintainable, and production-ready code
Required competencies and certifications
- 10+ years of experience in data engineering, Big Data development, and enterprise data platforms
- Proven hands-on expertise in Big Data technologies with ability to guide teams technically and troubleshoot complex challenges
- Strong proficiency in Python, PySpark, SQL, and distributed data processing frameworks
- Experience building and managing complex data pipelines, ETL/ELT workflows, and large-scale data processing solutions
- Demonstrated leadership in managing data engineering teams, deliverables, and driving technical execution
- Strong understanding of data modelling, data architecture, performance optimization, and platform scalability
- Excellent stakeholder management, communication, problem-solving, and leadership skills
Preferred competencies and qualifications
- Experience with cloud-based data engineering and modern data platforms
- Background in banking, financial services, regulatory reporting, risk analytics, or other data-intensive enterprise environments