Job Summary
Lead and manage data engineering teams to design and deliver scalable enterprise data platforms and pipelines, driving technical execution and collaboration with stakeholders to meet business requirements.
Responsibilities
Lead data engineering teams to design, develop, and deliver scalable enterprise data platforms and pipelines that meet business needs
Maintain hands-on expertise in Big Data technologies including Hadoop, Spark, PySpark, Hive, Kafka, and distributed data processing to solve complex engineering challenges
Design and implement robust ETL/ELT pipelines, data ingestion frameworks, data transformation, and integration solutions for large-scale data processing
Drive data engineering architecture, perform technical design, and optimize performance of data workloads to ensure scalability and efficiency
Manage implementations of data lakes, data warehouses, and modern cloud-based data platforms to support enterprise data needs
Utilize cloud platforms such as AWS, Azure, or GCP and technologies like Databricks to build and maintain data engineering solutions
Lead initiatives on data quality, data governance, metadata management, security, and data reliability to uphold enterprise standards
Collaborate with solution architects, data architects, business stakeholders, and project managers to translate business requirements into technical solutions
Manage project delivery including resource allocation, technical reviews, risk mitigation, and adherence to delivery timelines
Mentor engineers by establishing best practices and ensuring delivery of high-quality, maintainable, and production-ready code
Required competencies and certifications
10+ years of experience in data engineering, Big Data development, and enterprise data platforms
Proven hands-on expertise in Big Data technologies with ability to guide teams technically and troubleshoot complex challenges
Strong proficiency in Python, PySpark, SQL, and distributed data processing frameworks
Experience building and managing complex data pipelines, ETL/ELT workflows, and large-scale data processing solutions
Demonstrated leadership in managing data engineering teams, deliverables, and driving technical execution
Strong understanding of data modelling, data architecture, performance optimization, and platform scalability
Excellent stakeholder management, communication, problem-solving, and leadership skills
Preferred competencies and qualifications
Experience with cloud-based data engineering and modern data platforms
Background in banking, financial services, regulatory reporting, risk analytics, or other data-intensive enterprise environments