We are looking for a Data Engineer to design, develop and support enterprise-scale data engineering solutions. The role will focus on data pipelines, ETL processing, Big Data platforms, data migration, data quality and optimisation across banking and financial services environments.
Key Responsibilities
- Design, develop and optimise scalable data pipelines and ETL processes.
- Develop and maintain data processing solutions using Python, PySpark, SQL, Hive and Spark.
- Work with Big Data platforms including Hadoop HDFS, Hive and cloud/object storage environments.
- Perform data extraction, transformation and loading from multiple source systems.
- Design and implement data migration and modernisation solutions involving DataStage, Talend, Informatica and Hadoop.
- Develop and optimise batch processing and scheduling workflows using Autosys and Control-M.
- Perform data profiling, data quality checks and performance optimisation.
- Support data warehouse and dimensional data modelling activities.
- Implement CI/CD processes for data pipeline deployments.
- Work with business, application and infrastructure teams to deliver data solutions.
- Provide technical leadership and production support for enterprise data platforms.
Requirements
- Degree in Computer Science, Information Technology, Data Engineering or a related discipline.
- Strong experience in data engineering, ETL, data warehousing and Big Data technologies.
- Strong hands-on experience in Python, PySpark, SQL and Hive/HQL.
- Experience with Hadoop, Spark and enterprise data platforms.
- Experience with Talend, DataStage and/or Informatica.
- Experience with Teradata, Oracle or SQL Server.
- Experience with Unix/Linux and Shell scripting.
- Experience with CI/CD tools such as Git and Jenkins.
- Strong understanding of data modelling, data quality and data pipeline optimisation.
- Banking or financial services experience will be an advantage.
Preferred Skills
- Experience with Snowflake and Cloudera.
- Experience with Autosys, Control-M or similar scheduling tools.
- Experience in large-scale data migration and technology modernisation.
- Strong stakeholder management and technical leadership skills.