- Outram Central Region (Singapore) Singapore
Kerja Sepenuh Masa Senior Data Engineer – AWS - Databricks - PySpark, Gaji tinggi SGD 12,000 di ASTEK SINGAPORE INNOVATION TECHNOLOGY PTE. LTD. Central Region (Singapore) - Maukerja
Senior Data Engineer – AWS - Databricks - PySpark
ASTEK SINGAPORE INNOVATION TECHNOLOGY PTE. LTD.
Kongsi
Simpan
Lokasi Kerja
Penerangan Kerja
Tanggungjawab
Role Overview
We are seeking an experienced
Senior Data Engineer
to design, develop and maintain scalable data pipelines and reusable data products on an AWS and Databricks-based data platform. The role involves technical leadership in data engineering, lakehouse architecture, data quality, governance and performance optimisation to support analytics, reporting and AI/ML workloads.
Key Responsibilities
Design, develop and optimise scalable batch and streaming data pipelines using
AWS, Databricks, Python, PySpark and SQL
.
Build and maintain lakehouse architectures using
Delta Lake
, including Bronze, Silver and Gold data layers, curated datasets and reusable data products.
Implement data ingestion from databases, APIs, files and event streams, including incremental loading, Change Data Capture (CDC), backfills and recovery.
Establish automated data quality checks, schema validation, reconciliation, metadata management and data lineage.
Develop and automate data workflows using
Databricks Workflows, Lakeflow and AWS services
, incorporating CI/CD, automated testing and infrastructure as code.
Implement data governance, access controls and security measures using
Unity Catalog, AWS Glue Data Catalog, Lake Formation and IAM
.
Monitor pipeline performance, troubleshoot production issues and optimise Spark/SQL workloads for scalability, reliability and cost efficiency.
Lead technical design and code reviews, establish engineering standards and mentor junior engineers.
Collaborate with data scientists, analysts, architects and business stakeholders to deliver reliable data solutions.
Maintain technical documentation, data models, operational procedures and knowledge transfer materials.
Requirements
Degree in Computer Science, Software Engineering, Data Engineering, Information Systems or a related discipline.
6–10 years of data/software engineering experience
, including at least 2 years working with cloud-based data platforms.
Strong hands-on experience with
Python, PySpark, SQL, Apache Spark, Databricks and Delta Lake
.
Proficiency in AWS data services such as
S3, Glue, Lake Formation, Lambda, Step Functions, EventBridge, CloudWatch, Redshift, MSK/Kinesis, IAM and KMS
.
Experience designing data lake/lakehouse architectures, data models and production-grade batch or streaming pipelines.
Strong understanding of
ETL/ELT, CDC, incremental ingestion, data quality, schema validation, metadata and lineage
.
Experience with workflow orchestration, Git, CI/CD, automated testing and infrastructure as code, preferably
Terraform
.
Knowledge of data pipeline monitoring, observability, troubleshooting and performance optimisation.
Familiarity with cloud security, encryption, access management and data governance.
Proven ability to provide technical guidance, conduct code reviews and mentor engineers.
Good to Have
Experience with
Databricks Unity Catalog, Lakeflow, Delta Live Tables, Auto Loader, Structured Streaming, Delta Sharing and Databricks Asset Bundles
.
Familiarity with
Apache Airflow, dbt, Kafka, AWS MSK/Kinesis and AWS DMS
.
Knowledge of data contracts, data products, semantic layers, master/reference data and data mesh concepts.
Experience supporting ML/AI data pipelines, feature engineering, feature stores or MLOps.
Exposure to Singapore public-sector environments, including
IM8, VAPT, data classification and cross-agency data governance
.
Astek has been providing IT and engineering solutions for some of the world’s largest industrial and services groups for over 25 years.
We support our clients with the creation and development of their new products in four main areas of expertise:
Some of our clients include BNP Paribas, CA-CIB, Societe Generale, Amadeus, Air France, Alstom, Airbus, Thales, Safran, Faurecia, Valeo, Renault, Jaguar, PSA, BMW, Daimler, Schneider Electric, Areva, ENGIE, Veolia, Total…
Our team is dedicated to offering our global clients project support while guaranteeing a consistent level of service alongside a commitment to excellence and strong core values.
We’re always looking for smart, talented, driven, down-to-earth, fun-to-work-with people who want to make a positive and meaningful impact.
To join our community!
*************
We are seeking an experienced
Senior Data Engineer
to design, develop and maintain scalable data pipelines and reusable data products on an AWS and Databricks-based data platform. The role involves technical leadership in data engineering, lakehouse architecture, data quality, governance and performance optimisation to support analytics, reporting and AI/ML workloads.
Key Responsibilities
Design, develop and optimise scalable batch and streaming data pipelines using
AWS, Databricks, Python, PySpark and SQL
.
Build and maintain lakehouse architectures using
Delta Lake
, including Bronze, Silver and Gold data layers, curated datasets and reusable data products.
Implement data ingestion from databases, APIs, files and event streams, including incremental loading, Change Data Capture (CDC), backfills and recovery.
Establish automated data quality checks, schema validation, reconciliation, metadata management and data lineage.
Develop and automate data workflows using
Databricks Workflows, Lakeflow and AWS services
, incorporating CI/CD, automated testing and infrastructure as code.
Implement data governance, access controls and security measures using
Unity Catalog, AWS Glue Data Catalog, Lake Formation and IAM
.
Monitor pipeline performance, troubleshoot production issues and optimise Spark/SQL workloads for scalability, reliability and cost efficiency.
Lead technical design and code reviews, establish engineering standards and mentor junior engineers.
Collaborate with data scientists, analysts, architects and business stakeholders to deliver reliable data solutions.
Maintain technical documentation, data models, operational procedures and knowledge transfer materials.
Requirements
Degree in Computer Science, Software Engineering, Data Engineering, Information Systems or a related discipline.
6–10 years of data/software engineering experience
, including at least 2 years working with cloud-based data platforms.
Strong hands-on experience with
Python, PySpark, SQL, Apache Spark, Databricks and Delta Lake
.
Proficiency in AWS data services such as
S3, Glue, Lake Formation, Lambda, Step Functions, EventBridge, CloudWatch, Redshift, MSK/Kinesis, IAM and KMS
.
Experience designing data lake/lakehouse architectures, data models and production-grade batch or streaming pipelines.
Strong understanding of
ETL/ELT, CDC, incremental ingestion, data quality, schema validation, metadata and lineage
.
Experience with workflow orchestration, Git, CI/CD, automated testing and infrastructure as code, preferably
Terraform
.
Knowledge of data pipeline monitoring, observability, troubleshooting and performance optimisation.
Familiarity with cloud security, encryption, access management and data governance.
Proven ability to provide technical guidance, conduct code reviews and mentor engineers.
Good to Have
Experience with
Databricks Unity Catalog, Lakeflow, Delta Live Tables, Auto Loader, Structured Streaming, Delta Sharing and Databricks Asset Bundles
.
Familiarity with
Apache Airflow, dbt, Kafka, AWS MSK/Kinesis and AWS DMS
.
Knowledge of data contracts, data products, semantic layers, master/reference data and data mesh concepts.
Experience supporting ML/AI data pipelines, feature engineering, feature stores or MLOps.
Exposure to Singapore public-sector environments, including
IM8, VAPT, data classification and cross-agency data governance
.
Astek has been providing IT and engineering solutions for some of the world’s largest industrial and services groups for over 25 years.
We support our clients with the creation and development of their new products in four main areas of expertise:
- Information technology and systems (IT & IS);
- Industrial engineering and product development;
- Internet of Things (IoT);
- Digital, web and mobile solutions.
Some of our clients include BNP Paribas, CA-CIB, Societe Generale, Amadeus, Air France, Alstom, Airbus, Thales, Safran, Faurecia, Valeo, Renault, Jaguar, PSA, BMW, Daimler, Schneider Electric, Areva, ENGIE, Veolia, Total…
Our team is dedicated to offering our global clients project support while guaranteeing a consistent level of service alongside a commitment to excellence and strong core values.
We’re always looking for smart, talented, driven, down-to-earth, fun-to-work-with people who want to make a positive and meaningful impact.
To join our community!
*************
Peringatan Penting
Jangan pernah kongsikan maklumat bank atau kad kredit anda semasa memohon pekerjaan. Elakkan membuat sebarang pembayaran atau mengisi survey yang tidak berkaitan. Jika ada yang mencurigakan, sila laporkan iklan pekerjaan ini segera.
Lebih Lanjut