Full Time Data Engineer Jobs, in Job Expert Selangor - Maukerja

About the Role

We are looking for an experienced Data Engineer with strong expertise in Microsoft Azure, Azure Databricks, Python, Spark, and Azure DevOps. The successful candidate will work on cloud-based data engineering solutions, data processing, orchestration, CI/CD, and Azure data platform integration.

Key Responsibilities

• Design and develop scalable data engineering solutions using Microsoft Azure.
• Develop data processing solutions using Python and Spark within Azure Databricks.
• Manage Databricks workflows and orchestration pipelines.
• Work with Azure Data Lake Storage (ADLS) and mount-point integration with Azure Databricks.
• Manage and work with Hive Metastore.
• Work with Azure Key Vault for secret and key management.
• Integrate Azure Key Vault with Databricks and Azure DevOps service connections.
• Develop and manage CI/CD pipelines using Azure DevOps and GitHub Actions.
• Work with SQL and other programming/scripting languages for data engineering activities.
• Support monitoring and logging through Azure Log Analytics Workspace.
• Work with APIs, Databricks CLI, and AZ CLI.
• Collaborate with development teams using GitHub/Azure Repos and Agile/Scrum methodologies.

Requirements

• 7 years of relevant professional experience in Data Engineering.
• Azure DevOps – 7 years.
• Azure Databricks – 7 years.
• Python – 7 years.
• Strong experience with:
• Microsoft Azure
• Microsoft Entra ID / Azure Active Directory
• Azure Data Lake Storage
• Azure Databricks
• Azure Key Vault
• Python
• Spark
• SQL
• GitHub
• Azure DevOps
• Strong hands-on experience developing data processing solutions using Python and Spark.
• Experience managing Databricks workflows as orchestration pipelines.
• Understanding of pull requests, branching models, and policies.

Work Location: In person