Full Time Data Engineer Jobs, in Inter Island Group Selangor - Maukerja

DATA ENGINEER

Industry: Premium Automotive
Employment Type: Contract
Salary: RM13,500
Experience: 7 years

Role Overview

We are seeking an experienced Data Engineer with strong expertise in Microsoft Azure, Azure Databricks, Python, and Spark. The ideal candidate will have hands-on experience in cloud-native data development, data processing, API development, CI/CD, and modern data platform technologies.

Key Responsibilities

• Develop cloud-native data solutions using Microsoft Azure.
• Work with Microsoft Entra ID / Azure Active Directory and its integration with data platforms.
• Develop and manage data processing solutions using Azure Databricks.
• Work with Azure Data Lake Storage and its integration with Azure Databricks.
• Develop data processing solutions using Python and Spark.
• Manage Databricks workflows and orchestration pipelines.
• Work with Hive Metastore and Databricks compute clusters.
• Apply data processing techniques using Pandas and Polars.
• Manage secrets and keys using Azure Key Vault.
• Develop and maintain deployment pipelines using Azure DevOps and GitHub Actions.
• Develop APIs using FastAPI and related technologies.
• Work with Docker and Azure Container Registry.
• Implement and support asynchronous messaging and distributed caching using NATS and Redis.
• Work with SQL, Kusto (KQL), PowerShell, and Bash.
• Apply version control and collaboration practices using GitHub and Azure Repos.
• Support monitoring and logging using Azure Log Analytics Workspace.
• Work within a Scrum environment.

Requirements

• 7 years of relevant experience.
• Strong experience with Microsoft Azure.
• Strong experience with Azure Databricks.
• Strong Python and Spark development skills.
• Experience with Azure Data Lake Storage.
• Experience with Microsoft Entra ID / Azure Active Directory.
• Experience with Azure Key Vault.
• Strong SQL skills.
• Experience with GitHub and Azure Repos.
• Experience with Azure DevOps Pipeline / GitHub Actions and YAML-based deployment pipelines.
• Experience with FastAPI and REST API development.
• Experience with Docker and Azure Container Registry.
• Experience with NATS and Redis.
• Experience with Azure Log Analytics Workspace.
• Experience working within a Scrum framework.

Good to Have:

• Unity Catalog
• Medallion Architecture
• Pandas and Polars
• SAFE framework
• Databricks CLI
• AZ CLI
• PowerShell
• Bash

Experience:

• Azure DevOps: 7 years (Required)
• Azure Databricks: 7 years (Required)
• Phyton: 7 years (Required)

Work Location: In person