Kerja Sepenuh Masa Data Engineer, Gaji tinggi MYR 7,200 di MyValiant Selangor - Maukerja

Responsibilities

Build and manage the data asset using some of the most scalable and resilient open source big data technologies like Airflow, Spark, Apache Atlas, Kafka, Yarn, HDFS, ElasticSearch, Presto/Dremio, HDP, Visualisation layer,Snowflake and more.

Design and deliver the next-gen data lifecycle management suite of tools/frameworks, including ingestion and consumption on the top of the data lake to support real-time, API-based and serverless use-cases, along with batch (mini/micro) as relevant

Build and expose metadata catalogue for the Data Lake for easy exploration, profiling as well as lineage requirements

Enable Data Science teams to test and productionize various ML models, including propensity, risk and fraud models to better understand, serve and protect our customers

Lead technical discussions across the organisation through collaboration, including running RFC and architecture review sessions, tech talks on new technologies as well as retrospectives

Apply core software engineering and design concepts in creating operational as well as strategic technical roadmaps for business problems that are vague/not fully understood

Obsess security by ensuring all the components, from a platform, frameworks to the applications are fully secure and are compliant by the group’s infosec policies.

Requirements

At least 1-2+ years of relevant experience in developing scalable, secured, fault tolerant, resilient & mission-critical Big Data platforms.

Able to maintain and monitor the ecosystem with 99.9999% availability

Candidates will be aligned appropriately within the organisation depending on experience and depth of knowledge.

Must have sound understanding for all Big Data components & Administration Fundamentals. Hands-on in building a complete data platform using various open source technologies.

Must have good fundamental hands-on knowledge of Linux and building a big data stack on top of AWS/Azure using Kubernetes.

Strong understanding of big data and related technologies like HDFS, Spark, Presto, Airflow, apache atlas etc.

Good knowledge of Complex Event Processing (CEP) systems like Spark Streaming, Kafka, Apache Flink, Beam etc.

Experience with NoSQL databases – KV/Document/Graph and similar

Proven Ability to contribute to the open source community and up-to-date with the latest trends in the Big Data Space.

Able to drive devops best practices like CI/CD, containerization, blue-green deployments, 12-factor apps, secrets management etc in the Data ecosystem.

Able to develop an agile platform with auto scale capability up & down as well vertically and horizontally.

Must be in a position to create a monitoring ecosystem for all the components in use in the data ecosystem.

Proficiency in at least one of the programming languages Java, Scala, Python or Go along with a fair understanding of runtime complexities.

Pay: RM5,000.00 - RM7,200.00 per month

Benefits:

• Health insurance
• Opportunities for promotion
• Professional development

Work Location: In person