Conceptualize and own the data architecture for multiple large-scale projects, while evaluating design and operational cost-benefit tradeoffs within systems
Leads the architecture, design, and implementation of ETL processes, workflows, and data pipelines for the ingestion, transformation, and integration of data from multiple source agencies.
Leads the architecture, design, and implementation of ETL processes, workflows, and data pipelines for the ingestion, transformation, and integration of data from multiple source agencies.
Conceptualize and own the data architecture for multiple large-scale projects, while evaluating design and operational cost-benefit tradeoffs within systems
Engineer the People Data Middle Layer: Utilise Microsoft Fabric Lakehouse to consolidate various HR data sources. Drive heavy data transformations and build complex business logic primarily within Microsoft Fabric's scheduled pipelines and notebooks, leveraging PySpark/Python for efficient data ingestion, transformation, and distribution.
Design, build and enhance enterprise data platforms, including Data Lakes, Data Warehouses, data pipelines and analytics solutions that power business decisions across the organisation.
We are seeking an experienced Databricks Architect / Lead Data Engineer to lead the design, development, and delivery of enterprise-scale data platforms on Databricks and AWS.
Our data ecosystem will be migrating their products from Azure to AWS. Data Engineers needed to assist with new and current pipelines. All IT equipment are provided and it is the responsibilities of consultants to ensure the security of the organization's data and applications.
Architect end-to-end solution for business analytics product (dashboards or statistical model) from the acquiring of data, contextualizing data for business analytics and integrating of product with business process
Build and operate the Marketplace data warehouse end to end, covering batch and real-time ingestion, semantic layer design, and application-layer modeling for downstream consumers.