We are currently building a cutting-edge, large-scale environmental data platform. We are looking for a talented Senior Data Engineer to help us drive system resiliency, optimize CI/CD pipelines, and implement robust Site Reliability Engineering (SRE) practices across our cross-agency initiatives.
What You Will Do
As a Senior Data Engineer, you will bridge the gap between complex business requirements and highly resilient technical execution. Your core responsibilities will fall into three key pillars:
1. Data Pipeline Development & Architecture
- Architect & Build: Design and build robust batch and real-time ingestion pipelines to collect, clean, merge, and harmonise diverse data streams from various source systems.
- Translate Requirements: Partner with business users to translate their functional needs into scalable technical specifications, building reusable data models for end-users.
- Innovate: Research and propose cutting-edge technologies and processes to continuously upgrade our core data infrastructure.
- Secure Access: Design secure access mechanisms to ensure end-users and integrated systems can safely retrieve data assets.
2. Data Quality & Automated Testing
- Enforce Integrity: Implement end-to-end data quality checks, validation, and reconciliation processes across all pipeline stages.
- Automate Testing: Create and maintain automated test suites (covering functional, regression, and exploratory testing) integrated with CI/CD pipelines to catch bugs before they hit production.
- Support UAT: Coordinate with stakeholders to facilitate smooth User Acceptance Testing (UAT), ensuring data products fully meet business requirements.
3. Operations, Governance & SRE
- Optimize Operations: Conduct day-to-day monitoring, performance tuning, capacity planning, and issue diagnosis of databases and ETL systems.
- Data Governance: Collaborate with data stewards to maintain data catalogues, lineage, metadata, and enforce strict security measures (encryption, masking, access control).
- Drive DevOps: Champion CI/CD and Site Reliability Engineering (SRE) practices to ensure maximum system uptime and delivery efficiency.
What We Are Looking ForQualifications & Technical Skills
- Education: Bachelor’s Degree in Computer Science, Software Engineering, Information Technology, or a related discipline.
- Core Fundamentals: Deep understanding of system design, data structures, algorithms, data access, and data modeling.
- Programming & SQL: Highly proficient in Python, Shell Scripting, and SQL.
- Data Tools & Cloud:
- Experience with Databricks and orchestration frameworks (e.g., Airflow, Azure Data Factory).
- Hands-on experience with major cloud providers (AWS, Azure, or GCP).
- DevOps & QA:
- Familiarity with DevOps tools such as Docker, Git, and Terraform.
- Experience with testing automation, CI/CD integration, and defect tracking tools (like JIRA or equivalent).
Added Advantages (Nice-to-Haves)
- Experience working within public sector frameworks and government systems relating to data security/governance policies.
- Domain knowledge or prior experience working with Climate and Weather domains.
Pay: $8,000.00 - $9,000.00 per month
Work Location: In person