Role Overview
We are looking for an experienced Senior Databricks Consultant / Architect to lead the design, architecture, development, and implementation of enterprise-scale data engineering and analytics solutions using Azure Databricks. The ideal candidate should have strong expertise in Lakehouse architecture, PySpark, Delta Lake, cloud data platforms, and data governance.
Key Responsibilities
- Design and architect scalable, secure, and high-performance Lakehouse solutions using Databricks
- Lead end-to-end data engineering and data platform implementation
- Define enterprise data architecture, standards, best practices, and technical roadmaps
- Design and implement data pipelines using PySpark, Spark SQL, Delta Lake, and Databricks Workflows
- Develop and optimize ETL/ELT pipelines for batch and streaming data processing
- Implement Medallion Architecture (Bronze, Silver, Gold) and Delta Live Tables (DLT)
- Work extensively with Unity Catalog for data governance, security, access control, and lineage
- Integrate Databricks with Azure services such as ADLS Gen2, Azure Data Factory, Event Hubs, and Azure Key Vault
- Optimize Spark workloads, cluster configurations, SQL queries, and overall platform performance
- Implement CI/CD pipelines, automated deployments, and DevOps practices
- Provide technical leadership, conduct design reviews, and mentor data engineering teams
- Collaborate with business stakeholders, architects, and engineering teams to translate business requirements into technical solutions
Mandatory Technical Skills
- Strong hands-on experience with Azure Databricks and Apache Spark
- Expertise in PySpark, Spark SQL, Python, and SQL
- Strong knowledge of Delta Lake and Lakehouse architecture
- Experience implementing Medallion Architecture
- Expertise in Unity Catalog, Delta Live Tables (DLT), and Auto Loader
- Strong understanding of ETL/ELT design and data pipeline development
- Experience with Azure Data Lake Storage (ADLS Gen2) and Azure Data Factory
- Knowledge of Databricks Workflows, cluster management, and performance tuning
- Experience with data security, governance, and access management
- Hands-on experience with Git, Azure DevOps, and CI/CD
Good to Have
- Experience with AWS Databricks or GCP Databricks
- Knowledge of Kafka, Event Hubs, and real-time data streaming
- Experience with Databricks SQL and Lakehouse Monitoring
- Knowledge of Terraform and Infrastructure as Code
- Experience with MLflow and the Databricks Machine Learning ecosystem
- Databricks Professional-level certifications
Candidate Profile
- Strong experience in enterprise data architecture and large-scale data engineering implementations
- Ability to independently drive architecture decisions and technical delivery
- Excellent communication, stakeholder management, and problem-solving skills
- Ability to work with cross-functional teams and provide technical direction