Lead Databricks Data Engineer

Job Description: Lead Databricks Data Engineer
Job Title: Lead Databricks Data Engineer
Experience: 10-12 Years
Employment Type: Full-Time

Job Summary
We are looking for a highly experienced Lead Databricks Data Engineer with 10-12 years of experience in Data Engineering, Big Data, and Cloud Data Platforms. The ideal candidate will lead the design, development, modernization, and optimization of enterprise-scale data platforms using Databricks Lakehouse architecture. This role requires strong technical leadership, solution architecture expertise, stakeholder management skills, and hands-on experience building scalable cloud-native data ecosystems.
The candidate will be responsible for defining data engineering standards, mentoring teams, driving modernization initiatives, and delivering high-performance data solutions that support analytics, AI/ML, and business intelligence initiatives.

Key Responsibilities Data Engineering & Architecture

- Design and implement enterprise-scale data platforms using Databricks Lakehouse architecture.

- Architect and develop scalable ETL/ELT data pipelines for batch and real-time processing.

- Define enterprise data models, ingestion frameworks, and transformation strategies.

- Lead the implementation of Delta Lake, Unity Catalog, and data governance frameworks.

- Build and optimize highly scalable Spark and PySpark solutions handling large-volume datasets.

- Drive cloud data platform modernization and migration initiatives.
Technical Leadership

- Serve as the technical lead for Databricks and Data Engineering programs.

- Establish engineering best practices, coding standards, and architecture guidelines.

- Conduct architecture reviews, code reviews, and solution assessments.

- Mentor and guide Data Engineers, Technical Leads, and junior team members.

- Collaborate with Enterprise Architects, Product Owners, Data Scientists, and Business Stakeholders.
Cloud & Platform Engineering

- Design cloud-native data solutions on Azure, AWS, or GCP.

- Implement secure, scalable, and resilient data architectures.

- Optimize costs, performance, reliability, and availability of data platforms.

- Lead CI/CD and Infrastructure-as-Code implementation for data workloads.

- Define and implement monitoring, observability, and operational excellence practices.
Governance & Data Quality

- Establish data governance, lineage, cataloging, and security controls.

- Ensure compliance with organizational and regulatory requirements.

- Implement data quality frameworks and monitoring solutions.

- Collaborate with security and compliance teams to enforce data protection standards.
Stakeholder & Delivery Management

- Partner with business leaders to understand strategic data requirements.

- Translate business requirements into scalable technical solutions.

- Drive delivery planning, risk management, and resource coordination.

- Present architecture decisions and technical recommendations to leadership teams.

Required Technical Skills Databricks

- Databricks Lakehouse Platform

- Databricks Workflows

- Delta Lake

- Unity Catalog

- Structured Streaming

- Databricks SQL

- Delta Live Tables (DLT)

- Databricks Asset Bundles
Big Data Technologies

- Apache Spark

- PySpark

- Spark SQL

- Kafka

- Real-Time Streaming Architectures

- Data Lake & Lakehouse Architectures
Cloud Platforms (Strong Experience in One or More) Azure

- Azure Databricks

- Azure Data Factory (ADF)

- ADLS Gen2

- Azure Synapse Analytics

- Azure Key Vault

- Event Hub
AWS

- S3

- EMR

- Glue

- Redshift

- Lambda

- IAM
GCP

- BigQuery

- Cloud Storage

- Dataflow

- Pub/Sub

- Composer
Programming

- Python

- SQL

- PySpark

- Scala (Preferred)

- Shell Scripting
DevOps & Automation

- Git

- Azure DevOps

- GitHub Actions

- Jenkins

- Terraform

- Docker

- Kubernetes
Data Warehousing

- Dimensional Modeling

- Star & Snowflake Schema

- Data Vault

- Master Data Management

- Data Governance Frameworks

Required Experience

- 12-15 years of overall IT experience.

- Minimum 6+ years of hands-on Databricks experience.

- 7+ years of experience in Spark/PySpark-based data engineering.

- Experience designing enterprise-scale Data Lake and Lakehouse solutions.

- Strong experience leading large data transformation and modernization programs.

- Experience handling structured, semi-structured, and unstructured data processing.

- Experience supporting analytics, AI/ML, and reporting ecosystems.

- Proven experience managing technical teams and driving architectural decisions.

Preferred Qualifications

- Bachelor's or Master's Degree in Computer Science, Engineering, Information Technology, or related field.

- Databricks Certified Data Engineer Professional.

- Azure Data Engineer Associate.

- AWS Data Analytics Specialty.

- Google Professional Data Engineer Certification.

- TOGAF or Cloud Architecture certifications are a plus.

Nice to Have

- MLOps and AI Engineering exposure.

- Snowflake and Databricks integration experience.

- Microsoft Fabric exposure.

- Data Mesh implementation experience.

- Data Governance tools such as Collibra, Alation, or Informatica.

- Expertise in Data Quality frameworks such as Great Expectations or Deequ.

Key Competencies

- Solution Architecture

- Technical Leadership

- Stakeholder Management

- Program Delivery

- Data Strategy & Governance

- Team Mentoring & Coaching

- Problem Solving & Decision Making

- Communication & Executive Presentation Skills

- Performance Optimization

- Innovation & Continuous Improvement
Technology Classification
Technology Tower: Data & AI
Practice Area: Data Engineering
Primary Skill: Databricks Lakehouse Engineering
Secondary Skills: Apache Spark, PySpark, Cloud Data Platforms (Azure/AWS/GCP), Data Architecture, Data Governance, MLOps.

Responsibilities
Management of DB Systems, Performance Tuning, Security, Compliance Standards,

Qualifications
Graduate with 4 years BS or Professional Degree. 15 - 18 Years