Lead Data Engineer, Databricks
Listed by Jobgether for an employer it does not name
- LocationUS
- Work typeRemote
- EmploymentFull-time
- Posted16 November 2026
- SourceListed by Jobgether (Lever)
Last checked1 h ago: still listed on Jobgether. First seen 1 h ago.
Moves fastWhen Jobgether takes a role down early, it is usually about 8 days after posting. Apply soon.
At a glance
- Experience7+ years
- Skills and toolsDatabricks, Spark, SQL, Python, AWS, Azure, Google Cloud, Git
Read from the ad by RoleFinder, with rules and an AI reading; the description is the source.
About the role
This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Lead Data Engineer, Databricks based in the United States.
The Lead Data Engineer will drive the delivery of production-grade data engineering solutions for enterprise customers, taking ownership of technical workstreams from design through production handover. This role combines hands-on development with technical leadership, building scalable batch and streaming pipelines using Databricks, PySpark, SQL, and Delta Lake. You will establish engineering standards, guide implementation decisions, and ensure solutions meet performance, quality, and cost objectives. Working directly with customers and cross-functional technical teams, you will translate complex requirements into reliable data platforms. You will also mentor engineers, collaborate with data architects, and communicate technical trade-offs to stakeholders. This is a fully remote opportunity for an experienced engineer who enjoys combining technical excellence, customer engagement, and team leadership.
Accountabilities:
The Lead Data Engineer will serve as the technical owner of engineering workstreams across customer engagements, ensuring that data solutions are designed, implemented, optimized, and delivered to production. Key responsibilities include:
-
Lead data engineering activities across customer engagements, from initial design and implementation through production handover.
-
Build, maintain, and optimize batch and streaming data pipelines using PySpark, SQL, and Delta Lake on Databricks.
-
Implement Medallion architecture across cloud environments and establish consistent data layer conventions and engineering patterns.
-
Take ownership of platform performance and cost efficiency by benchmarking Spark jobs, tuning SQL queries, right-sizing compute resources, and evaluating technical trade-offs.
-
Implement data quality and validation processes as code, including Lakeflow expectations and automated testing within CI pipelines.
-
Establish and maintain engineering standards, ensuring the reliability, scalability, and quality of delivered solutions.
-
Mentor junior and mid-level engineers through code reviews, collaborative development, technical guidance, and design feedback.
-
Partner with Data Architects to validate architectural decisions and identify potential design limitations during implementation.
-
Communicate technical decisions, delivery progress, and implementation trade-offs clearly to customers and non-technical stakeholders.
-
Work closely with customer teams and relevant platform specialists to ensure successful delivery of enterprise data solutions.
Requirements :
The ideal candidate is a highly experienced data engineer with a strong background in Databricks and cloud-based data platforms, combined with proven technical leadership and customer-facing communication skills. Required qualifications include:
-
At least 7 years of professional data engineering experience, including experience leading projects or engineering teams.
-
Strong proficiency in Python, SQL, Apache Spark, and Delta Lake.
-
Hands-on production experience building both batch and streaming data pipelines at enterprise scale.
-
Practical experience with at least one major cloud platform: AWS, Microsoft Azure, or Google Cloud Platform (GCP), including its data services.
-
Experience with data modeling, Git, and CI/CD practices for data engineering workloads.
-
Demonstrated ability to optimize Spark performance, improve pipeline efficiency, and manage platform costs.
-
Strong understanding of scalable data architecture and modern data engineering practices.
-
Excellent client communication skills, including the ability to explain complex technical concepts and trade-offs to non-technical stakeholders.
-
Proven ability to guide engineers, provide constructive feedback, and take ownership of technical delivery.
Nice to Have:
-
Experience in healthcare or life sciences environments.
-
Databricks Certified Data Engineer Associate or Professional certification.
-
Hands-on experience with Unity Catalog, Lakeflow, or Databricks Asset Bundles.
-
Familiarity with dbt, Airflow, or equivalent orchestration and data transformation tools.
-
Experience migrating legacy ETL systems to modern cloud-based data platforms.
-
Previous consulting or client-facing engineering experience.
Benefits :
-
Unlimited paid time off.
-
Medical, dental, and vision coverage.
-
Retirement savings plan, including 401(k) and IRA options.
-
Basic and voluntary life insurance.
-
Accidental Death and Dismemberment (AD&D) coverage.
-
Maternity and paternity leave.
-
Short-term and long-term disability coverage.
-
Training and professional development opportunities.
-
Dedicated certification roadmap for technical employees.
-
In-house training delivered by Databricks-approved trainers.
-
Dedicated Employee Success Manager focused on supporting the employee experience.
How Jobgether works:
We use an AI-powered matching process to ensure your application is reviewed quickly, objectively, and fairly against the role's core requirements. Our system identifies the top-fitting candidates, and this shortlist is then shared directly with the hiring company. The final decision and next steps (interviews, assessments) are managed by their internal team.
We appreciate your interest and wish you the best!
Why Apply Through Jobgether?
Data Privacy Notice: By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time.
#LI-CL1
Get open roles ranked against your CV
Add your CV once. RoleFinder ranks the open roles where you want to work, plus remote ones, against it, shows your best 12 with a one-line reason each, and emails you new matches once a week. The file is read once by AI and not kept; employers never see it.
Sign in with Google or email to start. How we handle your CV