Senior Data Engineer

Data Engineer (Azure | Databricks | PySpark | SQL)

Contract | Remote | UK Public Sector Project

We are seeking a Data Engineer to support delivery of key data components within our evolving data ecosystem . This role will work closely with architects, analysts, engineers , and wider delivery teams to implement pipeline builds, data models, and integration patterns aligned to the organisation’s new Target Operating Model for Data .

You will contribute to the development of scalable data flows , optimise ingestion & transformation activities, and ensure solutions meet technical standards, performance expectations, and security controls . This is a hands-on engineering role ideal for someone who enjoys building, shaping, and improving modern data platforms .

Who We’re Looking For

A highly capable Data Engineer with hands-on experience building ETL pipelines , implementing Data Lake architectures , and working across Azure platform services . You will need strong problem-solving ability , a solid engineering mindset , and the ability to work collaboratively within a multi-disciplinary team .

If you’re confident writing production-grade code , applying design patterns , and building scalable, high-performing data products , this role offers technical autonomy, ownership, and challenge .

Key Responsibilities

- Build and maintain ETL pipelines and code in cloud data platforms.

- Support ingestion activity and onboarding of new data sources.

- Work with the Databricks Data Intelligence Platform and
develop
Data Engineering workloads.

- Construct raw, refined and curated data layers; catalogue assets appropriately.

- Validate solutions against functional and non-functional requirements.

- Deliver datasets , transformations and performance-optimised data products.

- Improve processes, engineering patterns, and reusable tooling.

- Monitor and measure pipeline performance; support incident resolution.

- Ensure documentation meets acceptance standards and is approved centrally.

- Actively engage in Agile ceremonies and governance forums.

Mandatory Requirements

- Strong experience with Python, PySpark & SQL for data engineering .

- Hands-on experience with Azure Databricks .

- Strong knowledge of Azure Data Factory & Fabric .

- Experience with Microsoft Purview .

- Familiarity with Event-driven data ingestion (Event Grid / Pub-Sub) .

- Understanding of SOLID principles , Async programming, Mediator/Factory patterns.

- Experience delivering unit + integration testing in Databricks .

- Knowledge of Secure ETL design with Entra IMID/SCIM integration.

- Understanding of Azure best practice, APIM, and platform governance .

- Ability to build and serve Power BI models via Databricks data sources .

Desirable

- Prior experience working within UK Public Sector environments .

Soft Skills

- Strong stakeholder communication and cross-team collaboration.

- Analytical and solution-focused mindset.

- Able to work independently, take ownership and drive progress.

- Commitment to clean, scalable, well-documented engineering.

- Adaptable, proactive, and comfortable working in dynamic delivery environments.

Contract Details

- Contract Type: Freelance / Contract

- Location: Remote (UK-based candidates preferred)

- Start Date: ASAP

- Clearance: Candidates must be eligible to work with UK Government departments / BPSS.