Our client is concerned with excessive charging attributed to the Databricks Lakehouse environment. Client would now like to implement some of the recommendations. These will be delivered in two workstreams:

Databricks Environment Changes - changes which can be made to Databricks environment setup including data deletion and control of instances Pipeline Rebuild POC - to understand the cost impact of some specific changes, rebuild one of the data pipelines to quantify the effect of these

changes and then extrapolate across other pipelines.

Responsibilities of contractor

• Leads the design, development, and optimization of the Silver Claims pipeline within the DAVE2 PoC environment.

• Evaluates and implements technical improvements such as SCD Type 2 modelling, removal of DLT components, caching, and code refactoring.

• Establishes monitoring and tagging to enable cost and performance comparison between baseline and optimized pipelines.

• Conducts parallel runs and validates output parity and performance consistency.

• Documents pipeline architecture, build standards, and optimization rationale for inclusion in deliverables.

• Provides technical guidance to other Data Engineers and ensures adherence to Nexus and AA data-engineering standards.

Requirements (experience, tech skills, etc.)

4+ years of experience Expert Spark/Databricks developer Deep understanding of data modelling, specifically the trade-offs between SCD Type 1 and Type 2 and DLT and traditional ELT pipelines on Delta Lake. Proven experience in performance tuning and refactoring complex data pipelines. Capable of benchmarking and clearly documenting performance/cost differences. Will lead

the technical implementation.

Show more
Show less