Case study

Off Scala: Rebuilding an Insurer's Data Platform on dbt and Snowflake

When I joined a UK specialty insurer, the data platform ran mostly on Scala, built and maintained by about a dozen engineers, possibly more. My own remit was the last mile: final SQL and reporting on tables the Scala engineers produced and handed over. I did not own the pipeline. A dbt-based approach had no mandate behind it.

The migration · Scala to dbt, Snowflake and Prefect

1. 01

   The starting position

   The platform ran mostly on Scala. From the last mile I could see the shape of the data, and it was nowhere near big data.

   Decision — Judge the tooling against the size of the actual problem, not against the platform already built around it.

2. 02

   Making the case for dbt

   I argued for a dbt-based approach from the layer I did own. It had no mandate, and making the case was slow and painful.

   Decision — Delivery time was the part of the argument that did not need an opinion about programming languages.

3. 03

   Executive buy-in

   Over time the approach gained attention, and then executive buy-in.

   Decision — Sponsorship was the constraint, not the argument. A live platform is not replaced until the direction is agreed above the team doing the work.

4. 04

   Ingestion

   PrefectAWSSnowflake

   Ingestion moved off Scala onto Prefect, landing data into Snowflake.

   Decision — Let the warehouse and the cloud provider do what they already do, and keep hand-written code for the parts that are genuinely specific.

5. 05

   Curated and conformed layers

   dbtSnowflakeSQL

   Both layers were rebuilt as dbt models. I was one of the leading engineers on the migration, working in the SSoT team that surfaces them.

   Decision — Express transformation as SQL against the warehouse, so changing a definition is a change to one model rather than a change to a job.

6. 06

   Nothing left on the old stack

   SnowflakeAWSPrefectdbt

   Scala is used nowhere in the platform now. Every layer is either Snowflake and AWS native, or Prefect and dbt.

   Decision — Leaving any component on Scala would have kept the old delivery speed for everything that touched it.

What shipped

• Ingestion moved off Scala and onto Prefect, landing into Snowflake
• The curated and conformed layers rebuilt as dbt models, surfaced by the SSoT team
• Every layer either Snowflake and AWS native, or Prefect and dbt
• Scala removed from the platform entirely, with no component left on the old stack
• A change or a new table delivered in hours, where the same work took weeks

Stack

The engineering was the straightforward half. Getting a mandate for it was the slow part.