Case study

Off Scala: An Insurer’s Data Platform on dbt and Snowflake

When I joined a UK specialty insurer, the data platform ran mostly on Scala, built and maintained by a team of around a dozen engineers. My own remit was the last mile: final SQL and reporting on tables the Scala engineers produced and handed over. I did not own the pipeline.

The migration · Scala to dbt, Snowflake and Fivetran

1. 01

   The starting position

   From the last mile I could see the volumes we were working with, and they sat comfortably inside what a warehouse handles directly.

   Decision · The volumes did not justify a distributed processing platform, so the tooling question was really a delivery-speed question.

2. 02

   Making the case for dbt

   I argued for a dbt-based approach from the layer I did own. Making the case was slow and painful.

   Decision · Delivery time was the part of the argument that did not need an opinion about programming languages.

3. 03

   Executive buy-in

   Over time the approach gained attention, and then executive buy-in.

   Decision · Sponsorship was the constraint, not the argument. A live platform is not replaced until the direction is agreed above the team doing the work.

4. 04

   Ingestion

   FivetranAWSSnowflake

   Ingestion moved off Scala onto Fivetran, landing data into Snowflake.

   Decision · Let the warehouse and the cloud provider do what they already do, and keep hand-written code for the parts that are genuinely specific.

5. 05

   Curated and conformed layers

   dbtSnowflakeSQL

   Both layers were rebuilt as dbt models. I was one of the engineers driving the migration, in the team that owns the single source of truth.

   Decision · Express transformation as SQL against the warehouse, so changing a definition is a change to one model rather than a change to a job.

6. 06

   Nothing left on the old stack

   SnowflakeAWSFivetrandbt

   Scala is used nowhere in the platform now. Ingestion, curated and conformed all moved across.

   Decision · Leaving any component on Scala would have kept the old delivery speed for everything that touched it.

What shipped

• Ingestion moved off Scala and onto Fivetran, landing into Snowflake
• The curated and conformed layers rebuilt as dbt models, owned by the single-source-of-truth team
• A change or a new table delivered in hours, where the same work took weeks

Stack

The engineering was the straightforward half. Getting a mandate for it was the slow part.