Incremental Models - Meteoric Impact

Core Argument

People think incremental means small. Taking tiny steps. This is not one of those stories. Sometimes incremental changes can open new opportunities.

The Mechanism

Incremental models only do work on the latest records. Instead of having to drop and reload your entire table every time, it only works with a small set of the data — a “micro-batch.”

Since your models are not doing as much work, you can run them more frequently without increasing your storage or compute costs. If you run them more frequently, your data can be more up to date. If your data is more fresh, then new decisions can be made on it.

Why It Matters

This is a chain reaction: incremental processing leads to lower compute costs, which enables higher frequency runs, which delivers fresher data, which unlocks new use cases. The “meteoric impact” comes not from the technique itself but from the cascade of possibilities it opens up.

This connects directly to the dbt transformation layer, where incremental materialization is a core pattern. The founder’s framing here — that a seemingly small technical choice can have outsized business impact — is a recurring theme in the Ray Data Co analytics-engineering philosophy.