Databricks Change Data Feed for Project Cost Control: Tracking BOQ and Actual-Cost Changes

Ask a project controls manager what they need from the data and it is rarely the current cost alone. It is the movement: which BOQ lines moved since last week, by how much, committed versus actual; when a budget line was revised and to what; whether this period’s spend is tracking the forecast per line. Cost control lives in the deltas, not the snapshot.

The client and the QS ask the same shape of question in reviews. "The forecast went up — against which lines, and when did that happen?" A number that is simply higher than last month, with no record of how it got there, invites exactly the dispute a project controls function exists to prevent.

So the data problem is not storing the current cost — any system does that. It is retaining the changes over time in a way you can query, attribute and defend. That is where a plain nightly snapshot quietly fails.

The common pattern is to load the cost and BOQ tables into the lakehouse each night, overwriting yesterday’s with today’s. That gives a correct current state and throws away the one thing cost control needs: what changed between the two. Once yesterday’s version is overwritten, "what moved since last week" can only be reconstructed by comparing saved copies, if anyone thought to keep them.

Teams work around this by snapshotting daily into dated tables and diffing them, which balloons storage and still only captures change at snapshot granularity — a line revised and revised back between snapshots shows no change at all. It is a lot of machinery that approximates what the source database already knew: the actual sequence of changes.

What you want is the change itself, recorded natively, queryable between any two points — not a pile of daily copies you diff by hand. On a Databricks estate, Delta Change Data Feed provides exactly that.

Delta Change Data Feed is a Delta Lake feature that, once enabled on a table, records the row-level changes made to it — inserts, updates and deletes, with the before and after of each changed row and the version and timestamp at which it changed. You can then query the changes between any two versions or points in time, rather than just the current state.

This is the mechanism cost control needs. Enable CDF on the BOQ and actual-cost tables and the lakehouse retains not just today’s figures but the full sequence of how they changed — every revision to a BOQ line, every movement in committed or actual cost, with when and what it moved from and to. The change history becomes a first-class, queryable object rather than something reconstructed after the fact.

Because it reads the changes Delta already tracks, CDF does this without a pile of dated snapshot tables. One feed, queried by time range, answers "what changed between the last report and this one" directly — which is the question the whole function is built around.

In practice, CDF on the cost tables lets you build the views project controls actually report from. A cost-movement view shows, per BOQ line, the change in committed and actual cost between two dates — the period delta, sourced from the change feed rather than a manual diff. A revision log shows when each budget line was changed and to what, so a forecast increase is explained line by line rather than asserted as a total.

It also gives earned-value and variance reporting a firmer footing: because the history of every line is retained, you can show how the current cost-to-complete was arrived at, and trace any figure in the forecast back through its changes to the transactions that drove them. When the client questions a number in review, the answer is a path, not an assertion.

This pairs naturally with the measurement and BOQ discipline an EPC business already runs. The change feed does not replace the commercial process — it records it faithfully, so the data reflects the same sequence of revisions and measurements the QS worked through, available to query rather than locked in spreadsheets.

EPC cost control is a change-tracking problem wearing a reporting label. A nightly snapshot answers "what is the cost now" and loses "what changed, when, and against which line" — which is the question project controls, the QS and the client all actually ask. Delta Change Data Feed on the BOQ and actual-cost tables keeps that change history as a queryable record, so period movements, budget revisions and the path from forecast back to transaction are all available without a pile of dated snapshots.

The payoff is a cost function that can defend every movement rather than restate a total. "The forecast rose on these three lines, revised on these dates, driven by these commitments" is a statement that ends a review dispute; "it is higher than last month" starts one. We build Microsoft-first by default — on Fabric the equivalent is a change feed over the Delta tables in OneLake — and deliver this on Databricks with Change Data Feed where a client’s estate runs there. Either way, cost control stops being a snapshot and becomes a traceable record of how the project reached today’s number.