The Data Warehouse Diagram Everyone Still Draws — And Why It’s Already Late

I used to draw this exact diagram on whiteboards. Data sources on the left — the ERP, the SQL databases, the IoT feeds, the spreadsheets nobody will admit to. An arrow into a staging area. ETL. A central warehouse in the middle. Data marts hanging off it. APIs and SQL feeding the BI tools on the right. Clean. Layered. Defensible.

For about fifteen years, it was the right answer. Storage was expensive, so you copied data sparingly and modelled it carefully. Compute was scarce, so you ran heavy transformations overnight when nobody was looking. The batch window wasn’t a compromise — it was the design.

The diagram hasn’t changed. The constraints that justified it have.

And that gap is why, in most industrial businesses I walk into, the dashboards are accurate and useless at the same time. Accurate, because the ETL is well built. Useless, because by the time a number reaches the person who could act on it, the moment has passed.

This is not a story about a tool being out of date. It’s about an architecture quietly imposing a tax that operations leaders have stopped noticing.

For years I defended this design, and I’d defend most of it again under the old constraints.

Staging exists for a good reason: you don’t want raw, inconsistent source data landing directly in the place your CFO runs the quarterly numbers from. ETL exists because source systems lie to each other — the same customer has three IDs across SAP, Dynamics and a legacy WMS, and someone has to reconcile that. The warehouse exists to give you one governed version. Data marts exist so the finance team isn’t querying across the entire estate to get a margin report.

Every box on that diagram solves a real problem. I’m not going to pretend otherwise, and I’m suspicious of anyone who tells you the whole model is obsolete. It isn’t.

What changed is the cost of the seams between the boxes.

Look at the arrows, not the boxes. Each arrow is a hand-off, and each hand-off is a delay.

Source to staging: a scheduled extract. Staging to warehouse: an ETL load. Warehouse to data mart: a refresh. Data mart to BI: a cache or an import. Four movements, each on its own schedule, each waiting for the one before it to finish.

In the classic build, those schedules are nightly. So the number an Operations Director reads at 8am is, at best, a snapshot of the plant at midnight. The night shift’s downtime, the 2am bearing failure, the line that ran 40 minutes behind on changeover — none of it is in the report. The report describes a factory that no longer exists.

The instinct, every time, is to attack the ETL. Tune the jobs. Tighten the window. Move from nightly to hourly. It works, briefly. You buy minutes. But you’re optimising the slowest part of a chain that has four slow parts, and the latency is structural — it lives in the seams between the layers, not inside any one of them.

I’ve watched teams spend a quarter shaving an ETL window from six hours to ninety minutes and still miss every decision that mattered, because the decision needed data that was minutes old, not ninety minutes old. That’s not a tuning problem. That’s the diagram.

Two things collapsed the constraints that made all this copying necessary.

The first is cheap, separated storage and compute. When storage costs almost nothing and compute scales on demand, you no longer need to ration copies of your data. The whole logic of “extract a little, transform carefully, load once” was a response to scarcity that no longer exists.

The second is the open table format — Delta Lake, sitting on Delta Parquet files. This is the quiet shift. It lets you keep raw data and governed, modelled data in the same place, with transactions, schema enforcement and time travel, without shunting it between a staging database, a warehouse engine and a mart. The governance the warehouse used to provide doesn’t disappear — it moves into the lake.

Put those together and the architecture inverts. Instead of copy → transform → copy → serve, you land once and model in place.

In the Microsoft estate, that’s Microsoft Fabric. Raw data lands in OneLake — a single logical store for the whole organisation, not a copy per tool. You model it in place with Delta Lake. And then the part that genuinely matters for the latency problem: Power BI reads from the lake over Direct Lake mode. No import. No separate mart to refresh. No cache to wait on. The report queries the modelled data directly.

The same idea exists outside the Microsoft world — Databricks and Snowflake solve the same collapse a different way, and if your estate already lives there, that’s the lower-friction path. The point isn’t the brand. The point is that the staging-warehouse-mart relay race was a workaround for constraints that have lifted.

Take a typical mid-market manufacturer. SAP S/4HANA for finance and production. Microsoft Dynamics 365 for sales and service. A WMS in each of three warehouses. SCADA on the plant floor, throwing sensor and OEE data. And, inevitably, a planner running the real demand forecast in a spreadsheet she doesn’t trust but can’t replace.

Under the classic diagram, getting an OEE-versus-plan view means: extract from SAP overnight, extract from SCADA on its own schedule, stage both, run an ETL job to align the work orders to the machine data, load the warehouse, refresh a manufacturing mart, then import it into Power BI. Six steps, three schedules, and a report that lands the next morning describing yesterday.

Land-once looks different. SAP work orders and SCADA telemetry land in OneLake. They’re modelled in place — work orders joined to machine runtime in Delta Lake, OEE calculated as a governed measure. Power BI serves it over Direct Lake. The shift supervisor sees OEE against plan while the shift is still running, not at the next morning’s standup.

And then the layer the original diagram never had: action. With the data already live in the lake, Power Platform — Power Automate and Power Apps — wires the insight to a response. A line dropping below an OEE threshold raises an alert and logs a maintenance task automatically. The number doesn’t sit in a report waiting to be noticed. It triggers something.

Unify the data. Predict with AI. Act with automation. In that order. The classic diagram stops at the first step and calls it finished.

I promised honesty, so here’s where the new model bites — because it does, and anyone selling you a frictionless version is selling you something.

Direct Lake is not magic. It has fallback behaviour: push it past its limits — very high cardinality, certain query shapes, models that outgrow the guardrails — and it quietly reverts to DirectQuery, and your fast report gets slow without telling you why. You have to design for it, not assume it.

Landing raw data once does not mean modelling it once. The reconciliation work — the three customer IDs, the unit-of-measure mismatches, the master data nobody owns — does not disappear because you changed the storage layer. It moves. If your master data is a mess, a lakehouse gives you a faster way to query a mess. The governance problem is still a governance problem wearing a new architecture.

Real-time isn’t free, and most operations don’t need it everywhere. Streaming SCADA data into the lake continuously costs more than a nightly batch, in money and in engineering. The skill is deciding which decisions need data that’s minutes old and which are genuinely fine on yesterday’s batch. Plenty are. Pretending everything needs to be live is how you burn budget.

And if you’re pre-ERP, or your data foundation is genuinely broken, none of this is your first fight. Collapsing the warehouse layers assumes you have something coherent to collapse.

If you run operations or sit over the people who do, the question isn’t “should we modernise the warehouse.” That’s an IT framing and it leads to an eighteen-month programme.

The question is narrower and more useful: which decision is being made late because the number arrives late? The changeover that’s always reactive. The stockout you see after it’s happened. The maintenance call that comes after the failure, not before. Pick the one that costs the most, and measure the lag between the event on the floor and the moment a trustworthy number reaches the person who acts.

If that lag is hours, no amount of ETL tuning fixes it, because the latency is in the architecture, not the job. If it’s minutes, you may not have a problem worth spending on. That distinction — not a platform preference — is where this decision actually lives.

The classic diagram isn’t wrong. It’s answering a question about cost and scarcity that stopped being the binding constraint years ago. The businesses pulling ahead aren’t running a faster version of it. They’re running fewer layers, and putting the action next to the insight instead of in a report nobody opens in time.

If you’ve read this far, you probably recognise your own estate in it somewhere — the overnight job, the mart that’s always one refresh behind, the dashboard the floor stopped trusting.

Thirty minutes is enough to find where the latency actually sits in your stack and whether it’s worth fixing. No slides. No pitch deck. No obligation to proceed. We map your sources, find the one decision that’s chronically late, and decide together whether it’s an architecture problem or something cheaper.

If that’s useful, book the call with Amit. Most operations leaders leave it with three things they hadn’t considered — and a clear view of whether collapsing those layers is worth the cost in their specific case.