What Is a Data Lakehouse? A Practitioner’s Definition

A data lakehouse is low-cost object storage holding open table formats — Delta Lake or Apache Iceberg — with a transaction log that gives those files ACID guarantees, schema enforcement and time travel, queried directly by multiple engines without copying the data out. That is the whole idea.

The reason it matters operationally is narrow and specific. When your maintenance history, your production counts and your invoice lines sit in the same governed storage layer at the same grain, the question "which line's downtime is actually costing us OTIF" has one answer rather than three. When they sit in three systems with three refresh cycles, it has three answers and a meeting.

A data lakehouse is a data architecture in which analytical tables are stored as open-format files in cheap cloud object storage, and a transaction log alongside those files supplies database behaviour — ACID transactions, schema enforcement, versioning and time travel. Multiple query engines read and write the same physical tables, so the data is not copied per tool. Four components have to be present; if one is missing, it is not a lakehouse.

The term was formalised in a 2021 CIDR paper arguing the two-tier lake-plus-warehouse pattern was structurally wasteful.

For about a decade the standard architecture had two tiers. Tier one was a data lake: raw files on cheap storage, holding everything, with no transactions — if a Spark job failed halfway through writing a partition, you had half a partition and no way to know, and a source system changing a column type silently poisoned every downstream read. That is how a data lake becomes a swamp.

Tier two was a warehouse: a subset of that data, loaded through ETL, with proper transactions and governance, serving finance and BI. It worked. It also meant the interesting data — sensor histories, weighbridge logs, telematics, quality inspection images — mostly never got loaded, because warehouse storage and ETL could not be justified for data that might be useful. So you paid twice, and the two tiers drifted. The lakehouse collapses the tiers by adding the missing database behaviour to the cheap tier instead of copying data up into the expensive one.

A transaction log is an ordered set of small metadata files stored next to the data files. Each commit records which files were added and removed. A reader loads the log first and only sees files listed in a committed version, so a half-finished write is invisible. Object storage cannot lock a folder — S3 and ADLS Gen2 do not offer "no one else touches this table while I update it" — which is why plain Parquet folders have no transactions.

Delta Lake never treats the folder listing as the table. The table is defined by the _delta_log directory: numbered JSON files each recording the files added and removed by one operation. The consequences follow directly. Atomicity — new Parquet files are written first, and only then is a single commit appended; if the job dies mid-write, no commit exists, so orphaned files are not part of the table. Isolation — readers pin to a version, so they never see inconsistent data. Concurrency — writers use optimistic concurrency, not locks. Time travel — reading yesterday's version means reading yesterday's file list. Schema enforcement — a write with an incompatible schema is rejected at commit, not discovered three weeks later in a dashboard.

A business-reader analogy that holds: the data files are stock in the warehouse, and the transaction log is the goods-receipt ledger. Stock that has not been booked in does not exist for counting, no matter that it is physically on the floor. Reversing the ledger to a prior date tells you exactly what you held that day.

The row that matters most is table format. A lakehouse is a data lake with a table format bolted on — not a diminishment; that one addition is what changes the estate from unreliable to auditable.

Open format means the storage layer is specified publicly and readable by engines from competing vendors, so the compute vendor does not own the data. If you stop paying that vendor, the Parquet files and the transaction log remain in your own storage account, readable by another engine. Two clarifications, because buyers get this wrong in both directions.

First, it is real. Snowflake's Iceberg tables can store data in an external volume in your own S3, GCS or Azure account. OneLake is built on ADLS Gen2 and stores tables in Delta Parquet or Iceberg, two open standards any tool can read. Amazon S3 Tables are managed Iceberg tables queryable from Athena, Redshift, Spark, Trino, Snowflake and DuckDB. Second, it is partial: your pipelines, orchestration, semantic models, RLS definitions and notebooks are not portable. Migrating a live estate between platforms is still a project measured in months, even when not a single byte of table data has to move. Open format lowers the exit cost materially — it does not make it zero, and any vendor implying otherwise is selling.

In Microsoft Fabric, the lakehouse is OneLake — storage built on ADLS Gen2 that stores tables as Delta Parquet by default, with Power BI reading those tables through Direct Lake, and shortcuts referencing data held in other clouds without copying it. Two features are what a buyer is actually paying for: Direct Lake, a Power BI storage mode that loads from Delta tables in OneLake where refresh copies only metadata (framing) rather than replicating data, so there is no overnight refresh window to protect; and shortcuts plus metadata virtualisation, where a shortcut references data elsewhere and OneLake generates virtual Delta metadata for Iceberg tables and vice versa.

Elsewhere the shape is identical and the components differ: Databricks pairs Delta Lake with Unity Catalog; Snowflake pairs Iceberg tables with external or managed volumes; AWS pairs S3 Tables with the Glue Data Catalog and automatic compaction.

You probably need one if three or more are true:

• You have data operationally valuable but too large or unstructured for your warehouse — MES/SCADA tag history, telematics, images, inspection logs
• You have more than one consuming engine: BI plus a data science team, or BI plus an application needing the same tables
• You run the same reconciliation between two systems every month, by hand
• You need audit-grade answers to "what did this table contain on this date"
• You have four or more source systems feeding analytics

You probably do not need one if your entire analytical estate is one ERP and a finance dashboard, your total data volume is under a few hundred gigabytes, and nobody is asking for machine learning. A well-modelled warehouse, or even a properly built Power BI semantic model over a SQL database, will serve you better, cost less and need far less skill to run. Building a lakehouse for an estate that small is buying complexity you then have to staff.

It is not automatically cheaper — storage gets cheaper; compute usually does not. The saving comes from removing a duplicate tier, not from the architecture itself; estates that add a lakehouse alongside an untouched warehouse pay for both. It does not fix data quality — ACID guarantees a write either fully happened or did not, and says nothing about whether the number was right. A duplicated customer master or a production count logged by an operator who was guessing commits perfectly.

Time travel is bounded, and shorter than people assume — Delta's default VACUUM retention is seven days, after which older versions are unrecoverable; seven years of auditable history is a deliberate retention design and a storage cost, not a default. Small files and maintenance are real operational work — frequent small writes produce thousands of small Parquet files, and compaction, V-Order and vacuum need scheduling. And serving has guardrails you must design against — Direct Lake limits vary by SKU (F2–F8 caps a table at 300 million rows; F64 raises it to 1,500 million), and Direct Lake on SQL falls back to DirectQuery against a SQL view or with SQL-based access control. Discovering this after go-live is the usual cause of "the report got slow".

Before anyone draws an architecture, answer five questions this week:

• Name the specific decision made late or wrongly because two systems disagree. If you cannot name one, stop here
• How many analytical copies of your ERP data exist today — extracts, staging, Power BI imports, spreadsheets? Count them
• What data are you deliberately not analysing because it is too large or awkward for your warehouse?
• How long do you need to reproduce a historical report, and does anyone currently have that ability?
• Who will own the tables — not the platform, the tables — after go-live?

Question five decides more outcomes than the technology choice does. We build these on Microsoft Fabric, OneLake and Power BI — first value in six weeks: one real slice, modelled properly, not a diagram of what could be built.