Delta Live Tables vs Fabric Data Pipelines: Late-Arriving IoT Data

A pattern from the plant estates I have worked in. The Wednesday shift report says the packing line ran in the high seventies on OEE. By Friday the same number has quietly dropped, because a gateway synced two days of buffered readings and the downtime they represent got booked into the wrong window.

This is the most corrosive failure mode in industrial analytics, because nothing is broken. No job failed. The pattern shows up wherever the sensor sits further from the network than the ERP does: a service fleet whose tablet syncs the next morning, an OPC-UA collector buffering to disk when the historian link drops, an MQTT broker on a remote pump station batching an hour of readings to save cellular cost.

So which handles this better — Databricks Delta Live Tables, or Microsoft Fabric? The honest answer starts with correcting the question.

Late-arriving data is a record whose event time — when the sensor took the reading — is materially earlier than its processing time, when the platform received it. A pipeline that groups by processing time will book Wednesday's downtime against Friday.

Three distinctions do most of the work. Event time versus processing time — every telemetry record should carry a device-generated timestamp, and if it does not, nothing downstream can fix it. Watermarking and out-of-order tolerance — any stateful stream computation (a five-minute average, a downtime duration, a join between machine-state and production-count streams) holds state open for a bounded period, and the watermark sets how long. And append versus restate — a late record appended to a raw table is easy and every tool handles it; the hard part is restating an aggregate that was already published.

Delta Live Tables — now Lakeflow pipelines — is a declarative ETL framework: you define streaming tables and materialized views, and the engine works out dependencies, incrementality, checkpoints and recovery. A Fabric Data Pipeline is an orchestrator — Microsoft's own decision guide describes pipeline Copy activity as built for orchestrated ingestion workflows where you define and manage components yourself.

Comparing Delta Live Tables to a Fabric Data Pipeline compares a stream processor to a scheduler. The fair comparison is against the Fabric capabilities that actually process streams: Eventstream and Eventhouse under Real-Time Intelligence, Spark Structured Streaming in Fabric notebooks, and Materialized Lake Views for declarative transformation.

Ingestion is Auto Loader (cloudFiles), which incrementally processes new files as they land, in directory-listing or file-notification mode, with exactly-once guarantees held in a RocksDB-backed checkpoint. Where state is involved — windowed aggregations, stream-stream joins, deduplication — you declare the tolerance in SQL: WATERMARK eventTimestamp DELAY OF INTERVAL 3 MINUTES.

Out-of-order corrections use AUTO CDC INTO with SEQUENCE BY, supporting SCD Type 1 and Type 2 with tunable tombstone retention. Data-quality constraints are declared inline as expectations with three actions — warn (invalid records still written, the default), ON VIOLATION DROP ROW, and ON VIOLATION FAIL UPDATE. Triggered pipeline mode suits refreshes every 10 minutes, hourly or daily and releases the cluster when done; continuous mode targets seconds-to-minutes on an always-running cluster.

Fabric has three genuinely different doors, and choosing the wrong one is the usual cause of disappointment. Eventstream ingests through 40-plus connectors including MQTT, Azure IoT Hub, Kafka and Event Hubs, routing to Eventhouse, Lakehouse, Activator or a Spark notebook, with a no-code processor and a SQL windowing operator in preview.

Eventhouse and KQL is where Fabric is strongest for out-of-order telemetry, and it is under-used by teams arriving from a BI background: a KQL materialized view stores a materialised part plus an unprocessed delta, and querying it combines both, so results are current regardless of when materialisation last ran. Fabric Spark notebooks give you the same Structured Streaming primitives Databricks builds on — withWatermark, tumbling/sliding/session windows, stream-stream joins, dropDuplicatesWithinWatermark, a RocksDB state store — the capability is there, the declarative wrapper is not. And Materialized Lake Views are the nearest analogue to Delta Live Tables' materialized views, carrying data-quality constraints with ON MISMATCH DROP or FAIL — but note watermarks are not available in MLVs, so event-time state belongs in Spark or KQL.

Databricks wins when the late-data logic is complex and must be expressed once: multi-stream joins across machine state, production counts and quality inspections, each with its own tolerance; SCD Type 2 history on equipment master arriving late from a CMMS; expectations that must produce auditable drop counts. Fabric wins when the telemetry is high-volume, high-cardinality and mostly queried by time and device — Eventhouse ingesting from IoT Hub or MQTT via Eventstream, with KQL materialized views for latest-state and rollups, gives sub-second interrogation of raw signal history a Delta-only architecture struggles to match.

Fabric Data Pipelines win at what they are for: sequencing the gateway file landing, triggering a notebook or MLV refresh on a OneLake event, calling the historian export and retrying it when the plant network drops. Do not ask them to hold event-time state. In practice: already on Azure Databricks with a data-engineering team — keep the stateful logic in Lakeflow and serve the result in Power BI. Standardised on Fabric with no Spark specialists — put telemetry into Eventhouse via Eventstream, keep KQL for signal history, and reserve Materialized Lake Views for the append-friendly gold layer.

Neither tool can invent a timestamp your device never sent — if the OPC-UA collector stamps records at read time rather than source time, every event-time calculation downstream is fiction. Every watermark is a decision to be wrong in one direction: set the tolerance at 15 minutes and a gateway that syncs after 9 hours contributes nothing to the aggregate. Restated numbers still need a governance answer — if Wednesday's OEE can change on Friday, somebody has to decide whether the shift report is a snapshot never revised or a living figure with a version stamp.

Two more: Fabric's incremental-refresh preconditions are stricter than they look — OEE state-duration logic leans on window functions, and window functions force a full refresh on Materialized Lake Views. And continuous processing has a standing cost — a Databricks continuous pipeline needs an always-running cluster; a Fabric F-SKU running an Eventhouse and a streaming notebook consumes units around the clock.

Answer these four questions with real data from the last 90 days, before choosing anything:

• What is the actual distribution of lateness? Compute ingestion_time − event_time per device and plot the tail
• Which numbers are allowed to change after publication, and who says so? Get that written down before you build
• Do all devices emit a trustworthy event timestamp, and are their clocks synchronised? Clock drift produces the same symptom as lateness and is fixed differently
• What decision would change if the number were right the first time? If nobody acts on the shift report inside the shift, correctness at T+6 hours is sufficient and the cheaper architecture wins

A useful first slice is one production line, one week of raw telemetry with both event time and ingestion time preserved, and a single OEE calculation built twice — once against processing time, once against event time with an explicit tolerance. The gap between the two curves is your business case, measured rather than asserted. We are equally willing to tell you the existing stack is fine and the real problem is a timestamp contract with your OT team.