Fabric Runtime 2.0 GA: What Changes for Spark ETL in Manufacturing

A notebook that reads a SCADA historian export and a set of MES production orders runs at 02:00, as it has every night for fourteen months. One morning it has failed. Nothing in the business changed overnight. What changed was the Spark runtime underneath the notebook — approved three weeks earlier as a housekeeping ticket by someone who reasonably assumed a runtime is like a Windows patch.

Microsoft Fabric Runtime 2.0 reached general availability on 14-Aug-2026, and it is a larger jump than any previous Fabric runtime change: Apache Spark 3.5 to 4.1, Delta Lake 3.2 to 4.2, Python 3.11 to 3.13, Java 11 to 21, Scala 2.12 to 2.13, and the base OS from Azure Linux 2.0 to 3.0.

The clock matters more than the features. Runtime 1.3 — what most production estates run today — has an end-of-support date of 30-Sep-2026, after which it moves into long-term support through March 2027.

Fabric Runtime 2.0 is the generally available Apache Spark execution environment for Fabric notebooks and Spark job definitions. It carries Apache Spark 4.1, Delta Lake 4.2, Python 3.13, Java 21, Scala 2.13 and R 4.5.2 on Azure Linux 3.0.

Runtime 2.0 was not the default at GA. Microsoft's documentation states it becomes the default for new workspaces and new Environment items in late September 2026 — which is exactly why the real risk is drift, not a switch-off.

Fabric gives a runtime roughly 24 months at GA, then 12 months of long-term support, with end of support announced six months before deprecation.

The failure mode is not a dramatic switch-off. It is drift: half your workspaces on 2.0 because they were created after the default changed, half on 1.3 because nobody touched them, and a notebook copied between them behaving differently for reasons that take a day to find.

The Python jump is the one that will bite you. The move to Python 3.13 breaks Environment items that carry Python and wheel libraries. In a manufacturing estate, your OT ingestion notebooks almost certainly use libraries outside the base runtime — an OPC-UA client, an MQTT client, a historian SDK, a vendor wheel for a proprietary MES export format, possibly a hand-built internal wheel a contractor produced in 2024 that nobody can rebuild. That is the honest heart of this upgrade: the risk is not in Spark, it is in your dependency inventory.

Scala and Java move too — Scala 2.12 to 2.13, Java 11 to 21. Scala minor versions are not binary compatible, so any custom JAR (a Kafka or OPC-UA connector, a bespoke deserialiser for a line-controller payload) has to be recompiled, not copied across.

Delta Lake 4.2 brings Z-ordering and liquid clustering for data layout, plus parallel Delta snapshot loading — genuinely useful for a Silver layer holding years of tag-level sensor history. The caveat is the important part: Microsoft states Delta 4.x-specific features are experimental, work only on Spark experiences, and should not be enabled where the same tables are used across multiple Fabric workloads. In a typical estate where Silver and Gold are written by Spark and read by a Power BI Direct Lake model, the SQL analytics endpoint and possibly a Warehouse, turning on a Delta 4.x feature because a blog said it was faster can put those tables into a state other workloads read badly.

What you can use now: the native execution engine runs on both 1.3 and 2.0 and is documented as up to six times faster than open-source Spark on TPC-DS, enabled at the Environment level under Spark compute → Acceleration. And resource profiles — the setting most teams have never looked at: new workspaces default to writeHeavy (V-Order off, for ingestion throughput), while readHeavyForPBI enables V-Order and optimizeWrite, which is what you want on tables a Direct Lake model reads.

The Environment item is what makes a safe test possible: it holds the runtime version, Spark compute config, libraries and resource files, and attaches to individual notebooks or acts as the workspace default. So you can run Runtime 2.0 on one notebook while everything else stays on 1.3.

• Inventory dependencies first — export the library list from every Environment
• Build a parallel workspace — clone the notebooks and job definitions, point them at read-only shortcuts to the same OneLake data, attach a new Environment set to 2.0
• Rebuild the Environment, do not edit it — given the Python breaking change, create it new
• Run a full night's cycle, not a sample — shift-boundary timestamps, a line down for maintenance, null sensor readings, a duplicated historian tag, month-end MES postings are the rows that break
• Regression-check the Silver layer numerically — row counts, distinct keys, null rates, min/max and sum on every measure, and a full-outer-join diff on the business key between 1.3 and 2.0 output
• Check downstream — refresh the Direct Lake model against the 2.0-written tables and query the SQL analytics endpoint
• Cut over by Environment, layer by layer — ingestion, then transformation, then anything serving reports

For ten to thirty notebooks and a handful of custom libraries, that is realistically 3–5 weeks elapsed with an engineer part-time — most of it waiting on test cycles and chasing library owners.

Fabric bills Spark against capacity — two Spark vCores per capacity unit, so an F64 gives 128 Spark vCores, rising to 384 with the 3x burst multiplier. When capacity is exhausted, background jobs queue FIFO and expire after 24 hours, while interactive notebook work is rejected. Two consequences during migration: running parallel 1.3 and 2.0 pipelines over the same window doubles Spark consumption, so if capacity is tight at 02:00, stagger the test rather than discovering the queue.

And if Spark is what pushes your capacity into throttling, autoscale billing for Spark is worth evaluating separately — it moves Spark onto pay-as-you-go serverless compute billed outside the capacity, but it does not burst from or fall back to capacity and does not use smoothing. It changes the cost profile rather than simply removing the constraint.

Every estate over two years old has them: a notebook a contractor built for a plant that has since changed MES vendors, a job feeding a report three people opened last quarter, an ingestion job for a decommissioned line whose historian tags still exist. A runtime deadline is the best excuse you will get to retire them.

Before migrating anything, pull the last 90 days of run history and downstream usage, and answer three questions per notebook and job definition: what does it write, who read that output last quarter, and who is the named owner. Anything failing all three goes on a disable-and-observe list — switch it off, leave it a fortnight, and migrate only what survives.

A runtime upgrade cannot fix undocumented dependencies — if an internal wheel was built by someone who has left and the source is gone, no Fabric configuration recovers it. Regression-testing OT data is harder than ERP data, which is transactional and reconcilable; sensor history is not. The new Delta features are not a free upgrade — Spark-experience-only, and capable of putting tables into a state other workloads read badly. The native execution engine is still preview and falls back silently, and a TPC-DS benchmark is not a forecast for a pipeline parsing MQTT JSON, a format the engine does not support.

And nothing here fixes a pipeline that was fragile to begin with — a notebook with no schema enforcement, no idempotency and no alerting failed at 02:00 on Runtime 1.3 too; it just failed less often. Capacity headroom is a constraint, not a detail: if your F-SKU already throttles, a parallel test cycle makes it worse before it makes it better.

Four questions to answer this week, before any change request is raised:

• Which runtime is every workspace on, and which will quietly start defaulting to 2.0 for items created after late September 2026?
• What is the complete library list across your Environments, and which entries are custom wheels or JARs with no identified owner or source?
• Which Spark notebooks and job definitions wrote data in the last 90 days, and which of those outputs were actually read?
• If your Silver layer produced numbers that differed by 0.5% after an upgrade, would anyone notice — and what would they be looking at?

The fourth question decides the shape of the exercise. If nothing downstream would catch a small numerical shift, the gap in your platform is verification, not runtime version. We run these upgrades the way we build the platforms — never moving a production layer without a numeric regression check behind it.