Lakehouse for EPC Document and Contract Data: Structuring Project Files in OneLake

Ask a commercial manager on a live EPC (Engineering, Procurement & Construction) project what the current entitlement position is, and you get a number with a caveat — usually "as far as we know". The answer is spread across a transmittal register in Aconex, an RFI log in Procore, claim letters in email, and a cost report in the ERP that lags all of them.

The ERP knows what has been committed and paid. It does not know that a design change instruction was issued on site nineteen days ago, that the notice clock is 28 days, and that nobody has raised the notice yet.

This is why EPC cost reports are structurally late rather than occasionally late. Exposure originates in documents, becomes an ERP entry weeks later when a human types it, and only then becomes reportable — by which point the entitlement window on some of it has closed.

Structuring unstructured EPC project files means landing documents — drawings, transmittals, RFIs, variation orders, NCRs — into the Files section of a Microsoft Fabric lakehouse in OneLake, then deriving two governed tables: one describing every document's metadata, one holding the commercial facts extracted from it.

Storing a million PDFs in OneLake buys almost nothing. Producing a governed table that says this is Variation Order VO-0412, contract package CP-03, discipline Mechanical, revision C, instructed 04-Jun-2026, valued AED 1.42m, notice not yet issued — that is the deliverable. Files and Tables live side by side in the lakehouse; the Tables section holds Delta tables, auto-registered and queryable.

An EPC estate has three categories of source, each with a different landing mechanism. Documents in Microsoft 365 — correspondence, method statements, site photo packs and claim submissions usually in SharePoint — are referenced through OneLake shortcuts at folder level, without copying. The EDMS — Aconex and Procore both publish REST APIs, so transmittals, registers, RFIs and correspondence are pulled on a schedule into Bronze via Azure Data Factory pipelines. And structured systems — Fabric mirroring covers SQL Server, Oracle and SAP, replicating ERP cost and commitments into OneLake as Delta with change capture.

Once files are addressable there are three realistic extraction routes, and the choice is not a matter of taste. Azure AI Document Intelligence suits stable templates — it has a prebuilt contract model extracting parties, jurisdictions, contract ID and title (English-language contracts only), plus invoice and layout models and custom neural models trainable from five examples.

Azure Content Understanding fits the messy end — subcontractor claim letters, extension-of-time submissions, correspondence where the same fact sits somewhere different in every document. It works zero-shot without labelled training data, supports inferred fields, and returns confidence scores and page-level grounding. And Fabric AI functions keep the work inside the lakehouse — ai.extract, ai.classify and ai.summarize run over pandas or PySpark DataFrames, with multimodal versions accepting file paths directly (PDF, JPG, PNG, WebP and text; Office files need conversion first).

Extracted commercial fields get the attention. The metadata spine is what makes the estate usable, and it is the part teams under-build.

Model this as a conformed dim_document alongside a fact_document_event at one-row-per-event grain — issued, transmitted, responded, superseded. That event table supports the metric commercial teams actually want: elapsed days between an instructing event and a notice being raised, against the contractual notice period.

Extraction handles the fields you knew to ask for; retrieval handles the question nobody anticipated. The Azure AI Search OneLake indexer is generally available and indexes content and metadata from the lakehouse Files location — PDF, Office formats, HTML, JSON, Markdown, email and plain text — including through ADLS Gen2, OneLake, S3 and GCS shortcuts, with incremental change detection.

That index becomes grounding for a Fabric Data Agent. An agent combines up to five data sources, with Azure AI Search supported in preview for unstructured retrieval alongside SQL sources, semantic models and Eventhouse — so one agent holds the cost model, the document metadata tables and the full-text index, and can answer "which variation orders on CP-03 have no notice raised" against all three.

This is where the build earns its budget, and it is mostly a keying problem. The join runs document → WBS/cost code → ERP commitment and actual → P6 activity, and it breaks where a document carries no cost code, where the ERP uses a different code structure than the drawing register, and where nobody owns the mapping between them.

Build the reconciliation as a Gold table with three columns that never get merged: value as extracted, value as assessed by the commercial team, and value as committed in the ERP. Variance between the first and the third, aged, is the exposure report — the thing the commercial manager cannot produce today without a week of chasing.

EPC document estates carry privileged claim strategy, subcontractor pricing under confidentiality and client information governed by the contract, so access design is not an afterthought. OneLake security is generally available with folder-level permissions in the Files section, inheriting recursively, plus table, row and column-level security on the Tables side, enforced across Spark notebooks, the SQL analytics endpoint in user-identity mode and Direct Lake semantic models.

One combination to plan around: the OneLake indexer does not support Fabric items carrying sensitivity labels, though individually labelled documents are supported. If the governance design labels the lakehouse item itself, the search index and the agent built on it stop working — so label documents, not the item.

A lakehouse does not fix a broken transmittal register — if transmittals go out by email outside the EDMS, or get registered days after issue, you now have a fast, well-governed view of an unreliable register. Revision-control chaos survives ingestion: Rev C, Rev C1, Rev C-Rev1 and "Rev C FINAL FINAL" all land, and someone has to define the survivorship rule. Scanned drawings extract badly, and the limits are documented — Document Intelligence needs roughly 8-point text at 150 DPI and page dimensions up to 17×17 inches.

Two commercial truths matter more than any feature. A derived figure is not a contractual position — an extracted variation value with a 0.86 confidence score is a management indicator, not a submitted claim, and will not survive adjudication on its own. And time bars are commercial, not technical — the platform can tell you day 19 of 28 has passed; a person still has to raise the notice. Capacity cost is real too: AI functions bill against Fabric capacity, and a full-estate extraction pass over several hundred thousand documents is not a background task.

Do not start with a platform migration. Answer four questions this month:

• On the last two projects, what was the median elapsed time between an instructing event on site and the corresponding ERP commitment entry?
• What percentage of transmittals were registered in the EDMS within 48 hours of issue?
• Can every document in your register be mapped to a contract package and a WBS code today — and who owns that mapping?
• Which single commercial decision would change if the entitlement position were visible weekly instead of monthly?

If the fourth answer is "none", this is an expensive archive. If it is "we would raise notices inside the contractual window instead of arguing at final account", the case writes itself. A sensible first slice is one contract package on one live project: SharePoint and EDMS documents landed, metadata modelled, one document type extracted end to end — variation orders are usually the highest-value choice — and one aged exposure report joining extracted values to ERP commitments.