Azure AI Search for Enterprise RAG: Hybrid Retrieval, Semantic Ranking and Metadata Filters

When an enterprise RAG assistant gives a wrong or vague answer, the instinct is to blame the language model or the prompt. In practice the failure is almost always one step earlier: the retrieval stage handed the model the wrong passages, and a capable model wrote a fluent, confident answer from poor source material. Fix retrieval and most hallucinations disappear on their own.

On the Microsoft stack, retrieval is Azure AI Search, and whether it is wired into a Copilot Studio agent, an Azure OpenAI app or Microsoft Fabric, the same three levers decide quality: how you search, how you re-rank, and how you filter. Get those right before touching the prompt.

A pure vector index is good at concepts and terrible at exact strings. Ask it for part number 'BRG-4471-A' or clause '7.3(b)' and vector similarity will happily return something that is about bearings or about contract clauses, but not the one you named. Keyword search (BM25) nails the exact token; vector search nails the paraphrase. Hybrid search runs both and fuses the results with Reciprocal Rank Fusion so a passage that scores well on either method rises.

For industrial and B2B content — spec sheets, SOPs, contracts, SAP document text — this matters more than the choice of embedding model, because so much of what users search for is a code, a tag or a named entity. Enable hybrid first. It is the single change that most improves answer accuracy, and it costs nothing beyond indexing both representations.

Hybrid search returns a strong candidate set, but the best passage is not always first. Semantic ranking applies a second-stage machine-learned re-ranker (an L2 model) over the top candidates, scoring each for genuine relevance to the query rather than lexical or vector proximity. It is the difference between 'the answer is somewhere in these ten chunks' and 'the answer is in chunk one'.

It is not free: semantic ranking is billed per query and adds latency, so apply it to the retrieved top-N rather than the whole index, and measure the accuracy lift before switching it on everywhere. For a grounded Copilot answering operational questions, the lift is usually worth it; for a high-volume, low-stakes search box, it may not be.

Retrieval accuracy and retrieval safety are the same problem viewed twice. Metadata filters — on plant, document type, effective date, language, business unit — narrow the search to the passages that can possibly be relevant, which both improves precision and stops the model grounding on the wrong context. A query scoped to 'current, UK, quality SOPs' will not return a superseded 2021 revision from another site.

Security trimming is the non-negotiable version of this: every chunk carries the identity groups allowed to see it, and the query filters on the caller's Entra ID group membership so the assistant can never surface a document the user has no right to read. Build this into the index schema from day one — retrofitting access control onto a live RAG index is painful, and an assistant that leaks a restricted contract clause is a governance incident, not a bug.

How you split documents into chunks sets the ceiling on retrieval quality. Chunks that are too large bury the answer in noise and dilute the embedding; too small and they lose the context that makes them meaningful. Splitting on semantic boundaries — headings, clauses, sections — with a small overlap beats fixed character windows for structured enterprise documents, and keeping the source heading in each chunk gives both the ranker and the model something to anchor on.

Tables, diagrams and scanned PDFs are where naive chunking quietly fails: a pipe-delimited table flattened into a text blob retrieves badly and reads worse. Extract tables as structured text, pair scanned pages with a document-cracking step, and keep a metadata pointer back to the source page so answers can cite it. The embedding model is the last thing to tune, not the first.

None of this fixes a bad corpus. If the authoritative answer is not in the indexed documents, or three contradictory versions are, retrieval cannot save you — you will reliably fetch the wrong-but-present passage. Governance of what goes into the index is the real work, and it is a content-ownership problem, not a search problem.

Stale content is the quiet killer: without an effective-date filter and a reindex discipline, last year's price list or a withdrawn SOP will keep surfacing with full confidence. And cost scales with usage — semantic ranking and large vector indexes have a per-query and per-GB price that a proof of concept never reveals, so model the steady-state query volume before you promise the business an always-on assistant.

Start with the corpus, not the configuration. Pick one well-bounded document set with a clear owner — say, current quality SOPs for one site — and get that right end to end before widening scope.

• Confirm the document set is current and authoritative, with superseded versions removed or date-stamped
• Design the index schema with metadata (site, type, effective date) and security groups from the first pass
• Enable hybrid search, then add semantic ranking over the top-N and measure the accuracy difference
• Fix chunking for tables and scanned pages before blaming the embedding model
• Build a small golden set of real questions with known-correct source passages, so every change is measured, not guessed