Azure AI Search at Scale: Index Design, Latency and Cost Optimisation for RAG

A proof of concept runs on a small index, one user at a time, with every convenient option switched on because it is cheap at that size. None of those conditions hold in production. The corpus is an order of magnitude larger, dozens of users query at once, and the options that were free at pilot scale now carry a per-query or per-gigabyte price that shows up on the Azure bill.

Scaling Azure AI Search well is mostly about reversing pilot habits: indexing less, storing less, and re-ranking selectively. The service tier matters, but the schema matters more — a well-designed index on a smaller tier will beat a careless one on a larger tier for both latency and cost.

Every field in an Azure AI Search index has independent attributes — searchable, filterable, sortable, facetable, retrievable — and each one you enable adds index size and, for searchable and filterable, write and query cost. The common mistake is marking everything searchable 'just in case'. A large body-text field that users never keyword-search against but do need returned should be retrievable, not searchable. A code you filter on should be filterable, not searchable.

Keep the authoritative content in the system of record and store only what retrieval needs in the index: the chunk text, its vector, and the metadata you filter and cite on. Pushing whole documents, duplicate copies and unused fields into the index is the quickest way to a slow, expensive search service that is painful to reindex.

Two levers scale the service, and they are not interchangeable. Partitions increase storage capacity and split the index for query throughput against large data; replicas increase query concurrency and provide availability (multiple replicas are required for a read SLA). A single-partition, single-replica service is fine for a pilot and wrong for a busy production assistant.

Size them to the actual shape of the workload: a large corpus with light concurrency needs partitions more than replicas; a modest corpus with many simultaneous users needs replicas more than partitions. Both bill continuously as search units, so over-provisioning is a standing cost, not a one-off — scale up when the measured query latency or storage demands it, not pre-emptively.

In a RAG index the vector fields typically dominate storage, and storage drives partition count, which drives cost. The dimension of the embedding, the number of chunks, and whether you store full-precision vectors all compound. Vector quantisation and compression reduce the footprint substantially with a small, measurable recall cost — often a good trade at scale.

Dimension choice matters too: a larger embedding is not automatically better for retrieval, and it is reliably more expensive to store and search. Measure recall on your golden set across a couple of embedding options rather than defaulting to the largest. The cheapest vector is the chunk you did not need to index because the document was duplicated or out of scope.

Semantic ranking and large retrieval depths are per-query costs. Applying semantic ranking to every query across the whole result set is the RAG equivalent of leaving every light on. Retrieve a sensible candidate set, re-rank the top-N, and return the top few — the accuracy gain from re-ranking 50 candidates versus 500 is usually negligible while the cost is not.

Cache where the same questions recur, which in operational settings they heavily do. A thin answer-or-retrieval cache for repeated queries cuts both latency and spend, and most enterprise assistants see a long tail of identical questions — shift handover, policy lookups, the same five SOPs — that never need to hit the full pipeline twice.

Optimisation has a floor set by the corpus and the SLA. If you genuinely must index millions of large documents with sub-second latency and high concurrency, the cost is what it is — no schema trick makes that free, and pretending otherwise just defers the finance conversation. Size it honestly and decide whether the use case justifies it.

Sizing cannot be modelled on paper with confidence: embedding footprint, query mix and concurrency interact in ways that only real load reveals. The reliable method is to load production-representative volume into a test service, replay realistic queries, and measure latency and search-unit consumption — then size from evidence. And reindexing a badly designed schema at scale is slow and disruptive, so the field-design decisions are worth getting right before the index is large.

Before scaling, instrument and measure — you cannot optimise a service you are not watching.

• Audit the index schema: turn off searchable/filterable on fields you do not query, and keep whole documents in the system of record, not the index
• Measure where storage goes — usually vectors — and test quantisation and a smaller dimension against recall on a golden set
• Right-size partitions (storage/throughput) and replicas (concurrency/availability) from measured load, not a guess
• Apply semantic ranking to the top-N only, and cache repeated operational queries
• Load production-representative volume into a test service and replay real queries before committing to a tier