The obvious optimisation
When a model-backed feature gets slow or expensive, the first suggestion is always to cache completions. It is easy to reason about and easy to implement: hash the prompt, store the answer, serve it again.
It also fails quietly. Prompts carry timestamps, user context and document versions, so the hit rate is far lower than anyone expects, and the hits you do get are the ones most likely to be stale in a way the user notices.
Where the time actually went
We traced one workflow end to end. Embedding the query and running retrieval with reranking accounted for well over half the wall-clock time and a meaningful share of the cost, before the model had produced a single token.
That step is also far more repetitive than the completion. Dozens of differently worded questions resolve to the same handful of source chunks, and those chunks change only when a document changes.
What we cache now
Query embeddings, keyed by normalised text. Retrieval results, keyed by the embedding plus the filter set, invalidated by document version rather than by a clock.
Completions we cache only for genuinely idempotent calls, such as summarising a specific document version with a specific prompt template. Everything else streams fresh.
The result
Median response on the main workflow dropped by roughly half, and the cost curve flattened because the repeated work stopped being repeated. Nothing became stale, because the invalidation key is the document, not a timer.
The general lesson holds beyond retrieval: profile the pipeline before caching the part that is easiest to cache.