Doc Values vs Stored Fields vs Term Vectors

If you’ve spent some time configuring fields in Lucene, you’ve probably bumped into all three of these: stored fields, doc values, and term vectors. They sound like they might overlap, all three are ways of keeping extra information about a field around, but they exist to answer completely different questions, and they’re stored on disk in completely different shapes.

This post walks through what each one actually is, why Lucene needs all three instead of just one, and how to think about which one you actually need for a given field.

They all answer a different question

Before getting into the mechanics, it helps to frame each one around the question it’s actually built to answer:

• Stored fields answer: “give me back the original value of this field for this specific document.”
• Doc values answer: “for this field, give me the value for every document quickly, especially across lots of documents at once,” which is what sorting, faceting, and aggregations actually need.
• Term vectors answer: “for this document, what terms does this field contain, and where exactly do they occur,” which is mainly useful for things like highlighting and “more like this” style features.

Same field, three very different jobs. Lucene keeps them as separate systems because each one is shaped for a different access pattern, and trying to force one structure to do all three jobs would make all of them slower.

Stored fields: getting the original value back

When you mark a field as stored, Lucene keeps the original value you gave it, untouched, so it can hand it back to you later as part of a search result. This is what lets you run a query and get back an actual title, body, or price, rather than just a document ID.

Internally, stored fields are organized by document, row by row, similar to how a row in a regular database table works. All of a document’s stored field values sit together on disk. This is great when you want everything about one document at once, you find the document, then grab all its stored fields in one go.

The catch is that this layout isn’t great if you only want one specific field’s value across many documents. To get the price field for a thousand different documents, Lucene still has to seek to each document’s row and read through it, even if you only care about that one field. Reading a single field from many documents this way involves more scattered disk access than you’d expect.

Doc values: getting values fast across documents

Doc values exist specifically to fix the scattered access problem stored fields run into for certain tasks. Instead of organizing data by document, doc values organize data by field. All the values for one field, across every document, sit together on disk, one after another, lined up by document number.

This is sometimes called a column based layout, as opposed to the row based layout stored fields use. If you want the price field for a thousand documents, doc values let you do something much closer to one continuous scan through the price column, instead of jumping all over a row based file.

This is exactly why doc values are what Lucene uses for sorting, faceting, and aggregations, the kinds of operations that touch the same field across huge numbers of documents, often without caring about any of the other fields on those documents at all. Numeric range filtering and grouping results by a category field both lean on doc values for the same reason.

One thing worth knowing: doc values aren’t meant for full text. They store fixed, simple values per document, numbers, dates, single strings, or small sets of values, not the kind of free text you’d run a phrase query against. If a field needs to be both searchable as text and sortable, it’s common to index it one way for search and add doc values separately for the sortable form.

Term vectors: a mini inverted index per document

Term vectors flip the normal inverted index inside out, but only for a single document at a time. Lucene’s main index answers “which documents contain this term,” term vectors answer “which terms does this document contain, and where.” For a given document and field, a term vector can tell you every term that occurred, how many times, and optionally the exact positions and character offsets where each occurrence happened.

This is genuinely a separate structure from the main postings data, even though it’s built from largely the same underlying analysis. You have to explicitly tell Lucene to store term vectors for a field, they aren’t there by default, and storing them does mean a bigger index, since you’re essentially keeping a second, per-document view of the same term data.

The main reason people reach for term vectors is highlighting (showing the matching snippet of text with the search terms marked) and “more like this” style similarity features, both of which need to know exactly where terms occurred inside a specific document rather than just knowing the document matched. Without stored offsets and positions, Lucene can still do some highlighting by re-analyzing the stored or original text on the fly, but for longer fields or fields that are expensive to re-analyze, having that data precomputed in a term vector can be a lot cheaper at query time.

For short fields, term vectors usually aren’t worth it, re-running the analyzer on a short bit of text at query time is cheap enough that paying the storage cost up front doesn’t pay off. They tend to make more sense on longer text fields, or fields where re-analyzing on every query would be noticeably expensive.

How to think about turning these on

None of these three are free, they all cost index size and some indexing time, so it’s worth being deliberate per field rather than turning everything on everywhere.

• Turn on stored fields for whatever you actually need to show back to the user in results: titles, snippets of body text, prices, whatever ends up rendered on a results page.
• Turn on doc values for any field you sort by, filter by range on, facet on, or aggregate over. If a field only ever gets searched with text queries and never touched for sorting or faceting, it usually doesn’t need doc values at all.
• Turn on term vectors mainly for fields where you’re doing highlighting or similarity matching on longer text, and where re-analyzing the text at query time would be noticeably slow. For short fields, or fields you never highlight, skip them, the storage cost usually isn’t worth it.

It’s also worth remembering these aren’t mutually exclusive. A single field, like a product description, might reasonably be indexed for search, stored so you can show it in results, and have doc values turned on for some derived sortable version of it, all at once. Each one is doing a different job, so there’s no rule that says you have to pick only one.

The mental model to keep

Three different questions, three different shapes on disk. Stored fields give you back what a document originally said, organized by document. Doc values let you scan one field across tons of documents fast, organized by field. Term vectors give you a detailed, per-document map of exactly where terms occurred, organized around the document but built like a mini inverted index. None of them substitutes for the others, and figuring out which question you’re actually trying to answer is usually enough to tell you which one (or which combination) you need.