Lucene File Formats Explained

Open up a Lucene index directory and you’ll see a pile of oddly named files. _0.si, _0.cfs, segments_3, maybe a write.lock. None of that is random. Every one of those files has a specific job, and understanding what they do makes a lot of Lucene’s behavior (merging, deletes, compound files) click into place.

This post walks through what these files are, grouped by what they’re actually for, rather than trying to memorize an exhaustive list of every extension.

Segments are the foundation

Before the files make sense, it helps to remember the big picture: a Lucene index is made up of segments, and each segment is basically its own mini index. Every segment gets its own set of files, all sharing the same name prefix, like _0 for the first segment, _1 for the next one, and so on. The different extensions after that prefix are the different pieces of data that make up that segment.

So when you see a folder full of _0.something, _1.something, those aren’t separate indexes, they’re separate pieces of the same two segments.

The files that point to everything: segments_N

At the top level, there’s a file like segments_1, or segments_2 after the next commit, and so on. This is the file that says, right now, these are the segments that make up the index. Every time you commit, Lucene writes a new one of these with a higher number, and that becomes the new “current” pointer.

This file matters a lot for safety. Lucene is built so that this file either fully represents a valid commit or it doesn’t get treated as valid at all, there’s no in-between state a reader can stumble into. If you’ve read about how commits work internally, this is the file all of that machinery is protecting.

There’s also a write.lock file sitting alongside it. It doesn’t store index data, it just exists to make sure only one writer touches the index at a time.

The files inside a segment

Within a single segment, the data is split across several different files, each handling one specific kind of information. Broadly, they fall into a few groups.

Segment metadata

Every segment has a small info file (commonly .si) that describes basic facts about the segment: how many documents it has, what codec version wrote it, and so on. There’s also a field info file (often .fnm) that lists out every field in the segment and some basic facts about how each one was indexed, like whether it’s stored, whether it has norms, whether it has doc values.

Think of these as the segment’s table of contents. Before Lucene can read anything else, it reads these to know what’s even in here.

The inverted index: terms and postings

This is the heart of search. For every term, Lucene needs to know which documents contain it, how often, and where. That information is split across a few files:

• A file holding the actual list of terms (sorted, so they can be looked up quickly), along with basic stats like how many documents each term appears in.
• A file holding the postings themselves, the actual list of document numbers that contain each term, plus frequency information.
• A separate file for positions, used when phrase queries or proximity search need to know exactly where in a document a term occurred.

These files together are what let Lucene answer “which documents contain this word” quickly without scanning every document.

Stored fields

Separately from the searchable, inverted data, Lucene also keeps the actual original field values you asked it to store, so it can hand them back to you in search results. This lives in its own pair of files: one holding the actual stored data (usually compressed), and one acting as an index into it so Lucene can jump straight to a given document’s stored fields instead of scanning from the start.

This is a genuinely different system from the postings files above. Searching and retrieving stored field values are two separate jobs internally, even though from the outside it feels like one index.

Doc values

Doc values are a column-oriented way of storing per-document values, mainly used for sorting, faceting, and aggregations rather than full text search. Instead of “for this term, which documents have it”, doc values answer “for this document, what’s its value in this field”, which is a very different access pattern and needs its own file format to be efficient.

Norms

Norms are small per-field, per-document values used in scoring, mainly to account for things like field length when ranking results. They get their own dedicated files too, separate from doc values, since they’re used differently internally even though conceptually they’re also “a value per document.”

Points, for numeric and spatial fields

Fields that need efficient range queries, numbers, dates, geographic points, are indexed using a structure based on KD-trees rather than the term based postings system used for text. This gets its own set of files as well, letting Lucene answer “give me everything between X and Y” efficiently without it looking anything like a text search.

Vector fields

Newer versions of Lucene support storing dense vectors for nearest neighbor search, the kind of thing used in semantic or embedding based search. These get their own files too: one for the raw vector data, one for the graph structure used to search the vectors efficiently (commonly an HNSW graph), and a small metadata file tying it together.

Live docs (deletions)

When a document gets deleted, Lucene doesn’t rewrite anything, it just flips a bit in a small file that tracks which documents in the segment are still alive. If a segment has no deletions at all, this file doesn’t even need to exist. We’ve covered this one in more detail in an earlier post on deletes, but it’s worth remembering it’s a real file on disk, not just an in memory thing.

The compound file: bundling it all together

If you’ve looked at a small or freshly written segment’s files, you may have noticed there often aren’t many separate extensions sitting around, instead there’s often just a .cfs file and a .cfe file. This is Lucene’s compound file format. Rather than leaving a dozen or more small files scattered in the directory per segment, Lucene can bundle most of them into one single file (the .cfs), with a small companion file (the .cfe) acting as a table of contents that says where each original file’s data starts and ends inside the bundle.

This exists mainly to cut down on the number of open file handles and file system overhead, especially when you have lots of small segments. Larger segments sometimes skip compound format, since the overhead matters less once a segment is big enough on its own.

A couple of files never get folded into the compound file: the segment info file and the live docs (deletions) file, since those need to be readable on their own without unpacking the whole bundle.

Why understanding this actually helps

You don’t need to memorize every extension to use Lucene well, but a few things click better once you’ve seen this layout:

• Why merges are expensive. A merge has to read through nearly all of these different file types from each source segment and rewrite fresh versions of them into the new segment. It’s not just copying one file, it’s rebuilding several different data structures at once.
• Why deletes are cheap but space isn’t reclaimed immediately. The live docs file is small and fast to update, but the actual stored fields, postings, and so on are untouched until a merge rewrites the segment without the deleted documents.
• Why file handle limits matter. Without compound file format, every segment is several open files, not one. A lot of small segments can add up to a surprising number of file handles fast.
• Why different field types have such different performance characteristics. Full text search, sorting, range queries, and vector search are all backed by genuinely different file formats under the hood, each suited to a different access pattern. That’s why, for example, a field optimized for full text search isn’t automatically efficient to sort on, they’re different data structures entirely.

Yes, you can actually look inside these files

All of this might sound like internal plumbing you’re not meant to touch, but you genuinely can open these files up and poke around, you just need the right tool rather than trying to read raw bytes yourself.

• Luke. This is the easiest place to start. It’s a GUI tool that lets you point at an index directory and browse it: see the list of segments, look at individual documents, check stored field values, look up a term and see its postings, inspect doc values, and run basic health checks. Luke used to be a separate download, but it’s now bundled directly with Lucene itself (it became an official part of the Lucene project starting with Lucene 8.1), so if you grab a Lucene release, you likely already have it.
• CheckIndex. This ships with Lucene as a command line tool, and it’s meant for verifying an index is healthy rather than casual browsing, but it’s still genuinely useful for understanding what’s going on. Run it against an index directory and it’ll walk through every segment, reporting document counts, deleted document counts, and whether each internal structure (postings, stored fields, doc values, and so on) checks out okay. It also has a repair mode for emergencies, though that one should be used carefully since it can permanently drop problematic segments.
• Clue. If you’re working on a remote box over SSH and a GUI isn’t an option, Clue is a command line tool built specifically for that case. It can show you directory contents, term postings, stored field values for a given document, and run searches, all without needing a graphical interface, and it plays nicely with piping into normal Unix tools like grep.

None of these tools require you to understand the raw byte layout of .tim or .fdt files yourself, they all sit on top of Lucene’s own codec classes to decode everything properly. That’s really the right way to look at these files anyway, the formats are versioned and somewhat involved, so reaching for a tool that already knows how to read them beats trying to parse them by hand.

The mental model to keep

A Lucene segment isn’t one file, it’s a small bundle of specialized files, each one a separate data structure built for a separate job: finding documents by term, retrieving stored values, sorting by a field, searching vectors, or tracking deletions. The compound file format just changes whether those pieces are bundled into one physical file or left as separate ones, it doesn’t change what they’re doing underneath.