How Lucene Handles Deleted Documents

Deleting a document sounds like it should be simple: the document is gone, end of story. But Lucene’s segments are immutable, meaning once a segment is written it can’t be changed. So what actually happens when you delete a document, if the file holding it can’t be edited?

This post walks through how Lucene handles deletes internally, why it works the way it does, and what that means for performance and disk usage.

Deleting doesn’t actually remove anything right away

When you call something like deleteDocuments() on an IndexWriter, Lucene doesn’t go find that document’s bytes and erase them. Instead, it flips a bit in a small per-segment structure that just keeps track of which documents in that segment are still “alive” versus deleted. This structure is usually called the live docs bitset, and on disk it’s stored in a file with a .liv extension (older Lucene versions used .del).

So a delete is really just: find the document, figure out which segment it’s in, and flip its bit to “not alive” in that segment’s live docs file. The actual document data, the text, the stored fields, all of it, is still sitting in the segment, untouched.

Why deletes can’t just remove the data immediately

This goes back to the immutability of segments. Once Lucene finishes writing a segment, it’s done, it doesn’t go back and rewrite parts of it. Actually removing a document would mean rewriting most of the segment’s internal structures (term dictionaries, postings lists, stored fields) just to get rid of one document’s data. Doing that on every single delete would be far too slow.

Flipping a bit in a small separate file is cheap by comparison. It’s a tiny, fast write instead of a full rewrite of a potentially large segment.

What this means while the document is still “deleted but present”

Until the segment containing it gets merged, a deleted document quietly sticks around:

• It still takes up disk space. The actual document bytes are still in the segment file.
• It still uses some memory. Per-document data structures, like norms or doc values, still get loaded for that document, even though it’ll never show up in results.
• Searches still pay a small cost for it. Every time a search checks a potential match, it has to check the live docs bitset to see if that document is actually still alive. That check happens for every potential hit, not just the deleted ones.
• Its terms can linger too. If a word only ever appeared in documents that are now deleted, that word can still technically exist in the segment’s term dictionary until a merge cleans it up. Some people call these ghost terms.

None of this is wrong or broken, it’s a deliberate tradeoff. Lucene accepts some wasted space and a small ongoing search cost in exchange for making deletes themselves nearly free.

How deletes actually get cleaned up: merging

The real cleanup happens during a segment merge. When Lucene merges a group of segments together into one new segment, it checks the live docs bitset for each source segment and simply skips over any document marked as deleted. Only the documents that are still alive get copied into the new, merged segment.

So a merge does double duty. It reduces the number of segments, which helps search speed, and it physically reclaims all the space that deleted documents in those segments were taking up. After the merge, the new segment doesn’t even have a live docs file, because there’s nothing deleted in it yet.

This is also why people sometimes say “Lucene deletes happen at merge time.” The bit flip happens immediately, but the actual disk space isn’t freed until something merges that segment.

A small optimization: dropping fully-deleted segments early

There’s one shortcut Lucene takes that’s worth knowing about. If every single document in a segment ends up deleted, meaning the live docs bitset is entirely zeros, Lucene can just drop that whole segment without waiting for a proper merge. There’s nothing left worth keeping, so there’s no point copying zero documents into a new segment. This is a nice fast path, though it only kicks in when a segment happens to be 100% deleted, which doesn’t happen all that often on its own.

Soft deletes: a different kind of delete

Everything above describes what’s sometimes called a hard delete, the normal kind triggered by deleteDocuments(). Lucene also supports something called a soft delete, which works a bit differently.

With a soft delete, instead of flipping a bit in a separate live docs file, you mark a document by giving it a value in a special field you’ve designated as the “soft deletes field.” As far as normal IndexWriter deletion logic is concerned, a soft-deleted document is treated the same as a hard-deleted one, it won’t show up in live docs once that marker is set.

The difference is what merges are allowed to do with it. With a special merge policy, like SoftDeletesRetentionMergePolicy, you can tell Lucene to actually keep some soft-deleted documents around during a merge instead of discarding them, based on a query you provide. This is useful if you want to retain a recent window of deleted or updated documents, for example to support point-in-time lookups or versioning, without keeping every single deleted document forever.

Hard deletes don’t give you this choice. Once a hard-deleted document is merged away, it’s gone. Soft deletes give you a way to say “treat this as deleted for search purposes, but don’t necessarily throw it away yet.”

Why this matters for you as a developer

A few practical takeaways fall out of all this:

• A high delete count in your index isn’t free, even before anything gets merged. It costs disk space and adds a small amount of overhead to every search.
• Merging is what actually reclaims space, so if your workload deletes a lot of documents, how aggressively your merge policy reclaims deletes matters for keeping the index lean. Lucene’s TieredMergePolicy has a setting for exactly this, a threshold for how many deleted documents a segment can accumulate before it gets prioritized for merging.
• If deletes never seem to clear up, it’s worth checking whether merges are actually happening. An index that only ever receives deletes and no new documents can, in some setups, end up not triggering merges the way you’d expect, since merge policies are usually tuned with new segments in mind.
• Soft deletes are a different tool for a different job. If you need to retain deleted documents temporarily for things like change tracking, look at soft deletes and a retention-aware merge policy rather than fighting against normal hard deletes.

The mental model to keep

A delete in Lucene isn’t an erase, it’s a flag. The document stays exactly where it was, just marked as something to skip. Nothing is actually thrown away until a merge comes along and rebuilds a segment without it. This is what lets deletes be fast, but it also means the real cost of a delete is something you pay later, during merging, rather than at the moment you call delete.