What Happens Internally During a Lucene Commit

Calling commit() looks like a single, simple step from the outside. What’s less obvious is what it actually does under the hood, and why it’s a much heavier operation than just “write some files to disk.”

This post walks through what happens internally during a Lucene commit, in plain terms, and why understanding it matters for both data safety and performance.

A quick word on fsync, since it will come up a lot here

Before going further, it helps to explain one term that shows up throughout this post: fsync.

When a program writes data to a file, the operating system often doesn’t send it straight to the disk right away. It holds the data in memory for a bit, in something called a write buffer, and writes it to the actual disk a little later. This makes writes feel fast, but it also means that if the machine loses power or crashes at the wrong moment, that “written” data can simply vanish, because it was never really on the disk in the first place.

fsync is a request you can make to the operating system that says, basically: don’t return until this data is actually, physically on the disk. Once fsync returns successfully, you can be confident the data will survive a crash or power loss. It’s slower than a normal write because it has to wait for real disk I/O instead of just handing data off to a buffer, but it’s the only way to get a real durability guarantee.

Keep that in mind. A lot of what makes a Lucene commit “heavy” comes down to calling fsync.

Flush and commit are not the same thing

This is the first distinction worth nailing down, because the two words get used loosely and it causes confusion.

• Flush means Lucene writes the documents sitting in memory out to a new segment file on disk. But it does not call fsync. So right after a flush, that data might still only be sitting in the OS’s write buffer, not safely on disk yet.
• Commit is the heavier step. It calls fsync on the relevant files, waits for that to finish, and only then marks this point as the new official, durable version of the index.

A flush can happen many times without a commit ever happening. In fact that’s normal, it happens automatically as documents fill up memory. A commit always makes sure anything pending gets flushed first, but it adds the fsync step and the “this is now official” step on top.

What causes a flush in the first place

Before commit even comes into the picture, it helps to know what causes a flush, since that’s what determines how much data is waiting around by the time you commit.

Lucene buffers documents you add in memory. Once that buffer gets too big, it gets written out to disk as a new segment. Two things can trigger this:

• Memory usage. Once the amount of memory used by buffered documents crosses a configured limit, a flush happens.
• Document count. If you’ve configured a maximum number of buffered documents, hitting that count triggers a flush too.

Deletes are handled a bit differently. They get recorded but don’t force a flush on their own the same way new documents do.

The two steps inside a commit

A Lucene commit actually happens in two steps internally. Most of the time you don’t see this because calling commit() does both steps for you, but it helps to know they’re separate.

Step one: getting everything ready

This step does almost all of the actual work:

1. Anything still sitting in memory gets flushed to disk as new segments.
2. Any pending deletes get applied.
3. fsync gets called on all the relevant files, so the data is genuinely safe on disk.
4. A new file gets written that lists exactly which segments make up this new commit, but this file is deliberately left slightly incomplete (missing one final piece of validation).

That last point matters more than it sounds like it should. Because that file is incomplete, it isn’t considered valid yet. If something tried to read the index at this exact moment, it would notice the file looks broken and would just fall back to the previous, still valid version instead. In other words, nothing has actually changed yet from the outside, even though almost all the real work is already done.

Step two: making it official

The second step is small on purpose. It fills in that one missing piece in the file from step one, and syncs it. Once that’s done, this is now the version of the index that anyone reading it will see. The previous version becomes safe to clean up later.

Splitting it this way is useful when Lucene’s commit needs to be coordinated with something else, like a database, where both things need to succeed or both need to fail together rather than one succeeding and the other not.

Why this two-step approach is safe even if something crashes

Because the “make it official” step is small and happens last, a Lucene commit is essentially all-or-nothing from the outside, even though a lot of separate file writes happen behind the scenes. A reader either sees the old version of the index or the fully complete new version. There’s no in-between state visible to anyone.

This also means a crash in the middle of committing isn’t usually a disaster. Worst case, you end up with some extra segment files on disk that nothing points to yet, and Lucene cleans those up later. The index itself still reflects whatever the last fully finished commit was.

A caveat worth knowing about fsync

It’s worth being upfront about this: fsync is only as trustworthy as the hardware underneath it. On most systems, once fsync returns, the data really is safe. But some disks and storage devices cheat a little, they report success to look faster, while quietly still holding the data in their own internal cache rather than the disk itself. If power is lost at that moment, that data can still be lost, even though fsync said it was fine. This isn’t something Lucene (or really any software) can fully protect against, it’s a property of the hardware.

Why commits are not called constantly

Since a commit involves fsync, often across several files, it costs real time and real disk I/O, much more than a flush does. This is exactly why systems built on top of Lucene, Elasticsearch being a well known example, don’t call commit on every single write. Instead they collect writes and rely on a separate log of recent changes to cover the gap, only calling the actual Lucene commit every so often. That kind of log lives above Lucene, it isn’t something Lucene provides by itself, it’s a pattern applications add when they want durability without paying the fsync cost constantly.

If you’re working with Lucene directly, the same idea applies in a simpler form: commit when you actually need a durable, official checkpoint, but don’t call it more often than you need to, since every call has a real cost.

A simple way to remember all this

Flushing moves documents from memory onto disk. Commit is what makes that move durable and official, using fsync to guarantee the data will survive a crash. The two-step process inside commit exists so that this “durable and official” moment happens all at once from the outside, instead of leaving the index in a half-changed state if something goes wrong partway through.

One more thing worth keeping separate in your head: calling commit() is not what makes new documents searchable. That’s a different mechanism (refresh, and opening a new reader), related to this but not the same thing.