[nick-boss-tech]

🤖 AI text below 🤖 (reviewed by Nick Shanin)

https://issues.apache.org/jira/browse/SOLR-18010

What happens today

A reader can see an empty or partial security.json, and a crash mid-write leaves one behind. persistConf at the merge base opens the file with truncation before it writes the new content. #5065 serializes edits inside one process. It does not protect the file on disk.

What this change does

An edit writes a temp file, syncs it, and renames it over security.json. Readers see the old file or the new one. The new file keeps the owner, group and mode (code).

• The temp file is owner-only until the old mode is restored, so it is never wider than the file it replaces.
• A failed write, sync or permission copy fails the edit and keeps the old file.
• The file is written in place, with a warning, for a symlink, a hard-linked file, a file system without POSIX attributes, a read-only SOLR_HOME, a refused move (for example a bind-mounted file), a file owned by another user, and a group that cannot be set.
• ACLs and SELinux labels are not kept. See the first choice below.
• The version marker stays.

Other projects use a similar pattern. ZooKeeper's AtomicFileOutputStream syncs, then renames. Kafka's atomicMoveWithFallback renames, then syncs the parent directory.

Choices to check

Renaming over security.json drops its ACLs and SELinux labels. Is that acceptable?

Main writes this file in place, so a crash can truncate it. A rename cannot leave a partial file. The cost is that the new file is a new inode, and Java on Linux cannot read or copy ACLs or SELinux labels. The reference guide says so.

• Rename (implemented). No partial file after a crash. An ACL or custom label is set again after each edit.
• Write in place (main). Labels are kept. A crash during the write can truncate the file.

How to decide:

1. Does security.json have an ACL or a custom SELinux label? Check with getfacl and ls -Z. If no, rename costs nothing.
2. If yes, can a deploy step set it again after each edit? If yes, rename and document that step.
3. If no, write in place, and accept that a crash can truncate the file.

Recommendation: rename. A partial file after a crash is worse than a label that has to be set again, and step 2 covers setups that need one.

Is losing ACLs and SELinux labels after an edit acceptable, with the step documented?

When an owner, a group or a move blocks the rename, should the edit write in place or fail?

Bind-mounted files, files owned by another user, and groups that cannot be set all save on main today, because main writes in place. The fallback brings back main's truncation risk, but only for those files.

How to decide:

1. Does the setup save on main today? If yes, failing the edit breaks a working deployment. Bind-mounted and other-owner files do.
2. Does the in-place write add risk beyond main? No. It is the write main makes, and its risk is a crash or a full disk during that write.
3. If you would rather break those setups than keep that risk, fail the edit instead.

Recommendation: keep the fallback, with the logged warning. Those setups keep working, and their risk is no worse than main.

Should these edits keep the in-place fallback, or fail?

A process killed during a write leaves a temporary file behind. Should Solr delete leftovers at startup?

Context: an edit that succeeds or fails with an error leaves no temporary file (the tests check this). Only a kill during the write leaves one, named security.json.<uuid>.tmp. In 75 of 100 kill trials it was left behind. It is as large as the file it was replacing, owner-only, and Solr never reads it.

How to decide:

1. Does the leftover hurt your deployment? If no, leave it.
2. If yes, can a startup sweep match only files with that exact name? A file with that name is one Solr wrote, so a sweep on that name cannot remove anything else. If yes, sweep at startup.
3. If a startup delete is out of scope for this change, leave the files and open a follow-up.

Recommendation: leave the files in this PR and add the sweep as a follow-up. A startup delete changes what Solr removes on boot, and it needs tests of its own. This PR changes the write path only.

Is leaving the leftovers now, with the sweep in a follow-up, acceptable?

Proof

The temp-file test fails on c34abaec94 and passes on 46869c4d8b (2026-10-23; production code unchanged since d8388c6519. The last commit, d742eb99dd, changes one line of a test file, an Error Prone fix).

• SecurityConfHandlerTest: 7 of 8 pass on Linux. The hard-link test skips on NFS and passes on local disk. The temp-file test fails on c34abaec94, where the temp file is 0644 while it is written. On main, the reader test and the concurrent-persist test fail.
• SecurityConfHandlerFailurePathsTest: 8 of 8 pass on Linux. It makes each failure step fail through a test file system that wraps the default one: the temp-file sync, the directory sync, the move, the permission copy, the owner check, and the group copy with its failure. Removing the temp-file sync, the owner check and the group copy from production fails 6 of the 8 tests. Removing the permission copy fails its test, and making a refused move throw fails its test.
• SecurityConfHandlerReaderTest: 1 of 1 on Linux. BasicAuthStandaloneTest: 2 of 2.
• Windows (build 26200, JDK 21.0.11) on d742eb99dd: 19 tests, 6 pass, 13 skipped, 0 failures, 0 errors. The skips are the POSIX-only tests: all 8 failure-path tests, 4 of 8 in SecurityConfHandlerTest, and the reader test. The other 6 pass, including the concurrency and edit tests. Windows writes in place, as on main.
• Outside the suite: kill -9 during 8 MB writes, 100 trials: 52 old, 48 new, 0 torn. In an earlier run with the same harness and window, main's in-place write gave 13 of 100 torn or empty. Disk full, with every inode in use or every byte in use: the edit fails, and the 1 MB file is unchanged and parses. Symlink: the edit writes through the link. Bind mount: the rename is refused, the in-place fallback writes, and the mounted file gets the new content.

Limits

Not covered: power loss, a real group change, and the symlink case.

• The group copy is tested with a fake group name in the test file system. My test host has one group, so a real group change on a multi-group host is not run here.
• Power loss is not tested. The kill -9 runs cannot show it, because the kernel keeps unsynced data after a process dies.
• The symlink case runs outside the suite. Solr's test security policy grants LinkPermission "hard" only. Leaving the policy as it is is the recommendation.
• SecurityConfHandlerReaderTest runs without Lucene's Windows-like test filesystem, which fails when a file is replaced while another handle is open. On NFS a read can fail with a stale handle during a replace. The test skips those reads, and a production reader gets an error, not partial content.
• Two processes sharing one SOLR_HOME are not coordinated, as with SOLR-18010: Fix security.json corruption from parallel edits #5065.
• Windows writes in place, as on main. The POSIX-only tests skip there (see Proof).

Changelog: changelog/unreleased/SOLR-18010-security-json-atomic-replace.yml

AI-generated description. Nick Shanin directed the work and takes responsibility for this contribution.

[nick-boss-tech]

I saw tests with Crave.io failed; that seems to be an unrelated flaky test, a PR for that is available:
#5098
I'll mark this PR ready to review after the other PR merges:
#5065