GH-3394: Reuse listed file status during split planning#3674
Open
dossett wants to merge 1 commit into
Open
Conversation
dossett
force-pushed
the
agent-reuse-file-status-for-splits
branch
from
July 17, 2026 21:04
6ba4e37 to
5aac156
Compare
Contributor
Author
|
cc @wangyum @steveloughran and @wgtmac who were involved in the prior PR. |
dossett
marked this pull request as ready for review
July 17, 2026 21:27
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I used CODEX to analyze this problem and create this PR. I've reviewed the code and tests and stand by them. This summary is written completely by a human (me) other than very light copy editing by an LLM.
This change is inspired by #3395, which has gone idle.
In general, caching the metadata of
Footers is reasonable. Somewhat confusingly the key to this cache is aFileStatusWrapperobject which containsFileStatusinformation, which can be useful. But it's fairly mutable so shouldn't actually be relied on.In the case of file listing and split generation it would be nice to retain the FileStatus between those stages since that can be done reliably. So this PR lumps the FileStatus and Footer together in a new type that use the
List<Footer>offooters.It's a little confusing and a much deeper refactor could certainly be considered, but this seems worthwhile. Saving one namenode rpc call or cloud object store metadata request per file can really add up.
CODEX generated summary follows:
Parquet's client-side split planning already obtains a
FileStatusfor every input while listing files. It later discards that status and callsgetFileStatus()again for each file before calculating block-aware splits. That adds one sequential NameNode RPC or remote metadata lookup per file, increasing job-startup latency and metadata-service load without changing the result.This change carries the status obtained during listing internally through footer processing and reuses it during split creation. It leaves
Footer's public API unchanged and preserves the existing lookup as a fallback for caller-created footers. The optimized path covers direct footer reads, footer-cache hits, summary metadata, and deprecated split planning.This addresses the same underlying problem as #3395 while covering the cache and summary-file paths and avoiding a public nullable
FileStatusAPI.Closes #3394.
Implementation
FileStatusobjects already supplied to footer reading using a private internal carrier.getFileStatus()when none is available.Testing
mvn -pl parquet-hadoop -Dtest=TestInputFormat,TestInputFormatFileStatusReuse,DeprecatedInputFormatTest,org.apache.parquet.hadoop.example.TestInputOutputFormat test(35 tests)mvn -pl parquet-hadoop test(740 tests, 24 skipped)mvn -pl parquet-hadoop -DskipTests spotless:check