Skip to content

GH-3394: Reuse listed file status during split planning#3674

Open
dossett wants to merge 1 commit into
apache:masterfrom
dossett:agent-reuse-file-status-for-splits
Open

GH-3394: Reuse listed file status during split planning#3674
dossett wants to merge 1 commit into
apache:masterfrom
dossett:agent-reuse-file-status-for-splits

Conversation

@dossett

@dossett dossett commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

I used CODEX to analyze this problem and create this PR. I've reviewed the code and tests and stand by them. This summary is written completely by a human (me) other than very light copy editing by an LLM.

This change is inspired by #3395, which has gone idle.

In general, caching the metadata of Footers is reasonable. Somewhat confusingly the key to this cache is a FileStatusWrapper object which contains FileStatus information, which can be useful. But it's fairly mutable so shouldn't actually be relied on.

In the case of file listing and split generation it would be nice to retain the FileStatus between those stages since that can be done reliably. So this PR lumps the FileStatus and Footer together in a new type that use the List<Footer> of footers.

It's a little confusing and a much deeper refactor could certainly be considered, but this seems worthwhile. Saving one namenode rpc call or cloud object store metadata request per file can really add up.

CODEX generated summary follows:

Parquet's client-side split planning already obtains a FileStatus for every input while listing files. It later discards that status and calls getFileStatus() again for each file before calculating block-aware splits. That adds one sequential NameNode RPC or remote metadata lookup per file, increasing job-startup latency and metadata-service load without changing the result.

This change carries the status obtained during listing internally through footer processing and reuses it during split creation. It leaves Footer's public API unchanged and preserves the existing lookup as a fallback for caller-created footers. The optimized path covers direct footer reads, footer-cache hits, summary metadata, and deprecated split planning.

This addresses the same underlying problem as #3395 while covering the cache and summary-file paths and avoiding a public nullable FileStatus API.

Closes #3394.

Implementation

  • Associate footer results with the FileStatus objects already supplied to footer reading using a private internal carrier.
  • Attach the current listed status when returning a cached footer rather than relying on cached status identity.
  • Reuse the associated status during split generation and fall back to getFileStatus() when none is available.

Testing

  • mvn -pl parquet-hadoop -Dtest=TestInputFormat,TestInputFormatFileStatusReuse,DeprecatedInputFormatTest,org.apache.parquet.hadoop.example.TestInputOutputFormat test (35 tests)
  • mvn -pl parquet-hadoop test (740 tests, 24 skipped)
  • mvn -pl parquet-hadoop -DskipTests spotless:check

@dossett
dossett force-pushed the agent-reuse-file-status-for-splits branch from 6ba4e37 to 5aac156 Compare July 17, 2026 21:04
@dossett

dossett commented Jul 17, 2026

Copy link
Copy Markdown
Contributor Author

cc @wangyum @steveloughran and @wgtmac who were involved in the prior PR.

@dossett
dossett marked this pull request as ready for review July 17, 2026 21:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cache FileStatus in Footer to reduce redundant NameNode RPC calls

1 participant