Repository navigation
fix: raise the executor's open file limit to the hard limit at startup - #2794
Merged
Merged
Conversation
andygrove
force-pushed
the
fix-executor-nofile-limit
branch
from
November 7, 2026 17:12
6f64753 to
a46e77b
Compare
A task holds a file per shuffle partition it reads or writes plus its spill files, so stages with a few hundred partitions need thousands of descriptors. With the common 1024 soft limit, TPC-H at 128+ partitions fails with "Too many open files". The JVM raises its soft limit at startup, so Spark executors are unaffected; do the same.
andygrove
force-pushed
the
fix-executor-nofile-limit
branch
from
November 7, 2026 17:57
a46e77b to
52c3136
Compare
andygrove
marked this pull request as ready for review
November 8, 2026 09:01
Member
Author
|
Thanks @milenkovicm |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
No single issue. Split out of #2792. Related: #2520 (performance review tracking).
Rationale for this change
A task holds a file per shuffle partition it reads or writes, plus its spill files, so a stage with a few hundred partitions needs thousands of descriptors per executor. With the common soft limit of 1024, TPC-H at 128 or more partitions fails with
Too many open files. The JVM raises its soft limit to the hard limit at startup, so Spark executors never see this.What changes are included in this PR?
A public
raise_open_file_limit()raisesRLIMIT_NOFILE's soft limit to the hard limit, capped at 2^20 descriptors (10240 on macOS, which rejects a higher soft limit even when the hard limit is unlimited). The cap keeps a container's billion-descriptor hard limit from making every close-all-descriptors loop in a child process crawl. It logs the old and new values and does nothing when the soft limit is already at the target. Unix only.The
ballista-executorbinary calls it after logging is set up.start_executor_processdoes not, because the limit is process-wide and the function is also the entry point for embedders; they callraise_open_file_limit()themselves if they want it. The standalone (in-process) executor is unchanged for the same reason.The test only checks that the call never lowers the limit: the limit is process-wide, so a stronger check would change it for every other test in the binary.
Are there any user-facing changes?
The executor binary starts with a higher open-file soft limit.
raise_open_file_limitis a new public function.