Repository navigation
Conversation
Perseus
force-pushed
the
anirudh.singh/iceberg-dyn-filtering-2
branch
5 times, most recently
from
November 6, 2026 13:36
6bdc438 to
58786d0
Compare
Perseus
commented
Nov 7, 2026
Perseus
commented
Nov 7, 2026
| @@ -18,6 +18,8 @@ extensions_options! { | |||
| pub column_stats_enabled: bool, default = false | |||
| /// Discover and retain file tasks during physical planning instead of execution. | |||
| pub eager_file_planning: bool, default = false | |||
| /// (Experimental) Whether to enable dynamic filtering. | |||
| pub dynamic_filtering_enabled: bool, default = false | |||
Contributor
Author
There was a problem hiding this comment.
This was added so that a rolling deploy for workers doesn't cause errors if the coordinator starts producing dynamic filters that the workers don't consume yet - I'm not sure if that's necessarily an issue in practice and this adds something to the public API of this datasource that might not be easily removed later, so I'm open to changing the approach to something better
Perseus
commented
Nov 7, 2026
Contributor
Author
There was a problem hiding this comment.
These tests are largely just checking that the plumbing works since the filters don't do anything yet
Perseus
marked this pull request as ready for review
November 7, 2026 17:23
DataFusion offers each scan the filters above it, including the dynamic filters that joins, top-N sorts and aggregates fill in while the query runs. Iceberg scans didn't keep them. Keeping them is the first step toward skipping files with them. Both scans now keep every pushed filter, ANDed into one predicate. They answer that they don't apply them, so DataFusion still checks every row. EXPLAIN shows the dynamic ones on the scan as dynamic_filters=, and the codecs carry all of them to workers. With iceberg.dynamic_filtering_enabled on, scans also report the filters through apply_expressions. Joins and aggregates fill in a dynamic filter only when a scan reports it, and the coordinator sends updates to the workers holding those scans. The setting is off by default, and the plan carries it to workers. Enable it only once every worker supports it: an older worker can fail a query whose scan reports a dynamic filter.
…ltering is on With iceberg.dynamic_filtering_enabled off, scans aren't consumers of the dynamic filters pushed into them, so those filters stay empty. Plans no longer print them in that case.
Perseus
force-pushed
the
anirudh.singh/iceberg-dyn-filtering-2
branch
from
November 10, 2026 10:21
d927948 to
cab7f00
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
This is part of a stack of PRs that implement dynamic filtering for Iceberg. The previous PR can be found here - #850
Relevant changes will also be ported to datafusion-iceberg.
Summary
This PR adds support for accepting and storing expressions, including dynamic filters, in both Iceberg datasources (Eager and Lazy).
Based on a config,
iceberg.dynamic_filtering_enabled, operators likeHashJoinExecare also informed that the datasource is a consumer of these dynamic filters.The dynamic filters also show up in
EXPLAIN ANALYZEplans in the DataSource in the following format:These filters are not yet used for any optimizations - they're only stored and reported.
Future PRs will use them to improve data pruning in the following order -
PhysicalExprtoiceberg::PredicateFileScanTasks