Repository navigation
Conversation
Perseus
force-pushed
the
anirudh.singh/iceberg-dyn-filtering-3
branch
from
November 9, 2026 21:11
6bfdfaf to
52659af
Compare
DataFusion offers each scan the filters above it, including the dynamic filters that joins, top-N sorts and aggregates fill in while the query runs. Iceberg scans didn't keep them. Keeping them is the first step toward skipping files with them. Both scans now keep every pushed filter, ANDed into one predicate. They answer that they don't apply them, so DataFusion still checks every row. EXPLAIN shows the dynamic ones on the scan as dynamic_filters=, and the codecs carry all of them to workers. With iceberg.dynamic_filtering_enabled on, scans also report the filters through apply_expressions. Joins and aggregates fill in a dynamic filter only when a scan reports it, and the coordinator sends updates to the workers holding those scans. The setting is off by default, and the plan carries it to workers. Enable it only once every worker supports it: an older worker can fail a query whose scan reports a dynamic filter.
…ltering is on With iceberg.dynamic_filtering_enabled off, scans aren't consumers of the dynamic filters pushed into them, so those filters stay empty. Plans no longer print them in that case.
Adds a translator from a DataFusion physical filter to an iceberg predicate. It handles comparisons with a value, IN lists, IS NULL, LIKE prefixes, isnan, starts_with, AND, OR, NOT and CASE. It also handles the filters hash joins build while the query runs: IN lists, struct(a, b) IN (...) for joins on several columns, and the CASE a partitioned join builds with one branch per partition. The predicate matches every row the filter keeps, and can match more, so the filter still runs on every row. An AND can drop a side it can't convert. A NOT is converted only when its inside was converted fully. Floats and times aren't converted, and neither are LIKE prefixes with non-ASCII text. iceberg-rust handles these in ways that could skip matching rows or fail the read. Nothing calls it yet. Partition skipping will use it next.
Perseus
force-pushed
the
anirudh.singh/iceberg-dyn-filtering-3
branch
from
November 10, 2026 10:22
52659af to
ef70cff
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
This is part of a stack of PRs that implement dynamic filtering for Iceberg.
The previous PRs can be found here - #850 , #855
Please review this PR starting at dbd4c77
Relevant changes will also be ported to datafusion-iceberg.
Summary
This PR
PhysicalExpr->Iceberg::PredicatetranslationSome of the code in this PR duplicates functionality from
iceberg-rustdue to those APIs not being public. I will raise PRs/issues in the repo to make them public so that the code can be deleted from here eventually'Notes