Filter pushdown on float columns drops rows with NaN or -0.0 #101

Describe the bug

Some filters on double and float columns return fewer rows from an Iceberg table than DataFusion returns for the same data. The missing rows are the ones that hold NaN or -0.0. The query succeeds, but those rows aren't in the result.

The two sides compare these values differently:

• DataFusion treats NaN as equal to itself and greater than any other value, and treats -0.0 as equal to 0.0. SELECT CAST('NaN' AS DOUBLE) > 1.0, CAST('NaN' AS DOUBLE) = CAST('NaN' AS DOUBLE), CAST('-0.0' AS DOUBLE) = 0.0 returns true for all three.
• iceberg-rust's InclusiveMetricsEvaluator decides that a data file holding only NaN can't match >, >=, = or IN, and NaN is left out of a file's upper bound. iceberg-rust also orders floats with total_cmp, so to it -0.0 and 0.0 are different values.

scalar_value_to_datum passes float literals through as Datum::double, and the comparison reaches the scan unchanged. So the scan skips files that contain rows DataFusion would match. Those files are never read, and filtering again in DataFusion can't bring the rows back.

To Reproduce

Using the sqllogictest engine, with one INSERT per data file:

CREATE TABLE default.default.t (id INT NOT NULL, x DOUBLE);
INSERT INTO default.default.t VALUES (1, CAST('NaN' AS DOUBLE));
INSERT INTO default.default.t VALUES (2, 0.5), (3, CAST('NaN' AS DOUBLE));
INSERT INTO default.default.t VALUES (4, 2.0);
INSERT INTO default.default.t VALUES (5, CAST('-0.0' AS DOUBLE));
INSERT INTO default.default.t VALUES (6, 0.0);
CREATE TABLE m AS SELECT * FROM default.default.t;

SELECT id FROM <table> WHERE <filter> returns:

The same table returns 1, 3, 4 for x + 0 > 1.0, which isn't pushed down.

Which rows go missing also depends on how they're laid out in files. Row 3 is skipped because its file's upper bound is 0.5. The same NaN in a file with 2.0 comes back.

Expected behavior

t returns the same rows as m.

I can think of two ways to get there:

1. Stop pushing down comparisons and IN lists that have a float literal. That's the simplest, but float columns lose file pruning.
2. Push a looser predicate that still covers every row DataFusion could match, and leave the exact check to DataFusion. For example, x > c becomes x > c OR x IS NAN and x = NaN becomes x IS NAN. Zero literals would cover both signs: x = 0.0 becomes x = -0.0 OR x = 0.0, x >= 0.0 becomes x >= -0.0, and x <= -0.0 becomes x <= 0.0.

Let me know which you'd prefer.

Additional context

• <, !=, NOT IN, isnan(x) and NOT isnan(x) return the same rows on both sides.
• NOT (x <= 1.0) hits this too, since it's simplified to x > 1.0 before pushdown.
• REAL columns are affected when the literal is also REAL, for example x > CAST(1 AS REAL). With a DOUBLE literal the column gets a cast, and since fix: refuse to look inside casts in expr_to_predicate #91 that comparison isn't pushed down.
• iceberg-rust main has the same NaN handling and float ordering, so updating the iceberg-rust dependency won't fix this.
• This is the same kind of problem as Filter pushdown strips casts on columns, which drops matching rows or fails valid queries #36 and convert_filters_to_predicate negates a weakened AND, producing a predicate that drops matching rows #44: the pushed predicate rejects rows that the filter keeps.