fix: restore column-vs-literal comparison in isGreaterThan/isLessThan family (#227)

[nikolauspschuetz]

Problem

isGreaterThanOrEqualTo (and the rest of the comparator family) regressed for column-vs-literal comparisons. This used to work:

check.isGreaterThanOrEqualTo("cluster_size", "1", hint="Cluster should have at least one element")

but now fails with the constraint message Input data does not include column 1! — the second operand is interpreted strictly as a column name. Reported in #227.

Root cause

This rode in with the bundled Deequ jar upgrade to 2.0.x (PyDeequ is pinned to com.amazon.deequ:deequ:2.0.8-... via pydeequ/configs.py). In Deequ 1.2.x the comparators built a plain SQL predicate ("<colA> >= <colB>") and passed no columns list, so a literal second operand worked. In Deequ 2.0.x the comparators pass columns = List(columnA, columnB), and Deequ validates that every entry is a real dataframe column — so literals fail. The PyDeequ wrappers themselves never changed; they just forward to the regressed Scala methods.

Fix

Route the whole comparator family (isLessThan, isLessThanOrEqualTo, isGreaterThan, isGreaterThanOrEqualTo) through Deequ's public satisfies(...) with an empty columns list — exactly the pre-2.0 behavior. Column-vs-column comparisons are unchanged; column-vs-literal/expression works again. Applied across the entire family for consistency.

Note: as with Deequ's own satisfies, columnB is treated as a Spark SQL expression, so string literals must be quoted by the caller (e.g. "'foo'"). This matches the original 1.2.x semantics.

Tests

Added to tests/test_checks.py:

• test_comparator_against_literal — column-vs-literal for all four comparators, expecting Success.
• test_fail_comparator_against_literal — a failing literal comparison, expecting Failure.

Validated against real Spark 3.5 / Deequ 2.0.8: the 2 new tests plus all 8 existing column-vs-column comparator tests pass (10 passed). CI will exercise the full pyspark 3.1/3.2/3.3/3.5 matrix.

Closes #227

[github-actions]

---

Generated by AI (model: us.anthropic.claude-opus-4-6-v1, prompt: 416310f3) — may not be fully accurate. Reply if this doesn't help.

[nikolauspschuetz]

Ready for review. Restores column-vs-literal comparison across the isGreaterThan*/isLessThan* family — a regression that rode in with the Deequ 2.0.x jar (it now routes through satisfies with an empty columns list, the pre-2.0 behavior). Validated locally on Spark 3.5: full test_checks.py → 88 passed. cc @sudsali @chenliu0831 — would appreciate your review. Closes #227.

[github-actions]

---

Generated by AI (model: us.anthropic.claude-opus-4-6-v1, prompt: 416310f3) — may not be fully accurate. Reply if this doesn't help.

[nikolauspschuetz]

Thanks for the automated review pass. The current revision addresses these findings:

• columnA quoting — the generated predicate backtick-quotes column A (`{columnA}` {operator} {columnB}), so column-A names with spaces/special characters or SQL reserved words stay valid.
• columnB left raw (intentional) — restoring the column-vs-literal usage from Regression in behavior of check comparator function isGreaterThanOrEqualTo #227 is the whole point: columnB may be a column name, a SQL literal, or a SQL expression, so quoting is the caller's responsibility — the same contract as Deequ's own satisfies. This is documented in the _column_comparison docstring.
• Default assertion — now routed through satisfies$default$3 (_ == 1.0), which is the correct default now that the comparator family genuinely goes through satisfies; it matches each comparator's own Deequ default.

Resolving the threads accordingly.

[github-actions]

This PR has been inactive for 60 days. It will be closed in 14 days if there is no further activity. If you are still working on this, please push an update or comment to keep it open.

[nikolauspschuetz]

rebased onto the latest master — still green and mergeable. this one restores the long-supported column-vs-literal usage in the isGreaterThan/isLessThan family (regressed in deequ 2.0.x), with tests. still relevant on my end and i'll keep it current; a review would be appreciated whenever there's bandwidth.

[github-actions]

No issues found.

---

Generated by AI (model: us.anthropic.claude-opus-4-8, prompt: 295226dd) — may not be fully accurate. Reply if this doesn't help.
Reviewed by Shadow · github.com/sudsali/shadow