feat: add Distance analyzer for categorical feature drift (L-infinity, chi-squared) (#164)

[nikolauspschuetz]

Problem

Deequ's Distance analyzer (com/amazon/deequ/analyzers/Distance.scala) computes distribution distance for feature-drift detection (L-infinity and chi-squared), but it was not exposed in PyDeequ. Requested in #164 (labeled good first issue / help wanted / feature request).

What this adds

A Distance helper plus a CategoricalDistanceMethod enum exposing categorical distance:

from pydeequ.analyzers import Distance, CategoricalDistanceMethod

dist = Distance(spark)
# distribution dicts as produced by the Histogram analyzer: {category: count}
d = dist.categoricalDistance({"a": 5, "b": 5}, {"a": 10},
                             method=CategoricalDistanceMethod.LInfinity)

Supports both L-infinity and chi-squared, with the full alpha / Yates / Cochran parameters.

Design note (please scrutinize)

Deequ's Distance is a plain object with static-style methods, not an Analyzer subclass, so it cannot go through AnalysisRunBuilder.addAnalyzer(...).run(). I exposed it faithfully as a helper rather than faking analyzer integration.

categoricalDistance requires a Scala mutable.Map[String, Long]. py4j auto-unboxes individual java.lang.Long back to Python ints (re-entering as Integer), which makes Deequ's e._2.toDouble throw ClassCastException. To keep values genuinely Long-typed, each dict is round-tripped through a one-row Spark DataFrame with a MapType(StringType, LongType) column and collected JVM-side. Tradeoff: a small Spark job per distribution — acceptable for a drift helper, and the only reliable path found. Open to a lighter approach if reviewers prefer one.

Scope: categorical distance only. numericalDistance requires constructing a JVM QuantileNonSample[Double] with no clean Python path and is intentionally left out (matches the issue).

Tests

Added to tests/test_analyzers.py: test_Distance_categorical_LInfinity and test_Distance_categorical_Chisquare. Validated against the live Deequ 2.0.8 jar on Spark 3.5 (2 passed; full analyzer suite 27 passed / 22 pre-existing xfails, no regressions). Docs updated in docs/analyzers.md.

Closes #164

[github-actions]

---

Generated by AI (model: us.anthropic.claude-opus-4-6-v1, prompt: 416310f3) — may not be fully accurate. Reply if this doesn't help.

[nikolauspschuetz]

Ready for review. Adds a Distance helper (categorical L-infinity + chi-squared) for feature-drift detection, exposing Deequ's Distance object faithfully (it's not an Analyzer, so it's a helper rather than addAnalyzer). Validated against the live Deequ 2.0.8 jar on Spark 3.5: test_analyzers.py → 27 passed / 22 xfailed. See the PR body for a reviewer note on the py4j Long-boxing handling. cc @sudsali @chenliu0831 @rdsharma26 — would appreciate your review. Closes #164.

[github-actions]

---

Generated by AI (model: us.anthropic.claude-opus-4-6-v1, prompt: 416310f3) — may not be fully accurate. Reply if this doesn't help.

[github-actions]

No issues found.

---

Generated by AI (model: us.anthropic.claude-opus-4-6-v1, prompt: 416310f3) — may not be fully accurate. Reply if this doesn't help.

[nikolauspschuetz]

Thanks for the automated review pass. The current revision addresses these findings:

• Map construction — _to_scala_mutable_long_map no longer uses createDataFrame/collectAsList. It builds the mutable.HashMap[String, Long] from JVM Long[] arrays via genericWrapArray/zip/toMap, so there is no per-call Spark job and no reliance on a java.util.Map -> Scala implicit conversion.
• Empty distributions — guarded: categoricalDistance raises ValueError if either distribution dict is empty.
• correctForLowNumberOfSamples docstring — matches Deequ semantics: True returns the raw/unscaled statistic, False returns the normalized (KS-corrected L-infinity / chi-squared) result.
• Seq.canBuildFrom (Scala 2.12) — documented as a known constraint in the method docstring; all supported Spark builds (3.1-3.5) ship Scala 2.12.
• Tests — added test_Distance_categorical_single_category, test_Distance_categorical_empty_dict_raises, and test_Distance_categorical_invalid_method_raises (plus an alpha test) covering the edge/error paths.

Resolving the threads accordingly.

[github-actions]

This PR has been inactive for 60 days. It will be closed in 14 days if there is no further activity. If you are still working on this, please push an update or comment to keep it open.

[nikolauspschuetz]

rebased onto the latest master — green and mergeable. adds a Distance analyzer wrapper for categorical feature drift (L-infinity, chi-squared). still relevant on my end and happy to keep it current; would appreciate a review when there's bandwidth.

[github-actions]

No issues found.

---

Generated by AI (model: us.anthropic.claude-opus-4-8, prompt: 295226dd) — may not be fully accurate. Reply if this doesn't help.
Reviewed by Shadow · github.com/sudsali/shadow