Skip to content

feat: OHE min_frequency knob to collapse rare categories - #68

Merged
rxavier merged 1 commit into
mainfrom
feat/preprocessing-p1-ohe-min-frequency
Aug 3, 2026
Merged

rxavier merged 1 commit into
mainfrom
feat/preprocessing-p1-ohe-min-frequency

Conversation

@rxavier

@rxavier rxavier commented Aug 3, 2026

Copy link
Copy Markdown
Owner

Summary

Implements P1-5 from the ML defaults improvements plan (docs/ml-defaults-improvements.md).

Changes

  • Added ohe_min_frequency parameter to PoniardPreprocessor (default 5).
    • int: absolute category count; float: fraction of samples; None: keep every category.
  • Default OHE is now OneHotEncoder(drop="if_binary", handle_unknown="ignore", sparse_output=False, min_frequency=5).
  • Rare categories collapse into sklearn's infrequent_sklearn bucket, composing correctly with handle_unknown="ignore" (unseen categories still map to all-zeros).
  • Disabled the knob in the two existing tests that assert exact OHE output on 5-row toy data (where every category is rarer than 5), so they keep testing their intended behaviors.
  • Documented the new default in the preprocessing guide.

Tests

  • Rare categories collapse into the infrequent column; None preserves every category; float thresholds work as fractions.

All 251 tests pass; coverage 89.4% (gate 85%); ruff + format clean.

- Add ohe_min_frequency (default 5) to PoniardPreprocessor; int is an
  absolute count, float is a fraction of samples, None keeps every category
- Rare categories collapse into sklearn's infrequent bucket, composing with
  the existing handle_unknown='ignore'
- Disable the knob in the two existing tests that assert exact OHE output on
  5-row toy data; document the default in the preprocessing guide

Closes P1-5 of the ML defaults improvements plan.
@rxavier
rxavier merged commit 64b2e82 into main Aug 3, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant