Skip to content

feat: Refine dy.Categorical to support pl.Categories - #378

Draft
Daniel Elsner (delsner) wants to merge 5 commits into
mainfrom
refine-categoricals
Draft

feat: Refine dy.Categorical to support pl.Categories#378
Daniel Elsner (delsner) wants to merge 5 commits into
mainfrom
refine-categoricals

Conversation

@delsner

@delsner Daniel Elsner (delsner) commented Jul 29, 2026

Copy link
Copy Markdown
Member

Motivation

Polars allows setting name, namespace and physical backing type for categoricals. Right now, dataframely users cannot specify these parameters, even though they're relevant for efficient categorical representation.

Changes

Add support for pl.Categories. I decided to create our own data structure since pl.Categories isn't serializable out of the box.

@github-actions github-actions Bot added the enhancement New feature or request label Jul 29, 2026
@codecov

codecov Bot commented Jul 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (febf6ce) to head (6267c85).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff            @@
##              main      #378   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           56        56           
  Lines         3507      3543   +36     
=========================================
+ Hits          3507      3543   +36     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@delsner Daniel Elsner (delsner) changed the title feat: Refine dy.Categorical to support pl.Categories feat: Refine dy.Categorical to support pl.Categories Jul 29, 2026
@delsner
Daniel Elsner (delsner) marked this pull request as ready for review July 29, 2026 15:22
Copilot AI review requested due to automatic review settings July 29, 2026 15:22

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR extends dy.Categorical so callers can specify Polars-style global category metadata (name/namespace and physical index type) via a new serializable dy.Categories helper, enabling more explicit and potentially more efficient categorical representations.

Changes:

  • Introduces dy.Categories and adds a categories parameter to dy.Categorical, mapping to pl.Categories when constructing the Polars dtype.
  • Extends column serialization/deserialization to roundtrip the new categorical metadata.
  • Adds unit tests covering dtype construction, equality/matching, and schema serialization roundtrips for categorical columns.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
tests/schema/test_serialization.py Adds categorical schemas to the general schema serialization roundtrip parametrization.
tests/column_types/test_categorical.py New tests for categorical dtype mapping, matching, dict roundtrip, and schema serialization roundtrip.
dataframely/columns/categorical.py Adds Categories dataclass, pl.Categories dtype support, and serialization hooks for categorical metadata.
dataframely/columns/init.py Re-exports Categories from the columns package.
dataframely/init.py Exposes Categories at the top-level dataframely API.

Comment thread dataframely/columns/categorical.py Outdated
@borchero Oliver Borchert (borchero) added this to the v3 milestone Jul 29, 2026
@borchero
Oliver Borchert (borchero) marked this pull request as draft July 29, 2026 23:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants