feat: Refine dy.Categorical to support pl.Categories - #378
Draft
Daniel Elsner (delsner) wants to merge 5 commits into
Draft
feat: Refine dy.Categorical to support pl.Categories#378Daniel Elsner (delsner) wants to merge 5 commits into
dy.Categorical to support pl.Categories#378Daniel Elsner (delsner) wants to merge 5 commits into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #378 +/- ##
=========================================
Coverage 100.00% 100.00%
=========================================
Files 56 56
Lines 3507 3543 +36
=========================================
+ Hits 3507 3543 +36 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
dy.Categorical to support pl.Categories
Daniel Elsner (delsner)
marked this pull request as ready for review
July 29, 2026 15:22
Daniel Elsner (delsner)
requested review from
Andreas Albert (AndreasAlbertQC) and
Oliver Borchert (borchero)
as code owners
July 29, 2026 15:22
Daniel Elsner (delsner)
requested a review
from Moritz Potthoff (MoritzPotthoffQC)
July 29, 2026 15:26
Contributor
There was a problem hiding this comment.
Pull request overview
This PR extends dy.Categorical so callers can specify Polars-style global category metadata (name/namespace and physical index type) via a new serializable dy.Categories helper, enabling more explicit and potentially more efficient categorical representations.
Changes:
- Introduces
dy.Categoriesand adds acategoriesparameter tody.Categorical, mapping topl.Categorieswhen constructing the Polars dtype. - Extends column serialization/deserialization to roundtrip the new categorical metadata.
- Adds unit tests covering dtype construction, equality/matching, and schema serialization roundtrips for categorical columns.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/schema/test_serialization.py | Adds categorical schemas to the general schema serialization roundtrip parametrization. |
| tests/column_types/test_categorical.py | New tests for categorical dtype mapping, matching, dict roundtrip, and schema serialization roundtrip. |
| dataframely/columns/categorical.py | Adds Categories dataclass, pl.Categories dtype support, and serialization hooks for categorical metadata. |
| dataframely/columns/init.py | Re-exports Categories from the columns package. |
| dataframely/init.py | Exposes Categories at the top-level dataframely API. |
Oliver Borchert (borchero)
marked this pull request as draft
July 29, 2026 23:58
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Polars allows setting name, namespace and physical backing type for categoricals. Right now, dataframely users cannot specify these parameters, even though they're relevant for efficient categorical representation.
Changes
Add support for
pl.Categories. I decided to create our own data structure sincepl.Categoriesisn't serializable out of the box.