[TEST] Adding Sparse Triangle to Unittest by henrydingliu · Pull Request #1157 · casact/chainladder-python

henrydingliu · 2026-07-26T05:03:32Z

Summary of Changes

Fixing test fixture to supply actual sparse triangles
sped up numpy conversion of prism in test_triangle
resolved failing tests in test_development
added numpy conversion in learning to satisfy skl requirement that sample_weight is dense
resolved failing test in cape_cod
added backend conversion in mack for sparse compatibility
added nanquantile and nanmedian to sparse
added new tests for nanquantile
added nan to nan_to_num in sparse

Related GitHub Issue(s)

fixes #1155

Additional Context for Reviewers

I passed tests locally for both code (uv run pytest) and documentation changes (uv run jb build docs --builder=custom --custom-builder=doctest)

Note

Medium Risk
Changes Mack reserve uncertainty recursion and sparse reduction helpers used across triangles; scope is mostly backend alignment and test coverage, but Mack and sparse quantile paths affect numerical results on sparse data.

Overview
This PR fixes sparse-backed triangle testing (issue #1155) by changing how fixtures supply backends and aligning code paths when sparse triangles meet numpy-only intermediates.

Test infrastructure: conftest no longer toggles global ARRAY_BACKEND; fixtures yield triangles via set_backend("sparse") or numpy per parametrization. The summed-prism fixture is renamed monthly; prism tests default to sparse-only runs.

Sparse utilities: Adds nanquantile / nanmedian for COO arrays and extends nan_to_num with a configurable NaN replacement—supporting quantile/median-style ops on sparse data.

Runtime fixes: DevelopmentML._prep_w_ml forces dense numpy/cupy weights for sklearn. MackChainladder re-applies set_backend before sums and summary_ concatenation so auto-sparse conversions don’t mix backends. Development test helpers densify sparse triangles before weighted regression.

Tests: Assertions increasingly use Triangle equality instead of raw array module tests; grain/auto_sparse tests are adjusted for real sparse inputs and a smaller prism slice for numpy auto_sparse checks.

^{Reviewed by Cursor Bugbot for commit 9445d09. Bugbot is set up for automated code reviews on this repo. Configure here.}

expecting errors

henrydingliu · 2026-07-26T05:06:21Z

@genedan can you take care of the 6 failing tests in test_slicing?

github-actions · 2026-07-26T05:07:40Z

Pyright Type Completeness

View the full pyright --verifytypes output for this commit

Project (full chainladder package, at this PR's head): 15.0% of exported symbols fully typed (196 / 1305)

	Known	Ambiguous	Unknown	Total
Project (head)	196	110	999	1305

Other symbols referenced but not exported by chainladder: 13

	Known	Ambiguous	Unknown	Total
Other (head)	3	1	9	13

Symbols without documentation:

Functions without docstring: 315
Functions without default param: 0
Classes without docstring: 10

Patch (exported symbols added or changed by this PR): 28.6% fully typed (2 / 7); 2 no longer exported

	Known	Ambiguous	Unknown	Total
Patch	2	0	5	7

Patch symbol details

Symbol	Status	Change
`chainladder.utils.sparse.nanmedian`	❌ unknown	new
`chainladder.utils.sparse.nanquantile`	❌ unknown	new
`chainladder.utils.tests.test_sparse.test_1D_nanquantile`	✅ known	new
`chainladder.utils.tests.test_sparse.test_keepdims_nanquantile`	✅ known	new
`chainladder.utils.tests.test_utilities.test_maximum_2`	❌ unknown	new
`chainladder.utils.tests.test_utilities.test_maximum_minimum_1`	❌ unknown	new
`chainladder.utils.tests.test_utilities.test_minimum_2`	❌ unknown	new
`chainladder.utils.tests.test_utilities.test_maximum`	—	no longer exported (was ❌ unknown)
`chainladder.utils.tests.test_utilities.test_minimum`	—	no longer exported (was ❌ unknown)

* fixing fixture expecting errors * fixing prism * trying another approach * some fixes * more fixes * more fixes * fix for learning * more learning fix * restoring test still fails

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 7c8e5fe. Configure here.}

henrydingliu · 2026-07-27T04:59:14Z

@genedan there are two more tests in test_triangle that you authored. i'm not sure if those tests should be sparse-compatible. could you please take a look?

* further improving prism_convert * capecod & mack fixes * fixing test_utilities and removing sparse.testing * more fixes * implementing nanquantile and nanmedian * improving comment, adding tests, fixing bugbot * fixing test * removing empty check

* Release prep * Updated release notes * feat(core): add index/columns/origin/development alternatives to Triangle.drop() Route the keyword alternatives to their axis, mirroring pandas. Refs #1060. * docs(core): add drop() examples and wrap TriangleProtocol.drop signature Address review on #1131: format the long drop() signature one arg per line and add testable Examples to the docstring. * fix(core): handle integer labels in Triangle.drop() Normalize any scalar (str or int) into a list so a bare int routes to its axis instead of raising TypeError. Addresses Bugbot review on #1131. * fix(core): preserve list-like handling in Triangle.drop() Use np.isscalar so pd.Index/ndarray labels pass through list() while scalars (str/int) are wrapped, and a bare drop() still raises. Addresses Bugbot review on #1131. * Sync from main, added a few more updates * docs(core): show original columns in drop() example Print the triangle's original columns before the drop, per review on #1131. * Added release date * Bump uv * Apply lognormal fix for apriori draws and add tests * Added sphinx-book-theme dependency * Added contributors full names * Added contributor fullnames and missing tickets * Tighten sphinx-book-theme version * bug bot fix * Fix documentation build warnings and cross-references * Restore local Sphinx extension path * Fix Development reference in user guide * Restore user guide section links. * Fix Triangle display crash when stored in a DataFrame cell (GH #142) A Triangle is a 4-D container but pandas' is_sequence() misclassified it as a 1-D sequence because the legacy __getitem__ iteration protocol plus __len__ made iter() succeed. Formatting a DataFrame that holds a Triangle in a cell then iterated the Triangle and raised, crashing both the text and HTML reprs. Declare Triangle non-iterable via __iter__ = None so pandas falls back to str(triangle) and renders the summary. Add regression tests. * Remove dead backend-reset code in _prep_columns (#1045) * FEAT: Explicitly forbid in-place assignment on sparse array. * FIX: Fix bug in column assignment with a sparse backend. * FEAT: Enable Triangle __array__ for sparse backend. * TEST: Fix tests. * FIX: Apply bugbot fix. * TST: Add concurrency guard. --------- Co-authored-by: Kenneth Hsu <[email protected]> Co-authored-by: priyam0k <[email protected]> Co-authored-by: Howard Friman <[email protected]> Co-authored-by: Anonymous <[email protected]> Co-authored-by: Gene Dan <[email protected]>

updating again

codecov · 2026-07-29T22:20:44Z

Codecov Report

❌ Patch coverage is 98.00000% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 91.14%. Comparing base (f48bd83) to head (9445d09).

Files with missing lines	Patch %	Lines
chainladder/utils/sparse.py	96.66%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1157      +/-   ##
==========================================
- Coverage   91.27%   91.14%   -0.13%     
==========================================
  Files          91       91              
  Lines        5422     5365      -57     
  Branches      691      681      -10     
==========================================
- Hits         4949     4890      -59     
- Misses        339      340       +1     
- Partials      134      135       +1

Flag	Coverage Δ
unittests	`91.14% <98.00%> (-0.12%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

genedan

Minor comments/questions and nitpicks on spacing.

genedan · 2026-07-31T01:02:59Z

-            origins = X.age_to_age.values.shape[2]
-            reg_x = X.incr_to_cum().values[...,:origins,:-1]
-            reg_y = X.incr_to_cum().values[...,:origins,1:]
+            #following precedent set by _set_fit_groups() from DevelopmentBase to force triangle to be dense


Would there be a use case where we'd need the triangle to sparse?

no, this was one of the main points behind why i think our long term strategy for the package is to abandon sparse altogether. sparse has never been used in estimating parameters, because there is zero point of doing so. a sparse triangle is by default a single claim. the age_to_age on a single claim is meaningless. ldf_ is always dense.

genedan · 2026-07-31T01:03:47Z

-    prism : Triangle
-        The prism sample data set Triangle.
+    prism_convert : Triangle
+        The prism sample data set Triangle, reduced for test speed.


It makes sense from this docstring why the new fixture was created, a comment in the corresponding definition in conftest.py would be helpful.

i'm actually gonna get rid of prism_convert, and follow your logic from test_auto_sparse_converts_numpy_to_sparse

genedan · 2026-07-31T01:11:35Z



-def nan_to_num(a):
+def nan_to_num(a,nan=0.0):


space after the comma

genedan · 2026-07-31T01:17:38Z

+    out : COO
+        result cast back to COO for consistency.
+    """
+    return nanquantile(a,0.5,axis,keepdims)


genedan · 2026-07-31T01:19:31Z

+
+    return COO(out)
+
+def nanmedian(a:COO, axis:int=0, keepdims:bool=False):


spaces after the colons and around the equal signs, e.g., keepdims: bool = False

genedan · 2026-07-31T01:28:01Z

    yield from _sample_fixture(request, "prism")

+@pytest.fixture
+def prism_convert(request):


Now that we have 3 different prism fixtures, I wonder if we can adjust the naming convention to prevent confusion and let people know the differences between them. Was prism_convert created as a version that would auto-sparsify, whereas prism_dense is one that always stays dense?

If so, maybe something like:

prism_auto_sparse (A dense version of prism above 30 MB that will auto sparsify after aggregation)

prism_always_dense (A dense version of prism below 30 MB that will always stay dense)

With accompanying docstrings to explain their creation - i.e., we have to do some amount of truncation or aggregation at definition time to prevent the fixture from eating up all the RAM.

prism_dense is actually only used for its monthly grain. renaming it to monthly instead.

…nladder-python into sparse_fixture

Sparse fixture

henrydingliu · 2026-07-31T03:04:54Z

@genedan thanks for the review. i tried to address all comments.

henrydingliu added 3 commits July 25, 2026 22:01

fixing fixture

661d2bb

expecting errors

fixing prism

5f6fa5a

trying another approach

29e1be1

Sparse test fixture (#16)

2718584

* fixing fixture expecting errors * fixing prism * trying another approach * some fixes * more fixes * more fixes * fix for learning * more learning fix * restoring test still fails

cursor Bot reviewed Jul 26, 2026

View reviewed changes

Comment thread conftest.py Outdated

henrydingliu added 4 commits July 26, 2026 17:23

further improving prism_convert

e8825f9

capecod & mack fixes

06f01a6

further improving prism_convert

e1c7f23

capecod & mack fixes

2d8435f

henrydingliu closed this Jul 27, 2026

henrydingliu reopened this Jul 27, 2026

henrydingliu added 6 commits July 27, 2026 02:53

fixing test_utilities and removing sparse.testing

7820832

more fixes

79318f0

implementing nanquantile and nanmedian

7003aee

fixing test_utilities and removing sparse.testing

09364a1

more fixes

14e1f09

implementing nanquantile and nanmedian

7c8e5fe

cursor Bot reviewed Jul 27, 2026

View reviewed changes

Comment thread chainladder/utils/sparse.py

genedan mentioned this pull request Jul 29, 2026

Sparse fixes #1164

Merged

1 task

henrydingliu and others added 6 commits July 29, 2026 03:39

improving comment, adding tests, fixing bugbot

900c3ad

fixing test

a5382a2

removing empty check

cfb901d

Sparse fixture (#1166)

ac9e1b5

* further improving prism_convert * capecod & mack fixes * fixing test_utilities and removing sparse.testing * more fixes * implementing nanquantile and nanmedian * improving comment, adding tests, fixing bugbot * fixing test * removing empty check

Merge pull request #1170 from casact/main

5098497

updating again

henrydingliu marked this pull request as ready for review July 29, 2026 22:24

henrydingliu requested review from genedan, jbogaardt and kennethshsu as code owners July 29, 2026 22:24

genedan reviewed Jul 31, 2026

View reviewed changes

henrydingliu and others added 3 commits July 31, 2026 02:44

Merge branch 'sparse_fixture' of https://github.com/henrydingliu/chai…

6576aa9

…nladder-python into sparse_fixture

address review

af2018c

Merge pull request #1174 from henrydingliu/sparse_fixture

9445d09

Sparse fixture

genedan approved these changes Aug 1, 2026

View reviewed changes

henrydingliu merged commit bb16128 into main Aug 1, 2026
22 checks passed

henrydingliu deleted the sparse_test branch August 1, 2026 00:41


		return COO(out)

		def nanmedian(a:COO, axis:int=0, keepdims:bool=False):

Uh oh!

Conversation

henrydingliu commented Jul 26, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary of Changes

Related GitHub Issue(s)

Additional Context for Reviewers

Uh oh!

henrydingliu commented Jul 26, 2026

Uh oh!

github-actions Bot commented Jul 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pyright Type Completeness

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

henrydingliu commented Jul 27, 2026

Uh oh!

codecov Bot commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

genedan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

henrydingliu commented Jul 31, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

henrydingliu commented Jul 26, 2026 •

edited by cursor Bot

Loading

github-actions Bot commented Jul 26, 2026 •

edited

Loading

codecov Bot commented Jul 29, 2026 •

edited

Loading