feat!: Introduce new summary section for data metrics by MoritzPotthoffQC · Pull Request #47 · Quantco/diffly

Moritz Potthoff (MoritzPotthoffQC) · 2026-07-17T12:01:53Z

Motivation

See this discussion: We want to display data metrics in their own section in the summary.

Changes

Add differentiation for the metrics
Add new section Data Inspection that gets all the data metrics

codecov · 2026-07-17T12:02:59Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (01cef28) to head (6a09e68).

Additional details and impacted files

@@            Coverage Diff            @@
##              main       #47   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           14        13    -1     
  Lines         1052      1127   +75     
=========================================
+ Hits          1052      1127   +75

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot

Pull request overview

This PR extends Diffly’s summary reporting to distinguish change metrics (computed over deltas) from data inspection metrics (describing left/right datasets), and renders the latter in a new dedicated “Data Inspection” section while also separating them in the JSON schema.

Changes:

Add Metric.kind ("change" vs "data") and mark the Null% preset as a "data" metric.
Split per-column metric outputs into metrics vs data_metrics in the summary JSON, and render "data" metrics in a new “Data Inspection” section.
Update summary tests and regenerate pretty-output fixtures to reflect the new section/layout.

Reviewed changes

Copilot reviewed 36 out of 36 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
diffly/metrics/_common.py	Adds `Metric.kind` to classify metrics by rendering section.
diffly/metrics/data.py	Marks `Null%` preset as a `"data"` metric.
diffly/summary.py	Routes metric labels into change vs data lists; adds “Data Inspection” rendering and JSON schema changes.
tests/summary/test_summary.py	Adds coverage for `metrics` vs `data_metrics` JSON routing; updates expected JSON shape.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_True_top_True_slim_True_sample_rows_True_sample_pk_True.txt	Updates fixture output to move `Null%` into “Data Inspection”.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_True_top_True_slim_True_sample_rows_False_sample_pk_False.txt	Updates fixture output to move `Null%` into “Data Inspection”.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_True_top_True_slim_False_sample_rows_True_sample_pk_True.txt	Updates fixture output to move `Null%` into “Data Inspection”.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_True_top_True_slim_False_sample_rows_False_sample_pk_False.txt	Updates fixture output to move `Null%` into “Data Inspection”.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_True_top_False_slim_True_sample_rows_True_sample_pk_False.txt	Updates fixture output to move `Null%` into “Data Inspection”.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_True_top_False_slim_True_sample_rows_False_sample_pk_False.txt	Updates fixture output to move `Null%` into “Data Inspection”.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_True_top_False_slim_False_sample_rows_True_sample_pk_False.txt	Updates fixture output to move `Null%` into “Data Inspection”.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_True_top_False_slim_False_sample_rows_False_sample_pk_False.txt	Updates fixture output to move `Null%` into “Data Inspection”.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_False_top_True_slim_True_sample_rows_True_sample_pk_True.txt	Updates fixture output to move `Null%` into “Data Inspection”.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_False_top_True_slim_True_sample_rows_False_sample_pk_False.txt	Updates fixture output to move `Null%` into “Data Inspection”.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_False_top_True_slim_False_sample_rows_True_sample_pk_True.txt	Updates fixture output to move `Null%` into “Data Inspection”.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_False_top_True_slim_False_sample_rows_False_sample_pk_False.txt	Updates fixture output to move `Null%` into “Data Inspection”.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_False_top_False_slim_True_sample_rows_True_sample_pk_False.txt	Updates fixture output to move `Null%` into “Data Inspection”.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_False_top_False_slim_True_sample_rows_False_sample_pk_False.txt	Updates fixture output to move `Null%` into “Data Inspection”.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_False_top_False_slim_False_sample_rows_True_sample_pk_False.txt	Updates fixture output to move `Null%` into “Data Inspection”.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_False_top_False_slim_False_sample_rows_False_sample_pk_False.txt	Updates fixture output to move `Null%` into “Data Inspection”.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_True_top_True_slim_True_sample_rows_True_sample_pk_True.txt	Updates fixture output to move `Null%` into “Data Inspection”.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_True_top_True_slim_True_sample_rows_False_sample_pk_False.txt	Updates fixture output to move `Null%` into “Data Inspection”.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_True_top_True_slim_False_sample_rows_True_sample_pk_True.txt	Updates fixture output to move `Null%` into “Data Inspection”.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_True_top_True_slim_False_sample_rows_False_sample_pk_False.txt	Updates fixture output to move `Null%` into “Data Inspection”.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_True_top_False_slim_True_sample_rows_True_sample_pk_False.txt	Updates fixture output to move `Null%` into “Data Inspection”.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_True_top_False_slim_True_sample_rows_False_sample_pk_False.txt	Updates fixture output to move `Null%` into “Data Inspection”.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_True_top_False_slim_False_sample_rows_True_sample_pk_False.txt	Updates fixture output to move `Null%` into “Data Inspection”.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_True_top_False_slim_False_sample_rows_False_sample_pk_False.txt	Updates fixture output to move `Null%` into “Data Inspection”.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_False_top_True_slim_True_sample_rows_True_sample_pk_True.txt	Updates fixture output to move `Null%` into “Data Inspection”.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_False_top_True_slim_True_sample_rows_False_sample_pk_False.txt	Updates fixture output to move `Null%` into “Data Inspection”.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_False_top_True_slim_False_sample_rows_True_sample_pk_True.txt	Updates fixture output to move `Null%` into “Data Inspection”.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_False_top_True_slim_False_sample_rows_False_sample_pk_False.txt	Updates fixture output to move `Null%` into “Data Inspection”.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_False_top_False_slim_True_sample_rows_True_sample_pk_False.txt	Updates fixture output to move `Null%` into “Data Inspection”.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_False_top_False_slim_True_sample_rows_False_sample_pk_False.txt	Updates fixture output to move `Null%` into “Data Inspection”.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_False_top_False_slim_False_sample_rows_True_sample_pk_False.txt	Updates fixture output to move `Null%` into “Data Inspection”.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_False_top_False_slim_False_sample_rows_False_sample_pk_False.txt	Updates fixture output to move `Null%` into “Data Inspection”.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

EgeKaraismailogluQC

Thank you Moritz Potthoff (@MoritzPotthoffQC)! In general I am happy with the layout. I added a few comments. In addition to those, I think it would be nice to have at least one summary fixture where we render two or more data metrics rather than just one, just to see what that looks like (maybe I missed it?).

Copilot

Pull request overview

Copilot reviewed 51 out of 52 changed files in this pull request and generated 1 comment.

Copilot

Pull request overview

Copilot reviewed 84 out of 85 changed files in this pull request and generated 2 comments.

Co-authored-by: Copilot Autofix powered by AI <[email protected]>

EgeKaraismailogluQC

Looks great, thank you Moritz Potthoff (@MoritzPotthoffQC)! There is a final issue with codecov. That can sometimes be a false positive and then would go away upon retrigger. Can you take a look? Otherwise, LGTM 😄

Marius Merkle (MariusMerkleQC)

Thanks a lot for working on the metrics in this and other PRs. I didn't review 100% of the changes in depth and trust EgeKaraismailogluQC's review for it. Just a few comments from my side.

Copilot

Pull request overview

Copilot reviewed 85 out of 86 changed files in this pull request and generated no new comments.

Marius Merkle (MariusMerkleQC)

Thank you, looks great now!

Co-authored-by: Marius Merkle <[email protected]>

…abels.py Co-authored-by: Marius Merkle <[email protected]>

…s_few.py Co-authored-by: Marius Merkle <[email protected]>

Co-authored-by: Marius Merkle <[email protected]>

…ts_many.py Co-authored-by: Marius Merkle <[email protected]>

init

bff2594

Moritz Potthoff (MoritzPotthoffQC) self-assigned this Jul 17, 2026

github-actions Bot added the enhancement New feature or request label Jul 17, 2026

add test

51fbda2

Moritz Potthoff (MoritzPotthoffQC) requested a review from Copilot July 17, 2026 12:09

Copilot started reviewing on behalf of Moritz Potthoff (MoritzPotthoffQC) July 17, 2026 12:09 View session

Copilot AI reviewed Jul 17, 2026

View reviewed changes

Comment thread diffly/summary.py Outdated

Moritz Potthoff (MoritzPotthoffQC) added 3 commits July 17, 2026 14:39

refactor

cb086ef

simplify

526c944

fix

5732c37

Moritz Potthoff (MoritzPotthoffQC) marked this pull request as ready for review July 17, 2026 13:56

Moritz Potthoff (MoritzPotthoffQC) requested review from EgeKaraismailogluQC, Marius Merkle (MariusMerkleQC) and Oliver Borchert (borchero) as code owners July 17, 2026 13:56

Moritz Potthoff (MoritzPotthoffQC) mentioned this pull request Jul 17, 2026

feat: Add data metrics for numeric columns #46

Merged

EgeKaraismailogluQC reviewed Jul 22, 2026

View reviewed changes

Comment thread diffly/summary.py Outdated

Comment thread diffly/metrics/_common.py Outdated

Comment thread diffly/metrics/_common.py Outdated

Comment thread diffly/metrics/data.py Outdated

Moritz Potthoff (MoritzPotthoffQC) added 2 commits July 22, 2026 19:07

review

5867021

review

6b705e7

Moritz Potthoff (MoritzPotthoffQC) marked this pull request as draft July 22, 2026 17:29

Moritz Potthoff (MoritzPotthoffQC) added 7 commits July 22, 2026 19:48

review

6c868d6

review

6802592

refactor

d827967

simplify

2a49d96

simplify

eef3d3b

simplify

e918bbf

fix

65d80d6

Moritz Potthoff (MoritzPotthoffQC) requested a review from Copilot July 23, 2026 12:35

Copilot started reviewing on behalf of Moritz Potthoff (MoritzPotthoffQC) July 23, 2026 12:36 View session

Copilot AI reviewed Jul 23, 2026

View reviewed changes

Comment thread diffly/metrics/__init__.py Outdated

Copilot started reviewing on behalf of Moritz Potthoff (MoritzPotthoffQC) July 25, 2026 13:14 View session

Copilot AI reviewed Jul 25, 2026

View reviewed changes

Comment thread diffly/comparison.py Outdated

Comment thread docs/api/metrics.rst

Potential fix for pull request finding

8c7c45a

Co-authored-by: Copilot Autofix powered by AI <[email protected]>

Moritz Potthoff (MoritzPotthoffQC) requested a review from EgeKaraismailogluQC July 25, 2026 13:24

EgeKaraismailogluQC approved these changes Jul 27, 2026

View reviewed changes

Marius Merkle (MariusMerkleQC) reviewed Jul 27, 2026

View reviewed changes

Comment thread diffly/metrics/__init__.py Outdated

Comment thread diffly/metrics/change.py Outdated

Comment thread tests/summary/test_summary.py Outdated

Moritz Potthoff (MoritzPotthoffQC) added 3 commits July 27, 2026 15:37

review

a6bb6d8

coverage

ddc31f8

coverage

69e5893

Marius Merkle (MariusMerkleQC) reviewed Jul 27, 2026

View reviewed changes

Comment thread diffly/comparison.py Outdated

Moritz Potthoff (MoritzPotthoffQC) added 2 commits July 27, 2026 16:32

fix

42787c7

review

b1ac321

Moritz Potthoff (MoritzPotthoffQC) requested a review from Copilot July 27, 2026 15:35

Copilot started reviewing on behalf of Moritz Potthoff (MoritzPotthoffQC) July 27, 2026 15:35 View session

Copilot AI reviewed Jul 27, 2026

View reviewed changes

Moritz Potthoff (MoritzPotthoffQC) requested a review from Marius Merkle (MariusMerkleQC) July 27, 2026 15:41

Marius Merkle (MariusMerkleQC) approved these changes Jul 27, 2026

View reviewed changes

Moritz Potthoff (MoritzPotthoffQC) and others added 11 commits July 28, 2026 10:13

Apply suggestions from code review

cab9baf

Co-authored-by: Marius Merkle <[email protected]>

Update tests/summary/fixtures/metrics_long_labels/test_metrics_long_l…

ee99ec1

…abels.py Co-authored-by: Marius Merkle <[email protected]>

Update tests/summary/fixtures/metrics_presets_few/test_metrics_preset…

41f8637

…s_few.py Co-authored-by: Marius Merkle <[email protected]>

Update tests/summary/test_summary.py

0958d6c

Co-authored-by: Marius Merkle <[email protected]>

Update tests/test_metrics.py

aeb3cd8

Co-authored-by: Marius Merkle <[email protected]>

Update tests/summary/test_summary.py

1d93a59

Co-authored-by: Marius Merkle <[email protected]>

Update tests/summary/fixtures/metrics_presets_many/test_metrics_prese…

e5fd01a

…ts_many.py Co-authored-by: Marius Merkle <[email protected]>

review

045ba31

fix

d150ba4

review

7835e7b

review

6a09e68

Moritz Potthoff (MoritzPotthoffQC) merged commit a626117 into main Jul 28, 2026
18 checks passed

Moritz Potthoff (MoritzPotthoffQC) deleted the data-inspection-section branch July 28, 2026 08:48

Uh oh!

Conversation

Moritz Potthoff (MoritzPotthoffQC) commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Uh oh!

codecov Bot commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

EgeKaraismailogluQC left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

EgeKaraismailogluQC left a comment

Choose a reason for hiding this comment

Uh oh!

Marius Merkle (MariusMerkleQC) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Marius Merkle (MariusMerkleQC) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Moritz Potthoff (MoritzPotthoffQC) commented Jul 17, 2026 •

edited

Loading

codecov Bot commented Jul 17, 2026 •

edited

Loading