feat: Expose comparison objects upon failure in testing assertions to enable interactive debugging

[Moritz Potthoff (MoritzPotthoffQC)]

Motivation

I often have situations where a data frame/collection equality assertion in a test fails, but the summary that is included in the assertion failure is not sufficient to understand the issue. I think it should be as easy as possible to get interactive access to the comparison object that is already constructed for evaluating the assertion for debugging. Currently, I often have to manually reconstruct it, which causes unnecessary overhead.

Changes

• Introduce subclasses of assertion errors that carry a/several comparison object(s). Since they are subclasses, this does not change existing behavior (e.g., when AssertionError is caught)
• Append comparison object(s) to the assertions that are raised
• Explain how to use it in the assertion failure text.

Usage Example

When a test fails, run it with --pdb and directly access the comparison(s):

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (3769d7d) to head (a35bdd5).

Additional details and impacted files

@@            Coverage Diff            @@
##              main       #48   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           13        13           
  Lines         1127      1138   +11     
=========================================
+ Hits          1127      1138   +11     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

This PR enhances Diffly’s testing helpers by raising richer assertion errors that carry the underlying DataFrameComparison objects, enabling interactive debugging when assert_frame_equal / assert_collection_equal fail.

Changes:

• Added ComparisonAssertionError base class plus FrameComparisonAssertionError and CollectionComparisonAssertionError that store comparison object(s).
• Updated assert_frame_equal / assert_collection_equal to raise these subclasses and include a debugging hint in the failure message.
• Added tests asserting that the comparison object(s) are exposed and usable after failures.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
diffly/testing.py	Introduces new assertion error subclasses, updates raised exception types, and appends interactive-debug hints to failure messages.
tests/test_assert_frame_equal.py	Adds a regression test verifying FrameComparisonAssertionError.comparison is exposed and usable.
tests/test_assert_collection_equal.py	Adds a regression test verifying CollectionComparisonAssertionError.comparisons exposes only failing member comparison(s).

Comments suppressed due to low confidence (3)

diffly/testing.py:56

• Similarly here, the Pdb snippet relies on $_exception, which is not available on Python 3.11. Use a version-agnostic approach (e.g. sys.last_value) and optionally mention $_exception for Python 3.12+.

    Access the failing member comparisons from the exception using ``e.comparisons``
    or from a post-mortem debugger via::

        (Pdb) cmps = $_exception.comparisons
        (Pdb) cmps["some_member"].joined_unequal()

diffly/testing.py:81

• Same issue for the collection debugging hint: $_exception is not available on Python 3.11, so the instructions should mention sys.last_value (and optionally $_exception for Python 3.12+).

_DEBUG_HINT_COLLECTION = (
    "To debug interactively (e.g. with `pytest --pdb`), access the failing member "
    "comparisons via `$_exception.comparisons` (a mapping from member name to "
    "comparison) at the debugger prompt or via the `.comparisons` attribute of this "
    "error."
)

diffly/testing.py:304

• The doctest/example text in this docstring still shows an AssertionError traceback, but the function now raises FrameComparisonAssertionError (a subclass of AssertionError). This can be confusing when users compare the docs to actual output.

Update the example traceback (and optionally the surrounding sentence) to reflect the new exception type.

        FrameComparisonAssertionError: If the data frames are not equal.

    Note:
        Contrary to :meth:`polars.testing.assert_frame_equal`, the data frames ``left``
        and ``right`` may both be either eager or lazy. They are not required to be the

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.