Skip to content

test: add architectural and schema conformance test suite - #955

Open
allenporter wants to merge 6 commits into
Python-roborock:mainfrom
allenporter:add_conformance_tests
Open

test: add architectural and schema conformance test suite#955
allenporter wants to merge 6 commits into
Python-roborock:mainfrom
allenporter:add_conformance_tests

Conversation

@allenporter

@allenporter allenporter commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR introduces an automated conformance test suite in tests/conformance/ using package reflection to enforce architectural invariants and prevent recurring regressions previously caught manually during PR reviews.

Related to Issue #953

Each conformance test directly targets patterns discussed in past code reviews (particularly with @Lash-L and @allenporter):


1. Data Model Serialization & RoborockBase Inheritance (test_model_conformance.py)

  • Background: Domain data models intended for serialization/deserialization need RoborockBase (which provides .as_dict(), camelCase conversion, and .from_dict() integration). When models are defined using plain @dataclass, standard serialization helpers fail or necessitate ad-hoc dictionary conversion workarounds.
  • Review Precedent & Incidents:
  • Enforcement: Reflects over roborock.data and ensures every declared domain @dataclass inherits from RoborockBase.

2. Wire Enum Fallback Resilience (test_enum_conformance.py)

  • Background: Roborock hardware protocols and cloud APIs frequently emit unannounced or firmware-specific enum values. Without fallback members or safe lookup methods, parsing fails abruptly when encountering new wire codes.
  • Review Precedent & Incidents:
  • Enforcement:
    • Ensures RoborockEnum subclasses provide an unknown fallback member, tested against dynamically derived sentinels (max(val) + 1) outside the enum's range.
    • Ensures RoborockModeEnum subclasses implement from_code_optional() or from_code(), tested against dynamic sentinels.
    • Validates that enums in wire *code_mappings* modules inherit from RoborockEnum or RoborockModeEnum (with 4 intentional command/domain exceptions allowlisted).
    • Baselined 53 legacy enums missing unknown using @pytest.mark.xfail(strict=True) so new enums must handle fallbacks without breaking existing legacy enums.

3. Trait Channel & Transport Boundaries (test_trait_boundary_conformance.py)

  • Background: Traits represent device capabilities and should coordinate over high-level channel abstractions (RoborockChannel / RPC dispatchers), rather than taking direct references to transport sockets, encryption keys, tokens, or IP addresses.
  • Review Precedent & Incidents:
  • Enforcement:
    • Inspects trait __init__ constructor parameter names to ensure no transport/credential parameters (socket, token, local_key, ip, key, etc.) are injected.
    • Recursively checks parameter type annotations against low-level transport/credential classes (socket.socket, asyncio.BaseTransport, LocalChannel, LocalChannelParams, MqttParams, MqttSession).

4. Exception Hierarchy Rooting (test_exception_conformance.py)

  • Background: Library exceptions raised by decoders and traits must inherit from RoborockException so caller reconnection and retry loops catch errors reliably without leaking unexpected standard exceptions.
  • Review Precedent & Incidents:
  • Enforcement: Ensures all custom exception classes defined in roborock.exceptions and across the library inherit from RoborockException.

5. Robust Package Discovery (discovery.py)

  • walk_modules() imports modules directly without swallowing ImportError or AttributeError, guaranteeing that syntax or module initialization errors fail fast in CI rather than causing modules to be silently skipped by reflection.

Verification

  • uv run pytest tests/conformance/ -v: 280 passed, 53 xfailed
  • uv run pytest: 1,263 passed, 53 xfailed, 92 snapshots passed
  • uv run pre-commit run --all-files: all passed (ruff, mypy, codespell, check-yaml, etc.)

Copilot AI lite review requested due to automatic review settings September 12, 2026 20:31
@allenporter
allenporter requested a review from Lash-L September 12, 2026 20:34

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Moderate issues leave reflection coverage and trait boundary enforcement incomplete.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds reflection-based conformance tests for models, enums, traits, and exceptions.

Changes:

  • Added shared discovery utilities and package marker.
  • Added architectural and schema conformance checks.
  • Added legacy enum fallback baselines.
File summaries
File Summary
tests/conformance/test_trait_boundary_conformance.py Checks trait constructor boundaries; type-based transport dependencies may bypass the checks.
tests/conformance/test_model_conformance.py Validates RoborockBase inheritance for models.
tests/conformance/test_exception_conformance.py Validates exception hierarchy rooting.
tests/conformance/test_enum_conformance.py Checks enum fallbacks; plain enums may be missed and the fixed sentinel may be valid.
tests/conformance/discovery.py Provides reflection helpers; import and attribute errors can silently skip modules.
tests/conformance/__init__.py Defines the conformance test package.
Review details

Suppressed comments (2)

tests/conformance/test_enum_conformance.py:87

  • These collections only discover classes that already inherit from RoborockEnum or RoborockModeEnum. A new wire/status enum declared as a plain Enum, IntEnum, or StrEnum is invisible to all three checks, so the suite cannot enforce the stated requirement that wire enums use the resilient bases. Add discovery or an explicit allowlist for wire code-mapping enums, while excluding intentional UI/domain enums.
_ALL_ROBOROCK_ENUMS = discover_subclasses(roborock, RoborockEnum, exclude=(RoborockEnum,))
_ALL_MODE_ENUMS = discover_subclasses(roborock, RoborockModeEnum, exclude=(RoborockModeEnum,))

tests/conformance/test_enum_conformance.py:98

  • 99999 is not guaranteed to be unknown: a future enum can legitimately add that firmware code, making this assertion exercise a valid member and stop testing the fallback. Derive a sentinel outside the enum's current values instead.
    assert enum_cls(99999) == enum_cls.unknown
  • Files reviewed: 6/6 changed files
  • Comments generated: 2
  • Review effort level: Lite

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread tests/conformance/discovery.py Outdated
Comment thread tests/conformance/test_trait_boundary_conformance.py Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants