Diagram.restrict: apply Part conditions through the Master to preserve master-part compositional integrity

[dimitri-yatsenko]

Summary

dj.Diagram.restrict is logically conjunctive (AND — it removes rows that fail the criteria).
Unlike the disjunctive trace/cascade (which include everything touched and keep/pull whole
compositional units), a conjunctive filter can cut into a master-part group, leaving partial
groups — a Master exported with only a subset of its Parts (or none). This preserves referential
integrity but breaks compositional integrity: the exported entity misrepresents the original.

Reproduction (structural)

dj.Diagram(schema).restrict(Master & 'm=1').restrict(Master.Part & 'p>3')

• Master → all of m=1.
• Master.Part → (Parts of m=1, from the master-keyed downstream restriction) AND p>3 → only
  the p>3 Parts.
• Result: Master m=1 whole, but only a subset of its Parts. A Master can even survive with zero
  Parts, since restrict does not walk up from Part to Master.

Why it matters

• Referential integrity — fine. A kept Part's FK to its Master is satisfied; a Master needs no Parts
  to be valid. The export re-imports without dangling foreign keys.
• Compositional integrity — broken. DataJoint's master-part contract treats a Master's Parts as one
  atomic entity. A recording exported with only some of its channels loads cleanly but is a
  corrupted/partial entity.

Root cause / model

In DataJoint's model the Master is the entity; Parts have no independent existence. The unit of
export is therefore the Master-with-all-its-Parts. A Part-level filter is either meaningless for export,
or should mean "Masters that have a matching Part."

Decision (v2.4)

In Diagram restriction, conditions on Part tables are applied through their Master:

• A restriction that selects Part rows lifts to an existential restriction on the Master — Masters
  that have a matching Part are selected.
• A selected Master brings in all of its Parts.
• Net: compositional units are never split; Parts are never filtered within a unit.

This is the compositional-integrity analogue, for read-only restriction/export, of what cascade's
part_integrity="cascade" does for delete.

Scope / relationships

• Not for 2.3.1 — behavior change deferred to v2.4.
• Distinct from docs(diagram): document restrict is downstream-only (no part->master walk-up) #1500 (which established that restrict does not perform cascade's deletion
  part→master walk-up). This issue is about filtering granularity — keeping units whole under a
  conjunctive filter — a different mechanism; the two are compatible.
• Optional 2.3.x stopgap: document the current behavior (restrict can yield partial Part groups; lift
  Part criteria to the Master via Master & (Part & cond)).

References

• Surfaced in the 2.3.1 dj.Diagram review (finding F12).
• Code: src/datajoint/diagram.py — restrict, _propagate_restrictions (mode="restrict"); contrast
  _propagate_part_to_master (cascade's upward walk).

[dimitri-yatsenko]

Scope note: this issue also subsumes documenting restrict's contract (the intended docstring/spec text). PR #1500 attempted that against current 2.3.1 behavior ('downstream-only; never walks Part→Master') but was closed unmerged — that wording contradicts the direction here (Part conditions lifting to the Master). The restrict documentation should be written as part of this v2.4 work, against the final semantics: Part conditions apply through the Master; a matching Part selects its Master and brings in all its Parts; use trace for upstream/ancestor closure.