Absolute CTR structure factor and reflectivity

doc/design/ctr_stage0_contract.md

@@ -0,0 +1,86 @@
+# CTR targeted implementation: Stage 0 contract
+
+Status: implementation note, 2026-09-16. Numerical behavior was captured at
+commit `d52ab8f5234e`, before Stage 1 of
+[`ctr_targeted_implementation_plan.md`](ctr_targeted_implementation_plan.md).
+
+This note records what the characterization fixture means and which metadata
+is not available. It does not claim that the frozen numbers independently
+validate the physics.
+
+## Frozen extraction behavior
+
+`orgui/app/test/fixtures/ctr_stage0_contract.json` and
+`test_ctr_stage0_contract.py` pin two compact cases:
+
+- A stationary extraction with detector counts and one-sigma count errors,
+  radian diffraction angles, an exposure-times-monitor divisor, and the
+  current stationary Lorentz-to-`F2_hkl` conversion.
+- A mechanical rocking extraction with a degree-valued motor axis, two
+  background regions, per-frame normalization inside the angular integral,
+  radian detector acceptance, and the current relative `F2_hkl` scale.
+
+The fixture also pins a pre-typed correction-settings dictionary. Its sample
+dimensions are meters and `beam_flux_density` remains photons per second per
+square meter. Missing fields remain missing; zero is not substituted for an
+unknown calibration. Existing tests in `test_config_data.py` and
+`test_peak1Dintegr.py` cover typed NeXus round trips, legacy JSON database
+dispatch, and both stored correction-group layouts used by the reducer.
+
+At Stage 0, strict expected-failure tests recorded the two Stage 1 defects
+without changing application code: the arm-following polarization evaluator
+ignored a nonzero polarization axis, and rocking acceptance ignored actual-arm
+angles. Stage 1 converts those reproductions into passing regressions.
+
+## Settings-loader audit
+
+All `examples/config_*` files use the legacy INI loader. They define detector,
+lattice and diffractometer values; none selects a normalization monitor or
+stores integration-correction provenance. `ConfigData.from_ini` therefore
+starts with an empty `CorrectionState`. Current database snapshots write the
+typed `integration_corrections` layout (schema version 2), while
+`ConfigData.from_nxdict` still recognizes the older group containing one
+opaque JSON dataset. Neither old layout distinguishes rate-like from
+frame-integrated monitors.
+
+## Backend counter-semantics audit
+
+The implementation assumption is that a selected incident-beam monitor is
+**rate-like**. It is therefore multiplied by exposure time in the fluence
+divisor. This matches the current normalization arithmetic. The present
+backend contract names arrays but does not persist a counter kind or unit, so
+the assumption must remain visible in provenance until a beamline-specific
+calibration establishes the counter-to-total-flux conversion.
+
+| Backend | Persisted candidates | Meaning established by code | Rate/integrated status |
+|---|---|---|---|
+| Base `Scan`, imported images | none | no monitor metadata | unavailable |
+| Built-in P212 | implicit `exposure_time` when present | frame exposure, seconds by loader convention | not a flux monitor; P212 flux counters are unavailable |
+| ID31 fast/BLISS variants | `exposure_time`, `time`/`elapsed_time`/`epoch`, `srcur`, `mondio`, plus electrochemistry values | exposure is seconds; time/epoch are coordinates; `srcur` and `mondio` are often divided by their scan mean and are therefore dimensionless relative factors | selected beam monitors are treated as rate-like relative-flux proxies |
+| Example CHESS QM2 | `diode`, `ic1`, `ic2`, `emon`, `nemon`, `pemon` | raw SPEC columns; `ic2` is ion chamber 2 and is the expected primary incident-beam monitor | `ic2` is treated as rate-like; a later calibrated factor converts its reading to total photons/s |
+| Example P212 | `beckvolt_1`, `beckvolt_2`, `eh3_entrance`, `oh2_diode1`, `oh2_diode2`, `petracurrent`, `timestamp` | raw scan columns; timestamp is not a flux monitor | a selected beam monitor is treated as rate-like; its calibration is unresolved |
+| Segmented/interlaced scans | intersection of segment counters, plus implicit exposure time | values are concatenated/reordered; semantics are inherited | selected monitors remain rate-like; units and calibration are not merged |
+
+`current`, `potential`, `scaled_potv2f`, motor positions and timestamps are not
+incident-flux monitors merely because they appear in `auxillary_counters`.
+Until Stage 3 adds an explicit monitor unit and calibration relation, the
+legacy behavior is exactly exposure time times the selected rate-like monitor
+product recorded by `monitor_corrections`; it must be shown as legacy rather
+than silently assigned an absolute photon scale. For QM2, the intended primary
+monitor is `ic2`, with a future scalar calibration relating its rate to total
+incident flux in photons/s.
+
+## Stage 1 inputs now pinned
+
+- Polarization: at the calibrated arm position,
+  `polarizationAtPoints` must match `polarizationArray` for nonzero axes and
+  mixed polarization fractions. The axis-zero baseline already passes.
+- Acceptance: scan arm values are degrees at the storage boundary and must be
+  converted to radians before calling `out_of_plane_acceptance`. The fallback
+  for old scans with no arm values must remain explicit.
+
+The rolled-detector case uses the published review parameters: 172 micrometer
+pixels, 0.15 meter distance, 30 degree detector roll, ROI center `(row,
+column) = (500, 440)`, height 60 pixels, incidence 10 degrees and actual
+`(gamma, delta)` arm angles `(30, 40)` degrees. It is a counterexample to arm
+invariance, not a statement about a typical experimental error.

doc/design/ctr_stage2_persistence_contract.md

@@ -0,0 +1,62 @@
+# CTR Stage 2 persistence contract
+
+Status: implemented, 2026-09-16.
+
+Stage 2 introduces persistence and dispatch boundaries only. It deliberately
+does not enable the total-flux calculation or move normalization/footprint
+between extraction and reduction. Existing numerical defaults and legacy
+dataset meanings remain unchanged.
+
+## Requested settings
+
+`configuration/orgui/integration_corrections` now has schema version 3. The
+legacy `beam_flux_density` field remains photons/(s m2). The distinct
+`total_incident_flux` field is photons/s and is never inferred from that
+density. Primary-monitor name, rate/integrated kind, unit, calibration reading
+and calibration exposure have separate fields. Horizontal interception is an
+explicit mode plus optional fraction. Missing values mean unknown or not
+configured; they do not mean zero flux, full interception or a unity divisor.
+
+Measured vertical profiles store their normalized positions [m] and density
+[1/m] in addition to the source path and import settings. Analytical profiles
+remain reproducible from their named shape and parameters.
+
+## Applied curve record
+
+New extractions retain all established groups and add a sibling named
+`ctr_curve_v3`. It is not named `rois` or `counters`, so an older reducer cannot
+mistake this branch for the legacy unnormalized input. The record separates:
+
+- identity: algorithm, output quantity and scale convention;
+- normalization: explicit status, exact divisor and contributing counters;
+- illumination: explicit status, exact divisor, named legacy/new convention,
+  overlap components and alpha in radians when available;
+- base: center/background sums and variances plus the reversible background-
+  subtracted scalar curve and variance before normalization/illumination;
+- pixel corrections: the established combined center/background factors,
+  with separate polarization-only fields reserved for Stage 4;
+- geometry: actual detector arms in radians and ROI location/size in pixels;
+- profile: embedded profile provenance rather than only an external path.
+
+Correction statuses are exactly `applied`, `not_applied`, `unavailable` or
+`unknown`. An absent dataset is never read as unity.
+
+Stationary records capture the scalar curve immediately before the existing
+normalization and active-area divisors. Rocking extraction currently applies
+neither divisor, so both statuses are explicitly `not_applied`; its current
+legacy reducer remains responsible for them until Stage 5.
+
+## Compatibility and dispatch
+
+The existing `rois`, `croibg`, `Cfactors_croi` and `Cfactors_bgroi` datasets
+retain their meanings. Current Stage 2 records identify their algorithm as
+`legacy_stationary_roi_v1` or `legacy_rocking_roi_v1`, allowing the existing
+paths to reproduce the same numbers.
+
+The rocking loader checks the sibling record before returning the legacy ROI
+curve. A record with a non-legacy algorithm (for example the later framewise
+total-flux convention) is refused by the legacy reducer instead of being
+silently reinterpreted. Incomplete older records remain accessible through the
+legacy diagnostic path with unknown provenance. Reduction outputs continue to
+be added as new measurement groups, so re-reduction does not overwrite the
+source extraction record.

doc/design/ctr_stage3_total_flux_contract.md

@@ -0,0 +1,81 @@
+# CTR Stage 3 total-flux numerical contract
+
+Status: implemented, 2026-09-16.
+
+Stage 3 adds pure numerical building blocks for calibrated total incident flux.
+It does not change either integration path, the current GUI/config behavior or
+the legacy flux-density APIs. Stage 5 will select and persist this convention
+during extraction.
+
+## Per-frame incident photons
+
+`corrections.normalization.frame_fluence` returns `Q_f` in photons. A constant
+total flux `Phi` [photons/s] gives `Q_f = Phi T_f`. A rate-like primary monitor
+uses
+
+```text
+Q_f = Phi_ref T_f M_f / M_ref.
+```
+
+An integrated monitor already contains the exposure and instead uses
+
+```text
+Q_f = Phi_ref T_ref U_f / U_ref.
+```
+
+There is no second multiplication by `T_f` in the integrated case. Monitor kind
+is explicit; the numerical core does not infer it from a counter name or unit.
+The beamline counters expected for the first wiring pass, including QM2 `ic2`,
+are rate-like after applying their configured conversion to total incident
+flux. `relative_frame_fluence` applies the same exposure rule without claiming
+photon units when no absolute calibration exists.
+
+All fluxes, exposures, monitor readings and calibration references must be
+finite and strictly positive. Calibration references correspond to the stated
+reference flux; no scan-mean normalization is introduced.
+
+## Beam/sample illumination
+
+For a normalized vertical beam profile, sample length `L` [m], incidence angle
+`alpha` [rad] and explicit horizontal intercepted fraction `f_x`,
+
+```text
+f_z   = profile.flux_on_sample(alpha, L)
+f_hit = f_z f_x
+H     = f_hit / sin(alpha).
+```
+
+`corrections.activearea.intercepted_fraction` returns `f_hit`, while
+`illumination_divisor` returns the dimensionless `H`. These are different
+quantities: photons physically hitting the sample are `Q_f f_hit`, whereas the
+retained CTR reduction divides counts by `Q_f H`.
+
+`BeamProfile.flux_over_sine` evaluates the ratio directly. Built-in measured,
+distribution and analytical Gaussian profiles continue it to the finite
+grazing-incidence limit, avoiding numerical zero divided by zero. Existing
+external profile subclasses remain instantiable through a non-abstract default
+implementation.
+
+The vertical profile supplies no horizontal geometry. `f_x` is therefore an
+explicit number in `(0, 1]`; one means the full horizontal beam is intercepted.
+No general two-dimensional clipping model is implied.
+
+## Structure-factor scale
+
+`corrections.measurement.total_flux_prefactor` returns
+
+```text
+K = r_e^2 lambda^2 / A_u^2,
+```
+
+with wavelength in Angstrom and surface unit-cell area in square Angstrom.
+`structure_factor_squared_from_photon_yield` reduces
+`Y = N_net / (Q_f H)` with the existing scan-mode angular factor and optional
+detector efficiency. `photon_yield_from_structure_factor` is its exact forward
+inverse.
+
+The total-flux identity is pinned independently for a separable 2-D Gaussian:
+total flux times `H` equals peak flux density times the existing effective beam
+area. Thus `Q H` replaces the legacy density-times-area fluence; it is not an
+extra footprint correction. `scale_factor`, `structure_factor_squared` and
+`integrated_intensity` retain their legacy density semantics unchanged.