diff --git a/CHANGELOG.md b/CHANGELOG.md index 4b87c62..ee44db8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,294 @@ This is the changelog for the software orGUI, written by Timo Fuchs Scientific and analysis additions: +- **The end-to-end CTR correction contract now has an independent + calibrated-pixel forward validation and published limits.** A finite-width + Gaussian rod is simulated directly on calibrated detector rays without using the + production normalization, illumination, angular-factor or structure-factor + helpers, then reduced through the stationary and rocking workflows. The + matrix covers changing flux/exposure, rate-like and integrated monitors, + measured-profile width and offset, moving detector arms, complete and + clipped ROIs, and convergence of the rocking quadrature before its tolerance + is chosen. Finite apertures are compared with their resolution-weighted + structure-factor integral rather than a point value. No distributable raw + CTR scan with independent absolute-flux calibration is present, so the docs + deliberately stop short of a real-data absolute-accuracy claim and identify + detector efficiency, external transmission and in-plane acceptance as the + remaining experimental inputs. + +- **CTR correction controls now state what is saved and what the next + operation will do.** The integration dialog separates detector signal, + incident beam and CTR result settings; new sessions use an explicitly + relative total-flux convention, while calibrated photon flux, one + rate-like or integrated primary monitor, its calibration reference and the + effective normalization formula are shown with units. Legacy flux density + and multi-monitor products remain labeled and are not reinterpreted. + Horizontal interception must be selected as full or as a known fraction; + unresolved imported settings stay ``Not specified``. The rocking reducer + shows the loaded curve's saved normalization/footprint provenance separately + from the next reduction and replaces the footprint checkbox with safe + keep/apply/remove actions. Unknown legacy provenance disables replacement + and removal and directs the user to re-extract rather than guessing. + +- **The numerical correction core now has an explicit total-incident-flux + convention.** New pure APIs calculate calibrated photons per frame from a + constant flux or an explicitly rate-like/integrated monitor, distinguish the + intercepted fraction from the dimensionless illumination divisor + ``H = f_hit / sin(alpha)``, and reduce photon-normalized yield with + ``K = r_e^2 lambda^2 / A_u^2``. Built-in beam profiles retain the finite + grazing-incidence limit, and horizontal interception must be stated + separately because a vertical profile cannot infer it. This stage adds the + unit-audited numerical path; the extraction/reduction wiring described + below activates it only for explicit version-3 settings. Legacy + flux-density APIs retain their original units and meaning. + +- **Stationary and rocking CTR integrations now share one framewise total-flux + policy.** An explicitly configured primary monitor is classified as a rate + or an integrated reading, so exposure enters its photon-fluence divisor + exactly once; an ``ic2``-style ion-chamber rate is therefore handled as + ``exposure * monitor_rate`` before applying its calibration factor. New + total-flux curves require both the stored incident-photon divisor ``Q`` and + the stated horizontal plus calculated vertical illumination divisor ``H`` + before they can be labeled ``F2_hkl``. Stationary scans form that result + immediately. Rocking scans keep the existing later reduction step, but the + reducer now recognizes ``framewise_ctr_total_flux_v1``, reconstructs the + curve from its immutable polarization-only base values and exact stored Q/H + arrays, and does not call the live legacy normalization or footprint path a + second time. Calibrated curves additionally use the stored wavelength and + surface unit-cell area with ``r_e^2 lambda^2 / A_u^2``; detector efficiency + and external transmission remain explicit unity assumptions. Footprint + keep/remove/replace operations always restart from the saved base curve, + reject unknown provenance, and require explicit permission to cross between + the legacy active-area and new total-flux conventions. Legacy databases and + reconstruction defaults retain their existing numerical behavior. + +- **Saved CTR curves now carry a versioned correction record.** This is a + persistence-only compatibility step: numerical defaults and the established + ``rois``, ``croibg`` and ``Cfactors_*`` meanings are unchanged. A separate + ``ctr_curve_v3`` branch records the reversible pre-normalization scalar + curve and variance, exact applied normalization and illumination divisors, + explicit applied/not-applied/unavailable/unknown states, pixel-factor + provenance, detector-arm/ROI geometry and embedded beam-profile data. The + requested correction settings schema is version 3 and stores total incident + flux separately from the legacy flux density, together with primary-monitor + kind/reference and horizontal-interception fields; absent values remain + unknown rather than physical zero or unity. The rocking reducer explicitly + accepts unchanged legacy-algorithm records, dispatches the supported + total-flux algorithm through its stored divisors, and refuses any other + normalized contract, preventing it from silently normalizing a new curve + twice. + +- **The polarization correction now follows the detector arm.** + *This changes saved numbers for scans that move the detector arm and for + arm-corrected integrations using a nonzero configured polarization axis; + the conventional axis-zero case is unchanged.* The per-pixel polarization + array is built from the calibrated + geometry, which is right only while the arm stays there. On a scan that + drives the arm -- a reflectivity curve, where it follows twice the incidence + angle -- the same pixel looks in a different direction on every frame, and + the calibrated-position value understated the correction by 3 % at a + scattering angle of 10 degrees, 10 % at 18 and 33 % at 30. Both integration + paths now apply a per-frame factor, + ``corrections.detector.polarization_arm_correction``, that moves the + correction onto the arm position each frame was measured at. It is exactly + one while the arm sits at its calibrated position when both paths use the + same configured polarization axis, so an axis-zero fixed-arm scan is + bit-identical and needs no switch. The factor is a ratio of two region + means rather than a rebuilt per-pixel array, which keeps the cost to a + region-sized evaluation per frame; the polarization is not flat across a + region at a large scattering angle, so the means matter. The arm-following + evaluator now rotates the incident electric-field basis by the configured + ``polarization_axis``, matching the calibrated array for zero, 90-degree and + intermediate axes and for mixed polarization fractions; it previously + assumed an axis of zero. The detector solid + angle needs no such correction: an arm rotation is a rigid rotation about + the sample, so every pixel keeps its distance and its obliquity to its own + line of sight, and the solid angle is invariant under it exactly. The + reciprocal-space reconstruction applies the polarization per pixel rather + than as a region mean and is unchanged; correcting it there would need the + array rebuilt per frame. + +- **The arm-following polarization correction no longer makes a mu-scan + rocking reduction unusably slow.** It evaluates once and broadcasts when + the incidence angle and the arm are constant across a curve, which covers + every fixed-arm scan and every th-scan rocking curve; a mu scan rocks the + incidence angle itself, and a reflectivity rocking curve additionally + tracks the arm at twice it, so nothing was constant and it fell back to a + Python loop calling the per-frame correction once per point -- about 20 + seconds for a thousand-point mu scan, a minute or more for several + thousand, with no error, just a reduction that looked stuck. It is now one + batched call per rocking curve instead of one per frame + (``corrections.detector.polarization_arm_correction_frames``, backed by + ``DetectorCalibration.Detector2D_SXRD._tthAzimuthAtArms``), verified + against the per-frame loop to floating-point precision and measured about + 7-8x faster end to end. A stationary integration tracking a rod across the + detector has no single region to batch on and keeps the per-frame loop, + unchanged. **No saved value changes** -- this is the same correction, + computed the same way, just not one frame at a time. + +- **Rocking and stationary integration now produce the same structure factor.** + *This changes saved numbers in both modes.* A rocking scan and a stationary + scan of the same rod previously differed by exactly exposure time times + monitor times the out-of-plane acceptance in degrees; they now agree. Four + corrections changed. The rocking path gained the per-frame exposure and + monitor normalization, applied inside the rocking integral so that a varying + counting time or a drifting monitor is handled correctly rather than only on + average; it now integrates the rocking angle in radian as the published + expressions require, rather than in degrees; and it divides by the + out-of-plane acceptance of its region of interest, without which a rod + measured with regions resized along the scan came out with a distorted + *shape* -- a factor 2.3 across a simulated Pt(111) rod -- and not merely a + wrong scale. Separately, the detector **solid-angle correction no longer + reaches a structure factor** in either mode: summing a region of interest + already yields the complete angular integral, with every pixel weighted by + the solid angle it subtends, so dividing by that solid angle again + double-counted the detector obliquity (0.7 % for a detector at 1 m, 7 % at + 0.3 m, varying across the detector and therefore a rod shape error). The + switch stays, because that correction is the right one for a broad or + diffuse feature where a differential cross section is wanted: it still + scales the intensity counters, while a direct polarization-only photon + branch forms ``F2_hkl``, so a structure factor is the same number whether or + not it was enabled. Its tooltip and the + ``SOLA`` status badge say so. The reciprocal-space reconstruction keeps + applying it uncompensated, since that path does form a differential cross + section per pixel. For rocking scans the correction is applied when the + curves are extracted. New extractions now derive a second CTR photon curve + from the same background-subtracted signal using only the polarization + correction, including its detector-arm adjustment; ``croibg`` retains the + combined solid-angle and polarization correction as the diagnostic + intensity. This direct branch removes the covariance error from dividing a + combined correction mean by a separately estimated solid-angle mean. Older + databases without the photon curve retain the configuration-driven scalar + fallback and warn if its correction state cannot be established. A partly + masked center ROI now also warns that nominal-area scaling preserves a flat + density but cannot physically reconstruct peak intensity hidden by a mask + or detector gap. Integrated rocking scans store a + ``reduction`` group beside ``F2_hkl`` recording the mode, the angle unit, + which normalizations were applied, the acceptance used, whether the + direct photon curve or legacy solid-angle compensation was used, and the + active-area assumption, so that a saved rod can be placed on a common scale + afterwards. Rocking reductions also average the joint Lorentz and + rod-intersection factor with the same angular quadrature as the counts, + avoiding a covariance residual from multiplying two separately averaged + factors. + Rocking normalization uses the counters stored with the scan, so it requires + a backend that declares ``exposure_time`` in ``auxillary_counters``; a + missing counter is skipped and recorded rather than failing the integration. + Existing configuration files load unchanged. + This equivalence has now been checked on real data as well as in simulation: + on a LaNiO3 rod measured both ways, the two modes agree to a median 1.03 + along the rod, with the remaining spread explained by the resolution + difference between them rather than by a normalization. At a Bragg peak on + the same rod they differ by a factor of five, which is expected — a + stationary region cannot collect a peak much wider than itself. + +- **The settings that decide what was integrated are now stored with every + integration.** A reduction could previously not be reproduced from its own + output: the background ROI margins, the automatic-sizing switches, the + projected sample size, and the effective ``delta_s`` of a rocking scan were + not written anywhere. They are now saved under + ``configuration/orgui/roi_integration`` as typed groups carrying their + units — ``region`` (sizes, margins, automatic sizing), ``advanced`` (sample + size in meter, offsets, the inclination and projection switches) and + ``rocking_scan`` (``delta_s`` and ``max_s``, with ``delta_s`` stored as the + value actually used after the resolution clipping, not as typed). The + correction switches move the same way: ``integration_corrections`` is now a + group of typed datasets instead of one opaque JSON string, so a stored + configuration can be read in any HDF5 browser. Databases written with the + JSON layout are still read. The Lorentz, footprint and normalization + switches are recorded alongside the others, and a configuration that predates + them leaves those controls as the user has them rather than silently + switching a correction off. New curve records save the combined and + polarization-only region factors separately, which makes the + diagnostic-intensity and CTR-photon branches reproducible; legacy reductions + continue to read ``C_solid_angle`` where it is available. + +- **The corrections dialog now asks for the sample size across the beam.** A + second size ``W`` sits beside the existing sample size ``L``, in millimeter. + It is the sample extent perpendicular to the beam in the surface plane, and + it is taken to be the width of the illuminated area, on the assumption that + the beam is at least as wide as the sample -- so the sample bounds the lit + area rather than the beam. Nothing in a *relative* structure factor uses it: + the active-area correction that is applied to an integrated intensity is + ``illuminated_area_fraction``, which is dimensionless and unchanged, so no + saved number moves. ``W`` supplies the missing input for the *absolute* + active area in square meter, ``IntegrationCorrectionsDialog.activeArea``, + which an absolutely scaled structure factor needs. The measurement is + assumed throughout to be made with open post-sample slits and an area + detector, which is what makes the footprint the active area; the + slit-limited case, where ``1/(sin(delta) cos(alpha - beta_in))`` would apply + instead, is not supported. Both size controls now say so in their tooltips. + +- **A beam flux input, and every footprint-dialog input, is now stored with + the configuration.** The corrections dialog gains a beam flux field, the + incident photon flux density needed together with the active area for an + absolutely scaled structure factor (issue #15); called "beam flux" rather + than ``Phi_0`` or ``phi`` to avoid confusion with the diffractometer's own + sample-circle ``phi``. It, and the existing ``L`` and ``W`` sample sizes, + are now recorded in a typed ``integration_corrections/footprint`` NeXus + group beside the correction switches -- previously none of the three + reached the saved configuration at all, so a stored active-area calculation + could not be reproduced from its own output. **The beam shape itself was + the same gap and is now closed too:** whether the beam is described + analytically or by a measured profile, the selected shape and its + parameters, the profile file and how to read it, and where the sample sits + in the beam are all recorded in a new ``integration_corrections/beam_shape`` + group, in the same units the dialog itself shows them in. A configuration + written before any of this existed, or a dialog that was never opened, + leaves the running dialog exactly as it is rather than resetting it to + defaults. + +- **All correction factors collected into one package.** Every factor between + detector counts and a structure factor now lives in + ``orgui.datautils.xrayutils.corrections``, split by what it depends on: + ``geometry`` (the z-axis Lorentz, rod-interception and area table), + ``beamprofile``, ``activearea``, ``detector`` (per-pixel solid angle and + polarization), ``normalization`` (counting time and monitor), ``roi``, and + ``measurement``. The rocking integration, the stationary integration and the + reciprocal-space reconstruction previously each carried their own copy of + several of these; they now share one definition, so they cannot drift onto + different scales. The package is physics only -- numbers in, numbers out -- + and reads no scan object, configuration or widget; ``orgui.app`` + ``integration_corrections`` is the adapter that supplies those. + ``orgui.datautils.xrayutils.geometrycorrections`` and + ``orgui.datautils.xrayutils.beamprofile`` keep working as aliases of the + moved modules. **No calculated value changes.** + +- **Out-of-plane detector acceptance.** + ``orgui.datautils.xrayutils.corrections.acceptance`` estimates + ``Delta_gamma``, the angular height of a region of interest as seen from the + sample, which a rocking-scan integrated intensity is proportional to (Vlieg + equations 20 and 42) and which orGUI previously did not compute at all. + Measured edge to edge, at the region's centre column where the rod crosses + the aperture, and vectorized over a scan because orGUI resizes regions per + detector position. ``gamma_range`` reports the span over the whole region as + a check on rolled-detector geometries, and ``pixel_acceptance`` the one-row + case. New rocking extractions store the true primary-beam detector-arm + angles for every source frame, in radians, and evaluate the acceptance at + the frame nearest each calculated peak. This matters for rolled or oblique + detectors, where the actual-arm span can differ by several percent from the + calibrated-position span. Older databases have no arm history and retain + the calibrated-position result with an explicit warning. The rocking + integration now divides by this acceptance -- see the mode-equivalence entry + below. + +- **One structure-factor scale for rocking scans, stationary scans, and + reflectivity.** The new public module + ``orgui.datautils.xrayutils.corrections.measurement`` reduces an integrated + intensity to ``|F_hkl|^2`` for any scan mode, following E. Vlieg, + *J. Appl. Cryst.* 30 (1997) 532 and J. Drnec et al., + *J. Appl. Cryst.* 47 (2014) 365. It normalizes counts by exposure time and + monitor, converts a rocking integral from degrees to radians, applies the + mode-dependent angular factor (rocking scans additionally require the + out-of-plane acceptance of the region of interest, stationary measurements + reject it), and, given the incident flux density and the illuminated area, + puts the result on the absolute electron-unit scale. It also converts + between ``|F_hkl|^2`` and absolute reflectivity, so a reflectivity curve and + a set of truncation rods can be brought onto one scale. + ``doc/design/ctr_structure_factor_scale.md`` records the full analysis with + the measured size of every correction, and + ``doc/physics/ctr_structure_factor_physics.tex`` is a typeset reference for + the normalizations and integration intervals. - **Added opt-in incoherent CTR models for large surface height domains.** A `PoissonSurface` distributes surface heights, and by default those heights add as amplitudes: the coherent limit, in which every height lies inside one @@ -29,7 +317,6 @@ Scientific and analysis additions: `prepareFit`. A component such as a water layer may be stacked above the target surface, where it is common to every domain at the mean surface height, as in the coherent model. - - **Added live DWBA predictions to CTR fitting.** ``CTROptimizer.set_dwba`` now evaluates the optimizer-owned crystal through the semi-infinite DWBA model, forms predictions independently in each dataset's stored F or @@ -671,6 +958,30 @@ A ***critical bug*** was fixed that affects bulk CTR calculations: exposure bounds already read each segment's own arm, and they now agree with the arm the rest of the program sees. +- **Reducing a rocking scan now uses the detector geometry stored with that + scan**, instead of whatever calibration the application happens to hold. + *This changes saved numbers for any reduction run against a different + calibration than the one the curves were extracted with.* The out-of-plane + acceptance and the solid-angle compensation are properties of the geometry + the data was taken with, so reducing from a script that has not loaded the + matching configuration, or after another calibration was opened, silently + produced a wrongly scaled ``F2_hkl``: on a real scan it scaled every + acceptance by 2.3, with nothing in the output to show for it. A scan that + stores no detector geometry warns and is left on the acceptance-blind scale + rather than using the wrong one. + +- **Integrating a scan whose rod leaves the detector no longer fails.** Frames + where the rod never reaches the detector carry region positions that are not + finite; the solid-angle and polarization corrections rejected those with + ``OverflowError`` or ``ValueError`` and aborted the integration. They now + yield a neutral factor for such frames, which carry no counts anyway. A + region size that is finite but not positive is still an error. + +- **The command-line interface starts again.** ``--cli`` selected the Qt + ``minimal`` platform plugin, which has no font database, so loading the + icon font raised ``FontError`` before any batch script could run. It now + uses ``offscreen``, and an explicitly set ``QT_QPA_PLATFORM`` is respected. + GUI changes: - The "Scan data" and "Reciprocal space navigation" panels no longer reserve @@ -738,6 +1049,25 @@ GUI fixes: opened. Both dialogs apply every edit immediately, so restoring the widgets alone left the edited values active, and the discarded configuration stayed in use until it was overwritten or a config file was loaded. +- The footprint correction dialog now fits a normal screen. Both the + analytical-shape and the measured-profile groups used to be given layout + space at once, only one of them disabled, and everything was stacked in a + single column below a full-width schematic image and a 220-pixel preview + plot -- together taller than most displays. It is now two columns, the + beam and sample inputs on the left and the sample position and preview on + the right; the inactive beam-model group is hidden rather than merely + disabled, so it stops reserving space it is not using; the schematic image + is capped; and the preview plot is a little shorter. +- Loading a stored configuration from the database now refreshes the ROI and + reflection overlays and the Q-plot. ``mu``/``chi``/``phi``, the UB matrix + and the detector geometry were already updated correctly and immediately -- + every live angle and HKL readout reads them fresh rather than from a cache + -- but nothing told the plot to redraw, so it kept showing reflection + markers, ROI positions and the reciprocal-space conversion computed from + the geometry active *before* the load, which looked like the angle-to-HKL + conversion itself had not updated. Loading a configuration now emits the + same replot request every interactive machine-parameter, crystal-parameter + or U-alignment change already does. ESRF ID31 beamline support and reciprocal-space display: @@ -1141,6 +1471,47 @@ Reciprocal-space reconstruction: times the headroom actually needed and still falls back to the heap if a block ever exceeds it, so this bounds memory, never correctness; mapped output is unchanged, verified record for record. +- **A reciprocal-space mapping run no longer stops partway through when + automatic mode retunes its thread split.** Changing the native threads + per image needs a whole new compute pool rather than a resize, and the + replacement used to be started before the outgoing one was retired. + Both then drew from the same queue, so the outgoing pool's shutdown + signals were picked up by the incoming workers, which stopped + immediately -- leaving the run with no compute workers, no error and no + progress, indefinitely. Affected runs hung at whichever frame the + retune landed on, typically within the first minute. The outgoing pool + is now fully retired before its replacement starts, and any leftover + signals are cleared while nothing is reading the queue. Mapped output + is unaffected. +- **Reciprocal-space mapping now discards a whole block of detector at a + time when it cannot reach the output volume.** The kernel could already + prove that an individual pixel misses the grid, but only one pixel at a + time, so every frame paid that proof for all 6.2 million of them -- + including the roughly half of a full rotation whose frames reach the + selected volume nowhere at all. The same test now runs once per work + block first, and a block it rejects is never opened. On the reference + job a frame that contributes nothing became 5.3x faster to reject and a + frame that does contribute 1.25x faster, 1.46x averaged over the scan; + a block that cannot be rejected costs about 0.1% extra. Mapped output + is identical, checked record for record on 21 frames across the scan. + The profile reports `skipped_bricks`, and `valid_pixels` now counts + only pixels actually visited. Applies to flat detectors without a + distortion spline. +- **A reciprocal-space mapping run no longer reads frames that cannot + reach the reconstruction volume.** Whether a frame's detector can reach + the selected volume at all depends only on the exposure's angles, the + detector geometry and the volume, so it is now decided for the whole + scan before the first frame is opened. A frame ruled out costs no read, + no correction and no mapping; on the reference job, a small volume + crossed by a full rotation, 1516 of 3651 frames are ruled out, and + deciding that for the whole scan takes 0.084 s. The test only ever + rules a frame *out*, never in, so a frame that is kept costs exactly + what it did before. Every one of those 1516 frames was mapped to + confirm it produces no records. Skipped frames still count toward their + checkpoint, so resuming an interrupted job is unaffected, and progress + still reaches 100%. A volume that no frame reaches now fails with the + existing empty-result error rather than mapping nothing. Set + `ORGUI_NO_FRAME_SKIP=1` to disable. ## [1.5.0] (2026-06-07) diff --git a/doc/design/ctr_stage0_contract.md b/doc/design/ctr_stage0_contract.md new file mode 100644 index 0000000..2f66fe4 --- /dev/null +++ b/doc/design/ctr_stage0_contract.md @@ -0,0 +1,86 @@ +# CTR targeted implementation: Stage 0 contract + +Status: implementation note, 2026-09-16. Numerical behavior was captured at +commit `d52ab8f5234e`, before Stage 1 of +[`ctr_targeted_implementation_plan.md`](ctr_targeted_implementation_plan.md). + +This note records what the characterization fixture means and which metadata +is not available. It does not claim that the frozen numbers independently +validate the physics. + +## Frozen extraction behavior + +`orgui/app/test/fixtures/ctr_stage0_contract.json` and +`test_ctr_stage0_contract.py` pin two compact cases: + +- A stationary extraction with detector counts and one-sigma count errors, + radian diffraction angles, an exposure-times-monitor divisor, and the + current stationary Lorentz-to-`F2_hkl` conversion. +- A mechanical rocking extraction with a degree-valued motor axis, two + background regions, per-frame normalization inside the angular integral, + radian detector acceptance, and the current relative `F2_hkl` scale. + +The fixture also pins a pre-typed correction-settings dictionary. Its sample +dimensions are meters and `beam_flux_density` remains photons per second per +square meter. Missing fields remain missing; zero is not substituted for an +unknown calibration. Existing tests in `test_config_data.py` and +`test_peak1Dintegr.py` cover typed NeXus round trips, legacy JSON database +dispatch, and both stored correction-group layouts used by the reducer. + +At Stage 0, strict expected-failure tests recorded the two Stage 1 defects +without changing application code: the arm-following polarization evaluator +ignored a nonzero polarization axis, and rocking acceptance ignored actual-arm +angles. Stage 1 converts those reproductions into passing regressions. + +## Settings-loader audit + +All `examples/config_*` files use the legacy INI loader. They define detector, +lattice and diffractometer values; none selects a normalization monitor or +stores integration-correction provenance. `ConfigData.from_ini` therefore +starts with an empty `CorrectionState`. Current database snapshots write the +typed `integration_corrections` layout (schema version 2), while +`ConfigData.from_nxdict` still recognizes the older group containing one +opaque JSON dataset. Neither old layout distinguishes rate-like from +frame-integrated monitors. + +## Backend counter-semantics audit + +The implementation assumption is that a selected incident-beam monitor is +**rate-like**. It is therefore multiplied by exposure time in the fluence +divisor. This matches the current normalization arithmetic. The present +backend contract names arrays but does not persist a counter kind or unit, so +the assumption must remain visible in provenance until a beamline-specific +calibration establishes the counter-to-total-flux conversion. + +| Backend | Persisted candidates | Meaning established by code | Rate/integrated status | +|---|---|---|---| +| Base `Scan`, imported images | none | no monitor metadata | unavailable | +| Built-in P212 | implicit `exposure_time` when present | frame exposure, seconds by loader convention | not a flux monitor; P212 flux counters are unavailable | +| ID31 fast/BLISS variants | `exposure_time`, `time`/`elapsed_time`/`epoch`, `srcur`, `mondio`, plus electrochemistry values | exposure is seconds; time/epoch are coordinates; `srcur` and `mondio` are often divided by their scan mean and are therefore dimensionless relative factors | selected beam monitors are treated as rate-like relative-flux proxies | +| Example CHESS QM2 | `diode`, `ic1`, `ic2`, `emon`, `nemon`, `pemon` | raw SPEC columns; `ic2` is ion chamber 2 and is the expected primary incident-beam monitor | `ic2` is treated as rate-like; a later calibrated factor converts its reading to total photons/s | +| Example P212 | `beckvolt_1`, `beckvolt_2`, `eh3_entrance`, `oh2_diode1`, `oh2_diode2`, `petracurrent`, `timestamp` | raw scan columns; timestamp is not a flux monitor | a selected beam monitor is treated as rate-like; its calibration is unresolved | +| Segmented/interlaced scans | intersection of segment counters, plus implicit exposure time | values are concatenated/reordered; semantics are inherited | selected monitors remain rate-like; units and calibration are not merged | + +`current`, `potential`, `scaled_potv2f`, motor positions and timestamps are not +incident-flux monitors merely because they appear in `auxillary_counters`. +Until Stage 3 adds an explicit monitor unit and calibration relation, the +legacy behavior is exactly exposure time times the selected rate-like monitor +product recorded by `monitor_corrections`; it must be shown as legacy rather +than silently assigned an absolute photon scale. For QM2, the intended primary +monitor is `ic2`, with a future scalar calibration relating its rate to total +incident flux in photons/s. + +## Stage 1 inputs now pinned + +- Polarization: at the calibrated arm position, + `polarizationAtPoints` must match `polarizationArray` for nonzero axes and + mixed polarization fractions. The axis-zero baseline already passes. +- Acceptance: scan arm values are degrees at the storage boundary and must be + converted to radians before calling `out_of_plane_acceptance`. The fallback + for old scans with no arm values must remain explicit. + +The rolled-detector case uses the published review parameters: 172 micrometer +pixels, 0.15 meter distance, 30 degree detector roll, ROI center `(row, +column) = (500, 440)`, height 60 pixels, incidence 10 degrees and actual +`(gamma, delta)` arm angles `(30, 40)` degrees. It is a counterexample to arm +invariance, not a statement about a typical experimental error. diff --git a/doc/design/ctr_stage2_persistence_contract.md b/doc/design/ctr_stage2_persistence_contract.md new file mode 100644 index 0000000..6ad24fe --- /dev/null +++ b/doc/design/ctr_stage2_persistence_contract.md @@ -0,0 +1,62 @@ +# CTR Stage 2 persistence contract + +Status: implemented, 2026-09-16. + +Stage 2 introduces persistence and dispatch boundaries only. It deliberately +does not enable the total-flux calculation or move normalization/footprint +between extraction and reduction. Existing numerical defaults and legacy +dataset meanings remain unchanged. + +## Requested settings + +`configuration/orgui/integration_corrections` now has schema version 3. The +legacy `beam_flux_density` field remains photons/(s m2). The distinct +`total_incident_flux` field is photons/s and is never inferred from that +density. Primary-monitor name, rate/integrated kind, unit, calibration reading +and calibration exposure have separate fields. Horizontal interception is an +explicit mode plus optional fraction. Missing values mean unknown or not +configured; they do not mean zero flux, full interception or a unity divisor. + +Measured vertical profiles store their normalized positions [m] and density +[1/m] in addition to the source path and import settings. Analytical profiles +remain reproducible from their named shape and parameters. + +## Applied curve record + +New extractions retain all established groups and add a sibling named +`ctr_curve_v3`. It is not named `rois` or `counters`, so an older reducer cannot +mistake this branch for the legacy unnormalized input. The record separates: + +- identity: algorithm, output quantity and scale convention; +- normalization: explicit status, exact divisor and contributing counters; +- illumination: explicit status, exact divisor, named legacy/new convention, + overlap components and alpha in radians when available; +- base: center/background sums and variances plus the reversible background- + subtracted scalar curve and variance before normalization/illumination; +- pixel corrections: the established combined center/background factors, + with separate polarization-only fields reserved for Stage 4; +- geometry: actual detector arms in radians and ROI location/size in pixels; +- profile: embedded profile provenance rather than only an external path. + +Correction statuses are exactly `applied`, `not_applied`, `unavailable` or +`unknown`. An absent dataset is never read as unity. + +Stationary records capture the scalar curve immediately before the existing +normalization and active-area divisors. Rocking extraction currently applies +neither divisor, so both statuses are explicitly `not_applied`; its current +legacy reducer remains responsible for them until Stage 5. + +## Compatibility and dispatch + +The existing `rois`, `croibg`, `Cfactors_croi` and `Cfactors_bgroi` datasets +retain their meanings. Current Stage 2 records identify their algorithm as +`legacy_stationary_roi_v1` or `legacy_rocking_roi_v1`, allowing the existing +paths to reproduce the same numbers. + +The rocking loader checks the sibling record before returning the legacy ROI +curve. A record with a non-legacy algorithm (for example the later framewise +total-flux convention) is refused by the legacy reducer instead of being +silently reinterpreted. Incomplete older records remain accessible through the +legacy diagnostic path with unknown provenance. Reduction outputs continue to +be added as new measurement groups, so re-reduction does not overwrite the +source extraction record. diff --git a/doc/design/ctr_stage3_total_flux_contract.md b/doc/design/ctr_stage3_total_flux_contract.md new file mode 100644 index 0000000..b123825 --- /dev/null +++ b/doc/design/ctr_stage3_total_flux_contract.md @@ -0,0 +1,81 @@ +# CTR Stage 3 total-flux numerical contract + +Status: implemented, 2026-09-16. + +Stage 3 adds pure numerical building blocks for calibrated total incident flux. +It does not change either integration path, the current GUI/config behavior or +the legacy flux-density APIs. Stage 5 will select and persist this convention +during extraction. + +## Per-frame incident photons + +`corrections.normalization.frame_fluence` returns `Q_f` in photons. A constant +total flux `Phi` [photons/s] gives `Q_f = Phi T_f`. A rate-like primary monitor +uses + +```text +Q_f = Phi_ref T_f M_f / M_ref. +``` + +An integrated monitor already contains the exposure and instead uses + +```text +Q_f = Phi_ref T_ref U_f / U_ref. +``` + +There is no second multiplication by `T_f` in the integrated case. Monitor kind +is explicit; the numerical core does not infer it from a counter name or unit. +The beamline counters expected for the first wiring pass, including QM2 `ic2`, +are rate-like after applying their configured conversion to total incident +flux. `relative_frame_fluence` applies the same exposure rule without claiming +photon units when no absolute calibration exists. + +All fluxes, exposures, monitor readings and calibration references must be +finite and strictly positive. Calibration references correspond to the stated +reference flux; no scan-mean normalization is introduced. + +## Beam/sample illumination + +For a normalized vertical beam profile, sample length `L` [m], incidence angle +`alpha` [rad] and explicit horizontal intercepted fraction `f_x`, + +```text +f_z = profile.flux_on_sample(alpha, L) +f_hit = f_z f_x +H = f_hit / sin(alpha). +``` + +`corrections.activearea.intercepted_fraction` returns `f_hit`, while +`illumination_divisor` returns the dimensionless `H`. These are different +quantities: photons physically hitting the sample are `Q_f f_hit`, whereas the +retained CTR reduction divides counts by `Q_f H`. + +`BeamProfile.flux_over_sine` evaluates the ratio directly. Built-in measured, +distribution and analytical Gaussian profiles continue it to the finite +grazing-incidence limit, avoiding numerical zero divided by zero. Existing +external profile subclasses remain instantiable through a non-abstract default +implementation. + +The vertical profile supplies no horizontal geometry. `f_x` is therefore an +explicit number in `(0, 1]`; one means the full horizontal beam is intercepted. +No general two-dimensional clipping model is implied. + +## Structure-factor scale + +`corrections.measurement.total_flux_prefactor` returns + +```text +K = r_e^2 lambda^2 / A_u^2, +``` + +with wavelength in Angstrom and surface unit-cell area in square Angstrom. +`structure_factor_squared_from_photon_yield` reduces +`Y = N_net / (Q_f H)` with the existing scan-mode angular factor and optional +detector efficiency. `photon_yield_from_structure_factor` is its exact forward +inverse. + +The total-flux identity is pinned independently for a separable 2-D Gaussian: +total flux times `H` equals peak flux density times the existing effective beam +area. Thus `Q H` replaces the legacy density-times-area fluence; it is not an +extra footprint correction. `scale_factor`, `structure_factor_squared` and +`integrated_intensity` retain their legacy density semantics unchanged. diff --git a/doc/design/ctr_structure_factor_handover.md b/doc/design/ctr_structure_factor_handover.md new file mode 100644 index 0000000..8189c82 --- /dev/null +++ b/doc/design/ctr_structure_factor_handover.md @@ -0,0 +1,346 @@ +# CTR structure-factor scale: implementation status and handover + +> **Status as of 2026-09-11.** Branch `claude/ctr-structure-factor-9633bc`, +> nothing pushed. `git log --oneline master..HEAD` is the commit list; a +> count written here goes stale on the commit that writes it. +> +> The physics analysis is complete and quantified, and the reduction is now +> **wired in**: a rocking scan and a stationary scan of the same rod come out +> with the same `F2_hkl`, asserted by +> `test_scan_mode_equivalence.py::test_rocking_and_stationary_paths_agree`. +> This **changed saved numbers in both modes**. The **real-data check has now +> been done** — LaNiO3 scan 61, both modes run from the raw images, agreeing +> to a median 1.033 along the CTR; see +> [`ctr_structure_factor_scale.md`](ctr_structure_factor_scale.md) section +> 5.1. [#82](https://github.com/tifuchs/orGUI/issues/82) is closed up to +> `C_det` (F7), which that check bounds but does not model. +> [#15](https://github.com/tifuchs/orGUI/issues/15) still needs the two +> absolute-scale inputs of section 6. +> +> This document is the handover: what exists, how to run it, what to do next, +> and which of my predictions turned out wrong. The physics itself is in +> [`ctr_structure_factor_scale.md`](ctr_structure_factor_scale.md) — read that +> first for *why*; this one is *where things stand*. + +## 1. The one-paragraph summary + +A rocking scan and a stationary scan of the same rod used to differ by exactly +`T_omega * monitor_omega * Delta_gamma_in_degrees`, measured to seven digits on +simulated data. Three things were missing from the rocking path: +exposure/monitor normalization, integration in radian rather than degrees, and +division by the out-of-plane detector acceptance. A fourth, F6, affected +**both** modes: a ROI sum is already a complete angular integral, so the +solid-angle correction had to be divided back out of `F2_hkl`. It stays +applied to the *intensity*, where a broad or diffuse feature needs it. All +four are now handled, and the two modes agree to `1e-6` on simulated data — +the residual is the trapezoidal sampling of the rocking profile, not a +correction factor. + +F5, the arm-blind polarization, is fixed too, in its own commit. It is +independent of mode equivalence — it cancels between the modes at the same +reflection — but it is up to a 33 % error on a scan that drives the detector +arm, which is exactly the reflectivity case. + +## 2. What is on the branch + +| commit | what it did | +|---|---| +| `777d887` | `refactor: collect correction factors into one corrections package` | +| `4193c80` | `feat: reduce integrated intensities to \|F_hkl\|^2 on one scale` | +| `e25b8df` | `docs: record the rocking/stationary structure-factor scale analysis` | +| `6959191` | `feat: estimate the out-of-plane detector acceptance` | +| `ecb3bb5` | `docs: settle F6 and add the structure-factor physics reference` | +| `0af9cb7` | `feat(phys)!: put rocking and stationary integration on one structure-factor scale` | +| *(this one)* | `fix(phys)!: follow the detector arm in the polarization correction` | + +The first three were split out of one working tree at the end, which required +*staged versions* of four files: `peak1Dintegr.py`, `integration_corrections.py`, +`corrections/__init__.py` and `test_corrections_package.py` all reference +`measurement`, which only exists from `4193c80`. In `777d887` they keep the old +inline Lorentz selection with only the import path updated. Each commit was +verified green on its own, not just the tip. + +### 2.1 The package + +``` +orgui/datautils/xrayutils/corrections/ + geometry.py 219 z-axis Lorentz x3, rod interception, area factor + beamprofile.py 1049 beam profile shapes and their integrals + activearea.py 190 active area in m^2, slit- and beam-limited + acceptance.py 255 Delta_gamma, gamma_range, pixel_acceptance + detector.py 319 per-pixel solid angle and polarization, their + region means, and the polarization arm correction + normalization.py 103 counting time and monitor, from values + roi.py 93 CorrectionFactors, roi_mean_correction + measurement.py 605 mode dispatch, master equation, reflectivity +``` + +`orgui/datautils/xrayutils/geometrycorrections.py` and `beamprofile.py` remain +as re-export aliases because both were released under those names. +`test_corrections_package.py` asserts the aliases hand out the *same objects*, +not merely that they import. + +### 2.2 What is wired + +| caller | uses the package for | still does its own thing | +|---|---|---| +| `orGUI.integrateROI` (stationary) | `pixel_factors`, `mode_components`, `normalization_divisor`, `C_illum_area`, `roi_mean_inverse_solid_angle`, `polarization_arm_correction` | — | +| `orGUI.rocking_integrate` | `pixel_factors`, `polarization_arm_correction` | — | +| `peak1Dintegr.integrate` (rocking) | `mode_components`, `normalization_divisor`, `normalized_intensity`, `out_of_plane_acceptance`, `roi_mean_inverse_solid_angle` | — | +| `reconstruction_job` | `pixel_factors` (solid angle applied and **not** compensated; polarization **not** arm-corrected) | own native-fused application, own normalization loop | + +Still uncalled outside the tests: `measurement.structure_factor_squared`, +`measurement.angular_factor` and `activearea.*`. That is deliberate rather +than a gap — the two integration paths form `F2_hkl` on a *relative* scale by +dividing out the mode-dependent factors they already hold as +interval-weighted means, and `structure_factor_squared` additionally divides +by the absolute prefactor, which needs the issue #15 inputs. `angular_factor` +rebuilds `eta` from point angles, which is the wrong thing for a path that +has ROI-weighted means of each component. + +The rocking path could not use `normalization_divisor(scan, ...)` as the +stationary path does: it runs off the database and has no scan object. It +builds the divisor from the stored `auxillary` counters instead, via +`_rocking_normalization`, using the same monitor-name setting +(`reconstruction_monitor_corrections`) the other two paths use. Exposure time +is only there if the backend declares `exposure_time` in +`auxillary_counters` — ID31 does, `P212_tools` and the base `Scan` do not, and +a missing counter is skipped and recorded rather than failing the job. + +## 3. Getting a green test run + +**A fresh checkout reports ~80 failures.** They are all the unbuilt native +extension, not the branch. With the extensions built the suite is +**1120 passed, 3 skipped, 0 failed** (the 3 skips are a missing `arviz`). +Building also *unlocks* roughly 230 tests that are not collected at all +without it, so the unbuilt number is not simply "the green ones". + +The environment on this machine is Python 3.14.7 (miniforge) with a working +orGUI already installed in `miniforge3\Lib\site-packages`. **Do not +`pip install -e .`** — it rebinds `import orgui` for that interpreter and the +repository owner wants the installed copy left alone. Build out of tree and +stage instead: + +```powershell +# build dir must be SHORT: a path under the session scratchpad overruns +# MAX_PATH and meson dies in check_clock_skew with a FileNotFoundError +$vc = 'C:\Program Files\Microsoft Visual Studio\18\Community\VC\Auxiliary\Build\vcvars64.bat' +$mf = 'C:\Users\timof\miniforge3' +$pre = "$mf;$mf\Scripts;$mf\Library\bin" +cmd /v:on /c "`"$vc`" >nul 2>&1 && set PATH=$pre;!PATH! && meson setup C:\Users\timof\obuild --buildtype=release" +cmd /v:on /c "`"$vc`" >nul 2>&1 && set PATH=$pre;!PATH! && meson compile -C C:\Users\timof\obuild" +cmd /v:on /c "`"$vc`" >nul 2>&1 && set PATH=$pre;!PATH! && meson install -C C:\Users\timof\obuild --destdir C:\Users\timof\ostage" + +cd C:\Users\timof\ostage\Lib\site-packages +python -m pytest orgui/app/test orgui/datautils/xrayutils/test -q +``` + +Four traps, each of which cost an attempt: + +1. `cl` is not on PATH; `vcvars64.bat` is required. It prints a harmless + `'vswhere.exe' is not recognized` line and works anyway. +2. **`%PATH%` expands at parse time.** `vcvars && set PATH=;%PATH%` + silently discards everything vcvars added, and meson then reports + `Unknown compiler(s)`. Hence `cmd /v:on` and `!PATH!`. +3. meson and python are not on cmd's PATH even though they are on the bash + shell's. The interpreter is in the miniforge root, the entry points in + `Scripts`. +4. MAX_PATH, as above. + +Do **not** copy the built `.pyd` into `orgui/` to make the worktree importable. +`install_subdir('orgui', ...)` ships whatever is in the source tree, so a stray +extension is then packaged over every future build — a trap this repository has +already been bitten by once. + +`meson install --destdir` cannot write outside the staging tree, so the +installed copy is provably untouched; verify by checking that +`corrections/` is *absent* from `site-packages\orgui\datautils\xrayutils`, +not by comparing mtimes. + +## 4. Rules this work established + +Two of these came from the repository owner during the session and are now +also in the `AGENTS.md` files. They are the constraints a follow-up must keep. + +* **`datautils` holds self-consistent physics modules. No UI or UI state may + leak in.** No widget, no configuration object, and no *scan object* in + `orgui/datautils/`. This is why `corrections/normalization.py` takes + exposure and monitor *values* while `app/integration_corrections.py` keeps + the `scan`-attribute lookup, and why the `use_lorentz` / `use_footprint` + switches stayed on the app side. +* **One definition per factor.** Before this branch the mode→factor mapping + lived in four places, the active area in three, the exposure/monitor divisor + in three, and the per-pixel array in three verbatim copies. When adding a + correction, put the physics in `corrections/` and call it; do not compute a + factor inline in `orGUI.py` or `peak1Dintegr.py`. +* **Behaviour-preserving moves are verified against a failure-count baseline, + not by inspection.** The refactor was checked at 80 failed / 782 passed + before and after. + +## 5. The wiring commit, as landed + +`feat(phys)!` — it changed saved numbers in **both** modes. What it did: + +1. **Normalization.** `peak1Dintegr.RockingPeakIntegrator._rocking_normalization` + builds the per-frame exposure/monitor divisor from the stored `auxillary` + counters (see section 2.2 for why not from a scan object) and it is applied + **inside** the rocking integral, not to the finished integral: with a + varying counting time or a drifting monitor the quantity Vlieg integrates is + `Int N(omega)/(T M) d omega`, and dividing the result by a mean is only + equivalent for constant counters. It rides the existing `C_corr` machinery, + which already carried `C_illum_area` per `(s, omega)` point, so the error + propagation followed for free. +2. **Radian.** `measurement.normalized_intensity(..., angle_unit="deg")` + converts when `F2_hkl` is formed. The stored `croibg` and `int_interval` + stay in the unit they were measured in — changing those would change the + meaning of two saved columns for no gain. +3. **Acceptance.** `_rocking_acceptance` calls `out_of_plane_acceptance` with + the region centre and vertical size stored per `s` point. Two traps here: + the coordinate order (`surfaceAnglesPoint` takes pyFAI dimension 1 first, + which is the *row*, and orGUI's `y` is the row while `x` is the column — + every call site in the application passes them swapped), and `vsize` being + the row extent, which follows from `detvsize, dethsize = detector.shape` + at `orGUI.py:1096`. It is evaluated at the calibrated arm position, which + costs nothing measurable because the span is arm-invariant to nine digits. + With no calibrated detector reachable it warns and leaves `F2_hkl` on the + acceptance-blind scale rather than failing the integration. +4. **Solid angle compensated, not removed.** The switch, its config key and + its badge are untouched, and it still scales the intensity — a broad or + diffuse feature wants a differential cross-section, and that capability was + worth keeping. What changed is that `F2_hkl` divides it back out, via the + new `detector.roi_mean_inverse_solid_angle`, measured over the same regions + from the calibrated geometry alone (no image, so it runs outside the + integration loop). Two things to know: the compensation is approximate at + the `1e-6` level, because `pixel_factors` fuses the solid angle with the + polarization so the applied mean is `<1/(Omega~ P)>` rather than a product + of means; and for a rocking scan the correction was applied at *extraction* + time, so whether to compensate is read from the configuration stored with + the scan (`configuration/orgui/integration_corrections/json`) rather than + from the current switch. An older database where that cannot be read warns + and is left uncompensated rather than guessed at. + + A related trap, found by breaking it first: `useSolidAngleBox` is *also* + the store the reconstruction persists its own solid-angle setting through + (`config_data.py:383`/`:467` map the `solidAngle` integration option onto + `CorrectionState.use_solid_angle`, and `ReconstructionDialog` mirrors its + checkbox via `scanSelector.get/set_integration_options`). Removing the key + would have silently disabled the correction in the one path that must keep + it, and that is invisible from either file alone. +5. **Provenance.** A `reduction` group beside `F2_hkl` records the mode, the + angle unit, which normalizations applied, whether the acceptance was + applied, whether the solid-angle correction was compensated, the + active-area assumption, and the acceptance array itself. + +`test_rocking_and_stationary_paths_differ_by_the_missing_normalizations` +became `test_rocking_and_stationary_paths_agree`, and +`test_a_resized_region_of_interest_distorts_the_rocking_rod` became +`..._no_longer_distorts_...`. Both keep the *old* behaviour as a contrast +assertion via a `reduce=False` switch on the helper, so the factor that used to +be left behind cannot come back unnoticed. + +## 6. After that + +* **Absolute scale (#15)** needs two user inputs that have no config field yet: + the incident flux density `Phi_0`, and the horizontal beam width for + `activearea.beam_limited_area`. The footprint dialog already asks for the + sample size and the beam profile. Everything else — `lambda`, `A_u`, `r_e` — + is available. +* **Reflectivity comes for free** once `|F|^2` is absolute; + `measurement.reflectivity_from_structure_factor` is the conversion. **F5 is + now fixed**, which mattered most here: a reflectivity scan drives the arm, + and the arm-blind polarization was a 10 % error at `2theta = 18` degrees and + 33 % at 30. The correction applies to the two direct-space integration + paths; the reciprocal-space reconstruction still evaluates the polarization + per pixel at the calibrated position. +* **`C_det` (F7)** is the only mechanism that can still break mode equivalence + after the wiring, and it cannot be validated on simulated data. The + real-data overlap comparison has now been run and **bounds** it: not + detectable against 3 % scatter along a CTR whose regions cover the peak, a + factor of 5 at the Bragg peak on the same rod. Modelling it is still open, + and the bound is an upper limit for one rod on one sample, not a general + result. + +## 7. Predictions that turned out wrong + +Recorded because each cost time and none was obvious in advance. + +* **`active_area_footprint` should live next to the existing area code.** It + should not exist at all. `w * min(L, h/sin(alpha))` is exactly + `w * L * illuminated_area_fraction(alpha, L)` for a top-hat profile and only + for that profile — verified to `3.3e-16`. A Gaussian of the same width + departs by up to 19 %. The function was deleted, not moved. +* **"The refactor will remove ~100 lines from `orGUI.py`."** It is 27 added, + 10 removed — net *longer*, because the explanatory comment is worth more + than the ten lines saved. Real length reduction there means extracting the + `integrateROI` / `rocking_integrate` driver bodies, which is separate work. +* **"A moving detector arm changes `Delta_gamma`."** It does not. Driving the + gamma arm shifts the exit angle at the ROI centre by the full arm angle but + leaves the *span* identical to nine digits, because the rotation is about the + axis gamma is measured around. A 40-degree delta-arm rotation moves it by one + part in `1e5`. This is the opposite of the polarization factor, and it means + an acceptance computed without arm bookkeeping is still usable. +* **"The run-to-run test-count drift is flakiness."** It was the interpreter + changing under the session when Python 3.14 was installed mid-work. +* **`if C_arr is None: C_arr = np.ones(...)` in `orGUI.py` is dead code.** It + is load-bearing: the branch that rebuilds `C_arr` sits inside `if HAS_ACCEL:`, + so the NumPy-only path would be handed `None`. It is commented as such now. +* **pyFAI accepts mismatched coordinate array shapes.** It asserts + `pos2.size == size`, so a scalar column with a per-frame region height — what + the rocking path will pass — dies inside the extension. + `acceptance._surface_gamma` broadcasts before the call. + +## 8. Deliberately not done + +* **The reconstruction's correction pass was not unified.** It is fused into + `apply_correction_factors` with a bit-for-bit native/NumPy contract and + variance propagation. It shares the *definition* (`pixel_factors`) but keeps + its own streaming application; that was the right boundary. +* **F5 was not fixed while moving the code.** `pixel_factors` reproduces the + historical arm-blind behaviour exactly. Changing it was a numerical fix that + belonged in its own `phys` commit, not smuggled into a refactor -- and it + landed as one. Note what it did *not* need: rebuilding the per-pixel array + per frame. Because the integration paths reduce the polarization to a region + mean anyway, the fix is a per-frame ratio of two region means, which is + exactly 1 at the calibrated arm position and so leaves every fixed-arm scan + bit-identical. `pixel_factors` itself is unchanged. +* **`meson.build` was not changed.** Its `exclude_directories: ['__pycache__']` + only excludes the top-level directory, so 72 stale `cpython-312.pyc` files + are sitting in the installed copy. Inert under 3.14, and the repository owner + had previously declined a `meson.build` change for the related stale-`.pyd` + problem, so it was left alone and raised separately. + +## 9. Reproducing the key numbers + +```powershell +pytest orgui/datautils/xrayutils/test/test_corrections_measurement.py +pytest orgui/datautils/xrayutils/test/test_corrections_acceptance.py +pytest orgui/datautils/xrayutils/test/test_corrections_activearea.py +pytest orgui/datautils/xrayutils/test/test_corrections_package.py +pytest orgui/app/test/test_scan_mode_equivalence.py +``` + +These five do not need the native extension. Everything else in the suite may, +so use section 3 before concluding anything from a failure. + +The real-data check of `ctr_structure_factor_scale.md` section 5.1 needs data +that is not in the repository (LaNiO3 scan 61). To repeat it on another pair +of scans, run orGUI with `--nogui -i