Skip to content

Plate-solve the full sensor, with a cached optical calibration - #554

Open
mrosseel wants to merge 2 commits into
brickbots:mainfrom
mrosseel:fullframe-solve
Open

Plate-solve the full sensor, with a cached optical calibration#554
mrosseel wants to merge 2 commits into
brickbots:mainfrom
mrosseel:fullframe-solve

Conversation

@mrosseel

Copy link
Copy Markdown
Collaborator

Two changes to the solver, both aimed at the same thing: stop throwing away sky, and stop rediscovering the optics on every frame.

1. Solve the full sensor

The camera cropped every exposure to a centred square before solving — on the imx462, 1920×1080 down to 980×980, so roughly half the sensor area was discarded. Cedar Solve has no trouble with the whole frame, so that area is free plate-solving sensitivity.

The camera now publishes two frames per exposure:

Size Consumed by
display frame 512×512 square crop, unchanged UI, focus, preview, align, SQM photometry
solve frame whole sensor, native scale the plate solver only

Bayer sensors bin 2×2 for the solve frame, which removes the RGGB modulation the old downscale to 512×512 was implicitly smoothing away.

Everything downstream of the solver is written against the display frame — SQM in particular scales matched centroids onto the raw photometry image and derotates them, a chain that only works starting from display-frame pixels. So a full-frame solve is projected back onto the display frame in one place (project_solution_to_display), and the projected copy is what the rest of the system sees. Matched stars outside the crop are dropped along with their catalogue counterparts: they contributed to the solve, which is the point, but they have no pixels in the crop to measure. SolveGeometry owns the mapping, composed from the affine transform of each pipeline stage.

Gated by solver_full_frame (default on), and disabled automatically when the camera has no sensor profile or the frame rotation is not a quarter turn — rotating a non-square frame by an arbitrary angle would clip exactly the corners this exists to recover.

2. Cache the optical calibration

The solver passed a fixed fov_estimate=12.0, fov_max_error=4.0 and never passed a distortion estimate at all, so Cedar re-derived both on every frame from scratch.

The first successful solve of a run now captures the measured FOV and lens distortion; later solves get a ±0.5° window and that distortion as a starting point. This speeds up pattern matching and improves centroid matching toward the frame corners — the part of the image the crop used to discard, and the part where lens distortion actually bites. A run of consecutive failures discards the calibration and reopens the window. Session-scoped; nothing is persisted.

What the field of view actually does

Worth stating, because it is counterintuitive: after the camera's quarter turn the solve frame's horizontal axis is the sensor's short side, which the square crop already spanned. So horizontal FOV grows only modestly (imx462 ~10.2° → ~11.2°) or not at all (imx296, HQ). Total area still roughly doubles — the extra sky arrives vertically. The FOV search window is therefore derived per camera from the known crop field rather than hard-coded.

Cost

Benchmarked over 220 archived sweep frames (imx462 and HQ), exposures 25 ms – 1.1 s, on an x86 dev box:

pixels extract solve total
512×512 crop (today) 0.26 Mpx 1.04 ms 6.5 ms 7.6 ms
full-sensor pixel count 0.52 Mpx 1.02 ms 4.9 ms 5.9 ms

2.0× the pixels cost −0.02 ms; 2.9× (HQ) cost −0.28 ms — nothing, within noise. Total extract+solve is 0.6–8% of one exposure cycle depending on exposure, so the CPU budget is not the constraint here.

Caveat on that benchmark: the sweep archive stores cropped raw, so the full-sensor arm is the real crop padded with its own background noise to the right pixel count. It measures the pixel cost honestly and the sensitivity benefit not at all — that needs a device run.

Testing

ruff clean, mypy clean on changed files, 1052 unit+smoke tests pass.

  • Unit tests drive a marker through both real pipelines for all four sensor profiles × both rotations, checking the affine mapping against what the code actually produces
  • Integration tests run a real Cedar solve of a test frame embedded in a larger canvas: RA/Dec/Roll match the crop solve, projected centroids land on the same stars (median 0.11 px), FOV projection round-trips
  • Ran headless with the debug camera: solves normally, SQM unaffected

Not yet run on hardware — the debug camera has no sensor profile, so that exercises the wiring but not the full-frame path itself.

…tion

The camera cropped every exposure to a centred square before solving, throwing
away roughly half the sensor area (imx462: 1920x1080 down to 980x980). Cedar
Solve has no trouble with the whole frame, so that discarded area is free
plate-solving sensitivity.

The camera now publishes two frames per exposure: the 512x512 display frame,
unchanged, and a solve frame covering the whole sensor at native scale. Bayer
sensors are binned 2x2 for the solve frame, which removes the RGGB modulation
the old downscale to 512x512 used to smooth away.

Everything downstream of the solver -- SQM photometry, the preview overlay, the
alignment marker -- is written against the display frame, and SQM in particular
is calibrated to its plate scale. So solves are projected back onto the display
frame in one place (project_solution_to_display), keeping those consumers
unchanged. SolveGeometry owns the mapping, composed from the affine transform of
each pipeline stage.

The solver also stopped rediscovering the optics on every frame. It passed a
fixed 12 +/- 4 degree window and never passed a distortion estimate at all. The
first successful solve of a run now captures the measured FOV and lens
distortion; later solves get a +/-0.5 degree window and the measured distortion
as a starting point, which speeds up pattern matching and improves centroid
matching towards the frame corners -- the part of the image the crop used to
discard. A run of failures discards the calibration and reopens the window.

Full frame is on by default and can be turned off with solver_full_frame. It is
disabled automatically when the camera has no sensor profile, or when
camera_rotation is set to something other than a quarter turn, since rotating a
non-square frame by an arbitrary angle would clip the corners.

The 2x2 Bayer bin is gone. It was there because the crop pipeline's downscale
had been smoothing the RGGB checkerboard away implicitly, and losing that was
assumed to bias centroids toward the green sites. Measured against 12 imx462
frames of real sky, the assumption was wrong and the bin was costing exactly
what this change exists to gain:

                  solve rate   median matches   median RMSE   solves down to
  crop (today)        8/12           15.5          15.4"         257 ms
  binned             10/12           20.0          40.5"         174 ms
  un-binned          11/12           26.0          41.1"          97 ms

Un-binned finds 30% more matches than binned and solves at 2.6x shorter
exposure than the crop -- against 1.5x for the binned version, so binning was
giving away half the sensitivity gain. Angular residuals are statistically
identical (41.1" vs 40.5"), so the centroid-bias worry does not survive
contact with data; the finer sampling is simply what lets marginal stars clear
the detection threshold.

Full frame does cost accuracy against the crop (15.4" -> 41"), from corner
distortion a single-k model cannot fully absorb. That is a real trade, though
41" is negligible against any eyepiece field, and it buys solves on frames the
crop cannot solve at all.
/api/camera/raw rendered a uint16 sensor frame with
Image.fromarray(arr, mode="L"), which reinterprets the 16-bit buffer as
8-bit. The result is interleaved-byte noise that still looks enough like an
image to be believed -- I spent a night's captures on it before checking the
histogram (median 1, p90 80, 1% at 255) and realising the frames were junk.

16-bit frames now render as mode "I;16" PNGs, preserving every ADU, and a
test pins the round trip.

Adds /api/camera/rawfull for the whole sensor including the margins the crop
discards. Naming follows the exposure sweep's TIFFs: raw is the crop,
rawfull is everything. The full frame is ~4 MB, so it is published on demand
rather than on every capture -- the endpoint asks, the camera fulfils the
request once and clears it.

Between them these make the frames the pipeline actually works on
retrievable from a running device, which is what debugging a solve failure
in the field needs and what nothing currently offers.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant