Skip to content

Acquisition and sequencing: fine-acquire settle and budget, offset classes, auto-ontarget, pick at readout, NEXP>1 stop, offset wait - #503

Open
cfremling wants to merge 10 commits into
mainfrom
feat/acquisition-sequencer-automation
Open

cfremling wants to merge 10 commits into
mainfrom
feat/acquisition-sequencer-automation

Conversation

@cfremling

@cfremling cfremling commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

Acquisition and sequencing changes that depend on each other, so they are one PR: the sequencer's wait for the target offset needs acamd's goal-change reporting and runs in the pick-at-readout exposure path, and the later sequencer changes build on the earlier ones. One commit per change, for review:

  1. fineacquire: wait until acamd reports the goal change applied (slicecamd, acamd, GOALSHIFT_PENDING in common/message_keys.h). The settle after a fine-acquire correction starts when the move has executed, not after a fixed timer.
  2. fineacquire: single-frame accept on the verification pass.
  3. acamd: classify guide, fine-acquire and deliberate-offset corrections. Goal changes carry a generation; only a correction computed for a deliberate offset may use ACQUIRE_TCS_MAX_PUTONSLIT_OFFSET, so an in-flight guide correction can no longer consume that allowance and leave the exposure on the offset star (2026-08-26, 10-06, 10-19).
  4. slicecamd: fine-acquire correction budget (FINE_ACQUIRE_MAX_ADJUSTMENTS, FINE_ACQUIRE_RELAXED_FACTOR).
  5. auto-ontarget: the TCS itself may satisfy the ontarget wait (tcsd passes ?ONTARGET through; sequencerd watches ?ONTARGET, tracking and the separation). The operator's ontarget still works and wins when first.
  6. sequencerd: pick the next target when the last readout starts, with a readout gate that holds every camerad command until READY; flexure set just before each exposure and between sub-exposures.
  7. sequencerd: auto-ontarget on by default; require the cass ring at the commanded angle.
  8. camerad, sequencerd: stop and abort an NEXP>1 series.
  9. sequencerd: expose once acamd has made the target offset (bounded at 30 s) instead of a fixed 3 s.

Config templates: ACQUIRE_OFFSET_THRESHOLD=1.0; FINE_ACQUIRE_SETTLE_SEC=1, FINE_ACQUIRE_SETTLE_FRAMES=0, FINE_ACQUIRE_MOVE_TIMEOUT=60.0, FINE_ACQUIRE_MAX_ADJUSTMENTS=3, FINE_ACQUIRE_RELAXED_FACTOR=2.0; TCS_AUTO_ONTARGET=yes, TCS_AUTO_ONTARGET_SEP=5.0, TCS_AUTO_ONTARGET_CASS=0.05, OFFSET_SETTLE_SEC=0.

To deploy: rebuild, then restart acamd, slicecamd, sequencerd, tcsd and camerad.

Tested on sky 2026-10-28/29 UT using branch cf/summit-20261020, 18 targets in 10:27–12:34 UT:

  • auto-ontarget continued 17 of 18 slews, 1.2–2.0 s after tcsd reported tracking (the operator pressed for the first);
  • all 17 target changes were picked at the start of the last readout;
  • 18 of 18 fine acquisitions converged: 17 with one correction, accepted on the frame after it; one on the budget's third correction (0.76", relaxed goal 0.8");
  • acamd made the target offset 5.6–9.2 s after it was sent for 15 targets, 12.9–22.8 s for 3; the 30 s bound never fired and no exposure was taken on the offset star;
  • not yet exercised on sky: NEXP>1 stop and abort, the flexure re-apply between sub-exposures, REPEAT, the ring warning; and the slicecamd move wait as revised after review (last commit), which was not part of that test.

Jira: PAL-166, PAL-171, PAL-182, PAL-144, PAL-181, PAL-152, PAL-142.

Replaces #479, #483, #497, #498, #499, #500, #501 and #502. Independent of #485, #493, #494, #495, #496 and #504: each of those applies to main on its own, before or after this one.

🤖 Generated with Claude Code

cfremling and others added 9 commits October 29, 2026 16:22
While ACAM guides, a fine-acquire correction only re-points acamd's
goal; the guide loop applies it at its next executed correction. The
settle was timed from the send, so with short slicecam exposures the
next sample median could be built from pre-move frames and the same
correction applied twice (UT 2026-08-06, BD+28d4211: exposed 2.6 arcsec
off slit).

acamd tracks the request: offsetgoal marks the goal change pending; the
guide loop clears it when it executes a correction computed from the
changed goal (the pending state is captured before the goal is read, so
a correction already in flight can never clear it). The flag is
published; do_fineacquire waits for pending false on a status newer
than its send before settling. A stop interrupts the wait; timeout
FINE_ACQUIRE_MOVE_TIMEOUT (default 60 s, 0 disables) proceeds on the
old timer. Also fixes the offset_acam_goal log line, which printed
degrees labelled arcsec on the guiding path.

Co-Authored-By: Claude Fable 5 <[email protected]>
After a correction has been applied, one frame already inside the goal is
accepted without waiting for the full sample set. A frame can only accept,
never reject: larger readings fall through to the existing median logic
unchanged. SETTLE_FRAMES config default 2 -> 0; the wall-clock settle after
the confirmed goal change covers the move.
A deliberate goal offset (the star-to-target offset after fine
acquisition, put-on-slit, the GUI offset) armed a one-shot allowance
that any sent correction cleared. A guide correction whose blocking
pt_offset was in flight when the offset arrived cleared it as it
completed, and the next frame's correction to the new goal was refused
at the guiding cap, so the exposure was taken on the offset star. The
cap selection also required acquire_mode == TARGET_GUIDE, which is a
race of its own.

Goal changes now carry a generation, and each correction is classified
by the goal it was computed from: guide, fine-acquire, or deliberate
offset. Only a deliberate-offset correction may use the larger
ACQUIRE_TCS_MAX_PUTONSLIT_OFFSET cap; once sent it has made the change
and clears goalshift_pending, unless a newer change arrived meanwhile.
A correction computed from an older goal can neither use nor spend a
newer goal's allowance.

acamd.cfg.in: ACQUIRE_OFFSET_THRESHOLD=1.0, the value the deployed
configuration uses.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
A target that kept landing just outside the goal was corrected again and
again until the sequencer's acquisition timeout. Two slicecamd.cfg keys:

FINE_ACQUIRE_MAX_ADJUSTMENTS=<n> (default 3, 0 = previous behaviour):
at most <n> corrections are commanded in one run, and the <n>-th decides.
It is sent only if the offset it removes is within
FINE_ACQUIRE_RELAXED_FACTOR x goal, and the run converges once that move
has executed and settled; otherwise the run fails at once without
sending it. A correction acamd does not confirm within
FINE_ACQUIRE_MOVE_TIMEOUT also fails a budgeted run. Nothing is sampled
or evaluated once no correction can follow.

FINE_ACQUIRE_RELAXED_FACTOR=<f> (default 2.0, at least 1).

Also: a single frame inside the goal after a correction converges the
run; FINE_ACQUIRE_SETTLE_SEC=1 (was 3.0) now that the settle starts when
the move has executed; every lock writes the same "fine acquisition
converged" line, with how it locked appended.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
?ONTARGET was invisible through tcsd because send_command translates
every non-whitelisted reply as a status code: the flag IS the reply, so
0 collided with TCS_SUCCESS ('success') and 1 fell through to
'tcs_undefined' ERROR. Whitelist it for verbatim passthrough.

sequencerd: config TCS_AUTO_ONTARGET (default no) spawns a watcher for
exactly the lifetime of move_to_target's TCSOP wait. It calls
ontarget() -- the operator's own path -- when, on two consecutive 1 Hz
reads: ?ONTARGET==1 AND motion==tracking AND |telescope-target| <
TCS_AUTO_ONTARGET_SEP arcsec. All three are required: shadow data
(UT 2026-08-25, 64 slews) shows the flag is position-agnostic, the
separation cannot see sub-threshold hops, and motion says nothing
about settle. Operator ack trailed the flag by median +9.7 s / p90
+18 s and never once preceded it. The operator's press is unchanged;
whichever arrives first wins.
When the last sub-exposure of a science target starts reading out, the
target is completed and the next one is picked through the normal path
(get_next, activate, publish, slew, acquire) while camerad reads out.
A readout gate holds every camerad command for the new target (camera
settings, expose) until camerad reports READY. There is no early pick in
single-step or single-target mode, after a cancel, with a thread error
set, or when the current row is a calibration.

Readouts are counted from camerad's PIXELCOUNT messages, so the count
cannot run ahead of the shutter; a PIXELCOUNT from the tail of the
previous readout cannot end the next target's EXPOSE wait. camera_set
runs in parallel with the slew and acquisition behind the readout gate
and is joined just before the exposure; a target is never exposed unless
its own camera settings were applied.

The flexure compensator is set just before each exposure, re-applied
during the readout between sub-exposures, and before a REPEAT exposes.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
… commanded angle

TCS_AUTO_ONTARGET defaults to yes. The automatic ontarget now also
requires |TCS cass angle - RINGGO angle| < TCS_AUTO_ONTARGET_CASS
(0.05 deg) on the same two consecutive reads: ACAM's goal is computed
from the cass angle at the acquire, about 8.5" per degree of ring still
to turn. When everything but the ring has held for 20 s, one warning
tells the operator to press ontarget once the ring is in place.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
stop: the sequencer accepts it for the whole series of the current
target (the EXPOSE wait covers only the first sub-exposure; camerad not
READY covers the rest) and no longer clears a cancel. camerad ends the
series after the frame in progress, cut short if it is integrating with
5 s or more left, instead of starting the next sub-exposure. abort no
longer starts the next sub-exposure either. A cancelled exposure's
remaining PIXELCOUNT messages cannot set the READOUT wait later.

camerad waits for the last frame's FITS write before acting on the next
command after READY.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
A target offset sent while ACAM guides only changes acamd's goal; the
guide loop's next correction makes the move. The sequencer exposed a
fixed OFFSET_SETTLE_SEC (3 s) after sending it, which could be before
the telescope had moved. It now waits until acamd's status shows the
goal change raised after the offset was sent and cleared again
(GOALSHIFT_PENDING), bounded at 30 s, after which it exposes anyway
with a warning; a cancel ends the wait. OFFSET_SETTLE_SEC is now an
extra wait after the move and defaults to 0.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-29T14:28:23.737626Z 8c25120 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8c251201c2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread slicecamd/slicecam_interface.cpp Outdated
Comment on lines +655 to +660
if ( this->last_acam_pubtime.load(std::memory_order_acquire) <= send_time ) return false;
if ( this->acam_goalshift_pending.load(std::memory_order_acquire) ) {
seen_pending = true;
return false;
}
return seen_pending;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Record goal-shift transitions in the telemetry handler

If acamd's pending=true and subsequent pending=false reports are both processed before this waiter evaluates its predicate, seen_pending remains false even though the correction completed successfully. This can happen before the synchronous offset_acam_goal() call returns or while the waiting thread is not scheduled: handletopic_acamd() retains only the latest pending value, and condition-variable notifications do not queue transitions. The acquisition then waits the full move timeout and, with the configured correction budget, incorrectly reports failure. Record the raised and cleared publish times in the telemetry handler, as the sequencer's new offset wait does, so completion does not depend on observing the intermediate state in this thread.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in e3414e7: the acamd telemetry handler records the publish times of the latest status with GOALSHIFT_PENDING set and of the latest with it clear, and the wait completes once a set status published after the send is followed by a clear one, the same rule as the sequencer's wait for the target offset. It no longer depends on observing the intermediate state.

The wait for acamd to apply a fine-acquire correction completed only if it
saw GOALSHIFT_PENDING set and then clear. Condition-variable notifications
do not queue, so if both statuses were handled before the waiting thread
re-checked, it waited out FINE_ACQUIRE_MOVE_TIMEOUT and, under a correction
budget, failed the run. The acamd telemetry handler now records the publish
times of the latest set and clear statuses, and the wait completes once a
set status published after the send is followed by a clear one, as the
sequencer's wait for the target offset does.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants