Skip to content

Fix 25 problems from the log (http, runner, db) - #220

Merged
danielehrhardt merged 3 commits into
mainfrom
godmode/64-fix-25-problems-from-the-log-http-runner
Oct 21, 2026
Merged

danielehrhardt merged 3 commits into
mainfrom
godmode/64-fix-25-problems-from-the-log-http-runner

Conversation

@danielehrhardt

Copy link
Copy Markdown
Contributor

Scheduled automations now catch up slots missed while Godmode was off or the Mac slept

The main item is fixed. When Godmode starts again, or the Mac wakes, an automation whose slot fell in the gap runs once. Before, it never ran and nobody was told, which is what happened to the two prospect-reply automations on 2026-10-20 (Godmode down 12:38–14:06Z). The typecheck passes (pnpm -r typecheck), and the test files I touched and the related ones pass, including the device-groups end-to-end tests.

Root cause (packages/core/src/scheduler/scheduler.ts)

  • On start, the scheduler only set up its jobs. The scheduling library never fires a slot that passed before its job was created.
  • For automations with a random start window, a slot whose window was still open but whose random start time had already passed was skipped until the next day. That's the 14:00Z slot with its 45-minute window, seen at 14:06Z.

How it works now

  • Heartbeat: the scheduler writes a "last alive" time every 60 s (scheduler.alive_at). Only slots after it count as missed, so slots skipped on purpose while it was running (previous run still going, budget used up) stay skipped.
  • Which slot runs: only the latest one. It runs if all of these hold:
    • it is at most 12 h old;
    • no run was started for it, here or on another computer;
    • the automation existed before the slot;
    • the next slot is more than 2 h away.
  • Start windows: if the random start time has already passed, it runs now. If it is still ahead, the normal timer runs it as planned.
  • Logging: each catch-up logs routine <id> caught up its missed <slot> slot (start|wake): run <runId>. Skipped ones log the reason (older than 12 h, or next slot within 2 h). The automation's history says "Caught up: Godmode wasn't running at the scheduled time" (or "...the computer was asleep...").
  • Wake: the Mac's wake is detected by a jump in the clock. The library already fires one late tick after a wake, so the wake catch-up waits 10 s and a per-slot guard skips a second run.
  • Device groups:
    • The catch-up first waits until the group has synced.
    • A computer standing in for its primary leaves catch-up to the existing stand-in path (runMissedSince), so it doesn't wrongly re-run slots the primary skipped on purpose.
    • The stand-in path now also runs window slots whose start time passed.
  • Never twice:
    • A catch-up stops starting runs once the scheduler stops or restarts.
    • A catch-up that fails to start is reported like a failed normal start.
    • The heartbeat is kept out of backups, so restoring an old backup can't trigger a wave of re-runs.
    • Fix what the review of spend and budgets found #63's sign-in retry (merged to main meanwhile) goes through the same start path, so the two can't both re-run the same slot. I resolved the merge conflict with it.

Tests (packages/core/test/scheduler.test.ts, fixed clock):

  • a slot inside the downtime gives exactly one run;
  • a second catch-up, on start or wake, gives none;
  • next slot within 2 h gives none, and a slot older than 12 h gives none;
  • a slot the running scheduler already saw gives none;
  • an automation created after its slot gives none, and one that already ran gives none;
  • a window whose start time passed runs once now; one whose start time is still ahead is left to its timer;
  • a computer standing in for its primary does nothing;
  • with no heartbeat yet, nothing is caught up.

There is no test yet that drives a real late tick through the per-slot guard; it is checked only through repeated catch-up calls.

First run after the update: the very first start on this build has no heartbeat yet, so it catches nothing up. It works from the second start on.

The other log lines need no code change:

Nothing for Daniel to do; it reaches the desktop app the next time he updates Godmode, since shipping is paused.


Task #64 · done by Self-Healer with Godmode Bot

A heartbeat (meta scheduler.alive_at) marks when the scheduler last ran. On
start and after a wake, each schedule automation whose latest slot (or drawn
start in a random window) fell after it runs once now, unless it ran already,
is older than 12 h or its next slot comes within 2 h. A late tick for a slot
that was caught up is skipped. The stand-in catch-up now also runs window
slots whose drawn start passed.
…wake, stop with the scheduler

A secondary standing in has no heartbeat of the primary's schedule, so its
start catch-up is left to runMissedSince. The wake catch-up waits until
croner's late ticks and the device group's wake check came first; a catch-up
stops starting runs once the scheduler stops or restarts. A failed catch-up
is told like a failed tick. The heartbeat stays out of backups.
…blems-from-the-log-http-runner

# Conflicts:
#	packages/core/src/scheduler/scheduler.ts
@danielehrhardt
danielehrhardt merged commit fd058c0 into main Oct 21, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant