Skip to content

Fix 25 problems from the log (http, runner, db) - #219

Merged
danielehrhardt merged 2 commits into
mainfrom
godmode/63-fix-25-problems-from-the-log-http-runner
Oct 21, 2026
Merged

danielehrhardt merged 2 commits into
mainfrom
godmode/63-fix-25-problems-from-the-log-http-runner

Conversation

@danielehrhardt

Copy link
Copy Markdown
Contributor

Done. Two commits on the branch, the branch is up to date with origin/main, the typecheck passes and the related core tests pass.

What changed

1. One notice per sign-out outage (services/claudeLogin.ts, services/runNotices.ts)

  • Main already sends one "Claude is signed out" notice per outage and keeps that state in the meta table (commit f44dcdd). The build that logged the outage, 0.2.20, didn't have it yet, which is why nobody was told.
  • The notice is now level error, the same path notify_user uses (inbox and desktop). It says scheduled automations are failing and how to sign in again: run claude in a terminal and type /login, or use one click in Godmode.
  • On a runner, the title names the computer ("Claude is signed out on …"). The runner's notice reaches the desktop once, and runners don't run the notices or the scheduler, so nothing is sent twice.
  • An automation that fails over the sign-in no longer sends its own "“X” failed" notice; the sign-out notice covers all of them. There is one exception: if no sign-out notice has gone out since anything last worked, the failure notice still goes out after 10 s.
  • "Notify a failure once until it works again" now ignores runs that failed over the sign-in.

2. One more run after the outage (new automations/signInRetry.ts, started next to the scheduler)

  • The outage ends with the first run Claude answers on that computer. This one and each runner are tracked separately, and runs on Z.ai don't count.
  • At that point, every scheduled automation whose latest run failed over the sign-in in the last 24 h runs once more, through the normal scheduler path. It is skipped when:
    • its next slot is less than 2 h away,
    • it or its agent is turned off,
    • it is already running,
    • or the failed run was itself such a retry, so it never loops.
  • Which failed runs were already handled is kept in the meta table, and retries are also marked on their automation event. A restart neither repeats nor forgets anything.
  • Failures from before this version are recorded as handled on the first start, so today's failed runs won't suddenly run hours after the update.
  • One log line (scope sign-in-retry) gives how many automations ran again and which were skipped, with the reason.

Tests: new test/sign-in-retry.test.ts with the fake Claude in AUTH_EXPIRED mode. It checks:

  • several failures give one error notice and no notices per automation;
  • the first answered run starts the yearly automation once;
  • the half-hourly one (next slot under 2 h) and the turned-off one are skipped;
  • the retry failing again is a second outage with its own notice;
  • the next answered run doesn't start it a third time;
  • a sign-in-style failure with no sign-out notice is still reported.

A review pass found no blocking defects; I fixed its three smaller findings (the second commit).

Log lines that are not code bugs (nothing changed)

  • Why Claude was signed out: the refresh token was rejected after the Mac woke up. Claude Code owns that token in the Keychain, so Godmode can't fix it.
  • Network after sleep: the ENOTFOUND failures, the GitHub "can't reach" and "timed out" warnings for sources and tasks, the browser "CDP connection closed" lines, and the 403 "Request not allowed" refusals. Resume and the 403 back-off already handle the retries.
  • Runner asleep: "could not copy the setup", "couldn't merge memory" and "couldn't copy browser sessions".
  • Watchdog stops: the "before its first step" burst was on 2026-10-17, before the build where the watchdog counts awake time only. The 60-minute stop was a background task that waited forever, which is a correct stop.
  • ElevenLabs audio timeout (2026-10-19): it happened during a network outage, and builds since 0.2.20 log it at info.
  • Slow queries and blocked event loop: none since 0.2.20 was installed (2026-10-20 10:09Z).

Next for Daniel: nothing. Supporting a long-lived claude setup-token login in Godmode would prevent this kind of outage, but that is a separate feature.


Task #63 · done by Self-Healer with Godmode Bot

… run once more after it

When Claude's sign-in ran out, every scheduled automation failed with its
own notice and nothing ran again once it worked.

- The sign-in notice is an error now, names the runner it comes from and
  says how to sign in again (claude, then /login, or one click in Godmode).
  Automation failures over the sign-in send no notice of their own.
- The first run Claude answers on that computer ends the outage: every
  scheduled automation whose latest run failed over the sign-in runs once
  more, unless its next slot is less than two hours away, it is turned off,
  it runs already, or that run was such a second try. Kept in the meta
  table, so a restart neither repeats nor forgets it.
…e, mark retries on their event

- A failure that reads like a sign-in problem but sent no sign-in notice
  (Claude Code didn't say so itself) is still told after a short wait.
- "Once until it works again" skips runs that failed over the sign-in.
- The one more run is recognised by its automation event too, so a
  computer that stood in or a full list doesn't start it a third time.
@danielehrhardt
danielehrhardt merged commit a8caa1e into main Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant