Isolate unreadable tick history and keep dispatch faults recoverable

docs/agents.md

@@ -77,13 +77,7 @@ python manage.py ox_health
 OK: backlog=0 oldest_age=none last_claim_age=none
 ```
 
-`manage.py check` reports schedule and backend policy errors through
-`django_ox.E002` to `E005`, `E010` and `E011`, and warnings through
-`django_ox.W003` and `django_ox.W004`, before deployment.
-Database acceptance of schedule arguments is determined at dispatch;
-alert on `schedule_dispatch_error` and `schedule_dispatch_failed` at runtime.
-Review `django_ox.W003` and `django_ox.W004` warnings too; `W004` asks for
-a non-bool integer from 1 to 32767 while retaining compatible legacy values.
+`manage.py check` reports schedule and backend policy errors through `django_ox.E002` to `E005`, `E010` and `E011`, and warnings through `django_ox.W003` and `django_ox.W004`, before deployment. Database acceptance of schedule arguments is determined at dispatch. Alert on `schedule_dispatch_error`, `schedule_dispatch_failed` and `schedule_tick_unreadable`. For stored schedules, also alert on `schedule_row_skipped`, `schedule_source_unavailable` and `schedule_boundary_heal_failed`. Review `django_ox.W003` and `django_ox.W004` warnings too; `W004` asks for a non-bool integer from 1 to 32767 while retaining compatible legacy values.
 
 Invalid per-task fields raise `InvalidTask` (`InvalidTaskError` on the
 backport) when the task is built, normally at import, rather than appearing

docs/production.md

@@ -369,14 +369,9 @@ schedule fires its first due tick. Use a long-running worker for more
 frequent schedule checks, but missed ticks are still coalesced; this does not
 guarantee that every tick runs. See [Missed ticks](recurring-tasks.md#missed-ticks).
 
-An unreachable server, a lost or changed database session, or a database
-error escaping a shared dispatch read does not count as an empty batch pass.
-Failed polling passes and abandoned dispatch passes are retried, as they are
-for a long-running worker. A batch keeps retrying until a dispatch pass
-completes and the normal queue-drain and idle conditions hold. A failed
-shared read, failed rollback or unusable session abandons the dispatch pass,
-logs `schedule_dispatch_failed` at WARNING and keeps the batch open until a
-pass completes.
+Failed polling and abandoned dispatch passes do not count as empty batch passes. An unreachable server, lost or changed session, failed shared read, failed rollback or unusable connection causes retries. Other unhandled dispatch exceptions now do too: unlike 1.8.0, they log `schedule_dispatch_failed` with `category=unexpected_exception` instead of ending the worker with exit 1.
+
+The batch keeps claiming queued work and retries dispatch once per schedule interval. Draining the queue does not end it while dispatch remains owed. Normal completion requires a completed dispatch pass and the usual drain and idle conditions. This includes a custom schedule source whose store fails after startup. Direct calls to `dispatch_schedules()` still raise. There is no failure threshold or new exit code; stopping the batch can return 0 despite continuing dispatch failures.
 
 A failure inside one schedule's transaction is reported as
 `schedule_dispatch_error` at ERROR if rollback succeeds and the same
@@ -390,13 +385,7 @@ each pass is abandoned at that schedule, preventing later schedules from
 being reached and holding the batch open. `schedule_dispatch_failed` does
 not identify the schedule being processed.
 
-Alert on `schedule_dispatch_error` and `schedule_dispatch_failed`, plus
-`schedule_row_skipped` and `schedule_source_unavailable` for stored schedules,
-regardless of the batch's exit. Each `--batch` invocation starts a new worker,
-so a schedule that keeps failing logs its first-failure traceback on every run
-and no `schedule_dispatch_recovered` event carries across runs; alert on the
-presence of `schedule_dispatch_error` in each run. Give the job runner a
-timeout: it is what bounds a run against a failing database.
+Alert on `schedule_dispatch_error`, `schedule_dispatch_failed` and `schedule_tick_unreadable`. For stored schedules, also alert on `schedule_row_skipped`, `schedule_source_unavailable` and `schedule_boundary_heal_failed`. Alert regardless of batch exit. Each invocation starts a new worker: continuing schedule failures get a first-failure traceback each run, and `schedule_dispatch_recovered` does not carry across runs. Alert on `schedule_dispatch_error` in each run. Give the job runner its own timeout for database, schedule-source and package faults. Clean termination does not prove successful dispatch.
 
 Both flags run a single process. `--processes` above 1 is rejected, because
 the supervisor restarts a worker that exits on its own; for more throughput in

docs/recurring-tasks.md

@@ -301,11 +301,7 @@ across backends:
 python manage.py check
 ```
 
-Database acceptance is determined at dispatch after validation. Alert on
-`schedule_dispatch_error` for schedule-scoped failures and
-`schedule_dispatch_failed` for abandoned passes. For stored schedules,
-also alert on `schedule_row_skipped` and `schedule_source_unavailable`.
-Use these events for schedule health monitoring.
+Database acceptance is determined at dispatch after validation. Alert on `schedule_dispatch_error`, `schedule_dispatch_failed` and `schedule_tick_unreadable`. For stored schedules, also alert on `schedule_row_skipped`, `schedule_source_unavailable` and `schedule_boundary_heal_failed`. These report schedule-scoped failures, abandoned passes, unusable history, skipped rows, source failures and blocked boundary healing.
 
 To see what has actually dispatched, read the tick log:
 

docs/stored-schedules.md

@@ -691,19 +691,8 @@ Worth knowing before you turn it on:
   such a write tells a worker to look. Set
   `OPTIONS["SCHEDULE_RECONCILE_INTERVAL"]` if you want that sooner; it is one
   indexed read of a small table.
-- **One extra query per pass.** Workers read a single row to learn whether
-  anything changed. They re-read the schedules when it did and at each
-  reconcile interval. If the change marker cannot be read, full reads still
-  run at each reconcile interval.
-- **Unreadable or invalid rows are skipped.** A row is left out if a value
-  cannot be converted, its start or end cannot be compared with the worker's
-  clock, or it cannot be built. The other schedules and the queue keep
-  running. The skip is logged as `schedule_row_skipped` with `schedule_pk`,
-  `schedule` and `reason`. If the name cannot be read, `schedule` may be
-  `None`. A conversion failure's reason names the column. Each row is logged
-  once, then at most once a minute per worker while it stays skipped. A skip
-  during the locked dispatch read includes a traceback only on its first
-  line.
+- **Marker reads and reconciles.** Workers read one change-marker row per pass. They re-read schedules when it changes and at `SCHEDULE_RECONCILE_INTERVAL`, normally 60 seconds. If the marker cannot be read, full reads still run at that interval. Stored schedule reads use more CPU than 1.8.0, and full reconciles now use batched queries. Reconciles do not run every tick. Latest tick decoding costs CPU each pass. Dispatch statement counts are unchanged, both with and without a due tick.
+- **Stored values django-ox cannot use.** Workers skip unreadable or invalid rows, including incomparable boundaries and a maximum `boundary_generation`; other schedules and the queue continue. `schedule_row_skipped` carries `schedule_pk`, `schedule`, `fields` and `reason`. An unreadable name may be `None`. Stored-value reasons name the field, quote its value and explain the problem. Reporting is once, then at most once a minute per row per worker. Locked dispatch skips include a traceback only on the first line.
 - **Skipped rows need repair.** Skipping does not repair invalid values. Find
   the row by `schedule_pk` and correct it in the admin change form or with
   `update_schedule`. Use SQL if the value is one the ORM cannot read. To pause
@@ -718,23 +707,16 @@ Worth knowing before you turn it on:
   and the same connection remains usable, later schedules are still attempted.
   The failed schedule is retried on subsequent passes under the usual
   due-tick and deadline rules; only its reporting is rate-limited.
-- **Database read failures are not bad rows.** A database error during a full
-  schedule read or a shared dispatch read, such as a lost connection, missing
-  table or lock error, is not attributed to an individual row. It stops the
-  pass rather than skipping a row.
-- **An abandoned pass stops further traversal.** A `django.db.DatabaseError`
-  escaping a shared dispatch read, failed rollback or unusable connection is
-  reported as `schedule_dispatch_failed`. The stored source's marker read and
-  boundary heal retain their own events. An unreadable change-marker value
-  reports `schedule_source_unavailable` once, then at most once a minute while
-  it remains unreadable, and the worker reads the schedules in full at each
-  reconcile interval during that time. A database error on the marker read
-  reports the same event on every dispatch pass, and the worker keeps using
-  the last schedules it read. A failed boundary heal reports
-  `schedule_boundary_heal_failed`. Dispatch is retried on a later pass.
-  Ticks already committed are not undone. Alert on `schedule_row_skipped`,
-  `schedule_dispatch_error`, `schedule_dispatch_failed` and
-  `schedule_source_unavailable`.
+- **Database read failures are not bad rows.** Database errors during full schedule or shared dispatch reads, including connection, missing-table and lock errors, stop the pass. They are not attributed to a row. Server-side errors propagate unchanged. In autocommit, recognized value-local failures may be isolated by diagnostic re-reads while the connection remains usable.
+
+  Inside a transaction, diagnostic re-reads are restricted: PostgreSQL requires a usable transaction; SQLite permits only its recognized UTF-8 decoding refusal; MySQL permits none. Unwind through the transaction or savepoint boundary before further queries where required. Reads create no savepoints. `update_schedule` and `delete_schedule` have their own atomic blocks; source reads and `ox_prune` do not. Unlike 1.8.0, a source read inside your transaction can leave a server error for you to roll back.
+- **An abandoned pass stops further traversal.** An escaping `django.db.Error`, other unhandled exception, failed rollback or unusable connection reports `schedule_dispatch_failed`. Dispatch retries on a later pass; committed ticks remain committed. Marker reads and boundary heals retain their own events.
+
+  An unreadable marker reports `schedule_source_unavailable` once, then at most once a minute; full reads continue at each reconcile interval. A database error on the marker read reports every pass while workers use the last schedules read. Failed boundary healing reports `schedule_boundary_heal_failed`, first with a traceback, then counted summaries at most once a minute per row per worker. Healing retries each pass; the affected schedule cannot dispatch meanwhile.
+
+  An unreadable newest tick within the pass's bound skips only its schedule with `schedule_tick_unreadable`. A settings schedule can also be blocked by its unreadable earliest tick. History is not treated as absent. The pass completes, so this skip does not hold a batch open. Follow the recovery procedure under [Configuration](configuration.md#ox_prune).
+
+  Alert on `schedule_dispatch_error`, `schedule_dispatch_failed` and `schedule_tick_unreadable`. For stored schedules, also alert on `schedule_row_skipped`, `schedule_source_unavailable` and `schedule_boundary_heal_failed`.
 - **`manage.py check` cannot see rows.** Checks run before `migrate`, so a bad
   schedule in the database is a log line, not a start-up error.
   Settings-declared schedules still fail fast for errors their checks can
@@ -761,21 +743,20 @@ failures. The full set, with every field, is on the
 | --- | --- |
 | `schedule_dispatched` | A tick enqueued its task. |
 | `schedule_tick_dropped` | A tick was past its starting deadline. Carries `late_seconds`. |
-| `schedule_row_skipped` | A row could not be built. A skip during a periodic full read carries `schedule`, `schedule_pk` and `reason`. A skip during the locked dispatch read carries `schedule_pk` and a traceback. |
+| `schedule_row_skipped` | A row holds values django-ox cannot use, has incomparable boundaries, or cannot be built. Full and locked reads carry `schedule`, `schedule_pk`, `fields` and `reason`. Reporting is rate-limited; locked reads include a traceback only on the first line. |
 | `schedule_dispatch_error` | A schedule-scoped failure, database or not. Its tick and task rolled back, and the pass continues. The schedule is retried under the usual due-tick and deadline rules. Reporting is rate-limited. |
-| `schedule_dispatch_failed` | A dispatch pass was abandoned. It is retried on a later pass. Reporting is rate-limited. |
+| `schedule_dispatch_failed` | A pass was abandoned and will be retried. `category` distinguishes `database` from `unexpected_exception`. Reporting is rate-limited. |
 | `schedule_dispatch_recovered` | A schedule that had failed on this worker committed a tick again. Carries `failures`. |
+| `schedule_dispatch_resumed` | A pass completed after abandoned passes. Carries their total in `failures`. |
+| `schedule_tick_unreadable` | A schedule was skipped because required tick history could not be read. History was not treated as absent; other schedules continue. |
+| `schedule_tick_readable` | This worker read the history again after skipped passes. Carries their total in `failures`. |
 
 Pausing a stored schedule preserves its failure state on each worker; deleting
 the row drops it without an event. A paused, fixed and resumed row reports
 `schedule_dispatch_recovered` when it next commits a tick on a worker that
 retained its failure state.
 
-Alert on `schedule_row_skipped`, `schedule_dispatch_error`,
-`schedule_dispatch_failed` and `schedule_source_unavailable`, regardless of a
-batch's exit code. `schedule_row_skipped` usually means a task key was removed
-from the code while a row still names it. A row can pass validation and still
-fail at dispatch, so that event alone is not enough.
+Alert on `schedule_dispatch_error`, `schedule_dispatch_failed` and `schedule_tick_unreadable`. For stored schedules, also alert on `schedule_row_skipped`, `schedule_source_unavailable` and `schedule_boundary_heal_failed`. Alert regardless of a batch's exit code. For `schedule_row_skipped`, inspect `fields` and `reason`: an unregistered task key is one possible cause. A row can pass validation and still fail at dispatch, so skip alerts alone are insufficient.
 
 Read `failures` and `suppressed` on dispatch failure events rather than
 counting log lines. `ox_health` has no schedule check, and a rolled-back