From c330c8091d42f40d6869c851d2bb1a5a3f5f9c3f Mon Sep 17 00:00:00 2001 From: Richard McDaniel <1130888+rmcdaniel@users.noreply.github.com> Date: Thu, 19 Nov 2026 05:48:29 +0000 Subject: [PATCH] Prepare native Server 3.14.0 Co-Authored-By: Claude Opus 5.5 --- docker-compose.published.yml | 2 +- docs/releases/3.14.0.md | 251 +++++++++++++++++++++++++++ docs/rust-server-upgrade-takeover.md | 2 +- docs/server-reference.md | 2 +- 4 files changed, 254 insertions(+), 3 deletions(-) create mode 100644 docs/releases/3.14.0.md diff --git a/docker-compose.published.yml b/docker-compose.published.yml index b4778ccaa..dfed35932 100644 --- a/docker-compose.published.yml +++ b/docker-compose.published.yml @@ -16,7 +16,7 @@ name: durable-workflow-server # release whose default names another image). Set DW_SERVER_IMAGE to run # any other native image, by tag or digest; pin a digest for production # change control. -x-server-image: &server-image ${DW_SERVER_IMAGE:-durableworkflow/server:3.13.0} +x-server-image: &server-image ${DW_SERVER_IMAGE:-durableworkflow/server:3.14.0} services: server: diff --git a/docs/releases/3.14.0.md b/docs/releases/3.14.0.md new file mode 100644 index 000000000..9aaaa94e0 --- /dev/null +++ b/docs/releases/3.14.0.md @@ -0,0 +1,251 @@ +# Durable Workflow Server 3.14.0 (native) + +Server 3.14.0 hands a run's open children to its continued run at +continue-as-new, answers queries on closed runs, guards worker build and task +queue retirement, exposes queue backlog to autoscalers at `GET /metrics`, gives +standalone activity starts a duplicate policy, keeps MariaDB completions within +the default packet, admits leased completion uploads while storage drains, and +honors the native long-poll caps. Its immutable images are +`durableworkflow/server:3.14.0` and `ghcr.io/durable-workflow/server:3.14.0`; +the published Compose default selects 3.14.0. Rolling aliases move separately +after published-image qualification. + +**Upgrade:** this release requires an explicit additive `schema-upgrade` from +3.13.x on every backend; see [Upgrade and rollback](#upgrade-and-rollback). +After the schema step, rolling back to 3.13.x requires restoring the +pre-upgrade backup: a 3.13.x binary refuses the upgraded database with +`native_schema_newer_than_binary`. Worker protocol and control-plane versions +are unchanged. + +## Behavior changes to review + +- A standalone activity start for an id that is still running now returns a + typed `409 activity_already_started` instead of `422 validation_failed`. + Clients that matched the 422 must match the 409, or send + `duplicate_policy: "use-existing"` (see + [Standalone activity duplicate policy](#standalone-activity-duplicate-policy)). +- Queries on failed, cancelled and terminated runs are answered instead of + refused with `409 run_not_active`. A run whose history was pruned or is + missing now gets `409 query_history_unavailable` instead of + `503 storage_unavailable`. +- A workflow-task completion receipt stores a digest of the request, not the + request. Do not run 3.13.x and 3.14.0 nodes against one database: a 3.13.x + node would answer an identical retry of a completion a 3.14.0 node recorded + with `409 conflicting_completion`. The schema step already rules out a mixed + fleet, because 3.13.x refuses the upgraded schema. + +## Open children at continue-as-new + +When a run continues as new while children it started are still open, the +continued run now takes over their supervision in the continue-as-new +transaction, as Core does since Workflow 2.26.0 (workflow#868, workflow#1107). +The child link, child call, child identity, deadlines and `parent_close_policy` +stay as recorded, and the child's `WorkflowStarted` still names the run that +started it. Each continue-as-new passes on every child the run still +supervises, including children an earlier run of the chain handed to it. + +When the chain's final run completes, fails, times out, is cancelled or is +terminated, it applies each handed-over child's `request_cancel`, `terminate` +or `request_cancellation` policy to the child's current run and records +`ParentClosePolicyApplied` or `ParentClosePolicyFailed` in its own history. +`abandon` children are handed over and left alone. A handed-over child that +fails while a run of the chain still supervises it is retried under the +starting run's recorded retry policy; once the chain has closed, the failure is +final. The continued run gets no handle on the child or its result, and a +closed predecessor gets no history or task when a handed-over child closes. + +Supervision is kept in the new native table `dw_child_supervisions`, which the +schema step creates empty. A chain continued before the upgrade has no +supervision rows, so its open children are left running and a later failure is +final, as before. No event is recorded at the hand-over, so existing histories +replay unchanged. Published PHP 2.x applies no policy to a child whose parent +continued as new (native divergence `continue-as-new-child-supervision`) +(#1184, #1246). + +## Queries on closed runs + +A retained run of any status, including failed, cancelled and terminated, is +queried read-only. The worker receives that run's id, status and full history up +to its terminal event; nothing is written to the run, its history or its tasks. +Describe's `actions.can_query` follows the same rule. A run whose history was +pruned by retention, or whose `WorkflowStarted` is missing, is refused with +`409 query_history_unavailable` (with `history_state`, `run_status` and +`is_terminal`) and no query task is admitted. The query contract lists the new +reason after `run_not_active`, which stays listed for older servers. +[Workflow queries](../server-reference.md#workflow-queries) now lists every query failure and +timeout reason. Published PHP keeps `run_not_active` (native divergence +`closed-run-queries`) (#587, #1240). + +## Guarded worker build and task queue retirement + +`GET /api/deployments/{namespace}/{queue}@{build}/retirement` and +`GET /api/task-queues/{queue}/retirement` return one read-only inventory of what +still depends on a build or queue: open, waiting, continued and child runs; +ready, leased and delayed tasks; open activities, update validations and +schedules; and open runs with no recorded queue. Each blocker has an exact +count, up to 20 samples and a recovery hint. + +`POST /api/deployments/{name}/retire` and `POST /api/task-queues/{queue}/retire` +repeat that inventory inside native's serialized transition and write only +when it is complete and empty, optionally bound to +`expected_inventory_revision`. Otherwise they return `409` with the inventory and +change nothing; there is no force path. A retired build is not selectable for +new starts and its workers stay draining. Discover the feature through +`capabilities.worker_build_retirement`. See +[worker build retirement](../worker-build-retirement.md) (#519, #1241). + +## Queue backlog for autoscalers + +`GET /metrics` serves Prometheus text format 0.0.4 for one namespace: ready, +due, delayed, leased and dispatchable tasks per queue, kind and build id; +active pollers and worker slots; admission state; activity retry pressure; +schedules; and run and activity gauges. Backlog is read from task rows, so due +work is visible while no worker runs. Series are bounded at 100 queues and 10 +build ids per queue and kind, with truncation disclosed. A failed read answers +`503` with no samples. + +The route accepts a new scrape-only `metrics` role, granted by +`DW_METRICS_TOKEN` (which must differ from every other token) or by a runtime +credential with `"roles":["metrics"]`; admin tokens can also scrape. Discover it +through `capabilities.prometheus_text_metrics`. KEDA, ServiceMonitor and scrape +examples are in `k8s/helm/examples/autoscaling/`. The packaged chart has no +first-class metrics values yet; pass the token through `auth.extra` or an +existing Secret (#534, #1242). + +## Standalone activity duplicate policy + +`POST /api/activities` accepts `duplicate_policy`, `fail` (default) or +`use-existing`, mirroring workflow start: + +| Activity id | `fail` | `use-existing` | +| --- | --- | --- | +| Still running | `409 activity_already_started`, `outcome: rejected_duplicate` | `200` with the running activity, `outcome: returned_existing_active` | +| Closed or new | `201`, `outcome: started_new` | `201`, `outcome: started_new` | + +**Behavior change:** before 3.14.0 a running id was refused with +`422 validation_failed` on `activity_id`. Neither duplicate answer writes +anything, and both are answered before the queue drain check. `use-existing` +does not compare input or type. An invalid policy gets `422`. The Server +reference now documents [standalone activities](../server-reference.md#standalone-activities) (#590, #1243). + +## MariaDB completions within the default packet + +A workflow-task completion receipt now stores a SHA-256 digest of the canonical +request and the created task ids instead of the whole request, so completions +up to the advertised 32 MiB `limits.max_workflow_task_completion_bytes` record +on MariaDB's default 16 MiB `max_allowed_packet`. The 3.13.0 advice to raise the +packet to 64 MiB is no longer needed for this release. An identical retry still +replays `already_completed`, any other completion is still refused, and +receipts written by earlier releases still match. On MySQL and MariaDB, startup +logs `database_packet_limit`, or `database_packet_limit_below_payload_headroom` +when the packet is under four times `DW_MAX_PAYLOAD_BYTES`. A completion the +database still refuses with a packet error is answered +`413 payload_too_large`, `limit_kind: database_max_allowed_packet`, and nothing +is recorded. + +**Mixed versions:** a 3.13.x node reads only full-request receipts, so it would +answer an identical retry of a completion recorded by a 3.14.0 node with +`409 conflicting_completion`. Because 3.14.0 requires the schema step and 3.13.x +refuses the upgraded schema, the two cannot serve one database; upgrade every +node together as described below (#1234, #1244). + +## Completion uploads while storage drains + +While storage admission is `draining`, a worker's activity-result upload +carrying `X-Durable-Workflow-Payload-Completion` is admitted for its current +activity lease within the per-lease allowance (128 slots, +`DW_EXTERNAL_PAYLOAD_COMPLETION_MAX_BYTES`). Uploads without the header and +workflow, query and activity-failure contexts are still refused. An exhausted +allowance while draining answers the storage refusal (`503 storage_pressure`, +`Retry-After: 5`) and writes nothing. Admission is rechecked after the body is +read and before each write (#951, #1245). + +## Native long-poll capacity + +Native now honors `DW_WORKER_LONG_POLL_MAX_CONCURRENT` (held workflow and +activity waits), `DW_QUERY_TASK_POLL_MAX_CONCURRENT` (held query-task and +update-validation waits) and `DW_WORKER_LONG_POLL_RESERVED_HTTP_WORKERS`, which +native applies as database connections reserved for health, readiness and admin +requests. A poll probes for work before it takes a slot; when no slot is free it +gets `429 long_poll_capacity_exhausted` with `Retry-After: 1`. Each cap is per +process. When none is set, behavior is unchanged. The per-namespace variants +remain ignored (#982, #1236). + +## Opt-in timeout cleanup (not yet advertised) + +Native implements deadline-triggered timeout cleanup (`timeout_cleanup_seconds` +on workflow and child starts), following embedded Workflow 2.29.0. Admission +requires a live worker on the queue that advertises the `timeout_cleanup` +capability; otherwise the start is refused with +`422 timeout_cleanup_not_supported`, never downgraded to a hard timeout. +`capabilities.timeout_cleanup` stays unadvertised until installed cross-SDK +acceptance passes. Starts without the option are unchanged and write no new key +or event (#1201, #1235). + +## Upgrade and rollback + +This release requires an explicit additive schema upgrade from 3.13.x on every +backend: + +| Backend | Schema step | +| --- | --- | +| SQLite | 15 → 16 | +| PostgreSQL | 16 → 17 | +| MariaDB | 16 → 17 | +| MySQL | 17 → 18 | + +Each step creates the empty `dw_child_supervisions` table and its indexes. It +rewrites no existing rows. Stop every Server replica, keep the database +running, and back up the database and every external-payload root. Select the +new immutable image and run that image's `schema-upgrade` once before +restarting every replica on the same version. Startup does not upgrade an older +schema automatically and refuses to serve it (`native_schema_upgrade_required`). +MySQL and MariaDB resume an interrupted upgrade when `schema-upgrade` is run +again; SQLite and PostgreSQL apply their step transactionally. From 3.11.x or +earlier, the same command also applies the intervening steps described in the +[3.12.0 upgrade notes](3.12.0.md#upgrade-and-rollback). + +For a single-node Compose deployment, stop Server while leaving MySQL running, +back up MySQL and `server_storage`, set `DW_SERVER_IMAGE` to the 3.14.0 image, +then pull and upgrade once: + +```bash +docker compose --env-file durable-workflow.prod.env \ + -f docker-compose.published.yml stop server +docker compose --env-file durable-workflow.prod.env \ + -f docker-compose.published.yml pull server +docker compose --env-file durable-workflow.prod.env \ + -f docker-compose.published.yml run --rm --no-deps server schema-upgrade +docker compose --env-file durable-workflow.prod.env \ + -f docker-compose.published.yml up -d --wait +``` + +From 3.13.x on MySQL, the upgrade reports `from_schema_version:17`, +`schema_version:18`, `status:upgraded`; a repeat reports `status:current`. +Then verify `/api/ready`, `/api/cluster/info` version `3.14.0`, and worker +registration. + +With Helm, run the 3.14.0 image's `schema-upgrade` through the opt-in +`native.schemaUpgrade.enabled` hook or the manual Job described in the chart's +native schema upgrade section, after scaling every Server replica to zero. The +chart that selects 3.14.0 by default is published separately; follow its +versioned upgrade notes, and make any explicit `image.tag` or `image.digest` +override select the intended image. + +Until `schema-upgrade` has run, going back to 3.13.x is an ordinary image +rollback. After it, rolling back requires restoring the pre-upgrade database +and payload-storage backup. A 3.13.x binary does not ignore the new table: it +refuses the upgraded database with `native_schema_newer_than_binary` and does +not start, so do not try to run one against it. Workflows, supervision +records and objects written after the upgrade are not in that backup. + +Worker protocol `1.20`, control-plane version `2`, codec and command shapes, and +the public external-payload reference shape are unchanged. New capabilities +are `worker_build_retirement` and `prometheus_text_metrics`. + +## Unchanged + +The 3.12.0 storage bindings, the 3.13.0 history budget, schedule run ids, +worker registration retention and identifier bounds are unchanged. The PHP +Server remains 2.11.8, retired and unpatched; see the +[retirement/security notice](../security/retired-php-server-2026-11-13.md). diff --git a/docs/rust-server-upgrade-takeover.md b/docs/rust-server-upgrade-takeover.md index 6f001e59e..39cf886e8 100644 --- a/docs/rust-server-upgrade-takeover.md +++ b/docs/rust-server-upgrade-takeover.md @@ -743,7 +743,7 @@ named. These are the ones a client moving from 2.x can observe: as final. A workflow that needs a loser's effect must await it before returning. The parity harness records this as the named divergence `run-close-settles-open-work`. -- **Open children at continue-as-new (native source, unreleased; Server +- **Open children at continue-as-new (native Server 3.14.0; Server #1184).** PHP 2.x leaves a child that is still open when its parent continues as new to itself: no run of the chain applies its parent-close policy, and a later failure is never retried. Native follows Core diff --git a/docs/server-reference.md b/docs/server-reference.md index 5db40c09c..3468c1841 100644 --- a/docs/server-reference.md +++ b/docs/server-reference.md @@ -1478,7 +1478,7 @@ work open, so a workflow that needs a loser's effect must await it before returning. When a run continues as new while children it started are still open (native -source, unreleased; Server #1184, Core workflow#868 in Workflow 2.26.0), the +Server 3.14.0; Server #1184, Core workflow#868 in Workflow 2.26.0), the continued run takes over their supervision in the same transaction. The child link, child call, child identity, deadlines and `parent_close_policy` stay as recorded, and the child's `WorkflowStarted` still names the run that started