Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docker-compose.published.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ name: durable-workflow-server
# release whose default names another image). Set DW_SERVER_IMAGE to run
# any other native image, by tag or digest; pin a digest for production
# change control.
x-server-image: &server-image ${DW_SERVER_IMAGE:-durableworkflow/server:3.13.0}
x-server-image: &server-image ${DW_SERVER_IMAGE:-durableworkflow/server:3.14.0}

services:
server:
Expand Down
251 changes: 251 additions & 0 deletions docs/releases/3.14.0.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,251 @@
# Durable Workflow Server 3.14.0 (native)

Server 3.14.0 hands a run's open children to its continued run at
continue-as-new, answers queries on closed runs, guards worker build and task
queue retirement, exposes queue backlog to autoscalers at `GET /metrics`, gives
standalone activity starts a duplicate policy, keeps MariaDB completions within
the default packet, admits leased completion uploads while storage drains, and
honors the native long-poll caps. Its immutable images are
`durableworkflow/server:3.14.0` and `ghcr.io/durable-workflow/server:3.14.0`;
the published Compose default selects 3.14.0. Rolling aliases move separately
after published-image qualification.

**Upgrade:** this release requires an explicit additive `schema-upgrade` from
3.13.x on every backend; see [Upgrade and rollback](#upgrade-and-rollback).
After the schema step, rolling back to 3.13.x requires restoring the
pre-upgrade backup: a 3.13.x binary refuses the upgraded database with
`native_schema_newer_than_binary`. Worker protocol and control-plane versions
are unchanged.

## Behavior changes to review

- A standalone activity start for an id that is still running now returns a
typed `409 activity_already_started` instead of `422 validation_failed`.
Clients that matched the 422 must match the 409, or send
`duplicate_policy: "use-existing"` (see
[Standalone activity duplicate policy](#standalone-activity-duplicate-policy)).
- Queries on failed, cancelled and terminated runs are answered instead of
refused with `409 run_not_active`. A run whose history was pruned or is
missing now gets `409 query_history_unavailable` instead of
`503 storage_unavailable`.
- A workflow-task completion receipt stores a digest of the request, not the
request. Do not run 3.13.x and 3.14.0 nodes against one database: a 3.13.x
node would answer an identical retry of a completion a 3.14.0 node recorded
with `409 conflicting_completion`. The schema step already rules out a mixed
fleet, because 3.13.x refuses the upgraded schema.

## Open children at continue-as-new

When a run continues as new while children it started are still open, the
continued run now takes over their supervision in the continue-as-new
transaction, as Core does since Workflow 2.26.0 (workflow#868, workflow#1107).
The child link, child call, child identity, deadlines and `parent_close_policy`
stay as recorded, and the child's `WorkflowStarted` still names the run that
started it. Each continue-as-new passes on every child the run still
supervises, including children an earlier run of the chain handed to it.

When the chain's final run completes, fails, times out, is cancelled or is
terminated, it applies each handed-over child's `request_cancel`, `terminate`
or `request_cancellation` policy to the child's current run and records
`ParentClosePolicyApplied` or `ParentClosePolicyFailed` in its own history.
`abandon` children are handed over and left alone. A handed-over child that
fails while a run of the chain still supervises it is retried under the
starting run's recorded retry policy; once the chain has closed, the failure is
final. The continued run gets no handle on the child or its result, and a
closed predecessor gets no history or task when a handed-over child closes.

Supervision is kept in the new native table `dw_child_supervisions`, which the
schema step creates empty. A chain continued before the upgrade has no
supervision rows, so its open children are left running and a later failure is
final, as before. No event is recorded at the hand-over, so existing histories
replay unchanged. Published PHP 2.x applies no policy to a child whose parent
continued as new (native divergence `continue-as-new-child-supervision`)
(#1184, #1246).

## Queries on closed runs

A retained run of any status, including failed, cancelled and terminated, is
queried read-only. The worker receives that run's id, status and full history up
to its terminal event; nothing is written to the run, its history or its tasks.
Describe's `actions.can_query` follows the same rule. A run whose history was
pruned by retention, or whose `WorkflowStarted` is missing, is refused with
`409 query_history_unavailable` (with `history_state`, `run_status` and
`is_terminal`) and no query task is admitted. The query contract lists the new
reason after `run_not_active`, which stays listed for older servers.
[Workflow queries](../server-reference.md#workflow-queries) now lists every query failure and
timeout reason. Published PHP keeps `run_not_active` (native divergence
`closed-run-queries`) (#587, #1240).

## Guarded worker build and task queue retirement

`GET /api/deployments/{namespace}/{queue}@{build}/retirement` and
`GET /api/task-queues/{queue}/retirement` return one read-only inventory of what
still depends on a build or queue: open, waiting, continued and child runs;
ready, leased and delayed tasks; open activities, update validations and
schedules; and open runs with no recorded queue. Each blocker has an exact
count, up to 20 samples and a recovery hint.

`POST /api/deployments/{name}/retire` and `POST /api/task-queues/{queue}/retire`
repeat that inventory inside native's serialized transition and write only
when it is complete and empty, optionally bound to
`expected_inventory_revision`. Otherwise they return `409` with the inventory and
change nothing; there is no force path. A retired build is not selectable for
new starts and its workers stay draining. Discover the feature through
`capabilities.worker_build_retirement`. See
[worker build retirement](../worker-build-retirement.md) (#519, #1241).

## Queue backlog for autoscalers

`GET /metrics` serves Prometheus text format 0.0.4 for one namespace: ready,
due, delayed, leased and dispatchable tasks per queue, kind and build id;
active pollers and worker slots; admission state; activity retry pressure;
schedules; and run and activity gauges. Backlog is read from task rows, so due
work is visible while no worker runs. Series are bounded at 100 queues and 10
build ids per queue and kind, with truncation disclosed. A failed read answers
`503` with no samples.

The route accepts a new scrape-only `metrics` role, granted by
`DW_METRICS_TOKEN` (which must differ from every other token) or by a runtime
credential with `"roles":["metrics"]`; admin tokens can also scrape. Discover it
through `capabilities.prometheus_text_metrics`. KEDA, ServiceMonitor and scrape
examples are in `k8s/helm/examples/autoscaling/`. The packaged chart has no
first-class metrics values yet; pass the token through `auth.extra` or an
existing Secret (#534, #1242).

## Standalone activity duplicate policy

`POST /api/activities` accepts `duplicate_policy`, `fail` (default) or
`use-existing`, mirroring workflow start:

| Activity id | `fail` | `use-existing` |
| --- | --- | --- |
| Still running | `409 activity_already_started`, `outcome: rejected_duplicate` | `200` with the running activity, `outcome: returned_existing_active` |
| Closed or new | `201`, `outcome: started_new` | `201`, `outcome: started_new` |

**Behavior change:** before 3.14.0 a running id was refused with
`422 validation_failed` on `activity_id`. Neither duplicate answer writes
anything, and both are answered before the queue drain check. `use-existing`
does not compare input or type. An invalid policy gets `422`. The Server
reference now documents [standalone activities](../server-reference.md#standalone-activities) (#590, #1243).

## MariaDB completions within the default packet

A workflow-task completion receipt now stores a SHA-256 digest of the canonical
request and the created task ids instead of the whole request, so completions
up to the advertised 32 MiB `limits.max_workflow_task_completion_bytes` record
on MariaDB's default 16 MiB `max_allowed_packet`. The 3.13.0 advice to raise the
packet to 64 MiB is no longer needed for this release. An identical retry still
replays `already_completed`, any other completion is still refused, and
receipts written by earlier releases still match. On MySQL and MariaDB, startup
logs `database_packet_limit`, or `database_packet_limit_below_payload_headroom`
when the packet is under four times `DW_MAX_PAYLOAD_BYTES`. A completion the
database still refuses with a packet error is answered
`413 payload_too_large`, `limit_kind: database_max_allowed_packet`, and nothing
is recorded.

**Mixed versions:** a 3.13.x node reads only full-request receipts, so it would
answer an identical retry of a completion recorded by a 3.14.0 node with
`409 conflicting_completion`. Because 3.14.0 requires the schema step and 3.13.x
refuses the upgraded schema, the two cannot serve one database; upgrade every
node together as described below (#1234, #1244).

## Completion uploads while storage drains

While storage admission is `draining`, a worker's activity-result upload
carrying `X-Durable-Workflow-Payload-Completion` is admitted for its current
activity lease within the per-lease allowance (128 slots,
`DW_EXTERNAL_PAYLOAD_COMPLETION_MAX_BYTES`). Uploads without the header and
workflow, query and activity-failure contexts are still refused. An exhausted
allowance while draining answers the storage refusal (`503 storage_pressure`,
`Retry-After: 5`) and writes nothing. Admission is rechecked after the body is
read and before each write (#951, #1245).

## Native long-poll capacity

Native now honors `DW_WORKER_LONG_POLL_MAX_CONCURRENT` (held workflow and
activity waits), `DW_QUERY_TASK_POLL_MAX_CONCURRENT` (held query-task and
update-validation waits) and `DW_WORKER_LONG_POLL_RESERVED_HTTP_WORKERS`, which
native applies as database connections reserved for health, readiness and admin
requests. A poll probes for work before it takes a slot; when no slot is free it
gets `429 long_poll_capacity_exhausted` with `Retry-After: 1`. Each cap is per
process. When none is set, behavior is unchanged. The per-namespace variants
remain ignored (#982, #1236).

## Opt-in timeout cleanup (not yet advertised)

Native implements deadline-triggered timeout cleanup (`timeout_cleanup_seconds`
on workflow and child starts), following embedded Workflow 2.29.0. Admission
requires a live worker on the queue that advertises the `timeout_cleanup`
capability; otherwise the start is refused with
`422 timeout_cleanup_not_supported`, never downgraded to a hard timeout.
`capabilities.timeout_cleanup` stays unadvertised until installed cross-SDK
acceptance passes. Starts without the option are unchanged and write no new key
or event (#1201, #1235).

## Upgrade and rollback

This release requires an explicit additive schema upgrade from 3.13.x on every
backend:

| Backend | Schema step |
| --- | --- |
| SQLite | 15 → 16 |
| PostgreSQL | 16 → 17 |
| MariaDB | 16 → 17 |
| MySQL | 17 → 18 |

Each step creates the empty `dw_child_supervisions` table and its indexes. It
rewrites no existing rows. Stop every Server replica, keep the database
running, and back up the database and every external-payload root. Select the
new immutable image and run that image's `schema-upgrade` once before
restarting every replica on the same version. Startup does not upgrade an older
schema automatically and refuses to serve it (`native_schema_upgrade_required`).
MySQL and MariaDB resume an interrupted upgrade when `schema-upgrade` is run
again; SQLite and PostgreSQL apply their step transactionally. From 3.11.x or
earlier, the same command also applies the intervening steps described in the
[3.12.0 upgrade notes](3.12.0.md#upgrade-and-rollback).

For a single-node Compose deployment, stop Server while leaving MySQL running,
back up MySQL and `server_storage`, set `DW_SERVER_IMAGE` to the 3.14.0 image,
then pull and upgrade once:

```bash
docker compose --env-file durable-workflow.prod.env \
-f docker-compose.published.yml stop server
docker compose --env-file durable-workflow.prod.env \
-f docker-compose.published.yml pull server
docker compose --env-file durable-workflow.prod.env \
-f docker-compose.published.yml run --rm --no-deps server schema-upgrade
docker compose --env-file durable-workflow.prod.env \
-f docker-compose.published.yml up -d --wait
```

From 3.13.x on MySQL, the upgrade reports `from_schema_version:17`,
`schema_version:18`, `status:upgraded`; a repeat reports `status:current`.
Then verify `/api/ready`, `/api/cluster/info` version `3.14.0`, and worker
registration.

With Helm, run the 3.14.0 image's `schema-upgrade` through the opt-in
`native.schemaUpgrade.enabled` hook or the manual Job described in the chart's
native schema upgrade section, after scaling every Server replica to zero. The
chart that selects 3.14.0 by default is published separately; follow its
versioned upgrade notes, and make any explicit `image.tag` or `image.digest`
override select the intended image.

Until `schema-upgrade` has run, going back to 3.13.x is an ordinary image
rollback. After it, rolling back requires restoring the pre-upgrade database
and payload-storage backup. A 3.13.x binary does not ignore the new table: it
refuses the upgraded database with `native_schema_newer_than_binary` and does
not start, so do not try to run one against it. Workflows, supervision
records and objects written after the upgrade are not in that backup.

Worker protocol `1.20`, control-plane version `2`, codec and command shapes, and
the public external-payload reference shape are unchanged. New capabilities
are `worker_build_retirement` and `prometheus_text_metrics`.

## Unchanged

The 3.12.0 storage bindings, the 3.13.0 history budget, schedule run ids,
worker registration retention and identifier bounds are unchanged. The PHP
Server remains 2.11.8, retired and unpatched; see the
[retirement/security notice](../security/retired-php-server-2026-11-13.md).
2 changes: 1 addition & 1 deletion docs/rust-server-upgrade-takeover.md
Original file line number Diff line number Diff line change
Expand Up @@ -743,7 +743,7 @@ named. These are the ones a client moving from 2.x can observe:
as final. A workflow that needs a loser's effect must await it before
returning. The parity harness records this as the named divergence
`run-close-settles-open-work`.
- **Open children at continue-as-new (native source, unreleased; Server
- **Open children at continue-as-new (native Server 3.14.0; Server
#1184).** PHP 2.x leaves a child that is still open when its parent
continues as new to itself: no run of the chain applies its parent-close
policy, and a later failure is never retried. Native follows Core
Expand Down
2 changes: 1 addition & 1 deletion docs/server-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -1478,7 +1478,7 @@ work open, so a workflow that needs a loser's effect must await it before
returning.

When a run continues as new while children it started are still open (native
source, unreleased; Server #1184, Core workflow#868 in Workflow 2.26.0), the
Server 3.14.0; Server #1184, Core workflow#868 in Workflow 2.26.0), the
continued run takes over their supervision in the same transaction. The child
link, child call, child identity, deadlines and `parent_close_policy` stay as
recorded, and the child's `WorkflowStarted` still names the run that started
Expand Down
Loading