Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docker-compose.published.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ name: durable-workflow-server
# release whose default names another image). Set DW_SERVER_IMAGE to run
# any other native image, by tag or digest; pin a digest for production
# change control.
x-server-image: &server-image ${DW_SERVER_IMAGE:-durableworkflow/server:3.11.0}
x-server-image: &server-image ${DW_SERVER_IMAGE:-durableworkflow/server:3.12.0}

services:
server:
Expand Down
175 changes: 175 additions & 0 deletions docs/releases/3.12.0.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,175 @@
# Durable Workflow Server 3.12.0 (native)

Server 3.12.0 keeps native local external payloads readable and reclaimable
after a namespace's storage write policy is disabled or moved. It also adds a
read-only query that counts and pages open runs by their retained
version-marker decision. Its immutable images are
`durableworkflow/server:3.12.0` and `ghcr.io/durable-workflow/server:3.12.0`;
the published Compose default selects 3.12.0. Rolling aliases move separately
after published-image qualification.

## Retained payloads across storage policy changes

In 3.11.x and earlier, disabling a namespace's local external-payload policy
or moving its location left objects written under the earlier policy
unresolvable. Reads, retention pruning, abandoned-upload cleanup and namespace
deletion resolved every stored object through the current policy.

The native local driver now records an immutable binding with each admitted
object: its namespace, storage generation and canonical root. Reads, run
pruning, abandoned-upload cleanup and namespace deletion use that original
binding. Disabling a policy stops new uploads only. Moving the location sends
new objects to the new root; existing objects stay where they were. Changing
the policy does not copy objects between roots. A disable request that omits
the location now keeps the configured location. An upload admitted before a
policy change can finish at its recorded root; later uploads must pass the
current enabled policy. Deletion journals keep their binding, so mixed
generations, backup holds and another namespace's ownership of the same
physical object are preserved.

Keep every retained root mounted and accessible to the Server and its
maintenance processes until its objects and committed deletion records are
reclaimed. A missing or substituted root returns the nonretryable
`external_payload_storage_binding_unavailable` outcome (HTTP 409). The runtime
does not recreate an empty root or infer storage authority from an arbitrary
URI. Backups must include the registry and deletion journals together with the
bytes of every retained root, including roots no longer selected for uploads.
See [native local-storage policy changes](../contracts/external-payload-storage.md#native-local-storage-policy-changes).

### Objects written before the upgrade

The schema step adds the binding column without backfilling it. An existing
registration is bound on first use only when the namespace's admitted local
policy reproduces its recorded URI exactly. A policy change first pins every
unbound registration to the current policy or, when the request names an
explicit location, to that location where the recorded URI matches. If any
unbound registration matches neither, the policy change is refused with
`external_payload_storage_binding_unavailable` and the policy is not replaced.

Objects whose namespace still resolves to their original root, including
through a disabled policy, become readable and are bound on first use. Under
3.11.x and earlier, moving the location, or disabling with a request that
omitted a custom location, replaced the root the Server resolves. Objects
stranded that way stay unavailable after the upgrade. To repair them, keep the
original root mounted and submit a policy change that names it explicitly
(enabled or disabled); that pins the stranded registrations to it, and an
enabled policy also sends new uploads there. A later policy change can move
new uploads again without unbinding them. Because one request can pin only to
the current and the requested location, stranded registrations under more than
one earlier root cause every policy change to be refused; this release
documents no repair for that case. This release does not relocate objects or
recover bytes that are no longer present at their recorded location.

## Open-run version usage

`GET /api/workflow-version-usage` reports, for one namespace, workflow type and
change ID, which open runs retain a version-marker decision below a proposed
`min_version` (`legacy`), at or above it (`current`), or have no authoritative
answer (`unknown`). Discover support by requiring
`capabilities.workflow_version_usage` to be exactly `true` on
`GET /api/cluster/info`; a missing capability or a failed request is not an
empty cohort. The endpoint requires an operator or admin token and the
existing control-plane version header.

Missing markers, pruned or unsupported history, malformed ranges and conflicting
decisions remain `unknown` with a specific reason. Counts cover the whole
open-run cohort before the optional state filter, and each response is one
read-only snapshot. Signed, opaque cursors page by ascending run ID under the
first page's fixed upper bound; invalid or mismatched cursors return HTTP 422.
Pages are a live observation, not a deployment fence, and the response carries
no safe-to-remove flag. The query reads no workflow arguments, results, memos or
external payloads and changes no marker production, history or replay
behavior. The shared v1 contract and paging rules are in
[open-run version usage](../workflow-version-usage.md).

## Portable native library builds

Unix-specific storage-admission observation checks are now compiled only on
Unix targets, so the native library builds on other platforms, such as the
CLI's Windows development runtime. Admission stays disabled by default. A
configured observation file on a non-Unix target fails closed with
`storage_admission_unavailable`; no fallback reader is introduced. The Unix
descriptor, ownership and mode checks are unchanged, and the published Linux
images behave as in 3.11.0. This does not add native image platforms. See
[storage admission](../storage-admission.md#admission-observation-file).

## Upgrade and rollback

This release requires an explicit additive schema upgrade from 3.11.x on every
backend:

| Backend | Schema step |
| --- | --- |
| SQLite | 14 → 15 |
| PostgreSQL | 15 → 16 |
| MariaDB | 15 → 16 |
| MySQL | 16 → 17 |

Each step adds the nullable `runtime_external_payloads.storage_binding` column
and `workflow_namespaces.external_payload_storage_generation` (default `0`).
It rewrites no existing rows. Stop every Server replica, keep the database
running, and back up the database and every external-payload root. Select the
new immutable image and run that image's `schema-upgrade` once before
restarting every replica on the same version. Startup does not upgrade an older
schema automatically and refuses to serve it. MySQL and MariaDB resume an
interrupted upgrade when `schema-upgrade` is run again; SQLite and PostgreSQL
apply their step transactionally. From 3.9.x or earlier, the same command also
applies the intervening steps described in the
[3.10.0 upgrade notes](3.10.0.md#upgrade-and-rollback).

For a single-node Compose deployment, stop Server while leaving MySQL running,
back up MySQL and `server_storage`, set `DW_SERVER_IMAGE` to the 3.12.0 image,
then pull and upgrade once:

```bash
docker compose --env-file durable-workflow.prod.env \
-f docker-compose.published.yml stop server
docker compose --env-file durable-workflow.prod.env \
-f docker-compose.published.yml pull server
docker compose --env-file durable-workflow.prod.env \
-f docker-compose.published.yml run --rm --no-deps server schema-upgrade
docker compose --env-file durable-workflow.prod.env \
-f docker-compose.published.yml up -d --wait
```

From 3.11.x on MySQL, the upgrade reports `from_schema_version:16`,
`schema_version:17`, `status:upgraded`; a repeat reports `status:current`.
Then verify `/api/ready`, `/api/cluster/info` version `3.12.0`, and worker
registration.

With Helm, run the 3.12.0 image's `schema-upgrade` through the opt-in
`native.schemaUpgrade.enabled` hook or the manual Job described in the chart's
native schema upgrade section, after scaling every Server replica to zero. The
chart that selects 3.12.0 by default is published separately; follow its
versioned upgrade notes, and make any explicit `image.tag` or `image.digest`
override select the intended image.

Rolling back requires restoring the pre-upgrade database and payload-storage
backup. An older binary refuses the upgraded database with
`native_schema_newer_than_binary`; do not try to run one against it. Objects
written and policy changes made after the upgrade are not in that backup.

Worker protocol `1.20`, control-plane version `2`, codec and command shapes, and
the public external-payload reference shape are unchanged. Existing
command-key, completion-byte, storage-admission and unsupported service-upsert
boundaries remain in force. The 3.11.0 fired-timer redrive rules are unchanged.

## Qualification and retained limits

The storage-binding fix is native only. Published PHP Server 2.11.8 remains
retired and unpatched and keeps its earlier behavior; see the
[retirement/security notice](../security/retired-php-server-2026-11-13.md).
The native parity contract records the retention and namespace-deletion
expectations that now differ from PHP. The PHP takeover runbook keeps its exact
historical version, backend and state limits.

CLI and Waterline adoption of the version-usage endpoint is tracked on
[#1220](https://github.com/durable-workflow/server/issues/1220) and remains
separately attributable. The CLI's Windows development runtime is a source
build and does not qualify a published Server image. Installed SDK timer-clock
evidence ([#980](https://github.com/durable-workflow/server/issues/980)) and the
MariaDB registration-timeout investigation
([#1204](https://github.com/durable-workflow/server/issues/1204)) are unchanged
by this release. Publication, exact-image corpus, the published Compose cells,
scans, protected rolling aliases and the final Release must qualify this new
schema image.
Loading