Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 69 additions & 0 deletions docs/contracts/server-capabilities-digest.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
# Server capabilities digest (opt-in)

Every worker-plane JSON answer carries `server_capabilities`, as PHP's
`WorkerProtocol::json` requires. A completion that carries a poll
(`activity_poll` or `workflow_poll`) carries a second copy inside its `answer`.
The object is about 17 KB, so it is most of a worker's response bytes. In a lab
run of the standard one-activity workflow it was about 306 KB of the 385 KB of
responses per workflow.

## Request

A worker opts in by sending

```
X-Durable-Workflow-Server-Capabilities-Digest: <digest>|none
```

on worker-plane requests. The value is the digest of the capabilities the
worker holds, or `none` if it holds none yet. A request without the header gets
exactly the same bytes as before.

## Answer

For an opted-in request, the server compares the request's header with the
digest of its current capabilities.

- **Match:** each copy of the current capabilities in the answer is replaced by
the digest string. That covers the top level and up to two objects deep, as in
a carried poll's `answer`.
- **No match:** each copy is sent in full with one added member, `digest`. Its
value is the digest of the capabilities without that member.
- Every opted-in answer also carries the current digest in the same response
header.

The digest is `sha256:` followed by the lowercase hex SHA-256 of the
capabilities' compact JSON. A `server_capabilities` value that differs from the
current capabilities is never replaced.

## Resolving a digest string: per request

The server replaces copies only with the digest **that request sent**. A digest
string in an answer therefore always names the capabilities the worker held when
it sent that request, never a value it learned later.

Requests can overlap. A worker can have a completion in flight under digest
`D1` while another answer moves it to `D2`; this happens during a rolling
deploy, or when the capabilities change at run time. A worker must handle this:

- Capture the digest and the capabilities object when it sends each request.
- Resolve a digest string in that request's answer against the captured pair,
not against whatever it holds when the answer arrives.
- Learn a newer digest only from a tagged full copy, and use it for requests
sent afterwards.

A worker that keeps only its latest digest would reject a correct `D1` answer
that arrives after it switched to `D2`. If that answer came after the server
had already committed the request, the worker would lose a committed outcome.

A digest string that matches neither the digest captured for that request nor a
tagged copy in the same answer is a protocol failure.

## Compatibility and layering

Servers that do not support the header ignore it and always answer in full, so
a worker that sends it works against both.

On the server, the digest layer runs after the worker envelope is complete and
inside response compression. Compression then encodes the smaller body. The
digest layer never rewrites an already-encoded body.
12 changes: 10 additions & 2 deletions docs/rust-server-port.md
Original file line number Diff line number Diff line change
Expand Up @@ -338,8 +338,16 @@ contract fixture before its correction.
Capabilities are true only where native serves the feature.
`workflow_activity_result_inspection` is now advertised: the
`cluster-capabilities` fixture qualifies the activity result read against
published 2.11.7. Native advertises no manifest it does not publish and no
response compression. `native_unqualified` names each member it still
published 2.11.7. Native advertises no manifest it does not publish.
Like PHP's `CompressResponse`, native gzip- or deflate-encodes JSON answers
of 1024 bytes or more when `Accept-Encoding` allows it, and advertises
`response_compression: ["gzip", "deflate"]`, unless
`DW_COMPRESSION_ENABLED` is false. A worker can opt out of receiving
`server_capabilities` again in every answer: it sends
`X-Durable-Workflow-Server-Capabilities-Digest` and gets the digest string in
place of copies it already holds
([contract](contracts/server-capabilities-digest.md)). That layer runs inside
compression, so the smaller body is still encoded. `native_unqualified` names each member it still
leaves out: `topology`, the diagnostics view, the external payload
transaction limit and the conformance manifests. Since server#1005 (3.1.0)
native serves `supported_sdk_versions` and `client_compatibility` from one
Expand Down
12 changes: 12 additions & 0 deletions docs/server-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -1462,6 +1462,18 @@ worker capability fields. Adding new workflow-task commands or optional
capability booleans is additive; removing or renaming a command or capability
requires a protocol version bump.

Native Server: a worker can send
`X-Durable-Workflow-Server-Capabilities-Digest: <digest>|none` on
worker-plane requests to stop receiving the same `server_capabilities` object
in every answer. When the header names the current digest, each copy of the
current capabilities (top level, or inside a carried poll's `answer`) is
replaced by the digest string. Otherwise each copy is sent in full with an added
`digest` member. Every opted-in answer also carries the current digest in the
same response header. A digest string always names the digest that request
sent, so a worker resolves it against the capabilities it captured when sending
that request. Requests without the header are unchanged. The full contract is in
`docs/contracts/server-capabilities-digest.md`.

Workflow task polling returns a leased task plus `workflow_task_attempt`. Clients
must echo both `workflow_task_attempt` and `lease_owner` on workflow-task
`heartbeat`, `complete`, and `fail` calls. Workflow-task completion supports
Expand Down
4 changes: 2 additions & 2 deletions rust/src/config_audit.rs
Original file line number Diff line number Diff line change
Expand Up @@ -581,7 +581,7 @@ pub const SETTINGS: &[Setting] = &[
Setting { name: "DW_BOOTSTRAP_RETRIES", legacy: Some("WORKFLOW_SERVER_BOOTSTRAP_RETRIES"), disposition: Ignored },
Setting { name: "DW_CALLER_LABEL_HEADER", legacy: Some("WORKFLOW_SERVER_CALLER_LABEL_HEADER"), disposition: Ignored },
Setting { name: "DW_CALLER_TYPE_HEADER", legacy: Some("WORKFLOW_SERVER_CALLER_TYPE_HEADER"), disposition: Ignored },
Setting { name: "DW_COMPRESSION_ENABLED", legacy: Some("WORKFLOW_SERVER_COMPRESSION_ENABLED"), disposition: Ignored },
Setting { name: "DW_COMPRESSION_ENABLED", legacy: Some("WORKFLOW_SERVER_COMPRESSION_ENABLED"), disposition: Honored },
Setting { name: "DW_DEFAULT_NAMESPACE", legacy: Some("WORKFLOW_SERVER_DEFAULT_NAMESPACE"), disposition: Refused(AnyCase("default"), DEFAULT_NAMESPACE) },
Setting { name: "DW_DUE_TIMER_RECOVERY_SCAN_LIMIT", legacy: Some("WORKFLOW_SERVER_DUE_TIMER_RECOVERY_SCAN_LIMIT"), disposition: Ignored },
Setting { name: "DW_ENV_AUDIT_STRICT", legacy: None, disposition: Honored },
Expand Down Expand Up @@ -995,7 +995,7 @@ mod tests {
&[("DW_V2_PIN_TO_RECORDED_FINGERPRINT", "no")],
&[("DW_SERIALIZER", "")],
&[("DW_STORAGE_CONNECTION", "pgsql"), ("DB_CONNECTION", "pgsql")],
// Ignored and reported settings take any value.
// Ignored and reported settings, and honored booleans, take any value.
&[("DW_MAX_PENDING_ACTIVITIES", "10"), ("DW_COMPRESSION_ENABLED", "false")],
];
for values in accepted {
Expand Down
222 changes: 222 additions & 0 deletions rust/src/runtime/capabilities_digest.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,222 @@
//! Worker answers without repeated server capabilities (opt-in).
//!
//! Every worker-plane JSON answer carries `server_capabilities` (PHP's
//! `WorkerProtocol::json`), and a completion that carries a poll carries a
//! second copy in its `answer`. The object is about 17 KB, so on a small
//! deployment it is most of a worker's response bytes. A worker that sends
//! `X-Durable-Workflow-Server-Capabilities-Digest` opts in:
//!
//! - the header names the digest of the capabilities the worker holds, or
//! `none`;
//! - when it names the current digest, each copy of the current capabilities
//! in the answer (top level, or one or two objects deep, as in a carried
//! poll's `answer`) is replaced by the digest string;
//! - otherwise each copy is sent in full with a `digest` member added, which
//! the worker keeps for its next request;
//! - every opted-in answer also has the current digest in the same header.
//!
//! A request without the header gets exactly the bytes it got before. The
//! digest is `sha256:` and the lowercase hex SHA-256 of the capabilities'
//! compact JSON. A copy that differs from the current capabilities is never
//! replaced.
use axum::{
body::{Body, HttpBody as _},
extract::Request,
http::{HeaderValue, header},
middleware::Next,
response::Response,
};
use serde_json::{json, value::RawValue};
use sha2::{Digest, Sha256};

pub(super) const HEADER: &str = "x-durable-workflow-server-capabilities-digest";
/// The largest answer that is rewritten; bigger ones pass through.
const MAX_BYTES: u64 = 64 * 1024 * 1024;
/// Top level, a carried poll, and its `answer`.
const MAX_DEPTH: usize = 3;

pub(super) fn digest(text: &str) -> String {
let hash = Sha256::digest(text.as_bytes());
let mut digest = String::with_capacity(7 + 64);
digest.push_str("sha256:");
for byte in hash {
digest.push_str(&format!("{byte:02x}"));
}
digest
}

pub(super) async fn elide(request: Request, next: Next) -> Response {
let path = request.uri().path();
let worker = path == "/api/worker" || path.starts_with("/api/worker/");
let held = worker
.then(|| request.headers().get(HEADER).and_then(|value| value.to_str().ok()).map(|value| value.trim().to_owned()))
.flatten();
let response = next.run(request).await;
match held {
Some(held) => rewrite(response, &held).await,
None => response,
}
}

async fn rewrite(response: Response, held: &str) -> Response {
let current = super::store::capabilities().to_string();
let current_digest = digest(&current);
let json = response
.headers()
.get(header::CONTENT_TYPE)
.and_then(|value| value.to_str().ok())
.is_some_and(|value| value.starts_with("application/json"));
let size = response.body().size_hint().exact();
let (mut parts, body) = response.into_parts();
if let Ok(value) = HeaderValue::from_str(&current_digest) {
parts.headers.insert(HEADER, value);
}
if !json || parts.headers.contains_key(header::CONTENT_ENCODING) || !size.is_some_and(|size| size <= MAX_BYTES) {
return Response::from_parts(parts, body);
}
let Ok(bytes) = axum::body::to_bytes(body, MAX_BYTES as usize).await else {
// An exact-sized in-memory body cannot fail to collect.
return Response::from_parts(parts, Body::empty());
};
let replacement = if held == current_digest {
json!(current_digest).to_string()
} else {
// `{...}` plus the digest member before the closing brace.
let mut tagged = current[..current.len() - 1].to_owned();
if tagged.len() > 1 {
tagged.push(',');
}
tagged.push_str(&format!("\"digest\":{}}}", json!(current_digest)));
tagged
};
match std::str::from_utf8(&bytes).ok().and_then(|text| replace(text, &current, &replacement, 1)) {
Some(rewritten) => {
parts.headers.remove(header::CONTENT_LENGTH);
Response::from_parts(parts, Body::from(rewritten))
}
None => Response::from_parts(parts, Body::from(bytes)),
}
}

/// The object with each `server_capabilities` member whose raw value is
/// `current` replaced, or `None` when nothing changed or it is not an object.
/// Members are rewritten from their raw text, so only whitespace between
/// them changes.
fn replace(text: &str, current: &str, replacement: &str, depth: usize) -> Option<String> {
if !text.trim_start().starts_with('{') {
return None;
}
let super::http::ObjectMembers(members) = serde_json::from_str(text).ok()?;
let mut changed = false;
let rendered: Vec<String> = members
.iter()
.map(|(key, raw): &(String, &RawValue)| {
let raw = raw.get();
let value = if key == "server_capabilities" && raw == current {
changed = true;
replacement.to_owned()
} else if depth < MAX_DEPTH {
match replace(raw, current, replacement, depth + 1) {
Some(nested) => {
changed = true;
nested
}
None => raw.to_owned(),
}
} else {
raw.to_owned()
};
format!("{}:{value}", json!(key))
})
.collect();
changed.then(|| format!("{{{}}}", rendered.join(",")))
}

#[cfg(test)]
mod tests {
use super::*;
use axum::{Router, middleware, routing::post};
use serde_json::Value;
use tower::ServiceExt as _;

fn caps() -> Value {
super::super::store::capabilities()
}

fn app() -> Router {
let carried = || async {
axum::Json(json!({"outcome": "completed", "server_capabilities": caps(),
"activity_poll": {"leased": true, "answer": {"task": {"id": 1}, "poll_status": "leased", "server_capabilities": caps()}},
"protocol_version": "1.20"}))
};
Router::new()
.route("/api/worker/activity-tasks/poll", post(|| async { axum::Json(json!({"task": null, "server_capabilities": caps()})) }))
.route("/api/worker/workflow-tasks/t/complete", post(carried))
.route("/api/worker/other", post(|| async { axum::Json(json!({"server_capabilities": {"different": true}})) }))
.route("/api/workflows", post(|| async { axum::Json(json!({"server_capabilities": caps()})) }))
.layer(middleware::from_fn(elide))
}

async fn call(path: &str, held: Option<&str>) -> (Option<String>, Value, Vec<u8>) {
let mut request = axum::http::Request::builder().method("POST").uri(path);
if let Some(held) = held {
request = request.header(HEADER, held);
}
let response = app().oneshot(request.body(Body::empty()).unwrap()).await.unwrap();
let header = response.headers().get(HEADER).map(|value| value.to_str().unwrap().to_owned());
let bytes = axum::body::to_bytes(response.into_body(), usize::MAX).await.unwrap().to_vec();
(header, serde_json::from_slice(&bytes).unwrap(), bytes)
}

#[tokio::test]
async fn answers_without_the_header_are_unchanged() {
for path in ["/api/worker/activity-tasks/poll", "/api/worker/workflow-tasks/t/complete"] {
let (header, body, _) = call(path, None).await;
assert_eq!(header, None);
assert_eq!(body["server_capabilities"], caps());
}
let (_, body, _) = call("/api/worker/workflow-tasks/t/complete", None).await;
assert_eq!(body["activity_poll"]["answer"]["server_capabilities"], caps());
}

#[tokio::test]
async fn a_worker_without_the_digest_gets_tagged_capabilities() {
let current = digest(&caps().to_string());
for held in ["none", "sha256:0000"] {
let (header, body, _) = call("/api/worker/workflow-tasks/t/complete", Some(held)).await;
assert_eq!(header.as_deref(), Some(current.as_str()));
let mut tagged = caps();
tagged["digest"] = json!(current);
assert_eq!(body["server_capabilities"], tagged);
assert_eq!(body["activity_poll"]["answer"]["server_capabilities"], tagged);
// The digest covers the capabilities without the digest member.
let mut untagged = body["server_capabilities"].clone();
untagged.as_object_mut().unwrap().remove("digest");
assert_eq!(digest(&untagged.to_string()), current);
}
}

#[tokio::test]
async fn a_worker_with_the_current_digest_gets_the_digest_in_place_of_each_copy() {
let current = digest(&caps().to_string());
let (header, body, bytes) = call("/api/worker/workflow-tasks/t/complete", Some(&current)).await;
assert_eq!(header.as_deref(), Some(current.as_str()));
assert_eq!(body["server_capabilities"], json!(current));
assert_eq!(body["activity_poll"]["answer"]["server_capabilities"], json!(current));
assert_eq!(body["activity_poll"]["answer"]["task"], json!({"id": 1}));
assert_eq!(body["outcome"], json!("completed"));
assert!(bytes.len() < 1024, "{}", bytes.len());
let (_, poll, _) = call("/api/worker/activity-tasks/poll", Some(&current)).await;
assert_eq!(poll, json!({"task": null, "server_capabilities": current}));
}

#[tokio::test]
async fn other_values_and_the_control_plane_are_never_replaced() {
let current = digest(&caps().to_string());
let (_, body, _) = call("/api/worker/other", Some(&current)).await;
assert_eq!(body["server_capabilities"], json!({"different": true}));
let (header, body, _) = call("/api/workflows", Some(&current)).await;
assert_eq!(header, None);
assert_eq!(body["server_capabilities"], caps());
}
}
6 changes: 3 additions & 3 deletions rust/src/runtime/cluster_info.rs
Original file line number Diff line number Diff line change
Expand Up @@ -134,8 +134,8 @@ pub(super) fn capabilities() -> Value {
json!({"schema": "durable_workflow.protocol.Value", "fingerprint": super::history_export::VALUE_FINGERPRINT,
"framing": "single_object", "magic_hex": "c301"}),
);
// Native sends uncompressed responses.
map.insert("response_compression".into(), json!([]));
// PHP's CompressResponse encodings, unless DW_COMPRESSION_ENABLED is false.
map.insert("response_compression".into(), json!(super::response_compression::advertised()));
Value::Object(map)
}

Expand Down Expand Up @@ -244,7 +244,7 @@ mod tests {
assert_eq!(caps["child_workflow_start_turn"], json!(true));
assert_eq!(caps["condition_wait_idempotent_reopen"], json!(true));
assert_eq!(caps["condition_wait_occurrence_deadline"], json!(true));
assert_eq!(caps["response_compression"], json!([]));
assert_eq!(caps["response_compression"], json!(["gzip", "deflate"]));
assert_eq!(caps["timer_runtime_contract"], json!(false));
assert_eq!(caps["principal_attribution_contract"], json!(true));
}
Expand Down
12 changes: 1 addition & 11 deletions rust/src/runtime/history_pages.rs
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,7 @@ pub(super) fn compress(payload: &mut Value, encoding: Option<&'static str>) -> R
}
let json = serde_json::to_vec(events)?;
let compressed = match encoding {
"gzip" => gzip(&json),
"gzip" => super::response_compression::gzip(&json),
_ => miniz_oxide::deflate::compress_to_vec(&json, LEVEL),
};
payload["history_events"] = json!([]);
Expand All @@ -122,16 +122,6 @@ pub(super) fn compress(payload: &mut Value, encoding: Option<&'static str>) -> R
Ok(())
}

/// `gzencode`: an RFC 1952 member around one raw DEFLATE stream.
fn gzip(data: &[u8]) -> Vec<u8> {
const CRC32: crc::Crc<u32> = crc::Crc::<u32>::new(&crc::CRC_32_ISO_HDLC);
// Magic, deflate method, no flags or time, default level, Unix.
let mut member = vec![0x1f, 0x8b, 8, 0, 0, 0, 0, 0, 0, 3];
member.extend(miniz_oxide::deflate::compress_to_vec(data, LEVEL));
member.extend(CRC32.checksum(data).to_le_bytes());
member.extend((data.len() as u32).to_le_bytes());
member
}

#[cfg(test)]
mod tests {
Expand Down
Loading