diff --git a/docs/API.md b/docs/API.md index eafb5a7..dceabf3 100644 --- a/docs/API.md +++ b/docs/API.md @@ -333,6 +333,8 @@ Success returns `204 No Content`. Missing resources return `404 guarded_resource Requires the authenticated creator frontend session. With no query, or with `graceful=true`, it starts or replays the durable graceful deletion job. Queued and running work returns `202` with only `lock_id` and `status`; a completed-and-forgotten absent lock returns `200 { "lock_id": "...", "status": "completed" }`. +Graceful withdrawal compares the current public lock bytes with the frozen canonical lock before publishing the tombstone. Pubky 0.9.3 does not provide an atomic conditional write, so this check and the tombstone `PUT` are not one operation. A replacement already visible to the check is preserved and fails the deletion closed, but an out-of-band creator replacement written after that check and before the `PUT` can be overwritten by the tombstone. This is an explicitly accepted limitation until Pubky supports conditional writes. Graceful cleanup does not delete replacement bytes; unconditional public deletion remains exclusive to `force=true`. + `force=true` is explicit and cannot be combined with `graceful=true`. With no active graceful job, force deletion synchronously stores the permanent blocking receipt, removes the public lock/tombstone, and then best-effort deletes guarded resources. A terminal graceful job supplies its frozen manifest for this synchronous cleanup. It returns exactly `{ "lock_id": "...", "lock_deleted": true, "failed_resource_paths": ["..."] }`. With a queued or running graceful job, it revokes any current worker claim, durably requeues the job for force escalation, and returns that redacted job at `202`. An in-flight canonical publication returns redacted `409 content_lock_path_conflict`; force has not started and no receipt exists, so the creator may retry after publication reconciles. Unknown query keys, malformed or false booleans, and ambiguous modes return `400 invalid_request`. The accepted wire forms are exactly no query, `graceful=true`, or `force=true`. @@ -341,7 +343,7 @@ Rust SDK callers use `CreatorLocks::delete_content_lock(DeleteContentLockRequest ### `GET /creator/content-locks/{lock_id}/deletion` -Returns only `{ "lock_id": "...", "status": "queued|running|completed|failed", "failure_code"?: "..." }`. The optional failure vocabulary is closed to `tombstone_missing`, `tombstone_replaced`, `retry_exhausted`, and `state_corrupt`. If neither a job nor permanent force receipt exists, it returns `404 content_lock_deletion_not_found`. +Returns only `{ "lock_id": "...", "status": "queued|running|completed|failed", "failure_code"?: "..." }`. The optional failure vocabulary is closed to `tombstone_missing`, `tombstone_replaced`, `resource_replaced`, `retry_exhausted`, and `state_corrupt`. If neither a job nor permanent force receipt exists, it returns `404 content_lock_deletion_not_found`. ### `POST /creator/content-locks` diff --git a/docs/RUNTIME.md b/docs/RUNTIME.md index 25e43e8..f14eb37 100644 --- a/docs/RUNTIME.md +++ b/docs/RUNTIME.md @@ -117,7 +117,7 @@ key_republisher_interval_seconds = 3600 `public_pubky_tls_port` and `public_icann_http_port` advertise externally reachable ports. `icann_domain` is browser/ICANN fallback target. Local testnet operators should set `pkarr_relays = ["http://localhost:15411"]`. -PKARR publishing starts when environment is `staging`/`production` or creator-authority acquisition is enabled, and republishes every `key_republisher_interval_seconds` seconds. +PKARR publishing starts when environment is `staging`/`production` or creator-authority acquisition is enabled, and republishes every `key_republisher_interval_seconds` seconds. Initial publication begins only after the HTTP listener binds successfully; it is cancellation-aware and bounded by `deletion_worker.shutdown_timeout_seconds`, the existing lifecycle deadline. `credentials.lock_server_secret_key` must contain: @@ -255,6 +255,19 @@ For `paykit-payment`, every Paykit status-call failure schedules a normal pendin Scheduled retries and crash recovery are separate mechanisms. Expected retryable results explicitly release the current claim and set `next_attempt_at`. If a worker crashes while a task is `in_progress`, another worker may reclaim it only after `claim_expires_at`; only the worker that still owns an active claim may schedule its retry. +The deletion worker has an independent closed runtime section: + +```toml +[deletion_worker] +enabled = true +poll_interval_ms = 250 +claim_timeout_seconds = 60 +shutdown_timeout_seconds = 30 +worker_id = "deletion-worker" +``` + +All three numeric values must be positive and `worker_id` must be nonblank. Queue polling cadence is independent from durable deletion retry backoff. `shutdown_timeout_seconds` bounds coordinated worker and HTTP shutdown; once shutdown begins, the deletion worker must stop acquiring new claims. + ## Health and readiness `GET /healthz` reports process liveness: @@ -263,7 +276,7 @@ Scheduled retries and crash recovery are separate mechanisms. Expected retryable { "status": "ok" } ``` -`GET /readyz` reports runtime dependency readiness: +`GET /readyz` reports runtime dependency and enabled-worker readiness: ```json { @@ -274,3 +287,5 @@ Scheduled retries and crash recovery are separate mechanisms. Expected retryable ``` Health/readiness responses must remain secret-free. They must not include database URLs, secret paths, worker IDs, task counts, public keys, credentials, raw errors, or submitted proof material. + +The response status is `not_ready` while an enabled worker is starting, stopping, or unexpectedly stopped. A transient deletion dependency failure reports `degraded` until successful dependency evidence. Ordinary pending deletion work, advisory-lock contention, and a correctly terminalized failed deletion job do not degrade readiness. diff --git a/docs/plans/2026-08-10-graceful-content-lock-deletion.md b/docs/plans/2026-08-10-graceful-content-lock-deletion.md index c0855e1..066035d 100644 --- a/docs/plans/2026-08-10-graceful-content-lock-deletion.md +++ b/docs/plans/2026-08-10-graceful-content-lock-deletion.md @@ -2,7 +2,7 @@ > **For Hermes:** Use subagent-driven-development to implement this plan one review-gated commit slice at a time. Stop after each slice; the user commits before the next slice. -**Goal:** Add a bounded `paykit-payment` deadline and creator-authorized graceful content-lock deletion that withdraws the public lock immediately, drains accepted payment and access obligations durably, removes guarded content, and safely permits later republication after complete graceful cleanup. +**Goal:** Add a bounded `paykit-payment` deadline and creator-authorized graceful content-lock deletion that withdraws the public lock immediately and drains accepted payment and access obligations durably. Graceful cleanup never deletes concurrent Pubky replacements, but tombstone publication has the explicitly accepted non-atomic overwrite limitation recorded below. **Architecture:** Locks owns the public tombstone, admission cutoff, verification tasks, credentials, guarded content, path ownership, and overall deletion job. Paykit Server owns invoice timestamps, Payment Request lifecycle classification, cancellation, Bitcoin observation, and a durable lock-wide payment drain. PostgreSQL stores retryable Locks workflow state; Pubky remains authoritative for public lock/tombstone and private guarded bytes. @@ -57,20 +57,22 @@ 23. Exact credential issuance replay returns the same random bearer. Persist a versioned encrypted envelope using a domain-separated key derived from the existing runtime master key; bind creator, Bundle ID, deletion job, and envelope version as AEAD context. 24. Deletion worker retries transient failures with durable exponential backoff: one second initial, five-minute cap, full jitter, ten attempts per phase by default. Attempts reset on phase advance. 25. Creator-visible job status is only `queued|running|completed|failed`; failed responses include a stable secret-free `failure_code` only. -26. Missing or replaced tombstone before destructive work halts as failed. Creator restores the exact tombstone and repeats graceful DELETE to resume the same job. +26. Missing or replaced tombstone before final verification halts as failed. Creator restores the exact tombstone and repeats graceful DELETE to resume the same job. 27. Guarded paths are exclusive to one managed lock. Enforce unique `(creator, guarded_path)` ownership in PostgreSQL. There is no historical backfill. 28. Lock publication uses best-effort ownership compensation and a durable opaque per-lock publication intent under the same PostgreSQL fence as deletion admission. Graceful/force deletion cannot start while publication is in flight. The intent is cleared only after ownership is durably published or failed publication is safely compensated. Process death can leave operator-reconciled intent/ownership state; do not claim cross-system atomicity. -29. Graceful final cleanup deletes guarded content first and tombstone last, purges Locks authorization/task/job state, asks Paykit to remove operational drain state, and releases path ownership. It forgets the deletion so the same canonical Lock ID may later be published fresh with new Bundle IDs. +29. Graceful finalization non-destructively verifies every frozen guarded-resource generation and the exact tombstone, then purges Locks authorization/task/job state and asks Paykit to remove operational drain state. The tombstone and guarded bytes remain on Pubky because its unconditional DELETE API cannot safely remove an expected generation in the presence of out-of-band creator writes. Later republication requires an explicit follow-up design rather than an unsafe graceful delete. 30. Paykit retains terminal financial invoice/payment history; delayed old lifecycle events cannot reactivate a fresh publication. 31. New force deletion is synchronous: persist a permanent minimal blocking receipt, delete lock/tombstone first, then best-effort guarded resources. Do not drain Paykit/tasks/credentials. Return failed paths. A force-deleted Lock ID can never be republished. 32. `force=true` against an active graceful job persists `force_requested`, revokes the current claim token/lease, requeues the same frozen job, and returns `202`; a fresh worker claim escalates asynchronously under exclusive action ownership, skips drains, deletes tombstone then content, and finishes forced. 33. Graceful job insertion/resume and permanent force-receipt establishment acquire the same canonical per-lock PostgreSQL fence. The durable result is either an active graceful job or a permanent force receipt, never both. Failed graceful replay requeues the same job and frozen manifest. Force against a terminal job atomically replaces that operational row with the permanent receipt before synchronous external deletion. -34. Any Content Lock fetched from Pubky for deletion must hash to the requested Lock ID and name the authenticated creator before its manifest is frozen or used for resource deletion. +34. Any Content Lock fetched from Pubky for deletion must hash to the requested Lock ID and name the authenticated creator before its manifest is frozen or used for resource verification or force deletion. 35. Runtime encryption uses one environment-only 32-byte unpadded-base64url master key selected by `secrets.runtime_master_key_env`. Creator-authority and final-credential encryption keys are derived from it with distinct fixed domain labels. The retired `creator_authority_key_env` key is rejected as unknown configuration; no compatibility alias is retained. 36. The closed `[deletion]` configuration contract is `retry_max_attempts = 10`, `retry_initial_backoff_seconds = 1`, `retry_max_backoff_seconds = 300`, `final_credential_issuance_window_seconds = 900`, and `final_read_window_seconds = 900` by default. All values are positive; initial backoff cannot exceed maximum backoff; both credential windows are bounded to at most 3600 seconds. Retry jitter remains an implementation policy rather than a configurable field. 37. Deletion admission immutably records whether each paid snapshot Bundle had any active credential at cutoff and enrolls every such ordinary credential with its original expiry. Enrolled ordinary credentials remain reusable against the frozen manifest until that expiry; they do not acquire one-shot resource-read rows. When the claimed job first enters `issue_final_credentials`, it persists `final_issuance_started_at`, `final_credential_issuance_deadline = final_issuance_started_at + final_credential_issuance_window`, and `final_read_deadline = final_credential_issuance_deadline + final_read_window` once; replay and later config changes never extend them. A paid snapshot resolved completed without an active ordinary credential at cutoff becomes durably final-credential eligible and receives exactly one encrypted replayable final credential expiring at `final_read_deadline`. Every final credential receives one claimable row per frozen manifest path. Final-read claims precede Pubky fetch, are released on pre-response failure, expire for crash recovery, and are consumed only after the complete HTTP response is constructed; consumption is permanent. Phase advancement waits until every enrolled ordinary credential is expired and every final resource is consumed or its credential/read window is expired. 38. Ordinary credential insertion and deletion admission acquire the same canonical per-lock fence. Deletion-first rejects the insert; insertion-first is attached and classified at cutoff. Database lock order is canonical per-lock fence, deletion job row, snapshot/credential row, then resource-read row. No transaction spans Pubky I/O. Final read claims use fixed 30-second leases clamped to credential expiry; stale claim tokens cannot consume or release a reclaimed row. 39. This pre-production migration intentionally has no creator-authority ciphertext compatibility path. Moving the same bytes to `runtime_master_key_env` changes the derived creator-authority key; existing local encrypted authority rows must be discarded and reacquired or the local database recreated. +40. Deletion-worker runtime configuration is a separate closed `[deletion_worker]` section with `enabled`, `poll_interval_ms`, `claim_timeout_seconds`, `shutdown_timeout_seconds`, and `worker_id`. Defaults are `true`, `250`, `60`, `30`, and `"deletion-worker"`. Enabled verification and deletion workers are independently tracked: starting, stopping, or unexpected exit is `not_ready`; a transient deletion dependency failure is `degraded` until successful dependency evidence; ordinary pending work, advisory-lock contention, and a correctly terminalized failed job do not degrade readiness. +41. **Accepted Pubky tombstone TOCTOU limitation:** the pinned Pubky 0.9.3 SDK exposes ETag metadata but no conditional write API, and the matching homeserver enforces `If-None-Match` for reads but not `If-Match` for writes. Graceful withdrawal therefore reads and compares the frozen canonical lock before an unconditional tombstone `PUT`. A replacement already visible at that read fails closed and is preserved; crash/reclaim replay also preserves any replacement it observes. However, an out-of-band creator replacement written after the comparison and before the `PUT` can be overwritten by the tombstone. Product explicitly accepts this race until Pubky provides atomic conditional writes. This exception does not authorize graceful deletion of replacement bytes; active force remains the only unconditional delete path. ### Source-derived constraints @@ -260,7 +262,7 @@ Reject `force=true&graceful=true`, unknown fields, malformed booleans, and dupli GET /creator/content-locks/{lock_id}/deletion ``` -Authenticated response contains Lock ID and `status`; include `failure_code` only for failed jobs. The closed stable vocabulary is exactly `tombstone_missing`, `tombstone_replaced`, `retry_exhausted`, and `state_corrupt`. Do not expose phases, leases, retries, Bundle IDs, readers, credentials, paths, Paykit IDs, or dependency errors. +Authenticated response contains Lock ID and `status`; include `failure_code` only for failed jobs. The closed stable vocabulary is exactly `tombstone_missing`, `tombstone_replaced`, `resource_replaced`, `retry_exhausted`, and `state_corrupt`. `resource_replaced` means a frozen guarded-resource path no longer contains the admitted generation and therefore graceful finalization failed closed without deleting the replacement. Do not expose phases, leases, retries, Bundle IDs, readers, credentials, paths, Paykit IDs, or dependency errors. If no job or force receipt exists, status returns `404 content_lock_deletion_not_found`. A permanent force receipt projects as `{ "lock_id": "...", "status": "completed" }` without exposing force mode. @@ -274,8 +276,8 @@ Internal phase names are not public API. The implementation should represent at 4. `drain_existing_credentials`: wait for credentials active at cutoff to expire. 5. `issue_final_credentials`: allow bounded issuance for eligible entitlements. 6. `drain_final_reads`: enforce per-path claims/consumption and read deadlines. -7. `delete_content`: idempotently delete every frozen resource while tombstone remains exact. -8. `delete_tombstone`: persist intent-to-remove phase before external delete so missing-on-retry is success. +7. `delete_content`: non-destructively verify every frozen resource generation while the tombstone remains exact. +8. `delete_tombstone`: non-destructively verify that the exact tombstone remains published before the purge handoff. 9. `purge_operational_state`: remove Paykit operational drain, then atomically purge Locks lock-scoped authorization/task/job state and release path ownership. Use separate durable `state`, `phase`, `attempt_count`, `next_attempt_at`, claim owner/token/expiry, and force-request fields. Use a per-job PostgreSQL advisory action lock where lease expiry must not permit overlapping external effects. SQLx advisory-lock connections must be close-on-drop and explicitly unlocked/closed. @@ -469,7 +471,7 @@ cargo test --workspace --no-run ### Task 9: Implement and supervise the deletion worker -**Objective:** Execute external phases retryably without overlapping destructive actions or breaking shutdown. +**Objective:** Execute external phases retryably without overlapping external actions or breaking shutdown. **Files:** - Create: `locks-server/src/deletion_worker.rs` @@ -479,17 +481,27 @@ cargo test --workspace --no-run - Modify: `locks-server/src/config/validation.rs` - Modify: `locks-server/src/app_state/readiness.rs` - Modify: `locks-server/src/api/runtime.rs` +- Modify: deletion application ports/use cases and PostgreSQL/Pubky/in-memory adapters required for exact tombstone I/O, per-job advisory action ownership, non-failure deferral, and worker materialization of final credentials - Test: worker unit tests and `locks-e2e/tests/postgres_runtime.rs` -**RED:** Crash/reclaim tests after every external side effect; advisory ownership exclusion; tombstone read-back/replacement failure; retry exhaustion/resume; force escalation; content-first/tombstone-last; missing tombstone allowed only after durable final-removal phase; readiness degradation; shutdown stops claims and bounds worker join. +**RED:** Crash/reclaim tests after every external side effect; PostgreSQL advisory ownership exclusion; exact tombstone publication/read-back and failure on replacements observed before publication or during replay; retry exhaustion/resume; force escalation; non-destructive sorted/deduplicated verification of every frozen guarded-resource generation followed by exact retained-tombstone verification; missing or replaced frozen resources and missing or replaced tombstones fail closed; active force deletes the canonical public path before best-effort private cleanup; readiness degradation; shutdown stops claims before HTTP drain and bounds the complete worker/HTTP join. These tests do not claim atomic replacement safety across the accepted read-to-unconditional-`PUT` window in decision 41. **GREEN:** Reuse existing worker configuration conventions but keep queue cadence and retry due time separate. Never log manifest, resource paths, Bundle IDs, credentials, readers, or Paykit payloads. +**Task 9 PostgreSQL crash/reclaim acceptance coverage map:** + +- Graceful public tombstone publication/read-back: `locks-e2e/tests/postgres_runtime.rs::postgres_graceful_withdraw_crash_reclaims_without_republishing_or_stale_advance` executes the production phase executor over the PostgreSQL job/lease repository, simulates process loss after publication, proves a fresh claim and advisory owner resume from exact read-back without a second publication, and fences the stale phase write. Exact missing/replaced byte classification remains covered by `locks-service/tests/content_lock_tombstones.rs` and the phase-executor failure tests. +- Payment-drain start and reconciliation: `payment_drain_reclaim_reconciles_external_start_before_local_persistence` covers remote start before local persistence and fresh-claim lookup reconciliation; `start_phase_replay_persists_monotonic_progress_after_crash_before_phase_advance`, `reclaim_first_fences_stale_initial_payment_drain_store`, `concurrent_force_winner_fences_stale_payment_drain_reconciliation`, and `concurrent_reclaim_winner_fences_stale_terminal_obligation_persistence` cover persisted aggregate replay plus stale start/reconcile/task writes against real PostgreSQL. +- Final credential generation/persistence/replay: `final_credentials_to_materialize_revalidates_exact_live_issue_claim_and_deadline` and `worker_final_issuance_is_exact_claim_fenced_in_winner_transaction` cover live-claim enumeration, fresh reclaimed ownership, stale/forced/deadline fencing, encrypted winner persistence, exact replay, and one-row cardinality against real PostgreSQL; `concurrent_final_issuers_replay_one_winner` independently covers concurrent winner replay. +- Frozen guarded-resource generation and retained-tombstone verification: `locks-e2e/tests/postgres_runtime.rs::postgres_guarded_generation_verification_crash_reclaims_and_replays_without_deletion` demonstrates two distinct loss points: after a frozen generation read but before its phase advance, and after exact retained-tombstone read-back but before the purge-handoff advance. Each boundary drops advisory ownership, recreates the PostgreSQL repository/runtime executor, reclaims with a fresh token, fences the stale advance, and idempotently replays without deleting private bytes. +- Active-force public-first/private cleanup: `locks-e2e/tests/postgres_runtime.rs::postgres_active_force_public_delete_crash_reclaims_before_private_cleanup` loses ownership immediately after canonical public deletion, before any private cleanup, then recreates the PostgreSQL repository/runtime executor, fences stale completion, replays public absence, performs private cleanup, and persists the terminal force receipt. `locks-e2e/tests/postgres_runtime.rs::postgres_active_force_private_delete_crash_reclaims_to_terminal_receipt` separately loses ownership after private deletion, then proves fresh-token reclaim, stale completion fencing, idempotent public/private replay, permanent receipt persistence, job removal, and no further claimable work. `execute_forced_content_lock_deletion::tests` supplies the sorted/deduplicated multi-resource and best-effort error matrix at the same production use-case seam. +- Every composed E2E case above drops the detached PostgreSQL advisory guard as the crash boundary and proves a later independent acquisition succeeds; `postgres_deletion_action_ownership_excludes_overlap_and_reacquires_after_release` separately proves overlap exclusion. + **Suggested commit:** `feat(server): run graceful deletion worker` ### Task 10: Purge graceful state and preserve force blocks -**Objective:** Complete graceful forget/republication without reactivating old authority, while permanently blocking force-deleted Lock IDs. +**Objective:** Purge graceful operational state without removing the durable external tombstone, while permanently blocking force-deleted Lock IDs. **Files:** - Create/modify: lock-scoped purge repository/use case in `locks-service/src/` @@ -560,7 +572,7 @@ Cross-service acceptance must additionally prove: - deletion crash recovery after every remote effect; - exact tombstone replacement halt/resume; - existing/final credential drain and concurrent per-path consumption; -- graceful same-ID republication with no old authorization revival; +- graceful tombstone preservation with no old authorization revival; - permanent force same-ID block. ## Remaining implementation-contract gates diff --git a/examples/js-sdk/README.md b/examples/js-sdk/README.md index 80fcfd7..6f083bf 100644 --- a/examples/js-sdk/README.md +++ b/examples/js-sdk/README.md @@ -211,6 +211,19 @@ npm --prefix examples/js-sdk run reset-paykit-demo Do not use `docker compose -f compose.paykit-local-demo.yaml down -v` unless you intentionally want to delete the persistent Lock Server identity volume. +### Migrating version 1 Compose secrets + +Compose secret schema version 1 stored only the retired creator-authority encryption key. It cannot be reinterpreted as the runtime master key because the current server derives domain-specific keys from that root. The bootstrap therefore fails closed rather than starting with a different key and leaving encrypted state unreadable. + +If startup reports version 1 secrets, perform this **destructive** reset from the repository root: + +```bash +docker compose -f compose.paykit-local-demo.yaml down --volumes --remove-orphans +rm -f .local/compose-secrets.json .local/locks-server/compose.env +``` + +Then run the normal Compose startup command again. This intentionally deletes the local Lock Server identity, Locks and Paykit databases, encrypted runtime state, and disposable Bitcoin/Fulcrum volumes. Do not use this migration reset when any of that local state must be retained; there is no safe automatic key migration for version 1 state. + ### Direct npm server ```bash diff --git a/examples/js-sdk/scripts/init-paykit-compose.mjs b/examples/js-sdk/scripts/init-paykit-compose.mjs index 615923e..8efb202 100644 --- a/examples/js-sdk/scripts/init-paykit-compose.mjs +++ b/examples/js-sdk/scripts/init-paykit-compose.mjs @@ -19,7 +19,7 @@ const SECRET_KEYS = [ 'paykitMasterKey', 'bitcoinRpcUser', 'bitcoinRpcPassword', - 'locksCreatorAuthKey', + 'locksRuntimeMasterKey', 'pubkyHomeserverAdminPassword', ]; const TOKEN = /^[A-Za-z0-9_-]{16,128}$/; @@ -31,25 +31,32 @@ function randomToken(bytes = 24) { export function createComposeSecrets() { return Object.freeze({ - version: 1, + version: 2, locksPostgresPassword: randomToken(), paykitPostgresPassword: randomToken(), paykitMasterKey: randomToken(32), bitcoinRpcUser: `bitcoin_${randomToken(12)}`, bitcoinRpcPassword: randomToken(32), - locksCreatorAuthKey: randomToken(32), + locksRuntimeMasterKey: randomToken(32), pubkyHomeserverAdminPassword: randomToken(24), }); } export function validateComposeSecrets(value) { + if (value !== null && typeof value === 'object' && !Array.isArray(value) && value.version === 1) { + const error = new Error( + 'persisted Compose secrets version 1 requires the destructive runtime-key reset documented in examples/js-sdk/README.md', + ); + error.code = 'INCOMPATIBLE_COMPOSE_SECRETS_VERSION'; + throw error; + } if ( value === null || typeof value !== 'object' || Array.isArray(value) || Object.keys(value).length !== SECRET_KEYS.length || !SECRET_KEYS.every((key) => Object.hasOwn(value, key)) - || value.version !== 1 + || value.version !== 2 ) { throw new Error('invalid persisted Compose secrets'); } @@ -64,7 +71,7 @@ export function validateComposeSecrets(value) { throw new Error('invalid persisted Compose secrets'); } } - for (const key of ['paykitMasterKey', 'locksCreatorAuthKey']) { + for (const key of ['paykitMasterKey', 'locksRuntimeMasterKey']) { if (typeof value[key] !== 'string' || !MASTER_KEY.test(value[key])) { throw new Error('invalid persisted Compose secrets'); } @@ -128,6 +135,7 @@ export async function initializePaykitCompose({ try { secrets = validateComposeSecrets(JSON.parse(await readPrivateText(paths.secrets))); } catch (error) { + if (error?.code === 'INCOMPATIBLE_COMPOSE_SECRETS_VERSION') throw error; if (error?.code !== 'ENOENT') throw new Error('persisted Compose secrets are invalid'); secrets = createComposeSecrets(); await writeSecure(paths.secrets, `${JSON.stringify(secrets, null, 2)}\n`); @@ -143,7 +151,7 @@ export async function initializePaykitCompose({ await Promise.all([ writeSecure(paths.locksPostgres, `POSTGRES_DB=locks_test\nPOSTGRES_USER=locks\nPOSTGRES_PASSWORD=${secrets.locksPostgresPassword}\n`), - writeSecure(paths.locksServer, `PUBKY_LOCK_DATABASE_URL=postgres://locks:${secrets.locksPostgresPassword}@postgres:5432/locks_test\nPUBKY_LOCK_CREATOR_AUTH_ENCRYPTION_KEY=${secrets.locksCreatorAuthKey}\n`), + writeSecure(paths.locksServer, `PUBKY_LOCK_DATABASE_URL=postgres://locks:${secrets.locksPostgresPassword}@postgres:5432/locks_test\nPUBKY_LOCK_RUNTIME_MASTER_KEY=${secrets.locksRuntimeMasterKey}\n`), writeSecure(paths.paykitPostgres, `POSTGRES_DB=paykit\nPOSTGRES_USER=paykit\nPOSTGRES_PASSWORD=${secrets.paykitPostgresPassword}\n`), writeSecure(paths.paykitServer, `PAYKIT_DATABASE_URL=${paykitDatabaseUrl}\nPAYKIT_MASTER_KEY=${secrets.paykitMasterKey}\n`), writeSecure(paths.bitcoinRpc, `BITCOIN_RPC_USER=${secrets.bitcoinRpcUser}\nBITCOIN_RPC_PASSWORD=${secrets.bitcoinRpcPassword}\nRPCUSER=${secrets.bitcoinRpcUser}\nRPCPASSWORD=${secrets.bitcoinRpcPassword}\n`), diff --git a/examples/js-sdk/scripts/smoke-paykit-compose.mjs b/examples/js-sdk/scripts/smoke-paykit-compose.mjs index 42ad8a0..cbf32a0 100644 --- a/examples/js-sdk/scripts/smoke-paykit-compose.mjs +++ b/examples/js-sdk/scripts/smoke-paykit-compose.mjs @@ -179,9 +179,31 @@ try { const paykitEnvironment = await readFile(join(generatedRoot, 'paykit-server/paykit.env'), 'utf8'); assert.match(paykitEnvironment, /^PAYKIT_DATABASE_URL=postgres:\/\/paykit:[A-Za-z0-9_-]+@paykit-postgres:5432\/paykit$/m); assert.match(paykitEnvironment, /^PAYKIT_MASTER_KEY=[A-Za-z0-9_-]{43}$/m); + const locksEnvironment = await readFile(join(generatedRoot, 'locks-server/compose.env'), 'utf8'); + assert.match(locksEnvironment, /^PUBKY_LOCK_RUNTIME_MASTER_KEY=[A-Za-z0-9_-]{43}$/m); + assert.doesNotMatch(locksEnvironment, /PUBKY_LOCK_CREATOR_AUTH_ENCRYPTION_KEY/); const firstSecrets = await readFile(join(generatedRoot, 'compose-secrets.json'), 'utf8'); + const parsedSecrets = JSON.parse(firstSecrets); + assert.equal(parsedSecrets.version, 2); + assert.match(parsedSecrets.locksRuntimeMasterKey, /^[A-Za-z0-9_-]{43}$/); + assert.equal(Object.hasOwn(parsedSecrets, 'locksCreatorAuthKey'), false); await initializePaykitCompose({ root: generatedRoot }); assert.equal(await readFile(join(generatedRoot, 'compose-secrets.json'), 'utf8'), firstSecrets); + + const versionOneSecrets = { ...parsedSecrets, version: 1 }; + versionOneSecrets.locksCreatorAuthKey = versionOneSecrets.locksRuntimeMasterKey; + delete versionOneSecrets.locksRuntimeMasterKey; + await writeFile( + join(generatedRoot, 'compose-secrets.json'), + `${JSON.stringify(versionOneSecrets)}\n`, + { mode: 0o600 }, + ); + await assert.rejects( + initializePaykitCompose({ root: generatedRoot }), + /version 1.*destructive runtime-key reset/, + ); + await writeFile(join(generatedRoot, 'compose-secrets.json'), firstSecrets, { mode: 0o600 }); + await chmod(join(generatedRoot, 'compose-secrets.json'), 0o644); await assert.rejects( initializePaykitCompose({ root: generatedRoot }), diff --git a/locks-e2e/tests/creator_publishing_http.rs b/locks-e2e/tests/creator_publishing_http.rs index 95a0130..6734a28 100644 --- a/locks-e2e/tests/creator_publishing_http.rs +++ b/locks-e2e/tests/creator_publishing_http.rs @@ -20,6 +20,7 @@ use locks_server::testing::TestServerApp; use locks_server::worker::{VerificationWorker, WorkerTick}; use locks_service::application::models::FrontendSessionToken; use locks_service::application::ports::CriterionVerifier; +use locks_service::infrastructure::memory::content_lock_tombstones::InMemoryContentLockTombstoneRepository; use locks_service::infrastructure::memory::content_locks::InMemoryContentLockRepository; use locks_service::infrastructure::memory::entitlements::InMemoryEntitlementRepository; use locks_service::infrastructure::memory::guarded_resources::InMemoryGuardedResourceRepository; @@ -41,6 +42,7 @@ async fn creator_publishing_http_flow_registers_locks_verifies_and_proxy_reads_g TestServerApp::from_state(AppState::new_empty_in_memory_with_creator_repositories( config, Arc::new(InMemoryContentLockRepository::new()), + Arc::new(InMemoryContentLockTombstoneRepository::new()), Arc::new(InMemoryGuardedResourceRepository::new()), Arc::new(InMemoryLockServicePointerRepository::new()), Arc::new(InMemoryEntitlementRepository::new()), @@ -335,6 +337,7 @@ async fn creator_publishing_http_rejects_invalid_guarded_path_before_lock_creati TestServerApp::from_state(AppState::new_empty_in_memory_with_creator_repositories( config, Arc::new(InMemoryContentLockRepository::new()), + Arc::new(InMemoryContentLockTombstoneRepository::new()), Arc::new(InMemoryGuardedResourceRepository::new()), Arc::new(InMemoryLockServicePointerRepository::new()), Arc::new(InMemoryEntitlementRepository::new()), @@ -486,6 +489,7 @@ async fn creator_publishing_http_paykit_payment_flow_creates_invoice_verifies_an let state = AppState::new_empty_in_memory_with_creator_repositories( config, Arc::new(InMemoryContentLockRepository::new()), + Arc::new(InMemoryContentLockTombstoneRepository::new()), Arc::new(InMemoryGuardedResourceRepository::new()), Arc::new(InMemoryLockServicePointerRepository::new()), Arc::new(InMemoryEntitlementRepository::new()), @@ -628,6 +632,7 @@ async fn creator_publishing_client() -> (TestServerApp, LocalCreatorPublishingCl TestServerApp::from_state(AppState::new_empty_in_memory_with_creator_repositories( config, Arc::new(InMemoryContentLockRepository::new()), + Arc::new(InMemoryContentLockTombstoneRepository::new()), Arc::new(InMemoryGuardedResourceRepository::new()), Arc::new(InMemoryLockServicePointerRepository::new()), Arc::new(InMemoryEntitlementRepository::new()), diff --git a/locks-e2e/tests/postgres_runtime.rs b/locks-e2e/tests/postgres_runtime.rs index c03a8be..da3251f 100644 --- a/locks-e2e/tests/postgres_runtime.rs +++ b/locks-e2e/tests/postgres_runtime.rs @@ -1,16 +1,19 @@ +use std::collections::HashMap; use std::net::{IpAddr, Ipv4Addr, SocketAddr}; use std::path::PathBuf; use std::str::FromStr; -use std::sync::Arc; use std::sync::atomic::{AtomicUsize, Ordering}; +use std::sync::{Arc, Mutex}; use async_trait::async_trait; use axum::body::{Body, to_bytes}; use axum::extract::ConnectInfo; use axum::http::{Request, StatusCode, header}; use axum::routing::post; +use locks_core::content_lock_deletion::ContentLockDeletionTombstone; use locks_core::ids::{ - BundleId, CreatorPubky, GuardedResourceHash, LockServerPubky, PubkyLockResource, TaskId, + BundleId, ContentLockPath, CreatorPubky, GuardedResourceHash, LockServerPubky, + PubkyLockResource, TaskId, }; use locks_core::lock_policy::{ AccessPolicy, CONTENT_LOCK_VERSION, ContentLock, Criterion, GuardedResource, LockLogic, @@ -25,33 +28,634 @@ use locks_server::config::{ LockServerRuntimeConfig, LoggingConfig, PaykitConfig, PkdnsConfig, PubkyConfig, RateLimitsConfig, RuntimeConfig, RuntimeEnvironment, SecretsConfig, WorkerConfig, }; +use locks_server::deletion_worker::{ClaimedDeletionExecutor, RuntimeClaimedDeletionExecutor}; +use locks_server::testing::TestServerApp; use locks_server::worker::{VerificationWorker, WorkerTick}; use locks_service::application::models::{ - AccessCredential, AccessCredentialLookupKey, ContentLockDeletionJob, + AccessCredential, AccessCredentialLookupKey, ClaimedContentLockDeletionJob, + ContentLockDeletionJob, ContentLockDeletionPhase, ContentLockDeletionState, ContentLockOwnershipStatus, CreatorAuthorityAuthKind, CreatorAuthorityRecord, - CreatorAuthoritySecret, VerificationTaskRecord, VerificationTaskStatus, + CreatorAuthoritySecret, FrontendSessionToken, GuardedResourceRecord, + PrepareForceDeletionResult, VerificationTaskRecord, VerificationTaskStatus, }; use locks_service::application::ports::{ - ContentLockDeletionRepository, VerificationTaskRepository, + ContentLockDeletionActionAcquireResult, ContentLockDeletionActionClaim, + ContentLockDeletionActionGuard, ContentLockDeletionActionOwnership, + ContentLockDeletionRepository, ContentLockTombstoneRepository, GuardedResourceReadback, + GuardedResourceRepository, TombstoneReadback, VerificationTaskRepository, }; use locks_service::infrastructure::final_credentials::FinalCredentialCipher; use locks_service::infrastructure::memory::{ + content_lock_tombstones::InMemoryContentLockTombstoneRepository, content_locks::InMemoryContentLockRepository, entitlements::InMemoryEntitlementRepository, guarded_resources::InMemoryGuardedResourceRepository, lock_service_pointers::InMemoryLockServicePointerRepository, }; use locks_service::infrastructure::postgres::{ - CreatorAuthoritySecretCipher, PostgresContentLockDeletionRepository, - PostgresVerificationTaskRepository, run_migrations, + CreatorAuthoritySecretCipher, PostgresContentLockDeletionActionOwnership, + PostgresContentLockDeletionRepository, PostgresVerificationTaskRepository, run_migrations, }; use serde_json::{Value, json}; use sqlx::postgres::PgPoolOptions; use sqlx::{Connection, Executor, PgConnection, PgPool}; -use time::macros::datetime; +use time::{OffsetDateTime, macros::datetime}; use tower::ServiceExt; const BUNDLE_ID: &str = "000G40R40M30E209185GR38E1W"; +#[tokio::test] +async fn postgres_deletion_action_ownership_excludes_overlap_and_reacquires_after_release() { + let Some(database) = TestDatabase::create().await else { + return; + }; + let first_owner = PostgresContentLockDeletionActionOwnership::new(database.pool().clone()); + let second_owner = PostgresContentLockDeletionActionOwnership::new(database.pool().clone()); + let deletions = PostgresContentLockDeletionRepository::new(database.pool().clone()); + let job = ContentLockDeletionJob::new( + uuid::Uuid::new_v4(), + content_lock(), + OffsetDateTime::now_utc(), + ) + .unwrap(); + deletions.insert_job(job).await.unwrap(); + let claimed = deletions + .claim_next("ownership-worker", time::Duration::minutes(1)) + .await + .unwrap() + .unwrap(); + + let first_guard = expect_action_acquired( + first_owner + .try_acquire(deletion_action_claim(&claimed, "ownership-worker", false)) + .await + .unwrap(), + ); + assert!(matches!( + second_owner + .try_acquire(deletion_action_claim(&claimed, "ownership-worker", false,)) + .await + .unwrap(), + ContentLockDeletionActionAcquireResult::Busy + )); + + first_guard.release().await.unwrap(); + let replacement_guard = expect_action_acquired( + second_owner + .try_acquire(deletion_action_claim(&claimed, "ownership-worker", false)) + .await + .unwrap(), + ); + replacement_guard.release().await.unwrap(); + + database.cleanup().await; +} + +#[tokio::test] +async fn postgres_missed_final_issuance_terminalizes_with_closed_creator_failure_without_external_action() + { + let Some(database) = TestDatabase::create().await else { + return; + }; + let lock = content_lock(); + let external = Arc::new(CrashExternalRepository::with_tombstone_and_resources(&lock)); + let state = deletion_app_state(database.pool().clone(), Arc::clone(&external)); + let tasks = PostgresVerificationTaskRepository::new(database.pool().clone()); + let now = OffsetDateTime::now_utc(); + let mut task = VerificationTaskRecord { + task_id: TaskId::from_str(&uuid::Uuid::new_v4().to_string()).unwrap(), + creator: lock.creator.clone(), + submitted_proof_bundle: submitted_proof_bundle_for(&lock), + status: VerificationTaskStatus::Completed, + submitted_at: now - time::Duration::hours(1), + started_at: Some(now - time::Duration::hours(1)), + completed_at: Some(now - time::Duration::minutes(30)), + failure_message: None, + }; + task.submitted_proof_bundle.proofs[0].verifier_type = VerifierType::PaykitPayment; + tasks.insert_verification_task(task).await.unwrap(); + + let job = ContentLockDeletionJob::new(uuid::Uuid::new_v4(), lock, now).unwrap(); + let deletions = PostgresContentLockDeletionRepository::new(database.pool().clone()); + deletions.insert_job(job.clone()).await.unwrap(); + sqlx::query( + "UPDATE content_lock_deletion_task_snapshot + SET resolved_status = 'completed', resolved_at = $2, + final_credential_eligible_at = $2 + WHERE deletion_job_id = $1", + ) + .bind(job.job_id) + .bind(now - time::Duration::minutes(30)) + .execute(database.pool()) + .await + .unwrap(); + sqlx::query( + "UPDATE content_lock_deletion_jobs + SET phase = 'issue_final_credentials', final_issuance_started_at = $2, + final_credential_issuance_deadline = $3, final_read_deadline = $4 + WHERE job_id = $1", + ) + .bind(job.job_id) + .bind(now - time::Duration::minutes(20)) + .bind(now - time::Duration::minutes(5)) + .bind(now + time::Duration::minutes(10)) + .execute(database.pool()) + .await + .unwrap(); + let claim = deletions + .claim_next( + "missed-issuance-worker", + (now + time::Duration::minutes(5)) - (now), + ) + .await + .unwrap() + .unwrap(); + + assert_eq!( + RuntimeClaimedDeletionExecutor::new(state.clone()) + .execute_claimed(claim, "missed-issuance-worker") + .await + .outcome, + locks_service::application::use_cases::execute_content_lock_deletion_phase::DeletionPhaseExecutionOutcome::TerminalFailed + ); + let failed = deletions + .get_job(&job.creator, &job.lock_id) + .await + .unwrap() + .unwrap(); + assert_eq!(failed.state, ContentLockDeletionState::Failed); + assert_eq!( + failed.failure_code.map(|code| code.as_str()), + Some("state_corrupt") + ); + assert!(external.operations().is_empty()); + assert_eq!(external.resource_read_count(), 0); + assert_eq!(external.resource_delete_count(), 0); + + let app = TestServerApp::from_state(state); + let token = "missed-issuance-creator-session"; + app.insert_frontend_session_for_test( + FrontendSessionToken::new(token), + job.creator.clone(), + now + time::Duration::hours(1), + ) + .await + .unwrap(); + let response = app + .router() + .oneshot( + Request::builder() + .method("GET") + .uri(format!("/creator/content-locks/{}/deletion", job.lock_id)) + .header(header::AUTHORIZATION, format!("Bearer {token}")) + .body(Body::empty()) + .unwrap(), + ) + .await + .unwrap(); + assert_eq!(response.status(), StatusCode::OK); + assert_eq!( + response_json(response).await, + json!({ + "lock_id": job.lock_id, + "status": "failed", + "failure_code": "state_corrupt" + }) + ); + + database.cleanup().await; +} + +#[tokio::test] +async fn postgres_graceful_withdraw_crash_reclaims_without_republishing_or_stale_advance() { + let Some(database) = TestDatabase::create().await else { + return; + }; + let external = Arc::new(CrashExternalRepository::with_original(content_lock())); + let deletions = PostgresContentLockDeletionRepository::new(database.pool().clone()); + let ownership = PostgresContentLockDeletionActionOwnership::new(database.pool().clone()); + let now = OffsetDateTime::now_utc(); + let job = ContentLockDeletionJob::new(uuid::Uuid::new_v4(), content_lock(), now).unwrap(); + deletions.insert_job(job.clone()).await.unwrap(); + + let (stale, crash_guard) = + claim_deletion_action(&deletions, &ownership, "withdraw-crashed", false).await; + let tombstone = ContentLockDeletionTombstone::new(job.lock_id.clone(), now); + assert_eq!( + external + .withdraw_content_lock( + job.creator.clone(), + ContentLockPath::from_lock_id(job.lock_id.clone()), + &job.frozen_content_lock, + &tombstone, + ) + .await + .unwrap(), + TombstoneReadback::Exact + ); + drop(crash_guard); + + let fresh = expire_and_reclaim_deletion( + database.pool(), + &deletions, + job.job_id, + "withdraw-reclaimed", + ) + .await; + assert_ne!(fresh.claim_token, stale.claim_token); + assert!(matches!( + deletions + .advance_phase( + job.job_id, + "withdraw-crashed", + stale.claim_token, + ContentLockDeletionPhase::StartPaymentDrain, + ) + .await + .unwrap(), + locks_service::application::models::AdvanceContentLockDeletionPhaseResult::ClaimLost + )); + + let recreated = deletion_app_state(database.pool().clone(), Arc::clone(&external)); + assert_eq!( + RuntimeClaimedDeletionExecutor::new(recreated) + .execute_claimed(fresh, "withdraw-reclaimed") + .await + .outcome, + locks_service::application::use_cases::execute_content_lock_deletion_phase::DeletionPhaseExecutionOutcome::Progressed + ); + assert_eq!(external.tombstone_write_count(), 1); + assert_eq!(external.withdraw_call_count(), 2); + assert_eq!( + deletions + .get_job(&job.creator, &job.lock_id) + .await + .unwrap() + .unwrap() + .phase, + ContentLockDeletionPhase::StartPaymentDrain + ); + let release_check = deletions + .claim_next("withdraw-release-check", time::Duration::minutes(1)) + .await + .unwrap() + .unwrap(); + expect_action_acquired( + ownership + .try_acquire(deletion_action_claim( + &release_check, + "withdraw-release-check", + false, + )) + .await + .unwrap(), + ) + .release() + .await + .unwrap(); + + database.cleanup().await; +} + +#[tokio::test] +async fn postgres_guarded_generation_verification_crash_reclaims_and_replays_without_deletion() { + let Some(database) = TestDatabase::create().await else { + return; + }; + let lock = content_lock(); + let external = Arc::new(CrashExternalRepository::with_tombstone_and_resources(&lock)); + let state = deletion_app_state(database.pool().clone(), Arc::clone(&external)); + let deletions = PostgresContentLockDeletionRepository::new(database.pool().clone()); + let ownership = PostgresContentLockDeletionActionOwnership::new(database.pool().clone()); + let now = OffsetDateTime::now_utc(); + let job = ContentLockDeletionJob::new(uuid::Uuid::new_v4(), lock.clone(), now).unwrap(); + deletions.insert_job(job.clone()).await.unwrap(); + set_deletion_phase(database.pool(), job.job_id, "delete_content").await; + + let (stale, crash_guard) = + claim_deletion_action(&deletions, &ownership, "verify-crashed", false).await; + let primary = lock.primary_resource.as_ref().unwrap(); + assert_eq!( + external + .read_guarded_resource_generation(&job.creator, &primary.path, &primary.hash) + .await + .unwrap(), + GuardedResourceReadback::Exact + ); + drop(crash_guard); + + let fresh = + expire_and_reclaim_deletion(database.pool(), &deletions, job.job_id, "verify-reclaimed") + .await; + assert!(matches!( + deletions + .advance_phase( + job.job_id, + "verify-crashed", + stale.claim_token, + ContentLockDeletionPhase::DeleteTombstone, + ) + .await + .unwrap(), + locks_service::application::models::AdvanceContentLockDeletionPhaseResult::ClaimLost + )); + assert_eq!( + RuntimeClaimedDeletionExecutor::new(state) + .execute_claimed(fresh, "verify-reclaimed") + .await + .outcome, + locks_service::application::use_cases::execute_content_lock_deletion_phase::DeletionPhaseExecutionOutcome::Progressed + ); + assert_eq!(external.resource_delete_count(), 0); + assert_eq!(external.resource_read_count(), 2); + assert_eq!( + deletions + .get_job(&job.creator, &job.lock_id) + .await + .unwrap() + .unwrap() + .phase, + ContentLockDeletionPhase::DeleteTombstone + ); + + let tombstone_crashed = deletions + .claim_next("tombstone-crashed", time::Duration::seconds(1)) + .await + .unwrap() + .unwrap(); + let tombstone_crash_guard = expect_action_acquired( + ownership + .try_acquire(deletion_action_claim( + &tombstone_crashed, + "tombstone-crashed", + false, + )) + .await + .unwrap(), + ); + let tombstone = ContentLockDeletionTombstone::new(job.lock_id.clone(), job.deletion_started_at); + assert_eq!( + external + .read_tombstone( + &job.creator, + &ContentLockPath::from_lock_id(job.lock_id.clone()), + &tombstone, + ) + .await + .unwrap(), + TombstoneReadback::Exact + ); + drop(tombstone_crash_guard); + + expire_deletion_claim(database.pool(), job.job_id).await; + let recreated_deletions = PostgresContentLockDeletionRepository::new(database.pool().clone()); + let retained_tombstone_claim = recreated_deletions + .claim_next("tombstone-reclaimed", time::Duration::minutes(1)) + .await + .unwrap() + .unwrap(); + assert_ne!( + retained_tombstone_claim.claim_token, + tombstone_crashed.claim_token + ); + assert!(matches!( + recreated_deletions + .advance_phase( + job.job_id, + "tombstone-crashed", + tombstone_crashed.claim_token, + ContentLockDeletionPhase::PurgeOperationalState, + ) + .await + .unwrap(), + locks_service::application::models::AdvanceContentLockDeletionPhaseResult::ClaimLost + )); + let recreated = deletion_app_state(database.pool().clone(), Arc::clone(&external)); + assert_eq!( + RuntimeClaimedDeletionExecutor::new(recreated) + .execute_claimed(retained_tombstone_claim, "tombstone-reclaimed") + .await + .outcome, + locks_service::application::use_cases::execute_content_lock_deletion_phase::DeletionPhaseExecutionOutcome::Progressed + ); + assert_eq!(external.tombstone_read_count(), 4); + assert_eq!(external.resource_delete_count(), 0); + assert_eq!( + recreated_deletions + .get_job(&job.creator, &job.lock_id) + .await + .unwrap() + .unwrap() + .phase, + ContentLockDeletionPhase::PurgeOperationalState + ); + let release_check = recreated_deletions + .claim_next("tombstone-release-check", time::Duration::minutes(1)) + .await + .unwrap() + .unwrap(); + expect_action_acquired( + ownership + .try_acquire(deletion_action_claim( + &release_check, + "tombstone-release-check", + false, + )) + .await + .unwrap(), + ) + .release() + .await + .unwrap(); + + database.cleanup().await; +} + +#[tokio::test] +async fn postgres_active_force_public_delete_crash_reclaims_before_private_cleanup() { + let Some(database) = TestDatabase::create().await else { + return; + }; + let lock = content_lock(); + let external = Arc::new(CrashExternalRepository::with_original_and_resources(&lock)); + let deletions = PostgresContentLockDeletionRepository::new(database.pool().clone()); + let ownership = PostgresContentLockDeletionActionOwnership::new(database.pool().clone()); + let now = OffsetDateTime::now_utc(); + let job = ContentLockDeletionJob::new(uuid::Uuid::new_v4(), lock, now).unwrap(); + deletions.insert_job(job.clone()).await.unwrap(); + assert!(matches!( + deletions + .prepare_force_deletion(&job.creator, &job.lock_id) + .await + .unwrap(), + PrepareForceDeletionResult::Active(_) + )); + + let (stale, crash_guard) = + claim_deletion_action(&deletions, &ownership, "public-crashed", true).await; + external + .force_delete_content_lock_and_verify_absent( + &job.creator, + &ContentLockPath::from_lock_id(job.lock_id.clone()), + ) + .await + .unwrap(); + assert_eq!(external.operations(), vec!["public"]); + drop(crash_guard); + + expire_deletion_claim(database.pool(), job.job_id).await; + let recreated_deletions = PostgresContentLockDeletionRepository::new(database.pool().clone()); + let fresh = recreated_deletions + .claim_next("public-reclaimed", time::Duration::minutes(1)) + .await + .unwrap() + .unwrap(); + let fresh_action_claim = fresh.clone(); + assert_ne!(fresh.claim_token, stale.claim_token); + assert!( + !deletions + .complete_force_deletion(job.job_id, "public-crashed", stale.claim_token,) + .await + .unwrap() + ); + + let recreated = deletion_app_state(database.pool().clone(), Arc::clone(&external)); + assert_eq!( + RuntimeClaimedDeletionExecutor::new(recreated) + .execute_claimed(fresh, "public-reclaimed") + .await + .outcome, + locks_service::application::use_cases::execute_content_lock_deletion_phase::DeletionPhaseExecutionOutcome::Progressed + ); + assert!( + recreated_deletions + .has_force_receipt(&job.creator, &job.lock_id) + .await + .unwrap() + ); + assert!( + recreated_deletions + .get_job(&job.creator, &job.lock_id) + .await + .unwrap() + .is_none() + ); + assert_eq!(external.operations(), vec!["public", "public", "private"]); + assert_eq!(external.resource_delete_count(), 1); + assert!(matches!( + ownership + .try_acquire(deletion_action_claim( + &fresh_action_claim, + "public-reclaimed", + true, + )) + .await + .unwrap(), + ContentLockDeletionActionAcquireResult::ClaimLost + )); + + database.cleanup().await; +} + +#[tokio::test] +async fn postgres_active_force_private_delete_crash_reclaims_to_terminal_receipt() { + let Some(database) = TestDatabase::create().await else { + return; + }; + let lock = content_lock(); + let external = Arc::new(CrashExternalRepository::with_original_and_resources(&lock)); + let deletions = PostgresContentLockDeletionRepository::new(database.pool().clone()); + let ownership = PostgresContentLockDeletionActionOwnership::new(database.pool().clone()); + let now = OffsetDateTime::now_utc(); + let job = ContentLockDeletionJob::new(uuid::Uuid::new_v4(), lock.clone(), now).unwrap(); + deletions.insert_job(job.clone()).await.unwrap(); + assert!(matches!( + deletions + .prepare_force_deletion(&job.creator, &job.lock_id) + .await + .unwrap(), + PrepareForceDeletionResult::Active(_) + )); + + let (stale, crash_guard) = + claim_deletion_action(&deletions, &ownership, "private-crashed", true).await; + external + .force_delete_content_lock_and_verify_absent( + &job.creator, + &ContentLockPath::from_lock_id(job.lock_id.clone()), + ) + .await + .unwrap(); + external + .delete_guarded_resource(&job.creator, &lock.primary_resource.as_ref().unwrap().path) + .await + .unwrap(); + assert_eq!(external.operations(), vec!["public", "private"]); + drop(crash_guard); + + expire_deletion_claim(database.pool(), job.job_id).await; + let recreated_deletions = PostgresContentLockDeletionRepository::new(database.pool().clone()); + let fresh = recreated_deletions + .claim_next("private-reclaimed", time::Duration::minutes(1)) + .await + .unwrap() + .unwrap(); + let fresh_action_claim = fresh.clone(); + assert_ne!(fresh.claim_token, stale.claim_token); + assert!( + !deletions + .complete_force_deletion(job.job_id, "private-crashed", stale.claim_token,) + .await + .unwrap() + ); + + let recreated = deletion_app_state(database.pool().clone(), Arc::clone(&external)); + assert_eq!( + RuntimeClaimedDeletionExecutor::new(recreated) + .execute_claimed(fresh, "private-reclaimed") + .await + .outcome, + locks_service::application::use_cases::execute_content_lock_deletion_phase::DeletionPhaseExecutionOutcome::Progressed + ); + assert!( + recreated_deletions + .has_force_receipt(&job.creator, &job.lock_id) + .await + .unwrap() + ); + assert!( + recreated_deletions + .get_job(&job.creator, &job.lock_id) + .await + .unwrap() + .is_none() + ); + assert!( + recreated_deletions + .claim_next("after-terminal", time::Duration::minutes(1)) + .await + .unwrap() + .is_none() + ); + assert_eq!( + external.operations(), + vec!["public", "private", "public", "private"] + ); + assert_eq!(external.resource_delete_count(), 2); + assert!(matches!( + ownership + .try_acquire(deletion_action_claim( + &fresh_action_claim, + "private-reclaimed", + true, + )) + .await + .unwrap(), + ContentLockDeletionActionAcquireResult::ClaimLost + )); + + database.cleanup().await; +} + #[tokio::test] async fn postgres_runtime_state_survives_app_state_recreation() { let Some(database) = TestDatabase::create().await else { @@ -176,6 +780,14 @@ async fn postgres_runtime_readyz_returns_ready_without_leaking_runtime_details() return; }; let state = app_state(database.pool().clone()); + state.record_worker_readiness( + locks_server::app_state::WorkerKind::Verification, + locks_server::app_state::WorkerReadinessEvidence::Ready, + ); + state.record_worker_readiness( + locks_server::app_state::WorkerKind::Deletion, + locks_server::app_state::WorkerReadinessEvidence::Ready, + ); let response = router(state) .oneshot(empty_request("GET", "/readyz")) .await @@ -282,6 +894,7 @@ async fn deletion_first_proof_submission_returns_409_without_calling_paykit() { FinalCredentialCipher::new([8; 32]), ), Arc::new(InMemoryContentLockRepository::new()), + Arc::new(InMemoryContentLockTombstoneRepository::new()), Arc::new(InMemoryGuardedResourceRepository::new()), Arc::new(InMemoryLockServicePointerRepository::new()), Arc::new(InMemoryEntitlementRepository::new()), @@ -377,6 +990,7 @@ async fn snapshotted_unready_paykit_replay_ignores_tombstoned_lock_and_reader_re FinalCredentialCipher::new([8; 32]), ), Arc::new(InMemoryContentLockRepository::new()), + Arc::new(InMemoryContentLockTombstoneRepository::new()), Arc::new(InMemoryGuardedResourceRepository::new()), Arc::new(InMemoryLockServicePointerRepository::new()), Arc::new(InMemoryEntitlementRepository::new()), @@ -415,6 +1029,7 @@ async fn snapshotted_unready_paykit_replay_ignores_tombstoned_lock_and_reader_re FinalCredentialCipher::new([8; 32]), ), Arc::new(InMemoryContentLockRepository::new()), + Arc::new(InMemoryContentLockTombstoneRepository::new()), Arc::new(InMemoryGuardedResourceRepository::new()), Arc::new(InMemoryLockServicePointerRepository::new()), Arc::new(InMemoryEntitlementRepository::new()), @@ -465,6 +1080,287 @@ async fn snapshotted_unready_paykit_replay_ignores_tombstoned_lock_and_reader_re database.cleanup().await; } +#[derive(Debug)] +struct CrashExternalRepository { + // 0 = frozen original, 1 = exact tombstone, 2 = absent, 3 = replacement. + public_state: AtomicUsize, + tombstone_writes: AtomicUsize, + withdraw_calls: AtomicUsize, + tombstone_reads: AtomicUsize, + resource_reads: AtomicUsize, + resource_deletes: AtomicUsize, + resources: Mutex>, + operations: Mutex>, +} + +impl CrashExternalRepository { + fn with_original(_lock: ContentLock) -> Self { + Self::new(0, None) + } + + fn with_tombstone_and_resources(lock: &ContentLock) -> Self { + Self::new(1, Some(lock)) + } + + fn with_original_and_resources(lock: &ContentLock) -> Self { + Self::new(0, Some(lock)) + } + + fn new(public_state: usize, lock: Option<&ContentLock>) -> Self { + let mut resources = HashMap::new(); + if let Some(lock) = lock + && let Some(primary) = &lock.primary_resource + { + resources.insert( + primary.path.clone(), + GuardedResourceRecord { + creator: lock.creator.clone(), + path: primary.path.clone(), + hash: primary.hash, + content_type: primary.content_type.clone(), + size: primary.size, + bytes: vec![7; primary.size as usize], + }, + ); + } + Self { + public_state: AtomicUsize::new(public_state), + tombstone_writes: AtomicUsize::new(0), + withdraw_calls: AtomicUsize::new(0), + tombstone_reads: AtomicUsize::new(0), + resource_reads: AtomicUsize::new(0), + resource_deletes: AtomicUsize::new(0), + resources: Mutex::new(resources), + operations: Mutex::new(Vec::new()), + } + } + + fn tombstone_write_count(&self) -> usize { + self.tombstone_writes.load(Ordering::SeqCst) + } + + fn withdraw_call_count(&self) -> usize { + self.withdraw_calls.load(Ordering::SeqCst) + } + + fn tombstone_read_count(&self) -> usize { + self.tombstone_reads.load(Ordering::SeqCst) + } + + fn resource_read_count(&self) -> usize { + self.resource_reads.load(Ordering::SeqCst) + } + + fn resource_delete_count(&self) -> usize { + self.resource_deletes.load(Ordering::SeqCst) + } + + fn operations(&self) -> Vec<&'static str> { + self.operations.lock().unwrap().clone() + } + + fn tombstone_readback(&self) -> TombstoneReadback { + match self.public_state.load(Ordering::SeqCst) { + 1 => TombstoneReadback::Exact, + 2 => TombstoneReadback::Missing, + _ => TombstoneReadback::Replaced, + } + } +} + +#[async_trait] +impl ContentLockTombstoneRepository for CrashExternalRepository { + async fn withdraw_content_lock( + &self, + _creator: CreatorPubky, + _content_lock_path: ContentLockPath, + _frozen_original: &ContentLock, + _tombstone: &ContentLockDeletionTombstone, + ) -> Result { + self.withdraw_calls.fetch_add(1, Ordering::SeqCst); + if self + .public_state + .compare_exchange(0, 1, Ordering::SeqCst, Ordering::SeqCst) + .is_ok() + { + self.tombstone_writes.fetch_add(1, Ordering::SeqCst); + } + Ok(self.tombstone_readback()) + } + + async fn read_tombstone( + &self, + _creator: &CreatorPubky, + _content_lock_path: &ContentLockPath, + _expected: &ContentLockDeletionTombstone, + ) -> Result { + self.tombstone_reads.fetch_add(1, Ordering::SeqCst); + Ok(self.tombstone_readback()) + } + + async fn force_delete_content_lock_and_verify_absent( + &self, + _creator: &CreatorPubky, + _content_lock_path: &ContentLockPath, + ) -> Result<(), locks_service::application::errors::ApplicationError> { + self.operations.lock().unwrap().push("public"); + self.public_state.store(2, Ordering::SeqCst); + Ok(()) + } +} + +#[async_trait] +impl GuardedResourceRepository for CrashExternalRepository { + async fn upsert_guarded_resource( + &self, + resource: GuardedResourceRecord, + ) -> Result<(), locks_service::application::errors::ApplicationError> { + self.resources + .lock() + .unwrap() + .insert(resource.path.clone(), resource); + Ok(()) + } + + async fn get_guarded_resource( + &self, + _creator: &CreatorPubky, + path: &str, + hash: &GuardedResourceHash, + ) -> Result, locks_service::application::errors::ApplicationError> + { + Ok(self + .resources + .lock() + .unwrap() + .get(path) + .filter(|record| record.hash == *hash) + .cloned()) + } + + async fn get_current_guarded_resource( + &self, + _creator: &CreatorPubky, + path: &str, + ) -> Result, locks_service::application::errors::ApplicationError> + { + self.resource_reads.fetch_add(1, Ordering::SeqCst); + Ok(self.resources.lock().unwrap().get(path).cloned()) + } + + async fn delete_guarded_resource( + &self, + _creator: &CreatorPubky, + path: &str, + ) -> Result { + self.operations.lock().unwrap().push("private"); + self.resource_deletes.fetch_add(1, Ordering::SeqCst); + Ok(self.resources.lock().unwrap().remove(path).is_some()) + } +} + +fn deletion_action_claim<'a>( + claimed: &ClaimedContentLockDeletionJob, + worker_id: &'a str, + force: bool, +) -> ContentLockDeletionActionClaim<'a> { + ContentLockDeletionActionClaim { + job_id: claimed.job.job_id, + worker_id, + claim_token: claimed.claim_token, + expected_phase: claimed.job.phase, + force, + } +} + +fn expect_action_acquired( + result: ContentLockDeletionActionAcquireResult, +) -> Box { + match result { + ContentLockDeletionActionAcquireResult::Acquired(guard) => guard, + ContentLockDeletionActionAcquireResult::Busy => { + panic!("live deletion action claim was unexpectedly contended") + } + ContentLockDeletionActionAcquireResult::ClaimLost => { + panic!("live deletion action claim was unexpectedly lost") + } + } +} + +async fn claim_deletion_action( + deletions: &PostgresContentLockDeletionRepository, + ownership: &PostgresContentLockDeletionActionOwnership, + worker_id: &str, + force: bool, +) -> ( + ClaimedContentLockDeletionJob, + Box, +) { + let claimed = deletions + .claim_next(worker_id, time::Duration::seconds(1)) + .await + .unwrap() + .unwrap(); + let guard = expect_action_acquired( + ownership + .try_acquire(deletion_action_claim(&claimed, worker_id, force)) + .await + .unwrap(), + ); + (claimed, guard) +} + +async fn set_deletion_phase(pool: &PgPool, job_id: uuid::Uuid, phase: &str) { + sqlx::query("UPDATE content_lock_deletion_jobs SET phase = $2 WHERE job_id = $1") + .bind(job_id) + .bind(phase) + .execute(pool) + .await + .unwrap(); +} + +async fn expire_deletion_claim(pool: &PgPool, job_id: uuid::Uuid) { + sqlx::query( + "UPDATE content_lock_deletion_jobs + SET claim_expires_at = clock_timestamp() - interval '1 second' + WHERE job_id = $1", + ) + .bind(job_id) + .execute(pool) + .await + .unwrap(); +} + +async fn expire_and_reclaim_deletion( + pool: &PgPool, + deletions: &PostgresContentLockDeletionRepository, + job_id: uuid::Uuid, + worker_id: &str, +) -> ClaimedContentLockDeletionJob { + expire_deletion_claim(pool, job_id).await; + deletions + .claim_next(worker_id, time::Duration::minutes(1)) + .await + .unwrap() + .unwrap() +} + +fn deletion_app_state(pool: PgPool, external: Arc) -> AppState { + AppState::new_with_postgres_runtime_and_creator_repositories( + test_config(), + pool, + RuntimeSecretCiphers::new( + CreatorAuthoritySecretCipher::new([7; 32]), + FinalCredentialCipher::new([8; 32]), + ), + Arc::new(InMemoryContentLockRepository::new()), + external.clone(), + external, + Arc::new(InMemoryLockServicePointerRepository::new()), + Arc::new(InMemoryEntitlementRepository::new()), + ) +} + struct TestDatabase { pool: PgPool, schema_name: String, @@ -537,6 +1433,7 @@ fn app_state(pool: PgPool) -> AppState { FinalCredentialCipher::new([8; 32]), ), std::sync::Arc::new(InMemoryContentLockRepository::new()), + std::sync::Arc::new(InMemoryContentLockTombstoneRepository::new()), std::sync::Arc::new(InMemoryGuardedResourceRepository::new()), std::sync::Arc::new(InMemoryLockServicePointerRepository::new()), std::sync::Arc::new(InMemoryEntitlementRepository::new()), @@ -574,6 +1471,7 @@ fn test_config() -> LockServerRuntimeConfig { run_migrations_on_startup: true, }, deletion: DeletionConfig::default(), + deletion_worker: locks_server::config::DeletionWorkerConfig::default(), worker: WorkerConfig { enabled: true, poll_interval_ms: 250, diff --git a/locks-e2e/tests/production_creator_publishing_http.rs b/locks-e2e/tests/production_creator_publishing_http.rs index 6e87814..c1e2f88 100644 --- a/locks-e2e/tests/production_creator_publishing_http.rs +++ b/locks-e2e/tests/production_creator_publishing_http.rs @@ -14,8 +14,8 @@ use locks_service::application::models::{CreatorAuthorityAuthKind, FrontendSessi use locks_service::application::ports::{CreatorAuthorityManager, CreatorAuthorityStatus}; use locks_service::infrastructure::pubky::{ AuthorizingPubkyHomeserverStorageClient, PubkyBytesResource, PubkyContentLockRepository, - PubkyEntitlementRepository, PubkyHomeserverStorageClient, PubkyLockServicePointerRepository, - PubkyPrivResourceRepository, + PubkyContentLockTombstoneRepository, PubkyEntitlementRepository, PubkyHomeserverStorageClient, + PubkyLockServicePointerRepository, PubkyPrivResourceRepository, }; use serde_json::json; use support::creator_publishing_client::LocalCreatorPublishingClient; @@ -39,6 +39,9 @@ async fn production_creator_publishing_http_flow_writes_to_pubky_storage_when_fr storage.clone(), manager.clone(), ))), + Arc::new(PubkyContentLockTombstoneRepository::new( + authorizing_storage(storage.clone(), manager.clone()), + )), Arc::new(PubkyPrivResourceRepository::new(authorizing_storage( storage.clone(), manager.clone(), @@ -162,6 +165,9 @@ async fn production_creator_publishing_http_returns_creator_authority_unavailabl storage.clone(), manager.clone(), ))), + Arc::new(PubkyContentLockTombstoneRepository::new( + authorizing_storage(storage.clone(), manager.clone()), + )), Arc::new(PubkyPrivResourceRepository::new(authorizing_storage( storage.clone(), manager.clone(), diff --git a/locks-e2e/tests/pubky_homeserver_repositories.rs b/locks-e2e/tests/pubky_homeserver_repositories.rs index ad73c3a..e0b6fbb 100644 --- a/locks-e2e/tests/pubky_homeserver_repositories.rs +++ b/locks-e2e/tests/pubky_homeserver_repositories.rs @@ -12,8 +12,9 @@ use locks_server::testing::TestServerApp; use locks_service::application::errors::ApplicationError; use locks_service::application::models::FrontendSessionToken; use locks_service::infrastructure::pubky::{ - PubkyBytesResource, PubkyContentLockRepository, PubkyEntitlementRepository, - PubkyHomeserverStorageClient, PubkyLockServicePointerRepository, PubkyPrivResourceRepository, + PubkyBytesResource, PubkyContentLockRepository, PubkyContentLockTombstoneRepository, + PubkyEntitlementRepository, PubkyHomeserverStorageClient, PubkyLockServicePointerRepository, + PubkyPrivResourceRepository, }; use serde_json::json; use support::creator_publishing_client::{LocalCreatorPublishingClient, response_bytes}; @@ -31,6 +32,7 @@ async fn pubky_homeserver_repository_flow_writes_to_fake_homeserver_storage() { let state = AppState::new_empty_in_memory_with_creator_repositories( config, Arc::new(PubkyContentLockRepository::new(storage.clone())), + Arc::new(PubkyContentLockTombstoneRepository::new(storage.clone())), Arc::new(PubkyPrivResourceRepository::new(storage.clone())), Arc::new(PubkyLockServicePointerRepository::new(storage.clone())), Arc::new(PubkyEntitlementRepository::new(storage.clone())), diff --git a/locks-server/config/example.dev.postgres.toml b/locks-server/config/example.dev.postgres.toml index efe1215..ef9fb45 100644 --- a/locks-server/config/example.dev.postgres.toml +++ b/locks-server/config/example.dev.postgres.toml @@ -16,6 +16,13 @@ poll_interval_ms = 250 claim_timeout_seconds = 60 worker_id = "default-worker" +[deletion_worker] +enabled = true +poll_interval_ms = 250 +claim_timeout_seconds = 60 +shutdown_timeout_seconds = 30 +worker_id = "deletion-worker" + [runtime] environment = "development" diff --git a/locks-server/src/api/creator_publishing.rs b/locks-server/src/api/creator_publishing.rs index d158fd6..66d2f87 100644 --- a/locks-server/src/api/creator_publishing.rs +++ b/locks-server/src/api/creator_publishing.rs @@ -219,7 +219,7 @@ async fn force_delete_content_lock( } let content_lock = match state .content_lock_deletions() - .prepare_force_deletion(&creator, &lock_id, state.clock().now()) + .prepare_force_deletion(&creator, &lock_id) .await? { PrepareForceDeletionResult::PublicationInProgress => { diff --git a/locks-server/src/api/errors.rs b/locks-server/src/api/errors.rs index bffe91b..ecddddb 100644 --- a/locks-server/src/api/errors.rs +++ b/locks-server/src/api/errors.rs @@ -240,6 +240,7 @@ impl From for ApiError { Self::new(ApiErrorCode::RateLimited, "rate limit exceeded") } ApplicationError::Storage { .. } + | ApplicationError::VerificationDependencyUnavailable | ApplicationError::FinalCredentialSecret { .. } | ApplicationError::InvalidContentLockDeletionState { .. } | ApplicationError::Verifier { .. } diff --git a/locks-server/src/api/routes/tests.rs b/locks-server/src/api/routes/tests.rs index 77f760c..16803d1 100644 --- a/locks-server/src/api/routes/tests.rs +++ b/locks-server/src/api/routes/tests.rs @@ -51,6 +51,7 @@ use crate::config::{ VerificationSubmissionRateLimitConfig, WorkerConfig, }; +use locks_service::infrastructure::memory::content_lock_tombstones::InMemoryContentLockTombstoneRepository; use locks_service::infrastructure::memory::content_locks::InMemoryContentLockRepository; use locks_service::infrastructure::memory::entitlements::InMemoryEntitlementRepository; use locks_service::infrastructure::memory::guarded_resources::InMemoryGuardedResourceRepository; @@ -130,8 +131,7 @@ impl AccessCredentialStore for ResponseBoundaryAccessCredentialStore { &self, _lookup_key: &AccessCredentialLookupKey, path: &str, - _now: time::OffsetDateTime, - _claim_expires_at: time::OffsetDateTime, + _claim_duration: time::Duration, ) -> Result, ApplicationError> { Ok(Some(DeletionReadAuthorization { claim_token: Some(self.claim_token), @@ -169,7 +169,6 @@ impl AccessCredentialStore for ResponseBoundaryAccessCredentialStore { _lookup_key: &AccessCredentialLookupKey, _path: &str, claim_token: Uuid, - _now: time::OffsetDateTime, ) -> Result { assert_eq!(claim_token, self.claim_token); self.consumes.fetch_add(1, Ordering::SeqCst); @@ -647,7 +646,7 @@ async fn force_after_terminal_graceful_failure_runs_synchronously_from_frozen_ma let now = state.clock().now(); let claimed = state .content_lock_deletions() - .claim_next("test-worker", now, now + time::Duration::minutes(1)) + .claim_next("test-worker", (now + time::Duration::minutes(1)) - (now)) .await .unwrap() .unwrap(); @@ -657,7 +656,6 @@ async fn force_after_terminal_graceful_failure_runs_synchronously_from_frozen_ma claimed.job.job_id, "test-worker", claimed.claim_token, - now, Some(ContentLockDeletionFailureCode::TombstoneMissing), ) .await @@ -736,7 +734,7 @@ async fn graceful_replay_of_failed_job_requeues_same_frozen_manifest() { let now = state.clock().now(); let claimed = state .content_lock_deletions() - .claim_next("test-worker", now, now + time::Duration::minutes(1)) + .claim_next("test-worker", (now + time::Duration::minutes(1)) - (now)) .await .unwrap() .unwrap(); @@ -746,7 +744,6 @@ async fn graceful_replay_of_failed_job_requeues_same_frozen_manifest() { claimed.job.job_id, "test-worker", claimed.claim_token, - now, Some(ContentLockDeletionFailureCode::TombstoneMissing), ) .await @@ -889,8 +886,30 @@ async fn creator_content_lock_deletion_status_reports_job_and_absence() { } #[tokio::test] -async fn readyz_returns_ready_for_ephemeral_runtime_without_secrets() { - let response = router(test_state()) +async fn readyz_requires_independent_ready_evidence_for_enabled_workers() { + let state = test_state(); + let starting = router(state.clone()) + .oneshot(empty_request("GET", "/readyz")) + .await + .unwrap(); + assert_eq!(starting.status(), StatusCode::SERVICE_UNAVAILABLE); + assert_eq!(response_json(starting).await["status"], "not_ready"); + + state.record_worker_readiness( + crate::app_state::WorkerKind::Verification, + crate::app_state::WorkerReadinessEvidence::Ready, + ); + let verification_only = router(state.clone()) + .oneshot(empty_request("GET", "/readyz")) + .await + .unwrap(); + assert_eq!(verification_only.status(), StatusCode::SERVICE_UNAVAILABLE); + + state.record_worker_readiness( + crate::app_state::WorkerKind::Deletion, + crate::app_state::WorkerReadinessEvidence::Ready, + ); + let response = router(state) .oneshot(empty_request("GET", "/readyz")) .await .unwrap(); @@ -923,6 +942,7 @@ async fn readyz_returns_ready_for_ephemeral_runtime_without_secrets() { async fn readyz_reports_worker_disabled_for_ephemeral_runtime() { let mut config = test_config(RuntimeEnvironment::Development, true); config.worker.enabled = false; + config.deletion_worker.enabled = false; let response = router(AppState::new_empty_in_memory(config)) .oneshot(empty_request("GET", "/readyz")) .await @@ -936,6 +956,31 @@ async fn readyz_reports_worker_disabled_for_ephemeral_runtime() { assert_eq!(body.as_object().unwrap().len(), 3); } +#[tokio::test] +async fn readyz_reports_degraded_when_a_worker_dependency_is_degraded() { + let state = test_state(); + state.record_worker_readiness( + crate::app_state::WorkerKind::Verification, + crate::app_state::WorkerReadinessEvidence::Ready, + ); + state.record_worker_readiness( + crate::app_state::WorkerKind::Deletion, + crate::app_state::WorkerReadinessEvidence::TransientDependencyFailure, + ); + + let response = router(state) + .oneshot(empty_request("GET", "/readyz")) + .await + .unwrap(); + + assert_eq!(response.status(), StatusCode::OK); + let body = response_json(response).await; + assert_eq!(body["status"], "degraded"); + assert_eq!(body["runtime_storage"], "ephemeral"); + assert_eq!(body["worker_enabled"], true); + assert_eq!(body.as_object().unwrap().len(), 3); +} + #[tokio::test] async fn readyz_returns_not_ready_for_persisted_runtime_when_pool_ping_fails() { let pool = PgPoolOptions::new() @@ -949,6 +994,14 @@ async fn readyz_returns_not_ready_for_persisted_runtime_when_pool_ping_fails() { CreatorAuthoritySecretCipher::new([7; 32]), locks_service::infrastructure::final_credentials::FinalCredentialCipher::new([8; 32]), ); + state.record_worker_readiness( + crate::app_state::WorkerKind::Verification, + crate::app_state::WorkerReadinessEvidence::Ready, + ); + state.record_worker_readiness( + crate::app_state::WorkerKind::Deletion, + crate::app_state::WorkerReadinessEvidence::TransientDependencyFailure, + ); let response = router(state) .oneshot(empty_request("GET", "/readyz")) @@ -1553,6 +1606,7 @@ async fn dev_pubky_homeserver_routes_mount_authenticated_creator_routes_and_manu let app = router(AppState::new_empty_in_memory_with_creator_repositories( config, Arc::new(InMemoryContentLockRepository::new()), + Arc::new(InMemoryContentLockTombstoneRepository::new()), Arc::new(InMemoryGuardedResourceRepository::new()), Arc::new(InMemoryLockServicePointerRepository::new()), Arc::new(InMemoryEntitlementRepository::new()), @@ -3475,6 +3529,7 @@ fn test_state_with_runtime( AppState::new_empty_in_memory_with_creator_repositories( config, Arc::new(InMemoryContentLockRepository::new()), + Arc::new(InMemoryContentLockTombstoneRepository::new()), Arc::new(InMemoryGuardedResourceRepository::new()), Arc::new(InMemoryLockServicePointerRepository::new()), Arc::new(InMemoryEntitlementRepository::new()), @@ -3509,6 +3564,7 @@ fn test_state_with_content_lock_limits(content_locks: ContentLocksConfig) -> App AppState::new_empty_in_memory_with_creator_repositories( config, Arc::new(InMemoryContentLockRepository::new()), + Arc::new(InMemoryContentLockTombstoneRepository::new()), Arc::new(InMemoryGuardedResourceRepository::new()), Arc::new(InMemoryLockServicePointerRepository::new()), Arc::new(InMemoryEntitlementRepository::new()), @@ -3523,6 +3579,7 @@ fn test_state_with_creator_repository_backend( AppState::new_empty_in_memory_with_creator_repositories( config, Arc::new(InMemoryContentLockRepository::new()), + Arc::new(InMemoryContentLockTombstoneRepository::new()), Arc::new(InMemoryGuardedResourceRepository::new()), Arc::new(InMemoryLockServicePointerRepository::new()), Arc::new(InMemoryEntitlementRepository::new()), @@ -3563,6 +3620,7 @@ fn test_config( rate_limits: RateLimitsConfig::default(), content_locks: ContentLocksConfig::default(), deletion: crate::config::DeletionConfig::default(), + deletion_worker: crate::config::DeletionWorkerConfig::default(), paykit: None, } } diff --git a/locks-server/src/api/runtime.rs b/locks-server/src/api/runtime.rs index 3a6d56c..881076b 100644 --- a/locks-server/src/api/runtime.rs +++ b/locks-server/src/api/runtime.rs @@ -6,48 +6,46 @@ use axum::response::{IntoResponse, Response}; use crate::api::dtos::{ HealthHttpResponse, ReadinessHttpResponse, WellKnownLocksServerHttpResponse, }; -use crate::app_state::{AppState, RuntimeStorageKind}; +use crate::app_state::{AppState, ReadinessStatus, RuntimeStorageKind}; pub(super) async fn healthz() -> Json { Json(HealthHttpResponse { status: "ok" }) } pub(super) async fn readyz(State(state): State) -> Response { - let worker_enabled = state.config().worker.enabled; - match state.private_runtime_storage_kind() { - RuntimeStorageKind::InMemory => Json(ReadinessHttpResponse { - status: "ready", - runtime_storage: "ephemeral", - worker_enabled, - }) - .into_response(), + let worker_enabled = state.config().worker.enabled || state.config().deletion_worker.enabled; + let (runtime_storage, database_ready) = match state.private_runtime_storage_kind() { + RuntimeStorageKind::InMemory => ("ephemeral", true), RuntimeStorageKind::Postgres => { - let is_ready = match state.postgres_pool() { + let database_ready = match state.postgres_pool() { Some(pool) => sqlx::query_scalar::<_, i32>("SELECT 1") .fetch_one(pool) .await .is_ok(), None => false, }; - if is_ready { - Json(ReadinessHttpResponse { - status: "ready", - runtime_storage: "persisted", - worker_enabled, - }) - .into_response() - } else { - ( - StatusCode::SERVICE_UNAVAILABLE, - Json(ReadinessHttpResponse { - status: "not_ready", - runtime_storage: "persisted", - worker_enabled, - }), - ) - .into_response() - } + ("persisted", database_ready) } + }; + + let status = if !database_ready { + ReadinessStatus::NotReady + } else { + state.worker_readiness_status() + }; + let response = Json(ReadinessHttpResponse { + status: match status { + ReadinessStatus::Ready => "ready", + ReadinessStatus::Degraded => "degraded", + ReadinessStatus::NotReady => "not_ready", + }, + runtime_storage, + worker_enabled, + }); + + match status { + ReadinessStatus::Ready | ReadinessStatus::Degraded => response.into_response(), + ReadinessStatus::NotReady => (StatusCode::SERVICE_UNAVAILABLE, response).into_response(), } } diff --git a/locks-server/src/app_state/creator_repositories.rs b/locks-server/src/app_state/creator_repositories.rs index 74613b4..f2389b1 100644 --- a/locks-server/src/app_state/creator_repositories.rs +++ b/locks-server/src/app_state/creator_repositories.rs @@ -2,17 +2,17 @@ use std::sync::Arc; use locks_service::{ application::ports::{ - ContentLockRepository, EntitlementRepository, GuardedResourceRepository, - LockServicePointerRepository, + ContentLockRepository, ContentLockTombstoneRepository, EntitlementRepository, + GuardedResourceRepository, LockServicePointerRepository, }, infrastructure::{ postgres::PostgresCreatorAuthorityStore, pubky::{ LegacyCookieCreatorScopedPubkyStorageProvider, ProviderBackedPubkyHomeserverStorageClient, PubkyContentLockRepository, - PubkyEntitlementRepository, PubkyHomeserverStorageClient, - PubkyLegacyCookieSessionImporter, PubkyLockServicePointerRepository, - PubkyPrivResourceRepository, + PubkyContentLockTombstoneRepository, PubkyEntitlementRepository, + PubkyHomeserverStorageClient, PubkyLegacyCookieSessionImporter, + PubkyLockServicePointerRepository, PubkyPrivResourceRepository, }, }, }; @@ -20,6 +20,7 @@ use locks_service::{ #[derive(Clone)] pub(super) struct CreatorRepositoryAdapters { pub(super) content_locks: Arc, + pub(super) content_lock_tombstones: Arc, pub(super) guarded_resources: Arc, pub(super) lock_service_pointers: Arc, pub(super) entitlements: Arc, @@ -28,12 +29,14 @@ pub(super) struct CreatorRepositoryAdapters { impl CreatorRepositoryAdapters { pub(super) fn new( content_locks: Arc, + content_lock_tombstones: Arc, guarded_resources: Arc, lock_service_pointers: Arc, entitlements: Arc, ) -> Self { Self { content_locks, + content_lock_tombstones, guarded_resources, lock_service_pointers, entitlements, @@ -52,6 +55,7 @@ impl CreatorRepositoryAdapters { Self::new( Arc::new(PubkyContentLockRepository::new(client.clone())), + Arc::new(PubkyContentLockTombstoneRepository::new(client.clone())), Arc::new(PubkyPrivResourceRepository::new(client.clone())), Arc::new(PubkyLockServicePointerRepository::new(client.clone())), Arc::new(PubkyEntitlementRepository::new(client)), diff --git a/locks-server/src/app_state/mod.rs b/locks-server/src/app_state/mod.rs index 22eee4a..4361420 100644 --- a/locks-server/src/app_state/mod.rs +++ b/locks-server/src/app_state/mod.rs @@ -17,38 +17,43 @@ use locks_service::{ errors::ApplicationError, models::AccessCredentialPolicy, ports::{ - AccessCredentialStore, Clock, ContentLockDeletionRepository, - ContentLockOwnershipRepository, ContentLockRepository, CreatorAuthorityManager, - CreatorAuthorityStore, CreatorConnectFlowStore, EntitlementRepository, - FrontendSessionCodeStore, FrontendSessionStore, GuardedResourceRepository, - LegacyCreatorConnectFlowClient, LockServicePointerRepository, PaymentDrainClient, - PaymentDrainRepository, VerificationTaskClaimer, VerificationTaskRepository, + AccessCredentialStore, Clock, ContentLockDeletionActionOwnership, + ContentLockDeletionRepository, ContentLockOwnershipRepository, ContentLockRepository, + ContentLockTombstoneRepository, CreatorAuthorityManager, CreatorAuthorityStore, + CreatorConnectFlowStore, EntitlementRepository, FrontendSessionCodeStore, + FrontendSessionStore, GuardedResourceRepository, LegacyCreatorConnectFlowClient, + LockServicePointerRepository, PaymentDrainClient, PaymentDrainRepository, + VerificationTaskClaimer, VerificationTaskRepository, }, }, infrastructure::{ final_credentials::FinalCredentialCipher, memory::{ access_credentials::InMemoryAccessCredentialStore, + content_lock_deletion_action_ownership::InMemoryContentLockDeletionActionOwnership, content_lock_deletions::InMemoryContentLockDeletionRepository, content_lock_ownership::InMemoryContentLockOwnershipRepository, + content_lock_tombstones::InMemoryContentLockTombstoneRepository, + content_locks::InMemoryContentLockRepository, + public_content_locks::InMemoryPublicContentLockStore, verification_task_claims::InMemoryVerificationTaskClaimer, verification_task_deletion_fence::InMemoryVerificationTaskDeletionFence, verification_tasks::InMemoryVerificationTaskRepository, }, postgres::{ CreatorAuthoritySecretCipher, PostgresAccessCredentialStore, - PostgresContentLockDeletionRepository, PostgresContentLockOwnershipRepository, - PostgresCreatorAuthorityStore, PostgresCreatorConnectFlowStore, - PostgresFrontendSessionCodeStore, PostgresFrontendSessionStore, - PostgresPaymentDrainRepository, PostgresVerificationTaskClaimer, - PostgresVerificationTaskRepository, + PostgresContentLockDeletionActionOwnership, PostgresContentLockDeletionRepository, + PostgresContentLockOwnershipRepository, PostgresCreatorAuthorityStore, + PostgresCreatorConnectFlowStore, PostgresFrontendSessionCodeStore, + PostgresFrontendSessionStore, PostgresPaymentDrainRepository, + PostgresVerificationTaskClaimer, PostgresVerificationTaskRepository, }, pubky::{ AuthorizingPubkyHomeserverStorageClient, LegacyCookieCreatorAuthorityManager, - PubkyBytesResource, PubkyContentLockRepository, PubkyEntitlementRepository, - PubkyHomeserverStorageClient, PubkyLegacyCookieSessionRevalidator, - PubkyLegacyCreatorConnectFlowClient, PubkyLockServicePointerRepository, - PubkyPrivResourceRepository, + PubkyBytesResource, PubkyContentLockRepository, PubkyContentLockTombstoneRepository, + PubkyEntitlementRepository, PubkyHomeserverStorageClient, + PubkyLegacyCookieSessionRevalidator, PubkyLegacyCreatorConnectFlowClient, + PubkyLockServicePointerRepository, PubkyPrivResourceRepository, }, verifiers::dev_static::DevStaticVerifier, verifiers::paykit_payment::PaykitPaymentVerifier, @@ -72,7 +77,10 @@ use crate::app_state::private_runtime::{ use crate::app_state::pubky_clients::{ build_pubky_client, build_pubky_http_client, pubky_auth_relay_for_network, }; -pub use crate::app_state::readiness::RuntimeStorageKind; +pub use crate::app_state::readiness::{ + ReadinessStatus, RuntimeStorageKind, WorkerKind, WorkerReadiness, WorkerReadinessEvidence, + WorkerReadinessState, +}; use crate::config::LockServerRuntimeConfig; use crate::paykit_http_client::PaykitHttpClient; use crate::rate_limit::InMemoryVerificationSubmissionRateLimiter; @@ -151,11 +159,14 @@ pub struct AppState { config: LockServerRuntimeConfig, private_runtime_storage_kind: RuntimeStorageKind, postgres_pool: Option, + worker_readiness: WorkerReadiness, content_locks: Arc, + content_lock_tombstones: Arc, guarded_resources: Arc, lock_service_pointers: Arc, content_lock_ownership: Arc, content_lock_deletions: Arc, + content_lock_deletion_action_ownership: Arc, payment_drains: Option>, verification_tasks: Arc, verification_task_claimer: Arc, @@ -255,21 +266,36 @@ impl AppState { creator_authority_store, NoopLegacyCookieSessionRevalidator, )); - let unavailable_storage = UnavailablePubkyHomeserverStorageClient; + let unavailable_storage: Arc = + Arc::new(UnavailablePubkyHomeserverStorageClient); + let public_content_locks = InMemoryPublicContentLockStore::new(); let creator_repositories = CreatorRepositoryAdapters::new( - Arc::new(PubkyContentLockRepository::new(unavailable_storage)), - Arc::new(PubkyPrivResourceRepository::new(unavailable_storage)), - Arc::new(PubkyLockServicePointerRepository::new(unavailable_storage)), + Arc::new(InMemoryContentLockRepository::with_public_store( + public_content_locks.clone(), + )), + Arc::new(InMemoryContentLockTombstoneRepository::with_public_store( + public_content_locks, + )), + Arc::new(PubkyPrivResourceRepository::new( + unavailable_storage.clone(), + )), + Arc::new(PubkyLockServicePointerRepository::new( + unavailable_storage.clone(), + )), Arc::new(PubkyEntitlementRepository::new(unavailable_storage)), ); + let content_lock_deletions = Arc::new( + InMemoryContentLockDeletionRepository::with_access_credentials_and_verification_task_fence( + Arc::clone(&access_credentials), + verification_task_deletion_fence, + ), + ); let private_runtime = PrivateRuntimeAdapters { content_lock_ownership: Arc::new(InMemoryContentLockOwnershipRepository::new()), - content_lock_deletions: Arc::new( - InMemoryContentLockDeletionRepository::with_access_credentials_and_verification_task_fence( - Arc::clone(&access_credentials), - verification_task_deletion_fence, - ), + content_lock_deletions: content_lock_deletions.clone(), + content_lock_deletion_action_ownership: Arc::new( + InMemoryContentLockDeletionActionOwnership::new(content_lock_deletions), ), verification_tasks, verification_task_claimer, @@ -294,6 +320,7 @@ impl AppState { pub fn new_empty_in_memory_with_creator_repositories( config: LockServerRuntimeConfig, content_locks: Arc, + content_lock_tombstones: Arc, guarded_resources: Arc, lock_service_pointers: Arc, entitlements: Arc, @@ -324,18 +351,23 @@ impl AppState { )); let creator_repositories = CreatorRepositoryAdapters::new( content_locks, + content_lock_tombstones, guarded_resources, lock_service_pointers, entitlements, ); + let content_lock_deletions = Arc::new( + InMemoryContentLockDeletionRepository::with_access_credentials_and_verification_task_fence( + Arc::clone(&access_credentials), + verification_task_deletion_fence, + ), + ); let private_runtime = PrivateRuntimeAdapters { content_lock_ownership: Arc::new(InMemoryContentLockOwnershipRepository::new()), - content_lock_deletions: Arc::new( - InMemoryContentLockDeletionRepository::with_access_credentials_and_verification_task_fence( - Arc::clone(&access_credentials), - verification_task_deletion_fence, - ), + content_lock_deletions: content_lock_deletions.clone(), + content_lock_deletion_action_ownership: Arc::new( + InMemoryContentLockDeletionActionOwnership::new(content_lock_deletions), ), verification_tasks, verification_task_claimer, @@ -388,37 +420,39 @@ impl AppState { creator_authority_store.clone(), NoopLegacyCookieSessionRevalidator, )); - let authorizing_storage = |storage: S| { - AuthorizingPubkyHomeserverStorageClient::new( + let authorizing_storage: Arc = + Arc::new(AuthorizingPubkyHomeserverStorageClient::new( storage, LegacyCookieCreatorAuthorityManager::new( creator_authority_store.clone(), NoopLegacyCookieSessionRevalidator, ), - ) - }; + )); let creator_repositories = CreatorRepositoryAdapters::new( - Arc::new(PubkyContentLockRepository::new(authorizing_storage( - storage.clone(), - ))), - Arc::new(PubkyPrivResourceRepository::new(authorizing_storage( - storage.clone(), - ))), - Arc::new(PubkyLockServicePointerRepository::new(authorizing_storage( - storage.clone(), - ))), - Arc::new(PubkyEntitlementRepository::new(authorizing_storage( - storage, - ))), + Arc::new(PubkyContentLockRepository::new(authorizing_storage.clone())), + Arc::new(PubkyContentLockTombstoneRepository::new( + authorizing_storage.clone(), + )), + Arc::new(PubkyPrivResourceRepository::new( + authorizing_storage.clone(), + )), + Arc::new(PubkyLockServicePointerRepository::new( + authorizing_storage.clone(), + )), + Arc::new(PubkyEntitlementRepository::new(authorizing_storage)), ); + let content_lock_deletions = Arc::new( + InMemoryContentLockDeletionRepository::with_access_credentials_and_verification_task_fence( + Arc::clone(&access_credentials), + verification_task_deletion_fence, + ), + ); let private_runtime = PrivateRuntimeAdapters { content_lock_ownership: Arc::new(InMemoryContentLockOwnershipRepository::new()), - content_lock_deletions: Arc::new( - InMemoryContentLockDeletionRepository::with_access_credentials_and_verification_task_fence( - Arc::clone(&access_credentials), - verification_task_deletion_fence, - ), + content_lock_deletions: content_lock_deletions.clone(), + content_lock_deletion_action_ownership: Arc::new( + InMemoryContentLockDeletionActionOwnership::new(content_lock_deletions), ), verification_tasks, verification_task_claimer, @@ -485,6 +519,9 @@ impl AppState { content_lock_deletions: Arc::new(PostgresContentLockDeletionRepository::new( pool.clone(), )), + content_lock_deletion_action_ownership: Arc::new( + PostgresContentLockDeletionActionOwnership::new(pool.clone()), + ), verification_tasks, verification_task_claimer, access_credentials, @@ -505,11 +542,13 @@ impl AppState { ) } + #[allow(clippy::too_many_arguments)] pub fn new_with_postgres_runtime_and_creator_repositories( config: LockServerRuntimeConfig, pool: PgPool, ciphers: RuntimeSecretCiphers, content_locks: Arc, + content_lock_tombstones: Arc, guarded_resources: Arc, lock_service_pointers: Arc, entitlements: Arc, @@ -545,6 +584,7 @@ impl AppState { }; let creator_repositories = CreatorRepositoryAdapters::new( content_locks, + content_lock_tombstones, guarded_resources, lock_service_pointers, entitlements, @@ -556,6 +596,9 @@ impl AppState { content_lock_deletions: Arc::new(PostgresContentLockDeletionRepository::new( pool.clone(), )), + content_lock_deletion_action_ownership: Arc::new( + PostgresContentLockDeletionActionOwnership::new(pool.clone()), + ), verification_tasks, verification_task_claimer, access_credentials, @@ -583,6 +626,8 @@ impl AppState { creator_repositories: CreatorRepositoryAdapters, private_runtime: PrivateRuntimeAdapters, ) -> Self { + let worker_readiness = + WorkerReadiness::new(config.worker.enabled, config.deletion_worker.enabled); let access_credential_policy = AccessCredentialPolicy::new(config.credentials.max_ttl_seconds); let verification_submission_rate_limiter = @@ -617,11 +662,15 @@ impl AppState { config, private_runtime_storage_kind, postgres_pool, + worker_readiness, content_locks: creator_repositories.content_locks, + content_lock_tombstones: creator_repositories.content_lock_tombstones, guarded_resources: creator_repositories.guarded_resources, lock_service_pointers: creator_repositories.lock_service_pointers, content_lock_ownership: private_runtime.content_lock_ownership, content_lock_deletions: private_runtime.content_lock_deletions, + content_lock_deletion_action_ownership: private_runtime + .content_lock_deletion_action_ownership, payment_drains, verification_tasks: private_runtime.verification_tasks, verification_task_claimer: private_runtime.verification_task_claimer, @@ -661,10 +710,26 @@ impl AppState { self.postgres_pool.as_ref() } + pub fn worker_readiness(&self) -> &WorkerReadiness { + &self.worker_readiness + } + + pub fn worker_readiness_status(&self) -> ReadinessStatus { + self.worker_readiness.status() + } + + pub fn record_worker_readiness(&self, worker: WorkerKind, evidence: WorkerReadinessEvidence) { + self.worker_readiness.record(worker, evidence); + } + pub fn content_locks(&self) -> &Arc { &self.content_locks } + pub fn content_lock_tombstones(&self) -> &Arc { + &self.content_lock_tombstones + } + pub fn guarded_resources(&self) -> &Arc { &self.guarded_resources } @@ -677,6 +742,12 @@ impl AppState { &self.content_lock_deletions } + pub fn content_lock_deletion_action_ownership( + &self, + ) -> &Arc { + &self.content_lock_deletion_action_ownership + } + pub fn payment_drains(&self) -> Option<&Arc> { self.payment_drains.as_ref() } diff --git a/locks-server/src/app_state/private_runtime.rs b/locks-server/src/app_state/private_runtime.rs index b0ce136..58d3196 100644 --- a/locks-server/src/app_state/private_runtime.rs +++ b/locks-server/src/app_state/private_runtime.rs @@ -11,10 +11,10 @@ use locks_service::application::{ PendingCreatorConnectFlowRecord, }, ports::{ - AccessCredentialStore, ContentLockDeletionRepository, ContentLockOwnershipRepository, - CreatorAuthorityManager, CreatorAuthorityStore, CreatorConnectFlowStore, - FrontendSessionCodeStore, FrontendSessionStore, LegacyCreatorConnectFlowClient, - VerificationTaskClaimer, VerificationTaskRepository, + AccessCredentialStore, ContentLockDeletionActionOwnership, ContentLockDeletionRepository, + ContentLockOwnershipRepository, CreatorAuthorityManager, CreatorAuthorityStore, + CreatorConnectFlowStore, FrontendSessionCodeStore, FrontendSessionStore, + LegacyCreatorConnectFlowClient, VerificationTaskClaimer, VerificationTaskRepository, }, }; use time::OffsetDateTime; @@ -24,6 +24,7 @@ use tokio::sync::RwLock; pub(super) struct PrivateRuntimeAdapters { pub(super) content_lock_ownership: Arc, pub(super) content_lock_deletions: Arc, + pub(super) content_lock_deletion_action_ownership: Arc, pub(super) verification_tasks: Arc, pub(super) verification_task_claimer: Arc, pub(super) access_credentials: Arc, diff --git a/locks-server/src/app_state/readiness.rs b/locks-server/src/app_state/readiness.rs index 96ac469..203dd95 100644 --- a/locks-server/src/app_state/readiness.rs +++ b/locks-server/src/app_state/readiness.rs @@ -1,6 +1,192 @@ +use std::sync::{Arc, RwLock}; + /// Runtime composition state for the Lock Server process. #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub enum RuntimeStorageKind { InMemory, Postgres, } + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum WorkerKind { + Verification, + Deletion, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum WorkerReadinessState { + Disabled, + Starting, + Ready, + Degraded, + NotReady, + Stopped, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum WorkerReadinessEvidence { + Starting, + Ready, + DependencySucceeded, + TransientDependencyFailure, + PendingWork, + LockContention, + TerminalBusinessFailure, + Stopping, + Stopped, + UnexpectedExit, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum ReadinessStatus { + Ready, + Degraded, + NotReady, +} + +#[derive(Debug, Clone)] +pub struct WorkerReadiness { + states: Arc>, +} + +impl WorkerReadiness { + pub fn new(verification_enabled: bool, deletion_enabled: bool) -> Self { + let initial = |enabled| { + if enabled { + WorkerReadinessState::Starting + } else { + WorkerReadinessState::Disabled + } + }; + Self { + states: Arc::new(RwLock::new([ + initial(verification_enabled), + initial(deletion_enabled), + ])), + } + } + + pub fn status(&self) -> ReadinessStatus { + let states = self.states.read().expect("worker readiness lock poisoned"); + if states.iter().any(|state| { + matches!( + state, + WorkerReadinessState::Starting + | WorkerReadinessState::NotReady + | WorkerReadinessState::Stopped + ) + }) { + ReadinessStatus::NotReady + } else if states.contains(&WorkerReadinessState::Degraded) { + ReadinessStatus::Degraded + } else { + ReadinessStatus::Ready + } + } + + pub fn worker_state(&self, worker: WorkerKind) -> WorkerReadinessState { + self.states.read().expect("worker readiness lock poisoned")[worker.index()] + } + + pub fn record(&self, worker: WorkerKind, evidence: WorkerReadinessEvidence) { + let state = match evidence { + WorkerReadinessEvidence::Starting => Some(WorkerReadinessState::Starting), + WorkerReadinessEvidence::Ready | WorkerReadinessEvidence::DependencySucceeded => { + Some(WorkerReadinessState::Ready) + } + WorkerReadinessEvidence::TransientDependencyFailure => { + Some(WorkerReadinessState::Degraded) + } + WorkerReadinessEvidence::Stopping | WorkerReadinessEvidence::UnexpectedExit => { + Some(WorkerReadinessState::NotReady) + } + WorkerReadinessEvidence::Stopped => Some(WorkerReadinessState::Stopped), + WorkerReadinessEvidence::PendingWork + | WorkerReadinessEvidence::LockContention + | WorkerReadinessEvidence::TerminalBusinessFailure => None, + }; + if let Some(state) = state { + self.states.write().expect("worker readiness lock poisoned")[worker.index()] = state; + } + } +} + +impl WorkerKind { + fn index(self) -> usize { + match self { + Self::Verification => 0, + Self::Deletion => 1, + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn enabled_workers_start_not_ready_and_require_independent_ready_evidence() { + let readiness = WorkerReadiness::new(true, true); + + assert_eq!(readiness.status(), ReadinessStatus::NotReady); + assert_eq!( + readiness.worker_state(WorkerKind::Verification), + WorkerReadinessState::Starting + ); + assert_eq!( + readiness.worker_state(WorkerKind::Deletion), + WorkerReadinessState::Starting + ); + + readiness.record(WorkerKind::Verification, WorkerReadinessEvidence::Ready); + assert_eq!(readiness.status(), ReadinessStatus::NotReady); + + readiness.record(WorkerKind::Deletion, WorkerReadinessEvidence::Ready); + assert_eq!(readiness.status(), ReadinessStatus::Ready); + } + + #[test] + fn deletion_dependency_failure_degrades_until_success_without_business_outcome_noise() { + let readiness = WorkerReadiness::new(false, true); + readiness.record(WorkerKind::Deletion, WorkerReadinessEvidence::Ready); + + readiness.record( + WorkerKind::Deletion, + WorkerReadinessEvidence::TransientDependencyFailure, + ); + assert_eq!(readiness.status(), ReadinessStatus::Degraded); + assert_eq!( + readiness.worker_state(WorkerKind::Deletion), + WorkerReadinessState::Degraded + ); + + for evidence in [ + WorkerReadinessEvidence::PendingWork, + WorkerReadinessEvidence::LockContention, + WorkerReadinessEvidence::TerminalBusinessFailure, + ] { + readiness.record(WorkerKind::Deletion, evidence); + assert_eq!(readiness.status(), ReadinessStatus::Degraded); + } + + readiness.record( + WorkerKind::Deletion, + WorkerReadinessEvidence::DependencySucceeded, + ); + assert_eq!(readiness.status(), ReadinessStatus::Ready); + } + + #[test] + fn stopping_stopped_and_unexpected_exit_are_not_ready() { + for evidence in [ + WorkerReadinessEvidence::Stopping, + WorkerReadinessEvidence::Stopped, + WorkerReadinessEvidence::UnexpectedExit, + ] { + let readiness = WorkerReadiness::new(false, true); + readiness.record(WorkerKind::Deletion, WorkerReadinessEvidence::Ready); + readiness.record(WorkerKind::Deletion, evidence); + assert_eq!(readiness.status(), ReadinessStatus::NotReady); + } + } +} diff --git a/locks-server/src/app_state/test_support.rs b/locks-server/src/app_state/test_support.rs index 80e19c1..0a7997c 100644 --- a/locks-server/src/app_state/test_support.rs +++ b/locks-server/src/app_state/test_support.rs @@ -3,7 +3,8 @@ use std::path::PathBuf; use std::str::FromStr; use std::sync::Arc; -use locks_core::ids::{CreatorPubky, LockServerPubky}; +use locks_core::content_lock_deletion::ContentLockDeletionTombstone; +use locks_core::ids::{ContentLockPath, CreatorPubky, LockId, LockServerPubky}; use sqlx::postgres::PgPoolOptions; use time::macros::datetime; @@ -18,9 +19,12 @@ use crate::config::{ }; use crate::rate_limit::VerificationSubmissionRateLimitKey; use locks_service::application::errors::ApplicationError; +use locks_service::application::models::ContentLockDeletionPhase; use locks_service::application::ports::{ + ContentLockDeletionActionAcquireResult, ContentLockDeletionActionClaim, + ContentLockDeletionActionOwnership, ContentLockTombstoneRepository, CreatorConnectFlowIdGenerator, FrontendSessionCodeGenerator, FrontendSessionTokenGenerator, - VerificationTaskIdGenerator, + TombstoneReadback, VerificationTaskIdGenerator, }; use locks_service::infrastructure::final_credentials::FinalCredentialCipher; use locks_service::infrastructure::postgres::CreatorAuthoritySecretCipher; @@ -52,8 +56,86 @@ async fn postgres_state_uses_postgres_for_private_runtime_adapters() { state.private_runtime_storage_kind(), RuntimeStorageKind::Postgres ); + assert_action_ownership_adapter(state.content_lock_deletion_action_ownership().as_ref()); + assert_tombstone_adapter(state.content_lock_tombstones().as_ref()); } +#[tokio::test] +async fn in_memory_state_exposes_callable_deletion_action_and_tombstone_adapters() { + let state = AppState::new_empty_in_memory(test_config()); + let result = state + .content_lock_deletion_action_ownership() + .try_acquire(ContentLockDeletionActionClaim { + job_id: uuid::Uuid::new_v4(), + worker_id: "test-worker", + claim_token: uuid::Uuid::new_v4(), + expected_phase: ContentLockDeletionPhase::Withdraw, + force: false, + }) + .await + .unwrap(); + assert!(matches!( + result, + ContentLockDeletionActionAcquireResult::ClaimLost + )); + + let lock_id = LockId::from_str("000G40R40M30E209185GR38E1W8124GK2GAHC5RR34D1P70X3RFG").unwrap(); + let path = ContentLockPath::from_lock_id(lock_id.clone()); + let tombstone = ContentLockDeletionTombstone::new(lock_id, datetime!(2026-08-12 05:00:00 UTC)); + assert_eq!( + state + .content_lock_tombstones() + .read_tombstone(&rate_limit_key().creator, &path, &tombstone) + .await + .unwrap(), + TombstoneReadback::Missing + ); + + let creator = rate_limit_key().creator; + let content_lock: locks_core::lock_policy::ContentLock = + serde_json::from_value(serde_json::json!({ + "version": 1, + "creator": creator, + "primary_resource": null, + "secondary_resources": {}, + "criteria": [], + "lock_logic": { "type": "all", "criteria": [] }, + "access_policy": { "requested_credential_ttl_seconds": 900 }, + "lock_server": { "override": null }, + "created_at": "2026-08-12T04:00:00Z" + })) + .unwrap(); + state + .content_locks() + .upsert_content_lock(creator.clone(), path.clone(), content_lock.clone()) + .await + .unwrap(); + state + .content_lock_tombstones() + .withdraw_content_lock(creator.clone(), path.clone(), &content_lock, &tombstone) + .await + .unwrap(); + assert!( + state + .content_locks() + .get_content_lock(&creator, &path) + .await + .is_err() + ); + assert_eq!( + state + .content_lock_tombstones() + .read_tombstone(&creator, &path, &tombstone) + .await + .unwrap(), + TombstoneReadback::Exact + ); +} + +fn assert_action_ownership_adapter(_: &dyn ContentLockDeletionActionOwnership) {} + +fn assert_tombstone_adapter(_: &dyn ContentLockTombstoneRepository) {} + #[tokio::test] async fn postgres_state_wires_legacy_connect_flow_runtime_state() { let pool = PgPoolOptions::new() @@ -307,6 +389,7 @@ fn test_config() -> LockServerRuntimeConfig { rate_limits: RateLimitsConfig::default(), content_locks: ContentLocksConfig::default(), deletion: crate::config::DeletionConfig::default(), + deletion_worker: crate::config::DeletionWorkerConfig::default(), paykit: None, } } diff --git a/locks-server/src/config/defaults.rs b/locks-server/src/config/defaults.rs index 82d3250..f246ad7 100644 --- a/locks-server/src/config/defaults.rs +++ b/locks-server/src/config/defaults.rs @@ -8,3 +8,7 @@ pub(super) const DEFAULT_DELETION_RETRY_INITIAL_BACKOFF_SECONDS: u64 = 1; pub(super) const DEFAULT_DELETION_RETRY_MAX_BACKOFF_SECONDS: u64 = 300; pub(super) const DEFAULT_FINAL_CREDENTIAL_ISSUANCE_WINDOW_SECONDS: u64 = 900; pub(super) const DEFAULT_FINAL_READ_WINDOW_SECONDS: u64 = 900; +pub(super) const DEFAULT_DELETION_WORKER_POLL_INTERVAL_MS: u64 = 250; +pub(super) const DEFAULT_DELETION_WORKER_CLAIM_TIMEOUT_SECONDS: u64 = 60; +pub(super) const DEFAULT_DELETION_WORKER_SHUTDOWN_TIMEOUT_SECONDS: u64 = 30; +pub(super) const DEFAULT_DELETION_WORKER_ID: &str = "deletion-worker"; diff --git a/locks-server/src/config/examples.rs b/locks-server/src/config/examples.rs index bbe8d35..a8a817b 100644 --- a/locks-server/src/config/examples.rs +++ b/locks-server/src/config/examples.rs @@ -233,6 +233,26 @@ fn rejects_zero_worker_poll_interval() { } } +#[test] +fn rejects_zero_pkarr_republisher_interval_before_runtime_startup() { + let temp_dir = tempdir().unwrap(); + let secret_path = temp_dir.path().join("secret.sess"); + let public_key = test_identity(&secret_path); + let config_path = temp_dir.path().join("config.toml"); + let config = minimal_config(&secret_path, &public_key, "development").replace( + "key_republisher_interval_seconds = 3600", + "key_republisher_interval_seconds = 0", + ); + std::fs::write(&config_path, config).unwrap(); + + let error = load_existing_config_from_path(&config_path).unwrap_err(); + + assert!(matches!( + error, + ConfigError::InvalidPkarrRepublisherInterval + )); +} + #[test] fn rejects_removed_creator_repositories_section() { let temp_dir = tempdir().unwrap(); @@ -340,6 +360,49 @@ fn accepts_closed_deletion_defaults_and_runtime_master_key_contract() { 900 ); assert_eq!(config.deletion.final_read_window_seconds, 900); + assert!(config.deletion_worker.enabled); + assert_eq!(config.deletion_worker.poll_interval_ms, 250); + assert_eq!(config.deletion_worker.claim_timeout_seconds, 60); + assert_eq!(config.deletion_worker.shutdown_timeout_seconds, 30); + assert_eq!(config.deletion_worker.worker_id, "deletion-worker"); +} + +#[test] +fn rejects_non_positive_or_blank_deletion_worker_settings() { + let temp_dir = tempdir().unwrap(); + let secret_path = temp_dir.path().join("secret.sess"); + let public_key = test_identity(&secret_path); + let base = minimal_config(&secret_path, &public_key, "development").replace( + "[runtime]", + "[deletion_worker]\nenabled = true\npoll_interval_ms = 250\nclaim_timeout_seconds = 60\nshutdown_timeout_seconds = 30\nworker_id = \"deletion-worker\"\n\n[runtime]", + ); + + for (name, from, to) in [ + ("poll", "poll_interval_ms = 250", "poll_interval_ms = 0"), + ( + "claim", + "claim_timeout_seconds = 60", + "claim_timeout_seconds = 0", + ), + ( + "shutdown", + "shutdown_timeout_seconds = 30", + "shutdown_timeout_seconds = 0", + ), + ( + "worker-id", + "worker_id = \"deletion-worker\"", + "worker_id = \" \"", + ), + ] { + let config_path = temp_dir.path().join(format!("{name}.toml")); + std::fs::write(&config_path, base.replace(from, to)).unwrap(); + let error = load_existing_config_from_path(&config_path).unwrap_err(); + assert!( + matches!(error, ConfigError::InvalidDeletionWorkerConfig), + "unexpected error for {name}: {error}" + ); + } } #[test] diff --git a/locks-server/src/config/loading.rs b/locks-server/src/config/loading.rs index a3d7396..18692f7 100644 --- a/locks-server/src/config/loading.rs +++ b/locks-server/src/config/loading.rs @@ -4,9 +4,9 @@ use super::defaults::{DEFAULT_CONFIG_FILE, DEFAULT_SECRET_FILE, DEFAULT_SERVICE_ use super::raw::RawConfig; use super::schema::{ ConfigError, ConfigPathResolution, ContentLocksConfig, CreatorAuthorityAcquisitionConfig, - DatabaseConfig, DeletionConfig, LockServerCredentialsConfig, LockServerRuntimeConfig, - LoggingConfig, PaykitConfig, PkdnsConfig, PubkyConfig, RateLimitsConfig, RuntimeConfig, - RuntimeEnvironment, SecretsConfig, WorkerConfig, + DatabaseConfig, DeletionConfig, DeletionWorkerConfig, LockServerCredentialsConfig, + LockServerRuntimeConfig, LoggingConfig, PaykitConfig, PkdnsConfig, PubkyConfig, + RateLimitsConfig, RuntimeConfig, RuntimeEnvironment, SecretsConfig, WorkerConfig, }; use super::secrets::{LockServerIdentityProvider, parse_lock_server_keypair_seed}; @@ -149,6 +149,7 @@ fn initialize_default_config( rate_limits: RateLimitsConfig::default(), content_locks: ContentLocksConfig::default(), deletion: DeletionConfig::default(), + deletion_worker: DeletionWorkerConfig::default(), paykit: Some(PaykitConfig { server_url: "http://127.0.0.1:3001/".to_owned(), minimum_confirmations: 0, @@ -200,6 +201,13 @@ retry_max_backoff_seconds = {} # Maximum durable retry delay in seconds. final_credential_issuance_window_seconds = {} # Bounded final-credential issuance window; must be 1..=3600. final_read_window_seconds = {} # Bounded one-read-per-resource window; must be 1..=3600. +[deletion_worker] +enabled = {} # true enables the in-process deletion worker; false leaves deletion jobs for another worker process. +poll_interval_ms = {} # Deletion queue polling interval in milliseconds. Must be > 0. +claim_timeout_seconds = {} # Seconds before another deletion worker may reclaim a stuck job. Must be > 0. +shutdown_timeout_seconds = {} # Maximum graceful deletion-worker shutdown wait in seconds. Must be > 0. +worker_id = "{}" # Stable, non-blank deletion-worker identity. Use a unique value per worker process. + [logging] level = "{}" # Tracing level/filter, e.g. error, warn, info, debug, trace, or EnvFilter syntax. Higher verbosity may expose operational detail in logs. @@ -257,6 +265,11 @@ max_total_resource_bytes = {} # Maximum combined bytes across resources in one c config.deletion.retry_max_backoff_seconds, config.deletion.final_credential_issuance_window_seconds, config.deletion.final_read_window_seconds, + config.deletion_worker.enabled, + config.deletion_worker.poll_interval_ms, + config.deletion_worker.claim_timeout_seconds, + config.deletion_worker.shutdown_timeout_seconds, + config.deletion_worker.worker_id, config.logging.level, config.pkdns.public_ip, config.pkdns.public_pubky_tls_port.unwrap_or(6287), diff --git a/locks-server/src/config/mod.rs b/locks-server/src/config/mod.rs index d4ddbd9..bc56164 100644 --- a/locks-server/src/config/mod.rs +++ b/locks-server/src/config/mod.rs @@ -10,7 +10,7 @@ mod validation; pub use loading::{load_existing_config_from_path, load_or_initialize_config, resolve_config_path}; pub use schema::{ ConfigError, ConfigPathResolution, ContentLocksConfig, CreatorAuthorityAcquisitionConfig, - CreatorAuthorityAcquisitionMethod, DatabaseConfig, DeletionConfig, + CreatorAuthorityAcquisitionMethod, DatabaseConfig, DeletionConfig, DeletionWorkerConfig, LegacyConnectAcquisitionConfig, LockServerCredentialsConfig, LockServerRuntimeConfig, LoggingConfig, PAYKIT_CONNECT_TIMEOUT_SECONDS, PAYKIT_REQUEST_TIMEOUT_SECONDS, PaykitConfig, PkdnsConfig, PubkyConfig, PubkyNetwork, RateLimitsConfig, RuntimeConfig, RuntimeEnvironment, diff --git a/locks-server/src/config/raw.rs b/locks-server/src/config/raw.rs index 4ea27d2..5669a29 100644 --- a/locks-server/src/config/raw.rs +++ b/locks-server/src/config/raw.rs @@ -9,12 +9,15 @@ use url::Url; use super::defaults::{ DEFAULT_DELETION_RETRY_INITIAL_BACKOFF_SECONDS, DEFAULT_DELETION_RETRY_MAX_ATTEMPTS, - DEFAULT_DELETION_RETRY_MAX_BACKOFF_SECONDS, DEFAULT_FINAL_CREDENTIAL_ISSUANCE_WINDOW_SECONDS, - DEFAULT_FINAL_READ_WINDOW_SECONDS, DEFAULT_RUNTIME_MASTER_KEY_ENV, PUBLIC_KEY_PLACEHOLDER, + DEFAULT_DELETION_RETRY_MAX_BACKOFF_SECONDS, DEFAULT_DELETION_WORKER_CLAIM_TIMEOUT_SECONDS, + DEFAULT_DELETION_WORKER_ID, DEFAULT_DELETION_WORKER_POLL_INTERVAL_MS, + DEFAULT_DELETION_WORKER_SHUTDOWN_TIMEOUT_SECONDS, + DEFAULT_FINAL_CREDENTIAL_ISSUANCE_WINDOW_SECONDS, DEFAULT_FINAL_READ_WINDOW_SECONDS, + DEFAULT_RUNTIME_MASTER_KEY_ENV, PUBLIC_KEY_PLACEHOLDER, }; use super::schema::{ ConfigError, ContentLocksConfig, CreatorAuthorityAcquisitionConfig, - CreatorAuthorityAcquisitionMethod, DatabaseConfig, DeletionConfig, + CreatorAuthorityAcquisitionMethod, DatabaseConfig, DeletionConfig, DeletionWorkerConfig, LegacyConnectAcquisitionConfig, LockServerCredentialsConfig, LockServerRuntimeConfig, LoggingConfig, MAX_DELETION_CREDENTIAL_WINDOW_SECONDS, PAYKIT_REQUEST_TIMEOUT_SECONDS, PaykitConfig, PkdnsConfig, PubkyConfig, PubkyNetwork, RateLimitsConfig, RuntimeConfig, @@ -46,6 +49,8 @@ pub(super) struct RawConfig { #[serde(default)] deletion: RawDeletionConfig, #[serde(default)] + deletion_worker: RawDeletionWorkerConfig, + #[serde(default)] paykit: Option, } @@ -129,6 +134,9 @@ impl Default for RawPkdnsConfig { impl RawPkdnsConfig { fn into_pkdns_config(self) -> Result { + if self.key_republisher_interval_seconds == 0 { + return Err(ConfigError::InvalidPkarrRepublisherInterval); + } Ok(PkdnsConfig { public_ip: self.public_ip, public_pubky_tls_port: self.public_pubky_tls_port, @@ -298,6 +306,72 @@ impl Default for RawDeletionConfig { } } +#[derive(Debug, Deserialize)] +#[serde(deny_unknown_fields)] +struct RawDeletionWorkerConfig { + #[serde(default = "default_deletion_worker_enabled")] + enabled: bool, + #[serde(default = "default_deletion_worker_poll_interval_ms")] + poll_interval_ms: u64, + #[serde(default = "default_deletion_worker_claim_timeout_seconds")] + claim_timeout_seconds: u64, + #[serde(default = "default_deletion_worker_shutdown_timeout_seconds")] + shutdown_timeout_seconds: u64, + #[serde(default = "default_deletion_worker_id")] + worker_id: String, +} + +impl Default for RawDeletionWorkerConfig { + fn default() -> Self { + Self { + enabled: default_deletion_worker_enabled(), + poll_interval_ms: default_deletion_worker_poll_interval_ms(), + claim_timeout_seconds: default_deletion_worker_claim_timeout_seconds(), + shutdown_timeout_seconds: default_deletion_worker_shutdown_timeout_seconds(), + worker_id: default_deletion_worker_id(), + } + } +} + +impl RawDeletionWorkerConfig { + fn into_deletion_worker_config(self) -> Result { + if self.poll_interval_ms == 0 + || self.claim_timeout_seconds == 0 + || self.shutdown_timeout_seconds == 0 + || self.worker_id.trim().is_empty() + { + return Err(ConfigError::InvalidDeletionWorkerConfig); + } + Ok(DeletionWorkerConfig { + enabled: self.enabled, + poll_interval_ms: self.poll_interval_ms, + claim_timeout_seconds: self.claim_timeout_seconds, + shutdown_timeout_seconds: self.shutdown_timeout_seconds, + worker_id: self.worker_id, + }) + } +} + +fn default_deletion_worker_enabled() -> bool { + true +} + +fn default_deletion_worker_poll_interval_ms() -> u64 { + DEFAULT_DELETION_WORKER_POLL_INTERVAL_MS +} + +fn default_deletion_worker_claim_timeout_seconds() -> u64 { + DEFAULT_DELETION_WORKER_CLAIM_TIMEOUT_SECONDS +} + +fn default_deletion_worker_shutdown_timeout_seconds() -> u64 { + DEFAULT_DELETION_WORKER_SHUTDOWN_TIMEOUT_SECONDS +} + +fn default_deletion_worker_id() -> String { + DEFAULT_DELETION_WORKER_ID.to_owned() +} + fn default_deletion_retry_max_attempts() -> u32 { DEFAULT_DELETION_RETRY_MAX_ATTEMPTS } @@ -594,6 +668,7 @@ impl RawConfig { let rate_limits = self.rate_limits.into_rate_limits_config()?; let content_locks = self.content_locks.into_content_locks_config()?; let deletion = self.deletion.into_deletion_config()?; + let deletion_worker = self.deletion_worker.into_deletion_worker_config()?; let paykit = self .paykit .map(RawPaykitConfig::into_paykit_config) @@ -638,6 +713,7 @@ impl RawConfig { rate_limits, content_locks, deletion, + deletion_worker, paykit, }) } diff --git a/locks-server/src/config/schema.rs b/locks-server/src/config/schema.rs index 1e3b772..6cd437f 100644 --- a/locks-server/src/config/schema.rs +++ b/locks-server/src/config/schema.rs @@ -7,8 +7,11 @@ use thiserror::Error; use super::defaults::{ DEFAULT_DELETION_RETRY_INITIAL_BACKOFF_SECONDS, DEFAULT_DELETION_RETRY_MAX_ATTEMPTS, - DEFAULT_DELETION_RETRY_MAX_BACKOFF_SECONDS, DEFAULT_FINAL_CREDENTIAL_ISSUANCE_WINDOW_SECONDS, - DEFAULT_FINAL_READ_WINDOW_SECONDS, DEFAULT_RUNTIME_MASTER_KEY_ENV, + DEFAULT_DELETION_RETRY_MAX_BACKOFF_SECONDS, DEFAULT_DELETION_WORKER_CLAIM_TIMEOUT_SECONDS, + DEFAULT_DELETION_WORKER_ID, DEFAULT_DELETION_WORKER_POLL_INTERVAL_MS, + DEFAULT_DELETION_WORKER_SHUTDOWN_TIMEOUT_SECONDS, + DEFAULT_FINAL_CREDENTIAL_ISSUANCE_WINDOW_SECONDS, DEFAULT_FINAL_READ_WINDOW_SECONDS, + DEFAULT_RUNTIME_MASTER_KEY_ENV, }; pub const PAYKIT_CONNECT_TIMEOUT_SECONDS: u64 = 5; @@ -29,6 +32,7 @@ pub struct LockServerRuntimeConfig { pub rate_limits: RateLimitsConfig, pub content_locks: ContentLocksConfig, pub deletion: DeletionConfig, + pub deletion_worker: DeletionWorkerConfig, pub paykit: Option, } @@ -56,6 +60,27 @@ impl Default for DeletionConfig { } } +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct DeletionWorkerConfig { + pub enabled: bool, + pub poll_interval_ms: u64, + pub claim_timeout_seconds: u64, + pub shutdown_timeout_seconds: u64, + pub worker_id: String, +} + +impl Default for DeletionWorkerConfig { + fn default() -> Self { + Self { + enabled: true, + poll_interval_ms: DEFAULT_DELETION_WORKER_POLL_INTERVAL_MS, + claim_timeout_seconds: DEFAULT_DELETION_WORKER_CLAIM_TIMEOUT_SECONDS, + shutdown_timeout_seconds: DEFAULT_DELETION_WORKER_SHUTDOWN_TIMEOUT_SECONDS, + worker_id: DEFAULT_DELETION_WORKER_ID.to_owned(), + } + } +} + #[derive(Debug, Clone, PartialEq, Eq)] pub struct PaykitConfig { pub server_url: String, @@ -349,6 +374,10 @@ pub enum ConfigError { InvalidDeletionRetryBackoffOrder, #[error("deletion credential windows must be between 1 and 3600 seconds")] InvalidDeletionCredentialWindow, + #[error( + "deletion_worker numeric settings must be greater than zero and deletion_worker.worker_id must not be blank" + )] + InvalidDeletionWorkerConfig, #[error( "creator_authority_acquisition.allowed_return_origins must contain http(s) origins without path, query, or fragment: {0}" )] @@ -357,6 +386,8 @@ pub enum ConfigError { "creator_authority_acquisition.allowed_return_origins must not be \"*\" when runtime.environment is production; list explicit origins" )] WildcardReturnOriginInProduction, + #[error("pkdns.key_republisher_interval_seconds must be greater than zero")] + InvalidPkarrRepublisherInterval, #[error("pkdns.pkarr_relays must contain valid http(s) URLs: {0}")] InvalidPkarrRelayUrl(String), #[error("paykit.server_url must be a valid http(s) URL: {0}")] diff --git a/locks-server/src/deletion_worker.rs b/locks-server/src/deletion_worker.rs new file mode 100644 index 0000000..7a56bb8 --- /dev/null +++ b/locks-server/src/deletion_worker.rs @@ -0,0 +1,1712 @@ +use std::time::Duration as StdDuration; + +use async_trait::async_trait; +use locks_service::application::{ + errors::ApplicationError, + models::{ClaimedContentLockDeletionJob, ContentLockDeletionFailureCode}, + ports::{Clock, ContentLockDeletionRepository}, + use_cases::{ + drain_lock_payments::DrainLockPaymentsUseCase, + execute_content_lock_deletion_phase::{ + ContentLockDeletionPhaseExecutor, ContentLockDeletionPhaseExecutorConfig, + ContentLockDeletionPhaseExecutorDependencies, ContentLockPaymentDrainExecutor, + DeletionDependencyEvidence, DeletionDependencySource, DeletionDependencyStatus, + DeletionPhaseExecution, DeletionPhaseExecutionOutcome, + }, + execute_forced_content_lock_deletion::{ + ExecuteForcedContentLockDeletionDependencies, ExecuteForcedContentLockDeletionUseCase, + ForcedContentLockDeletionOutcome, + }, + materialize_final_credentials::MaterializeFinalCredentialsUseCase, + no_paykit_deletion_drain::NoPaykitDeletionDrainUseCase, + }, +}; +use rand::Rng; +use time::{Duration, OffsetDateTime}; +use tokio::sync::watch; +use tracing::error; + +use crate::app_state::{AppState, WorkerKind, WorkerReadiness, WorkerReadinessEvidence}; + +/// Executes at most one bounded action for an already claimed deletion job. +#[async_trait] +pub trait ClaimedDeletionExecutor: Send + Sync { + async fn execute_claimed( + &self, + claim: ClaimedContentLockDeletionJob, + worker_id: &str, + ) -> DeletionPhaseExecution; +} + +#[derive(Clone)] +pub struct RuntimeClaimedDeletionExecutor { + state: AppState, +} + +impl RuntimeClaimedDeletionExecutor { + pub fn new(state: AppState) -> Self { + Self { state } + } + + async fn execute_graceful( + &self, + claim: ClaimedContentLockDeletionJob, + worker_id: &str, + ) -> DeletionPhaseExecution { + let materializer = MaterializeFinalCredentialsUseCase::new( + self.state.access_credentials().as_ref(), + self.state.credential_generator().as_ref(), + self.state.clock().as_ref(), + ); + let no_paykit = NoPaykitDeletionDrainUseCase::new( + self.state.content_lock_deletions().as_ref(), + self.state.clock().as_ref(), + ); + let real_paykit = match ( + self.state.payment_drains(), + self.state.payment_drain_client(), + self.state.config().paykit.as_ref(), + ) { + (Some(drains), Some(client), Some(config)) => Some(DrainLockPaymentsUseCase::new( + self.state.content_lock_deletions().as_ref(), + drains.as_ref(), + client.as_ref(), + self.state.entitlements().as_ref(), + self.state.clock().as_ref(), + self.state + .config() + .credentials + .lock_server_public_key + .clone(), + config.minimum_confirmations, + )), + _ => None, + }; + let payments: &dyn ContentLockPaymentDrainExecutor = + real_paykit.as_ref().map_or(&no_paykit, |drain| { + drain as &dyn ContentLockPaymentDrainExecutor + }); + let config = &self.state.config().deletion; + ContentLockDeletionPhaseExecutor::new( + ContentLockDeletionPhaseExecutorDependencies { + deletions: self.state.content_lock_deletions().as_ref(), + action_ownership: self.state.content_lock_deletion_action_ownership().as_ref(), + tombstones: self.state.content_lock_tombstones().as_ref(), + guarded_resources: self.state.guarded_resources().as_ref(), + access_credentials: self.state.access_credentials().as_ref(), + clock: self.state.clock().as_ref(), + payments, + final_credentials: &materializer, + }, + ContentLockDeletionPhaseExecutorConfig { + final_credential_issuance_window: Duration::seconds( + config.final_credential_issuance_window_seconds as i64, + ), + final_read_window: Duration::seconds(config.final_read_window_seconds as i64), + final_credential_batch_limit: 64, + }, + ) + .execute_with_evidence(claim, worker_id) + .await + } +} + +#[async_trait] +impl ClaimedDeletionExecutor for RuntimeClaimedDeletionExecutor { + async fn execute_claimed( + &self, + claim: ClaimedContentLockDeletionJob, + worker_id: &str, + ) -> DeletionPhaseExecution { + if claim.job.force_requested_at.is_some() { + let execution = ExecuteForcedContentLockDeletionUseCase::new( + ExecuteForcedContentLockDeletionDependencies { + action_ownership: self.state.content_lock_deletion_action_ownership().as_ref(), + tombstones: self.state.content_lock_tombstones().as_ref(), + guarded_resources: self.state.guarded_resources().as_ref(), + deletions: self.state.content_lock_deletions().as_ref(), + clock: self.state.clock().as_ref(), + }, + ) + .execute_with_evidence(claim, worker_id) + .await; + let phase_outcome = match execution.outcome { + ForcedContentLockDeletionOutcome::Completed => { + DeletionPhaseExecutionOutcome::Progressed + } + ForcedContentLockDeletionOutcome::Deferred => { + DeletionPhaseExecutionOutcome::Deferred + } + ForcedContentLockDeletionOutcome::ClaimLost => { + DeletionPhaseExecutionOutcome::ClaimLost + } + ForcedContentLockDeletionOutcome::TransientDependencyFailure => { + DeletionPhaseExecutionOutcome::TransientDependencyFailure + } + ForcedContentLockDeletionOutcome::FatalFailure => { + DeletionPhaseExecutionOutcome::FatalFailure + } + }; + return DeletionPhaseExecution::new(phase_outcome).with_evidence(execution.evidence); + } + self.execute_graceful(claim, worker_id).await + } +} + +/// Supplies a full-jitter delay in the inclusive range from zero through `cap`. +pub trait FullJitterSource: Send + Sync { + fn sample(&self, cap: StdDuration) -> StdDuration; +} + +/// Production full-jitter source. +#[derive(Debug, Default, Clone, Copy)] +pub struct RandomFullJitter; + +impl FullJitterSource for RandomFullJitter { + fn sample(&self, cap: StdDuration) -> StdDuration { + let cap_nanos = cap.as_nanos(); + if cap_nanos == 0 { + return StdDuration::ZERO; + } + let sampled = rand::thread_rng().gen_range(0..=cap_nanos); + StdDuration::new( + (sampled / 1_000_000_000).try_into().unwrap_or(u64::MAX), + (sampled % 1_000_000_000) as u32, + ) + } +} + +/// Runtime policy for the deletion worker core. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct DeletionWorkerConfig { + pub worker_id: String, + pub poll_interval: StdDuration, + pub claim_timeout: StdDuration, + pub retry_max_attempts: u32, + pub retry_initial_backoff: StdDuration, + pub retry_max_backoff: StdDuration, +} + +/// Secret-free result of one worker iteration. It intentionally carries no job IDs or paths. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum DeletionWorkerOutcome { + Idle, + Cancelled, + Progressed, + Deferred, + ClaimLost, + TerminalFailed, + RetryScheduled, + RetryExhausted, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +struct DeletionWorkerExecution { + outcome: DeletionWorkerOutcome, + evidence: DeletionDependencyEvidence, +} + +impl DeletionWorkerExecution { + fn new(outcome: DeletionWorkerOutcome, evidence: DeletionDependencyEvidence) -> Self { + Self { outcome, evidence } + } +} + +struct DeletionWorkerFailure { + error: ApplicationError, + evidence: DeletionDependencyEvidence, +} + +impl DeletionWorkerFailure { + fn repository( + error: ApplicationError, + source: DeletionDependencySource, + prior_evidence: DeletionDependencyEvidence, + ) -> Self { + Self { + error, + evidence: prior_evidence.merge(DeletionDependencyEvidence::unavailable(source)), + } + } + + fn fatal(error: ApplicationError) -> Self { + Self { + error, + evidence: DeletionDependencyEvidence::none(), + } + } +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum PollErrorDisposition { + RetryableRepository, + Fatal(&'static str), +} + +fn classify_poll_error(error: &ApplicationError) -> PollErrorDisposition { + match error { + ApplicationError::Storage { .. } => PollErrorDisposition::RetryableRepository, + ApplicationError::InvalidContentLockDeletionState { .. } => { + PollErrorDisposition::Fatal("invalid_deletion_state") + } + _ => PollErrorDisposition::Fatal("unexpected_application_error"), + } +} + +fn redacted_fatal_poll_error(class: &'static str) -> ApplicationError { + ApplicationError::InvalidContentLockDeletionState { + message: format!("deletion worker terminated after {class}"), + } +} + +#[derive(Debug, Default)] +struct DeletionReadinessRecovery { + degraded: [bool; DeletionDependencySource::ALL.len()], +} + +impl DeletionReadinessRecovery { + fn record_outcome( + &mut self, + outcome: DeletionWorkerOutcome, + evidence: DeletionDependencyEvidence, + readiness: &WorkerReadiness, + ) { + if outcome == DeletionWorkerOutcome::Cancelled { + return; + } + for source in DeletionDependencySource::ALL { + match evidence.status(source) { + Some(DeletionDependencyStatus::Healthy) => self.degraded[source.index()] = false, + Some(DeletionDependencyStatus::Unavailable) => self.degraded[source.index()] = true, + None => {} + } + } + readiness.record( + WorkerKind::Deletion, + if self.degraded.iter().any(|degraded| *degraded) { + WorkerReadinessEvidence::TransientDependencyFailure + } else { + WorkerReadinessEvidence::DependencySucceeded + }, + ); + } + + #[cfg(test)] + fn record_repository_failure( + &mut self, + source: DeletionDependencySource, + readiness: &WorkerReadiness, + ) { + self.record_outcome( + DeletionWorkerOutcome::RetryScheduled, + DeletionDependencyEvidence::unavailable(source), + readiness, + ); + } +} + +/// Polling, retry, and cancellation core for content-lock deletion jobs. +pub struct DeletionWorker<'a> { + deletions: &'a dyn ContentLockDeletionRepository, + clock: &'a dyn Clock, + executor: &'a dyn ClaimedDeletionExecutor, + jitter: &'a dyn FullJitterSource, + config: DeletionWorkerConfig, +} + +impl<'a> DeletionWorker<'a> { + pub fn new( + deletions: &'a dyn ContentLockDeletionRepository, + clock: &'a dyn Clock, + executor: &'a dyn ClaimedDeletionExecutor, + jitter: &'a dyn FullJitterSource, + config: DeletionWorkerConfig, + ) -> Self { + Self { + deletions, + clock, + executor, + jitter, + config, + } + } + + /// Runs at most one claim. A sticky shutdown signal is checked before claiming and + /// again after claim acquisition, before any external deletion action can begin. + pub async fn run_once( + &self, + shutdown: &watch::Receiver, + ) -> Result { + self.run_once_with_evidence(shutdown) + .await + .map(|execution| execution.outcome) + .map_err(|failure| failure.error) + } + + async fn run_once_with_evidence( + &self, + shutdown: &watch::Receiver, + ) -> Result { + if *shutdown.borrow() { + return Ok(DeletionWorkerExecution::new( + DeletionWorkerOutcome::Cancelled, + DeletionDependencyEvidence::none(), + )); + } + + let claim_started_at = self.clock.now(); + let claim_expires_at = claim_started_at + to_time_duration(self.config.claim_timeout); + let repository_healthy = + DeletionDependencyEvidence::healthy(DeletionDependencySource::RepositoryQueueClaim); + let Some(claim) = self + .deletions + .claim_next( + &self.config.worker_id, + (claim_expires_at) - (claim_started_at), + ) + .await + .map_err(|error| { + DeletionWorkerFailure::repository( + error, + DeletionDependencySource::RepositoryQueueClaim, + DeletionDependencyEvidence::none(), + ) + })? + else { + return Ok(DeletionWorkerExecution::new( + DeletionWorkerOutcome::Idle, + repository_healthy, + )); + }; + + if *shutdown.borrow() { + let outcome = self + .release_cancelled_claim(&claim) + .await + .map_err(|error| { + DeletionWorkerFailure::repository( + error, + DeletionDependencySource::RepositoryDefer, + repository_healthy, + ) + })?; + let mutation_evidence = + mutation_success_evidence(outcome, DeletionDependencySource::RepositoryDefer); + return Ok(DeletionWorkerExecution::new( + outcome, + repository_healthy.merge(mutation_evidence), + )); + } + + let claim_for_write = claim.clone(); + let execution = self + .executor + .execute_claimed(claim, &self.config.worker_id) + .await; + let mut mutation_evidence = DeletionDependencyEvidence::none(); + let outcome = match execution.outcome { + DeletionPhaseExecutionOutcome::Progressed => DeletionWorkerOutcome::Progressed, + DeletionPhaseExecutionOutcome::TerminalFailed => DeletionWorkerOutcome::TerminalFailed, + DeletionPhaseExecutionOutcome::ClaimLost => DeletionWorkerOutcome::ClaimLost, + DeletionPhaseExecutionOutcome::Deferred => { + let outcome = self.defer_claim(&claim_for_write).await.map_err(|error| { + DeletionWorkerFailure::repository( + error, + DeletionDependencySource::RepositoryDefer, + repository_healthy.merge(execution.evidence), + ) + })?; + mutation_evidence = + mutation_success_evidence(outcome, DeletionDependencySource::RepositoryDefer); + outcome + } + DeletionPhaseExecutionOutcome::TransientDependencyFailure => { + let source = if claim_for_write.job.attempt_count >= self.config.retry_max_attempts + { + DeletionDependencySource::RepositoryTerminalMutation + } else { + DeletionDependencySource::RepositoryRetry + }; + let outcome = self + .retry_or_exhaust(claim_for_write) + .await + .map_err(|error| { + DeletionWorkerFailure::repository( + error, + source, + repository_healthy.merge(execution.evidence), + ) + })?; + mutation_evidence = mutation_success_evidence(outcome, source); + outcome + } + DeletionPhaseExecutionOutcome::FatalFailure => { + return Err(DeletionWorkerFailure::fatal(redacted_fatal_poll_error( + "fatal_execution_failure", + ))); + } + }; + Ok(DeletionWorkerExecution::new( + outcome, + execution + .evidence + .merge(repository_healthy) + .merge(mutation_evidence), + )) + } + + /// Runs until shutdown, sleeping only after idle polls. Shutdown prevents every future claim, + /// including when it arrives while claim acquisition is blocked. + pub async fn run_until_shutdown( + &self, + mut shutdown: watch::Receiver, + ) -> Result<(), ApplicationError> { + loop { + if *shutdown.borrow() { + return Ok(()); + } + + match self.run_once(&shutdown).await? { + DeletionWorkerOutcome::Cancelled => return Ok(()), + DeletionWorkerOutcome::Idle => { + tokio::select! { + changed = shutdown.changed() => { + if changed.is_err() || *shutdown.borrow() { + return Ok(()); + } + } + _ = tokio::time::sleep(self.config.poll_interval) => {} + } + } + DeletionWorkerOutcome::Progressed + | DeletionWorkerOutcome::Deferred + | DeletionWorkerOutcome::ClaimLost + | DeletionWorkerOutcome::TerminalFailed + | DeletionWorkerOutcome::RetryScheduled + | DeletionWorkerOutcome::RetryExhausted => {} + } + } + } + + pub async fn run_until_shutdown_with_readiness( + &self, + mut shutdown: watch::Receiver, + readiness: &WorkerReadiness, + ) -> Result<(), ApplicationError> { + let mut recovery = DeletionReadinessRecovery::default(); + loop { + if *shutdown.borrow() { + readiness.record(WorkerKind::Deletion, WorkerReadinessEvidence::Stopped); + return Ok(()); + } + + match self.run_once_with_evidence(&shutdown).await { + Ok(DeletionWorkerExecution { + outcome: DeletionWorkerOutcome::Cancelled, + .. + }) => { + readiness.record(WorkerKind::Deletion, WorkerReadinessEvidence::Stopped); + return Ok(()); + } + Ok( + execution @ DeletionWorkerExecution { + outcome: DeletionWorkerOutcome::Idle, + .. + }, + ) => { + recovery.record_outcome(execution.outcome, execution.evidence, readiness); + tokio::select! { + _ = shutdown.changed() => {} + _ = tokio::time::sleep(self.config.poll_interval) => {} + } + } + Ok(execution) => { + recovery.record_outcome(execution.outcome, execution.evidence, readiness) + } + Err(failure) => match classify_poll_error(&failure.error) { + PollErrorDisposition::RetryableRepository => { + recovery.record_outcome( + DeletionWorkerOutcome::RetryScheduled, + failure.evidence, + readiness, + ); + error!( + operation = "deletion_queue_poll_or_transition", + error_class = "repository_unavailable", + retrying = true, + "deletion worker repository operation failed" + ); + tokio::select! { + _ = shutdown.changed() => {} + _ = tokio::time::sleep(self.config.poll_interval) => {} + } + } + PollErrorDisposition::Fatal(error_class) => { + readiness.record( + WorkerKind::Deletion, + WorkerReadinessEvidence::UnexpectedExit, + ); + error!( + operation = "deletion_queue_poll_or_transition", + error_class, + retrying = false, + "deletion worker terminated after unexpected application error" + ); + return Err(redacted_fatal_poll_error(error_class)); + } + }, + } + } + } + + async fn release_cancelled_claim( + &self, + claim: &ClaimedContentLockDeletionJob, + ) -> Result { + let now = self.clock.now(); + self.defer_at(claim, now, now, DeletionWorkerOutcome::Cancelled) + .await + } + + async fn defer_claim( + &self, + claim: &ClaimedContentLockDeletionJob, + ) -> Result { + let now = self.clock.now(); + let due = now + to_time_duration(self.config.poll_interval); + self.defer_at(claim, now, due, DeletionWorkerOutcome::Deferred) + .await + } + + async fn defer_at( + &self, + claim: &ClaimedContentLockDeletionJob, + now: OffsetDateTime, + due: OffsetDateTime, + success: DeletionWorkerOutcome, + ) -> Result { + let updated = self + .deletions + .defer( + claim.job.job_id, + &self.config.worker_id, + claim.claim_token, + (due) - (now), + ) + .await?; + Ok(if updated.is_some() { + success + } else { + DeletionWorkerOutcome::ClaimLost + }) + } + + async fn retry_or_exhaust( + &self, + claim: ClaimedContentLockDeletionJob, + ) -> Result { + let now = self.clock.now(); + if claim.job.attempt_count >= self.config.retry_max_attempts { + let updated = self + .deletions + .finish( + claim.job.job_id, + &self.config.worker_id, + claim.claim_token, + Some(ContentLockDeletionFailureCode::RetryExhausted), + ) + .await?; + return Ok(if updated.is_some() { + DeletionWorkerOutcome::RetryExhausted + } else { + DeletionWorkerOutcome::ClaimLost + }); + } + + let cap = retry_cap( + self.config.retry_initial_backoff, + self.config.retry_max_backoff, + claim.job.attempt_count, + ); + // Keep repository scheduling bounded even if a custom source violates its contract. + let delay = self.jitter.sample(cap).min(cap); + let next_attempt_at = now + to_time_duration(delay); + let updated = self + .deletions + .schedule_retry( + claim.job.job_id, + &self.config.worker_id, + claim.claim_token, + (next_attempt_at) - (now), + ) + .await?; + Ok(if updated.is_some() { + DeletionWorkerOutcome::RetryScheduled + } else { + DeletionWorkerOutcome::ClaimLost + }) + } +} + +fn mutation_success_evidence( + outcome: DeletionWorkerOutcome, + source: DeletionDependencySource, +) -> DeletionDependencyEvidence { + if outcome == DeletionWorkerOutcome::ClaimLost { + DeletionDependencyEvidence::none() + } else { + DeletionDependencyEvidence::healthy(source) + } +} + +fn retry_cap(initial: StdDuration, maximum: StdDuration, attempt_count: u32) -> StdDuration { + let mut cap = initial.min(maximum); + for _ in 1..attempt_count { + cap = cap.checked_mul(2).unwrap_or(maximum).min(maximum); + if cap == maximum { + break; + } + } + cap +} + +fn to_time_duration(duration: StdDuration) -> Duration { + Duration::new( + i64::try_from(duration.as_secs()).unwrap_or(i64::MAX), + duration.subsec_nanos() as i32, + ) +} + +#[cfg(test)] +mod tests { + use std::{ + collections::BTreeMap, + str::FromStr, + sync::{ + Arc, Mutex, + atomic::{AtomicBool, AtomicUsize, Ordering}, + }, + time::Duration as StdDuration, + }; + + use async_trait::async_trait; + use locks_core::{ + ids::{CreatorPubky, GuardedResourceHash, LockId}, + lock_policy::{ + AccessPolicy, CONTENT_LOCK_VERSION, ContentLock, GuardedResource, LockLogic, + LockServerConfig, + }, + }; + use locks_service::application::{ + errors::ApplicationError, + models::{ + AdvanceContentLockDeletionPhaseResult, ClaimedContentLockDeletionJob, + ContentLockDeletionFailureCode, ContentLockDeletionJob, ContentLockDeletionPhase, + ContentLockDeletionState, PrepareForceDeletionResult, + }, + ports::{Clock, ContentLockDeletionRepository}, + use_cases::execute_content_lock_deletion_phase::{ + DeletionDependencyEvidence, DeletionDependencySource, DeletionDependencyStatus, + DeletionPhaseExecution, DeletionPhaseExecutionOutcome, + }, + }; + use time::{OffsetDateTime, macros::datetime}; + use tokio::sync::{Notify, watch}; + use uuid::Uuid; + + use super::{ + ClaimedDeletionExecutor, DeletionReadinessRecovery, DeletionWorker, DeletionWorkerConfig, + DeletionWorkerOutcome, FullJitterSource, PollErrorDisposition, classify_poll_error, + retry_cap, + }; + use crate::app_state::{ReadinessStatus, WorkerKind, WorkerReadiness, WorkerReadinessState}; + + const NOW: OffsetDateTime = datetime!(2026-08-17 12:00:00 UTC); + const CREATOR: &str = "pubkytkrq8zmwb8a3m9k15csu3q17qmfgqnp9dskbrg9uq1rydpyxp7qy"; + + #[tokio::test] + async fn active_shutdown_performs_no_claim() { + let repository = FakeRepository::default(); + let executor = FakeExecutor::new(DeletionPhaseExecutionOutcome::Progressed); + let worker = worker(&repository, &executor, &FixedJitter(StdDuration::ZERO)); + let (shutdown_tx, shutdown_rx) = watch::channel(false); + shutdown_tx.send(true).unwrap(); + + assert_eq!( + worker.run_once(&shutdown_rx).await.unwrap(), + DeletionWorkerOutcome::Cancelled + ); + assert_eq!(repository.claim_calls.load(Ordering::SeqCst), 0); + assert_eq!(executor.calls.load(Ordering::SeqCst), 0); + } + + #[tokio::test] + async fn shutdown_while_claim_is_blocked_releases_claim_without_external_work() { + let repository = Arc::new(FakeRepository::with_claim(claim(1))); + repository.block_claim.store(true, Ordering::SeqCst); + let executor = Arc::new(FakeExecutor::new(DeletionPhaseExecutionOutcome::Progressed)); + let jitter = Arc::new(FixedJitter(StdDuration::ZERO)); + let worker = Arc::new(OwnedWorker::new( + Arc::clone(&repository), + Arc::clone(&executor), + jitter, + )); + let (shutdown_tx, shutdown_rx) = watch::channel(false); + let run = { + let worker = Arc::clone(&worker); + tokio::spawn(async move { worker.run_once(&shutdown_rx).await }) + }; + repository.claim_entered.notified().await; + shutdown_tx.send(true).unwrap(); + repository.claim_release.notify_one(); + + assert_eq!( + run.await.unwrap().unwrap(), + DeletionWorkerOutcome::Cancelled + ); + assert_eq!(executor.calls.load(Ordering::SeqCst), 0); + let defers = repository.defers.lock().unwrap(); + assert_eq!(defers.len(), 1); + assert_eq!(defers[0].0, NOW); + assert_eq!(defers[0].1, NOW); + } + + #[tokio::test] + async fn shutdown_during_first_claim_prevents_a_second_claim() { + let repository = Arc::new(FakeRepository::with_claim(claim(1))); + repository.block_claim.store(true, Ordering::SeqCst); + let executor = Arc::new(FakeExecutor::new(DeletionPhaseExecutionOutcome::Progressed)); + let worker = Arc::new(OwnedWorker::new( + Arc::clone(&repository), + executor, + Arc::new(FixedJitter(StdDuration::ZERO)), + )); + let (shutdown_tx, shutdown_rx) = watch::channel(false); + let run = { + let worker = Arc::clone(&worker); + tokio::spawn(async move { worker.run_until_shutdown(shutdown_rx).await }) + }; + repository.claim_entered.notified().await; + shutdown_tx.send(true).unwrap(); + repository.claim_release.notify_one(); + + run.await.unwrap().unwrap(); + assert_eq!(repository.claim_calls.load(Ordering::SeqCst), 1); + } + + #[tokio::test] + async fn deferred_work_releases_claim_on_poll_schedule_without_retry_write() { + let repository = FakeRepository::with_claim(claim(3)); + let executor = FakeExecutor::new(DeletionPhaseExecutionOutcome::Deferred); + let worker = worker(&repository, &executor, &FixedJitter(StdDuration::ZERO)); + let (_shutdown_tx, shutdown_rx) = watch::channel(false); + + assert_eq!( + worker.run_once(&shutdown_rx).await.unwrap(), + DeletionWorkerOutcome::Deferred + ); + assert!(repository.retries.lock().unwrap().is_empty()); + assert!(repository.finishes.lock().unwrap().is_empty()); + assert_eq!( + repository.defers.lock().unwrap().as_slice(), + &[(NOW, NOW + time::Duration::seconds(5))] + ); + } + + #[tokio::test] + async fn completed_terminal_and_lost_executor_outcomes_do_not_write_retry_state() { + for (execution, expected) in [ + ( + DeletionPhaseExecutionOutcome::Progressed, + DeletionWorkerOutcome::Progressed, + ), + ( + DeletionPhaseExecutionOutcome::TerminalFailed, + DeletionWorkerOutcome::TerminalFailed, + ), + ( + DeletionPhaseExecutionOutcome::ClaimLost, + DeletionWorkerOutcome::ClaimLost, + ), + ] { + let repository = FakeRepository::with_claim(claim(1)); + let executor = FakeExecutor::new(execution); + let worker = worker(&repository, &executor, &FixedJitter(StdDuration::ZERO)); + let (_shutdown_tx, shutdown_rx) = watch::channel(false); + + assert_eq!(worker.run_once(&shutdown_rx).await.unwrap(), expected); + assert!(repository.defers.lock().unwrap().is_empty()); + assert!(repository.retries.lock().unwrap().is_empty()); + assert!(repository.finishes.lock().unwrap().is_empty()); + } + } + + #[tokio::test] + async fn idle_sleep_wakes_immediately_for_shutdown() { + let repository = Arc::new(FakeRepository::default()); + let executor = Arc::new(FakeExecutor::new(DeletionPhaseExecutionOutcome::Progressed)); + let worker = Arc::new(OwnedWorker::new( + Arc::clone(&repository), + executor, + Arc::new(FixedJitter(StdDuration::ZERO)), + )); + let (shutdown_tx, shutdown_rx) = watch::channel(false); + let run = { + let worker = Arc::clone(&worker); + tokio::spawn(async move { worker.run_until_shutdown(shutdown_rx).await }) + }; + while repository.claim_calls.load(Ordering::SeqCst) == 0 { + tokio::task::yield_now().await; + } + + shutdown_tx.send(true).unwrap(); + tokio::time::timeout(StdDuration::from_millis(100), run) + .await + .expect("shutdown must interrupt the poll sleep") + .unwrap() + .unwrap(); + assert_eq!(repository.claim_calls.load(Ordering::SeqCst), 1); + } + + #[tokio::test] + async fn fatal_executor_failure_terminates_supervised_worker_without_retry_write() { + let repository = FakeRepository::with_claim(claim(1)); + let executor = FakeExecutor::new(DeletionPhaseExecutionOutcome::FatalFailure); + let worker = worker(&repository, &executor, &FixedJitter(StdDuration::ZERO)); + let readiness = WorkerReadiness::new(false, true); + let (_shutdown_tx, shutdown_rx) = watch::channel(false); + + let error = worker + .run_until_shutdown_with_readiness(shutdown_rx, &readiness) + .await + .unwrap_err(); + + assert_eq!(readiness.status(), ReadinessStatus::NotReady); + assert!(error.to_string().contains("invalid_deletion_state")); + assert!(repository.retries.lock().unwrap().is_empty()); + assert!(repository.finishes.lock().unwrap().is_empty()); + } + + #[tokio::test] + async fn transient_executor_failure_schedules_retry_and_marks_dependency_degraded() { + let repository = FakeRepository::with_claim(claim(1)); + let executor = FakeExecutor::new(DeletionPhaseExecutionOutcome::TransientDependencyFailure); + let worker = worker(&repository, &executor, &FixedJitter(StdDuration::ZERO)); + let (_shutdown_tx, shutdown_rx) = watch::channel(false); + + assert_eq!( + worker.run_once(&shutdown_rx).await.unwrap(), + DeletionWorkerOutcome::RetryScheduled + ); + assert_eq!(repository.retries.lock().unwrap().len(), 1); + } + + #[tokio::test] + async fn dependency_failure_exhausts_on_exact_max_attempt() { + let repository = FakeRepository::with_claim(claim(4)); + let executor = FakeExecutor::new(DeletionPhaseExecutionOutcome::TransientDependencyFailure); + let worker = worker(&repository, &executor, &FixedJitter(StdDuration::ZERO)); + let (_shutdown_tx, shutdown_rx) = watch::channel(false); + + assert_eq!( + worker.run_once(&shutdown_rx).await.unwrap(), + DeletionWorkerOutcome::RetryExhausted + ); + assert!(repository.retries.lock().unwrap().is_empty()); + assert_eq!( + repository.finishes.lock().unwrap().as_slice(), + &[Some(ContentLockDeletionFailureCode::RetryExhausted)] + ); + } + + #[tokio::test] + async fn full_jitter_is_bounded_by_exponential_cap() { + assert_eq!( + retry_cap(StdDuration::from_secs(10), StdDuration::from_secs(25), 1), + StdDuration::from_secs(10) + ); + assert_eq!( + retry_cap(StdDuration::from_secs(10), StdDuration::from_secs(25), 2), + StdDuration::from_secs(20) + ); + assert_eq!( + retry_cap(StdDuration::from_secs(10), StdDuration::from_secs(25), 3), + StdDuration::from_secs(25) + ); + + let repository = FakeRepository::with_claim(claim(2)); + let executor = FakeExecutor::new(DeletionPhaseExecutionOutcome::TransientDependencyFailure); + let jitter = FixedJitter(StdDuration::from_secs(99)); + let worker = worker(&repository, &executor, &jitter); + let (_shutdown_tx, shutdown_rx) = watch::channel(false); + worker.run_once(&shutdown_rx).await.unwrap(); + + assert_eq!( + repository.retries.lock().unwrap().as_slice(), + &[(NOW, NOW + time::Duration::seconds(20))] + ); + } + + #[tokio::test] + async fn retry_due_time_uses_jitter_not_poll_interval() { + let repository = FakeRepository::with_claim(claim(1)); + let executor = FakeExecutor::new(DeletionPhaseExecutionOutcome::TransientDependencyFailure); + let jitter = FixedJitter(StdDuration::from_secs(7)); + let worker = worker(&repository, &executor, &jitter); + let (_shutdown_tx, shutdown_rx) = watch::channel(false); + + assert_eq!( + worker.run_once(&shutdown_rx).await.unwrap(), + DeletionWorkerOutcome::RetryScheduled + ); + assert_eq!( + repository.retries.lock().unwrap().as_slice(), + &[(NOW, NOW + time::Duration::seconds(7))] + ); + } + + #[tokio::test] + async fn stale_retry_defer_and_finish_writes_map_to_claim_lost() { + for (execution, attempts) in [ + (DeletionPhaseExecutionOutcome::Deferred, 1), + (DeletionPhaseExecutionOutcome::TransientDependencyFailure, 1), + (DeletionPhaseExecutionOutcome::TransientDependencyFailure, 4), + ] { + let repository = FakeRepository::with_claim(claim(attempts)); + repository.stale_writes.store(true, Ordering::SeqCst); + let executor = FakeExecutor::new(execution); + let worker = worker(&repository, &executor, &FixedJitter(StdDuration::ZERO)); + let (_shutdown_tx, shutdown_rx) = watch::channel(false); + assert_eq!( + worker.run_once(&shutdown_rx).await.unwrap(), + DeletionWorkerOutcome::ClaimLost + ); + } + } + + #[tokio::test] + async fn successful_worker_mutations_emit_only_their_exact_healthy_slot() { + for (execution, attempts, expected_source) in [ + ( + DeletionPhaseExecutionOutcome::Deferred, + 1, + DeletionDependencySource::RepositoryDefer, + ), + ( + DeletionPhaseExecutionOutcome::TransientDependencyFailure, + 1, + DeletionDependencySource::RepositoryRetry, + ), + ( + DeletionPhaseExecutionOutcome::TransientDependencyFailure, + 4, + DeletionDependencySource::RepositoryTerminalMutation, + ), + ] { + let repository = FakeRepository::with_claim(claim(attempts)); + let executor = FakeExecutor::new(execution); + let worker = worker(&repository, &executor, &FixedJitter(StdDuration::ZERO)); + let (_shutdown_tx, shutdown_rx) = watch::channel(false); + + let execution = match worker.run_once_with_evidence(&shutdown_rx).await { + Ok(execution) => execution, + Err(_) => panic!("worker mutation should succeed"), + }; + + assert_eq!( + execution.evidence.status(expected_source), + Some(DeletionDependencyStatus::Healthy) + ); + for skipped in [ + DeletionDependencySource::RepositoryPhaseMutation, + DeletionDependencySource::RepositoryDefer, + DeletionDependencySource::RepositoryRetry, + DeletionDependencySource::RepositoryTerminalMutation, + ] { + if skipped != expected_source { + assert_eq!(execution.evidence.status(skipped), None); + } + } + } + } + + #[tokio::test] + async fn stale_worker_mutations_are_evidence_free_for_the_skipped_slot() { + for (execution, attempts, skipped_source) in [ + ( + DeletionPhaseExecutionOutcome::Deferred, + 1, + DeletionDependencySource::RepositoryDefer, + ), + ( + DeletionPhaseExecutionOutcome::TransientDependencyFailure, + 1, + DeletionDependencySource::RepositoryRetry, + ), + ( + DeletionPhaseExecutionOutcome::TransientDependencyFailure, + 4, + DeletionDependencySource::RepositoryTerminalMutation, + ), + ] { + let repository = FakeRepository::with_claim(claim(attempts)); + repository.stale_writes.store(true, Ordering::SeqCst); + let executor = FakeExecutor::new(execution); + let worker = worker(&repository, &executor, &FixedJitter(StdDuration::ZERO)); + let (_shutdown_tx, shutdown_rx) = watch::channel(false); + + let execution = match worker.run_once_with_evidence(&shutdown_rx).await { + Ok(execution) => execution, + Err(_) => panic!("worker mutation should succeed"), + }; + + assert_eq!(execution.outcome, DeletionWorkerOutcome::ClaimLost); + assert_eq!(execution.evidence.status(skipped_source), None); + } + } + + #[tokio::test] + async fn worker_mutation_failures_report_their_exact_repository_source() { + for (execution, attempts, write_failure, expected_source) in [ + ( + DeletionPhaseExecutionOutcome::Deferred, + 1, + RepositoryWriteFailure::Defer, + DeletionDependencySource::RepositoryDefer, + ), + ( + DeletionPhaseExecutionOutcome::TransientDependencyFailure, + 1, + RepositoryWriteFailure::Retry, + DeletionDependencySource::RepositoryRetry, + ), + ( + DeletionPhaseExecutionOutcome::TransientDependencyFailure, + 4, + RepositoryWriteFailure::Terminal, + DeletionDependencySource::RepositoryTerminalMutation, + ), + ] { + let repository = FakeRepository::with_claim(claim(attempts)); + *repository.write_failure.lock().unwrap() = Some(write_failure); + let executor = FakeExecutor::new(execution); + let worker = worker(&repository, &executor, &FixedJitter(StdDuration::ZERO)); + let (_shutdown_tx, shutdown_rx) = watch::channel(false); + + let failure = match worker.run_once_with_evidence(&shutdown_rx).await { + Ok(_) => panic!("repository mutation should fail"), + Err(failure) => failure, + }; + + assert_eq!( + failure.evidence.status(expected_source), + Some(DeletionDependencyStatus::Unavailable) + ); + for unrelated in [ + DeletionDependencySource::RepositoryDefer, + DeletionDependencySource::RepositoryRetry, + DeletionDependencySource::RepositoryTerminalMutation, + ] { + if unrelated != expected_source { + assert_eq!(failure.evidence.status(unrelated), None); + } + } + } + } + + #[test] + fn healthy_business_deferral_establishes_readiness_without_degradation() { + let readiness = WorkerReadiness::new(false, true); + let mut recovery = DeletionReadinessRecovery::default(); + + recovery.record_outcome( + DeletionWorkerOutcome::Deferred, + DeletionDependencyEvidence::healthy(DeletionDependencySource::RepositoryQueueClaim), + &readiness, + ); + + assert_eq!(readiness.status(), ReadinessStatus::Ready); + assert_eq!( + readiness.worker_state(WorkerKind::Deletion), + WorkerReadinessState::Ready + ); + } + + #[test] + fn paykit_degradation_survives_unrelated_pubky_or_repository_progress() { + let readiness = WorkerReadiness::new(false, true); + let mut recovery = DeletionReadinessRecovery::default(); + + recovery.record_outcome( + DeletionWorkerOutcome::RetryScheduled, + DeletionDependencyEvidence::unavailable(DeletionDependencySource::PaymentProvider), + &readiness, + ); + assert_eq!(readiness.status(), ReadinessStatus::Degraded); + + recovery.record_outcome( + DeletionWorkerOutcome::Progressed, + DeletionDependencyEvidence::healthy(DeletionDependencySource::PubkyReadback).merge( + DeletionDependencyEvidence::healthy(DeletionDependencySource::RepositoryQueueClaim), + ), + &readiness, + ); + + assert_eq!( + readiness.worker_state(WorkerKind::Deletion), + WorkerReadinessState::Degraded + ); + } + + #[test] + fn successful_active_paykit_drain_recovers_readiness_while_deferred() { + let readiness = WorkerReadiness::new(false, true); + let mut recovery = DeletionReadinessRecovery::default(); + + recovery.record_outcome( + DeletionWorkerOutcome::RetryScheduled, + DeletionDependencyEvidence::unavailable(DeletionDependencySource::PaymentProvider), + &readiness, + ); + recovery.record_outcome( + DeletionWorkerOutcome::Deferred, + DeletionDependencyEvidence::healthy(DeletionDependencySource::PaymentProvider).merge( + DeletionDependencyEvidence::healthy(DeletionDependencySource::RepositoryQueueClaim), + ), + &readiness, + ); + + assert_eq!(readiness.status(), ReadinessStatus::Ready); + } + + #[test] + fn dependency_sources_recover_independently() { + let readiness = WorkerReadiness::new(false, true); + let mut recovery = DeletionReadinessRecovery::default(); + + recovery.record_outcome( + DeletionWorkerOutcome::RetryScheduled, + DeletionDependencyEvidence::unavailable(DeletionDependencySource::PaymentProvider) + .merge(DeletionDependencyEvidence::unavailable( + DeletionDependencySource::PubkyReadback, + )), + &readiness, + ); + recovery.record_outcome( + DeletionWorkerOutcome::Progressed, + DeletionDependencyEvidence::healthy(DeletionDependencySource::PubkyReadback), + &readiness, + ); + assert_eq!(readiness.status(), ReadinessStatus::Degraded); + + recovery.record_outcome( + DeletionWorkerOutcome::Deferred, + DeletionDependencyEvidence::healthy(DeletionDependencySource::PaymentProvider), + &readiness, + ); + assert_eq!(readiness.status(), ReadinessStatus::Ready); + } + + #[test] + fn repository_degradation_is_sticky_across_non_recovery_outcomes() { + for non_recovery in [ + DeletionWorkerOutcome::Deferred, + DeletionWorkerOutcome::ClaimLost, + DeletionWorkerOutcome::TerminalFailed, + ] { + let readiness = WorkerReadiness::new(false, true); + let mut recovery = DeletionReadinessRecovery::default(); + + recovery.record_repository_failure( + DeletionDependencySource::RepositoryQueueClaim, + &readiness, + ); + recovery.record_outcome(non_recovery, DeletionDependencyEvidence::none(), &readiness); + assert_eq!(readiness.status(), ReadinessStatus::Degraded); + + recovery.record_outcome( + DeletionWorkerOutcome::Idle, + DeletionDependencyEvidence::healthy(DeletionDependencySource::RepositoryQueueClaim), + &readiness, + ); + assert_eq!(readiness.status(), ReadinessStatus::Ready); + } + } + + #[test] + fn idle_queue_poll_does_not_clear_repository_mutation_degradation() { + let readiness = WorkerReadiness::new(false, true); + let mut recovery = DeletionReadinessRecovery::default(); + + recovery.record_outcome( + DeletionWorkerOutcome::RetryScheduled, + DeletionDependencyEvidence::unavailable( + DeletionDependencySource::RepositoryPhaseMutation, + ), + &readiness, + ); + recovery.record_outcome( + DeletionWorkerOutcome::Idle, + DeletionDependencyEvidence::healthy(DeletionDependencySource::RepositoryQueueClaim), + &readiness, + ); + + assert_eq!(readiness.status(), ReadinessStatus::Degraded); + } + + #[test] + fn forced_contention_does_not_clear_prior_pubky_degradation() { + let readiness = WorkerReadiness::new(false, true); + let mut recovery = DeletionReadinessRecovery::default(); + + recovery.record_outcome( + DeletionWorkerOutcome::RetryScheduled, + DeletionDependencyEvidence::unavailable(DeletionDependencySource::PubkyReadback), + &readiness, + ); + recovery.record_outcome( + DeletionWorkerOutcome::Deferred, + DeletionDependencyEvidence::none(), + &readiness, + ); + + assert_eq!(readiness.status(), ReadinessStatus::Degraded); + } + + #[test] + fn retry_exhaustion_is_dependency_failure_evidence_even_without_prior_retry_tick() { + let readiness = WorkerReadiness::new(false, true); + let mut recovery = DeletionReadinessRecovery::default(); + + recovery.record_outcome( + DeletionWorkerOutcome::RetryExhausted, + DeletionDependencyEvidence::unavailable(DeletionDependencySource::PubkyReadback), + &readiness, + ); + + assert_eq!(readiness.status(), ReadinessStatus::Degraded); + } + + #[test] + fn polling_error_classification_retries_only_storage_failures_without_details() { + let secret = "postgres://user:password@example.test/locks"; + assert_eq!( + classify_poll_error(&ApplicationError::Storage { + message: secret.to_owned(), + }), + PollErrorDisposition::RetryableRepository + ); + assert_eq!( + classify_poll_error(&ApplicationError::InvalidContentLockDeletionState { + message: secret.to_owned(), + }), + PollErrorDisposition::Fatal("invalid_deletion_state") + ); + assert_eq!( + classify_poll_error(&ApplicationError::MissingRecord { + record: "content_lock_deletion", + }), + PollErrorDisposition::Fatal("unexpected_application_error") + ); + + for class in ["invalid_deletion_state", "unexpected_application_error"] { + let redacted = super::redacted_fatal_poll_error(class).to_string(); + assert!(redacted.contains(class)); + assert!(!redacted.contains(secret)); + } + } + + #[tokio::test] + async fn unexpected_poll_error_terminates_worker_not_ready_with_redacted_error() { + let secret = "pubky-secret-path"; + let repository = + FakeRepository::with_claim_error(ApplicationError::InvalidContentLockDeletionState { + message: secret.to_owned(), + }); + let executor = FakeExecutor::new(DeletionPhaseExecutionOutcome::Progressed); + let worker = worker(&repository, &executor, &FixedJitter(StdDuration::ZERO)); + let readiness = WorkerReadiness::new(false, true); + let (_shutdown_tx, shutdown_rx) = watch::channel(false); + + let error = worker + .run_until_shutdown_with_readiness(shutdown_rx, &readiness) + .await + .unwrap_err(); + + assert_eq!(readiness.status(), ReadinessStatus::NotReady); + assert!(error.to_string().contains("invalid_deletion_state")); + assert!(!error.to_string().contains(secret)); + } + + #[test] + fn public_outcome_debug_contains_no_job_identity() { + for outcome in [ + DeletionWorkerOutcome::Idle, + DeletionWorkerOutcome::Cancelled, + DeletionWorkerOutcome::Progressed, + DeletionWorkerOutcome::Deferred, + DeletionWorkerOutcome::ClaimLost, + DeletionWorkerOutcome::TerminalFailed, + DeletionWorkerOutcome::RetryScheduled, + DeletionWorkerOutcome::RetryExhausted, + ] { + let debug = format!("{outcome:?}"); + assert!(!debug.contains("pubky")); + assert!(!debug.contains('/')); + assert!(!debug.contains('-')); + } + } + + fn worker<'a>( + repository: &'a FakeRepository, + executor: &'a FakeExecutor, + jitter: &'a dyn FullJitterSource, + ) -> DeletionWorker<'a> { + DeletionWorker::new(repository, &FIXED_CLOCK, executor, jitter, config()) + } + + fn config() -> DeletionWorkerConfig { + DeletionWorkerConfig { + worker_id: "deletion-worker-test".to_owned(), + poll_interval: StdDuration::from_secs(5), + claim_timeout: StdDuration::from_secs(30), + retry_max_attempts: 4, + retry_initial_backoff: StdDuration::from_secs(10), + retry_max_backoff: StdDuration::from_secs(25), + } + } + + struct OwnedWorker { + repository: Arc, + executor: Arc, + jitter: Arc, + } + + impl OwnedWorker { + fn new( + repository: Arc, + executor: Arc, + jitter: Arc, + ) -> Self { + Self { + repository, + executor, + jitter, + } + } + + async fn run_once( + &self, + shutdown: &watch::Receiver, + ) -> Result { + DeletionWorker::new( + self.repository.as_ref(), + &FIXED_CLOCK, + self.executor.as_ref(), + self.jitter.as_ref(), + config(), + ) + .run_once(shutdown) + .await + } + + async fn run_until_shutdown( + &self, + shutdown: watch::Receiver, + ) -> Result<(), ApplicationError> { + DeletionWorker::new( + self.repository.as_ref(), + &FIXED_CLOCK, + self.executor.as_ref(), + self.jitter.as_ref(), + config(), + ) + .run_until_shutdown(shutdown) + .await + } + } + + struct FixedClock; + static FIXED_CLOCK: FixedClock = FixedClock; + impl Clock for FixedClock { + fn now(&self) -> OffsetDateTime { + NOW + } + } + + struct FixedJitter(StdDuration); + impl FullJitterSource for FixedJitter { + fn sample(&self, _cap: StdDuration) -> StdDuration { + self.0 + } + } + + struct FakeExecutor { + outcome: DeletionPhaseExecutionOutcome, + calls: AtomicUsize, + } + + impl FakeExecutor { + fn new(outcome: DeletionPhaseExecutionOutcome) -> Self { + Self { + outcome, + calls: AtomicUsize::new(0), + } + } + } + + #[async_trait] + impl ClaimedDeletionExecutor for FakeExecutor { + async fn execute_claimed( + &self, + _claim: ClaimedContentLockDeletionJob, + _worker_id: &str, + ) -> DeletionPhaseExecution { + self.calls.fetch_add(1, Ordering::SeqCst); + DeletionPhaseExecution::new(self.outcome) + } + } + + #[derive(Default)] + struct FakeRepository { + claim: Mutex>, + claim_error: Mutex>, + claim_calls: AtomicUsize, + block_claim: AtomicBool, + claim_entered: Notify, + claim_release: Notify, + stale_writes: AtomicBool, + defers: Mutex>, + retries: Mutex>, + finishes: Mutex>>, + write_failure: Mutex>, + } + + #[derive(Debug, Clone, Copy, PartialEq, Eq)] + enum RepositoryWriteFailure { + Defer, + Retry, + Terminal, + } + + impl FakeRepository { + fn with_claim(claim: ClaimedContentLockDeletionJob) -> Self { + Self { + claim: Mutex::new(Some(claim)), + ..Self::default() + } + } + + fn with_claim_error(error: ApplicationError) -> Self { + Self { + claim_error: Mutex::new(Some(error)), + ..Self::default() + } + } + + fn write_result(&self, job: ContentLockDeletionJob) -> Option { + (!self.stale_writes.load(Ordering::SeqCst)).then_some(job) + } + } + + #[async_trait] + impl ContentLockDeletionRepository for FakeRepository { + async fn begin_publication( + &self, + _: &CreatorPubky, + _: &LockId, + _: Uuid, + ) -> Result<(), ApplicationError> { + unreachable!() + } + async fn finish_publication( + &self, + _: &CreatorPubky, + _: &LockId, + _: Uuid, + ) -> Result { + unreachable!() + } + async fn abandon_publication( + &self, + _: &CreatorPubky, + _: &LockId, + _: Uuid, + ) -> Result { + unreachable!() + } + async fn publication_in_progress( + &self, + _: &CreatorPubky, + _: &LockId, + ) -> Result { + unreachable!() + } + async fn insert_job(&self, _: ContentLockDeletionJob) -> Result<(), ApplicationError> { + unreachable!() + } + async fn get_job( + &self, + _: &CreatorPubky, + _: &LockId, + ) -> Result, ApplicationError> { + unreachable!() + } + + async fn claim_next( + &self, + _: &str, + _: time::Duration, + ) -> Result, ApplicationError> { + self.claim_calls.fetch_add(1, Ordering::SeqCst); + if let Some(error) = self.claim_error.lock().unwrap().take() { + return Err(error); + } + if self.block_claim.load(Ordering::SeqCst) { + self.claim_entered.notify_one(); + self.claim_release.notified().await; + } + Ok(self.claim.lock().unwrap().take()) + } + + async fn schedule_retry( + &self, + _: Uuid, + _: &str, + _: Uuid, + retry_after: time::Duration, + ) -> Result, ApplicationError> { + if self.write_failure.lock().unwrap().take() == Some(RepositoryWriteFailure::Retry) { + return Err(ApplicationError::Storage { + message: "retry unavailable".to_owned(), + }); + } + self.retries.lock().unwrap().push((NOW, NOW + retry_after)); + Ok(self.write_result(job(1))) + } + + async fn defer( + &self, + _: Uuid, + _: &str, + _: Uuid, + defer_for: time::Duration, + ) -> Result, ApplicationError> { + if self.write_failure.lock().unwrap().take() == Some(RepositoryWriteFailure::Defer) { + return Err(ApplicationError::Storage { + message: "defer unavailable".to_owned(), + }); + } + self.defers.lock().unwrap().push((NOW, NOW + defer_for)); + Ok(self.write_result(job(1))) + } + + async fn advance_phase( + &self, + _: Uuid, + _: &str, + _: Uuid, + _: ContentLockDeletionPhase, + ) -> Result { + Ok(match self.write_result(job(1)) { + Some(job) => AdvanceContentLockDeletionPhaseResult::Advanced(Box::new(job)), + None => AdvanceContentLockDeletionPhaseResult::ClaimLost, + }) + } + + async fn finish( + &self, + _: Uuid, + _: &str, + _: Uuid, + code: Option, + ) -> Result, ApplicationError> { + if self.write_failure.lock().unwrap().take() == Some(RepositoryWriteFailure::Terminal) { + return Err(ApplicationError::Storage { + message: "terminal mutation unavailable".to_owned(), + }); + } + self.finishes.lock().unwrap().push(code); + Ok(self.write_result(job(1))) + } + + async fn resume_failed_job( + &self, + _: &CreatorPubky, + _: &LockId, + _: OffsetDateTime, + ) -> Result, ApplicationError> { + unreachable!() + } + async fn prepare_force_deletion( + &self, + _: &CreatorPubky, + _: &LockId, + ) -> Result { + unreachable!() + } + async fn complete_force_deletion( + &self, + _: Uuid, + _: &str, + _: Uuid, + ) -> Result { + unreachable!() + } + async fn has_force_receipt( + &self, + _: &CreatorPubky, + _: &LockId, + ) -> Result { + unreachable!() + } + } + + fn claim(attempt_count: u32) -> ClaimedContentLockDeletionJob { + ClaimedContentLockDeletionJob { + job: job(attempt_count), + claim_token: Uuid::from_u128(2), + } + } + + fn job(attempt_count: u32) -> ContentLockDeletionJob { + let frozen_content_lock = content_lock(); + ContentLockDeletionJob { + job_id: Uuid::from_u128(1), + creator: frozen_content_lock.creator.clone(), + lock_id: frozen_content_lock.lock_id().unwrap(), + frozen_content_lock, + deletion_started_at: NOW, + state: ContentLockDeletionState::Running, + phase: ContentLockDeletionPhase::Withdraw, + attempt_count, + next_attempt_at: None, + force_requested_at: None, + failure_code: None, + } + } + + fn content_lock() -> ContentLock { + ContentLock { + version: CONTENT_LOCK_VERSION, + creator: CreatorPubky::from_str(CREATOR).unwrap(), + primary_resource: Some( + GuardedResource::new( + "/priv/locks.app/content/post.json".to_owned(), + GuardedResourceHash::from_bytes([7; 32]), + "application/json".to_owned(), + 42, + ) + .unwrap(), + ), + secondary_resources: BTreeMap::new(), + criteria: vec![], + lock_logic: LockLogic::All { criteria: vec![] }, + access_policy: AccessPolicy { + requested_credential_ttl_seconds: 900, + }, + lock_server: LockServerConfig { override_: None }, + created_at: NOW, + } + } +} diff --git a/locks-server/src/lib.rs b/locks-server/src/lib.rs index f1805f4..c5b23d7 100644 --- a/locks-server/src/lib.rs +++ b/locks-server/src/lib.rs @@ -1,6 +1,7 @@ pub mod api; pub mod app_state; pub mod config; +pub mod deletion_worker; pub mod paykit_http_client; pub mod pkdns; pub mod rate_limit; diff --git a/locks-server/src/main.rs b/locks-server/src/main.rs index 6fba9f7..c2406bf 100644 --- a/locks-server/src/main.rs +++ b/locks-server/src/main.rs @@ -1,14 +1,17 @@ -use std::env; -use std::error::Error; +use std::{env, error::Error, time::Duration}; use locks_server::api::routes::router; use locks_server::config::{FilesystemLockServerIdentityProvider, load_or_initialize_config}; +use locks_server::deletion_worker::{ + DeletionWorker, DeletionWorkerConfig, RandomFullJitter, RuntimeClaimedDeletionExecutor, +}; use locks_server::pkdns::LockServerKeyRepublisher; -use locks_server::runtime::{home_dir_from_env, parse_config_arg}; +use locks_server::runtime::{ + InitialStartupOutcome, RuntimeTasks, ShutdownFuture, bind_listener_then_run_initial, + home_dir_from_env, parse_config_arg, supervise, wait_for_shutdown, +}; use locks_server::storage::build_runtime_state; use locks_server::worker::VerificationWorker; -use tokio::net::TcpListener; -use tokio::sync::watch; use tower_http::trace::TraceLayer; use tracing::{error, info}; use tracing_subscriber::EnvFilter; @@ -21,41 +24,117 @@ async fn main() -> Result<(), Box> { let config = load_or_initialize_config(config_path, &home_dir, &identity_provider)?; init_tracing(&config.logging.level); let bind_addr = config.bind_addr; - let _key_republisher = LockServerKeyRepublisher::start_if_required(&config).await?; let state = build_runtime_state(config).await?; - let worker_enabled = state.config().worker.enabled; - let (shutdown_tx, shutdown_rx) = watch::channel(false); - let worker_handle = if worker_enabled { - let worker_state = state.clone(); - Some(tokio::spawn(async move { - let worker = VerificationWorker::from_state(&worker_state); - if let Err(error) = worker.run_until_shutdown(shutdown_rx).await { - error!(%error, "verification worker stopped with error"); + let shutdown_timeout = + Duration::from_secs(state.config().deletion_worker.shutdown_timeout_seconds); + // Validate publication state before network startup, but do not advertise the service until + // secret validation, database startup, migrations, and listener binding have all succeeded. + let key_republisher = LockServerKeyRepublisher::build_if_required(state.config())?; + let mut shutdown: ShutdownFuture = Box::pin(shutdown_signal()); + let startup = bind_listener_then_run_initial( + bind_addr, + shutdown_timeout, + &mut shutdown, + move || async move { + if let Some(republisher) = &key_republisher { + republisher + .publish_initial() + .await + .map_err(anyhow::Error::from)?; } - })) - } else { - None - }; - let app = router(state).layer(TraceLayer::new_for_http()); - let listener = TcpListener::bind(bind_addr).await?; - - info!(%bind_addr, "starting locks-server"); - axum::serve( - listener, - app.into_make_service_with_connect_info::(), + Ok(key_republisher) + }, ) - .with_graceful_shutdown(shutdown_signal()) .await?; - let _ = shutdown_tx.send(true); - if let Some(worker_handle) = worker_handle { - worker_handle.await?; + let InitialStartupOutcome::Ready { + listener, + initial: key_republisher, + } = startup + else { + return Ok(()); + }; + let app = router(state.clone()).layer(TraceLayer::new_for_http()); + + let mut tasks = RuntimeTasks::new(shutdown_timeout).with_http(Box::new(move |shutdown| { + Box::pin(async move { + axum::serve( + listener, + app.into_make_service_with_connect_info::(), + ) + .with_graceful_shutdown(wait_for_shutdown(shutdown)) + .await + .map_err(anyhow::Error::from) + }) + })); + + if let Some(key_republisher) = key_republisher { + tasks = tasks.with_pkarr_republisher(Box::new(move |shutdown| { + Box::pin(async move { + key_republisher + .run_until_shutdown(shutdown) + .await + .map_err(anyhow::Error::from) + }) + })); } + + if state.config().worker.enabled { + let worker_state = state.clone(); + tasks = tasks.with_verification_worker(Box::new(move |context| { + Box::pin(async move { + VerificationWorker::from_state(&worker_state) + .run_until_shutdown_with_readiness( + context.shutdown(), + worker_state.worker_readiness(), + ) + .await + .map_err(anyhow::Error::from) + }) + })); + } + + if state.config().deletion_worker.enabled { + let worker_state = state.clone(); + tasks = tasks.with_deletion_worker(Box::new(move |context| { + Box::pin(async move { + let runtime = &worker_state.config().deletion_worker; + let retry = &worker_state.config().deletion; + let executor = RuntimeClaimedDeletionExecutor::new(worker_state.clone()); + let jitter = RandomFullJitter; + let config = DeletionWorkerConfig { + worker_id: runtime.worker_id.clone(), + poll_interval: Duration::from_millis(runtime.poll_interval_ms), + claim_timeout: Duration::from_secs(runtime.claim_timeout_seconds), + retry_max_attempts: retry.retry_max_attempts, + retry_initial_backoff: Duration::from_secs(retry.retry_initial_backoff_seconds), + retry_max_backoff: Duration::from_secs(retry.retry_max_backoff_seconds), + }; + DeletionWorker::new( + worker_state.content_lock_deletions().as_ref(), + worker_state.clock().as_ref(), + &executor, + &jitter, + config, + ) + .run_until_shutdown_with_readiness( + context.shutdown(), + worker_state.worker_readiness(), + ) + .await + .map_err(anyhow::Error::from) + }) + })); + } + + info!(%bind_addr, "starting locks-server"); + supervise(state.worker_readiness().clone(), shutdown, tasks).await?; Ok(()) } async fn shutdown_signal() { if let Err(error) = tokio::signal::ctrl_c().await { error!(%error, "failed to listen for shutdown signal"); + std::future::pending::<()>().await; } } diff --git a/locks-server/src/pkdns.rs b/locks-server/src/pkdns.rs index 2fdf4cd..2501bdb 100644 --- a/locks-server/src/pkdns.rs +++ b/locks-server/src/pkdns.rs @@ -22,23 +22,30 @@ pub enum LockServerKeyRepublisherError { PacketBuild(String), #[error("failed to build PKARR client: {0}")] ClientBuild(String), + #[error("PKARR republisher interval must be greater than zero")] + InvalidInterval, #[error("failed to publish lock server PKARR packet: {0}")] Publish(String), } -/// Background task that publishes and periodically republishes the Lock Server key's PKARR record. +/// Owned task that publishes and periodically republishes the Lock Server key's PKARR record. #[derive(Debug)] pub struct LockServerKeyRepublisher { - join_handle: tokio::task::JoinHandle<()>, + client: pkarr::Client, + signed_packet: pkarr::SignedPacket, + interval: std::time::Duration, } impl LockServerKeyRepublisher { - pub async fn start_if_required( + pub fn build_if_required( config: &LockServerRuntimeConfig, ) -> Result, LockServerKeyRepublisherError> { if !requires_lock_server_pkarr(config) { return Ok(None); } + if config.pkdns.key_republisher_interval_seconds == 0 { + return Err(LockServerKeyRepublisherError::InvalidInterval); + } let keypair = load_lock_server_keypair(&config.credentials)?; let signed_packet = create_signed_packet(&config.pkdns, &keypair)?; @@ -51,30 +58,35 @@ impl LockServerKeyRepublisher { let client = builder .build() .map_err(|error| LockServerKeyRepublisherError::ClientBuild(error.to_string()))?; - publish_once(&client, &signed_packet).await?; - - let interval_seconds = config.pkdns.key_republisher_interval_seconds; - let join_handle = tokio::spawn(async move { - let mut interval = - tokio::time::interval(std::time::Duration::from_secs(interval_seconds)); - interval.tick().await; - loop { - interval.tick().await; - let _ = publish_once(&client, &signed_packet).await; - } - }); - - Ok(Some(Self { join_handle })) + Ok(Some(Self { + client, + signed_packet, + interval: std::time::Duration::from_secs(config.pkdns.key_republisher_interval_seconds), + })) } - pub fn stop(&self) { - self.join_handle.abort(); + pub async fn publish_initial(&self) -> Result<(), LockServerKeyRepublisherError> { + publish_once(&self.client, &self.signed_packet).await } -} -impl Drop for LockServerKeyRepublisher { - fn drop(&mut self) { - self.stop(); + pub async fn run_until_shutdown( + self, + mut shutdown: tokio::sync::watch::Receiver, + ) -> Result<(), LockServerKeyRepublisherError> { + let mut interval = tokio::time::interval(self.interval); + interval.tick().await; + loop { + tokio::select! { + changed = shutdown.changed() => { + if changed.is_err() || *shutdown.borrow() { + return Ok(()); + } + } + _ = interval.tick() => { + publish_once(&self.client, &self.signed_packet).await?; + } + } + } } } @@ -275,6 +287,20 @@ mod tests { )); } + #[tokio::test] + async fn zero_republisher_interval_is_rejected_before_secret_or_publication_work() { + let mut config = test_config(); + config.creator_authority_acquisition.enabled = true; + config.pkdns.key_republisher_interval_seconds = 0; + + let error = LockServerKeyRepublisher::build_if_required(&config).unwrap_err(); + + assert!(matches!( + error, + LockServerKeyRepublisherError::InvalidInterval + )); + } + #[test] fn pkarr_republisher_is_required_for_production_lock_server_identity() { let mut config = test_config(); @@ -330,6 +356,7 @@ mod tests { rate_limits: RateLimitsConfig::default(), content_locks: ContentLocksConfig::default(), deletion: crate::config::DeletionConfig::default(), + deletion_worker: crate::config::DeletionWorkerConfig::default(), paykit: None, } } diff --git a/locks-server/src/runtime.rs b/locks-server/src/runtime.rs index fbc11d3..312ff37 100644 --- a/locks-server/src/runtime.rs +++ b/locks-server/src/runtime.rs @@ -1,6 +1,425 @@ -use std::path::PathBuf; +use std::{ + collections::HashMap, future::Future, net::SocketAddr, path::PathBuf, pin::Pin, time::Duration, +}; -use crate::config::ConfigError; +use tokio::{ + net::TcpListener, + sync::watch, + task::{Id, JoinError, JoinSet}, +}; + +use crate::{ + app_state::{WorkerKind, WorkerReadiness, WorkerReadinessEvidence}, + config::ConfigError, +}; + +pub type RuntimeTaskFuture = Pin> + Send + 'static>>; +pub type ShutdownFuture = Pin + Send + 'static>>; +pub type HttpTaskFactory = + Box) -> RuntimeTaskFuture + Send + 'static>; +pub type LifecycleTaskFactory = + Box) -> RuntimeTaskFuture + Send + 'static>; +pub type WorkerTaskFactory = + Box RuntimeTaskFuture + Send + 'static>; + +#[derive(Debug)] +pub enum InitialStartupOutcome { + Ready { listener: TcpListener, initial: T }, + ShutdownRequested, +} + +#[derive(Debug, thiserror::Error)] +pub enum InitialStartupError { + #[error("failed to bind HTTP listener: {0}")] + Bind(#[source] std::io::Error), + #[error("initial startup task failed: {0}")] + Initial(#[source] anyhow::Error), + #[error("initial startup exceeded lifecycle timeout {0:?}")] + TimedOut(Duration), +} + +pub async fn bind_listener_then_run_initial( + bind_addr: SocketAddr, + lifecycle_timeout: Duration, + shutdown: &mut Pin>, + initial: F, +) -> Result, InitialStartupError> +where + S: Future + ?Sized, + F: FnOnce() -> Fut, + Fut: Future>, +{ + let listener = tokio::select! { + biased; + _ = shutdown.as_mut() => return Ok(InitialStartupOutcome::ShutdownRequested), + result = TcpListener::bind(bind_addr) => result.map_err(InitialStartupError::Bind)?, + }; + + let initial = tokio::time::timeout(lifecycle_timeout, initial()); + tokio::pin!(initial); + tokio::select! { + biased; + _ = shutdown.as_mut() => Ok(InitialStartupOutcome::ShutdownRequested), + result = &mut initial => match result { + Ok(Ok(initial)) => Ok(InitialStartupOutcome::Ready { listener, initial }), + Ok(Err(error)) => Err(InitialStartupError::Initial(error)), + Err(_) => Err(InitialStartupError::TimedOut(lifecycle_timeout)), + }, + } +} + +#[derive(Clone)] +pub struct WorkerTaskContext { + shutdown: watch::Receiver, + readiness: WorkerReadiness, + worker: WorkerKind, +} + +impl WorkerTaskContext { + pub fn shutdown(&self) -> watch::Receiver { + self.shutdown.clone() + } + + pub fn record(&self, evidence: WorkerReadinessEvidence) { + self.readiness.record(self.worker, evidence); + } +} + +#[derive(Default)] +pub struct RuntimeTasks { + shutdown_timeout: Duration, + http: Option, + pkarr_republisher: Option, + verification_worker: Option, + deletion_worker: Option, +} + +impl RuntimeTasks { + pub fn new(shutdown_timeout: Duration) -> Self { + Self { + shutdown_timeout, + ..Self::default() + } + } + + pub fn with_http(mut self, factory: HttpTaskFactory) -> Self { + self.http = Some(factory); + self + } + + pub fn with_pkarr_republisher(mut self, factory: LifecycleTaskFactory) -> Self { + self.pkarr_republisher = Some(factory); + self + } + + pub fn with_verification_worker(mut self, factory: WorkerTaskFactory) -> Self { + self.verification_worker = Some(factory); + self + } + + pub fn with_deletion_worker(mut self, factory: WorkerTaskFactory) -> Self { + self.deletion_worker = Some(factory); + self + } +} + +#[derive(Debug, thiserror::Error)] +pub enum RuntimeError { + #[error("{0} exited unexpectedly")] + UnexpectedExit(&'static str), + #[error("{task} failed: {error}")] + TaskFailed { + task: &'static str, + error: anyhow::Error, + }, + #[error("{0} panicked")] + TaskPanicked(&'static str), + #[error("runtime shutdown exceeded {0:?}")] + ShutdownTimedOut(Duration), +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum OwnedTask { + Http, + PkarrRepublisher, + Verification, + Deletion, + Unknown, +} + +impl OwnedTask { + fn label(self) -> &'static str { + match self { + Self::Http => "HTTP server", + Self::PkarrRepublisher => "PKARR republisher", + Self::Verification => "verification worker", + Self::Deletion => "deletion worker", + Self::Unknown => "owned runtime task", + } + } + + fn worker(self) -> Option { + match self { + Self::Http | Self::PkarrRepublisher | Self::Unknown => None, + Self::Verification => Some(WorkerKind::Verification), + Self::Deletion => Some(WorkerKind::Deletion), + } + } +} + +pub async fn supervise( + readiness: WorkerReadiness, + mut shutdown: ShutdownFuture, + tasks: RuntimeTasks, +) -> Result<(), RuntimeError> { + let shutdown_timeout = tasks.shutdown_timeout; + let (worker_shutdown_tx, worker_shutdown_rx) = watch::channel(false); + let (service_shutdown_tx, service_shutdown_rx) = watch::channel(false); + let verification_enabled = tasks.verification_worker.is_some(); + let deletion_enabled = tasks.deletion_worker.is_some(); + let mut workers_remaining = usize::from(verification_enabled) + usize::from(deletion_enabled); + let mut roots = JoinSet::new(); + let mut root_tasks = HashMap::new(); + + if let Some(factory) = tasks.http { + spawn_root( + &mut roots, + &mut root_tasks, + OwnedTask::Http, + factory(service_shutdown_rx), + ); + } + if let Some(factory) = tasks.pkarr_republisher { + let shutdown = service_shutdown_tx.subscribe(); + spawn_root( + &mut roots, + &mut root_tasks, + OwnedTask::PkarrRepublisher, + factory(shutdown), + ); + } + if let Some(factory) = tasks.verification_worker { + let context = WorkerTaskContext { + shutdown: worker_shutdown_rx.clone(), + readiness: readiness.clone(), + worker: WorkerKind::Verification, + }; + spawn_root( + &mut roots, + &mut root_tasks, + OwnedTask::Verification, + factory(context), + ); + } + if let Some(factory) = tasks.deletion_worker { + let context = WorkerTaskContext { + shutdown: worker_shutdown_rx, + readiness: readiness.clone(), + worker: WorkerKind::Deletion, + }; + spawn_root( + &mut roots, + &mut root_tasks, + OwnedTask::Deletion, + factory(context), + ); + } + + let failure = tokio::select! { + biased; + _ = &mut shutdown => None, + completed = join_next_owned(&mut roots, &mut root_tasks) => { + if let Some((task, _)) = &completed + && task.worker().is_some() + { + workers_remaining = workers_remaining.saturating_sub(1); + } + Some(classify_premature(completed, &readiness)) + }, + }; + + begin_worker_shutdown( + &readiness, + verification_enabled, + deletion_enabled, + &worker_shutdown_tx, + ); + + let mut failure = failure; + let drain_result = tokio::time::timeout( + shutdown_timeout, + stop_workers_then_drain_services( + &mut roots, + &mut root_tasks, + workers_remaining, + &service_shutdown_tx, + &mut failure, + ), + ) + .await; + match drain_result { + Ok(()) => match failure { + Some(error) => Err(error), + None => Ok(()), + }, + Err(_) => { + roots.abort_all(); + while roots.join_next().await.is_some() {} + Err(failure.unwrap_or(RuntimeError::ShutdownTimedOut(shutdown_timeout))) + } + } +} + +fn spawn_root( + roots: &mut JoinSet<(OwnedTask, anyhow::Result<()>)>, + root_tasks: &mut HashMap, + task: OwnedTask, + future: RuntimeTaskFuture, +) { + let handle = roots.spawn(async move { (task, future.await) }); + root_tasks.insert(handle.id(), task); +} + +type OwnedCompletion = (OwnedTask, Result, JoinError>); + +async fn join_next_owned( + roots: &mut JoinSet<(OwnedTask, anyhow::Result<()>)>, + root_tasks: &mut HashMap, +) -> Option { + match roots.join_next_with_id().await { + Some(Ok((id, (task, result)))) => { + root_tasks.remove(&id); + Some((task, Ok(result))) + } + Some(Err(error)) => { + let task = root_tasks.remove(&error.id()).unwrap_or(OwnedTask::Unknown); + Some((task, Err(error))) + } + None => None, + } +} + +fn begin_worker_shutdown( + readiness: &WorkerReadiness, + verification_enabled: bool, + deletion_enabled: bool, + worker_shutdown_tx: &watch::Sender, +) { + if verification_enabled { + readiness.record(WorkerKind::Verification, WorkerReadinessEvidence::Stopping); + } + if deletion_enabled { + readiness.record(WorkerKind::Deletion, WorkerReadinessEvidence::Stopping); + } + worker_shutdown_tx.send_replace(true); +} + +async fn stop_workers_then_drain_services( + roots: &mut JoinSet<(OwnedTask, anyhow::Result<()>)>, + root_tasks: &mut HashMap, + mut workers_remaining: usize, + service_shutdown_tx: &watch::Sender, + failure: &mut Option, +) { + while workers_remaining > 0 { + match join_next_owned(roots, root_tasks).await { + Some((task, result)) if task.worker().is_some() => { + workers_remaining -= 1; + if let Some(error) = classify_shutdown_completion(task, result, false) { + failure.get_or_insert(error); + } + } + Some((task, result)) => { + if let Some(error) = classify_shutdown_completion(task, result, true) { + failure.get_or_insert(error); + } + } + None => { + failure.get_or_insert(RuntimeError::UnexpectedExit("runtime task set")); + break; + } + } + } + + // Worker root completion is explicit evidence that no future queue claim can begin. + // Only now may HTTP graceful draining and other service-root cancellation start. + service_shutdown_tx.send_replace(true); + drain(roots, root_tasks, failure).await; +} + +fn classify_premature( + completed: Option, + readiness: &WorkerReadiness, +) -> RuntimeError { + match completed { + Some((task, Ok(result))) => { + if let Some(worker) = task.worker() { + readiness.record(worker, WorkerReadinessEvidence::UnexpectedExit); + } + match result { + Ok(()) => RuntimeError::UnexpectedExit(task.label()), + Err(error) => RuntimeError::TaskFailed { + task: task.label(), + error, + }, + } + } + Some((task, Err(error))) => { + if let Some(worker) = task.worker() { + readiness.record(worker, WorkerReadinessEvidence::UnexpectedExit); + } + classify_join_error(task, error) + } + None => RuntimeError::UnexpectedExit("runtime task set"), + } +} + +fn classify_shutdown_completion( + task: OwnedTask, + result: Result, JoinError>, + success_is_unexpected: bool, +) -> Option { + match result { + Ok(Ok(())) if success_is_unexpected => Some(RuntimeError::UnexpectedExit(task.label())), + Ok(Ok(())) => None, + Ok(Err(error)) => Some(RuntimeError::TaskFailed { + task: task.label(), + error, + }), + Err(error) => Some(classify_join_error(task, error)), + } +} + +fn classify_join_error(task: OwnedTask, error: JoinError) -> RuntimeError { + if error.is_panic() { + RuntimeError::TaskPanicked(task.label()) + } else { + RuntimeError::UnexpectedExit(task.label()) + } +} + +async fn drain( + roots: &mut JoinSet<(OwnedTask, anyhow::Result<()>)>, + root_tasks: &mut HashMap, + failure: &mut Option, +) { + while let Some((task, result)) = join_next_owned(roots, root_tasks).await { + if let Some(error) = classify_shutdown_completion(task, result, false) { + failure.get_or_insert(error); + } + } +} + +pub async fn wait_for_shutdown(mut shutdown: watch::Receiver) { + if *shutdown.borrow() { + return; + } + while shutdown.changed().await.is_ok() { + if *shutdown.borrow() { + return; + } + } +} pub fn parse_config_arg(args: I) -> Result, ConfigError> where @@ -45,7 +464,348 @@ pub fn home_dir_from_env() -> Result { #[cfg(test)] mod tests { - use super::parse_config_arg; + use std::{ + sync::{Arc, Mutex}, + time::{Duration, Instant}, + }; + + use tokio::sync::{Notify, oneshot}; + + use crate::app_state::{ + WorkerKind, WorkerReadiness, WorkerReadinessEvidence, WorkerReadinessState, + }; + + use super::{RuntimeTasks, parse_config_arg, supervise, wait_for_shutdown}; + + #[tokio::test] + async fn premature_worker_success_fails_runtime_and_marks_worker_unexpected() { + let readiness = WorkerReadiness::new(true, false); + let observed = readiness.clone(); + let tasks = RuntimeTasks::new(Duration::from_millis(100)) + .with_verification_worker(Box::new(|_context| Box::pin(async { Ok(()) }))); + + let error = supervise(readiness, Box::pin(std::future::pending()), tasks) + .await + .unwrap_err(); + + assert!( + error + .to_string() + .contains("verification worker exited unexpectedly") + ); + assert_eq!( + observed.worker_state(WorkerKind::Verification), + WorkerReadinessState::NotReady + ); + } + + #[tokio::test] + async fn worker_error_and_panic_fail_runtime() { + for tasks in [ + RuntimeTasks::new(Duration::from_millis(100)).with_verification_worker(Box::new( + |_context| Box::pin(async { anyhow::bail!("dependency broke") }), + )), + RuntimeTasks::new(Duration::from_millis(100)) + .with_verification_worker(Box::new(|_context| Box::pin(async { panic!("boom") }))), + ] { + let error = supervise( + WorkerReadiness::new(true, false), + Box::pin(std::future::pending()), + tasks, + ) + .await + .unwrap_err(); + assert!(error.to_string().contains("failed") || error.to_string().contains("panicked")); + } + } + + #[tokio::test] + async fn worker_panic_starts_service_drain_promptly_and_preserves_panic_verdict() { + let http_started = Arc::new(Notify::new()); + let http_draining = Arc::new(Notify::new()); + let (panic_tx, panic_rx) = oneshot::channel(); + let tasks = RuntimeTasks::new(Duration::from_secs(5)) + .with_http(Box::new({ + let http_started = Arc::clone(&http_started); + let http_draining = Arc::clone(&http_draining); + move |shutdown| { + Box::pin(async move { + http_started.notify_one(); + wait_for_shutdown(shutdown).await; + http_draining.notify_one(); + Ok(()) + }) + } + })) + .with_verification_worker(Box::new(move |_context| { + Box::pin(async move { + let _ = panic_rx.await; + panic!("boom"); + }) + })); + let supervisor = tokio::spawn(supervise( + WorkerReadiness::new(true, false), + Box::pin(std::future::pending()), + tasks, + )); + + http_started.notified().await; + panic_tx.send(()).unwrap(); + let error = tokio::time::timeout(Duration::from_millis(200), supervisor) + .await + .expect("worker panic should not wait for the shutdown deadline") + .unwrap() + .unwrap_err(); + + tokio::time::timeout(Duration::from_millis(100), http_draining.notified()) + .await + .expect("worker panic should signal service draining"); + assert!(matches!( + error, + super::RuntimeError::TaskPanicked("verification worker") + )); + } + + #[tokio::test] + async fn pkarr_republisher_failure_fails_runtime() { + let tasks = RuntimeTasks::new(Duration::from_millis(100)).with_pkarr_republisher(Box::new( + |_shutdown| Box::pin(async { anyhow::bail!("publish failed") }), + )); + + let error = supervise( + WorkerReadiness::new(false, false), + Box::pin(std::future::pending()), + tasks, + ) + .await + .unwrap_err(); + + assert!(error.to_string().contains("PKARR republisher failed")); + } + + #[tokio::test] + async fn readiness_is_not_ready_before_workers_observe_sticky_cancellation() { + let readiness = WorkerReadiness::new(true, true); + let observed = readiness.clone(); + let observations = Arc::new(Mutex::new(Vec::new())); + let (shutdown_tx, shutdown_rx) = oneshot::channel(); + let tasks = RuntimeTasks::new(Duration::from_millis(100)) + .with_verification_worker(observing_worker( + observed.clone(), + Arc::clone(&observations), + WorkerKind::Verification, + )) + .with_deletion_worker(observing_worker( + observed, + Arc::clone(&observations), + WorkerKind::Deletion, + )); + shutdown_tx.send(()).unwrap(); + + supervise( + readiness, + Box::pin(async { + let _ = shutdown_rx.await; + }), + tasks, + ) + .await + .unwrap(); + + assert_eq!( + *observations.lock().unwrap(), + vec![ + WorkerReadinessState::NotReady, + WorkerReadinessState::NotReady + ] + ); + } + + fn observing_worker( + readiness: WorkerReadiness, + observations: Arc>>, + kind: WorkerKind, + ) -> super::WorkerTaskFactory { + Box::new(move |context| { + Box::pin(async move { + wait_for_shutdown(context.shutdown()).await; + observations + .lock() + .unwrap() + .push(readiness.worker_state(kind)); + Ok(()) + }) + }) + } + + #[tokio::test] + async fn http_drain_waits_for_explicit_worker_claim_stop() { + let worker_stopping = Arc::new(Notify::new()); + let allow_worker_stop = Arc::new(Notify::new()); + let http_draining = Arc::new(Notify::new()); + let (shutdown_tx, shutdown_rx) = oneshot::channel(); + let tasks = RuntimeTasks::new(Duration::from_secs(1)) + .with_http(Box::new({ + let http_draining = Arc::clone(&http_draining); + move |shutdown| { + Box::pin(async move { + wait_for_shutdown(shutdown).await; + http_draining.notify_one(); + Ok(()) + }) + } + })) + .with_verification_worker(Box::new({ + let worker_stopping = Arc::clone(&worker_stopping); + let allow_worker_stop = Arc::clone(&allow_worker_stop); + move |context| { + Box::pin(async move { + wait_for_shutdown(context.shutdown()).await; + worker_stopping.notify_one(); + allow_worker_stop.notified().await; + Ok(()) + }) + } + })); + let supervisor = tokio::spawn(supervise( + WorkerReadiness::new(true, false), + Box::pin(async { + let _ = shutdown_rx.await; + }), + tasks, + )); + + shutdown_tx.send(()).unwrap(); + worker_stopping.notified().await; + assert!( + tokio::time::timeout(Duration::from_millis(20), http_draining.notified()) + .await + .is_err(), + "HTTP draining started before the verification worker stopped claiming" + ); + allow_worker_stop.notify_one(); + supervisor.await.unwrap().unwrap(); + tokio::time::timeout(Duration::from_millis(100), http_draining.notified()) + .await + .unwrap(); + } + + #[tokio::test] + async fn graceful_shutdown_joins_all_roots_and_late_subscriber_exits() { + let joined = Arc::new(Mutex::new(Vec::new())); + let late_subscription_exited = Arc::new(Notify::new()); + let (shutdown_tx, shutdown_rx) = oneshot::channel(); + let tasks = RuntimeTasks::new(Duration::from_secs(1)) + .with_http(joining_task(Arc::clone(&joined), "http")) + .with_verification_worker(Box::new({ + let joined = Arc::clone(&joined); + let late_subscription_exited = Arc::clone(&late_subscription_exited); + move |context| { + Box::pin(async move { + wait_for_shutdown(context.shutdown()).await; + wait_for_shutdown(context.shutdown()).await; + late_subscription_exited.notify_one(); + joined.lock().unwrap().push("verification"); + Ok(()) + }) + } + })) + .with_deletion_worker(Box::new({ + let joined = Arc::clone(&joined); + move |context| { + Box::pin(async move { + wait_for_shutdown(context.shutdown()).await; + joined.lock().unwrap().push("deletion"); + Ok(()) + }) + } + })); + shutdown_tx.send(()).unwrap(); + + supervise( + WorkerReadiness::new(true, true), + Box::pin(async { + let _ = shutdown_rx.await; + }), + tasks, + ) + .await + .unwrap(); + tokio::time::timeout( + Duration::from_millis(100), + late_subscription_exited.notified(), + ) + .await + .unwrap(); + let mut roots = joined.lock().unwrap().clone(); + roots.sort_unstable(); + assert_eq!(roots, vec!["deletion", "http", "verification"]); + } + + fn joining_task( + joined: Arc>>, + name: &'static str, + ) -> super::HttpTaskFactory { + Box::new(move |shutdown| { + Box::pin(async move { + wait_for_shutdown(shutdown).await; + joined.lock().unwrap().push(name); + Ok(()) + }) + }) + } + + #[tokio::test] + async fn blocked_root_is_aborted_and_drained_at_deadline() { + let dropped = Arc::new(Notify::new()); + let (shutdown_tx, shutdown_rx) = oneshot::channel(); + let tasks = + RuntimeTasks::new(Duration::from_millis(20)).with_verification_worker(Box::new({ + let dropped = Arc::clone(&dropped); + move |_context| { + Box::pin(async move { + let _guard = DropSignal(dropped); + std::future::pending::<()>().await; + Ok(()) + }) + } + })); + shutdown_tx.send(()).unwrap(); + let started = Instant::now(); + + let error = supervise( + WorkerReadiness::new(true, false), + Box::pin(async { + let _ = shutdown_rx.await; + }), + tasks, + ) + .await + .unwrap_err(); + + assert!(error.to_string().contains("shutdown exceeded")); + assert!(started.elapsed() < Duration::from_millis(250)); + tokio::time::timeout(Duration::from_millis(100), dropped.notified()) + .await + .unwrap(); + } + + struct DropSignal(Arc); + impl Drop for DropSignal { + fn drop(&mut self) { + self.0.notify_one(); + } + } + + #[test] + fn readiness_evidence_can_be_recorded_through_worker_context_contract() { + let readiness = WorkerReadiness::new(true, false); + readiness.record(WorkerKind::Verification, WorkerReadinessEvidence::Ready); + assert_eq!( + readiness.worker_state(WorkerKind::Verification), + WorkerReadinessState::Ready + ); + } #[test] fn parse_config_arg_accepts_absent_or_explicit_config() { diff --git a/locks-server/src/testing.rs b/locks-server/src/testing.rs index 4159893..a40165d 100644 --- a/locks-server/src/testing.rs +++ b/locks-server/src/testing.rs @@ -11,9 +11,11 @@ use locks_service::application::models::{ }; use locks_service::application::ports::LegacyCreatorConnectFlowClient; use locks_service::infrastructure::memory::{ + content_lock_tombstones::InMemoryContentLockTombstoneRepository, content_locks::InMemoryContentLockRepository, entitlements::InMemoryEntitlementRepository, guarded_resources::InMemoryGuardedResourceRepository, lock_service_pointers::InMemoryLockServicePointerRepository, + public_content_locks::InMemoryPublicContentLockStore, }; use locks_service::infrastructure::pubky::PubkyHomeserverStorageClient; use time::OffsetDateTime; @@ -40,10 +42,16 @@ impl TestServerApp { } pub fn new_in_memory(config: LockServerRuntimeConfig) -> Self { + let public_content_locks = InMemoryPublicContentLockStore::new(); Self { state: AppState::new_empty_in_memory_with_creator_repositories( config, - Arc::new(InMemoryContentLockRepository::new()), + Arc::new(InMemoryContentLockRepository::with_public_store( + public_content_locks.clone(), + )), + Arc::new(InMemoryContentLockTombstoneRepository::with_public_store( + public_content_locks, + )), Arc::new(InMemoryGuardedResourceRepository::new()), Arc::new(InMemoryLockServicePointerRepository::new()), Arc::new(InMemoryEntitlementRepository::new()), @@ -101,6 +109,7 @@ impl TestServerApp { rate_limits: RateLimitsConfig::default(), content_locks: ContentLocksConfig::default(), deletion: crate::config::DeletionConfig::default(), + deletion_worker: crate::config::DeletionWorkerConfig::default(), paykit: None, } } diff --git a/locks-server/src/worker.rs b/locks-server/src/worker.rs index 35dfac3..dc70f51 100644 --- a/locks-server/src/worker.rs +++ b/locks-server/src/worker.rs @@ -1,5 +1,12 @@ +use std::sync::atomic::{AtomicU8, Ordering}; + +use async_trait::async_trait; use locks_core::ids::{LockServerPubky, TaskId}; +use locks_core::verification::CriterionVerificationResult; use locks_service::application::errors::ApplicationError; +use locks_service::application::models::{ + ClaimedVerificationTask, CriterionVerificationRequest, VerificationTaskRecord, +}; use locks_service::application::ports::{ Clock, ContentLockRepository, CriterionVerifier, EntitlementRepository, VerificationTaskClaimer, VerificationTaskRepository, @@ -11,15 +18,333 @@ use locks_service::infrastructure::verifiers::registry::StaticCriterionVerifierR use tokio::sync::watch; use tracing::{debug, error, info}; -use crate::app_state::AppState; +use crate::app_state::{AppState, WorkerKind, WorkerReadiness, WorkerReadinessEvidence}; const PENDING_VERIFICATION_RETRY_DELAY_SECONDS: i64 = 30; +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum PaykitProviderEvidence { + None, + HealthyResponse, + Unavailable, +} + +#[derive(Debug, Default, Clone, Copy, PartialEq, Eq)] +enum OperationEvidence { + #[default] + None, + Succeeded, +} + +#[derive(Debug, Default, Clone, Copy, PartialEq, Eq)] +struct RepositoryOperationEvidence { + queue_poll: OperationEvidence, + mutation: OperationEvidence, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +struct VerificationPoll { + tick: WorkerTick, + paykit_provider: PaykitProviderEvidence, + repository: RepositoryOperationEvidence, +} + +impl VerificationPoll { + fn without_provider_evidence(tick: WorkerTick) -> Self { + Self { + tick, + paykit_provider: PaykitProviderEvidence::None, + repository: RepositoryOperationEvidence::default(), + } + } +} + +#[derive(Debug)] +struct VerificationPollError { + operation: RepositoryOperation, + error: ApplicationError, + repository: RepositoryOperationEvidence, +} + +#[derive(Default)] +struct RepositoryOperationObserver { + queue_poll_succeeded: AtomicU8, + mutation_succeeded: AtomicU8, +} + +impl RepositoryOperationObserver { + fn record_queue_poll_success(&self) { + self.queue_poll_succeeded.store(1, Ordering::Relaxed); + } + + fn record_mutation_success(&self) { + self.mutation_succeeded.store(1, Ordering::Relaxed); + } + + fn evidence(&self) -> RepositoryOperationEvidence { + RepositoryOperationEvidence { + queue_poll: if self.queue_poll_succeeded.load(Ordering::Relaxed) == 1 { + OperationEvidence::Succeeded + } else { + OperationEvidence::None + }, + mutation: if self.mutation_succeeded.load(Ordering::Relaxed) == 1 { + OperationEvidence::Succeeded + } else { + OperationEvidence::None + }, + } + } +} + +struct ObservedVerificationTaskClaimer<'a> { + inner: &'a dyn VerificationTaskClaimer, + observer: &'a RepositoryOperationObserver, +} + +#[async_trait] +impl VerificationTaskClaimer for ObservedVerificationTaskClaimer<'_> { + async fn begin_claimed_entitlement_publication( + &self, + task_id: &TaskId, + worker_id: &str, + claim_token: &uuid::Uuid, + ) -> Result { + let result = self + .inner + .begin_claimed_entitlement_publication(task_id, worker_id, claim_token) + .await; + if matches!(result, Ok(true)) { + self.observer.record_mutation_success(); + } + result + } + + async fn claim_next_verification_task( + &self, + worker_id: &str, + claim_ttl: time::Duration, + ) -> Result, ApplicationError> { + let result = self + .inner + .claim_next_verification_task(worker_id, claim_ttl) + .await; + if result.is_ok() { + self.observer.record_queue_poll_success(); + } + result + } + + async fn schedule_verification_task_retry( + &self, + task_id: &TaskId, + worker_id: &str, + claim_token: &uuid::Uuid, + retry_after: time::Duration, + ) -> Result, ApplicationError> { + let result = self + .inner + .schedule_verification_task_retry(task_id, worker_id, claim_token, retry_after) + .await; + if matches!(result, Ok(Some(_))) { + self.observer.record_mutation_success(); + } + result + } + + async fn persist_claimed_verification_task_transition( + &self, + task: VerificationTaskRecord, + worker_id: &str, + claim_token: &uuid::Uuid, + ) -> Result, ApplicationError> { + let result = self + .inner + .persist_claimed_verification_task_transition(task, worker_id, claim_token) + .await; + if matches!(result, Ok(Some(_))) { + self.observer.record_mutation_success(); + } + result + } +} + +struct ObservedPaykitVerifier<'a> { + inner: &'a dyn CriterionVerifier, + evidence: AtomicU8, +} + +impl<'a> ObservedPaykitVerifier<'a> { + const HEALTHY: u8 = 1; + const UNAVAILABLE: u8 = 2; + + fn new(inner: &'a dyn CriterionVerifier) -> Self { + Self { + inner, + evidence: AtomicU8::new(0), + } + } + + fn evidence(&self) -> PaykitProviderEvidence { + match self.evidence.load(Ordering::Relaxed) { + Self::HEALTHY => PaykitProviderEvidence::HealthyResponse, + Self::UNAVAILABLE => PaykitProviderEvidence::Unavailable, + _ => PaykitProviderEvidence::None, + } + } +} + +#[async_trait] +impl CriterionVerifier for ObservedPaykitVerifier<'_> { + async fn verify( + &self, + request: CriterionVerificationRequest, + ) -> Result { + let result = self.inner.verify(request).await; + match &result { + Err(ApplicationError::VerificationDependencyUnavailable) => { + self.evidence.store(Self::UNAVAILABLE, Ordering::Relaxed); + } + Ok(_) | Err(ApplicationError::VerificationPending) => { + self.evidence + .compare_exchange(0, Self::HEALTHY, Ordering::Relaxed, Ordering::Relaxed) + .ok(); + } + Err(_) => {} + } + result + } +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum VerificationPollErrorDisposition { + RetryableRepository, + Fatal(&'static str), +} + +fn classify_verification_poll_error(error: &ApplicationError) -> VerificationPollErrorDisposition { + match error { + ApplicationError::Storage { .. } => VerificationPollErrorDisposition::RetryableRepository, + ApplicationError::InvalidVerificationTaskState { .. } + | ApplicationError::InvalidVerificationTaskTransition { .. } + | ApplicationError::InvalidVerificationTaskFailureMessage => { + VerificationPollErrorDisposition::Fatal("invalid_verification_task_state") + } + _ => VerificationPollErrorDisposition::Fatal("unexpected_application_error"), + } +} + +fn redacted_fatal_verification_error(error_class: &'static str) -> ApplicationError { + ApplicationError::InvalidVerificationTaskState { + message: format!("verification worker terminated after {error_class}"), + } +} + +fn is_terminal_business_failure(error: &ApplicationError) -> bool { + matches!( + error, + ApplicationError::ContentLockUnavailable + | ApplicationError::EntitlementNotSatisfied + | ApplicationError::UnsupportedVerifierType { .. } + | ApplicationError::Verifier { .. } + | ApplicationError::ContentLockHashMismatch { .. } + | ApplicationError::ContentLockCanonicalization { .. } + | ApplicationError::EmptyContentLockCriteria + | ApplicationError::DuplicateContentLockCriterion { .. } + | ApplicationError::DuplicateVerificationResultCriterion { .. } + | ApplicationError::UnknownVerificationResultCriterion { .. } + ) +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum RepositoryOperation { + QueuePoll, + Mutation, +} + +#[derive(Debug, Default)] +struct VerificationReadinessRecovery { + paykit_provider_degraded: bool, + queue_poll_degraded: bool, + mutation_degraded: bool, +} + +impl VerificationReadinessRecovery { + fn record_poll(&mut self, poll: VerificationPoll, readiness: &WorkerReadiness) { + self.record_repository_evidence(poll.repository, readiness); + match poll.paykit_provider { + PaykitProviderEvidence::HealthyResponse => { + self.paykit_provider_degraded = false; + self.record_ready_if_healthy(readiness); + } + PaykitProviderEvidence::Unavailable => { + self.paykit_provider_degraded = true; + readiness.record( + WorkerKind::Verification, + WorkerReadinessEvidence::TransientDependencyFailure, + ); + } + PaykitProviderEvidence::None => {} + } + } + + fn record_repository_evidence( + &mut self, + evidence: RepositoryOperationEvidence, + readiness: &WorkerReadiness, + ) { + if evidence.queue_poll == OperationEvidence::Succeeded { + self.record_repository_success(RepositoryOperation::QueuePoll, readiness); + } + if evidence.mutation == OperationEvidence::Succeeded { + self.record_repository_success(RepositoryOperation::Mutation, readiness); + } + } + + fn record_repository_failure( + &mut self, + operation: RepositoryOperation, + readiness: &WorkerReadiness, + ) { + match operation { + RepositoryOperation::QueuePoll => self.queue_poll_degraded = true, + RepositoryOperation::Mutation => self.mutation_degraded = true, + } + readiness.record( + WorkerKind::Verification, + WorkerReadinessEvidence::TransientDependencyFailure, + ); + } + + fn record_repository_success( + &mut self, + operation: RepositoryOperation, + readiness: &WorkerReadiness, + ) { + match operation { + RepositoryOperation::QueuePoll => self.queue_poll_degraded = false, + RepositoryOperation::Mutation => self.mutation_degraded = false, + } + self.record_ready_if_healthy(readiness); + } + + fn record_ready_if_healthy(&self, readiness: &WorkerReadiness) { + if !self.paykit_provider_degraded && !self.queue_poll_degraded && !self.mutation_degraded { + readiness.record( + WorkerKind::Verification, + WorkerReadinessEvidence::DependencySucceeded, + ); + } + } +} + /// Result of one worker polling attempt. #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub enum WorkerTick { + Cancelled, Idle, Completed(TaskId), + VerificationPendingRetryScheduled(TaskId), RetryScheduled(TaskId), Failed(TaskId), } @@ -96,24 +421,118 @@ impl<'a> VerificationWorker<'a> { } pub async fn run_once(&self) -> Result { + self.run_once_with_recovery_evidence() + .await + .map(|poll| poll.tick) + .map_err(|failure| failure.error) + } + + async fn run_once_with_recovery_evidence( + &self, + ) -> Result { + let (_shutdown_tx, shutdown_rx) = watch::channel(false); + self.run_once_until_shutdown_with_recovery_evidence(&shutdown_rx) + .await + } + + pub async fn run_once_until_shutdown( + &self, + shutdown: &watch::Receiver, + ) -> Result { + self.run_once_until_shutdown_with_recovery_evidence(shutdown) + .await + .map(|poll| poll.tick) + .map_err(|failure| failure.error) + } + + async fn run_once_until_shutdown_with_recovery_evidence( + &self, + shutdown: &watch::Receiver, + ) -> Result { + let observer = RepositoryOperationObserver::default(); + let claimer = ObservedVerificationTaskClaimer { + inner: self.claimer, + observer: &observer, + }; + match self.run_once_until_shutdown_inner(shutdown, &claimer).await { + Ok(mut poll) => { + poll.repository = observer.evidence(); + Ok(poll) + } + Err((operation, error)) => Err(VerificationPollError { + operation, + error, + repository: observer.evidence(), + }), + } + } + + async fn schedule_claimed_retry( + &self, + claimer: &dyn VerificationTaskClaimer, + task_id: &TaskId, + claim_token: &uuid::Uuid, + ) -> Result, ApplicationError> { + let retry_scheduled_at = self.clock.now(); + let next_attempt_at = retry_scheduled_at + retry_delay(); + claimer + .schedule_verification_task_retry( + task_id, + &self.worker_id, + claim_token, + next_attempt_at - retry_scheduled_at, + ) + .await + .map(|scheduled| scheduled.map(|_| next_attempt_at)) + } + + async fn run_once_until_shutdown_inner( + &self, + shutdown: &watch::Receiver, + claimer: &dyn VerificationTaskClaimer, + ) -> Result { + if *shutdown.borrow() { + return Ok(VerificationPoll::without_provider_evidence( + WorkerTick::Cancelled, + )); + } let now = self.clock.now(); let claim_expires_at = now + claim_timeout(self.claim_timeout_seconds); - let Some(claim) = self - .claimer - .claim_next_verification_task(&self.worker_id, now, claim_expires_at) - .await? + let Some(claim) = claimer + .claim_next_verification_task(&self.worker_id, (claim_expires_at) - (now)) + .await + .map_err(|error| (RepositoryOperation::QueuePoll, error))? else { - return Ok(WorkerTick::Idle); + return Ok(VerificationPoll::without_provider_evidence( + WorkerTick::Idle, + )); }; let task_id = claim.task.task_id; let claim_token = claim.claim_token; + if *shutdown.borrow() { + let _ = claimer + .schedule_verification_task_retry( + &task_id, + &self.worker_id, + &claim_token, + time::Duration::ZERO, + ) + .await + .map_err(|error| (RepositoryOperation::Mutation, error))?; + return Ok(VerificationPoll::without_provider_evidence( + WorkerTick::Cancelled, + )); + } debug!(%task_id, worker_id = %self.worker_id, "claimed verification task"); let mut verifiers = StaticCriterionVerifierRegistry::new(); if self.allow_dev_static_verifier { verifiers = verifiers.with_dev_static(self.dev_static_verifier); } - if let Some(paykit_payment_verifier) = self.paykit_payment_verifier { + let observed_paykit_verifier = self + .paykit_payment_verifier + .map(ObservedPaykitVerifier::new); + if let Some(paykit_payment_verifier) = observed_paykit_verifier.as_ref() { verifiers = verifiers.with_paykit_payment(paykit_payment_verifier); } let use_case = CompleteVerificationTaskUseCase::new( @@ -125,39 +544,66 @@ impl<'a> VerificationWorker<'a> { self.verified_by.clone(), ); - match use_case + let tick = match use_case .execute_claimed( CompleteVerificationTaskRequest { task_id }, claim, &self.worker_id, - self.claimer, + claimer, ) .await { Ok(completed) => { info!(%task_id, status = ?completed.status, "completed verification task"); - Ok(WorkerTick::Completed(task_id)) + WorkerTick::Completed(task_id) + } + Err(ApplicationError::VerificationDependencyUnavailable) => { + let Some(next_attempt_at) = self + .schedule_claimed_retry(claimer, &task_id, &claim_token) + .await + .map_err(|error| (RepositoryOperation::Mutation, error))? + else { + info!( + %task_id, + worker_id = %self.worker_id, + "verification task claim no longer owned; retry not scheduled" + ); + return Ok(VerificationPoll { + tick: WorkerTick::Idle, + paykit_provider: observed_paykit_verifier.as_ref().map_or( + PaykitProviderEvidence::None, + ObservedPaykitVerifier::evidence, + ), + repository: RepositoryOperationEvidence::default(), + }); + }; + debug!( + %task_id, + worker_id = %self.worker_id, + %next_attempt_at, + "scheduled verification task retry after dependency failure" + ); + WorkerTick::RetryScheduled(task_id) } Err(ApplicationError::VerificationPending) => { - let retry_scheduled_at = self.clock.now(); - let next_attempt_at = retry_scheduled_at + retry_delay(); - let Some(_) = self - .claimer - .schedule_verification_task_retry( - &task_id, - &self.worker_id, - &claim_token, - retry_scheduled_at, - next_attempt_at, - ) - .await? + let Some(next_attempt_at) = self + .schedule_claimed_retry(claimer, &task_id, &claim_token) + .await + .map_err(|error| (RepositoryOperation::Mutation, error))? else { info!( %task_id, worker_id = %self.worker_id, "verification task claim no longer owned; retry not scheduled" ); - return Ok(WorkerTick::Idle); + return Ok(VerificationPoll { + tick: WorkerTick::Idle, + paykit_provider: observed_paykit_verifier.as_ref().map_or( + PaykitProviderEvidence::None, + ObservedPaykitVerifier::evidence, + ), + repository: RepositoryOperationEvidence::default(), + }); }; debug!( %task_id, @@ -165,7 +611,7 @@ impl<'a> VerificationWorker<'a> { %next_attempt_at, "scheduled verification task retry" ); - Ok(WorkerTick::RetryScheduled(task_id)) + WorkerTick::VerificationPendingRetryScheduled(task_id) } Err(ApplicationError::VerificationTaskClaimLost) => { info!( @@ -173,13 +619,27 @@ impl<'a> VerificationWorker<'a> { worker_id = %self.worker_id, "verification task claim no longer owned; terminal state not persisted" ); - Ok(WorkerTick::Idle) + WorkerTick::Idle } - Err(error) => { - error!(%task_id, error = %error, "verification task failed"); - Ok(WorkerTick::Failed(task_id)) + Err(error) if is_terminal_business_failure(&error) => { + error!( + %task_id, + error_class = "terminal_business_failure", + retrying = false, + "verification task failed" + ); + WorkerTick::Failed(task_id) } - } + Err(error) => return Err((RepositoryOperation::Mutation, error)), + }; + Ok(VerificationPoll { + tick, + paykit_provider: observed_paykit_verifier.as_ref().map_or( + PaykitProviderEvidence::None, + ObservedPaykitVerifier::evidence, + ), + repository: RepositoryOperationEvidence::default(), + }) } pub async fn run_until_shutdown( @@ -191,7 +651,8 @@ impl<'a> VerificationWorker<'a> { return Ok(()); } - match self.run_once().await? { + match self.run_once_until_shutdown(&shutdown).await? { + WorkerTick::Cancelled => return Ok(()), WorkerTick::Idle => { tokio::select! { _ = shutdown.changed() => { @@ -203,11 +664,91 @@ impl<'a> VerificationWorker<'a> { } } WorkerTick::Completed(_) + | WorkerTick::VerificationPendingRetryScheduled(_) | WorkerTick::RetryScheduled(_) | WorkerTick::Failed(_) => {} } } } + + pub async fn run_until_shutdown_with_readiness( + &self, + mut shutdown: watch::Receiver, + readiness: &WorkerReadiness, + ) -> Result<(), ApplicationError> { + let mut recovery = VerificationReadinessRecovery::default(); + loop { + if *shutdown.borrow() { + readiness.record(WorkerKind::Verification, WorkerReadinessEvidence::Stopped); + return Ok(()); + } + + match self + .run_once_until_shutdown_with_recovery_evidence(&shutdown) + .await + { + Ok(VerificationPoll { + tick: WorkerTick::Cancelled, + .. + }) => { + readiness.record(WorkerKind::Verification, WorkerReadinessEvidence::Stopped); + return Ok(()); + } + Ok( + poll @ VerificationPoll { + tick: WorkerTick::Idle, + .. + }, + ) => { + recovery.record_poll(poll, readiness); + tokio::select! { + _ = shutdown.changed() => {} + _ = tokio::time::sleep(self.poll_interval) => {} + } + } + Ok(poll) => recovery.record_poll(poll, readiness), + Err(failure) => { + recovery.record_repository_evidence(failure.repository, readiness); + match classify_verification_poll_error(&failure.error) { + VerificationPollErrorDisposition::RetryableRepository => { + recovery.record_repository_failure(failure.operation, readiness); + error!( + operation = match failure.operation { + RepositoryOperation::QueuePoll => "verification_queue_poll", + RepositoryOperation::Mutation => + "verification_repository_mutation", + }, + error_class = "repository_unavailable", + retrying = true, + "verification worker repository operation failed" + ); + tokio::select! { + _ = shutdown.changed() => {} + _ = tokio::time::sleep(self.poll_interval) => {} + } + } + VerificationPollErrorDisposition::Fatal(error_class) => { + readiness.record( + WorkerKind::Verification, + WorkerReadinessEvidence::UnexpectedExit, + ); + error!( + operation = match failure.operation { + RepositoryOperation::QueuePoll => "verification_queue_poll", + RepositoryOperation::Mutation => + "verification_repository_mutation", + }, + error_class, + retrying = false, + "verification worker terminated after unexpected application error" + ); + return Err(redacted_fatal_verification_error(error_class)); + } + } + } + } + } + } } fn claim_timeout(seconds: u64) -> time::Duration { @@ -237,10 +778,12 @@ mod tests { }; use locks_service::application::errors::ApplicationError; use locks_service::application::models::{ - CriterionVerificationRequest, VerificationTaskRecord, VerificationTaskStatus, + ClaimedVerificationTask, CriterionVerificationRequest, VerificationTaskRecord, + VerificationTaskStatus, }; use locks_service::application::ports::{ - ContentLockRepository, CriterionVerifier, EntitlementRepository, VerificationTaskRepository, + ContentLockRepository, CriterionVerifier, EntitlementRepository, VerificationTaskClaimer, + VerificationTaskRepository, }; use locks_service::infrastructure::memory::{ content_locks::InMemoryContentLockRepository, entitlements::InMemoryEntitlementRepository, @@ -248,16 +791,23 @@ mod tests { verification_tasks::InMemoryVerificationTaskRepository, }; use locks_service::infrastructure::verifiers::dev_static::DevStaticVerifier; + use locks_service::infrastructure::verifiers::paykit_payment::{ + PaykitPaymentStatus, PaykitPaymentStatusClient, PaykitPaymentStatusError, + PaykitPaymentStatusKind, PaykitPaymentVerifier, + }; use time::macros::datetime; - use tokio::sync::watch; + use tokio::sync::{Notify, watch}; - use crate::app_state::{AppState, SystemClock}; + use crate::app_state::{AppState, ReadinessStatus, SystemClock, WorkerReadiness}; use crate::config::{ ContentLocksConfig, DatabaseConfig, LockServerCredentialsConfig, LockServerRuntimeConfig, LoggingConfig, PubkyConfig, RateLimitsConfig, RuntimeConfig, RuntimeEnvironment, SecretsConfig, WorkerConfig, }; - use crate::worker::{VerificationWorker, WorkerTick, retry_delay}; + use crate::worker::{ + VerificationPollErrorDisposition, VerificationWorker, WorkerTick, + classify_verification_poll_error, redacted_fatal_verification_error, retry_delay, + }; const TASK_ID: &str = "018fc6ec-2f3d-4f7e-8b7d-6f5c4b3a2d10"; const BUNDLE_ID: &str = "000G40R40M30E209185GR38E1W"; @@ -318,11 +868,51 @@ mod tests { fixture.seed_task().await; let verifier = RetryOnceVerifier::default(); let worker = fixture.worker_with_verifier(&verifier); + let readiness = WorkerReadiness::new(true, false); + let mut recovery = super::VerificationReadinessRecovery::default(); + let poll = worker.run_once_with_recovery_evidence().await.unwrap(); assert_eq!( - worker.run_once().await.unwrap(), - WorkerTick::RetryScheduled(task_id()) + poll.tick, + WorkerTick::VerificationPendingRetryScheduled(task_id()) ); + assert_eq!( + poll.repository.queue_poll, + super::OperationEvidence::Succeeded + ); + assert_eq!( + poll.repository.mutation, + super::OperationEvidence::Succeeded + ); + recovery.record_poll(poll, &readiness); + assert_eq!(readiness.status(), ReadinessStatus::Ready); + let stored = fixture + .tasks + .get_verification_task(&task_id()) + .await + .unwrap() + .unwrap(); + assert_eq!(stored.status, VerificationTaskStatus::Pending); + assert_eq!(stored.failure_message, None); + assert_eq!(worker.run_once().await.unwrap(), WorkerTick::Idle); + } + + #[tokio::test] + async fn paykit_provider_outage_schedules_retry_and_degrades_readiness() { + let fixture = WorkerFixture::new(paykit_content_lock()).await; + fixture.seed_task().await; + let status_client = FakePaykitStatusClient::failing(); + let verifier = PaykitPaymentVerifier::new(&status_client, 1); + let worker = fixture.worker_with_paykit_verifier(&verifier); + let readiness = WorkerReadiness::new(true, false); + let mut recovery = super::VerificationReadinessRecovery::default(); + + let poll = worker.run_once_with_recovery_evidence().await.unwrap(); + + assert_eq!(poll.tick, WorkerTick::RetryScheduled(task_id())); + recovery.record_poll(poll, &readiness); + assert_eq!(readiness.status(), ReadinessStatus::Degraded); + assert_eq!(status_client.calls.load(Ordering::SeqCst), 1); let stored = fixture .tasks .get_verification_task(&task_id()) @@ -334,6 +924,95 @@ mod tests { assert_eq!(worker.run_once().await.unwrap(), WorkerTick::Idle); } + #[tokio::test] + async fn healthy_paykit_pending_schedules_retry_without_degrading_readiness() { + let fixture = WorkerFixture::new(paykit_content_lock()).await; + fixture.seed_task().await; + let status_client = FakePaykitStatusClient::healthy_pending(); + let verifier = PaykitPaymentVerifier::new(&status_client, 1); + let worker = fixture.worker_with_paykit_verifier(&verifier); + let readiness = WorkerReadiness::new(true, false); + let mut recovery = super::VerificationReadinessRecovery::default(); + + let poll = worker.run_once_with_recovery_evidence().await.unwrap(); + + assert_eq!( + poll.tick, + WorkerTick::VerificationPendingRetryScheduled(task_id()) + ); + recovery.record_poll(poll, &readiness); + assert_eq!(readiness.status(), ReadinessStatus::Ready); + assert_eq!(status_client.calls.load(Ordering::SeqCst), 1); + let stored = fixture + .tasks + .get_verification_task(&task_id()) + .await + .unwrap() + .unwrap(); + assert_eq!(stored.status, VerificationTaskStatus::Pending); + assert_eq!(stored.failure_message, None); + assert_eq!(worker.run_once().await.unwrap(), WorkerTick::Idle); + } + + #[tokio::test] + async fn paykit_degradation_ignores_unrelated_work_until_healthy_paykit_response() { + let readiness = WorkerReadiness::new(true, false); + let mut recovery = super::VerificationReadinessRecovery::default(); + + let outage_fixture = WorkerFixture::new(paykit_content_lock()).await; + outage_fixture.seed_task().await; + let outage_client = FakePaykitStatusClient::failing(); + let outage_verifier = PaykitPaymentVerifier::new(&outage_client, 1); + let outage_worker = outage_fixture.worker_with_paykit_verifier(&outage_verifier); + recovery.record_poll( + outage_worker + .run_once_with_recovery_evidence() + .await + .unwrap(), + &readiness, + ); + assert_eq!(readiness.status(), ReadinessStatus::Degraded); + + let completed_fixture = WorkerFixture::new(content_lock(true)).await; + completed_fixture.seed_task().await; + recovery.record_poll( + completed_fixture + .worker() + .run_once_with_recovery_evidence() + .await + .unwrap(), + &readiness, + ); + assert_eq!(readiness.status(), ReadinessStatus::Degraded); + + let pending_fixture = WorkerFixture::new(content_lock(true)).await; + pending_fixture.seed_task().await; + let pending_verifier = RetryOnceVerifier::default(); + recovery.record_poll( + pending_fixture + .worker_with_verifier(&pending_verifier) + .run_once_with_recovery_evidence() + .await + .unwrap(), + &readiness, + ); + assert_eq!(readiness.status(), ReadinessStatus::Degraded); + + let healthy_fixture = WorkerFixture::new(paykit_content_lock()).await; + healthy_fixture.seed_task().await; + let healthy_client = FakePaykitStatusClient::healthy_pending(); + let healthy_verifier = PaykitPaymentVerifier::new(&healthy_client, 1); + recovery.record_poll( + healthy_fixture + .worker_with_paykit_verifier(&healthy_verifier) + .run_once_with_recovery_evidence() + .await + .unwrap(), + &readiness, + ); + assert_eq!(readiness.status(), ReadinessStatus::Ready); + } + #[tokio::test] async fn worker_without_dev_static_registration_fails_dev_static_tasks() { let fixture = WorkerFixture::new(content_lock(true)).await; @@ -391,11 +1070,185 @@ mod tests { .unwrap(); } + #[tokio::test] + async fn shutdown_during_claim_releases_claim_without_verification_execution() { + let fixture = WorkerFixture::new(content_lock(true)).await; + fixture.seed_task().await; + let blocking = BlockingClaimer::new(&fixture.claimer); + let worker = fixture.worker_with_claimer(&blocking); + let (shutdown_tx, shutdown_rx) = watch::channel(false); + + let (result, ()) = tokio::join!(worker.run_once_until_shutdown(&shutdown_rx), async { + blocking.claim_entered.notified().await; + shutdown_tx.send(true).unwrap(); + blocking.release_claim.notify_one(); + }); + + assert_eq!(result.unwrap(), WorkerTick::Cancelled); + assert_eq!(blocking.publication_calls.load(Ordering::SeqCst), 0); + assert_eq!(blocking.retry_calls.load(Ordering::SeqCst), 1); + } + #[test] fn pending_verification_retry_is_independent_of_queue_polling() { assert_eq!(retry_delay(), time::Duration::seconds(30)); } + #[tokio::test] + async fn unexpected_verification_error_terminates_not_ready_with_redacted_class() { + let fixture = WorkerFixture::new(content_lock(true)).await; + fixture.seed_task().await; + let verifier = UnexpectedVerifier("super-secret-verifier-detail".to_owned()); + let worker = fixture.worker_with_verifier(&verifier); + let readiness = WorkerReadiness::new(true, false); + let (_shutdown_tx, shutdown_rx) = watch::channel(false); + + let error = worker + .run_until_shutdown_with_readiness(shutdown_rx, &readiness) + .await + .unwrap_err(); + + assert_eq!(readiness.status(), ReadinessStatus::NotReady); + assert!( + error + .to_string() + .contains("invalid_verification_task_state") + ); + assert!(!error.to_string().contains("super-secret-verifier-detail")); + } + + #[test] + fn ordinary_verification_pending_retry_does_not_degrade_readiness() { + let readiness = WorkerReadiness::new(true, false); + let mut recovery = super::VerificationReadinessRecovery::default(); + + recovery.record_repository_success(super::RepositoryOperation::QueuePoll, &readiness); + recovery.record_poll( + super::VerificationPoll::without_provider_evidence( + WorkerTick::VerificationPendingRetryScheduled(task_id()), + ), + &readiness, + ); + + assert_eq!(readiness.status(), ReadinessStatus::Ready); + } + + #[tokio::test] + async fn successful_idle_poll_reports_only_queue_poll_recovery() { + let fixture = WorkerFixture::empty(); + + let poll = fixture + .worker() + .run_once_with_recovery_evidence() + .await + .unwrap(); + + assert_eq!(poll.tick, WorkerTick::Idle); + assert_eq!( + poll.repository.queue_poll, + super::OperationEvidence::Succeeded + ); + assert_eq!(poll.repository.mutation, super::OperationEvidence::None); + } + + #[test] + fn repository_mutation_failure_ignores_outcomes_until_mutation_succeeds() { + let readiness = WorkerReadiness::new(true, false); + let mut recovery = super::VerificationReadinessRecovery::default(); + + recovery.record_repository_failure(super::RepositoryOperation::Mutation, &readiness); + for tick in [ + WorkerTick::Idle, + WorkerTick::Completed(task_id()), + WorkerTick::VerificationPendingRetryScheduled(task_id()), + WorkerTick::Failed(task_id()), + ] { + recovery.record_poll( + super::VerificationPoll::without_provider_evidence(tick), + &readiness, + ); + assert_eq!(readiness.status(), ReadinessStatus::Degraded); + } + + recovery.record_repository_success(super::RepositoryOperation::Mutation, &readiness); + assert_eq!(readiness.status(), ReadinessStatus::Ready); + } + + #[test] + fn queue_poll_failure_recovers_only_from_successful_queue_poll() { + let readiness = WorkerReadiness::new(true, false); + let mut recovery = super::VerificationReadinessRecovery::default(); + + recovery.record_repository_failure(super::RepositoryOperation::QueuePoll, &readiness); + recovery.record_repository_success(super::RepositoryOperation::Mutation, &readiness); + assert_eq!(readiness.status(), ReadinessStatus::Degraded); + + recovery.record_repository_success(super::RepositoryOperation::QueuePoll, &readiness); + assert_eq!(readiness.status(), ReadinessStatus::Ready); + } + + #[test] + fn provider_failure_stays_degraded_across_business_outcomes_until_provider_success() { + let readiness = WorkerReadiness::new(true, false); + let mut recovery = super::VerificationReadinessRecovery::default(); + + recovery.record_poll( + super::VerificationPoll { + tick: WorkerTick::RetryScheduled(task_id()), + paykit_provider: super::PaykitProviderEvidence::Unavailable, + repository: super::RepositoryOperationEvidence::default(), + }, + &readiness, + ); + recovery.record_poll( + super::VerificationPoll::without_provider_evidence(WorkerTick::Failed(task_id())), + &readiness, + ); + recovery.record_poll( + super::VerificationPoll::without_provider_evidence(WorkerTick::Idle), + &readiness, + ); + assert_eq!(readiness.status(), ReadinessStatus::Degraded); + + recovery.record_poll( + super::VerificationPoll { + tick: WorkerTick::VerificationPendingRetryScheduled(task_id()), + paykit_provider: super::PaykitProviderEvidence::HealthyResponse, + repository: super::RepositoryOperationEvidence::default(), + }, + &readiness, + ); + assert_eq!(readiness.status(), ReadinessStatus::Ready); + } + + #[test] + fn verification_errors_retry_only_storage_and_redact_fatal_details() { + let secret = "postgres://user:password@example.test/locks"; + assert_eq!( + classify_verification_poll_error(&ApplicationError::Storage { + message: secret.to_owned(), + }), + VerificationPollErrorDisposition::RetryableRepository + ); + assert_eq!( + classify_verification_poll_error(&ApplicationError::InvalidVerificationTaskState { + message: secret.to_owned(), + }), + VerificationPollErrorDisposition::Fatal("invalid_verification_task_state") + ); + assert_eq!( + classify_verification_poll_error(&ApplicationError::MissingRecord { + record: "verification_task", + }), + VerificationPollErrorDisposition::Fatal("unexpected_application_error") + ); + + let redacted = + redacted_fatal_verification_error("invalid_verification_task_state").to_string(); + assert!(redacted.contains("invalid_verification_task_state")); + assert!(!redacted.contains(secret)); + } + #[tokio::test] async fn worker_tick_debug_does_not_expose_submitted_proof_payload() { let fixture = WorkerFixture::new(content_lock_with_payload(json_secret_payload())).await; @@ -458,6 +1311,29 @@ mod tests { self.worker_with_verifier(&self.verifier) } + fn worker_with_claimer<'a>( + &'a self, + claimer: &'a dyn VerificationTaskClaimer, + ) -> VerificationWorker<'a> { + VerificationWorker::new( + self.tasks.as_ref(), + claimer, + &self.content_locks, + &self.entitlements, + &self.verifier, + None, + true, + &self.clock, + LockServerPubky::from_str( + "pubky7ir1ttte48bcp4zjychjyscicrwi1j34mtt91ptsafdbjmr8g9eo", + ) + .unwrap(), + "test-worker".to_owned(), + std::time::Duration::from_millis(10), + 60, + ) + } + fn worker_with_verifier<'a>( &'a self, verifier: &'a dyn CriterionVerifier, @@ -481,6 +1357,29 @@ mod tests { ) } + fn worker_with_paykit_verifier<'a>( + &'a self, + verifier: &'a dyn CriterionVerifier, + ) -> VerificationWorker<'a> { + VerificationWorker::new( + self.tasks.as_ref(), + &self.claimer, + &self.content_locks, + &self.entitlements, + &self.verifier, + Some(verifier), + false, + &self.clock, + LockServerPubky::from_str( + "pubky7ir1ttte48bcp4zjychjyscicrwi1j34mtt91ptsafdbjmr8g9eo", + ) + .unwrap(), + "test-worker".to_owned(), + std::time::Duration::from_millis(10), + 60, + ) + } + fn worker_without_dev_static_registration(&self) -> VerificationWorker<'_> { VerificationWorker::new( self.tasks.as_ref(), @@ -524,6 +1423,91 @@ mod tests { } } + struct BlockingClaimer<'a> { + inner: &'a dyn VerificationTaskClaimer, + claim_entered: Notify, + release_claim: Notify, + publication_calls: std::sync::atomic::AtomicUsize, + retry_calls: std::sync::atomic::AtomicUsize, + } + + impl<'a> BlockingClaimer<'a> { + fn new(inner: &'a dyn VerificationTaskClaimer) -> Self { + Self { + inner, + claim_entered: Notify::new(), + release_claim: Notify::new(), + publication_calls: std::sync::atomic::AtomicUsize::new(0), + retry_calls: std::sync::atomic::AtomicUsize::new(0), + } + } + } + + #[async_trait] + impl VerificationTaskClaimer for BlockingClaimer<'_> { + async fn begin_claimed_entitlement_publication( + &self, + task_id: &TaskId, + worker_id: &str, + claim_token: &uuid::Uuid, + ) -> Result { + self.publication_calls.fetch_add(1, Ordering::SeqCst); + self.inner + .begin_claimed_entitlement_publication(task_id, worker_id, claim_token) + .await + } + + async fn claim_next_verification_task( + &self, + worker_id: &str, + claim_ttl: time::Duration, + ) -> Result, ApplicationError> { + self.claim_entered.notify_one(); + self.release_claim.notified().await; + self.inner + .claim_next_verification_task(worker_id, claim_ttl) + .await + } + + async fn schedule_verification_task_retry( + &self, + task_id: &TaskId, + worker_id: &str, + claim_token: &uuid::Uuid, + retry_after: time::Duration, + ) -> Result, ApplicationError> { + self.retry_calls.fetch_add(1, Ordering::SeqCst); + self.inner + .schedule_verification_task_retry(task_id, worker_id, claim_token, retry_after) + .await + } + + async fn persist_claimed_verification_task_transition( + &self, + task: VerificationTaskRecord, + worker_id: &str, + claim_token: &uuid::Uuid, + ) -> Result, ApplicationError> { + self.inner + .persist_claimed_verification_task_transition(task, worker_id, claim_token) + .await + } + } + + struct UnexpectedVerifier(String); + + #[async_trait] + impl CriterionVerifier for UnexpectedVerifier { + async fn verify( + &self, + _request: CriterionVerificationRequest, + ) -> Result { + Err(ApplicationError::InvalidVerificationTaskState { + message: self.0.clone(), + }) + } + } + #[derive(Default)] struct RetryOnceVerifier { returned_pending: AtomicBool, @@ -542,6 +1526,43 @@ mod tests { } } + struct FakePaykitStatusClient { + response: Result, + calls: std::sync::atomic::AtomicUsize, + } + + impl FakePaykitStatusClient { + fn failing() -> Self { + Self { + response: Err(PaykitPaymentStatusError), + calls: std::sync::atomic::AtomicUsize::new(0), + } + } + + fn healthy_pending() -> Self { + Self { + response: Ok(PaykitPaymentStatus { + status: PaykitPaymentStatusKind::Detected, + confirmations: 0, + amount_matched: true, + }), + calls: std::sync::atomic::AtomicUsize::new(0), + } + } + } + + #[async_trait] + impl PaykitPaymentStatusClient for &FakePaykitStatusClient { + async fn transaction_status( + &self, + _creator: &CreatorPubky, + _bundle_id: &BundleId, + ) -> Result { + self.calls.fetch_add(1, Ordering::SeqCst); + self.response + } + } + fn task_for(content_lock: &ContentLock, payload: serde_json::Value) -> VerificationTaskRecord { VerificationTaskRecord { task_id: task_id(), @@ -569,7 +1590,7 @@ mod tests { reader_public_key: None, proofs: vec![Proof { criterion_id: "criterion-1".to_owned(), - verifier_type: VerifierType::DevStatic, + verifier_type: content_lock.criteria[0].verifier_type, payload, }], } @@ -579,6 +1600,17 @@ mod tests { content_lock_with_payload(serde_json::json!({ "satisfied": satisfied })) } + fn paykit_content_lock() -> ContentLock { + let mut content_lock = content_lock_with_payload(serde_json::json!({ + "recipient_pubky": creator().to_string(), + "amount": "50000", + "asset": "BTC", + "payment_in": 24 + })); + content_lock.criteria[0].verifier_type = VerifierType::PaykitPayment; + content_lock + } + fn content_lock_with_payload(params: serde_json::Value) -> ContentLock { ContentLock { version: CONTENT_LOCK_VERSION, @@ -645,6 +1677,7 @@ mod tests { rate_limits: RateLimitsConfig::default(), content_locks: ContentLocksConfig::default(), deletion: crate::config::DeletionConfig::default(), + deletion_worker: crate::config::DeletionWorkerConfig::default(), paykit: None, } } diff --git a/locks-server/tests/startup_lifecycle.rs b/locks-server/tests/startup_lifecycle.rs new file mode 100644 index 0000000..250d6b0 --- /dev/null +++ b/locks-server/tests/startup_lifecycle.rs @@ -0,0 +1,84 @@ +use std::{ + net::SocketAddr, + sync::{ + Arc, + atomic::{AtomicUsize, Ordering}, + }, + time::{Duration, Instant}, +}; + +use locks_server::runtime::{InitialStartupOutcome, bind_listener_then_run_initial}; +use tokio::sync::{Notify, oneshot}; + +#[tokio::test] +async fn listener_bind_failure_does_not_attempt_initial_publication() { + let occupied = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let bind_addr = occupied.local_addr().unwrap(); + let publication_calls = Arc::new(AtomicUsize::new(0)); + let observed_calls = Arc::clone(&publication_calls); + let mut shutdown = Box::pin(std::future::pending()); + + let error = bind_listener_then_run_initial( + bind_addr, + Duration::from_millis(100), + &mut shutdown, + move || async move { + observed_calls.fetch_add(1, Ordering::SeqCst); + Ok::<_, anyhow::Error>(()) + }, + ) + .await + .unwrap_err(); + + assert!(error.to_string().contains("bind")); + assert_eq!(publication_calls.load(Ordering::SeqCst), 0); +} + +#[tokio::test] +async fn blocked_initial_publication_stops_when_shutdown_is_requested() { + let (shutdown_tx, shutdown_rx) = oneshot::channel(); + let mut shutdown = Box::pin(async move { + let _ = shutdown_rx.await; + }); + let publication_started = Arc::new(Notify::new()); + let observed_start = Arc::clone(&publication_started); + let bind_addr: SocketAddr = "127.0.0.1:0".parse().unwrap(); + + let startup = bind_listener_then_run_initial( + bind_addr, + Duration::from_secs(1), + &mut shutdown, + move || async move { + observed_start.notify_one(); + std::future::pending::>().await + }, + ); + let request_shutdown = async { + publication_started.notified().await; + shutdown_tx.send(()).unwrap(); + }; + let (result, ()) = tokio::join!(startup, request_shutdown); + + assert!(matches!( + result.unwrap(), + InitialStartupOutcome::ShutdownRequested + )); +} + +#[tokio::test] +async fn blocked_initial_publication_is_bounded_by_lifecycle_timeout() { + let mut shutdown = Box::pin(std::future::pending()); + let started = Instant::now(); + + let error = bind_listener_then_run_initial( + "127.0.0.1:0".parse().unwrap(), + Duration::from_millis(20), + &mut shutdown, + || async { std::future::pending::>().await }, + ) + .await + .unwrap_err(); + + assert!(error.to_string().contains("initial startup exceeded")); + assert!(started.elapsed() < Duration::from_millis(250)); +} diff --git a/locks-service/migrations/0017_content_lock_deletion_resource_replaced.sql b/locks-service/migrations/0017_content_lock_deletion_resource_replaced.sql new file mode 100644 index 0000000..a2b3b70 --- /dev/null +++ b/locks-service/migrations/0017_content_lock_deletion_resource_replaced.sql @@ -0,0 +1,15 @@ +ALTER TABLE content_lock_deletion_jobs + DROP CONSTRAINT content_lock_deletion_jobs_failure_valid; + +ALTER TABLE content_lock_deletion_jobs + ADD CONSTRAINT content_lock_deletion_jobs_failure_valid CHECK ( + (state = 'failed' AND failure_code IN ( + 'tombstone_missing', + 'tombstone_replaced', + 'resource_replaced', + 'retry_exhausted', + 'state_corrupt' + )) + OR + (state <> 'failed' AND failure_code IS NULL) + ); diff --git a/locks-service/src/application/errors.rs b/locks-service/src/application/errors.rs index 5937ed2..6417919 100644 --- a/locks-service/src/application/errors.rs +++ b/locks-service/src/application/errors.rs @@ -48,6 +48,9 @@ pub enum ApplicationError { /// Criterion verifier is not terminal yet and should be retried later. #[error("verification pending")] VerificationPending, + /// A verification provider is transiently unavailable and should be retried later. + #[error("verification dependency unavailable")] + VerificationDependencyUnavailable, /// Submitted payment proof does not match its canonical content lock criterion. #[error("invalid paykit payment submission")] InvalidPaykitPaymentSubmission, diff --git a/locks-service/src/application/models/access.rs b/locks-service/src/application/models/access.rs index 6caa741..9e75db9 100644 --- a/locks-service/src/application/models/access.rs +++ b/locks-service/src/application/models/access.rs @@ -70,6 +70,28 @@ pub struct FinalCredentialContext { pub bundle_id: BundleId, } +/// Secret-free identity of an eligible deletion snapshot awaiting final credential issuance. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct FinalCredentialMaterialization { + pub creator: CreatorPubky, + pub bundle_id: BundleId, +} + +/// Immutable final-access windows established by the storage serialization winner. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct FinalAccessWindows { + pub issuance_started_at: OffsetDateTime, + pub credential_issuance_deadline: OffsetDateTime, + pub read_deadline: OffsetDateTime, +} + +/// Closed result of claim-fenced final-access window initialization. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum InitializeFinalAccessWindowsResult { + Initialized(FinalAccessWindows), + ClaimLost, +} + /// A deletion credential returned only after its encrypted bearer is durable. #[derive(Clone, PartialEq, Eq)] pub struct IssuedDeletionCredential { diff --git a/locks-service/src/application/models/content_lock_deletion.rs b/locks-service/src/application/models/content_lock_deletion.rs index 56b24da..0cd2e46 100644 --- a/locks-service/src/application/models/content_lock_deletion.rs +++ b/locks-service/src/application/models/content_lock_deletion.rs @@ -23,6 +23,7 @@ pub enum ContentLockDeletionState { pub enum ContentLockDeletionFailureCode { TombstoneMissing, TombstoneReplaced, + ResourceReplaced, RetryExhausted, StateCorrupt, } @@ -33,6 +34,7 @@ impl ContentLockDeletionFailureCode { match self { Self::TombstoneMissing => "tombstone_missing", Self::TombstoneReplaced => "tombstone_replaced", + Self::ResourceReplaced => "resource_replaced", Self::RetryExhausted => "retry_exhausted", Self::StateCorrupt => "state_corrupt", } @@ -46,6 +48,7 @@ impl FromStr for ContentLockDeletionFailureCode { match value { "tombstone_missing" => Ok(Self::TombstoneMissing), "tombstone_replaced" => Ok(Self::TombstoneReplaced), + "resource_replaced" => Ok(Self::ResourceReplaced), "retry_exhausted" => Ok(Self::RetryExhausted), "state_corrupt" => Ok(Self::StateCorrupt), _ => Err(ApplicationError::InvalidContentLockDeletionState { @@ -109,6 +112,28 @@ pub struct ClaimedContentLockDeletionJob { pub claim_token: Uuid, } +/// Transactionally authoritative result of a requested deletion phase advance. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum AdvanceContentLockDeletionPhaseResult { + /// The exact live claim advanced and the returned job reflects the committed phase. + Advanced(Box), + /// The claim was no longer live when the repository serialized the transition. + ClaimLost, + /// The claim remains live, but healthy access obligations must drain before advancing. + ObligationsPending, + /// The claim remains live, but a closed failure requires immediate fail-closed terminalization. + TerminalFailure(ContentLockDeletionFailureCode), +} + +impl AdvanceContentLockDeletionPhaseResult { + pub fn advanced(self) -> Option { + match self { + Self::Advanced(job) => Some(*job), + Self::ClaimLost | Self::ObligationsPending | Self::TerminalFailure(_) => None, + } + } +} + /// Durable decision made while serializing force deletion against graceful lifecycle state. #[derive(Debug, Clone, PartialEq, Eq)] pub enum PrepareForceDeletionResult { diff --git a/locks-service/src/application/ports/access.rs b/locks-service/src/application/ports/access.rs index f5ad1e5..1699555 100644 --- a/locks-service/src/application/ports/access.rs +++ b/locks-service/src/application/ports/access.rs @@ -1,14 +1,25 @@ use async_trait::async_trait; use locks_core::ids::{BundleId, CreatorPubky, LockId}; -use time::OffsetDateTime; +use time::{Duration, OffsetDateTime}; use uuid::Uuid; use crate::application::errors::ApplicationError; use crate::application::models::{ AccessCredential, AccessCredentialLookupKey, AccessCredentialRecord, DeletionReadAuthorization, - IssuedDeletionCredential, + FinalCredentialMaterialization, InitializeFinalAccessWindowsResult, IssuedDeletionCredential, }; +/// Exact worker ownership and candidate material for one final-credential winner operation. +pub struct FinalCredentialWorkerIssueRequest<'a> { + pub deletion_job_id: Uuid, + pub worker_id: &'a str, + pub claim_token: Uuid, + pub creator: &'a CreatorPubky, + pub bundle_id: &'a BundleId, + pub now: OffsetDateTime, + pub candidate: AccessCredential, +} + /// Store for issued opaque access credentials. #[async_trait] pub trait AccessCredentialStore: Send + Sync { @@ -43,11 +54,22 @@ pub trait AccessCredentialStore: Send + Sync { _deletion_job_id: Uuid, _worker_id: &str, _claim_token: Uuid, - _now: OffsetDateTime, - _issuance_deadline: OffsetDateTime, - _read_deadline: OffsetDateTime, - ) -> Result { - Ok(false) + _issuance_window: Duration, + _read_window: Duration, + ) -> Result { + Ok(InitializeFinalAccessWindowsResult::ClaimLost) + } + + /// Enumerates eligible snapshots awaiting final credential materialization under an exact + /// live deletion-worker claim. Implementations return deterministic bundle ordering. + async fn final_credentials_to_materialize( + &self, + _deletion_job_id: Uuid, + _worker_id: &str, + _claim_token: Uuid, + _limit: usize, + ) -> Result, ApplicationError> { + Ok(Vec::new()) } async fn issue_or_replay_final_credential( @@ -60,6 +82,15 @@ pub trait AccessCredentialStore: Send + Sync { Ok(None) } + /// Issues or replays one final credential only while the exact deletion-worker claim remains + /// live. Implementations revalidate ownership and fresh time in the winner transaction. + async fn issue_or_replay_final_credential_for_worker( + &self, + _request: FinalCredentialWorkerIssueRequest<'_>, + ) -> Result, ApplicationError> { + Ok(None) + } + /// Reports whether this deletion Bundle may issue or replay its final credential now. async fn final_credential_available( &self, @@ -74,8 +105,7 @@ pub trait AccessCredentialStore: Send + Sync { &self, _lookup_key: &AccessCredentialLookupKey, _path: &str, - _now: OffsetDateTime, - _claim_expires_at: OffsetDateTime, + _claim_duration: Duration, ) -> Result, ApplicationError> { Ok(None) } @@ -104,7 +134,6 @@ pub trait AccessCredentialStore: Send + Sync { _lookup_key: &AccessCredentialLookupKey, _path: &str, _claim_token: Uuid, - _now: OffsetDateTime, ) -> Result { Ok(false) } diff --git a/locks-service/src/application/ports/content_lock_deletion.rs b/locks-service/src/application/ports/content_lock_deletion.rs index 972062e..3b9fce5 100644 --- a/locks-service/src/application/ports/content_lock_deletion.rs +++ b/locks-service/src/application/ports/content_lock_deletion.rs @@ -6,8 +6,9 @@ use uuid::Uuid; use crate::application::{ errors::ApplicationError, models::{ - ClaimedContentLockDeletionJob, ContentLockDeletionFailureCode, ContentLockDeletionJob, - ContentLockDeletionPhase, PrepareForceDeletionResult, + AdvanceContentLockDeletionPhaseResult, ClaimedContentLockDeletionJob, + ContentLockDeletionFailureCode, ContentLockDeletionJob, ContentLockDeletionPhase, + PrepareForceDeletionResult, }, }; @@ -56,8 +57,7 @@ pub trait ContentLockDeletionRepository: Send + Sync { async fn claim_next( &self, worker_id: &str, - now: OffsetDateTime, - claim_expires_at: OffsetDateTime, + claim_ttl: time::Duration, ) -> Result, ApplicationError>; async fn schedule_retry( @@ -65,8 +65,17 @@ pub trait ContentLockDeletionRepository: Send + Sync { job_id: Uuid, worker_id: &str, claim_token: Uuid, - now: OffsetDateTime, - next_attempt_at: OffsetDateTime, + retry_after: time::Duration, + ) -> Result, ApplicationError>; + + /// Releases a healthy-poll claim and schedules its next observation without + /// charging the claim-acquisition increment to the transient failure budget. + async fn defer( + &self, + job_id: Uuid, + worker_id: &str, + claim_token: Uuid, + defer_for: time::Duration, ) -> Result, ApplicationError>; async fn advance_phase( @@ -74,9 +83,21 @@ pub trait ContentLockDeletionRepository: Send + Sync { job_id: Uuid, worker_id: &str, claim_token: Uuid, - now: OffsetDateTime, next_phase: ContentLockDeletionPhase, - ) -> Result, ApplicationError>; + ) -> Result; + + /// Expires every unresolved frozen non-Paykit task under the exact live deletion claim. + async fn expire_unresolved_non_paykit_tasks( + &self, + job_id: Uuid, + worker_id: &str, + claim_token: Uuid, + ) -> Result { + let _ = (job_id, worker_id, claim_token); + Err(ApplicationError::InvalidContentLockDeletionState { + message: "non-Paykit deletion drain is not supported by this repository".to_owned(), + }) + } /// Persists terminal completion or a stable secret-free failure under the exact lease. async fn finish( @@ -84,7 +105,6 @@ pub trait ContentLockDeletionRepository: Send + Sync { job_id: Uuid, worker_id: &str, claim_token: Uuid, - now: OffsetDateTime, failure_code: Option, ) -> Result, ApplicationError>; @@ -101,9 +121,16 @@ pub trait ContentLockDeletionRepository: Send + Sync { &self, creator: &CreatorPubky, lock_id: &LockId, - forced_at: OffsetDateTime, ) -> Result; + /// Finalizes force deletion only for the exact live worker claim that observed the effects. + async fn complete_force_deletion( + &self, + job_id: Uuid, + worker_id: &str, + claim_token: Uuid, + ) -> Result; + async fn has_force_receipt( &self, creator: &CreatorPubky, diff --git a/locks-service/src/application/ports/content_lock_deletion_action_ownership.rs b/locks-service/src/application/ports/content_lock_deletion_action_ownership.rs new file mode 100644 index 0000000..3cb3c10 --- /dev/null +++ b/locks-service/src/application/ports/content_lock_deletion_action_ownership.rs @@ -0,0 +1,40 @@ +use async_trait::async_trait; +use uuid::Uuid; + +use crate::application::{errors::ApplicationError, models::ContentLockDeletionPhase}; + +/// Exact expected live claim and lifecycle state for one external action lane. +#[derive(Debug, Clone, Copy)] +pub struct ContentLockDeletionActionClaim<'a> { + pub job_id: Uuid, + pub worker_id: &'a str, + pub claim_token: Uuid, + pub expected_phase: ContentLockDeletionPhase, + pub force: bool, +} + +/// Owns one job's external side-effect lane until explicitly released or dropped. +#[async_trait] +pub trait ContentLockDeletionActionGuard: Send { + /// Releases ownership. Implementations must not return a still-locked + /// connection or equivalent resource to shared storage. + async fn release(self: Box) -> Result<(), ApplicationError>; +} + +/// Closed result of post-lock live-claim validation. +pub enum ContentLockDeletionActionAcquireResult { + Acquired(Box), + Busy, + ClaimLost, +} + +/// Nonblocking per-job ownership boundary for deletion external actions. +#[async_trait] +pub trait ContentLockDeletionActionOwnership: Send + Sync { + /// Acquires the per-job lane, then validates the exact claim against + /// storage-authoritative time and expected lifecycle state. + async fn try_acquire( + &self, + claim: ContentLockDeletionActionClaim<'_>, + ) -> Result; +} diff --git a/locks-service/src/application/ports/content_lock_tombstone.rs b/locks-service/src/application/ports/content_lock_tombstone.rs new file mode 100644 index 0000000..c61feee --- /dev/null +++ b/locks-service/src/application/ports/content_lock_tombstone.rs @@ -0,0 +1,74 @@ +use async_trait::async_trait; +use locks_core::content_lock_deletion::ContentLockDeletionTombstone; +use locks_core::ids::{ContentLockPath, CreatorPubky}; +use locks_core::lock_policy::ContentLock; + +use crate::application::errors::ApplicationError; + +/// Byte-for-byte state of a canonical public deletion tombstone. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum TombstoneReadback { + /// The public bytes exactly match the canonical expected tombstone bytes. + Exact, + /// The canonical public lock path is absent. + Missing, + /// The path contains bytes other than the canonical expected tombstone. + Replaced, +} + +/// Exact public tombstone publication and readback boundary. +#[async_trait] +pub trait ContentLockTombstoneRepository: Send + Sync { + /// Reconciles withdrawal from the current canonical public bytes. + /// + /// An exact tombstone is replay success without a write. The implementation compares current + /// bytes with the frozen original before publishing and fails closed on bytes already observed + /// as missing or replaced. Pubky 0.9.3 has no conditional write, so this is deliberately not a + /// CAS guarantee: an out-of-band replacement racing between the comparison and unconditional + /// tombstone PUT can be overwritten under the product's documented TOCTOU exception. + async fn withdraw_content_lock( + &self, + creator: CreatorPubky, + content_lock_path: ContentLockPath, + frozen_original: &ContentLock, + tombstone: &ContentLockDeletionTombstone, + ) -> Result; + + /// Classifies raw bytes at the canonical public lock path without parsing them as a lock. + async fn read_tombstone( + &self, + creator: &CreatorPubky, + content_lock_path: &ContentLockPath, + expected: &ContentLockDeletionTombstone, + ) -> Result; + + /// Force-only operation that unconditionally deletes whatever bytes occupy the canonical + /// public lock path and succeeds only after a raw read verifies that the path is absent. + /// + /// This deliberately does not compare or parse the current bytes: an original Content Lock, + /// the expected tombstone, or any replacement is deleted. + async fn force_delete_content_lock_and_verify_absent( + &self, + creator: &CreatorPubky, + content_lock_path: &ContentLockPath, + ) -> Result<(), ApplicationError>; +} + +pub(crate) fn canonical_tombstone_bytes( + tombstone: &ContentLockDeletionTombstone, +) -> Result, ApplicationError> { + serde_json::to_vec(tombstone).map_err(|error| ApplicationError::Storage { + message: format!("failed to serialize content lock deletion tombstone: {error}"), + }) +} + +pub(crate) fn classify_tombstone_bytes( + actual: Option<&[u8]>, + expected: &[u8], +) -> TombstoneReadback { + match actual { + None => TombstoneReadback::Missing, + Some(actual) if actual == expected => TombstoneReadback::Exact, + Some(_) => TombstoneReadback::Replaced, + } +} diff --git a/locks-service/src/application/ports/guarded_resources.rs b/locks-service/src/application/ports/guarded_resources.rs index b5241a8..e304770 100644 --- a/locks-service/src/application/ports/guarded_resources.rs +++ b/locks-service/src/application/ports/guarded_resources.rs @@ -4,6 +4,14 @@ use locks_core::ids::{CreatorPubky, GuardedResourceHash}; use crate::application::errors::ApplicationError; use crate::application::models::GuardedResourceRecord; +/// Closed, non-destructive classification of a frozen guarded-resource generation. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum GuardedResourceReadback { + Exact, + Missing, + Replaced, +} + /// Repository for guarded resource bytes used by the first retrieval/access slice. #[async_trait] pub trait GuardedResourceRepository: Send + Sync { @@ -42,4 +50,20 @@ pub trait GuardedResourceRepository: Send + Sync { creator: &CreatorPubky, path: &str, ) -> Result; + + /// Reads and classifies the current generation without deleting any bytes. + async fn read_guarded_resource_generation( + &self, + creator: &CreatorPubky, + path: &str, + expected_hash: &GuardedResourceHash, + ) -> Result { + Ok( + match self.get_current_guarded_resource(creator, path).await? { + None => GuardedResourceReadback::Missing, + Some(record) if record.hash == *expected_hash => GuardedResourceReadback::Exact, + Some(_) => GuardedResourceReadback::Replaced, + }, + ) + } } diff --git a/locks-service/src/application/ports/mod.rs b/locks-service/src/application/ports/mod.rs index a40450e..d5ce366 100644 --- a/locks-service/src/application/ports/mod.rs +++ b/locks-service/src/application/ports/mod.rs @@ -2,7 +2,9 @@ pub mod semantics {} mod access; mod content_lock_deletion; +pub mod content_lock_deletion_action_ownership; mod content_lock_ownership; +pub mod content_lock_tombstone; mod creator_authority; mod entitlement; mod guarded_resources; @@ -14,7 +16,9 @@ mod verification; pub use access::*; pub use content_lock_deletion::*; +pub use content_lock_deletion_action_ownership::*; pub use content_lock_ownership::*; +pub use content_lock_tombstone::*; pub use creator_authority::*; pub use entitlement::*; pub use guarded_resources::*; diff --git a/locks-service/src/application/ports/payment_drain_repository.rs b/locks-service/src/application/ports/payment_drain_repository.rs index a1239da..89de2b4 100644 --- a/locks-service/src/application/ports/payment_drain_repository.rs +++ b/locks-service/src/application/ports/payment_drain_repository.rs @@ -33,7 +33,6 @@ pub trait PaymentDrainRepository: Send + Sync { deletion_job_id: Uuid, worker_id: &str, claim_token: Uuid, - now: OffsetDateTime, summary: &PaymentDrainSummary, ) -> Result; @@ -47,7 +46,6 @@ pub trait PaymentDrainRepository: Send + Sync { deletion_job_id: Uuid, worker_id: &str, claim_token: Uuid, - now: OffsetDateTime, summary: &PaymentDrainSummary, ) -> Result; @@ -61,7 +59,6 @@ pub trait PaymentDrainRepository: Send + Sync { deletion_job_id: Uuid, worker_id: &str, claim_token: Uuid, - now: OffsetDateTime, task_id: &TaskId, ) -> Result, ApplicationError>; @@ -70,7 +67,6 @@ pub trait PaymentDrainRepository: Send + Sync { deletion_job_id: Uuid, worker_id: &str, claim_token: Uuid, - now: OffsetDateTime, task_id: &TaskId, transition: PaymentDrainTerminalTransition, ) -> Result; diff --git a/locks-service/src/application/ports/verification.rs b/locks-service/src/application/ports/verification.rs index 255faec..078bfd7 100644 --- a/locks-service/src/application/ports/verification.rs +++ b/locks-service/src/application/ports/verification.rs @@ -30,6 +30,28 @@ pub trait VerificationTaskRepository: Send + Sync { task: VerificationTaskRecord, ) -> Result<(), ApplicationError>; + /// Atomically updates a prevalidated set of verification tasks. + /// + /// Repositories that support multi-record updates must either apply every update or none. + /// The default supports the trivially atomic single-record case only. + async fn update_verification_tasks_atomically( + &self, + mut tasks: Vec, + ) -> Result<(), ApplicationError> { + if tasks.len() == 1 { + return self + .update_verification_task(tasks.pop().expect("length checked")) + .await; + } + if tasks.is_empty() { + return Ok(()); + } + Err(ApplicationError::Storage { + message: "atomic verification task batch update is not implemented by this repository" + .to_owned(), + }) + } + /// Loads a verification task by task ID. /// /// Returns `Ok(None)` when no task exists. @@ -70,19 +92,16 @@ pub trait VerificationTaskClaimer: Send + Sync { task_id: &TaskId, worker_id: &str, claim_token: &uuid::Uuid, - now: time::OffsetDateTime, ) -> Result; /// Claims one pending or expired in-progress verification task for a worker. /// /// Returns `Ok(None)` when no task is claimable. Every successful claim includes a fresh - /// opaque token. The `now` parameter defines expiration comparison time, and - /// `claim_expires_at` is the new lease expiry assigned to the claimed task. + /// opaque token. Implementations sample authoritative time after fencing the selected task. async fn claim_next_verification_task( &self, worker_id: &str, - now: time::OffsetDateTime, - claim_expires_at: time::OffsetDateTime, + claim_ttl: time::Duration, ) -> Result, ApplicationError>; /// Returns an actively owned in-progress task to pending with a durable retry due time. @@ -95,8 +114,7 @@ pub trait VerificationTaskClaimer: Send + Sync { task_id: &TaskId, worker_id: &str, claim_token: &uuid::Uuid, - now: time::OffsetDateTime, - next_attempt_at: time::OffsetDateTime, + retry_after: time::Duration, ) -> Result, ApplicationError>; /// Persists a terminal task transition only for the exact active lease incarnation. @@ -107,7 +125,6 @@ pub trait VerificationTaskClaimer: Send + Sync { task: VerificationTaskRecord, worker_id: &str, claim_token: &uuid::Uuid, - now: time::OffsetDateTime, ) -> Result, ApplicationError>; } diff --git a/locks-service/src/application/use_cases/complete_verification_task.rs b/locks-service/src/application/use_cases/complete_verification_task.rs index 252d3c5..ee318f0 100644 --- a/locks-service/src/application/use_cases/complete_verification_task.rs +++ b/locks-service/src/application/use_cases/complete_verification_task.rs @@ -51,7 +51,6 @@ struct ClaimFencedTaskRepository<'a> { claim: ClaimedVerificationTask, claimer: &'a dyn VerificationTaskClaimer, worker_id: &'a str, - clock: &'a dyn Clock, } struct ClaimFencedEntitlementRepository<'a> { @@ -59,7 +58,6 @@ struct ClaimFencedEntitlementRepository<'a> { claim: &'a ClaimedVerificationTask, claimer: &'a dyn VerificationTaskClaimer, worker_id: &'a str, - clock: &'a dyn Clock, } #[async_trait] @@ -74,7 +72,6 @@ impl EntitlementRepository for ClaimFencedEntitlementRepository<'_> { &self.claim.task.task_id, self.worker_id, &self.claim.claim_token, - self.clock.now(), ) .await? { @@ -122,7 +119,6 @@ impl VerificationTaskRepository for ClaimFencedTaskRepository<'_> { task, self.worker_id, &self.claim.claim_token, - self.clock.now(), ) .await? .ok_or(ApplicationError::VerificationTaskClaimLost)?; @@ -205,7 +201,11 @@ impl<'a> CompleteVerificationTaskUseCase<'a> { let verification_result = match self.verify_criteria(&task, &content_lock).await { Ok(verification_result) => verification_result, Err(error) => { - if matches!(error, ApplicationError::VerificationPending) { + if matches!( + error, + ApplicationError::VerificationPending + | ApplicationError::VerificationDependencyUnavailable + ) { return Err(error); } self.persist_failed_task(task, viewer_safe_failure_message(&error).to_owned()) @@ -299,14 +299,12 @@ impl<'a> CompleteVerificationTaskUseCase<'a> { claim: claim.clone(), claimer, worker_id, - clock: self.clock, }; let fenced_entitlements = ClaimFencedEntitlementRepository { inner: self.entitlements, claim: &claim, claimer, worker_id, - clock: self.clock, }; CompleteVerificationTaskUseCase::new( &fenced_tasks, @@ -408,6 +406,7 @@ fn viewer_safe_failure_message(error: &ApplicationError) -> &'static str { | ApplicationError::MissingRecord { .. } | ApplicationError::InvalidVerificationTaskTransition { .. } | ApplicationError::VerificationPending + | ApplicationError::VerificationDependencyUnavailable | ApplicationError::InvalidVerificationTaskState { .. } | ApplicationError::VerificationTaskClaimLost | ApplicationError::InvalidVerificationTaskFailureMessage @@ -721,8 +720,7 @@ mod tests { let claim = claimer .claim_next_verification_task( "worker-a", - datetime!(2026-05-29 12:04:00 UTC), - datetime!(2026-05-29 13:00:00 UTC), + (datetime!(2026-05-29 13:00:00 UTC)) - (datetime!(2026-05-29 12:04:00 UTC)), ) .await .unwrap() @@ -1237,7 +1235,6 @@ mod tests { _task_id: &TaskId, _worker_id: &str, _claim_token: &uuid::Uuid, - _now: time::OffsetDateTime, ) -> Result { Ok(true) } @@ -1245,8 +1242,7 @@ mod tests { async fn claim_next_verification_task( &self, _worker_id: &str, - _now: OffsetDateTime, - _claim_expires_at: OffsetDateTime, + _claim_ttl: time::Duration, ) -> Result, ApplicationError> { unreachable!("completion must not claim tasks") } @@ -1256,8 +1252,7 @@ mod tests { _task_id: &TaskId, _worker_id: &str, _claim_token: &uuid::Uuid, - _now: OffsetDateTime, - _next_attempt_at: OffsetDateTime, + _retry_after: time::Duration, ) -> Result, ApplicationError> { unreachable!("completion must not schedule retries") } @@ -1267,7 +1262,6 @@ mod tests { _task: VerificationTaskRecord, _worker_id: &str, _claim_token: &uuid::Uuid, - _now: OffsetDateTime, ) -> Result, ApplicationError> { Ok(None) } diff --git a/locks-service/src/application/use_cases/create_content_lock.rs b/locks-service/src/application/use_cases/create_content_lock.rs index 36526ee..417d5a0 100644 --- a/locks-service/src/application/use_cases/create_content_lock.rs +++ b/locks-service/src/application/use_cases/create_content_lock.rs @@ -336,7 +336,7 @@ mod tests { let lock_id = content_lock.lock_id().unwrap(); fixture .content_lock_deletions - .prepare_force_deletion(&request.creator, &lock_id, fixture.clock.now()) + .prepare_force_deletion(&request.creator, &lock_id) .await .unwrap(); @@ -652,7 +652,7 @@ mod tests { assert_eq!( fixture .content_lock_deletions - .prepare_force_deletion(&creator(), &lock_id, fixture.clock.now()) + .prepare_force_deletion(&creator(), &lock_id) .await .unwrap(), PrepareForceDeletionResult::PublicationInProgress @@ -664,7 +664,6 @@ mod tests { let fixture = Fixture::seeded().await; let probe = PublicationRaceProbe { deletions: &fixture.content_lock_deletions, - now: fixture.clock.now(), }; let use_case = CreateContentLockUseCase::new( &probe, @@ -682,7 +681,7 @@ mod tests { assert_eq!( fixture .content_lock_deletions - .prepare_force_deletion(&creator(), &created.lock_id, fixture.clock.now()) + .prepare_force_deletion(&creator(), &created.lock_id) .await .unwrap(), PrepareForceDeletionResult::Synchronous(None) @@ -691,7 +690,6 @@ mod tests { struct PublicationRaceProbe<'a> { deletions: &'a InMemoryContentLockDeletionRepository, - now: OffsetDateTime, } #[async_trait] @@ -705,7 +703,7 @@ mod tests { let lock_id = content_lock.lock_id().unwrap(); assert_eq!( self.deletions - .prepare_force_deletion(&creator, &lock_id, self.now) + .prepare_force_deletion(&creator, &lock_id) .await?, PrepareForceDeletionResult::PublicationInProgress ); diff --git a/locks-service/src/application/use_cases/credential_flow_tests.rs b/locks-service/src/application/use_cases/credential_flow_tests.rs index fd472cb..ecb0c07 100644 --- a/locks-service/src/application/use_cases/credential_flow_tests.rs +++ b/locks-service/src/application/use_cases/credential_flow_tests.rs @@ -1,3 +1,4 @@ +use std::collections::VecDeque; use std::str::FromStr; use std::sync::Mutex; @@ -22,7 +23,7 @@ use locks_core::verification::{ use crate::application::errors::ApplicationError; use crate::application::models::{ AccessCredential, AccessCredentialLookupKey, AccessCredentialPolicy, AccessCredentialRecord, - GuardedResourceRecord, VerificationTaskRecord, + GuardedResourceRecord, IssuedDeletionCredential, VerificationTaskRecord, }; use crate::application::ports::{ AccessCredentialGenerator, AccessCredentialStore, Clock, ContentLockRepository, @@ -37,6 +38,36 @@ use crate::application::use_cases::validate_access_credential::{ const BUNDLE_ID: &str = "000G40R40M30E209185GR38E1W"; +#[tokio::test] +async fn final_credential_winner_uses_fresh_time_after_generation() { + let before_generation = datetime!(2026-05-29 12:00:00 UTC); + let after_generation = datetime!(2026-05-29 12:00:01 UTC); + let entitlements = FakeEntitlements::new(None); + let content_locks = FakeContentLocks::new(None); + let store = FakeAccessCredentialStore::with_final_credential(); + let generator = FakeGenerator::new(AccessCredential::new("final-bearer")); + let clock = SequenceClock::new([before_generation, after_generation]); + let use_case = IssueAccessCredentialUseCase::new( + &entitlements, + &content_locks, + &store, + &generator, + &clock, + AccessCredentialPolicy::new(3600), + ); + + let issued = use_case + .execute(IssueAccessCredentialRequest { + creator: creator(), + bundle_id: bundle_id(), + }) + .await + .unwrap(); + + assert_eq!(issued.credential.as_str(), "final-bearer"); + assert_eq!(store.final_issue_time(), Some(after_generation)); +} + #[tokio::test] async fn issue_access_credential_rechecks_entitlement_and_stores_lookup_key() { let content_lock = content_lock_fixture(900); @@ -556,6 +587,8 @@ impl EntitlementRepository for FakeEntitlements { struct FakeAccessCredentialStore { record: Mutex>, deleted: Mutex>, + final_available: bool, + final_issue_time: Mutex>, } impl FakeAccessCredentialStore { @@ -563,9 +596,22 @@ impl FakeAccessCredentialStore { Self { record: Mutex::new(Some((lookup_key, record))), deleted: Mutex::new(None), + final_available: false, + final_issue_time: Mutex::new(None), + } + } + + fn with_final_credential() -> Self { + Self { + final_available: true, + ..Self::default() } } + fn final_issue_time(&self) -> Option { + *self.final_issue_time.lock().unwrap() + } + fn stored_record(&self) -> (AccessCredentialLookupKey, AccessCredentialRecord) { self.record.lock().unwrap().clone().unwrap() } @@ -612,6 +658,29 @@ impl AccessCredentialStore for FakeAccessCredentialStore { *self.record.lock().unwrap() = None; Ok(()) } + + async fn final_credential_available( + &self, + _creator: &CreatorPubky, + _bundle_id: &BundleId, + _now: OffsetDateTime, + ) -> Result { + Ok(self.final_available) + } + + async fn issue_or_replay_final_credential( + &self, + _creator: &CreatorPubky, + _bundle_id: &BundleId, + now: OffsetDateTime, + candidate: AccessCredential, + ) -> Result, ApplicationError> { + *self.final_issue_time.lock().unwrap() = Some(now); + Ok(Some(IssuedDeletionCredential { + credential: candidate, + expires_at: now + time::Duration::minutes(1), + })) + } } struct FakeGenerator { @@ -656,6 +725,28 @@ impl Clock for FakeClock { } } +struct SequenceClock { + values: Mutex>, +} + +impl SequenceClock { + fn new(values: impl IntoIterator) -> Self { + Self { + values: Mutex::new(values.into_iter().collect()), + } + } +} + +impl Clock for SequenceClock { + fn now(&self) -> OffsetDateTime { + self.values + .lock() + .unwrap() + .pop_front() + .expect("sequence clock exhausted") + } +} + #[allow(dead_code)] struct UnusedPortImplementations; diff --git a/locks-service/src/application/use_cases/deletion_test_fixtures.rs b/locks-service/src/application/use_cases/deletion_test_fixtures.rs new file mode 100644 index 0000000..af9f65a --- /dev/null +++ b/locks-service/src/application/use_cases/deletion_test_fixtures.rs @@ -0,0 +1,68 @@ +use std::{collections::BTreeMap, str::FromStr}; + +use locks_core::{ + ids::{CreatorPubky, GuardedResourceHash}, + lock_policy::{ + AccessPolicy, CONTENT_LOCK_VERSION, ContentLock, GuardedResource, LockLogic, + LockServerConfig, SecondaryGuardedResource, + }, +}; +use time::OffsetDateTime; +use uuid::Uuid; + +use crate::application::models::{ + ClaimedContentLockDeletionJob, ContentLockDeletionJob, ContentLockDeletionPhase, +}; + +pub(super) fn claimed_deletion_job( + now: OffsetDateTime, + phase: ContentLockDeletionPhase, + force: bool, +) -> ClaimedContentLockDeletionJob { + let mut secondary_resources = BTreeMap::new(); + for path in [ + "/priv/locks.app/content/z", + "/priv/locks.app/content/a", + "/priv/locks.app/content/m", + ] { + secondary_resources.insert( + path.to_owned(), + SecondaryGuardedResource { + hash: GuardedResourceHash::from_bytes([7; 32]), + content_type: "text/plain".to_owned(), + size: 1, + }, + ); + } + let lock = ContentLock { + version: CONTENT_LOCK_VERSION, + creator: CreatorPubky::from_str( + "pubkytkrq8zmwb8a3m9k15csu3q17qmfgqnp9dskbrg9uq1rydpyxp7qy", + ) + .unwrap(), + primary_resource: Some( + GuardedResource::new( + "/priv/locks.app/content/m", + GuardedResourceHash::from_bytes([8; 32]), + "text/plain", + 1, + ) + .unwrap(), + ), + secondary_resources, + criteria: vec![], + lock_logic: LockLogic::All { criteria: vec![] }, + access_policy: AccessPolicy { + requested_credential_ttl_seconds: 900, + }, + lock_server: LockServerConfig { override_: None }, + created_at: now, + }; + let mut job = ContentLockDeletionJob::new(Uuid::from_u128(1), lock, now).unwrap(); + job.phase = phase; + job.force_requested_at = force.then_some(now); + ClaimedContentLockDeletionJob { + job, + claim_token: Uuid::from_u128(2), + } +} diff --git a/locks-service/src/application/use_cases/drain_lock_payments.rs b/locks-service/src/application/use_cases/drain_lock_payments.rs index 5b46ba8..aa9a57c 100644 --- a/locks-service/src/application/use_cases/drain_lock_payments.rs +++ b/locks-service/src/application/use_cases/drain_lock_payments.rs @@ -7,7 +7,8 @@ use locks_core::verification::{ use crate::application::errors::ApplicationError; use crate::application::models::{ - ClaimedContentLockDeletionJob, ContentLockDeletionPhase, VerificationTaskStatus, + AdvanceContentLockDeletionPhaseResult, ClaimedContentLockDeletionJob, ContentLockDeletionPhase, + VerificationTaskStatus, }; use crate::application::ports::{ Clock, ContentLockDeletionRepository, EntitlementRepository, PaymentDrainClient, @@ -16,6 +17,11 @@ use crate::application::ports::{ same_entitlement_decision, }; +use super::execute_content_lock_deletion_phase::{ + DeletionDependencyEvidence, DeletionDependencySource, DeletionExecutionErrorClass, + DeletionPhaseExecution, DeletionPhaseExecutionOutcome, classify_deletion_execution_error, +}; + pub struct DrainLockPaymentsUseCase<'a> { deletions: &'a dyn ContentLockDeletionRepository, drains: &'a dyn PaymentDrainRepository, @@ -51,10 +57,50 @@ impl<'a> DrainLockPaymentsUseCase<'a> { &self, claim: ClaimedContentLockDeletionJob, worker_id: &str, + ) -> Result { + self.execute_claimed_observed(claim, worker_id, &mut DeletionDependencyEvidence::none()) + .await + } + + pub async fn execute_claimed_with_evidence( + &self, + claim: ClaimedContentLockDeletionJob, + worker_id: &str, + ) -> DeletionPhaseExecution { + let mut evidence = DeletionDependencyEvidence::none(); + match self + .execute_claimed_observed(claim, worker_id, &mut evidence) + .await + { + Ok(true) => DeletionPhaseExecution::new(DeletionPhaseExecutionOutcome::Progressed) + .with_evidence(evidence), + Ok(false) => DeletionPhaseExecution::new(DeletionPhaseExecutionOutcome::Deferred) + .with_evidence(evidence), + Err(error) => { + let outcome = match classify_deletion_execution_error(&error) { + DeletionExecutionErrorClass::TransientDependency => { + DeletionPhaseExecutionOutcome::TransientDependencyFailure + } + DeletionExecutionErrorClass::Fatal => { + DeletionPhaseExecutionOutcome::FatalFailure + } + }; + DeletionPhaseExecution::new(outcome).with_evidence(evidence) + } + } + } + + async fn execute_claimed_observed( + &self, + claim: ClaimedContentLockDeletionJob, + worker_id: &str, + evidence: &mut DeletionDependencyEvidence, ) -> Result { match claim.job.phase { - ContentLockDeletionPhase::StartPaymentDrain => self.start(claim, worker_id).await, - ContentLockDeletionPhase::DrainPayments => self.drain(claim, worker_id).await, + ContentLockDeletionPhase::StartPaymentDrain => { + self.start(claim, worker_id, evidence).await + } + ContentLockDeletionPhase::DrainPayments => self.drain(claim, worker_id, evidence).await, _ => Err(ApplicationError::InvalidContentLockDeletionState { message: "payment drain use case requires a payment drain phase".to_owned(), }), @@ -65,85 +111,143 @@ impl<'a> DrainLockPaymentsUseCase<'a> { &self, claim: ClaimedContentLockDeletionJob, worker_id: &str, + evidence: &mut DeletionDependencyEvidence, ) -> Result { let lock_resource = lock_resource(&claim); let summary = match self.paykit.start_payment_drain(&lock_resource).await { - Ok(summary) => summary, - Err(PaymentDrainClientError::Conflict) => self - .paykit - .lookup_payment_drain(&lock_resource) - .await - .map_err(map_client_error)? - .ok_or_else(|| ApplicationError::Verifier { - message: "Paykit payment drain conflict could not be reconciled".to_owned(), - })?, - Err(error) => return Err(map_client_error(error)), + Ok(summary) => { + observe_healthy(evidence, DeletionDependencySource::PaymentProvider); + summary + } + Err(PaymentDrainClientError::Conflict) => { + observe_healthy(evidence, DeletionDependencySource::PaymentProvider); + match self.paykit.lookup_payment_drain(&lock_resource).await { + Ok(Some(summary)) => { + observe_healthy(evidence, DeletionDependencySource::PaymentProvider); + summary + } + Ok(None) => { + observe_healthy(evidence, DeletionDependencySource::PaymentProvider); + return Err(ApplicationError::Verifier { + message: "Paykit payment drain conflict could not be reconciled" + .to_owned(), + }); + } + Err(error) => { + observe_unavailable(evidence, DeletionDependencySource::PaymentProvider); + return Err(map_client_error(error)); + } + } + } + Err(error) => { + observe_unavailable(evidence, DeletionDependencySource::PaymentProvider); + return Err(map_client_error(error)); + } }; - let now = self.clock.now(); - if !self + let _now = self.clock.now(); + match self .drains - .store_payment_drain( - claim.job.job_id, - worker_id, - claim.claim_token, - now, - &summary, - ) - .await? + .store_payment_drain(claim.job.job_id, worker_id, claim.claim_token, &summary) + .await { - return Ok(false); + Ok(true) => observe_healthy(evidence, DeletionDependencySource::PaymentDrainRepository), + Ok(false) => return Ok(false), + Err(error) => { + observe_unavailable(evidence, DeletionDependencySource::PaymentDrainRepository); + return Err(error); + } } - Ok(self + match self .deletions .advance_phase( claim.job.job_id, worker_id, claim.claim_token, - now, ContentLockDeletionPhase::DrainPayments, ) - .await? - .is_some()) + .await + { + Ok(AdvanceContentLockDeletionPhaseResult::Advanced(_)) => { + observe_healthy(evidence, DeletionDependencySource::RepositoryPhaseMutation); + Ok(true) + } + Ok(AdvanceContentLockDeletionPhaseResult::ClaimLost) => Ok(false), + Ok(_) => { + observe_healthy(evidence, DeletionDependencySource::RepositoryPhaseMutation); + Ok(false) + } + Err(error) => { + observe_unavailable(evidence, DeletionDependencySource::RepositoryPhaseMutation); + Err(error) + } + } } async fn drain( &self, claim: ClaimedContentLockDeletionJob, worker_id: &str, + evidence: &mut DeletionDependencyEvidence, ) -> Result { let lock_resource = lock_resource(&claim); - let summary = self - .paykit - .lookup_payment_drain(&lock_resource) - .await - .map_err(map_client_error)? - .ok_or_else(|| ApplicationError::Verifier { - message: "Paykit payment drain not found".to_owned(), - })?; - let persisted = self - .drains - .get_payment_drain(claim.job.job_id) - .await? - .ok_or_else(|| ApplicationError::InvalidContentLockDeletionState { - message: "payment drain cleanup token is missing".to_owned(), - })?; + let summary = match self.paykit.lookup_payment_drain(&lock_resource).await { + Ok(Some(summary)) => { + observe_healthy(evidence, DeletionDependencySource::PaymentProvider); + summary + } + Ok(None) => { + observe_healthy(evidence, DeletionDependencySource::PaymentProvider); + return Err(ApplicationError::Verifier { + message: "Paykit payment drain not found".to_owned(), + }); + } + Err(error) => { + observe_unavailable(evidence, DeletionDependencySource::PaymentProvider); + return Err(map_client_error(error)); + } + }; + let persisted = match self.drains.get_payment_drain(claim.job.job_id).await { + Ok(Some(persisted)) => { + observe_healthy(evidence, DeletionDependencySource::PaymentDrainRepository); + persisted + } + Ok(None) => { + observe_healthy(evidence, DeletionDependencySource::PaymentDrainRepository); + return Err(ApplicationError::InvalidContentLockDeletionState { + message: "payment drain cleanup token is missing".to_owned(), + }); + } + Err(error) => { + observe_unavailable(evidence, DeletionDependencySource::PaymentDrainRepository); + return Err(error); + } + }; validate_aggregate_progress(&persisted, &summary)?; - let now = self.clock.now(); - if !self + let _now = self.clock.now(); + match self .drains - .reconcile_payment_drain( - claim.job.job_id, - worker_id, - claim.claim_token, - now, - &summary, - ) - .await? + .reconcile_payment_drain(claim.job.job_id, worker_id, claim.claim_token, &summary) + .await { - return Ok(false); + Ok(true) => observe_healthy(evidence, DeletionDependencySource::PaymentDrainRepository), + Ok(false) => return Ok(false), + Err(error) => { + observe_unavailable(evidence, DeletionDependencySource::PaymentDrainRepository); + return Err(error); + } } - for obligation in self.drains.list_obligations(claim.job.job_id).await? { + let obligations = match self.drains.list_obligations(claim.job.job_id).await { + Ok(obligations) => { + observe_healthy(evidence, DeletionDependencySource::PaymentDrainRepository); + obligations + } + Err(error) => { + observe_unavailable(evidence, DeletionDependencySource::PaymentDrainRepository); + return Err(error); + } + }; + for obligation in obligations { if matches!( obligation.status, VerificationTaskStatus::Completed @@ -152,14 +256,26 @@ impl<'a> DrainLockPaymentsUseCase<'a> { ) { continue; } - let status = self + let status = match self .paykit .payment_request_status(&obligation.creator, &obligation.bundle_id) .await - .map_err(map_client_error)? - .ok_or_else(|| ApplicationError::Verifier { - message: "Paykit payment request not found".to_owned(), - })?; + { + Ok(Some(status)) => { + observe_healthy(evidence, DeletionDependencySource::PaymentProvider); + status + } + Ok(None) => { + observe_healthy(evidence, DeletionDependencySource::PaymentProvider); + return Err(ApplicationError::Verifier { + message: "Paykit payment request not found".to_owned(), + }); + } + Err(error) => { + observe_unavailable(evidence, DeletionDependencySource::PaymentProvider); + return Err(map_client_error(error)); + } + }; if status.invoice_created_at != obligation.invoice_created_at || status.payment_deadline != obligation.payment_deadline { @@ -174,18 +290,28 @@ impl<'a> DrainLockPaymentsUseCase<'a> { let now = self.clock.now(); let mut entitlement_publication_token = None; if next_status == VerificationTaskStatus::Completed { - let Some(publication_token) = self + let publication_token = match self .drains .begin_entitlement_publication( claim.job.job_id, worker_id, claim.claim_token, - now, &obligation.task_id, ) - .await? - else { - return Ok(false); + .await + { + Ok(Some(token)) => { + observe_healthy(evidence, DeletionDependencySource::PaymentDrainRepository); + token + } + Ok(None) => return Ok(false), + Err(error) => { + observe_unavailable( + evidence, + DeletionDependencySource::PaymentDrainRepository, + ); + return Err(error); + } }; let entitlement = VerifiedProofBundle { version: VERIFIED_PROOF_BUNDLE_VERSION, @@ -202,47 +328,72 @@ impl<'a> DrainLockPaymentsUseCase<'a> { }, entitlement_lifetime: EntitlementLifetime::Unbounded, }; - persist_entitlement(self.entitlements, entitlement).await?; + persist_entitlement(self.entitlements, entitlement, evidence).await?; entitlement_publication_token = Some(publication_token); } - if !self + match self .drains .persist_terminal_obligation( claim.job.job_id, worker_id, claim.claim_token, - now, &obligation.task_id, PaymentDrainTerminalTransition { status: next_status, entitlement_publication_token, }, ) - .await? + .await { - return Ok(false); + Ok(true) => { + observe_healthy(evidence, DeletionDependencySource::PaymentDrainRepository) + } + Ok(false) => return Ok(false), + Err(error) => { + observe_unavailable(evidence, DeletionDependencySource::PaymentDrainRepository); + return Err(error); + } } } - if summary.status != PaymentDrainStatus::Completed - || !self - .drains - .all_obligations_terminal(claim.job.job_id) - .await? - { + if summary.status != PaymentDrainStatus::Completed { return Ok(false); } - Ok(self + match self.drains.all_obligations_terminal(claim.job.job_id).await { + Ok(true) => observe_healthy(evidence, DeletionDependencySource::PaymentDrainRepository), + Ok(false) => { + observe_healthy(evidence, DeletionDependencySource::PaymentDrainRepository); + return Ok(false); + } + Err(error) => { + observe_unavailable(evidence, DeletionDependencySource::PaymentDrainRepository); + return Err(error); + } + } + match self .deletions .advance_phase( claim.job.job_id, worker_id, claim.claim_token, - self.clock.now(), ContentLockDeletionPhase::DrainExistingCredentials, ) - .await? - .is_some()) + .await + { + Ok(AdvanceContentLockDeletionPhaseResult::Advanced(_)) => { + observe_healthy(evidence, DeletionDependencySource::RepositoryPhaseMutation); + Ok(true) + } + Ok(AdvanceContentLockDeletionPhaseResult::ClaimLost) => Ok(false), + Ok(_) => { + observe_healthy(evidence, DeletionDependencySource::RepositoryPhaseMutation); + Ok(false) + } + Err(error) => { + observe_unavailable(evidence, DeletionDependencySource::RepositoryPhaseMutation); + Err(error) + } + } } } @@ -256,27 +407,59 @@ fn lock_resource(claim: &ClaimedContentLockDeletionJob) -> PubkyLockResource { async fn persist_entitlement( entitlements: &dyn EntitlementRepository, entitlement: VerifiedProofBundle, + evidence: &mut DeletionDependencyEvidence, ) -> Result<(), ApplicationError> { if let Err(error) = entitlements .insert_verified_proof_bundle(entitlement.clone()) .await { - let existing = entitlements + let expected_duplicate = matches!( + &error, + ApplicationError::DuplicateRecord { record } + if *record == "verified_proof_bundle" + ); + if !expected_duplicate { + observe_unavailable(evidence, DeletionDependencySource::EntitlementRepository); + } + let existing = match entitlements .get_verified_proof_bundle( entitlement.pubky_lock_resource.creator(), &entitlement.bundle_id, ) - .await?; + .await + { + Ok(existing) => { + observe_healthy(evidence, DeletionDependencySource::EntitlementRepository); + existing + } + Err(read_error) => { + observe_unavailable(evidence, DeletionDependencySource::EntitlementRepository); + return Err(read_error); + } + }; if !existing .as_ref() .is_some_and(|existing| same_entitlement_decision(existing, &entitlement)) { return Err(error); } + } else { + observe_healthy(evidence, DeletionDependencySource::EntitlementRepository); } Ok(()) } +fn observe_healthy(evidence: &mut DeletionDependencyEvidence, source: DeletionDependencySource) { + *evidence = evidence.merge(DeletionDependencyEvidence::healthy(source)); +} + +fn observe_unavailable( + evidence: &mut DeletionDependencyEvidence, + source: DeletionDependencySource, +) { + *evidence = evidence.merge(DeletionDependencyEvidence::unavailable(source)); +} + fn map_client_error(error: PaymentDrainClientError) -> ApplicationError { ApplicationError::Verifier { message: match error { @@ -362,17 +545,94 @@ pub(crate) fn classify_payment_task( #[cfg(test)] mod tests { + use std::str::FromStr; + + use async_trait::async_trait; use base64::Engine; use base64::engine::general_purpose::URL_SAFE_NO_PAD; use time::macros::datetime; + use locks_core::ids::{BundleId, ContentLockPath, CreatorPubky, PubkyLockResource}; + use locks_core::verification::{ + EntitlementLifetime, VERIFIED_PROOF_BUNDLE_VERSION, VerificationResult, VerifiedProofBundle, + }; + + use crate::application::errors::ApplicationError; use crate::application::models::VerificationTaskStatus; use crate::application::ports::{ - PaymentDrainCleanupToken, PaymentDrainStatus, PaymentDrainSummary, PaymentRequestState, - PaymentRequestStatus, PaymentState, + EntitlementRepository, PaymentDrainCleanupToken, PaymentDrainStatus, PaymentDrainSummary, + PaymentRequestState, PaymentRequestStatus, PaymentState, + }; + + use crate::application::use_cases::execute_content_lock_deletion_phase::DeletionDependencyStatus; + + use super::{ + DeletionDependencyEvidence, DeletionDependencySource, classify_payment_task, + persist_entitlement, validate_aggregate_progress, }; - use super::{classify_payment_task, validate_aggregate_progress}; + #[tokio::test] + async fn exact_duplicate_entitlement_replay_finishes_with_healthy_evidence() { + let entitlement = entitlement(); + let repository = DuplicateEntitlementRepository(entitlement.clone()); + let mut evidence = DeletionDependencyEvidence::none(); + + persist_entitlement(&repository, entitlement, &mut evidence) + .await + .unwrap(); + + assert_eq!( + evidence.status(DeletionDependencySource::EntitlementRepository), + Some(DeletionDependencyStatus::Healthy) + ); + } + + struct DuplicateEntitlementRepository(VerifiedProofBundle); + + #[async_trait] + impl EntitlementRepository for DuplicateEntitlementRepository { + async fn insert_verified_proof_bundle( + &self, + _: VerifiedProofBundle, + ) -> Result<(), ApplicationError> { + Err(ApplicationError::DuplicateRecord { + record: "verified_proof_bundle", + }) + } + + async fn get_verified_proof_bundle( + &self, + _: &CreatorPubky, + _: &BundleId, + ) -> Result, ApplicationError> { + Ok(Some(self.0.clone())) + } + + async fn delete_verified_proof_bundle( + &self, + _: &CreatorPubky, + _: &BundleId, + ) -> Result<(), ApplicationError> { + unreachable!() + } + } + + fn entitlement() -> VerifiedProofBundle { + let creator = + CreatorPubky::from_str("pubkytkrq8zmwb8a3m9k15csu3q17qmfgqnp9dskbrg9uq1rydpyxp7qy") + .unwrap(); + let path = ContentLockPath::from_str( + "/pub/locks.app/000G40R40M30E209185GR38E1W8124GK2GAHC5RR34D1P70X3RFG.json", + ) + .unwrap(); + VerifiedProofBundle { + version: VERIFIED_PROOF_BUNDLE_VERSION, + bundle_id: BundleId::from_str("000G40R40M30E209185GR38E1W").unwrap(), + pubky_lock_resource: PubkyLockResource::new(creator, path), + verification_result: VerificationResult { criteria: vec![] }, + entitlement_lifetime: EntitlementLifetime::Unbounded, + } + } #[test] fn aggregate_progress_accepts_only_equal_monotonic_transfer() { diff --git a/locks-service/src/application/use_cases/execute_content_lock_deletion_phase.rs b/locks-service/src/application/use_cases/execute_content_lock_deletion_phase.rs new file mode 100644 index 0000000..2782b32 --- /dev/null +++ b/locks-service/src/application/use_cases/execute_content_lock_deletion_phase.rs @@ -0,0 +1,771 @@ +use std::collections::BTreeMap; + +use async_trait::async_trait; +use locks_core::{ + content_lock_deletion::ContentLockDeletionTombstone, + ids::{ContentLockPath, GuardedResourceHash}, +}; +use time::Duration; + +use crate::application::{ + errors::ApplicationError, + models::{ + AdvanceContentLockDeletionPhaseResult, ClaimedContentLockDeletionJob, + ContentLockDeletionFailureCode, ContentLockDeletionPhase, + InitializeFinalAccessWindowsResult, + }, + ports::{ + AccessCredentialStore, Clock, ContentLockDeletionActionAcquireResult, + ContentLockDeletionActionClaim, ContentLockDeletionActionOwnership, + ContentLockDeletionRepository, ContentLockTombstoneRepository, GuardedResourceReadback, + GuardedResourceRepository, TombstoneReadback, + }, +}; + +use super::{ + drain_lock_payments::DrainLockPaymentsUseCase, + materialize_final_credentials::{ + MaterializeFinalCredentialsOutcome, MaterializeFinalCredentialsRequest, + MaterializeFinalCredentialsUseCase, + }, +}; + +/// Closed result of one bounded deletion-phase execution. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum DeletionPhaseExecutionOutcome { + Progressed, + Deferred, + ClaimLost, + TerminalFailed, + TransientDependencyFailure, + FatalFailure, +} + +/// Internal, identifier-free dependency classes observed by deletion execution. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum DeletionDependencySource { + PaymentProvider, + PaymentDrainRepository, + EntitlementRepository, + PubkyWithdrawal, + PubkyReadback, + PubkyResource, + PubkyForcePublic, + RepositoryQueueClaim, + RepositoryPhaseMutation, + RepositoryDefer, + RepositoryRetry, + RepositoryTerminalMutation, + RepositoryActionLock, + RepositoryActionLockRelease, + RepositoryForceReceipt, +} + +/// Health observed from one dependency during a bounded execution. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum DeletionDependencyStatus { + Healthy, + Unavailable, +} + +/// Closed dependency evidence carried from the real phase invocation to worker readiness. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] +pub struct DeletionDependencyEvidence { + statuses: [Option; DeletionDependencySource::ALL.len()], +} + +impl DeletionDependencyEvidence { + pub const fn none() -> Self { + Self { + statuses: [None; DeletionDependencySource::ALL.len()], + } + } + + pub fn healthy(source: DeletionDependencySource) -> Self { + Self::observed(source, DeletionDependencyStatus::Healthy) + } + + pub fn unavailable(source: DeletionDependencySource) -> Self { + Self::observed(source, DeletionDependencyStatus::Unavailable) + } + + pub fn status(self, source: DeletionDependencySource) -> Option { + self.statuses[source.index()] + } + + pub fn merge(mut self, other: Self) -> Self { + for source in DeletionDependencySource::ALL { + let index = source.index(); + self.statuses[index] = match (self.statuses[index], other.statuses[index]) { + (Some(DeletionDependencyStatus::Unavailable), _) + | (_, Some(DeletionDependencyStatus::Unavailable)) => { + Some(DeletionDependencyStatus::Unavailable) + } + (Some(status), None) | (None, Some(status)) => Some(status), + ( + Some(DeletionDependencyStatus::Healthy), + Some(DeletionDependencyStatus::Healthy), + ) => Some(DeletionDependencyStatus::Healthy), + (None, None) => None, + }; + } + self + } + + fn observed(source: DeletionDependencySource, status: DeletionDependencyStatus) -> Self { + let mut evidence = Self::none(); + evidence.statuses[source.index()] = Some(status); + evidence + } +} + +impl DeletionDependencySource { + pub const ALL: [Self; 15] = [ + Self::PaymentProvider, + Self::PaymentDrainRepository, + Self::EntitlementRepository, + Self::PubkyWithdrawal, + Self::PubkyReadback, + Self::PubkyResource, + Self::PubkyForcePublic, + Self::RepositoryQueueClaim, + Self::RepositoryPhaseMutation, + Self::RepositoryDefer, + Self::RepositoryRetry, + Self::RepositoryTerminalMutation, + Self::RepositoryActionLock, + Self::RepositoryActionLockRelease, + Self::RepositoryForceReceipt, + ]; + + pub const fn index(self) -> usize { + match self { + Self::PaymentProvider => 0, + Self::PaymentDrainRepository => 1, + Self::EntitlementRepository => 2, + Self::PubkyWithdrawal => 3, + Self::PubkyReadback => 4, + Self::PubkyResource => 5, + Self::PubkyForcePublic => 6, + Self::RepositoryQueueClaim => 7, + Self::RepositoryPhaseMutation => 8, + Self::RepositoryDefer => 9, + Self::RepositoryRetry => 10, + Self::RepositoryTerminalMutation => 11, + Self::RepositoryActionLock => 12, + Self::RepositoryActionLockRelease => 13, + Self::RepositoryForceReceipt => 14, + } + } +} + +/// One closed execution outcome plus source-aware, secret-free dependency evidence. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct DeletionPhaseExecution { + pub outcome: DeletionPhaseExecutionOutcome, + pub evidence: DeletionDependencyEvidence, +} + +impl DeletionPhaseExecution { + pub fn new(outcome: DeletionPhaseExecutionOutcome) -> Self { + Self { + outcome, + evidence: DeletionDependencyEvidence::none(), + } + } + + pub fn with_evidence(mut self, evidence: DeletionDependencyEvidence) -> Self { + self.evidence = self.evidence.merge(evidence); + self + } +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(super) enum DeletionExecutionErrorClass { + TransientDependency, + Fatal, +} + +pub(super) fn classify_deletion_execution_error( + error: &ApplicationError, +) -> DeletionExecutionErrorClass { + match error { + ApplicationError::Storage { .. } | ApplicationError::Verifier { .. } => { + DeletionExecutionErrorClass::TransientDependency + } + _ => DeletionExecutionErrorClass::Fatal, + } +} + +fn error_outcome(error: &ApplicationError) -> DeletionPhaseExecutionOutcome { + match classify_deletion_execution_error(error) { + DeletionExecutionErrorClass::TransientDependency => { + DeletionPhaseExecutionOutcome::TransientDependencyFailure + } + DeletionExecutionErrorClass::Fatal => DeletionPhaseExecutionOutcome::FatalFailure, + } +} + +fn error_execution( + error: &ApplicationError, + source: DeletionDependencySource, +) -> DeletionPhaseExecution { + let outcome = error_outcome(error); + DeletionPhaseExecution::new(outcome) + .with_evidence(DeletionDependencyEvidence::unavailable(source)) +} + +/// Bounded final-access settings used by the phase executor. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct ContentLockDeletionPhaseExecutorConfig { + pub final_credential_issuance_window: Duration, + pub final_read_window: Duration, + pub final_credential_batch_limit: usize, +} + +/// Object-safe payment-drain collaborator for phase-executor tests and runtime composition. +#[async_trait] +pub trait ContentLockPaymentDrainExecutor: Send + Sync { + /// Returns only evidence recorded by the concrete remote/repository operations it invoked. + async fn execute_claimed( + &self, + claim: ClaimedContentLockDeletionJob, + worker_id: &str, + ) -> DeletionPhaseExecution; +} + +#[async_trait] +impl ContentLockPaymentDrainExecutor for DrainLockPaymentsUseCase<'_> { + async fn execute_claimed( + &self, + claim: ClaimedContentLockDeletionJob, + worker_id: &str, + ) -> DeletionPhaseExecution { + DrainLockPaymentsUseCase::execute_claimed_with_evidence(self, claim, worker_id).await + } +} + +/// Object-safe bounded final-credential materialization collaborator. +#[async_trait] +pub trait FinalCredentialMaterializer: Send + Sync { + async fn materialize( + &self, + request: MaterializeFinalCredentialsRequest<'_>, + ) -> Result; +} + +#[async_trait] +impl FinalCredentialMaterializer for MaterializeFinalCredentialsUseCase<'_> { + async fn materialize( + &self, + request: MaterializeFinalCredentialsRequest<'_>, + ) -> Result { + self.execute(request).await + } +} + +/// Dependencies for one graceful-deletion phase executor. +pub struct ContentLockDeletionPhaseExecutorDependencies<'a> { + pub deletions: &'a dyn ContentLockDeletionRepository, + pub action_ownership: &'a dyn ContentLockDeletionActionOwnership, + pub tombstones: &'a dyn ContentLockTombstoneRepository, + pub guarded_resources: &'a dyn GuardedResourceRepository, + pub access_credentials: &'a dyn AccessCredentialStore, + pub clock: &'a dyn Clock, + pub payments: &'a dyn ContentLockPaymentDrainExecutor, + pub final_credentials: &'a dyn FinalCredentialMaterializer, +} + +/// Executes at most one bounded graceful-deletion phase while holding its action guard. +pub struct ContentLockDeletionPhaseExecutor<'a> { + dependencies: ContentLockDeletionPhaseExecutorDependencies<'a>, + config: ContentLockDeletionPhaseExecutorConfig, +} + +impl<'a> ContentLockDeletionPhaseExecutor<'a> { + pub fn new( + dependencies: ContentLockDeletionPhaseExecutorDependencies<'a>, + config: ContentLockDeletionPhaseExecutorConfig, + ) -> Self { + Self { + dependencies, + config, + } + } + + pub async fn execute( + &self, + claim: ClaimedContentLockDeletionJob, + worker_id: &str, + ) -> DeletionPhaseExecutionOutcome { + self.execute_with_evidence(claim, worker_id).await.outcome + } + + pub async fn execute_with_evidence( + &self, + claim: ClaimedContentLockDeletionJob, + worker_id: &str, + ) -> DeletionPhaseExecution { + let guard = match self + .dependencies + .action_ownership + .try_acquire(ContentLockDeletionActionClaim { + job_id: claim.job.job_id, + worker_id, + claim_token: claim.claim_token, + expected_phase: claim.job.phase, + force: false, + }) + .await + { + Ok(ContentLockDeletionActionAcquireResult::Acquired(guard)) => guard, + Ok(ContentLockDeletionActionAcquireResult::Busy) => { + return DeletionPhaseExecution::new(DeletionPhaseExecutionOutcome::Deferred); + } + Ok(ContentLockDeletionActionAcquireResult::ClaimLost) => { + return DeletionPhaseExecution::new(DeletionPhaseExecutionOutcome::ClaimLost); + } + Err(error) => { + return error_execution(&error, DeletionDependencySource::RepositoryActionLock); + } + }; + + let execution = self + .execute_guarded_with_evidence(claim, worker_id) + .await + .with_evidence(DeletionDependencyEvidence::healthy( + DeletionDependencySource::RepositoryActionLock, + )); + if let Err(error) = guard.release().await { + return error_execution( + &error, + DeletionDependencySource::RepositoryActionLockRelease, + ) + .with_evidence(execution.evidence); + } + execution.with_evidence(DeletionDependencyEvidence::healthy( + DeletionDependencySource::RepositoryActionLockRelease, + )) + } + + async fn execute_guarded_with_evidence( + &self, + claim: ClaimedContentLockDeletionJob, + worker_id: &str, + ) -> DeletionPhaseExecution { + match claim.job.phase { + ContentLockDeletionPhase::Withdraw => { + self.withdraw_with_evidence(&claim, worker_id).await + } + ContentLockDeletionPhase::StartPaymentDrain + | ContentLockDeletionPhase::DrainPayments => { + self.dependencies + .payments + .execute_claimed(claim, worker_id) + .await + } + ContentLockDeletionPhase::DrainExistingCredentials => self.repository_execution( + self.advance( + &claim, + worker_id, + ContentLockDeletionPhase::IssueFinalCredentials, + ) + .await, + ), + ContentLockDeletionPhase::IssueFinalCredentials => { + self.repository_execution(self.issue_final_credentials(&claim, worker_id).await) + } + ContentLockDeletionPhase::DrainFinalReads => self.repository_execution( + self.advance(&claim, worker_id, ContentLockDeletionPhase::DeleteContent) + .await, + ), + ContentLockDeletionPhase::DeleteContent => { + self.verify_frozen_content_with_evidence(&claim, worker_id) + .await + } + ContentLockDeletionPhase::DeleteTombstone => { + self.verify_tombstone_for_purge_with_evidence(&claim, worker_id) + .await + } + ContentLockDeletionPhase::PurgeOperationalState => { + DeletionPhaseExecution::new(DeletionPhaseExecutionOutcome::Deferred) + } + } + } + + async fn withdraw_with_evidence( + &self, + claim: &ClaimedContentLockDeletionJob, + worker_id: &str, + ) -> DeletionPhaseExecution { + let tombstone = tombstone(claim); + let content_lock_path = ContentLockPath::from_lock_id(claim.job.lock_id.clone()); + let readback = self + .dependencies + .tombstones + .withdraw_content_lock( + claim.job.creator.clone(), + content_lock_path, + &claim.job.frozen_content_lock, + &tombstone, + ) + .await; + let pubky_healthy = + DeletionDependencyEvidence::healthy(DeletionDependencySource::PubkyWithdrawal); + match readback { + Ok(TombstoneReadback::Exact) => self + .repository_execution( + self.advance( + claim, + worker_id, + ContentLockDeletionPhase::StartPaymentDrain, + ) + .await, + ) + .with_evidence(pubky_healthy), + Ok(TombstoneReadback::Missing) => self + .repository_execution( + self.finish_terminal( + claim, + worker_id, + ContentLockDeletionFailureCode::TombstoneMissing, + ) + .await, + ) + .with_evidence(pubky_healthy), + Ok(TombstoneReadback::Replaced) => self + .repository_execution( + self.finish_terminal( + claim, + worker_id, + ContentLockDeletionFailureCode::TombstoneReplaced, + ) + .await, + ) + .with_evidence(pubky_healthy), + Err(error) => error_execution(&error, DeletionDependencySource::PubkyWithdrawal), + } + } + + fn repository_execution( + &self, + outcome: DeletionPhaseExecutionOutcome, + ) -> DeletionPhaseExecution { + let evidence = match outcome { + DeletionPhaseExecutionOutcome::TransientDependencyFailure => { + DeletionDependencyEvidence::unavailable( + DeletionDependencySource::RepositoryPhaseMutation, + ) + } + DeletionPhaseExecutionOutcome::Progressed + | DeletionPhaseExecutionOutcome::Deferred + | DeletionPhaseExecutionOutcome::TerminalFailed => DeletionDependencyEvidence::healthy( + DeletionDependencySource::RepositoryPhaseMutation, + ), + DeletionPhaseExecutionOutcome::ClaimLost + | DeletionPhaseExecutionOutcome::FatalFailure => DeletionDependencyEvidence::none(), + }; + DeletionPhaseExecution::new(outcome).with_evidence(evidence) + } + + async fn issue_final_credentials( + &self, + claim: &ClaimedContentLockDeletionJob, + worker_id: &str, + ) -> DeletionPhaseExecutionOutcome { + let windows = match self + .dependencies + .access_credentials + .initialize_final_access_windows( + claim.job.job_id, + worker_id, + claim.claim_token, + self.config.final_credential_issuance_window, + self.config.final_read_window, + ) + .await + { + Ok(InitializeFinalAccessWindowsResult::Initialized(windows)) => windows, + Ok(InitializeFinalAccessWindowsResult::ClaimLost) => { + return DeletionPhaseExecutionOutcome::ClaimLost; + } + Err(error) => return error_outcome(&error), + }; + + let materialized = self + .dependencies + .final_credentials + .materialize(MaterializeFinalCredentialsRequest { + deletion_job_id: claim.job.job_id, + worker_id, + claim_token: claim.claim_token, + now: windows.issuance_started_at, + batch_limit: self.config.final_credential_batch_limit, + }) + .await; + match materialized { + Ok(outcome) + if self.config.final_credential_batch_limit > 0 + && outcome.materialized_count >= self.config.final_credential_batch_limit => + { + DeletionPhaseExecutionOutcome::Deferred + } + Ok(_) => { + self.advance(claim, worker_id, ContentLockDeletionPhase::DrainFinalReads) + .await + } + Err(error) => error_outcome(&error), + } + } + + async fn verify_frozen_content_with_evidence( + &self, + claim: &ClaimedContentLockDeletionJob, + worker_id: &str, + ) -> DeletionPhaseExecution { + let tombstone = tombstone(claim); + let content_lock_path = ContentLockPath::from_lock_id(claim.job.lock_id.clone()); + let mut resources = BTreeMap::::new(); + for (path, resource) in &claim.job.frozen_content_lock.secondary_resources { + resources.insert(path.clone(), resource.hash); + } + if let Some(primary) = &claim.job.frozen_content_lock.primary_resource { + resources.insert(primary.path.clone(), primary.hash); + } + + let pubky_healthy = + DeletionDependencyEvidence::healthy(DeletionDependencySource::PubkyReadback); + let mut resource_was_read = false; + for (path, expected_hash) in resources { + match self + .dependencies + .tombstones + .read_tombstone(&claim.job.creator, &content_lock_path, &tombstone) + .await + { + Ok(TombstoneReadback::Exact) => {} + Ok(TombstoneReadback::Missing) => { + return self + .repository_execution( + self.finish_terminal( + claim, + worker_id, + ContentLockDeletionFailureCode::TombstoneMissing, + ) + .await, + ) + .with_evidence(pubky_healthy); + } + Ok(TombstoneReadback::Replaced) => { + return self + .repository_execution( + self.finish_terminal( + claim, + worker_id, + ContentLockDeletionFailureCode::TombstoneReplaced, + ) + .await, + ) + .with_evidence(pubky_healthy); + } + Err(error) => { + return error_execution(&error, DeletionDependencySource::PubkyReadback); + } + } + match self + .dependencies + .guarded_resources + .read_guarded_resource_generation(&claim.job.creator, &path, &expected_hash) + .await + { + Ok(GuardedResourceReadback::Exact) => resource_was_read = true, + Ok(GuardedResourceReadback::Missing) => { + return self + .repository_execution( + self.finish_terminal( + claim, + worker_id, + ContentLockDeletionFailureCode::StateCorrupt, + ) + .await, + ) + .with_evidence(pubky_healthy) + .with_evidence(DeletionDependencyEvidence::healthy( + DeletionDependencySource::PubkyResource, + )); + } + Ok(GuardedResourceReadback::Replaced) => { + return self + .repository_execution( + self.finish_terminal( + claim, + worker_id, + ContentLockDeletionFailureCode::ResourceReplaced, + ) + .await, + ) + .with_evidence(pubky_healthy) + .with_evidence(DeletionDependencyEvidence::healthy( + DeletionDependencySource::PubkyResource, + )); + } + Err(error) => { + return error_execution(&error, DeletionDependencySource::PubkyResource) + .with_evidence(pubky_healthy); + } + } + } + + let resource_evidence = if resource_was_read { + DeletionDependencyEvidence::healthy(DeletionDependencySource::PubkyResource) + } else { + DeletionDependencyEvidence::none() + }; + + match self + .dependencies + .tombstones + .read_tombstone(&claim.job.creator, &content_lock_path, &tombstone) + .await + { + Ok(TombstoneReadback::Exact) => self + .repository_execution( + self.advance(claim, worker_id, ContentLockDeletionPhase::DeleteTombstone) + .await, + ) + .with_evidence(pubky_healthy) + .with_evidence(resource_evidence), + Ok(TombstoneReadback::Missing) => self + .repository_execution( + self.finish_terminal( + claim, + worker_id, + ContentLockDeletionFailureCode::TombstoneMissing, + ) + .await, + ) + .with_evidence(pubky_healthy) + .with_evidence(resource_evidence), + Ok(TombstoneReadback::Replaced) => self + .repository_execution( + self.finish_terminal( + claim, + worker_id, + ContentLockDeletionFailureCode::TombstoneReplaced, + ) + .await, + ) + .with_evidence(pubky_healthy) + .with_evidence(resource_evidence), + Err(error) => error_execution(&error, DeletionDependencySource::PubkyReadback) + .with_evidence(resource_evidence), + } + } + + async fn verify_tombstone_for_purge_with_evidence( + &self, + claim: &ClaimedContentLockDeletionJob, + worker_id: &str, + ) -> DeletionPhaseExecution { + let tombstone = tombstone(claim); + let content_lock_path = ContentLockPath::from_lock_id(claim.job.lock_id.clone()); + let readback = self + .dependencies + .tombstones + .read_tombstone(&claim.job.creator, &content_lock_path, &tombstone) + .await; + let pubky_healthy = + DeletionDependencyEvidence::healthy(DeletionDependencySource::PubkyReadback); + match readback { + Ok(TombstoneReadback::Exact) => self + .repository_execution( + self.advance( + claim, + worker_id, + ContentLockDeletionPhase::PurgeOperationalState, + ) + .await, + ) + .with_evidence(pubky_healthy), + Ok(TombstoneReadback::Missing) => self + .repository_execution( + self.finish_terminal( + claim, + worker_id, + ContentLockDeletionFailureCode::TombstoneMissing, + ) + .await, + ) + .with_evidence(pubky_healthy), + Ok(TombstoneReadback::Replaced) => self + .repository_execution( + self.finish_terminal( + claim, + worker_id, + ContentLockDeletionFailureCode::TombstoneReplaced, + ) + .await, + ) + .with_evidence(pubky_healthy), + Err(error) => error_execution(&error, DeletionDependencySource::PubkyReadback), + } + } + + async fn advance( + &self, + claim: &ClaimedContentLockDeletionJob, + worker_id: &str, + next_phase: ContentLockDeletionPhase, + ) -> DeletionPhaseExecutionOutcome { + match self + .dependencies + .deletions + .advance_phase(claim.job.job_id, worker_id, claim.claim_token, next_phase) + .await + { + Ok(AdvanceContentLockDeletionPhaseResult::Advanced(_)) => { + DeletionPhaseExecutionOutcome::Progressed + } + Ok(AdvanceContentLockDeletionPhaseResult::ClaimLost) => { + DeletionPhaseExecutionOutcome::ClaimLost + } + Ok(AdvanceContentLockDeletionPhaseResult::ObligationsPending) => { + DeletionPhaseExecutionOutcome::Deferred + } + Ok(AdvanceContentLockDeletionPhaseResult::TerminalFailure(failure_code)) => { + self.finish_terminal(claim, worker_id, failure_code).await + } + Err(error) => error_outcome(&error), + } + } + + async fn finish_terminal( + &self, + claim: &ClaimedContentLockDeletionJob, + worker_id: &str, + failure_code: ContentLockDeletionFailureCode, + ) -> DeletionPhaseExecutionOutcome { + match self + .dependencies + .deletions + .finish( + claim.job.job_id, + worker_id, + claim.claim_token, + Some(failure_code), + ) + .await + { + Ok(Some(_)) => DeletionPhaseExecutionOutcome::TerminalFailed, + Ok(None) => DeletionPhaseExecutionOutcome::ClaimLost, + Err(error) => error_outcome(&error), + } + } +} + +fn tombstone(claim: &ClaimedContentLockDeletionJob) -> ContentLockDeletionTombstone { + ContentLockDeletionTombstone::new(claim.job.lock_id.clone(), claim.job.deletion_started_at) +} + +#[cfg(test)] +mod tests; diff --git a/locks-service/src/application/use_cases/execute_content_lock_deletion_phase/tests.rs b/locks-service/src/application/use_cases/execute_content_lock_deletion_phase/tests.rs new file mode 100644 index 0000000..bfff89d --- /dev/null +++ b/locks-service/src/application/use_cases/execute_content_lock_deletion_phase/tests.rs @@ -0,0 +1,1084 @@ +use std::{ + collections::VecDeque, + sync::{ + Arc, Mutex, + atomic::{AtomicBool, Ordering}, + }, +}; + +use async_trait::async_trait; +use locks_core::{ + content_lock_deletion::ContentLockDeletionTombstone, + ids::{ContentLockPath, CreatorPubky, GuardedResourceHash, LockId}, + lock_policy::ContentLock, +}; +use time::{Duration, OffsetDateTime, macros::datetime}; +use uuid::Uuid; + +use super::*; +use crate::application::use_cases::deletion_test_fixtures::claimed_deletion_job; +use crate::application::{ + models::{ + AccessCredentialLookupKey, AccessCredentialRecord, ContentLockDeletionJob, + FinalAccessWindows, GuardedResourceRecord, PrepareForceDeletionResult, + }, + ports::{ + ContentLockDeletionActionAcquireResult, ContentLockDeletionActionClaim, + ContentLockDeletionActionGuard, GuardedResourceReadback, + }, +}; + +const NOW: OffsetDateTime = datetime!(2026-08-17 12:00:00 UTC); + +#[tokio::test] +async fn withdraw_exact_progresses_and_releases_guard() { + let h = Harness::new(); + let outcome = h.execute(ContentLockDeletionPhase::Withdraw).await; + assert_eq!(outcome, DeletionPhaseExecutionOutcome::Progressed); + assert_eq!( + h.deletions.advances(), + vec![ContentLockDeletionPhase::StartPaymentDrain] + ); + assert!(h.actions.released.load(Ordering::SeqCst)); +} + +#[tokio::test] +async fn execute_matches_evidence_aware_outcome_for_every_phase() { + for phase in [ + ContentLockDeletionPhase::Withdraw, + ContentLockDeletionPhase::StartPaymentDrain, + ContentLockDeletionPhase::DrainPayments, + ContentLockDeletionPhase::DrainExistingCredentials, + ContentLockDeletionPhase::IssueFinalCredentials, + ContentLockDeletionPhase::DrainFinalReads, + ContentLockDeletionPhase::DeleteContent, + ContentLockDeletionPhase::DeleteTombstone, + ContentLockDeletionPhase::PurgeOperationalState, + ] { + let outcome_harness = Harness::new(); + outcome_harness + .tombstones + .set_reads([TombstoneReadback::Exact; 4]); + let outcome = outcome_harness.execute(phase).await; + + let evidence_harness = Harness::new(); + evidence_harness + .tombstones + .set_reads([TombstoneReadback::Exact; 4]); + let evidence_outcome = evidence_harness.execute_with_evidence(phase).await.outcome; + + assert_eq!(outcome, evidence_outcome, "phase {phase:?}"); + } +} + +#[tokio::test] +async fn withdraw_reclaim_preserves_replacement_after_phase_advance_failure() { + let h = Harness::new(); + *h.deletions.advance_error.lock().unwrap() = Some(ApplicationError::Storage { + message: "phase persistence unavailable after publication".to_owned(), + }); + + let first = h + .executor() + .execute(claim(ContentLockDeletionPhase::Withdraw), "worker-a") + .await; + assert_eq!( + first, + DeletionPhaseExecutionOutcome::TransientDependencyFailure + ); + assert_eq!(h.tombstones.withdraw_count(), 1); + assert!(h.deletions.advances().is_empty()); + + h.tombstones.replace_public_bytes(); + + let mut reclaimed = claim(ContentLockDeletionPhase::Withdraw); + reclaimed.claim_token = Uuid::from_u128(3); + let second = h.executor().execute(reclaimed, "worker-b").await; + + assert_eq!(second, DeletionPhaseExecutionOutcome::TerminalFailed); + assert_eq!(h.tombstones.withdraw_count(), 1); + assert!(h.tombstones.replacement_is_present()); + assert_eq!( + h.deletions.finishes(), + vec![ContentLockDeletionFailureCode::TombstoneReplaced] + ); + assert_eq!( + h.deletions.advances(), + Vec::::new() + ); +} + +#[tokio::test] +async fn missing_tombstone_finishes_exact_claim_with_stable_code() { + let h = Harness::new(); + h.tombstones.set_reads([TombstoneReadback::Missing]); + let outcome = h + .executor() + .execute(claim(ContentLockDeletionPhase::DeleteContent), "worker") + .await; + assert_eq!(outcome, DeletionPhaseExecutionOutcome::TerminalFailed); + assert_eq!( + h.deletions.finishes(), + vec![ContentLockDeletionFailureCode::TombstoneMissing] + ); + assert!(h.resources.observed().is_empty()); +} + +#[tokio::test] +async fn replaced_tombstone_is_terminal() { + let h = Harness::new(); + h.tombstones.set_reads([TombstoneReadback::Replaced]); + let outcome = h + .executor() + .execute(claim(ContentLockDeletionPhase::DeleteTombstone), "worker") + .await; + assert_eq!(outcome, DeletionPhaseExecutionOutcome::TerminalFailed); + assert_eq!( + h.deletions.finishes(), + vec![ContentLockDeletionFailureCode::TombstoneReplaced] + ); + assert_eq!(h.tombstones.read_count(), 1); +} + +#[tokio::test] +async fn exact_checks_verify_every_sorted_deduplicated_content_generation() { + let h = Harness::new(); + h.tombstones.set_reads([TombstoneReadback::Exact; 4]); + let outcome = h + .executor() + .execute(claim(ContentLockDeletionPhase::DeleteContent), "worker") + .await; + assert_eq!(outcome, DeletionPhaseExecutionOutcome::Progressed); + assert_eq!( + h.resources.observed(), + vec![ + "/priv/locks.app/content/a".to_owned(), + "/priv/locks.app/content/m".to_owned(), + "/priv/locks.app/content/z".to_owned(), + ] + ); + assert_eq!(h.tombstones.read_count(), 4); + assert_eq!( + h.deletions.advances(), + vec![ContentLockDeletionPhase::DeleteTombstone] + ); +} + +#[tokio::test] +async fn preexisting_guarded_resource_replacement_is_terminal_and_not_deleted() { + let h = Harness::new(); + h.resources + .set_outcomes([GuardedResourceReadback::Replaced]); + h.tombstones.set_reads([TombstoneReadback::Exact]); + + let outcome = h + .executor() + .execute(claim(ContentLockDeletionPhase::DeleteContent), "worker") + .await; + + assert_eq!(outcome, DeletionPhaseExecutionOutcome::TerminalFailed); + assert_eq!( + h.deletions.finishes(), + vec![ContentLockDeletionFailureCode::ResourceReplaced] + ); + assert!(h.deletions.advances().is_empty()); + assert_eq!(h.resources.observed().len(), 1); +} + +#[tokio::test] +async fn final_tombstone_loss_after_content_deletion_fails_before_phase_advance() { + let h = Harness::new(); + h.tombstones.set_reads([ + TombstoneReadback::Exact, + TombstoneReadback::Exact, + TombstoneReadback::Exact, + TombstoneReadback::Missing, + ]); + let outcome = h + .executor() + .execute(claim(ContentLockDeletionPhase::DeleteContent), "worker") + .await; + + assert_eq!(outcome, DeletionPhaseExecutionOutcome::TerminalFailed); + assert!(h.deletions.advances().is_empty()); + assert_eq!( + h.deletions.finishes(), + vec![ContentLockDeletionFailureCode::TombstoneMissing] + ); +} + +#[tokio::test] +async fn exact_tombstone_remains_published_at_purge_handoff() { + let h = Harness::new(); + h.tombstones.set_reads([TombstoneReadback::Exact]); + let outcome = h + .executor() + .execute(claim(ContentLockDeletionPhase::DeleteTombstone), "worker") + .await; + assert_eq!(outcome, DeletionPhaseExecutionOutcome::Progressed); + assert_eq!( + h.deletions.advances(), + vec![ContentLockDeletionPhase::PurgeOperationalState] + ); +} + +#[tokio::test] +async fn busy_action_guard_defers_without_side_effects() { + let h = Harness::new(); + h.actions.busy.store(true, Ordering::SeqCst); + let outcome = h.execute(ContentLockDeletionPhase::Withdraw).await; + assert_eq!(outcome, DeletionPhaseExecutionOutcome::Deferred); + assert!(h.deletions.advances().is_empty()); + assert_eq!(h.tombstones.withdraw_count(), 0); +} + +#[tokio::test] +async fn action_ownership_unexpected_error_is_fatal() { + let h = Harness::new(); + *h.actions.error.lock().unwrap() = Some(ApplicationError::MissingRecord { + record: "deletion_action_guard", + }); + + let outcome = h.execute(ContentLockDeletionPhase::Withdraw).await; + + assert_eq!(outcome, DeletionPhaseExecutionOutcome::FatalFailure); + assert_eq!(h.tombstones.withdraw_count(), 0); +} + +#[tokio::test] +async fn tombstone_storage_error_is_transient_but_unexpected_error_is_fatal() { + for (error, expected) in [ + ( + ApplicationError::Storage { + message: "temporary homeserver outage".to_owned(), + }, + DeletionPhaseExecutionOutcome::TransientDependencyFailure, + ), + ( + ApplicationError::MissingRecord { + record: "content_lock_tombstone", + }, + DeletionPhaseExecutionOutcome::FatalFailure, + ), + ] { + let h = Harness::new(); + *h.tombstones.withdraw_error.lock().unwrap() = Some(error); + + let outcome = h + .executor() + .execute(claim(ContentLockDeletionPhase::Withdraw), "worker") + .await; + + assert_eq!(outcome, expected); + } +} + +#[tokio::test] +async fn final_credential_crypto_error_is_fatal() { + let h = Harness::new(); + *h.finals.error.lock().unwrap() = Some(ApplicationError::FinalCredentialSecret { + message: "secret detail".to_owned(), + }); + + let outcome = h + .executor() + .execute( + claim(ContentLockDeletionPhase::IssueFinalCredentials), + "worker", + ) + .await; + + assert_eq!(outcome, DeletionPhaseExecutionOutcome::FatalFailure); +} + +#[tokio::test] +async fn invalid_state_transition_is_fatal_instead_of_business_deferral() { + let h = Harness::new(); + *h.deletions.advance_error.lock().unwrap() = + Some(ApplicationError::InvalidContentLockDeletionState { + message: "invariant detail".to_owned(), + }); + + let outcome = h + .executor() + .execute( + claim(ContentLockDeletionPhase::DrainExistingCredentials), + "worker", + ) + .await; + + assert_eq!(outcome, DeletionPhaseExecutionOutcome::FatalFailure); +} + +#[tokio::test] +async fn repository_obligations_pending_is_healthy_deferral() { + let h = Harness::new(); + h.deletions + .obligations_pending + .store(true, Ordering::SeqCst); + + let outcome = h + .executor() + .execute( + claim(ContentLockDeletionPhase::DrainExistingCredentials), + "worker", + ) + .await; + + assert_eq!(outcome, DeletionPhaseExecutionOutcome::Deferred); +} + +#[tokio::test] +async fn missed_final_credential_issuance_deadline_terminalizes_without_advancing() { + let h = Harness::new(); + h.deletions + .issuance_deadline_missed + .store(true, Ordering::SeqCst); + + let outcome = h + .executor() + .execute( + claim(ContentLockDeletionPhase::IssueFinalCredentials), + "worker", + ) + .await; + + assert_eq!(outcome, DeletionPhaseExecutionOutcome::TerminalFailed); + assert_eq!( + h.deletions.finishes(), + vec![ContentLockDeletionFailureCode::StateCorrupt] + ); + assert!(h.deletions.advances().is_empty()); + assert!(h.resources.observed().is_empty()); +} + +#[tokio::test] +async fn transient_provider_error_remains_retryable_across_payment_boundary() { + let h = Harness::new(); + *h.payments.error.lock().unwrap() = Some(ApplicationError::Verifier { + message: "provider detail".to_owned(), + }); + + let outcome = h + .executor() + .execute(claim(ContentLockDeletionPhase::DrainPayments), "worker") + .await; + + assert_eq!( + outcome, + DeletionPhaseExecutionOutcome::TransientDependencyFailure + ); +} + +#[tokio::test] +async fn paykit_failure_carries_only_paykit_unavailable_evidence() { + let h = Harness::new(); + *h.payments.error.lock().unwrap() = Some(ApplicationError::Verifier { + message: "provider detail".to_owned(), + }); + + let execution = h + .executor() + .execute_with_evidence(claim(ContentLockDeletionPhase::DrainPayments), "worker") + .await; + + assert_eq!( + execution.outcome, + DeletionPhaseExecutionOutcome::TransientDependencyFailure + ); + assert_eq!( + execution + .evidence + .status(DeletionDependencySource::PaymentProvider), + Some(DeletionDependencyStatus::Unavailable) + ); + assert_eq!( + execution + .evidence + .status(DeletionDependencySource::PubkyReadback), + None + ); + assert_eq!( + execution + .evidence + .status(DeletionDependencySource::RepositoryPhaseMutation), + None + ); +} + +#[tokio::test] +async fn successful_active_paykit_drain_is_healthy_evidence_while_deferred() { + let h = Harness::new(); + + let execution = h + .executor() + .execute_with_evidence(claim(ContentLockDeletionPhase::DrainPayments), "worker") + .await; + + assert_eq!(execution.outcome, DeletionPhaseExecutionOutcome::Deferred); + assert_eq!( + execution + .evidence + .status(DeletionDependencySource::PaymentProvider), + Some(DeletionDependencyStatus::Healthy) + ); +} + +#[tokio::test] +async fn pubky_success_followed_by_repository_failure_preserves_source_independence() { + let h = Harness::new(); + *h.deletions.advance_error.lock().unwrap() = Some(ApplicationError::Storage { + message: "local persistence unavailable".to_owned(), + }); + + let execution = h + .execute_with_evidence(ContentLockDeletionPhase::Withdraw) + .await; + + assert_eq!( + execution.outcome, + DeletionPhaseExecutionOutcome::TransientDependencyFailure + ); + assert_eq!( + execution + .evidence + .status(DeletionDependencySource::PubkyReadback), + None + ); + assert_eq!( + execution + .evidence + .status(DeletionDependencySource::PubkyWithdrawal), + Some(DeletionDependencyStatus::Healthy) + ); + assert_eq!( + execution + .evidence + .status(DeletionDependencySource::RepositoryPhaseMutation), + Some(DeletionDependencyStatus::Unavailable) + ); + assert_eq!( + execution + .evidence + .status(DeletionDependencySource::PaymentProvider), + None + ); +} + +#[tokio::test] +async fn delete_content_pubky_success_followed_by_advance_failure_preserves_stage_evidence() { + let h = Harness::new(); + h.tombstones.set_reads([TombstoneReadback::Exact; 4]); + *h.deletions.advance_error.lock().unwrap() = Some(ApplicationError::Storage { + message: "phase persistence unavailable".to_owned(), + }); + + let execution = h + .executor() + .execute_with_evidence(claim(ContentLockDeletionPhase::DeleteContent), "worker") + .await; + + assert_eq!( + execution.outcome, + DeletionPhaseExecutionOutcome::TransientDependencyFailure + ); + assert_eq!( + execution + .evidence + .status(DeletionDependencySource::PubkyReadback), + Some(DeletionDependencyStatus::Healthy) + ); + assert_eq!( + execution + .evidence + .status(DeletionDependencySource::PubkyResource), + Some(DeletionDependencyStatus::Healthy) + ); + assert_eq!( + execution + .evidence + .status(DeletionDependencySource::RepositoryPhaseMutation), + Some(DeletionDependencyStatus::Unavailable) + ); + assert_eq!( + execution + .evidence + .status(DeletionDependencySource::RepositoryQueueClaim), + None + ); +} + +#[tokio::test] +async fn busy_action_guard_defers_without_dependency_health_evidence() { + let h = Harness::new(); + h.actions.busy.store(true, Ordering::SeqCst); + + let execution = h + .execute_with_evidence(ContentLockDeletionPhase::Withdraw) + .await; + + assert_eq!(execution.outcome, DeletionPhaseExecutionOutcome::Deferred); + for source in [ + DeletionDependencySource::PaymentProvider, + DeletionDependencySource::PubkyReadback, + DeletionDependencySource::RepositoryQueueClaim, + DeletionDependencySource::RepositoryPhaseMutation, + ] { + assert_eq!(execution.evidence.status(source), None); + } +} + +#[tokio::test] +async fn repository_none_is_claim_lost_and_guard_is_released() { + let h = Harness::new(); + h.deletions.claim_live.store(false, Ordering::SeqCst); + let outcome = h.execute(ContentLockDeletionPhase::Withdraw).await; + assert_eq!(outcome, DeletionPhaseExecutionOutcome::ClaimLost); + assert!(h.actions.released.load(Ordering::SeqCst)); +} + +#[tokio::test] +async fn purge_is_deferred_task_ten_handoff() { + let h = Harness::new(); + let outcome = h + .executor() + .execute( + claim(ContentLockDeletionPhase::PurgeOperationalState), + "worker", + ) + .await; + assert_eq!(outcome, DeletionPhaseExecutionOutcome::Deferred); + assert!(h.deletions.advances().is_empty()); + assert!(h.actions.released.load(Ordering::SeqCst)); +} + +#[tokio::test] +async fn operation_evidence_does_not_infer_phase_health_from_purge_or_claim_loss() { + let h = Harness::new(); + let purge = h + .executor() + .execute_with_evidence( + claim(ContentLockDeletionPhase::PurgeOperationalState), + "worker", + ) + .await; + assert_eq!(purge.outcome, DeletionPhaseExecutionOutcome::Deferred); + assert_eq!( + purge + .evidence + .status(DeletionDependencySource::RepositoryPhaseMutation), + None + ); + + h.deletions.claim_live.store(false, Ordering::SeqCst); + let lost = h + .execute_with_evidence(ContentLockDeletionPhase::Withdraw) + .await; + assert_eq!(lost.outcome, DeletionPhaseExecutionOutcome::ClaimLost); + assert_eq!( + lost.evidence + .status(DeletionDependencySource::RepositoryPhaseMutation), + None + ); + assert_eq!( + lost.evidence + .status(DeletionDependencySource::PubkyWithdrawal), + Some(DeletionDependencyStatus::Healthy) + ); +} + +struct Harness { + deletions: FakeDeletions, + actions: FakeActions, + tombstones: FakeTombstones, + resources: FakeResources, + access: FakeAccess, + clock: FixedClock, + payments: FakePayments, + finals: FakeFinals, +} + +impl Harness { + fn new() -> Self { + Self { + deletions: FakeDeletions::new(), + actions: FakeActions::new(), + tombstones: FakeTombstones::new(), + resources: FakeResources::default(), + access: FakeAccess, + clock: FixedClock, + payments: FakePayments::default(), + finals: FakeFinals::default(), + } + } + + async fn execute(&self, phase: ContentLockDeletionPhase) -> DeletionPhaseExecutionOutcome { + self.executor().execute(claim(phase), "worker").await + } + + async fn execute_with_evidence( + &self, + phase: ContentLockDeletionPhase, + ) -> DeletionPhaseExecution { + self.executor() + .execute_with_evidence(claim(phase), "worker") + .await + } + + fn executor(&self) -> ContentLockDeletionPhaseExecutor<'_> { + ContentLockDeletionPhaseExecutor::new( + ContentLockDeletionPhaseExecutorDependencies { + deletions: &self.deletions, + action_ownership: &self.actions, + tombstones: &self.tombstones, + guarded_resources: &self.resources, + access_credentials: &self.access, + clock: &self.clock, + payments: &self.payments, + final_credentials: &self.finals, + }, + ContentLockDeletionPhaseExecutorConfig { + final_credential_issuance_window: Duration::minutes(15), + final_read_window: Duration::minutes(15), + final_credential_batch_limit: 10, + }, + ) + } +} + +#[derive(Default)] +struct FakeDeletions { + claim_live: AtomicBool, + obligations_pending: AtomicBool, + issuance_deadline_missed: AtomicBool, + advances: Mutex>, + finishes: Mutex>, + advance_error: Mutex>, +} +impl FakeDeletions { + fn new() -> Self { + Self { + claim_live: AtomicBool::new(true), + ..Self::default() + } + } + fn advances(&self) -> Vec { + self.advances.lock().unwrap().clone() + } + fn finishes(&self) -> Vec { + self.finishes.lock().unwrap().clone() + } + fn result(&self) -> Option { + self.claim_live + .load(Ordering::SeqCst) + .then(|| claim(ContentLockDeletionPhase::Withdraw).job) + } +} + +#[async_trait] +impl ContentLockDeletionRepository for FakeDeletions { + async fn begin_publication( + &self, + _: &CreatorPubky, + _: &LockId, + _: Uuid, + ) -> Result<(), ApplicationError> { + unreachable!() + } + async fn finish_publication( + &self, + _: &CreatorPubky, + _: &LockId, + _: Uuid, + ) -> Result { + unreachable!() + } + async fn abandon_publication( + &self, + _: &CreatorPubky, + _: &LockId, + _: Uuid, + ) -> Result { + unreachable!() + } + async fn publication_in_progress( + &self, + _: &CreatorPubky, + _: &LockId, + ) -> Result { + unreachable!() + } + async fn insert_job(&self, _: ContentLockDeletionJob) -> Result<(), ApplicationError> { + unreachable!() + } + async fn get_job( + &self, + _: &CreatorPubky, + _: &LockId, + ) -> Result, ApplicationError> { + unreachable!() + } + async fn claim_next( + &self, + _: &str, + _: time::Duration, + ) -> Result, ApplicationError> { + unreachable!() + } + async fn schedule_retry( + &self, + _: Uuid, + _: &str, + _: Uuid, + _: time::Duration, + ) -> Result, ApplicationError> { + unreachable!() + } + async fn defer( + &self, + _: Uuid, + _: &str, + _: Uuid, + _: time::Duration, + ) -> Result, ApplicationError> { + unreachable!() + } + async fn advance_phase( + &self, + _: Uuid, + _: &str, + _: Uuid, + next: ContentLockDeletionPhase, + ) -> Result { + if let Some(error) = self.advance_error.lock().unwrap().take() { + return Err(error); + } + if self.obligations_pending.load(Ordering::SeqCst) { + return Ok(AdvanceContentLockDeletionPhaseResult::ObligationsPending); + } + if self.issuance_deadline_missed.load(Ordering::SeqCst) { + return Ok(AdvanceContentLockDeletionPhaseResult::TerminalFailure( + ContentLockDeletionFailureCode::StateCorrupt, + )); + } + self.advances.lock().unwrap().push(next); + Ok(match self.result() { + Some(job) => AdvanceContentLockDeletionPhaseResult::Advanced(Box::new(job)), + None => AdvanceContentLockDeletionPhaseResult::ClaimLost, + }) + } + async fn finish( + &self, + _: Uuid, + _: &str, + _: Uuid, + code: Option, + ) -> Result, ApplicationError> { + self.finishes.lock().unwrap().push(code.unwrap()); + Ok(self.result()) + } + async fn resume_failed_job( + &self, + _: &CreatorPubky, + _: &LockId, + _: OffsetDateTime, + ) -> Result, ApplicationError> { + unreachable!() + } + async fn prepare_force_deletion( + &self, + _: &CreatorPubky, + _: &LockId, + ) -> Result { + unreachable!() + } + async fn complete_force_deletion( + &self, + _: Uuid, + _: &str, + _: Uuid, + ) -> Result { + unreachable!() + } + async fn has_force_receipt( + &self, + _: &CreatorPubky, + _: &LockId, + ) -> Result { + unreachable!() + } +} + +struct FakeGuard(Arc); +#[async_trait] +impl ContentLockDeletionActionGuard for FakeGuard { + async fn release(self: Box) -> Result<(), ApplicationError> { + self.0.store(true, Ordering::SeqCst); + Ok(()) + } +} +struct FakeActions { + busy: AtomicBool, + released: Arc, + error: Mutex>, +} +impl FakeActions { + fn new() -> Self { + Self { + busy: AtomicBool::new(false), + released: Arc::new(AtomicBool::new(false)), + error: Mutex::new(None), + } + } +} +#[async_trait] +impl ContentLockDeletionActionOwnership for FakeActions { + async fn try_acquire( + &self, + _: ContentLockDeletionActionClaim<'_>, + ) -> Result { + if let Some(error) = self.error.lock().unwrap().take() { + return Err(error); + } + if self.busy.load(Ordering::SeqCst) { + Ok(ContentLockDeletionActionAcquireResult::Busy) + } else { + Ok(ContentLockDeletionActionAcquireResult::Acquired(Box::new( + FakeGuard(self.released.clone()), + ))) + } + } +} + +struct FakeTombstones { + public_state: Mutex, + withdraw_error: Mutex>, + reads: Mutex>, + withdraw_count: Mutex, + read_count: Mutex, +} + +#[derive(Clone, Copy, PartialEq, Eq)] +enum FakePublicLockState { + Original, + Tombstone, + Replacement, +} + +impl FakeTombstones { + fn new() -> Self { + Self { + public_state: Mutex::new(FakePublicLockState::Original), + withdraw_error: Mutex::new(None), + reads: Mutex::new(VecDeque::new()), + withdraw_count: Mutex::new(0), + read_count: Mutex::new(0), + } + } + + fn set_reads(&self, values: [TombstoneReadback; N]) { + *self.reads.lock().unwrap() = values.into(); + } + fn withdraw_count(&self) -> usize { + *self.withdraw_count.lock().unwrap() + } + fn replace_public_bytes(&self) { + *self.public_state.lock().unwrap() = FakePublicLockState::Replacement; + } + fn replacement_is_present(&self) -> bool { + *self.public_state.lock().unwrap() == FakePublicLockState::Replacement + } + fn read_count(&self) -> usize { + *self.read_count.lock().unwrap() + } +} +#[async_trait] +impl ContentLockTombstoneRepository for FakeTombstones { + async fn withdraw_content_lock( + &self, + _: CreatorPubky, + _: ContentLockPath, + _: &ContentLock, + _: &ContentLockDeletionTombstone, + ) -> Result { + if let Some(error) = self.withdraw_error.lock().unwrap().take() { + return Err(error); + } + let mut state = self.public_state.lock().unwrap(); + match *state { + FakePublicLockState::Original => { + *self.withdraw_count.lock().unwrap() += 1; + *state = FakePublicLockState::Tombstone; + Ok(TombstoneReadback::Exact) + } + FakePublicLockState::Tombstone => Ok(TombstoneReadback::Exact), + FakePublicLockState::Replacement => Ok(TombstoneReadback::Replaced), + } + } + async fn read_tombstone( + &self, + _: &CreatorPubky, + _: &ContentLockPath, + _: &ContentLockDeletionTombstone, + ) -> Result { + *self.read_count.lock().unwrap() += 1; + Ok(self.reads.lock().unwrap().pop_front().unwrap()) + } + + async fn force_delete_content_lock_and_verify_absent( + &self, + _: &CreatorPubky, + _: &ContentLockPath, + ) -> Result<(), ApplicationError> { + unreachable!() + } +} + +#[derive(Default)] +struct FakeResources { + observed: Mutex>, + outcomes: Mutex>, +} +impl FakeResources { + fn observed(&self) -> Vec { + self.observed.lock().unwrap().clone() + } + + fn set_outcomes(&self, outcomes: impl IntoIterator) { + *self.outcomes.lock().unwrap() = outcomes.into_iter().collect(); + } +} +#[async_trait] +impl GuardedResourceRepository for FakeResources { + async fn upsert_guarded_resource( + &self, + _: GuardedResourceRecord, + ) -> Result<(), ApplicationError> { + unreachable!() + } + async fn get_guarded_resource( + &self, + _: &CreatorPubky, + _: &str, + _: &GuardedResourceHash, + ) -> Result, ApplicationError> { + unreachable!() + } + async fn get_current_guarded_resource( + &self, + _: &CreatorPubky, + _: &str, + ) -> Result, ApplicationError> { + unreachable!() + } + async fn delete_guarded_resource( + &self, + _: &CreatorPubky, + path: &str, + ) -> Result { + self.observed.lock().unwrap().push(path.to_owned()); + Ok(false) + } + async fn read_guarded_resource_generation( + &self, + _: &CreatorPubky, + path: &str, + _: &GuardedResourceHash, + ) -> Result { + self.observed.lock().unwrap().push(path.to_owned()); + Ok(self + .outcomes + .lock() + .unwrap() + .pop_front() + .unwrap_or(GuardedResourceReadback::Exact)) + } +} + +struct FakeAccess; +#[async_trait] +impl AccessCredentialStore for FakeAccess { + async fn insert_access_credential( + &self, + _: &LockId, + _: AccessCredentialLookupKey, + _: AccessCredentialRecord, + ) -> Result<(), ApplicationError> { + unreachable!() + } + async fn get_access_credential( + &self, + _: &AccessCredentialLookupKey, + ) -> Result, ApplicationError> { + unreachable!() + } + async fn delete_access_credential( + &self, + _: &AccessCredentialLookupKey, + ) -> Result<(), ApplicationError> { + unreachable!() + } + async fn initialize_final_access_windows( + &self, + _: Uuid, + _: &str, + _: Uuid, + _: Duration, + _: Duration, + ) -> Result { + Ok(InitializeFinalAccessWindowsResult::Initialized( + FinalAccessWindows { + issuance_started_at: NOW, + credential_issuance_deadline: NOW + Duration::minutes(15), + read_deadline: NOW + Duration::minutes(30), + }, + )) + } +} +struct FixedClock; +impl Clock for FixedClock { + fn now(&self) -> OffsetDateTime { + NOW + } +} +#[derive(Default)] +struct FakePayments { + error: Mutex>, +} +#[async_trait] +impl ContentLockPaymentDrainExecutor for FakePayments { + async fn execute_claimed( + &self, + _: ClaimedContentLockDeletionJob, + _: &str, + ) -> DeletionPhaseExecution { + if let Some(error) = self.error.lock().unwrap().take() { + return error_execution(&error, DeletionDependencySource::PaymentProvider); + } + DeletionPhaseExecution::new(DeletionPhaseExecutionOutcome::Deferred).with_evidence( + DeletionDependencyEvidence::healthy(DeletionDependencySource::PaymentProvider), + ) + } +} +#[derive(Default)] +struct FakeFinals { + error: Mutex>, +} +#[async_trait] +impl FinalCredentialMaterializer for FakeFinals { + async fn materialize( + &self, + _: MaterializeFinalCredentialsRequest<'_>, + ) -> Result { + if let Some(error) = self.error.lock().unwrap().take() { + return Err(error); + } + Ok(MaterializeFinalCredentialsOutcome { + materialized_count: 0, + }) + } +} + +fn claim(phase: ContentLockDeletionPhase) -> ClaimedContentLockDeletionJob { + claimed_deletion_job(NOW, phase, false) +} diff --git a/locks-service/src/application/use_cases/execute_forced_content_lock_deletion.rs b/locks-service/src/application/use_cases/execute_forced_content_lock_deletion.rs new file mode 100644 index 0000000..b40a019 --- /dev/null +++ b/locks-service/src/application/use_cases/execute_forced_content_lock_deletion.rs @@ -0,0 +1,220 @@ +use std::collections::BTreeSet; + +use locks_core::ids::ContentLockPath; + +use crate::application::{ + errors::ApplicationError, + models::ClaimedContentLockDeletionJob, + ports::{ + Clock, ContentLockDeletionActionAcquireResult, ContentLockDeletionActionClaim, + ContentLockDeletionActionOwnership, ContentLockDeletionRepository, + ContentLockTombstoneRepository, GuardedResourceRepository, + }, +}; + +use super::execute_content_lock_deletion_phase::{ + DeletionDependencyEvidence, DeletionDependencySource, DeletionExecutionErrorClass, + classify_deletion_execution_error, +}; + +/// Closed, secret-free result of one active-force external execution attempt. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum ForcedContentLockDeletionOutcome { + Completed, + Deferred, + ClaimLost, + TransientDependencyFailure, + FatalFailure, +} + +fn error_outcome(error: &ApplicationError) -> ForcedContentLockDeletionOutcome { + match classify_deletion_execution_error(error) { + DeletionExecutionErrorClass::TransientDependency => { + ForcedContentLockDeletionOutcome::TransientDependencyFailure + } + DeletionExecutionErrorClass::Fatal => ForcedContentLockDeletionOutcome::FatalFailure, + } +} + +/// One force outcome plus exact, identifier-free dependency observations. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct ForcedContentLockDeletionExecution { + pub outcome: ForcedContentLockDeletionOutcome, + pub evidence: DeletionDependencyEvidence, +} + +impl ForcedContentLockDeletionExecution { + fn new(outcome: ForcedContentLockDeletionOutcome) -> Self { + Self { + outcome, + evidence: DeletionDependencyEvidence::none(), + } + } + + fn with_evidence(mut self, evidence: DeletionDependencyEvidence) -> Self { + self.evidence = self.evidence.merge(evidence); + self + } +} + +fn error_execution( + error: &ApplicationError, + source: DeletionDependencySource, +) -> ForcedContentLockDeletionExecution { + ForcedContentLockDeletionExecution::new(error_outcome(error)) + .with_evidence(DeletionDependencyEvidence::unavailable(source)) +} + +/// External dependencies required by active-force execution. +pub struct ExecuteForcedContentLockDeletionDependencies<'a> { + pub action_ownership: &'a dyn ContentLockDeletionActionOwnership, + pub tombstones: &'a dyn ContentLockTombstoneRepository, + pub guarded_resources: &'a dyn GuardedResourceRepository, + pub deletions: &'a dyn ContentLockDeletionRepository, + pub clock: &'a dyn Clock, +} + +/// Executes the force path for an already-claimed active deletion job. +pub struct ExecuteForcedContentLockDeletionUseCase<'a> { + dependencies: ExecuteForcedContentLockDeletionDependencies<'a>, +} + +impl<'a> ExecuteForcedContentLockDeletionUseCase<'a> { + pub fn new(dependencies: ExecuteForcedContentLockDeletionDependencies<'a>) -> Self { + Self { dependencies } + } + + pub async fn execute( + &self, + claim: ClaimedContentLockDeletionJob, + worker_id: &str, + ) -> ForcedContentLockDeletionOutcome { + self.execute_with_evidence(claim, worker_id).await.outcome + } + + pub async fn execute_with_evidence( + &self, + claim: ClaimedContentLockDeletionJob, + worker_id: &str, + ) -> ForcedContentLockDeletionExecution { + if claim.job.force_requested_at.is_none() { + return ForcedContentLockDeletionExecution::new( + ForcedContentLockDeletionOutcome::Deferred, + ); + } + + let guard = match self + .dependencies + .action_ownership + .try_acquire(ContentLockDeletionActionClaim { + job_id: claim.job.job_id, + worker_id, + claim_token: claim.claim_token, + expected_phase: claim.job.phase, + force: true, + }) + .await + { + Ok(ContentLockDeletionActionAcquireResult::Acquired(guard)) => guard, + Ok(ContentLockDeletionActionAcquireResult::Busy) => { + return ForcedContentLockDeletionExecution::new( + ForcedContentLockDeletionOutcome::Deferred, + ); + } + Ok(ContentLockDeletionActionAcquireResult::ClaimLost) => { + return ForcedContentLockDeletionExecution::new( + ForcedContentLockDeletionOutcome::ClaimLost, + ); + } + Err(error) => { + return error_execution(&error, DeletionDependencySource::RepositoryActionLock); + } + }; + + let execution = self.execute_guarded(&claim, worker_id).await.with_evidence( + DeletionDependencyEvidence::healthy(DeletionDependencySource::RepositoryActionLock), + ); + if let Err(error) = guard.release().await { + return error_execution( + &error, + DeletionDependencySource::RepositoryActionLockRelease, + ) + .with_evidence(execution.evidence); + } + execution.with_evidence(DeletionDependencyEvidence::healthy( + DeletionDependencySource::RepositoryActionLockRelease, + )) + } + + async fn execute_guarded( + &self, + claim: &ClaimedContentLockDeletionJob, + worker_id: &str, + ) -> ForcedContentLockDeletionExecution { + let public_path = ContentLockPath::from_lock_id(claim.job.lock_id.clone()); + if let Err(error) = self + .dependencies + .tombstones + .force_delete_content_lock_and_verify_absent(&claim.job.creator, &public_path) + .await + { + return error_execution(&error, DeletionDependencySource::PubkyForcePublic); + } + + let mut evidence = + DeletionDependencyEvidence::healthy(DeletionDependencySource::PubkyForcePublic); + for path in frozen_resource_paths(claim) { + evidence = evidence.merge( + match self + .dependencies + .guarded_resources + .delete_guarded_resource(&claim.job.creator, &path) + .await + { + Ok(_) => { + DeletionDependencyEvidence::healthy(DeletionDependencySource::PubkyResource) + } + Err(_) => DeletionDependencyEvidence::unavailable( + DeletionDependencySource::PubkyResource, + ), + }, + ); + } + + let execution = match self + .dependencies + .deletions + .complete_force_deletion(claim.job.job_id, worker_id, claim.claim_token) + .await + { + Ok(true) => { + ForcedContentLockDeletionExecution::new(ForcedContentLockDeletionOutcome::Completed) + .with_evidence(DeletionDependencyEvidence::healthy( + DeletionDependencySource::RepositoryForceReceipt, + )) + } + Ok(false) => { + ForcedContentLockDeletionExecution::new(ForcedContentLockDeletionOutcome::ClaimLost) + } + Err(error) => error_execution(&error, DeletionDependencySource::RepositoryForceReceipt), + }; + execution.with_evidence(evidence) + } +} + +fn frozen_resource_paths(claim: &ClaimedContentLockDeletionJob) -> BTreeSet { + let mut paths = claim + .job + .frozen_content_lock + .secondary_resources + .keys() + .cloned() + .collect::>(); + if let Some(primary) = &claim.job.frozen_content_lock.primary_resource { + paths.insert(primary.path.clone()); + } + paths +} + +#[cfg(test)] +mod tests; diff --git a/locks-service/src/application/use_cases/execute_forced_content_lock_deletion/tests.rs b/locks-service/src/application/use_cases/execute_forced_content_lock_deletion/tests.rs new file mode 100644 index 0000000..89f65da --- /dev/null +++ b/locks-service/src/application/use_cases/execute_forced_content_lock_deletion/tests.rs @@ -0,0 +1,611 @@ +use std::{ + collections::HashSet, + sync::{ + Arc, Mutex, + atomic::{AtomicBool, Ordering}, + }, +}; + +use async_trait::async_trait; +use locks_core::{ + content_lock_deletion::ContentLockDeletionTombstone, + ids::{ContentLockPath, CreatorPubky, GuardedResourceHash, LockId}, + lock_policy::ContentLock, +}; +use time::{OffsetDateTime, macros::datetime}; +use uuid::Uuid; + +use super::*; +use crate::application::use_cases::deletion_test_fixtures::claimed_deletion_job; +use crate::application::use_cases::execute_content_lock_deletion_phase::{ + DeletionDependencyEvidence, DeletionDependencySource, DeletionDependencyStatus, +}; +use crate::application::{ + errors::ApplicationError, + models::{ + AdvanceContentLockDeletionPhaseResult, ClaimedContentLockDeletionJob, + ContentLockDeletionFailureCode, ContentLockDeletionJob, ContentLockDeletionPhase, + GuardedResourceRecord, PrepareForceDeletionResult, + }, + ports::{ + ContentLockDeletionActionGuard, ContentLockDeletionActionOwnership, + ContentLockDeletionRepository, ContentLockTombstoneRepository, GuardedResourceRepository, + TombstoneReadback, + }, +}; + +const NOW: OffsetDateTime = datetime!(2026-08-17 12:00:00 UTC); + +#[test] +fn outcomes_have_closed_path_free_debug_output() { + assert_eq!( + format!("{:?}", ForcedContentLockDeletionOutcome::Completed), + "Completed" + ); + assert_eq!( + format!("{:?}", ForcedContentLockDeletionOutcome::Deferred), + "Deferred" + ); + assert_eq!( + format!("{:?}", ForcedContentLockDeletionOutcome::ClaimLost), + "ClaimLost" + ); + assert_eq!( + format!( + "{:?}", + ForcedContentLockDeletionOutcome::TransientDependencyFailure + ), + "TransientDependencyFailure" + ); + assert_eq!( + format!("{:?}", ForcedContentLockDeletionOutcome::FatalFailure), + "FatalFailure" + ); +} + +#[tokio::test] +async fn deletes_public_path_first_then_attempts_sorted_deduplicated_resources_and_completes() { + let h = Harness::new(); + + let outcome = h.execute().await; + + assert_eq!(outcome, ForcedContentLockDeletionOutcome::Completed); + assert_eq!( + h.operations(), + vec![ + "public".to_owned(), + "resource:/priv/locks.app/content/a".to_owned(), + "resource:/priv/locks.app/content/m".to_owned(), + "resource:/priv/locks.app/content/z".to_owned(), + "complete".to_owned(), + "release".to_owned(), + ] + ); +} + +#[tokio::test] +async fn resource_errors_are_best_effort_and_still_complete_the_force_receipt() { + let h = Harness::new(); + h.resources + .fail_paths + .lock() + .unwrap() + .insert("/priv/locks.app/content/m".to_owned()); + + let outcome = h.execute().await; + + assert_eq!(outcome, ForcedContentLockDeletionOutcome::Completed); + assert_eq!(h.deletions.complete_calls.load(Ordering::SeqCst), 1); + assert!( + h.operations() + .iter() + .any(|op| op == "resource:/priv/locks.app/content/z") + ); +} + +#[tokio::test] +async fn stale_claim_cannot_complete_after_all_external_attempts() { + let h = Harness::new(); + h.deletions.claim_live.store(false, Ordering::SeqCst); + + let outcome = h.execute().await; + + assert_eq!(outcome, ForcedContentLockDeletionOutcome::ClaimLost); + assert_eq!(h.deletions.complete_calls.load(Ordering::SeqCst), 1); + assert_eq!(h.operations().last().map(String::as_str), Some("release")); +} + +#[tokio::test] +async fn reclaimed_force_claim_replays_deletions_after_receipt_persistence_failure() { + let h = Harness::new(); + h.deletions.fail_complete_once.store(true, Ordering::SeqCst); + + let first = h.use_case().execute(force_claim(), "worker-a").await; + assert_eq!( + first, + ForcedContentLockDeletionOutcome::TransientDependencyFailure + ); + assert_eq!(h.deletions.complete_calls.load(Ordering::SeqCst), 1); + + let mut reclaimed = force_claim(); + reclaimed.claim_token = Uuid::from_u128(3); + let second = h.use_case().execute(reclaimed, "worker-b").await; + + assert_eq!(second, ForcedContentLockDeletionOutcome::Completed); + assert_eq!(h.deletions.complete_calls.load(Ordering::SeqCst), 2); + assert_eq!( + h.operations() + .iter() + .filter(|operation| operation.as_str() == "public") + .count(), + 2 + ); +} + +#[tokio::test] +async fn busy_guard_defers_without_effects() { + let h = Harness::new(); + h.actions.busy.store(true, Ordering::SeqCst); + + let outcome = h.execute().await; + + assert_eq!(outcome, ForcedContentLockDeletionOutcome::Deferred); + assert!(h.operations().is_empty()); + assert_eq!(h.deletions.complete_calls.load(Ordering::SeqCst), 0); +} + +#[tokio::test] +async fn post_lock_stale_force_claim_is_lost_without_external_action() { + let h = Harness::new(); + h.actions.claim_live.store(false, Ordering::SeqCst); + let outcome = h.execute().await; + assert_eq!(outcome, ForcedContentLockDeletionOutcome::ClaimLost); + assert!(h.operations().is_empty()); + assert_eq!(h.deletions.complete_calls.load(Ordering::SeqCst), 0); +} + +#[tokio::test] +async fn public_absence_failure_stops_before_resources_and_completion() { + let h = Harness::new(); + h.tombstones.fail.store(true, Ordering::SeqCst); + + let outcome = h.execute().await; + + assert_eq!( + outcome, + ForcedContentLockDeletionOutcome::TransientDependencyFailure + ); + assert_eq!( + h.operations(), + vec!["public".to_owned(), "release".to_owned()] + ); + assert_eq!(h.deletions.complete_calls.load(Ordering::SeqCst), 0); +} + +#[tokio::test] +async fn busy_guard_has_no_healthy_dependency_evidence() { + let h = Harness::new(); + h.actions.busy.store(true, Ordering::SeqCst); + + let execution = h.execute_with_evidence().await; + + assert_eq!( + execution.outcome, + ForcedContentLockDeletionOutcome::Deferred + ); + assert_eq!(execution.evidence, DeletionDependencyEvidence::none()); +} + +#[tokio::test] +async fn force_receipt_failure_reports_pubky_healthy_and_repository_mutation_unavailable() { + let h = Harness::new(); + h.deletions.fail_complete_once.store(true, Ordering::SeqCst); + + let execution = h.execute_with_evidence().await; + + assert_eq!( + execution.outcome, + ForcedContentLockDeletionOutcome::TransientDependencyFailure + ); + assert_eq!( + execution + .evidence + .status(DeletionDependencySource::PubkyForcePublic), + Some(DeletionDependencyStatus::Healthy) + ); + assert_eq!( + execution + .evidence + .status(DeletionDependencySource::RepositoryForceReceipt), + Some(DeletionDependencyStatus::Unavailable) + ); +} + +#[tokio::test] +async fn force_pubky_failure_does_not_degrade_repository_mutation() { + let h = Harness::new(); + h.tombstones.fail.store(true, Ordering::SeqCst); + + let execution = h.execute_with_evidence().await; + + assert_eq!( + execution + .evidence + .status(DeletionDependencySource::PubkyForcePublic), + Some(DeletionDependencyStatus::Unavailable) + ); + assert_ne!( + execution + .evidence + .status(DeletionDependencySource::RepositoryForceReceipt), + Some(DeletionDependencyStatus::Unavailable) + ); +} + +#[tokio::test] +async fn stale_force_receipt_and_guard_release_do_not_infer_force_receipt_health() { + let h = Harness::new(); + h.deletions.claim_live.store(false, Ordering::SeqCst); + + let execution = h.execute_with_evidence().await; + + assert_eq!( + execution.outcome, + ForcedContentLockDeletionOutcome::ClaimLost + ); + assert_eq!( + execution + .evidence + .status(DeletionDependencySource::RepositoryForceReceipt), + None + ); + assert_eq!( + execution + .evidence + .status(DeletionDependencySource::RepositoryActionLockRelease), + Some(DeletionDependencyStatus::Healthy) + ); +} + +struct Harness { + operations: Arc>>, + deletions: FakeDeletions, + actions: FakeActions, + tombstones: FakeTombstones, + resources: FakeResources, + clock: FixedClock, +} + +impl Harness { + fn new() -> Self { + let operations = Arc::new(Mutex::new(Vec::new())); + Self { + deletions: FakeDeletions::new(Arc::clone(&operations)), + actions: FakeActions::new(Arc::clone(&operations)), + tombstones: FakeTombstones::new(Arc::clone(&operations)), + resources: FakeResources::new(Arc::clone(&operations)), + clock: FixedClock, + operations, + } + } + + async fn execute(&self) -> ForcedContentLockDeletionOutcome { + self.use_case().execute(force_claim(), "worker").await + } + + async fn execute_with_evidence(&self) -> ForcedContentLockDeletionExecution { + self.use_case() + .execute_with_evidence(force_claim(), "worker") + .await + } + + fn use_case(&self) -> ExecuteForcedContentLockDeletionUseCase<'_> { + ExecuteForcedContentLockDeletionUseCase::new(ExecuteForcedContentLockDeletionDependencies { + action_ownership: &self.actions, + tombstones: &self.tombstones, + guarded_resources: &self.resources, + deletions: &self.deletions, + clock: &self.clock, + }) + } + + fn operations(&self) -> Vec { + self.operations.lock().unwrap().clone() + } +} + +struct FixedClock; +impl crate::application::ports::Clock for FixedClock { + fn now(&self) -> OffsetDateTime { + NOW + } +} + +struct FakeGuard(Arc>>); +#[async_trait] +impl ContentLockDeletionActionGuard for FakeGuard { + async fn release(self: Box) -> Result<(), ApplicationError> { + self.0.lock().unwrap().push("release".to_owned()); + Ok(()) + } +} + +struct FakeActions { + busy: AtomicBool, + claim_live: AtomicBool, + operations: Arc>>, +} +impl FakeActions { + fn new(operations: Arc>>) -> Self { + Self { + busy: AtomicBool::new(false), + claim_live: AtomicBool::new(true), + operations, + } + } +} +#[async_trait] +impl ContentLockDeletionActionOwnership for FakeActions { + async fn try_acquire( + &self, + _: ContentLockDeletionActionClaim<'_>, + ) -> Result { + if self.busy.load(Ordering::SeqCst) { + Ok(ContentLockDeletionActionAcquireResult::Busy) + } else if !self.claim_live.load(Ordering::SeqCst) { + Ok(ContentLockDeletionActionAcquireResult::ClaimLost) + } else { + Ok(ContentLockDeletionActionAcquireResult::Acquired(Box::new( + FakeGuard(Arc::clone(&self.operations)), + ))) + } + } +} + +struct FakeTombstones { + operations: Arc>>, + fail: AtomicBool, +} +impl FakeTombstones { + fn new(operations: Arc>>) -> Self { + Self { + operations, + fail: AtomicBool::new(false), + } + } +} +#[async_trait] +impl ContentLockTombstoneRepository for FakeTombstones { + async fn withdraw_content_lock( + &self, + _: CreatorPubky, + _: ContentLockPath, + _: &ContentLock, + _: &ContentLockDeletionTombstone, + ) -> Result { + unreachable!() + } + async fn read_tombstone( + &self, + _: &CreatorPubky, + _: &ContentLockPath, + _: &ContentLockDeletionTombstone, + ) -> Result { + unreachable!() + } + + async fn force_delete_content_lock_and_verify_absent( + &self, + _: &CreatorPubky, + _: &ContentLockPath, + ) -> Result<(), ApplicationError> { + self.operations.lock().unwrap().push("public".to_owned()); + if self.fail.load(Ordering::SeqCst) { + Err(ApplicationError::Storage { + message: "public dependency".to_owned(), + }) + } else { + Ok(()) + } + } +} + +struct FakeResources { + operations: Arc>>, + fail_paths: Mutex>, +} +impl FakeResources { + fn new(operations: Arc>>) -> Self { + Self { + operations, + fail_paths: Mutex::new(HashSet::new()), + } + } +} +#[async_trait] +impl GuardedResourceRepository for FakeResources { + async fn upsert_guarded_resource( + &self, + _: GuardedResourceRecord, + ) -> Result<(), ApplicationError> { + unreachable!() + } + async fn get_guarded_resource( + &self, + _: &CreatorPubky, + _: &str, + _: &GuardedResourceHash, + ) -> Result, ApplicationError> { + unreachable!() + } + async fn get_current_guarded_resource( + &self, + _: &CreatorPubky, + _: &str, + ) -> Result, ApplicationError> { + unreachable!() + } + async fn delete_guarded_resource( + &self, + _: &CreatorPubky, + path: &str, + ) -> Result { + self.operations + .lock() + .unwrap() + .push(format!("resource:{path}")); + if self.fail_paths.lock().unwrap().contains(path) { + Err(ApplicationError::Storage { + message: "resource dependency".to_owned(), + }) + } else { + Ok(false) + } + } +} + +struct FakeDeletions { + operations: Arc>>, + claim_live: AtomicBool, + complete_calls: AtomicUsize, + fail_complete_once: AtomicBool, +} +use std::sync::atomic::AtomicUsize; +impl FakeDeletions { + fn new(operations: Arc>>) -> Self { + Self { + operations, + claim_live: AtomicBool::new(true), + complete_calls: AtomicUsize::new(0), + fail_complete_once: AtomicBool::new(false), + } + } +} +#[async_trait] +impl ContentLockDeletionRepository for FakeDeletions { + async fn begin_publication( + &self, + _: &CreatorPubky, + _: &LockId, + _: Uuid, + ) -> Result<(), ApplicationError> { + unreachable!() + } + async fn finish_publication( + &self, + _: &CreatorPubky, + _: &LockId, + _: Uuid, + ) -> Result { + unreachable!() + } + async fn abandon_publication( + &self, + _: &CreatorPubky, + _: &LockId, + _: Uuid, + ) -> Result { + unreachable!() + } + async fn publication_in_progress( + &self, + _: &CreatorPubky, + _: &LockId, + ) -> Result { + unreachable!() + } + async fn insert_job(&self, _: ContentLockDeletionJob) -> Result<(), ApplicationError> { + unreachable!() + } + async fn get_job( + &self, + _: &CreatorPubky, + _: &LockId, + ) -> Result, ApplicationError> { + unreachable!() + } + async fn claim_next( + &self, + _: &str, + _: time::Duration, + ) -> Result, ApplicationError> { + unreachable!() + } + async fn schedule_retry( + &self, + _: Uuid, + _: &str, + _: Uuid, + _: time::Duration, + ) -> Result, ApplicationError> { + unreachable!() + } + async fn defer( + &self, + _: Uuid, + _: &str, + _: Uuid, + _: time::Duration, + ) -> Result, ApplicationError> { + unreachable!() + } + async fn advance_phase( + &self, + _: Uuid, + _: &str, + _: Uuid, + _: ContentLockDeletionPhase, + ) -> Result { + unreachable!() + } + async fn finish( + &self, + _: Uuid, + _: &str, + _: Uuid, + _: Option, + ) -> Result, ApplicationError> { + unreachable!() + } + async fn resume_failed_job( + &self, + _: &CreatorPubky, + _: &LockId, + _: OffsetDateTime, + ) -> Result, ApplicationError> { + unreachable!() + } + async fn prepare_force_deletion( + &self, + _: &CreatorPubky, + _: &LockId, + ) -> Result { + unreachable!() + } + async fn complete_force_deletion( + &self, + _: Uuid, + _: &str, + _: Uuid, + ) -> Result { + self.complete_calls.fetch_add(1, Ordering::SeqCst); + self.operations.lock().unwrap().push("complete".to_owned()); + if self.fail_complete_once.swap(false, Ordering::SeqCst) { + return Err(ApplicationError::Storage { + message: "force receipt persistence unavailable after deletion".to_owned(), + }); + } + Ok(self.claim_live.load(Ordering::SeqCst)) + } + async fn has_force_receipt( + &self, + _: &CreatorPubky, + _: &LockId, + ) -> Result { + unreachable!() + } +} + +fn force_claim() -> ClaimedContentLockDeletionJob { + claimed_deletion_job(NOW, ContentLockDeletionPhase::Withdraw, true) +} diff --git a/locks-service/src/application/use_cases/issue_access_credential.rs b/locks-service/src/application/use_cases/issue_access_credential.rs index e921bf8..32d5feb 100644 --- a/locks-service/src/application/use_cases/issue_access_credential.rs +++ b/locks-service/src/application/use_cases/issue_access_credential.rs @@ -75,12 +75,13 @@ impl<'a> IssueAccessCredentialUseCase<'a> { .credential_generator .generate_access_credential() .await?; + let final_issue_now = self.clock.now(); if let Some(final_credential) = self .credential_store .issue_or_replay_final_credential( &request.creator, &request.bundle_id, - now, + final_issue_now, candidate, ) .await? diff --git a/locks-service/src/application/use_cases/materialize_final_credentials.rs b/locks-service/src/application/use_cases/materialize_final_credentials.rs new file mode 100644 index 0000000..e5ba8c2 --- /dev/null +++ b/locks-service/src/application/use_cases/materialize_final_credentials.rs @@ -0,0 +1,326 @@ +use time::OffsetDateTime; +use uuid::Uuid; + +use crate::application::{ + errors::ApplicationError, + ports::{ + AccessCredentialGenerator, AccessCredentialStore, Clock, FinalCredentialWorkerIssueRequest, + }, +}; + +/// Exact worker claim and bounded batch used to materialize final deletion credentials. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct MaterializeFinalCredentialsRequest<'a> { + pub deletion_job_id: Uuid, + pub worker_id: &'a str, + pub claim_token: Uuid, + pub now: OffsetDateTime, + pub batch_limit: usize, +} + +/// Secret-free result of one bounded materialization pass. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct MaterializeFinalCredentialsOutcome { + pub materialized_count: usize, +} + +/// Materializes final credentials selected under a live deletion-worker claim. +pub struct MaterializeFinalCredentialsUseCase<'a> { + store: &'a dyn AccessCredentialStore, + generator: &'a dyn AccessCredentialGenerator, + clock: &'a dyn Clock, +} + +impl<'a> MaterializeFinalCredentialsUseCase<'a> { + pub fn new( + store: &'a dyn AccessCredentialStore, + generator: &'a dyn AccessCredentialGenerator, + clock: &'a dyn Clock, + ) -> Self { + Self { + store, + generator, + clock, + } + } + + pub async fn execute( + &self, + request: MaterializeFinalCredentialsRequest<'_>, + ) -> Result { + let pending = self + .store + .final_credentials_to_materialize( + request.deletion_job_id, + request.worker_id, + request.claim_token, + request.batch_limit, + ) + .await?; + let mut materialized_count = 0; + for item in pending { + let candidate = self.generator.generate_access_credential().await?; + let fresh_now = self.clock.now(); + if self + .store + .issue_or_replay_final_credential_for_worker(FinalCredentialWorkerIssueRequest { + deletion_job_id: request.deletion_job_id, + worker_id: request.worker_id, + claim_token: request.claim_token, + creator: &item.creator, + bundle_id: &item.bundle_id, + now: fresh_now, + candidate, + }) + .await? + .is_some() + { + materialized_count += 1; + } + } + Ok(MaterializeFinalCredentialsOutcome { materialized_count }) + } +} + +#[cfg(test)] +mod tests { + use std::{collections::HashMap, str::FromStr, sync::Mutex}; + + use async_trait::async_trait; + use locks_core::ids::{BundleId, CreatorPubky, LockId}; + use time::macros::datetime; + use uuid::Uuid; + + use super::{MaterializeFinalCredentialsRequest, MaterializeFinalCredentialsUseCase}; + use crate::application::{ + errors::ApplicationError, + models::{ + AccessCredential, AccessCredentialLookupKey, AccessCredentialRecord, + FinalCredentialMaterialization, IssuedDeletionCredential, + }, + ports::{ + AccessCredentialGenerator, AccessCredentialStore, Clock, + FinalCredentialWorkerIssueRequest, + }, + }; + + const NOW: time::OffsetDateTime = datetime!(2026-08-17 12:00:00 UTC); + + #[tokio::test] + async fn materializes_every_eligible_snapshot_in_deterministic_store_order() { + let creator = creator(); + let first_bundle = BundleId::from_str("000G40R40M30E209185GR38E1V").unwrap(); + let second_bundle = BundleId::from_str("000G40R40M30E209185GR38E1W").unwrap(); + let store = FakeStore::with_pending(vec![ + FinalCredentialMaterialization { + creator: creator.clone(), + bundle_id: first_bundle.clone(), + }, + FinalCredentialMaterialization { + creator: creator.clone(), + bundle_id: second_bundle.clone(), + }, + ]); + let generator = SequenceGenerator::new(["first-secret", "second-secret"]); + let request = request(10); + + let clock = FixedClock(NOW + time::Duration::seconds(1)); + let outcome = MaterializeFinalCredentialsUseCase::new(&store, &generator, &clock) + .execute(request) + .await + .unwrap(); + + assert_eq!(outcome.materialized_count, 2); + assert_eq!( + store.calls(), + vec![ + (first_bundle, "first-secret".to_owned()), + (second_bundle, "second-secret".to_owned()), + ] + ); + assert_eq!(generator.generated_count(), 2); + assert!(!format!("{outcome:?}").contains("secret")); + } + + #[tokio::test] + async fn retry_after_materialization_does_not_generate_or_persist_a_second_bearer() { + let creator = creator(); + let bundle_id = BundleId::from_str("000G40R40M30E209185GR38E1W").unwrap(); + let store = FakeStore::with_pending(vec![FinalCredentialMaterialization { + creator, + bundle_id: bundle_id.clone(), + }]); + let generator = SequenceGenerator::new(["persisted-secret", "must-not-be-generated"]); + + let clock = FixedClock(NOW + time::Duration::seconds(1)); + let first = MaterializeFinalCredentialsUseCase::new(&store, &generator, &clock) + .execute(request(1)) + .await + .unwrap(); + let retry = MaterializeFinalCredentialsUseCase::new(&store, &generator, &clock) + .execute(request(1)) + .await + .unwrap(); + + assert_eq!(first.materialized_count, 1); + assert_eq!(retry.materialized_count, 0); + assert_eq!(generator.generated_count(), 1); + assert_eq!( + store.calls(), + vec![(bundle_id, "persisted-secret".to_owned())] + ); + } + + fn request(batch_limit: usize) -> MaterializeFinalCredentialsRequest<'static> { + MaterializeFinalCredentialsRequest { + deletion_job_id: Uuid::from_u128(7), + worker_id: "worker-final", + claim_token: Uuid::from_u128(8), + now: NOW, + batch_limit, + } + } + + fn creator() -> CreatorPubky { + CreatorPubky::from_str("pubkytkrq8zmwb8a3m9k15csu3q17qmfgqnp9dskbrg9uq1rydpyxp7qy").unwrap() + } + + #[derive(Debug, Clone, Copy)] + struct FixedClock(time::OffsetDateTime); + + impl Clock for FixedClock { + fn now(&self) -> time::OffsetDateTime { + self.0 + } + } + + #[derive(Debug)] + struct SequenceGenerator { + values: Mutex>, + generated: Mutex, + } + + impl SequenceGenerator { + fn new(values: [&str; N]) -> Self { + Self { + values: Mutex::new( + values + .into_iter() + .rev() + .map(AccessCredential::new) + .collect(), + ), + generated: Mutex::new(0), + } + } + + fn generated_count(&self) -> usize { + *self.generated.lock().unwrap() + } + } + + #[async_trait] + impl AccessCredentialGenerator for SequenceGenerator { + async fn generate_access_credential(&self) -> Result { + *self.generated.lock().unwrap() += 1; + self.values.lock().unwrap().pop().ok_or_else(|| { + ApplicationError::CredentialGeneration { + message: "test generator exhausted".to_owned(), + } + }) + } + } + + #[derive(Debug)] + struct FakeStore { + pending: Mutex>, + winners: Mutex>, + calls: Mutex>, + } + + impl FakeStore { + fn with_pending(pending: Vec) -> Self { + Self { + pending: Mutex::new(pending), + winners: Mutex::new(HashMap::new()), + calls: Mutex::new(Vec::new()), + } + } + + fn calls(&self) -> Vec<(BundleId, String)> { + self.calls.lock().unwrap().clone() + } + } + + #[async_trait] + impl AccessCredentialStore for FakeStore { + async fn insert_access_credential( + &self, + _lock_id: &LockId, + _lookup_key: AccessCredentialLookupKey, + _record: AccessCredentialRecord, + ) -> Result<(), ApplicationError> { + unreachable!() + } + + async fn get_access_credential( + &self, + _lookup_key: &AccessCredentialLookupKey, + ) -> Result, ApplicationError> { + unreachable!() + } + + async fn delete_access_credential( + &self, + _lookup_key: &AccessCredentialLookupKey, + ) -> Result<(), ApplicationError> { + unreachable!() + } + + async fn final_credentials_to_materialize( + &self, + _deletion_job_id: Uuid, + _worker_id: &str, + _claim_token: Uuid, + limit: usize, + ) -> Result, ApplicationError> { + let mut pending = self.pending.lock().unwrap(); + let take = pending.len().min(limit); + Ok(pending.drain(..take).collect()) + } + + async fn issue_or_replay_final_credential_for_worker( + &self, + request: FinalCredentialWorkerIssueRequest<'_>, + ) -> Result, ApplicationError> { + let FinalCredentialWorkerIssueRequest { + deletion_job_id, + worker_id, + claim_token, + bundle_id, + now, + candidate, + .. + } = request; + assert_eq!(deletion_job_id, Uuid::from_u128(7)); + assert_eq!(worker_id, "worker-final"); + assert_eq!(claim_token, Uuid::from_u128(8)); + assert_eq!(now, NOW + time::Duration::seconds(1)); + self.calls + .lock() + .unwrap() + .push((bundle_id.clone(), candidate.as_str().to_owned())); + let winner = self + .winners + .lock() + .unwrap() + .entry(bundle_id.clone()) + .or_insert(candidate) + .clone(); + Ok(Some(IssuedDeletionCredential { + credential: winner, + expires_at: NOW + time::Duration::minutes(30), + })) + } + } +} diff --git a/locks-service/src/application/use_cases/mod.rs b/locks-service/src/application/use_cases/mod.rs index ae472c7..3322c53 100644 --- a/locks-service/src/application/use_cases/mod.rs +++ b/locks-service/src/application/use_cases/mod.rs @@ -4,12 +4,18 @@ pub mod create_content_lock; #[cfg(test)] mod credential_flow_tests; pub mod delete_guarded_resource; +#[cfg(test)] +mod deletion_test_fixtures; pub mod drain_lock_payments; mod entitlement_check; pub mod exchange_frontend_session_code; +pub mod execute_content_lock_deletion_phase; +pub mod execute_forced_content_lock_deletion; pub mod get_creator_authority_status; pub mod get_verification_task; pub mod issue_access_credential; +pub mod materialize_final_credentials; +pub mod no_paykit_deletion_drain; pub mod proxy_read_guarded_resource; pub mod register_guarded_resource; pub mod require_creator_authority_for_pubky_io; diff --git a/locks-service/src/application/use_cases/no_paykit_deletion_drain.rs b/locks-service/src/application/use_cases/no_paykit_deletion_drain.rs new file mode 100644 index 0000000..3bfa020 --- /dev/null +++ b/locks-service/src/application/use_cases/no_paykit_deletion_drain.rs @@ -0,0 +1,171 @@ +use async_trait::async_trait; + +use crate::application::{ + errors::ApplicationError, + models::{ + AdvanceContentLockDeletionPhaseResult, ClaimedContentLockDeletionJob, + ContentLockDeletionPhase, + }, + ports::{Clock, ContentLockDeletionRepository}, +}; + +use super::execute_content_lock_deletion_phase::{ + ContentLockPaymentDrainExecutor, DeletionDependencyEvidence, DeletionDependencySource, + DeletionExecutionErrorClass, DeletionPhaseExecution, DeletionPhaseExecutionOutcome, + classify_deletion_execution_error, +}; + +#[async_trait] +pub trait NoPaykitDeletionDrainExecutor: Send + Sync { + async fn execute_claimed( + &self, + claim: ClaimedContentLockDeletionJob, + worker_id: &str, + ) -> Result; +} + +pub struct NoPaykitDeletionDrainUseCase<'a> { + deletions: &'a dyn ContentLockDeletionRepository, + clock: &'a dyn Clock, +} + +impl<'a> NoPaykitDeletionDrainUseCase<'a> { + pub fn new(deletions: &'a dyn ContentLockDeletionRepository, clock: &'a dyn Clock) -> Self { + Self { deletions, clock } + } + + pub async fn execute_claimed( + &self, + claim: ClaimedContentLockDeletionJob, + worker_id: &str, + ) -> Result { + let next_phase = match claim.job.phase { + ContentLockDeletionPhase::StartPaymentDrain => ContentLockDeletionPhase::DrainPayments, + ContentLockDeletionPhase::DrainPayments => { + ContentLockDeletionPhase::DrainExistingCredentials + } + _ => { + return Err(ApplicationError::InvalidContentLockDeletionState { + message: "non-Paykit drain requires a payment drain phase".to_owned(), + }); + } + }; + let _now = self.clock.now(); + if !self + .deletions + .expire_unresolved_non_paykit_tasks(claim.job.job_id, worker_id, claim.claim_token) + .await? + { + return Ok(false); + } + Ok(matches!( + self.deletions + .advance_phase(claim.job.job_id, worker_id, claim.claim_token, next_phase) + .await?, + AdvanceContentLockDeletionPhaseResult::Advanced(_) + )) + } +} + +#[async_trait] +impl NoPaykitDeletionDrainExecutor for NoPaykitDeletionDrainUseCase<'_> { + async fn execute_claimed( + &self, + claim: ClaimedContentLockDeletionJob, + worker_id: &str, + ) -> Result { + NoPaykitDeletionDrainUseCase::execute_claimed(self, claim, worker_id).await + } +} + +#[async_trait] +impl ContentLockPaymentDrainExecutor for NoPaykitDeletionDrainUseCase<'_> { + async fn execute_claimed( + &self, + claim: ClaimedContentLockDeletionJob, + worker_id: &str, + ) -> DeletionPhaseExecution { + let next_phase = match claim.job.phase { + ContentLockDeletionPhase::StartPaymentDrain => ContentLockDeletionPhase::DrainPayments, + ContentLockDeletionPhase::DrainPayments => { + ContentLockDeletionPhase::DrainExistingCredentials + } + _ => return DeletionPhaseExecution::new(DeletionPhaseExecutionOutcome::FatalFailure), + }; + let _now = self.clock.now(); + let mut evidence = DeletionDependencyEvidence::none(); + match self + .deletions + .expire_unresolved_non_paykit_tasks(claim.job.job_id, worker_id, claim.claim_token) + .await + { + Ok(false) => { + return DeletionPhaseExecution::new(DeletionPhaseExecutionOutcome::ClaimLost); + } + Ok(true) => { + evidence = evidence.merge(DeletionDependencyEvidence::healthy( + DeletionDependencySource::PaymentDrainRepository, + )); + } + Err(error) => { + return observed_error( + &error, + DeletionDependencySource::PaymentDrainRepository, + evidence, + ); + } + } + + match self + .deletions + .advance_phase(claim.job.job_id, worker_id, claim.claim_token, next_phase) + .await + { + Ok(AdvanceContentLockDeletionPhaseResult::Advanced(_)) => { + observed_phase(DeletionPhaseExecutionOutcome::Progressed, evidence) + } + Ok(AdvanceContentLockDeletionPhaseResult::ClaimLost) => { + DeletionPhaseExecution::new(DeletionPhaseExecutionOutcome::ClaimLost) + .with_evidence(evidence) + } + Ok(AdvanceContentLockDeletionPhaseResult::ObligationsPending) => { + observed_phase(DeletionPhaseExecutionOutcome::Deferred, evidence) + } + Ok(AdvanceContentLockDeletionPhaseResult::TerminalFailure(_)) => { + observed_phase(DeletionPhaseExecutionOutcome::TerminalFailed, evidence) + } + Err(error) => observed_error( + &error, + DeletionDependencySource::RepositoryPhaseMutation, + evidence, + ), + } + } +} + +fn observed_phase( + outcome: DeletionPhaseExecutionOutcome, + evidence: DeletionDependencyEvidence, +) -> DeletionPhaseExecution { + DeletionPhaseExecution::new(outcome) + .with_evidence(evidence) + .with_evidence(DeletionDependencyEvidence::healthy( + DeletionDependencySource::RepositoryPhaseMutation, + )) +} + +fn observed_error( + error: &ApplicationError, + source: DeletionDependencySource, + evidence: DeletionDependencyEvidence, +) -> DeletionPhaseExecution { + let outcome = match classify_deletion_execution_error(error) { + DeletionExecutionErrorClass::TransientDependency => { + DeletionPhaseExecutionOutcome::TransientDependencyFailure + } + DeletionExecutionErrorClass::Fatal => DeletionPhaseExecutionOutcome::FatalFailure, + }; + DeletionPhaseExecution::new(outcome) + .with_evidence(evidence) + .with_evidence(DeletionDependencyEvidence::unavailable(source)) +} diff --git a/locks-service/src/application/use_cases/proxy_read_guarded_resource.rs b/locks-service/src/application/use_cases/proxy_read_guarded_resource.rs index 90eaa4b..e6546a0 100644 --- a/locks-service/src/application/use_cases/proxy_read_guarded_resource.rs +++ b/locks-service/src/application/use_cases/proxy_read_guarded_resource.rs @@ -72,16 +72,10 @@ impl<'a> ProxyReadGuardedResourceUseCase<'a> { &self, request: ProxyReadGuardedResourceRequest, ) -> Result { - let now = self.clock.now(); let lookup_key = AccessCredentialLookupKey::derive(&request.credential); if let Some(authorization) = self .credential_store - .prepare_deletion_read( - &lookup_key, - &request.path, - now, - now + time::Duration::seconds(30), - ) + .prepare_deletion_read(&lookup_key, &request.path, time::Duration::seconds(30)) .await? { let claim_token = authorization.claim_token; @@ -191,12 +185,7 @@ impl<'a> ProxyReadGuardedResourceUseCase<'a> { }; if !self .credential_store - .consume_deletion_read( - &claim.lookup_key, - &claim.path, - claim.claim_token, - self.clock.now(), - ) + .consume_deletion_read(&claim.lookup_key, &claim.path, claim.claim_token) .await? { return Err(ApplicationError::InvalidAccessCredential); @@ -533,8 +522,7 @@ mod tests { &self, _lookup_key: &AccessCredentialLookupKey, path: &str, - _now: OffsetDateTime, - _claim_expires_at: OffsetDateTime, + _claim_duration: time::Duration, ) -> Result, ApplicationError> { Ok((path == "/priv/locks.app/content/resource.txt").then(|| { DeletionReadAuthorization { @@ -569,7 +557,6 @@ mod tests { _lookup_key: &AccessCredentialLookupKey, _path: &str, claim_token: Uuid, - _now: OffsetDateTime, ) -> Result { assert_eq!(claim_token, self.claim_token); self.consumes.fetch_add(1, Ordering::SeqCst); diff --git a/locks-service/src/infrastructure/memory/access_credentials.rs b/locks-service/src/infrastructure/memory/access_credentials.rs index fc0c3e2..cfdb897 100644 --- a/locks-service/src/infrastructure/memory/access_credentials.rs +++ b/locks-service/src/infrastructure/memory/access_credentials.rs @@ -19,10 +19,11 @@ use crate::application::{ models::{ AccessCredential, AccessCredentialLookupKey, AccessCredentialRecord, ContentLockDeletionJob, ContentLockDeletionPhase, ContentLockDeletionState, - DeletionReadAuthorization, EncryptedFinalCredential, FinalCredentialContext, + DeletionReadAuthorization, EncryptedFinalCredential, FinalAccessWindows, + FinalCredentialContext, FinalCredentialMaterialization, InitializeFinalAccessWindowsResult, IssuedDeletionCredential, VerificationTaskStatus, }, - ports::{AccessCredentialStore, VerificationTaskRepository}, + ports::{AccessCredentialStore, FinalCredentialWorkerIssueRequest, VerificationTaskRepository}, }; use crate::infrastructure::{ final_credentials::FinalCredentialCipher, @@ -31,6 +32,13 @@ use crate::infrastructure::{ type JobKey = (CreatorPubky, LockId); +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum AccessPhaseAdvanceStatus { + Ready, + ObligationsPending, + FinalCredentialIssuanceMissed, +} + #[derive(Debug, Clone, Copy, PartialEq, Eq)] enum DrainCredentialKind { Ordinary, @@ -55,6 +63,7 @@ struct DeletionAccessState { claimed_by: Option, claim_token: Option, claim_expires_at: Option, + issuance_started_at: Option, issuance_deadline: Option, read_deadline: Option, payment_aggregate: Option, @@ -178,6 +187,14 @@ impl InMemoryAccessCredentialStore { } } + fn authoritative_winner_time(&self) -> OffsetDateTime { + self.verification_task_deletion_fence + .as_ref() + .map_or_else(OffsetDateTime::now_utc, |fence| { + fence.authoritative_cutoff() + }) + } + pub(crate) async fn register_deletion( &self, job: &ContentLockDeletionJob, @@ -242,6 +259,7 @@ impl InMemoryAccessCredentialStore { claimed_by: None, claim_token: None, claim_expires_at: None, + issuance_started_at: None, issuance_deadline: None, read_deadline: None, payment_aggregate: None, @@ -335,7 +353,7 @@ impl InMemoryAccessCredentialStore { && deletion.claim_token == Some(claim_token) && deletion .claim_expires_at - .is_some_and(|claim_expires_at| claim_expires_at >= now); + .is_some_and(|claim_expires_at| claim_expires_at > now); if !owns_live_claim { return Ok(false); } @@ -379,16 +397,82 @@ impl InMemoryAccessCredentialStore { Ok(true) } + pub(crate) async fn expire_unresolved_non_paykit_tasks( + &self, + deletion_job_id: Uuid, + worker_id: &str, + claim_token: Uuid, + now: OffsetDateTime, + ) -> Result { + let task_ids = { + let state = self.state.read().await; + let Some(deletion) = state.deletions.get(&deletion_job_id) else { + return Ok(false); + }; + if !owns_live_no_paykit_claim(deletion, worker_id, claim_token, now) { + return Ok(false); + } + if deletion + .bundle_snapshots + .values() + .any(|snapshot| snapshot.paykit_admission_required) + { + return Err(invalid_deletion_state( + "non-Paykit deletion drain cannot process a Paykit snapshot", + )); + } + deletion + .bundle_snapshots + .values() + .filter(|snapshot| snapshot.resolved_status.is_none()) + .map(|snapshot| snapshot.task_id) + .collect::>() + }; + + if let Some(tasks) = &self.verification_tasks { + let mut updates = Vec::with_capacity(task_ids.len()); + for task_id in &task_ids { + let Some(task) = tasks.get_verification_task(task_id).await? else { + return Err(invalid_deletion_state( + "frozen non-Paykit verification task is missing", + )); + }; + if matches!( + task.status, + VerificationTaskStatus::Pending | VerificationTaskStatus::InProgress + ) { + updates.push(task.transition_to(VerificationTaskStatus::Expired, now, None)?); + } + } + tasks.update_verification_tasks_atomically(updates).await?; + } + + let mut state = self.state.write().await; + let Some(deletion) = state.deletions.get_mut(&deletion_job_id) else { + return Ok(false); + }; + if !owns_live_no_paykit_claim(deletion, worker_id, claim_token, now) { + return Ok(false); + } + for snapshot in deletion.bundle_snapshots.values_mut() { + if snapshot.resolved_status.is_none() { + snapshot.resolved_status = Some(VerificationTaskStatus::Expired); + snapshot.resolved_at = Some(now); + } + } + Ok(true) + } + pub(crate) async fn check_phase_advance( &self, deletion_job_id: Uuid, current_phase: ContentLockDeletionPhase, next_phase: ContentLockDeletionPhase, now: OffsetDateTime, - ) -> Result<(), ApplicationError> { + ) -> Result { let state = self.state.read().await; let Some(deletion) = state.deletions.get(&deletion_job_id) else { - return Ok(()); + return Ok(AccessPhaseAdvanceStatus::Ready); }; if current_phase == ContentLockDeletionPhase::DrainPayments && next_phase == ContentLockDeletionPhase::DrainExistingCredentials @@ -415,9 +499,7 @@ impl InMemoryAccessCredentialStore { && stored.record.expires_at > now }) { - return Err(invalid_deletion_state( - "existing credentials must reach their original expiry before final issuance", - )); + return Ok(AccessPhaseAdvanceStatus::ObligationsPending); } if current_phase == ContentLockDeletionPhase::IssueFinalCredentials && next_phase == ContentLockDeletionPhase::DrainFinalReads @@ -425,19 +507,24 @@ impl InMemoryAccessCredentialStore { snapshot.permits_final_credential() && !snapshot.final_credential_issued }) { - return Err(invalid_deletion_state( - "final credential issuance must complete before final-read draining", - )); + return Ok( + if deletion + .issuance_deadline + .is_some_and(|deadline| now >= deadline) + { + AccessPhaseAdvanceStatus::FinalCredentialIssuanceMissed + } else { + AccessPhaseAdvanceStatus::ObligationsPending + }, + ); } if current_phase == ContentLockDeletionPhase::DrainFinalReads && next_phase == ContentLockDeletionPhase::DeleteContent && has_live_access_obligation(&state, deletion_job_id, now) { - return Err(invalid_deletion_state( - "credential expiry and final-read obligations must drain before destructive deletion", - )); + return Ok(AccessPhaseAdvanceStatus::ObligationsPending); } - Ok(()) + Ok(AccessPhaseAdvanceStatus::Ready) } pub(crate) async fn check_successful_finish( @@ -495,13 +582,38 @@ fn owns_live_payment_drain_claim( && deletion.claim_token == Some(claim_token) && deletion .claim_expires_at - .is_some_and(|claim_expires_at| claim_expires_at >= now) + .is_some_and(|claim_expires_at| claim_expires_at > now) +} + +fn owns_live_no_paykit_claim( + deletion: &DeletionAccessState, + worker_id: &str, + claim_token: Uuid, + now: OffsetDateTime, +) -> bool { + deletion.state == ContentLockDeletionState::Running + && matches!( + deletion.phase, + ContentLockDeletionPhase::StartPaymentDrain | ContentLockDeletionPhase::DrainPayments + ) + && !deletion.force_requested + && deletion.claimed_by.as_deref() == Some(worker_id) + && deletion.claim_token == Some(claim_token) + && deletion + .claim_expires_at + .is_some_and(|claim_expires_at| claim_expires_at > now) } fn payment_aggregate_completed(deletion: &DeletionAccessState) -> bool { - deletion - .payment_aggregate - .is_some_and(|aggregate| aggregate.completed && aggregate.accepted_count == 0) + deletion.payment_aggregate.map_or_else( + || { + !deletion + .bundle_snapshots + .values() + .any(|snapshot| snapshot.paykit_admission_required) + }, + |aggregate| aggregate.completed && aggregate.accepted_count == 0, + ) } #[async_trait] @@ -568,18 +680,18 @@ impl AccessCredentialStore for InMemoryAccessCredentialStore { deletion_job_id: Uuid, worker_id: &str, claim_token: Uuid, - now: OffsetDateTime, - issuance_deadline: OffsetDateTime, - read_deadline: OffsetDateTime, - ) -> Result { - if issuance_deadline <= now || read_deadline <= issuance_deadline { + issuance_window: Duration, + read_window: Duration, + ) -> Result { + if issuance_window <= Duration::ZERO || read_window <= Duration::ZERO { return Err(ApplicationError::Storage { - message: "invalid final access window ordering".to_owned(), + message: "final access window durations must be positive".to_owned(), }); } let mut state = self.state.write().await; + let now = self.authoritative_winner_time(); let Some(deletion) = state.deletions.get_mut(&deletion_job_id) else { - return Ok(false); + return Ok(InitializeFinalAccessWindowsResult::ClaimLost); }; let owns_live_claim = deletion.state == ContentLockDeletionState::Running && deletion.phase == ContentLockDeletionPhase::IssueFinalCredentials @@ -590,21 +702,109 @@ impl AccessCredentialStore for InMemoryAccessCredentialStore { .claim_expires_at .is_some_and(|claim_expires_at| claim_expires_at > now); if !owns_live_claim { - return Ok(false); + return Ok(InitializeFinalAccessWindowsResult::ClaimLost); } - deletion.issuance_deadline.get_or_insert(issuance_deadline); - deletion.read_deadline.get_or_insert(read_deadline); - Ok(true) + match ( + deletion.issuance_started_at, + deletion.issuance_deadline, + deletion.read_deadline, + ) { + ( + Some(issuance_started_at), + Some(credential_issuance_deadline), + Some(read_deadline), + ) => { + return Ok(InitializeFinalAccessWindowsResult::Initialized( + FinalAccessWindows { + issuance_started_at, + credential_issuance_deadline, + read_deadline, + }, + )); + } + (None, None, None) => {} + _ => { + return Err(ApplicationError::Storage { + message: "incomplete final access windows in memory".to_owned(), + }); + } + } + let credential_issuance_deadline = + now.checked_add(issuance_window) + .ok_or_else(|| ApplicationError::Storage { + message: "final credential issuance deadline overflow".to_owned(), + })?; + let read_deadline = credential_issuance_deadline + .checked_add(read_window) + .ok_or_else(|| ApplicationError::Storage { + message: "final read deadline overflow".to_owned(), + })?; + deletion.issuance_started_at = Some(now); + deletion.issuance_deadline = Some(credential_issuance_deadline); + deletion.read_deadline = Some(read_deadline); + Ok(InitializeFinalAccessWindowsResult::Initialized( + FinalAccessWindows { + issuance_started_at: now, + credential_issuance_deadline, + read_deadline, + }, + )) + } + + async fn final_credentials_to_materialize( + &self, + deletion_job_id: Uuid, + worker_id: &str, + claim_token: Uuid, + limit: usize, + ) -> Result, ApplicationError> { + if limit == 0 { + return Ok(Vec::new()); + } + let state = self.state.read().await; + let now = self.authoritative_winner_time(); + let Some(deletion) = state.deletions.get(&deletion_job_id) else { + return Ok(Vec::new()); + }; + let owns_live_issue_claim = deletion.state == ContentLockDeletionState::Running + && deletion.phase == ContentLockDeletionPhase::IssueFinalCredentials + && !deletion.force_requested + && deletion.claimed_by.as_deref() == Some(worker_id) + && deletion.claim_token == Some(claim_token) + && deletion + .claim_expires_at + .is_some_and(|claim_expires_at| claim_expires_at > now) + && deletion + .issuance_deadline + .is_some_and(|issuance_deadline| issuance_deadline > now); + if !owns_live_issue_claim { + return Ok(Vec::new()); + } + let mut pending: Vec<_> = deletion + .bundle_snapshots + .iter() + .filter(|(_, snapshot)| { + snapshot.permits_final_credential() && !snapshot.final_credential_issued + }) + .map(|(bundle_id, _)| FinalCredentialMaterialization { + creator: deletion.creator.clone(), + bundle_id: bundle_id.clone(), + }) + .collect(); + pending.sort_by(|left, right| left.bundle_id.as_str().cmp(right.bundle_id.as_str())); + pending.truncate(limit); + Ok(pending) } async fn issue_or_replay_final_credential( &self, creator: &CreatorPubky, bundle_id: &BundleId, - now: OffsetDateTime, + _caller_now: OffsetDateTime, candidate: AccessCredential, ) -> Result, ApplicationError> { let mut state = self.state.write().await; + let now = self.authoritative_winner_time(); let Some((deletion_job_id, _)) = state.deletions.iter().find(|(_, deletion)| { deletion.creator == *creator && deletion.bundle_snapshots.contains_key(bundle_id) }) else { @@ -715,6 +915,122 @@ impl AccessCredentialStore for InMemoryAccessCredentialStore { })) } + async fn issue_or_replay_final_credential_for_worker( + &self, + request: FinalCredentialWorkerIssueRequest<'_>, + ) -> Result, ApplicationError> { + let FinalCredentialWorkerIssueRequest { + deletion_job_id, + worker_id, + claim_token, + creator, + bundle_id, + now: _caller_now, + candidate, + } = request; + let mut state = self.state.write().await; + let now = self.authoritative_winner_time(); + let Some(deletion) = state.deletions.get(&deletion_job_id) else { + return Ok(None); + }; + let owns_live_issue_claim = deletion.creator == *creator + && deletion.state == ContentLockDeletionState::Running + && deletion.phase == ContentLockDeletionPhase::IssueFinalCredentials + && !deletion.force_requested + && deletion.claimed_by.as_deref() == Some(worker_id) + && deletion.claim_token == Some(claim_token) + && deletion + .claim_expires_at + .is_some_and(|claim_expires_at| claim_expires_at > now) + && deletion + .issuance_deadline + .is_some_and(|issuance_deadline| issuance_deadline > now) + && deletion + .read_deadline + .is_some_and(|read_deadline| read_deadline > now) + && deletion + .bundle_snapshots + .get(bundle_id) + .copied() + .is_some_and(DeletionBundleSnapshot::permits_final_credential); + if !owns_live_issue_claim { + return Ok(None); + } + + let context = FinalCredentialContext { + deletion_job_id, + creator: creator.clone(), + bundle_id: bundle_id.clone(), + }; + if let Some(existing) = state + .final_credentials + .get(&(deletion_job_id, bundle_id.clone())) + { + let credential = self + .final_credential_cipher + .decrypt(&context, &existing.encrypted_bearer)?; + return Ok(Some(IssuedDeletionCredential { + credential, + expires_at: existing.expires_at, + })); + } + + let deletion = state + .deletions + .get(&deletion_job_id) + .expect("worker-fenced deletion was validated under the write lock"); + let expires_at = deletion + .read_deadline + .expect("worker-fenced final issuance requires an initialized read deadline"); + let frozen_content_lock = deletion.frozen_content_lock.clone(); + let lock_id = deletion.lock_id.clone(); + let encrypted_bearer = self.final_credential_cipher.encrypt(&context, &candidate)?; + let lookup_key = AccessCredentialLookupKey::derive(&candidate); + if state.records.contains_key(&lookup_key) { + return Err(ApplicationError::DuplicateRecord { + record: "access_credential", + }); + } + let mut reads = HashMap::new(); + if let Some(resource) = frozen_content_lock.primary_resource { + reads.insert(resource.path, FinalReadState::default()); + } + for path in frozen_content_lock.secondary_resources.keys() { + reads.insert(path.clone(), FinalReadState::default()); + } + state.records.insert( + lookup_key.clone(), + StoredCredential { + record: AccessCredentialRecord { + creator: creator.clone(), + bundle_id: bundle_id.clone(), + expires_at, + }, + lock_id, + deletion: Some((deletion_job_id, DrainCredentialKind::Final)), + }, + ); + state.final_credentials.insert( + (deletion_job_id, bundle_id.clone()), + FinalCredentialRecord { + lookup_key, + encrypted_bearer, + expires_at, + reads, + }, + ); + state + .deletions + .get_mut(&deletion_job_id) + .and_then(|deletion| deletion.bundle_snapshots.get_mut(bundle_id)) + .expect("issued final credential must retain its immutable snapshot") + .final_credential_issued = true; + Ok(Some(IssuedDeletionCredential { + credential: candidate, + expires_at, + })) + } + async fn final_credential_available( &self, creator: &CreatorPubky, @@ -766,10 +1082,10 @@ impl AccessCredentialStore for InMemoryAccessCredentialStore { &self, lookup_key: &AccessCredentialLookupKey, path: &str, - now: OffsetDateTime, - claim_expires_at: OffsetDateTime, + claim_duration: Duration, ) -> Result, ApplicationError> { let mut state = self.state.write().await; + let now = self.authoritative_winner_time(); let Some(stored) = state.records.get(lookup_key).cloned() else { return Ok(None); }; @@ -838,7 +1154,11 @@ impl AccessCredentialStore for InMemoryAccessCredentialStore { { return Ok(None); } - let bounded_expiry = claim_expires_at + let bounded_expiry = now + .checked_add(claim_duration) + .ok_or_else(|| ApplicationError::Storage { + message: "final read claim expiry overflow".to_owned(), + })? .min(now + Duration::seconds(30)) .min(final_credential.expires_at) .min(read_deadline); @@ -898,9 +1218,9 @@ impl AccessCredentialStore for InMemoryAccessCredentialStore { lookup_key: &AccessCredentialLookupKey, path: &str, claim_token: Uuid, - now: OffsetDateTime, ) -> Result { let mut state = self.state.write().await; + let now = self.authoritative_winner_time(); let Some((deletion_job_id, bundle_id)) = final_credential_identity(&state, lookup_key) else { return Ok(false); @@ -984,13 +1304,99 @@ fn revoke_job_read_claims(state: &mut StoreState, job_id: Uuid) { #[cfg(test)] mod tests { - use std::str::FromStr; + use std::{collections::BTreeMap, str::FromStr, sync::Mutex as StdMutex}; + use serde_json::json; use time::macros::datetime; + use tokio::sync::RwLock; - use locks_core::ids::{BundleId, CreatorPubky, LockId}; + use locks_core::{ + ids::{BundleId, CreatorPubky, GuardedResourceHash, LockId, PubkyLockResource, TaskId}, + lock_policy::{ + AccessPolicy, CONTENT_LOCK_VERSION, ContentLock, GuardedResource, LockLogic, + LockServerConfig, VerifierType, + }, + verification::{Proof, SUBMITTED_PROOF_BUNDLE_VERSION, SubmittedProofBundle}, + }; use super::*; + use crate::application::models::VerificationTaskRecord; + + #[derive(Debug, Default)] + struct FailingVerificationTaskRepository { + records: RwLock>, + fail_update_for: RwLock>, + } + + #[async_trait] + impl VerificationTaskRepository for FailingVerificationTaskRepository { + async fn insert_verification_task( + &self, + task: VerificationTaskRecord, + ) -> Result<(), ApplicationError> { + self.records.write().await.insert(task.task_id, task); + Ok(()) + } + + async fn update_verification_task( + &self, + task: VerificationTaskRecord, + ) -> Result<(), ApplicationError> { + if *self.fail_update_for.read().await == Some(task.task_id) { + return Err(ApplicationError::Storage { + message: "injected verification task update failure".to_owned(), + }); + } + let mut records = self.records.write().await; + if !records.contains_key(&task.task_id) { + return Err(ApplicationError::MissingRecord { + record: "verification_task", + }); + } + records.insert(task.task_id, task); + Ok(()) + } + + async fn update_verification_tasks_atomically( + &self, + tasks: Vec, + ) -> Result<(), ApplicationError> { + let mut records = self.records.write().await; + let fail_update_for = *self.fail_update_for.read().await; + if tasks + .iter() + .any(|task| !records.contains_key(&task.task_id)) + { + return Err(ApplicationError::MissingRecord { + record: "verification_task", + }); + } + if tasks + .iter() + .any(|task| fail_update_for == Some(task.task_id)) + { + return Err(ApplicationError::Storage { + message: "injected verification task update failure".to_owned(), + }); + } + for task in tasks { + records.insert(task.task_id, task); + } + Ok(()) + } + + async fn get_verification_task( + &self, + task_id: &TaskId, + ) -> Result, ApplicationError> { + Ok(self.records.read().await.get(task_id).cloned()) + } + + async fn delete_verification_task(&self, task_id: &TaskId) -> Result<(), ApplicationError> { + self.records.write().await.remove(task_id); + Ok(()) + } + } #[tokio::test] async fn insert_rejects_duplicate_read_miss_is_none_delete_is_ensure_absent() { @@ -1030,6 +1436,711 @@ mod tests { ); } + #[tokio::test] + async fn non_paykit_expiry_missing_later_task_mutates_neither_tasks_nor_snapshots() { + let tasks = Arc::new(FailingVerificationTaskRepository::default()); + let store = non_paykit_expiry_store(tasks.clone()).await; + let task_ids = unresolved_task_ids(&store).await; + tasks + .insert_verification_task(non_paykit_task(task_ids[0])) + .await + .unwrap(); + + assert!(expire_non_paykit(&store).await.is_err()); + assert_eq!( + tasks + .get_verification_task(&task_ids[0]) + .await + .unwrap() + .unwrap() + .status, + VerificationTaskStatus::Pending + ); + assert_snapshots_unresolved(&store).await; + } + + #[tokio::test] + async fn non_paykit_expiry_later_update_failure_mutates_neither_tasks_nor_snapshots() { + let tasks = Arc::new(FailingVerificationTaskRepository::default()); + let store = non_paykit_expiry_store(tasks.clone()).await; + let task_ids = unresolved_task_ids(&store).await; + for task_id in task_ids.iter().copied() { + tasks + .insert_verification_task(non_paykit_task(task_id)) + .await + .unwrap(); + } + *tasks.fail_update_for.write().await = Some(task_ids[1]); + + assert!(expire_non_paykit(&store).await.is_err()); + for task_id in task_ids { + assert_eq!( + tasks + .get_verification_task(&task_id) + .await + .unwrap() + .unwrap() + .status, + VerificationTaskStatus::Pending + ); + } + assert_snapshots_unresolved(&store).await; + } + + async fn expire_non_paykit( + store: &InMemoryAccessCredentialStore, + ) -> Result { + store + .expire_unresolved_non_paykit_tasks( + Uuid::from_u128(7), + "worker-drain", + Uuid::from_u128(8), + datetime!(2026-08-17 12:00:00 UTC), + ) + .await + } + + async fn non_paykit_expiry_store( + tasks: Arc, + ) -> InMemoryAccessCredentialStore { + let store = InMemoryAccessCredentialStore::build(Some(tasks), None); + let snapshots = [ + ( + "000G40R40M30E209185GR38E1W", + "018fc6ec-2f3d-4f7e-8b7d-6f5c4b3a2d10", + ), + ( + "000G40R40M30E209185GR38E1V", + "018fc6ec-2f3d-4f7e-8b7d-6f5c4b3a2d11", + ), + ] + .into_iter() + .map(|(bundle_id, task_id)| { + ( + BundleId::from_str(bundle_id).unwrap(), + DeletionBundleSnapshot { + task_id: TaskId::from_str(task_id).unwrap(), + paykit_admission_required: false, + had_active_credential_at_cutoff: false, + status_at_cutoff: VerificationTaskStatus::Pending, + resolved_status: None, + resolved_at: None, + final_credential_eligible_at: None, + final_credential_issued: false, + }, + ) + }) + .collect(); + store.state.write().await.deletions.insert( + Uuid::from_u128(7), + DeletionAccessState { + creator: record().creator, + lock_id: LockId::from_str("000G40R40M30E209185GR38E1W8124GK2GAHC5RR34D1P70X3RFG") + .unwrap(), + frozen_content_lock: test_content_lock(), + state: ContentLockDeletionState::Running, + phase: ContentLockDeletionPhase::StartPaymentDrain, + force_requested: false, + claimed_by: Some("worker-drain".to_owned()), + claim_token: Some(Uuid::from_u128(8)), + claim_expires_at: Some(datetime!(2026-08-17 12:05:00 UTC)), + issuance_started_at: None, + issuance_deadline: None, + read_deadline: None, + payment_aggregate: None, + bundle_snapshots: snapshots, + }, + ); + store + } + + async fn unresolved_task_ids(store: &InMemoryAccessCredentialStore) -> Vec { + store + .state + .read() + .await + .deletions + .get(&Uuid::from_u128(7)) + .unwrap() + .bundle_snapshots + .values() + .map(|snapshot| snapshot.task_id) + .collect() + } + + async fn assert_snapshots_unresolved(store: &InMemoryAccessCredentialStore) { + assert!( + store + .state + .read() + .await + .deletions + .get(&Uuid::from_u128(7)) + .unwrap() + .bundle_snapshots + .values() + .all( + |snapshot| snapshot.resolved_status.is_none() && snapshot.resolved_at.is_none() + ) + ); + } + + fn non_paykit_task(task_id: TaskId) -> VerificationTaskRecord { + VerificationTaskRecord { + task_id, + creator: record().creator, + submitted_proof_bundle: SubmittedProofBundle { + version: SUBMITTED_PROOF_BUNDLE_VERSION, + bundle_id: BundleId::from_str("000G40R40M30E209185GR38E1W").unwrap(), + pubky_lock_resource: PubkyLockResource::from_str( + "pubkytkrq8zmwb8a3m9k15csu3q17qmfgqnp9dskbrg9uq1rydpyxp7qy/pub/locks.app/000G40R40M30E209185GR38E1W8124GK2GAHC5RR34D1P70X3RFG.json", + ) + .unwrap(), + reader_public_key: None, + proofs: vec![Proof { + criterion_id: "criterion-1".to_owned(), + verifier_type: VerifierType::DevStatic, + payload: json!({}), + }], + }, + status: VerificationTaskStatus::Pending, + submitted_at: datetime!(2026-08-17 11:00:00 UTC), + started_at: None, + completed_at: None, + failure_message: None, + } + } + + #[tokio::test] + async fn final_materialization_enumeration_is_ordered_eligible_bounded_and_claim_fenced() { + let store = final_materialization_store().await; + let job_id = Uuid::from_u128(7); + let claim_token = Uuid::from_u128(8); + + let selected = store + .final_credentials_to_materialize(job_id, "worker-final", claim_token, 1) + .await + .unwrap(); + assert_eq!(selected.len(), 1); + assert_eq!(selected[0].bundle_id.as_str(), "000G40R40M30E209185GR38E1R"); + + for (worker, token) in [ + ("wrong-worker", claim_token), + ("worker-final", Uuid::from_u128(9)), + ] { + assert!( + store + .final_credentials_to_materialize(job_id, worker, token, 10) + .await + .unwrap() + .is_empty() + ); + } + + { + let mut state = store.state.write().await; + state.deletions.get_mut(&job_id).unwrap().force_requested = true; + } + assert!( + store + .final_credentials_to_materialize(job_id, "worker-final", claim_token, 10) + .await + .unwrap() + .is_empty() + ); + { + let mut state = store.state.write().await; + let deletion = state.deletions.get_mut(&job_id).unwrap(); + deletion.force_requested = false; + deletion.phase = ContentLockDeletionPhase::DrainFinalReads; + } + assert!( + store + .final_credentials_to_materialize(job_id, "worker-final", claim_token, 10) + .await + .unwrap() + .is_empty() + ); + } + + #[tokio::test] + async fn final_materialization_enumeration_samples_time_after_state_fence() { + let initial = datetime!(2026-08-17 12:00:00 UTC); + let claim_expiry = datetime!(2026-08-17 12:05:00 UTC); + let clock = Arc::new(MutableClock::new(initial)); + let store = Arc::new(final_materialization_store_with_clock(clock.clone()).await); + let state_guard = store.state.write().await; + let waiting_store = Arc::clone(&store); + + let enumeration = tokio::spawn(async move { + waiting_store + .final_credentials_to_materialize( + Uuid::from_u128(7), + "worker-final", + Uuid::from_u128(8), + 10, + ) + .await + }); + tokio::task::yield_now().await; + assert!(!enumeration.is_finished()); + clock.set(claim_expiry); + drop(state_guard); + + assert!(enumeration.await.unwrap().unwrap().is_empty()); + } + + #[tokio::test] + async fn worker_final_issuance_revalidates_exact_live_claim_and_fresh_deadlines() { + let job_id = Uuid::from_u128(7); + let claim_token = Uuid::from_u128(8); + let now = datetime!(2026-08-17 12:00:00 UTC); + let bundle_id = BundleId::from_str("000G40R40M30E209185GR38E1R").unwrap(); + let creator = record().creator; + + for (worker, token, at) in [ + ("wrong-worker", claim_token, now), + ("worker-final", Uuid::from_u128(9), now), + ( + "worker-final", + claim_token, + datetime!(2026-08-17 12:05:00 UTC), + ), + ( + "worker-final", + claim_token, + datetime!(2026-08-17 12:15:00 UTC), + ), + ] { + let store = final_materialization_store_at(at).await; + let candidate = AccessCredential::new(format!("denied-{worker}-{at}")); + let lookup_key = AccessCredentialLookupKey::derive(&candidate); + assert!( + store + .issue_or_replay_final_credential_for_worker( + FinalCredentialWorkerIssueRequest { + deletion_job_id: job_id, + worker_id: worker, + claim_token: token, + creator: &creator, + bundle_id: &bundle_id, + now: at, + candidate, + }, + ) + .await + .unwrap() + .is_none() + ); + assert!( + store + .get_access_credential(&lookup_key) + .await + .unwrap() + .is_none() + ); + } + + let forced = final_materialization_store().await; + forced + .state + .write() + .await + .deletions + .get_mut(&job_id) + .unwrap() + .force_requested = true; + assert!( + forced + .issue_or_replay_final_credential_for_worker(FinalCredentialWorkerIssueRequest { + deletion_job_id: job_id, + worker_id: "worker-final", + claim_token, + creator: &creator, + bundle_id: &bundle_id, + now, + candidate: AccessCredential::new("force-loser"), + },) + .await + .unwrap() + .is_none() + ); + + let store = final_materialization_store().await; + let winner = AccessCredential::new("worker-winner"); + let issued = store + .issue_or_replay_final_credential_for_worker(FinalCredentialWorkerIssueRequest { + deletion_job_id: job_id, + worker_id: "worker-final", + claim_token, + creator: &creator, + bundle_id: &bundle_id, + now, + candidate: winner.clone(), + }) + .await + .unwrap() + .unwrap(); + let replay = store + .issue_or_replay_final_credential_for_worker(FinalCredentialWorkerIssueRequest { + deletion_job_id: job_id, + worker_id: "worker-final", + claim_token, + creator: &creator, + bundle_id: &bundle_id, + now, + candidate: AccessCredential::new("worker-loser"), + }) + .await + .unwrap() + .unwrap(); + assert_eq!(issued.credential, winner); + assert_eq!(replay, issued); + assert_eq!(store.state.read().await.final_credentials.len(), 1); + } + + #[tokio::test] + async fn final_winner_paths_sample_time_after_waiting_for_state_fence() { + let initial = datetime!(2026-08-17 12:00:00 UTC); + let clock = Arc::new(MutableClock::new(initial)); + let job_id = Uuid::from_u128(7); + let claim_token = Uuid::from_u128(8); + let bundle_id = BundleId::from_str("000G40R40M30E209185GR38E1R").unwrap(); + let creator = record().creator; + + let public_store = Arc::new(final_materialization_store_with_clock(clock.clone()).await); + let public_guard = public_store.state.write().await; + let issuing_store = Arc::clone(&public_store); + let issuing_creator = creator.clone(); + let issuing_bundle = bundle_id.clone(); + let public_issue = tokio::spawn(async move { + issuing_store + .issue_or_replay_final_credential( + &issuing_creator, + &issuing_bundle, + initial, + AccessCredential::new("stale-public-candidate"), + ) + .await + }); + tokio::task::yield_now().await; + assert!(!public_issue.is_finished()); + clock.set(datetime!(2026-08-17 12:15:00 UTC)); + drop(public_guard); + assert!(public_issue.await.unwrap().unwrap().is_none()); + assert!(public_store.state.read().await.final_credentials.is_empty()); + + clock.set(initial); + let worker_store = Arc::new(final_materialization_store_with_clock(clock.clone()).await); + let worker_guard = worker_store.state.write().await; + let issuing_store = Arc::clone(&worker_store); + let worker_issue = tokio::spawn(async move { + issuing_store + .issue_or_replay_final_credential_for_worker(FinalCredentialWorkerIssueRequest { + deletion_job_id: job_id, + worker_id: "worker-final", + claim_token, + creator: &creator, + bundle_id: &bundle_id, + now: initial, + candidate: AccessCredential::new("stale-worker-candidate"), + }) + .await + }); + tokio::task::yield_now().await; + assert!(!worker_issue.is_finished()); + clock.set(datetime!(2026-08-17 12:05:00 UTC)); + drop(worker_guard); + assert!(worker_issue.await.unwrap().unwrap().is_none()); + assert!(worker_store.state.read().await.final_credentials.is_empty()); + } + + #[tokio::test] + async fn final_access_window_initialization_samples_time_after_waiting_for_state_fence() { + let initial = datetime!(2026-08-17 12:00:00 UTC); + let claim_expiry = datetime!(2026-08-17 12:05:00 UTC); + let clock = Arc::new(MutableClock::new(initial)); + let store = Arc::new(final_materialization_store_with_clock(clock.clone()).await); + { + let mut state = store.state.write().await; + let deletion = state.deletions.get_mut(&Uuid::from_u128(7)).unwrap(); + deletion.issuance_deadline = None; + deletion.read_deadline = None; + } + + let state_guard = store.state.write().await; + let initializing_store = Arc::clone(&store); + let initialization = tokio::spawn(async move { + initializing_store + .initialize_final_access_windows( + Uuid::from_u128(7), + "worker-final", + Uuid::from_u128(8), + Duration::minutes(15), + Duration::minutes(15), + ) + .await + }); + tokio::task::yield_now().await; + assert!(!initialization.is_finished()); + clock.set(claim_expiry); + drop(state_guard); + + assert_eq!( + initialization.await.unwrap().unwrap(), + InitializeFinalAccessWindowsResult::ClaimLost + ); + let state = store.state.read().await; + let deletion = state.deletions.get(&Uuid::from_u128(7)).unwrap(); + assert!(deletion.issuance_deadline.is_none()); + assert!(deletion.read_deadline.is_none()); + } + + #[tokio::test] + async fn final_read_prepare_samples_time_after_state_fence_and_rejects_deadline_equality() { + let initial = datetime!(2026-08-17 12:00:00 UTC); + let deadline = datetime!(2026-08-17 12:30:00 UTC); + let clock = Arc::new(MutableClock::new(initial)); + let store = Arc::new(final_materialization_store_with_clock(clock.clone()).await); + let credential = AccessCredential::new("blocked-final-read"); + let lookup = AccessCredentialLookupKey::derive(&credential); + let creator = record().creator; + let bundle = BundleId::from_str("000G40R40M30E209185GR38E1R").unwrap(); + store + .issue_or_replay_final_credential(&creator, &bundle, initial, credential) + .await + .unwrap() + .unwrap(); + + let state_guard = store.state.write().await; + let waiting_store = Arc::clone(&store); + let waiting_lookup = lookup.clone(); + let prepare = tokio::spawn(async move { + waiting_store + .prepare_deletion_read( + &waiting_lookup, + "/priv/locks.app/content/post.json", + Duration::seconds(30), + ) + .await + }); + tokio::task::yield_now().await; + assert!(!prepare.is_finished()); + clock.set(deadline); + drop(state_guard); + + assert!(prepare.await.unwrap().unwrap().is_none()); + let state = store.state.read().await; + let read = &state + .final_credentials + .get(&(Uuid::from_u128(7), bundle)) + .unwrap() + .reads["/priv/locks.app/content/post.json"]; + assert!(read.claim_token.is_none()); + assert!(read.consumed_at.is_none()); + } + + #[tokio::test] + async fn final_read_consume_samples_time_after_state_fence_and_rejects_claim_expiry_equality() { + let initial = datetime!(2026-08-17 12:00:00 UTC); + let claim_expiry = initial + Duration::seconds(30); + let clock = Arc::new(MutableClock::new(initial)); + let store = Arc::new(final_materialization_store_with_clock(clock.clone()).await); + let credential = AccessCredential::new("blocked-final-consume"); + let lookup = AccessCredentialLookupKey::derive(&credential); + let creator = record().creator; + let bundle = BundleId::from_str("000G40R40M30E209185GR38E1R").unwrap(); + store + .issue_or_replay_final_credential(&creator, &bundle, initial, credential) + .await + .unwrap() + .unwrap(); + let claim_token = store + .prepare_deletion_read( + &lookup, + "/priv/locks.app/content/post.json", + Duration::seconds(30), + ) + .await + .unwrap() + .unwrap() + .claim_token + .unwrap(); + + let state_guard = store.state.write().await; + let waiting_store = Arc::clone(&store); + let waiting_lookup = lookup.clone(); + let consume = tokio::spawn(async move { + waiting_store + .consume_deletion_read( + &waiting_lookup, + "/priv/locks.app/content/post.json", + claim_token, + ) + .await + }); + tokio::task::yield_now().await; + assert!(!consume.is_finished()); + clock.set(claim_expiry); + drop(state_guard); + + assert!(!consume.await.unwrap().unwrap()); + let state = store.state.read().await; + let read = &state + .final_credentials + .get(&(Uuid::from_u128(7), bundle)) + .unwrap() + .reads["/priv/locks.app/content/post.json"]; + assert!(read.consumed_at.is_none()); + } + + #[tokio::test] + async fn final_materialization_enumeration_does_not_create_eligibility() { + let store = final_materialization_store().await; + let job_id = Uuid::from_u128(7); + let ineligible = BundleId::from_str("000G40R40M30E209185GR38E1X").unwrap(); + { + let mut state = store.state.write().await; + let snapshot = state + .deletions + .get_mut(&job_id) + .unwrap() + .bundle_snapshots + .get_mut(&ineligible) + .unwrap(); + snapshot.resolved_status = Some(VerificationTaskStatus::Completed); + assert!(snapshot.final_credential_eligible_at.is_none()); + } + + let selected = store + .final_credentials_to_materialize(job_id, "worker-final", Uuid::from_u128(8), 10) + .await + .unwrap(); + assert!(!selected.iter().any(|item| item.bundle_id == ineligible)); + assert!( + store + .state + .read() + .await + .deletions + .get(&job_id) + .unwrap() + .bundle_snapshots + .get(&ineligible) + .unwrap() + .final_credential_eligible_at + .is_none() + ); + } + + struct MutableClock(StdMutex); + + impl MutableClock { + fn new(now: OffsetDateTime) -> Self { + Self(StdMutex::new(now)) + } + + fn set(&self, now: OffsetDateTime) { + *self.0.lock().unwrap() = now; + } + } + + impl crate::application::ports::Clock for MutableClock { + fn now(&self) -> OffsetDateTime { + *self.0.lock().unwrap() + } + } + + async fn final_materialization_store() -> InMemoryAccessCredentialStore { + final_materialization_store_at(datetime!(2026-08-17 12:00:00 UTC)).await + } + + async fn final_materialization_store_at(now: OffsetDateTime) -> InMemoryAccessCredentialStore { + final_materialization_store_with_clock(Arc::new(MutableClock::new(now))).await + } + + async fn final_materialization_store_with_clock( + clock: Arc, + ) -> InMemoryAccessCredentialStore { + let fence = Arc::new(InMemoryVerificationTaskDeletionFence::with_clock(clock)); + let store = InMemoryAccessCredentialStore::build(None, Some(fence)); + let creator = + CreatorPubky::from_str("pubkytkrq8zmwb8a3m9k15csu3q17qmfgqnp9dskbrg9uq1rydpyxp7qy") + .unwrap(); + let eligible_at = datetime!(2026-08-17 11:00:00 UTC); + let snapshots = [ + ("000G40R40M30E209185GR38E1W", true), + ("000G40R40M30E209185GR38E1V", true), + ("000G40R40M30E209185GR38E1X", false), + ] + .into_iter() + .enumerate() + .map(|(index, (bundle, eligible))| { + ( + BundleId::from_str(bundle).unwrap(), + DeletionBundleSnapshot { + task_id: TaskId::from_str(&format!( + "550e8400-e29b-41d4-a716-44665544000{index}" + )) + .unwrap(), + paykit_admission_required: true, + had_active_credential_at_cutoff: false, + status_at_cutoff: VerificationTaskStatus::Completed, + resolved_status: Some(VerificationTaskStatus::Completed), + resolved_at: Some(eligible_at), + final_credential_eligible_at: eligible.then_some(eligible_at), + final_credential_issued: false, + }, + ) + }) + .collect(); + store.state.write().await.deletions.insert( + Uuid::from_u128(7), + DeletionAccessState { + creator, + lock_id: LockId::from_str("000G40R40M30E209185GR38E1W8124GK2GAHC5RR34D1P70X3RFG") + .unwrap(), + frozen_content_lock: test_content_lock(), + state: ContentLockDeletionState::Running, + phase: ContentLockDeletionPhase::IssueFinalCredentials, + force_requested: false, + claimed_by: Some("worker-final".to_owned()), + claim_token: Some(Uuid::from_u128(8)), + claim_expires_at: Some(datetime!(2026-08-17 12:05:00 UTC)), + issuance_started_at: Some(datetime!(2026-08-17 12:00:00 UTC)), + issuance_deadline: Some(datetime!(2026-08-17 12:15:00 UTC)), + read_deadline: Some(datetime!(2026-08-17 12:30:00 UTC)), + payment_aggregate: None, + bundle_snapshots: snapshots, + }, + ); + store + } + + fn test_content_lock() -> ContentLock { + ContentLock { + version: CONTENT_LOCK_VERSION, + creator: record().creator, + primary_resource: Some( + GuardedResource::new( + "/priv/locks.app/content/post.json".to_owned(), + GuardedResourceHash::from_bytes([7; 32]), + "application/json".to_owned(), + 42, + ) + .unwrap(), + ), + secondary_resources: BTreeMap::new(), + criteria: vec![], + lock_logic: LockLogic::All { criteria: vec![] }, + access_policy: AccessPolicy { + requested_credential_ttl_seconds: 900, + }, + lock_server: LockServerConfig { override_: None }, + created_at: datetime!(2026-08-17 11:00:00 UTC), + } + } + fn record() -> AccessCredentialRecord { AccessCredentialRecord { creator: CreatorPubky::from_str( diff --git a/locks-service/src/infrastructure/memory/content_lock_deletion_action_ownership.rs b/locks-service/src/infrastructure/memory/content_lock_deletion_action_ownership.rs new file mode 100644 index 0000000..a0fd6fa --- /dev/null +++ b/locks-service/src/infrastructure/memory/content_lock_deletion_action_ownership.rs @@ -0,0 +1,101 @@ +use std::{ + collections::HashSet, + sync::{Arc, Mutex}, +}; + +use async_trait::async_trait; +use uuid::Uuid; + +use crate::{ + application::{ + errors::ApplicationError, + ports::{ + ContentLockDeletionActionAcquireResult, ContentLockDeletionActionClaim, + ContentLockDeletionActionGuard, ContentLockDeletionActionOwnership, + }, + }, + infrastructure::memory::content_lock_deletions::InMemoryContentLockDeletionRepository, +}; + +/// Process-local parity adapter for per-job external action ownership. +#[derive(Debug, Clone)] +pub struct InMemoryContentLockDeletionActionOwnership { + deletions: Arc, + owned_jobs: Arc>>, +} + +impl InMemoryContentLockDeletionActionOwnership { + pub fn new(deletions: Arc) -> Self { + Self { + deletions, + owned_jobs: Arc::new(Mutex::new(HashSet::new())), + } + } +} + +#[async_trait] +impl ContentLockDeletionActionOwnership for InMemoryContentLockDeletionActionOwnership { + async fn try_acquire( + &self, + claim: ContentLockDeletionActionClaim<'_>, + ) -> Result { + { + let mut owned_jobs = self + .owned_jobs + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + if !owned_jobs.insert(claim.job_id) { + return Ok(ContentLockDeletionActionAcquireResult::Busy); + } + } + + if !self.deletions.action_claim_is_live(claim).await { + self.owned_jobs + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .remove(&claim.job_id); + return Ok(ContentLockDeletionActionAcquireResult::ClaimLost); + } + + Ok(ContentLockDeletionActionAcquireResult::Acquired(Box::new( + InMemoryContentLockDeletionActionGuard { + job_id: claim.job_id, + owned_jobs: Arc::clone(&self.owned_jobs), + released: false, + }, + ))) + } +} + +struct InMemoryContentLockDeletionActionGuard { + job_id: Uuid, + owned_jobs: Arc>>, + released: bool, +} + +impl InMemoryContentLockDeletionActionGuard { + fn release_inner(&mut self) { + if self.released { + return; + } + self.owned_jobs + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .remove(&self.job_id); + self.released = true; + } +} + +#[async_trait] +impl ContentLockDeletionActionGuard for InMemoryContentLockDeletionActionGuard { + async fn release(mut self: Box) -> Result<(), ApplicationError> { + self.release_inner(); + Ok(()) + } +} + +impl Drop for InMemoryContentLockDeletionActionGuard { + fn drop(&mut self) { + self.release_inner(); + } +} diff --git a/locks-service/src/infrastructure/memory/content_lock_deletions.rs b/locks-service/src/infrastructure/memory/content_lock_deletions.rs index b11508b..69c599e 100644 --- a/locks-service/src/infrastructure/memory/content_lock_deletions.rs +++ b/locks-service/src/infrastructure/memory/content_lock_deletions.rs @@ -1,24 +1,22 @@ -use std::{ - collections::{HashMap, HashSet}, - sync::Arc, -}; +use std::{collections::HashMap, sync::Arc}; use async_trait::async_trait; use locks_core::ids::{CreatorPubky, LockId}; use time::OffsetDateTime; -use tokio::sync::RwLock; +use tokio::sync::{Mutex, RwLock}; use uuid::Uuid; use crate::application::{ errors::ApplicationError, models::{ - ClaimedContentLockDeletionJob, ContentLockDeletionFailureCode, ContentLockDeletionJob, - ContentLockDeletionPhase, ContentLockDeletionState, PrepareForceDeletionResult, + AdvanceContentLockDeletionPhaseResult, ClaimedContentLockDeletionJob, + ContentLockDeletionFailureCode, ContentLockDeletionJob, ContentLockDeletionPhase, + ContentLockDeletionState, PrepareForceDeletionResult, }, - ports::ContentLockDeletionRepository, + ports::{ContentLockDeletionActionClaim, ContentLockDeletionRepository}, }; use crate::infrastructure::memory::{ - access_credentials::InMemoryAccessCredentialStore, + access_credentials::{AccessPhaseAdvanceStatus, InMemoryAccessCredentialStore}, verification_task_deletion_fence::InMemoryVerificationTaskDeletionFence, }; @@ -36,8 +34,9 @@ struct StoredJob { #[derive(Debug)] pub struct InMemoryContentLockDeletionRepository { jobs: RwLock>, - force_receipts: RwLock>, + force_receipts: RwLock>, publication_intents: RwLock>, + claim_transition_gate: Mutex<()>, verification_task_fence: Arc, access_credentials: Arc, } @@ -71,12 +70,31 @@ impl InMemoryContentLockDeletionRepository { ) -> Self { Self { jobs: RwLock::new(HashMap::new()), - force_receipts: RwLock::new(HashSet::new()), + force_receipts: RwLock::new(HashMap::new()), publication_intents: RwLock::new(HashMap::new()), + claim_transition_gate: Mutex::new(()), verification_task_fence, access_credentials, } } + + pub(super) async fn action_claim_is_live( + &self, + claim: ContentLockDeletionActionClaim<'_>, + ) -> bool { + let now = self.verification_task_fence.authoritative_cutoff(); + self.jobs.read().await.values().any(|stored| { + stored.job.job_id == claim.job_id + && stored.job.state == ContentLockDeletionState::Running + && stored.job.phase == claim.expected_phase + && stored.claimed_by.as_deref() == Some(claim.worker_id) + && stored.claim_token == Some(claim.claim_token) + && stored + .claim_expires_at + .is_some_and(|expires_at| expires_at > now) + && (stored.job.force_requested_at.is_some() == claim.force) + }) + } } #[async_trait] @@ -91,7 +109,7 @@ impl ContentLockDeletionRepository for InMemoryContentLockDeletionRepository { let mut intents = self.publication_intents.write().await; let jobs = self.jobs.read().await; let receipts = self.force_receipts.read().await; - if jobs.contains_key(&key) || receipts.contains(&key) { + if jobs.contains_key(&key) || receipts.contains_key(&key) { return Err(ApplicationError::ContentLockDeletionInProgress); } if intents.contains_key(&key) { @@ -158,7 +176,7 @@ impl ContentLockDeletionRepository for InMemoryContentLockDeletionRepository { let intents = self.publication_intents.read().await; let mut jobs = self.jobs.write().await; let receipts = self.force_receipts.read().await; - if intents.contains_key(&key) || receipts.contains(&key) { + if intents.contains_key(&key) || receipts.contains_key(&key) { return Err(ApplicationError::ContentLockDeletionInProgress); } if jobs.contains_key(&key) || jobs.values().any(|stored| stored.job.job_id == job.job_id) { @@ -236,10 +254,12 @@ impl ContentLockDeletionRepository for InMemoryContentLockDeletionRepository { async fn claim_next( &self, worker_id: &str, - now: OffsetDateTime, - claim_expires_at: OffsetDateTime, + claim_ttl: time::Duration, ) -> Result, ApplicationError> { + let _claim_transition = self.claim_transition_gate.lock().await; let mut jobs = self.jobs.write().await; + let now = self.verification_task_fence.authoritative_cutoff(); + let claim_expires_at = now + claim_ttl; let Some(stored) = jobs .values_mut() .filter(|stored| is_claimable(stored, now)) @@ -273,10 +293,12 @@ impl ContentLockDeletionRepository for InMemoryContentLockDeletionRepository { job_id: Uuid, worker_id: &str, claim_token: Uuid, - now: OffsetDateTime, - next_attempt_at: OffsetDateTime, + retry_after: time::Duration, ) -> Result, ApplicationError> { + let _claim_transition = self.claim_transition_gate.lock().await; let mut jobs = self.jobs.write().await; + let now = self.verification_task_fence.authoritative_cutoff(); + let next_attempt_at = now + retry_after; let Some(stored) = jobs .values_mut() .find(|stored| owns_claim(stored, job_id, worker_id, claim_token, now)) @@ -292,29 +314,69 @@ impl ContentLockDeletionRepository for InMemoryContentLockDeletionRepository { Ok(Some(stored.job.clone())) } - async fn advance_phase( + async fn defer( &self, job_id: Uuid, worker_id: &str, claim_token: Uuid, - now: OffsetDateTime, - next_phase: ContentLockDeletionPhase, + defer_for: time::Duration, ) -> Result, ApplicationError> { + let _claim_transition = self.claim_transition_gate.lock().await; let mut jobs = self.jobs.write().await; + let now = self.verification_task_fence.authoritative_cutoff(); + let next_attempt_at = now + defer_for; let Some(stored) = jobs .values_mut() .find(|stored| owns_claim(stored, job_id, worker_id, claim_token, now)) else { return Ok(None); }; + stored.job.attempt_count = stored.job.attempt_count.saturating_sub(1); + stored.job.state = ContentLockDeletionState::Queued; + stored.job.next_attempt_at = Some(next_attempt_at); + clear_claim(stored); + self.access_credentials + .synchronize_job(&stored.job, None, None, None) + .await; + Ok(Some(stored.job.clone())) + } + + async fn advance_phase( + &self, + job_id: Uuid, + worker_id: &str, + claim_token: Uuid, + next_phase: ContentLockDeletionPhase, + ) -> Result { + let _claim_transition = self.claim_transition_gate.lock().await; + let mut jobs = self.jobs.write().await; + let now = self.verification_task_fence.authoritative_cutoff(); + let Some(stored) = jobs + .values_mut() + .find(|stored| owns_claim(stored, job_id, worker_id, claim_token, now)) + else { + return Ok(AdvanceContentLockDeletionPhaseResult::ClaimLost); + }; if !stored.job.phase.permits(next_phase) { return Err(ApplicationError::InvalidContentLockDeletionState { message: "deletion phase must advance to its immediate successor".to_owned(), }); } - self.access_credentials + let access_status = self + .access_credentials .check_phase_advance(job_id, stored.job.phase, next_phase, now) .await?; + match access_status { + AccessPhaseAdvanceStatus::Ready => {} + AccessPhaseAdvanceStatus::ObligationsPending => { + return Ok(AdvanceContentLockDeletionPhaseResult::ObligationsPending); + } + AccessPhaseAdvanceStatus::FinalCredentialIssuanceMissed => { + return Ok(AdvanceContentLockDeletionPhaseResult::TerminalFailure( + ContentLockDeletionFailureCode::StateCorrupt, + )); + } + } stored.job.phase = next_phase; stored.job.state = ContentLockDeletionState::Queued; stored.job.attempt_count = 0; @@ -324,7 +386,22 @@ impl ContentLockDeletionRepository for InMemoryContentLockDeletionRepository { self.access_credentials .synchronize_job(&stored.job, None, None, None) .await; - Ok(Some(stored.job.clone())) + Ok(AdvanceContentLockDeletionPhaseResult::Advanced(Box::new( + stored.job.clone(), + ))) + } + + async fn expire_unresolved_non_paykit_tasks( + &self, + job_id: Uuid, + worker_id: &str, + claim_token: Uuid, + ) -> Result { + let _claim_transition = self.claim_transition_gate.lock().await; + let now = self.verification_task_fence.authoritative_cutoff(); + self.access_credentials + .expire_unresolved_non_paykit_tasks(job_id, worker_id, claim_token, now) + .await } async fn finish( @@ -332,10 +409,11 @@ impl ContentLockDeletionRepository for InMemoryContentLockDeletionRepository { job_id: Uuid, worker_id: &str, claim_token: Uuid, - now: OffsetDateTime, failure_code: Option, ) -> Result, ApplicationError> { + let _claim_transition = self.claim_transition_gate.lock().await; let mut jobs = self.jobs.write().await; + let now = self.verification_task_fence.authoritative_cutoff(); let Some(stored) = jobs .values_mut() .find(|stored| owns_claim(stored, job_id, worker_id, claim_token, now)) @@ -367,9 +445,10 @@ impl ContentLockDeletionRepository for InMemoryContentLockDeletionRepository { lock_id: &LockId, _resumed_at: OffsetDateTime, ) -> Result, ApplicationError> { + let _claim_transition = self.claim_transition_gate.lock().await; let mut jobs = self.jobs.write().await; let receipts = self.force_receipts.read().await; - if receipts.contains(&(creator.clone(), lock_id.clone())) { + if receipts.contains_key(&(creator.clone(), lock_id.clone())) { return Ok(None); } let Some(stored) = jobs.get_mut(&(creator.clone(), lock_id.clone())) else { @@ -392,8 +471,9 @@ impl ContentLockDeletionRepository for InMemoryContentLockDeletionRepository { &self, creator: &CreatorPubky, lock_id: &LockId, - forced_at: OffsetDateTime, ) -> Result { + let _claim_transition = self.claim_transition_gate.lock().await; + let forced_at = self.verification_task_fence.authoritative_cutoff(); let key = (creator.clone(), lock_id.clone()); let intents = self.publication_intents.read().await; if intents.contains_key(&key) { @@ -411,7 +491,7 @@ impl ContentLockDeletionRepository for InMemoryContentLockDeletionRepository { drop(verification_tasks); let mut jobs = self.jobs.write().await; let mut receipts = self.force_receipts.write().await; - if receipts.contains(&key) { + if receipts.contains_key(&key) { return Ok(PrepareForceDeletionResult::Synchronous( jobs.get(&key).map(|stored| stored.job.clone()), )); @@ -435,16 +515,56 @@ impl ContentLockDeletionRepository for InMemoryContentLockDeletionRepository { .block_key_and_disable_job(creator, lock_id, Some(job.job_id)) .await; jobs.remove(&key); - receipts.insert(key); + receipts.insert(key, forced_at); return Ok(PrepareForceDeletionResult::Synchronous(Some(job))); } self.access_credentials .block_key_and_disable_job(creator, lock_id, None) .await; - receipts.insert(key); + receipts.insert(key, forced_at); Ok(PrepareForceDeletionResult::Synchronous(None)) } + async fn complete_force_deletion( + &self, + job_id: Uuid, + worker_id: &str, + claim_token: Uuid, + ) -> Result { + let _claim_transition = self.claim_transition_gate.lock().await; + let now = self.verification_task_fence.authoritative_cutoff(); + let key = self + .jobs + .read() + .await + .iter() + .find_map(|(key, stored)| (stored.job.job_id == job_id).then(|| key.clone())); + let Some(key) = key else { + return Ok(false); + }; + let _admission = self + .verification_task_fence + .acquire_lock_admission(&key.0, &key.1) + .await; + let mut jobs = self.jobs.write().await; + let mut receipts = self.force_receipts.write().await; + let Some(stored) = jobs.get(&key) else { + return Ok(false); + }; + if !owns_claim(stored, job_id, worker_id, claim_token, now) { + return Ok(false); + } + let Some(forced_at) = stored.job.force_requested_at else { + return Ok(false); + }; + self.access_credentials + .block_key_and_disable_job(&key.0, &key.1, Some(job_id)) + .await; + receipts.insert(key.clone(), forced_at); + jobs.remove(&key); + Ok(true) + } + async fn has_force_receipt( &self, creator: &CreatorPubky, @@ -454,7 +574,7 @@ impl ContentLockDeletionRepository for InMemoryContentLockDeletionRepository { .force_receipts .read() .await - .contains(&(creator.clone(), lock_id.clone()))) + .contains_key(&(creator.clone(), lock_id.clone()))) } } @@ -481,7 +601,7 @@ fn is_claimable(stored: &StoredJob, now: OffsetDateTime) -> bool { .is_none_or(|next_attempt_at| next_attempt_at <= now), ContentLockDeletionState::Running => stored .claim_expires_at - .is_some_and(|claim_expires_at| claim_expires_at < now), + .is_some_and(|claim_expires_at| claim_expires_at <= now), ContentLockDeletionState::Completed | ContentLockDeletionState::Failed => false, } } @@ -499,7 +619,7 @@ fn owns_claim( && stored.claim_token == Some(claim_token) && stored .claim_expires_at - .is_some_and(|claim_expires_at| claim_expires_at >= now) + .is_some_and(|claim_expires_at| now < claim_expires_at) } fn clear_claim(stored: &mut StoredJob) { diff --git a/locks-service/src/infrastructure/memory/content_lock_tombstones.rs b/locks-service/src/infrastructure/memory/content_lock_tombstones.rs new file mode 100644 index 0000000..0b61a99 --- /dev/null +++ b/locks-service/src/infrastructure/memory/content_lock_tombstones.rs @@ -0,0 +1,98 @@ +use async_trait::async_trait; +use locks_core::content_lock_deletion::ContentLockDeletionTombstone; +use locks_core::ids::{ContentLockPath, CreatorPubky}; +use locks_core::lock_policy::ContentLock; + +use crate::application::errors::ApplicationError; +use crate::application::ports::content_lock_tombstone::{ + canonical_tombstone_bytes, classify_tombstone_bytes, +}; +use crate::application::ports::{ContentLockTombstoneRepository, TombstoneReadback}; +use crate::infrastructure::memory::public_content_locks::InMemoryPublicContentLockStore; + +/// In-memory exact-byte public tombstone adapter. +#[derive(Debug, Default)] +pub struct InMemoryContentLockTombstoneRepository { + public_store: InMemoryPublicContentLockStore, +} + +impl InMemoryContentLockTombstoneRepository { + pub fn new() -> Self { + Self::default() + } + + /// Creates a raw tombstone adapter over shared canonical public-path storage. + pub fn with_public_store(public_store: InMemoryPublicContentLockStore) -> Self { + Self { public_store } + } +} + +#[async_trait] +impl ContentLockTombstoneRepository for InMemoryContentLockTombstoneRepository { + async fn withdraw_content_lock( + &self, + creator: CreatorPubky, + content_lock_path: ContentLockPath, + frozen_original: &ContentLock, + tombstone: &ContentLockDeletionTombstone, + ) -> Result { + let expected = canonical_tombstone_bytes(tombstone)?; + let original = + frozen_original + .canonical_json_bytes() + .map_err(|error| ApplicationError::Storage { + message: format!("failed to serialize frozen content lock: {error}"), + })?; + match self + .public_store + .get(&creator, &content_lock_path) + .await + .as_deref() + { + Some(actual) if actual == expected => return Ok(TombstoneReadback::Exact), + Some(actual) if actual == original => {} + None => return Ok(TombstoneReadback::Missing), + Some(_) => return Ok(TombstoneReadback::Replaced), + } + self.public_store + .put(creator.clone(), content_lock_path.clone(), expected) + .await; + self.read_tombstone(&creator, &content_lock_path, tombstone) + .await + } + + async fn read_tombstone( + &self, + creator: &CreatorPubky, + content_lock_path: &ContentLockPath, + expected: &ContentLockDeletionTombstone, + ) -> Result { + let expected = canonical_tombstone_bytes(expected)?; + Ok(classify_tombstone_bytes( + self.public_store + .get(creator, content_lock_path) + .await + .as_deref(), + &expected, + )) + } + + async fn force_delete_content_lock_and_verify_absent( + &self, + creator: &CreatorPubky, + content_lock_path: &ContentLockPath, + ) -> Result<(), ApplicationError> { + self.public_store.remove(creator, content_lock_path).await; + if self + .public_store + .get(creator, content_lock_path) + .await + .is_some() + { + return Err(ApplicationError::Storage { + message: "forced public content lock deletion did not reach absence".to_owned(), + }); + } + Ok(()) + } +} diff --git a/locks-service/src/infrastructure/memory/content_locks.rs b/locks-service/src/infrastructure/memory/content_locks.rs index 5db6ced..fb0fb38 100644 --- a/locks-service/src/infrastructure/memory/content_locks.rs +++ b/locks-service/src/infrastructure/memory/content_locks.rs @@ -1,20 +1,16 @@ -use std::collections::HashMap; - use async_trait::async_trait; -use tokio::sync::RwLock; use locks_core::ids::{ContentLockPath, CreatorPubky}; use locks_core::lock_policy::ContentLock; use crate::application::errors::ApplicationError; use crate::application::ports::ContentLockRepository; - -type ContentLockKey = (CreatorPubky, ContentLockPath); +use crate::infrastructure::memory::public_content_locks::InMemoryPublicContentLockStore; /// In-memory content lock repository for the first retrieval/access slice. #[derive(Debug, Default)] pub struct InMemoryContentLockRepository { - records: RwLock>, + public_store: InMemoryPublicContentLockStore, } impl InMemoryContentLockRepository { @@ -22,6 +18,11 @@ impl InMemoryContentLockRepository { pub fn new() -> Self { Self::default() } + + /// Creates a typed adapter over shared canonical public-path storage. + pub fn with_public_store(public_store: InMemoryPublicContentLockStore) -> Self { + Self { public_store } + } } #[async_trait] @@ -32,10 +33,15 @@ impl ContentLockRepository for InMemoryContentLockRepository { content_lock_path: ContentLockPath, content_lock: ContentLock, ) -> Result<(), ApplicationError> { - self.records - .write() - .await - .insert((creator, content_lock_path), content_lock); + let bytes = + content_lock + .canonical_json_bytes() + .map_err(|error| ApplicationError::Storage { + message: format!("failed to serialize in-memory content lock: {error}"), + })?; + self.public_store + .put(creator, content_lock_path, bytes) + .await; Ok(()) } @@ -44,12 +50,15 @@ impl ContentLockRepository for InMemoryContentLockRepository { creator: &CreatorPubky, content_lock_path: &ContentLockPath, ) -> Result, ApplicationError> { - Ok(self - .records - .read() + self.public_store + .get(creator, content_lock_path) .await - .get(&(creator.clone(), content_lock_path.clone())) - .cloned()) + .map(|bytes| { + serde_json::from_slice(&bytes).map_err(|error| ApplicationError::Storage { + message: format!("failed to deserialize in-memory content lock: {error}"), + }) + }) + .transpose() } async fn delete_content_lock( @@ -58,10 +67,9 @@ impl ContentLockRepository for InMemoryContentLockRepository { content_lock_path: &ContentLockPath, ) -> Result { Ok(self - .records - .write() + .public_store + .remove(creator, content_lock_path) .await - .remove(&(creator.clone(), content_lock_path.clone())) .is_some()) } } diff --git a/locks-service/src/infrastructure/memory/mod.rs b/locks-service/src/infrastructure/memory/mod.rs index 1854ea9..3351b28 100644 --- a/locks-service/src/infrastructure/memory/mod.rs +++ b/locks-service/src/infrastructure/memory/mod.rs @@ -1,10 +1,13 @@ pub mod access_credentials; +pub mod content_lock_deletion_action_ownership; pub mod content_lock_deletions; pub mod content_lock_ownership; +pub mod content_lock_tombstones; pub mod content_locks; pub mod entitlements; pub mod guarded_resources; pub mod lock_service_pointers; +pub mod public_content_locks; pub mod verification_task_claims; pub mod verification_task_deletion_fence; pub mod verification_tasks; diff --git a/locks-service/src/infrastructure/memory/public_content_locks.rs b/locks-service/src/infrastructure/memory/public_content_locks.rs new file mode 100644 index 0000000..006c35f --- /dev/null +++ b/locks-service/src/infrastructure/memory/public_content_locks.rs @@ -0,0 +1,45 @@ +use std::{collections::HashMap, sync::Arc}; + +use locks_core::ids::{ContentLockPath, CreatorPubky}; +use tokio::sync::RwLock; + +pub(crate) type PublicContentLockKey = (CreatorPubky, ContentLockPath); + +/// Shared in-memory backing for the one canonical public content-lock path. +#[derive(Debug, Clone, Default)] +pub struct InMemoryPublicContentLockStore { + records: Arc>>>, +} + +impl InMemoryPublicContentLockStore { + pub fn new() -> Self { + Self::default() + } + + pub(crate) async fn put(&self, creator: CreatorPubky, path: ContentLockPath, bytes: Vec) { + self.records.write().await.insert((creator, path), bytes); + } + + pub(crate) async fn get( + &self, + creator: &CreatorPubky, + path: &ContentLockPath, + ) -> Option> { + self.records + .read() + .await + .get(&(creator.clone(), path.clone())) + .cloned() + } + + pub(crate) async fn remove( + &self, + creator: &CreatorPubky, + path: &ContentLockPath, + ) -> Option> { + self.records + .write() + .await + .remove(&(creator.clone(), path.clone())) + } +} diff --git a/locks-service/src/infrastructure/memory/verification_task_claims.rs b/locks-service/src/infrastructure/memory/verification_task_claims.rs index db07493..c6f4745 100644 --- a/locks-service/src/infrastructure/memory/verification_task_claims.rs +++ b/locks-service/src/infrastructure/memory/verification_task_claims.rs @@ -80,6 +80,16 @@ impl InMemoryVerificationTaskClaimer { /// Creates a claimer seeded with already-claimed task records. pub fn with_claimed_tasks( records: Vec<(VerificationTaskRecord, String, time::OffsetDateTime)>, + ) -> Self { + Self::with_claimed_tasks_and_clock( + records, + Arc::new(crate::infrastructure::memory::verification_task_deletion_fence::SystemClock), + ) + } + + fn with_claimed_tasks_and_clock( + records: Vec<(VerificationTaskRecord, String, time::OffsetDateTime)>, + clock: Arc, ) -> Self { let tasks = records .iter() @@ -101,7 +111,9 @@ impl InMemoryVerificationTaskClaimer { .collect(), ), task_repository: None, - deletion_fence: Arc::new(InMemoryVerificationTaskDeletionFence::from_tasks(&tasks)), + deletion_fence: Arc::new( + InMemoryVerificationTaskDeletionFence::from_tasks_with_clock(&tasks, clock), + ), } } } @@ -113,18 +125,16 @@ impl VerificationTaskClaimer for InMemoryVerificationTaskClaimer { task_id: &TaskId, worker_id: &str, claim_token: &uuid::Uuid, - now: time::OffsetDateTime, ) -> Result { let mut fence_records = self.deletion_fence.records.write().await; let records = self.records.read().await; + let now = self.deletion_fence.authoritative_cutoff(); let owned = records.iter().any(|record| { record.task.task_id == *task_id && record.task.status == VerificationTaskStatus::InProgress && record.claimed_by.as_deref() == Some(worker_id) && record.claim_token.as_ref() == Some(claim_token) - && record - .claim_expires_at - .is_some_and(|expires| expires >= now) + && record.claim_expires_at.is_some_and(|expires| now < expires) }); let Some(fence) = fence_records.get_mut(task_id) else { return Ok(false); @@ -139,11 +149,16 @@ impl VerificationTaskClaimer for InMemoryVerificationTaskClaimer { async fn claim_next_verification_task( &self, worker_id: &str, - now: time::OffsetDateTime, - claim_expires_at: time::OffsetDateTime, + claim_ttl: time::Duration, ) -> Result, ApplicationError> { let fence_records = self.deletion_fence.records.read().await; let mut records = self.records.write().await; + let now = self.deletion_fence.authoritative_cutoff(); + let claim_expires_at = + now.checked_add(claim_ttl) + .ok_or_else(|| ApplicationError::Storage { + message: "verification task claim expiry overflow".to_owned(), + })?; let Some(index) = records.iter().position(|record| { record.is_claimable_at(now) && fence_records @@ -183,8 +198,7 @@ impl VerificationTaskClaimer for InMemoryVerificationTaskClaimer { task_id: &TaskId, worker_id: &str, claim_token: &uuid::Uuid, - now: time::OffsetDateTime, - next_attempt_at: time::OffsetDateTime, + retry_after: time::Duration, ) -> Result, ApplicationError> { let fence_records = self.deletion_fence.records.read().await; if fence_records @@ -194,6 +208,12 @@ impl VerificationTaskClaimer for InMemoryVerificationTaskClaimer { return Ok(None); } let mut records = self.records.write().await; + let now = self.deletion_fence.authoritative_cutoff(); + let next_attempt_at = + now.checked_add(retry_after) + .ok_or_else(|| ApplicationError::Storage { + message: "verification task retry time overflow".to_owned(), + })?; let Some(record) = records.iter_mut().find(|record| { record.task.task_id == *task_id && record.task.status == VerificationTaskStatus::InProgress @@ -201,7 +221,7 @@ impl VerificationTaskClaimer for InMemoryVerificationTaskClaimer { && record.claim_token.as_ref() == Some(claim_token) && record .claim_expires_at - .is_some_and(|claim_expires_at| claim_expires_at >= now) + .is_some_and(|claim_expires_at| now < claim_expires_at) }) else { return Ok(None); }; @@ -227,7 +247,6 @@ impl VerificationTaskClaimer for InMemoryVerificationTaskClaimer { task: VerificationTaskRecord, worker_id: &str, claim_token: &uuid::Uuid, - now: time::OffsetDateTime, ) -> Result, ApplicationError> { if !matches!( task.status, @@ -247,6 +266,7 @@ impl VerificationTaskClaimer for InMemoryVerificationTaskClaimer { return Ok(None); } let mut records = self.records.write().await; + let now = self.deletion_fence.authoritative_cutoff(); let Some(record) = records.iter_mut().find(|record| { record.task.task_id == task.task_id && record.task.status == VerificationTaskStatus::InProgress @@ -254,7 +274,7 @@ impl VerificationTaskClaimer for InMemoryVerificationTaskClaimer { && record.claim_token.as_ref() == Some(claim_token) && record .claim_expires_at - .is_some_and(|claim_expires_at| claim_expires_at >= now) + .is_some_and(|claim_expires_at| now < claim_expires_at) }) else { return Ok(None); }; @@ -283,7 +303,7 @@ impl ClaimableVerificationTask { .is_none_or(|next_attempt_at| next_attempt_at <= now), VerificationTaskStatus::InProgress => self .claim_expires_at - .is_some_and(|claim_expires_at| claim_expires_at < now), + .is_some_and(|claim_expires_at| claim_expires_at <= now), VerificationTaskStatus::Completed | VerificationTaskStatus::Failed | VerificationTaskStatus::Expired => false, @@ -293,7 +313,11 @@ impl ClaimableVerificationTask { #[cfg(test)] mod tests { - use std::{collections::BTreeMap, str::FromStr, sync::Arc}; + use std::{ + collections::BTreeMap, + str::FromStr, + sync::{Arc, Mutex}, + }; use serde_json::json; use time::macros::datetime; @@ -314,7 +338,7 @@ mod tests { VerificationTaskStatus, }; use crate::application::ports::{ - ContentLockDeletionRepository, VerificationTaskClaimer, VerificationTaskRepository, + Clock, ContentLockDeletionRepository, VerificationTaskClaimer, VerificationTaskRepository, }; use crate::infrastructure::memory::content_lock_deletions::InMemoryContentLockDeletionRepository; use crate::infrastructure::memory::verification_task_deletion_fence::InMemoryVerificationTaskDeletionFence; @@ -324,13 +348,49 @@ mod tests { const NOW: time::OffsetDateTime = datetime!(2026-05-29 12:10:00 UTC); const CLAIM_EXPIRES_AT: time::OffsetDateTime = datetime!(2026-05-29 12:15:00 UTC); + #[derive(Debug)] + struct MutableClock(Mutex); + + impl MutableClock { + fn new(now: time::OffsetDateTime) -> Self { + Self(Mutex::new(now)) + } + + fn set(&self, now: time::OffsetDateTime) { + *self.0.lock().unwrap() = now; + } + } + + impl Clock for MutableClock { + fn now(&self) -> time::OffsetDateTime { + *self.0.lock().unwrap() + } + } + + fn claimer_with_clock( + records: Vec, + ) -> (InMemoryVerificationTaskClaimer, Arc) { + let clock = Arc::new(MutableClock::new(NOW)); + let fence = Arc::new( + InMemoryVerificationTaskDeletionFence::from_tasks_with_clock(&records, clock.clone()), + ); + ( + InMemoryVerificationTaskClaimer::with_deletion_fence(records, fence), + clock, + ) + } + + fn claimer_at_now(records: Vec) -> InMemoryVerificationTaskClaimer { + claimer_with_clock(records).0 + } + #[tokio::test] async fn no_pending_or_expired_in_progress_task_returns_none() { - let claimer = InMemoryVerificationTaskClaimer::new(vec![]); + let claimer = claimer_at_now(vec![]); assert_eq!( claimer - .claim_next_verification_task("worker-a", NOW, CLAIM_EXPIRES_AT) + .claim_next_verification_task("worker-a", (CLAIM_EXPIRES_AT) - (NOW)) .await .unwrap(), None @@ -352,7 +412,7 @@ mod tests { InMemoryVerificationTaskClaimer::with_deletion_fence(vec![task], Arc::clone(&fence)); let deletions = InMemoryContentLockDeletionRepository::with_verification_task_fence(fence); let claimed = claimer - .claim_next_verification_task("worker-a", NOW, CLAIM_EXPIRES_AT) + .claim_next_verification_task("worker-a", (CLAIM_EXPIRES_AT) - (NOW)) .await .unwrap() .unwrap(); @@ -362,7 +422,6 @@ mod tests { &claimed.task.task_id, "worker-a", &claimed.claim_token, - NOW, ) .await .unwrap() @@ -389,7 +448,7 @@ mod tests { InMemoryVerificationTaskClaimer::with_deletion_fence(vec![task], Arc::clone(&fence)); let deletions = InMemoryContentLockDeletionRepository::with_verification_task_fence(fence); let claimed = claimer - .claim_next_verification_task("worker-a", NOW, CLAIM_EXPIRES_AT) + .claim_next_verification_task("worker-a", (CLAIM_EXPIRES_AT) - (NOW)) .await .unwrap() .unwrap(); @@ -401,7 +460,6 @@ mod tests { &claimed.task.task_id, "worker-a", &claimed.claim_token, - NOW, ) .await .unwrap() @@ -416,7 +474,6 @@ mod tests { completed, "worker-a", &claimed.claim_token, - NOW, ) .await .unwrap(), @@ -448,7 +505,7 @@ mod tests { assert_eq!( deletions - .prepare_force_deletion(&job.creator, &job.lock_id, NOW) + .prepare_force_deletion(&job.creator, &job.lock_id) .await .unwrap(), PrepareForceDeletionResult::PublicationInProgress @@ -472,14 +529,18 @@ mod tests { VerificationTaskStatus::Pending, &job.lock_id, ); - let fence = Arc::new(InMemoryVerificationTaskDeletionFence::from_tasks( - std::slice::from_ref(&task), - )); + let clock = Arc::new(MutableClock::new(NOW)); + let fence = Arc::new( + InMemoryVerificationTaskDeletionFence::from_tasks_with_clock( + std::slice::from_ref(&task), + clock.clone(), + ), + ); let claimer = InMemoryVerificationTaskClaimer::with_deletion_fence(vec![task], Arc::clone(&fence)); let deletions = InMemoryContentLockDeletionRepository::with_verification_task_fence(fence); let first = claimer - .claim_next_verification_task("worker-a", NOW, CLAIM_EXPIRES_AT) + .claim_next_verification_task("worker-a", (CLAIM_EXPIRES_AT) - (NOW)) .await .unwrap() .unwrap(); @@ -489,7 +550,6 @@ mod tests { &first.task.task_id, "worker-a", &first.claim_token, - NOW, ) .await .unwrap() @@ -500,8 +560,7 @@ mod tests { &first.task.task_id, "worker-a", &first.claim_token, - NOW, - retry_at, + (retry_at) - (NOW), ) .await .unwrap() @@ -512,11 +571,11 @@ mod tests { Err(ApplicationError::ContentLockDeletionInProgress) ); + clock.set(retry_at); let second = claimer .claim_next_verification_task( "worker-b", - retry_at, - retry_at + time::Duration::minutes(5), + (retry_at + time::Duration::minutes(5)) - (retry_at), ) .await .unwrap() @@ -527,7 +586,6 @@ mod tests { &second.task.task_id, "worker-b", &second.claim_token, - retry_at, ) .await .unwrap() @@ -541,7 +599,6 @@ mod tests { completed, "worker-b", &second.claim_token, - retry_at, ) .await .unwrap() @@ -556,10 +613,10 @@ mod tests { "018fc6ec-2f3d-4f7e-8b7d-6f5c4b3a2d10", VerificationTaskStatus::Pending, ); - let claimer = InMemoryVerificationTaskClaimer::new(vec![pending.clone()]); + let claimer = claimer_at_now(vec![pending.clone()]); let claimed = claimer - .claim_next_verification_task("worker-a", NOW, CLAIM_EXPIRES_AT) + .claim_next_verification_task("worker-a", (CLAIM_EXPIRES_AT) - (NOW)) .await .unwrap() .expect("pending task is claimed"); @@ -571,7 +628,7 @@ mod tests { assert_eq!(claimed.task.failure_message, None); assert_eq!( claimer - .claim_next_verification_task("worker-b", NOW, CLAIM_EXPIRES_AT) + .claim_next_verification_task("worker-b", (CLAIM_EXPIRES_AT) - (NOW)) .await .unwrap(), None @@ -595,7 +652,7 @@ mod tests { ); claimer - .claim_next_verification_task("worker-a", NOW, CLAIM_EXPIRES_AT) + .claim_next_verification_task("worker-a", (CLAIM_EXPIRES_AT) - (NOW)) .await .unwrap() .unwrap(); @@ -625,7 +682,7 @@ mod tests { assert!( claimer - .claim_next_verification_task("worker-a", NOW, CLAIM_EXPIRES_AT) + .claim_next_verification_task("worker-a", (CLAIM_EXPIRES_AT) - (NOW)) .await .is_err() ); @@ -633,7 +690,7 @@ mod tests { assert!( claimer - .claim_next_verification_task("worker-a", NOW, CLAIM_EXPIRES_AT) + .claim_next_verification_task("worker-a", (CLAIM_EXPIRES_AT) - (NOW)) .await .unwrap() .is_some() @@ -647,18 +704,18 @@ mod tests { VerificationTaskStatus::Pending, ); let task_id = pending.task_id; - let claimer = InMemoryVerificationTaskClaimer::new(vec![pending]); + let (claimer, clock) = claimer_with_clock(vec![pending]); let first = claimer - .claim_next_verification_task("worker-a", NOW, CLAIM_EXPIRES_AT) + .claim_next_verification_task("worker-a", (CLAIM_EXPIRES_AT) - (NOW)) .await .unwrap() .unwrap(); let reclaimed_at = CLAIM_EXPIRES_AT + time::Duration::milliseconds(1); + clock.set(reclaimed_at); let second = claimer .claim_next_verification_task( "worker-a", - reclaimed_at, - reclaimed_at + time::Duration::minutes(5), + (reclaimed_at + time::Duration::minutes(5)) - (reclaimed_at), ) .await .unwrap() @@ -671,8 +728,7 @@ mod tests { &task_id, "worker-a", &first.claim_token, - reclaimed_at, - reclaimed_at + time::Duration::seconds(10), + (reclaimed_at + time::Duration::seconds(10)) - (reclaimed_at), ) .await .unwrap(), @@ -684,8 +740,7 @@ mod tests { &task_id, "worker-a", &second.claim_token, - reclaimed_at, - reclaimed_at + time::Duration::seconds(10), + (reclaimed_at + time::Duration::seconds(10)) - (reclaimed_at), ) .await .unwrap() @@ -699,18 +754,18 @@ mod tests { "018fc6ec-2f3d-4f7e-8b7d-6f5c4b3a2d10", VerificationTaskStatus::Pending, ); - let claimer = InMemoryVerificationTaskClaimer::new(vec![pending]); + let (claimer, clock) = claimer_with_clock(vec![pending]); let first = claimer - .claim_next_verification_task("worker-a", NOW, CLAIM_EXPIRES_AT) + .claim_next_verification_task("worker-a", (CLAIM_EXPIRES_AT) - (NOW)) .await .unwrap() .unwrap(); let reclaimed_at = CLAIM_EXPIRES_AT + time::Duration::milliseconds(1); + clock.set(reclaimed_at); let second = claimer .claim_next_verification_task( "worker-a", - reclaimed_at, - reclaimed_at + time::Duration::minutes(5), + (reclaimed_at + time::Duration::minutes(5)) - (reclaimed_at), ) .await .unwrap() @@ -741,7 +796,6 @@ mod tests { terminal, "worker-a", &first.claim_token, - reclaimed_at, ) .await .unwrap(), @@ -754,7 +808,6 @@ mod tests { completed.clone(), "worker-a", &second.claim_token, - reclaimed_at, ) .await .unwrap(), @@ -769,9 +822,9 @@ mod tests { VerificationTaskStatus::Pending, ); let task_id = pending.task_id; - let claimer = InMemoryVerificationTaskClaimer::new(vec![pending]); + let (claimer, clock) = claimer_with_clock(vec![pending]); let claim = claimer - .claim_next_verification_task("worker-a", NOW, CLAIM_EXPIRES_AT) + .claim_next_verification_task("worker-a", (CLAIM_EXPIRES_AT) - (NOW)) .await .unwrap() .unwrap(); @@ -783,21 +836,7 @@ mod tests { &task_id, "worker-b", &claim.claim_token, - NOW, - next_attempt_at, - ) - .await - .unwrap(), - None - ); - assert_eq!( - claimer - .schedule_verification_task_retry( - &task_id, - "worker-a", - &claim.claim_token, - CLAIM_EXPIRES_AT + time::Duration::nanoseconds(1), - next_attempt_at, + (next_attempt_at) - (NOW), ) .await .unwrap(), @@ -808,8 +847,7 @@ mod tests { &task_id, "worker-a", &claim.claim_token, - NOW, - next_attempt_at, + (next_attempt_at) - (NOW), ) .await .unwrap() @@ -823,19 +861,18 @@ mod tests { claimer .claim_next_verification_task( "worker-b", - NOW + time::Duration::seconds(9), - CLAIM_EXPIRES_AT, + (CLAIM_EXPIRES_AT) - (NOW + time::Duration::seconds(9)), ) .await .unwrap(), None ); + clock.set(next_attempt_at); assert!( claimer .claim_next_verification_task( "worker-b", - next_attempt_at, - CLAIM_EXPIRES_AT + time::Duration::seconds(10), + (CLAIM_EXPIRES_AT + time::Duration::seconds(10)) - (next_attempt_at), ) .await .unwrap() @@ -875,11 +912,11 @@ mod tests { ) .transition_to(VerificationTaskStatus::Expired, NOW, None) .unwrap(); - let claimer = InMemoryVerificationTaskClaimer::new(vec![completed, failed, expired]); + let claimer = claimer_at_now(vec![completed, failed, expired]); assert_eq!( claimer - .claim_next_verification_task("worker-a", NOW, CLAIM_EXPIRES_AT) + .claim_next_verification_task("worker-a", (CLAIM_EXPIRES_AT) - (NOW)) .await .unwrap(), None @@ -898,14 +935,17 @@ mod tests { None, ) .unwrap(); - let claimer = InMemoryVerificationTaskClaimer::with_claimed_tasks(vec![( - in_progress.clone(), - "worker-a".to_owned(), - datetime!(2026-05-29 12:05:00 UTC), - )]); + let claimer = InMemoryVerificationTaskClaimer::with_claimed_tasks_and_clock( + vec![( + in_progress.clone(), + "worker-a".to_owned(), + datetime!(2026-05-29 12:05:00 UTC), + )], + Arc::new(MutableClock::new(NOW)), + ); let reclaimed = claimer - .claim_next_verification_task("worker-b", NOW, CLAIM_EXPIRES_AT) + .claim_next_verification_task("worker-b", (CLAIM_EXPIRES_AT) - (NOW)) .await .unwrap() .expect("expired in-progress claim is reclaimed"); @@ -927,15 +967,18 @@ mod tests { None, ) .unwrap(); - let claimer = InMemoryVerificationTaskClaimer::with_claimed_tasks(vec![( - in_progress, - "worker-a".to_owned(), - datetime!(2026-05-29 12:11:00 UTC), - )]); + let claimer = InMemoryVerificationTaskClaimer::with_claimed_tasks_and_clock( + vec![( + in_progress, + "worker-a".to_owned(), + datetime!(2026-05-29 12:11:00 UTC), + )], + Arc::new(MutableClock::new(NOW)), + ); assert_eq!( claimer - .claim_next_verification_task("worker-b", NOW, CLAIM_EXPIRES_AT) + .claim_next_verification_task("worker-b", (CLAIM_EXPIRES_AT) - (NOW)) .await .unwrap(), None diff --git a/locks-service/src/infrastructure/memory/verification_task_deletion_fence.rs b/locks-service/src/infrastructure/memory/verification_task_deletion_fence.rs index dd1ca8b..a7bad85 100644 --- a/locks-service/src/infrastructure/memory/verification_task_deletion_fence.rs +++ b/locks-service/src/infrastructure/memory/verification_task_deletion_fence.rs @@ -14,7 +14,7 @@ use crate::application::{ type LockKey = (CreatorPubky, LockId); #[derive(Debug)] -struct SystemClock; +pub(crate) struct SystemClock; impl Clock for SystemClock { fn now(&self) -> OffsetDateTime { @@ -93,6 +93,13 @@ impl InMemoryVerificationTaskDeletionFence { } pub(crate) fn from_tasks(tasks: &[VerificationTaskRecord]) -> Self { + Self::from_tasks_with_clock(tasks, Arc::new(SystemClock)) + } + + pub(crate) fn from_tasks_with_clock( + tasks: &[VerificationTaskRecord], + clock: Arc, + ) -> Self { Self { records: RwLock::new( tasks @@ -124,7 +131,7 @@ impl InMemoryVerificationTaskDeletionFence { .collect(), ), lock_admissions: Mutex::new(HashMap::new()), - clock: Arc::new(SystemClock), + clock, } } } diff --git a/locks-service/src/infrastructure/memory/verification_tasks.rs b/locks-service/src/infrastructure/memory/verification_tasks.rs index 694873c..bd2a899 100644 --- a/locks-service/src/infrastructure/memory/verification_tasks.rs +++ b/locks-service/src/infrastructure/memory/verification_tasks.rs @@ -101,6 +101,29 @@ impl VerificationTaskRepository for InMemoryVerificationTaskRepository { Ok(()) } + async fn update_verification_tasks_atomically( + &self, + tasks: Vec, + ) -> Result<(), ApplicationError> { + let mut fence_records = self.deletion_fence.records.write().await; + let mut records = self.records.write().await; + if tasks + .iter() + .any(|task| !records.contains_key(&task.task_id)) + { + return Err(ApplicationError::MissingRecord { + record: "verification_task", + }); + } + for task in tasks { + if let Some(fence_record) = fence_records.get_mut(&task.task_id) { + fence_record.status = task.status; + } + records.insert(task.task_id, task); + } + Ok(()) + } + async fn get_verification_task( &self, task_id: &TaskId, diff --git a/locks-service/src/infrastructure/postgres/access_credentials.rs b/locks-service/src/infrastructure/postgres/access_credentials.rs index b7790f0..0bd5695 100644 --- a/locks-service/src/infrastructure/postgres/access_credentials.rs +++ b/locks-service/src/infrastructure/postgres/access_credentials.rs @@ -2,7 +2,7 @@ use std::str::FromStr; use async_trait::async_trait; use sqlx::{PgPool, Postgres, Row, Transaction}; -use time::OffsetDateTime; +use time::{Duration, OffsetDateTime}; use uuid::Uuid; use locks_core::{ @@ -13,9 +13,10 @@ use locks_core::{ use crate::application::errors::ApplicationError; use crate::application::models::{ AccessCredential, AccessCredentialLookupKey, AccessCredentialRecord, DeletionReadAuthorization, - EncryptedFinalCredential, FinalCredentialContext, IssuedDeletionCredential, + EncryptedFinalCredential, FinalAccessWindows, FinalCredentialContext, + FinalCredentialMaterialization, InitializeFinalAccessWindowsResult, IssuedDeletionCredential, }; -use crate::application::ports::AccessCredentialStore; +use crate::application::ports::{AccessCredentialStore, FinalCredentialWorkerIssueRequest}; use crate::infrastructure::final_credentials::FinalCredentialCipher; use super::proof_admission::lock_proof_admission; @@ -130,42 +131,225 @@ impl AccessCredentialStore for PostgresAccessCredentialStore { deletion_job_id: Uuid, worker_id: &str, claim_token: Uuid, - now: OffsetDateTime, - issuance_deadline: OffsetDateTime, - read_deadline: OffsetDateTime, - ) -> Result { - if issuance_deadline <= now || read_deadline <= issuance_deadline { + issuance_window: Duration, + read_window: Duration, + ) -> Result { + if issuance_window <= Duration::ZERO || read_window <= Duration::ZERO { return Err(ApplicationError::Storage { - message: "invalid final access window ordering".to_owned(), + message: "final access window durations must be positive".to_owned(), }); } - let updated = sqlx::query( - "UPDATE content_lock_deletion_jobs - SET final_issuance_started_at = COALESCE(final_issuance_started_at, $4), - final_credential_issuance_deadline = - COALESCE(final_credential_issuance_deadline, $5), - final_read_deadline = COALESCE(final_read_deadline, $6) - WHERE job_id = $1 AND claimed_by = $2 AND claim_token = $3 - AND claim_expires_at > $4 AND state = 'running' - AND force_requested_at IS NULL AND phase = 'issue_final_credentials'", + let mut transaction = self.pool.begin().await.map_err(storage_error)?; + let job = sqlx::query( + "SELECT state, phase, force_requested_at, claimed_by, claim_token, claim_expires_at, + final_issuance_started_at, final_credential_issuance_deadline, + final_read_deadline + FROM content_lock_deletion_jobs + WHERE job_id = $1 + FOR UPDATE", ) .bind(deletion_job_id) - .bind(worker_id) - .bind(claim_token) - .bind(now) - .bind(issuance_deadline) - .bind(read_deadline) - .execute(&self.pool) + .fetch_optional(&mut *transaction) .await .map_err(storage_error)?; - Ok(updated.rows_affected() == 1) + let Some(job) = job else { + transaction.commit().await.map_err(storage_error)?; + return Ok(InitializeFinalAccessWindowsResult::ClaimLost); + }; + let now: OffsetDateTime = sqlx::query_scalar("SELECT clock_timestamp()") + .fetch_one(&mut *transaction) + .await + .map_err(storage_error)?; + let owns_live_claim = job.try_get::("state").map_err(storage_error)? + == "running" + && job.try_get::("phase").map_err(storage_error)? + == "issue_final_credentials" + && job + .try_get::, _>("force_requested_at") + .map_err(storage_error)? + .is_none() + && job + .try_get::, _>("claimed_by") + .map_err(storage_error)? + .as_deref() + == Some(worker_id) + && job + .try_get::, _>("claim_token") + .map_err(storage_error)? + == Some(claim_token) + && job + .try_get::, _>("claim_expires_at") + .map_err(storage_error)? + .is_some_and(|expires_at| expires_at > now); + if !owns_live_claim { + transaction.commit().await.map_err(storage_error)?; + return Ok(InitializeFinalAccessWindowsResult::ClaimLost); + } + let existing = ( + job.try_get::, _>("final_issuance_started_at") + .map_err(storage_error)?, + job.try_get::, _>("final_credential_issuance_deadline") + .map_err(storage_error)?, + job.try_get::, _>("final_read_deadline") + .map_err(storage_error)?, + ); + let windows = match existing { + ( + Some(issuance_started_at), + Some(credential_issuance_deadline), + Some(read_deadline), + ) => FinalAccessWindows { + issuance_started_at, + credential_issuance_deadline, + read_deadline, + }, + (None, None, None) => { + let credential_issuance_deadline = + now.checked_add(issuance_window) + .ok_or_else(|| ApplicationError::Storage { + message: "final credential issuance deadline overflow".to_owned(), + })?; + let read_deadline = credential_issuance_deadline + .checked_add(read_window) + .ok_or_else(|| ApplicationError::Storage { + message: "final read deadline overflow".to_owned(), + })?; + sqlx::query( + "UPDATE content_lock_deletion_jobs + SET final_issuance_started_at = $2, + final_credential_issuance_deadline = $3, + final_read_deadline = $4 + WHERE job_id = $1", + ) + .bind(deletion_job_id) + .bind(now) + .bind(credential_issuance_deadline) + .bind(read_deadline) + .execute(&mut *transaction) + .await + .map_err(storage_error)?; + FinalAccessWindows { + issuance_started_at: now, + credential_issuance_deadline, + read_deadline, + } + } + _ => { + return Err(ApplicationError::Storage { + message: "incomplete final access windows in Postgres".to_owned(), + }); + } + }; + transaction.commit().await.map_err(storage_error)?; + Ok(InitializeFinalAccessWindowsResult::Initialized(windows)) + } + + async fn final_credentials_to_materialize( + &self, + deletion_job_id: Uuid, + worker_id: &str, + claim_token: Uuid, + limit: usize, + ) -> Result, ApplicationError> { + if limit == 0 { + return Ok(Vec::new()); + } + let limit = i64::try_from(limit).map_err(|_| ApplicationError::Storage { + message: "final credential materialization limit exceeds Postgres BIGINT".to_owned(), + })?; + let mut transaction = self.pool.begin().await.map_err(storage_error)?; + let job = sqlx::query( + "SELECT creator, state, phase, force_requested_at, claimed_by, claim_token, + claim_expires_at, final_credential_issuance_deadline + FROM content_lock_deletion_jobs + WHERE job_id = $1 + FOR UPDATE", + ) + .bind(deletion_job_id) + .fetch_optional(&mut *transaction) + .await + .map_err(storage_error)?; + let Some(job) = job else { + transaction.commit().await.map_err(storage_error)?; + return Ok(Vec::new()); + }; + let now: OffsetDateTime = sqlx::query_scalar("SELECT clock_timestamp()") + .fetch_one(&mut *transaction) + .await + .map_err(storage_error)?; + let owns_live_issue_claim = job.try_get::("state").map_err(storage_error)? + == "running" + && job.try_get::("phase").map_err(storage_error)? + == "issue_final_credentials" + && job + .try_get::, _>("force_requested_at") + .map_err(storage_error)? + .is_none() + && job + .try_get::, _>("claimed_by") + .map_err(storage_error)? + .as_deref() + == Some(worker_id) + && job + .try_get::, _>("claim_token") + .map_err(storage_error)? + == Some(claim_token) + && job + .try_get::, _>("claim_expires_at") + .map_err(storage_error)? + .is_some_and(|deadline| now < deadline) + && job + .try_get::, _>("final_credential_issuance_deadline") + .map_err(storage_error)? + .is_some_and(|deadline| now < deadline); + if !owns_live_issue_claim { + transaction.commit().await.map_err(storage_error)?; + return Ok(Vec::new()); + } + let creator: String = job.try_get("creator").map_err(storage_error)?; + let creator = + CreatorPubky::from_str(&creator).map_err(|error| ApplicationError::Storage { + message: format!("invalid deletion job creator stored in Postgres: {error}"), + })?; + let bundle_ids: Vec = sqlx::query_scalar( + "SELECT bundle_id + FROM content_lock_deletion_task_snapshot + WHERE deletion_job_id = $1 + AND resolved_status = 'completed' + AND final_credential_eligible_at IS NOT NULL + AND final_credential_issued_at IS NULL + ORDER BY bundle_id ASC + LIMIT $2", + ) + .bind(deletion_job_id) + .bind(limit) + .fetch_all(&mut *transaction) + .await + .map_err(storage_error)?; + let pending = bundle_ids + .into_iter() + .map(|bundle_id| { + BundleId::from_str(&bundle_id) + .map(|bundle_id| FinalCredentialMaterialization { + creator: creator.clone(), + bundle_id, + }) + .map_err(|error| ApplicationError::Storage { + message: format!( + "invalid final credential bundle_id stored in Postgres: {error}" + ), + }) + }) + .collect::, _>>()?; + transaction.commit().await.map_err(storage_error)?; + Ok(pending) } async fn issue_or_replay_final_credential( &self, creator: &CreatorPubky, bundle_id: &BundleId, - now: OffsetDateTime, + _caller_now: OffsetDateTime, candidate: AccessCredential, ) -> Result, ApplicationError> { let cipher = match &self.final_credential_cipher { @@ -181,7 +365,6 @@ impl AccessCredentialStore for PostgresAccessCredentialStore { AND job.state IN ('queued', 'running') AND job.force_requested_at IS NULL AND job.phase IN ('issue_final_credentials', 'drain_final_reads') - AND job.final_read_deadline > $3 AND EXISTS ( SELECT 1 FROM content_lock_deletion_task_snapshot AS snapshot @@ -194,7 +377,6 @@ impl AccessCredentialStore for PostgresAccessCredentialStore { ) .bind(creator.to_string()) .bind(bundle_id.to_string()) - .bind(now) .fetch_optional(&mut *transaction) .await .map_err(storage_error)?; @@ -202,6 +384,12 @@ impl AccessCredentialStore for PostgresAccessCredentialStore { transaction.commit().await.map_err(storage_error)?; return Ok(None); }; + // `transaction_timestamp()` is fixed at BEGIN; sample the wall clock only after the + // job-row lock has serialized this winner with force/phase/deadline transitions. + let now: OffsetDateTime = sqlx::query_scalar("SELECT clock_timestamp()") + .fetch_one(&mut *transaction) + .await + .map_err(storage_error)?; let deletion_job_id: Uuid = job.try_get("job_id").map_err(storage_error)?; let phase: String = job.try_get("phase").map_err(storage_error)?; let issuance_deadline: OffsetDateTime = job @@ -209,6 +397,10 @@ impl AccessCredentialStore for PostgresAccessCredentialStore { .map_err(storage_error)?; let expires_at: OffsetDateTime = job.try_get("final_read_deadline").map_err(storage_error)?; + if now >= expires_at { + transaction.commit().await.map_err(storage_error)?; + return Ok(None); + } let snapshot_exists = sqlx::query_scalar::<_, bool>( "SELECT TRUE FROM content_lock_deletion_task_snapshot @@ -337,6 +529,207 @@ impl AccessCredentialStore for PostgresAccessCredentialStore { })) } + async fn issue_or_replay_final_credential_for_worker( + &self, + request: FinalCredentialWorkerIssueRequest<'_>, + ) -> Result, ApplicationError> { + let FinalCredentialWorkerIssueRequest { + deletion_job_id, + worker_id, + claim_token, + creator, + bundle_id, + now: _caller_now, + candidate, + } = request; + let cipher = match &self.final_credential_cipher { + Some(cipher) => cipher, + None => return Ok(None), + }; + let mut transaction = self.pool.begin().await.map_err(storage_error)?; + let job = sqlx::query( + "SELECT job.job_id, job.creator, job.state, job.phase, job.force_requested_at, + job.claimed_by, job.claim_token, job.claim_expires_at, + job.final_credential_issuance_deadline, job.final_read_deadline, + job.frozen_content_lock + FROM content_lock_deletion_jobs AS job + WHERE job.job_id = $1 + FOR UPDATE OF job", + ) + .bind(deletion_job_id) + .fetch_optional(&mut *transaction) + .await + .map_err(storage_error)?; + let Some(job) = job else { + transaction.commit().await.map_err(storage_error)?; + return Ok(None); + }; + // PostgreSQL transaction time is fixed at BEGIN, so only clock_timestamp() sampled after + // the row lock is authoritative for a statement that may have waited behind another owner. + let now: OffsetDateTime = sqlx::query_scalar("SELECT clock_timestamp()") + .fetch_one(&mut *transaction) + .await + .map_err(storage_error)?; + let owns_live_issue_claim = job.try_get::("creator").map_err(storage_error)? + == creator.to_string() + && job.try_get::("state").map_err(storage_error)? == "running" + && job.try_get::("phase").map_err(storage_error)? + == "issue_final_credentials" + && job + .try_get::, _>("force_requested_at") + .map_err(storage_error)? + .is_none() + && job + .try_get::, _>("claimed_by") + .map_err(storage_error)? + .as_deref() + == Some(worker_id) + && job + .try_get::, _>("claim_token") + .map_err(storage_error)? + == Some(claim_token) + && job + .try_get::, _>("claim_expires_at") + .map_err(storage_error)? + .is_some_and(|deadline| deadline > now) + && job + .try_get::, _>("final_credential_issuance_deadline") + .map_err(storage_error)? + .is_some_and(|deadline| deadline > now) + && job + .try_get::, _>("final_read_deadline") + .map_err(storage_error)? + .is_some_and(|deadline| deadline > now); + if !owns_live_issue_claim { + transaction.commit().await.map_err(storage_error)?; + return Ok(None); + } + let expires_at: OffsetDateTime = + job.try_get("final_read_deadline").map_err(storage_error)?; + let snapshot_exists = sqlx::query_scalar::<_, bool>( + "SELECT TRUE + FROM content_lock_deletion_task_snapshot + WHERE deletion_job_id = $1 AND bundle_id = $2 + AND resolved_status = 'completed' + AND final_credential_eligible_at IS NOT NULL + FOR UPDATE", + ) + .bind(deletion_job_id) + .bind(bundle_id.to_string()) + .fetch_optional(&mut *transaction) + .await + .map_err(storage_error)? + .unwrap_or(false); + if !snapshot_exists { + transaction.commit().await.map_err(storage_error)?; + return Ok(None); + } + let existing_encrypted: Option = sqlx::query_scalar( + "SELECT encrypted_bearer + FROM content_lock_access_drain_credentials + WHERE deletion_job_id = $1 AND creator = $2 AND bundle_id = $3 + AND credential_kind = 'final' + FOR UPDATE", + ) + .bind(deletion_job_id) + .bind(creator.to_string()) + .bind(bundle_id.to_string()) + .fetch_optional(&mut *transaction) + .await + .map_err(storage_error)? + .flatten(); + let context = FinalCredentialContext { + deletion_job_id, + creator: creator.clone(), + bundle_id: bundle_id.clone(), + }; + if let Some(encrypted) = existing_encrypted { + let credential = cipher.decrypt(&context, &EncryptedFinalCredential::new(encrypted))?; + transaction.commit().await.map_err(storage_error)?; + return Ok(Some(IssuedDeletionCredential { + credential, + expires_at, + })); + } + let frozen: serde_json::Value = + job.try_get("frozen_content_lock").map_err(storage_error)?; + let frozen: ContentLock = + serde_json::from_value(frozen).map_err(|error| ApplicationError::Storage { + message: format!("invalid frozen content lock stored in Postgres: {error}"), + })?; + let encrypted = cipher.encrypt(&context, &candidate)?; + let lookup_key = AccessCredentialLookupKey::derive(&candidate); + let credential_id = Uuid::new_v4(); + sqlx::query( + "INSERT INTO access_credentials ( + lookup_key, creator, bundle_id, expires_at, deletion_job_id + ) VALUES ($1, $2, $3, $4, $5)", + ) + .bind(lookup_key.as_bytes().as_slice()) + .bind(creator.to_string()) + .bind(bundle_id.to_string()) + .bind(expires_at) + .bind(deletion_job_id) + .execute(&mut *transaction) + .await + .map_err(storage_error)?; + sqlx::query( + "INSERT INTO content_lock_access_drain_credentials ( + credential_id, deletion_job_id, lookup_key, creator, bundle_id, + credential_kind, issued_at, expires_at, encrypted_bearer + ) VALUES ($1, $2, $3, $4, $5, 'final', $6, $7, $8)", + ) + .bind(credential_id) + .bind(deletion_job_id) + .bind(lookup_key.as_bytes().as_slice()) + .bind(creator.to_string()) + .bind(bundle_id.to_string()) + .bind(now) + .bind(expires_at) + .bind(encrypted.as_str()) + .execute(&mut *transaction) + .await + .map_err(storage_error)?; + let mut resources: Vec = frozen.primary_resource.into_iter().collect(); + resources.extend( + frozen + .secondary_resources + .into_iter() + .map(|(path, resource)| { + GuardedResource::new(path, resource.hash, resource.content_type, resource.size) + .expect("persisted frozen manifest was validated at deletion admission") + }), + ); + for resource in resources { + sqlx::query( + "INSERT INTO content_lock_access_drain_reads (credential_id, guarded_path) + VALUES ($1, $2)", + ) + .bind(credential_id) + .bind(resource.path) + .execute(&mut *transaction) + .await + .map_err(storage_error)?; + } + sqlx::query( + "UPDATE content_lock_deletion_task_snapshot + SET final_credential_issued_at = $3 + WHERE deletion_job_id = $1 AND bundle_id = $2 + AND final_credential_issued_at IS NULL", + ) + .bind(deletion_job_id) + .bind(bundle_id.to_string()) + .bind(now) + .execute(&mut *transaction) + .await + .map_err(storage_error)?; + transaction.commit().await.map_err(storage_error)?; + Ok(Some(IssuedDeletionCredential { + credential: candidate, + expires_at, + })) + } + async fn final_credential_available( &self, creator: &CreatorPubky, @@ -387,8 +780,7 @@ impl AccessCredentialStore for PostgresAccessCredentialStore { &self, lookup_key: &AccessCredentialLookupKey, path: &str, - now: OffsetDateTime, - _claim_expires_at: OffsetDateTime, + claim_duration: Duration, ) -> Result, ApplicationError> { let mut transaction = self.pool.begin().await.map_err(storage_error)?; let Some(deletion_job_id) = lookup_deletion_job_id(&mut transaction, lookup_key).await? @@ -400,6 +792,12 @@ impl AccessCredentialStore for PostgresAccessCredentialStore { transaction.commit().await.map_err(storage_error)?; return Ok(None); }; + // Transaction time is fixed at BEGIN. Sample wall-clock time only after the exact job-row + // fence, so a wait cannot revive an expired credential, deadline, or read claim. + let now: OffsetDateTime = sqlx::query_scalar("SELECT clock_timestamp()") + .fetch_one(&mut *transaction) + .await + .map_err(storage_error)?; let credential = sqlx::query( "SELECT credential_id, credential_kind, creator, expires_at FROM content_lock_access_drain_credentials @@ -485,7 +883,12 @@ impl AccessCredentialStore for PostgresAccessCredentialStore { transaction.commit().await.map_err(storage_error)?; return Ok(None); }; - let bounded_expiry = (now + time::Duration::seconds(30)) + let bounded_expiry = now + .checked_add(claim_duration) + .ok_or_else(|| ApplicationError::Storage { + message: "final read claim expiry overflow".to_owned(), + })? + .min(now + time::Duration::seconds(30)) .min(credential_expiry) .min(read_deadline); if read_deadline <= now || bounded_expiry <= now { @@ -583,7 +986,6 @@ impl AccessCredentialStore for PostgresAccessCredentialStore { lookup_key: &AccessCredentialLookupKey, path: &str, claim_token: Uuid, - now: OffsetDateTime, ) -> Result { let mut transaction = self.pool.begin().await.map_err(storage_error)?; let Some(deletion_job_id) = lookup_deletion_job_id(&mut transaction, lookup_key).await? @@ -595,6 +997,10 @@ impl AccessCredentialStore for PostgresAccessCredentialStore { transaction.commit().await.map_err(storage_error)?; return Ok(false); }; + let now: OffsetDateTime = sqlx::query_scalar("SELECT clock_timestamp()") + .fetch_one(&mut *transaction) + .await + .map_err(storage_error)?; if !phase_allows_credential_access(&job.phase, "final") || job .final_read_deadline @@ -717,19 +1123,28 @@ fn storage_error(error: sqlx::Error) -> ApplicationError { #[cfg(test)] mod tests { - use std::str::FromStr; + use std::{collections::BTreeMap, str::FromStr}; use sqlx::Row; use time::macros::datetime; + use uuid::Uuid; - use locks_core::ids::{BundleId, CreatorPubky, LockId}; + use locks_core::{ + ids::{BundleId, CreatorPubky, GuardedResourceHash, LockId}, + lock_policy::{ + AccessPolicy, CONTENT_LOCK_VERSION, ContentLock, GuardedResource, LockLogic, + LockServerConfig, + }, + }; use super::PostgresAccessCredentialStore; use crate::application::errors::ApplicationError; use crate::application::models::{ AccessCredential, AccessCredentialLookupKey, AccessCredentialRecord, + InitializeFinalAccessWindowsResult, }; - use crate::application::ports::AccessCredentialStore; + use crate::application::ports::{AccessCredentialStore, FinalCredentialWorkerIssueRequest}; + use crate::infrastructure::final_credentials::FinalCredentialCipher; use crate::infrastructure::postgres::testing::TestDatabase; #[tokio::test] @@ -842,6 +1257,623 @@ mod tests { database.cleanup().await; } + #[tokio::test] + async fn final_credentials_to_materialize_returns_eligible_unissued_rows_in_order_with_limit() { + let database = TestDatabase::create().await; + let store = PostgresAccessCredentialStore::new(database.pool().clone()); + let now = sqlx::query_scalar::<_, time::OffsetDateTime>("SELECT clock_timestamp()") + .fetch_one(database.pool()) + .await + .unwrap(); + let claim_token = Uuid::new_v4(); + let job_id = insert_final_issuance_job(database.pool(), now, claim_token).await; + insert_final_snapshot( + database.pool(), + job_id, + "000G40R40M30E209185GR38E1W", + true, + false, + ) + .await; + insert_final_snapshot( + database.pool(), + job_id, + "000G40R40M30E209185GR38E1R", + true, + false, + ) + .await; + insert_final_snapshot( + database.pool(), + job_id, + "000G40R40M30E209185GR38E1M", + false, + false, + ) + .await; + insert_final_snapshot( + database.pool(), + job_id, + "000G40R40M30E209185GR38E1G", + true, + true, + ) + .await; + + let bounded = store + .final_credentials_to_materialize(job_id, "worker", claim_token, 1) + .await + .unwrap(); + assert_eq!(bounded.len(), 1); + assert_eq!(bounded[0].bundle_id.as_str(), "000G40R40M30E209185GR38E1R"); + assert_eq!(bounded[0].creator, creator()); + + let all = store + .final_credentials_to_materialize(job_id, "worker", claim_token, 10) + .await + .unwrap(); + assert_eq!( + all.iter() + .map(|pending| pending.bundle_id.as_str()) + .collect::>(), + vec!["000G40R40M30E209185GR38E1R", "000G40R40M30E209185GR38E1W"] + ); + assert!( + store + .final_credentials_to_materialize(job_id, "worker", claim_token, 0) + .await + .unwrap() + .is_empty() + ); + + database.cleanup().await; + } + + #[tokio::test] + async fn final_credentials_to_materialize_revalidates_exact_live_issue_claim_and_deadline() { + let database = TestDatabase::create().await; + let store = PostgresAccessCredentialStore::new(database.pool().clone()); + let now = sqlx::query_scalar::<_, time::OffsetDateTime>("SELECT clock_timestamp()") + .fetch_one(database.pool()) + .await + .unwrap(); + let claim_token = Uuid::new_v4(); + let job_id = insert_final_issuance_job(database.pool(), now, claim_token).await; + insert_final_snapshot( + database.pool(), + job_id, + "000G40R40M30E209185GR38E1W", + true, + false, + ) + .await; + + assert!( + store + .final_credentials_to_materialize(job_id, "worker", Uuid::new_v4(), 10) + .await + .unwrap() + .is_empty() + ); + + sqlx::query( + "UPDATE content_lock_deletion_jobs SET force_requested_at = $2 WHERE job_id = $1", + ) + .bind(job_id) + .bind(now) + .execute(database.pool()) + .await + .unwrap(); + assert!( + store + .final_credentials_to_materialize(job_id, "worker", claim_token, 10) + .await + .unwrap() + .is_empty() + ); + + sqlx::query( + "UPDATE content_lock_deletion_jobs + SET force_requested_at = NULL, phase = 'drain_final_reads' + WHERE job_id = $1", + ) + .bind(job_id) + .execute(database.pool()) + .await + .unwrap(); + assert!( + store + .final_credentials_to_materialize(job_id, "worker", claim_token, 10) + .await + .unwrap() + .is_empty() + ); + + sqlx::query( + "UPDATE content_lock_deletion_jobs + SET phase = 'issue_final_credentials', final_credential_issuance_deadline = $2 + WHERE job_id = $1", + ) + .bind(job_id) + .bind(now) + .execute(database.pool()) + .await + .unwrap(); + assert!( + store + .final_credentials_to_materialize(job_id, "worker", claim_token, 10) + .await + .unwrap() + .is_empty() + ); + + database.cleanup().await; + } + + #[tokio::test] + async fn final_credentials_to_materialize_samples_time_after_job_lock_and_rejects_equality() { + let database = TestDatabase::create().await; + let store = PostgresAccessCredentialStore::new(database.pool().clone()); + let now = sqlx::query_scalar::<_, time::OffsetDateTime>("SELECT clock_timestamp()") + .fetch_one(database.pool()) + .await + .unwrap(); + let claim_token = Uuid::new_v4(); + let job_id = insert_final_issuance_job(database.pool(), now, claim_token).await; + insert_final_snapshot( + database.pool(), + job_id, + "000G40R40M30E209185GR38E1W", + true, + false, + ) + .await; + + let mut blocker = database.pool().begin().await.unwrap(); + sqlx::query("SELECT job_id FROM content_lock_deletion_jobs WHERE job_id = $1 FOR UPDATE") + .bind(job_id) + .fetch_one(&mut *blocker) + .await + .unwrap(); + let waiting_store = store.clone(); + let enumeration = tokio::spawn(async move { + waiting_store + .final_credentials_to_materialize(job_id, "worker", claim_token, 10) + .await + }); + tokio::time::sleep(std::time::Duration::from_millis(25)).await; + assert!(!enumeration.is_finished()); + sqlx::query( + "UPDATE content_lock_deletion_jobs + SET claim_expires_at = clock_timestamp(), + final_credential_issuance_deadline = clock_timestamp() + WHERE job_id = $1", + ) + .bind(job_id) + .execute(&mut *blocker) + .await + .unwrap(); + blocker.commit().await.unwrap(); + + assert!(enumeration.await.unwrap().unwrap().is_empty()); + database.cleanup().await; + } + + #[tokio::test] + async fn worker_final_issuance_is_exact_claim_fenced_in_winner_transaction() { + let database = TestDatabase::create().await; + let store = PostgresAccessCredentialStore::with_final_credential_cipher( + database.pool().clone(), + FinalCredentialCipher::new([41; 32]), + ); + let now = sqlx::query_scalar::<_, time::OffsetDateTime>("SELECT CURRENT_TIMESTAMP") + .fetch_one(database.pool()) + .await + .unwrap(); + let claim_token = Uuid::new_v4(); + let job_id = insert_final_issuance_job(database.pool(), now, claim_token).await; + let bundle_id = BundleId::from_str("000G40R40M30E209185GR38E1R").unwrap(); + let creator = creator(); + insert_final_snapshot(database.pool(), job_id, bundle_id.as_str(), true, false).await; + + for (job, worker, token, at) in [ + (Uuid::new_v4(), "worker", claim_token, now), + (job_id, "reclaimer", claim_token, now), + (job_id, "worker", Uuid::new_v4(), now), + ] { + let candidate = AccessCredential::new(format!("denied-{job}-{worker}-{at}")); + assert!( + store + .issue_or_replay_final_credential_for_worker( + FinalCredentialWorkerIssueRequest { + deletion_job_id: job, + worker_id: worker, + claim_token: token, + creator: &creator, + bundle_id: &bundle_id, + now: at, + candidate, + }, + ) + .await + .unwrap() + .is_none() + ); + } + let count: i64 = sqlx::query_scalar("SELECT COUNT(*) FROM access_credentials") + .fetch_one(database.pool()) + .await + .unwrap(); + assert_eq!(count, 0); + + sqlx::query( + "UPDATE content_lock_deletion_jobs SET force_requested_at = $2 WHERE job_id = $1", + ) + .bind(job_id) + .bind(now) + .execute(database.pool()) + .await + .unwrap(); + assert!( + store + .issue_or_replay_final_credential_for_worker(FinalCredentialWorkerIssueRequest { + deletion_job_id: job_id, + worker_id: "worker", + claim_token, + creator: &creator, + bundle_id: &bundle_id, + now, + candidate: AccessCredential::new("force-loser"), + },) + .await + .unwrap() + .is_none() + ); + + let reclaimed_token = Uuid::new_v4(); + sqlx::query( + "UPDATE content_lock_deletion_jobs + SET force_requested_at = NULL, claimed_by = 'reclaimer', claim_token = $2, + claim_expires_at = $3 + WHERE job_id = $1", + ) + .bind(job_id) + .bind(reclaimed_token) + .bind(now + time::Duration::minutes(5)) + .execute(database.pool()) + .await + .unwrap(); + assert!( + store + .issue_or_replay_final_credential_for_worker(FinalCredentialWorkerIssueRequest { + deletion_job_id: job_id, + worker_id: "worker", + claim_token, + creator: &creator, + bundle_id: &bundle_id, + now, + candidate: AccessCredential::new("stale-reclaimed-loser"), + },) + .await + .unwrap() + .is_none() + ); + + let winner = AccessCredential::new("postgres-worker-winner"); + let issued = store + .issue_or_replay_final_credential_for_worker(FinalCredentialWorkerIssueRequest { + deletion_job_id: job_id, + worker_id: "reclaimer", + claim_token: reclaimed_token, + creator: &creator, + bundle_id: &bundle_id, + now, + candidate: winner.clone(), + }) + .await + .unwrap() + .unwrap(); + let replay = store + .issue_or_replay_final_credential_for_worker(FinalCredentialWorkerIssueRequest { + deletion_job_id: job_id, + worker_id: "reclaimer", + claim_token: reclaimed_token, + creator: &creator, + bundle_id: &bundle_id, + now, + candidate: AccessCredential::new("postgres-worker-loser"), + }) + .await + .unwrap() + .unwrap(); + assert_eq!(issued.credential, winner); + assert_eq!(replay, issued); + let count: i64 = sqlx::query_scalar("SELECT COUNT(*) FROM access_credentials") + .fetch_one(database.pool()) + .await + .unwrap(); + assert_eq!(count, 1); + + database.cleanup().await; + } + + #[tokio::test] + async fn public_final_issuance_rechecks_database_time_after_waiting_for_job_lock() { + let database = TestDatabase::create().await; + let store = PostgresAccessCredentialStore::with_final_credential_cipher( + database.pool().clone(), + FinalCredentialCipher::new([43; 32]), + ); + let caller_now = datetime!(2026-05-29 12:00:00 UTC); + let claim_token = Uuid::new_v4(); + let job_id = insert_final_issuance_job(database.pool(), caller_now, claim_token).await; + let bundle_id = BundleId::from_str("000G40R40M30E209185GR38E1R").unwrap(); + let creator = creator(); + insert_final_snapshot(database.pool(), job_id, bundle_id.as_str(), true, false).await; + + let mut blocker = database.pool().begin().await.unwrap(); + sqlx::query( + "UPDATE content_lock_deletion_jobs + SET final_credential_issuance_deadline = clock_timestamp() + interval '500 milliseconds', + final_read_deadline = clock_timestamp() + interval '10 minutes' + WHERE job_id = $1", + ) + .bind(job_id) + .execute(&mut *blocker) + .await + .unwrap(); + + let issuance = tokio::spawn(async move { + store + .issue_or_replay_final_credential( + &creator, + &bundle_id, + caller_now, + AccessCredential::new("must-not-persist-after-public-lock-wait"), + ) + .await + }); + sqlx::query("SELECT pg_sleep(1)") + .execute(&mut *blocker) + .await + .unwrap(); + blocker.commit().await.unwrap(); + + assert!(issuance.await.unwrap().unwrap().is_none()); + let count: i64 = sqlx::query_scalar("SELECT COUNT(*) FROM access_credentials") + .fetch_one(database.pool()) + .await + .unwrap(); + assert_eq!(count, 0); + + database.cleanup().await; + } + + #[tokio::test] + async fn worker_final_issuance_rechecks_database_time_after_waiting_for_job_lock() { + let database = TestDatabase::create().await; + let store = PostgresAccessCredentialStore::with_final_credential_cipher( + database.pool().clone(), + FinalCredentialCipher::new([42; 32]), + ); + let caller_now = datetime!(2026-05-29 12:00:00 UTC); + let claim_token = Uuid::new_v4(); + let job_id = insert_final_issuance_job(database.pool(), caller_now, claim_token).await; + let bundle_id = BundleId::from_str("000G40R40M30E209185GR38E1R").unwrap(); + let creator = creator(); + insert_final_snapshot(database.pool(), job_id, bundle_id.as_str(), true, false).await; + + let mut blocker = database.pool().begin().await.unwrap(); + sqlx::query( + "UPDATE content_lock_deletion_jobs + SET claim_expires_at = clock_timestamp() + interval '500 milliseconds', + final_credential_issuance_deadline = clock_timestamp() + interval '500 milliseconds', + final_read_deadline = clock_timestamp() + interval '10 minutes' + WHERE job_id = $1", + ) + .bind(job_id) + .execute(&mut *blocker) + .await + .unwrap(); + + let issuance = tokio::spawn(async move { + store + .issue_or_replay_final_credential_for_worker(FinalCredentialWorkerIssueRequest { + deletion_job_id: job_id, + worker_id: "worker", + claim_token, + creator: &creator, + bundle_id: &bundle_id, + now: caller_now, + candidate: AccessCredential::new("must-not-persist-after-lock-wait"), + }) + .await + }); + sqlx::query("SELECT pg_sleep(1)") + .execute(&mut *blocker) + .await + .unwrap(); + blocker.commit().await.unwrap(); + + assert!(issuance.await.unwrap().unwrap().is_none()); + let count: i64 = sqlx::query_scalar("SELECT COUNT(*) FROM access_credentials") + .fetch_one(database.pool()) + .await + .unwrap(); + assert_eq!(count, 0); + + database.cleanup().await; + } + + #[tokio::test] + async fn final_access_window_initialization_rechecks_database_time_after_job_lock() { + let database = TestDatabase::create().await; + let store = PostgresAccessCredentialStore::new(database.pool().clone()); + let fixture_now: time::OffsetDateTime = sqlx::query_scalar("SELECT clock_timestamp()") + .fetch_one(database.pool()) + .await + .unwrap(); + let claim_token = Uuid::new_v4(); + let job_id = insert_final_issuance_job(database.pool(), fixture_now, claim_token).await; + + let mut blocker = database.pool().begin().await.unwrap(); + sqlx::query( + "UPDATE content_lock_deletion_jobs + SET claim_expires_at = clock_timestamp() + interval '500 milliseconds', + final_issuance_started_at = NULL, + final_credential_issuance_deadline = NULL, + final_read_deadline = NULL + WHERE job_id = $1", + ) + .bind(job_id) + .execute(&mut *blocker) + .await + .unwrap(); + + let initialization = tokio::spawn(async move { + store + .initialize_final_access_windows( + job_id, + "worker", + claim_token, + time::Duration::minutes(15), + time::Duration::minutes(15), + ) + .await + }); + sqlx::query("SELECT pg_sleep(1)") + .execute(&mut *blocker) + .await + .unwrap(); + blocker.commit().await.unwrap(); + + assert_eq!( + initialization.await.unwrap().unwrap(), + InitializeFinalAccessWindowsResult::ClaimLost + ); + let windows: ( + Option, + Option, + Option, + ) = sqlx::query_as( + "SELECT final_issuance_started_at, final_credential_issuance_deadline, + final_read_deadline + FROM content_lock_deletion_jobs WHERE job_id = $1", + ) + .bind(job_id) + .fetch_one(database.pool()) + .await + .unwrap(); + assert_eq!(windows, (None, None, None)); + + database.cleanup().await; + } + + async fn insert_final_issuance_job( + pool: &sqlx::PgPool, + now: time::OffsetDateTime, + claim_token: Uuid, + ) -> Uuid { + let job_id = Uuid::new_v4(); + sqlx::query( + "INSERT INTO content_lock_deletion_jobs ( + job_id, creator, lock_id, frozen_content_lock, deletion_started_at, + state, phase, claimed_by, claim_token, claim_expires_at, + final_issuance_started_at, final_credential_issuance_deadline, + final_read_deadline + ) VALUES ($1, $2, $3, $4, $5, 'running', + 'issue_final_credentials', 'worker', $6, $7, $8, $9, $10)", + ) + .bind(job_id) + .bind(creator().to_string()) + .bind("000G40R40M30E209185GR38E1W8124GK2GAHC5RR34D1P70X3RFG") + .bind(serde_json::to_value(test_content_lock()).unwrap()) + .bind(now) + .bind(claim_token) + .bind(now + time::Duration::minutes(5)) + .bind(now - time::Duration::minutes(1)) + .bind(now + time::Duration::minutes(10)) + .bind(now + time::Duration::minutes(20)) + .execute(pool) + .await + .unwrap(); + job_id + } + + async fn insert_final_snapshot( + pool: &sqlx::PgPool, + job_id: Uuid, + bundle_id: &str, + eligible: bool, + issued: bool, + ) { + let task_id = Uuid::new_v4(); + let now = datetime!(2026-05-29 12:00:00 UTC); + sqlx::query( + "INSERT INTO verification_tasks ( + task_id, status, submitted_proof_bundle, submitted_at, creator, bundle_id, + deletion_job_id + ) VALUES ($1, 'completed', '{}'::jsonb, $2, $3, $4, $5)", + ) + .bind(task_id) + .bind(now) + .bind(creator().to_string()) + .bind(bundle_id) + .bind(job_id) + .execute(pool) + .await + .unwrap(); + sqlx::query( + "INSERT INTO content_lock_deletion_task_snapshot ( + deletion_job_id, verification_task_id, creator, bundle_id, + pubky_lock_resource, criterion_id, status_at_cutoff, + paykit_admission_required, resolved_status, resolved_at, + final_credential_eligible_at, final_credential_issued_at + ) VALUES ($1, $2, $3, $4, 'pubkycreator/pub/locks.app/lock.json', + 'payment', 'completed', TRUE, 'completed', $5, + CASE WHEN $6 THEN $5 END, + CASE WHEN $7 THEN $5 END)", + ) + .bind(job_id) + .bind(task_id) + .bind(creator().to_string()) + .bind(bundle_id) + .bind(now) + .bind(eligible) + .bind(issued) + .execute(pool) + .await + .unwrap(); + } + + fn creator() -> CreatorPubky { + CreatorPubky::from_str("pubkytkrq8zmwb8a3m9k15csu3q17qmfgqnp9dskbrg9uq1rydpyxp7qy").unwrap() + } + + fn test_content_lock() -> ContentLock { + ContentLock { + version: CONTENT_LOCK_VERSION, + creator: creator(), + primary_resource: Some( + GuardedResource::new( + "/priv/locks.app/content/post.json".to_owned(), + GuardedResourceHash::from_bytes([7; 32]), + "application/json".to_owned(), + 42, + ) + .unwrap(), + ), + secondary_resources: BTreeMap::new(), + criteria: vec![], + lock_logic: LockLogic::All { criteria: vec![] }, + access_policy: AccessPolicy { + requested_credential_ttl_seconds: 900, + }, + lock_server: LockServerConfig { override_: None }, + created_at: datetime!(2026-05-29 11:00:00 UTC), + } + } + async fn assert_stored_lookup_key_is_exact( pool: &sqlx::PgPool, lookup_key: &AccessCredentialLookupKey, diff --git a/locks-service/src/infrastructure/postgres/content_lock_deletion_action_ownership.rs b/locks-service/src/infrastructure/postgres/content_lock_deletion_action_ownership.rs new file mode 100644 index 0000000..21f6136 --- /dev/null +++ b/locks-service/src/infrastructure/postgres/content_lock_deletion_action_ownership.rs @@ -0,0 +1,152 @@ +use async_trait::async_trait; +use sqlx::{Connection, PgConnection, PgPool}; +use uuid::Uuid; + +use crate::application::{ + errors::ApplicationError, + models::ContentLockDeletionPhase, + ports::{ + ContentLockDeletionActionAcquireResult, ContentLockDeletionActionClaim, + ContentLockDeletionActionGuard, ContentLockDeletionActionOwnership, + }, +}; + +/// PostgreSQL session-advisory-lock ownership for deletion external actions. +#[derive(Debug, Clone)] +pub struct PostgresContentLockDeletionActionOwnership { + pool: PgPool, +} + +impl PostgresContentLockDeletionActionOwnership { + pub fn new(pool: PgPool) -> Self { + Self { pool } + } +} + +#[async_trait] +impl ContentLockDeletionActionOwnership for PostgresContentLockDeletionActionOwnership { + async fn try_acquire( + &self, + claim: ContentLockDeletionActionClaim<'_>, + ) -> Result { + let lock_key = action_lock_key(claim.job_id); + let mut pooled = self.pool.acquire().await.map_err(storage_error)?; + let acquired: bool = sqlx::query_scalar("SELECT pg_try_advisory_lock($1)") + .bind(lock_key) + .fetch_one(&mut *pooled) + .await + .map_err(storage_error)?; + if !acquired { + return Ok(ContentLockDeletionActionAcquireResult::Busy); + } + + // Detach immediately after locking. Any validation error then closes the + // session on drop instead of returning a locked connection to the pool. + let mut connection = pooled.detach(); + let live: bool = sqlx::query_scalar( + "SELECT EXISTS ( + SELECT 1 + FROM content_lock_deletion_jobs + WHERE job_id = $1 + AND state = 'running' + AND claimed_by = $2 + AND claim_token = $3 + AND claim_expires_at > clock_timestamp() + AND phase = $4 + AND (($5 AND force_requested_at IS NOT NULL) + OR (NOT $5 AND force_requested_at IS NULL)) + )", + ) + .bind(claim.job_id) + .bind(claim.worker_id) + .bind(claim.claim_token) + .bind(phase_to_database(claim.expected_phase)) + .bind(claim.force) + .fetch_one(&mut connection) + .await + .map_err(storage_error)?; + + if !live { + unlock_and_close(connection, lock_key).await?; + return Ok(ContentLockDeletionActionAcquireResult::ClaimLost); + } + + Ok(ContentLockDeletionActionAcquireResult::Acquired(Box::new( + PostgresContentLockDeletionActionGuard { + connection: Some(connection), + lock_key, + }, + ))) + } +} + +struct PostgresContentLockDeletionActionGuard { + connection: Option, + lock_key: i64, +} + +#[async_trait] +impl ContentLockDeletionActionGuard for PostgresContentLockDeletionActionGuard { + async fn release(mut self: Box) -> Result<(), ApplicationError> { + let connection = self + .connection + .take() + .ok_or_else(|| ApplicationError::Storage { + message: "content lock deletion action ownership was already released".to_owned(), + })?; + unlock_and_close(connection, self.lock_key).await + } +} + +impl Drop for PostgresContentLockDeletionActionGuard { + fn drop(&mut self) { + drop(self.connection.take()); + } +} + +async fn unlock_and_close( + mut connection: PgConnection, + lock_key: i64, +) -> Result<(), ApplicationError> { + let unlock_result = sqlx::query_scalar::<_, bool>("SELECT pg_advisory_unlock($1)") + .bind(lock_key) + .fetch_one(&mut connection) + .await; + let close_result = connection.close().await; + let unlocked = unlock_result.map_err(storage_error)?; + close_result.map_err(storage_error)?; + if !unlocked { + return Err(ApplicationError::Storage { + message: "content lock deletion action ownership was not held".to_owned(), + }); + } + Ok(()) +} + +fn phase_to_database(phase: ContentLockDeletionPhase) -> &'static str { + match phase { + ContentLockDeletionPhase::Withdraw => "withdraw", + ContentLockDeletionPhase::StartPaymentDrain => "start_payment_drain", + ContentLockDeletionPhase::DrainPayments => "drain_payments", + ContentLockDeletionPhase::DrainExistingCredentials => "drain_existing_credentials", + ContentLockDeletionPhase::IssueFinalCredentials => "issue_final_credentials", + ContentLockDeletionPhase::DrainFinalReads => "drain_final_reads", + ContentLockDeletionPhase::DeleteContent => "delete_content", + ContentLockDeletionPhase::DeleteTombstone => "delete_tombstone", + ContentLockDeletionPhase::PurgeOperationalState => "purge_operational_state", + } +} + +fn action_lock_key(job_id: Uuid) -> i64 { + let digest = blake3::derive_key( + "pubky-locks content-lock deletion external action ownership v1", + job_id.as_bytes(), + ); + i64::from_be_bytes(digest[..8].try_into().expect("eight-byte digest prefix")) +} + +fn storage_error(error: sqlx::Error) -> ApplicationError { + ApplicationError::Storage { + message: error.to_string(), + } +} diff --git a/locks-service/src/infrastructure/postgres/content_lock_deletions.rs b/locks-service/src/infrastructure/postgres/content_lock_deletions.rs index d0312ce..2988ef8 100644 --- a/locks-service/src/infrastructure/postgres/content_lock_deletions.rs +++ b/locks-service/src/infrastructure/postgres/content_lock_deletions.rs @@ -12,14 +12,16 @@ use uuid::Uuid; use crate::application::{ errors::ApplicationError, models::{ - ClaimedContentLockDeletionJob, ContentLockDeletionFailureCode, ContentLockDeletionJob, - ContentLockDeletionPhase, ContentLockDeletionState, PrepareForceDeletionResult, + AdvanceContentLockDeletionPhaseResult, ClaimedContentLockDeletionJob, + ContentLockDeletionFailureCode, ContentLockDeletionJob, ContentLockDeletionPhase, + ContentLockDeletionState, PrepareForceDeletionResult, }, ports::ContentLockDeletionRepository, }; use crate::infrastructure::postgres::proof_admission::lock_proof_admission; const ROW_COLUMNS: &str = "job_id, creator, lock_id, frozen_content_lock, deletion_started_at, state, phase, attempt_count, next_attempt_at, force_requested_at, failure_code, claimed_by, claim_token, claim_expires_at"; +const CLAIMED_ROW_COLUMNS: &str = "job.job_id, job.creator, job.lock_id, job.frozen_content_lock, job.deletion_started_at, job.state, job.phase, job.attempt_count, job.next_attempt_at, job.force_requested_at, job.failure_code, job.claimed_by, job.claim_token, job.claim_expires_at"; #[derive(Debug, FromRow)] struct DeletionJobRow { @@ -358,29 +360,32 @@ impl ContentLockDeletionRepository for PostgresContentLockDeletionRepository { async fn claim_next( &self, worker_id: &str, - now: OffsetDateTime, - claim_expires_at: OffsetDateTime, + claim_ttl: time::Duration, ) -> Result, ApplicationError> { let claim_token = Uuid::new_v4(); + let claim_ttl_seconds = claim_ttl.as_seconds_f64(); let sql = format!( - "UPDATE content_lock_deletion_jobs + "WITH winner AS MATERIALIZED (SELECT clock_timestamp() AS at), + candidate AS MATERIALIZED ( + SELECT job_id FROM content_lock_deletion_jobs, winner + WHERE (state = 'queued' AND (next_attempt_at IS NULL OR next_attempt_at <= winner.at)) + OR (state = 'running' AND claim_expires_at <= winner.at) + ORDER BY deletion_started_at + FOR UPDATE OF content_lock_deletion_jobs SKIP LOCKED LIMIT 1 + ) + UPDATE content_lock_deletion_jobs AS job SET state = 'running', claimed_by = $1, claim_token = $2, - claim_expires_at = $3, next_attempt_at = NULL, - attempt_count = attempt_count + 1, updated_at = $4 - WHERE job_id = ( - SELECT job_id FROM content_lock_deletion_jobs - WHERE (state = 'queued' AND (next_attempt_at IS NULL OR next_attempt_at <= $4)) - OR (state = 'running' AND claim_expires_at < $4) - ORDER BY deletion_started_at - FOR UPDATE SKIP LOCKED LIMIT 1 - ) - RETURNING {ROW_COLUMNS}" + claim_expires_at = winner.at + ($3 * interval '1 second'), + next_attempt_at = NULL, attempt_count = attempt_count + 1, + updated_at = winner.at + FROM candidate, winner + WHERE job.job_id = candidate.job_id + RETURNING {CLAIMED_ROW_COLUMNS}" ); let row = sqlx::query_as::<_, DeletionJobRow>(&sql) .bind(worker_id) .bind(claim_token) - .bind(claim_expires_at) - .bind(now) + .bind(claim_ttl_seconds) .fetch_optional(&self.pool) .await .map_err(storage_error)?; @@ -394,51 +399,83 @@ impl ContentLockDeletionRepository for PostgresContentLockDeletionRepository { job_id: Uuid, worker_id: &str, claim_token: Uuid, - now: OffsetDateTime, - next_attempt_at: OffsetDateTime, + retry_after: time::Duration, ) -> Result, ApplicationError> { + let mut transaction = self.pool.begin().await.map_err(storage_error)?; + let Some((_current, winner_time)) = + load_owned_claim(&mut transaction, job_id, worker_id, claim_token).await? + else { + transaction.rollback().await.map_err(storage_error)?; + return Ok(None); + }; let sql = format!( "UPDATE content_lock_deletion_jobs - SET state = 'queued', next_attempt_at = $5, claimed_by = NULL, - claim_token = NULL, claim_expires_at = NULL, updated_at = $4 - WHERE job_id = $1 AND state = 'running' AND claimed_by = $2 - AND claim_token = $3 AND claim_expires_at >= $4 - RETURNING {ROW_COLUMNS}" + SET state = 'queued', next_attempt_at = $2, claimed_by = NULL, + claim_token = NULL, claim_expires_at = NULL, updated_at = $3 + WHERE job_id = $1 RETURNING {ROW_COLUMNS}" ); - fetch_optional_job( - sqlx::query_as::<_, DeletionJobRow>(&sql) - .bind(job_id) - .bind(worker_id) - .bind(claim_token) - .bind(now) - .bind(next_attempt_at) - .fetch_optional(&self.pool) - .await - .map_err(storage_error)?, - ) + let row = sqlx::query_as::<_, DeletionJobRow>(&sql) + .bind(job_id) + .bind(winner_time + retry_after) + .bind(winner_time) + .fetch_one(&mut *transaction) + .await + .map_err(storage_error)?; + transaction.commit().await.map_err(storage_error)?; + fetch_optional_job(Some(row)) } - async fn advance_phase( + async fn defer( &self, job_id: Uuid, worker_id: &str, claim_token: Uuid, - now: OffsetDateTime, - next_phase: ContentLockDeletionPhase, + defer_for: time::Duration, ) -> Result, ApplicationError> { let mut transaction = self.pool.begin().await.map_err(storage_error)?; - let current = - load_owned_claim(&mut transaction, job_id, worker_id, claim_token, now).await?; - let Some(current) = current else { + let Some((_current, winner_time)) = + load_owned_claim(&mut transaction, job_id, worker_id, claim_token).await? + else { transaction.rollback().await.map_err(storage_error)?; return Ok(None); }; + let sql = format!( + "UPDATE content_lock_deletion_jobs + SET state = 'queued', attempt_count = GREATEST(attempt_count - 1, 0), + next_attempt_at = $2, claimed_by = NULL, claim_token = NULL, + claim_expires_at = NULL, updated_at = $3 + WHERE job_id = $1 RETURNING {ROW_COLUMNS}" + ); + let row = sqlx::query_as::<_, DeletionJobRow>(&sql) + .bind(job_id) + .bind(winner_time + defer_for) + .bind(winner_time) + .fetch_one(&mut *transaction) + .await + .map_err(storage_error)?; + transaction.commit().await.map_err(storage_error)?; + fetch_optional_job(Some(row)) + } + + async fn advance_phase( + &self, + job_id: Uuid, + worker_id: &str, + claim_token: Uuid, + next_phase: ContentLockDeletionPhase, + ) -> Result { + let mut transaction = self.pool.begin().await.map_err(storage_error)?; + let current = load_owned_claim(&mut transaction, job_id, worker_id, claim_token).await?; + let Some((current, now)) = current else { + transaction.rollback().await.map_err(storage_error)?; + return Ok(AdvanceContentLockDeletionPhaseResult::ClaimLost); + }; if !current.phase.permits(next_phase) { return Err(ApplicationError::InvalidContentLockDeletionState { message: "deletion phase must advance to its immediate successor".to_owned(), }); } - check_access_obligations_for_phase( + let access_status = check_access_obligations_for_phase( &mut transaction, job_id, current.phase, @@ -446,6 +483,19 @@ impl ContentLockDeletionRepository for PostgresContentLockDeletionRepository { now, ) .await?; + match access_status { + AccessPhaseAdvanceStatus::Ready => {} + AccessPhaseAdvanceStatus::ObligationsPending => { + transaction.commit().await.map_err(storage_error)?; + return Ok(AdvanceContentLockDeletionPhaseResult::ObligationsPending); + } + AccessPhaseAdvanceStatus::FinalCredentialIssuanceMissed => { + transaction.commit().await.map_err(storage_error)?; + return Ok(AdvanceContentLockDeletionPhaseResult::TerminalFailure( + ContentLockDeletionFailureCode::StateCorrupt, + )); + } + } if current.phase == ContentLockDeletionPhase::DrainPayments && next_phase == ContentLockDeletionPhase::DrainExistingCredentials { @@ -533,7 +583,80 @@ impl ContentLockDeletionRepository for PostgresContentLockDeletionRepository { .await .map_err(storage_error)?; transaction.commit().await.map_err(storage_error)?; - Ok(Some(row_to_job(updated)?)) + Ok(AdvanceContentLockDeletionPhaseResult::Advanced(Box::new( + row_to_job(updated)?, + ))) + } + + async fn expire_unresolved_non_paykit_tasks( + &self, + job_id: Uuid, + worker_id: &str, + claim_token: Uuid, + ) -> Result { + let mut transaction = self.pool.begin().await.map_err(storage_error)?; + let Some((current, now)) = + load_owned_claim(&mut transaction, job_id, worker_id, claim_token).await? + else { + transaction.rollback().await.map_err(storage_error)?; + return Ok(false); + }; + if !matches!( + current.phase, + ContentLockDeletionPhase::StartPaymentDrain | ContentLockDeletionPhase::DrainPayments + ) { + return Err(invalid_state( + "non-Paykit deletion drain requires a payment drain phase", + )); + } + let has_paykit_or_unknown: bool = sqlx::query_scalar( + "SELECT EXISTS ( + SELECT 1 FROM content_lock_deletion_task_snapshot + WHERE deletion_job_id = $1 + AND paykit_admission_required IS DISTINCT FROM FALSE + )", + ) + .bind(job_id) + .fetch_one(&mut *transaction) + .await + .map_err(storage_error)?; + if has_paykit_or_unknown { + return Err(invalid_state( + "non-Paykit deletion drain cannot process a Paykit snapshot", + )); + } + sqlx::query( + "UPDATE verification_tasks AS task + SET status = 'expired', completed_at = $2, failure_message = NULL, + claimed_by = NULL, claim_token = NULL, claim_expires_at = NULL, + entitlement_publication_claim_token = NULL, next_attempt_at = NULL, + last_attempt_error = NULL, updated_at = $2 + FROM content_lock_deletion_task_snapshot AS snapshot + WHERE snapshot.deletion_job_id = $1 + AND snapshot.verification_task_id = task.task_id + AND snapshot.paykit_admission_required = FALSE + AND snapshot.resolved_status IS NULL + AND task.status IN ('pending', 'in_progress')", + ) + .bind(job_id) + .bind(now) + .execute(&mut *transaction) + .await + .map_err(storage_error)?; + sqlx::query( + "UPDATE content_lock_deletion_task_snapshot + SET resolved_status = 'expired', resolved_at = $2 + WHERE deletion_job_id = $1 + AND paykit_admission_required = FALSE + AND resolved_status IS NULL", + ) + .bind(job_id) + .bind(now) + .execute(&mut *transaction) + .await + .map_err(storage_error)?; + transaction.commit().await.map_err(storage_error)?; + Ok(true) } async fn finish( @@ -541,13 +664,11 @@ impl ContentLockDeletionRepository for PostgresContentLockDeletionRepository { job_id: Uuid, worker_id: &str, claim_token: Uuid, - now: OffsetDateTime, failure_code: Option, ) -> Result, ApplicationError> { let mut transaction = self.pool.begin().await.map_err(storage_error)?; - let current = - load_owned_claim(&mut transaction, job_id, worker_id, claim_token, now).await?; - let Some(current) = current else { + let current = load_owned_claim(&mut transaction, job_id, worker_id, claim_token).await?; + let Some((current, now)) = current else { transaction.rollback().await.map_err(storage_error)?; return Ok(None); }; @@ -662,10 +783,13 @@ impl ContentLockDeletionRepository for PostgresContentLockDeletionRepository { &self, creator: &CreatorPubky, lock_id: &LockId, - forced_at: OffsetDateTime, ) -> Result { let mut transaction = self.pool.begin().await.map_err(storage_error)?; lock_proof_admission(&mut transaction, creator, lock_id).await?; + let forced_at: OffsetDateTime = sqlx::query_scalar("SELECT clock_timestamp()") + .fetch_one(&mut *transaction) + .await + .map_err(storage_error)?; let publication_in_progress = sqlx::query_scalar::<_, bool>( "SELECT EXISTS (SELECT 1 FROM content_lock_publication_intents WHERE creator = $1 AND lock_id = $2)", ) @@ -757,6 +881,68 @@ impl ContentLockDeletionRepository for PostgresContentLockDeletionRepository { Ok(PrepareForceDeletionResult::Synchronous(None)) } + async fn complete_force_deletion( + &self, + job_id: Uuid, + worker_id: &str, + claim_token: Uuid, + ) -> Result { + let key = sqlx::query_as::<_, (String, String)>( + "SELECT creator, lock_id FROM content_lock_deletion_jobs WHERE job_id = $1", + ) + .bind(job_id) + .fetch_optional(&self.pool) + .await + .map_err(storage_error)?; + let Some((creator, lock_id)) = key else { + return Ok(false); + }; + let creator = CreatorPubky::from_str(&creator).map_err(storage_display)?; + let lock_id = LockId::from_str(&lock_id).map_err(storage_display)?; + + let mut transaction = self.pool.begin().await.map_err(storage_error)?; + lock_proof_admission(&mut transaction, &creator, &lock_id).await?; + let current = load_owned_claim(&mut transaction, job_id, worker_id, claim_token).await?; + let Some((current, _now)) = current else { + transaction.rollback().await.map_err(storage_error)?; + return Ok(false); + }; + let Some(forced_at) = current.force_requested_at else { + transaction.rollback().await.map_err(storage_error)?; + return Ok(false); + }; + + sqlx::query( + "INSERT INTO content_lock_force_deletion_receipts (creator, lock_id, forced_at) + VALUES ($1, $2, $3)", + ) + .bind(current.creator.to_string()) + .bind(current.lock_id.to_string()) + .bind(forced_at) + .execute(&mut *transaction) + .await + .map_err(storage_error)?; + sqlx::query( + "UPDATE verification_tasks SET deletion_job_id = NULL WHERE deletion_job_id = $1", + ) + .bind(job_id) + .execute(&mut *transaction) + .await + .map_err(storage_error)?; + let deleted = sqlx::query("DELETE FROM content_lock_deletion_jobs WHERE job_id = $1") + .bind(job_id) + .execute(&mut *transaction) + .await + .map_err(storage_error)?; + if deleted.rows_affected() != 1 { + return Err(invalid_state( + "force completion lost its locked deletion job", + )); + } + transaction.commit().await.map_err(storage_error)?; + Ok(true) + } + async fn has_force_receipt( &self, creator: &CreatorPubky, @@ -774,13 +960,20 @@ impl ContentLockDeletionRepository for PostgresContentLockDeletionRepository { } } +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum AccessPhaseAdvanceStatus { + Ready, + ObligationsPending, + FinalCredentialIssuanceMissed, +} + async fn check_access_obligations_for_phase( transaction: &mut Transaction<'_, Postgres>, job_id: Uuid, current_phase: ContentLockDeletionPhase, next_phase: ContentLockDeletionPhase, now: OffsetDateTime, -) -> Result<(), ApplicationError> { +) -> Result { if current_phase == ContentLockDeletionPhase::DrainExistingCredentials && next_phase == ContentLockDeletionPhase::IssueFinalCredentials { @@ -797,9 +990,7 @@ async fn check_access_obligations_for_phase( .await .map_err(storage_error)?; if ordinary_active { - return Err(invalid_state( - "existing credentials must reach their original expiry before final issuance", - )); + return Ok(AccessPhaseAdvanceStatus::ObligationsPending); } } @@ -819,25 +1010,43 @@ async fn check_access_obligations_for_phase( .await .map_err(storage_error)?; if has_unissued_eligible { - return Err(invalid_state( - "every eligible final credential must be durably issued before final-read draining", - )); + let issuance_deadline: Option = sqlx::query_scalar( + "SELECT final_credential_issuance_deadline + FROM content_lock_deletion_jobs WHERE job_id = $1", + ) + .bind(job_id) + .fetch_one(&mut **transaction) + .await + .map_err(storage_error)?; + return Ok( + if issuance_deadline.is_some_and(|deadline| now >= deadline) { + AccessPhaseAdvanceStatus::FinalCredentialIssuanceMissed + } else { + AccessPhaseAdvanceStatus::ObligationsPending + }, + ); } } if current_phase == ContentLockDeletionPhase::DrainFinalReads && next_phase == ContentLockDeletionPhase::DeleteContent { - ensure_no_live_access_obligations(transaction, job_id, now).await?; + return Ok( + if has_live_access_obligations(transaction, job_id, now).await? { + AccessPhaseAdvanceStatus::ObligationsPending + } else { + AccessPhaseAdvanceStatus::Ready + }, + ); } - Ok(()) + Ok(AccessPhaseAdvanceStatus::Ready) } -async fn ensure_no_live_access_obligations( +async fn has_live_access_obligations( transaction: &mut Transaction<'_, Postgres>, job_id: Uuid, now: OffsetDateTime, -) -> Result<(), ApplicationError> { +) -> Result { sqlx::query( "UPDATE content_lock_access_drain_reads AS read SET claim_token = NULL, claim_expires_at = NULL @@ -884,7 +1093,15 @@ async fn ensure_no_live_access_obligations( .fetch_one(&mut **transaction) .await .map_err(storage_error)?; - if has_live_obligation { + Ok(has_live_obligation) +} + +async fn ensure_no_live_access_obligations( + transaction: &mut Transaction<'_, Postgres>, + job_id: Uuid, + now: OffsetDateTime, +) -> Result<(), ApplicationError> { + if has_live_access_obligations(transaction, job_id, now).await? { return Err(invalid_state( "credential expiry and final-read obligations must drain before destructive deletion", )); @@ -929,9 +1146,21 @@ async fn ensure_payment_drain_completed( .fetch_optional(&mut **transaction) .await .map_err(storage_error)?; - if !aggregate - .is_some_and(|(status, accepted_count)| status == "completed" && accepted_count == 0) - { + let completed = aggregate + .as_ref() + .is_some_and(|(status, accepted_count)| status == "completed" && *accepted_count == 0); + let has_paykit_snapshot: bool = sqlx::query_scalar( + "SELECT EXISTS ( + SELECT 1 FROM content_lock_deletion_task_snapshot + WHERE deletion_job_id = $1 + AND paykit_admission_required IS DISTINCT FROM FALSE + )", + ) + .bind(job_id) + .fetch_one(&mut **transaction) + .await + .map_err(storage_error)?; + if !completed && (aggregate.is_some() || has_paykit_snapshot) { return Err(invalid_state( "payment drain aggregate must be durably completed before credential draining", )); @@ -989,23 +1218,33 @@ async fn load_owned_claim( job_id: Uuid, worker_id: &str, claim_token: Uuid, - now: OffsetDateTime, -) -> Result, ApplicationError> { +) -> Result, ApplicationError> { let sql = format!( "SELECT {ROW_COLUMNS} FROM content_lock_deletion_jobs WHERE job_id = $1 AND state = 'running' AND claimed_by = $2 - AND claim_token = $3 AND claim_expires_at >= $4 FOR UPDATE" + AND claim_token = $3 FOR UPDATE" ); - sqlx::query_as::<_, DeletionJobRow>(&sql) + let row = sqlx::query_as::<_, DeletionJobRow>(&sql) .bind(job_id) .bind(worker_id) .bind(claim_token) - .bind(now) .fetch_optional(&mut **transaction) .await - .map_err(storage_error)? - .map(row_to_job) - .transpose() + .map_err(storage_error)?; + let Some(row) = row else { + return Ok(None); + }; + let winner_time: OffsetDateTime = sqlx::query_scalar("SELECT clock_timestamp()") + .fetch_one(&mut **transaction) + .await + .map_err(storage_error)?; + if row + .claim_expires_at + .is_none_or(|claim_expires_at| winner_time >= claim_expires_at) + { + return Ok(None); + } + Ok(Some((row_to_job(row)?, winner_time))) } fn fetch_optional_job( @@ -1117,59 +1356,426 @@ fn storage_error(error: sqlx::Error) -> ApplicationError { storage_display(error) } -fn storage_display(error: impl std::fmt::Display) -> ApplicationError { - ApplicationError::Storage { - message: error.to_string(), +fn storage_display(error: impl std::fmt::Display) -> ApplicationError { + ApplicationError::Storage { + message: error.to_string(), + } +} + +#[cfg(test)] +mod tests { + use std::{ + collections::BTreeMap, + str::FromStr, + sync::{ + Mutex, + atomic::{AtomicUsize, Ordering}, + }, + }; + + use async_trait::async_trait; + use locks_core::{ + ids::{ + BundleId, CreatorPubky, GuardedResourceHash, LockServerPubky, PubkyLockResource, TaskId, + }, + lock_policy::{ + AccessPolicy, CONTENT_LOCK_VERSION, ContentLock, GuardedResource, LockLogic, + LockServerConfig, VerifierType, + }, + verification::{Proof, SUBMITTED_PROOF_BUNDLE_VERSION, SubmittedProofBundle}, + }; + use time::macros::datetime; + use uuid::Uuid; + + use super::PostgresContentLockDeletionRepository; + use crate::{ + application::{ + errors::ApplicationError, + models::{ + AccessCredential, AccessCredentialLookupKey, AccessCredentialRecord, + AdvanceContentLockDeletionPhaseResult, ContentLockDeletionFailureCode, + ContentLockDeletionJob, ContentLockDeletionPhase, ContentLockDeletionState, + InitializeFinalAccessWindowsResult, PrepareForceDeletionResult, + VerificationTaskRecord, VerificationTaskStatus, + }, + ports::{ + AccessCredentialStore, Clock, ContentLockDeletionActionAcquireResult, + ContentLockDeletionActionClaim, ContentLockDeletionActionOwnership, + ContentLockDeletionRepository, EntitlementRepository, PaymentDrainCleanupToken, + PaymentDrainClient, PaymentDrainClientError, PaymentDrainRepository, + PaymentDrainStatus, PaymentDrainSummary, PaymentDrainTerminalTransition, + PaymentRequestState, PaymentRequestStatus, PaymentState, VerificationTaskClaimer, + VerificationTaskRepository, + }, + use_cases::drain_lock_payments::DrainLockPaymentsUseCase, + }, + infrastructure::memory::entitlements::InMemoryEntitlementRepository, + infrastructure::postgres::{ + PostgresAccessCredentialStore, PostgresContentLockDeletionActionOwnership, + PostgresPaymentDrainRepository, PostgresVerificationTaskClaimer, + PostgresVerificationTaskRepository, testing::TestDatabase, + }, + }; + + const CREATOR: &str = "pubkytkrq8zmwb8a3m9k15csu3q17qmfgqnp9dskbrg9uq1rydpyxp7qy"; + const NOW: time::OffsetDateTime = datetime!(2026-08-12 05:00:00 UTC); + + #[tokio::test] + async fn healthy_defer_restores_the_postgres_attempt_budget_and_fences_stale_tokens() { + let database = TestDatabase::create().await; + let repository = PostgresContentLockDeletionRepository::new(database.pool().clone()); + let job = ContentLockDeletionJob::new(Uuid::new_v4(), content_lock(), NOW).unwrap(); + repository.insert_job(job.clone()).await.unwrap(); + + let first = repository + .claim_next("worker", (NOW + time::Duration::minutes(5)) - (NOW)) + .await + .unwrap() + .unwrap(); + assert_eq!(first.job.attempt_count, 1); + assert!( + repository + .defer( + job.job_id, + "worker", + Uuid::new_v4(), + (NOW + time::Duration::minutes(1)) - (NOW), + ) + .await + .unwrap() + .is_none() + ); + + let deferred = repository + .defer( + job.job_id, + "worker", + first.claim_token, + time::Duration::minutes(1), + ) + .await + .unwrap() + .unwrap(); + assert_eq!(deferred.attempt_count, 0); + let (first_due, first_updated_at): (time::OffsetDateTime, time::OffsetDateTime) = + sqlx::query_as( + "SELECT next_attempt_at, updated_at + FROM content_lock_deletion_jobs WHERE job_id = $1", + ) + .bind(job.job_id) + .fetch_one(database.pool()) + .await + .unwrap(); + assert_eq!(deferred.next_attempt_at, Some(first_due)); + assert_eq!(first_due - first_updated_at, time::Duration::minutes(1)); + sqlx::query( + "UPDATE content_lock_deletion_jobs + SET next_attempt_at = clock_timestamp() + WHERE job_id = $1", + ) + .bind(job.job_id) + .execute(database.pool()) + .await + .unwrap(); + + let second = repository + .claim_next("worker", time::Duration::minutes(5)) + .await + .unwrap() + .unwrap(); + assert_eq!(second.job.attempt_count, 1); + let deferred = repository + .defer( + job.job_id, + "worker", + second.claim_token, + time::Duration::minutes(1), + ) + .await + .unwrap() + .unwrap(); + assert_eq!(deferred.attempt_count, 0); + let (second_due, second_updated_at): (time::OffsetDateTime, time::OffsetDateTime) = + sqlx::query_as( + "SELECT next_attempt_at, updated_at + FROM content_lock_deletion_jobs WHERE job_id = $1", + ) + .bind(job.job_id) + .fetch_one(database.pool()) + .await + .unwrap(); + assert_eq!(deferred.next_attempt_at, Some(second_due)); + assert_eq!(second_due - second_updated_at, time::Duration::minutes(1)); + + database.cleanup().await; + } + + #[tokio::test] + async fn active_force_completion_persists_original_receipt_and_removes_operational_job() { + let database = TestDatabase::create().await; + let repository = PostgresContentLockDeletionRepository::new(database.pool().clone()); + let lock = content_lock(); + let task = verification_task(&lock, BundleId::from_bytes([99; 16])); + PostgresVerificationTaskRepository::new(database.pool().clone()) + .insert_verification_task(task.clone()) + .await + .unwrap(); + let job = ContentLockDeletionJob::new(Uuid::new_v4(), lock, NOW).unwrap(); + repository.insert_job(job.clone()).await.unwrap(); + let revoked = repository + .claim_next("worker-old", (NOW + time::Duration::minutes(1)) - (NOW)) + .await + .unwrap() + .unwrap(); + let forced_at = match repository + .prepare_force_deletion(&job.creator, &job.lock_id) + .await + .unwrap() + { + PrepareForceDeletionResult::Active(job) => job.force_requested_at.unwrap(), + result => panic!("expected active force deletion, got {result:?}"), + }; + assert!(matches!( + repository + .prepare_force_deletion(&job.creator, &job.lock_id,) + .await + .unwrap(), + PrepareForceDeletionResult::Active(_) + )); + assert!( + !repository + .complete_force_deletion(job.job_id, "worker-old", revoked.claim_token) + .await + .unwrap() + ); + + let live = repository + .claim_next("worker-live", time::Duration::minutes(5)) + .await + .unwrap() + .unwrap(); + assert!( + !repository + .complete_force_deletion(job.job_id, "worker-live", Uuid::new_v4()) + .await + .unwrap() + ); + assert!( + repository + .complete_force_deletion(job.job_id, "worker-live", live.claim_token) + .await + .unwrap() + ); + + let receipt: (String, String, time::OffsetDateTime) = sqlx::query_as( + "SELECT creator, lock_id, forced_at + FROM content_lock_force_deletion_receipts", + ) + .fetch_one(database.pool()) + .await + .unwrap(); + assert_eq!(receipt.0, job.creator.to_string()); + assert_eq!(receipt.1, job.lock_id.to_string()); + assert_eq!(receipt.2, forced_at); + assert!( + repository + .get_job(&job.creator, &job.lock_id) + .await + .unwrap() + .is_none() + ); + let retained_task_job: Option = + sqlx::query_scalar("SELECT deletion_job_id FROM verification_tasks WHERE task_id = $1") + .bind(task.task_id.as_uuid()) + .fetch_one(database.pool()) + .await + .unwrap(); + assert_eq!(retained_task_job, None); + let snapshot_count: i64 = sqlx::query_scalar( + "SELECT COUNT(*) FROM content_lock_deletion_task_snapshot WHERE deletion_job_id = $1", + ) + .bind(job.job_id) + .fetch_one(database.pool()) + .await + .unwrap(); + assert_eq!(snapshot_count, 0); + assert!( + !repository + .complete_force_deletion(job.job_id, "worker-live", live.claim_token) + .await + .unwrap() + ); + assert_eq!( + repository + .begin_publication(&job.creator, &job.lock_id, Uuid::new_v4()) + .await, + Err(ApplicationError::ContentLockDeletionInProgress) + ); + + database.cleanup().await; + } + + #[tokio::test] + async fn expired_or_unforced_postgres_claim_cannot_finalize_force_deletion() { + let database = TestDatabase::create().await; + let repository = PostgresContentLockDeletionRepository::new(database.pool().clone()); + let unforced = ContentLockDeletionJob::new(Uuid::new_v4(), content_lock(), NOW).unwrap(); + repository.insert_job(unforced.clone()).await.unwrap(); + let claim = repository + .claim_next("worker", (NOW + time::Duration::minutes(1)) - (NOW)) + .await + .unwrap() + .unwrap(); + assert!( + !repository + .complete_force_deletion(unforced.job_id, "worker", claim.claim_token) + .await + .unwrap() + ); + repository + .prepare_force_deletion(&unforced.creator, &unforced.lock_id) + .await + .unwrap(); + let expiring = repository + .claim_next("worker", time::Duration::minutes(1)) + .await + .unwrap() + .unwrap(); + sqlx::query( + "UPDATE content_lock_deletion_jobs + SET claim_expires_at = clock_timestamp() + WHERE job_id = $1 AND claim_token = $2", + ) + .bind(unforced.job_id) + .bind(expiring.claim_token) + .execute(database.pool()) + .await + .unwrap(); + assert!( + !repository + .complete_force_deletion(unforced.job_id, "worker", expiring.claim_token,) + .await + .unwrap() + ); + assert!( + repository + .get_job(&unforced.creator, &unforced.lock_id) + .await + .unwrap() + .is_some() + ); + assert!( + !repository + .has_force_receipt(&unforced.creator, &unforced.lock_id) + .await + .unwrap() + ); + + database.cleanup().await; + } + + #[tokio::test] + async fn postgres_action_ownership_validates_exact_live_claim_after_locking() { + let database = TestDatabase::create().await; + let repository = PostgresContentLockDeletionRepository::new(database.pool().clone()); + let first_owner = PostgresContentLockDeletionActionOwnership::new(database.pool().clone()); + let second_owner = PostgresContentLockDeletionActionOwnership::new(database.pool().clone()); + let job = ContentLockDeletionJob::new(Uuid::new_v4(), content_lock(), NOW).unwrap(); + repository.insert_job(job).await.unwrap(); + let claimed = repository + .claim_next("worker", time::Duration::minutes(5)) + .await + .unwrap() + .unwrap(); + let request = || ContentLockDeletionActionClaim { + job_id: claimed.job.job_id, + worker_id: "worker", + claim_token: claimed.claim_token, + expected_phase: claimed.job.phase, + force: false, + }; + + let ContentLockDeletionActionAcquireResult::Acquired(first) = + first_owner.try_acquire(request()).await.unwrap() + else { + panic!("live claim must acquire") + }; + assert!(matches!( + second_owner.try_acquire(request()).await.unwrap(), + ContentLockDeletionActionAcquireResult::Busy + )); + + first.release().await.unwrap(); + let ContentLockDeletionActionAcquireResult::Acquired(reacquired) = + second_owner.try_acquire(request()).await.unwrap() + else { + panic!("released live claim must reacquire") + }; + reacquired.release().await.unwrap(); + + let stale = ContentLockDeletionActionClaim { + worker_id: "other-worker", + ..request() + }; + assert!(matches!( + second_owner.try_acquire(stale).await.unwrap(), + ContentLockDeletionActionAcquireResult::ClaimLost + )); + let wrong_phase = ContentLockDeletionActionClaim { + expected_phase: ContentLockDeletionPhase::DrainPayments, + ..request() + }; + assert!(matches!( + second_owner.try_acquire(wrong_phase).await.unwrap(), + ContentLockDeletionActionAcquireResult::ClaimLost + )); + let wrong_mode = ContentLockDeletionActionClaim { + force: true, + ..request() + }; + assert!(matches!( + second_owner.try_acquire(wrong_mode).await.unwrap(), + ContentLockDeletionActionAcquireResult::ClaimLost + )); + + database.cleanup().await; } -} - -#[cfg(test)] -mod tests { - use std::{collections::BTreeMap, str::FromStr, sync::Mutex}; - use async_trait::async_trait; - use locks_core::{ - ids::{ - BundleId, CreatorPubky, GuardedResourceHash, LockServerPubky, PubkyLockResource, TaskId, - }, - lock_policy::{ - AccessPolicy, CONTENT_LOCK_VERSION, ContentLock, GuardedResource, LockLogic, - LockServerConfig, VerifierType, - }, - verification::{Proof, SUBMITTED_PROOF_BUNDLE_VERSION, SubmittedProofBundle}, - }; - use time::macros::datetime; - use uuid::Uuid; + #[tokio::test] + async fn postgres_action_ownership_rejects_lease_expiry_equality_and_releases_session_lock() { + let database = TestDatabase::create().await; + let repository = PostgresContentLockDeletionRepository::new(database.pool().clone()); + let ownership = PostgresContentLockDeletionActionOwnership::new(database.pool().clone()); + let job = ContentLockDeletionJob::new(Uuid::new_v4(), content_lock(), NOW).unwrap(); + repository.insert_job(job).await.unwrap(); + let claimed = repository + .claim_next("worker", time::Duration::minutes(5)) + .await + .unwrap() + .unwrap(); + sqlx::query("UPDATE content_lock_deletion_jobs SET claim_expires_at = clock_timestamp() WHERE job_id = $1") + .bind(claimed.job.job_id) + .execute(database.pool()).await.unwrap(); + let request = ContentLockDeletionActionClaim { + job_id: claimed.job.job_id, + worker_id: "worker", + claim_token: claimed.claim_token, + expected_phase: claimed.job.phase, + force: false, + }; + assert!(matches!( + ownership.try_acquire(request).await.unwrap(), + ContentLockDeletionActionAcquireResult::ClaimLost + )); - use super::PostgresContentLockDeletionRepository; - use crate::{ - application::{ - errors::ApplicationError, - models::{ - AccessCredential, AccessCredentialLookupKey, AccessCredentialRecord, - ContentLockDeletionFailureCode, ContentLockDeletionJob, ContentLockDeletionPhase, - ContentLockDeletionState, PrepareForceDeletionResult, VerificationTaskRecord, - VerificationTaskStatus, - }, - ports::{ - AccessCredentialStore, Clock, ContentLockDeletionRepository, EntitlementRepository, - PaymentDrainCleanupToken, PaymentDrainClient, PaymentDrainClientError, - PaymentDrainRepository, PaymentDrainStatus, PaymentDrainSummary, - PaymentDrainTerminalTransition, PaymentRequestState, PaymentRequestStatus, - PaymentState, VerificationTaskClaimer, VerificationTaskRepository, - }, - use_cases::drain_lock_payments::DrainLockPaymentsUseCase, - }, - infrastructure::memory::entitlements::InMemoryEntitlementRepository, - infrastructure::postgres::{ - PostgresAccessCredentialStore, PostgresPaymentDrainRepository, - PostgresVerificationTaskClaimer, PostgresVerificationTaskRepository, - testing::TestDatabase, - }, - }; + let value: i32 = sqlx::query_scalar("SELECT 1") + .fetch_one(database.pool()) + .await + .unwrap(); + assert_eq!(value, 1); - const CREATOR: &str = "pubkytkrq8zmwb8a3m9k15csu3q17qmfgqnp9dskbrg9uq1rydpyxp7qy"; - const NOW: time::OffsetDateTime = datetime!(2026-08-12 05:00:00 UTC); + database.cleanup().await; + } #[tokio::test] async fn admission_cutoff_timestamp_is_established_after_the_canonical_fence() { @@ -1369,12 +1975,7 @@ mod tests { .unwrap() ); let authorization = credentials - .prepare_deletion_read( - &lookup_key, - &guarded_path, - NOW, - NOW + time::Duration::seconds(30), - ) + .prepare_deletion_read(&lookup_key, &guarded_path, time::Duration::seconds(30)) .await .unwrap() .unwrap(); @@ -1385,8 +1986,7 @@ mod tests { .prepare_deletion_read( &lookup_key, "/priv/locks.app/content/not-frozen.json", - NOW, - NOW + time::Duration::seconds(30), + time::Duration::seconds(30), ) .await .unwrap() @@ -1512,7 +2112,7 @@ mod tests { let (graceful, force) = tokio::join!( repository.insert_job(job.clone()), - repository.prepare_force_deletion(&job.creator, &job.lock_id, NOW) + repository.prepare_force_deletion(&job.creator, &job.lock_id) ); let persisted_job = repository .get_job(&job.creator, &job.lock_id) @@ -1558,7 +2158,7 @@ mod tests { .unwrap(); assert_eq!( repository - .prepare_force_deletion(&lock.creator, &lock_id, NOW) + .prepare_force_deletion(&lock.creator, &lock_id) .await .unwrap(), PrepareForceDeletionResult::PublicationInProgress @@ -1577,7 +2177,7 @@ mod tests { ); assert_eq!( repository - .prepare_force_deletion(&lock.creator, &lock_id, NOW) + .prepare_force_deletion(&lock.creator, &lock_id) .await .unwrap(), PrepareForceDeletionResult::Synchronous(None) @@ -1671,7 +2271,7 @@ mod tests { .await .unwrap(); let claim = repository - .claim_next("worker", NOW, NOW + time::Duration::seconds(60)) + .claim_next("worker", (NOW + time::Duration::seconds(60)) - (NOW)) .await .unwrap() .unwrap(); @@ -1681,7 +2281,6 @@ mod tests { claim.job.job_id, "worker", claim.claim_token, - NOW, ContentLockDeletionPhase::StartPaymentDrain, ) .await; @@ -1719,12 +2318,12 @@ mod tests { claim.job.job_id, "worker", claim.claim_token, - NOW, ContentLockDeletionPhase::StartPaymentDrain, ) .await .unwrap() - .unwrap(); + .advanced() + .expect("live claim should advance phase"); assert_eq!(advanced.phase, ContentLockDeletionPhase::StartPaymentDrain); database.cleanup().await; @@ -1757,7 +2356,7 @@ mod tests { .await .unwrap(); let claim = repository - .claim_next("worker", NOW, NOW + time::Duration::seconds(60)) + .claim_next("worker", (NOW + time::Duration::seconds(60)) - (NOW)) .await .unwrap() .unwrap(); @@ -1767,7 +2366,6 @@ mod tests { claim.job.job_id, "worker", claim.claim_token, - NOW, ContentLockDeletionPhase::StartPaymentDrain, ) .await; @@ -1842,23 +2440,29 @@ mod tests { assert!(reopened.insert_job(distinct_job).await.is_err()); let first = reopened - .claim_next("worker-a", NOW, datetime!(2026-08-12 05:05:00 UTC)) + .claim_next("worker-a", (datetime!(2026-08-12 05:05:00 UTC)) - (NOW)) .await .unwrap() .unwrap(); assert!( reopened - .claim_next("worker-b", NOW, datetime!(2026-08-12 05:05:00 UTC),) + .claim_next("worker-b", (datetime!(2026-08-12 05:05:00 UTC)) - (NOW)) .await .unwrap() .is_none() ); + sqlx::query( + "UPDATE content_lock_deletion_jobs + SET claim_expires_at = clock_timestamp() + WHERE job_id = $1 AND claim_token = $2", + ) + .bind(job.job_id) + .bind(first.claim_token) + .execute(database.pool()) + .await + .unwrap(); let reclaimed = reopened - .claim_next( - "worker-b", - datetime!(2026-08-12 05:05:01 UTC), - datetime!(2026-08-12 05:10:00 UTC), - ) + .claim_next("worker-b", time::Duration::minutes(5)) .await .unwrap() .unwrap(); @@ -1869,25 +2473,23 @@ mod tests { job.job_id, "worker-a", first.claim_token, - datetime!(2026-08-12 05:05:01 UTC), - datetime!(2026-08-12 05:06:00 UTC), + (datetime!(2026-08-12 05:06:00 UTC)) - (datetime!(2026-08-12 05:05:01 UTC)), ) .await .unwrap() .is_none() ); - assert!( + assert_eq!( reopened .advance_phase( job.job_id, "worker-a", first.claim_token, - datetime!(2026-08-12 05:05:01 UTC), ContentLockDeletionPhase::StartPaymentDrain, ) .await - .unwrap() - .is_none() + .unwrap(), + AdvanceContentLockDeletionPhaseResult::ClaimLost ); assert!( reopened @@ -1895,7 +2497,6 @@ mod tests { job.job_id, "worker-a", first.claim_token, - datetime!(2026-08-12 05:05:01 UTC), Some(ContentLockDeletionFailureCode::StateCorrupt), ) .await @@ -1908,20 +2509,19 @@ mod tests { job.job_id, "worker-b", reclaimed.claim_token, - datetime!(2026-08-12 05:06:00 UTC), ContentLockDeletionPhase::StartPaymentDrain, ) .await .unwrap() - .unwrap(); + .advanced() + .expect("live claim should advance phase"); assert_eq!(advanced.state, ContentLockDeletionState::Queued); assert_eq!(advanced.attempt_count, 0); let final_claim = reopened .claim_next( "worker-c", - datetime!(2026-08-12 05:06:01 UTC), - datetime!(2026-08-12 05:11:00 UTC), + (datetime!(2026-08-12 05:11:00 UTC)) - (datetime!(2026-08-12 05:06:01 UTC)), ) .await .unwrap() @@ -1931,7 +2531,6 @@ mod tests { job.job_id, "worker-c", final_claim.claim_token, - datetime!(2026-08-12 05:07:00 UTC), Some(ContentLockDeletionFailureCode::TombstoneMissing), ) .await @@ -1945,14 +2544,14 @@ mod tests { assert!(matches!( reopened - .prepare_force_deletion(&job.creator, &job.lock_id, NOW) + .prepare_force_deletion(&job.creator, &job.lock_id) .await .unwrap(), PrepareForceDeletionResult::Synchronous(Some(_)) )); assert!(matches!( reopened - .prepare_force_deletion(&job.creator, &job.lock_id, NOW) + .prepare_force_deletion(&job.creator, &job.lock_id) .await .unwrap(), PrepareForceDeletionResult::Synchronous(None) @@ -1988,8 +2587,8 @@ mod tests { repository.insert_job(job).await.unwrap(); let (left, right) = tokio::join!( - repository.claim_next("worker-a", NOW, datetime!(2026-08-12 05:05:00 UTC)), - repository.claim_next("worker-b", NOW, datetime!(2026-08-12 05:05:00 UTC)), + repository.claim_next("worker-a", (datetime!(2026-08-12 05:05:00 UTC)) - (NOW)), + repository.claim_next("worker-b", (datetime!(2026-08-12 05:05:00 UTC)) - (NOW)), ); assert_eq!( usize::from(left.unwrap().is_some()) + usize::from(right.unwrap().is_some()), @@ -2006,13 +2605,16 @@ mod tests { let job = ContentLockDeletionJob::new(Uuid::new_v4(), content_lock(), NOW).unwrap(); repository.insert_job(job.clone()).await.unwrap(); let claimed = repository - .claim_next("graceful-worker", NOW, NOW + time::Duration::minutes(1)) + .claim_next( + "graceful-worker", + (NOW + time::Duration::minutes(1)) - (NOW), + ) .await .unwrap() .unwrap(); let escalated = repository - .prepare_force_deletion(&job.creator, &job.lock_id, NOW) + .prepare_force_deletion(&job.creator, &job.lock_id) .await .unwrap(); let PrepareForceDeletionResult::Active(escalated) = escalated else { @@ -2027,8 +2629,7 @@ mod tests { job.job_id, "graceful-worker", claimed.claim_token, - NOW, - NOW + time::Duration::seconds(1), + (NOW + time::Duration::seconds(1)) - (NOW), ) .await .unwrap(), @@ -2040,29 +2641,22 @@ mod tests { job.job_id, "graceful-worker", claimed.claim_token, - NOW, ContentLockDeletionPhase::StartPaymentDrain, ) .await .unwrap(), - None + AdvanceContentLockDeletionPhaseResult::ClaimLost ); assert_eq!( repository - .finish( - job.job_id, - "graceful-worker", - claimed.claim_token, - NOW, - None, - ) + .finish(job.job_id, "graceful-worker", claimed.claim_token, None,) .await .unwrap(), None ); let force_claim = repository - .claim_next("force-worker", NOW, NOW + time::Duration::minutes(1)) + .claim_next("force-worker", (NOW + time::Duration::minutes(1)) - (NOW)) .await .unwrap() .unwrap(); @@ -2095,7 +2689,7 @@ mod tests { admissions.mark_ready(&task, window).await.unwrap(); let ordinary_claim = ordinary - .claim_next_verification_task("ordinary", NOW, NOW + time::Duration::minutes(5)) + .claim_next_verification_task("ordinary", (NOW + time::Duration::minutes(5)) - (NOW)) .await .unwrap() .unwrap(); @@ -2108,7 +2702,6 @@ mod tests { &ordinary_claim.task.task_id, "ordinary", &ordinary_claim.claim_token, - NOW, ) .await .unwrap() @@ -2125,7 +2718,6 @@ mod tests { ordinary_completed, "ordinary", &ordinary_claim.claim_token, - NOW, ) .await .unwrap() @@ -2133,7 +2725,7 @@ mod tests { ); let withdraw_claim = deletions - .claim_next("deletion", NOW, NOW + time::Duration::minutes(5)) + .claim_next("deletion", (NOW + time::Duration::minutes(5)) - (NOW)) .await .unwrap() .unwrap(); @@ -2142,14 +2734,14 @@ mod tests { job.job_id, "deletion", withdraw_claim.claim_token, - NOW, ContentLockDeletionPhase::StartPaymentDrain, ) .await .unwrap() - .unwrap(); + .advanced() + .expect("live claim should advance phase"); let start_claim = deletions - .claim_next("deletion", NOW, NOW + time::Duration::minutes(5)) + .claim_next("deletion", (NOW + time::Duration::minutes(5)) - (NOW)) .await .unwrap() .unwrap(); @@ -2164,25 +2756,13 @@ mod tests { }; assert!( drains - .store_payment_drain( - job.job_id, - "deletion", - start_claim.claim_token, - NOW, - &summary, - ) + .store_payment_drain(job.job_id, "deletion", start_claim.claim_token, &summary,) .await .unwrap() ); assert!( drains - .store_payment_drain( - job.job_id, - "deletion", - start_claim.claim_token, - NOW, - &summary, - ) + .store_payment_drain(job.job_id, "deletion", start_claim.claim_token, &summary,) .await .unwrap() ); @@ -2192,13 +2772,7 @@ mod tests { }; assert!( drains - .store_payment_drain( - job.job_id, - "deletion", - start_claim.claim_token, - NOW, - &divergent, - ) + .store_payment_drain(job.job_id, "deletion", start_claim.claim_token, &divergent,) .await .is_err() ); @@ -2218,14 +2792,14 @@ mod tests { job.job_id, "deletion", start_claim.claim_token, - NOW, ContentLockDeletionPhase::DrainPayments, ) .await .unwrap() - .unwrap(); + .advanced() + .expect("live claim should advance phase"); let drain_claim = deletions - .claim_next("deletion", NOW, NOW + time::Duration::minutes(5)) + .claim_next("deletion", (NOW + time::Duration::minutes(5)) - (NOW)) .await .unwrap() .unwrap(); @@ -2234,7 +2808,6 @@ mod tests { job.job_id, "deletion", drain_claim.claim_token, - NOW, &task.task_id, ) .await @@ -2246,7 +2819,6 @@ mod tests { job.job_id, "deletion", drain_claim.claim_token, - NOW, &task.task_id, ) .await @@ -2255,7 +2827,7 @@ mod tests { ); assert_eq!( deletions - .prepare_force_deletion(&job.creator, &job.lock_id, NOW) + .prepare_force_deletion(&job.creator, &job.lock_id) .await .unwrap(), PrepareForceDeletionResult::PublicationInProgress @@ -2266,7 +2838,6 @@ mod tests { job.job_id, "deletion", drain_claim.claim_token, - NOW, &task.task_id, PaymentDrainTerminalTransition { status: VerificationTaskStatus::Completed, @@ -2282,7 +2853,6 @@ mod tests { job.job_id, "deletion", drain_claim.claim_token, - NOW, &task.task_id, PaymentDrainTerminalTransition { status: VerificationTaskStatus::Completed, @@ -2303,15 +2873,30 @@ mod tests { .fetch_one(database.pool()) .await .unwrap(); - assert_eq!(final_credential_eligible_at, Some(NOW)); - let issuance_deadline = NOW + time::Duration::minutes(15); + let resolved_at: Option = sqlx::query_scalar( + "SELECT resolved_at FROM content_lock_deletion_task_snapshot + WHERE deletion_job_id = $1 AND verification_task_id = $2", + ) + .bind(job.job_id) + .bind(task.task_id.as_uuid()) + .fetch_one(database.pool()) + .await + .unwrap(); + assert_eq!(final_credential_eligible_at, resolved_at); + let final_access_started_at: time::OffsetDateTime = + sqlx::query_scalar("SELECT clock_timestamp()") + .fetch_one(database.pool()) + .await + .unwrap(); + let issuance_deadline = final_access_started_at + time::Duration::minutes(15); let read_deadline = issuance_deadline + time::Duration::minutes(15); sqlx::query( "UPDATE content_lock_deletion_jobs - SET phase = 'issue_final_credentials' + SET phase = 'issue_final_credentials', claim_expires_at = $2 WHERE job_id = $1", ) .bind(job.job_id) + .bind(read_deadline + time::Duration::minutes(5)) .execute(database.pool()) .await .unwrap(); @@ -2319,31 +2904,32 @@ mod tests { database.pool().clone(), crate::infrastructure::final_credentials::FinalCredentialCipher::new([8; 32]), ); - assert!( - access - .initialize_final_access_windows( - job.job_id, - "deletion", - drain_claim.claim_token, - NOW, - issuance_deadline, - read_deadline, - ) - .await - .unwrap() - ); - assert!( + let initialized = access + .initialize_final_access_windows( + job.job_id, + "deletion", + drain_claim.claim_token, + time::Duration::minutes(15), + time::Duration::minutes(15), + ) + .await + .unwrap(); + assert!(matches!( + initialized, + InitializeFinalAccessWindowsResult::Initialized(_) + )); + assert_eq!( access .initialize_final_access_windows( job.job_id, "deletion", drain_claim.claim_token, - NOW + time::Duration::seconds(1), - issuance_deadline + time::Duration::hours(1), - read_deadline + time::Duration::hours(1), + time::Duration::hours(1), + time::Duration::hours(1), ) .await - .unwrap() + .unwrap(), + initialized ); let persisted_windows: ( Option, @@ -2359,15 +2945,22 @@ mod tests { .fetch_one(database.pool()) .await .unwrap(); + let InitializeFinalAccessWindowsResult::Initialized(windows) = initialized else { + unreachable!() + }; assert_eq!( persisted_windows, - (Some(NOW), Some(issuance_deadline), Some(read_deadline)) + ( + Some(windows.issuance_started_at), + Some(windows.credential_issuance_deadline), + Some(windows.read_deadline), + ) ); let first = access .issue_or_replay_final_credential( &job.creator, &task.submitted_proof_bundle.bundle_id, - NOW, + final_access_started_at, AccessCredential::new("first-final-bearer"), ) .await @@ -2377,15 +2970,15 @@ mod tests { .issue_or_replay_final_credential( &job.creator, &task.submitted_proof_bundle.bundle_id, - NOW + time::Duration::seconds(1), + final_access_started_at + time::Duration::seconds(1), AccessCredential::new("different-retry-candidate"), ) .await .unwrap() .unwrap(); assert_eq!(first.credential, replay.credential); - assert_eq!(first.expires_at, read_deadline); - assert_eq!(replay.expires_at, read_deadline); + assert_eq!(first.expires_at, windows.read_deadline); + assert_eq!(replay.expires_at, windows.read_deadline); let late_replay = access .issue_or_replay_final_credential( &job.creator, @@ -2430,18 +3023,8 @@ mod tests { let final_lookup = AccessCredentialLookupKey::derive(&first.credential); let guarded_path = "/priv/locks.app/content/post.json"; let (first_attempt, second_attempt) = tokio::join!( - access.prepare_deletion_read( - &final_lookup, - guarded_path, - NOW, - NOW + time::Duration::seconds(30), - ), - access.prepare_deletion_read( - &final_lookup, - guarded_path, - NOW, - NOW + time::Duration::seconds(30), - ), + access.prepare_deletion_read(&final_lookup, guarded_path, time::Duration::seconds(30),), + access.prepare_deletion_read(&final_lookup, guarded_path, time::Duration::seconds(30),), ); let first_attempt = first_attempt.unwrap(); let second_attempt = second_attempt.unwrap(); @@ -2454,12 +3037,7 @@ mod tests { assert_eq!(first_claim.resource.path, guarded_path); assert!( access - .prepare_deletion_read( - &final_lookup, - guarded_path, - NOW + time::Duration::seconds(1), - NOW + time::Duration::seconds(31), - ) + .prepare_deletion_read(&final_lookup, guarded_path, time::Duration::seconds(30),) .await .unwrap() .is_none() @@ -2470,7 +3048,7 @@ mod tests { &final_lookup, guarded_path, Uuid::new_v4(), - NOW + time::Duration::seconds(1), + final_access_started_at + time::Duration::seconds(1), ) .await .unwrap() @@ -2481,18 +3059,13 @@ mod tests { &final_lookup, guarded_path, first_token, - NOW + time::Duration::seconds(1), + final_access_started_at + time::Duration::seconds(1), ) .await .unwrap() ); let second_claim = access - .prepare_deletion_read( - &final_lookup, - guarded_path, - NOW + time::Duration::seconds(2), - NOW + time::Duration::seconds(32), - ) + .prepare_deletion_read(&final_lookup, guarded_path, time::Duration::seconds(30)) .await .unwrap() .unwrap(); @@ -2500,98 +3073,223 @@ mod tests { assert_ne!(second_token, first_token); assert!( !access - .consume_deletion_read( - &final_lookup, - guarded_path, - first_token, - NOW + time::Duration::seconds(3), - ) + .consume_deletion_read(&final_lookup, guarded_path, first_token,) .await .unwrap() ); assert!( access - .consume_deletion_read( - &final_lookup, - guarded_path, - second_token, - NOW + time::Duration::seconds(3), + .consume_deletion_read(&final_lookup, guarded_path, second_token,) + .await + .unwrap() + ); + assert!( + access + .prepare_deletion_read(&final_lookup, guarded_path, time::Duration::seconds(30),) + .await + .unwrap() + .is_none() + ); + assert!( + access + .deletion_credential_enrolled(&final_lookup) + .await + .unwrap() + ); + let retained_marker: Option = sqlx::query_scalar( + "SELECT entitlement_publication_claim_token FROM verification_tasks WHERE task_id = $1", + ) + .bind(task.task_id.as_uuid()) + .fetch_one(database.pool()) + .await + .unwrap(); + assert_eq!(retained_marker, None); + + database.cleanup().await; + } + + #[tokio::test] + async fn entitlement_publication_fence_committed_first_blocks_deletion_cutoff() { + let database = TestDatabase::create().await; + let deletions = PostgresContentLockDeletionRepository::new(database.pool().clone()); + let tasks = PostgresVerificationTaskRepository::new(database.pool().clone()); + let ordinary = PostgresVerificationTaskClaimer::new(database.pool().clone()); + let lock = content_lock(); + let task = verification_task(&lock, BundleId::from_bytes([11; 16])); + tasks.insert_verification_task(task).await.unwrap(); + let claim = ordinary + .claim_next_verification_task("ordinary", (NOW + time::Duration::minutes(5)) - (NOW)) + .await + .unwrap() + .unwrap(); + assert!( + ordinary + .begin_claimed_entitlement_publication( + &claim.task.task_id, + "ordinary", + &claim.claim_token, ) .await .unwrap() ); - assert!( - access - .prepare_deletion_read( - &final_lookup, - guarded_path, - NOW + time::Duration::seconds(4), - NOW + time::Duration::seconds(34), - ) - .await - .unwrap() - .is_none() + + let job = ContentLockDeletionJob::new(Uuid::new_v4(), lock, NOW).unwrap(); + assert_eq!( + deletions.insert_job(job.clone()).await, + Err(ApplicationError::ContentLockDeletionInProgress) + ); + assert!( + deletions + .get_job(&job.creator, &job.lock_id) + .await + .unwrap() + .is_none() + ); + + database.cleanup().await; + } + + #[tokio::test] + async fn payment_drain_reclaim_reconciles_external_start_before_local_persistence() { + use crate::infrastructure::postgres::{ + PaykitInvoiceWindow, PostgresPaykitTaskAdmissionRepository, + }; + + let database = TestDatabase::create().await; + let deletions = PostgresContentLockDeletionRepository::new(database.pool().clone()); + let drains = PostgresPaymentDrainRepository::new(database.pool().clone()); + let admissions = PostgresPaykitTaskAdmissionRepository::new(database.pool().clone()); + let entitlements = InMemoryEntitlementRepository::new(); + let lock = content_lock(); + let mut task = verification_task(&lock, BundleId::from_bytes([11; 16])); + task.submitted_proof_bundle.proofs[0].verifier_type = VerifierType::PaykitPayment; + admissions.reserve(task.clone(), 24).await.unwrap(); + admissions + .mark_ready( + &task, + PaykitInvoiceWindow { + invoice_created_at: NOW, + payment_deadline: NOW + time::Duration::hours(24), + }, + ) + .await + .unwrap(); + + let job = ContentLockDeletionJob::new(Uuid::new_v4(), lock, NOW).unwrap(); + deletions.insert_job(job.clone()).await.unwrap(); + let withdraw = deletions + .claim_next("worker-a", (NOW + time::Duration::minutes(5)) - (NOW)) + .await + .unwrap() + .unwrap(); + deletions + .advance_phase( + job.job_id, + "worker-a", + withdraw.claim_token, + ContentLockDeletionPhase::StartPaymentDrain, + ) + .await + .unwrap() + .advanced() + .expect("live claim should advance phase"); + let reclaimed = deletions + .claim_next("worker-b", (NOW + time::Duration::minutes(5)) - (NOW)) + .await + .unwrap() + .unwrap(); + + let summary = PaymentDrainSummary { + status: PaymentDrainStatus::Active, + accepted_count: 1, + terminal_count: 0, + cancellation_enqueued_count: 0, + cleanup_token: PaymentDrainCleanupToken::parse( + "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", + ) + .unwrap(), + }; + let paykit = ReclaimingPaymentDrainClient { + summary, + start_calls: AtomicUsize::new(0), + }; + let lock_resource = PubkyLockResource::new( + job.creator.clone(), + locks_core::ids::ContentLockPath::from_lock_id(job.lock_id.clone()), + ); + paykit.start_payment_drain(&lock_resource).await.unwrap(); + + let use_case = DrainLockPaymentsUseCase::new( + &deletions, + &drains, + &paykit, + &entitlements, + &FixedClock(NOW), + LockServerPubky::from_str("pubky7ir1ttte48bcp4zjychjyscicrwi1j34mtt91ptsafdbjmr8g9eo") + .unwrap(), + 6, ); assert!( - access - .deletion_credential_enrolled(&final_lookup) + use_case + .execute_claimed(reclaimed, "worker-b") .await .unwrap() ); - let retained_marker: Option = sqlx::query_scalar( - "SELECT entitlement_publication_claim_token FROM verification_tasks WHERE task_id = $1", - ) - .bind(task.task_id.as_uuid()) - .fetch_one(database.pool()) - .await - .unwrap(); - assert_eq!(retained_marker, None); - - database.cleanup().await; - } - - #[tokio::test] - async fn entitlement_publication_fence_committed_first_blocks_deletion_cutoff() { - let database = TestDatabase::create().await; - let deletions = PostgresContentLockDeletionRepository::new(database.pool().clone()); - let tasks = PostgresVerificationTaskRepository::new(database.pool().clone()); - let ordinary = PostgresVerificationTaskClaimer::new(database.pool().clone()); - let lock = content_lock(); - let task = verification_task(&lock, BundleId::from_bytes([11; 16])); - tasks.insert_verification_task(task).await.unwrap(); - let claim = ordinary - .claim_next_verification_task("ordinary", NOW, NOW + time::Duration::minutes(5)) - .await - .unwrap() - .unwrap(); + assert_eq!(paykit.start_calls.load(Ordering::SeqCst), 2); assert!( - ordinary - .begin_claimed_entitlement_publication( - &claim.task.task_id, - "ordinary", - &claim.claim_token, - NOW, - ) + drains + .get_payment_drain(job.job_id) .await .unwrap() + .is_some() ); - - let job = ContentLockDeletionJob::new(Uuid::new_v4(), lock, NOW).unwrap(); assert_eq!( - deletions.insert_job(job.clone()).await, - Err(ApplicationError::ContentLockDeletionInProgress) - ); - assert!( deletions .get_job(&job.creator, &job.lock_id) .await .unwrap() - .is_none() + .unwrap() + .phase, + ContentLockDeletionPhase::DrainPayments ); database.cleanup().await; } + struct ReclaimingPaymentDrainClient { + summary: PaymentDrainSummary, + start_calls: AtomicUsize, + } + + #[async_trait] + impl PaymentDrainClient for ReclaimingPaymentDrainClient { + async fn start_payment_drain( + &self, + _lock_resource: &PubkyLockResource, + ) -> Result { + if self.start_calls.fetch_add(1, Ordering::SeqCst) == 0 { + Ok(self.summary.clone()) + } else { + Err(PaymentDrainClientError::Conflict) + } + } + + async fn lookup_payment_drain( + &self, + _lock_resource: &PubkyLockResource, + ) -> Result, PaymentDrainClientError> { + Ok(Some(self.summary.clone())) + } + + async fn payment_request_status( + &self, + _creator: &CreatorPubky, + _bundle_id: &BundleId, + ) -> Result, PaymentDrainClientError> { + unreachable!() + } + } + #[tokio::test] async fn completed_paykit_aggregate_waits_for_locks_confirmation_and_local_terminal_state() { use crate::infrastructure::postgres::{ @@ -2615,7 +3313,7 @@ mod tests { let job = ContentLockDeletionJob::new(Uuid::new_v4(), lock, NOW).unwrap(); deletions.insert_job(job.clone()).await.unwrap(); let withdraw = deletions - .claim_next("deletion", NOW, NOW + time::Duration::minutes(5)) + .claim_next("deletion", (NOW + time::Duration::minutes(5)) - (NOW)) .await .unwrap() .unwrap(); @@ -2624,14 +3322,14 @@ mod tests { job.job_id, "deletion", withdraw.claim_token, - NOW, ContentLockDeletionPhase::StartPaymentDrain, ) .await .unwrap() - .unwrap(); + .advanced() + .expect("live claim should advance phase"); let start = deletions - .claim_next("deletion", NOW, NOW + time::Duration::minutes(5)) + .claim_next("deletion", (NOW + time::Duration::minutes(5)) - (NOW)) .await .unwrap() .unwrap(); @@ -2645,13 +3343,7 @@ mod tests { cleanup_token: token.clone(), }; drains - .store_payment_drain( - job.job_id, - "deletion", - start.claim_token, - NOW, - &initial_summary, - ) + .store_payment_drain(job.job_id, "deletion", start.claim_token, &initial_summary) .await .unwrap(); deletions @@ -2659,14 +3351,14 @@ mod tests { job.job_id, "deletion", start.claim_token, - NOW, ContentLockDeletionPhase::DrainPayments, ) .await .unwrap() - .unwrap(); + .advanced() + .expect("live claim should advance phase"); let claim = deletions - .claim_next("deletion", NOW, NOW + time::Duration::minutes(5)) + .claim_next("deletion", (NOW + time::Duration::minutes(5)) - (NOW)) .await .unwrap() .unwrap(); @@ -2756,19 +3448,20 @@ mod tests { #[tokio::test] async fn concurrent_final_issuers_replay_one_winner() { let database = TestDatabase::create().await; - let (job, task, access) = eligible_final_credential_fixture(&database).await; + let (job, task, access, final_access_started_at) = + eligible_final_credential_fixture(&database).await; let (first, second) = tokio::join!( access.issue_or_replay_final_credential( &job.creator, &task.submitted_proof_bundle.bundle_id, - NOW, + final_access_started_at, AccessCredential::new("concurrent-final-one"), ), access.issue_or_replay_final_credential( &job.creator, &task.submitted_proof_bundle.bundle_id, - NOW, + final_access_started_at, AccessCredential::new("concurrent-final-two"), ), ); @@ -2791,12 +3484,13 @@ mod tests { #[tokio::test] async fn force_revokes_live_final_read_claim() { let database = TestDatabase::create().await; - let (job, task, access) = eligible_final_credential_fixture(&database).await; + let (job, task, access, final_access_started_at) = + eligible_final_credential_fixture(&database).await; let issued = access .issue_or_replay_final_credential( &job.creator, &task.submitted_proof_bundle.bundle_id, - NOW, + final_access_started_at, AccessCredential::new("final-before-force"), ) .await @@ -2811,7 +3505,7 @@ mod tests { .path .clone(); let prepared = access - .prepare_deletion_read(&lookup, &path, NOW, NOW + time::Duration::seconds(30)) + .prepare_deletion_read(&lookup, &path, time::Duration::seconds(30)) .await .unwrap() .unwrap(); @@ -2820,34 +3514,20 @@ mod tests { let force = PostgresContentLockDeletionRepository::new(database.pool().clone()); assert!(matches!( force - .prepare_force_deletion( - &job.creator, - &job.lock_id, - NOW + time::Duration::seconds(1) - ) + .prepare_force_deletion(&job.creator, &job.lock_id,) .await .unwrap(), PrepareForceDeletionResult::Active(_) )); assert!( !access - .consume_deletion_read( - &lookup, - &path, - read_token, - NOW + time::Duration::seconds(2), - ) + .consume_deletion_read(&lookup, &path, read_token,) .await .unwrap() ); assert!( access - .prepare_deletion_read( - &lookup, - &path, - NOW + time::Duration::seconds(2), - NOW + time::Duration::seconds(30), - ) + .prepare_deletion_read(&lookup, &path, time::Duration::seconds(28),) .await .unwrap() .is_none() @@ -2859,7 +3539,8 @@ mod tests { #[tokio::test] async fn final_issuance_waiting_on_snapshot_observes_force_winner() { let database = TestDatabase::create().await; - let (job, task, access) = eligible_final_credential_fixture(&database).await; + let (job, task, access, final_access_started_at) = + eligible_final_credential_fixture(&database).await; let mut blocker = database.pool().begin().await.unwrap(); sqlx::query( "SELECT job_id FROM content_lock_deletion_jobs @@ -2875,7 +3556,7 @@ mod tests { WHERE job_id = $1", ) .bind(job.job_id) - .bind(NOW) + .bind(final_access_started_at) .execute(&mut *blocker) .await .unwrap(); @@ -2888,7 +3569,7 @@ mod tests { .issue_or_replay_final_credential( &issuing_creator, &issuing_bundle, - NOW, + final_access_started_at, AccessCredential::new("must-not-escape-force"), ) .await @@ -2905,7 +3586,8 @@ mod tests { #[tokio::test] async fn phase_advancement_and_successful_finish_cannot_bypass_access_obligations() { let database = TestDatabase::create().await; - let (job, task, access) = eligible_final_credential_fixture(&database).await; + let (job, task, access, final_access_started_at) = + eligible_final_credential_fixture(&database).await; let repository = PostgresContentLockDeletionRepository::new(database.pool().clone()); let claim_token = Uuid::new_v4(); sqlx::query( @@ -2916,7 +3598,7 @@ mod tests { ) .bind(job.job_id) .bind(claim_token) - .bind(NOW + time::Duration::hours(1)) + .bind(final_access_started_at + time::Duration::hours(1)) .execute(database.pool()) .await .unwrap(); @@ -2927,15 +3609,14 @@ mod tests { job.job_id, "worker", claim_token, - NOW, ContentLockDeletionPhase::DrainFinalReads, ) .await, - Err(ApplicationError::InvalidContentLockDeletionState { .. }) + Ok(AdvanceContentLockDeletionPhaseResult::ObligationsPending) )); assert!(matches!( repository - .finish(job.job_id, "worker", claim_token, NOW, None) + .finish(job.job_id, "worker", claim_token, None,) .await, Err(ApplicationError::InvalidContentLockDeletionState { .. }) )); @@ -2944,7 +3625,7 @@ mod tests { .issue_or_replay_final_credential( &job.creator, &task.submitted_proof_bundle.bundle_id, - NOW, + final_access_started_at, AccessCredential::new("phase-obligation-final"), ) .await @@ -2956,15 +3637,18 @@ mod tests { job.job_id, "worker", claim_token, - NOW, ContentLockDeletionPhase::DrainFinalReads, ) .await .unwrap() + .advanced() .is_some() ); let drain_claim = repository - .claim_next("worker", NOW, NOW + time::Duration::hours(1)) + .claim_next( + "worker", + (final_access_started_at + time::Duration::hours(1)) - (final_access_started_at), + ) .await .unwrap() .unwrap(); @@ -2974,11 +3658,10 @@ mod tests { job.job_id, "worker", drain_claim.claim_token, - NOW, ContentLockDeletionPhase::DeleteContent, ) .await, - Err(ApplicationError::InvalidContentLockDeletionState { .. }) + Ok(AdvanceContentLockDeletionPhaseResult::ObligationsPending) )); let lookup = AccessCredentialLookupKey::derive(&issued.credential); let path = job @@ -2989,13 +3672,13 @@ mod tests { .path .clone(); let read = access - .prepare_deletion_read(&lookup, &path, NOW, NOW + time::Duration::seconds(30)) + .prepare_deletion_read(&lookup, &path, time::Duration::seconds(30)) .await .unwrap() .unwrap(); assert!( access - .consume_deletion_read(&lookup, &path, read.claim_token.unwrap(), NOW) + .consume_deletion_read(&lookup, &path, read.claim_token.unwrap(),) .await .unwrap() ); @@ -3005,11 +3688,11 @@ mod tests { job.job_id, "worker", drain_claim.claim_token, - NOW, ContentLockDeletionPhase::DeleteContent, ) .await .unwrap() + .advanced() .is_some() ); @@ -3017,30 +3700,129 @@ mod tests { } #[tokio::test] - async fn drain_payments_phase_requires_durable_completed_aggregate() { + async fn no_paykit_drain_expires_pending_without_creating_paykit_state() { let database = TestDatabase::create().await; let tasks = PostgresVerificationTaskRepository::new(database.pool().clone()); let lock = content_lock(); - let mut task = verification_task(&lock, BundleId::from_bytes([61; 16])); - task.status = VerificationTaskStatus::Completed; - task.started_at = Some(NOW); - task.completed_at = Some(NOW); + let task = verification_task(&lock, BundleId::from_bytes([60; 16])); + let task_id = task.task_id; tasks.insert_verification_task(task).await.unwrap(); let job = ContentLockDeletionJob::new(Uuid::new_v4(), lock, NOW).unwrap(); let repository = PostgresContentLockDeletionRepository::new(database.pool().clone()); repository.insert_job(job.clone()).await.unwrap(); + let claim_token = Uuid::new_v4(); sqlx::query( - "INSERT INTO content_lock_payment_drains ( - deletion_job_id, status, accepted_count, terminal_count, - cancellation_enqueued_count, cleanup_token, created_at, updated_at - ) VALUES ($1, 'active', 1, 0, 0, $2, $3, $3)", + "UPDATE content_lock_deletion_jobs + SET phase = 'drain_payments', state = 'running', claimed_by = 'worker', + claim_token = $2, claim_expires_at = $3 + WHERE job_id = $1", ) .bind(job.job_id) - .bind("AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA") - .bind(NOW) + .bind(claim_token) + .bind(database_now(&database).await + time::Duration::minutes(5)) .execute(database.pool()) .await .unwrap(); + + assert!( + !repository + .expire_unresolved_non_paykit_tasks(job.job_id, "worker", Uuid::new_v4()) + .await + .unwrap() + ); + assert!( + repository + .expire_unresolved_non_paykit_tasks(job.job_id, "worker", claim_token) + .await + .unwrap() + ); + assert!( + repository + .advance_phase( + job.job_id, + "worker", + claim_token, + ContentLockDeletionPhase::DrainExistingCredentials, + ) + .await + .unwrap() + .advanced() + .is_some() + ); + + let task_status: String = + sqlx::query_scalar("SELECT status FROM verification_tasks WHERE task_id = $1") + .bind(task_id.as_uuid()) + .fetch_one(database.pool()) + .await + .unwrap(); + let snapshot_status: Option = sqlx::query_scalar("SELECT resolved_status FROM content_lock_deletion_task_snapshot WHERE deletion_job_id = $1") + .bind(job.job_id).fetch_one(database.pool()).await.unwrap(); + let drain_count: i64 = sqlx::query_scalar( + "SELECT COUNT(*) FROM content_lock_payment_drains WHERE deletion_job_id = $1", + ) + .bind(job.job_id) + .fetch_one(database.pool()) + .await + .unwrap(); + assert_eq!(task_status, "expired"); + assert_eq!(snapshot_status.as_deref(), Some("expired")); + assert_eq!(drain_count, 0); + database.cleanup().await; + } + + #[tokio::test] + async fn no_paykit_drain_rejects_paykit_and_missing_paykit_aggregate_still_blocks() { + let database = TestDatabase::create().await; + let tasks = PostgresVerificationTaskRepository::new(database.pool().clone()); + let lock = content_lock(); + let mut task = verification_task(&lock, BundleId::from_bytes([59; 16])); + task.submitted_proof_bundle.proofs[0].verifier_type = VerifierType::PaykitPayment; + tasks.insert_verification_task(task).await.unwrap(); + let job = ContentLockDeletionJob::new(Uuid::new_v4(), lock, NOW).unwrap(); + let repository = PostgresContentLockDeletionRepository::new(database.pool().clone()); + repository.insert_job(job.clone()).await.unwrap(); + let claim_token = Uuid::new_v4(); + sqlx::query("UPDATE content_lock_deletion_jobs SET phase = 'drain_payments', state = 'running', claimed_by = 'worker', claim_token = $2, claim_expires_at = $3 WHERE job_id = $1") + .bind(job.job_id) + .bind(claim_token) + .bind(database_now(&database).await + time::Duration::minutes(5)) + .execute(database.pool()).await.unwrap(); + assert!(matches!( + repository + .expire_unresolved_non_paykit_tasks(job.job_id, "worker", claim_token) + .await, + Err(ApplicationError::InvalidContentLockDeletionState { .. }) + )); + sqlx::query("UPDATE content_lock_deletion_task_snapshot SET resolved_status = 'expired', resolved_at = $2 WHERE deletion_job_id = $1") + .bind(job.job_id).bind(NOW).execute(database.pool()).await.unwrap(); + assert!(matches!( + repository + .advance_phase( + job.job_id, + "worker", + claim_token, + ContentLockDeletionPhase::DrainExistingCredentials, + ) + .await, + Err(ApplicationError::InvalidContentLockDeletionState { .. }) + )); + database.cleanup().await; + } + + #[tokio::test] + async fn drain_payments_phase_allows_missing_aggregate_for_non_paykit_snapshots() { + let database = TestDatabase::create().await; + let tasks = PostgresVerificationTaskRepository::new(database.pool().clone()); + let lock = content_lock(); + let mut task = verification_task(&lock, BundleId::from_bytes([61; 16])); + task.status = VerificationTaskStatus::Completed; + task.started_at = Some(NOW); + task.completed_at = Some(NOW); + tasks.insert_verification_task(task).await.unwrap(); + let job = ContentLockDeletionJob::new(Uuid::new_v4(), lock, NOW).unwrap(); + let repository = PostgresContentLockDeletionRepository::new(database.pool().clone()); + repository.insert_job(job.clone()).await.unwrap(); let claim_token = Uuid::new_v4(); sqlx::query( "UPDATE content_lock_deletion_jobs @@ -3050,23 +3832,24 @@ mod tests { ) .bind(job.job_id) .bind(claim_token) - .bind(NOW + time::Duration::minutes(5)) + .bind(database_now(&database).await + time::Duration::minutes(5)) .execute(database.pool()) .await .unwrap(); - assert!(matches!( + assert!( repository .advance_phase( job.job_id, "worker", claim_token, - NOW, ContentLockDeletionPhase::DrainExistingCredentials, ) - .await, - Err(ApplicationError::InvalidContentLockDeletionState { .. }) - )); + .await + .unwrap() + .advanced() + .is_some() + ); database.cleanup().await; } @@ -3119,7 +3902,7 @@ mod tests { ) .bind(job.job_id) .bind(claim_token) - .bind(NOW + time::Duration::minutes(5)) + .bind(database_now(&database).await + time::Duration::minutes(5)) .execute(database.pool()) .await .unwrap(); @@ -3130,7 +3913,6 @@ mod tests { job.job_id, "worker", claim_token, - NOW, ContentLockDeletionPhase::DrainExistingCredentials, ) .await, @@ -3143,13 +3925,14 @@ mod tests { #[tokio::test] async fn expired_final_read_claim_does_not_wedge_destructive_phase_and_cannot_consume() { let database = TestDatabase::create().await; - let (job, task, access) = eligible_final_credential_fixture(&database).await; + let (job, task, access, final_access_started_at) = + eligible_final_credential_fixture(&database).await; let repository = PostgresContentLockDeletionRepository::new(database.pool().clone()); let issued = access .issue_or_replay_final_credential( &job.creator, &task.submitted_proof_bundle.bundle_id, - NOW, + final_access_started_at, AccessCredential::new("expires-with-final-read-window"), ) .await @@ -3164,7 +3947,7 @@ mod tests { .path .clone(); let issue_claim = Uuid::new_v4(); - let read_deadline = NOW + time::Duration::minutes(30); + let read_deadline = final_access_started_at + time::Duration::minutes(30); sqlx::query( "UPDATE content_lock_deletion_jobs SET state = 'running', claimed_by = 'worker', claim_token = $2, @@ -3182,32 +3965,60 @@ mod tests { job.job_id, "worker", issue_claim, - NOW, ContentLockDeletionPhase::DrainFinalReads, ) .await .unwrap() - .unwrap(); + .advanced() + .expect("live claim should advance phase"); let drain_claim = repository .claim_next( "worker", - read_deadline - time::Duration::seconds(10), - read_deadline + time::Duration::minutes(5), + (read_deadline + time::Duration::minutes(5)) + - (read_deadline - time::Duration::seconds(10)), ) .await .unwrap() .unwrap(); let read = access - .prepare_deletion_read( - &lookup, - &path, - read_deadline - time::Duration::seconds(10), - read_deadline + time::Duration::minutes(1), - ) + .prepare_deletion_read(&lookup, &path, time::Duration::seconds(70)) .await .unwrap() .unwrap(); let stale_read_token = read.claim_token.unwrap(); + sqlx::query( + "WITH anchor AS (SELECT clock_timestamp() AS at) + UPDATE content_lock_deletion_jobs + SET final_credential_issuance_deadline = final_issuance_started_at + + ((anchor.at - final_issuance_started_at) / 2), + final_read_deadline = anchor.at + FROM anchor + WHERE job_id = $1", + ) + .bind(job.job_id) + .execute(database.pool()) + .await + .unwrap(); + sqlx::query( + "UPDATE content_lock_access_drain_credentials + SET expires_at = clock_timestamp() + WHERE deletion_job_id = $1 AND credential_kind = 'final'", + ) + .bind(job.job_id) + .execute(database.pool()) + .await + .unwrap(); + sqlx::query( + "UPDATE content_lock_access_drain_reads AS read + SET claim_expires_at = clock_timestamp() + FROM content_lock_access_drain_credentials AS credential + WHERE read.credential_id = credential.credential_id + AND credential.deletion_job_id = $1", + ) + .bind(job.job_id) + .execute(database.pool()) + .await + .unwrap(); assert!( repository @@ -3215,16 +4026,16 @@ mod tests { job.job_id, "worker", drain_claim.claim_token, - read_deadline, ContentLockDeletionPhase::DeleteContent, ) .await .unwrap() + .advanced() .is_some() ); assert!( !access - .consume_deletion_read(&lookup, &path, stale_read_token, read_deadline,) + .consume_deletion_read(&lookup, &path, stale_read_token) .await .unwrap() ); @@ -3233,21 +4044,21 @@ mod tests { } #[tokio::test] - async fn unissued_eligible_snapshot_blocks_final_read_transition_after_issuance_deadline() { + async fn unissued_eligible_snapshot_reports_irrecoverable_miss_after_issuance_deadline() { let database = TestDatabase::create().await; - let (job, _task, _access) = eligible_final_credential_fixture(&database).await; + let (job, _task, _access, _final_access_started_at) = + eligible_final_credential_fixture(&database).await; let repository = PostgresContentLockDeletionRepository::new(database.pool().clone()); let claim_token = Uuid::new_v4(); - let after_deadline = NOW + time::Duration::minutes(16); sqlx::query( "UPDATE content_lock_deletion_jobs SET state = 'running', claimed_by = 'worker', claim_token = $2, - claim_expires_at = $3 + claim_expires_at = clock_timestamp() + INTERVAL '5 minutes', + final_credential_issuance_deadline = clock_timestamp() WHERE job_id = $1", ) .bind(job.job_id) .bind(claim_token) - .bind(after_deadline + time::Duration::minutes(5)) .execute(database.pool()) .await .unwrap(); @@ -3258,11 +4069,12 @@ mod tests { job.job_id, "worker", claim_token, - after_deadline, ContentLockDeletionPhase::DrainFinalReads, ) .await, - Err(ApplicationError::InvalidContentLockDeletionState { .. }) + Ok(AdvanceContentLockDeletionPhaseResult::TerminalFailure( + ContentLockDeletionFailureCode::StateCorrupt + )) )); database.cleanup().await; @@ -3316,14 +4128,14 @@ mod tests { ) .bind(job.job_id) .bind(claim_token) - .bind(NOW + time::Duration::minutes(5)) + .bind(database_now(&database).await + time::Duration::minutes(5)) .execute(database.pool()) .await .unwrap(); assert!(matches!( repository - .finish(job.job_id, "worker", claim_token, NOW, None) + .finish(job.job_id, "worker", claim_token, None) .await, Err(ApplicationError::InvalidContentLockDeletionState { .. }) )); @@ -3350,7 +4162,7 @@ mod tests { .unwrap(); assert!(matches!( repository - .finish(job.job_id, "worker", claim_token, NOW, None) + .finish(job.job_id, "worker", claim_token, None) .await, Err(ApplicationError::InvalidContentLockDeletionState { .. }) )); @@ -3365,7 +4177,7 @@ mod tests { .await .unwrap(); repository - .finish(job.job_id, "worker", claim_token, NOW, None) + .finish(job.job_id, "worker", claim_token, None) .await .unwrap(); @@ -3379,14 +4191,14 @@ mod tests { let job = ContentLockDeletionJob::new(Uuid::new_v4(), content_lock(), NOW).unwrap(); repository.insert_job(job.clone()).await.unwrap(); let claimed = repository - .claim_next("worker", NOW, NOW + time::Duration::minutes(5)) + .claim_next("worker", (NOW + time::Duration::minutes(5)) - (NOW)) .await .unwrap() .unwrap(); assert!(matches!( repository - .finish(job.job_id, "worker", claimed.claim_token, NOW, None) + .finish(job.job_id, "worker", claimed.claim_token, None) .await, Err(ApplicationError::InvalidContentLockDeletionState { .. }) )); @@ -3397,8 +4209,18 @@ mod tests { #[tokio::test] async fn issuance_deadline_is_half_open_and_transition_preserves_only_exact_replay() { let database = TestDatabase::create().await; - let (job, task, access) = eligible_final_credential_fixture(&database).await; - let issuance_deadline = NOW + time::Duration::minutes(15); + let (job, task, access, _final_access_started_at) = + eligible_final_credential_fixture(&database).await; + let issuance_deadline: time::OffsetDateTime = sqlx::query_scalar( + "UPDATE content_lock_deletion_jobs + SET final_credential_issuance_deadline = clock_timestamp() + WHERE job_id = $1 + RETURNING final_credential_issuance_deadline", + ) + .bind(job.job_id) + .fetch_one(database.pool()) + .await + .unwrap(); assert!( access @@ -3422,6 +4244,16 @@ mod tests { .unwrap(); assert_eq!(final_count, 0); + let issuance_deadline: time::OffsetDateTime = sqlx::query_scalar( + "UPDATE content_lock_deletion_jobs + SET final_credential_issuance_deadline = clock_timestamp() + INTERVAL '15 minutes' + WHERE job_id = $1 + RETURNING final_credential_issuance_deadline", + ) + .bind(job.job_id) + .fetch_one(database.pool()) + .await + .unwrap(); let issued = access .issue_or_replay_final_credential( &job.creator, @@ -3451,12 +4283,12 @@ mod tests { job.job_id, "worker", claim_token, - issuance_deadline, ContentLockDeletionPhase::DrainFinalReads, ) .await .unwrap() - .unwrap(); + .advanced() + .expect("live claim should advance phase"); let replay = access .issue_or_replay_final_credential( @@ -3474,14 +4306,15 @@ mod tests { } #[tokio::test] - async fn final_read_claim_lease_is_fixed_to_thirty_seconds_at_storage_boundary() { + async fn final_read_claim_lease_is_capped_at_thirty_seconds_by_storage_time() { let database = TestDatabase::create().await; - let (job, task, access) = eligible_final_credential_fixture(&database).await; + let (job, task, access, final_access_started_at) = + eligible_final_credential_fixture(&database).await; let issued = access .issue_or_replay_final_credential( &job.creator, &task.submitted_proof_bundle.bundle_id, - NOW, + final_access_started_at, AccessCredential::new("fixed-storage-lease"), ) .await @@ -3496,11 +4329,13 @@ mod tests { .path .clone(); + let before_claim = database_now(&database).await; access - .prepare_deletion_read(&lookup, &path, NOW, NOW + time::Duration::seconds(1)) + .prepare_deletion_read(&lookup, &path, time::Duration::seconds(70)) .await .unwrap() .unwrap(); + let after_claim = database_now(&database).await; let stored_expiry: time::OffsetDateTime = sqlx::query_scalar( "SELECT read.claim_expires_at FROM content_lock_access_drain_reads AS read @@ -3513,7 +4348,8 @@ mod tests { .fetch_one(database.pool()) .await .unwrap(); - assert_eq!(stored_expiry, NOW + time::Duration::seconds(30)); + assert!(stored_expiry >= before_claim + time::Duration::seconds(30)); + assert!(stored_expiry <= after_claim + time::Duration::seconds(30)); database.cleanup().await; } @@ -3579,6 +4415,7 @@ mod tests { ContentLockDeletionJob, VerificationTaskRecord, PostgresAccessCredentialStore, + time::OffsetDateTime, ) { let tasks = PostgresVerificationTaskRepository::new(database.pool().clone()); let lock = content_lock(); @@ -3590,6 +4427,11 @@ mod tests { .insert_job(job.clone()) .await .unwrap(); + let final_access_started_at: time::OffsetDateTime = + sqlx::query_scalar("SELECT clock_timestamp()") + .fetch_one(database.pool()) + .await + .unwrap(); sqlx::query( "UPDATE content_lock_deletion_task_snapshot SET resolved_status = 'completed', resolved_at = $2, @@ -3597,7 +4439,7 @@ mod tests { WHERE deletion_job_id = $1", ) .bind(job.job_id) - .bind(NOW) + .bind(final_access_started_at) .execute(database.pool()) .await .unwrap(); @@ -3608,9 +4450,9 @@ mod tests { WHERE job_id = $1", ) .bind(job.job_id) - .bind(NOW) - .bind(NOW + time::Duration::minutes(15)) - .bind(NOW + time::Duration::minutes(30)) + .bind(final_access_started_at) + .bind(final_access_started_at + time::Duration::minutes(15)) + .bind(final_access_started_at + time::Duration::minutes(30)) .execute(database.pool()) .await .unwrap(); @@ -3618,7 +4460,14 @@ mod tests { database.pool().clone(), crate::infrastructure::final_credentials::FinalCredentialCipher::new([9; 32]), ); - (job, task, access) + (job, task, access, final_access_started_at) + } + + async fn database_now(database: &TestDatabase) -> time::OffsetDateTime { + sqlx::query_scalar("SELECT clock_timestamp()") + .fetch_one(database.pool()) + .await + .unwrap() } fn verification_task(lock: &ContentLock, bundle_id: BundleId) -> VerificationTaskRecord { diff --git a/locks-service/src/infrastructure/postgres/mod.rs b/locks-service/src/infrastructure/postgres/mod.rs index a347b3f..5be1214 100644 --- a/locks-service/src/infrastructure/postgres/mod.rs +++ b/locks-service/src/infrastructure/postgres/mod.rs @@ -7,6 +7,7 @@ //! adapters or explicit production indexes are designed. pub mod access_credentials; +pub mod content_lock_deletion_action_ownership; pub mod content_lock_deletions; pub mod content_lock_ownership; pub mod creator_authority; @@ -23,6 +24,7 @@ pub mod verification_task_claims; pub mod verification_tasks; pub use access_credentials::PostgresAccessCredentialStore; +pub use content_lock_deletion_action_ownership::PostgresContentLockDeletionActionOwnership; pub use content_lock_deletions::PostgresContentLockDeletionRepository; pub use content_lock_ownership::PostgresContentLockOwnershipRepository; pub use creator_authority::{CreatorAuthoritySecretCipher, PostgresCreatorAuthorityStore}; diff --git a/locks-service/src/infrastructure/postgres/payment_drains.rs b/locks-service/src/infrastructure/postgres/payment_drains.rs index 1649437..99ae437 100644 --- a/locks-service/src/infrastructure/postgres/payment_drains.rs +++ b/locks-service/src/infrastructure/postgres/payment_drains.rs @@ -68,12 +68,11 @@ impl PaymentDrainRepository for PostgresPaymentDrainRepository { deletion_job_id: Uuid, worker_id: &str, claim_token: Uuid, - now: OffsetDateTime, summary: &PaymentDrainSummary, ) -> Result { let counts = summary_counts(summary)?; let mut transaction = self.pool.begin().await.map_err(storage_error)?; - let ownership = lock_deletion_ownership(&mut transaction, deletion_job_id).await?; + let (ownership, now) = lock_deletion_ownership(&mut transaction, deletion_job_id).await?; if !owns_live_drain_claim( ownership.as_ref(), worker_id, @@ -158,12 +157,11 @@ impl PaymentDrainRepository for PostgresPaymentDrainRepository { deletion_job_id: Uuid, worker_id: &str, claim_token: Uuid, - now: OffsetDateTime, summary: &PaymentDrainSummary, ) -> Result { let counts = summary_counts(summary)?; let mut transaction = self.pool.begin().await.map_err(storage_error)?; - let ownership = lock_deletion_ownership(&mut transaction, deletion_job_id).await?; + let (ownership, now) = lock_deletion_ownership(&mut transaction, deletion_job_id).await?; if !owns_live_drain_claim( ownership.as_ref(), worker_id, @@ -227,28 +225,18 @@ impl PaymentDrainRepository for PostgresPaymentDrainRepository { deletion_job_id: Uuid, worker_id: &str, claim_token: Uuid, - now: OffsetDateTime, task_id: &TaskId, ) -> Result, ApplicationError> { let publication_token = Uuid::new_v4(); let mut transaction = self.pool.begin().await.map_err(storage_error)?; - let owns_claim: bool = sqlx::query_scalar( - "SELECT EXISTS ( - SELECT 1 FROM content_lock_deletion_jobs - WHERE job_id = $1 AND state = 'running' AND claimed_by = $2 - AND claim_token = $3 AND claim_expires_at >= $4 - AND phase = 'drain_payments' AND force_requested_at IS NULL - FOR UPDATE - )", - ) - .bind(deletion_job_id) - .bind(worker_id) - .bind(claim_token) - .bind(now) - .fetch_one(&mut *transaction) - .await - .map_err(storage_error)?; - if !owns_claim { + let (ownership, now) = lock_deletion_ownership(&mut transaction, deletion_job_id).await?; + if !owns_live_drain_claim( + ownership.as_ref(), + worker_id, + claim_token, + now, + "drain_payments", + ) { transaction.rollback().await.map_err(storage_error)?; return Ok(None); } @@ -290,7 +278,6 @@ impl PaymentDrainRepository for PostgresPaymentDrainRepository { deletion_job_id: Uuid, worker_id: &str, claim_token: Uuid, - now: OffsetDateTime, task_id: &TaskId, transition: PaymentDrainTerminalTransition, ) -> Result { @@ -307,7 +294,7 @@ impl PaymentDrainRepository for PostgresPaymentDrainRepository { }); } let mut transaction = self.pool.begin().await.map_err(storage_error)?; - let ownership = lock_deletion_ownership(&mut transaction, deletion_job_id).await?; + let (ownership, now) = lock_deletion_ownership(&mut transaction, deletion_job_id).await?; if !owns_live_drain_claim( ownership.as_ref(), worker_id, @@ -415,8 +402,8 @@ impl PaymentDrainRepository for PostgresPaymentDrainRepository { async fn lock_deletion_ownership( transaction: &mut sqlx::Transaction<'_, sqlx::Postgres>, deletion_job_id: Uuid, -) -> Result, ApplicationError> { - sqlx::query_as::<_, DeletionOwnershipRow>( +) -> Result<(Option, OffsetDateTime), ApplicationError> { + let ownership = sqlx::query_as::<_, DeletionOwnershipRow>( "SELECT state, phase, force_requested_at, claimed_by, claim_token, claim_expires_at FROM content_lock_deletion_jobs WHERE job_id = $1 @@ -425,7 +412,12 @@ async fn lock_deletion_ownership( .bind(deletion_job_id) .fetch_optional(&mut **transaction) .await - .map_err(storage_error) + .map_err(storage_error)?; + let winner_time = sqlx::query_scalar("SELECT clock_timestamp()") + .fetch_one(&mut **transaction) + .await + .map_err(storage_error)?; + Ok((ownership, winner_time)) } fn owns_live_drain_claim( @@ -443,7 +435,7 @@ fn owns_live_drain_claim( && ownership.claim_token == Some(claim_token) && ownership .claim_expires_at - .is_some_and(|claim_expires_at| claim_expires_at >= now) + .is_some_and(|claim_expires_at| now < claim_expires_at) }) } @@ -626,7 +618,7 @@ mod tests { }; assert!( repository - .reconcile_payment_drain(job_id, "worker", claim_token, now, &completed) + .reconcile_payment_drain(job_id, "worker", claim_token, &completed) .await .unwrap() ); @@ -644,13 +636,13 @@ mod tests { }; assert!( !repository - .reconcile_payment_drain(job_id, "worker", claim_token, now, &divergent) + .reconcile_payment_drain(job_id, "worker", claim_token, &divergent) .await .unwrap() ); assert!( !repository - .reconcile_payment_drain(job_id, "worker", Uuid::new_v4(), now, &divergent) + .reconcile_payment_drain(job_id, "worker", Uuid::new_v4(), &divergent) .await .unwrap() ); @@ -689,7 +681,7 @@ mod tests { }; assert!( repository - .store_payment_drain(job_id, "worker", claim_token, now, &active) + .store_payment_drain(job_id, "worker", claim_token, &active) .await .unwrap() ); @@ -703,7 +695,7 @@ mod tests { }; assert!( repository - .store_payment_drain(job_id, "worker", claim_token, now, &completed) + .store_payment_drain(job_id, "worker", claim_token, &completed) .await .unwrap() ); @@ -752,7 +744,7 @@ mod tests { }; let stale = tokio::spawn(async move { repository - .store_payment_drain(job_id, "worker", stale_claim_token, now, &summary) + .store_payment_drain(job_id, "worker", stale_claim_token, &summary) .await }); tokio::time::sleep(std::time::Duration::from_millis(50)).await; @@ -813,7 +805,7 @@ mod tests { }; let stale = tokio::spawn(async move { repository - .store_payment_drain(job_id, "worker", stale_claim_token, now, &summary) + .store_payment_drain(job_id, "worker", stale_claim_token, &summary) .await }); tokio::time::sleep(std::time::Duration::from_millis(50)).await; @@ -880,7 +872,7 @@ mod tests { }; let stale = tokio::spawn(async move { repository - .reconcile_payment_drain(job_id, "worker", stale_claim_token, now, &completed) + .reconcile_payment_drain(job_id, "worker", stale_claim_token, &completed) .await }); tokio::time::sleep(std::time::Duration::from_millis(50)).await; @@ -946,7 +938,6 @@ mod tests { job_id, "worker", stale_claim_token, - now, &task_id, PaymentDrainTerminalTransition { status: VerificationTaskStatus::Completed, diff --git a/locks-service/src/infrastructure/postgres/verification_task_claims.rs b/locks-service/src/infrastructure/postgres/verification_task_claims.rs index 5ee74d3..971fbe7 100644 --- a/locks-service/src/infrastructure/postgres/verification_task_claims.rs +++ b/locks-service/src/infrastructure/postgres/verification_task_claims.rs @@ -31,13 +31,22 @@ impl VerificationTaskClaimer for PostgresVerificationTaskClaimer { task_id: &TaskId, worker_id: &str, claim_token: &uuid::Uuid, - now: time::OffsetDateTime, ) -> Result { + let mut transaction = self.pool.begin().await.map_err(storage_error)?; + sqlx::query("SELECT task_id FROM verification_tasks WHERE task_id = $1::uuid FOR UPDATE") + .bind(task_id.to_string()) + .fetch_optional(&mut *transaction) + .await + .map_err(storage_error)?; + let now: time::OffsetDateTime = sqlx::query_scalar("SELECT clock_timestamp()") + .fetch_one(&mut *transaction) + .await + .map_err(storage_error)?; let updated = sqlx::query( "UPDATE verification_tasks SET entitlement_publication_claim_token = $3, updated_at = $4 WHERE task_id = $1::uuid AND status = 'in_progress' - AND claimed_by = $2 AND claim_token = $3 AND claim_expires_at >= $4 + AND claimed_by = $2 AND claim_token = $3 AND claim_expires_at > $4 AND deletion_job_id IS NULL AND NOT EXISTS ( SELECT 1 FROM content_lock_deletion_task_snapshot AS snapshot @@ -48,74 +57,85 @@ impl VerificationTaskClaimer for PostgresVerificationTaskClaimer { .bind(worker_id) .bind(claim_token) .bind(now) - .execute(&self.pool) + .execute(&mut *transaction) .await .map_err(storage_error)?; + transaction.commit().await.map_err(storage_error)?; Ok(updated.rows_affected() == 1) } async fn claim_next_verification_task( &self, worker_id: &str, - now: time::OffsetDateTime, - claim_expires_at: time::OffsetDateTime, + claim_ttl: time::Duration, ) -> Result, ApplicationError> { let claim_token = uuid::Uuid::new_v4(); + let mut transaction = self.pool.begin().await.map_err(storage_error)?; + let candidates: Vec<( + uuid::Uuid, + String, + Option, + Option, + )> = sqlx::query_as( + "SELECT task_id, status, next_attempt_at, claim_expires_at + FROM verification_tasks + WHERE status IN ('pending', 'in_progress') + AND deletion_job_id IS NULL + AND NOT EXISTS ( + SELECT 1 FROM paykit_task_admissions + WHERE verification_task_id = verification_tasks.task_id + AND (ready = FALSE OR payment_in_hours IS NULL OR payment_in_hours <= 0 + OR invoice_created_at IS NULL OR payment_deadline IS NULL + OR invoice_created_at > payment_deadline) + ) + AND NOT EXISTS ( + SELECT 1 FROM content_lock_deletion_task_snapshot AS snapshot + WHERE snapshot.verification_task_id = verification_tasks.task_id + ) + AND creator = split_part(submitted_proof_bundle->>'pubky_lock_resource', '/', 1) + AND bundle_id = submitted_proof_bundle->>'bundle_id' + ORDER BY submitted_at + FOR UPDATE SKIP LOCKED", + ) + .fetch_all(&mut *transaction) + .await + .map_err(storage_error)?; + let now: time::OffsetDateTime = sqlx::query_scalar("SELECT clock_timestamp()") + .fetch_one(&mut *transaction) + .await + .map_err(storage_error)?; + let claim_expires_at = + now.checked_add(claim_ttl) + .ok_or_else(|| ApplicationError::Storage { + message: "verification task claim expiry overflow".to_owned(), + })?; + let Some((task_id, _, _, _)) = candidates.into_iter().find(|(_, status, next, expiry)| { + (status == "pending" && next.is_none_or(|due| due <= now)) + || (status == "in_progress" && expiry.is_some_and(|deadline| deadline <= now)) + }) else { + transaction.commit().await.map_err(storage_error)?; + return Ok(None); + }; let sql = format!( "UPDATE verification_tasks - SET - status = 'in_progress', - claimed_by = $1, - claim_expires_at = $2, - claim_token = $4, - next_attempt_at = NULL, - started_at = COALESCE(started_at, $3), - attempt_count = attempt_count + 1, - updated_at = $3 - WHERE task_id = ( - SELECT task_id - FROM verification_tasks - WHERE ((status = 'pending' - AND (next_attempt_at IS NULL OR next_attempt_at <= $3)) - OR (status = 'in_progress' AND claim_expires_at < $3)) - AND deletion_job_id IS NULL - AND NOT EXISTS ( - SELECT 1 FROM paykit_task_admissions - WHERE verification_task_id = verification_tasks.task_id - AND ( - ready = FALSE - OR payment_in_hours IS NULL - OR payment_in_hours <= 0 - OR invoice_created_at IS NULL - OR payment_deadline IS NULL - OR invoice_created_at > payment_deadline - ) - ) - AND NOT EXISTS ( - SELECT 1 - FROM content_lock_deletion_task_snapshot AS snapshot - WHERE snapshot.verification_task_id = verification_tasks.task_id - ) - AND creator = split_part(submitted_proof_bundle->>'pubky_lock_resource', '/', 1) - AND bundle_id = submitted_proof_bundle->>'bundle_id' - ORDER BY submitted_at - FOR UPDATE SKIP LOCKED - LIMIT 1 - ) - RETURNING {VERIFICATION_TASK_ROW_COLUMNS}" + SET status = 'in_progress', claimed_by = $2, claim_expires_at = $3, + claim_token = $4, next_attempt_at = NULL, + started_at = COALESCE(started_at, $5), attempt_count = attempt_count + 1, + updated_at = $5 + WHERE task_id = $1 + RETURNING {VERIFICATION_TASK_ROW_COLUMNS}" ); let row = sqlx::query_as::<_, VerificationTaskRow>(&sql) + .bind(task_id) .bind(worker_id) .bind(claim_expires_at) - .bind(now) .bind(claim_token) - .fetch_optional(&self.pool) + .bind(now) + .fetch_one(&mut *transaction) .await .map_err(storage_error)?; - - row.map(row_to_task) - .transpose() - .map(|task| task.map(|task| ClaimedVerificationTask { task, claim_token })) + transaction.commit().await.map_err(storage_error)?; + row_to_task(row).map(|task| Some(ClaimedVerificationTask { task, claim_token })) } async fn schedule_verification_task_retry( @@ -123,9 +143,23 @@ impl VerificationTaskClaimer for PostgresVerificationTaskClaimer { task_id: &TaskId, worker_id: &str, claim_token: &uuid::Uuid, - now: time::OffsetDateTime, - next_attempt_at: time::OffsetDateTime, + retry_after: time::Duration, ) -> Result, ApplicationError> { + let mut transaction = self.pool.begin().await.map_err(storage_error)?; + sqlx::query("SELECT task_id FROM verification_tasks WHERE task_id = $1::uuid FOR UPDATE") + .bind(task_id.to_string()) + .fetch_optional(&mut *transaction) + .await + .map_err(storage_error)?; + let now: time::OffsetDateTime = sqlx::query_scalar("SELECT clock_timestamp()") + .fetch_one(&mut *transaction) + .await + .map_err(storage_error)?; + let next_attempt_at = + now.checked_add(retry_after) + .ok_or_else(|| ApplicationError::Storage { + message: "verification task retry time overflow".to_owned(), + })?; let sql = format!( "UPDATE verification_tasks SET @@ -143,7 +177,7 @@ impl VerificationTaskClaimer for PostgresVerificationTaskClaimer { AND status = 'in_progress' AND claimed_by = $2 AND claim_token = $3 - AND claim_expires_at >= $4 + AND claim_expires_at > $4 AND deletion_job_id IS NULL AND NOT EXISTS ( SELECT 1 @@ -158,11 +192,12 @@ impl VerificationTaskClaimer for PostgresVerificationTaskClaimer { .bind(claim_token) .bind(now) .bind(next_attempt_at) - .fetch_optional(&self.pool) + .fetch_optional(&mut *transaction) .await .map_err(storage_error)?; - - row.map(row_to_task).transpose() + let result = row.map(row_to_task).transpose()?; + transaction.commit().await.map_err(storage_error)?; + Ok(result) } async fn persist_claimed_verification_task_transition( @@ -170,7 +205,6 @@ impl VerificationTaskClaimer for PostgresVerificationTaskClaimer { task: VerificationTaskRecord, worker_id: &str, claim_token: &uuid::Uuid, - now: time::OffsetDateTime, ) -> Result, ApplicationError> { if !matches!( task.status, @@ -182,6 +216,16 @@ impl VerificationTaskClaimer for PostgresVerificationTaskClaimer { message: "claimed task transition must be terminal".to_owned(), }); } + let mut transaction = self.pool.begin().await.map_err(storage_error)?; + sqlx::query("SELECT task_id FROM verification_tasks WHERE task_id = $1::uuid FOR UPDATE") + .bind(task.task_id.to_string()) + .fetch_optional(&mut *transaction) + .await + .map_err(storage_error)?; + let now: time::OffsetDateTime = sqlx::query_scalar("SELECT clock_timestamp()") + .fetch_one(&mut *transaction) + .await + .map_err(storage_error)?; let sql = format!( "UPDATE verification_tasks SET status = $5, @@ -199,7 +243,7 @@ impl VerificationTaskClaimer for PostgresVerificationTaskClaimer { AND status = 'in_progress' AND claimed_by = $2 AND claim_token = $3 - AND claim_expires_at >= $4 + AND claim_expires_at > $4 AND deletion_job_id IS NULL AND NOT EXISTS ( SELECT 1 @@ -217,11 +261,12 @@ impl VerificationTaskClaimer for PostgresVerificationTaskClaimer { .bind(task.started_at) .bind(task.completed_at) .bind(task.failure_message) - .fetch_optional(&self.pool) + .fetch_optional(&mut *transaction) .await .map_err(storage_error)?; - - row.map(row_to_task).transpose() + let result = row.map(row_to_task).transpose()?; + transaction.commit().await.map_err(storage_error)?; + Ok(result) } } @@ -270,7 +315,7 @@ mod tests { .await .unwrap(); let claim = claimer - .claim_next_verification_task("worker-a", NOW, CLAIM_EXPIRES_AT) + .claim_next_verification_task("worker-a", (CLAIM_EXPIRES_AT) - (NOW)) .await .unwrap() .unwrap(); @@ -287,7 +332,7 @@ mod tests { let claim_token = claim.claim_token; let publication = tokio::spawn(async move { waiting_claimer - .begin_claimed_entitlement_publication(&task_id, "worker-a", &claim_token, NOW) + .begin_claimed_entitlement_publication(&task_id, "worker-a", &claim_token) .await }); tokio::time::sleep(std::time::Duration::from_millis(25)).await; @@ -333,7 +378,7 @@ mod tests { .await .unwrap(); let first = claimer - .claim_next_verification_task("worker-a", NOW, CLAIM_EXPIRES_AT) + .claim_next_verification_task("worker-a", (CLAIM_EXPIRES_AT) - (NOW)) .await .unwrap() .unwrap(); @@ -343,7 +388,6 @@ mod tests { &pending.task_id, "worker-a", &first.claim_token, - NOW, ) .await .unwrap() @@ -354,8 +398,7 @@ mod tests { &pending.task_id, "worker-a", &first.claim_token, - NOW, - retry_at, + (retry_at) - (NOW), ) .await .unwrap() @@ -372,11 +415,11 @@ mod tests { .unwrap(); assert_eq!(retained, Some(first.claim_token)); + mark_retry_due(database.pool(), &pending.task_id).await; let second = claimer .claim_next_verification_task( "worker-b", - retry_at, - retry_at + time::Duration::minutes(5), + (retry_at + time::Duration::minutes(5)) - (retry_at), ) .await .unwrap() @@ -387,7 +430,6 @@ mod tests { &pending.task_id, "worker-b", &second.claim_token, - retry_at, ) .await .unwrap() @@ -401,7 +443,6 @@ mod tests { completed, "worker-b", &second.claim_token, - retry_at, ) .await .unwrap() @@ -442,15 +483,21 @@ mod tests { .await .unwrap(); + let before_claim = database_time(database.pool()).await; let claimed = claimer - .claim_next_verification_task("worker-a", NOW, CLAIM_EXPIRES_AT) + .claim_next_verification_task("worker-a", (CLAIM_EXPIRES_AT) - (NOW)) .await .unwrap() .expect("oldest pending task is claimed"); assert_eq!(claimed.task.task_id, older.task_id); assert_eq!(claimed.task.status, VerificationTaskStatus::InProgress); - assert_eq!(claimed.task.started_at, Some(NOW)); + let after_claim = database_time(database.pool()).await; + assert!( + claimed.task.started_at.is_some_and(|started_at| { + before_claim <= started_at && started_at <= after_claim + }) + ); database.cleanup().await; } @@ -472,7 +519,7 @@ mod tests { assert!(first.requires_paykit); assert!( claimer - .claim_next_verification_task("worker-a", NOW, CLAIM_EXPIRES_AT) + .claim_next_verification_task("worker-a", (CLAIM_EXPIRES_AT) - (NOW)) .await .unwrap() .is_none() @@ -507,7 +554,7 @@ mod tests { assert_eq!(ready_replay.invoice_window, Some(invoice_window)); assert_eq!( claimer - .claim_next_verification_task("worker-a", NOW, CLAIM_EXPIRES_AT) + .claim_next_verification_task("worker-a", (CLAIM_EXPIRES_AT) - (NOW)) .await .unwrap() .unwrap() @@ -554,7 +601,7 @@ mod tests { let claimer = PostgresVerificationTaskClaimer::new(database.pool().clone()); assert!( claimer - .claim_next_verification_task("worker-a", NOW, CLAIM_EXPIRES_AT) + .claim_next_verification_task("worker-a", (CLAIM_EXPIRES_AT) - (NOW)) .await .unwrap() .is_none() @@ -587,7 +634,7 @@ mod tests { assert_eq!( claimer - .claim_next_verification_task("worker-a", NOW, CLAIM_EXPIRES_AT) + .claim_next_verification_task("worker-a", (CLAIM_EXPIRES_AT) - (NOW)) .await .unwrap(), None @@ -616,7 +663,7 @@ mod tests { mark_claim_expired(database.pool(), &in_progress.task_id).await; let reclaimed = claimer - .claim_next_verification_task("worker-b", NOW, CLAIM_EXPIRES_AT) + .claim_next_verification_task("worker-b", (CLAIM_EXPIRES_AT) - (NOW)) .await .unwrap() .expect("expired in-progress task is reclaimed"); @@ -649,7 +696,7 @@ mod tests { assert_eq!( claimer - .claim_next_verification_task("worker-b", NOW, CLAIM_EXPIRES_AT) + .claim_next_verification_task("worker-b", (CLAIM_EXPIRES_AT) - (NOW)) .await .unwrap(), None @@ -675,8 +722,8 @@ mod tests { .unwrap(); let (claim_a, claim_b) = tokio::join!( - claimer_a.claim_next_verification_task("worker-a", NOW, CLAIM_EXPIRES_AT), - claimer_b.claim_next_verification_task("worker-b", NOW, CLAIM_EXPIRES_AT), + claimer_a.claim_next_verification_task("worker-a", (CLAIM_EXPIRES_AT) - (NOW)), + claimer_b.claim_next_verification_task("worker-b", (CLAIM_EXPIRES_AT) - (NOW)), ); let claimed = [claim_a.unwrap(), claim_b.unwrap()]; @@ -708,16 +755,16 @@ mod tests { .await .unwrap(); let first = claimer - .claim_next_verification_task("worker-a", NOW, CLAIM_EXPIRES_AT) + .claim_next_verification_task("worker-a", (CLAIM_EXPIRES_AT) - (NOW)) .await .unwrap() .unwrap(); - let reclaimed_at = CLAIM_EXPIRES_AT + time::Duration::milliseconds(1); + expire_claim(database.pool(), &first.task.task_id).await; + let reclaimed_at = database_time(database.pool()).await; let second = claimer .claim_next_verification_task( "worker-a", - reclaimed_at, - reclaimed_at + time::Duration::minutes(5), + (reclaimed_at + time::Duration::minutes(5)) - (reclaimed_at), ) .await .unwrap() @@ -730,8 +777,7 @@ mod tests { &pending.task_id, "worker-a", &first.claim_token, - reclaimed_at, - reclaimed_at + time::Duration::seconds(10), + (reclaimed_at + time::Duration::seconds(10)) - (reclaimed_at), ) .await .unwrap(), @@ -743,8 +789,7 @@ mod tests { &pending.task_id, "worker-a", &second.claim_token, - reclaimed_at, - reclaimed_at + time::Duration::seconds(10), + (reclaimed_at + time::Duration::seconds(10)) - (reclaimed_at), ) .await .unwrap() @@ -766,16 +811,16 @@ mod tests { ); repository.insert_verification_task(pending).await.unwrap(); let first = claimer - .claim_next_verification_task("worker-a", NOW, CLAIM_EXPIRES_AT) + .claim_next_verification_task("worker-a", (CLAIM_EXPIRES_AT) - (NOW)) .await .unwrap() .unwrap(); - let reclaimed_at = CLAIM_EXPIRES_AT + time::Duration::milliseconds(1); + expire_claim(database.pool(), &first.task.task_id).await; + let reclaimed_at = database_time(database.pool()).await; let second = claimer .claim_next_verification_task( "worker-a", - reclaimed_at, - reclaimed_at + time::Duration::minutes(5), + (reclaimed_at + time::Duration::minutes(5)) - (reclaimed_at), ) .await .unwrap() @@ -806,7 +851,6 @@ mod tests { terminal, "worker-a", &first.claim_token, - reclaimed_at, ) .await .unwrap(), @@ -819,7 +863,6 @@ mod tests { completed.clone(), "worker-a", &second.claim_token, - reclaimed_at, ) .await .unwrap(), @@ -844,7 +887,7 @@ mod tests { .await .unwrap(); let claim = claimer - .claim_next_verification_task("worker-a", NOW, CLAIM_EXPIRES_AT) + .claim_next_verification_task("worker-a", (CLAIM_EXPIRES_AT) - (NOW)) .await .unwrap() .expect("pending task is claimed"); @@ -856,21 +899,7 @@ mod tests { &pending.task_id, "worker-b", &claim.claim_token, - NOW, - next_attempt_at, - ) - .await - .unwrap(), - None - ); - assert_eq!( - claimer - .schedule_verification_task_retry( - &pending.task_id, - "worker-a", - &claim.claim_token, - CLAIM_EXPIRES_AT + time::Duration::milliseconds(1), - next_attempt_at, + (next_attempt_at) - (NOW), ) .await .unwrap(), @@ -881,8 +910,7 @@ mod tests { &pending.task_id, "worker-a", &claim.claim_token, - NOW, - next_attempt_at, + (next_attempt_at) - (NOW), ) .await .unwrap() @@ -896,19 +924,18 @@ mod tests { claimer .claim_next_verification_task( "worker-b", - next_attempt_at - time::Duration::milliseconds(1), - CLAIM_EXPIRES_AT, + (CLAIM_EXPIRES_AT) - (next_attempt_at - time::Duration::milliseconds(1)), ) .await .unwrap(), None ); + mark_retry_due(database.pool(), &pending.task_id).await; assert!( claimer .claim_next_verification_task( "worker-b", - next_attempt_at, - CLAIM_EXPIRES_AT + time::Duration::seconds(10), + (CLAIM_EXPIRES_AT + time::Duration::seconds(10)) - (next_attempt_at), ) .await .unwrap() @@ -930,11 +957,10 @@ mod tests { async fn mark_claim_expired(pool: &sqlx::PgPool, task_id: &TaskId) { sqlx::query( "UPDATE verification_tasks - SET claimed_by = 'worker-a', claim_expires_at = $2 + SET claimed_by = 'worker-a', claim_expires_at = clock_timestamp() WHERE task_id = $1::uuid", ) .bind(task_id.to_string()) - .bind(datetime!(2026-05-29 12:05:00 UTC)) .execute(pool) .await .expect("mark claim expired"); @@ -943,16 +969,47 @@ mod tests { async fn mark_claim_active(pool: &sqlx::PgPool, task_id: &TaskId) { sqlx::query( "UPDATE verification_tasks - SET claimed_by = 'worker-a', claim_expires_at = $2 + SET claimed_by = 'worker-a', + claim_expires_at = clock_timestamp() + INTERVAL '5 minutes' WHERE task_id = $1::uuid", ) .bind(task_id.to_string()) - .bind(datetime!(2026-05-29 12:11:00 UTC)) .execute(pool) .await .expect("mark claim active"); } + async fn expire_claim(pool: &sqlx::PgPool, task_id: &TaskId) { + sqlx::query( + "UPDATE verification_tasks + SET claim_expires_at = clock_timestamp() + WHERE task_id = $1::uuid", + ) + .bind(task_id.to_string()) + .execute(pool) + .await + .expect("expire claim at the database clock boundary"); + } + + async fn mark_retry_due(pool: &sqlx::PgPool, task_id: &TaskId) { + sqlx::query( + "UPDATE verification_tasks + SET next_attempt_at = clock_timestamp() + WHERE task_id = $1::uuid", + ) + .bind(task_id.to_string()) + .execute(pool) + .await + .expect("make retry due at the database clock boundary"); + } + + async fn database_time(pool: &sqlx::PgPool) -> time::OffsetDateTime { + sqlx::query_scalar("SELECT clock_timestamp()") + .fetch_one(pool) + .await + .expect("sample database clock") + } + fn terminal_task(task_id: &str, status: VerificationTaskStatus) -> VerificationTaskRecord { let started_at = datetime!(2026-05-29 12:00:00 UTC); let in_progress = task(task_id, VerificationTaskStatus::Pending, started_at) diff --git a/locks-service/src/infrastructure/pubky/content_lock_tombstones.rs b/locks-service/src/infrastructure/pubky/content_lock_tombstones.rs new file mode 100644 index 0000000..5115e9a --- /dev/null +++ b/locks-service/src/infrastructure/pubky/content_lock_tombstones.rs @@ -0,0 +1,111 @@ +use async_trait::async_trait; +use locks_core::content_lock_deletion::ContentLockDeletionTombstone; +use locks_core::ids::{ContentLockPath, CreatorPubky}; +use locks_core::lock_policy::ContentLock; + +use crate::application::errors::ApplicationError; +use crate::application::ports::content_lock_tombstone::{ + canonical_tombstone_bytes, classify_tombstone_bytes, +}; +use crate::application::ports::{ContentLockTombstoneRepository, TombstoneReadback}; +use crate::infrastructure::pubky::storage_client::PubkyHomeserverStorageClient; + +/// Pubky homeserver adapter for exact public content-lock tombstone bytes. +/// +/// Pubky 0.9.3 cannot condition a PUT on the bytes read below. The pre-write comparison protects +/// already-visible replacements and crash/reclaim replay, but an out-of-band replacement can race +/// between GET and PUT and be overwritten. This accepted limitation is documented in the active +/// graceful-deletion plan and public API reference; this adapter must not be described as CAS. +#[derive(Debug)] +pub struct PubkyContentLockTombstoneRepository { + client: C, +} + +impl PubkyContentLockTombstoneRepository { + pub fn new(client: C) -> Self { + Self { client } + } + + pub fn client(&self) -> &C { + &self.client + } +} + +#[async_trait] +impl ContentLockTombstoneRepository for PubkyContentLockTombstoneRepository +where + C: PubkyHomeserverStorageClient, +{ + async fn withdraw_content_lock( + &self, + creator: CreatorPubky, + content_lock_path: ContentLockPath, + frozen_original: &ContentLock, + tombstone: &ContentLockDeletionTombstone, + ) -> Result { + let tombstone_bytes = canonical_tombstone_bytes(tombstone)?; + let original_bytes = + frozen_original + .canonical_json_bytes() + .map_err(|error| ApplicationError::Storage { + message: format!("failed to serialize frozen content lock: {error}"), + })?; + let actual = self + .client + .get_bytes_as_creator(&creator, &content_lock_path.to_string()) + .await?; + match actual.as_ref().map(|resource| resource.bytes.as_slice()) { + Some(actual) if actual == tombstone_bytes => return Ok(TombstoneReadback::Exact), + Some(actual) if actual == original_bytes => {} + None => return Ok(TombstoneReadback::Missing), + Some(_) => return Ok(TombstoneReadback::Replaced), + } + self.client + .put_bytes_as_creator( + &creator, + &content_lock_path.to_string(), + tombstone_bytes, + "application/json", + ) + .await?; + self.read_tombstone(&creator, &content_lock_path, tombstone) + .await + } + + async fn read_tombstone( + &self, + creator: &CreatorPubky, + content_lock_path: &ContentLockPath, + expected: &ContentLockDeletionTombstone, + ) -> Result { + let expected = canonical_tombstone_bytes(expected)?; + let actual = self + .client + .get_bytes_as_creator(creator, &content_lock_path.to_string()) + .await?; + Ok(classify_tombstone_bytes( + actual.as_ref().map(|resource| resource.bytes.as_slice()), + &expected, + )) + } + + async fn force_delete_content_lock_and_verify_absent( + &self, + creator: &CreatorPubky, + content_lock_path: &ContentLockPath, + ) -> Result<(), ApplicationError> { + let path = content_lock_path.to_string(); + self.client.delete_as_creator(creator, &path).await?; + if self + .client + .get_bytes_as_creator(creator, &path) + .await? + .is_some() + { + return Err(ApplicationError::Storage { + message: "forced public content lock deletion did not reach absence".to_owned(), + }); + } + Ok(()) + } +} diff --git a/locks-service/src/infrastructure/pubky/mod.rs b/locks-service/src/infrastructure/pubky/mod.rs index 49c82c8..289d6d4 100644 --- a/locks-service/src/infrastructure/pubky/mod.rs +++ b/locks-service/src/infrastructure/pubky/mod.rs @@ -1,3 +1,4 @@ +pub mod content_lock_tombstones; pub mod content_locks; pub mod entitlements; pub mod legacy_connect_flow; @@ -6,6 +7,7 @@ pub mod lock_service_pointers; pub mod priv_resources; pub mod storage_client; +pub use content_lock_tombstones::PubkyContentLockTombstoneRepository; pub use content_locks::PubkyContentLockRepository; pub use entitlements::PubkyEntitlementRepository; pub use legacy_connect_flow::{ diff --git a/locks-service/src/infrastructure/pubky/priv_resources.rs b/locks-service/src/infrastructure/pubky/priv_resources.rs index 2e31ec3..803d021 100644 --- a/locks-service/src/infrastructure/pubky/priv_resources.rs +++ b/locks-service/src/infrastructure/pubky/priv_resources.rs @@ -138,7 +138,10 @@ fn record_from_resource( #[cfg(test)] mod tests { use std::str::FromStr; - use std::sync::Mutex; + use std::sync::{ + Mutex, + atomic::{AtomicBool, Ordering}, + }; use async_trait::async_trait; use locks_core::ids::{CreatorPubky, GuardedResourceHash}; @@ -334,6 +337,7 @@ mod tests { bytes_read: Mutex>, last_bytes: Mutex>>, operations: Mutex>, + fail_get: AtomicBool, } impl FakeStorageClient { @@ -394,6 +398,11 @@ mod tests { creator: &CreatorPubky, path: &str, ) -> Result, ApplicationError> { + if self.fail_get.load(Ordering::SeqCst) { + return Err(ApplicationError::InvalidGuardedResource { + message: "storage read failed".to_owned(), + }); + } self.operations .lock() .unwrap() @@ -410,6 +419,7 @@ mod tests { .lock() .unwrap() .push(format!("delete {creator} {path}")); + *self.bytes_read.lock().unwrap() = None; Ok(()) } } diff --git a/locks-service/src/infrastructure/verifiers/paykit_payment.rs b/locks-service/src/infrastructure/verifiers/paykit_payment.rs index a71875a..64ce80a 100644 --- a/locks-service/src/infrastructure/verifiers/paykit_payment.rs +++ b/locks-service/src/infrastructure/verifiers/paykit_payment.rs @@ -76,7 +76,7 @@ where .client .transaction_status(&request.creator, &request.bundle_id) .await - .map_err(|_| ApplicationError::VerificationPending)?; + .map_err(|_| ApplicationError::VerificationDependencyUnavailable)?; if !payment_status_satisfies(status, self.minimum_confirmations) { return Err(ApplicationError::VerificationPending); } @@ -230,12 +230,12 @@ mod tests { } #[tokio::test] - async fn status_client_errors_leave_task_pending() { + async fn status_client_errors_remain_distinct_from_healthy_pending() { let verifier = PaykitPaymentVerifier::new(FakeStatusClient::error(), 0); assert_eq!( verifier.verify(request()).await, - Err(ApplicationError::VerificationPending) + Err(ApplicationError::VerificationDependencyUnavailable) ); } diff --git a/locks-service/tests/content_lock_deletion_executor.rs b/locks-service/tests/content_lock_deletion_executor.rs new file mode 100644 index 0000000..9ed8947 --- /dev/null +++ b/locks-service/tests/content_lock_deletion_executor.rs @@ -0,0 +1,17 @@ +use locks_service::application::use_cases::execute_content_lock_deletion_phase::{ + ContentLockDeletionPhaseExecutor, DeletionPhaseExecutionOutcome, +}; + +#[test] +fn deletion_phase_executor_exposes_closed_outcomes() { + let _ = std::mem::size_of::>(); + let outcomes = [ + DeletionPhaseExecutionOutcome::Progressed, + DeletionPhaseExecutionOutcome::Deferred, + DeletionPhaseExecutionOutcome::ClaimLost, + DeletionPhaseExecutionOutcome::TerminalFailed, + DeletionPhaseExecutionOutcome::TransientDependencyFailure, + DeletionPhaseExecutionOutcome::FatalFailure, + ]; + assert_eq!(outcomes.len(), 6); +} diff --git a/locks-service/tests/content_lock_deletions.rs b/locks-service/tests/content_lock_deletions.rs index 8224055..9d315d6 100644 --- a/locks-service/tests/content_lock_deletions.rs +++ b/locks-service/tests/content_lock_deletions.rs @@ -12,17 +12,21 @@ use locks_service::{ application::{ models::{ AccessCredential, AccessCredentialLookupKey, AccessCredentialRecord, - ContentLockDeletionFailureCode, ContentLockDeletionJob, ContentLockDeletionPhase, - ContentLockDeletionState, PrepareForceDeletionResult, VerificationTaskRecord, - VerificationTaskStatus, + AdvanceContentLockDeletionPhaseResult, ContentLockDeletionFailureCode, + ContentLockDeletionJob, ContentLockDeletionPhase, ContentLockDeletionState, + FinalAccessWindows, InitializeFinalAccessWindowsResult, PrepareForceDeletionResult, + VerificationTaskRecord, VerificationTaskStatus, }, ports::{ - AccessCredentialStore, Clock, ContentLockDeletionRepository, VerificationTaskClaimer, - VerificationTaskRepository, + AccessCredentialStore, Clock, ContentLockDeletionActionAcquireResult, + ContentLockDeletionActionClaim, ContentLockDeletionActionOwnership, + ContentLockDeletionRepository, VerificationTaskClaimer, VerificationTaskRepository, }, + use_cases::no_paykit_deletion_drain::NoPaykitDeletionDrainUseCase, }, infrastructure::memory::{ access_credentials::InMemoryAccessCredentialStore, + content_lock_deletion_action_ownership::InMemoryContentLockDeletionActionOwnership, content_lock_deletions::InMemoryContentLockDeletionRepository, verification_task_claims::InMemoryVerificationTaskClaimer, verification_task_deletion_fence::InMemoryVerificationTaskDeletionFence, @@ -33,6 +37,63 @@ use serde_json::json; use time::macros::datetime; use uuid::Uuid; +struct PausingVerificationTaskRepository { + inner: Arc, + pause_next_get: std::sync::atomic::AtomicBool, + get_entered: tokio::sync::Notify, + release_get: tokio::sync::Notify, +} + +impl PausingVerificationTaskRepository { + fn new(inner: Arc) -> Self { + Self { + inner, + pause_next_get: std::sync::atomic::AtomicBool::new(true), + get_entered: tokio::sync::Notify::new(), + release_get: tokio::sync::Notify::new(), + } + } +} + +#[async_trait::async_trait] +impl VerificationTaskRepository for PausingVerificationTaskRepository { + async fn insert_verification_task( + &self, + task: VerificationTaskRecord, + ) -> Result<(), locks_service::application::errors::ApplicationError> { + self.inner.insert_verification_task(task).await + } + + async fn update_verification_task( + &self, + task: VerificationTaskRecord, + ) -> Result<(), locks_service::application::errors::ApplicationError> { + self.inner.update_verification_task(task).await + } + + async fn get_verification_task( + &self, + task_id: &TaskId, + ) -> Result, locks_service::application::errors::ApplicationError> + { + if self + .pause_next_get + .swap(false, std::sync::atomic::Ordering::SeqCst) + { + self.get_entered.notify_one(); + self.release_get.notified().await; + } + self.inner.get_verification_task(task_id).await + } + + async fn delete_verification_task( + &self, + task_id: &TaskId, + ) -> Result<(), locks_service::application::errors::ApplicationError> { + self.inner.delete_verification_task(task_id).await + } +} + const CREATOR: &str = "pubkytkrq8zmwb8a3m9k15csu3q17qmfgqnp9dskbrg9uq1rydpyxp7qy"; const NOW: time::OffsetDateTime = datetime!(2026-08-12 05:00:00 UTC); const LEASE_END: time::OffsetDateTime = datetime!(2026-08-12 05:05:00 UTC); @@ -46,6 +107,104 @@ impl Clock for FixedClock { } } +#[derive(Debug)] +struct MutableClock(std::sync::Mutex); + +impl MutableClock { + fn new(now: time::OffsetDateTime) -> Self { + Self(std::sync::Mutex::new(now)) + } + + fn set(&self, now: time::OffsetDateTime) { + *self.0.lock().unwrap() = now; + } +} + +impl Clock for MutableClock { + fn now(&self) -> time::OffsetDateTime { + *self.0.lock().unwrap() + } +} + +#[tokio::test] +async fn in_memory_action_ownership_is_exclusive_and_reacquirable() { + let repository = Arc::new( + InMemoryContentLockDeletionRepository::with_verification_task_fence(Arc::new( + InMemoryVerificationTaskDeletionFence::with_clock(Arc::new(FixedClock(NOW))), + )), + ); + let job = ContentLockDeletionJob::new(Uuid::new_v4(), content_lock(), NOW).unwrap(); + repository.insert_job(job).await.unwrap(); + let claimed = repository + .claim_next("worker", time::Duration::minutes(5)) + .await + .unwrap() + .unwrap(); + let ownership = InMemoryContentLockDeletionActionOwnership::new(repository); + let request = || ContentLockDeletionActionClaim { + job_id: claimed.job.job_id, + worker_id: "worker", + claim_token: claimed.claim_token, + expected_phase: claimed.job.phase, + force: false, + }; + + let ContentLockDeletionActionAcquireResult::Acquired(first) = + ownership.try_acquire(request()).await.unwrap() + else { + panic!("live claim must acquire") + }; + assert!(matches!( + ownership.try_acquire(request()).await.unwrap(), + ContentLockDeletionActionAcquireResult::Busy + )); + + first.release().await.unwrap(); + let ContentLockDeletionActionAcquireResult::Acquired(reacquired) = + ownership.try_acquire(request()).await.unwrap() + else { + panic!("released claim must reacquire") + }; + drop(reacquired); + assert!(matches!( + ownership.try_acquire(request()).await.unwrap(), + ContentLockDeletionActionAcquireResult::Acquired(_) + )); +} + +#[tokio::test] +async fn in_memory_action_ownership_rejects_expired_claim_at_half_open_boundary() { + let clock = Arc::new(MutableClock::new(NOW)); + let repository = Arc::new( + InMemoryContentLockDeletionRepository::with_verification_task_fence(Arc::new( + InMemoryVerificationTaskDeletionFence::with_clock(clock.clone()), + )), + ); + let job = ContentLockDeletionJob::new(Uuid::new_v4(), content_lock(), NOW).unwrap(); + repository.insert_job(job).await.unwrap(); + let claimed = repository + .claim_next("worker", time::Duration::minutes(5)) + .await + .unwrap() + .unwrap(); + clock.set(NOW + time::Duration::minutes(5)); + let ownership = InMemoryContentLockDeletionActionOwnership::new(repository); + let result = ownership + .try_acquire(ContentLockDeletionActionClaim { + job_id: claimed.job.job_id, + worker_id: "worker", + claim_token: claimed.claim_token, + expected_phase: claimed.job.phase, + force: false, + }) + .await + .unwrap(); + assert!(matches!( + result, + ContentLockDeletionActionAcquireResult::ClaimLost + )); +} + #[tokio::test] async fn frozen_manifest_identity_is_immutable_and_creator_lock_unique() { let repository = InMemoryContentLockDeletionRepository::with_verification_task_fence(Arc::new( @@ -86,14 +245,15 @@ async fn frozen_manifest_identity_is_immutable_and_creator_lock_unique() { #[tokio::test] async fn due_claims_reclaim_with_fresh_tokens_and_fence_stale_writes() { + let clock = Arc::new(MutableClock::new(NOW)); let repository = InMemoryContentLockDeletionRepository::with_verification_task_fence(Arc::new( - InMemoryVerificationTaskDeletionFence::with_clock(Arc::new(FixedClock(NOW))), + InMemoryVerificationTaskDeletionFence::with_clock(clock.clone()), )); let job = ContentLockDeletionJob::new(Uuid::new_v4(), content_lock(), NOW).unwrap(); repository.insert_job(job.clone()).await.unwrap(); let first = repository - .claim_next("worker-a", NOW, LEASE_END) + .claim_next("worker-a", (LEASE_END) - (NOW)) .await .unwrap() .unwrap(); @@ -101,17 +261,17 @@ async fn due_claims_reclaim_with_fresh_tokens_and_fence_stale_writes() { assert_eq!(first.job.attempt_count, 1); assert!( repository - .claim_next("worker-b", NOW, LEASE_END) + .claim_next("worker-b", (LEASE_END) - (NOW)) .await .unwrap() .is_none() ); + clock.set(LEASE_END); let reclaimed = repository .claim_next( "worker-b", - datetime!(2026-08-12 05:05:01 UTC), - datetime!(2026-08-12 05:10:00 UTC), + (datetime!(2026-08-12 05:10:00 UTC)) - (datetime!(2026-08-12 05:05:01 UTC)), ) .await .unwrap() @@ -119,39 +279,39 @@ async fn due_claims_reclaim_with_fresh_tokens_and_fence_stale_writes() { assert_ne!(first.claim_token, reclaimed.claim_token); assert_eq!(reclaimed.job.attempt_count, 2); - assert!( + assert_eq!( repository .advance_phase( job.job_id, "worker-a", first.claim_token, - datetime!(2026-08-12 05:05:01 UTC), ContentLockDeletionPhase::StartPaymentDrain, ) .await - .unwrap() - .is_none() + .unwrap(), + AdvanceContentLockDeletionPhaseResult::ClaimLost ); + clock.set(datetime!(2026-08-12 05:06:00 UTC)); let advanced = repository .advance_phase( job.job_id, "worker-b", reclaimed.claim_token, - datetime!(2026-08-12 05:06:00 UTC), ContentLockDeletionPhase::StartPaymentDrain, ) .await .unwrap() - .unwrap(); + .advanced() + .expect("live claim should advance phase"); assert_eq!(advanced.state, ContentLockDeletionState::Queued); assert_eq!(advanced.phase, ContentLockDeletionPhase::StartPaymentDrain); assert_eq!(advanced.attempt_count, 0); + clock.set(datetime!(2026-08-12 05:06:01 UTC)); let next_claim = repository .claim_next( "worker-c", - datetime!(2026-08-12 05:06:01 UTC), - datetime!(2026-08-12 05:11:00 UTC), + (datetime!(2026-08-12 05:11:00 UTC)) - (datetime!(2026-08-12 05:06:01 UTC)), ) .await .unwrap() @@ -162,18 +322,17 @@ async fn due_claims_reclaim_with_fresh_tokens_and_fence_stale_writes() { job.job_id, "worker-c", next_claim.claim_token, - datetime!(2026-08-12 05:07:00 UTC), ContentLockDeletionPhase::DeleteContent, ) .await .is_err() ); + clock.set(datetime!(2026-08-12 05:07:00 UTC)); let failed = repository .finish( job.job_id, "worker-c", next_claim.claim_token, - datetime!(2026-08-12 05:07:00 UTC), Some(ContentLockDeletionFailureCode::TombstoneMissing), ) .await @@ -186,6 +345,84 @@ async fn due_claims_reclaim_with_fresh_tokens_and_fence_stale_writes() { ); } +#[tokio::test] +async fn healthy_defer_does_not_accumulate_transient_attempts_across_polls() { + let clock = Arc::new(MutableClock::new(NOW)); + let repository = InMemoryContentLockDeletionRepository::with_verification_task_fence(Arc::new( + InMemoryVerificationTaskDeletionFence::with_clock(clock.clone()), + )); + let job = ContentLockDeletionJob::new(Uuid::new_v4(), content_lock(), NOW).unwrap(); + repository.insert_job(job.clone()).await.unwrap(); + + let mut due_at = NOW; + for poll in 1..=3 { + let claimed = repository + .claim_next("worker-a", (due_at + time::Duration::minutes(5)) - (due_at)) + .await + .unwrap() + .unwrap(); + assert_eq!(claimed.job.attempt_count, 1, "healthy poll {poll}"); + + due_at += time::Duration::minutes(1); + let deferred = repository + .defer( + job.job_id, + "worker-a", + claimed.claim_token, + time::Duration::minutes(1), + ) + .await + .unwrap() + .unwrap(); + assert_eq!(deferred.state, ContentLockDeletionState::Queued); + assert_eq!(deferred.attempt_count, 0); + assert_eq!(deferred.next_attempt_at, Some(due_at)); + clock.set(due_at); + } +} + +#[tokio::test] +async fn defer_is_fenced_by_the_live_claim_token() { + let repository = InMemoryContentLockDeletionRepository::with_verification_task_fence(Arc::new( + InMemoryVerificationTaskDeletionFence::with_clock(Arc::new(FixedClock(NOW))), + )); + let job = ContentLockDeletionJob::new(Uuid::new_v4(), content_lock(), NOW).unwrap(); + repository.insert_job(job.clone()).await.unwrap(); + let claimed = repository + .claim_next("worker-a", (LEASE_END) - (NOW)) + .await + .unwrap() + .unwrap(); + let next_poll = NOW + time::Duration::minutes(1); + + assert!( + repository + .defer(job.job_id, "worker-a", Uuid::new_v4(), (next_poll) - (NOW)) + .await + .unwrap() + .is_none() + ); + let still_running = repository + .get_job(&job.creator, &job.lock_id) + .await + .unwrap() + .unwrap(); + assert_eq!(still_running.state, ContentLockDeletionState::Running); + assert_eq!(still_running.attempt_count, 1); + + let deferred = repository + .defer( + job.job_id, + "worker-a", + claimed.claim_token, + (next_poll) - (NOW), + ) + .await + .unwrap() + .unwrap(); + assert_eq!(deferred.attempt_count, 0); +} + #[test] fn failure_codes_are_a_closed_stable_vocabulary() { for (code, wire) in [ @@ -197,6 +434,10 @@ fn failure_codes_are_a_closed_stable_vocabulary() { ContentLockDeletionFailureCode::TombstoneReplaced, "tombstone_replaced", ), + ( + ContentLockDeletionFailureCode::ResourceReplaced, + "resource_replaced", + ), ( ContentLockDeletionFailureCode::RetryExhausted, "retry_exhausted", @@ -217,36 +458,51 @@ fn failure_codes_are_a_closed_stable_vocabulary() { .parse::() .is_err() ); + assert!( + "final_credential_issuance_missed" + .parse::() + .is_err() + ); } #[tokio::test] async fn retry_due_time_and_force_receipts_are_durable_repository_facts() { + let clock = Arc::new(MutableClock::new(NOW)); let repository = InMemoryContentLockDeletionRepository::with_verification_task_fence(Arc::new( - InMemoryVerificationTaskDeletionFence::with_clock(Arc::new(FixedClock(NOW))), + InMemoryVerificationTaskDeletionFence::with_clock(clock.clone()), )); let job = ContentLockDeletionJob::new(Uuid::new_v4(), content_lock(), NOW).unwrap(); repository.insert_job(job.clone()).await.unwrap(); let claimed = repository - .claim_next("worker-a", NOW, LEASE_END) + .claim_next("worker-a", (LEASE_END) - (NOW)) .await .unwrap() .unwrap(); let retry_at = datetime!(2026-08-12 05:06:00 UTC); repository - .schedule_retry(job.job_id, "worker-a", claimed.claim_token, NOW, retry_at) + .schedule_retry( + job.job_id, + "worker-a", + claimed.claim_token, + (retry_at) - (NOW), + ) .await .unwrap() .unwrap(); assert!( repository - .claim_next("worker-b", NOW, LEASE_END) + .claim_next("worker-b", (LEASE_END) - (NOW)) .await .unwrap() .is_none() ); + clock.set(retry_at); assert!( repository - .claim_next("worker-b", retry_at, datetime!(2026-08-12 05:11:00 UTC)) + .claim_next( + "worker-b", + (datetime!(2026-08-12 05:11:00 UTC)) - (retry_at) + ) .await .unwrap() .is_some() @@ -254,11 +510,144 @@ async fn retry_due_time_and_force_receipts_are_durable_repository_facts() { assert!(matches!( repository - .prepare_force_deletion(&job.creator, &job.lock_id, NOW) + .prepare_force_deletion(&job.creator, &job.lock_id) + .await + .unwrap(), + PrepareForceDeletionResult::Active(_) + )); + assert!( + !repository + .has_force_receipt(&job.creator, &job.lock_id) + .await + .unwrap() + ); +} + +#[tokio::test] +async fn in_memory_active_force_completion_is_exactly_fenced_and_permanently_blocks_publication() { + let clock = Arc::new(MutableClock::new(NOW)); + let repository = InMemoryContentLockDeletionRepository::with_verification_task_fence(Arc::new( + InMemoryVerificationTaskDeletionFence::with_clock(clock.clone()), + )); + let job = ContentLockDeletionJob::new(Uuid::new_v4(), content_lock(), NOW).unwrap(); + repository.insert_job(job.clone()).await.unwrap(); + let revoked = repository + .claim_next("worker-old", (LEASE_END) - (NOW)) + .await + .unwrap() + .unwrap(); + let forced_at = NOW + time::Duration::seconds(1); + clock.set(forced_at); + assert!(matches!( + repository + .prepare_force_deletion(&job.creator, &job.lock_id) .await .unwrap(), PrepareForceDeletionResult::Active(_) )); + assert!( + !repository + .complete_force_deletion(job.job_id, "worker-old", revoked.claim_token) + .await + .unwrap() + ); + + let expiring = repository + .claim_next( + "worker-expiring", + (forced_at + time::Duration::minutes(1)) - (forced_at), + ) + .await + .unwrap() + .unwrap(); + assert!( + !repository + .complete_force_deletion(job.job_id, "worker-expiring", Uuid::new_v4()) + .await + .unwrap() + ); + let reclaim_at = forced_at + time::Duration::minutes(2); + clock.set(reclaim_at); + assert!( + !repository + .complete_force_deletion(job.job_id, "worker-expiring", expiring.claim_token,) + .await + .unwrap() + ); + let live = repository + .claim_next( + "worker-live", + (reclaim_at + time::Duration::minutes(5)) - (reclaim_at), + ) + .await + .unwrap() + .unwrap(); + assert!( + !repository + .complete_force_deletion(job.job_id, "worker-expiring", expiring.claim_token,) + .await + .unwrap() + ); + assert!( + repository + .complete_force_deletion(job.job_id, "worker-live", live.claim_token) + .await + .unwrap() + ); + + assert!( + repository + .get_job(&job.creator, &job.lock_id) + .await + .unwrap() + .is_none() + ); + assert!( + repository + .has_force_receipt(&job.creator, &job.lock_id) + .await + .unwrap() + ); + assert!( + !repository + .complete_force_deletion(job.job_id, "worker-live", live.claim_token) + .await + .unwrap() + ); + assert_eq!( + repository + .begin_publication(&job.creator, &job.lock_id, Uuid::new_v4()) + .await, + Err(locks_service::application::errors::ApplicationError::ContentLockDeletionInProgress) + ); +} + +#[tokio::test] +async fn in_memory_unforced_claim_cannot_create_a_force_receipt() { + let repository = InMemoryContentLockDeletionRepository::with_verification_task_fence(Arc::new( + InMemoryVerificationTaskDeletionFence::with_clock(Arc::new(FixedClock(NOW))), + )); + let job = ContentLockDeletionJob::new(Uuid::new_v4(), content_lock(), NOW).unwrap(); + repository.insert_job(job.clone()).await.unwrap(); + let claim = repository + .claim_next("worker", (LEASE_END) - (NOW)) + .await + .unwrap() + .unwrap(); + + assert!( + !repository + .complete_force_deletion(job.job_id, "worker", claim.claim_token) + .await + .unwrap() + ); + assert!( + repository + .get_job(&job.creator, &job.lock_id) + .await + .unwrap() + .is_some() + ); assert!( !repository .has_force_receipt(&job.creator, &job.lock_id) @@ -299,8 +688,7 @@ async fn in_memory_deletion_enrolls_existing_ordinary_credentials_and_blocks_lat .prepare_deletion_read( &ordinary_lookup, "/priv/locks.app/content/post.json", - NOW, - NOW + time::Duration::seconds(30), + time::Duration::seconds(30), ) .await .unwrap() @@ -309,8 +697,7 @@ async fn in_memory_deletion_enrolls_existing_ordinary_credentials_and_blocks_lat .prepare_deletion_read( &ordinary_lookup, "/priv/locks.app/content/post.json", - NOW + time::Duration::minutes(1), - NOW + time::Duration::minutes(2), + time::Duration::minutes(1), ) .await .unwrap() @@ -331,8 +718,7 @@ async fn in_memory_deletion_enrolls_existing_ordinary_credentials_and_blocks_lat .prepare_deletion_read( &ordinary_lookup, "/priv/locks.app/content/not-in-frozen-manifest.json", - NOW, - NOW + time::Duration::seconds(30), + time::Duration::seconds(30), ) .await .unwrap() @@ -367,18 +753,15 @@ async fn in_memory_deletion_enrolls_existing_ordinary_credentials_and_blocks_lat job.job_id, "worker-final", drain_existing_claim.claim_token, - NOW, ContentLockDeletionPhase::IssueFinalCredentials, ) .await, - Err( - locks_service::application::errors::ApplicationError::InvalidContentLockDeletionState { .. } - ) + Ok(AdvanceContentLockDeletionPhaseResult::ObligationsPending) )); assert!(matches!( deletions - .prepare_force_deletion(&job.creator, &job.lock_id, NOW) + .prepare_force_deletion(&job.creator, &job.lock_id) .await .unwrap(), PrepareForceDeletionResult::Active(_) @@ -517,7 +900,7 @@ async fn failed_access_registration_leaves_no_job_or_task_ownership() { assert!( tokio::time::timeout( std::time::Duration::from_secs(1), - claimer.claim_next_verification_task("worker", NOW, LEASE_END), + claimer.claim_next_verification_task("worker", (LEASE_END) - (NOW)), ) .await .expect("failed deletion admission must release task ownership locks") @@ -528,7 +911,8 @@ async fn failed_access_registration_leaves_no_job_or_task_ownership() { #[tokio::test] async fn in_memory_final_credential_is_exactly_replayable_and_reads_are_lease_fenced() { - let (verification_tasks, access, deletions) = in_memory_access_stack(); + let clock = Arc::new(MutableClock::new(NOW)); + let (verification_tasks, access, deletions) = in_memory_access_stack_with_clock(clock.clone()); let job = ContentLockDeletionJob::new(Uuid::new_v4(), content_lock(), NOW).unwrap(); let completed = verification_task(&job, VerificationTaskStatus::Pending) .transition_to(VerificationTaskStatus::InProgress, NOW, None) @@ -544,31 +928,36 @@ async fn in_memory_final_credential_is_exactly_replayable_and_reads_are_lease_fe let claimed = advance_to_final_issuance(&deletions, &access, job.job_id).await; let issuance_deadline = NOW + time::Duration::minutes(15); let read_deadline = NOW + time::Duration::minutes(30); - assert!( + let initialized = access + .initialize_final_access_windows( + job.job_id, + "worker-final", + claimed.claim_token, + time::Duration::minutes(15), + time::Duration::minutes(15), + ) + .await + .unwrap(); + assert_eq!( + initialized, + InitializeFinalAccessWindowsResult::Initialized(FinalAccessWindows { + issuance_started_at: NOW, + credential_issuance_deadline: issuance_deadline, + read_deadline, + }) + ); + assert_eq!( access .initialize_final_access_windows( job.job_id, "worker-final", claimed.claim_token, - NOW, - issuance_deadline, - read_deadline, + time::Duration::minutes(20), + time::Duration::minutes(20), ) .await - .unwrap() - ); - assert!( - access - .initialize_final_access_windows( - job.job_id, - "worker-final", - claimed.claim_token, - NOW, - NOW + time::Duration::minutes(20), - NOW + time::Duration::minutes(40), - ) - .await - .unwrap() + .unwrap(), + initialized ); let candidate = AccessCredential::new("final-secret-bearer"); @@ -608,12 +997,12 @@ async fn in_memory_final_credential_is_exactly_replayable_and_reads_are_lease_fe job.job_id, "worker-final", claimed.claim_token, - NOW, ContentLockDeletionPhase::DrainFinalReads, ) .await .unwrap() - .unwrap(); + .advanced() + .expect("live claim should advance phase"); assert_eq!(advanced.phase, ContentLockDeletionPhase::DrainFinalReads); let phase_replay = access .issue_or_replay_final_credential( @@ -630,19 +1019,15 @@ async fn in_memory_final_credential_is_exactly_replayable_and_reads_are_lease_fe let lookup = AccessCredentialLookupKey::derive(&first.credential); let path = "/priv/locks.app/content/post.json"; let first_claim = access - .prepare_deletion_read(&lookup, path, NOW, NOW + time::Duration::minutes(1)) + .prepare_deletion_read(&lookup, path, time::Duration::minutes(1)) .await .unwrap() .unwrap() .claim_token .unwrap(); + clock.set(NOW + time::Duration::seconds(30)); let equality_reclaim = access - .prepare_deletion_read( - &lookup, - path, - NOW + time::Duration::seconds(30), - NOW + time::Duration::minutes(2), - ) + .prepare_deletion_read(&lookup, path, time::Duration::seconds(90)) .await .unwrap() .unwrap() @@ -681,8 +1066,7 @@ async fn in_memory_final_credential_is_exactly_replayable_and_reads_are_lease_fe .prepare_deletion_read( &lookup, path, - NOW + time::Duration::seconds(30), - NOW + time::Duration::hours(1), + time::Duration::minutes(59) + time::Duration::seconds(30), ) .await .unwrap() @@ -690,14 +1074,19 @@ async fn in_memory_final_credential_is_exactly_replayable_and_reads_are_lease_fe .claim_token .unwrap(); let reclaim_time = NOW + time::Duration::seconds(60); + clock.set(reclaim_time); assert!( !access - .consume_deletion_read(&lookup, path, stale_claim, reclaim_time) + .consume_deletion_read(&lookup, path, stale_claim) .await .unwrap() ); let recovered_claim = access - .prepare_deletion_read(&lookup, path, reclaim_time, NOW + time::Duration::hours(1)) + .prepare_deletion_read( + &lookup, + path, + (NOW + time::Duration::hours(1)) - reclaim_time, + ) .await .unwrap() .unwrap() @@ -705,24 +1094,20 @@ async fn in_memory_final_credential_is_exactly_replayable_and_reads_are_lease_fe .unwrap(); assert!( !access - .consume_deletion_read(&lookup, path, stale_claim, reclaim_time) + .consume_deletion_read(&lookup, path, stale_claim) .await .unwrap() ); assert!( access - .consume_deletion_read(&lookup, path, recovered_claim, reclaim_time) + .consume_deletion_read(&lookup, path, recovered_claim) .await .unwrap() ); + clock.set(NOW + time::Duration::minutes(3)); assert!( access - .prepare_deletion_read( - &lookup, - path, - NOW + time::Duration::minutes(3), - NOW + time::Duration::minutes(4), - ) + .prepare_deletion_read(&lookup, path, time::Duration::minutes(1)) .await .unwrap() .is_none() @@ -807,14 +1192,14 @@ async fn mutable_task_completion_cannot_resolve_the_immutable_deletion_snapshot( job.job_id, "worker-final", drain_claim.claim_token, - NOW, ContentLockDeletionPhase::DrainExistingCredentials, ) .await .unwrap() - .unwrap(); + .advanced() + .expect("live claim should advance phase"); let existing_drain_claim = deletions - .claim_next("worker-final", NOW, LEASE_END) + .claim_next("worker-final", (LEASE_END) - (NOW)) .await .unwrap() .unwrap(); @@ -823,14 +1208,14 @@ async fn mutable_task_completion_cannot_resolve_the_immutable_deletion_snapshot( job.job_id, "worker-final", existing_drain_claim.claim_token, - NOW, ContentLockDeletionPhase::IssueFinalCredentials, ) .await .unwrap() - .unwrap(); + .advanced() + .expect("live claim should advance phase"); let final_claim = deletions - .claim_next("worker-final", NOW, LEASE_END) + .claim_next("worker-final", (LEASE_END) - (NOW)) .await .unwrap() .unwrap(); @@ -839,9 +1224,8 @@ async fn mutable_task_completion_cannot_resolve_the_immutable_deletion_snapshot( job.job_id, "worker-final", final_claim.claim_token, - NOW, - NOW + time::Duration::minutes(15), - NOW + time::Duration::minutes(30), + time::Duration::minutes(15), + time::Duration::minutes(15), ) .await .unwrap(); @@ -857,8 +1241,11 @@ async fn mutable_task_completion_cannot_resolve_the_immutable_deletion_snapshot( async fn in_memory_payment_drain_waits_for_pending_non_paykit_snapshot() { let (verification_tasks, access, deletions) = in_memory_access_stack(); let job = ContentLockDeletionJob::new(Uuid::new_v4(), content_lock(), NOW).unwrap(); - let mut pending = verification_task(&job, VerificationTaskStatus::Pending); - pending.submitted_proof_bundle.proofs[0].verifier_type = VerifierType::DevStatic; + let pending = verification_task_with_verifier( + &job, + VerificationTaskStatus::Pending, + VerifierType::DevStatic, + ); verification_tasks .insert_verification_task(pending) .await @@ -879,7 +1266,6 @@ async fn in_memory_payment_drain_waits_for_pending_non_paykit_snapshot() { job.job_id, "worker-final", drain_claim.claim_token, - NOW, ContentLockDeletionPhase::DrainExistingCredentials, ), ) @@ -891,6 +1277,362 @@ async fn in_memory_payment_drain_waits_for_pending_non_paykit_snapshot() { )); } +#[tokio::test] +async fn no_paykit_drain_expires_pending_task_and_advances_without_an_aggregate() { + let (verification_tasks, _access, deletions) = in_memory_access_stack(); + let job = ContentLockDeletionJob::new(Uuid::new_v4(), content_lock(), NOW).unwrap(); + let pending = verification_task_with_verifier( + &job, + VerificationTaskStatus::Pending, + VerifierType::DevStatic, + ); + let task_id = pending.task_id; + verification_tasks + .insert_verification_task(pending) + .await + .unwrap(); + deletions.insert_job(job.clone()).await.unwrap(); + + let withdraw = deletions + .claim_next("worker", (LEASE_END) - (NOW)) + .await + .unwrap() + .unwrap(); + deletions + .advance_phase( + job.job_id, + "worker", + withdraw.claim_token, + ContentLockDeletionPhase::StartPaymentDrain, + ) + .await + .unwrap() + .advanced() + .expect("live claim should advance phase"); + + let use_case = NoPaykitDeletionDrainUseCase::new(&deletions, &FixedClock(NOW)); + let start = deletions + .claim_next("worker", (LEASE_END) - (NOW)) + .await + .unwrap() + .unwrap(); + assert!(use_case.execute_claimed(start, "worker").await.unwrap()); + let drain = deletions + .claim_next("worker", (LEASE_END) - (NOW)) + .await + .unwrap() + .unwrap(); + assert!(use_case.execute_claimed(drain, "worker").await.unwrap()); + + assert_eq!( + verification_tasks + .get_verification_task(&task_id) + .await + .unwrap() + .unwrap() + .status, + VerificationTaskStatus::Expired + ); + assert_eq!( + deletions + .get_job(&job.creator, &job.lock_id) + .await + .unwrap() + .unwrap() + .phase, + ContentLockDeletionPhase::DrainExistingCredentials + ); +} + +#[tokio::test] +async fn no_paykit_drain_preserves_terminal_tasks() { + for status in [ + VerificationTaskStatus::Completed, + VerificationTaskStatus::Failed, + VerificationTaskStatus::Expired, + ] { + let (verification_tasks, _access, deletions) = in_memory_access_stack(); + let job = ContentLockDeletionJob::new(Uuid::new_v4(), content_lock(), NOW).unwrap(); + let mut task = verification_task_with_verifier(&job, status, VerifierType::DevStatic); + task.started_at = (status != VerificationTaskStatus::Expired).then_some(NOW); + task.completed_at = Some(NOW); + task.failure_message = + (status == VerificationTaskStatus::Failed).then(|| "closed failure".to_owned()); + let task_id = task.task_id; + verification_tasks + .insert_verification_task(task) + .await + .unwrap(); + deletions.insert_job(job.clone()).await.unwrap(); + let withdraw = deletions + .claim_next("worker", (LEASE_END) - (NOW)) + .await + .unwrap() + .unwrap(); + deletions + .advance_phase( + job.job_id, + "worker", + withdraw.claim_token, + ContentLockDeletionPhase::StartPaymentDrain, + ) + .await + .unwrap() + .advanced() + .expect("live claim should advance phase"); + let claim = deletions + .claim_next("worker", (LEASE_END) - (NOW)) + .await + .unwrap() + .unwrap(); + NoPaykitDeletionDrainUseCase::new(&deletions, &FixedClock(NOW)) + .execute_claimed(claim, "worker") + .await + .unwrap(); + assert_eq!( + verification_tasks + .get_verification_task(&task_id) + .await + .unwrap() + .unwrap() + .status, + status + ); + } +} + +#[tokio::test] +async fn no_paykit_drain_fences_stale_claim() { + let clock = Arc::new(MutableClock::new(NOW)); + let (verification_tasks, _access, deletions) = in_memory_access_stack_with_clock(clock.clone()); + let job = ContentLockDeletionJob::new(Uuid::new_v4(), content_lock(), NOW).unwrap(); + let task = verification_task_with_verifier( + &job, + VerificationTaskStatus::Pending, + VerifierType::DevStatic, + ); + let task_id = task.task_id; + verification_tasks + .insert_verification_task(task) + .await + .unwrap(); + deletions.insert_job(job.clone()).await.unwrap(); + let claim = advance_to_start_payment_drain(&deletions, job.job_id).await; + clock.set(LEASE_END + time::Duration::seconds(1)); + assert!( + !NoPaykitDeletionDrainUseCase::new(&deletions, &FixedClock(NOW)) + .execute_claimed(claim, "worker") + .await + .unwrap() + ); + assert_eq!( + verification_tasks + .get_verification_task(&task_id) + .await + .unwrap() + .unwrap() + .status, + VerificationTaskStatus::Pending + ); +} + +#[tokio::test] +async fn in_memory_payment_resolution_rejects_claim_at_expiry_equality() { + let (verification_tasks, access, deletions) = in_memory_access_stack(); + let job = ContentLockDeletionJob::new(Uuid::new_v4(), content_lock(), NOW).unwrap(); + let task = verification_task(&job, VerificationTaskStatus::Pending); + let task_id = task.task_id; + verification_tasks + .insert_verification_task(task) + .await + .unwrap(); + deletions.insert_job(job.clone()).await.unwrap(); + let claim = advance_to_phase( + &deletions, + &access, + job.job_id, + ContentLockDeletionPhase::DrainPayments, + ) + .await; + + assert!( + !access + .resolve_deletion_payment( + job.job_id, + "worker-final", + claim.claim_token, + LEASE_END, + &task_id, + VerificationTaskStatus::Completed, + ) + .await + .unwrap() + ); +} + +#[tokio::test] +async fn in_memory_payment_aggregate_completion_rejects_claim_at_expiry_equality() { + let (verification_tasks, access, deletions) = in_memory_access_stack(); + let job = ContentLockDeletionJob::new(Uuid::new_v4(), content_lock(), NOW).unwrap(); + verification_tasks + .insert_verification_task(verification_task(&job, VerificationTaskStatus::Completed)) + .await + .unwrap(); + deletions.insert_job(job.clone()).await.unwrap(); + let claim = advance_to_phase( + &deletions, + &access, + job.job_id, + ContentLockDeletionPhase::DrainPayments, + ) + .await; + + assert!( + !access + .complete_deletion_payment_aggregate( + job.job_id, + "worker-final", + claim.claim_token, + LEASE_END, + ) + .await + .unwrap() + ); +} + +#[tokio::test] +async fn in_memory_non_paykit_expiry_rejects_claim_at_expiry_equality() { + let clock = Arc::new(MutableClock::new(NOW)); + let (verification_tasks, _access, deletions) = in_memory_access_stack_with_clock(clock.clone()); + let job = ContentLockDeletionJob::new(Uuid::new_v4(), content_lock(), NOW).unwrap(); + let task = verification_task_with_verifier( + &job, + VerificationTaskStatus::Pending, + VerifierType::DevStatic, + ); + let task_id = task.task_id; + verification_tasks + .insert_verification_task(task) + .await + .unwrap(); + deletions.insert_job(job.clone()).await.unwrap(); + let claim = advance_to_start_payment_drain(&deletions, job.job_id).await; + + clock.set(LEASE_END); + assert!( + !deletions + .expire_unresolved_non_paykit_tasks(job.job_id, "worker", claim.claim_token) + .await + .unwrap() + ); + assert_eq!( + verification_tasks + .get_verification_task(&task_id) + .await + .unwrap() + .unwrap() + .status, + VerificationTaskStatus::Pending + ); +} + +#[tokio::test] +async fn no_paykit_task_expiry_serializes_with_force_claim_revocation() { + let fence = Arc::new(InMemoryVerificationTaskDeletionFence::with_clock(Arc::new( + FixedClock(NOW), + ))); + let verification_tasks = Arc::new(InMemoryVerificationTaskRepository::with_deletion_fence( + Arc::clone(&fence), + )); + let pausing_tasks = Arc::new(PausingVerificationTaskRepository::new( + verification_tasks.clone(), + )); + let access = Arc::new( + InMemoryAccessCredentialStore::with_verification_task_repository_and_deletion_fence( + pausing_tasks.clone(), + Arc::clone(&fence), + ), + ); + let deletions = Arc::new( + InMemoryContentLockDeletionRepository::with_access_credentials_and_verification_task_fence( + access, fence, + ), + ); + let job = ContentLockDeletionJob::new(Uuid::new_v4(), content_lock(), NOW).unwrap(); + let task = verification_task_with_verifier( + &job, + VerificationTaskStatus::Pending, + VerifierType::DevStatic, + ); + let task_id = task.task_id; + verification_tasks + .insert_verification_task(task) + .await + .unwrap(); + deletions.insert_job(job.clone()).await.unwrap(); + let claim = advance_to_start_payment_drain(&deletions, job.job_id).await; + + let expiry_repository = deletions.clone(); + let job_id = job.job_id; + let expiry = tokio::spawn(async move { + expiry_repository + .expire_unresolved_non_paykit_tasks(job_id, "worker", claim.claim_token) + .await + }); + pausing_tasks.get_entered.notified().await; + + let force_repository = deletions.clone(); + let creator = job.creator.clone(); + let lock_id = job.lock_id.clone(); + let mut force = tokio::spawn(async move { + force_repository + .prepare_force_deletion(&creator, &lock_id) + .await + }); + assert!( + tokio::time::timeout(std::time::Duration::from_millis(50), &mut force) + .await + .is_err(), + "force must wait until task and snapshot terminalization commits" + ); + + pausing_tasks.release_get.notify_one(); + assert!(expiry.await.unwrap().unwrap()); + assert!(matches!( + force.await.unwrap().unwrap(), + PrepareForceDeletionResult::Active(_) + )); + assert_eq!( + verification_tasks + .get_verification_task(&task_id) + .await + .unwrap() + .unwrap() + .status, + VerificationTaskStatus::Expired + ); +} + +#[tokio::test] +async fn no_paykit_drain_rejects_paykit_snapshot() { + let (verification_tasks, _access, deletions) = in_memory_access_stack(); + let job = ContentLockDeletionJob::new(Uuid::new_v4(), content_lock(), NOW).unwrap(); + verification_tasks + .insert_verification_task(verification_task(&job, VerificationTaskStatus::Pending)) + .await + .unwrap(); + deletions.insert_job(job.clone()).await.unwrap(); + let claim = advance_to_start_payment_drain(&deletions, job.job_id).await; + assert!(matches!( + NoPaykitDeletionDrainUseCase::new(&deletions, &FixedClock(NOW)) + .execute_claimed(claim, "worker") + .await, + Err( + locks_service::application::errors::ApplicationError::InvalidContentLockDeletionState { .. } + ) + )); +} + #[tokio::test] async fn in_memory_payment_drain_waits_for_pending_paykit_snapshot() { let (verification_tasks, access, deletions) = in_memory_access_stack(); @@ -915,7 +1657,6 @@ async fn in_memory_payment_drain_waits_for_pending_paykit_snapshot() { job.job_id, "worker-final", drain_claim.claim_token, - NOW, ContentLockDeletionPhase::DrainExistingCredentials, ), ) @@ -968,7 +1709,6 @@ async fn in_memory_payment_drain_waits_for_completed_aggregate() { job.job_id, "worker-final", drain_claim.claim_token, - NOW, ContentLockDeletionPhase::DrainExistingCredentials, ), ) @@ -982,7 +1722,8 @@ async fn in_memory_payment_drain_waits_for_completed_aggregate() { #[tokio::test] async fn in_memory_phase_and_success_finish_guards_preserve_access_obligations() { - let (verification_tasks, access, deletions) = in_memory_access_stack(); + let clock = Arc::new(MutableClock::new(NOW)); + let (verification_tasks, access, deletions) = in_memory_access_stack_with_clock(clock.clone()); let job = ContentLockDeletionJob::new(Uuid::new_v4(), content_lock(), NOW).unwrap(); let completed = verification_task(&job, VerificationTaskStatus::Pending) .transition_to(VerificationTaskStatus::InProgress, NOW, None) @@ -1008,18 +1749,17 @@ async fn in_memory_phase_and_success_finish_guards_preserve_access_obligations() job.job_id, "worker-final", issue_claim.claim_token, - NOW, - NOW + time::Duration::minutes(15), - NOW + time::Duration::minutes(30), + time::Duration::minutes(15), + time::Duration::minutes(15), ) .await .unwrap(); let deadline = NOW + time::Duration::minutes(15); + clock.set(deadline); let deadline_claim = deletions .claim_next( "worker-final", - deadline, - deadline + time::Duration::minutes(5), + (deadline + time::Duration::minutes(5)) - (deadline), ) .await .unwrap() @@ -1030,13 +1770,12 @@ async fn in_memory_phase_and_success_finish_guards_preserve_access_obligations() job.job_id, "worker-final", deadline_claim.claim_token, - deadline, ContentLockDeletionPhase::DrainFinalReads, ) .await, - Err( - locks_service::application::errors::ApplicationError::InvalidContentLockDeletionState { .. } - ) + Ok(AdvanceContentLockDeletionPhaseResult::TerminalFailure( + ContentLockDeletionFailureCode::StateCorrupt + )) )); assert!( access @@ -1052,13 +1791,7 @@ async fn in_memory_phase_and_success_finish_guards_preserve_access_obligations() ); assert!(matches!( deletions - .finish( - job.job_id, - "worker-final", - deadline_claim.claim_token, - deadline, - None - ) + .finish(job.job_id, "worker-final", deadline_claim.claim_token, None,) .await, Err( locks_service::application::errors::ApplicationError::InvalidContentLockDeletionState { .. } @@ -1069,7 +1802,8 @@ async fn in_memory_phase_and_success_finish_guards_preserve_access_obligations() #[tokio::test] async fn in_memory_final_credential_eligibility_does_not_change_when_cutoff_credential_is_deleted() { - let (verification_tasks, access, deletions) = in_memory_access_stack(); + let clock = Arc::new(MutableClock::new(NOW)); + let (verification_tasks, access, deletions) = in_memory_access_stack_with_clock(clock.clone()); let job = ContentLockDeletionJob::new(Uuid::new_v4(), content_lock(), NOW).unwrap(); let completed = verification_task(&job, VerificationTaskStatus::Pending) .transition_to(VerificationTaskStatus::InProgress, NOW, None) @@ -1105,31 +1839,29 @@ async fn in_memory_final_credential_eligibility_does_not_change_when_cutoff_cred ContentLockDeletionPhase::DrainExistingCredentials, ) .await; - assert!(matches!( + assert_eq!( deletions .advance_phase( job.job_id, "worker-final", drain_existing_claim.claim_token, - NOW, ContentLockDeletionPhase::IssueFinalCredentials, ) - .await, - Err( - locks_service::application::errors::ApplicationError::InvalidContentLockDeletionState { .. } - ) - )); + .await + .unwrap(), + AdvanceContentLockDeletionPhaseResult::ObligationsPending + ); let after_expiry = original_expiry; access .delete_access_credential(&ordinary_lookup) .await .unwrap(); + clock.set(after_expiry); let after_expiry_claim = deletions .claim_next( "worker-final", - after_expiry, - after_expiry + time::Duration::minutes(5), + (after_expiry + time::Duration::minutes(5)) - (after_expiry), ) .await .unwrap() @@ -1139,34 +1871,33 @@ async fn in_memory_final_credential_eligibility_does_not_change_when_cutoff_cred job.job_id, "worker-final", after_expiry_claim.claim_token, - after_expiry, ContentLockDeletionPhase::IssueFinalCredentials, ) .await .unwrap() - .unwrap(); + .advanced() + .expect("live claim should advance phase"); let claimed = deletions .claim_next( "worker-final", - after_expiry, - after_expiry + time::Duration::minutes(5), + (after_expiry + time::Duration::minutes(5)) - (after_expiry), ) .await .unwrap() .unwrap(); - assert!( + assert!(matches!( access .initialize_final_access_windows( job.job_id, "worker-final", claimed.claim_token, - after_expiry, - after_expiry + time::Duration::minutes(15), - after_expiry + time::Duration::minutes(30), + time::Duration::minutes(15), + time::Duration::minutes(15), ) .await - .unwrap() - ); + .unwrap(), + InitializeFinalAccessWindowsResult::Initialized(_) + )); access .delete_access_credential(&ordinary_lookup) @@ -1197,8 +1928,11 @@ async fn in_memory_final_credential_eligibility_does_not_change_when_cutoff_cred async fn in_memory_final_credential_rejects_completed_non_paykit_snapshot() { let (verification_tasks, access, deletions) = in_memory_access_stack(); let job = ContentLockDeletionJob::new(Uuid::new_v4(), content_lock(), NOW).unwrap(); - let mut task = verification_task(&job, VerificationTaskStatus::Completed); - task.submitted_proof_bundle.proofs[0].verifier_type = VerifierType::DevStatic; + let task = verification_task_with_verifier( + &job, + VerificationTaskStatus::Completed, + VerifierType::DevStatic, + ); let bundle_id = task.submitted_proof_bundle.bundle_id.clone(); verification_tasks .insert_verification_task(task.clone()) @@ -1206,19 +1940,19 @@ async fn in_memory_final_credential_rejects_completed_non_paykit_snapshot() { .unwrap(); deletions.insert_job(job.clone()).await.unwrap(); let claimed = advance_to_final_issuance(&deletions, &access, job.job_id).await; - assert!( + assert!(matches!( access .initialize_final_access_windows( job.job_id, "worker-final", claimed.claim_token, - NOW, - NOW + time::Duration::minutes(15), - NOW + time::Duration::minutes(30), + time::Duration::minutes(15), + time::Duration::minutes(15), ) .await - .unwrap() - ); + .unwrap(), + InitializeFinalAccessWindowsResult::Initialized(_) + )); assert!( !access @@ -1245,9 +1979,17 @@ fn in_memory_access_stack() -> ( Arc, InMemoryContentLockDeletionRepository, ) { - let fence = Arc::new(InMemoryVerificationTaskDeletionFence::with_clock(Arc::new( - FixedClock(NOW), - ))); + in_memory_access_stack_with_clock(Arc::new(FixedClock(NOW))) +} + +fn in_memory_access_stack_with_clock( + clock: Arc, +) -> ( + Arc, + Arc, + InMemoryContentLockDeletionRepository, +) { + let fence = Arc::new(InMemoryVerificationTaskDeletionFence::with_clock(clock)); let verification_tasks = Arc::new(InMemoryVerificationTaskRepository::with_deletion_fence( Arc::clone(&fence), )); @@ -1278,7 +2020,7 @@ async fn advance_to_phase( ContentLockDeletionPhase::IssueFinalCredentials, ] { let claimed = deletions - .claim_next("worker-final", NOW, LEASE_END) + .claim_next("worker-final", (LEASE_END) - (NOW)) .await .unwrap() .unwrap(); @@ -1296,13 +2038,14 @@ async fn advance_to_phase( ); } deletions - .advance_phase(job_id, "worker-final", claimed.claim_token, NOW, next_phase) + .advance_phase(job_id, "worker-final", claimed.claim_token, next_phase) .await .unwrap() - .unwrap(); + .advanced() + .expect("live claim should advance phase"); if next_phase == target { return deletions - .claim_next("worker-final", NOW, LEASE_END) + .claim_next("worker-final", (LEASE_END) - (NOW)) .await .unwrap() .unwrap(); @@ -1311,6 +2054,33 @@ async fn advance_to_phase( panic!("unsupported test target phase"); } +async fn advance_to_start_payment_drain( + deletions: &InMemoryContentLockDeletionRepository, + job_id: Uuid, +) -> locks_service::application::models::ClaimedContentLockDeletionJob { + let claimed = deletions + .claim_next("worker", LEASE_END - NOW) + .await + .unwrap() + .unwrap(); + deletions + .advance_phase( + job_id, + "worker", + claimed.claim_token, + ContentLockDeletionPhase::StartPaymentDrain, + ) + .await + .unwrap() + .advanced() + .expect("live claim should advance phase"); + deletions + .claim_next("worker", LEASE_END - NOW) + .await + .unwrap() + .unwrap() +} + async fn advance_to_final_issuance( deletions: &InMemoryContentLockDeletionRepository, access: &InMemoryAccessCredentialStore, @@ -1328,6 +2098,14 @@ async fn advance_to_final_issuance( fn verification_task( job: &ContentLockDeletionJob, status: VerificationTaskStatus, +) -> VerificationTaskRecord { + verification_task_with_verifier(job, status, VerifierType::PaykitPayment) +} + +fn verification_task_with_verifier( + job: &ContentLockDeletionJob, + status: VerificationTaskStatus, + verifier_type: VerifierType, ) -> VerificationTaskRecord { VerificationTaskRecord { task_id: TaskId::from_str("018fc6ec-2f3d-4f7e-8b7d-6f5c4b3a2d10").unwrap(), @@ -1343,7 +2121,7 @@ fn verification_task( reader_public_key: None, proofs: vec![Proof { criterion_id: "criterion-1".to_owned(), - verifier_type: VerifierType::PaykitPayment, + verifier_type, payload: json!({}), }], }, diff --git a/locks-service/tests/content_lock_tombstones.rs b/locks-service/tests/content_lock_tombstones.rs new file mode 100644 index 0000000..27a0940 --- /dev/null +++ b/locks-service/tests/content_lock_tombstones.rs @@ -0,0 +1,417 @@ +use std::str::FromStr; +use std::sync::Mutex; + +use async_trait::async_trait; +use locks_core::content_lock_deletion::ContentLockDeletionTombstone; +use locks_core::ids::{ContentLockPath, CreatorPubky, LockId}; +use locks_service::application::errors::ApplicationError; +use locks_service::application::ports::{ + ContentLockRepository, ContentLockTombstoneRepository, TombstoneReadback, +}; +use locks_service::infrastructure::memory::{ + content_lock_tombstones::InMemoryContentLockTombstoneRepository, + content_locks::InMemoryContentLockRepository, + public_content_locks::InMemoryPublicContentLockStore, +}; +use locks_service::infrastructure::pubky::{ + PubkyBytesResource, PubkyContentLockTombstoneRepository, PubkyHomeserverStorageClient, +}; +use time::macros::datetime; + +const LOCK_ID: &str = "000G40R40M30E209185GR38E1W8124GK2GAHC5RR34D1P70X3RFG"; + +#[tokio::test] +async fn memory_content_lock_and_tombstone_adapters_share_one_canonical_public_path() { + let store = InMemoryPublicContentLockStore::new(); + let content_locks = InMemoryContentLockRepository::with_public_store(store.clone()); + let tombstones = InMemoryContentLockTombstoneRepository::with_public_store(store); + let creator = creator(); + let path = path(); + let original: locks_core::lock_policy::ContentLock = + serde_json::from_value(serde_json::json!({ + "version": 1, + "creator": creator, + "primary_resource": null, + "secondary_resources": {}, + "criteria": [], + "lock_logic": { "type": "all", "criteria": [] }, + "access_policy": { "requested_credential_ttl_seconds": 900 }, + "lock_server": { "override": null }, + "created_at": "2026-08-12T04:00:00Z" + })) + .unwrap(); + content_locks + .upsert_content_lock(creator.clone(), path.clone(), original.clone()) + .await + .unwrap(); + + let tombstone = tombstone_at_five(); + assert_eq!( + tombstones + .withdraw_content_lock(creator.clone(), path.clone(), &original, &tombstone) + .await + .unwrap(), + TombstoneReadback::Exact + ); + assert!( + content_locks + .get_content_lock(&creator, &path) + .await + .is_err() + ); + + assert_eq!( + tombstones + .read_tombstone(&creator, &path, &tombstone) + .await + .unwrap(), + TombstoneReadback::Exact + ); + assert!( + content_locks + .get_content_lock(&creator, &path) + .await + .is_err() + ); +} + +#[tokio::test] +async fn memory_withdrawal_is_exact_and_replacement_is_classified_without_parsing() { + let store = InMemoryPublicContentLockStore::new(); + let content_locks = InMemoryContentLockRepository::with_public_store(store.clone()); + let repository = InMemoryContentLockTombstoneRepository::with_public_store(store); + let creator = creator(); + let path = path(); + let original = original_content_lock(); + let expected = tombstone_at_five(); + + assert_eq!( + repository + .read_tombstone(&creator, &path, &expected) + .await + .unwrap(), + TombstoneReadback::Missing + ); + assert_eq!( + repository + .withdraw_content_lock(creator.clone(), path.clone(), &original, &expected) + .await + .unwrap(), + TombstoneReadback::Missing + ); + + content_locks + .upsert_content_lock(creator.clone(), path.clone(), original.clone()) + .await + .unwrap(); + assert_eq!( + repository + .withdraw_content_lock(creator.clone(), path.clone(), &original, &expected) + .await + .unwrap(), + TombstoneReadback::Exact + ); + + let mut replacement = original.clone(); + replacement.created_at = datetime!(2026-08-12 05:00:01 UTC); + content_locks + .upsert_content_lock(creator.clone(), path.clone(), replacement.clone()) + .await + .unwrap(); + + assert_eq!( + repository + .read_tombstone(&creator, &path, &expected) + .await + .unwrap(), + TombstoneReadback::Replaced + ); + assert_eq!( + content_locks + .get_content_lock(&creator, &path) + .await + .unwrap(), + Some(replacement) + ); +} + +#[tokio::test] +async fn memory_force_delete_removes_tombstone_and_missing_retry_succeeds() { + let store = InMemoryPublicContentLockStore::new(); + let content_locks = InMemoryContentLockRepository::with_public_store(store.clone()); + let repository = InMemoryContentLockTombstoneRepository::with_public_store(store); + let creator = creator(); + let path = path(); + let original = original_content_lock(); + let expected = tombstone_at_five(); + content_locks + .upsert_content_lock(creator.clone(), path.clone(), original.clone()) + .await + .unwrap(); + repository + .withdraw_content_lock(creator.clone(), path.clone(), &original, &expected) + .await + .unwrap(); + + repository + .force_delete_content_lock_and_verify_absent(&creator, &path) + .await + .unwrap(); + repository + .force_delete_content_lock_and_verify_absent(&creator, &path) + .await + .unwrap(); + assert_eq!( + repository + .read_tombstone(&creator, &path, &expected) + .await + .unwrap(), + TombstoneReadback::Missing + ); +} + +#[tokio::test] +async fn pubky_withdrawal_writes_canonical_bytes_to_exact_path_and_reads_them_back() { + let repository = PubkyContentLockTombstoneRepository::new(FakePubkyStorage::default()); + let creator = creator(); + let path = path(); + let original = original_content_lock(); + let expected = tombstone_at_five(); + repository + .client() + .replace_bytes(original.canonical_json_bytes().unwrap()); + + assert_eq!( + repository + .withdraw_content_lock(creator.clone(), path.clone(), &original, &expected) + .await + .unwrap(), + TombstoneReadback::Exact + ); + assert_eq!( + repository.client().written_bytes(), + serde_json::to_vec(&expected).unwrap() + ); + assert_eq!( + repository.client().operations(), + vec![ + format!("get_bytes {creator} {path}"), + format!("put_bytes {creator} {path} application/json"), + format!("get_bytes {creator} {path}"), + ] + ); + + assert_eq!( + repository + .withdraw_content_lock(creator.clone(), path.clone(), &original, &expected) + .await + .unwrap(), + TombstoneReadback::Exact + ); + assert_eq!( + repository + .client() + .operations() + .iter() + .filter(|operation| operation.starts_with("put_bytes ")) + .count(), + 1 + ); +} + +#[tokio::test] +async fn pubky_readback_classifies_missing_and_non_tombstone_replacement_as_raw_bytes() { + let repository = PubkyContentLockTombstoneRepository::new(FakePubkyStorage::default()); + let creator = creator(); + let path = path(); + let expected = tombstone_at_five(); + + assert_eq!( + repository + .read_tombstone(&creator, &path, &expected) + .await + .unwrap(), + TombstoneReadback::Missing + ); + + repository + .client() + .replace_bytes(br#"{"version":1,"creator":"not-a-tombstone"}"#.to_vec()); + assert_eq!( + repository + .read_tombstone(&creator, &path, &expected) + .await + .unwrap(), + TombstoneReadback::Replaced + ); + + assert_eq!( + repository + .withdraw_content_lock( + creator.clone(), + path.clone(), + &original_content_lock(), + &expected, + ) + .await + .unwrap(), + TombstoneReadback::Replaced + ); + assert!( + repository + .client() + .operations() + .iter() + .all(|operation| !operation.starts_with("put_bytes ")) + ); +} + +#[tokio::test] +async fn pubky_force_delete_removes_original_bytes_tombstone_and_replacement_with_missing_retries() +{ + let repository = PubkyContentLockTombstoneRepository::new(FakePubkyStorage::default()); + let creator = creator(); + let path = path(); + + for bytes in [ + br#"{"version":1,"creator":"original-content-lock"}"#.to_vec(), + serde_json::to_vec(&tombstone_at_five()).unwrap(), + br#"replacement bytes that are not json"#.to_vec(), + ] { + repository.client().replace_bytes(bytes); + repository + .force_delete_content_lock_and_verify_absent(&creator, &path) + .await + .unwrap(); + } + + repository + .force_delete_content_lock_and_verify_absent(&creator, &path) + .await + .unwrap(); + let expected_last = format!("get_bytes {creator} {path}"); + assert_eq!( + repository.client().operations().last(), + Some(&expected_last) + ); +} + +#[derive(Debug, Default)] +struct FakePubkyStorage { + bytes: Mutex>>, + written_bytes: Mutex>>, + operations: Mutex>, +} + +impl FakePubkyStorage { + fn replace_bytes(&self, bytes: Vec) { + *self.bytes.lock().unwrap() = Some(bytes); + } + + fn written_bytes(&self) -> Vec { + self.written_bytes.lock().unwrap().clone().unwrap() + } + + fn operations(&self) -> Vec { + self.operations.lock().unwrap().clone() + } +} + +#[async_trait] +impl PubkyHomeserverStorageClient for FakePubkyStorage { + async fn put_json_value_as_creator( + &self, + _creator: &CreatorPubky, + _path: &str, + _body: serde_json::Value, + ) -> Result<(), ApplicationError> { + unreachable!("tombstones must use raw canonical bytes") + } + + async fn get_json_value_as_creator( + &self, + _creator: &CreatorPubky, + _path: &str, + ) -> Result, ApplicationError> { + unreachable!("tombstones must not be parsed as JSON or ContentLock") + } + + async fn put_bytes_as_creator( + &self, + creator: &CreatorPubky, + path: &str, + bytes: Vec, + content_type: &str, + ) -> Result<(), ApplicationError> { + self.operations + .lock() + .unwrap() + .push(format!("put_bytes {creator} {path} {content_type}")); + *self.written_bytes.lock().unwrap() = Some(bytes.clone()); + *self.bytes.lock().unwrap() = Some(bytes); + Ok(()) + } + + async fn get_bytes_as_creator( + &self, + creator: &CreatorPubky, + path: &str, + ) -> Result, ApplicationError> { + self.operations + .lock() + .unwrap() + .push(format!("get_bytes {creator} {path}")); + Ok(self + .bytes + .lock() + .unwrap() + .clone() + .map(|bytes| PubkyBytesResource { + bytes, + content_type: Some("application/json".to_owned()), + })) + } + + async fn delete_as_creator( + &self, + creator: &CreatorPubky, + path: &str, + ) -> Result<(), ApplicationError> { + self.operations + .lock() + .unwrap() + .push(format!("delete {creator} {path}")); + *self.bytes.lock().unwrap() = None; + Ok(()) + } +} + +fn creator() -> CreatorPubky { + CreatorPubky::from_str("pubkytkrq8zmwb8a3m9k15csu3q17qmfgqnp9dskbrg9uq1rydpyxp7qy").unwrap() +} + +fn path() -> ContentLockPath { + ContentLockPath::from_lock_id(LockId::from_str(LOCK_ID).unwrap()) +} + +fn tombstone_at_five() -> ContentLockDeletionTombstone { + ContentLockDeletionTombstone::new( + LockId::from_str(LOCK_ID).unwrap(), + datetime!(2026-08-12 05:00:00 UTC), + ) +} + +fn original_content_lock() -> locks_core::lock_policy::ContentLock { + serde_json::from_value(serde_json::json!({ + "version": 1, + "creator": creator(), + "primary_resource": null, + "secondary_resources": {}, + "criteria": [], + "lock_logic": { "type": "all", "criteria": [] }, + "access_policy": { "requested_credential_ttl_seconds": 900 }, + "lock_server": { "override": null }, + "created_at": "2026-08-12T04:00:00Z" + })) + .unwrap() +}