Skip to content

Add cluster_pause: every server pauses together while nobody plays, with Surface Export holding the pause for work in flight - #422

Merged
solarcloud7 merged 17 commits into
mainfrom
feat/cluster-pause
Oct 17, 2026
Merged

solarcloud7 merged 17 commits into
mainfrom
feat/cluster-pause

Conversation

@solarcloud7

@solarcloud7 solarcloud7 commented Oct 17, 2026 •

Copy link
Copy Markdown
Owner

Problem and change

Surface Export refuses transfers to and from a server with Factorio's auto_pause on, so the pve7 servers run while empty. Over about three weeks unattended, enemy evolution reached 0.88 and a fusion-powered platform burned all its fuel.

This adds cluster_pause, a separate Clusterio plugin in plugins/cluster_pause/. It pauses every server together once nobody is on any server, and any join resumes all of them.

  • Phases. running → countdown (cluster_pause.countdown_ms, 30000) → draining → paused. During draining, participants stop admitting new work, and the plugin waits until their holds clear before setting game.tick_paused on every ready instance. A join, the resume command, turning the feature off, or an unknown instance state returns the cluster to running.
  • Settings. cluster_pause.enabled (default false), cluster_pause.countdown_ms, cluster_pause.hold_check_interval_ms.
  • CLI. clusterioctl cluster-pause status|pause|resume [--watch]. ctl.mjs treats status as read-only.
  • Surface Export as an optional participant (lib/cluster-pause-participant.ts).
    • Holds: every unresolved transfer, pending intent, admission in progress, starting or reconciling instance and platform resolution.
    • While the cluster drains or is paused, new transfers, imports, exports and resolutions are refused. Admitted work, retries and recovery are not refused for the cluster pause.
    • Without cluster_pause loaded, Surface Export behaves as before, except for the paused-game check below.
  • Surface Export refuses a paused game, whatever paused it. Factorio saves keep game.tick_paused (measured below). The check:
    • Where it applies: when Surface Export admits a transfer, import or export; when a queued transfer starts; and before a transfer's import.
    • When it refuses: when the source or destination game is paused, or does not report whether it is (fail closed).
    • How it reads the state: the instance answers through the platform-list reply each admission already fetches.
    • Startup warning: an instance logs one when its save loads paused without cluster_pause.
    • Never unpauses: an admin's deliberate pause stays in place.
    • This was the owner's choice after the first review found that removing cluster_pause from a paused cluster left frozen servers that still admitted work.
  • Fix found in review. An instance that reports ready during the brief pausing window cancels the pause, and is now sent a resume as well. Before, a save paused by something else stayed frozen in that case.
  • Docs. docs/admins/cluster-pause.md covers settings, phases, commands, install, paused saves, the safe removal procedure, the paused-game check and the passenger reconnect gap. It is linked from docs/README.md.
  • Tooling.
    • build-plugin.ps1 -Plugin cluster_pause builds, lints and tests it with Surface Export's toolchain, and CI runs it.
    • tests/lua/cluster-pause.lua is in the Lua list.
    • tests/manual/cluster-pause/ (with a small additive option in docker-lab.mjs) is the disposable lab used below.

The branch was the local feat/cluster-pause (d6394510..b6b49273, based on 779bf9d). It is rebased onto main without conflicts. The devDependencies now pin Clusterio 2.0.2.

Verification

Checks at the PR head (2a0d50d):

Check Result
build-plugin.ps1 lint pass
build-plugin.ps1 test 1026 tests: 1013 pass, 0 fail, 13 skipped
build-plugin.ps1 lint -Plugin cluster_pause pass
build-plugin.ps1 test -Plugin cluster_pause 22 of 22 pass
run-lua-tests.ps1 (full CI list, including cluster-pause.lua) 279 PASS, 0 FAIL
Root npm test 920 tests: 899 pass, 0 fail, 21 skipped
check-pr-scope.ps1 -Base origin/main pass

Guard mutants. Each was applied in this worktree, run through the isolated test build, then restored. Every one was killed by the test aimed at it:

  • The admission check ignores tickPaused.
  • A paused game is not refused.
  • The instance reply reports false for an unreadable state.
  • The platform tree drops the state.
  • The startup warning ignores cluster_pause.
  • The pausing-window resume is removed. Its test fails on the code before the fix, and 22 of 22 pass with it.

Engine behaviour on pinned Factorio 2.1.20

Lab run se-manual-mvbpmseg-cde437df, verdict PASS, cleanup succeeded:

  • Setup: two disposable instances on the 2.0.2.r1 images, with packages built at c950875 (Surface Export 0.12.0-beta.3), plus the full 2.1.20 Linux client.
  • Since that build: only 2a0d50d is newer. It changes the unknown-state message wording, the pausing-window resume (unit-tested above) and docs.
  • How the results were read: every claim comes from game.tick, game.tick_paused and fixture state read on the servers over RCON, not from the plugins' own reports.
  • Reproducibility: an earlier run, se-manual-mvbntpoi-f7b9f879 at a9f79be, gave the same engine, cluster and client results.
Spec question Result
Can a script set and hold game.tick_paused on a dedicated server? Yes. 0 ticks advanced in two 4 s windows.
Do RCON commands run while paused? Yes. Each /sc took 670–699 ms, the same as while running; storage writes and chat worked. An RCON unpause advanced 161 ticks in 2 s.
Can a client join a paused server and resume cleanly? Yes. With the plugin off and the server paused by script, the client reached InGame and PlayerJoinGame was processed. connected_players was 1 and the tick stayed frozen (0 ticks in 4 s). After an RCON unpause the game ran at about 60 UPS (299 ticks in 4 s, 654 in 10 s) with the player still connected. The client log had no desync, drop or error lines, and screenshots show the client in-game.
Does a save keep tick_paused? Yes. game.server_save while paused wrote a valid save. Without cluster_pause, an instance started from it stayed frozen (0 ticks in 4 s). With cluster_pause loaded, it was unpaused when it reported ready.
Does Clusterio's stop still save while paused? Yes. The save hash changed, and a marker written while paused was present after restart.

Cluster behaviour (same run)

  • Countdown, then pause. After enabled=true: countdown (29.4 s remaining at first read), then draining, then paused, confirmed by both instances. Both servers advanced 0 ticks in 4 s.
  • Transfer in flight holds the pause. A transfer was admitted with start-transfer, then cluster-pause pause was run. The reply was draining with the hold transfer preparing <request id>. Controller log order: draining with the hold, then export_requested, then transfer_completed, then paused (holds_cleared).
    • The transfer completed, and the destination's physical cargo equals the independent fixture contract (inventories by quality, both belt lanes, fluid temperature).
    • The source platform was removed.
  • Admission while paused. start-transfer was refused: "The cluster is paused because no players are online … Nothing was locked or exported." The second fixture was still present, unlocked, with unchanged cargo.
  • Any join resumes all. A client joined Dev One. The cluster went to running (player_joined), and both servers advanced (281 and 284 ticks in 4 s).
  • Leave pauses again. When the client left: countdown (last_player_left), then paused 31.5 s later, with 0 ticks advanced on both servers.
  • resume unpaused both servers (162–163 ticks in 2 s).
  • status and --watch. status returns the phase, holds, confirmed and paused instances, and notReady/notLoaded. status --watch printed 33 lines, including each phase change and the draining hold.

Paused-game check (same run, cluster pause turned off)

  • Paused at runtime. Dev Two was paused with a plain /sc game.tick_paused = true (0 ticks in 4 s). start-transfer was refused: "The destination instance "Dev Two" … has its game paused (game.tick_paused), so work there would stop halfway. Work was not admitted. … Nothing was locked or exported." No transfer was recorded, and the source was present, unlocked, with unchanged cargo.
  • Restarted from the paused save with cluster_pause.load_plugin=false. The game stayed frozen (0 ticks in 4 s), the startup warning was logged, and the same refusal fired with the source unchanged.
  • After /sc game.tick_paused = false (161 ticks in 2 s), the retried transfer completed, its destination cargo matched the contract, and the source was removed.

Independent review

Read-only reviewing agents applied the data-integrity checklist twice: to the rebased branch, and to the paused-game delta. Neither found a cargo or ownership defect. They confirmed the following:

  • Every gated handler counts its admission before checking the gate, and admitted work registers its hold before that count drops.
  • startDraining publishes the phase to participants before holds are read.
  • Every refusal happens before an operation record, a lock or a dispatch, or it returns safeToUnlockSource.
  • Surface Export only reads tick_paused.
  • The new read cannot throw out of the platform-list handler, and it adds no new class of log spam.

The second review's doc corrections, its message wording and the pausing-window bug are fixed in 2a0d50d. The remaining points are under Limits.

Limits

  • The paused-game check also applies to admitted work when a queued transfer starts and before its import. If one RCON read fails there, the transfer fails and its source unlocks; if that unlock fails, the source stays locked for recovery (existing behaviour). This trades liveness for never importing into a frozen game.
  • Fail closed across versions. An instance whose Surface Export predates this check sends no paused state, so transfers, imports and exports there are refused until it restarts on the current version. The message says so. Upgrade the controller and hosts together.
  • Removing the plugin. A paused save outlives cluster_pause. Surface Export now refuses work there and the instance logs how to resume it, but nothing unpauses it automatically. The documented removal is: cluster-pause resume, check paused is empty, enabled=false, then remove.
  • Deleting cluster_pause.json while paused. The controller forgets which instances it paused. Clusterio 2.0.2 sends connect to instance plugins after a controller restart (connectors.js: ready → connect), so each instance re-announces ready and is resumed.
  • Passengers (accepted by the owner, documented). A passenger stays connected on the source during transit and is offline only while reconnecting to the destination (about 11 s measured from Connecting to InGame). If the reconnect outlasts the countdown, the cluster pauses, and the passenger's join resumes it.
    • Clusterio's join event comes from a Lua on_player_joined_game handler, and joins resumed a paused cluster here. Join handlers therefore run on a paused server.
    • Passenger deadlines are tick-based.
    • Not tested with a real passenger.
  • Lua-started route transfers. These are Gateway arrivals with nobody online. They lock and export in-game before the controller hears of them, so they are not holds, and a pause can freeze their export. The export only reads the source. On completion during the pause the controller refuses and the source unlocks over RCON. The platform then stays parked at the portal until someone acts.
  • Not shown in the platform tree. The tree does not display a paused game.
  • Not participants. research_sync, subspace_storage and player_sync were loaded in the lab without errors but are not participants. tools/surface-export/import-blueprint.mjs and export-blueprint.mjs drive RCON directly and bypass both checks.
  • Clients. A script pause shows the joining player no pause overlay; they see a frozen world until the join resumes it. The client used no Factorio account (the lab turned off require_user_verification).
  • Changelog. Surface Export's CHANGELOG is left to the next release-prep PR.

Build and deployment

See docs/admins/cluster-pause.md. In short:

  • Location. plugins/cluster_pause/ is outside docker/seed-data/external_plugins on purpose. Clusterio loads every package in external_plugins/ (findLocalPlugins), and the development cluster mounts that directory.
  • Build. ./tools/clusterio/build-plugin.ps1 node -Plugin cluster_pause -OutputDirectory ci-artifacts/<dir>.
  • Install. Copy the output as dist/, with module/ and package.json, into the cluster's external_plugins/cluster_pause/. Restart the controller and hosts, then set cluster_pause.enabled true.
  • pve7. It installs plugins from npm by version through homelab-iac, so it needs @solarcloud7/plugin-cluster-pause published, an entry in its desired state (vm.json), and cluster_pause.enabled: true. Its Surface Export would also need this version, for the paused-game check.
  • Status. Not merged, not deployed and not enabled anywhere.

… a cross-plugin state event and hold protocol

plugins/cluster_pause sits outside docker/seed-data/external_plugins, so the
development containers neither install nor load it. The controller counts online
players from playerEvent, runs running -> countdown -> draining -> paused, collects
holds from in-process participants (found through Symbol.for("cluster_pause.rendezvous")
on the controller), pauses every running instance that loads the plugin once no hold
remains, and resumes on any join or command. ClusterPauseStateEvent goes to
subscribed control connections, every instance and the participants. The instance
plugin applies game.tick_paused through its cluster_pause remote interface. The ctl
plugin adds cluster-pause status|pause|resume [--watch].
…lds the pause for work in flight and refuses new work while the cluster pauses

The participant registers through the cluster_pause rendezvous on the controller
object, without importing that package; when cluster_pause is not loaded it never
registers and nothing changes. It holds for unresolved or queued operations,
pending intents, passengers in transit, admission requests in progress, instances
reconciling their save, in-progress resolutions and unreadable transfer state.
While draining or paused, new transfers, uploaded imports, download exports and
new lineage resolutions are refused, as is new work for an instance whose resume
is unconfirmed. Queue-admitted transfers, replays of known work and recovery are
not refused.
…w plugin with the Surface Export toolchain; CI runs it and ctl.mjs reads cluster-pause status

The cluster_pause package has no lockfile of its own. The build container mounts the
same dependency volume at its node_modules, after the existing Surface Export lock
check; CI links Surface Export's installed node_modules. surface_export stays the
default target, and cluster_pause refuses web, smoke, staged packages and restarts.
…s starting, until its startup recovery finishes
…pause targets before ordering, and keeps a stopped paused instance listed

An instance that cannot run the remote interface (save patching or script commands
off, or started before the plugin) never reports ready, so it is neither paused nor
refused work through a resume that can never be confirmed; status lists it under
notReady, and instances re-announce on a new controller connection. The targets
are written to cluster_pause.json before any pause order, so a controller killed
mid-pause still resumes them, and a failed write sends no order. An instance stopped
while paused stays listed until it confirms a resume; a ready report resumes it even
while the plugin is turned off.
…, no longer lets a new request past the cluster pause
…sume retries instead of removing ids a newer attempt is pausing
… phases with a transfer in flight, and a client join on two disposable instances
…nstance whose game is paused, and warns when a save loads paused without cluster_pause
… saves and safe removal, and the passenger reconnect gap
…a paused game at runtime and after a paused restart, then a completed transfer after the unpause
…r is pausing; an unknown paused state names an outdated plugin; docs state the paused-game check's scope and every phase exit
@solarcloud7
solarcloud7 merged commit ed0c42f into main Oct 17, 2026
7 checks passed
@solarcloud7
solarcloud7 deleted the feat/cluster-pause branch October 17, 2026 02:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant