Repository navigation
Add cluster_pause: every server pauses together while nobody plays, with Surface Export holding the pause for work in flight - #422
Merged
Conversation
… a cross-plugin state event and hold protocol
plugins/cluster_pause sits outside docker/seed-data/external_plugins, so the
development containers neither install nor load it. The controller counts online
players from playerEvent, runs running -> countdown -> draining -> paused, collects
holds from in-process participants (found through Symbol.for("cluster_pause.rendezvous")
on the controller), pauses every running instance that loads the plugin once no hold
remains, and resumes on any join or command. ClusterPauseStateEvent goes to
subscribed control connections, every instance and the participants. The instance
plugin applies game.tick_paused through its cluster_pause remote interface. The ctl
plugin adds cluster-pause status|pause|resume [--watch].
…lds the pause for work in flight and refuses new work while the cluster pauses The participant registers through the cluster_pause rendezvous on the controller object, without importing that package; when cluster_pause is not loaded it never registers and nothing changes. It holds for unresolved or queued operations, pending intents, passengers in transit, admission requests in progress, instances reconciling their save, in-progress resolutions and unreadable transfer state. While draining or paused, new transfers, uploaded imports, download exports and new lineage resolutions are refused, as is new work for an instance whose resume is unconfirmed. Queue-admitted transfers, replays of known work and recovery are not refused.
…h its module mounted by run-lua-tests
…w plugin with the Surface Export toolchain; CI runs it and ctl.mjs reads cluster-pause status The cluster_pause package has no lockfile of its own. The build container mounts the same dependency volume at its node_modules, after the existing Surface Export lock check; CI links Surface Export's installed node_modules. surface_export stays the default target, and cluster_pause refuses web, smoke, staged packages and restarts.
…s starting, until its startup recovery finishes
…pause targets before ordering, and keeps a stopped paused instance listed An instance that cannot run the remote interface (save patching or script commands off, or started before the plugin) never reports ready, so it is neither paused nor refused work through a resume that can never be confirmed; status lists it under notReady, and instances re-announce on a new controller connection. The targets are written to cluster_pause.json before any pause order, so a controller killed mid-pause still resumes them, and a failed write sends no order. An instance stopped while paused stays listed until it confirms a resume; a ready report resumes it even while the plugin is turned off.
…, no longer lets a new request past the cluster pause
…sume retries instead of removing ids a newer attempt is pausing
… phases with a transfer in flight, and a client join on two disposable instances
…gainst the transfer in the controller log
…nstance whose game is paused, and warns when a save loads paused without cluster_pause
…wn one stays unknown
… saves and safe removal, and the passenger reconnect gap
…a paused game at runtime and after a paused restart, then a completed transfer after the unpause
…r is pausing; an unknown paused state names an outdated plugin; docs state the paused-game check's scope and every phase exit
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem and change
Surface Export refuses transfers to and from a server with Factorio's
auto_pauseon, so the pve7 servers run while empty. Over about three weeks unattended, enemy evolution reached 0.88 and a fusion-powered platform burned all its fuel.This adds
cluster_pause, a separate Clusterio plugin inplugins/cluster_pause/. It pauses every server together once nobody is on any server, and any join resumes all of them.running→countdown(cluster_pause.countdown_ms, 30000) →draining→paused. During draining, participants stop admitting new work, and the plugin waits until their holds clear before settinggame.tick_pausedon every ready instance. A join, theresumecommand, turning the feature off, or an unknown instance state returns the cluster torunning.cluster_pause.enabled(defaultfalse),cluster_pause.countdown_ms,cluster_pause.hold_check_interval_ms.clusterioctl cluster-pause status|pause|resume [--watch].ctl.mjstreatsstatusas read-only.lib/cluster-pause-participant.ts).game.tick_paused(measured below). The check:pausingwindow cancels the pause, and is now sent a resume as well. Before, a save paused by something else stayed frozen in that case.docs/admins/cluster-pause.mdcovers settings, phases, commands, install, paused saves, the safe removal procedure, the paused-game check and the passenger reconnect gap. It is linked fromdocs/README.md.build-plugin.ps1 -Plugin cluster_pausebuilds, lints and tests it with Surface Export's toolchain, and CI runs it.tests/lua/cluster-pause.luais in the Lua list.tests/manual/cluster-pause/(with a small additive option indocker-lab.mjs) is the disposable lab used below.The branch was the local
feat/cluster-pause(d6394510..b6b49273, based on 779bf9d). It is rebased onto main without conflicts. The devDependencies now pin Clusterio 2.0.2.Verification
Checks at the PR head (2a0d50d):
build-plugin.ps1 lintbuild-plugin.ps1 testbuild-plugin.ps1 lint -Plugin cluster_pausebuild-plugin.ps1 test -Plugin cluster_pauserun-lua-tests.ps1(full CI list, includingcluster-pause.lua)npm testcheck-pr-scope.ps1 -Base origin/mainGuard mutants. Each was applied in this worktree, run through the isolated test build, then restored. Every one was killed by the test aimed at it:
tickPaused.falsefor an unreadable state.pausing-window resume is removed. Its test fails on the code before the fix, and 22 of 22 pass with it.Engine behaviour on pinned Factorio 2.1.20
Lab run
se-manual-mvbpmseg-cde437df, verdict PASS, cleanup succeeded:pausing-window resume (unit-tested above) and docs.game.tick,game.tick_pausedand fixture state read on the servers over RCON, not from the plugins' own reports.se-manual-mvbntpoi-f7b9f879at a9f79be, gave the same engine, cluster and client results.game.tick_pausedon a dedicated server?/sctook 670–699 ms, the same as while running; storage writes and chat worked. An RCON unpause advanced 161 ticks in 2 s.InGameandPlayerJoinGamewas processed.connected_playerswas 1 and the tick stayed frozen (0 ticks in 4 s). After an RCON unpause the game ran at about 60 UPS (299 ticks in 4 s, 654 in 10 s) with the player still connected. The client log had no desync, drop or error lines, and screenshots show the client in-game.tick_paused?game.server_savewhile paused wrote a valid save. Without cluster_pause, an instance started from it stayed frozen (0 ticks in 4 s). With cluster_pause loaded, it was unpaused when it reported ready.Cluster behaviour (same run)
enabled=true:countdown(29.4 s remaining at first read), thendraining, thenpaused, confirmed by both instances. Both servers advanced 0 ticks in 4 s.start-transfer, thencluster-pause pausewas run. The reply wasdrainingwith the holdtransfer preparing <request id>. Controller log order: draining with the hold, thenexport_requested, thentransfer_completed, thenpaused (holds_cleared).start-transferwas refused: "The cluster is paused because no players are online … Nothing was locked or exported." The second fixture was still present, unlocked, with unchanged cargo.running (player_joined), and both servers advanced (281 and 284 ticks in 4 s).countdown (last_player_left), thenpaused31.5 s later, with 0 ticks advanced on both servers.resumeunpaused both servers (162–163 ticks in 2 s).statusand--watch.statusreturns the phase, holds, confirmed and paused instances, andnotReady/notLoaded.status --watchprinted 33 lines, including each phase change and the draining hold.Paused-game check (same run, cluster pause turned off)
/sc game.tick_paused = true(0 ticks in 4 s).start-transferwas refused: "The destination instance "Dev Two" … has its game paused (game.tick_paused), so work there would stop halfway. Work was not admitted. … Nothing was locked or exported." No transfer was recorded, and the source was present, unlocked, with unchanged cargo.cluster_pause.load_plugin=false. The game stayed frozen (0 ticks in 4 s), the startup warning was logged, and the same refusal fired with the source unchanged./sc game.tick_paused = false(161 ticks in 2 s), the retried transfer completed, its destination cargo matched the contract, and the source was removed.Independent review
Read-only reviewing agents applied the data-integrity checklist twice: to the rebased branch, and to the paused-game delta. Neither found a cargo or ownership defect. They confirmed the following:
startDrainingpublishes the phase to participants before holds are read.safeToUnlockSource.tick_paused.The second review's doc corrections, its message wording and the
pausing-window bug are fixed in 2a0d50d. The remaining points are under Limits.Limits
cluster-pause resume, checkpausedis empty,enabled=false, then remove.cluster_pause.jsonwhile paused. The controller forgets which instances it paused. Clusterio 2.0.2 sendsconnectto instance plugins after a controller restart (connectors.js:ready→connect), so each instance re-announces ready and is resumed.on_player_joined_gamehandler, and joins resumed a paused cluster here. Join handlers therefore run on a paused server.research_sync,subspace_storageandplayer_syncwere loaded in the lab without errors but are not participants.tools/surface-export/import-blueprint.mjsandexport-blueprint.mjsdrive RCON directly and bypass both checks.require_user_verification).Build and deployment
See
docs/admins/cluster-pause.md. In short:plugins/cluster_pause/is outsidedocker/seed-data/external_pluginson purpose. Clusterio loads every package inexternal_plugins/(findLocalPlugins), and the development cluster mounts that directory../tools/clusterio/build-plugin.ps1 node -Plugin cluster_pause -OutputDirectory ci-artifacts/<dir>.dist/, withmodule/andpackage.json, into the cluster'sexternal_plugins/cluster_pause/. Restart the controller and hosts, then setcluster_pause.enabled true.@solarcloud7/plugin-cluster-pausepublished, an entry in its desired state (vm.json), andcluster_pause.enabled: true. Its Surface Export would also need this version, for the paused-game check.