bug: fix panel bugs #507, #521, #567, #583, #613, #617, #618, #619

[lukepolo]

Fixes the open [BUG] issues from 5stack-panel #507 onward that need api changes. The web side is 5stackgg/web#576, and the Swiftly prefix fix (#616) is 5stackgg/game-server#181.

Each fix started with a test that reproduced the bug and failed, then went green.

Suite	Before	After
yarn test:unit	52 suites / 844 tests	59 suites / 927 tests
yarn test:sql (fresh container)	1321 tests	1361 tests

yarn typecheck:tests is clean.

Deploy notes

• Migration 1886000000619_tournament_substitutes_enabled (#619), then hasura metadata apply (new column on tournaments).
• Order: deploy api before web. The web tournament page selects substitutes_enabled.
• Functions, triggers and views re-apply on boot. There is no other migration or metadata change.
• First sweeps after deploy: ReconcileOnDemandServerJobs (#567) will delete every orphaned m-* match Job already in the cluster. Expect a burst of "removing orphaned on demand server job" warnings.

---

#507 / #617 / #618 Round robin stage advancement

#507 Round robin as stage 1 rejects valid group counts

Problem

• A 4-group round robin feeding an 8-team double elimination fails at 20 teams with Invalid Groups (4) for stage 1.
• It only fails when the RR's min changes, which is why min 16 / max 20 worked.

Root cause

• Saving stage 1 applied the elimination "halving" rule to later stages: the DE min was forced to 20/2 = 10, and then 10 % groups was required to be 0. Halving means nothing after a round robin, because how many advance is set by the next stage's size.
• The same rule inflated a round robin when a later stage was added: RR 4–8 plus SE 4 forced the RR min to 8.
• Seeding only mapped whole placements, so there were no "best runner-up" slots. With 6 groups feeding 8 teams, the last two seeds went to the G1 and G2 runners-up regardless of record.

Fix

• Halving is skipped across a round robin or a no-elimination Swiss.
• The group-divisibility check is skipped for round robins.
• A stage after either of those takes 2 up to that stage's max_teams, and that stage can't shrink below the next stage's size.
• New get_stage_qualifier_seeds:
  • Whole placements are filled first, cross-seeded by group, exactly as before.
  • Leftover seeds go to the best next-placement teams across all groups, ranked by win rate → map ratio → round ratio → KDR → seed → id.
  • Win rate rather than wins, because uneven groups play different numbers of matches.
• get_team_at_stage_rank had no other users and is dropped.

#617 Only 8 or more teams could advance to the next stage

Problem: a round robin couldn't feed a small playoff, for example two group winners into a single final.

Root cause

• validate_tournament_stage required the next stage's min to be at least the previous stage's last-round match count, groups × ⌊teams per group / 2⌋. That number means nothing for a round robin.
• The same check broke league playoffs built by start_league_season for any division of 6 or more teams, in both the round robin and no-elimination Swiss shapes.

Fix: after a round robin or no-elimination Swiss, only the next stage's size is validated, from 2 up to the previous stage's max.

#618 Round robin only accepted groups of 4 or more

Problem: 4 groups with 15 teams (groups of 4, 4, 4 and 3) was rejected.

Root cause: the first-stage rule required 4 teams per group for every stage type except Swiss.

Fix

• Round robins need 3 teams per group at any stage order. The group draw already builds uneven groups.
• SE and DE first stages still need 4 per group.

Also fixed in the same code

Swiss 16 → SE 8 advanced 3 teams

• get_stage_team_counts only special-cased round robins. For a Swiss it counted the last round, which is just the 2-2 pool (3 matches).

Valve Swiss was validated by the wrong round

• Too many teams: Swiss 16 → SE 16 passed validation and would have seeded 0-3 teams into the playoff.
• Too strict a minimum: the minimum came from the 2-2 last round, so Swiss 32 → SE 4 was rejected.
• Resizing: halving across the Swiss resized the playoff when the Swiss was edited.
• Now: a Valve Swiss is validated like the other ranked stages.
  • The next stage takes 2 up to ⌊max/2⌋, which is what generate_swiss_bracket's 3-0, 3-1 and 3-2 pools add up to.
  • Halving is skipped across the Swiss.
  • The playoff is sized from the Swiss's actual round-1 match count.

Tests

• test/tournament-group-stages.spec.ts (new, 11 tests):
  • resizing a 4-group RR to 20 teams
  • no RR inflation when a later stage is added
  • RR can't advance more teams than it holds
  • 6 groups → 8: winners plus the 2 best runners-up
  • 8-team RR and 10-team no-elimination Swiss → 2-seat playoff (the league shapes)
  • two group winners play a single final to Finished
  • 15 teams in groups of 4, 4, 4, 3 play to Finished
  • wildcards compare win rate
  • groups under 3 teams are rejected
  • shrinking below the next stage is rejected
• test/tournament-stages.spec.ts:
  • the halving test now uses an SE stage 1
  • Swiss 16 → SE 8 advances 8
  • Valve Swiss cap and 3-win-half minimum
  • resizing a Swiss leaves the next stage alone
  • under-filled Swiss sizing

Known limits

• No same-group rematch avoidance on odd layouts. It's no worse than before.
• Existing tournaments where a later stage's max is above the round robin's max will reject edits until the sizes are fixed.

---

#619 Stand-ins are possible in a Duel tournament

Problem

• Duel tournaments let teams add stand-ins: registration rostered up to 3 players, and every match lineup had 3 slots.
• There was also no way to turn substitutes off for any tournament.

Root cause

• Lineup capacity is the match type's minimum plus match_options.number_of_substitutes, with no Duel exception.
• The tournament forms stamp the global team substitute limit (default 2) onto the tournament's options whatever the type. That count is then cloned into stage and bracket match options, giving 1 + 2 = 3.

Fix

• New tournaments.substitutes_enabled, NOT NULL DEFAULT true, so every tournament keeps substitutes unless an organizer turns them off.
• tournament_max_players_per_lineup and match_max_players_per_lineup add 0 substitutes for Duel tournaments, or when the setting is off.
  • For matches the check reads the tournament, so editing a match's own copy of the options can't reopen slots.
  • Rostering, auto-fill, match seating, the lineup-count checks and the web slot UIs all go through these functions.
  • Non-tournament Duel matches and draft games are unchanged.
• The roster trigger now enforces the cap only when a player is added to a team. A roster already over a lowered cap can still check in and drop players.

Tests: test/tournament-substitutes.spec.ts (new, 11 tests). 10 failed before the fix; the guard (a non-tournament Duel match keeps its substitute slots) passed throughout.

---

#613 Manually added awards don't count for the leaderboard

Problem: hand-granted awards never appeared on the Awards leaderboard.

Root cause

• _leaderboard_awards counted medals by award_recipients.placement, and dropped rows without a placement or without a tournament/season. This was a deliberate "hand-granted awards must not move rankings" rule, and grantAward never sets a placement.
• Separately, get_player_leaderboard_rank ranked awards by gold alone, while the board sorts mvp → gold → silver → bronze, so jump-to-my-rank could land on the wrong page.

Fix

• The rule is removed.
• Each row counts by COALESCE(placement_tier, award tier). Special-tier awards have no column and don't count.
• Unscoped hand grants appear only when no match type filter is set, and are placed in time windows by grant date.
• Team grants count once per rostered player, never for the team row.
• Calculated rows produce exactly the same numbers as before.
• The rank function uses the board's medal-table order.

Tests: test/awards.spec.ts › "medal leaderboard":

• counts by tier
• team grant fans out to players
• rolling windows
• season scoping (2 tests)
• special tier left off
• match-type filter
• placement override (guard)
• rank order

Also added two guard tests for the web-only #609 and #610: a self-grant is allowed, and a team grant without a tournament stores the team row plus one row per roster player.

---

#567 Match jobs orphaned

Problem

• On-demand match servers (Jobs named m-<matchId>) could keep running after their match ended, was deleted or moved.
• Even idle, a leftover pod holds its CPU request (static CPU manager) and host ports, while the database already counts the slot as free.

Root cause: stopping relied on:

• a single event pass
• a single exec'd SIGUSR1
• a database release that ran whether or not the stop worked

Nothing ever checked the cluster, and several paths skipped the stop or sent it to the wrong server.

Fix, by orphan path

1. The graceful stop only sent a signal and swallowed failures.

• Before: an exec into a pod that wasn't Running failed silently. While setup.sh is PID 1 the signal is accepted and ignored. Kubernetes errors were logged, the row freed and the job resolved, so it never retried.
• Now:
  • A Job with no running pod is deleted.
  • A running pod is signalled, and a check-back tied to the Job's uid removes it after 2 minutes if it's still running.
  • Kubernetes errors are rethrown (5 attempts with backoff).
  • The row is released only once teardown succeeds.

2. Deleting a match never stopped its server.

• Before: Hasura sends DELETE with new = {}, so data.new.server_id was undefined.
• Now: it's read from old.

3. A cancel during assignment.

• Before: a cancel arriving before server_id was written stopped nothing, and the late write then tried to signal a pod that was still being created.
• Now:
  • Status is re-read inside the lock.
  • server_id is only written if the match hasn't ended; otherwise the new Job is removed and the row freed.
  • assignServer stops before the dedicated fallback for an ended match.

4. A reboot or reassignment freed the NEW server.

• Before: Job names are per match, so "server was removed" signalled the replacement's pod and freed every row the match held.
• Now:
  • Moving to another on-demand server frees only the old row, with no signal.
  • Moving to a dedicated server stops the old Job and frees only the old row.

5. End-of-match side-effect errors skipped the stop.

• Before: Discord, matchmaking, scrim, camera, ELO or voice errors, or a null server.is_dedicated, prevented the stop from being queued.
• Now:
  • The stop is queued first, and the is_dedicated read is null-safe.
  • Practice matches queue their stop before marking the session ended.

6. A late or retried stop killed a restarted match.

• Before: the stop waits out tv_delay and backoff on a single-worker queue, so an organizer can start the match again before it runs.
• Now:
  • The stop does nothing if the match exists and hasn't ended.
  • It only frees rows whose match is still ended when the write happens.
  • It only removes the exact Job (by uid) that it read.

7. The assignment lock expired mid-assignment.

• Before: the 10s lock expired inside a critical section of 15s or more.
• Now:
  • The lock is 45s: longer than the one 15s teardown, and short enough to expire within the retry window.
  • Failing to get the lock raises FailedToCreateOnDemandServer, so the existing retry loop runs.

8. remove=true deleted pods before the Job.

• Before: the Job controller replaced the deleted pods.
• Now: the Job is deleted first, then its pods.

Backstop: ReconcileOnDemandServerJobs, every minute

• Which Jobs it looks at: Jobs labelled app=game-server, role=match, match-id (labels added to new Jobs), or named m-<uuid> (existing Jobs).
• Skipped: finished Jobs, Jobs younger than 5 minutes, and non-match Jobs.
• Deletes a Job when:
  • its match row is gone, or
  • the match ended longer ago than ended_at/cancels_at + tv_delay + 10 min, or
  • the match hasn't ended but no longer holds a server, 5 minutes after the sweep first sees it.
• Safety:
  • Deletes use a uid precondition, so a Job replaced after the listing is left alone.
  • Only pods owned by that uid are deleted.
  • Rows are freed only after the sweep removed the Job, and the update re-checks the reservation.
  • It doesn't wait for deletion, so it can't stall veto auto-picks on the shared queue.
  • If listing Jobs fails, nothing is deleted.

Tests: 5 new specs:

• matches.controller.match-events
• match-assistant.on-demand-teardown
• match-assistant.reconcile-jobs
• StopOnDemandServer
• ReconcileOnDemandServerJobs

36 tests failed on the unfixed code, then 15 more on the first version of the fix after review. All 60 pass.

Open questions

• activeDeadlineSeconds for a match that hasn't ended but whose pod never boots (still kept today).
• backoffLimit: 10.
• Logs are lost for Jobs removed without a clean stop.
• An owner-token lock release in cache.service.ts.

---

#521 Playcast live stream fails

Problem: in TV mode with use_playcast, the game streamer never showed the match; the relay answered 404 "broadcast not found".

Root cause

• Since 600c87c, GameStreamerService.buildConnectEnv hardcoded PLAYCAST_URL=https://tv.5stack.gg/<matchId>.
• The game server broadcasts to https://${RELAY_DOMAIN}/<matchId>. So any install other than 5stack.gg (DEAFCS, dev) asked a relay with no broadcast for that match, both at pod start and on stream switching.
• The relay had also drifted from Valve's reference relay:
  • signup_fragment was always 0.
  • /sync returned tick/tps etc. as strings.
  • No 205 while the start data was still arriving.

Fix

• PLAYCAST_URL is built from appConfig.relayDomain.
• The relay records the real signup fragment and moves it when start is re-sent.
• It stores numeric query values as numbers, and returns 205 until the start data exists.

Tests

• game-streamer.service.spec.ts (relay domain). Before the fix it got https://tv.5stack.gg/m-1.
• New match-relay.service.spec.ts:
  • signup fragment and numeric /sync
  • start served only at the signup fragment
  • re-sent start
  • both 205 cases

---

#583 Playcast matches do not get marked as finished

Evidence (match f14e4aca, from the attached logs):

Time	Event
20:58:40	Knife → Live; cancels_at = NOW() + live_match_timeout armed
21:28:24	Game over 13-10; WaitingForTV published with the winner; "deferring HandleEndOfMap by 115s"
21:28:31–44	Players disconnect
21:29:19	Received signal to stop the match (SIGUSR1). HandleEndOfMap, UploadingDemo and Finished never happen

Root cause

• The only sender of SIGUSR1 is stopOnDemandServer, which fires with no delay when a match is Canceled. The timing fits a 30-minute live_match_timeout and the 21:29 run of CancelExpiredMatches.
• Nothing extended cancels_at once the map reached WaitingForTV.
• isAwaitingWarmup counted only Knife, Live, Overtime and Paused as started. So the played match:
  • looked like an unstarted lobby missing players
  • was cancelled ("not everyone showed up")
  • wrote abandoned_matches rows for everyone who left after the final round
  • had its server stopped immediately
  • would have been deleted by RemoveCancelledMatches a day later
• The plugin can't resume after a reload, because MatchManager stops at WaitingForTV.

Fix

• SQL:
  • Entering WaitingForTV or UploadingDemo pushes cancels_at to NOW() + tv_delay + live_match_timeout (Live matches only).
  • Finishing a map with more of the series to play gives the next map the Warmup window (auto_cancel_duration).
• Job:
  • End-of-map maps count as played, so there's no force start and no no-show rows.
  • An expired match stuck at the end of a map with a stored winner has that map set Finished, so update_match_state settles the match. The server is asked for the next map only if the match is still Live.
• Once a map has been played:
  • No-show penalties never apply.
  • A stalled tournament series goes to an organizer (MatchSupport notification) instead of a forfeit decided by readiness or coin toss. is_ready is check-in state, and a Canceled tournament match never advances the bracket.
  • A stalled non-tournament series is cancelled without penalties.
• MatchMapStatusEvent ignores a late WaitingForTV/UploadingDemo/Finished aimed at a map that isn't in play, so it can't overwrite the next, unplayed map.

Tests

• CancelExpiredMatches.spec.ts:
  • finishing a stuck map
  • no force start
  • no penalties after a played map
  • the second pass of a series
  • the organizer cases
• New MatchMapStatusEvent.spec.ts
• test/match-scoring.spec.ts › "the end-of-map window": both cancels_at windows, plus a guard that finishing a WaitingForTV map finishes the match.

Follow-ups, not in this PR

• game-streamer: poll /sync and retry before playcast.
• game-server: resume HandleEndOfMap after a plugin reload.
• The mp_match_restart_delay race on game nodes.
• Re-arming cancels_at on each round.