Skip to content

Add configuration for graceful shutdown of application services - #935

Open
eseliger wants to merge 2 commits into
mainfrom
syntax-highlighter-termination-grace
Open

Add configuration for graceful shutdown of application services#935
eseliger wants to merge 2 commits into
mainfrom
syntax-highlighter-termination-grace

Conversation

@eseliger

@eseliger eseliger commented Aug 29, 2026

Copy link
Copy Markdown
Member

Sourcegraph application services need an explicit shutdown contract so Kubernetes leaves enough time for endpoint propagation, request draining, and process cleanup. The chart currently relies mostly on implicit Kubernetes defaults and requires operators to know internal environment variable names.

This adds global chart settings:

  • sourcegraph.preShutdownPause (default 5s), rendered as SRC_PRE_SHUTDOWN_PAUSE
  • sourcegraph.gracefulShutdownTimeout (default 15s total, including the pause), rendered as SRC_GRACEFUL_SHUTDOWN_TIMEOUT
  • sourcegraph.terminationGracePeriodSeconds (default 30)

Frontend, Gitserver, Searcher, Worker, precise and syntactic code-intel workers, and syntax-highlighter inherit these settings. Each supports a service-level override using the same value name. Existing explicit entries in a service's env map take precedence, preserving current configurations.

The common timeline is:

  1. accept through t=5s
  2. drain through at most t=15s
  3. then retain Kubernetes margin until t=30s

Test plan

  • helm lint charts/sourcegraph
  • helm unittest -q charts/sourcegraph (122 tests)
  • Render tests cover global defaults, service overrides, explicit env precedence, worker replicas.

@eseliger
eseliger force-pushed the syntax-highlighter-termination-grace branch from 5b7af16 to b5327c7 Compare August 29, 2026 23:38
@eseliger eseliger changed the title fix/syntax-highlighter: configure termination grace period feat/kubernetes: configure graceful shutdown for Sourcegraph services Aug 29, 2026
@eseliger
eseliger force-pushed the syntax-highlighter-termination-grace branch from b5327c7 to b077f1e Compare August 29, 2026 23:39
Add global and per-service controls for the standard Sourcegraph pre-shutdown pause, total graceful shutdown timeout, and Kubernetes termination grace period. Syntax-highlighter uses the longer 5s/40s/45s profile required for large highlighting requests.

Cloud rollout notes: canary the released syntax-highlighter image first. Verify the rendered 45-second grace period and image digest, EndpointSlice transition timing, stabilizer shutdown logs, and zero terminal Frontend failures before rolling out to sourcegraph.com. Keep replica-count changes separate; sourcegraph.com already runs two replicas.

Amp-Thread-ID: https://ampcode.com/threads/T-01a04e71-0f57-74a8-91db-36cc3c22d05d
@eseliger
eseliger force-pushed the syntax-highlighter-termination-grace branch from b077f1e to 59f215f Compare August 29, 2026 23:57
@eseliger eseliger changed the title feat/kubernetes: configure graceful shutdown for Sourcegraph services feat/kubernetes: configure graceful shutdown for application services Aug 29, 2026
@eseliger eseliger changed the title feat/kubernetes: configure graceful shutdown for application services Add configuration for graceful shutdown of application services Aug 30, 2026
@eseliger
eseliger marked this pull request as ready for review August 30, 2026 00:02
@eseliger
eseliger requested review from a team and michaellzc August 30, 2026 00:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant