Skip to content

[chore][PLAT-1440] Raise termination grace periods so in-flight SLFs can finish #365

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
nataliechen-pixel merged 3 commits into from
Aug 20, 2026
Merged

[chore][PLAT-1440] Raise termination grace periods so in-flight SLFs can finish #365

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
675a147
[PLAT-1440] Raise termination grace periods so in-flight SLFs can finish
nataliechen-pixel Aug 20, 2026
240ff3d
[chore][PLAT-1440] Bump chart version to 6.11.21
nataliechen-pixel Aug 20, 2026
d79acad
[chore][PLAT-1440] Unify JS executor termination grace on 391
nataliechen-pixel Aug 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion charts/retool/Chart.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ apiVersion: v2
name: retool
description: A Helm chart for Kubernetes
type: application
version: 6.11.20
version: 6.11.21
maintainers:
- name: Retool Engineering
email: [email protected]
Expand Down
2 changes: 1 addition & 1 deletion charts/retool/templates/deployment_backend.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ spec:
{{ toYaml .Values.ui.labels | indent 8 }}
{{- end }}
spec:
terminationGracePeriodSeconds: {{ .Values.backend.terminationGracePeriodSeconds | default 136 }}

@mako-taco mako-taco Aug 20, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it possible to add comments in the YAML to explain how all of these magic numbers were arrived at?

@mako-taco mako-taco Aug 20, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah I see... they're documented in the default values. Do we need the | still, considering we have default values checked in?

terminationGracePeriodSeconds: {{ .Values.backend.terminationGracePeriodSeconds | default 391 }}
serviceAccountName: {{ template "retool.serviceAccountName" . }}
{{- if .Values.priorityClassName }}
priorityClassName: "{{ .Values.priorityClassName }}"
Expand Down
1 change: 1 addition & 0 deletions charts/retool/templates/deployment_code_executor.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ spec:
{{ toYaml .Values.codeExecutor.labels | indent 8 }}
{{- end }}
spec:
terminationGracePeriodSeconds: {{ .Values.codeExecutor.terminationGracePeriodSeconds | default 391 }}
serviceAccountName: {{ template "retool.serviceAccountName" . }}
{{- if .Values.priorityClassName }}
priorityClassName: "{{ .Values.priorityClassName }}"
Expand Down
1 change: 1 addition & 0 deletions charts/retool/templates/deployment_js_executor.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ spec:
{{ toYaml .Values.rr.jsExecutor.labels | indent 8 }}
{{- end }}
spec:
terminationGracePeriodSeconds: {{ .Values.rr.jsExecutor.terminationGracePeriodSeconds | default 391 }}
serviceAccountName: {{ template "retool.serviceAccountName" . }}
{{- if .Values.priorityClassName }}
priorityClassName: "{{ .Values.priorityClassName }}"
Expand Down
13 changes: 9 additions & 4 deletions charts/retool/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -432,9 +432,8 @@ backend:
labels: {}

# Make backend pods wait the specified time before kubernetes will forcibly kill them when they need to be rescheduled
# By default this is a 2 minutes + 15 seconds + 1 second (136) grace period to allow outstanding queries to complete
# Change this to 10 minutes 15 seconds + 1 second (616) if you are using long-running queries with a 10 minute timeout
terminationGracePeriodSeconds: 136
# By default this is a 6 minutes + 30 seconds + 1 second (391) grace period to allow outstanding queries to complete
terminationGracePeriodSeconds: 391

ui:
# Annotations for ui pods
Expand Down Expand Up @@ -538,7 +537,7 @@ dbconnector:
# kubernetes will forcibly kill them when they need to be rescheduled. Setting
# this to a long duration will minimize disruption to any long-running
# resource queries during updates, at the cost of making updates take longer.
# terminationGracePeriodSeconds: 960
terminationGracePeriodSeconds: 391

# If necessary, specify the resources to provision the workflows-backend pod
# separately from the main backend pods. If unspecified, uses the same
Expand Down Expand Up @@ -794,6 +793,9 @@ codeExecutor:

replicaCount: 1

# 6m in-flight request + 30s backend-style preStop buffer + 1s
terminationGracePeriodSeconds: 391

# Annotations for code executor pods
annotations: {}

Expand Down Expand Up @@ -868,6 +870,9 @@ rr:

replicaCount: 1

# 6m in-flight SLF + 30s drain buffer + 1s
terminationGracePeriodSeconds: 391

seccompLocalhostProfile: profiles/nsjail-seccomp.json

# JS-executor-specific environment; not inherited from the top-level
Expand Down
13 changes: 9 additions & 4 deletions values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -432,9 +432,8 @@ backend:
labels: {}

# Make backend pods wait the specified time before kubernetes will forcibly kill them when they need to be rescheduled
# By default this is a 2 minutes + 15 seconds + 1 second (136) grace period to allow outstanding queries to complete
# Change this to 10 minutes 15 seconds + 1 second (616) if you are using long-running queries with a 10 minute timeout
terminationGracePeriodSeconds: 136
# By default this is a 6 minutes + 30 seconds + 1 second (391) grace period to allow outstanding queries to complete
terminationGracePeriodSeconds: 391

ui:
# Annotations for ui pods
Expand Down Expand Up @@ -538,7 +537,7 @@ dbconnector:
# kubernetes will forcibly kill them when they need to be rescheduled. Setting
# this to a long duration will minimize disruption to any long-running
# resource queries during updates, at the cost of making updates take longer.
# terminationGracePeriodSeconds: 960
terminationGracePeriodSeconds: 391

# If necessary, specify the resources to provision the workflows-backend pod
# separately from the main backend pods. If unspecified, uses the same
Expand Down Expand Up @@ -794,6 +793,9 @@ codeExecutor:

replicaCount: 1

# 6m in-flight request + 30s backend-style preStop buffer + 1s
terminationGracePeriodSeconds: 391

# Annotations for code executor pods
annotations: {}

Expand Down Expand Up @@ -868,6 +870,9 @@ rr:

replicaCount: 1

# 6m in-flight SLF + 30s drain buffer + 1s
terminationGracePeriodSeconds: 391

seccompLocalhostProfile: profiles/nsjail-seccomp.json

# JS-executor-specific environment; not inherited from the top-level
Expand Down
Loading