Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions CONTEXT.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,34 @@ _Avoid_: runtime support
The shared Add-on state reported as `notInstalled`, `installing`, `ready`, `updating`, `uninstalling`, or `failed`, with an optional detail message and installed version.
_Avoid_: Add-on-specific status enum

**Non-removable Cluster Add-on**:
A Cluster Add-on that QuickStack can install, inspect, and update but deliberately cannot remove because removal would leave host-level runtime changes behind. Its lifecycle never enters `uninstalling`.
_Avoid_: broken Add-on, partially installed Add-on

**gVisor Runtime Add-on**:
A Trusted, Non-removable Cluster Add-on that makes the gVisor runtime available on supported cluster nodes. It does not select a runtime for Agent Sandboxes.
_Avoid_: gVisor plugin, Agent runtime selector

**gVisor Release Version**:
The explicit, date-based gVisor release selected by QuickStack for installation and display, such as `20260921`. It is the version of the gVisor Runtime Add-on.
_Avoid_: latest, unpinned runtime version, Add-on SemVer

**gVisor RuntimeClass**:
The optional Kubernetes RuntimeClass named `gvisor`, backed by `io.containerd.runsc.v1`. It is available alongside, and never replaces, the cluster's default runtime class.
_Avoid_: default runtime, runsc RuntimeClass

**gVisor-ready Node**:
A schedulable node on which QuickStack has statically verified the pinned gVisor installation, K3s configuration, active K3s service, and node readiness. QuickStack marks it with `quickstack.dev/gvisor=true` and `quickstack.dev/gvisor-version` after success.
_Avoid_: gVisor-tested node, RuntimeClass-proven node

**gVisor Node Version**:
The gVisor Release Version actually installed and statically verified on a gVisor-ready Node. It is represented by `quickstack.dev/gvisor-version` and does not change until that node's update succeeds.
_Avoid_: desired gVisor version, planned node version

**gVisor Add-on Configuration**:
The Add-on-owned Kubernetes ConfigMap that persistently declares the desired gVisor Release Version for the gVisor Runtime Add-on. It is not part of a generic Add-on configuration store.
_Avoid_: global Add-on configuration, runtime version cache

**Agent Sandbox Add-on**:
The Cluster Add-on that installs the Agent Sandbox controller and its extension CRDs, including SandboxTemplate and SandboxWarmPool. It does not create user workload resources.
_Avoid_: Agent, Agent runtime
Expand Down
3 changes: 3 additions & 0 deletions docs/adr/0009-gvisor-runtime-addon-lifecycle.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Keep gVisor as a non-removable runtime Add-on

QuickStack provides gVisor as a trusted Cluster Add-on that can be installed, inspected, and explicitly updated but never removed, because its installation changes node hosts and K3s runtime configuration. Its visible and installed version is the explicit, code-defined gVisor date-based release rather than a wrapper SemVer; its Add-on-owned ConfigMap persists that desired version. Each installation or update uses the official, pinned gVisor Debian repository and completes only when every schedulable node is ready. It provides the optional `gvisor` RuntimeClass beside, rather than in place of, the default runtime class. A privileged DaemonSet coordinates host changes through a cluster-wide Lease, preserving existing K3s templates by managing only a marked gVisor block. A new compatible `amd64` or `arm64` Ubuntu/Debian node changes the Add-on to `updating` while it is brought to the pinned version automatically and serially; an incompatible node or failed rollout changes it to `failed`. An explicit retry only processes unverified or divergent nodes, leaving a failed node cordoned without automatic rollback. Cordon never drains workloads. Readiness is based on static host and Kubernetes checks rather than a test workload, and successful nodes receive QuickStack gVisor labels whose version is the node's actual installed version; later static checks report manual drift without repairing it. The UI requires confirmation before an install or update because host mutations are irreversible and can briefly interrupt a single-server control plane.
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
-- AlterTable
ALTER TABLE "Agent" ADD COLUMN "runtimeClassName" TEXT;
1 change: 1 addition & 0 deletions prisma/schema.prisma
Original file line number Diff line number Diff line change
Expand Up @@ -289,6 +289,7 @@ model Agent {
containerCommand String?
containerArgs String?
workingDir String?
runtimeClassName String?
warmPoolReplicas Int @default(0)
deployFileBrowser Boolean @default(false)

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
import { execFileSync } from 'node:child_process';

const runLiveIntegration = process.env.RUN_GVISOR_LIVE_INTEGRATION === '1';
const describeLive = runLiveIntegration ? describe : describe.skip;

describeLive('gVisor live cluster installation', () => {
const kubectl = process.env.KUBECTL_PATH ?? '/home/node/.local/bin/kubectl';
const kubeconfig = process.env.KUBECONFIG ?? '/workspace/kube-config.config';
const namespace = 'quickstack-gvisor-system';

const kubectlOutput = (...args: string[]): string => execFileSync(kubectl, ['--kubeconfig', kubeconfig, ...args], {
encoding: 'utf8',
timeout: 30_000,
});

it('installs and verifies gVisor on every schedulable node', async () => {
const { default: gvisorAddonService } = await import('@/server/services/addons/gvisor-addon.service');
const initialNodes = JSON.parse(kubectlOutput('get', 'nodes', '-o', 'json')) as {
items: Array<{ metadata: { name: string }; spec?: { unschedulable?: boolean } }>;
};
const targetNodeNames = initialNodes.items
.filter((node) => !node.spec?.unschedulable)
.map((node) => node.metadata.name);
expect(targetNodeNames.length).toBeGreaterThan(0);

const initialStatus = await gvisorAddonService.getStatus();
if (initialStatus.status === 'notInstalled') {
const result = await gvisorAddonService.install();
expect(result.status).toBe('succeeded');
}

const deadline = Date.now() + 10 * 60_000;
while (Date.now() < deadline) {
const nodes = JSON.parse(kubectlOutput('get', 'nodes', '-o', 'json')) as {
items: Array<{ metadata: { name: string; labels?: Record<string, string> }; spec?: { unschedulable?: boolean } }>;
};
const targetsReady = targetNodeNames.every((name) => {
const node = nodes.items.find((candidate) => candidate.metadata.name === name);
return node && !node.spec?.unschedulable && node.metadata.labels?.['quickstack.dev/gvisor'] === 'true' &&
node.metadata.labels?.['quickstack.dev/gvisor-version'] === '20260921';
});
if (targetsReady) {
break;
}
await new Promise((resolve) => setTimeout(resolve, 5_000));
}

await expect(gvisorAddonService.getStatus()).resolves.toMatchObject({ status: 'ready' });
expect(kubectlOutput('-n', namespace, 'get', 'daemonset', 'quickstack-gvisor-installer')).toContain('quickstack-gvisor-installer');
expect(kubectlOutput('get', 'runtimeclass', 'gvisor', '-o', 'jsonpath={.handler}')).toBe('gvisor');

const nodes = JSON.parse(kubectlOutput('get', 'nodes', '-o', 'json')) as {
items: Array<{ metadata: { name: string; labels?: Record<string, string> }; spec?: { unschedulable?: boolean } }>;
};
const targetNodes = nodes.items.filter((node) => targetNodeNames.includes(node.metadata.name));
expect(targetNodes).toHaveLength(targetNodeNames.length);
for (const node of targetNodes) {
expect(node.spec?.unschedulable).not.toBe(true);
expect(node.metadata.labels?.['quickstack.dev/gvisor']).toBe('true');
expect(node.metadata.labels?.['quickstack.dev/gvisor-version']).toBe('20260921');
}
}, 11 * 60_000);
});
5 changes: 3 additions & 2 deletions src/app/project/agent/[agentId]/agent-detail-client.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -62,11 +62,12 @@ const buildAgentTabParams = (tab: string, section: ConfigurationSection) => {
return params;
};

export default function AgentDetailClient({ agent, role, templateInfo, storageClasses }: {
export default function AgentDetailClient({ agent, role, templateInfo, storageClasses, runtimeClasses }: {
agent: AgentExtendedModel;
templateInfo?: AgentSandboxTemplateInfo;
role: RolePermissionEnum | null;
storageClasses: string[];
runtimeClasses: string[];
}) {
const searchParams = useSearchParams();
const rawTabName = searchParams.get('tabName') || 'sandboxes';
Expand Down Expand Up @@ -127,7 +128,7 @@ export default function AgentDetailClient({ agent, role, templateInfo, storageCl
case 'container':
return (
<div className="space-y-4">
<AgentContainerConfigCard agent={agent} readonly={readonly} />
<AgentContainerConfigCard agent={agent} readonly={readonly} runtimeClasses={runtimeClasses} />
<AgentRateLimitsCard agent={agent} readonly={readonly} />
<HealthCheckSettings workload={agent} readonly={readonly} saveHealthCheck={saveAgentHealthCheck} />
</div>
Expand Down
9 changes: 9 additions & 0 deletions src/app/project/agent/[agentId]/general/actions.ts
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ import { ServiceException } from "@/shared/model/service.exception.model";
import agentGitSshKeyService from "@/server/services/agent-git-ssh-key.service";
import gitService from "@/server/services/git.service";
import { ContainerCommangArgsUtils } from "@/shared/utils/container-command-args.utils";
import clusterService from "@/server/services/cluster.service";
import { z } from "zod";
import { HealthCheckModel, healthCheckZodModel } from "@/shared/model/health-check.model";

Expand Down Expand Up @@ -176,6 +177,14 @@ export const saveAgentRateLimits = async (prevState: any, inputData: AgentRateLi
export const saveAgentContainerConfig = async (prevState: any, inputData: AgentContainerConfigModel, agentId: string) =>
saveFormAction(inputData, agentContainerConfigZodModel, async (validatedData) => {
await isAuthorizedWriteForWorkload(agentId);
if (validatedData.runtimeClassName) {
const runtimeClasses = await clusterService.getRuntimeClasses();
if (!runtimeClasses.includes(validatedData.runtimeClassName)) {
throw new FormValidationException('The selected RuntimeClass is no longer available in the cluster.', {
runtimeClassName: ['Select an available RuntimeClass.'],
});
}
}
await agentService.saveAgent({
...validatedData,
containerCommand: ContainerCommangArgsUtils.serializeContainerCommandItems(validatedData.containerCommand),
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import { SubmitButton } from "@/components/custom/submit-button";
import { Card, CardContent, CardDescription, CardFooter, CardHeader, CardTitle } from "@/components/ui/card";
import { Form, FormControl, FormField, FormItem, FormLabel, FormMessage } from "@/components/ui/form";
import { Input } from "@/components/ui/input";
import { Select, SelectContent, SelectItem, SelectTrigger, SelectValue } from "@/components/ui/select";
import { Switch } from "@/components/ui/switch";
import { TooltipProvider } from "@/components/ui/tooltip";
import { FormUtils } from "@/frontend/utils/form.utilts";
Expand All @@ -22,9 +23,12 @@ import { ServerActionResult } from "@/shared/model/server-action-error-return.mo
import { saveAgentContainerConfig } from "./actions";
import { ContainerCommangArgsUtils } from "@/shared/utils/container-command-args.utils";

export default function AgentContainerConfigCard({ agent, readonly }: {
const DEFAULT_RUNTIME_CLASS = '__quickstack_default__';

export default function AgentContainerConfigCard({ agent, readonly, runtimeClasses }: {
agent: AgentExtendedModel;
readonly: boolean;
runtimeClasses: string[];
}) {
const inputValue = (value: unknown) => typeof value === 'string' || typeof value === 'number' ? value : '';
const form = useForm<z.input<typeof agentContainerConfigZodModel>, unknown, z.output<typeof agentContainerConfigZodModel>>({
Expand All @@ -35,6 +39,7 @@ export default function AgentContainerConfigCard({ agent, readonly }: {
? JSON.parse(agent.containerArgs).map((arg: string) => ({ value: arg }))
: [],
workingDir: agent.workingDir ?? '',
runtimeClassName: agent.runtimeClassName ?? '',
warmPoolReplicas: agent.warmPoolReplicas ?? 0,
deployFileBrowser: agent.deployFileBrowser,
},
Expand Down Expand Up @@ -81,6 +86,43 @@ export default function AgentContainerConfigCard({ agent, readonly }: {
/>
</div>

<FormField
control={form.control}
name="runtimeClassName"
render={({ field }) => (
<FormItem>
<FormLabel>Runtime Class</FormLabel>
<Select
value={field.value || DEFAULT_RUNTIME_CLASS}
onValueChange={(value) => field.onChange(value === DEFAULT_RUNTIME_CLASS ? null : value)}
disabled={readonly}
items={[
{ value: DEFAULT_RUNTIME_CLASS, label: 'K3s default runtime' },
...runtimeClasses.map((runtimeClass) => ({ value: runtimeClass, label: runtimeClass })),
]}
>
<FormControl>
<SelectTrigger>
<SelectValue placeholder="K3s default runtime" />
</SelectTrigger>
</FormControl>
<SelectContent>
<SelectItem value={DEFAULT_RUNTIME_CLASS}>K3s default runtime</SelectItem>
{runtimeClasses.map((runtimeClass) => (
<SelectItem key={runtimeClass} value={runtimeClass}>
{runtimeClass}
</SelectItem>
))}
</SelectContent>
</Select>
<p className="text-sm text-muted-foreground">
Selects the Kubernetes RuntimeClass for new agent sandboxes.
</p>
<FormMessage />
</FormItem>
)}
/>

<FormField
control={form.control}
name="workingDir"
Expand Down
4 changes: 3 additions & 1 deletion src/app/project/agent/[agentId]/page.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,10 @@ export default async function AgentDetailPage({
ensureReadProjectWorkload(identity, resolvedParams.agentId);

const agent = await agentService.getById(resolvedParams.agentId);
const [templateDeploymentDetails, storageClasses, agents] = await Promise.all([
const [templateDeploymentDetails, storageClasses, runtimeClasses, agents] = await Promise.all([
CatchUtils.resultOrUndefined(() => agentService.getSandboxTemplateDeployInfo(agent.id)),
clusterService.getStorageClasses(),
clusterService.getRuntimeClasses(),
agentService.getAllByProjectId(agent.projectId),
]);
const relevantAgents = agents.filter((projectAgent) =>
Expand Down Expand Up @@ -56,6 +57,7 @@ export default async function AgentDetailPage({
role={role}
templateInfo={templateDeploymentDetails ?? undefined}
storageClasses={storageClasses}
runtimeClasses={runtimeClasses}
/>
</div>
);
Expand Down
Loading
Loading