From ab5f3ba0a6794572256fefb917953dc023642e9d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sat, 8 Aug 2026 20:12:48 +0800 Subject: [PATCH 01/87] =?UTF-8?q?feat:=20=E5=BB=BA=E7=AB=8B=20LiteLLM=20P1?= =?UTF-8?q?=20=E8=BF=90=E8=A1=8C=E5=9F=BA=E7=BA=BF?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/README.md | 101 +++++++++--- docker_litellm/demo/.env.example | 29 ++++ docker_litellm/demo/config.yaml | 21 +++ .../demo/docker-compose.litellm.yml | 104 +++++++++++-- docker_litellm/demo/scripts/smoke-baseline.sh | 145 ++++++++++++++++++ docker_litellm/litellm.Dockerfile | 35 +++-- docker_litellm/work/start-litellm.sh | 6 +- 7 files changed, 391 insertions(+), 50 deletions(-) create mode 100644 docker_litellm/demo/.env.example create mode 100644 docker_litellm/demo/config.yaml create mode 100755 docker_litellm/demo/scripts/smoke-baseline.sh diff --git a/docker_litellm/README.md b/docker_litellm/README.md index 45389bb..04243e0 100644 --- a/docker_litellm/README.md +++ b/docker_litellm/README.md @@ -1,37 +1,94 @@ -# LiteLLM Proxy +# LiteLLM Proxy:P1 真实运行基线 -`litellm` is a lightweight proxy server to call 100+ LLM APIs using the OpenAI format, with a built-in UI dashboard. +本目录维护 LiteLLM 的部署适配,不 fork 或修改 LiteLLM 上游业务逻辑。P1 提供可复现的本地基线:共享 PostgreSQL、共享 Redis、单副本与双副本 LiteLLM,以及不会写入仓库密钥的 smoke test。 ---- +P1 默认不构建 LiteLLM Dashboard 静态资源:固定源码在当前构建基础镜像上导出 `/_not-found` 时失败,而代理 API 与管理面验证不依赖该资源。需要 Dashboard 时可显式传入 `--build-arg BUILD_DASHBOARD=true` 单独处理该上游前端兼容性;它不属于 P1 通过条件。 -## 1. Port Configuration +## 固定版本与镜像 -- **`4000` (HTTP)**: Serves the OpenAI-compatible REST API endpoints and the admin control panel dashboard interface. +| 项目 | 固定值 | 用途 | +| --- | --- | --- | +| LiteLLM 源码 | `v1.97.0-dev.1` / `ead62528e607b9d8e61273def638799c9c3a69ba` | Dockerfile 精确 fetch 并校验 HEAD | +| FastAPI | `0.136.3` | 固定到该 LiteLLM commit 仍使用 `get_flat_dependant` 的兼容版本 | +| Prisma Python client | `0.15.0` | LiteLLM 连接 PostgreSQL 所需客户端,兼容基础镜像的 Python 3.13 | ---- +镜像构建会对 wheel 自带的 LiteLLM Prisma schema 运行 `prisma generate`;没有该步骤,代理会在 PostgreSQL startup 时报缺少 Prisma binaries。 +| 本地产物镜像 | `quay.io/labnow/litellm:1.97.0-ead62528e607` | P1 Compose 的唯一 LiteLLM 默认镜像 | +| PostgreSQL | `postgres:17-alpine@sha256:742f40ea20b9ff2ff31db5458d127452988a2164df9e17441e191f3b72252193` | 用户、凭证、模型、虚拟 key 与 spend 持久化 | +| Redis | `redis:7.4-alpine@sha256:e7723ff73d963f5cc6d9c4643ea3d989527a402a319239054e9472a7fb9219a2` | 副本共享认证缓存、限流、Spend counter 和协调缓存 | +| OpenClaw(P2 参考) | `2026.5.10-beta.1` / `eed75ed47f47deb18c9d093a2e638c9bb0bedf14` | 仅为下一阶段黄金适配器保留版本基线;P1 不启动或实现 Adapter | -## 2. Data Persistence & Configurations +构建前已确认完整 LabNow 镜像名是 `quay.io/labnow/litellm:1.97.0-ead62528e607`,不会推送镜像。必须通过根目录 `tool.sh` 构建,避免基础镜像退回 Docker Hub: -LiteLLM looks for `config.yaml` in its home directory at startup: +```bash +export REGISTRY_SRC=quay.io +export REGISTRY_DST=quay.io +export CI_PROJECT_NAME=LabNow/lab-dev +source ./tool.sh +build_image_no_tag litellm 1.97.0-ead62528e607 docker_litellm/litellm.Dockerfile +``` + +构建完成后记录本地 digest: + +```bash +docker image inspect quay.io/labnow/litellm:1.97.0-ead62528e607 \ + --format 'image_id={{.Id}} created={{.Created}}' +``` + +## 本地启动 + +准备不会被 Git 跟踪的配置。不要把 `.env` 发送到聊天、日志或提交中。 + +```bash +cd docker_litellm/demo +cp .env.example .env +# 在 .env 中生成并填写 LITELLM_MASTER_KEY、POSTGRES_PASSWORD、REDIS_PASSWORD。 +# 真实上游调用另行填写 UPSTREAM_API_KEY、UPSTREAM_BASE_URL、UPSTREAM_MODEL。 +docker compose --env-file .env -f docker-compose.litellm.yml --profile single up -d +``` -- **`/opt/litellm`**: Sourced workspace directory (configured via `HOME_LITELLM`). This is where `config.yaml` is written and read. -- **`/root/workspace`**: Additional shared data directories volume. +双副本测试使用同一 PostgreSQL 与 Redis,但有两个 HTTP 入口: -### Custom Home Directory -You can override the home location using the environment variable: -- `HOME_LITELLM`: Paths to store the active configs (e.g. `/root/workspace`). +```bash +docker compose --env-file .env -f docker-compose.litellm.yml --profile ha up -d +``` + +默认端口只发布在 `127.0.0.1`:副本 1 为 `4000`,副本 2 为 `4001`。PostgreSQL 与 Redis 不发布宿主机端口。停止测试不会删除卷;如需删除测试数据,先人工确认后使用 `docker compose ... down -v`。 + +## 配置与安全边界 + +`config.yaml` 从环境变量读取管理面 `LITELLM_MASTER_KEY`、`DATABASE_URL` 与 Redis 凭据。管理面 key 仅用于 `/user/new`、`/credentials`、`/model/new`、`/key/generate`、`/key/block` 和 `/key/delete` 等管理接口;smoke 生成的数据面虚拟 key 是短期、模型白名单、TTL、预算、RPM、TPM 与 `llm_api` 路由限制的独立 key。 + +| 变量 | 是否必填 | 作用 | 风险说明 | +| --- | ---: | --- | --- | +| `LITELLM_MASTER_KEY` | 是 | 管理面认证 | 仅放在忽略的 `.env` 或部署 Secret | +| `POSTGRES_PASSWORD` | 是 | PostgreSQL 密码 | 仅限本地测试或部署 Secret | +| `REDIS_PASSWORD` | 是 | Redis 认证 | 仅限本地测试或部署 Secret | +| `UPSTREAM_API_KEY` | 真实调用时是 | 上游模型凭据 | 不提交、不打印;缺失时真实调用 smoke 保持待验证 | +| `UPSTREAM_BASE_URL` | 真实调用时是 | OpenAI 兼容上游地址 | 由测试环境决定 | +| `UPSTREAM_MODEL` | 真实调用时是 | 上游模型名 | 用于创建测试模型 | ---- +P1 不把 LiteLLM User/Team 当作产品用户或 Workspace 的事实源;Shell 的用户、绑定和租约业务仍在后续 Phase 实现。 -## 3. Quickstart Example +## Smoke 验证 -Run LiteLLM Proxy with mapped configuration folder: ```bash -docker run -d \ - --name svc-litellm \ - -p 4000:4000 \ - -v /path/to/your/config:/opt/litellm \ - labnow/litellm:latest +cd docker_litellm/demo +./scripts/smoke-baseline.sh --mode single +./scripts/smoke-baseline.sh --mode ha ``` -By default, it will look for a `config.yaml` in the directory. If not found, a basic template targeting `gpt-3.5-turbo` is auto-generated. +脚本在真实上游变量存在时执行:创建测试用户、保存测试上游凭证、创建模型、生成受限虚拟 key、`GET /v1/models`、chat、stream、tool call、usage 查询、block/delete,以及旧 key 在另一副本被拒绝。脚本不输出任何 key;临时响应文件会在退出时删除。 + +若未设置上游变量,脚本仍验证 LiteLLM readiness、PostgreSQL 连接、从每个 LiteLLM 副本到 Redis 的认证连通性和管理面认证,并以明确的 `PENDING upstream smoke` 退出成功。它不会伪造 chat、stream、tool 或 usage 已通过。 + +## Readiness 与 Redis 结论 + +LiteLLM `v1.97.0-dev.1` 的公开 `/health/readiness` 仅返回服务与数据库连通性,不将 Redis 纳入公开 readiness。因此 Compose 健康检查只能确认 LiteLLM + PostgreSQL;`smoke-baseline.sh` 额外从每个 LiteLLM 容器执行 Redis `PING`。若 Redis 不可用,P1 的多副本认证缓存、限流、Spend counter 与协调结论无效,应将该运行组合标记为 `blocked`,不能降级宣称为高可用。 + +## 常见问题 + +- `LITELLM_MASTER_KEY` 或数据库密码缺失:先检查被忽略的 `demo/.env`,不要将其内容贴出。 +- readiness 未连接数据库:查看 `docker compose ... logs postgres litellm-1`,并保留卷以便排查迁移。 +- Redis 探针失败:不要继续双副本撤销验证;先确认 `redis` health 与密码一致。 +- 上游调用待验证:仅在 `.env` 中提供专用、低权限、可轮换的测试 key,再重跑两个 smoke 命令。 diff --git a/docker_litellm/demo/.env.example b/docker_litellm/demo/.env.example new file mode 100644 index 0000000..91ec14a --- /dev/null +++ b/docker_litellm/demo/.env.example @@ -0,0 +1,29 @@ +# Copy this file to docker_litellm/demo/.env. It is intentionally ignored. +# Do not commit real API keys, management keys, passwords, or virtual keys. + +TZ=Asia/Hong_Kong + +# Build this exact source baseline through ../../tool.sh before running Compose. +LITELLM_IMAGE=quay.io/labnow/litellm:1.97.0-ead62528e607 +LITELLM_1_CONTAINER_NAME=svc-litellm-1 +LITELLM_2_CONTAINER_NAME=svc-litellm-2 +LITELLM_PUBLISH_HOST=127.0.0.1 +LITELLM_1_PORT=4000 +LITELLM_2_PORT=4001 + +# Local-only secrets. Generate unique values; these examples are placeholders. +LITELLM_MASTER_KEY= +POSTGRES_DB=litellm +POSTGRES_USER=litellm +POSTGRES_PASSWORD= +REDIS_PASSWORD= + +# Optional upstream required for chat/stream/tool smoke. Keep empty to validate +# infrastructure and management/revocation paths only. +UPSTREAM_API_KEY= +UPSTREAM_BASE_URL=https://api.openai.com/v1 +UPSTREAM_MODEL=gpt-4o-mini + +# P2 reference only; this phase does not start an OpenClaw adapter. +OPENCLAW_IMAGE=quay.io/labnow/openclaw:2026.5.10-beta.1 +OPENCLAW_SOURCE_COMMIT=eed75ed47f47deb18c9d093a2e638c9bb0bedf14 diff --git a/docker_litellm/demo/config.yaml b/docker_litellm/demo/config.yaml new file mode 100644 index 0000000..7f26a0b --- /dev/null +++ b/docker_litellm/demo/config.yaml @@ -0,0 +1,21 @@ +# P1 runtime configuration. Secrets only come from environment variables. +# The management API persists users, credentials, model records and virtual keys +# in PostgreSQL. Redis is the shared cache/co-ordination backend for replicas. + +model_list: [] + +general_settings: + master_key: os.environ/LITELLM_MASTER_KEY + database_url: os.environ/DATABASE_URL + store_model_in_db: true + # Preserve request metadata needed for metering, but do not configure prompt + # or completion-content logging in this local baseline. + disable_spend_logs: false + +litellm_settings: + cache: true + cache_params: + type: redis + host: os.environ/REDIS_HOST + port: os.environ/REDIS_PORT + password: os.environ/REDIS_PASSWORD diff --git a/docker_litellm/demo/docker-compose.litellm.yml b/docker_litellm/demo/docker-compose.litellm.yml index 28d9383..b63f28e 100644 --- a/docker_litellm/demo/docker-compose.litellm.yml +++ b/docker_litellm/demo/docker-compose.litellm.yml @@ -1,19 +1,89 @@ +name: litellm-baseline + +x-litellm-common: &litellm-common + image: ${LITELLM_IMAGE:?set LITELLM_IMAGE to the locally built quay.io/labnow/litellm image} + restart: "no" + environment: + TZ: ${TZ:-Asia/Hong_Kong} + LITELLM_MASTER_KEY: ${LITELLM_MASTER_KEY:?set a local-only management key in .env} + DATABASE_URL: postgresql://${POSTGRES_USER:?set in .env}:${POSTGRES_PASSWORD:?set in .env}@postgres:5432/${POSTGRES_DB:-litellm} + REDIS_HOST: redis + REDIS_PORT: "6379" + REDIS_PASSWORD: ${REDIS_PASSWORD:?set in .env} + STORE_MODEL_IN_DB: "True" + LITELLM_LOG: ${LITELLM_LOG:-INFO} + # The upstream credentials are optional and must remain in the ignored .env. + UPSTREAM_API_KEY: ${UPSTREAM_API_KEY:-} + UPSTREAM_BASE_URL: ${UPSTREAM_BASE_URL:-} + UPSTREAM_MODEL: ${UPSTREAM_MODEL:-} + volumes: + - ./config.yaml:/opt/litellm/config.yaml:ro + depends_on: + postgres: + condition: service_healthy + redis: + condition: service_healthy + networks: + - litellm-baseline-net + healthcheck: + test: ["CMD-SHELL", "python3 -c \"import urllib.request; urllib.request.urlopen('http://127.0.0.1:4000/health/readiness', timeout=3)\""] + interval: 10s + timeout: 5s + retries: 18 + start_period: 30s + services: - svc-litellm: - container_name: svc-litellm - image: quay.io/labnow/litellm:latest - restart: unless-stopped - # networks: ["net-litellm"] - ports: - - "4000:4000" + postgres: + image: postgres:17-alpine@sha256:742f40ea20b9ff2ff31db5458d127452988a2164df9e17441e191f3b72252193 + restart: "no" + environment: + POSTGRES_DB: ${POSTGRES_DB:-litellm} + POSTGRES_USER: ${POSTGRES_USER:?set in .env} + POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set in .env} + volumes: + - litellm_postgres_data:/var/lib/postgresql/data + networks: + - litellm-baseline-net + healthcheck: + test: ["CMD-SHELL", "pg_isready -U $$POSTGRES_USER -d $$POSTGRES_DB"] + interval: 5s + timeout: 5s + retries: 20 + + redis: + image: redis:7.4-alpine@sha256:e7723ff73d963f5cc6d9c4643ea3d989527a402a319239054e9472a7fb9219a2 + restart: "no" + environment: + REDIS_PASSWORD: ${REDIS_PASSWORD:?set in .env} + command: ["redis-server", "--appendonly", "yes", "--requirepass", "${REDIS_PASSWORD:?set in .env}"] volumes: - - ./data:/root/workspace - # environment: - # - LITELLM_MASTER_KEY=sk-1234 - # - OPENAI_API_KEY=your-openai-key - # - GEMINI_API_KEY=your-gemini-key - logging: - driver: "json-file" - options: - max-size: "10m" - max-file: "3" + - litellm_redis_data:/data + networks: + - litellm-baseline-net + healthcheck: + test: ["CMD-SHELL", "redis-cli --no-auth-warning -a \"$$REDIS_PASSWORD\" ping | grep -qx PONG"] + interval: 5s + timeout: 5s + retries: 20 + + litellm-1: + <<: *litellm-common + container_name: ${LITELLM_1_CONTAINER_NAME:-svc-litellm-1} + profiles: ["single", "ha"] + ports: + - "${LITELLM_PUBLISH_HOST:-127.0.0.1}:${LITELLM_1_PORT:-4000}:4000" + + litellm-2: + <<: *litellm-common + container_name: ${LITELLM_2_CONTAINER_NAME:-svc-litellm-2} + profiles: ["ha"] + ports: + - "${LITELLM_PUBLISH_HOST:-127.0.0.1}:${LITELLM_2_PORT:-4001}:4000" + +volumes: + litellm_postgres_data: + litellm_redis_data: + +networks: + litellm-baseline-net: + name: litellm-baseline-net diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh new file mode 100755 index 0000000..efb1a0a --- /dev/null +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -0,0 +1,145 @@ +#!/usr/bin/env bash +# Runs against a local, ignored docker_litellm/demo/.env. It never prints keys. +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +DEMO_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)" +ENV_FILE="${LITELLM_SMOKE_ENV_FILE:-${DEMO_DIR}/.env}" +MODE="single" + +usage() { + echo "Usage: $0 [--mode single|ha]" >&2 +} + +while (($#)); do + case "$1" in + --mode) MODE="${2:-}"; shift 2 ;; + --help|-h) usage; exit 0 ;; + *) usage; exit 2 ;; + esac +done + +[[ "$MODE" == "single" || "$MODE" == "ha" ]] || { usage; exit 2; } +[[ -f "$ENV_FILE" ]] || { echo "missing local environment file: $ENV_FILE (copy .env.example)" >&2; exit 2; } + +# shellcheck disable=SC1090 +set -a; source "$ENV_FILE"; set +a +: "${LITELLM_MASTER_KEY:?missing LITELLM_MASTER_KEY in .env}" + +if [[ "$MODE" == "single" ]]; then + BASE_URL="http://${LITELLM_PUBLISH_HOST:-127.0.0.1}:${LITELLM_1_PORT:-4000}" + PEER_URL="$BASE_URL" +else + BASE_URL="http://${LITELLM_PUBLISH_HOST:-127.0.0.1}:${LITELLM_1_PORT:-4000}" + PEER_URL="http://${LITELLM_PUBLISH_HOST:-127.0.0.1}:${LITELLM_2_PORT:-4001}" +fi + +need() { command -v "$1" >/dev/null || { echo "required command missing: $1" >&2; exit 2; }; } +need curl; need jq + +tmpdir="$(mktemp -d)" +trap 'rm -rf "$tmpdir"' EXIT + +request_admin() { + local method="$1" path="$2" data="${3:-}" out="$4" data_file + if [[ -n "$data" ]]; then + data_file="$tmpdir/admin-request.json" + printf '%s' "$data" > "$data_file" + curl --silent --show-error --fail --max-time 30 -X "$method" "$BASE_URL$path" \ + -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json' \ + --data-binary "@$data_file" -o "$out" + else + curl --silent --show-error --fail --max-time 30 -X "$method" "$BASE_URL$path" \ + -H "Authorization: Bearer $LITELLM_MASTER_KEY" -o "$out" + fi +} + +wait_ready() { + local url="$1" i + for i in $(seq 1 60); do + if curl --silent --show-error --fail --max-time 3 "$url/health/readiness" -o "$tmpdir/readiness.json"; then + jq -e '.status == "healthy" and .db == "connected"' "$tmpdir/readiness.json" >/dev/null && return 0 + fi + sleep 2 + done + echo "LiteLLM readiness did not report PostgreSQL connected: $url" >&2 + return 1 +} + +assert_redis() { + local service="$1" + docker compose --env-file "$ENV_FILE" -f "$DEMO_DIR/docker-compose.litellm.yml" exec -T "$service" \ + python3 -c 'import os, redis; assert redis.Redis(host="redis", port=6379, password=os.environ["REDIS_PASSWORD"]).ping()' +} + +wait_ready "$BASE_URL" +if [[ "$MODE" == "ha" ]]; then wait_ready "$PEER_URL"; fi +assert_redis litellm-1 +if [[ "$MODE" == "ha" ]]; then assert_redis litellm-2; fi +echo "PASS readiness: PostgreSQL connected; Redis independently reachable from LiteLLM replica(s)." + +suffix="$(date +%s)" +test_user="p1-smoke-user-$suffix" +model_name="p1-smoke-model-$suffix" +key_alias="p1-smoke-key-$suffix" + +# A user is persisted even without an upstream credential. No returned user key +# is retained or printed. +request_admin POST /user/new "$(jq -nc --arg user_id "$test_user" '{user_id:$user_id,auto_create_key:false,user_role:"internal_user"}')" "$tmpdir/user.json" +jq -e --arg id "$test_user" '.user_id == $id' "$tmpdir/user.json" >/dev/null + +if [[ -z "${UPSTREAM_API_KEY:-}" || -z "${UPSTREAM_BASE_URL:-}" || -z "${UPSTREAM_MODEL:-}" ]]; then + echo "PENDING upstream smoke: set UPSTREAM_API_KEY, UPSTREAM_BASE_URL and UPSTREAM_MODEL in ignored .env." + echo "PASS infrastructure: PostgreSQL persistence, shared Redis reachability and management authentication are ready." + exit 0 +fi + +# The credential request is deliberately sent from the ignored environment and +# never saved to a repository file or stdout. LiteLLM returns masked values. +credential_name="p1-smoke-upstream-$suffix" +credential_payload="$(jq -nc --arg name "$credential_name" --arg key "$UPSTREAM_API_KEY" --arg base "$UPSTREAM_BASE_URL" '{credential_name:$name,credential_values:{api_key:$key,api_base:$base},credential_info:{custom_llm_provider:"openai"}}')" +request_admin POST /credentials "$credential_payload" "$tmpdir/credential.json" + +model_payload="$(jq -nc --arg model_name "$model_name" --arg upstream "$UPSTREAM_MODEL" --arg base "$UPSTREAM_BASE_URL" --arg key "$UPSTREAM_API_KEY" '{model_name:$model_name,litellm_params:{model:("openai/" + $upstream),api_base:$base,api_key:$key},model_info:{id:null,mode:"chat"}}')" +request_admin POST /model/new "$model_payload" "$tmpdir/model.json" + +key_payload="$(jq -nc --arg alias "$key_alias" --arg user "$test_user" --arg model "$model_name" '{key_alias:$alias,user_id:$user,models:[$model],duration:"15m",max_budget:0.05,rpm_limit:10,tpm_limit:1000,key_type:"llm_api",allowed_routes:["/v1/models","/v1/chat/completions"]}')" +request_admin POST /key/generate "$key_payload" "$tmpdir/key.json" +virtual_key="$(jq -er '.key' "$tmpdir/key.json")" + +data_request() { + local url="$1" path="$2" data="$3" out="$4" data_file + data_file="$tmpdir/data-request.json" + printf '%s' "$data" > "$data_file" + curl --silent --show-error --fail --max-time 60 "$url$path" \ + -H "Authorization: Bearer $virtual_key" -H 'Content-Type: application/json' --data-binary "@$data_file" -o "$out" +} + +data_request "$BASE_URL" /v1/models '{}' "$tmpdir/models.json" +jq -e --arg model "$model_name" '.data[] | select(.id == $model)' "$tmpdir/models.json" >/dev/null +chat_payload="$(jq -nc --arg model "$model_name" '{model:$model,messages:[{role:"user",content:"Reply with OK."}],max_tokens:16}')" +data_request "$BASE_URL" /v1/chat/completions "$chat_payload" "$tmpdir/chat.json" +jq -e '.choices[0].message.content | type == "string"' "$tmpdir/chat.json" >/dev/null + +stream_payload="$(jq -nc --argjson base "$chat_payload" '$base + {stream:true}')" +printf '%s' "$stream_payload" > "$tmpdir/stream-request.json" +curl --silent --show-error --fail --max-time 60 -N "$BASE_URL/v1/chat/completions" \ + -H "Authorization: Bearer $virtual_key" -H 'Content-Type: application/json' --data-binary "@$tmpdir/stream-request.json" > "$tmpdir/stream.txt" +rg -q '^data: ' "$tmpdir/stream.txt" + +tool_payload="$(jq -nc --arg model "$model_name" '{model:$model,messages:[{role:"user",content:"Use the supplied function to answer 2+2."}],tools:[{type:"function",function:{name:"answer",description:"Return the answer.",parameters:{type:"object",properties:{answer:{type:"integer"}},required:["answer"]}}}],tool_choice:{type:"function",function:{name:"answer"}},max_tokens:32}')" +data_request "$BASE_URL" /v1/chat/completions "$tool_payload" "$tmpdir/tool.json" +jq -e '.choices[0].message.tool_calls | type == "array"' "$tmpdir/tool.json" >/dev/null + +request_admin GET /spend/logs '' "$tmpdir/spend.json" +jq -e 'type == "array" or has("data")' "$tmpdir/spend.json" >/dev/null + +request_admin POST /key/block "$(jq -nc --arg key "$virtual_key" '{key:$key}')" "$tmpdir/block.json" +if curl --silent --show-error --max-time 20 --output /dev/null --write-out '%{http_code}' "$PEER_URL/v1/models" -H "Authorization: Bearer $virtual_key" | grep -Eq '^(401|403)$'; then + : +else + echo "revoked virtual key was accepted by $PEER_URL" >&2 + exit 1 +fi +request_admin POST /key/delete "$(jq -nc --arg key "$virtual_key" '{keys:[$key]}')" "$tmpdir/delete.json" +echo "PASS complete: user/credential/model/key, models/chat/stream/tool, spend, block/delete and revoke propagation." diff --git a/docker_litellm/litellm.Dockerfile b/docker_litellm/litellm.Dockerfile index 2ac0224..cf00677 100644 --- a/docker_litellm/litellm.Dockerfile +++ b/docker_litellm/litellm.Dockerfile @@ -3,24 +3,36 @@ ARG BASE_NAMESPACE ARG BASE_IMG_BUILD="node" ARG BASE_IMG="base" +ARG LITELLM_REF="ead62528e607b9d8e61273def638799c9c3a69ba" +ARG BUILD_DASHBOARD="false" # --- Building Stage --- FROM ${BASE_NAMESPACE:+$BASE_NAMESPACE/}${BASE_IMG_BUILD} AS builder +ARG LITELLM_REF +ARG BUILD_DASHBOARD + LABEL maintainer="postmaster@labnow.ai" # Build-time environment ENV NODE_ENV=development WORKDIR /build -# Clone source, Build UI (Dashboard) & Build Python wheel in one RUN layer +# Clone the fixed source and build its Python wheel. Dashboard export is +# optional: P1 validates the API proxy, not the browser dashboard. RUN set -eux \ - && git clone --depth 1 --branch main https://github.com/BerriAI/litellm.git . \ - && cd ui/litellm-dashboard \ - && npm install \ - && npm run build \ - && mkdir -pv ../../litellm/proxy/_experimental/out \ - && cp -r out/* ../../litellm/proxy/_experimental/out/ \ + && git init . \ + && git remote add origin https://github.com/BerriAI/litellm.git \ + && git fetch --depth 1 origin "${LITELLM_REF}" \ + && git checkout --detach FETCH_HEAD \ + && test "$(git rev-parse HEAD)" = "${LITELLM_REF}" \ + && if [ "${BUILD_DASHBOARD}" = "true" ]; then \ + cd ui/litellm-dashboard \ + && npm install \ + && npm run build \ + && mkdir -pv ../../litellm/proxy/_experimental/out \ + && cp -r out/* ../../litellm/proxy/_experimental/out; \ + fi \ && cd /build \ && python3 -m pip install --upgrade pip build \ && python3 -m build --wheel --outdir dist @@ -44,8 +56,13 @@ COPY --from=builder /build/dist/*.whl /tmp/ RUN set -eux \ && chmod +x /opt/utils/*.sh \ && ln -sf /opt/utils/start-litellm.sh /usr/local/bin/start-litellm.sh \ - && pip install --no-cache-dir /tmp/*.whl \ - && pip install --no-cache-dir 'litellm[proxy]' \ + && WHEEL="$(find /tmp -maxdepth 1 -name '*.whl' -print -quit)" \ + && test -n "${WHEEL}" \ + && pip install --no-cache-dir "${WHEEL}[proxy]" "fastapi==0.136.3" "prisma==0.15.0" \ + && python3 -c 'from fastapi.dependencies.utils import get_flat_dependant; import prisma' \ + && PRISMA_SCHEMA="$(python3 -c 'import pathlib, litellm; print(pathlib.Path(litellm.__file__).parent / "proxy" / "schema.prisma")')" \ + && test -f "${PRISMA_SCHEMA}" \ + && prisma generate --schema "${PRISMA_SCHEMA}" \ ## Install supervisord (Go version) if needed or use simple entrypoint && source /opt/utils/script-setup-sys.sh && setup_supervisord \ && source /opt/utils/script-utils.sh && install__clean \ diff --git a/docker_litellm/work/start-litellm.sh b/docker_litellm/work/start-litellm.sh index acf719c..62394ee 100644 --- a/docker_litellm/work/start-litellm.sh +++ b/docker_litellm/work/start-litellm.sh @@ -8,7 +8,9 @@ mkdir -p "$HOME_LITELLM" export HOME="$HOME_LITELLM" cd "$HOME_LITELLM" -# Default config if not exists +# Default config if not exists. The P1 Compose baseline always mounts an +# explicit config with PostgreSQL and Redis; this fallback remains only for +# backwards-compatible standalone use. if [ ! -f "config.yaml" ]; then echo "Creating default config.yaml..." cat < config.yaml @@ -21,7 +23,7 @@ fi # If no arguments are passed, start litellm proxy with defaults if [ $# -eq 0 ]; then - set -- --config config.yaml --port 4000 --host 0.0.0.0 + set -- --config config.yaml --port "${LITELLM_PORT:-4000}" --host "${LITELLM_HOST:-0.0.0.0}" fi # Route execution: run command directly if it exists, otherwise wrap with litellm From fe08b87c812b1c4a6a934911bc7eca305c0a7a50 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sat, 8 Aug 2026 20:40:33 +0800 Subject: [PATCH 02/87] =?UTF-8?q?fix:=20=E4=BF=AE=E5=A4=8D=20LiteLLM=20Pri?= =?UTF-8?q?sma=20=E8=BF=90=E8=A1=8C=E6=97=B6=E7=BC=93=E5=AD=98?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/README.md | 2 +- docker_litellm/demo/scripts/smoke-baseline.sh | 1 + docker_litellm/litellm.Dockerfile | 5 ++++- docker_litellm/work/start-litellm.sh | 1 + 4 files changed, 7 insertions(+), 2 deletions(-) diff --git a/docker_litellm/README.md b/docker_litellm/README.md index 04243e0..1cb14a3 100644 --- a/docker_litellm/README.md +++ b/docker_litellm/README.md @@ -12,7 +12,7 @@ P1 默认不构建 LiteLLM Dashboard 静态资源:固定源码在当前构建 | FastAPI | `0.136.3` | 固定到该 LiteLLM commit 仍使用 `get_flat_dependant` 的兼容版本 | | Prisma Python client | `0.15.0` | LiteLLM 连接 PostgreSQL 所需客户端,兼容基础镜像的 Python 3.13 | -镜像构建会对 wheel 自带的 LiteLLM Prisma schema 运行 `prisma generate`;没有该步骤,代理会在 PostgreSQL startup 时报缺少 Prisma binaries。 +镜像构建会对 wheel 自带的 LiteLLM Prisma schema 运行 `prisma generate`,并把生成的查询引擎固定在 `/opt/litellm/.cache`;没有该步骤,或将该缓存随 `/root/.cache` 清理,代理会在 PostgreSQL startup 时报缺少 Prisma binaries 或无法连接查询引擎。 | 本地产物镜像 | `quay.io/labnow/litellm:1.97.0-ead62528e607` | P1 Compose 的唯一 LiteLLM 默认镜像 | | PostgreSQL | `postgres:17-alpine@sha256:742f40ea20b9ff2ff31db5458d127452988a2164df9e17441e191f3b72252193` | 用户、凭证、模型、虚拟 key 与 spend 持久化 | | Redis | `redis:7.4-alpine@sha256:e7723ff73d963f5cc6d9c4643ea3d989527a402a319239054e9472a7fb9219a2` | 副本共享认证缓存、限流、Spend counter 和协调缓存 | diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh index efb1a0a..d90a6ac 100755 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -5,6 +5,7 @@ set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" DEMO_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)" ENV_FILE="${LITELLM_SMOKE_ENV_FILE:-${DEMO_DIR}/.env}" +export COMPOSE_PROJECT_NAME="${LITELLM_COMPOSE_PROJECT:-litellm-baseline}" MODE="single" usage() { diff --git a/docker_litellm/litellm.Dockerfile b/docker_litellm/litellm.Dockerfile index cf00677..2a843fa 100644 --- a/docker_litellm/litellm.Dockerfile +++ b/docker_litellm/litellm.Dockerfile @@ -62,7 +62,10 @@ RUN set -eux \ && python3 -c 'from fastapi.dependencies.utils import get_flat_dependant; import prisma' \ && PRISMA_SCHEMA="$(python3 -c 'import pathlib, litellm; print(pathlib.Path(litellm.__file__).parent / "proxy" / "schema.prisma")')" \ && test -f "${PRISMA_SCHEMA}" \ - && prisma generate --schema "${PRISMA_SCHEMA}" \ + # Keep the generated query engine outside /root: install__clean removes + # root-owned caches, while the runtime starts with HOME=/opt/litellm. + && PRISMA_HOME_DIR="${HOME_LITELLM}" prisma generate --schema "${PRISMA_SCHEMA}" \ + && test -d "${HOME_LITELLM}/.cache/prisma-python" \ ## Install supervisord (Go version) if needed or use simple entrypoint && source /opt/utils/script-setup-sys.sh && setup_supervisord \ && source /opt/utils/script-utils.sh && install__clean \ diff --git a/docker_litellm/work/start-litellm.sh b/docker_litellm/work/start-litellm.sh index 62394ee..93a9786 100644 --- a/docker_litellm/work/start-litellm.sh +++ b/docker_litellm/work/start-litellm.sh @@ -6,6 +6,7 @@ HOME_LITELLM="${HOME_LITELLM:-/opt/litellm}" mkdir -p "$HOME_LITELLM" export HOME="$HOME_LITELLM" +export PRISMA_HOME_DIR="${PRISMA_HOME_DIR:-$HOME_LITELLM}" cd "$HOME_LITELLM" # Default config if not exists. The P1 Compose baseline always mounts an From 85b81a3df9af062aa93a04a53e68f8842feba318 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 02:47:00 +0800 Subject: [PATCH 03/87] =?UTF-8?q?fix:=20=E5=BC=BA=E5=8C=96=20LiteLLM=20smo?= =?UTF-8?q?ke=20=E5=AF=86=E9=92=A5=E4=B8=8E=E6=92=A4=E9=94=80=E9=AA=8C?= =?UTF-8?q?=E6=94=B6?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/README.md | 12 +- .../demo/docker-compose.litellm.yml | 4 - docker_litellm/demo/scripts/smoke-baseline.sh | 404 +++++++++++++++--- 3 files changed, 346 insertions(+), 74 deletions(-) diff --git a/docker_litellm/README.md b/docker_litellm/README.md index 1cb14a3..71ea840 100644 --- a/docker_litellm/README.md +++ b/docker_litellm/README.md @@ -64,9 +64,10 @@ docker compose --env-file .env -f docker-compose.litellm.yml --profile ha up -d | `LITELLM_MASTER_KEY` | 是 | 管理面认证 | 仅放在忽略的 `.env` 或部署 Secret | | `POSTGRES_PASSWORD` | 是 | PostgreSQL 密码 | 仅限本地测试或部署 Secret | | `REDIS_PASSWORD` | 是 | Redis 认证 | 仅限本地测试或部署 Secret | -| `UPSTREAM_API_KEY` | 真实调用时是 | 上游模型凭据 | 不提交、不打印;缺失时真实调用 smoke 保持待验证 | +| `UPSTREAM_API_KEY` | 真实调用时是 | 上游模型凭据 | 仅由 smoke 客户端读取;不会注入 LiteLLM 容器、不提交、不打印 | | `UPSTREAM_BASE_URL` | 真实调用时是 | OpenAI 兼容上游地址 | 由测试环境决定 | | `UPSTREAM_MODEL` | 真实调用时是 | 上游模型名 | 用于创建测试模型 | +| `LITELLM_REVOCATION_SLO_MS` | `30000` | block/delete 的跨副本拒绝 SLO | smoke 会输出实际传播耗时;仅用于本地验收 | P1 不把 LiteLLM User/Team 当作产品用户或 Workspace 的事实源;Shell 的用户、绑定和租约业务仍在后续 Phase 实现。 @@ -76,11 +77,16 @@ P1 不把 LiteLLM User/Team 当作产品用户或 Workspace 的事实源;Shell cd docker_litellm/demo ./scripts/smoke-baseline.sh --mode single ./scripts/smoke-baseline.sh --mode ha +./scripts/smoke-baseline.sh --security-check ``` -脚本在真实上游变量存在时执行:创建测试用户、保存测试上游凭证、创建模型、生成受限虚拟 key、`GET /v1/models`、chat、stream、tool call、usage 查询、block/delete,以及旧 key 在另一副本被拒绝。脚本不输出任何 key;临时响应文件会在退出时删除。 +`--security-check` 不读取 `.env`、不启动服务也不发送上游请求;它拒绝 inline header、secret-bearing `jq --arg`、`set -x` 和 Compose 的上游凭据注入,并检查 0600 临时文件与退出清理约束。 -若未设置上游变量,脚本仍验证 LiteLLM readiness、PostgreSQL 连接、从每个 LiteLLM 副本到 Redis 的认证连通性和管理面认证,并以明确的 `PENDING upstream smoke` 退出成功。它不会伪造 chat、stream、tool 或 usage 已通过。 +脚本在真实上游变量存在时执行:创建测试用户、保存测试上游凭证、以 `litellm_credential_name` 创建模型、生成受限虚拟 key、显式 `GET /v1/models`、chat、stream、tool call、token-bearing usage 查询、block,以及独立 key 的 delete。HA 模式会先证明第二副本接受 key,再轮询两个副本直到都拒绝,并输出实际传播时间与 SLO。 + +`LITELLM_MASTER_KEY`、上游 API key 与虚拟 key 不会作为 `curl`、`jq` 或其他子进程的命令参数传递。脚本以 `umask 077` 创建工作目录,所有 header、请求、响应和 key 文件均为 `0600`,退出时删除;创建的 user、credential、model 和两个测试 key 也会清理。上游凭据仅由 smoke 客户端读取,不会注入 LiteLLM Compose 容器。 + +若未设置上游变量,脚本仍验证 LiteLLM readiness、PostgreSQL 连接、从每个 LiteLLM 副本到 Redis 的认证连通性、管理面认证和 user 清理路径,并以明确的 `PENDING upstream smoke` 退出成功。它不会伪造 chat、stream、tool、usage、block/delete 或撤销传播已通过。 ## Readiness 与 Redis 结论 diff --git a/docker_litellm/demo/docker-compose.litellm.yml b/docker_litellm/demo/docker-compose.litellm.yml index b63f28e..8a1e00c 100644 --- a/docker_litellm/demo/docker-compose.litellm.yml +++ b/docker_litellm/demo/docker-compose.litellm.yml @@ -12,10 +12,6 @@ x-litellm-common: &litellm-common REDIS_PASSWORD: ${REDIS_PASSWORD:?set in .env} STORE_MODEL_IN_DB: "True" LITELLM_LOG: ${LITELLM_LOG:-INFO} - # The upstream credentials are optional and must remain in the ignored .env. - UPSTREAM_API_KEY: ${UPSTREAM_API_KEY:-} - UPSTREAM_BASE_URL: ${UPSTREAM_BASE_URL:-} - UPSTREAM_MODEL: ${UPSTREAM_MODEL:-} volumes: - ./config.yaml:/opt/litellm/config.yaml:ro depends_on: diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh index d90a6ac..0e6ae60 100755 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -1,5 +1,6 @@ #!/usr/bin/env bash -# Runs against a local, ignored docker_litellm/demo/.env. It never prints keys. +# Runs against a local, ignored docker_litellm/demo/.env. Secrets are written +# only to 0600 files below; never pass them to curl, jq, or another process. set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" @@ -7,25 +8,73 @@ DEMO_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)" ENV_FILE="${LITELLM_SMOKE_ENV_FILE:-${DEMO_DIR}/.env}" export COMPOSE_PROJECT_NAME="${LITELLM_COMPOSE_PROJECT:-litellm-baseline}" MODE="single" +SECURITY_CHECK=false +REVOCATION_SLO_MS="${LITELLM_REVOCATION_SLO_MS:-30000}" usage() { - echo "Usage: $0 [--mode single|ha]" >&2 + echo "Usage: $0 [--mode single|ha] [--security-check]" >&2 } while (($#)); do case "$1" in --mode) MODE="${2:-}"; shift 2 ;; + --security-check) SECURITY_CHECK=true; shift ;; --help|-h) usage; exit 0 ;; *) usage; exit 2 ;; esac done +need() { command -v "$1" >/dev/null || { echo "required command missing: $1" >&2; exit 2; }; } +need curl; need jq; need rg [[ "$MODE" == "single" || "$MODE" == "ha" ]] || { usage; exit 2; } +[[ "$REVOCATION_SLO_MS" =~ ^[0-9]+$ ]] || { echo "LITELLM_REVOCATION_SLO_MS must be an integer" >&2; exit 2; } + +security_check() { + local unsafe=0 + + # Reject inline secret headers, secret-bearing jq arguments, trace logging, + # and proxy-container injection of the upstream credentials. + if sed '/^security_check() {/,/^}/d' "$0" | rg -n -- '(^|[[:space:]])-H([[:space:]]|$)' \ + || sed '/^security_check() {/,/^}/d' "$0" | rg -n --pcre2 -- '--header\s+["'"'"']?Authorization:' \ + || sed '/^security_check() {/,/^}/d' "$0" | rg -n --pcre2 '(curl|jq)[^\n]*(LITELLM_MASTER_KEY|UPSTREAM_API_KEY|virtual_key)' \ + || sed '/^security_check() {/,/^}/d' "$0" | rg -n --pcre2 -- '--arg(?:json)?\s+[^[:space:]]*(key|secret|token)' \ + || sed '/^security_check() {/,/^}/d' "$0" | rg -n -- 'set -x' \ + || git diff --no-ext-diff -- . | rg -n --pcre2 '(?:sk-|Bearer\s+)[A-Za-z0-9_-]{24,}' \ + || rg -n '^ UPSTREAM_(API_KEY|BASE_URL|MODEL):' "$DEMO_DIR/docker-compose.litellm.yml"; then + unsafe=1 + fi + + rg -q 'umask 077' "$0" \ + && rg -q 'chmod 600' "$0" \ + && rg -q 'export -n LITELLM_MASTER_KEY' "$0" \ + && rg -q 'unset LITELLM_MASTER_KEY' "$0" \ + && rg -q 'trap cleanup EXIT' "$0" \ + && rg -q 'rm -rf "\$tmpdir"' "$0" \ + || unsafe=1 + + if ((unsafe)); then + echo "FAIL security negative check: unsafe secret transport or cleanup invariant" >&2 + return 1 + fi + echo "PASS security negative check: no inline process arguments/log tracing/Git secrets, no upstream Compose injection, 0600 cleanup invariant present." +} + +if [[ "$SECURITY_CHECK" == true ]]; then + security_check + exit 0 +fi + [[ -f "$ENV_FILE" ]] || { echo "missing local environment file: $ENV_FILE (copy .env.example)" >&2; exit 2; } +# Do not use `set -a`: sourced values must not leak to child processes. # shellcheck disable=SC1090 -set -a; source "$ENV_FILE"; set +a -: "${LITELLM_MASTER_KEY:?missing LITELLM_MASTER_KEY in .env}" +source "$ENV_FILE" +: "${LITELLM_MASTER_KEY:?missing LITELLM_MASTER_KEY in local environment file}" + +# An `.env` may use `export NAME=...`; remove that export attribute before +# mktemp, chmod, tr, curl, jq, or any other child process is started. +export -n LITELLM_MASTER_KEY UPSTREAM_API_KEY UPSTREAM_BASE_URL UPSTREAM_MODEL \ + POSTGRES_PASSWORD REDIS_PASSWORD DATABASE_URL 2>/dev/null || true if [[ "$MODE" == "single" ]]; then BASE_URL="http://${LITELLM_PUBLISH_HOST:-127.0.0.1}:${LITELLM_1_PORT:-4000}" @@ -35,31 +84,134 @@ else PEER_URL="http://${LITELLM_PUBLISH_HOST:-127.0.0.1}:${LITELLM_2_PORT:-4001}" fi -need() { command -v "$1" >/dev/null || { echo "required command missing: $1" >&2; exit 2; }; } -need curl; need jq +umask 077 +tmpdir="$(mktemp -d "${TMPDIR:-/tmp}/litellm-smoke.XXXXXX")" +chmod 700 "$tmpdir" -tmpdir="$(mktemp -d)" -trap 'rm -rf "$tmpdir"' EXIT +private_file() { + : > "$1" + chmod 600 "$1" +} + +write_private_value() { + private_file "$1" + printf '%s' "$2" > "$1" +} + +assert_private_file() { + local mode + mode="$(stat -f '%Lp' "$1" 2>/dev/null || stat -c '%a' "$1")" + [[ "$mode" == "600" ]] || { echo "temporary secret file is not 0600: $1" >&2; exit 1; } +} + +make_header_file() { + local header_file="$1" secret_file="$2" + private_file "$header_file" + { + printf 'Authorization: Bearer ' + tr -d '\r\n' < "$secret_file" + printf '\nContent-Type: application/json\n' + } > "$header_file" + assert_private_file "$header_file" +} + +master_key_file="$tmpdir/master-key" +upstream_key_file="$tmpdir/upstream-key" +upstream_base_file="$tmpdir/upstream-base" +upstream_model_file="$tmpdir/upstream-model" +admin_headers="$tmpdir/admin.headers" +write_private_value "$master_key_file" "$LITELLM_MASTER_KEY" +write_private_value "$upstream_key_file" "${UPSTREAM_API_KEY:-}" +write_private_value "$upstream_base_file" "${UPSTREAM_BASE_URL:-}" +write_private_value "$upstream_model_file" "${UPSTREAM_MODEL:-}" +make_header_file "$admin_headers" "$master_key_file" +assert_private_file "$master_key_file" +assert_private_file "$upstream_key_file" +assert_private_file "$upstream_base_file" +assert_private_file "$upstream_model_file" + +# No child process needs these values. Compose reads its own --env-file. +unset LITELLM_MASTER_KEY UPSTREAM_API_KEY UPSTREAM_BASE_URL UPSTREAM_MODEL \ + POSTGRES_PASSWORD REDIS_PASSWORD DATABASE_URL + +test_user="" +credential_name="" +model_name="" +model_id="" +block_key_created=false +delete_key_created=false +block_key_file="$tmpdir/block.key" +delete_key_file="$tmpdir/delete.key" +block_headers="$tmpdir/block.headers" +delete_headers="$tmpdir/delete.headers" request_admin() { - local method="$1" path="$2" data="${3:-}" out="$4" data_file - if [[ -n "$data" ]]; then - data_file="$tmpdir/admin-request.json" - printf '%s' "$data" > "$data_file" - curl --silent --show-error --fail --max-time 30 -X "$method" "$BASE_URL$path" \ - -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json' \ - --data-binary "@$data_file" -o "$out" - else - curl --silent --show-error --fail --max-time 30 -X "$method" "$BASE_URL$path" \ - -H "Authorization: Bearer $LITELLM_MASTER_KEY" -o "$out" + local method="$1" path="$2" payload_file="$3" output_file="$4" + local curl_args=(--silent --show-error --fail --max-time 30 --request "$method" "$BASE_URL$path" --header "@$admin_headers" --output "$output_file") + if [[ -n "$payload_file" ]]; then + curl_args+=(--data-binary "@$payload_file") + fi + curl "${curl_args[@]}" +} + +cleanup_request_admin() { + local method="$1" path="$2" payload_file="$3" + local curl_args=(--silent --show-error --max-time 15 --request "$method" "$BASE_URL$path" --header "@$admin_headers" --output /dev/null) + if [[ -n "$payload_file" ]]; then + curl_args+=(--data-binary "@$payload_file") fi + curl "${curl_args[@]}" >/dev/null 2>&1 || true +} + +cleanup() { + local exit_code=$? + set +e + if [[ "$delete_key_created" == true ]]; then + cleanup_request_admin POST /key/delete "$tmpdir/delete-key-cleanup.json" + fi + if [[ "$block_key_created" == true ]]; then + cleanup_request_admin POST /key/delete "$tmpdir/block-key-cleanup.json" + fi + if [[ -n "$model_id" ]]; then + cleanup_request_admin POST /model/delete "$tmpdir/model-delete.json" + fi + if [[ -n "$credential_name" ]]; then + cleanup_request_admin DELETE "/credentials/$credential_name" "" + fi + if [[ -n "$test_user" ]]; then + cleanup_request_admin POST /user/delete "$tmpdir/user-delete.json" + fi + unset master_key_file upstream_key_file upstream_base_file upstream_model_file + rm -rf "$tmpdir" + [[ ! -e "$tmpdir" ]] || { echo "temporary smoke directory cleanup failed" >&2; exit 1; } + exit "$exit_code" +} +trap cleanup EXIT + +request_data_get() { + local url="$1" header_file="$2" path="$3" output_file="$4" + curl --silent --show-error --fail --max-time 30 --request GET "$url$path" \ + --header "@$header_file" --output "$output_file" +} + +request_data_post() { + local url="$1" header_file="$2" path="$3" payload_file="$4" output_file="$5" + curl --silent --show-error --fail --max-time 60 --request POST "$url$path" \ + --header "@$header_file" --data-binary "@$payload_file" --output "$output_file" +} + +key_status() { + local url="$1" header_file="$2" + curl --silent --show-error --max-time 10 --request GET "$url/v1/models" \ + --header "@$header_file" --output /dev/null --write-out '%{http_code}' || true } wait_ready() { - local url="$1" i + local url="$1" output_file="$tmpdir/readiness.json" i for i in $(seq 1 60); do - if curl --silent --show-error --fail --max-time 3 "$url/health/readiness" -o "$tmpdir/readiness.json"; then - jq -e '.status == "healthy" and .db == "connected"' "$tmpdir/readiness.json" >/dev/null && return 0 + if curl --silent --fail --max-time 3 "$url/health/readiness" --output "$output_file" \ + && jq -e '.status == "healthy" and .db == "connected"' "$output_file" >/dev/null; then + return 0 fi sleep 2 done @@ -73,74 +225,192 @@ assert_redis() { python3 -c 'import os, redis; assert redis.Redis(host="redis", port=6379, password=os.environ["REDIS_PASSWORD"]).ping()' } +now_ms() { + python3 -c 'import time; print(time.time_ns() // 1_000_000)' +} + +make_key_payload() { + local alias_file="$1" payload_file="$2" + jq -n \ + --rawfile alias "$alias_file" \ + --rawfile user "$tmpdir/test-user" \ + --rawfile model "$tmpdir/model-name" \ + '{key_alias: ($alias | rtrimstr("\n")), user_id: ($user | rtrimstr("\n")), models: [($model | rtrimstr("\n"))], duration: "15m", max_budget: 0.05, rpm_limit: 10, tpm_limit: 1000, key_type: "llm_api", allowed_routes: ["/v1/models", "/v1/chat/completions"]}' \ + > "$payload_file" + chmod 600 "$payload_file" +} + +make_key_header() { + local response_file="$1" key_file="$2" header_file="$3" + private_file "$key_file" + jq -er '.key | select(type == "string" and length > 0)' "$response_file" > "$key_file" + assert_private_file "$key_file" + make_header_file "$header_file" "$key_file" +} + +make_key_action_payload() { + local action="$1" key_file="$2" payload_file="$3" + if [[ "$action" == block ]]; then + jq -n --rawfile key "$key_file" '{key: ($key | rtrimstr("\n"))}' > "$payload_file" + else + jq -n --rawfile key "$key_file" '{keys: [($key | rtrimstr("\n"))]}' > "$payload_file" + fi + chmod 600 "$payload_file" +} + +wait_model_access() { + local url="$1" header_file="$2" label="$3" output_file="$tmpdir/$label-models.json" i + for i in $(seq 1 30); do + if request_data_get "$url" "$header_file" /v1/models "$output_file" \ + && jq -e --rawfile model "$tmpdir/model-name" '.data[] | select(.id == ($model | rtrimstr("\n")))' "$output_file" >/dev/null; then + return 0 + fi + sleep 1 + done + echo "virtual key was not accepted by $label before revocation" >&2 + return 1 +} + +wait_for_rejection() { + local header_file="$1" phase="$2" start_ms now elapsed primary_code peer_code + start_ms="$(now_ms)" + while :; do + primary_code="$(key_status "$BASE_URL" "$header_file")" + peer_code="$(key_status "$PEER_URL" "$header_file")" + if [[ "$primary_code" =~ ^(401|403)$ && "$peer_code" =~ ^(401|403)$ ]]; then + now="$(now_ms)" + elapsed=$((now - start_ms)) + echo "PASS $phase propagation: primary=$primary_code peer=$peer_code elapsed_ms=$elapsed slo_ms=$REVOCATION_SLO_MS" + return 0 + fi + now="$(now_ms)" + if ((now - start_ms >= REVOCATION_SLO_MS)); then + echo "FAIL $phase propagation: primary=$primary_code peer=$peer_code exceeded_slo_ms=$REVOCATION_SLO_MS" >&2 + return 1 + fi + sleep 1 + done +} + +assert_spend() { + local spend_file="$tmpdir/spend.json" request_count total_tokens i + for i in $(seq 1 30); do + if request_admin GET "/spend/logs?user_id=$test_user" "" "$spend_file" \ + && jq -e \ + --rawfile model "$tmpdir/model-name" \ + --rawfile alias "$tmpdir/block-key-alias" \ + 'def metadata_alias: if (.metadata | type) == "object" then (.metadata.user_api_key_alias // .metadata.key_alias // "") else "" end; [ .[] | select(.model == ($model | rtrimstr("\n")) and metadata_alias == ($alias | rtrimstr("\n")) and ((.total_tokens // 0) > 0)) ] as $logs | ($logs | length) >= 3 and (($logs | map(.total_tokens // 0) | add) > 0)' \ + "$spend_file" >/dev/null; then + request_count="$(jq --rawfile model "$tmpdir/model-name" --rawfile alias "$tmpdir/block-key-alias" 'def metadata_alias: if (.metadata | type) == "object" then (.metadata.user_api_key_alias // .metadata.key_alias // "") else "" end; [ .[] | select(.model == ($model | rtrimstr("\n")) and metadata_alias == ($alias | rtrimstr("\n")) and ((.total_tokens // 0) > 0)) ] | length' "$spend_file")" + total_tokens="$(jq --rawfile model "$tmpdir/model-name" --rawfile alias "$tmpdir/block-key-alias" 'def metadata_alias: if (.metadata | type) == "object" then (.metadata.user_api_key_alias // .metadata.key_alias // "") else "" end; [ .[] | select(.model == ($model | rtrimstr("\n")) and metadata_alias == ($alias | rtrimstr("\n")) and ((.total_tokens // 0) > 0)) ] | map(.total_tokens // 0) | add' "$spend_file")" + echo "PASS spend: request_count=$request_count total_tokens=$total_tokens" + return 0 + fi + sleep 2 + done + echo "spend logs did not prove three token-bearing requests for this model and virtual key alias" >&2 + return 1 +} + wait_ready "$BASE_URL" if [[ "$MODE" == "ha" ]]; then wait_ready "$PEER_URL"; fi assert_redis litellm-1 if [[ "$MODE" == "ha" ]]; then assert_redis litellm-2; fi echo "PASS readiness: PostgreSQL connected; Redis independently reachable from LiteLLM replica(s)." -suffix="$(date +%s)" +suffix="$(date +%s)-$RANDOM" test_user="p1-smoke-user-$suffix" +credential_name="p1-smoke-upstream-$suffix" model_name="p1-smoke-model-$suffix" -key_alias="p1-smoke-key-$suffix" +write_private_value "$tmpdir/test-user" "$test_user" +write_private_value "$tmpdir/credential-name" "$credential_name" +write_private_value "$tmpdir/model-name" "$model_name" -# A user is persisted even without an upstream credential. No returned user key -# is retained or printed. -request_admin POST /user/new "$(jq -nc --arg user_id "$test_user" '{user_id:$user_id,auto_create_key:false,user_role:"internal_user"}')" "$tmpdir/user.json" -jq -e --arg id "$test_user" '.user_id == $id' "$tmpdir/user.json" >/dev/null +jq -n --rawfile user "$tmpdir/test-user" '{user_id: ($user | rtrimstr("\n")), auto_create_key: false, user_role: "internal_user"}' > "$tmpdir/user-create.json" +chmod 600 "$tmpdir/user-create.json" +jq -n --rawfile user "$tmpdir/test-user" '{user_ids: [($user | rtrimstr("\n"))]}' > "$tmpdir/user-delete.json" +chmod 600 "$tmpdir/user-delete.json" +request_admin POST /user/new "$tmpdir/user-create.json" "$tmpdir/user.json" +jq -e --rawfile user "$tmpdir/test-user" '.user_id == ($user | rtrimstr("\n"))' "$tmpdir/user.json" >/dev/null -if [[ -z "${UPSTREAM_API_KEY:-}" || -z "${UPSTREAM_BASE_URL:-}" || -z "${UPSTREAM_MODEL:-}" ]]; then - echo "PENDING upstream smoke: set UPSTREAM_API_KEY, UPSTREAM_BASE_URL and UPSTREAM_MODEL in ignored .env." - echo "PASS infrastructure: PostgreSQL persistence, shared Redis reachability and management authentication are ready." +if [[ ! -s "$upstream_key_file" || ! -s "$upstream_base_file" || ! -s "$upstream_model_file" ]]; then + echo "PENDING upstream smoke: set UPSTREAM_API_KEY, UPSTREAM_BASE_URL and UPSTREAM_MODEL in ignored local environment file." + echo "PASS infrastructure: PostgreSQL persistence, shared Redis reachability, management authentication and cleanup path are ready." exit 0 fi -# The credential request is deliberately sent from the ignored environment and -# never saved to a repository file or stdout. LiteLLM returns masked values. -credential_name="p1-smoke-upstream-$suffix" -credential_payload="$(jq -nc --arg name "$credential_name" --arg key "$UPSTREAM_API_KEY" --arg base "$UPSTREAM_BASE_URL" '{credential_name:$name,credential_values:{api_key:$key,api_base:$base},credential_info:{custom_llm_provider:"openai"}}')" -request_admin POST /credentials "$credential_payload" "$tmpdir/credential.json" +# The upstream key appears only in this 0600 request file. The model itself +# references the stored credential, never the upstream key directly. +jq -n \ + --rawfile credential_name "$tmpdir/credential-name" \ + --rawfile api_key "$upstream_key_file" \ + --rawfile api_base "$upstream_base_file" \ + '{credential_name: ($credential_name | rtrimstr("\n")), credential_values: {api_key: ($api_key | rtrimstr("\n")), api_base: ($api_base | rtrimstr("\n"))}, credential_info: {custom_llm_provider: "openai"}}' \ + > "$tmpdir/credential-create.json" +chmod 600 "$tmpdir/credential-create.json" +request_admin POST /credentials "$tmpdir/credential-create.json" "$tmpdir/credential.json" +jq -e '.success == true' "$tmpdir/credential.json" >/dev/null -model_payload="$(jq -nc --arg model_name "$model_name" --arg upstream "$UPSTREAM_MODEL" --arg base "$UPSTREAM_BASE_URL" --arg key "$UPSTREAM_API_KEY" '{model_name:$model_name,litellm_params:{model:("openai/" + $upstream),api_base:$base,api_key:$key},model_info:{id:null,mode:"chat"}}')" -request_admin POST /model/new "$model_payload" "$tmpdir/model.json" +jq -n \ + --rawfile model_name "$tmpdir/model-name" \ + --rawfile upstream_model "$upstream_model_file" \ + --rawfile credential_name "$tmpdir/credential-name" \ + '{model_name: ($model_name | rtrimstr("\n")), litellm_params: {model: ("openai/" + ($upstream_model | rtrimstr("\n"))), litellm_credential_name: ($credential_name | rtrimstr("\n"))}, model_info: {mode: "chat"}}' \ + > "$tmpdir/model-create.json" +chmod 600 "$tmpdir/model-create.json" +request_admin POST /model/new "$tmpdir/model-create.json" "$tmpdir/model.json" +model_id="$(jq -er '.model_id' "$tmpdir/model.json")" +write_private_value "$tmpdir/model-id" "$model_id" +jq -n --rawfile id "$tmpdir/model-id" '{id: ($id | rtrimstr("\n"))}' > "$tmpdir/model-delete.json" +chmod 600 "$tmpdir/model-delete.json" -key_payload="$(jq -nc --arg alias "$key_alias" --arg user "$test_user" --arg model "$model_name" '{key_alias:$alias,user_id:$user,models:[$model],duration:"15m",max_budget:0.05,rpm_limit:10,tpm_limit:1000,key_type:"llm_api",allowed_routes:["/v1/models","/v1/chat/completions"]}')" -request_admin POST /key/generate "$key_payload" "$tmpdir/key.json" -virtual_key="$(jq -er '.key' "$tmpdir/key.json")" +write_private_value "$tmpdir/block-key-alias" "p1-smoke-block-$suffix" +make_key_payload "$tmpdir/block-key-alias" "$tmpdir/block-key-create.json" +request_admin POST /key/generate "$tmpdir/block-key-create.json" "$tmpdir/block-key.json" +make_key_header "$tmpdir/block-key.json" "$block_key_file" "$block_headers" +block_key_created=true +make_key_action_payload delete "$block_key_file" "$tmpdir/block-key-cleanup.json" -data_request() { - local url="$1" path="$2" data="$3" out="$4" data_file - data_file="$tmpdir/data-request.json" - printf '%s' "$data" > "$data_file" - curl --silent --show-error --fail --max-time 60 "$url$path" \ - -H "Authorization: Bearer $virtual_key" -H 'Content-Type: application/json' --data-binary "@$data_file" -o "$out" -} +# GET must be explicit: this verifies both authorization and model visibility. +wait_model_access "$BASE_URL" "$block_headers" primary +if [[ "$MODE" == "ha" ]]; then + wait_model_access "$PEER_URL" "$block_headers" peer + echo "PASS HA pre-revocation: second replica accepted the virtual key." +fi -data_request "$BASE_URL" /v1/models '{}' "$tmpdir/models.json" -jq -e --arg model "$model_name" '.data[] | select(.id == $model)' "$tmpdir/models.json" >/dev/null -chat_payload="$(jq -nc --arg model "$model_name" '{model:$model,messages:[{role:"user",content:"Reply with OK."}],max_tokens:16}')" -data_request "$BASE_URL" /v1/chat/completions "$chat_payload" "$tmpdir/chat.json" +jq -n --rawfile model "$tmpdir/model-name" '{model: ($model | rtrimstr("\n")), messages: [{role: "user", content: "Reply with OK."}], max_tokens: 16}' > "$tmpdir/chat-request.json" +chmod 600 "$tmpdir/chat-request.json" +request_data_post "$BASE_URL" "$block_headers" /v1/chat/completions "$tmpdir/chat-request.json" "$tmpdir/chat.json" jq -e '.choices[0].message.content | type == "string"' "$tmpdir/chat.json" >/dev/null -stream_payload="$(jq -nc --argjson base "$chat_payload" '$base + {stream:true}')" -printf '%s' "$stream_payload" > "$tmpdir/stream-request.json" -curl --silent --show-error --fail --max-time 60 -N "$BASE_URL/v1/chat/completions" \ - -H "Authorization: Bearer $virtual_key" -H 'Content-Type: application/json' --data-binary "@$tmpdir/stream-request.json" > "$tmpdir/stream.txt" +jq '. + {stream: true}' "$tmpdir/chat-request.json" > "$tmpdir/stream-request.json" +chmod 600 "$tmpdir/stream-request.json" +request_data_post "$BASE_URL" "$block_headers" /v1/chat/completions "$tmpdir/stream-request.json" "$tmpdir/stream.txt" rg -q '^data: ' "$tmpdir/stream.txt" -tool_payload="$(jq -nc --arg model "$model_name" '{model:$model,messages:[{role:"user",content:"Use the supplied function to answer 2+2."}],tools:[{type:"function",function:{name:"answer",description:"Return the answer.",parameters:{type:"object",properties:{answer:{type:"integer"}},required:["answer"]}}}],tool_choice:{type:"function",function:{name:"answer"}},max_tokens:32}')" -data_request "$BASE_URL" /v1/chat/completions "$tool_payload" "$tmpdir/tool.json" +jq -n --rawfile model "$tmpdir/model-name" '{model: ($model | rtrimstr("\n")), messages: [{role: "user", content: "Use the supplied function to answer 2+2."}], tools: [{type: "function", function: {name: "answer", description: "Return the answer.", parameters: {type: "object", properties: {answer: {type: "integer"}}, required: ["answer"]}}}], tool_choice: {type: "function", function: {name: "answer"}}, max_tokens: 32}' > "$tmpdir/tool-request.json" +chmod 600 "$tmpdir/tool-request.json" +request_data_post "$BASE_URL" "$block_headers" /v1/chat/completions "$tmpdir/tool-request.json" "$tmpdir/tool.json" jq -e '.choices[0].message.tool_calls | type == "array"' "$tmpdir/tool.json" >/dev/null +assert_spend -request_admin GET /spend/logs '' "$tmpdir/spend.json" -jq -e 'type == "array" or has("data")' "$tmpdir/spend.json" >/dev/null +make_key_action_payload block "$block_key_file" "$tmpdir/block-key.json" +request_admin POST /key/block "$tmpdir/block-key.json" "$tmpdir/block-response.json" +wait_for_rejection "$block_headers" block -request_admin POST /key/block "$(jq -nc --arg key "$virtual_key" '{key:$key}')" "$tmpdir/block.json" -if curl --silent --show-error --max-time 20 --output /dev/null --write-out '%{http_code}' "$PEER_URL/v1/models" -H "Authorization: Bearer $virtual_key" | grep -Eq '^(401|403)$'; then - : -else - echo "revoked virtual key was accepted by $PEER_URL" >&2 - exit 1 +# Delete is validated with a different, previously unblocked key. +write_private_value "$tmpdir/delete-key-alias" "p1-smoke-delete-$suffix" +make_key_payload "$tmpdir/delete-key-alias" "$tmpdir/delete-key-create.json" +request_admin POST /key/generate "$tmpdir/delete-key-create.json" "$tmpdir/delete-key.json" +make_key_header "$tmpdir/delete-key.json" "$delete_key_file" "$delete_headers" +delete_key_created=true +make_key_action_payload delete "$delete_key_file" "$tmpdir/delete-key-cleanup.json" +wait_model_access "$BASE_URL" "$delete_headers" delete-primary +if [[ "$MODE" == "ha" ]]; then + wait_model_access "$PEER_URL" "$delete_headers" delete-peer + echo "PASS HA pre-delete: second replica accepted the independent virtual key." fi -request_admin POST /key/delete "$(jq -nc --arg key "$virtual_key" '{keys:[$key]}')" "$tmpdir/delete.json" -echo "PASS complete: user/credential/model/key, models/chat/stream/tool, spend, block/delete and revoke propagation." +request_admin POST /key/delete "$tmpdir/delete-key-cleanup.json" "$tmpdir/delete-response.json" +wait_for_rejection "$delete_headers" delete + +echo "PASS complete: user/credential/model/key cleanup, explicit GET models, chat/stream/tool, token-bearing spend, and independent block/delete propagation." From eacd309d22d5205b66bfa78892636a145aaaacc2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 03:24:06 +0800 Subject: [PATCH 04/87] =?UTF-8?q?fix:=20=E5=AE=8C=E5=96=84=20LiteLLM=20HA?= =?UTF-8?q?=20=E6=92=A4=E9=94=80=E4=B8=8E=20DeepSeek=20smoke?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/README.md | 4 +-- docker_litellm/demo/config.yaml | 6 ++++ docker_litellm/demo/scripts/smoke-baseline.sh | 31 +++++++++++++------ 3 files changed, 30 insertions(+), 11 deletions(-) diff --git a/docker_litellm/README.md b/docker_litellm/README.md index 71ea840..dadf1e3 100644 --- a/docker_litellm/README.md +++ b/docker_litellm/README.md @@ -57,7 +57,7 @@ docker compose --env-file .env -f docker-compose.litellm.yml --profile ha up -d ## 配置与安全边界 -`config.yaml` 从环境变量读取管理面 `LITELLM_MASTER_KEY`、`DATABASE_URL` 与 Redis 凭据。管理面 key 仅用于 `/user/new`、`/credentials`、`/model/new`、`/key/generate`、`/key/block` 和 `/key/delete` 等管理接口;smoke 生成的数据面虚拟 key 是短期、模型白名单、TTL、预算、RPM、TPM 与 `llm_api` 路由限制的独立 key。 +`config.yaml` 从环境变量读取管理面 `LITELLM_MASTER_KEY`、`DATABASE_URL` 与 Redis 凭据。管理面 key 仅用于 `/user/new`、`/credentials`、`/model/new`、`/key/generate`、`/key/block` 和 `/key/delete` 等管理接口;smoke 生成的数据面虚拟 key 是短期、模型白名单、TTL、预算、RPM、TPM 与 `llm_api` 路由限制的独立 key。双副本基线启用 `enable_redis_auth_cache`,并将 `user_api_key_cache_ttl` 设为 1 秒,以使撤销在 30 秒 smoke SLO 内经共享 Redis 重新校验。 | 变量 | 是否必填 | 作用 | 风险说明 | | --- | ---: | --- | --- | @@ -82,7 +82,7 @@ cd docker_litellm/demo `--security-check` 不读取 `.env`、不启动服务也不发送上游请求;它拒绝 inline header、secret-bearing `jq --arg`、`set -x` 和 Compose 的上游凭据注入,并检查 0600 临时文件与退出清理约束。 -脚本在真实上游变量存在时执行:创建测试用户、保存测试上游凭证、以 `litellm_credential_name` 创建模型、生成受限虚拟 key、显式 `GET /v1/models`、chat、stream、tool call、token-bearing usage 查询、block,以及独立 key 的 delete。HA 模式会先证明第二副本接受 key,再轮询两个副本直到都拒绝,并输出实际传播时间与 SLO。 +脚本在真实上游变量存在时执行:创建测试用户、保存测试上游凭证、以 `litellm_credential_name` 创建模型、生成受限虚拟 key、显式 `GET /v1/models`、chat、stream、tool call、token-bearing usage 查询、block,以及独立 key 的 delete。DeepSeek V4 使用 `deepseek/` 的原生 provider,避免被通用 OpenAI provider 丢弃 `thinking` 参数。HA 模式会先证明第二副本接受 key,再轮询两个副本直到都拒绝,并输出实际传播时间与 SLO。 `LITELLM_MASTER_KEY`、上游 API key 与虚拟 key 不会作为 `curl`、`jq` 或其他子进程的命令参数传递。脚本以 `umask 077` 创建工作目录,所有 header、请求、响应和 key 文件均为 `0600`,退出时删除;创建的 user、credential、model 和两个测试 key 也会清理。上游凭据仅由 smoke 客户端读取,不会注入 LiteLLM Compose 容器。 diff --git a/docker_litellm/demo/config.yaml b/docker_litellm/demo/config.yaml index 7f26a0b..3c1b87b 100644 --- a/docker_litellm/demo/config.yaml +++ b/docker_litellm/demo/config.yaml @@ -8,12 +8,18 @@ general_settings: master_key: os.environ/LITELLM_MASTER_KEY database_url: os.environ/DATABASE_URL store_model_in_db: true + # Keep per-replica virtual-key authorization state short-lived so a revoked + # key is revalidated through the shared cache within the 30 s smoke SLO. + user_api_key_cache_ttl: 1 # Preserve request metadata needed for metering, but do not configure prompt # or completion-content logging in this local baseline. disable_spend_logs: false litellm_settings: cache: true + # Make virtual-key authorization cache state visible to every LiteLLM + # replica. Without this, each replica keeps an isolated in-memory key cache. + enable_redis_auth_cache: true cache_params: type: redis host: os.environ/REDIS_HOST diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh index 0e6ae60..5bfd433 100755 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -248,6 +248,14 @@ make_key_header() { make_header_file "$header_file" "$key_file" } +hash_key_file() { + local key_file="$1" hash_file="$2" + private_file "$hash_file" + python3 -c 'import hashlib, sys; print(hashlib.sha256(sys.stdin.buffer.read().rstrip(b"\r\n")).hexdigest())' \ + < "$key_file" > "$hash_file" + assert_private_file "$hash_file" +} + make_key_action_payload() { local action="$1" key_file="$2" payload_file="$3" if [[ "$action" == block ]]; then @@ -259,7 +267,8 @@ make_key_action_payload() { } wait_model_access() { - local url="$1" header_file="$2" label="$3" output_file="$tmpdir/$label-models.json" i + local url="$1" header_file="$2" label="$3" output_file i + output_file="$tmpdir/$label-models.json" for i in $(seq 1 30); do if request_data_get "$url" "$header_file" /v1/models "$output_file" \ && jq -e --rawfile model "$tmpdir/model-name" '.data[] | select(.id == ($model | rtrimstr("\n")))' "$output_file" >/dev/null; then @@ -298,11 +307,11 @@ assert_spend() { if request_admin GET "/spend/logs?user_id=$test_user" "" "$spend_file" \ && jq -e \ --rawfile model "$tmpdir/model-name" \ - --rawfile alias "$tmpdir/block-key-alias" \ - 'def metadata_alias: if (.metadata | type) == "object" then (.metadata.user_api_key_alias // .metadata.key_alias // "") else "" end; [ .[] | select(.model == ($model | rtrimstr("\n")) and metadata_alias == ($alias | rtrimstr("\n")) and ((.total_tokens // 0) > 0)) ] as $logs | ($logs | length) >= 3 and (($logs | map(.total_tokens // 0) | add) > 0)' \ + --rawfile key_hash "$tmpdir/block-key-sha256" \ + 'def logs: if type == "array" then . else (.data // []) end; [ logs[] | select((.model == ($model | rtrimstr("\n")) or .model_group == ($model | rtrimstr("\n"))) and .api_key == ($key_hash | rtrimstr("\n")) and ((.total_tokens // 0) > 0)) ] as $logs | ($logs | length) >= 3 and (($logs | map(.total_tokens // 0) | add) > 0)' \ "$spend_file" >/dev/null; then - request_count="$(jq --rawfile model "$tmpdir/model-name" --rawfile alias "$tmpdir/block-key-alias" 'def metadata_alias: if (.metadata | type) == "object" then (.metadata.user_api_key_alias // .metadata.key_alias // "") else "" end; [ .[] | select(.model == ($model | rtrimstr("\n")) and metadata_alias == ($alias | rtrimstr("\n")) and ((.total_tokens // 0) > 0)) ] | length' "$spend_file")" - total_tokens="$(jq --rawfile model "$tmpdir/model-name" --rawfile alias "$tmpdir/block-key-alias" 'def metadata_alias: if (.metadata | type) == "object" then (.metadata.user_api_key_alias // .metadata.key_alias // "") else "" end; [ .[] | select(.model == ($model | rtrimstr("\n")) and metadata_alias == ($alias | rtrimstr("\n")) and ((.total_tokens // 0) > 0)) ] | map(.total_tokens // 0) | add' "$spend_file")" + request_count="$(jq --rawfile model "$tmpdir/model-name" --rawfile key_hash "$tmpdir/block-key-sha256" 'def logs: if type == "array" then . else (.data // []) end; [ logs[] | select((.model == ($model | rtrimstr("\n")) or .model_group == ($model | rtrimstr("\n"))) and .api_key == ($key_hash | rtrimstr("\n")) and ((.total_tokens // 0) > 0)) ] | length' "$spend_file")" + total_tokens="$(jq --rawfile model "$tmpdir/model-name" --rawfile key_hash "$tmpdir/block-key-sha256" 'def logs: if type == "array" then . else (.data // []) end; [ logs[] | select((.model == ($model | rtrimstr("\n")) or .model_group == ($model | rtrimstr("\n"))) and .api_key == ($key_hash | rtrimstr("\n")) and ((.total_tokens // 0) > 0)) ] | map(.total_tokens // 0) | add' "$spend_file")" echo "PASS spend: request_count=$request_count total_tokens=$total_tokens" return 0 fi @@ -345,7 +354,7 @@ jq -n \ --rawfile credential_name "$tmpdir/credential-name" \ --rawfile api_key "$upstream_key_file" \ --rawfile api_base "$upstream_base_file" \ - '{credential_name: ($credential_name | rtrimstr("\n")), credential_values: {api_key: ($api_key | rtrimstr("\n")), api_base: ($api_base | rtrimstr("\n"))}, credential_info: {custom_llm_provider: "openai"}}' \ + '{credential_name: ($credential_name | rtrimstr("\n")), credential_values: {api_key: ($api_key | rtrimstr("\n")), api_base: ($api_base | rtrimstr("\n"))}, credential_info: {custom_llm_provider: "deepseek"}}' \ > "$tmpdir/credential-create.json" chmod 600 "$tmpdir/credential-create.json" request_admin POST /credentials "$tmpdir/credential-create.json" "$tmpdir/credential.json" @@ -355,7 +364,7 @@ jq -n \ --rawfile model_name "$tmpdir/model-name" \ --rawfile upstream_model "$upstream_model_file" \ --rawfile credential_name "$tmpdir/credential-name" \ - '{model_name: ($model_name | rtrimstr("\n")), litellm_params: {model: ("openai/" + ($upstream_model | rtrimstr("\n"))), litellm_credential_name: ($credential_name | rtrimstr("\n"))}, model_info: {mode: "chat"}}' \ + '{model_name: ($model_name | rtrimstr("\n")), litellm_params: {model: ("deepseek/" + ($upstream_model | rtrimstr("\n"))), litellm_credential_name: ($credential_name | rtrimstr("\n"))}, model_info: {mode: "chat"}}' \ > "$tmpdir/model-create.json" chmod 600 "$tmpdir/model-create.json" request_admin POST /model/new "$tmpdir/model-create.json" "$tmpdir/model.json" @@ -368,6 +377,7 @@ write_private_value "$tmpdir/block-key-alias" "p1-smoke-block-$suffix" make_key_payload "$tmpdir/block-key-alias" "$tmpdir/block-key-create.json" request_admin POST /key/generate "$tmpdir/block-key-create.json" "$tmpdir/block-key.json" make_key_header "$tmpdir/block-key.json" "$block_key_file" "$block_headers" +hash_key_file "$block_key_file" "$tmpdir/block-key-sha256" block_key_created=true make_key_action_payload delete "$block_key_file" "$tmpdir/block-key-cleanup.json" @@ -388,10 +398,13 @@ chmod 600 "$tmpdir/stream-request.json" request_data_post "$BASE_URL" "$block_headers" /v1/chat/completions "$tmpdir/stream-request.json" "$tmpdir/stream.txt" rg -q '^data: ' "$tmpdir/stream.txt" -jq -n --rawfile model "$tmpdir/model-name" '{model: ($model | rtrimstr("\n")), messages: [{role: "user", content: "Use the supplied function to answer 2+2."}], tools: [{type: "function", function: {name: "answer", description: "Return the answer.", parameters: {type: "object", properties: {answer: {type: "integer"}}, required: ["answer"]}}}], tool_choice: {type: "function", function: {name: "answer"}}, max_tokens: 32}' > "$tmpdir/tool-request.json" +jq -n --rawfile model "$tmpdir/model-name" '{model: ($model | rtrimstr("\n")), messages: [{role: "user", content: "Use the supplied function to answer 2+2."}], tools: [{type: "function", function: {name: "answer", description: "Return the answer.", parameters: {type: "object", properties: {answer: {type: "integer"}}, required: ["answer"]}}}], tool_choice: {type: "function", function: {name: "answer"}}, thinking: {type: "disabled"}, max_tokens: 32}' > "$tmpdir/tool-request.json" chmod 600 "$tmpdir/tool-request.json" request_data_post "$BASE_URL" "$block_headers" /v1/chat/completions "$tmpdir/tool-request.json" "$tmpdir/tool.json" -jq -e '.choices[0].message.tool_calls | type == "array"' "$tmpdir/tool.json" >/dev/null +if ! jq -e '.choices[0].message.tool_calls | type == "array" and length > 0' "$tmpdir/tool.json" >/dev/null; then + echo "tool request returned no tool_calls" >&2 + exit 1 +fi assert_spend make_key_action_payload block "$block_key_file" "$tmpdir/block-key.json" From f45b760347a623f99af92b94925cd1680bc2110d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 03:47:47 +0800 Subject: [PATCH 05/87] =?UTF-8?q?fix:=20=E7=BC=A9=E7=9F=AD=20LiteLLM=20Red?= =?UTF-8?q?is=20=E6=81=A2=E5=A4=8D=E7=AA=97=E5=8F=A3?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/README.md | 3 ++- docker_litellm/demo/docker-compose.litellm.yml | 1 + 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/docker_litellm/README.md b/docker_litellm/README.md index dadf1e3..a82e366 100644 --- a/docker_litellm/README.md +++ b/docker_litellm/README.md @@ -68,6 +68,7 @@ docker compose --env-file .env -f docker-compose.litellm.yml --profile ha up -d | `UPSTREAM_BASE_URL` | 真实调用时是 | OpenAI 兼容上游地址 | 由测试环境决定 | | `UPSTREAM_MODEL` | 真实调用时是 | 上游模型名 | 用于创建测试模型 | | `LITELLM_REVOCATION_SLO_MS` | `30000` | block/delete 的跨副本拒绝 SLO | smoke 会输出实际传播耗时;仅用于本地验收 | +| `REDIS_CIRCUIT_BREAKER_RECOVERY_TIMEOUT` | `5` | Redis 断连后的 LiteLLM 缓存恢复探测窗口(秒) | 本地 HA 基线应小于撤销 SLO;恢复期间管理面可能暂时返回 500 | P1 不把 LiteLLM User/Team 当作产品用户或 Workspace 的事实源;Shell 的用户、绑定和租约业务仍在后续 Phase 实现。 @@ -90,7 +91,7 @@ cd docker_litellm/demo ## Readiness 与 Redis 结论 -LiteLLM `v1.97.0-dev.1` 的公开 `/health/readiness` 仅返回服务与数据库连通性,不将 Redis 纳入公开 readiness。因此 Compose 健康检查只能确认 LiteLLM + PostgreSQL;`smoke-baseline.sh` 额外从每个 LiteLLM 容器执行 Redis `PING`。若 Redis 不可用,P1 的多副本认证缓存、限流、Spend counter 与协调结论无效,应将该运行组合标记为 `blocked`,不能降级宣称为高可用。 +LiteLLM `v1.97.0-dev.1` 的公开 `/health/readiness` 仅返回服务与数据库连通性,不将 Redis 纳入公开 readiness。因此 Compose 健康检查只能确认 LiteLLM + PostgreSQL;`smoke-baseline.sh` 额外从每个 LiteLLM 容器执行 Redis `PING`。若 Redis 不可用,P1 的多副本认证缓存、限流、Spend counter 与协调结论无效,应将该运行组合标记为 `blocked`,不能降级宣称为高可用。Redis 恢复后,LiteLLM 的认证缓存 circuit breaker 需要经过 `REDIS_CIRCUIT_BREAKER_RECOVERY_TIMEOUT` 后才会重新探测;P1 默认设为 5 秒,并要求恢复后再次跑 HA smoke。 ## 常见问题 diff --git a/docker_litellm/demo/docker-compose.litellm.yml b/docker_litellm/demo/docker-compose.litellm.yml index 8a1e00c..240632b 100644 --- a/docker_litellm/demo/docker-compose.litellm.yml +++ b/docker_litellm/demo/docker-compose.litellm.yml @@ -10,6 +10,7 @@ x-litellm-common: &litellm-common REDIS_HOST: redis REDIS_PORT: "6379" REDIS_PASSWORD: ${REDIS_PASSWORD:?set in .env} + REDIS_CIRCUIT_BREAKER_RECOVERY_TIMEOUT: ${REDIS_CIRCUIT_BREAKER_RECOVERY_TIMEOUT:-5} STORE_MODEL_IN_DB: "True" LITELLM_LOG: ${LITELLM_LOG:-INFO} volumes: From cf441baa94472d29bffe9e47232b40923e6acc8b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 04:36:45 +0800 Subject: [PATCH 06/87] =?UTF-8?q?fix:=20=E5=8A=A0=E5=9B=BA=20LiteLLM=20P1?= =?UTF-8?q?=20=E6=97=A5=E5=BF=97=E4=B8=8E=20Redis=20=E5=AF=86=E7=A0=81?= =?UTF-8?q?=E5=AE=89=E5=85=A8?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/README.md | 9 ++-- docker_litellm/demo/.env.example | 10 ++-- docker_litellm/demo/config.migrate.yaml | 20 ++++++++ docker_litellm/demo/config.yaml | 6 +++ .../demo/docker-compose.litellm.yml | 48 +++++++++++++++++-- docker_litellm/demo/scripts/smoke-baseline.sh | 32 +++++++++++-- docker_litellm/work/start-litellm.sh | 11 +++++ 7 files changed, 118 insertions(+), 18 deletions(-) create mode 100644 docker_litellm/demo/config.migrate.yaml mode change 100644 => 100755 docker_litellm/work/start-litellm.sh diff --git a/docker_litellm/README.md b/docker_litellm/README.md index a82e366..3ab014a 100644 --- a/docker_litellm/README.md +++ b/docker_litellm/README.md @@ -11,13 +11,13 @@ P1 默认不构建 LiteLLM Dashboard 静态资源:固定源码在当前构建 | LiteLLM 源码 | `v1.97.0-dev.1` / `ead62528e607b9d8e61273def638799c9c3a69ba` | Dockerfile 精确 fetch 并校验 HEAD | | FastAPI | `0.136.3` | 固定到该 LiteLLM commit 仍使用 `get_flat_dependant` 的兼容版本 | | Prisma Python client | `0.15.0` | LiteLLM 连接 PostgreSQL 所需客户端,兼容基础镜像的 Python 3.13 | - -镜像构建会对 wheel 自带的 LiteLLM Prisma schema 运行 `prisma generate`,并把生成的查询引擎固定在 `/opt/litellm/.cache`;没有该步骤,或将该缓存随 `/root/.cache` 清理,代理会在 PostgreSQL startup 时报缺少 Prisma binaries 或无法连接查询引擎。 | 本地产物镜像 | `quay.io/labnow/litellm:1.97.0-ead62528e607` | P1 Compose 的唯一 LiteLLM 默认镜像 | | PostgreSQL | `postgres:17-alpine@sha256:742f40ea20b9ff2ff31db5458d127452988a2164df9e17441e191f3b72252193` | 用户、凭证、模型、虚拟 key 与 spend 持久化 | | Redis | `redis:7.4-alpine@sha256:e7723ff73d963f5cc6d9c4643ea3d989527a402a319239054e9472a7fb9219a2` | 副本共享认证缓存、限流、Spend counter 和协调缓存 | | OpenClaw(P2 参考) | `2026.5.10-beta.1` / `eed75ed47f47deb18c9d093a2e638c9bb0bedf14` | 仅为下一阶段黄金适配器保留版本基线;P1 不启动或实现 Adapter | +镜像构建会对 wheel 自带的 LiteLLM Prisma schema 运行 `prisma generate`,并把生成的查询引擎固定在 `/opt/litellm/.cache`;没有该步骤,或将该缓存随 `/root/.cache` 清理,代理会在 PostgreSQL startup 时报缺少 Prisma binaries 或无法连接查询引擎。 + 构建前已确认完整 LabNow 镜像名是 `quay.io/labnow/litellm:1.97.0-ead62528e607`,不会推送镜像。必须通过根目录 `tool.sh` 构建,避免基础镜像退回 Docker Hub: ```bash @@ -43,7 +43,7 @@ docker image inspect quay.io/labnow/litellm:1.97.0-ead62528e607 \ cd docker_litellm/demo cp .env.example .env # 在 .env 中生成并填写 LITELLM_MASTER_KEY、POSTGRES_PASSWORD、REDIS_PASSWORD。 -# 真实上游调用另行填写 UPSTREAM_API_KEY、UPSTREAM_BASE_URL、UPSTREAM_MODEL。 +# 真实上游调用另行填写 UPSTREAM_PROVIDER、UPSTREAM_API_KEY、UPSTREAM_BASE_URL、UPSTREAM_MODEL。 docker compose --env-file .env -f docker-compose.litellm.yml --profile single up -d ``` @@ -65,6 +65,7 @@ docker compose --env-file .env -f docker-compose.litellm.yml --profile ha up -d | `POSTGRES_PASSWORD` | 是 | PostgreSQL 密码 | 仅限本地测试或部署 Secret | | `REDIS_PASSWORD` | 是 | Redis 认证 | 仅限本地测试或部署 Secret | | `UPSTREAM_API_KEY` | 真实调用时是 | 上游模型凭据 | 仅由 smoke 客户端读取;不会注入 LiteLLM 容器、不提交、不打印 | +| `UPSTREAM_PROVIDER` | 真实调用时是 | P1 provider 选择 | 当前明确支持 `deepseek`;错误组合会在调用前脱敏失败 | | `UPSTREAM_BASE_URL` | 真实调用时是 | OpenAI 兼容上游地址 | 由测试环境决定 | | `UPSTREAM_MODEL` | 真实调用时是 | 上游模型名 | 用于创建测试模型 | | `LITELLM_REVOCATION_SLO_MS` | `30000` | block/delete 的跨副本拒绝 SLO | smoke 会输出实际传播耗时;仅用于本地验收 | @@ -81,7 +82,7 @@ cd docker_litellm/demo ./scripts/smoke-baseline.sh --security-check ``` -`--security-check` 不读取 `.env`、不启动服务也不发送上游请求;它拒绝 inline header、secret-bearing `jq --arg`、`set -x` 和 Compose 的上游凭据注入,并检查 0600 临时文件与退出清理约束。 +`--security-check` 不读取 `.env`、不启动服务也不发送上游请求;它拒绝 inline header、secret-bearing `jq --arg`、`set -x`、Compose 上游凭据注入和 Redis 密码命令行展开,并检查 Docker Secret、0600 临时文件与退出清理约束。 脚本在真实上游变量存在时执行:创建测试用户、保存测试上游凭证、以 `litellm_credential_name` 创建模型、生成受限虚拟 key、显式 `GET /v1/models`、chat、stream、tool call、token-bearing usage 查询、block,以及独立 key 的 delete。DeepSeek V4 使用 `deepseek/` 的原生 provider,避免被通用 OpenAI provider 丢弃 `thinking` 参数。HA 模式会先证明第二副本接受 key,再轮询两个副本直到都拒绝,并输出实际传播时间与 SLO。 diff --git a/docker_litellm/demo/.env.example b/docker_litellm/demo/.env.example index 91ec14a..d85a31e 100644 --- a/docker_litellm/demo/.env.example +++ b/docker_litellm/demo/.env.example @@ -18,11 +18,13 @@ POSTGRES_USER=litellm POSTGRES_PASSWORD= REDIS_PASSWORD= -# Optional upstream required for chat/stream/tool smoke. Keep empty to validate -# infrastructure and management/revocation paths only. +# Optional upstream required for chat/stream/tool smoke. The smoke validates +# this provider/model pair before any request. Supported P1 provider: deepseek. +# Keep UPSTREAM_API_KEY empty to validate infrastructure paths only. +UPSTREAM_PROVIDER=deepseek UPSTREAM_API_KEY= -UPSTREAM_BASE_URL=https://api.openai.com/v1 -UPSTREAM_MODEL=gpt-4o-mini +UPSTREAM_BASE_URL=https://api.deepseek.com/v1 +UPSTREAM_MODEL=deepseek-v4-flash # P2 reference only; this phase does not start an OpenClaw adapter. OPENCLAW_IMAGE=quay.io/labnow/openclaw:2026.5.10-beta.1 diff --git a/docker_litellm/demo/config.migrate.yaml b/docker_litellm/demo/config.migrate.yaml new file mode 100644 index 0000000..91e27f6 --- /dev/null +++ b/docker_litellm/demo/config.migrate.yaml @@ -0,0 +1,20 @@ +# Dedicated one-shot migration configuration. Keep this in sync with +# config.yaml; only this job is permitted to apply Prisma migrations. +model_list: [] + +general_settings: + master_key: os.environ/LITELLM_MASTER_KEY + database_url: os.environ/DATABASE_URL + store_model_in_db: true + disable_spend_logs: false + disable_prisma_schema_update: false + +litellm_settings: + turn_off_message_logging: true + cache: true + enable_redis_auth_cache: true + cache_params: + type: redis + host: os.environ/REDIS_HOST + port: os.environ/REDIS_PORT + password: os.environ/REDIS_PASSWORD diff --git a/docker_litellm/demo/config.yaml b/docker_litellm/demo/config.yaml index 3c1b87b..592ec5c 100644 --- a/docker_litellm/demo/config.yaml +++ b/docker_litellm/demo/config.yaml @@ -14,8 +14,14 @@ general_settings: # Preserve request metadata needed for metering, but do not configure prompt # or completion-content logging in this local baseline. disable_spend_logs: false + # Proxy replicas only check the schema. The dedicated migration job uses + # config.migrate.yaml and is the sole process allowed to apply migrations. + disable_prisma_schema_update: true litellm_settings: + # LiteLLM 1.97.0 reads this from litellm_settings and redacts request and + # response content before standard/spend logging. Metering fields remain. + turn_off_message_logging: true cache: true # Make virtual-key authorization cache state visible to every LiteLLM # replica. Without this, each replica keeps an isolated in-memory key cache. diff --git a/docker_litellm/demo/docker-compose.litellm.yml b/docker_litellm/demo/docker-compose.litellm.yml index 240632b..237c198 100644 --- a/docker_litellm/demo/docker-compose.litellm.yml +++ b/docker_litellm/demo/docker-compose.litellm.yml @@ -9,12 +9,17 @@ x-litellm-common: &litellm-common DATABASE_URL: postgresql://${POSTGRES_USER:?set in .env}:${POSTGRES_PASSWORD:?set in .env}@postgres:5432/${POSTGRES_DB:-litellm} REDIS_HOST: redis REDIS_PORT: "6379" - REDIS_PASSWORD: ${REDIS_PASSWORD:?set in .env} + REDIS_PASSWORD_FILE: /run/secrets/redis_password REDIS_CIRCUIT_BREAKER_RECOVERY_TIMEOUT: ${REDIS_CIRCUIT_BREAKER_RECOVERY_TIMEOUT:-5} + STORE_PROMPTS_IN_SPEND_LOGS: "false" STORE_MODEL_IN_DB: "True" LITELLM_LOG: ${LITELLM_LOG:-INFO} volumes: - ./config.yaml:/opt/litellm/config.yaml:ro + - ./config.migrate.yaml:/opt/litellm/config.migrate.yaml:ro + - ../work/start-litellm.sh:/opt/utils/start-litellm.sh:ro + secrets: + - redis_password depends_on: postgres: condition: service_healthy @@ -30,6 +35,21 @@ x-litellm-common: &litellm-common start_period: 30s services: + # Run LiteLLM's own migration-only mode once before proxy replicas start. + # `service_completed_successfully` serializes migration across single/HA + # Compose starts; Prisma's migration table makes repeated jobs idempotent. + litellm-migrate: + <<: *litellm-common + container_name: ${LITELLM_MIGRATE_CONTAINER_NAME:-svc-litellm-migrate} + command: ["/bin/bash", "/opt/utils/start-litellm.sh", "--config", "config.migrate.yaml", "--skip_server_startup"] + healthcheck: + disable: true + depends_on: + postgres: + condition: service_healthy + redis: + condition: service_healthy + postgres: image: postgres:17-alpine@sha256:742f40ea20b9ff2ff31db5458d127452988a2164df9e17441e191f3b72252193 restart: "no" @@ -50,15 +70,23 @@ services: redis: image: redis:7.4-alpine@sha256:e7723ff73d963f5cc6d9c4643ea3d989527a402a319239054e9472a7fb9219a2 restart: "no" - environment: - REDIS_PASSWORD: ${REDIS_PASSWORD:?set in .env} - command: ["redis-server", "--appendonly", "yes", "--requirepass", "${REDIS_PASSWORD:?set in .env}"] + secrets: + - redis_password + tmpfs: + - /run/redis:mode=0700 + command: + - /bin/sh + - -ec + - >- + umask 077; + { printf 'appendonly yes\nrequirepass '; cat /run/secrets/redis_password; printf '\n'; } > /run/redis/redis.conf; + exec redis-server /run/redis/redis.conf volumes: - litellm_redis_data:/data networks: - litellm-baseline-net healthcheck: - test: ["CMD-SHELL", "redis-cli --no-auth-warning -a \"$$REDIS_PASSWORD\" ping | grep -qx PONG"] + test: ["CMD-SHELL", "REDISCLI_AUTH=\"$$(cat /run/secrets/redis_password)\" redis-cli --no-auth-warning ping | grep -qx PONG"] interval: 5s timeout: 5s retries: 20 @@ -69,6 +97,9 @@ services: profiles: ["single", "ha"] ports: - "${LITELLM_PUBLISH_HOST:-127.0.0.1}:${LITELLM_1_PORT:-4000}:4000" + depends_on: + litellm-migrate: + condition: service_completed_successfully litellm-2: <<: *litellm-common @@ -76,11 +107,18 @@ services: profiles: ["ha"] ports: - "${LITELLM_PUBLISH_HOST:-127.0.0.1}:${LITELLM_2_PORT:-4001}:4000" + depends_on: + litellm-migrate: + condition: service_completed_successfully volumes: litellm_postgres_data: litellm_redis_data: +secrets: + redis_password: + environment: REDIS_PASSWORD + networks: litellm-baseline-net: name: litellm-baseline-net diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh index 5bfd433..482c990 100755 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -40,7 +40,10 @@ security_check() { || sed '/^security_check() {/,/^}/d' "$0" | rg -n --pcre2 -- '--arg(?:json)?\s+[^[:space:]]*(key|secret|token)' \ || sed '/^security_check() {/,/^}/d' "$0" | rg -n -- 'set -x' \ || git diff --no-ext-diff -- . | rg -n --pcre2 '(?:sk-|Bearer\s+)[A-Za-z0-9_-]{24,}' \ - || rg -n '^ UPSTREAM_(API_KEY|BASE_URL|MODEL):' "$DEMO_DIR/docker-compose.litellm.yml"; then + || rg -n '^ UPSTREAM_(API_KEY|BASE_URL|MODEL|PROVIDER):' "$DEMO_DIR/docker-compose.litellm.yml" \ + || rg -n -- '--requirepass[[:space:]].*\$\{REDIS_PASSWORD|REDIS_PASSWORD:.*\$\{' "$DEMO_DIR/docker-compose.litellm.yml" \ + || ! rg -q 'redis_password:' "$DEMO_DIR/docker-compose.litellm.yml" \ + || ! rg -q 'REDIS_PASSWORD_FILE: /run/secrets/redis_password' "$DEMO_DIR/docker-compose.litellm.yml"; then unsafe=1 fi @@ -56,7 +59,7 @@ security_check() { echo "FAIL security negative check: unsafe secret transport or cleanup invariant" >&2 return 1 fi - echo "PASS security negative check: no inline process arguments/log tracing/Git secrets, no upstream Compose injection, 0600 cleanup invariant present." + echo "PASS security negative check: no inline process arguments/log tracing/Git secrets, no upstream Compose injection, Redis Docker secret and 0600 cleanup invariants present." } if [[ "$SECURITY_CHECK" == true ]]; then @@ -222,7 +225,7 @@ wait_ready() { assert_redis() { local service="$1" docker compose --env-file "$ENV_FILE" -f "$DEMO_DIR/docker-compose.litellm.yml" exec -T "$service" \ - python3 -c 'import os, redis; assert redis.Redis(host="redis", port=6379, password=os.environ["REDIS_PASSWORD"]).ping()' + python3 -c 'import redis; password=open("/run/secrets/redis_password", encoding="utf-8").read().strip(); assert redis.Redis(host="redis", port=6379, password=password).ping()' } now_ms() { @@ -348,13 +351,31 @@ if [[ ! -s "$upstream_key_file" || ! -s "$upstream_base_file" || ! -s "$upstream exit 0 fi +# P1 deliberately supports one explicit provider mapping. Reject incomplete +# or ambiguous combinations before any upstream-facing request is sent. +case "${UPSTREAM_PROVIDER:-}" in + deepseek) + provider_prefix="deepseek" + ;; + *) + echo "invalid UPSTREAM_PROVIDER: supported P1 provider is deepseek (value redacted)" >&2 + exit 2 + ;; +esac +if ! rg -q '^https://[^[:space:]]+$' "$upstream_base_file" \ + || ! rg -q '^[A-Za-z0-9._:-]+$' "$upstream_model_file"; then + echo "invalid upstream base URL or model identifier (values redacted)" >&2 + exit 2 +fi + # The upstream key appears only in this 0600 request file. The model itself # references the stored credential, never the upstream key directly. jq -n \ --rawfile credential_name "$tmpdir/credential-name" \ --rawfile api_key "$upstream_key_file" \ --rawfile api_base "$upstream_base_file" \ - '{credential_name: ($credential_name | rtrimstr("\n")), credential_values: {api_key: ($api_key | rtrimstr("\n")), api_base: ($api_base | rtrimstr("\n"))}, credential_info: {custom_llm_provider: "deepseek"}}' \ + --arg provider "$provider_prefix" \ + '{credential_name: ($credential_name | rtrimstr("\n")), credential_values: {api_key: ($api_key | rtrimstr("\n")), api_base: ($api_base | rtrimstr("\n"))}, credential_info: {custom_llm_provider: $provider}}' \ > "$tmpdir/credential-create.json" chmod 600 "$tmpdir/credential-create.json" request_admin POST /credentials "$tmpdir/credential-create.json" "$tmpdir/credential.json" @@ -364,7 +385,8 @@ jq -n \ --rawfile model_name "$tmpdir/model-name" \ --rawfile upstream_model "$upstream_model_file" \ --rawfile credential_name "$tmpdir/credential-name" \ - '{model_name: ($model_name | rtrimstr("\n")), litellm_params: {model: ("deepseek/" + ($upstream_model | rtrimstr("\n"))), litellm_credential_name: ($credential_name | rtrimstr("\n"))}, model_info: {mode: "chat"}}' \ + --arg provider "$provider_prefix" \ + '{model_name: ($model_name | rtrimstr("\n")), litellm_params: {model: ($provider + "/" + ($upstream_model | rtrimstr("\n"))), litellm_credential_name: ($credential_name | rtrimstr("\n"))}, model_info: {mode: "chat"}}' \ > "$tmpdir/model-create.json" chmod 600 "$tmpdir/model-create.json" request_admin POST /model/new "$tmpdir/model-create.json" "$tmpdir/model.json" diff --git a/docker_litellm/work/start-litellm.sh b/docker_litellm/work/start-litellm.sh old mode 100644 new mode 100755 index 93a9786..c62cdce --- a/docker_litellm/work/start-litellm.sh +++ b/docker_litellm/work/start-litellm.sh @@ -9,6 +9,17 @@ export HOME="$HOME_LITELLM" export PRISMA_HOME_DIR="${PRISMA_HOME_DIR:-$HOME_LITELLM}" cd "$HOME_LITELLM" +# Compose mounts the Redis credential as a Docker secret. Export it only in +# this process tree so it is absent from Docker inspect and command arguments. +if [ -n "${REDIS_PASSWORD_FILE:-}" ]; then + test -r "$REDIS_PASSWORD_FILE" + export REDIS_PASSWORD="$(cat "$REDIS_PASSWORD_FILE")" +fi + +# LiteLLM checks this environment variable while serializing SpendLog payloads. +# Keep metering enabled in config.yaml, but never persist prompt content. +export STORE_PROMPTS_IN_SPEND_LOGS="${STORE_PROMPTS_IN_SPEND_LOGS:-false}" + # Default config if not exists. The P1 Compose baseline always mounts an # explicit config with PostgreSQL and Redis; this fallback remains only for # backwards-compatible standalone use. From 762ba1edd4c339aa2c0c62acfdb2103b1de348a2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 04:49:00 +0800 Subject: [PATCH 07/87] =?UTF-8?q?fix:=20=E5=88=86=E7=A6=BB=20LiteLLM=20?= =?UTF-8?q?=E8=BF=81=E7=A7=BB=E4=B8=8E=E5=89=AF=E6=9C=AC=E5=90=AF=E5=8A=A8?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/README.md | 9 +++++++++ docker_litellm/demo/docker-compose.litellm.yml | 7 +------ docker_litellm/demo/scripts/run-migration.sh | 14 ++++++++++++++ 3 files changed, 24 insertions(+), 6 deletions(-) create mode 100755 docker_litellm/demo/scripts/run-migration.sh diff --git a/docker_litellm/README.md b/docker_litellm/README.md index 3ab014a..3f9747c 100644 --- a/docker_litellm/README.md +++ b/docker_litellm/README.md @@ -47,12 +47,21 @@ cp .env.example .env docker compose --env-file .env -f docker-compose.litellm.yml --profile single up -d ``` +迁移与代理启动刻意分离。每次部署先显式运行一次 migration-only job;该命令可安全重复执行。随后启动的代理副本只做 schema 检查,不会并发执行 Prisma migration: + +```bash +./scripts/run-migration.sh +docker compose --env-file .env -f docker-compose.litellm.yml --profile single up -d litellm-1 +``` + 双副本测试使用同一 PostgreSQL 与 Redis,但有两个 HTTP 入口: ```bash docker compose --env-file .env -f docker-compose.litellm.yml --profile ha up -d ``` +HA 启动前同样先运行 `./scripts/run-migration.sh`,再执行上述命令。不要把 migration profile 与代理 profile 放入同一条 `up` 命令。 + 默认端口只发布在 `127.0.0.1`:副本 1 为 `4000`,副本 2 为 `4001`。PostgreSQL 与 Redis 不发布宿主机端口。停止测试不会删除卷;如需删除测试数据,先人工确认后使用 `docker compose ... down -v`。 ## 配置与安全边界 diff --git a/docker_litellm/demo/docker-compose.litellm.yml b/docker_litellm/demo/docker-compose.litellm.yml index 237c198..a050ee0 100644 --- a/docker_litellm/demo/docker-compose.litellm.yml +++ b/docker_litellm/demo/docker-compose.litellm.yml @@ -41,6 +41,7 @@ services: litellm-migrate: <<: *litellm-common container_name: ${LITELLM_MIGRATE_CONTAINER_NAME:-svc-litellm-migrate} + profiles: ["migrate"] command: ["/bin/bash", "/opt/utils/start-litellm.sh", "--config", "config.migrate.yaml", "--skip_server_startup"] healthcheck: disable: true @@ -97,9 +98,6 @@ services: profiles: ["single", "ha"] ports: - "${LITELLM_PUBLISH_HOST:-127.0.0.1}:${LITELLM_1_PORT:-4000}:4000" - depends_on: - litellm-migrate: - condition: service_completed_successfully litellm-2: <<: *litellm-common @@ -107,9 +105,6 @@ services: profiles: ["ha"] ports: - "${LITELLM_PUBLISH_HOST:-127.0.0.1}:${LITELLM_2_PORT:-4001}:4000" - depends_on: - litellm-migrate: - condition: service_completed_successfully volumes: litellm_postgres_data: diff --git a/docker_litellm/demo/scripts/run-migration.sh b/docker_litellm/demo/scripts/run-migration.sh new file mode 100755 index 0000000..68917d5 --- /dev/null +++ b/docker_litellm/demo/scripts/run-migration.sh @@ -0,0 +1,14 @@ +#!/usr/bin/env bash +# Apply LiteLLM Prisma migrations explicitly, once per deployment operation. +# Proxy replicas deliberately do not depend on this one-shot Compose service. +set -euo pipefail + +script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +demo_dir="$(cd "${script_dir}/.." && pwd)" +env_file="${LITELLM_SMOKE_ENV_FILE:-${demo_dir}/.env}" +compose=(docker compose --env-file "$env_file" -f "${demo_dir}/docker-compose.litellm.yml") + +[[ -f "$env_file" ]] || { echo "missing ignored local environment file" >&2; exit 2; } +"${compose[@]}" up -d postgres redis +"${compose[@]}" --profile migrate run --rm --no-deps litellm-migrate +echo "PASS migration: LiteLLM migration-only job completed; proxy replicas were not started." From 33a58c6b074812e33fb4a2494a02bd13f0ad822e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 04:57:08 +0800 Subject: [PATCH 08/87] =?UTF-8?q?test:=20=E8=BE=93=E5=87=BA=20LiteLLM=20P1?= =?UTF-8?q?=20=E8=84=B1=E6=95=8F=E9=AA=8C=E8=AF=81=E6=91=98=E8=A6=81?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .gitignore | 1 + docker_litellm/demo/scripts/smoke-baseline.sh | 20 +++++++++++++++++++ 2 files changed, 21 insertions(+) diff --git a/.gitignore b/.gitignore index 0757bd3..92b961a 100644 --- a/.gitignore +++ b/.gitignore @@ -122,6 +122,7 @@ celerybeat.pid # Environments .env +docker_litellm/demo/artifacts/ .venv env/ venv/ diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh index 482c990..4040272 100755 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -10,6 +10,9 @@ export COMPOSE_PROJECT_NAME="${LITELLM_COMPOSE_PROJECT:-litellm-baseline}" MODE="single" SECURITY_CHECK=false REVOCATION_SLO_MS="${LITELLM_REVOCATION_SLO_MS:-30000}" +SUMMARY_FILE="${LITELLM_SMOKE_SUMMARY_FILE:-}" +block_elapsed_ms="" +delete_elapsed_ms="" usage() { echo "Usage: $0 [--mode single|ha] [--security-check]" >&2 @@ -293,6 +296,7 @@ wait_for_rejection() { now="$(now_ms)" elapsed=$((now - start_ms)) echo "PASS $phase propagation: primary=$primary_code peer=$peer_code elapsed_ms=$elapsed slo_ms=$REVOCATION_SLO_MS" + if [[ "$phase" == "block" ]]; then block_elapsed_ms="$elapsed"; else delete_elapsed_ms="$elapsed"; fi return 0 fi now="$(now_ms)" @@ -304,6 +308,21 @@ wait_for_rejection() { done } +write_summary() { + [[ -n "$SUMMARY_FILE" ]] || return 0 + mkdir -p "$(dirname "$SUMMARY_FILE")" + chmod 700 "$(dirname "$SUMMARY_FILE")" + jq -n \ + --arg commit "$(git -C "$DEMO_DIR/../.." rev-parse HEAD)" \ + --arg image_id "$(docker image inspect "${LITELLM_IMAGE:-}" --format '{{.Id}}' 2>/dev/null || true)" \ + --arg tested_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ + --arg mode "$MODE" --arg block_ms "$block_elapsed_ms" --arg delete_ms "$delete_elapsed_ms" \ + '{commit:$commit,image_id:$image_id,tested_at:$tested_at,mode:$mode,migration:"external run-migration.sh",chat:true,stream:true,tool:true,usage:true,block_elapsed_ms:($block_ms|tonumber?),delete_elapsed_ms:($delete_ms|tonumber?),cleanup:"completed",security_scan:"passed",content_redacted:true}' \ + > "$SUMMARY_FILE" + chmod 600 "$SUMMARY_FILE" + echo "PASS summary: $SUMMARY_FILE" +} + assert_spend() { local spend_file="$tmpdir/spend.json" request_count total_tokens i for i in $(seq 1 30); do @@ -448,4 +467,5 @@ fi request_admin POST /key/delete "$tmpdir/delete-key-cleanup.json" "$tmpdir/delete-response.json" wait_for_rejection "$delete_headers" delete +write_summary echo "PASS complete: user/credential/model/key cleanup, explicit GET models, chat/stream/tool, token-bearing spend, and independent block/delete propagation." From 24a7a5f42256ea8137a98189edff7561b9db0b4f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 05:04:13 +0800 Subject: [PATCH 09/87] =?UTF-8?q?test:=20=E9=AA=8C=E8=AF=81=20LiteLLM=20?= =?UTF-8?q?=E5=8F=8C=E5=89=AF=E6=9C=AC=E5=85=B1=E4=BA=AB=E9=99=90=E6=B5=81?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/demo/scripts/smoke-baseline.sh | 24 ++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh index 4040272..a201ccd 100755 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -146,10 +146,13 @@ model_name="" model_id="" block_key_created=false delete_key_created=false +rate_key_created=false block_key_file="$tmpdir/block.key" delete_key_file="$tmpdir/delete.key" +rate_key_file="$tmpdir/rate.key" block_headers="$tmpdir/block.headers" delete_headers="$tmpdir/delete.headers" +rate_headers="$tmpdir/rate.headers" request_admin() { local method="$1" path="$2" payload_file="$3" output_file="$4" @@ -175,6 +178,9 @@ cleanup() { if [[ "$delete_key_created" == true ]]; then cleanup_request_admin POST /key/delete "$tmpdir/delete-key-cleanup.json" fi + if [[ "$rate_key_created" == true ]]; then + cleanup_request_admin POST /key/delete "$tmpdir/rate-key-cleanup.json" + fi if [[ "$block_key_created" == true ]]; then cleanup_request_admin POST /key/delete "$tmpdir/block-key-cleanup.json" fi @@ -317,7 +323,7 @@ write_summary() { --arg image_id "$(docker image inspect "${LITELLM_IMAGE:-}" --format '{{.Id}}' 2>/dev/null || true)" \ --arg tested_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ --arg mode "$MODE" --arg block_ms "$block_elapsed_ms" --arg delete_ms "$delete_elapsed_ms" \ - '{commit:$commit,image_id:$image_id,tested_at:$tested_at,mode:$mode,migration:"external run-migration.sh",chat:true,stream:true,tool:true,usage:true,block_elapsed_ms:($block_ms|tonumber?),delete_elapsed_ms:($delete_ms|tonumber?),cleanup:"completed",security_scan:"passed",content_redacted:true}' \ + '{commit:$commit,image_id:$image_id,tested_at:$tested_at,mode:$mode,migration:"external run-migration.sh",chat:true,stream:true,tool:true,usage:true,shared_rpm_limit:($mode == "ha"),block_elapsed_ms:($block_ms|tonumber?),delete_elapsed_ms:($delete_ms|tonumber?),cleanup:"completed",security_scan:"passed",content_redacted:true}' \ > "$SUMMARY_FILE" chmod 600 "$SUMMARY_FILE" echo "PASS summary: $SUMMARY_FILE" @@ -448,6 +454,22 @@ if ! jq -e '.choices[0].message.tool_calls | type == "array" and length > 0' "$t fi assert_spend +if [[ "$MODE" == "ha" ]]; then + # One request reaches replica 1; the same key must be RPM-limited on replica 2. + write_private_value "$tmpdir/rate-key-alias" "p1-smoke-rate-$suffix" + make_key_payload "$tmpdir/rate-key-alias" "$tmpdir/rate-key-create.json" + jq '.rpm_limit = 1' "$tmpdir/rate-key-create.json" > "$tmpdir/rate-key-limited.json" + chmod 600 "$tmpdir/rate-key-limited.json" + request_admin POST /key/generate "$tmpdir/rate-key-limited.json" "$tmpdir/rate-key.json" + make_key_header "$tmpdir/rate-key.json" "$rate_key_file" "$rate_headers" + rate_key_created=true + make_key_action_payload delete "$rate_key_file" "$tmpdir/rate-key-cleanup.json" + request_data_post "$BASE_URL" "$rate_headers" /v1/chat/completions "$tmpdir/chat-request.json" "$tmpdir/rate-first.json" + rate_code="$(curl --silent --show-error --max-time 30 --request POST "$PEER_URL/v1/chat/completions" --header "@$rate_headers" --data-binary "@$tmpdir/chat-request.json" --output "$tmpdir/rate-second.json" --write-out '%{http_code}' || true)" + [[ "$rate_code" == "429" ]] || { echo "shared RPM limit was bypassed by peer: http=$rate_code" >&2; exit 1; } + echo "PASS HA shared RPM: peer rejected the second request with 429." +fi + make_key_action_payload block "$block_key_file" "$tmpdir/block-key.json" request_admin POST /key/block "$tmpdir/block-key.json" "$tmpdir/block-response.json" wait_for_rejection "$block_headers" block From 1b8f50f8fc7de4b69ce2d26d583e02aaa58532d8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 05:12:22 +0800 Subject: [PATCH 10/87] =?UTF-8?q?test:=20=E9=AA=8C=E8=AF=81=20LiteLLM=20Re?= =?UTF-8?q?dis=20=E4=B8=AD=E6=96=AD=E6=81=A2=E5=A4=8D?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/demo/scripts/smoke-baseline.sh | 28 +++++++++++++++++-- 1 file changed, 25 insertions(+), 3 deletions(-) diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh index a201ccd..c46000d 100755 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -13,6 +13,7 @@ REVOCATION_SLO_MS="${LITELLM_REVOCATION_SLO_MS:-30000}" SUMMARY_FILE="${LITELLM_SMOKE_SUMMARY_FILE:-}" block_elapsed_ms="" delete_elapsed_ms="" +redis_recovery_result="not_run" usage() { echo "Usage: $0 [--mode single|ha] [--security-check]" >&2 @@ -234,7 +235,7 @@ wait_ready() { assert_redis() { local service="$1" docker compose --env-file "$ENV_FILE" -f "$DEMO_DIR/docker-compose.litellm.yml" exec -T "$service" \ - python3 -c 'import redis; password=open("/run/secrets/redis_password", encoding="utf-8").read().strip(); assert redis.Redis(host="redis", port=6379, password=password).ping()' + python3 -c 'import redis; password=open("/run/secrets/redis_password", encoding="utf-8").read().strip(); assert redis.Redis(host="redis", port=6379, password=password, socket_connect_timeout=2, socket_timeout=2).ping()' } now_ms() { @@ -322,8 +323,8 @@ write_summary() { --arg commit "$(git -C "$DEMO_DIR/../.." rev-parse HEAD)" \ --arg image_id "$(docker image inspect "${LITELLM_IMAGE:-}" --format '{{.Id}}' 2>/dev/null || true)" \ --arg tested_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ - --arg mode "$MODE" --arg block_ms "$block_elapsed_ms" --arg delete_ms "$delete_elapsed_ms" \ - '{commit:$commit,image_id:$image_id,tested_at:$tested_at,mode:$mode,migration:"external run-migration.sh",chat:true,stream:true,tool:true,usage:true,shared_rpm_limit:($mode == "ha"),block_elapsed_ms:($block_ms|tonumber?),delete_elapsed_ms:($delete_ms|tonumber?),cleanup:"completed",security_scan:"passed",content_redacted:true}' \ + --arg mode "$MODE" --arg block_ms "$block_elapsed_ms" --arg delete_ms "$delete_elapsed_ms" --arg redis_recovery "$redis_recovery_result" \ + '{commit:$commit,image_id:$image_id,tested_at:$tested_at,mode:$mode,migration:"external run-migration.sh",chat:true,stream:true,tool:true,usage:true,shared_rpm_limit:($mode == "ha"),redis_recovery:$redis_recovery,block_elapsed_ms:($block_ms|tonumber?),delete_elapsed_ms:($delete_ms|tonumber?),cleanup:"completed",security_scan:"passed",content_redacted:true}' \ > "$SUMMARY_FILE" chmod 600 "$SUMMARY_FILE" echo "PASS summary: $SUMMARY_FILE" @@ -468,6 +469,27 @@ if [[ "$MODE" == "ha" ]]; then rate_code="$(curl --silent --show-error --max-time 30 --request POST "$PEER_URL/v1/chat/completions" --header "@$rate_headers" --data-binary "@$tmpdir/chat-request.json" --output "$tmpdir/rate-second.json" --write-out '%{http_code}' || true)" [[ "$rate_code" == "429" ]] || { echo "shared RPM limit was bypassed by peer: http=$rate_code" >&2; exit 1; } echo "PASS HA shared RPM: peer rejected the second request with 429." + + redis_container="$(docker compose --env-file "$ENV_FILE" -f "$DEMO_DIR/docker-compose.litellm.yml" ps -q redis)" + [[ -n "$redis_container" ]] || { echo "Redis container not found" >&2; exit 1; } + redis_network="$(docker inspect -f '{{range $name, $_ := .NetworkSettings.Networks}}{{$name}}{{end}}' "$redis_container")" + [[ -n "$redis_network" ]] || { echo "Redis network not found" >&2; exit 1; } + # Disconnecting the Redis endpoint keeps LiteLLM processes runnable, so the + # bounded probes can prove failure without docker exec freezing on a paused + # target container. + docker network disconnect "$redis_network" "$redis_container" + if assert_redis litellm-1 >/dev/null 2>&1 || assert_redis litellm-2 >/dev/null 2>&1; then + docker network connect --alias redis "$redis_network" "$redis_container" >/dev/null || true + echo "Redis interruption did not fail both bounded probes" >&2 + exit 1 + fi + docker network connect --alias redis "$redis_network" "$redis_container" + sleep $(( ${REDIS_CIRCUIT_BREAKER_RECOVERY_TIMEOUT:-5} + 1 )) + assert_redis litellm-1 + assert_redis litellm-2 + wait_model_access "$PEER_URL" "$block_headers" post-redis-recovery + redis_recovery_result="passed" + echo "PASS HA Redis recovery: both probes and peer key authorization recovered." fi make_key_action_payload block "$block_key_file" "$tmpdir/block-key.json" From 5a4139793a7844df878d968c18beddb5e97f92a8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 05:18:07 +0800 Subject: [PATCH 11/87] =?UTF-8?q?test:=20=E9=AA=8C=E8=AF=81=20LiteLLM=20?= =?UTF-8?q?=E5=A4=9A=E5=89=AF=E6=9C=AC=E6=A8=A1=E5=9E=8B=E4=B8=8E=E7=94=A8?= =?UTF-8?q?=E9=87=8F=E4=B8=80=E8=87=B4=E6=80=A7?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/demo/scripts/smoke-baseline.sh | 29 +++++++++++++++++-- 1 file changed, 26 insertions(+), 3 deletions(-) diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh index c46000d..c0b18fe 100755 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -14,6 +14,7 @@ SUMMARY_FILE="${LITELLM_SMOKE_SUMMARY_FILE:-}" block_elapsed_ms="" delete_elapsed_ms="" redis_recovery_result="not_run" +shared_spend_counter_result="not_run" usage() { echo "Usage: $0 [--mode single|ha] [--security-check]" >&2 @@ -323,15 +324,15 @@ write_summary() { --arg commit "$(git -C "$DEMO_DIR/../.." rev-parse HEAD)" \ --arg image_id "$(docker image inspect "${LITELLM_IMAGE:-}" --format '{{.Id}}' 2>/dev/null || true)" \ --arg tested_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ - --arg mode "$MODE" --arg block_ms "$block_elapsed_ms" --arg delete_ms "$delete_elapsed_ms" --arg redis_recovery "$redis_recovery_result" \ - '{commit:$commit,image_id:$image_id,tested_at:$tested_at,mode:$mode,migration:"external run-migration.sh",chat:true,stream:true,tool:true,usage:true,shared_rpm_limit:($mode == "ha"),redis_recovery:$redis_recovery,block_elapsed_ms:($block_ms|tonumber?),delete_elapsed_ms:($delete_ms|tonumber?),cleanup:"completed",security_scan:"passed",content_redacted:true}' \ + --arg mode "$MODE" --arg block_ms "$block_elapsed_ms" --arg delete_ms "$delete_elapsed_ms" --arg redis_recovery "$redis_recovery_result" --arg shared_spend_counter "$shared_spend_counter_result" \ + '{commit:$commit,image_id:$image_id,tested_at:$tested_at,mode:$mode,migration:"external run-migration.sh",chat:true,stream:true,tool:true,usage:true,shared_rpm_limit:($mode == "ha"),shared_spend_counter:$shared_spend_counter,redis_recovery:$redis_recovery,block_elapsed_ms:($block_ms|tonumber?),delete_elapsed_ms:($delete_ms|tonumber?),cleanup:"completed",security_scan:"passed",content_redacted:true}' \ > "$SUMMARY_FILE" chmod 600 "$SUMMARY_FILE" echo "PASS summary: $SUMMARY_FILE" } assert_spend() { - local spend_file="$tmpdir/spend.json" request_count total_tokens i + local spend_file="$tmpdir/spend.json" peer_spend_file="$tmpdir/peer-spend.json" request_count total_tokens peer_request_count peer_total_tokens i for i in $(seq 1 30); do if request_admin GET "/spend/logs?user_id=$test_user" "" "$spend_file" \ && jq -e \ @@ -341,6 +342,14 @@ assert_spend() { "$spend_file" >/dev/null; then request_count="$(jq --rawfile model "$tmpdir/model-name" --rawfile key_hash "$tmpdir/block-key-sha256" 'def logs: if type == "array" then . else (.data // []) end; [ logs[] | select((.model == ($model | rtrimstr("\n")) or .model_group == ($model | rtrimstr("\n"))) and .api_key == ($key_hash | rtrimstr("\n")) and ((.total_tokens // 0) > 0)) ] | length' "$spend_file")" total_tokens="$(jq --rawfile model "$tmpdir/model-name" --rawfile key_hash "$tmpdir/block-key-sha256" 'def logs: if type == "array" then . else (.data // []) end; [ logs[] | select((.model == ($model | rtrimstr("\n")) or .model_group == ($model | rtrimstr("\n"))) and .api_key == ($key_hash | rtrimstr("\n")) and ((.total_tokens // 0) > 0)) ] | map(.total_tokens // 0) | add' "$spend_file")" + if [[ "$MODE" == "ha" ]]; then + curl --silent --show-error --fail --max-time 30 --request GET "$PEER_URL/spend/logs?user_id=$test_user" --header "@$admin_headers" --output "$peer_spend_file" + peer_request_count="$(jq --rawfile model "$tmpdir/model-name" --rawfile key_hash "$tmpdir/block-key-sha256" 'def logs: if type == "array" then . else (.data // []) end; [ logs[] | select((.model == ($model | rtrimstr("\n")) or .model_group == ($model | rtrimstr("\n"))) and .api_key == ($key_hash | rtrimstr("\n")) and ((.total_tokens // 0) > 0)) ] | length' "$peer_spend_file")" + peer_total_tokens="$(jq --rawfile model "$tmpdir/model-name" --rawfile key_hash "$tmpdir/block-key-sha256" 'def logs: if type == "array" then . else (.data // []) end; [ logs[] | select((.model == ($model | rtrimstr("\n")) or .model_group == ($model | rtrimstr("\n"))) and .api_key == ($key_hash | rtrimstr("\n")) and ((.total_tokens // 0) > 0)) ] | map(.total_tokens // 0) | add' "$peer_spend_file")" + [[ "$request_count/$total_tokens" == "$peer_request_count/$peer_total_tokens" ]] || { echo "shared SpendLog mismatch: primary=$request_count/$total_tokens peer=$peer_request_count/$peer_total_tokens" >&2; return 1; } + shared_spend_counter_result="passed" + echo "PASS HA shared SpendLog: request_count=$request_count total_tokens=$total_tokens on both replicas." + fi echo "PASS spend: request_count=$request_count total_tokens=$total_tokens" return 0 fi @@ -441,6 +450,20 @@ chmod 600 "$tmpdir/chat-request.json" request_data_post "$BASE_URL" "$block_headers" /v1/chat/completions "$tmpdir/chat-request.json" "$tmpdir/chat.json" jq -e '.choices[0].message.content | type == "string"' "$tmpdir/chat.json" >/dev/null +if [[ "$MODE" == "ha" ]]; then + peer_completion_ready=false + for i in $(seq 1 30); do + if request_data_post "$PEER_URL" "$block_headers" /v1/chat/completions "$tmpdir/chat-request.json" "$tmpdir/peer-chat.json" \ + && jq -e '.choices[0].message.content | type == "string"' "$tmpdir/peer-chat.json" >/dev/null; then + peer_completion_ready=true + break + fi + sleep 1 + done + [[ "$peer_completion_ready" == true ]] || { echo "second replica did not load the newly created model for completion" >&2; exit 1; } + echo "PASS HA model propagation: second replica completed with the newly created model." +fi + jq '. + {stream: true}' "$tmpdir/chat-request.json" > "$tmpdir/stream-request.json" chmod 600 "$tmpdir/stream-request.json" request_data_post "$BASE_URL" "$block_headers" /v1/chat/completions "$tmpdir/stream-request.json" "$tmpdir/stream.txt" From d07b467ee8d03e328f9f2e148cb26ec185f29dfb Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 05:24:16 +0800 Subject: [PATCH 12/87] =?UTF-8?q?test:=20=E6=81=A2=E5=A4=8D=20Redis=20?= =?UTF-8?q?=E7=BD=91=E7=BB=9C=E4=B8=AD=E6=96=AD=E6=B8=85=E7=90=86=E4=BF=9D?= =?UTF-8?q?=E9=9A=9C?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/demo/scripts/smoke-baseline.sh | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh index c0b18fe..f9c5726 100755 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -15,6 +15,8 @@ block_elapsed_ms="" delete_elapsed_ms="" redis_recovery_result="not_run" shared_spend_counter_result="not_run" +redis_container="" +redis_network="" usage() { echo "Usage: $0 [--mode single|ha] [--security-check]" >&2 @@ -177,6 +179,9 @@ cleanup_request_admin() { cleanup() { local exit_code=$? set +e + if [[ -n "$redis_container" && -n "$redis_network" ]]; then + docker network connect --alias redis "$redis_network" "$redis_container" >/dev/null 2>&1 || true + fi if [[ "$delete_key_created" == true ]]; then cleanup_request_admin POST /key/delete "$tmpdir/delete-key-cleanup.json" fi From 0cd389b48a00bfc98d4b4c36e1d8a2e7d7c7da49 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 05:28:59 +0800 Subject: [PATCH 13/87] =?UTF-8?q?test:=20=E8=AE=B0=E5=BD=95=20LiteLLM=20sm?= =?UTF-8?q?oke=20=E5=A4=B1=E8=B4=A5=E9=98=B6=E6=AE=B5?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/demo/scripts/smoke-baseline.sh | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh index f9c5726..43eb8f6 100755 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -17,6 +17,8 @@ redis_recovery_result="not_run" shared_spend_counter_result="not_run" redis_container="" redis_network="" +smoke_phase="initializing" +smoke_exit_code="" usage() { echo "Usage: $0 [--mode single|ha] [--security-check]" >&2 @@ -200,6 +202,8 @@ cleanup() { if [[ -n "$test_user" ]]; then cleanup_request_admin POST /user/delete "$tmpdir/user-delete.json" fi + smoke_exit_code="$exit_code" + write_summary || true unset master_key_file upstream_key_file upstream_base_file upstream_model_file rm -rf "$tmpdir" [[ ! -e "$tmpdir" ]] || { echo "temporary smoke directory cleanup failed" >&2; exit 1; } @@ -329,8 +333,8 @@ write_summary() { --arg commit "$(git -C "$DEMO_DIR/../.." rev-parse HEAD)" \ --arg image_id "$(docker image inspect "${LITELLM_IMAGE:-}" --format '{{.Id}}' 2>/dev/null || true)" \ --arg tested_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ - --arg mode "$MODE" --arg block_ms "$block_elapsed_ms" --arg delete_ms "$delete_elapsed_ms" --arg redis_recovery "$redis_recovery_result" --arg shared_spend_counter "$shared_spend_counter_result" \ - '{commit:$commit,image_id:$image_id,tested_at:$tested_at,mode:$mode,migration:"external run-migration.sh",chat:true,stream:true,tool:true,usage:true,shared_rpm_limit:($mode == "ha"),shared_spend_counter:$shared_spend_counter,redis_recovery:$redis_recovery,block_elapsed_ms:($block_ms|tonumber?),delete_elapsed_ms:($delete_ms|tonumber?),cleanup:"completed",security_scan:"passed",content_redacted:true}' \ + --arg mode "$MODE" --arg block_ms "$block_elapsed_ms" --arg delete_ms "$delete_elapsed_ms" --arg redis_recovery "$redis_recovery_result" --arg shared_spend_counter "$shared_spend_counter_result" --arg phase "$smoke_phase" --arg exit_code "$smoke_exit_code" \ + '{commit:$commit,image_id:$image_id,tested_at:$tested_at,mode:$mode,result:(if $exit_code == "0" then "passed" else "failed" end),phase:$phase,migration:"external run-migration.sh",chat:true,stream:true,tool:true,usage:true,shared_rpm_limit:($mode == "ha"),shared_spend_counter:$shared_spend_counter,redis_recovery:$redis_recovery,block_elapsed_ms:($block_ms|tonumber?),delete_elapsed_ms:($delete_ms|tonumber?),cleanup:"completed",security_scan:"passed",content_redacted:true}' \ > "$SUMMARY_FILE" chmod 600 "$SUMMARY_FILE" echo "PASS summary: $SUMMARY_FILE" @@ -482,6 +486,7 @@ if ! jq -e '.choices[0].message.tool_calls | type == "array" and length > 0' "$t exit 1 fi assert_spend +smoke_phase="shared_limit_and_redis_recovery" if [[ "$MODE" == "ha" ]]; then # One request reaches replica 1; the same key must be RPM-limited on replica 2. @@ -521,6 +526,7 @@ if [[ "$MODE" == "ha" ]]; then fi make_key_action_payload block "$block_key_file" "$tmpdir/block-key.json" +smoke_phase="revocation" request_admin POST /key/block "$tmpdir/block-key.json" "$tmpdir/block-response.json" wait_for_rejection "$block_headers" block @@ -539,5 +545,7 @@ fi request_admin POST /key/delete "$tmpdir/delete-key-cleanup.json" "$tmpdir/delete-response.json" wait_for_rejection "$delete_headers" delete +smoke_phase="completed" +smoke_exit_code="0" write_summary echo "PASS complete: user/credential/model/key cleanup, explicit GET models, chat/stream/tool, token-bearing spend, and independent block/delete propagation." From fc0e3dc014cc3bbfc62745b267fb81b6540fc234 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 05:33:45 +0800 Subject: [PATCH 14/87] =?UTF-8?q?test:=20=E6=8B=86=E5=88=86=20Redis=20?= =?UTF-8?q?=E6=81=A2=E5=A4=8D=E9=AA=8C=E6=94=B6=E8=84=9A=E6=9C=AC?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../demo/scripts/smoke-redis-recovery.sh | 51 +++++++++++++++++++ 1 file changed, 51 insertions(+) create mode 100755 docker_litellm/demo/scripts/smoke-redis-recovery.sh diff --git a/docker_litellm/demo/scripts/smoke-redis-recovery.sh b/docker_litellm/demo/scripts/smoke-redis-recovery.sh new file mode 100755 index 0000000..6092e40 --- /dev/null +++ b/docker_litellm/demo/scripts/smoke-redis-recovery.sh @@ -0,0 +1,51 @@ +#!/usr/bin/env bash +# Isolated Redis outage/recovery proof for an already-running HA stack. +set -euo pipefail + +script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +demo_dir="$(cd "${script_dir}/.." && pwd)" +env_file="${LITELLM_SMOKE_ENV_FILE:-${demo_dir}/.env}" +summary_file="${LITELLM_REDIS_SUMMARY_FILE:-${demo_dir}/artifacts/p1-redis-recovery.json}" +recovery_timeout="${REDIS_CIRCUIT_BREAKER_RECOVERY_TIMEOUT:-5}" +container="" +network="" +phase="initializing" +result="failed" + +cleanup() { + set +e + if [[ -n "$container" && -n "$network" ]]; then + docker network connect --alias redis "$network" "$container" >/dev/null 2>&1 || true + fi + mkdir -p "$(dirname "$summary_file")" + chmod 700 "$(dirname "$summary_file")" + jq -n --arg commit "$(git -C "$demo_dir/../.." rev-parse HEAD)" --arg tested_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" --arg result "$result" --arg phase "$phase" \ + '{commit:$commit,tested_at:$tested_at,redis_recovery:$result,phase:$phase,credentials_or_content:false}' > "$summary_file" + chmod 600 "$summary_file" +} +trap cleanup EXIT + +probe() { + docker exec "$1" python3 -c 'import redis; p=open("/run/secrets/redis_password").read().strip(); assert redis.Redis(host="redis", password=p, socket_connect_timeout=2, socket_timeout=2).ping()' +} + +container="$(docker compose --env-file "$env_file" -f "$demo_dir/docker-compose.litellm.yml" ps -q redis)" +[[ -n "$container" ]] || { phase="redis_not_found"; exit 1; } +network="$(docker inspect -f '{{range $name, $_ := .NetworkSettings.Networks}}{{$name}}{{end}}' "$container")" +[[ -n "$network" ]] || { phase="network_not_found"; exit 1; } + +phase="disconnect" +docker network disconnect "$network" "$container" +if probe svc-litellm-1 >/dev/null 2>&1 || probe svc-litellm-2 >/dev/null 2>&1; then + phase="probe_unexpectedly_succeeded" + exit 1 +fi + +phase="recover" +docker network connect --alias redis "$network" "$container" +sleep $((recovery_timeout + 1)) +probe svc-litellm-1 +probe svc-litellm-2 +result="passed" +phase="completed" +echo "PASS Redis recovery: both bounded probes failed during outage and recovered afterward." From dffb3b24d4b8543bf17c9348b7789f26de95cbfb Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 11:38:44 +0800 Subject: [PATCH 15/87] =?UTF-8?q?docs:=20=E6=B1=87=E6=80=BB=20LiteLLM=20P1?= =?UTF-8?q?=20=E9=AA=8C=E8=AF=81=E8=AF=81=E6=8D=AE?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/README.md | 4 ++++ .../scripts/aggregate-verification-summary.sh | 24 +++++++++++++++++++ 2 files changed, 28 insertions(+) create mode 100755 docker_litellm/demo/scripts/aggregate-verification-summary.sh diff --git a/docker_litellm/README.md b/docker_litellm/README.md index 3f9747c..99cdaab 100644 --- a/docker_litellm/README.md +++ b/docker_litellm/README.md @@ -88,11 +88,15 @@ P1 不把 LiteLLM User/Team 当作产品用户或 Workspace 的事实源;Shell cd docker_litellm/demo ./scripts/smoke-baseline.sh --mode single ./scripts/smoke-baseline.sh --mode ha +./scripts/smoke-redis-recovery.sh +./scripts/aggregate-verification-summary.sh ./scripts/smoke-baseline.sh --security-check ``` `--security-check` 不读取 `.env`、不启动服务也不发送上游请求;它拒绝 inline header、secret-bearing `jq --arg`、`set -x`、Compose 上游凭据注入和 Redis 密码命令行展开,并检查 Docker Secret、0600 临时文件与退出清理约束。 +`smoke-redis-recovery.sh` 在已启动的 HA 栈中临时断开 Redis 网络端点,验证两个副本的有界认证探针均失败,再恢复 `redis` alias、等待 breaker 窗口并确认探针恢复。它有独立的恢复 trap,不会让故障测试影响主 smoke 的资源清理。`aggregate-verification-summary.sh` 将主 HA 与 Redis 独立报告组合为不含密钥、密码、提示词和响应正文的最终 JSON 摘要。 + 脚本在真实上游变量存在时执行:创建测试用户、保存测试上游凭证、以 `litellm_credential_name` 创建模型、生成受限虚拟 key、显式 `GET /v1/models`、chat、stream、tool call、token-bearing usage 查询、block,以及独立 key 的 delete。DeepSeek V4 使用 `deepseek/` 的原生 provider,避免被通用 OpenAI provider 丢弃 `thinking` 参数。HA 模式会先证明第二副本接受 key,再轮询两个副本直到都拒绝,并输出实际传播时间与 SLO。 `LITELLM_MASTER_KEY`、上游 API key 与虚拟 key 不会作为 `curl`、`jq` 或其他子进程的命令参数传递。脚本以 `umask 077` 创建工作目录,所有 header、请求、响应和 key 文件均为 `0600`,退出时删除;创建的 user、credential、model 和两个测试 key 也会清理。上游凭据仅由 smoke 客户端读取,不会注入 LiteLLM Compose 容器。 diff --git a/docker_litellm/demo/scripts/aggregate-verification-summary.sh b/docker_litellm/demo/scripts/aggregate-verification-summary.sh new file mode 100755 index 0000000..b872f96 --- /dev/null +++ b/docker_litellm/demo/scripts/aggregate-verification-summary.sh @@ -0,0 +1,24 @@ +#!/usr/bin/env bash +# Build a redacted final P1 summary from independently generated smoke reports. +set -euo pipefail + +script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +demo_dir="$(cd "${script_dir}/.." && pwd)" +artifacts_dir="${demo_dir}/artifacts" +main_report="${artifacts_dir}/p1-ha-summary.json" +redis_report="${artifacts_dir}/p1-redis-recovery.json" +output="${LITELLM_AGGREGATE_SUMMARY_FILE:-${artifacts_dir}/p1-final-summary.json}" + +[[ -f "$main_report" && -f "$redis_report" ]] || { echo "missing smoke summary input" >&2; exit 2; } +umask 077 +mkdir -p "$(dirname "$output")" +chmod 700 "$(dirname "$output")" +jq -n \ + --arg commit "$(git -C "$demo_dir/../.." rev-parse HEAD)" \ + --arg image_id "$(jq -r '.image_id' "$main_report")" \ + --arg generated_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ + --slurpfile main "$main_report" --slurpfile redis "$redis_report" \ + '{commit:$commit,image_id:$image_id,generated_at:$generated_at,single:"passed in prior real smoke",ha:$main[0],redis_recovery:$redis[0],idempotency:{native_api:"no verified Idempotency-Key contract",shell_follow_up:"stable request ID plus operation ledger and lookup recovery"},secrets_or_content:false}' \ + > "$output" +chmod 600 "$output" +echo "PASS aggregate summary: $output" From a62d09538df4cb354b69f7eef60caec21197b446 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 13:07:08 +0800 Subject: [PATCH 16/87] =?UTF-8?q?fix:=20=E4=BF=AE=E5=A4=8D=20LiteLLM=20P1?= =?UTF-8?q?=20=E5=8F=AF=E5=A4=8D=E7=8E=B0=E9=AA=8C=E6=94=B6=E6=8A=A5?= =?UTF-8?q?=E5=91=8A?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/README.md | 17 +- .../demo/docker-compose.litellm.yml | 7 +- .../scripts/aggregate-verification-summary.sh | 61 +++++- docker_litellm/demo/scripts/run-migration.sh | 41 +++- docker_litellm/demo/scripts/smoke-baseline.sh | 204 +++++++++++++----- .../demo/scripts/smoke-redis-recovery.sh | 73 ++++++- tool.sh | 10 +- 7 files changed, 334 insertions(+), 79 deletions(-) mode change 100755 => 100644 docker_litellm/demo/scripts/aggregate-verification-summary.sh mode change 100755 => 100644 docker_litellm/demo/scripts/smoke-redis-recovery.sh diff --git a/docker_litellm/README.md b/docker_litellm/README.md index 99cdaab..9f911ae 100644 --- a/docker_litellm/README.md +++ b/docker_litellm/README.md @@ -51,13 +51,18 @@ docker compose --env-file .env -f docker-compose.litellm.yml --profile single up ```bash ./scripts/run-migration.sh -docker compose --env-file .env -f docker-compose.litellm.yml --profile single up -d litellm-1 +docker compose --env-file .env -f docker-compose.litellm.yml --profile single up -d --wait +./scripts/smoke-baseline.sh --mode single ``` 双副本测试使用同一 PostgreSQL 与 Redis,但有两个 HTTP 入口: ```bash -docker compose --env-file .env -f docker-compose.litellm.yml --profile ha up -d +./scripts/run-migration.sh +docker compose --env-file .env -f docker-compose.litellm.yml --profile ha up -d --wait +./scripts/smoke-baseline.sh --mode ha +./scripts/smoke-redis-recovery.sh +./scripts/aggregate-verification-summary.sh ``` HA 启动前同样先运行 `./scripts/run-migration.sh`,再执行上述命令。不要把 migration profile 与代理 profile 放入同一条 `up` 命令。 @@ -93,15 +98,17 @@ cd docker_litellm/demo ./scripts/smoke-baseline.sh --security-check ``` +在全新 checkout 中按以下顺序执行时,脚本自己生成而非复用历史文件:`p1-migration-summary.json`、`p1-single-summary.json`、`p1-ha-summary.json`、`p1-redis-recovery.json` 与最终 `p1-final-summary.json`(均在被忽略的 `artifacts/`)。聚合脚本只接受当前 `HEAD`、相同 image ID、`result=passed`、`phase=completed` 且脱敏的四份输入;任何缺失、失败、跳过或模式不符都会被拒绝。 + `--security-check` 不读取 `.env`、不启动服务也不发送上游请求;它拒绝 inline header、secret-bearing `jq --arg`、`set -x`、Compose 上游凭据注入和 Redis 密码命令行展开,并检查 Docker Secret、0600 临时文件与退出清理约束。 -`smoke-redis-recovery.sh` 在已启动的 HA 栈中临时断开 Redis 网络端点,验证两个副本的有界认证探针均失败,再恢复 `redis` alias、等待 breaker 窗口并确认探针恢复。它有独立的恢复 trap,不会让故障测试影响主 smoke 的资源清理。`aggregate-verification-summary.sh` 将主 HA 与 Redis 独立报告组合为不含密钥、密码、提示词和响应正文的最终 JSON 摘要。 +`smoke-redis-recovery.sh` 在已启动的 HA 栈中临时断开 Redis 网络端点,验证两个副本的有界认证探针均失败,再恢复 `redis` alias、等待 breaker 窗口并确认认证后的 `GET /v1/models` 恢复。它有独立的恢复 trap,不会让故障测试影响主 smoke 的资源清理。`aggregate-verification-summary.sh` 将 migration、single、HA 与 Redis 独立报告组合为不含密钥、密码、提示词和响应正文的最终 JSON 摘要。 -脚本在真实上游变量存在时执行:创建测试用户、保存测试上游凭证、以 `litellm_credential_name` 创建模型、生成受限虚拟 key、显式 `GET /v1/models`、chat、stream、tool call、token-bearing usage 查询、block,以及独立 key 的 delete。DeepSeek V4 使用 `deepseek/` 的原生 provider,避免被通用 OpenAI provider 丢弃 `thinking` 参数。HA 模式会先证明第二副本接受 key,再轮询两个副本直到都拒绝,并输出实际传播时间与 SLO。 +脚本在真实上游变量存在时执行:创建测试用户、保存测试上游凭证、以 `litellm_credential_name` 创建模型、由调用方生成稳定高熵 virtual key 并故意丢弃首次创建响应,再用该 key 的 0600 Authorization header 调用 `/key/info` 恢复、验证相同 key 重试被拒绝而不会创建第二资源;随后显式 `GET /v1/models`、chat、stream、tool call、token-bearing usage 查询、block,以及独立 key 的 delete。DeepSeek V4 使用 `deepseek/` 的原生 provider,避免被通用 OpenAI provider 丢弃 `thinking` 参数。HA 模式会先证明第二副本接受 key,再验证跨副本 RPM 和 post-spend budget 限制均返回 `429`,最后轮询两个副本直到都拒绝,并输出实际传播时间与 SLO。 `LITELLM_MASTER_KEY`、上游 API key 与虚拟 key 不会作为 `curl`、`jq` 或其他子进程的命令参数传递。脚本以 `umask 077` 创建工作目录,所有 header、请求、响应和 key 文件均为 `0600`,退出时删除;创建的 user、credential、model 和两个测试 key 也会清理。上游凭据仅由 smoke 客户端读取,不会注入 LiteLLM Compose 容器。 -若未设置上游变量,脚本仍验证 LiteLLM readiness、PostgreSQL 连接、从每个 LiteLLM 副本到 Redis 的认证连通性、管理面认证和 user 清理路径,并以明确的 `PENDING upstream smoke` 退出成功。它不会伪造 chat、stream、tool、usage、block/delete 或撤销传播已通过。 +若未设置上游变量,脚本仍验证 LiteLLM readiness、PostgreSQL 连接、从每个 LiteLLM 副本到 Redis 的认证连通性、migration 证据和 user 清理路径,并以明确的 `result=skipped` / `phase=pending_upstream` 报告退出。它不会伪造 chat、stream、tool、usage、block/delete 或撤销传播已通过,最终聚合也会拒绝该报告。 ## Readiness 与 Redis 结论 diff --git a/docker_litellm/demo/docker-compose.litellm.yml b/docker_litellm/demo/docker-compose.litellm.yml index a050ee0..5a56663 100644 --- a/docker_litellm/demo/docker-compose.litellm.yml +++ b/docker_litellm/demo/docker-compose.litellm.yml @@ -35,9 +35,10 @@ x-litellm-common: &litellm-common start_period: 30s services: - # Run LiteLLM's own migration-only mode once before proxy replicas start. - # `service_completed_successfully` serializes migration across single/HA - # Compose starts; Prisma's migration table makes repeated jobs idempotent. + # Run LiteLLM's own migration-only mode explicitly before proxy replicas. + # Replicas intentionally do not depend on this profile service: the deploy + # workflow calls scripts/run-migration.sh after postgres/redis are healthy. + # Prisma's migration table makes repeated one-shot jobs idempotent. litellm-migrate: <<: *litellm-common container_name: ${LITELLM_MIGRATE_CONTAINER_NAME:-svc-litellm-migrate} diff --git a/docker_litellm/demo/scripts/aggregate-verification-summary.sh b/docker_litellm/demo/scripts/aggregate-verification-summary.sh old mode 100755 new mode 100644 index b872f96..7d890d8 --- a/docker_litellm/demo/scripts/aggregate-verification-summary.sh +++ b/docker_litellm/demo/scripts/aggregate-verification-summary.sh @@ -1,24 +1,71 @@ #!/usr/bin/env bash -# Build a redacted final P1 summary from independently generated smoke reports. +# Aggregate only reports made by the current checkout; never infer a result +# from a previous run or from missing inputs. set -euo pipefail script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" demo_dir="$(cd "${script_dir}/.." && pwd)" artifacts_dir="${demo_dir}/artifacts" -main_report="${artifacts_dir}/p1-ha-summary.json" +single_report="${artifacts_dir}/p1-single-summary.json" +ha_report="${artifacts_dir}/p1-ha-summary.json" redis_report="${artifacts_dir}/p1-redis-recovery.json" +migration_report="${artifacts_dir}/p1-migration-summary.json" output="${LITELLM_AGGREGATE_SUMMARY_FILE:-${artifacts_dir}/p1-final-summary.json}" +commit="$(git -C "$demo_dir/../.." rev-parse HEAD)" + +for report in "$single_report" "$ha_report" "$redis_report" "$migration_report"; do + [[ -f "$report" ]] || { echo "missing required report: $report" >&2; exit 2; } +done + +# Each input must be an independently successful and fully redacted result of +# this exact checkout. jq -e performs the gate before the final report exists. +jq -e --arg commit "$commit" ' + .mode == "migration" and .result == "passed" and .phase == "completed" and + .commit == $commit and (.image_id | type == "string" and length > 0) and + .content_redacted == true and .proxy_replicas_started == false +' "$migration_report" >/dev/null + +jq -e --arg commit "$commit" ' + .mode == "single" and .result == "passed" and .phase == "completed" and + .commit == $commit and (.image_id | type == "string" and length > 0) and + .content_redacted == true and .migration == "passed" and + .chat == "passed" and .stream == "passed" and .tool == "passed" and + .usage == "passed" and .block == "passed" and .delete == "passed" and + .cleanup == "passed" and .security_scan == "passed" +' "$single_report" >/dev/null + +jq -e --arg commit "$commit" ' + .mode == "ha" and .result == "passed" and .phase == "completed" and + .commit == $commit and (.image_id | type == "string" and length > 0) and + .content_redacted == true and .migration == "passed" and + .chat == "passed" and .stream == "passed" and .tool == "passed" and + .usage == "passed" and .block == "passed" and .delete == "passed" and + .shared_rpm_limit == "passed" and .shared_enforcement == "passed" and + .shared_spend_counter == "passed" and .idempotency_recovery == "passed" and + .cleanup == "passed" and .security_scan == "passed" +' "$ha_report" >/dev/null + +jq -e --arg commit "$commit" ' + .mode == "ha" and .result == "passed" and .phase == "completed" and + .commit == $commit and (.image_id | type == "string" and length > 0) and + .redis_recovery == "passed" and .content_redacted == true and + .security_scan == "passed" +' "$redis_report" >/dev/null + +image_id="$(jq -r '.image_id' "$migration_report")" +[[ "$image_id" == "$(jq -r '.image_id' "$single_report")" ]] || { echo "single image ID differs" >&2; exit 1; } +[[ "$image_id" == "$(jq -r '.image_id' "$ha_report")" ]] || { echo "HA image ID differs" >&2; exit 1; } +[[ "$image_id" == "$(jq -r '.image_id' "$redis_report")" ]] || { echo "Redis report image ID differs" >&2; exit 1; } -[[ -f "$main_report" && -f "$redis_report" ]] || { echo "missing smoke summary input" >&2; exit 2; } umask 077 mkdir -p "$(dirname "$output")" chmod 700 "$(dirname "$output")" jq -n \ - --arg commit "$(git -C "$demo_dir/../.." rev-parse HEAD)" \ - --arg image_id "$(jq -r '.image_id' "$main_report")" \ + --arg commit "$commit" --arg image_id "$image_id" \ --arg generated_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ - --slurpfile main "$main_report" --slurpfile redis "$redis_report" \ - '{commit:$commit,image_id:$image_id,generated_at:$generated_at,single:"passed in prior real smoke",ha:$main[0],redis_recovery:$redis[0],idempotency:{native_api:"no verified Idempotency-Key contract",shell_follow_up:"stable request ID plus operation ledger and lookup recovery"},secrets_or_content:false}' \ + --slurpfile migration "$migration_report" --slurpfile single "$single_report" \ + --slurpfile ha "$ha_report" --slurpfile redis "$redis_report" \ + '{commit:$commit,image_id:$image_id,generated_at:$generated_at,result:"passed",phase:"completed",migration:$migration[0],single:$single[0],ha:$ha[0],redis_recovery:$redis[0],content_redacted:true,local_only:true}' \ > "$output" chmod 600 "$output" echo "PASS aggregate summary: $output" diff --git a/docker_litellm/demo/scripts/run-migration.sh b/docker_litellm/demo/scripts/run-migration.sh index 68917d5..b24f8f5 100755 --- a/docker_litellm/demo/scripts/run-migration.sh +++ b/docker_litellm/demo/scripts/run-migration.sh @@ -7,8 +7,45 @@ script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" demo_dir="$(cd "${script_dir}/.." && pwd)" env_file="${LITELLM_SMOKE_ENV_FILE:-${demo_dir}/.env}" compose=(docker compose --env-file "$env_file" -f "${demo_dir}/docker-compose.litellm.yml") +summary_file="${LITELLM_MIGRATION_SUMMARY_FILE:-${demo_dir}/artifacts/p1-migration-summary.json}" [[ -f "$env_file" ]] || { echo "missing ignored local environment file" >&2; exit 2; } -"${compose[@]}" up -d postgres redis + +# Deliberately load only the non-secret image reference into this shell. The +# Compose invocation receives the ignored env file itself; no value is echoed. +# shellcheck disable=SC1090 +source "$env_file" +: "${LITELLM_IMAGE:?missing LITELLM_IMAGE in ignored local environment file}" +image_ref="$LITELLM_IMAGE" +export -n LITELLM_IMAGE LITELLM_MASTER_KEY POSTGRES_PASSWORD REDIS_PASSWORD \ + UPSTREAM_API_KEY UPSTREAM_BASE_URL UPSTREAM_MODEL 2>/dev/null || true +unset LITELLM_MASTER_KEY POSTGRES_PASSWORD REDIS_PASSWORD UPSTREAM_API_KEY \ + UPSTREAM_BASE_URL UPSTREAM_MODEL + +result="failed" +phase="initializing" +cleanup() { + local exit_code=$? + umask 077 + mkdir -p "$(dirname "$summary_file")" + chmod 700 "$(dirname "$summary_file")" + jq -n \ + --arg commit "$(git -C "$demo_dir/../.." rev-parse HEAD)" \ + --arg image_id "$(docker image inspect "$image_ref" --format '{{.Id}}' 2>/dev/null || true)" \ + --arg tested_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ + --arg result "$result" --arg phase "$phase" --argjson exit_code "$exit_code" \ + '{commit:$commit,image_id:$image_id,tested_at:$tested_at,mode:"migration",result:$result,phase:$phase,exit_code:$exit_code,proxy_replicas_started:false,content_redacted:true}' \ + > "$summary_file" + chmod 600 "$summary_file" +} +trap cleanup EXIT + +# A cold Compose start previously raced PostgreSQL/Redis readiness. `--wait` +# makes the dependency condition explicit before the one-shot job is run. +phase="waiting_dependencies" +"${compose[@]}" up -d --wait postgres redis +phase="migration_job" "${compose[@]}" --profile migrate run --rm --no-deps litellm-migrate -echo "PASS migration: LiteLLM migration-only job completed; proxy replicas were not started." +phase="completed" +result="passed" +echo "PASS migration: dependencies healthy; migration-only job completed; proxy replicas were not started." diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh index 43eb8f6..98aab16 100755 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -13,8 +13,20 @@ REVOCATION_SLO_MS="${LITELLM_REVOCATION_SLO_MS:-30000}" SUMMARY_FILE="${LITELLM_SMOKE_SUMMARY_FILE:-}" block_elapsed_ms="" delete_elapsed_ms="" -redis_recovery_result="not_run" -shared_spend_counter_result="not_run" +result="failed" +chat_result="not_run" +stream_result="not_run" +tool_result="not_run" +usage_result="not_run" +block_result="not_run" +delete_result="not_run" +shared_rpm_limit_result="not_applicable" +shared_enforcement_result="not_applicable" +shared_spend_counter_result="not_applicable" +idempotency_recovery_result="not_applicable" +migration_result="not_run" +security_scan_result="not_run" +cleanup_result="not_run" redis_container="" redis_network="" smoke_phase="initializing" @@ -33,6 +45,10 @@ while (($#)); do esac done +if [[ -z "$SUMMARY_FILE" ]]; then + SUMMARY_FILE="$DEMO_DIR/artifacts/p1-${MODE}-summary.json" +fi + need() { command -v "$1" >/dev/null || { echo "required command missing: $1" >&2; exit 2; }; } need curl; need jq; need rg [[ "$MODE" == "single" || "$MODE" == "ha" ]] || { usage; exit 2; } @@ -76,16 +92,24 @@ if [[ "$SECURITY_CHECK" == true ]]; then exit 0 fi +# Run the static negative checks in every real smoke too. A successful report +# cannot claim a security result that was not actually executed. +security_check +security_scan_result="static_passed" + [[ -f "$ENV_FILE" ]] || { echo "missing local environment file: $ENV_FILE (copy .env.example)" >&2; exit 2; } # Do not use `set -a`: sourced values must not leak to child processes. # shellcheck disable=SC1090 source "$ENV_FILE" : "${LITELLM_MASTER_KEY:?missing LITELLM_MASTER_KEY in local environment file}" +: "${LITELLM_IMAGE:?missing LITELLM_IMAGE in local environment file}" +image_ref="$LITELLM_IMAGE" +upstream_provider="${UPSTREAM_PROVIDER:-}" # An `.env` may use `export NAME=...`; remove that export attribute before # mktemp, chmod, tr, curl, jq, or any other child process is started. -export -n LITELLM_MASTER_KEY UPSTREAM_API_KEY UPSTREAM_BASE_URL UPSTREAM_MODEL \ +export -n LITELLM_IMAGE LITELLM_MASTER_KEY UPSTREAM_API_KEY UPSTREAM_BASE_URL UPSTREAM_MODEL UPSTREAM_PROVIDER \ POSTGRES_PASSWORD REDIS_PASSWORD DATABASE_URL 2>/dev/null || true if [[ "$MODE" == "single" ]]; then @@ -143,7 +167,7 @@ assert_private_file "$upstream_base_file" assert_private_file "$upstream_model_file" # No child process needs these values. Compose reads its own --env-file. -unset LITELLM_MASTER_KEY UPSTREAM_API_KEY UPSTREAM_BASE_URL UPSTREAM_MODEL \ +unset LITELLM_MASTER_KEY UPSTREAM_API_KEY UPSTREAM_BASE_URL UPSTREAM_MODEL UPSTREAM_PROVIDER \ POSTGRES_PASSWORD REDIS_PASSWORD DATABASE_URL test_user="" @@ -153,12 +177,15 @@ model_id="" block_key_created=false delete_key_created=false rate_key_created=false +enforcement_key_created=false block_key_file="$tmpdir/block.key" delete_key_file="$tmpdir/delete.key" rate_key_file="$tmpdir/rate.key" block_headers="$tmpdir/block.headers" delete_headers="$tmpdir/delete.headers" rate_headers="$tmpdir/rate.headers" +enforcement_key_file="$tmpdir/enforcement.key" +enforcement_headers="$tmpdir/enforcement.headers" request_admin() { local method="$1" path="$2" payload_file="$3" output_file="$4" @@ -175,38 +202,48 @@ cleanup_request_admin() { if [[ -n "$payload_file" ]]; then curl_args+=(--data-binary "@$payload_file") fi - curl "${curl_args[@]}" >/dev/null 2>&1 || true + curl "${curl_args[@]}" >/dev/null 2>&1 } cleanup() { local exit_code=$? + local cleanup_ok=true set +e if [[ -n "$redis_container" && -n "$redis_network" ]]; then docker network connect --alias redis "$redis_network" "$redis_container" >/dev/null 2>&1 || true fi if [[ "$delete_key_created" == true ]]; then - cleanup_request_admin POST /key/delete "$tmpdir/delete-key-cleanup.json" + cleanup_request_admin POST /key/delete "$tmpdir/delete-key-cleanup.json" || cleanup_ok=false fi if [[ "$rate_key_created" == true ]]; then - cleanup_request_admin POST /key/delete "$tmpdir/rate-key-cleanup.json" + cleanup_request_admin POST /key/delete "$tmpdir/rate-key-cleanup.json" || cleanup_ok=false + fi + if [[ "$enforcement_key_created" == true ]]; then + cleanup_request_admin POST /key/delete "$tmpdir/enforcement-key-cleanup.json" || cleanup_ok=false fi if [[ "$block_key_created" == true ]]; then - cleanup_request_admin POST /key/delete "$tmpdir/block-key-cleanup.json" + cleanup_request_admin POST /key/delete "$tmpdir/block-key-cleanup.json" || cleanup_ok=false fi if [[ -n "$model_id" ]]; then - cleanup_request_admin POST /model/delete "$tmpdir/model-delete.json" + cleanup_request_admin POST /model/delete "$tmpdir/model-delete.json" || cleanup_ok=false fi if [[ -n "$credential_name" ]]; then - cleanup_request_admin DELETE "/credentials/$credential_name" "" + cleanup_request_admin DELETE "/credentials/$credential_name" "" || cleanup_ok=false fi if [[ -n "$test_user" ]]; then - cleanup_request_admin POST /user/delete "$tmpdir/user-delete.json" + cleanup_request_admin POST /user/delete "$tmpdir/user-delete.json" || cleanup_ok=false fi - smoke_exit_code="$exit_code" - write_summary || true + if [[ "$cleanup_ok" == true ]]; then cleanup_result="passed"; else cleanup_result="failed"; exit_code=1; fi unset master_key_file upstream_key_file upstream_base_file upstream_model_file rm -rf "$tmpdir" - [[ ! -e "$tmpdir" ]] || { echo "temporary smoke directory cleanup failed" >&2; exit 1; } + if [[ -e "$tmpdir" ]]; then cleanup_result="failed"; exit_code=1; fi + smoke_exit_code="$exit_code" + if [[ "$exit_code" == 0 && "$smoke_phase" == completed && "$cleanup_result" == passed && "$security_scan_result" == passed ]]; then + result="passed" + elif [[ "$result" != skipped ]]; then + result="failed" + fi + write_summary || true exit "$exit_code" } trap cleanup EXIT @@ -252,14 +289,51 @@ now_ms() { python3 -c 'import time; print(time.time_ns() // 1_000_000)' } +assert_migration_evidence() { + local report="$DEMO_DIR/artifacts/p1-migration-summary.json" commit image_id + commit="$(git -C "$DEMO_DIR/../.." rev-parse HEAD)" + image_id="$(docker image inspect "$image_ref" --format '{{.Id}}' 2>/dev/null || true)" + [[ -f "$report" ]] || { echo "missing current migration report: $report" >&2; return 1; } + jq -e --arg commit "$commit" --arg image_id "$image_id" ' + .mode == "migration" and .result == "passed" and .phase == "completed" and + .commit == $commit and .image_id == $image_id and .proxy_replicas_started == false and + .content_redacted == true + ' "$report" >/dev/null +} + +runtime_security_check() { + local logs_file="$tmpdir/litellm-logs.txt" inspect_file="$tmpdir/inspect.json" process_file="$tmpdir/redis-processes.txt" + docker inspect svc-litellm-1 > "$inspect_file" + if [[ "$MODE" == ha ]]; then docker inspect svc-litellm-2 >> "$inspect_file"; fi + docker compose --env-file "$ENV_FILE" -f "$DEMO_DIR/docker-compose.litellm.yml" logs --no-color litellm-1 > "$logs_file" 2>&1 + if [[ "$MODE" == ha ]]; then docker compose --env-file "$ENV_FILE" -f "$DEMO_DIR/docker-compose.litellm.yml" logs --no-color litellm-2 >> "$logs_file" 2>&1; fi + docker exec "$(docker compose --env-file "$ENV_FILE" -f "$DEMO_DIR/docker-compose.litellm.yml" ps -q redis)" ps -eo args > "$process_file" + ! rg -q 'UPSTREAM_(API_KEY|BASE_URL|MODEL)=' "$inspect_file" && + ! rg -q -- '--requirepass[[:space:]]+[^[:space:]]+' "$process_file" && + ! rg -q --file "$tmpdir/prompt-marker" "$logs_file" && + ! rg -q --file "$tmpdir/tool-marker" "$logs_file" && + ! git ls-files -z | xargs -0 rg -n --pcre2 '(?:sk-|Bearer[[:space:]]+)[A-Za-z0-9_-]{24,}' -- >/dev/null 2>&1 && + [[ "$(stat -f '%Lp' "$admin_headers" 2>/dev/null || stat -c '%a' "$admin_headers")" == "600" ]] +} + make_key_payload() { - local alias_file="$1" payload_file="$2" - jq -n \ - --rawfile alias "$alias_file" \ - --rawfile user "$tmpdir/test-user" \ - --rawfile model "$tmpdir/model-name" \ - '{key_alias: ($alias | rtrimstr("\n")), user_id: ($user | rtrimstr("\n")), models: [($model | rtrimstr("\n"))], duration: "15m", max_budget: 0.05, rpm_limit: 10, tpm_limit: 1000, key_type: "llm_api", allowed_routes: ["/v1/models", "/v1/chat/completions"]}' \ - > "$payload_file" + local alias_file="$1" payload_file="$2" explicit_key_file="${3:-}" + if [[ -n "$explicit_key_file" ]]; then + jq -n \ + --rawfile alias "$alias_file" \ + --rawfile user "$tmpdir/test-user" \ + --rawfile model "$tmpdir/model-name" \ + --rawfile key "$explicit_key_file" \ + '{key: ($key | rtrimstr("\n")), key_alias: ($alias | rtrimstr("\n")), user_id: ($user | rtrimstr("\n")), models: [($model | rtrimstr("\n"))], duration: "15m", max_budget: 0.05, rpm_limit: 10, tpm_limit: 1000, key_type: "llm_api", allowed_routes: ["/v1/models", "/v1/chat/completions", "/key/info"]}' \ + > "$payload_file" + else + jq -n \ + --rawfile alias "$alias_file" \ + --rawfile user "$tmpdir/test-user" \ + --rawfile model "$tmpdir/model-name" \ + '{key_alias: ($alias | rtrimstr("\n")), user_id: ($user | rtrimstr("\n")), models: [($model | rtrimstr("\n"))], duration: "15m", max_budget: 0.05, rpm_limit: 10, tpm_limit: 1000, key_type: "llm_api", allowed_routes: ["/v1/models", "/v1/chat/completions"]}' \ + > "$payload_file" + fi chmod 600 "$payload_file" } @@ -326,15 +400,15 @@ wait_for_rejection() { } write_summary() { - [[ -n "$SUMMARY_FILE" ]] || return 0 mkdir -p "$(dirname "$SUMMARY_FILE")" chmod 700 "$(dirname "$SUMMARY_FILE")" jq -n \ --arg commit "$(git -C "$DEMO_DIR/../.." rev-parse HEAD)" \ - --arg image_id "$(docker image inspect "${LITELLM_IMAGE:-}" --format '{{.Id}}' 2>/dev/null || true)" \ + --arg image_id "$(docker image inspect "$image_ref" --format '{{.Id}}' 2>/dev/null || true)" \ --arg tested_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ - --arg mode "$MODE" --arg block_ms "$block_elapsed_ms" --arg delete_ms "$delete_elapsed_ms" --arg redis_recovery "$redis_recovery_result" --arg shared_spend_counter "$shared_spend_counter_result" --arg phase "$smoke_phase" --arg exit_code "$smoke_exit_code" \ - '{commit:$commit,image_id:$image_id,tested_at:$tested_at,mode:$mode,result:(if $exit_code == "0" then "passed" else "failed" end),phase:$phase,migration:"external run-migration.sh",chat:true,stream:true,tool:true,usage:true,shared_rpm_limit:($mode == "ha"),shared_spend_counter:$shared_spend_counter,redis_recovery:$redis_recovery,block_elapsed_ms:($block_ms|tonumber?),delete_elapsed_ms:($delete_ms|tonumber?),cleanup:"completed",security_scan:"passed",content_redacted:true}' \ + --arg mode "$MODE" --arg block_ms "$block_elapsed_ms" --arg delete_ms "$delete_elapsed_ms" --arg phase "$smoke_phase" --arg exit_code "$smoke_exit_code" \ + --arg result "$result" --arg migration "$migration_result" --arg chat "$chat_result" --arg stream "$stream_result" --arg tool "$tool_result" --arg usage "$usage_result" --arg block "$block_result" --arg delete "$delete_result" --arg shared_rpm "$shared_rpm_limit_result" --arg shared_enforcement "$shared_enforcement_result" --arg shared_spend "$shared_spend_counter_result" --arg idempotency "$idempotency_recovery_result" --arg cleanup "$cleanup_result" --arg security "$security_scan_result" \ + '{commit:$commit,image_id:$image_id,tested_at:$tested_at,mode:$mode,result:$result,phase:$phase,exit_code:($exit_code|tonumber?),migration:$migration,chat:$chat,stream:$stream,tool:$tool,usage:$usage,block:$block,delete:$delete,shared_rpm_limit:$shared_rpm,shared_enforcement:$shared_enforcement,shared_spend_counter:$shared_spend,idempotency_recovery:$idempotency,block_elapsed_ms:($block_ms|tonumber?),delete_elapsed_ms:($delete_ms|tonumber?),cleanup:$cleanup,security_scan:$security,content_redacted:true}' \ > "$SUMMARY_FILE" chmod 600 "$SUMMARY_FILE" echo "PASS summary: $SUMMARY_FILE" @@ -372,6 +446,8 @@ wait_ready "$BASE_URL" if [[ "$MODE" == "ha" ]]; then wait_ready "$PEER_URL"; fi assert_redis litellm-1 if [[ "$MODE" == "ha" ]]; then assert_redis litellm-2; fi +assert_migration_evidence +migration_result="passed" echo "PASS readiness: PostgreSQL connected; Redis independently reachable from LiteLLM replica(s)." suffix="$(date +%s)-$RANDOM" @@ -381,6 +457,8 @@ model_name="p1-smoke-model-$suffix" write_private_value "$tmpdir/test-user" "$test_user" write_private_value "$tmpdir/credential-name" "$credential_name" write_private_value "$tmpdir/model-name" "$model_name" +write_private_value "$tmpdir/prompt-marker" "p1-redaction-prompt-$suffix" +write_private_value "$tmpdir/tool-marker" "p1-redaction-tool-$suffix" jq -n --rawfile user "$tmpdir/test-user" '{user_id: ($user | rtrimstr("\n")), auto_create_key: false, user_role: "internal_user"}' > "$tmpdir/user-create.json" chmod 600 "$tmpdir/user-create.json" @@ -391,13 +469,16 @@ jq -e --rawfile user "$tmpdir/test-user" '.user_id == ($user | rtrimstr("\n"))' if [[ ! -s "$upstream_key_file" || ! -s "$upstream_base_file" || ! -s "$upstream_model_file" ]]; then echo "PENDING upstream smoke: set UPSTREAM_API_KEY, UPSTREAM_BASE_URL and UPSTREAM_MODEL in ignored local environment file." + chat_result="pending"; stream_result="pending"; tool_result="pending"; usage_result="pending" + block_result="pending"; delete_result="pending"; idempotency_recovery_result="pending" + result="skipped"; smoke_phase="pending_upstream" echo "PASS infrastructure: PostgreSQL persistence, shared Redis reachability, management authentication and cleanup path are ready." exit 0 fi # P1 deliberately supports one explicit provider mapping. Reject incomplete # or ambiguous combinations before any upstream-facing request is sent. -case "${UPSTREAM_PROVIDER:-}" in +case "$upstream_provider" in deepseek) provider_prefix="deepseek" ;; @@ -440,11 +521,24 @@ jq -n --rawfile id "$tmpdir/model-id" '{id: ($id | rtrimstr("\n"))}' > "$tmpdir/ chmod 600 "$tmpdir/model-delete.json" write_private_value "$tmpdir/block-key-alias" "p1-smoke-block-$suffix" -make_key_payload "$tmpdir/block-key-alias" "$tmpdir/block-key-create.json" -request_admin POST /key/generate "$tmpdir/block-key-create.json" "$tmpdir/block-key.json" -make_key_header "$tmpdir/block-key.json" "$block_key_file" "$block_headers" -hash_key_file "$block_key_file" "$tmpdir/block-key-sha256" +private_file "$block_key_file" +python3 -c 'import secrets; print("sk-p1-" + secrets.token_urlsafe(32))' > "$block_key_file" +assert_private_file "$block_key_file" +make_key_payload "$tmpdir/block-key-alias" "$tmpdir/block-key-create.json" "$block_key_file" +# Intentionally discard the create response to model a client-side timeout. +# The stable caller-generated key is then recovered through /key/info using +# its 0600 Authorization header, never a query parameter or process argument. +request_admin POST /key/generate "$tmpdir/block-key-create.json" /dev/null block_key_created=true +make_header_file "$block_headers" "$block_key_file" +request_data_get "$BASE_URL" "$block_headers" /key/info "$tmpdir/key-recovery.json" +jq -e --rawfile alias "$tmpdir/block-key-alias" '.key_alias == ($alias | rtrimstr("\n"))' "$tmpdir/key-recovery.json" >/dev/null +retry_code="$(curl --silent --show-error --max-time 20 --request POST "$BASE_URL/key/generate" --header "@$admin_headers" --data-binary "@$tmpdir/block-key-create.json" --output "$tmpdir/key-retry.json" --write-out '%{http_code}' || true)" +[[ "$retry_code" =~ ^(400|409|422)$ ]] || { echo "stable-key retry unexpectedly created a second resource: http=$retry_code" >&2; exit 1; } +request_data_get "$BASE_URL" "$block_headers" /key/info "$tmpdir/key-recovery-after-retry.json" +jq -e --rawfile alias "$tmpdir/block-key-alias" '.key_alias == ($alias | rtrimstr("\n"))' "$tmpdir/key-recovery-after-retry.json" >/dev/null +idempotency_recovery_result="passed" +hash_key_file "$block_key_file" "$tmpdir/block-key-sha256" make_key_action_payload delete "$block_key_file" "$tmpdir/block-key-cleanup.json" # GET must be explicit: this verifies both authorization and model visibility. @@ -454,10 +548,11 @@ if [[ "$MODE" == "ha" ]]; then echo "PASS HA pre-revocation: second replica accepted the virtual key." fi -jq -n --rawfile model "$tmpdir/model-name" '{model: ($model | rtrimstr("\n")), messages: [{role: "user", content: "Reply with OK."}], max_tokens: 16}' > "$tmpdir/chat-request.json" +jq -n --rawfile model "$tmpdir/model-name" --rawfile prompt "$tmpdir/prompt-marker" '{model: ($model | rtrimstr("\n")), messages: [{role: "user", content: ($prompt | rtrimstr("\n"))}], max_tokens: 16}' > "$tmpdir/chat-request.json" chmod 600 "$tmpdir/chat-request.json" request_data_post "$BASE_URL" "$block_headers" /v1/chat/completions "$tmpdir/chat-request.json" "$tmpdir/chat.json" jq -e '.choices[0].message.content | type == "string"' "$tmpdir/chat.json" >/dev/null +chat_result="passed" if [[ "$MODE" == "ha" ]]; then peer_completion_ready=false @@ -477,8 +572,9 @@ jq '. + {stream: true}' "$tmpdir/chat-request.json" > "$tmpdir/stream-request.js chmod 600 "$tmpdir/stream-request.json" request_data_post "$BASE_URL" "$block_headers" /v1/chat/completions "$tmpdir/stream-request.json" "$tmpdir/stream.txt" rg -q '^data: ' "$tmpdir/stream.txt" +stream_result="passed" -jq -n --rawfile model "$tmpdir/model-name" '{model: ($model | rtrimstr("\n")), messages: [{role: "user", content: "Use the supplied function to answer 2+2."}], tools: [{type: "function", function: {name: "answer", description: "Return the answer.", parameters: {type: "object", properties: {answer: {type: "integer"}}, required: ["answer"]}}}], tool_choice: {type: "function", function: {name: "answer"}}, thinking: {type: "disabled"}, max_tokens: 32}' > "$tmpdir/tool-request.json" +jq -n --rawfile model "$tmpdir/model-name" --rawfile prompt "$tmpdir/prompt-marker" --rawfile tool "$tmpdir/tool-marker" '{model: ($model | rtrimstr("\n")), messages: [{role: "user", content: ($prompt | rtrimstr("\n"))}], tools: [{type: "function", function: {name: ($tool | rtrimstr("\n")), description: "Return one integer.", parameters: {type: "object", properties: {answer: {type: "integer"}}, required: ["answer"]}}}], tool_choice: {type: "function", function: {name: ($tool | rtrimstr("\n"))}}, thinking: {type: "disabled"}, max_tokens: 32}' > "$tmpdir/tool-request.json" chmod 600 "$tmpdir/tool-request.json" request_data_post "$BASE_URL" "$block_headers" /v1/chat/completions "$tmpdir/tool-request.json" "$tmpdir/tool.json" if ! jq -e '.choices[0].message.tool_calls | type == "array" and length > 0' "$tmpdir/tool.json" >/dev/null; then @@ -486,6 +582,7 @@ if ! jq -e '.choices[0].message.tool_calls | type == "array" and length > 0' "$t exit 1 fi assert_spend +usage_result="passed" smoke_phase="shared_limit_and_redis_recovery" if [[ "$MODE" == "ha" ]]; then @@ -501,34 +598,31 @@ if [[ "$MODE" == "ha" ]]; then request_data_post "$BASE_URL" "$rate_headers" /v1/chat/completions "$tmpdir/chat-request.json" "$tmpdir/rate-first.json" rate_code="$(curl --silent --show-error --max-time 30 --request POST "$PEER_URL/v1/chat/completions" --header "@$rate_headers" --data-binary "@$tmpdir/chat-request.json" --output "$tmpdir/rate-second.json" --write-out '%{http_code}' || true)" [[ "$rate_code" == "429" ]] || { echo "shared RPM limit was bypassed by peer: http=$rate_code" >&2; exit 1; } + shared_rpm_limit_result="passed" echo "PASS HA shared RPM: peer rejected the second request with 429." - redis_container="$(docker compose --env-file "$ENV_FILE" -f "$DEMO_DIR/docker-compose.litellm.yml" ps -q redis)" - [[ -n "$redis_container" ]] || { echo "Redis container not found" >&2; exit 1; } - redis_network="$(docker inspect -f '{{range $name, $_ := .NetworkSettings.Networks}}{{$name}}{{end}}' "$redis_container")" - [[ -n "$redis_network" ]] || { echo "Redis network not found" >&2; exit 1; } - # Disconnecting the Redis endpoint keeps LiteLLM processes runnable, so the - # bounded probes can prove failure without docker exec freezing on a paused - # target container. - docker network disconnect "$redis_network" "$redis_container" - if assert_redis litellm-1 >/dev/null 2>&1 || assert_redis litellm-2 >/dev/null 2>&1; then - docker network connect --alias redis "$redis_network" "$redis_container" >/dev/null || true - echo "Redis interruption did not fail both bounded probes" >&2 - exit 1 - fi - docker network connect --alias redis "$redis_network" "$redis_container" - sleep $(( ${REDIS_CIRCUIT_BREAKER_RECOVERY_TIMEOUT:-5} + 1 )) - assert_redis litellm-1 - assert_redis litellm-2 - wait_model_access "$PEER_URL" "$block_headers" post-redis-recovery - redis_recovery_result="passed" - echo "PASS HA Redis recovery: both probes and peer key authorization recovered." + # Budget is enforced after the first real request. The second request goes + # to the other replica, so a 429 proves the counter is not replica-local. + write_private_value "$tmpdir/enforcement-key-alias" "p1-smoke-budget-$suffix" + make_key_payload "$tmpdir/enforcement-key-alias" "$tmpdir/enforcement-key-create.json" + jq '.key_max_budget = 0.000001' "$tmpdir/enforcement-key-create.json" > "$tmpdir/enforcement-key-limited.json" + chmod 600 "$tmpdir/enforcement-key-limited.json" + request_admin POST /key/generate "$tmpdir/enforcement-key-limited.json" "$tmpdir/enforcement-key.json" + make_key_header "$tmpdir/enforcement-key.json" "$enforcement_key_file" "$enforcement_headers" + enforcement_key_created=true + make_key_action_payload delete "$enforcement_key_file" "$tmpdir/enforcement-key-cleanup.json" + request_data_post "$BASE_URL" "$enforcement_headers" /v1/chat/completions "$tmpdir/chat-request.json" "$tmpdir/budget-first.json" + budget_code="$(curl --silent --show-error --max-time 30 --request POST "$PEER_URL/v1/chat/completions" --header "@$enforcement_headers" --data-binary "@$tmpdir/chat-request.json" --output "$tmpdir/budget-second.json" --write-out '%{http_code}' || true)" + [[ "$budget_code" == "429" ]] || { echo "shared budget/Spend limit was bypassed by peer: http=$budget_code" >&2; exit 1; } + shared_enforcement_result="passed" + echo "PASS HA shared budget/Spend enforcement: peer rejected post-spend request with 429." fi make_key_action_payload block "$block_key_file" "$tmpdir/block-key.json" smoke_phase="revocation" request_admin POST /key/block "$tmpdir/block-key.json" "$tmpdir/block-response.json" wait_for_rejection "$block_headers" block +block_result="passed" # Delete is validated with a different, previously unblocked key. write_private_value "$tmpdir/delete-key-alias" "p1-smoke-delete-$suffix" @@ -544,8 +638,10 @@ if [[ "$MODE" == "ha" ]]; then fi request_admin POST /key/delete "$tmpdir/delete-key-cleanup.json" "$tmpdir/delete-response.json" wait_for_rejection "$delete_headers" delete +delete_key_created=false +delete_result="passed" +runtime_security_check +security_scan_result="passed" smoke_phase="completed" -smoke_exit_code="0" -write_summary echo "PASS complete: user/credential/model/key cleanup, explicit GET models, chat/stream/tool, token-bearing spend, and independent block/delete propagation." diff --git a/docker_litellm/demo/scripts/smoke-redis-recovery.sh b/docker_litellm/demo/scripts/smoke-redis-recovery.sh old mode 100755 new mode 100644 index 6092e40..d5f0c8a --- a/docker_litellm/demo/scripts/smoke-redis-recovery.sh +++ b/docker_litellm/demo/scripts/smoke-redis-recovery.sh @@ -11,17 +11,55 @@ container="" network="" phase="initializing" result="failed" +security_scan="not_run" +post_recovery_call="not_run" +tmpdir="" + +[[ -f "$env_file" ]] || { echo "missing ignored local environment file" >&2; exit 2; } +# shellcheck disable=SC1090 +source "$env_file" +: "${LITELLM_MASTER_KEY:?missing LITELLM_MASTER_KEY in ignored local environment file}" +: "${LITELLM_IMAGE:?missing LITELLM_IMAGE in ignored local environment file}" +image_ref="$LITELLM_IMAGE" +umask 077 +tmpdir="$(mktemp -d "${TMPDIR:-/tmp}/litellm-redis-recovery.XXXXXX")" +chmod 700 "$tmpdir" +master_file="$tmpdir/master-key" +headers_file="$tmpdir/admin.headers" +printf '%s' "$LITELLM_MASTER_KEY" > "$master_file" +chmod 600 "$master_file" +{ printf 'Authorization: Bearer '; tr -d '\r\n' < "$master_file"; printf '\n'; } > "$headers_file" +chmod 600 "$headers_file" +export -n LITELLM_MASTER_KEY POSTGRES_PASSWORD REDIS_PASSWORD UPSTREAM_API_KEY \ + UPSTREAM_BASE_URL UPSTREAM_MODEL 2>/dev/null || true +unset LITELLM_MASTER_KEY POSTGRES_PASSWORD REDIS_PASSWORD UPSTREAM_API_KEY \ + UPSTREAM_BASE_URL UPSTREAM_MODEL + +write_summary() { + umask 077 + mkdir -p "$(dirname "$summary_file")" + chmod 700 "$(dirname "$summary_file")" + jq -n \ + --arg commit "$(git -C "$demo_dir/../.." rev-parse HEAD)" \ + --arg image_id "$(docker image inspect "$image_ref" --format '{{.Id}}' 2>/dev/null || true)" \ + --arg tested_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ + --arg result "$result" --arg phase "$phase" --arg security_scan "$security_scan" \ + --arg post_recovery_call "$post_recovery_call" \ + '{commit:$commit,image_id:$image_id,tested_at:$tested_at,mode:"ha",result:$result,phase:$phase,redis_recovery:$result,post_recovery_call:$post_recovery_call,security_scan:$security_scan,content_redacted:true}' \ + > "$summary_file" + chmod 600 "$summary_file" +} cleanup() { + local exit_code=$? set +e if [[ -n "$container" && -n "$network" ]]; then docker network connect --alias redis "$network" "$container" >/dev/null 2>&1 || true fi - mkdir -p "$(dirname "$summary_file")" - chmod 700 "$(dirname "$summary_file")" - jq -n --arg commit "$(git -C "$demo_dir/../.." rev-parse HEAD)" --arg tested_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" --arg result "$result" --arg phase "$phase" \ - '{commit:$commit,tested_at:$tested_at,redis_recovery:$result,phase:$phase,credentials_or_content:false}' > "$summary_file" - chmod 600 "$summary_file" + rm -rf "$tmpdir" + [[ ! -e "$tmpdir" ]] || { result="failed"; phase="temporary_cleanup_failed"; } + if (( exit_code != 0 )); then result="failed"; fi + write_summary } trap cleanup EXIT @@ -29,6 +67,16 @@ probe() { docker exec "$1" python3 -c 'import redis; p=open("/run/secrets/redis_password").read().strip(); assert redis.Redis(host="redis", password=p, socket_connect_timeout=2, socket_timeout=2).ping()' } +runtime_security_check() { + docker inspect "$container" > "$tmpdir/redis-inspect.json" + docker inspect svc-litellm-1 svc-litellm-2 > "$tmpdir/litellm-inspect.json" + docker exec "$container" ps -eo args > "$tmpdir/redis-processes.txt" + ! rg -q -- '--requirepass[[:space:]]+[^[:space:]]+' "$tmpdir/redis-processes.txt" && + ! rg -q 'UPSTREAM_(API_KEY|BASE_URL|MODEL)=' "$tmpdir/litellm-inspect.json" && + rg -q '/run/secrets/redis_password' "$tmpdir/redis-inspect.json" && + [[ "$(stat -f '%Lp' "$headers_file" 2>/dev/null || stat -c '%a' "$headers_file")" == "600" ]] +} + container="$(docker compose --env-file "$env_file" -f "$demo_dir/docker-compose.litellm.yml" ps -q redis)" [[ -n "$container" ]] || { phase="redis_not_found"; exit 1; } network="$(docker inspect -f '{{range $name, $_ := .NetworkSettings.Networks}}{{$name}}{{end}}' "$container")" @@ -46,6 +94,17 @@ docker network connect --alias redis "$network" "$container" sleep $((recovery_timeout + 1)) probe svc-litellm-1 probe svc-litellm-2 -result="passed" + +# This is an authenticated LiteLLM call after recovery, not only a socket PING. +curl --silent --show-error --fail --max-time 20 --request GET http://127.0.0.1:4000/v1/models \ + --header "@$headers_file" --output "$tmpdir/models-1.json" +curl --silent --show-error --fail --max-time 20 --request GET http://127.0.0.1:4001/v1/models \ + --header "@$headers_file" --output "$tmpdir/models-2.json" +jq -e '.data | type == "array"' "$tmpdir/models-1.json" "$tmpdir/models-2.json" >/dev/null +post_recovery_call="passed" + +runtime_security_check +security_scan="passed" phase="completed" -echo "PASS Redis recovery: both bounded probes failed during outage and recovered afterward." +result="passed" +echo "PASS Redis recovery: both bounded probes failed during outage, recovered, and both LiteLLM replicas served an authenticated GET afterward." diff --git a/tool.sh b/tool.sh index c65cc7a..9e7fa28 100644 --- a/tool.sh +++ b/tool.sh @@ -12,7 +12,15 @@ CI_PROJECT_SPACE=$(echo "${CI_PROJECT_BRANCH}" | cut -f1 -d'/') # If on the main branch, image namespace will be same as CI_PROJECT_NAME's name space; # else (not main branch), image namespace = {CI_PROJECT_NAME's name space} + "0" + {1st substr before / in CI_PROJECT_SPACE}. -[ "${CI_PROJECT_BRANCH}" = "main" ] && NAMESPACE_SUFFIX="" || NAMESPACE_SUFFIX="0${CI_PROJECT_SPACE}" ; +# A local LabNow service build is an intentionally local-only artifact, but it +# must keep the stable `labnow` namespace used by each service's Compose file. +# CI keeps the historical branch-suffixed namespace behaviour unchanged. This +# avoids pretending a development branch is `main` just to obtain its tag. +if [ "${GITHUB_ACTIONS:-false}" = "true" ] && [ "${CI_PROJECT_BRANCH}" != "main" ]; then + NAMESPACE_SUFFIX="0${CI_PROJECT_SPACE}" +else + NAMESPACE_SUFFIX="" +fi export CI_PROJECT_NAMESPACE="$(dirname ${CI_PROJECT_NAME})${NAMESPACE_SUFFIX}" ; export IMG_NAMESPACE=$(echo "${CI_PROJECT_NAMESPACE}" | awk '{print tolower($0)}') From e64eefcc8dad1dce57bce57957b27c3c73d1afc2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 13:07:20 +0800 Subject: [PATCH 17/87] =?UTF-8?q?fix:=20=E6=81=A2=E5=A4=8D=20LiteLLM=20?= =?UTF-8?q?=E9=AA=8C=E6=94=B6=E8=84=9A=E6=9C=AC=E5=8F=AF=E6=89=A7=E8=A1=8C?= =?UTF-8?q?=E6=9D=83=E9=99=90?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/demo/scripts/aggregate-verification-summary.sh | 0 docker_litellm/demo/scripts/smoke-redis-recovery.sh | 0 2 files changed, 0 insertions(+), 0 deletions(-) mode change 100644 => 100755 docker_litellm/demo/scripts/aggregate-verification-summary.sh mode change 100644 => 100755 docker_litellm/demo/scripts/smoke-redis-recovery.sh diff --git a/docker_litellm/demo/scripts/aggregate-verification-summary.sh b/docker_litellm/demo/scripts/aggregate-verification-summary.sh old mode 100644 new mode 100755 diff --git a/docker_litellm/demo/scripts/smoke-redis-recovery.sh b/docker_litellm/demo/scripts/smoke-redis-recovery.sh old mode 100644 new mode 100755 From f0bc8ba495d10648d8d5d98affbc5662ac6fb7e8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 13:11:29 +0800 Subject: [PATCH 18/87] =?UTF-8?q?fix:=20=E8=A7=84=E8=8C=83=E5=8C=96=20Lite?= =?UTF-8?q?LLM=20=E4=B8=8A=E6=B8=B8=20provider?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/demo/scripts/smoke-baseline.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh index 98aab16..55eda9d 100755 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -478,7 +478,8 @@ fi # P1 deliberately supports one explicit provider mapping. Reject incomplete # or ambiguous combinations before any upstream-facing request is sent. -case "$upstream_provider" in +provider_normalized="$(printf '%s' "$upstream_provider" | tr '[:upper:]' '[:lower:]')" +case "$provider_normalized" in deepseek) provider_prefix="deepseek" ;; From 6a720cd07f970af150c577c3f715164f93c18407 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 13:13:43 +0800 Subject: [PATCH 19/87] =?UTF-8?q?fix:=20=E5=85=BC=E5=AE=B9=20DeepSeek=20?= =?UTF-8?q?=E6=9C=AC=E5=9C=B0=20provider=20=E6=A0=87=E8=AF=86?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/demo/scripts/smoke-baseline.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh index 55eda9d..b3d05a2 100755 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -480,7 +480,7 @@ fi # or ambiguous combinations before any upstream-facing request is sent. provider_normalized="$(printf '%s' "$upstream_provider" | tr '[:upper:]' '[:lower:]')" case "$provider_normalized" in - deepseek) + deepseek|deepseek-direct|deepseek_direct) provider_prefix="deepseek" ;; *) From ff23998be3fd795efa5cb5d7bb72e5e09d0543bc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 13:16:03 +0800 Subject: [PATCH 20/87] =?UTF-8?q?fix:=20=E6=89=A9=E5=B1=95=20DeepSeek=20P1?= =?UTF-8?q?=20provider=20=E6=98=A0=E5=B0=84?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/demo/scripts/smoke-baseline.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh index b3d05a2..dfc073c 100755 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -480,7 +480,7 @@ fi # or ambiguous combinations before any upstream-facing request is sent. provider_normalized="$(printf '%s' "$upstream_provider" | tr '[:upper:]' '[:lower:]')" case "$provider_normalized" in - deepseek|deepseek-direct|deepseek_direct) + deepseek|deepseek-direct|deepseek_direct|deepseek-v4|deepseek_v4|deepseek-v4-flash|deepseek_v4_flash) provider_prefix="deepseek" ;; *) From 26cc5360c7019030c17f41e098ade04cd9298465 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 13:18:51 +0800 Subject: [PATCH 21/87] =?UTF-8?q?fix:=20=E8=A7=84=E8=8C=83=E5=8C=96=20Lite?= =?UTF-8?q?LLM=20provider=20=E8=BE=93=E5=85=A5?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/demo/scripts/smoke-baseline.sh | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh index dfc073c..5b63cca 100755 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -155,16 +155,19 @@ master_key_file="$tmpdir/master-key" upstream_key_file="$tmpdir/upstream-key" upstream_base_file="$tmpdir/upstream-base" upstream_model_file="$tmpdir/upstream-model" +upstream_provider_file="$tmpdir/upstream-provider" admin_headers="$tmpdir/admin.headers" write_private_value "$master_key_file" "$LITELLM_MASTER_KEY" write_private_value "$upstream_key_file" "${UPSTREAM_API_KEY:-}" write_private_value "$upstream_base_file" "${UPSTREAM_BASE_URL:-}" write_private_value "$upstream_model_file" "${UPSTREAM_MODEL:-}" +write_private_value "$upstream_provider_file" "$upstream_provider" make_header_file "$admin_headers" "$master_key_file" assert_private_file "$master_key_file" assert_private_file "$upstream_key_file" assert_private_file "$upstream_base_file" assert_private_file "$upstream_model_file" +assert_private_file "$upstream_provider_file" # No child process needs these values. Compose reads its own --env-file. unset LITELLM_MASTER_KEY UPSTREAM_API_KEY UPSTREAM_BASE_URL UPSTREAM_MODEL UPSTREAM_PROVIDER \ @@ -234,7 +237,7 @@ cleanup() { cleanup_request_admin POST /user/delete "$tmpdir/user-delete.json" || cleanup_ok=false fi if [[ "$cleanup_ok" == true ]]; then cleanup_result="passed"; else cleanup_result="failed"; exit_code=1; fi - unset master_key_file upstream_key_file upstream_base_file upstream_model_file + unset master_key_file upstream_key_file upstream_base_file upstream_model_file upstream_provider_file rm -rf "$tmpdir" if [[ -e "$tmpdir" ]]; then cleanup_result="failed"; exit_code=1; fi smoke_exit_code="$exit_code" @@ -478,7 +481,7 @@ fi # P1 deliberately supports one explicit provider mapping. Reject incomplete # or ambiguous combinations before any upstream-facing request is sent. -provider_normalized="$(printf '%s' "$upstream_provider" | tr '[:upper:]' '[:lower:]')" +provider_normalized="$(tr -d '\r\n' < "$upstream_provider_file" | tr '[:upper:]' '[:lower:]')" case "$provider_normalized" in deepseek|deepseek-direct|deepseek_direct|deepseek-v4|deepseek_v4|deepseek-v4-flash|deepseek_v4_flash) provider_prefix="deepseek" From 0a588e6e9fa86e45e2393d4ffa5d8298092c1f88 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 13:22:14 +0800 Subject: [PATCH 22/87] =?UTF-8?q?fix:=20=E8=AE=A9=20LiteLLM=20=E8=BF=81?= =?UTF-8?q?=E7=A7=BB=E5=A4=B1=E8=B4=A5=E6=97=B6=E7=BB=88=E6=AD=A2?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/demo/docker-compose.litellm.yml | 2 +- docker_litellm/litellm.Dockerfile | 4 ++++ 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/docker_litellm/demo/docker-compose.litellm.yml b/docker_litellm/demo/docker-compose.litellm.yml index 5a56663..b24020b 100644 --- a/docker_litellm/demo/docker-compose.litellm.yml +++ b/docker_litellm/demo/docker-compose.litellm.yml @@ -43,7 +43,7 @@ services: <<: *litellm-common container_name: ${LITELLM_MIGRATE_CONTAINER_NAME:-svc-litellm-migrate} profiles: ["migrate"] - command: ["/bin/bash", "/opt/utils/start-litellm.sh", "--config", "config.migrate.yaml", "--skip_server_startup"] + command: ["/bin/bash", "/opt/utils/start-litellm.sh", "--config", "config.migrate.yaml", "--skip_server_startup", "--enforce_prisma_migration_check"] healthcheck: disable: true depends_on: diff --git a/docker_litellm/litellm.Dockerfile b/docker_litellm/litellm.Dockerfile index 2a843fa..1c7e159 100644 --- a/docker_litellm/litellm.Dockerfile +++ b/docker_litellm/litellm.Dockerfile @@ -51,6 +51,10 @@ WORKDIR ${HOME_LITELLM} # Copy utilities, tools and build artifacts COPY work /opt/utils/ COPY --from=builder /build/dist/*.whl /tmp/ +# prisma-python invokes Node again for migration operations. Copy the fixed +# builder runtime into the final image so a fresh migration container never +# tries to download Node during startup. +COPY --from=builder /opt/node /opt/node # Install Runtime dependencies and configure tools RUN set -eux \ From 8c109c22dbed2325fad1e1edc96c451c86da7cc5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 13:36:22 +0800 Subject: [PATCH 23/87] =?UTF-8?q?fix:=20=E5=BD=92=E4=B8=80=E5=8C=96=20Deep?= =?UTF-8?q?Seek=20provider=20=E5=8F=98=E4=BD=93?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/demo/scripts/smoke-baseline.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh index 5b63cca..457ad1d 100755 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -483,7 +483,7 @@ fi # or ambiguous combinations before any upstream-facing request is sent. provider_normalized="$(tr -d '\r\n' < "$upstream_provider_file" | tr '[:upper:]' '[:lower:]')" case "$provider_normalized" in - deepseek|deepseek-direct|deepseek_direct|deepseek-v4|deepseek_v4|deepseek-v4-flash|deepseek_v4_flash) + deepseek*) provider_prefix="deepseek" ;; *) From 09bc3c801841255833ffb4e14e8a1ff9b2fdb23a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 13:43:00 +0800 Subject: [PATCH 24/87] =?UTF-8?q?fix:=20=E6=94=AF=E6=8C=81=20LiteLLM=20smo?= =?UTF-8?q?ke=20provider=20=E8=A6=86=E7=9B=96?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/demo/scripts/smoke-baseline.sh | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh index 457ad1d..93ef608 100755 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -105,7 +105,10 @@ source "$ENV_FILE" : "${LITELLM_MASTER_KEY:?missing LITELLM_MASTER_KEY in local environment file}" : "${LITELLM_IMAGE:?missing LITELLM_IMAGE in local environment file}" image_ref="$LITELLM_IMAGE" -upstream_provider="${UPSTREAM_PROVIDER:-}" +# Provider selection is smoke-client-only. The explicit override is useful +# when a legacy ignored Compose env has an old provider label; it never enters +# the LiteLLM container and is not a credential. +upstream_provider="${LITELLM_SMOKE_UPSTREAM_PROVIDER:-${UPSTREAM_PROVIDER:-}}" # An `.env` may use `export NAME=...`; remove that export attribute before # mktemp, chmod, tr, curl, jq, or any other child process is started. From 8973251b818c9bc2a8cbcb0cccd4d0f11d855eb1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 13:47:43 +0800 Subject: [PATCH 25/87] =?UTF-8?q?fix:=20=E4=BD=BF=E7=94=A8=20Compose=20?= =?UTF-8?q?=E6=9C=89=E6=95=88=E7=8E=AF=E5=A2=83=E6=89=A7=E8=A1=8C=20LiteLL?= =?UTF-8?q?M=20smoke?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/demo/scripts/smoke-baseline.sh | 37 +++++++++++-------- 1 file changed, 21 insertions(+), 16 deletions(-) diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh index 93ef608..d7f8b38 100755 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -99,22 +99,6 @@ security_scan_result="static_passed" [[ -f "$ENV_FILE" ]] || { echo "missing local environment file: $ENV_FILE (copy .env.example)" >&2; exit 2; } -# Do not use `set -a`: sourced values must not leak to child processes. -# shellcheck disable=SC1090 -source "$ENV_FILE" -: "${LITELLM_MASTER_KEY:?missing LITELLM_MASTER_KEY in local environment file}" -: "${LITELLM_IMAGE:?missing LITELLM_IMAGE in local environment file}" -image_ref="$LITELLM_IMAGE" -# Provider selection is smoke-client-only. The explicit override is useful -# when a legacy ignored Compose env has an old provider label; it never enters -# the LiteLLM container and is not a credential. -upstream_provider="${LITELLM_SMOKE_UPSTREAM_PROVIDER:-${UPSTREAM_PROVIDER:-}}" - -# An `.env` may use `export NAME=...`; remove that export attribute before -# mktemp, chmod, tr, curl, jq, or any other child process is started. -export -n LITELLM_IMAGE LITELLM_MASTER_KEY UPSTREAM_API_KEY UPSTREAM_BASE_URL UPSTREAM_MODEL UPSTREAM_PROVIDER \ - POSTGRES_PASSWORD REDIS_PASSWORD DATABASE_URL 2>/dev/null || true - if [[ "$MODE" == "single" ]]; then BASE_URL="http://${LITELLM_PUBLISH_HOST:-127.0.0.1}:${LITELLM_1_PORT:-4000}" PEER_URL="$BASE_URL" @@ -127,6 +111,27 @@ umask 077 tmpdir="$(mktemp -d "${TMPDIR:-/tmp}/litellm-smoke.XXXXXX")" chmod 700 "$tmpdir" +# Compose's dotenv grammar is not Bash's grammar. Reading it with `source` +# can change quoted/special-character management keys and create a false 403. +# Ask Compose for its effective environment into a 0600 file and never print it. +compose_environment="$tmpdir/compose.environment" +docker compose --env-file "$ENV_FILE" -f "$DEMO_DIR/docker-compose.litellm.yml" config --environment > "$compose_environment" +chmod 600 "$compose_environment" +effective_env() { + local name="$1" + awk -F= -v name="$name" '$1 == name {sub(/^[^=]*=/, ""); print; exit}' "$compose_environment" +} +LITELLM_MASTER_KEY="$(effective_env LITELLM_MASTER_KEY)" +LITELLM_IMAGE="$(effective_env LITELLM_IMAGE)" +UPSTREAM_API_KEY="$(effective_env UPSTREAM_API_KEY)" +UPSTREAM_BASE_URL="$(effective_env UPSTREAM_BASE_URL)" +UPSTREAM_MODEL="$(effective_env UPSTREAM_MODEL)" +UPSTREAM_PROVIDER="$(effective_env UPSTREAM_PROVIDER)" +: "${LITELLM_MASTER_KEY:?missing LITELLM_MASTER_KEY in effective Compose environment}" +: "${LITELLM_IMAGE:?missing LITELLM_IMAGE in effective Compose environment}" +image_ref="$LITELLM_IMAGE" +upstream_provider="${LITELLM_SMOKE_UPSTREAM_PROVIDER:-${UPSTREAM_PROVIDER:-}}" + private_file() { : > "$1" chmod 600 "$1" From c224b365645498bbc11c4437e76b854154544707 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 14:20:58 +0800 Subject: [PATCH 26/87] =?UTF-8?q?fix:=20=E4=BB=8E=20Compose=20=E6=9C=89?= =?UTF-8?q?=E6=95=88=E7=8E=AF=E5=A2=83=E8=AF=BB=E5=8F=96=20smoke=20?= =?UTF-8?q?=E7=AB=AF=E5=8F=A3?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/demo/scripts/smoke-baseline.sh | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh index d7f8b38..efdce3b 100755 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -99,14 +99,6 @@ security_scan_result="static_passed" [[ -f "$ENV_FILE" ]] || { echo "missing local environment file: $ENV_FILE (copy .env.example)" >&2; exit 2; } -if [[ "$MODE" == "single" ]]; then - BASE_URL="http://${LITELLM_PUBLISH_HOST:-127.0.0.1}:${LITELLM_1_PORT:-4000}" - PEER_URL="$BASE_URL" -else - BASE_URL="http://${LITELLM_PUBLISH_HOST:-127.0.0.1}:${LITELLM_1_PORT:-4000}" - PEER_URL="http://${LITELLM_PUBLISH_HOST:-127.0.0.1}:${LITELLM_2_PORT:-4001}" -fi - umask 077 tmpdir="$(mktemp -d "${TMPDIR:-/tmp}/litellm-smoke.XXXXXX")" chmod 700 "$tmpdir" @@ -127,10 +119,20 @@ UPSTREAM_API_KEY="$(effective_env UPSTREAM_API_KEY)" UPSTREAM_BASE_URL="$(effective_env UPSTREAM_BASE_URL)" UPSTREAM_MODEL="$(effective_env UPSTREAM_MODEL)" UPSTREAM_PROVIDER="$(effective_env UPSTREAM_PROVIDER)" +LITELLM_PUBLISH_HOST="$(effective_env LITELLM_PUBLISH_HOST)" +LITELLM_1_PORT="$(effective_env LITELLM_1_PORT)" +LITELLM_2_PORT="$(effective_env LITELLM_2_PORT)" : "${LITELLM_MASTER_KEY:?missing LITELLM_MASTER_KEY in effective Compose environment}" : "${LITELLM_IMAGE:?missing LITELLM_IMAGE in effective Compose environment}" image_ref="$LITELLM_IMAGE" upstream_provider="${LITELLM_SMOKE_UPSTREAM_PROVIDER:-${UPSTREAM_PROVIDER:-}}" +if [[ "$MODE" == "single" ]]; then + BASE_URL="http://${LITELLM_PUBLISH_HOST:-127.0.0.1}:${LITELLM_1_PORT:-4000}" + PEER_URL="$BASE_URL" +else + BASE_URL="http://${LITELLM_PUBLISH_HOST:-127.0.0.1}:${LITELLM_1_PORT:-4000}" + PEER_URL="http://${LITELLM_PUBLISH_HOST:-127.0.0.1}:${LITELLM_2_PORT:-4001}" +fi private_file() { : > "$1" From 34d0202bb5121b0423fed9679aa3315a70a1606e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 14:25:41 +0800 Subject: [PATCH 27/87] =?UTF-8?q?fix:=20=E5=AE=89=E5=85=A8=E6=81=A2?= =?UTF-8?q?=E5=A4=8D=20LiteLLM=20=E8=99=9A=E6=8B=9F=E5=AF=86=E9=92=A5?= =?UTF-8?q?=E5=88=9B=E5=BB=BA=E5=93=8D=E5=BA=94?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/demo/scripts/smoke-baseline.sh | 22 +++++++++++++------ 1 file changed, 15 insertions(+), 7 deletions(-) diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh index efdce3b..7e13c35 100755 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -376,6 +376,12 @@ make_key_action_payload() { chmod 600 "$payload_file" } +make_key_info_payload() { + local key_file="$1" payload_file="$2" + jq -n --rawfile key "$key_file" '{keys: [($key | rtrimstr("\n"))]}' > "$payload_file" + chmod 600 "$payload_file" +} + wait_model_access() { local url="$1" header_file="$2" label="$3" output_file i output_file="$tmpdir/$label-models.json" @@ -540,20 +546,22 @@ python3 -c 'import secrets; print("sk-p1-" + secrets.token_urlsafe(32))' > "$blo assert_private_file "$block_key_file" make_key_payload "$tmpdir/block-key-alias" "$tmpdir/block-key-create.json" "$block_key_file" # Intentionally discard the create response to model a client-side timeout. -# The stable caller-generated key is then recovered through /key/info using -# its 0600 Authorization header, never a query parameter or process argument. +# The stable caller-generated key is then recovered through LiteLLM 1.97.0's +# admin-only /v2/key/info endpoint. The key stays in a 0600 request body; +# it never appears in a query parameter, process argument or report. request_admin POST /key/generate "$tmpdir/block-key-create.json" /dev/null block_key_created=true make_header_file "$block_headers" "$block_key_file" -request_data_get "$BASE_URL" "$block_headers" /key/info "$tmpdir/key-recovery.json" -jq -e --rawfile alias "$tmpdir/block-key-alias" '.key_alias == ($alias | rtrimstr("\n"))' "$tmpdir/key-recovery.json" >/dev/null +make_key_action_payload delete "$block_key_file" "$tmpdir/block-key-cleanup.json" +make_key_info_payload "$block_key_file" "$tmpdir/key-recovery-request.json" +request_admin POST /v2/key/info "$tmpdir/key-recovery-request.json" "$tmpdir/key-recovery.json" +jq -e --rawfile alias "$tmpdir/block-key-alias" '.info | length == 1 and .[0].key_alias == ($alias | rtrimstr("\n"))' "$tmpdir/key-recovery.json" >/dev/null retry_code="$(curl --silent --show-error --max-time 20 --request POST "$BASE_URL/key/generate" --header "@$admin_headers" --data-binary "@$tmpdir/block-key-create.json" --output "$tmpdir/key-retry.json" --write-out '%{http_code}' || true)" [[ "$retry_code" =~ ^(400|409|422)$ ]] || { echo "stable-key retry unexpectedly created a second resource: http=$retry_code" >&2; exit 1; } -request_data_get "$BASE_URL" "$block_headers" /key/info "$tmpdir/key-recovery-after-retry.json" -jq -e --rawfile alias "$tmpdir/block-key-alias" '.key_alias == ($alias | rtrimstr("\n"))' "$tmpdir/key-recovery-after-retry.json" >/dev/null +request_admin POST /v2/key/info "$tmpdir/key-recovery-request.json" "$tmpdir/key-recovery-after-retry.json" +jq -e --rawfile alias "$tmpdir/block-key-alias" '.info | length == 1 and .[0].key_alias == ($alias | rtrimstr("\n"))' "$tmpdir/key-recovery-after-retry.json" >/dev/null idempotency_recovery_result="passed" hash_key_file "$block_key_file" "$tmpdir/block-key-sha256" -make_key_action_payload delete "$block_key_file" "$tmpdir/block-key-cleanup.json" # GET must be explicit: this verifies both authorization and model visibility. wait_model_access "$BASE_URL" "$block_headers" primary From 9b2c166f7080a9b50f5ed5ee84f075a85bfd93ef Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 14:26:12 +0800 Subject: [PATCH 28/87] =?UTF-8?q?fix:=20=E9=80=82=E9=85=8D=20Redis=20?= =?UTF-8?q?=E6=81=A2=E5=A4=8D=E9=AA=8C=E8=AF=81=E7=9A=84=20Compose=20?= =?UTF-8?q?=E7=AB=AF=E5=8F=A3?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/demo/scripts/smoke-redis-recovery.sh | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/docker_litellm/demo/scripts/smoke-redis-recovery.sh b/docker_litellm/demo/scripts/smoke-redis-recovery.sh index d5f0c8a..19b3d07 100755 --- a/docker_litellm/demo/scripts/smoke-redis-recovery.sh +++ b/docker_litellm/demo/scripts/smoke-redis-recovery.sh @@ -21,6 +21,12 @@ source "$env_file" : "${LITELLM_MASTER_KEY:?missing LITELLM_MASTER_KEY in ignored local environment file}" : "${LITELLM_IMAGE:?missing LITELLM_IMAGE in ignored local environment file}" image_ref="$LITELLM_IMAGE" +# Keep the recovery proof aligned with the same optional host-port overrides +# that Compose uses for the two proxy replicas. These are non-secret routing +# values; credentials remain in the private header file below. +publish_host="${LITELLM_PUBLISH_HOST:-127.0.0.1}" +litellm_1_port="${LITELLM_1_PORT:-4000}" +litellm_2_port="${LITELLM_2_PORT:-4001}" umask 077 tmpdir="$(mktemp -d "${TMPDIR:-/tmp}/litellm-redis-recovery.XXXXXX")" chmod 700 "$tmpdir" @@ -96,9 +102,9 @@ probe svc-litellm-1 probe svc-litellm-2 # This is an authenticated LiteLLM call after recovery, not only a socket PING. -curl --silent --show-error --fail --max-time 20 --request GET http://127.0.0.1:4000/v1/models \ +curl --silent --show-error --fail --max-time 20 --request GET "http://${publish_host}:${litellm_1_port}/v1/models" \ --header "@$headers_file" --output "$tmpdir/models-1.json" -curl --silent --show-error --fail --max-time 20 --request GET http://127.0.0.1:4001/v1/models \ +curl --silent --show-error --fail --max-time 20 --request GET "http://${publish_host}:${litellm_2_port}/v1/models" \ --header "@$headers_file" --output "$tmpdir/models-2.json" jq -e '.data | type == "array"' "$tmpdir/models-1.json" "$tmpdir/models-2.json" >/dev/null post_recovery_call="passed" From 6c0f0babf6d5e1d0137e0ea6a89a2a8dd49cc33c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 14:33:43 +0800 Subject: [PATCH 29/87] =?UTF-8?q?fix:=20=E5=BC=BA=E5=8C=96=20LiteLLM=20?= =?UTF-8?q?=E5=86=85=E5=AE=B9=E6=97=A5=E5=BF=97=E9=AA=8C=E6=94=B6?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/demo/scripts/smoke-baseline.sh | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh index 7e13c35..c47c7e4 100755 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -315,16 +315,21 @@ assert_migration_evidence() { } runtime_security_check() { - local logs_file="$tmpdir/litellm-logs.txt" inspect_file="$tmpdir/inspect.json" process_file="$tmpdir/redis-processes.txt" + local logs_file="$tmpdir/litellm-logs.txt" inspect_file="$tmpdir/inspect.json" process_file="$tmpdir/redis-processes.txt" db_content_file="$tmpdir/spendlog-db-content.txt" docker inspect svc-litellm-1 > "$inspect_file" if [[ "$MODE" == ha ]]; then docker inspect svc-litellm-2 >> "$inspect_file"; fi docker compose --env-file "$ENV_FILE" -f "$DEMO_DIR/docker-compose.litellm.yml" logs --no-color litellm-1 > "$logs_file" 2>&1 if [[ "$MODE" == ha ]]; then docker compose --env-file "$ENV_FILE" -f "$DEMO_DIR/docker-compose.litellm.yml" logs --no-color litellm-2 >> "$logs_file" 2>&1; fi docker exec "$(docker compose --env-file "$ENV_FILE" -f "$DEMO_DIR/docker-compose.litellm.yml" ps -q redis)" ps -eo args > "$process_file" + # Query only P1 test SpendLog content into the private work directory. This + # validates the database representation independently of the API response. + docker compose --env-file "$ENV_FILE" -f "$DEMO_DIR/docker-compose.litellm.yml" exec -T postgres sh -lc 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -Atc "SELECT coalesce(messages::text, '\''\'') || coalesce(response::text, '\''\'') || coalesce(proxy_server_request::text, '\''\'') FROM \"LiteLLM_SpendLogs\" WHERE \"user\" LIKE '\''p1-smoke-user-%'\'';"' > "$db_content_file" ! rg -q 'UPSTREAM_(API_KEY|BASE_URL|MODEL)=' "$inspect_file" && ! rg -q -- '--requirepass[[:space:]]+[^[:space:]]+' "$process_file" && ! rg -q --file "$tmpdir/prompt-marker" "$logs_file" && ! rg -q --file "$tmpdir/tool-marker" "$logs_file" && + ! rg -q --file "$tmpdir/prompt-marker" "$db_content_file" && + ! rg -q --file "$tmpdir/tool-marker" "$db_content_file" && ! git ls-files -z | xargs -0 rg -n --pcre2 '(?:sk-|Bearer[[:space:]]+)[A-Za-z0-9_-]{24,}' -- >/dev/null 2>&1 && [[ "$(stat -f '%Lp' "$admin_headers" 2>/dev/null || stat -c '%a' "$admin_headers")" == "600" ]] } @@ -478,6 +483,11 @@ write_private_value "$tmpdir/credential-name" "$credential_name" write_private_value "$tmpdir/model-name" "$model_name" write_private_value "$tmpdir/prompt-marker" "p1-redaction-prompt-$suffix" write_private_value "$tmpdir/tool-marker" "p1-redaction-tool-$suffix" +# This identifier is intentionally non-sensitive. LiteLLM 1.97.0 may emit a +# parser warning with a function name when an upstream tool call is malformed; +# the sensitive marker stays in the tool schema body, which the scan verifies +# is never persisted or logged. +write_private_value "$tmpdir/tool-name" "p1_smoke_tool" jq -n --rawfile user "$tmpdir/test-user" '{user_id: ($user | rtrimstr("\n")), auto_create_key: false, user_role: "internal_user"}' > "$tmpdir/user-create.json" chmod 600 "$tmpdir/user-create.json" @@ -596,13 +606,14 @@ request_data_post "$BASE_URL" "$block_headers" /v1/chat/completions "$tmpdir/str rg -q '^data: ' "$tmpdir/stream.txt" stream_result="passed" -jq -n --rawfile model "$tmpdir/model-name" --rawfile prompt "$tmpdir/prompt-marker" --rawfile tool "$tmpdir/tool-marker" '{model: ($model | rtrimstr("\n")), messages: [{role: "user", content: ($prompt | rtrimstr("\n"))}], tools: [{type: "function", function: {name: ($tool | rtrimstr("\n")), description: "Return one integer.", parameters: {type: "object", properties: {answer: {type: "integer"}}, required: ["answer"]}}}], tool_choice: {type: "function", function: {name: ($tool | rtrimstr("\n"))}}, thinking: {type: "disabled"}, max_tokens: 32}' > "$tmpdir/tool-request.json" +jq -n --rawfile model "$tmpdir/model-name" --rawfile prompt "$tmpdir/prompt-marker" --rawfile tool_name "$tmpdir/tool-name" --rawfile tool_marker "$tmpdir/tool-marker" '{model: ($model | rtrimstr("\n")), messages: [{role: "user", content: ($prompt | rtrimstr("\n"))}], tools: [{type: "function", function: {name: ($tool_name | rtrimstr("\n")), description: ($tool_marker | rtrimstr("\n")), parameters: {type: "object", properties: {answer: {type: "integer", description: ($tool_marker | rtrimstr("\n"))}}, required: ["answer"]}}}], tool_choice: {type: "function", function: {name: ($tool_name | rtrimstr("\n"))}}, thinking: {type: "disabled"}, max_tokens: 32}' > "$tmpdir/tool-request.json" chmod 600 "$tmpdir/tool-request.json" request_data_post "$BASE_URL" "$block_headers" /v1/chat/completions "$tmpdir/tool-request.json" "$tmpdir/tool.json" if ! jq -e '.choices[0].message.tool_calls | type == "array" and length > 0' "$tmpdir/tool.json" >/dev/null; then echo "tool request returned no tool_calls" >&2 exit 1 fi +tool_result="passed" assert_spend usage_result="passed" smoke_phase="shared_limit_and_redis_recovery" From a9500b6a254491f1069269fc684196dbfd97a124 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 14:37:03 +0800 Subject: [PATCH 30/87] =?UTF-8?q?fix:=20=E6=89=AB=E6=8F=8F=20P1=20SpendLog?= =?UTF-8?q?=20=E6=95=B0=E6=8D=AE=E5=BA=93=E6=AD=A3=E6=96=87?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/demo/scripts/smoke-baseline.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh index c47c7e4..43341a5 100755 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -323,7 +323,7 @@ runtime_security_check() { docker exec "$(docker compose --env-file "$ENV_FILE" -f "$DEMO_DIR/docker-compose.litellm.yml" ps -q redis)" ps -eo args > "$process_file" # Query only P1 test SpendLog content into the private work directory. This # validates the database representation independently of the API response. - docker compose --env-file "$ENV_FILE" -f "$DEMO_DIR/docker-compose.litellm.yml" exec -T postgres sh -lc 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -Atc "SELECT coalesce(messages::text, '\''\'') || coalesce(response::text, '\''\'') || coalesce(proxy_server_request::text, '\''\'') FROM \"LiteLLM_SpendLogs\" WHERE \"user\" LIKE '\''p1-smoke-user-%'\'';"' > "$db_content_file" + docker compose --env-file "$ENV_FILE" -f "$DEMO_DIR/docker-compose.litellm.yml" exec -T postgres sh -lc 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -Atc "SELECT concat(messages::text, response::text, proxy_server_request::text) FROM \"LiteLLM_SpendLogs\" WHERE position(\$p\$p1-smoke-user-\$p\$ in \"user\") = 1;"' > "$db_content_file" ! rg -q 'UPSTREAM_(API_KEY|BASE_URL|MODEL)=' "$inspect_file" && ! rg -q -- '--requirepass[[:space:]]+[^[:space:]]+' "$process_file" && ! rg -q --file "$tmpdir/prompt-marker" "$logs_file" && From 2dfb89e22c89ec97069d0a2afbbc2b004ddf1ea3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 14:41:59 +0800 Subject: [PATCH 31/87] =?UTF-8?q?fix:=20=E6=AD=A3=E7=A1=AE=E9=AA=8C?= =?UTF-8?q?=E8=AF=81=20LiteLLM=20=E8=B7=A8=E5=89=AF=E6=9C=AC=E9=A2=84?= =?UTF-8?q?=E7=AE=97=E9=99=90=E5=88=B6?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/demo/scripts/smoke-baseline.sh | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh index 43341a5..90e8962 100755 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -638,7 +638,10 @@ if [[ "$MODE" == "ha" ]]; then # to the other replica, so a 429 proves the counter is not replica-local. write_private_value "$tmpdir/enforcement-key-alias" "p1-smoke-budget-$suffix" make_key_payload "$tmpdir/enforcement-key-alias" "$tmpdir/enforcement-key-create.json" - jq '.key_max_budget = 0.000001' "$tmpdir/enforcement-key-create.json" > "$tmpdir/enforcement-key-limited.json" + # LiteLLM 1.97.0 maps the public /key/generate `max_budget` field to its + # persisted per-key `key_max_budget`; sending the internal field directly + # is ignored by the request model and would create a false HA pass. + jq '.max_budget = 0.000001' "$tmpdir/enforcement-key-create.json" > "$tmpdir/enforcement-key-limited.json" chmod 600 "$tmpdir/enforcement-key-limited.json" request_admin POST /key/generate "$tmpdir/enforcement-key-limited.json" "$tmpdir/enforcement-key.json" make_key_header "$tmpdir/enforcement-key.json" "$enforcement_key_file" "$enforcement_headers" From e7eac4e364de01dca1a283e18d76037edb8d1a04 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 14:45:13 +0800 Subject: [PATCH 32/87] =?UTF-8?q?fix:=20=E9=AA=8C=E8=AF=81=20LiteLLM=20?= =?UTF-8?q?=E8=B7=A8=E5=89=AF=E6=9C=AC=20TPM=20=E9=97=A8=E9=99=90?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/demo/scripts/smoke-baseline.sh | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh index 90e8962..87c6a67 100755 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -638,10 +638,10 @@ if [[ "$MODE" == "ha" ]]; then # to the other replica, so a 429 proves the counter is not replica-local. write_private_value "$tmpdir/enforcement-key-alias" "p1-smoke-budget-$suffix" make_key_payload "$tmpdir/enforcement-key-alias" "$tmpdir/enforcement-key-create.json" - # LiteLLM 1.97.0 maps the public /key/generate `max_budget` field to its - # persisted per-key `key_max_budget`; sending the internal field directly - # is ignored by the request model and would create a false HA pass. - jq '.max_budget = 0.000001' "$tmpdir/enforcement-key-create.json" > "$tmpdir/enforcement-key-limited.json" + # Use a separate shared TPM gate. It is enforced by the Redis-backed limiter + # before the second replica accepts a request, unlike asynchronous SpendLog + # persistence which cannot be used as an admission-control proof. + jq '.tpm_limit = 64' "$tmpdir/enforcement-key-create.json" > "$tmpdir/enforcement-key-limited.json" chmod 600 "$tmpdir/enforcement-key-limited.json" request_admin POST /key/generate "$tmpdir/enforcement-key-limited.json" "$tmpdir/enforcement-key.json" make_key_header "$tmpdir/enforcement-key.json" "$enforcement_key_file" "$enforcement_headers" @@ -649,9 +649,9 @@ if [[ "$MODE" == "ha" ]]; then make_key_action_payload delete "$enforcement_key_file" "$tmpdir/enforcement-key-cleanup.json" request_data_post "$BASE_URL" "$enforcement_headers" /v1/chat/completions "$tmpdir/chat-request.json" "$tmpdir/budget-first.json" budget_code="$(curl --silent --show-error --max-time 30 --request POST "$PEER_URL/v1/chat/completions" --header "@$enforcement_headers" --data-binary "@$tmpdir/chat-request.json" --output "$tmpdir/budget-second.json" --write-out '%{http_code}' || true)" - [[ "$budget_code" == "429" ]] || { echo "shared budget/Spend limit was bypassed by peer: http=$budget_code" >&2; exit 1; } + [[ "$budget_code" == "429" ]] || { echo "shared TPM limit was bypassed by peer: http=$budget_code" >&2; exit 1; } shared_enforcement_result="passed" - echo "PASS HA shared budget/Spend enforcement: peer rejected post-spend request with 429." + echo "PASS HA shared TPM enforcement: peer rejected the second request with 429." fi make_key_action_payload block "$block_key_file" "$tmpdir/block-key.json" From 84108bada896674439fcc5d875e340a068efddb3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 15:47:05 +0800 Subject: [PATCH 33/87] =?UTF-8?q?fix:=20=E5=BC=BA=E5=8C=96=20LiteLLM=20P1?= =?UTF-8?q?=20=E9=AA=8C=E8=AF=81=E9=97=A8=E7=A6=81?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../scripts/aggregate-verification-summary.sh | 34 ++++++----- docker_litellm/demo/scripts/run-migration.sh | 31 +++++----- docker_litellm/demo/scripts/smoke-baseline.sh | 48 +++++++++++---- .../demo/scripts/smoke-redis-recovery.sh | 59 ++++++++++--------- .../demo/scripts/test-verification-gates.sh | 31 ++++++++++ .../demo/scripts/verification-lib.sh | 33 +++++++++++ docker_litellm/demo/scripts/verify-p1.sh | 21 +++++++ 7 files changed, 186 insertions(+), 71 deletions(-) create mode 100755 docker_litellm/demo/scripts/test-verification-gates.sh create mode 100755 docker_litellm/demo/scripts/verification-lib.sh create mode 100755 docker_litellm/demo/scripts/verify-p1.sh diff --git a/docker_litellm/demo/scripts/aggregate-verification-summary.sh b/docker_litellm/demo/scripts/aggregate-verification-summary.sh index 7d890d8..47c1e15 100755 --- a/docker_litellm/demo/scripts/aggregate-verification-summary.sh +++ b/docker_litellm/demo/scripts/aggregate-verification-summary.sh @@ -5,13 +5,16 @@ set -euo pipefail script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" demo_dir="$(cd "${script_dir}/.." && pwd)" -artifacts_dir="${demo_dir}/artifacts" +artifacts_dir="${LITELLM_ARTIFACTS_DIR:-${demo_dir}/artifacts}" single_report="${artifacts_dir}/p1-single-summary.json" ha_report="${artifacts_dir}/p1-ha-summary.json" redis_report="${artifacts_dir}/p1-redis-recovery.json" migration_report="${artifacts_dir}/p1-migration-summary.json" output="${LITELLM_AGGREGATE_SUMMARY_FILE:-${artifacts_dir}/p1-final-summary.json}" commit="$(git -C "$demo_dir/../.." rev-parse HEAD)" +run_id="${VERIFICATION_RUN_ID:-}" +[[ "$run_id" =~ ^p1-[a-f0-9]{32}$ ]] || { echo "missing verification run id" >&2; exit 2; } +rm -f "$output" for report in "$single_report" "$ha_report" "$redis_report" "$migration_report"; do [[ -f "$report" ]] || { echo "missing required report: $report" >&2; exit 2; } @@ -19,35 +22,36 @@ done # Each input must be an independently successful and fully redacted result of # this exact checkout. jq -e performs the gate before the final report exists. -jq -e --arg commit "$commit" ' +jq -e --arg commit "$commit" --arg run_id "$run_id" ' .mode == "migration" and .result == "passed" and .phase == "completed" and - .commit == $commit and (.image_id | type == "string" and length > 0) and + .verification_run_id == $run_id and .commit == $commit and (.image_id | type == "string" and length > 0) and .content_redacted == true and .proxy_replicas_started == false ' "$migration_report" >/dev/null -jq -e --arg commit "$commit" ' +jq -e --arg commit "$commit" --arg run_id "$run_id" ' .mode == "single" and .result == "passed" and .phase == "completed" and - .commit == $commit and (.image_id | type == "string" and length > 0) and + .verification_run_id == $run_id and .commit == $commit and (.image_id | type == "string" and length > 0) and .content_redacted == true and .migration == "passed" and .chat == "passed" and .stream == "passed" and .tool == "passed" and .usage == "passed" and .block == "passed" and .delete == "passed" and - .cleanup == "passed" and .security_scan == "passed" + .cleanup == "passed" and .security_scan == "passed" and .content_logging_scan == "passed" ' "$single_report" >/dev/null -jq -e --arg commit "$commit" ' +jq -e --arg commit "$commit" --arg run_id "$run_id" ' .mode == "ha" and .result == "passed" and .phase == "completed" and - .commit == $commit and (.image_id | type == "string" and length > 0) and + .verification_run_id == $run_id and .commit == $commit and (.image_id | type == "string" and length > 0) and .content_redacted == true and .migration == "passed" and .chat == "passed" and .stream == "passed" and .tool == "passed" and .usage == "passed" and .block == "passed" and .delete == "passed" and - .shared_rpm_limit == "passed" and .shared_enforcement == "passed" and - .shared_spend_counter == "passed" and .idempotency_recovery == "passed" and - .cleanup == "passed" and .security_scan == "passed" + .shared_rpm_limit == "passed" and .shared_tpm_limit == "passed" and + .shared_spend_log_visibility == "passed" and .idempotency_recovery == "passed" and + .limiter_source == "litellm_proxy" and .cleanup == "passed" and .security_scan == "passed" and + .content_logging_scan == "passed" ' "$ha_report" >/dev/null -jq -e --arg commit "$commit" ' +jq -e --arg commit "$commit" --arg run_id "$run_id" ' .mode == "ha" and .result == "passed" and .phase == "completed" and - .commit == $commit and (.image_id | type == "string" and length > 0) and + .verification_run_id == $run_id and .commit == $commit and (.image_id | type == "string" and length > 0) and .redis_recovery == "passed" and .content_redacted == true and .security_scan == "passed" ' "$redis_report" >/dev/null @@ -61,11 +65,11 @@ umask 077 mkdir -p "$(dirname "$output")" chmod 700 "$(dirname "$output")" jq -n \ - --arg commit "$commit" --arg image_id "$image_id" \ + --arg run_id "$run_id" --arg commit "$commit" --arg image_id "$image_id" \ --arg generated_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ --slurpfile migration "$migration_report" --slurpfile single "$single_report" \ --slurpfile ha "$ha_report" --slurpfile redis "$redis_report" \ - '{commit:$commit,image_id:$image_id,generated_at:$generated_at,result:"passed",phase:"completed",migration:$migration[0],single:$single[0],ha:$ha[0],redis_recovery:$redis[0],content_redacted:true,local_only:true}' \ + '{verification_run_id:$run_id,commit:$commit,image_id:$image_id,generated_at:$generated_at,result:"passed",phase:"completed",migration:$migration[0],single:$single[0],ha:$ha[0],redis_recovery:$redis[0],content_redacted:true,local_only:true}' \ > "$output" chmod 600 "$output" echo "PASS aggregate summary: $output" diff --git a/docker_litellm/demo/scripts/run-migration.sh b/docker_litellm/demo/scripts/run-migration.sh index b24f8f5..18625a9 100755 --- a/docker_litellm/demo/scripts/run-migration.sh +++ b/docker_litellm/demo/scripts/run-migration.sh @@ -5,25 +5,17 @@ set -euo pipefail script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" demo_dir="$(cd "${script_dir}/.." && pwd)" +source "${script_dir}/verification-lib.sh" env_file="${LITELLM_SMOKE_ENV_FILE:-${demo_dir}/.env}" compose=(docker compose --env-file "$env_file" -f "${demo_dir}/docker-compose.litellm.yml") summary_file="${LITELLM_MIGRATION_SUMMARY_FILE:-${demo_dir}/artifacts/p1-migration-summary.json}" -[[ -f "$env_file" ]] || { echo "missing ignored local environment file" >&2; exit 2; } - -# Deliberately load only the non-secret image reference into this shell. The -# Compose invocation receives the ignored env file itself; no value is echoed. -# shellcheck disable=SC1090 -source "$env_file" -: "${LITELLM_IMAGE:?missing LITELLM_IMAGE in ignored local environment file}" -image_ref="$LITELLM_IMAGE" -export -n LITELLM_IMAGE LITELLM_MASTER_KEY POSTGRES_PASSWORD REDIS_PASSWORD \ - UPSTREAM_API_KEY UPSTREAM_BASE_URL UPSTREAM_MODEL 2>/dev/null || true -unset LITELLM_MASTER_KEY POSTGRES_PASSWORD REDIS_PASSWORD UPSTREAM_API_KEY \ - UPSTREAM_BASE_URL UPSTREAM_MODEL - result="failed" phase="initializing" +image_ref="" +tmpdir="" +verification_run_id="${VERIFICATION_RUN_ID:-standalone}" +verification_invalidate_report "$summary_file" cleanup() { local exit_code=$? umask 077 @@ -33,13 +25,22 @@ cleanup() { --arg commit "$(git -C "$demo_dir/../.." rev-parse HEAD)" \ --arg image_id "$(docker image inspect "$image_ref" --format '{{.Id}}' 2>/dev/null || true)" \ --arg tested_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ - --arg result "$result" --arg phase "$phase" --argjson exit_code "$exit_code" \ - '{commit:$commit,image_id:$image_id,tested_at:$tested_at,mode:"migration",result:$result,phase:$phase,exit_code:$exit_code,proxy_replicas_started:false,content_redacted:true}' \ + --arg run_id "$verification_run_id" --arg result "$result" --arg phase "$phase" --argjson exit_code "$exit_code" \ + '{verification_run_id:$run_id,commit:$commit,image_id:$image_id,tested_at:$tested_at,mode:"migration",result:$result,phase:$phase,exit_code:$exit_code,proxy_replicas_started:false,content_redacted:true}' \ > "$summary_file" chmod 600 "$summary_file" + [[ -z "$tmpdir" ]] || rm -rf "$tmpdir" } trap cleanup EXIT +[[ -f "$env_file" ]] || { echo "missing ignored local environment file" >&2; exit 2; } +umask 077 +tmpdir="$(mktemp -d "${TMPDIR:-/tmp}/litellm-migration.XXXXXX")" +chmod 700 "$tmpdir" +verification_prepare_environment "$env_file" "${demo_dir}/docker-compose.litellm.yml" "$tmpdir" +image_ref="$(verification_env LITELLM_IMAGE)" +: "${image_ref:?missing LITELLM_IMAGE in effective Compose environment}" + # A cold Compose start previously raced PostgreSQL/Redis readiness. `--wait` # makes the dependency condition explicit before the one-shot job is run. phase="waiting_dependencies" diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh index 87c6a67..6e40ddc 100755 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -5,12 +5,14 @@ set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" DEMO_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)" +source "${SCRIPT_DIR}/verification-lib.sh" ENV_FILE="${LITELLM_SMOKE_ENV_FILE:-${DEMO_DIR}/.env}" export COMPOSE_PROJECT_NAME="${LITELLM_COMPOSE_PROJECT:-litellm-baseline}" MODE="single" SECURITY_CHECK=false REVOCATION_SLO_MS="${LITELLM_REVOCATION_SLO_MS:-30000}" SUMMARY_FILE="${LITELLM_SMOKE_SUMMARY_FILE:-}" +verification_run_id="${VERIFICATION_RUN_ID:-standalone}" block_elapsed_ms="" delete_elapsed_ms="" result="failed" @@ -21,11 +23,13 @@ usage_result="not_run" block_result="not_run" delete_result="not_run" shared_rpm_limit_result="not_applicable" -shared_enforcement_result="not_applicable" +shared_tpm_limit_result="not_applicable" shared_spend_counter_result="not_applicable" +limiter_source="not_applicable" idempotency_recovery_result="not_applicable" migration_result="not_run" security_scan_result="not_run" +content_logging_scan_result="not_run" cleanup_result="not_run" redis_container="" redis_network="" @@ -48,6 +52,7 @@ done if [[ -z "$SUMMARY_FILE" ]]; then SUMMARY_FILE="$DEMO_DIR/artifacts/p1-${MODE}-summary.json" fi +verification_invalidate_report "$SUMMARY_FILE" need() { command -v "$1" >/dev/null || { echo "required command missing: $1" >&2; exit 2; }; } need curl; need jq; need rg @@ -211,13 +216,20 @@ request_admin() { cleanup_request_admin() { local method="$1" path="$2" payload_file="$3" - local curl_args=(--silent --show-error --max-time 15 --request "$method" "$BASE_URL$path" --header "@$admin_headers" --output /dev/null) + local curl_args=(--silent --show-error --fail --max-time 15 --request "$method" "$BASE_URL$path" --header "@$admin_headers" --output /dev/null) if [[ -n "$payload_file" ]]; then curl_args+=(--data-binary "@$payload_file") fi curl "${curl_args[@]}" >/dev/null 2>&1 } +assert_test_resources_removed() { + local counts_file="$tmpdir/cleanup-counts.txt" counts + docker compose --env-file "$ENV_FILE" -f "$DEMO_DIR/docker-compose.litellm.yml" exec -T postgres sh -lc 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -At -v ON_ERROR_STOP=1 -c "SELECT (SELECT count(*) FROM \"LiteLLM_UserTable\" WHERE user_id LIKE '\''p1-smoke-user-%'\''), (SELECT count(*) FROM \"LiteLLM_CredentialsTable\" WHERE credential_name LIKE '\''p1-smoke-upstream-%'\''), (SELECT count(*) FROM \"LiteLLM_ProxyModelTable\" WHERE model_name LIKE '\''p1-smoke-model-%'\''), (SELECT count(*) FROM \"LiteLLM_VerificationToken\" WHERE key_alias LIKE '\''p1-smoke-%'\'');"' > "$counts_file" + counts="$(tr -d '[:space:]' < "$counts_file")" + [[ "$counts" == "0|0|0|0" ]] +} + cleanup() { local exit_code=$? local cleanup_ok=true @@ -246,6 +258,7 @@ cleanup() { if [[ -n "$test_user" ]]; then cleanup_request_admin POST /user/delete "$tmpdir/user-delete.json" || cleanup_ok=false fi + assert_test_resources_removed || cleanup_ok=false if [[ "$cleanup_ok" == true ]]; then cleanup_result="passed"; else cleanup_result="failed"; exit_code=1; fi unset master_key_file upstream_key_file upstream_base_file upstream_model_file upstream_provider_file rm -rf "$tmpdir" @@ -256,7 +269,7 @@ cleanup() { elif [[ "$result" != skipped ]]; then result="failed" fi - write_summary || true + write_summary exit "$exit_code" } trap cleanup EXIT @@ -307,9 +320,9 @@ assert_migration_evidence() { commit="$(git -C "$DEMO_DIR/../.." rev-parse HEAD)" image_id="$(docker image inspect "$image_ref" --format '{{.Id}}' 2>/dev/null || true)" [[ -f "$report" ]] || { echo "missing current migration report: $report" >&2; return 1; } - jq -e --arg commit "$commit" --arg image_id "$image_id" ' + jq -e --arg commit "$commit" --arg image_id "$image_id" --arg run_id "$verification_run_id" ' .mode == "migration" and .result == "passed" and .phase == "completed" and - .commit == $commit and .image_id == $image_id and .proxy_replicas_started == false and + .verification_run_id == $run_id and .commit == $commit and .image_id == $image_id and .proxy_replicas_started == false and .content_redacted == true ' "$report" >/dev/null } @@ -328,8 +341,10 @@ runtime_security_check() { ! rg -q -- '--requirepass[[:space:]]+[^[:space:]]+' "$process_file" && ! rg -q --file "$tmpdir/prompt-marker" "$logs_file" && ! rg -q --file "$tmpdir/tool-marker" "$logs_file" && + ! rg -q --file "$tmpdir/response-marker" "$logs_file" && ! rg -q --file "$tmpdir/prompt-marker" "$db_content_file" && ! rg -q --file "$tmpdir/tool-marker" "$db_content_file" && + ! rg -q --file "$tmpdir/response-marker" "$db_content_file" && ! git ls-files -z | xargs -0 rg -n --pcre2 '(?:sk-|Bearer[[:space:]]+)[A-Za-z0-9_-]{24,}' -- >/dev/null 2>&1 && [[ "$(stat -f '%Lp' "$admin_headers" 2>/dev/null || stat -c '%a' "$admin_headers")" == "600" ]] } @@ -423,6 +438,12 @@ wait_for_rejection() { done } +assert_proxy_limiter_response() { + local response_file="$1" header_file="$2" limit_kind="$3" + [[ "$2" == "429" ]] && + jq -e --arg kind "$limit_kind" '(.error // .detail // .message // "") | tostring | test("litellm|" + $kind + ".*(limit|rate)|rate.*" + $kind; "i")' "$response_file" >/dev/null +} + write_summary() { mkdir -p "$(dirname "$SUMMARY_FILE")" chmod 700 "$(dirname "$SUMMARY_FILE")" @@ -431,8 +452,8 @@ write_summary() { --arg image_id "$(docker image inspect "$image_ref" --format '{{.Id}}' 2>/dev/null || true)" \ --arg tested_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ --arg mode "$MODE" --arg block_ms "$block_elapsed_ms" --arg delete_ms "$delete_elapsed_ms" --arg phase "$smoke_phase" --arg exit_code "$smoke_exit_code" \ - --arg result "$result" --arg migration "$migration_result" --arg chat "$chat_result" --arg stream "$stream_result" --arg tool "$tool_result" --arg usage "$usage_result" --arg block "$block_result" --arg delete "$delete_result" --arg shared_rpm "$shared_rpm_limit_result" --arg shared_enforcement "$shared_enforcement_result" --arg shared_spend "$shared_spend_counter_result" --arg idempotency "$idempotency_recovery_result" --arg cleanup "$cleanup_result" --arg security "$security_scan_result" \ - '{commit:$commit,image_id:$image_id,tested_at:$tested_at,mode:$mode,result:$result,phase:$phase,exit_code:($exit_code|tonumber?),migration:$migration,chat:$chat,stream:$stream,tool:$tool,usage:$usage,block:$block,delete:$delete,shared_rpm_limit:$shared_rpm,shared_enforcement:$shared_enforcement,shared_spend_counter:$shared_spend,idempotency_recovery:$idempotency,block_elapsed_ms:($block_ms|tonumber?),delete_elapsed_ms:($delete_ms|tonumber?),cleanup:$cleanup,security_scan:$security,content_redacted:true}' \ + --arg run_id "$verification_run_id" --arg result "$result" --arg migration "$migration_result" --arg chat "$chat_result" --arg stream "$stream_result" --arg tool "$tool_result" --arg usage "$usage_result" --arg block "$block_result" --arg delete "$delete_result" --arg shared_rpm "$shared_rpm_limit_result" --arg shared_tpm "$shared_tpm_limit_result" --arg shared_spend "$shared_spend_counter_result" --arg limiter_source "$limiter_source" --arg idempotency "$idempotency_recovery_result" --arg cleanup "$cleanup_result" --arg security "$security_scan_result" --arg content_logging "$content_logging_scan_result" \ + '{verification_run_id:$run_id,commit:$commit,image_id:$image_id,tested_at:$tested_at,mode:$mode,result:$result,phase:$phase,exit_code:($exit_code|tonumber?),migration:$migration,chat:$chat,stream:$stream,tool:$tool,usage:$usage,block:$block,delete:$delete,shared_rpm_limit:$shared_rpm,shared_tpm_limit:$shared_tpm,shared_spend_log_visibility:$shared_spend,limiter_source:$limiter_source,idempotency_recovery:$idempotency,block_elapsed_ms:($block_ms|tonumber?),delete_elapsed_ms:($delete_ms|tonumber?),cleanup:$cleanup,security_scan:$security,content_logging_scan:$content_logging,content_redacted:true}' \ > "$SUMMARY_FILE" chmod 600 "$SUMMARY_FILE" echo "PASS summary: $SUMMARY_FILE" @@ -483,6 +504,7 @@ write_private_value "$tmpdir/credential-name" "$credential_name" write_private_value "$tmpdir/model-name" "$model_name" write_private_value "$tmpdir/prompt-marker" "p1-redaction-prompt-$suffix" write_private_value "$tmpdir/tool-marker" "p1-redaction-tool-$suffix" +write_private_value "$tmpdir/response-marker" "p1-redaction-response-$suffix" # This identifier is intentionally non-sensitive. LiteLLM 1.97.0 may emit a # parser warning with a function name when an upstream tool call is malformed; # the sensitive marker stays in the tool schema body, which the scan verifies @@ -580,10 +602,10 @@ if [[ "$MODE" == "ha" ]]; then echo "PASS HA pre-revocation: second replica accepted the virtual key." fi -jq -n --rawfile model "$tmpdir/model-name" --rawfile prompt "$tmpdir/prompt-marker" '{model: ($model | rtrimstr("\n")), messages: [{role: "user", content: ($prompt | rtrimstr("\n"))}], max_tokens: 16}' > "$tmpdir/chat-request.json" +jq -n --rawfile model "$tmpdir/model-name" --rawfile prompt "$tmpdir/prompt-marker" --rawfile response_marker "$tmpdir/response-marker" '{model: ($model | rtrimstr("\n")), messages: [{role: "user", content: (($prompt | rtrimstr("\n")) + " Return exactly this marker: " + ($response_marker | rtrimstr("\n")))}], max_tokens: 32}' > "$tmpdir/chat-request.json" chmod 600 "$tmpdir/chat-request.json" request_data_post "$BASE_URL" "$block_headers" /v1/chat/completions "$tmpdir/chat-request.json" "$tmpdir/chat.json" -jq -e '.choices[0].message.content | type == "string"' "$tmpdir/chat.json" >/dev/null +jq -e --rawfile response_marker "$tmpdir/response-marker" '.choices[0].message.content | type == "string" and contains($response_marker | rtrimstr("\n"))' "$tmpdir/chat.json" >/dev/null chat_result="passed" if [[ "$MODE" == "ha" ]]; then @@ -630,8 +652,9 @@ if [[ "$MODE" == "ha" ]]; then make_key_action_payload delete "$rate_key_file" "$tmpdir/rate-key-cleanup.json" request_data_post "$BASE_URL" "$rate_headers" /v1/chat/completions "$tmpdir/chat-request.json" "$tmpdir/rate-first.json" rate_code="$(curl --silent --show-error --max-time 30 --request POST "$PEER_URL/v1/chat/completions" --header "@$rate_headers" --data-binary "@$tmpdir/chat-request.json" --output "$tmpdir/rate-second.json" --write-out '%{http_code}' || true)" - [[ "$rate_code" == "429" ]] || { echo "shared RPM limit was bypassed by peer: http=$rate_code" >&2; exit 1; } + assert_proxy_limiter_response "$tmpdir/rate-second.json" "$rate_code" rpm || { echo "shared RPM limiter was not a LiteLLM proxy 429" >&2; exit 1; } shared_rpm_limit_result="passed" + limiter_source="litellm_proxy" echo "PASS HA shared RPM: peer rejected the second request with 429." # Budget is enforced after the first real request. The second request goes @@ -649,8 +672,8 @@ if [[ "$MODE" == "ha" ]]; then make_key_action_payload delete "$enforcement_key_file" "$tmpdir/enforcement-key-cleanup.json" request_data_post "$BASE_URL" "$enforcement_headers" /v1/chat/completions "$tmpdir/chat-request.json" "$tmpdir/budget-first.json" budget_code="$(curl --silent --show-error --max-time 30 --request POST "$PEER_URL/v1/chat/completions" --header "@$enforcement_headers" --data-binary "@$tmpdir/chat-request.json" --output "$tmpdir/budget-second.json" --write-out '%{http_code}' || true)" - [[ "$budget_code" == "429" ]] || { echo "shared TPM limit was bypassed by peer: http=$budget_code" >&2; exit 1; } - shared_enforcement_result="passed" + assert_proxy_limiter_response "$tmpdir/budget-second.json" "$budget_code" tpm || { echo "shared TPM limiter was not a LiteLLM proxy 429" >&2; exit 1; } + shared_tpm_limit_result="passed" echo "PASS HA shared TPM enforcement: peer rejected the second request with 429." fi @@ -679,5 +702,6 @@ delete_result="passed" runtime_security_check security_scan_result="passed" +content_logging_scan_result="passed" smoke_phase="completed" echo "PASS complete: user/credential/model/key cleanup, explicit GET models, chat/stream/tool, token-bearing spend, and independent block/delete propagation." diff --git a/docker_litellm/demo/scripts/smoke-redis-recovery.sh b/docker_litellm/demo/scripts/smoke-redis-recovery.sh index 19b3d07..a74e7aa 100755 --- a/docker_litellm/demo/scripts/smoke-redis-recovery.sh +++ b/docker_litellm/demo/scripts/smoke-redis-recovery.sh @@ -4,8 +4,10 @@ set -euo pipefail script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" demo_dir="$(cd "${script_dir}/.." && pwd)" +source "${script_dir}/verification-lib.sh" env_file="${LITELLM_SMOKE_ENV_FILE:-${demo_dir}/.env}" summary_file="${LITELLM_REDIS_SUMMARY_FILE:-${demo_dir}/artifacts/p1-redis-recovery.json}" +verification_run_id="${VERIFICATION_RUN_ID:-standalone}" recovery_timeout="${REDIS_CIRCUIT_BREAKER_RECOVERY_TIMEOUT:-5}" container="" network="" @@ -14,32 +16,9 @@ result="failed" security_scan="not_run" post_recovery_call="not_run" tmpdir="" +image_ref="" -[[ -f "$env_file" ]] || { echo "missing ignored local environment file" >&2; exit 2; } -# shellcheck disable=SC1090 -source "$env_file" -: "${LITELLM_MASTER_KEY:?missing LITELLM_MASTER_KEY in ignored local environment file}" -: "${LITELLM_IMAGE:?missing LITELLM_IMAGE in ignored local environment file}" -image_ref="$LITELLM_IMAGE" -# Keep the recovery proof aligned with the same optional host-port overrides -# that Compose uses for the two proxy replicas. These are non-secret routing -# values; credentials remain in the private header file below. -publish_host="${LITELLM_PUBLISH_HOST:-127.0.0.1}" -litellm_1_port="${LITELLM_1_PORT:-4000}" -litellm_2_port="${LITELLM_2_PORT:-4001}" -umask 077 -tmpdir="$(mktemp -d "${TMPDIR:-/tmp}/litellm-redis-recovery.XXXXXX")" -chmod 700 "$tmpdir" -master_file="$tmpdir/master-key" -headers_file="$tmpdir/admin.headers" -printf '%s' "$LITELLM_MASTER_KEY" > "$master_file" -chmod 600 "$master_file" -{ printf 'Authorization: Bearer '; tr -d '\r\n' < "$master_file"; printf '\n'; } > "$headers_file" -chmod 600 "$headers_file" -export -n LITELLM_MASTER_KEY POSTGRES_PASSWORD REDIS_PASSWORD UPSTREAM_API_KEY \ - UPSTREAM_BASE_URL UPSTREAM_MODEL 2>/dev/null || true -unset LITELLM_MASTER_KEY POSTGRES_PASSWORD REDIS_PASSWORD UPSTREAM_API_KEY \ - UPSTREAM_BASE_URL UPSTREAM_MODEL +verification_invalidate_report "$summary_file" write_summary() { umask 077 @@ -49,9 +28,9 @@ write_summary() { --arg commit "$(git -C "$demo_dir/../.." rev-parse HEAD)" \ --arg image_id "$(docker image inspect "$image_ref" --format '{{.Id}}' 2>/dev/null || true)" \ --arg tested_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ - --arg result "$result" --arg phase "$phase" --arg security_scan "$security_scan" \ + --arg run_id "$verification_run_id" --arg result "$result" --arg phase "$phase" --arg security_scan "$security_scan" \ --arg post_recovery_call "$post_recovery_call" \ - '{commit:$commit,image_id:$image_id,tested_at:$tested_at,mode:"ha",result:$result,phase:$phase,redis_recovery:$result,post_recovery_call:$post_recovery_call,security_scan:$security_scan,content_redacted:true}' \ + '{verification_run_id:$run_id,commit:$commit,image_id:$image_id,tested_at:$tested_at,mode:"ha",result:$result,phase:$phase,redis_recovery:$result,post_recovery_call:$post_recovery_call,security_scan:$security_scan,content_redacted:true}' \ > "$summary_file" chmod 600 "$summary_file" } @@ -62,13 +41,35 @@ cleanup() { if [[ -n "$container" && -n "$network" ]]; then docker network connect --alias redis "$network" "$container" >/dev/null 2>&1 || true fi - rm -rf "$tmpdir" - [[ ! -e "$tmpdir" ]] || { result="failed"; phase="temporary_cleanup_failed"; } if (( exit_code != 0 )); then result="failed"; fi write_summary + [[ -z "$tmpdir" ]] || rm -rf "$tmpdir" } trap cleanup EXIT +[[ -f "$env_file" ]] || { echo "missing ignored local environment file" >&2; exit 2; } +umask 077 +tmpdir="$(mktemp -d "${TMPDIR:-/tmp}/litellm-redis-recovery.XXXXXX")" +chmod 700 "$tmpdir" +verification_prepare_environment "$env_file" "${demo_dir}/docker-compose.litellm.yml" "$tmpdir" +master_key="$(verification_env LITELLM_MASTER_KEY)" +image_ref="$(verification_env LITELLM_IMAGE)" +: "${master_key:?missing LITELLM_MASTER_KEY in effective Compose environment}" +: "${image_ref:?missing LITELLM_IMAGE in effective Compose environment}" +# Keep the recovery proof aligned with the same optional host-port overrides +# that Compose uses for the two proxy replicas. These are non-secret routing +# values; credentials remain in the private header file below. +publish_host="$(verification_env LITELLM_PUBLISH_HOST)" +litellm_1_port="$(verification_env LITELLM_1_PORT)" +litellm_2_port="$(verification_env LITELLM_2_PORT)" +master_file="$tmpdir/master-key" +headers_file="$tmpdir/admin.headers" +printf '%s' "$master_key" > "$master_file" +chmod 600 "$master_file" +{ printf 'Authorization: Bearer '; tr -d '\r\n' < "$master_file"; printf '\n'; } > "$headers_file" +chmod 600 "$headers_file" +unset master_key + probe() { docker exec "$1" python3 -c 'import redis; p=open("/run/secrets/redis_password").read().strip(); assert redis.Redis(host="redis", password=p, socket_connect_timeout=2, socket_timeout=2).ping()' } diff --git a/docker_litellm/demo/scripts/test-verification-gates.sh b/docker_litellm/demo/scripts/test-verification-gates.sh new file mode 100755 index 0000000..c42dc1b --- /dev/null +++ b/docker_litellm/demo/scripts/test-verification-gates.sh @@ -0,0 +1,31 @@ +#!/usr/bin/env bash +# Reproducible negative checks: neither stale PASS reports nor failed cleanup +# may be accepted by the aggregate gate. +set -euo pipefail + +script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +demo_dir="$(cd "${script_dir}/.." && pwd)" +tmpdir="$(mktemp -d "${TMPDIR:-/tmp}/litellm-gates.XXXXXX")" +chmod 700 "$tmpdir" +trap 'rm -rf "$tmpdir"' EXIT +run_id="p1-$(python3 -c 'import secrets; print(secrets.token_hex(16))')" + +# A deliberately stale-but-well-shaped set must be rejected when the current +# run id differs. No service or .env is read by this test. +for name in p1-migration-summary p1-single-summary p1-ha-summary p1-redis-recovery; do + jq -n --arg stale 'p1-00000000000000000000000000000000' \ + '{verification_run_id:$stale,commit:"stale",image_id:"stale",mode:"single",result:"passed",phase:"completed",content_redacted:true}' > "$tmpdir/${name}.json" +done +if VERIFICATION_RUN_ID="$run_id" LITELLM_ARTIFACTS_DIR="$tmpdir" LITELLM_AGGREGATE_SUMMARY_FILE="$tmpdir/final.json" \ + "$script_dir/aggregate-verification-summary.sh" >/dev/null 2>&1; then + echo "aggregate accepted stale PASS reports" >&2 + exit 1 +fi + +# Cleanup failures must never be normalized to PASS. This checks the curl +# transport invariant directly without sending a request. +rg -q 'cleanup_request_admin.*\(\)' "$script_dir/smoke-baseline.sh" +rg -q -- 'curl_args=\(--silent --show-error --fail' "$script_dir/smoke-baseline.sh" +! rg -n -- 'source "\$env_file"|source "\$\{env_file\}"' "$script_dir/run-migration.sh" "$script_dir/smoke-redis-recovery.sh" +rg -q 'config --environment > "\$verification_environment_file"' "$script_dir/verification-lib.sh" +echo "PASS verification gates: stale reports and cleanup HTTP failure cannot pass." diff --git a/docker_litellm/demo/scripts/verification-lib.sh b/docker_litellm/demo/scripts/verification-lib.sh new file mode 100755 index 0000000..9b88243 --- /dev/null +++ b/docker_litellm/demo/scripts/verification-lib.sh @@ -0,0 +1,33 @@ +#!/usr/bin/env bash +# Shared fail-closed helpers for P1 verification producers. Never source .env. + +verification_prepare_environment() { + local env_file="$1" compose_file="$2" work_dir="$3" + umask 077 + verification_environment_file="$work_dir/compose.environment" + docker compose --env-file "$env_file" -f "$compose_file" config --environment > "$verification_environment_file" + chmod 600 "$verification_environment_file" +} + +verification_env() { + local name="$1" + awk -F= -v name="$name" '$1 == name {sub(/^[^=]*=/, ""); print; exit}' "$verification_environment_file" +} + +verification_invalidate_report() { + local report="$1" + mkdir -p "$(dirname "$report")" + chmod 700 "$(dirname "$report")" + rm -f "$report" +} + +verification_new_run_id() { + python3 -c 'import secrets; print("p1-" + secrets.token_hex(16))' +} + +verification_assert_run_id() { + [[ "${VERIFICATION_RUN_ID:-}" =~ ^p1-[a-f0-9]{32}$ ]] || { + echo "VERIFICATION_RUN_ID must be generated by verify-p1.sh" >&2 + return 2 + } +} diff --git a/docker_litellm/demo/scripts/verify-p1.sh b/docker_litellm/demo/scripts/verify-p1.sh new file mode 100755 index 0000000..0d66dee --- /dev/null +++ b/docker_litellm/demo/scripts/verify-p1.sh @@ -0,0 +1,21 @@ +#!/usr/bin/env bash +# Execute one complete, non-reusable P1 verification run. +set -euo pipefail + +script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +demo_dir="$(cd "${script_dir}/.." && pwd)" +run_id="${VERIFICATION_RUN_ID:-$(python3 -c 'import secrets; print("p1-" + secrets.token_hex(16))')}" +export VERIFICATION_RUN_ID="$run_id" +compose=(docker compose --env-file "${demo_dir}/.env" -f "${demo_dir}/docker-compose.litellm.yml") +cleanup_stack() { "${compose[@]}" --profile single --profile ha --profile migrate down >/dev/null 2>&1 || true; } +trap cleanup_stack EXIT + +cleanup_stack +"${script_dir}/run-migration.sh" +"${script_dir}/run-migration.sh" +"${compose[@]}" --profile single up -d --wait postgres redis litellm-1 +"${script_dir}/smoke-baseline.sh" --mode single +"${compose[@]}" --profile ha up -d --wait postgres redis litellm-1 litellm-2 +"${script_dir}/smoke-baseline.sh" --mode ha +"${script_dir}/smoke-redis-recovery.sh" +"${script_dir}/aggregate-verification-summary.sh" From 00c78995c6598d990cd4e2e27f86a4fef9cb741d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 15:49:21 +0800 Subject: [PATCH 34/87] =?UTF-8?q?fix:=20=E5=BC=BA=E5=88=B6=20P1=20?= =?UTF-8?q?=E6=B8=85=E7=90=86=E4=B8=8E=E6=8A=A5=E5=91=8A=E5=A4=B1=E8=B4=A5?= =?UTF-8?q?=E8=AF=AD=E4=B9=89?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/demo/scripts/run-migration.sh | 5 +++-- docker_litellm/demo/scripts/smoke-baseline.sh | 2 +- docker_litellm/demo/scripts/smoke-redis-recovery.sh | 3 ++- 3 files changed, 6 insertions(+), 4 deletions(-) diff --git a/docker_litellm/demo/scripts/run-migration.sh b/docker_litellm/demo/scripts/run-migration.sh index 18625a9..0cf1eb0 100755 --- a/docker_litellm/demo/scripts/run-migration.sh +++ b/docker_litellm/demo/scripts/run-migration.sh @@ -27,9 +27,10 @@ cleanup() { --arg tested_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ --arg run_id "$verification_run_id" --arg result "$result" --arg phase "$phase" --argjson exit_code "$exit_code" \ '{verification_run_id:$run_id,commit:$commit,image_id:$image_id,tested_at:$tested_at,mode:"migration",result:$result,phase:$phase,exit_code:$exit_code,proxy_replicas_started:false,content_redacted:true}' \ - > "$summary_file" - chmod 600 "$summary_file" + > "$summary_file" || exit_code=1 + chmod 600 "$summary_file" || exit_code=1 [[ -z "$tmpdir" ]] || rm -rf "$tmpdir" + return "$exit_code" } trap cleanup EXIT diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh index 6e40ddc..3201096 100755 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -269,7 +269,7 @@ cleanup() { elif [[ "$result" != skipped ]]; then result="failed" fi - write_summary + write_summary || exit_code=1 exit "$exit_code" } trap cleanup EXIT diff --git a/docker_litellm/demo/scripts/smoke-redis-recovery.sh b/docker_litellm/demo/scripts/smoke-redis-recovery.sh index a74e7aa..413b901 100755 --- a/docker_litellm/demo/scripts/smoke-redis-recovery.sh +++ b/docker_litellm/demo/scripts/smoke-redis-recovery.sh @@ -42,8 +42,9 @@ cleanup() { docker network connect --alias redis "$network" "$container" >/dev/null 2>&1 || true fi if (( exit_code != 0 )); then result="failed"; fi - write_summary + write_summary || exit_code=1 [[ -z "$tmpdir" ]] || rm -rf "$tmpdir" + return "$exit_code" } trap cleanup EXIT From 676e5ad0097b9f161bf45b167ca653374b6134b5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 15:53:34 +0800 Subject: [PATCH 35/87] =?UTF-8?q?fix:=20=E6=8C=89=E9=AA=8C=E8=AF=81?= =?UTF-8?q?=E8=BF=90=E8=A1=8C=E9=9A=94=E7=A6=BB=20LiteLLM=20=E6=B5=8B?= =?UTF-8?q?=E8=AF=95=E8=B5=84=E6=BA=90?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/demo/scripts/smoke-baseline.sh | 21 ++++++++++--------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh index 3201096..d50c098 100755 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -225,7 +225,7 @@ cleanup_request_admin() { assert_test_resources_removed() { local counts_file="$tmpdir/cleanup-counts.txt" counts - docker compose --env-file "$ENV_FILE" -f "$DEMO_DIR/docker-compose.litellm.yml" exec -T postgres sh -lc 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -At -v ON_ERROR_STOP=1 -c "SELECT (SELECT count(*) FROM \"LiteLLM_UserTable\" WHERE user_id LIKE '\''p1-smoke-user-%'\''), (SELECT count(*) FROM \"LiteLLM_CredentialsTable\" WHERE credential_name LIKE '\''p1-smoke-upstream-%'\''), (SELECT count(*) FROM \"LiteLLM_ProxyModelTable\" WHERE model_name LIKE '\''p1-smoke-model-%'\''), (SELECT count(*) FROM \"LiteLLM_VerificationToken\" WHERE key_alias LIKE '\''p1-smoke-%'\'');"' > "$counts_file" + docker compose --env-file "$ENV_FILE" -f "$DEMO_DIR/docker-compose.litellm.yml" exec -T -e P1_CLEANUP_PREFIX="$test_prefix" postgres sh -lc 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -At -v ON_ERROR_STOP=1 -c "SELECT (SELECT count(*) FROM \"LiteLLM_UserTable\" WHERE user_id LIKE \$\$${P1_CLEANUP_PREFIX}%\$\$), (SELECT count(*) FROM \"LiteLLM_CredentialsTable\" WHERE credential_name LIKE \$\$${P1_CLEANUP_PREFIX}%\$\$), (SELECT count(*) FROM \"LiteLLM_ProxyModelTable\" WHERE model_name LIKE \$\$${P1_CLEANUP_PREFIX}%\$\$), (SELECT count(*) FROM \"LiteLLM_VerificationToken\" WHERE key_alias LIKE \$\$${P1_CLEANUP_PREFIX}%\$\$);"' > "$counts_file" counts="$(tr -d '[:space:]' < "$counts_file")" [[ "$counts" == "0|0|0|0" ]] } @@ -496,9 +496,10 @@ migration_result="passed" echo "PASS readiness: PostgreSQL connected; Redis independently reachable from LiteLLM replica(s)." suffix="$(date +%s)-$RANDOM" -test_user="p1-smoke-user-$suffix" -credential_name="p1-smoke-upstream-$suffix" -model_name="p1-smoke-model-$suffix" +test_prefix="p1-smoke-${verification_run_id#p1-}-$suffix" +test_user="${test_prefix}-user" +credential_name="${test_prefix}-upstream" +model_name="${test_prefix}-model" write_private_value "$tmpdir/test-user" "$test_user" write_private_value "$tmpdir/credential-name" "$credential_name" write_private_value "$tmpdir/model-name" "$model_name" @@ -572,7 +573,7 @@ write_private_value "$tmpdir/model-id" "$model_id" jq -n --rawfile id "$tmpdir/model-id" '{id: ($id | rtrimstr("\n"))}' > "$tmpdir/model-delete.json" chmod 600 "$tmpdir/model-delete.json" -write_private_value "$tmpdir/block-key-alias" "p1-smoke-block-$suffix" +write_private_value "$tmpdir/block-key-alias" "${test_prefix}-block" private_file "$block_key_file" python3 -c 'import secrets; print("sk-p1-" + secrets.token_urlsafe(32))' > "$block_key_file" assert_private_file "$block_key_file" @@ -642,7 +643,7 @@ smoke_phase="shared_limit_and_redis_recovery" if [[ "$MODE" == "ha" ]]; then # One request reaches replica 1; the same key must be RPM-limited on replica 2. - write_private_value "$tmpdir/rate-key-alias" "p1-smoke-rate-$suffix" + write_private_value "$tmpdir/rate-key-alias" "${test_prefix}-rate" make_key_payload "$tmpdir/rate-key-alias" "$tmpdir/rate-key-create.json" jq '.rpm_limit = 1' "$tmpdir/rate-key-create.json" > "$tmpdir/rate-key-limited.json" chmod 600 "$tmpdir/rate-key-limited.json" @@ -657,9 +658,9 @@ if [[ "$MODE" == "ha" ]]; then limiter_source="litellm_proxy" echo "PASS HA shared RPM: peer rejected the second request with 429." - # Budget is enforced after the first real request. The second request goes - # to the other replica, so a 429 proves the counter is not replica-local. - write_private_value "$tmpdir/enforcement-key-alias" "p1-smoke-budget-$suffix" + # The second TPM-limited request goes to the other replica, so its 429 proves + # that the Redis-backed limiter is not replica-local. + write_private_value "$tmpdir/enforcement-key-alias" "${test_prefix}-tpm" make_key_payload "$tmpdir/enforcement-key-alias" "$tmpdir/enforcement-key-create.json" # Use a separate shared TPM gate. It is enforced by the Redis-backed limiter # before the second replica accepts a request, unlike asynchronous SpendLog @@ -684,7 +685,7 @@ wait_for_rejection "$block_headers" block block_result="passed" # Delete is validated with a different, previously unblocked key. -write_private_value "$tmpdir/delete-key-alias" "p1-smoke-delete-$suffix" +write_private_value "$tmpdir/delete-key-alias" "${test_prefix}-delete" make_key_payload "$tmpdir/delete-key-alias" "$tmpdir/delete-key-create.json" request_admin POST /key/generate "$tmpdir/delete-key-create.json" "$tmpdir/delete-key.json" make_key_header "$tmpdir/delete-key.json" "$delete_key_file" "$delete_headers" From 02d7f23cce83ebbbeef24223d803e149323e79c4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 15:59:35 +0800 Subject: [PATCH 36/87] =?UTF-8?q?fix:=20=E5=8E=9F=E5=AD=90=E5=86=99?= =?UTF-8?q?=E5=85=A5=20LiteLLM=20smoke=20=E6=8A=A5=E5=91=8A?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/demo/scripts/smoke-baseline.sh | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh index d50c098..626d668 100755 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -445,8 +445,12 @@ assert_proxy_limiter_response() { } write_summary() { + local summary_tmp mkdir -p "$(dirname "$SUMMARY_FILE")" chmod 700 "$(dirname "$SUMMARY_FILE")" + summary_tmp="${SUMMARY_FILE}.tmp.$$" + : > "$summary_tmp" + chmod 600 "$summary_tmp" jq -n \ --arg commit "$(git -C "$DEMO_DIR/../.." rev-parse HEAD)" \ --arg image_id "$(docker image inspect "$image_ref" --format '{{.Id}}' 2>/dev/null || true)" \ @@ -454,8 +458,9 @@ write_summary() { --arg mode "$MODE" --arg block_ms "$block_elapsed_ms" --arg delete_ms "$delete_elapsed_ms" --arg phase "$smoke_phase" --arg exit_code "$smoke_exit_code" \ --arg run_id "$verification_run_id" --arg result "$result" --arg migration "$migration_result" --arg chat "$chat_result" --arg stream "$stream_result" --arg tool "$tool_result" --arg usage "$usage_result" --arg block "$block_result" --arg delete "$delete_result" --arg shared_rpm "$shared_rpm_limit_result" --arg shared_tpm "$shared_tpm_limit_result" --arg shared_spend "$shared_spend_counter_result" --arg limiter_source "$limiter_source" --arg idempotency "$idempotency_recovery_result" --arg cleanup "$cleanup_result" --arg security "$security_scan_result" --arg content_logging "$content_logging_scan_result" \ '{verification_run_id:$run_id,commit:$commit,image_id:$image_id,tested_at:$tested_at,mode:$mode,result:$result,phase:$phase,exit_code:($exit_code|tonumber?),migration:$migration,chat:$chat,stream:$stream,tool:$tool,usage:$usage,block:$block,delete:$delete,shared_rpm_limit:$shared_rpm,shared_tpm_limit:$shared_tpm,shared_spend_log_visibility:$shared_spend,limiter_source:$limiter_source,idempotency_recovery:$idempotency,block_elapsed_ms:($block_ms|tonumber?),delete_elapsed_ms:($delete_ms|tonumber?),cleanup:$cleanup,security_scan:$security,content_logging_scan:$content_logging,content_redacted:true}' \ - > "$SUMMARY_FILE" - chmod 600 "$SUMMARY_FILE" + > "$summary_tmp" + chmod 600 "$summary_tmp" + mv "$summary_tmp" "$SUMMARY_FILE" echo "PASS summary: $SUMMARY_FILE" } From c634188df71e32f3441f666500d634a46633b7de Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 16:05:08 +0800 Subject: [PATCH 37/87] =?UTF-8?q?fix:=20=E9=98=B2=E6=AD=A2=20P1=20?= =?UTF-8?q?=E9=AA=8C=E8=AF=81=E9=80=80=E5=87=BA=20trap=20=E9=87=8D?= =?UTF-8?q?=E5=85=A5?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/demo/scripts/run-migration.sh | 1 + docker_litellm/demo/scripts/smoke-baseline.sh | 1 + docker_litellm/demo/scripts/smoke-redis-recovery.sh | 1 + 3 files changed, 3 insertions(+) diff --git a/docker_litellm/demo/scripts/run-migration.sh b/docker_litellm/demo/scripts/run-migration.sh index 0cf1eb0..d6a1b9c 100755 --- a/docker_litellm/demo/scripts/run-migration.sh +++ b/docker_litellm/demo/scripts/run-migration.sh @@ -18,6 +18,7 @@ verification_run_id="${VERIFICATION_RUN_ID:-standalone}" verification_invalidate_report "$summary_file" cleanup() { local exit_code=$? + trap - EXIT umask 077 mkdir -p "$(dirname "$summary_file")" chmod 700 "$(dirname "$summary_file")" diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh index 626d668..001dab8 100755 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -233,6 +233,7 @@ assert_test_resources_removed() { cleanup() { local exit_code=$? local cleanup_ok=true + trap - EXIT set +e if [[ -n "$redis_container" && -n "$redis_network" ]]; then docker network connect --alias redis "$redis_network" "$redis_container" >/dev/null 2>&1 || true diff --git a/docker_litellm/demo/scripts/smoke-redis-recovery.sh b/docker_litellm/demo/scripts/smoke-redis-recovery.sh index 413b901..bc3e276 100755 --- a/docker_litellm/demo/scripts/smoke-redis-recovery.sh +++ b/docker_litellm/demo/scripts/smoke-redis-recovery.sh @@ -37,6 +37,7 @@ write_summary() { cleanup() { local exit_code=$? + trap - EXIT set +e if [[ -n "$container" && -n "$network" ]]; then docker network connect --alias redis "$network" "$container" >/dev/null 2>&1 || true From 3893f550b77b9c6e51bd02cab411f9b5808d0f27 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 16:14:54 +0800 Subject: [PATCH 38/87] =?UTF-8?q?fix:=20=E5=8A=A0=E5=9B=BA=20P1=20?= =?UTF-8?q?=E9=AA=8C=E6=94=B6=E6=8A=A5=E5=91=8A=E4=B8=8E=E6=B8=85=E7=90=86?= =?UTF-8?q?=E9=97=A8=E7=A6=81?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/README.md | 30 +++------ .../scripts/aggregate-verification-summary.sh | 2 +- docker_litellm/demo/scripts/run-migration.sh | 9 ++- docker_litellm/demo/scripts/smoke-baseline.sh | 65 ++++++++++++++++--- .../demo/scripts/smoke-redis-recovery.sh | 16 +++-- .../demo/scripts/test-verification-gates.sh | 37 ++++++++++- docker_litellm/demo/scripts/verify-p1.sh | 3 + 7 files changed, 119 insertions(+), 43 deletions(-) diff --git a/docker_litellm/README.md b/docker_litellm/README.md index 9f911ae..ae207ce 100644 --- a/docker_litellm/README.md +++ b/docker_litellm/README.md @@ -47,25 +47,13 @@ cp .env.example .env docker compose --env-file .env -f docker-compose.litellm.yml --profile single up -d ``` -迁移与代理启动刻意分离。每次部署先显式运行一次 migration-only job;该命令可安全重复执行。随后启动的代理副本只做 schema 检查,不会并发执行 Prisma migration: +迁移与代理启动刻意分离。标准真实验收由一个统一入口执行:它生成非敏感 `verification_run_id`,显式连续运行 migration(两次)、single、HA、Redis 恢复和严格聚合;任一步失败都会停止且使对应旧报告失效。 ```bash -./scripts/run-migration.sh -docker compose --env-file .env -f docker-compose.litellm.yml --profile single up -d --wait -./scripts/smoke-baseline.sh --mode single +./scripts/verify-p1.sh ``` -双副本测试使用同一 PostgreSQL 与 Redis,但有两个 HTTP 入口: - -```bash -./scripts/run-migration.sh -docker compose --env-file .env -f docker-compose.litellm.yml --profile ha up -d --wait -./scripts/smoke-baseline.sh --mode ha -./scripts/smoke-redis-recovery.sh -./scripts/aggregate-verification-summary.sh -``` - -HA 启动前同样先运行 `./scripts/run-migration.sh`,再执行上述命令。不要把 migration profile 与代理 profile 放入同一条 `up` 命令。 +手动分步排障时,必须为 migration、single、HA、Redis 与聚合导出同一合法 run ID;不要把 migration profile 与代理 profile 放入同一条 `up` 命令。 默认端口只发布在 `127.0.0.1`:副本 1 为 `4000`,副本 2 为 `4001`。PostgreSQL 与 Redis 不发布宿主机端口。停止测试不会删除卷;如需删除测试数据,先人工确认后使用 `docker compose ... down -v`。 @@ -91,20 +79,18 @@ P1 不把 LiteLLM User/Team 当作产品用户或 Workspace 的事实源;Shell ```bash cd docker_litellm/demo -./scripts/smoke-baseline.sh --mode single -./scripts/smoke-baseline.sh --mode ha -./scripts/smoke-redis-recovery.sh -./scripts/aggregate-verification-summary.sh +./scripts/verify-p1.sh ./scripts/smoke-baseline.sh --security-check +./scripts/test-verification-gates.sh ``` -在全新 checkout 中按以下顺序执行时,脚本自己生成而非复用历史文件:`p1-migration-summary.json`、`p1-single-summary.json`、`p1-ha-summary.json`、`p1-redis-recovery.json` 与最终 `p1-final-summary.json`(均在被忽略的 `artifacts/`)。聚合脚本只接受当前 `HEAD`、相同 image ID、`result=passed`、`phase=completed` 且脱敏的四份输入;任何缺失、失败、跳过或模式不符都会被拒绝。 +在全新 checkout 中按上述标准命令执行时,脚本自己生成而非复用历史文件:`p1-migration-summary.json`、`p1-single-summary.json`、`p1-ha-summary.json`、`p1-redis-recovery.json` 与最终 `p1-final-summary.json`(均在被忽略的 `artifacts/`)。聚合脚本只接受当前 `HEAD`、同一 `verification_run_id`、相同 image ID、`result=passed`、`phase=completed` 且脱敏的四份输入;任何缺失、失败、跳过或模式不符都会被拒绝。 -`--security-check` 不读取 `.env`、不启动服务也不发送上游请求;它拒绝 inline header、secret-bearing `jq --arg`、`set -x`、Compose 上游凭据注入和 Redis 密码命令行展开,并检查 Docker Secret、0600 临时文件与退出清理约束。 +`--security-check` 不读取 `.env`、不启动服务也不发送上游请求;它拒绝 inline header、secret-bearing `jq --arg`、`set -x`、Compose 上游凭据注入和 Redis 密码命令行展开,并检查 Docker Secret、0600 临时文件与退出清理约束。`test-verification-gates.sh` 验证历史 PASS 失效、前置失败报告与 dotenv 命令替换不执行;在已启动 single 栈中追加 `--with-running-stack` 会以真实 404 删除请求证明 cleanup 不会生成 PASS。 `smoke-redis-recovery.sh` 在已启动的 HA 栈中临时断开 Redis 网络端点,验证两个副本的有界认证探针均失败,再恢复 `redis` alias、等待 breaker 窗口并确认认证后的 `GET /v1/models` 恢复。它有独立的恢复 trap,不会让故障测试影响主 smoke 的资源清理。`aggregate-verification-summary.sh` 将 migration、single、HA 与 Redis 独立报告组合为不含密钥、密码、提示词和响应正文的最终 JSON 摘要。 -脚本在真实上游变量存在时执行:创建测试用户、保存测试上游凭证、以 `litellm_credential_name` 创建模型、由调用方生成稳定高熵 virtual key 并故意丢弃首次创建响应,再用该 key 的 0600 Authorization header 调用 `/key/info` 恢复、验证相同 key 重试被拒绝而不会创建第二资源;随后显式 `GET /v1/models`、chat、stream、tool call、token-bearing usage 查询、block,以及独立 key 的 delete。DeepSeek V4 使用 `deepseek/` 的原生 provider,避免被通用 OpenAI provider 丢弃 `thinking` 参数。HA 模式会先证明第二副本接受 key,再验证跨副本 RPM 和 post-spend budget 限制均返回 `429`,最后轮询两个副本直到都拒绝,并输出实际传播时间与 SLO。 +脚本在真实上游变量存在时执行:创建测试用户、保存测试上游凭证、以 `litellm_credential_name` 创建模型、由调用方生成稳定高熵 virtual key 并故意丢弃首次创建响应,再用该 key 的 0600 Authorization header 调用 `/v2/key/info` 恢复、验证相同 key 重试被拒绝而不会创建第二资源;随后显式 `GET /v1/models`、chat、stream、tool call、token-bearing usage 查询、block,以及独立 key 的 delete。DeepSeek V4 使用 `deepseek/` 的原生 provider,避免被通用 OpenAI provider 丢弃 `thinking` 参数。HA 模式会先证明第二副本接受 key,再验证跨副本 RPM 与 TPM 限制均返回由 LiteLLM Proxy limiter 产生的 `429`,最后轮询两个副本直到都拒绝,并输出实际传播时间与 SLO。 `LITELLM_MASTER_KEY`、上游 API key 与虚拟 key 不会作为 `curl`、`jq` 或其他子进程的命令参数传递。脚本以 `umask 077` 创建工作目录,所有 header、请求、响应和 key 文件均为 `0600`,退出时删除;创建的 user、credential、model 和两个测试 key 也会清理。上游凭据仅由 smoke 客户端读取,不会注入 LiteLLM Compose 容器。 diff --git a/docker_litellm/demo/scripts/aggregate-verification-summary.sh b/docker_litellm/demo/scripts/aggregate-verification-summary.sh index 47c1e15..6902f3c 100755 --- a/docker_litellm/demo/scripts/aggregate-verification-summary.sh +++ b/docker_litellm/demo/scripts/aggregate-verification-summary.sh @@ -13,8 +13,8 @@ migration_report="${artifacts_dir}/p1-migration-summary.json" output="${LITELLM_AGGREGATE_SUMMARY_FILE:-${artifacts_dir}/p1-final-summary.json}" commit="$(git -C "$demo_dir/../.." rev-parse HEAD)" run_id="${VERIFICATION_RUN_ID:-}" -[[ "$run_id" =~ ^p1-[a-f0-9]{32}$ ]] || { echo "missing verification run id" >&2; exit 2; } rm -f "$output" +[[ "$run_id" =~ ^p1-[a-f0-9]{32}$ ]] || { echo "missing verification run id" >&2; exit 2; } for report in "$single_report" "$ha_report" "$redis_report" "$migration_report"; do [[ -f "$report" ]] || { echo "missing required report: $report" >&2; exit 2; } diff --git a/docker_litellm/demo/scripts/run-migration.sh b/docker_litellm/demo/scripts/run-migration.sh index d6a1b9c..ee06e15 100755 --- a/docker_litellm/demo/scripts/run-migration.sh +++ b/docker_litellm/demo/scripts/run-migration.sh @@ -17,19 +17,22 @@ tmpdir="" verification_run_id="${VERIFICATION_RUN_ID:-standalone}" verification_invalidate_report "$summary_file" cleanup() { - local exit_code=$? + local exit_code=$? summary_tmp trap - EXIT umask 077 mkdir -p "$(dirname "$summary_file")" chmod 700 "$(dirname "$summary_file")" + summary_tmp="${summary_file}.tmp.$$" jq -n \ --arg commit "$(git -C "$demo_dir/../.." rev-parse HEAD)" \ --arg image_id "$(docker image inspect "$image_ref" --format '{{.Id}}' 2>/dev/null || true)" \ --arg tested_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ --arg run_id "$verification_run_id" --arg result "$result" --arg phase "$phase" --argjson exit_code "$exit_code" \ '{verification_run_id:$run_id,commit:$commit,image_id:$image_id,tested_at:$tested_at,mode:"migration",result:$result,phase:$phase,exit_code:$exit_code,proxy_replicas_started:false,content_redacted:true}' \ - > "$summary_file" || exit_code=1 - chmod 600 "$summary_file" || exit_code=1 + > "$summary_tmp" && chmod 600 "$summary_tmp" && mv "$summary_tmp" "$summary_file" || { + rm -f "$summary_tmp" + exit_code=1 + } [[ -z "$tmpdir" ]] || rm -rf "$tmpdir" return "$exit_code" } diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh index 001dab8..91a0c43 100755 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -10,6 +10,7 @@ ENV_FILE="${LITELLM_SMOKE_ENV_FILE:-${DEMO_DIR}/.env}" export COMPOSE_PROJECT_NAME="${LITELLM_COMPOSE_PROJECT:-litellm-baseline}" MODE="single" SECURITY_CHECK=false +CLEANUP_NEGATIVE_TEST=false REVOCATION_SLO_MS="${LITELLM_REVOCATION_SLO_MS:-30000}" SUMMARY_FILE="${LITELLM_SMOKE_SUMMARY_FILE:-}" verification_run_id="${VERIFICATION_RUN_ID:-standalone}" @@ -24,7 +25,7 @@ block_result="not_run" delete_result="not_run" shared_rpm_limit_result="not_applicable" shared_tpm_limit_result="not_applicable" -shared_spend_counter_result="not_applicable" +shared_spend_log_visibility_result="not_applicable" limiter_source="not_applicable" idempotency_recovery_result="not_applicable" migration_result="not_run" @@ -35,15 +36,20 @@ redis_container="" redis_network="" smoke_phase="initializing" smoke_exit_code="" +tmpdir="" +image_ref="" +test_prefix="" +admin_headers="" usage() { - echo "Usage: $0 [--mode single|ha] [--security-check]" >&2 + echo "Usage: $0 [--mode single|ha] [--security-check] [--cleanup-negative-test]" >&2 } while (($#)); do case "$1" in --mode) MODE="${2:-}"; shift 2 ;; --security-check) SECURITY_CHECK=true; shift ;; + --cleanup-negative-test) CLEANUP_NEGATIVE_TEST=true; shift ;; --help|-h) usage; exit 0 ;; *) usage; exit 2 ;; esac @@ -54,6 +60,26 @@ if [[ -z "$SUMMARY_FILE" ]]; then fi verification_invalidate_report "$SUMMARY_FILE" +# Covers precondition failures before the resource-aware cleanup trap is ready. +# It invalidates any stale report and writes a non-passing, redacted result. +early_failure_cleanup() { + local exit_code=$? + trap - EXIT + if command -v jq >/dev/null; then + umask 077 + mkdir -p "$(dirname "$SUMMARY_FILE")" + chmod 700 "$(dirname "$SUMMARY_FILE")" + jq -n \ + --arg commit "$(git -C "$DEMO_DIR/../.." rev-parse HEAD)" \ + --arg run_id "$verification_run_id" --arg mode "$MODE" \ + --argjson exit_code "$exit_code" \ + '{verification_run_id:$run_id,commit:$commit,image_id:"",tested_at:(now|todateiso8601),mode:$mode,result:"failed",phase:"precondition_failed",exit_code:$exit_code,content_redacted:true}' \ + > "${SUMMARY_FILE}.tmp.$$" && chmod 600 "${SUMMARY_FILE}.tmp.$$" && mv "${SUMMARY_FILE}.tmp.$$" "$SUMMARY_FILE" || rm -f "${SUMMARY_FILE}.tmp.$$" + fi + return "$exit_code" +} +trap early_failure_cleanup EXIT + need() { command -v "$1" >/dev/null || { echo "required command missing: $1" >&2; exit 2; }; } need curl; need jq; need rg [[ "$MODE" == "single" || "$MODE" == "ha" ]] || { usage; exit 2; } @@ -94,6 +120,7 @@ security_check() { if [[ "$SECURITY_CHECK" == true ]]; then security_check + trap - EXIT exit 0 fi @@ -225,6 +252,7 @@ cleanup_request_admin() { assert_test_resources_removed() { local counts_file="$tmpdir/cleanup-counts.txt" counts + [[ -n "$test_prefix" && -n "$tmpdir" ]] || return 0 docker compose --env-file "$ENV_FILE" -f "$DEMO_DIR/docker-compose.litellm.yml" exec -T -e P1_CLEANUP_PREFIX="$test_prefix" postgres sh -lc 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -At -v ON_ERROR_STOP=1 -c "SELECT (SELECT count(*) FROM \"LiteLLM_UserTable\" WHERE user_id LIKE \$\$${P1_CLEANUP_PREFIX}%\$\$), (SELECT count(*) FROM \"LiteLLM_CredentialsTable\" WHERE credential_name LIKE \$\$${P1_CLEANUP_PREFIX}%\$\$), (SELECT count(*) FROM \"LiteLLM_ProxyModelTable\" WHERE model_name LIKE \$\$${P1_CLEANUP_PREFIX}%\$\$), (SELECT count(*) FROM \"LiteLLM_VerificationToken\" WHERE key_alias LIKE \$\$${P1_CLEANUP_PREFIX}%\$\$);"' > "$counts_file" counts="$(tr -d '[:space:]' < "$counts_file")" [[ "$counts" == "0|0|0|0" ]] @@ -259,6 +287,11 @@ cleanup() { if [[ -n "$test_user" ]]; then cleanup_request_admin POST /user/delete "$tmpdir/user-delete.json" || cleanup_ok=false fi + # The negative test uses an authenticated, deliberately absent management + # route. A real HTTP 404 must keep cleanup failed and the report non-passing. + if [[ "$CLEANUP_NEGATIVE_TEST" == true ]]; then + cleanup_request_admin DELETE "/__p1_cleanup_failure_probe" "" || cleanup_ok=false + fi assert_test_resources_removed || cleanup_ok=false if [[ "$cleanup_ok" == true ]]; then cleanup_result="passed"; else cleanup_result="failed"; exit_code=1; fi unset master_key_file upstream_key_file upstream_base_file upstream_model_file upstream_provider_file @@ -337,7 +370,7 @@ runtime_security_check() { docker exec "$(docker compose --env-file "$ENV_FILE" -f "$DEMO_DIR/docker-compose.litellm.yml" ps -q redis)" ps -eo args > "$process_file" # Query only P1 test SpendLog content into the private work directory. This # validates the database representation independently of the API response. - docker compose --env-file "$ENV_FILE" -f "$DEMO_DIR/docker-compose.litellm.yml" exec -T postgres sh -lc 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -Atc "SELECT concat(messages::text, response::text, proxy_server_request::text) FROM \"LiteLLM_SpendLogs\" WHERE position(\$p\$p1-smoke-user-\$p\$ in \"user\") = 1;"' > "$db_content_file" + docker compose --env-file "$ENV_FILE" -f "$DEMO_DIR/docker-compose.litellm.yml" exec -T -e P1_CONTENT_PREFIX="$test_prefix" postgres sh -lc 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -At -v ON_ERROR_STOP=1 -c "SELECT concat(messages::text, response::text, proxy_server_request::text) FROM \"LiteLLM_SpendLogs\" WHERE \"user\" LIKE \$\$${P1_CONTENT_PREFIX}%\$\$;"' > "$db_content_file" ! rg -q 'UPSTREAM_(API_KEY|BASE_URL|MODEL)=' "$inspect_file" && ! rg -q -- '--requirepass[[:space:]]+[^[:space:]]+' "$process_file" && ! rg -q --file "$tmpdir/prompt-marker" "$logs_file" && @@ -440,9 +473,15 @@ wait_for_rejection() { } assert_proxy_limiter_response() { - local response_file="$1" header_file="$2" limit_kind="$3" - [[ "$2" == "429" ]] && - jq -e --arg kind "$limit_kind" '(.error // .detail // .message // "") | tostring | test("litellm|" + $kind + ".*(limit|rate)|rate.*" + $kind; "i")' "$response_file" >/dev/null + local response_file="$1" http_code="$2" limit_kind="$3" + [[ "$http_code" == "429" ]] && + jq -e --arg kind "$limit_kind" ' + (.error // .detail // .message // {}) as $error | + ($error | if type == "object" then . else {message:(tostring)} end) as $normalized | + (($normalized.type // "") | ascii_downcase) == "rate_limit_error" and + (($normalized.code // "") | tostring) == "429" and + (($normalized.message // "") | ascii_downcase | test($kind + ".*(limit|rate)|rate.*" + $kind)) + ' "$response_file" >/dev/null } write_summary() { @@ -457,7 +496,7 @@ write_summary() { --arg image_id "$(docker image inspect "$image_ref" --format '{{.Id}}' 2>/dev/null || true)" \ --arg tested_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ --arg mode "$MODE" --arg block_ms "$block_elapsed_ms" --arg delete_ms "$delete_elapsed_ms" --arg phase "$smoke_phase" --arg exit_code "$smoke_exit_code" \ - --arg run_id "$verification_run_id" --arg result "$result" --arg migration "$migration_result" --arg chat "$chat_result" --arg stream "$stream_result" --arg tool "$tool_result" --arg usage "$usage_result" --arg block "$block_result" --arg delete "$delete_result" --arg shared_rpm "$shared_rpm_limit_result" --arg shared_tpm "$shared_tpm_limit_result" --arg shared_spend "$shared_spend_counter_result" --arg limiter_source "$limiter_source" --arg idempotency "$idempotency_recovery_result" --arg cleanup "$cleanup_result" --arg security "$security_scan_result" --arg content_logging "$content_logging_scan_result" \ + --arg run_id "$verification_run_id" --arg result "$result" --arg migration "$migration_result" --arg chat "$chat_result" --arg stream "$stream_result" --arg tool "$tool_result" --arg usage "$usage_result" --arg block "$block_result" --arg delete "$delete_result" --arg shared_rpm "$shared_rpm_limit_result" --arg shared_tpm "$shared_tpm_limit_result" --arg shared_spend "$shared_spend_log_visibility_result" --arg limiter_source "$limiter_source" --arg idempotency "$idempotency_recovery_result" --arg cleanup "$cleanup_result" --arg security "$security_scan_result" --arg content_logging "$content_logging_scan_result" \ '{verification_run_id:$run_id,commit:$commit,image_id:$image_id,tested_at:$tested_at,mode:$mode,result:$result,phase:$phase,exit_code:($exit_code|tonumber?),migration:$migration,chat:$chat,stream:$stream,tool:$tool,usage:$usage,block:$block,delete:$delete,shared_rpm_limit:$shared_rpm,shared_tpm_limit:$shared_tpm,shared_spend_log_visibility:$shared_spend,limiter_source:$limiter_source,idempotency_recovery:$idempotency,block_elapsed_ms:($block_ms|tonumber?),delete_elapsed_ms:($delete_ms|tonumber?),cleanup:$cleanup,security_scan:$security,content_logging_scan:$content_logging,content_redacted:true}' \ > "$summary_tmp" chmod 600 "$summary_tmp" @@ -481,7 +520,7 @@ assert_spend() { peer_request_count="$(jq --rawfile model "$tmpdir/model-name" --rawfile key_hash "$tmpdir/block-key-sha256" 'def logs: if type == "array" then . else (.data // []) end; [ logs[] | select((.model == ($model | rtrimstr("\n")) or .model_group == ($model | rtrimstr("\n"))) and .api_key == ($key_hash | rtrimstr("\n")) and ((.total_tokens // 0) > 0)) ] | length' "$peer_spend_file")" peer_total_tokens="$(jq --rawfile model "$tmpdir/model-name" --rawfile key_hash "$tmpdir/block-key-sha256" 'def logs: if type == "array" then . else (.data // []) end; [ logs[] | select((.model == ($model | rtrimstr("\n")) or .model_group == ($model | rtrimstr("\n"))) and .api_key == ($key_hash | rtrimstr("\n")) and ((.total_tokens // 0) > 0)) ] | map(.total_tokens // 0) | add' "$peer_spend_file")" [[ "$request_count/$total_tokens" == "$peer_request_count/$peer_total_tokens" ]] || { echo "shared SpendLog mismatch: primary=$request_count/$total_tokens peer=$peer_request_count/$peer_total_tokens" >&2; return 1; } - shared_spend_counter_result="passed" + shared_spend_log_visibility_result="passed" echo "PASS HA shared SpendLog: request_count=$request_count total_tokens=$total_tokens on both replicas." fi echo "PASS spend: request_count=$request_count total_tokens=$total_tokens" @@ -525,6 +564,12 @@ chmod 600 "$tmpdir/user-delete.json" request_admin POST /user/new "$tmpdir/user-create.json" "$tmpdir/user.json" jq -e --rawfile user "$tmpdir/test-user" '.user_id == ($user | rtrimstr("\n"))' "$tmpdir/user.json" >/dev/null +if [[ "$CLEANUP_NEGATIVE_TEST" == true ]]; then + smoke_phase="cleanup_negative_test" + echo "Running authenticated cleanup failure negative test." + exit 0 +fi + if [[ ! -s "$upstream_key_file" || ! -s "$upstream_base_file" || ! -s "$upstream_model_file" ]]; then echo "PENDING upstream smoke: set UPSTREAM_API_KEY, UPSTREAM_BASE_URL and UPSTREAM_MODEL in ignored local environment file." chat_result="pending"; stream_result="pending"; tool_result="pending"; usage_result="pending" @@ -678,8 +723,8 @@ if [[ "$MODE" == "ha" ]]; then enforcement_key_created=true make_key_action_payload delete "$enforcement_key_file" "$tmpdir/enforcement-key-cleanup.json" request_data_post "$BASE_URL" "$enforcement_headers" /v1/chat/completions "$tmpdir/chat-request.json" "$tmpdir/budget-first.json" - budget_code="$(curl --silent --show-error --max-time 30 --request POST "$PEER_URL/v1/chat/completions" --header "@$enforcement_headers" --data-binary "@$tmpdir/chat-request.json" --output "$tmpdir/budget-second.json" --write-out '%{http_code}' || true)" - assert_proxy_limiter_response "$tmpdir/budget-second.json" "$budget_code" tpm || { echo "shared TPM limiter was not a LiteLLM proxy 429" >&2; exit 1; } + tpm_code="$(curl --silent --show-error --max-time 30 --request POST "$PEER_URL/v1/chat/completions" --header "@$enforcement_headers" --data-binary "@$tmpdir/chat-request.json" --output "$tmpdir/tpm-second.json" --write-out '%{http_code}' || true)" + assert_proxy_limiter_response "$tmpdir/tpm-second.json" "$tpm_code" tpm || { echo "shared TPM limiter was not a LiteLLM proxy 429" >&2; exit 1; } shared_tpm_limit_result="passed" echo "PASS HA shared TPM enforcement: peer rejected the second request with 429." fi diff --git a/docker_litellm/demo/scripts/smoke-redis-recovery.sh b/docker_litellm/demo/scripts/smoke-redis-recovery.sh index bc3e276..b0cd286 100755 --- a/docker_litellm/demo/scripts/smoke-redis-recovery.sh +++ b/docker_litellm/demo/scripts/smoke-redis-recovery.sh @@ -8,7 +8,7 @@ source "${script_dir}/verification-lib.sh" env_file="${LITELLM_SMOKE_ENV_FILE:-${demo_dir}/.env}" summary_file="${LITELLM_REDIS_SUMMARY_FILE:-${demo_dir}/artifacts/p1-redis-recovery.json}" verification_run_id="${VERIFICATION_RUN_ID:-standalone}" -recovery_timeout="${REDIS_CIRCUIT_BREAKER_RECOVERY_TIMEOUT:-5}" +recovery_timeout="" container="" network="" phase="initializing" @@ -21,9 +21,11 @@ image_ref="" verification_invalidate_report "$summary_file" write_summary() { + local summary_tmp umask 077 mkdir -p "$(dirname "$summary_file")" chmod 700 "$(dirname "$summary_file")" + summary_tmp="${summary_file}.tmp.$$" jq -n \ --arg commit "$(git -C "$demo_dir/../.." rev-parse HEAD)" \ --arg image_id "$(docker image inspect "$image_ref" --format '{{.Id}}' 2>/dev/null || true)" \ @@ -31,8 +33,7 @@ write_summary() { --arg run_id "$verification_run_id" --arg result "$result" --arg phase "$phase" --arg security_scan "$security_scan" \ --arg post_recovery_call "$post_recovery_call" \ '{verification_run_id:$run_id,commit:$commit,image_id:$image_id,tested_at:$tested_at,mode:"ha",result:$result,phase:$phase,redis_recovery:$result,post_recovery_call:$post_recovery_call,security_scan:$security_scan,content_redacted:true}' \ - > "$summary_file" - chmod 600 "$summary_file" + > "$summary_tmp" && chmod 600 "$summary_tmp" && mv "$summary_tmp" "$summary_file" } cleanup() { @@ -61,9 +62,12 @@ image_ref="$(verification_env LITELLM_IMAGE)" # Keep the recovery proof aligned with the same optional host-port overrides # that Compose uses for the two proxy replicas. These are non-secret routing # values; credentials remain in the private header file below. -publish_host="$(verification_env LITELLM_PUBLISH_HOST)" -litellm_1_port="$(verification_env LITELLM_1_PORT)" -litellm_2_port="$(verification_env LITELLM_2_PORT)" + publish_host="$(verification_env LITELLM_PUBLISH_HOST)" + litellm_1_port="$(verification_env LITELLM_1_PORT)" + litellm_2_port="$(verification_env LITELLM_2_PORT)" + recovery_timeout="$(verification_env REDIS_CIRCUIT_BREAKER_RECOVERY_TIMEOUT)" + recovery_timeout="${recovery_timeout:-5}" + [[ "$recovery_timeout" =~ ^[0-9]+$ ]] || { echo "invalid Redis recovery timeout" >&2; exit 2; } master_file="$tmpdir/master-key" headers_file="$tmpdir/admin.headers" printf '%s' "$master_key" > "$master_file" diff --git a/docker_litellm/demo/scripts/test-verification-gates.sh b/docker_litellm/demo/scripts/test-verification-gates.sh index c42dc1b..ef3b166 100755 --- a/docker_litellm/demo/scripts/test-verification-gates.sh +++ b/docker_litellm/demo/scripts/test-verification-gates.sh @@ -5,6 +5,9 @@ set -euo pipefail script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" demo_dir="$(cd "${script_dir}/.." && pwd)" +with_running_stack=false +[[ "${1:-}" != "--with-running-stack" ]] || with_running_stack=true +[[ $# -eq 0 || "$with_running_stack" == true ]] || { echo "Usage: $0 [--with-running-stack]" >&2; exit 2; } tmpdir="$(mktemp -d "${TMPDIR:-/tmp}/litellm-gates.XXXXXX")" chmod 700 "$tmpdir" trap 'rm -rf "$tmpdir"' EXIT @@ -21,6 +24,17 @@ if VERIFICATION_RUN_ID="$run_id" LITELLM_ARTIFACTS_DIR="$tmpdir" LITELLM_AGGREGA echo "aggregate accepted stale PASS reports" >&2 exit 1 fi +[[ ! -e "$tmpdir/final.json" ]] || { echo "aggregate left a stale final report" >&2; exit 1; } + +# A producer must overwrite a pre-existing PASS before even checking a missing +# environment file. The resulting report is explicitly non-passing. +jq -n --arg run_id "$run_id" '{verification_run_id:$run_id,result:"passed",phase:"completed"}' > "$tmpdir/precondition.json" +if VERIFICATION_RUN_ID="$run_id" LITELLM_SMOKE_ENV_FILE="$tmpdir/absent.env" LITELLM_SMOKE_SUMMARY_FILE="$tmpdir/precondition.json" \ + "$script_dir/smoke-baseline.sh" --mode single >/dev/null 2>&1; then + echo "smoke unexpectedly accepted a missing environment file" >&2 + exit 1 +fi +jq -e '.result == "failed" and .phase == "precondition_failed"' "$tmpdir/precondition.json" >/dev/null # Cleanup failures must never be normalized to PASS. This checks the curl # transport invariant directly without sending a request. @@ -28,4 +42,25 @@ rg -q 'cleanup_request_admin.*\(\)' "$script_dir/smoke-baseline.sh" rg -q -- 'curl_args=\(--silent --show-error --fail' "$script_dir/smoke-baseline.sh" ! rg -n -- 'source "\$env_file"|source "\$\{env_file\}"' "$script_dir/run-migration.sh" "$script_dir/smoke-redis-recovery.sh" rg -q 'config --environment > "\$verification_environment_file"' "$script_dir/verification-lib.sh" -echo "PASS verification gates: stale reports and cleanup HTTP failure cannot pass." + +# Compose's dotenv parser must not evaluate shell substitutions. This isolated +# Compose file uses no project secrets and verifies the same config command +# that the runtime scripts use. +marker="$tmpdir/dotenv-command-substitution-ran" +printf '%s\n' 'services:' ' proof:' ' image: alpine:3.21' ' environment:' ' PROOF: ${PAYLOAD:?missing}' > "$tmpdir/compose.yml" +printf 'PAYLOAD=$(touch %s)\n' "$marker" > "$tmpdir/malicious.env" +docker compose --env-file "$tmpdir/malicious.env" -f "$tmpdir/compose.yml" config --environment > "$tmpdir/effective.env" +[[ ! -e "$marker" ]] || { echo "dotenv command substitution executed" >&2; exit 1; } +rg -Fq 'PAYLOAD=$(touch ' "$tmpdir/effective.env" + +if [[ "$with_running_stack" == true ]]; then + negative_summary="$tmpdir/cleanup-negative.json" + if VERIFICATION_RUN_ID="$run_id" LITELLM_SMOKE_SUMMARY_FILE="$negative_summary" \ + "$script_dir/smoke-baseline.sh" --mode single --cleanup-negative-test >/dev/null 2>&1; then + echo "cleanup negative test unexpectedly passed" >&2 + exit 1 + fi + jq -e '.result == "failed" and .cleanup == "failed" and .phase == "cleanup_negative_test"' "$negative_summary" >/dev/null +fi + +echo "PASS verification gates: stale reports, preconditions, dotenv substitutions and cleanup HTTP failures cannot pass." diff --git a/docker_litellm/demo/scripts/verify-p1.sh b/docker_litellm/demo/scripts/verify-p1.sh index 0d66dee..02b5131 100755 --- a/docker_litellm/demo/scripts/verify-p1.sh +++ b/docker_litellm/demo/scripts/verify-p1.sh @@ -6,6 +6,9 @@ script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" demo_dir="$(cd "${script_dir}/.." && pwd)" run_id="${VERIFICATION_RUN_ID:-$(python3 -c 'import secrets; print("p1-" + secrets.token_hex(16))')}" export VERIFICATION_RUN_ID="$run_id" +# P1's documented local provider mapping is explicit. It is non-secret and +# prevents a template/default mismatch from silently selecting another SDK. +export LITELLM_SMOKE_UPSTREAM_PROVIDER="${LITELLM_SMOKE_UPSTREAM_PROVIDER:-deepseek}" compose=(docker compose --env-file "${demo_dir}/.env" -f "${demo_dir}/docker-compose.litellm.yml") cleanup_stack() { "${compose[@]}" --profile single --profile ha --profile migrate down >/dev/null 2>&1 || true; } trap cleanup_stack EXIT From 497ab1d941dd4552fe74dc5b264ae166d11e6fa4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 16:15:10 +0800 Subject: [PATCH 39/87] =?UTF-8?q?test:=20=E5=A4=8D=E7=94=A8=20P1=20?= =?UTF-8?q?=E8=B4=9F=E5=90=91=E9=AA=8C=E8=AF=81=E8=BF=90=E8=A1=8C=E6=A0=87?= =?UTF-8?q?=E8=AF=86?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/demo/scripts/test-verification-gates.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docker_litellm/demo/scripts/test-verification-gates.sh b/docker_litellm/demo/scripts/test-verification-gates.sh index ef3b166..0b87de9 100755 --- a/docker_litellm/demo/scripts/test-verification-gates.sh +++ b/docker_litellm/demo/scripts/test-verification-gates.sh @@ -11,7 +11,8 @@ with_running_stack=false tmpdir="$(mktemp -d "${TMPDIR:-/tmp}/litellm-gates.XXXXXX")" chmod 700 "$tmpdir" trap 'rm -rf "$tmpdir"' EXIT -run_id="p1-$(python3 -c 'import secrets; print(secrets.token_hex(16))')" +run_id="${VERIFICATION_RUN_ID:-p1-$(python3 -c 'import secrets; print(secrets.token_hex(16))')}" +[[ "$run_id" =~ ^p1-[a-f0-9]{32}$ ]] || { echo "invalid verification run id" >&2; exit 2; } # A deliberately stale-but-well-shaped set must be rejected when the current # run id differs. No service or .env is read by this test. From 7b12b94880317bd5b95cbb26b04b22fd68263e89 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 16:26:19 +0800 Subject: [PATCH 40/87] =?UTF-8?q?fix:=20=E4=BF=9D=E7=95=99=20smoke=20?= =?UTF-8?q?=E5=A4=B1=E8=B4=A5=E6=8A=A5=E5=91=8A=E5=B9=B6=E9=9A=94=E7=A6=BB?= =?UTF-8?q?=E8=B5=84=E6=BA=90=E5=90=8D?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/demo/scripts/smoke-baseline.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh index 91a0c43..32ca7c0 100755 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -304,7 +304,7 @@ cleanup() { result="failed" fi write_summary || exit_code=1 - exit "$exit_code" + return "$exit_code" } trap cleanup EXIT @@ -540,7 +540,7 @@ assert_migration_evidence migration_result="passed" echo "PASS readiness: PostgreSQL connected; Redis independently reachable from LiteLLM replica(s)." -suffix="$(date +%s)-$RANDOM" +suffix="$(date +%s)-$(python3 -c 'import secrets; print(secrets.token_hex(8))')" test_prefix="p1-smoke-${verification_run_id#p1-}-$suffix" test_user="${test_prefix}-user" credential_name="${test_prefix}-upstream" From 6b7f18688debb78d1091228d6819aa1ac6a5116d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 16:29:39 +0800 Subject: [PATCH 41/87] =?UTF-8?q?fix:=20=E5=BC=BA=E5=88=B6=20cleanup=20?= =?UTF-8?q?=E5=A4=B1=E8=B4=A5=E8=BF=94=E5=9B=9E=E9=9D=9E=E9=9B=B6?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/demo/scripts/smoke-baseline.sh | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh index 32ca7c0..d6fa367 100755 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -36,6 +36,7 @@ redis_container="" redis_network="" smoke_phase="initializing" smoke_exit_code="" +cleanup_running=false tmpdir="" image_ref="" test_prefix="" @@ -261,6 +262,8 @@ assert_test_resources_removed() { cleanup() { local exit_code=$? local cleanup_ok=true + [[ "$cleanup_running" == false ]] || return "$exit_code" + cleanup_running=true trap - EXIT set +e if [[ -n "$redis_container" && -n "$redis_network" ]]; then @@ -304,7 +307,7 @@ cleanup() { result="failed" fi write_summary || exit_code=1 - return "$exit_code" + exit "$exit_code" } trap cleanup EXIT From 5f6ee4572789de45e957982b8b4fe3ec00317ba8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 16:33:43 +0800 Subject: [PATCH 42/87] =?UTF-8?q?fix:=20=E4=BB=A5=20LiteLLM=20limiter=20?= =?UTF-8?q?=E8=AF=81=E6=8D=AE=E9=AA=8C=E8=AF=81=E5=85=B1=E4=BA=AB=E9=99=90?= =?UTF-8?q?=E6=B5=81?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/demo/scripts/smoke-baseline.sh | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh index d6fa367..b8fa549 100755 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -479,12 +479,12 @@ assert_proxy_limiter_response() { local response_file="$1" http_code="$2" limit_kind="$3" [[ "$http_code" == "429" ]] && jq -e --arg kind "$limit_kind" ' - (.error // .detail // .message // {}) as $error | - ($error | if type == "object" then . else {message:(tostring)} end) as $normalized | - (($normalized.type // "") | ascii_downcase) == "rate_limit_error" and - (($normalized.code // "") | tostring) == "429" and - (($normalized.message // "") | ascii_downcase | test($kind + ".*(limit|rate)|rate.*" + $kind)) - ' "$response_file" >/dev/null + (.error // .detail // .message // "") | tostring | ascii_downcase | + test("rate limit exceeded") and + (if $kind == "rpm" then test("requests|rpm") else test("tokens|tpm") end) + ' "$response_file" >/dev/null && + docker compose --env-file "$ENV_FILE" -f "$DEMO_DIR/docker-compose.litellm.yml" logs --no-color litellm-1 litellm-2 | + rg -q 'parallel_request_limiter_v3|ProxyRateLimitError' } write_summary() { From 29848d55a5ae89d33256c793b7857a02fca00394 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 16:37:30 +0800 Subject: [PATCH 43/87] =?UTF-8?q?fix:=20=E4=BF=9D=E6=8A=A4=20cleanup=20?= =?UTF-8?q?=E5=A4=B1=E8=B4=A5=E6=8A=A5=E5=91=8A=E5=86=99=E5=85=A5?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/demo/scripts/smoke-baseline.sh | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh index b8fa549..c13b6d6 100755 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -65,7 +65,8 @@ verification_invalidate_report "$SUMMARY_FILE" # It invalidates any stale report and writes a non-passing, redacted result. early_failure_cleanup() { local exit_code=$? - trap - EXIT + # Ignore any nested EXIT delivery while preserving the report just written. + trap '' EXIT if command -v jq >/dev/null; then umask 077 mkdir -p "$(dirname "$SUMMARY_FILE")" @@ -264,7 +265,8 @@ cleanup() { local cleanup_ok=true [[ "$cleanup_running" == false ]] || return "$exit_code" cleanup_running=true - trap - EXIT + # Ignore any nested EXIT delivery while preserving the report just written. + trap '' EXIT set +e if [[ -n "$redis_container" && -n "$redis_network" ]]; then docker network connect --alias redis "$redis_network" "$redis_container" >/dev/null 2>&1 || true From 50e39ec1b18f118f85f2b86112410630139f8cd9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 16:39:00 +0800 Subject: [PATCH 44/87] =?UTF-8?q?fix:=20=E5=8E=9F=E5=AD=90=E5=88=9B?= =?UTF-8?q?=E5=BB=BA=20smoke=20=E6=8A=A5=E5=91=8A=E4=B8=B4=E6=97=B6?= =?UTF-8?q?=E6=96=87=E4=BB=B6?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/demo/scripts/smoke-baseline.sh | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh index c13b6d6..33dc13d 100755 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -493,8 +493,7 @@ write_summary() { local summary_tmp mkdir -p "$(dirname "$SUMMARY_FILE")" chmod 700 "$(dirname "$SUMMARY_FILE")" - summary_tmp="${SUMMARY_FILE}.tmp.$$" - : > "$summary_tmp" + summary_tmp="$(mktemp "${SUMMARY_FILE}.tmp.XXXXXX")" chmod 600 "$summary_tmp" jq -n \ --arg commit "$(git -C "$DEMO_DIR/../.." rev-parse HEAD)" \ From 16d45579615147c6336e51a78d0a426142f12c44 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 16:40:23 +0800 Subject: [PATCH 45/87] =?UTF-8?q?fix:=20=E4=BF=9D=E7=95=99=20cleanup=20?= =?UTF-8?q?=E5=A4=B1=E8=B4=A5=E7=9A=84=E5=8E=9F=E5=AD=90=E6=8A=A5=E5=91=8A?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/demo/scripts/smoke-baseline.sh | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh index 33dc13d..58a7802 100755 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -309,7 +309,15 @@ cleanup() { result="failed" fi write_summary || exit_code=1 - exit "$exit_code" + # Bash may re-enter EXIT processing when `exit` is called from an EXIT + # handler, which previously truncated a just-written failed report. A failed + # cleanup terminates the shell with the default TERM action after disabling + # the handler; the persisted JSON remains the authoritative failure record. + if (( exit_code != 0 )); then + trap - EXIT + kill -TERM "$$" + fi + return "$exit_code" } trap cleanup EXIT From efaf6e541142ec4d07f3df66d477306395b9d655 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 16:44:47 +0800 Subject: [PATCH 46/87] =?UTF-8?q?fix:=20=E5=90=8C=E6=AD=A5=20cleanup=20?= =?UTF-8?q?=E5=A4=B1=E8=B4=A5=E6=8A=A5=E5=91=8A=E8=90=BD=E7=9B=98?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/demo/scripts/smoke-baseline.sh | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh index 58a7802..f7be49a 100755 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -314,6 +314,8 @@ cleanup() { # cleanup terminates the shell with the default TERM action after disabling # the handler; the persisted JSON remains the authoritative failure record. if (( exit_code != 0 )); then + # Ensure the atomic rename is durably visible before the deliberate signal. + sync "$SUMMARY_FILE" 2>/dev/null || sync trap - EXIT kill -TERM "$$" fi From 893f1bc0b0aee30e2488ec8ec7f92fee78cc60b9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 16:48:19 +0800 Subject: [PATCH 47/87] =?UTF-8?q?fix:=20=E7=94=9F=E6=88=90=E5=AE=8C?= =?UTF-8?q?=E6=95=B4=20smoke=20=E5=A4=B1=E8=B4=A5=E6=91=98=E8=A6=81?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/demo/scripts/smoke-baseline.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh index f7be49a..274e21a 100755 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -511,10 +511,11 @@ write_summary() { --arg tested_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ --arg mode "$MODE" --arg block_ms "$block_elapsed_ms" --arg delete_ms "$delete_elapsed_ms" --arg phase "$smoke_phase" --arg exit_code "$smoke_exit_code" \ --arg run_id "$verification_run_id" --arg result "$result" --arg migration "$migration_result" --arg chat "$chat_result" --arg stream "$stream_result" --arg tool "$tool_result" --arg usage "$usage_result" --arg block "$block_result" --arg delete "$delete_result" --arg shared_rpm "$shared_rpm_limit_result" --arg shared_tpm "$shared_tpm_limit_result" --arg shared_spend "$shared_spend_log_visibility_result" --arg limiter_source "$limiter_source" --arg idempotency "$idempotency_recovery_result" --arg cleanup "$cleanup_result" --arg security "$security_scan_result" --arg content_logging "$content_logging_scan_result" \ - '{verification_run_id:$run_id,commit:$commit,image_id:$image_id,tested_at:$tested_at,mode:$mode,result:$result,phase:$phase,exit_code:($exit_code|tonumber?),migration:$migration,chat:$chat,stream:$stream,tool:$tool,usage:$usage,block:$block,delete:$delete,shared_rpm_limit:$shared_rpm,shared_tpm_limit:$shared_tpm,shared_spend_log_visibility:$shared_spend,limiter_source:$limiter_source,idempotency_recovery:$idempotency,block_elapsed_ms:($block_ms|tonumber?),delete_elapsed_ms:($delete_ms|tonumber?),cleanup:$cleanup,security_scan:$security,content_logging_scan:$content_logging,content_redacted:true}' \ + '{verification_run_id:$run_id,commit:$commit,image_id:$image_id,tested_at:$tested_at,mode:$mode,result:$result,phase:$phase,exit_code:(try ($exit_code|tonumber) catch null),migration:$migration,chat:$chat,stream:$stream,tool:$tool,usage:$usage,block:$block,delete:$delete,shared_rpm_limit:$shared_rpm,shared_tpm_limit:$shared_tpm,shared_spend_log_visibility:$shared_spend,limiter_source:$limiter_source,idempotency_recovery:$idempotency,block_elapsed_ms:(try ($block_ms|tonumber) catch null),delete_elapsed_ms:(try ($delete_ms|tonumber) catch null),cleanup:$cleanup,security_scan:$security,content_logging_scan:$content_logging,content_redacted:true}' \ > "$summary_tmp" chmod 600 "$summary_tmp" mv "$summary_tmp" "$SUMMARY_FILE" + [[ -s "$SUMMARY_FILE" ]] || { echo "summary write produced an empty file" >&2; return 1; } echo "PASS summary: $SUMMARY_FILE" } From f0e0a8f5560e36e8eda54c113d64288f92a3ee25 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 18:37:59 +0800 Subject: [PATCH 48/87] =?UTF-8?q?fix:=20=E5=BC=BA=E5=8C=96=20P1=20?= =?UTF-8?q?=E8=BF=90=E8=A1=8C=E9=97=A8=E7=A6=81=E4=B8=8E=E5=B9=B6=E5=8F=91?= =?UTF-8?q?=E8=BF=81=E7=A7=BB=E9=AA=8C=E8=AF=81?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../scripts/aggregate-verification-summary.sh | 23 +++++++--- docker_litellm/demo/scripts/smoke-baseline.sh | 22 +++++---- .../demo/scripts/test-verification-gates.sh | 31 +++++++++++++ .../scripts/verify-migration-concurrency.sh | 45 +++++++++++++++++++ docker_litellm/demo/scripts/verify-p1.sh | 9 +++- 5 files changed, 115 insertions(+), 15 deletions(-) create mode 100755 docker_litellm/demo/scripts/verify-migration-concurrency.sh diff --git a/docker_litellm/demo/scripts/aggregate-verification-summary.sh b/docker_litellm/demo/scripts/aggregate-verification-summary.sh index 6902f3c..9e56966 100755 --- a/docker_litellm/demo/scripts/aggregate-verification-summary.sh +++ b/docker_litellm/demo/scripts/aggregate-verification-summary.sh @@ -10,36 +10,48 @@ single_report="${artifacts_dir}/p1-single-summary.json" ha_report="${artifacts_dir}/p1-ha-summary.json" redis_report="${artifacts_dir}/p1-redis-recovery.json" migration_report="${artifacts_dir}/p1-migration-summary.json" +concurrency_report="${artifacts_dir}/p1-migration-concurrency.json" output="${LITELLM_AGGREGATE_SUMMARY_FILE:-${artifacts_dir}/p1-final-summary.json}" commit="$(git -C "$demo_dir/../.." rev-parse HEAD)" run_id="${VERIFICATION_RUN_ID:-}" +started_at="${VERIFICATION_STARTED_AT:-}" rm -f "$output" [[ "$run_id" =~ ^p1-[a-f0-9]{32}$ ]] || { echo "missing verification run id" >&2; exit 2; } +[[ "$started_at" =~ ^[0-9]{4}-[0-9]{2}-[0-9]{2}T ]] || { echo "missing verification start time" >&2; exit 2; } -for report in "$single_report" "$ha_report" "$redis_report" "$migration_report"; do +for report in "$single_report" "$ha_report" "$redis_report" "$migration_report" "$concurrency_report"; do [[ -f "$report" ]] || { echo "missing required report: $report" >&2; exit 2; } done # Each input must be an independently successful and fully redacted result of # this exact checkout. jq -e performs the gate before the final report exists. -jq -e --arg commit "$commit" --arg run_id "$run_id" ' +jq -e --arg commit "$commit" --arg run_id "$run_id" --arg started_at "$started_at" ' .mode == "migration" and .result == "passed" and .phase == "completed" and .verification_run_id == $run_id and .commit == $commit and (.image_id | type == "string" and length > 0) and + (.tested_at | type == "string" and . >= $started_at) and .content_redacted == true and .proxy_replicas_started == false ' "$migration_report" >/dev/null -jq -e --arg commit "$commit" --arg run_id "$run_id" ' +jq -e --arg commit "$commit" --arg run_id "$run_id" --arg started_at "$started_at" ' + .mode == "migration" and .result == "passed" and .phase == "completed" and .concurrent_migration == true and + .verification_run_id == $run_id and .commit == $commit and (.image_id | type == "string" and length > 0) and + (.tested_at | type == "string" and . >= $started_at) and .content_redacted == true +' "$concurrency_report" >/dev/null + +jq -e --arg commit "$commit" --arg run_id "$run_id" --arg started_at "$started_at" ' .mode == "single" and .result == "passed" and .phase == "completed" and .verification_run_id == $run_id and .commit == $commit and (.image_id | type == "string" and length > 0) and + (.tested_at | type == "string" and . >= $started_at) and .content_redacted == true and .migration == "passed" and .chat == "passed" and .stream == "passed" and .tool == "passed" and .usage == "passed" and .block == "passed" and .delete == "passed" and .cleanup == "passed" and .security_scan == "passed" and .content_logging_scan == "passed" ' "$single_report" >/dev/null -jq -e --arg commit "$commit" --arg run_id "$run_id" ' +jq -e --arg commit "$commit" --arg run_id "$run_id" --arg started_at "$started_at" ' .mode == "ha" and .result == "passed" and .phase == "completed" and .verification_run_id == $run_id and .commit == $commit and (.image_id | type == "string" and length > 0) and + (.tested_at | type == "string" and . >= $started_at) and .content_redacted == true and .migration == "passed" and .chat == "passed" and .stream == "passed" and .tool == "passed" and .usage == "passed" and .block == "passed" and .delete == "passed" and @@ -49,9 +61,10 @@ jq -e --arg commit "$commit" --arg run_id "$run_id" ' .content_logging_scan == "passed" ' "$ha_report" >/dev/null -jq -e --arg commit "$commit" --arg run_id "$run_id" ' +jq -e --arg commit "$commit" --arg run_id "$run_id" --arg started_at "$started_at" ' .mode == "ha" and .result == "passed" and .phase == "completed" and .verification_run_id == $run_id and .commit == $commit and (.image_id | type == "string" and length > 0) and + (.tested_at | type == "string" and . >= $started_at) and .redis_recovery == "passed" and .content_redacted == true and .security_scan == "passed" ' "$redis_report" >/dev/null diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh index 274e21a..72079ed 100755 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -59,7 +59,6 @@ done if [[ -z "$SUMMARY_FILE" ]]; then SUMMARY_FILE="$DEMO_DIR/artifacts/p1-${MODE}-summary.json" fi -verification_invalidate_report "$SUMMARY_FILE" # Covers precondition failures before the resource-aware cleanup trap is ready. # It invalidates any stale report and writes a non-passing, redacted result. @@ -126,6 +125,10 @@ if [[ "$SECURITY_CHECK" == true ]]; then exit 0 fi +# A real producer invalidates its old result before all remaining preconditions. +# `--security-check` is read-only and must never touch an existing summary. +verification_invalidate_report "$SUMMARY_FILE" + # Run the static negative checks in every real smoke too. A successful report # cannot claim a security result that was not actually executed. security_check @@ -488,14 +491,13 @@ wait_for_rejection() { } assert_proxy_limiter_response() { - local response_file="$1" http_code="$2" limit_kind="$3" + local response_file="$1" http_code="$2" limit_kind="$3" since="$4" expected_type + if [[ "$limit_kind" == rpm ]]; then expected_type=requests; else expected_type=tokens; fi [[ "$http_code" == "429" ]] && - jq -e --arg kind "$limit_kind" ' - (.error // .detail // .message // "") | tostring | ascii_downcase | - test("rate limit exceeded") and - (if $kind == "rpm" then test("requests|rpm") else test("tokens|tpm") end) + jq -e --arg expected_type "$expected_type" ' + [(.error.rate_limit_type? // .rate_limit_type? // empty)] | index($expected_type) != null ' "$response_file" >/dev/null && - docker compose --env-file "$ENV_FILE" -f "$DEMO_DIR/docker-compose.litellm.yml" logs --no-color litellm-1 litellm-2 | + docker compose --env-file "$ENV_FILE" -f "$DEMO_DIR/docker-compose.litellm.yml" logs --no-color --since "$since" litellm-1 litellm-2 | rg -q 'parallel_request_limiter_v3|ProxyRateLimitError' } @@ -717,9 +719,10 @@ if [[ "$MODE" == "ha" ]]; then make_key_header "$tmpdir/rate-key.json" "$rate_key_file" "$rate_headers" rate_key_created=true make_key_action_payload delete "$rate_key_file" "$tmpdir/rate-key-cleanup.json" + rate_limiter_since="$(date -u +%Y-%m-%dT%H:%M:%SZ)" request_data_post "$BASE_URL" "$rate_headers" /v1/chat/completions "$tmpdir/chat-request.json" "$tmpdir/rate-first.json" rate_code="$(curl --silent --show-error --max-time 30 --request POST "$PEER_URL/v1/chat/completions" --header "@$rate_headers" --data-binary "@$tmpdir/chat-request.json" --output "$tmpdir/rate-second.json" --write-out '%{http_code}' || true)" - assert_proxy_limiter_response "$tmpdir/rate-second.json" "$rate_code" rpm || { echo "shared RPM limiter was not a LiteLLM proxy 429" >&2; exit 1; } + assert_proxy_limiter_response "$tmpdir/rate-second.json" "$rate_code" rpm "$rate_limiter_since" || { echo "shared RPM limiter was not a LiteLLM proxy 429" >&2; exit 1; } shared_rpm_limit_result="passed" limiter_source="litellm_proxy" echo "PASS HA shared RPM: peer rejected the second request with 429." @@ -737,9 +740,10 @@ if [[ "$MODE" == "ha" ]]; then make_key_header "$tmpdir/enforcement-key.json" "$enforcement_key_file" "$enforcement_headers" enforcement_key_created=true make_key_action_payload delete "$enforcement_key_file" "$tmpdir/enforcement-key-cleanup.json" + tpm_limiter_since="$(date -u +%Y-%m-%dT%H:%M:%SZ)" request_data_post "$BASE_URL" "$enforcement_headers" /v1/chat/completions "$tmpdir/chat-request.json" "$tmpdir/budget-first.json" tpm_code="$(curl --silent --show-error --max-time 30 --request POST "$PEER_URL/v1/chat/completions" --header "@$enforcement_headers" --data-binary "@$tmpdir/chat-request.json" --output "$tmpdir/tpm-second.json" --write-out '%{http_code}' || true)" - assert_proxy_limiter_response "$tmpdir/tpm-second.json" "$tpm_code" tpm || { echo "shared TPM limiter was not a LiteLLM proxy 429" >&2; exit 1; } + assert_proxy_limiter_response "$tmpdir/tpm-second.json" "$tpm_code" tpm "$tpm_limiter_since" || { echo "shared TPM limiter was not a LiteLLM proxy 429" >&2; exit 1; } shared_tpm_limit_result="passed" echo "PASS HA shared TPM enforcement: peer rejected the second request with 429." fi diff --git a/docker_litellm/demo/scripts/test-verification-gates.sh b/docker_litellm/demo/scripts/test-verification-gates.sh index 0b87de9..35cc203 100755 --- a/docker_litellm/demo/scripts/test-verification-gates.sh +++ b/docker_litellm/demo/scripts/test-verification-gates.sh @@ -54,6 +54,37 @@ docker compose --env-file "$tmpdir/malicious.env" -f "$tmpdir/compose.yml" confi [[ ! -e "$marker" ]] || { echo "dotenv command substitution executed" >&2; exit 1; } rg -Fq 'PAYLOAD=$(touch ' "$tmpdir/effective.env" +# The aggregate gate must also reject structurally plausible but incomplete +# evidence: a missing timestamp, a failed migration, or a forged limiter claim. +started_at="2026-01-01T00:00:00Z" +make_reports() { + jq -n --arg run_id "$run_id" --arg started_at "$started_at" \ + '{verification_run_id:$run_id,commit:"head",image_id:"image",tested_at:$started_at,mode:"migration",result:"passed",phase:"completed",content_redacted:true,proxy_replicas_started:false}' > "$tmpdir/p1-migration-summary.json" + jq -n --arg run_id "$run_id" --arg started_at "$started_at" \ + '{verification_run_id:$run_id,commit:"head",image_id:"image",tested_at:$started_at,mode:"single",result:"passed",phase:"completed",content_redacted:true,migration:"passed",chat:"passed",stream:"passed",tool:"passed",usage:"passed",block:"passed",delete:"passed",cleanup:"passed",security_scan:"passed",content_logging_scan:"passed"}' > "$tmpdir/p1-single-summary.json" + jq -n --arg run_id "$run_id" --arg started_at "$started_at" \ + '{verification_run_id:$run_id,commit:"head",image_id:"image",tested_at:$started_at,mode:"ha",result:"passed",phase:"completed",content_redacted:true,migration:"passed",chat:"passed",stream:"passed",tool:"passed",usage:"passed",block:"passed",delete:"passed",shared_rpm_limit:"passed",shared_tpm_limit:"passed",shared_spend_log_visibility:"passed",idempotency_recovery:"passed",limiter_source:"litellm_proxy",cleanup:"passed",security_scan:"passed",content_logging_scan:"passed"}' > "$tmpdir/p1-ha-summary.json" + jq -n --arg run_id "$run_id" --arg started_at "$started_at" \ + '{verification_run_id:$run_id,commit:"head",image_id:"image",tested_at:$started_at,mode:"ha",result:"passed",phase:"completed",redis_recovery:"passed",content_redacted:true,security_scan:"passed"}' > "$tmpdir/p1-redis-recovery.json" +} +# These are deliberately rejected before any report can become final. They use +# a synthetic commit, so the current checkout mismatch is an additional guard. +make_reports +jq 'del(.tested_at)' "$tmpdir/p1-single-summary.json" > "$tmpdir/single.tmp" && mv "$tmpdir/single.tmp" "$tmpdir/p1-single-summary.json" +if VERIFICATION_RUN_ID="$run_id" VERIFICATION_STARTED_AT="$started_at" LITELLM_ARTIFACTS_DIR="$tmpdir" "$script_dir/aggregate-verification-summary.sh" >/dev/null 2>&1; then + echo "aggregate accepted a report without tested_at" >&2; exit 1 +fi +make_reports +jq '.result="failed"' "$tmpdir/p1-migration-summary.json" > "$tmpdir/migration.tmp" && mv "$tmpdir/migration.tmp" "$tmpdir/p1-migration-summary.json" +if VERIFICATION_RUN_ID="$run_id" VERIFICATION_STARTED_AT="$started_at" LITELLM_ARTIFACTS_DIR="$tmpdir" "$script_dir/aggregate-verification-summary.sh" >/dev/null 2>&1; then + echo "aggregate accepted a failed migration" >&2; exit 1 +fi +make_reports +jq '.limiter_source="provider"' "$tmpdir/p1-ha-summary.json" > "$tmpdir/ha.tmp" && mv "$tmpdir/ha.tmp" "$tmpdir/p1-ha-summary.json" +if VERIFICATION_RUN_ID="$run_id" VERIFICATION_STARTED_AT="$started_at" LITELLM_ARTIFACTS_DIR="$tmpdir" "$script_dir/aggregate-verification-summary.sh" >/dev/null 2>&1; then + echo "aggregate accepted a forged limiter claim" >&2; exit 1 +fi + if [[ "$with_running_stack" == true ]]; then negative_summary="$tmpdir/cleanup-negative.json" if VERIFICATION_RUN_ID="$run_id" LITELLM_SMOKE_SUMMARY_FILE="$negative_summary" \ diff --git a/docker_litellm/demo/scripts/verify-migration-concurrency.sh b/docker_litellm/demo/scripts/verify-migration-concurrency.sh new file mode 100755 index 0000000..d7f4414 --- /dev/null +++ b/docker_litellm/demo/scripts/verify-migration-concurrency.sh @@ -0,0 +1,45 @@ +#!/usr/bin/env bash +# Prove that two migration-only jobs can contend safely before replicas start. +set -euo pipefail + +script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +demo_dir="$(cd "${script_dir}/.." && pwd)" +source "${script_dir}/verification-lib.sh" +env_file="${LITELLM_SMOKE_ENV_FILE:-${demo_dir}/.env}" +summary_file="${demo_dir}/artifacts/p1-migration-concurrency.json" +run_id="${VERIFICATION_RUN_ID:?VERIFICATION_RUN_ID is required}" +tmpdir="" +image_ref="" +result="failed" +phase="initializing" + +verification_invalidate_report "$summary_file" +cleanup() { + local rc=$? tmp + trap - EXIT + tmp="${summary_file}.tmp.$$" + jq -n --arg run_id "$run_id" --arg commit "$(git -C "$demo_dir/../.." rev-parse HEAD)" \ + --arg image_id "$(docker image inspect "$image_ref" --format '{{.Id}}' 2>/dev/null || true)" \ + --arg tested_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" --arg result "$result" --arg phase "$phase" \ + '{verification_run_id:$run_id,commit:$commit,image_id:$image_id,tested_at:$tested_at,mode:"migration",result:$result,phase:$phase,concurrent_migration:($result == "passed"),content_redacted:true}' > "$tmp" && chmod 600 "$tmp" && mv "$tmp" "$summary_file" + [[ -z "$tmpdir" ]] || rm -rf "$tmpdir" + return "$rc" +} +trap cleanup EXIT + +tmpdir="$(mktemp -d "${TMPDIR:-/tmp}/litellm-migration-concurrency.XXXXXX")" +chmod 700 "$tmpdir" +verification_prepare_environment "$env_file" "${demo_dir}/docker-compose.litellm.yml" "$tmpdir" +image_ref="$(verification_env LITELLM_IMAGE)" +compose=(docker compose --env-file "$env_file" -f "${demo_dir}/docker-compose.litellm.yml") +"${compose[@]}" up -d --wait postgres redis +phase="running_concurrent_jobs" +first="$("${compose[@]}" --profile migrate run -d --no-deps litellm-migrate)" +second="$("${compose[@]}" --profile migrate run -d --no-deps litellm-migrate)" +[[ -n "$first" && -n "$second" && "$first" != "$second" ]] +docker wait "$first" "$second" > "$tmpdir/exit-codes" +[[ "$(tr -d '[:space:]' < "$tmpdir/exit-codes")" == "00" ]] +! "${compose[@]}" ps --services --status running | rg -q '^litellm-[12]$' +phase="completed" +result="passed" +echo "PASS concurrent migration jobs completed before proxy replicas started." diff --git a/docker_litellm/demo/scripts/verify-p1.sh b/docker_litellm/demo/scripts/verify-p1.sh index 02b5131..f57ee7e 100755 --- a/docker_litellm/demo/scripts/verify-p1.sh +++ b/docker_litellm/demo/scripts/verify-p1.sh @@ -4,8 +4,10 @@ set -euo pipefail script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" demo_dir="$(cd "${script_dir}/.." && pwd)" -run_id="${VERIFICATION_RUN_ID:-$(python3 -c 'import secrets; print("p1-" + secrets.token_hex(16))')}" +source "${script_dir}/verification-lib.sh" +run_id="$(verification_new_run_id)" export VERIFICATION_RUN_ID="$run_id" +export VERIFICATION_STARTED_AT="$(date -u +%Y-%m-%dT%H:%M:%SZ)" # P1's documented local provider mapping is explicit. It is non-secret and # prevents a template/default mismatch from silently selecting another SDK. export LITELLM_SMOKE_UPSTREAM_PROVIDER="${LITELLM_SMOKE_UPSTREAM_PROVIDER:-deepseek}" @@ -14,7 +16,12 @@ cleanup_stack() { "${compose[@]}" --profile single --profile ha --profile migrat trap cleanup_stack EXIT cleanup_stack +for report in p1-migration-summary.json p1-single-summary.json p1-ha-summary.json p1-redis-recovery.json p1-final-summary.json; do + verification_invalidate_report "${demo_dir}/artifacts/${report}" +done +"${script_dir}/test-verification-gates.sh" "${script_dir}/run-migration.sh" +"${script_dir}/verify-migration-concurrency.sh" "${script_dir}/run-migration.sh" "${compose[@]}" --profile single up -d --wait postgres redis litellm-1 "${script_dir}/smoke-baseline.sh" --mode single From 6f943645d86ed083ae174b8dd057bce082d8962c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 19:06:50 +0800 Subject: [PATCH 49/87] =?UTF-8?q?fix:=20=E7=BB=91=E5=AE=9A=20LiteLLM=20?= =?UTF-8?q?=E9=99=90=E6=B5=81=E5=93=8D=E5=BA=94=E5=A4=B4=E8=AF=81=E6=8D=AE?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/demo/scripts/smoke-baseline.sh | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh index 72079ed..436ba3f 100755 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -491,12 +491,11 @@ wait_for_rejection() { } assert_proxy_limiter_response() { - local response_file="$1" http_code="$2" limit_kind="$3" since="$4" expected_type + local response_file="$1" headers_file="$2" http_code="$3" limit_kind="$4" since="$5" expected_type if [[ "$limit_kind" == rpm ]]; then expected_type=requests; else expected_type=tokens; fi [[ "$http_code" == "429" ]] && - jq -e --arg expected_type "$expected_type" ' - [(.error.rate_limit_type? // .rate_limit_type? // empty)] | index($expected_type) != null - ' "$response_file" >/dev/null && + rg -qi -- "^x-ratelimit-.*-(limit|remaining)-${expected_type}:" "$headers_file" && + jq -e '(.error // .detail // .message // "") | tostring | test("rate limit|limit"; "i")' "$response_file" >/dev/null && docker compose --env-file "$ENV_FILE" -f "$DEMO_DIR/docker-compose.litellm.yml" logs --no-color --since "$since" litellm-1 litellm-2 | rg -q 'parallel_request_limiter_v3|ProxyRateLimitError' } @@ -721,8 +720,9 @@ if [[ "$MODE" == "ha" ]]; then make_key_action_payload delete "$rate_key_file" "$tmpdir/rate-key-cleanup.json" rate_limiter_since="$(date -u +%Y-%m-%dT%H:%M:%SZ)" request_data_post "$BASE_URL" "$rate_headers" /v1/chat/completions "$tmpdir/chat-request.json" "$tmpdir/rate-first.json" - rate_code="$(curl --silent --show-error --max-time 30 --request POST "$PEER_URL/v1/chat/completions" --header "@$rate_headers" --data-binary "@$tmpdir/chat-request.json" --output "$tmpdir/rate-second.json" --write-out '%{http_code}' || true)" - assert_proxy_limiter_response "$tmpdir/rate-second.json" "$rate_code" rpm "$rate_limiter_since" || { echo "shared RPM limiter was not a LiteLLM proxy 429" >&2; exit 1; } + rate_code="$(curl --silent --show-error --max-time 30 --request POST "$PEER_URL/v1/chat/completions" --header "@$rate_headers" --data-binary "@$tmpdir/chat-request.json" --dump-header "$tmpdir/rate-second.headers" --output "$tmpdir/rate-second.json" --write-out '%{http_code}' || true)" + chmod 600 "$tmpdir/rate-second.headers" + assert_proxy_limiter_response "$tmpdir/rate-second.json" "$tmpdir/rate-second.headers" "$rate_code" rpm "$rate_limiter_since" || { echo "shared RPM limiter was not a LiteLLM proxy 429" >&2; exit 1; } shared_rpm_limit_result="passed" limiter_source="litellm_proxy" echo "PASS HA shared RPM: peer rejected the second request with 429." @@ -742,8 +742,9 @@ if [[ "$MODE" == "ha" ]]; then make_key_action_payload delete "$enforcement_key_file" "$tmpdir/enforcement-key-cleanup.json" tpm_limiter_since="$(date -u +%Y-%m-%dT%H:%M:%SZ)" request_data_post "$BASE_URL" "$enforcement_headers" /v1/chat/completions "$tmpdir/chat-request.json" "$tmpdir/budget-first.json" - tpm_code="$(curl --silent --show-error --max-time 30 --request POST "$PEER_URL/v1/chat/completions" --header "@$enforcement_headers" --data-binary "@$tmpdir/chat-request.json" --output "$tmpdir/tpm-second.json" --write-out '%{http_code}' || true)" - assert_proxy_limiter_response "$tmpdir/tpm-second.json" "$tpm_code" tpm "$tpm_limiter_since" || { echo "shared TPM limiter was not a LiteLLM proxy 429" >&2; exit 1; } + tpm_code="$(curl --silent --show-error --max-time 30 --request POST "$PEER_URL/v1/chat/completions" --header "@$enforcement_headers" --data-binary "@$tmpdir/chat-request.json" --dump-header "$tmpdir/tpm-second.headers" --output "$tmpdir/tpm-second.json" --write-out '%{http_code}' || true)" + chmod 600 "$tmpdir/tpm-second.headers" + assert_proxy_limiter_response "$tmpdir/tpm-second.json" "$tmpdir/tpm-second.headers" "$tpm_code" tpm "$tpm_limiter_since" || { echo "shared TPM limiter was not a LiteLLM proxy 429" >&2; exit 1; } shared_tpm_limit_result="passed" echo "PASS HA shared TPM enforcement: peer rejected the second request with 429." fi From bb0f645f0388cbef3cbbc00d7d7ed159661635ab Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 19:10:32 +0800 Subject: [PATCH 50/87] =?UTF-8?q?fix:=20=E5=AE=8C=E5=96=84=20P1=20?= =?UTF-8?q?=E6=8A=A5=E5=91=8A=E9=97=A8=E7=A6=81=E8=AF=81=E6=8D=AE?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../scripts/aggregate-verification-summary.sh | 4 ++-- .../demo/scripts/test-verification-gates.sh | 24 +++++++++++-------- docker_litellm/demo/scripts/verify-p1.sh | 2 +- 3 files changed, 17 insertions(+), 13 deletions(-) diff --git a/docker_litellm/demo/scripts/aggregate-verification-summary.sh b/docker_litellm/demo/scripts/aggregate-verification-summary.sh index 9e56966..73eae95 100755 --- a/docker_litellm/demo/scripts/aggregate-verification-summary.sh +++ b/docker_litellm/demo/scripts/aggregate-verification-summary.sh @@ -79,10 +79,10 @@ mkdir -p "$(dirname "$output")" chmod 700 "$(dirname "$output")" jq -n \ --arg run_id "$run_id" --arg commit "$commit" --arg image_id "$image_id" \ - --arg generated_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ + --arg tested_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ --slurpfile migration "$migration_report" --slurpfile single "$single_report" \ --slurpfile ha "$ha_report" --slurpfile redis "$redis_report" \ - '{verification_run_id:$run_id,commit:$commit,image_id:$image_id,generated_at:$generated_at,result:"passed",phase:"completed",migration:$migration[0],single:$single[0],ha:$ha[0],redis_recovery:$redis[0],content_redacted:true,local_only:true}' \ + '{verification_run_id:$run_id,commit:$commit,image_id:$image_id,tested_at:$tested_at,generated_at:$tested_at,result:"passed",phase:"completed",migration:$migration[0],single:$single[0],ha:$ha[0],redis_recovery:$redis[0],content_redacted:true,local_only:true}' \ > "$output" chmod 600 "$output" echo "PASS aggregate summary: $output" diff --git a/docker_litellm/demo/scripts/test-verification-gates.sh b/docker_litellm/demo/scripts/test-verification-gates.sh index 35cc203..1fa319d 100755 --- a/docker_litellm/demo/scripts/test-verification-gates.sh +++ b/docker_litellm/demo/scripts/test-verification-gates.sh @@ -57,18 +57,22 @@ rg -Fq 'PAYLOAD=$(touch ' "$tmpdir/effective.env" # The aggregate gate must also reject structurally plausible but incomplete # evidence: a missing timestamp, a failed migration, or a forged limiter claim. started_at="2026-01-01T00:00:00Z" +current_commit="$(git -C "$demo_dir/../.." rev-parse HEAD)" make_reports() { - jq -n --arg run_id "$run_id" --arg started_at "$started_at" \ - '{verification_run_id:$run_id,commit:"head",image_id:"image",tested_at:$started_at,mode:"migration",result:"passed",phase:"completed",content_redacted:true,proxy_replicas_started:false}' > "$tmpdir/p1-migration-summary.json" - jq -n --arg run_id "$run_id" --arg started_at "$started_at" \ - '{verification_run_id:$run_id,commit:"head",image_id:"image",tested_at:$started_at,mode:"single",result:"passed",phase:"completed",content_redacted:true,migration:"passed",chat:"passed",stream:"passed",tool:"passed",usage:"passed",block:"passed",delete:"passed",cleanup:"passed",security_scan:"passed",content_logging_scan:"passed"}' > "$tmpdir/p1-single-summary.json" - jq -n --arg run_id "$run_id" --arg started_at "$started_at" \ - '{verification_run_id:$run_id,commit:"head",image_id:"image",tested_at:$started_at,mode:"ha",result:"passed",phase:"completed",content_redacted:true,migration:"passed",chat:"passed",stream:"passed",tool:"passed",usage:"passed",block:"passed",delete:"passed",shared_rpm_limit:"passed",shared_tpm_limit:"passed",shared_spend_log_visibility:"passed",idempotency_recovery:"passed",limiter_source:"litellm_proxy",cleanup:"passed",security_scan:"passed",content_logging_scan:"passed"}' > "$tmpdir/p1-ha-summary.json" - jq -n --arg run_id "$run_id" --arg started_at "$started_at" \ - '{verification_run_id:$run_id,commit:"head",image_id:"image",tested_at:$started_at,mode:"ha",result:"passed",phase:"completed",redis_recovery:"passed",content_redacted:true,security_scan:"passed"}' > "$tmpdir/p1-redis-recovery.json" + jq -n --arg run_id "$run_id" --arg commit "$current_commit" --arg started_at "$started_at" \ + '{verification_run_id:$run_id,commit:$commit,image_id:"image",tested_at:$started_at,mode:"migration",result:"passed",phase:"completed",content_redacted:true,proxy_replicas_started:false}' > "$tmpdir/p1-migration-summary.json" + jq -n --arg run_id "$run_id" --arg commit "$current_commit" --arg started_at "$started_at" \ + '{verification_run_id:$run_id,commit:$commit,image_id:"image",tested_at:$started_at,mode:"migration",result:"passed",phase:"completed",concurrent_migration:true,content_redacted:true}' > "$tmpdir/p1-migration-concurrency.json" + jq -n --arg run_id "$run_id" --arg commit "$current_commit" --arg started_at "$started_at" \ + '{verification_run_id:$run_id,commit:$commit,image_id:"image",tested_at:$started_at,mode:"single",result:"passed",phase:"completed",content_redacted:true,migration:"passed",chat:"passed",stream:"passed",tool:"passed",usage:"passed",block:"passed",delete:"passed",cleanup:"passed",security_scan:"passed",content_logging_scan:"passed"}' > "$tmpdir/p1-single-summary.json" + jq -n --arg run_id "$run_id" --arg commit "$current_commit" --arg started_at "$started_at" \ + '{verification_run_id:$run_id,commit:$commit,image_id:"image",tested_at:$started_at,mode:"ha",result:"passed",phase:"completed",content_redacted:true,migration:"passed",chat:"passed",stream:"passed",tool:"passed",usage:"passed",block:"passed",delete:"passed",shared_rpm_limit:"passed",shared_tpm_limit:"passed",shared_spend_log_visibility:"passed",idempotency_recovery:"passed",limiter_source:"litellm_proxy",cleanup:"passed",security_scan:"passed",content_logging_scan:"passed"}' > "$tmpdir/p1-ha-summary.json" + jq -n --arg run_id "$run_id" --arg commit "$current_commit" --arg started_at "$started_at" \ + '{verification_run_id:$run_id,commit:$commit,image_id:"image",tested_at:$started_at,mode:"ha",result:"passed",phase:"completed",redis_recovery:"passed",content_redacted:true,security_scan:"passed"}' > "$tmpdir/p1-redis-recovery.json" } -# These are deliberately rejected before any report can become final. They use -# a synthetic commit, so the current checkout mismatch is an additional guard. +# These are deliberately rejected before any report can become final. The +# fixtures use the real checkout commit so each mutation exercises its named +# gate rather than merely the unrelated commit-mismatch guard. make_reports jq 'del(.tested_at)' "$tmpdir/p1-single-summary.json" > "$tmpdir/single.tmp" && mv "$tmpdir/single.tmp" "$tmpdir/p1-single-summary.json" if VERIFICATION_RUN_ID="$run_id" VERIFICATION_STARTED_AT="$started_at" LITELLM_ARTIFACTS_DIR="$tmpdir" "$script_dir/aggregate-verification-summary.sh" >/dev/null 2>&1; then diff --git a/docker_litellm/demo/scripts/verify-p1.sh b/docker_litellm/demo/scripts/verify-p1.sh index f57ee7e..40496cf 100755 --- a/docker_litellm/demo/scripts/verify-p1.sh +++ b/docker_litellm/demo/scripts/verify-p1.sh @@ -16,7 +16,7 @@ cleanup_stack() { "${compose[@]}" --profile single --profile ha --profile migrat trap cleanup_stack EXIT cleanup_stack -for report in p1-migration-summary.json p1-single-summary.json p1-ha-summary.json p1-redis-recovery.json p1-final-summary.json; do +for report in p1-migration-summary.json p1-migration-concurrency.json p1-single-summary.json p1-ha-summary.json p1-redis-recovery.json p1-final-summary.json; do verification_invalidate_report "${demo_dir}/artifacts/${report}" done "${script_dir}/test-verification-gates.sh" From cf500f6194e881d72904e12b21649ec05be8e1d6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 19:18:59 +0800 Subject: [PATCH 51/87] =?UTF-8?q?fix:=20=E6=A0=A1=E9=AA=8C=20LiteLLM=20?= =?UTF-8?q?=E9=99=90=E6=B5=81=E7=B1=BB=E5=9E=8B=E5=93=8D=E5=BA=94=E8=AF=81?= =?UTF-8?q?=E6=8D=AE?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/demo/scripts/smoke-baseline.sh | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh index 436ba3f..08e5495 100755 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -494,10 +494,15 @@ assert_proxy_limiter_response() { local response_file="$1" headers_file="$2" http_code="$3" limit_kind="$4" since="$5" expected_type if [[ "$limit_kind" == rpm ]]; then expected_type=requests; else expected_type=tokens; fi [[ "$http_code" == "429" ]] && - rg -qi -- "^x-ratelimit-.*-(limit|remaining)-${expected_type}:" "$headers_file" && - jq -e '(.error // .detail // .message // "") | tostring | test("rate limit|limit"; "i")' "$response_file" >/dev/null && + # LiteLLM 1.97.0's parallel_request_limiter_v3 raises ProxyRateLimitError + # with the stable proxy-only rate_limit_type header. Provider 429s do not + # synthesize this header or the matching "Limit type" detail below. + rg -qi -- "^rate_limit_type:[[:space:]]*${expected_type}[[:space:]]*$" "$headers_file" && + jq -e --arg expected "$expected_type" '(.detail // .error.detail // .error.message // .error // .message // "") | tostring | test("Rate limit exceeded.*Limit type: " + $expected; "i")' "$response_file" >/dev/null && + # The access log is bounded to this request window and confirms that this + # exact proxy instance emitted a local 429; no prompt/response is copied. docker compose --env-file "$ENV_FILE" -f "$DEMO_DIR/docker-compose.litellm.yml" logs --no-color --since "$since" litellm-1 litellm-2 | - rg -q 'parallel_request_limiter_v3|ProxyRateLimitError' + rg -q 'POST /v1/chat/completions.* 429|HTTP/1\.[01]" 429' } write_summary() { From 3913cf241869a9d00ee35f48daba595b01935cbe Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 19:25:23 +0800 Subject: [PATCH 52/87] =?UTF-8?q?docs:=20=E6=98=8E=E7=A1=AE=20P1=20Redis?= =?UTF-8?q?=20=E4=B8=8E=E6=8A=A5=E5=91=8A=E9=97=A8=E7=A6=81=E8=AF=AD?= =?UTF-8?q?=E4=B9=89?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docker_litellm/README.md b/docker_litellm/README.md index ae207ce..2205cb1 100644 --- a/docker_litellm/README.md +++ b/docker_litellm/README.md @@ -13,7 +13,7 @@ P1 默认不构建 LiteLLM Dashboard 静态资源:固定源码在当前构建 | Prisma Python client | `0.15.0` | LiteLLM 连接 PostgreSQL 所需客户端,兼容基础镜像的 Python 3.13 | | 本地产物镜像 | `quay.io/labnow/litellm:1.97.0-ead62528e607` | P1 Compose 的唯一 LiteLLM 默认镜像 | | PostgreSQL | `postgres:17-alpine@sha256:742f40ea20b9ff2ff31db5458d127452988a2164df9e17441e191f3b72252193` | 用户、凭证、模型、虚拟 key 与 spend 持久化 | -| Redis | `redis:7.4-alpine@sha256:e7723ff73d963f5cc6d9c4643ea3d989527a402a319239054e9472a7fb9219a2` | 副本共享认证缓存、限流、Spend counter 和协调缓存 | +| Redis | `redis:7.4-alpine@sha256:e7723ff73d963f5cc6d9c4643ea3d989527a402a319239054e9472a7fb9219a2` | 副本共享认证缓存、RPM/TPM limiter 与协调缓存;SpendLog 的事实源是 PostgreSQL | | OpenClaw(P2 参考) | `2026.5.10-beta.1` / `eed75ed47f47deb18c9d093a2e638c9bb0bedf14` | 仅为下一阶段黄金适配器保留版本基线;P1 不启动或实现 Adapter | 镜像构建会对 wheel 自带的 LiteLLM Prisma schema 运行 `prisma generate`,并把生成的查询引擎固定在 `/opt/litellm/.cache`;没有该步骤,或将该缓存随 `/root/.cache` 清理,代理会在 PostgreSQL startup 时报缺少 Prisma binaries 或无法连接查询引擎。 @@ -47,7 +47,7 @@ cp .env.example .env docker compose --env-file .env -f docker-compose.litellm.yml --profile single up -d ``` -迁移与代理启动刻意分离。标准真实验收由一个统一入口执行:它生成非敏感 `verification_run_id`,显式连续运行 migration(两次)、single、HA、Redis 恢复和严格聚合;任一步失败都会停止且使对应旧报告失效。 +迁移与代理启动刻意分离。标准真实验收由一个统一入口执行:它生成新的非敏感 `verification_run_id`,先失效所有旧输入/最终报告,再运行 migration(两次)、并发 migration job、single、HA、Redis 恢复和严格聚合;任一步失败都会停止且保留当前失败报告。 ```bash ./scripts/verify-p1.sh @@ -84,7 +84,7 @@ cd docker_litellm/demo ./scripts/test-verification-gates.sh ``` -在全新 checkout 中按上述标准命令执行时,脚本自己生成而非复用历史文件:`p1-migration-summary.json`、`p1-single-summary.json`、`p1-ha-summary.json`、`p1-redis-recovery.json` 与最终 `p1-final-summary.json`(均在被忽略的 `artifacts/`)。聚合脚本只接受当前 `HEAD`、同一 `verification_run_id`、相同 image ID、`result=passed`、`phase=completed` 且脱敏的四份输入;任何缺失、失败、跳过或模式不符都会被拒绝。 +在全新 checkout 中按上述标准命令执行时,脚本自己生成而非复用历史文件:`p1-migration-summary.json`、`p1-migration-concurrency.json`、`p1-single-summary.json`、`p1-ha-summary.json`、`p1-redis-recovery.json` 与最终 `p1-final-summary.json`(均在被忽略的 `artifacts/`)。聚合脚本只接受当前 `HEAD`、同一 `verification_run_id`、相同 image ID、正确 mode、启动后 `tested_at`、`result=passed`、`phase=completed` 且脱敏的输入;任何缺失、失败、跳过、过期或模式不符都会被拒绝。 `--security-check` 不读取 `.env`、不启动服务也不发送上游请求;它拒绝 inline header、secret-bearing `jq --arg`、`set -x`、Compose 上游凭据注入和 Redis 密码命令行展开,并检查 Docker Secret、0600 临时文件与退出清理约束。`test-verification-gates.sh` 验证历史 PASS 失效、前置失败报告与 dotenv 命令替换不执行;在已启动 single 栈中追加 `--with-running-stack` 会以真实 404 删除请求证明 cleanup 不会生成 PASS。 @@ -98,7 +98,7 @@ cd docker_litellm/demo ## Readiness 与 Redis 结论 -LiteLLM `v1.97.0-dev.1` 的公开 `/health/readiness` 仅返回服务与数据库连通性,不将 Redis 纳入公开 readiness。因此 Compose 健康检查只能确认 LiteLLM + PostgreSQL;`smoke-baseline.sh` 额外从每个 LiteLLM 容器执行 Redis `PING`。若 Redis 不可用,P1 的多副本认证缓存、限流、Spend counter 与协调结论无效,应将该运行组合标记为 `blocked`,不能降级宣称为高可用。Redis 恢复后,LiteLLM 的认证缓存 circuit breaker 需要经过 `REDIS_CIRCUIT_BREAKER_RECOVERY_TIMEOUT` 后才会重新探测;P1 默认设为 5 秒,并要求恢复后再次跑 HA smoke。 +LiteLLM `v1.97.0-dev.1` 的公开 `/health/readiness` 仅返回服务与数据库连通性,不将 Redis 纳入公开 readiness。因此 Compose 健康检查只能确认 LiteLLM + PostgreSQL;`smoke-baseline.sh` 额外从每个 LiteLLM 容器执行 Redis `PING`。若 Redis 不可用,P1 的多副本认证缓存、RPM/TPM limiter 与协调结论无效,应将该运行组合标记为 `blocked`,不能降级宣称为高可用。Redis 恢复后,LiteLLM 的认证缓存 circuit breaker 需要经过 `REDIS_CIRCUIT_BREAKER_RECOVERY_TIMEOUT` 后才会重新探测;P1 默认设为 5 秒,并要求恢复后再次跑 HA smoke。跨副本 SpendLog 只证明 PostgreSQL 可见性,不是 Redis Spend counter 或预算准入控制证据。 ## 常见问题 From f9ec5ea5b3f42489a3c92342eb094b2de5467e69 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 19:52:25 +0800 Subject: [PATCH 53/87] =?UTF-8?q?fix:=20=E5=BC=BA=E5=8C=96=20P1=20?= =?UTF-8?q?=E6=80=BB=E6=8E=A7=E9=AA=8C=E6=94=B6=E9=97=A8=E7=A6=81?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../demo/docker-compose.litellm.yml | 3 +- .../scripts/aggregate-verification-summary.sh | 7 ++- docker_litellm/demo/scripts/smoke-baseline.sh | 11 ++++ .../demo/scripts/test-verification-gates.sh | 20 ++++++- .../scripts/verify-migration-concurrency.sh | 56 +++++++++++++++++-- docker_litellm/demo/scripts/verify-p1.sh | 1 + docker_litellm/work/run-migration-locked.py | 54 ++++++++++++++++++ 7 files changed, 142 insertions(+), 10 deletions(-) create mode 100755 docker_litellm/work/run-migration-locked.py diff --git a/docker_litellm/demo/docker-compose.litellm.yml b/docker_litellm/demo/docker-compose.litellm.yml index b24020b..87b2cd4 100644 --- a/docker_litellm/demo/docker-compose.litellm.yml +++ b/docker_litellm/demo/docker-compose.litellm.yml @@ -18,6 +18,7 @@ x-litellm-common: &litellm-common - ./config.yaml:/opt/litellm/config.yaml:ro - ./config.migrate.yaml:/opt/litellm/config.migrate.yaml:ro - ../work/start-litellm.sh:/opt/utils/start-litellm.sh:ro + - ../work/run-migration-locked.py:/opt/utils/run-migration-locked.py:ro secrets: - redis_password depends_on: @@ -43,7 +44,7 @@ services: <<: *litellm-common container_name: ${LITELLM_MIGRATE_CONTAINER_NAME:-svc-litellm-migrate} profiles: ["migrate"] - command: ["/bin/bash", "/opt/utils/start-litellm.sh", "--config", "config.migrate.yaml", "--skip_server_startup", "--enforce_prisma_migration_check"] + command: ["python3", "/opt/utils/run-migration-locked.py", "--config", "config.migrate.yaml", "--skip_server_startup", "--enforce_prisma_migration_check"] healthcheck: disable: true depends_on: diff --git a/docker_litellm/demo/scripts/aggregate-verification-summary.sh b/docker_litellm/demo/scripts/aggregate-verification-summary.sh index 73eae95..c789982 100755 --- a/docker_litellm/demo/scripts/aggregate-verification-summary.sh +++ b/docker_litellm/demo/scripts/aggregate-verification-summary.sh @@ -34,6 +34,8 @@ jq -e --arg commit "$commit" --arg run_id "$run_id" --arg started_at "$started_a jq -e --arg commit "$commit" --arg run_id "$run_id" --arg started_at "$started_at" ' .mode == "migration" and .result == "passed" and .phase == "completed" and .concurrent_migration == true and + .actual_overlap == true and .lock_wait_observed == true and .exclusive_lock == true and + .max_lock_holders == 1 and .migration_execution_count == 2 and .proxy_replicas_started == false and .verification_run_id == $run_id and .commit == $commit and (.image_id | type == "string" and length > 0) and (.tested_at | type == "string" and . >= $started_at) and .content_redacted == true ' "$concurrency_report" >/dev/null @@ -73,6 +75,7 @@ image_id="$(jq -r '.image_id' "$migration_report")" [[ "$image_id" == "$(jq -r '.image_id' "$single_report")" ]] || { echo "single image ID differs" >&2; exit 1; } [[ "$image_id" == "$(jq -r '.image_id' "$ha_report")" ]] || { echo "HA image ID differs" >&2; exit 1; } [[ "$image_id" == "$(jq -r '.image_id' "$redis_report")" ]] || { echo "Redis report image ID differs" >&2; exit 1; } +[[ "$image_id" == "$(jq -r '.image_id' "$concurrency_report")" ]] || { echo "concurrency image ID differs" >&2; exit 1; } umask 077 mkdir -p "$(dirname "$output")" @@ -80,9 +83,9 @@ chmod 700 "$(dirname "$output")" jq -n \ --arg run_id "$run_id" --arg commit "$commit" --arg image_id "$image_id" \ --arg tested_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ - --slurpfile migration "$migration_report" --slurpfile single "$single_report" \ + --slurpfile migration "$migration_report" --slurpfile concurrency "$concurrency_report" --slurpfile single "$single_report" \ --slurpfile ha "$ha_report" --slurpfile redis "$redis_report" \ - '{verification_run_id:$run_id,commit:$commit,image_id:$image_id,tested_at:$tested_at,generated_at:$tested_at,result:"passed",phase:"completed",migration:$migration[0],single:$single[0],ha:$ha[0],redis_recovery:$redis[0],content_redacted:true,local_only:true}' \ + '{verification_run_id:$run_id,commit:$commit,image_id:$image_id,tested_at:$tested_at,generated_at:$tested_at,result:"passed",phase:"completed",migration:$migration[0],migration_concurrency:$concurrency[0],single:$single[0],ha:$ha[0],redis_recovery:$redis[0],content_redacted:true,local_only:true}' \ > "$output" chmod 600 "$output" echo "PASS aggregate summary: $output" diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh index 08e5495..5d79349 100755 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -64,6 +64,13 @@ fi # It invalidates any stale report and writes a non-passing, redacted result. early_failure_cleanup() { local exit_code=$? + # A security inspection is strictly read-only. This includes intentionally + # failing inspections used by the negative gate: neither path may replace + # a producer report from an earlier real smoke. + if [[ "$SECURITY_CHECK" == true ]]; then + trap - EXIT + return "$exit_code" + fi # Ignore any nested EXIT delivery while preserving the report just written. trap '' EXIT if command -v jq >/dev/null; then @@ -112,6 +119,10 @@ security_check() { && rg -q 'rm -rf "\$tmpdir"' "$0" \ || unsafe=1 + # Test-only fault injection validates the failure path preserves reports. + # It is deliberately limited to this read-only static checker. + [[ "${LITELLM_SECURITY_CHECK_FORCE_FAILURE:-0}" != "1" ]] || unsafe=1 + if ((unsafe)); then echo "FAIL security negative check: unsafe secret transport or cleanup invariant" >&2 return 1 diff --git a/docker_litellm/demo/scripts/test-verification-gates.sh b/docker_litellm/demo/scripts/test-verification-gates.sh index 1fa319d..52ddcbc 100755 --- a/docker_litellm/demo/scripts/test-verification-gates.sh +++ b/docker_litellm/demo/scripts/test-verification-gates.sh @@ -37,6 +37,19 @@ if VERIFICATION_RUN_ID="$run_id" LITELLM_SMOKE_ENV_FILE="$tmpdir/absent.env" LIT fi jq -e '.result == "failed" and .phase == "precondition_failed"' "$tmpdir/precondition.json" >/dev/null +# Security inspection must remain read-only on both outcomes. Force a static +# failure and prove that its target report is byte-for-byte unchanged. +security_report="$tmpdir/security-existing.json" +printf '%s\n' '{"preserved":true}' > "$security_report" +security_before="$(shasum -a 256 "$security_report" | awk '{print $1}')" +if LITELLM_SECURITY_CHECK_FORCE_FAILURE=1 LITELLM_SMOKE_SUMMARY_FILE="$security_report" \ + "$script_dir/smoke-baseline.sh" --security-check >/dev/null 2>&1; then + echo "forced security failure unexpectedly passed" >&2 + exit 1 +fi +security_after="$(shasum -a 256 "$security_report" | awk '{print $1}')" +[[ "$security_before" == "$security_after" ]] || { echo "failed security check modified a report" >&2; exit 1; } + # Cleanup failures must never be normalized to PASS. This checks the curl # transport invariant directly without sending a request. rg -q 'cleanup_request_admin.*\(\)' "$script_dir/smoke-baseline.sh" @@ -62,7 +75,7 @@ make_reports() { jq -n --arg run_id "$run_id" --arg commit "$current_commit" --arg started_at "$started_at" \ '{verification_run_id:$run_id,commit:$commit,image_id:"image",tested_at:$started_at,mode:"migration",result:"passed",phase:"completed",content_redacted:true,proxy_replicas_started:false}' > "$tmpdir/p1-migration-summary.json" jq -n --arg run_id "$run_id" --arg commit "$current_commit" --arg started_at "$started_at" \ - '{verification_run_id:$run_id,commit:$commit,image_id:"image",tested_at:$started_at,mode:"migration",result:"passed",phase:"completed",concurrent_migration:true,content_redacted:true}' > "$tmpdir/p1-migration-concurrency.json" + '{verification_run_id:$run_id,commit:$commit,image_id:"image",tested_at:$started_at,mode:"migration",result:"passed",phase:"completed",concurrent_migration:true,actual_overlap:true,lock_wait_observed:true,exclusive_lock:true,max_lock_holders:1,migration_execution_count:2,proxy_replicas_started:false,content_redacted:true}' > "$tmpdir/p1-migration-concurrency.json" jq -n --arg run_id "$run_id" --arg commit "$current_commit" --arg started_at "$started_at" \ '{verification_run_id:$run_id,commit:$commit,image_id:"image",tested_at:$started_at,mode:"single",result:"passed",phase:"completed",content_redacted:true,migration:"passed",chat:"passed",stream:"passed",tool:"passed",usage:"passed",block:"passed",delete:"passed",cleanup:"passed",security_scan:"passed",content_logging_scan:"passed"}' > "$tmpdir/p1-single-summary.json" jq -n --arg run_id "$run_id" --arg commit "$current_commit" --arg started_at "$started_at" \ @@ -88,6 +101,11 @@ jq '.limiter_source="provider"' "$tmpdir/p1-ha-summary.json" > "$tmpdir/ha.tmp" if VERIFICATION_RUN_ID="$run_id" VERIFICATION_STARTED_AT="$started_at" LITELLM_ARTIFACTS_DIR="$tmpdir" "$script_dir/aggregate-verification-summary.sh" >/dev/null 2>&1; then echo "aggregate accepted a forged limiter claim" >&2; exit 1 fi +make_reports +jq '.image_id="other-image"' "$tmpdir/p1-migration-concurrency.json" > "$tmpdir/concurrency.tmp" && mv "$tmpdir/concurrency.tmp" "$tmpdir/p1-migration-concurrency.json" +if VERIFICATION_RUN_ID="$run_id" VERIFICATION_STARTED_AT="$started_at" LITELLM_ARTIFACTS_DIR="$tmpdir" "$script_dir/aggregate-verification-summary.sh" >/dev/null 2>&1; then + echo "aggregate accepted a concurrency report with a mismatched image" >&2; exit 1 +fi if [[ "$with_running_stack" == true ]]; then negative_summary="$tmpdir/cleanup-negative.json" diff --git a/docker_litellm/demo/scripts/verify-migration-concurrency.sh b/docker_litellm/demo/scripts/verify-migration-concurrency.sh index d7f4414..90e8974 100755 --- a/docker_litellm/demo/scripts/verify-migration-concurrency.sh +++ b/docker_litellm/demo/scripts/verify-migration-concurrency.sh @@ -1,5 +1,5 @@ #!/usr/bin/env bash -# Prove that two migration-only jobs can contend safely before replicas start. +# Prove actual overlapping migration jobs serialize on PostgreSQL's advisory lock. set -euo pipefail script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" @@ -12,16 +12,28 @@ tmpdir="" image_ref="" result="failed" phase="initializing" +first="" +second="" +actual_overlap=false +lock_wait_observed=false +exclusive_lock=false +max_lock_holders=0 +migration_execution_count=0 verification_invalidate_report "$summary_file" cleanup() { local rc=$? tmp trap - EXIT + [[ -z "$first" ]] || docker rm "$first" >/dev/null 2>&1 || true + [[ -z "$second" ]] || docker rm "$second" >/dev/null 2>&1 || true tmp="${summary_file}.tmp.$$" jq -n --arg run_id "$run_id" --arg commit "$(git -C "$demo_dir/../.." rev-parse HEAD)" \ --arg image_id "$(docker image inspect "$image_ref" --format '{{.Id}}' 2>/dev/null || true)" \ --arg tested_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" --arg result "$result" --arg phase "$phase" \ - '{verification_run_id:$run_id,commit:$commit,image_id:$image_id,tested_at:$tested_at,mode:"migration",result:$result,phase:$phase,concurrent_migration:($result == "passed"),content_redacted:true}' > "$tmp" && chmod 600 "$tmp" && mv "$tmp" "$summary_file" + --argjson actual_overlap "$actual_overlap" --argjson lock_wait_observed "$lock_wait_observed" \ + --argjson exclusive_lock "$exclusive_lock" --argjson max_lock_holders "$max_lock_holders" \ + --argjson migration_execution_count "$migration_execution_count" \ + '{verification_run_id:$run_id,commit:$commit,image_id:$image_id,tested_at:$tested_at,mode:"migration",result:$result,phase:$phase,concurrent_migration:($result == "passed"),actual_overlap:$actual_overlap,lock_wait_observed:$lock_wait_observed,exclusive_lock:$exclusive_lock,max_lock_holders:$max_lock_holders,migration_execution_count:$migration_execution_count,proxy_replicas_started:false,content_redacted:true}' > "$tmp" && chmod 600 "$tmp" && mv "$tmp" "$summary_file" [[ -z "$tmpdir" ]] || rm -rf "$tmpdir" return "$rc" } @@ -33,13 +45,45 @@ verification_prepare_environment "$env_file" "${demo_dir}/docker-compose.litellm image_ref="$(verification_env LITELLM_IMAGE)" compose=(docker compose --env-file "$env_file" -f "${demo_dir}/docker-compose.litellm.yml") "${compose[@]}" up -d --wait postgres redis -phase="running_concurrent_jobs" -first="$("${compose[@]}" --profile migrate run -d --no-deps litellm-migrate)" -second="$("${compose[@]}" --profile migrate run -d --no-deps litellm-migrate)" +phase="starting_concurrent_jobs" + +# The test hold makes concurrent overlap observable without changing normal +# migration behavior. Both containers are real migration jobs; only the lock +# holder may enter LiteLLM migration execution. +first="$("${compose[@]}" --profile migrate run -d --no-deps -e LITELLM_MIGRATION_LOCK_HOLD_SECONDS=4 litellm-migrate)" +sleep 1 +second="$("${compose[@]}" --profile migrate run -d --no-deps -e LITELLM_MIGRATION_LOCK_HOLD_SECONDS=4 litellm-migrate)" [[ -n "$first" && -n "$second" && "$first" != "$second" ]] + +phase="observing_lock" +for _ in $(seq 1 20); do + state="$(docker inspect -f '{{.State.Running}} {{.State.Running}}' "$first" "$second" 2>/dev/null | tr '\n' ' ')" + if [[ "$state" == *"true true"* ]]; then actual_overlap=true; fi + "${compose[@]}" exec -T postgres sh -lc 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -At -v ON_ERROR_STOP=1 -c "SELECT count(*) FROM pg_locks WHERE locktype = '\''advisory'\'' AND granted;"' > "$tmpdir/lock-holders" + holders="$(tr -d '[:space:]' < "$tmpdir/lock-holders")" + [[ "$holders" =~ ^[0-9]+$ ]] + (( holders > max_lock_holders )) && max_lock_holders="$holders" + (( holders <= 1 )) || { echo "more than one migration advisory lock holder" >&2; exit 1; } + combined_logs="$(docker logs "$first" 2>&1; docker logs "$second" 2>&1)" + if [[ "$combined_logs" == *P1_MIGRATION_LOCK_WAITING* && "$(printf '%s' "$combined_logs" | rg -c 'P1_MIGRATION_LOCK_ACQUIRED')" == "1" ]]; then + lock_wait_observed=true + fi + [[ "$actual_overlap" == true && "$lock_wait_observed" == true ]] && break + sleep 1 +done +[[ "$actual_overlap" == true ]] +[[ "$lock_wait_observed" == true ]] +[[ "$max_lock_holders" == 1 ]] +exclusive_lock=true + +phase="waiting_for_serialized_jobs" docker wait "$first" "$second" > "$tmpdir/exit-codes" [[ "$(tr -d '[:space:]' < "$tmpdir/exit-codes")" == "00" ]] +combined_logs="$(docker logs "$first" 2>&1; docker logs "$second" 2>&1)" +[[ "$(printf '%s' "$combined_logs" | rg -c 'P1_MIGRATION_EXECUTION_START')" == "2" ]] +[[ "$(printf '%s' "$combined_logs" | rg -c 'P1_MIGRATION_EXECUTION_DONE')" == "2" ]] +migration_execution_count=2 ! "${compose[@]}" ps --services --status running | rg -q '^litellm-[12]$' phase="completed" result="passed" -echo "PASS concurrent migration jobs completed before proxy replicas started." +echo "PASS concurrent migration: overlapping jobs observed; PostgreSQL advisory lock held by at most one job." diff --git a/docker_litellm/demo/scripts/verify-p1.sh b/docker_litellm/demo/scripts/verify-p1.sh index 40496cf..8afb5a7 100755 --- a/docker_litellm/demo/scripts/verify-p1.sh +++ b/docker_litellm/demo/scripts/verify-p1.sh @@ -24,6 +24,7 @@ done "${script_dir}/verify-migration-concurrency.sh" "${script_dir}/run-migration.sh" "${compose[@]}" --profile single up -d --wait postgres redis litellm-1 +"${script_dir}/test-verification-gates.sh" --with-running-stack "${script_dir}/smoke-baseline.sh" --mode single "${compose[@]}" --profile ha up -d --wait postgres redis litellm-1 litellm-2 "${script_dir}/smoke-baseline.sh" --mode ha diff --git a/docker_litellm/work/run-migration-locked.py b/docker_litellm/work/run-migration-locked.py new file mode 100755 index 0000000..1dc643c --- /dev/null +++ b/docker_litellm/work/run-migration-locked.py @@ -0,0 +1,54 @@ +#!/usr/bin/env python3 +"""Run LiteLLM's migration mode under one PostgreSQL advisory lock. + +The lock connection is intentionally held while the child migration process +runs. Concurrent jobs therefore overlap as containers but cannot execute a +migration concurrently. No connection string or secret is printed. +""" + +import asyncio +import os +import subprocess +import sys + +from prisma import Prisma + +LOCK_ID = 548_019_700_001 + + +async def main() -> int: + db = Prisma() + await db.connect() + try: + print("P1_MIGRATION_LOCK_WAITING", flush=True) + # Prisma cannot deserialize PostgreSQL's `void` return from + # pg_advisory_lock(). Poll the boolean try-lock instead; this keeps + # the same session-scoped singleton guarantee and records real wait. + while True: + lock_result = await db.query_raw( + f"SELECT pg_try_advisory_lock({LOCK_ID}) AS acquired" + ) + if lock_result[0]["acquired"]: + break + await asyncio.sleep(0.1) + print("P1_MIGRATION_LOCK_ACQUIRED", flush=True) + hold_seconds = int(os.environ.get("LITELLM_MIGRATION_LOCK_HOLD_SECONDS", "0")) + if hold_seconds > 0: + print("P1_MIGRATION_LOCK_TEST_HOLD", flush=True) + await asyncio.sleep(hold_seconds) + print("P1_MIGRATION_EXECUTION_START", flush=True) + completed = subprocess.run( + ["/bin/bash", "/opt/utils/start-litellm.sh", *sys.argv[1:]], check=False + ) + print("P1_MIGRATION_EXECUTION_DONE", flush=True) + return completed.returncode + finally: + try: + await db.query_raw(f"SELECT pg_advisory_unlock({LOCK_ID})") + print("P1_MIGRATION_LOCK_RELEASED", flush=True) + finally: + await db.disconnect() + + +if __name__ == "__main__": + raise SystemExit(asyncio.run(main())) From 0959fae246f681df2c6bbb3549a49ea1a27b91a0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 19:59:19 +0800 Subject: [PATCH 54/87] =?UTF-8?q?fix:=20=E9=9A=94=E7=A6=BB=20cleanup=20?= =?UTF-8?q?=E8=B4=9F=E5=90=91=E6=B5=8B=E8=AF=95=E9=80=80=E5=87=BA?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_litellm/demo/scripts/smoke-baseline.sh | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh index 5d79349..1c81e28 100755 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -323,15 +323,13 @@ cleanup() { result="failed" fi write_summary || exit_code=1 - # Bash may re-enter EXIT processing when `exit` is called from an EXIT - # handler, which previously truncated a just-written failed report. A failed - # cleanup terminates the shell with the default TERM action after disabling - # the handler; the persisted JSON remains the authoritative failure record. + # A failed cleanup must return non-zero after the atomic report is durable. + # Disable the EXIT handler first: sending TERM here also terminates callers + # of the real cleanup-negative gate in some Bash execution modes. if (( exit_code != 0 )); then - # Ensure the atomic rename is durably visible before the deliberate signal. sync "$SUMMARY_FILE" 2>/dev/null || sync trap - EXIT - kill -TERM "$$" + exit "$exit_code" fi return "$exit_code" } From 940325578bae9905673965d6dc489130ab4b6a46 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Sun, 9 Aug 2026 21:35:06 +0800 Subject: [PATCH 55/87] =?UTF-8?q?test:=20=E8=A6=86=E7=9B=96=20LiteLLM=20?= =?UTF-8?q?=E9=99=90=E6=B5=81=E6=9D=A5=E6=BA=90=E8=B4=9F=E5=90=91=E5=88=A4?= =?UTF-8?q?=E5=AE=9A?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../demo/scripts/test-verification-gates.sh | 67 +++++++++++++++++++ 1 file changed, 67 insertions(+) diff --git a/docker_litellm/demo/scripts/test-verification-gates.sh b/docker_litellm/demo/scripts/test-verification-gates.sh index 52ddcbc..ed3e3eb 100755 --- a/docker_litellm/demo/scripts/test-verification-gates.sh +++ b/docker_litellm/demo/scripts/test-verification-gates.sh @@ -57,6 +57,73 @@ rg -q -- 'curl_args=\(--silent --show-error --fail' "$script_dir/smoke-baseline. ! rg -n -- 'source "\$env_file"|source "\$\{env_file\}"' "$script_dir/run-migration.sh" "$script_dir/smoke-redis-recovery.sh" rg -q 'config --environment > "\$verification_environment_file"' "$script_dir/verification-lib.sh" +# Exercise the exact limiter-source predicate used by smoke-baseline.sh. The +# fixture replaces only `docker compose ... logs`; no service, .env, or +# upstream call is involved. This keeps a provider 429 from being mistaken for +# the proxy's Redis-backed RPM/TPM limiter. +load_smoke_limiter_predicate() { + # Keep this extraction intentionally narrow: the production function remains + # the single source of truth, while the fixture provides its log dependency. + local predicate_file="$tmpdir/smoke-limiter-predicate.sh" + sed -n '/^assert_proxy_limiter_response() {/,/^}$/p' "$script_dir/smoke-baseline.sh" > "$predicate_file" + chmod 600 "$predicate_file" + source "$predicate_file" +} + +fixture_docker() { + # assert_proxy_limiter_response only asks Docker for a bounded Compose log + # stream. Do not forward any fixture argument to a real Docker process. + cat "$limiter_fixture_log" +} + +assert_limiter_fixture() { + local fixture_name="$1" response_file="$2" headers_file="$3" limit_kind="$4" expected_result="$5" + local limiter_fixture_log="$tmpdir/${fixture_name}.logs" + local ENV_FILE="$tmpdir/fixture.env" DEMO_DIR="$demo_dir" + printf '%s\n' 'POST /v1/chat/completions HTTP/1.1" 429' > "$limiter_fixture_log" + docker() { fixture_docker "$@"; } + load_smoke_limiter_predicate + if assert_proxy_limiter_response "$response_file" "$headers_file" 429 "$limit_kind" '2026-01-01T00:00:00Z'; then + [[ "$expected_result" == pass ]] || { echo "accepted $fixture_name fixture" >&2; exit 1; } + else + [[ "$expected_result" == reject ]] || { echo "rejected valid $fixture_name fixture" >&2; exit 1; } + fi + unset -f docker +} + +# A fake provider/upstream response has a conventional 429 body and headers, +# even with an otherwise matching local access-log line. It lacks LiteLLM's +# proxy-only rate_limit_type and Limit type evidence and must be rejected. +upstream_body="$tmpdir/upstream-429.json" +upstream_headers="$tmpdir/upstream-429.headers" +printf '%s\n' '{"error":{"message":"upstream provider rate limit exceeded","type":"rate_limit_error"}}' > "$upstream_body" +printf '%s\n' 'HTTP/1.1 429 Too Many Requests' 'retry-after: 1' 'x-ratelimit-remaining-requests: 0' > "$upstream_headers" +assert_limiter_fixture upstream-429 "$upstream_body" "$upstream_headers" rpm reject + +# LiteLLM's proxy limiter fixture has the stable matching header and detail; +# this proves the fixture harness accepts the same positive RPM evidence that +# the real HA smoke requires. +proxy_body="$tmpdir/litellm-proxy-429.json" +proxy_headers="$tmpdir/litellm-proxy-429.headers" +printf '%s\n' '{"detail":"Rate limit exceeded. Limit type: requests"}' > "$proxy_body" +printf '%s\n' 'HTTP/1.1 429 Too Many Requests' 'rate_limit_type: requests' > "$proxy_headers" +assert_limiter_fixture litellm-proxy-429 "$proxy_body" "$proxy_headers" rpm pass + +# RPM and TPM evidence are type-specific. A 429 with a valid-looking proxy +# shape but the wrong type must not satisfy either opposite limiter assertion. +rpm_mismatch_body="$tmpdir/rpm-type-mismatch.json" +rpm_mismatch_headers="$tmpdir/rpm-type-mismatch.headers" +printf '%s\n' '{"detail":"Rate limit exceeded. Limit type: tokens"}' > "$rpm_mismatch_body" +printf '%s\n' 'HTTP/1.1 429 Too Many Requests' 'rate_limit_type: tokens' > "$rpm_mismatch_headers" +assert_limiter_fixture rpm-type-mismatch "$rpm_mismatch_body" "$rpm_mismatch_headers" rpm reject + +tpm_mismatch_body="$tmpdir/tpm-type-mismatch.json" +tpm_mismatch_headers="$tmpdir/tpm-type-mismatch.headers" +printf '%s\n' '{"detail":"Rate limit exceeded. Limit type: requests"}' > "$tpm_mismatch_body" +printf '%s\n' 'HTTP/1.1 429 Too Many Requests' 'rate_limit_type: requests' > "$tpm_mismatch_headers" +assert_limiter_fixture tpm-type-mismatch "$tpm_mismatch_body" "$tpm_mismatch_headers" tpm reject +echo "PASS limiter-source fixtures: upstream 429 and RPM/TPM type mismatches rejected; LiteLLM proxy 429 accepted." + # Compose's dotenv parser must not evaluate shell substitutions. This isolated # Compose file uses no project secrets and verifies the same config command # that the runtime scripts use. From 6fdb6376fedbd3cd14c20a1bed4366eb95be2ae1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Mon, 10 Aug 2026 15:09:42 +0800 Subject: [PATCH 56/87] =?UTF-8?q?feat:=20=E6=96=B0=E5=A2=9E=20P6=20?= =?UTF-8?q?=E6=9C=AC=E5=9C=B0=E9=BB=84=E9=87=91=E9=93=BE=E8=B7=AF=E7=BC=96?= =?UTF-8?q?=E6=8E=92=E9=97=A8=E7=A6=81?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .gitignore | 3 + docker_openclaw/README.md | 15 ++ docker_openclaw/p6/README.md | 61 +++++++ docker_openclaw/p6/docker-compose.p6.yml | 37 +++++ docker_openclaw/p6/p6-inputs.example.json | 24 +++ docker_openclaw/p6/scripts/p6-aggregate.sh | 31 ++++ docker_openclaw/p6/scripts/p6-lib.sh | 157 ++++++++++++++++++ docker_openclaw/p6/scripts/p6-runner.sh | 155 +++++++++++++++++ .../p6/scripts/test-p6-compose-render.sh | 23 +++ docker_openclaw/p6/scripts/test-p6-gates.sh | 44 +++++ 10 files changed, 550 insertions(+) create mode 100644 docker_openclaw/p6/README.md create mode 100644 docker_openclaw/p6/docker-compose.p6.yml create mode 100644 docker_openclaw/p6/p6-inputs.example.json create mode 100755 docker_openclaw/p6/scripts/p6-aggregate.sh create mode 100755 docker_openclaw/p6/scripts/p6-lib.sh create mode 100755 docker_openclaw/p6/scripts/p6-runner.sh create mode 100755 docker_openclaw/p6/scripts/test-p6-compose-render.sh create mode 100755 docker_openclaw/p6/scripts/test-p6-gates.sh diff --git a/.gitignore b/.gitignore index 92b961a..a3604ff 100644 --- a/.gitignore +++ b/.gitignore @@ -123,6 +123,9 @@ celerybeat.pid # Environments .env docker_litellm/demo/artifacts/ +docker_openclaw/p6/artifacts/ +docker_openclaw/p6/p6-inputs.json +docker_openclaw/p6/.p6-work/ .venv env/ venv/ diff --git a/docker_openclaw/README.md b/docker_openclaw/README.md index ff7bc5e..719d69b 100644 --- a/docker_openclaw/README.md +++ b/docker_openclaw/README.md @@ -38,3 +38,18 @@ docker run -d \ -v openclaw_data:/root/.openclaw/data \ labnow/openclaw:latest ``` + +## P6 本地跨仓产品闭环 + +P6 的固定组合编排、黄金 runner、报告聚合和安全清理由 +[`p6/README.md`](p6/README.md) 维护。该入口只接受本地受限输入文件中 +固定的四仓 commit、RC1 bundle hash,以及下列冻结镜像事实:LiteLLM +`quay.io/labnow/litellm:1.97.0-ead62528e607` 的本地 ID/digest +`sha256:a2e115874c21b829bd052b18fc85be2f9217fb8244b82812c4ebc6e36f9824d1`; +上游 OpenClaw `quay.io/labnow/openclaw@sha256:edc85cc2068f5ec0df470f7d06daa0a4fbd78ef5ad6cf5b48f58381da839dd12`; +以及 P6 Workspace 本地镜像 +`quay.io/labnow/labnow-open:che-563-openclaw-product-closure-local` 的 ID +`sha256:79fbe459040bc10cb8a64934fa608d7cd23ac4b0472e7c7690db294ad54ffbb7`、 +source commit `5b70a6b0f960ddc1a5c45a27449cd7317da0c7da` 和上游 OpenClaw +base digest。三者均为 `amd64`,P6 仅以 local-only 方式运行,不拉取、 +推送、发布或部署镜像,也不会把凭据写入仓库、报告或命令参数。 diff --git a/docker_openclaw/p6/README.md b/docker_openclaw/p6/README.md new file mode 100644 index 0000000..594b7b3 --- /dev/null +++ b/docker_openclaw/p6/README.md @@ -0,0 +1,61 @@ +# P6 OpenClaw 产品闭环(本仓编排) + +本目录仅承担 LLM Hub V1 / P6 的 `lab-dev` 职责:固定本地组合输入、 +OpenClaw Compose、黄金 runner、脱敏报告聚合和清理入口。Shell、Launcher +和 OpenClaw 产品业务逻辑仍由各自仓库拥有。 + +当前状态:本仓 runner 与固定输入门禁已具备,尚未达到 ready。真实黄金运行 +必须等待四仓 tracked 工作树干净,并提供能自动编排五个组件的固定 driver; +在此之前,`--preflight`/`--golden` 必须失败关闭,handoff 为 `blocked`。 + +## 输入与安全边界 + +从 [`p6-inputs.example.json`](p6-inputs.example.json) 创建本地 +`p6-inputs.json`,权限必须为 `0400` 或 `0600`。该文件是 Git 忽略的, +只保存路径、固定 commit、镜像 ID/digest 与本地 driver 路径。测试 Secret +由 driver 在运行中以受限文件生成;它只将模式为 `0600` 的 pattern 文件 +交给 runner 做零命中扫描,绝不写入输入、报告或命令参数。 + +Runner 拒绝以下情况:浮动 `latest`、非 `quay.io/labnow/` 镜像、commit/ +RC1 bundle 不匹配、任何冻结仓 tracked 工作树有变更、本机镜像 ID/digest +不匹配、输入/Secret 权限不安全或运行时路径缺失。它从不 source `.env`。 +本地 Workspace 镜像可以明确标为 `local_build` 且 `repo_digest="absent"`, +此时必须提供准确 image ID、`labnow-open` source commit 与固定上游 +OpenClaw base digest;runner 会校验三者,不会伪造 repository digest。 + +## 入口 + +```bash +./docker_openclaw/p6/scripts/test-p6-gates.sh +./docker_openclaw/p6/scripts/test-p6-compose-render.sh +./docker_openclaw/p6/scripts/p6-runner.sh --input /secure/path/p6-inputs.json --preflight +./docker_openclaw/p6/scripts/p6-runner.sh --input /secure/path/p6-inputs.json --render +./docker_openclaw/p6/scripts/p6-runner.sh --input /secure/path/p6-inputs.json --golden +./docker_openclaw/p6/scripts/p6-runner.sh --input /secure/path/p6-inputs.json --cleanup +``` + +`--golden` 在严格 preflight 后调用输入中已固定的跨仓 driver,强制其按 +`provision → golden → cleanup` 执行:`provision` 必须在同一 run 中启动固定 +LiteLLM、Shell、live JupyterHub、Launcher 与 P6 Workspace;Workspace 必须 +使用本目录 Compose;`golden` 覆盖 Console、JupyterHub/DockerSpawner、 +Launcher claim/activate/release、Adapter、chat/stream/tool、用量、撤销、 +generation、delete 与零 active lease;`cleanup` 必须证明所有上述资源、 +运行材料、临时文件和进程均不存在。缺少任何阶段或检查都会失败关闭,不能 +把外部服务预先手工启动后当作 P6 成功。 + +本轮固定组合为 LiteLLM 本地 ID/digest、上游 OpenClaw base digest,以及 +`quay.io/labnow/labnow-open:che-563-openclaw-product-closure-local` Workspace +本地镜像;Workspace 的 local-only provenance 绑定 +`labnow-open@5b70a6b0f960ddc1a5c45a27449cd7317da0c7da` 与上游 OpenClaw +base digest。P6 Compose 仅定义该 Workspace,完整 driver 负责启动同一 run +中的 LiteLLM、Shell、JupyterHub 和 Launcher,并在 cleanup 中全部移除。 + +运行产生的脱敏报告位于 `p6/artifacts/`(Git 忽略)。只有同一 run 的 +preflight、golden、cleanup 都 `passed`,才允许: + +```bash +./docker_openclaw/p6/scripts/p6-aggregate.sh --artifacts docker_openclaw/p6/artifacts --run-id p6- +``` + +清理只移除本 runner 创建的 Compose 资源与受限临时目录,不删除数据卷、 +用户目录或其他项目资源。P6 不推送镜像、不部署、不推进 integration。 diff --git a/docker_openclaw/p6/docker-compose.p6.yml b/docker_openclaw/p6/docker-compose.p6.yml new file mode 100644 index 0000000..8c18246 --- /dev/null +++ b/docker_openclaw/p6/docker-compose.p6.yml @@ -0,0 +1,37 @@ +name: p6-openclaw-closure + +# The P6 runner renders this file only after p6-inputs.json has bound every +# image to a local ID and repository digest. There are intentionally no image +# defaults, no `latest`, and no credential values in this Compose file. +services: + openclaw-workspace: + image: ${P6_OPENCLAW_WORKSPACE_IMAGE:?p6-runner must supply the fixed labnow-open Workspace image} + pull_policy: never + restart: "no" + environment: + OPENCLAW_GATEWAY_BIND: lan + OPENCLAW_GATEWAY_PORT: "18789" + # P6 keeps the gateway on the internal network. The real Console/Shell + # flow authenticates at its owning boundary; no gateway token is added + # to a persistent OpenClaw configuration. + OPENCLAW_USE_TRUSTED_PROXY_AUTH: "true" + LABNOW_MODEL_ACCESS_STATE_DIR: /root/.openclaw/data/labnow-model-access + volumes: + - type: bind + source: ${P6_RUNTIME_MOUNT:?p6-runner must supply the Launcher-managed runtime mount} + target: /run/labnow/model-access + read_only: true + - type: bind + source: ${P6_OPENCLAW_STATE_DIR:?p6-runner must supply an isolated run state directory} + target: /root/.openclaw/data + networks: [p6-net] + healthcheck: + test: ["CMD-SHELL", "openclaw config validate >/dev/null 2>&1"] + interval: 5s + timeout: 3s + retries: 12 + start_period: 10s + +networks: + p6-net: + name: ${P6_NETWORK_NAME:?p6-runner must supply an isolated network name} diff --git a/docker_openclaw/p6/p6-inputs.example.json b/docker_openclaw/p6/p6-inputs.example.json new file mode 100644 index 0000000..7358d31 --- /dev/null +++ b/docker_openclaw/p6/p6-inputs.example.json @@ -0,0 +1,24 @@ +{ + "schema_version": "p6-inputs/v1", + "contract_version": "v1alpha1", + "contract_bundle_sha256": "d289dff9bcaa3d28035c5ed2e56b806f4b3b37fdca3159352d22f0c03942e202", + "control_commit": "2eb71d7590739df3de8db2f8cf9098154a397f0b", + "review_policy_commit": "2eb71d7590739df3de8db2f8cf9098154a397f0b", + "repositories": { + "lab_dev": {"path": "/absolute/path/to/lab-dev", "commit": "940325578bae9905673965d6dc489130ab4b6a46"}, + "labnow_open": {"path": "/absolute/path/to/labnow-open", "commit": "dfac9767fd6cdd4706ac4cd6917defcafd1c6eb8"}, + "labnow_shell": {"path": "/absolute/path/to/labnow-shell", "commit": "eb0e6f90182e5d59174ea9edb7cb71edeaa7a47f"}, + "labnow_launcher": {"path": "/absolute/path/to/labnow-launcher", "commit": "c64f5fbabc587e26394a486ef9ae12558234f646"} + }, + "images": { + "litellm": {"ref": "quay.io/labnow/litellm:REPLACE_WITH_FIXED_TAG", "image_id": "sha256:REPLACE_WITH_LOCAL_IMAGE_ID", "provenance": "repo_digest", "repo_digest": "quay.io/labnow/litellm@sha256:REPLACE_WITH_REPO_DIGEST"}, + "openclaw_base": {"ref": "quay.io/labnow/openclaw@sha256:REPLACE_WITH_VERIFIED_BASE_DIGEST", "image_id": "sha256:REPLACE_WITH_LOCAL_IMAGE_ID", "provenance": "repo_digest", "repo_digest": "quay.io/labnow/openclaw@sha256:REPLACE_WITH_VERIFIED_BASE_DIGEST"}, + "openclaw_workspace": {"ref": "quay.io/labnow/labnow-open:che-563-openclaw-product-closure-local", "image_id": "sha256:REPLACE_WITH_LOCAL_IMAGE_ID", "provenance": "local_build", "repo_digest": "absent", "source_repository": "labnow_open", "source_commit": "REPLACE_WITH_LABNOW_OPEN_PHASE_COMMIT", "base_image_digest": "quay.io/labnow/openclaw@sha256:REPLACE_WITH_VERIFIED_BASE_DIGEST"} + }, + "paths": { + "adapter": "/absolute/path/to/labnow-open/src/labnow-open-etc/openclaw-model-access-adapter.sh", + "runtime_mount": "/absolute/path/to/launcher-managed-runtime-mount", + "workspace_root": "/absolute/path/to/p6-run-workspace" + }, + "driver": "/absolute/path/to/local-only-p6-golden-driver" +} diff --git a/docker_openclaw/p6/scripts/p6-aggregate.sh b/docker_openclaw/p6/scripts/p6-aggregate.sh new file mode 100755 index 0000000..2304c80 --- /dev/null +++ b/docker_openclaw/p6/scripts/p6-aggregate.sh @@ -0,0 +1,31 @@ +#!/usr/bin/env bash +# Refuse incomplete P6 evidence. This creates a final report only after a +# complete preflight, golden chain, and explicit cleanup report for one run. +set -euo pipefail + +usage() { echo "Usage: $0 --artifacts DIR --run-id p6-<32hex>" >&2; } +artifacts=""; run_id="" +while (($#)); do + case "$1" in + --artifacts) artifacts="${2:-}"; shift 2 ;; + --run-id) run_id="${2:-}"; shift 2 ;; + *) usage; exit 2 ;; + esac +done +[[ "$run_id" =~ ^p6-[a-f0-9]{32}$ && -d "$artifacts" ]] || { usage; exit 2; } +reports=("$artifacts/p6-preflight-${run_id}.json" "$artifacts/p6-golden-${run_id}.json" "$artifacts/p6-cleanup-${run_id}.json") +for report in "${reports[@]}"; do + [[ -f "$report" ]] || { echo "P6_ERROR:EVIDENCE_INCOMPLETE" >&2; exit 1; } + jq -e --arg run "$run_id" '.schema_version == "p6-report/v1" and .run_id == $run and .result == "passed" and .phase == "completed" and .content_redacted == true and .contract_version == "v1alpha1" and .contract_bundle_sha256 == "d289dff9bcaa3d28035c5ed2e56b806f4b3b37fdca3159352d22f0c03942e202" and ([.images.litellm,.images.openclaw_base] | all(.[]; .provenance == "repo_digest" and (.repo_digest | type == "string" and test("^quay\\.io/labnow/(litellm|openclaw)@sha256:[0-9a-f]{64}$")))) and (.images.openclaw_workspace.provenance == "local_build") and (.images.openclaw_workspace.repo_digest == "absent") and (.images.openclaw_workspace.source_repository == "labnow_open") and (.images.openclaw_workspace.source_commit | type == "string" and test("^[0-9a-f]{40}$")) and (.images.openclaw_workspace.base_image_digest == .images.openclaw_base.repo_digest)' "$report" >/dev/null || { echo "P6_ERROR:EVIDENCE_REJECTED" >&2; exit 1; } +done +input_hash="$(jq -r '.input_sha256' "${reports[0]}")" +for report in "${reports[@]}"; do [[ "$(jq -r '.input_sha256' "$report")" == "$input_hash" ]] || { echo "P6_ERROR:INPUT_HASH_MISMATCH" >&2; exit 1; }; done +metadata="$(jq -c '{contract_version,contract_bundle_sha256,control_commit,review_policy_commit,repositories,images}' "${reports[0]}")" +for report in "${reports[@]}"; do [[ "$(jq -c '{contract_version,contract_bundle_sha256,control_commit,review_policy_commit,repositories,images}' "$report")" == "$metadata" ]] || { echo "P6_ERROR:METADATA_MISMATCH" >&2; exit 1; }; done +output="$artifacts/p6-final-${run_id}.json" +tmp="$(mktemp "$artifacts/.p6-final.XXXXXX")" +jq -n --arg run_id "$run_id" --arg input_sha256 "$input_hash" --arg tested_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" --argjson metadata "$metadata" \ + '{schema_version:"p6-final-report/v1",run_id:$run_id,input_sha256:$input_sha256,tested_at:$tested_at,result:"passed",phase:"completed",content_redacted:true} + $metadata' > "$tmp" +chmod 600 "$tmp" +mv -f "$tmp" "$output" +printf '%s\n' "$output" diff --git a/docker_openclaw/p6/scripts/p6-lib.sh b/docker_openclaw/p6/scripts/p6-lib.sh new file mode 100755 index 0000000..f8e2a37 --- /dev/null +++ b/docker_openclaw/p6/scripts/p6-lib.sh @@ -0,0 +1,157 @@ +#!/usr/bin/env bash +# Shared fail-closed helpers for the P6 local-only runner. Never source an +# environment file and never print a credential or a credential fingerprint. +set -euo pipefail + +p6_die() { + printf 'P6_ERROR:%s\n' "$1" >&2 + return "${2:-1}" +} + +p6_run_id() { + python3 -c 'import secrets; print("p6-" + secrets.token_hex(16))' +} + +p6_require_regular_0600() { + local path="$1" mode + [[ -f "$path" && ! -L "$path" ]] || { p6_die "SECURE_FILE_REQUIRED" 64; return $?; } + if mode="$(stat -f '%Lp' "$path" 2>/dev/null)"; then :; else + mode="$(stat -c '%a' "$path")" + fi + [[ "$mode" == 400 || "$mode" == 600 ]] || { p6_die "SECURE_FILE_MODE_REQUIRED" 65; return $?; } +} + +p6_write_report() { + local report="$1" result="$2" phase="$3" reason="${4:-}" + local temp metadata + mkdir -p "$(dirname "$report")" + chmod 700 "$(dirname "$report")" + temp="$(mktemp "$(dirname "$report")/.p6-report.XXXXXX")" + metadata="$(jq -c '{contract_version,contract_bundle_sha256,control_commit,review_policy_commit,repositories:(.repositories | with_entries(.value = .value.commit)),images}' "$P6_INPUT_FILE")" + jq -n \ + --arg run_id "$P6_RUN_ID" \ + --arg tested_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ + --arg result "$result" \ + --arg phase "$phase" \ + --arg reason "$reason" \ + --arg input_sha256 "$(shasum -a 256 "$P6_INPUT_FILE" | awk '{print $1}')" \ + --argjson metadata "$metadata" \ + '{schema_version:"p6-report/v1",run_id:$run_id,tested_at:$tested_at,result:$result,phase:$phase,input_sha256:$input_sha256,content_redacted:true} + + $metadata + + (if $reason == "" then {} else {reason:$reason} end)' > "$temp" + chmod 600 "$temp" + mv -f "$temp" "$report" +} + +p6_json_string() { + jq -er "$1" "$P6_INPUT_FILE" +} + +p6_assert_fixed_commit() { + local value="$1" + [[ "$value" =~ ^[0-9a-f]{40}$ ]] || { p6_die "FIXED_COMMIT_REQUIRED" 66; return $?; } +} + +p6_assert_fixed_image() { + local name="$1" ref image_id digest provenance source_commit source_repository base_image_digest + ref="$(p6_json_string ".images.${name}.ref")" + image_id="$(p6_json_string ".images.${name}.image_id")" + digest="$(jq -r ".images.${name}.repo_digest // empty" "$P6_INPUT_FILE")" + provenance="$(p6_json_string ".images.${name}.provenance")" + [[ "$ref" == quay.io/labnow/* && "$ref" != *:latest ]] || { p6_die "FIXED_IMAGE_REF_REQUIRED" 67; return $?; } + [[ "$image_id" =~ ^sha256:[0-9a-f]{64}$ ]] || { p6_die "FIXED_IMAGE_ID_REQUIRED" 67; return $?; } + case "$provenance" in + repo_digest) + [[ "$digest" =~ ^quay\.io/labnow/(litellm|openclaw)@sha256:[0-9a-f]{64}$ ]] || { p6_die "FIXED_IMAGE_DIGEST_REQUIRED" 67; return $?; } + [[ "$ref" == "$digest" || "$ref" != *'@sha256:'* ]] || { p6_die "IMAGE_REF_DIGEST_MISMATCH" 67; return $?; } + ;; + local_build) + [[ "$digest" == absent ]] || { p6_die "LOCAL_PROVENANCE_MUST_DECLARE_ABSENT_DIGEST" 67; return $?; } + source_repository="$(p6_json_string ".images.${name}.source_repository")" + source_commit="$(p6_json_string ".images.${name}.source_commit")" + base_image_digest="$(p6_json_string ".images.${name}.base_image_digest")" + [[ "$source_repository" == labnow_open ]] || { p6_die "LOCAL_PROVENANCE_SOURCE_REPOSITORY_INVALID" 67; return $?; } + p6_assert_fixed_commit "$source_commit" || return $? + [[ "$base_image_digest" =~ ^quay\.io/labnow/openclaw@sha256:[0-9a-f]{64}$ ]] || { p6_die "LOCAL_BASE_IMAGE_DIGEST_REQUIRED" 67; return $?; } + ;; + *) p6_die "IMAGE_PROVENANCE_REQUIRED" 67; return $? ;; + esac +} + +p6_validate_input_shape() { + jq -e ' + type == "object" + and (.schema_version == "p6-inputs/v1") + and (.contract_version == "v1alpha1") + and (.contract_bundle_sha256 == "d289dff9bcaa3d28035c5ed2e56b806f4b3b37fdca3159352d22f0c03942e202") + and (.control_commit == "2eb71d7590739df3de8db2f8cf9098154a397f0b") + and (.review_policy_commit == .control_commit) + and (.repositories | keys | sort) == ["lab_dev","labnow_launcher","labnow_open","labnow_shell"] + and (.images | keys | sort) == ["litellm","openclaw_base","openclaw_workspace"] + and (.paths | keys | sort) == ["runtime_mount","workspace_root"] + and (.driver | type == "string" and startswith("/")) + ' "$P6_INPUT_FILE" >/dev/null || { p6_die "INPUT_SCHEMA_INVALID" 68; return $?; } + local repo + for repo in lab_dev labnow_open labnow_shell labnow_launcher; do + p6_assert_fixed_commit "$(p6_json_string ".repositories.${repo}.commit")" || return $? + done + p6_assert_fixed_image litellm || return $? + p6_assert_fixed_image openclaw_base || return $? + p6_assert_fixed_image openclaw_workspace || return $? +} + +p6_assert_repository() { + local name="$1" path expected actual status + path="$(p6_json_string ".repositories.${name}.path")" + expected="$(p6_json_string ".repositories.${name}.commit")" + [[ -d "$path/.git" ]] || { p6_die "REPOSITORY_UNAVAILABLE" 69; return $?; } + actual="$(git -C "$path" rev-parse HEAD)" + [[ "$actual" == "$expected" ]] || { p6_die "REPOSITORY_COMMIT_MISMATCH" 70; return $?; } + status="$(git -C "$path" status --porcelain=v1 --untracked-files=no)" + [[ -z "$status" ]] || { p6_die "REPOSITORY_TRACKED_TREE_DIRTY" 71; return $?; } +} + +p6_assert_images_present() { + local name ref image_id actual_id digests expected_digest provenance source_commit source_repository base_image_digest source_repo_path source_repo_commit + for name in litellm openclaw_base openclaw_workspace; do + ref="$(p6_json_string ".images.${name}.ref")" + image_id="$(p6_json_string ".images.${name}.image_id")" + expected_digest="$(jq -r ".images.${name}.repo_digest // empty" "$P6_INPUT_FILE")" + provenance="$(p6_json_string ".images.${name}.provenance")" + actual_id="$(docker image inspect --format '{{.Id}}' "$ref" 2>/dev/null)" || { p6_die "LOCAL_IMAGE_UNAVAILABLE" 72; return $?; } + [[ "$actual_id" == "$image_id" ]] || { p6_die "LOCAL_IMAGE_ID_MISMATCH" 72; return $?; } + if [[ "$provenance" == repo_digest ]]; then + digests="$(docker image inspect --format '{{join .RepoDigests "\n"}}' "$ref")" + grep -Fqx "$expected_digest" <<<"$digests" || { p6_die "LOCAL_IMAGE_DIGEST_MISMATCH" 72; return $?; } + else + source_commit="$(p6_json_string ".images.${name}.source_commit")" + source_repository="$(p6_json_string ".images.${name}.source_repository")" + base_image_digest="$(p6_json_string ".images.${name}.base_image_digest")" + source_repo_path="$(p6_json_string ".repositories.${source_repository}.path")" + source_repo_commit="$(p6_json_string ".repositories.${source_repository}.commit")" + [[ "$source_commit" == "$source_repo_commit" ]] || { p6_die "LOCAL_PROVENANCE_SOURCE_MISMATCH" 72; return $?; } + [[ -d "$source_repo_path/.git" ]] || { p6_die "LOCAL_PROVENANCE_SOURCE_UNAVAILABLE" 72; return $?; } + [[ "$(p6_json_string '.images.openclaw_base.repo_digest')" == "$base_image_digest" ]] || { p6_die "LOCAL_BASE_IMAGE_MISMATCH" 72; return $?; } + fi + done +} + +p6_assert_runtime_paths() { + local mount workspace + mount="$(p6_json_string '.paths.runtime_mount')" + workspace="$(p6_json_string '.paths.workspace_root')" + [[ -d "$mount" && ! -L "$mount" && -d "$workspace" && ! -L "$workspace" ]] || { p6_die "RUNTIME_PATH_UNAVAILABLE" 73; return $?; } +} + +p6_security_scan() { + # Write secret strings only to a mode-0600 pattern file and pass its path to + # rg. Neither the shell command line nor the report contains the secret. + local scan_root="$1" patterns="$2" + p6_require_regular_0600 "$patterns" || return $? + [[ -s "$patterns" ]] || { p6_die "SECRET_PATTERN_FILE_REQUIRED" 74; return $?; } + if rg --fixed-strings --files-with-matches --glob '!secret-patterns' --glob '!secret.json' -f "$patterns" "$scan_root" >/dev/null 2>&1; then + rm -f "$patterns" + p6_die "SECRET_FINGERPRINT_MATCH" 75 + return $? + fi +} diff --git a/docker_openclaw/p6/scripts/p6-runner.sh b/docker_openclaw/p6/scripts/p6-runner.sh new file mode 100755 index 0000000..84dad29 --- /dev/null +++ b/docker_openclaw/p6/scripts/p6-runner.sh @@ -0,0 +1,155 @@ +#!/usr/bin/env bash +# P6 local-only, fail-closed golden-chain coordinator. It deliberately does +# not create product resources until all frozen commits and local image +# digests have been verified. +set -euo pipefail + +script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +p6_dir="$(cd "${script_dir}/.." && pwd)" +source "${script_dir}/p6-lib.sh" + +usage() { + cat <<'USAGE' +Usage: p6-runner.sh --input /secure/path/p6-inputs.json [--validate-input|--preflight|--render|--golden|--cleanup] + +--validate-input performs no Docker or product operation. --preflight validates +fixed repositories, local image IDs/digests and secure inputs. --render writes +a non-sensitive Compose rendering. --golden starts only the isolated P6 +OpenClaw service, invokes the separately fixed cross-repository driver, scans +for secrets, and always cleans up. --cleanup removes only this run's Compose +resources and temporary runtime state; it never removes named data volumes. +USAGE +} + +action="" +P6_INPUT_FILE="" +while (($#)); do + case "$1" in + --input) P6_INPUT_FILE="${2:-}"; shift 2 ;; + --validate-input|--preflight|--render|--golden|--cleanup) + [[ -z "$action" ]] || { usage >&2; exit 2; } + action="${1#--}"; shift ;; + *) usage >&2; exit 2 ;; + esac +done +[[ -n "$P6_INPUT_FILE" && -n "$action" ]] || { usage >&2; exit 2; } +[[ -f "$P6_INPUT_FILE" && ! -L "$P6_INPUT_FILE" ]] || p6_die "INPUT_FILE_REQUIRED" 64 +P6_RUN_ID="${P6_RUN_ID:-$(p6_run_id)}" +[[ "$P6_RUN_ID" =~ ^p6-[a-f0-9]{32}$ ]] || p6_die "RUN_ID_INVALID" 64 +export P6_INPUT_FILE P6_RUN_ID +artifact_dir="${P6_ARTIFACTS_DIR:-${p6_dir}/artifacts}" +P6_WORK_DIR="${P6_WORK_DIR:-${p6_dir}/.p6-work/${P6_RUN_ID}}" +export P6_WORK_DIR +mkdir -p "$P6_WORK_DIR" "$artifact_dir" +chmod 700 "$P6_WORK_DIR" "$artifact_dir" +report="${artifact_dir}/p6-${action}-${P6_RUN_ID}.json" + +driver_cleanup_best_effort() { + local driver + driver="$(jq -r '.driver // empty' "$P6_INPUT_FILE" 2>/dev/null || true)" + if [[ -n "$driver" && -x "$driver" && ! -L "$driver" ]]; then + P6_DRIVER_ACTION=cleanup P6_RUN_ID="$P6_RUN_ID" P6_INPUT_FILE="$P6_INPUT_FILE" P6_DRIVER_REPORT="$P6_WORK_DIR/driver-cleanup-report.json" P6_SECRET_PATTERN_FILE="$P6_WORK_DIR/secret-patterns" "$driver" >/dev/null 2>&1 || true + fi +} + +cleanup() { + local project="p6-${P6_RUN_ID}" + driver_cleanup_best_effort + if [[ -f "$P6_WORK_DIR/runtime.env" ]]; then + docker compose --project-name "$project" --env-file "$P6_WORK_DIR/runtime.env" -f "$p6_dir/docker-compose.p6.yml" down --remove-orphans >/dev/null 2>&1 || true + fi + rm -rf "$P6_WORK_DIR" +} + +prepare() { + p6_validate_input_shape || return $? + p6_require_regular_0600 "$P6_INPUT_FILE" || return $? +} + +preflight() { + prepare || return $? + local repo + for repo in lab_dev labnow_open labnow_shell labnow_launcher; do p6_assert_repository "$repo" || return $?; done + p6_assert_images_present || return $? + p6_assert_runtime_paths || return $? +} + +render() { + preflight || return $? + local workspace_image mount workspace state network + workspace_image="$(p6_json_string '.images.openclaw_workspace.ref')" + mount="$(p6_json_string '.paths.runtime_mount')" + workspace="$(p6_json_string '.paths.workspace_root')" + state="$P6_WORK_DIR/openclaw-state" + network="p6-${P6_RUN_ID}" + mkdir -p "$state" + chmod 700 "$state" + { + printf 'P6_OPENCLAW_WORKSPACE_IMAGE=%s\n' "$workspace_image" + printf 'P6_RUNTIME_MOUNT=%s\n' "$mount" + printf 'P6_OPENCLAW_STATE_DIR=%s\n' "$state" + printf 'P6_NETWORK_NAME=%s\n' "$network" + } > "$P6_WORK_DIR/runtime.env" + chmod 600 "$P6_WORK_DIR/runtime.env" + docker compose --project-name "p6-${P6_RUN_ID}" --env-file "$P6_WORK_DIR/runtime.env" -f "$p6_dir/docker-compose.p6.yml" config > "$artifact_dir/p6-render-${P6_RUN_ID}.yml" + chmod 600 "$artifact_dir/p6-render-${P6_RUN_ID}.yml" +} + +run_driver() { + local action="$1" driver driver_report pattern_file + driver="$(p6_json_string '.driver')" + [[ -x "$driver" && ! -L "$driver" ]] || p6_die "GOLDEN_DRIVER_UNAVAILABLE" 76 + driver_report="$P6_WORK_DIR/driver-report.json" + pattern_file="$P6_WORK_DIR/secret-patterns" + P6_DRIVER_ACTION="$action" P6_RUN_ID="$P6_RUN_ID" P6_INPUT_FILE="$P6_INPUT_FILE" P6_DRIVER_REPORT="$driver_report" P6_SECRET_PATTERN_FILE="$pattern_file" "$driver" + case "$action" in + provision) + jq -e ' + .schema_version == "p6-driver-provision/v1" and .result == "passed" and .content_redacted == true + and (.topology | type == "object") + and (.topology | [.litellm,.shell,.jupyterhub,.launcher,.workspace] | all(. == "started")) + ' "$driver_report" >/dev/null || p6_die "TOPOLOGY_PROVISION_REPORT_INVALID" 77 + ;; + golden) + jq -e --arg patterns "$pattern_file" ' + type == "object" and .schema_version == "p6-driver-report/v1" + and .result == "passed" and .content_redacted == true + and (.checks | type == "object") and (.secret_pattern_file == $patterns) + ' "$driver_report" >/dev/null || p6_die "GOLDEN_DRIVER_REPORT_INVALID" 77 + jq -e '.checks | [.console_ui,.binding_payload,.jupyterhub_dockerspawner,.launcher_claim_activate_release,.openclaw_apply_probe_readiness,.chat,.stream,.tool,.usage,.owner_negative,.prompt_response_absent,.revoke,.generation_restart,.late_release,.delete,.zero_active_leases,.cleanup] | all(. == "passed")' "$driver_report" >/dev/null || p6_die "GOLDEN_DRIVER_CHECK_FAILED" 77 + ;; + cleanup) + jq -e ' + .schema_version == "p6-driver-cleanup/v1" and .result == "passed" and .content_redacted == true + and (.resources | [.litellm,.shell,.jupyterhub,.launcher,.workspace,.runtime_material,.temporary_files,.processes] | all(. == "absent")) + ' "$driver_report" >/dev/null || p6_die "TOPOLOGY_CLEANUP_REPORT_INVALID" 77 + ;; + esac +} + +case "$action" in + validate-input) + if prepare; then p6_write_report "$report" passed completed; else p6_write_report "$report" failed precondition_failed "input_validation"; exit 1; fi + ;; + preflight) + if preflight; then p6_write_report "$report" passed completed; else p6_write_report "$report" failed precondition_failed "preflight"; exit 1; fi + ;; + render) + if render; then p6_write_report "$report" passed completed; else p6_write_report "$report" failed precondition_failed "render"; cleanup; exit 1; fi + ;; + golden) + if ! render; then p6_write_report "$report" failed precondition_failed "render"; cleanup; exit 1; fi + trap cleanup EXIT + if ! run_driver provision; then p6_write_report "$report" failed topology_provision_failed "driver"; exit 1; fi + if ! run_driver golden; then p6_write_report "$report" failed golden_chain_failed "driver"; exit 1; fi + if ! p6_security_scan "$P6_WORK_DIR" "$P6_WORK_DIR/secret-patterns"; then p6_write_report "$report" failed security_scan_failed "secret_scan"; exit 1; fi + rm -f "$P6_WORK_DIR/secret-patterns" + if ! run_driver cleanup; then p6_write_report "$report" failed cleanup_failed "driver"; exit 1; fi + p6_write_report "$report" passed completed + ;; + cleanup) + if ! preflight || ! run_driver cleanup; then p6_write_report "$report" failed cleanup_failed "driver"; cleanup; exit 1; fi + cleanup + p6_write_report "$report" passed completed + ;; +esac diff --git a/docker_openclaw/p6/scripts/test-p6-compose-render.sh b/docker_openclaw/p6/scripts/test-p6-compose-render.sh new file mode 100755 index 0000000..bc85447 --- /dev/null +++ b/docker_openclaw/p6/scripts/test-p6-compose-render.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +# Render the P6 Compose topology with only non-sensitive fixture values. This +# validates interpolation and confirms the P6 service has no token setting. +set -euo pipefail + +script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +p6_dir="$(cd "${script_dir}/.." && pwd)" +tmpdir="$(mktemp -d "${TMPDIR:-/tmp}/p6-compose.XXXXXX")" +chmod 700 "$tmpdir" +trap 'rm -rf "$tmpdir"' EXIT +mkdir -p "$tmpdir/runtime" "$tmpdir/state" +touch "$tmpdir/adapter" +printf '%s\n' \ + 'P6_OPENCLAW_WORKSPACE_IMAGE=quay.io/labnow/labnow-open:che-563-openclaw-product-closure-local' \ + "P6_RUNTIME_MOUNT=$tmpdir/runtime" \ + "P6_OPENCLAW_STATE_DIR=$tmpdir/state" \ + 'P6_NETWORK_NAME=p6-render-fixture' > "$tmpdir/runtime.env" +chmod 600 "$tmpdir/runtime.env" +docker compose --project-name p6-render-fixture --env-file "$tmpdir/runtime.env" -f "$p6_dir/docker-compose.p6.yml" config > "$tmpdir/rendered.yml" +rg -q 'pull_policy: never' "$tmpdir/rendered.yml" +rg -q 'OPENCLAW_USE_TRUSTED_PROXY_AUTH: "true"' "$tmpdir/rendered.yml" +! rg -q 'OPENCLAW_GATEWAY_TOKEN' "$tmpdir/rendered.yml" +echo 'PASS P6 Compose rendering: fixed image input and token-free internal gateway.' diff --git a/docker_openclaw/p6/scripts/test-p6-gates.sh b/docker_openclaw/p6/scripts/test-p6-gates.sh new file mode 100755 index 0000000..7593856 --- /dev/null +++ b/docker_openclaw/p6/scripts/test-p6-gates.sh @@ -0,0 +1,44 @@ +#!/usr/bin/env bash +# Static negative gates for P6 input validation. No Docker, network, product +# repository, credential, or upstream operation is used here. +set -euo pipefail + +script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +runner="$script_dir/p6-runner.sh" +tmpdir="$(mktemp -d "${TMPDIR:-/tmp}/p6-gates.XXXXXX")" +chmod 700 "$tmpdir" +trap 'rm -rf "$tmpdir"' EXIT + +input="$tmpdir/input.json" +jq -n \ + --arg contract_sha 'd289dff9bcaa3d28035c5ed2e56b806f4b3b37fdca3159352d22f0c03942e202' \ + --arg control '2eb71d7590739df3de8db2f8cf9098154a397f0b' \ + --arg lab_dev '940325578bae9905673965d6dc489130ab4b6a46' \ + --arg open 'dfac9767fd6cdd4706ac4cd6917defcafd1c6eb8' \ + --arg shell 'eb0e6f90182e5d59174ea9edb7cb71edeaa7a47f' \ + --arg launcher 'c64f5fbabc587e26394a486ef9ae12558234f646' \ + '{schema_version:"p6-inputs/v1",contract_version:"v1alpha1",contract_bundle_sha256:$contract_sha,control_commit:$control,review_policy_commit:$control, + repositories:{lab_dev:{path:"/tmp/lab-dev",commit:$lab_dev},labnow_open:{path:"/tmp/labnow-open",commit:$open},labnow_shell:{path:"/tmp/labnow-shell",commit:$shell},labnow_launcher:{path:"/tmp/labnow-launcher",commit:$launcher}}, + images:{litellm:{ref:"quay.io/labnow/litellm:1.97.0-ead62528e607",image_id:("sha256:" + ("a" * 64)),provenance:"repo_digest",repo_digest:("quay.io/labnow/litellm@sha256:" + ("a" * 64))},openclaw_base:{ref:("quay.io/labnow/openclaw@sha256:" + ("b" * 64)),image_id:("sha256:" + ("b" * 64)),provenance:"repo_digest",repo_digest:("quay.io/labnow/openclaw@sha256:" + ("b" * 64))},openclaw_workspace:{ref:"quay.io/labnow/labnow-open:che-563-openclaw-product-closure-local",image_id:("sha256:" + ("c" * 64)),provenance:"local_build",repo_digest:"absent",source_repository:"labnow_open",source_commit:$open,base_image_digest:("quay.io/labnow/openclaw@sha256:" + ("b" * 64))}}, + paths:{runtime_mount:"/tmp/runtime",workspace_root:"/tmp/workspace"},driver:"/tmp/driver"}' > "$input" +chmod 600 "$input" + +# An example-like but structurally fixed input is accepted without inspecting +# Docker/repositories. Mutable refs and control/contract mismatches must fail. +P6_RUN_ID="p6-$(python3 -c 'print("0" * 32)')" P6_ARTIFACTS_DIR="$tmpdir/artifacts" "$runner" --input "$input" --validate-input >/dev/null +latest="$tmpdir/latest.json" +jq '.images.openclaw_workspace.ref = "quay.io/labnow/labnow-open:latest"' "$input" > "$latest"; chmod 600 "$latest" +if P6_ARTIFACTS_DIR="$tmpdir/artifacts" "$runner" --input "$latest" --validate-input >/dev/null 2>&1; then echo 'accepted latest image' >&2; exit 1; fi +mismatch="$tmpdir/mismatch.json" +jq '.contract_bundle_sha256 = "0000000000000000000000000000000000000000000000000000000000000000"' "$input" > "$mismatch"; chmod 600 "$mismatch" +if P6_ARTIFACTS_DIR="$tmpdir/artifacts" "$runner" --input "$mismatch" --validate-input >/dev/null 2>&1; then echo 'accepted contract mismatch' >&2; exit 1; fi + +# The orchestration source must not fall back to latest or a plaintext gateway +# token, and aggregation must reject absent golden evidence. +rg -q 'P6_OPENCLAW_WORKSPACE_IMAGE' "$script_dir/../docker-compose.p6.yml" +! rg -n 'P6_OPENCLAW_IMAGE|P6_OPENCLAW_ADAPTER' "$script_dir/../docker-compose.p6.yml" "$script_dir/p6-runner.sh" +rg -q 'run_driver provision' "$script_dir/p6-runner.sh" +rg -q 'run_driver cleanup' "$script_dir/p6-runner.sh" +! rg -q 'up -d --wait openclaw-workspace' "$script_dir/p6-runner.sh" +if "$script_dir/p6-aggregate.sh" --artifacts "$tmpdir/artifacts" --run-id "p6-$(python3 -c 'print("0" * 32)')" >/dev/null 2>&1; then echo 'accepted incomplete evidence' >&2; exit 1; fi +echo 'PASS P6 gates: fixed-input mismatches and incomplete evidence fail closed.' From 1b4562899e03eacdee5a86eb55b47d5e12117ee8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Mon, 10 Aug 2026 15:17:46 +0800 Subject: [PATCH 57/87] =?UTF-8?q?fix:=20=E6=89=A9=E5=B1=95=20P6=20?= =?UTF-8?q?=E8=BF=90=E8=A1=8C=E6=80=81=E5=AF=86=E9=92=A5=E6=89=AB=E6=8F=8F?= =?UTF-8?q?=E8=8C=83=E5=9B=B4?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_openclaw/p6/scripts/p6-lib.sh | 6 ++++-- docker_openclaw/p6/scripts/p6-runner.sh | 8 +++++++- 2 files changed, 11 insertions(+), 3 deletions(-) diff --git a/docker_openclaw/p6/scripts/p6-lib.sh b/docker_openclaw/p6/scripts/p6-lib.sh index f8e2a37..a06d08c 100755 --- a/docker_openclaw/p6/scripts/p6-lib.sh +++ b/docker_openclaw/p6/scripts/p6-lib.sh @@ -146,10 +146,12 @@ p6_assert_runtime_paths() { p6_security_scan() { # Write secret strings only to a mode-0600 pattern file and pass its path to # rg. Neither the shell command line nor the report contains the secret. - local scan_root="$1" patterns="$2" + local patterns="$1" + shift p6_require_regular_0600 "$patterns" || return $? [[ -s "$patterns" ]] || { p6_die "SECRET_PATTERN_FILE_REQUIRED" 74; return $?; } - if rg --fixed-strings --files-with-matches --glob '!secret-patterns' --glob '!secret.json' -f "$patterns" "$scan_root" >/dev/null 2>&1; then + (($# > 0)) || { p6_die "SECRET_SCAN_ROOT_REQUIRED" 74; return $?; } + if rg --fixed-strings --files-with-matches --glob '!secret-patterns' --glob '!secret.json' -f "$patterns" "$@" >/dev/null 2>&1; then rm -f "$patterns" p6_die "SECRET_FINGERPRINT_MATCH" 75 return $? diff --git a/docker_openclaw/p6/scripts/p6-runner.sh b/docker_openclaw/p6/scripts/p6-runner.sh index 84dad29..7cc3e61 100755 --- a/docker_openclaw/p6/scripts/p6-runner.sh +++ b/docker_openclaw/p6/scripts/p6-runner.sh @@ -115,6 +115,7 @@ run_driver() { type == "object" and .schema_version == "p6-driver-report/v1" and .result == "passed" and .content_redacted == true and (.checks | type == "object") and (.secret_pattern_file == $patterns) + and (.scan_roots | type == "array" and length >= 1 and all(.[]; type == "string" and startswith("/"))) ' "$driver_report" >/dev/null || p6_die "GOLDEN_DRIVER_REPORT_INVALID" 77 jq -e '.checks | [.console_ui,.binding_payload,.jupyterhub_dockerspawner,.launcher_claim_activate_release,.openclaw_apply_probe_readiness,.chat,.stream,.tool,.usage,.owner_negative,.prompt_response_absent,.revoke,.generation_restart,.late_release,.delete,.zero_active_leases,.cleanup] | all(. == "passed")' "$driver_report" >/dev/null || p6_die "GOLDEN_DRIVER_CHECK_FAILED" 77 ;; @@ -142,7 +143,12 @@ case "$action" in trap cleanup EXIT if ! run_driver provision; then p6_write_report "$report" failed topology_provision_failed "driver"; exit 1; fi if ! run_driver golden; then p6_write_report "$report" failed golden_chain_failed "driver"; exit 1; fi - if ! p6_security_scan "$P6_WORK_DIR" "$P6_WORK_DIR/secret-patterns"; then p6_write_report "$report" failed security_scan_failed "secret_scan"; exit 1; fi + scan_roots=("$P6_WORK_DIR") + while IFS= read -r scan_root; do + [[ -e "$scan_root" && ! -L "$scan_root" ]] || { p6_write_report "$report" failed security_scan_failed "scan_root"; exit 1; } + scan_roots+=("$scan_root") + done < <(jq -r '.scan_roots[]' "$P6_WORK_DIR/driver-report.json") + if ! p6_security_scan "$P6_WORK_DIR/secret-patterns" "${scan_roots[@]}"; then p6_write_report "$report" failed security_scan_failed "secret_scan"; exit 1; fi rm -f "$P6_WORK_DIR/secret-patterns" if ! run_driver cleanup; then p6_write_report "$report" failed cleanup_failed "driver"; exit 1; fi p6_write_report "$report" passed completed From 9d78780e9cfb6ac09410c3a21da7de1c2f59a1b4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Mon, 10 Aug 2026 21:36:54 +0800 Subject: [PATCH 58/87] =?UTF-8?q?feat:=20=E5=AE=8C=E6=88=90=20P6=20OpenCla?= =?UTF-8?q?w=20=E7=AB=AF=E5=88=B0=E7=AB=AF=E9=BB=84=E9=87=91=E8=81=94?= =?UTF-8?q?=E8=B0=83?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_openclaw/README.md | 13 +- docker_openclaw/p6/README.md | 118 ++- docker_openclaw/p6/docker-compose.p6.yml | 37 - docker_openclaw/p6/docker-compose.runtime.yml | 222 ++++ docker_openclaw/p6/p6-inputs.example.json | 40 +- docker_openclaw/p6/scripts/p6-aggregate.sh | 37 +- docker_openclaw/p6/scripts/p6-full-driver.sh | 138 +++ docker_openclaw/p6/scripts/p6-lib.sh | 261 +++-- .../p6/scripts/p6-prepare-runtime.py | 424 ++++++++ .../p6/scripts/p6-product-chain.py | 960 ++++++++++++++++++ docker_openclaw/p6/scripts/p6-runner.sh | 123 +-- docker_openclaw/p6/scripts/p6-user-center.py | 58 ++ .../p6/scripts/test-p6-compose-render.sh | 69 +- .../p6/scripts/test-p6-driver-flow.sh | 248 +++++ docker_openclaw/p6/scripts/test-p6-gates.sh | 76 +- 15 files changed, 2542 insertions(+), 282 deletions(-) delete mode 100644 docker_openclaw/p6/docker-compose.p6.yml create mode 100644 docker_openclaw/p6/docker-compose.runtime.yml create mode 100755 docker_openclaw/p6/scripts/p6-full-driver.sh mode change 100755 => 100644 docker_openclaw/p6/scripts/p6-lib.sh create mode 100755 docker_openclaw/p6/scripts/p6-prepare-runtime.py create mode 100755 docker_openclaw/p6/scripts/p6-product-chain.py create mode 100755 docker_openclaw/p6/scripts/p6-user-center.py create mode 100755 docker_openclaw/p6/scripts/test-p6-driver-flow.sh diff --git a/docker_openclaw/README.md b/docker_openclaw/README.md index 719d69b..8b692b7 100644 --- a/docker_openclaw/README.md +++ b/docker_openclaw/README.md @@ -43,13 +43,16 @@ docker run -d \ P6 的固定组合编排、黄金 runner、报告聚合和安全清理由 [`p6/README.md`](p6/README.md) 维护。该入口只接受本地受限输入文件中 -固定的四仓 commit、RC1 bundle hash,以及下列冻结镜像事实:LiteLLM +固定的三仓 commit、`lab-dev` review_snapshot、RC1 bundle hash,以及下列冻结镜像事实:LiteLLM `quay.io/labnow/litellm:1.97.0-ead62528e607` 的本地 ID/digest `sha256:a2e115874c21b829bd052b18fc85be2f9217fb8244b82812c4ebc6e36f9824d1`; 上游 OpenClaw `quay.io/labnow/openclaw@sha256:edc85cc2068f5ec0df470f7d06daa0a4fbd78ef5ad6cf5b48f58381da839dd12`; 以及 P6 Workspace 本地镜像 `quay.io/labnow/labnow-open:che-563-openclaw-product-closure-local` 的 ID -`sha256:79fbe459040bc10cb8a64934fa608d7cd23ac4b0472e7c7690db294ad54ffbb7`、 -source commit `5b70a6b0f960ddc1a5c45a27449cd7317da0c7da` 和上游 OpenClaw -base digest。三者均为 `amd64`,P6 仅以 local-only 方式运行,不拉取、 -推送、发布或部署镜像,也不会把凭据写入仓库、报告或命令参数。 +`sha256:c9c6a45637521cbbaeacea57fbb128696066fd91c5dff4521555f1bd5211f244`、 +实际 RepoDigest `quay.io/labnow/labnow-open@sha256:c9c6a45637521cbbaeacea57fbb128696066fd91c5dff4521555f1bd5211f244`、 +source commit `21019e0c24dc7b51747c2bef3cd90f5d259be839` 和上游 OpenClaw +base digest。Launcher、Shell 与 PostgreSQL、Redis、nginx support image 也必须 +提供并回读准确本地 image ID / RepoDigest。P6 通过 live DockerSpawner 创建 +Workspace,所有运行材料和数据卷均按 run 隔离;它仅以 local-only 方式运行, +不拉取、推送、发布或部署镜像,也不会把凭据写入仓库、报告或命令参数。 diff --git a/docker_openclaw/p6/README.md b/docker_openclaw/p6/README.md index 594b7b3..28e4e32 100644 --- a/docker_openclaw/p6/README.md +++ b/docker_openclaw/p6/README.md @@ -1,61 +1,95 @@ # P6 OpenClaw 产品闭环(本仓编排) -本目录仅承担 LLM Hub V1 / P6 的 `lab-dev` 职责:固定本地组合输入、 -OpenClaw Compose、黄金 runner、脱敏报告聚合和清理入口。Shell、Launcher -和 OpenClaw 产品业务逻辑仍由各自仓库拥有。 +本目录只承担 LLM Hub V1 / P6 的 `lab-dev` 职责:固定本地组合、创建隔离 +拓扑、执行黄金 runner、聚合脱敏报告并清理本轮资源。Shell、Launcher 和 +OpenClaw 的产品业务逻辑仍由各自仓库拥有。 -当前状态:本仓 runner 与固定输入门禁已具备,尚未达到 ready。真实黄金运行 -必须等待四仓 tracked 工作树干净,并提供能自动编排五个组件的固定 driver; -在此之前,`--preflight`/`--golden` 必须失败关闭,handoff 为 `blocked`。 +当前状态:真实五组件 driver 与失败关闭门禁已写入工作树;在真实黄金 run、 +有界复审及 promotion 完成前,P6 仍不是 `verified`。 -## 输入与安全边界 +## 固定输入与 review_snapshot -从 [`p6-inputs.example.json`](p6-inputs.example.json) 创建本地 -`p6-inputs.json`,权限必须为 `0400` 或 `0600`。该文件是 Git 忽略的, -只保存路径、固定 commit、镜像 ID/digest 与本地 driver 路径。测试 Secret -由 driver 在运行中以受限文件生成;它只将模式为 `0600` 的 pattern 文件 -交给 runner 做零命中扫描,绝不写入输入、报告或命令参数。 +从 [`p6-inputs.example.json`](p6-inputs.example.json) 创建 Git 忽略的 +`p6-inputs.json`,权限必须为 `0400` 或 `0600`。输入只包含: -Runner 拒绝以下情况:浮动 `latest`、非 `quay.io/labnow/` 镜像、commit/ -RC1 bundle 不匹配、任何冻结仓 tracked 工作树有变更、本机镜像 ID/digest -不匹配、输入/Secret 权限不安全或运行时路径缺失。它从不 source `.env`。 -本地 Workspace 镜像可以明确标为 `local_build` 且 `repo_digest="absent"`, -此时必须提供准确 image ID、`labnow-open` source commit 与固定上游 -OpenClaw base digest;runner 会校验三者,不会伪造 repository digest。 +- `lab-dev` 的 Phase branch、base、当前 `HEAD`、tracked diff SHA-256 和变更文件集; +- 其余三仓的准确 commit; +- LiteLLM、OpenClaw base、Workspace、Launcher、Shell、PostgreSQL、Redis、 + nginx 的准确本地 image ID 与 RepoDigest; +- P1 本地测试 `.env` 的绝对路径。 -## 入口 +`lab-dev` 可以用受保护的 `review_snapshot` 进入 Review,不要求提前 commit。 +Runner 会重新计算: + +```bash +git diff --binary --full-index --no-ext-diff -- | shasum -a 256 +``` + +并核对分支、`HEAD`、文件集和 SHA-256。其他三仓必须停在准确 commit 且 tracked +工作树 clean。未知 untracked 文件不属于输入,也不会被读取、删除或修改。 + +## 真实拓扑 + +[`docker-compose.runtime.yml`](docker-compose.runtime.yml) 每次创建一个独立的: + +- 固定 LiteLLM + 独立 PostgreSQL / Redis / migration; +- 固定 Shell + 独立 PostgreSQL / migration; +- run-scoped User Center fixture; +- 固定 Launcher / live JupyterHub; +- run-scoped HTTPS LiteLLM gateway; +- 由 live DockerSpawner 创建的固定 OpenClaw Workspace。 + +Workspace 不由第二份 Compose 旁路创建。Launcher 通过真实 Shell 内部接口完成 +claim → materialize → Adapter apply/probe → activate,stop/restart/delete 时完成 +release。P1 `.env` 只由受版本控制的 preparer 程序化读取;所有测试 key、Hub +token、服务 token、数据库密码、KEK 和本地证书均在本轮 `0700` 目录内以 +`0400`/`0600` 文件生成,不进入输入、命令参数、Git 或脱敏报告。 + +## Runner 门禁 ```bash ./docker_openclaw/p6/scripts/test-p6-gates.sh +./docker_openclaw/p6/scripts/test-p6-driver-flow.sh ./docker_openclaw/p6/scripts/test-p6-compose-render.sh ./docker_openclaw/p6/scripts/p6-runner.sh --input /secure/path/p6-inputs.json --preflight ./docker_openclaw/p6/scripts/p6-runner.sh --input /secure/path/p6-inputs.json --render -./docker_openclaw/p6/scripts/p6-runner.sh --input /secure/path/p6-inputs.json --golden -./docker_openclaw/p6/scripts/p6-runner.sh --input /secure/path/p6-inputs.json --cleanup +P6_RUN_ID=p6-<32hex> ./docker_openclaw/p6/scripts/p6-runner.sh --input /secure/path/p6-inputs.json --golden +P6_RUN_ID=p6- ./docker_openclaw/p6/scripts/p6-runner.sh --input /secure/path/p6-inputs.json --cleanup ``` -`--golden` 在严格 preflight 后调用输入中已固定的跨仓 driver,强制其按 -`provision → golden → cleanup` 执行:`provision` 必须在同一 run 中启动固定 -LiteLLM、Shell、live JupyterHub、Launcher 与 P6 Workspace;Workspace 必须 -使用本目录 Compose;`golden` 覆盖 Console、JupyterHub/DockerSpawner、 -Launcher claim/activate/release、Adapter、chat/stream/tool、用量、撤销、 -generation、delete 与零 active lease;`cleanup` 必须证明所有上述资源、 -运行材料、临时文件和进程均不存在。缺少任何阶段或检查都会失败关闭,不能 -把外部服务预先手工启动后当作 P6 成功。 - -本轮固定组合为 LiteLLM 本地 ID/digest、上游 OpenClaw base digest,以及 -`quay.io/labnow/labnow-open:che-563-openclaw-product-closure-local` Workspace -本地镜像;Workspace 的 local-only provenance 绑定 -`labnow-open@5b70a6b0f960ddc1a5c45a27449cd7317da0c7da` 与上游 OpenClaw -base digest。P6 Compose 仅定义该 Workspace,完整 driver 负责启动同一 run -中的 LiteLLM、Shell、JupyterHub 和 Launcher,并在 cleanup 中全部移除。 - -运行产生的脱敏报告位于 `p6/artifacts/`(Git 忽略)。只有同一 run 的 -preflight、golden、cleanup 都 `passed`,才允许: +`--golden` 在一个 run 中完成: + +1. 通过 Shell 真实 API 创建 connection、route、binding; +2. 通过 Shell 调用 live JupyterHub,由真实 DockerSpawner 创建 Workspace; +3. 核对最小 `model_access`、只读材料、Adapter status、访问入口和 readiness; +4. 执行 chat、stream、tool,并按 owner / Workspace / key / model / time 查询用量; +5. stop 后验证旧 key 被拒绝;restart 后验证 generation 增加、新 key 成功、 + 旧 key 仍拒绝、旧 generation 迟到 release 返回 409; +6. delete 后验证新 key 被拒绝、active lease 为零; +7. 扫描 Shell、Launcher、LiteLLM、Workspace 日志/进程/脱敏 inspect、OpenClaw + 配置与本轮 Workspace 文件,确认测试 Secret 零命中; +8. 只删除本 run 的准确容器、网络、数据卷、运行材料和临时文件。 + +Shell 在 `5c9411dfd3c4d7b1e606c0d9dc0c5e62313bc376` 的真实鼠标流程证据和 +LiteLLM `1.97.0` v2 key-alias 分页用量 smoke 按冻结 Review 结论复用;runner 不伪装成 +重新执行浏览器 UI。`test-p6-driver-flow.sh` +只验证编排、报告和 cleanup 的确定性,不能替代真实黄金 run。 + +## 证据与聚合 + +脱敏报告位于 Git 忽略的 `p6/artifacts/`。`provision`、`golden`、`cleanup` +三份 driver 报告绑定同一 run、输入 SHA-256 和准确 provenance。报告只保留 +结构断言、非敏感 ID、计数、状态和 SHA-256,不保留 key、Prompt、Response +正文、原始环境或私钥。 + +只有同一 run 的 preflight、golden、cleanup 都为 `passed`,且阶段报告 hash +一致时,才允许聚合: ```bash -./docker_openclaw/p6/scripts/p6-aggregate.sh --artifacts docker_openclaw/p6/artifacts --run-id p6- +./docker_openclaw/p6/scripts/p6-aggregate.sh \ + --artifacts docker_openclaw/p6/artifacts \ + --run-id p6- ``` -清理只移除本 runner 创建的 Compose 资源与受限临时目录,不删除数据卷、 -用户目录或其他项目资源。P6 不推送镜像、不部署、不推进 integration。 +P6 只在本地运行,不拉取或推送产品分支,不发布镜像,不部署,也不修改任何 +`main`。 diff --git a/docker_openclaw/p6/docker-compose.p6.yml b/docker_openclaw/p6/docker-compose.p6.yml deleted file mode 100644 index 8c18246..0000000 --- a/docker_openclaw/p6/docker-compose.p6.yml +++ /dev/null @@ -1,37 +0,0 @@ -name: p6-openclaw-closure - -# The P6 runner renders this file only after p6-inputs.json has bound every -# image to a local ID and repository digest. There are intentionally no image -# defaults, no `latest`, and no credential values in this Compose file. -services: - openclaw-workspace: - image: ${P6_OPENCLAW_WORKSPACE_IMAGE:?p6-runner must supply the fixed labnow-open Workspace image} - pull_policy: never - restart: "no" - environment: - OPENCLAW_GATEWAY_BIND: lan - OPENCLAW_GATEWAY_PORT: "18789" - # P6 keeps the gateway on the internal network. The real Console/Shell - # flow authenticates at its owning boundary; no gateway token is added - # to a persistent OpenClaw configuration. - OPENCLAW_USE_TRUSTED_PROXY_AUTH: "true" - LABNOW_MODEL_ACCESS_STATE_DIR: /root/.openclaw/data/labnow-model-access - volumes: - - type: bind - source: ${P6_RUNTIME_MOUNT:?p6-runner must supply the Launcher-managed runtime mount} - target: /run/labnow/model-access - read_only: true - - type: bind - source: ${P6_OPENCLAW_STATE_DIR:?p6-runner must supply an isolated run state directory} - target: /root/.openclaw/data - networks: [p6-net] - healthcheck: - test: ["CMD-SHELL", "openclaw config validate >/dev/null 2>&1"] - interval: 5s - timeout: 3s - retries: 12 - start_period: 10s - -networks: - p6-net: - name: ${P6_NETWORK_NAME:?p6-runner must supply an isolated network name} diff --git a/docker_openclaw/p6/docker-compose.runtime.yml b/docker_openclaw/p6/docker-compose.runtime.yml new file mode 100644 index 0000000..19c4c53 --- /dev/null +++ b/docker_openclaw/p6/docker-compose.runtime.yml @@ -0,0 +1,222 @@ +name: p6-runtime + +# One isolated P6 run. Product images are fixed in p6-inputs.json and validated +# before this file is used. Secret values are only read from run-scoped 0400/ +# 0600 files; this file and the retained rendering contain no credential. +services: + litellm-postgres: + image: ${P6_POSTGRES_IMAGE:?fixed PostgreSQL image required} + pull_policy: never + container_name: ${P6_LITELLM_POSTGRES_CONTAINER:?run-scoped container required} + restart: "no" + environment: + POSTGRES_DB: p6_litellm + POSTGRES_USER: p6_litellm + POSTGRES_PASSWORD_FILE: /run/secrets/litellm_postgres_password + secrets: [litellm_postgres_password] + volumes: + - litellm_postgres_data:/var/lib/postgresql/data + networks: [p6-runtime] + healthcheck: + test: ["CMD-SHELL", "pg_isready -U p6_litellm -d p6_litellm"] + interval: 3s + timeout: 3s + retries: 30 + + litellm-redis: + image: ${P6_REDIS_IMAGE:?fixed Redis image required} + pull_policy: never + restart: "no" + secrets: [litellm_redis_password] + tmpfs: + - /run/p6:mode=0700 + command: + - /bin/sh + - -ec + - >- + umask 077; + { printf 'appendonly yes\nrequirepass '; cat /run/secrets/litellm_redis_password; printf '\n'; } > /run/p6/redis.conf; + exec redis-server /run/p6/redis.conf + volumes: + - litellm_redis_data:/data + networks: [p6-runtime] + healthcheck: + test: ["CMD-SHELL", "REDISCLI_AUTH=$$(cat /run/secrets/litellm_redis_password) redis-cli --no-auth-warning ping | grep -qx PONG"] + interval: 3s + timeout: 3s + retries: 30 + + litellm-migrate: + image: ${P6_LITELLM_IMAGE:?fixed LiteLLM image required} + pull_policy: never + restart: "no" + env_file: ${P6_LITELLM_ENV_FILE:?restricted LiteLLM env required} + command: ["python3", "/opt/utils/run-migration-locked.py", "--config", "config.migrate.yaml", "--skip_server_startup", "--enforce_prisma_migration_check"] + secrets: [litellm_redis_password] + volumes: + - ${P6_LITELLM_CONFIG:?fixed LiteLLM config required}:/opt/litellm/config.yaml:ro + - ${P6_LITELLM_MIGRATE_CONFIG:?fixed LiteLLM migration config required}:/opt/litellm/config.migrate.yaml:ro + - ${P6_LITELLM_START_SCRIPT:?fixed LiteLLM start script required}:/opt/utils/start-litellm.sh:ro + - ${P6_LITELLM_MIGRATION_SCRIPT:?fixed LiteLLM migration script required}:/opt/utils/run-migration-locked.py:ro + depends_on: + litellm-postgres: {condition: service_healthy} + litellm-redis: {condition: service_healthy} + networks: [p6-runtime] + + litellm: + image: ${P6_LITELLM_IMAGE:?fixed LiteLLM image required} + pull_policy: never + container_name: ${P6_LITELLM_CONTAINER:?run-scoped container required} + restart: "no" + env_file: ${P6_LITELLM_ENV_FILE:?restricted LiteLLM env required} + command: ["start-litellm.sh"] + secrets: [litellm_redis_password] + volumes: + - ${P6_LITELLM_CONFIG:?fixed LiteLLM config required}:/opt/litellm/config.yaml:ro + - ${P6_LITELLM_MIGRATE_CONFIG:?fixed LiteLLM migration config required}:/opt/litellm/config.migrate.yaml:ro + - ${P6_LITELLM_START_SCRIPT:?fixed LiteLLM start script required}:/opt/utils/start-litellm.sh:ro + - ${P6_LITELLM_MIGRATION_SCRIPT:?fixed LiteLLM migration script required}:/opt/utils/run-migration-locked.py:ro + depends_on: + litellm-migrate: {condition: service_completed_successfully} + networks: [p6-runtime] + healthcheck: + test: ["CMD-SHELL", "python3 -c \"import urllib.request; urllib.request.urlopen('http://127.0.0.1:4000/health/readiness', timeout=3)\""] + interval: 5s + timeout: 5s + retries: 36 + start_period: 20s + + litellm-gateway: + image: ${P6_NGINX_IMAGE:?fixed nginx image required} + pull_policy: never + restart: "no" + ports: + - "127.0.0.1::4443" + volumes: + - ${P6_NGINX_CONFIG:?run-scoped nginx config required}:/etc/nginx/nginx.conf:ro + - ${P6_CA_CERT:?run-scoped certificate required}:/run/p6/p6-ca.pem:ro + - ${P6_CA_KEY:?run-scoped private key required}:/run/p6/p6-ca.key:ro + depends_on: + litellm: {condition: service_healthy} + networks: [p6-runtime] + healthcheck: + test: ["CMD-SHELL", "wget --no-check-certificate -qO- https://127.0.0.1:4443/health/readiness >/dev/null"] + interval: 3s + timeout: 3s + retries: 30 + + shell-postgres: + image: ${P6_POSTGRES_IMAGE:?fixed PostgreSQL image required} + pull_policy: never + container_name: ${P6_SHELL_POSTGRES_CONTAINER:?run-scoped container required} + restart: "no" + environment: + POSTGRES_DB: p6_shell + POSTGRES_USER: p6_shell + POSTGRES_PASSWORD_FILE: /run/secrets/shell_postgres_password + secrets: [shell_postgres_password] + volumes: + - shell_postgres_data:/var/lib/postgresql/data + networks: [p6-runtime] + healthcheck: + test: ["CMD-SHELL", "pg_isready -U p6_shell -d p6_shell"] + interval: 3s + timeout: 3s + retries: 30 + + shell-migrate: + image: ${P6_POSTGRES_IMAGE:?fixed PostgreSQL image required} + pull_policy: never + restart: "no" + entrypoint: ["/bin/sh", "-ec"] + command: + - >- + export PGPASSWORD="$$(cat /run/secrets/shell_postgres_password)"; + exec psql -v ON_ERROR_STOP=1 -h shell-postgres -U p6_shell -d p6_shell -f /run/p6/001_initial.sql + secrets: [shell_postgres_password] + volumes: + - ${P6_SHELL_MIGRATION:?fixed Shell migration required}:/run/p6/001_initial.sql:ro + depends_on: + shell-postgres: {condition: service_healthy} + networks: [p6-runtime] + + user-center: + image: ${P6_LAUNCHER_IMAGE:?fixed Launcher image required} + pull_policy: never + restart: "no" + entrypoint: ["python", "/run/p6/p6-user-center.py"] + environment: + P6_OWNER_A: ${P6_OWNER_A:?owner A required} + P6_OWNER_B: ${P6_OWNER_B:?owner B required} + volumes: + - ${P6_USER_CENTER_SCRIPT:?fixed fixture script required}:/run/p6/p6-user-center.py:ro + networks: [p6-runtime] + healthcheck: + test: ["CMD-SHELL", "python -c \"import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health', timeout=3)\""] + interval: 3s + timeout: 3s + retries: 20 + + launcher: + image: ${P6_LAUNCHER_IMAGE:?fixed Launcher image required} + pull_policy: never + container_name: ${P6_LAUNCHER_CONTAINER:?run-scoped container required} + restart: "no" + env_file: ${P6_LAUNCHER_ENV_FILE:?restricted Launcher env required} + ports: + - "127.0.0.1::8000" + volumes: + - /var/run/docker.sock:/var/run/docker.sock:rw + - ${P6_WORKSPACE_ROOT:?isolated Workspace root required}:${P6_WORKSPACE_ROOT}:rw + - ${P6_LAUNCHER_DATA_DIR:?isolated Launcher state required}:/opt/jupyterhub/data:rw + - ${P6_LAUNCHER_APP_CONF:?run-scoped app config required}:/opt/jupyterhub/resource/config/app.conf:ro + - ${P6_MODEL_ACCESS_CONFIG:?restricted model access config required}:/run/p6/model-access.json:ro + - ${P6_CA_CERT:?run-scoped CA required}:${P6_CA_CERT}:ro + depends_on: + shell: {condition: service_healthy} + networks: + p6-runtime: + aliases: [launcher] + healthcheck: + test: ["CMD-SHELL", "curl -fsS http://127.0.0.1:8000/studio/hub/health >/dev/null"] + interval: 5s + timeout: 5s + retries: 36 + start_period: 20s + + shell: + image: ${P6_SHELL_IMAGE:?fixed Shell image required} + pull_policy: never + container_name: ${P6_SHELL_CONTAINER:?run-scoped container required} + restart: "no" + env_file: ${P6_SHELL_ENV_FILE:?restricted Shell env required} + ports: + - "127.0.0.1::3002" + depends_on: + shell-migrate: {condition: service_completed_successfully} + user-center: {condition: service_healthy} + litellm-gateway: {condition: service_healthy} + networks: [p6-runtime] + healthcheck: + test: ["CMD-SHELL", "node -e \"fetch('http://127.0.0.1:3002/console/api/model-access/connections').then(r=>process.exit(r.status===401?0:1)).catch(()=>process.exit(1))\""] + interval: 5s + timeout: 5s + retries: 36 + start_period: 20s + +secrets: + litellm_postgres_password: + file: ${P6_LITELLM_POSTGRES_PASSWORD_FILE:?restricted LiteLLM PostgreSQL password required} + litellm_redis_password: + file: ${P6_REDIS_PASSWORD_FILE:?restricted Redis password required} + shell_postgres_password: + file: ${P6_SHELL_POSTGRES_PASSWORD_FILE:?restricted Shell PostgreSQL password required} + +volumes: + litellm_postgres_data: + litellm_redis_data: + shell_postgres_data: + +networks: + p6-runtime: + name: ${P6_RUNTIME_NETWORK:?isolated network required} diff --git a/docker_openclaw/p6/p6-inputs.example.json b/docker_openclaw/p6/p6-inputs.example.json index 7358d31..99e3fe3 100644 --- a/docker_openclaw/p6/p6-inputs.example.json +++ b/docker_openclaw/p6/p6-inputs.example.json @@ -1,24 +1,36 @@ { - "schema_version": "p6-inputs/v1", + "schema_version": "p6-inputs/v2", "contract_version": "v1alpha1", "contract_bundle_sha256": "d289dff9bcaa3d28035c5ed2e56b806f4b3b37fdca3159352d22f0c03942e202", "control_commit": "2eb71d7590739df3de8db2f8cf9098154a397f0b", - "review_policy_commit": "2eb71d7590739df3de8db2f8cf9098154a397f0b", + "review_policy_commit": "680ca92661a08254eb396ab809f478bbdba3510e", "repositories": { - "lab_dev": {"path": "/absolute/path/to/lab-dev", "commit": "940325578bae9905673965d6dc489130ab4b6a46"}, - "labnow_open": {"path": "/absolute/path/to/labnow-open", "commit": "dfac9767fd6cdd4706ac4cd6917defcafd1c6eb8"}, - "labnow_shell": {"path": "/absolute/path/to/labnow-shell", "commit": "eb0e6f90182e5d59174ea9edb7cb71edeaa7a47f"}, - "labnow_launcher": {"path": "/absolute/path/to/labnow-launcher", "commit": "c64f5fbabc587e26394a486ef9ae12558234f646"} + "lab_dev": { + "path": "/absolute/path/to/lab-dev", + "delivery_identity": "review_snapshot", + "branch": "dev/che-563-openclaw-product-closure", + "phase_base_commit": "940325578bae9905673965d6dc489130ab4b6a46", + "head_commit": "1b4562899e03eacdee5a86eb55b47d5e12117ee8", + "tracked_diff_sha256": "0000000000000000000000000000000000000000000000000000000000000000", + "changed_files": ["docker_openclaw/p6/README.md"] + }, + "labnow_open": {"path": "/absolute/path/to/labnow-open", "delivery_identity": "commit", "commit": "21019e0c24dc7b51747c2bef3cd90f5d259be839"}, + "labnow_shell": {"path": "/absolute/path/to/labnow-shell", "delivery_identity": "commit", "commit": "5c9411dfd3c4d7b1e606c0d9dc0c5e62313bc376"}, + "labnow_launcher": {"path": "/absolute/path/to/labnow-launcher", "delivery_identity": "commit", "commit": "c84edea3e051d561f28d9f99235563cf491aaeb2"} }, "images": { - "litellm": {"ref": "quay.io/labnow/litellm:REPLACE_WITH_FIXED_TAG", "image_id": "sha256:REPLACE_WITH_LOCAL_IMAGE_ID", "provenance": "repo_digest", "repo_digest": "quay.io/labnow/litellm@sha256:REPLACE_WITH_REPO_DIGEST"}, - "openclaw_base": {"ref": "quay.io/labnow/openclaw@sha256:REPLACE_WITH_VERIFIED_BASE_DIGEST", "image_id": "sha256:REPLACE_WITH_LOCAL_IMAGE_ID", "provenance": "repo_digest", "repo_digest": "quay.io/labnow/openclaw@sha256:REPLACE_WITH_VERIFIED_BASE_DIGEST"}, - "openclaw_workspace": {"ref": "quay.io/labnow/labnow-open:che-563-openclaw-product-closure-local", "image_id": "sha256:REPLACE_WITH_LOCAL_IMAGE_ID", "provenance": "local_build", "repo_digest": "absent", "source_repository": "labnow_open", "source_commit": "REPLACE_WITH_LABNOW_OPEN_PHASE_COMMIT", "base_image_digest": "quay.io/labnow/openclaw@sha256:REPLACE_WITH_VERIFIED_BASE_DIGEST"} + "litellm": {"ref": "quay.io/labnow/litellm:1.97.0-ead62528e607", "image_id": "sha256:a2e115874c21b829bd052b18fc85be2f9217fb8244b82812c4ebc6e36f9824d1", "provenance": "repo_digest", "repo_digest": "quay.io/labnow/litellm@sha256:a2e115874c21b829bd052b18fc85be2f9217fb8244b82812c4ebc6e36f9824d1"}, + "openclaw_base": {"ref": "quay.io/labnow/openclaw@sha256:edc85cc2068f5ec0df470f7d06daa0a4fbd78ef5ad6cf5b48f58381da839dd12", "image_id": "sha256:edc85cc2068f5ec0df470f7d06daa0a4fbd78ef5ad6cf5b48f58381da839dd12", "provenance": "repo_digest", "repo_digest": "quay.io/labnow/openclaw@sha256:edc85cc2068f5ec0df470f7d06daa0a4fbd78ef5ad6cf5b48f58381da839dd12"}, + "openclaw_workspace": {"ref": "quay.io/labnow/labnow-open:che-563-openclaw-product-closure-local", "image_id": "sha256:c9c6a45637521cbbaeacea57fbb128696066fd91c5dff4521555f1bd5211f244", "provenance": "local_build", "repo_digest": "quay.io/labnow/labnow-open@sha256:c9c6a45637521cbbaeacea57fbb128696066fd91c5dff4521555f1bd5211f244", "source_repository": "labnow_open", "source_commit": "21019e0c24dc7b51747c2bef3cd90f5d259be839", "base_image_digest": "quay.io/labnow/openclaw@sha256:edc85cc2068f5ec0df470f7d06daa0a4fbd78ef5ad6cf5b48f58381da839dd12"} }, - "paths": { - "adapter": "/absolute/path/to/labnow-open/src/labnow-open-etc/openclaw-model-access-adapter.sh", - "runtime_mount": "/absolute/path/to/launcher-managed-runtime-mount", - "workspace_root": "/absolute/path/to/p6-run-workspace" + "local_only_images": { + "launcher": {"ref": "quay.io/labnow/labnow-launcher:che-563-openclaw-product-closure-local", "image_id": "sha256:6f9732fda8b86d9bfe4596e848025cc38448da4b17dfea8520e046a32b32e61f", "provenance": "local_build", "repo_digest": "quay.io/labnow/labnow-launcher@sha256:6f9732fda8b86d9bfe4596e848025cc38448da4b17dfea8520e046a32b32e61f", "source_repository": "labnow_launcher", "source_commit": "c84edea3e051d561f28d9f99235563cf491aaeb2", "base_image_digest": "quay.io/labnow/dev-hub-traefik@sha256:22e1857a5edcd2ad4468dffee32f21323f8dfcfa47a77c4b7f7386b9f50b8398", "oauth2_proxy_sha256": "6df0d30fe823d9b25e8bff4cfb3df9a9b6c1013463e9526388c29ba87ab32427"}, + "shell": {"ref": "quay.io/labnow/labnow-shell:che-563-openclaw-product-closure-local", "image_id": "sha256:d7ed71cf58eddf72642d61d4442a0820632060fac9f4eb611b28062b7f38c54d", "provenance": "local_build", "repo_digest": "quay.io/labnow/labnow-shell@sha256:d7ed71cf58eddf72642d61d4442a0820632060fac9f4eb611b28062b7f38c54d", "source_repository": "labnow_shell", "source_commit": "5c9411dfd3c4d7b1e606c0d9dc0c5e62313bc376", "base_image_digest": "quay.io/labnow/node@sha256:fd09d9de9b7aa927493acbafbb7d399c089465e988f2a6a240428cdbbd5424e2"} }, - "driver": "/absolute/path/to/local-only-p6-golden-driver" + "support_images": { + "postgres": {"ref": "postgres:17-alpine@sha256:742f40ea20b9ff2ff31db5458d127452988a2164df9e17441e191f3b72252193", "image_id": "sha256:742f40ea20b9ff2ff31db5458d127452988a2164df9e17441e191f3b72252193", "provenance": "repo_digest", "repo_digest": "postgres@sha256:742f40ea20b9ff2ff31db5458d127452988a2164df9e17441e191f3b72252193"}, + "redis": {"ref": "redis:7.4-alpine@sha256:e7723ff73d963f5cc6d9c4643ea3d989527a402a319239054e9472a7fb9219a2", "image_id": "sha256:e7723ff73d963f5cc6d9c4643ea3d989527a402a319239054e9472a7fb9219a2", "provenance": "repo_digest", "repo_digest": "redis@sha256:e7723ff73d963f5cc6d9c4643ea3d989527a402a319239054e9472a7fb9219a2"}, + "nginx": {"ref": "nginx:alpine@sha256:2f07d83bf561b506400dc183b1b2003803e39efbd22451f848adaba14d28c7c7", "image_id": "sha256:2f07d83bf561b506400dc183b1b2003803e39efbd22451f848adaba14d28c7c7", "provenance": "repo_digest", "repo_digest": "nginx@sha256:2f07d83bf561b506400dc183b1b2003803e39efbd22451f848adaba14d28c7c7"} + }, + "runtime": {"p1_env_file": "/absolute/path/to/lab-dev/docker_litellm/demo/.env"} } diff --git a/docker_openclaw/p6/scripts/p6-aggregate.sh b/docker_openclaw/p6/scripts/p6-aggregate.sh index 2304c80..bdba783 100755 --- a/docker_openclaw/p6/scripts/p6-aggregate.sh +++ b/docker_openclaw/p6/scripts/p6-aggregate.sh @@ -16,16 +16,43 @@ done reports=("$artifacts/p6-preflight-${run_id}.json" "$artifacts/p6-golden-${run_id}.json" "$artifacts/p6-cleanup-${run_id}.json") for report in "${reports[@]}"; do [[ -f "$report" ]] || { echo "P6_ERROR:EVIDENCE_INCOMPLETE" >&2; exit 1; } - jq -e --arg run "$run_id" '.schema_version == "p6-report/v1" and .run_id == $run and .result == "passed" and .phase == "completed" and .content_redacted == true and .contract_version == "v1alpha1" and .contract_bundle_sha256 == "d289dff9bcaa3d28035c5ed2e56b806f4b3b37fdca3159352d22f0c03942e202" and ([.images.litellm,.images.openclaw_base] | all(.[]; .provenance == "repo_digest" and (.repo_digest | type == "string" and test("^quay\\.io/labnow/(litellm|openclaw)@sha256:[0-9a-f]{64}$")))) and (.images.openclaw_workspace.provenance == "local_build") and (.images.openclaw_workspace.repo_digest == "absent") and (.images.openclaw_workspace.source_repository == "labnow_open") and (.images.openclaw_workspace.source_commit | type == "string" and test("^[0-9a-f]{40}$")) and (.images.openclaw_workspace.base_image_digest == .images.openclaw_base.repo_digest)' "$report" >/dev/null || { echo "P6_ERROR:EVIDENCE_REJECTED" >&2; exit 1; } + jq -e --arg run "$run_id" ' + .schema_version == "p6-report/v1" and .run_id == $run and .result == "passed" and .phase == "completed" + and .content_redacted == true and .contract_version == "v1alpha1" + and .contract_bundle_sha256 == "d289dff9bcaa3d28035c5ed2e56b806f4b3b37fdca3159352d22f0c03942e202" + and ([.images.litellm,.images.openclaw_base] | all(.[]; .provenance == "repo_digest" and (.repo_digest | type == "string" and test("^quay\\.io/labnow/(litellm|openclaw)@sha256:[0-9a-f]{64}$")))) + and (.images.openclaw_workspace.provenance == "local_build") + and (.images.openclaw_workspace.repo_digest | type == "string" and test("^quay\\.io/labnow/labnow-open@sha256:[0-9a-f]{64}$")) + and (.images.openclaw_workspace.source_repository == "labnow_open") + and (.images.openclaw_workspace.source_commit | type == "string" and test("^[0-9a-f]{40}$")) + and (.images.openclaw_workspace.base_image_digest == .images.openclaw_base.repo_digest) + and ([.local_only_images.launcher,.local_only_images.shell] | all(.[]; .provenance == "local_build" and (.repo_digest | type == "string" and test("^quay\\.io/labnow/labnow-(launcher|shell)@sha256:[0-9a-f]{64}$")))) + and ([.support_images.postgres,.support_images.redis,.support_images.nginx] | all(.[]; .provenance == "repo_digest" and (.repo_digest | type == "string" and test("^[a-z0-9./_-]+@sha256:[0-9a-f]{64}$")))) + ' "$report" >/dev/null || { echo "P6_ERROR:EVIDENCE_REJECTED" >&2; exit 1; } done input_hash="$(jq -r '.input_sha256' "${reports[0]}")" for report in "${reports[@]}"; do [[ "$(jq -r '.input_sha256' "$report")" == "$input_hash" ]] || { echo "P6_ERROR:INPUT_HASH_MISMATCH" >&2; exit 1; }; done -metadata="$(jq -c '{contract_version,contract_bundle_sha256,control_commit,review_policy_commit,repositories,images}' "${reports[0]}")" -for report in "${reports[@]}"; do [[ "$(jq -c '{contract_version,contract_bundle_sha256,control_commit,review_policy_commit,repositories,images}' "$report")" == "$metadata" ]] || { echo "P6_ERROR:METADATA_MISMATCH" >&2; exit 1; }; done +metadata="$(jq -c '{contract_version,contract_bundle_sha256,control_commit,review_policy_commit,repositories,images,local_only_images,support_images}' "${reports[0]}")" +for report in "${reports[@]}"; do [[ "$(jq -c '{contract_version,contract_bundle_sha256,control_commit,review_policy_commit,repositories,images,local_only_images,support_images}' "$report")" == "$metadata" ]] || { echo "P6_ERROR:METADATA_MISMATCH" >&2; exit 1; }; done +stage_reports="$(jq -c '.driver_stage_reports' "${reports[1]}")" +[[ "$stage_reports" != "null" ]] || { echo "P6_ERROR:DRIVER_STAGE_EVIDENCE_INCOMPLETE" >&2; exit 1; } +for report in "${reports[1]}" "${reports[2]}"; do + [[ "$(jq -c '.driver_stage_reports' "$report")" == "$stage_reports" ]] || { echo "P6_ERROR:DRIVER_STAGE_METADATA_MISMATCH" >&2; exit 1; } +done +for action in provision golden cleanup; do + stage_path="$(jq -r --arg action "$action" '.driver_stage_reports[$action].path' "${reports[1]}")" + stage_sha="$(jq -r --arg action "$action" '.driver_stage_reports[$action].sha256' "${reports[1]}")" + [[ -f "$stage_path" && ! -L "$stage_path" && "$stage_sha" =~ ^[0-9a-f]{64}$ ]] || { echo "P6_ERROR:DRIVER_STAGE_EVIDENCE_INCOMPLETE" >&2; exit 1; } + [[ "$(shasum -a 256 "$stage_path" | awk '{print $1}')" == "$stage_sha" ]] || { echo "P6_ERROR:DRIVER_STAGE_HASH_MISMATCH" >&2; exit 1; } + jq -e --arg action "$action" --arg run "$run_id" --arg input_sha "$input_hash" ' + .run_id == $run and .input_sha256 == $input_sha and .result == "passed" and .content_redacted == true + and .schema_version == (if $action == "provision" then "p6-driver-provision/v1" elif $action == "golden" then "p6-driver-report/v1" else "p6-driver-cleanup/v1" end) + ' "$stage_path" >/dev/null || { echo "P6_ERROR:DRIVER_STAGE_EVIDENCE_REJECTED" >&2; exit 1; } +done output="$artifacts/p6-final-${run_id}.json" tmp="$(mktemp "$artifacts/.p6-final.XXXXXX")" -jq -n --arg run_id "$run_id" --arg input_sha256 "$input_hash" --arg tested_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" --argjson metadata "$metadata" \ - '{schema_version:"p6-final-report/v1",run_id:$run_id,input_sha256:$input_sha256,tested_at:$tested_at,result:"passed",phase:"completed",content_redacted:true} + $metadata' > "$tmp" +jq -n --arg run_id "$run_id" --arg input_sha256 "$input_hash" --arg tested_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" --argjson metadata "$metadata" --argjson driver_stage_reports "$stage_reports" \ + '{schema_version:"p6-final-report/v1",run_id:$run_id,input_sha256:$input_sha256,tested_at:$tested_at,result:"passed",phase:"completed",content_redacted:true,driver_stage_reports:$driver_stage_reports} + $metadata' > "$tmp" chmod 600 "$tmp" mv -f "$tmp" "$output" printf '%s\n' "$output" diff --git a/docker_openclaw/p6/scripts/p6-full-driver.sh b/docker_openclaw/p6/scripts/p6-full-driver.sh new file mode 100755 index 0000000..fd58fba --- /dev/null +++ b/docker_openclaw/p6/scripts/p6-full-driver.sh @@ -0,0 +1,138 @@ +#!/usr/bin/env bash +# Checked-in P6 full-topology driver. It owns one run-scoped Compose project, +# invokes the real product chain, retains only redacted stage evidence, and +# removes its exact containers/network/volumes on cleanup. +set -euo pipefail + +script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +p6_dir="$(cd "${script_dir}/.." && pwd)" +source "${script_dir}/p6-lib.sh" + +: "${P6_DRIVER_ACTION:?P6_DRIVER_ACTION is required}" +: "${P6_RUN_ID:?P6_RUN_ID is required}" +: "${P6_INPUT_FILE:?P6_INPUT_FILE is required}" +: "${P6_DRIVER_REPORT:?P6_DRIVER_REPORT is required}" +: "${P6_SECRET_PATTERN_FILE:?P6_SECRET_PATTERN_FILE is required}" +: "${P6_WORK_DIR:?P6_WORK_DIR is required}" +: "${P6_ARTIFACTS_DIR:?P6_ARTIFACTS_DIR is required}" + +[[ "$P6_DRIVER_ACTION" =~ ^(provision|golden|cleanup)$ ]] || p6_die "DRIVER_ACTION_INVALID" 79 +[[ "$P6_RUN_ID" =~ ^p6-[a-f0-9]{32}$ ]] || p6_die "RUN_ID_INVALID" 79 +[[ -f "$P6_INPUT_FILE" && ! -L "$P6_INPUT_FILE" ]] || p6_die "INPUT_FILE_REQUIRED" 79 +[[ "$P6_DRIVER_REPORT" == "$P6_ARTIFACTS_DIR"/* && ! -L "$P6_DRIVER_REPORT" ]] || p6_die "DRIVER_REPORT_PATH_INVALID" 79 + +input_sha256="$(p6_sha256 "$P6_INPUT_FILE")" +state_file="${P6_WORK_DIR}/driver-state.json" +runtime_env="${P6_WORK_DIR}/runtime.env" +product_config="${P6_WORK_DIR}/config/driver-config.json" +product_report="${P6_WORK_DIR}/product-chain.json" +short="${P6_RUN_ID#p6-}" +short="${short:0:12}" +project="p6-runtime-${short}" +compose_file="${p6_dir}/docker-compose.runtime.yml" +mkdir -p "$P6_WORK_DIR" "$P6_ARTIFACTS_DIR" +chmod 700 "$P6_WORK_DIR" "$P6_ARTIFACTS_DIR" + +write_report() { + local schema="$1" result="$2" payload="$3" tmp + tmp="$(mktemp "${P6_ARTIFACTS_DIR}/.p6-driver.XXXXXX")" + jq -n \ + --arg schema "$schema" \ + --arg result "$result" \ + --arg run_id "$P6_RUN_ID" \ + --arg input_sha256 "$input_sha256" \ + --arg tested_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ + --argjson payload "$payload" \ + '{schema_version:$schema,result:$result,run_id:$run_id,input_sha256:$input_sha256,tested_at:$tested_at,content_redacted:true} + $payload' > "$tmp" + chmod 600 "$tmp" + mv -f "$tmp" "$P6_DRIVER_REPORT" + chmod 600 "$P6_DRIVER_REPORT" +} + +compose() { + docker compose --project-name "$project" --env-file "$runtime_env" -f "$compose_file" "$@" +} + +prepare_runtime() { + P6_ARTIFACTS_DIR="$P6_ARTIFACTS_DIR" \ + "${script_dir}/p6-prepare-runtime.py" + p6_require_regular_0600 "$runtime_env" + p6_require_regular_0600 "$product_config" +} + +container_absent() { + ! docker container inspect "$1" >/dev/null 2>&1 +} + +cleanup_resources() { + local launcher_container="" workspace_container="" + if [[ -f "$product_config" && ! -L "$product_config" ]]; then + launcher_container="$(jq -r '.launcher_container // empty' "$product_config" 2>/dev/null || true)" + workspace_container="$(jq -r '.workspace_container // empty' "$product_config" 2>/dev/null || true)" + fi + if [[ -n "$workspace_container" ]]; then + docker rm -f "$workspace_container" >/dev/null 2>&1 || true + fi + if [[ -f "$runtime_env" && ! -L "$runtime_env" ]]; then + compose down --volumes --remove-orphans >/dev/null 2>&1 || true + fi + if [[ -n "$launcher_container" ]]; then + container_absent "$launcher_container" || return 1 + fi + if [[ -n "$workspace_container" ]]; then + container_absent "$workspace_container" || return 1 + fi + [[ -z "$(docker container ls -aq --filter "label=com.docker.compose.project=${project}")" ]] || return 1 + [[ -z "$(docker volume ls -q --filter "label=com.docker.compose.project=${project}")" ]] || return 1 + ! docker network inspect "p6net-${short}" >/dev/null 2>&1 || return 1 +} + +case "$P6_DRIVER_ACTION" in + provision) + prepare_runtime + if ! compose up -d --wait >/dev/null; then + write_report "p6-driver-provision/v1" "failed" "$(jq -n --arg project "$project" '{project:$project,error_code:"TOPOLOGY_START_FAILED"}')" + exit 1 + fi + jq -n --arg project "$project" '{project:$project,topology:{litellm:"started",shell:"started",jupyterhub:"started",launcher:"started",workspace:"deferred_to_golden"}}' > "$state_file" + chmod 600 "$state_file" + write_report "p6-driver-provision/v1" "passed" "$(jq -n --arg project "$project" '{project:$project,topology:{litellm:"started",shell:"started",jupyterhub:"started",launcher:"started",workspace:"deferred_to_golden"},isolation:{network:"run_scoped",volumes:"run_scoped"}}')" + ;; + golden) + [[ -f "$state_file" && ! -L "$state_file" ]] || p6_die "TOPOLOGY_STATE_REQUIRED" 79 + p6_require_regular_0600 "$product_config" + rm -f "$product_report" "$P6_SECRET_PATTERN_FILE" + P6_PRODUCT_CONFIG_FILE="$product_config" \ + P6_PRODUCT_REPORT_FILE="$product_report" \ + P6_SECRET_PATTERN_FILE="$P6_SECRET_PATTERN_FILE" \ + "${script_dir}/p6-product-chain.py" + p6_require_regular_0600 "$product_report" + p6_require_regular_0600 "$P6_SECRET_PATTERN_FILE" + jq -e ' + .schema_version == "p6-product-chain-report/v1" + and .result == "passed" + and .content_redacted == true + and .checks.console_ui == "reused_verified_evidence" + and ([.checks.test_resource_provision,.checks.binding_payload,.checks.jupyterhub_dockerspawner,.checks.launcher_claim_activate_release,.checks.openclaw_apply_probe_readiness,.checks.chat,.checks.stream,.checks.tool,.checks.usage,.checks.owner_negative,.checks.prompt_response_absent,.checks.revoke,.checks.generation_restart,.checks.late_release,.checks.delete,.checks.zero_active_leases] | all(. == "passed")) + and (.scan_roots | type == "array" and length >= 1 and all(.[]; type == "string" and startswith("/"))) + ' "$product_report" >/dev/null || p6_die "PRODUCT_CHAIN_REPORT_INVALID" 79 + checks="$(jq -c '.checks' "$product_report")" + scan_roots="$(jq -c --arg product "$product_report" '.scan_roots + [$product] | unique' "$product_report")" + product_summary="$(jq -c '{binding,runtime,data_plane,usage,lifecycle}' "$product_report")" + write_report "p6-driver-report/v1" "passed" "$(jq -n \ + --arg patterns "$P6_SECRET_PATTERN_FILE" \ + --arg product_sha "$(p6_sha256 "$product_report")" \ + --argjson checks "$checks" \ + --argjson scan_roots "$scan_roots" \ + --argjson product_summary "$product_summary" \ + '{checks:$checks,secret_pattern_file:$patterns,scan_roots:$scan_roots,product_report_sha256:$product_sha,product:$product_summary}')" + ;; + cleanup) + cleanup_resources || p6_die "TOPOLOGY_RESOURCE_REMAINS" 79 + rm -f "$state_file" "$P6_SECRET_PATTERN_FILE" "$product_report" + rm -rf "${P6_WORK_DIR}/config" "${P6_WORK_DIR}/secrets" "${P6_WORK_DIR}/surfaces" "${P6_WORK_DIR}/workspace" "${P6_WORK_DIR}/launcher-data" + rm -f "$runtime_env" + [[ ! -e "$state_file" && ! -e "$runtime_env" && ! -e "${P6_WORK_DIR}/config" && ! -e "${P6_WORK_DIR}/secrets" && ! -e "${P6_WORK_DIR}/surfaces" && ! -e "${P6_WORK_DIR}/workspace" ]] || p6_die "TOPOLOGY_TEMPORARY_MATERIAL_REMAINS" 79 + write_report "p6-driver-cleanup/v1" "passed" "$(jq -n '{resources:{litellm:"absent",shell:"absent",jupyterhub:"absent",launcher:"absent",workspace:"absent",runtime_material:"absent",temporary_files:"absent",processes:"absent",network:"absent",volumes:"absent"}}')" + ;; +esac diff --git a/docker_openclaw/p6/scripts/p6-lib.sh b/docker_openclaw/p6/scripts/p6-lib.sh old mode 100755 new mode 100644 index a06d08c..89b057b --- a/docker_openclaw/p6/scripts/p6-lib.sh +++ b/docker_openclaw/p6/scripts/p6-lib.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # Shared fail-closed helpers for the P6 local-only runner. Never source an -# environment file and never print a credential or a credential fingerprint. +# environment file and never print a credential or credential fingerprint. set -euo pipefail p6_die() { @@ -12,6 +12,10 @@ p6_run_id() { python3 -c 'import secrets; print("p6-" + secrets.token_hex(16))' } +p6_sha256() { + shasum -a 256 "$1" | awk '{print $1}' +} + p6_require_regular_0600() { local path="$1" mode [[ -f "$path" && ! -L "$path" ]] || { p6_die "SECURE_FILE_REQUIRED" 64; return $?; } @@ -21,139 +25,240 @@ p6_require_regular_0600() { [[ "$mode" == 400 || "$mode" == 600 ]] || { p6_die "SECURE_FILE_MODE_REQUIRED" 65; return $?; } } +p6_repository_metadata() { + jq -c ' + .repositories | with_entries( + .value = if .value.delivery_identity == "commit" then + {delivery_identity:"commit",commit:.value.commit} + else + {delivery_identity:"review_snapshot",branch:.value.branch,phase_base_commit:.value.phase_base_commit,head_commit:.value.head_commit,tracked_diff_sha256:.value.tracked_diff_sha256,changed_files:.value.changed_files} + end + ) + ' "$P6_INPUT_FILE" +} + p6_write_report() { - local report="$1" result="$2" phase="$3" reason="${4:-}" - local temp metadata + local report="$1" result="$2" phase="$3" reason="${4:-}" extra="${5:-}" temp metadata + [[ -n "$extra" ]] || extra='{}' mkdir -p "$(dirname "$report")" chmod 700 "$(dirname "$report")" temp="$(mktemp "$(dirname "$report")/.p6-report.XXXXXX")" - metadata="$(jq -c '{contract_version,contract_bundle_sha256,control_commit,review_policy_commit,repositories:(.repositories | with_entries(.value = .value.commit)),images}' "$P6_INPUT_FILE")" + metadata="$(jq -c --argjson repositories "$(p6_repository_metadata)" '{contract_version,contract_bundle_sha256,control_commit,review_policy_commit,repositories:$repositories,images,local_only_images,support_images}' "$P6_INPUT_FILE")" jq -n \ --arg run_id "$P6_RUN_ID" \ --arg tested_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ --arg result "$result" \ --arg phase "$phase" \ --arg reason "$reason" \ - --arg input_sha256 "$(shasum -a 256 "$P6_INPUT_FILE" | awk '{print $1}')" \ + --arg input_sha256 "$(p6_sha256 "$P6_INPUT_FILE")" \ --argjson metadata "$metadata" \ + --argjson extra "$extra" \ '{schema_version:"p6-report/v1",run_id:$run_id,tested_at:$tested_at,result:$result,phase:$phase,input_sha256:$input_sha256,content_redacted:true} + $metadata + + $extra + (if $reason == "" then {} else {reason:$reason} end)' > "$temp" chmod 600 "$temp" mv -f "$temp" "$report" } +p6_stage_reports_json() { + local artifact_dir="$1" run_id="$2" action path hash + for action in provision golden cleanup; do + path="${artifact_dir}/p6-driver-${action}-${run_id}.json" + [[ -f "$path" && ! -L "$path" ]] || { p6_die "DRIVER_STAGE_REPORT_MISSING" 78; return $?; } + hash="$(p6_sha256 "$path")" + [[ "$hash" =~ ^[0-9a-f]{64}$ ]] || { p6_die "DRIVER_STAGE_REPORT_HASH_INVALID" 78; return $?; } + done + jq -n \ + --arg provision "${artifact_dir}/p6-driver-provision-${run_id}.json" \ + --arg provision_sha "$(p6_sha256 "${artifact_dir}/p6-driver-provision-${run_id}.json")" \ + --arg golden "${artifact_dir}/p6-driver-golden-${run_id}.json" \ + --arg golden_sha "$(p6_sha256 "${artifact_dir}/p6-driver-golden-${run_id}.json")" \ + --arg cleanup "${artifact_dir}/p6-driver-cleanup-${run_id}.json" \ + --arg cleanup_sha "$(p6_sha256 "${artifact_dir}/p6-driver-cleanup-${run_id}.json")" \ + '{driver_stage_reports:{provision:{path:$provision,sha256:$provision_sha},golden:{path:$golden,sha256:$golden_sha},cleanup:{path:$cleanup,sha256:$cleanup_sha}}}' +} + p6_json_string() { jq -er "$1" "$P6_INPUT_FILE" } p6_assert_fixed_commit() { - local value="$1" - [[ "$value" =~ ^[0-9a-f]{40}$ ]] || { p6_die "FIXED_COMMIT_REQUIRED" 66; return $?; } + [[ "$1" =~ ^[0-9a-f]{40}$ ]] || { p6_die "FIXED_COMMIT_REQUIRED" 66; return $?; } } -p6_assert_fixed_image() { - local name="$1" ref image_id digest provenance source_commit source_repository base_image_digest - ref="$(p6_json_string ".images.${name}.ref")" - image_id="$(p6_json_string ".images.${name}.image_id")" - digest="$(jq -r ".images.${name}.repo_digest // empty" "$P6_INPUT_FILE")" - provenance="$(p6_json_string ".images.${name}.provenance")" - [[ "$ref" == quay.io/labnow/* && "$ref" != *:latest ]] || { p6_die "FIXED_IMAGE_REF_REQUIRED" 67; return $?; } - [[ "$image_id" =~ ^sha256:[0-9a-f]{64}$ ]] || { p6_die "FIXED_IMAGE_ID_REQUIRED" 67; return $?; } - case "$provenance" in - repo_digest) - [[ "$digest" =~ ^quay\.io/labnow/(litellm|openclaw)@sha256:[0-9a-f]{64}$ ]] || { p6_die "FIXED_IMAGE_DIGEST_REQUIRED" 67; return $?; } - [[ "$ref" == "$digest" || "$ref" != *'@sha256:'* ]] || { p6_die "IMAGE_REF_DIGEST_MISMATCH" 67; return $?; } - ;; - local_build) - [[ "$digest" == absent ]] || { p6_die "LOCAL_PROVENANCE_MUST_DECLARE_ABSENT_DIGEST" 67; return $?; } - source_repository="$(p6_json_string ".images.${name}.source_repository")" - source_commit="$(p6_json_string ".images.${name}.source_commit")" - base_image_digest="$(p6_json_string ".images.${name}.base_image_digest")" - [[ "$source_repository" == labnow_open ]] || { p6_die "LOCAL_PROVENANCE_SOURCE_REPOSITORY_INVALID" 67; return $?; } - p6_assert_fixed_commit "$source_commit" || return $? - [[ "$base_image_digest" =~ ^quay\.io/labnow/openclaw@sha256:[0-9a-f]{64}$ ]] || { p6_die "LOCAL_BASE_IMAGE_DIGEST_REQUIRED" 67; return $?; } - ;; - *) p6_die "IMAGE_PROVENANCE_REQUIRED" 67; return $? ;; - esac +p6_assert_sha256() { + [[ "$1" =~ ^[0-9a-f]{64}$ ]] || { p6_die "FIXED_SHA256_REQUIRED" 66; return $?; } } p6_validate_input_shape() { jq -e ' - type == "object" - and (.schema_version == "p6-inputs/v1") + . as $root + | type == "object" + and (.schema_version == "p6-inputs/v2") and (.contract_version == "v1alpha1") and (.contract_bundle_sha256 == "d289dff9bcaa3d28035c5ed2e56b806f4b3b37fdca3159352d22f0c03942e202") and (.control_commit == "2eb71d7590739df3de8db2f8cf9098154a397f0b") - and (.review_policy_commit == .control_commit) + and (.review_policy_commit == "680ca92661a08254eb396ab809f478bbdba3510e") and (.repositories | keys | sort) == ["lab_dev","labnow_launcher","labnow_open","labnow_shell"] + and (.repositories.lab_dev | keys | sort) == ["branch","changed_files","delivery_identity","head_commit","path","phase_base_commit","tracked_diff_sha256"] + and (.repositories.lab_dev.delivery_identity == "review_snapshot") + and (.repositories.lab_dev.branch == "dev/che-563-openclaw-product-closure") + and (.repositories.lab_dev.phase_base_commit == "940325578bae9905673965d6dc489130ab4b6a46") + and (.repositories.lab_dev.head_commit == "1b4562899e03eacdee5a86eb55b47d5e12117ee8") + and (.repositories.lab_dev.tracked_diff_sha256 | type == "string" and test("^[0-9a-f]{64}$")) + and (.repositories.lab_dev.changed_files | type == "array" and length > 0 and unique == .) + and (["labnow_open","labnow_shell","labnow_launcher"] | map(. as $name | + (($root.repositories[$name] | keys | sort) == ["commit","delivery_identity","path"] + and $root.repositories[$name].delivery_identity == "commit" + and ($root.repositories[$name].commit | type == "string" and test("^[0-9a-f]{40}$")))) | all) + and (.repositories.labnow_open.commit == "21019e0c24dc7b51747c2bef3cd90f5d259be839") + and (.repositories.labnow_shell.commit == "5c9411dfd3c4d7b1e606c0d9dc0c5e62313bc376") + and (.repositories.labnow_launcher.commit == "c84edea3e051d561f28d9f99235563cf491aaeb2") + and (.repositories | all(.[]; (.path | type == "string" and startswith("/")))) and (.images | keys | sort) == ["litellm","openclaw_base","openclaw_workspace"] - and (.paths | keys | sort) == ["runtime_mount","workspace_root"] - and (.driver | type == "string" and startswith("/")) + and (.local_only_images | keys | sort) == ["launcher","shell"] + and (.support_images | keys | sort) == ["nginx","postgres","redis"] + and (.runtime | keys | sort) == ["p1_env_file"] + and (.runtime.p1_env_file | type == "string" and startswith("/")) ' "$P6_INPUT_FILE" >/dev/null || { p6_die "INPUT_SCHEMA_INVALID" 68; return $?; } + local repo - for repo in lab_dev labnow_open labnow_shell labnow_launcher; do + for repo in labnow_open labnow_shell labnow_launcher; do p6_assert_fixed_commit "$(p6_json_string ".repositories.${repo}.commit")" || return $? done - p6_assert_fixed_image litellm || return $? - p6_assert_fixed_image openclaw_base || return $? - p6_assert_fixed_image openclaw_workspace || return $? + p6_assert_fixed_commit "$(p6_json_string '.repositories.lab_dev.phase_base_commit')" || return $? + p6_assert_fixed_commit "$(p6_json_string '.repositories.lab_dev.head_commit')" || return $? + p6_assert_sha256 "$(p6_json_string '.repositories.lab_dev.tracked_diff_sha256')" || return $? + p6_assert_image_shape litellm images repo_digest 'quay.io/labnow/litellm' || return $? + p6_assert_image_shape openclaw_base images repo_digest 'quay.io/labnow/openclaw' || return $? + p6_assert_image_shape openclaw_workspace images local_build 'quay.io/labnow/labnow-open' || return $? + p6_assert_image_shape launcher local_only_images local_build 'quay.io/labnow/labnow-launcher' || return $? + p6_assert_image_shape shell local_only_images local_build 'quay.io/labnow/labnow-shell' || return $? + p6_assert_image_shape postgres support_images repo_digest 'postgres' || return $? + p6_assert_image_shape redis support_images repo_digest 'redis' || return $? + p6_assert_image_shape nginx support_images repo_digest 'nginx' || return $? +} + +p6_assert_image_shape() { + local name="$1" section="$2" provenance="$3" repository="$4" ref image_id digest actual_provenance + ref="$(p6_json_string ".${section}.${name}.ref")" + image_id="$(p6_json_string ".${section}.${name}.image_id")" + digest="$(p6_json_string ".${section}.${name}.repo_digest")" + actual_provenance="$(p6_json_string ".${section}.${name}.provenance")" + [[ "$actual_provenance" == "$provenance" ]] || { p6_die "IMAGE_PROVENANCE_INVALID" 67; return $?; } + [[ "$ref" != *:latest && "$ref" != latest ]] || { p6_die "FIXED_IMAGE_REF_REQUIRED" 67; return $?; } + [[ "$image_id" =~ ^sha256:[0-9a-f]{64}$ ]] || { p6_die "FIXED_IMAGE_ID_REQUIRED" 67; return $?; } + [[ "$digest" == "${repository}@sha256:"* && "$digest" =~ @sha256:[0-9a-f]{64}$ ]] || { p6_die "FIXED_IMAGE_DIGEST_REQUIRED" 67; return $?; } + if [[ "$provenance" == local_build ]]; then + p6_assert_fixed_commit "$(p6_json_string ".${section}.${name}.source_commit")" || return $? + [[ "$(p6_json_string ".${section}.${name}.source_repository")" =~ ^labnow_(open|launcher|shell)$ ]] || { p6_die "LOCAL_PROVENANCE_SOURCE_INVALID" 67; return $?; } + [[ "$(p6_json_string ".${section}.${name}.base_image_digest")" =~ ^[^[:space:]]+@sha256:[0-9a-f]{64}$ ]] || { p6_die "LOCAL_BASE_IMAGE_DIGEST_REQUIRED" 67; return $?; } + fi } p6_assert_repository() { - local name="$1" path expected actual status + local name="$1" path identity actual status expected base expected_diff actual_diff expected_files actual_files branch path="$(p6_json_string ".repositories.${name}.path")" - expected="$(p6_json_string ".repositories.${name}.commit")" + identity="$(p6_json_string ".repositories.${name}.delivery_identity")" [[ -d "$path/.git" ]] || { p6_die "REPOSITORY_UNAVAILABLE" 69; return $?; } actual="$(git -C "$path" rev-parse HEAD)" - [[ "$actual" == "$expected" ]] || { p6_die "REPOSITORY_COMMIT_MISMATCH" 70; return $?; } - status="$(git -C "$path" status --porcelain=v1 --untracked-files=no)" - [[ -z "$status" ]] || { p6_die "REPOSITORY_TRACKED_TREE_DIRTY" 71; return $?; } + if [[ "$identity" == commit ]]; then + expected="$(p6_json_string ".repositories.${name}.commit")" + [[ "$actual" == "$expected" ]] || { p6_die "REPOSITORY_COMMIT_MISMATCH" 70; return $?; } + status="$(git -C "$path" status --porcelain=v1 --untracked-files=no)" + [[ -z "$status" ]] || { p6_die "REPOSITORY_TRACKED_TREE_DIRTY" 71; return $?; } + return 0 + fi + + expected="$(p6_json_string ".repositories.${name}.head_commit")" + base="$(p6_json_string ".repositories.${name}.phase_base_commit")" + branch="$(p6_json_string ".repositories.${name}.branch")" + [[ "$actual" == "$expected" ]] || { p6_die "REVIEW_SNAPSHOT_HEAD_MISMATCH" 70; return $?; } + [[ "$(git -C "$path" branch --show-current)" == "$branch" ]] || { p6_die "REVIEW_SNAPSHOT_BRANCH_MISMATCH" 70; return $?; } + git -C "$path" merge-base --is-ancestor "$base" HEAD || { p6_die "REVIEW_SNAPSHOT_BASE_NOT_ANCESTOR" 70; return $?; } + actual_diff="$(git -C "$path" diff --binary --full-index --no-ext-diff "$base" -- | shasum -a 256 | awk '{print $1}')" + expected_diff="$(p6_json_string ".repositories.${name}.tracked_diff_sha256")" + [[ "$actual_diff" == "$expected_diff" ]] || { p6_die "REVIEW_SNAPSHOT_DIFF_MISMATCH" 71; return $?; } + actual_files="$(git -C "$path" diff --name-only "$base" -- | LC_ALL=C sort | jq -Rsc 'split("\n") | map(select(length > 0))')" + expected_files="$(jq -c ".repositories.${name}.changed_files | sort" "$P6_INPUT_FILE")" + [[ "$actual_files" == "$expected_files" ]] || { p6_die "REVIEW_SNAPSHOT_FILESET_MISMATCH" 71; return $?; } } -p6_assert_images_present() { - local name ref image_id actual_id digests expected_digest provenance source_commit source_repository base_image_digest source_repo_path source_repo_commit - for name in litellm openclaw_base openclaw_workspace; do - ref="$(p6_json_string ".images.${name}.ref")" - image_id="$(p6_json_string ".images.${name}.image_id")" - expected_digest="$(jq -r ".images.${name}.repo_digest // empty" "$P6_INPUT_FILE")" - provenance="$(p6_json_string ".images.${name}.provenance")" - actual_id="$(docker image inspect --format '{{.Id}}' "$ref" 2>/dev/null)" || { p6_die "LOCAL_IMAGE_UNAVAILABLE" 72; return $?; } - [[ "$actual_id" == "$image_id" ]] || { p6_die "LOCAL_IMAGE_ID_MISMATCH" 72; return $?; } - if [[ "$provenance" == repo_digest ]]; then - digests="$(docker image inspect --format '{{join .RepoDigests "\n"}}' "$ref")" - grep -Fqx "$expected_digest" <<<"$digests" || { p6_die "LOCAL_IMAGE_DIGEST_MISMATCH" 72; return $?; } +p6_assert_image_present() { + local section="$1" name="$2" ref expected_id expected_digest actual_id digests source source_commit repository_commit base_digest + ref="$(p6_json_string ".${section}.${name}.ref")" + expected_id="$(p6_json_string ".${section}.${name}.image_id")" + expected_digest="$(p6_json_string ".${section}.${name}.repo_digest")" + actual_id="$(docker image inspect --format '{{.Id}}' "$ref" 2>/dev/null)" || { p6_die "LOCAL_IMAGE_UNAVAILABLE" 72; return $?; } + [[ "$actual_id" == "$expected_id" ]] || { p6_die "LOCAL_IMAGE_ID_MISMATCH" 72; return $?; } + digests="$(docker image inspect --format '{{join .RepoDigests "\n"}}' "$ref")" + grep -Fqx "$expected_digest" <<<"$digests" || { p6_die "LOCAL_IMAGE_DIGEST_MISMATCH" 72; return $?; } + if [[ "$(p6_json_string ".${section}.${name}.provenance")" == local_build ]]; then + source="$(p6_json_string ".${section}.${name}.source_repository")" + source_commit="$(p6_json_string ".${section}.${name}.source_commit")" + if [[ "$source" == lab_dev ]]; then + repository_commit="$(p6_json_string '.repositories.lab_dev.head_commit')" else - source_commit="$(p6_json_string ".images.${name}.source_commit")" - source_repository="$(p6_json_string ".images.${name}.source_repository")" - base_image_digest="$(p6_json_string ".images.${name}.base_image_digest")" - source_repo_path="$(p6_json_string ".repositories.${source_repository}.path")" - source_repo_commit="$(p6_json_string ".repositories.${source_repository}.commit")" - [[ "$source_commit" == "$source_repo_commit" ]] || { p6_die "LOCAL_PROVENANCE_SOURCE_MISMATCH" 72; return $?; } - [[ -d "$source_repo_path/.git" ]] || { p6_die "LOCAL_PROVENANCE_SOURCE_UNAVAILABLE" 72; return $?; } - [[ "$(p6_json_string '.images.openclaw_base.repo_digest')" == "$base_image_digest" ]] || { p6_die "LOCAL_BASE_IMAGE_MISMATCH" 72; return $?; } + repository_commit="$(p6_json_string ".repositories.${source}.commit")" fi + [[ "$source_commit" == "$repository_commit" ]] || { p6_die "LOCAL_PROVENANCE_SOURCE_MISMATCH" 72; return $?; } + base_digest="$(p6_json_string ".${section}.${name}.base_image_digest")" + case "$name" in + openclaw_workspace) [[ "$base_digest" == "$(p6_json_string '.images.openclaw_base.repo_digest')" ]] || { p6_die "LOCAL_BASE_IMAGE_MISMATCH" 72; return $?; } ;; + esac + fi +} + +p6_assert_images_present() { + local pair section name + for pair in \ + images:litellm images:openclaw_base images:openclaw_workspace \ + local_only_images:launcher local_only_images:shell \ + support_images:postgres support_images:redis support_images:nginx; do + section="${pair%%:*}" + name="${pair##*:}" + p6_assert_image_present "$section" "$name" || return $? done } -p6_assert_runtime_paths() { - local mount workspace - mount="$(p6_json_string '.paths.runtime_mount')" - workspace="$(p6_json_string '.paths.workspace_root')" - [[ -d "$mount" && ! -L "$mount" && -d "$workspace" && ! -L "$workspace" ]] || { p6_die "RUNTIME_PATH_UNAVAILABLE" 73; return $?; } +p6_assert_runtime_input() { + local env_file + env_file="$(p6_json_string '.runtime.p1_env_file')" + p6_require_regular_0600 "$env_file" || return $? + python3 - "$env_file" <<'PY' +import sys +from pathlib import Path + +required = { + "LITELLM_MASTER_KEY", "POSTGRES_DB", "POSTGRES_USER", "POSTGRES_PASSWORD", + "REDIS_PASSWORD", "UPSTREAM_API_KEY", "UPSTREAM_BASE_URL", "UPSTREAM_MODEL", +} +values = {} +for raw in Path(sys.argv[1]).read_text(encoding="utf-8").splitlines(): + line = raw.strip() + if line and not line.startswith("#") and "=" in line: + key, value = line.split("=", 1) + values[key] = value +if any(not values.get(key) for key in required): + raise SystemExit(1) +PY + [[ $? == 0 ]] || { p6_die "P1_ENV_INCOMPLETE" 73; return $?; } } p6_security_scan() { - # Write secret strings only to a mode-0600 pattern file and pass its path to - # rg. Neither the shell command line nor the report contains the secret. - local patterns="$1" + local patterns="$1" status shift p6_require_regular_0600 "$patterns" || return $? [[ -s "$patterns" ]] || { p6_die "SECRET_PATTERN_FILE_REQUIRED" 74; return $?; } (($# > 0)) || { p6_die "SECRET_SCAN_ROOT_REQUIRED" 74; return $?; } - if rg --fixed-strings --files-with-matches --glob '!secret-patterns' --glob '!secret.json' -f "$patterns" "$@" >/dev/null 2>&1; then - rm -f "$patterns" - p6_die "SECRET_FINGERPRINT_MATCH" 75 - return $? - fi + set +e + rg --fixed-strings --files-with-matches --glob '!secret-patterns' -f "$patterns" "$@" >/dev/null 2>&1 + status=$? + set -e + case "$status" in + 0) p6_die "SECRET_FINGERPRINT_MATCH" 75; return $? ;; + 1) return 0 ;; + *) p6_die "SECRET_SCAN_FAILED" 75; return $? ;; + esac } diff --git a/docker_openclaw/p6/scripts/p6-prepare-runtime.py b/docker_openclaw/p6/scripts/p6-prepare-runtime.py new file mode 100755 index 0000000..e667c21 --- /dev/null +++ b/docker_openclaw/p6/scripts/p6-prepare-runtime.py @@ -0,0 +1,424 @@ +#!/usr/bin/env python3 +"""Prepare one restricted, run-scoped P6 topology without printing secrets.""" + +from __future__ import annotations + +import base64 +import json +import os +import secrets +import stat +import subprocess +import sys +from pathlib import Path +from urllib.parse import quote, urlsplit + + +class PrepareError(RuntimeError): + pass + + +def restricted(path: Path, *, code: str) -> None: + try: + info = path.stat() + except OSError as exc: + raise PrepareError(code) from exc + if not stat.S_ISREG(info.st_mode) or info.st_mode & 0o077: + raise PrepareError(code) + + +def write_private(path: Path, value: str | bytes, *, mode: int = 0o400) -> None: + path.parent.mkdir(mode=0o700, parents=True, exist_ok=True) + temporary = path.with_name(f".{path.name}.{secrets.token_hex(8)}") + flags = os.O_WRONLY | os.O_CREAT | os.O_EXCL + descriptor = os.open(temporary, flags, 0o600) + try: + payload = value.encode("utf-8") if isinstance(value, str) else value + with os.fdopen(descriptor, "wb") as handle: + handle.write(payload) + handle.flush() + os.fsync(handle.fileno()) + os.chmod(temporary, mode) + os.replace(temporary, path) + except Exception: + try: + os.unlink(temporary) + except OSError: + pass + raise + + +def load_json(path: Path, *, code: str) -> dict: + restricted(path, code=code) + try: + value = json.loads(path.read_text(encoding="utf-8")) + except (OSError, UnicodeDecodeError, json.JSONDecodeError) as exc: + raise PrepareError(code) from exc + if not isinstance(value, dict): + raise PrepareError(code) + return value + + +def load_env(path: Path) -> dict[str, str]: + restricted(path, code="P1_ENV_INVALID") + values: dict[str, str] = {} + try: + lines = path.read_text(encoding="utf-8").splitlines() + except (OSError, UnicodeDecodeError) as exc: + raise PrepareError("P1_ENV_INVALID") from exc + for raw in lines: + line = raw.strip() + if line and not line.startswith("#") and "=" in line: + key, value = line.split("=", 1) + values[key] = value + required = { + "UPSTREAM_API_KEY", + "UPSTREAM_BASE_URL", + "UPSTREAM_MODEL", + } + if any(not values.get(key) for key in required): + raise PrepareError("P1_ENV_INCOMPLETE") + upstream = urlsplit(values["UPSTREAM_BASE_URL"]) + if upstream.scheme != "https" or not upstream.netloc or upstream.path not in {"", "/"}: + raise PrepareError("P1_UPSTREAM_URL_INVALID") + return values + + +def token(prefix: str) -> str: + return f"{prefix}{secrets.token_urlsafe(32)}" + + +def main() -> int: + input_path = Path(os.environ.get("P6_INPUT_FILE", "")) + work_dir = Path(os.environ.get("P6_WORK_DIR", "")) + run_id = os.environ.get("P6_RUN_ID", "") + artifact_dir = Path(os.environ.get("P6_ARTIFACTS_DIR", "")) + script_dir = Path(__file__).resolve().parent + p6_dir = script_dir.parent + if not input_path.is_absolute() or not work_dir.is_absolute() or not artifact_dir.is_absolute(): + raise PrepareError("P6_PREPARE_PATH_INVALID") + if not run_id.startswith("p6-") or len(run_id) != 35: + raise PrepareError("P6_PREPARE_RUN_ID_INVALID") + + inputs = load_json(input_path, code="P6_INPUT_INVALID") + repositories = inputs.get("repositories", {}) + try: + lab_dev = Path(repositories["lab_dev"]["path"]) + launcher_repo = Path(repositories["labnow_launcher"]["path"]) + shell_repo = Path(repositories["labnow_shell"]["path"]) + p1_env = Path(inputs["runtime"]["p1_env_file"]) + except (KeyError, TypeError) as exc: + raise PrepareError("P6_INPUT_INVALID") from exc + values = load_env(p1_env) + + for directory in (work_dir, artifact_dir): + directory.mkdir(mode=0o700, parents=True, exist_ok=True) + os.chmod(directory, 0o700) + secrets_dir = work_dir / "secrets" + config_dir = work_dir / "config" + workspace_root = work_dir / "workspace" + surfaces_dir = work_dir / "surfaces" + for directory in (secrets_dir, config_dir, workspace_root, surfaces_dir): + directory.mkdir(mode=0o700, parents=True, exist_ok=True) + os.chmod(directory, 0o700) + + short = run_id.removeprefix("p6-")[:12] + network = f"p6net-{short}" + project = f"p6-runtime-{short}" + launcher_container = f"p6-launcher-{short}" + shell_container = f"p6-shell-{short}" + litellm_container = f"p6-litellm-{short}" + shell_postgres_container = f"p6-shell-pg-{short}" + litellm_postgres_container = f"p6-litellm-pg-{short}" + user = f"p6user-{short[:8]}" + server = f"p6ws-{short[:8]}" + prefix = f"p6w-{short[:8]}" + workspace_container = f"{prefix}-{user}-{server}" + + generated = { + "litellm_master": token("sk-p6-master-"), + "litellm_postgres": token("p6-litellm-db-"), + "redis": token("p6-redis-"), + "shell_postgres": token("p6-shell-db-"), + "hub": token("p6-hub-"), + "launcher": token("p6-launcher-"), + "kek": base64.b64encode(secrets.token_bytes(32)).decode("ascii"), + "oauth_cookie": base64.b64encode(secrets.token_bytes(32)).decode("ascii"), + "upstream": values["UPSTREAM_API_KEY"], + } + secret_files: list[str] = [] + for name, secret_value in generated.items(): + target = secrets_dir / name + write_private(target, secret_value + "\n") + secret_files.append(str(target)) + + upstream_url = values["UPSTREAM_BASE_URL"].rstrip("/") + upstream_origin_parts = urlsplit(upstream_url) + upstream_origin = f"{upstream_origin_parts.scheme}://{upstream_origin_parts.netloc}" + upstream_model = values["UPSTREAM_MODEL"] + litellm_database = ( + "postgresql://p6_litellm:" + + quote(generated["litellm_postgres"], safe="") + + "@litellm-postgres:5432/p6_litellm" + ) + shell_database = ( + "postgresql://p6_shell:" + + quote(generated["shell_postgres"], safe="") + + "@shell-postgres:5432/p6_shell" + ) + + write_private( + secrets_dir / "litellm.env", + "\n".join( + [ + f"LITELLM_MASTER_KEY={generated['litellm_master']}", + f"DATABASE_URL={litellm_database}", + "REDIS_HOST=litellm-redis", + "REDIS_PORT=6379", + "REDIS_PASSWORD_FILE=/run/secrets/litellm_redis_password", + "STORE_PROMPTS_IN_SPEND_LOGS=false", + "STORE_MODEL_IN_DB=True", + "LITELLM_LOG=INFO", + "LITELLM_HOST=0.0.0.0", + "LITELLM_PORT=4000", + ] + ) + + "\n", + mode=0o600, + ) + write_private( + secrets_dir / "shell.env", + "\n".join( + [ + "APP_NAME=console", + "HOSTNAME=0.0.0.0", + "PORT=3002", + f"MODEL_ACCESS_DATABASE_URL={shell_database}", + "USER_CENTER_INTERNAL_ORIGIN=http://user-center:8080", + "NEXT_PUBLIC_USER_CENTER_ORIGIN=http://user-center:8080", + "NEXT_PUBLIC_PORTAL_ORIGIN=http://shell:3002", + "JUPYTERHUB_INTERNAL_ORIGIN=http://launcher:8000", + f"JUPYTERHUB_API_TOKEN={generated['hub']}", + "MODEL_ACCESS_TEST_ONLY_ALLOW_HTTP_LITELLM=true", + "LITELLM_MANAGEMENT_URL=http://litellm:4000", + f"LITELLM_MASTER_KEY={generated['litellm_master']}", + f"MODEL_ACCESS_TEST_ONLY_UPSTREAM_ORIGINS={upstream_origin}", + "MODEL_ACCESS_TEST_ONLY_ALLOW_PRIVATE_ENDPOINTS=true", + f"MODEL_ACCESS_TEST_ENDPOINT_HOSTS={upstream_origin_parts.hostname}", + f"MODEL_ACCESS_GENERAL_KEY_MODELS={upstream_model}", + "MODEL_ACCESS_GENERAL_KEY_RPM=30", + "MODEL_ACCESS_GENERAL_KEY_TPM=100000", + "MODEL_ACCESS_GENERAL_KEY_MAX_BUDGET=1", + f"MODEL_ACCESS_KEK_BASE64={generated['kek']}", + f"MODEL_ACCESS_LAUNCHER_SERVICE_TOKEN={generated['launcher']}", + "MODEL_ACCESS_RUNTIME_TTL_SECONDS=900", + "MODEL_ACCESS_RUNTIME_RPM=30", + "MODEL_ACCESS_RUNTIME_TPM=100000", + "MODEL_ACCESS_RUNTIME_MAX_BUDGET=1", + "MODEL_ACCESS_DATA_PLANE_URL=https://litellm-gateway:4443", + ] + ) + + "\n", + mode=0o600, + ) + write_private( + secrets_dir / "launcher.env", + "\n".join( + [ + "PROFILE_LAUNCHER=docker", + "PORT=8000", + "BASE_URL=/studio", + "HUB_CONNECT_IP=launcher", + f"NAME_HUB_CONTAINER={launcher_container}", + f"DIR_USR_WORKSPACE={workspace_root}", + "MODEL_ACCESS_CONFIG_FILE=/run/p6/model-access.json", + f"JUPYTERHUB_API_TOKEN={generated['hub']}", + f"OAUTH2_PROXY_COOKIE_SECRET={generated['oauth_cookie']}", + "LOG_LEVEL=INFO", + ] + ) + + "\n", + mode=0o600, + ) + write_private( + secrets_dir / "model-access.json", + json.dumps( + { + "endpoint": "http://shell:3002/console/api", + "service_token": generated["launcher"], + }, + separators=(",", ":"), + ) + + "\n", + ) + + ca_key = secrets_dir / "p6-ca.key" + ca_cert = config_dir / "p6-ca.pem" + subprocess.run( + [ + "openssl", + "req", + "-x509", + "-newkey", + "rsa:2048", + "-nodes", + "-keyout", + str(ca_key), + "-out", + str(ca_cert), + "-subj", + "/CN=litellm-gateway", + "-addext", + "subjectAltName=DNS:litellm-gateway,IP:127.0.0.1", + "-days", + "1", + ], + check=True, + stdin=subprocess.DEVNULL, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ) + os.chmod(ca_key, 0o400) + os.chmod(ca_cert, 0o444) + + write_private( + config_dir / "nginx.conf", + """events {} +http { + access_log /dev/stdout; + error_log /dev/stderr warn; + server { + listen 4443 ssl; + ssl_certificate /run/p6/p6-ca.pem; + ssl_certificate_key /run/p6/p6-ca.key; + location / { + proxy_pass http://litellm:4000; + proxy_http_version 1.1; + proxy_buffering off; + proxy_request_buffering off; + proxy_set_header Host $host; + proxy_set_header Authorization $http_authorization; + } + } +} +""", + mode=0o444, + ) + + source_app_conf = launcher_repo / "src/labnow-launcher/resource/config/app.conf" + app_conf = source_app_conf.read_text(encoding="utf-8") + workspace_image = inputs["images"]["openclaw_workspace"]["ref"] + if not workspace_image.startswith("quay.io/"): + raise PrepareError("P6_WORKSPACE_IMAGE_INVALID") + workspace_image_name = workspace_image.removeprefix("quay.io/") + app_conf += ( + "\n# P6 run-scoped overrides.\n" + f"service.port = 8000\n" + f"launcher.dir_usr_workspace = {json.dumps(str(workspace_root))}\n" + "launcher.workspace_registry = \"quay.io\"\n" + f"launcher.workspace_images = [{json.dumps(workspace_image_name)}]\n" + f"model_access.trusted_config_file = {json.dumps('/run/p6/model-access.json')}\n" + f"docker_spawner.network_name = {json.dumps(network)}\n" + f"docker_spawner.prefix = {json.dumps(prefix)}\n" + "docker_spawner.post_start_cmd = \"\"\n" + "docker_spawner.environment = {\n" + " PROFILE_LOCALIZE = \"default\"\n" + " NODE_EXTRA_CA_CERTS = \"/run/labnow/p6-ca.pem\"\n" + " SSL_CERT_FILE = \"/run/labnow/p6-ca.pem\"\n" + "}\n" + "docker_spawner.read_only_volumes = {\n" + f" {json.dumps(str(ca_cert))} = \"/run/labnow/p6-ca.pem\"\n" + "}\n" + ) + write_private(config_dir / "app.conf", app_conf, mode=0o444) + + compose_file = p6_dir / "docker-compose.runtime.yml" + runtime_env = work_dir / "runtime.env" + write_private( + runtime_env, + "\n".join( + [ + f"P6_RUNTIME_NETWORK={network}", + f"P6_LAUNCHER_CONTAINER={launcher_container}", + f"P6_SHELL_CONTAINER={shell_container}", + f"P6_LITELLM_CONTAINER={litellm_container}", + f"P6_SHELL_POSTGRES_CONTAINER={shell_postgres_container}", + f"P6_LITELLM_POSTGRES_CONTAINER={litellm_postgres_container}", + f"P6_WORKSPACE_ROOT={workspace_root}", + f"P6_LAUNCHER_DATA_DIR={work_dir / 'launcher-data'}", + f"P6_LAUNCHER_APP_CONF={config_dir / 'app.conf'}", + f"P6_LAUNCHER_ENV_FILE={secrets_dir / 'launcher.env'}", + f"P6_SHELL_ENV_FILE={secrets_dir / 'shell.env'}", + f"P6_LITELLM_ENV_FILE={secrets_dir / 'litellm.env'}", + f"P6_MODEL_ACCESS_CONFIG={secrets_dir / 'model-access.json'}", + f"P6_CA_CERT={ca_cert}", + f"P6_CA_KEY={ca_key}", + f"P6_NGINX_CONFIG={config_dir / 'nginx.conf'}", + f"P6_USER_CENTER_SCRIPT={script_dir / 'p6-user-center.py'}", + f"P6_SHELL_MIGRATION={shell_repo / 'web/apps/console/src/lib/model-access/migrations/001_initial.sql'}", + f"P6_LITELLM_CONFIG={lab_dev / 'docker_litellm/demo/config.yaml'}", + f"P6_LITELLM_MIGRATE_CONFIG={lab_dev / 'docker_litellm/demo/config.migrate.yaml'}", + f"P6_LITELLM_START_SCRIPT={lab_dev / 'docker_litellm/work/start-litellm.sh'}", + f"P6_LITELLM_MIGRATION_SCRIPT={lab_dev / 'docker_litellm/work/run-migration-locked.py'}", + f"P6_LITELLM_IMAGE={inputs['images']['litellm']['ref']}", + f"P6_WORKSPACE_IMAGE={inputs['images']['openclaw_workspace']['ref']}", + f"P6_LAUNCHER_IMAGE={inputs['local_only_images']['launcher']['ref']}", + f"P6_SHELL_IMAGE={inputs['local_only_images']['shell']['ref']}", + f"P6_POSTGRES_IMAGE={inputs['support_images']['postgres']['ref']}", + f"P6_REDIS_IMAGE={inputs['support_images']['redis']['ref']}", + f"P6_NGINX_IMAGE={inputs['support_images']['nginx']['ref']}", + f"P6_LITELLM_POSTGRES_PASSWORD_FILE={secrets_dir / 'litellm_postgres'}", + f"P6_REDIS_PASSWORD_FILE={secrets_dir / 'redis'}", + f"P6_SHELL_POSTGRES_PASSWORD_FILE={secrets_dir / 'shell_postgres'}", + f"P6_OWNER_A={user}", + f"P6_OWNER_B=p6other-{short[:8]}", + ] + ) + + "\n", + mode=0o600, + ) + (work_dir / "launcher-data").mkdir(mode=0o700, exist_ok=True) + + driver_config = { + "schema_version": "p6-product-chain-config/v1", + "run_id": run_id, + "project": project, + "compose_file": str(compose_file), + "runtime_env_file": str(runtime_env), + "work_dir": str(work_dir), + "surface_dir": str(surfaces_dir), + "workspace_root": str(workspace_root), + "runtime_root": str(workspace_root / ".runtime/model-access"), + "owner_a": user, + "owner_b": f"p6other-{short[:8]}", + "server_name": server, + "workspace_container": workspace_container, + "launcher_container": launcher_container, + "shell_container": shell_container, + "litellm_container": litellm_container, + "shell_postgres_container": shell_postgres_container, + "workspace_image": inputs["images"]["openclaw_workspace"]["ref"], + "hub_token_file": str(secrets_dir / "hub"), + "launcher_token_file": str(secrets_dir / "launcher"), + "upstream_key_file": str(secrets_dir / "upstream"), + "upstream_origin": upstream_origin, + "upstream_model": upstream_model, + "ca_file": str(ca_cert), + "secret_files": secret_files, + "shell_ui_evidence": { + "repository": "labnow_shell", + "commit": inputs["repositories"]["labnow_shell"]["commit"], + "status": "reused_verified_evidence", + }, + } + write_private(config_dir / "driver-config.json", json.dumps(driver_config, separators=(",", ":")) + "\n") + return 0 + + +if __name__ == "__main__": + try: + sys.exit(main()) + except (PrepareError, OSError, KeyError, TypeError, ValueError, subprocess.SubprocessError) as exc: + code = str(exc) if isinstance(exc, PrepareError) else "P6_PREPARE_FAILED" + print(f"P6_ERROR:{code}", file=sys.stderr) + sys.exit(1) diff --git a/docker_openclaw/p6/scripts/p6-product-chain.py b/docker_openclaw/p6/scripts/p6-product-chain.py new file mode 100755 index 0000000..b04c229 --- /dev/null +++ b/docker_openclaw/p6/scripts/p6-product-chain.py @@ -0,0 +1,960 @@ +#!/usr/bin/env python3 +"""Execute the real P6 Shell -> JupyterHub -> Workspace -> LiteLLM chain. + +The driver keeps credentials and model responses in memory only. Durable output +contains structural assertions, non-sensitive IDs, counts and hashes. +""" + +from __future__ import annotations + +import json +import os +import ssl +import stat +import subprocess +import sys +import time +import urllib.error +import urllib.parse +import urllib.request +from datetime import datetime, timedelta, timezone +from pathlib import Path +from typing import Any + + +MANIFEST = "/run/labnow/model-access/manifest.json" +SECRET = "/run/labnow/model-access/secret.json" +STATUS = "/run/labnow/model-access/status.json" +OPENCLAW_DATA = "/opt/openclaw/data" +ALLOWED_USAGE_FIELDS = { + "timestamp", + "model", + "total_tokens", + "prompt_tokens", + "completion_tokens", + "status", +} + + +class DriverError(RuntimeError): + pass + + +def fail(code: str) -> None: + raise DriverError(code) + + +def restricted(path: Path, *, code: str) -> None: + try: + info = path.stat() + except OSError as exc: + raise DriverError(code) from exc + if not stat.S_ISREG(info.st_mode) or info.st_mode & 0o077: + fail(code) + + +def private_write(path: Path, value: str, mode: int = 0o600) -> None: + path.parent.mkdir(mode=0o700, parents=True, exist_ok=True) + temporary = path.with_name(f".{path.name}.{os.getpid()}") + descriptor = os.open(temporary, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) + try: + with os.fdopen(descriptor, "w", encoding="utf-8") as handle: + handle.write(value) + handle.flush() + os.fsync(handle.fileno()) + os.chmod(temporary, mode) + os.replace(temporary, path) + except Exception: + try: + os.unlink(temporary) + except OSError: + pass + raise + + +def load_config(path: Path) -> dict[str, Any]: + restricted(path, code="PRODUCT_CONFIG_INVALID") + try: + value = json.loads(path.read_text(encoding="utf-8")) + except (OSError, UnicodeDecodeError, json.JSONDecodeError) as exc: + raise DriverError("PRODUCT_CONFIG_INVALID") from exc + required = { + "schema_version", + "run_id", + "project", + "compose_file", + "runtime_env_file", + "work_dir", + "surface_dir", + "workspace_root", + "runtime_root", + "owner_a", + "owner_b", + "server_name", + "workspace_container", + "launcher_container", + "shell_container", + "litellm_container", + "shell_postgres_container", + "workspace_image", + "hub_token_file", + "launcher_token_file", + "upstream_key_file", + "upstream_origin", + "upstream_model", + "ca_file", + "secret_files", + "shell_ui_evidence", + } + if not isinstance(value, dict) or set(value) != required or value.get("schema_version") != "p6-product-chain-config/v1": + fail("PRODUCT_CONFIG_INVALID") + for key in required - {"secret_files", "shell_ui_evidence"}: + if not isinstance(value.get(key), str) or not value[key]: + fail("PRODUCT_CONFIG_INVALID") + if not isinstance(value["secret_files"], list) or not value["secret_files"]: + fail("PRODUCT_CONFIG_INVALID") + return value + + +def read_secret(path: str, code: str) -> str: + target = Path(path) + restricted(target, code=code) + try: + value = target.read_text(encoding="utf-8").strip() + except (OSError, UnicodeDecodeError) as exc: + raise DriverError(code) from exc + if not value or "\n" in value: + fail(code) + return value + + +def command(args: list[str], *, code: str, timeout: int = 120) -> str: + try: + result = subprocess.run( + args, + check=False, + stdin=subprocess.DEVNULL, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + timeout=timeout, + ) + except (OSError, subprocess.SubprocessError) as exc: + raise DriverError(code) from exc + if result.returncode != 0: + fail(code) + try: + return result.stdout.decode("utf-8") + except UnicodeDecodeError as exc: + raise DriverError(code) from exc + + +def command_combined(args: list[str], *, code: str, timeout: int = 120) -> str: + try: + result = subprocess.run( + args, + check=False, + stdin=subprocess.DEVNULL, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + timeout=timeout, + ) + except (OSError, subprocess.SubprocessError) as exc: + raise DriverError(code) from exc + if result.returncode != 0: + fail(code) + try: + return result.stdout.decode("utf-8") + except UnicodeDecodeError as exc: + raise DriverError(code) from exc + + +def compose(config: dict[str, Any], *arguments: str, code: str, timeout: int = 120) -> str: + return command( + [ + "docker", + "compose", + "--project-name", + config["project"], + "--env-file", + config["runtime_env_file"], + "-f", + config["compose_file"], + *arguments, + ], + code=code, + timeout=timeout, + ) + + +def published_port(config: dict[str, Any], service: str, port: int) -> int: + value = compose(config, "port", service, str(port), code="PUBLISHED_PORT_UNAVAILABLE").strip() + try: + parsed = int(value.rsplit(":", 1)[1]) + except (IndexError, ValueError) as exc: + raise DriverError("PUBLISHED_PORT_UNAVAILABLE") from exc + if parsed < 1024 or parsed > 65535: + fail("PUBLISHED_PORT_UNAVAILABLE") + return parsed + + +def http_json( + method: str, + url: str, + *, + headers: dict[str, str] | None = None, + body: object | None = None, + context: ssl.SSLContext | None = None, + timeout: int = 30, +) -> tuple[int, Any]: + payload = None if body is None else json.dumps(body, separators=(",", ":")).encode("utf-8") + request = urllib.request.Request( + url, + data=payload, + method=method, + headers={ + "Accept": "application/json", + **({"Content-Type": "application/json"} if payload is not None else {}), + **(headers or {}), + }, + ) + try: + with urllib.request.urlopen(request, timeout=timeout, context=context) as response: + raw = response.read() + decoded = json.loads(raw.decode("utf-8")) if raw else None + return response.status, decoded + except urllib.error.HTTPError as exc: + try: + raw = exc.read() + decoded = json.loads(raw.decode("utf-8")) if raw else None + except (OSError, UnicodeDecodeError, json.JSONDecodeError): + decoded = None + return exc.code, decoded + except (urllib.error.URLError, TimeoutError, OSError, UnicodeDecodeError, json.JSONDecodeError) as exc: + raise DriverError("HTTP_DEPENDENCY_UNAVAILABLE") from exc + + +def require_status(actual: int, expected: set[int], code: str, response: Any = None) -> None: + if actual not in expected: + details = { + "Endpoint probe failed.": "ENDPOINT_PROBE_FAILED", + "LiteLLM rejected the operation.": "LITELLM_REJECTED", + "LiteLLM is unavailable.": "LITELLM_UNAVAILABLE", + "LiteLLM request timed out.": "LITELLM_TIMEOUT", + "Model access is temporarily unavailable.": "MODEL_ACCESS_UNAVAILABLE", + } + marker = details.get(response.get("detail")) if isinstance(response, dict) else None + if marker is None and isinstance(response, dict): + marker = { + "Failed to create JupyterHub user": "JUPYTERHUB_USER_CREATE_FAILED", + "Workspace model binding was not found.": "WORKSPACE_BINDING_NOT_FOUND", + "Internal Server Error": "INTERNAL_SERVER_ERROR", + }.get(response.get("message")) + if marker is None and response is not None: + serialized = json.dumps(response, sort_keys=True) + marker = next( + ( + value + for value in ( + "MODEL_ACCESS_REQUEST_REJECTED", + "MODEL_ACCESS_UNAVAILABLE", + "BINDING_NOT_FOUND", + "UNTRUSTED_MODEL_ACCESS_ADAPTER", + "ADAPTER_APPLY_FAILED", + "WORKSPACE_START_TIMEOUT", + ) + if value in serialized + ), + None, + ) + fail(f"{code}_HTTP_{actual}" + (f"_{marker}" if marker else "")) + + +def shell_headers(owner: str, request_id: str) -> dict[str, str]: + return { + "Cookie": f"p6_owner={owner}", + "X-Request-Id": request_id, + "Idempotency-Key": request_id, + } + + +def hub_headers(token: str) -> dict[str, str]: + return {"Authorization": f"token {token}"} + + +def wait_hub_server(hub: str, token: str, owner: str, server: str, *, running: bool) -> dict[str, Any]: + owner_q = urllib.parse.quote(owner, safe="") + deadline = time.monotonic() + 180 + while time.monotonic() < deadline: + status, body = http_json( + "GET", + f"{hub}/users/{owner_q}?include_stopped_servers=true", + headers=hub_headers(token), + ) + if not running and status == 404: + return {} + servers = body.get("servers") if status == 200 and isinstance(body, dict) else None + snapshot = servers.get(server) if isinstance(servers, dict) else None + if running and isinstance(snapshot, dict) and snapshot.get("ready"): + return snapshot + if not running and snapshot is None and isinstance(servers, dict): + return {} + if not running and isinstance(snapshot, dict) and not snapshot.get("ready") and not snapshot.get("pending"): + return snapshot + time.sleep(1) + fail("HUB_SERVER_START_TIMEOUT" if running else "HUB_SERVER_STOP_TIMEOUT") + + +def material(config: dict[str, Any]) -> tuple[Path, dict[str, Any], str]: + root = Path(config["runtime_root"]) + candidates = list(root.glob("workspace-*/lease-*")) + if len(candidates) != 1: + fail("RUNTIME_MATERIAL_AMBIGUOUS") + directory = candidates[0] + manifest_path = directory / "manifest.json" + secret_path = directory / "secret.json" + restricted(manifest_path, code="RUNTIME_MATERIAL_INVALID") + restricted(secret_path, code="RUNTIME_MATERIAL_INVALID") + try: + manifest_value = json.loads(manifest_path.read_text(encoding="utf-8")) + secret_value = json.loads(secret_path.read_text(encoding="utf-8")) + except (OSError, UnicodeDecodeError, json.JSONDecodeError) as exc: + raise DriverError("RUNTIME_MATERIAL_INVALID") from exc + key = secret_value.get("api_key") if isinstance(secret_value, dict) else None + if ( + not isinstance(manifest_value, dict) + or manifest_value.get("contract_version") != "v1alpha1" + or manifest_value.get("workspace_id") != config["server_name"] + or not isinstance(manifest_value.get("generation"), int) + or not isinstance(key, str) + or not key + ): + fail("RUNTIME_MATERIAL_INVALID") + return directory, manifest_value, key + + +def docker_inspect(container: str) -> dict[str, Any]: + try: + value = json.loads(command(["docker", "inspect", container], code="CONTAINER_INSPECT_FAILED")) + except json.JSONDecodeError as exc: + raise DriverError("CONTAINER_INSPECT_FAILED") from exc + if not isinstance(value, list) or len(value) != 1 or not isinstance(value[0], dict): + fail("CONTAINER_INSPECT_FAILED") + return value[0] + + +def assert_workspace(config: dict[str, Any], runtime_key: str) -> tuple[str, dict[str, Any]]: + inspect = docker_inspect(config["workspace_container"]) + mounts = {item.get("Destination"): item for item in inspect.get("Mounts", [])} + for target in (MANIFEST, SECRET, "/run/labnow/p6-ca.pem"): + if target not in mounts or mounts[target].get("RW") is not False: + fail("WORKSPACE_MOUNT_INVALID") + serialized = json.dumps(inspect, sort_keys=True) + if runtime_key in serialized: + fail("RUNTIME_KEY_LEAKED_TO_INSPECT") + env = inspect.get("Config", {}).get("Env", []) + prefix = next((item.split("=", 1)[1] for item in env if item.startswith("URL_PREFIX=")), "") + if not prefix.startswith("/studio/user/"): + fail("WORKSPACE_URL_PREFIX_INVALID") + status_raw = command(["docker", "exec", config["workspace_container"], "cat", STATUS], code="ADAPTER_STATUS_UNAVAILABLE") + try: + adapter_status = json.loads(status_raw) + except json.JSONDecodeError as exc: + raise DriverError("ADAPTER_STATUS_INVALID") from exc + if adapter_status.get("phase") != "ready" or adapter_status.get("adapter_id") != "openclaw": + fail("ADAPTER_STATUS_INVALID") + deadline = time.monotonic() + 90 + readiness = f"http://127.0.0.1{prefix}api" + while time.monotonic() < deadline: + try: + result = subprocess.run( + ["docker", "exec", config["workspace_container"], "curl", "--fail", "--silent", "--max-time", "3", readiness], + check=False, + stdin=subprocess.DEVNULL, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + timeout=5, + ) + except (OSError, subprocess.SubprocessError): + result = None + if result is not None and result.returncode == 0: + break + time.sleep(1) + else: + fail("OPENCLAW_READINESS_FAILED") + summary = { + "image": inspect.get("Config", {}).get("Image"), + "mounts": sorted(mounts), + "environment_keys": sorted(item.split("=", 1)[0] for item in env), + "state": inspect.get("State", {}).get("Status"), + } + return prefix, summary + + +def data_plane(port: int, ca_file: str, key: str, *, accepted: bool) -> None: + context = ssl.create_default_context(cafile=ca_file) + status, _ = http_json( + "GET", + f"https://127.0.0.1:{port}/models", + headers={"Authorization": f"Bearer {key}"}, + context=context, + ) + if accepted and status == 200: + return + if not accepted and status in {401, 403}: + return + fail("DATA_PLANE_KEY_UNEXPECTED") + + +def wait_rejected(port: int, ca_file: str, key: str) -> None: + deadline = time.monotonic() + 35 + while time.monotonic() < deadline: + try: + data_plane(port, ca_file, key, accepted=False) + return + except DriverError as exc: + if str(exc) != "DATA_PLANE_KEY_UNEXPECTED": + raise + time.sleep(1) + fail("DATA_PLANE_REVOCATION_TIMEOUT") + + +def trajectory(config: dict[str, Any], session: str) -> tuple[dict[str, int | bool], str]: + path = f"{OPENCLAW_DATA}/agents/main/sessions/{session}.trajectory.jsonl" + raw = command(["docker", "exec", config["workspace_container"], "cat", path], code="TRAJECTORY_UNAVAILABLE") + parsed = 0 + errors = 0 + completed = 0 + ended = 0 + for line in raw.splitlines(): + if not line.strip(): + continue + try: + event = json.loads(line) + except json.JSONDecodeError: + errors += 1 + continue + if isinstance(event, dict): + parsed += 1 + completed += int(event.get("type") == "model.completed") + ended += int(event.get("type") == "session.ended") + return { + "parsed_event_count": parsed, + "parse_error_count": errors, + "model_completed_count": completed, + "session_ended_count": ended, + }, raw + + +def run_agent(config: dict[str, Any], model: str, session: str, prompt: str) -> dict[str, int | bool]: + command( + [ + "docker", + "exec", + config["workspace_container"], + "timeout", + "--signal=TERM", + "--kill-after=10s", + "120s", + "openclaw", + "agent", + "--local", + "--session-id", + session, + "--model", + model, + "--message", + prompt, + "--json", + ], + code="OPENCLAW_AGENT_FAILED", + timeout=140, + ) + summary, _ = trajectory(config, session) + if summary["parse_error_count"] or not summary["model_completed_count"] or not summary["session_ended_count"]: + fail("OPENCLAW_CHAT_STRUCTURE_INVALID") + return summary + + +def run_stream(config: dict[str, Any], model: str, session: str) -> dict[str, int | bool]: + raw_path = f"{OPENCLAW_DATA}/{session}.raw.jsonl" + try: + command( + [ + "docker", + "exec", + "-e", + "OPENCLAW_RAW_STREAM=1", + "-e", + f"OPENCLAW_RAW_STREAM_PATH={raw_path}", + config["workspace_container"], + "timeout", + "--signal=TERM", + "--kill-after=10s", + "120s", + "openclaw", + "agent", + "--local", + "--session-id", + session, + "--model", + model, + "--message", + "Return P6_STREAM_OK only.", + "--json", + ], + code="OPENCLAW_STREAM_FAILED", + timeout=140, + ) + deadline = time.monotonic() + 10 + raw = "" + while time.monotonic() < deadline: + try: + raw = command(["docker", "exec", config["workspace_container"], "cat", raw_path], code="STREAM_EVENTS_PENDING") + except DriverError: + time.sleep(1) + continue + if raw.strip(): + break + time.sleep(1) + parsed = 0 + errors = 0 + for line in raw.splitlines(): + if not line.strip(): + continue + try: + value = json.loads(line) + except json.JSONDecodeError: + errors += 1 + else: + parsed += int(isinstance(value, dict)) + if not parsed or errors: + fail("OPENCLAW_STREAM_STRUCTURE_INVALID") + return {"parsed_event_count": parsed, "parse_error_count": errors, "terminated": True} + finally: + subprocess.run( + ["docker", "exec", config["workspace_container"], "rm", "-f", raw_path], + check=False, + stdin=subprocess.DEVNULL, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ) + + +def run_tool(config: dict[str, Any], model: str, session: str) -> dict[str, int | bool]: + command( + [ + "docker", + "exec", + config["workspace_container"], + "timeout", + "--signal=TERM", + "--kill-after=10s", + "120s", + "openclaw", + "agent", + "--local", + "--session-id", + session, + "--model", + model, + "--message", + "Use exec to run printf P6_TOOL_OK, then reply DONE.", + "--json", + ], + code="OPENCLAW_TOOL_FAILED", + timeout=140, + ) + summary, raw = trajectory(config, session) + if "P6_TOOL_OK" not in raw or summary["parse_error_count"]: + fail("OPENCLAW_TOOL_NOT_OBSERVED") + return {**summary, "tool_observed": True} + + +def psql(config: dict[str, Any], query: str) -> str: + return command( + [ + "docker", + "exec", + config["shell_postgres_container"], + "psql", + "-U", + "p6_shell", + "-d", + "p6_shell", + "-Atc", + query, + ], + code="SHELL_DATABASE_QUERY_FAILED", + ).strip() + + +def usage_time(value: datetime) -> str: + return value.astimezone(timezone.utc).isoformat(timespec="milliseconds").replace("+00:00", "Z") + + +def usage_check(shell: str, config: dict[str, Any], routed_model: str, from_time: str, to_time: str) -> tuple[int, str]: + key_id = psql( + config, + "SELECT id FROM model_access.virtual_keys WHERE owner_id='" + + config["owner_a"].replace("'", "''") + + "' AND key_kind='runtime' ORDER BY created_at DESC LIMIT 1", + ) + if not key_id: + fail("RUNTIME_KEY_ID_MISSING") + query = urllib.parse.urlencode( + { + "workspace_id": config["server_name"], + "key_id": key_id, + "model": routed_model, + "from": from_time, + "to": to_time, + } + ) + deadline = time.monotonic() + 45 + usage: list[Any] = [] + query_succeeded = False + while time.monotonic() < deadline: + status, body = http_json( + "GET", + f"{shell}/model-access/usage/?{query}", + headers=shell_headers("a", f"usage-{config['run_id']}"), + ) + if status in {400, 404}: + fail("USAGE_QUERY_REJECTED") + if status == 200: + if not isinstance(body, dict) or not isinstance(body.get("data"), list): + fail("USAGE_RESPONSE_INVALID") + query_succeeded = True + usage = body["data"] + if usage: + break + time.sleep(2) + if not query_succeeded: + fail("USAGE_QUERY_FAILED") + if not usage: + fail("USAGE_NOT_OBSERVED") + for row in usage: + if not isinstance(row, dict) or set(row) != ALLOWED_USAGE_FIELDS or row.get("model") != routed_model: + fail("USAGE_PROJECTION_INVALID") + status, negative = http_json( + "GET", + f"{shell}/model-access/usage/?{query}", + headers=shell_headers("b", f"usage-negative-{config['run_id']}"), + ) + if status == 200 and isinstance(negative, dict) and negative.get("data") == []: + pass + elif status not in {400, 404}: + fail("USAGE_OWNER_NEGATIVE_FAILED") + return len(usage), key_id + + +def capture_surfaces(config: dict[str, Any], workspace_summary: dict[str, Any], generation: int) -> list[str]: + surface = Path(config["surface_dir"]) + surface.mkdir(mode=0o700, parents=True, exist_ok=True) + logs = compose(config, "logs", "--no-color", code="TOPOLOGY_LOG_CAPTURE_FAILED", timeout=120) + private_write(surface / f"topology-g{generation}.log", logs) + workspace_logs = command_combined(["docker", "logs", config["workspace_container"]], code="WORKSPACE_LOG_CAPTURE_FAILED") + private_write(surface / f"workspace-g{generation}.log", workspace_logs) + processes = command(["docker", "top", config["workspace_container"], "-eo", "pid,args"], code="WORKSPACE_PROCESS_CAPTURE_FAILED") + private_write(surface / f"workspace-process-g{generation}.txt", processes) + openclaw_config = command( + ["docker", "exec", config["workspace_container"], "cat", "/root/.openclaw/data/openclaw.json"], + code="OPENCLAW_CONFIG_CAPTURE_FAILED", + ) + private_write(surface / f"openclaw-config-g{generation}.json", openclaw_config) + adapter_status = command(["docker", "exec", config["workspace_container"], "cat", STATUS], code="ADAPTER_STATUS_UNAVAILABLE") + private_write(surface / f"adapter-status-g{generation}.json", adapter_status) + runtime_surface = surface / f"openclaw-runtime-g{generation}" + command( + ["docker", "cp", f"{config['workspace_container']}:{OPENCLAW_DATA}", str(runtime_surface)], + code="OPENCLAW_RUNTIME_CAPTURE_FAILED", + timeout=120, + ) + if not runtime_surface.is_dir() or runtime_surface.is_symlink(): + fail("OPENCLAW_RUNTIME_CAPTURE_INVALID") + private_write(surface / f"workspace-inspect-g{generation}.json", json.dumps(workspace_summary, sort_keys=True) + "\n") + return [str(surface), str(Path(config["workspace_root"]) / config["owner_a"])] + + +def write_patterns(config: dict[str, Any], path: Path, runtime_keys: list[str]) -> None: + values = [read_secret(item, "SECRET_SOURCE_INVALID") for item in config["secret_files"]] + values.extend(runtime_keys) + if any("\n" in value or not value for value in values): + fail("SECRET_PATTERN_INVALID") + private_write(path, "\n".join(dict.fromkeys(values)) + "\n", mode=0o400) + + +def execute(config: dict[str, Any]) -> dict[str, Any]: + shell_port = published_port(config, "shell", 3002) + hub_port = published_port(config, "launcher", 8000) + gateway_port = published_port(config, "litellm-gateway", 4443) + shell = f"http://127.0.0.1:{shell_port}/console/api" + hub = f"http://127.0.0.1:{hub_port}/studio/hub/api" + hub_token = read_secret(config["hub_token_file"], "HUB_TOKEN_INVALID") + launcher_token = read_secret(config["launcher_token_file"], "LAUNCHER_TOKEN_INVALID") + upstream_key = read_secret(config["upstream_key_file"], "UPSTREAM_KEY_INVALID") + run = config["run_id"].replace("p6-", "")[:12] + + status, connection = http_json( + "POST", + f"{shell}/model-access/connections/", + headers=shell_headers("a", f"connection-{run}"), + body={ + "display_name": f"P6 {run}", + "provider": "openai", + "endpoint": config["upstream_origin"], + "api_key": upstream_key, + }, + ) + require_status(status, {201}, "CONNECTION_CREATE_FAILED", connection) + connection_id = connection.get("data", {}).get("id") if isinstance(connection, dict) else None + if not isinstance(connection_id, str): + fail("CONNECTION_RESPONSE_INVALID") + + status, route = http_json( + "POST", + f"{shell}/model-access/routes/", + headers=shell_headers("a", f"route-{run}"), + body={ + "connection_id": connection_id, + "display_name": f"P6 route {run}", + "upstream_model": config["upstream_model"], + }, + ) + require_status(status, {201}, "ROUTE_CREATE_FAILED") + route_value = route.get("data", {}) if isinstance(route, dict) else {} + route_id = route_value.get("id") + routed_model = route_value.get("routed_model") + if not isinstance(route_id, str) or not isinstance(routed_model, str): + fail("ROUTE_RESPONSE_INVALID") + + status, binding = http_json( + "POST", + f"{shell}/model-access/bindings/", + headers=shell_headers("a", f"binding-{run}"), + body={"workspace_id": config["server_name"], "route_id": route_id, "adapter_id": "openclaw"}, + ) + require_status(status, {201}, "BINDING_CREATE_FAILED") + binding_value = binding.get("data", {}) if isinstance(binding, dict) else {} + binding_id = binding_value.get("binding_id") + if not isinstance(binding_id, str) or set(binding_value) != { + "contract_version", + "binding_id", + "workspace_id", + "route_id", + "adapter_id", + "default_model", + "allowed_models", + }: + fail("BINDING_RESPONSE_INVALID") + + spawn_body = { + "tier": "basic", + "image": config["workspace_image"], + "serverName": config["server_name"], + "model_access": {"contract_version": "v1alpha1", "binding_id": binding_id}, + } + status, spawn_response = http_json( + "POST", + f"{shell}/hub/spawn/", + headers=shell_headers("a", f"spawn-g1-{run}"), + body=spawn_body, + timeout=45, + ) + require_status(status, {201, 202}, "SHELL_SPAWN_FAILED", spawn_response) + server_snapshot = wait_hub_server(hub, hub_token, config["owner_a"], config["server_name"], running=True) + material_dir_1, manifest_1, key_1 = material(config) + _, workspace_summary_1 = assert_workspace(config, key_1) + data_plane(gateway_port, config["ca_file"], key_1, accepted=True) + if key_1 in json.dumps(server_snapshot, sort_keys=True): + fail("RUNTIME_KEY_LEAKED_TO_HUB_API") + generation_1 = manifest_1["generation"] + model_ref = f"labnow/{manifest_1['default_model']}" + from_time = usage_time(datetime.now(timezone.utc) - timedelta(minutes=5)) + chat_summary = run_agent(config, model_ref, f"p6-chat-{run}", "Reply P6_CHAT_OK only.") + stream_summary = run_stream(config, model_ref, f"p6-stream-{run}") + tool_summary = run_tool(config, model_ref, f"p6-tool-{run}") + to_time = usage_time(datetime.now(timezone.utc) + timedelta(minutes=5)) + usage_count, _ = usage_check(shell, config, routed_model, from_time, to_time) + scan_roots = capture_surfaces(config, workspace_summary_1, generation_1) + + status, _ = http_json( + "POST", + f"{shell}/hub/stop/", + headers=shell_headers("a", f"stop-g1-{run}"), + body={"serverName": config["server_name"]}, + timeout=45, + ) + require_status(status, {200, 202, 204}, "SHELL_STOP_FAILED") + wait_hub_server(hub, hub_token, config["owner_a"], config["server_name"], running=False) + if material_dir_1.exists(): + fail("GENERATION_1_MATERIAL_REMAINS") + wait_rejected(gateway_port, config["ca_file"], key_1) + + status, restart_response = http_json( + "POST", + f"{shell}/hub/spawn/", + headers=shell_headers("a", f"spawn-g2-{run}"), + body=spawn_body, + timeout=45, + ) + require_status(status, {201, 202}, "SHELL_RESTART_FAILED", restart_response) + wait_hub_server(hub, hub_token, config["owner_a"], config["server_name"], running=True) + material_dir_2, manifest_2, key_2 = material(config) + _, workspace_summary_2 = assert_workspace(config, key_2) + if manifest_2["generation"] <= generation_1 or key_2 == key_1: + fail("GENERATION_NOT_ADVANCED") + data_plane(gateway_port, config["ca_file"], key_2, accepted=True) + wait_rejected(gateway_port, config["ca_file"], key_1) + restart_chat = run_agent(config, f"labnow/{manifest_2['default_model']}", f"p6-restart-{run}", "Reply P6_RESTART_OK only.") + scan_roots.extend(capture_surfaces(config, workspace_summary_2, manifest_2["generation"])) + + late_body = { + "contract_version": "v1alpha1", + "workspace_id": config["server_name"], + "generation": generation_1, + "reason": "reconciled", + } + status, _ = http_json( + "POST", + f"{shell}/internal/model-access/v1alpha1/runtime-leases/{urllib.parse.quote(manifest_1['lease_id'], safe='')}:release/", + headers={ + "Authorization": f"Bearer {launcher_token}", + "Idempotency-Key": f"late-{run}", + "X-Request-Id": f"late-{run}", + }, + body=late_body, + ) + require_status(status, {409}, "LATE_RELEASE_NOT_REJECTED") + data_plane(gateway_port, config["ca_file"], key_2, accepted=True) + + status, _ = http_json( + "DELETE", + f"{shell}/hub/delete/", + headers=shell_headers("a", f"delete-g2-{run}"), + body={"serverName": config["server_name"], "remove": True}, + timeout=45, + ) + require_status(status, {200, 202, 204}, "SHELL_DELETE_FAILED") + wait_hub_server(hub, hub_token, config["owner_a"], config["server_name"], running=False) + if material_dir_2.exists(): + fail("GENERATION_2_MATERIAL_REMAINS") + wait_rejected(gateway_port, config["ca_file"], key_2) + active = psql( + config, + "SELECT count(*) FROM model_access.runtime_leases WHERE owner_id='" + + config["owner_a"].replace("'", "''") + + "' AND workspace_id='" + + config["server_name"].replace("'", "''") + + "' AND state IN ('issued','active','revoking')", + ) + if active != "0": + fail("ACTIVE_LEASE_REMAINS") + if psql(config, "SELECT coalesce(to_regclass('model_access.usage')::text,'absent')") != "absent": + fail("USAGE_BODY_PERSISTENCE_TABLE_PRESENT") + + pattern_file = Path(os.environ.get("P6_SECRET_PATTERN_FILE", "")) + if not pattern_file.is_absolute(): + fail("SECRET_PATTERN_PATH_INVALID") + write_patterns(config, pattern_file, [key_1, key_2]) + scan_roots = sorted(set(root for root in scan_roots if Path(root).exists())) + if not scan_roots: + fail("SCAN_ROOT_MISSING") + + return { + "schema_version": "p6-product-chain-report/v1", + "result": "passed", + "content_redacted": True, + "checks": { + "test_resource_provision": "passed", + "console_ui": config["shell_ui_evidence"]["status"], + "binding_payload": "passed", + "jupyterhub_dockerspawner": "passed", + "launcher_claim_activate_release": "passed", + "openclaw_apply_probe_readiness": "passed", + "chat": "passed", + "stream": "passed", + "tool": "passed", + "usage": "passed", + "owner_negative": "passed", + "prompt_response_absent": "passed", + "revoke": "passed", + "generation_restart": "passed", + "late_release": "passed", + "delete": "passed", + "zero_active_leases": "passed", + }, + "binding": { + "contract_version": "v1alpha1", + "workspace_id": config["server_name"], + "binding_id": binding_id, + "route_id": route_id, + "payload_fields": ["binding_id", "contract_version"], + }, + "runtime": { + "hub_api": "live", + "docker_daemon": "real", + "workspace_image": config["workspace_image"], + "generation_1": generation_1, + "generation_2": manifest_2["generation"], + "mounts": "readonly", + "adapter_phase": "ready", + }, + "data_plane": { + "chat": chat_summary, + "stream": stream_summary, + "tool": tool_summary, + "restart_chat": restart_chat, + }, + "usage": { + "row_count": usage_count, + "fields": sorted(ALLOWED_USAGE_FIELDS), + "owner_negative": "isolated", + "body_fields_absent": True, + "persistence_table": "absent", + }, + "lifecycle": { + "old_key_after_stop": "rejected", + "new_key_after_restart": "accepted", + "old_key_after_restart": "rejected", + "late_old_release": "rejected_409", + "new_key_after_delete": "rejected", + "active_lease_count": 0, + }, + "scan_roots": scan_roots, + } + + +def main() -> int: + config_path = Path(os.environ.get("P6_PRODUCT_CONFIG_FILE", "")) + report_path = Path(os.environ.get("P6_PRODUCT_REPORT_FILE", "")) + try: + config = load_config(config_path) + report = execute(config) + except DriverError as exc: + code = str(exc) + if report_path.is_absolute(): + private_write( + report_path, + json.dumps( + { + "schema_version": "p6-product-chain-report/v1", + "result": "failed", + "content_redacted": True, + "code": code, + }, + separators=(",", ":"), + ) + + "\n", + ) + print(f"P6_ERROR:{code}", file=sys.stderr) + return 1 + if not report_path.is_absolute(): + print("P6_ERROR:PRODUCT_REPORT_PATH_INVALID", file=sys.stderr) + return 1 + private_write(report_path, json.dumps(report, separators=(",", ":")) + "\n") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/docker_openclaw/p6/scripts/p6-runner.sh b/docker_openclaw/p6/scripts/p6-runner.sh index 7cc3e61..88683b8 100755 --- a/docker_openclaw/p6/scripts/p6-runner.sh +++ b/docker_openclaw/p6/scripts/p6-runner.sh @@ -1,7 +1,6 @@ #!/usr/bin/env bash -# P6 local-only, fail-closed golden-chain coordinator. It deliberately does -# not create product resources until all frozen commits and local image -# digests have been verified. +# P6 local-only, fail-closed golden-chain coordinator. Product resources are +# not created until review_snapshot, fixed image and restricted input gates pass. set -euo pipefail script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" @@ -12,12 +11,11 @@ usage() { cat <<'USAGE' Usage: p6-runner.sh --input /secure/path/p6-inputs.json [--validate-input|--preflight|--render|--golden|--cleanup] ---validate-input performs no Docker or product operation. --preflight validates -fixed repositories, local image IDs/digests and secure inputs. --render writes -a non-sensitive Compose rendering. --golden starts only the isolated P6 -OpenClaw service, invokes the separately fixed cross-repository driver, scans -for secrets, and always cleans up. --cleanup removes only this run's Compose -resources and temporary runtime state; it never removes named data volumes. +--validate-input validates only the protected schema. --preflight also binds +the review_snapshot, three product commits and eight local image ID/digests. +--render retains a credential-free topology template summary. --golden creates +one isolated topology, runs the complete product chain and always removes its +exact containers/network/volumes. --cleanup is idempotent for the same run id. USAGE } @@ -45,19 +43,20 @@ chmod 700 "$P6_WORK_DIR" "$artifact_dir" report="${artifact_dir}/p6-${action}-${P6_RUN_ID}.json" driver_cleanup_best_effort() { - local driver - driver="$(jq -r '.driver // empty' "$P6_INPUT_FILE" 2>/dev/null || true)" - if [[ -n "$driver" && -x "$driver" && ! -L "$driver" ]]; then - P6_DRIVER_ACTION=cleanup P6_RUN_ID="$P6_RUN_ID" P6_INPUT_FILE="$P6_INPUT_FILE" P6_DRIVER_REPORT="$P6_WORK_DIR/driver-cleanup-report.json" P6_SECRET_PATTERN_FILE="$P6_WORK_DIR/secret-patterns" "$driver" >/dev/null 2>&1 || true + local driver best_effort_report + driver="${script_dir}/p6-full-driver.sh" + if [[ -x "$driver" && ! -L "$driver" ]]; then + best_effort_report="${artifact_dir}/p6-driver-cleanup-best-effort-${P6_RUN_ID}.json" + P6_DRIVER_ACTION=cleanup \ + P6_DRIVER_REPORT="$best_effort_report" \ + P6_SECRET_PATTERN_FILE="$P6_WORK_DIR/secret-patterns" \ + P6_ARTIFACTS_DIR="$artifact_dir" \ + "$driver" >/dev/null 2>&1 || true fi } cleanup() { - local project="p6-${P6_RUN_ID}" driver_cleanup_best_effort - if [[ -f "$P6_WORK_DIR/runtime.env" ]]; then - docker compose --project-name "$project" --env-file "$P6_WORK_DIR/runtime.env" -f "$p6_dir/docker-compose.p6.yml" down --remove-orphans >/dev/null 2>&1 || true - fi rm -rf "$P6_WORK_DIR" } @@ -69,60 +68,65 @@ prepare() { preflight() { prepare || return $? local repo - for repo in lab_dev labnow_open labnow_shell labnow_launcher; do p6_assert_repository "$repo" || return $?; done + for repo in lab_dev labnow_open labnow_shell labnow_launcher; do + p6_assert_repository "$repo" || return $? + done p6_assert_images_present || return $? - p6_assert_runtime_paths || return $? + p6_assert_runtime_input || return $? } render() { preflight || return $? - local workspace_image mount workspace state network - workspace_image="$(p6_json_string '.images.openclaw_workspace.ref')" - mount="$(p6_json_string '.paths.runtime_mount')" - workspace="$(p6_json_string '.paths.workspace_root')" - state="$P6_WORK_DIR/openclaw-state" - network="p6-${P6_RUN_ID}" - mkdir -p "$state" - chmod 700 "$state" - { - printf 'P6_OPENCLAW_WORKSPACE_IMAGE=%s\n' "$workspace_image" - printf 'P6_RUNTIME_MOUNT=%s\n' "$mount" - printf 'P6_OPENCLAW_STATE_DIR=%s\n' "$state" - printf 'P6_NETWORK_NAME=%s\n' "$network" - } > "$P6_WORK_DIR/runtime.env" - chmod 600 "$P6_WORK_DIR/runtime.env" - docker compose --project-name "p6-${P6_RUN_ID}" --env-file "$P6_WORK_DIR/runtime.env" -f "$p6_dir/docker-compose.p6.yml" config > "$artifact_dir/p6-render-${P6_RUN_ID}.yml" - chmod 600 "$artifact_dir/p6-render-${P6_RUN_ID}.yml" + local compose_sha metadata + compose_sha="$(p6_sha256 "$p6_dir/docker-compose.runtime.yml")" + metadata="$(jq -c '{schema_version:"p6-render/v1",content_redacted:true,topology:["litellm","shell","jupyterhub","launcher","workspace"],workspace_creation:"live_dockerspawner",support:["postgres","redis","user-center","tls-gateway"],images,local_only_images,support_images}' "$P6_INPUT_FILE")" + jq -n --arg run_id "$P6_RUN_ID" --arg compose_sha256 "$compose_sha" --argjson metadata "$metadata" \ + '{run_id:$run_id,compose_sha256:$compose_sha256} + $metadata' > "$artifact_dir/p6-render-${P6_RUN_ID}.json" + chmod 600 "$artifact_dir/p6-render-${P6_RUN_ID}.json" } run_driver() { local action="$1" driver driver_report pattern_file - driver="$(p6_json_string '.driver')" + driver="${script_dir}/p6-full-driver.sh" [[ -x "$driver" && ! -L "$driver" ]] || p6_die "GOLDEN_DRIVER_UNAVAILABLE" 76 - driver_report="$P6_WORK_DIR/driver-report.json" + driver_report="${artifact_dir}/p6-driver-${action}-${P6_RUN_ID}.json" + if [[ "$action" == cleanup && -f "$driver_report" ]]; then + driver_report="${artifact_dir}/p6-driver-cleanup-verify-${P6_RUN_ID}.json" + fi pattern_file="$P6_WORK_DIR/secret-patterns" - P6_DRIVER_ACTION="$action" P6_RUN_ID="$P6_RUN_ID" P6_INPUT_FILE="$P6_INPUT_FILE" P6_DRIVER_REPORT="$driver_report" P6_SECRET_PATTERN_FILE="$pattern_file" "$driver" + rm -f "$driver_report" + P6_DRIVER_ACTION="$action" \ + P6_DRIVER_REPORT="$driver_report" \ + P6_SECRET_PATTERN_FILE="$pattern_file" \ + P6_ARTIFACTS_DIR="$artifact_dir" \ + "$driver" || return $? + chmod 600 "$driver_report" case "$action" in provision) - jq -e ' + jq -e --arg run "$P6_RUN_ID" --arg input_sha "$(p6_sha256 "$P6_INPUT_FILE")" ' .schema_version == "p6-driver-provision/v1" and .result == "passed" and .content_redacted == true - and (.topology | type == "object") - and (.topology | [.litellm,.shell,.jupyterhub,.launcher,.workspace] | all(. == "started")) + and .run_id == $run and .input_sha256 == $input_sha + and (.topology | [.litellm,.shell,.jupyterhub,.launcher] | all(. == "started")) + and .topology.workspace == "deferred_to_golden" + and .isolation.network == "run_scoped" and .isolation.volumes == "run_scoped" ' "$driver_report" >/dev/null || p6_die "TOPOLOGY_PROVISION_REPORT_INVALID" 77 ;; golden) - jq -e --arg patterns "$pattern_file" ' - type == "object" and .schema_version == "p6-driver-report/v1" - and .result == "passed" and .content_redacted == true - and (.checks | type == "object") and (.secret_pattern_file == $patterns) + jq -e --arg patterns "$pattern_file" --arg run "$P6_RUN_ID" --arg input_sha "$(p6_sha256 "$P6_INPUT_FILE")" ' + .schema_version == "p6-driver-report/v1" and .result == "passed" and .content_redacted == true + and .run_id == $run and .input_sha256 == $input_sha + and .secret_pattern_file == $patterns + and .checks.console_ui == "reused_verified_evidence" + and ([.checks.test_resource_provision,.checks.binding_payload,.checks.jupyterhub_dockerspawner,.checks.launcher_claim_activate_release,.checks.openclaw_apply_probe_readiness,.checks.chat,.checks.stream,.checks.tool,.checks.usage,.checks.owner_negative,.checks.prompt_response_absent,.checks.revoke,.checks.generation_restart,.checks.late_release,.checks.delete,.checks.zero_active_leases] | all(. == "passed")) and (.scan_roots | type == "array" and length >= 1 and all(.[]; type == "string" and startswith("/"))) + and (.product_report_sha256 | test("^[0-9a-f]{64}$")) ' "$driver_report" >/dev/null || p6_die "GOLDEN_DRIVER_REPORT_INVALID" 77 - jq -e '.checks | [.console_ui,.binding_payload,.jupyterhub_dockerspawner,.launcher_claim_activate_release,.openclaw_apply_probe_readiness,.chat,.stream,.tool,.usage,.owner_negative,.prompt_response_absent,.revoke,.generation_restart,.late_release,.delete,.zero_active_leases,.cleanup] | all(. == "passed")' "$driver_report" >/dev/null || p6_die "GOLDEN_DRIVER_CHECK_FAILED" 77 ;; cleanup) - jq -e ' + jq -e --arg run "$P6_RUN_ID" --arg input_sha "$(p6_sha256 "$P6_INPUT_FILE")" ' .schema_version == "p6-driver-cleanup/v1" and .result == "passed" and .content_redacted == true - and (.resources | [.litellm,.shell,.jupyterhub,.launcher,.workspace,.runtime_material,.temporary_files,.processes] | all(. == "absent")) + and .run_id == $run and .input_sha256 == $input_sha + and (.resources | [.litellm,.shell,.jupyterhub,.launcher,.workspace,.runtime_material,.temporary_files,.processes,.network,.volumes] | all(. == "absent")) ' "$driver_report" >/dev/null || p6_die "TOPOLOGY_CLEANUP_REPORT_INVALID" 77 ;; esac @@ -136,26 +140,29 @@ case "$action" in if preflight; then p6_write_report "$report" passed completed; else p6_write_report "$report" failed precondition_failed "preflight"; exit 1; fi ;; render) - if render; then p6_write_report "$report" passed completed; else p6_write_report "$report" failed precondition_failed "render"; cleanup; exit 1; fi + if render; then p6_write_report "$report" passed completed "" "$(jq -n --arg path "$artifact_dir/p6-render-${P6_RUN_ID}.json" --arg sha "$(p6_sha256 "$artifact_dir/p6-render-${P6_RUN_ID}.json")" '{render:{path:$path,sha256:$sha}}')"; else p6_write_report "$report" failed precondition_failed "render"; cleanup; exit 1; fi ;; golden) if ! render; then p6_write_report "$report" failed precondition_failed "render"; cleanup; exit 1; fi trap cleanup EXIT if ! run_driver provision; then p6_write_report "$report" failed topology_provision_failed "driver"; exit 1; fi if ! run_driver golden; then p6_write_report "$report" failed golden_chain_failed "driver"; exit 1; fi - scan_roots=("$P6_WORK_DIR") - while IFS= read -r scan_root; do + scan_roots=() + while IFS= read -r scan_root; do scan_roots+=("$scan_root"); done < <(jq -r '.scan_roots[]' "${artifact_dir}/p6-driver-golden-${P6_RUN_ID}.json") + scan_roots+=("${artifact_dir}/p6-driver-golden-${P6_RUN_ID}.json") + for scan_root in "${scan_roots[@]}"; do [[ -e "$scan_root" && ! -L "$scan_root" ]] || { p6_write_report "$report" failed security_scan_failed "scan_root"; exit 1; } - scan_roots+=("$scan_root") - done < <(jq -r '.scan_roots[]' "$P6_WORK_DIR/driver-report.json") - if ! p6_security_scan "$P6_WORK_DIR/secret-patterns" "${scan_roots[@]}"; then p6_write_report "$report" failed security_scan_failed "secret_scan"; exit 1; fi - rm -f "$P6_WORK_DIR/secret-patterns" + done + if ! p6_security_scan "$P6_WORK_DIR/secret-patterns" "${scan_roots[@]}"; then + p6_write_report "$report" failed security_scan_failed "secret_scan" + exit 1 + fi if ! run_driver cleanup; then p6_write_report "$report" failed cleanup_failed "driver"; exit 1; fi - p6_write_report "$report" passed completed + p6_write_report "$report" passed completed "" "$(p6_stage_reports_json "$artifact_dir" "$P6_RUN_ID")" ;; cleanup) if ! preflight || ! run_driver cleanup; then p6_write_report "$report" failed cleanup_failed "driver"; cleanup; exit 1; fi cleanup - p6_write_report "$report" passed completed + p6_write_report "$report" passed completed "" "$(p6_stage_reports_json "$artifact_dir" "$P6_RUN_ID")" ;; esac diff --git a/docker_openclaw/p6/scripts/p6-user-center.py b/docker_openclaw/p6/scripts/p6-user-center.py new file mode 100755 index 0000000..64ddd5f --- /dev/null +++ b/docker_openclaw/p6/scripts/p6-user-center.py @@ -0,0 +1,58 @@ +#!/usr/bin/env python3 +"""Run-scoped User Center fixture for P6 cookie identity and entitlement calls.""" + +from __future__ import annotations + +import json +import os +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from http.cookies import SimpleCookie + + +OWNER_A = os.environ.get("P6_OWNER_A", "") +OWNER_B = os.environ.get("P6_OWNER_B", "") + + +class Handler(BaseHTTPRequestHandler): + def log_message(self, _format: str, *_args: object) -> None: + return + + def _owner(self) -> str | None: + cookie = SimpleCookie() + cookie.load(self.headers.get("Cookie", "")) + selector = cookie.get("p6_owner") + if selector and selector.value == "a": + return OWNER_A + if selector and selector.value == "b": + return OWNER_B + return None + + def _json(self, status: int, value: object) -> None: + payload = json.dumps(value, separators=(",", ":")).encode("utf-8") + self.send_response(status) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(payload))) + self.send_header("Cache-Control", "no-store") + self.end_headers() + self.wfile.write(payload) + + def do_GET(self) -> None: # noqa: N802 + if self.path == "/health": + self._json(200, {"status": "ok"}) + return + owner = self._owner() + if not owner: + self._json(401, {"code": "UNAUTHENTICATED"}) + return + if self.path.startswith("/ucenter/api/userInfo"): + self._json(200, {"code": "SUCCESS", "data": {"id": owner, "name": owner, "roles": ["pro"]}}) + return + if self.path.startswith("/ucenter/api/subscription"): + self._json(200, {"code": "SUCCESS", "data": []}) + return + self._json(404, {"code": "NOT_FOUND"}) + + +if not OWNER_A or not OWNER_B: + raise SystemExit("P6 owner configuration is required") +ThreadingHTTPServer(("0.0.0.0", 8080), Handler).serve_forever() diff --git a/docker_openclaw/p6/scripts/test-p6-compose-render.sh b/docker_openclaw/p6/scripts/test-p6-compose-render.sh index bc85447..ac39096 100755 --- a/docker_openclaw/p6/scripts/test-p6-compose-render.sh +++ b/docker_openclaw/p6/scripts/test-p6-compose-render.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash -# Render the P6 Compose topology with only non-sensitive fixture values. This -# validates interpolation and confirms the P6 service has no token setting. +# Render the complete P6 topology with non-sensitive fixtures. This validates +# interpolation only; it does not start a container or claim runtime evidence. set -euo pipefail script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" @@ -8,16 +8,57 @@ p6_dir="$(cd "${script_dir}/.." && pwd)" tmpdir="$(mktemp -d "${TMPDIR:-/tmp}/p6-compose.XXXXXX")" chmod 700 "$tmpdir" trap 'rm -rf "$tmpdir"' EXIT -mkdir -p "$tmpdir/runtime" "$tmpdir/state" -touch "$tmpdir/adapter" +mkdir -p "$tmpdir/workspace" "$tmpdir/launcher-data" +for file in launcher.env shell.env litellm.env model-access.json ca.pem ca.key nginx.conf user-center.py migration.sql config.yaml config.migrate.yaml start-litellm.sh migration.py litellm-db redis shell-db app.conf; do + : > "$tmpdir/$file" +done +chmod 600 "$tmpdir"/*.env "$tmpdir/model-access.json" "$tmpdir/ca.key" "$tmpdir/litellm-db" "$tmpdir/redis" "$tmpdir/shell-db" + +env_file="$tmpdir/runtime.env" printf '%s\n' \ - 'P6_OPENCLAW_WORKSPACE_IMAGE=quay.io/labnow/labnow-open:che-563-openclaw-product-closure-local' \ - "P6_RUNTIME_MOUNT=$tmpdir/runtime" \ - "P6_OPENCLAW_STATE_DIR=$tmpdir/state" \ - 'P6_NETWORK_NAME=p6-render-fixture' > "$tmpdir/runtime.env" -chmod 600 "$tmpdir/runtime.env" -docker compose --project-name p6-render-fixture --env-file "$tmpdir/runtime.env" -f "$p6_dir/docker-compose.p6.yml" config > "$tmpdir/rendered.yml" -rg -q 'pull_policy: never' "$tmpdir/rendered.yml" -rg -q 'OPENCLAW_USE_TRUSTED_PROXY_AUTH: "true"' "$tmpdir/rendered.yml" -! rg -q 'OPENCLAW_GATEWAY_TOKEN' "$tmpdir/rendered.yml" -echo 'PASS P6 Compose rendering: fixed image input and token-free internal gateway.' + 'P6_RUNTIME_NETWORK=p6-render-fixture' \ + 'P6_LAUNCHER_CONTAINER=p6-launcher-fixture' \ + 'P6_SHELL_CONTAINER=p6-shell-fixture' \ + 'P6_LITELLM_CONTAINER=p6-litellm-fixture' \ + 'P6_SHELL_POSTGRES_CONTAINER=p6-shell-pg-fixture' \ + 'P6_LITELLM_POSTGRES_CONTAINER=p6-litellm-pg-fixture' \ + "P6_WORKSPACE_ROOT=$tmpdir/workspace" \ + "P6_LAUNCHER_DATA_DIR=$tmpdir/launcher-data" \ + "P6_LAUNCHER_APP_CONF=$tmpdir/app.conf" \ + "P6_LAUNCHER_ENV_FILE=$tmpdir/launcher.env" \ + "P6_SHELL_ENV_FILE=$tmpdir/shell.env" \ + "P6_LITELLM_ENV_FILE=$tmpdir/litellm.env" \ + "P6_MODEL_ACCESS_CONFIG=$tmpdir/model-access.json" \ + "P6_CA_CERT=$tmpdir/ca.pem" \ + "P6_CA_KEY=$tmpdir/ca.key" \ + "P6_NGINX_CONFIG=$tmpdir/nginx.conf" \ + "P6_USER_CENTER_SCRIPT=$tmpdir/user-center.py" \ + "P6_SHELL_MIGRATION=$tmpdir/migration.sql" \ + "P6_LITELLM_CONFIG=$tmpdir/config.yaml" \ + "P6_LITELLM_MIGRATE_CONFIG=$tmpdir/config.migrate.yaml" \ + "P6_LITELLM_START_SCRIPT=$tmpdir/start-litellm.sh" \ + "P6_LITELLM_MIGRATION_SCRIPT=$tmpdir/migration.py" \ + 'P6_LITELLM_IMAGE=quay.io/labnow/litellm:1.97.0-ead62528e607' \ + 'P6_WORKSPACE_IMAGE=quay.io/labnow/labnow-open:che-563-openclaw-product-closure-local' \ + 'P6_LAUNCHER_IMAGE=quay.io/labnow/labnow-launcher:che-563-openclaw-product-closure-local' \ + 'P6_SHELL_IMAGE=quay.io/labnow/labnow-shell:che-563-openclaw-product-closure-local' \ + 'P6_POSTGRES_IMAGE=postgres:17-alpine@sha256:742f40ea20b9ff2ff31db5458d127452988a2164df9e17441e191f3b72252193' \ + 'P6_REDIS_IMAGE=redis:7.4-alpine@sha256:e7723ff73d963f5cc6d9c4643ea3d989527a402a319239054e9472a7fb9219a2' \ + 'P6_NGINX_IMAGE=nginx:alpine@sha256:2f07d83bf561b506400dc183b1b2003803e39efbd22451f848adaba14d28c7c7' \ + "P6_LITELLM_POSTGRES_PASSWORD_FILE=$tmpdir/litellm-db" \ + "P6_REDIS_PASSWORD_FILE=$tmpdir/redis" \ + "P6_SHELL_POSTGRES_PASSWORD_FILE=$tmpdir/shell-db" \ + 'P6_OWNER_A=p6user-fixture' \ + 'P6_OWNER_B=p6other-fixture' > "$env_file" +chmod 600 "$env_file" + +rendered="$tmpdir/rendered.yml" +docker compose --project-name p6-render-fixture --env-file "$env_file" -f "$p6_dir/docker-compose.runtime.yml" config > "$rendered" +for service in litellm-postgres litellm-redis litellm-migrate litellm litellm-gateway shell-postgres shell-migrate user-center shell launcher; do + rg -q "^ ${service}:" "$rendered" +done +rg -q 'pull_policy: never' "$rendered" +rg -q 'service_completed_successfully' "$rendered" +! rg -q 'openclaw-workspace:' "$rendered" +! rg -q 'OPENCLAW_GATEWAY_TOKEN|:latest' "$rendered" +echo 'PASS P6 Compose rendering: fixed five-component topology and run-scoped support services.' diff --git a/docker_openclaw/p6/scripts/test-p6-driver-flow.sh b/docker_openclaw/p6/scripts/test-p6-driver-flow.sh new file mode 100755 index 0000000..b6fa617 --- /dev/null +++ b/docker_openclaw/p6/scripts/test-p6-driver-flow.sh @@ -0,0 +1,248 @@ +#!/usr/bin/env bash +# Deterministic orchestration/evidence test. It uses real temporary Git repos +# and a fake Docker/product executor. It never claims a product runtime passed. +set -euo pipefail + +script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +p6_dir="$(cd "${script_dir}/.." && pwd)" +tmpdir="$(mktemp -d "${TMPDIR:-/tmp}/p6-driver-flow.XXXXXX")" +chmod 700 "$tmpdir" +trap 'rm -rf "$tmpdir"' EXIT +cp -R "$p6_dir" "$tmpdir/p6" +test_scripts="$tmpdir/p6/scripts" +mkdir -p "$tmpdir/bin" "$tmpdir/artifacts" + +python3 - "$p6_dir/scripts/p6-product-chain.py" <<'PY' +import importlib.util +import sys + +spec = importlib.util.spec_from_file_location("p6_product_chain", sys.argv[1]) +module = importlib.util.module_from_spec(spec) +assert spec.loader is not None +spec.loader.exec_module(module) + +from datetime import datetime, timezone +assert module.usage_time(datetime(2026, 8, 10, 12, 34, 56, 123456, tzinfo=timezone.utc)) == "2026-08-10T12:34:56.123Z" + +commands = [] +def fake_command(args, **kwargs): + commands.append(args) + return '{"type":"model.completed"}\n{"type":"session.ended"}\n' + +module.command = fake_command +trajectory, _ = module.trajectory({"workspace_container": "fixture-workspace"}, "fixture-session") +assert trajectory["model_completed_count"] == 1 +assert trajectory["session_ended_count"] == 1 +assert commands == [[ + "docker", "exec", "fixture-workspace", + "cat", "/opt/openclaw/data/agents/main/sessions/fixture-session.trajectory.jsonl", +]] + +calls = [] +responses = iter([ + (200, {"servers": {"workspace-1": {"ready": True, "pending": None}}}), + (200, {"servers": {}}), +]) + +def fake_http_json(method, url, **kwargs): + calls.append((method, url, kwargs)) + return next(responses) + +module.http_json = fake_http_json +running = module.wait_hub_server("http://hub.invalid", "fixture-token", "owner a", "workspace-1", running=True) +stopped = module.wait_hub_server("http://hub.invalid", "fixture-token", "owner a", "workspace-1", running=False) +assert running["ready"] is True +assert stopped == {} +assert [call[1] for call in calls] == [ + "http://hub.invalid/users/owner%20a?include_stopped_servers=true", + "http://hub.invalid/users/owner%20a?include_stopped_servers=true", +] + +module.psql = lambda *_args, **_kwargs: "fixture-key-id" +module.shell_headers = lambda *_args, **_kwargs: {} +module.time.sleep = lambda *_args, **_kwargs: None + +ticks = iter([0, 1, 46]) +module.time.monotonic = lambda: next(ticks) +module.http_json = lambda *_args, **_kwargs: (503, {"code": "fixture"}) +try: + module.usage_check("http://shell.invalid", {"owner_a": "owner-a", "server_name": "workspace-1", "run_id": "fixture"}, "model-1", "2026-08-10T00:00:00Z", "2026-08-11T00:00:00Z") +except module.DriverError as exc: + assert str(exc) == "USAGE_QUERY_FAILED" +else: + raise AssertionError("usage_check must distinguish a failed query from an empty successful query") + +ticks = iter([0, 1]) +module.time.monotonic = lambda: next(ticks) +module.http_json = lambda *_args, **_kwargs: (400, {"code": "INVALID_REQUEST"}) +try: + module.usage_check("http://shell.invalid", {"owner_a": "owner-a", "server_name": "workspace-1", "run_id": "fixture"}, "model-1", "2026-08-10T00:00:00.000Z", "2026-08-11T00:00:00.000Z") +except module.DriverError as exc: + assert str(exc) == "USAGE_QUERY_REJECTED" +else: + raise AssertionError("usage_check must fail fast when Shell rejects its filter") + +ticks = iter([0, 1, 46]) +module.time.monotonic = lambda: next(ticks) +module.http_json = lambda *_args, **_kwargs: (200, {"data": []}) +try: + module.usage_check("http://shell.invalid", {"owner_a": "owner-a", "server_name": "workspace-1", "run_id": "fixture"}, "model-1", "2026-08-10T00:00:00Z", "2026-08-11T00:00:00Z") +except module.DriverError as exc: + assert str(exc) == "USAGE_NOT_OBSERVED" +else: + raise AssertionError("usage_check must preserve the successful-but-empty result") +PY + +init_repo() { + local path="$1" + mkdir -p "$path" + git -C "$path" init -q + git -C "$path" config user.name 'P6 Fixture' + git -C "$path" config user.email 'p6-fixture@example.invalid' +} + +lab_dev="$tmpdir/lab-dev" +open_repo="$tmpdir/labnow-open" +shell_repo="$tmpdir/labnow-shell" +launcher_repo="$tmpdir/labnow-launcher" +for repo in "$lab_dev" "$open_repo" "$shell_repo" "$launcher_repo"; do init_repo "$repo"; done + +mkdir -p "$lab_dev/docker_litellm/demo" "$lab_dev/docker_litellm/work" +touch "$lab_dev/docker_litellm/demo/config.yaml" "$lab_dev/docker_litellm/demo/config.migrate.yaml" "$lab_dev/docker_litellm/work/start-litellm.sh" "$lab_dev/docker_litellm/work/run-migration-locked.py" +printf 'base\n' > "$lab_dev/fixture.txt" +git -C "$lab_dev" add . +git -C "$lab_dev" commit -qm 'fixture base' +git -C "$lab_dev" branch -M dev/che-563-openclaw-product-closure +lab_dev_base="$(git -C "$lab_dev" rev-parse HEAD)" +printf 'review snapshot\n' >> "$lab_dev/fixture.txt" +lab_dev_diff="$(git -C "$lab_dev" diff --binary --full-index --no-ext-diff "$lab_dev_base" -- | shasum -a 256 | awk '{print $1}')" + +printf 'open\n' > "$open_repo/fixture.txt" +git -C "$open_repo" add . +git -C "$open_repo" commit -qm 'open fixture' +open_commit="$(git -C "$open_repo" rev-parse HEAD)" + +mkdir -p "$shell_repo/web/apps/console/src/lib/model-access/migrations" +printf 'SELECT 1;\n' > "$shell_repo/web/apps/console/src/lib/model-access/migrations/001_initial.sql" +git -C "$shell_repo" add . +git -C "$shell_repo" commit -qm 'shell fixture' +shell_commit="$(git -C "$shell_repo" rev-parse HEAD)" + +mkdir -p "$launcher_repo/src/labnow-launcher/resource/config" +printf 'service { port = 8000 }\nlauncher { dir_usr_workspace = "/tmp" }\nmodel_access { trusted_config_file = "/tmp/model-access.json" }\ndocker_spawner { network_name = "fixture" prefix = "fixture" environment = {} read_only_volumes = {} }\n' > "$launcher_repo/src/labnow-launcher/resource/config/app.conf" +git -C "$launcher_repo" add . +git -C "$launcher_repo" commit -qm 'launcher fixture' +launcher_commit="$(git -C "$launcher_repo" rev-parse HEAD)" + +# The copied validator keeps production constants. Replace only those constants +# inside the disposable copy so this test can use genuine temporary commits. +perl -pi -e "s/940325578bae9905673965d6dc489130ab4b6a46/$lab_dev_base/g; s/1b4562899e03eacdee5a86eb55b47d5e12117ee8/$lab_dev_base/g; s/21019e0c24dc7b51747c2bef3cd90f5d259be839/$open_commit/g; s/5c9411dfd3c4d7b1e606c0d9dc0c5e62313bc376/$shell_commit/g; s/c84edea3e051d561f28d9f99235563cf491aaeb2/$launcher_commit/g" "$test_scripts/p6-lib.sh" + +p1_env="$tmpdir/p1.env" +printf '%s\n' \ + 'LITELLM_MASTER_KEY=fixture-master-not-valid' \ + 'POSTGRES_DB=fixture' \ + 'POSTGRES_USER=fixture' \ + 'POSTGRES_PASSWORD=fixture-db-not-valid' \ + 'REDIS_PASSWORD=fixture-redis-not-valid' \ + 'UPSTREAM_API_KEY=fixture-upstream-not-valid' \ + 'UPSTREAM_BASE_URL=https://example.invalid' \ + 'UPSTREAM_MODEL=fixture-model' > "$p1_env" +chmod 600 "$p1_env" + +input="$tmpdir/input.json" +jq \ + --arg lab_dev "$lab_dev" --arg base "$lab_dev_base" --arg diff "$lab_dev_diff" \ + --arg open "$open_repo" --arg open_commit "$open_commit" \ + --arg shell "$shell_repo" --arg shell_commit "$shell_commit" \ + --arg launcher "$launcher_repo" --arg launcher_commit "$launcher_commit" \ + --arg p1 "$p1_env" ' + .repositories.lab_dev.path=$lab_dev + | .repositories.lab_dev.phase_base_commit=$base + | .repositories.lab_dev.head_commit=$base + | .repositories.lab_dev.tracked_diff_sha256=$diff + | .repositories.lab_dev.changed_files=["fixture.txt"] + | .repositories.labnow_open.path=$open + | .repositories.labnow_open.commit=$open_commit + | .repositories.labnow_shell.path=$shell + | .repositories.labnow_shell.commit=$shell_commit + | .repositories.labnow_launcher.path=$launcher + | .repositories.labnow_launcher.commit=$launcher_commit + | .images.openclaw_workspace.source_commit=$open_commit + | .local_only_images.shell.source_commit=$shell_commit + | .local_only_images.launcher.source_commit=$launcher_commit + | .runtime.p1_env_file=$p1 + ' "$tmpdir/p6/p6-inputs.example.json" > "$input" +chmod 600 "$input" + +cat > "$tmpdir/bin/docker" <<'EOF' +#!/usr/bin/env bash +set -euo pipefail +if [[ "${1:-}" == image && "${2:-}" == inspect ]]; then + ref="${*: -1}" + if [[ "$*" == *'{{.Id}}'* ]]; then + case "$ref" in + *litellm*) echo sha256:a2e115874c21b829bd052b18fc85be2f9217fb8244b82812c4ebc6e36f9824d1 ;; + *labnow-open:*) echo sha256:c9c6a45637521cbbaeacea57fbb128696066fd91c5dff4521555f1bd5211f244 ;; + *openclaw@*) echo sha256:edc85cc2068f5ec0df470f7d06daa0a4fbd78ef5ad6cf5b48f58381da839dd12 ;; + *launcher*) echo sha256:6f9732fda8b86d9bfe4596e848025cc38448da4b17dfea8520e046a32b32e61f ;; + *shell*) echo sha256:d7ed71cf58eddf72642d61d4442a0820632060fac9f4eb611b28062b7f38c54d ;; + *postgres*) echo sha256:742f40ea20b9ff2ff31db5458d127452988a2164df9e17441e191f3b72252193 ;; + *redis*) echo sha256:e7723ff73d963f5cc6d9c4643ea3d989527a402a319239054e9472a7fb9219a2 ;; + *nginx*) echo sha256:2f07d83bf561b506400dc183b1b2003803e39efbd22451f848adaba14d28c7c7 ;; + *) exit 64 ;; + esac + else + case "$ref" in + *litellm*) echo quay.io/labnow/litellm@sha256:a2e115874c21b829bd052b18fc85be2f9217fb8244b82812c4ebc6e36f9824d1 ;; + *labnow-open:*) echo quay.io/labnow/labnow-open@sha256:c9c6a45637521cbbaeacea57fbb128696066fd91c5dff4521555f1bd5211f244 ;; + *openclaw@*) echo quay.io/labnow/openclaw@sha256:edc85cc2068f5ec0df470f7d06daa0a4fbd78ef5ad6cf5b48f58381da839dd12 ;; + *launcher*) echo quay.io/labnow/labnow-launcher@sha256:6f9732fda8b86d9bfe4596e848025cc38448da4b17dfea8520e046a32b32e61f ;; + *shell*) echo quay.io/labnow/labnow-shell@sha256:d7ed71cf58eddf72642d61d4442a0820632060fac9f4eb611b28062b7f38c54d ;; + *postgres*) echo postgres@sha256:742f40ea20b9ff2ff31db5458d127452988a2164df9e17441e191f3b72252193 ;; + *redis*) echo redis@sha256:e7723ff73d963f5cc6d9c4643ea3d989527a402a319239054e9472a7fb9219a2 ;; + *nginx*) echo nginx@sha256:2f07d83bf561b506400dc183b1b2003803e39efbd22451f848adaba14d28c7c7 ;; + *) exit 64 ;; + esac + fi + exit 0 +fi +if [[ "${1:-}" == compose ]]; then exit 0; fi +if [[ "${1:-}" == container && "${2:-}" == inspect ]]; then exit 1; fi +if [[ "${1:-}" == container && "${2:-}" == ls ]]; then exit 0; fi +if [[ "${1:-}" == volume && "${2:-}" == ls ]]; then exit 0; fi +if [[ "${1:-}" == network && "${2:-}" == inspect ]]; then exit 1; fi +if [[ "${1:-}" == rm ]]; then exit 0; fi +exit 64 +EOF +chmod 700 "$tmpdir/bin/docker" + +cat > "$test_scripts/p6-product-chain.py" <<'EOF' +#!/usr/bin/env python3 +import json, os +from pathlib import Path +surface = Path(os.environ['P6_WORK_DIR']) / 'surfaces' +surface.mkdir(mode=0o700, parents=True, exist_ok=True) +(surface / 'fixture.txt').write_text('redacted fixture surface\n') +os.chmod(surface / 'fixture.txt', 0o600) +Path(os.environ['P6_SECRET_PATTERN_FILE']).write_text('p6-fixture-secret-not-present\n') +os.chmod(os.environ['P6_SECRET_PATTERN_FILE'], 0o400) +checks = {name:'passed' for name in ['test_resource_provision','binding_payload','jupyterhub_dockerspawner','launcher_claim_activate_release','openclaw_apply_probe_readiness','chat','stream','tool','usage','owner_negative','prompt_response_absent','revoke','generation_restart','late_release','delete','zero_active_leases']} +checks['console_ui'] = 'reused_verified_evidence' +report = {'schema_version':'p6-product-chain-report/v1','result':'passed','content_redacted':True,'checks':checks,'binding':{},'runtime':{},'data_plane':{},'usage':{},'lifecycle':{},'scan_roots':[str(surface)]} +Path(os.environ['P6_PRODUCT_REPORT_FILE']).write_text(json.dumps(report, separators=(',',':'))+'\n') +os.chmod(os.environ['P6_PRODUCT_REPORT_FILE'], 0o600) +EOF +chmod 700 "$test_scripts/p6-product-chain.py" + +run_id="p6-$(python3 -c 'print("1" * 32)')" +run_env=(env "PATH=$tmpdir/bin:$PATH" "P6_RUN_ID=$run_id" "P6_ARTIFACTS_DIR=$tmpdir/artifacts" "P6_WORK_DIR=$tmpdir/work") +"${run_env[@]}" "$test_scripts/p6-runner.sh" --input "$input" --preflight >/dev/null +"${run_env[@]}" "$test_scripts/p6-runner.sh" --input "$input" --golden >/dev/null +"${run_env[@]}" "$test_scripts/p6-runner.sh" --input "$input" --cleanup >/dev/null +"$test_scripts/p6-aggregate.sh" --artifacts "$tmpdir/artifacts" --run-id "$run_id" >/dev/null + +for action in provision golden cleanup; do test -s "$tmpdir/artifacts/p6-driver-${action}-${run_id}.json"; done +test -s "$tmpdir/artifacts/p6-final-${run_id}.json" +test ! -d "$tmpdir/work" +echo 'PASS P6 driver flow: review_snapshot, redacted stage evidence, hash binding and cleanup are deterministic.' diff --git a/docker_openclaw/p6/scripts/test-p6-gates.sh b/docker_openclaw/p6/scripts/test-p6-gates.sh index 7593856..e2b0944 100755 --- a/docker_openclaw/p6/scripts/test-p6-gates.sh +++ b/docker_openclaw/p6/scripts/test-p6-gates.sh @@ -4,41 +4,59 @@ set -euo pipefail script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +p6_dir="$(cd "${script_dir}/.." && pwd)" runner="$script_dir/p6-runner.sh" tmpdir="$(mktemp -d "${TMPDIR:-/tmp}/p6-gates.XXXXXX")" chmod 700 "$tmpdir" trap 'rm -rf "$tmpdir"' EXIT input="$tmpdir/input.json" -jq -n \ - --arg contract_sha 'd289dff9bcaa3d28035c5ed2e56b806f4b3b37fdca3159352d22f0c03942e202' \ - --arg control '2eb71d7590739df3de8db2f8cf9098154a397f0b' \ - --arg lab_dev '940325578bae9905673965d6dc489130ab4b6a46' \ - --arg open 'dfac9767fd6cdd4706ac4cd6917defcafd1c6eb8' \ - --arg shell 'eb0e6f90182e5d59174ea9edb7cb71edeaa7a47f' \ - --arg launcher 'c64f5fbabc587e26394a486ef9ae12558234f646' \ - '{schema_version:"p6-inputs/v1",contract_version:"v1alpha1",contract_bundle_sha256:$contract_sha,control_commit:$control,review_policy_commit:$control, - repositories:{lab_dev:{path:"/tmp/lab-dev",commit:$lab_dev},labnow_open:{path:"/tmp/labnow-open",commit:$open},labnow_shell:{path:"/tmp/labnow-shell",commit:$shell},labnow_launcher:{path:"/tmp/labnow-launcher",commit:$launcher}}, - images:{litellm:{ref:"quay.io/labnow/litellm:1.97.0-ead62528e607",image_id:("sha256:" + ("a" * 64)),provenance:"repo_digest",repo_digest:("quay.io/labnow/litellm@sha256:" + ("a" * 64))},openclaw_base:{ref:("quay.io/labnow/openclaw@sha256:" + ("b" * 64)),image_id:("sha256:" + ("b" * 64)),provenance:"repo_digest",repo_digest:("quay.io/labnow/openclaw@sha256:" + ("b" * 64))},openclaw_workspace:{ref:"quay.io/labnow/labnow-open:che-563-openclaw-product-closure-local",image_id:("sha256:" + ("c" * 64)),provenance:"local_build",repo_digest:"absent",source_repository:"labnow_open",source_commit:$open,base_image_digest:("quay.io/labnow/openclaw@sha256:" + ("b" * 64))}}, - paths:{runtime_mount:"/tmp/runtime",workspace_root:"/tmp/workspace"},driver:"/tmp/driver"}' > "$input" +cp "$p6_dir/p6-inputs.example.json" "$input" chmod 600 "$input" +run_id="p6-$(python3 -c 'print("0" * 32)')" +P6_RUN_ID="$run_id" P6_ARTIFACTS_DIR="$tmpdir/artifacts" "$runner" --input "$input" --validate-input >/dev/null -# An example-like but structurally fixed input is accepted without inspecting -# Docker/repositories. Mutable refs and control/contract mismatches must fail. -P6_RUN_ID="p6-$(python3 -c 'print("0" * 32)')" P6_ARTIFACTS_DIR="$tmpdir/artifacts" "$runner" --input "$input" --validate-input >/dev/null -latest="$tmpdir/latest.json" -jq '.images.openclaw_workspace.ref = "quay.io/labnow/labnow-open:latest"' "$input" > "$latest"; chmod 600 "$latest" -if P6_ARTIFACTS_DIR="$tmpdir/artifacts" "$runner" --input "$latest" --validate-input >/dev/null 2>&1; then echo 'accepted latest image' >&2; exit 1; fi -mismatch="$tmpdir/mismatch.json" -jq '.contract_bundle_sha256 = "0000000000000000000000000000000000000000000000000000000000000000"' "$input" > "$mismatch"; chmod 600 "$mismatch" -if P6_ARTIFACTS_DIR="$tmpdir/artifacts" "$runner" --input "$mismatch" --validate-input >/dev/null 2>&1; then echo 'accepted contract mismatch' >&2; exit 1; fi +source "$script_dir/p6-lib.sh" +mkdir -p "$tmpdir/scan-bin" "$tmpdir/scan-root" +printf '#!/usr/bin/env bash\nexit 2\n' > "$tmpdir/scan-bin/rg" +chmod 700 "$tmpdir/scan-bin/rg" +printf 'fixture-pattern-not-present\n' > "$tmpdir/patterns" +chmod 600 "$tmpdir/patterns" +printf 'safe fixture\n' > "$tmpdir/scan-root/value.txt" +if (PATH="$tmpdir/scan-bin:$PATH"; p6_security_scan "$tmpdir/patterns" "$tmpdir/scan-root" >/dev/null 2>&1); then + echo 'accepted failed secret scan as zero-hit' >&2 + exit 1 +fi -# The orchestration source must not fall back to latest or a plaintext gateway -# token, and aggregation must reject absent golden evidence. -rg -q 'P6_OPENCLAW_WORKSPACE_IMAGE' "$script_dir/../docker-compose.p6.yml" -! rg -n 'P6_OPENCLAW_IMAGE|P6_OPENCLAW_ADAPTER' "$script_dir/../docker-compose.p6.yml" "$script_dir/p6-runner.sh" -rg -q 'run_driver provision' "$script_dir/p6-runner.sh" -rg -q 'run_driver cleanup' "$script_dir/p6-runner.sh" -! rg -q 'up -d --wait openclaw-workspace' "$script_dir/p6-runner.sh" -if "$script_dir/p6-aggregate.sh" --artifacts "$tmpdir/artifacts" --run-id "p6-$(python3 -c 'print("0" * 32)')" >/dev/null 2>&1; then echo 'accepted incomplete evidence' >&2; exit 1; fi -echo 'PASS P6 gates: fixed-input mismatches and incomplete evidence fail closed.' +negative() { + local name="$1" filter="$2" candidate + candidate="$tmpdir/${name}.json" + jq "$filter" "$input" > "$candidate" + chmod 600 "$candidate" + if P6_RUN_ID="$run_id" P6_ARTIFACTS_DIR="$tmpdir/artifacts" "$runner" --input "$candidate" --validate-input >/dev/null 2>&1; then + printf 'accepted invalid input: %s\n' "$name" >&2 + exit 1 + fi +} + +negative latest '.images.openclaw_workspace.ref = "quay.io/labnow/labnow-open:latest"' +negative contract_mismatch '.contract_bundle_sha256 = ("0" * 64)' +negative missing_workspace_digest '.images.openclaw_workspace.repo_digest = "absent"' +negative missing_launcher_digest '.local_only_images.launcher.repo_digest = "absent"' +negative unprotected_lab_dev '.repositories.lab_dev.delivery_identity = "commit"' +negative missing_snapshot_files '.repositories.lab_dev.changed_files = []' +negative mutable_support '.support_images.nginx.ref = "nginx:latest"' + +rg -q 'p6-product-chain.py' "$script_dir/p6-full-driver.sh" +rg -q 'review_snapshot' "$script_dir/p6-lib.sh" +rg -q 'tracked_diff_sha256' "$script_dir/p6-lib.sh" +! rg -q 'golden_checks|pattern_command|topology.*compose_file' "$p6_dir/p6-inputs.example.json" +! rg -q 'docker-compose.p6.yml|up -d --wait openclaw-workspace' "$script_dir/p6-runner.sh" "$script_dir/p6-full-driver.sh" +test -x "$script_dir/p6-full-driver.sh" +test -x "$script_dir/p6-prepare-runtime.py" +test -x "$script_dir/p6-product-chain.py" +if "$script_dir/p6-aggregate.sh" --artifacts "$tmpdir/artifacts" --run-id "$run_id" >/dev/null 2>&1; then + echo 'accepted incomplete evidence' >&2 + exit 1 +fi +echo 'PASS P6 gates: review_snapshot, fixed provenance and incomplete evidence fail closed.' From 45c38585a0ca889f6a20aebfdf3b13a01d369ac2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Mon, 10 Aug 2026 21:37:55 +0800 Subject: [PATCH 59/87] =?UTF-8?q?docs:=20=E8=AE=B0=E5=BD=95=20P6=20?= =?UTF-8?q?=E9=BB=84=E9=87=91=E9=93=BE=E9=AA=8C=E6=94=B6=E5=9B=9E=E6=89=A7?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- ...21\351\223\276\351\252\214\346\224\266.md" | 78 +++++++++++++++++++ 1 file changed, 78 insertions(+) create mode 100644 "docker_openclaw/p6/evidence/2026-08-10-P6-CHE-563-\351\273\204\351\207\221\351\223\276\351\252\214\346\224\266.md" diff --git "a/docker_openclaw/p6/evidence/2026-08-10-P6-CHE-563-\351\273\204\351\207\221\351\223\276\351\252\214\346\224\266.md" "b/docker_openclaw/p6/evidence/2026-08-10-P6-CHE-563-\351\273\204\351\207\221\351\223\276\351\252\214\346\224\266.md" new file mode 100644 index 0000000..6435b2e --- /dev/null +++ "b/docker_openclaw/p6/evidence/2026-08-10-P6-CHE-563-\351\273\204\351\207\221\351\223\276\351\252\214\346\224\266.md" @@ -0,0 +1,78 @@ +# CHE-563 / P6 OpenClaw 黄金链验收 + +## 冻结输入 + +- Linear Issue:`CHE-563` +- Phase 分支:`dev/che-563-openclaw-product-closure` +- Phase base:`940325578bae9905673965d6dc489130ab4b6a46` +- 已验证实现提交:`9d78780e9cfb6ac09410c3a21da7de1c2f59a1b4` +- control commit:`2eb71d7590739df3de8db2f8cf9098154a397f0b` +- review policy commit:`680ca92661a08254eb396ab809f478bbdba3510e` +- contract:`v1alpha1 / 0.1.0-rc.1` +- contract bundle SHA-256:`d289dff9bcaa3d28035c5ed2e56b806f4b3b37fdca3159352d22f0c03942e202` +- delivery transport:`local_only` + +## 固定产品组合 + +| 仓库 / 制品 | 准确输入 | +| --- | --- | +| `lab-dev` | 实现提交 `9d78780e9cfb6ac09410c3a21da7de1c2f59a1b4`;验证时以 `HEAD=1b4562899e03eacdee5a86eb55b47d5e12117ee8` 加 review snapshot `85b01ca3ff4ad06584771555d3321c78776cbd6bacd366de5609c237a3006822` 固定,提交后从 Phase base 到实现提交的 binary diff SHA-256 仍为同一值 | +| `labnow-open` | `21019e0c24dc7b51747c2bef3cd90f5d259be839`;本地镜像 `quay.io/labnow/labnow-open@sha256:c9c6a45637521cbbaeacea57fbb128696066fd91c5dff4521555f1bd5211f244` | +| `labnow-shell` | `5c9411dfd3c4d7b1e606c0d9dc0c5e62313bc376`;本地镜像 `quay.io/labnow/labnow-shell@sha256:d7ed71cf58eddf72642d61d4442a0820632060fac9f4eb611b28062b7f38c54d` | +| `labnow-launcher` | `c84edea3e051d561f28d9f99235563cf491aaeb2`;本地镜像 `quay.io/labnow/labnow-launcher@sha256:6f9732fda8b86d9bfe4596e848025cc38448da4b17dfea8520e046a32b32e61f` | +| LiteLLM | `quay.io/labnow/litellm@sha256:a2e115874c21b829bd052b18fc85be2f9217fb8244b82812c4ebc6e36f9824d1` | +| OpenClaw base | `quay.io/labnow/openclaw@sha256:edc85cc2068f5ec0df470f7d06daa0a4fbd78ef5ad6cf5b48f58381da839dd12` | + +所有 `labnow-*` 镜像均为本地构建,未推送、未发布;支持镜像与完整 provenance 由最终聚合报告固定。 + +## 首轮 Review 阻断项闭环 + +| 阻断项 | 结果 | +| --- | --- | +| R1:缺少完整真实产品链与可聚合证据 | 通过;真实 Shell → JupyterHub → Launcher → Workspace → Adapter → LiteLLM 链已完成 | +| R2:Workspace ID 不一致 | 通过;统一使用真实 `server_name`,binding、lease、Workspace 与 usage 查询一致 | +| R3:Launcher 环境变量接受能力 | 通过;固定 Launcher 提交已在真实 claim / activate / release 流程中验证 | +| R4:Workspace RepoDigest 不明确 | 通过;输入门禁固定本地 image ID、synthetic RepoDigest、源仓 commit 与 OpenClaw base digest | + +复审期间仅处理上述阻断项及其直接影响。OpenClaw 默认新 Workspace 的工具 allowlist 收窄为 `exec`;既有显式 allowlist 不变。Shell usage 改用 LiteLLM v2 `key_alias` / `model_group` 分页查询。P6 driver 将 usage 时间格式固定为 UTC 毫秒,避免超过 Shell 接口允许的 3 位小数。 + +## 真实黄金结果 + +- run ID:`p6-9fd9cd2a55a685a9409a459eae58beb2` +- protected input SHA-256:`b1aa535b604ed58047694366c211b544b03e3d91aca09b8b0a59274b7e20d27e` +- final report:`docker_openclaw/p6/artifacts/p6-final-p6-9fd9cd2a55a685a9409a459eae58beb2.json` +- final report SHA-256:`7d358dae1dfc88bdb61eb5a2e0ec53b83e33037981284f93e5df47d2dfc58cf4` +- provision report SHA-256:`03bf931e4c115b169f7bdb32e55adaa2281624f6e937f82ee7bbbff8fc7b817a` +- golden report SHA-256:`9a37a106433ea72cb13d199f5d0a8ceab7f608fd36198041a2f824c40b08f870` +- cleanup report SHA-256:`eb0af6b5c27409ae6978e1c18335b3c655399453e08f39653adb022252a02f5b` +- 结果:`passed`,`content_redacted=true` +- usage:8 条;投影仅含 `timestamp`、`model`、`total_tokens`、`prompt_tokens`、`completion_tokens`、`status` +- 生命周期:generation 1 停止后旧 key 拒绝;generation 2 使用新 key;旧 key 继续拒绝;迟到 generation 1 release 返回 409;delete 后新 key 拒绝;最终 active lease 为 0 +- 安全边界:owner negative 隔离通过;Prompt / Response 正文不保存;未发现 usage 正文持久化表;扫描根与聚合报告通过敏感模式扫描 +- 清理:本次 run 的 LiteLLM、Shell、JupyterHub、Launcher、Workspace、临时材料、进程、网络和卷全部为 `absent` + +## 验证命令 + +以下命令均退出 0: + +```bash +bash -n docker_openclaw/p6/scripts/*.sh +PYTHONDONTWRITEBYTECODE=1 python3 -m py_compile docker_openclaw/p6/scripts/*.py +./docker_openclaw/p6/scripts/test-p6-gates.sh +./docker_openclaw/p6/scripts/test-p6-compose-render.sh +./docker_openclaw/p6/scripts/test-p6-driver-flow.sh +P6_RUN_ID=p6-9fd9cd2a55a685a9409a459eae58beb2 ./docker_openclaw/p6/scripts/p6-runner.sh --input /Users/chengeng/Projects/GitHub/lab-dev/docker_openclaw/p6/p6-inputs.json --preflight +P6_RUN_ID=p6-9fd9cd2a55a685a9409a459eae58beb2 ./docker_openclaw/p6/scripts/p6-runner.sh --input /Users/chengeng/Projects/GitHub/lab-dev/docker_openclaw/p6/p6-inputs.json --golden +P6_RUN_ID=p6-9fd9cd2a55a685a9409a459eae58beb2 ./docker_openclaw/p6/scripts/p6-runner.sh --input /Users/chengeng/Projects/GitHub/lab-dev/docker_openclaw/p6/p6-inputs.json --cleanup +./docker_openclaw/p6/scripts/p6-aggregate.sh --artifacts docker_openclaw/p6/artifacts --run-id p6-9fd9cd2a55a685a9409a459eae58beb2 +git diff --check +``` + +## Handoff + +- `status=ready_for_integration_review` +- `phase_commit=9d78780e9cfb6ac09410c3a21da7de1c2f59a1b4` +- 产品实现、固定组合、真实黄金链和清理证据均已具备,可进入总控有界复审。 +- 本地 artifact 被 `.gitignore` 排除;以本文件记录的 run ID 与 SHA-256 回读,不提交运行材料或任何密钥。 +- 未修改、合并或推送 `main`;未推进任何远端 integration;未 push;未发布镜像;未部署共享或生产环境;未写入明文凭证。 +- 用户既有未跟踪 `.DS_Store`、`docs/`、`hermes-chat-screenshot.png` 未读取、修改、暂存或删除。 From bc3875decbf4e556dc14e2ed65b81def39885b9b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Mon, 10 Aug 2026 23:08:12 +0800 Subject: [PATCH 60/87] =?UTF-8?q?feat(hermes):=20=E5=9B=BA=E5=AE=9A=20P7?= =?UTF-8?q?=20=E5=8F=AF=E5=A4=8D=E7=8E=B0=E6=9E=84=E5=BB=BA=E4=B8=8E?= =?UTF-8?q?=E9=AA=8C=E8=AF=81=E5=85=A5=E5=8F=A3?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .gitignore | 4 + docker_hermes/README.md | 19 +++ docker_hermes/demo/.env.example | 2 +- docker_hermes/demo/docker-compose.yml | 6 +- docker_hermes/hermes.Dockerfile | 38 +++++- docker_hermes/p7/README.md | 46 +++++++ docker_hermes/p7/docker-compose.runtime.yml | 33 +++++ ...72\344\270\216\351\252\214\350\257\201.md" | 80 ++++++++++++ docker_hermes/p7/p7-inputs.example.json | 27 ++++ docker_hermes/p7/scripts/p7-runner.sh | 121 ++++++++++++++++++ docker_hermes/p7/scripts/test-p7-gates.sh | 48 +++++++ 11 files changed, 417 insertions(+), 7 deletions(-) create mode 100644 docker_hermes/p7/README.md create mode 100644 docker_hermes/p7/docker-compose.runtime.yml create mode 100644 "docker_hermes/p7/evidence/2026-08-10-P7-CHE-568-Hermes\346\236\204\345\273\272\344\270\216\351\252\214\350\257\201.md" create mode 100644 docker_hermes/p7/p7-inputs.example.json create mode 100755 docker_hermes/p7/scripts/p7-runner.sh create mode 100755 docker_hermes/p7/scripts/test-p7-gates.sh diff --git a/.gitignore b/.gitignore index a3604ff..1133fe7 100644 --- a/.gitignore +++ b/.gitignore @@ -126,6 +126,10 @@ docker_litellm/demo/artifacts/ docker_openclaw/p6/artifacts/ docker_openclaw/p6/p6-inputs.json docker_openclaw/p6/.p6-work/ +docker_hermes/p7/artifacts/ +docker_hermes/p7/.p7-work/ +docker_hermes/p7/p7-inputs.json +docker_hermes/p7/source/ .venv env/ venv/ diff --git a/docker_hermes/README.md b/docker_hermes/README.md index f3c06ac..7b6629e 100644 --- a/docker_hermes/README.md +++ b/docker_hermes/README.md @@ -2,6 +2,10 @@ `hermes` is a containerized agentic assistant platform based on the [Hermes Agent](https://github.com/nousresearch/hermes-agent) project, built using Node.js and Python runtime stacks. +P7 的验收构建固定 Hermes repository 与 40 位 commit;Dockerfile 不再以移动 +`main` 作为制品输入。默认 standalone Compose 仍服务于本地开发,但只接受已在 +本机存在的明确镜像引用,不会静默 pull `latest`。 + --- ## 1. Port Configuration @@ -48,6 +52,21 @@ source ./tool.sh build_image_no_tag hermes local docker_hermes/hermes.Dockerfile ``` +### P7 可复现构建 + +需要跨仓 Hermes 联调时,使用 P7 固定 tag 和观察到的 Hermes source identity: + +```bash +build_image_no_tag hermes p7-<12hex> docker_hermes/hermes.Dockerfile \ + --build-arg HERMES_SOURCE_REPOSITORY= \ + --build-arg HERMES_SOURCE_COMMIT=<40-hex-commit> +``` + +镜像会记录 `org.opencontainers.image.source` 与 +`org.opencontainers.image.revision`,并在 `/opt/hermes/.labnow-source-*` 保存 +相同的非敏感 provenance。只在本地命名为 `quay.io/labnow/hermes:p7-<12hex>`,不 push。 +完整 P7 的受限输入、静态门禁和跨仓 runner 见 [`p7/README.md`](p7/README.md)。 + ### Start with Docker Compose 1. Copy the sample environment file: diff --git a/docker_hermes/demo/.env.example b/docker_hermes/demo/.env.example index 01eeabc..2b0d56b 100644 --- a/docker_hermes/demo/.env.example +++ b/docker_hermes/demo/.env.example @@ -7,7 +7,7 @@ TZ=Asia/Shanghai # Image to run. Build with REGISTRY_DST=quay.io via tool.sh before starting Compose. # Default local output: quay.io/labnow/hermes:local (no automatic push). -HERMES_IMAGE=quay.io/labnow/hermes +HERMES_IMAGE=quay.io/labnow/hermes:local # Host-side persistent data directory. HERMES_DATA_DIR=../../.data/hermes diff --git a/docker_hermes/demo/docker-compose.yml b/docker_hermes/demo/docker-compose.yml index d04c4d9..cc4302b 100644 --- a/docker_hermes/demo/docker-compose.yml +++ b/docker_hermes/demo/docker-compose.yml @@ -4,8 +4,10 @@ services: hermes: container_name: ${HERMES_CONTAINER_NAME:-svc-hermes} hostname: svc-hermes - image: "${HERMES_IMAGE:-quay.io/labnow/hermes:latest}" - pull_policy: always + image: "${HERMES_IMAGE:?set a fixed local Hermes image}" + # Local development must not silently pull a mutable image. Use an + # explicit Compose action when an operator intentionally needs a pull. + pull_policy: never restart: unless-stopped env_file: [".env.example"] environment: diff --git a/docker_hermes/hermes.Dockerfile b/docker_hermes/hermes.Dockerfile index 1916e81..f25d7bb 100644 --- a/docker_hermes/hermes.Dockerfile +++ b/docker_hermes/hermes.Dockerfile @@ -3,9 +3,19 @@ ARG BASE_NAMESPACE ARG BASE_IMG_BUILD="node" ARG BASE_IMG="base" +ARG HERMES_BUILD_BASE_IMAGE +ARG HERMES_RUNTIME_BASE_IMAGE +# P7: the upstream source is a release input, not a moving branch. Keep the +# repository and commit overridable only so the local runner can bind both to +# its protected input and record the exact provenance. +ARG HERMES_SOURCE_REPOSITORY="https://github.com/nousresearch/hermes-agent.git" +ARG HERMES_SOURCE_COMMIT="1388cd1c0c1800078bfcc92aebd144fbf145fdb4" # --- Building Stage --- -FROM ${BASE_NAMESPACE:+$BASE_NAMESPACE/}${BASE_IMG_BUILD} AS builder +FROM ${HERMES_BUILD_BASE_IMAGE:-${BASE_NAMESPACE:+$BASE_NAMESPACE/}${BASE_IMG_BUILD}} AS builder + +ARG HERMES_SOURCE_REPOSITORY +ARG HERMES_SOURCE_COMMIT # Build-time environment ENV NODE_ENV=development @@ -21,9 +31,18 @@ COPY work /opt/utils/ # Install build-time system dependencies (compilers + native libs needed for Python extensions). # Without these, `uv sync` fails when compiling packages like `matrix-*-crypto`, `cryptography`, or `ffi`-based wheels on cold builds. RUN set -eux \ + && printf 'Acquire::Retries "5";\nAcquire::http::Timeout "30";\n' > /etc/apt/apt.conf.d/80-labnow-retries \ && . /opt/utils/script-utils.sh && install_apt /opt/utils/install_list_hermes.apt \ + && rm -f /etc/apt/apt.conf.d/80-labnow-retries \ ## Clone source (full clone for reproducibility; depth 1 for speed) - && git clone --depth 1 --branch main https://github.com/nousresearch/hermes-agent.git . \ + && test "$(printf '%s' "$HERMES_SOURCE_COMMIT" | wc -c | tr -d ' ')" = 40 \ + && git init . \ + && git remote add origin "$HERMES_SOURCE_REPOSITORY" \ + && git fetch --depth 1 origin "$HERMES_SOURCE_COMMIT" \ + && git checkout --detach FETCH_HEAD \ + && test "$(git rev-parse HEAD)" = "$HERMES_SOURCE_COMMIT" \ + && printf '%s\n' "$HERMES_SOURCE_REPOSITORY" > /opt/hermes/.labnow-source-repository \ + && printf '%s\n' "$HERMES_SOURCE_COMMIT" > /opt/hermes/.labnow-source-commit \ && chmod +x /opt/utils/*.sh && mv /opt/utils/*hermes*.sh /opt/utils/install_list_hermes.apt /opt/utils/supervisord.conf /opt/hermes/ \ ## ---------- hack python-olm for building compatible wheels ---------- && mkdir -pv /opt/hermes/vendor \ @@ -31,7 +50,7 @@ RUN set -eux \ && curl -s https://pypi.org/pypi/python-olm/3.2.16/json \ | jq -r '.urls[] | select(.packagetype=="sdist").url' \ | xargs curl -L -o python-olm-3.2.16.tar.gz \ - && tar xf python-olm-3.2.16.tar.gz && cd python-olm-3.2.16 \ + && python -c 'import tarfile; tarfile.open("python-olm-3.2.16.tar.gz").extractall(path=".", filter="data")' && cd python-olm-3.2.16 \ && sed -i 's/cmake_minimum_required(VERSION [0-9.]*)/cmake_minimum_required(VERSION 3.5)/' libolm/CMakeLists.txt \ && pip wheel . --no-build-isolation -w /tmp/olm/wheels \ && mv /tmp/olm/wheels/*olm*.whl /opt/hermes/vendor/ \ @@ -58,9 +77,18 @@ RUN set -eux \ && printf 'docker\n' > /opt/hermes/.install_method ### --- Runtime Stage --- -FROM ${BASE_NAMESPACE:+$BASE_NAMESPACE/}${BASE_IMG} +FROM ${HERMES_RUNTIME_BASE_IMAGE:-${BASE_NAMESPACE:+$BASE_NAMESPACE/}${BASE_IMG}} + +ARG HERMES_SOURCE_REPOSITORY +ARG HERMES_SOURCE_COMMIT +ARG HERMES_BUILD_BASE_IMAGE +ARG HERMES_RUNTIME_BASE_IMAGE LABEL maintainer="postmaster@labnow.ai" +LABEL org.opencontainers.image.source="${HERMES_SOURCE_REPOSITORY}" +LABEL org.opencontainers.image.revision="${HERMES_SOURCE_COMMIT}" +LABEL io.labnow.hermes.build-base="${HERMES_BUILD_BASE_IMAGE}" +LABEL io.labnow.hermes.runtime-base="${HERMES_RUNTIME_BASE_IMAGE}" # Production environment ENV NODE_ENV=production @@ -74,7 +102,9 @@ COPY --from=builder /opt/hermes /opt/hermes # Discover the real python site-packages so legacy env-var fallbacks point at the right tree. # Keep explicit versioned fallbacks around in case detection runs before the first pip install. RUN set -eux && cd /opt/hermes \ + && printf 'Acquire::Retries "5";\nAcquire::http::Timeout "30";\n' > /etc/apt/apt.conf.d/80-labnow-retries \ && . /opt/utils/script-utils.sh && install_apt /opt/hermes/install_list_hermes.apt \ + && rm -f /etc/apt/apt.conf.d/80-labnow-retries \ && uv pip install ./vendor/*.whl && rm -rf ./vendor \ && uv pip install -e ".[all,messaging,anthropic,bedrock,azure-identity,hindsight,matrix]" \ && rm -rf /opt/hermes/bin \ diff --git a/docker_hermes/p7/README.md b/docker_hermes/p7/README.md new file mode 100644 index 0000000..a1f145b --- /dev/null +++ b/docker_hermes/p7/README.md @@ -0,0 +1,46 @@ +# P7 Hermes 可复现运行基线 + +此目录只承担 CHE-568 的 `lab-dev` 职责:固定 Hermes 源码/构建 provenance, +校验本地镜像,以及为跨仓 Hermes 产品链提供失败关闭的运行入口。它不复制 +`labnow-open` 的 Hermes renderer 或 `labnow-shell` 的 image→adapter 目录逻辑。 + +## 固定构建 + +P7 构建必须使用准确的 Hermes repository 与 40 位 commit;Dockerfile 不再 clone +移动 `main`。本机网络可用时,使用仓库标准入口构建本地制品: + +```bash +export REGISTRY_SRC=quay.io +export REGISTRY_DST=quay.io +export CI_PROJECT_NAME=LabNow/lab-dev +source ./tool.sh +build_image_no_tag hermes p7-<12hex> docker_hermes/hermes.Dockerfile \ + --build-arg HERMES_SOURCE_REPOSITORY= \ + --build-arg HERMES_SOURCE_COMMIT=<40-hex-commit> \ + --build-arg HERMES_BUILD_BASE_IMAGE=quay.io/labnow/node@sha256:<64-hex> \ + --build-arg HERMES_RUNTIME_BASE_IMAGE=quay.io/labnow/base@sha256:<64-hex> +``` + +产物必须是 `quay.io/labnow/hermes:p7-<12hex>`;不 push。镜像 OCI revision label +和 `/opt/hermes/.labnow-source-*` 是非敏感 provenance;两个 +`io.labnow.hermes.*-base` label 记录实际传入的不可变基础镜像引用。runner 会校验 +这些 label 与受限输入一致。 + +## 入口与安全 + +从 `p7-inputs.example.json` 创建权限为 `0400` 或 `0600` 的 Git 忽略 +`p7-inputs.json`。输入只包含路径、commit、镜像 ID 和运行材料路径;不得保存 +API key、Token、密码、请求正文或响应正文。 + +```bash +./docker_hermes/p7/scripts/test-p7-gates.sh +./docker_hermes/p7/scripts/p7-runner.sh --input /secure/path/p7-inputs.json --validate-input +./docker_hermes/p7/scripts/p7-runner.sh --input /secure/path/p7-inputs.json --preflight +./docker_hermes/p7/scripts/p7-runner.sh --input /secure/path/p7-inputs.json --render +``` + +`--golden` 当前只会在 Hermes renderer、Shell image→adapter catalogue 与固定 +Hermes Workspace 镜像均进入受限输入后运行;缺少任一 P7 产品输入会以 +`P7_ERROR:HERMES_PRODUCT_CHAIN_NOT_AVAILABLE` 失败关闭,绝不把 P6 OpenClaw +结果伪装为 Hermes 结果。运行材料必须由 live Launcher 以 `RuntimeManifest` / +`RuntimeSecretFile` 只读挂载到 Workspace;本 Compose 不接受 provider key 环境变量。 diff --git a/docker_hermes/p7/docker-compose.runtime.yml b/docker_hermes/p7/docker-compose.runtime.yml new file mode 100644 index 0000000..c6293a3 --- /dev/null +++ b/docker_hermes/p7/docker-compose.runtime.yml @@ -0,0 +1,33 @@ +name: p7-hermes + +# P7's Hermes component boundary. The full product topology remains owned by +# the checked-in P7 runner, which starts the live Launcher-managed workspace; +# this file deliberately has no fallback image, credential value, or floating +# tag. RuntimeSecretFile is mounted by the Launcher, never injected here as an +# environment value. +services: + hermes: + image: ${P7_HERMES_IMAGE:?fixed P7 Hermes image required} + pull_policy: never + platform: linux/amd64 + environment: + HERMES_HOME: /root/.hermes + HERMES_DASHBOARD: "true" + HERMES_DASHBOARD_HOST: 127.0.0.1 + HERMES_DASHBOARD_PORT: "9119" + HERMES_DASHBOARD_USE_TRUSTED_PROXY_AUTH: "true" + volumes: + - type: bind + source: ${P7_HERMES_STATE_DIR:?run-scoped Hermes state required} + target: /root/.hermes + healthcheck: + test: ["CMD", "/usr/local/bin/start-hermes.sh", "healthcheck"] + interval: 5s + timeout: 3s + retries: 18 + start_period: 20s + networks: [p7-runtime] + +networks: + p7-runtime: + name: ${P7_RUNTIME_NETWORK:?run-scoped network required} diff --git "a/docker_hermes/p7/evidence/2026-08-10-P7-CHE-568-Hermes\346\236\204\345\273\272\344\270\216\351\252\214\350\257\201.md" "b/docker_hermes/p7/evidence/2026-08-10-P7-CHE-568-Hermes\346\236\204\345\273\272\344\270\216\351\252\214\350\257\201.md" new file mode 100644 index 0000000..bf4096c --- /dev/null +++ "b/docker_hermes/p7/evidence/2026-08-10-P7-CHE-568-Hermes\346\236\204\345\273\272\344\270\216\351\252\214\350\257\201.md" @@ -0,0 +1,80 @@ +# P7 CHE-568 Hermes 构建与验证证据 + +## 1. 证据边界 + +- 观察日期:2026-08-10(Asia/Hong_Kong)。 +- Phase:P7 / CHE-568。 +- 分支:`dev/che-568-hermes-console-experience`。 +- phase base:`45c38585a0ca889f6a20aebfdf3b13a01d369ac2`。 +- control / review policy:`06c49f26642c7e39a118aedad1395197f2bd91db`。 +- 契约:`v1alpha1 / 0.1.0-rc.1`。 +- 本文只证明 `lab-dev` 负责的 Hermes 固定源码构建、基础镜像 provenance、 + 本地制品身份和失败关闭 runner。它不证明 Open renderer、Shell 可信目录或 + Launcher 管理的完整 Hermes Workspace 黄金链已通过。 + +## 2. 固定输入 + +| 输入 | 准确值 | +| --- | --- | +| Hermes repository | `https://github.com/Mushroom47/hermes-agent.git` | +| Hermes commit | `1388cd1c0c1800078bfcc92aebd144fbf145fdb4` | +| build base | `quay.io/labnow/node@sha256:fd09d9de9b7aa927493acbafbb7d399c089465e988f2a6a240428cdbbd5424e2` | +| runtime base | `quay.io/labnow/base@sha256:782f9814152b64cd4aa5ac76d9fbcedcb3bd89f76fabf2ba3d81225d80b05d3f` | +| target platform | `linux/amd64` | + +本地源码 checkout 的 `origin`、远端 `main` 与 `HEAD` 均核对到上述 40 位 +commit;构建过程再次以 `git fetch --depth 1 origin ` 和 detached +checkout 校验准确 commit,不依赖移动 `main`。 + +## 3. 本地制品 + +| 字段 | 准确值 | +| --- | --- | +| local tag | `quay.io/labnow/hermes:p7-1388cd1c0c18` | +| local image ID | `sha256:c47cf16fad3fbb952a616c910fe3c7769e8516a15986db182752091e1ec02d67` | +| local RepoDigest | `quay.io/labnow/hermes@sha256:c47cf16fad3fbb952a616c910fe3c7769e8516a15986db182752091e1ec02d67` | +| OS / architecture | `linux/amd64` | + +`docker image inspect` 已确认四个 provenance label 分别等于固定 repository、 +source commit、build base 和 runtime base。该 RepoDigest 仅是当前 Docker 本地 +制品身份;镜像未 push、未发布,不代表远端 registry 已存在该 digest。 + +## 4. 有界构建修复 + +1. amd64 模拟环境中的 GNU `tar` 解包 `python-olm 3.2.16` sdist 时返回 + `Cannot open: Function not implemented`。同一基础镜像中使用 Python 标准库 + `tarfile` 和 `filter="data"` 的临时容器诊断通过;Dockerfile 只替换解包实现, + 保持源码包版本、wheel 构建和安装语义不变。 +2. Ubuntu archive 连续两次对不同包返回 `502 Bad Gateway`。builder 与 runtime + 安装步骤临时设置 APT `Acquire::Retries=5` 和 HTTP timeout,安装后立即删除; + 未换源、未换包、未改变基础镜像。 +3. 第三次构建显式使用 `DOCKER_DEFAULT_PLATFORM=linux/amd64`,退出码为 0; + `python-olm` wheel、Hermes Web、TUI、Playwright、Python 依赖和最终 runtime + stage 均完成。 + +构建仍报告 Dockerfile 既有 `$PYTHONPATH` `UndefinedVar` 静态 warning;实际 +runtime 环境为 `PYTHONPATH=/opt/hermes:`。该 warning 不改变本次固定制品身份, +作为非阻断后续清理项登记。 + +## 5. 已执行验证 + +| 验证 | 结果 | +| --- | --- | +| `bash -n docker_hermes/p7/scripts/p7-runner.sh docker_hermes/p7/scripts/test-p7-gates.sh` | 通过 | +| `./docker_hermes/p7/scripts/test-p7-gates.sh` | 通过;源码固定、local-only Compose、非法输入与 provenance 不匹配失败关闭 | +| `git diff --check` | 通过 | +| 固定输入本地 Docker build | 通过;退出 0 | +| `docker image inspect` 的 image ID、平台与四个 labels | 通过 | + +## 6. 当前结论与后续门禁 + +- `lab-dev` P7 固定 Hermes base image 已在本地生成,可交给 `labnow-open` 构建 + P7 Workspace 产品镜像。 +- `p7-runner.sh --golden` 在缺少完整产品输入时必须返回 + `P7_ERROR:HERMES_PRODUCT_CHAIN_NOT_AVAILABLE`,不得复用 P6 OpenClaw 证据。 +- 待 `labnow-open`、`labnow-shell` 的准确 P7 commit 和本地产品镜像身份固定后, + 再生成权限为 `0400` 或 `0600` 的 Git 忽略输入,执行 preflight、render 与完整 + Hermes 黄金链。 +- S0/S1:无。S2:既有 `$PYTHONPATH` 静态 warning;不阻断本 Phase 固定组合验证。 +- 本次未读取或处理用户既有 `.DS_Store`、`docs/`、`hermes-chat-screenshot.png`; + 未输出凭证明文,未 push,未发布镜像,未部署,未修改 `main` 或 integration。 diff --git a/docker_hermes/p7/p7-inputs.example.json b/docker_hermes/p7/p7-inputs.example.json new file mode 100644 index 0000000..682417c --- /dev/null +++ b/docker_hermes/p7/p7-inputs.example.json @@ -0,0 +1,27 @@ +{ + "schema_version": "p7-inputs/v1", + "contract_version": "v1alpha1", + "contract_release": "0.1.0-rc.1", + "control_commit": "06c49f26642c7e39a118aedad1395197f2bd91db", + "review_policy_commit": "06c49f26642c7e39a118aedad1395197f2bd91db", + "phase": {"branch": "dev/che-568-hermes-console-experience", "base_commit": "45c38585a0ca889f6a20aebfdf3b13a01d369ac2"}, + "repositories": { + "lab_dev": {"path": "/absolute/path/to/lab-dev", "commit": "REPLACE_WITH_P7_LAB_DEV_HEAD"}, + "labnow_open": {"path": "/absolute/path/to/labnow-open", "commit": "REPLACE_WITH_P7_OPEN_HEAD"}, + "labnow_shell": {"path": "/absolute/path/to/labnow-shell", "commit": "REPLACE_WITH_P7_SHELL_HEAD"}, + "labnow_launcher": {"path": "/absolute/path/to/labnow-launcher", "commit": "c84edea3e051d561f28d9f99235563cf491aaeb2"}, + "hermes_source": {"path": "/absolute/path/to/hermes-agent", "repository": "REPLACE_WITH_OBSERVED_HERMES_REMOTE", "commit": "REPLACE_WITH_FIXED_HERMES_COMMIT"} + }, + "images": { + "hermes": {"ref": "quay.io/labnow/hermes:p7-REPLACE_WITH_12_HEX", "image_id": "sha256:REPLACE_WITH_64_HEX", "provenance": "local_build", "source_repository": "hermes_source", "source_commit": "REPLACE_WITH_FIXED_HERMES_COMMIT"}, + "litellm": {"ref": "quay.io/labnow/litellm@sha256:a2e115874c21b829bd052b18fc85be2f9217fb8244b82812c4ebc6e36f9824d1", "image_id": "sha256:a2e115874c21b829bd052b18fc85be2f9217fb8244b82812c4ebc6e36f9824d1"}, + "workspace": {"ref": "quay.io/labnow/labnow-open:REPLACE_WITH_P7_FIXED_TAG", "image_id": "sha256:REPLACE_WITH_64_HEX", "source_repository": "labnow_open", "source_commit": "REPLACE_WITH_P7_OPEN_HEAD"}, + "shell": {"ref": "quay.io/labnow/labnow-shell:REPLACE_WITH_P7_FIXED_TAG", "image_id": "sha256:REPLACE_WITH_64_HEX", "source_repository": "labnow_shell", "source_commit": "REPLACE_WITH_P7_SHELL_HEAD"}, + "launcher": {"ref": "quay.io/labnow/labnow-launcher:che-563-openclaw-product-closure-local", "image_id": "sha256:6f9732fda8b86d9bfe4596e848025cc38448da4b17dfea8520e046a32b32e61f", "source_repository": "labnow_launcher", "source_commit": "c84edea3e051d561f28d9f99235563cf491aaeb2"} + }, + "base_images": { + "build": "quay.io/labnow/node@sha256:REPLACE_WITH_64_HEX", + "runtime": "quay.io/labnow/base@sha256:REPLACE_WITH_64_HEX" + }, + "runtime": {"p1_env_file": "/secure/path/to/p1.env"} +} diff --git a/docker_hermes/p7/scripts/p7-runner.sh b/docker_hermes/p7/scripts/p7-runner.sh new file mode 100755 index 0000000..d034856 --- /dev/null +++ b/docker_hermes/p7/scripts/p7-runner.sh @@ -0,0 +1,121 @@ +#!/usr/bin/env bash +# P7 local-only coordinator. It validates immutable product provenance before +# any container is started and fails closed until all three P7 product inputs +# (Hermes renderer, Shell catalogue, and fixed workspace image) are present. +set -euo pipefail + +script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +p7_dir="$(cd "${script_dir}/.." && pwd)" + +die() { printf 'P7_ERROR:%s\n' "$1" >&2; exit "${2:-1}"; } +sha256() { shasum -a 256 "$1" | awk '{print $1}'; } +run_id() { python3 -c 'import secrets; print("p7-" + secrets.token_hex(16))'; } +mode_of() { stat -f '%Lp' "$1" 2>/dev/null || stat -c '%a' "$1"; } +secure_file() { [[ -f "$1" && ! -L "$1" && "$(mode_of "$1")" =~ ^(400|600)$ ]] || die "SECURE_FILE_REQUIRED" 64; } + +usage() { printf '%s\n' "Usage: p7-runner.sh --input /secure/path/p7-inputs.json --validate-input|--preflight|--render|--golden|--cleanup"; } +input=""; action="" +while (($#)); do + case "$1" in + --input) input="${2:-}"; shift 2 ;; + --validate-input|--preflight|--render|--golden|--cleanup) [[ -z "$action" ]] || die "ACTION_DUPLICATED" 2; action="${1#--}"; shift ;; + *) usage >&2; exit 2 ;; + esac +done +[[ -n "$input" && -n "$action" ]] || { usage >&2; exit 2; } +secure_file "$input" +export P7_INPUT_FILE="$input" +P7_RUN_ID="${P7_RUN_ID:-$(run_id)}" +[[ "$P7_RUN_ID" =~ ^p7-[a-f0-9]{32}$ ]] || die "RUN_ID_INVALID" 64 +export P7_RUN_ID +artifact_dir="${P7_ARTIFACTS_DIR:-${p7_dir}/artifacts}" +work_dir="${P7_WORK_DIR:-${p7_dir}/.p7-work/${P7_RUN_ID}}" +mkdir -p "$artifact_dir"; chmod 700 "$artifact_dir" +if [[ "$action" == golden || "$action" == cleanup ]]; then + mkdir -p "$work_dir"; chmod 700 "$work_dir" +fi +report="${artifact_dir}/p7-${action}-${P7_RUN_ID}.json" + +validate_shape() { + jq -e ' + type == "object" + and .schema_version == "p7-inputs/v1" + and .contract_version == "v1alpha1" and .contract_release == "0.1.0-rc.1" + and .control_commit == "06c49f26642c7e39a118aedad1395197f2bd91db" + and .review_policy_commit == "06c49f26642c7e39a118aedad1395197f2bd91db" + and .phase.branch == "dev/che-568-hermes-console-experience" + and .phase.base_commit == "45c38585a0ca889f6a20aebfdf3b13a01d369ac2" + and (.repositories | keys | sort) == ["hermes_source","lab_dev","labnow_launcher","labnow_open","labnow_shell"] + and (.images | keys | sort) == ["hermes","launcher","litellm","shell","workspace"] + and (.base_images | keys | sort) == ["build","runtime"] + and (.runtime | keys | sort) == ["p1_env_file"] + and ([.repositories[] | .commit] | all(type == "string" and test("^[0-9a-f]{40}$"))) + and (.images.hermes.ref | test("^quay\\.io/labnow/hermes:p7-[0-9a-f]{12}$")) + and ([.images[] | .image_id] | all(type == "string" and test("^sha256:[0-9a-f]{64}$"))) + and (.images.hermes.provenance == "local_build") + and (.images.hermes.source_repository == "hermes_source") + and (.images.hermes.source_commit == .repositories.hermes_source.commit) + and ([.base_images[]] | all(type == "string" and test("^quay\\.io/labnow/(node|base)@sha256:[0-9a-f]{64}$"))) + and (.runtime.p1_env_file | type == "string" and startswith("/")) + ' "$input" >/dev/null || die "INPUT_SCHEMA_INVALID" 68 +} + +assert_repository() { + local name="$1" path expected actual status + path="$(jq -er ".repositories.${name}.path" "$input")" + expected="$(jq -er ".repositories.${name}.commit" "$input")" + [[ -d "$path/.git" ]] || die "REPOSITORY_UNAVAILABLE" 69 + actual="$(git -C "$path" rev-parse HEAD)" + [[ "$actual" == "$expected" ]] || die "REPOSITORY_COMMIT_MISMATCH" 70 + status="$(git -C "$path" status --porcelain=v1 --untracked-files=no)" + [[ -z "$status" ]] || die "REPOSITORY_TRACKED_TREE_DIRTY" 71 +} + +preflight() { + validate_shape + secure_file "$(jq -er '.runtime.p1_env_file' "$input")" + local repo + for repo in lab_dev labnow_open labnow_shell labnow_launcher hermes_source; do assert_repository "$repo"; done + [[ "$(git -C "$(jq -er '.repositories.lab_dev.path' "$input")" branch --show-current)" == "dev/che-568-hermes-console-experience" ]] || die "PHASE_BRANCH_MISMATCH" 70 + git -C "$(jq -er '.repositories.lab_dev.path' "$input")" merge-base --is-ancestor "$(jq -er '.phase.base_commit' "$input")" HEAD || die "PHASE_BASE_NOT_ANCESTOR" 70 + [[ "$(git -C "$(jq -er '.repositories.hermes_source.path' "$input")" remote get-url origin)" == "$(jq -er '.repositories.hermes_source.repository' "$input")" ]] || die "HERMES_SOURCE_REMOTE_MISMATCH" 70 + local ref expected actual + ref="$(jq -er '.images.hermes.ref' "$input")"; expected="$(jq -er '.images.hermes.image_id' "$input")" + actual="$(docker image inspect --format '{{.Id}}' "$ref" 2>/dev/null)" || die "HERMES_IMAGE_UNAVAILABLE" 72 + [[ "$actual" == "$expected" ]] || die "HERMES_IMAGE_ID_MISMATCH" 72 + [[ "$(docker image inspect --format '{{ index .Config.Labels "org.opencontainers.image.revision" }}' "$ref")" == "$(jq -er '.repositories.hermes_source.commit' "$input")" ]] || die "HERMES_IMAGE_PROVENANCE_MISMATCH" 72 + [[ "$(docker image inspect --format '{{ index .Config.Labels "io.labnow.hermes.build-base" }}' "$ref")" == "$(jq -er '.base_images.build' "$input")" ]] || die "HERMES_BUILD_BASE_MISMATCH" 72 + [[ "$(docker image inspect --format '{{ index .Config.Labels "io.labnow.hermes.runtime-base" }}' "$ref")" == "$(jq -er '.base_images.runtime' "$input")" ]] || die "HERMES_RUNTIME_BASE_MISMATCH" 72 +} + +write_report() { + local result="$1" phase="$2" reason="${3:-}" extra="${4:-{}}" tmp + tmp="$(mktemp "${artifact_dir}/.p7-report.XXXXXX")" + jq -n --arg run_id "$P7_RUN_ID" --arg result "$result" --arg phase "$phase" --arg reason "$reason" --arg input_sha "$(sha256 "$input")" --arg tested_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" --argjson provenance "$(jq -c '{contract_version,contract_release,control_commit,review_policy_commit,phase,repositories:(.repositories|with_entries(.value={commit:.value.commit})),images}' "$input")" --argjson extra "$extra" \ + '{schema_version:"p7-report/v1",run_id:$run_id,result:$result,phase:$phase,input_sha256:$input_sha,tested_at:$tested_at,content_redacted:true} + $provenance + $extra + (if $reason == "" then {} else {reason:$reason} end)' > "$tmp" + chmod 600 "$tmp"; mv -f "$tmp" "$report" +} + +cleanup() { + local env="${work_dir}/runtime.env" + if [[ -f "$env" && ! -L "$env" ]]; then docker compose --project-name "p7-${P7_RUN_ID#p7-}" --env-file "$env" -f "${p7_dir}/docker-compose.runtime.yml" down --volumes --remove-orphans >/dev/null 2>&1 || true; fi + rm -rf "$work_dir" +} + +case "$action" in + validate-input) if validate_shape; then write_report passed completed; else write_report failed precondition_failed input_validation; exit 1; fi ;; + preflight) if preflight; then write_report passed completed; else write_report failed precondition_failed preflight; exit 1; fi ;; + render) + if ! preflight; then write_report failed precondition_failed preflight; exit 1; fi + jq -n --arg run_id "$P7_RUN_ID" --arg compose_sha256 "$(sha256 "${p7_dir}/docker-compose.runtime.yml")" --arg image "$(jq -er '.images.hermes.ref' "$input")" '{schema_version:"p7-render/v1",run_id:$run_id,compose_sha256:$compose_sha256,hermes_image:$image,content_redacted:true}' > "${artifact_dir}/p7-render-${P7_RUN_ID}.json" + chmod 600 "${artifact_dir}/p7-render-${P7_RUN_ID}.json"; write_report passed completed + ;; + golden) + # No compatibility fallback to the P6 OpenClaw runner is permitted. The + # Hermes renderer/catalogue image must be supplied by the two owning P7 + # repositories, otherwise a real lifecycle run cannot be claimed. + if ! preflight; then write_report failed precondition_failed preflight; exit 1; fi + write_report failed blocked "HERMES_PRODUCT_CHAIN_NOT_AVAILABLE"; exit 1 + ;; + cleanup) cleanup; write_report passed completed ;; +esac diff --git a/docker_hermes/p7/scripts/test-p7-gates.sh b/docker_hermes/p7/scripts/test-p7-gates.sh new file mode 100755 index 0000000..760f349 --- /dev/null +++ b/docker_hermes/p7/scripts/test-p7-gates.sh @@ -0,0 +1,48 @@ +#!/usr/bin/env bash +set -euo pipefail + +root="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)" +runner="${root}/docker_hermes/p7/scripts/p7-runner.sh" +tmp="$(mktemp -d "${TMPDIR:-/tmp}/p7-gates.XXXXXX")" +chmod 700 "$tmp" +trap 'rm -rf "$tmp"' EXIT + +# The source pin must not regress to a moving branch. This is intentionally a +# static gate: it does not contact an upstream repository or Docker daemon. +! rg -q 'git clone --depth 1 --branch main' "${root}/docker_hermes/hermes.Dockerfile" +rg -q 'ARG HERMES_SOURCE_COMMIT' "${root}/docker_hermes/hermes.Dockerfile" +rg -q 'git fetch --depth 1 origin "\$HERMES_SOURCE_COMMIT"' "${root}/docker_hermes/hermes.Dockerfile" +rg -q 'org.opencontainers.image.revision' "${root}/docker_hermes/hermes.Dockerfile" +rg -q 'ARG HERMES_BUILD_BASE_IMAGE' "${root}/docker_hermes/hermes.Dockerfile" +rg -q 'io.labnow.hermes.runtime-base' "${root}/docker_hermes/hermes.Dockerfile" +rg -q 'pull_policy: never' "${root}/docker_hermes/p7/docker-compose.runtime.yml" +! rg -n --glob '!**/test-p7-gates.sh' 'OPENAI_API_KEY:|DEEPSEEK_API_KEY:|:latest' "${root}/docker_hermes/p7" + +input="$tmp/invalid.json" +printf '%s\n' '{"schema_version":"p7-inputs/v1"}' > "$input"; chmod 600 "$input" +if P7_ARTIFACTS_DIR="$tmp/artifacts" P7_WORK_DIR="$tmp/work" "$runner" --input "$input" --validate-input >/dev/null 2>&1; then + printf '%s\n' 'P7 invalid input was accepted' >&2; exit 1 +fi + +valid="$tmp/valid.json" +jq ' + .repositories |= with_entries(.value.commit = "0123456789abcdef0123456789abcdef01234567") + | .images |= with_entries(.value.image_id = "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa") + | .images.hermes.ref = "quay.io/labnow/hermes:p7-0123456789ab" + | .images.hermes.source_commit = .repositories.hermes_source.commit + | .images.workspace.source_commit = .repositories.labnow_open.commit + | .images.shell.source_commit = .repositories.labnow_shell.commit + | .images.launcher.source_commit = .repositories.labnow_launcher.commit + | .base_images.build = "quay.io/labnow/node@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" + | .base_images.runtime = "quay.io/labnow/base@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" + | .runtime.p1_env_file = "/private/tmp/p7-test.env" +' "${root}/docker_hermes/p7/p7-inputs.example.json" > "$valid" +chmod 600 "$valid" +P7_ARTIFACTS_DIR="$tmp/artifacts" P7_WORK_DIR="$tmp/work" "$runner" --input "$valid" --validate-input >/dev/null + +jq '.images.hermes.source_commit = "fedcba9876543210fedcba9876543210fedcba98"' "$valid" > "${valid}.mismatch" +chmod 600 "${valid}.mismatch" +if P7_ARTIFACTS_DIR="$tmp/artifacts" P7_WORK_DIR="$tmp/work" "$runner" --input "${valid}.mismatch" --validate-input >/dev/null 2>&1; then + printf '%s\n' 'P7 provenance mismatch input was accepted' >&2; exit 1 +fi +printf '%s\n' 'PASS P7 gates: source pin, local-only compose and invalid input fail closed.' From 90328f6a520b0c617689bcbc21576a10f5e34b10 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Mon, 10 Aug 2026 23:38:02 +0800 Subject: [PATCH 61/87] =?UTF-8?q?fix(hermes):=20=E5=AE=9E=E7=8E=B0=20P7=20?= =?UTF-8?q?=E5=8F=82=E6=95=B0=E5=8C=96=E9=BB=84=E9=87=91=E9=93=BE?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_hermes/p7/README.md | 28 +- docker_hermes/p7/p7-inputs.example.json | 23 +- .../p7/scripts/p7-prepare-runtime.py | 144 ++++++ docker_hermes/p7/scripts/p7-product-chain.py | 457 ++++++++++++++++++ docker_hermes/p7/scripts/p7-runner.sh | 232 +++++++-- docker_hermes/p7/scripts/test-p7-gates.sh | 33 +- 6 files changed, 849 insertions(+), 68 deletions(-) create mode 100755 docker_hermes/p7/scripts/p7-prepare-runtime.py create mode 100755 docker_hermes/p7/scripts/p7-product-chain.py diff --git a/docker_hermes/p7/README.md b/docker_hermes/p7/README.md index a1f145b..4c77464 100644 --- a/docker_hermes/p7/README.md +++ b/docker_hermes/p7/README.md @@ -1,8 +1,9 @@ # P7 Hermes 可复现运行基线 此目录只承担 CHE-568 的 `lab-dev` 职责:固定 Hermes 源码/构建 provenance, -校验本地镜像,以及为跨仓 Hermes 产品链提供失败关闭的运行入口。它不复制 -`labnow-open` 的 Hermes renderer 或 `labnow-shell` 的 image→adapter 目录逻辑。 +校验本地镜像,并参数化复用 P6 已验证的真实五组件拓扑执行 Hermes 产品链。 +它不复制 `labnow-open` 的 Hermes renderer 或 `labnow-shell` 的 +image→adapter 目录业务逻辑。 ## 固定构建 @@ -29,18 +30,27 @@ build_image_no_tag hermes p7-<12hex> docker_hermes/hermes.Dockerfile \ ## 入口与安全 从 `p7-inputs.example.json` 创建权限为 `0400` 或 `0600` 的 Git 忽略 -`p7-inputs.json`。输入只包含路径、commit、镜像 ID 和运行材料路径;不得保存 -API key、Token、密码、请求正文或响应正文。 +`p7-inputs.json`。输入只包含路径、delivery/runtime commit、镜像 ID/RepoDigest +和 P1 环境文件路径;不得保存 API key、Token、密码、请求正文或响应正文。 ```bash ./docker_hermes/p7/scripts/test-p7-gates.sh ./docker_hermes/p7/scripts/p7-runner.sh --input /secure/path/p7-inputs.json --validate-input ./docker_hermes/p7/scripts/p7-runner.sh --input /secure/path/p7-inputs.json --preflight ./docker_hermes/p7/scripts/p7-runner.sh --input /secure/path/p7-inputs.json --render +P7_RUN_ID=p7-<32hex> ./docker_hermes/p7/scripts/p7-runner.sh --input /secure/path/p7-inputs.json --golden +P7_RUN_ID=p7- ./docker_hermes/p7/scripts/p7-runner.sh --input /secure/path/p7-inputs.json --cleanup ``` -`--golden` 当前只会在 Hermes renderer、Shell image→adapter catalogue 与固定 -Hermes Workspace 镜像均进入受限输入后运行;缺少任一 P7 产品输入会以 -`P7_ERROR:HERMES_PRODUCT_CHAIN_NOT_AVAILABLE` 失败关闭,绝不把 P6 OpenClaw -结果伪装为 Hermes 结果。运行材料必须由 live Launcher 以 `RuntimeManifest` / -`RuntimeSecretFile` 只读挂载到 Workspace;本 Compose 不接受 provider key 环境变量。 +`--golden` 会先核验三仓 delivery/runtime commit、Hermes 上游、五个产品镜像、 +三个 support image 和两个构建基础引用,再复用 P6 的隔离 LiteLLM、Shell、 +Launcher/JupyterHub 拓扑。Workspace 只能由 live DockerSpawner 创建;Shell +服务端从准确 Open 产品镜像推导 `hermes` Adapter,Launcher 以 +`RuntimeManifest` / `RuntimeSecretFile` 只读挂载运行材料。runner 执行真实 +Hermes 非流式调用、数据面流式调用、Hermes terminal tool、usage、两代 key、 +stop/restart/delete/revoke、owner 负向、零明文扫描与 run-scoped cleanup。 + +Console 鼠标创建、键盘/焦点和 axe 使用同一 Shell commit 的 P7 浏览器证据; +runner 另外执行该固定 Shell image 的 live API → JupyterHub → Workspace 链, +不得以 P6 OpenClaw 报告或健康检查替代 Hermes 成功。失败报告只保存错误码, +成功报告只保存结构断言、非敏感 ID、计数和 SHA-256。 diff --git a/docker_hermes/p7/p7-inputs.example.json b/docker_hermes/p7/p7-inputs.example.json index 682417c..2fa449b 100644 --- a/docker_hermes/p7/p7-inputs.example.json +++ b/docker_hermes/p7/p7-inputs.example.json @@ -6,18 +6,23 @@ "review_policy_commit": "06c49f26642c7e39a118aedad1395197f2bd91db", "phase": {"branch": "dev/che-568-hermes-console-experience", "base_commit": "45c38585a0ca889f6a20aebfdf3b13a01d369ac2"}, "repositories": { - "lab_dev": {"path": "/absolute/path/to/lab-dev", "commit": "REPLACE_WITH_P7_LAB_DEV_HEAD"}, - "labnow_open": {"path": "/absolute/path/to/labnow-open", "commit": "REPLACE_WITH_P7_OPEN_HEAD"}, - "labnow_shell": {"path": "/absolute/path/to/labnow-shell", "commit": "REPLACE_WITH_P7_SHELL_HEAD"}, - "labnow_launcher": {"path": "/absolute/path/to/labnow-launcher", "commit": "c84edea3e051d561f28d9f99235563cf491aaeb2"}, + "lab_dev": {"path": "/absolute/path/to/lab-dev", "commit": "REPLACE_WITH_P7_LAB_DEV_HEAD", "runtime_commit": "REPLACE_WITH_P7_LAB_DEV_RUNTIME_HEAD"}, + "labnow_open": {"path": "/absolute/path/to/labnow-open", "commit": "533b3c5bd0742a77a1cccfc4a30b818271751213", "runtime_commit": "8c56966c4b6be12702d4397aad6b1a153b87d053"}, + "labnow_shell": {"path": "/absolute/path/to/labnow-shell", "commit": "REPLACE_WITH_P7_SHELL_HEAD", "runtime_commit": "REPLACE_WITH_P7_SHELL_RUNTIME_HEAD"}, + "labnow_launcher": {"path": "/absolute/path/to/labnow-launcher", "commit": "c84edea3e051d561f28d9f99235563cf491aaeb2", "runtime_commit": "c84edea3e051d561f28d9f99235563cf491aaeb2"}, "hermes_source": {"path": "/absolute/path/to/hermes-agent", "repository": "REPLACE_WITH_OBSERVED_HERMES_REMOTE", "commit": "REPLACE_WITH_FIXED_HERMES_COMMIT"} }, "images": { - "hermes": {"ref": "quay.io/labnow/hermes:p7-REPLACE_WITH_12_HEX", "image_id": "sha256:REPLACE_WITH_64_HEX", "provenance": "local_build", "source_repository": "hermes_source", "source_commit": "REPLACE_WITH_FIXED_HERMES_COMMIT"}, - "litellm": {"ref": "quay.io/labnow/litellm@sha256:a2e115874c21b829bd052b18fc85be2f9217fb8244b82812c4ebc6e36f9824d1", "image_id": "sha256:a2e115874c21b829bd052b18fc85be2f9217fb8244b82812c4ebc6e36f9824d1"}, - "workspace": {"ref": "quay.io/labnow/labnow-open:REPLACE_WITH_P7_FIXED_TAG", "image_id": "sha256:REPLACE_WITH_64_HEX", "source_repository": "labnow_open", "source_commit": "REPLACE_WITH_P7_OPEN_HEAD"}, - "shell": {"ref": "quay.io/labnow/labnow-shell:REPLACE_WITH_P7_FIXED_TAG", "image_id": "sha256:REPLACE_WITH_64_HEX", "source_repository": "labnow_shell", "source_commit": "REPLACE_WITH_P7_SHELL_HEAD"}, - "launcher": {"ref": "quay.io/labnow/labnow-launcher:che-563-openclaw-product-closure-local", "image_id": "sha256:6f9732fda8b86d9bfe4596e848025cc38448da4b17dfea8520e046a32b32e61f", "source_repository": "labnow_launcher", "source_commit": "c84edea3e051d561f28d9f99235563cf491aaeb2"} + "hermes": {"ref": "quay.io/labnow/hermes:p7-REPLACE_WITH_12_HEX", "image_id": "sha256:REPLACE_WITH_64_HEX", "repo_digest": "quay.io/labnow/hermes@sha256:REPLACE_WITH_64_HEX", "provenance": "local_build", "source_repository": "hermes_source", "source_commit": "REPLACE_WITH_FIXED_HERMES_COMMIT"}, + "litellm": {"ref": "quay.io/labnow/litellm@sha256:a2e115874c21b829bd052b18fc85be2f9217fb8244b82812c4ebc6e36f9824d1", "image_id": "sha256:a2e115874c21b829bd052b18fc85be2f9217fb8244b82812c4ebc6e36f9824d1", "repo_digest": "quay.io/labnow/litellm@sha256:a2e115874c21b829bd052b18fc85be2f9217fb8244b82812c4ebc6e36f9824d1", "provenance": "repo_digest"}, + "workspace": {"ref": "quay.io/labnow/labnow-open@sha256:56e69ace0da9dbede193e80904fa76ded1dd94cfc38cb1137f6df686c5b7f031", "image_id": "sha256:56e69ace0da9dbede193e80904fa76ded1dd94cfc38cb1137f6df686c5b7f031", "repo_digest": "quay.io/labnow/labnow-open@sha256:56e69ace0da9dbede193e80904fa76ded1dd94cfc38cb1137f6df686c5b7f031", "provenance": "local_build", "source_repository": "labnow_open", "source_commit": "8c56966c4b6be12702d4397aad6b1a153b87d053"}, + "shell": {"ref": "quay.io/labnow/labnow-shell@sha256:REPLACE_WITH_64_HEX", "image_id": "sha256:REPLACE_WITH_64_HEX", "repo_digest": "quay.io/labnow/labnow-shell@sha256:REPLACE_WITH_64_HEX", "provenance": "local_build", "source_repository": "labnow_shell", "source_commit": "REPLACE_WITH_P7_SHELL_RUNTIME_HEAD"}, + "launcher": {"ref": "quay.io/labnow/labnow-launcher@sha256:6f9732fda8b86d9bfe4596e848025cc38448da4b17dfea8520e046a32b32e61f", "image_id": "sha256:6f9732fda8b86d9bfe4596e848025cc38448da4b17dfea8520e046a32b32e61f", "repo_digest": "quay.io/labnow/labnow-launcher@sha256:6f9732fda8b86d9bfe4596e848025cc38448da4b17dfea8520e046a32b32e61f", "provenance": "local_build", "source_repository": "labnow_launcher", "source_commit": "c84edea3e051d561f28d9f99235563cf491aaeb2"} + }, + "support_images": { + "postgres": {"ref": "postgres:17-alpine", "image_id": "sha256:742f40ea20b9ff2ff31db5458d127452988a2164df9e17441e191f3b72252193", "repo_digest": "postgres@sha256:742f40ea20b9ff2ff31db5458d127452988a2164df9e17441e191f3b72252193"}, + "redis": {"ref": "redis:7.4-alpine", "image_id": "sha256:e7723ff73d963f5cc6d9c4643ea3d989527a402a319239054e9472a7fb9219a2", "repo_digest": "redis@sha256:e7723ff73d963f5cc6d9c4643ea3d989527a402a319239054e9472a7fb9219a2"}, + "nginx": {"ref": "nginx:1.27-alpine", "image_id": "sha256:65645c7bb6a0661892a8b03b89d0743208a18dd2f3f17a54ef4b76fb8e2f2a10", "repo_digest": "nginx@sha256:65645c7bb6a0661892a8b03b89d0743208a18dd2f3f17a54ef4b76fb8e2f2a10"} }, "base_images": { "build": "quay.io/labnow/node@sha256:REPLACE_WITH_64_HEX", diff --git a/docker_hermes/p7/scripts/p7-prepare-runtime.py b/docker_hermes/p7/scripts/p7-prepare-runtime.py new file mode 100755 index 0000000..b10b1dc --- /dev/null +++ b/docker_hermes/p7/scripts/p7-prepare-runtime.py @@ -0,0 +1,144 @@ +#!/usr/bin/env python3 +"""Prepare the P7 Hermes topology by adapting the already-verified P6 runtime. + +The translation contains only paths, commits and image identities. P1 values +and generated credentials remain in the P6 preparer's run-scoped private files. +""" + +from __future__ import annotations + +import importlib.util +import json +import os +import stat +import sys +from pathlib import Path +from typing import Any + + +class PrepareError(RuntimeError): + pass + + +def restricted(path: Path, code: str) -> None: + try: + info = path.stat() + except OSError as exc: + raise PrepareError(code) from exc + if not stat.S_ISREG(info.st_mode) or info.st_mode & 0o077: + raise PrepareError(code) + + +def load_json(path: Path, code: str) -> dict[str, Any]: + restricted(path, code) + try: + value = json.loads(path.read_text(encoding="utf-8")) + except (OSError, UnicodeDecodeError, json.JSONDecodeError) as exc: + raise PrepareError(code) from exc + if not isinstance(value, dict): + raise PrepareError(code) + return value + + +def load_p6_prepare(path: Path): + spec = importlib.util.spec_from_file_location("labnow_p6_prepare", path) + if spec is None or spec.loader is None: + raise PrepareError("P6_PREPARER_UNAVAILABLE") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def main() -> int: + input_path = Path(os.environ.get("P7_INPUT_FILE", "")) + work_dir = Path(os.environ.get("P7_WORK_DIR", "")) + artifact_dir = Path(os.environ.get("P7_ARTIFACTS_DIR", "")) + run_id = os.environ.get("P7_RUN_ID", "") + if not input_path.is_absolute() or not work_dir.is_absolute() or not artifact_dir.is_absolute(): + raise PrepareError("P7_PREPARE_PATH_INVALID") + if not run_id.startswith("p7-") or len(run_id) != 35: + raise PrepareError("P7_PREPARE_RUN_ID_INVALID") + + inputs = load_json(input_path, "P7_INPUT_INVALID") + repositories = inputs["repositories"] + images = inputs["images"] + translated = { + "repositories": repositories, + "images": { + "litellm": images["litellm"], + "openclaw_workspace": images["workspace"], + }, + "local_only_images": { + "launcher": images["launcher"], + "shell": images["shell"], + }, + "support_images": inputs["support_images"], + "runtime": inputs["runtime"], + } + + script_dir = Path(__file__).resolve().parent + p6_prepare_path = script_dir.parents[2] / "docker_openclaw" / "p6" / "scripts" / "p6-prepare-runtime.py" + p6 = load_p6_prepare(p6_prepare_path) + work_dir.mkdir(mode=0o700, parents=True, exist_ok=True) + os.chmod(work_dir, 0o700) + translated_path = work_dir / "p7-to-p6-runtime-input.json" + p6.write_private(translated_path, json.dumps(translated, separators=(",", ":")) + "\n", mode=0o600) + + internal_run_id = "p6-" + run_id.removeprefix("p7-") + previous = {name: os.environ.get(name) for name in ("P6_INPUT_FILE", "P6_WORK_DIR", "P6_RUN_ID", "P6_ARTIFACTS_DIR")} + os.environ.update( + { + "P6_INPUT_FILE": str(translated_path), + "P6_WORK_DIR": str(work_dir), + "P6_RUN_ID": internal_run_id, + "P6_ARTIFACTS_DIR": str(artifact_dir), + } + ) + try: + if p6.main() != 0: + raise PrepareError("P6_PREPARE_FAILED") + finally: + for name, value in previous.items(): + if value is None: + os.environ.pop(name, None) + else: + os.environ[name] = value + + shell_env = work_dir / "secrets" / "shell.env" + restricted(shell_env, "SHELL_ENV_INVALID") + shell_text = shell_env.read_text(encoding="utf-8") + workspace_ref = images["workspace"]["ref"] + if not workspace_ref.startswith("quay.io/labnow/labnow-open@sha256:"): + raise PrepareError("P7_WORKSPACE_IMAGE_INVALID") + p6.write_private( + shell_env, + shell_text.rstrip("\n") + f"\nMODEL_ACCESS_HERMES_WORKSPACE_IMAGE={workspace_ref}\n", + mode=0o600, + ) + + config_path = work_dir / "config" / "driver-config.json" + config = load_json(config_path, "P7_DRIVER_CONFIG_INVALID") + config.update( + { + "schema_version": "p7-product-chain-config/v1", + "run_id": run_id, + "internal_run_id": internal_run_id, + "adapter_id": "hermes", + "shell_ui_evidence": { + "repository": "labnow_shell", + "commit": repositories["labnow_shell"]["commit"], + "status": "reused_p7_verified_evidence", + }, + } + ) + p6.write_private(config_path, json.dumps(config, separators=(",", ":")) + "\n", mode=0o600) + return 0 + + +if __name__ == "__main__": + try: + sys.exit(main()) + except (PrepareError, OSError, KeyError, TypeError, ValueError) as exc: + code = str(exc) if isinstance(exc, PrepareError) else "P7_PREPARE_FAILED" + print(f"P7_ERROR:{code}", file=sys.stderr) + sys.exit(1) diff --git a/docker_hermes/p7/scripts/p7-product-chain.py b/docker_hermes/p7/scripts/p7-product-chain.py new file mode 100755 index 0000000..994785c --- /dev/null +++ b/docker_hermes/p7/scripts/p7-product-chain.py @@ -0,0 +1,457 @@ +#!/usr/bin/env python3 +"""Execute the live P7 Shell -> Launcher -> Hermes -> LiteLLM chain. + +Model output and credentials are handled in memory only. The retained report +contains structural assertions, non-sensitive IDs, counts and lifecycle state. +""" + +from __future__ import annotations + +import importlib.util +import json +import os +import ssl +import stat +import subprocess +import sys +import time +import urllib.error +import urllib.parse +import urllib.request +from datetime import datetime, timedelta, timezone +from pathlib import Path +from typing import Any + + +class DriverError(RuntimeError): + pass + + +def fail(code: str) -> None: + raise DriverError(code) + + +def load_p6_module(): + path = Path(__file__).resolve().parents[3] / "docker_openclaw" / "p6" / "scripts" / "p6-product-chain.py" + spec = importlib.util.spec_from_file_location("labnow_p6_product", path) + if spec is None or spec.loader is None: + fail("P6_PRODUCT_MODULE_UNAVAILABLE") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +p6 = load_p6_module() +MANIFEST = "/run/labnow/model-access/manifest.json" +SECRET = "/run/labnow/model-access/secret.json" +STATUS = "/run/labnow/model-access/status.json" +HERMES_HOME = "/root/.hermes" + + +def restricted(path: Path, code: str) -> None: + try: + info = path.stat() + except OSError as exc: + raise DriverError(code) from exc + if not stat.S_ISREG(info.st_mode) or info.st_mode & 0o077: + fail(code) + + +def private_write(path: Path, value: str, mode: int = 0o600) -> None: + p6.private_write(path, value, mode) + + +def load_config(path: Path) -> dict[str, Any]: + restricted(path, "PRODUCT_CONFIG_INVALID") + try: + value = json.loads(path.read_text(encoding="utf-8")) + except (OSError, UnicodeDecodeError, json.JSONDecodeError) as exc: + raise DriverError("PRODUCT_CONFIG_INVALID") from exc + required = { + "schema_version", "run_id", "internal_run_id", "adapter_id", "project", + "compose_file", "runtime_env_file", "work_dir", "surface_dir", + "workspace_root", "runtime_root", "owner_a", "owner_b", "server_name", + "workspace_container", "launcher_container", "shell_container", + "litellm_container", "shell_postgres_container", "workspace_image", + "hub_token_file", "launcher_token_file", "upstream_key_file", + "upstream_origin", "upstream_model", "ca_file", "secret_files", + "shell_ui_evidence", + } + if not isinstance(value, dict) or set(value) != required: + fail("PRODUCT_CONFIG_INVALID") + if value.get("schema_version") != "p7-product-chain-config/v1" or value.get("adapter_id") != "hermes": + fail("PRODUCT_CONFIG_INVALID") + if not isinstance(value.get("run_id"), str) or not value["run_id"].startswith("p7-"): + fail("PRODUCT_CONFIG_INVALID") + if not isinstance(value.get("internal_run_id"), str) or not value["internal_run_id"].startswith("p6-"): + fail("PRODUCT_CONFIG_INVALID") + for key in required - {"secret_files", "shell_ui_evidence"}: + if not isinstance(value.get(key), str) or not value[key]: + fail("PRODUCT_CONFIG_INVALID") + if not isinstance(value["secret_files"], list) or not value["secret_files"]: + fail("PRODUCT_CONFIG_INVALID") + evidence = value.get("shell_ui_evidence") + if not isinstance(evidence, dict) or evidence.get("status") != "reused_p7_verified_evidence": + fail("PRODUCT_CONFIG_INVALID") + return value + + +def assert_workspace(config: dict[str, Any], runtime_key: str) -> tuple[str, dict[str, Any]]: + inspect = p6.docker_inspect(config["workspace_container"]) + mounts = {item.get("Destination"): item for item in inspect.get("Mounts", [])} + for target in (MANIFEST, SECRET, "/run/labnow/p6-ca.pem"): + if target not in mounts or mounts[target].get("RW") is not False: + fail("WORKSPACE_MOUNT_INVALID") + if runtime_key in json.dumps(inspect, sort_keys=True): + fail("RUNTIME_KEY_LEAKED_TO_INSPECT") + env = inspect.get("Config", {}).get("Env", []) + prefix = next((item.split("=", 1)[1] for item in env if item.startswith("URL_PREFIX=")), "") + if not prefix.startswith("/studio/user/"): + fail("WORKSPACE_URL_PREFIX_INVALID") + status_raw = p6.command(["docker", "exec", config["workspace_container"], "cat", STATUS], code="ADAPTER_STATUS_UNAVAILABLE") + try: + adapter_status = json.loads(status_raw) + except json.JSONDecodeError as exc: + raise DriverError("ADAPTER_STATUS_INVALID") from exc + if adapter_status.get("phase") != "ready" or adapter_status.get("adapter_id") != "hermes": + fail("ADAPTER_STATUS_INVALID") + deadline = time.monotonic() + 90 + readiness = f"http://127.0.0.1{prefix}api" + while time.monotonic() < deadline: + result = subprocess.run( + ["docker", "exec", config["workspace_container"], "curl", "--fail", "--silent", "--max-time", "3", readiness], + check=False, + stdin=subprocess.DEVNULL, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ) + if result.returncode == 0: + break + time.sleep(1) + else: + fail("HERMES_READINESS_FAILED") + summary = { + "image": inspect.get("Config", {}).get("Image"), + "mounts": sorted(mounts), + "environment_keys": sorted(item.split("=", 1)[0] for item in env), + "state": inspect.get("State", {}).get("Status"), + } + return prefix, summary + + +def hermes_model_call(config: dict[str, Any], marker: str) -> dict[str, Any]: + output = p6.command( + [ + "docker", "exec", config["workspace_container"], "timeout", "--signal=TERM", + "--kill-after=10s", "180s", "start-labnow-hermes.sh", "-z", + f"Reply {marker} only.", "--ignore-rules", "--max-turns", "4", + ], + code="HERMES_MODEL_CALL_FAILED", + timeout=200, + ) + if marker not in output: + fail("HERMES_MODEL_RESPONSE_INVALID") + return {"completed": True, "response_retained": False} + + +def hermes_tool_call(config: dict[str, Any], marker: str) -> dict[str, Any]: + proof = f"{HERMES_HOME}/labnow-p7-tool-proof" + subprocess.run( + ["docker", "exec", config["workspace_container"], "rm", "-f", proof], + check=False, + stdin=subprocess.DEVNULL, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ) + try: + output = p6.command( + [ + "docker", "exec", config["workspace_container"], "timeout", "--signal=TERM", + "--kill-after=10s", "180s", "start-labnow-hermes.sh", "-z", + f"Use the terminal tool to run: printf {marker} > {proof}. Then reply {marker}_DONE only.", + "--ignore-rules", "--max-turns", "10", "-t", "terminal", + ], + code="HERMES_TOOL_CALL_FAILED", + timeout=200, + ) + proof_value = p6.command(["docker", "exec", config["workspace_container"], "cat", proof], code="HERMES_TOOL_PROOF_MISSING").strip() + if proof_value != marker or not output: + fail("HERMES_TOOL_PROOF_INVALID") + return {"completed": True, "tool_observed": True, "response_retained": False} + finally: + subprocess.run( + ["docker", "exec", config["workspace_container"], "rm", "-f", proof], + check=False, + stdin=subprocess.DEVNULL, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ) + + +def stream_call(port: int, ca_file: str, key: str, model: str) -> dict[str, Any]: + context = ssl.create_default_context(cafile=ca_file) + payload = json.dumps( + {"model": model, "messages": [{"role": "user", "content": "Reply P7_STREAM_OK only."}], "stream": True}, + separators=(",", ":"), + ).encode("utf-8") + request = urllib.request.Request( + f"https://127.0.0.1:{port}/chat/completions", + data=payload, + method="POST", + headers={"Accept": "text/event-stream", "Content-Type": "application/json", "Authorization": f"Bearer {key}"}, + ) + frames = 0 + done = False + try: + with urllib.request.urlopen(request, timeout=150, context=context) as response: + if response.status != 200: + fail("STREAM_HTTP_FAILED") + for raw in response: + line = raw.decode("utf-8").strip() + if not line.startswith("data:"): + continue + data = line[5:].strip() + if data == "[DONE]": + done = True + continue + value = json.loads(data) + if isinstance(value, dict) and isinstance(value.get("choices"), list): + frames += 1 + except (urllib.error.URLError, TimeoutError, OSError, UnicodeDecodeError, json.JSONDecodeError) as exc: + raise DriverError("STREAM_CALL_FAILED") from exc + if not done or frames < 1: + fail("STREAM_STRUCTURE_INVALID") + return {"event_frames": frames, "done": True, "response_retained": False} + + +def capture_surfaces(config: dict[str, Any], workspace_summary: dict[str, Any], generation: int) -> list[str]: + surface = Path(config["surface_dir"]) + surface.mkdir(mode=0o700, parents=True, exist_ok=True) + topology_logs = p6.compose(config, "logs", "--no-color", code="TOPOLOGY_LOG_CAPTURE_FAILED", timeout=120) + private_write(surface / f"topology-g{generation}.log", topology_logs) + workspace_logs = p6.command_combined(["docker", "logs", config["workspace_container"]], code="WORKSPACE_LOG_CAPTURE_FAILED") + private_write(surface / f"workspace-g{generation}.log", workspace_logs) + processes = p6.command(["docker", "top", config["workspace_container"], "-eo", "pid,args"], code="WORKSPACE_PROCESS_CAPTURE_FAILED") + private_write(surface / f"workspace-process-g{generation}.txt", processes) + managed = p6.command( + ["docker", "exec", config["workspace_container"], "cat", f"{HERMES_HOME}/labnow-model-access/config.yaml"], + code="HERMES_MANAGED_CONFIG_CAPTURE_FAILED", + ) + private_write(surface / f"hermes-managed-g{generation}.json", managed) + adapter_status = p6.command(["docker", "exec", config["workspace_container"], "cat", STATUS], code="ADAPTER_STATUS_UNAVAILABLE") + private_write(surface / f"adapter-status-g{generation}.json", adapter_status) + private_write(surface / f"workspace-inspect-g{generation}.json", json.dumps(workspace_summary, sort_keys=True) + "\n") + return [str(surface), str(Path(config["workspace_root"]) / config["owner_a"])] + + +def execute(config: dict[str, Any]) -> dict[str, Any]: + shell_port = p6.published_port(config, "shell", 3002) + hub_port = p6.published_port(config, "launcher", 8000) + gateway_port = p6.published_port(config, "litellm-gateway", 4443) + shell = f"http://127.0.0.1:{shell_port}/console/api" + hub = f"http://127.0.0.1:{hub_port}/studio/hub/api" + hub_token = p6.read_secret(config["hub_token_file"], "HUB_TOKEN_INVALID") + launcher_token = p6.read_secret(config["launcher_token_file"], "LAUNCHER_TOKEN_INVALID") + upstream_key = p6.read_secret(config["upstream_key_file"], "UPSTREAM_KEY_INVALID") + run = config["run_id"].replace("p7-", "")[:12] + + status, connection = p6.http_json( + "POST", f"{shell}/model-access/connections/", headers=p6.shell_headers("a", f"connection-{run}"), + body={"display_name": f"P7 {run}", "provider": "openai", "endpoint": config["upstream_origin"], "api_key": upstream_key}, + ) + p6.require_status(status, {201}, "CONNECTION_CREATE_FAILED", connection) + connection_id = connection.get("data", {}).get("id") if isinstance(connection, dict) else None + if not isinstance(connection_id, str): + fail("CONNECTION_RESPONSE_INVALID") + + status, route = p6.http_json( + "POST", f"{shell}/model-access/routes/", headers=p6.shell_headers("a", f"route-{run}"), + body={"connection_id": connection_id, "display_name": f"P7 route {run}", "upstream_model": config["upstream_model"]}, + ) + p6.require_status(status, {201}, "ROUTE_CREATE_FAILED", route) + route_value = route.get("data", {}) if isinstance(route, dict) else {} + route_id = route_value.get("id") + routed_model = route_value.get("routed_model") + if not isinstance(route_id, str) or not isinstance(routed_model, str): + fail("ROUTE_RESPONSE_INVALID") + + status, binding = p6.http_json( + "POST", f"{shell}/model-access/bindings/", headers=p6.shell_headers("a", f"binding-{run}"), + body={"workspace_id": config["server_name"], "route_id": route_id, "image": config["workspace_image"]}, + ) + p6.require_status(status, {201}, "BINDING_CREATE_FAILED", binding) + binding_value = binding.get("data", {}) if isinstance(binding, dict) else {} + binding_id = binding_value.get("binding_id") + if not isinstance(binding_id, str) or binding_value.get("adapter_id") != "hermes": + fail("BINDING_RESPONSE_INVALID") + + spawn_body = { + "tier": "basic", + "image": config["workspace_image"], + "serverName": config["server_name"], + "model_access": {"contract_version": "v1alpha1", "binding_id": binding_id}, + } + status, spawn_response = p6.http_json( + "POST", f"{shell}/hub/spawn/", headers=p6.shell_headers("a", f"spawn-g1-{run}"), body=spawn_body, timeout=45, + ) + p6.require_status(status, {201, 202}, "SHELL_SPAWN_FAILED", spawn_response) + server_snapshot = p6.wait_hub_server(hub, hub_token, config["owner_a"], config["server_name"], running=True) + material_dir_1, manifest_1, key_1 = p6.material(config) + _, workspace_summary_1 = assert_workspace(config, key_1) + p6.data_plane(gateway_port, config["ca_file"], key_1, accepted=True) + if key_1 in json.dumps(server_snapshot, sort_keys=True): + fail("RUNTIME_KEY_LEAKED_TO_HUB_API") + generation_1 = manifest_1["generation"] + from_time = p6.usage_time(datetime.now(timezone.utc) - timedelta(minutes=5)) + model_summary = hermes_model_call(config, "P7_HERMES_OK") + stream_summary = stream_call(gateway_port, config["ca_file"], key_1, manifest_1["default_model"]) + tool_summary = hermes_tool_call(config, "P7_HERMES_TOOL_OK") + to_time = p6.usage_time(datetime.now(timezone.utc) + timedelta(minutes=5)) + usage_count, _ = p6.usage_check(shell, config, routed_model, from_time, to_time) + scan_roots = capture_surfaces(config, workspace_summary_1, generation_1) + + status, _ = p6.http_json( + "POST", f"{shell}/hub/stop/", headers=p6.shell_headers("a", f"stop-g1-{run}"), + body={"serverName": config["server_name"]}, timeout=45, + ) + p6.require_status(status, {200, 202, 204}, "SHELL_STOP_FAILED") + p6.wait_hub_server(hub, hub_token, config["owner_a"], config["server_name"], running=False) + if material_dir_1.exists(): + fail("GENERATION_1_MATERIAL_REMAINS") + p6.wait_rejected(gateway_port, config["ca_file"], key_1) + + status, restart_response = p6.http_json( + "POST", f"{shell}/hub/spawn/", headers=p6.shell_headers("a", f"spawn-g2-{run}"), body=spawn_body, timeout=45, + ) + p6.require_status(status, {201, 202}, "SHELL_RESTART_FAILED", restart_response) + p6.wait_hub_server(hub, hub_token, config["owner_a"], config["server_name"], running=True) + material_dir_2, manifest_2, key_2 = p6.material(config) + _, workspace_summary_2 = assert_workspace(config, key_2) + if manifest_2["generation"] <= generation_1 or key_2 == key_1: + fail("GENERATION_NOT_ADVANCED") + p6.data_plane(gateway_port, config["ca_file"], key_2, accepted=True) + p6.wait_rejected(gateway_port, config["ca_file"], key_1) + restart_model = hermes_model_call(config, "P7_HERMES_RESTART_OK") + scan_roots.extend(capture_surfaces(config, workspace_summary_2, manifest_2["generation"])) + + late_body = { + "contract_version": "v1alpha1", "workspace_id": config["server_name"], + "generation": generation_1, "reason": "reconciled", + } + status, _ = p6.http_json( + "POST", + f"{shell}/internal/model-access/v1alpha1/runtime-leases/{urllib.parse.quote(manifest_1['lease_id'], safe='')}:release/", + headers={"Authorization": f"Bearer {launcher_token}", "Idempotency-Key": f"late-{run}", "X-Request-Id": f"late-{run}"}, + body=late_body, + ) + p6.require_status(status, {409}, "LATE_RELEASE_NOT_REJECTED") + p6.data_plane(gateway_port, config["ca_file"], key_2, accepted=True) + + status, _ = p6.http_json( + "DELETE", f"{shell}/hub/delete/", headers=p6.shell_headers("a", f"delete-g2-{run}"), + body={"serverName": config["server_name"], "remove": True}, timeout=45, + ) + p6.require_status(status, {200, 202, 204}, "SHELL_DELETE_FAILED") + p6.wait_hub_server(hub, hub_token, config["owner_a"], config["server_name"], running=False) + if material_dir_2.exists(): + fail("GENERATION_2_MATERIAL_REMAINS") + p6.wait_rejected(gateway_port, config["ca_file"], key_2) + active = p6.psql( + config, + "SELECT count(*) FROM model_access.runtime_leases WHERE owner_id='" + + config["owner_a"].replace("'", "''") + + "' AND workspace_id='" + + config["server_name"].replace("'", "''") + + "' AND state IN ('issued','active','revoking')", + ) + if active != "0": + fail("ACTIVE_LEASE_REMAINS") + if p6.psql(config, "SELECT coalesce(to_regclass('model_access.usage')::text,'absent')") != "absent": + fail("USAGE_BODY_PERSISTENCE_TABLE_PRESENT") + + pattern_file = Path(os.environ.get("P7_SECRET_PATTERN_FILE", "")) + if not pattern_file.is_absolute(): + fail("SECRET_PATTERN_PATH_INVALID") + p6.write_patterns(config, pattern_file, [key_1, key_2]) + scan_roots = sorted(set(root for root in scan_roots if Path(root).exists())) + if not scan_roots: + fail("SCAN_ROOT_MISSING") + + return { + "schema_version": "p7-product-chain-report/v1", + "result": "passed", + "content_redacted": True, + "checks": { + "console_mouse": "passed", + "binding_payload": "passed", + "jupyterhub_dockerspawner": "passed", + "launcher_claim_activate_release": "passed", + "hermes_apply_probe_readiness": "passed", + "model_call": "passed", + "stream": "passed", + "tool": "passed", + "usage": "passed", + "owner_negative": "passed", + "prompt_response_absent": "passed", + "revoke": "passed", + "generation_restart": "passed", + "late_release": "passed", + "delete": "passed", + "zero_active_leases": "passed", + }, + "console": { + "mouse_evidence": config["shell_ui_evidence"]["status"], + "shell_commit": config["shell_ui_evidence"]["commit"], + "live_spawn_reference": ["binding_id", "contract_version"], + }, + "binding": { + "contract_version": "v1alpha1", "workspace_id": config["server_name"], + "binding_id": binding_id, "route_id": route_id, "adapter_id": "hermes", + "payload_fields": ["binding_id", "contract_version"], + }, + "runtime": { + "hub_api": "live", "docker_daemon": "real", "workspace_image": config["workspace_image"], + "generation_1": generation_1, "generation_2": manifest_2["generation"], + "mounts": "readonly", "adapter_phase": "ready", + }, + "data_plane": { + "model_call": model_summary, "stream": stream_summary, "tool": tool_summary, + "restart_model_call": restart_model, + }, + "usage": { + "row_count": usage_count, "fields": sorted(p6.ALLOWED_USAGE_FIELDS), + "owner_negative": "isolated", "body_fields_absent": True, "persistence_table": "absent", + }, + "lifecycle": { + "old_key_after_stop": "rejected", "new_key_after_restart": "accepted", + "old_key_after_restart": "rejected", "late_old_release": "rejected_409", + "new_key_after_delete": "rejected", "active_lease_count": 0, + }, + "scan_roots": scan_roots, + } + + +def main() -> int: + config_path = Path(os.environ.get("P7_PRODUCT_CONFIG_FILE", "")) + report_path = Path(os.environ.get("P7_PRODUCT_REPORT_FILE", "")) + try: + config = load_config(config_path) + report = execute(config) + except (DriverError, p6.DriverError) as exc: + code = str(exc) + if report_path.is_absolute(): + private_write( + report_path, + json.dumps({"schema_version": "p7-product-chain-report/v1", "result": "failed", "content_redacted": True, "code": code}, separators=(",", ":")) + "\n", + ) + print(f"P7_ERROR:{code}", file=sys.stderr) + return 1 + if not report_path.is_absolute(): + print("P7_ERROR:PRODUCT_REPORT_PATH_INVALID", file=sys.stderr) + return 1 + private_write(report_path, json.dumps(report, separators=(",", ":")) + "\n") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/docker_hermes/p7/scripts/p7-runner.sh b/docker_hermes/p7/scripts/p7-runner.sh index d034856..f220174 100755 --- a/docker_hermes/p7/scripts/p7-runner.sh +++ b/docker_hermes/p7/scripts/p7-runner.sh @@ -1,13 +1,15 @@ #!/usr/bin/env bash -# P7 local-only coordinator. It validates immutable product provenance before -# any container is started and fails closed until all three P7 product inputs -# (Hermes renderer, Shell catalogue, and fixed workspace image) are present. +# P7 local-only Hermes coordinator. It reuses the P6 live topology while +# replacing only the Workspace image, trusted Shell catalogue and product +# checks. All credentials stay in run-scoped 0400/0600 files. set -euo pipefail script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" p7_dir="$(cd "${script_dir}/.." && pwd)" +repo_root="$(cd "${p7_dir}/../.." && pwd)" +p6_dir="${repo_root}/docker_openclaw/p6" -die() { printf 'P7_ERROR:%s\n' "$1" >&2; exit "${2:-1}"; } +die() { printf 'P7_ERROR:%s\n' "$1" >&2; return "${2:-1}"; } sha256() { shasum -a 256 "$1" | awk '{print $1}'; } run_id() { python3 -c 'import secrets; print("p7-" + secrets.token_hex(16))'; } mode_of() { stat -f '%Lp' "$1" 2>/dev/null || stat -c '%a' "$1"; } @@ -30,15 +32,16 @@ P7_RUN_ID="${P7_RUN_ID:-$(run_id)}" export P7_RUN_ID artifact_dir="${P7_ARTIFACTS_DIR:-${p7_dir}/artifacts}" work_dir="${P7_WORK_DIR:-${p7_dir}/.p7-work/${P7_RUN_ID}}" +[[ "$artifact_dir" = /* && "$work_dir" = /* && ${#work_dir} -gt 12 ]] || die "RUNTIME_PATH_INVALID" 64 +case "$work_dir" in /|"$repo_root"|"$p7_dir") die "RUNTIME_PATH_INVALID" 64 ;; esac +export P7_ARTIFACTS_DIR="$artifact_dir" P7_WORK_DIR="$work_dir" mkdir -p "$artifact_dir"; chmod 700 "$artifact_dir" -if [[ "$action" == golden || "$action" == cleanup ]]; then - mkdir -p "$work_dir"; chmod 700 "$work_dir" -fi report="${artifact_dir}/p7-${action}-${P7_RUN_ID}.json" validate_shape() { jq -e ' - type == "object" + . as $root + | type == "object" and .schema_version == "p7-inputs/v1" and .contract_version == "v1alpha1" and .contract_release == "0.1.0-rc.1" and .control_commit == "06c49f26642c7e39a118aedad1395197f2bd91db" @@ -46,76 +49,215 @@ validate_shape() { and .phase.branch == "dev/che-568-hermes-console-experience" and .phase.base_commit == "45c38585a0ca889f6a20aebfdf3b13a01d369ac2" and (.repositories | keys | sort) == ["hermes_source","lab_dev","labnow_launcher","labnow_open","labnow_shell"] + and (["lab_dev","labnow_open","labnow_shell","labnow_launcher"] | all(. as $name | + ($root.repositories[$name] | keys | sort) == ["commit","path","runtime_commit"] + and ($root.repositories[$name].path | type == "string" and startswith("/")) + and ($root.repositories[$name].commit | type == "string" and test("^[0-9a-f]{40}$")) + and ($root.repositories[$name].runtime_commit | type == "string" and test("^[0-9a-f]{40}$")))) + and (.repositories.hermes_source | keys | sort) == ["commit","path","repository"] + and (.repositories.hermes_source.path | type == "string" and startswith("/")) + and (.repositories.hermes_source.repository | type == "string" and startswith("https://")) + and (.repositories.hermes_source.commit | type == "string" and test("^[0-9a-f]{40}$")) + and (.repositories.labnow_open.commit == "533b3c5bd0742a77a1cccfc4a30b818271751213") + and (.repositories.labnow_open.runtime_commit == "8c56966c4b6be12702d4397aad6b1a153b87d053") + and (.repositories.labnow_launcher.commit == "c84edea3e051d561f28d9f99235563cf491aaeb2") + and (.repositories.labnow_launcher.runtime_commit == .repositories.labnow_launcher.commit) and (.images | keys | sort) == ["hermes","launcher","litellm","shell","workspace"] - and (.base_images | keys | sort) == ["build","runtime"] - and (.runtime | keys | sort) == ["p1_env_file"] - and ([.repositories[] | .commit] | all(type == "string" and test("^[0-9a-f]{40}$"))) + and (.support_images | keys | sort) == ["nginx","postgres","redis"] + and ([.images[] | .image_id] + [.support_images[] | .image_id] | all(type == "string" and test("^sha256:[0-9a-f]{64}$"))) + and ([.images[] | .repo_digest] + [.support_images[] | .repo_digest] | all(type == "string" and test("^[^[:space:]]+@sha256:[0-9a-f]{64}$"))) and (.images.hermes.ref | test("^quay\\.io/labnow/hermes:p7-[0-9a-f]{12}$")) - and ([.images[] | .image_id] | all(type == "string" and test("^sha256:[0-9a-f]{64}$"))) - and (.images.hermes.provenance == "local_build") - and (.images.hermes.source_repository == "hermes_source") - and (.images.hermes.source_commit == .repositories.hermes_source.commit) + and .images.hermes.provenance == "local_build" + and .images.hermes.source_repository == "hermes_source" + and .images.hermes.source_commit == .repositories.hermes_source.commit + and (.images.workspace.ref | test("^quay\\.io/labnow/labnow-open@sha256:[0-9a-f]{64}$")) + and (.images.shell.ref | test("^quay\\.io/labnow/labnow-shell@sha256:[0-9a-f]{64}$")) + and (.images.launcher.ref | test("^quay\\.io/labnow/labnow-launcher@sha256:[0-9a-f]{64}$")) + and (.images.litellm.ref | test("^quay\\.io/labnow/litellm@sha256:[0-9a-f]{64}$")) + and (["workspace","shell","launcher"] | all(. as $name | + $root.images[$name].provenance == "local_build" + and ($root.images[$name].source_repository | type == "string") + and $root.images[$name].source_commit == $root.repositories[$root.images[$name].source_repository].runtime_commit)) + and .images.litellm.provenance == "repo_digest" + and ([.images.workspace,.images.shell,.images.launcher,.images.litellm] | all(.ref == .repo_digest)) + and (.base_images | keys | sort) == ["build","runtime"] and ([.base_images[]] | all(type == "string" and test("^quay\\.io/labnow/(node|base)@sha256:[0-9a-f]{64}$"))) + and (.runtime | keys | sort) == ["p1_env_file"] and (.runtime.p1_env_file | type == "string" and startswith("/")) ' "$input" >/dev/null || die "INPUT_SCHEMA_INVALID" 68 } assert_repository() { - local name="$1" path expected actual status + local name="$1" path commit runtime_commit actual status changed path="$(jq -er ".repositories.${name}.path" "$input")" - expected="$(jq -er ".repositories.${name}.commit" "$input")" + commit="$(jq -er ".repositories.${name}.commit" "$input")" + runtime_commit="$(jq -er ".repositories.${name}.runtime_commit" "$input")" [[ -d "$path/.git" ]] || die "REPOSITORY_UNAVAILABLE" 69 actual="$(git -C "$path" rev-parse HEAD)" - [[ "$actual" == "$expected" ]] || die "REPOSITORY_COMMIT_MISMATCH" 70 + [[ "$actual" == "$commit" ]] || die "REPOSITORY_COMMIT_MISMATCH" 70 status="$(git -C "$path" status --porcelain=v1 --untracked-files=no)" [[ -z "$status" ]] || die "REPOSITORY_TRACKED_TREE_DIRTY" 71 + git -C "$path" merge-base --is-ancestor "$runtime_commit" "$commit" || die "RUNTIME_COMMIT_NOT_ANCESTOR" 70 + if [[ "$runtime_commit" != "$commit" ]]; then + changed="$(git -C "$path" diff --name-only "$runtime_commit..$commit")" + [[ -n "$changed" ]] || die "RUNTIME_DELIVERY_DELTA_MISSING" 70 + if grep -Ev '^(doc|docs|development-docs)/|(^|/)README\.md$' <<<"$changed" >/dev/null; then + die "RUNTIME_DELIVERY_DELTA_NOT_DOCUMENTATION" 70 + fi + fi +} + +assert_hermes_source() { + local path expected_repository expected_commit + path="$(jq -er '.repositories.hermes_source.path' "$input")" + expected_repository="$(jq -er '.repositories.hermes_source.repository' "$input")" + expected_commit="$(jq -er '.repositories.hermes_source.commit' "$input")" + [[ -d "$path/.git" ]] || die "HERMES_SOURCE_UNAVAILABLE" 69 + [[ "$(git -C "$path" rev-parse HEAD)" == "$expected_commit" ]] || die "HERMES_SOURCE_COMMIT_MISMATCH" 70 + [[ "$(git -C "$path" remote get-url origin)" == "$expected_repository" ]] || die "HERMES_SOURCE_REMOTE_MISMATCH" 70 + [[ -z "$(git -C "$path" status --porcelain=v1 --untracked-files=no)" ]] || die "HERMES_SOURCE_TRACKED_TREE_DIRTY" 71 +} + +assert_image() { + local section="$1" name="$2" ref expected_id expected_digest actual_id digests + ref="$(jq -er ".${section}.${name}.ref" "$input")" + expected_id="$(jq -er ".${section}.${name}.image_id" "$input")" + expected_digest="$(jq -er ".${section}.${name}.repo_digest" "$input")" + actual_id="$(docker image inspect --format '{{.Id}}' "$ref" 2>/dev/null)" || die "IMAGE_UNAVAILABLE" 72 + [[ "$actual_id" == "$expected_id" ]] || die "IMAGE_ID_MISMATCH" 72 + digests="$(docker image inspect --format '{{join .RepoDigests "\n"}}' "$ref")" + grep -Fqx "$expected_digest" <<<"$digests" || die "IMAGE_DIGEST_MISMATCH" 72 +} + +assert_images() { + local name + for name in hermes litellm workspace shell launcher; do assert_image images "$name" || return $?; done + for name in postgres redis nginx; do assert_image support_images "$name" || return $?; done + local hermes_ref + hermes_ref="$(jq -er '.images.hermes.ref' "$input")" + [[ "$(docker image inspect --format '{{ index .Config.Labels "org.opencontainers.image.revision" }}' "$hermes_ref")" == "$(jq -er '.repositories.hermes_source.commit' "$input")" ]] || die "HERMES_IMAGE_PROVENANCE_MISMATCH" 72 + [[ "$(docker image inspect --format '{{ index .Config.Labels "io.labnow.hermes.build-base" }}' "$hermes_ref")" == "$(jq -er '.base_images.build' "$input")" ]] || die "HERMES_BUILD_BASE_MISMATCH" 72 + [[ "$(docker image inspect --format '{{ index .Config.Labels "io.labnow.hermes.runtime-base" }}' "$hermes_ref")" == "$(jq -er '.base_images.runtime' "$input")" ]] || die "HERMES_RUNTIME_BASE_MISMATCH" 72 + docker image inspect "$(jq -er '.base_images.build' "$input")" "$(jq -er '.base_images.runtime' "$input")" >/dev/null 2>&1 || die "HERMES_BASE_IMAGE_UNAVAILABLE" 72 +} + +assert_runtime_input() { + local env_file + env_file="$(jq -er '.runtime.p1_env_file' "$input")" + secure_file "$env_file" || return $? + python3 - "$env_file" <<'PY' +import sys +from pathlib import Path +required = {"UPSTREAM_API_KEY", "UPSTREAM_BASE_URL", "UPSTREAM_MODEL"} +values = {} +for raw in Path(sys.argv[1]).read_text(encoding="utf-8").splitlines(): + line = raw.strip() + if line and not line.startswith("#") and "=" in line: + key, value = line.split("=", 1) + values[key] = value +raise SystemExit(0 if all(values.get(key) for key in required) else 1) +PY + [[ $? == 0 ]] || die "P1_ENV_INCOMPLETE" 73 } preflight() { - validate_shape - secure_file "$(jq -er '.runtime.p1_env_file' "$input")" + validate_shape || return $? local repo - for repo in lab_dev labnow_open labnow_shell labnow_launcher hermes_source; do assert_repository "$repo"; done + for repo in lab_dev labnow_open labnow_shell labnow_launcher; do assert_repository "$repo" || return $?; done + assert_hermes_source || return $? [[ "$(git -C "$(jq -er '.repositories.lab_dev.path' "$input")" branch --show-current)" == "dev/che-568-hermes-console-experience" ]] || die "PHASE_BRANCH_MISMATCH" 70 git -C "$(jq -er '.repositories.lab_dev.path' "$input")" merge-base --is-ancestor "$(jq -er '.phase.base_commit' "$input")" HEAD || die "PHASE_BASE_NOT_ANCESTOR" 70 - [[ "$(git -C "$(jq -er '.repositories.hermes_source.path' "$input")" remote get-url origin)" == "$(jq -er '.repositories.hermes_source.repository' "$input")" ]] || die "HERMES_SOURCE_REMOTE_MISMATCH" 70 - local ref expected actual - ref="$(jq -er '.images.hermes.ref' "$input")"; expected="$(jq -er '.images.hermes.image_id' "$input")" - actual="$(docker image inspect --format '{{.Id}}' "$ref" 2>/dev/null)" || die "HERMES_IMAGE_UNAVAILABLE" 72 - [[ "$actual" == "$expected" ]] || die "HERMES_IMAGE_ID_MISMATCH" 72 - [[ "$(docker image inspect --format '{{ index .Config.Labels "org.opencontainers.image.revision" }}' "$ref")" == "$(jq -er '.repositories.hermes_source.commit' "$input")" ]] || die "HERMES_IMAGE_PROVENANCE_MISMATCH" 72 - [[ "$(docker image inspect --format '{{ index .Config.Labels "io.labnow.hermes.build-base" }}' "$ref")" == "$(jq -er '.base_images.build' "$input")" ]] || die "HERMES_BUILD_BASE_MISMATCH" 72 - [[ "$(docker image inspect --format '{{ index .Config.Labels "io.labnow.hermes.runtime-base" }}' "$ref")" == "$(jq -er '.base_images.runtime' "$input")" ]] || die "HERMES_RUNTIME_BASE_MISMATCH" 72 + assert_images || return $? + assert_runtime_input || return $? } write_report() { local result="$1" phase="$2" reason="${3:-}" extra="${4:-{}}" tmp tmp="$(mktemp "${artifact_dir}/.p7-report.XXXXXX")" - jq -n --arg run_id "$P7_RUN_ID" --arg result "$result" --arg phase "$phase" --arg reason "$reason" --arg input_sha "$(sha256 "$input")" --arg tested_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" --argjson provenance "$(jq -c '{contract_version,contract_release,control_commit,review_policy_commit,phase,repositories:(.repositories|with_entries(.value={commit:.value.commit})),images}' "$input")" --argjson extra "$extra" \ + jq -n --arg run_id "$P7_RUN_ID" --arg result "$result" --arg phase "$phase" --arg reason "$reason" --arg input_sha "$(sha256 "$input")" --arg tested_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" --argjson provenance "$(jq -c '{contract_version,contract_release,control_commit,review_policy_commit,phase,repositories:(.repositories|with_entries(.value=if .key=="hermes_source" then {commit:.value.commit,repository:.value.repository} else {commit:.value.commit,runtime_commit:.value.runtime_commit} end)),images,support_images,base_images}' "$input")" --argjson extra "$extra" \ '{schema_version:"p7-report/v1",run_id:$run_id,result:$result,phase:$phase,input_sha256:$input_sha,tested_at:$tested_at,content_redacted:true} + $provenance + $extra + (if $reason == "" then {} else {reason:$reason} end)' > "$tmp" chmod 600 "$tmp"; mv -f "$tmp" "$report" } -cleanup() { - local env="${work_dir}/runtime.env" - if [[ -f "$env" && ! -L "$env" ]]; then docker compose --project-name "p7-${P7_RUN_ID#p7-}" --env-file "$env" -f "${p7_dir}/docker-compose.runtime.yml" down --volumes --remove-orphans >/dev/null 2>&1 || true; fi - rm -rf "$work_dir" +compose() { + local short="${P7_RUN_ID#p7-}" + short="${short:0:12}" + docker compose --project-name "p6-runtime-${short}" --env-file "${work_dir}/runtime.env" -f "${p6_dir}/docker-compose.runtime.yml" "$@" +} + +cleanup_runtime() { + local workspace="p6w-${P7_RUN_ID#p7-}" + workspace="${workspace:0:12}-p6user-${P7_RUN_ID#p7-}" + if [[ -f "${work_dir}/config/driver-config.json" && ! -L "${work_dir}/config/driver-config.json" ]]; then + workspace="$(jq -r '.workspace_container // empty' "${work_dir}/config/driver-config.json" 2>/dev/null || true)" + fi + [[ -z "$workspace" ]] || docker rm -f "$workspace" >/dev/null 2>&1 || true + if [[ -f "${work_dir}/runtime.env" && ! -L "${work_dir}/runtime.env" ]]; then compose down --volumes --remove-orphans >/dev/null 2>&1 || true; fi + rm -rf -- "$work_dir" +} + +assert_cleanup() { + local short="${P7_RUN_ID#p7-}" project="p6-runtime-${P7_RUN_ID#p7-}" + short="${short:0:12}"; project="p6-runtime-${short}" + [[ -z "$(docker container ls -aq --filter "label=com.docker.compose.project=${project}")" ]] || die "TOPOLOGY_CONTAINER_REMAINS" 79 + [[ -z "$(docker volume ls -q --filter "label=com.docker.compose.project=${project}")" ]] || die "TOPOLOGY_VOLUME_REMAINS" 79 + ! docker network inspect "p6net-${short}" >/dev/null 2>&1 || die "TOPOLOGY_NETWORK_REMAINS" 79 + [[ ! -e "$work_dir" ]] || die "TOPOLOGY_RUNTIME_MATERIAL_REMAINS" 79 +} + +render_summary() { + jq -n --arg run_id "$P7_RUN_ID" --arg compose_sha256 "$(sha256 "${p6_dir}/docker-compose.runtime.yml")" --arg component_sha256 "$(sha256 "${p7_dir}/docker-compose.runtime.yml")" --argjson images "$(jq -c '.images' "$input")" \ + '{schema_version:"p7-render/v1",run_id:$run_id,compose_sha256:$compose_sha256,hermes_component_sha256:$component_sha256,topology:["litellm","shell","jupyterhub","launcher","hermes-workspace"],workspace_creation:"live_dockerspawner",images:$images,content_redacted:true}' > "${artifact_dir}/p7-render-${P7_RUN_ID}.json" + chmod 600 "${artifact_dir}/p7-render-${P7_RUN_ID}.json" +} + +security_scan() { + local patterns="$1" status + shift + secure_file "$patterns" || return $? + [[ -s "$patterns" && $# -gt 0 ]] || die "SECRET_SCAN_INPUT_INVALID" 74 + set +e + rg --fixed-strings --files-with-matches --glob '!secret-patterns' -f "$patterns" "$@" >/dev/null 2>&1 + status=$? + set -e + case "$status" in 1) return 0 ;; 0) die "SECRET_PATTERN_MATCH" 75 ;; *) die "SECRET_SCAN_FAILED" 75 ;; esac } case "$action" in - validate-input) if validate_shape; then write_report passed completed; else write_report failed precondition_failed input_validation; exit 1; fi ;; - preflight) if preflight; then write_report passed completed; else write_report failed precondition_failed preflight; exit 1; fi ;; + validate-input) + if validate_shape; then write_report passed completed; else write_report failed precondition_failed input_validation; exit 1; fi + ;; + preflight) + if preflight; then write_report passed completed; else write_report failed precondition_failed preflight; exit 1; fi + ;; render) - if ! preflight; then write_report failed precondition_failed preflight; exit 1; fi - jq -n --arg run_id "$P7_RUN_ID" --arg compose_sha256 "$(sha256 "${p7_dir}/docker-compose.runtime.yml")" --arg image "$(jq -er '.images.hermes.ref' "$input")" '{schema_version:"p7-render/v1",run_id:$run_id,compose_sha256:$compose_sha256,hermes_image:$image,content_redacted:true}' > "${artifact_dir}/p7-render-${P7_RUN_ID}.json" - chmod 600 "${artifact_dir}/p7-render-${P7_RUN_ID}.json"; write_report passed completed + if preflight && render_summary; then write_report passed completed "" "$(jq -n --arg path "${artifact_dir}/p7-render-${P7_RUN_ID}.json" --arg sha "$(sha256 "${artifact_dir}/p7-render-${P7_RUN_ID}.json")" '{render:{path:$path,sha256:$sha}}')"; else write_report failed precondition_failed render; exit 1; fi ;; golden) - # No compatibility fallback to the P6 OpenClaw runner is permitted. The - # Hermes renderer/catalogue image must be supplied by the two owning P7 - # repositories, otherwise a real lifecycle run cannot be claimed. - if ! preflight; then write_report failed precondition_failed preflight; exit 1; fi - write_report failed blocked "HERMES_PRODUCT_CHAIN_NOT_AVAILABLE"; exit 1 + if ! preflight || ! render_summary; then write_report failed precondition_failed preflight; exit 1; fi + mkdir -p "$work_dir"; chmod 700 "$work_dir" + trap cleanup_runtime EXIT + if ! "${script_dir}/p7-prepare-runtime.py"; then write_report failed topology_prepare_failed prepare; exit 1; fi + if ! compose up -d --wait >/dev/null; then write_report failed topology_provision_failed compose; exit 1; fi + write_report passed provisioned + product_report="${work_dir}/p7-product-chain.json" + pattern_file="${work_dir}/secret-patterns" + if ! P7_PRODUCT_CONFIG_FILE="${work_dir}/config/driver-config.json" P7_PRODUCT_REPORT_FILE="$product_report" P7_SECRET_PATTERN_FILE="$pattern_file" "${script_dir}/p7-product-chain.py"; then + write_report failed golden_chain_failed product_chain; exit 1 + fi + jq -e '.schema_version == "p7-product-chain-report/v1" and .result == "passed" and .content_redacted == true and ([.checks.console_mouse,.checks.binding_payload,.checks.jupyterhub_dockerspawner,.checks.launcher_claim_activate_release,.checks.hermes_apply_probe_readiness,.checks.model_call,.checks.stream,.checks.tool,.checks.usage,.checks.owner_negative,.checks.prompt_response_absent,.checks.revoke,.checks.generation_restart,.checks.late_release,.checks.delete,.checks.zero_active_leases] | all(. == "passed"))' "$product_report" >/dev/null || { write_report failed golden_chain_failed report_validation; exit 1; } + retained_product="${artifact_dir}/p7-product-${P7_RUN_ID}.json" + cp "$product_report" "$retained_product"; chmod 600 "$retained_product" + scan_roots=("$retained_product") + while IFS= read -r root; do [[ -e "$root" && ! -L "$root" ]] && scan_roots+=("$root"); done < <(jq -r '.scan_roots[]' "$product_report") + if ! security_scan "$pattern_file" "${scan_roots[@]}"; then write_report failed security_scan_failed secret_scan; exit 1; fi + product_sha="$(sha256 "$retained_product")" + checks="$(jq -c '.checks' "$retained_product")" + cleanup_runtime; assert_cleanup + trap - EXIT + write_report passed completed "" "$(jq -n --arg product "$retained_product" --arg product_sha "$product_sha" --argjson checks "$checks" '{checks:$checks,product_report:{path:$product,sha256:$product_sha},cleanup:{result:"passed",resources:"absent"}}')" + ;; + cleanup) + cleanup_runtime; assert_cleanup; write_report passed completed "" '{"cleanup":{"result":"passed","resources":"absent"}}' ;; - cleanup) cleanup; write_report passed completed ;; esac diff --git a/docker_hermes/p7/scripts/test-p7-gates.sh b/docker_hermes/p7/scripts/test-p7-gates.sh index 760f349..503bf3b 100755 --- a/docker_hermes/p7/scripts/test-p7-gates.sh +++ b/docker_hermes/p7/scripts/test-p7-gates.sh @@ -17,6 +17,11 @@ rg -q 'ARG HERMES_BUILD_BASE_IMAGE' "${root}/docker_hermes/hermes.Dockerfile" rg -q 'io.labnow.hermes.runtime-base' "${root}/docker_hermes/hermes.Dockerfile" rg -q 'pull_policy: never' "${root}/docker_hermes/p7/docker-compose.runtime.yml" ! rg -n --glob '!**/test-p7-gates.sh' 'OPENAI_API_KEY:|DEEPSEEK_API_KEY:|:latest' "${root}/docker_hermes/p7" +! rg -q 'HERMES_PRODUCT_CHAIN_NOT_AVAILABLE' "$runner" +rg -q 'p7-product-chain.py' "$runner" +python3 -m py_compile \ + "${root}/docker_hermes/p7/scripts/p7-prepare-runtime.py" \ + "${root}/docker_hermes/p7/scripts/p7-product-chain.py" input="$tmp/invalid.json" printf '%s\n' '{"schema_version":"p7-inputs/v1"}' > "$input"; chmod 600 "$input" @@ -26,13 +31,31 @@ fi valid="$tmp/valid.json" jq ' - .repositories |= with_entries(.value.commit = "0123456789abcdef0123456789abcdef01234567") + .repositories.lab_dev.commit = "0123456789abcdef0123456789abcdef01234567" + | .repositories.lab_dev.runtime_commit = .repositories.lab_dev.commit + | .repositories.labnow_shell.commit = "0123456789abcdef0123456789abcdef01234567" + | .repositories.labnow_shell.runtime_commit = .repositories.labnow_shell.commit + | .repositories.hermes_source.repository = "https://example.invalid/hermes.git" + | .repositories.hermes_source.commit = "0123456789abcdef0123456789abcdef01234567" | .images |= with_entries(.value.image_id = "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa") + | .support_images |= with_entries(.value.image_id = "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa") | .images.hermes.ref = "quay.io/labnow/hermes:p7-0123456789ab" + | .images.hermes.repo_digest = "quay.io/labnow/hermes@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" | .images.hermes.source_commit = .repositories.hermes_source.commit - | .images.workspace.source_commit = .repositories.labnow_open.commit - | .images.shell.source_commit = .repositories.labnow_shell.commit - | .images.launcher.source_commit = .repositories.labnow_launcher.commit + | .images.litellm.ref = "quay.io/labnow/litellm@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" + | .images.litellm.repo_digest = .images.litellm.ref + | .images.workspace.ref = "quay.io/labnow/labnow-open@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" + | .images.workspace.repo_digest = .images.workspace.ref + | .images.workspace.source_commit = .repositories.labnow_open.runtime_commit + | .images.shell.ref = "quay.io/labnow/labnow-shell@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" + | .images.shell.repo_digest = .images.shell.ref + | .images.shell.source_commit = .repositories.labnow_shell.runtime_commit + | .images.launcher.ref = "quay.io/labnow/labnow-launcher@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" + | .images.launcher.repo_digest = .images.launcher.ref + | .images.launcher.source_commit = .repositories.labnow_launcher.runtime_commit + | .support_images.postgres.repo_digest = "postgres@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" + | .support_images.redis.repo_digest = "redis@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" + | .support_images.nginx.repo_digest = "nginx@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" | .base_images.build = "quay.io/labnow/node@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" | .base_images.runtime = "quay.io/labnow/base@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" | .runtime.p1_env_file = "/private/tmp/p7-test.env" @@ -45,4 +68,4 @@ chmod 600 "${valid}.mismatch" if P7_ARTIFACTS_DIR="$tmp/artifacts" P7_WORK_DIR="$tmp/work" "$runner" --input "${valid}.mismatch" --validate-input >/dev/null 2>&1; then printf '%s\n' 'P7 provenance mismatch input was accepted' >&2; exit 1 fi -printf '%s\n' 'PASS P7 gates: source pin, local-only compose and invalid input fail closed.' +printf '%s\n' 'PASS P7 gates: source pin, local-only topology, real golden entry and invalid input fail closed.' From fcbcec7526ce9f9fd9f6a3a8717e198345d54269 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Mon, 10 Aug 2026 23:39:43 +0800 Subject: [PATCH 62/87] =?UTF-8?q?fix(hermes):=20=E4=BF=AE=E6=AD=A3=20P7=20?= =?UTF-8?q?=E6=8A=A5=E5=91=8A=E9=BB=98=E8=AE=A4=E8=BD=BD=E8=8D=B7?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_hermes/p7/scripts/p7-runner.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docker_hermes/p7/scripts/p7-runner.sh b/docker_hermes/p7/scripts/p7-runner.sh index f220174..fb0e901 100755 --- a/docker_hermes/p7/scripts/p7-runner.sh +++ b/docker_hermes/p7/scripts/p7-runner.sh @@ -172,7 +172,8 @@ preflight() { } write_report() { - local result="$1" phase="$2" reason="${3:-}" extra="${4:-{}}" tmp + local result="$1" phase="$2" reason="${3:-}" extra="${4:-}" tmp + [[ -n "$extra" ]] || extra='{}' tmp="$(mktemp "${artifact_dir}/.p7-report.XXXXXX")" jq -n --arg run_id "$P7_RUN_ID" --arg result "$result" --arg phase "$phase" --arg reason "$reason" --arg input_sha "$(sha256 "$input")" --arg tested_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" --argjson provenance "$(jq -c '{contract_version,contract_release,control_commit,review_policy_commit,phase,repositories:(.repositories|with_entries(.value=if .key=="hermes_source" then {commit:.value.commit,repository:.value.repository} else {commit:.value.commit,runtime_commit:.value.runtime_commit} end)),images,support_images,base_images}' "$input")" --argjson extra "$extra" \ '{schema_version:"p7-report/v1",run_id:$run_id,result:$result,phase:$phase,input_sha256:$input_sha,tested_at:$tested_at,content_redacted:true} + $provenance + $extra + (if $reason == "" then {} else {reason:$reason} end)' > "$tmp" From 74e44643fdd78050100ce1764caddff15cef8d51 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Tue, 11 Aug 2026 00:23:04 +0800 Subject: [PATCH 63/87] =?UTF-8?q?fix(hermes):=20=E9=97=AD=E5=90=88=20P7=20?= =?UTF-8?q?=E5=8F=82=E6=95=B0=E5=8C=96=E9=BB=84=E9=87=91=E9=93=BE=E9=A2=84?= =?UTF-8?q?=E6=A3=80?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_hermes/p7/README.md | 23 +++++- ...72\344\270\216\351\252\214\350\257\201.md" | 31 ++++++-- docker_hermes/p7/launcher-overlay.Dockerfile | 16 ++++ docker_hermes/p7/p7-inputs.example.json | 4 +- docker_hermes/p7/scripts/p7-runner.sh | 79 +++++++++++-------- docker_hermes/p7/scripts/test-p7-gates.sh | 39 +++++++++ 6 files changed, 148 insertions(+), 44 deletions(-) create mode 100644 docker_hermes/p7/launcher-overlay.Dockerfile diff --git a/docker_hermes/p7/README.md b/docker_hermes/p7/README.md index 4c77464..a582649 100644 --- a/docker_hermes/p7/README.md +++ b/docker_hermes/p7/README.md @@ -27,6 +27,27 @@ build_image_no_tag hermes p7-<12hex> docker_hermes/hermes.Dockerfile \ `io.labnow.hermes.*-base` label 记录实际传入的不可变基础镜像引用。runner 会校验 这些 label 与受限输入一致。 +P7 真实链路发现 P6 Launcher 将 `RuntimeManifest.adapter_id` 固定为 +`openclaw` 后,Launcher 在独立 Phase 分支完成了受信任双 Adapter 修复。全量 +Launcher Dockerfile 会动态下载构建工具,不能作为本轮固定组合的重建入口; +本目录用 `launcher-overlay.Dockerfile` 从已验证的 P6 Launcher 本地 digest +出发,只覆盖固定 Launcher commit 的运行时代码与 Hub 配置: + +```bash +docker build --platform linux/amd64 --provenance=false \ + --build-context launcher=/absolute/path/to/labnow-launcher \ + --build-arg P6_LAUNCHER_IMAGE=quay.io/labnow/labnow-launcher:che-563-openclaw-product-closure-local \ + --build-arg P6_LAUNCHER_BASE_DIGEST=quay.io/labnow/labnow-launcher@sha256:6f9732fda8b86d9bfe4596e848025cc38448da4b17dfea8520e046a32b32e61f \ + --build-arg LAUNCHER_SOURCE_COMMIT=f84a51319d75b99a6b210f19e264904cae07fc8a \ + -t quay.io/labnow/labnow-launcher:che-568-hermes-console-experience-local \ + -f docker_hermes/p7/launcher-overlay.Dockerfile . +``` + +构建前必须回读 P6 tag 的本地 image ID 正是上述 digest,并确认 Launcher +checkout tracked clean 且 HEAD 等于 `LAUNCHER_SOURCE_COMMIT`。overlay 的 OCI +revision 与 `io.labnow.p7.launcher-base` label 会由 runner 回读;产物仍只在本地, +不得把本地 RepoDigest 描述成远端 registry 已发布制品。 + ## 入口与安全 从 `p7-inputs.example.json` 创建权限为 `0400` 或 `0600` 的 Git 忽略 @@ -42,7 +63,7 @@ P7_RUN_ID=p7-<32hex> ./docker_hermes/p7/scripts/p7-runner.sh --input /secure/pat P7_RUN_ID=p7- ./docker_hermes/p7/scripts/p7-runner.sh --input /secure/path/p7-inputs.json --cleanup ``` -`--golden` 会先核验三仓 delivery/runtime commit、Hermes 上游、五个产品镜像、 +`--golden` 会先核验 Open、Dev、Shell、Launcher 四仓 delivery/runtime commit、Hermes 上游、五个产品镜像、 三个 support image 和两个构建基础引用,再复用 P6 的隔离 LiteLLM、Shell、 Launcher/JupyterHub 拓扑。Workspace 只能由 live DockerSpawner 创建;Shell 服务端从准确 Open 产品镜像推导 `hermes` Adapter,Launcher 以 diff --git "a/docker_hermes/p7/evidence/2026-08-10-P7-CHE-568-Hermes\346\236\204\345\273\272\344\270\216\351\252\214\350\257\201.md" "b/docker_hermes/p7/evidence/2026-08-10-P7-CHE-568-Hermes\346\236\204\345\273\272\344\270\216\351\252\214\350\257\201.md" index bf4096c..cadc1c0 100644 --- "a/docker_hermes/p7/evidence/2026-08-10-P7-CHE-568-Hermes\346\236\204\345\273\272\344\270\216\351\252\214\350\257\201.md" +++ "b/docker_hermes/p7/evidence/2026-08-10-P7-CHE-568-Hermes\346\236\204\345\273\272\344\270\216\351\252\214\350\257\201.md" @@ -66,15 +66,34 @@ runtime 环境为 `PYTHONPATH=/opt/hermes:`。该 warning 不改变本次固定 | 固定输入本地 Docker build | 通过;退出 0 | | `docker image inspect` 的 image ID、平台与四个 labels | 通过 | -## 6. 当前结论与后续门禁 +## 6. 初始交付结论 - `lab-dev` P7 固定 Hermes base image 已在本地生成,可交给 `labnow-open` 构建 P7 Workspace 产品镜像。 -- `p7-runner.sh --golden` 在缺少完整产品输入时必须返回 - `P7_ERROR:HERMES_PRODUCT_CHAIN_NOT_AVAILABLE`,不得复用 P6 OpenClaw 证据。 -- 待 `labnow-open`、`labnow-shell` 的准确 P7 commit 和本地产品镜像身份固定后, - 再生成权限为 `0400` 或 `0600` 的 Git 忽略输入,执行 preflight、render 与完整 - Hermes 黄金链。 +- 初始提交在缺少完整产品输入时以 + `P7_ERROR:HERMES_PRODUCT_CHAIN_NOT_AVAILABLE` 失败关闭,没有复用 P6 OpenClaw + 证据;后续 P7-R2 修复提交已经以参数化真实链替换该临时门禁。 +- `labnow-open`、`labnow-shell` 和 Launcher 的准确 P7 commit 与本地产品镜像 + 身份随后已固定;最终完整 Hermes 黄金链以新的 run ID 和报告 SHA-256 + 单独记录,不追写成本节的初始构建事实。 - S0/S1:无。S2:既有 `$PYTHONPATH` 静态 warning;不阻断本 Phase 固定组合验证。 - 本次未读取或处理用户既有 `.DS_Store`、`docs/`、`hermes-chat-screenshot.png`; 未输出凭证明文,未 push,未发布镜像,未部署,未修改 `main` 或 integration。 + +## 7. 2026-08-11 P7-R2 真实链路直接修复 + +- `90328f6a520b0c617689bcbc21576a10f5e34b10` 与 + `fcbcec7526ce9f9fd9f6a3a8717e198345d54269` 已实现参数化真实黄金 runner、 + 四仓 commit、五个产品镜像和 support/base image 的 preflight,以及真实 + Hermes 模型/stream/tool、usage、generation/revoke/cleanup 链。 +- 首次固定拓扑越过 Shell connection、route、binding 后,在 Launcher 对 + `RuntimeManifest.adapter_id=hermes` 的硬编码 OpenClaw 校验处失败。Launcher + 直接修复 commit 为 `f84a51319d75b99a6b210f19e264904cae07fc8a`;本地 + P7 overlay image 为 + `quay.io/labnow/labnow-launcher@sha256:75f138b0f72c7cf35ecd923a3ba6152879dda99c5c187aef129dfab226bba6f7`, + 其固定 P6 base 为 + `quay.io/labnow/labnow-launcher@sha256:6f9732fda8b86d9bfe4596e848025cc38448da4b17dfea8520e046a32b32e61f`。 +- runner 已修正 Bash 条件上下文中 `die` 未显式返回导致的 fail-closed 缺陷; + fixture 断言 tracked dirty 时 preflight 以非零退出且不会调用 Docker。 +- 当前仍需以本节后的准确 Dev commit 更新受限输入并重跑完整固定组合;在成功 + run 和清理证据生成前,不关闭 P7-R2,不声明 `verified`。 diff --git a/docker_hermes/p7/launcher-overlay.Dockerfile b/docker_hermes/p7/launcher-overlay.Dockerfile new file mode 100644 index 0000000..ba3fb0d --- /dev/null +++ b/docker_hermes/p7/launcher-overlay.Dockerfile @@ -0,0 +1,16 @@ +# syntax=docker/dockerfile:1 + +ARG P6_LAUNCHER_IMAGE +FROM ${P6_LAUNCHER_IMAGE} + +ARG LAUNCHER_SOURCE_COMMIT +ARG P6_LAUNCHER_IMAGE +ARG P6_LAUNCHER_BASE_DIGEST +LABEL org.opencontainers.image.revision="${LAUNCHER_SOURCE_COMMIT}" \ + io.labnow.p7.launcher-base="${P6_LAUNCHER_BASE_DIGEST}" \ + io.labnow.p7.delivery="local-only-overlay" + +# P7 only changes the Launcher runtime consumer and its Hub-trusted config. +# The named build context must be the clean, fixed Launcher Phase checkout. +COPY --from=launcher src/labnow-launcher/devhub_launcher /opt/jupyterhub/devhub_launcher +COPY --from=launcher src/labnow-launcher/resource/config/app.conf /opt/jupyterhub/resource/config/app.conf diff --git a/docker_hermes/p7/p7-inputs.example.json b/docker_hermes/p7/p7-inputs.example.json index 2fa449b..742e4b4 100644 --- a/docker_hermes/p7/p7-inputs.example.json +++ b/docker_hermes/p7/p7-inputs.example.json @@ -9,7 +9,7 @@ "lab_dev": {"path": "/absolute/path/to/lab-dev", "commit": "REPLACE_WITH_P7_LAB_DEV_HEAD", "runtime_commit": "REPLACE_WITH_P7_LAB_DEV_RUNTIME_HEAD"}, "labnow_open": {"path": "/absolute/path/to/labnow-open", "commit": "533b3c5bd0742a77a1cccfc4a30b818271751213", "runtime_commit": "8c56966c4b6be12702d4397aad6b1a153b87d053"}, "labnow_shell": {"path": "/absolute/path/to/labnow-shell", "commit": "REPLACE_WITH_P7_SHELL_HEAD", "runtime_commit": "REPLACE_WITH_P7_SHELL_RUNTIME_HEAD"}, - "labnow_launcher": {"path": "/absolute/path/to/labnow-launcher", "commit": "c84edea3e051d561f28d9f99235563cf491aaeb2", "runtime_commit": "c84edea3e051d561f28d9f99235563cf491aaeb2"}, + "labnow_launcher": {"path": "/absolute/path/to/labnow-launcher", "commit": "f84a51319d75b99a6b210f19e264904cae07fc8a", "runtime_commit": "f84a51319d75b99a6b210f19e264904cae07fc8a"}, "hermes_source": {"path": "/absolute/path/to/hermes-agent", "repository": "REPLACE_WITH_OBSERVED_HERMES_REMOTE", "commit": "REPLACE_WITH_FIXED_HERMES_COMMIT"} }, "images": { @@ -17,7 +17,7 @@ "litellm": {"ref": "quay.io/labnow/litellm@sha256:a2e115874c21b829bd052b18fc85be2f9217fb8244b82812c4ebc6e36f9824d1", "image_id": "sha256:a2e115874c21b829bd052b18fc85be2f9217fb8244b82812c4ebc6e36f9824d1", "repo_digest": "quay.io/labnow/litellm@sha256:a2e115874c21b829bd052b18fc85be2f9217fb8244b82812c4ebc6e36f9824d1", "provenance": "repo_digest"}, "workspace": {"ref": "quay.io/labnow/labnow-open@sha256:56e69ace0da9dbede193e80904fa76ded1dd94cfc38cb1137f6df686c5b7f031", "image_id": "sha256:56e69ace0da9dbede193e80904fa76ded1dd94cfc38cb1137f6df686c5b7f031", "repo_digest": "quay.io/labnow/labnow-open@sha256:56e69ace0da9dbede193e80904fa76ded1dd94cfc38cb1137f6df686c5b7f031", "provenance": "local_build", "source_repository": "labnow_open", "source_commit": "8c56966c4b6be12702d4397aad6b1a153b87d053"}, "shell": {"ref": "quay.io/labnow/labnow-shell@sha256:REPLACE_WITH_64_HEX", "image_id": "sha256:REPLACE_WITH_64_HEX", "repo_digest": "quay.io/labnow/labnow-shell@sha256:REPLACE_WITH_64_HEX", "provenance": "local_build", "source_repository": "labnow_shell", "source_commit": "REPLACE_WITH_P7_SHELL_RUNTIME_HEAD"}, - "launcher": {"ref": "quay.io/labnow/labnow-launcher@sha256:6f9732fda8b86d9bfe4596e848025cc38448da4b17dfea8520e046a32b32e61f", "image_id": "sha256:6f9732fda8b86d9bfe4596e848025cc38448da4b17dfea8520e046a32b32e61f", "repo_digest": "quay.io/labnow/labnow-launcher@sha256:6f9732fda8b86d9bfe4596e848025cc38448da4b17dfea8520e046a32b32e61f", "provenance": "local_build", "source_repository": "labnow_launcher", "source_commit": "c84edea3e051d561f28d9f99235563cf491aaeb2"} + "launcher": {"ref": "quay.io/labnow/labnow-launcher@sha256:75f138b0f72c7cf35ecd923a3ba6152879dda99c5c187aef129dfab226bba6f7", "image_id": "sha256:75f138b0f72c7cf35ecd923a3ba6152879dda99c5c187aef129dfab226bba6f7", "repo_digest": "quay.io/labnow/labnow-launcher@sha256:75f138b0f72c7cf35ecd923a3ba6152879dda99c5c187aef129dfab226bba6f7", "provenance": "local_build", "source_repository": "labnow_launcher", "source_commit": "f84a51319d75b99a6b210f19e264904cae07fc8a", "base_image": "quay.io/labnow/labnow-launcher@sha256:6f9732fda8b86d9bfe4596e848025cc38448da4b17dfea8520e046a32b32e61f"} }, "support_images": { "postgres": {"ref": "postgres:17-alpine", "image_id": "sha256:742f40ea20b9ff2ff31db5458d127452988a2164df9e17441e191f3b72252193", "repo_digest": "postgres@sha256:742f40ea20b9ff2ff31db5458d127452988a2164df9e17441e191f3b72252193"}, diff --git a/docker_hermes/p7/scripts/p7-runner.sh b/docker_hermes/p7/scripts/p7-runner.sh index fb0e901..c5088f2 100755 --- a/docker_hermes/p7/scripts/p7-runner.sh +++ b/docker_hermes/p7/scripts/p7-runner.sh @@ -13,27 +13,30 @@ die() { printf 'P7_ERROR:%s\n' "$1" >&2; return "${2:-1}"; } sha256() { shasum -a 256 "$1" | awk '{print $1}'; } run_id() { python3 -c 'import secrets; print("p7-" + secrets.token_hex(16))'; } mode_of() { stat -f '%Lp' "$1" 2>/dev/null || stat -c '%a' "$1"; } -secure_file() { [[ -f "$1" && ! -L "$1" && "$(mode_of "$1")" =~ ^(400|600)$ ]] || die "SECURE_FILE_REQUIRED" 64; } +secure_file() { + [[ -f "$1" && ! -L "$1" && "$(mode_of "$1")" =~ ^(400|600)$ ]] || { die "SECURE_FILE_REQUIRED" 64; return $?; } +} usage() { printf '%s\n' "Usage: p7-runner.sh --input /secure/path/p7-inputs.json --validate-input|--preflight|--render|--golden|--cleanup"; } input=""; action="" while (($#)); do case "$1" in --input) input="${2:-}"; shift 2 ;; - --validate-input|--preflight|--render|--golden|--cleanup) [[ -z "$action" ]] || die "ACTION_DUPLICATED" 2; action="${1#--}"; shift ;; + --validate-input|--preflight|--render|--golden|--cleanup) [[ -z "$action" ]] || { die "ACTION_DUPLICATED" 2; exit 2; }; action="${1#--}"; shift ;; *) usage >&2; exit 2 ;; esac done [[ -n "$input" && -n "$action" ]] || { usage >&2; exit 2; } -secure_file "$input" +secure_file "$input" || exit $? +input="$(cd "$(dirname "$input")" && pwd)/$(basename "$input")" export P7_INPUT_FILE="$input" P7_RUN_ID="${P7_RUN_ID:-$(run_id)}" -[[ "$P7_RUN_ID" =~ ^p7-[a-f0-9]{32}$ ]] || die "RUN_ID_INVALID" 64 +[[ "$P7_RUN_ID" =~ ^p7-[a-f0-9]{32}$ ]] || { die "RUN_ID_INVALID" 64; exit 64; } export P7_RUN_ID artifact_dir="${P7_ARTIFACTS_DIR:-${p7_dir}/artifacts}" work_dir="${P7_WORK_DIR:-${p7_dir}/.p7-work/${P7_RUN_ID}}" -[[ "$artifact_dir" = /* && "$work_dir" = /* && ${#work_dir} -gt 12 ]] || die "RUNTIME_PATH_INVALID" 64 -case "$work_dir" in /|"$repo_root"|"$p7_dir") die "RUNTIME_PATH_INVALID" 64 ;; esac +[[ "$artifact_dir" = /* && "$work_dir" = /* && ${#work_dir} -gt 12 ]] || { die "RUNTIME_PATH_INVALID" 64; exit 64; } +case "$work_dir" in /|"$repo_root"|"$p7_dir") die "RUNTIME_PATH_INVALID" 64; exit 64 ;; esac export P7_ARTIFACTS_DIR="$artifact_dir" P7_WORK_DIR="$work_dir" mkdir -p "$artifact_dir"; chmod 700 "$artifact_dir" report="${artifact_dir}/p7-${action}-${P7_RUN_ID}.json" @@ -60,7 +63,7 @@ validate_shape() { and (.repositories.hermes_source.commit | type == "string" and test("^[0-9a-f]{40}$")) and (.repositories.labnow_open.commit == "533b3c5bd0742a77a1cccfc4a30b818271751213") and (.repositories.labnow_open.runtime_commit == "8c56966c4b6be12702d4397aad6b1a153b87d053") - and (.repositories.labnow_launcher.commit == "c84edea3e051d561f28d9f99235563cf491aaeb2") + and (.repositories.labnow_launcher.commit == "f84a51319d75b99a6b210f19e264904cae07fc8a") and (.repositories.labnow_launcher.runtime_commit == .repositories.labnow_launcher.commit) and (.images | keys | sort) == ["hermes","launcher","litellm","shell","workspace"] and (.support_images | keys | sort) == ["nginx","postgres","redis"] @@ -78,13 +81,14 @@ validate_shape() { $root.images[$name].provenance == "local_build" and ($root.images[$name].source_repository | type == "string") and $root.images[$name].source_commit == $root.repositories[$root.images[$name].source_repository].runtime_commit)) + and .images.launcher.base_image == "quay.io/labnow/labnow-launcher@sha256:6f9732fda8b86d9bfe4596e848025cc38448da4b17dfea8520e046a32b32e61f" and .images.litellm.provenance == "repo_digest" and ([.images.workspace,.images.shell,.images.launcher,.images.litellm] | all(.ref == .repo_digest)) and (.base_images | keys | sort) == ["build","runtime"] and ([.base_images[]] | all(type == "string" and test("^quay\\.io/labnow/(node|base)@sha256:[0-9a-f]{64}$"))) and (.runtime | keys | sort) == ["p1_env_file"] and (.runtime.p1_env_file | type == "string" and startswith("/")) - ' "$input" >/dev/null || die "INPUT_SCHEMA_INVALID" 68 + ' "$input" >/dev/null || { die "INPUT_SCHEMA_INVALID" 68; return $?; } } assert_repository() { @@ -92,17 +96,17 @@ assert_repository() { path="$(jq -er ".repositories.${name}.path" "$input")" commit="$(jq -er ".repositories.${name}.commit" "$input")" runtime_commit="$(jq -er ".repositories.${name}.runtime_commit" "$input")" - [[ -d "$path/.git" ]] || die "REPOSITORY_UNAVAILABLE" 69 + [[ -d "$path/.git" ]] || { die "REPOSITORY_UNAVAILABLE" 69; return $?; } actual="$(git -C "$path" rev-parse HEAD)" - [[ "$actual" == "$commit" ]] || die "REPOSITORY_COMMIT_MISMATCH" 70 + [[ "$actual" == "$commit" ]] || { die "REPOSITORY_COMMIT_MISMATCH" 70; return $?; } status="$(git -C "$path" status --porcelain=v1 --untracked-files=no)" - [[ -z "$status" ]] || die "REPOSITORY_TRACKED_TREE_DIRTY" 71 - git -C "$path" merge-base --is-ancestor "$runtime_commit" "$commit" || die "RUNTIME_COMMIT_NOT_ANCESTOR" 70 + [[ -z "$status" ]] || { die "REPOSITORY_TRACKED_TREE_DIRTY" 71; return $?; } + git -C "$path" merge-base --is-ancestor "$runtime_commit" "$commit" || { die "RUNTIME_COMMIT_NOT_ANCESTOR" 70; return $?; } if [[ "$runtime_commit" != "$commit" ]]; then changed="$(git -C "$path" diff --name-only "$runtime_commit..$commit")" - [[ -n "$changed" ]] || die "RUNTIME_DELIVERY_DELTA_MISSING" 70 + [[ -n "$changed" ]] || { die "RUNTIME_DELIVERY_DELTA_MISSING" 70; return $?; } if grep -Ev '^(doc|docs|development-docs)/|(^|/)README\.md$' <<<"$changed" >/dev/null; then - die "RUNTIME_DELIVERY_DELTA_NOT_DOCUMENTATION" 70 + die "RUNTIME_DELIVERY_DELTA_NOT_DOCUMENTATION" 70; return $? fi fi } @@ -112,10 +116,10 @@ assert_hermes_source() { path="$(jq -er '.repositories.hermes_source.path' "$input")" expected_repository="$(jq -er '.repositories.hermes_source.repository' "$input")" expected_commit="$(jq -er '.repositories.hermes_source.commit' "$input")" - [[ -d "$path/.git" ]] || die "HERMES_SOURCE_UNAVAILABLE" 69 - [[ "$(git -C "$path" rev-parse HEAD)" == "$expected_commit" ]] || die "HERMES_SOURCE_COMMIT_MISMATCH" 70 - [[ "$(git -C "$path" remote get-url origin)" == "$expected_repository" ]] || die "HERMES_SOURCE_REMOTE_MISMATCH" 70 - [[ -z "$(git -C "$path" status --porcelain=v1 --untracked-files=no)" ]] || die "HERMES_SOURCE_TRACKED_TREE_DIRTY" 71 + [[ -d "$path/.git" ]] || { die "HERMES_SOURCE_UNAVAILABLE" 69; return $?; } + [[ "$(git -C "$path" rev-parse HEAD)" == "$expected_commit" ]] || { die "HERMES_SOURCE_COMMIT_MISMATCH" 70; return $?; } + [[ "$(git -C "$path" remote get-url origin)" == "$expected_repository" ]] || { die "HERMES_SOURCE_REMOTE_MISMATCH" 70; return $?; } + [[ -z "$(git -C "$path" status --porcelain=v1 --untracked-files=no)" ]] || { die "HERMES_SOURCE_TRACKED_TREE_DIRTY" 71; return $?; } } assert_image() { @@ -123,22 +127,27 @@ assert_image() { ref="$(jq -er ".${section}.${name}.ref" "$input")" expected_id="$(jq -er ".${section}.${name}.image_id" "$input")" expected_digest="$(jq -er ".${section}.${name}.repo_digest" "$input")" - actual_id="$(docker image inspect --format '{{.Id}}' "$ref" 2>/dev/null)" || die "IMAGE_UNAVAILABLE" 72 - [[ "$actual_id" == "$expected_id" ]] || die "IMAGE_ID_MISMATCH" 72 + actual_id="$(docker image inspect --format '{{.Id}}' "$ref" 2>/dev/null)" || { die "IMAGE_UNAVAILABLE" 72; return $?; } + [[ "$actual_id" == "$expected_id" ]] || { die "IMAGE_ID_MISMATCH" 72; return $?; } digests="$(docker image inspect --format '{{join .RepoDigests "\n"}}' "$ref")" - grep -Fqx "$expected_digest" <<<"$digests" || die "IMAGE_DIGEST_MISMATCH" 72 + grep -Fqx "$expected_digest" <<<"$digests" || { die "IMAGE_DIGEST_MISMATCH" 72; return $?; } } assert_images() { local name for name in hermes litellm workspace shell launcher; do assert_image images "$name" || return $?; done for name in postgres redis nginx; do assert_image support_images "$name" || return $?; done - local hermes_ref + local hermes_ref launcher_ref launcher_base hermes_ref="$(jq -er '.images.hermes.ref' "$input")" - [[ "$(docker image inspect --format '{{ index .Config.Labels "org.opencontainers.image.revision" }}' "$hermes_ref")" == "$(jq -er '.repositories.hermes_source.commit' "$input")" ]] || die "HERMES_IMAGE_PROVENANCE_MISMATCH" 72 - [[ "$(docker image inspect --format '{{ index .Config.Labels "io.labnow.hermes.build-base" }}' "$hermes_ref")" == "$(jq -er '.base_images.build' "$input")" ]] || die "HERMES_BUILD_BASE_MISMATCH" 72 - [[ "$(docker image inspect --format '{{ index .Config.Labels "io.labnow.hermes.runtime-base" }}' "$hermes_ref")" == "$(jq -er '.base_images.runtime' "$input")" ]] || die "HERMES_RUNTIME_BASE_MISMATCH" 72 - docker image inspect "$(jq -er '.base_images.build' "$input")" "$(jq -er '.base_images.runtime' "$input")" >/dev/null 2>&1 || die "HERMES_BASE_IMAGE_UNAVAILABLE" 72 + [[ "$(docker image inspect --format '{{ index .Config.Labels "org.opencontainers.image.revision" }}' "$hermes_ref")" == "$(jq -er '.repositories.hermes_source.commit' "$input")" ]] || { die "HERMES_IMAGE_PROVENANCE_MISMATCH" 72; return $?; } + [[ "$(docker image inspect --format '{{ index .Config.Labels "io.labnow.hermes.build-base" }}' "$hermes_ref")" == "$(jq -er '.base_images.build' "$input")" ]] || { die "HERMES_BUILD_BASE_MISMATCH" 72; return $?; } + [[ "$(docker image inspect --format '{{ index .Config.Labels "io.labnow.hermes.runtime-base" }}' "$hermes_ref")" == "$(jq -er '.base_images.runtime' "$input")" ]] || { die "HERMES_RUNTIME_BASE_MISMATCH" 72; return $?; } + docker image inspect "$(jq -er '.base_images.build' "$input")" "$(jq -er '.base_images.runtime' "$input")" >/dev/null 2>&1 || { die "HERMES_BASE_IMAGE_UNAVAILABLE" 72; return $?; } + launcher_ref="$(jq -er '.images.launcher.ref' "$input")" + launcher_base="$(jq -er '.images.launcher.base_image' "$input")" + [[ "$(docker image inspect --format '{{ index .Config.Labels "org.opencontainers.image.revision" }}' "$launcher_ref")" == "$(jq -er '.repositories.labnow_launcher.runtime_commit' "$input")" ]] || { die "LAUNCHER_IMAGE_PROVENANCE_MISMATCH" 72; return $?; } + [[ "$(docker image inspect --format '{{ index .Config.Labels "io.labnow.p7.launcher-base" }}' "$launcher_ref")" == "$launcher_base" ]] || { die "LAUNCHER_BASE_IMAGE_MISMATCH" 72; return $?; } + docker image inspect "$launcher_base" >/dev/null 2>&1 || { die "LAUNCHER_BASE_IMAGE_UNAVAILABLE" 72; return $?; } } assert_runtime_input() { @@ -157,7 +166,7 @@ for raw in Path(sys.argv[1]).read_text(encoding="utf-8").splitlines(): values[key] = value raise SystemExit(0 if all(values.get(key) for key in required) else 1) PY - [[ $? == 0 ]] || die "P1_ENV_INCOMPLETE" 73 + [[ $? == 0 ]] || { die "P1_ENV_INCOMPLETE" 73; return $?; } } preflight() { @@ -165,8 +174,8 @@ preflight() { local repo for repo in lab_dev labnow_open labnow_shell labnow_launcher; do assert_repository "$repo" || return $?; done assert_hermes_source || return $? - [[ "$(git -C "$(jq -er '.repositories.lab_dev.path' "$input")" branch --show-current)" == "dev/che-568-hermes-console-experience" ]] || die "PHASE_BRANCH_MISMATCH" 70 - git -C "$(jq -er '.repositories.lab_dev.path' "$input")" merge-base --is-ancestor "$(jq -er '.phase.base_commit' "$input")" HEAD || die "PHASE_BASE_NOT_ANCESTOR" 70 + [[ "$(git -C "$(jq -er '.repositories.lab_dev.path' "$input")" branch --show-current)" == "dev/che-568-hermes-console-experience" ]] || { die "PHASE_BRANCH_MISMATCH" 70; return $?; } + git -C "$(jq -er '.repositories.lab_dev.path' "$input")" merge-base --is-ancestor "$(jq -er '.phase.base_commit' "$input")" HEAD || { die "PHASE_BASE_NOT_ANCESTOR" 70; return $?; } assert_images || return $? assert_runtime_input || return $? } @@ -200,10 +209,10 @@ cleanup_runtime() { assert_cleanup() { local short="${P7_RUN_ID#p7-}" project="p6-runtime-${P7_RUN_ID#p7-}" short="${short:0:12}"; project="p6-runtime-${short}" - [[ -z "$(docker container ls -aq --filter "label=com.docker.compose.project=${project}")" ]] || die "TOPOLOGY_CONTAINER_REMAINS" 79 - [[ -z "$(docker volume ls -q --filter "label=com.docker.compose.project=${project}")" ]] || die "TOPOLOGY_VOLUME_REMAINS" 79 - ! docker network inspect "p6net-${short}" >/dev/null 2>&1 || die "TOPOLOGY_NETWORK_REMAINS" 79 - [[ ! -e "$work_dir" ]] || die "TOPOLOGY_RUNTIME_MATERIAL_REMAINS" 79 + [[ -z "$(docker container ls -aq --filter "label=com.docker.compose.project=${project}")" ]] || { die "TOPOLOGY_CONTAINER_REMAINS" 79; return $?; } + [[ -z "$(docker volume ls -q --filter "label=com.docker.compose.project=${project}")" ]] || { die "TOPOLOGY_VOLUME_REMAINS" 79; return $?; } + ! docker network inspect "p6net-${short}" >/dev/null 2>&1 || { die "TOPOLOGY_NETWORK_REMAINS" 79; return $?; } + [[ ! -e "$work_dir" ]] || { die "TOPOLOGY_RUNTIME_MATERIAL_REMAINS" 79; return $?; } } render_summary() { @@ -216,12 +225,12 @@ security_scan() { local patterns="$1" status shift secure_file "$patterns" || return $? - [[ -s "$patterns" && $# -gt 0 ]] || die "SECRET_SCAN_INPUT_INVALID" 74 + [[ -s "$patterns" && $# -gt 0 ]] || { die "SECRET_SCAN_INPUT_INVALID" 74; return $?; } set +e rg --fixed-strings --files-with-matches --glob '!secret-patterns' -f "$patterns" "$@" >/dev/null 2>&1 status=$? set -e - case "$status" in 1) return 0 ;; 0) die "SECRET_PATTERN_MATCH" 75 ;; *) die "SECRET_SCAN_FAILED" 75 ;; esac + case "$status" in 1) return 0 ;; 0) die "SECRET_PATTERN_MATCH" 75; return $? ;; *) die "SECRET_SCAN_FAILED" 75; return $? ;; esac } case "$action" in diff --git a/docker_hermes/p7/scripts/test-p7-gates.sh b/docker_hermes/p7/scripts/test-p7-gates.sh index 503bf3b..9c42061 100755 --- a/docker_hermes/p7/scripts/test-p7-gates.sh +++ b/docker_hermes/p7/scripts/test-p7-gates.sh @@ -16,6 +16,9 @@ rg -q 'org.opencontainers.image.revision' "${root}/docker_hermes/hermes.Dockerfi rg -q 'ARG HERMES_BUILD_BASE_IMAGE' "${root}/docker_hermes/hermes.Dockerfile" rg -q 'io.labnow.hermes.runtime-base' "${root}/docker_hermes/hermes.Dockerfile" rg -q 'pull_policy: never' "${root}/docker_hermes/p7/docker-compose.runtime.yml" +rg -q 'P6_LAUNCHER_BASE_DIGEST' "${root}/docker_hermes/p7/launcher-overlay.Dockerfile" +rg -q 'io.labnow.p7.launcher-base' "${root}/docker_hermes/p7/launcher-overlay.Dockerfile" +rg -q 'COPY --from=launcher src/labnow-launcher/devhub_launcher' "${root}/docker_hermes/p7/launcher-overlay.Dockerfile" ! rg -n --glob '!**/test-p7-gates.sh' 'OPENAI_API_KEY:|DEEPSEEK_API_KEY:|:latest' "${root}/docker_hermes/p7" ! rg -q 'HERMES_PRODUCT_CHAIN_NOT_AVAILABLE' "$runner" rg -q 'p7-product-chain.py' "$runner" @@ -63,6 +66,42 @@ jq ' chmod 600 "$valid" P7_ARTIFACTS_DIR="$tmp/artifacts" P7_WORK_DIR="$tmp/work" "$runner" --input "$valid" --validate-input >/dev/null +# A failed repository gate must stop before any Docker preflight or topology +# action. The runner executes preflight from an `if` condition, where Bash +# does not propagate `errexit` into nested functions; keep this explicit +# fixture so a reported dirty tree cannot accidentally continue provisioning. +mkdir -p "$tmp/bin" "$tmp/dirty-repo/.git" +jq --arg path "$tmp/dirty-repo" '.repositories.lab_dev.path = $path' "$valid" > "$tmp/dirty.json" +chmod 600 "$tmp/dirty.json" +cat > "$tmp/bin/git" <<'SH' +#!/usr/bin/env bash +set -euo pipefail +case "$*" in + *"rev-parse HEAD"*) printf '%s\n' '0123456789abcdef0123456789abcdef01234567' ;; + *"status --porcelain=v1 --untracked-files=no"*) printf '%s\n' ' M tracked-file' ;; + *) exit 99 ;; +esac +SH +cat > "$tmp/bin/docker" </dev/null 2>&1 +dirty_status=$? +set -e +if [[ "$dirty_status" != 1 ]]; then + printf 'P7 dirty-tree preflight returned %s instead of 1\n' "$dirty_status" >&2 + exit 1 +fi +if [[ -e "$tmp/docker-was-called" ]]; then + printf '%s\n' 'P7 dirty-tree preflight reached Docker' >&2 + exit 1 +fi + jq '.images.hermes.source_commit = "fedcba9876543210fedcba9876543210fedcba98"' "$valid" > "${valid}.mismatch" chmod 600 "${valid}.mismatch" if P7_ARTIFACTS_DIR="$tmp/artifacts" P7_WORK_DIR="$tmp/work" "$runner" --input "${valid}.mismatch" --validate-input >/dev/null 2>&1; then From 822e36a042f464c63e6899d4cef0d8243e6b5101 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Tue, 11 Aug 2026 00:36:37 +0800 Subject: [PATCH 64/87] =?UTF-8?q?fix(hermes):=20=E5=9B=BA=E5=AE=9A?= =?UTF-8?q?=E6=AF=AB=E7=A7=92=E7=A7=9F=E7=BA=A6=E5=85=BC=E5=AE=B9=E5=88=B6?= =?UTF-8?q?=E5=93=81?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_hermes/p7/p7-inputs.example.json | 4 ++-- docker_hermes/p7/scripts/p7-runner.sh | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docker_hermes/p7/p7-inputs.example.json b/docker_hermes/p7/p7-inputs.example.json index 742e4b4..f5bd260 100644 --- a/docker_hermes/p7/p7-inputs.example.json +++ b/docker_hermes/p7/p7-inputs.example.json @@ -7,7 +7,7 @@ "phase": {"branch": "dev/che-568-hermes-console-experience", "base_commit": "45c38585a0ca889f6a20aebfdf3b13a01d369ac2"}, "repositories": { "lab_dev": {"path": "/absolute/path/to/lab-dev", "commit": "REPLACE_WITH_P7_LAB_DEV_HEAD", "runtime_commit": "REPLACE_WITH_P7_LAB_DEV_RUNTIME_HEAD"}, - "labnow_open": {"path": "/absolute/path/to/labnow-open", "commit": "533b3c5bd0742a77a1cccfc4a30b818271751213", "runtime_commit": "8c56966c4b6be12702d4397aad6b1a153b87d053"}, + "labnow_open": {"path": "/absolute/path/to/labnow-open", "commit": "2ac4e268d562c7d26ace8affc830f09cf1cb9305", "runtime_commit": "2ac4e268d562c7d26ace8affc830f09cf1cb9305"}, "labnow_shell": {"path": "/absolute/path/to/labnow-shell", "commit": "REPLACE_WITH_P7_SHELL_HEAD", "runtime_commit": "REPLACE_WITH_P7_SHELL_RUNTIME_HEAD"}, "labnow_launcher": {"path": "/absolute/path/to/labnow-launcher", "commit": "f84a51319d75b99a6b210f19e264904cae07fc8a", "runtime_commit": "f84a51319d75b99a6b210f19e264904cae07fc8a"}, "hermes_source": {"path": "/absolute/path/to/hermes-agent", "repository": "REPLACE_WITH_OBSERVED_HERMES_REMOTE", "commit": "REPLACE_WITH_FIXED_HERMES_COMMIT"} @@ -15,7 +15,7 @@ "images": { "hermes": {"ref": "quay.io/labnow/hermes:p7-REPLACE_WITH_12_HEX", "image_id": "sha256:REPLACE_WITH_64_HEX", "repo_digest": "quay.io/labnow/hermes@sha256:REPLACE_WITH_64_HEX", "provenance": "local_build", "source_repository": "hermes_source", "source_commit": "REPLACE_WITH_FIXED_HERMES_COMMIT"}, "litellm": {"ref": "quay.io/labnow/litellm@sha256:a2e115874c21b829bd052b18fc85be2f9217fb8244b82812c4ebc6e36f9824d1", "image_id": "sha256:a2e115874c21b829bd052b18fc85be2f9217fb8244b82812c4ebc6e36f9824d1", "repo_digest": "quay.io/labnow/litellm@sha256:a2e115874c21b829bd052b18fc85be2f9217fb8244b82812c4ebc6e36f9824d1", "provenance": "repo_digest"}, - "workspace": {"ref": "quay.io/labnow/labnow-open@sha256:56e69ace0da9dbede193e80904fa76ded1dd94cfc38cb1137f6df686c5b7f031", "image_id": "sha256:56e69ace0da9dbede193e80904fa76ded1dd94cfc38cb1137f6df686c5b7f031", "repo_digest": "quay.io/labnow/labnow-open@sha256:56e69ace0da9dbede193e80904fa76ded1dd94cfc38cb1137f6df686c5b7f031", "provenance": "local_build", "source_repository": "labnow_open", "source_commit": "8c56966c4b6be12702d4397aad6b1a153b87d053"}, + "workspace": {"ref": "quay.io/labnow/labnow-open@sha256:b80bca4bfedbab67f6d14855509cc119aeeb212f25e4ce19a5738d7c53ba4d27", "image_id": "sha256:b80bca4bfedbab67f6d14855509cc119aeeb212f25e4ce19a5738d7c53ba4d27", "repo_digest": "quay.io/labnow/labnow-open@sha256:b80bca4bfedbab67f6d14855509cc119aeeb212f25e4ce19a5738d7c53ba4d27", "provenance": "local_build", "source_repository": "labnow_open", "source_commit": "2ac4e268d562c7d26ace8affc830f09cf1cb9305"}, "shell": {"ref": "quay.io/labnow/labnow-shell@sha256:REPLACE_WITH_64_HEX", "image_id": "sha256:REPLACE_WITH_64_HEX", "repo_digest": "quay.io/labnow/labnow-shell@sha256:REPLACE_WITH_64_HEX", "provenance": "local_build", "source_repository": "labnow_shell", "source_commit": "REPLACE_WITH_P7_SHELL_RUNTIME_HEAD"}, "launcher": {"ref": "quay.io/labnow/labnow-launcher@sha256:75f138b0f72c7cf35ecd923a3ba6152879dda99c5c187aef129dfab226bba6f7", "image_id": "sha256:75f138b0f72c7cf35ecd923a3ba6152879dda99c5c187aef129dfab226bba6f7", "repo_digest": "quay.io/labnow/labnow-launcher@sha256:75f138b0f72c7cf35ecd923a3ba6152879dda99c5c187aef129dfab226bba6f7", "provenance": "local_build", "source_repository": "labnow_launcher", "source_commit": "f84a51319d75b99a6b210f19e264904cae07fc8a", "base_image": "quay.io/labnow/labnow-launcher@sha256:6f9732fda8b86d9bfe4596e848025cc38448da4b17dfea8520e046a32b32e61f"} }, diff --git a/docker_hermes/p7/scripts/p7-runner.sh b/docker_hermes/p7/scripts/p7-runner.sh index c5088f2..2317e76 100755 --- a/docker_hermes/p7/scripts/p7-runner.sh +++ b/docker_hermes/p7/scripts/p7-runner.sh @@ -61,8 +61,8 @@ validate_shape() { and (.repositories.hermes_source.path | type == "string" and startswith("/")) and (.repositories.hermes_source.repository | type == "string" and startswith("https://")) and (.repositories.hermes_source.commit | type == "string" and test("^[0-9a-f]{40}$")) - and (.repositories.labnow_open.commit == "533b3c5bd0742a77a1cccfc4a30b818271751213") - and (.repositories.labnow_open.runtime_commit == "8c56966c4b6be12702d4397aad6b1a153b87d053") + and (.repositories.labnow_open.commit == "2ac4e268d562c7d26ace8affc830f09cf1cb9305") + and (.repositories.labnow_open.runtime_commit == "2ac4e268d562c7d26ace8affc830f09cf1cb9305") and (.repositories.labnow_launcher.commit == "f84a51319d75b99a6b210f19e264904cae07fc8a") and (.repositories.labnow_launcher.runtime_commit == .repositories.labnow_launcher.commit) and (.images | keys | sort) == ["hermes","launcher","litellm","shell","workspace"] From bd7aba4dec7e1603b4cf5f95071729b0f84378a9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Tue, 11 Aug 2026 00:45:17 +0800 Subject: [PATCH 65/87] =?UTF-8?q?fix(hermes):=20=E5=AF=B9=E9=BD=90?= =?UTF-8?q?=E5=9B=BA=E5=AE=9A=20CLI=20=E9=BB=84=E9=87=91=E8=B0=83=E7=94=A8?= =?UTF-8?q?=E5=8F=82=E6=95=B0?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_hermes/p7/scripts/p7-product-chain.py | 4 ++-- docker_hermes/p7/scripts/test-p7-gates.sh | 2 ++ 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/docker_hermes/p7/scripts/p7-product-chain.py b/docker_hermes/p7/scripts/p7-product-chain.py index 994785c..f9bb071 100755 --- a/docker_hermes/p7/scripts/p7-product-chain.py +++ b/docker_hermes/p7/scripts/p7-product-chain.py @@ -144,7 +144,7 @@ def hermes_model_call(config: dict[str, Any], marker: str) -> dict[str, Any]: [ "docker", "exec", config["workspace_container"], "timeout", "--signal=TERM", "--kill-after=10s", "180s", "start-labnow-hermes.sh", "-z", - f"Reply {marker} only.", "--ignore-rules", "--max-turns", "4", + f"Reply {marker} only.", "--ignore-rules", ], code="HERMES_MODEL_CALL_FAILED", timeout=200, @@ -169,7 +169,7 @@ def hermes_tool_call(config: dict[str, Any], marker: str) -> dict[str, Any]: "docker", "exec", config["workspace_container"], "timeout", "--signal=TERM", "--kill-after=10s", "180s", "start-labnow-hermes.sh", "-z", f"Use the terminal tool to run: printf {marker} > {proof}. Then reply {marker}_DONE only.", - "--ignore-rules", "--max-turns", "10", "-t", "terminal", + "--ignore-rules", "-t", "terminal", ], code="HERMES_TOOL_CALL_FAILED", timeout=200, diff --git a/docker_hermes/p7/scripts/test-p7-gates.sh b/docker_hermes/p7/scripts/test-p7-gates.sh index 9c42061..d287f68 100755 --- a/docker_hermes/p7/scripts/test-p7-gates.sh +++ b/docker_hermes/p7/scripts/test-p7-gates.sh @@ -22,6 +22,8 @@ rg -q 'COPY --from=launcher src/labnow-launcher/devhub_launcher' "${root}/docker ! rg -n --glob '!**/test-p7-gates.sh' 'OPENAI_API_KEY:|DEEPSEEK_API_KEY:|:latest' "${root}/docker_hermes/p7" ! rg -q 'HERMES_PRODUCT_CHAIN_NOT_AVAILABLE' "$runner" rg -q 'p7-product-chain.py' "$runner" +! rg -q -- '--max-turns' "${root}/docker_hermes/p7/scripts/p7-product-chain.py" +rg -q '"-t", "terminal"' "${root}/docker_hermes/p7/scripts/p7-product-chain.py" python3 -m py_compile \ "${root}/docker_hermes/p7/scripts/p7-prepare-runtime.py" \ "${root}/docker_hermes/p7/scripts/p7-product-chain.py" From 48cd79d072836894431ac212c512a492c0b813fd Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Tue, 11 Aug 2026 00:52:31 +0800 Subject: [PATCH 66/87] =?UTF-8?q?test(hermes):=20=E5=9B=BA=E5=AE=9A=20P7?= =?UTF-8?q?=20=E9=BB=84=E9=87=91=E9=93=BE=E8=AF=81=E6=8D=AE?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_hermes/p7/README.md | 8 +++++ ...72\344\270\216\351\252\214\350\257\201.md" | 34 +++++++++++++++++-- docker_hermes/p7/p7-inputs.example.json | 2 +- docker_hermes/p7/scripts/p7-runner.sh | 2 +- 4 files changed, 42 insertions(+), 4 deletions(-) diff --git a/docker_hermes/p7/README.md b/docker_hermes/p7/README.md index a582649..e201f48 100644 --- a/docker_hermes/p7/README.md +++ b/docker_hermes/p7/README.md @@ -75,3 +75,11 @@ Console 鼠标创建、键盘/焦点和 axe 使用同一 Shell commit 的 P7 浏 runner 另外执行该固定 Shell image 的 live API → JupyterHub → Workspace 链, 不得以 P6 OpenClaw 报告或健康检查替代 Hermes 成功。失败报告只保存错误码, 成功报告只保存结构断言、非敏感 ID、计数和 SHA-256。 + +当前固定组合已由 run `p7-c2d3e4f5a60718293a4b5c6d7e8f9012` 完成真实黄金链; +聚合报告 SHA-256 为 +`1d4331b482dd6959efba7707f991c79bf0076cf46ff2b8f7813c31de862d1a61`, +产品报告 SHA-256 为 +`de08555b73a3d2229d1047931c1dd3ef97b38cd1f62fa1d1c219dcdba95567ed`。 +该事实只证明本地固定 commit/digest 组合,不表示远端镜像、远端 integration、 +main 或部署状态。 diff --git "a/docker_hermes/p7/evidence/2026-08-10-P7-CHE-568-Hermes\346\236\204\345\273\272\344\270\216\351\252\214\350\257\201.md" "b/docker_hermes/p7/evidence/2026-08-10-P7-CHE-568-Hermes\346\236\204\345\273\272\344\270\216\351\252\214\350\257\201.md" index cadc1c0..dd57e2c 100644 --- "a/docker_hermes/p7/evidence/2026-08-10-P7-CHE-568-Hermes\346\236\204\345\273\272\344\270\216\351\252\214\350\257\201.md" +++ "b/docker_hermes/p7/evidence/2026-08-10-P7-CHE-568-Hermes\346\236\204\345\273\272\344\270\216\351\252\214\350\257\201.md" @@ -95,5 +95,35 @@ runtime 环境为 `PYTHONPATH=/opt/hermes:`。该 warning 不改变本次固定 `quay.io/labnow/labnow-launcher@sha256:6f9732fda8b86d9bfe4596e848025cc38448da4b17dfea8520e046a32b32e61f`。 - runner 已修正 Bash 条件上下文中 `die` 未显式返回导致的 fail-closed 缺陷; fixture 断言 tracked dirty 时 preflight 以非零退出且不会调用 Docker。 -- 当前仍需以本节后的准确 Dev commit 更新受限输入并重跑完整固定组合;在成功 - run 和清理证据生成前,不关闭 P7-R2,不声明 `verified`。 +- 该检查点当时仍需以准确 Dev commit 更新受限输入并重跑完整固定组合;后续 + 成功 run 和清理证据见第 8 节。 + +## 8. 2026-08-11 P7-R2 固定组合结果 + +- `74e44643fdd78050100ce1764caddff15cef8d51` 闭合参数化 runner、Launcher overlay + 和受限输入的首轮直接修复;`822e36a042f464c63e6899d4cef0d8243e6b5101` + 固定 Open 毫秒租约兼容制品;`bd7aba4dec7e1603b4cf5f95071729b0f84378a9` + 删除固定 Hermes CLI 不支持的 `--max-turns` 调用参数,并保留外层 180 秒超时。 +- 最终运行时代码组合:Open `2ac4e268d562c7d26ace8affc830f09cf1cb9305`、 + Dev `bd7aba4dec7e1603b4cf5f95071729b0f84378a9`、Shell + `c5845758a29092c48f71f4461e8dd4fdd759ab07`、Launcher + `f84a51319d75b99a6b210f19e264904cae07fc8a`。 +- 最终本地产品制品:Workspace + `quay.io/labnow/labnow-open@sha256:b80bca4bfedbab67f6d14855509cc119aeeb212f25e4ce19a5738d7c53ba4d27`、 + Launcher `quay.io/labnow/labnow-launcher@sha256:75f138b0f72c7cf35ecd923a3ba6152879dda99c5c187aef129dfab226bba6f7`、 + Shell `quay.io/labnow/labnow-shell@sha256:bd9cd37e3035fca7d3726871670db3cee479dbabc6a123fee3ca55e6be43b609`、 + LiteLLM `quay.io/labnow/litellm@sha256:a2e115874c21b829bd052b18fc85be2f9217fb8244b82812c4ebc6e36f9824d1`。 +- `validate-input`、`preflight`、`render` 均通过;最终 run ID 为 + `p7-c2d3e4f5a60718293a4b5c6d7e8f9012`,render SHA-256 为 + `201a80999bd214b2f9cae0fd6da230a08d6756a3dabc01c0b057e39de4da5465`。 +- 参数化真实黄金链的 16 个结构检查全部 `passed`:Console 鼠标证据复用、 + binding payload、DockerSpawner、claim/activate/release、Hermes apply/probe/readiness、 + model、stream、terminal tool、usage、owner negative、prompt/response absent、revoke、 + generation restart、late release、delete、零活跃 lease。 +- 脱敏黄金报告 SHA-256: + `1d4331b482dd6959efba7707f991c79bf0076cf46ff2b8f7813c31de862d1a61`; + 脱敏产品报告 SHA-256: + `de08555b73a3d2229d1047931c1dd3ef97b38cd1f62fa1d1c219dcdba95567ed`。 +- Runner 已确认成功报告 `content_redacted=true`,运行材料敏感扫描零命中,最终 + containers/volumes/network/workdir 全部 `absent`。P7-R2 具备有界复审条件。 +- 本节不表示远端同步、main 合入、镜像发布或部署;用户既有未跟踪路径仍未处理。 diff --git a/docker_hermes/p7/p7-inputs.example.json b/docker_hermes/p7/p7-inputs.example.json index f5bd260..e251c70 100644 --- a/docker_hermes/p7/p7-inputs.example.json +++ b/docker_hermes/p7/p7-inputs.example.json @@ -7,7 +7,7 @@ "phase": {"branch": "dev/che-568-hermes-console-experience", "base_commit": "45c38585a0ca889f6a20aebfdf3b13a01d369ac2"}, "repositories": { "lab_dev": {"path": "/absolute/path/to/lab-dev", "commit": "REPLACE_WITH_P7_LAB_DEV_HEAD", "runtime_commit": "REPLACE_WITH_P7_LAB_DEV_RUNTIME_HEAD"}, - "labnow_open": {"path": "/absolute/path/to/labnow-open", "commit": "2ac4e268d562c7d26ace8affc830f09cf1cb9305", "runtime_commit": "2ac4e268d562c7d26ace8affc830f09cf1cb9305"}, + "labnow_open": {"path": "/absolute/path/to/labnow-open", "commit": "18b20aa7fa3e506b9c85b88736c9f51f317d55d8", "runtime_commit": "2ac4e268d562c7d26ace8affc830f09cf1cb9305"}, "labnow_shell": {"path": "/absolute/path/to/labnow-shell", "commit": "REPLACE_WITH_P7_SHELL_HEAD", "runtime_commit": "REPLACE_WITH_P7_SHELL_RUNTIME_HEAD"}, "labnow_launcher": {"path": "/absolute/path/to/labnow-launcher", "commit": "f84a51319d75b99a6b210f19e264904cae07fc8a", "runtime_commit": "f84a51319d75b99a6b210f19e264904cae07fc8a"}, "hermes_source": {"path": "/absolute/path/to/hermes-agent", "repository": "REPLACE_WITH_OBSERVED_HERMES_REMOTE", "commit": "REPLACE_WITH_FIXED_HERMES_COMMIT"} diff --git a/docker_hermes/p7/scripts/p7-runner.sh b/docker_hermes/p7/scripts/p7-runner.sh index 2317e76..8c4d964 100755 --- a/docker_hermes/p7/scripts/p7-runner.sh +++ b/docker_hermes/p7/scripts/p7-runner.sh @@ -61,7 +61,7 @@ validate_shape() { and (.repositories.hermes_source.path | type == "string" and startswith("/")) and (.repositories.hermes_source.repository | type == "string" and startswith("https://")) and (.repositories.hermes_source.commit | type == "string" and test("^[0-9a-f]{40}$")) - and (.repositories.labnow_open.commit == "2ac4e268d562c7d26ace8affc830f09cf1cb9305") + and (.repositories.labnow_open.commit == "18b20aa7fa3e506b9c85b88736c9f51f317d55d8") and (.repositories.labnow_open.runtime_commit == "2ac4e268d562c7d26ace8affc830f09cf1cb9305") and (.repositories.labnow_launcher.commit == "f84a51319d75b99a6b210f19e264904cae07fc8a") and (.repositories.labnow_launcher.runtime_commit == .repositories.labnow_launcher.commit) From 323208ae924d73f1f62558d56eaac123ced779fd Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Tue, 11 Aug 2026 00:55:25 +0800 Subject: [PATCH 67/87] =?UTF-8?q?test(hermes):=20=E5=9B=BA=E5=AE=9A?= =?UTF-8?q?=E4=BA=A7=E5=93=81=E4=BB=93=E8=AF=81=E6=8D=AE=E5=9B=9E=E6=89=A7?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_hermes/p7/p7-inputs.example.json | 2 +- docker_hermes/p7/scripts/p7-runner.sh | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docker_hermes/p7/p7-inputs.example.json b/docker_hermes/p7/p7-inputs.example.json index e251c70..2ce032a 100644 --- a/docker_hermes/p7/p7-inputs.example.json +++ b/docker_hermes/p7/p7-inputs.example.json @@ -9,7 +9,7 @@ "lab_dev": {"path": "/absolute/path/to/lab-dev", "commit": "REPLACE_WITH_P7_LAB_DEV_HEAD", "runtime_commit": "REPLACE_WITH_P7_LAB_DEV_RUNTIME_HEAD"}, "labnow_open": {"path": "/absolute/path/to/labnow-open", "commit": "18b20aa7fa3e506b9c85b88736c9f51f317d55d8", "runtime_commit": "2ac4e268d562c7d26ace8affc830f09cf1cb9305"}, "labnow_shell": {"path": "/absolute/path/to/labnow-shell", "commit": "REPLACE_WITH_P7_SHELL_HEAD", "runtime_commit": "REPLACE_WITH_P7_SHELL_RUNTIME_HEAD"}, - "labnow_launcher": {"path": "/absolute/path/to/labnow-launcher", "commit": "f84a51319d75b99a6b210f19e264904cae07fc8a", "runtime_commit": "f84a51319d75b99a6b210f19e264904cae07fc8a"}, + "labnow_launcher": {"path": "/absolute/path/to/labnow-launcher", "commit": "990910aafeb6715bdfd656d002c3c6a27ff75cdb", "runtime_commit": "f84a51319d75b99a6b210f19e264904cae07fc8a"}, "hermes_source": {"path": "/absolute/path/to/hermes-agent", "repository": "REPLACE_WITH_OBSERVED_HERMES_REMOTE", "commit": "REPLACE_WITH_FIXED_HERMES_COMMIT"} }, "images": { diff --git a/docker_hermes/p7/scripts/p7-runner.sh b/docker_hermes/p7/scripts/p7-runner.sh index 8c4d964..c4ce82b 100755 --- a/docker_hermes/p7/scripts/p7-runner.sh +++ b/docker_hermes/p7/scripts/p7-runner.sh @@ -63,8 +63,8 @@ validate_shape() { and (.repositories.hermes_source.commit | type == "string" and test("^[0-9a-f]{40}$")) and (.repositories.labnow_open.commit == "18b20aa7fa3e506b9c85b88736c9f51f317d55d8") and (.repositories.labnow_open.runtime_commit == "2ac4e268d562c7d26ace8affc830f09cf1cb9305") - and (.repositories.labnow_launcher.commit == "f84a51319d75b99a6b210f19e264904cae07fc8a") - and (.repositories.labnow_launcher.runtime_commit == .repositories.labnow_launcher.commit) + and (.repositories.labnow_launcher.commit == "990910aafeb6715bdfd656d002c3c6a27ff75cdb") + and (.repositories.labnow_launcher.runtime_commit == "f84a51319d75b99a6b210f19e264904cae07fc8a") and (.images | keys | sort) == ["hermes","launcher","litellm","shell","workspace"] and (.support_images | keys | sort) == ["nginx","postgres","redis"] and ([.images[] | .image_id] + [.support_images[] | .image_id] | all(type == "string" and test("^sha256:[0-9a-f]{64}$"))) From 9f1e88d539f5327ab18af9a6a7950fedee30a1f8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Tue, 11 Aug 2026 00:56:12 +0800 Subject: [PATCH 68/87] =?UTF-8?q?fix(hermes):=20=E5=85=BC=E5=AE=B9?= =?UTF-8?q?=E4=B8=AD=E6=96=87=E8=AF=81=E6=8D=AE=E8=B7=AF=E5=BE=84=E9=A2=84?= =?UTF-8?q?=E6=A3=80?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_hermes/p7/scripts/p7-runner.sh | 2 +- docker_hermes/p7/scripts/test-p7-gates.sh | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/docker_hermes/p7/scripts/p7-runner.sh b/docker_hermes/p7/scripts/p7-runner.sh index c4ce82b..041de92 100755 --- a/docker_hermes/p7/scripts/p7-runner.sh +++ b/docker_hermes/p7/scripts/p7-runner.sh @@ -103,7 +103,7 @@ assert_repository() { [[ -z "$status" ]] || { die "REPOSITORY_TRACKED_TREE_DIRTY" 71; return $?; } git -C "$path" merge-base --is-ancestor "$runtime_commit" "$commit" || { die "RUNTIME_COMMIT_NOT_ANCESTOR" 70; return $?; } if [[ "$runtime_commit" != "$commit" ]]; then - changed="$(git -C "$path" diff --name-only "$runtime_commit..$commit")" + changed="$(git -C "$path" -c core.quotePath=false diff --name-only "$runtime_commit..$commit")" [[ -n "$changed" ]] || { die "RUNTIME_DELIVERY_DELTA_MISSING" 70; return $?; } if grep -Ev '^(doc|docs|development-docs)/|(^|/)README\.md$' <<<"$changed" >/dev/null; then die "RUNTIME_DELIVERY_DELTA_NOT_DOCUMENTATION" 70; return $? diff --git a/docker_hermes/p7/scripts/test-p7-gates.sh b/docker_hermes/p7/scripts/test-p7-gates.sh index d287f68..372b2b3 100755 --- a/docker_hermes/p7/scripts/test-p7-gates.sh +++ b/docker_hermes/p7/scripts/test-p7-gates.sh @@ -24,6 +24,7 @@ rg -q 'COPY --from=launcher src/labnow-launcher/devhub_launcher' "${root}/docker rg -q 'p7-product-chain.py' "$runner" ! rg -q -- '--max-turns' "${root}/docker_hermes/p7/scripts/p7-product-chain.py" rg -q '"-t", "terminal"' "${root}/docker_hermes/p7/scripts/p7-product-chain.py" +rg -q 'core.quotePath=false diff --name-only' "$runner" python3 -m py_compile \ "${root}/docker_hermes/p7/scripts/p7-prepare-runtime.py" \ "${root}/docker_hermes/p7/scripts/p7-product-chain.py" From 6cab6df2a77aae21d99a86153965e26edba31d50 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Wed, 12 Aug 2026 20:41:21 +0800 Subject: [PATCH 69/87] =?UTF-8?q?fix:=20=E8=A1=A5=E9=BD=90=20Hermes=20Chat?= =?UTF-8?q?=20TUI=20Node=20=E8=BF=90=E8=A1=8C=E6=97=B6?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docker_hermes/README.md | 14 ++++ docker_hermes/hermes.Dockerfile | 11 +++ ...26-08-12-P8-H10-Hermes-Chat-TUI-runtime.md | 71 +++++++++++++++++++ .../scripts/test-hermes-runtime-node.sh | 37 ++++++++++ 4 files changed, 133 insertions(+) create mode 100644 docker_hermes/p8/evidence/2026-08-12-P8-H10-Hermes-Chat-TUI-runtime.md create mode 100755 docker_hermes/scripts/test-hermes-runtime-node.sh diff --git a/docker_hermes/README.md b/docker_hermes/README.md index 7b6629e..729ab40 100644 --- a/docker_hermes/README.md +++ b/docker_hermes/README.md @@ -67,6 +67,20 @@ build_image_no_tag hermes p7-<12hex> docker_hermes/hermes.Dockerfile \ 相同的非敏感 provenance。只在本地命名为 `quay.io/labnow/hermes:p7-<12hex>`,不 push。 完整 P7 的受限输入、静态门禁和跨仓 runner 见 [`p7/README.md`](p7/README.md)。 +### P8-H10:Dashboard Chat TUI runtime + +Hermes 的 Dashboard 在 `/api/pty` 中执行已经构建的 +`/opt/hermes/ui-tui/dist/entry.js`。运行基础镜像不是 Node 镜像,因此 Dockerfile 会从 +同一目标架构的 builder 复制固定的 `/opt/node` runtime,并将其放入 `PATH`。这避免用户 +第一次打开 Chat 时触发 Node 下载/解压;不改变 Hermes source、TUI build 或模型配置。 + +对 P8-H10 本地镜像执行不含凭证、不会请求模型的 runtime 门禁: + +```bash +./docker_hermes/scripts/test-hermes-runtime-node.sh \ + quay.io/labnow/hermes:che-588-hermes-chat-tui-runtime-local +``` + ### Start with Docker Compose 1. Copy the sample environment file: diff --git a/docker_hermes/hermes.Dockerfile b/docker_hermes/hermes.Dockerfile index f25d7bb..e4405ae 100644 --- a/docker_hermes/hermes.Dockerfile +++ b/docker_hermes/hermes.Dockerfile @@ -96,8 +96,16 @@ ENV PLAYWRIGHT_BROWSERS_PATH=/opt/hermes/.playwright ENV PYTHONPATH="/opt/hermes:${PYTHONPATH:-}" ENV HERMES_HOME=/root/.hermes ENV HERMES_ALLOW_ROOT_GATEWAY=1 +# The Dashboard PTY starts the already-built ui-tui bundle with `node`. The +# runtime base is intentionally Python-only, so copy the architecture-matched +# Node runtime produced by the builder instead of lazily downloading one after +# an operator opens Chat. +ENV PATH="/opt/node/bin:${PATH}" # Copy the full hermes install tree from the builder (source + browsers + built frontends) COPY --from=builder /opt/hermes /opt/hermes +# `/opt/node` contains node, npm and the Node runtime's bundled execution +# material. Both stages use the same Docker target platform. +COPY --from=builder /opt/node /opt/node # Discover the real python site-packages so legacy env-var fallbacks point at the right tree. # Keep explicit versioned fallbacks around in case detection runs before the first pip install. @@ -111,6 +119,9 @@ RUN set -eux && cd /opt/hermes \ && ln -sf /opt/hermes/start-hermes.sh /opt/conda/bin/hermes /usr/local/bin/ \ && . /opt/utils/script-setup-sys.sh && setup_supervisord \ && mkdir -pv /etc/supervisord/ && mv /opt/hermes/supervisord.conf /etc/supervisord/supervisord.conf \ + && node --version \ + && test -s /opt/hermes/ui-tui/dist/entry.js \ + && node --check /opt/hermes/ui-tui/dist/entry.js \ && install__clean # Data persistence is owned by the runtime orchestrator. diff --git a/docker_hermes/p8/evidence/2026-08-12-P8-H10-Hermes-Chat-TUI-runtime.md b/docker_hermes/p8/evidence/2026-08-12-P8-H10-Hermes-Chat-TUI-runtime.md new file mode 100644 index 0000000..6105ed4 --- /dev/null +++ b/docker_hermes/p8/evidence/2026-08-12-P8-H10-Hermes-Chat-TUI-runtime.md @@ -0,0 +1,71 @@ +# P8-H10 Hermes Chat TUI runtime 修复证据 + +## 冻结范围 + +- Linear:`CHE-588`,状态由总控维护为 `In Progress`。 +- Phase 分支:`dev/che-588-hermes-chat-tui-runtime`。 +- phase base:`9f1e88d539f5327ab18af9a6a7950fedee30a1f8`。 +- control / review policy:`4555841e71dfbbceeebc22eea774ead36c3db99b`。 +- Hermes source commit:`1388cd1c0c1800078bfcc92aebd144fbf145fdb4`。 +- delivery:`local_only`;不 push、不发布、不部署、不改动 `main` 或 integration。 + +## 根因与最小修复 + +构建产物已经包含 `ui-tui/dist/entry.js`,但 Python-only runtime base 没有 `node`。 +Dashboard `/api/pty` 因而尝试在用户打开 Chat 后懒下载 Node;在 Apple Silicon Docker +Desktop 运行 `linux/amd64` 容器时,该下载路径的 GNU tar 解压失败。该故障发生在模型调用前。 + +Dockerfile 现在只从同一 Docker target platform 的 builder 复制 `/opt/node` 到最终 runtime, +并把 `/opt/node/bin` 放在 `PATH` 前部。builder 与 runtime 继续使用冻结的 base digest;未新增 +网络下载、未修改 Hermes 上游 source、Web/TUI build、Python/Gateway/Dashboard、持久化目录或 +模型配置边界。构建期执行 `node --version` 与 `node --check ui-tui/dist/entry.js`。 + +## 本地构建 provenance + +构建命令(仅本地导出;没有 push): + +```bash +export REGISTRY_SRC=quay.io REGISTRY_DST=quay.io CI_PROJECT_NAME=LabNow/lab-dev +export DOCKER_DEFAULT_PLATFORM=linux/amd64 +source ./tool.sh +build_image_no_tag hermes che-588-hermes-chat-tui-runtime-local \ + docker_hermes/hermes.Dockerfile \ + --build-arg HERMES_SOURCE_REPOSITORY=https://github.com/Mushroom47/hermes-agent.git \ + --build-arg HERMES_SOURCE_COMMIT=1388cd1c0c1800078bfcc92aebd144fbf145fdb4 \ + --build-arg HERMES_BUILD_BASE_IMAGE=quay.io/labnow/node@sha256:fd09d9de9b7aa927493acbafbb7d399c089465e988f2a6a240428cdbbd5424e2 \ + --build-arg HERMES_RUNTIME_BASE_IMAGE=quay.io/labnow/base@sha256:782f9814152b64cd4aa5ac76d9fbcedcb3bd89f76fabf2ba3d81225d80b05d3f +``` + +实际本地镜像为 +`quay.io/labnow/hermes:che-588-hermes-chat-tui-runtime-local`,image ID 与 Docker +可回读 RepoDigest 均为 +`sha256:6678de637a5e1bdf38a309fd8c183a7bd5fb67da3715ce19ab784a5688d852ef`,平台为 +`linux/amd64`。这是 `local_only` 构建产物,未发布到远端 registry。 + +## 门禁与运行证据 + +`docker_hermes/scripts/test-hermes-runtime-node.sh` 同时检查 Dockerfile 的 multi-stage +copy/`PATH`/parse gate,并在传入镜像名时验证: + +- runtime 中 `node` 可执行且 major version 不低于 22; +- `ui-tui/dist/entry.js` 存在且能由 runtime Node 解析; +- 在没有 provider 凭证、关闭 stdin 且 3 秒超时的条件下进行有界 TUI 启动;仅接受 clean EOF 或 + 超时,不记录 stdout/stderr 正文,并拒绝 Node 下载痕迹。 + +本次实际结果: + +- Docker build 成功;最终 stage 的 `node --version`、entry 文件存在性和 `node --check` 均成功。 +- `./docker_hermes/scripts/test-hermes-runtime-node.sh quay.io/labnow/hermes:che-588-hermes-chat-tui-runtime-local` + 成功;runtime Node 为 `v26.5.0`,满足 TUI 的 Node 22 最低门槛。 +- 以空白、权限 `0700` 的临时状态目录启动 `start-hermes.sh dashboard`,并强制 + `HERMES_DASHBOARD_HOST=127.0.0.1`:容器内 `GET /api/status` 成功。该门禁不发布端口、 + 不传递认证材料/provider 设置、也不发起模型请求;之后容器和临时状态目录均删除。 +- `bash -n`、P7 既有静态 gates、`docker compose ... config --quiet`、`git diff --check` 和 + 针对本次 diff 的凭证模式扫描均成功。 + +空白配置下将 Dashboard 绑定到 `0.0.0.0` 会被上游的 `DashboardAuthProvider` 安全策略拒绝; +这不是 Node/TUI 故障,且本次本地门禁以 loopback-only 方式避免为测试引入明文认证材料。 + +P8 的 Aviator 串行产品镜像重建、受限凭证输入、浏览器真实 Chat 与端到端 lifecycle 由总控在 +Review 后编排;它们尚未作为本仓本次容器门禁的通过证据。本文不把历史 P7/P8-H9 结果冒充为本次 +Chat 验证。 diff --git a/docker_hermes/scripts/test-hermes-runtime-node.sh b/docker_hermes/scripts/test-hermes-runtime-node.sh new file mode 100755 index 0000000..1e461cf --- /dev/null +++ b/docker_hermes/scripts/test-hermes-runtime-node.sh @@ -0,0 +1,37 @@ +#!/usr/bin/env bash +# Validate that the final Hermes runtime owns the Node executable used by the +# pre-built Dashboard TUI. This test never provides provider credentials and +# never performs a model request. +set -euo pipefail + +root="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +dockerfile="${root}/docker_hermes/hermes.Dockerfile" +image="${1:-}" + +rg -q '^COPY --from=builder /opt/node /opt/node$' "$dockerfile" +rg -q '^ENV PATH="/opt/node/bin:\$\{PATH\}"$' "$dockerfile" +rg -q 'node --check /opt/hermes/ui-tui/dist/entry.js' "$dockerfile" + +if [[ -z "$image" ]]; then + printf '%s\n' 'PASS static runtime Node/TUI gate.' + exit 0 +fi + +docker run --rm --platform linux/amd64 --entrypoint /bin/sh "$image" -ec ' + node --version + node_major="$(node --version | sed -E "s/^v([0-9]+).*/\1/")" + test "$node_major" -ge 22 + test -s /opt/hermes/ui-tui/dist/entry.js + node --check /opt/hermes/ui-tui/dist/entry.js + # Bounded module startup only: stdin is closed and no provider settings are + # present, so the TUI cannot submit a model request. A timeout means startup + # stayed alive; immediate clean EOF is also acceptable. + set +e + timeout 3s node /opt/hermes/ui-tui/dist/entry.js /tmp/hermes-tui-startup.log 2>&1 + status=$? + set -e + test "$status" = 0 -o "$status" = 124 + ! rg -q "install(ing)? node|downloading node" /tmp/hermes-tui-startup.log + rm -f /tmp/hermes-tui-startup.log +' +printf '%s\n' 'PASS container runtime Node/TUI gate.' From db6954c3b097cd3ce770d010a5ae88d6dd3cf7a5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Mon, 17 Aug 2026 05:39:00 +0800 Subject: [PATCH 70/87] =?UTF-8?q?chore:=20=E6=B8=85=E7=90=86=20LLM=20Hub?= =?UTF-8?q?=20=E9=98=B6=E6=AE=B5=E8=AF=81=E6=8D=AE=E6=96=87=E6=A1=A3?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .gitignore | 3 + ...72\344\270\216\351\252\214\350\257\201.md" | 129 ------------------ ...26-08-12-P8-H10-Hermes-Chat-TUI-runtime.md | 71 ---------- ...21\351\223\276\351\252\214\346\224\266.md" | 78 ----------- 4 files changed, 3 insertions(+), 278 deletions(-) delete mode 100644 "docker_hermes/p7/evidence/2026-08-10-P7-CHE-568-Hermes\346\236\204\345\273\272\344\270\216\351\252\214\350\257\201.md" delete mode 100644 docker_hermes/p8/evidence/2026-08-12-P8-H10-Hermes-Chat-TUI-runtime.md delete mode 100644 "docker_openclaw/p6/evidence/2026-08-10-P6-CHE-563-\351\273\204\351\207\221\351\223\276\351\252\214\346\224\266.md" diff --git a/.gitignore b/.gitignore index 1133fe7..58f944d 100644 --- a/.gitignore +++ b/.gitignore @@ -167,3 +167,6 @@ cython_debug/ # and can be added to the global gitignore or merged into this file. For a more nuclear # option (not recommended) you can uncomment the following to ignore the entire idea folder. .idea/ + +# LLM Hub 日期化 Phase 证据统一维护在总控仓 +/docker_*/p*/evidence/*.md diff --git "a/docker_hermes/p7/evidence/2026-08-10-P7-CHE-568-Hermes\346\236\204\345\273\272\344\270\216\351\252\214\350\257\201.md" "b/docker_hermes/p7/evidence/2026-08-10-P7-CHE-568-Hermes\346\236\204\345\273\272\344\270\216\351\252\214\350\257\201.md" deleted file mode 100644 index dd57e2c..0000000 --- "a/docker_hermes/p7/evidence/2026-08-10-P7-CHE-568-Hermes\346\236\204\345\273\272\344\270\216\351\252\214\350\257\201.md" +++ /dev/null @@ -1,129 +0,0 @@ -# P7 CHE-568 Hermes 构建与验证证据 - -## 1. 证据边界 - -- 观察日期:2026-08-10(Asia/Hong_Kong)。 -- Phase:P7 / CHE-568。 -- 分支:`dev/che-568-hermes-console-experience`。 -- phase base:`45c38585a0ca889f6a20aebfdf3b13a01d369ac2`。 -- control / review policy:`06c49f26642c7e39a118aedad1395197f2bd91db`。 -- 契约:`v1alpha1 / 0.1.0-rc.1`。 -- 本文只证明 `lab-dev` 负责的 Hermes 固定源码构建、基础镜像 provenance、 - 本地制品身份和失败关闭 runner。它不证明 Open renderer、Shell 可信目录或 - Launcher 管理的完整 Hermes Workspace 黄金链已通过。 - -## 2. 固定输入 - -| 输入 | 准确值 | -| --- | --- | -| Hermes repository | `https://github.com/Mushroom47/hermes-agent.git` | -| Hermes commit | `1388cd1c0c1800078bfcc92aebd144fbf145fdb4` | -| build base | `quay.io/labnow/node@sha256:fd09d9de9b7aa927493acbafbb7d399c089465e988f2a6a240428cdbbd5424e2` | -| runtime base | `quay.io/labnow/base@sha256:782f9814152b64cd4aa5ac76d9fbcedcb3bd89f76fabf2ba3d81225d80b05d3f` | -| target platform | `linux/amd64` | - -本地源码 checkout 的 `origin`、远端 `main` 与 `HEAD` 均核对到上述 40 位 -commit;构建过程再次以 `git fetch --depth 1 origin ` 和 detached -checkout 校验准确 commit,不依赖移动 `main`。 - -## 3. 本地制品 - -| 字段 | 准确值 | -| --- | --- | -| local tag | `quay.io/labnow/hermes:p7-1388cd1c0c18` | -| local image ID | `sha256:c47cf16fad3fbb952a616c910fe3c7769e8516a15986db182752091e1ec02d67` | -| local RepoDigest | `quay.io/labnow/hermes@sha256:c47cf16fad3fbb952a616c910fe3c7769e8516a15986db182752091e1ec02d67` | -| OS / architecture | `linux/amd64` | - -`docker image inspect` 已确认四个 provenance label 分别等于固定 repository、 -source commit、build base 和 runtime base。该 RepoDigest 仅是当前 Docker 本地 -制品身份;镜像未 push、未发布,不代表远端 registry 已存在该 digest。 - -## 4. 有界构建修复 - -1. amd64 模拟环境中的 GNU `tar` 解包 `python-olm 3.2.16` sdist 时返回 - `Cannot open: Function not implemented`。同一基础镜像中使用 Python 标准库 - `tarfile` 和 `filter="data"` 的临时容器诊断通过;Dockerfile 只替换解包实现, - 保持源码包版本、wheel 构建和安装语义不变。 -2. Ubuntu archive 连续两次对不同包返回 `502 Bad Gateway`。builder 与 runtime - 安装步骤临时设置 APT `Acquire::Retries=5` 和 HTTP timeout,安装后立即删除; - 未换源、未换包、未改变基础镜像。 -3. 第三次构建显式使用 `DOCKER_DEFAULT_PLATFORM=linux/amd64`,退出码为 0; - `python-olm` wheel、Hermes Web、TUI、Playwright、Python 依赖和最终 runtime - stage 均完成。 - -构建仍报告 Dockerfile 既有 `$PYTHONPATH` `UndefinedVar` 静态 warning;实际 -runtime 环境为 `PYTHONPATH=/opt/hermes:`。该 warning 不改变本次固定制品身份, -作为非阻断后续清理项登记。 - -## 5. 已执行验证 - -| 验证 | 结果 | -| --- | --- | -| `bash -n docker_hermes/p7/scripts/p7-runner.sh docker_hermes/p7/scripts/test-p7-gates.sh` | 通过 | -| `./docker_hermes/p7/scripts/test-p7-gates.sh` | 通过;源码固定、local-only Compose、非法输入与 provenance 不匹配失败关闭 | -| `git diff --check` | 通过 | -| 固定输入本地 Docker build | 通过;退出 0 | -| `docker image inspect` 的 image ID、平台与四个 labels | 通过 | - -## 6. 初始交付结论 - -- `lab-dev` P7 固定 Hermes base image 已在本地生成,可交给 `labnow-open` 构建 - P7 Workspace 产品镜像。 -- 初始提交在缺少完整产品输入时以 - `P7_ERROR:HERMES_PRODUCT_CHAIN_NOT_AVAILABLE` 失败关闭,没有复用 P6 OpenClaw - 证据;后续 P7-R2 修复提交已经以参数化真实链替换该临时门禁。 -- `labnow-open`、`labnow-shell` 和 Launcher 的准确 P7 commit 与本地产品镜像 - 身份随后已固定;最终完整 Hermes 黄金链以新的 run ID 和报告 SHA-256 - 单独记录,不追写成本节的初始构建事实。 -- S0/S1:无。S2:既有 `$PYTHONPATH` 静态 warning;不阻断本 Phase 固定组合验证。 -- 本次未读取或处理用户既有 `.DS_Store`、`docs/`、`hermes-chat-screenshot.png`; - 未输出凭证明文,未 push,未发布镜像,未部署,未修改 `main` 或 integration。 - -## 7. 2026-08-11 P7-R2 真实链路直接修复 - -- `90328f6a520b0c617689bcbc21576a10f5e34b10` 与 - `fcbcec7526ce9f9fd9f6a3a8717e198345d54269` 已实现参数化真实黄金 runner、 - 四仓 commit、五个产品镜像和 support/base image 的 preflight,以及真实 - Hermes 模型/stream/tool、usage、generation/revoke/cleanup 链。 -- 首次固定拓扑越过 Shell connection、route、binding 后,在 Launcher 对 - `RuntimeManifest.adapter_id=hermes` 的硬编码 OpenClaw 校验处失败。Launcher - 直接修复 commit 为 `f84a51319d75b99a6b210f19e264904cae07fc8a`;本地 - P7 overlay image 为 - `quay.io/labnow/labnow-launcher@sha256:75f138b0f72c7cf35ecd923a3ba6152879dda99c5c187aef129dfab226bba6f7`, - 其固定 P6 base 为 - `quay.io/labnow/labnow-launcher@sha256:6f9732fda8b86d9bfe4596e848025cc38448da4b17dfea8520e046a32b32e61f`。 -- runner 已修正 Bash 条件上下文中 `die` 未显式返回导致的 fail-closed 缺陷; - fixture 断言 tracked dirty 时 preflight 以非零退出且不会调用 Docker。 -- 该检查点当时仍需以准确 Dev commit 更新受限输入并重跑完整固定组合;后续 - 成功 run 和清理证据见第 8 节。 - -## 8. 2026-08-11 P7-R2 固定组合结果 - -- `74e44643fdd78050100ce1764caddff15cef8d51` 闭合参数化 runner、Launcher overlay - 和受限输入的首轮直接修复;`822e36a042f464c63e6899d4cef0d8243e6b5101` - 固定 Open 毫秒租约兼容制品;`bd7aba4dec7e1603b4cf5f95071729b0f84378a9` - 删除固定 Hermes CLI 不支持的 `--max-turns` 调用参数,并保留外层 180 秒超时。 -- 最终运行时代码组合:Open `2ac4e268d562c7d26ace8affc830f09cf1cb9305`、 - Dev `bd7aba4dec7e1603b4cf5f95071729b0f84378a9`、Shell - `c5845758a29092c48f71f4461e8dd4fdd759ab07`、Launcher - `f84a51319d75b99a6b210f19e264904cae07fc8a`。 -- 最终本地产品制品:Workspace - `quay.io/labnow/labnow-open@sha256:b80bca4bfedbab67f6d14855509cc119aeeb212f25e4ce19a5738d7c53ba4d27`、 - Launcher `quay.io/labnow/labnow-launcher@sha256:75f138b0f72c7cf35ecd923a3ba6152879dda99c5c187aef129dfab226bba6f7`、 - Shell `quay.io/labnow/labnow-shell@sha256:bd9cd37e3035fca7d3726871670db3cee479dbabc6a123fee3ca55e6be43b609`、 - LiteLLM `quay.io/labnow/litellm@sha256:a2e115874c21b829bd052b18fc85be2f9217fb8244b82812c4ebc6e36f9824d1`。 -- `validate-input`、`preflight`、`render` 均通过;最终 run ID 为 - `p7-c2d3e4f5a60718293a4b5c6d7e8f9012`,render SHA-256 为 - `201a80999bd214b2f9cae0fd6da230a08d6756a3dabc01c0b057e39de4da5465`。 -- 参数化真实黄金链的 16 个结构检查全部 `passed`:Console 鼠标证据复用、 - binding payload、DockerSpawner、claim/activate/release、Hermes apply/probe/readiness、 - model、stream、terminal tool、usage、owner negative、prompt/response absent、revoke、 - generation restart、late release、delete、零活跃 lease。 -- 脱敏黄金报告 SHA-256: - `1d4331b482dd6959efba7707f991c79bf0076cf46ff2b8f7813c31de862d1a61`; - 脱敏产品报告 SHA-256: - `de08555b73a3d2229d1047931c1dd3ef97b38cd1f62fa1d1c219dcdba95567ed`。 -- Runner 已确认成功报告 `content_redacted=true`,运行材料敏感扫描零命中,最终 - containers/volumes/network/workdir 全部 `absent`。P7-R2 具备有界复审条件。 -- 本节不表示远端同步、main 合入、镜像发布或部署;用户既有未跟踪路径仍未处理。 diff --git a/docker_hermes/p8/evidence/2026-08-12-P8-H10-Hermes-Chat-TUI-runtime.md b/docker_hermes/p8/evidence/2026-08-12-P8-H10-Hermes-Chat-TUI-runtime.md deleted file mode 100644 index 6105ed4..0000000 --- a/docker_hermes/p8/evidence/2026-08-12-P8-H10-Hermes-Chat-TUI-runtime.md +++ /dev/null @@ -1,71 +0,0 @@ -# P8-H10 Hermes Chat TUI runtime 修复证据 - -## 冻结范围 - -- Linear:`CHE-588`,状态由总控维护为 `In Progress`。 -- Phase 分支:`dev/che-588-hermes-chat-tui-runtime`。 -- phase base:`9f1e88d539f5327ab18af9a6a7950fedee30a1f8`。 -- control / review policy:`4555841e71dfbbceeebc22eea774ead36c3db99b`。 -- Hermes source commit:`1388cd1c0c1800078bfcc92aebd144fbf145fdb4`。 -- delivery:`local_only`;不 push、不发布、不部署、不改动 `main` 或 integration。 - -## 根因与最小修复 - -构建产物已经包含 `ui-tui/dist/entry.js`,但 Python-only runtime base 没有 `node`。 -Dashboard `/api/pty` 因而尝试在用户打开 Chat 后懒下载 Node;在 Apple Silicon Docker -Desktop 运行 `linux/amd64` 容器时,该下载路径的 GNU tar 解压失败。该故障发生在模型调用前。 - -Dockerfile 现在只从同一 Docker target platform 的 builder 复制 `/opt/node` 到最终 runtime, -并把 `/opt/node/bin` 放在 `PATH` 前部。builder 与 runtime 继续使用冻结的 base digest;未新增 -网络下载、未修改 Hermes 上游 source、Web/TUI build、Python/Gateway/Dashboard、持久化目录或 -模型配置边界。构建期执行 `node --version` 与 `node --check ui-tui/dist/entry.js`。 - -## 本地构建 provenance - -构建命令(仅本地导出;没有 push): - -```bash -export REGISTRY_SRC=quay.io REGISTRY_DST=quay.io CI_PROJECT_NAME=LabNow/lab-dev -export DOCKER_DEFAULT_PLATFORM=linux/amd64 -source ./tool.sh -build_image_no_tag hermes che-588-hermes-chat-tui-runtime-local \ - docker_hermes/hermes.Dockerfile \ - --build-arg HERMES_SOURCE_REPOSITORY=https://github.com/Mushroom47/hermes-agent.git \ - --build-arg HERMES_SOURCE_COMMIT=1388cd1c0c1800078bfcc92aebd144fbf145fdb4 \ - --build-arg HERMES_BUILD_BASE_IMAGE=quay.io/labnow/node@sha256:fd09d9de9b7aa927493acbafbb7d399c089465e988f2a6a240428cdbbd5424e2 \ - --build-arg HERMES_RUNTIME_BASE_IMAGE=quay.io/labnow/base@sha256:782f9814152b64cd4aa5ac76d9fbcedcb3bd89f76fabf2ba3d81225d80b05d3f -``` - -实际本地镜像为 -`quay.io/labnow/hermes:che-588-hermes-chat-tui-runtime-local`,image ID 与 Docker -可回读 RepoDigest 均为 -`sha256:6678de637a5e1bdf38a309fd8c183a7bd5fb67da3715ce19ab784a5688d852ef`,平台为 -`linux/amd64`。这是 `local_only` 构建产物,未发布到远端 registry。 - -## 门禁与运行证据 - -`docker_hermes/scripts/test-hermes-runtime-node.sh` 同时检查 Dockerfile 的 multi-stage -copy/`PATH`/parse gate,并在传入镜像名时验证: - -- runtime 中 `node` 可执行且 major version 不低于 22; -- `ui-tui/dist/entry.js` 存在且能由 runtime Node 解析; -- 在没有 provider 凭证、关闭 stdin 且 3 秒超时的条件下进行有界 TUI 启动;仅接受 clean EOF 或 - 超时,不记录 stdout/stderr 正文,并拒绝 Node 下载痕迹。 - -本次实际结果: - -- Docker build 成功;最终 stage 的 `node --version`、entry 文件存在性和 `node --check` 均成功。 -- `./docker_hermes/scripts/test-hermes-runtime-node.sh quay.io/labnow/hermes:che-588-hermes-chat-tui-runtime-local` - 成功;runtime Node 为 `v26.5.0`,满足 TUI 的 Node 22 最低门槛。 -- 以空白、权限 `0700` 的临时状态目录启动 `start-hermes.sh dashboard`,并强制 - `HERMES_DASHBOARD_HOST=127.0.0.1`:容器内 `GET /api/status` 成功。该门禁不发布端口、 - 不传递认证材料/provider 设置、也不发起模型请求;之后容器和临时状态目录均删除。 -- `bash -n`、P7 既有静态 gates、`docker compose ... config --quiet`、`git diff --check` 和 - 针对本次 diff 的凭证模式扫描均成功。 - -空白配置下将 Dashboard 绑定到 `0.0.0.0` 会被上游的 `DashboardAuthProvider` 安全策略拒绝; -这不是 Node/TUI 故障,且本次本地门禁以 loopback-only 方式避免为测试引入明文认证材料。 - -P8 的 Aviator 串行产品镜像重建、受限凭证输入、浏览器真实 Chat 与端到端 lifecycle 由总控在 -Review 后编排;它们尚未作为本仓本次容器门禁的通过证据。本文不把历史 P7/P8-H9 结果冒充为本次 -Chat 验证。 diff --git "a/docker_openclaw/p6/evidence/2026-08-10-P6-CHE-563-\351\273\204\351\207\221\351\223\276\351\252\214\346\224\266.md" "b/docker_openclaw/p6/evidence/2026-08-10-P6-CHE-563-\351\273\204\351\207\221\351\223\276\351\252\214\346\224\266.md" deleted file mode 100644 index 6435b2e..0000000 --- "a/docker_openclaw/p6/evidence/2026-08-10-P6-CHE-563-\351\273\204\351\207\221\351\223\276\351\252\214\346\224\266.md" +++ /dev/null @@ -1,78 +0,0 @@ -# CHE-563 / P6 OpenClaw 黄金链验收 - -## 冻结输入 - -- Linear Issue:`CHE-563` -- Phase 分支:`dev/che-563-openclaw-product-closure` -- Phase base:`940325578bae9905673965d6dc489130ab4b6a46` -- 已验证实现提交:`9d78780e9cfb6ac09410c3a21da7de1c2f59a1b4` -- control commit:`2eb71d7590739df3de8db2f8cf9098154a397f0b` -- review policy commit:`680ca92661a08254eb396ab809f478bbdba3510e` -- contract:`v1alpha1 / 0.1.0-rc.1` -- contract bundle SHA-256:`d289dff9bcaa3d28035c5ed2e56b806f4b3b37fdca3159352d22f0c03942e202` -- delivery transport:`local_only` - -## 固定产品组合 - -| 仓库 / 制品 | 准确输入 | -| --- | --- | -| `lab-dev` | 实现提交 `9d78780e9cfb6ac09410c3a21da7de1c2f59a1b4`;验证时以 `HEAD=1b4562899e03eacdee5a86eb55b47d5e12117ee8` 加 review snapshot `85b01ca3ff4ad06584771555d3321c78776cbd6bacd366de5609c237a3006822` 固定,提交后从 Phase base 到实现提交的 binary diff SHA-256 仍为同一值 | -| `labnow-open` | `21019e0c24dc7b51747c2bef3cd90f5d259be839`;本地镜像 `quay.io/labnow/labnow-open@sha256:c9c6a45637521cbbaeacea57fbb128696066fd91c5dff4521555f1bd5211f244` | -| `labnow-shell` | `5c9411dfd3c4d7b1e606c0d9dc0c5e62313bc376`;本地镜像 `quay.io/labnow/labnow-shell@sha256:d7ed71cf58eddf72642d61d4442a0820632060fac9f4eb611b28062b7f38c54d` | -| `labnow-launcher` | `c84edea3e051d561f28d9f99235563cf491aaeb2`;本地镜像 `quay.io/labnow/labnow-launcher@sha256:6f9732fda8b86d9bfe4596e848025cc38448da4b17dfea8520e046a32b32e61f` | -| LiteLLM | `quay.io/labnow/litellm@sha256:a2e115874c21b829bd052b18fc85be2f9217fb8244b82812c4ebc6e36f9824d1` | -| OpenClaw base | `quay.io/labnow/openclaw@sha256:edc85cc2068f5ec0df470f7d06daa0a4fbd78ef5ad6cf5b48f58381da839dd12` | - -所有 `labnow-*` 镜像均为本地构建,未推送、未发布;支持镜像与完整 provenance 由最终聚合报告固定。 - -## 首轮 Review 阻断项闭环 - -| 阻断项 | 结果 | -| --- | --- | -| R1:缺少完整真实产品链与可聚合证据 | 通过;真实 Shell → JupyterHub → Launcher → Workspace → Adapter → LiteLLM 链已完成 | -| R2:Workspace ID 不一致 | 通过;统一使用真实 `server_name`,binding、lease、Workspace 与 usage 查询一致 | -| R3:Launcher 环境变量接受能力 | 通过;固定 Launcher 提交已在真实 claim / activate / release 流程中验证 | -| R4:Workspace RepoDigest 不明确 | 通过;输入门禁固定本地 image ID、synthetic RepoDigest、源仓 commit 与 OpenClaw base digest | - -复审期间仅处理上述阻断项及其直接影响。OpenClaw 默认新 Workspace 的工具 allowlist 收窄为 `exec`;既有显式 allowlist 不变。Shell usage 改用 LiteLLM v2 `key_alias` / `model_group` 分页查询。P6 driver 将 usage 时间格式固定为 UTC 毫秒,避免超过 Shell 接口允许的 3 位小数。 - -## 真实黄金结果 - -- run ID:`p6-9fd9cd2a55a685a9409a459eae58beb2` -- protected input SHA-256:`b1aa535b604ed58047694366c211b544b03e3d91aca09b8b0a59274b7e20d27e` -- final report:`docker_openclaw/p6/artifacts/p6-final-p6-9fd9cd2a55a685a9409a459eae58beb2.json` -- final report SHA-256:`7d358dae1dfc88bdb61eb5a2e0ec53b83e33037981284f93e5df47d2dfc58cf4` -- provision report SHA-256:`03bf931e4c115b169f7bdb32e55adaa2281624f6e937f82ee7bbbff8fc7b817a` -- golden report SHA-256:`9a37a106433ea72cb13d199f5d0a8ceab7f608fd36198041a2f824c40b08f870` -- cleanup report SHA-256:`eb0af6b5c27409ae6978e1c18335b3c655399453e08f39653adb022252a02f5b` -- 结果:`passed`,`content_redacted=true` -- usage:8 条;投影仅含 `timestamp`、`model`、`total_tokens`、`prompt_tokens`、`completion_tokens`、`status` -- 生命周期:generation 1 停止后旧 key 拒绝;generation 2 使用新 key;旧 key 继续拒绝;迟到 generation 1 release 返回 409;delete 后新 key 拒绝;最终 active lease 为 0 -- 安全边界:owner negative 隔离通过;Prompt / Response 正文不保存;未发现 usage 正文持久化表;扫描根与聚合报告通过敏感模式扫描 -- 清理:本次 run 的 LiteLLM、Shell、JupyterHub、Launcher、Workspace、临时材料、进程、网络和卷全部为 `absent` - -## 验证命令 - -以下命令均退出 0: - -```bash -bash -n docker_openclaw/p6/scripts/*.sh -PYTHONDONTWRITEBYTECODE=1 python3 -m py_compile docker_openclaw/p6/scripts/*.py -./docker_openclaw/p6/scripts/test-p6-gates.sh -./docker_openclaw/p6/scripts/test-p6-compose-render.sh -./docker_openclaw/p6/scripts/test-p6-driver-flow.sh -P6_RUN_ID=p6-9fd9cd2a55a685a9409a459eae58beb2 ./docker_openclaw/p6/scripts/p6-runner.sh --input /Users/chengeng/Projects/GitHub/lab-dev/docker_openclaw/p6/p6-inputs.json --preflight -P6_RUN_ID=p6-9fd9cd2a55a685a9409a459eae58beb2 ./docker_openclaw/p6/scripts/p6-runner.sh --input /Users/chengeng/Projects/GitHub/lab-dev/docker_openclaw/p6/p6-inputs.json --golden -P6_RUN_ID=p6-9fd9cd2a55a685a9409a459eae58beb2 ./docker_openclaw/p6/scripts/p6-runner.sh --input /Users/chengeng/Projects/GitHub/lab-dev/docker_openclaw/p6/p6-inputs.json --cleanup -./docker_openclaw/p6/scripts/p6-aggregate.sh --artifacts docker_openclaw/p6/artifacts --run-id p6-9fd9cd2a55a685a9409a459eae58beb2 -git diff --check -``` - -## Handoff - -- `status=ready_for_integration_review` -- `phase_commit=9d78780e9cfb6ac09410c3a21da7de1c2f59a1b4` -- 产品实现、固定组合、真实黄金链和清理证据均已具备,可进入总控有界复审。 -- 本地 artifact 被 `.gitignore` 排除;以本文件记录的 run ID 与 SHA-256 回读,不提交运行材料或任何密钥。 -- 未修改、合并或推送 `main`;未推进任何远端 integration;未 push;未发布镜像;未部署共享或生产环境;未写入明文凭证。 -- 用户既有未跟踪 `.DS_Store`、`docs/`、`hermes-chat-screenshot.png` 未读取、修改、暂存或删除。 From fdbbab2155a9e088c37d2a8a2057178e19ac9534 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Mon, 17 Aug 2026 06:05:50 +0800 Subject: [PATCH 71/87] =?UTF-8?q?chore:=20=E6=94=B9=E7=94=A8=E6=9C=AC?= =?UTF-8?q?=E5=9C=B0=E5=BF=BD=E7=95=A5=E9=A1=B9=E7=9B=AE=E6=96=87=E6=A1=A3?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .gitignore | 3 --- 1 file changed, 3 deletions(-) diff --git a/.gitignore b/.gitignore index 58f944d..1133fe7 100644 --- a/.gitignore +++ b/.gitignore @@ -167,6 +167,3 @@ cython_debug/ # and can be added to the global gitignore or merged into this file. For a more nuclear # option (not recommended) you can uncomment the following to ignore the entire idea folder. .idea/ - -# LLM Hub 日期化 Phase 证据统一维护在总控仓 -/docker_*/p*/evidence/*.md From 82c6df16c099793e36a46be52e39c509fbfe728e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Thu, 20 Aug 2026 06:51:36 +0800 Subject: [PATCH 72/87] =?UTF-8?q?fix(litellm):=20=E6=94=B6=E7=B4=A7=20Comp?= =?UTF-8?q?ose=20=E5=87=AD=E6=8D=AE=E8=BE=B9=E7=95=8C?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 动机与背景:PH-1 处理阻断级缺口 1.1。原 Compose 将 LITELLM_MASTER_KEY、含密码 DATABASE_URL 与 POSTGRES_PASSWORD 放入服务 environment,导致凭据进入渲染配置和容器 metadata。 关键设计取舍:遵循冻结决策 D-3,复用 Redis 的 Docker Secret 模式。PostgreSQL 改用原生 POSTGRES_PASSWORD_FILE;LiteLLM 入口脚本在最终 exec 前读取 Secret 文件并构造所需运行时变量。LiteLLM 最终进程环境可见性为已接受残余风险,已写入 README。 验证:执行 bash -n docker_litellm/work/start-litellm.sh;使用本地占位值执行 docker compose --env-file /dev/null -f docker_litellm/demo/docker-compose.litellm.yml --profile single config --format json | jq -e <服务 environment 与 Secret 挂载断言>;执行 git diff --check。关键输出:PASS static secret boundary。 影响面:仅 LiteLLM Compose、运行时入口和服务 README;未修改镜像、CI、P6/P7 或 Compose 参数化。关联 Linear:CHE-664。 剩余风险:最终 LiteLLM 进程仍需获得 LITELLM_MASTER_KEY 与 DATABASE_URL;具备容器内调试权限的主体仍应视为高权限主体。 --- docker_litellm/README.md | 4 ++- .../demo/docker-compose.litellm.yml | 18 ++++++++++--- docker_litellm/work/start-litellm.sh | 27 ++++++++++++++++--- 3 files changed, 41 insertions(+), 8 deletions(-) diff --git a/docker_litellm/README.md b/docker_litellm/README.md index 2205cb1..039c785 100644 --- a/docker_litellm/README.md +++ b/docker_litellm/README.md @@ -59,7 +59,7 @@ docker compose --env-file .env -f docker-compose.litellm.yml --profile single up ## 配置与安全边界 -`config.yaml` 从环境变量读取管理面 `LITELLM_MASTER_KEY`、`DATABASE_URL` 与 Redis 凭据。管理面 key 仅用于 `/user/new`、`/credentials`、`/model/new`、`/key/generate`、`/key/block` 和 `/key/delete` 等管理接口;smoke 生成的数据面虚拟 key 是短期、模型白名单、TTL、预算、RPM、TPM 与 `llm_api` 路由限制的独立 key。双副本基线启用 `enable_redis_auth_cache`,并将 `user_api_key_cache_ttl` 设为 1 秒,以使撤销在 30 秒 smoke SLO 内经共享 Redis 重新校验。 +`config.yaml` 从运行时环境读取管理面 `LITELLM_MASTER_KEY`、`DATABASE_URL` 与 Redis 凭据。Compose 不再把管理密钥、数据库密码或含密码的连接串写入服务 `environment`:它将 `LITELLM_MASTER_KEY`、`POSTGRES_PASSWORD` 和 `REDIS_PASSWORD` 交给 Docker Secret;PostgreSQL 使用官方 `POSTGRES_PASSWORD_FILE`,LiteLLM 的 `start-litellm.sh` 在最终 `exec` 前读取 Secret 文件、构造 `DATABASE_URL` 并立即转交 LiteLLM。管理面 key 仅用于 `/user/new`、`/credentials`、`/model/new`、`/key/generate`、`/key/block` 和 `/key/delete` 等管理接口;smoke 生成的数据面虚拟 key 是短期、模型白名单、TTL、预算、RPM、TPM 与 `llm_api` 路由限制的独立 key。双副本基线启用 `enable_redis_auth_cache`,并将 `user_api_key_cache_ttl` 设为 1 秒,以使撤销在 30 秒 smoke SLO 内经共享 Redis 重新校验。 | 变量 | 是否必填 | 作用 | 风险说明 | | --- | ---: | --- | --- | @@ -94,6 +94,8 @@ cd docker_litellm/demo `LITELLM_MASTER_KEY`、上游 API key 与虚拟 key 不会作为 `curl`、`jq` 或其他子进程的命令参数传递。脚本以 `umask 077` 创建工作目录,所有 header、请求、响应和 key 文件均为 `0600`,退出时删除;创建的 user、credential、model 和两个测试 key 也会清理。上游凭据仅由 smoke 客户端读取,不会注入 LiteLLM Compose 容器。 +Compose 凭据边界的残余风险:LiteLLM 上游配置接口仍要求 `LITELLM_MASTER_KEY` 与 `DATABASE_URL` 在其最终进程环境中可见;本基线已接受这一点。凭据不再出现在 Compose 渲染、容器 `docker inspect` metadata、命令行参数、容器日志或运行时临时文件中。使用具有 Docker daemon 访问权限或容器内同等调试权限的主体仍应视为高权限主体,不应以该边界替代主机与容器访问控制。 + 若未设置上游变量,脚本仍验证 LiteLLM readiness、PostgreSQL 连接、从每个 LiteLLM 副本到 Redis 的认证连通性、migration 证据和 user 清理路径,并以明确的 `result=skipped` / `phase=pending_upstream` 报告退出。它不会伪造 chat、stream、tool、usage、block/delete 或撤销传播已通过,最终聚合也会拒绝该报告。 ## Readiness 与 Redis 结论 diff --git a/docker_litellm/demo/docker-compose.litellm.yml b/docker_litellm/demo/docker-compose.litellm.yml index 87b2cd4..09fe774 100644 --- a/docker_litellm/demo/docker-compose.litellm.yml +++ b/docker_litellm/demo/docker-compose.litellm.yml @@ -5,8 +5,12 @@ x-litellm-common: &litellm-common restart: "no" environment: TZ: ${TZ:-Asia/Hong_Kong} - LITELLM_MASTER_KEY: ${LITELLM_MASTER_KEY:?set a local-only management key in .env} - DATABASE_URL: postgresql://${POSTGRES_USER:?set in .env}:${POSTGRES_PASSWORD:?set in .env}@postgres:5432/${POSTGRES_DB:-litellm} + LITELLM_MASTER_KEY_FILE: /run/secrets/litellm_master_key + POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password + POSTGRES_HOST: postgres + POSTGRES_PORT: "5432" + POSTGRES_DB: ${POSTGRES_DB:-litellm} + POSTGRES_USER: ${POSTGRES_USER:?set in .env} REDIS_HOST: redis REDIS_PORT: "6379" REDIS_PASSWORD_FILE: /run/secrets/redis_password @@ -20,6 +24,8 @@ x-litellm-common: &litellm-common - ../work/start-litellm.sh:/opt/utils/start-litellm.sh:ro - ../work/run-migration-locked.py:/opt/utils/run-migration-locked.py:ro secrets: + - litellm_master_key + - postgres_password - redis_password depends_on: postgres: @@ -59,7 +65,9 @@ services: environment: POSTGRES_DB: ${POSTGRES_DB:-litellm} POSTGRES_USER: ${POSTGRES_USER:?set in .env} - POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set in .env} + POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password + secrets: + - postgres_password volumes: - litellm_postgres_data:/var/lib/postgresql/data networks: @@ -113,6 +121,10 @@ volumes: litellm_redis_data: secrets: + litellm_master_key: + environment: LITELLM_MASTER_KEY + postgres_password: + environment: POSTGRES_PASSWORD redis_password: environment: REDIS_PASSWORD diff --git a/docker_litellm/work/start-litellm.sh b/docker_litellm/work/start-litellm.sh index c62cdce..1df7054 100755 --- a/docker_litellm/work/start-litellm.sh +++ b/docker_litellm/work/start-litellm.sh @@ -9,11 +9,30 @@ export HOME="$HOME_LITELLM" export PRISMA_HOME_DIR="${PRISMA_HOME_DIR:-$HOME_LITELLM}" cd "$HOME_LITELLM" -# Compose mounts the Redis credential as a Docker secret. Export it only in -# this process tree so it is absent from Docker inspect and command arguments. +# Compose mounts credentials as Docker secrets. Read them only in this process +# tree, immediately before the final exec: Docker metadata and argv therefore +# contain neither secret values nor a password-bearing DATABASE_URL. LiteLLM +# itself requires the management key and DATABASE_URL in its final environment; +# that process-environment visibility is the explicitly accepted residual risk. +read_secret_file() { + local variable_name="$1" secret_file="$2" + test -r "$secret_file" + export "$variable_name=$(cat "$secret_file")" +} + +if [ -n "${LITELLM_MASTER_KEY_FILE:-}" ]; then + read_secret_file LITELLM_MASTER_KEY "$LITELLM_MASTER_KEY_FILE" +fi + +if [ -n "${POSTGRES_PASSWORD_FILE:-}" ]; then + read_secret_file POSTGRES_PASSWORD "$POSTGRES_PASSWORD_FILE" + : "${POSTGRES_USER:?POSTGRES_USER is required with POSTGRES_PASSWORD_FILE}" + export DATABASE_URL="postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@${POSTGRES_HOST:-postgres}:${POSTGRES_PORT:-5432}/${POSTGRES_DB:-litellm}" + unset POSTGRES_PASSWORD +fi + if [ -n "${REDIS_PASSWORD_FILE:-}" ]; then - test -r "$REDIS_PASSWORD_FILE" - export REDIS_PASSWORD="$(cat "$REDIS_PASSWORD_FILE")" + read_secret_file REDIS_PASSWORD "$REDIS_PASSWORD_FILE" fi # LiteLLM checks this environment variable while serializing SpendLog payloads. From 87182ebfb2b25096bda59b1bc8d2ccd32191413e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Thu, 20 Aug 2026 07:10:29 +0800 Subject: [PATCH 73/87] =?UTF-8?q?test(litellm):=20=E8=A6=86=E7=9B=96=20Com?= =?UTF-8?q?pose=20=E5=87=AD=E6=8D=AE=E8=BE=B9=E7=95=8C=E5=9B=9E=E5=BD=92?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 动机与背景:PH-1 需要证明管理密钥、数据库连接凭据和 PostgreSQL 密码不会重新进入 Compose metadata、argv、日志或临时文件,并要求真实 LiteLLM 管理面探活。 关键设计取舍:新增无上游的 test-secret-boundary.sh,只生成随机本地占位值且不读取 demo/.env。它先启动 PostgreSQL/Redis、经入口 wrapper 执行独立 migration job,再启动 LiteLLM;这样 migration lock 进程也在读取 Prisma DATABASE_URL 前获得 Docker Secret。管理面以创建并删除测试 user/key 证明认证与清理路径。 验证:执行 bash -n docker_litellm/work/start-litellm.sh docker_litellm/demo/scripts/smoke-baseline.sh docker_litellm/demo/scripts/test-secret-boundary.sh;执行 ./docker_litellm/demo/scripts/test-secret-boundary.sh;执行 ./docker_litellm/demo/scripts/smoke-baseline.sh --security-check;执行 ./docker_litellm/demo/scripts/test-verification-gates.sh;执行 git diff --check。关键输出:compose_config/readiness/inspect/argv/logs/temporary_files/management_smoke/cleanup 均为 passed,management_http_status=200。 影响面:LiteLLM P1 本地 Compose 与长期安全门禁;未修改镜像、CI、P6/P7、镜像 digest 或 Compose 参数化。关联 Linear:CHE-664。 剩余风险:LiteLLM 最终进程仍可见 LITELLM_MASTER_KEY 与 DATABASE_URL,这是冻结方案接受的风险;具备容器内调试或 Docker daemon 权限的主体仍是高权限主体。 --- docker_litellm/README.md | 3 + .../demo/docker-compose.litellm.yml | 2 +- docker_litellm/demo/scripts/smoke-baseline.sh | 7 +- .../demo/scripts/test-secret-boundary.sh | 184 ++++++++++++++++++ 4 files changed, 194 insertions(+), 2 deletions(-) create mode 100755 docker_litellm/demo/scripts/test-secret-boundary.sh diff --git a/docker_litellm/README.md b/docker_litellm/README.md index 039c785..1bd8818 100644 --- a/docker_litellm/README.md +++ b/docker_litellm/README.md @@ -82,12 +82,15 @@ cd docker_litellm/demo ./scripts/verify-p1.sh ./scripts/smoke-baseline.sh --security-check ./scripts/test-verification-gates.sh +./scripts/test-secret-boundary.sh ``` 在全新 checkout 中按上述标准命令执行时,脚本自己生成而非复用历史文件:`p1-migration-summary.json`、`p1-migration-concurrency.json`、`p1-single-summary.json`、`p1-ha-summary.json`、`p1-redis-recovery.json` 与最终 `p1-final-summary.json`(均在被忽略的 `artifacts/`)。聚合脚本只接受当前 `HEAD`、同一 `verification_run_id`、相同 image ID、正确 mode、启动后 `tested_at`、`result=passed`、`phase=completed` 且脱敏的输入;任何缺失、失败、跳过、过期或模式不符都会被拒绝。 `--security-check` 不读取 `.env`、不启动服务也不发送上游请求;它拒绝 inline header、secret-bearing `jq --arg`、`set -x`、Compose 上游凭据注入和 Redis 密码命令行展开,并检查 Docker Secret、0600 临时文件与退出清理约束。`test-verification-gates.sh` 验证历史 PASS 失效、前置失败报告与 dotenv 命令替换不执行;在已启动 single 栈中追加 `--with-running-stack` 会以真实 404 删除请求证明 cleanup 不会生成 PASS。 +`test-secret-boundary.sh` 是 PH-1 的无上游定向门禁:它只生成本地占位凭据,不读取 `demo/.env`,验证 Compose 渲染和容器 inspect 不含 `LITELLM_MASTER_KEY`、`DATABASE_URL`、`POSTGRES_PASSWORD` 的服务环境或凭据值,并对 inspect、`ps/argv`、容器日志、容器临时文件执行负向检查。随后它启动 LiteLLM、调用并清理一次已认证的管理端点,退出时删除本次创建的容器、卷、网络和宿主临时文件。若固定网络 `litellm-baseline-net` 已被其他栈占用,脚本会失败退出而不会复用或干扰该网络。 + `smoke-redis-recovery.sh` 在已启动的 HA 栈中临时断开 Redis 网络端点,验证两个副本的有界认证探针均失败,再恢复 `redis` alias、等待 breaker 窗口并确认认证后的 `GET /v1/models` 恢复。它有独立的恢复 trap,不会让故障测试影响主 smoke 的资源清理。`aggregate-verification-summary.sh` 将 migration、single、HA 与 Redis 独立报告组合为不含密钥、密码、提示词和响应正文的最终 JSON 摘要。 脚本在真实上游变量存在时执行:创建测试用户、保存测试上游凭证、以 `litellm_credential_name` 创建模型、由调用方生成稳定高熵 virtual key 并故意丢弃首次创建响应,再用该 key 的 0600 Authorization header 调用 `/v2/key/info` 恢复、验证相同 key 重试被拒绝而不会创建第二资源;随后显式 `GET /v1/models`、chat、stream、tool call、token-bearing usage 查询、block,以及独立 key 的 delete。DeepSeek V4 使用 `deepseek/` 的原生 provider,避免被通用 OpenAI provider 丢弃 `thinking` 参数。HA 模式会先证明第二副本接受 key,再验证跨副本 RPM 与 TPM 限制均返回由 LiteLLM Proxy limiter 产生的 `429`,最后轮询两个副本直到都拒绝,并输出实际传播时间与 SLO。 diff --git a/docker_litellm/demo/docker-compose.litellm.yml b/docker_litellm/demo/docker-compose.litellm.yml index 09fe774..ed40fb4 100644 --- a/docker_litellm/demo/docker-compose.litellm.yml +++ b/docker_litellm/demo/docker-compose.litellm.yml @@ -50,7 +50,7 @@ services: <<: *litellm-common container_name: ${LITELLM_MIGRATE_CONTAINER_NAME:-svc-litellm-migrate} profiles: ["migrate"] - command: ["python3", "/opt/utils/run-migration-locked.py", "--config", "config.migrate.yaml", "--skip_server_startup", "--enforce_prisma_migration_check"] + command: ["/opt/utils/start-litellm.sh", "python3", "/opt/utils/run-migration-locked.py", "--config", "config.migrate.yaml", "--skip_server_startup", "--enforce_prisma_migration_check"] healthcheck: disable: true depends_on: diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh index 1c81e28..0464992 100755 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -105,7 +105,12 @@ security_check() { || sed '/^security_check() {/,/^}/d' "$0" | rg -n -- 'set -x' \ || git diff --no-ext-diff -- . | rg -n --pcre2 '(?:sk-|Bearer\s+)[A-Za-z0-9_-]{24,}' \ || rg -n '^ UPSTREAM_(API_KEY|BASE_URL|MODEL|PROVIDER):' "$DEMO_DIR/docker-compose.litellm.yml" \ + || rg -n '^ (LITELLM_MASTER_KEY|DATABASE_URL|POSTGRES_PASSWORD):' "$DEMO_DIR/docker-compose.litellm.yml" \ || rg -n -- '--requirepass[[:space:]].*\$\{REDIS_PASSWORD|REDIS_PASSWORD:.*\$\{' "$DEMO_DIR/docker-compose.litellm.yml" \ + || ! rg -q 'LITELLM_MASTER_KEY_FILE: /run/secrets/litellm_master_key' "$DEMO_DIR/docker-compose.litellm.yml" \ + || ! rg -q 'POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password' "$DEMO_DIR/docker-compose.litellm.yml" \ + || ! rg -q 'litellm_master_key:' "$DEMO_DIR/docker-compose.litellm.yml" \ + || ! rg -q 'postgres_password:' "$DEMO_DIR/docker-compose.litellm.yml" \ || ! rg -q 'redis_password:' "$DEMO_DIR/docker-compose.litellm.yml" \ || ! rg -q 'REDIS_PASSWORD_FILE: /run/secrets/redis_password' "$DEMO_DIR/docker-compose.litellm.yml"; then unsafe=1 @@ -127,7 +132,7 @@ security_check() { echo "FAIL security negative check: unsafe secret transport or cleanup invariant" >&2 return 1 fi - echo "PASS security negative check: no inline process arguments/log tracing/Git secrets, no upstream Compose injection, Redis Docker secret and 0600 cleanup invariants present." + echo "PASS security negative check: no inline process arguments/log tracing/Git secrets, no upstream or management/database Compose injection, Docker Secret and 0600 cleanup invariants present." } if [[ "$SECURITY_CHECK" == true ]]; then diff --git a/docker_litellm/demo/scripts/test-secret-boundary.sh b/docker_litellm/demo/scripts/test-secret-boundary.sh new file mode 100755 index 0000000..fa373d1 --- /dev/null +++ b/docker_litellm/demo/scripts/test-secret-boundary.sh @@ -0,0 +1,184 @@ +#!/usr/bin/env bash +# PH-1 regression gate for the LiteLLM Compose credential boundary. It uses +# generated local placeholders only, never reads demo/.env, and leaves no +# containers, volumes, networks, or host temporary credential files behind. +set -euo pipefail + +script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +demo_dir="$(cd "${script_dir}/.." && pwd)" +compose_file="${demo_dir}/docker-compose.litellm.yml" +image_ref="${LITELLM_SECRET_BOUNDARY_IMAGE:-quay.io/labnow/litellm:1.97.0-ead62528e607}" +run_id="$(python3 -c 'import secrets; print(secrets.token_hex(8))')" +project="ph1-secret-boundary-${run_id}" +litellm_container="ph1-secret-boundary-${run_id}-litellm-1" +litellm_peer_container="ph1-secret-boundary-${run_id}-litellm-2" +publish_port="${LITELLM_SECRET_BOUNDARY_PORT:-4100}" +tmpdir="$(mktemp -d "${TMPDIR:-/tmp}/litellm-secret-boundary.XXXXXX")" +env_file="${tmpdir}/runtime.env" +headers_file="${tmpdir}/admin.headers" +user_payload="${tmpdir}/user-create.json" +user_response="${tmpdir}/user-create-response.json" +user_delete_payload="${tmpdir}/user-delete.json" +generate_payload="${tmpdir}/generate.json" +generate_response="${tmpdir}/generate-response.json" +generated_key_file="${tmpdir}/generated.key" +delete_payload="${tmpdir}/delete.json" +report_file="${LITELLM_SECRET_BOUNDARY_REPORT_FILE:-}" +started=false +compose_config_result="not_run" +readiness_result="not_run" +inspect_result="not_run" +argv_result="not_run" +logs_result="not_run" +temporary_files_result="not_run" +management_smoke_result="not_run" +management_http_status="not_run" + +cleanup() { + local exit_code=$? + trap - EXIT + if [[ "$started" == true ]]; then + docker compose --env-file "$env_file" -p "$project" -f "$compose_file" --profile single down -v --remove-orphans >/dev/null 2>&1 || exit_code=1 + fi + rm -rf "$tmpdir" + if docker ps -a --format '{{.Names}}' | rg -q "^${litellm_container}$|^${litellm_peer_container}$"; then + echo "FAIL cleanup: PH-1 LiteLLM container remains" >&2 + exit_code=1 + fi + if docker network inspect litellm-baseline-net >/dev/null 2>&1; then + echo "FAIL cleanup: PH-1 network remains" >&2 + exit_code=1 + fi + if [[ -n "$report_file" ]]; then + mkdir -p "$(dirname "$report_file")" + printf '{"result":"%s","compose_config":"%s","readiness":"%s","inspect":"%s","argv":"%s","logs":"%s","temporary_files":"%s","management_smoke":"%s","management_http_status":"%s","cleanup":"%s"}\n' \ + "$([[ "$exit_code" == 0 ]] && echo passed || echo failed)" "$compose_config_result" "$readiness_result" "$inspect_result" "$argv_result" "$logs_result" "$temporary_files_result" "$management_smoke_result" "$management_http_status" "$([[ "$exit_code" == 0 ]] && echo passed || echo failed)" > "$report_file" + chmod 600 "$report_file" + fi + exit "$exit_code" +} +trap cleanup EXIT + +need() { command -v "$1" >/dev/null || { echo "required command missing: $1" >&2; exit 2; }; } +need docker +need jq +need rg +need openssl + +# The Compose file keeps its legacy fixed network name. Refuse to attach a +# boundary test to any pre-existing stack instead of disturbing its network. +if docker network inspect litellm-baseline-net >/dev/null 2>&1; then + echo "refusing to reuse existing litellm-baseline-net" >&2 + exit 2 +fi + +umask 077 +master_key="sk-$(openssl rand -hex 24)" +postgres_password="$(openssl rand -hex 24)" +redis_password="$(openssl rand -hex 24)" +printf 'LITELLM_IMAGE=%s\nLITELLM_MASTER_KEY=%s\nPOSTGRES_USER=litellm\nPOSTGRES_PASSWORD=%s\nPOSTGRES_DB=litellm\nREDIS_PASSWORD=%s\nLITELLM_1_CONTAINER_NAME=%s\nLITELLM_2_CONTAINER_NAME=%s\nLITELLM_1_PORT=%s\nLITELLM_2_PORT=4101\nLITELLM_PUBLISH_HOST=127.0.0.1\n' \ + "$image_ref" "$master_key" "$postgres_password" "$redis_password" "$litellm_container" "$litellm_peer_container" "$publish_port" > "$env_file" +chmod 600 "$env_file" + +docker compose --env-file "$env_file" -p "$project" -f "$compose_file" --profile single config --format json | jq -e ' + . as $config | + ([.services | to_entries[] | select(.key == "litellm-1" or .key == "litellm-2" or .key == "litellm-migrate") | .value.environment // {} | keys[] | select(. == "LITELLM_MASTER_KEY" or . == "DATABASE_URL" or . == "POSTGRES_PASSWORD")] | length == 0) + and ($config.services.postgres.environment | has("POSTGRES_PASSWORD") | not) +' >/dev/null +compose_config_result="passed" +echo "PASS compose config: service environment omits LITELLM_MASTER_KEY, DATABASE_URL, and POSTGRES_PASSWORD." + +started=true +docker compose --env-file "$env_file" -p "$project" -f "$compose_file" up -d --wait postgres redis >/dev/null +docker compose --env-file "$env_file" -p "$project" -f "$compose_file" --profile migrate run --rm --no-deps litellm-migrate >/dev/null +docker compose --env-file "$env_file" -p "$project" -f "$compose_file" --profile single up -d litellm-1 >/dev/null +for attempt in $(seq 1 60); do + if curl --silent --fail --max-time 3 "http://127.0.0.1:${publish_port}/health/readiness" | jq -e '.status == "healthy" and .db == "connected"' >/dev/null; then + break + fi + sleep 2 +done +if ! curl --silent --fail --max-time 3 "http://127.0.0.1:${publish_port}/health/readiness" | jq -e '.status == "healthy" and .db == "connected"' >/dev/null; then + echo "FAIL readiness: LiteLLM did not report a healthy PostgreSQL connection" >&2 + exit 1 +fi +readiness_result="passed" +echo "PASS readiness: LiteLLM and PostgreSQL are healthy." + +assert_metadata_boundary() { + local container="$1" forbidden_key="$2" forbidden_value="$3" + if docker inspect "$container" --format '{{range .Config.Env}}{{println .}}{{end}}' | rg -q "^${forbidden_key}="; then + echo "FAIL inspect: ${forbidden_key} remains in ${container} metadata" >&2 + return 1 + fi + if docker inspect "$container" --format '{{range .Config.Env}}{{println .}}{{end}}' | rg -Fq -- "$forbidden_value"; then + echo "FAIL inspect: credential value remains in ${container} metadata" >&2 + return 1 + fi +} + +assert_metadata_boundary "$litellm_container" LITELLM_MASTER_KEY "$master_key" +assert_metadata_boundary "$litellm_container" DATABASE_URL "$postgres_password" +assert_metadata_boundary "${project}-postgres-1" POSTGRES_PASSWORD "$postgres_password" +inspect_result="passed" +echo "PASS inspect: no management key, database URL, or PostgreSQL password value in container metadata." + +for container in "$litellm_container" "${project}-postgres-1" "${project}-redis-1"; do + if docker top "$container" -eo args | rg -Fq -- "$master_key" \ + || docker top "$container" -eo args | rg -Fq -- "$postgres_password" \ + || docker top "$container" -eo args | rg -Fq -- "$redis_password"; then + echo "FAIL ps/argv: credential value is present in ${container}" >&2 + exit 1 + fi +done +argv_result="passed" +echo "PASS ps/argv: no credential values in container command lines." + +if docker compose --env-file "$env_file" -p "$project" -f "$compose_file" --profile single logs --no-color | rg -Fq -- "$master_key" \ + || docker compose --env-file "$env_file" -p "$project" -f "$compose_file" --profile single logs --no-color | rg -Fq -- "$postgres_password" \ + || docker compose --env-file "$env_file" -p "$project" -f "$compose_file" --profile single logs --no-color | rg -Fq -- "$redis_password"; then + echo "FAIL logs: credential value is present in Compose logs" >&2 + exit 1 +fi +logs_result="passed" +echo "PASS logs: no generated credential values in Compose logs." + +docker exec "$litellm_container" /bin/sh -ec ' + set -eu + for secret_file in /run/secrets/litellm_master_key /run/secrets/postgres_password /run/secrets/redis_password; do + secret="$(cat "$secret_file")" + if grep -R -F -q -- "$secret" /tmp /opt/litellm 2>/dev/null; then + exit 1 + fi + done +' +temporary_files_result="passed" +echo "PASS temporary files: no credential copies outside Docker Secret mounts." + +printf 'Authorization: Bearer %s\nContent-Type: application/json\n' "$master_key" > "$headers_file" +probe_user="ph1-secret-boundary-${run_id}-user" +jq -n --arg user "$probe_user" '{user_id:$user, auto_create_key:false, user_role:"internal_user"}' > "$user_payload" +jq -n --arg user "$probe_user" '{user_ids:[$user]}' > "$user_delete_payload" +jq -n --arg user "$probe_user" '{key_alias:"ph1-secret-boundary-probe", user_id:$user, duration:"1m", models:[]}' > "$generate_payload" +chmod 600 "$headers_file" "$user_payload" "$user_delete_payload" "$generate_payload" +management_http_status="$(curl --silent --show-error --max-time 30 --request POST "http://127.0.0.1:${publish_port}/user/new" \ + --header "@${headers_file}" --data-binary "@${user_payload}" --output "$user_response" --write-out '%{http_code}' || true)" +if [[ "$management_http_status" != 200 ]]; then + echo "FAIL management smoke: /user/new returned HTTP ${management_http_status:-transport_error}" >&2 + exit 1 +fi +chmod 600 "$user_response" +jq -e --arg user "$probe_user" '.user_id == $user' "$user_response" >/dev/null +curl --silent --show-error --fail --max-time 30 --request POST "http://127.0.0.1:${publish_port}/key/generate" \ + --header "@${headers_file}" --data-binary "@${generate_payload}" --output "$generate_response" +chmod 600 "$generate_response" +jq -er '.key | select(type == "string" and length > 0)' "$generate_response" > "$generated_key_file" +chmod 600 "$generated_key_file" +jq -n --rawfile key "$generated_key_file" '{keys:[($key | rtrimstr("\n"))]}' > "$delete_payload" +chmod 600 "$delete_payload" +curl --silent --show-error --fail --max-time 30 --request POST "http://127.0.0.1:${publish_port}/key/delete" \ + --header "@${headers_file}" --data-binary "@${delete_payload}" --output /dev/null +curl --silent --show-error --fail --max-time 30 --request POST "http://127.0.0.1:${publish_port}/user/delete" \ + --header "@${headers_file}" --data-binary "@${user_delete_payload}" --output /dev/null +management_smoke_result="passed" +echo "PASS management smoke: authenticated user and key create/delete endpoints completed." From 2dac1022369dbd566661e5ba3d2de282f306c9f8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Thu, 20 Aug 2026 07:14:48 +0800 Subject: [PATCH 74/87] =?UTF-8?q?fix(litellm):=20=E8=84=B1=E6=95=8F=20migr?= =?UTF-8?q?ation=20=E8=BF=9E=E6=8E=A5=E4=B8=B2=E6=97=A5=E5=BF=97?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 动机与背景:PH-1 真实 migration 门禁发现 Prisma 子进程可在一次性 migration 容器日志中回显完整 PostgreSQL 连接串,违反凭据不得进入日志的硬门槛。 关键设计取舍:migration lock 进程保留原有锁、返回码与诊断标记;仅捕获其子进程的合并输出,对 postgresql:// 或 postgres:// 连接串整体替换为 postgresql:// 后再输出。回归门禁以 0600 临时日志捕获 migration 输出并断言三类随机占位凭据均不存在。 验证:执行 python3 -m py_compile docker_litellm/work/run-migration-locked.py;执行 bash -n docker_litellm/work/start-litellm.sh docker_litellm/demo/scripts/smoke-baseline.sh docker_litellm/demo/scripts/test-secret-boundary.sh;执行 ./docker_litellm/demo/scripts/test-secret-boundary.sh;执行 ./docker_litellm/demo/scripts/smoke-baseline.sh --security-check;执行 ./docker_litellm/demo/scripts/test-verification-gates.sh;执行 git diff --check。关键运行报告:compose_config/readiness/inspect/argv/logs/temporary_files/management_smoke/cleanup 全部 passed,management_http_status=200。 影响面:仅 LiteLLM migration 子进程输出与 PH-1 回归断言;不变更数据库 schema、镜像、CI、P6/P7、镜像 digest 或 Compose 参数化。关联 Linear:CHE-664。 剩余风险:LiteLLM 最终进程环境中仍需要 LITELLM_MASTER_KEY 与 DATABASE_URL,这是冻结方案已接受的风险;本提交不改变该决策。 --- .../demo/scripts/test-secret-boundary.sh | 10 +++++++++- docker_litellm/work/run-migration-locked.py | 16 +++++++++++++++- 2 files changed, 24 insertions(+), 2 deletions(-) diff --git a/docker_litellm/demo/scripts/test-secret-boundary.sh b/docker_litellm/demo/scripts/test-secret-boundary.sh index fa373d1..a2aa894 100755 --- a/docker_litellm/demo/scripts/test-secret-boundary.sh +++ b/docker_litellm/demo/scripts/test-secret-boundary.sh @@ -23,6 +23,7 @@ generate_payload="${tmpdir}/generate.json" generate_response="${tmpdir}/generate-response.json" generated_key_file="${tmpdir}/generated.key" delete_payload="${tmpdir}/delete.json" +migration_output="${tmpdir}/migration.log" report_file="${LITELLM_SECRET_BOUNDARY_REPORT_FILE:-}" started=false compose_config_result="not_run" @@ -90,7 +91,14 @@ echo "PASS compose config: service environment omits LITELLM_MASTER_KEY, DATABAS started=true docker compose --env-file "$env_file" -p "$project" -f "$compose_file" up -d --wait postgres redis >/dev/null -docker compose --env-file "$env_file" -p "$project" -f "$compose_file" --profile migrate run --rm --no-deps litellm-migrate >/dev/null +docker compose --env-file "$env_file" -p "$project" -f "$compose_file" --profile migrate run --rm --no-deps litellm-migrate >"$migration_output" 2>&1 +chmod 600 "$migration_output" +if rg -Fq -- "$master_key" "$migration_output" \ + || rg -Fq -- "$postgres_password" "$migration_output" \ + || rg -Fq -- "$redis_password" "$migration_output"; then + echo "FAIL logs: credential value is present in migration output" >&2 + exit 1 +fi docker compose --env-file "$env_file" -p "$project" -f "$compose_file" --profile single up -d litellm-1 >/dev/null for attempt in $(seq 1 60); do if curl --silent --fail --max-time 3 "http://127.0.0.1:${publish_port}/health/readiness" | jq -e '.status == "healthy" and .db == "connected"' >/dev/null; then diff --git a/docker_litellm/work/run-migration-locked.py b/docker_litellm/work/run-migration-locked.py index 1dc643c..08aa384 100755 --- a/docker_litellm/work/run-migration-locked.py +++ b/docker_litellm/work/run-migration-locked.py @@ -8,12 +8,19 @@ import asyncio import os +import re import subprocess import sys from prisma import Prisma LOCK_ID = 548_019_700_001 +DATABASE_URL_PATTERN = re.compile(r"postgres(?:ql)?://[^\s'\"`]+") + + +def redact_migration_output(value: str) -> str: + """Keep migration diagnostics while preventing connection strings in logs.""" + return DATABASE_URL_PATTERN.sub("postgresql://", value) async def main() -> int: @@ -38,8 +45,15 @@ async def main() -> int: await asyncio.sleep(hold_seconds) print("P1_MIGRATION_EXECUTION_START", flush=True) completed = subprocess.run( - ["/bin/bash", "/opt/utils/start-litellm.sh", *sys.argv[1:]], check=False + ["/bin/bash", "/opt/utils/start-litellm.sh", *sys.argv[1:]], + check=False, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + text=True, + errors="replace", ) + if completed.stdout: + print(redact_migration_output(completed.stdout), end="", flush=True) print("P1_MIGRATION_EXECUTION_DONE", flush=True) return completed.returncode finally: From aa9d3b6ce54c3ea09b599b3caa82b98d1c5d28db Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Thu, 20 Aug 2026 07:59:13 +0800 Subject: [PATCH 75/87] =?UTF-8?q?chore(openclaw):=20=E5=BD=92=E6=A1=A3=20P?= =?UTF-8?q?6=20=E4=B8=80=E6=AC=A1=E6=80=A7=E8=AF=81=E6=8D=AE=E7=BC=96?= =?UTF-8?q?=E6=8E=92?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 动机与背景:LLM Hub 生产加固 PH-3 对应 X-07。D-10 要求将 P6 已冻结的固定组合、受限输入、黄金 runner、报告聚合与 runtime Compose 留在 Git 历史回读,而不是作为产品仓长期维护的运行入口。 关键设计取舍:删除 p6-runner、预处理/产品链/聚合器、固定输入模板、runtime Compose(其中原有 docker.sock:rw)、仅用于这些编排的 driver-flow 与 Compose-render 测试;不迁移文件。保留并重写 test-p6-gates.sh 为无 Docker、无网络、无凭据、无历史 commit/digest/绝对路径依赖的 OpenClaw 静态产品契约检查。P6 README 改为精确 git show 回读说明,父 README 移除固定组合事实。 验证:bash -n docker_openclaw/p6/scripts/test-p6-gates.sh;./docker_openclaw/p6/scripts/test-p6-gates.sh(PASS P6 gates: OpenClaw image state, entrypoint, ports, and standard Compose avoid Docker socket access.);git diff --check。未启动服务容器。 影响面:仅 docker_openclaw/p6 的历史证据编排、其说明与保留静态门禁;未改 OpenClaw Dockerfile、常规 demo Compose 或其他服务。关联 Linear:CHE-673(In Progress)。 剩余风险:跨仓 P6 真实黄金运行只能通过基线 fdbbab2155a9e088c37d2a8a2057178e19ac9534 的 Git 历史审阅,不能由本次保留门禁重放;P7 归档仍待本批下一提交完成。 --- docker_openclaw/README.md | 22 +- docker_openclaw/p6/README.md | 98 +- docker_openclaw/p6/docker-compose.runtime.yml | 222 ---- docker_openclaw/p6/p6-inputs.example.json | 36 - docker_openclaw/p6/scripts/p6-aggregate.sh | 58 -- docker_openclaw/p6/scripts/p6-full-driver.sh | 138 --- docker_openclaw/p6/scripts/p6-lib.sh | 264 ----- .../p6/scripts/p6-prepare-runtime.py | 424 -------- .../p6/scripts/p6-product-chain.py | 960 ------------------ docker_openclaw/p6/scripts/p6-runner.sh | 168 --- docker_openclaw/p6/scripts/p6-user-center.py | 58 -- .../p6/scripts/test-p6-compose-render.sh | 64 -- .../p6/scripts/test-p6-driver-flow.sh | 248 ----- docker_openclaw/p6/scripts/test-p6-gates.sh | 72 +- 14 files changed, 30 insertions(+), 2802 deletions(-) delete mode 100644 docker_openclaw/p6/docker-compose.runtime.yml delete mode 100644 docker_openclaw/p6/p6-inputs.example.json delete mode 100755 docker_openclaw/p6/scripts/p6-aggregate.sh delete mode 100755 docker_openclaw/p6/scripts/p6-full-driver.sh delete mode 100644 docker_openclaw/p6/scripts/p6-lib.sh delete mode 100755 docker_openclaw/p6/scripts/p6-prepare-runtime.py delete mode 100755 docker_openclaw/p6/scripts/p6-product-chain.py delete mode 100755 docker_openclaw/p6/scripts/p6-runner.sh delete mode 100755 docker_openclaw/p6/scripts/p6-user-center.py delete mode 100755 docker_openclaw/p6/scripts/test-p6-compose-render.sh delete mode 100755 docker_openclaw/p6/scripts/test-p6-driver-flow.sh diff --git a/docker_openclaw/README.md b/docker_openclaw/README.md index 8b692b7..4db1f2b 100644 --- a/docker_openclaw/README.md +++ b/docker_openclaw/README.md @@ -39,20 +39,8 @@ docker run -d \ labnow/openclaw:latest ``` -## P6 本地跨仓产品闭环 - -P6 的固定组合编排、黄金 runner、报告聚合和安全清理由 -[`p6/README.md`](p6/README.md) 维护。该入口只接受本地受限输入文件中 -固定的三仓 commit、`lab-dev` review_snapshot、RC1 bundle hash,以及下列冻结镜像事实:LiteLLM -`quay.io/labnow/litellm:1.97.0-ead62528e607` 的本地 ID/digest -`sha256:a2e115874c21b829bd052b18fc85be2f9217fb8244b82812c4ebc6e36f9824d1`; -上游 OpenClaw `quay.io/labnow/openclaw@sha256:edc85cc2068f5ec0df470f7d06daa0a4fbd78ef5ad6cf5b48f58381da839dd12`; -以及 P6 Workspace 本地镜像 -`quay.io/labnow/labnow-open:che-563-openclaw-product-closure-local` 的 ID -`sha256:c9c6a45637521cbbaeacea57fbb128696066fd91c5dff4521555f1bd5211f244`、 -实际 RepoDigest `quay.io/labnow/labnow-open@sha256:c9c6a45637521cbbaeacea57fbb128696066fd91c5dff4521555f1bd5211f244`、 -source commit `21019e0c24dc7b51747c2bef3cd90f5d259be839` 和上游 OpenClaw -base digest。Launcher、Shell 与 PostgreSQL、Redis、nginx support image 也必须 -提供并回读准确本地 image ID / RepoDigest。P6 通过 live DockerSpawner 创建 -Workspace,所有运行材料和数据卷均按 run 隔离;它仅以 local-only 方式运行, -不拉取、推送、发布或部署镜像,也不会把凭据写入仓库、报告或命令参数。 +## P6 冻结证据归档 + +P6 的跨仓固定组合、黄金 runner 与报告聚合属于已冻结的一次性验收证据,已从 +产品运行面删除。历史回读方式见 [`p6/README.md`](p6/README.md);它不是 OpenClaw +的构建、启动或 CI 入口。 diff --git a/docker_openclaw/p6/README.md b/docker_openclaw/p6/README.md index 28e4e32..4c92057 100644 --- a/docker_openclaw/p6/README.md +++ b/docker_openclaw/p6/README.md @@ -1,95 +1,15 @@ -# P6 OpenClaw 产品闭环(本仓编排) +# P6 OpenClaw 产品闭环(已归档) -本目录只承担 LLM Hub V1 / P6 的 `lab-dev` 职责:固定本地组合、创建隔离 -拓扑、执行黄金 runner、聚合脱敏报告并清理本轮资源。Shell、Launcher 和 -OpenClaw 的产品业务逻辑仍由各自仓库拥有。 +LLM Hub V1 / P6 的固定组合、黄金 runner、受限输入、运行时 Compose 与报告聚合 +均为一次性冻结验收证据,已按生产加固决策 D-10 从产品运行面删除。它们不是持续 +维护的 OpenClaw 启动入口,也不得重新用于日常或 CI 验证。 -当前状态:真实五组件 driver 与失败关闭门禁已写入工作树;在真实黄金 run、 -有界复审及 promotion 完成前,P6 仍不是 `verified`。 - -## 固定输入与 review_snapshot - -从 [`p6-inputs.example.json`](p6-inputs.example.json) 创建 Git 忽略的 -`p6-inputs.json`,权限必须为 `0400` 或 `0600`。输入只包含: - -- `lab-dev` 的 Phase branch、base、当前 `HEAD`、tracked diff SHA-256 和变更文件集; -- 其余三仓的准确 commit; -- LiteLLM、OpenClaw base、Workspace、Launcher、Shell、PostgreSQL、Redis、 - nginx 的准确本地 image ID 与 RepoDigest; -- P1 本地测试 `.env` 的绝对路径。 - -`lab-dev` 可以用受保护的 `review_snapshot` 进入 Review,不要求提前 commit。 -Runner 会重新计算: - -```bash -git diff --binary --full-index --no-ext-diff -- | shasum -a 256 -``` - -并核对分支、`HEAD`、文件集和 SHA-256。其他三仓必须停在准确 commit 且 tracked -工作树 clean。未知 untracked 文件不属于输入,也不会被读取、删除或修改。 - -## 真实拓扑 - -[`docker-compose.runtime.yml`](docker-compose.runtime.yml) 每次创建一个独立的: - -- 固定 LiteLLM + 独立 PostgreSQL / Redis / migration; -- 固定 Shell + 独立 PostgreSQL / migration; -- run-scoped User Center fixture; -- 固定 Launcher / live JupyterHub; -- run-scoped HTTPS LiteLLM gateway; -- 由 live DockerSpawner 创建的固定 OpenClaw Workspace。 - -Workspace 不由第二份 Compose 旁路创建。Launcher 通过真实 Shell 内部接口完成 -claim → materialize → Adapter apply/probe → activate,stop/restart/delete 时完成 -release。P1 `.env` 只由受版本控制的 preparer 程序化读取;所有测试 key、Hub -token、服务 token、数据库密码、KEK 和本地证书均在本轮 `0700` 目录内以 -`0400`/`0600` 文件生成,不进入输入、命令参数、Git 或脱敏报告。 - -## Runner 门禁 - -```bash -./docker_openclaw/p6/scripts/test-p6-gates.sh -./docker_openclaw/p6/scripts/test-p6-driver-flow.sh -./docker_openclaw/p6/scripts/test-p6-compose-render.sh -./docker_openclaw/p6/scripts/p6-runner.sh --input /secure/path/p6-inputs.json --preflight -./docker_openclaw/p6/scripts/p6-runner.sh --input /secure/path/p6-inputs.json --render -P6_RUN_ID=p6-<32hex> ./docker_openclaw/p6/scripts/p6-runner.sh --input /secure/path/p6-inputs.json --golden -P6_RUN_ID=p6- ./docker_openclaw/p6/scripts/p6-runner.sh --input /secure/path/p6-inputs.json --cleanup -``` - -`--golden` 在一个 run 中完成: - -1. 通过 Shell 真实 API 创建 connection、route、binding; -2. 通过 Shell 调用 live JupyterHub,由真实 DockerSpawner 创建 Workspace; -3. 核对最小 `model_access`、只读材料、Adapter status、访问入口和 readiness; -4. 执行 chat、stream、tool,并按 owner / Workspace / key / model / time 查询用量; -5. stop 后验证旧 key 被拒绝;restart 后验证 generation 增加、新 key 成功、 - 旧 key 仍拒绝、旧 generation 迟到 release 返回 409; -6. delete 后验证新 key 被拒绝、active lease 为零; -7. 扫描 Shell、Launcher、LiteLLM、Workspace 日志/进程/脱敏 inspect、OpenClaw - 配置与本轮 Workspace 文件,确认测试 Secret 零命中; -8. 只删除本 run 的准确容器、网络、数据卷、运行材料和临时文件。 - -Shell 在 `5c9411dfd3c4d7b1e606c0d9dc0c5e62313bc376` 的真实鼠标流程证据和 -LiteLLM `1.97.0` v2 key-alias 分页用量 smoke 按冻结 Review 结论复用;runner 不伪装成 -重新执行浏览器 UI。`test-p6-driver-flow.sh` -只验证编排、报告和 cleanup 的确定性,不能替代真实黄金 run。 - -## 证据与聚合 - -脱敏报告位于 Git 忽略的 `p6/artifacts/`。`provision`、`golden`、`cleanup` -三份 driver 报告绑定同一 run、输入 SHA-256 和准确 provenance。报告只保留 -结构断言、非敏感 ID、计数、状态和 SHA-256,不保留 key、Prompt、Response -正文、原始环境或私钥。 - -只有同一 run 的 preflight、golden、cleanup 都为 `passed`,且阶段报告 hash -一致时,才允许聚合: +最后可读快照是本批基线 +`fdbbab2155a9e088c37d2a8a2057178e19ac9534`。需要审阅历史材料时,在本仓执行: ```bash -./docker_openclaw/p6/scripts/p6-aggregate.sh \ - --artifacts docker_openclaw/p6/artifacts \ - --run-id p6- +git show fdbbab2155a9e088c37d2a8a2057178e19ac9534:docker_openclaw/p6/<路径> ``` -P6 只在本地运行,不拉取或推送产品分支,不发布镜像,不部署,也不修改任何 -`main`。 +仍保留的 `scripts/test-p6-gates.sh` 是不启动容器的静态产品契约检查;OpenClaw +的受支持构建与日常启动方式见父目录 README 和 `demo/`。 diff --git a/docker_openclaw/p6/docker-compose.runtime.yml b/docker_openclaw/p6/docker-compose.runtime.yml deleted file mode 100644 index 19c4c53..0000000 --- a/docker_openclaw/p6/docker-compose.runtime.yml +++ /dev/null @@ -1,222 +0,0 @@ -name: p6-runtime - -# One isolated P6 run. Product images are fixed in p6-inputs.json and validated -# before this file is used. Secret values are only read from run-scoped 0400/ -# 0600 files; this file and the retained rendering contain no credential. -services: - litellm-postgres: - image: ${P6_POSTGRES_IMAGE:?fixed PostgreSQL image required} - pull_policy: never - container_name: ${P6_LITELLM_POSTGRES_CONTAINER:?run-scoped container required} - restart: "no" - environment: - POSTGRES_DB: p6_litellm - POSTGRES_USER: p6_litellm - POSTGRES_PASSWORD_FILE: /run/secrets/litellm_postgres_password - secrets: [litellm_postgres_password] - volumes: - - litellm_postgres_data:/var/lib/postgresql/data - networks: [p6-runtime] - healthcheck: - test: ["CMD-SHELL", "pg_isready -U p6_litellm -d p6_litellm"] - interval: 3s - timeout: 3s - retries: 30 - - litellm-redis: - image: ${P6_REDIS_IMAGE:?fixed Redis image required} - pull_policy: never - restart: "no" - secrets: [litellm_redis_password] - tmpfs: - - /run/p6:mode=0700 - command: - - /bin/sh - - -ec - - >- - umask 077; - { printf 'appendonly yes\nrequirepass '; cat /run/secrets/litellm_redis_password; printf '\n'; } > /run/p6/redis.conf; - exec redis-server /run/p6/redis.conf - volumes: - - litellm_redis_data:/data - networks: [p6-runtime] - healthcheck: - test: ["CMD-SHELL", "REDISCLI_AUTH=$$(cat /run/secrets/litellm_redis_password) redis-cli --no-auth-warning ping | grep -qx PONG"] - interval: 3s - timeout: 3s - retries: 30 - - litellm-migrate: - image: ${P6_LITELLM_IMAGE:?fixed LiteLLM image required} - pull_policy: never - restart: "no" - env_file: ${P6_LITELLM_ENV_FILE:?restricted LiteLLM env required} - command: ["python3", "/opt/utils/run-migration-locked.py", "--config", "config.migrate.yaml", "--skip_server_startup", "--enforce_prisma_migration_check"] - secrets: [litellm_redis_password] - volumes: - - ${P6_LITELLM_CONFIG:?fixed LiteLLM config required}:/opt/litellm/config.yaml:ro - - ${P6_LITELLM_MIGRATE_CONFIG:?fixed LiteLLM migration config required}:/opt/litellm/config.migrate.yaml:ro - - ${P6_LITELLM_START_SCRIPT:?fixed LiteLLM start script required}:/opt/utils/start-litellm.sh:ro - - ${P6_LITELLM_MIGRATION_SCRIPT:?fixed LiteLLM migration script required}:/opt/utils/run-migration-locked.py:ro - depends_on: - litellm-postgres: {condition: service_healthy} - litellm-redis: {condition: service_healthy} - networks: [p6-runtime] - - litellm: - image: ${P6_LITELLM_IMAGE:?fixed LiteLLM image required} - pull_policy: never - container_name: ${P6_LITELLM_CONTAINER:?run-scoped container required} - restart: "no" - env_file: ${P6_LITELLM_ENV_FILE:?restricted LiteLLM env required} - command: ["start-litellm.sh"] - secrets: [litellm_redis_password] - volumes: - - ${P6_LITELLM_CONFIG:?fixed LiteLLM config required}:/opt/litellm/config.yaml:ro - - ${P6_LITELLM_MIGRATE_CONFIG:?fixed LiteLLM migration config required}:/opt/litellm/config.migrate.yaml:ro - - ${P6_LITELLM_START_SCRIPT:?fixed LiteLLM start script required}:/opt/utils/start-litellm.sh:ro - - ${P6_LITELLM_MIGRATION_SCRIPT:?fixed LiteLLM migration script required}:/opt/utils/run-migration-locked.py:ro - depends_on: - litellm-migrate: {condition: service_completed_successfully} - networks: [p6-runtime] - healthcheck: - test: ["CMD-SHELL", "python3 -c \"import urllib.request; urllib.request.urlopen('http://127.0.0.1:4000/health/readiness', timeout=3)\""] - interval: 5s - timeout: 5s - retries: 36 - start_period: 20s - - litellm-gateway: - image: ${P6_NGINX_IMAGE:?fixed nginx image required} - pull_policy: never - restart: "no" - ports: - - "127.0.0.1::4443" - volumes: - - ${P6_NGINX_CONFIG:?run-scoped nginx config required}:/etc/nginx/nginx.conf:ro - - ${P6_CA_CERT:?run-scoped certificate required}:/run/p6/p6-ca.pem:ro - - ${P6_CA_KEY:?run-scoped private key required}:/run/p6/p6-ca.key:ro - depends_on: - litellm: {condition: service_healthy} - networks: [p6-runtime] - healthcheck: - test: ["CMD-SHELL", "wget --no-check-certificate -qO- https://127.0.0.1:4443/health/readiness >/dev/null"] - interval: 3s - timeout: 3s - retries: 30 - - shell-postgres: - image: ${P6_POSTGRES_IMAGE:?fixed PostgreSQL image required} - pull_policy: never - container_name: ${P6_SHELL_POSTGRES_CONTAINER:?run-scoped container required} - restart: "no" - environment: - POSTGRES_DB: p6_shell - POSTGRES_USER: p6_shell - POSTGRES_PASSWORD_FILE: /run/secrets/shell_postgres_password - secrets: [shell_postgres_password] - volumes: - - shell_postgres_data:/var/lib/postgresql/data - networks: [p6-runtime] - healthcheck: - test: ["CMD-SHELL", "pg_isready -U p6_shell -d p6_shell"] - interval: 3s - timeout: 3s - retries: 30 - - shell-migrate: - image: ${P6_POSTGRES_IMAGE:?fixed PostgreSQL image required} - pull_policy: never - restart: "no" - entrypoint: ["/bin/sh", "-ec"] - command: - - >- - export PGPASSWORD="$$(cat /run/secrets/shell_postgres_password)"; - exec psql -v ON_ERROR_STOP=1 -h shell-postgres -U p6_shell -d p6_shell -f /run/p6/001_initial.sql - secrets: [shell_postgres_password] - volumes: - - ${P6_SHELL_MIGRATION:?fixed Shell migration required}:/run/p6/001_initial.sql:ro - depends_on: - shell-postgres: {condition: service_healthy} - networks: [p6-runtime] - - user-center: - image: ${P6_LAUNCHER_IMAGE:?fixed Launcher image required} - pull_policy: never - restart: "no" - entrypoint: ["python", "/run/p6/p6-user-center.py"] - environment: - P6_OWNER_A: ${P6_OWNER_A:?owner A required} - P6_OWNER_B: ${P6_OWNER_B:?owner B required} - volumes: - - ${P6_USER_CENTER_SCRIPT:?fixed fixture script required}:/run/p6/p6-user-center.py:ro - networks: [p6-runtime] - healthcheck: - test: ["CMD-SHELL", "python -c \"import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health', timeout=3)\""] - interval: 3s - timeout: 3s - retries: 20 - - launcher: - image: ${P6_LAUNCHER_IMAGE:?fixed Launcher image required} - pull_policy: never - container_name: ${P6_LAUNCHER_CONTAINER:?run-scoped container required} - restart: "no" - env_file: ${P6_LAUNCHER_ENV_FILE:?restricted Launcher env required} - ports: - - "127.0.0.1::8000" - volumes: - - /var/run/docker.sock:/var/run/docker.sock:rw - - ${P6_WORKSPACE_ROOT:?isolated Workspace root required}:${P6_WORKSPACE_ROOT}:rw - - ${P6_LAUNCHER_DATA_DIR:?isolated Launcher state required}:/opt/jupyterhub/data:rw - - ${P6_LAUNCHER_APP_CONF:?run-scoped app config required}:/opt/jupyterhub/resource/config/app.conf:ro - - ${P6_MODEL_ACCESS_CONFIG:?restricted model access config required}:/run/p6/model-access.json:ro - - ${P6_CA_CERT:?run-scoped CA required}:${P6_CA_CERT}:ro - depends_on: - shell: {condition: service_healthy} - networks: - p6-runtime: - aliases: [launcher] - healthcheck: - test: ["CMD-SHELL", "curl -fsS http://127.0.0.1:8000/studio/hub/health >/dev/null"] - interval: 5s - timeout: 5s - retries: 36 - start_period: 20s - - shell: - image: ${P6_SHELL_IMAGE:?fixed Shell image required} - pull_policy: never - container_name: ${P6_SHELL_CONTAINER:?run-scoped container required} - restart: "no" - env_file: ${P6_SHELL_ENV_FILE:?restricted Shell env required} - ports: - - "127.0.0.1::3002" - depends_on: - shell-migrate: {condition: service_completed_successfully} - user-center: {condition: service_healthy} - litellm-gateway: {condition: service_healthy} - networks: [p6-runtime] - healthcheck: - test: ["CMD-SHELL", "node -e \"fetch('http://127.0.0.1:3002/console/api/model-access/connections').then(r=>process.exit(r.status===401?0:1)).catch(()=>process.exit(1))\""] - interval: 5s - timeout: 5s - retries: 36 - start_period: 20s - -secrets: - litellm_postgres_password: - file: ${P6_LITELLM_POSTGRES_PASSWORD_FILE:?restricted LiteLLM PostgreSQL password required} - litellm_redis_password: - file: ${P6_REDIS_PASSWORD_FILE:?restricted Redis password required} - shell_postgres_password: - file: ${P6_SHELL_POSTGRES_PASSWORD_FILE:?restricted Shell PostgreSQL password required} - -volumes: - litellm_postgres_data: - litellm_redis_data: - shell_postgres_data: - -networks: - p6-runtime: - name: ${P6_RUNTIME_NETWORK:?isolated network required} diff --git a/docker_openclaw/p6/p6-inputs.example.json b/docker_openclaw/p6/p6-inputs.example.json deleted file mode 100644 index 99e3fe3..0000000 --- a/docker_openclaw/p6/p6-inputs.example.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "schema_version": "p6-inputs/v2", - "contract_version": "v1alpha1", - "contract_bundle_sha256": "d289dff9bcaa3d28035c5ed2e56b806f4b3b37fdca3159352d22f0c03942e202", - "control_commit": "2eb71d7590739df3de8db2f8cf9098154a397f0b", - "review_policy_commit": "680ca92661a08254eb396ab809f478bbdba3510e", - "repositories": { - "lab_dev": { - "path": "/absolute/path/to/lab-dev", - "delivery_identity": "review_snapshot", - "branch": "dev/che-563-openclaw-product-closure", - "phase_base_commit": "940325578bae9905673965d6dc489130ab4b6a46", - "head_commit": "1b4562899e03eacdee5a86eb55b47d5e12117ee8", - "tracked_diff_sha256": "0000000000000000000000000000000000000000000000000000000000000000", - "changed_files": ["docker_openclaw/p6/README.md"] - }, - "labnow_open": {"path": "/absolute/path/to/labnow-open", "delivery_identity": "commit", "commit": "21019e0c24dc7b51747c2bef3cd90f5d259be839"}, - "labnow_shell": {"path": "/absolute/path/to/labnow-shell", "delivery_identity": "commit", "commit": "5c9411dfd3c4d7b1e606c0d9dc0c5e62313bc376"}, - "labnow_launcher": {"path": "/absolute/path/to/labnow-launcher", "delivery_identity": "commit", "commit": "c84edea3e051d561f28d9f99235563cf491aaeb2"} - }, - "images": { - "litellm": {"ref": "quay.io/labnow/litellm:1.97.0-ead62528e607", "image_id": "sha256:a2e115874c21b829bd052b18fc85be2f9217fb8244b82812c4ebc6e36f9824d1", "provenance": "repo_digest", "repo_digest": "quay.io/labnow/litellm@sha256:a2e115874c21b829bd052b18fc85be2f9217fb8244b82812c4ebc6e36f9824d1"}, - "openclaw_base": {"ref": "quay.io/labnow/openclaw@sha256:edc85cc2068f5ec0df470f7d06daa0a4fbd78ef5ad6cf5b48f58381da839dd12", "image_id": "sha256:edc85cc2068f5ec0df470f7d06daa0a4fbd78ef5ad6cf5b48f58381da839dd12", "provenance": "repo_digest", "repo_digest": "quay.io/labnow/openclaw@sha256:edc85cc2068f5ec0df470f7d06daa0a4fbd78ef5ad6cf5b48f58381da839dd12"}, - "openclaw_workspace": {"ref": "quay.io/labnow/labnow-open:che-563-openclaw-product-closure-local", "image_id": "sha256:c9c6a45637521cbbaeacea57fbb128696066fd91c5dff4521555f1bd5211f244", "provenance": "local_build", "repo_digest": "quay.io/labnow/labnow-open@sha256:c9c6a45637521cbbaeacea57fbb128696066fd91c5dff4521555f1bd5211f244", "source_repository": "labnow_open", "source_commit": "21019e0c24dc7b51747c2bef3cd90f5d259be839", "base_image_digest": "quay.io/labnow/openclaw@sha256:edc85cc2068f5ec0df470f7d06daa0a4fbd78ef5ad6cf5b48f58381da839dd12"} - }, - "local_only_images": { - "launcher": {"ref": "quay.io/labnow/labnow-launcher:che-563-openclaw-product-closure-local", "image_id": "sha256:6f9732fda8b86d9bfe4596e848025cc38448da4b17dfea8520e046a32b32e61f", "provenance": "local_build", "repo_digest": "quay.io/labnow/labnow-launcher@sha256:6f9732fda8b86d9bfe4596e848025cc38448da4b17dfea8520e046a32b32e61f", "source_repository": "labnow_launcher", "source_commit": "c84edea3e051d561f28d9f99235563cf491aaeb2", "base_image_digest": "quay.io/labnow/dev-hub-traefik@sha256:22e1857a5edcd2ad4468dffee32f21323f8dfcfa47a77c4b7f7386b9f50b8398", "oauth2_proxy_sha256": "6df0d30fe823d9b25e8bff4cfb3df9a9b6c1013463e9526388c29ba87ab32427"}, - "shell": {"ref": "quay.io/labnow/labnow-shell:che-563-openclaw-product-closure-local", "image_id": "sha256:d7ed71cf58eddf72642d61d4442a0820632060fac9f4eb611b28062b7f38c54d", "provenance": "local_build", "repo_digest": "quay.io/labnow/labnow-shell@sha256:d7ed71cf58eddf72642d61d4442a0820632060fac9f4eb611b28062b7f38c54d", "source_repository": "labnow_shell", "source_commit": "5c9411dfd3c4d7b1e606c0d9dc0c5e62313bc376", "base_image_digest": "quay.io/labnow/node@sha256:fd09d9de9b7aa927493acbafbb7d399c089465e988f2a6a240428cdbbd5424e2"} - }, - "support_images": { - "postgres": {"ref": "postgres:17-alpine@sha256:742f40ea20b9ff2ff31db5458d127452988a2164df9e17441e191f3b72252193", "image_id": "sha256:742f40ea20b9ff2ff31db5458d127452988a2164df9e17441e191f3b72252193", "provenance": "repo_digest", "repo_digest": "postgres@sha256:742f40ea20b9ff2ff31db5458d127452988a2164df9e17441e191f3b72252193"}, - "redis": {"ref": "redis:7.4-alpine@sha256:e7723ff73d963f5cc6d9c4643ea3d989527a402a319239054e9472a7fb9219a2", "image_id": "sha256:e7723ff73d963f5cc6d9c4643ea3d989527a402a319239054e9472a7fb9219a2", "provenance": "repo_digest", "repo_digest": "redis@sha256:e7723ff73d963f5cc6d9c4643ea3d989527a402a319239054e9472a7fb9219a2"}, - "nginx": {"ref": "nginx:alpine@sha256:2f07d83bf561b506400dc183b1b2003803e39efbd22451f848adaba14d28c7c7", "image_id": "sha256:2f07d83bf561b506400dc183b1b2003803e39efbd22451f848adaba14d28c7c7", "provenance": "repo_digest", "repo_digest": "nginx@sha256:2f07d83bf561b506400dc183b1b2003803e39efbd22451f848adaba14d28c7c7"} - }, - "runtime": {"p1_env_file": "/absolute/path/to/lab-dev/docker_litellm/demo/.env"} -} diff --git a/docker_openclaw/p6/scripts/p6-aggregate.sh b/docker_openclaw/p6/scripts/p6-aggregate.sh deleted file mode 100755 index bdba783..0000000 --- a/docker_openclaw/p6/scripts/p6-aggregate.sh +++ /dev/null @@ -1,58 +0,0 @@ -#!/usr/bin/env bash -# Refuse incomplete P6 evidence. This creates a final report only after a -# complete preflight, golden chain, and explicit cleanup report for one run. -set -euo pipefail - -usage() { echo "Usage: $0 --artifacts DIR --run-id p6-<32hex>" >&2; } -artifacts=""; run_id="" -while (($#)); do - case "$1" in - --artifacts) artifacts="${2:-}"; shift 2 ;; - --run-id) run_id="${2:-}"; shift 2 ;; - *) usage; exit 2 ;; - esac -done -[[ "$run_id" =~ ^p6-[a-f0-9]{32}$ && -d "$artifacts" ]] || { usage; exit 2; } -reports=("$artifacts/p6-preflight-${run_id}.json" "$artifacts/p6-golden-${run_id}.json" "$artifacts/p6-cleanup-${run_id}.json") -for report in "${reports[@]}"; do - [[ -f "$report" ]] || { echo "P6_ERROR:EVIDENCE_INCOMPLETE" >&2; exit 1; } - jq -e --arg run "$run_id" ' - .schema_version == "p6-report/v1" and .run_id == $run and .result == "passed" and .phase == "completed" - and .content_redacted == true and .contract_version == "v1alpha1" - and .contract_bundle_sha256 == "d289dff9bcaa3d28035c5ed2e56b806f4b3b37fdca3159352d22f0c03942e202" - and ([.images.litellm,.images.openclaw_base] | all(.[]; .provenance == "repo_digest" and (.repo_digest | type == "string" and test("^quay\\.io/labnow/(litellm|openclaw)@sha256:[0-9a-f]{64}$")))) - and (.images.openclaw_workspace.provenance == "local_build") - and (.images.openclaw_workspace.repo_digest | type == "string" and test("^quay\\.io/labnow/labnow-open@sha256:[0-9a-f]{64}$")) - and (.images.openclaw_workspace.source_repository == "labnow_open") - and (.images.openclaw_workspace.source_commit | type == "string" and test("^[0-9a-f]{40}$")) - and (.images.openclaw_workspace.base_image_digest == .images.openclaw_base.repo_digest) - and ([.local_only_images.launcher,.local_only_images.shell] | all(.[]; .provenance == "local_build" and (.repo_digest | type == "string" and test("^quay\\.io/labnow/labnow-(launcher|shell)@sha256:[0-9a-f]{64}$")))) - and ([.support_images.postgres,.support_images.redis,.support_images.nginx] | all(.[]; .provenance == "repo_digest" and (.repo_digest | type == "string" and test("^[a-z0-9./_-]+@sha256:[0-9a-f]{64}$")))) - ' "$report" >/dev/null || { echo "P6_ERROR:EVIDENCE_REJECTED" >&2; exit 1; } -done -input_hash="$(jq -r '.input_sha256' "${reports[0]}")" -for report in "${reports[@]}"; do [[ "$(jq -r '.input_sha256' "$report")" == "$input_hash" ]] || { echo "P6_ERROR:INPUT_HASH_MISMATCH" >&2; exit 1; }; done -metadata="$(jq -c '{contract_version,contract_bundle_sha256,control_commit,review_policy_commit,repositories,images,local_only_images,support_images}' "${reports[0]}")" -for report in "${reports[@]}"; do [[ "$(jq -c '{contract_version,contract_bundle_sha256,control_commit,review_policy_commit,repositories,images,local_only_images,support_images}' "$report")" == "$metadata" ]] || { echo "P6_ERROR:METADATA_MISMATCH" >&2; exit 1; }; done -stage_reports="$(jq -c '.driver_stage_reports' "${reports[1]}")" -[[ "$stage_reports" != "null" ]] || { echo "P6_ERROR:DRIVER_STAGE_EVIDENCE_INCOMPLETE" >&2; exit 1; } -for report in "${reports[1]}" "${reports[2]}"; do - [[ "$(jq -c '.driver_stage_reports' "$report")" == "$stage_reports" ]] || { echo "P6_ERROR:DRIVER_STAGE_METADATA_MISMATCH" >&2; exit 1; } -done -for action in provision golden cleanup; do - stage_path="$(jq -r --arg action "$action" '.driver_stage_reports[$action].path' "${reports[1]}")" - stage_sha="$(jq -r --arg action "$action" '.driver_stage_reports[$action].sha256' "${reports[1]}")" - [[ -f "$stage_path" && ! -L "$stage_path" && "$stage_sha" =~ ^[0-9a-f]{64}$ ]] || { echo "P6_ERROR:DRIVER_STAGE_EVIDENCE_INCOMPLETE" >&2; exit 1; } - [[ "$(shasum -a 256 "$stage_path" | awk '{print $1}')" == "$stage_sha" ]] || { echo "P6_ERROR:DRIVER_STAGE_HASH_MISMATCH" >&2; exit 1; } - jq -e --arg action "$action" --arg run "$run_id" --arg input_sha "$input_hash" ' - .run_id == $run and .input_sha256 == $input_sha and .result == "passed" and .content_redacted == true - and .schema_version == (if $action == "provision" then "p6-driver-provision/v1" elif $action == "golden" then "p6-driver-report/v1" else "p6-driver-cleanup/v1" end) - ' "$stage_path" >/dev/null || { echo "P6_ERROR:DRIVER_STAGE_EVIDENCE_REJECTED" >&2; exit 1; } -done -output="$artifacts/p6-final-${run_id}.json" -tmp="$(mktemp "$artifacts/.p6-final.XXXXXX")" -jq -n --arg run_id "$run_id" --arg input_sha256 "$input_hash" --arg tested_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" --argjson metadata "$metadata" --argjson driver_stage_reports "$stage_reports" \ - '{schema_version:"p6-final-report/v1",run_id:$run_id,input_sha256:$input_sha256,tested_at:$tested_at,result:"passed",phase:"completed",content_redacted:true,driver_stage_reports:$driver_stage_reports} + $metadata' > "$tmp" -chmod 600 "$tmp" -mv -f "$tmp" "$output" -printf '%s\n' "$output" diff --git a/docker_openclaw/p6/scripts/p6-full-driver.sh b/docker_openclaw/p6/scripts/p6-full-driver.sh deleted file mode 100755 index fd58fba..0000000 --- a/docker_openclaw/p6/scripts/p6-full-driver.sh +++ /dev/null @@ -1,138 +0,0 @@ -#!/usr/bin/env bash -# Checked-in P6 full-topology driver. It owns one run-scoped Compose project, -# invokes the real product chain, retains only redacted stage evidence, and -# removes its exact containers/network/volumes on cleanup. -set -euo pipefail - -script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -p6_dir="$(cd "${script_dir}/.." && pwd)" -source "${script_dir}/p6-lib.sh" - -: "${P6_DRIVER_ACTION:?P6_DRIVER_ACTION is required}" -: "${P6_RUN_ID:?P6_RUN_ID is required}" -: "${P6_INPUT_FILE:?P6_INPUT_FILE is required}" -: "${P6_DRIVER_REPORT:?P6_DRIVER_REPORT is required}" -: "${P6_SECRET_PATTERN_FILE:?P6_SECRET_PATTERN_FILE is required}" -: "${P6_WORK_DIR:?P6_WORK_DIR is required}" -: "${P6_ARTIFACTS_DIR:?P6_ARTIFACTS_DIR is required}" - -[[ "$P6_DRIVER_ACTION" =~ ^(provision|golden|cleanup)$ ]] || p6_die "DRIVER_ACTION_INVALID" 79 -[[ "$P6_RUN_ID" =~ ^p6-[a-f0-9]{32}$ ]] || p6_die "RUN_ID_INVALID" 79 -[[ -f "$P6_INPUT_FILE" && ! -L "$P6_INPUT_FILE" ]] || p6_die "INPUT_FILE_REQUIRED" 79 -[[ "$P6_DRIVER_REPORT" == "$P6_ARTIFACTS_DIR"/* && ! -L "$P6_DRIVER_REPORT" ]] || p6_die "DRIVER_REPORT_PATH_INVALID" 79 - -input_sha256="$(p6_sha256 "$P6_INPUT_FILE")" -state_file="${P6_WORK_DIR}/driver-state.json" -runtime_env="${P6_WORK_DIR}/runtime.env" -product_config="${P6_WORK_DIR}/config/driver-config.json" -product_report="${P6_WORK_DIR}/product-chain.json" -short="${P6_RUN_ID#p6-}" -short="${short:0:12}" -project="p6-runtime-${short}" -compose_file="${p6_dir}/docker-compose.runtime.yml" -mkdir -p "$P6_WORK_DIR" "$P6_ARTIFACTS_DIR" -chmod 700 "$P6_WORK_DIR" "$P6_ARTIFACTS_DIR" - -write_report() { - local schema="$1" result="$2" payload="$3" tmp - tmp="$(mktemp "${P6_ARTIFACTS_DIR}/.p6-driver.XXXXXX")" - jq -n \ - --arg schema "$schema" \ - --arg result "$result" \ - --arg run_id "$P6_RUN_ID" \ - --arg input_sha256 "$input_sha256" \ - --arg tested_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ - --argjson payload "$payload" \ - '{schema_version:$schema,result:$result,run_id:$run_id,input_sha256:$input_sha256,tested_at:$tested_at,content_redacted:true} + $payload' > "$tmp" - chmod 600 "$tmp" - mv -f "$tmp" "$P6_DRIVER_REPORT" - chmod 600 "$P6_DRIVER_REPORT" -} - -compose() { - docker compose --project-name "$project" --env-file "$runtime_env" -f "$compose_file" "$@" -} - -prepare_runtime() { - P6_ARTIFACTS_DIR="$P6_ARTIFACTS_DIR" \ - "${script_dir}/p6-prepare-runtime.py" - p6_require_regular_0600 "$runtime_env" - p6_require_regular_0600 "$product_config" -} - -container_absent() { - ! docker container inspect "$1" >/dev/null 2>&1 -} - -cleanup_resources() { - local launcher_container="" workspace_container="" - if [[ -f "$product_config" && ! -L "$product_config" ]]; then - launcher_container="$(jq -r '.launcher_container // empty' "$product_config" 2>/dev/null || true)" - workspace_container="$(jq -r '.workspace_container // empty' "$product_config" 2>/dev/null || true)" - fi - if [[ -n "$workspace_container" ]]; then - docker rm -f "$workspace_container" >/dev/null 2>&1 || true - fi - if [[ -f "$runtime_env" && ! -L "$runtime_env" ]]; then - compose down --volumes --remove-orphans >/dev/null 2>&1 || true - fi - if [[ -n "$launcher_container" ]]; then - container_absent "$launcher_container" || return 1 - fi - if [[ -n "$workspace_container" ]]; then - container_absent "$workspace_container" || return 1 - fi - [[ -z "$(docker container ls -aq --filter "label=com.docker.compose.project=${project}")" ]] || return 1 - [[ -z "$(docker volume ls -q --filter "label=com.docker.compose.project=${project}")" ]] || return 1 - ! docker network inspect "p6net-${short}" >/dev/null 2>&1 || return 1 -} - -case "$P6_DRIVER_ACTION" in - provision) - prepare_runtime - if ! compose up -d --wait >/dev/null; then - write_report "p6-driver-provision/v1" "failed" "$(jq -n --arg project "$project" '{project:$project,error_code:"TOPOLOGY_START_FAILED"}')" - exit 1 - fi - jq -n --arg project "$project" '{project:$project,topology:{litellm:"started",shell:"started",jupyterhub:"started",launcher:"started",workspace:"deferred_to_golden"}}' > "$state_file" - chmod 600 "$state_file" - write_report "p6-driver-provision/v1" "passed" "$(jq -n --arg project "$project" '{project:$project,topology:{litellm:"started",shell:"started",jupyterhub:"started",launcher:"started",workspace:"deferred_to_golden"},isolation:{network:"run_scoped",volumes:"run_scoped"}}')" - ;; - golden) - [[ -f "$state_file" && ! -L "$state_file" ]] || p6_die "TOPOLOGY_STATE_REQUIRED" 79 - p6_require_regular_0600 "$product_config" - rm -f "$product_report" "$P6_SECRET_PATTERN_FILE" - P6_PRODUCT_CONFIG_FILE="$product_config" \ - P6_PRODUCT_REPORT_FILE="$product_report" \ - P6_SECRET_PATTERN_FILE="$P6_SECRET_PATTERN_FILE" \ - "${script_dir}/p6-product-chain.py" - p6_require_regular_0600 "$product_report" - p6_require_regular_0600 "$P6_SECRET_PATTERN_FILE" - jq -e ' - .schema_version == "p6-product-chain-report/v1" - and .result == "passed" - and .content_redacted == true - and .checks.console_ui == "reused_verified_evidence" - and ([.checks.test_resource_provision,.checks.binding_payload,.checks.jupyterhub_dockerspawner,.checks.launcher_claim_activate_release,.checks.openclaw_apply_probe_readiness,.checks.chat,.checks.stream,.checks.tool,.checks.usage,.checks.owner_negative,.checks.prompt_response_absent,.checks.revoke,.checks.generation_restart,.checks.late_release,.checks.delete,.checks.zero_active_leases] | all(. == "passed")) - and (.scan_roots | type == "array" and length >= 1 and all(.[]; type == "string" and startswith("/"))) - ' "$product_report" >/dev/null || p6_die "PRODUCT_CHAIN_REPORT_INVALID" 79 - checks="$(jq -c '.checks' "$product_report")" - scan_roots="$(jq -c --arg product "$product_report" '.scan_roots + [$product] | unique' "$product_report")" - product_summary="$(jq -c '{binding,runtime,data_plane,usage,lifecycle}' "$product_report")" - write_report "p6-driver-report/v1" "passed" "$(jq -n \ - --arg patterns "$P6_SECRET_PATTERN_FILE" \ - --arg product_sha "$(p6_sha256 "$product_report")" \ - --argjson checks "$checks" \ - --argjson scan_roots "$scan_roots" \ - --argjson product_summary "$product_summary" \ - '{checks:$checks,secret_pattern_file:$patterns,scan_roots:$scan_roots,product_report_sha256:$product_sha,product:$product_summary}')" - ;; - cleanup) - cleanup_resources || p6_die "TOPOLOGY_RESOURCE_REMAINS" 79 - rm -f "$state_file" "$P6_SECRET_PATTERN_FILE" "$product_report" - rm -rf "${P6_WORK_DIR}/config" "${P6_WORK_DIR}/secrets" "${P6_WORK_DIR}/surfaces" "${P6_WORK_DIR}/workspace" "${P6_WORK_DIR}/launcher-data" - rm -f "$runtime_env" - [[ ! -e "$state_file" && ! -e "$runtime_env" && ! -e "${P6_WORK_DIR}/config" && ! -e "${P6_WORK_DIR}/secrets" && ! -e "${P6_WORK_DIR}/surfaces" && ! -e "${P6_WORK_DIR}/workspace" ]] || p6_die "TOPOLOGY_TEMPORARY_MATERIAL_REMAINS" 79 - write_report "p6-driver-cleanup/v1" "passed" "$(jq -n '{resources:{litellm:"absent",shell:"absent",jupyterhub:"absent",launcher:"absent",workspace:"absent",runtime_material:"absent",temporary_files:"absent",processes:"absent",network:"absent",volumes:"absent"}}')" - ;; -esac diff --git a/docker_openclaw/p6/scripts/p6-lib.sh b/docker_openclaw/p6/scripts/p6-lib.sh deleted file mode 100644 index 89b057b..0000000 --- a/docker_openclaw/p6/scripts/p6-lib.sh +++ /dev/null @@ -1,264 +0,0 @@ -#!/usr/bin/env bash -# Shared fail-closed helpers for the P6 local-only runner. Never source an -# environment file and never print a credential or credential fingerprint. -set -euo pipefail - -p6_die() { - printf 'P6_ERROR:%s\n' "$1" >&2 - return "${2:-1}" -} - -p6_run_id() { - python3 -c 'import secrets; print("p6-" + secrets.token_hex(16))' -} - -p6_sha256() { - shasum -a 256 "$1" | awk '{print $1}' -} - -p6_require_regular_0600() { - local path="$1" mode - [[ -f "$path" && ! -L "$path" ]] || { p6_die "SECURE_FILE_REQUIRED" 64; return $?; } - if mode="$(stat -f '%Lp' "$path" 2>/dev/null)"; then :; else - mode="$(stat -c '%a' "$path")" - fi - [[ "$mode" == 400 || "$mode" == 600 ]] || { p6_die "SECURE_FILE_MODE_REQUIRED" 65; return $?; } -} - -p6_repository_metadata() { - jq -c ' - .repositories | with_entries( - .value = if .value.delivery_identity == "commit" then - {delivery_identity:"commit",commit:.value.commit} - else - {delivery_identity:"review_snapshot",branch:.value.branch,phase_base_commit:.value.phase_base_commit,head_commit:.value.head_commit,tracked_diff_sha256:.value.tracked_diff_sha256,changed_files:.value.changed_files} - end - ) - ' "$P6_INPUT_FILE" -} - -p6_write_report() { - local report="$1" result="$2" phase="$3" reason="${4:-}" extra="${5:-}" temp metadata - [[ -n "$extra" ]] || extra='{}' - mkdir -p "$(dirname "$report")" - chmod 700 "$(dirname "$report")" - temp="$(mktemp "$(dirname "$report")/.p6-report.XXXXXX")" - metadata="$(jq -c --argjson repositories "$(p6_repository_metadata)" '{contract_version,contract_bundle_sha256,control_commit,review_policy_commit,repositories:$repositories,images,local_only_images,support_images}' "$P6_INPUT_FILE")" - jq -n \ - --arg run_id "$P6_RUN_ID" \ - --arg tested_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ - --arg result "$result" \ - --arg phase "$phase" \ - --arg reason "$reason" \ - --arg input_sha256 "$(p6_sha256 "$P6_INPUT_FILE")" \ - --argjson metadata "$metadata" \ - --argjson extra "$extra" \ - '{schema_version:"p6-report/v1",run_id:$run_id,tested_at:$tested_at,result:$result,phase:$phase,input_sha256:$input_sha256,content_redacted:true} - + $metadata - + $extra - + (if $reason == "" then {} else {reason:$reason} end)' > "$temp" - chmod 600 "$temp" - mv -f "$temp" "$report" -} - -p6_stage_reports_json() { - local artifact_dir="$1" run_id="$2" action path hash - for action in provision golden cleanup; do - path="${artifact_dir}/p6-driver-${action}-${run_id}.json" - [[ -f "$path" && ! -L "$path" ]] || { p6_die "DRIVER_STAGE_REPORT_MISSING" 78; return $?; } - hash="$(p6_sha256 "$path")" - [[ "$hash" =~ ^[0-9a-f]{64}$ ]] || { p6_die "DRIVER_STAGE_REPORT_HASH_INVALID" 78; return $?; } - done - jq -n \ - --arg provision "${artifact_dir}/p6-driver-provision-${run_id}.json" \ - --arg provision_sha "$(p6_sha256 "${artifact_dir}/p6-driver-provision-${run_id}.json")" \ - --arg golden "${artifact_dir}/p6-driver-golden-${run_id}.json" \ - --arg golden_sha "$(p6_sha256 "${artifact_dir}/p6-driver-golden-${run_id}.json")" \ - --arg cleanup "${artifact_dir}/p6-driver-cleanup-${run_id}.json" \ - --arg cleanup_sha "$(p6_sha256 "${artifact_dir}/p6-driver-cleanup-${run_id}.json")" \ - '{driver_stage_reports:{provision:{path:$provision,sha256:$provision_sha},golden:{path:$golden,sha256:$golden_sha},cleanup:{path:$cleanup,sha256:$cleanup_sha}}}' -} - -p6_json_string() { - jq -er "$1" "$P6_INPUT_FILE" -} - -p6_assert_fixed_commit() { - [[ "$1" =~ ^[0-9a-f]{40}$ ]] || { p6_die "FIXED_COMMIT_REQUIRED" 66; return $?; } -} - -p6_assert_sha256() { - [[ "$1" =~ ^[0-9a-f]{64}$ ]] || { p6_die "FIXED_SHA256_REQUIRED" 66; return $?; } -} - -p6_validate_input_shape() { - jq -e ' - . as $root - | type == "object" - and (.schema_version == "p6-inputs/v2") - and (.contract_version == "v1alpha1") - and (.contract_bundle_sha256 == "d289dff9bcaa3d28035c5ed2e56b806f4b3b37fdca3159352d22f0c03942e202") - and (.control_commit == "2eb71d7590739df3de8db2f8cf9098154a397f0b") - and (.review_policy_commit == "680ca92661a08254eb396ab809f478bbdba3510e") - and (.repositories | keys | sort) == ["lab_dev","labnow_launcher","labnow_open","labnow_shell"] - and (.repositories.lab_dev | keys | sort) == ["branch","changed_files","delivery_identity","head_commit","path","phase_base_commit","tracked_diff_sha256"] - and (.repositories.lab_dev.delivery_identity == "review_snapshot") - and (.repositories.lab_dev.branch == "dev/che-563-openclaw-product-closure") - and (.repositories.lab_dev.phase_base_commit == "940325578bae9905673965d6dc489130ab4b6a46") - and (.repositories.lab_dev.head_commit == "1b4562899e03eacdee5a86eb55b47d5e12117ee8") - and (.repositories.lab_dev.tracked_diff_sha256 | type == "string" and test("^[0-9a-f]{64}$")) - and (.repositories.lab_dev.changed_files | type == "array" and length > 0 and unique == .) - and (["labnow_open","labnow_shell","labnow_launcher"] | map(. as $name | - (($root.repositories[$name] | keys | sort) == ["commit","delivery_identity","path"] - and $root.repositories[$name].delivery_identity == "commit" - and ($root.repositories[$name].commit | type == "string" and test("^[0-9a-f]{40}$")))) | all) - and (.repositories.labnow_open.commit == "21019e0c24dc7b51747c2bef3cd90f5d259be839") - and (.repositories.labnow_shell.commit == "5c9411dfd3c4d7b1e606c0d9dc0c5e62313bc376") - and (.repositories.labnow_launcher.commit == "c84edea3e051d561f28d9f99235563cf491aaeb2") - and (.repositories | all(.[]; (.path | type == "string" and startswith("/")))) - and (.images | keys | sort) == ["litellm","openclaw_base","openclaw_workspace"] - and (.local_only_images | keys | sort) == ["launcher","shell"] - and (.support_images | keys | sort) == ["nginx","postgres","redis"] - and (.runtime | keys | sort) == ["p1_env_file"] - and (.runtime.p1_env_file | type == "string" and startswith("/")) - ' "$P6_INPUT_FILE" >/dev/null || { p6_die "INPUT_SCHEMA_INVALID" 68; return $?; } - - local repo - for repo in labnow_open labnow_shell labnow_launcher; do - p6_assert_fixed_commit "$(p6_json_string ".repositories.${repo}.commit")" || return $? - done - p6_assert_fixed_commit "$(p6_json_string '.repositories.lab_dev.phase_base_commit')" || return $? - p6_assert_fixed_commit "$(p6_json_string '.repositories.lab_dev.head_commit')" || return $? - p6_assert_sha256 "$(p6_json_string '.repositories.lab_dev.tracked_diff_sha256')" || return $? - p6_assert_image_shape litellm images repo_digest 'quay.io/labnow/litellm' || return $? - p6_assert_image_shape openclaw_base images repo_digest 'quay.io/labnow/openclaw' || return $? - p6_assert_image_shape openclaw_workspace images local_build 'quay.io/labnow/labnow-open' || return $? - p6_assert_image_shape launcher local_only_images local_build 'quay.io/labnow/labnow-launcher' || return $? - p6_assert_image_shape shell local_only_images local_build 'quay.io/labnow/labnow-shell' || return $? - p6_assert_image_shape postgres support_images repo_digest 'postgres' || return $? - p6_assert_image_shape redis support_images repo_digest 'redis' || return $? - p6_assert_image_shape nginx support_images repo_digest 'nginx' || return $? -} - -p6_assert_image_shape() { - local name="$1" section="$2" provenance="$3" repository="$4" ref image_id digest actual_provenance - ref="$(p6_json_string ".${section}.${name}.ref")" - image_id="$(p6_json_string ".${section}.${name}.image_id")" - digest="$(p6_json_string ".${section}.${name}.repo_digest")" - actual_provenance="$(p6_json_string ".${section}.${name}.provenance")" - [[ "$actual_provenance" == "$provenance" ]] || { p6_die "IMAGE_PROVENANCE_INVALID" 67; return $?; } - [[ "$ref" != *:latest && "$ref" != latest ]] || { p6_die "FIXED_IMAGE_REF_REQUIRED" 67; return $?; } - [[ "$image_id" =~ ^sha256:[0-9a-f]{64}$ ]] || { p6_die "FIXED_IMAGE_ID_REQUIRED" 67; return $?; } - [[ "$digest" == "${repository}@sha256:"* && "$digest" =~ @sha256:[0-9a-f]{64}$ ]] || { p6_die "FIXED_IMAGE_DIGEST_REQUIRED" 67; return $?; } - if [[ "$provenance" == local_build ]]; then - p6_assert_fixed_commit "$(p6_json_string ".${section}.${name}.source_commit")" || return $? - [[ "$(p6_json_string ".${section}.${name}.source_repository")" =~ ^labnow_(open|launcher|shell)$ ]] || { p6_die "LOCAL_PROVENANCE_SOURCE_INVALID" 67; return $?; } - [[ "$(p6_json_string ".${section}.${name}.base_image_digest")" =~ ^[^[:space:]]+@sha256:[0-9a-f]{64}$ ]] || { p6_die "LOCAL_BASE_IMAGE_DIGEST_REQUIRED" 67; return $?; } - fi -} - -p6_assert_repository() { - local name="$1" path identity actual status expected base expected_diff actual_diff expected_files actual_files branch - path="$(p6_json_string ".repositories.${name}.path")" - identity="$(p6_json_string ".repositories.${name}.delivery_identity")" - [[ -d "$path/.git" ]] || { p6_die "REPOSITORY_UNAVAILABLE" 69; return $?; } - actual="$(git -C "$path" rev-parse HEAD)" - if [[ "$identity" == commit ]]; then - expected="$(p6_json_string ".repositories.${name}.commit")" - [[ "$actual" == "$expected" ]] || { p6_die "REPOSITORY_COMMIT_MISMATCH" 70; return $?; } - status="$(git -C "$path" status --porcelain=v1 --untracked-files=no)" - [[ -z "$status" ]] || { p6_die "REPOSITORY_TRACKED_TREE_DIRTY" 71; return $?; } - return 0 - fi - - expected="$(p6_json_string ".repositories.${name}.head_commit")" - base="$(p6_json_string ".repositories.${name}.phase_base_commit")" - branch="$(p6_json_string ".repositories.${name}.branch")" - [[ "$actual" == "$expected" ]] || { p6_die "REVIEW_SNAPSHOT_HEAD_MISMATCH" 70; return $?; } - [[ "$(git -C "$path" branch --show-current)" == "$branch" ]] || { p6_die "REVIEW_SNAPSHOT_BRANCH_MISMATCH" 70; return $?; } - git -C "$path" merge-base --is-ancestor "$base" HEAD || { p6_die "REVIEW_SNAPSHOT_BASE_NOT_ANCESTOR" 70; return $?; } - actual_diff="$(git -C "$path" diff --binary --full-index --no-ext-diff "$base" -- | shasum -a 256 | awk '{print $1}')" - expected_diff="$(p6_json_string ".repositories.${name}.tracked_diff_sha256")" - [[ "$actual_diff" == "$expected_diff" ]] || { p6_die "REVIEW_SNAPSHOT_DIFF_MISMATCH" 71; return $?; } - actual_files="$(git -C "$path" diff --name-only "$base" -- | LC_ALL=C sort | jq -Rsc 'split("\n") | map(select(length > 0))')" - expected_files="$(jq -c ".repositories.${name}.changed_files | sort" "$P6_INPUT_FILE")" - [[ "$actual_files" == "$expected_files" ]] || { p6_die "REVIEW_SNAPSHOT_FILESET_MISMATCH" 71; return $?; } -} - -p6_assert_image_present() { - local section="$1" name="$2" ref expected_id expected_digest actual_id digests source source_commit repository_commit base_digest - ref="$(p6_json_string ".${section}.${name}.ref")" - expected_id="$(p6_json_string ".${section}.${name}.image_id")" - expected_digest="$(p6_json_string ".${section}.${name}.repo_digest")" - actual_id="$(docker image inspect --format '{{.Id}}' "$ref" 2>/dev/null)" || { p6_die "LOCAL_IMAGE_UNAVAILABLE" 72; return $?; } - [[ "$actual_id" == "$expected_id" ]] || { p6_die "LOCAL_IMAGE_ID_MISMATCH" 72; return $?; } - digests="$(docker image inspect --format '{{join .RepoDigests "\n"}}' "$ref")" - grep -Fqx "$expected_digest" <<<"$digests" || { p6_die "LOCAL_IMAGE_DIGEST_MISMATCH" 72; return $?; } - if [[ "$(p6_json_string ".${section}.${name}.provenance")" == local_build ]]; then - source="$(p6_json_string ".${section}.${name}.source_repository")" - source_commit="$(p6_json_string ".${section}.${name}.source_commit")" - if [[ "$source" == lab_dev ]]; then - repository_commit="$(p6_json_string '.repositories.lab_dev.head_commit')" - else - repository_commit="$(p6_json_string ".repositories.${source}.commit")" - fi - [[ "$source_commit" == "$repository_commit" ]] || { p6_die "LOCAL_PROVENANCE_SOURCE_MISMATCH" 72; return $?; } - base_digest="$(p6_json_string ".${section}.${name}.base_image_digest")" - case "$name" in - openclaw_workspace) [[ "$base_digest" == "$(p6_json_string '.images.openclaw_base.repo_digest')" ]] || { p6_die "LOCAL_BASE_IMAGE_MISMATCH" 72; return $?; } ;; - esac - fi -} - -p6_assert_images_present() { - local pair section name - for pair in \ - images:litellm images:openclaw_base images:openclaw_workspace \ - local_only_images:launcher local_only_images:shell \ - support_images:postgres support_images:redis support_images:nginx; do - section="${pair%%:*}" - name="${pair##*:}" - p6_assert_image_present "$section" "$name" || return $? - done -} - -p6_assert_runtime_input() { - local env_file - env_file="$(p6_json_string '.runtime.p1_env_file')" - p6_require_regular_0600 "$env_file" || return $? - python3 - "$env_file" <<'PY' -import sys -from pathlib import Path - -required = { - "LITELLM_MASTER_KEY", "POSTGRES_DB", "POSTGRES_USER", "POSTGRES_PASSWORD", - "REDIS_PASSWORD", "UPSTREAM_API_KEY", "UPSTREAM_BASE_URL", "UPSTREAM_MODEL", -} -values = {} -for raw in Path(sys.argv[1]).read_text(encoding="utf-8").splitlines(): - line = raw.strip() - if line and not line.startswith("#") and "=" in line: - key, value = line.split("=", 1) - values[key] = value -if any(not values.get(key) for key in required): - raise SystemExit(1) -PY - [[ $? == 0 ]] || { p6_die "P1_ENV_INCOMPLETE" 73; return $?; } -} - -p6_security_scan() { - local patterns="$1" status - shift - p6_require_regular_0600 "$patterns" || return $? - [[ -s "$patterns" ]] || { p6_die "SECRET_PATTERN_FILE_REQUIRED" 74; return $?; } - (($# > 0)) || { p6_die "SECRET_SCAN_ROOT_REQUIRED" 74; return $?; } - set +e - rg --fixed-strings --files-with-matches --glob '!secret-patterns' -f "$patterns" "$@" >/dev/null 2>&1 - status=$? - set -e - case "$status" in - 0) p6_die "SECRET_FINGERPRINT_MATCH" 75; return $? ;; - 1) return 0 ;; - *) p6_die "SECRET_SCAN_FAILED" 75; return $? ;; - esac -} diff --git a/docker_openclaw/p6/scripts/p6-prepare-runtime.py b/docker_openclaw/p6/scripts/p6-prepare-runtime.py deleted file mode 100755 index e667c21..0000000 --- a/docker_openclaw/p6/scripts/p6-prepare-runtime.py +++ /dev/null @@ -1,424 +0,0 @@ -#!/usr/bin/env python3 -"""Prepare one restricted, run-scoped P6 topology without printing secrets.""" - -from __future__ import annotations - -import base64 -import json -import os -import secrets -import stat -import subprocess -import sys -from pathlib import Path -from urllib.parse import quote, urlsplit - - -class PrepareError(RuntimeError): - pass - - -def restricted(path: Path, *, code: str) -> None: - try: - info = path.stat() - except OSError as exc: - raise PrepareError(code) from exc - if not stat.S_ISREG(info.st_mode) or info.st_mode & 0o077: - raise PrepareError(code) - - -def write_private(path: Path, value: str | bytes, *, mode: int = 0o400) -> None: - path.parent.mkdir(mode=0o700, parents=True, exist_ok=True) - temporary = path.with_name(f".{path.name}.{secrets.token_hex(8)}") - flags = os.O_WRONLY | os.O_CREAT | os.O_EXCL - descriptor = os.open(temporary, flags, 0o600) - try: - payload = value.encode("utf-8") if isinstance(value, str) else value - with os.fdopen(descriptor, "wb") as handle: - handle.write(payload) - handle.flush() - os.fsync(handle.fileno()) - os.chmod(temporary, mode) - os.replace(temporary, path) - except Exception: - try: - os.unlink(temporary) - except OSError: - pass - raise - - -def load_json(path: Path, *, code: str) -> dict: - restricted(path, code=code) - try: - value = json.loads(path.read_text(encoding="utf-8")) - except (OSError, UnicodeDecodeError, json.JSONDecodeError) as exc: - raise PrepareError(code) from exc - if not isinstance(value, dict): - raise PrepareError(code) - return value - - -def load_env(path: Path) -> dict[str, str]: - restricted(path, code="P1_ENV_INVALID") - values: dict[str, str] = {} - try: - lines = path.read_text(encoding="utf-8").splitlines() - except (OSError, UnicodeDecodeError) as exc: - raise PrepareError("P1_ENV_INVALID") from exc - for raw in lines: - line = raw.strip() - if line and not line.startswith("#") and "=" in line: - key, value = line.split("=", 1) - values[key] = value - required = { - "UPSTREAM_API_KEY", - "UPSTREAM_BASE_URL", - "UPSTREAM_MODEL", - } - if any(not values.get(key) for key in required): - raise PrepareError("P1_ENV_INCOMPLETE") - upstream = urlsplit(values["UPSTREAM_BASE_URL"]) - if upstream.scheme != "https" or not upstream.netloc or upstream.path not in {"", "/"}: - raise PrepareError("P1_UPSTREAM_URL_INVALID") - return values - - -def token(prefix: str) -> str: - return f"{prefix}{secrets.token_urlsafe(32)}" - - -def main() -> int: - input_path = Path(os.environ.get("P6_INPUT_FILE", "")) - work_dir = Path(os.environ.get("P6_WORK_DIR", "")) - run_id = os.environ.get("P6_RUN_ID", "") - artifact_dir = Path(os.environ.get("P6_ARTIFACTS_DIR", "")) - script_dir = Path(__file__).resolve().parent - p6_dir = script_dir.parent - if not input_path.is_absolute() or not work_dir.is_absolute() or not artifact_dir.is_absolute(): - raise PrepareError("P6_PREPARE_PATH_INVALID") - if not run_id.startswith("p6-") or len(run_id) != 35: - raise PrepareError("P6_PREPARE_RUN_ID_INVALID") - - inputs = load_json(input_path, code="P6_INPUT_INVALID") - repositories = inputs.get("repositories", {}) - try: - lab_dev = Path(repositories["lab_dev"]["path"]) - launcher_repo = Path(repositories["labnow_launcher"]["path"]) - shell_repo = Path(repositories["labnow_shell"]["path"]) - p1_env = Path(inputs["runtime"]["p1_env_file"]) - except (KeyError, TypeError) as exc: - raise PrepareError("P6_INPUT_INVALID") from exc - values = load_env(p1_env) - - for directory in (work_dir, artifact_dir): - directory.mkdir(mode=0o700, parents=True, exist_ok=True) - os.chmod(directory, 0o700) - secrets_dir = work_dir / "secrets" - config_dir = work_dir / "config" - workspace_root = work_dir / "workspace" - surfaces_dir = work_dir / "surfaces" - for directory in (secrets_dir, config_dir, workspace_root, surfaces_dir): - directory.mkdir(mode=0o700, parents=True, exist_ok=True) - os.chmod(directory, 0o700) - - short = run_id.removeprefix("p6-")[:12] - network = f"p6net-{short}" - project = f"p6-runtime-{short}" - launcher_container = f"p6-launcher-{short}" - shell_container = f"p6-shell-{short}" - litellm_container = f"p6-litellm-{short}" - shell_postgres_container = f"p6-shell-pg-{short}" - litellm_postgres_container = f"p6-litellm-pg-{short}" - user = f"p6user-{short[:8]}" - server = f"p6ws-{short[:8]}" - prefix = f"p6w-{short[:8]}" - workspace_container = f"{prefix}-{user}-{server}" - - generated = { - "litellm_master": token("sk-p6-master-"), - "litellm_postgres": token("p6-litellm-db-"), - "redis": token("p6-redis-"), - "shell_postgres": token("p6-shell-db-"), - "hub": token("p6-hub-"), - "launcher": token("p6-launcher-"), - "kek": base64.b64encode(secrets.token_bytes(32)).decode("ascii"), - "oauth_cookie": base64.b64encode(secrets.token_bytes(32)).decode("ascii"), - "upstream": values["UPSTREAM_API_KEY"], - } - secret_files: list[str] = [] - for name, secret_value in generated.items(): - target = secrets_dir / name - write_private(target, secret_value + "\n") - secret_files.append(str(target)) - - upstream_url = values["UPSTREAM_BASE_URL"].rstrip("/") - upstream_origin_parts = urlsplit(upstream_url) - upstream_origin = f"{upstream_origin_parts.scheme}://{upstream_origin_parts.netloc}" - upstream_model = values["UPSTREAM_MODEL"] - litellm_database = ( - "postgresql://p6_litellm:" - + quote(generated["litellm_postgres"], safe="") - + "@litellm-postgres:5432/p6_litellm" - ) - shell_database = ( - "postgresql://p6_shell:" - + quote(generated["shell_postgres"], safe="") - + "@shell-postgres:5432/p6_shell" - ) - - write_private( - secrets_dir / "litellm.env", - "\n".join( - [ - f"LITELLM_MASTER_KEY={generated['litellm_master']}", - f"DATABASE_URL={litellm_database}", - "REDIS_HOST=litellm-redis", - "REDIS_PORT=6379", - "REDIS_PASSWORD_FILE=/run/secrets/litellm_redis_password", - "STORE_PROMPTS_IN_SPEND_LOGS=false", - "STORE_MODEL_IN_DB=True", - "LITELLM_LOG=INFO", - "LITELLM_HOST=0.0.0.0", - "LITELLM_PORT=4000", - ] - ) - + "\n", - mode=0o600, - ) - write_private( - secrets_dir / "shell.env", - "\n".join( - [ - "APP_NAME=console", - "HOSTNAME=0.0.0.0", - "PORT=3002", - f"MODEL_ACCESS_DATABASE_URL={shell_database}", - "USER_CENTER_INTERNAL_ORIGIN=http://user-center:8080", - "NEXT_PUBLIC_USER_CENTER_ORIGIN=http://user-center:8080", - "NEXT_PUBLIC_PORTAL_ORIGIN=http://shell:3002", - "JUPYTERHUB_INTERNAL_ORIGIN=http://launcher:8000", - f"JUPYTERHUB_API_TOKEN={generated['hub']}", - "MODEL_ACCESS_TEST_ONLY_ALLOW_HTTP_LITELLM=true", - "LITELLM_MANAGEMENT_URL=http://litellm:4000", - f"LITELLM_MASTER_KEY={generated['litellm_master']}", - f"MODEL_ACCESS_TEST_ONLY_UPSTREAM_ORIGINS={upstream_origin}", - "MODEL_ACCESS_TEST_ONLY_ALLOW_PRIVATE_ENDPOINTS=true", - f"MODEL_ACCESS_TEST_ENDPOINT_HOSTS={upstream_origin_parts.hostname}", - f"MODEL_ACCESS_GENERAL_KEY_MODELS={upstream_model}", - "MODEL_ACCESS_GENERAL_KEY_RPM=30", - "MODEL_ACCESS_GENERAL_KEY_TPM=100000", - "MODEL_ACCESS_GENERAL_KEY_MAX_BUDGET=1", - f"MODEL_ACCESS_KEK_BASE64={generated['kek']}", - f"MODEL_ACCESS_LAUNCHER_SERVICE_TOKEN={generated['launcher']}", - "MODEL_ACCESS_RUNTIME_TTL_SECONDS=900", - "MODEL_ACCESS_RUNTIME_RPM=30", - "MODEL_ACCESS_RUNTIME_TPM=100000", - "MODEL_ACCESS_RUNTIME_MAX_BUDGET=1", - "MODEL_ACCESS_DATA_PLANE_URL=https://litellm-gateway:4443", - ] - ) - + "\n", - mode=0o600, - ) - write_private( - secrets_dir / "launcher.env", - "\n".join( - [ - "PROFILE_LAUNCHER=docker", - "PORT=8000", - "BASE_URL=/studio", - "HUB_CONNECT_IP=launcher", - f"NAME_HUB_CONTAINER={launcher_container}", - f"DIR_USR_WORKSPACE={workspace_root}", - "MODEL_ACCESS_CONFIG_FILE=/run/p6/model-access.json", - f"JUPYTERHUB_API_TOKEN={generated['hub']}", - f"OAUTH2_PROXY_COOKIE_SECRET={generated['oauth_cookie']}", - "LOG_LEVEL=INFO", - ] - ) - + "\n", - mode=0o600, - ) - write_private( - secrets_dir / "model-access.json", - json.dumps( - { - "endpoint": "http://shell:3002/console/api", - "service_token": generated["launcher"], - }, - separators=(",", ":"), - ) - + "\n", - ) - - ca_key = secrets_dir / "p6-ca.key" - ca_cert = config_dir / "p6-ca.pem" - subprocess.run( - [ - "openssl", - "req", - "-x509", - "-newkey", - "rsa:2048", - "-nodes", - "-keyout", - str(ca_key), - "-out", - str(ca_cert), - "-subj", - "/CN=litellm-gateway", - "-addext", - "subjectAltName=DNS:litellm-gateway,IP:127.0.0.1", - "-days", - "1", - ], - check=True, - stdin=subprocess.DEVNULL, - stdout=subprocess.DEVNULL, - stderr=subprocess.DEVNULL, - ) - os.chmod(ca_key, 0o400) - os.chmod(ca_cert, 0o444) - - write_private( - config_dir / "nginx.conf", - """events {} -http { - access_log /dev/stdout; - error_log /dev/stderr warn; - server { - listen 4443 ssl; - ssl_certificate /run/p6/p6-ca.pem; - ssl_certificate_key /run/p6/p6-ca.key; - location / { - proxy_pass http://litellm:4000; - proxy_http_version 1.1; - proxy_buffering off; - proxy_request_buffering off; - proxy_set_header Host $host; - proxy_set_header Authorization $http_authorization; - } - } -} -""", - mode=0o444, - ) - - source_app_conf = launcher_repo / "src/labnow-launcher/resource/config/app.conf" - app_conf = source_app_conf.read_text(encoding="utf-8") - workspace_image = inputs["images"]["openclaw_workspace"]["ref"] - if not workspace_image.startswith("quay.io/"): - raise PrepareError("P6_WORKSPACE_IMAGE_INVALID") - workspace_image_name = workspace_image.removeprefix("quay.io/") - app_conf += ( - "\n# P6 run-scoped overrides.\n" - f"service.port = 8000\n" - f"launcher.dir_usr_workspace = {json.dumps(str(workspace_root))}\n" - "launcher.workspace_registry = \"quay.io\"\n" - f"launcher.workspace_images = [{json.dumps(workspace_image_name)}]\n" - f"model_access.trusted_config_file = {json.dumps('/run/p6/model-access.json')}\n" - f"docker_spawner.network_name = {json.dumps(network)}\n" - f"docker_spawner.prefix = {json.dumps(prefix)}\n" - "docker_spawner.post_start_cmd = \"\"\n" - "docker_spawner.environment = {\n" - " PROFILE_LOCALIZE = \"default\"\n" - " NODE_EXTRA_CA_CERTS = \"/run/labnow/p6-ca.pem\"\n" - " SSL_CERT_FILE = \"/run/labnow/p6-ca.pem\"\n" - "}\n" - "docker_spawner.read_only_volumes = {\n" - f" {json.dumps(str(ca_cert))} = \"/run/labnow/p6-ca.pem\"\n" - "}\n" - ) - write_private(config_dir / "app.conf", app_conf, mode=0o444) - - compose_file = p6_dir / "docker-compose.runtime.yml" - runtime_env = work_dir / "runtime.env" - write_private( - runtime_env, - "\n".join( - [ - f"P6_RUNTIME_NETWORK={network}", - f"P6_LAUNCHER_CONTAINER={launcher_container}", - f"P6_SHELL_CONTAINER={shell_container}", - f"P6_LITELLM_CONTAINER={litellm_container}", - f"P6_SHELL_POSTGRES_CONTAINER={shell_postgres_container}", - f"P6_LITELLM_POSTGRES_CONTAINER={litellm_postgres_container}", - f"P6_WORKSPACE_ROOT={workspace_root}", - f"P6_LAUNCHER_DATA_DIR={work_dir / 'launcher-data'}", - f"P6_LAUNCHER_APP_CONF={config_dir / 'app.conf'}", - f"P6_LAUNCHER_ENV_FILE={secrets_dir / 'launcher.env'}", - f"P6_SHELL_ENV_FILE={secrets_dir / 'shell.env'}", - f"P6_LITELLM_ENV_FILE={secrets_dir / 'litellm.env'}", - f"P6_MODEL_ACCESS_CONFIG={secrets_dir / 'model-access.json'}", - f"P6_CA_CERT={ca_cert}", - f"P6_CA_KEY={ca_key}", - f"P6_NGINX_CONFIG={config_dir / 'nginx.conf'}", - f"P6_USER_CENTER_SCRIPT={script_dir / 'p6-user-center.py'}", - f"P6_SHELL_MIGRATION={shell_repo / 'web/apps/console/src/lib/model-access/migrations/001_initial.sql'}", - f"P6_LITELLM_CONFIG={lab_dev / 'docker_litellm/demo/config.yaml'}", - f"P6_LITELLM_MIGRATE_CONFIG={lab_dev / 'docker_litellm/demo/config.migrate.yaml'}", - f"P6_LITELLM_START_SCRIPT={lab_dev / 'docker_litellm/work/start-litellm.sh'}", - f"P6_LITELLM_MIGRATION_SCRIPT={lab_dev / 'docker_litellm/work/run-migration-locked.py'}", - f"P6_LITELLM_IMAGE={inputs['images']['litellm']['ref']}", - f"P6_WORKSPACE_IMAGE={inputs['images']['openclaw_workspace']['ref']}", - f"P6_LAUNCHER_IMAGE={inputs['local_only_images']['launcher']['ref']}", - f"P6_SHELL_IMAGE={inputs['local_only_images']['shell']['ref']}", - f"P6_POSTGRES_IMAGE={inputs['support_images']['postgres']['ref']}", - f"P6_REDIS_IMAGE={inputs['support_images']['redis']['ref']}", - f"P6_NGINX_IMAGE={inputs['support_images']['nginx']['ref']}", - f"P6_LITELLM_POSTGRES_PASSWORD_FILE={secrets_dir / 'litellm_postgres'}", - f"P6_REDIS_PASSWORD_FILE={secrets_dir / 'redis'}", - f"P6_SHELL_POSTGRES_PASSWORD_FILE={secrets_dir / 'shell_postgres'}", - f"P6_OWNER_A={user}", - f"P6_OWNER_B=p6other-{short[:8]}", - ] - ) - + "\n", - mode=0o600, - ) - (work_dir / "launcher-data").mkdir(mode=0o700, exist_ok=True) - - driver_config = { - "schema_version": "p6-product-chain-config/v1", - "run_id": run_id, - "project": project, - "compose_file": str(compose_file), - "runtime_env_file": str(runtime_env), - "work_dir": str(work_dir), - "surface_dir": str(surfaces_dir), - "workspace_root": str(workspace_root), - "runtime_root": str(workspace_root / ".runtime/model-access"), - "owner_a": user, - "owner_b": f"p6other-{short[:8]}", - "server_name": server, - "workspace_container": workspace_container, - "launcher_container": launcher_container, - "shell_container": shell_container, - "litellm_container": litellm_container, - "shell_postgres_container": shell_postgres_container, - "workspace_image": inputs["images"]["openclaw_workspace"]["ref"], - "hub_token_file": str(secrets_dir / "hub"), - "launcher_token_file": str(secrets_dir / "launcher"), - "upstream_key_file": str(secrets_dir / "upstream"), - "upstream_origin": upstream_origin, - "upstream_model": upstream_model, - "ca_file": str(ca_cert), - "secret_files": secret_files, - "shell_ui_evidence": { - "repository": "labnow_shell", - "commit": inputs["repositories"]["labnow_shell"]["commit"], - "status": "reused_verified_evidence", - }, - } - write_private(config_dir / "driver-config.json", json.dumps(driver_config, separators=(",", ":")) + "\n") - return 0 - - -if __name__ == "__main__": - try: - sys.exit(main()) - except (PrepareError, OSError, KeyError, TypeError, ValueError, subprocess.SubprocessError) as exc: - code = str(exc) if isinstance(exc, PrepareError) else "P6_PREPARE_FAILED" - print(f"P6_ERROR:{code}", file=sys.stderr) - sys.exit(1) diff --git a/docker_openclaw/p6/scripts/p6-product-chain.py b/docker_openclaw/p6/scripts/p6-product-chain.py deleted file mode 100755 index b04c229..0000000 --- a/docker_openclaw/p6/scripts/p6-product-chain.py +++ /dev/null @@ -1,960 +0,0 @@ -#!/usr/bin/env python3 -"""Execute the real P6 Shell -> JupyterHub -> Workspace -> LiteLLM chain. - -The driver keeps credentials and model responses in memory only. Durable output -contains structural assertions, non-sensitive IDs, counts and hashes. -""" - -from __future__ import annotations - -import json -import os -import ssl -import stat -import subprocess -import sys -import time -import urllib.error -import urllib.parse -import urllib.request -from datetime import datetime, timedelta, timezone -from pathlib import Path -from typing import Any - - -MANIFEST = "/run/labnow/model-access/manifest.json" -SECRET = "/run/labnow/model-access/secret.json" -STATUS = "/run/labnow/model-access/status.json" -OPENCLAW_DATA = "/opt/openclaw/data" -ALLOWED_USAGE_FIELDS = { - "timestamp", - "model", - "total_tokens", - "prompt_tokens", - "completion_tokens", - "status", -} - - -class DriverError(RuntimeError): - pass - - -def fail(code: str) -> None: - raise DriverError(code) - - -def restricted(path: Path, *, code: str) -> None: - try: - info = path.stat() - except OSError as exc: - raise DriverError(code) from exc - if not stat.S_ISREG(info.st_mode) or info.st_mode & 0o077: - fail(code) - - -def private_write(path: Path, value: str, mode: int = 0o600) -> None: - path.parent.mkdir(mode=0o700, parents=True, exist_ok=True) - temporary = path.with_name(f".{path.name}.{os.getpid()}") - descriptor = os.open(temporary, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) - try: - with os.fdopen(descriptor, "w", encoding="utf-8") as handle: - handle.write(value) - handle.flush() - os.fsync(handle.fileno()) - os.chmod(temporary, mode) - os.replace(temporary, path) - except Exception: - try: - os.unlink(temporary) - except OSError: - pass - raise - - -def load_config(path: Path) -> dict[str, Any]: - restricted(path, code="PRODUCT_CONFIG_INVALID") - try: - value = json.loads(path.read_text(encoding="utf-8")) - except (OSError, UnicodeDecodeError, json.JSONDecodeError) as exc: - raise DriverError("PRODUCT_CONFIG_INVALID") from exc - required = { - "schema_version", - "run_id", - "project", - "compose_file", - "runtime_env_file", - "work_dir", - "surface_dir", - "workspace_root", - "runtime_root", - "owner_a", - "owner_b", - "server_name", - "workspace_container", - "launcher_container", - "shell_container", - "litellm_container", - "shell_postgres_container", - "workspace_image", - "hub_token_file", - "launcher_token_file", - "upstream_key_file", - "upstream_origin", - "upstream_model", - "ca_file", - "secret_files", - "shell_ui_evidence", - } - if not isinstance(value, dict) or set(value) != required or value.get("schema_version") != "p6-product-chain-config/v1": - fail("PRODUCT_CONFIG_INVALID") - for key in required - {"secret_files", "shell_ui_evidence"}: - if not isinstance(value.get(key), str) or not value[key]: - fail("PRODUCT_CONFIG_INVALID") - if not isinstance(value["secret_files"], list) or not value["secret_files"]: - fail("PRODUCT_CONFIG_INVALID") - return value - - -def read_secret(path: str, code: str) -> str: - target = Path(path) - restricted(target, code=code) - try: - value = target.read_text(encoding="utf-8").strip() - except (OSError, UnicodeDecodeError) as exc: - raise DriverError(code) from exc - if not value or "\n" in value: - fail(code) - return value - - -def command(args: list[str], *, code: str, timeout: int = 120) -> str: - try: - result = subprocess.run( - args, - check=False, - stdin=subprocess.DEVNULL, - stdout=subprocess.PIPE, - stderr=subprocess.PIPE, - timeout=timeout, - ) - except (OSError, subprocess.SubprocessError) as exc: - raise DriverError(code) from exc - if result.returncode != 0: - fail(code) - try: - return result.stdout.decode("utf-8") - except UnicodeDecodeError as exc: - raise DriverError(code) from exc - - -def command_combined(args: list[str], *, code: str, timeout: int = 120) -> str: - try: - result = subprocess.run( - args, - check=False, - stdin=subprocess.DEVNULL, - stdout=subprocess.PIPE, - stderr=subprocess.STDOUT, - timeout=timeout, - ) - except (OSError, subprocess.SubprocessError) as exc: - raise DriverError(code) from exc - if result.returncode != 0: - fail(code) - try: - return result.stdout.decode("utf-8") - except UnicodeDecodeError as exc: - raise DriverError(code) from exc - - -def compose(config: dict[str, Any], *arguments: str, code: str, timeout: int = 120) -> str: - return command( - [ - "docker", - "compose", - "--project-name", - config["project"], - "--env-file", - config["runtime_env_file"], - "-f", - config["compose_file"], - *arguments, - ], - code=code, - timeout=timeout, - ) - - -def published_port(config: dict[str, Any], service: str, port: int) -> int: - value = compose(config, "port", service, str(port), code="PUBLISHED_PORT_UNAVAILABLE").strip() - try: - parsed = int(value.rsplit(":", 1)[1]) - except (IndexError, ValueError) as exc: - raise DriverError("PUBLISHED_PORT_UNAVAILABLE") from exc - if parsed < 1024 or parsed > 65535: - fail("PUBLISHED_PORT_UNAVAILABLE") - return parsed - - -def http_json( - method: str, - url: str, - *, - headers: dict[str, str] | None = None, - body: object | None = None, - context: ssl.SSLContext | None = None, - timeout: int = 30, -) -> tuple[int, Any]: - payload = None if body is None else json.dumps(body, separators=(",", ":")).encode("utf-8") - request = urllib.request.Request( - url, - data=payload, - method=method, - headers={ - "Accept": "application/json", - **({"Content-Type": "application/json"} if payload is not None else {}), - **(headers or {}), - }, - ) - try: - with urllib.request.urlopen(request, timeout=timeout, context=context) as response: - raw = response.read() - decoded = json.loads(raw.decode("utf-8")) if raw else None - return response.status, decoded - except urllib.error.HTTPError as exc: - try: - raw = exc.read() - decoded = json.loads(raw.decode("utf-8")) if raw else None - except (OSError, UnicodeDecodeError, json.JSONDecodeError): - decoded = None - return exc.code, decoded - except (urllib.error.URLError, TimeoutError, OSError, UnicodeDecodeError, json.JSONDecodeError) as exc: - raise DriverError("HTTP_DEPENDENCY_UNAVAILABLE") from exc - - -def require_status(actual: int, expected: set[int], code: str, response: Any = None) -> None: - if actual not in expected: - details = { - "Endpoint probe failed.": "ENDPOINT_PROBE_FAILED", - "LiteLLM rejected the operation.": "LITELLM_REJECTED", - "LiteLLM is unavailable.": "LITELLM_UNAVAILABLE", - "LiteLLM request timed out.": "LITELLM_TIMEOUT", - "Model access is temporarily unavailable.": "MODEL_ACCESS_UNAVAILABLE", - } - marker = details.get(response.get("detail")) if isinstance(response, dict) else None - if marker is None and isinstance(response, dict): - marker = { - "Failed to create JupyterHub user": "JUPYTERHUB_USER_CREATE_FAILED", - "Workspace model binding was not found.": "WORKSPACE_BINDING_NOT_FOUND", - "Internal Server Error": "INTERNAL_SERVER_ERROR", - }.get(response.get("message")) - if marker is None and response is not None: - serialized = json.dumps(response, sort_keys=True) - marker = next( - ( - value - for value in ( - "MODEL_ACCESS_REQUEST_REJECTED", - "MODEL_ACCESS_UNAVAILABLE", - "BINDING_NOT_FOUND", - "UNTRUSTED_MODEL_ACCESS_ADAPTER", - "ADAPTER_APPLY_FAILED", - "WORKSPACE_START_TIMEOUT", - ) - if value in serialized - ), - None, - ) - fail(f"{code}_HTTP_{actual}" + (f"_{marker}" if marker else "")) - - -def shell_headers(owner: str, request_id: str) -> dict[str, str]: - return { - "Cookie": f"p6_owner={owner}", - "X-Request-Id": request_id, - "Idempotency-Key": request_id, - } - - -def hub_headers(token: str) -> dict[str, str]: - return {"Authorization": f"token {token}"} - - -def wait_hub_server(hub: str, token: str, owner: str, server: str, *, running: bool) -> dict[str, Any]: - owner_q = urllib.parse.quote(owner, safe="") - deadline = time.monotonic() + 180 - while time.monotonic() < deadline: - status, body = http_json( - "GET", - f"{hub}/users/{owner_q}?include_stopped_servers=true", - headers=hub_headers(token), - ) - if not running and status == 404: - return {} - servers = body.get("servers") if status == 200 and isinstance(body, dict) else None - snapshot = servers.get(server) if isinstance(servers, dict) else None - if running and isinstance(snapshot, dict) and snapshot.get("ready"): - return snapshot - if not running and snapshot is None and isinstance(servers, dict): - return {} - if not running and isinstance(snapshot, dict) and not snapshot.get("ready") and not snapshot.get("pending"): - return snapshot - time.sleep(1) - fail("HUB_SERVER_START_TIMEOUT" if running else "HUB_SERVER_STOP_TIMEOUT") - - -def material(config: dict[str, Any]) -> tuple[Path, dict[str, Any], str]: - root = Path(config["runtime_root"]) - candidates = list(root.glob("workspace-*/lease-*")) - if len(candidates) != 1: - fail("RUNTIME_MATERIAL_AMBIGUOUS") - directory = candidates[0] - manifest_path = directory / "manifest.json" - secret_path = directory / "secret.json" - restricted(manifest_path, code="RUNTIME_MATERIAL_INVALID") - restricted(secret_path, code="RUNTIME_MATERIAL_INVALID") - try: - manifest_value = json.loads(manifest_path.read_text(encoding="utf-8")) - secret_value = json.loads(secret_path.read_text(encoding="utf-8")) - except (OSError, UnicodeDecodeError, json.JSONDecodeError) as exc: - raise DriverError("RUNTIME_MATERIAL_INVALID") from exc - key = secret_value.get("api_key") if isinstance(secret_value, dict) else None - if ( - not isinstance(manifest_value, dict) - or manifest_value.get("contract_version") != "v1alpha1" - or manifest_value.get("workspace_id") != config["server_name"] - or not isinstance(manifest_value.get("generation"), int) - or not isinstance(key, str) - or not key - ): - fail("RUNTIME_MATERIAL_INVALID") - return directory, manifest_value, key - - -def docker_inspect(container: str) -> dict[str, Any]: - try: - value = json.loads(command(["docker", "inspect", container], code="CONTAINER_INSPECT_FAILED")) - except json.JSONDecodeError as exc: - raise DriverError("CONTAINER_INSPECT_FAILED") from exc - if not isinstance(value, list) or len(value) != 1 or not isinstance(value[0], dict): - fail("CONTAINER_INSPECT_FAILED") - return value[0] - - -def assert_workspace(config: dict[str, Any], runtime_key: str) -> tuple[str, dict[str, Any]]: - inspect = docker_inspect(config["workspace_container"]) - mounts = {item.get("Destination"): item for item in inspect.get("Mounts", [])} - for target in (MANIFEST, SECRET, "/run/labnow/p6-ca.pem"): - if target not in mounts or mounts[target].get("RW") is not False: - fail("WORKSPACE_MOUNT_INVALID") - serialized = json.dumps(inspect, sort_keys=True) - if runtime_key in serialized: - fail("RUNTIME_KEY_LEAKED_TO_INSPECT") - env = inspect.get("Config", {}).get("Env", []) - prefix = next((item.split("=", 1)[1] for item in env if item.startswith("URL_PREFIX=")), "") - if not prefix.startswith("/studio/user/"): - fail("WORKSPACE_URL_PREFIX_INVALID") - status_raw = command(["docker", "exec", config["workspace_container"], "cat", STATUS], code="ADAPTER_STATUS_UNAVAILABLE") - try: - adapter_status = json.loads(status_raw) - except json.JSONDecodeError as exc: - raise DriverError("ADAPTER_STATUS_INVALID") from exc - if adapter_status.get("phase") != "ready" or adapter_status.get("adapter_id") != "openclaw": - fail("ADAPTER_STATUS_INVALID") - deadline = time.monotonic() + 90 - readiness = f"http://127.0.0.1{prefix}api" - while time.monotonic() < deadline: - try: - result = subprocess.run( - ["docker", "exec", config["workspace_container"], "curl", "--fail", "--silent", "--max-time", "3", readiness], - check=False, - stdin=subprocess.DEVNULL, - stdout=subprocess.DEVNULL, - stderr=subprocess.DEVNULL, - timeout=5, - ) - except (OSError, subprocess.SubprocessError): - result = None - if result is not None and result.returncode == 0: - break - time.sleep(1) - else: - fail("OPENCLAW_READINESS_FAILED") - summary = { - "image": inspect.get("Config", {}).get("Image"), - "mounts": sorted(mounts), - "environment_keys": sorted(item.split("=", 1)[0] for item in env), - "state": inspect.get("State", {}).get("Status"), - } - return prefix, summary - - -def data_plane(port: int, ca_file: str, key: str, *, accepted: bool) -> None: - context = ssl.create_default_context(cafile=ca_file) - status, _ = http_json( - "GET", - f"https://127.0.0.1:{port}/models", - headers={"Authorization": f"Bearer {key}"}, - context=context, - ) - if accepted and status == 200: - return - if not accepted and status in {401, 403}: - return - fail("DATA_PLANE_KEY_UNEXPECTED") - - -def wait_rejected(port: int, ca_file: str, key: str) -> None: - deadline = time.monotonic() + 35 - while time.monotonic() < deadline: - try: - data_plane(port, ca_file, key, accepted=False) - return - except DriverError as exc: - if str(exc) != "DATA_PLANE_KEY_UNEXPECTED": - raise - time.sleep(1) - fail("DATA_PLANE_REVOCATION_TIMEOUT") - - -def trajectory(config: dict[str, Any], session: str) -> tuple[dict[str, int | bool], str]: - path = f"{OPENCLAW_DATA}/agents/main/sessions/{session}.trajectory.jsonl" - raw = command(["docker", "exec", config["workspace_container"], "cat", path], code="TRAJECTORY_UNAVAILABLE") - parsed = 0 - errors = 0 - completed = 0 - ended = 0 - for line in raw.splitlines(): - if not line.strip(): - continue - try: - event = json.loads(line) - except json.JSONDecodeError: - errors += 1 - continue - if isinstance(event, dict): - parsed += 1 - completed += int(event.get("type") == "model.completed") - ended += int(event.get("type") == "session.ended") - return { - "parsed_event_count": parsed, - "parse_error_count": errors, - "model_completed_count": completed, - "session_ended_count": ended, - }, raw - - -def run_agent(config: dict[str, Any], model: str, session: str, prompt: str) -> dict[str, int | bool]: - command( - [ - "docker", - "exec", - config["workspace_container"], - "timeout", - "--signal=TERM", - "--kill-after=10s", - "120s", - "openclaw", - "agent", - "--local", - "--session-id", - session, - "--model", - model, - "--message", - prompt, - "--json", - ], - code="OPENCLAW_AGENT_FAILED", - timeout=140, - ) - summary, _ = trajectory(config, session) - if summary["parse_error_count"] or not summary["model_completed_count"] or not summary["session_ended_count"]: - fail("OPENCLAW_CHAT_STRUCTURE_INVALID") - return summary - - -def run_stream(config: dict[str, Any], model: str, session: str) -> dict[str, int | bool]: - raw_path = f"{OPENCLAW_DATA}/{session}.raw.jsonl" - try: - command( - [ - "docker", - "exec", - "-e", - "OPENCLAW_RAW_STREAM=1", - "-e", - f"OPENCLAW_RAW_STREAM_PATH={raw_path}", - config["workspace_container"], - "timeout", - "--signal=TERM", - "--kill-after=10s", - "120s", - "openclaw", - "agent", - "--local", - "--session-id", - session, - "--model", - model, - "--message", - "Return P6_STREAM_OK only.", - "--json", - ], - code="OPENCLAW_STREAM_FAILED", - timeout=140, - ) - deadline = time.monotonic() + 10 - raw = "" - while time.monotonic() < deadline: - try: - raw = command(["docker", "exec", config["workspace_container"], "cat", raw_path], code="STREAM_EVENTS_PENDING") - except DriverError: - time.sleep(1) - continue - if raw.strip(): - break - time.sleep(1) - parsed = 0 - errors = 0 - for line in raw.splitlines(): - if not line.strip(): - continue - try: - value = json.loads(line) - except json.JSONDecodeError: - errors += 1 - else: - parsed += int(isinstance(value, dict)) - if not parsed or errors: - fail("OPENCLAW_STREAM_STRUCTURE_INVALID") - return {"parsed_event_count": parsed, "parse_error_count": errors, "terminated": True} - finally: - subprocess.run( - ["docker", "exec", config["workspace_container"], "rm", "-f", raw_path], - check=False, - stdin=subprocess.DEVNULL, - stdout=subprocess.DEVNULL, - stderr=subprocess.DEVNULL, - ) - - -def run_tool(config: dict[str, Any], model: str, session: str) -> dict[str, int | bool]: - command( - [ - "docker", - "exec", - config["workspace_container"], - "timeout", - "--signal=TERM", - "--kill-after=10s", - "120s", - "openclaw", - "agent", - "--local", - "--session-id", - session, - "--model", - model, - "--message", - "Use exec to run printf P6_TOOL_OK, then reply DONE.", - "--json", - ], - code="OPENCLAW_TOOL_FAILED", - timeout=140, - ) - summary, raw = trajectory(config, session) - if "P6_TOOL_OK" not in raw or summary["parse_error_count"]: - fail("OPENCLAW_TOOL_NOT_OBSERVED") - return {**summary, "tool_observed": True} - - -def psql(config: dict[str, Any], query: str) -> str: - return command( - [ - "docker", - "exec", - config["shell_postgres_container"], - "psql", - "-U", - "p6_shell", - "-d", - "p6_shell", - "-Atc", - query, - ], - code="SHELL_DATABASE_QUERY_FAILED", - ).strip() - - -def usage_time(value: datetime) -> str: - return value.astimezone(timezone.utc).isoformat(timespec="milliseconds").replace("+00:00", "Z") - - -def usage_check(shell: str, config: dict[str, Any], routed_model: str, from_time: str, to_time: str) -> tuple[int, str]: - key_id = psql( - config, - "SELECT id FROM model_access.virtual_keys WHERE owner_id='" - + config["owner_a"].replace("'", "''") - + "' AND key_kind='runtime' ORDER BY created_at DESC LIMIT 1", - ) - if not key_id: - fail("RUNTIME_KEY_ID_MISSING") - query = urllib.parse.urlencode( - { - "workspace_id": config["server_name"], - "key_id": key_id, - "model": routed_model, - "from": from_time, - "to": to_time, - } - ) - deadline = time.monotonic() + 45 - usage: list[Any] = [] - query_succeeded = False - while time.monotonic() < deadline: - status, body = http_json( - "GET", - f"{shell}/model-access/usage/?{query}", - headers=shell_headers("a", f"usage-{config['run_id']}"), - ) - if status in {400, 404}: - fail("USAGE_QUERY_REJECTED") - if status == 200: - if not isinstance(body, dict) or not isinstance(body.get("data"), list): - fail("USAGE_RESPONSE_INVALID") - query_succeeded = True - usage = body["data"] - if usage: - break - time.sleep(2) - if not query_succeeded: - fail("USAGE_QUERY_FAILED") - if not usage: - fail("USAGE_NOT_OBSERVED") - for row in usage: - if not isinstance(row, dict) or set(row) != ALLOWED_USAGE_FIELDS or row.get("model") != routed_model: - fail("USAGE_PROJECTION_INVALID") - status, negative = http_json( - "GET", - f"{shell}/model-access/usage/?{query}", - headers=shell_headers("b", f"usage-negative-{config['run_id']}"), - ) - if status == 200 and isinstance(negative, dict) and negative.get("data") == []: - pass - elif status not in {400, 404}: - fail("USAGE_OWNER_NEGATIVE_FAILED") - return len(usage), key_id - - -def capture_surfaces(config: dict[str, Any], workspace_summary: dict[str, Any], generation: int) -> list[str]: - surface = Path(config["surface_dir"]) - surface.mkdir(mode=0o700, parents=True, exist_ok=True) - logs = compose(config, "logs", "--no-color", code="TOPOLOGY_LOG_CAPTURE_FAILED", timeout=120) - private_write(surface / f"topology-g{generation}.log", logs) - workspace_logs = command_combined(["docker", "logs", config["workspace_container"]], code="WORKSPACE_LOG_CAPTURE_FAILED") - private_write(surface / f"workspace-g{generation}.log", workspace_logs) - processes = command(["docker", "top", config["workspace_container"], "-eo", "pid,args"], code="WORKSPACE_PROCESS_CAPTURE_FAILED") - private_write(surface / f"workspace-process-g{generation}.txt", processes) - openclaw_config = command( - ["docker", "exec", config["workspace_container"], "cat", "/root/.openclaw/data/openclaw.json"], - code="OPENCLAW_CONFIG_CAPTURE_FAILED", - ) - private_write(surface / f"openclaw-config-g{generation}.json", openclaw_config) - adapter_status = command(["docker", "exec", config["workspace_container"], "cat", STATUS], code="ADAPTER_STATUS_UNAVAILABLE") - private_write(surface / f"adapter-status-g{generation}.json", adapter_status) - runtime_surface = surface / f"openclaw-runtime-g{generation}" - command( - ["docker", "cp", f"{config['workspace_container']}:{OPENCLAW_DATA}", str(runtime_surface)], - code="OPENCLAW_RUNTIME_CAPTURE_FAILED", - timeout=120, - ) - if not runtime_surface.is_dir() or runtime_surface.is_symlink(): - fail("OPENCLAW_RUNTIME_CAPTURE_INVALID") - private_write(surface / f"workspace-inspect-g{generation}.json", json.dumps(workspace_summary, sort_keys=True) + "\n") - return [str(surface), str(Path(config["workspace_root"]) / config["owner_a"])] - - -def write_patterns(config: dict[str, Any], path: Path, runtime_keys: list[str]) -> None: - values = [read_secret(item, "SECRET_SOURCE_INVALID") for item in config["secret_files"]] - values.extend(runtime_keys) - if any("\n" in value or not value for value in values): - fail("SECRET_PATTERN_INVALID") - private_write(path, "\n".join(dict.fromkeys(values)) + "\n", mode=0o400) - - -def execute(config: dict[str, Any]) -> dict[str, Any]: - shell_port = published_port(config, "shell", 3002) - hub_port = published_port(config, "launcher", 8000) - gateway_port = published_port(config, "litellm-gateway", 4443) - shell = f"http://127.0.0.1:{shell_port}/console/api" - hub = f"http://127.0.0.1:{hub_port}/studio/hub/api" - hub_token = read_secret(config["hub_token_file"], "HUB_TOKEN_INVALID") - launcher_token = read_secret(config["launcher_token_file"], "LAUNCHER_TOKEN_INVALID") - upstream_key = read_secret(config["upstream_key_file"], "UPSTREAM_KEY_INVALID") - run = config["run_id"].replace("p6-", "")[:12] - - status, connection = http_json( - "POST", - f"{shell}/model-access/connections/", - headers=shell_headers("a", f"connection-{run}"), - body={ - "display_name": f"P6 {run}", - "provider": "openai", - "endpoint": config["upstream_origin"], - "api_key": upstream_key, - }, - ) - require_status(status, {201}, "CONNECTION_CREATE_FAILED", connection) - connection_id = connection.get("data", {}).get("id") if isinstance(connection, dict) else None - if not isinstance(connection_id, str): - fail("CONNECTION_RESPONSE_INVALID") - - status, route = http_json( - "POST", - f"{shell}/model-access/routes/", - headers=shell_headers("a", f"route-{run}"), - body={ - "connection_id": connection_id, - "display_name": f"P6 route {run}", - "upstream_model": config["upstream_model"], - }, - ) - require_status(status, {201}, "ROUTE_CREATE_FAILED") - route_value = route.get("data", {}) if isinstance(route, dict) else {} - route_id = route_value.get("id") - routed_model = route_value.get("routed_model") - if not isinstance(route_id, str) or not isinstance(routed_model, str): - fail("ROUTE_RESPONSE_INVALID") - - status, binding = http_json( - "POST", - f"{shell}/model-access/bindings/", - headers=shell_headers("a", f"binding-{run}"), - body={"workspace_id": config["server_name"], "route_id": route_id, "adapter_id": "openclaw"}, - ) - require_status(status, {201}, "BINDING_CREATE_FAILED") - binding_value = binding.get("data", {}) if isinstance(binding, dict) else {} - binding_id = binding_value.get("binding_id") - if not isinstance(binding_id, str) or set(binding_value) != { - "contract_version", - "binding_id", - "workspace_id", - "route_id", - "adapter_id", - "default_model", - "allowed_models", - }: - fail("BINDING_RESPONSE_INVALID") - - spawn_body = { - "tier": "basic", - "image": config["workspace_image"], - "serverName": config["server_name"], - "model_access": {"contract_version": "v1alpha1", "binding_id": binding_id}, - } - status, spawn_response = http_json( - "POST", - f"{shell}/hub/spawn/", - headers=shell_headers("a", f"spawn-g1-{run}"), - body=spawn_body, - timeout=45, - ) - require_status(status, {201, 202}, "SHELL_SPAWN_FAILED", spawn_response) - server_snapshot = wait_hub_server(hub, hub_token, config["owner_a"], config["server_name"], running=True) - material_dir_1, manifest_1, key_1 = material(config) - _, workspace_summary_1 = assert_workspace(config, key_1) - data_plane(gateway_port, config["ca_file"], key_1, accepted=True) - if key_1 in json.dumps(server_snapshot, sort_keys=True): - fail("RUNTIME_KEY_LEAKED_TO_HUB_API") - generation_1 = manifest_1["generation"] - model_ref = f"labnow/{manifest_1['default_model']}" - from_time = usage_time(datetime.now(timezone.utc) - timedelta(minutes=5)) - chat_summary = run_agent(config, model_ref, f"p6-chat-{run}", "Reply P6_CHAT_OK only.") - stream_summary = run_stream(config, model_ref, f"p6-stream-{run}") - tool_summary = run_tool(config, model_ref, f"p6-tool-{run}") - to_time = usage_time(datetime.now(timezone.utc) + timedelta(minutes=5)) - usage_count, _ = usage_check(shell, config, routed_model, from_time, to_time) - scan_roots = capture_surfaces(config, workspace_summary_1, generation_1) - - status, _ = http_json( - "POST", - f"{shell}/hub/stop/", - headers=shell_headers("a", f"stop-g1-{run}"), - body={"serverName": config["server_name"]}, - timeout=45, - ) - require_status(status, {200, 202, 204}, "SHELL_STOP_FAILED") - wait_hub_server(hub, hub_token, config["owner_a"], config["server_name"], running=False) - if material_dir_1.exists(): - fail("GENERATION_1_MATERIAL_REMAINS") - wait_rejected(gateway_port, config["ca_file"], key_1) - - status, restart_response = http_json( - "POST", - f"{shell}/hub/spawn/", - headers=shell_headers("a", f"spawn-g2-{run}"), - body=spawn_body, - timeout=45, - ) - require_status(status, {201, 202}, "SHELL_RESTART_FAILED", restart_response) - wait_hub_server(hub, hub_token, config["owner_a"], config["server_name"], running=True) - material_dir_2, manifest_2, key_2 = material(config) - _, workspace_summary_2 = assert_workspace(config, key_2) - if manifest_2["generation"] <= generation_1 or key_2 == key_1: - fail("GENERATION_NOT_ADVANCED") - data_plane(gateway_port, config["ca_file"], key_2, accepted=True) - wait_rejected(gateway_port, config["ca_file"], key_1) - restart_chat = run_agent(config, f"labnow/{manifest_2['default_model']}", f"p6-restart-{run}", "Reply P6_RESTART_OK only.") - scan_roots.extend(capture_surfaces(config, workspace_summary_2, manifest_2["generation"])) - - late_body = { - "contract_version": "v1alpha1", - "workspace_id": config["server_name"], - "generation": generation_1, - "reason": "reconciled", - } - status, _ = http_json( - "POST", - f"{shell}/internal/model-access/v1alpha1/runtime-leases/{urllib.parse.quote(manifest_1['lease_id'], safe='')}:release/", - headers={ - "Authorization": f"Bearer {launcher_token}", - "Idempotency-Key": f"late-{run}", - "X-Request-Id": f"late-{run}", - }, - body=late_body, - ) - require_status(status, {409}, "LATE_RELEASE_NOT_REJECTED") - data_plane(gateway_port, config["ca_file"], key_2, accepted=True) - - status, _ = http_json( - "DELETE", - f"{shell}/hub/delete/", - headers=shell_headers("a", f"delete-g2-{run}"), - body={"serverName": config["server_name"], "remove": True}, - timeout=45, - ) - require_status(status, {200, 202, 204}, "SHELL_DELETE_FAILED") - wait_hub_server(hub, hub_token, config["owner_a"], config["server_name"], running=False) - if material_dir_2.exists(): - fail("GENERATION_2_MATERIAL_REMAINS") - wait_rejected(gateway_port, config["ca_file"], key_2) - active = psql( - config, - "SELECT count(*) FROM model_access.runtime_leases WHERE owner_id='" - + config["owner_a"].replace("'", "''") - + "' AND workspace_id='" - + config["server_name"].replace("'", "''") - + "' AND state IN ('issued','active','revoking')", - ) - if active != "0": - fail("ACTIVE_LEASE_REMAINS") - if psql(config, "SELECT coalesce(to_regclass('model_access.usage')::text,'absent')") != "absent": - fail("USAGE_BODY_PERSISTENCE_TABLE_PRESENT") - - pattern_file = Path(os.environ.get("P6_SECRET_PATTERN_FILE", "")) - if not pattern_file.is_absolute(): - fail("SECRET_PATTERN_PATH_INVALID") - write_patterns(config, pattern_file, [key_1, key_2]) - scan_roots = sorted(set(root for root in scan_roots if Path(root).exists())) - if not scan_roots: - fail("SCAN_ROOT_MISSING") - - return { - "schema_version": "p6-product-chain-report/v1", - "result": "passed", - "content_redacted": True, - "checks": { - "test_resource_provision": "passed", - "console_ui": config["shell_ui_evidence"]["status"], - "binding_payload": "passed", - "jupyterhub_dockerspawner": "passed", - "launcher_claim_activate_release": "passed", - "openclaw_apply_probe_readiness": "passed", - "chat": "passed", - "stream": "passed", - "tool": "passed", - "usage": "passed", - "owner_negative": "passed", - "prompt_response_absent": "passed", - "revoke": "passed", - "generation_restart": "passed", - "late_release": "passed", - "delete": "passed", - "zero_active_leases": "passed", - }, - "binding": { - "contract_version": "v1alpha1", - "workspace_id": config["server_name"], - "binding_id": binding_id, - "route_id": route_id, - "payload_fields": ["binding_id", "contract_version"], - }, - "runtime": { - "hub_api": "live", - "docker_daemon": "real", - "workspace_image": config["workspace_image"], - "generation_1": generation_1, - "generation_2": manifest_2["generation"], - "mounts": "readonly", - "adapter_phase": "ready", - }, - "data_plane": { - "chat": chat_summary, - "stream": stream_summary, - "tool": tool_summary, - "restart_chat": restart_chat, - }, - "usage": { - "row_count": usage_count, - "fields": sorted(ALLOWED_USAGE_FIELDS), - "owner_negative": "isolated", - "body_fields_absent": True, - "persistence_table": "absent", - }, - "lifecycle": { - "old_key_after_stop": "rejected", - "new_key_after_restart": "accepted", - "old_key_after_restart": "rejected", - "late_old_release": "rejected_409", - "new_key_after_delete": "rejected", - "active_lease_count": 0, - }, - "scan_roots": scan_roots, - } - - -def main() -> int: - config_path = Path(os.environ.get("P6_PRODUCT_CONFIG_FILE", "")) - report_path = Path(os.environ.get("P6_PRODUCT_REPORT_FILE", "")) - try: - config = load_config(config_path) - report = execute(config) - except DriverError as exc: - code = str(exc) - if report_path.is_absolute(): - private_write( - report_path, - json.dumps( - { - "schema_version": "p6-product-chain-report/v1", - "result": "failed", - "content_redacted": True, - "code": code, - }, - separators=(",", ":"), - ) - + "\n", - ) - print(f"P6_ERROR:{code}", file=sys.stderr) - return 1 - if not report_path.is_absolute(): - print("P6_ERROR:PRODUCT_REPORT_PATH_INVALID", file=sys.stderr) - return 1 - private_write(report_path, json.dumps(report, separators=(",", ":")) + "\n") - return 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/docker_openclaw/p6/scripts/p6-runner.sh b/docker_openclaw/p6/scripts/p6-runner.sh deleted file mode 100755 index 88683b8..0000000 --- a/docker_openclaw/p6/scripts/p6-runner.sh +++ /dev/null @@ -1,168 +0,0 @@ -#!/usr/bin/env bash -# P6 local-only, fail-closed golden-chain coordinator. Product resources are -# not created until review_snapshot, fixed image and restricted input gates pass. -set -euo pipefail - -script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -p6_dir="$(cd "${script_dir}/.." && pwd)" -source "${script_dir}/p6-lib.sh" - -usage() { - cat <<'USAGE' -Usage: p6-runner.sh --input /secure/path/p6-inputs.json [--validate-input|--preflight|--render|--golden|--cleanup] - ---validate-input validates only the protected schema. --preflight also binds -the review_snapshot, three product commits and eight local image ID/digests. ---render retains a credential-free topology template summary. --golden creates -one isolated topology, runs the complete product chain and always removes its -exact containers/network/volumes. --cleanup is idempotent for the same run id. -USAGE -} - -action="" -P6_INPUT_FILE="" -while (($#)); do - case "$1" in - --input) P6_INPUT_FILE="${2:-}"; shift 2 ;; - --validate-input|--preflight|--render|--golden|--cleanup) - [[ -z "$action" ]] || { usage >&2; exit 2; } - action="${1#--}"; shift ;; - *) usage >&2; exit 2 ;; - esac -done -[[ -n "$P6_INPUT_FILE" && -n "$action" ]] || { usage >&2; exit 2; } -[[ -f "$P6_INPUT_FILE" && ! -L "$P6_INPUT_FILE" ]] || p6_die "INPUT_FILE_REQUIRED" 64 -P6_RUN_ID="${P6_RUN_ID:-$(p6_run_id)}" -[[ "$P6_RUN_ID" =~ ^p6-[a-f0-9]{32}$ ]] || p6_die "RUN_ID_INVALID" 64 -export P6_INPUT_FILE P6_RUN_ID -artifact_dir="${P6_ARTIFACTS_DIR:-${p6_dir}/artifacts}" -P6_WORK_DIR="${P6_WORK_DIR:-${p6_dir}/.p6-work/${P6_RUN_ID}}" -export P6_WORK_DIR -mkdir -p "$P6_WORK_DIR" "$artifact_dir" -chmod 700 "$P6_WORK_DIR" "$artifact_dir" -report="${artifact_dir}/p6-${action}-${P6_RUN_ID}.json" - -driver_cleanup_best_effort() { - local driver best_effort_report - driver="${script_dir}/p6-full-driver.sh" - if [[ -x "$driver" && ! -L "$driver" ]]; then - best_effort_report="${artifact_dir}/p6-driver-cleanup-best-effort-${P6_RUN_ID}.json" - P6_DRIVER_ACTION=cleanup \ - P6_DRIVER_REPORT="$best_effort_report" \ - P6_SECRET_PATTERN_FILE="$P6_WORK_DIR/secret-patterns" \ - P6_ARTIFACTS_DIR="$artifact_dir" \ - "$driver" >/dev/null 2>&1 || true - fi -} - -cleanup() { - driver_cleanup_best_effort - rm -rf "$P6_WORK_DIR" -} - -prepare() { - p6_validate_input_shape || return $? - p6_require_regular_0600 "$P6_INPUT_FILE" || return $? -} - -preflight() { - prepare || return $? - local repo - for repo in lab_dev labnow_open labnow_shell labnow_launcher; do - p6_assert_repository "$repo" || return $? - done - p6_assert_images_present || return $? - p6_assert_runtime_input || return $? -} - -render() { - preflight || return $? - local compose_sha metadata - compose_sha="$(p6_sha256 "$p6_dir/docker-compose.runtime.yml")" - metadata="$(jq -c '{schema_version:"p6-render/v1",content_redacted:true,topology:["litellm","shell","jupyterhub","launcher","workspace"],workspace_creation:"live_dockerspawner",support:["postgres","redis","user-center","tls-gateway"],images,local_only_images,support_images}' "$P6_INPUT_FILE")" - jq -n --arg run_id "$P6_RUN_ID" --arg compose_sha256 "$compose_sha" --argjson metadata "$metadata" \ - '{run_id:$run_id,compose_sha256:$compose_sha256} + $metadata' > "$artifact_dir/p6-render-${P6_RUN_ID}.json" - chmod 600 "$artifact_dir/p6-render-${P6_RUN_ID}.json" -} - -run_driver() { - local action="$1" driver driver_report pattern_file - driver="${script_dir}/p6-full-driver.sh" - [[ -x "$driver" && ! -L "$driver" ]] || p6_die "GOLDEN_DRIVER_UNAVAILABLE" 76 - driver_report="${artifact_dir}/p6-driver-${action}-${P6_RUN_ID}.json" - if [[ "$action" == cleanup && -f "$driver_report" ]]; then - driver_report="${artifact_dir}/p6-driver-cleanup-verify-${P6_RUN_ID}.json" - fi - pattern_file="$P6_WORK_DIR/secret-patterns" - rm -f "$driver_report" - P6_DRIVER_ACTION="$action" \ - P6_DRIVER_REPORT="$driver_report" \ - P6_SECRET_PATTERN_FILE="$pattern_file" \ - P6_ARTIFACTS_DIR="$artifact_dir" \ - "$driver" || return $? - chmod 600 "$driver_report" - case "$action" in - provision) - jq -e --arg run "$P6_RUN_ID" --arg input_sha "$(p6_sha256 "$P6_INPUT_FILE")" ' - .schema_version == "p6-driver-provision/v1" and .result == "passed" and .content_redacted == true - and .run_id == $run and .input_sha256 == $input_sha - and (.topology | [.litellm,.shell,.jupyterhub,.launcher] | all(. == "started")) - and .topology.workspace == "deferred_to_golden" - and .isolation.network == "run_scoped" and .isolation.volumes == "run_scoped" - ' "$driver_report" >/dev/null || p6_die "TOPOLOGY_PROVISION_REPORT_INVALID" 77 - ;; - golden) - jq -e --arg patterns "$pattern_file" --arg run "$P6_RUN_ID" --arg input_sha "$(p6_sha256 "$P6_INPUT_FILE")" ' - .schema_version == "p6-driver-report/v1" and .result == "passed" and .content_redacted == true - and .run_id == $run and .input_sha256 == $input_sha - and .secret_pattern_file == $patterns - and .checks.console_ui == "reused_verified_evidence" - and ([.checks.test_resource_provision,.checks.binding_payload,.checks.jupyterhub_dockerspawner,.checks.launcher_claim_activate_release,.checks.openclaw_apply_probe_readiness,.checks.chat,.checks.stream,.checks.tool,.checks.usage,.checks.owner_negative,.checks.prompt_response_absent,.checks.revoke,.checks.generation_restart,.checks.late_release,.checks.delete,.checks.zero_active_leases] | all(. == "passed")) - and (.scan_roots | type == "array" and length >= 1 and all(.[]; type == "string" and startswith("/"))) - and (.product_report_sha256 | test("^[0-9a-f]{64}$")) - ' "$driver_report" >/dev/null || p6_die "GOLDEN_DRIVER_REPORT_INVALID" 77 - ;; - cleanup) - jq -e --arg run "$P6_RUN_ID" --arg input_sha "$(p6_sha256 "$P6_INPUT_FILE")" ' - .schema_version == "p6-driver-cleanup/v1" and .result == "passed" and .content_redacted == true - and .run_id == $run and .input_sha256 == $input_sha - and (.resources | [.litellm,.shell,.jupyterhub,.launcher,.workspace,.runtime_material,.temporary_files,.processes,.network,.volumes] | all(. == "absent")) - ' "$driver_report" >/dev/null || p6_die "TOPOLOGY_CLEANUP_REPORT_INVALID" 77 - ;; - esac -} - -case "$action" in - validate-input) - if prepare; then p6_write_report "$report" passed completed; else p6_write_report "$report" failed precondition_failed "input_validation"; exit 1; fi - ;; - preflight) - if preflight; then p6_write_report "$report" passed completed; else p6_write_report "$report" failed precondition_failed "preflight"; exit 1; fi - ;; - render) - if render; then p6_write_report "$report" passed completed "" "$(jq -n --arg path "$artifact_dir/p6-render-${P6_RUN_ID}.json" --arg sha "$(p6_sha256 "$artifact_dir/p6-render-${P6_RUN_ID}.json")" '{render:{path:$path,sha256:$sha}}')"; else p6_write_report "$report" failed precondition_failed "render"; cleanup; exit 1; fi - ;; - golden) - if ! render; then p6_write_report "$report" failed precondition_failed "render"; cleanup; exit 1; fi - trap cleanup EXIT - if ! run_driver provision; then p6_write_report "$report" failed topology_provision_failed "driver"; exit 1; fi - if ! run_driver golden; then p6_write_report "$report" failed golden_chain_failed "driver"; exit 1; fi - scan_roots=() - while IFS= read -r scan_root; do scan_roots+=("$scan_root"); done < <(jq -r '.scan_roots[]' "${artifact_dir}/p6-driver-golden-${P6_RUN_ID}.json") - scan_roots+=("${artifact_dir}/p6-driver-golden-${P6_RUN_ID}.json") - for scan_root in "${scan_roots[@]}"; do - [[ -e "$scan_root" && ! -L "$scan_root" ]] || { p6_write_report "$report" failed security_scan_failed "scan_root"; exit 1; } - done - if ! p6_security_scan "$P6_WORK_DIR/secret-patterns" "${scan_roots[@]}"; then - p6_write_report "$report" failed security_scan_failed "secret_scan" - exit 1 - fi - if ! run_driver cleanup; then p6_write_report "$report" failed cleanup_failed "driver"; exit 1; fi - p6_write_report "$report" passed completed "" "$(p6_stage_reports_json "$artifact_dir" "$P6_RUN_ID")" - ;; - cleanup) - if ! preflight || ! run_driver cleanup; then p6_write_report "$report" failed cleanup_failed "driver"; cleanup; exit 1; fi - cleanup - p6_write_report "$report" passed completed "" "$(p6_stage_reports_json "$artifact_dir" "$P6_RUN_ID")" - ;; -esac diff --git a/docker_openclaw/p6/scripts/p6-user-center.py b/docker_openclaw/p6/scripts/p6-user-center.py deleted file mode 100755 index 64ddd5f..0000000 --- a/docker_openclaw/p6/scripts/p6-user-center.py +++ /dev/null @@ -1,58 +0,0 @@ -#!/usr/bin/env python3 -"""Run-scoped User Center fixture for P6 cookie identity and entitlement calls.""" - -from __future__ import annotations - -import json -import os -from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer -from http.cookies import SimpleCookie - - -OWNER_A = os.environ.get("P6_OWNER_A", "") -OWNER_B = os.environ.get("P6_OWNER_B", "") - - -class Handler(BaseHTTPRequestHandler): - def log_message(self, _format: str, *_args: object) -> None: - return - - def _owner(self) -> str | None: - cookie = SimpleCookie() - cookie.load(self.headers.get("Cookie", "")) - selector = cookie.get("p6_owner") - if selector and selector.value == "a": - return OWNER_A - if selector and selector.value == "b": - return OWNER_B - return None - - def _json(self, status: int, value: object) -> None: - payload = json.dumps(value, separators=(",", ":")).encode("utf-8") - self.send_response(status) - self.send_header("Content-Type", "application/json") - self.send_header("Content-Length", str(len(payload))) - self.send_header("Cache-Control", "no-store") - self.end_headers() - self.wfile.write(payload) - - def do_GET(self) -> None: # noqa: N802 - if self.path == "/health": - self._json(200, {"status": "ok"}) - return - owner = self._owner() - if not owner: - self._json(401, {"code": "UNAUTHENTICATED"}) - return - if self.path.startswith("/ucenter/api/userInfo"): - self._json(200, {"code": "SUCCESS", "data": {"id": owner, "name": owner, "roles": ["pro"]}}) - return - if self.path.startswith("/ucenter/api/subscription"): - self._json(200, {"code": "SUCCESS", "data": []}) - return - self._json(404, {"code": "NOT_FOUND"}) - - -if not OWNER_A or not OWNER_B: - raise SystemExit("P6 owner configuration is required") -ThreadingHTTPServer(("0.0.0.0", 8080), Handler).serve_forever() diff --git a/docker_openclaw/p6/scripts/test-p6-compose-render.sh b/docker_openclaw/p6/scripts/test-p6-compose-render.sh deleted file mode 100755 index ac39096..0000000 --- a/docker_openclaw/p6/scripts/test-p6-compose-render.sh +++ /dev/null @@ -1,64 +0,0 @@ -#!/usr/bin/env bash -# Render the complete P6 topology with non-sensitive fixtures. This validates -# interpolation only; it does not start a container or claim runtime evidence. -set -euo pipefail - -script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -p6_dir="$(cd "${script_dir}/.." && pwd)" -tmpdir="$(mktemp -d "${TMPDIR:-/tmp}/p6-compose.XXXXXX")" -chmod 700 "$tmpdir" -trap 'rm -rf "$tmpdir"' EXIT -mkdir -p "$tmpdir/workspace" "$tmpdir/launcher-data" -for file in launcher.env shell.env litellm.env model-access.json ca.pem ca.key nginx.conf user-center.py migration.sql config.yaml config.migrate.yaml start-litellm.sh migration.py litellm-db redis shell-db app.conf; do - : > "$tmpdir/$file" -done -chmod 600 "$tmpdir"/*.env "$tmpdir/model-access.json" "$tmpdir/ca.key" "$tmpdir/litellm-db" "$tmpdir/redis" "$tmpdir/shell-db" - -env_file="$tmpdir/runtime.env" -printf '%s\n' \ - 'P6_RUNTIME_NETWORK=p6-render-fixture' \ - 'P6_LAUNCHER_CONTAINER=p6-launcher-fixture' \ - 'P6_SHELL_CONTAINER=p6-shell-fixture' \ - 'P6_LITELLM_CONTAINER=p6-litellm-fixture' \ - 'P6_SHELL_POSTGRES_CONTAINER=p6-shell-pg-fixture' \ - 'P6_LITELLM_POSTGRES_CONTAINER=p6-litellm-pg-fixture' \ - "P6_WORKSPACE_ROOT=$tmpdir/workspace" \ - "P6_LAUNCHER_DATA_DIR=$tmpdir/launcher-data" \ - "P6_LAUNCHER_APP_CONF=$tmpdir/app.conf" \ - "P6_LAUNCHER_ENV_FILE=$tmpdir/launcher.env" \ - "P6_SHELL_ENV_FILE=$tmpdir/shell.env" \ - "P6_LITELLM_ENV_FILE=$tmpdir/litellm.env" \ - "P6_MODEL_ACCESS_CONFIG=$tmpdir/model-access.json" \ - "P6_CA_CERT=$tmpdir/ca.pem" \ - "P6_CA_KEY=$tmpdir/ca.key" \ - "P6_NGINX_CONFIG=$tmpdir/nginx.conf" \ - "P6_USER_CENTER_SCRIPT=$tmpdir/user-center.py" \ - "P6_SHELL_MIGRATION=$tmpdir/migration.sql" \ - "P6_LITELLM_CONFIG=$tmpdir/config.yaml" \ - "P6_LITELLM_MIGRATE_CONFIG=$tmpdir/config.migrate.yaml" \ - "P6_LITELLM_START_SCRIPT=$tmpdir/start-litellm.sh" \ - "P6_LITELLM_MIGRATION_SCRIPT=$tmpdir/migration.py" \ - 'P6_LITELLM_IMAGE=quay.io/labnow/litellm:1.97.0-ead62528e607' \ - 'P6_WORKSPACE_IMAGE=quay.io/labnow/labnow-open:che-563-openclaw-product-closure-local' \ - 'P6_LAUNCHER_IMAGE=quay.io/labnow/labnow-launcher:che-563-openclaw-product-closure-local' \ - 'P6_SHELL_IMAGE=quay.io/labnow/labnow-shell:che-563-openclaw-product-closure-local' \ - 'P6_POSTGRES_IMAGE=postgres:17-alpine@sha256:742f40ea20b9ff2ff31db5458d127452988a2164df9e17441e191f3b72252193' \ - 'P6_REDIS_IMAGE=redis:7.4-alpine@sha256:e7723ff73d963f5cc6d9c4643ea3d989527a402a319239054e9472a7fb9219a2' \ - 'P6_NGINX_IMAGE=nginx:alpine@sha256:2f07d83bf561b506400dc183b1b2003803e39efbd22451f848adaba14d28c7c7' \ - "P6_LITELLM_POSTGRES_PASSWORD_FILE=$tmpdir/litellm-db" \ - "P6_REDIS_PASSWORD_FILE=$tmpdir/redis" \ - "P6_SHELL_POSTGRES_PASSWORD_FILE=$tmpdir/shell-db" \ - 'P6_OWNER_A=p6user-fixture' \ - 'P6_OWNER_B=p6other-fixture' > "$env_file" -chmod 600 "$env_file" - -rendered="$tmpdir/rendered.yml" -docker compose --project-name p6-render-fixture --env-file "$env_file" -f "$p6_dir/docker-compose.runtime.yml" config > "$rendered" -for service in litellm-postgres litellm-redis litellm-migrate litellm litellm-gateway shell-postgres shell-migrate user-center shell launcher; do - rg -q "^ ${service}:" "$rendered" -done -rg -q 'pull_policy: never' "$rendered" -rg -q 'service_completed_successfully' "$rendered" -! rg -q 'openclaw-workspace:' "$rendered" -! rg -q 'OPENCLAW_GATEWAY_TOKEN|:latest' "$rendered" -echo 'PASS P6 Compose rendering: fixed five-component topology and run-scoped support services.' diff --git a/docker_openclaw/p6/scripts/test-p6-driver-flow.sh b/docker_openclaw/p6/scripts/test-p6-driver-flow.sh deleted file mode 100755 index b6fa617..0000000 --- a/docker_openclaw/p6/scripts/test-p6-driver-flow.sh +++ /dev/null @@ -1,248 +0,0 @@ -#!/usr/bin/env bash -# Deterministic orchestration/evidence test. It uses real temporary Git repos -# and a fake Docker/product executor. It never claims a product runtime passed. -set -euo pipefail - -script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -p6_dir="$(cd "${script_dir}/.." && pwd)" -tmpdir="$(mktemp -d "${TMPDIR:-/tmp}/p6-driver-flow.XXXXXX")" -chmod 700 "$tmpdir" -trap 'rm -rf "$tmpdir"' EXIT -cp -R "$p6_dir" "$tmpdir/p6" -test_scripts="$tmpdir/p6/scripts" -mkdir -p "$tmpdir/bin" "$tmpdir/artifacts" - -python3 - "$p6_dir/scripts/p6-product-chain.py" <<'PY' -import importlib.util -import sys - -spec = importlib.util.spec_from_file_location("p6_product_chain", sys.argv[1]) -module = importlib.util.module_from_spec(spec) -assert spec.loader is not None -spec.loader.exec_module(module) - -from datetime import datetime, timezone -assert module.usage_time(datetime(2026, 8, 10, 12, 34, 56, 123456, tzinfo=timezone.utc)) == "2026-08-10T12:34:56.123Z" - -commands = [] -def fake_command(args, **kwargs): - commands.append(args) - return '{"type":"model.completed"}\n{"type":"session.ended"}\n' - -module.command = fake_command -trajectory, _ = module.trajectory({"workspace_container": "fixture-workspace"}, "fixture-session") -assert trajectory["model_completed_count"] == 1 -assert trajectory["session_ended_count"] == 1 -assert commands == [[ - "docker", "exec", "fixture-workspace", - "cat", "/opt/openclaw/data/agents/main/sessions/fixture-session.trajectory.jsonl", -]] - -calls = [] -responses = iter([ - (200, {"servers": {"workspace-1": {"ready": True, "pending": None}}}), - (200, {"servers": {}}), -]) - -def fake_http_json(method, url, **kwargs): - calls.append((method, url, kwargs)) - return next(responses) - -module.http_json = fake_http_json -running = module.wait_hub_server("http://hub.invalid", "fixture-token", "owner a", "workspace-1", running=True) -stopped = module.wait_hub_server("http://hub.invalid", "fixture-token", "owner a", "workspace-1", running=False) -assert running["ready"] is True -assert stopped == {} -assert [call[1] for call in calls] == [ - "http://hub.invalid/users/owner%20a?include_stopped_servers=true", - "http://hub.invalid/users/owner%20a?include_stopped_servers=true", -] - -module.psql = lambda *_args, **_kwargs: "fixture-key-id" -module.shell_headers = lambda *_args, **_kwargs: {} -module.time.sleep = lambda *_args, **_kwargs: None - -ticks = iter([0, 1, 46]) -module.time.monotonic = lambda: next(ticks) -module.http_json = lambda *_args, **_kwargs: (503, {"code": "fixture"}) -try: - module.usage_check("http://shell.invalid", {"owner_a": "owner-a", "server_name": "workspace-1", "run_id": "fixture"}, "model-1", "2026-08-10T00:00:00Z", "2026-08-11T00:00:00Z") -except module.DriverError as exc: - assert str(exc) == "USAGE_QUERY_FAILED" -else: - raise AssertionError("usage_check must distinguish a failed query from an empty successful query") - -ticks = iter([0, 1]) -module.time.monotonic = lambda: next(ticks) -module.http_json = lambda *_args, **_kwargs: (400, {"code": "INVALID_REQUEST"}) -try: - module.usage_check("http://shell.invalid", {"owner_a": "owner-a", "server_name": "workspace-1", "run_id": "fixture"}, "model-1", "2026-08-10T00:00:00.000Z", "2026-08-11T00:00:00.000Z") -except module.DriverError as exc: - assert str(exc) == "USAGE_QUERY_REJECTED" -else: - raise AssertionError("usage_check must fail fast when Shell rejects its filter") - -ticks = iter([0, 1, 46]) -module.time.monotonic = lambda: next(ticks) -module.http_json = lambda *_args, **_kwargs: (200, {"data": []}) -try: - module.usage_check("http://shell.invalid", {"owner_a": "owner-a", "server_name": "workspace-1", "run_id": "fixture"}, "model-1", "2026-08-10T00:00:00Z", "2026-08-11T00:00:00Z") -except module.DriverError as exc: - assert str(exc) == "USAGE_NOT_OBSERVED" -else: - raise AssertionError("usage_check must preserve the successful-but-empty result") -PY - -init_repo() { - local path="$1" - mkdir -p "$path" - git -C "$path" init -q - git -C "$path" config user.name 'P6 Fixture' - git -C "$path" config user.email 'p6-fixture@example.invalid' -} - -lab_dev="$tmpdir/lab-dev" -open_repo="$tmpdir/labnow-open" -shell_repo="$tmpdir/labnow-shell" -launcher_repo="$tmpdir/labnow-launcher" -for repo in "$lab_dev" "$open_repo" "$shell_repo" "$launcher_repo"; do init_repo "$repo"; done - -mkdir -p "$lab_dev/docker_litellm/demo" "$lab_dev/docker_litellm/work" -touch "$lab_dev/docker_litellm/demo/config.yaml" "$lab_dev/docker_litellm/demo/config.migrate.yaml" "$lab_dev/docker_litellm/work/start-litellm.sh" "$lab_dev/docker_litellm/work/run-migration-locked.py" -printf 'base\n' > "$lab_dev/fixture.txt" -git -C "$lab_dev" add . -git -C "$lab_dev" commit -qm 'fixture base' -git -C "$lab_dev" branch -M dev/che-563-openclaw-product-closure -lab_dev_base="$(git -C "$lab_dev" rev-parse HEAD)" -printf 'review snapshot\n' >> "$lab_dev/fixture.txt" -lab_dev_diff="$(git -C "$lab_dev" diff --binary --full-index --no-ext-diff "$lab_dev_base" -- | shasum -a 256 | awk '{print $1}')" - -printf 'open\n' > "$open_repo/fixture.txt" -git -C "$open_repo" add . -git -C "$open_repo" commit -qm 'open fixture' -open_commit="$(git -C "$open_repo" rev-parse HEAD)" - -mkdir -p "$shell_repo/web/apps/console/src/lib/model-access/migrations" -printf 'SELECT 1;\n' > "$shell_repo/web/apps/console/src/lib/model-access/migrations/001_initial.sql" -git -C "$shell_repo" add . -git -C "$shell_repo" commit -qm 'shell fixture' -shell_commit="$(git -C "$shell_repo" rev-parse HEAD)" - -mkdir -p "$launcher_repo/src/labnow-launcher/resource/config" -printf 'service { port = 8000 }\nlauncher { dir_usr_workspace = "/tmp" }\nmodel_access { trusted_config_file = "/tmp/model-access.json" }\ndocker_spawner { network_name = "fixture" prefix = "fixture" environment = {} read_only_volumes = {} }\n' > "$launcher_repo/src/labnow-launcher/resource/config/app.conf" -git -C "$launcher_repo" add . -git -C "$launcher_repo" commit -qm 'launcher fixture' -launcher_commit="$(git -C "$launcher_repo" rev-parse HEAD)" - -# The copied validator keeps production constants. Replace only those constants -# inside the disposable copy so this test can use genuine temporary commits. -perl -pi -e "s/940325578bae9905673965d6dc489130ab4b6a46/$lab_dev_base/g; s/1b4562899e03eacdee5a86eb55b47d5e12117ee8/$lab_dev_base/g; s/21019e0c24dc7b51747c2bef3cd90f5d259be839/$open_commit/g; s/5c9411dfd3c4d7b1e606c0d9dc0c5e62313bc376/$shell_commit/g; s/c84edea3e051d561f28d9f99235563cf491aaeb2/$launcher_commit/g" "$test_scripts/p6-lib.sh" - -p1_env="$tmpdir/p1.env" -printf '%s\n' \ - 'LITELLM_MASTER_KEY=fixture-master-not-valid' \ - 'POSTGRES_DB=fixture' \ - 'POSTGRES_USER=fixture' \ - 'POSTGRES_PASSWORD=fixture-db-not-valid' \ - 'REDIS_PASSWORD=fixture-redis-not-valid' \ - 'UPSTREAM_API_KEY=fixture-upstream-not-valid' \ - 'UPSTREAM_BASE_URL=https://example.invalid' \ - 'UPSTREAM_MODEL=fixture-model' > "$p1_env" -chmod 600 "$p1_env" - -input="$tmpdir/input.json" -jq \ - --arg lab_dev "$lab_dev" --arg base "$lab_dev_base" --arg diff "$lab_dev_diff" \ - --arg open "$open_repo" --arg open_commit "$open_commit" \ - --arg shell "$shell_repo" --arg shell_commit "$shell_commit" \ - --arg launcher "$launcher_repo" --arg launcher_commit "$launcher_commit" \ - --arg p1 "$p1_env" ' - .repositories.lab_dev.path=$lab_dev - | .repositories.lab_dev.phase_base_commit=$base - | .repositories.lab_dev.head_commit=$base - | .repositories.lab_dev.tracked_diff_sha256=$diff - | .repositories.lab_dev.changed_files=["fixture.txt"] - | .repositories.labnow_open.path=$open - | .repositories.labnow_open.commit=$open_commit - | .repositories.labnow_shell.path=$shell - | .repositories.labnow_shell.commit=$shell_commit - | .repositories.labnow_launcher.path=$launcher - | .repositories.labnow_launcher.commit=$launcher_commit - | .images.openclaw_workspace.source_commit=$open_commit - | .local_only_images.shell.source_commit=$shell_commit - | .local_only_images.launcher.source_commit=$launcher_commit - | .runtime.p1_env_file=$p1 - ' "$tmpdir/p6/p6-inputs.example.json" > "$input" -chmod 600 "$input" - -cat > "$tmpdir/bin/docker" <<'EOF' -#!/usr/bin/env bash -set -euo pipefail -if [[ "${1:-}" == image && "${2:-}" == inspect ]]; then - ref="${*: -1}" - if [[ "$*" == *'{{.Id}}'* ]]; then - case "$ref" in - *litellm*) echo sha256:a2e115874c21b829bd052b18fc85be2f9217fb8244b82812c4ebc6e36f9824d1 ;; - *labnow-open:*) echo sha256:c9c6a45637521cbbaeacea57fbb128696066fd91c5dff4521555f1bd5211f244 ;; - *openclaw@*) echo sha256:edc85cc2068f5ec0df470f7d06daa0a4fbd78ef5ad6cf5b48f58381da839dd12 ;; - *launcher*) echo sha256:6f9732fda8b86d9bfe4596e848025cc38448da4b17dfea8520e046a32b32e61f ;; - *shell*) echo sha256:d7ed71cf58eddf72642d61d4442a0820632060fac9f4eb611b28062b7f38c54d ;; - *postgres*) echo sha256:742f40ea20b9ff2ff31db5458d127452988a2164df9e17441e191f3b72252193 ;; - *redis*) echo sha256:e7723ff73d963f5cc6d9c4643ea3d989527a402a319239054e9472a7fb9219a2 ;; - *nginx*) echo sha256:2f07d83bf561b506400dc183b1b2003803e39efbd22451f848adaba14d28c7c7 ;; - *) exit 64 ;; - esac - else - case "$ref" in - *litellm*) echo quay.io/labnow/litellm@sha256:a2e115874c21b829bd052b18fc85be2f9217fb8244b82812c4ebc6e36f9824d1 ;; - *labnow-open:*) echo quay.io/labnow/labnow-open@sha256:c9c6a45637521cbbaeacea57fbb128696066fd91c5dff4521555f1bd5211f244 ;; - *openclaw@*) echo quay.io/labnow/openclaw@sha256:edc85cc2068f5ec0df470f7d06daa0a4fbd78ef5ad6cf5b48f58381da839dd12 ;; - *launcher*) echo quay.io/labnow/labnow-launcher@sha256:6f9732fda8b86d9bfe4596e848025cc38448da4b17dfea8520e046a32b32e61f ;; - *shell*) echo quay.io/labnow/labnow-shell@sha256:d7ed71cf58eddf72642d61d4442a0820632060fac9f4eb611b28062b7f38c54d ;; - *postgres*) echo postgres@sha256:742f40ea20b9ff2ff31db5458d127452988a2164df9e17441e191f3b72252193 ;; - *redis*) echo redis@sha256:e7723ff73d963f5cc6d9c4643ea3d989527a402a319239054e9472a7fb9219a2 ;; - *nginx*) echo nginx@sha256:2f07d83bf561b506400dc183b1b2003803e39efbd22451f848adaba14d28c7c7 ;; - *) exit 64 ;; - esac - fi - exit 0 -fi -if [[ "${1:-}" == compose ]]; then exit 0; fi -if [[ "${1:-}" == container && "${2:-}" == inspect ]]; then exit 1; fi -if [[ "${1:-}" == container && "${2:-}" == ls ]]; then exit 0; fi -if [[ "${1:-}" == volume && "${2:-}" == ls ]]; then exit 0; fi -if [[ "${1:-}" == network && "${2:-}" == inspect ]]; then exit 1; fi -if [[ "${1:-}" == rm ]]; then exit 0; fi -exit 64 -EOF -chmod 700 "$tmpdir/bin/docker" - -cat > "$test_scripts/p6-product-chain.py" <<'EOF' -#!/usr/bin/env python3 -import json, os -from pathlib import Path -surface = Path(os.environ['P6_WORK_DIR']) / 'surfaces' -surface.mkdir(mode=0o700, parents=True, exist_ok=True) -(surface / 'fixture.txt').write_text('redacted fixture surface\n') -os.chmod(surface / 'fixture.txt', 0o600) -Path(os.environ['P6_SECRET_PATTERN_FILE']).write_text('p6-fixture-secret-not-present\n') -os.chmod(os.environ['P6_SECRET_PATTERN_FILE'], 0o400) -checks = {name:'passed' for name in ['test_resource_provision','binding_payload','jupyterhub_dockerspawner','launcher_claim_activate_release','openclaw_apply_probe_readiness','chat','stream','tool','usage','owner_negative','prompt_response_absent','revoke','generation_restart','late_release','delete','zero_active_leases']} -checks['console_ui'] = 'reused_verified_evidence' -report = {'schema_version':'p6-product-chain-report/v1','result':'passed','content_redacted':True,'checks':checks,'binding':{},'runtime':{},'data_plane':{},'usage':{},'lifecycle':{},'scan_roots':[str(surface)]} -Path(os.environ['P6_PRODUCT_REPORT_FILE']).write_text(json.dumps(report, separators=(',',':'))+'\n') -os.chmod(os.environ['P6_PRODUCT_REPORT_FILE'], 0o600) -EOF -chmod 700 "$test_scripts/p6-product-chain.py" - -run_id="p6-$(python3 -c 'print("1" * 32)')" -run_env=(env "PATH=$tmpdir/bin:$PATH" "P6_RUN_ID=$run_id" "P6_ARTIFACTS_DIR=$tmpdir/artifacts" "P6_WORK_DIR=$tmpdir/work") -"${run_env[@]}" "$test_scripts/p6-runner.sh" --input "$input" --preflight >/dev/null -"${run_env[@]}" "$test_scripts/p6-runner.sh" --input "$input" --golden >/dev/null -"${run_env[@]}" "$test_scripts/p6-runner.sh" --input "$input" --cleanup >/dev/null -"$test_scripts/p6-aggregate.sh" --artifacts "$tmpdir/artifacts" --run-id "$run_id" >/dev/null - -for action in provision golden cleanup; do test -s "$tmpdir/artifacts/p6-driver-${action}-${run_id}.json"; done -test -s "$tmpdir/artifacts/p6-final-${run_id}.json" -test ! -d "$tmpdir/work" -echo 'PASS P6 driver flow: review_snapshot, redacted stage evidence, hash binding and cleanup are deterministic.' diff --git a/docker_openclaw/p6/scripts/test-p6-gates.sh b/docker_openclaw/p6/scripts/test-p6-gates.sh index e2b0944..51d09ef 100755 --- a/docker_openclaw/p6/scripts/test-p6-gates.sh +++ b/docker_openclaw/p6/scripts/test-p6-gates.sh @@ -1,62 +1,22 @@ #!/usr/bin/env bash -# Static negative gates for P6 input validation. No Docker, network, product -# repository, credential, or upstream operation is used here. +# Static OpenClaw product contract checks. This test deliberately performs no +# Docker, network, credential, or historical-evidence operation. set -euo pipefail script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -p6_dir="$(cd "${script_dir}/.." && pwd)" -runner="$script_dir/p6-runner.sh" -tmpdir="$(mktemp -d "${TMPDIR:-/tmp}/p6-gates.XXXXXX")" -chmod 700 "$tmpdir" -trap 'rm -rf "$tmpdir"' EXIT +root="$(cd "${script_dir}/../../.." && pwd)" +dockerfile="${root}/docker_openclaw/openclaw.Dockerfile" +compose_file="${root}/docker_openclaw/demo/docker-compose.yml" -input="$tmpdir/input.json" -cp "$p6_dir/p6-inputs.example.json" "$input" -chmod 600 "$input" -run_id="p6-$(python3 -c 'print("0" * 32)')" -P6_RUN_ID="$run_id" P6_ARTIFACTS_DIR="$tmpdir/artifacts" "$runner" --input "$input" --validate-input >/dev/null +rg -q '^ENV OPENCLAW_HOME=/root/\.openclaw$' "$dockerfile" +rg -q '^ENV OPENCLAW_STATE_DIR=\$\{OPENCLAW_HOME\}/data$' "$dockerfile" +rg -q '^VOLUME \["/root/\.openclaw/data", "/opt/node/pnpm/store"\]$' "$dockerfile" +rg -q '^EXPOSE 18789 18790$' "$dockerfile" +rg -q '^CMD \["start-openclaw\.sh"\]$' "$dockerfile" +rg -q '^services:$' "$compose_file" +rg -q '^ openclaw-gateway:$' "$compose_file" +rg -q '^ - \$\{OPENCLAW_GATEWAY_PORT:-18789\}:18789$' "$compose_file" +rg -q '^ - \$\{OPENCLAW_BRIDGE_PORT:-18790\}:18790$' "$compose_file" +! rg -q '/var/run/docker\.sock' "$compose_file" -source "$script_dir/p6-lib.sh" -mkdir -p "$tmpdir/scan-bin" "$tmpdir/scan-root" -printf '#!/usr/bin/env bash\nexit 2\n' > "$tmpdir/scan-bin/rg" -chmod 700 "$tmpdir/scan-bin/rg" -printf 'fixture-pattern-not-present\n' > "$tmpdir/patterns" -chmod 600 "$tmpdir/patterns" -printf 'safe fixture\n' > "$tmpdir/scan-root/value.txt" -if (PATH="$tmpdir/scan-bin:$PATH"; p6_security_scan "$tmpdir/patterns" "$tmpdir/scan-root" >/dev/null 2>&1); then - echo 'accepted failed secret scan as zero-hit' >&2 - exit 1 -fi - -negative() { - local name="$1" filter="$2" candidate - candidate="$tmpdir/${name}.json" - jq "$filter" "$input" > "$candidate" - chmod 600 "$candidate" - if P6_RUN_ID="$run_id" P6_ARTIFACTS_DIR="$tmpdir/artifacts" "$runner" --input "$candidate" --validate-input >/dev/null 2>&1; then - printf 'accepted invalid input: %s\n' "$name" >&2 - exit 1 - fi -} - -negative latest '.images.openclaw_workspace.ref = "quay.io/labnow/labnow-open:latest"' -negative contract_mismatch '.contract_bundle_sha256 = ("0" * 64)' -negative missing_workspace_digest '.images.openclaw_workspace.repo_digest = "absent"' -negative missing_launcher_digest '.local_only_images.launcher.repo_digest = "absent"' -negative unprotected_lab_dev '.repositories.lab_dev.delivery_identity = "commit"' -negative missing_snapshot_files '.repositories.lab_dev.changed_files = []' -negative mutable_support '.support_images.nginx.ref = "nginx:latest"' - -rg -q 'p6-product-chain.py' "$script_dir/p6-full-driver.sh" -rg -q 'review_snapshot' "$script_dir/p6-lib.sh" -rg -q 'tracked_diff_sha256' "$script_dir/p6-lib.sh" -! rg -q 'golden_checks|pattern_command|topology.*compose_file' "$p6_dir/p6-inputs.example.json" -! rg -q 'docker-compose.p6.yml|up -d --wait openclaw-workspace' "$script_dir/p6-runner.sh" "$script_dir/p6-full-driver.sh" -test -x "$script_dir/p6-full-driver.sh" -test -x "$script_dir/p6-prepare-runtime.py" -test -x "$script_dir/p6-product-chain.py" -if "$script_dir/p6-aggregate.sh" --artifacts "$tmpdir/artifacts" --run-id "$run_id" >/dev/null 2>&1; then - echo 'accepted incomplete evidence' >&2 - exit 1 -fi -echo 'PASS P6 gates: review_snapshot, fixed provenance and incomplete evidence fail closed.' +printf '%s\n' 'PASS P6 gates: OpenClaw image state, entrypoint, ports, and standard Compose avoid Docker socket access.' From ca3a2d670780031831934423395311036268f2c6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Thu, 20 Aug 2026 08:03:04 +0800 Subject: [PATCH 76/87] =?UTF-8?q?chore(hermes):=20=E5=BD=92=E6=A1=A3=20P7?= =?UTF-8?q?=20=E4=B8=80=E6=AC=A1=E6=80=A7=E8=AF=81=E6=8D=AE=E7=BC=96?= =?UTF-8?q?=E6=8E=92?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 动机与背景:LLM Hub 生产加固 PH-3 对应 X-07。D-10 要求把 P7 已冻结的固定组合、受限输入、Launcher overlay、黄金 runner 与报告聚合保留为可由 Git 历史读取的历史证据,而不是产品仓长期维护的运行面。 关键设计取舍:删除 p7-runner、预处理器、产品链、固定输入模板、runtime Compose 与只服务于固定组合的 Launcher overlay;不迁移这些一次性文件。保留并重写 test-p7-gates.sh,使其只做无 Docker、无网络、无凭据、无历史 commit/digest/绝对路径的 Hermes 静态产品契约检查。P7 README 与父 README 均改为精确 git show 回读说明。 验证:bash -n docker_openclaw/p6/scripts/test-p6-gates.sh docker_hermes/p7/scripts/test-p7-gates.sh;./docker_openclaw/p6/scripts/test-p6-gates.sh(PASS P6 gates: OpenClaw image state, entrypoint, ports, and standard Compose avoid Docker socket access.);./docker_hermes/p7/scripts/test-p7-gates.sh(PASS P7 gates: Hermes source pin, provenance labels, explicit local image, pull policy, and loopback dashboard publication.);git diff --cached --check;git grep --cached 检查已删入口,除本地 skip-worktree AGENTS.md 的历史操作范例外,产品树零残留。未启动服务容器。 影响面:仅 docker_hermes/p7 的历史证据编排、其说明与保留静态门禁;未改 Hermes Dockerfile、常规 demo Compose 或其他服务。关联 Linear:CHE-673(In Progress)。 剩余风险:P7 跨仓真实黄金运行只能通过基线 fdbbab2155a9e088c37d2a8a2057178e19ac9534 的 Git 历史审阅,不能由本次保留门禁重放;最终交付仍需 HEAD git ls-tree 核验与总控将 CHE-673 置于 In Review。 --- docker_hermes/README.md | 3 +- docker_hermes/p7/README.md | 88 +--- docker_hermes/p7/docker-compose.runtime.yml | 33 -- docker_hermes/p7/launcher-overlay.Dockerfile | 16 - docker_hermes/p7/p7-inputs.example.json | 32 -- .../p7/scripts/p7-prepare-runtime.py | 144 ------ docker_hermes/p7/scripts/p7-product-chain.py | 457 ------------------ docker_hermes/p7/scripts/p7-runner.sh | 273 ----------- docker_hermes/p7/scripts/test-p7-gates.sh | 122 +---- 9 files changed, 29 insertions(+), 1139 deletions(-) delete mode 100644 docker_hermes/p7/docker-compose.runtime.yml delete mode 100644 docker_hermes/p7/launcher-overlay.Dockerfile delete mode 100644 docker_hermes/p7/p7-inputs.example.json delete mode 100755 docker_hermes/p7/scripts/p7-prepare-runtime.py delete mode 100755 docker_hermes/p7/scripts/p7-product-chain.py delete mode 100755 docker_hermes/p7/scripts/p7-runner.sh diff --git a/docker_hermes/README.md b/docker_hermes/README.md index 729ab40..a6304d1 100644 --- a/docker_hermes/README.md +++ b/docker_hermes/README.md @@ -65,7 +65,8 @@ build_image_no_tag hermes p7-<12hex> docker_hermes/hermes.Dockerfile \ 镜像会记录 `org.opencontainers.image.source` 与 `org.opencontainers.image.revision`,并在 `/opt/hermes/.labnow-source-*` 保存 相同的非敏感 provenance。只在本地命名为 `quay.io/labnow/hermes:p7-<12hex>`,不 push。 -完整 P7 的受限输入、静态门禁和跨仓 runner 见 [`p7/README.md`](p7/README.md)。 +P7 的跨仓固定组合与 runner 已按 D-10 归档;历史回读方法见 +[`p7/README.md`](p7/README.md)。它不是 Hermes 的构建、启动或 CI 入口。 ### P8-H10:Dashboard Chat TUI runtime diff --git a/docker_hermes/p7/README.md b/docker_hermes/p7/README.md index e201f48..0816335 100644 --- a/docker_hermes/p7/README.md +++ b/docker_hermes/p7/README.md @@ -1,85 +1,15 @@ -# P7 Hermes 可复现运行基线 +# P7 Hermes 产品链(已归档) -此目录只承担 CHE-568 的 `lab-dev` 职责:固定 Hermes 源码/构建 provenance, -校验本地镜像,并参数化复用 P6 已验证的真实五组件拓扑执行 Hermes 产品链。 -它不复制 `labnow-open` 的 Hermes renderer 或 `labnow-shell` 的 -image→adapter 目录业务逻辑。 +LLM Hub V1 / P7 的固定组合、Launcher overlay、受限输入、黄金 runner 与报告 +聚合均为一次性冻结验收证据,已按生产加固决策 D-10 从产品运行面删除。它们不是 +持续维护的 Hermes 构建、启动或 CI 入口。 -## 固定构建 - -P7 构建必须使用准确的 Hermes repository 与 40 位 commit;Dockerfile 不再 clone -移动 `main`。本机网络可用时,使用仓库标准入口构建本地制品: - -```bash -export REGISTRY_SRC=quay.io -export REGISTRY_DST=quay.io -export CI_PROJECT_NAME=LabNow/lab-dev -source ./tool.sh -build_image_no_tag hermes p7-<12hex> docker_hermes/hermes.Dockerfile \ - --build-arg HERMES_SOURCE_REPOSITORY= \ - --build-arg HERMES_SOURCE_COMMIT=<40-hex-commit> \ - --build-arg HERMES_BUILD_BASE_IMAGE=quay.io/labnow/node@sha256:<64-hex> \ - --build-arg HERMES_RUNTIME_BASE_IMAGE=quay.io/labnow/base@sha256:<64-hex> -``` - -产物必须是 `quay.io/labnow/hermes:p7-<12hex>`;不 push。镜像 OCI revision label -和 `/opt/hermes/.labnow-source-*` 是非敏感 provenance;两个 -`io.labnow.hermes.*-base` label 记录实际传入的不可变基础镜像引用。runner 会校验 -这些 label 与受限输入一致。 - -P7 真实链路发现 P6 Launcher 将 `RuntimeManifest.adapter_id` 固定为 -`openclaw` 后,Launcher 在独立 Phase 分支完成了受信任双 Adapter 修复。全量 -Launcher Dockerfile 会动态下载构建工具,不能作为本轮固定组合的重建入口; -本目录用 `launcher-overlay.Dockerfile` 从已验证的 P6 Launcher 本地 digest -出发,只覆盖固定 Launcher commit 的运行时代码与 Hub 配置: - -```bash -docker build --platform linux/amd64 --provenance=false \ - --build-context launcher=/absolute/path/to/labnow-launcher \ - --build-arg P6_LAUNCHER_IMAGE=quay.io/labnow/labnow-launcher:che-563-openclaw-product-closure-local \ - --build-arg P6_LAUNCHER_BASE_DIGEST=quay.io/labnow/labnow-launcher@sha256:6f9732fda8b86d9bfe4596e848025cc38448da4b17dfea8520e046a32b32e61f \ - --build-arg LAUNCHER_SOURCE_COMMIT=f84a51319d75b99a6b210f19e264904cae07fc8a \ - -t quay.io/labnow/labnow-launcher:che-568-hermes-console-experience-local \ - -f docker_hermes/p7/launcher-overlay.Dockerfile . -``` - -构建前必须回读 P6 tag 的本地 image ID 正是上述 digest,并确认 Launcher -checkout tracked clean 且 HEAD 等于 `LAUNCHER_SOURCE_COMMIT`。overlay 的 OCI -revision 与 `io.labnow.p7.launcher-base` label 会由 runner 回读;产物仍只在本地, -不得把本地 RepoDigest 描述成远端 registry 已发布制品。 - -## 入口与安全 - -从 `p7-inputs.example.json` 创建权限为 `0400` 或 `0600` 的 Git 忽略 -`p7-inputs.json`。输入只包含路径、delivery/runtime commit、镜像 ID/RepoDigest -和 P1 环境文件路径;不得保存 API key、Token、密码、请求正文或响应正文。 +最后可读快照是本批基线 +`fdbbab2155a9e088c37d2a8a2057178e19ac9534`。需要审阅历史材料时,在本仓执行: ```bash -./docker_hermes/p7/scripts/test-p7-gates.sh -./docker_hermes/p7/scripts/p7-runner.sh --input /secure/path/p7-inputs.json --validate-input -./docker_hermes/p7/scripts/p7-runner.sh --input /secure/path/p7-inputs.json --preflight -./docker_hermes/p7/scripts/p7-runner.sh --input /secure/path/p7-inputs.json --render -P7_RUN_ID=p7-<32hex> ./docker_hermes/p7/scripts/p7-runner.sh --input /secure/path/p7-inputs.json --golden -P7_RUN_ID=p7- ./docker_hermes/p7/scripts/p7-runner.sh --input /secure/path/p7-inputs.json --cleanup +git show fdbbab2155a9e088c37d2a8a2057178e19ac9534:docker_hermes/p7/<路径> ``` -`--golden` 会先核验 Open、Dev、Shell、Launcher 四仓 delivery/runtime commit、Hermes 上游、五个产品镜像、 -三个 support image 和两个构建基础引用,再复用 P6 的隔离 LiteLLM、Shell、 -Launcher/JupyterHub 拓扑。Workspace 只能由 live DockerSpawner 创建;Shell -服务端从准确 Open 产品镜像推导 `hermes` Adapter,Launcher 以 -`RuntimeManifest` / `RuntimeSecretFile` 只读挂载运行材料。runner 执行真实 -Hermes 非流式调用、数据面流式调用、Hermes terminal tool、usage、两代 key、 -stop/restart/delete/revoke、owner 负向、零明文扫描与 run-scoped cleanup。 - -Console 鼠标创建、键盘/焦点和 axe 使用同一 Shell commit 的 P7 浏览器证据; -runner 另外执行该固定 Shell image 的 live API → JupyterHub → Workspace 链, -不得以 P6 OpenClaw 报告或健康检查替代 Hermes 成功。失败报告只保存错误码, -成功报告只保存结构断言、非敏感 ID、计数和 SHA-256。 - -当前固定组合已由 run `p7-c2d3e4f5a60718293a4b5c6d7e8f9012` 完成真实黄金链; -聚合报告 SHA-256 为 -`1d4331b482dd6959efba7707f991c79bf0076cf46ff2b8f7813c31de862d1a61`, -产品报告 SHA-256 为 -`de08555b73a3d2229d1047931c1dd3ef97b38cd1f62fa1d1c219dcdba95567ed`。 -该事实只证明本地固定 commit/digest 组合,不表示远端镜像、远端 integration、 -main 或部署状态。 +仍保留的 `scripts/test-p7-gates.sh` 是不启动容器的 Hermes 静态产品契约检查; +受支持构建与日常启动方式见父目录 README 和 `demo/`。 diff --git a/docker_hermes/p7/docker-compose.runtime.yml b/docker_hermes/p7/docker-compose.runtime.yml deleted file mode 100644 index c6293a3..0000000 --- a/docker_hermes/p7/docker-compose.runtime.yml +++ /dev/null @@ -1,33 +0,0 @@ -name: p7-hermes - -# P7's Hermes component boundary. The full product topology remains owned by -# the checked-in P7 runner, which starts the live Launcher-managed workspace; -# this file deliberately has no fallback image, credential value, or floating -# tag. RuntimeSecretFile is mounted by the Launcher, never injected here as an -# environment value. -services: - hermes: - image: ${P7_HERMES_IMAGE:?fixed P7 Hermes image required} - pull_policy: never - platform: linux/amd64 - environment: - HERMES_HOME: /root/.hermes - HERMES_DASHBOARD: "true" - HERMES_DASHBOARD_HOST: 127.0.0.1 - HERMES_DASHBOARD_PORT: "9119" - HERMES_DASHBOARD_USE_TRUSTED_PROXY_AUTH: "true" - volumes: - - type: bind - source: ${P7_HERMES_STATE_DIR:?run-scoped Hermes state required} - target: /root/.hermes - healthcheck: - test: ["CMD", "/usr/local/bin/start-hermes.sh", "healthcheck"] - interval: 5s - timeout: 3s - retries: 18 - start_period: 20s - networks: [p7-runtime] - -networks: - p7-runtime: - name: ${P7_RUNTIME_NETWORK:?run-scoped network required} diff --git a/docker_hermes/p7/launcher-overlay.Dockerfile b/docker_hermes/p7/launcher-overlay.Dockerfile deleted file mode 100644 index ba3fb0d..0000000 --- a/docker_hermes/p7/launcher-overlay.Dockerfile +++ /dev/null @@ -1,16 +0,0 @@ -# syntax=docker/dockerfile:1 - -ARG P6_LAUNCHER_IMAGE -FROM ${P6_LAUNCHER_IMAGE} - -ARG LAUNCHER_SOURCE_COMMIT -ARG P6_LAUNCHER_IMAGE -ARG P6_LAUNCHER_BASE_DIGEST -LABEL org.opencontainers.image.revision="${LAUNCHER_SOURCE_COMMIT}" \ - io.labnow.p7.launcher-base="${P6_LAUNCHER_BASE_DIGEST}" \ - io.labnow.p7.delivery="local-only-overlay" - -# P7 only changes the Launcher runtime consumer and its Hub-trusted config. -# The named build context must be the clean, fixed Launcher Phase checkout. -COPY --from=launcher src/labnow-launcher/devhub_launcher /opt/jupyterhub/devhub_launcher -COPY --from=launcher src/labnow-launcher/resource/config/app.conf /opt/jupyterhub/resource/config/app.conf diff --git a/docker_hermes/p7/p7-inputs.example.json b/docker_hermes/p7/p7-inputs.example.json deleted file mode 100644 index 2ce032a..0000000 --- a/docker_hermes/p7/p7-inputs.example.json +++ /dev/null @@ -1,32 +0,0 @@ -{ - "schema_version": "p7-inputs/v1", - "contract_version": "v1alpha1", - "contract_release": "0.1.0-rc.1", - "control_commit": "06c49f26642c7e39a118aedad1395197f2bd91db", - "review_policy_commit": "06c49f26642c7e39a118aedad1395197f2bd91db", - "phase": {"branch": "dev/che-568-hermes-console-experience", "base_commit": "45c38585a0ca889f6a20aebfdf3b13a01d369ac2"}, - "repositories": { - "lab_dev": {"path": "/absolute/path/to/lab-dev", "commit": "REPLACE_WITH_P7_LAB_DEV_HEAD", "runtime_commit": "REPLACE_WITH_P7_LAB_DEV_RUNTIME_HEAD"}, - "labnow_open": {"path": "/absolute/path/to/labnow-open", "commit": "18b20aa7fa3e506b9c85b88736c9f51f317d55d8", "runtime_commit": "2ac4e268d562c7d26ace8affc830f09cf1cb9305"}, - "labnow_shell": {"path": "/absolute/path/to/labnow-shell", "commit": "REPLACE_WITH_P7_SHELL_HEAD", "runtime_commit": "REPLACE_WITH_P7_SHELL_RUNTIME_HEAD"}, - "labnow_launcher": {"path": "/absolute/path/to/labnow-launcher", "commit": "990910aafeb6715bdfd656d002c3c6a27ff75cdb", "runtime_commit": "f84a51319d75b99a6b210f19e264904cae07fc8a"}, - "hermes_source": {"path": "/absolute/path/to/hermes-agent", "repository": "REPLACE_WITH_OBSERVED_HERMES_REMOTE", "commit": "REPLACE_WITH_FIXED_HERMES_COMMIT"} - }, - "images": { - "hermes": {"ref": "quay.io/labnow/hermes:p7-REPLACE_WITH_12_HEX", "image_id": "sha256:REPLACE_WITH_64_HEX", "repo_digest": "quay.io/labnow/hermes@sha256:REPLACE_WITH_64_HEX", "provenance": "local_build", "source_repository": "hermes_source", "source_commit": "REPLACE_WITH_FIXED_HERMES_COMMIT"}, - "litellm": {"ref": "quay.io/labnow/litellm@sha256:a2e115874c21b829bd052b18fc85be2f9217fb8244b82812c4ebc6e36f9824d1", "image_id": "sha256:a2e115874c21b829bd052b18fc85be2f9217fb8244b82812c4ebc6e36f9824d1", "repo_digest": "quay.io/labnow/litellm@sha256:a2e115874c21b829bd052b18fc85be2f9217fb8244b82812c4ebc6e36f9824d1", "provenance": "repo_digest"}, - "workspace": {"ref": "quay.io/labnow/labnow-open@sha256:b80bca4bfedbab67f6d14855509cc119aeeb212f25e4ce19a5738d7c53ba4d27", "image_id": "sha256:b80bca4bfedbab67f6d14855509cc119aeeb212f25e4ce19a5738d7c53ba4d27", "repo_digest": "quay.io/labnow/labnow-open@sha256:b80bca4bfedbab67f6d14855509cc119aeeb212f25e4ce19a5738d7c53ba4d27", "provenance": "local_build", "source_repository": "labnow_open", "source_commit": "2ac4e268d562c7d26ace8affc830f09cf1cb9305"}, - "shell": {"ref": "quay.io/labnow/labnow-shell@sha256:REPLACE_WITH_64_HEX", "image_id": "sha256:REPLACE_WITH_64_HEX", "repo_digest": "quay.io/labnow/labnow-shell@sha256:REPLACE_WITH_64_HEX", "provenance": "local_build", "source_repository": "labnow_shell", "source_commit": "REPLACE_WITH_P7_SHELL_RUNTIME_HEAD"}, - "launcher": {"ref": "quay.io/labnow/labnow-launcher@sha256:75f138b0f72c7cf35ecd923a3ba6152879dda99c5c187aef129dfab226bba6f7", "image_id": "sha256:75f138b0f72c7cf35ecd923a3ba6152879dda99c5c187aef129dfab226bba6f7", "repo_digest": "quay.io/labnow/labnow-launcher@sha256:75f138b0f72c7cf35ecd923a3ba6152879dda99c5c187aef129dfab226bba6f7", "provenance": "local_build", "source_repository": "labnow_launcher", "source_commit": "f84a51319d75b99a6b210f19e264904cae07fc8a", "base_image": "quay.io/labnow/labnow-launcher@sha256:6f9732fda8b86d9bfe4596e848025cc38448da4b17dfea8520e046a32b32e61f"} - }, - "support_images": { - "postgres": {"ref": "postgres:17-alpine", "image_id": "sha256:742f40ea20b9ff2ff31db5458d127452988a2164df9e17441e191f3b72252193", "repo_digest": "postgres@sha256:742f40ea20b9ff2ff31db5458d127452988a2164df9e17441e191f3b72252193"}, - "redis": {"ref": "redis:7.4-alpine", "image_id": "sha256:e7723ff73d963f5cc6d9c4643ea3d989527a402a319239054e9472a7fb9219a2", "repo_digest": "redis@sha256:e7723ff73d963f5cc6d9c4643ea3d989527a402a319239054e9472a7fb9219a2"}, - "nginx": {"ref": "nginx:1.27-alpine", "image_id": "sha256:65645c7bb6a0661892a8b03b89d0743208a18dd2f3f17a54ef4b76fb8e2f2a10", "repo_digest": "nginx@sha256:65645c7bb6a0661892a8b03b89d0743208a18dd2f3f17a54ef4b76fb8e2f2a10"} - }, - "base_images": { - "build": "quay.io/labnow/node@sha256:REPLACE_WITH_64_HEX", - "runtime": "quay.io/labnow/base@sha256:REPLACE_WITH_64_HEX" - }, - "runtime": {"p1_env_file": "/secure/path/to/p1.env"} -} diff --git a/docker_hermes/p7/scripts/p7-prepare-runtime.py b/docker_hermes/p7/scripts/p7-prepare-runtime.py deleted file mode 100755 index b10b1dc..0000000 --- a/docker_hermes/p7/scripts/p7-prepare-runtime.py +++ /dev/null @@ -1,144 +0,0 @@ -#!/usr/bin/env python3 -"""Prepare the P7 Hermes topology by adapting the already-verified P6 runtime. - -The translation contains only paths, commits and image identities. P1 values -and generated credentials remain in the P6 preparer's run-scoped private files. -""" - -from __future__ import annotations - -import importlib.util -import json -import os -import stat -import sys -from pathlib import Path -from typing import Any - - -class PrepareError(RuntimeError): - pass - - -def restricted(path: Path, code: str) -> None: - try: - info = path.stat() - except OSError as exc: - raise PrepareError(code) from exc - if not stat.S_ISREG(info.st_mode) or info.st_mode & 0o077: - raise PrepareError(code) - - -def load_json(path: Path, code: str) -> dict[str, Any]: - restricted(path, code) - try: - value = json.loads(path.read_text(encoding="utf-8")) - except (OSError, UnicodeDecodeError, json.JSONDecodeError) as exc: - raise PrepareError(code) from exc - if not isinstance(value, dict): - raise PrepareError(code) - return value - - -def load_p6_prepare(path: Path): - spec = importlib.util.spec_from_file_location("labnow_p6_prepare", path) - if spec is None or spec.loader is None: - raise PrepareError("P6_PREPARER_UNAVAILABLE") - module = importlib.util.module_from_spec(spec) - spec.loader.exec_module(module) - return module - - -def main() -> int: - input_path = Path(os.environ.get("P7_INPUT_FILE", "")) - work_dir = Path(os.environ.get("P7_WORK_DIR", "")) - artifact_dir = Path(os.environ.get("P7_ARTIFACTS_DIR", "")) - run_id = os.environ.get("P7_RUN_ID", "") - if not input_path.is_absolute() or not work_dir.is_absolute() or not artifact_dir.is_absolute(): - raise PrepareError("P7_PREPARE_PATH_INVALID") - if not run_id.startswith("p7-") or len(run_id) != 35: - raise PrepareError("P7_PREPARE_RUN_ID_INVALID") - - inputs = load_json(input_path, "P7_INPUT_INVALID") - repositories = inputs["repositories"] - images = inputs["images"] - translated = { - "repositories": repositories, - "images": { - "litellm": images["litellm"], - "openclaw_workspace": images["workspace"], - }, - "local_only_images": { - "launcher": images["launcher"], - "shell": images["shell"], - }, - "support_images": inputs["support_images"], - "runtime": inputs["runtime"], - } - - script_dir = Path(__file__).resolve().parent - p6_prepare_path = script_dir.parents[2] / "docker_openclaw" / "p6" / "scripts" / "p6-prepare-runtime.py" - p6 = load_p6_prepare(p6_prepare_path) - work_dir.mkdir(mode=0o700, parents=True, exist_ok=True) - os.chmod(work_dir, 0o700) - translated_path = work_dir / "p7-to-p6-runtime-input.json" - p6.write_private(translated_path, json.dumps(translated, separators=(",", ":")) + "\n", mode=0o600) - - internal_run_id = "p6-" + run_id.removeprefix("p7-") - previous = {name: os.environ.get(name) for name in ("P6_INPUT_FILE", "P6_WORK_DIR", "P6_RUN_ID", "P6_ARTIFACTS_DIR")} - os.environ.update( - { - "P6_INPUT_FILE": str(translated_path), - "P6_WORK_DIR": str(work_dir), - "P6_RUN_ID": internal_run_id, - "P6_ARTIFACTS_DIR": str(artifact_dir), - } - ) - try: - if p6.main() != 0: - raise PrepareError("P6_PREPARE_FAILED") - finally: - for name, value in previous.items(): - if value is None: - os.environ.pop(name, None) - else: - os.environ[name] = value - - shell_env = work_dir / "secrets" / "shell.env" - restricted(shell_env, "SHELL_ENV_INVALID") - shell_text = shell_env.read_text(encoding="utf-8") - workspace_ref = images["workspace"]["ref"] - if not workspace_ref.startswith("quay.io/labnow/labnow-open@sha256:"): - raise PrepareError("P7_WORKSPACE_IMAGE_INVALID") - p6.write_private( - shell_env, - shell_text.rstrip("\n") + f"\nMODEL_ACCESS_HERMES_WORKSPACE_IMAGE={workspace_ref}\n", - mode=0o600, - ) - - config_path = work_dir / "config" / "driver-config.json" - config = load_json(config_path, "P7_DRIVER_CONFIG_INVALID") - config.update( - { - "schema_version": "p7-product-chain-config/v1", - "run_id": run_id, - "internal_run_id": internal_run_id, - "adapter_id": "hermes", - "shell_ui_evidence": { - "repository": "labnow_shell", - "commit": repositories["labnow_shell"]["commit"], - "status": "reused_p7_verified_evidence", - }, - } - ) - p6.write_private(config_path, json.dumps(config, separators=(",", ":")) + "\n", mode=0o600) - return 0 - - -if __name__ == "__main__": - try: - sys.exit(main()) - except (PrepareError, OSError, KeyError, TypeError, ValueError) as exc: - code = str(exc) if isinstance(exc, PrepareError) else "P7_PREPARE_FAILED" - print(f"P7_ERROR:{code}", file=sys.stderr) - sys.exit(1) diff --git a/docker_hermes/p7/scripts/p7-product-chain.py b/docker_hermes/p7/scripts/p7-product-chain.py deleted file mode 100755 index f9bb071..0000000 --- a/docker_hermes/p7/scripts/p7-product-chain.py +++ /dev/null @@ -1,457 +0,0 @@ -#!/usr/bin/env python3 -"""Execute the live P7 Shell -> Launcher -> Hermes -> LiteLLM chain. - -Model output and credentials are handled in memory only. The retained report -contains structural assertions, non-sensitive IDs, counts and lifecycle state. -""" - -from __future__ import annotations - -import importlib.util -import json -import os -import ssl -import stat -import subprocess -import sys -import time -import urllib.error -import urllib.parse -import urllib.request -from datetime import datetime, timedelta, timezone -from pathlib import Path -from typing import Any - - -class DriverError(RuntimeError): - pass - - -def fail(code: str) -> None: - raise DriverError(code) - - -def load_p6_module(): - path = Path(__file__).resolve().parents[3] / "docker_openclaw" / "p6" / "scripts" / "p6-product-chain.py" - spec = importlib.util.spec_from_file_location("labnow_p6_product", path) - if spec is None or spec.loader is None: - fail("P6_PRODUCT_MODULE_UNAVAILABLE") - module = importlib.util.module_from_spec(spec) - spec.loader.exec_module(module) - return module - - -p6 = load_p6_module() -MANIFEST = "/run/labnow/model-access/manifest.json" -SECRET = "/run/labnow/model-access/secret.json" -STATUS = "/run/labnow/model-access/status.json" -HERMES_HOME = "/root/.hermes" - - -def restricted(path: Path, code: str) -> None: - try: - info = path.stat() - except OSError as exc: - raise DriverError(code) from exc - if not stat.S_ISREG(info.st_mode) or info.st_mode & 0o077: - fail(code) - - -def private_write(path: Path, value: str, mode: int = 0o600) -> None: - p6.private_write(path, value, mode) - - -def load_config(path: Path) -> dict[str, Any]: - restricted(path, "PRODUCT_CONFIG_INVALID") - try: - value = json.loads(path.read_text(encoding="utf-8")) - except (OSError, UnicodeDecodeError, json.JSONDecodeError) as exc: - raise DriverError("PRODUCT_CONFIG_INVALID") from exc - required = { - "schema_version", "run_id", "internal_run_id", "adapter_id", "project", - "compose_file", "runtime_env_file", "work_dir", "surface_dir", - "workspace_root", "runtime_root", "owner_a", "owner_b", "server_name", - "workspace_container", "launcher_container", "shell_container", - "litellm_container", "shell_postgres_container", "workspace_image", - "hub_token_file", "launcher_token_file", "upstream_key_file", - "upstream_origin", "upstream_model", "ca_file", "secret_files", - "shell_ui_evidence", - } - if not isinstance(value, dict) or set(value) != required: - fail("PRODUCT_CONFIG_INVALID") - if value.get("schema_version") != "p7-product-chain-config/v1" or value.get("adapter_id") != "hermes": - fail("PRODUCT_CONFIG_INVALID") - if not isinstance(value.get("run_id"), str) or not value["run_id"].startswith("p7-"): - fail("PRODUCT_CONFIG_INVALID") - if not isinstance(value.get("internal_run_id"), str) or not value["internal_run_id"].startswith("p6-"): - fail("PRODUCT_CONFIG_INVALID") - for key in required - {"secret_files", "shell_ui_evidence"}: - if not isinstance(value.get(key), str) or not value[key]: - fail("PRODUCT_CONFIG_INVALID") - if not isinstance(value["secret_files"], list) or not value["secret_files"]: - fail("PRODUCT_CONFIG_INVALID") - evidence = value.get("shell_ui_evidence") - if not isinstance(evidence, dict) or evidence.get("status") != "reused_p7_verified_evidence": - fail("PRODUCT_CONFIG_INVALID") - return value - - -def assert_workspace(config: dict[str, Any], runtime_key: str) -> tuple[str, dict[str, Any]]: - inspect = p6.docker_inspect(config["workspace_container"]) - mounts = {item.get("Destination"): item for item in inspect.get("Mounts", [])} - for target in (MANIFEST, SECRET, "/run/labnow/p6-ca.pem"): - if target not in mounts or mounts[target].get("RW") is not False: - fail("WORKSPACE_MOUNT_INVALID") - if runtime_key in json.dumps(inspect, sort_keys=True): - fail("RUNTIME_KEY_LEAKED_TO_INSPECT") - env = inspect.get("Config", {}).get("Env", []) - prefix = next((item.split("=", 1)[1] for item in env if item.startswith("URL_PREFIX=")), "") - if not prefix.startswith("/studio/user/"): - fail("WORKSPACE_URL_PREFIX_INVALID") - status_raw = p6.command(["docker", "exec", config["workspace_container"], "cat", STATUS], code="ADAPTER_STATUS_UNAVAILABLE") - try: - adapter_status = json.loads(status_raw) - except json.JSONDecodeError as exc: - raise DriverError("ADAPTER_STATUS_INVALID") from exc - if adapter_status.get("phase") != "ready" or adapter_status.get("adapter_id") != "hermes": - fail("ADAPTER_STATUS_INVALID") - deadline = time.monotonic() + 90 - readiness = f"http://127.0.0.1{prefix}api" - while time.monotonic() < deadline: - result = subprocess.run( - ["docker", "exec", config["workspace_container"], "curl", "--fail", "--silent", "--max-time", "3", readiness], - check=False, - stdin=subprocess.DEVNULL, - stdout=subprocess.DEVNULL, - stderr=subprocess.DEVNULL, - ) - if result.returncode == 0: - break - time.sleep(1) - else: - fail("HERMES_READINESS_FAILED") - summary = { - "image": inspect.get("Config", {}).get("Image"), - "mounts": sorted(mounts), - "environment_keys": sorted(item.split("=", 1)[0] for item in env), - "state": inspect.get("State", {}).get("Status"), - } - return prefix, summary - - -def hermes_model_call(config: dict[str, Any], marker: str) -> dict[str, Any]: - output = p6.command( - [ - "docker", "exec", config["workspace_container"], "timeout", "--signal=TERM", - "--kill-after=10s", "180s", "start-labnow-hermes.sh", "-z", - f"Reply {marker} only.", "--ignore-rules", - ], - code="HERMES_MODEL_CALL_FAILED", - timeout=200, - ) - if marker not in output: - fail("HERMES_MODEL_RESPONSE_INVALID") - return {"completed": True, "response_retained": False} - - -def hermes_tool_call(config: dict[str, Any], marker: str) -> dict[str, Any]: - proof = f"{HERMES_HOME}/labnow-p7-tool-proof" - subprocess.run( - ["docker", "exec", config["workspace_container"], "rm", "-f", proof], - check=False, - stdin=subprocess.DEVNULL, - stdout=subprocess.DEVNULL, - stderr=subprocess.DEVNULL, - ) - try: - output = p6.command( - [ - "docker", "exec", config["workspace_container"], "timeout", "--signal=TERM", - "--kill-after=10s", "180s", "start-labnow-hermes.sh", "-z", - f"Use the terminal tool to run: printf {marker} > {proof}. Then reply {marker}_DONE only.", - "--ignore-rules", "-t", "terminal", - ], - code="HERMES_TOOL_CALL_FAILED", - timeout=200, - ) - proof_value = p6.command(["docker", "exec", config["workspace_container"], "cat", proof], code="HERMES_TOOL_PROOF_MISSING").strip() - if proof_value != marker or not output: - fail("HERMES_TOOL_PROOF_INVALID") - return {"completed": True, "tool_observed": True, "response_retained": False} - finally: - subprocess.run( - ["docker", "exec", config["workspace_container"], "rm", "-f", proof], - check=False, - stdin=subprocess.DEVNULL, - stdout=subprocess.DEVNULL, - stderr=subprocess.DEVNULL, - ) - - -def stream_call(port: int, ca_file: str, key: str, model: str) -> dict[str, Any]: - context = ssl.create_default_context(cafile=ca_file) - payload = json.dumps( - {"model": model, "messages": [{"role": "user", "content": "Reply P7_STREAM_OK only."}], "stream": True}, - separators=(",", ":"), - ).encode("utf-8") - request = urllib.request.Request( - f"https://127.0.0.1:{port}/chat/completions", - data=payload, - method="POST", - headers={"Accept": "text/event-stream", "Content-Type": "application/json", "Authorization": f"Bearer {key}"}, - ) - frames = 0 - done = False - try: - with urllib.request.urlopen(request, timeout=150, context=context) as response: - if response.status != 200: - fail("STREAM_HTTP_FAILED") - for raw in response: - line = raw.decode("utf-8").strip() - if not line.startswith("data:"): - continue - data = line[5:].strip() - if data == "[DONE]": - done = True - continue - value = json.loads(data) - if isinstance(value, dict) and isinstance(value.get("choices"), list): - frames += 1 - except (urllib.error.URLError, TimeoutError, OSError, UnicodeDecodeError, json.JSONDecodeError) as exc: - raise DriverError("STREAM_CALL_FAILED") from exc - if not done or frames < 1: - fail("STREAM_STRUCTURE_INVALID") - return {"event_frames": frames, "done": True, "response_retained": False} - - -def capture_surfaces(config: dict[str, Any], workspace_summary: dict[str, Any], generation: int) -> list[str]: - surface = Path(config["surface_dir"]) - surface.mkdir(mode=0o700, parents=True, exist_ok=True) - topology_logs = p6.compose(config, "logs", "--no-color", code="TOPOLOGY_LOG_CAPTURE_FAILED", timeout=120) - private_write(surface / f"topology-g{generation}.log", topology_logs) - workspace_logs = p6.command_combined(["docker", "logs", config["workspace_container"]], code="WORKSPACE_LOG_CAPTURE_FAILED") - private_write(surface / f"workspace-g{generation}.log", workspace_logs) - processes = p6.command(["docker", "top", config["workspace_container"], "-eo", "pid,args"], code="WORKSPACE_PROCESS_CAPTURE_FAILED") - private_write(surface / f"workspace-process-g{generation}.txt", processes) - managed = p6.command( - ["docker", "exec", config["workspace_container"], "cat", f"{HERMES_HOME}/labnow-model-access/config.yaml"], - code="HERMES_MANAGED_CONFIG_CAPTURE_FAILED", - ) - private_write(surface / f"hermes-managed-g{generation}.json", managed) - adapter_status = p6.command(["docker", "exec", config["workspace_container"], "cat", STATUS], code="ADAPTER_STATUS_UNAVAILABLE") - private_write(surface / f"adapter-status-g{generation}.json", adapter_status) - private_write(surface / f"workspace-inspect-g{generation}.json", json.dumps(workspace_summary, sort_keys=True) + "\n") - return [str(surface), str(Path(config["workspace_root"]) / config["owner_a"])] - - -def execute(config: dict[str, Any]) -> dict[str, Any]: - shell_port = p6.published_port(config, "shell", 3002) - hub_port = p6.published_port(config, "launcher", 8000) - gateway_port = p6.published_port(config, "litellm-gateway", 4443) - shell = f"http://127.0.0.1:{shell_port}/console/api" - hub = f"http://127.0.0.1:{hub_port}/studio/hub/api" - hub_token = p6.read_secret(config["hub_token_file"], "HUB_TOKEN_INVALID") - launcher_token = p6.read_secret(config["launcher_token_file"], "LAUNCHER_TOKEN_INVALID") - upstream_key = p6.read_secret(config["upstream_key_file"], "UPSTREAM_KEY_INVALID") - run = config["run_id"].replace("p7-", "")[:12] - - status, connection = p6.http_json( - "POST", f"{shell}/model-access/connections/", headers=p6.shell_headers("a", f"connection-{run}"), - body={"display_name": f"P7 {run}", "provider": "openai", "endpoint": config["upstream_origin"], "api_key": upstream_key}, - ) - p6.require_status(status, {201}, "CONNECTION_CREATE_FAILED", connection) - connection_id = connection.get("data", {}).get("id") if isinstance(connection, dict) else None - if not isinstance(connection_id, str): - fail("CONNECTION_RESPONSE_INVALID") - - status, route = p6.http_json( - "POST", f"{shell}/model-access/routes/", headers=p6.shell_headers("a", f"route-{run}"), - body={"connection_id": connection_id, "display_name": f"P7 route {run}", "upstream_model": config["upstream_model"]}, - ) - p6.require_status(status, {201}, "ROUTE_CREATE_FAILED", route) - route_value = route.get("data", {}) if isinstance(route, dict) else {} - route_id = route_value.get("id") - routed_model = route_value.get("routed_model") - if not isinstance(route_id, str) or not isinstance(routed_model, str): - fail("ROUTE_RESPONSE_INVALID") - - status, binding = p6.http_json( - "POST", f"{shell}/model-access/bindings/", headers=p6.shell_headers("a", f"binding-{run}"), - body={"workspace_id": config["server_name"], "route_id": route_id, "image": config["workspace_image"]}, - ) - p6.require_status(status, {201}, "BINDING_CREATE_FAILED", binding) - binding_value = binding.get("data", {}) if isinstance(binding, dict) else {} - binding_id = binding_value.get("binding_id") - if not isinstance(binding_id, str) or binding_value.get("adapter_id") != "hermes": - fail("BINDING_RESPONSE_INVALID") - - spawn_body = { - "tier": "basic", - "image": config["workspace_image"], - "serverName": config["server_name"], - "model_access": {"contract_version": "v1alpha1", "binding_id": binding_id}, - } - status, spawn_response = p6.http_json( - "POST", f"{shell}/hub/spawn/", headers=p6.shell_headers("a", f"spawn-g1-{run}"), body=spawn_body, timeout=45, - ) - p6.require_status(status, {201, 202}, "SHELL_SPAWN_FAILED", spawn_response) - server_snapshot = p6.wait_hub_server(hub, hub_token, config["owner_a"], config["server_name"], running=True) - material_dir_1, manifest_1, key_1 = p6.material(config) - _, workspace_summary_1 = assert_workspace(config, key_1) - p6.data_plane(gateway_port, config["ca_file"], key_1, accepted=True) - if key_1 in json.dumps(server_snapshot, sort_keys=True): - fail("RUNTIME_KEY_LEAKED_TO_HUB_API") - generation_1 = manifest_1["generation"] - from_time = p6.usage_time(datetime.now(timezone.utc) - timedelta(minutes=5)) - model_summary = hermes_model_call(config, "P7_HERMES_OK") - stream_summary = stream_call(gateway_port, config["ca_file"], key_1, manifest_1["default_model"]) - tool_summary = hermes_tool_call(config, "P7_HERMES_TOOL_OK") - to_time = p6.usage_time(datetime.now(timezone.utc) + timedelta(minutes=5)) - usage_count, _ = p6.usage_check(shell, config, routed_model, from_time, to_time) - scan_roots = capture_surfaces(config, workspace_summary_1, generation_1) - - status, _ = p6.http_json( - "POST", f"{shell}/hub/stop/", headers=p6.shell_headers("a", f"stop-g1-{run}"), - body={"serverName": config["server_name"]}, timeout=45, - ) - p6.require_status(status, {200, 202, 204}, "SHELL_STOP_FAILED") - p6.wait_hub_server(hub, hub_token, config["owner_a"], config["server_name"], running=False) - if material_dir_1.exists(): - fail("GENERATION_1_MATERIAL_REMAINS") - p6.wait_rejected(gateway_port, config["ca_file"], key_1) - - status, restart_response = p6.http_json( - "POST", f"{shell}/hub/spawn/", headers=p6.shell_headers("a", f"spawn-g2-{run}"), body=spawn_body, timeout=45, - ) - p6.require_status(status, {201, 202}, "SHELL_RESTART_FAILED", restart_response) - p6.wait_hub_server(hub, hub_token, config["owner_a"], config["server_name"], running=True) - material_dir_2, manifest_2, key_2 = p6.material(config) - _, workspace_summary_2 = assert_workspace(config, key_2) - if manifest_2["generation"] <= generation_1 or key_2 == key_1: - fail("GENERATION_NOT_ADVANCED") - p6.data_plane(gateway_port, config["ca_file"], key_2, accepted=True) - p6.wait_rejected(gateway_port, config["ca_file"], key_1) - restart_model = hermes_model_call(config, "P7_HERMES_RESTART_OK") - scan_roots.extend(capture_surfaces(config, workspace_summary_2, manifest_2["generation"])) - - late_body = { - "contract_version": "v1alpha1", "workspace_id": config["server_name"], - "generation": generation_1, "reason": "reconciled", - } - status, _ = p6.http_json( - "POST", - f"{shell}/internal/model-access/v1alpha1/runtime-leases/{urllib.parse.quote(manifest_1['lease_id'], safe='')}:release/", - headers={"Authorization": f"Bearer {launcher_token}", "Idempotency-Key": f"late-{run}", "X-Request-Id": f"late-{run}"}, - body=late_body, - ) - p6.require_status(status, {409}, "LATE_RELEASE_NOT_REJECTED") - p6.data_plane(gateway_port, config["ca_file"], key_2, accepted=True) - - status, _ = p6.http_json( - "DELETE", f"{shell}/hub/delete/", headers=p6.shell_headers("a", f"delete-g2-{run}"), - body={"serverName": config["server_name"], "remove": True}, timeout=45, - ) - p6.require_status(status, {200, 202, 204}, "SHELL_DELETE_FAILED") - p6.wait_hub_server(hub, hub_token, config["owner_a"], config["server_name"], running=False) - if material_dir_2.exists(): - fail("GENERATION_2_MATERIAL_REMAINS") - p6.wait_rejected(gateway_port, config["ca_file"], key_2) - active = p6.psql( - config, - "SELECT count(*) FROM model_access.runtime_leases WHERE owner_id='" - + config["owner_a"].replace("'", "''") - + "' AND workspace_id='" - + config["server_name"].replace("'", "''") - + "' AND state IN ('issued','active','revoking')", - ) - if active != "0": - fail("ACTIVE_LEASE_REMAINS") - if p6.psql(config, "SELECT coalesce(to_regclass('model_access.usage')::text,'absent')") != "absent": - fail("USAGE_BODY_PERSISTENCE_TABLE_PRESENT") - - pattern_file = Path(os.environ.get("P7_SECRET_PATTERN_FILE", "")) - if not pattern_file.is_absolute(): - fail("SECRET_PATTERN_PATH_INVALID") - p6.write_patterns(config, pattern_file, [key_1, key_2]) - scan_roots = sorted(set(root for root in scan_roots if Path(root).exists())) - if not scan_roots: - fail("SCAN_ROOT_MISSING") - - return { - "schema_version": "p7-product-chain-report/v1", - "result": "passed", - "content_redacted": True, - "checks": { - "console_mouse": "passed", - "binding_payload": "passed", - "jupyterhub_dockerspawner": "passed", - "launcher_claim_activate_release": "passed", - "hermes_apply_probe_readiness": "passed", - "model_call": "passed", - "stream": "passed", - "tool": "passed", - "usage": "passed", - "owner_negative": "passed", - "prompt_response_absent": "passed", - "revoke": "passed", - "generation_restart": "passed", - "late_release": "passed", - "delete": "passed", - "zero_active_leases": "passed", - }, - "console": { - "mouse_evidence": config["shell_ui_evidence"]["status"], - "shell_commit": config["shell_ui_evidence"]["commit"], - "live_spawn_reference": ["binding_id", "contract_version"], - }, - "binding": { - "contract_version": "v1alpha1", "workspace_id": config["server_name"], - "binding_id": binding_id, "route_id": route_id, "adapter_id": "hermes", - "payload_fields": ["binding_id", "contract_version"], - }, - "runtime": { - "hub_api": "live", "docker_daemon": "real", "workspace_image": config["workspace_image"], - "generation_1": generation_1, "generation_2": manifest_2["generation"], - "mounts": "readonly", "adapter_phase": "ready", - }, - "data_plane": { - "model_call": model_summary, "stream": stream_summary, "tool": tool_summary, - "restart_model_call": restart_model, - }, - "usage": { - "row_count": usage_count, "fields": sorted(p6.ALLOWED_USAGE_FIELDS), - "owner_negative": "isolated", "body_fields_absent": True, "persistence_table": "absent", - }, - "lifecycle": { - "old_key_after_stop": "rejected", "new_key_after_restart": "accepted", - "old_key_after_restart": "rejected", "late_old_release": "rejected_409", - "new_key_after_delete": "rejected", "active_lease_count": 0, - }, - "scan_roots": scan_roots, - } - - -def main() -> int: - config_path = Path(os.environ.get("P7_PRODUCT_CONFIG_FILE", "")) - report_path = Path(os.environ.get("P7_PRODUCT_REPORT_FILE", "")) - try: - config = load_config(config_path) - report = execute(config) - except (DriverError, p6.DriverError) as exc: - code = str(exc) - if report_path.is_absolute(): - private_write( - report_path, - json.dumps({"schema_version": "p7-product-chain-report/v1", "result": "failed", "content_redacted": True, "code": code}, separators=(",", ":")) + "\n", - ) - print(f"P7_ERROR:{code}", file=sys.stderr) - return 1 - if not report_path.is_absolute(): - print("P7_ERROR:PRODUCT_REPORT_PATH_INVALID", file=sys.stderr) - return 1 - private_write(report_path, json.dumps(report, separators=(",", ":")) + "\n") - return 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/docker_hermes/p7/scripts/p7-runner.sh b/docker_hermes/p7/scripts/p7-runner.sh deleted file mode 100755 index 041de92..0000000 --- a/docker_hermes/p7/scripts/p7-runner.sh +++ /dev/null @@ -1,273 +0,0 @@ -#!/usr/bin/env bash -# P7 local-only Hermes coordinator. It reuses the P6 live topology while -# replacing only the Workspace image, trusted Shell catalogue and product -# checks. All credentials stay in run-scoped 0400/0600 files. -set -euo pipefail - -script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -p7_dir="$(cd "${script_dir}/.." && pwd)" -repo_root="$(cd "${p7_dir}/../.." && pwd)" -p6_dir="${repo_root}/docker_openclaw/p6" - -die() { printf 'P7_ERROR:%s\n' "$1" >&2; return "${2:-1}"; } -sha256() { shasum -a 256 "$1" | awk '{print $1}'; } -run_id() { python3 -c 'import secrets; print("p7-" + secrets.token_hex(16))'; } -mode_of() { stat -f '%Lp' "$1" 2>/dev/null || stat -c '%a' "$1"; } -secure_file() { - [[ -f "$1" && ! -L "$1" && "$(mode_of "$1")" =~ ^(400|600)$ ]] || { die "SECURE_FILE_REQUIRED" 64; return $?; } -} - -usage() { printf '%s\n' "Usage: p7-runner.sh --input /secure/path/p7-inputs.json --validate-input|--preflight|--render|--golden|--cleanup"; } -input=""; action="" -while (($#)); do - case "$1" in - --input) input="${2:-}"; shift 2 ;; - --validate-input|--preflight|--render|--golden|--cleanup) [[ -z "$action" ]] || { die "ACTION_DUPLICATED" 2; exit 2; }; action="${1#--}"; shift ;; - *) usage >&2; exit 2 ;; - esac -done -[[ -n "$input" && -n "$action" ]] || { usage >&2; exit 2; } -secure_file "$input" || exit $? -input="$(cd "$(dirname "$input")" && pwd)/$(basename "$input")" -export P7_INPUT_FILE="$input" -P7_RUN_ID="${P7_RUN_ID:-$(run_id)}" -[[ "$P7_RUN_ID" =~ ^p7-[a-f0-9]{32}$ ]] || { die "RUN_ID_INVALID" 64; exit 64; } -export P7_RUN_ID -artifact_dir="${P7_ARTIFACTS_DIR:-${p7_dir}/artifacts}" -work_dir="${P7_WORK_DIR:-${p7_dir}/.p7-work/${P7_RUN_ID}}" -[[ "$artifact_dir" = /* && "$work_dir" = /* && ${#work_dir} -gt 12 ]] || { die "RUNTIME_PATH_INVALID" 64; exit 64; } -case "$work_dir" in /|"$repo_root"|"$p7_dir") die "RUNTIME_PATH_INVALID" 64; exit 64 ;; esac -export P7_ARTIFACTS_DIR="$artifact_dir" P7_WORK_DIR="$work_dir" -mkdir -p "$artifact_dir"; chmod 700 "$artifact_dir" -report="${artifact_dir}/p7-${action}-${P7_RUN_ID}.json" - -validate_shape() { - jq -e ' - . as $root - | type == "object" - and .schema_version == "p7-inputs/v1" - and .contract_version == "v1alpha1" and .contract_release == "0.1.0-rc.1" - and .control_commit == "06c49f26642c7e39a118aedad1395197f2bd91db" - and .review_policy_commit == "06c49f26642c7e39a118aedad1395197f2bd91db" - and .phase.branch == "dev/che-568-hermes-console-experience" - and .phase.base_commit == "45c38585a0ca889f6a20aebfdf3b13a01d369ac2" - and (.repositories | keys | sort) == ["hermes_source","lab_dev","labnow_launcher","labnow_open","labnow_shell"] - and (["lab_dev","labnow_open","labnow_shell","labnow_launcher"] | all(. as $name | - ($root.repositories[$name] | keys | sort) == ["commit","path","runtime_commit"] - and ($root.repositories[$name].path | type == "string" and startswith("/")) - and ($root.repositories[$name].commit | type == "string" and test("^[0-9a-f]{40}$")) - and ($root.repositories[$name].runtime_commit | type == "string" and test("^[0-9a-f]{40}$")))) - and (.repositories.hermes_source | keys | sort) == ["commit","path","repository"] - and (.repositories.hermes_source.path | type == "string" and startswith("/")) - and (.repositories.hermes_source.repository | type == "string" and startswith("https://")) - and (.repositories.hermes_source.commit | type == "string" and test("^[0-9a-f]{40}$")) - and (.repositories.labnow_open.commit == "18b20aa7fa3e506b9c85b88736c9f51f317d55d8") - and (.repositories.labnow_open.runtime_commit == "2ac4e268d562c7d26ace8affc830f09cf1cb9305") - and (.repositories.labnow_launcher.commit == "990910aafeb6715bdfd656d002c3c6a27ff75cdb") - and (.repositories.labnow_launcher.runtime_commit == "f84a51319d75b99a6b210f19e264904cae07fc8a") - and (.images | keys | sort) == ["hermes","launcher","litellm","shell","workspace"] - and (.support_images | keys | sort) == ["nginx","postgres","redis"] - and ([.images[] | .image_id] + [.support_images[] | .image_id] | all(type == "string" and test("^sha256:[0-9a-f]{64}$"))) - and ([.images[] | .repo_digest] + [.support_images[] | .repo_digest] | all(type == "string" and test("^[^[:space:]]+@sha256:[0-9a-f]{64}$"))) - and (.images.hermes.ref | test("^quay\\.io/labnow/hermes:p7-[0-9a-f]{12}$")) - and .images.hermes.provenance == "local_build" - and .images.hermes.source_repository == "hermes_source" - and .images.hermes.source_commit == .repositories.hermes_source.commit - and (.images.workspace.ref | test("^quay\\.io/labnow/labnow-open@sha256:[0-9a-f]{64}$")) - and (.images.shell.ref | test("^quay\\.io/labnow/labnow-shell@sha256:[0-9a-f]{64}$")) - and (.images.launcher.ref | test("^quay\\.io/labnow/labnow-launcher@sha256:[0-9a-f]{64}$")) - and (.images.litellm.ref | test("^quay\\.io/labnow/litellm@sha256:[0-9a-f]{64}$")) - and (["workspace","shell","launcher"] | all(. as $name | - $root.images[$name].provenance == "local_build" - and ($root.images[$name].source_repository | type == "string") - and $root.images[$name].source_commit == $root.repositories[$root.images[$name].source_repository].runtime_commit)) - and .images.launcher.base_image == "quay.io/labnow/labnow-launcher@sha256:6f9732fda8b86d9bfe4596e848025cc38448da4b17dfea8520e046a32b32e61f" - and .images.litellm.provenance == "repo_digest" - and ([.images.workspace,.images.shell,.images.launcher,.images.litellm] | all(.ref == .repo_digest)) - and (.base_images | keys | sort) == ["build","runtime"] - and ([.base_images[]] | all(type == "string" and test("^quay\\.io/labnow/(node|base)@sha256:[0-9a-f]{64}$"))) - and (.runtime | keys | sort) == ["p1_env_file"] - and (.runtime.p1_env_file | type == "string" and startswith("/")) - ' "$input" >/dev/null || { die "INPUT_SCHEMA_INVALID" 68; return $?; } -} - -assert_repository() { - local name="$1" path commit runtime_commit actual status changed - path="$(jq -er ".repositories.${name}.path" "$input")" - commit="$(jq -er ".repositories.${name}.commit" "$input")" - runtime_commit="$(jq -er ".repositories.${name}.runtime_commit" "$input")" - [[ -d "$path/.git" ]] || { die "REPOSITORY_UNAVAILABLE" 69; return $?; } - actual="$(git -C "$path" rev-parse HEAD)" - [[ "$actual" == "$commit" ]] || { die "REPOSITORY_COMMIT_MISMATCH" 70; return $?; } - status="$(git -C "$path" status --porcelain=v1 --untracked-files=no)" - [[ -z "$status" ]] || { die "REPOSITORY_TRACKED_TREE_DIRTY" 71; return $?; } - git -C "$path" merge-base --is-ancestor "$runtime_commit" "$commit" || { die "RUNTIME_COMMIT_NOT_ANCESTOR" 70; return $?; } - if [[ "$runtime_commit" != "$commit" ]]; then - changed="$(git -C "$path" -c core.quotePath=false diff --name-only "$runtime_commit..$commit")" - [[ -n "$changed" ]] || { die "RUNTIME_DELIVERY_DELTA_MISSING" 70; return $?; } - if grep -Ev '^(doc|docs|development-docs)/|(^|/)README\.md$' <<<"$changed" >/dev/null; then - die "RUNTIME_DELIVERY_DELTA_NOT_DOCUMENTATION" 70; return $? - fi - fi -} - -assert_hermes_source() { - local path expected_repository expected_commit - path="$(jq -er '.repositories.hermes_source.path' "$input")" - expected_repository="$(jq -er '.repositories.hermes_source.repository' "$input")" - expected_commit="$(jq -er '.repositories.hermes_source.commit' "$input")" - [[ -d "$path/.git" ]] || { die "HERMES_SOURCE_UNAVAILABLE" 69; return $?; } - [[ "$(git -C "$path" rev-parse HEAD)" == "$expected_commit" ]] || { die "HERMES_SOURCE_COMMIT_MISMATCH" 70; return $?; } - [[ "$(git -C "$path" remote get-url origin)" == "$expected_repository" ]] || { die "HERMES_SOURCE_REMOTE_MISMATCH" 70; return $?; } - [[ -z "$(git -C "$path" status --porcelain=v1 --untracked-files=no)" ]] || { die "HERMES_SOURCE_TRACKED_TREE_DIRTY" 71; return $?; } -} - -assert_image() { - local section="$1" name="$2" ref expected_id expected_digest actual_id digests - ref="$(jq -er ".${section}.${name}.ref" "$input")" - expected_id="$(jq -er ".${section}.${name}.image_id" "$input")" - expected_digest="$(jq -er ".${section}.${name}.repo_digest" "$input")" - actual_id="$(docker image inspect --format '{{.Id}}' "$ref" 2>/dev/null)" || { die "IMAGE_UNAVAILABLE" 72; return $?; } - [[ "$actual_id" == "$expected_id" ]] || { die "IMAGE_ID_MISMATCH" 72; return $?; } - digests="$(docker image inspect --format '{{join .RepoDigests "\n"}}' "$ref")" - grep -Fqx "$expected_digest" <<<"$digests" || { die "IMAGE_DIGEST_MISMATCH" 72; return $?; } -} - -assert_images() { - local name - for name in hermes litellm workspace shell launcher; do assert_image images "$name" || return $?; done - for name in postgres redis nginx; do assert_image support_images "$name" || return $?; done - local hermes_ref launcher_ref launcher_base - hermes_ref="$(jq -er '.images.hermes.ref' "$input")" - [[ "$(docker image inspect --format '{{ index .Config.Labels "org.opencontainers.image.revision" }}' "$hermes_ref")" == "$(jq -er '.repositories.hermes_source.commit' "$input")" ]] || { die "HERMES_IMAGE_PROVENANCE_MISMATCH" 72; return $?; } - [[ "$(docker image inspect --format '{{ index .Config.Labels "io.labnow.hermes.build-base" }}' "$hermes_ref")" == "$(jq -er '.base_images.build' "$input")" ]] || { die "HERMES_BUILD_BASE_MISMATCH" 72; return $?; } - [[ "$(docker image inspect --format '{{ index .Config.Labels "io.labnow.hermes.runtime-base" }}' "$hermes_ref")" == "$(jq -er '.base_images.runtime' "$input")" ]] || { die "HERMES_RUNTIME_BASE_MISMATCH" 72; return $?; } - docker image inspect "$(jq -er '.base_images.build' "$input")" "$(jq -er '.base_images.runtime' "$input")" >/dev/null 2>&1 || { die "HERMES_BASE_IMAGE_UNAVAILABLE" 72; return $?; } - launcher_ref="$(jq -er '.images.launcher.ref' "$input")" - launcher_base="$(jq -er '.images.launcher.base_image' "$input")" - [[ "$(docker image inspect --format '{{ index .Config.Labels "org.opencontainers.image.revision" }}' "$launcher_ref")" == "$(jq -er '.repositories.labnow_launcher.runtime_commit' "$input")" ]] || { die "LAUNCHER_IMAGE_PROVENANCE_MISMATCH" 72; return $?; } - [[ "$(docker image inspect --format '{{ index .Config.Labels "io.labnow.p7.launcher-base" }}' "$launcher_ref")" == "$launcher_base" ]] || { die "LAUNCHER_BASE_IMAGE_MISMATCH" 72; return $?; } - docker image inspect "$launcher_base" >/dev/null 2>&1 || { die "LAUNCHER_BASE_IMAGE_UNAVAILABLE" 72; return $?; } -} - -assert_runtime_input() { - local env_file - env_file="$(jq -er '.runtime.p1_env_file' "$input")" - secure_file "$env_file" || return $? - python3 - "$env_file" <<'PY' -import sys -from pathlib import Path -required = {"UPSTREAM_API_KEY", "UPSTREAM_BASE_URL", "UPSTREAM_MODEL"} -values = {} -for raw in Path(sys.argv[1]).read_text(encoding="utf-8").splitlines(): - line = raw.strip() - if line and not line.startswith("#") and "=" in line: - key, value = line.split("=", 1) - values[key] = value -raise SystemExit(0 if all(values.get(key) for key in required) else 1) -PY - [[ $? == 0 ]] || { die "P1_ENV_INCOMPLETE" 73; return $?; } -} - -preflight() { - validate_shape || return $? - local repo - for repo in lab_dev labnow_open labnow_shell labnow_launcher; do assert_repository "$repo" || return $?; done - assert_hermes_source || return $? - [[ "$(git -C "$(jq -er '.repositories.lab_dev.path' "$input")" branch --show-current)" == "dev/che-568-hermes-console-experience" ]] || { die "PHASE_BRANCH_MISMATCH" 70; return $?; } - git -C "$(jq -er '.repositories.lab_dev.path' "$input")" merge-base --is-ancestor "$(jq -er '.phase.base_commit' "$input")" HEAD || { die "PHASE_BASE_NOT_ANCESTOR" 70; return $?; } - assert_images || return $? - assert_runtime_input || return $? -} - -write_report() { - local result="$1" phase="$2" reason="${3:-}" extra="${4:-}" tmp - [[ -n "$extra" ]] || extra='{}' - tmp="$(mktemp "${artifact_dir}/.p7-report.XXXXXX")" - jq -n --arg run_id "$P7_RUN_ID" --arg result "$result" --arg phase "$phase" --arg reason "$reason" --arg input_sha "$(sha256 "$input")" --arg tested_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" --argjson provenance "$(jq -c '{contract_version,contract_release,control_commit,review_policy_commit,phase,repositories:(.repositories|with_entries(.value=if .key=="hermes_source" then {commit:.value.commit,repository:.value.repository} else {commit:.value.commit,runtime_commit:.value.runtime_commit} end)),images,support_images,base_images}' "$input")" --argjson extra "$extra" \ - '{schema_version:"p7-report/v1",run_id:$run_id,result:$result,phase:$phase,input_sha256:$input_sha,tested_at:$tested_at,content_redacted:true} + $provenance + $extra + (if $reason == "" then {} else {reason:$reason} end)' > "$tmp" - chmod 600 "$tmp"; mv -f "$tmp" "$report" -} - -compose() { - local short="${P7_RUN_ID#p7-}" - short="${short:0:12}" - docker compose --project-name "p6-runtime-${short}" --env-file "${work_dir}/runtime.env" -f "${p6_dir}/docker-compose.runtime.yml" "$@" -} - -cleanup_runtime() { - local workspace="p6w-${P7_RUN_ID#p7-}" - workspace="${workspace:0:12}-p6user-${P7_RUN_ID#p7-}" - if [[ -f "${work_dir}/config/driver-config.json" && ! -L "${work_dir}/config/driver-config.json" ]]; then - workspace="$(jq -r '.workspace_container // empty' "${work_dir}/config/driver-config.json" 2>/dev/null || true)" - fi - [[ -z "$workspace" ]] || docker rm -f "$workspace" >/dev/null 2>&1 || true - if [[ -f "${work_dir}/runtime.env" && ! -L "${work_dir}/runtime.env" ]]; then compose down --volumes --remove-orphans >/dev/null 2>&1 || true; fi - rm -rf -- "$work_dir" -} - -assert_cleanup() { - local short="${P7_RUN_ID#p7-}" project="p6-runtime-${P7_RUN_ID#p7-}" - short="${short:0:12}"; project="p6-runtime-${short}" - [[ -z "$(docker container ls -aq --filter "label=com.docker.compose.project=${project}")" ]] || { die "TOPOLOGY_CONTAINER_REMAINS" 79; return $?; } - [[ -z "$(docker volume ls -q --filter "label=com.docker.compose.project=${project}")" ]] || { die "TOPOLOGY_VOLUME_REMAINS" 79; return $?; } - ! docker network inspect "p6net-${short}" >/dev/null 2>&1 || { die "TOPOLOGY_NETWORK_REMAINS" 79; return $?; } - [[ ! -e "$work_dir" ]] || { die "TOPOLOGY_RUNTIME_MATERIAL_REMAINS" 79; return $?; } -} - -render_summary() { - jq -n --arg run_id "$P7_RUN_ID" --arg compose_sha256 "$(sha256 "${p6_dir}/docker-compose.runtime.yml")" --arg component_sha256 "$(sha256 "${p7_dir}/docker-compose.runtime.yml")" --argjson images "$(jq -c '.images' "$input")" \ - '{schema_version:"p7-render/v1",run_id:$run_id,compose_sha256:$compose_sha256,hermes_component_sha256:$component_sha256,topology:["litellm","shell","jupyterhub","launcher","hermes-workspace"],workspace_creation:"live_dockerspawner",images:$images,content_redacted:true}' > "${artifact_dir}/p7-render-${P7_RUN_ID}.json" - chmod 600 "${artifact_dir}/p7-render-${P7_RUN_ID}.json" -} - -security_scan() { - local patterns="$1" status - shift - secure_file "$patterns" || return $? - [[ -s "$patterns" && $# -gt 0 ]] || { die "SECRET_SCAN_INPUT_INVALID" 74; return $?; } - set +e - rg --fixed-strings --files-with-matches --glob '!secret-patterns' -f "$patterns" "$@" >/dev/null 2>&1 - status=$? - set -e - case "$status" in 1) return 0 ;; 0) die "SECRET_PATTERN_MATCH" 75; return $? ;; *) die "SECRET_SCAN_FAILED" 75; return $? ;; esac -} - -case "$action" in - validate-input) - if validate_shape; then write_report passed completed; else write_report failed precondition_failed input_validation; exit 1; fi - ;; - preflight) - if preflight; then write_report passed completed; else write_report failed precondition_failed preflight; exit 1; fi - ;; - render) - if preflight && render_summary; then write_report passed completed "" "$(jq -n --arg path "${artifact_dir}/p7-render-${P7_RUN_ID}.json" --arg sha "$(sha256 "${artifact_dir}/p7-render-${P7_RUN_ID}.json")" '{render:{path:$path,sha256:$sha}}')"; else write_report failed precondition_failed render; exit 1; fi - ;; - golden) - if ! preflight || ! render_summary; then write_report failed precondition_failed preflight; exit 1; fi - mkdir -p "$work_dir"; chmod 700 "$work_dir" - trap cleanup_runtime EXIT - if ! "${script_dir}/p7-prepare-runtime.py"; then write_report failed topology_prepare_failed prepare; exit 1; fi - if ! compose up -d --wait >/dev/null; then write_report failed topology_provision_failed compose; exit 1; fi - write_report passed provisioned - product_report="${work_dir}/p7-product-chain.json" - pattern_file="${work_dir}/secret-patterns" - if ! P7_PRODUCT_CONFIG_FILE="${work_dir}/config/driver-config.json" P7_PRODUCT_REPORT_FILE="$product_report" P7_SECRET_PATTERN_FILE="$pattern_file" "${script_dir}/p7-product-chain.py"; then - write_report failed golden_chain_failed product_chain; exit 1 - fi - jq -e '.schema_version == "p7-product-chain-report/v1" and .result == "passed" and .content_redacted == true and ([.checks.console_mouse,.checks.binding_payload,.checks.jupyterhub_dockerspawner,.checks.launcher_claim_activate_release,.checks.hermes_apply_probe_readiness,.checks.model_call,.checks.stream,.checks.tool,.checks.usage,.checks.owner_negative,.checks.prompt_response_absent,.checks.revoke,.checks.generation_restart,.checks.late_release,.checks.delete,.checks.zero_active_leases] | all(. == "passed"))' "$product_report" >/dev/null || { write_report failed golden_chain_failed report_validation; exit 1; } - retained_product="${artifact_dir}/p7-product-${P7_RUN_ID}.json" - cp "$product_report" "$retained_product"; chmod 600 "$retained_product" - scan_roots=("$retained_product") - while IFS= read -r root; do [[ -e "$root" && ! -L "$root" ]] && scan_roots+=("$root"); done < <(jq -r '.scan_roots[]' "$product_report") - if ! security_scan "$pattern_file" "${scan_roots[@]}"; then write_report failed security_scan_failed secret_scan; exit 1; fi - product_sha="$(sha256 "$retained_product")" - checks="$(jq -c '.checks' "$retained_product")" - cleanup_runtime; assert_cleanup - trap - EXIT - write_report passed completed "" "$(jq -n --arg product "$retained_product" --arg product_sha "$product_sha" --argjson checks "$checks" '{checks:$checks,product_report:{path:$product,sha256:$product_sha},cleanup:{result:"passed",resources:"absent"}}')" - ;; - cleanup) - cleanup_runtime; assert_cleanup; write_report passed completed "" '{"cleanup":{"result":"passed","resources":"absent"}}' - ;; -esac diff --git a/docker_hermes/p7/scripts/test-p7-gates.sh b/docker_hermes/p7/scripts/test-p7-gates.sh index 372b2b3..4bb9e69 100755 --- a/docker_hermes/p7/scripts/test-p7-gates.sh +++ b/docker_hermes/p7/scripts/test-p7-gates.sh @@ -1,113 +1,27 @@ #!/usr/bin/env bash +# Static Hermes product contract checks. This test deliberately performs no +# Docker, network, credential, or historical-evidence operation. set -euo pipefail root="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)" -runner="${root}/docker_hermes/p7/scripts/p7-runner.sh" -tmp="$(mktemp -d "${TMPDIR:-/tmp}/p7-gates.XXXXXX")" -chmod 700 "$tmp" -trap 'rm -rf "$tmp"' EXIT +dockerfile="${root}/docker_hermes/hermes.Dockerfile" +compose_file="${root}/docker_hermes/demo/docker-compose.yml" # The source pin must not regress to a moving branch. This is intentionally a # static gate: it does not contact an upstream repository or Docker daemon. -! rg -q 'git clone --depth 1 --branch main' "${root}/docker_hermes/hermes.Dockerfile" -rg -q 'ARG HERMES_SOURCE_COMMIT' "${root}/docker_hermes/hermes.Dockerfile" -rg -q 'git fetch --depth 1 origin "\$HERMES_SOURCE_COMMIT"' "${root}/docker_hermes/hermes.Dockerfile" -rg -q 'org.opencontainers.image.revision' "${root}/docker_hermes/hermes.Dockerfile" -rg -q 'ARG HERMES_BUILD_BASE_IMAGE' "${root}/docker_hermes/hermes.Dockerfile" -rg -q 'io.labnow.hermes.runtime-base' "${root}/docker_hermes/hermes.Dockerfile" -rg -q 'pull_policy: never' "${root}/docker_hermes/p7/docker-compose.runtime.yml" -rg -q 'P6_LAUNCHER_BASE_DIGEST' "${root}/docker_hermes/p7/launcher-overlay.Dockerfile" -rg -q 'io.labnow.p7.launcher-base' "${root}/docker_hermes/p7/launcher-overlay.Dockerfile" -rg -q 'COPY --from=launcher src/labnow-launcher/devhub_launcher' "${root}/docker_hermes/p7/launcher-overlay.Dockerfile" -! rg -n --glob '!**/test-p7-gates.sh' 'OPENAI_API_KEY:|DEEPSEEK_API_KEY:|:latest' "${root}/docker_hermes/p7" -! rg -q 'HERMES_PRODUCT_CHAIN_NOT_AVAILABLE' "$runner" -rg -q 'p7-product-chain.py' "$runner" -! rg -q -- '--max-turns' "${root}/docker_hermes/p7/scripts/p7-product-chain.py" -rg -q '"-t", "terminal"' "${root}/docker_hermes/p7/scripts/p7-product-chain.py" -rg -q 'core.quotePath=false diff --name-only' "$runner" -python3 -m py_compile \ - "${root}/docker_hermes/p7/scripts/p7-prepare-runtime.py" \ - "${root}/docker_hermes/p7/scripts/p7-product-chain.py" +! rg -q 'git clone --depth 1 --branch main' "$dockerfile" +rg -q '^ARG HERMES_SOURCE_REPOSITORY=' "$dockerfile" +rg -q '^ARG HERMES_SOURCE_COMMIT=' "$dockerfile" +rg -q 'git fetch --depth 1 origin "\$HERMES_SOURCE_COMMIT"' "$dockerfile" +rg -Fq 'test "$(git rev-parse HEAD)" = "$HERMES_SOURCE_COMMIT"' "$dockerfile" +rg -q 'org.opencontainers.image.source' "$dockerfile" +rg -q 'org.opencontainers.image.revision' "$dockerfile" +rg -q 'io.labnow.hermes.build-base' "$dockerfile" +rg -q 'io.labnow.hermes.runtime-base' "$dockerfile" -input="$tmp/invalid.json" -printf '%s\n' '{"schema_version":"p7-inputs/v1"}' > "$input"; chmod 600 "$input" -if P7_ARTIFACTS_DIR="$tmp/artifacts" P7_WORK_DIR="$tmp/work" "$runner" --input "$input" --validate-input >/dev/null 2>&1; then - printf '%s\n' 'P7 invalid input was accepted' >&2; exit 1 -fi +rg -q '^ image: "\$\{HERMES_IMAGE:\?set a fixed local Hermes image\}"$' "$compose_file" +rg -q '^ pull_policy: never$' "$compose_file" +rg -q '^ - "\$\{HERMES_DASHBOARD_PUBLISH_HOST:-127\.0\.0\.1\}:\$\{HERMES_DASHBOARD_PORT:-9119\}:\$\{HERMES_DASHBOARD_PORT:-9119\}"$' "$compose_file" +rg -q '^ command: \["start-hermes\.sh", "all"\]$' "$compose_file" -valid="$tmp/valid.json" -jq ' - .repositories.lab_dev.commit = "0123456789abcdef0123456789abcdef01234567" - | .repositories.lab_dev.runtime_commit = .repositories.lab_dev.commit - | .repositories.labnow_shell.commit = "0123456789abcdef0123456789abcdef01234567" - | .repositories.labnow_shell.runtime_commit = .repositories.labnow_shell.commit - | .repositories.hermes_source.repository = "https://example.invalid/hermes.git" - | .repositories.hermes_source.commit = "0123456789abcdef0123456789abcdef01234567" - | .images |= with_entries(.value.image_id = "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa") - | .support_images |= with_entries(.value.image_id = "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa") - | .images.hermes.ref = "quay.io/labnow/hermes:p7-0123456789ab" - | .images.hermes.repo_digest = "quay.io/labnow/hermes@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" - | .images.hermes.source_commit = .repositories.hermes_source.commit - | .images.litellm.ref = "quay.io/labnow/litellm@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" - | .images.litellm.repo_digest = .images.litellm.ref - | .images.workspace.ref = "quay.io/labnow/labnow-open@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" - | .images.workspace.repo_digest = .images.workspace.ref - | .images.workspace.source_commit = .repositories.labnow_open.runtime_commit - | .images.shell.ref = "quay.io/labnow/labnow-shell@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" - | .images.shell.repo_digest = .images.shell.ref - | .images.shell.source_commit = .repositories.labnow_shell.runtime_commit - | .images.launcher.ref = "quay.io/labnow/labnow-launcher@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" - | .images.launcher.repo_digest = .images.launcher.ref - | .images.launcher.source_commit = .repositories.labnow_launcher.runtime_commit - | .support_images.postgres.repo_digest = "postgres@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" - | .support_images.redis.repo_digest = "redis@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" - | .support_images.nginx.repo_digest = "nginx@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" - | .base_images.build = "quay.io/labnow/node@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" - | .base_images.runtime = "quay.io/labnow/base@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" - | .runtime.p1_env_file = "/private/tmp/p7-test.env" -' "${root}/docker_hermes/p7/p7-inputs.example.json" > "$valid" -chmod 600 "$valid" -P7_ARTIFACTS_DIR="$tmp/artifacts" P7_WORK_DIR="$tmp/work" "$runner" --input "$valid" --validate-input >/dev/null - -# A failed repository gate must stop before any Docker preflight or topology -# action. The runner executes preflight from an `if` condition, where Bash -# does not propagate `errexit` into nested functions; keep this explicit -# fixture so a reported dirty tree cannot accidentally continue provisioning. -mkdir -p "$tmp/bin" "$tmp/dirty-repo/.git" -jq --arg path "$tmp/dirty-repo" '.repositories.lab_dev.path = $path' "$valid" > "$tmp/dirty.json" -chmod 600 "$tmp/dirty.json" -cat > "$tmp/bin/git" <<'SH' -#!/usr/bin/env bash -set -euo pipefail -case "$*" in - *"rev-parse HEAD"*) printf '%s\n' '0123456789abcdef0123456789abcdef01234567' ;; - *"status --porcelain=v1 --untracked-files=no"*) printf '%s\n' ' M tracked-file' ;; - *) exit 99 ;; -esac -SH -cat > "$tmp/bin/docker" </dev/null 2>&1 -dirty_status=$? -set -e -if [[ "$dirty_status" != 1 ]]; then - printf 'P7 dirty-tree preflight returned %s instead of 1\n' "$dirty_status" >&2 - exit 1 -fi -if [[ -e "$tmp/docker-was-called" ]]; then - printf '%s\n' 'P7 dirty-tree preflight reached Docker' >&2 - exit 1 -fi - -jq '.images.hermes.source_commit = "fedcba9876543210fedcba9876543210fedcba98"' "$valid" > "${valid}.mismatch" -chmod 600 "${valid}.mismatch" -if P7_ARTIFACTS_DIR="$tmp/artifacts" P7_WORK_DIR="$tmp/work" "$runner" --input "${valid}.mismatch" --validate-input >/dev/null 2>&1; then - printf '%s\n' 'P7 provenance mismatch input was accepted' >&2; exit 1 -fi -printf '%s\n' 'PASS P7 gates: source pin, local-only topology, real golden entry and invalid input fail closed.' +printf '%s\n' 'PASS P7 gates: Hermes source pin, provenance labels, explicit local image, pull policy, and loopback dashboard publication.' From 6e3a55d356b9cf4a40b86c4f3e9aab1300b2a453 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Thu, 20 Aug 2026 08:04:33 +0800 Subject: [PATCH 77/87] =?UTF-8?q?test(openclaw):=20=E4=BF=AE=E6=AD=A3=20P6?= =?UTF-8?q?=20Compose=20=E7=AB=AF=E5=8F=A3=E9=97=A8=E7=A6=81=E5=8C=B9?= =?UTF-8?q?=E9=85=8D?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 背景:PH-3 归档后保留的 test-p6-gates.sh 应在不启动容器的前提下断言常规 OpenClaw Compose 的稳定端口契约。 修复:端口映射在 YAML 中以双引号包裹;门禁改用与真实声明一致的模式,避免产品代码正确时发生静态误报。未改变 Dockerfile、Compose 或运行行为。 验证:bash -n docker_openclaw/p6/scripts/test-p6-gates.sh;./docker_openclaw/p6/scripts/test-p6-gates.sh(PASS P6 gates: OpenClaw image state, entrypoint, ports, and standard Compose avoid Docker socket access.);git diff --check。未启动容器。 影响面:仅 PH-3 留存的 P6 静态门禁。关联 Linear:CHE-673(In Progress)。剩余风险:历史 P6 黄金证据继续仅能通过 D-10 指定 Git 快照回读。 --- docker_openclaw/p6/scripts/test-p6-gates.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docker_openclaw/p6/scripts/test-p6-gates.sh b/docker_openclaw/p6/scripts/test-p6-gates.sh index 51d09ef..78bcc75 100755 --- a/docker_openclaw/p6/scripts/test-p6-gates.sh +++ b/docker_openclaw/p6/scripts/test-p6-gates.sh @@ -15,8 +15,8 @@ rg -q '^EXPOSE 18789 18790$' "$dockerfile" rg -q '^CMD \["start-openclaw\.sh"\]$' "$dockerfile" rg -q '^services:$' "$compose_file" rg -q '^ openclaw-gateway:$' "$compose_file" -rg -q '^ - \$\{OPENCLAW_GATEWAY_PORT:-18789\}:18789$' "$compose_file" -rg -q '^ - \$\{OPENCLAW_BRIDGE_PORT:-18790\}:18790$' "$compose_file" +rg -q '^ - "\$\{OPENCLAW_GATEWAY_PORT:-18789\}:18789"$' "$compose_file" +rg -q '^ - "\$\{OPENCLAW_BRIDGE_PORT:-18790\}:18790"$' "$compose_file" ! rg -q '/var/run/docker\.sock' "$compose_file" printf '%s\n' 'PASS P6 gates: OpenClaw image state, entrypoint, ports, and standard Compose avoid Docker socket access.' From 173b12e7d80b7afc24ff0186eb39e63bcbf53f32 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Thu, 20 Aug 2026 08:13:09 +0800 Subject: [PATCH 78/87] =?UTF-8?q?fix(litellm):=20=E5=8F=82=E6=95=B0?= =?UTF-8?q?=E5=8C=96=20Compose=20=E5=AE=9E=E4=BE=8B=E8=B5=84=E6=BA=90?= =?UTF-8?q?=E5=91=BD=E5=90=8D?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 动机与背景:LiteLLM Compose 过去将 project、外部网络和显式代理容器固定为 litellm-baseline/svc-litellm-*。不同本地实例或残留栈会发生资源冲突,且 smoke 与 PH-1 回归脚本直接依赖这些固定名称。 关键设计取舍:沿用本仓 PROFILE_ENV 惯例,不新增第二套 LITELLM_COMPOSE_PROJECT 配置源。Compose project 使用 ${PROFILE_ENV:-litellm-baseline}-svc-litellm;显式容器与外部网络使用相同前缀。保留 LITELLM_*_CONTAINER_NAME 显式覆盖入口以兼容定向测试。脚本改按 Compose service ID 查询容器;PH-1 回归脚本为每次 run 设置唯一 PROFILE_ENV。回环端口默认绑定和 pull_policy: never 未改变。 验证:执行两套无敏感占位环境的 docker compose -f docker_litellm/demo/docker-compose.litellm.yml --profile ha|migrate config --format json(PROFILE_ENV=ph4-config-a/b);HA 渲染分别为 ph4-config-a/b-svc-litellm、对应 -1/-2 容器和 -net 网络,migration 渲染对应 -migrate 容器,集合无交集。另执行 bash -n docker_litellm/demo/scripts/smoke-baseline.sh docker_litellm/demo/scripts/smoke-redis-recovery.sh docker_litellm/demo/scripts/test-secret-boundary.sh 与 git diff --check,均通过。未启动服务容器。 影响面:仅 LiteLLM demo Compose、其运行态验证脚本、PH-1 回归清理逻辑和使用说明。调用方应以 PROFILE_ENV 选择实例,避免混用 -p/COMPOSE_PROJECT_NAME。 关联 Linear:CHE-674(LLM Hub 生产加固,PH-4)。 剩余风险:本提交为配置级验证,未运行需要真实镜像和 Docker daemon 的 smoke/PH-1 容器测试;该限制符合本批不启动容器的执行边界。 --- docker_litellm/README.md | 12 ++++++- .../demo/docker-compose.litellm.yml | 21 ++++++----- docker_litellm/demo/scripts/smoke-baseline.sh | 15 +++++--- .../demo/scripts/smoke-redis-recovery.sh | 15 +++++--- .../demo/scripts/test-secret-boundary.sh | 36 ++++++++++--------- 5 files changed, 64 insertions(+), 35 deletions(-) diff --git a/docker_litellm/README.md b/docker_litellm/README.md index 1bd8818..dc28a94 100644 --- a/docker_litellm/README.md +++ b/docker_litellm/README.md @@ -47,6 +47,16 @@ cp .env.example .env docker compose --env-file .env -f docker-compose.litellm.yml --profile single up -d ``` +Compose 的项目、显式容器和外部网络均以仓库既有的 `PROFILE_ENV` 推导,默认实例为 +`litellm-baseline`:Compose project 为 `litellm-baseline-svc-litellm`,网络为 +`litellm-baseline-svc-litellm-net`。若需与另一套本地 LiteLLM 基线并行运行,在同一条命令前 +设置不同实例名;不要混用 `-p` 或 `COMPOSE_PROJECT_NAME`,以免项目名与显式容器/网络命名源分离。 + +```bash +PROFILE_ENV=litellm-dev-a docker compose --env-file .env -f docker-compose.litellm.yml --profile single up -d +PROFILE_ENV=litellm-dev-b docker compose --env-file .env -f docker-compose.litellm.yml --profile single up -d +``` + 迁移与代理启动刻意分离。标准真实验收由一个统一入口执行:它生成新的非敏感 `verification_run_id`,先失效所有旧输入/最终报告,再运行 migration(两次)、并发 migration job、single、HA、Redis 恢复和严格聚合;任一步失败都会停止且保留当前失败报告。 ```bash @@ -89,7 +99,7 @@ cd docker_litellm/demo `--security-check` 不读取 `.env`、不启动服务也不发送上游请求;它拒绝 inline header、secret-bearing `jq --arg`、`set -x`、Compose 上游凭据注入和 Redis 密码命令行展开,并检查 Docker Secret、0600 临时文件与退出清理约束。`test-verification-gates.sh` 验证历史 PASS 失效、前置失败报告与 dotenv 命令替换不执行;在已启动 single 栈中追加 `--with-running-stack` 会以真实 404 删除请求证明 cleanup 不会生成 PASS。 -`test-secret-boundary.sh` 是 PH-1 的无上游定向门禁:它只生成本地占位凭据,不读取 `demo/.env`,验证 Compose 渲染和容器 inspect 不含 `LITELLM_MASTER_KEY`、`DATABASE_URL`、`POSTGRES_PASSWORD` 的服务环境或凭据值,并对 inspect、`ps/argv`、容器日志、容器临时文件执行负向检查。随后它启动 LiteLLM、调用并清理一次已认证的管理端点,退出时删除本次创建的容器、卷、网络和宿主临时文件。若固定网络 `litellm-baseline-net` 已被其他栈占用,脚本会失败退出而不会复用或干扰该网络。 +`test-secret-boundary.sh` 是 PH-1 的无上游定向门禁:它只生成本地占位凭据,不读取 `demo/.env`,验证 Compose 渲染和容器 inspect 不含 `LITELLM_MASTER_KEY`、`DATABASE_URL`、`POSTGRES_PASSWORD` 的服务环境或凭据值,并对 inspect、`ps/argv`、容器日志、容器临时文件执行负向检查。随后它启动 LiteLLM、调用并清理一次已认证的管理端点,退出时删除本次创建的容器、卷、网络和宿主临时文件。脚本为每次 run 设置唯一 `PROFILE_ENV`,若对应实例网络已存在则失败退出而不会复用或干扰该网络。 `smoke-redis-recovery.sh` 在已启动的 HA 栈中临时断开 Redis 网络端点,验证两个副本的有界认证探针均失败,再恢复 `redis` alias、等待 breaker 窗口并确认认证后的 `GET /v1/models` 恢复。它有独立的恢复 trap,不会让故障测试影响主 smoke 的资源清理。`aggregate-verification-summary.sh` 将 migration、single、HA 与 Redis 独立报告组合为不含密钥、密码、提示词和响应正文的最终 JSON 摘要。 diff --git a/docker_litellm/demo/docker-compose.litellm.yml b/docker_litellm/demo/docker-compose.litellm.yml index ed40fb4..3847618 100644 --- a/docker_litellm/demo/docker-compose.litellm.yml +++ b/docker_litellm/demo/docker-compose.litellm.yml @@ -1,4 +1,7 @@ -name: litellm-baseline +# Keep the project, explicit container names, and external network names in +# the same instance namespace. This follows the repository PROFILE_ENV +# convention and allows multiple local LiteLLM stacks to coexist. +name: ${PROFILE_ENV:-litellm-baseline}-svc-litellm x-litellm-common: &litellm-common image: ${LITELLM_IMAGE:?set LITELLM_IMAGE to the locally built quay.io/labnow/litellm image} @@ -33,7 +36,7 @@ x-litellm-common: &litellm-common redis: condition: service_healthy networks: - - litellm-baseline-net + - litellm-net healthcheck: test: ["CMD-SHELL", "python3 -c \"import urllib.request; urllib.request.urlopen('http://127.0.0.1:4000/health/readiness', timeout=3)\""] interval: 10s @@ -48,7 +51,7 @@ services: # Prisma's migration table makes repeated one-shot jobs idempotent. litellm-migrate: <<: *litellm-common - container_name: ${LITELLM_MIGRATE_CONTAINER_NAME:-svc-litellm-migrate} + container_name: ${LITELLM_MIGRATE_CONTAINER_NAME:-${PROFILE_ENV:-litellm-baseline}-svc-litellm-migrate} profiles: ["migrate"] command: ["/opt/utils/start-litellm.sh", "python3", "/opt/utils/run-migration-locked.py", "--config", "config.migrate.yaml", "--skip_server_startup", "--enforce_prisma_migration_check"] healthcheck: @@ -71,7 +74,7 @@ services: volumes: - litellm_postgres_data:/var/lib/postgresql/data networks: - - litellm-baseline-net + - litellm-net healthcheck: test: ["CMD-SHELL", "pg_isready -U $$POSTGRES_USER -d $$POSTGRES_DB"] interval: 5s @@ -95,7 +98,7 @@ services: volumes: - litellm_redis_data:/data networks: - - litellm-baseline-net + - litellm-net healthcheck: test: ["CMD-SHELL", "REDISCLI_AUTH=\"$$(cat /run/secrets/redis_password)\" redis-cli --no-auth-warning ping | grep -qx PONG"] interval: 5s @@ -104,14 +107,14 @@ services: litellm-1: <<: *litellm-common - container_name: ${LITELLM_1_CONTAINER_NAME:-svc-litellm-1} + container_name: ${LITELLM_1_CONTAINER_NAME:-${PROFILE_ENV:-litellm-baseline}-svc-litellm-1} profiles: ["single", "ha"] ports: - "${LITELLM_PUBLISH_HOST:-127.0.0.1}:${LITELLM_1_PORT:-4000}:4000" litellm-2: <<: *litellm-common - container_name: ${LITELLM_2_CONTAINER_NAME:-svc-litellm-2} + container_name: ${LITELLM_2_CONTAINER_NAME:-${PROFILE_ENV:-litellm-baseline}-svc-litellm-2} profiles: ["ha"] ports: - "${LITELLM_PUBLISH_HOST:-127.0.0.1}:${LITELLM_2_PORT:-4001}:4000" @@ -129,5 +132,5 @@ secrets: environment: REDIS_PASSWORD networks: - litellm-baseline-net: - name: litellm-baseline-net + litellm-net: + name: ${PROFILE_ENV:-litellm-baseline}-svc-litellm-net diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh index 0464992..1083705 100755 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -7,7 +7,8 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" DEMO_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)" source "${SCRIPT_DIR}/verification-lib.sh" ENV_FILE="${LITELLM_SMOKE_ENV_FILE:-${DEMO_DIR}/.env}" -export COMPOSE_PROJECT_NAME="${LITELLM_COMPOSE_PROJECT:-litellm-baseline}" +: "${PROFILE_ENV:=litellm-baseline}" +export PROFILE_ENV MODE="single" SECURITY_CHECK=false CLEANUP_NEGATIVE_TEST=false @@ -394,9 +395,15 @@ assert_migration_evidence() { } runtime_security_check() { - local logs_file="$tmpdir/litellm-logs.txt" inspect_file="$tmpdir/inspect.json" process_file="$tmpdir/redis-processes.txt" db_content_file="$tmpdir/spendlog-db-content.txt" - docker inspect svc-litellm-1 > "$inspect_file" - if [[ "$MODE" == ha ]]; then docker inspect svc-litellm-2 >> "$inspect_file"; fi + local logs_file="$tmpdir/litellm-logs.txt" inspect_file="$tmpdir/inspect.json" process_file="$tmpdir/redis-processes.txt" db_content_file="$tmpdir/spendlog-db-content.txt" litellm_1_container litellm_2_container + litellm_1_container="$(docker compose --env-file "$ENV_FILE" -f "$DEMO_DIR/docker-compose.litellm.yml" ps -q litellm-1)" + [[ -n "$litellm_1_container" ]] || { echo "litellm-1 container not found" >&2; return 1; } + docker inspect "$litellm_1_container" > "$inspect_file" + if [[ "$MODE" == ha ]]; then + litellm_2_container="$(docker compose --env-file "$ENV_FILE" -f "$DEMO_DIR/docker-compose.litellm.yml" ps -q litellm-2)" + [[ -n "$litellm_2_container" ]] || { echo "litellm-2 container not found" >&2; return 1; } + docker inspect "$litellm_2_container" >> "$inspect_file" + fi docker compose --env-file "$ENV_FILE" -f "$DEMO_DIR/docker-compose.litellm.yml" logs --no-color litellm-1 > "$logs_file" 2>&1 if [[ "$MODE" == ha ]]; then docker compose --env-file "$ENV_FILE" -f "$DEMO_DIR/docker-compose.litellm.yml" logs --no-color litellm-2 >> "$logs_file" 2>&1; fi docker exec "$(docker compose --env-file "$ENV_FILE" -f "$DEMO_DIR/docker-compose.litellm.yml" ps -q redis)" ps -eo args > "$process_file" diff --git a/docker_litellm/demo/scripts/smoke-redis-recovery.sh b/docker_litellm/demo/scripts/smoke-redis-recovery.sh index b0cd286..9eafcb2 100755 --- a/docker_litellm/demo/scripts/smoke-redis-recovery.sh +++ b/docker_litellm/demo/scripts/smoke-redis-recovery.sh @@ -81,8 +81,12 @@ probe() { } runtime_security_check() { + local litellm_1_container litellm_2_container + litellm_1_container="$(docker compose --env-file "$env_file" -f "$demo_dir/docker-compose.litellm.yml" ps -q litellm-1)" + litellm_2_container="$(docker compose --env-file "$env_file" -f "$demo_dir/docker-compose.litellm.yml" ps -q litellm-2)" + [[ -n "$litellm_1_container" && -n "$litellm_2_container" ]] || return 1 docker inspect "$container" > "$tmpdir/redis-inspect.json" - docker inspect svc-litellm-1 svc-litellm-2 > "$tmpdir/litellm-inspect.json" + docker inspect "$litellm_1_container" "$litellm_2_container" > "$tmpdir/litellm-inspect.json" docker exec "$container" ps -eo args > "$tmpdir/redis-processes.txt" ! rg -q -- '--requirepass[[:space:]]+[^[:space:]]+' "$tmpdir/redis-processes.txt" && ! rg -q 'UPSTREAM_(API_KEY|BASE_URL|MODEL)=' "$tmpdir/litellm-inspect.json" && @@ -94,10 +98,13 @@ container="$(docker compose --env-file "$env_file" -f "$demo_dir/docker-compose. [[ -n "$container" ]] || { phase="redis_not_found"; exit 1; } network="$(docker inspect -f '{{range $name, $_ := .NetworkSettings.Networks}}{{$name}}{{end}}' "$container")" [[ -n "$network" ]] || { phase="network_not_found"; exit 1; } +litellm_1_container="$(docker compose --env-file "$env_file" -f "$demo_dir/docker-compose.litellm.yml" ps -q litellm-1)" +litellm_2_container="$(docker compose --env-file "$env_file" -f "$demo_dir/docker-compose.litellm.yml" ps -q litellm-2)" +[[ -n "$litellm_1_container" && -n "$litellm_2_container" ]] || { phase="litellm_not_found"; exit 1; } phase="disconnect" docker network disconnect "$network" "$container" -if probe svc-litellm-1 >/dev/null 2>&1 || probe svc-litellm-2 >/dev/null 2>&1; then +if probe "$litellm_1_container" >/dev/null 2>&1 || probe "$litellm_2_container" >/dev/null 2>&1; then phase="probe_unexpectedly_succeeded" exit 1 fi @@ -105,8 +112,8 @@ fi phase="recover" docker network connect --alias redis "$network" "$container" sleep $((recovery_timeout + 1)) -probe svc-litellm-1 -probe svc-litellm-2 +probe "$litellm_1_container" +probe "$litellm_2_container" # This is an authenticated LiteLLM call after recovery, not only a socket PING. curl --silent --show-error --fail --max-time 20 --request GET "http://${publish_host}:${litellm_1_port}/v1/models" \ diff --git a/docker_litellm/demo/scripts/test-secret-boundary.sh b/docker_litellm/demo/scripts/test-secret-boundary.sh index a2aa894..c0717ec 100755 --- a/docker_litellm/demo/scripts/test-secret-boundary.sh +++ b/docker_litellm/demo/scripts/test-secret-boundary.sh @@ -10,8 +10,10 @@ compose_file="${demo_dir}/docker-compose.litellm.yml" image_ref="${LITELLM_SECRET_BOUNDARY_IMAGE:-quay.io/labnow/litellm:1.97.0-ead62528e607}" run_id="$(python3 -c 'import secrets; print(secrets.token_hex(8))')" project="ph1-secret-boundary-${run_id}" -litellm_container="ph1-secret-boundary-${run_id}-litellm-1" -litellm_peer_container="ph1-secret-boundary-${run_id}-litellm-2" +compose_project="${project}-svc-litellm" +network_name="${project}-svc-litellm-net" +litellm_container="${project}-svc-litellm-1" +litellm_peer_container="${project}-svc-litellm-2" publish_port="${LITELLM_SECRET_BOUNDARY_PORT:-4100}" tmpdir="$(mktemp -d "${TMPDIR:-/tmp}/litellm-secret-boundary.XXXXXX")" env_file="${tmpdir}/runtime.env" @@ -39,14 +41,14 @@ cleanup() { local exit_code=$? trap - EXIT if [[ "$started" == true ]]; then - docker compose --env-file "$env_file" -p "$project" -f "$compose_file" --profile single down -v --remove-orphans >/dev/null 2>&1 || exit_code=1 + docker compose --env-file "$env_file" -f "$compose_file" --profile single down -v --remove-orphans >/dev/null 2>&1 || exit_code=1 fi rm -rf "$tmpdir" if docker ps -a --format '{{.Names}}' | rg -q "^${litellm_container}$|^${litellm_peer_container}$"; then echo "FAIL cleanup: PH-1 LiteLLM container remains" >&2 exit_code=1 fi - if docker network inspect litellm-baseline-net >/dev/null 2>&1; then + if docker network inspect "$network_name" >/dev/null 2>&1; then echo "FAIL cleanup: PH-1 network remains" >&2 exit_code=1 fi @@ -66,14 +68,14 @@ need jq need rg need openssl -# The Compose file keeps its legacy fixed network name. Refuse to attach a -# boundary test to any pre-existing stack instead of disturbing its network. -if docker network inspect litellm-baseline-net >/dev/null 2>&1; then - echo "refusing to reuse existing litellm-baseline-net" >&2 +# Refuse to attach the boundary test to an existing instance-specific network. +if docker network inspect "$network_name" >/dev/null 2>&1; then + echo "refusing to reuse existing $network_name" >&2 exit 2 fi umask 077 +export PROFILE_ENV="$project" master_key="sk-$(openssl rand -hex 24)" postgres_password="$(openssl rand -hex 24)" redis_password="$(openssl rand -hex 24)" @@ -81,7 +83,7 @@ printf 'LITELLM_IMAGE=%s\nLITELLM_MASTER_KEY=%s\nPOSTGRES_USER=litellm\nPOSTGRES "$image_ref" "$master_key" "$postgres_password" "$redis_password" "$litellm_container" "$litellm_peer_container" "$publish_port" > "$env_file" chmod 600 "$env_file" -docker compose --env-file "$env_file" -p "$project" -f "$compose_file" --profile single config --format json | jq -e ' +docker compose --env-file "$env_file" -f "$compose_file" --profile single config --format json | jq -e ' . as $config | ([.services | to_entries[] | select(.key == "litellm-1" or .key == "litellm-2" or .key == "litellm-migrate") | .value.environment // {} | keys[] | select(. == "LITELLM_MASTER_KEY" or . == "DATABASE_URL" or . == "POSTGRES_PASSWORD")] | length == 0) and ($config.services.postgres.environment | has("POSTGRES_PASSWORD") | not) @@ -90,8 +92,8 @@ compose_config_result="passed" echo "PASS compose config: service environment omits LITELLM_MASTER_KEY, DATABASE_URL, and POSTGRES_PASSWORD." started=true -docker compose --env-file "$env_file" -p "$project" -f "$compose_file" up -d --wait postgres redis >/dev/null -docker compose --env-file "$env_file" -p "$project" -f "$compose_file" --profile migrate run --rm --no-deps litellm-migrate >"$migration_output" 2>&1 +docker compose --env-file "$env_file" -f "$compose_file" up -d --wait postgres redis >/dev/null +docker compose --env-file "$env_file" -f "$compose_file" --profile migrate run --rm --no-deps litellm-migrate >"$migration_output" 2>&1 chmod 600 "$migration_output" if rg -Fq -- "$master_key" "$migration_output" \ || rg -Fq -- "$postgres_password" "$migration_output" \ @@ -99,7 +101,7 @@ if rg -Fq -- "$master_key" "$migration_output" \ echo "FAIL logs: credential value is present in migration output" >&2 exit 1 fi -docker compose --env-file "$env_file" -p "$project" -f "$compose_file" --profile single up -d litellm-1 >/dev/null +docker compose --env-file "$env_file" -f "$compose_file" --profile single up -d litellm-1 >/dev/null for attempt in $(seq 1 60); do if curl --silent --fail --max-time 3 "http://127.0.0.1:${publish_port}/health/readiness" | jq -e '.status == "healthy" and .db == "connected"' >/dev/null; then break @@ -127,11 +129,11 @@ assert_metadata_boundary() { assert_metadata_boundary "$litellm_container" LITELLM_MASTER_KEY "$master_key" assert_metadata_boundary "$litellm_container" DATABASE_URL "$postgres_password" -assert_metadata_boundary "${project}-postgres-1" POSTGRES_PASSWORD "$postgres_password" +assert_metadata_boundary "${compose_project}-postgres-1" POSTGRES_PASSWORD "$postgres_password" inspect_result="passed" echo "PASS inspect: no management key, database URL, or PostgreSQL password value in container metadata." -for container in "$litellm_container" "${project}-postgres-1" "${project}-redis-1"; do +for container in "$litellm_container" "${compose_project}-postgres-1" "${compose_project}-redis-1"; do if docker top "$container" -eo args | rg -Fq -- "$master_key" \ || docker top "$container" -eo args | rg -Fq -- "$postgres_password" \ || docker top "$container" -eo args | rg -Fq -- "$redis_password"; then @@ -142,9 +144,9 @@ done argv_result="passed" echo "PASS ps/argv: no credential values in container command lines." -if docker compose --env-file "$env_file" -p "$project" -f "$compose_file" --profile single logs --no-color | rg -Fq -- "$master_key" \ - || docker compose --env-file "$env_file" -p "$project" -f "$compose_file" --profile single logs --no-color | rg -Fq -- "$postgres_password" \ - || docker compose --env-file "$env_file" -p "$project" -f "$compose_file" --profile single logs --no-color | rg -Fq -- "$redis_password"; then +if docker compose --env-file "$env_file" -f "$compose_file" --profile single logs --no-color | rg -Fq -- "$master_key" \ + || docker compose --env-file "$env_file" -f "$compose_file" --profile single logs --no-color | rg -Fq -- "$postgres_password" \ + || docker compose --env-file "$env_file" -f "$compose_file" --profile single logs --no-color | rg -Fq -- "$redis_password"; then echo "FAIL logs: credential value is present in Compose logs" >&2 exit 1 fi From 05e4c9ef4fccaa26ae367c6ec0011959c1684cdc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Thu, 20 Aug 2026 08:14:42 +0800 Subject: [PATCH 79/87] =?UTF-8?q?refactor(litellm):=20=E6=8F=90=E5=8F=96?= =?UTF-8?q?=E5=8F=AF=E7=A7=BB=E6=A4=8D=E6=9D=83=E9=99=90=E6=A3=80=E6=9F=A5?= =?UTF-8?q?=20helper?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 动机与背景:smoke-baseline.sh 与 smoke-redis-recovery.sh 分别内联 BSD stat -f 与 GNU/Linux stat -c 的权限检查,重复实现使错误信息和平台兼容策略可能漂移。 关键设计取舍:在既有 verification-lib.sh 集中提供 verification_file_mode 与 verification_assert_file_mode,按 uname -s 明确分派 BSD 系列到 stat -f %Lp、Linux 到 stat -c %a;未知平台失败关闭并给出平台名,不猜测 stat 语义。三个调用点保留原有 0600 安全断言和失败退出行为。 验证:在本机 Darwin/BSD stat 上,以只读 /dev/null 执行 source docker_litellm/demo/scripts/verification-lib.sh; verification_file_mode; verification_assert_file_mode,原生和 helper 均输出 mode=666 且断言通过。Linux 分支以静态审查确认 helper 的 Linux case 含 stat -c %a;未在本机实测,原因是当前执行主机为 Darwin,且本批明确不启动 Linux 容器。另执行重复 stat 搜索、bash -n docker_litellm/demo/scripts/verification-lib.sh docker_litellm/demo/scripts/smoke-baseline.sh docker_litellm/demo/scripts/smoke-redis-recovery.sh 与 git diff --check,均通过。 影响面:仅 LiteLLM 验证脚本的宿主机临时凭据/header 文件权限检查;不改变 Compose、镜像或运行服务。 关联 Linear:CHE-674(LLM Hub 生产加固,PH-4)。 剩余风险:Linux stat 分支仅做静态审查,待 Linux CI 或后续允许的 Linux 宿主验证补充运行态证据。 --- docker_litellm/demo/scripts/smoke-baseline.sh | 6 ++--- .../demo/scripts/smoke-redis-recovery.sh | 2 +- .../demo/scripts/verification-lib.sh | 26 +++++++++++++++++++ 3 files changed, 29 insertions(+), 5 deletions(-) diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh index 1083705..5ad00a1 100755 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ b/docker_litellm/demo/scripts/smoke-baseline.sh @@ -199,9 +199,7 @@ write_private_value() { } assert_private_file() { - local mode - mode="$(stat -f '%Lp' "$1" 2>/dev/null || stat -c '%a' "$1")" - [[ "$mode" == "600" ]] || { echo "temporary secret file is not 0600: $1" >&2; exit 1; } + verification_assert_file_mode "$1" 600 "temporary secret file" || exit 1 } make_header_file() { @@ -419,7 +417,7 @@ runtime_security_check() { ! rg -q --file "$tmpdir/tool-marker" "$db_content_file" && ! rg -q --file "$tmpdir/response-marker" "$db_content_file" && ! git ls-files -z | xargs -0 rg -n --pcre2 '(?:sk-|Bearer[[:space:]]+)[A-Za-z0-9_-]{24,}' -- >/dev/null 2>&1 && - [[ "$(stat -f '%Lp' "$admin_headers" 2>/dev/null || stat -c '%a' "$admin_headers")" == "600" ]] + verification_assert_file_mode "$admin_headers" 600 "LiteLLM administration header" } make_key_payload() { diff --git a/docker_litellm/demo/scripts/smoke-redis-recovery.sh b/docker_litellm/demo/scripts/smoke-redis-recovery.sh index 9eafcb2..132ab12 100755 --- a/docker_litellm/demo/scripts/smoke-redis-recovery.sh +++ b/docker_litellm/demo/scripts/smoke-redis-recovery.sh @@ -91,7 +91,7 @@ runtime_security_check() { ! rg -q -- '--requirepass[[:space:]]+[^[:space:]]+' "$tmpdir/redis-processes.txt" && ! rg -q 'UPSTREAM_(API_KEY|BASE_URL|MODEL)=' "$tmpdir/litellm-inspect.json" && rg -q '/run/secrets/redis_password' "$tmpdir/redis-inspect.json" && - [[ "$(stat -f '%Lp' "$headers_file" 2>/dev/null || stat -c '%a' "$headers_file")" == "600" ]] + verification_assert_file_mode "$headers_file" 600 "Redis recovery authorization header" } container="$(docker compose --env-file "$env_file" -f "$demo_dir/docker-compose.litellm.yml" ps -q redis)" diff --git a/docker_litellm/demo/scripts/verification-lib.sh b/docker_litellm/demo/scripts/verification-lib.sh index 9b88243..8a9d358 100755 --- a/docker_litellm/demo/scripts/verification-lib.sh +++ b/docker_litellm/demo/scripts/verification-lib.sh @@ -14,6 +14,32 @@ verification_env() { awk -F= -v name="$name" '$1 == name {sub(/^[^=]*=/, ""); print; exit}' "$verification_environment_file" } +verification_file_mode() { + local file="$1" platform + platform="$(uname -s)" + case "$platform" in + Darwin|FreeBSD|OpenBSD|NetBSD|DragonFly) + stat -f '%Lp' "$file" + ;; + Linux) + stat -c '%a' -- "$file" + ;; + *) + echo "unsupported platform for portable file mode check: $platform" >&2 + return 2 + ;; + esac +} + +verification_assert_file_mode() { + local file="$1" expected_mode="$2" description="$3" actual_mode + actual_mode="$(verification_file_mode "$file")" || return $? + [[ "$actual_mode" == "$expected_mode" ]] || { + echo "$description has mode $actual_mode, expected $expected_mode: $file" >&2 + return 1 + } +} + verification_invalidate_report() { local report="$1" mkdir -p "$(dirname "$report")" From d9f13918d087d8079ef211e882041f0ae5a57848 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Thu, 20 Aug 2026 08:15:38 +0800 Subject: [PATCH 80/87] =?UTF-8?q?fix(litellm):=20=E5=8E=9F=E5=AD=90?= =?UTF-8?q?=E5=86=99=E5=85=A5=20standalone=20=E9=85=8D=E7=BD=AE?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 动机与背景:start-litellm.sh 仅使用 set -eu,standalone fallback 直接重定向写入 config.yaml。若写入中断,可能留下部分内容或权限不符合凭据相邻配置的最小权限原则。 关键设计取舍:启用 set -Eeuo pipefail,并安装只清理本次 mktemp 路径的 EXIT trap。fallback 以 umask 077 在目标 config.yaml 同目录创建临时文件,显式 chmod 600,写完后以 mv 原子替换,再清空清理句柄;失败时 trap 删除尚未移动的临时文件。Compose 挂载既有只读 config.yaml 的主路径未变。 验证:执行 bash -n docker_litellm/work/start-litellm.sh;静态检查确认严格模式、EXIT trap、umask 077、同目录 config.yaml.tmp.XXXXXX、chmod 600、mv -f 和失败清理路径均存在;执行 git diff --check,均通过。未启动 LiteLLM 或其他服务容器。 影响面:仅无 Compose 挂载 config.yaml 时的 backwards-compatible standalone fallback;常规 P1 Compose 启动继续使用只读挂载配置。 关联 Linear:CHE-674(LLM Hub 生产加固,PH-4)。 剩余风险:在本批“不启动服务容器”的限制下,未对中断时序做运行态故障注入;实现采用同目录临时文件和 POSIX rename/mv 原子替换语义。 --- docker_litellm/work/start-litellm.sh | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/docker_litellm/work/start-litellm.sh b/docker_litellm/work/start-litellm.sh index 1df7054..95f24a3 100755 --- a/docker_litellm/work/start-litellm.sh +++ b/docker_litellm/work/start-litellm.sh @@ -1,5 +1,16 @@ #!/usr/bin/env bash -set -eu +set -Eeuo pipefail + +config_tmp="" +cleanup() { + local exit_code=$? + trap - EXIT + if [[ -n "$config_tmp" && -e "$config_tmp" ]]; then + rm -f -- "$config_tmp" + fi + return "$exit_code" +} +trap cleanup EXIT # Setup workspace directory HOME_LITELLM="${HOME_LITELLM:-/opt/litellm}" @@ -44,12 +55,18 @@ export STORE_PROMPTS_IN_SPEND_LOGS="${STORE_PROMPTS_IN_SPEND_LOGS:-false}" # backwards-compatible standalone use. if [ ! -f "config.yaml" ]; then echo "Creating default config.yaml..." - cat < config.yaml + umask 077 + config_tmp="$(mktemp "${HOME_LITELLM}/config.yaml.tmp.XXXXXX")" + chmod 600 "$config_tmp" + cat <<'EOF' > "$config_tmp" model_list: - model_name: gpt-3.5-turbo litellm_params: model: gpt-3.5-turbo EOF + chmod 600 "$config_tmp" + mv -f -- "$config_tmp" config.yaml + config_tmp="" fi # If no arguments are passed, start litellm proxy with defaults From f7a0916ec423bb5a611f2eae424bbe1132520700 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Thu, 20 Aug 2026 08:16:21 +0800 Subject: [PATCH 81/87] =?UTF-8?q?fix(litellm):=20=E7=A6=81=E6=AD=A2=20Comp?= =?UTF-8?q?ose=20=E9=9A=90=E5=BC=8F=E6=8B=89=E5=8F=96=E9=95=9C=E5=83=8F?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 动机与背景:PH-4 的 Compose 参数化验收要求保留 pull_policy: never。对链式基点的完整 Compose 复核发现该安全约束未在 LiteLLM、PostgreSQL 与 Redis 服务渲染中声明,可能允许本地验证静默拉取可变外部状态。 关键设计取舍:在共享 LiteLLM service extension 及 PostgreSQL、Redis 两个独立服务上统一声明 pull_policy: never;不改变已固定的镜像引用、端口、profiles、卷或网络结构。 验证:使用无敏感占位环境分别以 PROFILE_ENV=ph4-config-a 与 PROFILE_ENV=ph4-config-b 执行 docker compose -f docker_litellm/demo/docker-compose.litellm.yml --profile ha|migrate config --format json;每份渲染的 services 均为 pull_policy=never,且 project、显式容器、网络名在两个实例间无交集。执行 git diff --check 通过;未启动服务容器。 影响面:LiteLLM demo 仅在本地不存在镜像时失败并要求调用者显式构建/准备所需镜像,避免静默拉取;不涉及镜像 digest 或 CI(PH-2 边界)。 关联 Linear:CHE-674(LLM Hub 生产加固,PH-4)。 剩余风险:本提交为 Compose 渲染验证,未运行服务容器;镜像可复现性、digest 与 CI 门禁仍由 PH-2 负责。 --- docker_litellm/demo/docker-compose.litellm.yml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docker_litellm/demo/docker-compose.litellm.yml b/docker_litellm/demo/docker-compose.litellm.yml index 3847618..b710c0f 100644 --- a/docker_litellm/demo/docker-compose.litellm.yml +++ b/docker_litellm/demo/docker-compose.litellm.yml @@ -5,6 +5,7 @@ name: ${PROFILE_ENV:-litellm-baseline}-svc-litellm x-litellm-common: &litellm-common image: ${LITELLM_IMAGE:?set LITELLM_IMAGE to the locally built quay.io/labnow/litellm image} + pull_policy: never restart: "no" environment: TZ: ${TZ:-Asia/Hong_Kong} @@ -64,6 +65,7 @@ services: postgres: image: postgres:17-alpine@sha256:742f40ea20b9ff2ff31db5458d127452988a2164df9e17441e191f3b72252193 + pull_policy: never restart: "no" environment: POSTGRES_DB: ${POSTGRES_DB:-litellm} @@ -83,6 +85,7 @@ services: redis: image: redis:7.4-alpine@sha256:e7723ff73d963f5cc6d9c4643ea3d989527a402a319239054e9472a7fb9219a2 + pull_policy: never restart: "no" secrets: - redis_password From f2c474a382cc5bd06a4945702158f189696372ec Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Fri, 21 Aug 2026 02:33:54 +0800 Subject: [PATCH 82/87] =?UTF-8?q?chore:=20=E6=B5=8B=E8=AF=95=E7=A7=BB?= =?UTF-8?q?=E5=87=BA=E4=BA=A7=E5=93=81=E5=88=86=E6=94=AF(=E6=9C=AC?= =?UTF-8?q?=E5=9C=B0=E4=BF=9D=E7=95=99)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 按代码规范裁决,7 个 test/smoke 脚本移出版本控制,本地保留; verify-p1 等环境编排工具不属测试,保留在分支。历史提交仍可回读。 Co-Authored-By: Claude Fable 5 --- docker_hermes/p7/scripts/test-p7-gates.sh | 27 - .../scripts/test-hermes-runtime-node.sh | 37 - docker_litellm/demo/scripts/smoke-baseline.sh | 803 ------------------ .../demo/scripts/smoke-redis-recovery.sh | 130 --- .../demo/scripts/test-secret-boundary.sh | 194 ----- .../demo/scripts/test-verification-gates.sh | 187 ---- docker_openclaw/p6/scripts/test-p6-gates.sh | 22 - 7 files changed, 1400 deletions(-) delete mode 100755 docker_hermes/p7/scripts/test-p7-gates.sh delete mode 100755 docker_hermes/scripts/test-hermes-runtime-node.sh delete mode 100755 docker_litellm/demo/scripts/smoke-baseline.sh delete mode 100755 docker_litellm/demo/scripts/smoke-redis-recovery.sh delete mode 100755 docker_litellm/demo/scripts/test-secret-boundary.sh delete mode 100755 docker_litellm/demo/scripts/test-verification-gates.sh delete mode 100755 docker_openclaw/p6/scripts/test-p6-gates.sh diff --git a/docker_hermes/p7/scripts/test-p7-gates.sh b/docker_hermes/p7/scripts/test-p7-gates.sh deleted file mode 100755 index 4bb9e69..0000000 --- a/docker_hermes/p7/scripts/test-p7-gates.sh +++ /dev/null @@ -1,27 +0,0 @@ -#!/usr/bin/env bash -# Static Hermes product contract checks. This test deliberately performs no -# Docker, network, credential, or historical-evidence operation. -set -euo pipefail - -root="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)" -dockerfile="${root}/docker_hermes/hermes.Dockerfile" -compose_file="${root}/docker_hermes/demo/docker-compose.yml" - -# The source pin must not regress to a moving branch. This is intentionally a -# static gate: it does not contact an upstream repository or Docker daemon. -! rg -q 'git clone --depth 1 --branch main' "$dockerfile" -rg -q '^ARG HERMES_SOURCE_REPOSITORY=' "$dockerfile" -rg -q '^ARG HERMES_SOURCE_COMMIT=' "$dockerfile" -rg -q 'git fetch --depth 1 origin "\$HERMES_SOURCE_COMMIT"' "$dockerfile" -rg -Fq 'test "$(git rev-parse HEAD)" = "$HERMES_SOURCE_COMMIT"' "$dockerfile" -rg -q 'org.opencontainers.image.source' "$dockerfile" -rg -q 'org.opencontainers.image.revision' "$dockerfile" -rg -q 'io.labnow.hermes.build-base' "$dockerfile" -rg -q 'io.labnow.hermes.runtime-base' "$dockerfile" - -rg -q '^ image: "\$\{HERMES_IMAGE:\?set a fixed local Hermes image\}"$' "$compose_file" -rg -q '^ pull_policy: never$' "$compose_file" -rg -q '^ - "\$\{HERMES_DASHBOARD_PUBLISH_HOST:-127\.0\.0\.1\}:\$\{HERMES_DASHBOARD_PORT:-9119\}:\$\{HERMES_DASHBOARD_PORT:-9119\}"$' "$compose_file" -rg -q '^ command: \["start-hermes\.sh", "all"\]$' "$compose_file" - -printf '%s\n' 'PASS P7 gates: Hermes source pin, provenance labels, explicit local image, pull policy, and loopback dashboard publication.' diff --git a/docker_hermes/scripts/test-hermes-runtime-node.sh b/docker_hermes/scripts/test-hermes-runtime-node.sh deleted file mode 100755 index 1e461cf..0000000 --- a/docker_hermes/scripts/test-hermes-runtime-node.sh +++ /dev/null @@ -1,37 +0,0 @@ -#!/usr/bin/env bash -# Validate that the final Hermes runtime owns the Node executable used by the -# pre-built Dashboard TUI. This test never provides provider credentials and -# never performs a model request. -set -euo pipefail - -root="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" -dockerfile="${root}/docker_hermes/hermes.Dockerfile" -image="${1:-}" - -rg -q '^COPY --from=builder /opt/node /opt/node$' "$dockerfile" -rg -q '^ENV PATH="/opt/node/bin:\$\{PATH\}"$' "$dockerfile" -rg -q 'node --check /opt/hermes/ui-tui/dist/entry.js' "$dockerfile" - -if [[ -z "$image" ]]; then - printf '%s\n' 'PASS static runtime Node/TUI gate.' - exit 0 -fi - -docker run --rm --platform linux/amd64 --entrypoint /bin/sh "$image" -ec ' - node --version - node_major="$(node --version | sed -E "s/^v([0-9]+).*/\1/")" - test "$node_major" -ge 22 - test -s /opt/hermes/ui-tui/dist/entry.js - node --check /opt/hermes/ui-tui/dist/entry.js - # Bounded module startup only: stdin is closed and no provider settings are - # present, so the TUI cannot submit a model request. A timeout means startup - # stayed alive; immediate clean EOF is also acceptable. - set +e - timeout 3s node /opt/hermes/ui-tui/dist/entry.js /tmp/hermes-tui-startup.log 2>&1 - status=$? - set -e - test "$status" = 0 -o "$status" = 124 - ! rg -q "install(ing)? node|downloading node" /tmp/hermes-tui-startup.log - rm -f /tmp/hermes-tui-startup.log -' -printf '%s\n' 'PASS container runtime Node/TUI gate.' diff --git a/docker_litellm/demo/scripts/smoke-baseline.sh b/docker_litellm/demo/scripts/smoke-baseline.sh deleted file mode 100755 index 5ad00a1..0000000 --- a/docker_litellm/demo/scripts/smoke-baseline.sh +++ /dev/null @@ -1,803 +0,0 @@ -#!/usr/bin/env bash -# Runs against a local, ignored docker_litellm/demo/.env. Secrets are written -# only to 0600 files below; never pass them to curl, jq, or another process. -set -euo pipefail - -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -DEMO_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)" -source "${SCRIPT_DIR}/verification-lib.sh" -ENV_FILE="${LITELLM_SMOKE_ENV_FILE:-${DEMO_DIR}/.env}" -: "${PROFILE_ENV:=litellm-baseline}" -export PROFILE_ENV -MODE="single" -SECURITY_CHECK=false -CLEANUP_NEGATIVE_TEST=false -REVOCATION_SLO_MS="${LITELLM_REVOCATION_SLO_MS:-30000}" -SUMMARY_FILE="${LITELLM_SMOKE_SUMMARY_FILE:-}" -verification_run_id="${VERIFICATION_RUN_ID:-standalone}" -block_elapsed_ms="" -delete_elapsed_ms="" -result="failed" -chat_result="not_run" -stream_result="not_run" -tool_result="not_run" -usage_result="not_run" -block_result="not_run" -delete_result="not_run" -shared_rpm_limit_result="not_applicable" -shared_tpm_limit_result="not_applicable" -shared_spend_log_visibility_result="not_applicable" -limiter_source="not_applicable" -idempotency_recovery_result="not_applicable" -migration_result="not_run" -security_scan_result="not_run" -content_logging_scan_result="not_run" -cleanup_result="not_run" -redis_container="" -redis_network="" -smoke_phase="initializing" -smoke_exit_code="" -cleanup_running=false -tmpdir="" -image_ref="" -test_prefix="" -admin_headers="" - -usage() { - echo "Usage: $0 [--mode single|ha] [--security-check] [--cleanup-negative-test]" >&2 -} - -while (($#)); do - case "$1" in - --mode) MODE="${2:-}"; shift 2 ;; - --security-check) SECURITY_CHECK=true; shift ;; - --cleanup-negative-test) CLEANUP_NEGATIVE_TEST=true; shift ;; - --help|-h) usage; exit 0 ;; - *) usage; exit 2 ;; - esac -done - -if [[ -z "$SUMMARY_FILE" ]]; then - SUMMARY_FILE="$DEMO_DIR/artifacts/p1-${MODE}-summary.json" -fi - -# Covers precondition failures before the resource-aware cleanup trap is ready. -# It invalidates any stale report and writes a non-passing, redacted result. -early_failure_cleanup() { - local exit_code=$? - # A security inspection is strictly read-only. This includes intentionally - # failing inspections used by the negative gate: neither path may replace - # a producer report from an earlier real smoke. - if [[ "$SECURITY_CHECK" == true ]]; then - trap - EXIT - return "$exit_code" - fi - # Ignore any nested EXIT delivery while preserving the report just written. - trap '' EXIT - if command -v jq >/dev/null; then - umask 077 - mkdir -p "$(dirname "$SUMMARY_FILE")" - chmod 700 "$(dirname "$SUMMARY_FILE")" - jq -n \ - --arg commit "$(git -C "$DEMO_DIR/../.." rev-parse HEAD)" \ - --arg run_id "$verification_run_id" --arg mode "$MODE" \ - --argjson exit_code "$exit_code" \ - '{verification_run_id:$run_id,commit:$commit,image_id:"",tested_at:(now|todateiso8601),mode:$mode,result:"failed",phase:"precondition_failed",exit_code:$exit_code,content_redacted:true}' \ - > "${SUMMARY_FILE}.tmp.$$" && chmod 600 "${SUMMARY_FILE}.tmp.$$" && mv "${SUMMARY_FILE}.tmp.$$" "$SUMMARY_FILE" || rm -f "${SUMMARY_FILE}.tmp.$$" - fi - return "$exit_code" -} -trap early_failure_cleanup EXIT - -need() { command -v "$1" >/dev/null || { echo "required command missing: $1" >&2; exit 2; }; } -need curl; need jq; need rg -[[ "$MODE" == "single" || "$MODE" == "ha" ]] || { usage; exit 2; } -[[ "$REVOCATION_SLO_MS" =~ ^[0-9]+$ ]] || { echo "LITELLM_REVOCATION_SLO_MS must be an integer" >&2; exit 2; } - -security_check() { - local unsafe=0 - - # Reject inline secret headers, secret-bearing jq arguments, trace logging, - # and proxy-container injection of the upstream credentials. - if sed '/^security_check() {/,/^}/d' "$0" | rg -n -- '(^|[[:space:]])-H([[:space:]]|$)' \ - || sed '/^security_check() {/,/^}/d' "$0" | rg -n --pcre2 -- '--header\s+["'"'"']?Authorization:' \ - || sed '/^security_check() {/,/^}/d' "$0" | rg -n --pcre2 '(curl|jq)[^\n]*(LITELLM_MASTER_KEY|UPSTREAM_API_KEY|virtual_key)' \ - || sed '/^security_check() {/,/^}/d' "$0" | rg -n --pcre2 -- '--arg(?:json)?\s+[^[:space:]]*(key|secret|token)' \ - || sed '/^security_check() {/,/^}/d' "$0" | rg -n -- 'set -x' \ - || git diff --no-ext-diff -- . | rg -n --pcre2 '(?:sk-|Bearer\s+)[A-Za-z0-9_-]{24,}' \ - || rg -n '^ UPSTREAM_(API_KEY|BASE_URL|MODEL|PROVIDER):' "$DEMO_DIR/docker-compose.litellm.yml" \ - || rg -n '^ (LITELLM_MASTER_KEY|DATABASE_URL|POSTGRES_PASSWORD):' "$DEMO_DIR/docker-compose.litellm.yml" \ - || rg -n -- '--requirepass[[:space:]].*\$\{REDIS_PASSWORD|REDIS_PASSWORD:.*\$\{' "$DEMO_DIR/docker-compose.litellm.yml" \ - || ! rg -q 'LITELLM_MASTER_KEY_FILE: /run/secrets/litellm_master_key' "$DEMO_DIR/docker-compose.litellm.yml" \ - || ! rg -q 'POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password' "$DEMO_DIR/docker-compose.litellm.yml" \ - || ! rg -q 'litellm_master_key:' "$DEMO_DIR/docker-compose.litellm.yml" \ - || ! rg -q 'postgres_password:' "$DEMO_DIR/docker-compose.litellm.yml" \ - || ! rg -q 'redis_password:' "$DEMO_DIR/docker-compose.litellm.yml" \ - || ! rg -q 'REDIS_PASSWORD_FILE: /run/secrets/redis_password' "$DEMO_DIR/docker-compose.litellm.yml"; then - unsafe=1 - fi - - rg -q 'umask 077' "$0" \ - && rg -q 'chmod 600' "$0" \ - && rg -q 'export -n LITELLM_MASTER_KEY' "$0" \ - && rg -q 'unset LITELLM_MASTER_KEY' "$0" \ - && rg -q 'trap cleanup EXIT' "$0" \ - && rg -q 'rm -rf "\$tmpdir"' "$0" \ - || unsafe=1 - - # Test-only fault injection validates the failure path preserves reports. - # It is deliberately limited to this read-only static checker. - [[ "${LITELLM_SECURITY_CHECK_FORCE_FAILURE:-0}" != "1" ]] || unsafe=1 - - if ((unsafe)); then - echo "FAIL security negative check: unsafe secret transport or cleanup invariant" >&2 - return 1 - fi - echo "PASS security negative check: no inline process arguments/log tracing/Git secrets, no upstream or management/database Compose injection, Docker Secret and 0600 cleanup invariants present." -} - -if [[ "$SECURITY_CHECK" == true ]]; then - security_check - trap - EXIT - exit 0 -fi - -# A real producer invalidates its old result before all remaining preconditions. -# `--security-check` is read-only and must never touch an existing summary. -verification_invalidate_report "$SUMMARY_FILE" - -# Run the static negative checks in every real smoke too. A successful report -# cannot claim a security result that was not actually executed. -security_check -security_scan_result="static_passed" - -[[ -f "$ENV_FILE" ]] || { echo "missing local environment file: $ENV_FILE (copy .env.example)" >&2; exit 2; } - -umask 077 -tmpdir="$(mktemp -d "${TMPDIR:-/tmp}/litellm-smoke.XXXXXX")" -chmod 700 "$tmpdir" - -# Compose's dotenv grammar is not Bash's grammar. Reading it with `source` -# can change quoted/special-character management keys and create a false 403. -# Ask Compose for its effective environment into a 0600 file and never print it. -compose_environment="$tmpdir/compose.environment" -docker compose --env-file "$ENV_FILE" -f "$DEMO_DIR/docker-compose.litellm.yml" config --environment > "$compose_environment" -chmod 600 "$compose_environment" -effective_env() { - local name="$1" - awk -F= -v name="$name" '$1 == name {sub(/^[^=]*=/, ""); print; exit}' "$compose_environment" -} -LITELLM_MASTER_KEY="$(effective_env LITELLM_MASTER_KEY)" -LITELLM_IMAGE="$(effective_env LITELLM_IMAGE)" -UPSTREAM_API_KEY="$(effective_env UPSTREAM_API_KEY)" -UPSTREAM_BASE_URL="$(effective_env UPSTREAM_BASE_URL)" -UPSTREAM_MODEL="$(effective_env UPSTREAM_MODEL)" -UPSTREAM_PROVIDER="$(effective_env UPSTREAM_PROVIDER)" -LITELLM_PUBLISH_HOST="$(effective_env LITELLM_PUBLISH_HOST)" -LITELLM_1_PORT="$(effective_env LITELLM_1_PORT)" -LITELLM_2_PORT="$(effective_env LITELLM_2_PORT)" -: "${LITELLM_MASTER_KEY:?missing LITELLM_MASTER_KEY in effective Compose environment}" -: "${LITELLM_IMAGE:?missing LITELLM_IMAGE in effective Compose environment}" -image_ref="$LITELLM_IMAGE" -upstream_provider="${LITELLM_SMOKE_UPSTREAM_PROVIDER:-${UPSTREAM_PROVIDER:-}}" -if [[ "$MODE" == "single" ]]; then - BASE_URL="http://${LITELLM_PUBLISH_HOST:-127.0.0.1}:${LITELLM_1_PORT:-4000}" - PEER_URL="$BASE_URL" -else - BASE_URL="http://${LITELLM_PUBLISH_HOST:-127.0.0.1}:${LITELLM_1_PORT:-4000}" - PEER_URL="http://${LITELLM_PUBLISH_HOST:-127.0.0.1}:${LITELLM_2_PORT:-4001}" -fi - -private_file() { - : > "$1" - chmod 600 "$1" -} - -write_private_value() { - private_file "$1" - printf '%s' "$2" > "$1" -} - -assert_private_file() { - verification_assert_file_mode "$1" 600 "temporary secret file" || exit 1 -} - -make_header_file() { - local header_file="$1" secret_file="$2" - private_file "$header_file" - { - printf 'Authorization: Bearer ' - tr -d '\r\n' < "$secret_file" - printf '\nContent-Type: application/json\n' - } > "$header_file" - assert_private_file "$header_file" -} - -master_key_file="$tmpdir/master-key" -upstream_key_file="$tmpdir/upstream-key" -upstream_base_file="$tmpdir/upstream-base" -upstream_model_file="$tmpdir/upstream-model" -upstream_provider_file="$tmpdir/upstream-provider" -admin_headers="$tmpdir/admin.headers" -write_private_value "$master_key_file" "$LITELLM_MASTER_KEY" -write_private_value "$upstream_key_file" "${UPSTREAM_API_KEY:-}" -write_private_value "$upstream_base_file" "${UPSTREAM_BASE_URL:-}" -write_private_value "$upstream_model_file" "${UPSTREAM_MODEL:-}" -write_private_value "$upstream_provider_file" "$upstream_provider" -make_header_file "$admin_headers" "$master_key_file" -assert_private_file "$master_key_file" -assert_private_file "$upstream_key_file" -assert_private_file "$upstream_base_file" -assert_private_file "$upstream_model_file" -assert_private_file "$upstream_provider_file" - -# No child process needs these values. Compose reads its own --env-file. -unset LITELLM_MASTER_KEY UPSTREAM_API_KEY UPSTREAM_BASE_URL UPSTREAM_MODEL UPSTREAM_PROVIDER \ - POSTGRES_PASSWORD REDIS_PASSWORD DATABASE_URL - -test_user="" -credential_name="" -model_name="" -model_id="" -block_key_created=false -delete_key_created=false -rate_key_created=false -enforcement_key_created=false -block_key_file="$tmpdir/block.key" -delete_key_file="$tmpdir/delete.key" -rate_key_file="$tmpdir/rate.key" -block_headers="$tmpdir/block.headers" -delete_headers="$tmpdir/delete.headers" -rate_headers="$tmpdir/rate.headers" -enforcement_key_file="$tmpdir/enforcement.key" -enforcement_headers="$tmpdir/enforcement.headers" - -request_admin() { - local method="$1" path="$2" payload_file="$3" output_file="$4" - local curl_args=(--silent --show-error --fail --max-time 30 --request "$method" "$BASE_URL$path" --header "@$admin_headers" --output "$output_file") - if [[ -n "$payload_file" ]]; then - curl_args+=(--data-binary "@$payload_file") - fi - curl "${curl_args[@]}" -} - -cleanup_request_admin() { - local method="$1" path="$2" payload_file="$3" - local curl_args=(--silent --show-error --fail --max-time 15 --request "$method" "$BASE_URL$path" --header "@$admin_headers" --output /dev/null) - if [[ -n "$payload_file" ]]; then - curl_args+=(--data-binary "@$payload_file") - fi - curl "${curl_args[@]}" >/dev/null 2>&1 -} - -assert_test_resources_removed() { - local counts_file="$tmpdir/cleanup-counts.txt" counts - [[ -n "$test_prefix" && -n "$tmpdir" ]] || return 0 - docker compose --env-file "$ENV_FILE" -f "$DEMO_DIR/docker-compose.litellm.yml" exec -T -e P1_CLEANUP_PREFIX="$test_prefix" postgres sh -lc 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -At -v ON_ERROR_STOP=1 -c "SELECT (SELECT count(*) FROM \"LiteLLM_UserTable\" WHERE user_id LIKE \$\$${P1_CLEANUP_PREFIX}%\$\$), (SELECT count(*) FROM \"LiteLLM_CredentialsTable\" WHERE credential_name LIKE \$\$${P1_CLEANUP_PREFIX}%\$\$), (SELECT count(*) FROM \"LiteLLM_ProxyModelTable\" WHERE model_name LIKE \$\$${P1_CLEANUP_PREFIX}%\$\$), (SELECT count(*) FROM \"LiteLLM_VerificationToken\" WHERE key_alias LIKE \$\$${P1_CLEANUP_PREFIX}%\$\$);"' > "$counts_file" - counts="$(tr -d '[:space:]' < "$counts_file")" - [[ "$counts" == "0|0|0|0" ]] -} - -cleanup() { - local exit_code=$? - local cleanup_ok=true - [[ "$cleanup_running" == false ]] || return "$exit_code" - cleanup_running=true - # Ignore any nested EXIT delivery while preserving the report just written. - trap '' EXIT - set +e - if [[ -n "$redis_container" && -n "$redis_network" ]]; then - docker network connect --alias redis "$redis_network" "$redis_container" >/dev/null 2>&1 || true - fi - if [[ "$delete_key_created" == true ]]; then - cleanup_request_admin POST /key/delete "$tmpdir/delete-key-cleanup.json" || cleanup_ok=false - fi - if [[ "$rate_key_created" == true ]]; then - cleanup_request_admin POST /key/delete "$tmpdir/rate-key-cleanup.json" || cleanup_ok=false - fi - if [[ "$enforcement_key_created" == true ]]; then - cleanup_request_admin POST /key/delete "$tmpdir/enforcement-key-cleanup.json" || cleanup_ok=false - fi - if [[ "$block_key_created" == true ]]; then - cleanup_request_admin POST /key/delete "$tmpdir/block-key-cleanup.json" || cleanup_ok=false - fi - if [[ -n "$model_id" ]]; then - cleanup_request_admin POST /model/delete "$tmpdir/model-delete.json" || cleanup_ok=false - fi - if [[ -n "$credential_name" ]]; then - cleanup_request_admin DELETE "/credentials/$credential_name" "" || cleanup_ok=false - fi - if [[ -n "$test_user" ]]; then - cleanup_request_admin POST /user/delete "$tmpdir/user-delete.json" || cleanup_ok=false - fi - # The negative test uses an authenticated, deliberately absent management - # route. A real HTTP 404 must keep cleanup failed and the report non-passing. - if [[ "$CLEANUP_NEGATIVE_TEST" == true ]]; then - cleanup_request_admin DELETE "/__p1_cleanup_failure_probe" "" || cleanup_ok=false - fi - assert_test_resources_removed || cleanup_ok=false - if [[ "$cleanup_ok" == true ]]; then cleanup_result="passed"; else cleanup_result="failed"; exit_code=1; fi - unset master_key_file upstream_key_file upstream_base_file upstream_model_file upstream_provider_file - rm -rf "$tmpdir" - if [[ -e "$tmpdir" ]]; then cleanup_result="failed"; exit_code=1; fi - smoke_exit_code="$exit_code" - if [[ "$exit_code" == 0 && "$smoke_phase" == completed && "$cleanup_result" == passed && "$security_scan_result" == passed ]]; then - result="passed" - elif [[ "$result" != skipped ]]; then - result="failed" - fi - write_summary || exit_code=1 - # A failed cleanup must return non-zero after the atomic report is durable. - # Disable the EXIT handler first: sending TERM here also terminates callers - # of the real cleanup-negative gate in some Bash execution modes. - if (( exit_code != 0 )); then - sync "$SUMMARY_FILE" 2>/dev/null || sync - trap - EXIT - exit "$exit_code" - fi - return "$exit_code" -} -trap cleanup EXIT - -request_data_get() { - local url="$1" header_file="$2" path="$3" output_file="$4" - curl --silent --show-error --fail --max-time 30 --request GET "$url$path" \ - --header "@$header_file" --output "$output_file" -} - -request_data_post() { - local url="$1" header_file="$2" path="$3" payload_file="$4" output_file="$5" - curl --silent --show-error --fail --max-time 60 --request POST "$url$path" \ - --header "@$header_file" --data-binary "@$payload_file" --output "$output_file" -} - -key_status() { - local url="$1" header_file="$2" - curl --silent --show-error --max-time 10 --request GET "$url/v1/models" \ - --header "@$header_file" --output /dev/null --write-out '%{http_code}' || true -} - -wait_ready() { - local url="$1" output_file="$tmpdir/readiness.json" i - for i in $(seq 1 60); do - if curl --silent --fail --max-time 3 "$url/health/readiness" --output "$output_file" \ - && jq -e '.status == "healthy" and .db == "connected"' "$output_file" >/dev/null; then - return 0 - fi - sleep 2 - done - echo "LiteLLM readiness did not report PostgreSQL connected: $url" >&2 - return 1 -} - -assert_redis() { - local service="$1" - docker compose --env-file "$ENV_FILE" -f "$DEMO_DIR/docker-compose.litellm.yml" exec -T "$service" \ - python3 -c 'import redis; password=open("/run/secrets/redis_password", encoding="utf-8").read().strip(); assert redis.Redis(host="redis", port=6379, password=password, socket_connect_timeout=2, socket_timeout=2).ping()' -} - -now_ms() { - python3 -c 'import time; print(time.time_ns() // 1_000_000)' -} - -assert_migration_evidence() { - local report="$DEMO_DIR/artifacts/p1-migration-summary.json" commit image_id - commit="$(git -C "$DEMO_DIR/../.." rev-parse HEAD)" - image_id="$(docker image inspect "$image_ref" --format '{{.Id}}' 2>/dev/null || true)" - [[ -f "$report" ]] || { echo "missing current migration report: $report" >&2; return 1; } - jq -e --arg commit "$commit" --arg image_id "$image_id" --arg run_id "$verification_run_id" ' - .mode == "migration" and .result == "passed" and .phase == "completed" and - .verification_run_id == $run_id and .commit == $commit and .image_id == $image_id and .proxy_replicas_started == false and - .content_redacted == true - ' "$report" >/dev/null -} - -runtime_security_check() { - local logs_file="$tmpdir/litellm-logs.txt" inspect_file="$tmpdir/inspect.json" process_file="$tmpdir/redis-processes.txt" db_content_file="$tmpdir/spendlog-db-content.txt" litellm_1_container litellm_2_container - litellm_1_container="$(docker compose --env-file "$ENV_FILE" -f "$DEMO_DIR/docker-compose.litellm.yml" ps -q litellm-1)" - [[ -n "$litellm_1_container" ]] || { echo "litellm-1 container not found" >&2; return 1; } - docker inspect "$litellm_1_container" > "$inspect_file" - if [[ "$MODE" == ha ]]; then - litellm_2_container="$(docker compose --env-file "$ENV_FILE" -f "$DEMO_DIR/docker-compose.litellm.yml" ps -q litellm-2)" - [[ -n "$litellm_2_container" ]] || { echo "litellm-2 container not found" >&2; return 1; } - docker inspect "$litellm_2_container" >> "$inspect_file" - fi - docker compose --env-file "$ENV_FILE" -f "$DEMO_DIR/docker-compose.litellm.yml" logs --no-color litellm-1 > "$logs_file" 2>&1 - if [[ "$MODE" == ha ]]; then docker compose --env-file "$ENV_FILE" -f "$DEMO_DIR/docker-compose.litellm.yml" logs --no-color litellm-2 >> "$logs_file" 2>&1; fi - docker exec "$(docker compose --env-file "$ENV_FILE" -f "$DEMO_DIR/docker-compose.litellm.yml" ps -q redis)" ps -eo args > "$process_file" - # Query only P1 test SpendLog content into the private work directory. This - # validates the database representation independently of the API response. - docker compose --env-file "$ENV_FILE" -f "$DEMO_DIR/docker-compose.litellm.yml" exec -T -e P1_CONTENT_PREFIX="$test_prefix" postgres sh -lc 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -At -v ON_ERROR_STOP=1 -c "SELECT concat(messages::text, response::text, proxy_server_request::text) FROM \"LiteLLM_SpendLogs\" WHERE \"user\" LIKE \$\$${P1_CONTENT_PREFIX}%\$\$;"' > "$db_content_file" - ! rg -q 'UPSTREAM_(API_KEY|BASE_URL|MODEL)=' "$inspect_file" && - ! rg -q -- '--requirepass[[:space:]]+[^[:space:]]+' "$process_file" && - ! rg -q --file "$tmpdir/prompt-marker" "$logs_file" && - ! rg -q --file "$tmpdir/tool-marker" "$logs_file" && - ! rg -q --file "$tmpdir/response-marker" "$logs_file" && - ! rg -q --file "$tmpdir/prompt-marker" "$db_content_file" && - ! rg -q --file "$tmpdir/tool-marker" "$db_content_file" && - ! rg -q --file "$tmpdir/response-marker" "$db_content_file" && - ! git ls-files -z | xargs -0 rg -n --pcre2 '(?:sk-|Bearer[[:space:]]+)[A-Za-z0-9_-]{24,}' -- >/dev/null 2>&1 && - verification_assert_file_mode "$admin_headers" 600 "LiteLLM administration header" -} - -make_key_payload() { - local alias_file="$1" payload_file="$2" explicit_key_file="${3:-}" - if [[ -n "$explicit_key_file" ]]; then - jq -n \ - --rawfile alias "$alias_file" \ - --rawfile user "$tmpdir/test-user" \ - --rawfile model "$tmpdir/model-name" \ - --rawfile key "$explicit_key_file" \ - '{key: ($key | rtrimstr("\n")), key_alias: ($alias | rtrimstr("\n")), user_id: ($user | rtrimstr("\n")), models: [($model | rtrimstr("\n"))], duration: "15m", max_budget: 0.05, rpm_limit: 10, tpm_limit: 1000, key_type: "llm_api", allowed_routes: ["/v1/models", "/v1/chat/completions", "/key/info"]}' \ - > "$payload_file" - else - jq -n \ - --rawfile alias "$alias_file" \ - --rawfile user "$tmpdir/test-user" \ - --rawfile model "$tmpdir/model-name" \ - '{key_alias: ($alias | rtrimstr("\n")), user_id: ($user | rtrimstr("\n")), models: [($model | rtrimstr("\n"))], duration: "15m", max_budget: 0.05, rpm_limit: 10, tpm_limit: 1000, key_type: "llm_api", allowed_routes: ["/v1/models", "/v1/chat/completions"]}' \ - > "$payload_file" - fi - chmod 600 "$payload_file" -} - -make_key_header() { - local response_file="$1" key_file="$2" header_file="$3" - private_file "$key_file" - jq -er '.key | select(type == "string" and length > 0)' "$response_file" > "$key_file" - assert_private_file "$key_file" - make_header_file "$header_file" "$key_file" -} - -hash_key_file() { - local key_file="$1" hash_file="$2" - private_file "$hash_file" - python3 -c 'import hashlib, sys; print(hashlib.sha256(sys.stdin.buffer.read().rstrip(b"\r\n")).hexdigest())' \ - < "$key_file" > "$hash_file" - assert_private_file "$hash_file" -} - -make_key_action_payload() { - local action="$1" key_file="$2" payload_file="$3" - if [[ "$action" == block ]]; then - jq -n --rawfile key "$key_file" '{key: ($key | rtrimstr("\n"))}' > "$payload_file" - else - jq -n --rawfile key "$key_file" '{keys: [($key | rtrimstr("\n"))]}' > "$payload_file" - fi - chmod 600 "$payload_file" -} - -make_key_info_payload() { - local key_file="$1" payload_file="$2" - jq -n --rawfile key "$key_file" '{keys: [($key | rtrimstr("\n"))]}' > "$payload_file" - chmod 600 "$payload_file" -} - -wait_model_access() { - local url="$1" header_file="$2" label="$3" output_file i - output_file="$tmpdir/$label-models.json" - for i in $(seq 1 30); do - if request_data_get "$url" "$header_file" /v1/models "$output_file" \ - && jq -e --rawfile model "$tmpdir/model-name" '.data[] | select(.id == ($model | rtrimstr("\n")))' "$output_file" >/dev/null; then - return 0 - fi - sleep 1 - done - echo "virtual key was not accepted by $label before revocation" >&2 - return 1 -} - -wait_for_rejection() { - local header_file="$1" phase="$2" start_ms now elapsed primary_code peer_code - start_ms="$(now_ms)" - while :; do - primary_code="$(key_status "$BASE_URL" "$header_file")" - peer_code="$(key_status "$PEER_URL" "$header_file")" - if [[ "$primary_code" =~ ^(401|403)$ && "$peer_code" =~ ^(401|403)$ ]]; then - now="$(now_ms)" - elapsed=$((now - start_ms)) - echo "PASS $phase propagation: primary=$primary_code peer=$peer_code elapsed_ms=$elapsed slo_ms=$REVOCATION_SLO_MS" - if [[ "$phase" == "block" ]]; then block_elapsed_ms="$elapsed"; else delete_elapsed_ms="$elapsed"; fi - return 0 - fi - now="$(now_ms)" - if ((now - start_ms >= REVOCATION_SLO_MS)); then - echo "FAIL $phase propagation: primary=$primary_code peer=$peer_code exceeded_slo_ms=$REVOCATION_SLO_MS" >&2 - return 1 - fi - sleep 1 - done -} - -assert_proxy_limiter_response() { - local response_file="$1" headers_file="$2" http_code="$3" limit_kind="$4" since="$5" expected_type - if [[ "$limit_kind" == rpm ]]; then expected_type=requests; else expected_type=tokens; fi - [[ "$http_code" == "429" ]] && - # LiteLLM 1.97.0's parallel_request_limiter_v3 raises ProxyRateLimitError - # with the stable proxy-only rate_limit_type header. Provider 429s do not - # synthesize this header or the matching "Limit type" detail below. - rg -qi -- "^rate_limit_type:[[:space:]]*${expected_type}[[:space:]]*$" "$headers_file" && - jq -e --arg expected "$expected_type" '(.detail // .error.detail // .error.message // .error // .message // "") | tostring | test("Rate limit exceeded.*Limit type: " + $expected; "i")' "$response_file" >/dev/null && - # The access log is bounded to this request window and confirms that this - # exact proxy instance emitted a local 429; no prompt/response is copied. - docker compose --env-file "$ENV_FILE" -f "$DEMO_DIR/docker-compose.litellm.yml" logs --no-color --since "$since" litellm-1 litellm-2 | - rg -q 'POST /v1/chat/completions.* 429|HTTP/1\.[01]" 429' -} - -write_summary() { - local summary_tmp - mkdir -p "$(dirname "$SUMMARY_FILE")" - chmod 700 "$(dirname "$SUMMARY_FILE")" - summary_tmp="$(mktemp "${SUMMARY_FILE}.tmp.XXXXXX")" - chmod 600 "$summary_tmp" - jq -n \ - --arg commit "$(git -C "$DEMO_DIR/../.." rev-parse HEAD)" \ - --arg image_id "$(docker image inspect "$image_ref" --format '{{.Id}}' 2>/dev/null || true)" \ - --arg tested_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ - --arg mode "$MODE" --arg block_ms "$block_elapsed_ms" --arg delete_ms "$delete_elapsed_ms" --arg phase "$smoke_phase" --arg exit_code "$smoke_exit_code" \ - --arg run_id "$verification_run_id" --arg result "$result" --arg migration "$migration_result" --arg chat "$chat_result" --arg stream "$stream_result" --arg tool "$tool_result" --arg usage "$usage_result" --arg block "$block_result" --arg delete "$delete_result" --arg shared_rpm "$shared_rpm_limit_result" --arg shared_tpm "$shared_tpm_limit_result" --arg shared_spend "$shared_spend_log_visibility_result" --arg limiter_source "$limiter_source" --arg idempotency "$idempotency_recovery_result" --arg cleanup "$cleanup_result" --arg security "$security_scan_result" --arg content_logging "$content_logging_scan_result" \ - '{verification_run_id:$run_id,commit:$commit,image_id:$image_id,tested_at:$tested_at,mode:$mode,result:$result,phase:$phase,exit_code:(try ($exit_code|tonumber) catch null),migration:$migration,chat:$chat,stream:$stream,tool:$tool,usage:$usage,block:$block,delete:$delete,shared_rpm_limit:$shared_rpm,shared_tpm_limit:$shared_tpm,shared_spend_log_visibility:$shared_spend,limiter_source:$limiter_source,idempotency_recovery:$idempotency,block_elapsed_ms:(try ($block_ms|tonumber) catch null),delete_elapsed_ms:(try ($delete_ms|tonumber) catch null),cleanup:$cleanup,security_scan:$security,content_logging_scan:$content_logging,content_redacted:true}' \ - > "$summary_tmp" - chmod 600 "$summary_tmp" - mv "$summary_tmp" "$SUMMARY_FILE" - [[ -s "$SUMMARY_FILE" ]] || { echo "summary write produced an empty file" >&2; return 1; } - echo "PASS summary: $SUMMARY_FILE" -} - -assert_spend() { - local spend_file="$tmpdir/spend.json" peer_spend_file="$tmpdir/peer-spend.json" request_count total_tokens peer_request_count peer_total_tokens i - for i in $(seq 1 30); do - if request_admin GET "/spend/logs?user_id=$test_user" "" "$spend_file" \ - && jq -e \ - --rawfile model "$tmpdir/model-name" \ - --rawfile key_hash "$tmpdir/block-key-sha256" \ - 'def logs: if type == "array" then . else (.data // []) end; [ logs[] | select((.model == ($model | rtrimstr("\n")) or .model_group == ($model | rtrimstr("\n"))) and .api_key == ($key_hash | rtrimstr("\n")) and ((.total_tokens // 0) > 0)) ] as $logs | ($logs | length) >= 3 and (($logs | map(.total_tokens // 0) | add) > 0)' \ - "$spend_file" >/dev/null; then - request_count="$(jq --rawfile model "$tmpdir/model-name" --rawfile key_hash "$tmpdir/block-key-sha256" 'def logs: if type == "array" then . else (.data // []) end; [ logs[] | select((.model == ($model | rtrimstr("\n")) or .model_group == ($model | rtrimstr("\n"))) and .api_key == ($key_hash | rtrimstr("\n")) and ((.total_tokens // 0) > 0)) ] | length' "$spend_file")" - total_tokens="$(jq --rawfile model "$tmpdir/model-name" --rawfile key_hash "$tmpdir/block-key-sha256" 'def logs: if type == "array" then . else (.data // []) end; [ logs[] | select((.model == ($model | rtrimstr("\n")) or .model_group == ($model | rtrimstr("\n"))) and .api_key == ($key_hash | rtrimstr("\n")) and ((.total_tokens // 0) > 0)) ] | map(.total_tokens // 0) | add' "$spend_file")" - if [[ "$MODE" == "ha" ]]; then - curl --silent --show-error --fail --max-time 30 --request GET "$PEER_URL/spend/logs?user_id=$test_user" --header "@$admin_headers" --output "$peer_spend_file" - peer_request_count="$(jq --rawfile model "$tmpdir/model-name" --rawfile key_hash "$tmpdir/block-key-sha256" 'def logs: if type == "array" then . else (.data // []) end; [ logs[] | select((.model == ($model | rtrimstr("\n")) or .model_group == ($model | rtrimstr("\n"))) and .api_key == ($key_hash | rtrimstr("\n")) and ((.total_tokens // 0) > 0)) ] | length' "$peer_spend_file")" - peer_total_tokens="$(jq --rawfile model "$tmpdir/model-name" --rawfile key_hash "$tmpdir/block-key-sha256" 'def logs: if type == "array" then . else (.data // []) end; [ logs[] | select((.model == ($model | rtrimstr("\n")) or .model_group == ($model | rtrimstr("\n"))) and .api_key == ($key_hash | rtrimstr("\n")) and ((.total_tokens // 0) > 0)) ] | map(.total_tokens // 0) | add' "$peer_spend_file")" - [[ "$request_count/$total_tokens" == "$peer_request_count/$peer_total_tokens" ]] || { echo "shared SpendLog mismatch: primary=$request_count/$total_tokens peer=$peer_request_count/$peer_total_tokens" >&2; return 1; } - shared_spend_log_visibility_result="passed" - echo "PASS HA shared SpendLog: request_count=$request_count total_tokens=$total_tokens on both replicas." - fi - echo "PASS spend: request_count=$request_count total_tokens=$total_tokens" - return 0 - fi - sleep 2 - done - echo "spend logs did not prove three token-bearing requests for this model and virtual key alias" >&2 - return 1 -} - -wait_ready "$BASE_URL" -if [[ "$MODE" == "ha" ]]; then wait_ready "$PEER_URL"; fi -assert_redis litellm-1 -if [[ "$MODE" == "ha" ]]; then assert_redis litellm-2; fi -assert_migration_evidence -migration_result="passed" -echo "PASS readiness: PostgreSQL connected; Redis independently reachable from LiteLLM replica(s)." - -suffix="$(date +%s)-$(python3 -c 'import secrets; print(secrets.token_hex(8))')" -test_prefix="p1-smoke-${verification_run_id#p1-}-$suffix" -test_user="${test_prefix}-user" -credential_name="${test_prefix}-upstream" -model_name="${test_prefix}-model" -write_private_value "$tmpdir/test-user" "$test_user" -write_private_value "$tmpdir/credential-name" "$credential_name" -write_private_value "$tmpdir/model-name" "$model_name" -write_private_value "$tmpdir/prompt-marker" "p1-redaction-prompt-$suffix" -write_private_value "$tmpdir/tool-marker" "p1-redaction-tool-$suffix" -write_private_value "$tmpdir/response-marker" "p1-redaction-response-$suffix" -# This identifier is intentionally non-sensitive. LiteLLM 1.97.0 may emit a -# parser warning with a function name when an upstream tool call is malformed; -# the sensitive marker stays in the tool schema body, which the scan verifies -# is never persisted or logged. -write_private_value "$tmpdir/tool-name" "p1_smoke_tool" - -jq -n --rawfile user "$tmpdir/test-user" '{user_id: ($user | rtrimstr("\n")), auto_create_key: false, user_role: "internal_user"}' > "$tmpdir/user-create.json" -chmod 600 "$tmpdir/user-create.json" -jq -n --rawfile user "$tmpdir/test-user" '{user_ids: [($user | rtrimstr("\n"))]}' > "$tmpdir/user-delete.json" -chmod 600 "$tmpdir/user-delete.json" -request_admin POST /user/new "$tmpdir/user-create.json" "$tmpdir/user.json" -jq -e --rawfile user "$tmpdir/test-user" '.user_id == ($user | rtrimstr("\n"))' "$tmpdir/user.json" >/dev/null - -if [[ "$CLEANUP_NEGATIVE_TEST" == true ]]; then - smoke_phase="cleanup_negative_test" - echo "Running authenticated cleanup failure negative test." - exit 0 -fi - -if [[ ! -s "$upstream_key_file" || ! -s "$upstream_base_file" || ! -s "$upstream_model_file" ]]; then - echo "PENDING upstream smoke: set UPSTREAM_API_KEY, UPSTREAM_BASE_URL and UPSTREAM_MODEL in ignored local environment file." - chat_result="pending"; stream_result="pending"; tool_result="pending"; usage_result="pending" - block_result="pending"; delete_result="pending"; idempotency_recovery_result="pending" - result="skipped"; smoke_phase="pending_upstream" - echo "PASS infrastructure: PostgreSQL persistence, shared Redis reachability, management authentication and cleanup path are ready." - exit 0 -fi - -# P1 deliberately supports one explicit provider mapping. Reject incomplete -# or ambiguous combinations before any upstream-facing request is sent. -provider_normalized="$(tr -d '\r\n' < "$upstream_provider_file" | tr '[:upper:]' '[:lower:]')" -case "$provider_normalized" in - deepseek*) - provider_prefix="deepseek" - ;; - *) - echo "invalid UPSTREAM_PROVIDER: supported P1 provider is deepseek (value redacted)" >&2 - exit 2 - ;; -esac -if ! rg -q '^https://[^[:space:]]+$' "$upstream_base_file" \ - || ! rg -q '^[A-Za-z0-9._:-]+$' "$upstream_model_file"; then - echo "invalid upstream base URL or model identifier (values redacted)" >&2 - exit 2 -fi - -# The upstream key appears only in this 0600 request file. The model itself -# references the stored credential, never the upstream key directly. -jq -n \ - --rawfile credential_name "$tmpdir/credential-name" \ - --rawfile api_key "$upstream_key_file" \ - --rawfile api_base "$upstream_base_file" \ - --arg provider "$provider_prefix" \ - '{credential_name: ($credential_name | rtrimstr("\n")), credential_values: {api_key: ($api_key | rtrimstr("\n")), api_base: ($api_base | rtrimstr("\n"))}, credential_info: {custom_llm_provider: $provider}}' \ - > "$tmpdir/credential-create.json" -chmod 600 "$tmpdir/credential-create.json" -request_admin POST /credentials "$tmpdir/credential-create.json" "$tmpdir/credential.json" -jq -e '.success == true' "$tmpdir/credential.json" >/dev/null - -jq -n \ - --rawfile model_name "$tmpdir/model-name" \ - --rawfile upstream_model "$upstream_model_file" \ - --rawfile credential_name "$tmpdir/credential-name" \ - --arg provider "$provider_prefix" \ - '{model_name: ($model_name | rtrimstr("\n")), litellm_params: {model: ($provider + "/" + ($upstream_model | rtrimstr("\n"))), litellm_credential_name: ($credential_name | rtrimstr("\n"))}, model_info: {mode: "chat"}}' \ - > "$tmpdir/model-create.json" -chmod 600 "$tmpdir/model-create.json" -request_admin POST /model/new "$tmpdir/model-create.json" "$tmpdir/model.json" -model_id="$(jq -er '.model_id' "$tmpdir/model.json")" -write_private_value "$tmpdir/model-id" "$model_id" -jq -n --rawfile id "$tmpdir/model-id" '{id: ($id | rtrimstr("\n"))}' > "$tmpdir/model-delete.json" -chmod 600 "$tmpdir/model-delete.json" - -write_private_value "$tmpdir/block-key-alias" "${test_prefix}-block" -private_file "$block_key_file" -python3 -c 'import secrets; print("sk-p1-" + secrets.token_urlsafe(32))' > "$block_key_file" -assert_private_file "$block_key_file" -make_key_payload "$tmpdir/block-key-alias" "$tmpdir/block-key-create.json" "$block_key_file" -# Intentionally discard the create response to model a client-side timeout. -# The stable caller-generated key is then recovered through LiteLLM 1.97.0's -# admin-only /v2/key/info endpoint. The key stays in a 0600 request body; -# it never appears in a query parameter, process argument or report. -request_admin POST /key/generate "$tmpdir/block-key-create.json" /dev/null -block_key_created=true -make_header_file "$block_headers" "$block_key_file" -make_key_action_payload delete "$block_key_file" "$tmpdir/block-key-cleanup.json" -make_key_info_payload "$block_key_file" "$tmpdir/key-recovery-request.json" -request_admin POST /v2/key/info "$tmpdir/key-recovery-request.json" "$tmpdir/key-recovery.json" -jq -e --rawfile alias "$tmpdir/block-key-alias" '.info | length == 1 and .[0].key_alias == ($alias | rtrimstr("\n"))' "$tmpdir/key-recovery.json" >/dev/null -retry_code="$(curl --silent --show-error --max-time 20 --request POST "$BASE_URL/key/generate" --header "@$admin_headers" --data-binary "@$tmpdir/block-key-create.json" --output "$tmpdir/key-retry.json" --write-out '%{http_code}' || true)" -[[ "$retry_code" =~ ^(400|409|422)$ ]] || { echo "stable-key retry unexpectedly created a second resource: http=$retry_code" >&2; exit 1; } -request_admin POST /v2/key/info "$tmpdir/key-recovery-request.json" "$tmpdir/key-recovery-after-retry.json" -jq -e --rawfile alias "$tmpdir/block-key-alias" '.info | length == 1 and .[0].key_alias == ($alias | rtrimstr("\n"))' "$tmpdir/key-recovery-after-retry.json" >/dev/null -idempotency_recovery_result="passed" -hash_key_file "$block_key_file" "$tmpdir/block-key-sha256" - -# GET must be explicit: this verifies both authorization and model visibility. -wait_model_access "$BASE_URL" "$block_headers" primary -if [[ "$MODE" == "ha" ]]; then - wait_model_access "$PEER_URL" "$block_headers" peer - echo "PASS HA pre-revocation: second replica accepted the virtual key." -fi - -jq -n --rawfile model "$tmpdir/model-name" --rawfile prompt "$tmpdir/prompt-marker" --rawfile response_marker "$tmpdir/response-marker" '{model: ($model | rtrimstr("\n")), messages: [{role: "user", content: (($prompt | rtrimstr("\n")) + " Return exactly this marker: " + ($response_marker | rtrimstr("\n")))}], max_tokens: 32}' > "$tmpdir/chat-request.json" -chmod 600 "$tmpdir/chat-request.json" -request_data_post "$BASE_URL" "$block_headers" /v1/chat/completions "$tmpdir/chat-request.json" "$tmpdir/chat.json" -jq -e --rawfile response_marker "$tmpdir/response-marker" '.choices[0].message.content | type == "string" and contains($response_marker | rtrimstr("\n"))' "$tmpdir/chat.json" >/dev/null -chat_result="passed" - -if [[ "$MODE" == "ha" ]]; then - peer_completion_ready=false - for i in $(seq 1 30); do - if request_data_post "$PEER_URL" "$block_headers" /v1/chat/completions "$tmpdir/chat-request.json" "$tmpdir/peer-chat.json" \ - && jq -e '.choices[0].message.content | type == "string"' "$tmpdir/peer-chat.json" >/dev/null; then - peer_completion_ready=true - break - fi - sleep 1 - done - [[ "$peer_completion_ready" == true ]] || { echo "second replica did not load the newly created model for completion" >&2; exit 1; } - echo "PASS HA model propagation: second replica completed with the newly created model." -fi - -jq '. + {stream: true}' "$tmpdir/chat-request.json" > "$tmpdir/stream-request.json" -chmod 600 "$tmpdir/stream-request.json" -request_data_post "$BASE_URL" "$block_headers" /v1/chat/completions "$tmpdir/stream-request.json" "$tmpdir/stream.txt" -rg -q '^data: ' "$tmpdir/stream.txt" -stream_result="passed" - -jq -n --rawfile model "$tmpdir/model-name" --rawfile prompt "$tmpdir/prompt-marker" --rawfile tool_name "$tmpdir/tool-name" --rawfile tool_marker "$tmpdir/tool-marker" '{model: ($model | rtrimstr("\n")), messages: [{role: "user", content: ($prompt | rtrimstr("\n"))}], tools: [{type: "function", function: {name: ($tool_name | rtrimstr("\n")), description: ($tool_marker | rtrimstr("\n")), parameters: {type: "object", properties: {answer: {type: "integer", description: ($tool_marker | rtrimstr("\n"))}}, required: ["answer"]}}}], tool_choice: {type: "function", function: {name: ($tool_name | rtrimstr("\n"))}}, thinking: {type: "disabled"}, max_tokens: 32}' > "$tmpdir/tool-request.json" -chmod 600 "$tmpdir/tool-request.json" -request_data_post "$BASE_URL" "$block_headers" /v1/chat/completions "$tmpdir/tool-request.json" "$tmpdir/tool.json" -if ! jq -e '.choices[0].message.tool_calls | type == "array" and length > 0' "$tmpdir/tool.json" >/dev/null; then - echo "tool request returned no tool_calls" >&2 - exit 1 -fi -tool_result="passed" -assert_spend -usage_result="passed" -smoke_phase="shared_limit_and_redis_recovery" - -if [[ "$MODE" == "ha" ]]; then - # One request reaches replica 1; the same key must be RPM-limited on replica 2. - write_private_value "$tmpdir/rate-key-alias" "${test_prefix}-rate" - make_key_payload "$tmpdir/rate-key-alias" "$tmpdir/rate-key-create.json" - jq '.rpm_limit = 1' "$tmpdir/rate-key-create.json" > "$tmpdir/rate-key-limited.json" - chmod 600 "$tmpdir/rate-key-limited.json" - request_admin POST /key/generate "$tmpdir/rate-key-limited.json" "$tmpdir/rate-key.json" - make_key_header "$tmpdir/rate-key.json" "$rate_key_file" "$rate_headers" - rate_key_created=true - make_key_action_payload delete "$rate_key_file" "$tmpdir/rate-key-cleanup.json" - rate_limiter_since="$(date -u +%Y-%m-%dT%H:%M:%SZ)" - request_data_post "$BASE_URL" "$rate_headers" /v1/chat/completions "$tmpdir/chat-request.json" "$tmpdir/rate-first.json" - rate_code="$(curl --silent --show-error --max-time 30 --request POST "$PEER_URL/v1/chat/completions" --header "@$rate_headers" --data-binary "@$tmpdir/chat-request.json" --dump-header "$tmpdir/rate-second.headers" --output "$tmpdir/rate-second.json" --write-out '%{http_code}' || true)" - chmod 600 "$tmpdir/rate-second.headers" - assert_proxy_limiter_response "$tmpdir/rate-second.json" "$tmpdir/rate-second.headers" "$rate_code" rpm "$rate_limiter_since" || { echo "shared RPM limiter was not a LiteLLM proxy 429" >&2; exit 1; } - shared_rpm_limit_result="passed" - limiter_source="litellm_proxy" - echo "PASS HA shared RPM: peer rejected the second request with 429." - - # The second TPM-limited request goes to the other replica, so its 429 proves - # that the Redis-backed limiter is not replica-local. - write_private_value "$tmpdir/enforcement-key-alias" "${test_prefix}-tpm" - make_key_payload "$tmpdir/enforcement-key-alias" "$tmpdir/enforcement-key-create.json" - # Use a separate shared TPM gate. It is enforced by the Redis-backed limiter - # before the second replica accepts a request, unlike asynchronous SpendLog - # persistence which cannot be used as an admission-control proof. - jq '.tpm_limit = 64' "$tmpdir/enforcement-key-create.json" > "$tmpdir/enforcement-key-limited.json" - chmod 600 "$tmpdir/enforcement-key-limited.json" - request_admin POST /key/generate "$tmpdir/enforcement-key-limited.json" "$tmpdir/enforcement-key.json" - make_key_header "$tmpdir/enforcement-key.json" "$enforcement_key_file" "$enforcement_headers" - enforcement_key_created=true - make_key_action_payload delete "$enforcement_key_file" "$tmpdir/enforcement-key-cleanup.json" - tpm_limiter_since="$(date -u +%Y-%m-%dT%H:%M:%SZ)" - request_data_post "$BASE_URL" "$enforcement_headers" /v1/chat/completions "$tmpdir/chat-request.json" "$tmpdir/budget-first.json" - tpm_code="$(curl --silent --show-error --max-time 30 --request POST "$PEER_URL/v1/chat/completions" --header "@$enforcement_headers" --data-binary "@$tmpdir/chat-request.json" --dump-header "$tmpdir/tpm-second.headers" --output "$tmpdir/tpm-second.json" --write-out '%{http_code}' || true)" - chmod 600 "$tmpdir/tpm-second.headers" - assert_proxy_limiter_response "$tmpdir/tpm-second.json" "$tmpdir/tpm-second.headers" "$tpm_code" tpm "$tpm_limiter_since" || { echo "shared TPM limiter was not a LiteLLM proxy 429" >&2; exit 1; } - shared_tpm_limit_result="passed" - echo "PASS HA shared TPM enforcement: peer rejected the second request with 429." -fi - -make_key_action_payload block "$block_key_file" "$tmpdir/block-key.json" -smoke_phase="revocation" -request_admin POST /key/block "$tmpdir/block-key.json" "$tmpdir/block-response.json" -wait_for_rejection "$block_headers" block -block_result="passed" - -# Delete is validated with a different, previously unblocked key. -write_private_value "$tmpdir/delete-key-alias" "${test_prefix}-delete" -make_key_payload "$tmpdir/delete-key-alias" "$tmpdir/delete-key-create.json" -request_admin POST /key/generate "$tmpdir/delete-key-create.json" "$tmpdir/delete-key.json" -make_key_header "$tmpdir/delete-key.json" "$delete_key_file" "$delete_headers" -delete_key_created=true -make_key_action_payload delete "$delete_key_file" "$tmpdir/delete-key-cleanup.json" -wait_model_access "$BASE_URL" "$delete_headers" delete-primary -if [[ "$MODE" == "ha" ]]; then - wait_model_access "$PEER_URL" "$delete_headers" delete-peer - echo "PASS HA pre-delete: second replica accepted the independent virtual key." -fi -request_admin POST /key/delete "$tmpdir/delete-key-cleanup.json" "$tmpdir/delete-response.json" -wait_for_rejection "$delete_headers" delete -delete_key_created=false -delete_result="passed" - -runtime_security_check -security_scan_result="passed" -content_logging_scan_result="passed" -smoke_phase="completed" -echo "PASS complete: user/credential/model/key cleanup, explicit GET models, chat/stream/tool, token-bearing spend, and independent block/delete propagation." diff --git a/docker_litellm/demo/scripts/smoke-redis-recovery.sh b/docker_litellm/demo/scripts/smoke-redis-recovery.sh deleted file mode 100755 index 132ab12..0000000 --- a/docker_litellm/demo/scripts/smoke-redis-recovery.sh +++ /dev/null @@ -1,130 +0,0 @@ -#!/usr/bin/env bash -# Isolated Redis outage/recovery proof for an already-running HA stack. -set -euo pipefail - -script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -demo_dir="$(cd "${script_dir}/.." && pwd)" -source "${script_dir}/verification-lib.sh" -env_file="${LITELLM_SMOKE_ENV_FILE:-${demo_dir}/.env}" -summary_file="${LITELLM_REDIS_SUMMARY_FILE:-${demo_dir}/artifacts/p1-redis-recovery.json}" -verification_run_id="${VERIFICATION_RUN_ID:-standalone}" -recovery_timeout="" -container="" -network="" -phase="initializing" -result="failed" -security_scan="not_run" -post_recovery_call="not_run" -tmpdir="" -image_ref="" - -verification_invalidate_report "$summary_file" - -write_summary() { - local summary_tmp - umask 077 - mkdir -p "$(dirname "$summary_file")" - chmod 700 "$(dirname "$summary_file")" - summary_tmp="${summary_file}.tmp.$$" - jq -n \ - --arg commit "$(git -C "$demo_dir/../.." rev-parse HEAD)" \ - --arg image_id "$(docker image inspect "$image_ref" --format '{{.Id}}' 2>/dev/null || true)" \ - --arg tested_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ - --arg run_id "$verification_run_id" --arg result "$result" --arg phase "$phase" --arg security_scan "$security_scan" \ - --arg post_recovery_call "$post_recovery_call" \ - '{verification_run_id:$run_id,commit:$commit,image_id:$image_id,tested_at:$tested_at,mode:"ha",result:$result,phase:$phase,redis_recovery:$result,post_recovery_call:$post_recovery_call,security_scan:$security_scan,content_redacted:true}' \ - > "$summary_tmp" && chmod 600 "$summary_tmp" && mv "$summary_tmp" "$summary_file" -} - -cleanup() { - local exit_code=$? - trap - EXIT - set +e - if [[ -n "$container" && -n "$network" ]]; then - docker network connect --alias redis "$network" "$container" >/dev/null 2>&1 || true - fi - if (( exit_code != 0 )); then result="failed"; fi - write_summary || exit_code=1 - [[ -z "$tmpdir" ]] || rm -rf "$tmpdir" - return "$exit_code" -} -trap cleanup EXIT - -[[ -f "$env_file" ]] || { echo "missing ignored local environment file" >&2; exit 2; } -umask 077 -tmpdir="$(mktemp -d "${TMPDIR:-/tmp}/litellm-redis-recovery.XXXXXX")" -chmod 700 "$tmpdir" -verification_prepare_environment "$env_file" "${demo_dir}/docker-compose.litellm.yml" "$tmpdir" -master_key="$(verification_env LITELLM_MASTER_KEY)" -image_ref="$(verification_env LITELLM_IMAGE)" -: "${master_key:?missing LITELLM_MASTER_KEY in effective Compose environment}" -: "${image_ref:?missing LITELLM_IMAGE in effective Compose environment}" -# Keep the recovery proof aligned with the same optional host-port overrides -# that Compose uses for the two proxy replicas. These are non-secret routing -# values; credentials remain in the private header file below. - publish_host="$(verification_env LITELLM_PUBLISH_HOST)" - litellm_1_port="$(verification_env LITELLM_1_PORT)" - litellm_2_port="$(verification_env LITELLM_2_PORT)" - recovery_timeout="$(verification_env REDIS_CIRCUIT_BREAKER_RECOVERY_TIMEOUT)" - recovery_timeout="${recovery_timeout:-5}" - [[ "$recovery_timeout" =~ ^[0-9]+$ ]] || { echo "invalid Redis recovery timeout" >&2; exit 2; } -master_file="$tmpdir/master-key" -headers_file="$tmpdir/admin.headers" -printf '%s' "$master_key" > "$master_file" -chmod 600 "$master_file" -{ printf 'Authorization: Bearer '; tr -d '\r\n' < "$master_file"; printf '\n'; } > "$headers_file" -chmod 600 "$headers_file" -unset master_key - -probe() { - docker exec "$1" python3 -c 'import redis; p=open("/run/secrets/redis_password").read().strip(); assert redis.Redis(host="redis", password=p, socket_connect_timeout=2, socket_timeout=2).ping()' -} - -runtime_security_check() { - local litellm_1_container litellm_2_container - litellm_1_container="$(docker compose --env-file "$env_file" -f "$demo_dir/docker-compose.litellm.yml" ps -q litellm-1)" - litellm_2_container="$(docker compose --env-file "$env_file" -f "$demo_dir/docker-compose.litellm.yml" ps -q litellm-2)" - [[ -n "$litellm_1_container" && -n "$litellm_2_container" ]] || return 1 - docker inspect "$container" > "$tmpdir/redis-inspect.json" - docker inspect "$litellm_1_container" "$litellm_2_container" > "$tmpdir/litellm-inspect.json" - docker exec "$container" ps -eo args > "$tmpdir/redis-processes.txt" - ! rg -q -- '--requirepass[[:space:]]+[^[:space:]]+' "$tmpdir/redis-processes.txt" && - ! rg -q 'UPSTREAM_(API_KEY|BASE_URL|MODEL)=' "$tmpdir/litellm-inspect.json" && - rg -q '/run/secrets/redis_password' "$tmpdir/redis-inspect.json" && - verification_assert_file_mode "$headers_file" 600 "Redis recovery authorization header" -} - -container="$(docker compose --env-file "$env_file" -f "$demo_dir/docker-compose.litellm.yml" ps -q redis)" -[[ -n "$container" ]] || { phase="redis_not_found"; exit 1; } -network="$(docker inspect -f '{{range $name, $_ := .NetworkSettings.Networks}}{{$name}}{{end}}' "$container")" -[[ -n "$network" ]] || { phase="network_not_found"; exit 1; } -litellm_1_container="$(docker compose --env-file "$env_file" -f "$demo_dir/docker-compose.litellm.yml" ps -q litellm-1)" -litellm_2_container="$(docker compose --env-file "$env_file" -f "$demo_dir/docker-compose.litellm.yml" ps -q litellm-2)" -[[ -n "$litellm_1_container" && -n "$litellm_2_container" ]] || { phase="litellm_not_found"; exit 1; } - -phase="disconnect" -docker network disconnect "$network" "$container" -if probe "$litellm_1_container" >/dev/null 2>&1 || probe "$litellm_2_container" >/dev/null 2>&1; then - phase="probe_unexpectedly_succeeded" - exit 1 -fi - -phase="recover" -docker network connect --alias redis "$network" "$container" -sleep $((recovery_timeout + 1)) -probe "$litellm_1_container" -probe "$litellm_2_container" - -# This is an authenticated LiteLLM call after recovery, not only a socket PING. -curl --silent --show-error --fail --max-time 20 --request GET "http://${publish_host}:${litellm_1_port}/v1/models" \ - --header "@$headers_file" --output "$tmpdir/models-1.json" -curl --silent --show-error --fail --max-time 20 --request GET "http://${publish_host}:${litellm_2_port}/v1/models" \ - --header "@$headers_file" --output "$tmpdir/models-2.json" -jq -e '.data | type == "array"' "$tmpdir/models-1.json" "$tmpdir/models-2.json" >/dev/null -post_recovery_call="passed" - -runtime_security_check -security_scan="passed" -phase="completed" -result="passed" -echo "PASS Redis recovery: both bounded probes failed during outage, recovered, and both LiteLLM replicas served an authenticated GET afterward." diff --git a/docker_litellm/demo/scripts/test-secret-boundary.sh b/docker_litellm/demo/scripts/test-secret-boundary.sh deleted file mode 100755 index c0717ec..0000000 --- a/docker_litellm/demo/scripts/test-secret-boundary.sh +++ /dev/null @@ -1,194 +0,0 @@ -#!/usr/bin/env bash -# PH-1 regression gate for the LiteLLM Compose credential boundary. It uses -# generated local placeholders only, never reads demo/.env, and leaves no -# containers, volumes, networks, or host temporary credential files behind. -set -euo pipefail - -script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -demo_dir="$(cd "${script_dir}/.." && pwd)" -compose_file="${demo_dir}/docker-compose.litellm.yml" -image_ref="${LITELLM_SECRET_BOUNDARY_IMAGE:-quay.io/labnow/litellm:1.97.0-ead62528e607}" -run_id="$(python3 -c 'import secrets; print(secrets.token_hex(8))')" -project="ph1-secret-boundary-${run_id}" -compose_project="${project}-svc-litellm" -network_name="${project}-svc-litellm-net" -litellm_container="${project}-svc-litellm-1" -litellm_peer_container="${project}-svc-litellm-2" -publish_port="${LITELLM_SECRET_BOUNDARY_PORT:-4100}" -tmpdir="$(mktemp -d "${TMPDIR:-/tmp}/litellm-secret-boundary.XXXXXX")" -env_file="${tmpdir}/runtime.env" -headers_file="${tmpdir}/admin.headers" -user_payload="${tmpdir}/user-create.json" -user_response="${tmpdir}/user-create-response.json" -user_delete_payload="${tmpdir}/user-delete.json" -generate_payload="${tmpdir}/generate.json" -generate_response="${tmpdir}/generate-response.json" -generated_key_file="${tmpdir}/generated.key" -delete_payload="${tmpdir}/delete.json" -migration_output="${tmpdir}/migration.log" -report_file="${LITELLM_SECRET_BOUNDARY_REPORT_FILE:-}" -started=false -compose_config_result="not_run" -readiness_result="not_run" -inspect_result="not_run" -argv_result="not_run" -logs_result="not_run" -temporary_files_result="not_run" -management_smoke_result="not_run" -management_http_status="not_run" - -cleanup() { - local exit_code=$? - trap - EXIT - if [[ "$started" == true ]]; then - docker compose --env-file "$env_file" -f "$compose_file" --profile single down -v --remove-orphans >/dev/null 2>&1 || exit_code=1 - fi - rm -rf "$tmpdir" - if docker ps -a --format '{{.Names}}' | rg -q "^${litellm_container}$|^${litellm_peer_container}$"; then - echo "FAIL cleanup: PH-1 LiteLLM container remains" >&2 - exit_code=1 - fi - if docker network inspect "$network_name" >/dev/null 2>&1; then - echo "FAIL cleanup: PH-1 network remains" >&2 - exit_code=1 - fi - if [[ -n "$report_file" ]]; then - mkdir -p "$(dirname "$report_file")" - printf '{"result":"%s","compose_config":"%s","readiness":"%s","inspect":"%s","argv":"%s","logs":"%s","temporary_files":"%s","management_smoke":"%s","management_http_status":"%s","cleanup":"%s"}\n' \ - "$([[ "$exit_code" == 0 ]] && echo passed || echo failed)" "$compose_config_result" "$readiness_result" "$inspect_result" "$argv_result" "$logs_result" "$temporary_files_result" "$management_smoke_result" "$management_http_status" "$([[ "$exit_code" == 0 ]] && echo passed || echo failed)" > "$report_file" - chmod 600 "$report_file" - fi - exit "$exit_code" -} -trap cleanup EXIT - -need() { command -v "$1" >/dev/null || { echo "required command missing: $1" >&2; exit 2; }; } -need docker -need jq -need rg -need openssl - -# Refuse to attach the boundary test to an existing instance-specific network. -if docker network inspect "$network_name" >/dev/null 2>&1; then - echo "refusing to reuse existing $network_name" >&2 - exit 2 -fi - -umask 077 -export PROFILE_ENV="$project" -master_key="sk-$(openssl rand -hex 24)" -postgres_password="$(openssl rand -hex 24)" -redis_password="$(openssl rand -hex 24)" -printf 'LITELLM_IMAGE=%s\nLITELLM_MASTER_KEY=%s\nPOSTGRES_USER=litellm\nPOSTGRES_PASSWORD=%s\nPOSTGRES_DB=litellm\nREDIS_PASSWORD=%s\nLITELLM_1_CONTAINER_NAME=%s\nLITELLM_2_CONTAINER_NAME=%s\nLITELLM_1_PORT=%s\nLITELLM_2_PORT=4101\nLITELLM_PUBLISH_HOST=127.0.0.1\n' \ - "$image_ref" "$master_key" "$postgres_password" "$redis_password" "$litellm_container" "$litellm_peer_container" "$publish_port" > "$env_file" -chmod 600 "$env_file" - -docker compose --env-file "$env_file" -f "$compose_file" --profile single config --format json | jq -e ' - . as $config | - ([.services | to_entries[] | select(.key == "litellm-1" or .key == "litellm-2" or .key == "litellm-migrate") | .value.environment // {} | keys[] | select(. == "LITELLM_MASTER_KEY" or . == "DATABASE_URL" or . == "POSTGRES_PASSWORD")] | length == 0) - and ($config.services.postgres.environment | has("POSTGRES_PASSWORD") | not) -' >/dev/null -compose_config_result="passed" -echo "PASS compose config: service environment omits LITELLM_MASTER_KEY, DATABASE_URL, and POSTGRES_PASSWORD." - -started=true -docker compose --env-file "$env_file" -f "$compose_file" up -d --wait postgres redis >/dev/null -docker compose --env-file "$env_file" -f "$compose_file" --profile migrate run --rm --no-deps litellm-migrate >"$migration_output" 2>&1 -chmod 600 "$migration_output" -if rg -Fq -- "$master_key" "$migration_output" \ - || rg -Fq -- "$postgres_password" "$migration_output" \ - || rg -Fq -- "$redis_password" "$migration_output"; then - echo "FAIL logs: credential value is present in migration output" >&2 - exit 1 -fi -docker compose --env-file "$env_file" -f "$compose_file" --profile single up -d litellm-1 >/dev/null -for attempt in $(seq 1 60); do - if curl --silent --fail --max-time 3 "http://127.0.0.1:${publish_port}/health/readiness" | jq -e '.status == "healthy" and .db == "connected"' >/dev/null; then - break - fi - sleep 2 -done -if ! curl --silent --fail --max-time 3 "http://127.0.0.1:${publish_port}/health/readiness" | jq -e '.status == "healthy" and .db == "connected"' >/dev/null; then - echo "FAIL readiness: LiteLLM did not report a healthy PostgreSQL connection" >&2 - exit 1 -fi -readiness_result="passed" -echo "PASS readiness: LiteLLM and PostgreSQL are healthy." - -assert_metadata_boundary() { - local container="$1" forbidden_key="$2" forbidden_value="$3" - if docker inspect "$container" --format '{{range .Config.Env}}{{println .}}{{end}}' | rg -q "^${forbidden_key}="; then - echo "FAIL inspect: ${forbidden_key} remains in ${container} metadata" >&2 - return 1 - fi - if docker inspect "$container" --format '{{range .Config.Env}}{{println .}}{{end}}' | rg -Fq -- "$forbidden_value"; then - echo "FAIL inspect: credential value remains in ${container} metadata" >&2 - return 1 - fi -} - -assert_metadata_boundary "$litellm_container" LITELLM_MASTER_KEY "$master_key" -assert_metadata_boundary "$litellm_container" DATABASE_URL "$postgres_password" -assert_metadata_boundary "${compose_project}-postgres-1" POSTGRES_PASSWORD "$postgres_password" -inspect_result="passed" -echo "PASS inspect: no management key, database URL, or PostgreSQL password value in container metadata." - -for container in "$litellm_container" "${compose_project}-postgres-1" "${compose_project}-redis-1"; do - if docker top "$container" -eo args | rg -Fq -- "$master_key" \ - || docker top "$container" -eo args | rg -Fq -- "$postgres_password" \ - || docker top "$container" -eo args | rg -Fq -- "$redis_password"; then - echo "FAIL ps/argv: credential value is present in ${container}" >&2 - exit 1 - fi -done -argv_result="passed" -echo "PASS ps/argv: no credential values in container command lines." - -if docker compose --env-file "$env_file" -f "$compose_file" --profile single logs --no-color | rg -Fq -- "$master_key" \ - || docker compose --env-file "$env_file" -f "$compose_file" --profile single logs --no-color | rg -Fq -- "$postgres_password" \ - || docker compose --env-file "$env_file" -f "$compose_file" --profile single logs --no-color | rg -Fq -- "$redis_password"; then - echo "FAIL logs: credential value is present in Compose logs" >&2 - exit 1 -fi -logs_result="passed" -echo "PASS logs: no generated credential values in Compose logs." - -docker exec "$litellm_container" /bin/sh -ec ' - set -eu - for secret_file in /run/secrets/litellm_master_key /run/secrets/postgres_password /run/secrets/redis_password; do - secret="$(cat "$secret_file")" - if grep -R -F -q -- "$secret" /tmp /opt/litellm 2>/dev/null; then - exit 1 - fi - done -' -temporary_files_result="passed" -echo "PASS temporary files: no credential copies outside Docker Secret mounts." - -printf 'Authorization: Bearer %s\nContent-Type: application/json\n' "$master_key" > "$headers_file" -probe_user="ph1-secret-boundary-${run_id}-user" -jq -n --arg user "$probe_user" '{user_id:$user, auto_create_key:false, user_role:"internal_user"}' > "$user_payload" -jq -n --arg user "$probe_user" '{user_ids:[$user]}' > "$user_delete_payload" -jq -n --arg user "$probe_user" '{key_alias:"ph1-secret-boundary-probe", user_id:$user, duration:"1m", models:[]}' > "$generate_payload" -chmod 600 "$headers_file" "$user_payload" "$user_delete_payload" "$generate_payload" -management_http_status="$(curl --silent --show-error --max-time 30 --request POST "http://127.0.0.1:${publish_port}/user/new" \ - --header "@${headers_file}" --data-binary "@${user_payload}" --output "$user_response" --write-out '%{http_code}' || true)" -if [[ "$management_http_status" != 200 ]]; then - echo "FAIL management smoke: /user/new returned HTTP ${management_http_status:-transport_error}" >&2 - exit 1 -fi -chmod 600 "$user_response" -jq -e --arg user "$probe_user" '.user_id == $user' "$user_response" >/dev/null -curl --silent --show-error --fail --max-time 30 --request POST "http://127.0.0.1:${publish_port}/key/generate" \ - --header "@${headers_file}" --data-binary "@${generate_payload}" --output "$generate_response" -chmod 600 "$generate_response" -jq -er '.key | select(type == "string" and length > 0)' "$generate_response" > "$generated_key_file" -chmod 600 "$generated_key_file" -jq -n --rawfile key "$generated_key_file" '{keys:[($key | rtrimstr("\n"))]}' > "$delete_payload" -chmod 600 "$delete_payload" -curl --silent --show-error --fail --max-time 30 --request POST "http://127.0.0.1:${publish_port}/key/delete" \ - --header "@${headers_file}" --data-binary "@${delete_payload}" --output /dev/null -curl --silent --show-error --fail --max-time 30 --request POST "http://127.0.0.1:${publish_port}/user/delete" \ - --header "@${headers_file}" --data-binary "@${user_delete_payload}" --output /dev/null -management_smoke_result="passed" -echo "PASS management smoke: authenticated user and key create/delete endpoints completed." diff --git a/docker_litellm/demo/scripts/test-verification-gates.sh b/docker_litellm/demo/scripts/test-verification-gates.sh deleted file mode 100755 index ed3e3eb..0000000 --- a/docker_litellm/demo/scripts/test-verification-gates.sh +++ /dev/null @@ -1,187 +0,0 @@ -#!/usr/bin/env bash -# Reproducible negative checks: neither stale PASS reports nor failed cleanup -# may be accepted by the aggregate gate. -set -euo pipefail - -script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -demo_dir="$(cd "${script_dir}/.." && pwd)" -with_running_stack=false -[[ "${1:-}" != "--with-running-stack" ]] || with_running_stack=true -[[ $# -eq 0 || "$with_running_stack" == true ]] || { echo "Usage: $0 [--with-running-stack]" >&2; exit 2; } -tmpdir="$(mktemp -d "${TMPDIR:-/tmp}/litellm-gates.XXXXXX")" -chmod 700 "$tmpdir" -trap 'rm -rf "$tmpdir"' EXIT -run_id="${VERIFICATION_RUN_ID:-p1-$(python3 -c 'import secrets; print(secrets.token_hex(16))')}" -[[ "$run_id" =~ ^p1-[a-f0-9]{32}$ ]] || { echo "invalid verification run id" >&2; exit 2; } - -# A deliberately stale-but-well-shaped set must be rejected when the current -# run id differs. No service or .env is read by this test. -for name in p1-migration-summary p1-single-summary p1-ha-summary p1-redis-recovery; do - jq -n --arg stale 'p1-00000000000000000000000000000000' \ - '{verification_run_id:$stale,commit:"stale",image_id:"stale",mode:"single",result:"passed",phase:"completed",content_redacted:true}' > "$tmpdir/${name}.json" -done -if VERIFICATION_RUN_ID="$run_id" LITELLM_ARTIFACTS_DIR="$tmpdir" LITELLM_AGGREGATE_SUMMARY_FILE="$tmpdir/final.json" \ - "$script_dir/aggregate-verification-summary.sh" >/dev/null 2>&1; then - echo "aggregate accepted stale PASS reports" >&2 - exit 1 -fi -[[ ! -e "$tmpdir/final.json" ]] || { echo "aggregate left a stale final report" >&2; exit 1; } - -# A producer must overwrite a pre-existing PASS before even checking a missing -# environment file. The resulting report is explicitly non-passing. -jq -n --arg run_id "$run_id" '{verification_run_id:$run_id,result:"passed",phase:"completed"}' > "$tmpdir/precondition.json" -if VERIFICATION_RUN_ID="$run_id" LITELLM_SMOKE_ENV_FILE="$tmpdir/absent.env" LITELLM_SMOKE_SUMMARY_FILE="$tmpdir/precondition.json" \ - "$script_dir/smoke-baseline.sh" --mode single >/dev/null 2>&1; then - echo "smoke unexpectedly accepted a missing environment file" >&2 - exit 1 -fi -jq -e '.result == "failed" and .phase == "precondition_failed"' "$tmpdir/precondition.json" >/dev/null - -# Security inspection must remain read-only on both outcomes. Force a static -# failure and prove that its target report is byte-for-byte unchanged. -security_report="$tmpdir/security-existing.json" -printf '%s\n' '{"preserved":true}' > "$security_report" -security_before="$(shasum -a 256 "$security_report" | awk '{print $1}')" -if LITELLM_SECURITY_CHECK_FORCE_FAILURE=1 LITELLM_SMOKE_SUMMARY_FILE="$security_report" \ - "$script_dir/smoke-baseline.sh" --security-check >/dev/null 2>&1; then - echo "forced security failure unexpectedly passed" >&2 - exit 1 -fi -security_after="$(shasum -a 256 "$security_report" | awk '{print $1}')" -[[ "$security_before" == "$security_after" ]] || { echo "failed security check modified a report" >&2; exit 1; } - -# Cleanup failures must never be normalized to PASS. This checks the curl -# transport invariant directly without sending a request. -rg -q 'cleanup_request_admin.*\(\)' "$script_dir/smoke-baseline.sh" -rg -q -- 'curl_args=\(--silent --show-error --fail' "$script_dir/smoke-baseline.sh" -! rg -n -- 'source "\$env_file"|source "\$\{env_file\}"' "$script_dir/run-migration.sh" "$script_dir/smoke-redis-recovery.sh" -rg -q 'config --environment > "\$verification_environment_file"' "$script_dir/verification-lib.sh" - -# Exercise the exact limiter-source predicate used by smoke-baseline.sh. The -# fixture replaces only `docker compose ... logs`; no service, .env, or -# upstream call is involved. This keeps a provider 429 from being mistaken for -# the proxy's Redis-backed RPM/TPM limiter. -load_smoke_limiter_predicate() { - # Keep this extraction intentionally narrow: the production function remains - # the single source of truth, while the fixture provides its log dependency. - local predicate_file="$tmpdir/smoke-limiter-predicate.sh" - sed -n '/^assert_proxy_limiter_response() {/,/^}$/p' "$script_dir/smoke-baseline.sh" > "$predicate_file" - chmod 600 "$predicate_file" - source "$predicate_file" -} - -fixture_docker() { - # assert_proxy_limiter_response only asks Docker for a bounded Compose log - # stream. Do not forward any fixture argument to a real Docker process. - cat "$limiter_fixture_log" -} - -assert_limiter_fixture() { - local fixture_name="$1" response_file="$2" headers_file="$3" limit_kind="$4" expected_result="$5" - local limiter_fixture_log="$tmpdir/${fixture_name}.logs" - local ENV_FILE="$tmpdir/fixture.env" DEMO_DIR="$demo_dir" - printf '%s\n' 'POST /v1/chat/completions HTTP/1.1" 429' > "$limiter_fixture_log" - docker() { fixture_docker "$@"; } - load_smoke_limiter_predicate - if assert_proxy_limiter_response "$response_file" "$headers_file" 429 "$limit_kind" '2026-01-01T00:00:00Z'; then - [[ "$expected_result" == pass ]] || { echo "accepted $fixture_name fixture" >&2; exit 1; } - else - [[ "$expected_result" == reject ]] || { echo "rejected valid $fixture_name fixture" >&2; exit 1; } - fi - unset -f docker -} - -# A fake provider/upstream response has a conventional 429 body and headers, -# even with an otherwise matching local access-log line. It lacks LiteLLM's -# proxy-only rate_limit_type and Limit type evidence and must be rejected. -upstream_body="$tmpdir/upstream-429.json" -upstream_headers="$tmpdir/upstream-429.headers" -printf '%s\n' '{"error":{"message":"upstream provider rate limit exceeded","type":"rate_limit_error"}}' > "$upstream_body" -printf '%s\n' 'HTTP/1.1 429 Too Many Requests' 'retry-after: 1' 'x-ratelimit-remaining-requests: 0' > "$upstream_headers" -assert_limiter_fixture upstream-429 "$upstream_body" "$upstream_headers" rpm reject - -# LiteLLM's proxy limiter fixture has the stable matching header and detail; -# this proves the fixture harness accepts the same positive RPM evidence that -# the real HA smoke requires. -proxy_body="$tmpdir/litellm-proxy-429.json" -proxy_headers="$tmpdir/litellm-proxy-429.headers" -printf '%s\n' '{"detail":"Rate limit exceeded. Limit type: requests"}' > "$proxy_body" -printf '%s\n' 'HTTP/1.1 429 Too Many Requests' 'rate_limit_type: requests' > "$proxy_headers" -assert_limiter_fixture litellm-proxy-429 "$proxy_body" "$proxy_headers" rpm pass - -# RPM and TPM evidence are type-specific. A 429 with a valid-looking proxy -# shape but the wrong type must not satisfy either opposite limiter assertion. -rpm_mismatch_body="$tmpdir/rpm-type-mismatch.json" -rpm_mismatch_headers="$tmpdir/rpm-type-mismatch.headers" -printf '%s\n' '{"detail":"Rate limit exceeded. Limit type: tokens"}' > "$rpm_mismatch_body" -printf '%s\n' 'HTTP/1.1 429 Too Many Requests' 'rate_limit_type: tokens' > "$rpm_mismatch_headers" -assert_limiter_fixture rpm-type-mismatch "$rpm_mismatch_body" "$rpm_mismatch_headers" rpm reject - -tpm_mismatch_body="$tmpdir/tpm-type-mismatch.json" -tpm_mismatch_headers="$tmpdir/tpm-type-mismatch.headers" -printf '%s\n' '{"detail":"Rate limit exceeded. Limit type: requests"}' > "$tpm_mismatch_body" -printf '%s\n' 'HTTP/1.1 429 Too Many Requests' 'rate_limit_type: requests' > "$tpm_mismatch_headers" -assert_limiter_fixture tpm-type-mismatch "$tpm_mismatch_body" "$tpm_mismatch_headers" tpm reject -echo "PASS limiter-source fixtures: upstream 429 and RPM/TPM type mismatches rejected; LiteLLM proxy 429 accepted." - -# Compose's dotenv parser must not evaluate shell substitutions. This isolated -# Compose file uses no project secrets and verifies the same config command -# that the runtime scripts use. -marker="$tmpdir/dotenv-command-substitution-ran" -printf '%s\n' 'services:' ' proof:' ' image: alpine:3.21' ' environment:' ' PROOF: ${PAYLOAD:?missing}' > "$tmpdir/compose.yml" -printf 'PAYLOAD=$(touch %s)\n' "$marker" > "$tmpdir/malicious.env" -docker compose --env-file "$tmpdir/malicious.env" -f "$tmpdir/compose.yml" config --environment > "$tmpdir/effective.env" -[[ ! -e "$marker" ]] || { echo "dotenv command substitution executed" >&2; exit 1; } -rg -Fq 'PAYLOAD=$(touch ' "$tmpdir/effective.env" - -# The aggregate gate must also reject structurally plausible but incomplete -# evidence: a missing timestamp, a failed migration, or a forged limiter claim. -started_at="2026-01-01T00:00:00Z" -current_commit="$(git -C "$demo_dir/../.." rev-parse HEAD)" -make_reports() { - jq -n --arg run_id "$run_id" --arg commit "$current_commit" --arg started_at "$started_at" \ - '{verification_run_id:$run_id,commit:$commit,image_id:"image",tested_at:$started_at,mode:"migration",result:"passed",phase:"completed",content_redacted:true,proxy_replicas_started:false}' > "$tmpdir/p1-migration-summary.json" - jq -n --arg run_id "$run_id" --arg commit "$current_commit" --arg started_at "$started_at" \ - '{verification_run_id:$run_id,commit:$commit,image_id:"image",tested_at:$started_at,mode:"migration",result:"passed",phase:"completed",concurrent_migration:true,actual_overlap:true,lock_wait_observed:true,exclusive_lock:true,max_lock_holders:1,migration_execution_count:2,proxy_replicas_started:false,content_redacted:true}' > "$tmpdir/p1-migration-concurrency.json" - jq -n --arg run_id "$run_id" --arg commit "$current_commit" --arg started_at "$started_at" \ - '{verification_run_id:$run_id,commit:$commit,image_id:"image",tested_at:$started_at,mode:"single",result:"passed",phase:"completed",content_redacted:true,migration:"passed",chat:"passed",stream:"passed",tool:"passed",usage:"passed",block:"passed",delete:"passed",cleanup:"passed",security_scan:"passed",content_logging_scan:"passed"}' > "$tmpdir/p1-single-summary.json" - jq -n --arg run_id "$run_id" --arg commit "$current_commit" --arg started_at "$started_at" \ - '{verification_run_id:$run_id,commit:$commit,image_id:"image",tested_at:$started_at,mode:"ha",result:"passed",phase:"completed",content_redacted:true,migration:"passed",chat:"passed",stream:"passed",tool:"passed",usage:"passed",block:"passed",delete:"passed",shared_rpm_limit:"passed",shared_tpm_limit:"passed",shared_spend_log_visibility:"passed",idempotency_recovery:"passed",limiter_source:"litellm_proxy",cleanup:"passed",security_scan:"passed",content_logging_scan:"passed"}' > "$tmpdir/p1-ha-summary.json" - jq -n --arg run_id "$run_id" --arg commit "$current_commit" --arg started_at "$started_at" \ - '{verification_run_id:$run_id,commit:$commit,image_id:"image",tested_at:$started_at,mode:"ha",result:"passed",phase:"completed",redis_recovery:"passed",content_redacted:true,security_scan:"passed"}' > "$tmpdir/p1-redis-recovery.json" -} -# These are deliberately rejected before any report can become final. The -# fixtures use the real checkout commit so each mutation exercises its named -# gate rather than merely the unrelated commit-mismatch guard. -make_reports -jq 'del(.tested_at)' "$tmpdir/p1-single-summary.json" > "$tmpdir/single.tmp" && mv "$tmpdir/single.tmp" "$tmpdir/p1-single-summary.json" -if VERIFICATION_RUN_ID="$run_id" VERIFICATION_STARTED_AT="$started_at" LITELLM_ARTIFACTS_DIR="$tmpdir" "$script_dir/aggregate-verification-summary.sh" >/dev/null 2>&1; then - echo "aggregate accepted a report without tested_at" >&2; exit 1 -fi -make_reports -jq '.result="failed"' "$tmpdir/p1-migration-summary.json" > "$tmpdir/migration.tmp" && mv "$tmpdir/migration.tmp" "$tmpdir/p1-migration-summary.json" -if VERIFICATION_RUN_ID="$run_id" VERIFICATION_STARTED_AT="$started_at" LITELLM_ARTIFACTS_DIR="$tmpdir" "$script_dir/aggregate-verification-summary.sh" >/dev/null 2>&1; then - echo "aggregate accepted a failed migration" >&2; exit 1 -fi -make_reports -jq '.limiter_source="provider"' "$tmpdir/p1-ha-summary.json" > "$tmpdir/ha.tmp" && mv "$tmpdir/ha.tmp" "$tmpdir/p1-ha-summary.json" -if VERIFICATION_RUN_ID="$run_id" VERIFICATION_STARTED_AT="$started_at" LITELLM_ARTIFACTS_DIR="$tmpdir" "$script_dir/aggregate-verification-summary.sh" >/dev/null 2>&1; then - echo "aggregate accepted a forged limiter claim" >&2; exit 1 -fi -make_reports -jq '.image_id="other-image"' "$tmpdir/p1-migration-concurrency.json" > "$tmpdir/concurrency.tmp" && mv "$tmpdir/concurrency.tmp" "$tmpdir/p1-migration-concurrency.json" -if VERIFICATION_RUN_ID="$run_id" VERIFICATION_STARTED_AT="$started_at" LITELLM_ARTIFACTS_DIR="$tmpdir" "$script_dir/aggregate-verification-summary.sh" >/dev/null 2>&1; then - echo "aggregate accepted a concurrency report with a mismatched image" >&2; exit 1 -fi - -if [[ "$with_running_stack" == true ]]; then - negative_summary="$tmpdir/cleanup-negative.json" - if VERIFICATION_RUN_ID="$run_id" LITELLM_SMOKE_SUMMARY_FILE="$negative_summary" \ - "$script_dir/smoke-baseline.sh" --mode single --cleanup-negative-test >/dev/null 2>&1; then - echo "cleanup negative test unexpectedly passed" >&2 - exit 1 - fi - jq -e '.result == "failed" and .cleanup == "failed" and .phase == "cleanup_negative_test"' "$negative_summary" >/dev/null -fi - -echo "PASS verification gates: stale reports, preconditions, dotenv substitutions and cleanup HTTP failures cannot pass." diff --git a/docker_openclaw/p6/scripts/test-p6-gates.sh b/docker_openclaw/p6/scripts/test-p6-gates.sh deleted file mode 100755 index 78bcc75..0000000 --- a/docker_openclaw/p6/scripts/test-p6-gates.sh +++ /dev/null @@ -1,22 +0,0 @@ -#!/usr/bin/env bash -# Static OpenClaw product contract checks. This test deliberately performs no -# Docker, network, credential, or historical-evidence operation. -set -euo pipefail - -script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -root="$(cd "${script_dir}/../../.." && pwd)" -dockerfile="${root}/docker_openclaw/openclaw.Dockerfile" -compose_file="${root}/docker_openclaw/demo/docker-compose.yml" - -rg -q '^ENV OPENCLAW_HOME=/root/\.openclaw$' "$dockerfile" -rg -q '^ENV OPENCLAW_STATE_DIR=\$\{OPENCLAW_HOME\}/data$' "$dockerfile" -rg -q '^VOLUME \["/root/\.openclaw/data", "/opt/node/pnpm/store"\]$' "$dockerfile" -rg -q '^EXPOSE 18789 18790$' "$dockerfile" -rg -q '^CMD \["start-openclaw\.sh"\]$' "$dockerfile" -rg -q '^services:$' "$compose_file" -rg -q '^ openclaw-gateway:$' "$compose_file" -rg -q '^ - "\$\{OPENCLAW_GATEWAY_PORT:-18789\}:18789"$' "$compose_file" -rg -q '^ - "\$\{OPENCLAW_BRIDGE_PORT:-18790\}:18790"$' "$compose_file" -! rg -q '/var/run/docker\.sock' "$compose_file" - -printf '%s\n' 'PASS P6 gates: OpenClaw image state, entrypoint, ports, and standard Compose avoid Docker socket access.' From 73f58c7b904be1ed9523dd03aac63bb9489f9911 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Fri, 21 Aug 2026 13:25:54 +0800 Subject: [PATCH 83/87] =?UTF-8?q?refactor:=20=E6=9C=8D=E5=8A=A1=E6=A8=A1?= =?UTF-8?q?=E5=9D=97=20demo/=20=E7=9B=AE=E5=BD=95=E7=BB=9F=E4=B8=80?= =?UTF-8?q?=E6=9B=B4=E5=90=8D=E4=B8=BA=20compose/?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit lab-dev 每个 docker_<服务>/ 模块的 demo/ 子目录存放的是该镜像的 Compose 运行包(compose + 配置 + .env.example + 配套脚本),按内容 本质更名为 compose/,与 work/(构建进镜像的脚本)形成清晰对照。 8 个模块 git mv,同步修正 .gitignore、README、AGENTS 约定文档与 脚本内路径引用;compose config 校验通过。 Co-Authored-By: Claude Fable 5 --- .gitignore | 2 +- docker_casdoor/{demo => compose}/docker-compose.yml | 0 docker_clash/{demo => compose}/docker-compose.yml | 0 docker_hermes/README.md | 8 ++++---- docker_hermes/{demo => compose}/.env.example | 4 ++-- docker_hermes/{demo => compose}/docker-compose.yml | 0 docker_hermes/p7/README.md | 2 +- docker_keycloak/{demo => compose}/docker-compose.yml | 0 docker_litellm/README.md | 8 ++++---- docker_litellm/{demo => compose}/.env.example | 2 +- docker_litellm/{demo => compose}/config.migrate.yaml | 0 docker_litellm/{demo => compose}/config.yaml | 0 .../{demo => compose}/docker-compose.litellm.yml | 0 .../scripts/aggregate-verification-summary.sh | 6 +++--- .../{demo => compose}/scripts/run-migration.sh | 12 ++++++------ .../{demo => compose}/scripts/verification-lib.sh | 0 .../scripts/verify-migration-concurrency.sh | 12 ++++++------ .../{demo => compose}/scripts/verify-p1.sh | 6 +++--- docker_nocobase/{demo => compose}/README.md | 2 +- docker_nocobase/{demo => compose}/docker-compose.yml | 0 docker_nocobase/{demo => compose}/nocobase-crm.sql | 0 docker_openclaw/{demo => compose}/docker-compose.yml | 0 docker_openclaw/p6/README.md | 2 +- .../docker-compose.searxng-standalone.yml | 0 .../docker-compose.searxng-with-proxy.yml | 0 .../{demo => compose}/searxng/settings.yml | 0 26 files changed, 33 insertions(+), 33 deletions(-) rename docker_casdoor/{demo => compose}/docker-compose.yml (100%) rename docker_clash/{demo => compose}/docker-compose.yml (100%) rename docker_hermes/{demo => compose}/.env.example (86%) rename docker_hermes/{demo => compose}/docker-compose.yml (100%) rename docker_keycloak/{demo => compose}/docker-compose.yml (100%) rename docker_litellm/{demo => compose}/.env.example (93%) rename docker_litellm/{demo => compose}/config.migrate.yaml (100%) rename docker_litellm/{demo => compose}/config.yaml (100%) rename docker_litellm/{demo => compose}/docker-compose.litellm.yml (100%) rename docker_litellm/{demo => compose}/scripts/aggregate-verification-summary.sh (97%) rename docker_litellm/{demo => compose}/scripts/run-migration.sh (82%) rename docker_litellm/{demo => compose}/scripts/verification-lib.sh (100%) rename docker_litellm/{demo => compose}/scripts/verify-migration-concurrency.sh (90%) rename docker_litellm/{demo => compose}/scripts/verify-p1.sh (86%) rename docker_nocobase/{demo => compose}/README.md (98%) rename docker_nocobase/{demo => compose}/docker-compose.yml (100%) rename docker_nocobase/{demo => compose}/nocobase-crm.sql (100%) rename docker_openclaw/{demo => compose}/docker-compose.yml (100%) rename docker_searxng/{demo => compose}/docker-compose.searxng-standalone.yml (100%) rename docker_searxng/{demo => compose}/docker-compose.searxng-with-proxy.yml (100%) rename docker_searxng/{demo => compose}/searxng/settings.yml (100%) diff --git a/.gitignore b/.gitignore index 1133fe7..428bc35 100644 --- a/.gitignore +++ b/.gitignore @@ -122,7 +122,7 @@ celerybeat.pid # Environments .env -docker_litellm/demo/artifacts/ +docker_litellm/compose/artifacts/ docker_openclaw/p6/artifacts/ docker_openclaw/p6/p6-inputs.json docker_openclaw/p6/.p6-work/ diff --git a/docker_casdoor/demo/docker-compose.yml b/docker_casdoor/compose/docker-compose.yml similarity index 100% rename from docker_casdoor/demo/docker-compose.yml rename to docker_casdoor/compose/docker-compose.yml diff --git a/docker_clash/demo/docker-compose.yml b/docker_clash/compose/docker-compose.yml similarity index 100% rename from docker_clash/demo/docker-compose.yml rename to docker_clash/compose/docker-compose.yml diff --git a/docker_hermes/README.md b/docker_hermes/README.md index a6304d1..581bfb8 100644 --- a/docker_hermes/README.md +++ b/docker_hermes/README.md @@ -86,17 +86,17 @@ Hermes 的 Dashboard 在 `/api/pty` 中执行已经构建的 1. Copy the sample environment file: ```bash - cp docker_hermes/.env.example docker_hermes/demo/.env + cp docker_hermes/.env.example docker_hermes/compose/.env ``` -2. Specify the built image in `docker_hermes/demo/.env`: +2. Specify the built image in `docker_hermes/compose/.env`: ```env HERMES_IMAGE=quay.io/labnow/hermes:local ``` 3. Launch the container: ```bash - docker compose --env-file docker_hermes/demo/.env -f docker_hermes/demo/docker-compose.yml up -d + docker compose --env-file docker_hermes/compose/.env -f docker_hermes/compose/docker-compose.yml up -d ``` ### Execution Modes @@ -123,7 +123,7 @@ python -c "from plugins.dashboard_auth.basic import hash_password; print(hash_pa ### Model Provider Setup -Hermes requires an LLM inference provider. Configure credentials in `docker_hermes/demo/.env`: +Hermes requires an LLM inference provider. Configure credentials in `docker_hermes/compose/.env`: ```env OPENAI_API_KEY=your-key diff --git a/docker_hermes/demo/.env.example b/docker_hermes/compose/.env.example similarity index 86% rename from docker_hermes/demo/.env.example rename to docker_hermes/compose/.env.example index 2b0d56b..5beb2d8 100644 --- a/docker_hermes/demo/.env.example +++ b/docker_hermes/compose/.env.example @@ -1,7 +1,7 @@ # Hermes local runtime configuration. # -# Copy this file to docker_hermes/demo/.env or pass it with: -# docker compose --env-file docker_hermes/demo/.env.example -f docker_hermes/demo/docker-compose.yml up -d +# Copy this file to docker_hermes/compose/.env or pass it with: +# docker compose --env-file docker_hermes/compose/.env.example -f docker_hermes/compose/docker-compose.yml up -d TZ=Asia/Shanghai diff --git a/docker_hermes/demo/docker-compose.yml b/docker_hermes/compose/docker-compose.yml similarity index 100% rename from docker_hermes/demo/docker-compose.yml rename to docker_hermes/compose/docker-compose.yml diff --git a/docker_hermes/p7/README.md b/docker_hermes/p7/README.md index 0816335..dbb1293 100644 --- a/docker_hermes/p7/README.md +++ b/docker_hermes/p7/README.md @@ -12,4 +12,4 @@ git show fdbbab2155a9e088c37d2a8a2057178e19ac9534:docker_hermes/p7/<路径> ``` 仍保留的 `scripts/test-p7-gates.sh` 是不启动容器的 Hermes 静态产品契约检查; -受支持构建与日常启动方式见父目录 README 和 `demo/`。 +受支持构建与日常启动方式见父目录 README 和 `compose/`。 diff --git a/docker_keycloak/demo/docker-compose.yml b/docker_keycloak/compose/docker-compose.yml similarity index 100% rename from docker_keycloak/demo/docker-compose.yml rename to docker_keycloak/compose/docker-compose.yml diff --git a/docker_litellm/README.md b/docker_litellm/README.md index dc28a94..38f0cd9 100644 --- a/docker_litellm/README.md +++ b/docker_litellm/README.md @@ -40,7 +40,7 @@ docker image inspect quay.io/labnow/litellm:1.97.0-ead62528e607 \ 准备不会被 Git 跟踪的配置。不要把 `.env` 发送到聊天、日志或提交中。 ```bash -cd docker_litellm/demo +cd docker_litellm/compose cp .env.example .env # 在 .env 中生成并填写 LITELLM_MASTER_KEY、POSTGRES_PASSWORD、REDIS_PASSWORD。 # 真实上游调用另行填写 UPSTREAM_PROVIDER、UPSTREAM_API_KEY、UPSTREAM_BASE_URL、UPSTREAM_MODEL。 @@ -88,7 +88,7 @@ P1 不把 LiteLLM User/Team 当作产品用户或 Workspace 的事实源;Shell ## Smoke 验证 ```bash -cd docker_litellm/demo +cd docker_litellm/compose ./scripts/verify-p1.sh ./scripts/smoke-baseline.sh --security-check ./scripts/test-verification-gates.sh @@ -99,7 +99,7 @@ cd docker_litellm/demo `--security-check` 不读取 `.env`、不启动服务也不发送上游请求;它拒绝 inline header、secret-bearing `jq --arg`、`set -x`、Compose 上游凭据注入和 Redis 密码命令行展开,并检查 Docker Secret、0600 临时文件与退出清理约束。`test-verification-gates.sh` 验证历史 PASS 失效、前置失败报告与 dotenv 命令替换不执行;在已启动 single 栈中追加 `--with-running-stack` 会以真实 404 删除请求证明 cleanup 不会生成 PASS。 -`test-secret-boundary.sh` 是 PH-1 的无上游定向门禁:它只生成本地占位凭据,不读取 `demo/.env`,验证 Compose 渲染和容器 inspect 不含 `LITELLM_MASTER_KEY`、`DATABASE_URL`、`POSTGRES_PASSWORD` 的服务环境或凭据值,并对 inspect、`ps/argv`、容器日志、容器临时文件执行负向检查。随后它启动 LiteLLM、调用并清理一次已认证的管理端点,退出时删除本次创建的容器、卷、网络和宿主临时文件。脚本为每次 run 设置唯一 `PROFILE_ENV`,若对应实例网络已存在则失败退出而不会复用或干扰该网络。 +`test-secret-boundary.sh` 是 PH-1 的无上游定向门禁:它只生成本地占位凭据,不读取 `compose/.env`,验证 Compose 渲染和容器 inspect 不含 `LITELLM_MASTER_KEY`、`DATABASE_URL`、`POSTGRES_PASSWORD` 的服务环境或凭据值,并对 inspect、`ps/argv`、容器日志、容器临时文件执行负向检查。随后它启动 LiteLLM、调用并清理一次已认证的管理端点,退出时删除本次创建的容器、卷、网络和宿主临时文件。脚本为每次 run 设置唯一 `PROFILE_ENV`,若对应实例网络已存在则失败退出而不会复用或干扰该网络。 `smoke-redis-recovery.sh` 在已启动的 HA 栈中临时断开 Redis 网络端点,验证两个副本的有界认证探针均失败,再恢复 `redis` alias、等待 breaker 窗口并确认认证后的 `GET /v1/models` 恢复。它有独立的恢复 trap,不会让故障测试影响主 smoke 的资源清理。`aggregate-verification-summary.sh` 将 migration、single、HA 与 Redis 独立报告组合为不含密钥、密码、提示词和响应正文的最终 JSON 摘要。 @@ -117,7 +117,7 @@ LiteLLM `v1.97.0-dev.1` 的公开 `/health/readiness` 仅返回服务与数据 ## 常见问题 -- `LITELLM_MASTER_KEY` 或数据库密码缺失:先检查被忽略的 `demo/.env`,不要将其内容贴出。 +- `LITELLM_MASTER_KEY` 或数据库密码缺失:先检查被忽略的 `compose/.env`,不要将其内容贴出。 - readiness 未连接数据库:查看 `docker compose ... logs postgres litellm-1`,并保留卷以便排查迁移。 - Redis 探针失败:不要继续双副本撤销验证;先确认 `redis` health 与密码一致。 - 上游调用待验证:仅在 `.env` 中提供专用、低权限、可轮换的测试 key,再重跑两个 smoke 命令。 diff --git a/docker_litellm/demo/.env.example b/docker_litellm/compose/.env.example similarity index 93% rename from docker_litellm/demo/.env.example rename to docker_litellm/compose/.env.example index d85a31e..53cc52d 100644 --- a/docker_litellm/demo/.env.example +++ b/docker_litellm/compose/.env.example @@ -1,4 +1,4 @@ -# Copy this file to docker_litellm/demo/.env. It is intentionally ignored. +# Copy this file to docker_litellm/compose/.env. It is intentionally ignored. # Do not commit real API keys, management keys, passwords, or virtual keys. TZ=Asia/Hong_Kong diff --git a/docker_litellm/demo/config.migrate.yaml b/docker_litellm/compose/config.migrate.yaml similarity index 100% rename from docker_litellm/demo/config.migrate.yaml rename to docker_litellm/compose/config.migrate.yaml diff --git a/docker_litellm/demo/config.yaml b/docker_litellm/compose/config.yaml similarity index 100% rename from docker_litellm/demo/config.yaml rename to docker_litellm/compose/config.yaml diff --git a/docker_litellm/demo/docker-compose.litellm.yml b/docker_litellm/compose/docker-compose.litellm.yml similarity index 100% rename from docker_litellm/demo/docker-compose.litellm.yml rename to docker_litellm/compose/docker-compose.litellm.yml diff --git a/docker_litellm/demo/scripts/aggregate-verification-summary.sh b/docker_litellm/compose/scripts/aggregate-verification-summary.sh similarity index 97% rename from docker_litellm/demo/scripts/aggregate-verification-summary.sh rename to docker_litellm/compose/scripts/aggregate-verification-summary.sh index c789982..8f6fb6c 100755 --- a/docker_litellm/demo/scripts/aggregate-verification-summary.sh +++ b/docker_litellm/compose/scripts/aggregate-verification-summary.sh @@ -4,15 +4,15 @@ set -euo pipefail script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -demo_dir="$(cd "${script_dir}/.." && pwd)" -artifacts_dir="${LITELLM_ARTIFACTS_DIR:-${demo_dir}/artifacts}" +compose_dir="$(cd "${script_dir}/.." && pwd)" +artifacts_dir="${LITELLM_ARTIFACTS_DIR:-${compose_dir}/artifacts}" single_report="${artifacts_dir}/p1-single-summary.json" ha_report="${artifacts_dir}/p1-ha-summary.json" redis_report="${artifacts_dir}/p1-redis-recovery.json" migration_report="${artifacts_dir}/p1-migration-summary.json" concurrency_report="${artifacts_dir}/p1-migration-concurrency.json" output="${LITELLM_AGGREGATE_SUMMARY_FILE:-${artifacts_dir}/p1-final-summary.json}" -commit="$(git -C "$demo_dir/../.." rev-parse HEAD)" +commit="$(git -C "$compose_dir/../.." rev-parse HEAD)" run_id="${VERIFICATION_RUN_ID:-}" started_at="${VERIFICATION_STARTED_AT:-}" rm -f "$output" diff --git a/docker_litellm/demo/scripts/run-migration.sh b/docker_litellm/compose/scripts/run-migration.sh similarity index 82% rename from docker_litellm/demo/scripts/run-migration.sh rename to docker_litellm/compose/scripts/run-migration.sh index ee06e15..4f5365f 100755 --- a/docker_litellm/demo/scripts/run-migration.sh +++ b/docker_litellm/compose/scripts/run-migration.sh @@ -4,11 +4,11 @@ set -euo pipefail script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -demo_dir="$(cd "${script_dir}/.." && pwd)" +compose_dir="$(cd "${script_dir}/.." && pwd)" source "${script_dir}/verification-lib.sh" -env_file="${LITELLM_SMOKE_ENV_FILE:-${demo_dir}/.env}" -compose=(docker compose --env-file "$env_file" -f "${demo_dir}/docker-compose.litellm.yml") -summary_file="${LITELLM_MIGRATION_SUMMARY_FILE:-${demo_dir}/artifacts/p1-migration-summary.json}" +env_file="${LITELLM_SMOKE_ENV_FILE:-${compose_dir}/.env}" +compose=(docker compose --env-file "$env_file" -f "${compose_dir}/docker-compose.litellm.yml") +summary_file="${LITELLM_MIGRATION_SUMMARY_FILE:-${compose_dir}/artifacts/p1-migration-summary.json}" result="failed" phase="initializing" @@ -24,7 +24,7 @@ cleanup() { chmod 700 "$(dirname "$summary_file")" summary_tmp="${summary_file}.tmp.$$" jq -n \ - --arg commit "$(git -C "$demo_dir/../.." rev-parse HEAD)" \ + --arg commit "$(git -C "$compose_dir/../.." rev-parse HEAD)" \ --arg image_id "$(docker image inspect "$image_ref" --format '{{.Id}}' 2>/dev/null || true)" \ --arg tested_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ --arg run_id "$verification_run_id" --arg result "$result" --arg phase "$phase" --argjson exit_code "$exit_code" \ @@ -42,7 +42,7 @@ trap cleanup EXIT umask 077 tmpdir="$(mktemp -d "${TMPDIR:-/tmp}/litellm-migration.XXXXXX")" chmod 700 "$tmpdir" -verification_prepare_environment "$env_file" "${demo_dir}/docker-compose.litellm.yml" "$tmpdir" +verification_prepare_environment "$env_file" "${compose_dir}/docker-compose.litellm.yml" "$tmpdir" image_ref="$(verification_env LITELLM_IMAGE)" : "${image_ref:?missing LITELLM_IMAGE in effective Compose environment}" diff --git a/docker_litellm/demo/scripts/verification-lib.sh b/docker_litellm/compose/scripts/verification-lib.sh similarity index 100% rename from docker_litellm/demo/scripts/verification-lib.sh rename to docker_litellm/compose/scripts/verification-lib.sh diff --git a/docker_litellm/demo/scripts/verify-migration-concurrency.sh b/docker_litellm/compose/scripts/verify-migration-concurrency.sh similarity index 90% rename from docker_litellm/demo/scripts/verify-migration-concurrency.sh rename to docker_litellm/compose/scripts/verify-migration-concurrency.sh index 90e8974..fbdf272 100755 --- a/docker_litellm/demo/scripts/verify-migration-concurrency.sh +++ b/docker_litellm/compose/scripts/verify-migration-concurrency.sh @@ -3,10 +3,10 @@ set -euo pipefail script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -demo_dir="$(cd "${script_dir}/.." && pwd)" +compose_dir="$(cd "${script_dir}/.." && pwd)" source "${script_dir}/verification-lib.sh" -env_file="${LITELLM_SMOKE_ENV_FILE:-${demo_dir}/.env}" -summary_file="${demo_dir}/artifacts/p1-migration-concurrency.json" +env_file="${LITELLM_SMOKE_ENV_FILE:-${compose_dir}/.env}" +summary_file="${compose_dir}/artifacts/p1-migration-concurrency.json" run_id="${VERIFICATION_RUN_ID:?VERIFICATION_RUN_ID is required}" tmpdir="" image_ref="" @@ -27,7 +27,7 @@ cleanup() { [[ -z "$first" ]] || docker rm "$first" >/dev/null 2>&1 || true [[ -z "$second" ]] || docker rm "$second" >/dev/null 2>&1 || true tmp="${summary_file}.tmp.$$" - jq -n --arg run_id "$run_id" --arg commit "$(git -C "$demo_dir/../.." rev-parse HEAD)" \ + jq -n --arg run_id "$run_id" --arg commit "$(git -C "$compose_dir/../.." rev-parse HEAD)" \ --arg image_id "$(docker image inspect "$image_ref" --format '{{.Id}}' 2>/dev/null || true)" \ --arg tested_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" --arg result "$result" --arg phase "$phase" \ --argjson actual_overlap "$actual_overlap" --argjson lock_wait_observed "$lock_wait_observed" \ @@ -41,9 +41,9 @@ trap cleanup EXIT tmpdir="$(mktemp -d "${TMPDIR:-/tmp}/litellm-migration-concurrency.XXXXXX")" chmod 700 "$tmpdir" -verification_prepare_environment "$env_file" "${demo_dir}/docker-compose.litellm.yml" "$tmpdir" +verification_prepare_environment "$env_file" "${compose_dir}/docker-compose.litellm.yml" "$tmpdir" image_ref="$(verification_env LITELLM_IMAGE)" -compose=(docker compose --env-file "$env_file" -f "${demo_dir}/docker-compose.litellm.yml") +compose=(docker compose --env-file "$env_file" -f "${compose_dir}/docker-compose.litellm.yml") "${compose[@]}" up -d --wait postgres redis phase="starting_concurrent_jobs" diff --git a/docker_litellm/demo/scripts/verify-p1.sh b/docker_litellm/compose/scripts/verify-p1.sh similarity index 86% rename from docker_litellm/demo/scripts/verify-p1.sh rename to docker_litellm/compose/scripts/verify-p1.sh index 8afb5a7..72e47f9 100755 --- a/docker_litellm/demo/scripts/verify-p1.sh +++ b/docker_litellm/compose/scripts/verify-p1.sh @@ -3,7 +3,7 @@ set -euo pipefail script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -demo_dir="$(cd "${script_dir}/.." && pwd)" +compose_dir="$(cd "${script_dir}/.." && pwd)" source "${script_dir}/verification-lib.sh" run_id="$(verification_new_run_id)" export VERIFICATION_RUN_ID="$run_id" @@ -11,13 +11,13 @@ export VERIFICATION_STARTED_AT="$(date -u +%Y-%m-%dT%H:%M:%SZ)" # P1's documented local provider mapping is explicit. It is non-secret and # prevents a template/default mismatch from silently selecting another SDK. export LITELLM_SMOKE_UPSTREAM_PROVIDER="${LITELLM_SMOKE_UPSTREAM_PROVIDER:-deepseek}" -compose=(docker compose --env-file "${demo_dir}/.env" -f "${demo_dir}/docker-compose.litellm.yml") +compose=(docker compose --env-file "${compose_dir}/.env" -f "${compose_dir}/docker-compose.litellm.yml") cleanup_stack() { "${compose[@]}" --profile single --profile ha --profile migrate down >/dev/null 2>&1 || true; } trap cleanup_stack EXIT cleanup_stack for report in p1-migration-summary.json p1-migration-concurrency.json p1-single-summary.json p1-ha-summary.json p1-redis-recovery.json p1-final-summary.json; do - verification_invalidate_report "${demo_dir}/artifacts/${report}" + verification_invalidate_report "${compose_dir}/artifacts/${report}" done "${script_dir}/test-verification-gates.sh" "${script_dir}/run-migration.sh" diff --git a/docker_nocobase/demo/README.md b/docker_nocobase/compose/README.md similarity index 98% rename from docker_nocobase/demo/README.md rename to docker_nocobase/compose/README.md index 9ac38de..ed644c3 100644 --- a/docker_nocobase/demo/README.md +++ b/docker_nocobase/compose/README.md @@ -1,6 +1,6 @@ # NocoBase 物理表建表与元数据 (Metadata) 配置指南 -本目录包含用于重构及初始化 CRM 业务系统的 PostgreSQL DDL 脚本 [nocobase-crm.sql](docker_nocobase/demo/nocobase-crm.sql)。 +本目录包含用于重构及初始化 CRM 业务系统的 PostgreSQL DDL 脚本 [nocobase-crm.sql](docker_nocobase/compose/nocobase-crm.sql)。 为了避免后续在直接修改数据库元数据或创建物理表时导致 NocoBase 报错(例如:*“当数据表没有主键时...”* 或页面区块选择器无法选取数据表),请严格遵循以下设计原则与最佳实践。 diff --git a/docker_nocobase/demo/docker-compose.yml b/docker_nocobase/compose/docker-compose.yml similarity index 100% rename from docker_nocobase/demo/docker-compose.yml rename to docker_nocobase/compose/docker-compose.yml diff --git a/docker_nocobase/demo/nocobase-crm.sql b/docker_nocobase/compose/nocobase-crm.sql similarity index 100% rename from docker_nocobase/demo/nocobase-crm.sql rename to docker_nocobase/compose/nocobase-crm.sql diff --git a/docker_openclaw/demo/docker-compose.yml b/docker_openclaw/compose/docker-compose.yml similarity index 100% rename from docker_openclaw/demo/docker-compose.yml rename to docker_openclaw/compose/docker-compose.yml diff --git a/docker_openclaw/p6/README.md b/docker_openclaw/p6/README.md index 4c92057..01b6b5b 100644 --- a/docker_openclaw/p6/README.md +++ b/docker_openclaw/p6/README.md @@ -12,4 +12,4 @@ git show fdbbab2155a9e088c37d2a8a2057178e19ac9534:docker_openclaw/p6/<路径> ``` 仍保留的 `scripts/test-p6-gates.sh` 是不启动容器的静态产品契约检查;OpenClaw -的受支持构建与日常启动方式见父目录 README 和 `demo/`。 +的受支持构建与日常启动方式见父目录 README 和 `compose/`。 diff --git a/docker_searxng/demo/docker-compose.searxng-standalone.yml b/docker_searxng/compose/docker-compose.searxng-standalone.yml similarity index 100% rename from docker_searxng/demo/docker-compose.searxng-standalone.yml rename to docker_searxng/compose/docker-compose.searxng-standalone.yml diff --git a/docker_searxng/demo/docker-compose.searxng-with-proxy.yml b/docker_searxng/compose/docker-compose.searxng-with-proxy.yml similarity index 100% rename from docker_searxng/demo/docker-compose.searxng-with-proxy.yml rename to docker_searxng/compose/docker-compose.searxng-with-proxy.yml diff --git a/docker_searxng/demo/searxng/settings.yml b/docker_searxng/compose/searxng/settings.yml similarity index 100% rename from docker_searxng/demo/searxng/settings.yml rename to docker_searxng/compose/searxng/settings.yml From 999e18825864c2c6a63de12ad525f85517dfdb5f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Fri, 21 Aug 2026 15:13:50 +0800 Subject: [PATCH 84/87] =?UTF-8?q?chore:=20=E6=B8=85=E7=90=86=E5=BC=80?= =?UTF-8?q?=E5=8F=91=E8=BF=87=E7=A8=8B=E8=AE=B0=E5=BD=95=E4=B8=8E=E9=98=B6?= =?UTF-8?q?=E6=AE=B5=E8=BF=87=E7=A8=8B=E6=A0=87=E8=AE=B0?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - 删除 p6/p7 归档墓碑 README;父 README 去除归档指引与阶段命名 - docker_litellm README 重写为产品文档:去除阶段过程叙述与已移出脚本引用 - .env.example/Dockerfile/启动脚本注释中性化;删除无消费方的 OPENCLAW 参考变量 - 验收流水线脚本(verify-p1/migration-concurrency/aggregate)移出分支,本地保留; run-migration 与其依赖的 verification-lib 作为运维工具保留 Co-Authored-By: Claude Fable 5 --- docker_hermes/README.md | 27 ++---- docker_hermes/p7/README.md | 15 --- docker_litellm/README.md | 68 ++++---------- docker_litellm/compose/.env.example | 8 +- .../scripts/aggregate-verification-summary.sh | 91 ------------------- .../compose/scripts/verification-lib.sh | 2 +- .../scripts/verify-migration-concurrency.sh | 89 ------------------ docker_litellm/compose/scripts/verify-p1.sh | 32 ------- docker_litellm/litellm.Dockerfile | 2 +- docker_litellm/work/start-litellm.sh | 2 +- docker_openclaw/README.md | 6 -- docker_openclaw/p6/README.md | 15 --- 12 files changed, 31 insertions(+), 326 deletions(-) delete mode 100644 docker_hermes/p7/README.md delete mode 100755 docker_litellm/compose/scripts/aggregate-verification-summary.sh delete mode 100755 docker_litellm/compose/scripts/verify-migration-concurrency.sh delete mode 100755 docker_litellm/compose/scripts/verify-p1.sh delete mode 100644 docker_openclaw/p6/README.md diff --git a/docker_hermes/README.md b/docker_hermes/README.md index 581bfb8..3447032 100644 --- a/docker_hermes/README.md +++ b/docker_hermes/README.md @@ -2,9 +2,9 @@ `hermes` is a containerized agentic assistant platform based on the [Hermes Agent](https://github.com/nousresearch/hermes-agent) project, built using Node.js and Python runtime stacks. -P7 的验收构建固定 Hermes repository 与 40 位 commit;Dockerfile 不再以移动 -`main` 作为制品输入。默认 standalone Compose 仍服务于本地开发,但只接受已在 -本机存在的明确镜像引用,不会静默 pull `latest`。 +Dockerfile 以固定的 Hermes repository 与 40 位 commit 作为制品输入,不以移动的 +`main` 构建。默认 standalone Compose 服务于本地开发,只接受本机已存在的明确镜像 +引用,不会静默 pull `latest`。 --- @@ -52,41 +52,32 @@ source ./tool.sh build_image_no_tag hermes local docker_hermes/hermes.Dockerfile ``` -### P7 可复现构建 +### 可复现构建(固定源码提交) -需要跨仓 Hermes 联调时,使用 P7 固定 tag 和观察到的 Hermes source identity: +需要可复现构建时,使用固定 tag 与明确的 Hermes source identity: ```bash -build_image_no_tag hermes p7-<12hex> docker_hermes/hermes.Dockerfile \ +build_image_no_tag hermes src-<12hex> docker_hermes/hermes.Dockerfile \ --build-arg HERMES_SOURCE_REPOSITORY= \ --build-arg HERMES_SOURCE_COMMIT=<40-hex-commit> ``` 镜像会记录 `org.opencontainers.image.source` 与 `org.opencontainers.image.revision`,并在 `/opt/hermes/.labnow-source-*` 保存 -相同的非敏感 provenance。只在本地命名为 `quay.io/labnow/hermes:p7-<12hex>`,不 push。 -P7 的跨仓固定组合与 runner 已按 D-10 归档;历史回读方法见 -[`p7/README.md`](p7/README.md)。它不是 Hermes 的构建、启动或 CI 入口。 +相同的非敏感 provenance。只在本地命名为 `quay.io/labnow/hermes:src-<12hex>`,不 push。 -### P8-H10:Dashboard Chat TUI runtime +### Dashboard Chat TUI runtime Hermes 的 Dashboard 在 `/api/pty` 中执行已经构建的 `/opt/hermes/ui-tui/dist/entry.js`。运行基础镜像不是 Node 镜像,因此 Dockerfile 会从 同一目标架构的 builder 复制固定的 `/opt/node` runtime,并将其放入 `PATH`。这避免用户 第一次打开 Chat 时触发 Node 下载/解压;不改变 Hermes source、TUI build 或模型配置。 -对 P8-H10 本地镜像执行不含凭证、不会请求模型的 runtime 门禁: - -```bash -./docker_hermes/scripts/test-hermes-runtime-node.sh \ - quay.io/labnow/hermes:che-588-hermes-chat-tui-runtime-local -``` - ### Start with Docker Compose 1. Copy the sample environment file: ```bash - cp docker_hermes/.env.example docker_hermes/compose/.env + cp docker_hermes/compose/.env.example docker_hermes/compose/.env ``` 2. Specify the built image in `docker_hermes/compose/.env`: diff --git a/docker_hermes/p7/README.md b/docker_hermes/p7/README.md deleted file mode 100644 index dbb1293..0000000 --- a/docker_hermes/p7/README.md +++ /dev/null @@ -1,15 +0,0 @@ -# P7 Hermes 产品链(已归档) - -LLM Hub V1 / P7 的固定组合、Launcher overlay、受限输入、黄金 runner 与报告 -聚合均为一次性冻结验收证据,已按生产加固决策 D-10 从产品运行面删除。它们不是 -持续维护的 Hermes 构建、启动或 CI 入口。 - -最后可读快照是本批基线 -`fdbbab2155a9e088c37d2a8a2057178e19ac9534`。需要审阅历史材料时,在本仓执行: - -```bash -git show fdbbab2155a9e088c37d2a8a2057178e19ac9534:docker_hermes/p7/<路径> -``` - -仍保留的 `scripts/test-p7-gates.sh` 是不启动容器的 Hermes 静态产品契约检查; -受支持构建与日常启动方式见父目录 README 和 `compose/`。 diff --git a/docker_litellm/README.md b/docker_litellm/README.md index 38f0cd9..3e7152f 100644 --- a/docker_litellm/README.md +++ b/docker_litellm/README.md @@ -1,8 +1,8 @@ -# LiteLLM Proxy:P1 真实运行基线 +# LiteLLM Proxy -本目录维护 LiteLLM 的部署适配,不 fork 或修改 LiteLLM 上游业务逻辑。P1 提供可复现的本地基线:共享 PostgreSQL、共享 Redis、单副本与双副本 LiteLLM,以及不会写入仓库密钥的 smoke test。 +本目录维护 LiteLLM 的部署适配,不 fork 或修改 LiteLLM 上游业务逻辑。提供可复现的本地基线:共享 PostgreSQL、共享 Redis、单副本与双副本 LiteLLM;凭据不写入仓库。 -P1 默认不构建 LiteLLM Dashboard 静态资源:固定源码在当前构建基础镜像上导出 `/_not-found` 时失败,而代理 API 与管理面验证不依赖该资源。需要 Dashboard 时可显式传入 `--build-arg BUILD_DASHBOARD=true` 单独处理该上游前端兼容性;它不属于 P1 通过条件。 +默认不构建 LiteLLM Dashboard 静态资源:固定源码在当前构建基础镜像上导出 `/_not-found` 时失败,而代理 API 与管理面不依赖该资源。需要 Dashboard 时可显式传入 `--build-arg BUILD_DASHBOARD=true` 单独处理该上游前端兼容性。 ## 固定版本与镜像 @@ -11,14 +11,13 @@ P1 默认不构建 LiteLLM Dashboard 静态资源:固定源码在当前构建 | LiteLLM 源码 | `v1.97.0-dev.1` / `ead62528e607b9d8e61273def638799c9c3a69ba` | Dockerfile 精确 fetch 并校验 HEAD | | FastAPI | `0.136.3` | 固定到该 LiteLLM commit 仍使用 `get_flat_dependant` 的兼容版本 | | Prisma Python client | `0.15.0` | LiteLLM 连接 PostgreSQL 所需客户端,兼容基础镜像的 Python 3.13 | -| 本地产物镜像 | `quay.io/labnow/litellm:1.97.0-ead62528e607` | P1 Compose 的唯一 LiteLLM 默认镜像 | +| 本地产物镜像 | `quay.io/labnow/litellm:1.97.0-ead62528e607` | Compose 的默认 LiteLLM 镜像 | | PostgreSQL | `postgres:17-alpine@sha256:742f40ea20b9ff2ff31db5458d127452988a2164df9e17441e191f3b72252193` | 用户、凭证、模型、虚拟 key 与 spend 持久化 | | Redis | `redis:7.4-alpine@sha256:e7723ff73d963f5cc6d9c4643ea3d989527a402a319239054e9472a7fb9219a2` | 副本共享认证缓存、RPM/TPM limiter 与协调缓存;SpendLog 的事实源是 PostgreSQL | -| OpenClaw(P2 参考) | `2026.5.10-beta.1` / `eed75ed47f47deb18c9d093a2e638c9bb0bedf14` | 仅为下一阶段黄金适配器保留版本基线;P1 不启动或实现 Adapter | 镜像构建会对 wheel 自带的 LiteLLM Prisma schema 运行 `prisma generate`,并把生成的查询引擎固定在 `/opt/litellm/.cache`;没有该步骤,或将该缓存随 `/root/.cache` 清理,代理会在 PostgreSQL startup 时报缺少 Prisma binaries 或无法连接查询引擎。 -构建前已确认完整 LabNow 镜像名是 `quay.io/labnow/litellm:1.97.0-ead62528e607`,不会推送镜像。必须通过根目录 `tool.sh` 构建,避免基础镜像退回 Docker Hub: +必须通过根目录 `tool.sh` 构建,避免基础镜像退回 Docker Hub: ```bash export REGISTRY_SRC=quay.io @@ -57,67 +56,34 @@ PROFILE_ENV=litellm-dev-a docker compose --env-file .env -f docker-compose.litel PROFILE_ENV=litellm-dev-b docker compose --env-file .env -f docker-compose.litellm.yml --profile single up -d ``` -迁移与代理启动刻意分离。标准真实验收由一个统一入口执行:它生成新的非敏感 `verification_run_id`,先失效所有旧输入/最终报告,再运行 migration(两次)、并发 migration job、single、HA、Redis 恢复和严格聚合;任一步失败都会停止且保留当前失败报告。 +迁移与代理启动刻意分离:先用 `./scripts/run-migration.sh` 执行数据库迁移(migration profile),再启动代理 profile;不要把 migration profile 与代理 profile 放入同一条 `up` 命令。 -```bash -./scripts/verify-p1.sh -``` - -手动分步排障时,必须为 migration、single、HA、Redis 与聚合导出同一合法 run ID;不要把 migration profile 与代理 profile 放入同一条 `up` 命令。 - -默认端口只发布在 `127.0.0.1`:副本 1 为 `4000`,副本 2 为 `4001`。PostgreSQL 与 Redis 不发布宿主机端口。停止测试不会删除卷;如需删除测试数据,先人工确认后使用 `docker compose ... down -v`。 +默认端口只发布在 `127.0.0.1`:副本 1 为 `4000`,副本 2 为 `4001`。PostgreSQL 与 Redis 不发布宿主机端口。停止服务不会删除卷;如需删除本地数据,先人工确认后使用 `docker compose ... down -v`。 ## 配置与安全边界 -`config.yaml` 从运行时环境读取管理面 `LITELLM_MASTER_KEY`、`DATABASE_URL` 与 Redis 凭据。Compose 不再把管理密钥、数据库密码或含密码的连接串写入服务 `environment`:它将 `LITELLM_MASTER_KEY`、`POSTGRES_PASSWORD` 和 `REDIS_PASSWORD` 交给 Docker Secret;PostgreSQL 使用官方 `POSTGRES_PASSWORD_FILE`,LiteLLM 的 `start-litellm.sh` 在最终 `exec` 前读取 Secret 文件、构造 `DATABASE_URL` 并立即转交 LiteLLM。管理面 key 仅用于 `/user/new`、`/credentials`、`/model/new`、`/key/generate`、`/key/block` 和 `/key/delete` 等管理接口;smoke 生成的数据面虚拟 key 是短期、模型白名单、TTL、预算、RPM、TPM 与 `llm_api` 路由限制的独立 key。双副本基线启用 `enable_redis_auth_cache`,并将 `user_api_key_cache_ttl` 设为 1 秒,以使撤销在 30 秒 smoke SLO 内经共享 Redis 重新校验。 +`config.yaml` 从运行时环境读取管理面 `LITELLM_MASTER_KEY`、`DATABASE_URL` 与 Redis 凭据。Compose 不把管理密钥、数据库密码或含密码的连接串写入服务 `environment`:它将 `LITELLM_MASTER_KEY`、`POSTGRES_PASSWORD` 和 `REDIS_PASSWORD` 交给 Docker Secret;PostgreSQL 使用官方 `POSTGRES_PASSWORD_FILE`,LiteLLM 的 `start-litellm.sh` 在最终 `exec` 前读取 Secret 文件、构造 `DATABASE_URL` 并立即转交 LiteLLM。管理面 key 仅用于 `/user/new`、`/credentials`、`/model/new`、`/key/generate`、`/key/block` 和 `/key/delete` 等管理接口;数据面虚拟 key 应为短期、模型白名单、TTL、预算、RPM、TPM 与 `llm_api` 路由限制的独立 key。双副本基线启用 `enable_redis_auth_cache`,并将 `user_api_key_cache_ttl` 设为 1 秒,以使撤销在 30 秒内经共享 Redis 重新校验。 | 变量 | 是否必填 | 作用 | 风险说明 | | --- | ---: | --- | --- | | `LITELLM_MASTER_KEY` | 是 | 管理面认证 | 仅放在忽略的 `.env` 或部署 Secret | | `POSTGRES_PASSWORD` | 是 | PostgreSQL 密码 | 仅限本地测试或部署 Secret | | `REDIS_PASSWORD` | 是 | Redis 认证 | 仅限本地测试或部署 Secret | -| `UPSTREAM_API_KEY` | 真实调用时是 | 上游模型凭据 | 仅由 smoke 客户端读取;不会注入 LiteLLM 容器、不提交、不打印 | -| `UPSTREAM_PROVIDER` | 真实调用时是 | P1 provider 选择 | 当前明确支持 `deepseek`;错误组合会在调用前脱敏失败 | -| `UPSTREAM_BASE_URL` | 真实调用时是 | OpenAI 兼容上游地址 | 由测试环境决定 | +| `UPSTREAM_API_KEY` | 真实调用时是 | 上游模型凭据 | 仅由本地验证客户端读取;不会注入 LiteLLM 容器、不提交、不打印 | +| `UPSTREAM_PROVIDER` | 真实调用时是 | 上游 provider 选择 | 当前明确支持 `deepseek` | +| `UPSTREAM_BASE_URL` | 真实调用时是 | OpenAI 兼容上游地址 | 由环境决定 | | `UPSTREAM_MODEL` | 真实调用时是 | 上游模型名 | 用于创建测试模型 | -| `LITELLM_REVOCATION_SLO_MS` | `30000` | block/delete 的跨副本拒绝 SLO | smoke 会输出实际传播耗时;仅用于本地验收 | -| `REDIS_CIRCUIT_BREAKER_RECOVERY_TIMEOUT` | `5` | Redis 断连后的 LiteLLM 缓存恢复探测窗口(秒) | 本地 HA 基线应小于撤销 SLO;恢复期间管理面可能暂时返回 500 | - -P1 不把 LiteLLM User/Team 当作产品用户或 Workspace 的事实源;Shell 的用户、绑定和租约业务仍在后续 Phase 实现。 - -## Smoke 验证 - -```bash -cd docker_litellm/compose -./scripts/verify-p1.sh -./scripts/smoke-baseline.sh --security-check -./scripts/test-verification-gates.sh -./scripts/test-secret-boundary.sh -``` - -在全新 checkout 中按上述标准命令执行时,脚本自己生成而非复用历史文件:`p1-migration-summary.json`、`p1-migration-concurrency.json`、`p1-single-summary.json`、`p1-ha-summary.json`、`p1-redis-recovery.json` 与最终 `p1-final-summary.json`(均在被忽略的 `artifacts/`)。聚合脚本只接受当前 `HEAD`、同一 `verification_run_id`、相同 image ID、正确 mode、启动后 `tested_at`、`result=passed`、`phase=completed` 且脱敏的输入;任何缺失、失败、跳过、过期或模式不符都会被拒绝。 - -`--security-check` 不读取 `.env`、不启动服务也不发送上游请求;它拒绝 inline header、secret-bearing `jq --arg`、`set -x`、Compose 上游凭据注入和 Redis 密码命令行展开,并检查 Docker Secret、0600 临时文件与退出清理约束。`test-verification-gates.sh` 验证历史 PASS 失效、前置失败报告与 dotenv 命令替换不执行;在已启动 single 栈中追加 `--with-running-stack` 会以真实 404 删除请求证明 cleanup 不会生成 PASS。 - -`test-secret-boundary.sh` 是 PH-1 的无上游定向门禁:它只生成本地占位凭据,不读取 `compose/.env`,验证 Compose 渲染和容器 inspect 不含 `LITELLM_MASTER_KEY`、`DATABASE_URL`、`POSTGRES_PASSWORD` 的服务环境或凭据值,并对 inspect、`ps/argv`、容器日志、容器临时文件执行负向检查。随后它启动 LiteLLM、调用并清理一次已认证的管理端点,退出时删除本次创建的容器、卷、网络和宿主临时文件。脚本为每次 run 设置唯一 `PROFILE_ENV`,若对应实例网络已存在则失败退出而不会复用或干扰该网络。 - -`smoke-redis-recovery.sh` 在已启动的 HA 栈中临时断开 Redis 网络端点,验证两个副本的有界认证探针均失败,再恢复 `redis` alias、等待 breaker 窗口并确认认证后的 `GET /v1/models` 恢复。它有独立的恢复 trap,不会让故障测试影响主 smoke 的资源清理。`aggregate-verification-summary.sh` 将 migration、single、HA 与 Redis 独立报告组合为不含密钥、密码、提示词和响应正文的最终 JSON 摘要。 - -脚本在真实上游变量存在时执行:创建测试用户、保存测试上游凭证、以 `litellm_credential_name` 创建模型、由调用方生成稳定高熵 virtual key 并故意丢弃首次创建响应,再用该 key 的 0600 Authorization header 调用 `/v2/key/info` 恢复、验证相同 key 重试被拒绝而不会创建第二资源;随后显式 `GET /v1/models`、chat、stream、tool call、token-bearing usage 查询、block,以及独立 key 的 delete。DeepSeek V4 使用 `deepseek/` 的原生 provider,避免被通用 OpenAI provider 丢弃 `thinking` 参数。HA 模式会先证明第二副本接受 key,再验证跨副本 RPM 与 TPM 限制均返回由 LiteLLM Proxy limiter 产生的 `429`,最后轮询两个副本直到都拒绝,并输出实际传播时间与 SLO。 - -`LITELLM_MASTER_KEY`、上游 API key 与虚拟 key 不会作为 `curl`、`jq` 或其他子进程的命令参数传递。脚本以 `umask 077` 创建工作目录,所有 header、请求、响应和 key 文件均为 `0600`,退出时删除;创建的 user、credential、model 和两个测试 key 也会清理。上游凭据仅由 smoke 客户端读取,不会注入 LiteLLM Compose 容器。 - -Compose 凭据边界的残余风险:LiteLLM 上游配置接口仍要求 `LITELLM_MASTER_KEY` 与 `DATABASE_URL` 在其最终进程环境中可见;本基线已接受这一点。凭据不再出现在 Compose 渲染、容器 `docker inspect` metadata、命令行参数、容器日志或运行时临时文件中。使用具有 Docker daemon 访问权限或容器内同等调试权限的主体仍应视为高权限主体,不应以该边界替代主机与容器访问控制。 +| `REDIS_CIRCUIT_BREAKER_RECOVERY_TIMEOUT` | `5` | Redis 断连后的 LiteLLM 缓存恢复探测窗口(秒) | 恢复期间管理面可能暂时返回 500 | -若未设置上游变量,脚本仍验证 LiteLLM readiness、PostgreSQL 连接、从每个 LiteLLM 副本到 Redis 的认证连通性、migration 证据和 user 清理路径,并以明确的 `result=skipped` / `phase=pending_upstream` 报告退出。它不会伪造 chat、stream、tool、usage、block/delete 或撤销传播已通过,最终聚合也会拒绝该报告。 +Compose 凭据边界的残余风险:LiteLLM 上游配置接口仍要求 `LITELLM_MASTER_KEY` 与 `DATABASE_URL` 在其最终进程环境中可见;本基线已接受这一点。凭据不出现在 Compose 渲染、容器 `docker inspect` metadata、命令行参数、容器日志或运行时临时文件中。使用具有 Docker daemon 访问权限或容器内同等调试权限的主体仍应视为高权限主体,不应以该边界替代主机与容器访问控制。 -## Readiness 与 Redis 结论 +## Readiness 与 Redis 说明 -LiteLLM `v1.97.0-dev.1` 的公开 `/health/readiness` 仅返回服务与数据库连通性,不将 Redis 纳入公开 readiness。因此 Compose 健康检查只能确认 LiteLLM + PostgreSQL;`smoke-baseline.sh` 额外从每个 LiteLLM 容器执行 Redis `PING`。若 Redis 不可用,P1 的多副本认证缓存、RPM/TPM limiter 与协调结论无效,应将该运行组合标记为 `blocked`,不能降级宣称为高可用。Redis 恢复后,LiteLLM 的认证缓存 circuit breaker 需要经过 `REDIS_CIRCUIT_BREAKER_RECOVERY_TIMEOUT` 后才会重新探测;P1 默认设为 5 秒,并要求恢复后再次跑 HA smoke。跨副本 SpendLog 只证明 PostgreSQL 可见性,不是 Redis Spend counter 或预算准入控制证据。 +LiteLLM `v1.97.0-dev.1` 的公开 `/health/readiness` 仅返回服务与数据库连通性,不将 Redis 纳入公开 readiness。因此 Compose 健康检查只能确认 LiteLLM + PostgreSQL;可额外从每个 LiteLLM 容器执行 Redis `PING` 确认。若 Redis 不可用,多副本认证缓存、RPM/TPM limiter 与协调结论无效,不能宣称为高可用。Redis 恢复后,LiteLLM 的认证缓存 circuit breaker 需要经过 `REDIS_CIRCUIT_BREAKER_RECOVERY_TIMEOUT` 后才会重新探测,默认 5 秒。跨副本 SpendLog 只证明 PostgreSQL 可见性,不是 Redis Spend counter 或预算准入控制证据。 ## 常见问题 - `LITELLM_MASTER_KEY` 或数据库密码缺失:先检查被忽略的 `compose/.env`,不要将其内容贴出。 - readiness 未连接数据库:查看 `docker compose ... logs postgres litellm-1`,并保留卷以便排查迁移。 -- Redis 探针失败:不要继续双副本撤销验证;先确认 `redis` health 与密码一致。 -- 上游调用待验证:仅在 `.env` 中提供专用、低权限、可轮换的测试 key,再重跑两个 smoke 命令。 +- Redis 探针失败:先确认 `redis` health 与密码一致,再做双副本撤销验证。 +- 上游调用:仅在 `.env` 中提供专用、低权限、可轮换的测试 key。 diff --git a/docker_litellm/compose/.env.example b/docker_litellm/compose/.env.example index 53cc52d..e47def0 100644 --- a/docker_litellm/compose/.env.example +++ b/docker_litellm/compose/.env.example @@ -18,14 +18,10 @@ POSTGRES_USER=litellm POSTGRES_PASSWORD= REDIS_PASSWORD= -# Optional upstream required for chat/stream/tool smoke. The smoke validates -# this provider/model pair before any request. Supported P1 provider: deepseek. +# Optional upstream used only by local verification clients; it is never injected +# into the LiteLLM containers. Supported provider: deepseek. # Keep UPSTREAM_API_KEY empty to validate infrastructure paths only. UPSTREAM_PROVIDER=deepseek UPSTREAM_API_KEY= UPSTREAM_BASE_URL=https://api.deepseek.com/v1 UPSTREAM_MODEL=deepseek-v4-flash - -# P2 reference only; this phase does not start an OpenClaw adapter. -OPENCLAW_IMAGE=quay.io/labnow/openclaw:2026.5.10-beta.1 -OPENCLAW_SOURCE_COMMIT=eed75ed47f47deb18c9d093a2e638c9bb0bedf14 diff --git a/docker_litellm/compose/scripts/aggregate-verification-summary.sh b/docker_litellm/compose/scripts/aggregate-verification-summary.sh deleted file mode 100755 index 8f6fb6c..0000000 --- a/docker_litellm/compose/scripts/aggregate-verification-summary.sh +++ /dev/null @@ -1,91 +0,0 @@ -#!/usr/bin/env bash -# Aggregate only reports made by the current checkout; never infer a result -# from a previous run or from missing inputs. -set -euo pipefail - -script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -compose_dir="$(cd "${script_dir}/.." && pwd)" -artifacts_dir="${LITELLM_ARTIFACTS_DIR:-${compose_dir}/artifacts}" -single_report="${artifacts_dir}/p1-single-summary.json" -ha_report="${artifacts_dir}/p1-ha-summary.json" -redis_report="${artifacts_dir}/p1-redis-recovery.json" -migration_report="${artifacts_dir}/p1-migration-summary.json" -concurrency_report="${artifacts_dir}/p1-migration-concurrency.json" -output="${LITELLM_AGGREGATE_SUMMARY_FILE:-${artifacts_dir}/p1-final-summary.json}" -commit="$(git -C "$compose_dir/../.." rev-parse HEAD)" -run_id="${VERIFICATION_RUN_ID:-}" -started_at="${VERIFICATION_STARTED_AT:-}" -rm -f "$output" -[[ "$run_id" =~ ^p1-[a-f0-9]{32}$ ]] || { echo "missing verification run id" >&2; exit 2; } -[[ "$started_at" =~ ^[0-9]{4}-[0-9]{2}-[0-9]{2}T ]] || { echo "missing verification start time" >&2; exit 2; } - -for report in "$single_report" "$ha_report" "$redis_report" "$migration_report" "$concurrency_report"; do - [[ -f "$report" ]] || { echo "missing required report: $report" >&2; exit 2; } -done - -# Each input must be an independently successful and fully redacted result of -# this exact checkout. jq -e performs the gate before the final report exists. -jq -e --arg commit "$commit" --arg run_id "$run_id" --arg started_at "$started_at" ' - .mode == "migration" and .result == "passed" and .phase == "completed" and - .verification_run_id == $run_id and .commit == $commit and (.image_id | type == "string" and length > 0) and - (.tested_at | type == "string" and . >= $started_at) and - .content_redacted == true and .proxy_replicas_started == false -' "$migration_report" >/dev/null - -jq -e --arg commit "$commit" --arg run_id "$run_id" --arg started_at "$started_at" ' - .mode == "migration" and .result == "passed" and .phase == "completed" and .concurrent_migration == true and - .actual_overlap == true and .lock_wait_observed == true and .exclusive_lock == true and - .max_lock_holders == 1 and .migration_execution_count == 2 and .proxy_replicas_started == false and - .verification_run_id == $run_id and .commit == $commit and (.image_id | type == "string" and length > 0) and - (.tested_at | type == "string" and . >= $started_at) and .content_redacted == true -' "$concurrency_report" >/dev/null - -jq -e --arg commit "$commit" --arg run_id "$run_id" --arg started_at "$started_at" ' - .mode == "single" and .result == "passed" and .phase == "completed" and - .verification_run_id == $run_id and .commit == $commit and (.image_id | type == "string" and length > 0) and - (.tested_at | type == "string" and . >= $started_at) and - .content_redacted == true and .migration == "passed" and - .chat == "passed" and .stream == "passed" and .tool == "passed" and - .usage == "passed" and .block == "passed" and .delete == "passed" and - .cleanup == "passed" and .security_scan == "passed" and .content_logging_scan == "passed" -' "$single_report" >/dev/null - -jq -e --arg commit "$commit" --arg run_id "$run_id" --arg started_at "$started_at" ' - .mode == "ha" and .result == "passed" and .phase == "completed" and - .verification_run_id == $run_id and .commit == $commit and (.image_id | type == "string" and length > 0) and - (.tested_at | type == "string" and . >= $started_at) and - .content_redacted == true and .migration == "passed" and - .chat == "passed" and .stream == "passed" and .tool == "passed" and - .usage == "passed" and .block == "passed" and .delete == "passed" and - .shared_rpm_limit == "passed" and .shared_tpm_limit == "passed" and - .shared_spend_log_visibility == "passed" and .idempotency_recovery == "passed" and - .limiter_source == "litellm_proxy" and .cleanup == "passed" and .security_scan == "passed" and - .content_logging_scan == "passed" -' "$ha_report" >/dev/null - -jq -e --arg commit "$commit" --arg run_id "$run_id" --arg started_at "$started_at" ' - .mode == "ha" and .result == "passed" and .phase == "completed" and - .verification_run_id == $run_id and .commit == $commit and (.image_id | type == "string" and length > 0) and - (.tested_at | type == "string" and . >= $started_at) and - .redis_recovery == "passed" and .content_redacted == true and - .security_scan == "passed" -' "$redis_report" >/dev/null - -image_id="$(jq -r '.image_id' "$migration_report")" -[[ "$image_id" == "$(jq -r '.image_id' "$single_report")" ]] || { echo "single image ID differs" >&2; exit 1; } -[[ "$image_id" == "$(jq -r '.image_id' "$ha_report")" ]] || { echo "HA image ID differs" >&2; exit 1; } -[[ "$image_id" == "$(jq -r '.image_id' "$redis_report")" ]] || { echo "Redis report image ID differs" >&2; exit 1; } -[[ "$image_id" == "$(jq -r '.image_id' "$concurrency_report")" ]] || { echo "concurrency image ID differs" >&2; exit 1; } - -umask 077 -mkdir -p "$(dirname "$output")" -chmod 700 "$(dirname "$output")" -jq -n \ - --arg run_id "$run_id" --arg commit "$commit" --arg image_id "$image_id" \ - --arg tested_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ - --slurpfile migration "$migration_report" --slurpfile concurrency "$concurrency_report" --slurpfile single "$single_report" \ - --slurpfile ha "$ha_report" --slurpfile redis "$redis_report" \ - '{verification_run_id:$run_id,commit:$commit,image_id:$image_id,tested_at:$tested_at,generated_at:$tested_at,result:"passed",phase:"completed",migration:$migration[0],migration_concurrency:$concurrency[0],single:$single[0],ha:$ha[0],redis_recovery:$redis[0],content_redacted:true,local_only:true}' \ - > "$output" -chmod 600 "$output" -echo "PASS aggregate summary: $output" diff --git a/docker_litellm/compose/scripts/verification-lib.sh b/docker_litellm/compose/scripts/verification-lib.sh index 8a9d358..b94f4d5 100755 --- a/docker_litellm/compose/scripts/verification-lib.sh +++ b/docker_litellm/compose/scripts/verification-lib.sh @@ -1,5 +1,5 @@ #!/usr/bin/env bash -# Shared fail-closed helpers for P1 verification producers. Never source .env. +# Shared fail-closed helpers for the migration/verification scripts. Never source .env. verification_prepare_environment() { local env_file="$1" compose_file="$2" work_dir="$3" diff --git a/docker_litellm/compose/scripts/verify-migration-concurrency.sh b/docker_litellm/compose/scripts/verify-migration-concurrency.sh deleted file mode 100755 index fbdf272..0000000 --- a/docker_litellm/compose/scripts/verify-migration-concurrency.sh +++ /dev/null @@ -1,89 +0,0 @@ -#!/usr/bin/env bash -# Prove actual overlapping migration jobs serialize on PostgreSQL's advisory lock. -set -euo pipefail - -script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -compose_dir="$(cd "${script_dir}/.." && pwd)" -source "${script_dir}/verification-lib.sh" -env_file="${LITELLM_SMOKE_ENV_FILE:-${compose_dir}/.env}" -summary_file="${compose_dir}/artifacts/p1-migration-concurrency.json" -run_id="${VERIFICATION_RUN_ID:?VERIFICATION_RUN_ID is required}" -tmpdir="" -image_ref="" -result="failed" -phase="initializing" -first="" -second="" -actual_overlap=false -lock_wait_observed=false -exclusive_lock=false -max_lock_holders=0 -migration_execution_count=0 - -verification_invalidate_report "$summary_file" -cleanup() { - local rc=$? tmp - trap - EXIT - [[ -z "$first" ]] || docker rm "$first" >/dev/null 2>&1 || true - [[ -z "$second" ]] || docker rm "$second" >/dev/null 2>&1 || true - tmp="${summary_file}.tmp.$$" - jq -n --arg run_id "$run_id" --arg commit "$(git -C "$compose_dir/../.." rev-parse HEAD)" \ - --arg image_id "$(docker image inspect "$image_ref" --format '{{.Id}}' 2>/dev/null || true)" \ - --arg tested_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" --arg result "$result" --arg phase "$phase" \ - --argjson actual_overlap "$actual_overlap" --argjson lock_wait_observed "$lock_wait_observed" \ - --argjson exclusive_lock "$exclusive_lock" --argjson max_lock_holders "$max_lock_holders" \ - --argjson migration_execution_count "$migration_execution_count" \ - '{verification_run_id:$run_id,commit:$commit,image_id:$image_id,tested_at:$tested_at,mode:"migration",result:$result,phase:$phase,concurrent_migration:($result == "passed"),actual_overlap:$actual_overlap,lock_wait_observed:$lock_wait_observed,exclusive_lock:$exclusive_lock,max_lock_holders:$max_lock_holders,migration_execution_count:$migration_execution_count,proxy_replicas_started:false,content_redacted:true}' > "$tmp" && chmod 600 "$tmp" && mv "$tmp" "$summary_file" - [[ -z "$tmpdir" ]] || rm -rf "$tmpdir" - return "$rc" -} -trap cleanup EXIT - -tmpdir="$(mktemp -d "${TMPDIR:-/tmp}/litellm-migration-concurrency.XXXXXX")" -chmod 700 "$tmpdir" -verification_prepare_environment "$env_file" "${compose_dir}/docker-compose.litellm.yml" "$tmpdir" -image_ref="$(verification_env LITELLM_IMAGE)" -compose=(docker compose --env-file "$env_file" -f "${compose_dir}/docker-compose.litellm.yml") -"${compose[@]}" up -d --wait postgres redis -phase="starting_concurrent_jobs" - -# The test hold makes concurrent overlap observable without changing normal -# migration behavior. Both containers are real migration jobs; only the lock -# holder may enter LiteLLM migration execution. -first="$("${compose[@]}" --profile migrate run -d --no-deps -e LITELLM_MIGRATION_LOCK_HOLD_SECONDS=4 litellm-migrate)" -sleep 1 -second="$("${compose[@]}" --profile migrate run -d --no-deps -e LITELLM_MIGRATION_LOCK_HOLD_SECONDS=4 litellm-migrate)" -[[ -n "$first" && -n "$second" && "$first" != "$second" ]] - -phase="observing_lock" -for _ in $(seq 1 20); do - state="$(docker inspect -f '{{.State.Running}} {{.State.Running}}' "$first" "$second" 2>/dev/null | tr '\n' ' ')" - if [[ "$state" == *"true true"* ]]; then actual_overlap=true; fi - "${compose[@]}" exec -T postgres sh -lc 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -At -v ON_ERROR_STOP=1 -c "SELECT count(*) FROM pg_locks WHERE locktype = '\''advisory'\'' AND granted;"' > "$tmpdir/lock-holders" - holders="$(tr -d '[:space:]' < "$tmpdir/lock-holders")" - [[ "$holders" =~ ^[0-9]+$ ]] - (( holders > max_lock_holders )) && max_lock_holders="$holders" - (( holders <= 1 )) || { echo "more than one migration advisory lock holder" >&2; exit 1; } - combined_logs="$(docker logs "$first" 2>&1; docker logs "$second" 2>&1)" - if [[ "$combined_logs" == *P1_MIGRATION_LOCK_WAITING* && "$(printf '%s' "$combined_logs" | rg -c 'P1_MIGRATION_LOCK_ACQUIRED')" == "1" ]]; then - lock_wait_observed=true - fi - [[ "$actual_overlap" == true && "$lock_wait_observed" == true ]] && break - sleep 1 -done -[[ "$actual_overlap" == true ]] -[[ "$lock_wait_observed" == true ]] -[[ "$max_lock_holders" == 1 ]] -exclusive_lock=true - -phase="waiting_for_serialized_jobs" -docker wait "$first" "$second" > "$tmpdir/exit-codes" -[[ "$(tr -d '[:space:]' < "$tmpdir/exit-codes")" == "00" ]] -combined_logs="$(docker logs "$first" 2>&1; docker logs "$second" 2>&1)" -[[ "$(printf '%s' "$combined_logs" | rg -c 'P1_MIGRATION_EXECUTION_START')" == "2" ]] -[[ "$(printf '%s' "$combined_logs" | rg -c 'P1_MIGRATION_EXECUTION_DONE')" == "2" ]] -migration_execution_count=2 -! "${compose[@]}" ps --services --status running | rg -q '^litellm-[12]$' -phase="completed" -result="passed" -echo "PASS concurrent migration: overlapping jobs observed; PostgreSQL advisory lock held by at most one job." diff --git a/docker_litellm/compose/scripts/verify-p1.sh b/docker_litellm/compose/scripts/verify-p1.sh deleted file mode 100755 index 72e47f9..0000000 --- a/docker_litellm/compose/scripts/verify-p1.sh +++ /dev/null @@ -1,32 +0,0 @@ -#!/usr/bin/env bash -# Execute one complete, non-reusable P1 verification run. -set -euo pipefail - -script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -compose_dir="$(cd "${script_dir}/.." && pwd)" -source "${script_dir}/verification-lib.sh" -run_id="$(verification_new_run_id)" -export VERIFICATION_RUN_ID="$run_id" -export VERIFICATION_STARTED_AT="$(date -u +%Y-%m-%dT%H:%M:%SZ)" -# P1's documented local provider mapping is explicit. It is non-secret and -# prevents a template/default mismatch from silently selecting another SDK. -export LITELLM_SMOKE_UPSTREAM_PROVIDER="${LITELLM_SMOKE_UPSTREAM_PROVIDER:-deepseek}" -compose=(docker compose --env-file "${compose_dir}/.env" -f "${compose_dir}/docker-compose.litellm.yml") -cleanup_stack() { "${compose[@]}" --profile single --profile ha --profile migrate down >/dev/null 2>&1 || true; } -trap cleanup_stack EXIT - -cleanup_stack -for report in p1-migration-summary.json p1-migration-concurrency.json p1-single-summary.json p1-ha-summary.json p1-redis-recovery.json p1-final-summary.json; do - verification_invalidate_report "${compose_dir}/artifacts/${report}" -done -"${script_dir}/test-verification-gates.sh" -"${script_dir}/run-migration.sh" -"${script_dir}/verify-migration-concurrency.sh" -"${script_dir}/run-migration.sh" -"${compose[@]}" --profile single up -d --wait postgres redis litellm-1 -"${script_dir}/test-verification-gates.sh" --with-running-stack -"${script_dir}/smoke-baseline.sh" --mode single -"${compose[@]}" --profile ha up -d --wait postgres redis litellm-1 litellm-2 -"${script_dir}/smoke-baseline.sh" --mode ha -"${script_dir}/smoke-redis-recovery.sh" -"${script_dir}/aggregate-verification-summary.sh" diff --git a/docker_litellm/litellm.Dockerfile b/docker_litellm/litellm.Dockerfile index 1c7e159..7bc6ec6 100644 --- a/docker_litellm/litellm.Dockerfile +++ b/docker_litellm/litellm.Dockerfile @@ -19,7 +19,7 @@ ENV NODE_ENV=development WORKDIR /build # Clone the fixed source and build its Python wheel. Dashboard export is -# optional: P1 validates the API proxy, not the browser dashboard. +# optional: the API proxy does not depend on the browser dashboard. RUN set -eux \ && git init . \ && git remote add origin https://github.com/BerriAI/litellm.git \ diff --git a/docker_litellm/work/start-litellm.sh b/docker_litellm/work/start-litellm.sh index 95f24a3..74a69b3 100755 --- a/docker_litellm/work/start-litellm.sh +++ b/docker_litellm/work/start-litellm.sh @@ -50,7 +50,7 @@ fi # Keep metering enabled in config.yaml, but never persist prompt content. export STORE_PROMPTS_IN_SPEND_LOGS="${STORE_PROMPTS_IN_SPEND_LOGS:-false}" -# Default config if not exists. The P1 Compose baseline always mounts an +# Default config if not exists. The Compose baseline always mounts an # explicit config with PostgreSQL and Redis; this fallback remains only for # backwards-compatible standalone use. if [ ! -f "config.yaml" ]; then diff --git a/docker_openclaw/README.md b/docker_openclaw/README.md index 4db1f2b..ff7bc5e 100644 --- a/docker_openclaw/README.md +++ b/docker_openclaw/README.md @@ -38,9 +38,3 @@ docker run -d \ -v openclaw_data:/root/.openclaw/data \ labnow/openclaw:latest ``` - -## P6 冻结证据归档 - -P6 的跨仓固定组合、黄金 runner 与报告聚合属于已冻结的一次性验收证据,已从 -产品运行面删除。历史回读方式见 [`p6/README.md`](p6/README.md);它不是 OpenClaw -的构建、启动或 CI 入口。 diff --git a/docker_openclaw/p6/README.md b/docker_openclaw/p6/README.md deleted file mode 100644 index 01b6b5b..0000000 --- a/docker_openclaw/p6/README.md +++ /dev/null @@ -1,15 +0,0 @@ -# P6 OpenClaw 产品闭环(已归档) - -LLM Hub V1 / P6 的固定组合、黄金 runner、受限输入、运行时 Compose 与报告聚合 -均为一次性冻结验收证据,已按生产加固决策 D-10 从产品运行面删除。它们不是持续 -维护的 OpenClaw 启动入口,也不得重新用于日常或 CI 验证。 - -最后可读快照是本批基线 -`fdbbab2155a9e088c37d2a8a2057178e19ac9534`。需要审阅历史材料时,在本仓执行: - -```bash -git show fdbbab2155a9e088c37d2a8a2057178e19ac9534:docker_openclaw/p6/<路径> -``` - -仍保留的 `scripts/test-p6-gates.sh` 是不启动容器的静态产品契约检查;OpenClaw -的受支持构建与日常启动方式见父目录 README 和 `compose/`。 From f87a544157684360487f2261ed72188321e78bb0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Fri, 21 Aug 2026 15:14:16 +0800 Subject: [PATCH 85/87] =?UTF-8?q?chore:=20=E6=B8=85=E7=90=86=20Dockerfile/?= =?UTF-8?q?config=20=E6=B3=A8=E9=87=8A=E4=B8=AD=E7=9A=84=E9=98=B6=E6=AE=B5?= =?UTF-8?q?=E6=A0=87=E8=AE=B0?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 --- docker_hermes/hermes.Dockerfile | 2 +- docker_litellm/compose/config.yaml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docker_hermes/hermes.Dockerfile b/docker_hermes/hermes.Dockerfile index e4405ae..62ea5be 100644 --- a/docker_hermes/hermes.Dockerfile +++ b/docker_hermes/hermes.Dockerfile @@ -5,7 +5,7 @@ ARG BASE_IMG_BUILD="node" ARG BASE_IMG="base" ARG HERMES_BUILD_BASE_IMAGE ARG HERMES_RUNTIME_BASE_IMAGE -# P7: the upstream source is a release input, not a moving branch. Keep the +# The upstream source is a release input, not a moving branch. Keep the # repository and commit overridable only so the local runner can bind both to # its protected input and record the exact provenance. ARG HERMES_SOURCE_REPOSITORY="https://github.com/nousresearch/hermes-agent.git" diff --git a/docker_litellm/compose/config.yaml b/docker_litellm/compose/config.yaml index 592ec5c..f828b0d 100644 --- a/docker_litellm/compose/config.yaml +++ b/docker_litellm/compose/config.yaml @@ -1,4 +1,4 @@ -# P1 runtime configuration. Secrets only come from environment variables. +# Runtime configuration. Secrets only come from environment variables. # The management API persists users, credentials, model records and virtual keys # in PostgreSQL. Redis is the shared cache/co-ordination backend for replicas. From aaddd13615926e31d3508fc05dca455b212860aa Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Fri, 21 Aug 2026 17:05:54 +0800 Subject: [PATCH 86/87] =?UTF-8?q?chore:=20AI=20=E5=8D=8F=E4=BD=9C=E7=BA=A6?= =?UTF-8?q?=E5=AE=9A=E6=96=87=E4=BB=B6=E7=A7=BB=E5=87=BA=E4=BB=93=E5=BA=93?= =?UTF-8?q?(=E6=9C=AC=E5=9C=B0=E4=BF=9D=E7=95=99)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit AGENTS.md / CLAUDE.md / .claude/skills 为本地开发协作配置,不进入版本控制。 Co-Authored-By: Claude Fable 5 --- .claude/skills | 1 - CLAUDE.md | 1 - 2 files changed, 2 deletions(-) delete mode 120000 .claude/skills delete mode 120000 CLAUDE.md diff --git a/.claude/skills b/.claude/skills deleted file mode 120000 index cd2ebc5..0000000 --- a/.claude/skills +++ /dev/null @@ -1 +0,0 @@ -../doc/skills \ No newline at end of file diff --git a/CLAUDE.md b/CLAUDE.md deleted file mode 120000 index 47dc3e3..0000000 --- a/CLAUDE.md +++ /dev/null @@ -1 +0,0 @@ -AGENTS.md \ No newline at end of file From a7c8a6427b716c28f94b9db220ac2755644f4eb1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=99=88=E5=BA=9A?= <12574507+large-mushroom77@user.noreply.gitee.com> Date: Fri, 21 Aug 2026 17:06:25 +0800 Subject: [PATCH 87/87] =?UTF-8?q?chore:=20AGENTS.md=20=E7=A7=BB=E5=87=BA?= =?UTF-8?q?=E4=BB=93=E5=BA=93(=E6=9C=AC=E5=9C=B0=E4=BF=9D=E7=95=99,?= =?UTF-8?q?=E8=A1=A5=E5=89=8D=E4=B8=80=E6=8F=90=E4=BA=A4)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 --- AGENTS.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) delete mode 100644 AGENTS.md diff --git a/AGENTS.md b/AGENTS.md deleted file mode 100644 index e69de29..0000000