Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

## Unreleased — security remediation pass

- Fix unattended Codex inspection with `approvalPolicy: never` under unchanged external Docker isolation. Keep client approval/tool grants denied and fail on denied/missing inspection, incomplete context, inconclusive/quarantined/truncated analysis or invalid review locations. Preserve safe separate COMMENT publication. Add per-review `--model` with model-specific effort validation and existing quota/idempotency binding. No schema, credential or permission migration; coordinate runner/Worker/caller pins.

- Classify rejected Responses replies with their actual upstream status (including 2xx), fixed Content-Type category and body presence. Extract only allowlisted codes/parameters from bounded JSON errors; keep non-SSE success responses fail-closed and provider metadata confined to the credential-proxy boundary.

- Failed inference now fails the review job even after recording a safe COMMENT fallback. Preserve separate protected publication, distinguish fallback publication from completed AI review, and carry only allowlisted Docker/Codex/RPC/proxy diagnostics. Local AppError statuses are not presented as provider HTTP responses. Runner changes require a coordinated immutable source/Worker/caller pin upgrade; no credential or database migration.
Expand Down
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ Review authentication, authorization, migrations and compatibility.

Configure a trigger plus up to five aliases matching `@[A-Za-z0-9]{1,39}`. Matching ignores case. These are text commands and can notify unrelated GitHub users with the same name; choose distinctive aliases. Quoted examples and fenced commands do not trigger reviews. Model and reasoning effort are separate; omitted effort preserves Codex's default.

Use `--model MODEL_ID` (or `--model=MODEL_ID`) to override the repository's model for one review; replace MODEL_ID with an available Codex catalog identifier. An unsupported model or effort fails without substitution or billing fallback. Missing inspection, incomplete context or an inconclusive result fails the review job while the separate publisher may still publish a safe COMMENT.

Set `bot.trigger: "@CBReview"` to use the example above. The legacy default `@codexbot` remains supported; new setup chooses `@CBReview<repository_id>`. Use a name without “codex” when another Codex integration is installed. CodexBot cannot control that independent connector; see [trigger isolation and rollout](docs/review-rollout.md).

## Status
Expand Down
4 changes: 3 additions & 1 deletion docs/acceptance.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,11 +70,13 @@ CHATGPT_DEPLOYMENT_AUTHORIZED=true npm run acceptance:provider -- --mode oauth \

# Separate intentional paid API test:
npm run acceptance:provider -- --mode api-key --credential-file /private/operator/api-key.json \
--model YOUR_MODEL --max-requests 1 --allow-paid-api
--model YOUR_MODEL --max-requests 3 --allow-paid-api
```

Build the pinned image first and use a nonroot Linux host. The fixture sends no real private source or GitHub writes. Requests are limited to an explicit 1–3, each input to 50 KB; the API-key harness adds a 2048-output-token cap. These are usage-unit ceilings, not a guaranteed currency ceiling: select the model using current pricing and an operator-controlled project/account budget. OAuth must omit unsupported API fields. Failure or truncation never becomes a passing check.

The provider harness now reviews a synthetic authorization regression, not an empty no-tool prompt. It supplies a diff, both source versions and trusted project instructions; the change makes a role check always truthy. Success requires actual inspection, complete schema-valid findings and a finding on `src/access.js` RIGHT line 3. The model is passed through the same per-review override path. Reading and reporting normally need at least two provider requests; a one-request budget is an intentional failure test. A model that exceeds the chosen three-request ceiling fails rather than borrowing more quota. This real-provider gate is opt-in and has not been executed for the approval-policy fix; scripted Responses in CI prove native command execution/isolation, not AI defect recognition. Never run this harness with an API key without explicit paid-usage authorization.

For OAuth success, require real streamed inference to a valid completed structured result using **only** the eligible SIWC access credential and verify the corresponding ChatGPT usage allowance/account/workspace. Confirm no API-key secret exists and no API-billed inference occurred in that isolated test window. A synthetic stream, successful token exchange, model list or HTTP 200 alone is insufficient. Check explicit denied plan scope, expired/revoked grants, actual refresh/rotation across two runs, disconnect/revoke and exhausted allowance; each must fail closed without paid fallback. Repeat API-key success and revoked/invalid-key behavior independently with bounded intentional billing.

## Resource and release gates
Expand Down
2 changes: 2 additions & 0 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,8 @@ The manifest distinguishes `diffComplete`, `requiredContextComplete` and supplem

Nonroot Docker containers use read-only root filesystems, resource caps, no capabilities, no-new-privileges and private temporary homes. The reviewer has network=none and only context/socket read-only mounts. It has no Docker socket, GitHub/OIDC token, API key, refresh token or vault key. A separate audited proxy owns the access credential, only permits stateless streaming Responses inference at one HTTPS route, rejects hosted network tools, pins model/effort/service-tier selection and replaces provider errors. Default limits are 100 requests and a 20-minute turn. The host owns encrypted session rotation and all GitHub API access.

Codex 0.169.0 uses `approvalPolicy: never` for unattended inspection. The thread's read-only default remains, and each review turn explicitly selects `externalSandbox` with restricted network, relying on the unchanged Docker boundary. `never` is not a client approval grant: all server-initiated approval/tool/permission requests still receive JSON-RPC -32601 and cause the analysis to fail. Native declined commands also fail. At least one completed command with exit 0 is necessary before a turn can produce a review; this is execution evidence, not proof of full semantic coverage. Both the isolated entrypoint and trusted host validate completeness, exact head and changed-line references. Incomplete/quarantined/truncated/inconclusive output fails the review job; the protected publisher can still publish a non-approving fallback.

## Durable identity and side effects

D1 receipts distinguish receipt/processing/retryable/done/failed. A request stores a dispatch-attempt UUID before the external POST. Workflow run names contain request plus attempt UUID; reconciliation verifies run identity, App actor, repository, event and immutable workflow SHA. Lost dispatch responses remain usable: prepare accepts an authorized created/dispatched request. A run-ID CAS admits one runner. A bounded resend can launch an extra trusted workflow after ambiguous provider failure; its prepare is denied once another run wins. Distributed exactly-once dispatch is not claimed.
Expand Down
6 changes: 4 additions & 2 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,17 +45,19 @@ Patterns are full-string, case-insensitive RE2. `*` matches a sequence; `?` one

## Commands and aliases

The first nonempty top-level line must begin with a configured trigger and `review`. Four-space/tab indents, quotations, inline examples and code fences do not invoke. Flags may occur in any order on that first line: `--merge`, `--effort LEVEL`, `--effort=LEVEL`, `--rerun`, and `--` for literal context. Duplicates, unknown flags and unclosed quotes are errors. Later lines are advisory context, never flags or authorization.
The first nonempty top-level line must begin with a configured trigger and `review`. Four-space/tab indents, quotations, inline examples and code fences do not invoke. Flags may occur in any order on that first line: `--merge`, `--model MODEL_ID`, `--model=MODEL_ID`, `--effort LEVEL`, `--effort=LEVEL`, `--rerun`, and `--` for literal context. Duplicates, unknown flags and unclosed quotes are errors. Later lines are advisory context, never flags or authorization.

Aliases are textual commands, not guaranteed GitHub App mentions. A custom alias can notify an unrelated GitHub account with the same name. Existing @codexbot installations remain compatible. New automated setup uses @CBReview followed by the numeric repository ID to reduce collisions; it remains editable. Configure `bot: {trigger: "@CBReview", aliases: []}` for a short explicit alternative. Avoid adding the old name as an alias if the purpose is avoiding an independent connector. Changing a trigger never changes the App's publishing identity. See [trigger isolation](review-rollout.md).

Effort syntax recognizes none/minimal/low/medium/high/xhigh/max, but the selected model must actually advertise the requested value. Omission preserves Codex defaults. Catalog discovery does not prove account entitlement; completed inference is required. Never substitute an unsupported value or change billing modes.

Model precedence is the explicit command override, then `review.model`, then the Codex catalog default. Model identifiers use 1–100 ASCII letters, digits, dots, underscores or hyphens, cannot begin with a flag hyphen and must match an actual catalog entry. Effort is validated against that selected model; a model override never silently changes an unsupported requested effort. Existing snapshots without `command.model` retain the previous behavior. Overrides are included in the existing request idempotency key and remain subject to normal authorization and quotas; they grant no merge permissions.

## Review decision table

| Evidence | Native decision |
|---|---|
| Incomplete diff/essential context/model analysis, malformed output, quarantine or output loss | COMMENT |
| Incomplete diff/essential context/model analysis, malformed output, quarantine or output loss | COMMENT; review job fails |
| Complete critical/high finding at or above blocking confidence | REQUEST_CHANGES, or COMMENT when disabled |
| Critical/high below threshold, or any medium/low finding | COMMENT; never approve by ignoring uncertainty |
| Complete clean result, zero findings, approval enabled | APPROVE |
Expand Down
2 changes: 2 additions & 0 deletions docs/development.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,8 @@ check runs strict TypeScript, ESLint, Node tests and production compilation. SQL

test:workers runs real workerd, applies migrations to real disposable local D1, checks HMAC rejection, durable receipts/deduplication, payload cleanup and scheduled recovery. No Node-only D1 mock substitutes for this check. test:sandbox drives a real Codex turn through the Unix-socket proxy against synthetic SSE in the built image. It checks credential/network/filesystem isolation, not real inference or account entitlement. Production orchestration needs a nonroot Linux runner; Windows Docker Desktop can execute the fixture test.

The sandbox suite also executes the pinned `functions.exec` code-mode tool and its `tools.exec_command` against an isolated synthetic repository. A negative control restores only `untrusted` and observes an actual rejected approval; `never` reads the exact diff/base/head/trusted-instruction bytes (verified by nonce-specific SHA-256), while write/network/credential probes remain blocked. Scripted complete/findings/incomplete results test schema/line validation and exit behavior; complete JSON without inspection fails. No source, tool output or raw Codex stderr is printed. Use the separately authorized provider acceptance harness to establish real AI recognition of the introduced bug.

For disposable local Worker development copy .dev.vars.example to ignored .dev.vars, configure only test credentials, apply D1 migrations with --local and run npm run dev. Never copy production credentials. Actual provisioning requires a controlled HTTPS origin. Production remote migration/deploy, live GitHub writes and inference are never part of check or CI.

Keep changes focused and retain deterministic security checks. Review [architecture](architecture.md), [security](security.md), [operations](operations.md) and [acceptance](acceptance.md) before changing workload identity, credential rotation, quotas or publication. No artifact should contain secrets/private source; do not broaden workflow permissions or run PR code on the credential-bearing host.
10 changes: 10 additions & 0 deletions docs/validation.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,15 @@
# Validation record and release gates

## Unattended inspection, completion gates and per-review model

The reported completed-but-inconclusive production review is explained by `thread/start` using `approvalPolicy: untrusted` while the client rejects every server-initiated approval request. The pinned [0.169.0 exec-policy implementation](https://github.com/openai/codex/blob/rust-v0.169.0/codex-rs/core/src/exec_policy.rs) prompts for unmatched commands under UnlessTrusted and allows non-dangerous unmatched commands under Never, relying on the sandbox. The generated native protocol confirms `never`, `externalSandbox` and `commandExecution` completed/declined states. Docker tests change only the native thread's approval setting for the negative control: untrusted triggers an actual rejected approval; never completes file access. The external Docker isolation and client -32601 refusal stay unchanged.

The real pinned app-server code-mode tool (`functions.exec` calling `tools.exec_command`) reads nonce-specific synthetic diff, base/head source and trusted instructions, verified by SHA-256 without raw source/tool-output logs. The same command verifies denied repository/runtime writes, no external network, absent credential environment/file and absent Docker socket. Scripted provider responses then exercise a schema-valid finding at changed RIGHT line 3, a genuine clean fixture and an inconclusive fixture. Complete JSON without command execution fails; incomplete results fail despite completed inference. Host tests independently reject inconclusive/contradictory results and missing essential context, and verify fallback recording plus release. Worker tests show separate COMMENT publication for a failed review and APPROVE only for a complete clean result. Per-review model tests cover flag syntax/order/quoting, validation, precedence, supported effort, persistence and quota/idempotency binding.

Passed locally: `npm run check` (148 tests, strict typecheck, lint, build), `npm run test:clean`, real workerd/D1, 41 targeted execution/security/Worker tests, Docker build, `npm run test:sandbox`, Actionlint, dependency audit, strict HIGH/CRITICAL image scan and Wrangler dry-run. These tests prove actual native inspection/approval behavior and isolation; the model replies are scripted, so they do **not** prove live AI recognition of the introduced defect. The opt-in real-provider harness now requires a complete review of the same introduced authorization bug and a valid changed-line finding. It was not run: no operator credential input was supplied, and production execution of the new source requires an approved pin/deployment cutover. No paid inference, reconnect, deployment, protected merge or secret change was performed. First successful production AI review therefore remains an external acceptance gate.

Rollout requires a normally reviewed/merged source PR, then the actual merged source SHA in both Worker trust plus exact backend test expectation and the protected caller. Drain old reviews; deploy only with explicit approval, coordinate caller cutover without relaxing OIDC, compare live pins, run credential-only diagnostics, then obtain owner authorization for a fresh advisory/no-merge review. Pin drafts must remain blocked until their provisional feature SHA is replaced by the verified source merge SHA. No D1/session/credential migration is required. Old snapshots without a model override remain compatible; they cannot bypass the immutable source pin.

## Provider response-format diagnostics — 2026-11-14

Inspected `inferenceProxy` at pinned source `0e937476323bbe35e3c33dc3a8572e62d1e544e5`: the observed format error occurs after `upstream.ok`, when the response body is absent or the SSE Content-Type guard fails. The original diagnostic contains no actual status, header category or body-presence evidence; a specific authentication, entitlement, model, HTML or JSON cause is unproven. The observed production workflow correctly failed analysis while its separate non-approving publisher succeeded.
Expand Down
16 changes: 9 additions & 7 deletions scripts/provider-acceptance.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,8 @@ import { readPrivateFile, atomicPrivateFile, withFileLock } from '../dist/src/au
import { OAuth, validateSession } from '../dist/src/auth/oauth.js';
import { isolatedReview } from '../dist/src/runner/docker.js';
import { DEFAULT_CONFIG } from '../dist/src/core/config.js';
import { normalizeReview } from '../dist/src/core/review.js';
import { normalizeReview, validateReview } from '../dist/src/core/review.js';
import { writeInspectionFixture } from '../tests/inspection-fixture.mjs';
const { values } = parseArgs({ options: { mode: { type: 'string' }, 'credential-file': { type: 'string' }, model: { type: 'string' }, effort: { type: 'string' }, image: { type: 'string', default: 'codexbot-review:local' }, 'allow-live-oauth': { type: 'boolean' }, 'allow-paid-api': { type: 'boolean' }, 'max-requests': { type: 'string' } } });
let directory;
try {
Expand All @@ -31,12 +32,13 @@ try {
const context = join(directory, 'context'), sockets = join(directory, 'sockets'), key = join(directory, 'credential');
await mkdir(context, { mode: 0o700 }); await mkdir(sockets, { mode: 0o700 }); await mkdir(join(context, 'source'), { mode: 0o700 });
await writeFile(key, credential, { mode: 0o600 });
const head = 'a'.repeat(40), config = globalThis.structuredClone(DEFAULT_CONFIG); config.credentials.mode = oauth ? 'chatgpt' : 'api-key'; config.review.model = values.model;
const request = { head, base: 'b'.repeat(40), config, command: { effort: values.effort ?? null, merge: false, instructions: 'Provider acceptance only. Empty change fixture; do not claim real repository analysis. Complete the output JSON without tool calls.' } };
for (const [file, value] of [['request.json', request], ['manifest.json', { complete: true, diffComplete: true, requiredContextComplete: true }], ['diff.json', []], ['trusted-instructions.json', []]]) await writeFile(join(context, file), JSON.stringify(value), { mode: 0o600 });
const output = normalizeReview(await isolatedReview(values.image, context, key, sockets, { maxRequests, maxInputBytes: 50_000, ...(!oauth ? { maxOutputTokens: 2048 } : {}) }), head);
assert.ok(output.complete && !output.quarantined && !output.output_truncated);
process.stdout.write(`Real ${oauth ? 'OAuth' : 'API-key'} provider completed structured inference. No GitHub writes performed.\n`);
const head = 'a'.repeat(40), config = globalThis.structuredClone(DEFAULT_CONFIG); config.credentials.mode = oauth ? 'chatgpt' : 'api-key';
const request = { head, base: 'b'.repeat(40), config, command: { model: values.model, effort: values.effort ?? null, merge: false, instructions: 'Acceptance review of this synthetic repository only. Inspect the provided diff, both source versions and trusted instructions. Do not execute project code or claim tests passed.' } };
const fixture = await writeInspectionFixture(context, request);
const output = validateReview(normalizeReview(await isolatedReview(values.image, context, key, sockets, { maxRequests, maxInputBytes: 50_000, ...(!oauth ? { maxOutputTokens: 2048 } : {}) }), head), head, fixture.files);
assert.ok(output.complete && output.result === 'findings' && !output.quarantined && !output.output_truncated);
assert.ok(output.findings.some(finding => finding.path === 'src/access.js' && finding.line === 3 && finding.side === 'RIGHT'));
process.stdout.write(`Real ${oauth ? 'OAuth' : 'API-key'} provider inspected a synthetic repository and reported its changed authorization line. No GitHub writes performed.\n`);
if (oauth) process.stdout.write('Launch gate remains pending until the operator verifies ChatGPT-plan usage and absence of API billing for this request.\n');
});
} catch { process.stderr.write('Provider acceptance failed or was not explicitly enabled. No raw provider output or credentials printed.\n'); process.exitCode = 1; }
Expand Down
Loading