Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

## Unreleased — security remediation pass

- Failed inference now fails the review job even after recording a safe COMMENT fallback. Preserve separate protected publication, distinguish fallback publication from completed AI review, and carry only allowlisted Docker/Codex/RPC/proxy diagnostics. Local AppError statuses are not presented as provider HTTP responses. Runner changes require a coordinated immutable source/Worker/caller pin upgrade; no credential or database migration.

- Validate broker default-branch identity using the authoritative repository API, not the minimal repository embedded in Actions runs. Preserve signed OIDC and all run/App dispatch checks across prepare, failure, publication and cleanup operations.

- Stage the hosted origin on codexbot.appsolves.workers.dev with Workers.dev enabled, previews disabled and no custom-domain routes. Default CLI/generated callers and the onboarding page/manifest to the new URL. Reject all privileged requests on noncanonical aliases, keep exact OIDC audiences and require fresh origin-bound ChatGPT connection. The retired domain is removed; rollback must stay on Workers.dev. No code deployment, zone-security change or credential migration is automatic; see docs/origin-migration.md.
Expand Down
14 changes: 14 additions & 0 deletions docs/review-rollout.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,20 @@ OpenAI's connector is independent. Its [documented manual command](https://learn

## Coordinated immutable-pin upgrade

### Isolated failure diagnostics and workflow conclusions

Failed analysis must fail the review job, including after successfully recording an incomplete fallback. The protected publisher still runs after a failed review job and may successfully publish that COMMENT; this does not make the overall workflow green. The publisher also durably invalidates complete context when `review_job_failed` is true. Progress explicitly calls a published fallback a failed analysis, while displaying GitHub's actual workflow conclusion (including success for historical buggy runs).

The runner's `execution` record identifies Docker start/exit/output/timeout, Codex process/RPC, model selection or turn failure. Optional `proxy_failure` is the **last observed** proxy failure for that same ephemeral run, not proof that it caused the terminal turn failure. `upstream_status` is populated only from an actual HTTP response at the credential proxy. RPC numeric codes, methods, exit codes, signals, provider codes and parameter names are projected through fixed vocabularies/ranges. Unknown details are omitted. A local AppError default 400 proves no upstream HTTP status; broker HTTP statuses remain separate.

Neither raw stderr nor provider `message`, `detail`, headers, URLs, request IDs, bodies, prompts, source or model output are logged. Diagnostic line/total budgets are 2 KiB/64 KB; provider error inspection is capped at 8 KiB. Unknown/oversized/non-JSON provider errors retain only the actual HTTP status. Codex child stderr is drained and discarded. Public progress contains fixed generic recovery text, not these internal diagnostics. A lost fallback callback preserves the primary job failure and emits a fixed reporting-deferred record; it cannot approve. Existing run-bound leases, expiry/cron recovery and publication retries remain in force; never reset an active production session to retry inference.

The original generic isolated-review log cannot establish whether the provider rejected authentication, entitlement, model/capability, usage or whether the process failed locally. Do not reconnect OAuth or change endpoints based on that code alone. The official [Responses limitations](https://developers.openai.com/siwc/token-sharing-open-source/preview-limitations) require `store:false`, streaming and explicit input history, prohibit several fields/tools, and retain the public Responses endpoint. [Documented recovery codes](https://developers.openai.com/siwc/token-sharing-open-source/errors-and-recovery) distinguish scoped entitlement/usage failures; an unknown 403 alone is not an entitlement diagnosis. The pinned Codex catalog is not account entitlement proof. There is no paid fallback or proxy-policy relaxation.

This fix changes runner source even though the reusable YAML does not need changing. A backend-only deployment cannot fix an old pinned runner. Prepare source and rollout PRs separately. Rollout PRs may provisionally reference the **actual reviewed feature commit** for review, but must remain Draft until the source PR has merged. If GitHub squash/rebase changes its SHA, update both rollout PRs to the actual merged source SHA before approval. Never deploy a guessed future SHA or accept two identities to hide a mismatch. No migration, reconnect, secret change or App permission increase is needed for this patch.

After approval, drain existing reviews, deploy the merged Worker source with its reviewed exact `TRUSTED_WORKFLOW`, and merge the caller pin PR through normal protection. During cutover any pin mismatch must fail closed. Compare both pins, run a fresh credential-only diagnostic, and only with explicit owner authorization request a fresh advisory/no-merge review. Verify failed synthetic/adversarial inference leaves a failed workflow plus a COMMENT fallback; a real complete advisory review remains a separate live gate. Roll back both pins together to the prior verified source; do not rerun old production attempts or reset D1/session state.

1. Validate and merge the feature branch. Obtain the **actual merged commit SHA** from GitHub and verify that it contains the reviewed reusable workflow. Never substitute an example, branch name, guessed SHA or an unrelated last commit.
2. Drain active old reviews. Apply migration 0003 using pinned Wrangler's remote migrations command. It is additive; do not reapply ALTER statements individually. Keep `MERGE_DEPLOYMENT_AUTHORIZED=false` and repository merge disabled.
3. Prepare a caller PR using that exact SHA plus the environment-free credential-scope guard, required dependency, named secret references and optional credential-only input. Preserve its run-name, permissions, concurrency and default-branch protection. Update the textual trigger separately if desired; do not change authorization or merge policy.
Expand Down
10 changes: 10 additions & 0 deletions docs/validation.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,15 @@
# Validation record and release gates

## Isolated review failure and misleading success — 2026-11-14

Verified root cause of the green workflow: `reviewJob` caught inference failure, saved the incomplete fallback and returned successfully. It now always rethrows the primary failure, even when reporting fails; the existing separately protected publisher remains eligible and can only publish a non-approving report. Original container/Codex/provider diagnostics were discarded, so the production inference failure's specific cause remains unknown. Its old `400` was a local AppError default, not evidence of upstream HTTP 400.

Regression coverage in `execution.test.ts` exercises the actual review job through synthetic GitHub/OIDC/Docker boundaries, fallback persistence, primary-error preservation and session release. It also covers Docker exit classes/stderr drain, RPC rejection/process exit, model selection and failed/interrupted/malformed turns. Proxy/stream tests distinguish actual HTTP status and documented entitlement/scope/usage/model codes from validation/stream failures and discard malicious/unknown/oversized details. Progress tests distinguish failed analysis plus successful fallback publication from completed AI analysis, including historical green workflow status. Workflow tests enforce failed-job publication without `continue-on-error`; existing Worker/security tests enforce permanent non-approval after a failed job.

`test:sandbox` executes real Docker isolation and pinned Codex 0.169.0 with synthetic Responses, first successfully and then with a simulated unsupported-capability HTTP 400. It checks exit 1, absent review output, safe engine/proxy envelopes and absence of the private provider message. No stderr/provider-body dumps are used, including test failures. This is protocol/containment evidence, not a claim about any production account, model entitlement or non-API-billed inference. Current official [SIWC constraints](https://developers.openai.com/siwc/token-sharing-open-source/preview-limitations) are retained; no undocumented endpoint is used.

Local validation: `npm run check` (140 tests, typecheck, lint, build), `npm run test:clean`, `npm run test:workers`, actionlint, Docker image build, `npm run test:sandbox`, `npm audit --audit-level=moderate` and Wrangler dry-run passed. See the [coordinated rollout](review-rollout.md#isolated-failure-diagnostics-and-workflow-conclusions). Source, Worker trust and caller pins must move together after normal approval. Deployment and fresh credential-only/advisory provider acceptance remain external gates; no production request was triggered by these tests.

## Workers.dev origin migration

The configured hosted origin, CLI and generated caller defaults, manifest and onboarding website now use `https://codexbot.appsolves.workers.dev`. Focused tests cover the real CLI default/self-hosted override/profile separation, exact OIDC audience, HMAC on the new origin, strict Origin/host rejection, nonredirecting retired aliases and inability to decrypt old-origin OAuth ciphertext under new AAD. The workerd/D1 test also exercises onboarding, Origin rejection and authenticated endpoint denial without a bearer. [Cutover/rollback](origin-migration.md) separates local evidence from pending deployment, GitHub App ping delivery, protected-default-branch credential diagnostic and fresh OAuth connection. No production change or inference is performed by these tests.
Expand Down
15 changes: 12 additions & 3 deletions src/core/diagnostics.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import { AppError } from './errors.js';
import { ExecutionError, executionFailure, type ExecutionFailure } from './execution.js';

export const REVIEW_STAGES = ['environment','prepare','context','credentials','analyzing','result','publish','release'] as const;
export const FAILURE_CODES = new Set(['missing_environment_credentials','ambiguous_environment_credentials','credential_owner_mismatch','missing_read_token',
Expand All @@ -25,10 +26,17 @@ github_actions_oidc_unavailable invalid_oidc_endpoint github_oidc_request_failed
export class BrokerError extends AppError {
constructor(public readonly operation: string, status: number, public readonly remoteCode: string | null) { super(`broker_${operation}_failed`, status); }
}
export function diagnostic(error: unknown, stage: string): { event: string; stage: string; code: string; status?: number; broker_code?: string } {
const result: { event: string; stage: string; code: string; status?: number; broker_code?: string } = {
export function diagnostic(error: unknown, stage: string): { event: string; stage: string; code: string; status?: number; broker_code?: string; execution?: ExecutionFailure; proxy_failure?: ExecutionFailure } {
const result: ReturnType<typeof diagnostic> = {
event: 'review_failed', stage: REVIEW_STAGES.includes(stage as typeof REVIEW_STAGES[number]) ? stage : 'environment', code: 'external_or_execution_failure'
};
if (error instanceof ExecutionError) {
result.code = error.code;
result.execution = executionFailure(error.detail)!;
const proxy = executionFailure(error.proxyFailure);
if (proxy) result.proxy_failure = proxy;
return result;
}
if (error instanceof BrokerError && OPERATIONS.has(error.operation)) {
result.code = error.code;
if (error.remoteCode && BROKER_CODES.has(error.remoteCode)) result.broker_code = error.remoteCode;
Expand All @@ -41,6 +49,7 @@ export function diagnostic(error: unknown, stage: string): { event: string; stag
cause = 'cause' in cause ? cause.cause : null;
}
}
if (error && typeof error === 'object' && 'status' in error && typeof error.status === 'number' && Number.isInteger(error.status) && error.status >= 400 && error.status <= 599) result.status = error.status;
// AppError's default 400 is a local validation status, not an upstream HTTP response.
if ((error instanceof BrokerError || !(error instanceof AppError) || error.code === 'github_oidc_request_failed') && error && typeof error === 'object' && 'status' in error && typeof error.status === 'number' && Number.isInteger(error.status) && error.status >= 400 && error.status <= 599) result.status = error.status;
return result;
}
87 changes: 87 additions & 0 deletions src/core/execution.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
import { AppError } from './errors.js';

// This is a projection, never a redaction of arbitrary process/provider text.
const COMPONENTS = new Set(['docker', 'proxy', 'provider', 'codex_process', 'codex_rpc', 'model', 'turn', 'engine']);
const CODES = new Set(`container_start_failed container_exit_failed container_timeout container_output_limit docker_unavailable
proxy_start_failed proxy_startup_failed proxy_route_denied proxy_body_too_large proxy_invalid_json proxy_unsupported_request_field
proxy_requires_private_streaming_request proxy_requires_explicit_input_array proxy_invalid_client_metadata proxy_invalid_tool_list
proxy_hosted_tools_denied proxy_hosted_tool_choice_denied proxy_invalid_tool_choice proxy_acceptance_input_budget
proxy_model_or_effort_change_denied proxy_request_limit proxy_concurrent_request_denied proxy_transport_failed
provider_http_failed provider_stream_failed provider_stream_content_type_invalid provider_stream_truncated provider_stream_limit
provider_frame_limit invalid_provider_event provider_event_type_missing provider_credential_echo provider_stream_incomplete
codex_process_failed codex_process_exited codex_rpc_message_too_large codex_rpc_rejected invalid_codex_protocol
configured_model_not_in_codex_catalog effort_not_supported_by_selected_model codex_invalid_model_catalog codex_invalid_model_cursor
codex_thread_id_missing codex_turn_failed codex_turn_interrupted codex_turn_timeout codex_output_not_json engine_failed`.split(/\s+/));
const PROVIDER_CODES = new Set(`subscription_sharing_user_not_eligible subscription_sharing_usage_limit_exceeded
subscription_sharing_usage_unavailable subscription_sharing_unsupported_capability subscription_sharing_route_not_supported
subscription_sharing_invalid_user chatpass_v2_scope_not_authorized chatpass_v2_invalid_authorization_context
subscription_sharing_user_unavailable invalid_api_key insufficient_quota rate_limit_exceeded model_not_found
unsupported_parameter unsupported_value invalid_request_error`.split(/\s+/));
const PROVIDER_PARAMS = new Set(['model', 'reasoning.effort', 'tools', 'tool_choice', 'input', 'text.format', 'store', 'stream',
'max_output_tokens', 'previous_response_id', 'temperature', 'prompt_cache_retention', 'metadata']);
const METHODS = new Set(['initialize', 'model/list', 'thread/start', 'turn/start']);
const SIGNALS = new Set(['SIGKILL', 'SIGTERM', 'SIGABRT', 'SIGSEGV', 'SIGINT']);
export interface ExecutionFailure {
component: string; code: string; exit_code?: number; signal?: string; rpc_code?: number; rpc_method?: string;
upstream_status?: number; provider_code?: string; provider_param?: string;
}
function object(value: unknown): Record<string, unknown> {
return value !== null && typeof value === 'object' && !Array.isArray(value) ? value as Record<string, unknown> : {};
}
export function executionFailure(value: unknown): ExecutionFailure | null {
const source = object(value);
const { component, code } = source;
if (typeof component !== 'string' || !COMPONENTS.has(component) || typeof code !== 'string' || !CODES.has(code)) return null;
const result: ExecutionFailure = { component, code };
if (Number.isInteger(source.exit_code) && Number(source.exit_code) >= 0 && Number(source.exit_code) <= 255) result.exit_code = Number(source.exit_code);
if (typeof source.signal === 'string' && SIGNALS.has(source.signal)) result.signal = source.signal;
if (typeof source.rpc_method === 'string' && METHODS.has(source.rpc_method)) result.rpc_method = source.rpc_method;
if (Number.isInteger(source.rpc_code) && Number(source.rpc_code) >= -32768 && Number(source.rpc_code) <= -32000) result.rpc_code = Number(source.rpc_code);
if (Number.isInteger(source.upstream_status) && Number(source.upstream_status) >= 400 && Number(source.upstream_status) <= 599) result.upstream_status = Number(source.upstream_status);
if (typeof source.provider_code === 'string' && PROVIDER_CODES.has(source.provider_code)) result.provider_code = source.provider_code;
if (typeof source.provider_param === 'string' && PROVIDER_PARAMS.has(source.provider_param)) result.provider_param = source.provider_param;
return result;
}
export class ExecutionError extends AppError {
readonly detail: ExecutionFailure;
proxyFailure?: ExecutionFailure;
constructor(detail: ExecutionFailure) {
const safe = executionFailure(detail) ?? { component: 'engine', code: 'engine_failed' };
super('isolated_review_failed'); this.detail = safe;
}
}
export function executionError(error: unknown, component: string, fallback: string): ExecutionError {
return error instanceof ExecutionError ? error : new ExecutionError({ component, code: error instanceof AppError && CODES.has(error.code) ? error.code : fallback });
}
export function executionEnvelope(detail: ExecutionFailure): string {
return JSON.stringify({ event: 'codexbot_execution_failure', ...executionFailure(detail) });
}
/** Bounded line reader for our own envelope. Arbitrary stderr is discarded, including oversized lines. */
export class ExecutionDiagnostics {
private line = '';
private dropping = false;
private bytes = 0;
last: ExecutionFailure | null = null;
write(chunk: string): void {
this.bytes += Buffer.byteLength(chunk);
if (this.bytes > 64_000) return;
for (const part of chunk.split(/(\n)/)) {
if (part === '\n') {
if (!this.dropping) {
try { const value = object(JSON.parse(this.line)); if (value.event === 'codexbot_execution_failure') this.last = executionFailure(value) ?? this.last; } catch { /* Untrusted stderr is never forwarded. */ }
}
this.line = ''; this.dropping = false;
} else if (!this.dropping) {
this.line += part;
if (this.line.length > 2048) { this.line = ''; this.dropping = true; }
}
}
}
}
/** Inspect only documented, exact error codes/parameter names, never message/detail text. */
export function providerFailure(value: unknown, upstreamStatus?: number): ExecutionFailure {
const source = object(value), response = object(source.response);
const error = object(source.error ?? response.error ?? source);
return executionFailure({ component: 'provider', code: upstreamStatus === undefined ? 'provider_stream_failed' : 'provider_http_failed',
upstream_status: upstreamStatus, provider_code: error.code, provider_param: error.param })!;
}
Loading