Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,11 @@
# Security and behavior changes

## Unreleased — inference diagnostics

- Preserve correlated per-request primary and cleanup errors, bounded HTTP/SSE metadata and allowlisted transport causes. A consumer closing after a completed SSE response is no longer mislabeled as a provider failure; successful retries clear stale failures.
- Distinguish fixed engine rejection reasons for incomplete/inconclusive results, invalid schema/head, quarantine and truncation. Incomplete reviews still fail CI and publish only a non-approving fallback.
- Add native pinned-Codex Docker regressions for completed-but-open HTTP streams and malformed structured output. No prompts, inference timeouts, context budgets, authorization or review/merge policy change.

## 0.1.0 — verified PNG input and large-PR context

- Verify PNG evidence separately from textual patches and supply immutable base/head images through official Codex localImage inputs. Require image-capable models and actual image transport; independently verify publication evidence. Image findings appear in the summary without invented inline locations. Unsupported formats and incomplete visual coverage remain non-approving.
Expand Down
11 changes: 11 additions & 0 deletions docs/review-rollout.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,3 +74,14 @@ After approval, drain existing reviews, deploy the merged Worker source with its
4. Report local/CI validation before deployment. Deploy the reviewed Worker with `TRUSTED_WORKFLOW=OWNER/codexbot/.github/workflows/review.yml@THAT_SAME_SHA`, preserving existing bindings, OAuth gate, secrets and domain settings. Merge the caller PR using normal protection checks. There is no atomic transaction across Cloudflare and GitHub: during the short mismatch window requests fail closed. Do not relax either SHA check to hide it.
5. Read back both actual pins and compare them byte-for-byte. Run the credential-only workflow check under the owning actor/installation Environment. Then request a new advisory/no-merge review for end-to-end broker/provider acceptance. Old Actions review attempts are not rerun to borrow a new credential/pin; use a new authorized comment.
6. Inspect the same progress comment, workflow link, sanitized failures and native review. Provider inference remains a distinct live test with the owner's authorization/allowance, never an API-key fallback. Roll back both pins together on failure; keep additive DB columns. A backend/caller protected-branch or deployment-permission block is a release blocker, not permission to bypass safeguards.


## Correlated inference diagnostics

The credential proxy emits an allowlisted `codexbot_proxy_request` record at request start and finish, with an ordinal (1–101), separate primary/cleanup errors, observed HTTP status/category/body presence, bounded elapsed time/byte/frame counts, SSE terminal state and owned abort origin. No provider IDs, arbitrary headers, exception messages, prompts, code or response bodies enter diagnostics. Diagnostic counts are capped; inference resource budgets are unchanged.

Only the latest request is authoritative. A started or successful later request clears an older recovered error; late completion of an older request cannot replace it. On failure, the Docker host forwards at most four sanitized failed-request records plus the latest request state to runner logs. It exposes the final safe request state as `proxy_request`, plus independent `proxy_failure` and `proxy_cleanup` when present. Startup failures keep the existing process-level envelope. An HTTP consumer closing after consuming `response.completed` is not itself a provider outage. Premature EOF, stream/frame limits, upstream events/read errors, deadline aborts, downstream disconnections, internal writes and cleanup remain separately diagnosed.

Engine `rejection_reason` is a fixed enum: `context_incomplete`, `schema_invalid`, `head_mismatch`, `quarantined`, `output_truncated`, `model_incomplete`, `model_inconclusive`, or `output_invalid`. Malformed JSON retains `codex_output_not_json`. These fields explain the existing failure decision; they never authorize approval or merging. Original model text remains private. The historical two production failures cannot be retrospectively classified from their old generic diagnostics.

The native Docker harness holds valid completed Responses streams open for clean/incomplete fixtures, uses the actual pinned Codex binary and exercises schema/JSON-invalid output. All provider responses are synthetic. The diagnostic rollout needs no D1 migration or new live inference; use the existing closed-merge-gate pin coordination and restore the gate after actual Worker/caller pins agree. Do not initiate a repository review merely to test diagnostics.
9 changes: 5 additions & 4 deletions src/core/diagnostics.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import { AppError } from './errors.js';
import { ExecutionError, executionFailure, type ExecutionFailure } from './execution.js';
import { ExecutionError, executionFailure, proxyRequestDiagnostic, type ProxyRequestDiagnostic, type ExecutionFailure } from './execution.js';

export const REVIEW_STAGES = ['environment','prepare','context','credentials','analyzing','result','publish','release'] as const;
export const FAILURE_CODES = new Set(['missing_environment_credentials','ambiguous_environment_credentials','credential_owner_mismatch','missing_read_token',
Expand Down Expand Up @@ -30,15 +30,16 @@ github_actions_oidc_unavailable invalid_oidc_endpoint github_oidc_request_failed
export class BrokerError extends AppError {
constructor(public readonly operation: string, status: number, public readonly remoteCode: string | null) { super(`broker_${operation}_failed`, status); }
}
export function diagnostic(error: unknown, stage: string): { event: string; stage: string; code: string; status?: number; broker_code?: string; execution?: ExecutionFailure; proxy_failure?: ExecutionFailure } {
export function diagnostic(error: unknown, stage: string): { event: string; stage: string; code: string; status?: number; broker_code?: string; execution?: ExecutionFailure; proxy_failure?: ExecutionFailure; proxy_cleanup?: ExecutionFailure; proxy_request?: Omit<ProxyRequestDiagnostic,'primary'|'cleanup'> } {
const result: ReturnType<typeof diagnostic> = {
event: 'review_failed', stage: REVIEW_STAGES.includes(stage as typeof REVIEW_STAGES[number]) ? stage : 'environment', code: 'external_or_execution_failure'
};
if (error instanceof ExecutionError) {
result.code = error.code;
result.execution = executionFailure(error.detail)!;
const proxy = executionFailure(error.proxyFailure);
if (proxy) result.proxy_failure = proxy;
const request=proxyRequestDiagnostic(error.proxyRequest);
if(request){const {primary,cleanup,...state}=request;result.proxy_request=state;if(primary)result.proxy_failure=primary;if(cleanup)result.proxy_cleanup=cleanup;}
else {const proxy=executionFailure(error.proxyFailure),cleanup=executionFailure(error.proxyCleanup);if(proxy)result.proxy_failure=proxy;if(cleanup)result.proxy_cleanup=cleanup;}
return result;
}
if (error instanceof BrokerError && OPERATIONS.has(error.operation)) {
Expand Down
55 changes: 48 additions & 7 deletions src/core/execution.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ proxy_start_failed proxy_startup_failed proxy_route_denied proxy_body_too_large
proxy_requires_private_streaming_request proxy_requires_explicit_input_array proxy_invalid_client_metadata proxy_invalid_tool_list
proxy_hosted_tools_denied proxy_hosted_tool_choice_denied proxy_invalid_tool_choice proxy_acceptance_input_budget
proxy_model_or_effort_change_denied proxy_request_limit proxy_concurrent_request_denied proxy_transport_failed
provider_http_failed provider_stream_failed provider_stream_content_type_invalid provider_stream_truncated provider_stream_limit
proxy_pipeline_failed proxy_client_disconnected proxy_deadline_exceeded proxy_stream_cleanup_failed provider_stream_read_failed provider_http_failed provider_stream_failed provider_stream_content_type_invalid provider_stream_truncated provider_stream_limit
provider_frame_limit invalid_provider_event provider_event_type_missing provider_credential_echo provider_stream_incomplete
codex_process_failed codex_process_exited codex_rpc_message_too_large codex_rpc_rejected invalid_codex_protocol
image_context_invalid image_context_limit configured_model_does_not_support_images proxy_unsupported_media_input proxy_image_source_denied proxy_image_inputs_missing configured_model_not_in_codex_catalog effort_not_supported_by_selected_model codex_invalid_model_catalog codex_invalid_model_cursor
Expand All @@ -22,13 +22,48 @@ const PROVIDER_PARAMS = new Set(['model', 'reasoning.effort', 'tools', 'tool_cho
const METHODS = new Set(['initialize', 'model/list', 'thread/start', 'turn/start']);
const SIGNALS = new Set(['SIGKILL', 'SIGTERM', 'SIGABRT', 'SIGSEGV', 'SIGINT']);
const CONTENT_TYPES = new Set(['event-stream', 'json', 'html', 'missing', 'other']);
export interface ExecutionFailure {
export type TerminalEvent = 'none' | 'completed' | 'failed' | 'incomplete' | 'error';
const ORIGINS = new Set(['validation','transport','upstream_event','upstream_read','framing','limit','deadline','downstream','pipeline','cleanup']);
const TRANSPORT_CODES = new Set(['ECONNRESET','ECONNREFUSED','ENOTFOUND','EPIPE','ETIMEDOUT','UND_ERR_SOCKET','UND_ERR_BODY_TIMEOUT','UND_ERR_HEADERS_TIMEOUT','ABORT_ERR','TIMEOUT_ERR','ERR_STREAM_PREMATURE_CLOSE','ERR_STREAM_DESTROYED','ERR_STREAM_WRITE_AFTER_END']);
const REJECTION_REASONS = ['context_incomplete','schema_invalid','head_mismatch','quarantined','output_truncated','model_incomplete','model_inconclusive','output_invalid'] as const;
export type EngineRejectionReason = typeof REJECTION_REASONS[number];
const REJECTIONS = new Set<string>(REJECTION_REASONS);
interface RequestMetadata {
request_index?: number; elapsed_ms?: number; stream_bytes?: number; stream_frames?: number; terminal_event?: TerminalEvent;
upstream_status?: number; content_type_category?: string; body_present?: boolean; abort_origin?: 'none' | 'deadline' | 'downstream' | 'pipeline';
}
export interface ProxyRequestDiagnostic extends RequestMetadata { request_index: number; phase: 'started' | 'finished'; primary: ExecutionFailure | null; cleanup: ExecutionFailure | null }
export interface ExecutionFailure extends RequestMetadata {
component: string; code: string; exit_code?: number; signal?: string; rpc_code?: number; rpc_method?: string;
upstream_status?: number; content_type_category?: string; body_present?: boolean; provider_code?: string; provider_param?: string;
provider_code?: string; provider_param?: string; origin?: string; transport_code?: string; rejection_reason?: EngineRejectionReason;
}
function object(value: unknown): Record<string, unknown> {
return value !== null && typeof value === 'object' && !Array.isArray(value) ? value as Record<string, unknown> : {};
}
function projectRequestMetadata(source: Record<string, unknown>, result: RequestMetadata): void {
for (const [key, maximum] of [['request_index',101],['elapsed_ms',1_320_000],['stream_bytes',20_000_001],['stream_frames',1_000_000],['upstream_status',599]] as const) {
const value=source[key]; if(typeof value==='number'&&Number.isInteger(value)&&value>=(key==='request_index'?1:key==='upstream_status'?200:0)&&value<=maximum)result[key]=value;
}
if(typeof source.content_type_category==='string'&&CONTENT_TYPES.has(source.content_type_category))result.content_type_category=source.content_type_category;
if(typeof source.body_present==='boolean')result.body_present=source.body_present;
if(typeof source.terminal_event==='string'&&['none','completed','failed','incomplete','error'].includes(source.terminal_event))result.terminal_event=source.terminal_event as TerminalEvent;
if(typeof source.abort_origin==='string'&&['none','deadline','downstream','pipeline'].includes(source.abort_origin))result.abort_origin=source.abort_origin as NonNullable<RequestMetadata['abort_origin']>;
}
/** Only exact machine codes leave the exception chain; never messages or arbitrary names. */
export function transportCode(error: unknown): string | undefined {
for(let depth=0;depth<4;depth++) {const value=object(error);if(typeof value.code==='string'&&TRANSPORT_CODES.has(value.code))return value.code; if(value.name==='AbortError')return 'ABORT_ERR';if(value.name==='TimeoutError')return 'TIMEOUT_ERR';error=value.cause;}
}
export function proxyRequestDiagnostic(value: unknown): ProxyRequestDiagnostic | null {
const source=object(value);if(source.phase!=='started'&&source.phase!=='finished')return null;
const result:ProxyRequestDiagnostic={request_index:0,phase:source.phase,primary:executionFailure(source.primary),cleanup:executionFailure(source.cleanup)};
projectRequestMetadata(source,result);if(!result.request_index)return null;
if((source.primary!==null&&!result.primary)||(source.cleanup!==null&&!result.cleanup))return null;
if([result.primary,result.cleanup].some(detail=>detail&&!['proxy','provider'].includes(detail.component)))return null;
if([result.primary,result.cleanup].some(detail=>detail?.request_index!==undefined&&detail.request_index!==result.request_index))return null;
if(result.phase==='started'&&(result.primary||result.cleanup))return null;
return result;
}
export function proxyRequestEnvelope(value: ProxyRequestDiagnostic): string {return JSON.stringify({event:'codexbot_proxy_request',...proxyRequestDiagnostic(value)});}
export function executionFailure(value: unknown): ExecutionFailure | null {
const source = object(value);
const { component, code } = source;
Expand All @@ -38,10 +73,11 @@ export function executionFailure(value: unknown): ExecutionFailure | null {
if (typeof source.signal === 'string' && SIGNALS.has(source.signal)) result.signal = source.signal;
if (typeof source.rpc_method === 'string' && METHODS.has(source.rpc_method)) result.rpc_method = source.rpc_method;
if (Number.isInteger(source.rpc_code) && Number(source.rpc_code) >= -32768 && Number(source.rpc_code) <= -32000) result.rpc_code = Number(source.rpc_code);
if(typeof source.rejection_reason==='string'&&component==='engine'&&REJECTIONS.has(source.rejection_reason))result.rejection_reason=source.rejection_reason as EngineRejectionReason;
if((component==='proxy'||component==='provider')&&typeof source.origin==='string'&&ORIGINS.has(source.origin))result.origin=source.origin;
if((component==='proxy'||component==='provider')&&typeof source.transport_code==='string'&&TRANSPORT_CODES.has(source.transport_code))result.transport_code=source.transport_code;
if(component==='proxy'||component==='provider')projectRequestMetadata(source,result);
if (component === 'provider') {
if (Number.isInteger(source.upstream_status) && Number(source.upstream_status) >= 200 && Number(source.upstream_status) <= 599) result.upstream_status = Number(source.upstream_status);
if (typeof source.content_type_category === 'string' && CONTENT_TYPES.has(source.content_type_category)) result.content_type_category = source.content_type_category;
if (typeof source.body_present === 'boolean') result.body_present = source.body_present;
if (typeof source.provider_code === 'string' && PROVIDER_CODES.has(source.provider_code)) result.provider_code = source.provider_code;
if (typeof source.provider_param === 'string' && PROVIDER_PARAMS.has(source.provider_param)) result.provider_param = source.provider_param;
}
Expand All @@ -50,6 +86,8 @@ export function executionFailure(value: unknown): ExecutionFailure | null {
export class ExecutionError extends AppError {
readonly detail: ExecutionFailure;
proxyFailure?: ExecutionFailure;
proxyCleanup?: ExecutionFailure;
proxyRequest?: ProxyRequestDiagnostic;
constructor(detail: ExecutionFailure) {
const safe = executionFailure(detail) ?? { component: 'engine', code: 'engine_failed' };
super('isolated_review_failed'); this.detail = safe;
Expand All @@ -67,13 +105,16 @@ export class ExecutionDiagnostics {
private dropping = false;
private bytes = 0;
last: ExecutionFailure | null = null;
request: ProxyRequestDiagnostic | null = null;
history: ProxyRequestDiagnostic[] = [];
write(chunk: string): void {
this.bytes += Buffer.byteLength(chunk);
if (this.bytes > 64_000) return;
for (const part of chunk.split(/(\n)/)) {
if (part === '\n') {
if (!this.dropping) {
try { const value = object(JSON.parse(this.line)); if (value.event === 'codexbot_execution_failure') this.last = executionFailure(value) ?? this.last; } catch { /* Untrusted stderr is never forwarded. */ }
try { const value = object(JSON.parse(this.line)); if (value.event === 'codexbot_execution_failure') this.last = executionFailure(value) ?? this.last;
if(value.event==='codexbot_proxy_request'){const request=proxyRequestDiagnostic(value);if(request){if(request.phase==='finished'&&(request.primary||request.cleanup))this.history=[...this.history.filter(x=>x.request_index!==request.request_index),request].slice(-4);if(request.request_index>=(this.request?.request_index??0)){this.request=request;this.last=request.primary;}}} } catch { /* Untrusted stderr is never forwarded. */ }
}
this.line = ''; this.dropping = false;
} else if (!this.dropping) {
Expand Down
14 changes: 9 additions & 5 deletions src/core/review.ts
Original file line number Diff line number Diff line change
Expand Up @@ -46,14 +46,17 @@ const validate = new Ajv().compile<Review>({ ...REVIEW_SCHEMA, properties: { ...
output_truncated: { type: 'boolean' }, quarantined: { type: 'boolean' } } });

/** Applied before transport, persistence and publication. No suspicious source text escapes. */
export function normalizeReview(value: unknown, head: string): Review {
export type ReviewRejectionReason = 'schema_invalid' | 'head_mismatch' | 'quarantined' | 'output_truncated';
export function normalizeReview(value: unknown, head: string, onReject: (reason: ReviewRejectionReason) => void = () => {}): Review {
const fallback = failedReview(head);
if (scanSecrets(value).length) return { ...fallback, quarantined: true };
if (!validate(value) || value.head_sha !== head) return fallback;
if (scanSecrets(value).length) {onReject('quarantined');return { ...fallback, quarantined: true };}
if (!validate(value)) {onReject('schema_invalid');return fallback;}
if (value.head_sha !== head) {onReject('head_mismatch');return fallback;}
const review = structuredClone(value);
const size = () => Buffer.byteLength(JSON.stringify(review));
if (size() > MAX_RAW_REVIEW_BYTES) return { ...fallback, output_truncated: true };
if (size() > MAX_RAW_REVIEW_BYTES) {onReject('output_truncated');return { ...fallback, output_truncated: true };}
if (size() > MAX_REVIEW_BYTES) {
onReject('output_truncated');
// Retain severity, confidence and locations. Any information loss disables approval.
review.complete = false; review.output_truncated = true;
const shorten = (text: string, bytes: number) => {
Expand All @@ -65,7 +68,8 @@ export function normalizeReview(value: unknown, head: string): Review {
}
if (review.result === 'clean') review.result = 'inconclusive';
}
return size() <= MAX_REVIEW_BYTES ? review : { ...fallback, output_truncated: true };
if(size()<=MAX_REVIEW_BYTES)return review;
onReject('output_truncated');return { ...fallback, output_truncated: true };
}

export function safePath(path: string): boolean {
Expand Down
Loading