From bf7b317bcbd3d38cd50c30d8ec7945ce8312690e Mon Sep 17 00:00:00 2001 From: AppSolves Date: Sat, 14 Nov 2026 19:22:56 +0100 Subject: [PATCH] fix: classify rejected provider responses with safe metadata --- CHANGELOG.md | 2 ++ docs/acceptance.md | 4 +++ docs/validation.md | 10 ++++++++ src/core/execution.ts | 13 +++++++--- src/runner/proxy.ts | 48 ++++++++++++++++++++++------------- tests/execution.test.ts | 18 +++++++++++-- tests/proxy.test.ts | 56 +++++++++++++++++++++++++++++++++++++++++ tests/sandbox-check.mjs | 50 ++++++++++++++++++++---------------- 8 files changed, 156 insertions(+), 45 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 82ae029..c3f1e6a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,8 @@ ## Unreleased — security remediation pass +- Classify rejected Responses replies with their actual upstream status (including 2xx), fixed Content-Type category and body presence. Extract only allowlisted codes/parameters from bounded JSON errors; keep non-SSE success responses fail-closed and provider metadata confined to the credential-proxy boundary. + - Failed inference now fails the review job even after recording a safe COMMENT fallback. Preserve separate protected publication, distinguish fallback publication from completed AI review, and carry only allowlisted Docker/Codex/RPC/proxy diagnostics. Local AppError statuses are not presented as provider HTTP responses. Runner changes require a coordinated immutable source/Worker/caller pin upgrade; no credential or database migration. - Validate broker default-branch identity using the authoritative repository API, not the minimal repository embedded in Actions runs. Preserve signed OIDC and all run/App dispatch checks across prepare, failure, publication and cleanup operations. diff --git a/docs/acceptance.md b/docs/acceptance.md index 76c273e..6cc1134 100644 --- a/docs/acceptance.md +++ b/docs/acceptance.md @@ -57,6 +57,10 @@ Enable MERGE_DEPLOYMENT_AUTHORIZED only after recording the applicable rows as p First use official local SIWC through the normal CLI. Hosted/ephemeral Actions eligibility is an external prerequisite: without it the OAuth launch gate is **BLOCKED**. Never reinterpret native CLI auth.json. Model discovery is bundled catalog discovery, not subscription entitlement. +For a model-access investigation, the [official account-specific catalog](https://developers.openai.com/siwc/token-sharing-open-source/models-and-inference) is `GET https://api.openai.com/v1/models` using the **same owning account/workspace OAuth access token** as inference. Its `models` array contains `slug` and `visibility`; compare the selected configured model (or Codex's bundled default) locally against those slugs. Codex `model/list` instead returns its own `data` catalog and supported efforts, including with an empty credential home in our pinned-binary test. That catalog does not establish account membership, entitlement, capability support or completed inference. Do not substitute the API-key catalog shape/credential or silently pick another model. A read-only catalog comparison performs no inference but still requires an authorized owner credential context; do not export central ciphertext or Environment secrets to obtain it, print the catalog/user data, or reconnect solely because of a response-format error. This check has not been run against the production OAuth session. + +`provider_stream_content_type_invalid` at source `0e937476323bbe35e3c33dc3a8572e62d1e544e5` proves a 2xx response with a missing body or a failing SSE Content-Type check. It does not distinguish JSON, HTML or absent headers, and proves no specific model/account/provider failure. New diagnostics retain only `upstream_status`, `content_type_category` (`event-stream`, `json`, `html`, `missing`, `other`), `body_present`, and optional allowlisted provider codes/parameters. A `json` category is a header classification, not proof of a valid error envelope; missing/other headers can also carry a bounded JSON error. MIME parameters and arbitrary provider text are discarded. A zero-length stream can have `body_present:true`; this is presence, not a content/completeness claim. After the coordinated immutable-pin rollout, a fresh authorized advisory test is needed to establish the production response shape. Keep the exact public Responses endpoint, `stream:true`, `store:false` and input-array constraints from the [SIWC limitations](https://developers.openai.com/siwc/token-sharing-open-source/preview-limitations); a 2xx JSON response is never accepted as completed review output. + The optional Linux-only harness accepts an operator-controlled owner-only JSON credential file outside the repository: a validated official SIWC OAuthSession or `{ "api_key": "..." }`. This is an acceptance input, not a new mandatory onboarding export. Provision it only through your own official OAuth/vault tooling; never paste it into chat, PRs or shell arguments. The harness does not export sessions from other users or the hosted service. It holds an exclusive lock, durably persists refresh rotation before use and prints only a fixed completion message. ```sh diff --git a/docs/validation.md b/docs/validation.md index 34bcb55..ed0282d 100644 --- a/docs/validation.md +++ b/docs/validation.md @@ -1,5 +1,15 @@ # Validation record and release gates +## Provider response-format diagnostics — 2026-11-14 + +Inspected `inferenceProxy` at pinned source `0e937476323bbe35e3c33dc3a8572e62d1e544e5`: the observed format error occurs after `upstream.ok`, when the response body is absent or the SSE Content-Type guard fails. The original diagnostic contains no actual status, header category or body-presence evidence; a specific authentication, entitlement, model, HTML or JSON cause is unproven. The observed production workflow correctly failed analysis while its separate non-approving publisher succeeded. + +The new response projection retains actual 200–599 HTTP status, a five-value MIME category and a boolean body-presence field. Rejected responses are inspected for at most 8 KiB of JSON, projecting only existing exact code/parameter allowlists. Unknown text, MIME parameters, credentials, user data, headers and response bodies are never emitted. Provider fields are removed from non-provider diagnostics, and the Docker boundary still rejects provider attestations from the unprivileged reviewer. Non-SSE 2xx responses remain rejected; an SSE-looking MIME prefix is no longer sufficient without the exact media type. No session, schema, authorization, endpoint, credential or merge-policy migration is involved. + +Local verification passed: `npm run check` (143 tests, typecheck, lint, build), `npm run test:clean`, real workerd/D1, targeted execution/security regressions (21 tests), Actionlint, Docker build plus synthetic Codex integration, `npm audit --audit-level=moderate`, strict HIGH/CRITICAL image scan and Wrangler dry-run. No deployment, reconnect, paid inference or production review was performed. + +`proxy.test.ts` covers successful SSE (including case/parameters), 2xx JSON success/error envelopes, 204/missing bodies, HTML, missing/misleading headers, oversized/malformed/unknown JSON and read/cancellation errors. `execution.test.ts` covers metadata validation, envelope/outer-log round trips and provenance. `test:sandbox` uses pinned Codex 0.169.0 and isolated Docker with synthetic successful SSE, HTTP 400 JSON and HTTP 200 JSON; both failure cases require exit 1 and absent review output, with safe proxy metadata. No real provider request is made. The pinned credential-free model-catalog test and official documentation show why bundled `model/list` is not account entitlement; the production account-specific catalog and completed inference remain unverified. See [the owner-only acceptance steps](acceptance.md#optional-real-openai-protocol-acceptance). + ## Isolated review failure and misleading success — 2026-11-14 Verified root cause of the green workflow: `reviewJob` caught inference failure, saved the incomplete fallback and returned successfully. It now always rethrows the primary failure, even when reporting fails; the existing separately protected publisher remains eligible and can only publish a non-approving report. Original container/Codex/provider diagnostics were discarded, so the production inference failure's specific cause remains unknown. Its old `400` was a local AppError default, not evidence of upstream HTTP 400. diff --git a/src/core/execution.ts b/src/core/execution.ts index e4b3476..2fcc7ef 100644 --- a/src/core/execution.ts +++ b/src/core/execution.ts @@ -21,9 +21,10 @@ const PROVIDER_PARAMS = new Set(['model', 'reasoning.effort', 'tools', 'tool_cho 'max_output_tokens', 'previous_response_id', 'temperature', 'prompt_cache_retention', 'metadata']); const METHODS = new Set(['initialize', 'model/list', 'thread/start', 'turn/start']); const SIGNALS = new Set(['SIGKILL', 'SIGTERM', 'SIGABRT', 'SIGSEGV', 'SIGINT']); +const CONTENT_TYPES = new Set(['event-stream', 'json', 'html', 'missing', 'other']); export interface ExecutionFailure { component: string; code: string; exit_code?: number; signal?: string; rpc_code?: number; rpc_method?: string; - upstream_status?: number; provider_code?: string; provider_param?: string; + upstream_status?: number; content_type_category?: string; body_present?: boolean; provider_code?: string; provider_param?: string; } function object(value: unknown): Record { return value !== null && typeof value === 'object' && !Array.isArray(value) ? value as Record : {}; @@ -37,9 +38,13 @@ export function executionFailure(value: unknown): ExecutionFailure | null { if (typeof source.signal === 'string' && SIGNALS.has(source.signal)) result.signal = source.signal; if (typeof source.rpc_method === 'string' && METHODS.has(source.rpc_method)) result.rpc_method = source.rpc_method; if (Number.isInteger(source.rpc_code) && Number(source.rpc_code) >= -32768 && Number(source.rpc_code) <= -32000) result.rpc_code = Number(source.rpc_code); - if (Number.isInteger(source.upstream_status) && Number(source.upstream_status) >= 400 && Number(source.upstream_status) <= 599) result.upstream_status = Number(source.upstream_status); - if (typeof source.provider_code === 'string' && PROVIDER_CODES.has(source.provider_code)) result.provider_code = source.provider_code; - if (typeof source.provider_param === 'string' && PROVIDER_PARAMS.has(source.provider_param)) result.provider_param = source.provider_param; + if (component === 'provider') { + if (Number.isInteger(source.upstream_status) && Number(source.upstream_status) >= 200 && Number(source.upstream_status) <= 599) result.upstream_status = Number(source.upstream_status); + if (typeof source.content_type_category === 'string' && CONTENT_TYPES.has(source.content_type_category)) result.content_type_category = source.content_type_category; + if (typeof source.body_present === 'boolean') result.body_present = source.body_present; + if (typeof source.provider_code === 'string' && PROVIDER_CODES.has(source.provider_code)) result.provider_code = source.provider_code; + if (typeof source.provider_param === 'string' && PROVIDER_PARAMS.has(source.provider_param)) result.provider_param = source.provider_param; + } return result; } export class ExecutionError extends AppError { diff --git a/src/runner/proxy.ts b/src/runner/proxy.ts index 29d0261..647dc17 100644 --- a/src/runner/proxy.ts +++ b/src/runner/proxy.ts @@ -15,6 +15,31 @@ function checkTools(tools: unknown, depth = 0): void { if (tool.type === 'namespace') checkTools(tool.tools, depth + 1); } } +function contentTypeCategory(header: string | null): string { + const type = header?.split(';', 1)[0]?.trim().toLowerCase(); + if (!type) return 'missing'; + if (type === 'text/event-stream') return 'event-stream'; + if (/^application\/(?:json|[a-z0-9!#$&^_.+-]+\+json)$/.test(type)) return 'json'; + if (type === 'text/html' || type === 'application/xhtml+xml') return 'html'; + return 'other'; +} +async function responseFailure(upstream: Response): Promise { + // Inspect at most 8 KiB; only the existing exact code/parameter projection leaves this function. + let errorBody: unknown; + const reader = upstream.body?.getReader(); + if (reader) { + try { + const limited: Uint8Array[] = []; let size = 0; + for (;;) { const chunk = await reader.read(); if (chunk.done) break; size += chunk.value.length; if (size > 8192) break; limited.push(chunk.value); } + if (size <= 8192) errorBody = JSON.parse(Buffer.concat(limited).toString('utf8')) as unknown; + } catch { /* Non-JSON, oversized or failed reads still retain the observed HTTP metadata. */ } + finally { + try { await reader.cancel(); } catch { /* Cancellation failure must not erase the original response diagnosis. */ } + finally { reader.releaseLock(); } + } + } + return providerFailure(errorBody, upstream.status); +} export async function inferenceProxy(request: IncomingMessage, response: ServerResponse, token: string, fetcher: typeof fetch = fetch, enforceSelection: (body: Record) => void = () => {}, report: (failure: ExecutionFailure) => void = () => {}): Promise { const disconnected = new AbortController(); @@ -53,27 +78,16 @@ export async function inferenceProxy(request: IncomingMessage, response: ServerR ...(typeof request.headers['originator'] === 'string' && request.headers['originator'] === 'CodexBot' ? { originator: 'CodexBot' } : {}) }, body: JSON.stringify(body), redirect: 'error', signal: AbortSignal.any([AbortSignal.timeout(20 * 60_000), disconnected.signal]) }); - if (!upstream.ok) { - // Read a small JSON error envelope only to project exact known codes. Never retain/forward its text. - let errorBody: unknown; - const reader = upstream.body?.getReader(); - if (reader) { - try { - const limited: Uint8Array[] = []; let size = 0; - for (;;) { const chunk = await reader.read(); if (chunk.done) break; size += chunk.value.length; if (size > 8192) break; limited.push(chunk.value); } - if (size <= 8192) errorBody = JSON.parse(Buffer.concat(limited).toString('utf8')) as unknown; - } catch { /* Unknown/non-JSON errors are represented by their actual HTTP status alone. */ } - finally { try { await reader.cancel(); } finally { reader.releaseLock(); } } - } - report(providerFailure(errorBody, upstream.status)); - response.writeHead(upstream.status, { 'Content-Type': 'application/json', 'Cache-Control': 'no-store' }); - response.end('{"error":{"message":"OpenAI inference unavailable; check authentication, entitlement or usage allowance.","type":"codexbot_inference_error"}}'); + const metadata = { upstream_status: upstream.status, content_type_category: contentTypeCategory(upstream.headers.get('content-type')), body_present: upstream.body !== null }; + if (!upstream.ok || !upstream.body || metadata.content_type_category !== 'event-stream') { + report({ ...await responseFailure(upstream), ...metadata, code: upstream.ok ? 'provider_stream_content_type_invalid' : 'provider_http_failed' }); + response.writeHead(upstream.ok ? 403 : upstream.status, { 'Content-Type': 'application/json', 'Cache-Control': 'no-store' }); + response.end(upstream.ok ? '{"error":"inference_proxy_denied_or_unavailable"}' : '{"error":{"message":"OpenAI inference unavailable; check authentication, entitlement or usage allowance.","type":"codexbot_inference_error"}}'); return; } stage = 'stream'; - requireCondition(upstream.body && upstream.headers.get('content-type')?.toLowerCase().startsWith('text/event-stream'), 'provider_stream_content_type_invalid'); response.writeHead(upstream.status, { 'Content-Type': 'text/event-stream', 'Cache-Control': 'no-store' }); - await pipeline(Readable.from(safeEvents(upstream.body, token, report)), response); + await pipeline(Readable.from(safeEvents(upstream.body, token, failure => report({ ...failure, ...metadata }))), response); } catch (error) { report(executionError(error, stage === 'stream' ? 'provider' : 'proxy', stage === 'transport' ? 'proxy_transport_failed' : stage === 'stream' ? 'provider_stream_failed' : 'proxy_invalid_json').detail); if (!response.headersSent) response.writeHead(403, { 'Content-Type': 'application/json' }); response.end('{"error":"inference_proxy_denied_or_unavailable"}'); diff --git a/tests/execution.test.ts b/tests/execution.test.ts index cc3cb69..82f4ea9 100644 --- a/tests/execution.test.ts +++ b/tests/execution.test.ts @@ -40,6 +40,20 @@ await test('execution projection discards arbitrary fields/text and never labels assert.deepEqual(diagnostic(error, 'analyzing'), { event: 'review_failed', stage: 'analyzing', code: 'isolated_review_failed', execution: error.detail, proxy_failure: safe }); assert.ok(!JSON.stringify(parser.last).includes(sensitive)); }); +await test('provider response metadata is exact, bounded and survives the envelope and outer diagnostic unchanged', () => { + for (const category of ['event-stream','json','html','missing','other']) for (const present of [true, false]) { + const detail = { component: 'provider', code: 'provider_stream_content_type_invalid', upstream_status: 200, content_type_category: category, body_present: present }; + const parser = new ExecutionDiagnostics(); parser.write(executionEnvelope({ ...detail }) + '\n'); + assert.deepEqual(parser.last, detail); + const error = new ExecutionError({ component: 'turn', code: 'codex_turn_failed' }); error.proxyFailure = detail; + assert.deepEqual(diagnostic(error, 'analyzing').proxy_failure, detail); + } + assert.deepEqual(executionFailure({ component: 'provider', code: 'provider_stream_content_type_invalid', upstream_status: 600, + content_type_category: sensitive, body_present: 'true', headers: sensitive, body: sensitive }), { component: 'provider', code: 'provider_stream_content_type_invalid' }); + assert.deepEqual(executionFailure({ component: 'provider', code: 'provider_stream_content_type_invalid', upstream_status: 199, + content_type_category: ['json'], body_present: 1 }), { component: 'provider', code: 'provider_stream_content_type_invalid' }); + assert.deepEqual(executionFailure({ component: 'turn', code: 'codex_turn_failed', upstream_status: 200, content_type_category: 'json', body_present: true }), { component: 'turn', code: 'codex_turn_failed' }); +}); await test('Docker diagnostics distinguish startup, exit, output bounds, and sanitized engine failure after stderr drains', async () => { for (const [exit, code] of [[125, 'container_start_failed'], [126, 'container_start_failed'], [127, 'container_start_failed'], [137, 'container_exit_failed'], [0, 'container_output_limit'], [1, 'codex_turn_failed'], [2, 'container_exit_failed']] as const) { const spawn = mock.method(childProcess, 'spawn', () => { @@ -47,14 +61,14 @@ await test('Docker diagnostics distinguish startup, exit, output bounds, and san queueMicrotask(() => { process.stderr.write(sensitive + '\n'); process.emit('exit', exit, null); - if (exit === 1) process.stderr.write(executionEnvelope({ component: 'turn', code, upstream_status: 401, provider_code: 'invalid_api_key' }) + '\n'); + if (exit === 1) process.stderr.write(JSON.stringify({ event: 'codexbot_execution_failure', component: 'turn', code, upstream_status: 200, provider_code: 'invalid_api_key', content_type_category: 'json', body_present: true }) + '\n'); if (exit === 2) process.stderr.write(executionEnvelope({ component: 'provider', code: 'provider_http_failed', upstream_status: 401 }) + '\n'); if (exit === 0) process.stdout.write('x'.repeat(101)); process.stdout.end(); process.stderr.end(); process.emit('close', exit, null); }); return process; }); syncBuiltinESMExports(); - try { await assert.rejects(docker(['run'], 100), error => { assert.ok(error instanceof ExecutionError); assert.equal(error.detail.code, code); assert.equal(error.detail.upstream_status, undefined); assert.equal(error.detail.provider_code, undefined); assert.ok(!JSON.stringify(diagnostic(error, 'analyzing')).includes(sensitive)); return true; }); } + try { await assert.rejects(docker(['run'], 100), error => { assert.ok(error instanceof ExecutionError); assert.equal(error.detail.code, code); assert.equal(error.detail.upstream_status, undefined); assert.equal(error.detail.provider_code, undefined); assert.equal(error.detail.content_type_category, undefined); assert.equal(error.detail.body_present, undefined); assert.ok(!JSON.stringify(diagnostic(error, 'analyzing')).includes(sensitive)); return true; }); } finally { spawn.mock.restore(); syncBuiltinESMExports(); } } }); diff --git a/tests/proxy.test.ts b/tests/proxy.test.ts index 8f921cc..d49371b 100644 --- a/tests/proxy.test.ts +++ b/tests/proxy.test.ts @@ -50,6 +50,7 @@ await test('proxy diagnostics retain only proven HTTP status and exact provider const post = (value: unknown) => fetch(`http://127.0.0.1:${address.port}/v1/responses`, { method: 'POST', body: JSON.stringify(value) }); const result = await post(body); assert.equal(result.status, status); assert.ok(!(await result.text()).includes(secret)); assert.deepEqual(diagnostics[0], { component: 'provider', code: 'provider_http_failed', upstream_status: status, + content_type_category: 'other', body_present: true, ...(!['oversized', secret].includes(code) ? { provider_code: code } : {}), ...(param && param !== secret ? { provider_param: param } : {}) }); const denied = await post({ ...body, [secret]: secret }); assert.equal(denied.status, 403); await denied.text(); assert.deepEqual(diagnostics[1], { component: 'proxy', code: 'proxy_unsupported_request_field' }); @@ -57,6 +58,61 @@ await test('proxy diagnostics retain only proven HTTP status and exact provider } finally { server.closeAllConnections(); server.close(); } } }); +await test('upstream 2xx requires a real SSE body; only classified response metadata and approved JSON codes escape', async () => { + const secret = 'Bearer fixture-private-token OAuth-code prompt source user-data'; + const json = JSON.stringify({ error: { code: 'model_not_found', param: 'model', message: secret, headers: secret }, detail: secret }); + const cases: { status: number; type: string | null; body: string | null; category: string; approvedCode?: boolean }[] = [ + { status: 200, type: 'text/event-stream', body: 'data: {"type":"response.completed"}\n\n', category: 'event-stream' }, + { status: 201, type: 'Text/Event-Stream; charset=utf-8', body: 'data: {"type":"response.completed"}\n\n', category: 'event-stream' }, + { status: 200, type: 'application/json; charset=utf-8', body: json, category: 'json', approvedCode: true }, + { status: 202, type: 'application/problem+json', body: json, category: 'json', approvedCode: true }, + { status: 200, type: 'application/json', body: JSON.stringify({ status: 'completed', output: [{ type: 'message', content: [{ type: 'output_text', text: secret }] }] }), category: 'json' }, + { status: 200, type: 'application/json', body: JSON.stringify({ error: { code: secret, param: secret }, detail: secret }), category: 'json' }, + { status: 200, type: 'application/json', body: JSON.stringify({ response: { error: { code: ['model_not_found'], param: ['model'] } } }), category: 'json' }, + { status: 200, type: 'application/json', body: '{"detail":"' + secret + '"}', category: 'json' }, + { status: 200, type: 'application/json', body: json + ' '.repeat(8192), category: 'json' }, + { status: 200, type: 'application/json', body: 'invalid-json ' + secret, category: 'json' }, + { status: 200, type: 'text/html; private=' + secret, body: '' + secret + '', category: 'html' }, + { status: 200, type: 'application/xhtml+xml', body: '' + secret + '', category: 'html' }, + { status: 204, type: 'text/event-stream', body: null, category: 'event-stream' }, + { status: 200, type: 'application/json', body: null, category: 'json' }, + { status: 200, type: null, body: null, category: 'missing' }, + { status: 200, type: null, body: json, category: 'missing', approvedCode: true }, + { status: 200, type: 'text/event-stream-private-' + secret, body: secret, category: 'other' } + ]; + for (const input of cases) { + const diagnostics: ExecutionFailure[] = []; + const upstream: typeof fetch = async () => { + const response = new Response(input.body, { status: input.status }); + if (input.type === null) response.headers.delete('content-type'); else response.headers.set('content-type', input.type); + response.headers.set('x-private-header', secret); return response; + }; + const server = createServer((request, response) => { void inferenceProxy(request, response, 'fixture-only', upstream, () => {}, value => diagnostics.push(value)); }); + server.listen(0, '127.0.0.1'); await once(server, 'listening'); const address = server.address(); assert.ok(address && typeof address !== 'string'); + try { + const result = await fetch(`http://127.0.0.1:${address.port}/v1/responses`, { method: 'POST', body: JSON.stringify({ model: 'fixture', store: false, stream: true, input: [] }) }); + const output = await result.text(), valid = input.category === 'event-stream' && input.body !== null; + assert.equal(result.status, valid ? input.status : 403); + if (valid) { assert.match(output, /response.completed/); assert.deepEqual(diagnostics, []); } + else assert.deepEqual(diagnostics, [{ component: 'provider', code: 'provider_stream_content_type_invalid', upstream_status: input.status, + content_type_category: input.category, body_present: input.body !== null, ...(input.approvedCode ? { provider_code: 'model_not_found', provider_param: 'model' } : {}) }]); + assert.ok(!output.includes(secret)); assert.ok(!JSON.stringify(diagnostics).includes(secret)); + assert.equal(result.headers.get('x-private-header'), null); + } finally { server.closeAllConnections(); server.close(); } + } +}); +await test('unreadable/cancellation-failed upstream bodies preserve only observed response metadata', async () => { + const secret = 'fixture-private-body-error'; + const diagnostics: ExecutionFailure[] = []; + const upstream: typeof fetch = async () => new Response(new ReadableStream({ pull() { throw new Error(secret); } }), { status: 200, headers: { 'content-type': 'application/json' } }); + const server = createServer((request, response) => { void inferenceProxy(request, response, 'fixture-only', upstream, () => {}, value => diagnostics.push(value)); }); + server.listen(0, '127.0.0.1'); await once(server, 'listening'); const address = server.address(); assert.ok(address && typeof address !== 'string'); + try { + const result = await fetch(`http://127.0.0.1:${address.port}/v1/responses`, { method: 'POST', body: JSON.stringify({ model: 'fixture', store: false, stream: true, input: [] }) }); + assert.equal(result.status, 403); assert.ok(!(await result.text()).includes(secret)); + assert.deepEqual(diagnostics, [{ component: 'provider', code: 'provider_stream_content_type_invalid', upstream_status: 200, content_type_category: 'json', body_present: true }]); + } finally { server.closeAllConnections(); server.close(); } +}); await test('inference locks model/effort and denies stored prompts and additional hosted tools', async () => { const localFetch = globalThis.fetch; let providerCalls = 0; const intercepted = mock.method(globalThis, 'fetch', async () => { providerCalls++; return new Response('data: {"type":"response.completed"}\n\n', { headers: { 'content-type': 'text/event-stream' } }); }); diff --git a/tests/sandbox-check.mjs b/tests/sandbox-check.mjs index e8ba790..35ef729 100644 --- a/tests/sandbox-check.mjs +++ b/tests/sandbox-check.mjs @@ -38,7 +38,9 @@ import {executionEnvelope} from '/app/dist/src/core/execution.js'; let calls=0; globalThis.fetch = async (url, init) => { if(url !== 'https://api.openai.com/v1/responses' || init.headers.Authorization !== 'Bearer fixture-only') throw new Error('Invalid inference boundary'); - if(++calls > 1) return Response.json({error:{code:'subscription_sharing_unsupported_capability',param:'tools',message:'private-fixture-provider-body'}},{status:400}); + const request=JSON.parse(init.body); + if(request.store!==false || request.stream!==true || !Array.isArray(request.input) || request.input.some(item=>item.role==='system')) throw new Error('Invalid SIWC request contract'); + if(++calls > 1) return Response.json({error:{code:calls===2?'subscription_sharing_unsupported_capability':'model_not_found',param:calls===2?'tools':'model',message:'private-fixture-provider-body'}},{status:calls===2?400:200}); return new Response(${JSON.stringify(events)}, {headers:{'content-type':'text/event-stream'}}); }; createInferenceServer('fixture-only',{},detail=>process.stderr.write(executionEnvelope(detail)+'\\n')).listen('/run/proxy/inference.sock',()=>process.stdout.write('ready\\n'));`; @@ -77,28 +79,32 @@ try { assert.ok(actual, 'Synthetic successful Codex turn did not complete; raw process output withheld.'); assert.deepEqual(JSON.parse(actual), result); process.stdout.write('Pinned Codex turn, Unix-socket inference proxy and structured output verified with synthetic SSE.\n'); - docker(['rm', review]); - docker(['create', '--name', review, ...common, '--mount', `type=bind,src=${directory},dst=/workspace,readonly`, '--mount', `type=volume,src=${volume},dst=/run/proxy,readonly`, 'codexbot-review:local', 'node', 'dist/src/runner/engine-main.js']); - let failed = false; - try { docker(['start', '-a', review]); } - catch (error) { - failed = true; - assert.equal(error.status, 1); - const engine = new ExecutionDiagnostics(); engine.write(String(error.stderr ?? '')); - assert.deepEqual(engine.last, { component: 'turn', code: 'codex_turn_failed' }); - assert.equal(String(error.stdout ?? '').trim(), ''); - assert.ok(!String(error.stderr ?? '').includes('private-fixture-provider-body')); + for (const status of [400, 200]) { + docker(['rm', review]); + docker(['create', '--name', review, ...common, '--mount', `type=bind,src=${directory},dst=/workspace,readonly`, '--mount', `type=volume,src=${volume},dst=/run/proxy,readonly`, 'codexbot-review:local', 'node', 'dist/src/runner/engine-main.js']); + let failed = false; + try { docker(['start', '-a', review]); } + catch (error) { + failed = true; + assert.equal(error.status, 1); + const engine = new ExecutionDiagnostics(); engine.write(String(error.stderr ?? '')); + assert.deepEqual(engine.last, { component: 'turn', code: 'codex_turn_failed' }); + assert.equal(String(error.stdout ?? '').trim(), ''); + assert.ok(!String(error.stderr ?? '').includes('private-fixture-provider-body')); + } + assert.equal(failed, true, 'Failed inference must exit nonzero with no review output'); + // Read via Node spawn to collect the safe stderr envelope without printing raw logs. + const observed = new ExecutionDiagnostics(); + await new Promise((resolveLogs, reject) => { + const logs = spawn('docker', ['--config', dockerConfig, 'logs', '--tail', '32', proxy], { stdio: ['ignore', 'ignore', 'pipe'] }); + logs.stderr.setEncoding('utf8'); logs.stderr.on('data', chunk => observed.write(chunk)); + logs.on('error', reject); logs.on('close', code => code === 0 ? resolveLogs() : reject(new Error('Synthetic proxy observation failed'))); + }); + assert.deepEqual(observed.last, { component: 'provider', code: status === 400 ? 'provider_http_failed' : 'provider_stream_content_type_invalid', + upstream_status: status, content_type_category: 'json', body_present: true, + provider_code: status === 400 ? 'subscription_sharing_unsupported_capability' : 'model_not_found', provider_param: status === 400 ? 'tools' : 'model' }); } - assert.equal(failed, true, 'Failed inference must exit nonzero with no review output'); - // Read via Node spawn to collect the safe stderr envelope without printing raw logs. - const observed = new ExecutionDiagnostics(); - await new Promise((resolveLogs, reject) => { - const logs = spawn('docker', ['--config', dockerConfig, 'logs', '--tail', '32', proxy], { stdio: ['ignore', 'ignore', 'pipe'] }); - logs.stderr.setEncoding('utf8'); logs.stderr.on('data', chunk => observed.write(chunk)); - logs.on('error', reject); logs.on('close', code => code === 0 ? resolveLogs() : reject(new Error('Synthetic proxy observation failed'))); - }); - assert.deepEqual(observed.last, { component: 'provider', code: 'provider_http_failed', upstream_status: 400, provider_code: 'subscription_sharing_unsupported_capability', provider_param: 'tools' }); - process.stdout.write('Pinned Codex failed turn and proxy upstream HTTP/code diagnostics verified; no raw provider body or successful output.\n'); + process.stdout.write('Pinned Codex failed turns for HTTP 400 and HTTP 200 JSON preserve safe proxy metadata, exit 1 and no review output.\n'); } finally { await cleanupReviewContainers(id); await cleanupReviewContainers(partial); docker(['volume', 'rm', volume]); await rm(directory, { recursive: true, force: true });