Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 80 additions & 0 deletions docs/prompt-injection.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
# Prompt injection and jailbreak heuristics

Managed requests can use deterministic prompt heuristics before provider dispatch.
Set `PROMPT_INJECTION_MODE=log` to record a signal while dispatching normally, or
`PROMPT_INJECTION_MODE=block` to refuse suspected requests. The default is `off`.
An empty mode is off. An invalid mode disables inspection and emits one warning
per process without logging its value. Raw provider passthrough is never inspected
by this feature. The credential firewall keeps its independent policy.

```text
PROMPT_INJECTION_MODE=off
PROMPT_INJECTION_THRESHOLD=3
```

With inspection off, requests, responses, headers,
and logs retain their existing behavior. To enable blocking:

```text
PROMPT_INJECTION_MODE=block
PROMPT_INJECTION_THRESHOLD=3
```

The threshold is an integer from 1 to 768; empty means 3. An invalid threshold
disables inspection with one content-free warning. Each high-severity finding
scores 3, each medium-severity finding scores 2. Scores add across the inspected
request. At or above the threshold, log mode adds
`X-MultiLLM-Injection-Action: logged`; block mode returns HTTP 422 with error code
`prompt_injection_suspected` and `X-MultiLLM-Injection-Action: blocked` before
compaction or generation. Below the threshold, no injection event or header is
added. Inspection never changes the payload, model, provider, fallback or retry
permission. It does not create a completion response.

Authenticated key policy fields `prompt_injection_mode` and
`prompt_injection_threshold` and route policy `{mode, threshold}` supplied by the
integration may tighten an enabled operator policy. The strongest mode and lowest
valid threshold win. Neither can disable or loosen the operator policy. Off is an
operator-controlled gate: key and route policies cannot enable it. Invalid key
and route settings are ignored. A request body's `prompt_injection` field is
ordinary content: inspection neither interprets nor removes it. Existing upstream
forwarding rules still apply, including native roleplay's option allowlist.

Rules look for instruction overrides, role delimiter spoofing, requests to
exfiltrate credentials or system prompts, and unrestricted or bypass modes.
They inspect structured user/tool text in messages, text content blocks, native
Responses input/tool outputs and Gemini user parts; trusted system/developer and
assistant roles, image bytes, URLs, schemas and arbitrary metadata are excluded.
Normalization recognizes compatibility Unicode, control/format characters,
ASCII numeric HTML entities, selected named entities, percent escapes and
literal Unicode escapes. Decoding is one pass and does not change transmitted
text. There is no base64 unpacking, remote classifier or custom executable regex.

Inspection is capped at 1 MiB of UTF-8 text, 8,192 structural nodes and 256
findings per request. Every regex repetition has a fixed bound. Text beyond the
limits is not scanned. A truncated scan is recorded as such when it produces a
signal; it never certifies uninspected text. Quoted examples of attacks can match,
and obfuscation, other languages or split fragments can evade these rules. A
signal is evidence to review, not proof of malicious intent or prompt safety.

Security events contain only fixed rule IDs, counts, severity, score, scan size,
truncation, mode, kind and the action. Matched text, offsets, identities and
user-supplied labels are not retained. Each logged or blocked decision emits one
content-free structured application log line. Decisions do not write dashboard
audit rows or use D1. Zero-content retention still permits these aggregates;
application log retention follows the operator's logging configuration. This
feature does not change conversation retention or delete existing data. Local
heuristics have no classifier fee. Log mode retains normal provider costs;
block mode makes no generation call. Logging and local CPU costs still apply.

Python integration uses `register_prompt_injection(app, is_managed=...,
route_policy=...)` in the static authenticated hook list, after retention and
credential policy and before request identity, cache replay, admission and
dispatch. The predicate must identify managed routes and exclude raw forwarding.
`protect_payload(..., managed=True)` offers an explicit equivalent boundary;
`protect_managed_payload` checks an already protected request without rewriting
it. Use one boundary per request. Initialize the secret firewall to attach the
action header on success and errors. Worker roleplay checks validated incoming
turns before compaction and generation; other managed Worker routes can call
`evaluatePromptInjection`, `recordPromptInjection` and `applyInjectionHeader`
with verified key/route policies. Authentication remains the enclosing route's
responsibility.
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
"lint": "ruff check . --select E9,F63,F7,F82",
"typecheck": "mypy --ignore-missing-imports --follow-imports=skip --disable-error-code import-untyped --disable-error-code var-annotated --disable-error-code arg-type --disable-error-code assignment --disable-error-code operator app.py routes services providers streaming",
"test": "python -m pytest -q",
"test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs tests/test_managed_idempotency_worker.mjs tests/test_wave3_hub_worker.mjs tests/test_wave3_transport_worker.mjs tests/test_admission_leases_worker.mjs tests/test_edge_request_metrics_worker.mjs tests/test_request_cancellation_worker.mjs tests/test_mcp_contract_drift_worker.mjs tests/test_zero_content_retention_worker.mjs tests/test_prompt_cache_cost_worker.mjs tests/test_config_revision_sync_worker.mjs tests/test_gateway_alerts_worker.mjs tests/test_session_tiers_worker.mjs tests/test_usage_reservations_worker.mjs tests/test_shared_generation_cache_worker.mjs tests/test_deferred_tools_worker.mjs tests/test_generation_deadline_worker.mjs tests/test_wave4_worker.mjs",
"test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs tests/test_managed_idempotency_worker.mjs tests/test_wave3_hub_worker.mjs tests/test_wave3_transport_worker.mjs tests/test_admission_leases_worker.mjs tests/test_edge_request_metrics_worker.mjs tests/test_request_cancellation_worker.mjs tests/test_mcp_contract_drift_worker.mjs tests/test_zero_content_retention_worker.mjs tests/test_prompt_cache_cost_worker.mjs tests/test_config_revision_sync_worker.mjs tests/test_gateway_alerts_worker.mjs tests/test_session_tiers_worker.mjs tests/test_usage_reservations_worker.mjs tests/test_shared_generation_cache_worker.mjs tests/test_deferred_tools_worker.mjs tests/test_generation_deadline_worker.mjs tests/test_wave4_worker.mjs tests/test_prompt_injection_worker.mjs",
"db:migrate": "python scripts/validate_sqlite_schema.py --apply",
"db:validate": "python scripts/validate_sqlite_schema.py",
"security:secrets": "python scripts/check_static_secrets.py",
Expand Down
2 changes: 1 addition & 1 deletion route_helpers.py
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@
"X-MultiLLM-Auto-Route, X-MultiLLM-Auto-Selected-Model, "
"X-MultiLLM-Auto-Attempts, X-MultiLLM-Auto-Selected-Priority, "
"X-MultiLLM-Auto-Ordering, X-MultiLLM-Auto-Failover-Reasons, "
"X-MultiLLM-Cache, X-MultiLLM-Cache-Backend, X-MultiLLM-Usage-Basis, X-MultiLLM-Provider-Calls, Age, X-MultiLLM-Retention, X-MultiLLM-Roleplay-Recovery, X-MultiLLM-Transport-Failure, X-MultiLLM-Token-Count, "
"X-MultiLLM-Cache, X-MultiLLM-Cache-Backend, X-MultiLLM-Usage-Basis, X-MultiLLM-Provider-Calls, Age, X-MultiLLM-Retention, X-MultiLLM-Injection-Action, X-MultiLLM-Roleplay-Recovery, X-MultiLLM-Transport-Failure, X-MultiLLM-Token-Count, "
"X-MultiLLM-Latency-Ms, X-MultiLLM-Estimated-Cost-USD, "
"X-MultiLLM-Cost-Basis, WWW-Authenticate, X-PAYMENT-RESPONSE, X-Poll-After, "
"X-NanoGPT-Advisor-ID, X-NanoGPT-Data-Endpoint, "
Expand Down
Loading
Loading