Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
103 changes: 103 additions & 0 deletions docs/observability-exports.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
# External observability exports

The gateway can send content-free request observations to explicit Langfuse and
Helicone collectors. `OBSERVABILITY_EXPORTERS_JSON` defaults to `[]`, which sends
nothing. An unset or empty value also disables export. Malformed JSON, unsupported
exporter types, unknown fields or invalid destinations disable all exporters and
produce one warning without the setting's value.

Configure each destination with exactly these fields:

```json
[
{
"type": "langfuse",
"endpoint": "https://langfuse.example/api/public/ingestion",
"allowed_origins": ["https://langfuse.example"],
"credential_env": "LANGFUSE_EXPORT_CREDENTIAL"
},
{
"type": "helicone",
"endpoint": "https://helicone.example/custom/v1/log",
"allowed_origins": ["https://helicone.example"],
"credential_env": "HELICONE_EXPORT_CREDENTIAL"
}
]
```

Replace the example destinations with the collector ingestion endpoints you
operate. There are no implicit hosts or fallback destinations. Endpoints must use
HTTPS and match an exact configured origin, including a non-default port. Wildcards,
URL credentials, query strings, fragments and redirects are refused. Origin entries
have no path beyond `/`. Configuration is limited to 32 KiB, eight exporters and
sixteen origins per exporter. Duplicate destinations with the same type and
credential reference are invalid.

`credential_env` names a private environment variable whose name ends in
`_EXPORT_CREDENTIAL`, so a provider or administrator secret cannot be sent to a
collector by mistake. For Langfuse its value is
the public key and secret key joined by a colon; it is encoded as HTTP Basic
authentication at delivery time. For Helicone its value is the collector token,
sent as HTTP Bearer authentication. Values appear only in outbound authentication
headers. They never appear in configuration examples, export bodies, status or logs.
An absent or empty credential causes a delivery failure without an HTTP call.
The Flask Container must receive the configured credential variables through its
explicit environment allowlist. Configure Worker bindings separately.

Observations contain request and correlation IDs, a hashed principal identifier,
the selected authorized model, request kind, status, a fixed error classification,
timings, nullable token counts, nullable cost, and usage/cost provenance. They never
include prompts, generated output, tools, API-key prefixes, raw request or response
headers, arbitrary metadata or upstream error messages. Model names come from
trusted finalization metadata. Zero token counts and zero costs remain distinct
from unknown values. Usage provenance remains `unknown` when unavailable; no price
is inferred from a collector response. Shadow and canary observations retain their
own kind and cost instead of being folded into production request totals.

Langfuse receives `generation-create` ingestion events with stable IDs and the
observation in generation metadata. Helicone receives content-free custom log
envelopes, split seconds/milliseconds timing, and the observation as the
`Helicone-Property-observation` property. The structural provider URL is the neutral
`https://multillm.invalid`; it is never used for a generation or network request.
Collectors requiring prompt/output fields cannot use this export mode. There is
no content-capture option.

Each process or Worker environment has an in-memory queue of at most 1,000
observations. Delivery takes at most 50 observations per batch; Helicone's custom
endpoint uses one HTTP envelope per observation. Every delivery attempt has a
two-second timeout and at most two retries, limited to transport failures, HTTP
429 and HTTP 5xx. Redirects and other HTTP errors are terminal. Export retries
never invoke a generation or change its retry permission. Stable event IDs are
reused on collector retries, but remote duplicate suppression depends on the
collector. The queue holds no durable telemetry: restarts or Worker eviction can
lose accepted records. No D1 or R2 table is created for exports.

Flask submissions only enqueue; a daemon thread delivers them independently of
OTLP. The existing OTLP payloads, headers, queue, return values and configuration
remain unchanged with exporters enabled or disabled. Native Worker finalization
enqueues its own observation and schedules export with the request execution
context's `waitUntil`. Forwarded requests export from Flask; their private D1
ledger flush never exports again. Worker collection and scheduling require the
native finalization hook to supply that execution context. Without scheduling,
acceptance only buffers the observation until an explicit flush.

`OBSERVABILITY_EXPORTER.status()` in Flask and
`getObservabilityExporter(env).status()` in the Worker return accepted, queued and
dropped counts. Each exporter reports its type, credential variable name,
attempt count, acknowledged count, failed count and delivery status (`idle`,
`acknowledged`, `partial`, `failed` or `credential_unavailable`). These are internal
operator interfaces, not new public endpoints. Queue acceptance is **accepted**,
never delivered. A 2xx HTTP acknowledgement indicates collector acceptance, not
proof of persistent storage. Langfuse HTTP 207 receipts count only successful
event IDs belonging to the submitted batch and exclude IDs listed as errors;
missing, malformed or oversized receipts acknowledge nothing. Partial receipts
are not retried. Collector response bodies and exception details are never logged.

Delivery failures are independent across exporters and do not alter generation
responses, accounting, storage or authorization. The queue drops newest items
when full and counts those drops. Deduplication retains the most recent 1,000
accepted request/owner/kind identities; it is process-local and does not promise
distributed exactly-once delivery. Content-free export is compatible with zero
content retention, but identifiers, model names, timings and usage still leave
the gateway at the operator's explicit direction. Local tests use fake collectors;
they do not establish live collector compatibility, persistence or deployment.
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
"lint": "ruff check . --select E9,F63,F7,F82",
"typecheck": "mypy --ignore-missing-imports --follow-imports=skip --disable-error-code import-untyped --disable-error-code var-annotated --disable-error-code arg-type --disable-error-code assignment --disable-error-code operator app.py routes services providers streaming",
"test": "python -m pytest -q",
"test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs tests/test_managed_idempotency_worker.mjs tests/test_wave3_hub_worker.mjs tests/test_wave3_transport_worker.mjs tests/test_admission_leases_worker.mjs tests/test_edge_request_metrics_worker.mjs tests/test_request_cancellation_worker.mjs tests/test_mcp_contract_drift_worker.mjs tests/test_zero_content_retention_worker.mjs tests/test_prompt_cache_cost_worker.mjs tests/test_config_revision_sync_worker.mjs tests/test_gateway_alerts_worker.mjs tests/test_session_tiers_worker.mjs tests/test_usage_reservations_worker.mjs tests/test_shared_generation_cache_worker.mjs tests/test_deferred_tools_worker.mjs tests/test_generation_deadline_worker.mjs tests/test_wave4_worker.mjs",
"test:worker": "node --test tests/test_cloudflare_worker.mjs tests/test_cascades_worker.mjs tests/test_cascades_dashboard.mjs tests/test_image_quality_worker.mjs tests/test_secret_scan_worker.mjs tests/test_secret_firewall_worker.mjs tests/test_secret_firewall_coverage_worker.mjs tests/test_roleplay_compaction_budget_worker.mjs tests/test_roleplay_compaction_model_worker.mjs tests/test_roleplay_stream_validation_worker.mjs tests/test_aihubmix_worker.mjs tests/test_gpt_image_moderation_worker.mjs tests/test_dashboard_scripts.mjs tests/test_operations_dashboard.mjs tests/test_opencode_reasoning_worker.mjs tests/test_opencode_thinking_worker.mjs tests/test_nanogpt_reasoning_worker.mjs tests/test_nanogpt_speed_routing_worker.mjs tests/test_sse_heartbeat_worker.mjs tests/test_roleplay_worker.mjs tests/test_roleplay_config_worker.mjs tests/test_roleplay_glm_failover_worker.mjs tests/test_roleplay_intelligence_worker.mjs tests/test_roleplay_memory_worker.mjs tests/test_roleplay_compatibility_worker.mjs tests/test_roleplay_history_reconciliation_worker.mjs tests/test_roleplay_output_contract_analysis.mjs tests/test_roleplay_output_contract_cleanup_worker.mjs tests/test_roleplay_output_contract_nonstream_worker.mjs tests/test_roleplay_output_contract_worker.mjs tests/test_roleplay_performance_worker.mjs tests/test_roleplay_quality_routing_worker.mjs tests/test_roleplay_reasoning_worker.mjs tests/test_roleplay_session_storage_worker.mjs tests/test_roleplay_transport_failure_worker.mjs tests/test_roleplay_unlimited_worker.mjs tests/test_roleplay_workbench_worker.mjs tests/test_roleplay_candidate_context_worker.mjs tests/test_roleplay_operator_worker.mjs tests/test_workbench_stream.mjs tests/test_intelligence_worker.mjs tests/test_intelligence_d1_worker.mjs tests/test_intelligence_auth_worker.mjs tests/test_intelligence_outbound_worker.mjs tests/test_intelligence_operator.mjs tests/test_control_users_worker.mjs tests/test_control_audit_worker.mjs tests/test_access_sso_worker.mjs tests/test_d1_schema_worker.mjs tests/test_shadow_eval_worker.mjs tests/test_auto_routes_worker.mjs tests/test_ai_outbound_worker.mjs tests/test_knowledge_*.mjs tests/test_control_state_worker.mjs tests/test_route_health_worker.mjs tests/test_gateway_mcp_worker.mjs tests/test_usage_ledger_worker.mjs tests/test_media_outbound_worker.mjs tests/test_media_files_worker.mjs tests/test_media_jobs_worker.mjs tests/test_tool_repair_worker.mjs tests/test_prometheus_worker.mjs tests/test_evaluation_noise_floor_worker.mjs tests/test_prompt_templates_worker.mjs tests/test_config_snapshots_worker.mjs tests/test_managed_idempotency_worker.mjs tests/test_wave3_hub_worker.mjs tests/test_wave3_transport_worker.mjs tests/test_admission_leases_worker.mjs tests/test_edge_request_metrics_worker.mjs tests/test_request_cancellation_worker.mjs tests/test_mcp_contract_drift_worker.mjs tests/test_zero_content_retention_worker.mjs tests/test_prompt_cache_cost_worker.mjs tests/test_config_revision_sync_worker.mjs tests/test_gateway_alerts_worker.mjs tests/test_session_tiers_worker.mjs tests/test_usage_reservations_worker.mjs tests/test_shared_generation_cache_worker.mjs tests/test_deferred_tools_worker.mjs tests/test_generation_deadline_worker.mjs tests/test_wave4_worker.mjs tests/test_observability_exports_worker.mjs",
"db:migrate": "python scripts/validate_sqlite_schema.py --apply",
"db:validate": "python scripts/validate_sqlite_schema.py",
"security:secrets": "python scripts/check_static_secrets.py",
Expand Down
Loading
Loading