Make AI quality observable.
OpenSpanGuard adds quality signals to your original AI traces. This OTLP/HTTP JSON enrichment gateway evaluates completed generation spans, preserves their trace/span IDs and timestamps, and exposes evaluation coverage alongside relevance in Prometheus and Grafana.
Developer Preview · Apache-2.0 · Node.js 22 / 24 tested · Validate on a canary before production
Try the demo ↓ · Read the integration guide · Website · Français
Why OpenSpanGuard · Demo · Docker stack · Integration · Signals · Architecture · Docs and community
- Keep trace context. Inspect quality signals on the exported generation span, with its original identity and timing.
- See evaluation gaps. Missing content, disabled evaluation and timeouts remain explicit; successful scores never hide low coverage.
- Use your observability stack. Forward enriched traces to your collector and query bounded cohort metrics in Prometheus/Grafana.
Built for engineers operating instrumented AI applications. With a BatchSpanProcessor, evaluation runs on the telemetry export path after generation completes. Export and flush still incur evaluation and delivery latency. Start with the demo, then validate a permissioned canary against your own labels.
Use Node.js 22 or 24 and npm:
git clone https://github.com/alebgl77/openspanguard.git
cd openspanguard
npm ci
npm run build
npm run demoOnce dependencies are installed, the demo runs with loopback services and requires no provider API key or Docker. It checks original IDs/timestamps, concurrent replay suppression, missing-content coverage, recognized-content redaction, a real OpenTelemetry JSON exporter, and streaming handling. The final JSON contains "result": "pass". All demo scores are synthetic.
See the recorded offline case study for the tested commit, commands, 81 passing tests, observed demo output and limits.
If PowerShell blocks npm.ps1, use npm.cmd for the same commands. See troubleshooting if setup fails.
Requires Docker Engine, Docker Compose and Node.js for the smoke scripts. From the cloned repository:
docker compose up --build -d --wait
node examples/stack-smoke.mjs
node examples/send-demo.mjs 180Open Grafana. Three minutes of synthetic traffic populate relevance, confidence, nonsense, coverage, token and export-health panels. The smoke script checks delivery to Tempo and injects a synthetic alert to verify the local Alertmanager/webhook route.
Ports bind to loopback. Grafana is an anonymous read-only demo; Tempo storage is ephemeral. Stop with docker compose down; see deployment for storage, cleanup and alert-rule validation.
Use OTLPTraceExporter from @opentelemetry/exporter-trace-otlp-http in your existing provider's BatchSpanProcessor, targeting http://127.0.0.1:4318/v1/traces. This preview accepts HTTP/JSON. The -proto exporter sends protobuf and receives HTTP 415.
The SDK helper is included in this source repository. The package is private; the quickstart builds locally. For an existing span, a JavaScript entry point in the cloned repository root can use:
import { annotateGeneration } from './dist/src/sdk.js';
span.setAttribute('gen_ai.operation.name', 'chat');
annotateGeneration(span, {
prompt: 'What is a trace?',
response: 'A trace records an operation.',
cohort: 'canary',
});
span.end();Capture content explicitly before ending the span. Supported gen_ai.input.messages and gen_ai.output.messages fields also work. Model names and token counts alone cannot supply evaluation content. With a running gateway, npx tsx examples/sdk.ts demonstrates batch export and streaming accumulation. See integration for auth, lifecycle handling and privacy boundaries.
If the full demo stack is running, stop it with docker compose down first to free gateway port 4318. Then start only the local collector/Tempo with docker compose up -d collector tempo. Build with npm run build. In a POSIX shell, with OSG_JEV_API_KEY already supplied by your secret manager:
export OSG_EVALUATOR=jev
export OSG_ALLOW_REMOTE_CONTENT=true
export OSG_OTLP_ENDPOINT=http://127.0.0.1:4319/v1/traces
export OSG_JEV_API_KEY
npm startPowerShell setup is available. The process reads environment variables; copying .env.example does not load them. Jev sends captured content to the configured evaluation endpoint before recognized content is removed from onward traces, and provider charges may apply. Review configuration and measurement limits first.
| Signal | What it tells you |
|---|---|
| Relevance | A five-level rubric result, normalized from 0–4 to 0–1 |
| Confidence | The evaluator's confidence in its relevance decision |
| Nonsense probability | Jev's noul result for this project's specific nonsense question |
| Coverage | Successful first-observation evaluations / first-observed generation spans, using the bounded local identity window |
These are evaluator signals, not proof of correctness. Nonsense probability is not a general hallucination probability. The sample relevance alert requires 20 evaluations in 5 minutes, at least 80% coverage, and mean relevance below 0.6 for 2 minutes. Calibrate those starting values against your workload before paging. Metrics record the first observed outcome for an identity; a successful retry does not revise that outcome while the metric identity window still suppresses it.
Read measurement limits for held-out evaluation, false positives, sample bias and benchmark methodology.
Technical diagram and editable sources
flowchart LR
A["Application + OTel SDK<br/>explicit content capture"] -->|completed spans / HTTP JSON| E["OpenSpanGuard<br/>bounded evaluation"]
E -.->|optional / explicit remote opt-in| J["Jev<br/>content before redaction"]
J -.->|evaluation result| E
E --> R["Remove recognized content<br/>enrich exported spans"]
R -->|original trace and span IDs| C["OTel Collector"]
C --> T["Tempo / trace backend"]
P["Prometheus"] -->|scrape metrics| E
G["Grafana"] -->|query metrics| P
G -->|query traces| T
P -->|alerts| AM["Alertmanager"]
AM --> W["Local demo webhook"]
Editable diagrams: architecture · quality loop.
Evaluation enriches exported OTLP; it does not modify the ended SDK span. gen_ai.evaluation.result is a span-event compatibility representation, not an emitted standard LogRecord. Recognized content is removed before forwarding; unrelated custom fields can retain sensitive data.
Cache and deduplication are bounded and process-local. Expiry, eviction, restart or another replica can cause re-evaluation. There is no disk queue or global exactly-once guarantee. See architecture and the production checklist.
| Need | Start here |
|---|---|
| Connect an application | Integration · SDK example |
| Operate the gateway | Configuration · Deployment |
| Understand evidence and limits | Measurement · FAQ |
| Fix an integration | Troubleshooting · Support |
| Discuss or contribute | Discussions · Contributing · Roadmap |
| Share the project | Media kit · Launch kit |
To verify a change, run npm run typecheck, npm test, npm run build and npm run demo. npm run benchmark -- 200 reports synthetic loopback measurements, not live Jev accuracy or production capacity.
Independent open-source project; no official OpenTelemetry, Grafana or Jev endorsement is implied. Apache-2.0 license · Security · Code of conduct.
Built and maintained by Alexandre Beguel, Agentic AI Architect and external CAIO based in Paris. He works on production GenAI, LLMOps, LLM FinOps and evidence-first AI governance. beguel.com · LinkedIn · Profile in English, French and Spanish