PROBE

A privacy-safe collective operational memory for software vendors.

A vendor ships one product — an HMS, an ERP, a GitLab-style platform — into hundreds of isolated, often regulated customer deployments. Each one breaks alone; logs are siloed and usually can't be shared outside the deployment for privacy/compliance reasons, so the same incident gets re-diagnosed from scratch at Site A, then B, then C. PROBE is the shared "HIVE" that stops that: it learns incident→fix patterns across a vendor's whole fleet — from sanitized fingerprints, never raw logs — so the 500th site never rediscovers a problem the 1st already solved.

👉 Open the landing page for the full architecture, the sanitization/matching boundary, the Jira-driven detect→match→suggest→ validate→learn loop, the two deployment modes, and the Elastic+AWS stack.

Built for Forge the Future 2026 (Elastic × AWS).

What's in this repo today

This repo implements two concrete stages of that pipeline end to end:

• Detector — seven independent anomaly-detection methods (z-score, change-point, Elastic ML jobs, a supervised classifier, and combinations of them) validated against a live OpenTelemetry demo fleet. Start at DETECTORS.md.
• Remediator — the FastAPI + React app below, which files the Jira ticket once a root cause is known. This is the rest of this README.

No Correlator (the LLM-via-Bedrock reasoning stage that ranks a detector's candidates and drives the HIVE match/suggest loop) or shared HIVE index exists in this repo yet — causal-changepoint-detection/'s heuristic ranking is the current stand-in for it. See DETECTORS.md for the honest state of each piece.

---

Remediator: configure Jira from your own UI, not from Kibana

Configure Elasticsearch's Jira integrations from your own UI, not from Kibana.

Enter your Jira credentials once. Probe uses them to provision both halves of the Elastic ↔ Jira story, and keeps its own Jira access so ticket creation works regardless of your Elastic licence.

• FastAPI backend (backend/)
• React + Vite frontend (frontend/)
• Elasticsearch + Kibana + elastic-connectors via docker-compose.yml

This is the Remediator stage of the full PROBE design (files the ticket once a cause is known). The Detector stage — four independent methods that find the anomaly in the first place, validated against a live OpenTelemetry demo — lives alongside it in this repo: DETECTORS.md (overview + comparison table), RUNNING_LOCALLY.md (self-hosted, no cloud account), or RUNNING_ON_ELASTIC_CLOUD.md (same demo + detectors, pointed at a real Elastic Cloud/Serverless deployment).

---

Read this first

Elastic has two unrelated things called "connectors", and only one of them can create a Jira issue:

	Content connector (service_type: jira)	Action connector (.jira)
Direction	Jira → Elasticsearch	Elasticsearch → Jira
Purpose	index tickets for search / dedupe	create tickets
Lives on	Elasticsearch	Kibana
Licence	Basic (free)	Gold

The Kibana Jira action connector is registered minimumLicenseRequired: 'gold' (elastic/kibana#67178), and Gold is discontinued for new customers. On a free/Basic stack, Elasticsearch cannot create a Jira ticket.

So Probe runs two execution paths behind one interface, and picks between them by reading enabled_in_license from GET /api/actions/connector_types at runtime:

1. elastic_connector — Kibana's .jira connector does the write (Gold+).
2. direct_jira — Probe calls POST /rest/api/3/issue itself (always works).

Everything works end to end on free tiers. Only the component making the outbound call changes, and the UI says which one it used.

Full findings: docs/RESEARCH.md. Every key, its free-tier status and a placeholder value: docs/KEYS.md. Design: docs/ARCHITECTURE.md.

What it does

• One setup wizard collects Elastic + Jira credentials and validates both live.
• Checks the Jira account really holds BROWSE_PROJECTS and CREATE_ISSUES before finishing, instead of failing on the first ticket.
• Provisions the Elasticsearch content connector, pushes the Jira settings into its configuration, schedules a sync and mints the connector service's API key.
• Provisions the Kibana .jira action connector when the licence allows it, and says plainly why it didn't when it doesn't.
• Creates tickets, warning about likely duplicates found in the synced index.
• Searches synced tickets, falling back to JQL before the first sync lands.
• Encrypts credentials at rest; the API only ever returns masked values.

Quick start

1. Elastic stack (optional — point at an existing one instead)

docker compose up -d elasticsearch kibana

2. Backend

cd backend
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env          # then edit it

# a real encryption key:
python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"

uvicorn app.main:app --reload --port 8000

API docs at http://localhost:8000/docs.

3. Frontend

cd frontend
npm install
cp .env.example .env          # VITE_PROBE_API_KEY must match PROBE_API_KEY
npm run dev

Open http://localhost:5173 and complete the wizard.

4. Connector service

Provisioning shows an API key once. Paste it into connectors-config/config.yml (see config.yml.example), then:

docker compose up -d connectors

Until this service is running, Elasticsearch has a connector record but no configuration schema, and Jira settings cannot be written. Probe reports this explicitly rather than leaving you with a connector that looks fine and syncs nothing.

Credentials you need

Summary — see docs/KEYS.md for the full table with placeholders.

	Where from	Free?
Jira API token	https://id.atlassian.com/manage-profile/security/api-tokens	✅ Free plan
Jira account email	the account that owns the token	✅
Jira project key	must grant that account BROWSE_PROJECTS + CREATE_ISSUES	✅
Elastic API key	needs manage_connector, manage_api_key, monitor	✅ Basic
PROBE_SECRET_KEY	Fernet.generate_key()	—

No OAuth app, no Marketplace add-on, no paid Elastic tier.

Jira API tokens use HTTP Basic auth, where OAuth scopes are not enforced — the token inherits the account's permissions. There is no token setting that grants "create issues"; a Jira admin grants it to the account.

API

Method	Path	Purpose
GET	/health	liveness (no auth)
GET	/api/setup/defaults	seed the setup form
POST	/api/validate/elastic	test ES + Kibana, report the licence gate
POST	/api/validate/jira	test Jira auth, permissions, issue types
POST	/api/setup	store credentials (encrypted)
GET	/api/connection	masked stored state
DELETE	/api/connection	forget credentials
GET	/api/connector/capabilities	licence → capability matrix → route
POST	/api/connector/provision	provision both connectors
GET	/api/connector/status	sync state, doc count, last error
POST	/api/connector/sync	trigger a full sync
GET	/api/connector/service-config	config.yml for the connector service
GET	/api/jira/projects	project picker
GET	/api/jira/issue-types	issue-type picker
GET	/api/jira/permissions	permission pre-flight
POST	/api/tickets	create a ticket
POST	/api/tickets/search	search synced tickets
POST	/api/tickets/similar	duplicate check

All /api/* routes require X-Probe-Key when PROBE_API_KEY is set. When it is unset the guard is inert, so first-run local development needs no secret.

Tests

cd backend && pytest -q        # 58 tests

They cover the licence gate and both routing paths, provisioning step reporting, ADF conversion, JQL escaping, encryption at rest and secret masking, using respx to stand in for Elasticsearch, Kibana and Jira.

Limits

• Jira Cloud only. The content connector supports Server/Data Center (data_source), but the setup flow and validation assume Cloud.
• One connection profile per deployment — state is a single encrypted file.
• Duplicate detection uses a fixed Lucene score threshold (8.0). Lucene scores are not normalised, so tune DUPLICATE_SCORE_THRESHOLD to your corpus.
• The content connector's document shape varies by connector version; the search mapping uses wildcards to stay resilient, which costs some precision.