Self-hosted AI discovery, with evidence behind every finding.
Understand which AI services and tools appear in your environment, where each signal came from, and what it can actually prove. Bring together network metadata, endpoint inventory and Microsoft directory signals in an interface built for investigation.
Expert implementation handbook · Get started · Professional documentation hub · Roadmap (English reference)
15 October audit remediation and operator checks · Repository governance
Early-stage software for evaluation and controlled pilots. One organization per deployment. Docker, Kubernetes and live Microsoft environments require operational validation in your environment; no certification or production SLA is claimed.
This README describes maintained source aligned on 2026-10-15, commit d159ed68012281a2c9e887e1281d44f0fb75112c (merged PR #17), with backend preview 0.3.0 and standalone agent 0.3.0. The latest published prerelease remains v0.3.0 (2026-10-07, 49a58d7, agent 0.1.0); it predates the extension inventory, localization, cartography and 15 October corrections.
The console offers English, French and Simplified Chinese before and after sign-in. A validated browser-local language preference is separate from authentication; English is the default when no preference is available or storage is blocked. Raw source data and machine/API identifiers retain their original values. See language coverage.
Use the professional documentation hub for the implementation reading order and source/release availability, then follow the expert handbook. Version 0.3.0 alone does not identify the delivered code: record the exact deployed commit, configuration, image digests and CI/build evidence. Later changes require their own qualification; successful CI does not publish images or certify your target environment.
Three silent French diagrams explain the AI footprint, evidence levels and collector recovery. These editorial illustrations use synthetic data; they are separate from the actual interface shown above.
Explore all three diagrams (English reference) · Open/download the MP4 · View the static poster
- Evidence you can inspect. Distinguish network observations, installed assets, directory inventory and instrumented usage.
- Infrastructure you control. Self-host the API, queue, event store and interface.
- Progressive adoption. Start with one log source or a small managed-device pilot.
- An open catalog. Review and contribute the signatures behind classifications.
- Honest gaps. A missing signal remains unknown; a DNS request is not a prompt or a bill.
See the footprint. Understand the risk. Keep control. The six-step roadmap (English reference) sets out richer discovery, agent visibility, optional local analysis, investigations that cite evidence, deliberate usage controls and continuous qualification. Each future step has measurable delivery gates; candidate technologies remain proposals for evaluation.
Prerequisite: Node.js 22.22.2+ (22.x). Launch the isolated interface demo:
cd frontend
npm ci
npm run demoOpen the /demo route at the local address printed by the development server. This is a synthetic demo session: it needs no backend credentials and does not connect to your directories or fleet. To continue with the full installation, stop the demo server and run cd .. first.
Prerequisites: Git, Python 3.12+ and a working Docker Engine with Compose v2. No Docker runtime is bundled. The initial interface binds to localhost over HTTP; remote agents require an HTTPS reverse proxy.
set -e
git clone https://github.com/alebgl77/open-shadow-ai.git
cd open-shadow-ai
bash scripts/bootstrap.sh
docker compose config --quiet
docker compose up -d --build api ingest-worker correlation-worker purge-worker frontend
for attempt in $(seq 1 30); do
if MSYS2_ARG_CONV_EXCL='*' docker compose exec -T api python /app/entrypoint.py python -m shadai.workers.redis_lifecycle reconcile --execute --legacy-writers-stopped; then break; fi
if [ "$attempt" = 30 ]; then exit 1; fi
sleep 2
done
docker compose up -d --wait --wait-timeout 180 api ingest-worker correlation-worker purge-worker frontend
docker compose run --rm api python -m shadai.cli create-adminOn Windows, prepare a trusted checkout and run ./scripts/bootstrap.ps1 in PowerShell first. In Git Bash, use that same checkout and Docker context: run set -e, then the Compose commands in the block above; skip bash scripts/bootstrap.sh. The retry loop requires Bash and cannot run directly in PowerShell. The per-call MSYS2_ARG_CONV_EXCL='*' prevents Git Bash path conversion from rewriting container paths; it does not change global environment settings. Both bootstraps preserve existing secrets and configuration; use -DryRun or --dry-run to preview.
Wait for the new workers to initialize their actual Redis groups before reconciliation; retry if initialization is still in progress. On upgrades, stop all legacy writers/workers/replay clients before starting the indexed version. Readiness requires the reconciled retention schema. See queue retention (English reference) and qualification (English reference) for failure behavior and the isolated rehearsal.
Open localhost:3000. PostgreSQL migrations run in the migrate service before the API starts; the three ClickHouse schema files load on a fresh event-store volume. Updating an existing installation (English reference) requires separate ClickHouse migrations.
Enabling collectors is deliberate:
docker compose --profile syslog up -d
# After configuring Entra IDs and its separately provisioned client-secret file:
docker compose --profile entra up -dAD exports and endpoint installs are operator-run workflows, documented in the Microsoft guide (English reference). Installing the server does not automatically scan the fleet.
Administrators can enroll scoped collectors in Sources & coverage, provision their one-time keys privately and rotate or revoke them. Endpoint, network and syslog collectors have bounded persistent delivery queues; pipeline status separates pending work, retained entries and unknown capture loss. Follow the operations guide (English reference) for enrollment, restart recovery, authenticated monitoring and identity retention.
Optional OIDC sign-in and SCIM provisioning (English reference) manage access to the console, including Microsoft Entra ID. They are disabled in the base deployment and use separate credentials from the inventory collector. Local administrator sign-in remains available for recovery.
The capabilities below describe the maintained source baseline above. Check release availability separately.
| Capability | Status and practical boundary |
|---|---|
| React investigation interface, catalog and governance workflow | Implemented; evaluate with your data |
| DNS/proxy parsers and syslog collector | Implemented; parser and listener must match the source |
| Passive network analysis (English reference) | Zeek/Suricata/TShark metadata imports, optional offline PCAP or explicit live sensor, and /network evidence view; observed names do not prove AI requests |
| AI cartography | /cartography connects network observations and reported host installations to catalog services, with 3D/list navigation, retained-event evidence and protocol/catalog-family radars over loaded observations; bounded coverage, presence and observations do not prove use |
| Endpoint agent | Implemented process/container/runtime/extension inventory; OS and fleet rollout verification pending |
| Browser extension inventory | Implemented per-installation state (version, enabled, install source, user, profile) for accessible local users/profiles on agent 0.3.0 hosts, version-change history and audited export in Extensions; presence evidence kept for the identity window, not usage |
| Microsoft Entra collector | Implemented service-principal inventory and optional grants; AI matching requires reviewed application-ID mappings; live tenant validation pending |
| On-prem Active Directory | OU-scoped read-only PowerShell exporter; RSAT/live AD validation pending |
| Generic event ingestion | Authenticated, bounded batches; custom adapters remain your integration work |
| Collector enrollment and recovery | Scoped one-time credentials, rotation/revocation, bounded local spools and explicit dead-letter replay; native/store CI results must be checked for the deployed commit |
| Model/token/cost evidence | Accepts instrumented metadata where supplied; network logs cannot produce these values |
| Other directories (LDAP, Okta, Google Workspace) | Custom ingestion contract available; native connectors planned |
| Docker Compose | Deployment package and CI service smoke provided; validate the first run |
| Kubernetes | Baseline for external stores; cluster, ingress, backup and scaling validation pending |
| OIDC SSO and SCIM 2.0 console provisioning | Implemented optional subset with Entra setup guide; live tenant interoperability validation pending |
| Isolated multi-tenancy, automated fleet management | Planned; one organization per install today |
| Inline blocking, prompt DLP, full behavioral analysis | Outside the current product |
Editable draw.io diagram · Mermaid and data flow (English reference)
The Python package and environment variable prefix remain shadai for compatibility. The public project name is Open Shadow AI.
The default configuration never stores URL paths, query strings or user agents. At ingestion, paths (without their query string) and user agents are compared in memory with catalog patterns scoped to each product's hosts; only the matched catalog entry is kept. Set privacy.match_transient_signals: false to skip that comparison. Treat usernames, device identifiers and directory exports as personal or organizational data. Set retention and access according to your deployment.
Per-observation identity membership expires independently (default at most 30 days); displayed counts are distinct observed identifiers, not staff headcount. Optional server pseudonymization affects new processing and does not scrub history or plaintext local spools. Risk scores retain their last calculated snapshot and expose a nullable calculation time and stale marker. Historical cleanup is an explicit bounded maintenance operation; see identity retention and pseudonymization (English reference).
DNS resolution indicates contact with a domain, not a completed AI interaction. An installed extension or directory application indicates presence or permission, not usage. The browser extension inventory supports incident review, not employee monitoring: its rows and export are limited to analysts and administrators, the current privacy setting applies to newly processed host/user labels while existing labels can retain their previous form until the next snapshot, and rows expire with the identity window. Model identifiers are declared evidence; token totals and costs require instrumentation. Cost calculations remain estimates unless reconciled against provider billing.
Unmanaged devices, unobserved encrypted DNS, local tools, shared domains, embedded SaaS AI and gateway bypasses can leave gaps. See the coverage guide (English reference) and market analysis (English reference).
Read CONTRIBUTING.md, submit a small reproducible change and include synthetic fixtures for collectors. Report vulnerabilities privately using SECURITY.md.
Licensed under Apache-2.0. Copyright Open Shadow AI contributors.
For sizing and rollout, see capacity planning (English reference). Throughput claims require measurements on your workload.