diff --git a/README.md b/README.md index 8e49683..5509e42 100644 --- a/README.md +++ b/README.md @@ -1,34 +1,76 @@ -SentinelAI -image +

+ SentinelAI interface +

-# SentinelAI — Enterprise AI Reliability & Governance Platform +# SentinelAI — Reproducible Drift-Monitoring Reference System [![CI](https://github.com/CoreyLeath-code/SentinelAI/actions/workflows/ci-cd.yml/badge.svg)](https://github.com/CoreyLeath-code/SentinelAI/actions/workflows/ci-cd.yml) -[![Benchmarks](https://github.com/CoreyLeath-code/SentinelAI/actions/workflows/benchmarks.yml/badge.svg)](https://github.com/CoreyLeath-code/SentinelAI/actions/workflows/benchmarks.yml) +[![Research benchmark](https://github.com/CoreyLeath-code/SentinelAI/actions/workflows/benchmarks.yml/badge.svg)](https://github.com/CoreyLeath-code/SentinelAI/actions/workflows/benchmarks.yml) [![Security](https://github.com/CoreyLeath-code/SentinelAI/actions/workflows/security.yml/badge.svg)](https://github.com/CoreyLeath-code/SentinelAI/actions/workflows/security.yml) [![SAST](https://github.com/CoreyLeath-code/SentinelAI/actions/workflows/sast.yml/badge.svg)](https://github.com/CoreyLeath-code/SentinelAI/actions/workflows/sast.yml) -[![Schema Validation](https://github.com/CoreyLeath-code/SentinelAI/actions/workflows/data-validation.yml/badge.svg)](https://github.com/CoreyLeath-code/SentinelAI/actions/workflows/data-validation.yml) -[![Release](https://github.com/CoreyLeath-code/SentinelAI/actions/workflows/release.yml/badge.svg)](https://github.com/CoreyLeath-code/SentinelAI/actions/workflows/release.yml) -[![Benchmark](https://img.shields.io/badge/reference%20p95-54.7%20%C2%B5s-6f42c1)](benchmarks/benchmark_report.md) -[![Throughput](https://img.shields.io/badge/reference%20throughput-23.0k%20ops%2Fs-2ea44f)](benchmarks/benchmark_report.md) -[![Python](https://img.shields.io/badge/Python-3.11-blue.svg)](https://www.python.org/) +[![Schema validation](https://github.com/CoreyLeath-code/SentinelAI/actions/workflows/data-validation.yml/badge.svg)](https://github.com/CoreyLeath-code/SentinelAI/actions/workflows/data-validation.yml) +[![Reference p95](https://img.shields.io/badge/Python_reference_p95-54.7_%C2%B5s-6f42c1)](benchmarks/benchmark_report.md) +[![Synthetic decision F1](https://img.shields.io/badge/synthetic_decision_F1-1.000-2ea44f)](benchmarks/benchmark_report.md) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE) ---- -- Detect model drift -- Monitor inference anomalies -- Track LLM hallucination risk -- Provide real-time observability -- Automate AI governance workflows -It combines statistical ML monitoring with LLM-powered incident intelligence. +## Abstract + +SentinelAI is a multi-service observability prototype for AI systems. Its directly implemented statistical component compares an expected and an observed histogram with Population Stability Index (PSI) and a Kolmogorov–Smirnov (KS) CDF distance, then raises a drift flag when either configured threshold is crossed. + +The versioned evidence measures a portable Python reference of that decision rule on seeded 32-bin synthetic histograms—not native C++ execution, HTTP latency, concurrent service load, or production drift-detection accuracy. The benchmark is a repeatable regression signal, not a claim of real-world model quality. + +## Formal decision rule + +Let $p=(p_1,...,p_B)$ and $q=(q_1,...,q_B)$ be expected and actual histogram-bin weights. The C++ engine computes + +\[ +\operatorname{PSI}(p,q) = \sum_{i=1}^{B}(q_i-p_i)\log\left(\frac{q_i}{p_i}\right), +\] + +summing only bins where both values are positive. It also forms normalized cumulative distributions: + +\[ +P_k=\sum_{i=1}^{k}\frac{p_i}{\sum_jp_j},\qquad Q_k=\sum_{i=1}^{k}\frac{q_i}{\sum_jq_j},\qquad \operatorname{KS}(p,q)=\max_{1\leq k\leq B}|P_k-Q_k|. +\] + +The implementation emits a drift event when + +\[ +\operatorname{drift}(p,q)=[\operatorname{PSI}(p,q)>0.20]\lor[\operatorname{KS}(p,q)>0.10]. +\] + +This logic is implemented in [drift-engine/drift_engine.cpp](drift-engine/drift_engine.cpp) and mirrored by [benchmarks/run_benchmark.py](benchmarks/run_benchmark.py). The benchmark supplies normalized positive bins; the service does not add smoothing to zero-valued PSI bins, so zero handling is an explicit limitation of the present implementation. + +For $B$ bins, PSI and KS each make one linear pass: $O(B)$ time and $O(1)$ auxiliary working memory (apart from the input vectors). The calculation is not a hypothesis test: the two thresholds are fixed decision settings, not calibrated p-values. +## Evidence snapshot + +| Measured quantity | Value | What it measures | +|---|---:|---| +| Timed reference evaluations | 20,000 | Seeded, single-process Python decision evaluations after 100 warm-ups | +| Reference latency (median / p95 / p99) | 39.700 / 54.700 / 76.200 µs | Per-decision Python reference latency | +| Reference throughput | 23,031.13 operations/s | Same Python reference workload | +| Peak traced memory | 0.623 MiB | Python allocations reported by tracemalloc | +| Synthetic decision precision / recall / F1 | 1.000 / 1.000 / 1.000 | 2,000 deliberately separated seeded perturbation cases | +| Decision thresholds | PSI > 0.20 or KS > 0.10 | Thresholds hard-coded by the C++ reference implementation | + +The full protocol, environment (CPython 3.12.13 on Windows 11), raw confusion matrix, and limitations are versioned in [benchmarks/latest.json](benchmarks/latest.json) and explained in [benchmarks/benchmark_report.md](benchmarks/benchmark_report.md). Reproduce the artifact with: + +```bash +python benchmarks/run_benchmark.py --output benchmarks/latest.json +``` + +## Research questions + +1. Under a leakage-free, labeled production dataset, how well do the fixed PSI/KS thresholds detect meaningful distribution shift? +2. How sensitive are false positives and false negatives to bin count, threshold selection, and zero-bin handling? +3. How does native C++ and end-to-end service latency scale with histogram size and concurrent requests? +4. Which operational metrics best distinguish feature drift from changes in data volume or service latency? ## Production Readiness Guide > This section is the portfolio audit entry point for **SentinelAI**. It describes an engineering promotion path; it is not a claim that the repository is already production-authorized. -[![CI](https://img.shields.io/github/actions/workflow/status/CoreyLeath-code/SentinelAI/ci.yml?branch=main&label=CI)](https://github.com/CoreyLeath-code/SentinelAI/actions) [![License](https://img.shields.io/github/license/CoreyLeath-code/SentinelAI)](https://github.com/CoreyLeath-code/SentinelAI/blob/main/LICENSE) - ### Architecture flowchart ```mermaid @@ -38,11 +80,11 @@ flowchart LR ### Quickstart and local validation -The supported local path should be reproducible from a clean checkout. The inferred stack for this repository is **C++**. +The repository uses Python services, a Go ingestion service, and a small C++ drift executable. Reproduce the portable evidence or build the statistical engine directly: ```bash -cmake -S . -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build -ctest --test-dir build --output-on-failure +python benchmarks/run_benchmark.py --output benchmarks/latest.json +g++ -std=c++17 drift-engine/drift_engine.cpp -o drift-engine/drift_engine ``` If the project uses external services, model artifacts, cloud credentials, or private data, start them through documented local fixtures or mocks. Never place secrets or identifiable records in the repository. @@ -73,7 +115,7 @@ Use the linked production-readiness issue for this repository as the checklist. ## 🏛️ Advanced Platform Architecture & Telemetry Decoupling -To guarantee enterprise-grade performance, SentinelAI enforces strict architectural separation between primary inference loops and the intelligent evaluation layers. +SentinelAI separates primary inference paths from telemetry and evaluation layers in its local architecture. [ Incoming User Query ] ───► [ Async Proxy Gateway ] ───► [ Downstream Application ] │ (Non-Blocking Telemetry Mirror)