From dc76ea1ba25b9a68a5267e581367a6d8099a75a8 Mon Sep 17 00:00:00 2001 From: Corey Leath Date: Wed, 12 Aug 2026 20:46:34 -0400 Subject: [PATCH 1/4] Document drift metrics and mathematical decision logic --- README.md | 75 +++++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 59 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index 8e49683..188e19c 100644 --- a/README.md +++ b/README.md @@ -1,28 +1,71 @@ -SentinelAI -image +

+ SentinelAI interface +

-# SentinelAI — Enterprise AI Reliability & Governance Platform +# SentinelAI — Reproducible Drift-Monitoring Reference System [![CI](https://github.com/CoreyLeath-code/SentinelAI/actions/workflows/ci-cd.yml/badge.svg)](https://github.com/CoreyLeath-code/SentinelAI/actions/workflows/ci-cd.yml) -[![Benchmarks](https://github.com/CoreyLeath-code/SentinelAI/actions/workflows/benchmarks.yml/badge.svg)](https://github.com/CoreyLeath-code/SentinelAI/actions/workflows/benchmarks.yml) +[![Research benchmark](https://github.com/CoreyLeath-code/SentinelAI/actions/workflows/benchmarks.yml/badge.svg)](https://github.com/CoreyLeath-code/SentinelAI/actions/workflows/benchmarks.yml) [![Security](https://github.com/CoreyLeath-code/SentinelAI/actions/workflows/security.yml/badge.svg)](https://github.com/CoreyLeath-code/SentinelAI/actions/workflows/security.yml) [![SAST](https://github.com/CoreyLeath-code/SentinelAI/actions/workflows/sast.yml/badge.svg)](https://github.com/CoreyLeath-code/SentinelAI/actions/workflows/sast.yml) -[![Schema Validation](https://github.com/CoreyLeath-code/SentinelAI/actions/workflows/data-validation.yml/badge.svg)](https://github.com/CoreyLeath-code/SentinelAI/actions/workflows/data-validation.yml) -[![Release](https://github.com/CoreyLeath-code/SentinelAI/actions/workflows/release.yml/badge.svg)](https://github.com/CoreyLeath-code/SentinelAI/actions/workflows/release.yml) -[![Benchmark](https://img.shields.io/badge/reference%20p95-54.7%20%C2%B5s-6f42c1)](benchmarks/benchmark_report.md) -[![Throughput](https://img.shields.io/badge/reference%20throughput-23.0k%20ops%2Fs-2ea44f)](benchmarks/benchmark_report.md) -[![Python](https://img.shields.io/badge/Python-3.11-blue.svg)](https://www.python.org/) +[![Schema validation](https://github.com/CoreyLeath-code/SentinelAI/actions/workflows/data-validation.yml/badge.svg)](https://github.com/CoreyLeath-code/SentinelAI/actions/workflows/data-validation.yml) +[![Reference p95](https://img.shields.io/badge/Python_reference_p95-54.7_%C2%B5s-6f42c1)](benchmarks/benchmark_report.md) +[![Synthetic decision F1](https://img.shields.io/badge/synthetic_decision_F1-1.000-2ea44f)](benchmarks/benchmark_report.md) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE) ---- -- Detect model drift -- Monitor inference anomalies -- Track LLM hallucination risk -- Provide real-time observability -- Automate AI governance workflows -It combines statistical ML monitoring with LLM-powered incident intelligence. +## Abstract + +SentinelAI is a multi-service observability prototype for AI systems. Its directly implemented statistical component compares an expected and an observed histogram with Population Stability Index (PSI) and a Kolmogorov–Smirnov (KS) CDF distance, then raises a drift flag when either configured threshold is crossed. + +The versioned evidence measures a portable Python reference of that decision rule on seeded 32-bin synthetic histograms—not native C++ execution, HTTP latency, concurrent service load, or production drift-detection accuracy. The benchmark is a repeatable regression signal, not a claim of real-world model quality. + +## Formal decision rule + +Let $p=(p_1,...,p_B)$ and $q=(q_1,...,q_B)$ be expected and actual histogram-bin weights. The C++ engine computes + +$ +PSI(p,q) = sum_(i=1)^B (q_i-p_i) log(q_i/p_i), +$ + +summing only bins where both values are positive. It also forms normalized cumulative distributions: + +$ +P_k=sum_(i=1)^k p_i/(sum_j p_j), Q_k=sum_(i=1)^k q_i/(sum_j q_j), KS(p,q)=max_(1<=k<=B)|P_k-Q_k|. +$ + +The implementation emits a drift event when + +$ +drift(p,q) = [PSI(p,q)>0.20] OR [KS(p,q)>0.10]. +$ + +This logic is implemented in [drift-engine/drift_engine.cpp](drift-engine/drift_engine.cpp) and mirrored by [benchmarks/run_benchmark.py](benchmarks/run_benchmark.py). The benchmark supplies normalized positive bins; the service does not add smoothing to zero-valued PSI bins, so zero handling is an explicit limitation of the present implementation. + +For $B$ bins, PSI and KS each make one linear pass: $O(B)$ time and $O(1)$ auxiliary working memory (apart from the input vectors). The calculation is not a hypothesis test: the two thresholds are fixed decision settings, not calibrated p-values. + +## Evidence snapshot + +| Measured quantity | Value | What it measures | +|---|---:|---| +| Timed reference evaluations | 20,000 | Seeded, single-process Python decision evaluations after 100 warm-ups | +| Reference latency (median / p95 / p99) | 39.700 / 54.700 / 76.200 µs | Per-decision Python reference latency | +| Reference throughput | 23,031.13 operations/s | Same Python reference workload | +| Peak traced memory | 0.623 MiB | Python allocations reported by tracemalloc | +| Synthetic decision precision / recall / F1 | 1.000 / 1.000 / 1.000 | 2,000 deliberately separated seeded perturbation cases | +| Decision thresholds | PSI > 0.20 or KS > 0.10 | Thresholds hard-coded by the C++ reference implementation | + +The full protocol, environment (CPython 3.12.13 on Windows 11), raw confusion matrix, and limitations are versioned in [benchmarks/latest.json](benchmarks/latest.json) and explained in [benchmarks/benchmark_report.md](benchmarks/benchmark_report.md). Reproduce the artifact with: + +```bash +python benchmarks/run_benchmark.py --output benchmarks/latest.json +``` +## Research questions +1. Under a leakage-free, labeled production dataset, how well do the fixed PSI/KS thresholds detect meaningful distribution shift? +2. How sensitive are false positives and false negatives to bin count, threshold selection, and zero-bin handling? +3. How does native C++ and end-to-end service latency scale with histogram size and concurrent requests? +4. Which operational metrics best distinguish feature drift from changes in data volume or service latency? ## Production Readiness Guide > This section is the portfolio audit entry point for **SentinelAI**. It describes an engineering promotion path; it is not a claim that the repository is already production-authorized. From 403e536cc561bd11858cb6627b646a5053dbe29a Mon Sep 17 00:00:00 2001 From: Corey Leath Date: Wed, 12 Aug 2026 20:46:48 -0400 Subject: [PATCH 2/4] Render README drift equations with TeX notation --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 188e19c..7705aa2 100644 --- a/README.md +++ b/README.md @@ -24,19 +24,19 @@ The versioned evidence measures a portable Python reference of that decision rul Let $p=(p_1,...,p_B)$ and $q=(q_1,...,q_B)$ be expected and actual histogram-bin weights. The C++ engine computes $ -PSI(p,q) = sum_(i=1)^B (q_i-p_i) log(q_i/p_i), +\operatorname{PSI}(p,q) = \sum_{i=1}^{B}(q_i-p_i)\log\left(\frac{q_i}{p_i}\right), $ summing only bins where both values are positive. It also forms normalized cumulative distributions: $ -P_k=sum_(i=1)^k p_i/(sum_j p_j), Q_k=sum_(i=1)^k q_i/(sum_j q_j), KS(p,q)=max_(1<=k<=B)|P_k-Q_k|. +P_k=\sum_{i=1}^{k}\frac{p_i}{\sum_jp_j},\qquad Q_k=\sum_{i=1}^{k}\frac{q_i}{\sum_jq_j},\qquad \operatorname{KS}(p,q)=\max_{1\leq k\leq B}|P_k-Q_k|. $ The implementation emits a drift event when $ -drift(p,q) = [PSI(p,q)>0.20] OR [KS(p,q)>0.10]. +\operatorname{drift}(p,q)=[\operatorname{PSI}(p,q)>0.20]\lor[\operatorname{KS}(p,q)>0.10]. $ This logic is implemented in [drift-engine/drift_engine.cpp](drift-engine/drift_engine.cpp) and mirrored by [benchmarks/run_benchmark.py](benchmarks/run_benchmark.py). The benchmark supplies normalized positive bins; the service does not add smoothing to zero-valued PSI bins, so zero handling is an explicit limitation of the present implementation. From 7a7f5bf8258449cec9115c4db4319e8b1ae9d445 Mon Sep 17 00:00:00 2001 From: Corey Leath Date: Wed, 12 Aug 2026 20:47:12 -0400 Subject: [PATCH 3/4] Clarify reproducibility and README evidence scope --- README.md | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 7705aa2..8f503fc 100644 --- a/README.md +++ b/README.md @@ -66,12 +66,11 @@ python benchmarks/run_benchmark.py --output benchmarks/latest.json 2. How sensitive are false positives and false negatives to bin count, threshold selection, and zero-bin handling? 3. How does native C++ and end-to-end service latency scale with histogram size and concurrent requests? 4. Which operational metrics best distinguish feature drift from changes in data volume or service latency? + ## Production Readiness Guide > This section is the portfolio audit entry point for **SentinelAI**. It describes an engineering promotion path; it is not a claim that the repository is already production-authorized. -[![CI](https://img.shields.io/github/actions/workflow/status/CoreyLeath-code/SentinelAI/ci.yml?branch=main&label=CI)](https://github.com/CoreyLeath-code/SentinelAI/actions) [![License](https://img.shields.io/github/license/CoreyLeath-code/SentinelAI)](https://github.com/CoreyLeath-code/SentinelAI/blob/main/LICENSE) - ### Architecture flowchart ```mermaid @@ -81,11 +80,11 @@ flowchart LR ### Quickstart and local validation -The supported local path should be reproducible from a clean checkout. The inferred stack for this repository is **C++**. +The repository uses Python services, a Go ingestion service, and a small C++ drift executable. Reproduce the portable evidence or build the statistical engine directly: ```bash -cmake -S . -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build -ctest --test-dir build --output-on-failure +python benchmarks/run_benchmark.py --output benchmarks/latest.json +g++ -std=c++17 drift-engine/drift_engine.cpp -o drift-engine/drift_engine ``` If the project uses external services, model artifacts, cloud credentials, or private data, start them through documented local fixtures or mocks. Never place secrets or identifiable records in the repository. @@ -116,7 +115,7 @@ Use the linked production-readiness issue for this repository as the checklist. ## 🏛️ Advanced Platform Architecture & Telemetry Decoupling -To guarantee enterprise-grade performance, SentinelAI enforces strict architectural separation between primary inference loops and the intelligent evaluation layers. +SentinelAI separates primary inference paths from telemetry and evaluation layers in its local architecture. [ Incoming User Query ] ───► [ Async Proxy Gateway ] ───► [ Downstream Application ] │ (Non-Blocking Telemetry Mirror) From e66a6d1101f3c8cbc678f5d0b3e5bdf81de93dcc Mon Sep 17 00:00:00 2001 From: Corey Leath Date: Wed, 12 Aug 2026 20:47:36 -0400 Subject: [PATCH 4/4] Use supported display-math delimiters in README --- README.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 8f503fc..5509e42 100644 --- a/README.md +++ b/README.md @@ -23,21 +23,21 @@ The versioned evidence measures a portable Python reference of that decision rul Let $p=(p_1,...,p_B)$ and $q=(q_1,...,q_B)$ be expected and actual histogram-bin weights. The C++ engine computes -$ +\[ \operatorname{PSI}(p,q) = \sum_{i=1}^{B}(q_i-p_i)\log\left(\frac{q_i}{p_i}\right), -$ +\] summing only bins where both values are positive. It also forms normalized cumulative distributions: -$ +\[ P_k=\sum_{i=1}^{k}\frac{p_i}{\sum_jp_j},\qquad Q_k=\sum_{i=1}^{k}\frac{q_i}{\sum_jq_j},\qquad \operatorname{KS}(p,q)=\max_{1\leq k\leq B}|P_k-Q_k|. -$ +\] The implementation emits a drift event when -$ +\[ \operatorname{drift}(p,q)=[\operatorname{PSI}(p,q)>0.20]\lor[\operatorname{KS}(p,q)>0.10]. -$ +\] This logic is implemented in [drift-engine/drift_engine.cpp](drift-engine/drift_engine.cpp) and mirrored by [benchmarks/run_benchmark.py](benchmarks/run_benchmark.py). The benchmark supplies normalized positive bins; the service does not add smoothing to zero-valued PSI bins, so zero handling is an explicit limitation of the present implementation.