Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,21 @@ All notable changes to CyberAI are documented here.

### Added

- **The prose that states a measured count is held against that count.** Two
numbers describe where the typing scope ends, and a test walked the imports
and red when either moved in the tree. It never read the page that states
them, so a move was paid for by rewriting the constant while the sentence
quoting it stayed behind -- green, silent and wrong. Three numbers on that
page had drifted exactly that way before anyone looked, and the guard's own
docstring had drifted with them, claiming a scope size the constant beside
it contradicted. The new guard runs the other way round: the constant is the
standard and the prose is what is checked, which is sound only because the
older test holds that constant against the tree. It reads the constants by
importing that test rather than recomputing the crossings, so the two cannot
answer from different methods. Dated sentences stay out of reach on purpose
-- a count that says when it was measured is a record, not a claim about
today.

- **A finding's confidence reaches the machine-readable export.** The field
carries what the agents did not fully evidence: an out-of-band attempt with
no callback, a redteam report the pipeline could not confirm. Both renderers
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
![Python](https://img.shields.io/badge/python-3.11%20%7C%203.12%20%7C%203.13%20%7C%203.14-blue)
![License](https://img.shields.io/badge/license-Apache_2.0-blue)
![Version](https://img.shields.io/badge/version-v1.7.0-brightgreen)
![Tests](https://img.shields.io/badge/tests-3241%20collected-brightgreen)
![Tests](https://img.shields.io/badge/tests-3245%20collected-brightgreen)
![Mypy](https://img.shields.io/badge/mypy-strict%3A%20118%2F175%20modules-blue)
![LLM](https://img.shields.io/badge/LLM-OpenAI%20%7C%20Anthropic%20%7C%20Ollama-blueviolet)
![Air-Gapped](https://img.shields.io/badge/air--gapped-ready-success)
Expand Down
2 changes: 1 addition & 1 deletion blog/launch-post-draft.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ not exist yet, it says so.
CyberAI is a multi-agent offensive-security platform: eight agents (recon,
intel, exploit, report, planner, mcp-scan, redteam, web3) run a typed, audited
pipeline over a shared knowledge base.
3241 tests collected under the gated selection run before every commit, with the
3245 tests collected under the gated selection run before every commit, with the
slow and smoke tests deselected there and run separately, `mypy --strict`
clean over 118 of 175 modules, Apache-2.0.

Expand Down
23 changes: 14 additions & 9 deletions docs/architecture/typing-scope.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,14 @@ than repeatable as written; the mechanism it showed is unchanged. So a name
crossing the boundary is typed by a module nothing checks, and adding a
module to the scope costs its import closure rather than its own error count
-- `mypy` on three single-error modules alone reports 90 errors across 26
files. The crossing is measured on every run by the test named below, which
reds when either count moves, so the paragraph cannot drift away from it. Both numbers are measured, not chosen, and they
are measured on the runner: the typecheck job prints them on every run.
files. Two tests hold the pair above, and they hold different things.
`tests/architecture/test_typing_scope_boundary_is_declared.py` walks the
imports and reds when either count moves in the tree.
`tests/architecture/test_the_prose_quotes_the_measured_edge.py` reds when
the sentence above stops quoting what that one holds, which is the drift
that fixing the first test alone used to leave behind. Both numbers are
measured, not chosen, and they are measured on the runner: the typecheck
job prints them on every run.

Four modules were carried into the scope rather than found there. The call at
`cyberai/cli/bench.py` that reported `Cannot call function of unknown type`
Expand Down Expand Up @@ -122,9 +127,9 @@ mypy --strict --python-version 3.11 --ignore-missing-imports cyberai

The cache purge is not decoration. A scoped run that reuses a cache left by a
wider run re-emits errors for modules outside the scope, and reports them as
if the declared set were dirty. Measured: cold cache gives `Success` on 97
modules, the same command after a full-package run gives 239 errors in 54
files, and every one of those files lies outside the scope.
if the declared set were dirty. Measured 2026-10-30: cold cache gives
`Success` on 118 modules, the same command after a full-package run gives
216 errors in 44 files, and every one of those files lies outside the scope.

`scripts/typing_scope_drift.py` runs the same partition without the hazard.
It reads the flags from `[tool.mypy]` instead of repeating them, so this page
Expand All @@ -133,9 +138,9 @@ wide run a cache directory of its own instead of purging the shared one.

## What the numbers depend on

The partition moves with the checker. Measured on mypy 1.19.1 the clean side
holds 98 modules; a later release moved it by one module in the other
direction. The dev extra therefore bounds the checker rather than naming a
The partition moves with the checker. Measured 2026-10-30 on mypy 1.19.1 the
clean side holds 118 modules; a later release moved it by one module in the
other direction. The dev extra therefore bounds the checker rather than naming a
floor and admitting every future release.

The checker is not the only version these counts depend on, and on 2026-09-18
Expand Down
84 changes: 84 additions & 0 deletions tests/architecture/test_the_prose_quotes_the_measured_edge.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
"""The prose beside a measured count must quote that count.

`test_typing_scope_boundary_is_declared.py` holds two numbers against the
tree and reds when either moves. What it does not do is read the page. A
move therefore reds there, the constant gets rewritten by whoever moved it,
and the sentence in docs/architecture/typing-scope.md that states the same
two numbers stays behind -- green, silent and wrong. Three numbers on that
page drifted exactly that way before anyone noticed.

The direction of this check is the opposite of the one next door, and that
is deliberate. There the tree is the standard and the prose is not read, so
the test cannot agree with itself. Here the constant is the standard and the
prose is what is checked, which is sound only because the constant is itself
held against the tree by that other test. Remove it and this one measures
nothing.

Dated counts are out of reach on purpose. A sentence that says when it was
measured is a record, not a claim about today, and pinning one would red on
every release of the checker.
"""

import importlib.util
import pathlib
import re
import sys

_ROOT = pathlib.Path(__file__).resolve().parents[2]
_PAGE = _ROOT / "docs" / "architecture" / "typing-scope.md"
_NEIGHBOUR = _ROOT / "tests" / "architecture" / "test_typing_scope_boundary_is_declared.py"


def _neighbour_constants() -> tuple[int, int]:
spec = importlib.util.spec_from_file_location("_boundary_guard", _NEIGHBOUR)
assert spec is not None and spec.loader is not None, f"cannot load {_NEIGHBOUR}"
module = importlib.util.module_from_spec(spec)
sys.modules[spec.name] = module
spec.loader.exec_module(module)
return int(module._EXPECTED_CROSSERS), int(module._EXPECTED_REACHED)


_CURRENT = re.compile(
r"(\d+) import a module outside it at module level,\s*\n"
r"and between them they reach (\d+) such modules"
)


def test_the_page_states_the_edge_in_one_undated_sentence() -> None:
"""One match, so a second phrasing cannot drift unseen behind the first."""
found = _CURRENT.findall(_PAGE.read_text(encoding="utf-8"))
assert len(found) == 1, f"the page states the current edge {len(found)} times, not once"


def test_the_page_quotes_the_edge_the_neighbour_holds() -> None:
found = _CURRENT.search(_PAGE.read_text(encoding="utf-8"))
assert found, "the page no longer states the edge in the form this guard reads"
quoted = (int(found.group(1)), int(found.group(2)))
assert quoted == _neighbour_constants(), (
f"the page says {quoted} while the guard beside it holds {_neighbour_constants()}"
)


def test_the_page_names_tests_that_exist() -> None:
"""The page points the reader at two files by path.

A renamed or deleted guard leaves the sentence pointing at nothing, and
the reader who follows it finds out before the page does. The earlier
wording said "the test named below" and named none, which is the same
break with nothing to check it against.
"""
named = re.findall(r"`(tests/architecture/\S+?\.py)`", _PAGE.read_text(encoding="utf-8"))
assert named, "the page names no guard for the counts it states"
missing = [path for path in named if not (_ROOT / path).is_file()]
assert not missing, f"the page names guards that do not exist: {missing}"


def test_the_neighbour_prose_quotes_its_own_constants() -> None:
"""The guard's own docstring states the counts and drifted away from them."""
crossers, _ = _neighbour_constants()
head = _NEIGHBOUR.read_text(encoding="utf-8").split('"""')[1]
match = re.search(r"(\d+) of them import a module outside the scope", head)
assert match, "the guard's docstring no longer states its crosser count in digits"
assert int(match.group(1)) == crossers, (
f"the docstring says {match.group(1)} crossers while the constant holds {crossers}"
)
8 changes: 4 additions & 4 deletions tests/architecture/test_typing_scope_boundary_is_declared.py
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
"""The typing scope has an edge, and until now nothing said where.

`[tool.mypy] files` resolves to 107 modules and the run reports `Success` on
exactly 107, which reads like a guarantee about those modules and is one only
up to the boundary. Twenty-two of them import a module outside the scope at
`[tool.mypy] files` resolves to 118 modules and the run reports `Success` on
exactly 118, which reads like a guarantee about those modules and is one only
up to the boundary. 16 of them import a module outside the scope at
module level. Both numbers were 95 and 19 when this was written and are
restated here on 2026-09-23; the assertions below are what holds them. mypy follows such an import to resolve the name and stays silent about
restated here on 2026-10-30; the assertions below are what holds them. mypy follows such an import to resolve the name and stays silent about
what it finds: appending an unannotated function to `cyberai/core/config.py`,
which is outside the scope and imported from inside it, changed nothing about
the output on a cold cache. A name crossing the edge is therefore typed by a
Expand Down
Loading