Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions .github/workflows/sol-discovery.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
name: Sol and discovery regression
on:
pull_request:
branches: [main]
push:
branches: [main]
permissions:
contents: read
jobs:
regression:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v6
- uses: actions/checkout@v6
with:
repository: NousResearch/hermes-agent
ref: 188a1a5efd098f067638032fea6e24c42d06178f
path: upstream-hermes
persist-credentials: false
- uses: actions/setup-python@v6
with:
python-version: '3.13'
- run: python -m pip install -e '.[dev]'
- name: Actual Sol pipeline and actual host-loader race regressions
env:
TT_HERMES_LOADER: ${{ github.workspace }}/upstream-hermes/plugins/plugin_loader.py
run: python -m pytest tests/test_gpt61_tokenizer.py tests/test_discovery_repair.py --junitxml=sol-discovery-tests.xml
- uses: actions/upload-artifact@v6
if: always()
with:
name: sol-discovery-verification
path: sol-discovery-tests.xml
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,12 @@
# Changelog

## 0.11.2 - 2026-10-05

- Added empirically validated GPT-6.1-Sol tokenizer compatibility, without overriding upstream mappings or guessing future model IDs.
- Fixed concurrent TT directory discovery and cross-profile module reuse through an explicit, source-guarded installer repair with exact backup.
- Added non-skipping Sol pipeline and real-loader concurrency CI regressions.
- Preserved call scope, protected context, exact recovery and strict final gates.

## 0.11.1 - 2026-10-04

- Fixed the Hermes live-compression adapter's missing `_coerce_threshold_tokens_cap`
Expand Down
2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "token-terminator"
version = "0.11.1"
version = "0.11.2"
edition = "2021"
rust-version = "1.78"
description = "Rust interoperability helpers for Token Terminator artifact identities and reduction invariants"
Expand Down
17 changes: 17 additions & 0 deletions MIGRATION.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,20 @@
# Migration to Token Terminator 0.11.2

Install with the managed Hermes interpreter:

```bash
python -m pip install --upgrade 'git+https://github.com/AronAxe/[email protected]'
python -m rtk_hermes_plus.cli install-context-engine
```

The explicit engine installer now requests the reviewed host-loader repair; inspect
its `discovery_repair` result before restarting the appropriate idle backend once.
A byte-exact host backup is retained. Unknown host sources are refused. No new keys,
manual Sol encoding override or data migration is required. See
[repair/rollback](docs/HERMES_DISCOVERY.md) and [counter evidence](docs/TOKENIZER_COMPAT.md).

---

# Migration to Token Terminator 0.11.1

## 0.11.0 to 0.11.1
Expand Down
35 changes: 25 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,10 @@

<p align="center">
<a href="https://github.com/AronAxe/Token-Terminator/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/AronAxe/Token-Terminator/actions/workflows/ci.yml/badge.svg"></a>
<a href="https://github.com/AronAxe/Token-Terminator/releases/tag/v0.11.1"><img alt="Release v0.11.1" src="https://img.shields.io/badge/release-v0.11.1-ef2b25"></a>
<a href="https://github.com/AronAxe/Token-Terminator/releases/tag/v0.11.2"><img alt="Release v0.11.2" src="https://img.shields.io/badge/release-v0.11.2-ef2b25"></a>
<a href="https://github.com/AronAxe/Token-Terminator/wiki/Jev-Semantic-Context-Gate" title="Optional JEV integration via OpenRouter or direct TypeSafe; off by default"><img alt="JEV-powered" src="https://img.shields.io/badge/JEV-powered-8b5cf6"></a>
<a href="https://github.com/AronAxe/Token-Terminator/wiki"><img alt="GitHub Wiki" src="https://img.shields.io/badge/docs-GitHub%20Wiki-181717?logo=github"></a>
<a href="https://crates.io/crates/token-terminator"><img alt="crates.io" src="https://img.shields.io/badge/crates.io-v0.11.1-orange?logo=rust"></a>
<a href="https://crates.io/crates/token-terminator"><img alt="crates.io" src="https://img.shields.io/badge/crates.io-v0.11.2-orange?logo=rust"></a>
<a href="https://docs.rs/token-terminator"><img alt="docs.rs" src="https://img.shields.io/docsrs/token-terminator?logo=docs.rs"></a>
<img alt="Python 3.10–3.13" src="https://img.shields.io/badge/Python-3.10%E2%80%933.13-3776AB?logo=python&logoColor=white">
<a href="LICENSE"><img alt="MIT License" src="https://img.shields.io/badge/License-MIT-22c55e.svg"></a>
Expand All @@ -27,6 +27,21 @@ Token Terminator is an agent-runtime optimization layer. It removes token bloat

Documentation: see the [GitHub Wiki](https://github.com/AronAxe/Token-Terminator/wiki) for quick start, architecture, configuration, recovery, security, troubleshooting, migration, and release notes.

## Patch v0.11.2: GPT-6.1-Sol and multiplex startup

**GPT-6.1-Sol now reaches TT/JEV context reduction without a manual tokenizer
override.** A provider-checked local compatibility mapping fills the missing
upstream model-name entry. Exact recovery and strict token/character gates remain.

**Concurrent TT discovery is repaired during explicit `install-context-engine`.**
For the reviewed affected Hermes loader, the installer makes a narrowly scoped,
backed-up host-file change: initialization is synchronized and profile module
caches are separated. Unknown host implementations are refused, not overwritten.
`install-dashboard` alone does not apply this repair.

[Release notes](docs/releases/v0.11.2.md) · [Tokenizer validation](docs/TOKENIZER_COMPAT.md) ·
[Discovery repair and rollback](docs/HERMES_DISCOVERY.md)

## Patch v0.11.1: Hermes live-compression compatibility

Repairs the missing `_coerce_threshold_tokens_cap` adapter method and live
Expand Down Expand Up @@ -159,12 +174,12 @@ See [JEV roles and implemented scope](docs/JEV_ROLES.md) and

## Select Token Terminator as the Hermes ContextEngine

Install **v0.11.1** in the Python environment that runs Hermes, then install its
Install **v0.11.2** in the Python environment that runs Hermes, then install its
managed adapter. Replace `python` below with that environment's interpreter:

```bash
python -m pip install --upgrade \
'git+https://github.com/AronAxe/[email protected].1'
'git+https://github.com/AronAxe/[email protected].2'
python -m rtk_hermes_plus.cli install-context-engine
```

Expand Down Expand Up @@ -322,9 +337,9 @@ The optional working-state block defaults to zero characters, even in `balanced`

## Install: Hermes Agent (turnkey)

Token Terminator 0.11.1 builds on v0.9.0 and replaces the earlier `rtk-hermes-plus` distribution. `token-terminator` and `rtk-hermes-plus` must not coexist because both own the `rtk_hermes_plus` Python import package.
Token Terminator 0.11.2 builds on v0.9.0 and replaces the earlier `rtk-hermes-plus` distribution. `token-terminator` and `rtk-hermes-plus` must not coexist because both own the `rtk_hermes_plus` Python import package.

This is the supported zero-glue installation: the repository already contains the Hermes hooks, slash command, recovery tool, and lifecycle accounting. The commands below pin the `v0.11.1` release tag.
This is the supported zero-glue installation: the repository already contains the Hermes hooks, slash command, recovery tool, and lifecycle accounting. The commands below pin the `v0.11.2` release tag.

### 1. Install RTK when using terminal rewriting

Expand All @@ -346,7 +361,7 @@ HERMES_PY="$HOME/.hermes/hermes-agent/venv/bin/python"
hermes plugins disable rtk-plus
"$HERMES_PY" -m pip uninstall -y rtk-hermes-plus token-terminator
"$HERMES_PY" -m pip install \
'git+https://github.com/AronAxe/[email protected].1'
'git+https://github.com/AronAxe/[email protected].2'
```

Windows example:
Expand All @@ -355,7 +370,7 @@ Windows example:
$HermesPy = "$env:LOCALAPPDATA\hermes\hermes-agent\venv\Scripts\python.exe"
hermes plugins disable rtk-plus
& $HermesPy -m pip uninstall -y rtk-hermes-plus token-terminator
& $HermesPy -m pip install "git+https://github.com/AronAxe/[email protected].1"
& $HermesPy -m pip install "git+https://github.com/AronAxe/[email protected].2"
```

`tiktoken` now ships with Token Terminator and is used automatically for supported OpenAI-family models, including common provider-qualified model IDs. Hugging Face `tokenizers` remains optional when pointing Token Terminator at a local `tokenizer.json`:
Expand Down Expand Up @@ -387,7 +402,7 @@ Installation and enablement are separate operations. Disabling affects subsequen
Rust agent hosts can use the supported `token-terminator` companion crate for the stable cross-language pieces of the Token Terminator contract:

```bash
cargo add [email protected].1
cargo add [email protected].2
```

```rust
Expand All @@ -409,7 +424,7 @@ Install the same distribution in the environment that owns your agent loop:

```bash
python -m pip install \
'git+https://github.com/AronAxe/[email protected].1'
'git+https://github.com/AronAxe/[email protected].2'
```

Then connect your runtime's tool-result and final-request hooks to `Runtime`. The adapter must map equivalent tools to Token Terminator's canonical names (`search_files`, `process`, and optionally `read_file`) and expose `Runtime.tool` to the model for exact recovery.
Expand Down
10 changes: 10 additions & 0 deletions SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -180,3 +180,13 @@ silently route a remote dashboard to this machine's localhost. Installation and
enable are separate and opt-in. See [DASHBOARD.md](docs/DASHBOARD.md) for limits,
accounting qualifications and rollback. The dashboard adds no service credential,
background training or publication.

## v0.11.2 explicit Hermes loader repair

The engine-install CLI can modify only the reviewed affected plugin_loader.py,
with a normalized source-hash guard, adjacent exact backup, and atomic replacement.
This resolves a race before TT plugin initialization; it is not a hidden import-time
patch. The wrapper targets only TT user ContextEngine loads and separates profile
module identities. Other plugin loads retain their host behavior. Unknown/modified
source is refused. No credentials, config, conversation or vault files are changed.
The library installer API requires explicit repair_discovery=True for this action.
6 changes: 6 additions & 0 deletions docs/CONTEXT_ENGINE.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,11 @@
# Token Terminator ContextEngine — v0.11.0

> **v0.11.2 install change:** The explicit engine installer also repairs the reviewed
> affected Hermes directory loader, with an exact source guard and backup. This is
> a narrow host-file repair for concurrent TT loading, not a replacement Hermes
> runtime. See the repository `docs/HERMES_DISCOVERY.md` before installation.


**Experimental and explicitly selected.** Available in v0.11.0; ordinary
middleware mode remains the default. Installing the managed adapter does not
change Hermes core, select a context engine, enable plugins, alter LCM data or
Expand Down
44 changes: 44 additions & 0 deletions docs/HERMES_DISCOVERY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# Hermes TT discovery repair

The reviewed Hermes shared directory loader inserts a new module into sys.modules
before running it, but treats a cached module with __file__ as ready. Concurrent
requests can receive that half-module. User engine module names also contain only
the plugin slug, not the profile path. These are initialization and isolation bugs.

## What 0.11.2 installs

The explicit `token-terminator install-context-engine` command requests the repair.
It patches only a reviewed loader file with normalized SHA-256
`402d3fd6f6077c344f15713ac3b793fbccbe0602181bceff7a23690270e3ff35`, checked against
Hermes `188a1a5efd098f067638032fea6e24c42d06178f` and the installed affected source.

The appended wrapper applies only to token-terminator in the user ContextEngine
namespace. A lock covers the original load until initialization finishes. Other
profiles receive path-derived namespaces; unrelated plugins retain original
behavior. Recursive same-thread loads return unavailable, never a partial module.
Waiting is bounded to ten seconds. Failed imports remain retryable.

No plugin-only hook can repair a lookup that happens before that plugin starts.
This release therefore makes the host-file change explicit. It is not a general
Hermes plugin-loader rewrite, new runtime dependency, or replacement context engine.

The original bytes are backed up beside the source as
`plugin_loader.py.tt-before-0.11.2`. The installer refuses unknown source versions,
modified patches, symlinks, mismatching backups and concurrent source changes.
Package import and ordinary middleware do not write host files. The Python
`install_context_engine` API defaults to `repair_discovery=False`; callers that
want the explicit CLI behavior pass `repair_discovery=True`.

## Verification and rollback

`tests/test_discovery_repair.py` deterministically holds one initialization open
while another lookup arrives. It also tests 64 parallel calls and profile isolation.
Set TT_HERMES_LOADER to a copy or checkout of the actual reviewed plugin_loader.py
to test its real implementation; tests only modify temporary files.

After installation, restart the correct idle backend once and verify discovery and
provider-request measurements. Do not delete history, vaults or unrelated plugins.
To roll back, stop the affected backend and restore the adjacent original backup
only after checking that no subsequent Hermes changes would be overwritten. If
Hermes upgrades its loader, review that new implementation rather than forcing this
patch onto it. A successful health check alone is not proof of token reduction.
38 changes: 38 additions & 0 deletions docs/TOKENIZER_COMPAT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# GPT-6.1-Sol tokenizer compatibility

Upstream tiktoken's model-name lookup did not contain GPT-6.1-Sol at the time of this
repair. TT 0.11.1 consequently returned no token count and its ContextEngine exited
before JEV. Version 0.11.2 adds a local compatibility mapping for that exact model,
not a blanket rule for future GPT versions. The upstream library remains unmodified.

The supported provider prefixes are bare, openai/, openai:, openai-codex/ and
openrouter/openai/. Cache keys retain the original provider identity. A recognized
upstream tokenizer takes precedence over TT's compatibility mapping. Local explicit
encoding/Hugging Face configuration keeps its existing behavior.

## Empirical validation

`tests/fixtures/gpt61-token-counts.json` contains five synthetic public inputs and
actual responses from https://api.openai.com/v1/responses/input_tokens on 2026-10-05.
All content counts matched o200k_base, with a constant six-token message framing:

| Case | Provider | Local content | Framing |
|---|---:|---:|---:|
| English | 13 | 7 | 6 |
| Multilingual and emoji | 330 | 324 | 6 |
| Code and values | 633 | 627 | 6 |
| JSON records | 610 | 604 | 6 |
| Literal markers | 204 | 198 | 6 |

This is an empirically validated compatibility mapping, not an official upstream
model declaration or universal proof. No private conversations or generation calls
were involved. The runtime does not call a remote counter or need a new key.

TT reports `tiktoken:compat:gpt-6.1-sol:o200k_base` (with the normalized model for
qualified routes). Its gate measures complete canonical provider-request JSON;
provider-internal framing, image tokenization and invoices remain outside that
measurement claim. Strict token AND character decrease, protected context and
exact recovery are unchanged. The full pipeline regression must actually reach
JEV and reduce the request, not merely return a numeric count.

Reference: https://developers.openai.com/api/docs/guides/token-counting
58 changes: 58 additions & 0 deletions docs/releases/v0.11.2.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
# Token Terminator v0.11.2 — Sol counting and reliable TT discovery

This repair addresses two blockers left in 0.11.1. It does not change JEV keys,
semantic policies, the evidence vault, Context IR, or the main-call-only boundary.

## GPT-6.1-Sol

The exact `gpt-6.1-sol` ID and known OpenAI/Codex/OpenRouter-qualified forms now
resolve to `o200k_base` when the installed tiktoken model-name table lacks the ID.
An upstream mapping still wins, and an explicit operator encoding remains supported.
Unrelated providers, invented versions and sibling names are not inferred.

Five synthetic inputs were checked against OpenAI's documented input-token counter:
English, multilingual/emoji text, code/numbers, structured JSON and literal special
markers. All matched local content counts plus the same six-token message framing.
The recorded corpus distinguishes o200k_base from cl100k_base. This is empirical
compatibility evidence, not an upstream tiktoken declaration or proof about every
possible input. TT still measures complete canonical request JSON, not hidden
provider framing or billed tokens. No generation requests were used in that check.

## Concurrent Hermes discovery

The affected shared loader can return a module before its initialization completes,
and its slug-only cache can reuse one profile's module for another. TT's explicit
`install-context-engine` command now applies a source-hash-guarded repair to the
reviewed affected loader. TT loads are synchronized, recursive discovery cannot
consume a partial module, and each resolved profile path has a distinct namespace.
Other plugins use the original loader. Failed initialization can be retried.

This is a narrow host-file compatibility patch, not merely a plugin import change:
the race occurs before the plugin can run. An adjacent byte-exact backup is kept.
Unknown or modified host implementations are refused rather than overwritten.
The Python installer API and `install-dashboard` do not patch the host by default.
No changes occur on package import, and configuration/history are not rewritten.

## Install

Use the interpreter/environment actually loaded by the managed Hermes launcher:

```bash
python -m pip install --upgrade 'git+https://github.com/AronAxe/[email protected]'
python -m rtk_hermes_plus.cli install-context-engine
```

The second command reports `discovery_repair`; for an affected installed host it
must say `repaired` or `already_repaired`. Then restart the appropriate idle backend
through its supervisor once. A package download alone cannot repair a running process.
Existing explicit engine selection and JEV/IR opt-ins remain unchanged.

Regression checks exercise the actual GPT-6.1-Sol model name without encoding
overrides, JEV selection, strict complete-request reduction and exact recovery.
Discovery checks include controlled overlapping initialization, 64 simultaneous
loads, profile separation, recursive loading, failed-load retry and safe installation.
A dedicated CI job runs those same tests against the actual pinned Hermes loader
source, not an optional skipped module. Full CI results are recorded on the PR.

The Rust companion is version-aligned only. Existing tags, Terminator artwork and
schematics remain unchanged. See docs/TOKENIZER_COMPAT.md and docs/HERMES_DISCOVERY.md.
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"

[project]
name = "token-terminator"
version = "0.11.1"
version = "0.11.2"
description = "Fail-open token reduction, exact recovery, and request compilation for Hermes Agent"
readme = "README.md"
requires-python = ">=3.10"
Expand Down
2 changes: 1 addition & 1 deletion src/rtk_hermes_plus/_version.py
Original file line number Diff line number Diff line change
@@ -1 +1 @@
__version__ = "0.11.1"
__version__ = "0.11.2"
8 changes: 7 additions & 1 deletion src/rtk_hermes_plus/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,13 @@ def run(args: argparse.Namespace) -> int:
if args.command in {"install-context-engine", "install-dashboard"}:
from .engine_install import install_context_engine

_emit(install_context_engine(args.hermes_home), as_json=args.json)
_emit(
install_context_engine(
args.hermes_home,
repair_discovery=args.command == "install-context-engine",
),
as_json=args.json,
)
return 0
config, store, graph, compiler = _runtime()
if args.command == "status":
Expand Down
Loading
Loading