token-visualization

🐼 Steer your LLMs and agents at the token level: A web tool for token visualization & control, model inspection, and LLM & agent trajectory annotation.

Rust CLI and library that flags low-confidence spans in LLM answers from token logprobs (offline or live via any OpenAI-compatible API) and renders per-token confidence as terminal, HTML or Markdown.

Explore how Hugging Face tokenizers work across models like LLaMA, PHI-3, and StarCoder2. Includes examples for encoding, decoding, chat formatting, and token visualization. Ideal for understanding text preprocessing in LLMs.

Token Pie runs locally on each developer's machine, reads telemetry Copilot already writes, Nothing leaves the machine. No server, no account, no telemetry of our own.