Skip to content

feat: add a two-headed token tagger for intent spans and slots - #15

Merged
belambert merged 3 commits into
mainfrom
feat/encoder-tagger
Oct 25, 2026
Merged

belambert merged 3 commits into
mainfrom
feat/encoder-tagger

Conversation

@belambert

@belambert belambert commented Oct 4, 2026 •

Copy link
Copy Markdown
Owner

Adds a non-generative baseline: a BERT-style sequence tagger with two classification heads on a shared backbone, one tagging intent spans and one tagging slots, each as BIO labels over subword tokens.

Intents never nest and slots sit inside intents, so two flat tag layers capture the whole annotation. Decoding puts each slot under the intent span that contains it and returns the same Annotation the generative formats produce, so metrics.score grades it identically.

Changes

  • tagging.py: converts spans to BIO tags and back, using token offsets rather than a tokenizer, so it's testable without downloads.
    • Decoding is lenient: a stray I- tag starts a new span.
    • A slot in a gap stretches the nearest intent to cover it; one straddling an intent's edge is clipped to it.
    • A slot the schema doesn't allow under its intent is dropped.
  • tagger.py: the model, training via HF Trainer, prediction, and save/load. The loss is the sum of both heads' cross-entropies, with --slot-weight to rebalance it.
    • Any AutoModel backbone works, decoders included. The default is ModernBERT-base.
  • tag-train and tag-predict commands. tag-predict writes the same JSONL as predict, rendered through xml, so score and --detail work unchanged.
  • scripts/compare-taggers.sh: trains and scores ModernBERT, BERT, RoBERTa and Qwen3.5-0.8B. Qwen sees left context only, which is what that comparison tests.
  • tag-predict prints the average inference time per utterance after the scores, reusing the timing helpers from feat: report average inference time per utterance #16.
  • Tests and README updated.

Testing

  • uv run pytest passes (73 tests), along with black, isort and mypy.
  • Gold annotations from the full validation split survive conversion to tags and back with no losses, using the ModernBERT, BERT, RoBERTa and Qwen3.5 tokenizers.
  • A smoke run trained, saved, loaded, predicted and re-scored end to end.

To Do

  • Add the full-run test-set results to the README (ModernBERT-base: 83.17% exact match, 96.03% slot F1)

@belambert
belambert force-pushed the feat/encoder-tagger branch 2 times, most recently from 8e2f38d to e009261 Compare October 4, 2026 23:38
@belambert
belambert force-pushed the feat/encoder-tagger branch from e009261 to d08ce68 Compare October 23, 2026 02:47
@belambert
belambert force-pushed the feat/encoder-tagger branch 3 times, most recently from 5354822 to 7ae23ba Compare October 24, 2026 01:51
@belambert
belambert force-pushed the feat/encoder-tagger branch from 7ae23ba to e81ab6b Compare October 24, 2026 01:53
@belambert
belambert marked this pull request as ready for review October 25, 2026 16:39
@belambert
belambert merged commit 1a796e1 into main Oct 25, 2026
2 checks passed
@belambert
belambert deleted the feat/encoder-tagger branch October 25, 2026 16:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant