Multi-intent NLU with generative language models. A single utterance can carry several intents, each with its own slots:
book a table in ames and play jazz
Instead of predicting intents and BIO slot tags separately, this repo has a generative model write the whole annotation down in one pass — and compares several ways of writing it down, since the choice of output format is itself a modeling decision. The first is inline XML: the model copies the utterance back and adds tags around the intents and slots it finds.
<BookRestaurant>book a table in <city>ames</city></BookRestaurant> and <PlayMusic>play <genre>jazz</genre></PlayMusic>
Every format parses back to the same span-based Annotation, so one parser-
agnostic scorer grades them all and the numbers are directly comparable — across
formats, and between few-shot prompting and LoRA fine-tuning.
The default dataset is blambert/mixsnips-intent-spans — MixSNIPS utterances with 7 intents and 39 slots, one to three intents each, annotated as character offsets rather than in any one output format.
uv sync
Models and datasets come from the Hugging Face Hub; run hf auth login (or set
HF_TOKEN) if either is private.
A format owns both halves of the contract — the instructions telling the model
what to emit, and the parser reading a generation back into spans — so adding an
approach means adding one Format and nothing else. --format picks one, and
multi-nlu formats lists what is implemented.
| Format | Status | The model emits |
|---|---|---|
xml |
done | the utterance copied verbatim with inline XML tags added |
dict |
done | [{"PlayMusic": {"genre": …}}], slots copied verbatim; a repeated slot label collapses |
xml and dict both exist today. The token tagger baseline
sits outside this interface, since it classifies tokens instead of generating
text.
Prompt a base model few-shot and score it:
uv run multi-nlu predict -m Qwen/Qwen3.5-2B --shots 8 --split test -n 200
Fine-tune a LoRA adapter, then evaluate it:
uv run multi-nlu train -m Qwen/Qwen3.5-2B -o runs/qwen3.5-2b
uv run multi-nlu predict -m Qwen/Qwen3.5-2B --adapter runs/qwen3.5-2b --split test
Both commands take --format, and training an adapter on a format evaluates
through the same one:
uv run multi-nlu train --format xml -o runs/xml
Training logs to Weights & Biases by default (project multi-nlu), tracking
the hyperparameters and loss curves; pass --no-wandb to disable it and
--wandb-project to log elsewhere.
Predictions can be saved and re-scored without rerunning the model; each row
records the format that produced it, so score needs no flags:
uv run multi-nlu predict ... -o preds.jsonl
uv run multi-nlu score preds.jsonl
predict -o also writes preds.meta.json beside the predictions: the model,
adapter, format, sample, batch size, device and inference time. score reads it
to report the time per utterance again. The time is the whole batched
generation, excluding model loading, so it only compares across runs on the
same device and batch size.
Both predict and score take --detail (-d) to print the examples
themselves — errors for the ones that went wrong, all for every one:
uv run multi-nlu score preds.jsonl -d errors
Each example shows its status and a diff of its intents and slots. Matches
collapse to a count so the errors stand out, one per line; - marks something
gold had and the prediction missed, + something the prediction invented.
Reading a - against the + under it usually names the mistake.
A correct example matches gold on every intent and slot, partial gets some
of them, wrong none, and malformed did not parse at all. Each status has its
own colour on a terminal — green, yellow, red and magenta — so a long run can be
skimmed. Only a malformed
example prints the generation itself, since nothing could be read out of it;
anywhere else the marks and the diff below say more than the raw output does.
A format like xml asks the model to copy the utterance back verbatim and only
add tags, so when it paraphrases instead, every character offset after the
change stops lining up with gold. Those examples get a two-line word diff of
the utterance against the text the model actually copied. The text the two
share is dimmed, so what stands out is the words that differ — green for what
the utterance says, red for what the prediction put there:
want ... in panama for two people and give the current book a zero of 6
copied ... in panama for two people and please give the current book a zero of 6
Colour is dropped when the output is not a terminal, so piping to a file or a pager stays readable.
Other commands: show prints annotated examples in a format and the system
prompt, formats and schemas list what is bundled, derive-schema builds a
new schema. Every command takes --help.
scripts/compare-few-shot.sh prompts each instruct Qwen3.5 model under 40B with
8 shots, once per format on the same sample, printing each one's scores in turn
and saving its predictions under runs/few-shot/ for re-scoring with score:
./scripts/compare-few-shot.sh
scripts/compare-finetuned.sh does the same for LoRA fine-tunes, training an
adapter per model and format under runs/ and then evaluating it:
./scripts/compare-finetuned.sh
Both take model ids to run only those, e.g.
./scripts/compare-finetuned.sh Qwen/Qwen3.5-0.8B. Both are safe to rerun: runs
with saved predictions are skipped and finished adapters are reused, so delete
a run's files to redo it. A failed run is listed at the end rather than stopping
the others. Training logs to wandb, so run wandb login first or set
WANDB_MODE=offline on a machine that isn't logged in.
The device is chosen automatically — cuda, then mps, then cpu — and --device
overrides it. Training uses bf16 on cuda and fp32 elsewhere.
The non-generative baseline is a BERT-style sequence tagger with two heads on a shared backbone. One head tags intent spans and the other tags slots, each with BIO labels over subword tokens:
book a table in ames and play jazz
B-Book I-Book I-Book I-Book I-Book O B-PlayMusic I-PlayMusic
O O O O B-city O O B-genre
Intents never nest, and slots sit inside intents, so these two flat layers
capture the whole annotation. Decoding puts each slot under the intent span
that contains it. A slot just outside every intent stretches the nearest one to
cover it, and a slot the schema doesn't allow under its intent is dropped. The
loss is the sum of both heads' cross-entropies; --slot-weight rebalances it.
uv run multi-nlu tag-train -m answerdotai/ModernBERT-base -o runs/tagger
uv run multi-nlu tag-predict -t runs/tagger --split test -o preds.jsonl
tag-predict writes the same JSONL and sidecar as predict, rendered through
xml, so score and --detail work on it unchanged. A tagger is well formed
and should be faithful by construction, but Flan-T5 taggers are not yet (see
Token Tagger results).
Any backbone AutoModel can load works, decoders included; an encoder-decoder
like T5 contributes only its encoder. scripts/compare-taggers.sh trains and
scores encoders (ModernBERT, BERT, RoBERTa, DeBERTa-v3, EuroBERT and the
Flan-T5 encoder), Ettin encoder and decoder pairs from 150M to 1B, and Qwen3.5
0.8B and 2B. Decoders tag with left context only, which is the point of the
comparison, and Ettin's pairs are trained alike so that is the only difference.
Backbones above roughly 3B are commented out, as are runs that already have
predictions, which the script would skip anyway:
./scripts/compare-taggers.sh
Scoring happens on parsed annotations, never on raw text, so every format is graded identically. Generated text can drift from the input utterance, which shifts every character offset after the drift, so slot errors compare span values rather than positions. Every score is an error rate, so lower is better.
| Metric | Counts an error when |
|---|---|
| total errors | any intent or slot error, pooled over both |
| intent errors | a gold intent is missed, or a predicted one is spurious |
| slot errors | a gold slot (label and text) is missed, or one is spurious |
| sentence errors | any intent or slot in the utterance is wrong |
| malformed | the output does not parse in its format |
| unfaithful | the output does not reproduce the utterance, whitespace aside |
Intent and slot errors are divided by the number of gold items and broken down
into false negatives (FN, missed) and false positives (FP, spurious), which
--detail marks - and +. A wrong label counts as one of each, so a model
that invents many slots can exceed 100%.
Total errors divides the intent and slot errors together by the gold intents and slots together, so it is the one number to rank models by. Slots outnumber intents, so it mostly tracks slot errors, and a missed intent costs its slots too. It resembles the semantic error rate, but that aligns predictions to gold and counts a wrong label once, as a substitution, where this counts it twice.
malformed and unfaithful measure whether the model respected the format at
all; they are usually the first thing fine-tuning fixes. Formats that emit
offsets rather than a copy of the utterance are faithful by construction, which
is part of what the comparison is meant to expose.
All results are on the full test split (n = 2199).
Few-shot prompting (8 shots), from scripts/compare-few-shot.sh. Every score is
an error rate (see Metrics), timed as for the fine-tuned runs below.
The re-run with error rates has so far reached 9B; larger models follow.
Sent., Malf. and Unf. are sentence, malformed and unfaithful errors, and ms is
the time per utterance.
| Qwen3.5 | Fmt | Total | Intent | Slot | Sent. | Malf. | Unf. | ms |
|---|---|---|---|---|---|---|---|---|
| 0.8B | xml |
95.97% | 40.76% | 117.37% | 97.68% | 14.55% | 84.99% | 568 |
| 0.8B | dict |
88.01% | 35.79% | 108.26% | 97.91% | 12.14% | 12.14% | 358 |
| 2B | xml |
79.57% | 22.28% | 101.78% | 95.09% | 2.82% | 74.49% | 749 |
| 2B | dict |
77.73% | 30.80% | 95.93% | 96.45% | 0.32% | 0.32% | 457 |
| 4B | xml |
54.29% | 22.26% | 66.72% | 84.81% | 1.59% | 31.56% | 1771 |
| 4B | dict |
50.68% | 13.58% | 65.06% | 90.90% | 0.00% | 0.00% | 1058 |
| 9B | xml |
47.11% | 14.21% | 59.87% | 80.54% | 0.86% | 24.83% | 2417 |
| 9B | dict |
46.12% | 12.70% | 59.08% | 87.86% | 0.00% | 0.00% | 1460 |
| 27B | xml |
|||||||
| 27B | dict |
|||||||
| 35B-A3B | xml |
|||||||
| 35B-A3B | dict |
Few-shot errors stay far above fine-tuned ones: slot errors exceed 100% below
4B, since the models both miss most gold slots and invent spurious ones. Size
helps most between 2B and 4B (about 25 points of total errors), much less from
4B to 9B (4-7 points). dict has fewer total errors at every size so far (1-8
points) and runs about 40% faster, but xml gets more utterances fully right at
4B and 9B (80.54% sentence errors against 87.86% at 9B). xml paraphrases the
utterance often (25-85%), while dict is unfaithful only when malformed.
One epoch of LoRA per model and format, from scripts/compare-finetuned.sh.
Every score is an error rate (see Metrics); no run produced
malformed output. Time per utterance is batched generation (batch size 8) on a
DGX Spark.
| Model | Format | Total | Intent | Slot | Sentence | Unfaithful | ms / utt |
|---|---|---|---|---|---|---|---|
| Qwen3.5-0.8B | xml |
7.74% | 3.66% | 9.32% | 20.10% | 0.68% | 388 |
| Qwen3.5-0.8B | dict |
7.32% | 3.44% | 8.83% | 20.60% | 0.00% | 240 |
| Qwen3.5-2B | xml |
7.08% | 2.43% | 8.88% | 20.01% | 0.68% | 566 |
| Qwen3.5-2B | dict |
6.85% | 2.74% | 8.44% | 19.01% | 0.00% | 345 |
| Qwen3.5-4B | xml |
5.77% | 2.88% | 6.90% | 15.64% | 0.00% | 1330 |
| Qwen3.5-4B | dict |
5.99% | 3.42% | 6.99% | 16.19% | 0.00% | 817 |
| Qwen3.5-9B | xml |
5.14% | 2.88% | 6.02% | 13.60% | 0.27% | 1914 |
| Qwen3.5-9B | dict |
5.40% | 2.56% | 6.50% | 15.05% | 0.00% | 1167 |
Fine-tuning closes most of the gap that few-shot prompting leaves: even the
0.8B model has 7-8% total errors, against at least 46% for few-shot 9B. Errors
fall steadily with size, most between 2B and 4B (about 1.1 points of total
errors), reaching 5.14% with 9B xml.
The two formats stay close at every size. dict is slightly more accurate at
0.8B and 2B and xml at 4B and 9B, by 0.2-0.4 points of total errors. dict is
35-40% faster, since it copies only slot text rather than the whole utterance,
and it is always faithful, while xml paraphrases the utterance in up to 0.7%
of examples.
Fine-tuning Qwen3.5-27B runs out of memory on the DGX Spark's 128 GB of unified memory with the current settings, so it has no results yet.
Three epochs of full fine-tuning with tag-train defaults, scored by
tag-predict on the full test split, from scripts/compare-taggers.sh. Every
score is an error rate; no tagger is malformed. Time per utterance is at batch
size 64 on a DGX Spark. Backbones above roughly 3B are not run yet.
Encoders, including the Flan-T5 encoder:
| Backbone | Total | Intent | Slot | Sentence | Unfaithful | ms / utt |
|---|---|---|---|---|---|---|
| ModernBERT-base | 7.58% | 3.84% | 9.03% | 18.96% | 0.00% | 1.9 |
| ModernBERT-large | 6.14% | 3.96% | 6.98% | 15.33% | 0.00% | 4.2 |
| BERT-base | 5.69% | 2.68% | 6.85% | 14.92% | 0.00% | 1.1 |
| BERT-large | 5.74% | 3.21% | 6.71% | 16.01% | 0.00% | 2.9 |
| RoBERTa-base | 7.35% | 3.28% | 8.93% | 19.42% | 0.00% | 1.1 |
| RoBERTa-large | 5.72% | 2.59% | 6.94% | 15.10% | 0.00% | 3.1 |
| DeBERTa-v3-base | 5.11% | 2.56% | 6.10% | 13.87% | 0.00% | 1.9 |
| DeBERTa-v3-large | 5.40% | 2.95% | 6.35% | 14.64% | 0.00% | 4.7 |
| Ettin-150M | 7.35% | 3.06% | 9.01% | 19.05% | 0.00% | 1.7 |
| Ettin-400M | 6.75% | 4.43% | 7.65% | 16.10% | 0.00% | 4.2 |
| Ettin-1B | 5.75% | 3.10% | 6.77% | 14.14% | 0.00% | 9.4 |
| EuroBERT-210M | 7.82% | 2.54% | 9.87% | 20.15% | 0.00% | 1.7 |
| EuroBERT-610M | 7.44% | 3.78% | 8.86% | 19.42% | 0.00% | 5.2 |
| EuroBERT-2.1B | 7.56% | 3.98% | 8.95% | 19.83% | 0.00% | 19.7 |
| Flan-T5-large | 11.86% | 3.46% | 15.12% | 29.79% | 4.50% | 5.7 |
| Flan-T5-XL | 5.92% | 2.02% | 7.44% | 16.73% | 0.50% | 16.6 |
Decoders, which tag with left context only (none is unfaithful):
| Backbone | Total | Intent | Slot | Sentence | ms / utt |
|---|---|---|---|---|---|
| Ettin-150M | 37.27% | 33.95% | 38.56% | 71.67% | 1.8 |
| Ettin-400M | 36.43% | 34.89% | 37.02% | 71.40% | 4.3 |
| Ettin-1B | 34.03% | 31.99% | 34.82% | 70.40% | 9.7 |
| Qwen3.5-0.8B | 33.11% | 33.23% | 33.07% | 68.80% | 11.1 |
| Qwen3.5-2B | 33.78% | 34.49% | 33.50% | 69.71% | 18.3 |
DeBERTa-v3-base is the best tagger, with 5.11% total errors, as good as LoRA Qwen3.5-9B (5.14%) and with about the same sentence errors. Most large encoders land between 5.4% and 6.1%, near LoRA 4B. Size helps little: DeBERTa-v3-base beats its large model, and EuroBERT does not improve from 210M to 2.1B. The encoders tag in 1-20 ms per utterance, against at least 240 ms for LoRA generation, since they need one forward pass and no generation.
Every decoder has 33-37% total errors, whatever its family or size, against 5.7-7.4% for the matching Ettin encoders. Nearly all of their intent errors are spurious (31-34% FP against at most 1.1% FN). That fits a tagger that cannot see where a span ends, but a bug in the decoder path would look the same, so these numbers need a sanity check.
Flan-T5 is unfaithful because its tokenizer can split a word into a bare ▁
and the rest, and the bare ▁ gets the same character offsets as the first
letter, so decoding repeats that letter ("samoa" becomes "ssamoa").
A schema is a YAML file naming the intents, the slots each intent takes, and the
dataset it came from. The bundled one lives in src/multi_nlu/schemas/:
name: mixsnips
dataset:
path: blambert/mixsnips-intent-spans
text_column: text
intents_column: intents
intents:
GetWeather:
- city
- timeRange
intents_column holds the offset annotations: a list of
{intent, start, end, slots: [{name, start, end}]}.
To use a different dataset, write a schema for it — or derive one from its gold annotations — and pass it to any command:
uv run multi-nlu derive-schema org/my-dataset -o my-dataset.yaml
uv run multi-nlu predict --schema my-dataset.yaml -m Qwen/Qwen3.5-2B
derive-schema drops slots seen in under 1% of an intent's occurrences
(--min-freq), which filters annotation noise. Read the result before using it.
- Re-run the few-shot experiments to get error-rate figures
- Fit 27B fine-tuning in memory (e.g. 4-bit loading or gradient checkpointing)
- Do the modeling with an encoder model
- Bidirectional attention for decoder taggers, to separate the weights from the mask
- Sanity-check the decoder taggers' 33-37% error rates for a bug
- Fix Flan-T5 taggers repeating a letter after a bare
▁token - Switch to alignment-based evaluation using an external library
- Fix the 2 train rows whose slot span overruns its intent span (dataset repo)
src/multi_nlu/
annotation.py the format-neutral spans everything converts to
schema.py schemas and the datasets they describe
formats/ one module per way of writing an annotation down
base.py the Format interface and shared prompt preamble
xml.py inline XML tagging
dict_format.py structured-JSON tagging
data.py loading examples, sampling few-shot demonstrations
device.py device and dtype selection, timing
metrics.py scoring, on annotations rather than text
report.py saved predictions, their scores and per-example diffs
cli.py command line interface
generative/ prompting and LoRA fine-tuning decoder LMs
models.py causal LM and tokenizer loading
prompts.py the message layout shared by prompting and fine-tuning
predict.py batched generation
train.py LoRA fine-tuning
tagger/ token tagging over a pretrained backbone
tags.py spans as two layers of BIO tags, and back
model.py the two-headed token tagger: model, training, prediction
uv run pytest
uv run black . && uv run isort . && uv run mypy src
The tests cover the span model, the xml and dict formats, BIO tagging,
scoring, per-example diffs, prompting and schema handling, and need neither a
model nor a GPU. One dict test and one tagging test pull real examples from
the dataset to check round-trip fidelity, so they need network access the first
time it isn't cached.