A web application that is operated by speech. The browser records a spoken question, the server transcribes it with Whisper, a fine-tuned DistilBERT model classifies the transcript into one of 16 food and nutrition intents, and the answer is looked up in a corpus of 39,447 real recipes. The recognised speech and the reply are both displayed, along with the predicted intent and its confidence.
Ask it how many calories are in butter chicken and the figure it quotes is real, read from the corpus rather than written into the code.
Live application: https://voicebot-slp.vercel.app
microphone -> recorded audio -> POST /api/voice
|
faster-whisper base.en (int8, CTranslate2)
| recognised text
DistilBERT intent classifier (int8 ONNX, 41 classes)
| intent + confidence
response template -> transcript, intent and reply shown
Neither model needs PyTorch at inference time, so the deployed service stays small enough for a free hosting tier.
Two datasets, doing different jobs.
CLINC150 (Larson et al., EMNLP 2019) trains the classifier. It is a
benchmark built for intent classification that includes an explicit out-of-scope
class. This project uses one domain in full - kitchen and dining, 15 intents -
plus the oos class, giving 16 classes.
recipes-with-nutrition (datahiveai) supplies the answers: 39,447 recipes
with servings, energy, nutrient breakdowns, ingredients and dietary labels. It
cannot train an intent classifier, since it holds recipes rather than user
utterances, but it is the knowledge base the cooking and nutrition intents
answer from. train/build_recipes.py reduces the published 450 MB CSV to the
2.4 MB gzipped table the app ships.
| Split | Utterances | Per in-scope intent | Out-of-scope |
|---|---|---|---|
| Train | 4,250 | 100 | 250 |
| Validation | 900 | 20 | 100 |
| Test | 1,500 | 30 | 300 |
Where an utterance names a dish, the cooking and nutrition intents fill their response from the corpus. Where it does not, or where the corpus holds nothing relevant - it records no cooking times - a written template is used instead. Templates are chosen so no placeholder ever reaches the user unfilled.
Four classifiers were trained on identical splits so the value of the transformer could be measured rather than assumed. Only the last is deployed.
| Model | Parameters | Test accuracy | Macro F1 |
|---|---|---|---|
| TF-IDF + Logistic Regression | 485,071 | 0.8800 | 0.9055 |
| Bag-of-words + MLP | 375,081 | 0.8747 | 0.8981 |
| Embedding + BiLSTM | 273,833 | 0.8860 | 0.9015 |
| DistilBERT, fine-tuned | 66,985,001 | 0.9460 | 0.9556 |
| DistilBERT, int8 ONNX (deployed) | - | 0.9473 | 0.9562 |
Speech recognition uses Whisper base.en with int8 quantisation, through the
faster-whisper implementation on the CTranslate2 engine.
Out-of-scope rejection. A prediction whose softmax probability falls below a threshold is answered as out-of-scope rather than guessed at. The threshold was chosen on the validation split and only then measured on test.
| Setting | In-scope accuracy | Out-of-scope recall | Macro F1 |
|---|---|---|---|
| No threshold | 0.9742 | 0.8333 | 0.9556 |
| Threshold 0.44 | 0.9700 | 0.9033 | 0.9651 |
End-to-end through speech. 86 held-out test utterances were synthesised to audio and run through the deployed pipeline.
| Metric | Value |
|---|---|
| Word error rate | 0.0394 |
| Transcribed with no errors | 80.2% |
| Intent accuracy from clean text | 0.9767 |
| Intent accuracy from speech | 0.9651 |
| Mean speech recognition latency | 1089 ms |
| Mean intent inference latency | 12 ms |
| Real-time factor | 0.35 |
Recognition errors cost 1.2 accuracy points, because they tend to fall on words that do not determine the intent.
Figures and tables are regenerated by train/evaluate.py and
train/eval_voice.py into results/, and reproduced in report/report.md.
app/ inference pipeline, FastAPI service, model weights
pipeline.py Whisper and the ONNX classifier, loaded once and reused
main.py API endpoints
responses.json intent to response templates
train/ data preparation, training, export and evaluation
web/ React front end
data/ generated CLINC150 splits
results/ metrics, figures, summary tables
report/ written report and viva preparation notes
pip install -r requirements-train.txt
python train/prepare_data.py # build the CLINC150 food subset
python train/build_recipes.py # condense the recipe corpus to a lookup table
python train/train_baselines.py # the three reference models
python train/train_distilbert.py # the transformer
python train/predict_split.py val # validation predictions for the threshold
python train/evaluate.py # figures and results tables
python train/export_onnx.py # int8 ONNX build that gets deployed
python train/eval_voice.py # end-to-end speech evaluation
cd app && uvicorn main:app --port 8080 # backend
cd web && npm install && npm run dev # front end on :5173All randomness is seeded, so the reported numbers reproduce.
| Method | Path | Purpose |
|---|---|---|
| GET | /health |
liveness probe |
| GET | /api/info |
model metadata and the intent list |
| POST | /api/chat |
{"text": "..."} to intent and reply |
| POST | /api/voice |
audio upload to transcript, intent and reply |
| POST | /api/transcribe |
audio upload to transcript only |
The backend runs as a Docker container on Railway and the front end is a static
build on Vercel. DEPLOY.md has the exact commands, including the Google Cloud
Run alternative.
- Larson, S., Mahendran, A., Peper, J. J., et al. An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction. EMNLP 2019.
- Sanh, V., Debut, L., Chaumond, J., Wolf, T. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. NeurIPS EMC^2 Workshop, 2019.
- Radford, A., Kim, J. W., Xu, T., et al. Robust Speech Recognition via Large-Scale Weak Supervision. OpenAI, 2022.
- Devlin, J., Chang, M.-W., Lee, K., Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL 2019.