Skip to content

feat: report average inference time per utterance - #16

Merged
belambert merged 1 commit into
mainfrom
feat/inference-timing
Oct 4, 2026
Merged

belambert merged 1 commit into
mainfrom
feat/inference-timing

Conversation

@belambert

@belambert belambert commented Oct 4, 2026 •

Copy link
Copy Markdown
Owner

predict now prints the average wall-clock inference time per utterance after the scores, for example:

time / utterance           41.2 ms
  • Only the model run is timed, not model or dataset loading or scoring.
  • On cuda and mps, the device is synchronized before the clock is read, so asynchronous kernels aren't undercounted.
  • The first batch is included in the average, so on a GPU a small -n will overstate the time while kernels warm up.
  • score and the saved JSONL are unchanged, since a re-scored file has no timing to report.

This is the predict half of the timing commit on #15, which also times tag-predict. The helpers are identical in both, so whichever PR merges second should rebase cleanly.

Testing

@belambert
belambert force-pushed the feat/inference-timing branch from 6c9a189 to 4c6096f Compare October 4, 2026 16:50
@belambert
belambert merged commit a45536e into main Oct 4, 2026
2 checks passed
@belambert
belambert deleted the feat/inference-timing branch October 6, 2026 01:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant