Skip to content

CLI --load: opt-in error-rate / latency thresholds that set the exit code (for CI) #2745

Open

Description

@cliffhall

Follow-up to #2439 / #2743.

Problem

A --load run that completes exits 0 whatever its error rate (documented in the CLI README). To use it as a CI gate today you have to parse the summary with --format json and jq and do the comparison yourself.

Proposal

Add an opt-in threshold so the exit code carries the verdict, for example:

  • --load-max-error-rate <0..1>: fail when errorRate is above the value (it is already the share of calls that were tool errors or thrown errors).
  • Possibly --load-max-p99 <ms> for a latency SLO. Decide whether that belongs in the same change.

When a threshold is breached, still print the full summary, then exit with a new dedicated exit code (or an existing one if one fits; check core/cli exit codes) and a structured error code naming which threshold failed. With no threshold set, behavior is unchanged.

Validation: the threshold flags require --load, and the value must be in range.

If #2744 lands first, add the flags there too, since both clients share core/cli/handlers/load-test.ts.

Priority score

Severity 2 (easy workaround with jq) + urgency 3 (wanted this milestone) + 1 (milestoned) = 6 → Medium.

Activity

  1. added this to the v2.12.0 milestone on Oct 20, 2026
  2. modified the milestones: v2.12.0, v2.13.0 on Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature requestv2Issues and PRs for v2

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions