Skip to content

Rate limit the job submission request - #866

Open
hellolittlej wants to merge 1 commit into
masterfrom
rate-limit-job-submission
Open

Rate limit the job submission request#866
hellolittlej wants to merge 1 commit into
masterfrom
rate-limit-job-submission

Conversation

@hellolittlej

@hellolittlej hellolittlej commented Jul 30, 2026

Copy link
Copy Markdown
Collaborator

What

Adds an opt-in, per route based bucket rate limit.For jobsRoute, add the rate limit in front of the three v1 job-creation endpoints:

  • POST /api/v1/jobs
  • POST /api/v1/jobClusters/{cluster}/jobs
  • POST /api/v1/jobs/actions/quickSubmit

The rate

Configuration

Property Default Notes
mantis.master.api.v1.submitJob.throttle.enabled false Off by default; no limiter is constructed when off
mantis.master.api.v1.submitJob.permitsPerSecond 1000 Re-readable at runtime, no deploy needed
  • Guava RateLimiter, non-blocking tryAcquire(); token-bucket with the usual ~1s of burst accumulation, so a 1000/s limit admits up to a ~1000-request burst after an idle period.
  • Shed → HTTP 429, checked in BaseRoute.withThrottle:141 before the entity is unmarshalled — a storm costs a permit check, not a parse plus an actor ask.

Off by default because the right ceiling depends on cluster size and traffic —
operators opt in once they've measured their own submit rate. The rate is backed by a
LongDynamicProperty, so it re-reads on the dynamic-property refresh interval
(mantis.config.dynamic.refreshSecs, default 30s).

Metrics

MasterApiMetrics/throttledRequestCount gains a source tag (submitJob,
resourceCluster), so an operator can tell which limit is shedding. Previously the
route-level throttle and the resource-cluster gateway's RequestThrottledException
both incremented the same untagged counter.

Checklist

  • ./gradlew build compiles code correctly
  • Added new tests where applicable
  • ./gradlew test passes all tests
  • Extended README or added javadocs where applicable

@hellolittlej hellolittlej changed the title Rate limit the job submission request [TESTING] Rate limit the job submission request Jul 30, 2026
@github-actions

github-actions Bot commented Jul 30, 2026

Copy link
Copy Markdown

Test Results

167 files  + 3  167 suites  +3   10m 43s ⏱️ +25s
814 tests +17  802 ✅ +16  11 💤 ±0  1 ❌ +1 
815 runs  +18  803 ✅ +17  11 💤 ±0  1 ❌ +1 

For more details on these failures, see this check.

Results for commit 8d0199a. ± Comparison against base commit 25ff35c.

♻️ This comment has been updated with latest results.

@hellolittlej
hellolittlej force-pushed the rate-limit-job-submission branch 2 times, most recently from 9cfd8a5 to a8c0223 Compare August 4, 2026 20:30
@hellolittlej hellolittlej changed the title [TESTING] Rate limit the job submission request Rate limit the job submission request Aug 4, 2026
@hellolittlej
hellolittlej force-pushed the rate-limit-job-submission branch 2 times, most recently from b9faf36 to ca6bbe8 Compare August 5, 2026 05:56
@hellolittlej
hellolittlej force-pushed the rate-limit-job-submission branch from ca6bbe8 to 8d0199a Compare August 5, 2026 06:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant