Skip to content

feat: run vLLM startup through prestart hooks - #358

Merged
velaraptor-runpod merged 1 commit into
mainfrom
feat/prestart-hooks
Nov 3, 2026
Merged

velaraptor-runpod merged 1 commit into
mainfrom
feat/prestart-hooks

Conversation

@jasonwang-runpod

@jasonwang-runpod jasonwang-runpod commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Problem

vLLM starts before the SDK, so unrecognized startup failures stay in worker logs instead of reaching queued requests.

Solution

Run startup under SDK prestart supervision while keeping jobs queued until startup settles. Preserve existing retries and startup_error responses for known configuration failures. Unrecognized failures now fail one queued job with prestart_failed, include vLLM output, and exit so the platform can restart the worker.

Requires runpod 2.0.0, which main already pins.

Testing

  • 184 worker tests and lint passed with runpod==2.0.0 from PyPI.
  • Live A40 / vLLM 0.32.0 / runpod==2.0.0: prompt, chat, and streaming jobs queued during a cold start, were claimed after startup finished, and completed within a 20-second execution limit.
  • Earlier live run against SDK 2.0 main also covered known and unknown startup failure reporting. Existing local-input and realtime SDK limitations remain.

@velaraptor-runpod
velaraptor-runpod self-requested a review October 20, 2026 01:20
@jasonwang-runpod
jasonwang-runpod marked this pull request as ready for review November 3, 2026 22:11
@velaraptor-runpod
velaraptor-runpod merged commit e8b857d into main Nov 3, 2026
9 of 10 checks passed
@velaraptor-runpod
velaraptor-runpod deleted the feat/prestart-hooks branch November 3, 2026 22:20
@promptless

promptless Bot commented Nov 3, 2026

Copy link
Copy Markdown

Promptless documentation updates

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants