remote/lambda/start/index.ts's wake() de-duplicates a concurrent start purely by looking up the environment's instance by tag (findManagedInstance, an eventually-consistent DescribeInstances call). Two wake() invocations for the same environment close enough together can each miss the other's not-yet-visible instance and each reach launchAcrossAzs, launching two EC2 instances for one environment — a real double-billed launch, not a cosmetic race.
#222 closes this gap for calls originating from one gateway process (a Go-side singleflight.Group coalesces concurrent wakes before they ever reach the control plane), but the control plane itself is still unprotected against:
- two separate gateway processes pointed at the same environment
- a
spinloop remote start CLI call racing a gateway wake
- any other direct caller
Fix
An idempotency lock in wake(), acquired before findManagedInstance and released (try/finally) across every one of its early-return paths (unconfigured, undeployed, the seeding gate, no-capacity, boot-failed, deadline-timeout, ready).
Cheapest option: an SSM parameter (e.g. /cloud-vm-llm/<env>/wake-lock) written with Overwrite: false as a create-if-absent primitive — the existing SSM IAM policy on the start Lambda's role is already namespace-wide (arn:...:parameter/cloud-vm-llm/*), so no new IAM grant is needed. ParameterAlreadyExists means another wake is in flight (or crashed): read the existing lock's expiry and either return the existing 503-retry reply, or overwrite and proceed if it's stale. The TTL needs to exceed the Lambda's own 900s timeout, since a legitimate wake can run that long.
A DynamoDB-backed lock (conditional PutItem, TTL attribute) is the more conventional alternative, at the cost of a new CDK construct and IAM grants — no DynamoDB table exists in the stack today.
Ruled out: reserved concurrency = 1 on the Lambda serializes every environment's wakes through one slot, not just the racing one. An SQS FIFO queue keyed by environment would be correctly scoped, but means converting the Lambda from its current synchronous poll-until-ready HTTP contract to an async one — a bigger redesign than the lock itself.
remote/lambda/start/index.ts'swake()de-duplicates a concurrent start purely by looking up the environment's instance by tag (findManagedInstance, an eventually-consistentDescribeInstancescall). Twowake()invocations for the same environment close enough together can each miss the other's not-yet-visible instance and each reachlaunchAcrossAzs, launching two EC2 instances for one environment — a real double-billed launch, not a cosmetic race.#222 closes this gap for calls originating from one gateway process (a Go-side
singleflight.Groupcoalesces concurrent wakes before they ever reach the control plane), but the control plane itself is still unprotected against:spinloop remote startCLI call racing a gateway wakeFix
An idempotency lock in
wake(), acquired beforefindManagedInstanceand released (try/finally) across every one of its early-return paths (unconfigured, undeployed, the seeding gate, no-capacity, boot-failed, deadline-timeout, ready).Cheapest option: an SSM parameter (e.g.
/cloud-vm-llm/<env>/wake-lock) written withOverwrite: falseas a create-if-absent primitive — the existing SSM IAM policy on the start Lambda's role is already namespace-wide (arn:...:parameter/cloud-vm-llm/*), so no new IAM grant is needed.ParameterAlreadyExistsmeans another wake is in flight (or crashed): read the existing lock's expiry and either return the existing 503-retry reply, or overwrite and proceed if it's stale. The TTL needs to exceed the Lambda's own 900s timeout, since a legitimate wake can run that long.A DynamoDB-backed lock (conditional
PutItem, TTL attribute) is the more conventional alternative, at the cost of a new CDK construct and IAM grants — no DynamoDB table exists in the stack today.Ruled out: reserved concurrency = 1 on the Lambda serializes every environment's wakes through one slot, not just the racing one. An SQS FIFO queue keyed by environment would be correctly scoped, but means converting the Lambda from its current synchronous poll-until-ready HTTP contract to an async one — a bigger redesign than the lock itself.