Skip to content

Run e2e tests in parallel chains on dedicated deployments - #164

Merged
jeremybeard merged 1 commit into
mainfrom
e2e-parallel
Oct 19, 2026
Merged

jeremybeard merged 1 commit into
mainfrom
e2e-parallel

Conversation

@jeremybeard

@jeremybeard jeremybeard commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

Cuts the e2e run from ~43 to ~13 minutes.

  • Tests run in three chains per deployment kind (standard, hibernate), each chain on its own deployment, plus one deployment each for the preview and remote tests: 9 at once. Tests that compare a deployment's versions before and after still never share one concurrently.
  • The deployments are created one at a time (creates return immediately), retrying rate-limited requests, and published as a slot→ID map; one job deletes them all, along with any deployment preview a failed or cancelled test left behind.
  • A new input validation job runs the action against a stub astro that records its calls, so a rejection only passes if it came before the action touched Astro. A control case must be accepted.
  • The diff-base patch now applies to every run shape, which needed the deploy-type step in action.yaml to read the before/after SHAs through its existing shell variables. No behavior change.
  • Fixes a timestamp check that always passed and an output name typo that made one assertion compare against an empty value.
  • Runs the e2e deployments on runtime 3.3-2, with 15-minute health waits.

@jeremybeard
jeremybeard added this pull request to stack #169 October 15, 2026 17:25
@jeremybeard jeremybeard changed the title e2e parallel Run e2e tests in parallel chains on dedicated deployments Oct 15, 2026
@jeremybeard
jeremybeard marked this pull request as ready for review October 16, 2026 19:11
@jeremybeard
jeremybeard requested a review from a team as a code owner October 16, 2026 19:11

@wendtek wendtek left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

@neel-astro neel-astro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for all the cleanup 🚀

name: deploy-action-non-dev-e2e
description: ""
runtime_version: 3.1-14
runtime_version: 3.3-2

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we can write some auto update CI job to keep these runtime versions to latest

@jeremybeard
jeremybeard merged commit fab181a into main Oct 19, 2026
41 checks passed
@jeremybeard
jeremybeard deleted the e2e-parallel branch October 21, 2026 14:51

This branch was successfully deployed

1 active deployment
e2e-test — b9afb99f Deployed Oct 16, 2026 by jeremybeard via Redact Inputs #361
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants