Skip to content

Add a runbook for one benchmark run on AWS - #3

Merged
RobGrafrath merged 1 commit into
mainfrom
add-benchmark-run-runbook
Oct 10, 2026
Merged

RobGrafrath merged 1 commit into
mainfrom
add-benchmark-run-runbook

Conversation

@RobGrafrath

Copy link
Copy Markdown
Contributor

What this changes and why

This adds docs/run-a-benchmark.md, the routine for one benchmark run on an AWS sandbox host, from signing in to destroying the host and recording what it cost. It links the page from the README and corrects one sentence in the cost guardrails. The repository explains how to build the account and each stack, but not the routine for a single run, and the first real run will need one.

The page is deliberately generic and names no experiment.

Writing it turned up three behaviors that the existing docs did not state:

  • max_lifetime_hours is part of the first-boot script and user_data_replace_on_change is true, so raising the value and applying again replaces the instance and loses the run. The cost guardrails page said to "raise the variable on purpose and apply again". It now points to the supported way to extend a running host from the inside.
  • The image lookup uses the newest Ubuntu image, so a plan against a running host can show a replacement when a newer image is published. The runbook says never to apply against a running host.
  • First boot installs the PostgreSQL client but neither the server tools nor a terminal multiplexer, so pg_test_fsync and tmux are not on the host until someone installs them.

Does it make a host easier to leave running or easier to reach?

It documents one way to lengthen a running host's life (sudo shutdown -c; sudo shutdown -h +N). The stated reason is that applying again would destroy the run, and the page requires scheduling a new timer in the same command line and noting the extension in the run log. It changes no stack, policy, or first-boot code, opens no port, and adds no permission.

How I checked it

scripts/check-repo.py and tofu fmt -check -recursive pass locally, and CI runs validate. I have not run the commands in the runbook against a real host, so the first real run should follow the page and correct it. The vCPU quota code (L-1216C47A) and the behavior of shutdown --show are from the AWS and systemd documentation as I know it, so please confirm the quota code in the Service Quotas console.

Left out on purpose

  • Adding tmux to shared/cloud-init.yaml, so a host has a terminal multiplexer on first boot. It changes first-boot code, so it should be reviewed on its own.
  • A small script that lists leftover resources, to replace the manual commands in step 9.

Checklist

  • tofu fmt -recursive and tofu validate pass in each stack I changed, or I am relying on CI. No stack changed.
  • No state files, .tfvars files, keys, tokens, account IDs, or personal addresses are committed.
  • Docs match the change, and follow the house style (US English, no em or en dashes).
  • Commits are signed off (DCO). If I forgot, I will post the retroactive sign-off comment described in CONTRIBUTING.md.
  • I license this contribution under Apache 2.0.

🤖 Generated with Claude Code

The routine from sign-in to teardown: confirm the account, configure and
plan, wait for first boot, copy code up, run under tmux, watch the
lifetime timer, copy results off, destroy and verify, and record the
cost.

It also records two behaviors the existing docs did not state. Changing
max_lifetime_hours replaces the host, because the value is part of the
first-boot script, and a plan against a running host can replace it when
a newer Ubuntu image appears. The cost guardrails page said to raise the
variable and apply again, which would have destroyed a running host's
data, so it now points to the supported way to extend a running host.

The runbook is deliberately generic and names no experiment.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Signed-off-by: Rob Grafrath <[email protected]>
@RobGrafrath
RobGrafrath marked this pull request as ready for review October 10, 2026 22:42
@RobGrafrath
RobGrafrath merged commit ab8c25b into main Oct 10, 2026
2 checks passed
@RobGrafrath
RobGrafrath deleted the add-benchmark-run-runbook branch October 10, 2026 22:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant