Skip to content

feat(AUTO-1062): ask sandbox for variable mutations only and rebuild scopes on the host - #1568

Open
Kart03 wants to merge 1 commit into
developfrom
feat/mutations-only-result
Open

Kart03 wants to merge 1 commit into
developfrom
feat/mutations-only-result

Conversation

@Kart03

@Kart03 Kart03 commented Oct 15, 2026

Copy link
Copy Markdown

Context

AUTO-1062 / AUTO-60. After each script the sandbox sends back the full environment, globals and collectionVariables, and runtime only applies their mutations. With ~4 MB of variables that copy costs ~22 ms and ~22 MB per script. One sandbox thread serves all VUs, so it also caps throughput: in a 44-VU, 20-minute CLI run, a prototype cut the worker peak from 2.65 to 1.92 GB and raised throughput from 32 to 40 rps.

Summary of the change

  • event.command.js passes variableMutationsOnly: true to host.execute (needs feat(AUTO-1062): add variableMutationsOnly option to return only scope mutations postman-sandbox#1116).
  • When a result scope comes back with no values, runtime rebuilds it with applyScopeMutations: a copy of contextToUse[scope] with the mutations applied, and .mutations kept. The script trigger still gets full scopes with mutations, which nested-request and consumers read.
  • contextToUse is the masked clone when _forbiddenSecretKeys apply, so masked secrets stay masked in the script result. A sandbox-only prototype failed 9 tests here (8 vault masking, 1 pm.variables).
  • Tracked scopes are still updated through result[scope].mutations → applyOn(payload.context[scope]), as before.
  • The rebuild copies Variable objects but shares their string values, so it stays cheap even with large values.

Breaking changes

None. An older sandbox ignores the option and sends full scopes, which take the old path (detected by values). An older runtime never sets the option.

Dependencies

postmanlabs/postman-sandbox#1116 (variableMutationsOnly). No gain until runtime bumps postman-sandbox. Until then this PR is behaviour-neutral.

Test plan

Suite sandbox 6.7.4 sandbox branch
test-unit 571 passing 571 passing
test-integration 1901 passing, 0 failing 1901 passing, 0 failing
test-integration-legacy not run 33 passing
test-lint ok ok

Bench: node copybench.js <runtimeDir> 7 100 (1 request, prerequest + test script, 4.08 MB collection vars, 200 scripts):

ms/script host alloc MB/script sandbox→host MB/script
develop 22.8 / 22.7 / 25.3 22.4 / 22.5 / 22.9 4.55
this + sandbox branch 13.4 / 13.6 / 13.8 14.2 / 14.1 / 14.1 0

Additional notes

Draft to develop. Hold it until the sandbox release, then add the dependency bump.

Generated with Claude Code

Pass variableMutationsOnly to host.execute so the sandbox sends back only
mutations for environment, globals and collectionVariables. The full
post-script scopes for the script trigger are rebuilt from the context
sent to the sandbox (the masked clone when forbidden secrets apply) with
the mutations applied. An older sandbox ignores the option and sends the
full scopes, which are used as before.

Co-Authored-By: Claude <[email protected]>
@codecov

codecov Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 46.15385% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 77.42%. Comparing base (9c2d10d) to head (c6ba538).

Files with missing lines Patch % Lines
lib/runner/extensions/event.command.js 46.15% 6 Missing and 1 partial ⚠️

❌ Your patch check has failed because the patch coverage (46.15%) is below the target coverage (100.00%). You can increase the patch coverage or adjust the target coverage.
❌ Your project check has failed because the head coverage (34.28%) is below the target coverage (35.00%). You can increase the head coverage or adjust the target coverage.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #1568      +/-   ##
===========================================
- Coverage    77.54%   77.42%   -0.12%     
===========================================
  Files           51       51              
  Lines         4338     4346       +8     
  Branches      1223     1225       +2     
===========================================
+ Hits          3364     3365       +1     
- Misses         731      737       +6     
- Partials       243      244       +1     
Flag Coverage Δ
integration 68.61% <46.15%> (-0.11%) ⬇️
legacy 34.28% <46.15%> (-0.05%) ⬇️
unit 46.82% <30.76%> (-0.05%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@Kart03

Kart03 commented Oct 15, 2026

Copy link
Copy Markdown
Author

Full-run results, for reviewers. This was a copy of the PPC Governor collection (AUTO-1062) through a local CLI build: 44 VUs, fixed load, 20 min, 6 GiB / 4 CPU. The run used a prototype that patched the sandbox the same way as this PR (only mutations back for environment, globals and collectionVariables), and the comparison run was identical apart from that.

Full scopes back Mutations only
Per script run, ~4 MB of variables (micro-benchmark) 22.5 ms, 22.4 MB allocated 13.3 ms, 14.0 MB allocated
Worker peak 2.65 GB 1.92 GB
Worker avg, minutes 4–12 / after the ACS step 1.20 / 2.23 GB 0.97 / 1.58 GB
Container peak 3.45 GB 2.81 GB
rps after the ACS step 31.7 40.0
Requests outside the time-bound loop 80,828 80,828
Assertions failed / passed 220 / 21,648 220 / 21,648

Why it helps: the allocation profile of the host thread shows the uvm bridge making about 6 copies of the variables per script run (Flatted.stringify + join + postMessage + receive + Flatted.parse), with the sandbox thread mirroring them. One sandbox thread serves all VUs, so the copies also capped throughput. This removes the return trip.

Reviewer guide covering all the AUTO-1062 changes: https://claude.ai/artifact/S7vBMoFxTA9kjzQHhQTXpt

@Kart03
Kart03 marked this pull request as ready for review October 15, 2026 14:54

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant