Initial implementation of remote persistent workers by MarcusSorealheis · Pull Request #1922 · TraceMachina/nativelink

MarcusSorealheis · 2025-09-03T15:52:10Z

Description

Remote persistent workers are coming to Nativelink! Not there yet.

Fixes # (issue)

Type of change

Please delete options that aren't relevant.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to
not work as expected)
This change requires a documentation update

How Has This Been Tested?

Please also list any relevant details for your test configuration

Checklist

Updated documentation if needed
Tests added/amended
bazel test //... passes locally
PR is contained in a single commit, using git amend see some docs

This change is

palfrey · 2025-09-11T14:12:10Z

None of this is being run in CI, unlike the buildstream/mongo tests. Does this need adding?

palfrey · 2025-09-11T14:12:35Z

+  int32 verbosity = 5;
+
+  // Sandbox directory for this request
+  string sandbox_dir = 6;


IIRC this needs removing?

It does need removing.

You know, I'm not sure it does. At first, when I gave my internal presentation, that was my understanding. However, I have been reading the docs again and reading the Bazel source code. Take a look at: https://bazel.build/remote/persistent#options

remote persistent actions will use the worker strategy but all others could use sandboxed or local.

should still be removed

palfrey · 2025-09-11T14:15:53Z

Not seeing any tests for this

It's very hard to test the worker runner.

palfrey · 2025-09-11T14:16:06Z

Need to test a few more things that just the key :)

palfrey · 2025-10-03T14:39:41Z

+
+```bash
+npm run build
+node dist/e2e.test.js


This doesn't exist in dist after running build.

This is not ready. I need to work on this more. Sorry for confusion

Implements remote persistent workers feature for NativeLink, enabling persistent worker processes to be reused across multiple actions for improved build performance, particularly for JVM-based builds. Core implementation: - Persistent worker protocol based on Bazel worker protocol - Worker lifecycle management with automatic cleanup - Scheduler integration for persistent worker routing - Platform properties-based worker key matching Components: - PersistentWorkerManager: Pool management with LRU eviction - PersistentWorkerInstance: Individual worker process lifecycle - PersistentWorkerRunner: On-demand worker process spawning - WorkRequest/WorkResponse protocol implementation Scheduler integration: - Dedicated worker pools per persistentWorkerKey - Modified worker matching to check persistent pools first - Automatic pool management on worker registration/removal Testing: - Full unit test coverage - Integration tests for worker protocol - Integration tests for scheduler matching - All 72 tests passing (bazel test //...) Code quality improvements: - All clippy warnings resolved with -D warnings - Proper code formatting - Correct license headers (BSL 1.1 for enterprise features)

- Change clippy cast_possible_truncation from deny to allow - Format Cargo.toml exclude array with alphabetical ordering

Remove clippy flags that pre-commit doesn't want: - await_holding_lock - elidable_lifetime_names - explicit_into_iter_loop - redundant_closure_for_method_calls - semicolon_if_nothing_returned Change todo from deny (-D) to warn (-W)

- Apply generate-bazel-rc auto-fixes to .bazelrc - Remove unused dependencies (hyper-util, tracing-test) - Fix Vale documentation style issues in README - Replace e.g., with 'for example' - Use backticks for filenames and compiler names - Fix namespace terminology

The tracing-test dependency is required by the #[nativelink_test] macro. Cargo-machete incorrectly flagged it as unused because it's used by the macro expansion, not directly in code. Added cargo-machete ignore metadata to prevent future false positives.

palfrey · 2025-11-05T10:36:18Z

This and various other config files appear to have been reset to ~0.7.1 era (given the versions) and they remove a bunch of fixes I've done in that time. Possibly merge issues?

Yes, that's correct. Let me try to get this fixed.

palfrey · 2025-11-05T10:37:03Z

Not sure why this is here?

MarcusSorealheis · 2025-11-06T16:11:14Z

I need more time to work on this

MarcusSorealheis · 2025-11-16T01:25:19Z

closing this approach in favor of a more secure implementation.

MarcusSorealheis temporarily deployed to production September 3, 2025 15:52 — with GitHub Actions Inactive

MarcusSorealheis had a problem deploying to production September 3, 2025 15:52 — with GitHub Actions Failure

MarcusSorealheis marked this pull request as draft September 3, 2025 23:41

MarcusSorealheis force-pushed the remote-persistent-workers branch from 42c8b8e to c69655d Compare September 4, 2025 01:12

MarcusSorealheis temporarily deployed to production September 4, 2025 01:12 — with GitHub Actions Inactive

MarcusSorealheis had a problem deploying to production September 4, 2025 01:12 — with GitHub Actions Failure

MarcusSorealheis temporarily deployed to production September 4, 2025 01:12 — with GitHub Actions Inactive

MarcusSorealheis had a problem deploying to production September 4, 2025 01:12 — with GitHub Actions Failure

MarcusSorealheis temporarily deployed to production September 4, 2025 02:05 — with GitHub Actions Inactive

MarcusSorealheis had a problem deploying to production September 4, 2025 02:05 — with GitHub Actions Error

MarcusSorealheis temporarily deployed to production September 4, 2025 02:14 — with GitHub Actions Inactive

MarcusSorealheis temporarily deployed to production September 4, 2025 02:17 — with GitHub Actions Inactive

MarcusSorealheis temporarily deployed to production September 4, 2025 03:08 — with GitHub Actions Inactive

MarcusSorealheis temporarily deployed to production September 4, 2025 09:05 — with GitHub Actions Inactive

MarcusSorealheis had a problem deploying to production September 4, 2025 09:05 — with GitHub Actions Failure

MarcusSorealheis temporarily deployed to production September 4, 2025 09:05 — with GitHub Actions Inactive

MarcusSorealheis temporarily deployed to production September 4, 2025 14:36 — with GitHub Actions Inactive

MarcusSorealheis temporarily deployed to production September 5, 2025 01:48 — with GitHub Actions Inactive

MarcusSorealheis force-pushed the remote-persistent-workers branch from c3fde9b to be7c3f4 Compare September 5, 2025 07:16

MarcusSorealheis temporarily deployed to production September 5, 2025 07:16 — with GitHub Actions Inactive

MarcusSorealheis had a problem deploying to production September 5, 2025 07:16 — with GitHub Actions Error

MarcusSorealheis temporarily deployed to production September 5, 2025 07:16 — with GitHub Actions Inactive

MarcusSorealheis marked this pull request as ready for review September 5, 2025 07:30

MarcusSorealheis temporarily deployed to production September 5, 2025 07:30 — with GitHub Actions Inactive

palfrey requested changes Sep 11, 2025

View reviewed changes

palfrey reviewed Oct 3, 2025

View reviewed changes

Comment thread integration_tests/persistent-workers-test/package.json Outdated

palfrey reviewed Oct 3, 2025

View reviewed changes

MarcusSorealheis added 8 commits November 4, 2025 09:19

Merge branch 'main' into remote-persistent-workers

81f6d9c

Fix pre-commit formatting issues

afee1ff

- Change clippy cast_possible_truncation from deny to allow - Format Cargo.toml exclude array with alphabetical ordering

Fix additional pre-commit clippy flags

80c64f2

Remove clippy flags that pre-commit doesn't want: - await_holding_lock - elidable_lifetime_names - explicit_into_iter_loop - redundant_closure_for_method_calls - semicolon_if_nothing_returned Change todo from deny (-D) to warn (-W)

fix bazelrc

90ff13d

Fix pre-commit issues

c35b3c2

- Apply generate-bazel-rc auto-fixes to .bazelrc - Remove unused dependencies (hyper-util, tracing-test) - Fix Vale documentation style issues in README - Replace e.g., with 'for example' - Use backticks for filenames and compiler names - Fix namespace terminology

fixed the github actions

899a788

palfrey reviewed Nov 5, 2025

View reviewed changes

Comment thread nativelink-store/MODULE.bazel

palfrey Nov 5, 2025

Copy link
Copy Markdown

Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure why this is here?

Conversation

MarcusSorealheis commented Sep 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

How Has This Been Tested?

Checklist

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MarcusSorealheis commented Nov 6, 2025

Uh oh!

MarcusSorealheis commented Nov 16, 2025

Uh oh!

This branch was previously deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

MarcusSorealheis commented Sep 3, 2025 •

edited

Loading