Skip to content

fix(worker): measure and kill an action by its process tree #2922

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
rohnnyjoy wants to merge 3 commits into
base: main
Choose a base branch
from
Open

fix(worker): measure and kill an action by its process tree #2922

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b982f6d
fix(worker): measure and kill an action by its process tree
rohnnyjoy Oct 10, 2026
266aea6
chore(proto): regenerate the remote_execution bindings
rohnnyjoy Oct 16, 2026
13a4494
test(worker): skip the detached-job tests without setsid or python3
rohnnyjoy Oct 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion nativelink-config/src/cas_server.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -916,7 +916,7 @@ pub struct EndpointConfig {
pub enum MemoryEnforcement {
/// Measure only.
None,
/// Kill the action's process group once two consecutive samples exceed
/// Kill the action's processes once two consecutive samples exceed
/// the reservation plus headroom, or one sample exceeds twice it.
#[default]
Soft,
Expand Down
4 changes: 2 additions & 2 deletions nativelink-proto/com/github/trace_machina/nativelink/remote_execution/worker_api.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -178,8 +178,8 @@ message ActionResourceUsage {
string worker_id = 4;

/// Total CPU time consumed by the action process tree, in milliseconds.
/// User and system time combined, summed across every process in the
/// action's process group.
/// User and system time combined, of every process the action started:
/// the kernel's total of everything the action's stub reaped.
uint64 cpu_time_ms = 5;

/// Wall time from spawn to exit, in milliseconds; cores used is
Expand Down
4 changes: 2 additions & 2 deletions nativelink-proto/genproto/com.github.trace_machina.nativelink.remote_execution.pb.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -146,8 +146,8 @@ pub struct ActionResourceUsage {
#[prost(string, tag = "4")]
pub worker_id: ::prost::alloc::string::String,
/// / Total CPU time consumed by the action process tree, in milliseconds.
/// / User and system time combined, summed across every process in the
/// / action's process group.
/// / User and system time combined, of every process the action started:
/// / the kernel's total of everything the action's stub reaped.
#[prost(uint64, tag = "5")]
pub cpu_time_ms: u64,
/// / Wall time from spawn to exit, in milliseconds; cores used is
Expand Down
6 changes: 5 additions & 1 deletion nativelink-worker/BUILD.bazel
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,13 +19,17 @@ rust_library(
"src/persistent_worker/mod.rs",
"src/persistent_worker/pool.rs",
"src/persistent_worker/protocol.rs",
"src/process_tree.rs",
"src/qos.rs",
"src/reaper.rs",
"src/running_actions_manager.rs",
"src/worker_api_client_wrapper.rs",
"src/worker_utils.rs",
] + select({
"@platforms//os:linux": ["src/namespace_utils.rs"],
"@platforms//os:linux": [
"src/action_stub.rs",
"src/namespace_utils.rs",
],
"//conditions:default": [],
}),
proc_macro_deps = [
Expand Down
Loading
Loading