Repository navigation
state_exec: deep-data work a slice per step, and the budget by kind - #658
Merged
Merged
Conversation
… P0) Declare the state_exec s-expression grammar in docs/STATE_EXEC_GRAMMAR.md (W3C EBNF, Unicode source rules, literal tables, and a table of every intentional change from the hand-written parser, tagged by phase P2-P4). The parser itself is unchanged; later phases build it from this grammar. state_exec_grammar_test tags each case with the phase it holds from: landed cases must hold, pending cases must still differ, the doc's EBNF must be closed, and every production needs a case. state_exec_corpus_test pins how the 597 DSL scripts the repo carries parse today (state_exec_corpus/, regenerated with gen_state_exec_corpus.py and CVC_UPDATE_GOLDENS=1). Also correct parser.h (no quasiquote) and the developer guide (false and #f read as nil, bytes literals exist).
Spec only: parse_options::max_depth on parse()/parse_all()/parser and execute_options::max_parse_depth for submitted scripts, both from P2. Records why the limit exists (the reader recurses on the native stack; the evaluators are stackless) and the corpus's real depths (max 13, median 2).
Spec only. max_depth resolves from state_exec.schedulers.<id>.parser.max_depth, then state_exec.defaults.parser.max_depth, then 256, matching scheduler::load_settings(); parse_options remains for callers without a tree. Drops the execute_options::max_parse_depth idea so the tree is the single source.
…grammar P1b) The parser now stops at parser.max_depth nested lists and quotes (default 256) with a parse_error, instead of recursing until the native stack overflows. Like the scheduler's other settings, the limit lives on the state tree: state_exec.schedulers.<id>.parser.max_depth, then state_exec.defaults.parser.max_depth, then 256; a missing, malformed or zero value is skipped. - parser: parse_options (max_depth) on parse()/parse_all()/parser, a depth guard on lists and quotes, and parse_options_from_state(). - scheduler: load_settings() resolves parser.* and publishes the effective value under state_exec.schedulers.<id>.parser.max_depth; execute() parses with it. - async_scheduler (the app's exec scheduler) reads the defaults key on every submit; Ariadne predicates read it from the app tree. - stackless evaluators: create_state(script, env, parse_options). The grammar test moves from a numeric kLandedPhase to named phases; the 257-deep case is P1b and holds now. Tests cover the default, a custom depth, quotes, 100000 open parens, the error position, tree resolution, bad values, and both schedulers.
…e read) Lists and quotes were parsed by native recursion, so input depth was bounded by the thread's stack. The parser now keeps open lists and quotes on a heap stack of frames and reads in a loop: native stack depth is constant however deeply the input nests (a test reads a million open parens and reports the unclosed list instead of crashing). The loop is a resumable read(out, budget): it reads up to `budget` tokens and returns datum / end / more, carrying open frames between calls, so a scheduler can parse a script a slice at a time. parse() and parse_all() call it with no budget. Output is unchanged, quirks included (a stray top-level `)` still reads as a symbol until P2): the conformance cases, the 597-script corpus and the evaluator suites all pass. parser.max_depth stays on the state tree; it now bounds parsed data for the code that still walks values recursively (list destruction, the snapshot codec, to_string) rather than the parser's own stack. parse_fib20_expr: about 3.4 us vs 3.1 us for the recursive parser (median of 6, MSVC Release, loaded machine).
The stackless parser's state is an input position plus a heap stack of open frames, so it can pause anywhere. New setting parser.step_tokens (on the state tree, resolved like parser.max_depth; default 0) lets a scheduler parse a submitted script inside the new process instead of at submit: each scheduler step reads N tokens, and the step that reaches the end sets up evaluation exactly as parsing up front would have. - Reading steps are process steps: they count against max_steps and max_time and interleave with other processes. - A syntax error surfaces like a runtime error: the async scheduler kills the process with it as the exit error; the sync scheduler throws it from step()/run(). With step_tokens = 0, execute() still throws. - Signals and watch events wait until the script is read (their handlers run on the evaluation stack, which doesn't exist yet). - The snapshot codec stores a mid-read process's source and settings and re-reads it after a restore. Adds parse_options::step_tokens, script_reader (owns its source and drives parser::read() a budget at a time) and evaluator_state::reading; stackless_evaluator::step() reads until the script is read. Tests cover the reader, the evaluator, the codec and both schedulers (results, syntax errors, max_steps, deferred signals).
…ens) app::exec_scheduler(), which runs Ariadne's actions and residents, now defaults parser.step_tokens to 1024, so it parses submitted scripts inside their processes. async_scheduler gains set_parser_defaults(): the built-in values under its watch root's state_exec.defaults.parser.* keys. A plain async_scheduler (pycvc's Exec) and the sync scheduler stay at 0, so they still report syntax errors at submit. Why 1024: every script the repo carries reads in one slice (the largest is 159 tokens, .ari scripts at most 24), so a normal action pays one extra step of its 5000; a slice costs about 80 us at the parser's measured ~80 ns/token, so a pathologically large script is spread over steps and frames instead of stalling a drain. Ariadne now says why a process died: a killed action's warning carries its exit error, and the per-frame sweep reports a killed resident (a syntax error in its body, a runtime error, its per-step cap) once, instead of letting it vanish silently.
…uler
pycvc's Exec runs programs on a private async_scheduler that already
watches the app's state tree, but its built-in parser defaults were
plain parse_options{} while app::exec_scheduler() read scripts in
1024-token steps. Both now take async_scheduler::host_parser_defaults(),
one definition (with the reasoning for 1024) for every libcvc host, and
the app tree's state_exec.defaults.parser.* keys override both.
Exec.run() still raises a RuntimeError with the parse error's message:
from the killed process's exit error when the script is read in steps,
or from execute() when the tree sets step_tokens to 0.
Tests: test_pycvc_exec.py checks the parse error's message and that
max_depth / step_tokens on the app tree configure Exec; the integration
test pins the app scheduler to host_parser_defaults().
pycvc's state_exec surface was only Exec; the parser settings these PRs add had no Python counterpart, so the defaults weren't even observable. In pycvc's direct-wrap style (the real C++ types, no facade): - pycvc.parse_options: the real cvc::state_exec::parse_options struct (max_depth, step_tokens), C++ defaults included. - pycvc.async_scheduler.host_parser_defaults(): the real static, declared to SWIG on its own (async_scheduler isn't wrapped yet, so no constructor). - pycvc.parse_options_from_state(app, scheduler_id="", base=...): the C++ resolver, taking the app where C++ takes the root (pycvc's convention). - Exec.parser_defaults() / Exec.set_parser_defaults(): the same calls as the async_scheduler Exec wraps. test_pycvc_exec.py covers each against the C++ behaviour; the README gains a state_exec section. Built locally (SWIG 4.4.1, Python 3.11): exec, exec_async, exec_coro and ari tests, 26 passed.
…ut native recursion The parser's nesting limit (parser.max_depth, default 256) exists because the reader recursed; it can rise only once nothing else recurses once per level of a value. Freeing, printing, comparing, copying, macro-expanding and snapshotting a value all did. - Release: value_tag and environment destructors drop the last owner of a list, dict, closure, generator or enclosing scope through detail::release_nested. The first release on a thread runs a loop over a local queue (8 owners inline, the rest on the heap); owners dropped while it frees something are queued, not freed inline. Dropping an owner that is not the last is still a plain decrement. - environment::lookup and set_existing are loops. - to_string, values_equal, deep_copy (state-data-get) and the stackless evaluator's macro substitute keep their work on heap stacks. The 1000-level caps of to_string, values_equal and deep_copy are gone: they did not protect small stacks (to_string overflowed a 256 KiB stack at 800 levels) and would have become functional limits once max_depth rises. to_string's 64 KiB length cap stays. - Snapshot codec: encode_value and decode_value are iterative. cvc::state itself recurses once per tree level on every write (fullName() and the childChanged climb to the root), so a list or dict more than 8 levels below where its stretch of nesting began goes into a __spill__ table under the encode root, referenced by __type__ "spill_ref" and the entry's index. Values at most 8 levels deep encode exactly as before. A value that contains itself throws instead of overflowing; decode rejects bad or reused references. __len__ is written once per container, not once per element. New state_exec_deep_data_test builds values 100000 levels deep at runtime and drops, prints, compares, macro-expands and snapshots them (10000 levels for the codec) on a 256 KiB thread. state_exec_bench_test gains ValueDropThroughput.
Freeing, printing, comparing, copying and macro-expanding a value no longer recurse, but their work still grows with the value. Like reading a script (parser.step_tokens), a scheduler can now run it a slice per step, so it counts against max_steps and max_time and other processes run between the slices. Each process also reports what its budget went on: reading, evaluating or deep-data work. - data_job (new): the walks as resumable jobs -- compare, print, copy, concat, macro substitution. A job holds the lists and dicts it walks by owning pointer and reads them by index, so a value changed between two slices changes the answer, never memory safety. to_string, values_equal and the state-data-get copy are these walks run whole. - data.step_nodes on the state tree (data_options, data_options_from_state), resolved like parser.*. The sync scheduler resolves and publishes it in load_settings(); the async scheduler reads it on every submit over set_data_defaults(). The app's exec scheduler and pycvc's Exec use host_data_defaults(): 1024 nodes a step (a slice is 0.05-0.4 ms at the measured 45-380 ns a node). - With N > 0, =, !=, str, str-concat, print and state-data-get hand their walk to the evaluator (run_data_job), and macro expansion does the same. One slice runs in the call's step; whatever is left runs in data steps while the call's frame waits. - Freeing: last owners dropped during a step go to the process's release backlog, freed at the step's end with what is left of the slice. More than that is freed in data steps before evaluation resumes. - Signals and watch events wait while a walk is pending. Snapshots keep data_step_nodes; a walk in flight runs again after a restore. - Budget by kind: evaluation_stats counts steps by step_kind (eval, read, data), deep-data nodes, and the seconds the schedulers charge to each kind. process_info, inspect and the new (budget-usage) report them. - pycvc: data_options, async_scheduler.host_data_defaults(), data_options_from_state and Exec.data_defaults/set_data_defaults, under the C++ names. New state_exec_data_steps_test covers the jobs (sliced = whole, values changed between slices, sharing and cycles), the evaluator (native calls, macros, freeing, max_steps, snapshots) and both schedulers (settings, max_steps, interleaving, signals, budget-usage/inspect). state_exec_bench_test gains DataWalkThroughput.
Wrap the real cvc::state_exec::parse_options_from_state(state&, id, base) instead of a pycvc adapter that took the app: Python passes the tree root, pycvc.State.instance(app), exactly as C++ passes cvc::state&. This is the first decision of the pycvc/C++ parity approach (C++ cvc::state& parameters take a pycvc.State in Python).
…e State root ffa9a26 made pycvc's parse_options_from_state the real C++ function, taking a pycvc.State root as C++ takes a cvc::state&. data_options_from_state follows: pycvc.i declares the real cvc::state_exec::data_options_from_state(cvc::state &root, ...) next to it, and the app-taking adapter in pycvc_exec.h/.cpp is gone. Exec.data_defaults()/set_data_defaults() are unchanged. The test and the README use pycvc.State.instance(app).
This was referenced Oct 17, 2026
# Conflicts: # docs/STATE_EXEC_GRAMMAR.md # src/cvc/tests/CMakeLists.txt # src/cvc/tests/state_exec_grammar_test.cpp
…xec-data-steps #653's branch merged master (bc20f8e), which brings #657 as 3003734. That is the content this branch already carried as the cherry-pick 4d3bc6b, so in every file where #658 builds on #657 (types.h/.cpp, intrinsics.cpp, stackless_evaluator.cpp, state_exec_bench_test.cpp) #653's side was byte-identical to 4d3bc6b, and #658's version stands. In src/cvc/tests/CMakeLists.txt, master's new test targets join #658's state_exec_data_steps_test lines.
# Conflicts: # src/cvc/state/state_exec/scheduler.cpp
…into HEAD # Conflicts: # src/cvc/state/state_exec/scheduler.cpp
…c-data-steps #653's branch picked up #660 (stdlib modules in every environment, with a per-call allocation limit). Conflicts: - scheduler.cpp load_settings(): data.step_nodes and limits.max_alloc_bytes, both resolved and both published. - types.cpp: the release backlog and the allocation guard, side by side. - builtins.cpp str-concat: #658's sliced path for list/dict arguments, then #660's checked fast path. The sliced path built its string in a concat job without #660's check, so passing the same big string many times next to one list would get past the limit. The concat job now checks every append against eval_alloc_check (the guard is armed around each scheduler step, data steps included), naming the builtin (str-concat, print): make_concat_job() takes it. New DataJob.ConcatChecksTheAllocationLimit.
Retargeting a PR is an 'edited' event, which ci.yml does not run on.
transfix
enabled auto-merge (squash)
October 18, 2026 01:58
This was referenced Oct 18, 2026
transfix
added a commit
that referenced
this pull request
Oct 18, 2026
…rap #658's data settings and budget fields on the schedulers
transfix
added a commit
that referenced
this pull request
Oct 18, 2026
…cles (#667) * state_exec: grammar spec, conformance cases and parse corpus (grammar P0) Declare the state_exec s-expression grammar in docs/STATE_EXEC_GRAMMAR.md (W3C EBNF, Unicode source rules, literal tables, and a table of every intentional change from the hand-written parser, tagged by phase P2-P4). The parser itself is unchanged; later phases build it from this grammar. state_exec_grammar_test tags each case with the phase it holds from: landed cases must hold, pending cases must still differ, the doc's EBNF must be closed, and every production needs a case. state_exec_corpus_test pins how the 597 DSL scripts the repo carries parse today (state_exec_corpus/, regenerated with gen_state_exec_corpus.py and CVC_UPDATE_GOLDENS=1). Also correct parser.h (no quasiquote) and the developer guide (false and #f read as nil, bytes literals exist). * state_exec grammar: make the nesting limit configurable (default 256) Spec only: parse_options::max_depth on parse()/parse_all()/parser and execute_options::max_parse_depth for submitted scripts, both from P2. Records why the limit exists (the reader recurses on the native stack; the evaluators are stackless) and the corpus's real depths (max 13, median 2). * state_exec grammar: parser settings live on the state tree Spec only. max_depth resolves from state_exec.schedulers.<id>.parser.max_depth, then state_exec.defaults.parser.max_depth, then 256, matching scheduler::load_settings(); parse_options remains for callers without a tree. Drops the execute_options::max_parse_depth idea so the tree is the single source. * state_exec: parser settings on the state tree, with a nesting limit (grammar P1b) The parser now stops at parser.max_depth nested lists and quotes (default 256) with a parse_error, instead of recursing until the native stack overflows. Like the scheduler's other settings, the limit lives on the state tree: state_exec.schedulers.<id>.parser.max_depth, then state_exec.defaults.parser.max_depth, then 256; a missing, malformed or zero value is skipped. - parser: parse_options (max_depth) on parse()/parse_all()/parser, a depth guard on lists and quotes, and parse_options_from_state(). - scheduler: load_settings() resolves parser.* and publishes the effective value under state_exec.schedulers.<id>.parser.max_depth; execute() parses with it. - async_scheduler (the app's exec scheduler) reads the defaults key on every submit; Ariadne predicates read it from the app tree. - stackless evaluators: create_state(script, env, parse_options). The grammar test moves from a numeric kLandedPhase to named phases; the 257-deep case is P1b and holds now. Tests cover the default, a custom depth, quotes, 100000 open parens, the error position, tree resolution, bad values, and both schedulers. * state_exec: make the parser stackless (explicit frame stack, resumable read) Lists and quotes were parsed by native recursion, so input depth was bounded by the thread's stack. The parser now keeps open lists and quotes on a heap stack of frames and reads in a loop: native stack depth is constant however deeply the input nests (a test reads a million open parens and reports the unclosed list instead of crashing). The loop is a resumable read(out, budget): it reads up to `budget` tokens and returns datum / end / more, carrying open frames between calls, so a scheduler can parse a script a slice at a time. parse() and parse_all() call it with no budget. Output is unchanged, quirks included (a stray top-level `)` still reads as a symbol until P2): the conformance cases, the 597-script corpus and the evaluator suites all pass. parser.max_depth stays on the state tree; it now bounds parsed data for the code that still walks values recursively (list destruction, the snapshot codec, to_string) rather than the parser's own stack. parse_fib20_expr: about 3.4 us vs 3.1 us for the recursive parser (median of 6, MSVC Release, loaded machine). * state_exec: read submitted scripts inside the process, a slice per step The stackless parser's state is an input position plus a heap stack of open frames, so it can pause anywhere. New setting parser.step_tokens (on the state tree, resolved like parser.max_depth; default 0) lets a scheduler parse a submitted script inside the new process instead of at submit: each scheduler step reads N tokens, and the step that reaches the end sets up evaluation exactly as parsing up front would have. - Reading steps are process steps: they count against max_steps and max_time and interleave with other processes. - A syntax error surfaces like a runtime error: the async scheduler kills the process with it as the exit error; the sync scheduler throws it from step()/run(). With step_tokens = 0, execute() still throws. - Signals and watch events wait until the script is read (their handlers run on the evaluation stack, which doesn't exist yet). - The snapshot codec stores a mid-read process's source and settings and re-reads it after a restore. Adds parse_options::step_tokens, script_reader (owns its source and drives parser::read() a budget at a time) and evaluator_state::reading; stackless_evaluator::step() reads until the script is read. Tests cover the reader, the evaluator, the codec and both schedulers (results, syntax errors, max_steps, deferred signals). * state_exec: the app's exec scheduler reads scripts in steps (1024 tokens) app::exec_scheduler(), which runs Ariadne's actions and residents, now defaults parser.step_tokens to 1024, so it parses submitted scripts inside their processes. async_scheduler gains set_parser_defaults(): the built-in values under its watch root's state_exec.defaults.parser.* keys. A plain async_scheduler (pycvc's Exec) and the sync scheduler stay at 0, so they still report syntax errors at submit. Why 1024: every script the repo carries reads in one slice (the largest is 159 tokens, .ari scripts at most 24), so a normal action pays one extra step of its 5000; a slice costs about 80 us at the parser's measured ~80 ns/token, so a pathologically large script is spread over steps and frames instead of stalling a drain. Ariadne now says why a process died: a killed action's warning carries its exit error, and the per-frame sweep reports a killed resident (a syntax error in its body, a runtime error, its per-step cap) once, instead of letting it vanish silently. * pycvc: give Exec the same state_exec defaults as the app's exec scheduler pycvc's Exec runs programs on a private async_scheduler that already watches the app's state tree, but its built-in parser defaults were plain parse_options{} while app::exec_scheduler() read scripts in 1024-token steps. Both now take async_scheduler::host_parser_defaults(), one definition (with the reasoning for 1024) for every libcvc host, and the app tree's state_exec.defaults.parser.* keys override both. Exec.run() still raises a RuntimeError with the parse error's message: from the killed process's exit error when the script is read in steps, or from execute() when the tree sets step_tokens to 0. Tests: test_pycvc_exec.py checks the parse error's message and that max_depth / step_tokens on the app tree configure Exec; the integration test pins the app scheduler to host_parser_defaults(). * pycvc: expose the state_exec parser-settings surface under its C++ names pycvc's state_exec surface was only Exec; the parser settings these PRs add had no Python counterpart, so the defaults weren't even observable. In pycvc's direct-wrap style (the real C++ types, no facade): - pycvc.parse_options: the real cvc::state_exec::parse_options struct (max_depth, step_tokens), C++ defaults included. - pycvc.async_scheduler.host_parser_defaults(): the real static, declared to SWIG on its own (async_scheduler isn't wrapped yet, so no constructor). - pycvc.parse_options_from_state(app, scheduler_id="", base=...): the C++ resolver, taking the app where C++ takes the root (pycvc's convention). - Exec.parser_defaults() / Exec.set_parser_defaults(): the same calls as the async_scheduler Exec wraps. test_pycvc_exec.py covers each against the C++ behaviour; the README gains a state_exec section. Built locally (SWIG 4.4.1, Python 3.11): exec, exec_async, exec_coro and ari tests, 26 passed. * state_exec: free, print, compare, copy and snapshot deep values without native recursion The parser's nesting limit (parser.max_depth, default 256) exists because the reader recursed; it can rise only once nothing else recurses once per level of a value. Freeing, printing, comparing, copying, macro-expanding and snapshotting a value all did. - Release: value_tag and environment destructors drop the last owner of a list, dict, closure, generator or enclosing scope through detail::release_nested. The first release on a thread runs a loop over a local queue (8 owners inline, the rest on the heap); owners dropped while it frees something are queued, not freed inline. Dropping an owner that is not the last is still a plain decrement. - environment::lookup and set_existing are loops. - to_string, values_equal, deep_copy (state-data-get) and the stackless evaluator's macro substitute keep their work on heap stacks. The 1000-level caps of to_string, values_equal and deep_copy are gone: they did not protect small stacks (to_string overflowed a 256 KiB stack at 800 levels) and would have become functional limits once max_depth rises. to_string's 64 KiB length cap stays. - Snapshot codec: encode_value and decode_value are iterative. cvc::state itself recurses once per tree level on every write (fullName() and the childChanged climb to the root), so a list or dict more than 8 levels below where its stretch of nesting began goes into a __spill__ table under the encode root, referenced by __type__ "spill_ref" and the entry's index. Values at most 8 levels deep encode exactly as before. A value that contains itself throws instead of overflowing; decode rejects bad or reused references. __len__ is written once per container, not once per element. New state_exec_deep_data_test builds values 100000 levels deep at runtime and drops, prints, compares, macro-expands and snapshots them (10000 levels for the codec) on a 256 KiB thread. state_exec_bench_test gains ValueDropThroughput. * state_exec: deep-data work a slice per step, and the budget by kind Freeing, printing, comparing, copying and macro-expanding a value no longer recurse, but their work still grows with the value. Like reading a script (parser.step_tokens), a scheduler can now run it a slice per step, so it counts against max_steps and max_time and other processes run between the slices. Each process also reports what its budget went on: reading, evaluating or deep-data work. - data_job (new): the walks as resumable jobs -- compare, print, copy, concat, macro substitution. A job holds the lists and dicts it walks by owning pointer and reads them by index, so a value changed between two slices changes the answer, never memory safety. to_string, values_equal and the state-data-get copy are these walks run whole. - data.step_nodes on the state tree (data_options, data_options_from_state), resolved like parser.*. The sync scheduler resolves and publishes it in load_settings(); the async scheduler reads it on every submit over set_data_defaults(). The app's exec scheduler and pycvc's Exec use host_data_defaults(): 1024 nodes a step (a slice is 0.05-0.4 ms at the measured 45-380 ns a node). - With N > 0, =, !=, str, str-concat, print and state-data-get hand their walk to the evaluator (run_data_job), and macro expansion does the same. One slice runs in the call's step; whatever is left runs in data steps while the call's frame waits. - Freeing: last owners dropped during a step go to the process's release backlog, freed at the step's end with what is left of the slice. More than that is freed in data steps before evaluation resumes. - Signals and watch events wait while a walk is pending. Snapshots keep data_step_nodes; a walk in flight runs again after a restore. - Budget by kind: evaluation_stats counts steps by step_kind (eval, read, data), deep-data nodes, and the seconds the schedulers charge to each kind. process_info, inspect and the new (budget-usage) report them. - pycvc: data_options, async_scheduler.host_data_defaults(), data_options_from_state and Exec.data_defaults/set_data_defaults, under the C++ names. New state_exec_data_steps_test covers the jobs (sliced = whole, values changed between slices, sharing and cycles), the evaluator (native calls, macros, freeing, max_steps, snapshots) and both schedulers (settings, max_steps, interleaving, signals, budget-usage/inspect). state_exec_bench_test gains DataWalkThroughput. * pycvc: parse_options_from_state takes the State root, as in C++ Wrap the real cvc::state_exec::parse_options_from_state(state&, id, base) instead of a pycvc adapter that took the app: Python passes the tree root, pycvc.State.instance(app), exactly as C++ passes cvc::state&. This is the first decision of the pycvc/C++ parity approach (C++ cvc::state& parameters take a pycvc.State in Python). * state_exec: load the stdlib modules in every environment, with a per-call allocation limit The string, bytes, math and collections modules in stdlib.cpp were never reachable from DSL programs: nothing outside tests referenced stdlib_registry, so (string.length "x") failed with "unknown function" on both schedulers, in Ariadne and in pycvc's Exec. - stdlib_registry::shared_scope() builds one frozen scope with every stdlib function, once per process. builtins::make_default_environment() links it as its outer scope, so every host reaches the modules without copying them into each environment. - environment::frozen: set() refuses a frozen scope, and set_existing() binds in the outermost unfrozen scope instead of writing into it. A program's (set string.length ...) or (defun string.upper ...) shadows the name for that program only, and threads can share the scope. - Ariadne predicates get the pure, bounded subset: string.length, byte-length, char-at, substring, upper, lower, trim, starts-with, ends-with, contains; bytes.length, byte-at; every math.* except math.random. The materializers stay out until they run as scheduled, sliced work. - Per-call allocation limit. A native call builds its result inside one step, where max_memory (between steps, state-tree writes only) cannot see it. New eval_alloc_guard/eval_alloc_check, next to the evaluation deadline: the schedulers arm it per step with the tighter of the tree setting limits.max_alloc_bytes (state_exec.schedulers.<id>, then state_exec.defaults; default 16 MiB, 0 = none of its own) and what is left of max_memory. string.split/join/replace, bytes.concat, collections.range/zip/flatten/dict-merge, str-concat and collect check their result size before allocating. - collections.range counts its elements in unsigned arithmetic, so it no longer overflows at the int64 ends. string.replace builds its result in one pass. collections.unique and dict-merge poll the deadline. - Developer guide section 11 rewritten: how the modules load, the allocation limit, the predicate subset, and corrected examples. * ariadne_runtime_test: keep master's include-comment alignment A full-file clang-format realigned an untouched include comment, which the changed-lines check then flagged. * ariadne: the action memory budget is a tree setting, default 4 MiB kActionMaxBytes (a fixed 256 KiB) becomes ariadne.limits.action_max_memory, read on every submission (bytes; 0 = no limit; a missing or unparsable value keeps the 4 MiB default). With the per-step allocation limit taking what is left of max_memory, 256 KiB would also have capped every native call in an action at 256 KiB. * tests: repair the state_exec_stdlib_test registration from the cherry-pick * state_exec: max_memory covers the measured heap; collect reference cycles max_memory counted only state-tree writes, so a program could hold any amount of memory in lists, dicts, strings and closures. And reference counting never freed a cycle: every defun, (set f (lambda ...)), let-local recursive lambda, defclass, or (append l l) leaked its whole scope. Heap share (heap.h): - A process is charged its share of each node it can reach: size x (its references / the node's owner count), passed down to children in Kahn order. Shares over all holders add up to the node's size, so a value shared by k processes (a broadcast, a fork) costs each about 1/k. - The measurement is a data job (#658) the scheduler starts after a step when natives reported enough allocation since the last one (max(64 KiB, last/2)), when that could cross the budget, or every limits.heap_check_steps steps (default 4096). It runs as the process's own sliced data steps. Kills use measurements only, so dropped temporaries never count. Only processes with max_memory > 0 are measured. - eval_alloc_guard::allocated() collects what natives report (every eval_alloc_check, plus eval_alloc_note for copies like cons and slice). - max_memory = state-tree bytes + heap share; the per-step allocation limit takes what is left of both. process_info and inspect report heap_memory. Cycle collector: - Trial deletion, as in CPython, over a per-thread registry of the environments closures/generators/class methods capture and the containers changed in place to hold a container. Objects with more owners than their internal references are held from outside and kept, with all they reach; the rest is cleared, and frees. - defclass's natives are named functor types (closure_natives.h) so the collector sees their captures via std::function::target. - Schedulers collect between steps when as many objects were tracked as survived the last collection (min 256), or within a second of a process ending. collect_cycles() is a no-op inside an evaluator step. - A finished process releases its evaluation state once its last step is over (process::release_state), so what it leaked becomes garbage. - Ariadne reaps finished actions from the app scheduler; it never did, so every action's process lived forever. * state_exec: keep heap tracking and the step guard off the common path - The cycle registry dedups by address (checking the control block, so a reused address is not mistaken for a live object) instead of only the last entry: round-robin processes appending to their lists re-tracked every append. - A collection is due only after max(256, survivors, last walk / 8) new objects, so its walk costs a bounded amount per tracked object even when candidates reach a large heap. - Schedulers check whether a collection is due every 64 steps. - The per-step allocation guard is armed only when it changes something (a memory budget, or a non-default limits.max_alloc_bytes), and heap accounting runs only for processes with a budget: no memory-tracker lookups or thread-local writes per step otherwise. Against #658's build (MSVC Release, medians of 5 interleaved runs): 100000 closures in a loop 0.98x, a plain counting loop 1.01x; state_exec_bench_test memory_pressure_200 1.31x -> 1.10x (it now frees each finished process's lists inside the timed run). * Run the master CI on #658 after retargeting it to master Retargeting a PR is an 'edited' event, which ci.yml does not run on.
transfix
added a commit
that referenced
this pull request
Oct 18, 2026
…rap #658's step_kind and the per-kind stats
transfix
added a commit
that referenced
this pull request
Oct 18, 2026
…edicates get them (#665) string.split/join/replace/encode, bytes.slice/concat/decode and all of collections.* do their work as data jobs, a slice per step (data.step_nodes) counted against max_steps/max_time, and Ariadne predicates now allow every stdlib function except math.random. - data_job call protocol: a job stops for a function call; run_data_job makes native calls inside the slice, the evaluator runs a closure as frames above the waiting call. collections.map/filter/reduce, apply and send take DSL functions (call_function in data_job.h). - pending_data_work records its waiting frame and chains; the codec snapshots up to the outermost waiting frame; fork refuses while such work is pending (not for a heap measurement). - collections.sort is a stable merge sort, unique uses a sliced equality walk, dict-merge keeps a key index. - Ariadne predicates always slice (state_exec.defaults.data.step_nodes, 1024 for 0). - With #667: heap_measure gets a real base and returns from step_data's finished branch. - Fixes a master bug from #658: a native function called by map that left deep-data work replaced the whole result.
This was referenced Oct 18, 2026
transfix
added a commit
that referenced
this pull request
Oct 18, 2026
…y-depth values) (#679) * state_exec: grammar spec, conformance cases and parse corpus (grammar P0) Declare the state_exec s-expression grammar in docs/STATE_EXEC_GRAMMAR.md (W3C EBNF, Unicode source rules, literal tables, and a table of every intentional change from the hand-written parser, tagged by phase P2-P4). The parser itself is unchanged; later phases build it from this grammar. state_exec_grammar_test tags each case with the phase it holds from: landed cases must hold, pending cases must still differ, the doc's EBNF must be closed, and every production needs a case. state_exec_corpus_test pins how the 597 DSL scripts the repo carries parse today (state_exec_corpus/, regenerated with gen_state_exec_corpus.py and CVC_UPDATE_GOLDENS=1). Also correct parser.h (no quasiquote) and the developer guide (false and #f read as nil, bytes literals exist). * state_exec grammar: make the nesting limit configurable (default 256) Spec only: parse_options::max_depth on parse()/parse_all()/parser and execute_options::max_parse_depth for submitted scripts, both from P2. Records why the limit exists (the reader recurses on the native stack; the evaluators are stackless) and the corpus's real depths (max 13, median 2). * state_exec grammar: parser settings live on the state tree Spec only. max_depth resolves from state_exec.schedulers.<id>.parser.max_depth, then state_exec.defaults.parser.max_depth, then 256, matching scheduler::load_settings(); parse_options remains for callers without a tree. Drops the execute_options::max_parse_depth idea so the tree is the single source. * state_exec: parser settings on the state tree, with a nesting limit (grammar P1b) The parser now stops at parser.max_depth nested lists and quotes (default 256) with a parse_error, instead of recursing until the native stack overflows. Like the scheduler's other settings, the limit lives on the state tree: state_exec.schedulers.<id>.parser.max_depth, then state_exec.defaults.parser.max_depth, then 256; a missing, malformed or zero value is skipped. - parser: parse_options (max_depth) on parse()/parse_all()/parser, a depth guard on lists and quotes, and parse_options_from_state(). - scheduler: load_settings() resolves parser.* and publishes the effective value under state_exec.schedulers.<id>.parser.max_depth; execute() parses with it. - async_scheduler (the app's exec scheduler) reads the defaults key on every submit; Ariadne predicates read it from the app tree. - stackless evaluators: create_state(script, env, parse_options). The grammar test moves from a numeric kLandedPhase to named phases; the 257-deep case is P1b and holds now. Tests cover the default, a custom depth, quotes, 100000 open parens, the error position, tree resolution, bad values, and both schedulers. * state_exec: make the parser stackless (explicit frame stack, resumable read) Lists and quotes were parsed by native recursion, so input depth was bounded by the thread's stack. The parser now keeps open lists and quotes on a heap stack of frames and reads in a loop: native stack depth is constant however deeply the input nests (a test reads a million open parens and reports the unclosed list instead of crashing). The loop is a resumable read(out, budget): it reads up to `budget` tokens and returns datum / end / more, carrying open frames between calls, so a scheduler can parse a script a slice at a time. parse() and parse_all() call it with no budget. Output is unchanged, quirks included (a stray top-level `)` still reads as a symbol until P2): the conformance cases, the 597-script corpus and the evaluator suites all pass. parser.max_depth stays on the state tree; it now bounds parsed data for the code that still walks values recursively (list destruction, the snapshot codec, to_string) rather than the parser's own stack. parse_fib20_expr: about 3.4 us vs 3.1 us for the recursive parser (median of 6, MSVC Release, loaded machine). * state_exec: read submitted scripts inside the process, a slice per step The stackless parser's state is an input position plus a heap stack of open frames, so it can pause anywhere. New setting parser.step_tokens (on the state tree, resolved like parser.max_depth; default 0) lets a scheduler parse a submitted script inside the new process instead of at submit: each scheduler step reads N tokens, and the step that reaches the end sets up evaluation exactly as parsing up front would have. - Reading steps are process steps: they count against max_steps and max_time and interleave with other processes. - A syntax error surfaces like a runtime error: the async scheduler kills the process with it as the exit error; the sync scheduler throws it from step()/run(). With step_tokens = 0, execute() still throws. - Signals and watch events wait until the script is read (their handlers run on the evaluation stack, which doesn't exist yet). - The snapshot codec stores a mid-read process's source and settings and re-reads it after a restore. Adds parse_options::step_tokens, script_reader (owns its source and drives parser::read() a budget at a time) and evaluator_state::reading; stackless_evaluator::step() reads until the script is read. Tests cover the reader, the evaluator, the codec and both schedulers (results, syntax errors, max_steps, deferred signals). * state_exec: the app's exec scheduler reads scripts in steps (1024 tokens) app::exec_scheduler(), which runs Ariadne's actions and residents, now defaults parser.step_tokens to 1024, so it parses submitted scripts inside their processes. async_scheduler gains set_parser_defaults(): the built-in values under its watch root's state_exec.defaults.parser.* keys. A plain async_scheduler (pycvc's Exec) and the sync scheduler stay at 0, so they still report syntax errors at submit. Why 1024: every script the repo carries reads in one slice (the largest is 159 tokens, .ari scripts at most 24), so a normal action pays one extra step of its 5000; a slice costs about 80 us at the parser's measured ~80 ns/token, so a pathologically large script is spread over steps and frames instead of stalling a drain. Ariadne now says why a process died: a killed action's warning carries its exit error, and the per-frame sweep reports a killed resident (a syntax error in its body, a runtime error, its per-step cap) once, instead of letting it vanish silently. * pycvc: give Exec the same state_exec defaults as the app's exec scheduler pycvc's Exec runs programs on a private async_scheduler that already watches the app's state tree, but its built-in parser defaults were plain parse_options{} while app::exec_scheduler() read scripts in 1024-token steps. Both now take async_scheduler::host_parser_defaults(), one definition (with the reasoning for 1024) for every libcvc host, and the app tree's state_exec.defaults.parser.* keys override both. Exec.run() still raises a RuntimeError with the parse error's message: from the killed process's exit error when the script is read in steps, or from execute() when the tree sets step_tokens to 0. Tests: test_pycvc_exec.py checks the parse error's message and that max_depth / step_tokens on the app tree configure Exec; the integration test pins the app scheduler to host_parser_defaults(). * pycvc: expose the state_exec parser-settings surface under its C++ names pycvc's state_exec surface was only Exec; the parser settings these PRs add had no Python counterpart, so the defaults weren't even observable. In pycvc's direct-wrap style (the real C++ types, no facade): - pycvc.parse_options: the real cvc::state_exec::parse_options struct (max_depth, step_tokens), C++ defaults included. - pycvc.async_scheduler.host_parser_defaults(): the real static, declared to SWIG on its own (async_scheduler isn't wrapped yet, so no constructor). - pycvc.parse_options_from_state(app, scheduler_id="", base=...): the C++ resolver, taking the app where C++ takes the root (pycvc's convention). - Exec.parser_defaults() / Exec.set_parser_defaults(): the same calls as the async_scheduler Exec wraps. test_pycvc_exec.py covers each against the C++ behaviour; the README gains a state_exec section. Built locally (SWIG 4.4.1, Python 3.11): exec, exec_async, exec_coro and ari tests, 26 passed. * pycvc: parse_options_from_state takes the State root, as in C++ Wrap the real cvc::state_exec::parse_options_from_state(state&, id, base) instead of a pycvc adapter that took the app: Python passes the tree root, pycvc.State.instance(app), exactly as C++ passes cvc::state&. This is the first decision of the pycvc/C++ parity approach (C++ cvc::state& parameters take a pycvc.State in Python). * pycvc: state_exec parity part 1 (values, parsing, parse_error, value codec) Start bringing pycvc's state_exec API to the C++ surface under the C++ names (docs/PYCVC_STATE_EXEC_PARITY.md: the audit of all 20 state_exec headers, the agreed approach, and the PR plan). - One strict value_t <-> Python conversion (pycvc_value.h), used by every API and by Exec: native types for data, pycvc.symbol (the real struct) for symbols, and pycvc.value (the real value_t, opaque) for closures, native fns, data objects and generators, which convert back unchanged. Non-str dict keys and unsupported types raise TypeError, ints beyond 64 bits OverflowError, and nesting past 1000 levels or a cycle ValueError. This replaces the lossy converter (symbols became str, closures "<closure>", anything else str(obj)), which could also put a null item in a list on invalid UTF-8 and leave an OverflowError pending. - pycvc.parse / parse_all, pycvc.parse_error (a RuntimeError with .line and .col), pycvc.script_reader, to_string, values_equal, and the value codec encode_value / decode_value on a State node. - Exec.run raises pycvc.parse_error for a syntax error whether the script was parsed at submit or read inside the process (it re-parses a killed process's script to recover the position). - The pycvc README's Status/Design sections described a v0 facade; they now describe the direct-wrap design. test_pycvc_state_exec.py (13 tests) covers each against the C++ behaviour. Locally: those plus pycvc's exec, async, coro, ari, state and core suites pass (87), except test_volume_rawiv_roundtrip, which fails the same way on the unmodified build (a Windows temp-file lock). * pycvc: state_exec parity part 2 (environments, builtins, schedulers) Wrap the scheduler surface of cvc::state_exec under the C++ names: - environment (shared, as environment_ptr) and builtins: make_default_environment, and register_fn taking a Python callable or a pycvc.value holding a native fn. - execute_options (on_complete is a Python callable, signal_handlers a dict), process_info, scheduler_stats, percent_encode/percent_decode. - process_status and scheduling_policy as enum.IntEnum classes; an enum argument must be a member, so to_string(process_status) and to_string(value_t) keep their shared name. - scheduler and async_scheduler as real classes with their non-coroutine members. run/step and sync_run/sync_step release the GIL; Python callbacks retake it. - Exec.scheduler() returns Exec's async_scheduler, which keeps the Exec alive. Empty optionals are None, except for values: get_result, pop_pending_message and environment.lookup raise KeyError, since None is nil. The value_t typecheck now runs last, so execute(str) picks the script overload. test_pycvc_state_exec_sched.py repeats the C++ scheduler tests on both classes. The parity doc and README are updated. * Rerun CI on master (the merge push ran while the PR still targeted #653's branch) * pycvc: environment.frozen (#660), and the set_existing note now that its comment is fixed * pycvc: state_exec parity part 3 (evaluators, processes, policy, intrinsics) Wrap the rest of cvc::state_exec's public surface under the C++ names: - stackless_evaluator and evaluator, with evaluator_state, eval_frame, eval_phase (an IntEnum) and evaluation_stats; evaluation_interrupted and evaluation_timeout as RuntimeError subclasses. The evaluators release the GIL while they evaluate, so another thread can interrupt, pause or resume them. on_complete is a Python callable. - The process struct (shared, as process_ptr), watch_event and watch_handler_entry, make_process, and the scheduler calls that take them: current_process() and queue_watch_event(). scheduler_base is the common base of the two schedulers. - encode_/decode_environment and encode_/decode_evaluator_state. - resource_policy (mode as an IntEnum), process_limits, validate_limits. - intrinsics_context, channel_policy, register_intrinsics, apply_chroot, resolve_channel_key, make_future, future_channel_of. Every function register_intrinsics adds keeps its context alive, and the context keeps the scheduler and channel policy assigned to it. - From #660: kDefaultMaxAllocBytes, eval_alloc_limit, eval_alloc_check, alloc_bytes, scheduler.max_alloc_bytes(), max_alloc_bytes_from_state. A reference into an object (evaluator_state.stats, evaluator.stats(), process.state) keeps its owner alive; other members read and write as copies. evaluator_state is move-only, so it is moved into Python. test_pycvc_state_exec_eval.py repeats the C++ tests. The parity doc, whose inventory now includes what landed on master since the audit, and the README are updated. * pycvc: state_exec parity part 4 (data jobs, heap, stdlib registry, any-depth values) - The value conversion walks with an explicit stack instead of recursing, so a value nested any depth crosses both ways (the parser's max_depth default is going to 100000). A list or dict shared within a value converts to one object, both ways: sharing survives the trip, a shared structure converts in time proportional to its size, and a DSL list that holds itself becomes a Python list that holds itself. A Python value that holds itself still raises ValueError. - The deep-data jobs (#658, #665): data_job, the make_*_job factories, run_to_completion, run_data_job (a Python callable as `then`), call_function, is_compound, count_data_nodes, kDataNodeBytes. A job is handed to Python owned (std::unique_ptr); `who` is kept for good, as C++ holds the pointer. - The heap (#667): heap_options and its tree settings, measure_heap_share, make_heap_share_job, heap_remeasure_bytes, and the per-thread cycle collector; the heap fields on process and process_info, evaluator_state.heap_measured, stackless_evaluator.global_env(), eval_alloc_note. - stdlib_registry: standard() (a copy), shared_scope(), register_function (a Python callable or a pycvc.value native), list_*, import_module, lookup_qualified (KeyError where C++ returns nullptr). - SWIG_STD_MOVE moves on MSVC too (it reports __cplusplus 199711L without /Zc:__cplusplus), so a unique_ptr argument can be passed. test_pycvc_state_exec_data.py covers the new calls, and test_pycvc_state_exec.py the any-depth, sharing conversion. The parity doc and README are updated. * pycvc: wrap #682/#683's call_function continuation, make_generator_job and the native nesting limits
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Based on master, which now has #653 (sliced reading), #657 (iterative walkers) and #660 (per-call allocation limit). The branch merged them in rather than rebasing, because #665 and #667 build on it; the diff against master is this PR's own 27 files. With #660 in, the sliced
str-concatcheckslimits.max_alloc_byteson every append, as the unsliced one does (ConcatChecksTheAllocationLimit).Why
#657 took deep-data handling off the native stack, but its cost still grows with the value: comparing, printing, copying, macro-expanding or freeing a million-node value is one long step. This makes that work schedulable, the same way #653 schedules reading:
max_stepsandmax_time;What
data_job(new). The walks as resumable jobs: compare, print, copy, concat and macro substitution. A job holds the lists and dicts it walks by owning pointer and reads them by index, checking bounds at each node. A value changed between two slices (by another process) changes the answer, never memory safety.to_string,values_equaland thestate-data-getcopy are these same walks run whole.data.step_nodeson the state tree. It lives indata_options/data_options_from_state(), resolved likeparser.*.load_settings().set_data_defaults().Execusehost_data_defaults(): 1024 nodes a step.=,!=,str,str-concat,printandstate-data-gethand their walk to the evaluator (run_data_job), and macro expansion does the same. One slice runs in the call's own step. Anything left runs in data steps while the call's frame waits. A walk under N nodes takes no extra steps.data_step_nodes. A walk in flight runs again after a restore.evaluation_statscounts steps bystep_kind(eval, read, data), deep-data nodes, and the seconds the schedulers charge to each kind.process_infogainsread_steps,eval_steps,data_steps,data_nodes,read_time,eval_timeanddata_time.inspectreports them, and the new(budget-usage)reports them for the running process.data_options,async_scheduler.host_data_defaults(),data_options_from_state(root, ...)(the real C++ function, taking apycvc.Stateroot,pycvc.State.instance(app), as state_exec: read submitted scripts inside the process, a slice per step #653 now does forparse_options_from_state), andExec.data_defaults()/set_data_defaults().docs/STATE_EXEC_GRAMMAR.mdgains "Deep data on the executor" and "Budget by kind".Why 1024 nodes
state_exec_bench_testDataWalkThroughput(MSVC Release, medians of 5) measures, per node: ~45 ns to free, ~70 ns to print, ~250 ns to compare, ~380 ns to copy.Tests
state_exec_data_steps_test(17).max_stepsmid-walk, a snapshot mid-walk.max_steps, other processes finishing between slices, a signal waiting for the walk, andbudget-usage/inspectfields from a scheduled process.test_pycvc_exec.py. Exec, exec_async and exec_coro give 30 passed.test_pycvc_exec.pygives 20 passed.cvc_cli_testpasses 171 of 173;Vol2ImgExportsSlicesandImg2VolImportsSlicesfail the same way on a master build here (no PNG slice I/O in the local deps);check_test_targets.pyis OK;test_pycvc_ari/test_pycvc_app/test_pycvc_asyncpass.test_pycvc.py::test_volume_rawiv_roundtripfails here with a Windows file-lock error removing its temp.rawivfile. That is volume I/O, which this doesn't touch; it was not checked on the base.Overhead with slicing off
state_exec_bench_test, this branch against its base commit 4d3bc6b, interleaved over 8 rounds; values are medians on a busy machine:Not in this PR
cvc::state's per-levelfullName()/childChangedwork. That pairs with making thatcvc::staterecursion iterative, a separate libcvc-wide change.collections.*,state-setcoercion) still run whole inside their step. Their nodes are counted.