Repository navigation
state_exec: the stdlib materializers run as sliced data steps, and predicates get them - #665
Merged
Merged
Conversation
… P0) Declare the state_exec s-expression grammar in docs/STATE_EXEC_GRAMMAR.md (W3C EBNF, Unicode source rules, literal tables, and a table of every intentional change from the hand-written parser, tagged by phase P2-P4). The parser itself is unchanged; later phases build it from this grammar. state_exec_grammar_test tags each case with the phase it holds from: landed cases must hold, pending cases must still differ, the doc's EBNF must be closed, and every production needs a case. state_exec_corpus_test pins how the 597 DSL scripts the repo carries parse today (state_exec_corpus/, regenerated with gen_state_exec_corpus.py and CVC_UPDATE_GOLDENS=1). Also correct parser.h (no quasiquote) and the developer guide (false and #f read as nil, bytes literals exist).
Spec only: parse_options::max_depth on parse()/parse_all()/parser and execute_options::max_parse_depth for submitted scripts, both from P2. Records why the limit exists (the reader recurses on the native stack; the evaluators are stackless) and the corpus's real depths (max 13, median 2).
Spec only. max_depth resolves from state_exec.schedulers.<id>.parser.max_depth, then state_exec.defaults.parser.max_depth, then 256, matching scheduler::load_settings(); parse_options remains for callers without a tree. Drops the execute_options::max_parse_depth idea so the tree is the single source.
…grammar P1b) The parser now stops at parser.max_depth nested lists and quotes (default 256) with a parse_error, instead of recursing until the native stack overflows. Like the scheduler's other settings, the limit lives on the state tree: state_exec.schedulers.<id>.parser.max_depth, then state_exec.defaults.parser.max_depth, then 256; a missing, malformed or zero value is skipped. - parser: parse_options (max_depth) on parse()/parse_all()/parser, a depth guard on lists and quotes, and parse_options_from_state(). - scheduler: load_settings() resolves parser.* and publishes the effective value under state_exec.schedulers.<id>.parser.max_depth; execute() parses with it. - async_scheduler (the app's exec scheduler) reads the defaults key on every submit; Ariadne predicates read it from the app tree. - stackless evaluators: create_state(script, env, parse_options). The grammar test moves from a numeric kLandedPhase to named phases; the 257-deep case is P1b and holds now. Tests cover the default, a custom depth, quotes, 100000 open parens, the error position, tree resolution, bad values, and both schedulers.
…e read) Lists and quotes were parsed by native recursion, so input depth was bounded by the thread's stack. The parser now keeps open lists and quotes on a heap stack of frames and reads in a loop: native stack depth is constant however deeply the input nests (a test reads a million open parens and reports the unclosed list instead of crashing). The loop is a resumable read(out, budget): it reads up to `budget` tokens and returns datum / end / more, carrying open frames between calls, so a scheduler can parse a script a slice at a time. parse() and parse_all() call it with no budget. Output is unchanged, quirks included (a stray top-level `)` still reads as a symbol until P2): the conformance cases, the 597-script corpus and the evaluator suites all pass. parser.max_depth stays on the state tree; it now bounds parsed data for the code that still walks values recursively (list destruction, the snapshot codec, to_string) rather than the parser's own stack. parse_fib20_expr: about 3.4 us vs 3.1 us for the recursive parser (median of 6, MSVC Release, loaded machine).
The stackless parser's state is an input position plus a heap stack of open frames, so it can pause anywhere. New setting parser.step_tokens (on the state tree, resolved like parser.max_depth; default 0) lets a scheduler parse a submitted script inside the new process instead of at submit: each scheduler step reads N tokens, and the step that reaches the end sets up evaluation exactly as parsing up front would have. - Reading steps are process steps: they count against max_steps and max_time and interleave with other processes. - A syntax error surfaces like a runtime error: the async scheduler kills the process with it as the exit error; the sync scheduler throws it from step()/run(). With step_tokens = 0, execute() still throws. - Signals and watch events wait until the script is read (their handlers run on the evaluation stack, which doesn't exist yet). - The snapshot codec stores a mid-read process's source and settings and re-reads it after a restore. Adds parse_options::step_tokens, script_reader (owns its source and drives parser::read() a budget at a time) and evaluator_state::reading; stackless_evaluator::step() reads until the script is read. Tests cover the reader, the evaluator, the codec and both schedulers (results, syntax errors, max_steps, deferred signals).
…ens) app::exec_scheduler(), which runs Ariadne's actions and residents, now defaults parser.step_tokens to 1024, so it parses submitted scripts inside their processes. async_scheduler gains set_parser_defaults(): the built-in values under its watch root's state_exec.defaults.parser.* keys. A plain async_scheduler (pycvc's Exec) and the sync scheduler stay at 0, so they still report syntax errors at submit. Why 1024: every script the repo carries reads in one slice (the largest is 159 tokens, .ari scripts at most 24), so a normal action pays one extra step of its 5000; a slice costs about 80 us at the parser's measured ~80 ns/token, so a pathologically large script is spread over steps and frames instead of stalling a drain. Ariadne now says why a process died: a killed action's warning carries its exit error, and the per-frame sweep reports a killed resident (a syntax error in its body, a runtime error, its per-step cap) once, instead of letting it vanish silently.
…uler
pycvc's Exec runs programs on a private async_scheduler that already
watches the app's state tree, but its built-in parser defaults were
plain parse_options{} while app::exec_scheduler() read scripts in
1024-token steps. Both now take async_scheduler::host_parser_defaults(),
one definition (with the reasoning for 1024) for every libcvc host, and
the app tree's state_exec.defaults.parser.* keys override both.
Exec.run() still raises a RuntimeError with the parse error's message:
from the killed process's exit error when the script is read in steps,
or from execute() when the tree sets step_tokens to 0.
Tests: test_pycvc_exec.py checks the parse error's message and that
max_depth / step_tokens on the app tree configure Exec; the integration
test pins the app scheduler to host_parser_defaults().
pycvc's state_exec surface was only Exec; the parser settings these PRs add had no Python counterpart, so the defaults weren't even observable. In pycvc's direct-wrap style (the real C++ types, no facade): - pycvc.parse_options: the real cvc::state_exec::parse_options struct (max_depth, step_tokens), C++ defaults included. - pycvc.async_scheduler.host_parser_defaults(): the real static, declared to SWIG on its own (async_scheduler isn't wrapped yet, so no constructor). - pycvc.parse_options_from_state(app, scheduler_id="", base=...): the C++ resolver, taking the app where C++ takes the root (pycvc's convention). - Exec.parser_defaults() / Exec.set_parser_defaults(): the same calls as the async_scheduler Exec wraps. test_pycvc_exec.py covers each against the C++ behaviour; the README gains a state_exec section. Built locally (SWIG 4.4.1, Python 3.11): exec, exec_async, exec_coro and ari tests, 26 passed.
…ut native recursion The parser's nesting limit (parser.max_depth, default 256) exists because the reader recursed; it can rise only once nothing else recurses once per level of a value. Freeing, printing, comparing, copying, macro-expanding and snapshotting a value all did. - Release: value_tag and environment destructors drop the last owner of a list, dict, closure, generator or enclosing scope through detail::release_nested. The first release on a thread runs a loop over a local queue (8 owners inline, the rest on the heap); owners dropped while it frees something are queued, not freed inline. Dropping an owner that is not the last is still a plain decrement. - environment::lookup and set_existing are loops. - to_string, values_equal, deep_copy (state-data-get) and the stackless evaluator's macro substitute keep their work on heap stacks. The 1000-level caps of to_string, values_equal and deep_copy are gone: they did not protect small stacks (to_string overflowed a 256 KiB stack at 800 levels) and would have become functional limits once max_depth rises. to_string's 64 KiB length cap stays. - Snapshot codec: encode_value and decode_value are iterative. cvc::state itself recurses once per tree level on every write (fullName() and the childChanged climb to the root), so a list or dict more than 8 levels below where its stretch of nesting began goes into a __spill__ table under the encode root, referenced by __type__ "spill_ref" and the entry's index. Values at most 8 levels deep encode exactly as before. A value that contains itself throws instead of overflowing; decode rejects bad or reused references. __len__ is written once per container, not once per element. New state_exec_deep_data_test builds values 100000 levels deep at runtime and drops, prints, compares, macro-expands and snapshots them (10000 levels for the codec) on a 256 KiB thread. state_exec_bench_test gains ValueDropThroughput.
Freeing, printing, comparing, copying and macro-expanding a value no longer recurse, but their work still grows with the value. Like reading a script (parser.step_tokens), a scheduler can now run it a slice per step, so it counts against max_steps and max_time and other processes run between the slices. Each process also reports what its budget went on: reading, evaluating or deep-data work. - data_job (new): the walks as resumable jobs -- compare, print, copy, concat, macro substitution. A job holds the lists and dicts it walks by owning pointer and reads them by index, so a value changed between two slices changes the answer, never memory safety. to_string, values_equal and the state-data-get copy are these walks run whole. - data.step_nodes on the state tree (data_options, data_options_from_state), resolved like parser.*. The sync scheduler resolves and publishes it in load_settings(); the async scheduler reads it on every submit over set_data_defaults(). The app's exec scheduler and pycvc's Exec use host_data_defaults(): 1024 nodes a step (a slice is 0.05-0.4 ms at the measured 45-380 ns a node). - With N > 0, =, !=, str, str-concat, print and state-data-get hand their walk to the evaluator (run_data_job), and macro expansion does the same. One slice runs in the call's step; whatever is left runs in data steps while the call's frame waits. - Freeing: last owners dropped during a step go to the process's release backlog, freed at the step's end with what is left of the slice. More than that is freed in data steps before evaluation resumes. - Signals and watch events wait while a walk is pending. Snapshots keep data_step_nodes; a walk in flight runs again after a restore. - Budget by kind: evaluation_stats counts steps by step_kind (eval, read, data), deep-data nodes, and the seconds the schedulers charge to each kind. process_info, inspect and the new (budget-usage) report them. - pycvc: data_options, async_scheduler.host_data_defaults(), data_options_from_state and Exec.data_defaults/set_data_defaults, under the C++ names. New state_exec_data_steps_test covers the jobs (sliced = whole, values changed between slices, sharing and cycles), the evaluator (native calls, macros, freeing, max_steps, snapshots) and both schedulers (settings, max_steps, interleaving, signals, budget-usage/inspect). state_exec_bench_test gains DataWalkThroughput.
Wrap the real cvc::state_exec::parse_options_from_state(state&, id, base) instead of a pycvc adapter that took the app: Python passes the tree root, pycvc.State.instance(app), exactly as C++ passes cvc::state&. This is the first decision of the pycvc/C++ parity approach (C++ cvc::state& parameters take a pycvc.State in Python).
…call allocation limit The string, bytes, math and collections modules in stdlib.cpp were never reachable from DSL programs: nothing outside tests referenced stdlib_registry, so (string.length "x") failed with "unknown function" on both schedulers, in Ariadne and in pycvc's Exec. - stdlib_registry::shared_scope() builds one frozen scope with every stdlib function, once per process. builtins::make_default_environment() links it as its outer scope, so every host reaches the modules without copying them into each environment. - environment::frozen: set() refuses a frozen scope, and set_existing() binds in the outermost unfrozen scope instead of writing into it. A program's (set string.length ...) or (defun string.upper ...) shadows the name for that program only, and threads can share the scope. - Ariadne predicates get the pure, bounded subset: string.length, byte-length, char-at, substring, upper, lower, trim, starts-with, ends-with, contains; bytes.length, byte-at; every math.* except math.random. The materializers stay out until they run as scheduled, sliced work. - Per-call allocation limit. A native call builds its result inside one step, where max_memory (between steps, state-tree writes only) cannot see it. New eval_alloc_guard/eval_alloc_check, next to the evaluation deadline: the schedulers arm it per step with the tighter of the tree setting limits.max_alloc_bytes (state_exec.schedulers.<id>, then state_exec.defaults; default 16 MiB, 0 = none of its own) and what is left of max_memory. string.split/join/replace, bytes.concat, collections.range/zip/flatten/dict-merge, str-concat and collect check their result size before allocating. - collections.range counts its elements in unsigned arithmetic, so it no longer overflows at the int64 ends. string.replace builds its result in one pass. collections.unique and dict-merge poll the deadline. - Developer guide section 11 rewritten: how the modules load, the allocation limit, the predicate subset, and corrected examples.
…e State root ffa9a26 made pycvc's parse_options_from_state the real C++ function, taking a pycvc.State root as C++ takes a cvc::state&. data_options_from_state follows: pycvc.i declares the real cvc::state_exec::data_options_from_state(cvc::state &root, ...) next to it, and the app-taking adapter in pycvc_exec.h/.cpp is gone. Exec.data_defaults()/set_data_defaults() are unchanged. The test and the README use pycvc.State.instance(app).
…edicates get them
string.split/join/replace/encode, bytes.slice/concat/decode and every
collections.* function now do their work as data jobs (data_job.h), a slice
per scheduler step, counted against max_steps and max_time as data steps.
Ariadne predicates can therefore use them.
- A slice is data.step_nodes nodes: one per element visited, one per 256
bytes (kDataNodeBytes) scanned or copied. The per-call allocation check
still comes first, as soon as the result's size is known (after a sliced
count for split, replace, join and flatten). A job reads its string
arguments in place during the call's own slice, and copies them only if
it outlives the call (own_inputs).
- A job can stop for a function call (take_call / call_returned).
run_data_job makes native calls inside the slice; the stackless evaluator
makes the rest. A DSL closure runs as frames above the waiting call, in
eval steps, and the job resumes in the data step after it returns. So
collections.map/filter/reduce take a lambda or a defun as well as a native
function, with the argument order unchanged. return and break in such a
function stop at the waiting call. Work that a native callee leaves of its
own chains to the job that called it (deferred_data_job::waits_on,
pending_data_work::outer / feeds_outer).
- pending_data_work records its waiting frame (base, frame_args), and
evaluator_state::data_work is the innermost of a chain. The snapshot
codec leaves out the frames of a function a job was calling and trims the
waiting frame to its arguments, so a restore runs the call again. fork
returns -1 while deep-data work is pending, since the job is not copied.
- collections.sort is a stable merge sort (n log n nodes). unique compares
lists and dicts with a sliced equality walk. dict-merge keeps a key index,
so it is linear, with the same result.
- With no evaluator (C++ calling the registry's function directly) a job
makes native calls itself, with deferral off, and refuses a closure.
- Ariadne: every stdlib function except math.random is in kAllowed. Each
predicate eval takes data_step_nodes from state_exec.defaults.data.step_nodes,
with 1024 for 0 or unset, so its step and time caps cut a huge call. A
lambda passed to map runs on the predicate's own, gated evaluator.
- Tests:
- through a scheduler: the same results at slices 0/1/3/64/1024, data
steps, max_steps and max_time, other processes between slices, the
allocation check, lambdas;
- on the evaluator: frames above the call, a snapshot mid-lambda, the
special-form gate, fork;
- Ariadne predicates and the action lane, and pycvc Exec.
- Bench StdlibJobThroughput.
- Docs: developer guide section 11 (Sliced work, In Ariadne predicates,
collections with lambdas) and "Deep data on the executor" in
STATE_EXEC_GRAMMAR.md.
… feat/state-exec-sliced-stdlib
The builtins apply and send called only native functions. A closure threw
"closure dispatch requires evaluator", while collections.map/filter/reduce
already took one.
- New call_function(fn, args, who) in data_job.h. A native function is
called directly. A closure goes through a one-call data job (call_walk),
so inside a stackless evaluator step its body runs as the process's own
frames, gated and budgeted like the rest of the program. With no
evaluator a closure throws, naming the caller.
- apply uses it, and so does send for a closure stored as a method.
- Tests:
- scheduler scripts at every slice size: apply with a lambda, return
inside it, apply of collections.map, map over apply, send to a lambda
method;
- the no-evaluator errors;
- the Ariadne action lane;
- pycvc, with a lambda and a registered Python function.
- Docs: the builtins list and "Sliced work" in the developer guide; the
function-calls bullet in STATE_EXEC_GRAMMAR.md.
…g builds PredicateMaterializerIsCutByTheEvalBudget expected a 120000-element collections.range to finish inside the 5 ms predicate time cap once the slice is back at 1024 nodes. The Linux and macOS Debug lanes were too slow for that and hid the widget. That half of the test now checks a 1000-element range, which fits in any build; the half that shows a one-node slice being cut by the eval budget is unchanged.
# Conflicts: # docs/STATE_EXEC_GRAMMAR.md # src/cvc/tests/CMakeLists.txt # src/cvc/tests/state_exec_grammar_test.cpp
…xec-data-steps #653's branch merged master (bc20f8e), which brings #657 as 3003734. That is the content this branch already carried as the cherry-pick 4d3bc6b, so in every file where #658 builds on #657 (types.h/.cpp, intrinsics.cpp, stackless_evaluator.cpp, state_exec_bench_test.cpp) #653's side was byte-identical to 4d3bc6b, and #658's version stands. In src/cvc/tests/CMakeLists.txt, master's new test targets join #658's state_exec_data_steps_test lines.
# Conflicts: # src/cvc/state/state_exec/scheduler.cpp
…into HEAD # Conflicts: # src/cvc/state/state_exec/scheduler.cpp
…c-data-steps #653's branch picked up #660 (stdlib modules in every environment, with a per-call allocation limit). Conflicts: - scheduler.cpp load_settings(): data.step_nodes and limits.max_alloc_bytes, both resolved and both published. - types.cpp: the release backlog and the allocation guard, side by side. - builtins.cpp str-concat: #658's sliced path for list/dict arguments, then #660's checked fast path. The sliced path built its string in a concat job without #660's check, so passing the same big string many times next to one list would get past the limit. The concat job now checks every append against eval_alloc_check (the guard is armed around each scheduler step, data steps included), naming the builtin (str-concat, print): make_concat_job() takes it. New DataJob.ConcatChecksTheAllocationLimit.
… feat/state-exec-sliced-stdlib
Retargeting a PR is an 'edited' event, which ci.yml does not run on.
… feat/state-exec-sliced-stdlib
This was referenced Oct 18, 2026
…eat/state-exec-sliced-stdlib - step_data: the work loop from this branch; a finished heap_measure records state.heap_measured and returns, as in #667. The comment on pending_data_work names all three kinds. - heap.cpp: a measurement gets base = the stack's size and frame_args = the top frame's values. It waits on no frame, and an aggregate-built one (base 0) would have made a snapshot keep no frames. - state_value_codec: a heap_measure work never shortens the encoded stack. - fork (both schedulers): a pending measurement does not refuse it; only work a call's frame waits on does. - #667's track_captured_env calls and defclass functors are kept. - Tests: HeapBudgetTest.AMeasurementWaitsOnNoFrame and AProcessForksDuringAMeasurement (which fails with the old fork guard), and DataStepsEvaluator.ANativeFunctionsOwnWorkIsItsCallsValue, the regression the deep-data session pointed out: on master with slicing, (collections.map (list a b) str) came back as one string. - pycvc test and developer guide: both sides kept.
transfix
added a commit
that referenced
this pull request
Oct 18, 2026
transfix
added a commit
that referenced
this pull request
Oct 18, 2026
…it is reported (#680) PredicateMaterializerIsCutByTheEvalBudget (from #665) runs a one-node-a-step collections.range 120000 in a predicate and expects it cut. On the Linux Debug lane the time cap fired inside the job's own deadline poll, which reports "collections.range: evaluation exceeded time budget" through the job's error rather than the evaluator's "exceeded the eval budget" (seen on #678, job 117587719214). Both are the predicate being cut; the test now accepts either.
transfix
added a commit
that referenced
this pull request
Oct 18, 2026
…y-depth values) (#679) * state_exec: grammar spec, conformance cases and parse corpus (grammar P0) Declare the state_exec s-expression grammar in docs/STATE_EXEC_GRAMMAR.md (W3C EBNF, Unicode source rules, literal tables, and a table of every intentional change from the hand-written parser, tagged by phase P2-P4). The parser itself is unchanged; later phases build it from this grammar. state_exec_grammar_test tags each case with the phase it holds from: landed cases must hold, pending cases must still differ, the doc's EBNF must be closed, and every production needs a case. state_exec_corpus_test pins how the 597 DSL scripts the repo carries parse today (state_exec_corpus/, regenerated with gen_state_exec_corpus.py and CVC_UPDATE_GOLDENS=1). Also correct parser.h (no quasiquote) and the developer guide (false and #f read as nil, bytes literals exist). * state_exec grammar: make the nesting limit configurable (default 256) Spec only: parse_options::max_depth on parse()/parse_all()/parser and execute_options::max_parse_depth for submitted scripts, both from P2. Records why the limit exists (the reader recurses on the native stack; the evaluators are stackless) and the corpus's real depths (max 13, median 2). * state_exec grammar: parser settings live on the state tree Spec only. max_depth resolves from state_exec.schedulers.<id>.parser.max_depth, then state_exec.defaults.parser.max_depth, then 256, matching scheduler::load_settings(); parse_options remains for callers without a tree. Drops the execute_options::max_parse_depth idea so the tree is the single source. * state_exec: parser settings on the state tree, with a nesting limit (grammar P1b) The parser now stops at parser.max_depth nested lists and quotes (default 256) with a parse_error, instead of recursing until the native stack overflows. Like the scheduler's other settings, the limit lives on the state tree: state_exec.schedulers.<id>.parser.max_depth, then state_exec.defaults.parser.max_depth, then 256; a missing, malformed or zero value is skipped. - parser: parse_options (max_depth) on parse()/parse_all()/parser, a depth guard on lists and quotes, and parse_options_from_state(). - scheduler: load_settings() resolves parser.* and publishes the effective value under state_exec.schedulers.<id>.parser.max_depth; execute() parses with it. - async_scheduler (the app's exec scheduler) reads the defaults key on every submit; Ariadne predicates read it from the app tree. - stackless evaluators: create_state(script, env, parse_options). The grammar test moves from a numeric kLandedPhase to named phases; the 257-deep case is P1b and holds now. Tests cover the default, a custom depth, quotes, 100000 open parens, the error position, tree resolution, bad values, and both schedulers. * state_exec: make the parser stackless (explicit frame stack, resumable read) Lists and quotes were parsed by native recursion, so input depth was bounded by the thread's stack. The parser now keeps open lists and quotes on a heap stack of frames and reads in a loop: native stack depth is constant however deeply the input nests (a test reads a million open parens and reports the unclosed list instead of crashing). The loop is a resumable read(out, budget): it reads up to `budget` tokens and returns datum / end / more, carrying open frames between calls, so a scheduler can parse a script a slice at a time. parse() and parse_all() call it with no budget. Output is unchanged, quirks included (a stray top-level `)` still reads as a symbol until P2): the conformance cases, the 597-script corpus and the evaluator suites all pass. parser.max_depth stays on the state tree; it now bounds parsed data for the code that still walks values recursively (list destruction, the snapshot codec, to_string) rather than the parser's own stack. parse_fib20_expr: about 3.4 us vs 3.1 us for the recursive parser (median of 6, MSVC Release, loaded machine). * state_exec: read submitted scripts inside the process, a slice per step The stackless parser's state is an input position plus a heap stack of open frames, so it can pause anywhere. New setting parser.step_tokens (on the state tree, resolved like parser.max_depth; default 0) lets a scheduler parse a submitted script inside the new process instead of at submit: each scheduler step reads N tokens, and the step that reaches the end sets up evaluation exactly as parsing up front would have. - Reading steps are process steps: they count against max_steps and max_time and interleave with other processes. - A syntax error surfaces like a runtime error: the async scheduler kills the process with it as the exit error; the sync scheduler throws it from step()/run(). With step_tokens = 0, execute() still throws. - Signals and watch events wait until the script is read (their handlers run on the evaluation stack, which doesn't exist yet). - The snapshot codec stores a mid-read process's source and settings and re-reads it after a restore. Adds parse_options::step_tokens, script_reader (owns its source and drives parser::read() a budget at a time) and evaluator_state::reading; stackless_evaluator::step() reads until the script is read. Tests cover the reader, the evaluator, the codec and both schedulers (results, syntax errors, max_steps, deferred signals). * state_exec: the app's exec scheduler reads scripts in steps (1024 tokens) app::exec_scheduler(), which runs Ariadne's actions and residents, now defaults parser.step_tokens to 1024, so it parses submitted scripts inside their processes. async_scheduler gains set_parser_defaults(): the built-in values under its watch root's state_exec.defaults.parser.* keys. A plain async_scheduler (pycvc's Exec) and the sync scheduler stay at 0, so they still report syntax errors at submit. Why 1024: every script the repo carries reads in one slice (the largest is 159 tokens, .ari scripts at most 24), so a normal action pays one extra step of its 5000; a slice costs about 80 us at the parser's measured ~80 ns/token, so a pathologically large script is spread over steps and frames instead of stalling a drain. Ariadne now says why a process died: a killed action's warning carries its exit error, and the per-frame sweep reports a killed resident (a syntax error in its body, a runtime error, its per-step cap) once, instead of letting it vanish silently. * pycvc: give Exec the same state_exec defaults as the app's exec scheduler pycvc's Exec runs programs on a private async_scheduler that already watches the app's state tree, but its built-in parser defaults were plain parse_options{} while app::exec_scheduler() read scripts in 1024-token steps. Both now take async_scheduler::host_parser_defaults(), one definition (with the reasoning for 1024) for every libcvc host, and the app tree's state_exec.defaults.parser.* keys override both. Exec.run() still raises a RuntimeError with the parse error's message: from the killed process's exit error when the script is read in steps, or from execute() when the tree sets step_tokens to 0. Tests: test_pycvc_exec.py checks the parse error's message and that max_depth / step_tokens on the app tree configure Exec; the integration test pins the app scheduler to host_parser_defaults(). * pycvc: expose the state_exec parser-settings surface under its C++ names pycvc's state_exec surface was only Exec; the parser settings these PRs add had no Python counterpart, so the defaults weren't even observable. In pycvc's direct-wrap style (the real C++ types, no facade): - pycvc.parse_options: the real cvc::state_exec::parse_options struct (max_depth, step_tokens), C++ defaults included. - pycvc.async_scheduler.host_parser_defaults(): the real static, declared to SWIG on its own (async_scheduler isn't wrapped yet, so no constructor). - pycvc.parse_options_from_state(app, scheduler_id="", base=...): the C++ resolver, taking the app where C++ takes the root (pycvc's convention). - Exec.parser_defaults() / Exec.set_parser_defaults(): the same calls as the async_scheduler Exec wraps. test_pycvc_exec.py covers each against the C++ behaviour; the README gains a state_exec section. Built locally (SWIG 4.4.1, Python 3.11): exec, exec_async, exec_coro and ari tests, 26 passed. * pycvc: parse_options_from_state takes the State root, as in C++ Wrap the real cvc::state_exec::parse_options_from_state(state&, id, base) instead of a pycvc adapter that took the app: Python passes the tree root, pycvc.State.instance(app), exactly as C++ passes cvc::state&. This is the first decision of the pycvc/C++ parity approach (C++ cvc::state& parameters take a pycvc.State in Python). * pycvc: state_exec parity part 1 (values, parsing, parse_error, value codec) Start bringing pycvc's state_exec API to the C++ surface under the C++ names (docs/PYCVC_STATE_EXEC_PARITY.md: the audit of all 20 state_exec headers, the agreed approach, and the PR plan). - One strict value_t <-> Python conversion (pycvc_value.h), used by every API and by Exec: native types for data, pycvc.symbol (the real struct) for symbols, and pycvc.value (the real value_t, opaque) for closures, native fns, data objects and generators, which convert back unchanged. Non-str dict keys and unsupported types raise TypeError, ints beyond 64 bits OverflowError, and nesting past 1000 levels or a cycle ValueError. This replaces the lossy converter (symbols became str, closures "<closure>", anything else str(obj)), which could also put a null item in a list on invalid UTF-8 and leave an OverflowError pending. - pycvc.parse / parse_all, pycvc.parse_error (a RuntimeError with .line and .col), pycvc.script_reader, to_string, values_equal, and the value codec encode_value / decode_value on a State node. - Exec.run raises pycvc.parse_error for a syntax error whether the script was parsed at submit or read inside the process (it re-parses a killed process's script to recover the position). - The pycvc README's Status/Design sections described a v0 facade; they now describe the direct-wrap design. test_pycvc_state_exec.py (13 tests) covers each against the C++ behaviour. Locally: those plus pycvc's exec, async, coro, ari, state and core suites pass (87), except test_volume_rawiv_roundtrip, which fails the same way on the unmodified build (a Windows temp-file lock). * pycvc: state_exec parity part 2 (environments, builtins, schedulers) Wrap the scheduler surface of cvc::state_exec under the C++ names: - environment (shared, as environment_ptr) and builtins: make_default_environment, and register_fn taking a Python callable or a pycvc.value holding a native fn. - execute_options (on_complete is a Python callable, signal_handlers a dict), process_info, scheduler_stats, percent_encode/percent_decode. - process_status and scheduling_policy as enum.IntEnum classes; an enum argument must be a member, so to_string(process_status) and to_string(value_t) keep their shared name. - scheduler and async_scheduler as real classes with their non-coroutine members. run/step and sync_run/sync_step release the GIL; Python callbacks retake it. - Exec.scheduler() returns Exec's async_scheduler, which keeps the Exec alive. Empty optionals are None, except for values: get_result, pop_pending_message and environment.lookup raise KeyError, since None is nil. The value_t typecheck now runs last, so execute(str) picks the script overload. test_pycvc_state_exec_sched.py repeats the C++ scheduler tests on both classes. The parity doc and README are updated. * Rerun CI on master (the merge push ran while the PR still targeted #653's branch) * pycvc: environment.frozen (#660), and the set_existing note now that its comment is fixed * pycvc: state_exec parity part 3 (evaluators, processes, policy, intrinsics) Wrap the rest of cvc::state_exec's public surface under the C++ names: - stackless_evaluator and evaluator, with evaluator_state, eval_frame, eval_phase (an IntEnum) and evaluation_stats; evaluation_interrupted and evaluation_timeout as RuntimeError subclasses. The evaluators release the GIL while they evaluate, so another thread can interrupt, pause or resume them. on_complete is a Python callable. - The process struct (shared, as process_ptr), watch_event and watch_handler_entry, make_process, and the scheduler calls that take them: current_process() and queue_watch_event(). scheduler_base is the common base of the two schedulers. - encode_/decode_environment and encode_/decode_evaluator_state. - resource_policy (mode as an IntEnum), process_limits, validate_limits. - intrinsics_context, channel_policy, register_intrinsics, apply_chroot, resolve_channel_key, make_future, future_channel_of. Every function register_intrinsics adds keeps its context alive, and the context keeps the scheduler and channel policy assigned to it. - From #660: kDefaultMaxAllocBytes, eval_alloc_limit, eval_alloc_check, alloc_bytes, scheduler.max_alloc_bytes(), max_alloc_bytes_from_state. A reference into an object (evaluator_state.stats, evaluator.stats(), process.state) keeps its owner alive; other members read and write as copies. evaluator_state is move-only, so it is moved into Python. test_pycvc_state_exec_eval.py repeats the C++ tests. The parity doc, whose inventory now includes what landed on master since the audit, and the README are updated. * pycvc: state_exec parity part 4 (data jobs, heap, stdlib registry, any-depth values) - The value conversion walks with an explicit stack instead of recursing, so a value nested any depth crosses both ways (the parser's max_depth default is going to 100000). A list or dict shared within a value converts to one object, both ways: sharing survives the trip, a shared structure converts in time proportional to its size, and a DSL list that holds itself becomes a Python list that holds itself. A Python value that holds itself still raises ValueError. - The deep-data jobs (#658, #665): data_job, the make_*_job factories, run_to_completion, run_data_job (a Python callable as `then`), call_function, is_compound, count_data_nodes, kDataNodeBytes. A job is handed to Python owned (std::unique_ptr); `who` is kept for good, as C++ holds the pointer. - The heap (#667): heap_options and its tree settings, measure_heap_share, make_heap_share_job, heap_remeasure_bytes, and the per-thread cycle collector; the heap fields on process and process_info, evaluator_state.heap_measured, stackless_evaluator.global_env(), eval_alloc_note. - stdlib_registry: standard() (a copy), shared_scope(), register_function (a Python callable or a pycvc.value native), list_*, import_module, lookup_qualified (KeyError where C++ returns nullptr). - SWIG_STD_MOVE moves on MSVC too (it reports __cplusplus 199711L without /Zc:__cplusplus), so a unique_ptr argument can be passed. test_pycvc_state_exec_data.py covers the new calls, and test_pycvc_state_exec.py the any-depth, sharing conversion. The parity doc and README are updated. * pycvc: wrap #682/#683's call_function continuation, make_generator_job and the native nesting limits
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #660, built on #658's data steps. The stdlib functions #660 left out of Ariadne predicates now do their work as sliced data steps, and predicates get them.
Base. Now targets master: #660 and #658 are merged (4908c65, 4e57321). Against master the diff is this PR's 17 files. The change itself is e3f5ee9, cb0460d and 86fa687; the rest are merges.
With #667 (heap accounting, merged first as agreed) — 5eb8826 merges it in:
step_datakeeps this branch's work loop; a finishedheap_measurerecordsheap_measuredand returns, as in state_exec: max_memory covers the measured heap; collect reference cycles #667.baseandframe_args(heap.cpp). It waits on no frame, so a snapshot keeps every frame; the codec also never shortens the stack for one.forkrefuses only for work a call's frame waits on, not for a pending measurement.track_captured_envcalls and defclass functors are kept.HeapBudgetTest.AMeasurementWaitsOnNoFrame,HeapBudgetTest.AProcessForksDuringAMeasurement(fails with the old fork guard).Fixes a master bug from #658 (found by the deep-data session). With
data.step_nodes> 0, a native function called bycollections.map/filter/reducethat left deep-data work of its own (str,=,printon a value bigger than the slice) replaced the whole map's result:(collections.map (list a b) str)came back as one string. Here such work feeds the call's value back into the job. Regression test:DataStepsEvaluator.ANativeFunctionsOwnWorkIsItsCallsValue.Locally on 5eb8826: 27 state_exec/Ariadne/async_lane/state_list/heap suites (1429 tests), pycvc 62, clang-format clean.
What changes
string.split/join/replace/encode,bytes.slice/concat/decodeand everycollections.*function are nowdata_jobs, run a slice per scheduler step (data.step_nodes). The work counts againstmax_stepsandmax_timeas data steps, and other processes run between slices.kDataNodeBytes) scanned or copied.eval_alloc_checkstill comes before any building. For split, replace, join and flatten it runs right after a sliced count.own_inputs).take_call/call_returned).run_data_jobmakes native calls inside the slice.collections.map/filter/reducetake a lambda or a defun. The argument order is still(list fn [init]).return/breakin such a function stop at the waiting call.deferred_data_job::waits_on,pending_data_work::outer/feeds_outer).applyandsendtake a DSL function too (cb0460d). The newcall_function(fn, args, who)indata_job.hcalls a native function directly, and runs a closure through a one-call job as the program's own frames.applyuses it, and so doessendfor a closure stored as a method. A lambda now works wherever a native function does. Before this, both threw "closure dispatch requires evaluator".pending_data_workrecords its waiting frame (base,frame_args), anddata_workis the innermost work of a chain.forkreturns -1 while deep-data work is pending, since the job is not copied.collections.sortis a stable merge sort (n log n nodes). It replacesstd::sort, whose mixed-type comparator wasn't a strict weak order.uniquecompares lists and dicts with a sliced equality walk.dict-mergeuses a key index, so it is linear; the result is the same, including duplicate keys.math.randomis inkAllowed.data_step_nodesfromstate_exec.defaults.data.step_nodes; 0 or unset gives 1024. The predicate therefore always slices, and its step and time caps cut a huge call.collections.mapruns on the predicate's own evaluator, so the special-form gate and the reduced environment hold inside it.Tests (local, MSVC 14.38 Release, from the cvcpkg prefix)
state_exec_stdlib_test, 8 new):max_stepsandmax_time, other processes running between slices;state_exec_data_steps_test, 4 new): frames above the waiting call, a snapshot mid-lambda, the special-form gate, fork.ariadne_runtime_test): predicates run the materializers but notmath.random, a huge call is cut by the eval budget, lambdas stay inside the predicate rules, and the action lane maps with a lambda. state_exec: load the stdlib modules in every environment, with a per-call allocation limit #660's "excludes materializers" test is replaced.test_pycvc_exec.py, 2 new): lambdas and a registered Python function in map/filter/reduce; data steps reported by(budget-usage)at host, 16 and 0 slicing.Results:
cvc_cli_test171/173 (the two ImageMagick slice tests fail on this box's Magick lookup, as on state_exec: load the stdlib modules in every environment, with a per-call allocation limit #660);check_test_targets.pyOK;Bench
StdlibJobThroughput, new, at 1024-node steps:string.encode, including first-touch page faults)mapwith a native functionmapwith(lambda (x) x)Small calls, timed with
cvc execagainst a build of the base commit 811c954 (7 interleaved rounds, medians). Each loop runs 100000 iterations of one small call:collections.mapwithmath.absNot in it
max_memory; state_exec: load the stdlib modules in every environment, with a per-call allocation limit #660 bounds each allocation only.state-setand the codec's own walk still run whole, as in state_exec: deep-data work a slice per step, and the budget by kind #658.