pycvc: state_exec parity part 3 (evaluators, processes, policy, intrinsics) by transfix · Pull Request #675 · transfix/libcvc

transfix · 2026-10-18T02:35:10Z

Targets master. #661 and #671 (parts 1 and 2) merged as 50c6d02 and 1087b09, and master is merged in (5e59800), so the diff here is part 3 only. #679 (part 4) is stacked on this PR.

Part 3 of bringing pycvc's state_exec API to the C++ surface, under the C++ names. With it, everything in docs/PYCVC_STATE_EXEC_PARITY.md's inventory is wrapped or listed as an exception with its reason, except stdlib_registry and the API that #658, #667 and #665 add (PR 4).

This PR

The evaluators: pycvc.stackless_evaluator and pycvc.evaluator.
- With evaluator_state, eval_frame, eval_phase (an IntEnum) and evaluation_stats.
- evaluation_interrupted and evaluation_timeout are RuntimeError subclasses.
- Evaluating releases the GIL, so another thread can call interrupt(), pause() and resume().
- on_complete is a Python callable.
- restrict_special_forms takes any iterable of str, or None; all_special_forms() returns a set.
pycvc.process: shared, as process_ptr.
- With watch_event and watch_handler_entry (also process.watch_event, ...) and make_process().
- The schedulers gain the calls that take it: current_process() and queue_watch_event().
- state is a reference into the process, and inbox reads as a list.
- The steady_clock time points aren't wrapped; elapsed_time() and accumulated_time carry the same information.
pycvc.scheduler_base: the common base of the two schedulers, which is what intrinsics_context.sched holds.
The codec: encode_environment/decode_environment and encode_evaluator_state/decode_evaluator_state.
pycvc.resource_policy: its mode is pycvc.resource_policy.mode, an IntEnum. Also process_limits and validate_limits.
The intrinsics:
- intrinsics_context, channel_policy (global is global_, since global is a Python keyword) and register_intrinsics;
- apply_chroot, resolve_channel_key, make_future and future_channel_of.
From state_exec: load the stdlib modules in every environment, with a per-call allocation limit #660, which landed after the audit:
- kDefaultMaxAllocBytes, eval_alloc_limit, eval_alloc_check and alloc_bytes;
- scheduler.max_alloc_bytes() and max_alloc_bytes_from_state.
A Python function a program calls runs on the stepping thread, where the step's limit is armed, so it can check the limit the way a builtin does.

Lifetimes (approach items 10 and 11 in the doc)

A reference keeps its owner alive. That covers evaluator_state.stats, evaluator.stats() and process.state. Other members read and write as copies.
register_intrinsics holds its context. C++ requires the caller to keep the context valid while the environment is used. In Python, each function it adds holds the context, and the context holds the scheduler and channel policy assigned to it. A test drops the context and the scheduler, collects garbage, then runs the environment's intrinsics on another scheduler.
evaluator_state is move-only (its stats hold a mutex). SWIG returns objects by copy, so a typemap moves a returned state into its Python object.

Not wrapped, with reasons in the doc

eval_alloc_guard: RAII that must nest last-in first-out, which a Python object's lifetime doesn't promise.
return_exception: the internal unwinding of (return ...).
document_scope: its slots are C++ templates.
park_on_channel: it takes raw process pointers and is valid only inside a running native function.
The context's tracker, document and watches.

Tests

bindings/pycvc/test_pycvc_state_exec_eval.py (18 tests, registered with CTest) repeats the C++ tests in state_exec_stackless_evaluator_test.cpp, state_exec_evaluator_test.cpp, state_exec_codec_test.cpp and state_exec_intrinsics_test.cpp. Among them:

an interrupt from another thread;
pause and resume;
a mid-run snapshot that resumes to the right result (with builtins restored on every scope chain, as PauseResumeProducesCorrectResult does);
the process struct's fields;
current_process() seen from a Python function during a run;
max_alloc_bytes from the tree, and the step's allocation limit seen from a Python function;
validate_limits;
intrinsics over a context, under a chroot, and after the context and scheduler are dropped.

Local results (Windows MSVC 14.38 Release, SWIG 4.4.1, Python 3.11, cvcpkg prefix, on the merged master):

Run	Result
`test_pycvc_state_exec_eval.py`	18 passed
all non-GL pycvc suites, including parts 1 and 2	276 passed, 1 failed: `test_volume_rawiv_roundtrip`, the Windows file lock #663 fixes (not in this stack)
SWIG	no warnings from `pycvc.i`

clang-format 18 is clean on the C++ files, and ruff (E, F, W) is clean on the new test.

pycvc_gl: the build (cvcGL, pycvc and pycvc_gl, VTK-Python bridge off) succeeds, with the same SWIG warnings as #661's GL build.

7 of the 12 scripts pass, with the usual environment skips; the state_exec suites also pass on that build.
5 scripts fail, but they fail on master too:
- test_pycvc_gl.py: the volume scattering default;
- test_pycvc_gl_world.py: the grid node's bounds;
- test_pycvc_gl_lod.py: an access violation;
- test_pycvc_gl_streaming.py and test_pycvc_gl_clip.py: heap corruption.
A control build with master's own bindings (1d07677) on the same core objects fails the same five scripts, with the same assertions and exit codes. All 12 passed on master base 8d6e4d4.
So this is a regression on master, from something after ariadne(media): stream-play-audio DSL verb (conference Phase 1 / §4a) #639 in cvcGL or the state tree. PR CI never builds pycvc_gl, which is why it wasn't caught. A separate session to bisect and fix it has been proposed.

Since opening

state_exec: deep-data work a slice per step, and the budget by kind #658 landed. Master is merged into pycvc: state_exec parity part 1 (values, parsing, parse_error, value codec) #661 and pycvc: state_exec parity part 2 (environments, builtins, schedulers) #671 and carried here (7fd982d):
- pycvc: state_exec parity part 2 (environments, builtins, schedulers) #671 also wraps state_exec: deep-data work a slice per step, and the budget by kind #658's scheduler data settings (data_defaults, set_data_defaults, data_settings) and process_info's steps and times by kind;
- this PR wraps state_exec: deep-data work a slice per step, and the budget by kind #658's step_kind (an IntEnum) and evaluation_stats' counts by kind (steps, step_time, data_nodes, last_step_kind, add_*), plus evaluator_state.data_step_nodes;
- data_work/backlog/pending_data_work are listed as exceptions: deep-data work in flight between steps.
Locally on the merged tree:
- 19 tests in test_pycvc_state_exec_eval.py pass, including the new test_stats_by_kind_of_step;
- the non-GL suites give 280 passed and 3 failed: the rawiv lock (volume: close the file after a successful rawiv write #663), and test_pycvc_lod's two GIL-timing tests;
- those two pass when run alone. The machine had about 30 compiler and Python processes from other sessions running at the time.
state_exec: max_memory covers the measured heap; collect reference cycles #667 has since landed. Its heap API goes to PR 4 with state_exec: the stdlib materializers run as sliced data steps, and predicates get them #665's.

Coordination

Agreed with the sessions working on #658, #667 and #665: their chain lands first. Then:

I merge master into pycvc: state_exec parity part 1 (values, parsing, parse_error, value codec) #661, pycvc: state_exec parity part 2 (environments, builtins, schedulers) #671 and this PR, resolving the pycvc.i/pycvc_exec conflicts with state_exec: deep-data work a slice per step, and the budget by kind #658's data_options wrap.
PR 4 wraps the API they add, from the inventories they sent:
- from state_exec: deep-data work a slice per step, and the budget by kind #658: data_job.h, step_kind and evaluation_stats.steps(), process_info's per-kind steps and times, and the data defaults;
- from state_exec: max_memory covers the measured heap; collect reference cycles #667: heap.h, and the heap fields on process/process_info;
- from state_exec: the stdlib materializers run as sliced data steps, and predicates get them #665: its data_job.h additions.
PR 4 also wraps stdlib_registry.

… P0) Declare the state_exec s-expression grammar in docs/STATE_EXEC_GRAMMAR.md (W3C EBNF, Unicode source rules, literal tables, and a table of every intentional change from the hand-written parser, tagged by phase P2-P4). The parser itself is unchanged; later phases build it from this grammar. state_exec_grammar_test tags each case with the phase it holds from: landed cases must hold, pending cases must still differ, the doc's EBNF must be closed, and every production needs a case. state_exec_corpus_test pins how the 597 DSL scripts the repo carries parse today (state_exec_corpus/, regenerated with gen_state_exec_corpus.py and CVC_UPDATE_GOLDENS=1). Also correct parser.h (no quasiquote) and the developer guide (false and #f read as nil, bytes literals exist).

Spec only: parse_options::max_depth on parse()/parse_all()/parser and execute_options::max_parse_depth for submitted scripts, both from P2. Records why the limit exists (the reader recurses on the native stack; the evaluators are stackless) and the corpus's real depths (max 13, median 2).

Spec only. max_depth resolves from state_exec.schedulers.<id>.parser.max_depth, then state_exec.defaults.parser.max_depth, then 256, matching scheduler::load_settings(); parse_options remains for callers without a tree. Drops the execute_options::max_parse_depth idea so the tree is the single source.

…grammar P1b) The parser now stops at parser.max_depth nested lists and quotes (default 256) with a parse_error, instead of recursing until the native stack overflows. Like the scheduler's other settings, the limit lives on the state tree: state_exec.schedulers.<id>.parser.max_depth, then state_exec.defaults.parser.max_depth, then 256; a missing, malformed or zero value is skipped. - parser: parse_options (max_depth) on parse()/parse_all()/parser, a depth guard on lists and quotes, and parse_options_from_state(). - scheduler: load_settings() resolves parser.* and publishes the effective value under state_exec.schedulers.<id>.parser.max_depth; execute() parses with it. - async_scheduler (the app's exec scheduler) reads the defaults key on every submit; Ariadne predicates read it from the app tree. - stackless evaluators: create_state(script, env, parse_options). The grammar test moves from a numeric kLandedPhase to named phases; the 257-deep case is P1b and holds now. Tests cover the default, a custom depth, quotes, 100000 open parens, the error position, tree resolution, bad values, and both schedulers.

…e read) Lists and quotes were parsed by native recursion, so input depth was bounded by the thread's stack. The parser now keeps open lists and quotes on a heap stack of frames and reads in a loop: native stack depth is constant however deeply the input nests (a test reads a million open parens and reports the unclosed list instead of crashing). The loop is a resumable read(out, budget): it reads up to `budget` tokens and returns datum / end / more, carrying open frames between calls, so a scheduler can parse a script a slice at a time. parse() and parse_all() call it with no budget. Output is unchanged, quirks included (a stray top-level `)` still reads as a symbol until P2): the conformance cases, the 597-script corpus and the evaluator suites all pass. parser.max_depth stays on the state tree; it now bounds parsed data for the code that still walks values recursively (list destruction, the snapshot codec, to_string) rather than the parser's own stack. parse_fib20_expr: about 3.4 us vs 3.1 us for the recursive parser (median of 6, MSVC Release, loaded machine).

The stackless parser's state is an input position plus a heap stack of open frames, so it can pause anywhere. New setting parser.step_tokens (on the state tree, resolved like parser.max_depth; default 0) lets a scheduler parse a submitted script inside the new process instead of at submit: each scheduler step reads N tokens, and the step that reaches the end sets up evaluation exactly as parsing up front would have. - Reading steps are process steps: they count against max_steps and max_time and interleave with other processes. - A syntax error surfaces like a runtime error: the async scheduler kills the process with it as the exit error; the sync scheduler throws it from step()/run(). With step_tokens = 0, execute() still throws. - Signals and watch events wait until the script is read (their handlers run on the evaluation stack, which doesn't exist yet). - The snapshot codec stores a mid-read process's source and settings and re-reads it after a restore. Adds parse_options::step_tokens, script_reader (owns its source and drives parser::read() a budget at a time) and evaluator_state::reading; stackless_evaluator::step() reads until the script is read. Tests cover the reader, the evaluator, the codec and both schedulers (results, syntax errors, max_steps, deferred signals).

…ens) app::exec_scheduler(), which runs Ariadne's actions and residents, now defaults parser.step_tokens to 1024, so it parses submitted scripts inside their processes. async_scheduler gains set_parser_defaults(): the built-in values under its watch root's state_exec.defaults.parser.* keys. A plain async_scheduler (pycvc's Exec) and the sync scheduler stay at 0, so they still report syntax errors at submit. Why 1024: every script the repo carries reads in one slice (the largest is 159 tokens, .ari scripts at most 24), so a normal action pays one extra step of its 5000; a slice costs about 80 us at the parser's measured ~80 ns/token, so a pathologically large script is spread over steps and frames instead of stalling a drain. Ariadne now says why a process died: a killed action's warning carries its exit error, and the per-frame sweep reports a killed resident (a syntax error in its body, a runtime error, its per-step cap) once, instead of letting it vanish silently.

…uler pycvc's Exec runs programs on a private async_scheduler that already watches the app's state tree, but its built-in parser defaults were plain parse_options{} while app::exec_scheduler() read scripts in 1024-token steps. Both now take async_scheduler::host_parser_defaults(), one definition (with the reasoning for 1024) for every libcvc host, and the app tree's state_exec.defaults.parser.* keys override both. Exec.run() still raises a RuntimeError with the parse error's message: from the killed process's exit error when the script is read in steps, or from execute() when the tree sets step_tokens to 0. Tests: test_pycvc_exec.py checks the parse error's message and that max_depth / step_tokens on the app tree configure Exec; the integration test pins the app scheduler to host_parser_defaults().

pycvc's state_exec surface was only Exec; the parser settings these PRs add had no Python counterpart, so the defaults weren't even observable. In pycvc's direct-wrap style (the real C++ types, no facade): - pycvc.parse_options: the real cvc::state_exec::parse_options struct (max_depth, step_tokens), C++ defaults included. - pycvc.async_scheduler.host_parser_defaults(): the real static, declared to SWIG on its own (async_scheduler isn't wrapped yet, so no constructor). - pycvc.parse_options_from_state(app, scheduler_id="", base=...): the C++ resolver, taking the app where C++ takes the root (pycvc's convention). - Exec.parser_defaults() / Exec.set_parser_defaults(): the same calls as the async_scheduler Exec wraps. test_pycvc_exec.py covers each against the C++ behaviour; the README gains a state_exec section. Built locally (SWIG 4.4.1, Python 3.11): exec, exec_async, exec_coro and ari tests, 26 passed.

Wrap the real cvc::state_exec::parse_options_from_state(state&, id, base) instead of a pycvc adapter that took the app: Python passes the tree root, pycvc.State.instance(app), exactly as C++ passes cvc::state&. This is the first decision of the pycvc/C++ parity approach (C++ cvc::state& parameters take a pycvc.State in Python).

…codec) Start bringing pycvc's state_exec API to the C++ surface under the C++ names (docs/PYCVC_STATE_EXEC_PARITY.md: the audit of all 20 state_exec headers, the agreed approach, and the PR plan). - One strict value_t <-> Python conversion (pycvc_value.h), used by every API and by Exec: native types for data, pycvc.symbol (the real struct) for symbols, and pycvc.value (the real value_t, opaque) for closures, native fns, data objects and generators, which convert back unchanged. Non-str dict keys and unsupported types raise TypeError, ints beyond 64 bits OverflowError, and nesting past 1000 levels or a cycle ValueError. This replaces the lossy converter (symbols became str, closures "<closure>", anything else str(obj)), which could also put a null item in a list on invalid UTF-8 and leave an OverflowError pending. - pycvc.parse / parse_all, pycvc.parse_error (a RuntimeError with .line and .col), pycvc.script_reader, to_string, values_equal, and the value codec encode_value / decode_value on a State node. - Exec.run raises pycvc.parse_error for a syntax error whether the script was parsed at submit or read inside the process (it re-parses a killed process's script to recover the position). - The pycvc README's Status/Design sections described a v0 facade; they now describe the direct-wrap design. test_pycvc_state_exec.py (13 tests) covers each against the C++ behaviour. Locally: those plus pycvc's exec, async, coro, ari, state and core suites pass (87), except test_volume_rawiv_roundtrip, which fails the same way on the unmodified build (a Windows temp-file lock).

Wrap the scheduler surface of cvc::state_exec under the C++ names: - environment (shared, as environment_ptr) and builtins: make_default_environment, and register_fn taking a Python callable or a pycvc.value holding a native fn. - execute_options (on_complete is a Python callable, signal_handlers a dict), process_info, scheduler_stats, percent_encode/percent_decode. - process_status and scheduling_policy as enum.IntEnum classes; an enum argument must be a member, so to_string(process_status) and to_string(value_t) keep their shared name. - scheduler and async_scheduler as real classes with their non-coroutine members. run/step and sync_run/sync_step release the GIL; Python callbacks retake it. - Exec.scheduler() returns Exec's async_scheduler, which keeps the Exec alive. Empty optionals are None, except for values: get_result, pop_pending_message and environment.lookup raise KeyError, since None is nil. The value_t typecheck now runs last, so execute(str) picks the script overload. test_pycvc_state_exec_sched.py repeats the C++ scheduler tests on both classes. The parity doc and README are updated.

…c-parity-2

…'s branch)

…its comment is fixed

…nsics) Wrap the rest of cvc::state_exec's public surface under the C++ names: - stackless_evaluator and evaluator, with evaluator_state, eval_frame, eval_phase (an IntEnum) and evaluation_stats; evaluation_interrupted and evaluation_timeout as RuntimeError subclasses. The evaluators release the GIL while they evaluate, so another thread can interrupt, pause or resume them. on_complete is a Python callable. - The process struct (shared, as process_ptr), watch_event and watch_handler_entry, make_process, and the scheduler calls that take them: current_process() and queue_watch_event(). scheduler_base is the common base of the two schedulers. - encode_/decode_environment and encode_/decode_evaluator_state. - resource_policy (mode as an IntEnum), process_limits, validate_limits. - intrinsics_context, channel_policy, register_intrinsics, apply_chroot, resolve_channel_key, make_future, future_channel_of. Every function register_intrinsics adds keeps its context alive, and the context keeps the scheduler and channel policy assigned to it. - From #660: kDefaultMaxAllocBytes, eval_alloc_limit, eval_alloc_check, alloc_bytes, scheduler.max_alloc_bytes(), max_alloc_bytes_from_state. A reference into an object (evaluator_state.stats, evaluator.stats(), process.state) keeps its owner alive; other members read and write as copies. evaluator_state is move-only, so it is moved into Python. test_pycvc_state_exec_eval.py repeats the C++ tests. The parity doc, whose inventory now includes what landed on master since the audit, and the README are updated.

… data_options wrap)

…rap #658's data settings and budget fields on the schedulers

…rap #658's step_kind and the per-kind stats

…o feat/pycvc-state-exec-parity-2

…-state-exec-parity-3

…ec-parity-3

…p-finish Conflicts: STATE_EXEC_GRAMMAR.md's native-calls list (state-json/state-from-json from #681, state-set here: both kept), and the tests #681/#682 and this branch each added to state_exec_data_steps_test.cpp and state_exec_deep_data_test.cpp (all kept). #675's pycvc tests asserted the old parser.max_depth default of 256; they now compare with pycvc.parse_options().max_depth, as agreed with the parity session.

…-exec-no-nested-eval Master has #682 as one squash commit; this branch had its original commits. The merge's tree is c4ca22c applied onto master: one conflict, the tests #681 and this branch each appended to state_exec_data_steps_test.cpp, both kept.

transfix added 20 commits October 16, 2026 23:58

Merge master into feat/pycvc-state-exec-parity (#653 landed)

cd292e0

Merge branch 'feat/pycvc-state-exec-parity' into feat/pycvc-state-exe…

40d0ce2

…c-parity-2

Rerun CI on master (the merge push ran while the PR still targeted #653…

0c24a14

…'s branch)

pycvc: environment.frozen (#660), and the set_existing note now that …

b2b772a

…its comment is fixed

Merge master into feat/pycvc-state-exec-parity (#658 landed: keep its…

41411dc

… data_options wrap)

Merge #661 (with master: #658) into feat/pycvc-state-exec-parity-2; w…

4f05226

…rap #658's data settings and budget fields on the schedulers

Merge #671 (with master: #658) into feat/pycvc-state-exec-parity-3; w…

7fd982d

…rap #658's step_kind and the per-kind stats

transfix mentioned this pull request Oct 18, 2026

pycvc: state_exec parity part 1 (values, parsing, parse_error, value codec) #661

Merged

Merge master (#661 squash-merged as 50c6d02, plus #665/#667/#663) int…

8b18056

…o feat/pycvc-state-exec-parity-2

transfix mentioned this pull request Oct 18, 2026

pycvc: state_exec parity part 2 (environments, builtins, schedulers) #671

Merged

Merge #671 (now on master, after #661's squash-merge) into feat/pycvc…

76e8c60

…-state-exec-parity-3

transfix mentioned this pull request Oct 18, 2026

pycvc: state_exec parity part 4 (data jobs, heap, stdlib registry, any-depth values) #679

Open

Merge master (#671 squash-merged as 1087b09) into feat/pycvc-state-ex…

5e59800

…ec-parity-3

transfix changed the base branch from feat/pycvc-state-exec-parity-2 to master October 18, 2026 05:20

transfix enabled auto-merge (squash) October 18, 2026 05:27

transfix merged commit c2a144b into master Oct 18, 2026
16 checks passed

Navigation Menu

Repository navigation

pycvc: state_exec parity part 3 (evaluators, processes, policy, intrinsics) - #675

pycvc: state_exec parity part 3 (evaluators, processes, policy, intrinsics)#675
transfix merged 23 commits into
masterfrom
feat/pycvc-state-exec-parity-3

transfix commented Oct 18, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

transfix commented Oct 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

This PR

Lifetimes (approach items 10 and 11 in the doc)

Not wrapped, with reasons in the doc

Tests

Since opening

Coordination

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

transfix commented Oct 18, 2026 •

edited

Loading