pycvc: state_exec parity part 2 (environments, builtins, schedulers) by transfix · Pull Request #671 · transfix/libcvc

transfix · 2026-10-18T00:17:12Z

Targets master. #661 (part 1) merged as 50c6d02, and master is merged in (8b18056), so the diff here is part 2 only. #675 (part 3) is stacked on this PR.

Part 2 of bringing pycvc's state_exec API to the C++ surface, under the C++ names. docs/PYCVC_STATE_EXEC_PARITY.md has the audit, the approach and the plan; this PR updates its inventory and adds four rules to the approach (enums, empty results, callables, the GIL).

This PR

pycvc.environment: the real struct, shared like environment_ptr. It has set, set_existing, extend, outer and lookup, and bindings reads and writes as a dict.
pycvc.builtins:
- make_default_environment();
- register_fn(env, name, fn), where fn is a Python callable, or a pycvc.value holding a native fn (so env.lookup("+") can be registered under another name).
pycvc.execute_options, pycvc.process_info, pycvc.scheduler_stats, with the C++ fields and defaults:
- on_complete is a Python callable, and reads back as the callable that was set;
- signal_handlers is a dict;
- the uint64_t fields are Python ints over their full range.
pycvc.process_status and pycvc.scheduling_policy: enum.IntEnum classes with the C++ names and values, so pycvc.process_status.killed is process_status::killed. Also to_string(process_status) and parse_process_status.
pycvc.scheduler and pycvc.async_scheduler: the real classes, with every non-coroutine member:
- execute, step/run, process control, limits, signals, fork, messaging, the queries, watch handlers;
- set_id, load_settings, parser_settings and the settings_root/defaults_prefix/sched_prefix constants on scheduler;
- kill_owner, wake_awaiting, post_message, drain_ingress, get_exit_error, reap and the parser defaults on async_scheduler.
pycvc.percent_encode / percent_decode.
Exec.scheduler(): returns the async_scheduler that Exec.run drives. The returned object keeps the Exec alive.

Rules this PR follows

Enums are strict: an enum argument must be a member of its IntEnum. A plain int stays a value, so pycvc.to_string(2) is "2" and pycvc.to_string(process_status.waiting) is "waiting".
Empty results: an empty std::optional is None, except for a value. None is nil, so get_result, pop_pending_message and environment.lookup raise KeyError(pid / path / name) where C++ returns std::nullopt or nullptr. A program that returns nil still gives None.
Callables: a Python callable runs on the thread stepping the scheduler and takes the GIL for its call. A Python exception becomes the C++ error a throwing callback would be:
- the sync scheduler throws it from run();
- the async scheduler kills the process and keeps the message as its exit error.
The GIL: scheduler.run/step and async_scheduler.sync_run/sync_step release it while programs run. Thread safety is otherwise the C++ one: only post_message may be called while a scheduler runs.
Overloads: the value_t typecheck now runs last (precedence 10000, was 0). A str goes to execute(script) rather than being executed as a string value, and an enum goes to to_string(process_status).

Not in this PR

PR 3: the process struct, and with it current_process() and queue_watch_event(). They come with evaluator_state, which a process holds.
mem_tracker(): memory_tracker is internal to the schedulers.
The coroutine step()/run(): Python has sync_step()/sync_run().
Exec.environment(): not added. Its intrinsics point into the Exec, so a stored environment could outlive it. PR 3's register_intrinsics gives a program submitted to Exec.scheduler() the intrinsics safely.

Tests

bindings/pycvc/test_pycvc_state_exec_sched.py (23 tests, registered with CTest) repeats the C++ tests in state_exec_scheduler_test.cpp and state_exec_async_test.cpp on both classes where both have the call. The GIL test checks that another Python thread makes progress in the middle third of a 0.3 s run(). Measured locally: a call that holds the GIL (a 60 MB percent_encode) gives 0 ticks there, while run/sync_run give about 400k.

Local results, Windows MSVC 14.38 Release, SWIG 4.4.1, Python 3.11, cvcpkg prefix:

Run	Result
`test_pycvc_state_exec_sched.py`	23 passed
state_exec, exec, async, coro, Ariadne and state suites	81 passed
the other non-GL suites	174 passed, 1 failed: `test_volume_rawiv_roundtrip`, the Windows file lock #663 fixes (not in this stack)
cvcGL + pycvc + `pycvc_gl` build (`build-win-gl`, VTK-Python bridge off as in #661)	builds; the same SWIG warnings as #661's GL build, none from the state_exec section
the 12 `pycvc_gl` scripts, plus `test_pycvc_state_exec`, `_sched`, `_exec` and `_state` on that build	16/16 pass, with the same environment skips as #661

clang-format 18 is clean on the C++ files, and ruff (E, F, W) is clean on the new test.

Notes on C++ behaviour the tests pin

environment::set_existing on a name bound nowhere sets it in the outermost unfrozen scope. When this PR opened, the header comment said "in this scope"; state_exec: load the stdlib modules in every environment, with a per-call allocation limit #660 has since corrected it.
On the sync scheduler, a process whose native fn threw stays running, so a later run() steps it and throws again. The async scheduler kills it instead. Both are the existing C++ behaviour.

Since opening

state_exec: read submitted scripts inside the process, a slice per step #653 landed. Master is merged into pycvc: state_exec parity part 1 (values, parsing, parse_error, value codec) #661 and, through pycvc: state_exec parity part 1 (values, parsing, parse_error, value codec) #661, into this branch (40d0ce2). On that merged tree, the parity, exec, async, coro, Ariadne and state suites pass: 83 tests.
environment.frozen: state_exec: load the stdlib modules in every environment, with a per-call allocation limit #660 added frozen scopes to environment. The wrapper now has the field, and test_a_frozen_scope_is_read_only checks the two rules: set raises RuntimeError (C++ throws std::logic_error), and set_existing shadows the name in an unfrozen scope instead of writing into the frozen one.
Merge order: agreed with the sessions working on state_exec: deep-data work a slice per step, and the budget by kind #658, state_exec: max_memory covers the measured heap; collect reference cycles #667 and state_exec: the stdlib materializers run as sliced data steps, and predicates get them #665. Their chain lands first; I then merge master into pycvc: state_exec parity part 1 (values, parsing, parse_error, value codec) #661/pycvc: state_exec parity part 2 (environments, builtins, schedulers) #671 and resolve the conflicts with state_exec: deep-data work a slice per step, and the budget by kind #658's data_options wrap in pycvc.i/pycvc_exec.
state_exec: deep-data work a slice per step, and the budget by kind #658 landed (2026-10-18). Master is merged in through pycvc: state_exec parity part 1 (values, parsing, parse_error, value codec) #661 (4f05226). On the merged tree, this PR also wraps the state_exec: deep-data work a slice per step, and the budget by kind #658 API next to what it already wraps:
- async_scheduler.data_defaults/set_data_defaults/data_settings beside host_data_defaults;
- scheduler.data_settings();
- process_info's steps and times by kind (read_steps, eval_steps, data_steps, data_nodes, read_time, eval_time, data_time).
test_data_settings_and_the_budget_by_kind covers them. Locally, 89 tests pass across the parity, exec, async, coro, Ariadne and state suites.

… P0) Declare the state_exec s-expression grammar in docs/STATE_EXEC_GRAMMAR.md (W3C EBNF, Unicode source rules, literal tables, and a table of every intentional change from the hand-written parser, tagged by phase P2-P4). The parser itself is unchanged; later phases build it from this grammar. state_exec_grammar_test tags each case with the phase it holds from: landed cases must hold, pending cases must still differ, the doc's EBNF must be closed, and every production needs a case. state_exec_corpus_test pins how the 597 DSL scripts the repo carries parse today (state_exec_corpus/, regenerated with gen_state_exec_corpus.py and CVC_UPDATE_GOLDENS=1). Also correct parser.h (no quasiquote) and the developer guide (false and #f read as nil, bytes literals exist).

Spec only: parse_options::max_depth on parse()/parse_all()/parser and execute_options::max_parse_depth for submitted scripts, both from P2. Records why the limit exists (the reader recurses on the native stack; the evaluators are stackless) and the corpus's real depths (max 13, median 2).

Spec only. max_depth resolves from state_exec.schedulers.<id>.parser.max_depth, then state_exec.defaults.parser.max_depth, then 256, matching scheduler::load_settings(); parse_options remains for callers without a tree. Drops the execute_options::max_parse_depth idea so the tree is the single source.

…grammar P1b) The parser now stops at parser.max_depth nested lists and quotes (default 256) with a parse_error, instead of recursing until the native stack overflows. Like the scheduler's other settings, the limit lives on the state tree: state_exec.schedulers.<id>.parser.max_depth, then state_exec.defaults.parser.max_depth, then 256; a missing, malformed or zero value is skipped. - parser: parse_options (max_depth) on parse()/parse_all()/parser, a depth guard on lists and quotes, and parse_options_from_state(). - scheduler: load_settings() resolves parser.* and publishes the effective value under state_exec.schedulers.<id>.parser.max_depth; execute() parses with it. - async_scheduler (the app's exec scheduler) reads the defaults key on every submit; Ariadne predicates read it from the app tree. - stackless evaluators: create_state(script, env, parse_options). The grammar test moves from a numeric kLandedPhase to named phases; the 257-deep case is P1b and holds now. Tests cover the default, a custom depth, quotes, 100000 open parens, the error position, tree resolution, bad values, and both schedulers.

…e read) Lists and quotes were parsed by native recursion, so input depth was bounded by the thread's stack. The parser now keeps open lists and quotes on a heap stack of frames and reads in a loop: native stack depth is constant however deeply the input nests (a test reads a million open parens and reports the unclosed list instead of crashing). The loop is a resumable read(out, budget): it reads up to `budget` tokens and returns datum / end / more, carrying open frames between calls, so a scheduler can parse a script a slice at a time. parse() and parse_all() call it with no budget. Output is unchanged, quirks included (a stray top-level `)` still reads as a symbol until P2): the conformance cases, the 597-script corpus and the evaluator suites all pass. parser.max_depth stays on the state tree; it now bounds parsed data for the code that still walks values recursively (list destruction, the snapshot codec, to_string) rather than the parser's own stack. parse_fib20_expr: about 3.4 us vs 3.1 us for the recursive parser (median of 6, MSVC Release, loaded machine).

The stackless parser's state is an input position plus a heap stack of open frames, so it can pause anywhere. New setting parser.step_tokens (on the state tree, resolved like parser.max_depth; default 0) lets a scheduler parse a submitted script inside the new process instead of at submit: each scheduler step reads N tokens, and the step that reaches the end sets up evaluation exactly as parsing up front would have. - Reading steps are process steps: they count against max_steps and max_time and interleave with other processes. - A syntax error surfaces like a runtime error: the async scheduler kills the process with it as the exit error; the sync scheduler throws it from step()/run(). With step_tokens = 0, execute() still throws. - Signals and watch events wait until the script is read (their handlers run on the evaluation stack, which doesn't exist yet). - The snapshot codec stores a mid-read process's source and settings and re-reads it after a restore. Adds parse_options::step_tokens, script_reader (owns its source and drives parser::read() a budget at a time) and evaluator_state::reading; stackless_evaluator::step() reads until the script is read. Tests cover the reader, the evaluator, the codec and both schedulers (results, syntax errors, max_steps, deferred signals).

…ens) app::exec_scheduler(), which runs Ariadne's actions and residents, now defaults parser.step_tokens to 1024, so it parses submitted scripts inside their processes. async_scheduler gains set_parser_defaults(): the built-in values under its watch root's state_exec.defaults.parser.* keys. A plain async_scheduler (pycvc's Exec) and the sync scheduler stay at 0, so they still report syntax errors at submit. Why 1024: every script the repo carries reads in one slice (the largest is 159 tokens, .ari scripts at most 24), so a normal action pays one extra step of its 5000; a slice costs about 80 us at the parser's measured ~80 ns/token, so a pathologically large script is spread over steps and frames instead of stalling a drain. Ariadne now says why a process died: a killed action's warning carries its exit error, and the per-frame sweep reports a killed resident (a syntax error in its body, a runtime error, its per-step cap) once, instead of letting it vanish silently.

…uler pycvc's Exec runs programs on a private async_scheduler that already watches the app's state tree, but its built-in parser defaults were plain parse_options{} while app::exec_scheduler() read scripts in 1024-token steps. Both now take async_scheduler::host_parser_defaults(), one definition (with the reasoning for 1024) for every libcvc host, and the app tree's state_exec.defaults.parser.* keys override both. Exec.run() still raises a RuntimeError with the parse error's message: from the killed process's exit error when the script is read in steps, or from execute() when the tree sets step_tokens to 0. Tests: test_pycvc_exec.py checks the parse error's message and that max_depth / step_tokens on the app tree configure Exec; the integration test pins the app scheduler to host_parser_defaults().

pycvc's state_exec surface was only Exec; the parser settings these PRs add had no Python counterpart, so the defaults weren't even observable. In pycvc's direct-wrap style (the real C++ types, no facade): - pycvc.parse_options: the real cvc::state_exec::parse_options struct (max_depth, step_tokens), C++ defaults included. - pycvc.async_scheduler.host_parser_defaults(): the real static, declared to SWIG on its own (async_scheduler isn't wrapped yet, so no constructor). - pycvc.parse_options_from_state(app, scheduler_id="", base=...): the C++ resolver, taking the app where C++ takes the root (pycvc's convention). - Exec.parser_defaults() / Exec.set_parser_defaults(): the same calls as the async_scheduler Exec wraps. test_pycvc_exec.py covers each against the C++ behaviour; the README gains a state_exec section. Built locally (SWIG 4.4.1, Python 3.11): exec, exec_async, exec_coro and ari tests, 26 passed.

Wrap the real cvc::state_exec::parse_options_from_state(state&, id, base) instead of a pycvc adapter that took the app: Python passes the tree root, pycvc.State.instance(app), exactly as C++ passes cvc::state&. This is the first decision of the pycvc/C++ parity approach (C++ cvc::state& parameters take a pycvc.State in Python).

…codec) Start bringing pycvc's state_exec API to the C++ surface under the C++ names (docs/PYCVC_STATE_EXEC_PARITY.md: the audit of all 20 state_exec headers, the agreed approach, and the PR plan). - One strict value_t <-> Python conversion (pycvc_value.h), used by every API and by Exec: native types for data, pycvc.symbol (the real struct) for symbols, and pycvc.value (the real value_t, opaque) for closures, native fns, data objects and generators, which convert back unchanged. Non-str dict keys and unsupported types raise TypeError, ints beyond 64 bits OverflowError, and nesting past 1000 levels or a cycle ValueError. This replaces the lossy converter (symbols became str, closures "<closure>", anything else str(obj)), which could also put a null item in a list on invalid UTF-8 and leave an OverflowError pending. - pycvc.parse / parse_all, pycvc.parse_error (a RuntimeError with .line and .col), pycvc.script_reader, to_string, values_equal, and the value codec encode_value / decode_value on a State node. - Exec.run raises pycvc.parse_error for a syntax error whether the script was parsed at submit or read inside the process (it re-parses a killed process's script to recover the position). - The pycvc README's Status/Design sections described a v0 facade; they now describe the direct-wrap design. test_pycvc_state_exec.py (13 tests) covers each against the C++ behaviour. Locally: those plus pycvc's exec, async, coro, ari, state and core suites pass (87), except test_volume_rawiv_roundtrip, which fails the same way on the unmodified build (a Windows temp-file lock).

Wrap the scheduler surface of cvc::state_exec under the C++ names: - environment (shared, as environment_ptr) and builtins: make_default_environment, and register_fn taking a Python callable or a pycvc.value holding a native fn. - execute_options (on_complete is a Python callable, signal_handlers a dict), process_info, scheduler_stats, percent_encode/percent_decode. - process_status and scheduling_policy as enum.IntEnum classes; an enum argument must be a member, so to_string(process_status) and to_string(value_t) keep their shared name. - scheduler and async_scheduler as real classes with their non-coroutine members. run/step and sync_run/sync_step release the GIL; Python callbacks retake it. - Exec.scheduler() returns Exec's async_scheduler, which keeps the Exec alive. Empty optionals are None, except for values: get_result, pop_pending_message and environment.lookup raise KeyError, since None is nil. The value_t typecheck now runs last, so execute(str) picks the script overload. test_pycvc_state_exec_sched.py repeats the C++ scheduler tests on both classes. The parity doc and README are updated.

…c-parity-2

…'s branch)

…its comment is fixed

… data_options wrap)

…rap #658's data settings and budget fields on the schedulers

…rap #658's step_kind and the per-kind stats

…o feat/pycvc-state-exec-parity-2

…-state-exec-parity-3

…ec-parity-3

transfix added 14 commits October 16, 2026 23:58

Merge master into feat/pycvc-state-exec-parity (#653 landed)

cd292e0

Merge branch 'feat/pycvc-state-exec-parity' into feat/pycvc-state-exe…

40d0ce2

…c-parity-2

transfix mentioned this pull request Oct 18, 2026

pycvc: state_exec parity part 1 (values, parsing, parse_error, value codec) #661

Merged

transfix added 2 commits October 17, 2026 20:47

Rerun CI on master (the merge push ran while the PR still targeted #653…

0c24a14

…'s branch)

pycvc: environment.frozen (#660), and the set_existing note now that …

b2b772a

…its comment is fixed

transfix mentioned this pull request Oct 18, 2026

pycvc: state_exec parity part 3 (evaluators, processes, policy, intrinsics) #675

Merged

transfix added 2 commits October 17, 2026 21:55

Merge master into feat/pycvc-state-exec-parity (#658 landed: keep its…

41411dc

… data_options wrap)

Merge #661 (with master: #658) into feat/pycvc-state-exec-parity-2; w…

4f05226

…rap #658's data settings and budget fields on the schedulers

transfix added a commit that referenced this pull request Oct 18, 2026

Merge #671 (with master: #658) into feat/pycvc-state-exec-parity-3; w…

7fd982d

…rap #658's step_kind and the per-kind stats

Merge master (#661 squash-merged as 50c6d02, plus #665/#667/#663) int…

8b18056

…o feat/pycvc-state-exec-parity-2

transfix changed the base branch from feat/pycvc-state-exec-parity to master October 18, 2026 04:43

transfix added a commit that referenced this pull request Oct 18, 2026

Merge #671 (now on master, after #661's squash-merge) into feat/pycvc…

76e8c60

…-state-exec-parity-3

transfix merged commit 1087b09 into master Oct 18, 2026
16 checks passed

transfix added a commit that referenced this pull request Oct 18, 2026

Merge master (#671 squash-merged as 1087b09) into feat/pycvc-state-ex…

5e59800

…ec-parity-3

Navigation Menu

Repository navigation

pycvc: state_exec parity part 2 (environments, builtins, schedulers) - #671

pycvc: state_exec parity part 2 (environments, builtins, schedulers)#671
transfix merged 19 commits into
masterfrom
feat/pycvc-state-exec-parity-2

transfix commented Oct 18, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

transfix commented Oct 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

This PR

Rules this PR follows

Not in this PR

Tests

Notes on C++ behaviour the tests pin

Since opening

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

transfix commented Oct 18, 2026 •

edited

Loading