Repository navigation
state_exec: class methods and generator bodies run inside the calling process - #683
Merged
Merged
Conversation
…chain iteratively A native function that calls another function itself, or runs a nested evaluator, nests on the native stack. Several of these could ask for unbounded nesting and crash the process: - (apply apply L) with L containing itself; - send re-sending through itself; - a class whose init is its own constructor; - a long chain of inherited inits; - methods calling methods, and generators of generators. detail::native_nesting now counts these levels: - where: call_function, call_here, run_data_job's job calls, the class constructor's init call, inherited init/finalize, class methods' nested evaluator and generator_next; - the limit: a level throws past kMaxNativeNesting (256) levels, or when less than kNativeStackReserve of the thread's native stack is left; - the reserve is 128 KiB on Windows, where MSVC runs a catch block before unwinding the frames above it, and 32 KiB elsewhere; - native_stack.cpp asks the platform where the stack ends: VirtualQuery, pthread_getattr_np, pthread_get_stackaddr_np or emscripten_stack_get_end. Measured per level in a Release build: about 0.5 KiB for apply, 2-4 KiB for a nested generator, and over 10 KiB for a class method's evaluator. That spread is why a level count alone could not fit a 64 KiB stack. class_method, class_constructor and class_inherited_method drop what they hold through drop_owner, as value_t does. Freeing a long superclass chain used to take a native frame per class. The constructor and inherited methods also call a copy of the function, not a reference into a class dict the callee may change. Tests (StateExecNativeNesting): - each runaway case throws on a 64 KiB stack instead of crashing, and leaves no level counted; - legitimate nesting still works on 1 MiB; - a 4 MiB stack stops at the level count; - freeing a 10000-class chain on 64 KiB crashed before this change.
The codec wrote each closure's scope chain inside the closure, and each binding in it with a fresh encode_value. That cost one native level per closure nesting. A function whose scope binds the function itself made encode_environment recurse forever. Any defun'd function is one, so a snapshot of a paused process that had defined one overflowed the stack. Values, closure bodies and scopes are now one walk with one explicit stack (value_writer): - each scope the walk reaches is an entry of an __envs__ table under the node it started at, holding the scope's bindings (sorted by name) and an __outer__ reference; - a closure's __env__ is a reference to an entry (__type__ "env_ref") or nil; - encode_environment writes a reference at its node, with the table under it. value_reader decodes it the same way: - a referenced entry becomes one environment, filled once, so closures that shared a scope share it again; - a bad reference, or an __outer__ chain that loops back on itself, throws (the loop is unlinked first, so nothing leaks). Older snapshots still decode. A closure whose __env__ is missing or an inline env_chain is read the old way: its body expressions and scope bindings each with their own __spill__ table. Tests: - AFunctionItsOwnScopeBinds: a defun'd factorial round-trips, keeps its scope, and runs; - ClosuresThatShareAScopeStillShareIt; - OlderSnapshotsStillDecode: the old layout, built by hand, spill table included; - BadScopeReferencesThrow; - StateExecDeepDataCodec.ClosureChainRoundTrip: 10000 closures, each bound in the next one's scope, on the small stack.
… process A class method used to run to completion on a temporary evaluator inside one native call. A DSL generator's body ran to its next yield the same way. Neither counted against the process's max_steps; only the time deadline bounded them. A method also cost over 10 KiB of native stack per nested call. Methods: - A method is now a closure in its class's dict, taking self first (detail::class_method is gone). send already calls through call_function, so the body runs as frames of the process. - The constructor runs init through a new call_function overload with a continuation. Its value is the instance once init has run, even when init's frames or deep-data work take several steps. - Inherited init/finalize tail-call through call_function. - super's native path shares step_apply's handling (apply_native), so a native super method that defers works. - Methods calling methods no longer nest natively: 10000 nested sends run on a stack that could not hold two before. Generators: - make_generator_job advances a generator as a data job. Each step of a DSL generator's evaluator, or each value of a native one, is a node. - next and collect run that job. collect checks the allocation limit per value, and the deadline per value when nothing slices. - A for loop over a DSL generator gets each value from a child native call, (%generator-next %generator) in a scope of the loop frame's own, so the job slices and waits like any other. - With data.step_nodes = N a generator body runs N steps per process step, in data steps, against max_steps and max_time. The generator's own deep-data work slices the same way. - Native generators (range) in for and next still advance inline. Measured: a loop over 300000 values of DSL generators costs about 15% more (3.1 s -> 3.6 s, cvc exec), for the child frame and job per value. Evaluation benches were within this busy machine's run-to-run noise. Tests: - DataStepsScheduler.MethodStepsCountAgainstMaxSteps; - GeneratorBodiesRunASlicePerStep (and killed at max_steps); - ForOverAGeneratorGivesTheSameSlicedOrNot; - ConstructorReturnsTheInstanceAfterInitDefers; - StateExecNativeNesting.MethodsCallingMethodsAreFrames; - the nesting tests' small stack is sized from the platform's reserve; - SchedulerTest.TimeBudgetStopsALoopingInitMethod: a looping init is now killed by max_time like any loop, instead of throwing out of run().
transfix
enabled auto-merge (squash)
October 18, 2026 07:06
transfix
added a commit
that referenced
this pull request
Oct 18, 2026
… on macOS (#687) ApplyOnASelfContainingList expected the stack, not the 256-level count, to stop (apply apply L) on a stack 64 KiB above the reserve. An apply level is about 0.5 KiB with MSVC but under 0.25 KiB with Apple clang, so on macOS all 256 levels fit and recipe / macos-latest / Release failed after #683. The check now runs 16 KiB above the reserve.
transfix
added a commit
that referenced
this pull request
Oct 18, 2026
…b and the native nesting limits
transfix
added a commit
that referenced
this pull request
Oct 18, 2026
…y-depth values) (#679) * state_exec: grammar spec, conformance cases and parse corpus (grammar P0) Declare the state_exec s-expression grammar in docs/STATE_EXEC_GRAMMAR.md (W3C EBNF, Unicode source rules, literal tables, and a table of every intentional change from the hand-written parser, tagged by phase P2-P4). The parser itself is unchanged; later phases build it from this grammar. state_exec_grammar_test tags each case with the phase it holds from: landed cases must hold, pending cases must still differ, the doc's EBNF must be closed, and every production needs a case. state_exec_corpus_test pins how the 597 DSL scripts the repo carries parse today (state_exec_corpus/, regenerated with gen_state_exec_corpus.py and CVC_UPDATE_GOLDENS=1). Also correct parser.h (no quasiquote) and the developer guide (false and #f read as nil, bytes literals exist). * state_exec grammar: make the nesting limit configurable (default 256) Spec only: parse_options::max_depth on parse()/parse_all()/parser and execute_options::max_parse_depth for submitted scripts, both from P2. Records why the limit exists (the reader recurses on the native stack; the evaluators are stackless) and the corpus's real depths (max 13, median 2). * state_exec grammar: parser settings live on the state tree Spec only. max_depth resolves from state_exec.schedulers.<id>.parser.max_depth, then state_exec.defaults.parser.max_depth, then 256, matching scheduler::load_settings(); parse_options remains for callers without a tree. Drops the execute_options::max_parse_depth idea so the tree is the single source. * state_exec: parser settings on the state tree, with a nesting limit (grammar P1b) The parser now stops at parser.max_depth nested lists and quotes (default 256) with a parse_error, instead of recursing until the native stack overflows. Like the scheduler's other settings, the limit lives on the state tree: state_exec.schedulers.<id>.parser.max_depth, then state_exec.defaults.parser.max_depth, then 256; a missing, malformed or zero value is skipped. - parser: parse_options (max_depth) on parse()/parse_all()/parser, a depth guard on lists and quotes, and parse_options_from_state(). - scheduler: load_settings() resolves parser.* and publishes the effective value under state_exec.schedulers.<id>.parser.max_depth; execute() parses with it. - async_scheduler (the app's exec scheduler) reads the defaults key on every submit; Ariadne predicates read it from the app tree. - stackless evaluators: create_state(script, env, parse_options). The grammar test moves from a numeric kLandedPhase to named phases; the 257-deep case is P1b and holds now. Tests cover the default, a custom depth, quotes, 100000 open parens, the error position, tree resolution, bad values, and both schedulers. * state_exec: make the parser stackless (explicit frame stack, resumable read) Lists and quotes were parsed by native recursion, so input depth was bounded by the thread's stack. The parser now keeps open lists and quotes on a heap stack of frames and reads in a loop: native stack depth is constant however deeply the input nests (a test reads a million open parens and reports the unclosed list instead of crashing). The loop is a resumable read(out, budget): it reads up to `budget` tokens and returns datum / end / more, carrying open frames between calls, so a scheduler can parse a script a slice at a time. parse() and parse_all() call it with no budget. Output is unchanged, quirks included (a stray top-level `)` still reads as a symbol until P2): the conformance cases, the 597-script corpus and the evaluator suites all pass. parser.max_depth stays on the state tree; it now bounds parsed data for the code that still walks values recursively (list destruction, the snapshot codec, to_string) rather than the parser's own stack. parse_fib20_expr: about 3.4 us vs 3.1 us for the recursive parser (median of 6, MSVC Release, loaded machine). * state_exec: read submitted scripts inside the process, a slice per step The stackless parser's state is an input position plus a heap stack of open frames, so it can pause anywhere. New setting parser.step_tokens (on the state tree, resolved like parser.max_depth; default 0) lets a scheduler parse a submitted script inside the new process instead of at submit: each scheduler step reads N tokens, and the step that reaches the end sets up evaluation exactly as parsing up front would have. - Reading steps are process steps: they count against max_steps and max_time and interleave with other processes. - A syntax error surfaces like a runtime error: the async scheduler kills the process with it as the exit error; the sync scheduler throws it from step()/run(). With step_tokens = 0, execute() still throws. - Signals and watch events wait until the script is read (their handlers run on the evaluation stack, which doesn't exist yet). - The snapshot codec stores a mid-read process's source and settings and re-reads it after a restore. Adds parse_options::step_tokens, script_reader (owns its source and drives parser::read() a budget at a time) and evaluator_state::reading; stackless_evaluator::step() reads until the script is read. Tests cover the reader, the evaluator, the codec and both schedulers (results, syntax errors, max_steps, deferred signals). * state_exec: the app's exec scheduler reads scripts in steps (1024 tokens) app::exec_scheduler(), which runs Ariadne's actions and residents, now defaults parser.step_tokens to 1024, so it parses submitted scripts inside their processes. async_scheduler gains set_parser_defaults(): the built-in values under its watch root's state_exec.defaults.parser.* keys. A plain async_scheduler (pycvc's Exec) and the sync scheduler stay at 0, so they still report syntax errors at submit. Why 1024: every script the repo carries reads in one slice (the largest is 159 tokens, .ari scripts at most 24), so a normal action pays one extra step of its 5000; a slice costs about 80 us at the parser's measured ~80 ns/token, so a pathologically large script is spread over steps and frames instead of stalling a drain. Ariadne now says why a process died: a killed action's warning carries its exit error, and the per-frame sweep reports a killed resident (a syntax error in its body, a runtime error, its per-step cap) once, instead of letting it vanish silently. * pycvc: give Exec the same state_exec defaults as the app's exec scheduler pycvc's Exec runs programs on a private async_scheduler that already watches the app's state tree, but its built-in parser defaults were plain parse_options{} while app::exec_scheduler() read scripts in 1024-token steps. Both now take async_scheduler::host_parser_defaults(), one definition (with the reasoning for 1024) for every libcvc host, and the app tree's state_exec.defaults.parser.* keys override both. Exec.run() still raises a RuntimeError with the parse error's message: from the killed process's exit error when the script is read in steps, or from execute() when the tree sets step_tokens to 0. Tests: test_pycvc_exec.py checks the parse error's message and that max_depth / step_tokens on the app tree configure Exec; the integration test pins the app scheduler to host_parser_defaults(). * pycvc: expose the state_exec parser-settings surface under its C++ names pycvc's state_exec surface was only Exec; the parser settings these PRs add had no Python counterpart, so the defaults weren't even observable. In pycvc's direct-wrap style (the real C++ types, no facade): - pycvc.parse_options: the real cvc::state_exec::parse_options struct (max_depth, step_tokens), C++ defaults included. - pycvc.async_scheduler.host_parser_defaults(): the real static, declared to SWIG on its own (async_scheduler isn't wrapped yet, so no constructor). - pycvc.parse_options_from_state(app, scheduler_id="", base=...): the C++ resolver, taking the app where C++ takes the root (pycvc's convention). - Exec.parser_defaults() / Exec.set_parser_defaults(): the same calls as the async_scheduler Exec wraps. test_pycvc_exec.py covers each against the C++ behaviour; the README gains a state_exec section. Built locally (SWIG 4.4.1, Python 3.11): exec, exec_async, exec_coro and ari tests, 26 passed. * pycvc: parse_options_from_state takes the State root, as in C++ Wrap the real cvc::state_exec::parse_options_from_state(state&, id, base) instead of a pycvc adapter that took the app: Python passes the tree root, pycvc.State.instance(app), exactly as C++ passes cvc::state&. This is the first decision of the pycvc/C++ parity approach (C++ cvc::state& parameters take a pycvc.State in Python). * pycvc: state_exec parity part 1 (values, parsing, parse_error, value codec) Start bringing pycvc's state_exec API to the C++ surface under the C++ names (docs/PYCVC_STATE_EXEC_PARITY.md: the audit of all 20 state_exec headers, the agreed approach, and the PR plan). - One strict value_t <-> Python conversion (pycvc_value.h), used by every API and by Exec: native types for data, pycvc.symbol (the real struct) for symbols, and pycvc.value (the real value_t, opaque) for closures, native fns, data objects and generators, which convert back unchanged. Non-str dict keys and unsupported types raise TypeError, ints beyond 64 bits OverflowError, and nesting past 1000 levels or a cycle ValueError. This replaces the lossy converter (symbols became str, closures "<closure>", anything else str(obj)), which could also put a null item in a list on invalid UTF-8 and leave an OverflowError pending. - pycvc.parse / parse_all, pycvc.parse_error (a RuntimeError with .line and .col), pycvc.script_reader, to_string, values_equal, and the value codec encode_value / decode_value on a State node. - Exec.run raises pycvc.parse_error for a syntax error whether the script was parsed at submit or read inside the process (it re-parses a killed process's script to recover the position). - The pycvc README's Status/Design sections described a v0 facade; they now describe the direct-wrap design. test_pycvc_state_exec.py (13 tests) covers each against the C++ behaviour. Locally: those plus pycvc's exec, async, coro, ari, state and core suites pass (87), except test_volume_rawiv_roundtrip, which fails the same way on the unmodified build (a Windows temp-file lock). * pycvc: state_exec parity part 2 (environments, builtins, schedulers) Wrap the scheduler surface of cvc::state_exec under the C++ names: - environment (shared, as environment_ptr) and builtins: make_default_environment, and register_fn taking a Python callable or a pycvc.value holding a native fn. - execute_options (on_complete is a Python callable, signal_handlers a dict), process_info, scheduler_stats, percent_encode/percent_decode. - process_status and scheduling_policy as enum.IntEnum classes; an enum argument must be a member, so to_string(process_status) and to_string(value_t) keep their shared name. - scheduler and async_scheduler as real classes with their non-coroutine members. run/step and sync_run/sync_step release the GIL; Python callbacks retake it. - Exec.scheduler() returns Exec's async_scheduler, which keeps the Exec alive. Empty optionals are None, except for values: get_result, pop_pending_message and environment.lookup raise KeyError, since None is nil. The value_t typecheck now runs last, so execute(str) picks the script overload. test_pycvc_state_exec_sched.py repeats the C++ scheduler tests on both classes. The parity doc and README are updated. * Rerun CI on master (the merge push ran while the PR still targeted #653's branch) * pycvc: environment.frozen (#660), and the set_existing note now that its comment is fixed * pycvc: state_exec parity part 3 (evaluators, processes, policy, intrinsics) Wrap the rest of cvc::state_exec's public surface under the C++ names: - stackless_evaluator and evaluator, with evaluator_state, eval_frame, eval_phase (an IntEnum) and evaluation_stats; evaluation_interrupted and evaluation_timeout as RuntimeError subclasses. The evaluators release the GIL while they evaluate, so another thread can interrupt, pause or resume them. on_complete is a Python callable. - The process struct (shared, as process_ptr), watch_event and watch_handler_entry, make_process, and the scheduler calls that take them: current_process() and queue_watch_event(). scheduler_base is the common base of the two schedulers. - encode_/decode_environment and encode_/decode_evaluator_state. - resource_policy (mode as an IntEnum), process_limits, validate_limits. - intrinsics_context, channel_policy, register_intrinsics, apply_chroot, resolve_channel_key, make_future, future_channel_of. Every function register_intrinsics adds keeps its context alive, and the context keeps the scheduler and channel policy assigned to it. - From #660: kDefaultMaxAllocBytes, eval_alloc_limit, eval_alloc_check, alloc_bytes, scheduler.max_alloc_bytes(), max_alloc_bytes_from_state. A reference into an object (evaluator_state.stats, evaluator.stats(), process.state) keeps its owner alive; other members read and write as copies. evaluator_state is move-only, so it is moved into Python. test_pycvc_state_exec_eval.py repeats the C++ tests. The parity doc, whose inventory now includes what landed on master since the audit, and the README are updated. * pycvc: state_exec parity part 4 (data jobs, heap, stdlib registry, any-depth values) - The value conversion walks with an explicit stack instead of recursing, so a value nested any depth crosses both ways (the parser's max_depth default is going to 100000). A list or dict shared within a value converts to one object, both ways: sharing survives the trip, a shared structure converts in time proportional to its size, and a DSL list that holds itself becomes a Python list that holds itself. A Python value that holds itself still raises ValueError. - The deep-data jobs (#658, #665): data_job, the make_*_job factories, run_to_completion, run_data_job (a Python callable as `then`), call_function, is_compound, count_data_nodes, kDataNodeBytes. A job is handed to Python owned (std::unique_ptr); `who` is kept for good, as C++ holds the pointer. - The heap (#667): heap_options and its tree settings, measure_heap_share, make_heap_share_job, heap_remeasure_bytes, and the per-thread cycle collector; the heap fields on process and process_info, evaluator_state.heap_measured, stackless_evaluator.global_env(), eval_alloc_note. - stdlib_registry: standard() (a copy), shared_scope(), register_function (a Python callable or a pycvc.value native), list_*, import_module, lookup_qualified (KeyError where C++ returns nullptr). - SWIG_STD_MOVE moves on MSVC too (it reports __cplusplus 199711L without /Zc:__cplusplus), so a unique_ptr argument can be passed. test_pycvc_state_exec_data.py covers the new calls, and test_pycvc_state_exec.py the any-depth, sharing conversion. The parity doc and README are updated. * pycvc: wrap #682/#683's call_function continuation, make_generator_job and the native nesting limits
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #682 (B1). Review the last commit, c4ca22c. This is the last part of finishing the deep-data work (#678, #682, this).
Why
A class method ran to completion on a temporary evaluator inside one native call. A DSL generator's body ran to its next yield the same way. Neither counted against the process's
max_steps; only the time deadline bounded them. A method also cost over 10 KiB of native stack per nested call, the heaviest kind of level #682 has to limit.Methods are closures, run as the process's frames
selffirst.detail::class_methodand its temporary evaluator are gone.sendalready calls throughcall_function(state_exec: the stdlib materializers run as sliced data steps, and predicates get them #665), so a method's body runs as frames of the calling process.initthrough a newcall_function(fn, args, who, then). Its value is the instance onceinithas run, even wheninit's frames or deep-data work take several steps.init/finalize. They tail-call throughcall_function.super. Its native path now sharesstep_apply's handling (apply_native), so a native super method that defers works.sends run on a stack that could not hold two before.Generators are advanced as data jobs
make_generator_jobadvances a generator a node at a time. A node is one step of a DSL generator's evaluator, or one value of a native generator.nextandcollectrun that job.collectchecks the allocation limit per value, and the deadline per value when nothing slices.forover a DSL generator gets each value from a child native call,(%generator-next %generator), in a scope of the loop frame's own. The job slices and waits like any other.data.step_nodes= N, a generator body runs N steps per process step, in data steps, againstmax_steps/max_time. Its own deep-data work slices the same way.range) inforandnextstill advance inline.Behavior changes
max_stepsis now killed at that budget.initused to throw out ofscheduler::run()at the deadline. Nowmax_timekills the process like any loop (SchedulerTest.TimeBudgetStopsALoopingInitMethod, which replacesDeadlineBoundsUncappedNestedMethodStep).cvc exec), for the child frame and job per value.EvalArithmeticLoop,EvalFibonacciandSchedulerThroughputwere within this busy machine's run-to-run noise (up to 2x between runs).Tests
DataStepsScheduler:MethodStepsCountAgainstMaxSteps;GeneratorBodiesRunASlicePerStep: values, data nodes and data steps, and killed atmax_steps;ForOverAGeneratorGivesTheSameSlicedOrNot;ConstructorReturnsTheInstanceAfterInitDefers: an inherited closureinitthat compares deep lists as data steps.StateExecNativeNesting.MethodsCallingMethodsAreFrames.kNativeStackReserverecords this.docs/STATE_EXEC_GRAMMAR.mdnow covers methods and generators.Local results (MSVC 14.38 Release, Windows)
state_exec_deep_data_testpasses at 64 KiB.