state_exec: a limit on native call nesting; class teardown and the closure codec without recursion

[transfix]

The second part of finishing the deep-data work (the first is #678). A sweep of the repo for native recursion found three problems that #657/#658 left. Each can crash a process from any full-environment script.

1. Native call nesting had no limit

A native function that calls another function itself (apply, send, a class constructor running init, an inherited init running its superclass's), or runs a nested evaluator (a class method, a generator advancing), nests on the native stack. Several scripts nest without bound and crash the process:

• (apply apply L) with L containing itself;
• send re-sending through itself;
• a class whose init is its own constructor;
• a long chain of inherited inits;
• methods calling methods;
• generators of generators.

detail::native_nesting now counts these levels. It runs in call_function, call_here, run_data_job's job calls, the class constructor, inherited methods, class methods' nested evaluator and generator_next. A level throws "<who>: native calls nested too deeply (N levels; ...)" in either of two cases:

• past kMaxNativeNesting (256) levels;
• when less than kNativeStackReserve of the thread's native stack is left: 128 KiB on Windows, 32 KiB elsewhere.

native_stack.cpp asks the platform where the stack ends: VirtualQuery, pthread_getattr_np, pthread_get_stackaddr_np or emscripten_stack_get_end.

Why both checks, measured in a Release build:

Kind of level	Stack per level
apply	about 0.5 KiB
nested generator	2-4 KiB
class method's evaluator	over 10 KiB

• No single level count fits a 64 KiB stack with the heavy kinds, and the light kinds can still take thousands of levels on a big stack.
• With only the stack check, a throw from about 7600 levels deep crashed on Windows. MSVC runs a catch block before unwinding the frames above it, so whoever catches the error (a scheduler killing the process) runs on top of the full stack. The level count bounds that depth; the larger Windows reserve leaves the catch block room.

DSL functions calling DSL functions are not levels; they are evaluator frames on the heap.

2. Freeing a long class chain recursed

class_constructor, class_inherited_method and class_method held their class dict or closure as a raw shared_ptr. Freeing a superclass chain therefore took a native frame per class. They now drop it through drop_owner, as value_t does. The constructor and inherited methods also call a copy of the function, not a reference into a class dict the callee may change (set-attr on __cls_dict__).

3. The snapshot codec recursed through closures, forever for a defun

The codec wrote each closure's scope chain inside the closure, and each binding with a fresh encode_value. That cost one native level per closure nesting. A function whose scope binds the function itself made encode_environment recurse forever. Any defun'd function is one, so snapshotting a paused process that had defined one overflowed the stack. (Nothing in production snapshots processes yet; tests and future migration do.)

Values, closure bodies and scopes are now one walk with one explicit stack:

• Writing. Each scope the walk reaches is an entry of an __envs__ table under the node the walk started at, holding its bindings (sorted) and an __outer__ reference. A closure's __env__ is a reference to an entry (env_ref), or nil.
• Reading. A referenced entry becomes one environment, so closures that shared a scope share it again. A bad reference, or an __outer__ chain that loops, throws (unlinked first, so nothing leaks).
• Older snapshots still decode. A closure with no __env__, or an inline env_chain, is read the old way, its body expressions and bindings each with their own __spill__ table.

Tests

• StateExecNativeNesting, 7 tests:
  • each runaway case throws on a 64 KiB stack instead of crashing, and leaves no level counted;
  • legitimate nesting still works on 1 MiB;
  • a 4 MiB stack stops at the level count;
  • FreeingALongClassChain (10000 classes on 64 KiB) crashed before fix 2 (checked by removing the destructors).
• EnvironmentCodecTest:
  • AFunctionItsOwnScopeBinds: a defun'd factorial round-trips, keeps its scope, and runs;
  • ClosuresThatShareAScopeStillShareIt;
  • OlderSnapshotsStillDecode: the old layout built by hand, spill table included;
  • BadScopeReferencesThrow.
• StateExecDeepDataCodec.ClosureChainRoundTrip: 10000 closures, each bound in the next one's scope, on the small stack.
• Docs: "Native call nesting" and "Snapshots of closures" in docs/STATE_EXEC_GRAMMAR.md.

Local results (MSVC 14.38 Release, Windows)

• All 28 state_exec, Ariadne, state, state_list and app suites pass (1671 tests).
• state_exec_deep_data_test passes at 64 KiB.
• clang-format 18 is clean on the changed lines.

Next (B2): class methods as closures run through call_function (frames, not a temporary evaluator), and generators advanced as data jobs. That removes the heaviest nesting levels and makes a generator body's steps sliced and counted.