state_exec: a limit on native call nesting; class teardown and the closure codec without recursion

docs/STATE_EXEC_GRAMMAR.md

@@ -261,6 +261,25 @@ Measured per node (MSVC Release, `state_exec_bench_test` `DataWalkThroughput`):
 The stdlib jobs (`StdlibJobThroughput`) cost 5-20 ns a node for elements and 200-370 ns for a
 256-byte copy.
 
+**Snapshots of closures.** The codec writes each scope a value's closures capture once, in an
+`__envs__` table under the snapshot node, and a closure refers to its scope's entry. A function
+that `defun` bound in the scope it captures is no cycle, and closures that share a scope share
+it again when decoded. Older snapshots, with each closure's scope chain written inside it, still
+decode.
+
+### Native call nesting
+
+Some native functions call other functions themselves: `apply`, `send`, a class constructor
+running `init`, an inherited `init` running its superclass's. Some run a nested evaluator: a class
+method, a generator advancing. Each of these nests on the native stack, so a value or class that
+refers to itself could ask for unbounded nesting: `(apply apply L)` with `L` containing itself, or
+a class whose `init` is its own constructor. A level is refused with an error ("native calls nested
+too deeply") past 256 levels (`kMaxNativeNesting`), or when less of the thread's native stack is
+left than `kNativeStackReserve`: 128 KiB on Windows, where a catch block runs before the frames
+above it unwind, and 32 KiB elsewhere. Levels cost from about 0.5 KiB (`apply`) to over 10 KiB (a
+class method) in a release build. DSL functions calling DSL functions are not levels: they are
+evaluator frames on the heap.
+
 ### Budget by kind
 
 Each process's steps are split by what they were spent on (`step_kind`):

inc/cvc/state/state_exec/data_job.h

@@ -143,12 +143,46 @@ value_t run_data_job(std::unique_ptr<data_job> job, std::function<value_t(value_
 /// throws. `who` names the caller in errors.
 value_t call_function(const value_t &fn, std::span<const value_t> args, const char *who);
 
+/// How native calls may nest on one thread's native stack. A level is a native function calling
+/// another function itself (apply, send, a class constructor running init, an inherited init
+/// running its superclass's, a job's call made where the job runs) or running a nested evaluator
+/// (a class method, a generator advancing). The evaluator applying a native function is not a
+/// level. A self-containing argument list, a class whose init is its own constructor, or a long
+/// enough chain of such calls asks for unbounded nesting, so a level is refused (it throws) past
+/// kMaxNativeNesting levels, or when less than kNativeStackReserve bytes of the thread's native
+/// stack are left (where the platform says where its stack ends). Levels cost from about 0.5 KiB
+/// (apply) to over 10 KiB (a class method's nested evaluator) in a release build.
+constexpr std::size_t kMaxNativeNesting = 256;
+
+/// The native stack a level leaves for the error it may throw. MSVC runs a catch block before
+/// unwinding the frames above it, so on Windows whatever catches the error (a scheduler killing
+/// the process, say) runs at the depth it was thrown from and needs this room; elsewhere a throw
+/// unwinds first and needs little.
+#ifdef _WIN32
+constexpr std::size_t kNativeStackReserve = 128 * 1024;
+#else
+constexpr std::size_t kNativeStackReserve = 32 * 1024;
+#endif
+
 /// Count `nodes` units of deep-data work done in the current evaluator step (a walk run to
 /// completion counts itself; the release backlog counts what it frees).
 void count_data_nodes(std::size_t nodes);
 
 namespace detail {
 
+/// RAII: one level of native nesting on this thread (see kNativeStackReserve). Throws, naming
+/// `who`, when the stack is too close to its end for another.
+class native_nesting {
+public:
+  explicit native_nesting(const char *who);
+  ~native_nesting();
+  native_nesting(const native_nesting &) = delete;
+  native_nesting &operator=(const native_nesting &) = delete;
+
+  /// Levels this thread is nested now.
+  static std::size_t depth();
+};
+
 /// A native call's deep-data job left to run after the call returned (see run_data_job).
 struct deferred_data_job {
   std::unique_ptr<data_job> job;

inc/cvc/state/state_exec/generator.h

@@ -65,6 +65,7 @@ inline std::optional<value_t> generator_next(generator &gen) {
   // (generator (lambda () (while t 1)))) would spin here inside one native step, so honour
   // the evaluation deadline (a no-op when none is armed) rather than looping forever.
   gen.state.yielded = false;
+  detail::native_nesting level("generator"); // a nested evaluator
   while (!gen.state.done && !gen.state.yielded) {
     if (eval_deadline_expired())
       throw std::runtime_error("generator: evaluation exceeded time budget");

inc/cvc/state/state_exec/state_value_codec.h

@@ -25,7 +25,9 @@ namespace cvc::state_exec {
 
 /// Encode a value_t into a state subtree.
 /// native_fn values are stored as a marker ("__native__") since functions
-/// cannot be serialized.
+/// cannot be serialized. The scopes the value's closures capture are written once each,
+/// in an `__envs__` table under `node` that the closures refer to, so closures that share a
+/// scope share it again when decoded, and a closure its own scope binds is no cycle.
 void encode_value(cvc::state &node, const value_t &val);
 
 /// Decode a value_t from a state subtree previously written by encode_value.
@@ -36,12 +38,14 @@ value_t decode_value(cvc::state &node);
 // Environment codec
 // ---------------------------------------------------------------------------
 
-/// Encode an environment chain into a state subtree.
-/// Flattens the scope chain into an indexed list of scope nodes, each
-/// containing its bindings.  native_fn bindings are skipped.
+/// Encode an environment chain into a state subtree: `node` refers to an entry of an
+/// `__envs__` table under it, and each scope in the chain (and each scope a closure bound
+/// in one captures) is an entry with its bindings and a reference to its enclosing scope.
+/// native_fn bindings are skipped.
 void encode_environment(cvc::state &node, const environment_ptr &env);
 
-/// Decode an environment chain from a state subtree.
+/// Decode an environment chain from a state subtree, written by encode_environment (or, in
+/// an older snapshot, as an inline list of scopes).
 environment_ptr decode_environment(cvc::state &node);
 
 // ---------------------------------------------------------------------------

src/cvc/CMakeLists.txt

@@ -212,6 +212,8 @@ set(SOURCE_FILES
   state/state_exec/async_stackless_evaluator.cpp
   state/state_exec/state_value_codec.cpp
   state/state_exec/data_job.cpp
+  state/state_exec/native_stack.cpp
+  state/state_exec/native_stack.h
   state/state_exec/heap.cpp
   state/state_exec/closure_natives.h
   state/state_exec/process.cpp

src/cvc/state/state_exec/closure_natives.h

@@ -11,19 +11,25 @@
 
 namespace cvc::state_exec::detail {
 
+// Each drops what it holds through drop_owner, as value_t does: a class dict holds these natives,
+// and they hold the next class dict up the chain, so plain shared_ptr destructors would free a
+// long superclass chain with a native stack frame per class.
+
 /// A defclass method: runs `method` (self first, then the call's arguments) on a fresh evaluator
 /// over `global_env`.
 struct class_method {
   closure_ptr method;
   environment_ptr global_env;
   value_t operator()(std::span<const value_t> args) const;
+  ~class_method() { drop_owner(method); }
 };
 
 /// A defclass constructor: a new instance dict whose __class__ is `cls_dict`, passed to the
 /// class's init method if it has one.
 struct class_constructor {
   dict_ptr cls_dict;
   value_t operator()(std::span<const value_t> init_args) const;
+  ~class_constructor() { drop_owner(cls_dict); }
 };
 
 /// The init or finalize a class gets when it defines none: the superclass's `name` method, if
@@ -32,6 +38,7 @@ struct class_inherited_method {
   dict_ptr super_cls; // may be null: no superclass
   std::string name;   // "init" or "finalize"
   value_t operator()(std::span<const value_t> args) const;
+  ~class_inherited_method() { drop_owner(super_cls); }
 };
 
 } // namespace cvc::state_exec::detail

src/cvc/state/state_exec/data_job.cpp

@@ -5,6 +5,8 @@
              work on the heap and run a slice at a time.
 */
 
+#include "native_stack.h"
+
 #include <algorithm>
 #include <charconv>
 #include <cstdint>
@@ -512,6 +514,9 @@ class call_walk final : public data_job {
 // The data side of the evaluator step running on this thread, if any.
 thread_local detail::data_step_scope *t_scope = nullptr;
 
+// Levels of native nesting on this thread (detail::native_nesting).
+thread_local std::size_t t_native_depth = 0;
+
 // Run `walk` to completion and count its nodes into the current step.
 template <class Walk> Walk &finish_walk(Walk &walk) {
   walk.step(0);
@@ -594,6 +599,7 @@ value_t call_here(const data_job::call_request &call) {
           scope->allow_defer(true);
       }
     } guard;
+    detail::native_nesting level(call.who);
     return (*fn)(call.args);
   }
   if (std::holds_alternative<closure_ptr>(call.fn->v))
@@ -661,7 +667,11 @@ value_t run_data_job(std::unique_ptr<data_job> job, std::function<value_t(value_
     if (!fn || (slicing && scope->nodes_ >= scope->step_nodes_))
       break;
     const data_job::call_request call = *job->take_call();
-    value_t value = (*fn)(call.args);
+    value_t value;
+    {
+      detail::native_nesting level(call.who);
+      value = (*fn)(call.args);
+    }
     if (scope->deferred_) {
       waits_on = std::move(scope->deferred_);
       break;
@@ -675,8 +685,10 @@ value_t run_data_job(std::unique_ptr<data_job> job, std::function<value_t(value_
 }
 
 value_t call_function(const value_t &fn, std::span<const value_t> args, const char *who) {
-  if (const auto *native = std::get_if<native_fn>(&fn.v))
+  if (const auto *native = std::get_if<native_fn>(&fn.v)) {
+    detail::native_nesting level(who);
     return (*native)(args);
+  }
   if (const auto *cls = std::get_if<closure_ptr>(&fn.v); cls && *cls)
     return run_data_job(
         std::make_unique<call_walk>(fn, std::vector<value_t>(args.begin(), args.end()), who));
@@ -692,6 +704,20 @@ void count_data_nodes(std::size_t nodes) {
   }
 }
 
+// -- native_nesting --------------------------------------------------------------
+
+detail::native_nesting::native_nesting(const char *who) {
+  if (t_native_depth >= kMaxNativeNesting || native_stack_left() < kNativeStackReserve)
+    throw std::runtime_error(std::string(who) + ": native calls nested too deeply (" +
+                             std::to_string(t_native_depth) +
+                             " levels; a value or class that refers to itself?)");
+  ++t_native_depth;
+}
+
+detail::native_nesting::~native_nesting() { --t_native_depth; }
+
+std::size_t detail::native_nesting::depth() { return t_native_depth; }
+
 // -- data_step_scope -------------------------------------------------------------
 
 detail::data_step_scope::data_step_scope(std::size_t step_nodes, evaluation_stats &stats,

src/cvc/state/state_exec/native_stack.cpp

@@ -0,0 +1,73 @@
+/*
+  Copyright 2026 The University of Texas at Austin
+
+  native_stack — where the calling thread's native stack ends, so nesting native calls can stop
+                 short of it (detail::native_nesting).
+*/
+
+#include "native_stack.h"
+
+#include <cstdint>
+#include <limits>
+
+#if defined(_WIN32)
+#ifndef WIN32_LEAN_AND_MEAN
+#define WIN32_LEAN_AND_MEAN
+#endif
+#ifndef NOMINMAX
+#define NOMINMAX
+#endif
+#include <windows.h>
+#elif defined(__EMSCRIPTEN__)
+#include <emscripten/stack.h>
+#elif defined(__APPLE__) || defined(__linux__)
+#include <pthread.h>
+#endif
+
+namespace cvc::state_exec::detail {
+
+namespace {
+
+// The lowest address of this thread's stack, or 0 when the platform does not say.
+std::uintptr_t stack_low() {
+#if defined(_WIN32)
+  // The reservation a stack address belongs to starts at the stack's lowest address (the guard
+  // pages are inside it). VirtualQuery, unlike GetCurrentThreadStackLimits, is in every Windows
+  // the build targets (_WIN32_WINNT is 0x0601).
+  MEMORY_BASIC_INFORMATION info{};
+  const char here = 0;
+  if (VirtualQuery(&here, &info, sizeof info) == 0)
+    return 0;
+  return reinterpret_cast<std::uintptr_t>(info.AllocationBase);
+#elif defined(__EMSCRIPTEN__)
+  return static_cast<std::uintptr_t>(emscripten_stack_get_end());
+#elif defined(__APPLE__)
+  const pthread_t self = pthread_self();
+  const auto high = reinterpret_cast<std::uintptr_t>(pthread_get_stackaddr_np(self));
+  return high - pthread_get_stacksize_np(self);
+#elif defined(__linux__)
+  pthread_attr_t attr;
+  if (pthread_getattr_np(pthread_self(), &attr) != 0)
+    return 0;
+  void *addr = nullptr;
+  std::size_t size = 0;
+  const int rc = pthread_attr_getstack(&attr, &addr, &size);
+  pthread_attr_destroy(&attr);
+  return rc == 0 ? reinterpret_cast<std::uintptr_t>(addr) : 0;
+#else
+  return 0;
+#endif
+}
+
+} // namespace
+
+std::size_t native_stack_left() {
+  thread_local const std::uintptr_t low = stack_low();
+  if (low == 0)
+    return std::numeric_limits<std::size_t>::max();
+  const char here = 0;
+  const auto sp = reinterpret_cast<std::uintptr_t>(&here);
+  return sp > low ? static_cast<std::size_t>(sp - low) : 0;
+}
+
+} // namespace cvc::state_exec::detail

src/cvc/state/state_exec/native_stack.h

@@ -0,0 +1,15 @@
+// Private: how much native stack the calling thread has left, for detail::native_nesting.
+#ifndef CVC_STATE_EXEC_NATIVE_STACK_H
+#define CVC_STATE_EXEC_NATIVE_STACK_H
+
+#include <cstddef>
+
+namespace cvc::state_exec::detail {
+
+/// Bytes of native stack left below the caller on this thread (stacks grow down on every
+/// platform libcvc builds for), or SIZE_MAX when the platform does not say where its stack ends.
+std::size_t native_stack_left();
+
+} // namespace cvc::state_exec::detail
+
+#endif // CVC_STATE_EXEC_NATIVE_STACK_H

src/cvc/state/state_exec/stackless_evaluator.cpp

@@ -1262,6 +1262,7 @@ value_t detail::class_method::operator()(std::span<const value_t> call_args) con
     for (size_t j = 0; j < method->params.size(); ++j)
       local->set(method->params[j].name, all_args[j]);
   }
+  detail::native_nesting level("method"); // a nested evaluator
   stackless_evaluator temp(global_env);
   if (method->body.size() == 1)
     return temp.evaluate(method->body[0], local);
@@ -1281,8 +1282,11 @@ value_t detail::class_constructor::operator()(std::span<const value_t> init_args
       std::vector<value_t> a;
       a.push_back(inst_val);
       a.insert(a.end(), init_args.begin(), init_args.end());
-      if (auto *fn = std::get_if<native_fn>(&v.v))
-        (*fn)(a);
+      if (auto *fn = std::get_if<native_fn>(&v.v)) {
+        const native_fn init = *fn; // init may change the class dict under `v`
+        detail::native_nesting level("constructor");
+        init(a);
+      }
       break;
     }
   }
@@ -1293,8 +1297,11 @@ value_t detail::class_inherited_method::operator()(std::span<const value_t> args
   if (super_cls) {
     for (auto &[k, v] : *super_cls) {
       if (k == name) {
-        if (auto *fn = std::get_if<native_fn>(&v.v))
-          return (*fn)(args);
+        if (auto *fn = std::get_if<native_fn>(&v.v)) {
+          const native_fn inherited = *fn; // it may change the class dict under `v`
+          detail::native_nesting level(name.c_str());
+          return inherited(args);
+        }
         break;
       }
     }