state_exec: class methods and generator bodies run inside the calling process

docs/STATE_EXEC_GRAMMAR.md

@@ -270,15 +270,25 @@ decode.
 ### Native call nesting
 
 Some native functions call other functions themselves: `apply`, `send`, a class constructor
-running `init`, an inherited `init` running its superclass's. Some run a nested evaluator: a class
-method, a generator advancing. Each of these nests on the native stack, so a value or class that
-refers to itself could ask for unbounded nesting: `(apply apply L)` with `L` containing itself, or
-a class whose `init` is its own constructor. A level is refused with an error ("native calls nested
-too deeply") past 256 levels (`kMaxNativeNesting`), or when less of the thread's native stack is
-left than `kNativeStackReserve`: 128 KiB on Windows, where a catch block runs before the frames
-above it unwind, and 32 KiB elsewhere. Levels cost from about 0.5 KiB (`apply`) to over 10 KiB (a
-class method) in a release build. DSL functions calling DSL functions are not levels: they are
-evaluator frames on the heap.
+running `init`, an inherited `init` running its superclass's. A generator advancing steps an
+evaluator of its own. Each of these nests on the native stack, so a value or class that refers to
+itself could ask for unbounded nesting: `(apply apply L)` with `L` containing itself, or a class
+whose `init` is its own constructor. A level is refused with an error ("native calls nested too
+deeply") past 256 levels (`kMaxNativeNesting`), or when less of the thread's native stack is left
+than `kNativeStackReserve`: 128 KiB on Windows, where a catch block runs before the frames above
+it unwind, and 32 KiB elsewhere. Levels cost from about 0.5 KiB (`apply`) to a few KiB (a
+generator) in a release build.
+
+DSL functions are not levels, and neither are class methods. A method is a closure in its class's
+dict, taking `self` first. `send`, `super` and a constructor's `init` call it through the
+evaluator (`call_function`), so its body runs as frames of the calling process: on the heap, a
+step at a time, against the process's budget.
+
+**Generators.** A DSL generator's body runs on an evaluator of its own, but `next`, `collect` and
+`for` advance it as a job (`make_generator_job`). Each step of its body, or each value of a native
+generator like `range`, is a deep-data node. So with `data.step_nodes` = N, the body runs N
+steps per process step, in data steps, against `max_steps` and `max_time`. A generator's own
+deep-data work slices the same way.
 
 ### Budget by kind
 

inc/cvc/state/state_exec/data_job.h

@@ -143,27 +143,44 @@ value_t run_data_job(std::unique_ptr<data_job> job, std::function<value_t(value_
 /// throws. `who` names the caller in errors.
 value_t call_function(const value_t &fn, std::span<const value_t> args, const char *who);
 
+/// As call_function, with `then` applied to the call's value: for a native function that has
+/// more to do after the call, such as a class constructor returning the new instance once init
+/// has run. The call is a one-call job either way (run_data_job), so `then` runs after the
+/// function finishes, however the evaluator ran it. `who` must outlive the call (a literal).
+value_t call_function(const value_t &fn, std::span<const value_t> args, const char *who,
+                      std::function<value_t(value_t)> then);
+
 /// How native calls may nest on one thread's native stack. A level is a native function calling
 /// another function itself (apply, send, a class constructor running init, an inherited init
-/// running its superclass's, a job's call made where the job runs) or running a nested evaluator
-/// (a class method, a generator advancing). The evaluator applying a native function is not a
-/// level. A self-containing argument list, a class whose init is its own constructor, or a long
-/// enough chain of such calls asks for unbounded nesting, so a level is refused (it throws) past
-/// kMaxNativeNesting levels, or when less than kNativeStackReserve bytes of the thread's native
-/// stack are left (where the platform says where its stack ends). Levels cost from about 0.5 KiB
-/// (apply) to over 10 KiB (a class method's nested evaluator) in a release build.
+/// running its superclass's, a job's call made where the job runs) or stepping a generator's own
+/// evaluator. The evaluator applying a native function is not a level, and neither is a DSL
+/// function or class method: those run as evaluator frames, on the heap. A self-containing
+/// argument list, a class whose init is its own constructor, or a long enough chain of such calls
+/// asks for unbounded nesting, so a level is refused (it throws) past kMaxNativeNesting levels, or
+/// when less than kNativeStackReserve bytes of the thread's native stack are left (where the
+/// platform says where its stack ends). Levels cost from about 0.5 KiB (apply) to a few KiB (a
+/// generator's evaluator) in a release build.
 constexpr std::size_t kMaxNativeNesting = 256;
 
 /// The native stack a level leaves for the error it may throw. MSVC runs a catch block before
 /// unwinding the frames above it, so on Windows whatever catches the error (a scheduler killing
-/// the process, say) runs at the depth it was thrown from and needs this room; elsewhere a throw
-/// unwinds first and needs little.
+/// the process, say) runs at the depth it was thrown from and needs this room: a throw with 32 KiB
+/// left crashed in a test, 64 KiB did not, and this doubles that for heavier catch blocks.
+/// Elsewhere a throw unwinds first and needs little.
 #ifdef _WIN32
 constexpr std::size_t kNativeStackReserve = 128 * 1024;
 #else
 constexpr std::size_t kNativeStackReserve = 32 * 1024;
 #endif
 
+/// Advance `gen` as a job: each step of a DSL generator's own evaluator (or each value of a
+/// native one) is a node, so a generator body runs a slice per step of the process advancing
+/// it, counted against its budget, instead of to its next yield inside one step. The result is
+/// a one-element list holding the value yielded, or an empty list once the generator is
+/// exhausted. With `collect`, the job runs the generator to exhaustion and its result is the
+/// list of everything it yielded.
+std::unique_ptr<data_job> make_generator_job(generator_ptr gen, bool collect = false);
+
 /// Count `nodes` units of deep-data work done in the current evaluator step (a walk run to
 /// completion counts itself; the release backlog counts what it frees).
 void count_data_nodes(std::size_t nodes);

inc/cvc/state/state_exec/stackless_evaluator.h

@@ -23,6 +23,7 @@
 #include <mutex>
 #include <optional>
 #include <set>
+#include <span>
 #include <string>
 #include <unordered_map>
 #include <vector>
@@ -223,6 +224,12 @@ class stackless_evaluator {
   void step_break_unwind(evaluator_state &state, eval_frame &frame);
 
   // Helpers
+  // A for loop over a DSL generator: push the call that advances it as a job.
+  void advance_generator(evaluator_state &state, eval_frame &frame);
+  // Apply native `fn` to `args` as the top frame's call: its value pops the frame, or, if it
+  // left deep-data work (a job, or a DSL function to call), the frame waits for that.
+  void apply_native(evaluator_state &state, eval_frame &frame, const native_fn &fn,
+                    std::span<const value_t> args);
   void call_closure(evaluator_state &state, eval_frame &frame, const closure_ptr &cls,
                     const std::vector<value_t> &args);
   // Push the frame that evaluates `cls`'s body in `local` (its parameters already bound).

src/cvc/state/state_exec/builtins.cpp

@@ -715,8 +715,15 @@ value_t builtin_next(std::span<const value_t> args) {
   auto *gp = std::get_if<generator_ptr>(&args[0].v);
   if (!gp || !*gp)
     throw std::runtime_error("next: expected generator");
-  auto result = generator_next(**gp);
-  return result ? *result : nil_value;
+  if ((*gp)->native_step) { // range and friends: one cheap value
+    auto result = generator_next(**gp);
+    return result ? *result : nil_value;
+  }
+  // A DSL generator's body runs as a job, a slice per step when the process slices.
+  return run_data_job(make_generator_job(*gp), [](value_t got) {
+    const auto &l = *std::get<list_ptr>(got.v);
+    return l.empty() ? nil_value : l.front();
+  });
 }
 
 value_t builtin_is_generator(std::span<const value_t> args) {
@@ -778,17 +785,10 @@ value_t builtin_collect(std::span<const value_t> args) {
   auto *gp = std::get_if<generator_ptr>(&args[0].v);
   if (!gp || !*gp)
     throw std::runtime_error("collect: expected generator");
-  std::vector<value_t> result;
-  while (auto next = generator_next(**gp)) {
-    // Draining an unbounded/huge generator (e.g. (collect (range 0 1e11))) runs entirely
-    // inside this one native step; honour the evaluation deadline so it aborts at the time
-    // budget instead of hanging/OOMing (a no-op when no deadline is armed).
-    if (eval_deadline_expired())
-      throw std::runtime_error("collect: evaluation exceeded time budget");
-    eval_alloc_check(alloc_bytes(result.size() + 1, sizeof(value_t)), "collect");
-    result.push_back(std::move(*next));
-  }
-  return make_list(std::move(result));
+  // Draining a generator is a job: a value (or a step of a DSL generator's body) is a node, so
+  // a huge or unbounded one (e.g. (collect (range 0 1e11))) spreads over the process's steps,
+  // within its budget, and the allocation limit checks every value kept.
+  return run_data_job(make_generator_job(*gp, true));
 }
 
 // ---------------------------------------------------------------------------

src/cvc/state/state_exec/closure_natives.h

@@ -13,16 +13,8 @@ namespace cvc::state_exec::detail {
 
 // Each drops what it holds through drop_owner, as value_t does: a class dict holds these natives,
 // and they hold the next class dict up the chain, so plain shared_ptr destructors would free a
-// long superclass chain with a native stack frame per class.
-
-/// A defclass method: runs `method` (self first, then the call's arguments) on a fresh evaluator
-/// over `global_env`.
-struct class_method {
-  closure_ptr method;
-  environment_ptr global_env;
-  value_t operator()(std::span<const value_t> args) const;
-  ~class_method() { drop_owner(method); }
-};
+// long superclass chain with a native stack frame per class. (A class's own methods are plain
+// closures in its dict.)
 
 /// A defclass constructor: a new instance dict whose __class__ is `cls_dict`, passed to the
 /// class's init method if it has one.

src/cvc/state/state_exec/data_job.cpp

@@ -511,9 +511,86 @@ class call_walk final : public data_job {
   value_t result_;
 };
 
+// A generator advanced a node at a time (make_generator_job).
+class generator_walk final : public data_job {
+public:
+  generator_walk(generator_ptr gen, bool collect) : gen_(std::move(gen)), collect_(collect) {}
+
+  bool step(std::size_t nodes) override;
+
+  value_t result() override { return make_list(std::move(values_)); }
+
+private:
+  // Keep a value it yielded. True when the walk goes on (collect).
+  bool keep(value_t v) {
+    if (collect_)
+      eval_alloc_check(alloc_bytes(values_.size() + 1, sizeof(value_t)), "collect");
+    values_.push_back(std::move(v));
+    return collect_;
+  }
+
+  generator_ptr gen_;
+  bool collect_;
+  bool advancing_ = false; // a DSL generator is part way to its next yield
+  std::vector<value_t> values_;
+};
+
 // The data side of the evaluator step running on this thread, if any.
 thread_local detail::data_step_scope *t_scope = nullptr;
 
+bool generator_walk::step(std::size_t nodes) {
+  generator &g = *gen_;
+  const std::size_t start = nodes_;
+  auto spent = [&] { return nodes > 0 && nodes_ - start >= nodes; };
+  for (;;) {
+    if (g.exhausted)
+      return true;
+    if (g.native_step) { // range and friends: a value is a node
+      if (spent())
+        return false;
+      if (eval_deadline_expired()) // (collect (range 0 1e11)) when nothing slices
+        throw std::runtime_error(std::string(collect_ ? "collect" : "next") +
+                                 ": evaluation exceeded time budget");
+      std::optional<value_t> v = g.native_step();
+      ++nodes_;
+      if (!v) {
+        g.exhausted = true;
+        return true;
+      }
+      if (!keep(std::move(*v)))
+        return true;
+      continue;
+    }
+    if (!advancing_) {
+      g.state.yielded = false;
+      advancing_ = true;
+      // Deep-data work in the body slices like the process's own.
+      if (detail::data_step_scope *scope = t_scope)
+        g.state.data_step_nodes = scope->step_nodes();
+    }
+    while (!g.state.done && !g.state.yielded) {
+      if (spent())
+        return false;
+      if (eval_deadline_expired())
+        throw std::runtime_error("generator: evaluation exceeded time budget");
+      {
+        detail::native_nesting level("generator"); // its own evaluator, on this stack
+        g.evaluator.step(g.state);
+      }
+      ++nodes_;
+    }
+    advancing_ = false;
+    if (g.state.yielded) {
+      // A yield that was also the body's last expression counts; the next advance ends it.
+      if (!keep(g.state.result))
+        return true;
+      continue;
+    }
+    g.exhausted = true;
+    return true;
+  }
+}
+
 // Levels of native nesting on this thread (detail::native_nesting).
 thread_local std::size_t t_native_depth = 0;
 
@@ -582,6 +659,12 @@ std::unique_ptr<data_job> make_substitute_job(value_t tmpl,
   return std::make_unique<substitute_walk>(std::move(tmpl), std::move(subst));
 }
 
+std::unique_ptr<data_job> make_generator_job(generator_ptr gen, bool collect) {
+  if (!gen)
+    throw std::runtime_error(collect ? "collect: expected generator" : "next: expected generator");
+  return std::make_unique<generator_walk>(std::move(gen), collect);
+}
+
 namespace {
 // Make a call a job running here (not on an evaluator) waits for. A native function's own
 // deep-data work must run here too, or its result would be lost: deferral is off for the call.
@@ -695,6 +778,16 @@ value_t call_function(const value_t &fn, std::span<const value_t> args, const ch
   throw std::runtime_error(std::string(who) + ": not a function: " + fn.type_name());
 }
 
+value_t call_function(const value_t &fn, std::span<const value_t> args, const char *who,
+                      std::function<value_t(value_t)> then) {
+  const auto *cls = std::get_if<closure_ptr>(&fn.v);
+  if (!std::holds_alternative<native_fn>(fn.v) && !(cls && *cls))
+    throw std::runtime_error(std::string(who) + ": not a function: " + fn.type_name());
+  return run_data_job(
+      std::make_unique<call_walk>(fn, std::vector<value_t>(args.begin(), args.end()), who),
+      std::move(then));
+}
+
 void count_data_nodes(std::size_t nodes) {
   if (nodes == 0)
     return;

src/cvc/state/state_exec/heap.cpp

@@ -58,11 +58,7 @@ template <class F> void value_edges(const value_t &v, F &f) {
           if (x)
             f(node_kind::data, x);
         } else if constexpr (std::is_same_v<T, native_fn>) {
-          if (const auto *m = x.template target<detail::class_method>()) {
-            if (m->method)
-              f(node_kind::closure, m->method);
-            env_edge(m->global_env, f);
-          } else if (const auto *c = x.template target<detail::class_constructor>()) {
+          if (const auto *c = x.template target<detail::class_constructor>()) {
             if (c->cls_dict)
               f(node_kind::dict, c->cls_dict);
           } else if (const auto *i = x.template target<detail::class_inherited_method>()) {