Skip to content

MILAB-6913: exec fallbacks in the workflow SDK - #1948

Open
blackcat wants to merge 44 commits into
mainfrom
MILAB-6913_exec-fallbacks
Open

blackcat wants to merge 44 commits into
mainfrom
MILAB-6913_exec-fallbacks

Conversation

@blackcat

@blackcat blackcat commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

SDK part of exec fallbacks (spec; implementation decisions). Backend: milaboratory/pl#2332 (WorkdirSave v3, restarts) on top of #2288 (RunCommand v3, already on main).

What changes

  • exec.builder():
    • New methods:
      • copy()
      • fallbacks(policy), keyed by failure reason, with exitCode, messageMatch, backendHasGPU and timeout attributes
      • verify(), which checks for cycles and that every handler declares the same outputs as the root
      • plan()
    • The result object gains getAttempts().
    • An exec without .fallbacks() renders byte-identical exec.exec inputs; its CID is pinned by a test.
  • Every exec on a RunCommand v3 backend goes through the outcome router. A software failure is an outcome; the router picks a handler or raises today's error text plus the attempt history. Each fallback attempt is a full exec-impl render inside the root render, from a fresh workdir, within the backend's attempt cap.
  • Restart after a lost working directory (spec §14). With workdirSaveV3, the wd-retrier reads the WorkdirSave v3 outcome map and runs the same plan again on WorkdirMissing / FileMissing, up to the backend's restartLimit. With workdir processors, a probe save heads the chain so a lost directory restarts instead of stalling.
  • Streams. With .fallbacks(), fileStreams[name] is a template-managed StreamCollection/1 holding attempt<K> managers, outcome<K> entries and a stream mirror for old desktops. Blocks need no change.
  • Logs driver. getLogHandle returns a composite handle for a collection: every attempt's text, a short delimiter after each failed one, the running attempt live. It is encoded as an ordinary live/ready log handle, so old models and uikit PlLogView accept it.
  • Compatibility.
    • Old backends: v2 path; .fallbacks() is ignored with a notice.
    • Old desktops: they follow the stream mirror.
    • Old SDKs on a new backend: unchanged.
    • Cost: a plain exec renders 4 templates (3 before).
  • Fixed in passing: workdir.save().saveFileSetContent never keyed the content by name.

Accepted limitations are listed in the decisions file (D12, D14), e.g. on a v3 backend a never-written saveFile makes all of the exec's outputs read as errors (on v1 some resolved by timing).

Most critical files

  • sdk/workflow-tengo/src/exec/index.lib.tengo: builder (fallbacks, verify, copy, plan, getAttempts)
  • sdk/workflow-tengo/src/exec/outcome-router.tpl.tengo: the decision per attempt
  • sdk/workflow-tengo/src/exec/wd-retrier.tpl.tengo, wave2.lib.tengo, save-chain.lib.tengo: save and restart
  • sdk/workflow-tengo/src/exec/fallback.lib.tengo: policy validation, matching, error text
  • sdk/workflow-tengo/src/exec/exec.tpl.tengo, exec-impl.tpl.tengo, prepare.lib.tengo: attempt rendering
  • sdk/workflow-tengo/src/exec/stream.lib.tengo, failed-streams.tpl.tengo: stream collections and failed-attempt logs
  • lib/node/pl-drivers/src/drivers/logs.ts, logs_composite.ts, helpers/logs_handle.ts; lib/model/common/src/drivers/log.ts: composite log handle

New resource structures

Dumped from this SDK on the backend of milaboratory/pl#2332; frames mark the resources the feature adds.

Plain exec on a v3 backend (no fallbacks)
plain v3 exec

Fallback: attempt 0 failed, attempt 1 running
fallback attempt 1 running

Fallback: job finished
fallback finished

Restart after a lost working directory
restart finished

A failed attempt's stream that cannot be saved
failed stream save

Tests

tests/workflow-tengo/src/exec/ covers:

Area Cases
RunCommand v3 AC1.1–1.3
Fallback policy AC2.1–2.5; GPUMissed → CPU; messageMatch; nested handlers
Streams and logs AC2.6, both old-driver and composite views, with the err.log variants; AC2.7
Compatibility AC3.1 (CID pinned, with and without fallbacks); AC3.3
Restart lost workdir; with a processor; of a fallback attempt; restart limit
Error isolation never-written files; an unsaveable failed-attempt log
Unit tests Tengo unit tests for the policy; 17 composite log handle tests
  • All 116 tests pass locally against the pl#2332 backend.
  • In CI, the restart tests skip until workdirSaveV3 is on pl main.
  • Independent review iterations: 5 (no open Major); review files are kept locally, not committed.

A v3 command finishes a software failure as an outcome instead of an error,
which is what exec fallbacks route on. The builder gets v3(attempt, jobId)
and the built command exposes the outcome output; limits gets
getRunCommandV3Type(queue), the capability check: the type is present in
the backend's resource types iff the backend has it.
WorkdirSave v3 reports a lost working directory or file as an outcome, not
an error, so the exec's wd-retrier can restart the step after a database
restore (decision D2). workdir.save(...).v3() and the save builder's v3
option select it and expose the outcome map; workdir processors save with v3
when the exec's save chain asks for it.
Pure functions shared by the exec builder and the outcome router: policy
validation and normalization, plan keys, picking the handler for an
outcome, and composing the final error from the structured outcome plus
the attempt history (spec §9, §10.4, §12.3).
…), plan()

Fallback plans are builders derived from the root, so the builder state moves
into one map and copy() clones it. fallbacks() validates and stores the policy;
verify() refuses cycles and handlers whose saved, streamed or processed outputs
differ from the root's, and run() calls it before any render. plan() is the old
run() up to the render inputs, now computed on a clone so it can be called for
every handler; with fallbacks on a RunCommand v3 backend it adds the
fallbackPlans and fallbackPolicy inputs, otherwise the inputs stay exactly as
before (AC3.1). The result gains getAttempts(). _forceRunCommandV2() is a test
seam that renders the exec as on a backend without RunCommand v3.
… wd-retrier, stream collections

On a backend with RunCommand v3 every exec-impl builds the command as v3 and
renders exec.outcome-router on its outcome. On success the router runs the
save chain (WorkdirSave v3) and exec.wd-retrier, which hands out the outputs
or, when the working directory was lost, renders the same plan again up to
the backend's restartLimit (spec §14). On a failure it renders the handler's
plan as the next attempt, an ephemeral exec-impl under the root render,
prepared by exec.prepare exactly as exec.exec prepares attempt 0; when no
handler applies or the backend attempt cap is reached it raises the composed
error with the history.

With fallbacks every streamed file is a StreamCollection/1 kept by the
templates: attempt<K> holds each attempt's manager, outcome<K> its history
entry, and stream the running attempt's live tail, then the winner's file,
so today's log driver follows the running attempt. The collection's exec
input carries the exec.exec inputs map as a stand-in for the render's CID.

Without RunCommand v3 exec-impl takes the previous path unchanged, plus the
one-entry attempts output, and fallbacks are ignored with a notice.
AC1.1-AC1.3 on RunCommand v3 directly; AC2.1-AC2.5 and a GPUMissed handler
on the exec builder; the old log driver's view of a fallback stream and the
collection after success (AC2.6, AC2.7); getAttempts() and a bare stream
manager without RunCommand v3 (AC3.3, through the _forceRunCommandV2 seam).
AC2.3 needs a backend with --runner-max-job-attempts 1 and runs only when
PL_ADDRESS_ATTEMPT_CAP_1 is set.
…e restart

AC3.1 pins the exec.exec input names of an exec without fallbacks and the
values that do not depend on the test software's version; the full dump was
compared with the SDK before the feature. The restart tests follow decision
D5: a first render runs a command, the test deletes its working directory, a
second render runs the save chain and the wd-retrier on it. They need the
backend's work storage root in PL_TEST_WORK_STORAGE_ROOT.
A minor release of workflow-tengo: new builder API and behaviour on backends
with RunCommand v3.
Decision D3 revised after the backend review: the save side has its own
capability key in getResourceTypesFor, so a backend may have RunCommand v3
without WorkdirSave v3. Fallbacks stay gated on runCommandV3; the save chain
uses WorkdirSave v3, and the wd-retrier restarts a lost working directory,
only when workdirSaveV3 is published. Without it the success path saves with
v1 and a lost directory is an error, as before. _forceWorkdirSaveV1() is a
test seam for that combination; it holds for the whole job.
…rSave v3

AC2.1 and AC2.6/AC2.7 run twice: as the backend has it, and through the
_forceWorkdirSaveV1 seam, a backend with RunCommand v3 but without
WorkdirSave v3.
The restart depends on its own capability since D3 was revised.
With .fallbacks() an exec's fileStreams entry is a StreamCollection/1
holding one StreamManager per attempt. Following its `stream` mirror
shows only the running attempt, so a failed attempt's log disappears
as soon as the fallback starts. LogsDriver.getLogHandle now returns a
composite handle for the collection: the attempts' logs in order, a
delimiter from each failed attempt's outcome, the last attempt live.
readText/lastLines map the viewer's offsets into the joined text, so
PlLogView reads it unchanged. The handle keeps the live/ready prefixes
so old block models still recognise it; a bare StreamManager keeps
today's handle.
@changeset-bot

changeset-bot Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 8f7495b

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 23 packages
Name Type
@milaboratories/pl-drivers Minor
@milaboratories/pl-model-common Minor
@platforma-sdk/model Minor
@platforma-sdk/workflow-tengo Minor
@milaboratories/pl-middle-layer Patch
@milaboratories/pl-model-middle-layer Patch
@milaboratories/pl-crash-recorder Patch
@milaboratories/pf-driver Patch
@milaboratories/pf-spec Patch
@milaboratories/pf-spec-driver Patch
@milaboratories/columns-collection-driver Patch
@milaboratories/pl-client Patch
@platforma-open/milaboratories.software-ptabler.schema Patch
@platforma-sdk/block-tools Patch
@platforma-sdk/ui-vue Minor
@milaboratories/uikit Patch
@platforma-sdk/test Minor
@platforma-sdk/pl-cli Patch
@milaboratories/pl-model-backend Patch
@milaboratories/pl-errors Patch
@milaboratories/pl-tree Patch
@milaboratories/ptabler-expression-js Patch
@platforma-sdk/tengo-builder Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@notion-workspace

Copy link
Copy Markdown

Review finding M1. A workdir processor reads the post-run directory through
a save of its own before the main save does; on a lost directory that save
never hands the directory on, the main WorkdirSave v3 never runs, and the
wd-retrier waited for its outcome forever. With processors, a WorkdirSave v3
with no rules now heads the save chain (a directory takes one save at a time,
so it is a link of the chain, not a save beside it); the wd-retrier decides
on that probe first and, when the directory is there, renders itself again
on the main save's outcomes. The save chain's outputs reach the wd-retrier in
one resource the input unmarshaller does not descend into, so a processor
error or an output that never resolves cannot fail or stall the decision.

Processors no longer get a saveVersion input (finding m5): the probe makes v3
saves in processors unnecessary, so processor templates, old or with closed
input schemas, are rendered exactly as before. Wave 2 moves into
:exec.wave2 so tests render the same code. A failed attempt saves its
streamed files in one save again, as a directory takes one save. The router
reads an absent attemptLimit as 1 (finding m9).
…llection

Review finding M2. A StreamCollection has the identity of its exec.exec
render, and a render with the same identity is normally reused as a whole.
When it is not (a failed render executed again by error recovery, or a
render that is gone while a block's previous outputs still hold its
collection), the new job reused the collection and left the earlier job's
attempt<K>/outcome<K> behind, so the composite log showed stale attempts and
never became final. Attempt 0 of a job now removes them before writing
attempt0. exec-impl also lists streamOwn among its outputs (nit n1).
…le collections

A lost or kept working directory with a workdir processor that reads a file
(M1); a restarted fallback attempt keeping its index and plan and writing
attempt<K> of the collection; the restart counter carried across attempts;
a job start removing an earlier job's attempts from a collection (M2). The
restart tests skip themselves on a backend without WorkdirSave v3.
…parts

Review findings m3 and m4. When an earlier attempt's part answers that its
handle is outdated (its live stream ended), the composite read now answers
the same, so the viewer fetches a new handle instead of seeing that
attempt's text vanish from the joined text. The running attempt's part is
exempt: its file may not exist yet, which reads the same way and must read
as empty (spec §13.1). A composite handle carries at most 256 parts: a
client-supplied handle with more is refused, and a collection with more
keeps the latest.
getAttempts() documents that a result reused from an exec rendered by an SDK
before fallbacks has no attempts (m1). fallbacks() documents that every
handler plan's inputs are awaited before attempt 0 starts (m8), and a
handler in a queue without RunCommand v3, while the root's has it, is a
render-time error: it would run without the router, its stream would never
join the collection and its own fallbacks would be ignored (m6). The history
text names a handler without a reason cleanly (n3); prepare notes why the
policy is passed decoded (n6); pick() gets first-match-wins unit cases (T8).
… without a second backend

The AC2.6 test now also reads the collection through the current LogsDriver:
while the handler runs the composite text is attempt 0's, the delimiter,
then the handler's lines; after success the same text from the files. Its
err.log variant, written by the root only, ends with the delimiter. AC3.1
pins the canonical id of an exec.exec render, the same with the SDK before
fallbacks (origin/main 314721c) and with this one. AC2.3 also runs on any
backend: a chain longer than the default cap of 3 ends at the cap. AC2.4
gains the differing processWorkdir case; an exec saving a file named
workdir, the key WorkdirSave v3 reserves, works as before.
The pl-compose backend's main root is bind-mounted from the runner and the
container runs as the runner's user, so the tests can delete a working
directory under <main root>/work, as the restart tests do (decision D5).
turbo passes PL_TEST_WORK_STORAGE_ROOT and the optional
PL_ADDRESS_ATTEMPT_CAP_1 through to the tests.
@codecov

codecov Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 83.87097% with 35 lines in your changes missing coverage. Please review.
✅ Project coverage is 61.24%. Comparing base (af78d3a) to head (8f7495b).
⚠️ Report is 62 commits behind head on main.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
lib/node/pl-drivers/src/drivers/logs_composite.ts 88.95% 2 Missing and 16 partials ⚠️
...node/pl-drivers/src/drivers/helpers/logs_handle.ts 67.64% 5 Missing and 6 partials ⚠️
lib/node/pl-drivers/src/drivers/logs.ts 76.47% 1 Missing and 3 partials ⚠️
lib/model/common/src/drivers/log.ts 33.33% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1948      +/-   ##
==========================================
+ Coverage   60.09%   61.24%   +1.14%     
==========================================
  Files         442      445       +3     
  Lines       23461    24350     +889     
  Branches     5482     5774     +292     
==========================================
+ Hits        14099    14912     +813     
+ Misses       7767     7741      -26     
- Partials     1595     1697     +102     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Review 2, finding M1, and the stream error found in the tree dumps. A
template with an error anywhere among its inputs fails as a whole, and so
does whatever reads an output of a render through a future while a sibling
output carries an error. The wd-retrier took the save chain's outputs as
inputs, so one saved file the software never wrote failed every output of
the exec on a WorkdirSave v3 backend, with or without fallbacks.

The wd-retrier now decides on the outcome maps alone, whose entries are
values in the backend's contract (a failure that is not a loss is an entry
with reason Error), and outputs its decision. Each output goes out through
a forwarder of its own that picks the saved output or the restart's, so an
error stays in the output it is in. Without file sets the main save is built
in place rather than in a template, so its outcome is a field of the save
itself. Only WorkdirMissing and FileMissing restart.

A streamed file an attempt never wrote, or did not save, now gets a JSON
stand-in as its final content instead of a missing key or a failed save:
both were errors that spread from the stream manager to every output of the
exec.
The workflow SDK now sets a JSON stand-in as the final content of a stream
whose file the run never wrote; LogsDriver reads it as no log, and the
composite handle as an empty part. A failed attempt whose manager has no
stream at all is final as well (review 2, m7), so a composite handle whose
earlier attempt skipped a streamed file ends ready instead of live.
…tputs

Review 2, m1. The one-entry attempts output of the v2 path was a constant,
resolved before the command ran and on a failure too. It now resolves once
the working directory is saved, and a failed command's error reaches it as
it reaches the other outputs (spec §10.2).
- A handler in a queue without RunCommand v3 no longer fails the render: the
  exec runs without fallbacks, with a notice, as on a backend without v3 (m2).
- fallbacks() documents that handler plans, quota and formulas included, are
  part of the exec's CID (m3); the changeset says not to await getAttempts()
  as a completion signal (m9).
- A file set matching a file named workdir saves it under another key on v3
  and maps it back, since v3 reserves that key (m4).
- No exit code reads "Failed to start" only for a reason that never starts
  (GPUMissed, Unallocated), "Failed without an exit code" otherwise (m5);
  an empty log tail says "No output was saved to logs" as before (m6).
- The history counts attempts, not entries, so restarts do not inflate it (n1).
A saved file never written is that file's error, with and without
fallbacks (M1); a file set with a file named workdir (m4); a messageMatch
handler end to end; a handler without RunCommand v3 dropping the fallbacks
(m2); the reverse err.log variant ending ready (m7); a failed v2 run's
getAttempts() erroring (m1); a FileMissing restart; the job's other outputs
resolving while a handler never writes a streamed file. The AC2.3 test that
needed a second backend is dropped in favour of the chain longer than the
cap on the test backend itself (m8); the restart test deletes only under the
work storage root (n7).
Review 3, M1. A plain exec on a v3 backend rendered exec.exec, exec-impl, the
router, the wd-retrier, six forwarders and a stream template. Per-output
forwarders bought nothing a block could see: every output is read through a
future on the exec.exec render, which an error in any one output marks too
(D12), so they are gone. The wd-retrier takes the save chain's outputs in
one resource the unmarshaller does not open and returns references into
it, writes the stream collections itself, and runs only with WorkdirSave
v3: without it the router returns the save chain's outputs directly, with
the history as attempts (D13 revised). The router and the wd-retrier keep
the attempt's exec-impl inputs opaque (a registered unmarshaller; tpl.light
now starts with an empty unmarshaller list), and exec-impl passes its own
inputs map for a restart instead of building a copy of it.

Templates per exec: plain v3 with WorkdirSave v3: 4 (exec.exec, exec-impl,
router, wd-retrier; +1 with processors); v3 without it: 4 (save.tpl
instead of the wd-retrier); v2: 4 (+v2-attempts). A streamed file written
but not saved ends on the stand-in: a manager may not reference a live
stream as its final content (m1).
… segment

Review 3, m4 and n5. A manager starts with its live stream and loses it only
when its final content errors, which the router's error does to the last
attempt of a failed job: such a part reads as empty and final, and a last
part ended that way makes the handle ready, not live forever. The composite
handle's path segment is '@composite/': a resource type name never starts
with '@', so no single-log handle can look composite.
AC1.x render the RunCommand v3 with the test shell software, which has a
docker image, instead of a plain command, which a Kubernetes executor cannot
run (CI: 'docker is not set'). AC3.1 reads the exec.exec render's canonical
id once its inputs are locked, without running the exec: the k8s failure was
the same docker error, not a different id. A saved file never written is
pinned on v2 and v3, with and without fallbacks and file sets (T1, T2);
nested handlers succeeding two levels deep (T6); a streamed file not saved
ending on the stand-in (m1).
Review 4, m1. A manager has no stream until its CreateStream has the working
directory, which can take minutes while a large workdir fills, and the
composite handle read that as a failed job and went ready. Only an attempt
whose manager's final content errored (the router's error reaches it there)
or that has an outcome counts as ended now; one that is still starting keeps
the handle live.
…etContent

- RunCommand v3 without WorkdirSave v3: getAttempts() resolves once the save
  is done and errors with it, through the v2 template now given the history
  (review 4, m2; spec §12.4).
- A failed attempt's streamed file that fails to save (a lost directory, a
  path that is not a regular file) failed the successful fallback's outputs:
  its error sat among the router's outputs. With WorkdirSave v3,
  exec.failed-streams now decides on that save's outcomes and saves again
  only what saved, so such a file ends on the stand-in and loses that log
  only (review 4, m3; spec §14.3).
- workdir.save().saveFileSetContent(name, regex) stored the regex as a set
  member, so every use failed in the save template; it is keyed by name now
  (review 4, m6; on origin/main since 2024).
A failed save failing getAttempts() with and without WorkdirSave v3 (m2); a
failed attempt's unsaveable stream not failing the fallback (m3);
saveFileSetContent (m6); a handler whose RAM is a resource formula; the CID
of an exec with fallbacks, pinned.
… logs hardened

- On success the collections' stream ends on the winner's final content for
  every streamed name, the stand-in when the winner never wrote the file, not
  on the winner's live tail, which an old driver read as live forever (found
  in the tree dumps).
- exec.failed-streams reads the first save's outcome map entry by entry and
  saves again with WorkdirSave v3, so a directory lost between the two saves
  leaves blobs unset instead of errored (review 5, m1). The second save stays:
  the first save's blobs, which may hold the error, cannot be a template's
  inputs without failing the template as a whole.
- The history's backend cap is the one the router applied when the outcome
  lacks it (n3); doc comments say what the code does (n1, n5, n6).
Review 5, n2 and n6: the StreamCollection and newCompositeLogHandle comments
sat on the wrong declarations; isNotWrittenStandIn lives beside the
composite reader, which uses it too.
The unsaveable failed-attempt stream test reads the WorkdirSave v3
capability explicitly instead of inferring it from the failure (m2), and
asserts that err.log's collection stream and the root's manager end on the
stand-in and the composite log is the delimiter only, ready (T1). A
handler's RAM changes the CID of an exec with fallbacks (T7).
CI runs the tests against the pl main image, which has RunCommand v3 but
not WorkdirSave v3: there the v3 path saves with WorkdirSave v1 and a
never-written file reads as on the v2 path, so the partial-save pins now
read the capability. On Kubernetes the log tail is the job script's own
lines, not the process output, so the messageMatch and log-tail limit
tests are skipped there; nested handlers get a longer timeout for the
slower cluster.
Without fallbacks and without WorkdirSave v3 the outcome router has
nothing to do: no fallback can run and a lost working directory cannot
be restarted. It still added renders between every command and its save,
and on a loaded backend (CI runs against pl main, which has RunCommand v3
but not WorkdirSave v3) that template load made an existing pframes test
time out. Such an exec now renders as before exec fallbacks; its error
carries no attempt history, so the test that pins the history reads the
capability.
> let's finally verify behavior:
> - on workdir save backend verifies directory is in place, if not, retries
> - on every file index backend checks if workdir is in place, if no retries, if yes then it's software/block/other problem not retries
> For me it looks like we don't need ListedInWorkdir flag

The backend (pl#2332) no longer reports FileMissing: a WorkdirSave v3 entry is WorkdirMissing
when the working directory is gone and Error otherwise. The wd-retrier now restarts on
WorkdirMissing only; a saved file gone from a directory that is still there fails the job with
the file's error, as with WorkdirSave v1. The restart test for that case now expects the error
and no restart, and Test_lostWorkdir checks that an Error entry restarts nothing.
> push, check both PRs ci is green

platforma#1948 conflicted with main, so its CI did not run.

# Conflicts:
#	sdk/workflow-tengo/src/exec/runcmd.lib.tengo
…imeout

> push, check both PRs ci is green

On Kubernetes each attempt is a pod, and this test runs two (the failing attempt and the
handler). It had vitest's default 15 s and timed out there; every other two-attempt fallback test
already has 120 s.
> for old backends... in case of OOM on old backend get biggest OOM case
> branch from fallbacks.

A backend without RunCommand v3 sees no failure, so fallbacks never run there.
Instead of ignoring the policy, the builder now runs the OOM plan with the most
RAM (any depth of OOM handlers; a formula compares by its static fallback; a
tie goes deeper) in place of the root. Without OOM handlers the root runs, as
before. getAttempts() names the plan that ran.
> yes, check why CI failed, ensure it became green

main made canonicalId part of the resource data pl-tree applies; the test's
hand-built resources lacked it, so three tests failed on tree update.
> yes, check why CI failed, ensure it became green

main links every RunCommand to the template's action context. The RunCommand
v3 schema on pl (main, v5.0.0) has no optional actionContext field, so the
backend refused the field and every v3 exec failed to render. Linked on v2
only until pl adds the field to the v3 schema.

The k8s job hit a heap OOM because ml-v3's read-logs test loops while the
error stays stable; it passes again with this fix.
@blackcat
blackcat marked this pull request as ready for review November 13, 2026 10:06
@greptile-apps

greptile-apps Bot commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

Automatic reviews are disabled for this repository. Configure in Code Review Settings.

@blackcat
blackcat enabled auto-merge November 13, 2026 11:44

shellSw := assets.importSoftware("@platforma-open/milaboratories.software-test-utils:shell")

self.body(func(inputs) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i dont get this function

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a template or boilerplate for other utility commands. Instead of writing ok and root execs fully from start to end we put common part in a function.
ok and root are used in different tests.

@blackcat
blackcat added this pull request to the merge queue Nov 13, 2026
@blackcat
blackcat removed this pull request from the merge queue due to a manual request Nov 13, 2026

@DenKoren DenKoren left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes because of three points: mandatory CPU and RAM requests (index.lib.tengo:447), negative and zero resource requests (prepare.lib.tengo:76), and the GPU fallback on backends without RunCommand v3 (index.lib.tengo:2008).

The inline comments cover the SDK code. The points below are about the backend part (milaboratory/pl#2332, already merged) and the PR description.

Backend (pl#2332)

  1. On Kubernetes the fail reason is computed inside telemetry code. collectRunTelemetry runs describeRunRecord under guardTelemetry, which turns a panic into one log line (controllers/runner/internal/drivers/k8s/run_record.go:33-41). If that code panics, the outcome has no failDetails, and no fallback handler can match it.
    Question: maybe we need a separate reason for this case? Or is it a regular error (panic), or InfraError?

  2. A command that is not found is a start error (a resource error) with local_exec (controllers/runner/internal/drivers/local_exec/command.go:466-468), but an outcome SoftwareError{exitCode: 127} on Kubernetes. A block with a catch-all SoftwareError handler falls back on a broken package on Kubernetes and fails locally. The drivers should agree.

  3. The run result (RunCommandOutcomeV1.FailDetails, controllers/shared/resources/run_command_outcome_v1.go) is the telemetry struct events.FailDetails. A change made for telemetry changes what blocks receive, and blocks are never rebuilt. This has already happened: Cancelled (MILAB-6926) is set by the Kubernetes telemetry code and reaches the outcome through lastFailDetails. Please add a comment at the struct definition that it is a contract with workflow-tengo.
    Question: is Cancelled listed in workflow-tengo? At least on the SDK-facing side, not the developer-facing one.

PR description

The description says the wd-retrier restarts on WorkdirMissing / FileMissing. There is no FileMissing reason: WorkdirSave v3 reports WorkdirMissing or Error, and only WorkdirMissing restarts (sdk/workflow-tengo/src/exec/save-chain.lib.tengo:22).

}
}

quotaData := {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We better make CPU and RAM requests to be mandatory for all run commands since now.
You provided all possible tools to make this choice wise: telemetry, block status panel with runtime info from commands, formulas to calculate requests based on input file info.

There's just no point to run a command without explicit CPU and RAM requirements. The only reason I can imagine is laziness. Let's fight with it! :)

*
* @param rootPlanned: map - the root's own plan, _planWithoutFallbacks()
*/
_planWithoutRunCommandV3: func(rootPlanned) {

@DenKoren DenKoren Nov 13, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On a backend without RunCommand v3 only OOM handlers are considered. A GPU root with a GPUMissed → CPU fallback runs the GPU plan, and a runner without GPU refuses it ("no GPU is available"). The SDK should check feats.hasGpu and run the GPUMissed plan, or always run the CPU plan.

Note: I prefer to just do 2nd, as we know that 'out of stock' can be a problem.

*/
REASONS := {
SoftwareError: { exitCode: "int", messageMatch: "string" },
OOM: { exitCode: "int" },

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OOM fail reason fallback must not have exit code split. It is an event without standard code in general, though some scenarios could provide it.
Example: on Kubernetes a container killed at once is reported as OOM with no exitCode, so a handler { OOM, exitCode: 137 } misses it. The same applies to the builder doc (index.lib.tengo:1846), the tests (fallback.test.tengo:29, :154; tests/workflow-tengo/src/exec/fallback/old_backend.tpl.tengo:35) and the spec's reason table.

TimeIsUp: {},
WorkdirOutOfSpace: {},
ScratchOutOfSpace: {},
InfraError: {}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

InfraError: what should a block developer do with it? I have no idea, and I believe a block developer has none too. It is not a part of the public API. We can keep it in an inner list of fail reasons that the SDK handles, but not block developers.

}
}
if !is_undefined(memFormula) {
quota.ram = string(execFormula.evaluateResource(memFormula, resolver, "ram")) + "b"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A negative formula result is not visible for us. A zero result fails the same way. The check fails in the template, before any RunCommand exists, so no run record is sent.

Related: a zero request is silently replaced with the queue default on the backend side (def.Val in ctl_notify_compute.go:75-76). A RAM size string is not range-checked (index.lib.tengo:57), so .mem("0GiB"), or a formula's static fallback of "0GiB", runs with the default allocation. This is wrong: a zero request should be an error.

planValues := inputs.fallbackPlans[next]
planRaw := raw.fallbackPlans.getValue().inputs()[next].getValue()
planInputs := maps.merge(planValues.inputs, planValues.meta)
impl := prepare.implInputs(planInputs, planRaw.inputs().inputs.getValue().inputs())

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For a fallback plan the formula is evaluated here, after the previous attempt failed. A non-positive result panics in evaluateResource, and that panic replaces the failed attempt's error text and the attempt history.

next := fallback.pick(inputs.fallbackPolicy, oc.failDetails, oc.lastLogLines)
// the error text is the one the backend's error had before RunCommand v3, plus the history
if is_undefined(next) {
ll.panic("%s", fallback.exhaustedMessage(oc, hist, "no handler", attemptLimit))

@DenKoren DenKoren Nov 13, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: The end user sees → no handler in the attempt history. That is developer jargon: it describes a block setting, not what went wrong. Please use wording that makes sense to the end user.

streamOwn = smart.ephemeralMapBuilder().build()
}

stream.attemptEnded(collections, ctx.attempt, entry, undefined)

@DenKoren DenKoren Nov 13, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Potential bug: on a terminal failure the router panics at :92 or :95 before it reaches this line. So the collection's stream stays on the failed attempt's live stream (set by attemptStarted, stream.lib.tengo:218) for good. Desktop's getLastLogs and getProgressLog* read only stream (lib/node/pl-drivers/src/drivers/logs.ts:46, :232) and keep showing the dead attempt as running.

This may keep the working directory because of stream

_OUTCOME_WORKDIR := "workdir"

// Outcome reasons of WorkdirSave v3 that mean the working directory's content is lost.
_LOST_REASONS := { WorkdirMissing: true }

@DenKoren DenKoren Nov 13, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Either rename WorkdirMissing to Retry, or add Retry for generic retry because of something. Say, pod eviction is 'retry', but not missing workdir.
Today an evicted pod reaches the block as SoftwareError (exit 143 after SIGTERM) or UnknownError, but you plan to add fair pod status watching which will be able to catch this fail reason natively.

for name, f in impl.metaInputs() {
meta[name] = f.getValue()
}
pure.attemptCtx = {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A restart reuses the same attempt and jobId. The Kubernetes run record builds RunID as <jobId>-<attempt>.<subAttempt> (pl controllers/runner/internal/drivers/k8s/run_record.go:132), and subAttempt starts from 0 again in the fresh working directory. So a restart sends the same RunID as the run it replaces, and telemetry deduplication can merge two different runs. The restart index should be part of the identity sent to the backend, or a field of its own.

if is_undefined(next) {
ll.panic("%s", fallback.exhaustedMessage(oc, hist, "no handler", attemptLimit))
}
if ctx.attempt + 1 >= attemptLimit {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: maybe we should change this. An attempt that never started (GPUMissed: the backend refused the request or no GPU node came in time) still counts toward the attempt cap, here and in the backend backstop (pl controllers/runner/internal/runctl/ctl_notify_run.go:629). We did not try anything, did not burn any money, seems safe to just not count.

…d takes it

> add actionContext to the RunCommand v3 and ensure block data will always reach CH

A run record carries its block only when the RunCommand is linked to the block's action
context. pl (MILAB-6913_run-command-v3-action-context) declares the link on the v3 schema and
reports the build flag runCommandV3ActionContext; feats exposes it.

- runcmd links a v3 command when the flag is set (v1/v2 as before).
- exec-impl routes an exec without fallbacks to RunCommand v3 only when the flag is set too:
  on an older backend (pl 5.0.x) it stays on v2, so its block still reaches ClickHouse. The
  lost-workdir restart of such an exec is not worth losing the block. An exec with fallbacks
  needs v3 and runs without the link there.
- Tests: the capability template reports plainExecV3 and the fallback tests branch on it; a new
  action-context test checks that a command with fallbacks reports Running and Completed into
  the block's context on a backend with the flag.
…e flag

> We didn't install 5.0 to anybody. It's staging only. Lets just fix an error

pl 5.0 runs on staging only, so there is no backend to protect: runcmd links every RunCommand,
v3 included, and exec-impl routes as before the flag. The flag (runCommandV3ActionContext), the
plainExecV3 capability and the test branches on it are gone; the action-context test of a command
with fallbacks now always checks the link. RunCommand v3 needs milaboratory/pl#2471.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants