Repository navigation
Conversation
A v3 command finishes a software failure as an outcome instead of an error, which is what exec fallbacks route on. The builder gets v3(attempt, jobId) and the built command exposes the outcome output; limits gets getRunCommandV3Type(queue), the capability check: the type is present in the backend's resource types iff the backend has it.
WorkdirSave v3 reports a lost working directory or file as an outcome, not an error, so the exec's wd-retrier can restart the step after a database restore (decision D2). workdir.save(...).v3() and the save builder's v3 option select it and expose the outcome map; workdir processors save with v3 when the exec's save chain asks for it.
Pure functions shared by the exec builder and the outcome router: policy validation and normalization, plan keys, picking the handler for an outcome, and composing the final error from the structured outcome plus the attempt history (spec §9, §10.4, §12.3).
…), plan() Fallback plans are builders derived from the root, so the builder state moves into one map and copy() clones it. fallbacks() validates and stores the policy; verify() refuses cycles and handlers whose saved, streamed or processed outputs differ from the root's, and run() calls it before any render. plan() is the old run() up to the render inputs, now computed on a clone so it can be called for every handler; with fallbacks on a RunCommand v3 backend it adds the fallbackPlans and fallbackPolicy inputs, otherwise the inputs stay exactly as before (AC3.1). The result gains getAttempts(). _forceRunCommandV2() is a test seam that renders the exec as on a backend without RunCommand v3.
… wd-retrier, stream collections On a backend with RunCommand v3 every exec-impl builds the command as v3 and renders exec.outcome-router on its outcome. On success the router runs the save chain (WorkdirSave v3) and exec.wd-retrier, which hands out the outputs or, when the working directory was lost, renders the same plan again up to the backend's restartLimit (spec §14). On a failure it renders the handler's plan as the next attempt, an ephemeral exec-impl under the root render, prepared by exec.prepare exactly as exec.exec prepares attempt 0; when no handler applies or the backend attempt cap is reached it raises the composed error with the history. With fallbacks every streamed file is a StreamCollection/1 kept by the templates: attempt<K> holds each attempt's manager, outcome<K> its history entry, and stream the running attempt's live tail, then the winner's file, so today's log driver follows the running attempt. The collection's exec input carries the exec.exec inputs map as a stand-in for the render's CID. Without RunCommand v3 exec-impl takes the previous path unchanged, plus the one-entry attempts output, and fallbacks are ignored with a notice.
AC1.1-AC1.3 on RunCommand v3 directly; AC2.1-AC2.5 and a GPUMissed handler on the exec builder; the old log driver's view of a fallback stream and the collection after success (AC2.6, AC2.7); getAttempts() and a bare stream manager without RunCommand v3 (AC3.3, through the _forceRunCommandV2 seam). AC2.3 needs a backend with --runner-max-job-attempts 1 and runs only when PL_ADDRESS_ATTEMPT_CAP_1 is set.
…e restart AC3.1 pins the exec.exec input names of an exec without fallbacks and the values that do not depend on the test software's version; the full dump was compared with the SDK before the feature. The restart tests follow decision D5: a first render runs a command, the test deletes its working directory, a second render runs the save chain and the wd-retrier on it. They need the backend's work storage root in PL_TEST_WORK_STORAGE_ROOT.
A minor release of workflow-tengo: new builder API and behaviour on backends with RunCommand v3.
Decision D3 revised after the backend review: the save side has its own capability key in getResourceTypesFor, so a backend may have RunCommand v3 without WorkdirSave v3. Fallbacks stay gated on runCommandV3; the save chain uses WorkdirSave v3, and the wd-retrier restarts a lost working directory, only when workdirSaveV3 is published. Without it the success path saves with v1 and a lost directory is an error, as before. _forceWorkdirSaveV1() is a test seam for that combination; it holds for the whole job.
…rSave v3 AC2.1 and AC2.6/AC2.7 run twice: as the backend has it, and through the _forceWorkdirSaveV1 seam, a backend with RunCommand v3 but without WorkdirSave v3.
The restart depends on its own capability since D3 was revised.
With .fallbacks() an exec's fileStreams entry is a StreamCollection/1 holding one StreamManager per attempt. Following its `stream` mirror shows only the running attempt, so a failed attempt's log disappears as soon as the fallback starts. LogsDriver.getLogHandle now returns a composite handle for the collection: the attempts' logs in order, a delimiter from each failed attempt's outcome, the last attempt live. readText/lastLines map the viewer's offsets into the joined text, so PlLogView reads it unchanged. The handle keeps the live/ready prefixes so old block models still recognise it; a bare StreamManager keeps today's handle.
🦋 Changeset detectedLatest commit: 8f7495b The changes in this PR will be included in the next version bump. This PR includes changesets to release 23 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Review finding M1. A workdir processor reads the post-run directory through a save of its own before the main save does; on a lost directory that save never hands the directory on, the main WorkdirSave v3 never runs, and the wd-retrier waited for its outcome forever. With processors, a WorkdirSave v3 with no rules now heads the save chain (a directory takes one save at a time, so it is a link of the chain, not a save beside it); the wd-retrier decides on that probe first and, when the directory is there, renders itself again on the main save's outcomes. The save chain's outputs reach the wd-retrier in one resource the input unmarshaller does not descend into, so a processor error or an output that never resolves cannot fail or stall the decision. Processors no longer get a saveVersion input (finding m5): the probe makes v3 saves in processors unnecessary, so processor templates, old or with closed input schemas, are rendered exactly as before. Wave 2 moves into :exec.wave2 so tests render the same code. A failed attempt saves its streamed files in one save again, as a directory takes one save. The router reads an absent attemptLimit as 1 (finding m9).
…llection Review finding M2. A StreamCollection has the identity of its exec.exec render, and a render with the same identity is normally reused as a whole. When it is not (a failed render executed again by error recovery, or a render that is gone while a block's previous outputs still hold its collection), the new job reused the collection and left the earlier job's attempt<K>/outcome<K> behind, so the composite log showed stale attempts and never became final. Attempt 0 of a job now removes them before writing attempt0. exec-impl also lists streamOwn among its outputs (nit n1).
…le collections A lost or kept working directory with a workdir processor that reads a file (M1); a restarted fallback attempt keeping its index and plan and writing attempt<K> of the collection; the restart counter carried across attempts; a job start removing an earlier job's attempts from a collection (M2). The restart tests skip themselves on a backend without WorkdirSave v3.
…parts Review findings m3 and m4. When an earlier attempt's part answers that its handle is outdated (its live stream ended), the composite read now answers the same, so the viewer fetches a new handle instead of seeing that attempt's text vanish from the joined text. The running attempt's part is exempt: its file may not exist yet, which reads the same way and must read as empty (spec §13.1). A composite handle carries at most 256 parts: a client-supplied handle with more is refused, and a collection with more keeps the latest.
getAttempts() documents that a result reused from an exec rendered by an SDK before fallbacks has no attempts (m1). fallbacks() documents that every handler plan's inputs are awaited before attempt 0 starts (m8), and a handler in a queue without RunCommand v3, while the root's has it, is a render-time error: it would run without the router, its stream would never join the collection and its own fallbacks would be ignored (m6). The history text names a handler without a reason cleanly (n3); prepare notes why the policy is passed decoded (n6); pick() gets first-match-wins unit cases (T8).
… without a second backend The AC2.6 test now also reads the collection through the current LogsDriver: while the handler runs the composite text is attempt 0's, the delimiter, then the handler's lines; after success the same text from the files. Its err.log variant, written by the root only, ends with the delimiter. AC3.1 pins the canonical id of an exec.exec render, the same with the SDK before fallbacks (origin/main 314721c) and with this one. AC2.3 also runs on any backend: a chain longer than the default cap of 3 ends at the cap. AC2.4 gains the differing processWorkdir case; an exec saving a file named workdir, the key WorkdirSave v3 reserves, works as before.
The pl-compose backend's main root is bind-mounted from the runner and the container runs as the runner's user, so the tests can delete a working directory under <main root>/work, as the restart tests do (decision D5). turbo passes PL_TEST_WORK_STORAGE_ROOT and the optional PL_ADDRESS_ATTEMPT_CAP_1 through to the tests.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #1948 +/- ##
==========================================
+ Coverage 60.09% 61.24% +1.14%
==========================================
Files 442 445 +3
Lines 23461 24350 +889
Branches 5482 5774 +292
==========================================
+ Hits 14099 14912 +813
+ Misses 7767 7741 -26
- Partials 1595 1697 +102 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Review 2, finding M1, and the stream error found in the tree dumps. A template with an error anywhere among its inputs fails as a whole, and so does whatever reads an output of a render through a future while a sibling output carries an error. The wd-retrier took the save chain's outputs as inputs, so one saved file the software never wrote failed every output of the exec on a WorkdirSave v3 backend, with or without fallbacks. The wd-retrier now decides on the outcome maps alone, whose entries are values in the backend's contract (a failure that is not a loss is an entry with reason Error), and outputs its decision. Each output goes out through a forwarder of its own that picks the saved output or the restart's, so an error stays in the output it is in. Without file sets the main save is built in place rather than in a template, so its outcome is a field of the save itself. Only WorkdirMissing and FileMissing restart. A streamed file an attempt never wrote, or did not save, now gets a JSON stand-in as its final content instead of a missing key or a failed save: both were errors that spread from the stream manager to every output of the exec.
The workflow SDK now sets a JSON stand-in as the final content of a stream whose file the run never wrote; LogsDriver reads it as no log, and the composite handle as an empty part. A failed attempt whose manager has no stream at all is final as well (review 2, m7), so a composite handle whose earlier attempt skipped a streamed file ends ready instead of live.
…tputs Review 2, m1. The one-entry attempts output of the v2 path was a constant, resolved before the command ran and on a failure too. It now resolves once the working directory is saved, and a failed command's error reaches it as it reaches the other outputs (spec §10.2).
- A handler in a queue without RunCommand v3 no longer fails the render: the exec runs without fallbacks, with a notice, as on a backend without v3 (m2). - fallbacks() documents that handler plans, quota and formulas included, are part of the exec's CID (m3); the changeset says not to await getAttempts() as a completion signal (m9). - A file set matching a file named workdir saves it under another key on v3 and maps it back, since v3 reserves that key (m4). - No exit code reads "Failed to start" only for a reason that never starts (GPUMissed, Unallocated), "Failed without an exit code" otherwise (m5); an empty log tail says "No output was saved to logs" as before (m6). - The history counts attempts, not entries, so restarts do not inflate it (n1).
A saved file never written is that file's error, with and without fallbacks (M1); a file set with a file named workdir (m4); a messageMatch handler end to end; a handler without RunCommand v3 dropping the fallbacks (m2); the reverse err.log variant ending ready (m7); a failed v2 run's getAttempts() erroring (m1); a FileMissing restart; the job's other outputs resolving while a handler never writes a streamed file. The AC2.3 test that needed a second backend is dropped in favour of the chain longer than the cap on the test backend itself (m8); the restart test deletes only under the work storage root (n7).
Review 3, M1. A plain exec on a v3 backend rendered exec.exec, exec-impl, the router, the wd-retrier, six forwarders and a stream template. Per-output forwarders bought nothing a block could see: every output is read through a future on the exec.exec render, which an error in any one output marks too (D12), so they are gone. The wd-retrier takes the save chain's outputs in one resource the unmarshaller does not open and returns references into it, writes the stream collections itself, and runs only with WorkdirSave v3: without it the router returns the save chain's outputs directly, with the history as attempts (D13 revised). The router and the wd-retrier keep the attempt's exec-impl inputs opaque (a registered unmarshaller; tpl.light now starts with an empty unmarshaller list), and exec-impl passes its own inputs map for a restart instead of building a copy of it. Templates per exec: plain v3 with WorkdirSave v3: 4 (exec.exec, exec-impl, router, wd-retrier; +1 with processors); v3 without it: 4 (save.tpl instead of the wd-retrier); v2: 4 (+v2-attempts). A streamed file written but not saved ends on the stand-in: a manager may not reference a live stream as its final content (m1).
… segment Review 3, m4 and n5. A manager starts with its live stream and loses it only when its final content errors, which the router's error does to the last attempt of a failed job: such a part reads as empty and final, and a last part ended that way makes the handle ready, not live forever. The composite handle's path segment is '@composite/': a resource type name never starts with '@', so no single-log handle can look composite.
AC1.x render the RunCommand v3 with the test shell software, which has a docker image, instead of a plain command, which a Kubernetes executor cannot run (CI: 'docker is not set'). AC3.1 reads the exec.exec render's canonical id once its inputs are locked, without running the exec: the k8s failure was the same docker error, not a different id. A saved file never written is pinned on v2 and v3, with and without fallbacks and file sets (T1, T2); nested handlers succeeding two levels deep (T6); a streamed file not saved ending on the stand-in (m1).
Review 4, m1. A manager has no stream until its CreateStream has the working directory, which can take minutes while a large workdir fills, and the composite handle read that as a failed job and went ready. Only an attempt whose manager's final content errored (the router's error reaches it there) or that has an outcome counts as ended now; one that is still starting keeps the handle live.
…etContent - RunCommand v3 without WorkdirSave v3: getAttempts() resolves once the save is done and errors with it, through the v2 template now given the history (review 4, m2; spec §12.4). - A failed attempt's streamed file that fails to save (a lost directory, a path that is not a regular file) failed the successful fallback's outputs: its error sat among the router's outputs. With WorkdirSave v3, exec.failed-streams now decides on that save's outcomes and saves again only what saved, so such a file ends on the stand-in and loses that log only (review 4, m3; spec §14.3). - workdir.save().saveFileSetContent(name, regex) stored the regex as a set member, so every use failed in the save template; it is keyed by name now (review 4, m6; on origin/main since 2024).
A failed save failing getAttempts() with and without WorkdirSave v3 (m2); a failed attempt's unsaveable stream not failing the fallback (m3); saveFileSetContent (m6); a handler whose RAM is a resource formula; the CID of an exec with fallbacks, pinned.
… logs hardened - On success the collections' stream ends on the winner's final content for every streamed name, the stand-in when the winner never wrote the file, not on the winner's live tail, which an old driver read as live forever (found in the tree dumps). - exec.failed-streams reads the first save's outcome map entry by entry and saves again with WorkdirSave v3, so a directory lost between the two saves leaves blobs unset instead of errored (review 5, m1). The second save stays: the first save's blobs, which may hold the error, cannot be a template's inputs without failing the template as a whole. - The history's backend cap is the one the router applied when the outcome lacks it (n3); doc comments say what the code does (n1, n5, n6).
Review 5, n2 and n6: the StreamCollection and newCompositeLogHandle comments sat on the wrong declarations; isNotWrittenStandIn lives beside the composite reader, which uses it too.
The unsaveable failed-attempt stream test reads the WorkdirSave v3 capability explicitly instead of inferring it from the failure (m2), and asserts that err.log's collection stream and the root's manager end on the stand-in and the composite log is the delimiter only, ready (T1). A handler's RAM changes the CID of an exec with fallbacks (T7).
CI runs the tests against the pl main image, which has RunCommand v3 but not WorkdirSave v3: there the v3 path saves with WorkdirSave v1 and a never-written file reads as on the v2 path, so the partial-save pins now read the capability. On Kubernetes the log tail is the job script's own lines, not the process output, so the messageMatch and log-tail limit tests are skipped there; nested handlers get a longer timeout for the slower cluster.
Without fallbacks and without WorkdirSave v3 the outcome router has nothing to do: no fallback can run and a lost working directory cannot be restarted. It still added renders between every command and its save, and on a loaded backend (CI runs against pl main, which has RunCommand v3 but not WorkdirSave v3) that template load made an existing pframes test time out. Such an exec now renders as before exec fallbacks; its error carries no attempt history, so the test that pins the history reads the capability.
34bcbe5 to
2e0209c
Compare
> let's finally verify behavior: > - on workdir save backend verifies directory is in place, if not, retries > - on every file index backend checks if workdir is in place, if no retries, if yes then it's software/block/other problem not retries > For me it looks like we don't need ListedInWorkdir flag The backend (pl#2332) no longer reports FileMissing: a WorkdirSave v3 entry is WorkdirMissing when the working directory is gone and Error otherwise. The wd-retrier now restarts on WorkdirMissing only; a saved file gone from a directory that is still there fails the job with the file's error, as with WorkdirSave v1. The restart test for that case now expects the error and no restart, and Test_lostWorkdir checks that an Error entry restarts nothing.
> push, check both PRs ci is green platforma#1948 conflicted with main, so its CI did not run. # Conflicts: # sdk/workflow-tengo/src/exec/runcmd.lib.tengo
…imeout > push, check both PRs ci is green On Kubernetes each attempt is a pod, and this test runs two (the failing attempt and the handler). It had vitest's default 15 s and timed out there; every other two-attempt fallback test already has 120 s.
> for old backends... in case of OOM on old backend get biggest OOM case > branch from fallbacks. A backend without RunCommand v3 sees no failure, so fallbacks never run there. Instead of ignoring the policy, the builder now runs the OOM plan with the most RAM (any depth of OOM handlers; a formula compares by its static fallback; a tie goes deeper) in place of the root. Without OOM handlers the root runs, as before. getAttempts() names the plan that ran.
> yes, check why CI failed, ensure it became green main made canonicalId part of the resource data pl-tree applies; the test's hand-built resources lacked it, so three tests failed on tree update.
> yes, check why CI failed, ensure it became green main links every RunCommand to the template's action context. The RunCommand v3 schema on pl (main, v5.0.0) has no optional actionContext field, so the backend refused the field and every v3 exec failed to render. Linked on v2 only until pl adds the field to the v3 schema. The k8s job hit a heap OOM because ml-v3's read-logs test loops while the error stays stable; it passes again with this fix.
|
Automatic reviews are disabled for this repository. Configure in Code Review Settings. |
|
|
||
| shellSw := assets.importSoftware("@platforma-open/milaboratories.software-test-utils:shell") | ||
|
|
||
| self.body(func(inputs) { |
There was a problem hiding this comment.
It's a template or boilerplate for other utility commands. Instead of writing ok and root execs fully from start to end we put common part in a function.
ok and root are used in different tests.
DenKoren
left a comment
There was a problem hiding this comment.
Requesting changes because of three points: mandatory CPU and RAM requests (index.lib.tengo:447), negative and zero resource requests (prepare.lib.tengo:76), and the GPU fallback on backends without RunCommand v3 (index.lib.tengo:2008).
The inline comments cover the SDK code. The points below are about the backend part (milaboratory/pl#2332, already merged) and the PR description.
Backend (pl#2332)
-
On Kubernetes the fail reason is computed inside telemetry code.
collectRunTelemetryrunsdescribeRunRecordunderguardTelemetry, which turns a panic into one log line (controllers/runner/internal/drivers/k8s/run_record.go:33-41). If that code panics, the outcome has nofailDetails, and no fallback handler can match it.
Question: maybe we need a separate reason for this case? Or is it a regular error (panic), orInfraError? -
A command that is not found is a start error (a resource error) with
local_exec(controllers/runner/internal/drivers/local_exec/command.go:466-468), but an outcomeSoftwareError{exitCode: 127}on Kubernetes. A block with a catch-allSoftwareErrorhandler falls back on a broken package on Kubernetes and fails locally. The drivers should agree. -
The run result (
RunCommandOutcomeV1.FailDetails,controllers/shared/resources/run_command_outcome_v1.go) is the telemetry structevents.FailDetails. A change made for telemetry changes what blocks receive, and blocks are never rebuilt. This has already happened:Cancelled(MILAB-6926) is set by the Kubernetes telemetry code and reaches the outcome throughlastFailDetails. Please add a comment at the struct definition that it is a contract with workflow-tengo.
Question: isCancelledlisted in workflow-tengo? At least on the SDK-facing side, not the developer-facing one.
PR description
The description says the wd-retrier restarts on WorkdirMissing / FileMissing. There is no FileMissing reason: WorkdirSave v3 reports WorkdirMissing or Error, and only WorkdirMissing restarts (sdk/workflow-tengo/src/exec/save-chain.lib.tengo:22).
| } | ||
| } | ||
|
|
||
| quotaData := { |
There was a problem hiding this comment.
We better make CPU and RAM requests to be mandatory for all run commands since now.
You provided all possible tools to make this choice wise: telemetry, block status panel with runtime info from commands, formulas to calculate requests based on input file info.
There's just no point to run a command without explicit CPU and RAM requirements. The only reason I can imagine is laziness. Let's fight with it! :)
| * | ||
| * @param rootPlanned: map - the root's own plan, _planWithoutFallbacks() | ||
| */ | ||
| _planWithoutRunCommandV3: func(rootPlanned) { |
There was a problem hiding this comment.
On a backend without RunCommand v3 only OOM handlers are considered. A GPU root with a GPUMissed → CPU fallback runs the GPU plan, and a runner without GPU refuses it ("no GPU is available"). The SDK should check feats.hasGpu and run the GPUMissed plan, or always run the CPU plan.
Note: I prefer to just do 2nd, as we know that 'out of stock' can be a problem.
| */ | ||
| REASONS := { | ||
| SoftwareError: { exitCode: "int", messageMatch: "string" }, | ||
| OOM: { exitCode: "int" }, |
There was a problem hiding this comment.
OOM fail reason fallback must not have exit code split. It is an event without standard code in general, though some scenarios could provide it.
Example: on Kubernetes a container killed at once is reported as OOM with no exitCode, so a handler { OOM, exitCode: 137 } misses it. The same applies to the builder doc (index.lib.tengo:1846), the tests (fallback.test.tengo:29, :154; tests/workflow-tengo/src/exec/fallback/old_backend.tpl.tengo:35) and the spec's reason table.
| TimeIsUp: {}, | ||
| WorkdirOutOfSpace: {}, | ||
| ScratchOutOfSpace: {}, | ||
| InfraError: {} |
There was a problem hiding this comment.
InfraError: what should a block developer do with it? I have no idea, and I believe a block developer has none too. It is not a part of the public API. We can keep it in an inner list of fail reasons that the SDK handles, but not block developers.
| } | ||
| } | ||
| if !is_undefined(memFormula) { | ||
| quota.ram = string(execFormula.evaluateResource(memFormula, resolver, "ram")) + "b" |
There was a problem hiding this comment.
A negative formula result is not visible for us. A zero result fails the same way. The check fails in the template, before any RunCommand exists, so no run record is sent.
Related: a zero request is silently replaced with the queue default on the backend side (def.Val in ctl_notify_compute.go:75-76). A RAM size string is not range-checked (index.lib.tengo:57), so .mem("0GiB"), or a formula's static fallback of "0GiB", runs with the default allocation. This is wrong: a zero request should be an error.
| planValues := inputs.fallbackPlans[next] | ||
| planRaw := raw.fallbackPlans.getValue().inputs()[next].getValue() | ||
| planInputs := maps.merge(planValues.inputs, planValues.meta) | ||
| impl := prepare.implInputs(planInputs, planRaw.inputs().inputs.getValue().inputs()) |
There was a problem hiding this comment.
For a fallback plan the formula is evaluated here, after the previous attempt failed. A non-positive result panics in evaluateResource, and that panic replaces the failed attempt's error text and the attempt history.
| next := fallback.pick(inputs.fallbackPolicy, oc.failDetails, oc.lastLogLines) | ||
| // the error text is the one the backend's error had before RunCommand v3, plus the history | ||
| if is_undefined(next) { | ||
| ll.panic("%s", fallback.exhaustedMessage(oc, hist, "no handler", attemptLimit)) |
There was a problem hiding this comment.
nit: The end user sees → no handler in the attempt history. That is developer jargon: it describes a block setting, not what went wrong. Please use wording that makes sense to the end user.
| streamOwn = smart.ephemeralMapBuilder().build() | ||
| } | ||
|
|
||
| stream.attemptEnded(collections, ctx.attempt, entry, undefined) |
There was a problem hiding this comment.
Potential bug: on a terminal failure the router panics at :92 or :95 before it reaches this line. So the collection's stream stays on the failed attempt's live stream (set by attemptStarted, stream.lib.tengo:218) for good. Desktop's getLastLogs and getProgressLog* read only stream (lib/node/pl-drivers/src/drivers/logs.ts:46, :232) and keep showing the dead attempt as running.
This may keep the working directory because of stream
| _OUTCOME_WORKDIR := "workdir" | ||
|
|
||
| // Outcome reasons of WorkdirSave v3 that mean the working directory's content is lost. | ||
| _LOST_REASONS := { WorkdirMissing: true } |
There was a problem hiding this comment.
Either rename WorkdirMissing to Retry, or add Retry for generic retry because of something. Say, pod eviction is 'retry', but not missing workdir.
Today an evicted pod reaches the block as SoftwareError (exit 143 after SIGTERM) or UnknownError, but you plan to add fair pod status watching which will be able to catch this fail reason natively.
| for name, f in impl.metaInputs() { | ||
| meta[name] = f.getValue() | ||
| } | ||
| pure.attemptCtx = { |
There was a problem hiding this comment.
A restart reuses the same attempt and jobId. The Kubernetes run record builds RunID as <jobId>-<attempt>.<subAttempt> (pl controllers/runner/internal/drivers/k8s/run_record.go:132), and subAttempt starts from 0 again in the fresh working directory. So a restart sends the same RunID as the run it replaces, and telemetry deduplication can merge two different runs. The restart index should be part of the identity sent to the backend, or a field of its own.
| if is_undefined(next) { | ||
| ll.panic("%s", fallback.exhaustedMessage(oc, hist, "no handler", attemptLimit)) | ||
| } | ||
| if ctx.attempt + 1 >= attemptLimit { |
There was a problem hiding this comment.
nit: maybe we should change this. An attempt that never started (GPUMissed: the backend refused the request or no GPU node came in time) still counts toward the attempt cap, here and in the backend backstop (pl controllers/runner/internal/runctl/ctl_notify_run.go:629). We did not try anything, did not burn any money, seems safe to just not count.
…d takes it > add actionContext to the RunCommand v3 and ensure block data will always reach CH A run record carries its block only when the RunCommand is linked to the block's action context. pl (MILAB-6913_run-command-v3-action-context) declares the link on the v3 schema and reports the build flag runCommandV3ActionContext; feats exposes it. - runcmd links a v3 command when the flag is set (v1/v2 as before). - exec-impl routes an exec without fallbacks to RunCommand v3 only when the flag is set too: on an older backend (pl 5.0.x) it stays on v2, so its block still reaches ClickHouse. The lost-workdir restart of such an exec is not worth losing the block. An exec with fallbacks needs v3 and runs without the link there. - Tests: the capability template reports plainExecV3 and the fallback tests branch on it; a new action-context test checks that a command with fallbacks reports Running and Completed into the block's context on a backend with the flag.
…e flag > We didn't install 5.0 to anybody. It's staging only. Lets just fix an error pl 5.0 runs on staging only, so there is no backend to protect: runcmd links every RunCommand, v3 included, and exec-impl routes as before the flag. The flag (runCommandV3ActionContext), the plainExecV3 capability and the test branches on it are gone; the action-context test of a command with fallbacks now always checks the link. RunCommand v3 needs milaboratory/pl#2471.
SDK part of exec fallbacks (spec; implementation decisions). Backend: milaboratory/pl#2332 (
WorkdirSavev3, restarts) on top of #2288 (RunCommandv3, already on main).What changes
exec.builder():copy()fallbacks(policy), keyed by failure reason, withexitCode,messageMatch,backendHasGPUandtimeoutattributesverify(), which checks for cycles and that every handler declares the same outputs as the rootplan()getAttempts()..fallbacks()renders byte-identicalexec.execinputs; its CID is pinned by a test.RunCommandv3 backend goes through the outcome router. A software failure is an outcome; the router picks a handler or raises today's error text plus the attempt history. Each fallback attempt is a fullexec-implrender inside the root render, from a fresh workdir, within the backend's attempt cap.workdirSaveV3, the wd-retrier reads theWorkdirSavev3 outcome map and runs the same plan again onWorkdirMissing/FileMissing, up to the backend'srestartLimit. With workdir processors, a probe save heads the chain so a lost directory restarts instead of stalling..fallbacks(),fileStreams[name]is a template-managedStreamCollection/1holdingattempt<K>managers,outcome<K>entries and astreammirror for old desktops. Blocks need no change.getLogHandlereturns a composite handle for a collection: every attempt's text, a short delimiter after each failed one, the running attempt live. It is encoded as an ordinary live/ready log handle, so old models and uikitPlLogViewaccept it..fallbacks()is ignored with a notice.streammirror.workdir.save().saveFileSetContentnever keyed the content by name.Accepted limitations are listed in the decisions file (D12, D14), e.g. on a v3 backend a never-written
saveFilemakes all of the exec's outputs read as errors (on v1 some resolved by timing).Most critical files
sdk/workflow-tengo/src/exec/index.lib.tengo: builder (fallbacks,verify,copy,plan,getAttempts)sdk/workflow-tengo/src/exec/outcome-router.tpl.tengo: the decision per attemptsdk/workflow-tengo/src/exec/wd-retrier.tpl.tengo,wave2.lib.tengo,save-chain.lib.tengo: save and restartsdk/workflow-tengo/src/exec/fallback.lib.tengo: policy validation, matching, error textsdk/workflow-tengo/src/exec/exec.tpl.tengo,exec-impl.tpl.tengo,prepare.lib.tengo: attempt renderingsdk/workflow-tengo/src/exec/stream.lib.tengo,failed-streams.tpl.tengo: stream collections and failed-attempt logslib/node/pl-drivers/src/drivers/logs.ts,logs_composite.ts,helpers/logs_handle.ts;lib/model/common/src/drivers/log.ts: composite log handleNew resource structures
Dumped from this SDK on the backend of milaboratory/pl#2332; frames mark the resources the feature adds.
Plain exec on a v3 backend (no fallbacks)

Fallback: attempt 0 failed, attempt 1 running

Fallback: job finished

Restart after a lost working directory

A failed attempt's stream that cannot be saved

Tests
tests/workflow-tengo/src/exec/covers:RunCommandv3messageMatch; nested handlerserr.logvariants; AC2.7workdirSaveV3is on pl main.