TS engine extensions (epic integration branch)#416
Draft
gordonwoodhull wants to merge 10 commits into
Draft
Conversation
…pi foundation, TsEngineHost, EngineRegistry Arc migration Restructure the engine-extension epic: split Plan 1a into protocol/host/engine sub-plans, extract Plan 2A, migrate the EngineRegistry definition into plan1a-engine, and rename `quarto build-ts-extension` -> `q2 build-ts-extension`. Engine-A foundation: - Publish @quarto/api + @quarto/types skeletons: config key-lists, faithful metadata partition, and the platform/text/format/crypto/ mappedString/markdownRegex/console/path/system namespaces. - Add the TS engine-host JSON protocol types; remove the vestigial FormatIdentifier::Custom variant. - Land quarto_runtime_dir(), ExecutionError::Timeout/ProcessCrashed, and the TsEngineHost transport split + demux. - plan1a-engine Phases 3/3.5/4: trait-discovery surface, resolve_engines + AST language scan + enforcement, and the EngineRegistry -> Arc migration + two-step TsEngine lifecycle. (This is the Arc/field-type half of the pipeline.rs merge conflict.)
…ion + RTQ wire surfaces Q1 engine-lifecycle ground-truth research; plan1a/1b reconciliation (embedding model, concurrency, return-based deps); the RTQ course-correction split (host-bugs plan + Plan 5 pooling stub); the engine surface-coverage audit and design questions. Code (RTQ items): - item-A: split EngineHostContext into Init/HostGlobalConfig + LaunchEngine/EngineProjectContext. - eng-1: complete the discovery tier (generates_figures, can_freeze, quarto_required on LoadEngineResult). - fc-1: carry metadata/pandoc/resourceFiles/preserve + postProcess on the execute-result wire. - fc-2: deferred-dependencies wire surface (Dependencies verb).
…ineExecutionStage + resolve_engines + EngineClaimsFileStage The engine-host-deno package: newline-framed JSON transport, MappedString rehydration, metadata-as-format partition, the Deno PlatformHost, buildQuartoAPI + engine-loader, the run-loop, execute-dispatch, a per-engine execute queue with cooperative Cancel + poison/transparent relaunch, and the committed esbuild bundle + xtask. MappedString unification into @quarto/types (Plan 1b.1). Plan 1c Tasks 1-14: - engine-contribution types + static-claim conversion; parse contributes.engines with validation/warnings. - TsEngineHost observability; TsEngine static claims + execute-time validation; quarto_data_dir / is_interactive / running_in_ci. - registry contribution_order + shutdown_all; build the real engine registry on ProjectContext. - Task 8: thread the project registry onto StageContext; make EngineExecutionStage stateless. - Task 9: resolve_engines drives execution (claimed short-circuit, contribution_order); delete KNOWN_ENGINES. - Task 10: EngineClaimsFileStage + LoadedSource.conversion + C'. - Task 11: P2-12/P2-13 failure model; Task 12: q2 build-ts-extension CLI; Tasks 13/14: echo/echo-legacy fixtures + E2E render gate. (This is the structural half of the pipeline.rs conflict: builder-signature removal, engine-stage reconstruction, EngineClaimsFileStage insertion, pipeline-length bump.)
… Julia & marimo engine validation, preview-capture fixes Plan 2 (Phase A/B): deferred launch-context bodies + global seam, execProcess restore (mergeOutput/stderrFilter/timeout; stdin is a mode), LanguageClaim type + claim constructors, and derived QuartoAPI conformance (retire the harness cast). Plan 3: full @quarto/api/jupyter port (13 tasks) — MIME dispatch, tags, labels, cell-options, preserve, widgets, to-markdown, percent-script, assets, makeJupyter; wire makeJupyter into buildQuartoAPI (cast-free). Decision: rename claims-files -> claims-extensions. Plan 4 (Julia, Phases 4A-4J): minimal render, figures, multi-cell/error handling, engine-file directory expansion, symlink-cycle hardening, frozen test seams; plus preview validation. Plan 4c (marimo, Phases 4cA-4cH): fixture, minimal render, sql-interop, the bare-sql ownership-gate fix (handledLanguages is leave-alone), and SC8-SC21 seam tests. Preview-capture (bd-h4rhohhy): Bug A/B/C repro + fixes — realistic .cell wrappers, bound splice, reader stray-line escalation bound, error-path coverage. Widen static engine claims to Vec-per-language.
…aims-extensions Plan 9 (q2 call engine) and Plan 10 (checkInstallation / q2 check): Q1 spec extraction, implementation research, and frozen-test-seam plans. The engine:/engines: two-key grammar design contract. Plan 6 (Pass-1 engine resolution) plan build + review rounds. Fix registry BUILTIN_ORDER to match the resolver. Code (plan1c.2 P2/P4): - typed FileClaim + parse-time extension normalization + to_wire_ext. - RenderableExtensions seam + claimed_file_extensions; discover statically-claimed engine extensions in projects. - pub(crate) contribution_order getter; real-process crash-path E2E (ProcessCrashed with stderr).
…ests, marimo preview splice Plan 4b/4d: case-4 becomes engine self-enforcement; owned_languages positive-ownership wire field. Plan 4b Phases A-F: - synthetic resolution-shaped + behave fixtures; resolution-tier matrix (pure-unit rows + real-Deno rows). - splice project _quarto.yml engines: into ordering (Phase C). - jupyter ANSI-strip binding + widget E2E (Phase E). - cooperative-cancel poison, crash-triggered relaunch (resettable transport, generation guard), Dependencies-verb round-trip (Phase F). Marimo q2 preview capture-splice fix (splice unwrapped engine islands, bd-5jxcio5d) + engine include-in-header delivery so widgets hydrate (bd-5oyk1xce). Promote plan1a.6 (off-stdout -> loopback TCP) to a plan of record. Relax over-required _extension.yml fields (title/author) to match Q1.
…ofile
Plan 6 Phases 1-5:
- claim-schema list shorthand + pub(crate) claim parsers.
- resolve_engines_inner consults generated-languages; whole-table claim
replacement; load-free predicate + no-load claim surface.
- Stamp Pass-1 engine resolution onto DocumentProfile (the
engine_resolution field + DOCUMENT_PROFILE_VERSION bump).
- Thread _extension.yml provenance into the engine-extension cache key;
add the Pass-1 counter + fall-through warning.
- Accept bare kind-string claim map values ({r: primary}); user-facing
docs for the engine:/engines: keys.
(This bucket owns the document_profile.rs version-bump conflict.)
…d-ts-extension' + fixture cleanup plan1a.6 (off-stdout -> loopback TCP), Phases 1-4: - TcpTransport newline-JSON framing, accept_and_handshake, spawn_into_tcp, shutdown + crash-reap, large-payload deadlock-freedom. - Deno dial-back: connectControl + main.ts selection. - Flip the production engine-host to loopback TCP; malformed control-socket frame is fatal. - Delete the stdio transport; retire the "stdout-is-the-protocol" / console.log-corrupts contract. plan1c3: rename `q2 build-ts-extension` -> `q2 call build-ts-extension`, extract the extension-bundle build lib (CLI becomes a wrapper), and hermetic self-regenerating synth fixtures (regenerate at test time). Capture-splice fixes: keep the earlier engine's cell output in multi-engine folds (bd-5oyk1xce Bug B); splice an echo:true cell's full output run. Julia fixture cleanup: drop Plots.jl, remove the orphan plot.qmd trap, surface swallowed render errors.
… main Mechanical compile/test fixups after rebasing the 8 engine commits onto main. These live in main-authored files that no branch commit owns, so they can only be applied as a post-rebase sweep (kept separate to preserve the 8 rebased commits intact): - Arc<EngineRegistry> migration call-sites main added since the merge-base: quarto-hub-provider Provider.registry field + new() param + 3 test call-sites; preview_record test. - EngineCapture.files (#410) on branch-side literals in capture_splice tests. - ProjectContext new fields (binary_dependencies/extensions/registry/…): ..Default::default() on test literals (pipeline, date_normalize, mermaid, language_pipeline, printable_render, video_shortcode_preview). - CodeBlock.code_start (#412) in the jupyter text_execute test helper. - Adapt main's #410 FigureWritingTestEngine to the branch's claims-based engine-resolution model (implement claims_language) so capture_embeds_engine_supporting_file_bytes exercises the engine again. - Drop stale `pub mod ts_process_framing_probe;` (file deleted by plan1c3 fixture cleanup; kept title_block_pipeline + video_shortcode_preview, which are main modules whose files exist). - Regenerated: package-lock.json (npm install), .braid/snapshot.jsonl.
…tone Plan 8 Grand plan (2026-04-16-ts-engine-extensions-subprocess.md): - Mark Plans 4b, 6, 1a.6 complete; add missing Plan 1c3 row (complete). - Tombstone Plan 8 row; refresh totals + status-key legend (add tombstoned). Plan 8: TOMBSTONED — diagrams are not engines. Mermaid is now an AST transform on main (transforms/mermaid.rs), HANDLED_LANGUAGES is gone, and 2026-07-20-mermaid-regular-rendering.md (bd-5m4ga0s1) pivots mermaid off the engine model. Both Part A/B depended on that premise. Banner + status added; original content preserved. Plan 6: check off the user-facing-docs box — the docs landed in the Plan 6 commit itself (docs/guides/authoring/engines.qmd) but the box was missed. Plan 5: add a Plan 1a.6 (loopback-TCP) interaction note (vindicates the EngineTransport seam; re-target the measure-first gate to include the TCP handshake; crash path must use TCP liveness not socket-EOF; kernel-survival premise unchanged) and fix the stale StdioTransport -> TcpTransport reference.
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Integration branch for the TypeScript engine-extensions epic. Draft — not ready to merge; open mainly to exercise CI.
Grand plan:
claude-notes/plans/2026-04-16-ts-engine-extensions-subprocess.md. Latest commit reconciles plan status (marks completions, tombstones Plan 8 — diagrams are not engines).