From a53a2b626765f6568579f160b0cedb5c3f781acb Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 19 Jul 2026 06:08:09 +0000 Subject: [PATCH 01/37] docs: fix bootstrap prompt code fence rendering The prompt block used a triple-backtick ```text fence containing a nested ```bash fence. mdBook's parser treated the inner closing ``` as closing the outer block, so everything after the curl example rendered as prose instead of inside the prompt code block. Switch the outer fence to four backticks (the pattern already used elsewhere in the site) so the nested fence is literal. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- docs-site/src/tools/bootstrap-prompt.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs-site/src/tools/bootstrap-prompt.md b/docs-site/src/tools/bootstrap-prompt.md index 6c0331741..49e573c63 100644 --- a/docs-site/src/tools/bootstrap-prompt.md +++ b/docs-site/src/tools/bootstrap-prompt.md @@ -12,7 +12,7 @@ can seed the model from a design prototype in the same session — nothing to ma ## The prompt -```text +````text This is an empty repo. Provision it as a Mendix app developed with mxcli: 1. Ensure `mxcli` is available. It should be pre-installed by the environment; if @@ -43,7 +43,7 @@ This is an empty repo. Provision it as a Mendix app developed with mxcli: the app answers HTTP 200 at http://localhost:8080/ and report. (Optional) Seed the domain model, pages, and microflows from this prototype: . -``` +```` ## Which mxcli version gets installed From e9efef075dbb4a89ed8b9ab358b0f246d24149da Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 19 Jul 2026 11:52:34 +0000 Subject: [PATCH 02/37] docs(skills): stop recommending destructive bare `mx update-widgets` MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CE0463 remediation in the synced skills told agents to run `mx update-widgets` directly. On an MPRv2 project (an `mprcontents/`-folder project — everything `mxcli new` creates) that command silently rewrites the project to single-file v1 and deletes `mprcontents/`, corrupting a git working tree, breaking a running `mxcli run --local` loop, and sometimes leaving Studio Pro unable to open the project. This actually bit a warm-loop app-generation run (CE0463 on Image widgets → agent ran `mx update-widgets` per the skill). With PR #764 (issue #763) merged, `mxcli docker check`/`build` snapshot and restore the v2 storage around the widget normalization, so they are the safe remediation. Point every CE0463/update-widgets mention at those commands and warn that bare `mx update-widgets` is only safe on a v1 project or a throwaway diagnostic copy. - create-page.md, custom-widgets.md, migrate-design-prototype.md, download-marketplace-content.md (user-synced): steer to docker check/build, warn about the v2->v1 deletion. - debug-bson.md (dev): fix the "safety net"/"quick workaround" framing; add a "use a COPY" caution to the reference-extraction blocks. - fix-issue.md: add a symptom-table row for the mprcontents/ data-loss class (issue #763 / PR #764). Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- .claude/skills/debug-bson.md | 14 ++++++++------ .claude/skills/fix-issue.md | 1 + .claude/skills/mendix/create-page.md | 13 ++++++++++--- .claude/skills/mendix/custom-widgets.md | 14 ++++++++++---- .../skills/mendix/download-marketplace-content.md | 2 +- .claude/skills/mendix/migrate-design-prototype.md | 9 ++++++--- 6 files changed, 36 insertions(+), 17 deletions(-) diff --git a/.claude/skills/debug-bson.md b/.claude/skills/debug-bson.md index f5a7fd5b9..76bb2f7db 100644 --- a/.claude/skills/debug-bson.md +++ b/.claude/skills/debug-bson.md @@ -37,12 +37,14 @@ Note the exact error code (CE0463, CE0642, etc.) and which widget triggers it. ### Step 2: Get a Known-Good Reference -Create a working example in Studio Pro and update it: +Create a working example in Studio Pro and update it. **Do this on a COPY** — both +commands below mutate the project, and `update-widgets` converts an MPRv2 project to +single-file v1 (deletes `mprcontents/`): ```bash -# Convert project to latest format and update widget definitions -reference/mxbuild/modeler/mx convert -p /path/to/app.mpr -reference/mxbuild/modeler/mx update-widgets /path/to/app.mpr +# Convert project to latest format and update widget definitions (mutates in place) +reference/mxbuild/modeler/mx convert -p /path/to/copy.mpr +reference/mxbuild/modeler/mx update-widgets /path/to/copy.mpr ``` Then extract the widget's BSON to compare against your generated output. @@ -323,7 +325,7 @@ for t, props in crash_props.items(): **Investigation methodology used for v0.10 CE0463 fixes** — see [WIDGET_BSON_VERSION_COMPATIBILITY.md](../../docs/03-development/WIDGET_BSON_VERSION_COMPATIBILITY.md) for the full case study and version-resilience model. -**Quick workaround** (if you can't fix the root cause): Run `mx update-widgets` after creating pages. +**Quick workaround** (if you can't fix the root cause): normalize with `mxcli docker check`/`build`, which run the widget update **and preserve MPRv2 storage** (they snapshot/restore `.mpr` + `mprcontents/`). Do **not** run bare `mx update-widgets` on a v2 project you care about — it converts to single-file v1 and deletes `mprcontents/` (corrupts git, breaks `mxcli run --local`). Raw `mx update-widgets` is fine only on a throwaway diagnostic copy or a v1 project. ### CE0642: Property X Is Required @@ -388,7 +390,7 @@ for _, pm := range a.ParameterMappings { 2. **Mode-dependent properties must be consistent**: When changing a mode-switching property (e.g., `showContentAs`), all dependent properties must be updated to match. -3. **`mx update-widgets` is the safety net**: Running this post-processing step normalizes all widget Objects to match mpk definitions. Use it as a fallback. +3. **Widget normalization is the safety net — via `mxcli docker check`/`build`**: they run the update-widgets normalization *and* preserve MPRv2 storage (snapshot/restore). Bare `mx update-widgets` does the same normalization but rewrites a v2 project to v1 and deletes `mprcontents/` — only use it on a v1 project or a throwaway diagnostic copy. 4. **The mpk is the source of truth**: The XML schema defines property types/defaults, the editorConfig.js defines visibility rules. Together they specify the complete expected Object structure. diff --git a/.claude/skills/fix-issue.md b/.claude/skills/fix-issue.md index 57b7fb52b..820aae001 100644 --- a/.claude/skills/fix-issue.md +++ b/.claude/skills/fix-issue.md @@ -79,6 +79,7 @@ to the symptom table below, so the next similar issue costs fewer reads. | CE0463 on a pluggable widget whose TextTemplate property is *conditionally hidden* by an enum/boolean toggle (VideoPlayer `videoUrl`/`posterUrl` when `type=expression`; Timeline `title`/`description`/`timeIndication` when `customVisualization=true`) — engine clones the template's populated `ClientTemplate` for a property Studio Pro hides and nulls | Engine has no per-property visibility metadata; the hide rules live in the widget's compiled `editorConfig.js` (`hidePropertyIn`/`hidePropertiesIn`), not `widget.xml` | `mdl/executor/widget_defs.go` `widgetVisibilityRules` table + `mdl/backend/mpr/widget_builder.go` `ApplyPropertyVisibility` | Extract the widget's `.mpk` `*.editorConfig.js` (`unzip` + grep `hidePropertiesIn`), transcribe the rule into `widgetVisibilityRules[widgetID]` as `{propertyKey, hiddenWhen:{propertyKey, operator eq/ne/truthy/falsy, value}}`; the engine nulls hidden TextTemplate-typed props at build time. Bump `WidgetDefGeneratorVersion` so stale project `.def.json` auto-refresh. Issue #574 | | CE0463 on a `datagrid` whose column uses `ColumnWidth: manual` + `Size: N` — Studio Pro resets the column `size` to `1` | The MDL `ColumnWidth:` keyword isn't mapped to the schema `width` enum, so `width` stays at its `autoFill` default; `size` only applies when `width=manual`, so the value is inconsistent. Regression from the Stream B keyword-path consolidation (the deleted `datagrid_builder.go` did `colPropString(col.Properties, "ColumnWidth", "autoFill")`) | `mdl/executor/widget_defs.go` `itemPropertyAliases` | Add the MDL→schema alias under `[datagrid]["columns"]`: `"width": {"ColumnWidth"}`. Bump `WidgetDefGeneratorVersion` so stale `.def.json` regenerate. General rule when a column/object-list property's MDL keyword differs from the `.mpk` schema key (not just case), add it to `itemPropertyAliases`; cross-check against the pre-B3 `datagrid_builder.go` `colProp*` calls for any other dropped mappings | | CE0463 on an MDL-created **Combobox** (or other platform widget) at `mxcli docker build/check` time — but the SAME widget passes `mx check` on a project whose installed `.mpk` matches mxcli's embedded template (Mendix 11.6 / combobox 2.5.0). NOT the old incomplete-template bug (#112, fixed — a matching-version combobox is clean) | Widget-VERSION mismatch: mxcli emits the embedded 2.5.0-shaped PropertyTypes, but the project has a NEWER combobox (e.g. 2.8.1) that reorders/regroups properties. `augmentFromMPK` patches presence + enum values but can't restructure the baseline; `GenerateFromMPK` is less faithful still (fails even on a matching version). The designed remediation is `mx update-widgets` (docker build/check runs it before `mx check`) | `cmd/mxcli/docker/check.go` + `build.go` (`updateWidgetsPathArg`) — the update-widgets *invocation*, not the widget emission | The real trap was that `mx update-widgets ` **crashed** (`AddProjectDirAsAllowedPath` → `Path.GetDirectoryName("app.mpr")` = "" → `System.ArgumentNullException`) and some mx builds exit 0 after printing it, so the migration silently no-op'd and CE0463 survived. Pass an **absolute** path to update-widgets (always has a directory component). `mx check` is unaffected. Diagnosis: run the bundled `mx update-widgets ` yourself — if it throws ArgumentNullException on AddProjectDirAsAllowedPath, the path lacks a dir. Faithful multi-version widget emission is the larger fix (#529). Issue #112; repro `mdl-examples/bug-tests/112-combobox-enum-ce0463-widget-version.mdl` | +| `mxcli docker check`/`build` (or a bare `mx update-widgets`) **silently deletes `mprcontents/`** and rewrites an MPRv2 project into single-file v1 — `check` reports **0 errors** and looks successful, but the git working tree diverges from tracked files, a running `mxcli run --local` loop breaks (it watches `mprcontents/`), and Studio Pro may crash on open (`LibGit2RepositoryProvider.WriteBaseFile`). Triggered by following the CE0463 remediation on a `mxcli new` (always v2) project | The pre-check `mx update-widgets` step (run to suppress false CE0463) **performs the conversion**: it inlines every unit into the `.mpr` (`Unit.Contents` column) and deletes `mprcontents/`. The `check` itself is read-only; `update-widgets` is the mutator. docker check/build invoked it with no storage-format protection | `cmd/mxcli/docker/check.go` (`snapshotStorageFormat` / `copyFile` / `copyDir`) + `build.go` | Snapshot `.mpr` + `mprcontents/` to a temp dir before `update-widgets`, `defer restore()` after the check (restore removes the post-conversion single-file `.mpr` residue and puts the v2 tree back); MPRv1 projects need no protection. The check still runs against the widget-normalized model, so CE0463 stays suppressed — only the on-disk format is preserved. **Never tell an agent to run bare `mx update-widgets` on a v2 project** — the synced skills (`create-page.md`, `custom-widgets.md`, `migrate-design-prototype.md`, `download-marketplace-content.md`) + dev `debug-bson.md` route to `mxcli docker check`/`build` (v2-safe) instead. Issue #763 / PR #764 | | Nightly `mx check` reports `CE0117 "Error(s) in expression." at Log message activity 'Log message (warning)'` on Mendix 10.24.19+ but not 10.24.16 or 11.x | Mendix 10.24.19 tightened expression validation: `toString()` is now a type error (toString expects a non-string input). An example called `toString($OrderNumber)` where `$OrderNumber` was already a string parameter | The offending `log warning ... with ({1} = toString($stringVar))` — find via `~/.mxcli/mxbuild/{ver}/modeler/mx check`, then bisect with `drop microflow ...` until CE0117 disappears | Remove the redundant `toString()` wrapper around already-string values. Only wrap non-string values (integers, decimals, dates, enums) in `toString()`. The Mendix 11.x parser is more lenient and lets this slide, but 10.24.19+ rejects it | | A page-level property can't be set — `ALTER PAGE X { SET PopupWidth = 800; }` (or any page-level prop other than Title/Url) fails with `unsupported page-level property: …` | The page-level SET handler only special-cased a couple of properties; everything else fell through to the default error | `mdl/backend/pagemutator/mutator.go` → `applyPageLevelSetMut` (shared by both engines) | Add a `case` writing the field at the top level of the Forms$Page doc via `dSetOrAppend` with the on-disk BSON type (int64 for PopupWidth/PopupHeight, bool for PopupResizable — verify against a Studio Pro page with `mxcli bson dump --format bson`). Page-level prop names are **case-sensitive**. For DESCRIBE roundtrip, emit the values back in the CREATE PAGE header. CREATE-time support: add a generic `IDENTIFIER COLON propertyValueV3` to `pageHeaderPropertyV3` (regen grammar), recognize the keys in `parsePageHeaderV3` (`applyGenericPageHeaderProp`, error on unknown), carry `*int`/`*bool` on `CreatePageStmtV3`, default to 600/600/false in `buildPageV3`, and have both writers honour `page.Popup*` (legacy `sdk/mpr/writer_pages.go` int64; codec `mdl/backend/modelsdk/page_write.go` int32 via gen — tolerated by mx check). The MCP backend has its **own** `mcpPageMutator` (pg content tree, not raw BSON) — page-level SET there reaches `SetWidgetProperty("")`; map it or reject honestly (it rejects, pending a `pg_read_page` probe of the pop-up keys). Issue #661 | | `PopupWidth: 0` / `PopupHeight: 0` rejected ("must be a positive number") on CREATE or ALTER PAGE; user can't make an auto-size pop-up | 0 is actually Studio Pro's **default** for pop-up dimensions (auto-size) — verified live on 11.12: a pg-created PopupLayout page stores 0/0 and `mx check` = 0 errors. Two validators rejected ≤0, and both writers coerced ≤0→600, so even an allowed 0 became 600 | `mdl/visitor/visitor_page_v3.go` (`popupDimensionValue`) + `mdl/backend/pagemutator/mutator.go` (`coercePopupDimension`) + `mdl/executor/cmd_pages_builder_v3.go` (builder default) + `sdk/mpr/writer_pages.go` & `mdl/backend/modelsdk/page_write.go` (`popupDimension`) + `mdl/executor/cmd_pages_describe.go` | Relax both validators to reject only **negative**; default the builder to **0** (not 600, matching Studio Pro); drop the `≤0→600` coercion in both writers (clamp only negatives to 0); have DESCRIBE suppress only the real default 0 (emit an explicit 600). No `*int` needed — 0 is a valid stored value. Bug-test `mdl-examples/bug-tests/713-popup-zero-dimensions.mdl`. Issue #713 | diff --git a/.claude/skills/mendix/create-page.md b/.claude/skills/mendix/create-page.md index 46ce76e34..ec5e3400e 100644 --- a/.claude/skills/mendix/create-page.md +++ b/.claude/skills/mendix/create-page.md @@ -831,9 +831,16 @@ Notes: PieChart use the same `series` shape. - **CE0463 at `mx check`**: charts can report "widget definition changed" from widget-version drift (embedded template vs the installed `Charts.mpk`), even for a - chart with no series. `mxcli docker check`/`build` fix this automatically by - running `mx update-widgets` first; if you invoke `mx check` directly, run - `mx update-widgets ` (absolute path) beforehand. + chart with no series. Clear it with **`mxcli docker check`/`build`**, which normalize + the widgets and preserve your storage format. + **Do NOT run bare `mx update-widgets` on an MPRv2 project** (one with an + `mprcontents/` folder — everything `mxcli new` creates). `mx update-widgets` rewrites + the project into the single-file v1 format and **deletes `mprcontents/`**, which + corrupts a git working tree, breaks a running `mxcli run --local` loop, and can leave + Studio Pro unable to open the project. `mxcli docker check`/`build` snapshot and + restore the v2 files around the normalization, so they are safe; raw + `mx update-widgets` is only safe on a v1 project (or on a throwaway copy used purely + for diagnosis). - LineChart/BubbleChart/HeatMap (the `line`/`scalecolor` object-lists) are not yet authorable via MDL — use Studio Pro for those. diff --git a/.claude/skills/mendix/custom-widgets.md b/.claude/skills/mendix/custom-widgets.md index fc57b32db..e45d6e0f2 100644 --- a/.claude/skills/mendix/custom-widgets.md +++ b/.claude/skills/mendix/custom-widgets.md @@ -123,10 +123,16 @@ MDL032). **CE0463 "update this widget" is EXPECTED after generating charts.** mxcli writes the WidgetType from an embedded 11.6 baseline; the installed Charts.mpk is a -different version, so Studio Pro/mxbuild flags drift. Clear it with -`mx update-widgets` (or just use `mxcli docker check`/`build`, which run it first) -before `mx check`. The whole `mdl-examples/doctype-tests/34-chart-widget-examples.mdl` -builds **0 errors** after update-widgets. +different version, so Studio Pro/mxbuild flags drift. Clear it with **`mxcli docker +check`/`build`** (they normalize the widgets and preserve your storage format). The +whole `mdl-examples/doctype-tests/34-chart-widget-examples.mdl` builds **0 errors** +after normalization. +**Do NOT run bare `mx update-widgets` on an MPRv2 project** (an `mprcontents/`-folder +project — what `mxcli new` creates): it converts the project to single-file v1 and +**deletes `mprcontents/`**, corrupting git, breaking a running `mxcli run --local` +loop, and sometimes making the project unopenable in Studio Pro. `mxcli docker +check`/`build` snapshot/restore the v2 files around the normalization; raw +`mx update-widgets` is only safe on a v1 project or a throwaway diagnostic copy. **DESCRIBE round-trips** series/line/scalecolor object-lists (item names are synthesized, e.g. `series1`); a Pie/HeatMap's widget-level `SeriesName`/datasource diff --git a/.claude/skills/mendix/download-marketplace-content.md b/.claude/skills/mendix/download-marketplace-content.md index 7917103bd..e5e87a105 100644 --- a/.claude/skills/mendix/download-marketplace-content.md +++ b/.claude/skills/mendix/download-marketplace-content.md @@ -60,7 +60,7 @@ mxcli marketplace install -p app.mpr [--version X.Y.Z] | Content type | Behavior | |---|---| -| **Widget** | Copied into `widgets/` (overwrites on update). Reload in Studio Pro or run `mx update-widgets`. | +| **Widget** | Copied into `widgets/` (overwrites on update). Reload in Studio Pro, or run `mxcli docker check`/`build` to normalize (v2-safe). Do **not** run bare `mx update-widgets` on an `mprcontents/` project — it converts to v1 and deletes `mprcontents/`. | | **Module** (new) | Imported via `mx module-import` — needs a matching mxbuild (`mxcli setup mxbuild -p app.mpr`). | | **Module** (already present) | **Reported, not modified** — see the caveat below. | | Theme / Starter App / Sample | Downloaded with import instructions (import via Studio Pro). | diff --git a/.claude/skills/mendix/migrate-design-prototype.md b/.claude/skills/mendix/migrate-design-prototype.md index 997587044..ad15fd51a 100644 --- a/.claude/skills/mendix/migrate-design-prototype.md +++ b/.claude/skills/mendix/migrate-design-prototype.md @@ -256,7 +256,9 @@ too — each `series`/`line` binds its own OQL-view datasource + X/Y attributes; Pie/HeatMap bind at the widget level (`ValueAttribute:`, Pie needs `SeriesName:`). See **[Custom & Pluggable Widgets → Charts](custom-widgets.md)** for the chart-type → id table, per-chart required-property gotchas (TimeSeries needs a datetime X, -Bubble needs a size attribute), and the **CE0463 → `mx update-widgets`** step. +Bubble needs a size attribute), and the **CE0463 → `mxcli docker check`/`build`** step +(these normalize widgets *and* preserve MPRv2 storage — never run bare +`mx update-widgets` on a `mxcli new` project; it deletes `mprcontents/`). `mdl-examples/doctype-tests/34-chart-widget-examples.mdl` is the full showcase. **Pluggable-widget gotchas:** @@ -502,8 +504,9 @@ the fast index so a design migration doesn't rediscover them. (`Charts.mpk`: column/bar/line/area/pie)** now author via MDL — each `series` (an object-list item inside the chart) binds a datasource plus X/Y attributes: `series s1 (staticDataSource: database from Module.View, staticXAttribute: "X", staticYAttribute: "Y")` - (a per-series OQL view works too). `mxcli docker check`/`build` run `mx update-widgets`, which - clears the widget-version-drift CE0463. Still lighter when the design allows: a **CSS-background + (a per-series OQL view works too). `mxcli docker check`/`build` clear the + widget-version-drift CE0463 (they normalize the widgets and preserve MPRv2 storage — + do not run bare `mx update-widgets`, which deletes `mprcontents/`). Still lighter when the design allows: a **CSS-background SVG** container (or `HTMLElement`) for sparklines/trends — no datasource — and `ProgressCircle` (`type: expression`, `expressionCurrentValue: '$currentObject/Rate'`, min `'0'` / max `'100'`, `labelType: percentage`) for a single-value gauge. From 549c44f8ac6a990cab7f22cebcffe7f48e092346 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 19 Jul 2026 15:22:47 +0000 Subject: [PATCH 03/37] fix(widgets): clear stale Image template default causing CE0463 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Authoring an Image widget on any Mendix 11.7+ project and running `mx check` (without `mx update-widgets`) reported CE0463 "the definition of this widget has changed". The documented remedy — `mx update-widgets` — rewrites an MPRv2 project to v1 and deletes mprcontents/ (issue #763), so this stale default quietly funnelled users into data loss. Root cause (found by ruling out every other theory empirically): the embedded image.json carried a spurious default value — a WidgetValue.Image pointing at `Atlas_Core.Content.Mendix` (Atlas's Mendix logo), captured when the template was extracted from a project that had it set. The installed 11.12 Image widget expects that field empty, so MxBuild flags the definition as changed. It was NOT a version stamp, Type $IDs, property order, or missing properties — those were all disproven (a passing datagrid is reordered by update-widgets too, and authoring with a "correct" extracted template + stableIds still failed). Fix: clear the default to "" in both engine templates. Verified: the widget now passes `mx check --no-update-widgets` with 0 errors on Mendix 11.12.1. Adds a bug-test repro and a fix-issue.md symptom row documenting the order-independent before/after-update-widgets diff method used to isolate it (reusable for the rest of this CE0463 class, incl. DataGrid2 #600). Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- .claude/skills/fix-issue.md | 1 + .../bug-tests/image-ce0463-stale-default.mdl | 55 +++++++++++++++++++ .../widgets/templates/mendix-11.6/image.json | 2 +- sdk/widgets/templates/mendix-11.6/image.json | 2 +- 4 files changed, 58 insertions(+), 2 deletions(-) create mode 100644 mdl-examples/bug-tests/image-ce0463-stale-default.mdl diff --git a/.claude/skills/fix-issue.md b/.claude/skills/fix-issue.md index 820aae001..ca0cac36b 100644 --- a/.claude/skills/fix-issue.md +++ b/.claude/skills/fix-issue.md @@ -79,6 +79,7 @@ to the symptom table below, so the next similar issue costs fewer reads. | CE0463 on a pluggable widget whose TextTemplate property is *conditionally hidden* by an enum/boolean toggle (VideoPlayer `videoUrl`/`posterUrl` when `type=expression`; Timeline `title`/`description`/`timeIndication` when `customVisualization=true`) — engine clones the template's populated `ClientTemplate` for a property Studio Pro hides and nulls | Engine has no per-property visibility metadata; the hide rules live in the widget's compiled `editorConfig.js` (`hidePropertyIn`/`hidePropertiesIn`), not `widget.xml` | `mdl/executor/widget_defs.go` `widgetVisibilityRules` table + `mdl/backend/mpr/widget_builder.go` `ApplyPropertyVisibility` | Extract the widget's `.mpk` `*.editorConfig.js` (`unzip` + grep `hidePropertiesIn`), transcribe the rule into `widgetVisibilityRules[widgetID]` as `{propertyKey, hiddenWhen:{propertyKey, operator eq/ne/truthy/falsy, value}}`; the engine nulls hidden TextTemplate-typed props at build time. Bump `WidgetDefGeneratorVersion` so stale project `.def.json` auto-refresh. Issue #574 | | CE0463 on a `datagrid` whose column uses `ColumnWidth: manual` + `Size: N` — Studio Pro resets the column `size` to `1` | The MDL `ColumnWidth:` keyword isn't mapped to the schema `width` enum, so `width` stays at its `autoFill` default; `size` only applies when `width=manual`, so the value is inconsistent. Regression from the Stream B keyword-path consolidation (the deleted `datagrid_builder.go` did `colPropString(col.Properties, "ColumnWidth", "autoFill")`) | `mdl/executor/widget_defs.go` `itemPropertyAliases` | Add the MDL→schema alias under `[datagrid]["columns"]`: `"width": {"ColumnWidth"}`. Bump `WidgetDefGeneratorVersion` so stale `.def.json` regenerate. General rule when a column/object-list property's MDL keyword differs from the `.mpk` schema key (not just case), add it to `itemPropertyAliases`; cross-check against the pre-B3 `datagrid_builder.go` `colProp*` calls for any other dropped mappings | | CE0463 on an MDL-created **Combobox** (or other platform widget) at `mxcli docker build/check` time — but the SAME widget passes `mx check` on a project whose installed `.mpk` matches mxcli's embedded template (Mendix 11.6 / combobox 2.5.0). NOT the old incomplete-template bug (#112, fixed — a matching-version combobox is clean) | Widget-VERSION mismatch: mxcli emits the embedded 2.5.0-shaped PropertyTypes, but the project has a NEWER combobox (e.g. 2.8.1) that reorders/regroups properties. `augmentFromMPK` patches presence + enum values but can't restructure the baseline; `GenerateFromMPK` is less faithful still (fails even on a matching version). The designed remediation is `mx update-widgets` (docker build/check runs it before `mx check`) | `cmd/mxcli/docker/check.go` + `build.go` (`updateWidgetsPathArg`) — the update-widgets *invocation*, not the widget emission | The real trap was that `mx update-widgets ` **crashed** (`AddProjectDirAsAllowedPath` → `Path.GetDirectoryName("app.mpr")` = "" → `System.ArgumentNullException`) and some mx builds exit 0 after printing it, so the migration silently no-op'd and CE0463 survived. Pass an **absolute** path to update-widgets (always has a directory component). `mx check` is unaffected. Diagnosis: run the bundled `mx update-widgets ` yourself — if it throws ArgumentNullException on AddProjectDirAsAllowedPath, the path lacks a dir. Faithful multi-version widget emission is the larger fix (#529). Issue #112; repro `mdl-examples/bug-tests/112-combobox-enum-ce0463-widget-version.mdl` | +| **CE0463 "widget definition changed" at an `Image` widget** on Mendix 11.7+ (`mx check` without `update-widgets`) — a plain mxcli-authored Image, no custom config. `update-widgets` clears it (and on v2 destroys `mprcontents/`, so it's a data-loss trap) | NOT a version stamp, Type `$ID`s, property order, or missing properties (all ruled out empirically). The embedded `image.json` carried a **spurious default value**: a `WidgetValue.Image` pointing at `Atlas_Core.Content.Mendix` (Atlas's Mendix logo), captured when the template was extracted from a project that had it set. The installed 11.12 Image widget expects that field empty, so MxBuild flags the definition as changed | `modelsdk/widgets/templates/mendix-11.6/image.json` + `sdk/widgets/templates/mendix-11.6/image.json` (line ~63) | Clear the stale default: `"Image": "Atlas_Core.Content.Mendix"` → `"Image": ""` in both engine templates. **Diagnosis method for this whole CE0463 class**: dump the widget BSON, `mx update-widgets` on a COPY, dump again, diff the `CustomWidgets$CustomWidget` subtree **order-independently** (canonicalise key order + mask `$ID`/`TypePointer` blobs). Reordering and generic instance chrome (`LabelTemplate`, `Appearance.DesignProperties`) are cosmetic — a *passing* widget gets reordered/those-added too; the real cause is whatever value/structure survives that normalisation. Other hand-extracted templates likely hide similar stale defaults (audit with the same diff). Repro `mdl-examples/bug-tests/image-ce0463-stale-default.mdl`. DataGrid2 custom-content CE0463 (#600) is a *separate*, more complex delta — same method, own fix | | `mxcli docker check`/`build` (or a bare `mx update-widgets`) **silently deletes `mprcontents/`** and rewrites an MPRv2 project into single-file v1 — `check` reports **0 errors** and looks successful, but the git working tree diverges from tracked files, a running `mxcli run --local` loop breaks (it watches `mprcontents/`), and Studio Pro may crash on open (`LibGit2RepositoryProvider.WriteBaseFile`). Triggered by following the CE0463 remediation on a `mxcli new` (always v2) project | The pre-check `mx update-widgets` step (run to suppress false CE0463) **performs the conversion**: it inlines every unit into the `.mpr` (`Unit.Contents` column) and deletes `mprcontents/`. The `check` itself is read-only; `update-widgets` is the mutator. docker check/build invoked it with no storage-format protection | `cmd/mxcli/docker/check.go` (`snapshotStorageFormat` / `copyFile` / `copyDir`) + `build.go` | Snapshot `.mpr` + `mprcontents/` to a temp dir before `update-widgets`, `defer restore()` after the check (restore removes the post-conversion single-file `.mpr` residue and puts the v2 tree back); MPRv1 projects need no protection. The check still runs against the widget-normalized model, so CE0463 stays suppressed — only the on-disk format is preserved. **Never tell an agent to run bare `mx update-widgets` on a v2 project** — the synced skills (`create-page.md`, `custom-widgets.md`, `migrate-design-prototype.md`, `download-marketplace-content.md`) + dev `debug-bson.md` route to `mxcli docker check`/`build` (v2-safe) instead. Issue #763 / PR #764 | | Nightly `mx check` reports `CE0117 "Error(s) in expression." at Log message activity 'Log message (warning)'` on Mendix 10.24.19+ but not 10.24.16 or 11.x | Mendix 10.24.19 tightened expression validation: `toString()` is now a type error (toString expects a non-string input). An example called `toString($OrderNumber)` where `$OrderNumber` was already a string parameter | The offending `log warning ... with ({1} = toString($stringVar))` — find via `~/.mxcli/mxbuild/{ver}/modeler/mx check`, then bisect with `drop microflow ...` until CE0117 disappears | Remove the redundant `toString()` wrapper around already-string values. Only wrap non-string values (integers, decimals, dates, enums) in `toString()`. The Mendix 11.x parser is more lenient and lets this slide, but 10.24.19+ rejects it | | A page-level property can't be set — `ALTER PAGE X { SET PopupWidth = 800; }` (or any page-level prop other than Title/Url) fails with `unsupported page-level property: …` | The page-level SET handler only special-cased a couple of properties; everything else fell through to the default error | `mdl/backend/pagemutator/mutator.go` → `applyPageLevelSetMut` (shared by both engines) | Add a `case` writing the field at the top level of the Forms$Page doc via `dSetOrAppend` with the on-disk BSON type (int64 for PopupWidth/PopupHeight, bool for PopupResizable — verify against a Studio Pro page with `mxcli bson dump --format bson`). Page-level prop names are **case-sensitive**. For DESCRIBE roundtrip, emit the values back in the CREATE PAGE header. CREATE-time support: add a generic `IDENTIFIER COLON propertyValueV3` to `pageHeaderPropertyV3` (regen grammar), recognize the keys in `parsePageHeaderV3` (`applyGenericPageHeaderProp`, error on unknown), carry `*int`/`*bool` on `CreatePageStmtV3`, default to 600/600/false in `buildPageV3`, and have both writers honour `page.Popup*` (legacy `sdk/mpr/writer_pages.go` int64; codec `mdl/backend/modelsdk/page_write.go` int32 via gen — tolerated by mx check). The MCP backend has its **own** `mcpPageMutator` (pg content tree, not raw BSON) — page-level SET there reaches `SetWidgetProperty("")`; map it or reject honestly (it rejects, pending a `pg_read_page` probe of the pop-up keys). Issue #661 | diff --git a/mdl-examples/bug-tests/image-ce0463-stale-default.mdl b/mdl-examples/bug-tests/image-ce0463-stale-default.mdl new file mode 100644 index 000000000..5829a1f1a --- /dev/null +++ b/mdl-examples/bug-tests/image-ce0463-stale-default.mdl @@ -0,0 +1,55 @@ +-- ============================================================================ +-- Image widget CE0463 "widget definition changed" from a stale template default +-- ============================================================================ +-- +-- Symptom (before fix): authoring an Image widget on any Mendix 11.7+ project +-- and running `mx check` (without `mx update-widgets`) reports: +-- +-- [CE0463] "The definition of this widget has changed. Update this widget..." +-- at Image 'img' +-- +-- The documented remedy was `mx update-widgets`, which on an MPRv2 project +-- rewrites storage to v1 and deletes mprcontents/ (issue #763) — so this stale +-- default quietly funnelled users into data loss. +-- +-- Root cause: NOT a version stamp, Type $IDs, property order, or missing +-- properties (all ruled out empirically). The embedded template +-- `image.json` carried a spurious default value — a `WidgetValue.Image` +-- pointing at `Atlas_Core.Content.Mendix` (Atlas's Mendix logo) — captured +-- when the template was extracted from a project that happened to have it set. +-- The installed 11.12 Image widget expects that field empty (""), so MxBuild +-- flagged the definition as changed. Clearing the default to "" makes the +-- widget pass with 0 errors and no update-widgets. +-- +-- Diagnosis method (reusable for this whole CE0463 class): dump the widget +-- BSON, run `mx update-widgets` on a COPY, dump again, and diff the +-- CustomWidget subtree ORDER-INDEPENDENTLY (canonicalise key order + mask +-- $ID/TypePointer blobs). Reordering and generic instance chrome +-- (LabelTemplate, Appearance/DesignProperties) are cosmetic — a passing +-- widget gets those too. The real cause is whatever value/structure survives +-- that normalisation. +-- +-- Fix: modelsdk/widgets/templates/mendix-11.6/image.json (+ sdk/widgets/...): +-- "Image": "Atlas_Core.Content.Mendix" -> "Image": "" +-- +-- Verify: +-- mxcli exec image-ce0463-stale-default.mdl -p <11.12 app.mpr> +-- mxcli docker check -p --no-update-widgets -> 0 errors +-- (before the fix: 1 error, CE0463 at Image 'img') +-- ============================================================================ + +create page MyFirstModule.ImgPage ( + Title: 'Image CE0463 repro', + Layout: Atlas_Core.Atlas_Default +) { + layoutgrid lg { + row r { + column c (DesktopWidth: 12) { + pluggablewidget 'com.mendix.widget.web.image.Image' img ( + ImageType: 'imageUrl', + imageUrl: 'https://example.com/x.png' + ) + } + } + } +} diff --git a/modelsdk/widgets/templates/mendix-11.6/image.json b/modelsdk/widgets/templates/mendix-11.6/image.json index 1baf84acb..a75d08f62 100644 --- a/modelsdk/widgets/templates/mendix-11.6/image.json +++ b/modelsdk/widgets/templates/mendix-11.6/image.json @@ -60,7 +60,7 @@ "Expression": "", "Form": "", "Icon": null, - "Image": "Atlas_Core.Content.Mendix", + "Image": "", "Microflow": "", "Nanoflow": "", "Objects": [ diff --git a/sdk/widgets/templates/mendix-11.6/image.json b/sdk/widgets/templates/mendix-11.6/image.json index 297ccf09c..8287cdbe2 100644 --- a/sdk/widgets/templates/mendix-11.6/image.json +++ b/sdk/widgets/templates/mendix-11.6/image.json @@ -60,7 +60,7 @@ "Expression": "", "Form": "", "Icon": null, - "Image": "Atlas_Core.Content.Mendix", + "Image": "", "Microflow": "", "Nanoflow": "", "Objects": [ From f7c87aae67fe0c644486ccbde53f6935386bc425 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 19 Jul 2026 16:06:17 +0000 Subject: [PATCH 04/37] test(widgets): broaden dirty-template guard to catch image/page/action defaults MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The dirty-template guard (from the ComboBox fix, 827bffd4b) only flagged concrete entity/attribute/flow refs in a template's Object — which is exactly why the Image dirty default (WidgetValue.Image = "Atlas_Core.Content.Mendix", fixed in 549c44f) shipped and caused CE0463. Extend `dirtyBindings` to also flag image-asset refs, page refs, and configured client actions (Action != Forms$NoAction) on a WidgetValue — the broader dirty-default class the tolerance spike identified. - Scans BOTH engines' template sets (modelsdk embed + sdk sibling on disk), so a regression in either fails the build. - Non-vacuous: detect-dirt cases for all four classes (entity/image/page/action) + a clean-is-clean case guarding against false positives (an empty DataSource slot on a data widget is legitimate and must not flag). - Verified it catches the real Image regression (reintroducing the stale default fails the test; clean passes). Kept WidgetValue-scoped (false-positive-free) rather than any-node: a broader scan surfaces a configured delete Forms$ActionButton baked into datagrid.json, but that nested-Forms-widget dirt is an unverified object-list-widget issue (#600 / Open Question 1) — documented as a lead in the proposal, not flagged here. Updates PROPOSAL_multi_version_pluggable_widgets.md: Image recorded as the second cleaned template (Phase 1), the guard marked built+broadened, the before/after-update-widgets diagnosis method recorded, and the datagrid ActionButton captured as a #600 lead. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- ...ROPOSAL_multi_version_pluggable_widgets.md | 39 +++- modelsdk/widgets/dirty_template_test.go | 204 +++++++++++++----- 2 files changed, 191 insertions(+), 52 deletions(-) diff --git a/docs/11-proposals/PROPOSAL_multi_version_pluggable_widgets.md b/docs/11-proposals/PROPOSAL_multi_version_pluggable_widgets.md index 1cebae95a..b1863a4a7 100644 --- a/docs/11-proposals/PROPOSAL_multi_version_pluggable_widgets.md +++ b/docs/11-proposals/PROPOSAL_multi_version_pluggable_widgets.md @@ -47,6 +47,23 @@ Two real, narrow gaps remain: builder applied `attribute: Country` on top without resetting them → an Object inconsistent with the schema → `CE0463`. The fix that worked (commit `827bffd4b`) was simply swapping in the legacy template's **clean/neutral** Object defaults. + **A second confirmed instance (commit `549c44f`): `image.json` carried a baked-in + static image `WidgetValue.Image = "Atlas_Core.Content.Mendix"` (Atlas's Mendix logo, + captured from a configured instance) where a fresh Image widget has `""`.** Any + Image authored on 11.7+ tripped `CE0463`; the documented remedy (`mx update-widgets`) + then destroyed MPRv2 storage (#763) — so a dirty default was silently funnelling + users into data loss. Isolated by the reusable method below and fixed by clearing + the value; verified `mx check --no-update-widgets` = 0 errors. The dirty-default + class is therefore broader than datasources: it includes **image-asset refs, page + refs, and configured client actions**, not just entity/attribute/flow bindings. + + **Reusable diagnosis method (this whole `CE0463` class).** Dump the widget BSON, + `mx update-widgets` on a **copy**, dump again, and diff the + `CustomWidgets$CustomWidget` subtree **order-independently** (canonicalise key order + + mask `$ID`/`TypePointer` blobs). Confirms the spike's tolerance result *per case*: + reordering and generic instance chrome (`LabelTemplate`, `Appearance.DesignProperties`) + are cosmetic — a *passing* widget gets those too; the real cause is whatever + value/structure survives that normalisation. For Image it was exactly one field. - **No cross-version guarantee.** Nothing tests that creation stays `CE0463`-free as Mendix minors and widget versions move; regressions surface in the field. @@ -154,8 +171,8 @@ changes. Applies to both engines. | File | Change | |------|--------| -| `sdk/widgets/templates/`, `modelsdk/widgets/templates/` | Audit every embedded template's Object for instance bindings; re-extract any dirty one from a fresh Studio-Pro widget (as done for ComboBox in `827bffd4b`) | -| `modelsdk/widgets/` (+ `sdk/widgets`) | New dirty-template guard: a test/`widget init` check that rejects Objects with populated `AttributeRef`/`DataSource`/`EntityRef` | +| `sdk/widgets/templates/`, `modelsdk/widgets/templates/` | Audit every embedded template's Object for instance bindings; re-extract any dirty one from a fresh Studio-Pro widget (ComboBox `827bffd4b`; **Image `549c44f` — done**) | +| `modelsdk/widgets/dirty_template_test.go` | **Built + broadened (this cycle).** Dirty-template guard (`dirtyBindings` + `TestEmbeddedTemplates_NoDirtyBindings`) rejects Objects with concrete entity/attribute/flow refs **and now image-asset / page / configured-action bindings** — the class the original guard missed, which is how the Image bug shipped. Scans **both** engines' template sets; proven non-vacuous (detect-dirt + clean-is-clean cases) and proven to catch the real Image regression. Follow-up: promote `dirtyBindings` from test-only to a runtime `widget init` check | | `modelsdk/widgets/loader.go`, `sdk/widgets/loader.go` | Consolidate so both engines share one augment path | | `sdk/widgets/augment.go` / `modelsdk/widgets/augment.go` | Reconcile to a single implementation | | `mdl/backend/widgetobj/builder.go` | Ensure the builder fully overrides every Object slot it sets (no bleed-through) | @@ -164,8 +181,13 @@ changes. Applies to both engines. ### Phasing -1. **Audit embedded templates for dirty Objects** + add the guard. ComboBox is - already clean (`827bffd4b`); confirm DataGrid2/Gallery/the rest. +1. **Audit embedded templates for dirty Objects** + add the guard. **In progress:** + ComboBox clean (`827bffd4b`); Image cleaned (`549c44f`); the guard is built and + broadened to the image/page/action class over both engines. Remaining: a broader + any-node scan surfaces a **configured delete `Forms$ActionButton`** + (`Forms$DeleteClientAction` + `Atlas_Core.Atlas.trash-can`) baked into + `datagrid.json` — nested-Forms-widget dirt, out of the WidgetValue-scoped guard's + safe reach (see Open Question 1). Confirm/clean it as part of the DataGrid2 pass. 2. Consolidate the two engines' widget loaders / augment onto one path. 3. Cross-version validation matrix. 4. (Optional) Evaluate whether a fresh-instance extraction could *generate* clean @@ -207,6 +229,15 @@ regenerate gen from the target version's reflection-data. object-list widgets likely need a clean template **per widget-version**, not one shared 11.6 base. Open: root-cause the Gallery nested diff, then decide per-version template vs deeper (nested) augment. + **Concrete lead (from the dirty-template audit):** `datagrid.json`'s Object has 0 + default `Forms$GridColumn`s but **1 configured `Forms$ActionButton`** — a + `Forms$DeleteClientAction` with an `Atlas_Core.Atlas.trash-can` icon, clearly + extracted from a configured grid. `#600`'s DataGrid2 custom-content `CE0463` still + reproduces on 11.12.1 (`datagrid` with an attribute column passes, but adding a + custom-content column fails; `update-widgets` clears it) and is likely this + nested-object-list dirt — which the flat WidgetValue-scoped guard deliberately does + NOT flag (false-positive risk on nested Forms widgets). Root-cause via the + before/after-`update-widgets` subtree diff, then clean or template-per-version. 2. **Long-tail marketplace widgets.** Built-ins can be hand-extracted clean. For arbitrary marketplace widgets with no embedded template, the only correct Object source is a fresh extraction — can `widget init` extract from a freshly-dropped diff --git a/modelsdk/widgets/dirty_template_test.go b/modelsdk/widgets/dirty_template_test.go index 817f9b270..507c2842c 100644 --- a/modelsdk/widgets/dirty_template_test.go +++ b/modelsdk/widgets/dirty_template_test.go @@ -4,6 +4,8 @@ package widgets import ( "encoding/json" + "os" + "path/filepath" "strings" "testing" ) @@ -13,11 +15,15 @@ import ( // instance rather than a freshly-dropped one. The ComboBox template (fixed in // 827bffd4b) had carried a System.UserRole association datasource as its "default"; // the page builder then applied the new widget's properties on top without resetting -// it, producing an Object inconsistent with the schema → CE0463. +// it, producing an Object inconsistent with the schema → CE0463. The Image template +// (fixed in 549c44f) carried a baked-in static image (Atlas_Core.Content.Mendix) for +// the same reason. // -// A neutral template's Object must reference NO concrete entity, attribute, or -// microflow/nanoflow. This is the reliable dirty signal: it is deterministic and -// false-positive-free (a genuinely clean template has zero concrete references). +// A neutral template's Object must reference NO concrete entity, attribute, +// microflow/nanoflow, image asset, page, or configured client action. These are +// reliable dirty signals: deterministic and false-positive-free (a genuinely clean +// template has none of them — an unconfigured widget's action is Forms$NoAction and +// its image/page/binding slots are empty). // // Note: this does NOT guard non-default *primitive* drift (e.g. ComboBox's // optionsSourceType="association"). That is not a reliable dirty signal — a fresh @@ -26,76 +32,162 @@ import ( // Phase-1 result), so guarding on it would flag correct templates. The cross-version // mx-check matrix (proposed separately) is what catches primitive-level problems. // -// The test is self-discovering: it scans every embedded template, so new templates -// are covered automatically. +// The test is self-discovering (scans every template) and covers BOTH engines' +// template sets, so a regression in either is caught. func TestEmbeddedTemplates_NoDirtyBindings(t *testing.T) { - const dir = "templates/mendix-11.6" - entries, err := templateFS.ReadDir(dir) - if err != nil { - t.Fatalf("read embedded templates: %v", err) - } - + // Both engines' template dirs, relative to this package. The modelsdk set is + // read via the embed FS (the exact bytes the binary ships); the legacy sdk set + // is read from disk (kept byte-identical in tandem, but guarded independently). scanned := 0 - for _, entry := range entries { - if entry.IsDir() || !strings.HasSuffix(entry.Name(), ".json") { - continue - } + scan := func(label string, data []byte) { scanned++ - name := entry.Name() - t.Run(name, func(t *testing.T) { - data, err := templateFS.ReadFile(dir + "/" + name) - if err != nil { - t.Fatalf("read %s: %v", name, err) - } + t.Run(label, func(t *testing.T) { var tmpl WidgetTemplate if err := json.Unmarshal(data, &tmpl); err != nil { - t.Fatalf("parse %s: %v", name, err) + t.Fatalf("parse %s: %v", label, err) } for _, b := range dirtyBindings(tmpl.Object) { t.Errorf("%s: template Object contains a concrete instance binding %q — "+ - "templates must be extracted from a FRESH (unconfigured) Studio Pro widget", name, b) + "templates must be extracted from a FRESH (unconfigured) Studio Pro widget. "+ + "Re-extract clean or clear the offending default. See "+ + "PROPOSAL_multi_version_pluggable_widgets.md (Phase 1).", label, b) } }) } + // modelsdk engine — embedded bytes. + const embDir = "templates/mendix-11.6" + entries, err := templateFS.ReadDir(embDir) + if err != nil { + t.Fatalf("read embedded templates: %v", err) + } + for _, e := range entries { + if e.IsDir() || !strings.HasSuffix(e.Name(), ".json") { + continue + } + data, err := templateFS.ReadFile(embDir + "/" + e.Name()) + if err != nil { + t.Fatalf("read %s: %v", e.Name(), err) + } + scan("modelsdk/"+e.Name(), data) + } + + // legacy sdk engine — sibling package's template files on disk (kept + // byte-identical to the modelsdk set, but guarded independently). + const sdkDir = "../../sdk/widgets/templates/mendix-11.6" + sdkEntries, err := os.ReadDir(sdkDir) + if err != nil { + t.Fatalf("read sdk templates (%s): %v", sdkDir, err) + } + for _, e := range sdkEntries { + if e.IsDir() || !strings.HasSuffix(e.Name(), ".json") { + continue + } + data, err := os.ReadFile(filepath.Join(sdkDir, e.Name())) + if err != nil { + t.Fatalf("read %s: %v", e.Name(), err) + } + scan("sdk/"+e.Name(), data) + } + if scanned == 0 { - t.Fatal("no embedded templates found to audit") + t.Fatal("no templates found to audit") } } -// TestDirtyBindings_DetectsDirt proves the guard is not vacuous: a synthetic Object -// shaped like the original dirty ComboBox (a System.UserRole association datasource) -// must be flagged. +// TestDirtyBindings_DetectsDirt proves the guard is not vacuous across every dirty +// class it covers: an entity-ref datasource (the original ComboBox bug), an image +// asset (the Image bug, 549c44f), a page reference, and a configured client action. func TestDirtyBindings_DetectsDirt(t *testing.T) { - dirty := map[string]any{ - "$Type": "CustomWidgets$WidgetObject", - "Properties": []any{float64(2), - map[string]any{ - "$Type": "CustomWidgets$WidgetProperty", - "Value": map[string]any{ - "$Type": "CustomWidgets$WidgetValue", - "DataSource": map[string]any{ - "$Type": "CustomWidgets$CustomWidgetXPathSource", - "EntityRef": map[string]any{ - "$Type": "DomainModels$DirectEntityRef", - "Entity": "System.UserRole", - }, + cases := []struct { + name string + value map[string]any + want string + }{ + { + name: "entity-ref datasource", + value: map[string]any{ + "$Type": "CustomWidgets$WidgetValue", + "DataSource": map[string]any{ + "$Type": "CustomWidgets$CustomWidgetXPathSource", + "EntityRef": map[string]any{ + "$Type": "DomainModels$DirectEntityRef", + "Entity": "System.UserRole", }, }, }, + want: "DirectEntityRef→System.UserRole", + }, + { + name: "image asset", + value: map[string]any{"$Type": "CustomWidgets$WidgetValue", "Image": "Atlas_Core.Content.Mendix"}, + want: "Image→Atlas_Core.Content.Mendix", + }, + { + name: "page reference", + value: map[string]any{"$Type": "CustomWidgets$WidgetValue", "Form": "MyModule.SomePage"}, + want: "Form→MyModule.SomePage", + }, + { + name: "configured client action", + value: map[string]any{ + "$Type": "CustomWidgets$WidgetValue", + "Action": map[string]any{"$Type": "Forms$DeleteClientAction"}, + }, + want: "Action→Forms$DeleteClientAction", }, } - got := dirtyBindings(dirty) - if len(got) == 0 { - t.Fatal("dirtyBindings failed to detect a System.UserRole binding") + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + obj := map[string]any{ + "$Type": "CustomWidgets$WidgetObject", + "Properties": []any{float64(2), map[string]any{"$Type": "CustomWidgets$WidgetProperty", "Value": tc.value}}, + } + got := dirtyBindings(obj) + if len(got) == 0 { + t.Fatalf("dirtyBindings failed to detect %s", tc.name) + } + found := false + for _, g := range got { + if g == tc.want { + found = true + } + } + if !found { + t.Errorf("dirtyBindings = %v, want to contain %q", got, tc.want) + } + }) } - if got[0] != "DirectEntityRef→System.UserRole" { - t.Errorf("dirtyBindings = %v, want [DirectEntityRef→System.UserRole]", got) +} + +// TestDirtyBindings_CleanIsClean proves a neutral WidgetValue (empty slots, +// Forms$NoAction) produces no findings — guarding against false positives. +func TestDirtyBindings_CleanIsClean(t *testing.T) { + clean := map[string]any{ + "$Type": "CustomWidgets$WidgetObject", + "Properties": []any{float64(2), map[string]any{ + "$Type": "CustomWidgets$WidgetProperty", + "Value": map[string]any{ + "$Type": "CustomWidgets$WidgetValue", + "AttributeRef": nil, + "DataSource": map[string]any{ // empty source slot — legitimate on data widgets + "$Type": "CustomWidgets$CustomWidgetXPathSource", + "EntityRef": nil, "XPathConstraint": "", + }, + "EntityRef": nil, "Image": "", "Form": "", "Microflow": "", "Nanoflow": "", + "Action": map[string]any{"$Type": "Forms$NoAction"}, + "PrimitiveValue": "image", "Selection": "None", // legitimate non-empty defaults + }, + }}, + } + if got := dirtyBindings(clean); len(got) != 0 { + t.Errorf("neutral Object flagged as dirty: %v", got) } } -// dirtyBindings returns every concrete entity/attribute/flow reference found -// anywhere in a widget Object. A neutral (freshly-extracted) template has none. +// dirtyBindings returns every concrete entity/attribute/flow/image/page/action +// reference found anywhere in a widget Object. A neutral (freshly-extracted) +// template has none. func dirtyBindings(obj map[string]any) []string { var out []string var walk func(any) @@ -118,6 +210,22 @@ func dirtyBindings(obj map[string]any) []string { if a, _ := v["Attribute"].(string); a != "" { out = append(out, "AttributeRef→"+a) } + case "CustomWidgets$WidgetValue": + // A concrete image-collection asset (the Image dirty-default class, + // 549c44f). A fresh widget's image slot is empty. + if img, _ := v["Image"].(string); img != "" { + out = append(out, "Image→"+img) + } + // A concrete page reference — a fresh widget targets no page. + if form, _ := v["Form"].(string); form != "" { + out = append(out, "Form→"+form) + } + // A configured client action — a fresh widget's action is Forms$NoAction. + if act, ok := v["Action"].(map[string]any); ok { + if at, _ := act["$Type"].(string); at != "" && at != "Forms$NoAction" { + out = append(out, "Action→"+at) + } + } } for k, val := range v { if (k == "Microflow" || k == "Nanoflow") && val != nil { From 7687eaf6190b85132d1eb4471cf685be1d2a9064 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 19 Jul 2026 17:14:12 +0000 Subject: [PATCH 05/37] =?UTF-8?q?docs(proposal):=20root-cause=20object-lis?= =?UTF-8?q?t=20CE0463=20=E2=80=94=20DataGrid2=20clean,=20Gallery@10.24=20s?= =?UTF-8?q?chema=20drift?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Verified pluggable-widget CE0463 directly against cached mxbuild 10.24.19 / 11.10.0 / 11.12.1: - DataGrid2 is clean across all three versions (attribute columns — the exact #600 repro — plus custom content, filters, and selection). #600 was filed on v0.11.0 and is resolved on HEAD. Corrects an earlier note that claimed DataGrid2 custom-content still failed on 11.12.1; that was a test-syntax artifact (Content: showContentAs), not a defect. - Gallery custom content fails CE0463 on 10.24 (clean on 11.10/11.12). The before/after-update-widgets subtree diff shows it is within-key PropertyType DEFINITION drift (pagingPosition enum values + default changed, a Category renamed, a Type/order shift), not a dirty default and not merely nested. augmentFromMPK reconciles key presence but not within-key definition changes. Frames the fix (deeper augment reconciling full PropertyType from the .mpk vs per-version templates) and notes this generalises — DataGrid2 will drift the same way on a far-enough version, so the deeper-augment fix future-proofs it. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- ...ROPOSAL_multi_version_pluggable_widgets.md | 51 ++++++++++++------- 1 file changed, 33 insertions(+), 18 deletions(-) diff --git a/docs/11-proposals/PROPOSAL_multi_version_pluggable_widgets.md b/docs/11-proposals/PROPOSAL_multi_version_pluggable_widgets.md index b1863a4a7..e74906b69 100644 --- a/docs/11-proposals/PROPOSAL_multi_version_pluggable_widgets.md +++ b/docs/11-proposals/PROPOSAL_multi_version_pluggable_widgets.md @@ -220,24 +220,39 @@ regenerate gen from the target version's reflection-data. ## Open Questions -1. **Per-widget-version templates for object-list widgets.** The cross-version - matrix (`scripts/widget-version-matrix.sh`) answered part of this concretely: - ComboBox and DataGrid2 reconcile across versions via augment (flat PropertyKeys), - but **Gallery `3.0.1`@10.24 produces CE0463 on both engines** — the 11.6-extracted - template's *nested object-list* sub-schema (items/content) doesn't match the - 10.24-installed widget, and augment's flat-key add/remove doesn't reach it. So - object-list widgets likely need a clean template **per widget-version**, not one - shared 11.6 base. Open: root-cause the Gallery nested diff, then decide - per-version template vs deeper (nested) augment. - **Concrete lead (from the dirty-template audit):** `datagrid.json`'s Object has 0 - default `Forms$GridColumn`s but **1 configured `Forms$ActionButton`** — a - `Forms$DeleteClientAction` with an `Atlas_Core.Atlas.trash-can` icon, clearly - extracted from a configured grid. `#600`'s DataGrid2 custom-content `CE0463` still - reproduces on 11.12.1 (`datagrid` with an attribute column passes, but adding a - custom-content column fails; `update-widgets` clears it) and is likely this - nested-object-list dirt — which the flat WidgetValue-scoped guard deliberately does - NOT flag (false-positive risk on nested Forms widgets). Root-cause via the - before/after-`update-widgets` subtree diff, then clean or template-per-version. +1. **Within-key PropertyType drift on object-list widgets — root-caused.** Verified + directly against cached mxbuild (10.24.19 / 11.10.0 / 11.12.1): + - **DataGrid2 is clean across all three** — attribute columns (the exact `#600` + repro), custom content (dynamictext / actionbutton / multi-widget), column + filters, and selection all produce **0 CE0463**. `#600` was filed on the old + v0.11.0 and is resolved on HEAD by the accumulated fixes (NestedKeyOrder + `f12aba2`, custom-content `58508d4`, …). (An earlier note here that DataGrid2 + custom-content "still reproduces on 11.12.1" was a **test-syntax artifact** — + `Content: showContentAs` injected a spurious TextTemplate — not a real defect.) + - **Gallery custom content produces CE0463 on 10.24** (clean on 11.10 / 11.12). + The before/after-`update-widgets` subtree diff shows the cause is **NOT** a dirty + default and **NOT** merely nested: the 11.6 template and the 10.24-installed + Gallery share the same PropertyType *keys* but differ in their *definitions* — + `pagingPosition` enum values changed (`bottom`/`top` → `below`/`above`, default + `bottom`→`below`), a `Category` was renamed (`General::Pagination` → + `General::Items`), and a property's `Type`/order shifted. `augmentFromMPK` + reconciles key *presence* (add/remove) but **not within-key definition changes**, + so the emitted Type ≠ the installed widget → CE0463. + + **This is the general object-list drift, not a Gallery quirk.** DataGrid2 passes on + the tested versions only because its 3.x schema stayed close to the 11.6 template; + the same mechanism will bite it on a far-enough version (a future 11.x, or another + old release). **Fix options:** (a) **deeper augment** — reconcile each matched + PropertyType's full definition (Category, DefaultValue, enum values, Type) from the + installed `.mpk`, not just key presence, so the emitted Type equals what + `update-widgets` produces; the general fix, future-proofs every object-list widget; + or (b) **per-version templates** (`gallery-10.24.json`) — simpler, maintenance-heavy, + doesn't generalise. Recommend (a), validated by `scripts/widget-version-matrix.sh`. + + **Also noted (dirty-template audit):** `datagrid.json`'s Object carries a configured + delete `Forms$ActionButton` (`Forms$DeleteClientAction` + `Atlas_Core.Atlas.trash-can`) + — a separate nested-Forms dirty default the flat WidgetValue-scoped guard deliberately + does not flag; benign on the tested versions but worth cleaning during the object-list pass. 2. **Long-tail marketplace widgets.** Built-ins can be hand-extracted clean. For arbitrary marketplace widgets with no embedded template, the only correct Object source is a fresh extraction — can `widget init` extract from a freshly-dropped From a48147e1092bfa600bbce8b7c6ac87c589c35e66 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 19 Jul 2026 17:34:47 +0000 Subject: [PATCH 06/37] docs(proposal): #600 DataGrid2 is Data-Widgets drift, not an mxcli-version bug MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The latest #600 report (mxcli v0.16.0, Mendix 11.12.0, "still present") is reproduced-by-diagnosis: the user updated Data Widgets from the marketplace (bundled 3.4.0 → 3.10.0), and that newer schema drifts from the 11.6 template exactly like Gallery@10.24 — within-key PropertyType definition changes that augmentFromMPK's key-presence reconciliation misses. Material facts established this session: - mxcli widget code is IDENTICAL v0.16.0→HEAD (only Image fix + guard differ), so no mxcli upgrade fixes the reporter's CE0463. - All BUNDLED Data Widgets versions (3.0.0@10.24, 3.4.0@11.10/11.12) reconcile and pass; a marketplace-updated newer widget is what drifts. Corrects the prior "DataGrid2 is clean across all three versions" to "clean with the bundled Data Widgets" and confirms the deeper-augment fix is the reporter's actual bug, not a Gallery-only concern. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- ...ROPOSAL_multi_version_pluggable_widgets.md | 21 ++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/docs/11-proposals/PROPOSAL_multi_version_pluggable_widgets.md b/docs/11-proposals/PROPOSAL_multi_version_pluggable_widgets.md index e74906b69..25d7cf544 100644 --- a/docs/11-proposals/PROPOSAL_multi_version_pluggable_widgets.md +++ b/docs/11-proposals/PROPOSAL_multi_version_pluggable_widgets.md @@ -222,13 +222,20 @@ regenerate gen from the target version's reflection-data. 1. **Within-key PropertyType drift on object-list widgets — root-caused.** Verified directly against cached mxbuild (10.24.19 / 11.10.0 / 11.12.1): - - **DataGrid2 is clean across all three** — attribute columns (the exact `#600` - repro), custom content (dynamictext / actionbutton / multi-widget), column - filters, and selection all produce **0 CE0463**. `#600` was filed on the old - v0.11.0 and is resolved on HEAD by the accumulated fixes (NestedKeyOrder - `f12aba2`, custom-content `58508d4`, …). (An earlier note here that DataGrid2 - custom-content "still reproduces on 11.12.1" was a **test-syntax artifact** — - `Content: showContentAs` injected a spurious TextTemplate — not a real defect.) + - **DataGrid2 is clean with the *bundled* Data Widgets** (3.0.0@10.24, 3.4.0@11.10/ + 11.12) across attribute columns (the exact `#600` repro), custom content, filters, + and selection — all **0 CE0463**. The old-v0.11.0 mxcli-side bugs (NestedKeyOrder + `f12aba2`, custom-content `58508d4`) are resolved on HEAD. **But DataGrid2 drifts + exactly like Gallery when the installed Data Widgets is *updated from the + marketplace* past what the 11.6 template reconciles with.** `#600`'s latest report + (3 days ago: mxcli **v0.16.0**, Mendix **11.12.0**, "still present") is this case — + the original reporter noted **Data Widgets 3.10.0**, newer than the 3.4.0 bundled + with 11.12.0. Confirmed material facts: mxcli's widget code is **identical + v0.16.0→HEAD** (the only deltas are the Image fix + guard), so **no mxcli upgrade + fixes it** — it is a widget-version drift needing the deeper-augment fix below. + (An earlier note that DataGrid2 custom-content "still reproduces on 11.12.1" was a + **test-syntax artifact** — `Content: showContentAs` injected a spurious + TextTemplate — not a real defect; the real driver is the updated `.mpk`.) - **Gallery custom content produces CE0463 on 10.24** (clean on 11.10 / 11.12). The before/after-`update-widgets` subtree diff shows the cause is **NOT** a dirty default and **NOT** merely nested: the 11.6 template and the 10.24-installed From 8b65f06b6913bc1222ff80dc63e62429f03719b5 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 19 Jul 2026 17:46:04 +0000 Subject: [PATCH 07/37] fix(widgets): reconcile PropertyType metadata from .mpk (within-key CE0463 drift) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit augmentFromMPK reconciled property-KEY presence (add/remove) and enum OPTION SETS, but not the rest of a matched PropertyType's definition. When a project's installed widget is updated past the embedded 11.6 template (e.g. Data Widgets 3.4.0 → 3.10.0 from the marketplace — the #600 case), the schema drifts WITHIN existing keys: - DefaultValue goes stale: reconcileEnumValues rebuilds Gallery pagingPosition's options to {below,above} but the default stays "bottom" — a value the installed widget no longer defines, so the PropertyType is inconsistent → CE0463. - Category drifts (General::Pagination → General::Items across 5 Gallery props). - Caption drifts. Add reconcilePropertyMetadata: for every PropertyType whose key the installed .mpk defines, overwrite Category, Caption, and ValueType.DefaultValue from the .mpk (only non-empty .mpk values, so a template value is never clobbered by a blank). The .mpk is authoritative for a freshly-created instance — this is what update-widgets does. Verified: after this pass, every Gallery@10.24 PropertyType matches update-widgets' output exactly (was: Category + pagingPosition DefaultValue drift). Existing widget tests green; DataGrid2 + Gallery still 0 errors on 11.12.0 (no regression). This is the likely complete fix for the #600 DataGrid2-with-updated-Data-Widgets case (that drift is PropertyType metadata). Gallery@10.24 additionally drifts in its datasource structure (Forms$GridSortBar.SortItems, CustomWidgetXPathSource. SourceVariable) — a separate codec-TypeDefaults axis, tracked in the proposal. modelsdk (default) engine only; the legacy sdk engine lacks even reconcileEnumValues. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- modelsdk/widgets/augment.go | 65 ++++++++++++++++++ modelsdk/widgets/augment_metadata_test.go | 83 +++++++++++++++++++++++ 2 files changed, 148 insertions(+) create mode 100644 modelsdk/widgets/augment_metadata_test.go diff --git a/modelsdk/widgets/augment.go b/modelsdk/widgets/augment.go index bc97c476d..4fae65e8b 100644 --- a/modelsdk/widgets/augment.go +++ b/modelsdk/widgets/augment.go @@ -169,9 +169,74 @@ func AugmentTemplate(tmpl *WidgetTemplate, def *mpk.WidgetDefinition) error { // authoritative, so overwrite each enum PropertyType's option set from it. reconcileEnumValues(tmpl.Type, mpkEnumValuesByKey(def)) + // Reconcile per-property metadata (Category, Caption) and the DefaultValue of + // existing PropertyTypes against the .mpk. reconcileEnumValues above rebuilds an + // enum's OPTION SET, but leaves a stale DefaultValue: e.g. Gallery pagingPosition's + // options reconcile to {below,above} while its default stays "bottom" — a value + // the installed widget no longer defines, so the PropertyType is inconsistent → + // CE0463. Category likewise drifts across widget versions (e.g. + // "General::Pagination" → "General::Items" on Data Widgets 3.x). The .mpk is + // authoritative for a freshly-created instance, so overwrite these from it. This is + // the within-key definition drift behind the marketplace-updated-widget CE0463 + // (issue #600 / Gallery@10.24) — augment previously reconciled key presence and + // enum options but not the rest of each matched PropertyType's definition. + reconcilePropertyMetadata(tmpl.Type, mpkPropDefsByKey(def)) + return nil } +// mpkPropDefsByKey indexes a widget's PropertyDefs by key, across both top-level and +// nested (object-list) properties. +func mpkPropDefsByKey(def *mpk.WidgetDefinition) map[string]mpk.PropertyDef { + out := map[string]mpk.PropertyDef{} + var add func([]mpk.PropertyDef) + add = func(props []mpk.PropertyDef) { + for _, p := range props { + out[p.Key] = p + if len(p.Children) > 0 { + add(p.Children) + } + } + } + add(def.Properties) + return out +} + +// reconcilePropertyMetadata walks a widget Type and, for every PropertyType whose key +// the installed .mpk defines, overwrites its Category, Caption, and ValueType +// DefaultValue from the .mpk so the emitted definition matches the installed widget. +// Only non-empty .mpk values are applied (the .mpk always carries a category/caption; +// DefaultValue is present for enumeration/boolean/integer types). +func reconcilePropertyMetadata(node any, byKey map[string]mpk.PropertyDef) { + switch v := node.(type) { + case map[string]any: + if v["$Type"] == "CustomWidgets$WidgetPropertyType" { + if key, _ := v["PropertyKey"].(string); key != "" { + if pd, ok := byKey[key]; ok { + if pd.Category != "" { + v["Category"] = pd.Category + } + if pd.Caption != "" { + v["Caption"] = pd.Caption + } + if pd.DefaultValue != "" { + if vt, ok := v["ValueType"].(map[string]any); ok { + vt["DefaultValue"] = pd.DefaultValue + } + } + } + } + } + for _, val := range v { + reconcilePropertyMetadata(val, byKey) + } + case []any: + for _, item := range v { + reconcilePropertyMetadata(item, byKey) + } + } +} + // mpkEnumValuesByKey indexes a widget's enumeration option sets by property key, // across both top-level and nested (object-list) properties. func mpkEnumValuesByKey(def *mpk.WidgetDefinition) map[string][]mpk.EnumValue { diff --git a/modelsdk/widgets/augment_metadata_test.go b/modelsdk/widgets/augment_metadata_test.go new file mode 100644 index 000000000..aaa4563bc --- /dev/null +++ b/modelsdk/widgets/augment_metadata_test.go @@ -0,0 +1,83 @@ +// SPDX-License-Identifier: Apache-2.0 + +package widgets + +import ( + "testing" + + "github.com/mendixlabs/mxcli/modelsdk/widgets/mpk" +) + +// TestReconcilePropertyMetadata verifies that augment overwrites an existing +// PropertyType's Category, Caption, and ValueType DefaultValue from the installed +// .mpk. This closes the within-key definition drift behind the marketplace-updated +// widget CE0463 (issue #600 / Gallery@10.24): reconcileEnumValues rebuilds an enum's +// OPTION SET but leaves a stale DefaultValue (e.g. pagingPosition options reconcile to +// {below,above} while the default stays "bottom" — a value the widget no longer +// defines), and Category drifts across widget versions (General::Pagination → +// General::Items). Confirmed empirically: after this reconciliation every Gallery@10.24 +// PropertyType matches what `mx update-widgets` produces. +func TestReconcilePropertyMetadata(t *testing.T) { + // A PropertyType carrying stale metadata from an older template extraction. + typ := map[string]any{ + "$Type": "CustomWidgets$ObjectType", + "PropertyTypes": []any{float64(2), + map[string]any{ + "$Type": "CustomWidgets$WidgetPropertyType", + "PropertyKey": "pagingPosition", + "Category": "General::Pagination", // stale + "Caption": "Position of pagination", // stale + "ValueType": map[string]any{ + "$Type": "CustomWidgets$WidgetValueType", + "Type": "Enumeration", + "DefaultValue": "bottom", // stale — not in the installed option set + }, + }, + }, + } + byKey := map[string]mpk.PropertyDef{ + "pagingPosition": { + Key: "pagingPosition", + Category: "General::Items", + Caption: "Position of paging buttons", + DefaultValue: "below", + }, + } + + reconcilePropertyMetadata(typ, byKey) + + pt := typ["PropertyTypes"].([]any)[1].(map[string]any) + if got := pt["Category"]; got != "General::Items" { + t.Errorf("Category = %v, want General::Items", got) + } + if got := pt["Caption"]; got != "Position of paging buttons" { + t.Errorf("Caption = %v, want 'Position of paging buttons'", got) + } + vt := pt["ValueType"].(map[string]any) + if got := vt["DefaultValue"]; got != "below" { + t.Errorf("DefaultValue = %v, want below (a value in the installed option set)", got) + } +} + +// TestReconcilePropertyMetadata_LeavesUnknownKeys verifies a PropertyType whose key +// the .mpk does not define is left untouched (no false rewrites), and that an empty +// .mpk value does not clobber a template value. +func TestReconcilePropertyMetadata_LeavesUnknownKeys(t *testing.T) { + typ := map[string]any{ + "$Type": "CustomWidgets$WidgetPropertyType", + "PropertyKey": "somethingElse", + "Category": "General::Custom", + "ValueType": map[string]any{"$Type": "CustomWidgets$WidgetValueType", "DefaultValue": "keep"}, + } + // byKey has the key but with empty DefaultValue — must not clobber "keep". + byKey := map[string]mpk.PropertyDef{ + "somethingElse": {Key: "somethingElse", Category: "General::New", DefaultValue: ""}, + } + reconcilePropertyMetadata(typ, byKey) + if got := typ["Category"]; got != "General::New" { + t.Errorf("Category = %v, want General::New (present in .mpk)", got) + } + if got := typ["ValueType"].(map[string]any)["DefaultValue"]; got != "keep" { + t.Errorf("DefaultValue = %v, want keep (empty .mpk value must not clobber)", got) + } +} From 4596b8d5aba33a9639131517b731824381d553fd Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 19 Jul 2026 17:46:38 +0000 Subject: [PATCH 08/37] docs(proposal): object-list CE0463 has two drift axes; metadata axis fixed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Records the root-cause split from the key-indexed PropertyType diff: Axis 1 (within-key PropertyType metadata — Category/DefaultValue/Caption) fixed in 8b65f06 and verified to make every Gallery@10.24 PropertyType match update-widgets; Axis 2 (datasource structure — GridSortBar.SortItems / XPathSource.SourceVariable, a codec TypeDefaults gap) remaining. Notes Axis 1 alone likely closes the #600 DataGrid2 case, since DataGrid2 does not hit the datasource axis that Gallery does. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- ...ROPOSAL_multi_version_pluggable_widgets.md | 36 ++++++++++++++----- 1 file changed, 27 insertions(+), 9 deletions(-) diff --git a/docs/11-proposals/PROPOSAL_multi_version_pluggable_widgets.md b/docs/11-proposals/PROPOSAL_multi_version_pluggable_widgets.md index 25d7cf544..47129edb8 100644 --- a/docs/11-proposals/PROPOSAL_multi_version_pluggable_widgets.md +++ b/docs/11-proposals/PROPOSAL_multi_version_pluggable_widgets.md @@ -246,15 +246,33 @@ regenerate gen from the target version's reflection-data. reconciles key *presence* (add/remove) but **not within-key definition changes**, so the emitted Type ≠ the installed widget → CE0463. - **This is the general object-list drift, not a Gallery quirk.** DataGrid2 passes on - the tested versions only because its 3.x schema stayed close to the 11.6 template; - the same mechanism will bite it on a far-enough version (a future 11.x, or another - old release). **Fix options:** (a) **deeper augment** — reconcile each matched - PropertyType's full definition (Category, DefaultValue, enum values, Type) from the - installed `.mpk`, not just key presence, so the emitted Type equals what - `update-widgets` produces; the general fix, future-proofs every object-list widget; - or (b) **per-version templates** (`gallery-10.24.json`) — simpler, maintenance-heavy, - doesn't generalise. Recommend (a), validated by `scripts/widget-version-matrix.sh`. + **This is the general object-list drift, not a Gallery quirk** — DataGrid2 with a + marketplace-updated Data Widgets (3.10.0, the #600 case) drifts the same way. + Root-caused via the key-indexed PropertyType diff into **two independent axes:** + + - **Axis 1 — within-key PropertyType metadata drift. FIXED (`8b65f06`).** + `augment` reconciled key presence and enum option sets but left the rest of a + matched PropertyType stale: a `DefaultValue` outside the reconciled enum set + (Gallery `pagingPosition` options → `{below,above}` but default stayed `bottom`), + a `Category` rename (`General::Pagination`→`General::Items`), a `Caption` change. + `reconcilePropertyMetadata` now overwrites Category/Caption/DefaultValue from the + `.mpk` for every matched key. Verified: after it, **every Gallery@10.24 + PropertyType matches `update-widgets`**; no regression on 11.12. This is the + likely complete fix for the #600 DataGrid2 case (its drift is PropertyType + metadata). *(modelsdk engine; the legacy sdk `augment` lacks even + `reconcileEnumValues` and is behind — consolidate per item 3 above.)* + - **Axis 2 — datasource-structure drift. REMAINING.** Gallery@10.24 *additionally* + drifts in its datasource: mxcli emits `Forms$GridSortBar` without `SortItems` and + `CustomWidgets$CustomWidgetXPathSource` without `SourceVariable`; the 10.24 widget + expects both (empty/null). This is a **codec `TypeDefaults`** gap in + `mdl/backend/modelsdk/widget_write.go`, not augment — a different axis from the + PropertyType schema. DataGrid2 has not been observed to hit it (its datasource + stayed stable), which is why Axis 1 alone likely closes #600. Fixing Gallery@10.24 + end-to-end needs Axis 2: register the empty `SortItems` / null `SourceVariable` + defaults so the emitted datasource matches the installed widget. + + Validate both against `scripts/widget-version-matrix.sh`. (Per-version templates + remain a fallback but are maintenance-heavy and don't generalise.) **Also noted (dirty-template audit):** `datagrid.json`'s Object carries a configured delete `Forms$ActionButton` (`Forms$DeleteClientAction` + `Atlas_Core.Atlas.trash-can`) From 10b5bcf73615ef13008a5cf7cde6413b3712a62f Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 19 Jul 2026 18:22:01 +0000 Subject: [PATCH 09/37] =?UTF-8?q?fix(widgets):=20complete=20object-list=20?= =?UTF-8?q?CE0463=20fix=20=E2=80=94=20datasource=20defaults=20+=20Property?= =?UTF-8?q?Type=20order?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two remaining axes of the object-list within-key drift (after the metadata fix 8b65f06), both needed to make Gallery@10.24 pass end-to-end without update-widgets: 1. Datasource structure (codec TypeDefaults, widget_write.go): a pluggable-widget XPath datasource always serializes SourceVariable as null when unbound, and its GridSortBar always emits an empty SortItems list (marker 2). An older embedded template omits both; a project whose installed widget expects them (Data Widgets updated past the 11.6 template) reports CE0463. Register the defaults for CustomWidgets$CustomWidgetXPathSource (NullFields: SourceVariable) and Forms$GridSortBar (MandatoryListMarkers: SortItems=2). 2. PropertyType order (augment.go): augment adds/removes/reconciles by KEY but kept the template's original order. When the installed widget reordered its properties across versions (Gallery 3.x moved pagingPosition ahead of showTotalCount), the emitted Type's PropertyType order != the installed widget -> CE0463. The WidgetType's PropertyType order IS checked (unlike the WidgetObject's Properties order, which Studio Pro tolerates). reorderPropertyTypes sorts the top-level PropertyTypes to the .mpk declaration order; Object<->Type refs are by $ID so reordering the Type list is safe. Verified end-to-end: Gallery@10.24 (previously CE0463) now 0 errors with NO update-widgets. Full regression clean — DataGrid2 (attribute + filter + custom-content) and Gallery both 0 errors on 10.24 / 11.10 / 11.12.0 / 11.12.1. Unit + executor + both widget-engine suites green. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- mdl/backend/modelsdk/widget_write.go | 13 +++++++ modelsdk/widgets/augment.go | 45 +++++++++++++++++++++++ modelsdk/widgets/augment_metadata_test.go | 37 +++++++++++++++++++ 3 files changed, 95 insertions(+) diff --git a/mdl/backend/modelsdk/widget_write.go b/mdl/backend/modelsdk/widget_write.go index ce47b6914..d07df6470 100644 --- a/mdl/backend/modelsdk/widget_write.go +++ b/mdl/backend/modelsdk/widget_write.go @@ -35,6 +35,19 @@ func init() { codec.RegisterTypeDefaults("Forms$ClientTemplate", codec.TypeDefaults{ MandatoryListMarkers: map[string]int32{"Parameters": 2}, }) + // A pluggable-widget XPath datasource (DataGrid2 / Gallery / …) always serializes + // SourceVariable as null when unbound, and its GridSortBar always emits an (empty) + // SortItems list with marker 2. An older embedded template extracted before these + // existed omits them, so a project whose installed widget expects them (e.g. Data + // Widgets updated past the 11.6 template — Gallery@10.24) reports CE0463. Registering + // the defaults makes the emitted datasource match the installed widget. (Axis 2 of + // the object-list drift — see PROPOSAL_multi_version_pluggable_widgets.md.) + codec.RegisterTypeDefaults("CustomWidgets$CustomWidgetXPathSource", codec.TypeDefaults{ + NullFields: []string{"SourceVariable"}, + }) + codec.RegisterTypeDefaults("Forms$GridSortBar", codec.TypeDefaults{ + MandatoryListMarkers: map[string]int32{"SortItems": 2}, + }) // LayoutGrid and its rows carry a null ConditionalVisibilitySettings; the grid, // rows, and columns all use the typed-array marker 2 in their parent lists. for _, t := range []string{"Forms$LayoutGrid", "Forms$LayoutGridRow"} { diff --git a/modelsdk/widgets/augment.go b/modelsdk/widgets/augment.go index 4fae65e8b..33309d172 100644 --- a/modelsdk/widgets/augment.go +++ b/modelsdk/widgets/augment.go @@ -5,6 +5,7 @@ package widgets import ( "encoding/json" "fmt" + "sort" "sync/atomic" "github.com/mendixlabs/mxcli/modelsdk/widgets/mpk" @@ -182,9 +183,53 @@ func AugmentTemplate(tmpl *WidgetTemplate, def *mpk.WidgetDefinition) error { // enum options but not the rest of each matched PropertyType's definition. reconcilePropertyMetadata(tmpl.Type, mpkPropDefsByKey(def)) + // Reorder the top-level PropertyTypes to match the installed .mpk's declaration + // order. augment above adds/removes/reconciles by KEY but leaves the template's + // original order; when the installed widget reordered its properties across + // versions (e.g. Gallery 3.x moved pagingPosition ahead of showTotalCount), the + // emitted Type's PropertyType order ≠ the installed widget → CE0463. Unlike the + // WidgetObject's Properties order (which Studio Pro tolerates — see the spike), + // the WidgetType's PropertyType order is checked. Object↔Type references are by + // $ID, so reordering the Type list is safe. The .mpk is authoritative. + reorderPropertyTypes(tmpl.Type, def) + return nil } +// reorderPropertyTypes reorders the top-level ObjectType.PropertyTypes to match the +// installed .mpk's property declaration order. Leading array markers are preserved; +// keys the .mpk does not declare (system/accessibility props absent from +// def.Properties) keep their relative order after the declared ones (stable sort). +func reorderPropertyTypes(tmplType map[string]any, def *mpk.WidgetDefinition) { + objType, ok := getMapField(tmplType, "ObjectType") + if !ok { + return + } + propTypes, ok := getArrayField(objType, "PropertyTypes") + if !ok { + return + } + order := make(map[string]int, len(def.Properties)) + for i, p := range def.Properties { + order[p.Key] = i + } + rank := func(pt any) int { + m, ok := pt.(map[string]any) + if !ok { + return -1 // markers sort first, keeping their leading position + } + key, _ := m["PropertyKey"].(string) + if pos, ok := order[key]; ok { + return pos + } + return 1 << 30 // not declared by the .mpk (system/accessibility): keep after declared + } + sort.SliceStable(propTypes, func(i, j int) bool { + return rank(propTypes[i]) < rank(propTypes[j]) + }) + setArrayField(objType, "PropertyTypes", propTypes) +} + // mpkPropDefsByKey indexes a widget's PropertyDefs by key, across both top-level and // nested (object-list) properties. func mpkPropDefsByKey(def *mpk.WidgetDefinition) map[string]mpk.PropertyDef { diff --git a/modelsdk/widgets/augment_metadata_test.go b/modelsdk/widgets/augment_metadata_test.go index aaa4563bc..f7e924906 100644 --- a/modelsdk/widgets/augment_metadata_test.go +++ b/modelsdk/widgets/augment_metadata_test.go @@ -81,3 +81,40 @@ func TestReconcilePropertyMetadata_LeavesUnknownKeys(t *testing.T) { t.Errorf("DefaultValue = %v, want keep (empty .mpk value must not clobber)", got) } } + +// TestReorderPropertyTypes verifies the top-level PropertyTypes are reordered to the +// installed .mpk's declaration order (leading array marker preserved, keys absent from +// the .mpk kept after the declared ones), closing the order axis of the object-list +// CE0463 drift (e.g. Gallery 3.x moved pagingPosition ahead of showTotalCount). +func TestReorderPropertyTypes(t *testing.T) { + pt := func(key string) map[string]any { + return map[string]any{"$Type": "CustomWidgets$WidgetPropertyType", "PropertyKey": key} + } + typ := map[string]any{ + "ObjectType": map[string]any{ + // template order: c, a, sys, b — marker 2 leads the list + "PropertyTypes": []any{float64(2), pt("c"), pt("a"), pt("sys"), pt("b")}, + }, + } + def := &mpk.WidgetDefinition{ + Properties: []mpk.PropertyDef{{Key: "a"}, {Key: "b"}, {Key: "c"}}, + } + reorderPropertyTypes(typ, def) + + got := typ["ObjectType"].(map[string]any)["PropertyTypes"].([]any) + if m, ok := got[0].(float64); !ok || m != 2 { + t.Fatalf("marker not preserved at head: %v", got[0]) + } + var order []string + for _, e := range got[1:] { + order = append(order, e.(map[string]any)["PropertyKey"].(string)) + } + // declared keys in .mpk order (a,b,c), then undeclared "sys" kept after + want := []string{"a", "b", "c", "sys"} + for i := range want { + if order[i] != want[i] { + t.Errorf("order = %v, want %v", order, want) + break + } + } +} From 5657a11cfa3f8dc5001c20437e7d8ce396474656 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 19 Jul 2026 18:22:32 +0000 Subject: [PATCH 10/37] docs(proposal): object-list CE0463 fully fixed (all 3 axes); Gallery@10.24 green Marks Axis 2 (datasource TypeDefaults + PropertyType order) fixed in 10b5bcf. Gallery@10.24 now passes mx check with 0 errors and no update-widgets; full regression clean across 10.24/11.10/11.12.0/11.12.1 for DataGrid2 and Gallery. Remaining: fold into widget-version-matrix.sh as a gate; #600 DataGrid2 end-to-end validation awaits a Data Widgets 3.10.0 project. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- ...ROPOSAL_multi_version_pluggable_widgets.md | 33 ++++++++++++------- 1 file changed, 21 insertions(+), 12 deletions(-) diff --git a/docs/11-proposals/PROPOSAL_multi_version_pluggable_widgets.md b/docs/11-proposals/PROPOSAL_multi_version_pluggable_widgets.md index 47129edb8..514b0af47 100644 --- a/docs/11-proposals/PROPOSAL_multi_version_pluggable_widgets.md +++ b/docs/11-proposals/PROPOSAL_multi_version_pluggable_widgets.md @@ -261,18 +261,27 @@ regenerate gen from the target version's reflection-data. likely complete fix for the #600 DataGrid2 case (its drift is PropertyType metadata). *(modelsdk engine; the legacy sdk `augment` lacks even `reconcileEnumValues` and is behind — consolidate per item 3 above.)* - - **Axis 2 — datasource-structure drift. REMAINING.** Gallery@10.24 *additionally* - drifts in its datasource: mxcli emits `Forms$GridSortBar` without `SortItems` and - `CustomWidgets$CustomWidgetXPathSource` without `SourceVariable`; the 10.24 widget - expects both (empty/null). This is a **codec `TypeDefaults`** gap in - `mdl/backend/modelsdk/widget_write.go`, not augment — a different axis from the - PropertyType schema. DataGrid2 has not been observed to hit it (its datasource - stayed stable), which is why Axis 1 alone likely closes #600. Fixing Gallery@10.24 - end-to-end needs Axis 2: register the empty `SortItems` / null `SourceVariable` - defaults so the emitted datasource matches the installed widget. - - Validate both against `scripts/widget-version-matrix.sh`. (Per-version templates - remain a fallback but are maintenance-heavy and don't generalise.) + - **Axis 2 — datasource structure + PropertyType order. FIXED (`10b5bcf`).** + Gallery@10.24 *additionally* drifted two ways: (a) mxcli emitted `Forms$GridSortBar` + without `SortItems` and `CustomWidgets$CustomWidgetXPathSource` without + `SourceVariable` (the 10.24 widget expects both, empty/null) — fixed with codec + `TypeDefaults` in `widget_write.go` (`MandatoryListMarkers: SortItems=2`, + `NullFields: SourceVariable`); and (b) the top-level PropertyType **order** differed + — the 3.x widget moved `pagingPosition` ahead of `showTotalCount`, and augment kept + the template order. **The WidgetType's PropertyType order IS checked by CE0463** + (unlike the WidgetObject's Properties order, which the spike proved tolerated); + `reorderPropertyTypes` now sorts the Type's PropertyTypes to the `.mpk` declaration + order (refs are by `$ID`, so it's safe). + + **Result: Gallery@10.24 now passes `mx check` with 0 errors and NO `update-widgets`.** + Full regression clean — DataGrid2 (attribute + filter + custom-content) and Gallery + both 0 errors on 10.24 / 11.10 / 11.12.0 / 11.12.1. All three axes (metadata, datasource, + order) are the general within-key/order drift that any object-list widget hits when its + installed version moves past the 11.6 template — including the #600 DataGrid2 case. + *(modelsdk engine; the legacy sdk `augment` remains behind — consolidate per item 3.)* + Remaining: fold these into `scripts/widget-version-matrix.sh` as a standing gate; and + direct end-to-end validation of the #600 DataGrid2 case awaits a project with Data + Widgets 3.10.0. **Also noted (dirty-template audit):** `datagrid.json`'s Object carries a configured delete `Forms$ActionButton` (`Forms$DeleteClientAction` + `Atlas_Core.Atlas.trash-can`) From 105c22836cf39be1939c36ad8eb65ca7350f4077 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 20 Jul 2026 04:39:06 +0000 Subject: [PATCH 11/37] =?UTF-8?q?docs(proposal):=20#600=20DataGrid2@3.10.0?= =?UTF-8?q?=20reproduced=20=E2=80=94=203=20axes=20necessary=20but=20not=20?= =?UTF-8?q?sufficient?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit End-to-end validation against the reporter's exact stack (marketplace Data Widgets 3.10.0 swapped into a Mendix 11.12.0 project): a DataGrid2 authored with the fixed binary STILL reports CE0463. Isolated from the Atlas template pages, the installed 3.10.0 schema differs from the 11.6-era template across far more than the three fixed axes reconcile — AllowUpload (79), ValueType.Type (15), Translations (14), DefaultValue (13), Required (7), EnumerationValues (7), AllowedTypes (4), and more. Corrects the earlier claim that Axis 1 alone would close #600: augment's key-presence + metadata/enum/order reconciliation scales to a MODERATE drift (Gallery@10.24, fully fixed) but not to a LARGE version jump (DataGrid2 11.6-era -> 3.10.0). The Phase-1 "56=56 for ComboBox" measurement held only because that was a small delta. Records the two paths to close the large-drift case, both now testable against real DW 3.10.0: (a) full-ValueType augment (overwrite the entire ValueType from the .mpk; recommended), or (b) per-version templates (fallback). Notes marketplace download now makes the version matrix a feasible standing gate. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- ...ROPOSAL_multi_version_pluggable_widgets.md | 66 +++++++++++++++---- 1 file changed, 53 insertions(+), 13 deletions(-) diff --git a/docs/11-proposals/PROPOSAL_multi_version_pluggable_widgets.md b/docs/11-proposals/PROPOSAL_multi_version_pluggable_widgets.md index 514b0af47..e94fd1971 100644 --- a/docs/11-proposals/PROPOSAL_multi_version_pluggable_widgets.md +++ b/docs/11-proposals/PROPOSAL_multi_version_pluggable_widgets.md @@ -257,10 +257,11 @@ regenerate gen from the target version's reflection-data. a `Category` rename (`General::Pagination`→`General::Items`), a `Caption` change. `reconcilePropertyMetadata` now overwrites Category/Caption/DefaultValue from the `.mpk` for every matched key. Verified: after it, **every Gallery@10.24 - PropertyType matches `update-widgets`**; no regression on 11.12. This is the - likely complete fix for the #600 DataGrid2 case (its drift is PropertyType - metadata). *(modelsdk engine; the legacy sdk `augment` lacks even - `reconcileEnumValues` and is behind — consolidate per item 3 above.)* + PropertyType matches `update-widgets`**; no regression on 11.12. (An earlier draft + claimed this was "the likely complete fix for #600" — **disproven**; see the + large-drift measurement below, which shows the #600 DataGrid2 case needs more.) + *(modelsdk engine; the legacy sdk `augment` lacks even `reconcileEnumValues` and is + behind — consolidate per item 3 above.)* - **Axis 2 — datasource structure + PropertyType order. FIXED (`10b5bcf`).** Gallery@10.24 *additionally* drifted two ways: (a) mxcli emitted `Forms$GridSortBar` without `SortItems` and `CustomWidgets$CustomWidgetXPathSource` without @@ -273,15 +274,54 @@ regenerate gen from the target version's reflection-data. `reorderPropertyTypes` now sorts the Type's PropertyTypes to the `.mpk` declaration order (refs are by `$ID`, so it's safe). - **Result: Gallery@10.24 now passes `mx check` with 0 errors and NO `update-widgets`.** - Full regression clean — DataGrid2 (attribute + filter + custom-content) and Gallery - both 0 errors on 10.24 / 11.10 / 11.12.0 / 11.12.1. All three axes (metadata, datasource, - order) are the general within-key/order drift that any object-list widget hits when its - installed version moves past the 11.6 template — including the #600 DataGrid2 case. - *(modelsdk engine; the legacy sdk `augment` remains behind — consolidate per item 3.)* - Remaining: fold these into `scripts/widget-version-matrix.sh` as a standing gate; and - direct end-to-end validation of the #600 DataGrid2 case awaits a project with Data - Widgets 3.10.0. + **Result (moderate drift): Gallery@10.24 passes `mx check` with 0 errors and NO + `update-widgets`.** Regression clean — DataGrid2 (attribute + filter + custom-content) + and Gallery both 0 errors on 10.24 / 11.10 / 11.12.0 / 11.12.1 with the *bundled* + Data Widgets. *(modelsdk engine; the legacy sdk `augment` remains behind — item 3.)* + + **The three axes are necessary but NOT sufficient for large version jumps — measured + against the #600 reporter's exact stack.** Downloaded Data Widgets **3.10.0** from the + marketplace, swapped it into a Mendix 11.12.0 project (over the bundled 3.4.0), and + authored a DataGrid2 with the fixed binary: **still CE0463.** Isolating that one grid + from the Atlas template pages (which legitimately drift and are a separate "update + widgets" concern), the installed 3.10.0 schema differs from the 11.6-era template in + **far more than the three axes reconcile** — an aggregated field diff over the + `CustomWidgetType`: + + | Differing `ValueType` field | # properties | + |---|---| + | `AllowUpload` (null vs false) | 79 | + | `Type` (property type changed) | 15 | + | `Translations` | 14 | + | `DefaultValue` | 13 | + | `Required` | 7 | + | `EnumerationValues` | 7 | + | `AllowedTypes` | 4 | + | + `Category`, `Description`, `ReturnType`, nested `ObjectType.PropertyTypes` | | + + So **augment's key-presence + metadata/enum/order reconciliation scales to a *moderate* + drift (Gallery@10.24: Category + one DefaultValue + order + datasource) but not to a + *large* one (DataGrid2 11.6-era → 3.10.0).** This directly disproves the earlier + working assumption that Axis 1 alone would close #600. The Phase-1 measurement that + "augment already delivers version-correct schemas (56=56 for ComboBox 2.4.3)" held only + because that was a *small* version delta; a big jump exposes per-field schema evolution + (`Type`, `Required`, `AllowedTypes`, `Translations`) that key-level augment never touches. + + **Two paths to close the large-drift case (both now testable against real DW 3.10.0):** + - **(a) Full-`ValueType` augment** — for every matched key, overwrite the *entire* + `ValueType` (`Type`, `Required`, `AllowedTypes`, `DefaultValue`, `EnumerationValues`) + from the `.mpk`, not just today's few fields. The `.mpk` is the authoritative schema, + so this generalises to any version. Open risk: `AllowUpload` / `Translations` are not + obviously in the `.mpk` — but the tolerance spike measured `AllowUpload` presence/absence + as *tolerated*, so full-`ValueType` reconciliation of the `.mpk`-backed fields may reach + 0 errors regardless. Recommended; validate directly on DW 3.10.0. + - **(b) Per-version templates** — extract a clean `datagrid-.json` per Data Widgets + release. Reliable but doesn't scale (3.11.2 already shipped); a fallback if (a) can't + reconcile the `.mpk`-absent fields. + + Remaining: pick (a)/(b), fold the whole matrix into `scripts/widget-version-matrix.sh` + as a standing gate (now feasible — `mxcli marketplace download` can fetch any widget + version for the fixture), and validate end-to-end on DW 3.10.0 (the reporter's exact case). **Also noted (dirty-template audit):** `datagrid.json`'s Object carries a configured delete `Forms$ActionButton` (`Forms$DeleteClientAction` + `Atlas_Core.Atlas.trash-can`) From 0351bc2c609ae22bac19b9e785a10bdb5b908b51 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 20 Jul 2026 05:03:44 +0000 Subject: [PATCH 12/37] =?UTF-8?q?docs(proposal):=20resolve=20#600=20DataGr?= =?UTF-8?q?id2@3.10.0=20=E2=80=94=20widget=20envelope=20is=20mxbuild-only?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit End-to-end reproduction of the #600 stack (DW 3.10.0 over bundled 3.4.0 in an 11.12.0 project, DataGrid2 authored with mxcli, isolated from the Atlas template pages) with an order-independent, ID-masked before/after-update-widgets subtree diff proves the parser/template/augment approach is structurally capped below large version jumps: - Nested-column PropertyKey sets are already identical before/after (augment adds every new 3.10.0 key correctly). - Residual drift is dominated by fields ABSENT from the widget XML source: AllowUpload (x105), DesignProperties, LabelTemplate, and a mxbuild-COMPUTED Required (XML declares 3 required="true"; mxbuild emits 54). - update-widgets on the same copy takes the grid CE0463 -> 0. Since these fields are not derivable from the .mpk, no augment (including the proposed full-ValueType path a) can reproduce a version-faithful WidgetType. A speculative reconcileValueTypeSchema was implemented and reverted (setting Required from the XML attr would mis-set the 51 computed keys to false). Revised path forward: delegate the envelope to mxbuild's own update-widgets made v2-safe (adopt mendixlabs PR #764), keep the moderate-drift augment as-is. Records the rejected approach inline in augment.go and updates the proposal Open Question #1. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- ...ROPOSAL_multi_version_pluggable_widgets.md | 54 ++++++++++++++----- modelsdk/widgets/augment.go | 11 ++++ 2 files changed, 52 insertions(+), 13 deletions(-) diff --git a/docs/11-proposals/PROPOSAL_multi_version_pluggable_widgets.md b/docs/11-proposals/PROPOSAL_multi_version_pluggable_widgets.md index e94fd1971..a24eb396d 100644 --- a/docs/11-proposals/PROPOSAL_multi_version_pluggable_widgets.md +++ b/docs/11-proposals/PROPOSAL_multi_version_pluggable_widgets.md @@ -307,21 +307,49 @@ regenerate gen from the target version's reflection-data. because that was a *small* version delta; a big jump exposes per-field schema evolution (`Type`, `Required`, `AllowedTypes`, `Translations`) that key-level augment never touches. - **Two paths to close the large-drift case (both now testable against real DW 3.10.0):** - - **(a) Full-`ValueType` augment** — for every matched key, overwrite the *entire* - `ValueType` (`Type`, `Required`, `AllowedTypes`, `DefaultValue`, `EnumerationValues`) - from the `.mpk`, not just today's few fields. The `.mpk` is the authoritative schema, - so this generalises to any version. Open risk: `AllowUpload` / `Translations` are not - obviously in the `.mpk` — but the tolerance spike measured `AllowUpload` presence/absence - as *tolerated*, so full-`ValueType` reconciliation of the `.mpk`-backed fields may reach - 0 errors regardless. Recommended; validate directly on DW 3.10.0. + **RESOLVED — path (a) is structurally impossible; only mxbuild can produce the + envelope.** Full end-to-end reproduction of the #600 stack (fresh 11.12.0 project, DW + 3.10.0 swapped in over the bundled 3.4.0, DataGrid2 `dg` authored with the fixed binary, + isolated from the Atlas template pages) with an order-independent, `$ID`/`TypePointer`-masked + before/after-`update-widgets` subtree diff of `dg` alone shows the residual drift is + dominated by fields that **do not exist in the widget XML source at all**: + + | Residual `dg` drift after augment | count | in `Datagrid.xml`? | + |---|---|---| + | `AllowUpload` (absent → `false`) | 105 | **no** — not in source | + | `Required` (`false` → `true`) | 54 | partially — XML has **3** `required="true"`, mxbuild emits **54** | + | `DesignProperties`, `LabelTemplate` | several | **no** — not in source | + | `ReturnType` (`null` → `{Type,IsList,…}`), `Type`, `DefaultValue` | ~30 | computed/partial | + + The nested-column **PropertyKey sets are already identical** before/after (augment adds + every new 3.10.0 key — `exportType`, `exportNumberFormat`, `exportDateFormat`, … — + correctly). What augment *cannot* produce is the mxbuild-**computed** BSON envelope: + `AllowUpload`/`DesignProperties`/`LabelTemplate` appear **nowhere** in `Datagrid.xml`, and + `Required` is computed (3 declared vs 54 emitted). Since these fields are not derivable + from the `.mpk`, **no parser/template/augment approach — including path (a), full-`ValueType` + reconciliation — can ever reproduce a version-faithful `WidgetType`.** (A speculative + `reconcileValueTypeSchema` was implemented and reverted: setting `Required` from the XML + `required` attr would mis-set the 51 mxbuild-computed keys to `false`.) `update-widgets` + on the same copy takes `dg` from CE0463 → **0** — confirming only mxbuild owns the envelope. + + **Revised path forward — delegate the envelope to mxbuild, made v2-safe (was path (d)):** + - **(c) v2-safe `update-widgets` as the CE0463 remediation.** The augment layer already + closes *moderate* drift (Gallery@10.24, bundled DW) and that stays. For *large* jumps, + the correct fix is to run mxbuild's own `update-widgets` — which is exactly what the + skills now route users to (`mxcli docker check`/`build`, post the earlier skill-guidance + fix). The remaining blocker is that bare `mx update-widgets` destroys MPRv2 (deletes + `mprcontents/`, converts to v1 — issue #763), which **mendixlabs PR #764** fixes. + Adopt/port #764 so `mxcli docker check`/`build` (and the warm local loop) can reconcile + widgets without downgrading the project. This is the only path that is faithful for + *arbitrary* widget versions, because only mxbuild computes the envelope. - **(b) Per-version templates** — extract a clean `datagrid-.json` per Data Widgets - release. Reliable but doesn't scale (3.11.2 already shipped); a fallback if (a) can't - reconcile the `.mpk`-absent fields. + release. Rejected as a primary fix: doesn't scale (3.11.2 already shipped) and still + wouldn't carry the computed `Required`; kept only as a note. - Remaining: pick (a)/(b), fold the whole matrix into `scripts/widget-version-matrix.sh` - as a standing gate (now feasible — `mxcli marketplace download` can fetch any widget - version for the fixture), and validate end-to-end on DW 3.10.0 (the reporter's exact case). + Remaining: implement (c) (v2-safe `update-widgets` via #764), keep the moderate-drift + augment as-is, and fold the matrix into `scripts/widget-version-matrix.sh` as a standing + gate (feasible now — `mxcli marketplace download` fetches any widget version for the + fixture) asserting *bundled*-DW authoring stays at 0 CE0463. **Also noted (dirty-template audit):** `datagrid.json`'s Object carries a configured delete `Forms$ActionButton` (`Forms$DeleteClientAction` + `Atlas_Core.Atlas.trash-can`) diff --git a/modelsdk/widgets/augment.go b/modelsdk/widgets/augment.go index 33309d172..068ca43bd 100644 --- a/modelsdk/widgets/augment.go +++ b/modelsdk/widgets/augment.go @@ -183,6 +183,17 @@ func AugmentTemplate(tmpl *WidgetTemplate, def *mpk.WidgetDefinition) error { // enum options but not the rest of each matched PropertyType's definition. reconcilePropertyMetadata(tmpl.Type, mpkPropDefsByKey(def)) + // NOTE: reconciling further ValueType scalars (Required, AllowedTypes, IsList) from + // the .mpk was evaluated and rejected — see the DataGrid2@3.10.0 investigation in + // PROPOSAL_multi_version_pluggable_widgets.md (Open Question #1). On large version + // jumps the installed widget's WidgetType BSON carries mxbuild-computed fields that + // are absent from the widget XML source entirely (AllowUpload, DesignProperties, + // LabelTemplate) and a computed Required (the DW 3.10.0 XML declares 3 required="true" + // but mxbuild emits 54), so no parser/template augment can reproduce it. The only + // faithful remediation for large drift is mxbuild's own `update-widgets`; the augment + // reconciliations here (keys, enum options, metadata, order) close moderate drift + // (Gallery@10.24) but are structurally capped below the large-jump case. + // Reorder the top-level PropertyTypes to match the installed .mpk's declaration // order. augment above adds/removes/reconciles by KEY but leaves the template's // original order; when the installed widget reordered its properties across From c7fe714700bdf27eb05da820d3fce70096a29a81 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 20 Jul 2026 05:41:39 +0000 Subject: [PATCH 13/37] fix(widgets): reconcile widget definition to installed .mpk generically (#600) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit update-widgets is fully generic — it has no per-widget knowledge, so everything needed to reconcile a widget across a large version jump is derivable from the widget package plus the generic metamodel defaults. This reworks augment to do the same, closing the DEFINITION side of the DataGrid2@3.10.0 CE0463: the emitted CustomWidgetType is now byte-identical (canonically) to mxbuild's update-widgets output for the #600 reporter's exact stack (DW 3.10.0 on Mendix 11.12.0). Parser (mpk): - Required defaults to TRUE when the XML omits required= (Mendix pluggable-widget spec default; mxbuild emits Required=true for every property without the attr). Defaulting missing->false caused within-key drift on augment-added keys. - Parse (expression props) and (shipped caption localizations). - Nested (object-list) child properties now inherit their propertyGroup caption chain as Category, matching how mxbuild derives nested categories. Augment: - reconcileValueTypesFromMPK: overwrite each matched PropertyType's ValueType scalars (Type, Required, DefaultValue, AllowedTypes, IsList, DataSourceProperty) from the .mpk; when a Type changes (wrong-typed exemplar clone), reset the matching Object WidgetValue so it stays schema-consistent; normalize the mutually-exclusive type-specific fields (EnumerationValues cleared on non-enum, ReturnType built only for Expression) so stale clone data cannot survive. - reconcilePropertyMetadata also reconciles Description. - completeValueTypeEnvelope emits the generic AllowUpload envelope default (mxbuild writes it on every ValueType; the 11.6-era template predates it). - reorderPropertyTypes now recurses into nested ObjectTypes. Verified end-to-end: definition diff vs update-widgets output = 0; no regression on bundled Data Widgets (3.4.0/3.0.0) across 11.12.0 / 11.10.0 / 10.24. Note the instance-side envelope (Forms$Appearance.DesignProperties, CustomWidget. LabelTemplate, Object-side translations) is not yet emitted; see follow-up. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- modelsdk/widgets/augment.go | 250 ++++++++++++++++++++-- modelsdk/widgets/augment_metadata_test.go | 64 ++++++ modelsdk/widgets/mpk/mpk.go | 76 ++++++- 3 files changed, 371 insertions(+), 19 deletions(-) diff --git a/modelsdk/widgets/augment.go b/modelsdk/widgets/augment.go index 068ca43bd..edd1e7b41 100644 --- a/modelsdk/widgets/augment.go +++ b/modelsdk/widgets/augment.go @@ -183,16 +183,26 @@ func AugmentTemplate(tmpl *WidgetTemplate, def *mpk.WidgetDefinition) error { // enum options but not the rest of each matched PropertyType's definition. reconcilePropertyMetadata(tmpl.Type, mpkPropDefsByKey(def)) - // NOTE: reconciling further ValueType scalars (Required, AllowedTypes, IsList) from - // the .mpk was evaluated and rejected — see the DataGrid2@3.10.0 investigation in - // PROPOSAL_multi_version_pluggable_widgets.md (Open Question #1). On large version - // jumps the installed widget's WidgetType BSON carries mxbuild-computed fields that - // are absent from the widget XML source entirely (AllowUpload, DesignProperties, - // LabelTemplate) and a computed Required (the DW 3.10.0 XML declares 3 required="true" - // but mxbuild emits 54), so no parser/template augment can reproduce it. The only - // faithful remediation for large drift is mxbuild's own `update-widgets`; the augment - // reconciliations here (keys, enum options, metadata, order) close moderate drift - // (Gallery@10.24) but are structurally capped below the large-jump case. + // Reconcile the schema-derived scalar fields of each matched PropertyType's ValueType + // against the .mpk — Type, Required, DefaultValue, AllowedTypes, IsList, + // DataSourceProperty — and, where the Type changes, reset the corresponding Object + // WidgetValue so it stays consistent with the schema. This closes the large-version + // -jump within-key drift behind issue #600 (DataGrid2 11.6-era template → Data Widgets + // 3.10.0). update-widgets is fully generic (it has no widget-specific knowledge), so + // everything it produces is derivable from the widget package + the generic metamodel + // defaults — nothing here is hardcoded per widget. Confirmed empirically against the + // #600 stack: after this reconciliation the Type/DefaultValue drift reconciles to zero + // and Required matches the spec default (absent→true, fixed in the mpk parser). + reconcileValueTypesFromMPK(tmpl, mpkPropDefsByKey(def)) + + // Emit the generic WidgetValueType envelope fields Studio Pro's current metamodel + // always serializes but that an older extracted template predates — today AllowUpload + // (default false, present on every ValueType in mxbuild output). These are generic + // metamodel defaults, not widget-specific, so they are added to every ValueType that + // lacks them regardless of key. Missing envelope fields are a within-key definition + // mismatch → CE0463 on large version jumps (issue #600: DW 3.10.0 emits AllowUpload + // on all 105 DataGrid2 ValueTypes; the 11.6-era template has none). + completeValueTypeEnvelope(tmpl.Type) // Reorder the top-level PropertyTypes to match the installed .mpk's declaration // order. augment above adds/removes/reconciles by KEY but leaves the template's @@ -216,13 +226,28 @@ func reorderPropertyTypes(tmplType map[string]any, def *mpk.WidgetDefinition) { if !ok { return } + reorderObjectTypePropertyTypes(objType, def.Properties) +} + +// reorderObjectTypePropertyTypes reorders one ObjectType's PropertyTypes to the given +// .mpk property order, then recurses into each object-list property's nested ObjectType +// using that property's children order. Leading array markers keep their position; keys +// the .mpk does not declare are kept after the declared ones (stable sort). Nested order +// matters as much as top-level: an object-list column whose child PropertyTypes are in the +// template's old order (not the installed widget's) is a within-key definition mismatch → +// CE0463 (issue #600: DataGrid2 3.10.0 reordered/added the column export* properties). +func reorderObjectTypePropertyTypes(objType map[string]any, props []mpk.PropertyDef) { propTypes, ok := getArrayField(objType, "PropertyTypes") if !ok { return } - order := make(map[string]int, len(def.Properties)) - for i, p := range def.Properties { + order := make(map[string]int, len(props)) + childrenByKey := make(map[string][]mpk.PropertyDef) + for i, p := range props { order[p.Key] = i + if len(p.Children) > 0 { + childrenByKey[p.Key] = p.Children + } } rank := func(pt any) int { m, ok := pt.(map[string]any) @@ -239,6 +264,28 @@ func reorderPropertyTypes(tmplType map[string]any, def *mpk.WidgetDefinition) { return rank(propTypes[i]) < rank(propTypes[j]) }) setArrayField(objType, "PropertyTypes", propTypes) + + // Recurse into nested ObjectTypes of object-list properties. + for _, pt := range propTypes { + m, ok := pt.(map[string]any) + if !ok { + continue + } + key, _ := m["PropertyKey"].(string) + kids, ok := childrenByKey[key] + if !ok { + continue + } + vt, ok := getMapField(m, "ValueType") + if !ok { + continue + } + nestedObjType, ok := getMapField(vt, "ObjectType") + if !ok { + continue + } + reorderObjectTypePropertyTypes(nestedObjType, kids) + } } // mpkPropDefsByKey indexes a widget's PropertyDefs by key, across both top-level and @@ -275,6 +322,9 @@ func reconcilePropertyMetadata(node any, byKey map[string]mpk.PropertyDef) { if pd.Caption != "" { v["Caption"] = pd.Caption } + if pd.Description != "" { + v["Description"] = pd.Description + } if pd.DefaultValue != "" { if vt, ok := v["ValueType"].(map[string]any); ok { vt["DefaultValue"] = pd.DefaultValue @@ -293,6 +343,182 @@ func reconcilePropertyMetadata(node any, byKey map[string]mpk.PropertyDef) { } } +// reconcileValueTypesFromMPK overwrites the schema-derived scalar fields of every +// matched PropertyType's ValueType from the .mpk (Type, Required, DefaultValue, +// AllowedTypes, IsList, DataSourceProperty), and — because an in-place Type change would +// otherwise leave the Object's WidgetValue shaped for the old type (the Object↔schema +// inconsistency that triggers CE0463) — resets each affected Object WidgetValue. +// +// The .mpk is the authoritative schema for a freshly-created instance; mxbuild's generic +// update-widgets derives exactly these values from the same package. DefaultValue is set +// unconditionally (including to "" when the .mpk omits it) so a stale template default — +// e.g. a value the installed widget no longer defines — is cleared, not preserved. +func reconcileValueTypesFromMPK(tmpl *WidgetTemplate, byKey map[string]mpk.PropertyDef) { + // PropertyType $IDs whose Type changed → their Object WidgetValue must be reset. + changedTypeIDs := map[string]mpk.PropertyDef{} + + var walk func(any) + walk = func(node any) { + switch v := node.(type) { + case map[string]any: + if v["$Type"] == "CustomWidgets$WidgetPropertyType" { + if key, _ := v["PropertyKey"].(string); key != "" { + if pd, ok := byKey[key]; ok { + if vt, ok := v["ValueType"].(map[string]any); ok { + bsonType := xmlTypeToBSONType(pd.Type) + if bsonType != "" { + if old, _ := vt["Type"].(string); old != bsonType { + if id, _ := v["$ID"].(string); id != "" { + changedTypeIDs[id] = pd + } + } + vt["Type"] = bsonType + } + vt["Required"] = pd.Required + vt["IsList"] = pd.IsList + vt["DefaultValue"] = pd.DefaultValue + vt["AllowedTypes"] = buildAllowedTypesArray(pd.AllowedTypes) + vt["DataSourceProperty"] = pd.DataSource + // Normalize the mutually-exclusive type-specific fields to the + // authoritative .mpk type. A ValueType cloned from a wrong-typed + // exemplar (or whose Type changed across widget versions) otherwise + // keeps stale fields that don't apply to its current type — e.g. a + // TextTemplate carrying EnumerationValues from an Enumeration + // exemplar, or a Widgets property carrying a cloned ReturnType. + // mxbuild emits these empty for the non-matching type, so a + // leftover is a within-key definition mismatch → CE0463. + switch bsonType { + case "Enumeration": + vt["EnumerationValues"] = buildEnumValuesArray(pd.EnumValues) + vt["ReturnType"] = nil + case "Expression": + vt["EnumerationValues"] = []any{float64(2)} + if pd.ReturnType != "" { + vt["ReturnType"] = buildReturnType(pd.ReturnType) + } else { + vt["ReturnType"] = nil + } + default: + vt["EnumerationValues"] = []any{float64(2)} + vt["ReturnType"] = nil + } + // Widget-shipped caption/template translations (from the .mpk + // ), emitted into the definition as a + // WidgetTranslation list. Absent here → CE0463 on widgets that + // ship localized captions (DataGrid2 nl_NL). + if len(pd.Translations) > 0 { + vt["Translations"] = buildTranslationsArray(pd.Translations) + } + } + } + } + } + for _, val := range v { + walk(val) + } + case []any: + for _, item := range v { + walk(item) + } + } + } + walk(tmpl.Type) + + if len(changedTypeIDs) == 0 { + return + } + // Reset each Object WidgetValue whose PropertyType's Type changed, so the Object + // matches the reconciled schema. The WidgetProperty's TypePointer references the + // PropertyType $ID (IDs are remapped later, so they still match at this stage). + var resetWalk func(any) + resetWalk = func(node any) { + switch v := node.(type) { + case map[string]any: + if v["$Type"] == "CustomWidgets$WidgetProperty" { + if tp, _ := v["TypePointer"].(string); tp != "" { + if pd, ok := changedTypeIDs[tp]; ok { + if val, ok := v["Value"].(map[string]any); ok { + resetPropertyValue(val, pd) + } + } + } + } + for _, child := range v { + resetWalk(child) + } + case []any: + for _, item := range v { + resetWalk(item) + } + } + } + resetWalk(tmpl.Object) +} + +// completeValueTypeEnvelope walks a widget Type and ensures every WidgetValueType carries +// the generic metamodel envelope fields Studio Pro's current version always serializes but +// an older extracted template may predate. These are widget-agnostic defaults (mxbuild +// emits them on every ValueType regardless of widget), so they are added wherever absent +// and never overwrite an explicit value. +func completeValueTypeEnvelope(node any) { + switch v := node.(type) { + case map[string]any: + if v["$Type"] == "CustomWidgets$WidgetValueType" { + if _, ok := v["AllowUpload"]; !ok { + v["AllowUpload"] = false + } + } + for _, val := range v { + completeValueTypeEnvelope(val) + } + case []any: + for _, item := range v { + completeValueTypeEnvelope(item) + } + } +} + +// buildReturnType builds a CustomWidgets$WidgetReturnType from a .mpk expression +// property's declared . The placeholder $ID is remapped to a +// fresh UUID by the loader's ID phase. +func buildReturnType(typ string) map[string]any { + return map[string]any{ + "$ID": placeholderID(), + "$Type": "CustomWidgets$WidgetReturnType", + "AssignableTo": "", + "EntityProperty": "", + "IsList": false, + "Type": typ, + } +} + +// buildTranslationsArray builds a ValueType.Translations list (leading Mendix array +// marker 2 followed by CustomWidgets$WidgetTranslation entries) from a .mpk property's +// declared . Placeholder $IDs are remapped by the loader's ID phase. +func buildTranslationsArray(trans []mpk.Translation) []any { + arr := []any{float64(2)} + for _, t := range trans { + arr = append(arr, map[string]any{ + "$ID": placeholderID(), + "$Type": "CustomWidgets$WidgetTranslation", + "LanguageCode": t.Lang, + "Text": t.Text, + }) + } + return arr +} + +// buildAllowedTypesArray builds a ValueType.AllowedTypes list (leading Mendix array +// marker 1 followed by the allowed Mendix type names) from a .mpk property's declared +// attributeTypes. Mirrors the construction in createDefaultValueType. +func buildAllowedTypesArray(types []string) []any { + arr := []any{float64(1)} + for _, t := range types { + arr = append(arr, t) + } + return arr +} + // mpkEnumValuesByKey indexes a widget's enumeration option sets by property key, // across both top-level and nested (object-list) properties. func mpkEnumValuesByKey(def *mpk.WidgetDefinition) map[string][]mpk.EnumValue { diff --git a/modelsdk/widgets/augment_metadata_test.go b/modelsdk/widgets/augment_metadata_test.go index f7e924906..e18c12ada 100644 --- a/modelsdk/widgets/augment_metadata_test.go +++ b/modelsdk/widgets/augment_metadata_test.go @@ -82,6 +82,70 @@ func TestReconcilePropertyMetadata_LeavesUnknownKeys(t *testing.T) { } } +// TestReconcileValueTypesFromMPK verifies the schema-derived ValueType reconciliation +// that closes the large-version-jump within-key CE0463 (issue #600 / DataGrid2@3.10.0): +// a Type change (from a wrong-typed exemplar clone) rewrites the ValueType Type AND +// resets the matching Object WidgetValue; the mutually-exclusive type-specific fields are +// normalized to the .mpk type (EnumerationValues cleared on a non-enum, ReturnType built +// only for Expression and cleared otherwise). +func TestReconcileValueTypesFromMPK(t *testing.T) { + pt := func(id, key, vtType string, extra map[string]any) map[string]any { + vt := map[string]any{"$Type": "CustomWidgets$WidgetValueType", "Type": vtType} + for k, v := range extra { + vt[k] = v + } + return map[string]any{"$Type": "CustomWidgets$WidgetPropertyType", "$ID": id, "PropertyKey": key, "ValueType": vt} + } + tmpl := &WidgetTemplate{ + Type: map[string]any{"ObjectType": map[string]any{"PropertyTypes": []any{float64(2), + // stale Enumeration clone for a key that is really a textTemplate + pt("pt1", "lbl", "Enumeration", map[string]any{"EnumerationValues": []any{float64(2), + map[string]any{"$Type": "CustomWidgets$WidgetEnumerationValue", "_Key": "none", "Caption": "None"}}}), + pt("pt2", "expr", "Expression", map[string]any{"ReturnType": nil}), + // stale ReturnType on a widgets-typed property + pt("pt3", "pag", "Widgets", map[string]any{"ReturnType": map[string]any{"$Type": "CustomWidgets$WidgetReturnType", "Type": "String"}}), + }}}, + Object: map[string]any{"Properties": []any{float64(2), + map[string]any{"$Type": "CustomWidgets$WidgetProperty", "TypePointer": "pt1", + "Value": map[string]any{"$Type": "CustomWidgets$WidgetValue", "PrimitiveValue": "none", "TextTemplate": nil}}, + }}, + } + byKey := map[string]mpk.PropertyDef{ + "lbl": {Key: "lbl", Type: "textTemplate"}, + "expr": {Key: "expr", Type: "expression", ReturnType: "String"}, + "pag": {Key: "pag", Type: "widgets"}, + } + reconcileValueTypesFromMPK(tmpl, byKey) + + pts := tmpl.Type["ObjectType"].(map[string]any)["PropertyTypes"].([]any) + vtOf := func(i int) map[string]any { return pts[i].(map[string]any)["ValueType"].(map[string]any) } + + // lbl: retyped to TextTemplate, enum cleared to the empty [2] marker. + if got := vtOf(1)["Type"]; got != "TextTemplate" { + t.Errorf("lbl Type = %v, want TextTemplate", got) + } + if ev, _ := vtOf(1)["EnumerationValues"].([]any); len(ev) != 1 { + t.Errorf("lbl EnumerationValues = %v, want empty [2] marker", vtOf(1)["EnumerationValues"]) + } + // lbl Object value reset: TextTemplate structure present, PrimitiveValue cleared. + objVal := tmpl.Object["Properties"].([]any)[1].(map[string]any)["Value"].(map[string]any) + if objVal["TextTemplate"] == nil { + t.Errorf("lbl Object value not reset: TextTemplate still nil after retype") + } + if objVal["PrimitiveValue"] != "" { + t.Errorf("lbl Object PrimitiveValue = %v, want cleared", objVal["PrimitiveValue"]) + } + // expr: ReturnType built for Expression. + rt, ok := vtOf(2)["ReturnType"].(map[string]any) + if !ok || rt["Type"] != "String" { + t.Errorf("expr ReturnType = %v, want WidgetReturnType{Type:String}", vtOf(2)["ReturnType"]) + } + // pag: stale ReturnType cleared on a non-Expression type. + if got := vtOf(3)["ReturnType"]; got != nil { + t.Errorf("pag ReturnType = %v, want nil (widgets type)", got) + } +} + // TestReorderPropertyTypes verifies the top-level PropertyTypes are reordered to the // installed .mpk's declaration order (leading array marker preserved, keys absent from // the .mpk kept after the declared ones), closing the order axis of the object-list diff --git a/modelsdk/widgets/mpk/mpk.go b/modelsdk/widgets/mpk/mpk.go index 20512f712..6cca36acd 100644 --- a/modelsdk/widgets/mpk/mpk.go +++ b/modelsdk/widgets/mpk/mpk.go @@ -27,8 +27,10 @@ type PropertyDef struct { IsList bool IsSystem bool // true for elements DataSource string // dataSource attribute reference + ReturnType string // for expression properties: the Mendix type AllowedTypes []string // for attribute properties: Mendix type names ("String", "Decimal", etc.) EnumValues []EnumValue // for enumeration properties: the declared options (key + caption) + Translations []Translation // widget-shipped caption/template translations () Children []PropertyDef // nested properties for object-type properties } @@ -38,6 +40,12 @@ type EnumValue struct { Caption string } +// Translation is one localized caption/template string of a widget property. +type Translation struct { + Lang string + Text string +} + // WidgetDefinition holds the parsed definition of a pluggable widget from an .mpk file. type WidgetDefinition struct { ID string // e.g. "com.mendix.widget.web.combobox.Combobox" @@ -114,10 +122,24 @@ type xmlProperty struct { Description string `xml:"description"` AttributeTypes []xmlAttributeType `xml:"attributeTypes>attributeType"` EnumValues []xmlEnumValue `xml:"enumerationValues>enumerationValue"` + ReturnType xmlReturnType `xml:"returnType"` + Translations []xmlTranslation `xml:"translations>translation"` // Nested properties for object type NestedProps []xmlPropGroup `xml:"properties>propertyGroup"` } +// xmlReturnType represents on an expression property. +type xmlReturnType struct { + Type string `xml:"type,attr"` +} + +// xmlTranslation represents Text — a widget-shipped +// caption/template translation. +type xmlTranslation struct { + Lang string `xml:"lang,attr"` + Text string `xml:",chardata"` +} + // xmlEnumValue represents Caption. type xmlEnumValue struct { Key string `xml:"key,attr"` @@ -266,18 +288,25 @@ func walkPropertyGroup(pg xmlPropGroup, parentCategory string, def *WidgetDefini Caption: p.Caption, Description: p.Description, Category: category, - Required: p.Required == "true", + // Mendix pluggable-widget spec: `required` defaults to true when the + // attribute is absent (mxbuild's update-widgets emits Required=true for + // every property that omits required=, e.g. DataGrid2 showContentAs). Only + // an explicit required="false" is optional. Defaulting missing→false here + // caused within-key CE0463 drift on augment-added keys (issue #600). + Required: p.Required != "false", DefaultValue: p.DefaultValue, IsList: p.IsList == "true", DataSource: p.DataSource, + ReturnType: p.ReturnType.Type, AllowedTypes: allowedTypes, EnumValues: enumValues, + Translations: toTranslations(p.Translations), } // Parse nested properties for object-type properties if p.Type == "object" && len(p.NestedProps) > 0 { for _, npg := range p.NestedProps { - collectNestedProperties(npg, &prop) + collectNestedProperties(npg, &prop, "") } } @@ -300,8 +329,17 @@ func walkPropertyGroup(pg xmlPropGroup, parentCategory string, def *WidgetDefini } // collectNestedProperties extracts child properties from nested propertyGroups -// within an object-type property and appends them to the parent PropertyDef. -func collectNestedProperties(pg xmlPropGroup, parent *PropertyDef) { +// within an object-type property and appends them to the parent PropertyDef. The +// property group's caption chain becomes each child's Category (joined with "::", +// mirroring walkPropertyGroup) — mxbuild derives nested categories the same way, so a +// missing category here is a within-key CE0463 drift on augment-added nested props. +func collectNestedProperties(pg xmlPropGroup, parent *PropertyDef, parentCategory string) { + category := pg.Caption + if parentCategory != "" && category != "" { + category = parentCategory + "::" + category + } else if parentCategory != "" { + category = parentCategory + } for _, p := range pg.Properties { var allowedTypes []string for _, at := range p.AttributeTypes { @@ -318,25 +356,49 @@ func collectNestedProperties(pg xmlPropGroup, parent *PropertyDef) { Type: p.Type, Caption: p.Caption, Description: p.Description, - Required: p.Required == "true", + Category: category, + // Mendix pluggable-widget spec: `required` defaults to true when the + // attribute is absent (mxbuild's update-widgets emits Required=true for + // every property that omits required=, e.g. DataGrid2 showContentAs). Only + // an explicit required="false" is optional. Defaulting missing→false here + // caused within-key CE0463 drift on augment-added keys (issue #600). + Required: p.Required != "false", DefaultValue: p.DefaultValue, IsList: p.IsList == "true", DataSource: p.DataSource, + ReturnType: p.ReturnType.Type, AllowedTypes: allowedTypes, EnumValues: enumValues, + Translations: toTranslations(p.Translations), } // Nested object-type properties can themselves contain object lists. if p.Type == "object" && len(p.NestedProps) > 0 { for _, npg := range p.NestedProps { - collectNestedProperties(npg, &child) + collectNestedProperties(npg, &child, "") } } parent.Children = append(parent.Children, child) } for _, sub := range pg.SubGroups { - collectNestedProperties(sub, parent) + collectNestedProperties(sub, parent, category) + } +} + +// toTranslations converts parsed XML elements to Translation records, +// trimming caption whitespace (the XML pretty-prints chardata with indentation). +func toTranslations(xts []xmlTranslation) []Translation { + if len(xts) == 0 { + return nil + } + out := make([]Translation, 0, len(xts)) + for _, xt := range xts { + if xt.Lang == "" { + continue + } + out = append(out, Translation{Lang: xt.Lang, Text: strings.TrimSpace(xt.Text)}) } + return out } // FindMPK looks in the project's widgets/ directory for an .mpk matching the widgetID. From 31fbde8610917987cd7309c9812d37b66490b15d Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 20 Jul 2026 06:29:31 +0000 Subject: [PATCH 14/37] =?UTF-8?q?docs(proposal):=20#600=20instance=20last-?= =?UTF-8?q?mile=20=E2=80=94=20Object=20rebuild=20rejected,=20editor-logic?= =?UTF-8?q?=20boundary?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Corrects the earlier (wrong) "only mxbuild can produce the envelope" conclusion: the WidgetType IS generically reproducible and is now byte-identical to update-widgets output (committed c7fe714). Records the instance-side investigation: - The residual CE0463 (definition matching) is textTemplate default-template instantiation in the WidgetObject. Even a minimal DataGrid2 reproduces it. - Which textTemplates get a default template is config-conditional applicability (aria labels always; clearSelectionButtonLabel/loadMoreButtonCaption/ singleSelectionColumnLabel only when their feature is active) — a distinction mxbuild derives from the widget's editor code, NOT the declarative XML (the null-vs-populated property defs are structurally identical). always-populate closes 9->3, Required-gate 3->9; neither matches because the rule isn't declarative. - The Object-from-definition rebuild was tried and REJECTED: it discards the byte-exact extracted template Object and regressed bundled DataGrid2 0->1 on both 11.12.0 and 10.24. Path forward for the instance last-mile: v2-safe update-widgets (#764) on top of mxcli's now-correct definition — the applicability logic lives in the widget's compiled editor code, which mxbuild executes and mxcli cannot. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- ...ROPOSAL_multi_version_pluggable_widgets.md | 105 +++++++++++------- 1 file changed, 62 insertions(+), 43 deletions(-) diff --git a/docs/11-proposals/PROPOSAL_multi_version_pluggable_widgets.md b/docs/11-proposals/PROPOSAL_multi_version_pluggable_widgets.md index a24eb396d..aecda2984 100644 --- a/docs/11-proposals/PROPOSAL_multi_version_pluggable_widgets.md +++ b/docs/11-proposals/PROPOSAL_multi_version_pluggable_widgets.md @@ -307,49 +307,68 @@ regenerate gen from the target version's reflection-data. because that was a *small* version delta; a big jump exposes per-field schema evolution (`Type`, `Required`, `AllowedTypes`, `Translations`) that key-level augment never touches. - **RESOLVED — path (a) is structurally impossible; only mxbuild can produce the - envelope.** Full end-to-end reproduction of the #600 stack (fresh 11.12.0 project, DW - 3.10.0 swapped in over the bundled 3.4.0, DataGrid2 `dg` authored with the fixed binary, - isolated from the Atlas template pages) with an order-independent, `$ID`/`TypePointer`-masked - before/after-`update-widgets` subtree diff of `dg` alone shows the residual drift is - dominated by fields that **do not exist in the widget XML source at all**: - - | Residual `dg` drift after augment | count | in `Datagrid.xml`? | - |---|---|---| - | `AllowUpload` (absent → `false`) | 105 | **no** — not in source | - | `Required` (`false` → `true`) | 54 | partially — XML has **3** `required="true"`, mxbuild emits **54** | - | `DesignProperties`, `LabelTemplate` | several | **no** — not in source | - | `ReturnType` (`null` → `{Type,IsList,…}`), `Type`, `DefaultValue` | ~30 | computed/partial | - - The nested-column **PropertyKey sets are already identical** before/after (augment adds - every new 3.10.0 key — `exportType`, `exportNumberFormat`, `exportDateFormat`, … — - correctly). What augment *cannot* produce is the mxbuild-**computed** BSON envelope: - `AllowUpload`/`DesignProperties`/`LabelTemplate` appear **nowhere** in `Datagrid.xml`, and - `Required` is computed (3 declared vs 54 emitted). Since these fields are not derivable - from the `.mpk`, **no parser/template/augment approach — including path (a), full-`ValueType` - reconciliation — can ever reproduce a version-faithful `WidgetType`.** (A speculative - `reconcileValueTypeSchema` was implemented and reverted: setting `Required` from the XML - `required` attr would mis-set the 51 mxbuild-computed keys to `false`.) `update-widgets` - on the same copy takes `dg` from CE0463 → **0** — confirming only mxbuild owns the envelope. - - **Revised path forward — delegate the envelope to mxbuild, made v2-safe (was path (d)):** - - **(c) v2-safe `update-widgets` as the CE0463 remediation.** The augment layer already - closes *moderate* drift (Gallery@10.24, bundled DW) and that stays. For *large* jumps, - the correct fix is to run mxbuild's own `update-widgets` — which is exactly what the - skills now route users to (`mxcli docker check`/`build`, post the earlier skill-guidance - fix). The remaining blocker is that bare `mx update-widgets` destroys MPRv2 (deletes - `mprcontents/`, converts to v1 — issue #763), which **mendixlabs PR #764** fixes. - Adopt/port #764 so `mxcli docker check`/`build` (and the warm local loop) can reconcile - widgets without downgrading the project. This is the only path that is faithful for - *arbitrary* widget versions, because only mxbuild computes the envelope. - - **(b) Per-version templates** — extract a clean `datagrid-.json` per Data Widgets - release. Rejected as a primary fix: doesn't scale (3.11.2 already shipped) and still - wouldn't carry the computed `Required`; kept only as a note. - - Remaining: implement (c) (v2-safe `update-widgets` via #764), keep the moderate-drift - augment as-is, and fold the matrix into `scripts/widget-version-matrix.sh` as a standing - gate (feasible now — `mxcli marketplace download` fetches any widget version for the - fixture) asserting *bundled*-DW authoring stays at 0 CE0463. + **RESOLVED (definition side) — the `WidgetType` IS generically reproducible; an + earlier "only mxbuild can produce the envelope" conclusion was WRONG.** `update-widgets` + is fully generic (no per-widget knowledge), so everything it emits is derivable from the + package + the generic metamodel/spec defaults — the "not in `Datagrid.xml`" fields are + generic defaults, not widget-specific computation: + - `AllowUpload` (on **all 105** ValueTypes) is a generic `WidgetValueType` metamodel + default (`false`) — emitted for every ValueType, widget-agnostic. + - `Required` "computed 3→54" is just the **pluggable-widget spec default**: `required` + defaults to **true** when the attribute is absent. The `.mpk` parser had the wrong + default (missing→false); fixed. + - `Type`/`DefaultValue` drift was augment cloning new keys from **wrong-typed exemplars** + (our bug), plus stale template values — fixed by reconciling each matched ValueType's + `Type`/`Required`/`DefaultValue`/`AllowedTypes`/`IsList`/`ReturnType`/`Translations` + from the `.mpk` and normalizing the mutually-exclusive type-specific fields. + - `Translations`/`ReturnType`/nested `Category`/nested order — all parsed from the + widget XML (``, ``, propertyGroup captions) and emitted. + + **Committed (`c7fe714`): the emitted `CustomWidgetType` is byte-identical (canonical, + `$ID`/`TypePointer`-masked) to `update-widgets` output for the #600 stack (DW 3.10.0 / + Mendix 11.12.0) — definition diff 326 → 0 — with no regression on bundled DW across + 11.12.0 / 11.10.0 / 10.24.** The generic-reconciliation thesis holds for the definition. + + **NOT resolved (instance side) — the last-mile Object default-template instantiation is + config-conditional applicability that lives in the widget's editor code, not the + declarative package.** With the definition matching, the residual CE0463 is entirely in + the `WidgetObject`: a handful of `textTemplate` properties whose default template + (`Forms$ClientTemplate` with the shipped caption translations) mxbuild instantiates in + the instance. Even a **minimal** DataGrid2 (Selection:None, one column) reproduces it — + definition identical, only ~9 `textTemplate` default-templates differ, `update-widgets` → + 0. **Which textTemplates get a default template is config-dependent**: aria/status labels + are always instantiated, but `clearSelectionButtonLabel` / `loadMoreButtonCaption` / + `singleSelectionColumnLabel` stay `null` because their feature is off — a distinction + mxbuild derives from the widget's **editor applicability logic**, not from anything in + `Datagrid.xml` (the property defs of the null-vs-populated ones are structurally + identical). Empirically: an "always-populate" rule closes 9→3 but over-populates the 3 + feature-gated ones (still CE0463); a `Required`-gate closes 3→9 the other way. Neither + declarative rule matches, because the rule isn't declarative. + + **Object-from-definition rebuild — tried, REJECTED (regresses the common case).** + Regenerating the whole `WidgetObject` from the reconciled definition (one WidgetValue per + PropertyType, defaults built from each ValueType) makes the count match (127=127) and is + architecturally clean, but it **discards the byte-exact extracted template Object** — and + because it can't replicate the editor-applicability rule, it **regressed bundled DataGrid2 + from 0 → 1 CE0463 on both 11.12.0 and 10.24**. The extracted template Object is the best + available source for the no-drift case; a rebuild trades that away for an approximation. + Reverted. + + **Path forward for the instance last-mile:** + - **(c) v2-safe `update-widgets` finish, on top of mxcli's correct definition.** The + applicability logic that decides instance defaults lives in the widget's compiled editor + code, which **mxbuild executes and mxcli (pure Go) cannot**. So the instance last-mile is + genuinely mxbuild's job: run its `update-widgets` to finalize instance defaults, made + v2-safe (bare `update-widgets` destroys MPRv2 — issue #763; **mendixlabs PR #764** fixes + it). This is *not* a retreat from the generic thesis — the definition is now fully + generic in mxcli; only the editor-logic-dependent instance defaults need mxbuild. + - **(b) Per-version templates** — rejected: doesn't scale and still wouldn't carry the + config-conditional instance defaults. + + Remaining: implement (c) (v2-safe `update-widgets` via #764) for the instance last-mile, + keep the committed definition reconciliation, and fold the matrix into + `scripts/widget-version-matrix.sh` as a standing gate (feasible now — `mxcli marketplace + download` fetches any widget version) asserting *bundled*-DW authoring stays at 0 CE0463. **Also noted (dirty-template audit):** `datagrid.json`'s Object carries a configured delete `Forms$ActionButton` (`Forms$DeleteClientAction` + `Atlas_Core.Atlas.trash-can`) From 515e24bd56beb5f27276da8d8c4cd659ff7aa26e Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 20 Jul 2026 12:45:34 +0000 Subject: [PATCH 15/37] =?UTF-8?q?docs(proposal):=20#600=20=E2=80=94=20conf?= =?UTF-8?q?irm=20editorConfig.js=20is=20the=20instance-default=20mechanism?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Upgrades the instance last-mile finding from inference to confirmed, with the concrete evidence: the widget .mpk ships Datagrid.editorConfig.js whose getProperties() calls hidePropertyIn/hidePropertiesIn conditionally on the instance's values. The verbatim hides for the three always-null properties are present ("Multi"!==selection hides clearSelectionButtonLabel; "loadMore"!== pagination hides loadMoreButtonCaption; singleSelectionColumnLabel conditional), mapping 1:1 onto the measured null-vs-populated pattern. Always-populated props (selectRowLabel, cancelExportLabel) appear 0x in editorConfig.js. A hidden property does not get its default template instantiated — so the applicability rule is imperative JS keyed on instance config, not declarative XML. Adds path (d): execute editorConfig.js in-process via a Go JS engine (goja) as a research spike for a no-mxbuild-dependency path; kept deferred behind the recommended (c) v2-safe update-widgets (#764). Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- ...ROPOSAL_multi_version_pluggable_widgets.md | 54 ++++++++++++++----- 1 file changed, 41 insertions(+), 13 deletions(-) diff --git a/docs/11-proposals/PROPOSAL_multi_version_pluggable_widgets.md b/docs/11-proposals/PROPOSAL_multi_version_pluggable_widgets.md index aecda2984..d711b4f5d 100644 --- a/docs/11-proposals/PROPOSAL_multi_version_pluggable_widgets.md +++ b/docs/11-proposals/PROPOSAL_multi_version_pluggable_widgets.md @@ -338,12 +338,31 @@ regenerate gen from the target version's reflection-data. definition identical, only ~9 `textTemplate` default-templates differ, `update-widgets` → 0. **Which textTemplates get a default template is config-dependent**: aria/status labels are always instantiated, but `clearSelectionButtonLabel` / `loadMoreButtonCaption` / - `singleSelectionColumnLabel` stay `null` because their feature is off — a distinction - mxbuild derives from the widget's **editor applicability logic**, not from anything in - `Datagrid.xml` (the property defs of the null-vs-populated ones are structurally - identical). Empirically: an "always-populate" rule closes 9→3 but over-populates the 3 - feature-gated ones (still CE0463); a `Required`-gate closes 3→9 the other way. Neither - declarative rule matches, because the rule isn't declarative. + `singleSelectionColumnLabel` stay `null` because their feature is off. Empirically: an + "always-populate" rule closes 9→3 but over-populates the 3 feature-gated ones (still + CE0463); a `Required`-gate closes 3→9 the other way. Neither declarative rule matches, + because the rule isn't declarative. + + **CONFIRMED mechanism — the widget's compiled `editorConfig.js` decides applicability.** + The `.mpk` ships `Datagrid.editorConfig.js` (16 KB) alongside `Datagrid.xml`. Its + `getProperties(values, defaultProperties)` function calls `hidePropertyIn` (24×) / + `hidePropertiesIn` (6×) / `changePropertyIn` conditionally on the *instance's* current + values. The exact hides for the three null properties are present verbatim: + + ```js + "Multi" !== r && hidePropertiesIn(e, t, ["selectionCounterPosition","clearSelectionButtonLabel","enableSelectAll"]) + "loadMore" !== e.pagination && hidePropertyIn(t, e, "loadMoreButtonCaption") + hidePropertyIn(e, t, "singleSelectionColumnLabel") // conditional + ``` + + This maps 1:1 onto the measurements: our minimal grid has `Selection:None` (≠ "Multi") → + `clearSelectionButtonLabel` hidden → `null`; `Pagination:buttons` (≠ "loadMore") → + `loadMoreButtonCaption` hidden → `null`. The always-populated properties + (`selectRowLabel`, `cancelExportLabel`) appear **0×** in `editorConfig.js` — never hidden, + so their default is instantiated. **A hidden property does not get its default template + instantiated in the Object.** The applicability graph is imperative JS keyed on the + instance config, not declarative XML — which is why no `.mpk` parsing reproduces it, and + why the null-vs-populated property *definitions* in `Datagrid.xml` are byte-identical. **Object-from-definition rebuild — tried, REJECTED (regresses the common case).** Regenerating the whole `WidgetObject` from the reconciled definition (one WidgetValue per @@ -355,13 +374,22 @@ regenerate gen from the target version's reflection-data. Reverted. **Path forward for the instance last-mile:** - - **(c) v2-safe `update-widgets` finish, on top of mxcli's correct definition.** The - applicability logic that decides instance defaults lives in the widget's compiled editor - code, which **mxbuild executes and mxcli (pure Go) cannot**. So the instance last-mile is - genuinely mxbuild's job: run its `update-widgets` to finalize instance defaults, made - v2-safe (bare `update-widgets` destroys MPRv2 — issue #763; **mendixlabs PR #764** fixes - it). This is *not* a retreat from the generic thesis — the definition is now fully - generic in mxcli; only the editor-logic-dependent instance defaults need mxbuild. + - **(c) v2-safe `update-widgets` finish, on top of mxcli's correct definition + (recommended).** The applicability logic that decides instance defaults is the widget's + compiled `editorConfig.js`, which **mxbuild executes (it embeds a JS runtime for widget + editors) and mxcli (pure Go) does not**. So the instance last-mile is genuinely + mxbuild's job: run its `update-widgets` to finalize instance defaults, made v2-safe + (bare `update-widgets` destroys MPRv2 — issue #763; **mendixlabs PR #764** fixes it). + This is *not* a retreat from the generic thesis — the definition is now fully generic in + mxcli; only the editor-logic-dependent instance defaults need mxbuild. + - **(d) Execute `editorConfig.js` in-process via a Go JS engine (research spike).** Now + that the mechanism is confirmed, a pure-Go path exists in principle: embed a JS + interpreter (`goja`), load each widget's `Datagrid.editorConfig.js`, and run + `getProperties(instanceValues, defaultProperties)` with a shim implementing the + widget-editor API (`hidePropertyIn`, `hidePropertiesIn`, `changePropertyIn`, …). The + resulting visibility set then drives which defaults are instantiated. Feasible but a + substantial subsystem, and it makes mxcli execute third-party widget JS; worthwhile only + if mxcli must author widgets with **no** mxbuild dependency. Defer behind (c). - **(b) Per-version templates** — rejected: doesn't scale and still wouldn't carry the config-conditional instance defaults. From 4b8c4f5ee88151d27addae9d12f3ed62e8073809 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 20 Jul 2026 13:39:52 +0000 Subject: [PATCH 16/37] feat(widgets): lift editorConfig.js visibility rules to close DataGrid2 CE0463 (#574) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Automates what the hand-transcribed widgetVisibilityRules table did for VideoPlayer/Timeline: a static extractor lifts a widget's compiled editorConfig.js property-hide logic (hidePropertyIn/hidePropertiesIn) into declarative WidgetVisibilityRules, which the builder's existing ApplyPropertyVisibility consumes to null the TextTemplate of properties the widget hides under the current config. This is the instance last-mile that the definition reconciliation (c7fe714) could not reach — the applicability logic lives in the widget's editor JS, not the declarative .mpk XML. Extractor (mdl/executor/editorconfig_extract.go): - Recognizes the dominant getProperties idioms: "V"===/!==ref && hide, ref && hide, ref || hide, ref ? hide : … — with scoped alias resolution (var r=e.selection) so minified single-letter identifiers don't leak across functions. - Skips object-list-nested hides and compound/ternary-nested guards (boundary check), degrading safely to "not hidden" rather than emitting a partial rule. - Reports coverage stats. On the real DW 3.10.0 Datagrid.editorConfig.js: 9/28 hide calls lifted (12 nested, 7 complex) — including the three that drive #600. Wiring: built-in widgets (DataGrid2/Gallery) skip .def.json generation, so rules are resolved on the fly from the project's installed .mpk editorConfig at build time (mpk.ReadEditorConfig + resolveWidgetVisibilityRules, cached), then applied. Object-side textTemplate defaults are populated with the .mpk's shipped caption translations (fixes CE4899 required-textTemplate) and the visibility pass nulls the hidden ones afterward. Result: minimal DataGrid2 on Mendix 11.12.0 + Data Widgets 3.10.0 (the #600 stack) → 0 errors (was CE0463). No regression on bundled Data Widgets across 11.12.0 / 11.10.0 / 10.24. Remaining: the full repro's nested textfilter widgets (separate widget type) still show 1 CE0463 — same mechanism, follow-up. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JXnEgoc2NQP1Y2TWMCMXC4 --- mdl/backend/widgetobj/builder.go | 1 + mdl/executor/editorconfig_extract.go | 428 ++++++++++++++++++ mdl/executor/editorconfig_extract_test.go | 116 +++++ .../testdata/datagrid.editorConfig.js | 1 + mdl/executor/widget_engine.go | 13 +- modelsdk/widgets/augment.go | 34 +- modelsdk/widgets/mpk/mpk.go | 100 +++- 7 files changed, 676 insertions(+), 17 deletions(-) create mode 100644 mdl/executor/editorconfig_extract.go create mode 100644 mdl/executor/editorconfig_extract_test.go create mode 100644 mdl/executor/testdata/datagrid.editorConfig.js diff --git a/mdl/backend/widgetobj/builder.go b/mdl/backend/widgetobj/builder.go index bfca48302..bef30809c 100644 --- a/mdl/backend/widgetobj/builder.go +++ b/mdl/backend/widgetobj/builder.go @@ -607,6 +607,7 @@ func (ob *Builder) EnsureRequiredObjectLists() { // TextTemplate. The widget's current primitive values (read from the assembled // object) drive rule evaluation, so a rule keyed on e.g. `type` sees the value // just set from MDL. + func (ob *Builder) ApplyPropertyVisibility(rules []types.WidgetVisibilityRule) { if len(rules) == 0 { return diff --git a/mdl/executor/editorconfig_extract.go b/mdl/executor/editorconfig_extract.go new file mode 100644 index 000000000..74a9e8e6a --- /dev/null +++ b/mdl/executor/editorconfig_extract.go @@ -0,0 +1,428 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "path/filepath" + "regexp" + "strings" + "sync" + + "github.com/mendixlabs/mxcli/mdl/types" + "github.com/mendixlabs/mxcli/modelsdk/widgets/mpk" +) + +// editorConfigExtractStats reports how much of a widget's editorConfig.js the +// extractor could lift into declarative WidgetVisibilityRules — the coverage +// number that tells `check`/serialization how far to trust the rules and when +// to fall back to mxbuild's update-widgets. +type editorConfigExtractStats struct { + TotalHideCalls int // hidePropertyIn + hidePropertiesIn call sites seen + Recognized int // lifted into a top-level WidgetVisibilityRule + SkippedNested int // object-list-nested (e.g. hidePropertyIn(...,"columns",n,"key")) + SkippedComplex int // ternary/compound guard, or an alias we couldn't resolve +} + +// hideCallRE locates a hidePropertyIn / hidePropertiesIn call and captures its +// (balanced-enough) argument list. hideNestedPropertiesIn is deliberately not +// matched — it only ever targets object-list items (Phase 2). +var hideCallRE = regexp.MustCompile(`hidePropert(?:y|ies)In\(`) + +// aliasAssignRE finds `IDENT=OBJ.PROP` (a `var x=e.selection`-style alias). +// Resolution is scoped to the enclosing function body (see enclosingAliases), +// because minified editorConfig reuses single-letter identifiers across scopes. +var aliasAssignRE = regexp.MustCompile(`([A-Za-z_$][\w$]*)=([A-Za-z_$][\w$]*)\.([A-Za-z_$][\w$]*)`) + +// stringLitRE matches a double-quoted JS string literal (no escapes in the +// property/enum keys we care about). +var stringLitRE = regexp.MustCompile(`"([^"\\]*)"`) + +// visibilityCache memoizes extracted rules per (projectPath, widgetID) so the +// editorConfig.js is parsed once per build session, not per widget instance. +var ( + visibilityCache = map[string][]types.WidgetVisibilityRule{} + visibilityCacheMu sync.Mutex +) + +// resolveWidgetVisibilityRules returns the property-visibility rules for a +// widget, lifted from its installed .mpk's editorConfig.js. Used to enrich +// built-in widget definitions (DataGrid2, Gallery, …) — which the .def.json +// generator skips — with the version-specific applicability logic of the Data +// Widgets package actually installed in the project. Returns nil when the .mpk +// or its editor config can't be found (degrades to "no rules" → template +// default, exactly today's behaviour). Best-effort: see extractVisibilityRules- +// FromJS for coverage limits. +func resolveWidgetVisibilityRules(projectPath, widgetID string) []types.WidgetVisibilityRule { + if projectPath == "" || widgetID == "" { + return nil + } + key := projectPath + "\x00" + widgetID + visibilityCacheMu.Lock() + if r, ok := visibilityCache[key]; ok { + visibilityCacheMu.Unlock() + return r + } + visibilityCacheMu.Unlock() + + // getProjectPath() yields the .mpr file path; FindMPK wants the directory + // that contains widgets/. + projectDir := projectPath + if strings.EqualFold(filepath.Ext(projectDir), ".mpr") { + projectDir = filepath.Dir(projectDir) + } + + var rules []types.WidgetVisibilityRule + if mpkPath, err := mpk.FindMPK(projectDir, widgetID); err == nil && mpkPath != "" { + if js, err := mpk.ReadEditorConfig(mpkPath, widgetID); err == nil && js != "" { + rules, _ = extractVisibilityRulesFromJS(js) + } + } + + visibilityCacheMu.Lock() + visibilityCache[key] = rules + visibilityCacheMu.Unlock() + return rules +} + +// extractVisibilityRulesFromJS lifts top-level property-hide rules from a +// widget's compiled editorConfig.js into declarative WidgetVisibilityRules. +// +// It recognizes the dominant `getProperties` idioms — `"V"===ref && hide(...)`, +// `"V"!==ref && hide(...)`, `ref && hide(...)`, `ref || hide(...)`, and +// `ref ? hide(...) : …` — where `ref` is `obj.prop` or a locally-aliased +// identifier resolved within the enclosing function scope. Everything it cannot +// lift (object-list-nested hides, compound/computed guards, unresolved aliases) +// is counted in the returned stats so callers can gauge coverage; unrecognized +// hides simply produce no rule, which degrades safely to "not hidden". +func extractVisibilityRulesFromJS(js string) ([]types.WidgetVisibilityRule, editorConfigExtractStats) { + var rules []types.WidgetVisibilityRule + var stats editorConfigExtractStats + seen := map[string]bool{} // dedupe propertyKey+condition + + for _, loc := range hideCallRE.FindAllStringIndex(js, -1) { + stats.TotalHideCalls++ + callStart, argsOpen := loc[0], loc[1] // argsOpen points just past '(' + args, ok := balancedArgs(js, argsOpen) + if !ok { + stats.SkippedComplex++ + continue + } + keys, nested := hideTargetKeys(args) + if nested { + stats.SkippedNested++ + continue + } + if len(keys) == 0 { + stats.SkippedComplex++ + continue + } + cond, ok := parseGuard(js, callStart) + if !ok { + stats.SkippedComplex++ + continue + } + stats.Recognized++ + for _, key := range keys { + sig := key + "\x00" + cond.PropertyKey + cond.Operator + cond.Value + if seen[sig] { + continue + } + seen[sig] = true + c := cond // copy per rule + rules = append(rules, types.WidgetVisibilityRule{PropertyKey: key, HiddenWhen: &c}) + } + } + return rules, stats +} + +// hideTargetKeys returns the property key(s) a hide call targets and whether the +// call is object-list-nested (which we skip). A top-level hidePropertyIn has a +// single string arg (the key); a top-level hidePropertiesIn has one array of +// string keys. A `"columns"`/`"..."`-prefixed string arg followed by more args +// marks the nested form. +func hideTargetKeys(args string) (keys []string, nested bool) { + parts := splitTopLevelCommas(args) + // Collect string-literal positional args and any array literal. + var stringArgs []string + var arrayKeys []string + for _, p := range parts { + p = strings.TrimSpace(p) + if strings.HasPrefix(p, "[") { + for _, m := range stringLitRE.FindAllStringSubmatch(p, -1) { + arrayKeys = append(arrayKeys, m[1]) + } + continue + } + if m := stringLitRE.FindStringSubmatch(p); m != nil && strings.HasPrefix(p, `"`) { + stringArgs = append(stringArgs, m[1]) + } + } + if len(arrayKeys) > 0 { + // hidePropertiesIn(obj, obj, [keys]) — nested if a leading string arg + // (e.g. "columns") also appears. + if len(stringArgs) > 0 { + return nil, true + } + return arrayKeys, false + } + switch len(stringArgs) { + case 1: + return stringArgs, false // hidePropertyIn(obj, obj, "key") + case 0: + return nil, false + default: + return nil, true // "columns","key" etc. — nested + } +} + +// parseGuard reads the guard expression immediately preceding a hide call and +// converts it to a WidgetVisibilityCondition. callStart points at the hide +// function name; the connector just before it is `&&`, `||`, or `?`. +func parseGuard(js string, callStart int) (types.WidgetVisibilityCondition, bool) { + // Strip an optional `_.` namespace prefix before the function name. + end := callStart + pre := strings.TrimRight(js[:end], " ") + if strings.HasSuffix(pre, "_.") { + pre = pre[:len(pre)-2] + } + // Identify the connector. + var falsy bool // || connector or !prefix ⇒ hide when guard is falsy + switch { + case strings.HasSuffix(pre, "&&"): + pre = pre[:len(pre)-2] + case strings.HasSuffix(pre, "?"): + pre = pre[:len(pre)-1] + case strings.HasSuffix(pre, "||"): + pre = pre[:len(pre)-2] + falsy = true + default: + return types.WidgetVisibilityCondition{}, false + } + guard, boundary := lastGuardExpr(pre) + if guard == "" { + return types.WidgetVisibilityCondition{}, false + } + // Skip guards nested inside a larger expression. A clean statement-level guard + // is bounded by a statement separator (`,`, `;`, `{`, or start-of-input); a + // boundary of `(`/`?`/`:`/`&`/`|` means the guard is one operand of a compound + // or ternary condition (e.g. `"web"===r ? (e.advanced || hide(...))`), of which + // we'd capture only a fragment — producing a WRONG rule that over-fires. Better + // to emit no rule (→ "not hidden" → template default), which is safe. + switch boundary { + case 0, ',', ';', '{': + // clean + default: + return types.WidgetVisibilityCondition{}, false + } + aliases := enclosingAliases(js, callStart) + return guardToCondition(guard, falsy, aliases) +} + +// lastGuardExpr returns the balanced expression ending at the end of `pre`, +// bounded by the previous top-level separator, and the boundary byte that +// terminated it (0 for start-of-input). The boundary lets the caller reject +// guards nested inside a compound/ternary expression. +func lastGuardExpr(pre string) (string, byte) { + depth := 0 + var boundary byte + i := len(pre) - 1 + for ; i >= 0; i-- { + c := pre[i] + switch c { + case ')', ']', '}': + depth++ + case '(', '[', '{': + if depth == 0 { + boundary = c + goto done + } + depth-- + case ',', ';', ':', '?', '&', '|': + if depth == 0 { + boundary = c + goto done + } + } + } +done: + return strings.TrimSpace(pre[i+1:]), boundary +} + +var ( + eqCmpRE = regexp.MustCompile(`^"([^"]*)"===([A-Za-z_$][\w$.]*)$`) + neCmpRE = regexp.MustCompile(`^"([^"]*)"!==([A-Za-z_$][\w$.]*)$`) + eqCmpRE2 = regexp.MustCompile(`^([A-Za-z_$][\w$.]*)==="([^"]*)"$`) + neCmpRE2 = regexp.MustCompile(`^([A-Za-z_$][\w$.]*)!=="([^"]*)"$`) + refRE = regexp.MustCompile(`^(!?)([A-Za-z_$][\w$.]*)$`) +) + +// guardToCondition parses a single guard expression into a visibility +// condition, resolving a bare identifier through the scope's alias map. +func guardToCondition(guard string, falsy bool, aliases map[string]string) (types.WidgetVisibilityCondition, bool) { + // "V" === ref / ref === "V" + if m := eqCmpRE.FindStringSubmatch(guard); m != nil { + if key, ok := resolveRef(m[2], aliases); ok { + return types.WidgetVisibilityCondition{PropertyKey: key, Operator: "eq", Value: m[1]}, true + } + return types.WidgetVisibilityCondition{}, false + } + if m := eqCmpRE2.FindStringSubmatch(guard); m != nil { + if key, ok := resolveRef(m[1], aliases); ok { + return types.WidgetVisibilityCondition{PropertyKey: key, Operator: "eq", Value: m[2]}, true + } + return types.WidgetVisibilityCondition{}, false + } + if m := neCmpRE.FindStringSubmatch(guard); m != nil { + if key, ok := resolveRef(m[2], aliases); ok { + return types.WidgetVisibilityCondition{PropertyKey: key, Operator: "ne", Value: m[1]}, true + } + return types.WidgetVisibilityCondition{}, false + } + if m := neCmpRE2.FindStringSubmatch(guard); m != nil { + if key, ok := resolveRef(m[1], aliases); ok { + return types.WidgetVisibilityCondition{PropertyKey: key, Operator: "ne", Value: m[2]}, true + } + return types.WidgetVisibilityCondition{}, false + } + // bare ref (truthy) or !ref (falsy), combined with the connector polarity: + // ref && hide → hide when ref truthy + // ref || hide → hide when ref falsy (falsy==true here) + // !ref && hide → hide when ref falsy + // ref ? hide:… → hide when ref truthy + if m := refRE.FindStringSubmatch(guard); m != nil { + key, ok := resolveRef(m[2], aliases) + if !ok { + return types.WidgetVisibilityCondition{}, false + } + neg := (m[1] == "!") + wantFalsy := falsy != neg // XOR: || or ! flips polarity (both flips cancel) + op := "truthy" + if wantFalsy { + op = "falsy" + } + return types.WidgetVisibilityCondition{PropertyKey: key, Operator: op}, true + } + return types.WidgetVisibilityCondition{}, false +} + +// resolveRef turns a guard reference into a widget property key: `obj.prop` +// yields `prop`; a bare identifier is looked up in the scope alias map. A bare +// identifier with no alias (e.g. a computed local) is unresolvable. +func resolveRef(ref string, aliases map[string]string) (string, bool) { + if i := strings.LastIndexByte(ref, '.'); i >= 0 { + return ref[i+1:], true + } + if key, ok := aliases[ref]; ok { + return key, true + } + return "", false +} + +// enclosingAliases returns the `ident → property` aliases declared in the +// function body that encloses the hide call at pos, resolved by scanning back +// to the nearest unbalanced `{`. Scoping matters: minified editorConfig reuses +// identifiers like `r`/`n` across functions, so only same-scope `var r=e.prop` +// assignments are trustworthy. +func enclosingAliases(js string, pos int) map[string]string { + // Walk back to the enclosing block's opening brace. + depth := 0 + open := 0 + for i := pos - 1; i >= 0; i-- { + switch js[i] { + case '}', ')', ']': + depth++ + case '{', '(', '[': + if depth == 0 { + open = i + goto found + } + depth-- + } + } +found: + body := js[open:pos] + aliases := map[string]string{} + ambiguous := map[string]bool{} + for _, m := range aliasAssignRE.FindAllStringSubmatch(body, -1) { + ident, prop := m[1], m[3] + if ambiguous[ident] { + continue + } + if existing, ok := aliases[ident]; ok && existing != prop { + delete(aliases, ident) + ambiguous[ident] = true + continue + } + aliases[ident] = prop + } + return aliases +} + +// balancedArgs returns the argument-list text between the '(' just before +// `open` and its matching ')'. Handles nested (), [], {} and string literals. +func balancedArgs(js string, open int) (string, bool) { + depth := 1 + inStr := byte(0) + for i := open; i < len(js); i++ { + c := js[i] + if inStr != 0 { + if c == '\\' { + i++ + continue + } + if c == inStr { + inStr = 0 + } + continue + } + switch c { + case '"', '\'', '`': + inStr = c + case '(', '[', '{': + depth++ + case ')', ']', '}': + depth-- + if depth == 0 { + return js[open:i], true + } + } + } + return "", false +} + +// splitTopLevelCommas splits an argument list on commas that are not nested +// inside (), [], {}, or string literals. +func splitTopLevelCommas(args string) []string { + var parts []string + depth := 0 + inStr := byte(0) + start := 0 + for i := 0; i < len(args); i++ { + c := args[i] + if inStr != 0 { + if c == '\\' { + i++ + continue + } + if c == inStr { + inStr = 0 + } + continue + } + switch c { + case '"', '\'', '`': + inStr = c + case '(', '[', '{': + depth++ + case ')', ']', '}': + depth-- + case ',': + if depth == 0 { + parts = append(parts, args[start:i]) + start = i + 1 + } + } + } + parts = append(parts, args[start:]) + return parts +} diff --git a/mdl/executor/editorconfig_extract_test.go b/mdl/executor/editorconfig_extract_test.go new file mode 100644 index 000000000..0cbc209c2 --- /dev/null +++ b/mdl/executor/editorconfig_extract_test.go @@ -0,0 +1,116 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "os" + "testing" + + "github.com/mendixlabs/mxcli/mdl/types" +) + +// find returns the first rule for the given property key, or nil. +func findRule(rules []types.WidgetVisibilityRule, key string) *types.WidgetVisibilityRule { + for i := range rules { + if rules[i].PropertyKey == key { + return &rules[i] + } + } + return nil +} + +// TestExtractVisibility_DataGrid runs the extractor against the real +// (minified) Data Widgets 3.10.0 DataGrid2 editorConfig.js and asserts it lifts +// the three top-level textTemplate hides that drive the #600 CE0463 — the ones +// the hand-transcribed table never covered — with the correct conditions. +func TestExtractVisibility_DataGrid(t *testing.T) { + js, err := os.ReadFile("testdata/datagrid.editorConfig.js") + if err != nil { + t.Fatalf("read fixture: %v", err) + } + rules, stats := extractVisibilityRulesFromJS(string(js)) + t.Logf("coverage: %d/%d hide calls recognized (nested=%d complex=%d), %d rules", + stats.Recognized, stats.TotalHideCalls, stats.SkippedNested, stats.SkippedComplex, len(rules)) + + want := []struct { + key, condKey, op, val string + }{ + {"clearSelectionButtonLabel", "itemSelection", "ne", "Multi"}, + {"singleSelectionColumnLabel", "itemSelection", "ne", "Single"}, + {"loadMoreButtonCaption", "pagination", "ne", "loadMore"}, + } + for _, w := range want { + r := findRule(rules, w.key) + if r == nil || r.HiddenWhen == nil { + t.Errorf("%s: no rule extracted", w.key) + continue + } + c := r.HiddenWhen + if c.PropertyKey != w.condKey || c.Operator != w.op || c.Value != w.val { + t.Errorf("%s: got {%s %s %q}, want {%s %s %q}", + w.key, c.PropertyKey, c.Operator, c.Value, w.condKey, w.op, w.val) + } + } + + // A couple more expected top-level lifts (non-textTemplate, but proves the + // idioms generalize): emptyPlaceholder hidden when showEmptyPlaceholder=="none". + if r := findRule(rules, "emptyPlaceholder"); r == nil || r.HiddenWhen == nil || + r.HiddenWhen.PropertyKey != "showEmptyPlaceholder" || r.HiddenWhen.Operator != "eq" || r.HiddenWhen.Value != "none" { + t.Errorf("emptyPlaceholder rule missing/wrong: %+v", r) + } +} + +// TestExtractVisibility_VideoPlayerPattern reproduces the hand-transcribed +// VideoPlayer rule: `"expression"===e.type && hidePropertiesIn(t,e,["videoUrl","posterUrl"])`. +func TestExtractVisibility_VideoPlayerPattern(t *testing.T) { + js := `exports.getProperties=function(e,t){"expression"===e.type&&_.hidePropertiesIn(t,e,["videoUrl","posterUrl"]);return t}` + rules, _ := extractVisibilityRulesFromJS(js) + for _, key := range []string{"videoUrl", "posterUrl"} { + r := findRule(rules, key) + if r == nil || r.HiddenWhen == nil || r.HiddenWhen.PropertyKey != "type" || + r.HiddenWhen.Operator != "eq" || r.HiddenWhen.Value != "expression" { + t.Errorf("%s: got %+v, want hidden when type==expression", key, r) + } + } +} + +// TestExtractVisibility_TimelinePattern reproduces the hand-transcribed Timeline +// rule, which uses a ternary truthy guard: +// `e.customVisualization ? hidePropertiesIn(t,e,["title","description"]) : x`. +func TestExtractVisibility_TimelinePattern(t *testing.T) { + js := `exports.getProperties=function(e,t){e.customVisualization?_.hidePropertiesIn(t,e,["title","description","timeIndication"]):null;return t}` + rules, _ := extractVisibilityRulesFromJS(js) + for _, key := range []string{"title", "description", "timeIndication"} { + r := findRule(rules, key) + if r == nil || r.HiddenWhen == nil || r.HiddenWhen.PropertyKey != "customVisualization" || + r.HiddenWhen.Operator != "truthy" { + t.Errorf("%s: got %+v, want hidden when customVisualization truthy", key, r) + } + } +} + +// TestExtractVisibility_ScopedAlias proves alias resolution is scoped: the same +// identifier `r` aliased to different properties in two functions must not leak. +func TestExtractVisibility_ScopedAlias(t *testing.T) { + js := `f1=function(e,t){var r=Object.keys(e);return r};` + + `f2=function(e,t){var r=t.itemSelection;"Multi"!==r&&_.hidePropertyIn(e,t,"clearSelectionButtonLabel")}` + rules, _ := extractVisibilityRulesFromJS(js) + r := findRule(rules, "clearSelectionButtonLabel") + if r == nil || r.HiddenWhen == nil || r.HiddenWhen.PropertyKey != "itemSelection" || + r.HiddenWhen.Operator != "ne" || r.HiddenWhen.Value != "Multi" { + t.Fatalf("scoped alias not resolved: %+v", r) + } +} + +// TestExtractVisibility_SkipsNested confirms object-list-nested hides +// (hidePropertyIn(...,"columns",n,"key")) are not lifted as top-level rules. +func TestExtractVisibility_SkipsNested(t *testing.T) { + js := `g=function(e,t){e.columns.forEach(function(r,n){e.columnsSortable||_.hidePropertyIn(t,e,"columns",n,"sortable")})}` + rules, stats := extractVisibilityRulesFromJS(js) + if findRule(rules, "sortable") != nil { + t.Error("nested column hide must not produce a top-level rule") + } + if stats.SkippedNested == 0 { + t.Error("expected the nested hide to be counted as skipped") + } +} diff --git a/mdl/executor/testdata/datagrid.editorConfig.js b/mdl/executor/testdata/datagrid.editorConfig.js new file mode 100644 index 000000000..a62245e7e --- /dev/null +++ b/mdl/executor/testdata/datagrid.editorConfig.js @@ -0,0 +1 @@ +"use strict";function e(t){return e="function"==typeof Symbol&&"symbol"==typeof Symbol.iterator?function(e){return typeof e}:function(e){return e&&"function"==typeof Symbol&&e.constructor===Symbol&&e!==Symbol.prototype?"symbol":typeof e},e(t)}function t(e,t){var r=Object.keys(e);if(Object.getOwnPropertySymbols){var n=Object.getOwnPropertySymbols(e);t&&(n=n.filter(function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable})),r.push.apply(r,n)}return r}function r(e){for(var r=1;re.length)&&(t=e.length);for(var r=0,n=Array(t);r0&&void 0!==arguments[0]?arguments[0]:"";try{return e.split(";").reduce(function(e,t){var r=t.split(":");2===r.length&&(e[r[0].trim().replace(/(-.)/g,function(e){return e[1].toUpperCase()})]=r[1].trim());return e},{})}catch(e){return{}}},m}var g,v,P={},w={};function C(){return g||(g=1,Object.defineProperty(w,"__esModule",{value:!0})),w}function S(){return v||(v=1,function(e){var t=P&&P.__createBinding||(Object.create?function(e,t,r,n){void 0===n&&(n=r);var o=Object.getOwnPropertyDescriptor(t,r);o&&!("get"in o?!t.__esModule:o.writable||o.configurable)||(o={enumerable:!0,get:function(){return t[r]}}),Object.defineProperty(e,n,o)}:function(e,t,r,n){void 0===n&&(n=r),e[n]=t[r]}),r=P&&P.__exportStar||function(e,r){for(var n in e)"default"===n||Object.prototype.hasOwnProperty.call(r,n)||t(r,e,n)};Object.defineProperty(e,"__esModule",{value:!0}),r(C(),e)}(P)),P}var O,F,A,j={},D={};function I(){if(O)return D;function e(e,r,n,o,i){t(function(e,t,r){return r.splice(t,1)},e,n,o,i)}function t(e,r,n,o,i){r.forEach(function(r){var a;r.propertyGroups&&t(e,r.propertyGroups,n,o,i),null===(a=r.properties)||void 0===a||a.forEach(function(r,a,u){r.key===n&&(void 0===o||void 0===i?e(r,a,u):r.objects?t(e,r.objects[o].properties,i):r.properties&&t(e,r.properties[o],i))})})}return O=1,Object.defineProperty(D,"__esModule",{value:!0}),D.hidePropertyIn=e,D.hidePropertiesIn=function(e,r,n){n.forEach(function(r){return t(function(e,t,r){return r.splice(t,1)},e,r,void 0,void 0)})},D.hideNestedPropertiesIn=function(t,r,n,o,i){i.forEach(function(i){return e(t,r,n,o,i)})},D.changePropertyIn=function(e,r,n,o,i,a){t(n,e,o,i,a)},D.transformGroupsIntoTabs=function(e){var t=[];e.forEach(function(e){e.propertyGroups&&(t.push.apply(t,i(e.propertyGroups)),e.propertyGroups=[])}),e.push.apply(e,t)},D.moveProperty=function(e,t,r){e>=0&&t>=0&&e