Feature/reconcile - #386
Open
awagner-mainz wants to merge 35 commits into
Open
awagner-mainz wants to merge 35 commits into
awagner-mainz wants to merge 35 commits into
Conversation
… lookup Add the generic OpenRefine Reconciliation API connector as a fallback alongside the existing GND connector for the person register, in both annotate-tei.html and annotate-jats.html's authority-config template (previously only present in the orphaned annotate-authorities.html demo file). Endpoint is decoupled/configurable, defaulting here to this repo's own local "reconcile" demo app.
… caveat New doc/reconcile.xml explains the reconcile profile's server-side Reconciliation Service API (endpoints, adding it to an app, customizing the entity/property model), cross-linked from the Reference Manual's "Extending the API" section and the annotation editor docs. The Reconciliation Service connector docs claimed it "cannot be used as a data provider for the Custom connector" / "it is not possible to use [it]... in the context of the Custom connector" - stale since 2021; Custom's createConnectors() has always dispatched on nested connectors generically, with no special-casing, and this has been actively exercised since 2023. Replaced both caveats with a positive example, and fixed connector="Reconciliation" -> the actual attribute value "ReconciliationService" recognized by connectors.js (a separate, pre-existing inaccuracy in the same examples).
…ency Matches reconcile/config.json dropping "registers" from depends: the "view" field's redirect is now checked against the app's actual installed-profiles list at request time, falling back to a built-in preview otherwise. Update the extends example, endpoint description, and add a "view" customization subsection covering the new function-item override.
…e-filtering Matches reconcile/modules/reconcile-config.xql's new "labels" and "fulltext-search" fields and reconcile-scoring.xql's edit-distance fallback tier.
Matches reconcile/modules/reconcile-conditions.xql - a query can now disambiguate/refine via property conditions or match directly by id, not just a plain name, reusing the same "properties" already defined for /extend.
Mirrors the new "Production hardening" section in profiles/reconcile/doc/README.md: request-size caps, reverse-proxy rate limiting, and opt-in auth via x-constraints.
…tributes The Reconciliation Service connector section didn't warn that an unrecognized `connector` attribute value silently falls back to the unrelated Metagrid connector, with no error - a real, easy-to-hit footgun (e.g. `connector="Reconciliation"` instead of the exact `connector="ReconciliationService"`; the docs' own prior example used the wrong, non-"Service" name). Add a caution note pointing at the exact, case-sensitive value required and the concrete symptom (queries going to api.metagrid.ch) so this is diagnosable from the docs alone next time. Also document the connector's `type` and `limit` attributes (verified against the actual reconciliation.js source, not assumed), and that results are now labeled with the connector's `prefix` when set, to disambiguate multiple reconciliation-style authorities in the same app. reconcile.xml: note that malformed batch entries (e.g. a blank spreadsheet cell serialized as JSON null) degrade to an empty query rather than failing or misaligning the whole batch, preserving the spec's required positional correspondence between queries and results.
…ibutes Generalize the entity-authority editor to write any number of a selected match's properties into separate output attributes instead of always writing just the id into one hardcoded field. annotations.js's authoritySelected() now fans out a properties object rather than assuming a single bare ref string; tei-annotation-config.tpl.xqm copies whatever attributes the client sent instead of hardcoding [$key]. Demo config in annotate-tei.html shows an example person mapping (escaped preferred name as @key, id as @ref, GND identifier fetched via /extend as a third attribute). Also documents this, plus the suggest-endpoint type-ahead and extend-sourcing behavior, in the annotations doc page. Co-Authored-By: Claude Sonnet 5 <[email protected]>
…on UI Document that OpenRefine's own reconciliation client only understands the 0.2 manifest field names (extend.propose_properties.service_url/ service_path, suggest.property.service_url/service_path) for its "Add columns from reconciled values" dialog - it has no code path for 1.0-draft's boolean-only equivalents, so no manifest content on our side can make that feature work against the default 1.0-draft manifest. Recommends adding the service with ?version=0.2 instead of mixing 0.2-only fields into the spec-pure 1.0-draft manifest. Also documents the new /suggest/flyout endpoint and why it exists (a related OpenRefine client quirk around the 0.2 spec's flyout_service_path field). Co-Authored-By: Claude Sonnet 5 <[email protected]>
New subsection under "Customizing what gets reconciled" explaining that a property's "value" is a plain function item with full XQuery/collection access, not restricted to the matched entity's own subtree - covers the mechanism, a verified worked example (average title length of documents mentioning a person, via this demo's persName/@key convention), and performance guidance (index the lookup or cache the aggregate once it's actually a bottleneck, same options already used for fulltext-search). Co-Authored-By: Claude Sonnet 5 <[email protected]>
…y list, images Explains that /preview now rewrites every link/image URL to an absolute one regardless of how relative the underlying ODD transform's own output is, why that was needed (the shipped connector injects the fetched HTML into an existing page via innerHTML rather than loading it as its own document, so a relative link resolved against the wrong page), and that the default preview now lists every configured property with a value, rendering image-shaped values as thumbnails (image: true()/false() to override the guess). Co-Authored-By: Claude Sonnet 5 <[email protected]>
…ain preview overflow Root cause of a bug reported 2026-07-24 (real repro: a user-created persName ref="kbga-actors-329" key="Rade-Martin-1857-1940" showed "Entity not found" in its detail popup, while an old-style persName key="kbga-actors-27" - id directly in @key, no @ref - worked fine): pb-view-annotate's own detail-lookup reads the entity id from whichever annotation attribute its "key"/"key-map" config names, default "key". That was correct under the OLD single-attribute convention this profile shipped before this session's field-mapping feature, where @key always held the id - but a fields="key=label,ref=id,..." mapping (the profile's own shipped "person" example, see annotate-tei.html) repurposes @key for the *label* instead, with the id moving to @ref. The lookup never got told about that: it kept reading @key, tried to look up a label-slug as if it were an id, and correctly reported "not found". Fixed with a "person": "ref" entry in config.json's features.annotate.configs.tei.keyMap, wired onto <pb-view-annotate key-map="..."> in annotate.html - pb-view-annotate.js's own getKey(type) already supported this per-type override, it just wasn't configured. Other types (place/organization/work) keep the old key=id default unaffected, since they don't use the new field-mapping convention. Also fixes a second, unrelated, previously-latent bug this investigation surfaced: annotate.css's containment rule for the entity-info preview area used an ID selector ("#authority-info") that never matched anything - the template only ever sets class="authority-info" - so the intended max-height/overflow:auto never applied. Harmless while preview content was a line or two; once the /preview response started listing every configured property (see the reconcile profile's own preview enhancement, same investigation), the unbounded growth pushed the reconciliation search results below the visible viewport, making a pre-existing Cypress test intermittently/consistently fail on a "clipped by an overflow: hidden ancestor" visibility check - not a regression in the search itself, just a long-dormant CSS selector typo finally mattering. Fixed the selector to match the actual class. Also caps the info area of the small "click to view" Tippy popup, which had no size bound at all, for the same reason. One new Cypress regression test reproduces the original bug against a real, previously-user-created live entity (sermons/27003.xml's "Rade" annotation) rather than a synthetic fixture. Full suite re-verified: 25 XQSuite, 43 Cypress API+GUI (was intermittently failing 1 before either CSS fix, now green with no retries needed). Co-Authored-By: Claude Sonnet 5 <[email protected]>
…ce/work Only "person" actually used fields="key=label,ref=id,..." and the matching keyMap override - organization/place/work still used the implicit default (key=id, the old single-attribute convention), and so had not hit the "Entity not found" bug fixed for person, but also weren't getting the richer escaped-name-as-key/id-as-ref split. Registry.buildProperties (tei-publisher-components) already made this mechanism generic across every connector, including "Custom" (which these three types use, wrapping GND/GeoNames as a fallback) - only this config needed catching up. Adds fields="key=label,ref=id" to the outer Custom connector for organization/place/work (skipping an extend:-sourced property for any of them: neither Custom's own local-register results nor its GND/ GeoNames fallbacks currently support real data-extension fetching, so an extend: mapping would just silently never resolve - unlike person's gnd=extend:gnd, which is meaningful there because it can be nested with a real ReconciliationService connector). Adds matching "ref" entries for all three types to config.json's keyMap. Verified live: a real "place" search-and-select (Castilla -> Madrid) correctly writes key="Madrid"/ref="geo-3117735" via the shared buildProperties mechanism, and a place linked via @ref correctly shows its real preview in the click-to-view detail popup instead of "Entity not found" - confirming the keyMap fix generalizes, not just the mechanism. See the annotate_select_save_not_verified project memory for a separate, pre-existing gap this investigation surfaced (not fixed here): Cypress's forced-click re-selection of an already-annotated span doesn't reliably persist to the document, for any type - worked around in the new regression test by setting up the linked state directly via XQuery instead. Co-Authored-By: Claude Sonnet 5 <[email protected]>
Explains why moving the id off the reference/key attribute (e.g. fields="key=label,ref=id") needs a matching keyMap entry in the app's own config.json, or the click-to-view detail popup shows "Entity not found" for an otherwise perfectly valid annotation - the bug fixed 2026-07-25, now documented so the next person configuring a custom fields mapping doesn't have to rediscover it. Co-Authored-By: Claude Sonnet 5 <[email protected]>
Adds a hidden <input name="link"> to the shared annotation-form template (alongside the existing ref/gnd ones) and extends organization/place/ work's fields= to "key=label,ref=id,link=extend:link", exercising tei-publisher-components' newly-added GND.fetchExtend/GeoNames. fetchExtend (an external reference URL fetched from that connector's own record) - the GND/GeoNames equivalent of person's existing "gnd=extend:gnd" sourced from a nested ReconciliationService. Also the fix half of this session's `jinks update --reinstall` verification: this profile's Jinks-registered copy had never actually been kept in sync with local edits all session (only the live app was patched) - a real regeneration silently reverted every annotate fix from today. Deployed every changed annotate profile file to /db/apps/jinks/profiles/annotate/... to match the pattern already used for the reconcile profile, then re-verified with --reinstall that everything (keyMap, fields=, the CSS fix) now correctly survives a clean regeneration. See the jinks_profile_sync_and_annotate_template_gotchas project memory for the full story, including a second, separate finding (templates/pages/annotate.html is a base template never meant to be requested directly - use annotate-tei.html, fixed in the test suite separately). Co-Authored-By: Claude Sonnet 5 <[email protected]>
Person's Custom connector wraps both a GND connector and a nested ReconciliationService, and a single search can surface a match from either one - the field mapping only resolves consistently across both because of a matching fix on the tei-publisher-components side (GND.getRecord() now aliases its "link" property as "gnd" too). Expand the doc comment to explain why, so a future edit doesn't "simplify" gnd=extend:gnd to extend:link and reintroduce the bug. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01GJ79F6UanhKHMctVm12ATN
Full read-through pass over every annotate-profile file touched during this project (config wiring for connectors/fields/keyMap per document type), aimed at a reader unfamiliar with the session history. Clarifies that the server-side $anno:reference-key/anno:get-key (always literal @key) and the client-side keyMap (per-type, drives the editor's popup) are two separate mechanisms; moves the person-specific "why gnd=extend:gnd needs GND.js's dual alias" explanation to sit directly above the person connector block instead of a shared comment above all four types. No logic/markup/config value changes. Flags one thing worth a follow-up, not fixed here (comments-only pass): anno:occurrences groups by literal @key, which for a field-mapped type now holds a slugified label rather than a stable id - two different people with the same name would incorrectly count as the same entity's occurrences. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01GJ79F6UanhKHMctVm12ATN
…dd screenshots - annotations.xml: update the keyMap example to all four field-mapped types (was person-only, stale since organization/place/work were added); soften the "Entity not found" caution now that pb-view-annotate.js falls back to @key when a type's configured id attribute is missing; add a tip on the Custom connector explaining that an extend:propertyId field source is resolved by property id alone, so every nested connector needs to expose a value under the same name (the concrete reason GND.js aliases "link" as "gnd" for person). - reconcile.xml / annotations.xml: add real screenshots (captured live against tp-reconc) for the service manifest, the annotate editor's federated search results (local + GND badges), and the click-to-view detail popup - the reconcile-specific docs previously had none, unlike the rest of annotations.xml's extensive ann-*.png screenshots. Verified via the docs site's own content-fetch API and a full page render after syncing to the running apps/tei-publisher app. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01GJ79F6UanhKHMctVm12ATN
The third issue flagged during the recent comment-audit pass: for a field-mapped type (see annotate-tei.html's `fields=` attributes), an entity's id lands in @ref while @key holds a slugified label instead - anno:occurrences only ever matched @key, so it undercounted to zero for any field-mapped type's occurrence badge in the search results list. NOTE: this app's live annotation-config.xqm (imported directly by annotations.xql) is actually a separate, flat, doctype-agnostic module bundled in the existdb/teipublisher base image, not generated from this file - the dispatcher that's supposed to delegate to it (annotation-config.tpl.xqm) was never registered with Jinks. This fix is applied here as the correct source-of-truth for when that gets wired up properly, and was additionally live-patched directly into the deployed flat module so the running demo app actually reflects it today - see the annotation_config_dispatcher_dormant project memory for the full investigation and why fully swapping in the dispatcher wasn't attempted here (its anno:annotations has a different, incompatible arity from what's actually deployed). Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01GJ79F6UanhKHMctVm12ATN
…in tests Three of the annotate profile's own shipped Cypress tests (api/annotations.cy.js, gui/annotate.cy.js, gui/custom-annotate.cy.js) all called cy.uploadXml(path, xmlContent) - an old 2-arg convention that no longer matches commands.js's actual signature, cy.uploadXml(url, filename, xml, opts) (multipart, POSTs to an upload endpoint). Every one of these tests therefore failed in its own before()/beforeEach() hook (405 posting to the literal fixture path) before ever reaching its real assertions - meaning this test coverage, including 11 real regression cases for /api/annotations/merge (delete+wrap, nested notes, choice/abbr/expan, app/lem/rdg, standOff/listAnnotation presence), had likely never actually run/passed in this project's context. Fixed all three call sites to a direct REST PUT instead, matching this project's own established fixture-deployment pattern. Also fixed gui/annotate.cy.js's separate, already-known-stale `?template=annotate.html` (base template, empty blocks) to `annotate-tei.html` (see the jinks_profile_sync_and_annotate_template_gotchas project memory). Found while thoroughly re-verifying the annotation-config dispatcher fix (see prior commits + the annotation_config_dispatcher_dormant project memory) - this is exactly the previously-missing save-path coverage that investigation needed and didn't have. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01GJ79F6UanhKHMctVm12ATN
…ences The "Configuring Annotations" section (and the "Custom Connector" section's local-authority-file note, and the authority-config location note earlier in the page) still told readers to edit modules/annotation-config.xqm directly and templates/pages/annotate.html - both wrong now that the per-doctype dispatch split (annotation-config.tpl.xqm delegating to tei-annotation-config.xqm, annotate.html as an empty base template extended by annotate-tei.html) is actually wired up and working, not dormant. Six reference fixes total, plus a caution note that the walkthrough's toolbar HTML/class names (header-toolbar, icon-group) predate TEI Publisher 10's current page-template layout and may not match annotate-tei.html verbatim - flagged rather than rewritten, since verifying/rewriting the full walkthrough is a separate, larger task. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01GJ79F6UanhKHMctVm12ATN
annotation-blocks.html's toggle link built its URL from $context?doc?type,
which comes back empty in this app's actually-deployed base10 (drifted from
this repo's own history, same class of issue as the
annotation_config_dispatcher_dormant finding) - producing a broken
"template=annotate-.html" link. Compute the doctype directly from the
reliably-present $context?doc?content node instead.
forms/config.json never set features.forms.enabled, the flag
forms-blocks.html gates fore.js/fore.css loading on - so the annotation
editor's <fx-fore> elements never upgraded and raw Fore markup (JS bodies,
instance data, a stuck "Cannot save to local register" message) rendered as
plain visible text. Confirmed via a Playwright check
(customElements.get('fx-fore') was undefined) and fixed by setting the flag.
Found while verifying the reconciliation demo Docker image against a fully
clean rebuild.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01GJ79F6UanhKHMctVm12ATN
The annotation toolbar's buttons (person, save, undo, ...) rendered as blank pills - their <svg><use href="#person-fill"> markup needs the icon sprite's <symbol> defs present somewhere in the document. annotate.html's frontmatter already declared theme.icons: ["annotate-icons.svg"], the convention base10's layouts/base.html is meant to consume (the same pattern metadata-editor uses successfully) - but this app's actually- deployed base10 has no such include loop at all (confirmed by fetching the live templates/layouts/base.html: no icon-related include, unlike its menu.html/toolbar.html ones). Fixed pragmatically: annotate's own content-top template now directly includes the sprite itself, independent of base10 support. Left the config.json theme.icons declaration in place too since it's harmless and correct for any base10 that does implement the loop. Found while re-verifying the reconciliation demo Docker image after fixing the XQDY0025 crash that was previously masking this bug. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01GJ79F6UanhKHMctVm12ATN
…panel
The annotation editor's right-hand panel (search/edit forms, occurrences,
commit) looked completely broken - misshapen "cards", missing borders,
oversized empty space around normal-sized icons, invisible labels. Root
cause: annotate.css's `fx-fore fx-group { display: grid; ... }` rule
(specificity 0,0,2) that drives each card's grow/shrink animation was
losing the cascade to @jinntec/fore's own generic `[relevant] { display:
block; }` rule (specificity 0,1,0, wins ties regardless of load order
since attribute selectors always outrank type selectors) - every group
Fore marks relevant="" silently fell back to block flow instead of the
grid layout the whole card design depends on.
Confirmed via Playwright: before the fix, a "card" fx-group computed to
display:block/height:36px while its own child header was 54px tall
(overflowing); after, the panel renders as a normal, properly laid out
form with visible borders, labels, and correctly sized icons.
Reproduced and fixed outside Docker first (a plain dev-container app on
the base image), then confirmed the same fix resolves it in the
reconciliation demo Docker image too - this bug predates and is unrelated
to that packaging work; it's been latent since the annotate profile
started depending on the "forms"/Fore integration (2026-02-18), just
never visually verified until now since the automated GUI regression
tests only assert functional behavior (request URLs, form values), not
CSS.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01GJ79F6UanhKHMctVm12ATN
Two more polish issues found by real use of the panel now that it
actually renders (see the CSS-specificity commit): rounded corners were
huge (90px, looking like speech bubbles) and there was no way to dismiss
an in-progress edit/annotation once opened.
Corners: this app's deployed base10 controls.css ships (undrifted from
the local tei-publisher-jinks history's own fix) a bare `[role="group"]
{ border-radius: var(--jinks-form-border-radius) }` rule (a 90px pill
meant for small toolbar segments, normally excluded via `:not(fx-group)`
- the deployed copy lacks that exclusion). Every fx-group card here also
carries role="group" (Fore's own convention), so it inherited the pill
radius instead of base.css's normal 1rem .card radius. Fixed the same way
as the earlier display:grid fix: fx-group[role="group"] (specificity
0,1,1) reliably beats the base10 rule (0,1,0).
Close button: added a small "x" (reusing the existing #dialog-close
sprite icon) to each of the four card headers (commit, annotation,
authority, occurrences), wired via <fx-trigger><button>...<fx-call
action="reset-panels"></fx-call></fx-trigger> - the exact same pattern
already used by the proven "save" button in the commit panel, calling the
same pre-existing reset-panels action every other panel-opening flow in
this file already relies on.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01GJ79F6UanhKHMctVm12ATN
Custom.query() (tei-publisher-components/src/authority/custom.js) concatenates nested-connector results in document order, so this profile's own reconciliation service now appears ahead of GND in the candidate list for person annotations, instead of after. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01GJ79F6UanhKHMctVm12ATN
Two bugs found via real interaction (both confirmed by the user, one initially misdiagnosed by me as a test-harness artifact before finding the real cause with a corrected element selector): 1. Clicking a card's close button set the right instance values (confirmed directly) but the [relevant]/[nonrelevant] DOM attributes never updated, leaving the panel visibly open. reset-panels' plain <fx-refresh> (Fore's default, unforced form) doesn't reliably re-render when invoked via <fx-call> from *inside* one of the very cards it's about to hide - a forced fx-fore.refresh(true) cleared it correctly every time in direct testing. Added force="" to both <fx-refresh> calls in annotate.html (reset-panels and pb-before-save) - a strict superset of the unforced form, safe for every existing caller. 2. Adding the close button as a 4th icon to headers that already held 3 (occurrences' mark-all/search-collection/save-all) tipped an unprotected flex row into squeezing its icons instead of the text label - confirmed directly: iron-icon width collapsing to 8px while height stayed 24px (visibly squashed, not just smaller). Added flex-shrink:0 to header icon buttons and let the label truncate instead, which degrades better than an illegible icon. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01GJ79F6UanhKHMctVm12ATN
Four more issues found by real use, all confirmed live before/after:
- Card headers ("Annotation Details", "Revision Description", ...) and
the annotation-form's plain <input> (not wrapped in <fx-control>, so it
never got that selector's border/background) were visually
indistinguishable from surrounding text - both now get a real title-bar
treatment and a visible bordered input respectively.
- The commit panel's Status <select> always rendered two blank options:
<option> can only reliably host text, not a nested custom element, and
<pb-i18n> inside it never rendered. Switched to data-i18n's plain
(no [attr] prefix) form, which pb-page.js applies via a direct
innerHTML set - works on any element, <option> included.
- The #form-save icon (and any other icon-only button outside a <header>)
suffered the same flex-squeeze already fixed for header icons.
Broadened the flex-shrink:0 rule from ".preview header" to ".preview"
generally instead of special-casing each container.
- The real bug: selecting a reconciliation match for a person already
present in the local KBGA register created a second, separate,
near-empty register entry under a "reconc-"-prefixed id instead of
resolving to the existing rich entry - confirmed live (the resulting
persName showed only its raw @key and "entity not found", while a
pre-existing mention of the literal same person, tagged with the
unprefixed id, showed full name/bio). Root cause: this profile's own
reconcile server matches directly against the same local register
(reconcile-config.xql's "person" entities are exactly
collection(...)/id(...)//tei:person[@xml:id]), so a candidate's id
already *is* the register's own xml:id - prefixing it was giving two
different-looking ids to the identical entity. Removed prefix="reconc"
from the ReconciliationService connector; verified end-to-end
(Custom.query()/select()/buildProperties()) that a match now correctly
resolves to the existing entry (POST .../kbga-actors-27 -> "found", no
duplicate) and produces ref="kbga-actors-27", not a new id.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01GJ79F6UanhKHMctVm12ATN
…w info You were right to suspect @key vs @ref. Two distinct, compounding bugs, both now confirmed fixed live: 1. Live "Entity not found" preview immediately after selecting a candidate. The visible, user-facing reference input (class="form-ref", the one whose "input" event triggers the live pb-authority-lookup preview) was named from features.annotate.configs.tei.key ("key", a single, type-agnostic value) - NOT from keyMap, which correctly maps person/organization/place/work to "ref". A separate <input type="hidden" name="ref"> already existed and silently received the real id correctly; the visible field instead received "key=label"'s human-readable slug (e.g. "Barth-Peter-1888-1940") and ran the live lookup against *that*, which 404s. Fixed by swapping which field carries the "form-ref" class/visible role: the one the user actually sees and that drives the live lookup is now literally named "ref" (matching pb-view-annotate's own default key - confirmed via its source: `this.key = 'ref'` - so this is correct for every type sharing this form, not just the four in keyMap). 2. Full entity info missing from an already-saved annotation's click/hover popup after a page reload - a *different* rendering path than (1): base10's generic teipublisher.odd (shared by persName/placeName/orgName/rs, not owned by this project) resolves an entity via `head((@key, @ref))` - @key if present, else @ref - the opposite priority from this app's own keyMap. With a real id in @ref but a human-readable slug in @key (from "key=label"), that expression picks the slug, fails to resolve it as an id, and falls back to displaying the bare slug string with no lookup at all - confirmed exactly reproducible/fixable by toggling @key's presence on an existing saved persName, live, before touching any source. Rather than overriding a shared, comparatively risky base10 ODD rule, removed the "key=label" mapping outright: with @key never written for these types, head((@key, @ref)) simply returns @ref, matching this app's convention. The existing "already-linked entity" Cypress regression test is unaffected - it builds its own test fixture directly via XQuery, independent of this fields= mapping. Verified end-to-end together: selecting "Barth, Peter (1888-1940)" now shows the real name/bio in the live preview immediately (no more "Entity not found"), and produces properties {ref: "kbga-actors-27"} with no "key" property at all. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01GJ79F6UanhKHMctVm12ATN
…odes
annotate.html's pb-commit dispatch called string() on the "message"
fx-property but not on "user", "export", or "status", so those carried
raw XPath attribute-node results into the CustomEvent detail object.
JSON.stringify() collapses such a node to "{}" (DOM/Fore node wrappers
expose no own enumerable properties), so the server received log.user/
log.status as an empty JSON object instead of a string or absent value.
tei-annotation-config.xqm's anno:extend-header() then tried to atomize
that empty map inside a `{...}` attribute value template
(who="{$log?user}" status="{$log?status}"), and eXist throws err:FOTY0013
"A function item other than an array cannot be atomized" - maps are
function items in XQuery 3.1, and only arrays are exempted from that
error. Reproduced the exact error via an ad-hoc query atomizing a bare
map{} the same way, then confirmed the client dispatch was the source.
Fixed at the client (string() on all four fx-property expr attributes)
and added anno:sanitize-log-value() server-side as defense in depth, so
a malformed request body can never 500 this endpoint regardless of what
a future caller sends. Verified against the live merge endpoint with the
exact crash-inducing payload (log.user/status as {}): now returns 200
with who=""/status="" instead of 500.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01GJ79F6UanhKHMctVm12ATN
…xpansion
The previous commit's doc comment on anno:sanitize-log-value used `{...}`
(backtick immediately followed by a curly brace) to describe an XQuery
attribute value template. That's not just markdown styling here - Jinks's
templates.xqm wraps an entire .tpl.xqm file's source in an eXist string
constructor and treats that exact character sequence as the start of a
real interpolation, regardless of whether it's inside a comment. Building
a new app (jinks create, i.e. every container's first boot) failed outright
with err:XPST0003 "unexpected token: ." while template-expanding this file,
so the demo container never started - a full regression, not just a
degraded feature.
Reworded the comment to avoid the sequence and added a warning against
reintroducing it. Swept both the reconcile and annotate profile's .tpl.*
files for the same pattern - no other occurrences. Verified end-to-end:
rebuilt docker/Dockerfile locally, the generator step (jinks create) now
completes cleanly with no FOTY0013/XPST0003, and the running container
answers the manifest, suggest, and merge endpoints correctly (re-tested
the exact payload that motivated the previous fix).
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01GJ79F6UanhKHMctVm12ATN
… literals
Two real, confirmed stored-XSS spots in the annotate editor, both the same
class of bug: raw template-literal HTML assigned straight to .innerHTML
with an interpolated value that isn't developer-controlled.
1. The .form-ref input's "loading"/"loading failed" status messages built
`<pb-i18n key="..."/> ${ref}...` (and "...${msg}") directly. `ref` is
pre-populated from the document's own @ref/@key attribute when opening
an already-tagged entity - a malicious @ref value in a document would
fire for any user who later opened that entity's panel, not just
someone typing it themselves. Fixed with a new setStatusMessage() helper
that appends the interpolated text as a real text node instead.
2. The "note" annotation-detail popup set `container.innerHTML =
data.properties.note` - plain document text pulled from the ODD's own
[[note]] template interpolation, not pre-sanitized markup. Changed to
.textContent. Also changed the sibling error-message innerHTML
assignment (h.innerHTML = msg) to .textContent/a real <pb-i18n> element
for the same reason, even though that one's less directly
attacker-reachable.
Moves the reconciliation server profile from its standalone home in
mpilhlt/tei-publisher-reconcile into this repo as profiles/reconcile/,
matching the structure every other feature profile already uses (compare
to profiles/dts/: config.json with $schema/type/category/depends/api,
modules/, doc/README.md). Dropped expath-pkg.xml/repo.xml/build.xml and
the now-redundant "skipSource" config.json entry - standalone-repo
packaging scaffolding a profile living inside this monorepo doesn't need.
Tests moved to fit this repo's existing per-profile Cypress layout
(test/cypress/e2e/{api,gui}/*.cy.js, test/cypress/fixtures/schemas/ - see
base10's own test/cypress/schemas/openapi-3.0.json for the established
"a profile can ship its own JSON Schema fixtures" precedent):
- test/cypress/e2e/api/reconcile.cy.js and test/cypress/fixtures/schemas/
{0.2,1.0}/*.json - the reconciliation protocol's own API tests, staying
with the server profile.
- test/cypress/e2e/gui/annotate-reconciliation.cy.js moved to
profiles/annotate/test/cypress/e2e/gui/ instead, alongside that
profile's other GUI tests - it exercises the annotate editor's client
behavior (via a live ReconciliationService connector), not the
reconcile profile's own server code.
- test/xqsuite/*.xqm kept as-is (no other profile here uses XQSuite, but
these are pure-logic unit tests - matches-value/normalize-1.0/normalize-
0.2/evaluate - with no natural Cypress equivalent; dropping them would
be a real coverage loss, not actual convention-following).
Scrubbed a handful of doc-comment references to paths and files specific
to the origin repo's own layout (skills/teipublisher-reconciliation-
testing/SKILL.md, reconc-specs/ as a local sibling clone,
README_MANUAL_TESTING.md) in favor of repo-agnostic wording (e.g.
pointing at https://github.com/reconciliation-api/specs directly) - none
of those paths exist here.
No code inside the .xql modules needed changing: every cross-module
import (config.xqm, pm-config.xql, etc.) already resolves against the
generated app's flattened modules/ directory regardless of which
profile's source tree it came from, exactly like every other profile's
own modules already do.
Not yet verified end-to-end in this repo (podman networking in the
current sandbox can't set up rootless containers right now - unrelated
infrastructure issue, not a code problem); the moved profile is otherwise
identical in behavior to the extensively-tested copy still living at
mpilhlt/tei-publisher-reconcile's own reconcile/ - see that repo's
AGENTS.md and README_MANUAL_TESTING.md for the full conformance history.
Re-run the full suite (XQSuite + Cypress API/GUI + cors-check.sh + both
local reconciliation-api testbenches) against a throwaway app generated
from this profile before treating this as PR-ready.
Prep work for a future PR to eeditiones/jinks - see the
fork_rebases_aug2026 and related project memory in
mpilhlt/tei-publisher-reconcile for the full upstreaming context. The
original copy stays in place at mpilhlt/tei-publisher-reconcile's own
reconcile/ for now; that repo's Docker demo image keeps building from
there until this is actually merged upstream.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01XhBiqPTawcDo2p7niXwNgR
Refactor comments for clarity and readability.
This was referenced Sep 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reconciliation update
The main point of the major reconciliation overhaul is adding a "reconcile" profile that makes TEI Publisher act as a reconciliation server. Thus, the data that a project maintains can be used by other projects (or internally, for that matter) as authority data. Often these data are quite specialized and the project is well-established in its research domain and community, but the data don't meet all (e.g. notability or entity type) requirements that large and "official" authority databases require. Then, the domain community can reconcile their OpenRefine or TP annotation activities against the project's TEI Publisher instance.
A second point is an update of the existing "annotate" profile for reconciliation services. This goes along with a PR (390) for tei-publisher-components. It adds explicit support for more recent versions of the Reconciliation API (v0.2 and v1.0-draft), plus some new features. Most notably, data extensions, which allows reconciliation clients to augment their data with more properties of the reconciled entities that are maintained and supplied by the authority database. This required revising how fields are mapped to TEI attributes: while formerly, it was hardcoded that the retrieved entity id would go to the
@keyattribute, it is now possible to specify a mapping in the authority lookup definition like this:For more details, see PR 390 in tei-publisher-components.
New reconcile profile for adding the API
profiles/reconcile/README.mdprofiles/docs/data/doc/reconcile*: documentation for the reconciliation API serverprofiles/reconcile/config.json: profile configprofiles/reconcile/modules/reconcile-api.tpl.json: OpenAPI-compatible spec (template)profiles/reconcile/modules/reconcile-api.xql: implementing the various endpoint functions (manifest, query, preview, suggest, extend etc.)profiles/reconcile/modules/reconcile-config.xql: configure$reconc-config:TYPEmap specifying how to handle, query, rank and offer different entity types in reconciliation (and some limits)profiles/reconcile/modules/reconcile-conditions.xql: harmonize/normalize query conditions between various query request shapes of the different supported API versionsprofiles/reconcile/modules/reconcile-fulltext.xql: add (optionally extend) fuzzy Lucene fulltext search that can be used in reconciliation queriesprofiles/reconcile/modules/reconcile-scoring.xql: add (optionally extend) scoring functions that can take into account Levenshtein distancesUpdate annotate profile
profiles/annotate/config.json:@keyby default) and add an icon to the themeprofiles/annotate/modules/annotations/tei-annotation-config.tpl.xqm:profiles/annotate/templates/pages/annotate.html)profiles/annotate/resources/css/annotate.css:profiles/annotate/resources/scripts/annotations/annotations.js:document.addEventListener("pb-page-loaded"...)profiles/annotate/templates/annotation-blocks.html:profiles/annotate/templates/pages/annotate-jats.html:profiles/annotate/templates/pages/annotate-tei.html:profiles/annotate/templates/pages/annotate.html:profiles/annotate/modules/annotations/tei-annotation-config.tpl.xqm)Other changes
profiles/forms/config.json: