diff --git a/.zenodo.json b/.zenodo.json index 43a0bdc0..95dbd376 100644 --- a/.zenodo.json +++ b/.zenodo.json @@ -1,8 +1,8 @@ { - "title": "Catalog of Second Chances (CSC) - Digital Database and Corresponding Interfaces for ReUse of Architectural Components (v0.6.4.0)", - "version": "0.6.4.0", + "title": "Catalog of Second Chances (CSC) - Digital Database and Corresponding Interfaces for ReUse of Architectural Components (v0.6.4.1)", + "version": "0.6.4.1", "upload_type": "software", - "description": "

The Catalog of Second Chances (CSC) is a prototypical digital component database. It catalogs digital representations of uniquely identifiable architectural components for reuse. Furthermore, it provides corresponding interfaces and tools to leverage them via a backend REST-API, a web-frontend as well as a interface for direct interaction within Rhino Grasshopper.

Changes in v0.6.4.0 (2026-10-23)

Search and the location map.

Added

Changed

Fixed

Security

Server notes

Full history: CHANGELOG.md

", + "description": "

The Catalog of Second Chances (CSC) is a prototypical digital component database. It catalogs digital representations of uniquely identifiable architectural components for reuse. Furthermore, it provides corresponding interfaces and tools to leverage them via a backend REST-API, a web-frontend as well as a interface for direct interaction within Rhino Grasshopper.

Changes in v0.6.4.1 (2026-10-27)

Hardening: the MCP connector, the component page and dependencies.

Added

Changed

Security

Server notes

Full history: CHANGELOG.md

", "creators": [ { "name": "Eschenbach, Max Benjamin", diff --git a/CHANGELOG.md b/CHANGELOG.md index 31eb3f49..3d43c277 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,60 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [0.6.4.1] - 2026-10-27 + +Hardening: the MCP connector, the component page and dependencies. + +### Added + +- **Search by evidence method:** the catalog list takes `evidence_method=` (one name of the + evidence vocabulary, an unknown name is a 422) and keeps only pieces with at least one + published, current record of that method, for example `reinforcement_layout` for the pieces + with a reinforcement layout. It combines with every other filter, also with fit and near. The + MCP tools `search_components`, `catalog_overview` and `similar_components` take it as + `has_evidence`. A filter in Browse follows later. +- **A reinforcement layout in the viewer:** the card of a reinforcement layout in the evidence + list has "Show in viewer", for every status, so a submitted draft can be seen before it is + published. The viewer opens the version the bars belong to and draws that layout on top of the + published ones, which turn faint; the legend names it with its status, and an x takes it out + again. The link can be sent on: whoever may not see the record gets a short note and no bars. + +### Changed + +- **MCP connector:** the server introduces itself as the Catalog of Second Chances, with a link to + the website and an icon, and its instructions tell a chat to use these tools for reused building + components before a web search (in English and German). Every tool has a title and is marked + read-only, and the descriptions name the evidence methods. +- **MCP, reinforcement:** `get_component` and `get_version` count the reinforcement layouts of the + version from its evidence records (records, bars, diameters) instead of the old inline field, + which always gave 0. In `get_evidence` a layout record carries a summary of its bars, and + `performed_by` shows the organization. +- **Similar pieces:** on a wide screen the card sits in the left rail below Provenance, with + smaller tiles, three to a row, so it no longer pushes the evidence records down. On a narrow + screen it keeps its place after Photos and location. "Show more" appears only when there are + more than six similar pieces. +- **Statistics:** the size facet of `/identities/stats` (`bbxX`) buckets the largest side of each + piece in millimetres (0, 100, 250, 500, 1000, 2000, 5000, 10000 and more); the old edges came + from the metre era and put nearly every piece in the last bucket. Pieces without a size are + left out instead of counting as the largest. +- **Write model:** creating or changing a version rejects the old inline field + `geometry.reinforcements`; a reinforcement layout is an evidence record. + +### Security + +- **E-mail verification** accepts only the hashed token. The fallback to a token stored in plain + text is gone (such tokens expired a day after 0.6.1.1); an account that is still unverified can + ask for a new mail. +- **Dependencies:** the frontend lockfile takes `undici` 7.30.1 and `fast-uri` 3.1.9 (one high, two + medium and two low advisories; both come only through the development tool `shadcn`, the + production install was not affected). The local import scripts move to Pillow 13.0.0. + +### Server notes + +- No step needed: no new setting, collection or index. +- In claude.ai the connector can be renamed to "Catalog of Second Chances" (Settings, Connectors); + the name the server gives itself is shown when it is added again. + ## [0.6.4.0] - 2026-10-23 Search and the location map. diff --git a/README.md b/README.md index 5ab02022..f90a21c3 100644 --- a/README.md +++ b/README.md @@ -29,8 +29,8 @@ framework ## Current Versions -- **CSC**: 0.6.4.0 --- backend, web frontend and Grasshopper interface are released together - under one version (tag `v0.6.4.0`); the single source is the `VERSION` file. +- **CSC**: 0.6.4.1 --- backend, web frontend and Grasshopper interface are released together + under one version (tag `v0.6.4.1`); the single source is the `VERSION` file. See `CHANGELOG.md` for release notes. @@ -548,15 +548,18 @@ sent back to) before you press Allow, and press Deny when it is not the tool you just connected. **What it can do.** Ten read-only tools: `search_components` (the filters of -Browse, a page at a time), `catalog_overview` (how many pieces, counted by -material, original function, dataset, shape class and circulation, the range of -each sorted side in millimetres, and how many have a location; it takes the same -filters), `get_component`, `list_versions`, `get_version` (geometry as counts -and a link, never the meshes), `get_evidence`, `list_datasets`, `get_dataset`, -`get_timeline` and `similar_components` (the pieces nearest in shape over the -whole descriptor, not on the two-dimensional similarity map, with their distance -in standardised units; it takes the same filters, which apply before the -ranking). Every answer +Browse and `has_evidence`, which keeps only pieces with a published, current +record of one evidence method, for example `reinforcement_layout` for "with +reinforcement", a page at a time), `catalog_overview` (how many pieces, counted +by material, original function, dataset, shape class and circulation, the range +of each sorted side in millimetres, and how many have a location; it takes the +same filters), `get_component` (with the reinforcement layouts of the current +version as counts), `list_versions`, `get_version` (geometry as counts and a +link, never the meshes; the reinforcement layouts of the version as counts), +`get_evidence`, `list_datasets`, `get_dataset`, `get_timeline` and +`similar_components` (the pieces nearest in shape over the whole descriptor, not +on the two-dimensional similarity map, with their distance in standardised +units; it takes the same filters, which apply before the ranking). Every answer carries the web link of the record, and a record that is not public is marked (`visibility` private, `status` other than published). There are no write tools and no tool for users, invitations or tokens; no answer holds an e-mail address diff --git a/VERSION b/VERSION index 31d34d20..f4362155 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.6.4.0 +0.6.4.1 diff --git a/docs/adr/DESIGN_DECISIONS.md b/docs/adr/DESIGN_DECISIONS.md index ee0dbe0e..a73aff43 100644 --- a/docs/adr/DESIGN_DECISIONS.md +++ b/docs/adr/DESIGN_DECISIONS.md @@ -308,9 +308,11 @@ into one **metadata object**. | 8.171 | **Location map: Leaflet with the tiles of openstreetmap.de** (user 2026-10-22; 0.6.4 (c), amends 8.164). Leaflet and `leaflet.markercluster` come from npm and are bundled with the web app (no CDN, the script rules of the CSP unchanged). The basemap is `https://tile.openstreetmap.de/{z}/{x}/{y}.png` of FOSSGIS e.V. (Germany), keyless, with the attribution to OpenStreetMap contributors and FOSSGIS. The CSP adds `https://tile.openstreetmap.de` to `img-src`, nothing else. Only the location map page loads tiles; the privacy page names FOSSGIS e.V. as the receiver of the visitor's IP address when that page is opened. The page (user 2026-10-22): the filter bar of Browse with its state in the URL and a "Show as list" link to Browse with the same filters; the Current / Origin layer switch (8.170) with clustered markers that show a count; colour-by and legend shared with the Similarity Map (`lib/mapColors`); a click on a marker opens the shared preview panel; a click on an empty spot offers "Search near here", which opens Browse with the Near filter set; visibility as on Browse. Amended by 8.176. | Same choice as the ReCreate digital map of precast concrete systems (D1.1, https://doi.org/10.5281/zenodo.19492868): a self-hosted Leaflet with an EU tile server. Rejected: MapLibre with self-hosted PMTiles (no third party at all, but some GB of extract on the server and about a day more); the public `tile.openstreetmap.org` of the OpenStreetMap Foundation (UK; light use only). The traffic of the map page stays light, within the fair use of the FOSSGIS servers. The privacy page change goes into the legal check of the privacy page that is pending with the university's data protection office. | | 8.172 | **Invite-only registration, and the open domains named instead of TU Darmstadt** (user 2026-10-22; 0.6.4, amends 8.14). (a) `CSC_OPEN_REGISTRATION_DOMAINS` without a domain means invite only: the default becomes empty instead of `tu-darmstadt.de`, so an empty or missing variable no longer brings an open domain back (until now `env_ignore_empty` let an empty value fall back to the default, and only a value such as `,` closed registration). (b) A public endpoint tells the web app the open domains (an empty list = invite only); the domains are not secret. (c) With no open domain the register page is an invite-only page: it says registration is by invitation only (ask a dataset moderator), and the registration form shows only with an invitation code (from the link `?code=...` or typed in); nothing suggests open registration, and the "Register" links in the sign-in form and the sidebar lead to that page in words that fit (for example "Have an invitation?"). (d) With open domains the current register page stays, but its texts no longer name TU Darmstadt: "Registration is open only to addresses of certain domains", with a question-mark tooltip (hover and tap) that lists the domains currently set; the invitation code field is labelled for addresses outside those domains. (e) The placeholders with a TU Darmstadt address on the register and verification-pending pages become neutral (`name@example.org`); the admin users page names the open domains or "invite only". Default confirmed (user 2026-10-22): empty, invite only unless the variable is set; the 0.6.4 release steps name setting it in the production backend environment. | The setting can now be closed by emptying it, as one would expect. Release step for the maintainer: production keeps open registration only if `CSC_OPEN_REGISTRATION_DOMAINS` is set explicitly in the backend environment before the deploy; without it production becomes invite only. | | 8.173 | **Sorting: volume, distance and match** (user 2026-10-22; 0.6.4 (d)). The list gains three sort keys: `volume` (the product of the three `bbx` sides), `distance` (only while a near search is set, 8.170) and `match` (only while a dimension search is set, 8.169: closest first for *about this size*, least waste first for *can hold*). A dimension search sorts by `match` by default, a near search by `distance`; with both, the dimension search filters and `distance` sorts; any other sort stays one choice away. The web keeps the sortable table headers and gives the card view a sort menu with the same keys. MCP `search_components` gets `sort` (`catalog_number`, `date`, `created`, `volume`, `distance`, `match`) and `order` (`asc` or `desc`). Overview (user 2026-10-22): an MCP tool `catalog_overview` on `GET /identities/stats` as the token's owner, with the filters of `search_components`; it returns the total, counts by material (with labels), function, dataset, shape class and circulation, the range (smallest and largest) of each sorted side, and how many pieces have a current and an origin location; the answer says how many of the counted pieces are not public. | Until now MCP always sorted by catalog number, and the card view had no sort. A piece without `bbx` sorts last under `volume`. | -| 8.174 | **True similarity: kNN on the descriptors, behind a retrieval check** (user 2026-10-22; 0.6.4 (e)). Similarity is the distance in the full descriptor space, not on the 2D map: the bases of the Similarity Map (`radial_signature` by default, `scalars`, `hks`), each dimension standardised over the catalog, a NumPy kNN over a cached feature matrix rebuilt by the same cron as the map cache. MCP `similar_components` switches to it: a `basis` parameter, the descriptor distance per result, and the filters of `search_components` including the dimension and near search ("similar in shape, can hold 1200 x 600 x 200, within 50 km"). The component page gets a "Similar pieces" card with the 6 nearest visible pieces (thumbnails) and a link to more. Stop point: before the card ships the implementer checks the neighbours of a few pieces of every dataset by eye and reports them; if they do not look right, only MCP gets the kNN. Visibility as on Browse. Grasshopper (user 2026-10-22): the kNN search is also exposed in Grasshopper, through a REST endpoint that the MCP tool and the GH client share; the GH component is grilled before its task. GH components (user 2026-10-22; T6, after T4): a new `FetchSimilarComponents` in "2 Catalog Interface" (inputs `IdentityID`, `Basis` default `radial_signature`, `Count` default 10, and optionally the filters of `FetchFilteredComponents`; outputs `ComponentPassport` and `Distance` in the same order); `FetchFilteredComponents` gains the dimension search (mode, size, tolerance) and the near search (point, layer, radius) so it keeps mirroring the web filter menu, with the deviation or distance per result. The changed user objects need the maintainer's export in Rhino 8 and `bump-version --gh`; the Session is unchanged; the server is deployed before users run `CSC_Update`. Amended (user 2026-10-22, after the T4 check): instead of a link to more, the card has a "Show more" button that loads up to 24 nearest pieces into the card and a "Show fewer" button that returns to 6, since no list sorts by similarity. Near-identical pieces of a series (the SAS bricks, same size to 0.1 mm, distance rounding to 0.00) stay separate tiles: for a piece of a series its siblings are the honest answer. | Until now `similar_components` ranked by the distance on the cached UMAP layout, which distorts distances. About 2300 pieces keep a query in milliseconds without Atlas Vector Search. The same retrieval rule as for HKS v2 in 0.6.7; HKS gains from this kNN later. | +| 8.174 | **True similarity: kNN on the descriptors, behind a retrieval check** (user 2026-10-22; 0.6.4 (e)). Similarity is the distance in the full descriptor space, not on the 2D map: the bases of the Similarity Map (`radial_signature` by default, `scalars`, `hks`), each dimension standardised over the catalog, a NumPy kNN over a cached feature matrix rebuilt by the same cron as the map cache. MCP `similar_components` switches to it: a `basis` parameter, the descriptor distance per result, and the filters of `search_components` including the dimension and near search ("similar in shape, can hold 1200 x 600 x 200, within 50 km"). The component page gets a "Similar pieces" card with the 6 nearest visible pieces (thumbnails) and a link to more. Stop point: before the card ships the implementer checks the neighbours of a few pieces of every dataset by eye and reports them; if they do not look right, only MCP gets the kNN. Visibility as on Browse. Grasshopper (user 2026-10-22): the kNN search is also exposed in Grasshopper, through a REST endpoint that the MCP tool and the GH client share; the GH component is grilled before its task. GH components (user 2026-10-22; T6, after T4): a new `FetchSimilarComponents` in "2 Catalog Interface" (inputs `IdentityID`, `Basis` default `radial_signature`, `Count` default 10, and optionally the filters of `FetchFilteredComponents`; outputs `ComponentPassport` and `Distance` in the same order); `FetchFilteredComponents` gains the dimension search (mode, size, tolerance) and the near search (point, layer, radius) so it keeps mirroring the web filter menu, with the deviation or distance per result. The changed user objects need the maintainer's export in Rhino 8 and `bump-version --gh`; the Session is unchanged; the server is deployed before users run `CSC_Update`. Amended (user 2026-10-22, after the T4 check): instead of a link to more, the card has a "Show more" button that loads up to 24 nearest pieces into the card and a "Show fewer" button that returns to 6, since no list sorts by similarity. Near-identical pieces of a series (the SAS bricks, same size to 0.1 mm, distance rounding to 0.00) stay separate tiles: for a piece of a series its siblings are the honest answer. Amended (user 2026-10-27, 0.6.4.1): the card took too much room next to the records. On a wide screen it sits in the left rail below Provenance, with smaller tiles (three to a row, a lower preview, the name and one line); on a narrow screen it keeps its place in the order, after Photos and location. | Until now `similar_components` ranked by the distance on the cached UMAP layout, which distorts distances. About 2300 pieces keep a query in milliseconds without Atlas Vector Search. The same retrieval rule as for HKS v2 in 0.6.7; HKS gains from this kNN later. | | 8.175 | **Vertex colours from the texture when a mesh is staged** (user 2026-10-22; added to 0.6.4). When CreateComponentIdentity or CreateComponentSnapshot stages a mesh, it computes vertex colours from the texture of the mesh, so a textured scan keeps its look in the Preview, Reduced and Original files. The texture comes only from the render material of the Rhino object the Geometry input references; there is no extra input. Precedence (user 2026-10-22): the texture wins over vertex colours the mesh already has; the component says in a remark that it replaced them. Sampling (user 2026-10-22): bilinear at each vertex's texture coordinate; the Preview, Reduced and Original levels are each coloured from their own texture coordinates. Build (user 2026-10-22): part of T6, the Grasshopper task after T4, so one Rhino 8 export and one `bump-version --gh` cover both. | Until now staging kept only a mesh's own vertex colours (`export_mesh` in `csc_gh/rhino.py`) and ignored textures: a textured scan was staged in the flat piece colour. A Grasshopper mesh carries its texture coordinates but not the image; the image belongs to the material of the Rhino object, reachable only while the input references that object. An internalised or Grasshopper-made mesh therefore keeps the old behaviour. | | 8.176 | **Privacy and imprint: the maintainer is the controller; no Google map preview** (user 2026-10-23; added to 0.6.4, amends 8.164 and 8.171). The privacy page and the imprint state that the catalog, part of a doctoral project at the DDU of TU Darmstadt, is run by the maintainer in person, who alone is the controller; TU Darmstadt does not operate the site and is not responsible for it. The hover preview of a location in the Browse table, an embedded Google map, is removed; the location stays a link that opens Google Maps in a new tab only on a click. With no Google frame left, the CSP drops its `frame-src` hosts. The cookies row no longer claims that nothing is loaded from other sites (the Location Map loads tiles from FOSSGIS, 8.171), and the imprint cites the DDG instead of the repealed TMG. | The embedded map sent every visitor who pointed at a location to Google (Ireland and USA) without a click and was not named on the privacy page; in Germany such an embed is widely held to need consent first. The Location Map (8.171) now covers seeing pieces on a map, so the only third party a visitor reaches without a click is the tile server of FOSSGIS, on that page. | +| 8.177 | **Search by evidence method** (user 2026-10-27; 0.6.4.1). The catalog list takes `evidence_method=` (one name of the evidence vocabulary; an unknown name is a 422): only pieces with at least one published, current evidence record of that method are listed (current: not superseded by a later record, `superseded_by` empty; user 2026-10-27). It combines with every other filter, also with fit and near. The MCP tools `search_components`, `catalog_overview` and `similar_components` take it as `has_evidence`. The match runs on the identity: the ids come from the evidence collection (status `published`, not superseded, the method), which has an index on status and method; the access of the list to the pieces stays as it is. A filter in Browse follows later. | Asking for pieces "with reinforcement" took one `get_evidence` call per piece (chat report, 2026-10-27). `properties.rebar_diameter` is no substitute: an archival claim sets it without any bars. A general method filter serves every method, not only `reinforcement_layout`. | +| 8.178 | **A reinforcement layout in the viewer, whatever its status** (user 2026-10-27; 0.6.4.1). The card of a `reinforcement_layout` record in the evidence list has "Show in viewer". It opens the component page on `?snapshot=&layout=`: the viewer shows the version the bars belong to and draws that one record in addition to the published layouts, whatever its status (draft, pending, rejected, superseded, published). The legend names the record and its status, and its bars stand out from the others. The server decides who may load the record, as for any record (7.0); a link to a record the caller may not see draws nothing more and says so. Without `layout=` the viewer stays as it is: published layouts only. Only reinforcement layouts for now; the same link for other positioned records later, if needed. | A submitted draft layout could be read as a card but not seen: the viewer draws published layouts only, while a draft version of a piece opens in the viewer through `?snapshot=`. A global "Show drafts" toggle was rejected: it mixes moderation states into the view of the piece, cannot tell several drafts apart, and would have to cover marks and versions too. One record per link can be sent to a moderator for review. | **Facts confirmed (user, 2026-09-28):** the 35 ZirKuS reinforcement bars were modelled after the original drawing --> `basis: drawing` (step 6d). The 2024-07-24 deinstallation applies to all 16 diff --git a/docs/adr/IMPLEMENTATION_PLAN_0.6.md b/docs/adr/IMPLEMENTATION_PLAN_0.6.md index 914fab8c..0315d71f 100644 --- a/docs/adr/IMPLEMENTATION_PLAN_0.6.md +++ b/docs/adr/IMPLEMENTATION_PLAN_0.6.md @@ -980,11 +980,17 @@ fold (once it is clear which quantities are searched); Atlas Search for typo-tol **0.6.4 decided (grilling from 2026-10-21):** (a) dimensions (8.169): two modes on the sorted sides of the current `bbx`, *about this size* (within a tolerance, closest first by the largest relative deviation) and *can hold this size* (every side at least the asked one, least waste first); the per-axis filters and `bbx` sorts stay. Tolerance: one per query, mm or %, default 10 %. Web: a "Fits" section in Browse's "More filters" (size in any order, mode, tolerance), the list sorted by the match with deviation or waste per row, state in the URL. MCP: `fit_mode`, `size_mm`, `tolerance` on `search_components`, no `find_fitting` tool. (b) and (c) location (8.170): the current and the origin location both as GeoJSON points with a `2dsphere` index each; near search names which one; the map has both as layers, Current by default; filling in origin coordinates is a separate data task. Web: latitude and longitude of the origin place editable in the origin form (a pasted "lat, lon" accepted). Map (8.171): Leaflet and `leaflet.markercluster` bundled from npm, tiles from `tile.openstreetmap.de` (FOSSGIS e.V.), CSP `img-src` plus that host, one paragraph on the privacy page. Near: `near_lat`, `near_lon`, `near_layer`, `radius_km` on `search_components` (no `find_near`), a "Near" section in Browse (point, layer, radius; "Near this piece" from a component page), sorted by distance. Registration (8.172, added by the user): an empty `CSC_OPEN_REGISTRATION_DOMAINS` (the new default) means invite only, with an invite-only register page; with domains set the register page says it is open to certain domains and lists them in a tooltip; no TU Darmstadt texts or placeholders. Map page: Browse's filter bar (URL state, "Show as list"), layer switch, clusters with counts, colour-by and legend of the Similarity Map, marker click opens the preview panel, empty-spot click offers "Search near here". (d) Sorting (8.173): keys `volume`, `distance`, `match`; Fits sorts by match, Near by distance, both: distance; a sort menu in the card view; `sort` and `order` on `search_components`. Overview: MCP `catalog_overview` (stats with the search filters: total, counts by material, function, dataset, shape class, circulation, side ranges, location coverage, non-public count). (e) Similarity (8.174): kNN on the standardised descriptors (basis `radial_signature` by default), `similar_components` with `basis` and the search filters, a "Similar pieces" card behind a retrieval check. Build order (user 2026-10-22): the registration task (8.172) first as hardening, then dimensions and sorting, location, the location map, overview and similarity; the kNN search also in Grasshopper. T6 (GH, after T4): `FetchSimilarComponents` (IdentityID, Basis, Count, filters; ComponentPassport, Distance) and Fit and Near in `FetchFilteredComponents`. Waste share (8.169, user 2026-10-22): at most 99 % unless exactly 100 %. Vertex colours from the texture (8.175, added by the user 2026-10-22): the Create components compute vertex colours from the texture of the referenced Rhino object's render material when they stage a mesh; no extra input. The texture wins over existing vertex colours, with a remark. Bilinear sampling at each vertex's texture coordinate, each level from its own coordinates. Built inside T6 (one Rhino 8 export, one `bump-version --gh`). Coordinate boxes (8.170 amended, user 2026-10-22): one shared latitude and longitude pair in every location form, a pasted pair fills both, a comma jumps to longitude, no separate paste field. Similar pieces card (8.174 amended): "Show more" up to 24 in the card with "Show fewer"; series siblings stay separate tiles. Privacy and imprint (8.176, user 2026-10-23): the maintainer is the controller, TU Darmstadt is not; the Google map hover preview in Browse goes (the link stays), the CSP drops frame-src; cookies row corrected; imprint on the DDG. +**0.6.4.1 --- hardening (user 2026-10-27, before 0.6.5):** first the MCP connector, from two chat reports. (a) Search by evidence method (8.177): `evidence_method` on the catalog list, `has_evidence` on the MCP search tools; Browse later. (b) Recognizable connector: instructions that name the domain in English and German (reused and salvaged building components, Bauteile, Wiederverwendung, concrete walls and beams, reinforcement / Bewehrung) and say to use these tools before a web search for such pieces; `website_url` and `icons` on the server; a title and `readOnlyHint` on every tool; tool descriptions that name the evidence methods. (c) The geometry summary drops the `reinforcements` count (always 0 since step 6d moved the bars to evidence, 7.8) and gains the reinforcement layouts of the version (records and bars) with a pointer to `get_evidence`. (d) The evidence list rows of a `reinforcement_layout` carry the bar count, basis, specs, diameters and cover; `performed_by` shows an organization instead of `null`. (e) The write model no longer accepts `geometry.reinforcements` (7.8). (b) to (e) confirmed by the user 2026-10-27; the server icon is the web app's `favicon.ico` until there is a CSC logo. (f) The "Similar pieces" card (8.174 amended): in the left rail on a wide screen, smaller tiles. (g) The `bbx` facet of `/identities/stats` buckets the largest side with edges from the metre era (0 to 1000, then ">=1000"), so in millimetres nearly every piece lands in the last bucket: edges in millimetres instead (for example 100, 250, 500, 1000, 2000, 5000, 10000), checked against the analytics page. (h) The Similar pieces card offers "Show more" only when the first answer says there are more than 6 neighbours, not when there are exactly 6. (g) and (h) added by the user 2026-10-27. (i) Frontend advisories: `undici` (high, fixed in 7.30.1) and `fast-uri` (two medium, two low, fixed in 3.1.9) come only through `shadcn`, the component generator, which is a `devDependency` already; the lockfile takes the fixed versions. (j) The import scripts (`scripts/import/requirements-import.txt`) move to Pillow 13.0.0 (20 alerts); they run locally, the accepted risk of the backend pin stays as it is. (k) E-mail verification drops the fallback to a token stored in plain text (the TODO for 0.6.2 in `auth.py`; such tokens expired 24 hours after 0.6.1.1). (i) to (k) added by the user 2026-10-27. (e) stays with `geometry.reinforcements` only (user 2026-10-27): `geometry.marker_points` is still written by the robot rubble scan pipeline, which is not in use now; both go to the backlog below. (l) Seven tests in `tests/grasshopper` hard-code the client `gh-userobjects/0.6.3.0` and fail since the version bump of 0.6.4.0 (CI does not run `tests/grasshopper`): they read the expected client from the `CSC_CLIENT` line of `DDU_CSC_Session.py` and check that it is not newer than `VERSION` (the rule of `check_version.py`), so a bump with or without `--gh` no longer breaks them; found during T1, built with T3 and a fix round after the bump. (m) "Show in viewer" on the card of a reinforcement layout (8.178, user 2026-10-27): the viewer draws that one record on its version whatever its status, so a submitted draft can be seen and checked; built as T4. + **0.6.5 --- reinforcement from Grasshopper (about 6--8 days; moved from 0.6.4 by the user 2026-10-21):** (a) units in GH: every component scales by the document unit (a prerequisite for placing scans); (b) RL8, a GH builder for the `rebar_spec` claim and its display, grilled together with (c); (c) Ferroscan to reinforcement layout, parts (a) report reader, detection, placement and a draft through `AddEvidence`, (b) the report image and PDF -as attachments, (d) several scans of one face; (c) "the image in 3D" later. +as attachments, (d) several scans of one face; (c) "the image in 3D" later. (e) `ApplyFrame` stays as it +is (user 2026-10-27): its `Frame` output is the frame used, in the stored coordinates (where the piece +sat before the move), not the frame of the moved piece. The descriptions of the component and of the +output say so plainly and point to `DisassembleComponent`, whose `Frame` applies `csc_placement` and so +gives the frame where the moved piece sits; exported with the user objects of 0.6.5. **0.6.6 --- interoperability (about 5--7 days):** the IFC export (`compose?format=ifc`; first the `ifcopenshell` wheel check for the server, then IFC4 vs IFC4x3 and single piece vs selection); the timeline as PROV-O / @@ -1005,8 +1011,12 @@ a canvas has the inputs its source declares. The access and the type hint of an annotation in the source's `RunScript` signature (`FitSize: list[object]` is list access without a type hint), but `CSC_Update` leaves the inputs of a component already on a canvas as they were, so new inputs are missing there. +**Later, not yet scheduled --- the robot rubble scan pipeline (user 2026-10-27):** `src/py_modules/csc_robot_rubble_scan_processing` still builds the 0.5 shape (`type`, `assembly`, `processes`, `iframe`) and puts the scan markers into `geometry.marker_points`, which `upload_robot_scan.py` posts unchanged to `POST /identities`. Before it runs again: the 0.6 create request, the markers as `capture.markers` with role `rig` (7.7), a trial upload against the local stack; then the write model rejects `geometry.marker_points`, as it rejects `geometry.reinforcements` since 0.6.4.1. + **Outside a release (the maintainer):** the Atlas traffic measurement of 0.6.1.0 (c); the backend CI job back to `ubuntu-latest` once MongoDB -8.0 starts on the runner's kernel (SERVER-121912). +8.0 starts on the runner's kernel (SERVER-121912); `shadcn` (dev only) lifted to >=4.22.1 once that +version is about two weeks old, which clears the six remaining dev findings of `npm audit` (found in the +review of 0.6.4.1). **Later or ideas:** the GH sign-in on macOS (Keychain); Ferroscan "the image in 3D"; MCP opened to members (after 0.6.3.1); the CSC logo and brand, then the logo in mails, Open Graph and the favicon; the partners on diff --git a/scripts/backup/pull_production.ps1 b/scripts/backup/pull_production.ps1 new file mode 100644 index 00000000..97975c67 --- /dev/null +++ b/scripts/backup/pull_production.ps1 @@ -0,0 +1,53 @@ +# Pull a backup of production to this computer over ssh. +# +# .\scripts\backup\pull_production.ps1 -Server @ -What json +# the collections as JSON into mongodb_collections_local\ +# .\scripts\backup\pull_production.ps1 -Server @ -What archive -Dest D:\backups +# mongodump archive (gzip) to \\csc_db_.archive.gz +# .\scripts\backup\pull_production.ps1 -Server @ -What assets -Dest D:\backups +# the asset folders as \\csc_assets_.tgz +# +# Needs ~/bin/csc_dump.sh on the server (uberspaceconfig/bin/csc_dump.sh). +# Binary streams go through cmd.exe so PowerShell does not re-encode them. + +param( + [Parameter(Mandatory = $true)][string]$Server, + [Parameter(Mandatory = $true)][ValidateSet('json', 'archive', 'assets')][string]$What, + [string]$Dest = '' +) + +$ErrorActionPreference = 'Stop' +$day = Get-Date -Format yyMMdd +$repo = Resolve-Path (Join-Path $PSScriptRoot '..\..') + +function Pull([string]$remote, [string]$file) { + cmd /c "ssh $Server $remote > `"$file`"" + if ($LASTEXITCODE -ne 0) { throw "ssh failed ($LASTEXITCODE): $remote" } + $size = (Get-Item $file).Length + if ($size -eq 0) { throw "empty download: $file" } + Write-Host "$file ($size bytes)" +} + +if ($What -eq 'json') { + $target = Join-Path $repo "mongodb_collections_local\$day" + if (Test-Path $target) { throw "$target exists already; move it away first" } + New-Item -ItemType Directory -Force $target | Out-Null + $tgz = Join-Path $env:TEMP "csc_json_$day.tgz" + Pull 'bin/csc_dump.sh json' $tgz + tar xzf $tgz -C $target + Remove-Item $tgz + Get-ChildItem $target | Select-Object Name, Length | Format-Table -AutoSize + exit 0 +} + +if (-not $Dest) { throw '-Dest is required for archive and assets' } +$folder = Join-Path $Dest $day +New-Item -ItemType Directory -Force $folder | Out-Null + +if ($What -eq 'archive') { + Pull 'bin/csc_dump.sh archive' (Join-Path $folder "csc_db_$day.archive.gz") +} else { + $file = Join-Path $folder "csc_assets_$day.tgz" + Pull 'tar czf - -C ~ csc_assets_private html/csc_assets' $file + tar tzf $file | Select-Object -First 3 +} diff --git a/scripts/import/requirements-import.txt b/scripts/import/requirements-import.txt index 475122f1..f08fea51 100644 --- a/scripts/import/requirements-import.txt +++ b/scripts/import/requirements-import.txt @@ -30,7 +30,7 @@ cycler==0.12.1 fonttools==4.67.0 kiwisolver==1.5.1 packaging==26.3 -pillow==12.3.0 +pillow==13.0.0 pyparsing==3.3.3 python-dateutil==2.9.0.post0 six==1.17.0 diff --git a/src/backend/apps/catalog/api/auth.py b/src/backend/apps/catalog/api/auth.py index cd07ac2d..8eb06cab 100644 --- a/src/backend/apps/catalog/api/auth.py +++ b/src/backend/apps/catalog/api/auth.py @@ -395,7 +395,7 @@ async def register_user( '_id': new_id, 'username': username, 'full_name': full_name, 'email': email, 'hashed_password': get_password_hash(password), 'disabled': False, 'role': 'user', 'email_verified': True, - 'verification_token': None, 'verification_token_expires': None, + 'verification_token_expires': None, 'invitation_id': invitation['_id'], } await _insert_account(users, doc) @@ -451,14 +451,10 @@ async def verify_email( if not token: raise HTTPException(400, 'Verification token is required') - # Find the user by the hash of the token (stored since 0.6.1.1); a token - # mailed before that is stored as it is and works until it expires, at - # most 24 hours. TODO 0.6.2: drop the plain-token fallback. + # Find the user by the hash of the token; the plain token is never stored user = await users.find_one({ 'verification_token_sha256': token_sha256(token), }) - if not user: - user = await users.find_one({'verification_token': token}) if not user: raise HTTPException(400, 'Invalid or expired verification token') diff --git a/src/backend/apps/catalog/api/catalog_list.py b/src/backend/apps/catalog/api/catalog_list.py index 7b49db6f..894c425d 100644 --- a/src/backend/apps/catalog/api/catalog_list.py +++ b/src/backend/apps/catalog/api/catalog_list.py @@ -130,11 +130,21 @@ def _format_list_rows( async def _pipelines_args(request: Request, filters: CatalogFilters, user: Optional[User]) -> Dict[str, Any]: """The filters plus the visibility rule (3.6) for this caller.""" + identity_match = and_match( + filters.identity_match(), + await visible_identity_match(request, viewer_of(user))) + if filters.evidence_method: + # 8.177: the pieces with a published, current record of this method + # (not superseded; None matches a missing field too), matched on the + # identity (the evidence collection has an index on status and + # method); the rest of the match keeps the access rules + ids = await request.app.mongodb_component_evidence.distinct( + 'identity_id', {'status': 'published', 'superseded_by': None, + 'method': filters.evidence_method}) + identity_match = and_match(identity_match, {'_id': {'$in': ids}}) return { 'snapshots_collection': request.app.mongodb_component_snapshots.name, - 'identity_match': and_match( - filters.identity_match(), - await visible_identity_match(request, viewer_of(user))), + 'identity_match': identity_match, 'snapshot_match': filters.snapshot_match(), 'fit_stages': filters.fit_pipeline_stages(), 'near': filters.near, diff --git a/src/backend/apps/catalog/api/identity_filters.py b/src/backend/apps/catalog/api/identity_filters.py index bbcfdd62..908cff16 100644 --- a/src/backend/apps/catalog/api/identity_filters.py +++ b/src/backend/apps/catalog/api/identity_filters.py @@ -14,6 +14,7 @@ from fastapi import HTTPException, Query +from ..vocab import EvidenceMethod from .dimension_fit import ( DEFAULT_TOLERANCE, FitError, @@ -85,6 +86,10 @@ class CatalogFilters: bbx_max_y: Optional[float] = None bbx_max_z: Optional[float] = None q: str = '' + # evidence method (8.177): only pieces with a published, current evidence + # record (not superseded) of this method; the ids come from the evidence collection, see + # ``catalog_list._pipelines_args`` + evidence_method: Optional[str] = None # dimension search (8.169): the mode, the asked sides (largest first) and # the tolerance of ``about`` as (``percent`` | ``mm``, value) fit_mode: Optional[str] = None @@ -222,6 +227,11 @@ def catalog_filters( bbx_max_z: Optional[float] = Query(None), q: str = Query('', max_length=FILTER_MAX_LENGTH, description='name, catalogue number or id'), + evidence_method: Optional[EvidenceMethod] = Query( + None, description='only pieces with at least one published, current ' + 'evidence record of this method (8.177), for example ' + 'reinforcement_layout; combines with every other ' + 'filter'), fit_mode: Optional[FitMode] = Query( None, description='dimension search (8.169): about = every side ' 'within fit_tol of fit_size; can_hold = every side ' diff --git a/src/backend/apps/catalog/api/identity_query.py b/src/backend/apps/catalog/api/identity_query.py index c744db36..b6fceab4 100644 --- a/src/backend/apps/catalog/api/identity_query.py +++ b/src/backend/apps/catalog/api/identity_query.py @@ -225,8 +225,6 @@ def build_count_pipeline( return pipeline -_BBX_ARRAY = {'$cond': [{'$isArray': '$bbx'}, '$bbx', []]} - _STATS_FACET_TEMPLATE: Dict[str, Any] = { 'total': [{'$count': 'count'}], 'byOriginalFunction': [ @@ -339,22 +337,19 @@ def build_count_pipeline( }}, ], 'bbx': [ - { - '$project': { - # a bbx that is no array reads as an empty one: $arrayElemAt - # fails on it, and one bad document must not 500 the stats - 'x': {'$arrayElemAt': [_BBX_ARRAY, 0]}, - 'y': {'$arrayElemAt': [_BBX_ARRAY, 1]}, - 'z': {'$arrayElemAt': [_BBX_ARRAY, 2]}, - } - }, + # the largest side (``_sides``, sorted largest first, as in the + # ``sides`` facet); a piece without three numbers has no size and is + # in no bucket, not in the default one + {'$match': {'_sides': {'$type': 'array'}}}, { '$bucket': { - 'groupBy': '$x', + 'groupBy': {'$arrayElemAt': ['$_sides', 0]}, + # millimetres, as bbx is stored: a bucket is labelled with + # its lower edge, the rest is '>=10000' 'boundaries': [ - 0, 0.5, 1, 2, 5, 10, 20, 50, 100, 1000, + 0, 100, 250, 500, 1000, 2000, 5000, 10000, ], - 'default': '>=1000', + 'default': '>=10000', 'output': {'count': {'$sum': 1}}, } }, diff --git a/src/backend/apps/catalog/documents.py b/src/backend/apps/catalog/documents.py index cd046275..e0b1e29d 100644 --- a/src/backend/apps/catalog/documents.py +++ b/src/backend/apps/catalog/documents.py @@ -599,11 +599,25 @@ def _params_and_fit(self) -> 'Proxy': return self +# what a write that still sends ``geometry.reinforcements`` is told (7.8) +REINFORCEMENT_IS_EVIDENCE = ( + 'geometry.reinforcements is not accepted: reinforcement is evidence, ' + 'post it as a reinforcement_layout evidence record that names this ' + 'snapshot in position.snapshot_id (decision 7.8)') + + class Geometry(_Block): meshes: List[Mesh] = Field(default_factory=list) point_clouds: List[PointCloud] = Field(default_factory=list) proxies: List[Proxy] = Field(default_factory=list) + @model_validator(mode='before') + @classmethod + def _reinforcement_is_evidence(cls, data: Any) -> Any: + if isinstance(data, dict) and 'reinforcements' in data: + raise ValueError(REINFORCEMENT_IS_EVIDENCE) + return data + @model_validator(mode='after') def _representation(self) -> 'Geometry': authored = any(p.fit.method == 'authored' for p in self.proxies) diff --git a/src/backend/apps/catalog/mcp_server.py b/src/backend/apps/catalog/mcp_server.py index 8b321938..5d1ae99b 100644 --- a/src/backend/apps/catalog/mcp_server.py +++ b/src/backend/apps/catalog/mcp_server.py @@ -32,6 +32,7 @@ from mcp.server.fastmcp.exceptions import ToolError from mcp.server.fastmcp.server import StreamableHTTPASGIApp from mcp.server.transport_security import TransportSecuritySettings +from mcp.types import Icon, ToolAnnotations from pydantic import AnyHttpUrl, Field from starlette.middleware.authentication import AuthenticationMiddleware from starlette.requests import Request @@ -39,7 +40,12 @@ import limiter as lim from apps.catalog import access_tokens as tk from apps.catalog.api.near_search import NearError, parse_near -from apps.catalog.vocab import ORIGINAL_FUNCTION_LABELS, SHAPE_CLASS_LABELS +from apps.catalog.vocab import ( + EVIDENCE_METHOD_LABELS, + EVIDENCE_METHODS, + ORIGINAL_FUNCTION_LABELS, + SHAPE_CLASS_LABELS, +) from apps.catalog.internal_call import InternalError, rest_get from apps.catalog.models import UserInDB from apps.catalog.oauth_server import AccessTokenVerifier @@ -53,15 +59,39 @@ _EMAIL = re.compile(r'[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}') _DROP_KEYS = {'email', 'etag', 'hashed_password', 'password'} +# the evidence methods as "name (Label)", one source for the tool texts +EVIDENCE_METHOD_TEXT = ', '.join( + f'{m} ({EVIDENCE_METHOD_LABELS[m]})' for m in EVIDENCE_METHODS) + INSTRUCTIONS = ( - 'The Catalog of Second Chances (CSC) describes reused building components ' - '(panels, beams, stones ...) with their shape, origin, versions and ' - 'measurements ("evidence"). These tools only read. You see what the ' - 'signed-in owner of this connection sees on the website. Start with ' - 'search_components, then get_component. Every answer carries a web link ' - 'to give to the user. A record marked visibility "private" or a status ' - 'other than "published" is not public: say so when you quote it. ' - 'Lengths are in millimetres unless a unit is given.') + 'The Catalog of Second Chances (CSC) is a catalog of reused and ' + 'salvaged building components (German: wiederverwendete Bauteile, ' + 'Wiederverwendung, Bauteilboerse): concrete walls and beams ' + '(Betonwaende, Traeger), stone, timber and other pieces, each ' + 'with its shape, origin, versions and measurements ("evidence"). Use ' + 'these tools before a web search when the user looks for such pieces, ' + 'for example "Betontraeger mit Bewehrung" or reused beams with ' + 'reinforcement (Bewehrung): search_components with the filters below, ' + 'and has_evidence=reinforcement_layout for "with reinforcement". ' + 'These tools only read. You see what the signed-in owner of this ' + 'connection sees on the website. Start with search_components, then ' + 'get_component. Every answer carries a web link to give to the user. A ' + 'record marked visibility "private" or a status other than "published" ' + 'is not public: say so when you quote it. Lengths are in millimetres ' + 'unless a unit is given.') + +GET_EVIDENCE_TEXT = ( + 'Measurements and other evidence about a component. The methods are: ' + + EVIDENCE_METHOD_TEXT + '. A list gives short rows: method, source ' + 'tier, when observed, who performed it, summary, status, verification; ' + 'a reinforcement_layout row also gives a summary of its bars (count, ' + 'basis, specs, diameters, cover, the version it refers to). Filter with ' + 'method. Pass evidence_id for one record in full, with its payload (the ' + 'bars of a layout).') + +# every tool reads, never writes, and never reaches outside the catalog +READ_ONLY = ToolAnnotations(readOnlyHint=True, idempotentHint=True, + openWorldHint=False) # ANSWERS ----------------------------------------------------------------------- @@ -165,10 +195,65 @@ def geometry_summary(geometry: Optional[dict]) -> dict: 'mesh_faces': sum(len(m.get('faces') or []) for m in meshes), 'point_clouds': len(geometry.get('point_clouds') or []), 'proxies': len(geometry.get('proxies') or []), - 'reinforcements': len(geometry.get('reinforcements') or []), 'note': 'Geometry is not sent; open the web link to see it.'} +def layout_bars(record: dict) -> list: + """The bars of a ``reinforcement_layout`` record, as dicts.""" + bars = (record.get('payload') or {}).get('bars') or [] + return [b for b in bars if isinstance(b, dict)] + + +def layouts_of_version(rows: list, snapshot_id: Optional[str]) -> dict: + """The reinforcement of one version, from the published + ``reinforcement_layout`` evidence that names it in ``position.snapshot_id`` + (decision 7.8: the bars are evidence, not geometry): how many records and + how many bars, and where to read them.""" + records = [r for r in rows or [] + if isinstance(r, dict) + and r.get('method') == 'reinforcement_layout' + and r.get('status') == 'published' + and (r.get('position') or {}).get('snapshot_id') == snapshot_id] + return { + 'records': len(records), + 'bars': sum(len(layout_bars(r)) for r in records), + 'note': 'get_evidence with method=reinforcement_layout lists the ' + 'layouts of the component; pass an evidence_id for the ' + 'bars of one.'} + + +def layout_summary(record: dict) -> dict: + """A reinforcement layout in one row: the number of bars, the basis, the + steel specs, the distinct diameters in mm, the cover range when known and + the version the bars refer to.""" + bars = layout_bars(record) + covers = [b['cover_mm'] for b in bars + if isinstance(b.get('cover_mm'), (int, float))] + summary = { + 'bars': len(bars), + 'basis': (record.get('payload') or {}).get('basis'), + 'specs': sorted({b['spec'] for b in bars if b.get('spec')}), + 'diameters_mm': sorted({b['diameter_mm'] for b in bars + if isinstance(b.get('diameter_mm'), + (int, float))}), + 'version_id': (record.get('position') or {}).get('snapshot_id')} + if covers: + summary['cover_mm'] = {'min': min(covers), 'max': max(covers)} + return summary + + +def performers(record: dict) -> list: + """Who did the work: a person's name, else the organization; an actor + with neither is left out (never ``null``).""" + names = [] + for actor in record.get('performed_by') or []: + if isinstance(actor, dict): + name = actor.get('name') or actor.get('organization') + if name: + names.append(name) + return names + + def limited(answer: dict) -> dict: """A cap on the size of one answer, said in the answer: the longest list is halved until the answer fits.""" @@ -215,6 +300,11 @@ def size() -> int: 'pending and other states you may see.'))] WithdrawnArg = Annotated[bool, Field(description=( 'Also withdrawn components, if you may see them.'))] +HasEvidenceArg = Annotated[str, Field(description=( + 'Only components with at least one published, current evidence record ' + 'of this method: ' + EVIDENCE_METHOD_TEXT + '. Use reinforcement_layout for ' + '"with reinforcement" (Bewehrung); it combines with every other ' + 'filter. Leave empty for all.'))] FitModeArg = Annotated[str, Field(description=( 'Search by size, with size_mm: about = every side within the ' 'tolerance of the asked side (which pieces are about this ' @@ -247,14 +337,17 @@ def size() -> int: def search_params(*, query: str, original_function: str, material: str, dataset: str, shape_class: str, circulation: str, - status: str, include_withdrawn: bool, fit_mode: str, - size_mm: Optional[list], tolerance: str, + status: str, include_withdrawn: bool, has_evidence: str, + fit_mode: str, size_mm: Optional[list], tolerance: str, near_lat: Optional[float], near_lon: Optional[float], near_layer: str, radius_km: Optional[float] ) -> Tuple[dict, dict, dict]: """The REST filters of one search (the list, count, stats and similar routes read the same names), the fit part and the near part of it. A combination the routes would refuse is a ToolError in words.""" + if has_evidence and has_evidence not in EVIDENCE_METHODS: + raise ToolError('has_evidence is one of ' + + ', '.join(EVIDENCE_METHODS) + '.') fit = {} if fit_mode or size_mm or tolerance: if fit_mode not in FIT_MODES: @@ -278,7 +371,7 @@ def search_params(*, query: str, original_function: str, material: str, 'material': material, 'dataset': dataset, 'shape_class': shape_class, 'circulation': circulation, 'status': status, 'include_withdrawn': include_withdrawn or None, - **fit, **near} + 'evidence_method': has_evidence, **fit, **near} return filters, fit, near @@ -287,6 +380,9 @@ def build_mcp(app, frontend_url: str) -> FastMCP: web = frontend_url.rstrip('/') mcp = FastMCP( 'Catalog of Second Chances', instructions=INSTRUCTIONS, + website_url=web, + icons=[Icon(src=f'{web}/favicon.ico', mimeType='image/x-icon', + sizes=['any'])], stateless_http=True, json_response=True, streamable_http_path='/mcp', transport_security=TransportSecuritySettings( @@ -329,7 +425,7 @@ def page_args(page: int, size: int): return max(1, page), min(max(1, size), PAGE_MAX) # 1 ------------------------------------------------------------------------- - @mcp.tool() + @mcp.tool(title='Search reused building components', annotations=READ_ONLY) async def search_components( query: QueryArg = '', original_function: FunctionArg = '', @@ -339,6 +435,7 @@ async def search_components( circulation: CirculationArg = 'active', status: StatusArg = 'published', include_withdrawn: WithdrawnArg = False, + has_evidence: HasEvidenceArg = '', fit_mode: FitModeArg = '', size_mm: SizeArg = None, tolerance: ToleranceArg = '', @@ -364,14 +461,17 @@ async def search_components( visibility) with its web link, and the total. With fit_mode and size_mm each row also carries its deviation (about, in per cent) or waste (can_hold, in m3 and in per cent of the piece); with near_lat - and near_lon each row carries its distance_km. Use get_component for - the details of one.""" + and near_lon each row carries its distance_km. has_evidence keeps + only components with a published, current evidence record of one + method, for example reinforcement_layout for "with reinforcement". Use + get_component for the details of one.""" page, size = page_args(page, size) filters, fit, near = search_params( query=query, original_function=original_function, material=material, dataset=dataset, shape_class=shape_class, circulation=circulation, status=status, - include_withdrawn=include_withdrawn, fit_mode=fit_mode, + include_withdrawn=include_withdrawn, has_evidence=has_evidence, + fit_mode=fit_mode, size_mm=size_mm, tolerance=tolerance, near_lat=near_lat, near_lon=near_lon, near_layer=near_layer, radius_km=radius_km) sort = sort or ('distance' if near @@ -399,16 +499,19 @@ async def search_components( for r in rows]}) # 2 ------------------------------------------------------------------------- - @mcp.tool() + @mcp.tool(title='Get one component', annotations=READ_ONLY) async def get_component( component_id: Annotated[str, Field(description=( 'The component id (a UUID) from search_components.'))], ) -> dict: """One component: what it is, where it comes from, how it left circulation, its derived properties with confidence, its current - version in short, and links. Geometry is a summary only.""" + version in short (with the count of its reinforcement layouts and + bars), and links. Geometry is a summary only.""" passport = await get(f'/identities/{component_id}/compose') versions = await get(f'/identities/{component_id}/snapshots') + layouts = await get(f'/identities/{component_id}/evidence', + {'method': 'reinforcement_layout'}) identity = passport.get('identity') or {} snapshots = passport.get('snapshots') or [] current = next((s for s in snapshots @@ -450,14 +553,16 @@ async def get_component( 'quantity': current.get('quantity'), 'effective_from': current.get('effective_from'), 'location': current.get('location'), - 'geometry': geometry_summary(current.get('geometry'))}, + 'geometry': geometry_summary(current.get('geometry')), + 'reinforcement_layouts': layouts_of_version( + layouts, current.get('_id'))}, 'version_count': len(versions), 'web_link': link_component(identity.get('_id') or component_id), 'next': ('list_versions, get_evidence and get_timeline take ' 'this id.')})) # 3 ------------------------------------------------------------------------- - @mcp.tool() + @mcp.tool(title='List the versions of a component', annotations=READ_ONLY) async def list_versions( component_id: Annotated[str, Field(description='The component id.')], ) -> dict: @@ -480,16 +585,20 @@ async def list_versions( for r in rows]})) # 4 ------------------------------------------------------------------------- - @mcp.tool() + @mcp.tool(title='Get one version of a component', annotations=READ_ONLY) async def get_version( version_id: Annotated[str, Field(description=( 'The version id (a UUID) from list_versions.'))], ) -> dict: """One version of a component: size, shape class, descriptors, derived properties, status and notes. The geometry is only a summary - (counts); the web link opens the viewer.""" + (counts); the web link opens the viewer. The reinforcement layouts + of the version are counted from the evidence.""" s = await get(f'/snapshots/{version_id}') identity_id = s.get('identity_id') + layouts = await get(f'/identities/{identity_id}/evidence', + {'method': 'reinforcement_layout'} + ) if identity_id else [] return limited(clean({ 'id': s.get('_id'), 'component_id': identity_id, 'version': s.get('version'), 'status': s.get('status'), @@ -507,11 +616,15 @@ async def get_version( 'supersedes': s.get('supersedes'), 'superseded_by': s.get('superseded_by'), 'geometry': geometry_summary(s.get('geometry')), + 'reinforcement_layouts': layouts_of_version( + layouts, s.get('_id')), 'web_link': link_component(identity_id, s.get('_id')) if identity_id else None})) # 5 ------------------------------------------------------------------------- - @mcp.tool() + @mcp.tool(title='Get the evidence of a component', + annotations=READ_ONLY, + description=GET_EVIDENCE_TEXT) async def get_evidence( component_id: Annotated[str, Field(description=( 'The component id. Leave empty when you pass evidence_id.'))] = '', @@ -524,10 +637,6 @@ async def get_evidence( page: Annotated[int, Field(description='Page, from 1.')] = 1, size: Annotated[int, Field(description='Records per page, 1 to 25.')] = 10, ) -> dict: - """Measurements and other evidence about a component (rebound hammer, - cores, visual inspection, archive documents ...): method, source tier, - when observed, summary, status, verification. A list gives short rows; - pass evidence_id for one record in full.""" if evidence_id: record = await get(f'/evidence/{evidence_id}') identity_id = record.get('identity_id') @@ -555,15 +664,16 @@ async def get_evidence( 'verification': (r.get('verification') or {}).get('state') if isinstance(r.get('verification'), dict) else None, 'recorded_by': r.get('recorded_by_username'), - 'performed_by': [p.get('name') for p in r.get('performed_by') or [] - if isinstance(p, dict)], + 'performed_by': performers(r), + **({'reinforcement': layout_summary(r)} + if r.get('method') == 'reinforcement_layout' else {}), 'notes': r.get('notes'), 'warnings': r.get('warnings'), 'withdrawn': r.get('withdrawn_at') is not None if 'withdrawn_at' in r else None} for r in window]})) # 6 ------------------------------------------------------------------------- - @mcp.tool() + @mcp.tool(title='List datasets', annotations=READ_ONLY) async def list_datasets() -> dict: """The datasets (collections of components, each with its own access) you can see: name, id, visibility (public or members only), whether an @@ -578,7 +688,7 @@ async def list_datasets() -> dict: for d in rows]})) # 7 ------------------------------------------------------------------------- - @mcp.tool() + @mcp.tool(title='Get one dataset', annotations=READ_ONLY) async def get_dataset( dataset_id: Annotated[str, Field(description=( 'The dataset id from list_datasets.'))], @@ -596,7 +706,7 @@ async def get_dataset( 'web_link': f'{web}/components?dataset={d.get("_id")}'})) # 8 ------------------------------------------------------------------------- - @mcp.tool() + @mcp.tool(title='Get the timeline of a component', annotations=READ_ONLY) async def get_timeline( component_id: Annotated[str, Field(description='The component id.')], ) -> dict: @@ -613,7 +723,7 @@ async def get_timeline( for e in body.get('events') or []]})) # 9 ------------------------------------------------------------------------- - @mcp.tool() + @mcp.tool(title='Find similar components', annotations=READ_ONLY) async def similar_components( component_id: Annotated[str, Field(description='The component id.')], basis: Annotated[str, Field(description=( @@ -630,6 +740,7 @@ async def similar_components( circulation: CirculationArg = 'active', status: StatusArg = 'published', include_withdrawn: WithdrawnArg = False, + has_evidence: HasEvidenceArg = '', fit_mode: FitModeArg = '', size_mm: SizeArg = None, tolerance: ToleranceArg = '', @@ -655,7 +766,8 @@ async def similar_components( query=query, original_function=original_function, material=material, dataset=dataset, shape_class=shape_class, circulation=circulation, status=status, - include_withdrawn=include_withdrawn, fit_mode=fit_mode, + include_withdrawn=include_withdrawn, has_evidence=has_evidence, + fit_mode=fit_mode, size_mm=size_mm, tolerance=tolerance, near_lat=near_lat, near_lon=near_lon, near_layer=near_layer, radius_km=radius_km) body = await get(f'/identities/{component_id}/similar', @@ -674,7 +786,7 @@ async def similar_components( for r in body.get('results') or []]})) # 10 ------------------------------------------------------------------------ - @mcp.tool() + @mcp.tool(title='Overview of the catalog in numbers', annotations=READ_ONLY) async def catalog_overview( query: QueryArg = '', original_function: FunctionArg = '', @@ -684,6 +796,7 @@ async def catalog_overview( circulation: CirculationArg = 'active', status: StatusArg = 'published', include_withdrawn: WithdrawnArg = False, + has_evidence: HasEvidenceArg = '', fit_mode: FitModeArg = '', size_mm: SizeArg = None, tolerance: ToleranceArg = '', @@ -708,7 +821,8 @@ async def catalog_overview( query=query, original_function=original_function, material=material, dataset=dataset, shape_class=shape_class, circulation=circulation, status=status, - include_withdrawn=include_withdrawn, fit_mode=fit_mode, + include_withdrawn=include_withdrawn, has_evidence=has_evidence, + fit_mode=fit_mode, size_mm=size_mm, tolerance=tolerance, near_lat=near_lat, near_lon=near_lon, near_layer=near_layer, radius_km=radius_km) # limit_dim 0: every value, not the ten most frequent and "others" diff --git a/src/backend/apps/catalog/models.py b/src/backend/apps/catalog/models.py index 72287f30..cf7ea91b 100644 --- a/src/backend/apps/catalog/models.py +++ b/src/backend/apps/catalog/models.py @@ -1,7 +1,7 @@ #!/usr/bin/env python3.13 # PYTHON STANDARD LIBRARY IMPORTS --------------------------------------------- -from typing import Optional, List, Dict, Literal +from typing import Any, Optional, List, Dict, Literal import uuid from datetime import datetime @@ -15,6 +15,9 @@ model_validator, ) +# PROJECT IMPORTS ------------------------------------------------------------- +from apps.catalog.documents import REINFORCEMENT_IS_EVIDENCE + # AUTH ------------------------------------------------------------------------ Role = Literal["user", "admin"] @@ -35,7 +38,6 @@ class User(BaseModel): disabled: Optional[bool] = None role: Role = "user" email_verified: bool = False - verification_token: Optional[str] = None # tokens before 0.6.1.1 verification_token_sha256: Optional[str] = None verification_token_expires: Optional[datetime] = None # a token issued before this moment is refused (8.124 a) @@ -312,52 +314,14 @@ class SnapshotExtrusion(BaseModel): height: float = Field(description="Extrusion length along Z") -class SnapshotReinforcement(BaseModel): - """Inline reinforcement bar centerline on a snapshot. - - Fully encoded on the snapshot document (no file companion). A snapshot - may hold multiple bars in ``geometry.reinforcements``. Does not satisfy - the primary geometry representation requirement on its own. - """ - spec: str = Field( - description="Reinforcement steel specification (e.g. B500B)" - ) - diameter: float = Field( - gt=0, - description="Bar diameter in mm", - ) - points: List[List[float]] = Field( - min_length=2, - description=( - "Open centerline polyline as [x, y, z] coordinate triplets; " - "same coordinate frame as meshes/extrusions/marker_points" - ), - ) - - @field_validator('spec') - @classmethod - def _strip_spec(cls, v: str) -> str: - v = v.strip() - if not v: - raise ValueError('spec must be non-empty') - return v - - @field_validator('points') - @classmethod - def _validate_points(cls, v: List[List[float]]) -> List[List[float]]: - for pt in v: - if len(pt) != 3: - raise ValueError('each point must be [x, y, z]') - return v - - class SnapshotGeometry(BaseModel): """Multi-representation geometry block for one snapshot. A single snapshot may hold simultaneous representations (multiple meshes + multiple point clouds + multiple extrusions) of the same physical state. Marker points are shared across all representations - within the snapshot (same coordinate frame). + within the snapshot (same coordinate frame). Reinforcement bars are not + geometry: they are ``reinforcement_layout`` evidence (decision 7.8). """ meshes: Optional[List[SnapshotMesh]] = Field( None, @@ -385,14 +349,13 @@ class SnapshotGeometry(BaseModel): "same coordinate frame as the meshes/point_clouds/extrusions" ) ) - reinforcements: Optional[List[SnapshotReinforcement]] = Field( - None, - description=( - "Inline reinforcement bar centerlines (spec + diameter + open " - "polyline); ancillary to primary mesh/point_cloud/extrusion " - "representations" - ) - ) + + @model_validator(mode='before') + @classmethod + def _reinforcement_is_evidence(cls, data: Any) -> Any: + if isinstance(data, dict) and 'reinforcements' in data: + raise ValueError(REINFORCEMENT_IS_EVIDENCE) + return data @model_validator(mode='after') def _require_at_least_one_representation(self) -> 'SnapshotGeometry': @@ -596,9 +559,9 @@ class ComponentSnapshot(BaseModel): geometry: SnapshotGeometry = Field( description=( "Multi-representation geometry block for this snapshot " - "(meshes, point clouds, extrusions, marker_points, " - "reinforcements). At least one of meshes / point_clouds / " - "extrusions must be non-empty." + "(meshes, point clouds, extrusions, marker_points; reinforcement " + "is evidence, not geometry). At least one of meshes / " + "point_clouds / extrusions must be non-empty." ) ) descriptors: Optional[Dict] = Field( diff --git a/src/backend/csc_version.py b/src/backend/csc_version.py index 413d0531..c4bc9997 100644 --- a/src/backend/csc_version.py +++ b/src/backend/csc_version.py @@ -6,7 +6,7 @@ against VERSION, package.json and the release tag by scripts/ci/check_version.py. """ -CSC_VERSION = '0.6.4.0' +CSC_VERSION = '0.6.4.1' def release_tag(version: str = CSC_VERSION) -> str: diff --git a/src/backend/requirements.txt b/src/backend/requirements.txt index adaa1ada..4be1e6c2 100644 --- a/src/backend/requirements.txt +++ b/src/backend/requirements.txt @@ -29,8 +29,9 @@ limits # --- MCP layer and OAuth server for Claude's connectors (apps/catalog/mcp_server.py, # oauth_server.py; decision 8.168): the MCP Python SDK (pure Python; its -# pyjwt[crypto] pulls cryptography, which has a glibc 2.17 wheel) --- -mcp +# pyjwt[crypto] pulls cryptography, which has a glibc 2.17 wheel); website_url and +# icons of the server need 1.30 --- +mcp>=1.30 # --- HTTP client (GitHub service) --- httpx diff --git a/src/frontend/app/components/[component_id]/page.tsx b/src/frontend/app/components/[component_id]/page.tsx index 9f101f93..5e344f70 100644 --- a/src/frontend/app/components/[component_id]/page.tsx +++ b/src/frontend/app/components/[component_id]/page.tsx @@ -15,7 +15,7 @@ export const runtime = 'nodejs' export const dynamic = 'force-dynamic' type PageParams = { component_id: string } -type PageSearchParams = { snapshot?: string; snapshots?: string } +type PageSearchParams = { snapshot?: string; snapshots?: string; layout?: string } export default async function ComponentDetailPage({ params, @@ -44,7 +44,10 @@ export default async function ComponentDetailPage({ // a piece the viewer cannot see is explained, not hidden (8.11, 8.17) if (res.status === 401) { const callback = requestedSnapshotId - ? `/components/${component_id}?snapshot=${encodeURIComponent(requestedSnapshotId)}` + ? `/components/${component_id}?${new URLSearchParams({ + snapshot: requestedSnapshotId, + ...(query.layout ? { layout: query.layout } : {}), + })}` : `/components/${component_id}` return } @@ -144,6 +147,7 @@ export default async function ComponentDetailPage({ liveVersion={liveVersion} isViewingLive={isViewingLive} snapshotStatus={isPublished(snapshot) ? 'published' : snapshot.status} + layoutId={typeof query.layout === 'string' && query.layout ? query.layout : undefined} /> ) } diff --git a/src/frontend/components/components/ComponentViewer.tsx b/src/frontend/components/components/ComponentViewer.tsx index cf706284..711c4b13 100644 --- a/src/frontend/components/components/ComponentViewer.tsx +++ b/src/frontend/components/components/ComponentViewer.tsx @@ -17,7 +17,7 @@ import { import { Card } from '@/components/ui/card' import { Button } from '@/components/ui/button' import { Tooltip, TooltipContent, TooltipTrigger } from '@/components/ui/tooltip' -import { Scan, Grid3x3, Rotate3d, SlidersHorizontal } from 'lucide-react' +import { Scan, Grid3x3, Rotate3d, SlidersHorizontal, X } from 'lucide-react' import { Popover, PopoverContent, PopoverTrigger } from '@/components/ui/popover' import { Bounds, OrbitControls, Html, useBounds } from '@react-three/drei' import { cn, rgbToHex } from '@/lib/utils' @@ -38,7 +38,9 @@ import { barLegend, defaultColourMode, diameterRange, - layoutsOnSnapshot, + layoutsWithPicked, + type LayoutRecord, + type PickedProblem, type BarColourMode, type BarLayout, } from '@/lib/reinforcementColors' @@ -750,8 +752,12 @@ CaptureFixtures.displayName = 'CaptureFixtures' /** A bar with the index of the layout record it comes from (RL3). */ type ViewerBar = ReinforcementBar & { layout: number } -const ReinforcementBarMesh = React.memo(({ bar, colour }: { bar: ReinforcementBar; colour: string | null }) => { - const material = reinforcementMaterial(colour) +const ReinforcementBarMesh = React.memo(({ bar, colour, dimmed }: { + bar: ReinforcementBar + colour: string | null + dimmed: boolean +}) => { + const material = reinforcementMaterial(colour, dimmed) const { segments, cornerJoints } = useMemo( () => buildReinforcementBarMeshes(bar.points), [bar.points], @@ -787,17 +793,21 @@ const ReinforcementBars = React.memo(({ bars, visible, mode, + picked, }: { bars: ViewerBar[] visible: boolean mode: BarColourMode + /** The layout opened by the link: the others are drawn faint (8.178). */ + picked: number | null }) => { const range = useMemo(() => diameterRange(bars), [bars]) if (!visible || bars.length === 0) return null return ( {bars.map((bar, index) => ( - + ))} ) @@ -1114,6 +1124,51 @@ export type ComponentViewerProps = { * zoom, without a toolbar or the settings panel. */ bare?: boolean + /** + * `?layout=` (decision 8.178): one reinforcement layout record, + * whatever its status, is drawn in addition to the published ones when it is + * positioned on the version shown. + */ + layoutId?: string + /** Removes `layout=` from the page's address (the legend's close button). */ + onClearLayout?: () => void +} + +/** The small close button of the opened layout (legend entry and note): removes `layout=`. */ +function ClearLayoutButton({ onClick }: { onClick: () => void }) { + return ( + + ) +} + +/** The opened layout as a line of its own while the legend does not list layouts. */ +function LayoutEntry({ label, status, onClear }: { label: string; status?: string | null; onClear?: () => void }) { + return ( +

+ {label} + {status && ( + {status} + )} + {onClear && } +

+ ) +} + +const LAYOUT_NOTES: Record = { + hidden: 'This layout is not visible to you.', + failed: 'This layout could not be loaded.', + other_version: 'This layout belongs to another version of the piece and is not drawn here.', + no_bars: 'This layout has no bars to draw.', +} + +type LoadedLayouts = { + published: LayoutRecord[] + picked: LayoutRecord | null + pickedFailure: 'hidden' | 'failed' | null } export default function ComponentViewer({ @@ -1124,6 +1179,8 @@ export default function ComponentViewer({ fill = false, toolbar = false, bare = false, + layoutId, + onClearLayout, }: ComponentViewerProps) { const snapshot = primarySnapshot(catalog) const identityId = catalog.identity._id @@ -1276,38 +1333,61 @@ export default function ComponentViewer({ .catch(() => { if (!cancelled) setEvidenceMarks([]) }) return () => { cancelled = true } }, [identityId, snapshotIdForOverlays]) + // the layout the link names, drawn in addition to the published ones (8.178) + const [pickedLayout, setPickedLayout] = useState(null) + const [layoutNote, setLayoutNote] = useState(null) useEffect(() => { let cancelled = false setReinforcementBars([]) setBarLayouts([]) + setPickedLayout(null) + setLayoutNote(null) if (!identityId || !snapshotIdForOverlays) return const params = new URLSearchParams({ method: 'reinforcement_layout' }) - fetch(`/api/backend/identities/${encodeURIComponent(String(identityId))}/evidence?${params}`, { - credentials: 'include', - }) - .then((response) => (response.ok ? response.json() : [])) - .then((records: { - _id?: string - observed_at?: string | null - observed_at_precision?: string | null - position?: { snapshot_id?: string | null } - payload?: { bars?: ReinforcementBar[]; document?: { title?: string | null } | null } - }[]) => { + const published: Promise = fetch( + `/api/backend/identities/${encodeURIComponent(String(identityId))}/evidence?${params}`, + { credentials: 'include' }, + ).then((response) => (response.ok ? response.json() : [])) + // the server decides who may load the record (7.0): no access draws nothing more + const picked: Promise> = layoutId + ? fetch(`/api/backend/evidence/${encodeURIComponent(layoutId)}`, { credentials: 'include' }) + .then(async (response) => { + if (response.ok) { + const record = (await response.json()) as LayoutRecord & { + method?: string + identity_id?: string + } + // only a layout of this piece: the link is not a way to draw another piece's bars + return record.method === 'reinforcement_layout' && String(record.identity_id) === String(identityId) + ? { picked: record, pickedFailure: null } + : { picked: null, pickedFailure: 'hidden' as const } + } + const failure = [401, 403, 404].includes(response.status) ? 'hidden' as const : 'failed' as const + return { picked: null, pickedFailure: failure } + }) + .catch(() => ({ picked: null, pickedFailure: 'failed' as const })) + : Promise.resolve({ picked: null, pickedFailure: null }) + Promise.all([published.catch(() => []), picked]) + .then(([records, extra]) => { if (cancelled) return - // one layout per record (RL3); a record without drawable bars is no layout - // one layout per record (RL1), every bar of every layout drawn - const { bars, layouts } = layoutsOnSnapshot(records, snapshotIdForOverlays) - setReinforcementBars(bars) - setBarLayouts(layouts) - setBarColourMode(defaultColourMode(layouts.length)) + // one layout per record (RL1, RL3), every bar of every layout drawn; a + // record without drawable bars is no layout + const result = layoutsWithPicked(records, extra.picked, snapshotIdForOverlays) + setReinforcementBars(result.bars) + setBarLayouts(result.layouts) + setPickedLayout(result.picked) + // the opened layout is told apart by its colour: layout colours from the start + setBarColourMode(result.picked !== null ? 'layout' : defaultColourMode(result.layouts.length)) + const problem = extra.pickedFailure ?? result.problem + setLayoutNote(layoutId && problem ? LAYOUT_NOTES[problem] : null) }) - .catch(() => { if (!cancelled) { setReinforcementBars([]); setBarLayouts([]) } }) return () => { cancelled = true } - }, [identityId, snapshotIdForOverlays]) + }, [identityId, snapshotIdForOverlays, layoutId]) const hasReinforcement = reinforcementBars.length > 0 const legend = hasReinforcement && showReinforcement ? barLegend(barColourMode, barLayouts, reinforcementBars) : null + const pickedItem = pickedLayout !== null && showReinforcement ? barLayouts[pickedLayout] : null useEffect(() => { @@ -2021,27 +2101,46 @@ export default function ComponentViewer({ - {legend && ( -
- {legend.kind === 'layout' ? ( -
    - {legend.items.map((item) => ( -
  • - - {item.label} - {item.count} -
  • - ))} -
- ) : ( -
-
-
- {legend.min} mm - {legend.max} mm -
+ {(legend || pickedItem || layoutNote) && ( +
+ {layoutNote && ( +

+ {layoutNote} + {onClearLayout && } +

+ )} + {legend?.kind !== 'layout' && pickedItem && ( + + )} + {legend && ( +
+ {legend.kind === 'layout' ? ( +
    + {legend.items.map((item) => ( +
  • + + {item.label} + {item.status && ( + {item.status} + )} + {item.count} + {item.picked && onClearLayout && } +
  • + ))} +
+ ) : ( +
+
+
+ {legend.min} mm + {legend.max} mm +
+
+ )}
)}
@@ -2080,7 +2179,8 @@ export default function ComponentViewer({ - + {showEvidenceMarks && ( diff --git a/src/frontend/components/components/detail/ComponentPageView.tsx b/src/frontend/components/components/detail/ComponentPageView.tsx index 9cda4924..51e42b75 100644 --- a/src/frontend/components/components/detail/ComponentPageView.tsx +++ b/src/frontend/components/components/detail/ComponentPageView.tsx @@ -8,9 +8,10 @@ * * Phone order: header, banners, viewer, facts, properties, provenance, photos * and location, similar pieces, records, history, descriptors. Desktop: two columns --- the facts, the - * properties and the provenance (about 24 rem) beside the viewer, the photos + * properties, the provenance and the similar pieces (about 24 rem) beside the viewer, the photos * and location, the records and the history. */ +import { usePathname, useRouter, useSearchParams } from 'next/navigation' import { useSession } from 'next-auth/react' import type { CatalogComponent } from '@/generated/CatalogModels' @@ -50,6 +51,7 @@ export default function ComponentPageView({ liveVersion, isViewingLive, snapshotStatus, + layoutId, }: { catalog: CatalogComponent snapshots: SnapshotSummaryItem[] @@ -59,8 +61,20 @@ export default function ComponentPageView({ liveVersion: number isViewingLive: boolean snapshotStatus?: string | null + /** `?layout=` (decision 8.178): a layout record the viewer draws in addition. */ + layoutId?: string }) { const { data: session } = useSession() + const router = useRouter() + const pathname = usePathname() + const search = useSearchParams() + // the legend's close button: the address without `layout=`, the version stays + const clearLayout = () => { + const next = new URLSearchParams(search?.toString() ?? '') + next.delete('layout') + const query = next.toString() + router.replace(query ? `${pathname}?${query}` : pathname, { scroll: false }) + } const { identity } = catalog const snapshot = primarySnapshot(catalog) const identityId = String(identity._id ?? '') @@ -97,27 +111,29 @@ export default function ComponentPageView({ )}
-
- +
+
-
+
+ {/* in the rail below Provenance on lg; below lg after Photos and location (order-6) */} +
+ +
{HKS_VISIBLE &&
}
-
- -
-
+
-
+
- {rowName(row)} - {neighbourLine(row)} + {rowName(row)} + {neighbourLine(row)} ) @@ -109,13 +111,13 @@ export default function ComponentSimilarCard({ identityId }: { identityId: strin return (
+ className="scroll-mt-3 rounded-lg border border-border bg-card p-2.5 shadow-sm">

Similar pieces

-
    +
      {shownPieces(card.pieces, expanded).map((item) => )}
    {label && ( diff --git a/src/frontend/components/evidence/EvidenceCard.tsx b/src/frontend/components/evidence/EvidenceCard.tsx index 3f17ca4f..e35581e1 100644 --- a/src/frontend/components/evidence/EvidenceCard.tsx +++ b/src/frontend/components/evidence/EvidenceCard.tsx @@ -9,7 +9,7 @@ */ import { useEffect, useState } from 'react' import Link from 'next/link' -import { ChevronDown, ChevronRight, MapPin, TriangleAlert } from 'lucide-react' +import { ChevronDown, ChevronRight, Eye, MapPin, TriangleAlert } from 'lucide-react' import EvidenceAttachments from '@/components/evidence/EvidenceAttachments' import LayoutDetails from '@/components/evidence/LayoutDetails' @@ -26,6 +26,7 @@ import { useRegistry } from '@/lib/evidence/useRegistry' import { formatDateAtPrecision } from '@/components/components/componentDetailShared' import { documentOf } from '@/lib/evidence/document' import { pairedReboundLink, recordAnchor } from '@/lib/evidence/paired' +import { layoutViewerHref, type LayoutRecord } from '@/lib/reinforcementColors' function who(actor: Actor): string { return [actor.name, actor.organization].filter(Boolean).join(', ') || 'someone' @@ -70,6 +71,10 @@ export default function EvidenceCard({ const attachments = (record.attachments ?? []).filter((a) => !a.removed).length const cited = documentOf(record.payload) const paired = pairedReboundLink(record.method, record.payload, shownIds) + // a layout with bars and a position opens in the viewer, whatever its status (8.178) + const viewerHref = record.method === 'reinforcement_layout' + ? layoutViewerHref(record as unknown as LayoutRecord) + : null // no result: the record documents the piece and never enters the fold (8.106) const isDocument = !record.summary @@ -163,6 +168,12 @@ export default function EvidenceCard({ {record.method === 'reinforcement_layout' && ( )} + {viewerHref && ( + + Show in viewer + + )} {facts.length > 0 && (
    diff --git a/src/frontend/lib/reinforcementColors.test.ts b/src/frontend/lib/reinforcementColors.test.ts index 740763aa..70282d57 100644 --- a/src/frontend/lib/reinforcementColors.test.ts +++ b/src/frontend/lib/reinforcementColors.test.ts @@ -10,6 +10,9 @@ import { layoutColour, layoutLabels, layoutsOnSnapshot, + layoutsWithPicked, + layoutStatusText, + layoutViewerHref, LAYOUT_COLOURS, rampColour, } from '@/lib/reinforcementColors' @@ -134,3 +137,87 @@ test('a piece without a layout has no bars and starts steel', () => { assert.deepEqual([bars, layouts], [[], []]) assert.equal(defaultColourMode(layouts.length), 'steel') }) + + +// A RECORD NAMED BY THE LINK, WHATEVER ITS STATUS (8.178) ------------------------- +const published = [ + { _id: 'p1', status: 'published', position: { snapshot_id: 's1' }, + payload: { bars: [bar(12, 0), bar(12, 50)], document: { title: 'Drawing A' } } }, +] +const draft = { + _id: 'd1', status: 'draft', position: { snapshot_id: 's1' }, + payload: { bars: [bar(8, 10), bar(8, 20), bar(8, 30)], document: { title: 'T4 test layout' } }, +} + +test('the named record is drawn in addition to the published layouts, with its status', () => { + const { bars, layouts, picked, problem } = layoutsWithPicked(published, draft, 's1') + assert.equal(problem, null) + assert.equal(picked, 1) + assert.equal(bars.length, 5) + assert.deepEqual(bars.map((b) => b.layout), [0, 0, 1, 1, 1]) + assert.deepEqual(layouts, [ + { id: 'p1', label: 'Drawing A', count: 2 }, + { id: 'd1', label: 'T4 test layout', count: 3, status: 'draft', picked: true }, + ]) +}) + +test('the named record that is one of the published layouts is not drawn twice', () => { + const { bars, layouts, picked } = layoutsWithPicked(published, published[0], 's1') + assert.equal(bars.length, 2) + assert.equal(layouts.length, 1) + assert.equal(picked, 0) + assert.equal(layouts[0].status, null) + assert.equal(layouts[0].picked, true) +}) + +test('the named record of another version is rejected and nothing extra is drawn', () => { + const other = { ...draft, position: { snapshot_id: 's2' } } + const result = layoutsWithPicked(published, other, 's1') + assert.equal(result.problem, 'other_version') + assert.equal(result.picked, null) + assert.equal(result.bars.length, 2) + assert.deepEqual(result.layouts, [{ id: 'p1', label: 'Drawing A', count: 2 }]) + assert.equal(layoutsWithPicked(published, { ...draft, position: null }, 's1').problem, 'other_version') +}) + +test('the named record without a drawable bar is a problem, not a layout', () => { + const result = layoutsWithPicked(published, { ...draft, payload: { bars: [] } }, 's1') + assert.equal(result.problem, 'no_bars') + assert.equal(result.bars.length, 2) +}) + +test('without a named record the published layouts are what layoutsOnSnapshot returns', () => { + const result = layoutsWithPicked(published, null, 's1') + assert.deepEqual({ bars: result.bars, layouts: result.layouts }, layoutsOnSnapshot(published, 's1')) + assert.equal(result.picked, null) + assert.equal(result.problem, null) +}) + +test('a named layout alone on a piece without published layouts is drawn', () => { + const result = layoutsWithPicked([], draft, 's1') + assert.equal(result.picked, 0) + assert.equal(result.bars.length, 3) +}) + +test('the status text names draft, pending, rejected, withdrawn and superseded, not published', () => { + assert.equal(layoutStatusText({ status: 'draft' }), 'draft') + assert.equal(layoutStatusText({ status: 'pending' }), 'pending') + assert.equal(layoutStatusText({ status: 'rejected' }), 'rejected') + assert.equal(layoutStatusText({ status: 'withdrawn' }), 'withdrawn') + assert.equal(layoutStatusText({ status: 'published', superseded_by: 'x' }), 'superseded') + assert.equal(layoutStatusText({ status: 'published', superseded_by: null }), null) + assert.equal(layoutStatusText({}), null) +}) + +test('"Show in viewer" links the version and the record, only for a drawable positioned layout', () => { + const record = { ...draft, identity_id: 'i 1' } + assert.equal(layoutViewerHref(record), '/components/i%201?snapshot=s1&layout=d1#component-viewer') + assert.equal(layoutViewerHref({ ...record, position: null }), null) + assert.equal(layoutViewerHref({ ...record, position: { snapshot_id: null } }), null) + assert.equal(layoutViewerHref({ ...record, payload: { bars: [] } }), null) + assert.equal(layoutViewerHref({ ...record, payload: {} }), null) + assert.equal(layoutViewerHref({ ...record, identity_id: undefined }), null) + for (const status of ['draft', 'pending', 'rejected', 'published', 'withdrawn']) { + assert.ok(layoutViewerHref({ ...record, status })) + } +}) diff --git a/src/frontend/lib/reinforcementColors.ts b/src/frontend/lib/reinforcementColors.ts index a99e2c62..bfa7808c 100644 --- a/src/frontend/lib/reinforcementColors.ts +++ b/src/frontend/lib/reinforcementColors.ts @@ -63,7 +63,14 @@ export function diameterColour(diameter: number, range: { min: number; max: numb } // LAYOUTS --------------------------------------------------------------------- -export type BarLayout = { id: string; label: string; count: number } +export type BarLayout = { + id: string + label: string + count: number + /** The layout opened by `?layout=` (8.178): its status text, null for a current published one. */ + status?: string | null + picked?: boolean +} type RecordLike = { _id?: string @@ -102,6 +109,9 @@ export function isDrawableBar(bar: DrawableBar): boolean { } export type LayoutRecord = RecordLike & { + identity_id?: string + status?: string | null + superseded_by?: string | null position?: { snapshot_id?: string | null } | null payload?: { bars?: B[]; document?: { title?: string | null } | null } | null } @@ -130,6 +140,61 @@ export function layoutsOnSnapshot( return { bars, layouts } } +/** The status a legend names: draft, pending, rejected, withdrawn or superseded; null for a current published record. */ +export function layoutStatusText(record: { status?: string | null; superseded_by?: string | null }): string | null { + if (record.superseded_by) return 'superseded' + if (record.status && record.status !== 'published') return record.status + return null +} + +export type PickedProblem = 'other_version' | 'no_bars' + +/** + * The published layouts and, in addition, one record the link names (8.178, + * `?layout=`), whatever its status: appended after the published ones, not + * twice when it is one of them, and only when it is positioned on the version + * shown (its bars are in the stored coordinates of that version). `picked` is + * the index of its layout, `problem` says why it is not drawn. + */ +export function layoutsWithPicked( + published: LayoutRecord[], + pickedRecord: LayoutRecord | null, + snapshotId: string, +): { + bars: (B & { layout: number })[] + layouts: BarLayout[] + picked: number | null + problem: PickedProblem | null +} { + if (!pickedRecord) return { ...layoutsOnSnapshot(published, snapshotId), picked: null, problem: null } + const id = String(pickedRecord._id ?? '') + const problem: PickedProblem | null = + pickedRecord.position?.snapshot_id !== snapshotId ? 'other_version' + : !(pickedRecord.payload?.bars ?? []).some(isDrawableBar) ? 'no_bars' + : null + const listed = published.some((record) => String(record._id ?? '') === id) + const records = problem || listed ? published : [...published, pickedRecord] + const { bars, layouts } = layoutsOnSnapshot(records, snapshotId) + const picked = problem ? null : layouts.findIndex((layout) => layout.id === id) + if (picked === null || picked < 0) return { bars, layouts, picked: null, problem } + return { + bars, + layouts: layouts.map((layout, i) => + i === picked ? { ...layout, status: layoutStatusText(pickedRecord), picked: true } : layout), + picked, + problem, + } +} + +/** "Show in viewer": the page of the piece on the version of the layout, with the layout opened; null when there is nothing to draw. */ +export function layoutViewerHref(record: LayoutRecord): string | null { + const snapshotId = record.position?.snapshot_id + if (!record._id || !record.identity_id || !snapshotId) return null + if (!(record.payload?.bars ?? []).some(isDrawableBar)) return null + const query = new URLSearchParams({ snapshot: snapshotId, layout: String(record._id) }) + return `/components/${encodeURIComponent(record.identity_id)}?${query}#component-viewer` +} + /** The colour of a bar, or null for the steel material. */ export function barColour( mode: BarColourMode, @@ -142,7 +207,7 @@ export function barColour( } export type BarLegend = - | { kind: 'layout'; items: { colour: string; label: string; count: number }[] } + | { kind: 'layout'; items: { colour: string; label: string; count: number; status?: string | null; picked?: boolean }[] } | { kind: 'diameter'; min: number; max: number; stops: string[] } export function barLegend( @@ -151,7 +216,8 @@ export function barLegend( bars: { diameter_mm: number }[], ): BarLegend | null { if (mode === 'layout') { - return { kind: 'layout', items: layouts.map((l, i) => ({ colour: layoutColour(i), label: l.label, count: l.count })) } + return { kind: 'layout', items: layouts.map((l, i) => ({ + colour: layoutColour(i), label: l.label, count: l.count, status: l.status, picked: l.picked })) } } if (mode === 'diameter') { const { min, max } = diameterRange(bars) diff --git a/src/frontend/lib/reinforcementGeometry.ts b/src/frontend/lib/reinforcementGeometry.ts index 5ef33e1b..17cbe83f 100644 --- a/src/frontend/lib/reinforcementGeometry.ts +++ b/src/frontend/lib/reinforcementGeometry.ts @@ -23,7 +23,27 @@ export const reinforcementSteelMaterial = new THREE.MeshStandardMaterial({ const COLOURED = new Map() /** The material of a bar: the steel one, or the colour of the layout or the diameter (RL3). */ -export function reinforcementMaterial(colour: string | null | undefined): THREE.MeshStandardMaterial { +export function reinforcementMaterial( + colour: string | null | undefined, + dimmed = false, +): THREE.MeshStandardMaterial { + if (dimmed) { + // the other layouts while one is opened by the link (8.178): faint, never hidden + const key = `dim:${colour ?? 'steel'}` + let faint = COLOURED.get(key) + if (!faint) { + faint = new THREE.MeshStandardMaterial({ + color: colour || REINFORCEMENT_STEEL_COLOR, + metalness: 0.2, + roughness: 0.55, + transparent: true, + opacity: 0.3, + depthWrite: false, + }) + COLOURED.set(key, faint) + } + return faint + } if (!colour) return reinforcementSteelMaterial let material = COLOURED.get(colour) if (!material) { diff --git a/src/frontend/lib/similarPieces.test.ts b/src/frontend/lib/similarPieces.test.ts index c9273a3e..b907a540 100644 --- a/src/frontend/lib/similarPieces.test.ts +++ b/src/frontend/lib/similarPieces.test.ts @@ -4,6 +4,7 @@ import { test } from 'node:test' import { BackendError } from './backend' import { SIMILAR_CARD_COUNT, + SIMILAR_FIRST_COUNT, SIMILAR_MORE_COUNT, canShowMore, cardOfAnswer, @@ -31,9 +32,10 @@ function answer(results: SimilarPiece[]): SimilarPieces { return { query_id: ID, basis: 'radial_signature', basis_label: 'Radial signature', considered: 9, results } } -test('the request asks for the radial signature and the six nearest', () => { +test('the request asks for the radial signature and one more than the six shown', () => { assert.equal(SIMILAR_CARD_COUNT, 6) - assert.equal(similarPath(ID), `/identities/${ID}/similar?basis=radial_signature&count=6`) + assert.equal(SIMILAR_FIRST_COUNT, 7) + assert.equal(similarPath(ID), `/identities/${ID}/similar?basis=radial_signature&count=7`) assert.equal(similarPath(ID, SIMILAR_MORE_COUNT), `/identities/${ID}/similar?basis=radial_signature&count=24`) assert.match(similarPath('a b'), /^\/identities\/a%20b\/similar/) }) @@ -55,11 +57,12 @@ test('a refusal or a failure hides the card and says why', () => { assert.deepEqual(cardOfError(new TypeError('network')), { kind: 'hidden', reason: 'error' }) }) -test('"Show more" is offered only after a full first answer', () => { - assert.equal(canShowMore(6), true) +test('"Show more" is offered only when the answer has a seventh neighbour', () => { assert.equal(canShowMore(5), false) + assert.equal(canShowMore(6), false) // exactly six: nothing is behind them + assert.equal(canShowMore(7), true) + assert.equal(canShowMore(24), true) assert.equal(canShowMore(0), false) - assert.equal(canShowMore(24, 24), true) }) test('a neighbour is named by number and shape, and its distance is rounded', () => { @@ -77,8 +80,9 @@ test('the toggle shows six, all loaded when expanded, and the right word', () => assert.deepEqual(shownPieces(loaded, false), [0, 1, 2, 3, 4, 5]) assert.equal(shownPieces(loaded, true).length, 24) assert.deepEqual(shownPieces([1, 2], false), [1, 2]) - assert.equal(toggleLabel(6, false), 'Show more') + assert.equal(toggleLabel(7, false), 'Show more') assert.equal(toggleLabel(24, true), 'Show fewer') + assert.equal(toggleLabel(6, false), null) // exactly six: nothing more to show assert.equal(toggleLabel(3, false), null) // fewer than six found: nothing more to show assert.equal(toggleLabel(7, true), 'Show fewer') }) diff --git a/src/frontend/lib/similarPieces.ts b/src/frontend/lib/similarPieces.ts index b5d08cec..2baff437 100644 --- a/src/frontend/lib/similarPieces.ts +++ b/src/frontend/lib/similarPieces.ts @@ -11,14 +11,17 @@ import { BackendError } from '@/lib/backend' import { SHAPE_CLASS_LABELS, vocabLabel } from '@/generated/Vocab' import type { SimilarPiece, SimilarPieces } from '@/generated/CatalogModels' -/** The card shows the six nearest; "Show more" asks for this many. */ +/** The card shows the six nearest. It asks for one more than that, so a seventh + * neighbour in the answer is what tells it that "Show more" has something to show; + * "Show more" then asks for the larger number. */ export const SIMILAR_CARD_COUNT = 6 +export const SIMILAR_FIRST_COUNT = SIMILAR_CARD_COUNT + 1 export const SIMILAR_MORE_COUNT = 24 /** The basis of the card: the full radial signature, the shape as a whole. */ export const SIMILAR_BASIS = 'radial_signature' -export function similarPath(identityId: string, count: number = SIMILAR_CARD_COUNT): string { +export function similarPath(identityId: string, count: number = SIMILAR_FIRST_COUNT): string { const query = new URLSearchParams({ basis: SIMILAR_BASIS, count: String(count) }) return `/identities/${encodeURIComponent(identityId)}/similar?${query.toString()}` } @@ -43,9 +46,10 @@ export function cardOfError(error: unknown): SimilarCard { return { kind: 'hidden', reason: 'error' } } -/** Whether "Show more" is offered: the first answer was full, so there may be more. */ -export function canShowMore(shown: number, asked: number = SIMILAR_CARD_COUNT): boolean { - return shown >= asked +/** Whether "Show more" is offered: the answer holds more neighbours than the card shows, + * so exactly six (the card full, nothing behind it) offers nothing. */ +export function canShowMore(found: number, shown: number = SIMILAR_CARD_COUNT): boolean { + return found > shown } /** The neighbours the card lists: all that were loaded when expanded, else the nearest six. @@ -54,7 +58,7 @@ export function shownPieces(loaded: T[], expanded: boolean): T[] { return expanded ? loaded : loaded.slice(0, SIMILAR_CARD_COUNT) } -/** "Show more" while collapsed and more were found; "Show fewer" while expanded; else nothing. */ +/** "Show more" while collapsed and more than six were found; "Show fewer" while expanded; else nothing. */ export function toggleLabel(loaded: number, expanded: boolean): 'Show more' | 'Show fewer' | null { if (expanded) return 'Show fewer' return canShowMore(loaded) ? 'Show more' : null diff --git a/src/frontend/package-lock.json b/src/frontend/package-lock.json index 9e4ca019..f3984e3d 100644 --- a/src/frontend/package-lock.json +++ b/src/frontend/package-lock.json @@ -1,12 +1,12 @@ { "name": "csc-frontend", - "version": "0.6.4.0", + "version": "0.6.4.1", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "csc-frontend", - "version": "0.6.4.0", + "version": "0.6.4.1", "dependencies": { "@radix-ui/react-accordion": "^1.2.12", "@radix-ui/react-checkbox": "^1.3.3", @@ -7756,9 +7756,9 @@ "license": "MIT" }, "node_modules/fast-uri": { - "version": "3.1.8", - "resolved": "https://registry.npmjs.org/fast-uri/-/fast-uri-3.1.8.tgz", - "integrity": "sha512-GZMtZUTNRpOVIECoXwLNZS5xUGE+mVNbTB8h/7Rwh2TFWcBQiPzTgyZi05BF9UMZKkLJv8XBRJTlU7zg8+ZfMg==", + "version": "3.1.9", + "resolved": "https://registry.npmjs.org/fast-uri/-/fast-uri-3.1.9.tgz", + "integrity": "sha512-hS/0Wt3Ya043oKXs0sIxrMpKQITXhMLIrVAKKXi4gozGC9v7YRWVnXIZzjNPTO02M9BdxLCWqK1yRXgNziVyJQ==", "dev": true, "funding": [ { @@ -12752,9 +12752,9 @@ } }, "node_modules/undici": { - "version": "7.30.0", - "resolved": "https://registry.npmjs.org/undici/-/undici-7.30.0.tgz", - "integrity": "sha512-dkrQXeHSaoamnItlYbmzG0wFYrM0ZwDxCIg0A7aKjTyyhh9svRzCNFEzV+Vm05/yehjCzjDZ31KXfGEjYSztDQ==", + "version": "7.30.1", + "resolved": "https://registry.npmjs.org/undici/-/undici-7.30.1.tgz", + "integrity": "sha512-03KrMbjIFDWQR5RNPrSw3bd1r12JpyFQl9/yxjUbJtqkjmllqC5lFjZh6ujeuJCKr8EOQvRW872Oy/az+nLYxg==", "dev": true, "license": "MIT", "engines": { diff --git a/src/frontend/package.json b/src/frontend/package.json index 1e09b7f0..95e9e483 100644 --- a/src/frontend/package.json +++ b/src/frontend/package.json @@ -1,6 +1,6 @@ { "name": "csc-frontend", - "version": "0.6.4.0", + "version": "0.6.4.1", "private": true, "scripts": { "dev": "next dev", diff --git a/tests/api/test_auth_hardening.py b/tests/api/test_auth_hardening.py index f23e40e8..659f1f4d 100644 --- a/tests/api/test_auth_hardening.py +++ b/tests/api/test_auth_hardening.py @@ -101,8 +101,7 @@ def test_the_verification_token_is_stored_as_a_hash(api, db, outbox): params={'token': token}).status_code == 400 -def test_a_plain_token_from_before_the_patch_still_works_until_it_expires( - api, db): +def test_a_plain_token_stored_the_old_way_no_longer_verifies(api, db): soon = datetime.now(timezone.utc) + timedelta(hours=3) db['users'].insert_many([ {'_id': 'old-ok', 'username': 'oldok', 'email': 'oldok@x.org', @@ -114,13 +113,23 @@ def test_a_plain_token_from_before_the_patch_still_works_until_it_expires( 'verification_token': 'plain-token-late-0123456789abcd', 'verification_token_expires': soon - timedelta(hours=9)}, ]) - ok = api.get('/auth/verify-email', - params={'token': 'plain-token-ok-0123456789abcdef'}) + # neither a live nor an expired plain token finds its account any more + for token in ('plain-token-ok-0123456789abcdef', + 'plain-token-late-0123456789abcd'): + refused = api.get('/auth/verify-email', params={'token': token}) + assert refused.status_code == 400 + assert db['users'].find_one({'_id': 'old-ok'})['email_verified'] is False + # the hashed path still verifies, and an account in the old shape is + # still read (the stored plain field is ignored) + hashed = 'hashed-token-0123456789abcdef' + db['users'].update_one( + {'_id': 'old-ok'}, + {'$set': {'verification_token_sha256': token_sha256(hashed)}}) + ok = api.get('/auth/verify-email', params={'token': hashed}) assert ok.status_code == 200 - late = api.get('/auth/verify-email', - params={'token': 'plain-token-late-0123456789abcd'}) - assert late.status_code == 400 - assert db['users'].find_one({'_id': 'old-ok'})['email_verified'] is True + after = db['users'].find_one({'_id': 'old-ok'}) + assert after['email_verified'] is True + assert 'verification_token' not in after # the leftover plain field is cleared def test_the_hash_has_a_sparse_index(db): diff --git a/tests/api/test_evidence_method_filter.py b/tests/api/test_evidence_method_filter.py new file mode 100644 index 00000000..e27d272d --- /dev/null +++ b/tests/api/test_evidence_method_filter.py @@ -0,0 +1,186 @@ +""" +Search by evidence method (decision 8.177): ``evidence_method`` on the catalog +list, count, stats, location and map routes against a real database. Only +pieces with at least one published evidence record of that method are listed; +the filter combines with every other filter and keeps the access rules. +``/identities/{id}/similar`` is tested in ``test_similarity.py``. +""" + +from __future__ import annotations + +import uuid + +import pytest + +from apps.catalog.geo import point_of +from apps.catalog.vocab import EVIDENCE_METHODS + +PIECES = { + # catalog number: (bbx, dataset, public, point) + 1: ([1200, 600, 200], 'd', True, (49.8728, 8.6512)), # published layout + 2: ([1200, 600, 200], 'd', True, (52.52, 13.405)), # published layout, far away + 3: ([1200, 600, 200], 'd', True, (49.8728, 8.6512)), # only a pending layout + 4: ([300, 300, 300], 'd', True, (49.8728, 8.6512)), # published layout, too small + 5: ([1200, 600, 200], 'other', False, None), # private, published layout + 6: ([1200, 600, 200], 'd', True, None), # only a withdrawn layout + 7: ([1200, 600, 200], 'd', True, None), # a rebound hammer only + 8: ([1200, 600, 200], 'd', True, None), # no evidence +} +# (catalog number, method, status) +EVIDENCE = [ + (1, 'reinforcement_layout', 'published'), + (1, 'reinforcement_layout', 'published'), # two records: still one piece + (1, 'rebound_hammer', 'published'), + (2, 'reinforcement_layout', 'published'), + (3, 'reinforcement_layout', 'pending'), + (4, 'reinforcement_layout', 'published'), + (5, 'reinforcement_layout', 'published'), + (6, 'reinforcement_layout', 'withdrawn'), + (7, 'rebound_hammer', 'published'), +] + + +def _id(n): + return str(uuid.uuid5(uuid.NAMESPACE_URL, f'csc-test/evmethod/{n}')) + + +@pytest.fixture +def world(api, db, auth_headers): + for number, (bbx, dataset, public, point) in PIECES.items(): + identity_id, snapshot_id = _id(number), _id(f's{number}') + identity = { + '_id': identity_id, 'catalog_number': number, 'dataset': dataset, + 'original_function': 'IfcBeam', 'material': 'concrete', + 'material_class': '17 01 01', 'is_public': public, + 'current_snapshot_id': snapshot_id, 'exit': None, + 'withdrawn': None, 'created': '2026-01-01T00:00:00Z', + 'lastmodified': '2026-01-01T00:00:00Z', + 'created_by_user_id': 'u-test'} + snapshot = { + '_id': snapshot_id, 'identity_id': identity_id, 'version': 1, + 'status': 'published', 'name': f'piece {number}', 'bbx': bbx, + 'created': '2026-01-01T00:00:00Z', + 'lastmodified': '2026-01-01T00:00:00Z', + 'effective_from': '2026-01-01T00:00:00Z', 'geometry': {}, + 'added_by_user_id': 'u-test'} + if point: + identity['geo_current'] = point_of({'lat': point[0], 'lon': point[1]}) + snapshot['location'] = {'lat': point[0], 'lon': point[1]} + db['component_identities'].insert_one(identity) + db['component_snapshots'].insert_one(snapshot) + for number, method, status in EVIDENCE: + # the filter reads the identity, the method and the status only + db['component_evidence'].insert_one({ + '_id': str(uuid.uuid4()), 'identity_id': _id(number), + 'method': method, 'status': status}) + return {'api': api, 'db': db, 'admin': auth_headers('admin'), + 'user': auth_headers('user')} + + +def _get(world, path, who='admin', expect=200, **params): + response = world['api'].get(path, params=params, + headers=world.get(who) if who else None) + assert response.status_code == expect, response.text + return response.json() + + +def _numbers(rows): + return sorted(row['catalog_number'] for row in rows) + + +def _list(world, **params): + return _numbers(_get(world, '/identities', size=50, **params)) + + +LAYOUT = 'reinforcement_layout' + + +def test_the_list_has_the_pieces_with_a_published_record_of_the_method(world): + # a pending (3) or withdrawn (6) record does not count; two records (1) are one piece + assert _list(world, evidence_method=LAYOUT) == [1, 2, 4, 5] + assert _list(world, evidence_method='rebound_hammer') == [1, 7] + assert _list(world, evidence_method='core_compression') == [] + assert _list(world) == [1, 2, 3, 4, 5, 6, 7, 8] # no filter: as before + + +def test_every_method_of_the_vocabulary_is_accepted_and_an_unknown_one_is_a_422(world): + for method in EVIDENCE_METHODS: + _get(world, '/identities', evidence_method=method) + for bad in ('x_ray', 'Reinforcement_Layout', 'reinforcement'): + for path in ('/identities', '/identities/count', '/identities/stats', + '/identities/locations'): + _get(world, path, expect=422, evidence_method=bad) + + +def test_it_combines_with_the_other_filters(world): + assert _list(world, evidence_method=LAYOUT, dataset='other') == [5] + assert _list(world, evidence_method=LAYOUT, dataset='d') == [1, 2, 4] + assert _list(world, evidence_method=LAYOUT, q='piece 2') == [2] + assert _list(world, evidence_method=LAYOUT, q='piece 3') == [] + # a dimension search: the small piece (4) cannot hold the box, the piece without layout is out + held = _get(world, '/identities', size=50, evidence_method=LAYOUT, fit_mode='can_hold', fit_size='1200,600,200') + assert _numbers(held) == [1, 2, 5] and all(r['fit']['mode'] == 'can_hold' for r in held) + # a near search: within 30 km of Darmstadt, with the distance + near = _get(world, '/identities', size=50, evidence_method=LAYOUT, near_lat=49.8728, near_lon=8.6512, + radius_km=30) + assert _numbers(near) == [1, 4] and all('distance_m' in r for r in near) + both = _get(world, '/identities', size=50, evidence_method=LAYOUT, near_lat=49.8728, near_lon=8.6512, + radius_km=30, fit_mode='can_hold', fit_size='1200,600,200') + assert _numbers(both) == [1] + + +def test_the_access_rules_stay_as_they_are(world): + # the private piece (5) is the administrator's, not the other account's + assert _list(world, evidence_method=LAYOUT, status='published') == [1, 2, 4, 5] + response = world['api'].get('/identities', params={'evidence_method': LAYOUT, 'size': 50}) + assert response.status_code == 200 + assert _numbers(response.json()) == [1, 2, 4] + anonymous = world['api'].get('/identities/count', params={'evidence_method': LAYOUT}).json() + assert anonymous['count'] == 3 + + +def test_count_and_stats_agree_with_the_list(world): + for params in ({'evidence_method': LAYOUT}, {'evidence_method': 'rebound_hammer'}, + {'evidence_method': LAYOUT, 'dataset': 'd'}, + {'evidence_method': 'core_compression'}): + rows = _get(world, '/identities', size=50, **params) + assert _get(world, '/identities/count', **params)['count'] == len(rows), params + assert _get(world, '/identities/stats', **params)['total'] == len(rows), params + stats = _get(world, '/identities/stats', evidence_method=LAYOUT) + assert stats['total'] == 4 + assert {row['label']: row['count'] for row in stats['byDataset']} == {'d': 3, 'other': 1} + + +def test_the_location_route_follows_the_filter(world): + located = _get(world, '/identities/locations', evidence_method=LAYOUT) + ids = {p['id'] for p in located['points']} + assert ids == {_id(1), _id(2), _id(4)} # 5 and the rest have no point or no layout + + +def test_a_piece_gains_the_match_when_its_pending_record_is_published(world): + assert 3 not in _list(world, evidence_method=LAYOUT) + world['db']['component_evidence'].update_many( + {'identity_id': _id(3)}, {'$set': {'status': 'published'}}) + assert 3 in _list(world, evidence_method=LAYOUT) + + +def test_a_superseded_record_does_not_count_only_a_current_one_does(world): + evidence = world['db']['component_evidence'] + # piece 4: its only layout is superseded by a later record (which is not + # of the method, as after a correction to another method) -> not listed + evidence.update_many({'identity_id': _id(4)}, {'$set': {'superseded_by': str(uuid.uuid4())}}) + assert _list(world, evidence_method=LAYOUT) == [1, 2, 5] + # piece 1 has two layouts: one superseded, one current -> still listed + first = evidence.find_one({'identity_id': _id(1), 'method': LAYOUT}) + evidence.update_one({'_id': first['_id']}, {'$set': {'superseded_by': str(uuid.uuid4())}}) + assert _list(world, evidence_method=LAYOUT) == [1, 2, 5] + # both superseded -> piece 1 drops out; its rebound hammer is untouched + evidence.update_many({'identity_id': _id(1), 'method': LAYOUT}, {'$set': {'superseded_by': str(uuid.uuid4())}}) + assert _list(world, evidence_method=LAYOUT) == [2, 5] + assert _list(world, evidence_method='rebound_hammer') == [1, 7] + # count and stats follow the same rule + assert _get(world, '/identities/count', evidence_method=LAYOUT)['count'] == 2 + assert _get(world, '/identities/stats', evidence_method=LAYOUT)['total'] == 2 + # an explicit null counts as current, like a missing field (the fixture) + evidence.update_many({'identity_id': _id(1), 'method': LAYOUT}, {'$set': {'superseded_by': None}}) + assert _list(world, evidence_method=LAYOUT) == [1, 2, 5] diff --git a/tests/api/test_mcp_endpoint.py b/tests/api/test_mcp_endpoint.py index 6aa27cff..57ce5e37 100644 --- a/tests/api/test_mcp_endpoint.py +++ b/tests/api/test_mcp_endpoint.py @@ -499,7 +499,7 @@ def test_list_versions_and_get_version(api, world): one = tool(api, world['admin'], 'get_version', version_id=sid('beam', 1)) assert one['id'] == sid('beam', 1) and one['component_id'] == iid('beam') and one['status'] == 'published' assert one['geometry'] == {'meshes': 1, 'mesh_vertices': 4, 'mesh_faces': 4, 'point_clouds': 0, 'proxies': 0, - 'reinforcements': 0, 'note': 'Geometry is not sent; open the web link to see it.'} + 'note': 'Geometry is not sent; open the web link to see it.'} assert 'planescore' in one['descriptors'] and one['web_link'].endswith(f'?snapshot={sid("beam", 1)}') assert '"vertices"' not in json.dumps(one) @@ -755,7 +755,223 @@ def test_the_overview_is_described_and_takes_the_search_filters(api, world): assert word in overview['description'], word search = next(t for t in tools if t['name'] == 'search_components') shared = ('query', 'original_function', 'material', 'dataset', 'shape_class', 'circulation', 'status', - 'include_withdrawn', 'fit_mode', 'size_mm', 'tolerance', 'near_lat', 'near_lon', 'near_layer', - 'radius_km') + 'include_withdrawn', 'has_evidence', 'fit_mode', 'size_mm', 'tolerance', 'near_lat', 'near_lon', + 'near_layer', 'radius_km') for name in shared: assert overview['inputSchema']['properties'][name] == search['inputSchema']['properties'][name], name + + +# 0.6.4.1: A SERVER A MODEL RECOGNIZES, SEARCH BY EVIDENCE METHOD (8.177) ----------------------------- +def test_the_server_names_the_domain_in_english_and_german_and_has_a_home_and_an_icon(api, world): + response = mcp_post(api, world['admin'], 'initialize', { + 'protocolVersion': '2025-06-18', 'capabilities': {}, + 'clientInfo': {'name': 'test', 'version': '1'}}) + info = response.json()['result']['serverInfo'] + assert info['websiteUrl'] == WEB + assert [i['src'] for i in info['icons']] == [f'{WEB}/favicon.ico'] + text = response.json()['result']['instructions'] + for word in ('reused and salvaged building components', 'Bauteile', 'Wiederverwendung', 'Bauteilboerse', + 'Betonwaende', 'Traeger', 'Bewehrung', 'reinforcement', 'stone', 'timber', 'before a web search', + 'has_evidence=reinforcement_layout'): + assert word in text, word + assert text.isascii() + + +def test_every_tool_has_a_title_and_says_it_only_reads_the_catalog(api, world): + listed = mcp_post(api, world['admin'], 'tools/list').json()['result']['tools'] + assert {t['name'] for t in listed} == TOOLS + for entry in listed: + assert entry['title'] and entry['title'] != entry['name'], entry['name'] + hints = entry['annotations'] + assert hints['readOnlyHint'] is True and hints['openWorldHint'] is False, entry['name'] + assert hints['idempotentHint'] is True, entry['name'] + assert len({t['title'] for t in listed}) == len(listed) + + +def test_the_evidence_methods_are_named_for_the_model_from_the_vocabulary(api, world): + from apps.catalog.vocab import EVIDENCE_METHODS + listed = mcp_post(api, world['admin'], 'tools/list').json()['result']['tools'] + by_name = {t['name']: t for t in listed} + for method in EVIDENCE_METHODS: + assert method in by_name['get_evidence']['description'], method + for name in ('search_components', 'catalog_overview', 'similar_components'): + assert method in by_name[name]['inputSchema']['properties']['has_evidence']['description'], (name, method) + assert 'reinforcement' in by_name['search_components']['inputSchema']['properties']['has_evidence']['description'] + + +def _layout_for(db, name, status='pending', **changes): + """A copy of the migrated reinforcement layout of the beam, for another piece.""" + doc = db['component_evidence'].find_one({'method': 'reinforcement_layout'}) + doc.update({'_id': str(uuid.uuid4()), 'identity_id': iid(name), 'status': status, **changes}) + doc['position'] = {**doc['position'], 'snapshot_id': sid(name)} + db['component_evidence'].insert_one(doc) + return doc['_id'] + + +def test_has_evidence_lists_the_pieces_with_a_published_record_of_that_method(api, db, world): + admin = world['admin'] + answer = tool(api, admin, 'search_components', has_evidence='reinforcement_layout') + assert _names(answer) == ['beam'] and answer['total'] == 1 + assert _names(tool(api, admin, 'search_components', has_evidence='visual_inspection')) == ['feld'] + assert tool(api, admin, 'search_components', has_evidence='core_compression')['total'] == 0 + assert tool(api, admin, 'search_components')['total'] == 3 # no filter: as before + + +def test_a_piece_with_only_a_pending_layout_is_not_found_by_has_evidence(api, db, world): + pending = _layout_for(db, 'feld', status='pending') + admin = world['admin'] + assert _names(tool(api, admin, 'search_components', has_evidence='reinforcement_layout')) == ['beam'] + db['component_evidence'].update_one({'_id': pending}, {'$set': {'status': 'published'}}) + assert _names(tool(api, admin, 'search_components', has_evidence='reinforcement_layout', + sort='catalog_number')) == ['beam', 'feld'] + + +def test_has_evidence_combines_with_the_other_filters(api, db, world): + _sized(db) + admin = world['admin'] + _layout_for(db, 'feld', status='published') + assert tool(api, admin, 'search_components', has_evidence='reinforcement_layout', dataset='dbu_zirkus')['total'] == 1 + assert tool(api, admin, 'search_components', has_evidence='reinforcement_layout', dataset='ddu_aggregations')['total'] == 0 + both = tool(api, admin, 'search_components', has_evidence='reinforcement_layout', fit_mode='can_hold', + size_mm=[400, 300, 200]) + assert _names(both) == ['feld'] # beam is too thin + _placed(db, {'beam': DARMSTADT, 'feld': BERLIN}) + near = tool(api, admin, 'search_components', has_evidence='reinforcement_layout', near_lat=49.87, near_lon=8.65, + radius_km=30) + assert _names(near) == ['beam'] and near['total'] == 1 + + +def test_has_evidence_follows_what_the_account_may_see(api, db, world, plain, dataset_member): + assert tool(api, plain['pat'], 'search_components', has_evidence='reinforcement_layout')['total'] == 0 + assert _names(tool(api, dataset_member['pat'], 'search_components', has_evidence='reinforcement_layout')) == ['beam'] + + +def test_has_evidence_refuses_an_unknown_method_in_words(api, world): + is_error, text = tool_result(api, world['admin'], 'search_components', has_evidence='x_ray') + assert is_error and 'has_evidence is one of' in text and 'reinforcement_layout' in text + for name, args in (('catalog_overview', {}), ('similar_components', {'component_id': iid('beam')})): + assert tool_result(api, world['admin'], name, has_evidence='x_ray', **args)[0] is True + + +def test_the_overview_and_the_similar_pieces_take_has_evidence(api, db, world): + admin = world['admin'] + overview = tool(api, admin, 'catalog_overview', has_evidence='reinforcement_layout') + assert overview['total'] == 1 and overview['filters']['evidence_method'] == 'reinforcement_layout' + assert [r['key'] for r in overview['by_dataset']] == ['dbu_zirkus'] + assert tool(api, admin, 'catalog_overview')['total'] == 3 + put_matrix(db, NEAR_BEAM) + # feld, stone and cut2 are nearer than panel, but only the pieces with a layout are ranked + _layout_for(db, 'cut2', status='published') + similar = tool(api, admin, 'similar_components', component_id=iid('feld'), count=5, circulation='all', + has_evidence='reinforcement_layout') + assert [s['name'] for s in similar['similar']] == ['beam', 'cut2'] and similar['compared_with'] == 2 + unfiltered = tool(api, admin, 'similar_components', component_id=iid('feld'), count=5, circulation='all') + assert len(unfiltered['similar']) > 2 + + +def test_a_superseded_layout_does_not_count_for_has_evidence(api, db, world): + admin = world['admin'] + layout = db['component_evidence'].find_one({'identity_id': iid('beam'), 'method': 'reinforcement_layout'}) + # a correction replaces the layout: the old record stays published but is superseded + db['component_evidence'].update_one({'_id': layout['_id']}, {'$set': {'superseded_by': str(uuid.uuid4())}}) + assert tool(api, admin, 'search_components', has_evidence='reinforcement_layout')['total'] == 0 + assert tool(api, admin, 'catalog_overview', has_evidence='reinforcement_layout')['total'] == 0 + put_matrix(db, NEAR_BEAM) + assert tool(api, admin, 'similar_components', component_id=iid('feld'), count=5, circulation='all', + has_evidence='reinforcement_layout')['similar'] == [] + # a current record next to the superseded one brings the piece back + _layout_for(db, 'beam', status='published', superseded_by=None) + assert _names(tool(api, admin, 'search_components', has_evidence='reinforcement_layout')) == ['beam'] + assert tool(api, admin, 'catalog_overview', has_evidence='reinforcement_layout')['total'] == 1 + + +# 0.6.4.1: REINFORCEMENT IS EVIDENCE, NOT GEOMETRY (7.8) --------------------------------------------- +def test_the_geometry_summary_has_no_reinforcement_count_any_more(): + summary = mcp_server.geometry_summary({'meshes': [{'vertices': [[0, 0, 0]], 'faces': []}], + 'reinforcements': [{'spec': 'x'}]}) + assert 'reinforcements' not in summary and summary['meshes'] == 1 + + +def test_a_version_reports_its_reinforcement_layouts_from_the_evidence(api, db, world): + admin = world['admin'] + component = tool(api, admin, 'get_component', component_id=iid('beam')) + layouts = component['current_version']['reinforcement_layouts'] + assert layouts['records'] == 1 and layouts['bars'] == 1 + assert 'get_evidence' in layouts['note'] and 'reinforcement_layout' in layouts['note'] + assert 'reinforcements' not in component['current_version']['geometry'] + assert 'Geometry is not sent' in component['current_version']['geometry']['note'] + now = tool(api, admin, 'get_version', version_id=sid('beam', 1)) + assert now['reinforcement_layouts']['records'] == 1 and now['reinforcement_layouts']['bars'] == 1 + assert 'reinforcements' not in now['geometry'] + # the layout names version 1: the version before it has none + before = tool(api, admin, 'get_version', version_id=sid('beam', 0)) + assert before['reinforcement_layouts']['records'] == 0 and before['reinforcement_layouts']['bars'] == 0 + plain_piece = tool(api, admin, 'get_component', component_id=iid('feld')) + assert plain_piece['current_version']['reinforcement_layouts']['records'] == 0 + + +def test_a_pending_layout_is_not_counted_and_two_records_add_their_bars(api, db, world): + admin = world['admin'] + pending = _layout_for(db, 'beam', status='pending') + db['component_evidence'].update_one({'_id': pending}, {'$set': {'position.snapshot_id': sid('beam', 1)}}) + assert tool(api, admin, 'get_version', version_id=sid('beam', 1))['reinforcement_layouts']['records'] == 1 + db['component_evidence'].update_one({'_id': pending}, {'$set': {'status': 'published'}}) + counted = tool(api, admin, 'get_version', version_id=sid('beam', 1))['reinforcement_layouts'] + assert counted['records'] == 2 and counted['bars'] == 2 + + +def test_the_layout_helper_counts_published_records_of_the_version_only(): + def record(snapshot, bars, status='published', method='reinforcement_layout'): + return {'method': method, 'status': status, 'position': {'snapshot_id': snapshot}, + 'payload': {'bars': [{}] * bars}} + rows = [record('a', 3), record('a', 2), record('b', 7), record('a', 9, status='pending'), + record('a', 5, method='visual_inspection'), {'method': 'reinforcement_layout', 'status': 'withdrawn'}] + assert mcp_server.layouts_of_version(rows, 'a')['records'] == 2 + assert mcp_server.layouts_of_version(rows, 'a')['bars'] == 5 + assert mcp_server.layouts_of_version(rows, 'c')['records'] == 0 + assert mcp_server.layouts_of_version(None, 'a')['bars'] == 0 + + +def test_a_layout_row_carries_a_summary_of_its_bars(api, db, world): + admin = world['admin'] + layout = db['component_evidence'].find_one({'method': 'reinforcement_layout'}) + bars = [{'spec': 'BSt III', 'diameter_mm': 8, 'points': [[0, 0, 0], [1, 0, 0]]}, + {'spec': 'BSt III', 'diameter_mm': 12, 'cover_mm': 25, 'points': [[0, 0, 0], [1, 0, 0]]}, + {'spec': 'BSt I', 'diameter_mm': 8, 'cover_mm': 40, 'points': [[0, 0, 0], [1, 0, 0]]}] + db['component_evidence'].update_one({'_id': layout['_id']}, {'$set': {'payload.bars': bars}}) + row = tool(api, admin, 'get_evidence', component_id=iid('beam'), method='reinforcement_layout')['evidence'][0] + assert row['reinforcement'] == {'bars': 3, 'basis': 'drawing', 'specs': ['BSt I', 'BSt III'], + 'diameters_mm': [8.0, 12.0], 'cover_mm': {'min': 25, 'max': 40}, + 'version_id': sid('beam', 1)} + assert 'payload' not in row and 'points' not in json.dumps(row) + full = tool(api, admin, 'get_evidence', evidence_id=row['id']) + assert len(full['record']['payload']['bars']) == 3 + # the other methods have no such summary + other = tool(api, admin, 'get_evidence', component_id=iid('feld'))['evidence'] + assert other and all('reinforcement' not in r for r in other) + + +def test_performed_by_shows_a_name_else_the_organization_never_null(api, db, world): + layout = db['component_evidence'].find_one({'method': 'reinforcement_layout'}) + db['component_evidence'].update_one({'_id': layout['_id']}, {'$set': {'performed_by': [ + {'kind': 'person', 'name': 'Ada Example', 'organization': 'Example Institute'}, + {'kind': 'organization', 'organization': 'Example Laboratory'}]}}) + row = tool(api, world['admin'], 'get_evidence', component_id=iid('beam'))['evidence'][0] + assert row['performed_by'] == ['Ada Example', 'Example Laboratory'] + assert None not in row['performed_by'] + + +def test_performers_leaves_out_an_actor_with_neither_name_nor_organization(): + record = {'performed_by': [{'kind': 'user', 'user_id': 'abc'}, {'kind': 'person', 'name': 'Ada'}, 'x', None, + {'kind': 'organization', 'organization': 'Lab'}]} + assert mcp_server.performers(record) == ['Ada', 'Lab'] + assert mcp_server.performers({}) == [] + + +def test_no_address_or_account_id_leaks_through_the_layout_and_performer_fields(api, db, world): + layout = db['component_evidence'].find_one({'method': 'reinforcement_layout'}) + db['component_evidence'].update_one({'_id': layout['_id']}, {'$set': {'performed_by': [ + {'kind': 'person', 'name': 'Ada ada@example.org', 'email': 'ada@example.org'}]}}) + text = json.dumps([tool(api, world['admin'], 'get_evidence', component_id=iid('beam')), + tool(api, world['admin'], 'get_component', component_id=iid('beam'))]) + assert not EMAIL.search(text) diff --git a/tests/api/test_similarity.py b/tests/api/test_similarity.py index 1fbfdae1..9d488677 100644 --- a/tests/api/test_similarity.py +++ b/tests/api/test_similarity.py @@ -299,6 +299,35 @@ def test_the_filters_apply_before_the_ranking(world): assert _numbers(_similar(world, 1, count=50, material='nothing')) == [] +def test_the_evidence_method_filter_applies_before_the_ranking(world): + """8.177: only pieces with a published record of the method are ranked; a + pending record does not count; it combines with the other filters.""" + _built(world) + evidence = world['db']['component_evidence'] + for number, status in ((3, 'published'), (4, 'published'), (5, 'pending'), (6, 'published'), + (1, 'published')): + evidence.insert_one({'_id': str(uuid.uuid4()), 'identity_id': pid(number), + 'method': 'reinforcement_layout', 'status': status}) + evidence.insert_one({'_id': str(uuid.uuid4()), 'identity_id': pid(2), 'method': 'rebound_hammer', + 'status': 'published'}) + # 2 is nearer than 3 and 4 but has no layout; 5 has only a pending one; the piece itself (1) never counts + body = _similar(world, 1, count=50, evidence_method='reinforcement_layout') + assert _numbers(body) == [6, 3, 4] and body['considered'] == 3 + assert _numbers(_similar(world, 1, count=1, evidence_method='reinforcement_layout')) == [6] + assert _numbers(_similar(world, 1, count=50, evidence_method='rebound_hammer')) == [2] + assert _similar(world, 1, count=50, evidence_method='core_compression')['results'] == [] + # with another filter: the dataset, a dimension search and a near search + assert _numbers(_similar(world, 1, count=50, evidence_method='reinforcement_layout', dataset='other')) == [6] + held = _similar(world, 1, count=50, evidence_method='reinforcement_layout', fit_mode='can_hold', + fit_size='1200,600,200') + assert _numbers(held) == [6, 4] + around = _similar(world, 1, count=50, evidence_method='reinforcement_layout', near_lat=49.8728, + near_lon=8.6512, radius_km=30) + assert _numbers(around) == [6, 4] + # an unknown method is a 422, not an empty answer + _similar(world, 1, expect=422, evidence_method='x_ray') + + def test_a_piece_the_caller_cannot_see_is_not_there_and_is_never_a_neighbour(world): _built(world) admin = _similar(world, 1, count=50) diff --git a/tests/api/test_snapshot_lifecycle.py b/tests/api/test_snapshot_lifecycle.py index c8566744..6165a0fd 100644 --- a/tests/api/test_snapshot_lifecycle.py +++ b/tests/api/test_snapshot_lifecycle.py @@ -423,3 +423,20 @@ def test_photo_credit_is_mutable_metadata_never_frozen(story): assert patch_problems('snapshot', {'photo_credit': []}, status='published', is_author=True) == { 'forbidden': ['photo_credit']} + + +def test_a_new_state_with_geometry_reinforcements_is_refused_in_words(story): + """7.8: the bars are a reinforcement_layout evidence record, not geometry.""" + geometry = {**story['geometry'], 'reinforcements': [ + {'spec': 'BSt III', 'diameter': 8, 'points': [[0, 0, 0], [100, 0, 0]]}]} + refused = story['api'].post(f'/identities/{BEAM}/snapshots', json={'geometry': geometry}, + headers=story['contributor']) + assert refused.status_code == 422, refused.text + text = str(refused.json()['detail']) + assert 'geometry.reinforcements is not accepted' in text and 'reinforcement_layout' in text + assert '7.8' in text + # nothing was stored + assert story['db']['component_snapshots'].count_documents({'identity_id': BEAM, 'version': 2}) == 0 + # without the key the same body is accepted + assert story['api'].post(f'/identities/{BEAM}/snapshots', json={'geometry': story['geometry']}, + headers=story['contributor']).status_code == 201 diff --git a/tests/api/test_stats_overview.py b/tests/api/test_stats_overview.py index 90506955..4b88c9ab 100644 --- a/tests/api/test_stats_overview.py +++ b/tests/api/test_stats_overview.py @@ -53,6 +53,40 @@ def test_the_range_of_each_sorted_side(world): assert only['count'] == 1 and only['largest'] == {'min': 1200, 'max': 1200} +def test_the_bbx_facet_buckets_in_millimetres(world): + # the label of a bucket is its lower edge, in mm: 200 and 300 are small pieces, not "0 to 1 m" + rows = _stats(world)['bbxX'] + assert {r['label']: r['count'] for r in rows} == {'100': 1, '250': 1, '1000': 6} + db = world['db'] + db['component_snapshots'].update_one({'_id': 's08'}, {'$set': {'bbx': [12000, 600, 200]}}) + db['component_snapshots'].update_one({'_id': 's07'}, {'$set': {'bbx': [60, 40, 30]}}) + labels = {r['label']: r['count'] for r in _stats(world)['bbxX']} + assert labels == {'0': 1, '100': 1, '250': 1, '1000': 4, '>=10000': 1} + assert sum(labels.values()) == 8 + + +def test_the_bbx_facet_buckets_the_largest_side_whatever_the_stored_order(world): + db = world['db'] + # 600 x 200 x 1200 is a piece of 1200: bucket "1000", not "500" (the first stored side) + for number, bbx in ((1, [600, 200, 1200]), (2, [200, 1200, 600]), (3, [1200, 600, 200])): + db['component_snapshots'].update_one({'_id': f's{number:02d}'}, {'$set': {'bbx': bbx}}) + labels = {r['label']: r['count'] for r in _stats(world, dataset='d', q='piece 1')['bbxX']} + assert labels == {'1000': 1} + mixed = {r['label']: r['count'] for r in _stats(world)['bbxX']} + assert mixed == {'250': 1, '1000': 7} # piece 5 is 300^3, the rest 1200 and more + + +def test_a_piece_without_a_size_is_in_no_bbx_bucket(world): + db = world['db'] + db['component_snapshots'].update_one({'_id': 's01'}, {'$unset': {'bbx': ''}}) + db['component_snapshots'].update_one({'_id': 's02'}, {'$set': {'bbx': [1, 2]}}) + db['component_snapshots'].update_one({'_id': 's03'}, {'$set': {'bbx': [1, 'x', 3]}}) + stats = _stats(world) + labels = {r['label']: r['count'] for r in stats['bbxX']} + assert labels == {'250': 1, '1000': 4} # 5 (300^3) and 4, 6, 7, 8; not 1, 2, 3 + assert '>=10000' not in labels and stats['total'] == 8 + + def test_a_piece_without_three_numbers_has_no_size_and_is_not_in_a_range(world): db = world['db'] db['component_snapshots'].update_one({'_id': 's01'}, {'$unset': {'bbx': ''}}) diff --git a/tests/catalog/test_snapshot_geometry.py b/tests/catalog/test_snapshot_geometry.py index 7f532e58..815632d2 100644 --- a/tests/catalog/test_snapshot_geometry.py +++ b/tests/catalog/test_snapshot_geometry.py @@ -5,7 +5,8 @@ import pytest from pydantic import ValidationError -from apps.catalog.models import SnapshotGeometry, SnapshotReinforcement +from apps.catalog.documents import Geometry +from apps.catalog.models import SnapshotGeometry def _minimal_extrusion_geometry() -> dict: @@ -17,116 +18,60 @@ def _minimal_extrusion_geometry() -> dict: } -def test_reinforcement_valid_object(): - bar = SnapshotReinforcement( - spec='B500B', - diameter=12.0, - points=[[0, 0, 0], [2400, 0, 0]], - ) - assert bar.spec == 'B500B' - assert bar.diameter == 12.0 - assert len(bar.points) == 2 - - -def test_reinforcement_strips_spec(): - bar = SnapshotReinforcement( - spec=' B500B ', - diameter=8.0, - points=[[0, 0, 0], [1, 0, 0]], - ) - assert bar.spec == 'B500B' - - -def test_reinforcement_rejects_empty_spec(): - with pytest.raises(ValidationError): - SnapshotReinforcement( - spec=' ', - diameter=12.0, - points=[[0, 0, 0], [1, 0, 0]], - ) - - -def test_reinforcement_rejects_non_positive_diameter(): - with pytest.raises(ValidationError): - SnapshotReinforcement( - spec='B500B', - diameter=0, - points=[[0, 0, 0], [1, 0, 0]], - ) - - -def test_reinforcement_requires_at_least_two_points(): - with pytest.raises(ValidationError): - SnapshotReinforcement( - spec='B500B', - diameter=12.0, - points=[[0, 0, 0]], - ) +def _bar() -> dict: + return {'spec': 'B500B', 'diameter': 12.0, + 'points': [[0, 0, 0], [100, 0, 0]]} -def test_reinforcement_rejects_malformed_points(): - with pytest.raises(ValidationError): - SnapshotReinforcement( - spec='B500B', - diameter=12.0, - points=[[0, 0], [1, 0, 0]], - ) +def test_geometry_accepts_extrusion(): + geometry = SnapshotGeometry(**_minimal_extrusion_geometry()) + assert len(geometry.extrusions) == 1 -def test_geometry_accepts_mesh_with_reinforcements(): +def test_geometry_accepts_mesh_with_marker_points(): geometry = SnapshotGeometry( meshes=[{ 'vertices': [[0, 0, 0], [1, 0, 0], [0, 1, 0]], 'faces': [[0, 1, 2]], }], - reinforcements=[{ - 'spec': 'B500B', - 'diameter': 12.0, - 'points': [[0, 0, 0], [100, 0, 0]], - }], + marker_points=[[0, 0, 0]], ) - assert geometry.reinforcements is not None - assert len(geometry.reinforcements) == 1 + assert geometry.marker_points == [[0, 0, 0]] -def test_geometry_rejects_reinforcements_only(): +def test_geometry_rejects_marker_points_only(): with pytest.raises(ValidationError, match='geometry must include'): - SnapshotGeometry( - reinforcements=[{ - 'spec': 'B500B', - 'diameter': 12.0, - 'points': [[0, 0, 0], [100, 0, 0]], - }], - ) - - -def test_geometry_accepts_helper_style_reinforcement_payload(): - """Shape produced by CreateReinforcement + CreateComponent* centering.""" - geometry = SnapshotGeometry( - **_minimal_extrusion_geometry(), - reinforcements=[{ - 'spec': 'B500B', - 'diameter': 12.0, - 'points': [[0.0, 0.0, 0.0], [2400.0, 0.0, 0.0]], - }], - ) - assert geometry.reinforcements[0].spec == 'B500B' - - -def test_geometry_accepts_extrusion_with_multiple_reinforcements(): - geometry = SnapshotGeometry( - **_minimal_extrusion_geometry(), - reinforcements=[ - { - 'spec': 'B500B', - 'diameter': 12.0, - 'points': [[0, 0, 0], [100, 0, 0]], - }, - { - 'spec': 'B500A', - 'diameter': 8.0, - 'points': [[0, 0, 5], [0, 50, 5], [0, 50, -5]], - }, - ], - ) - assert len(geometry.reinforcements or []) == 2 + SnapshotGeometry(marker_points=[[0, 0, 0]]) + + +@pytest.mark.parametrize('extra', [ + {}, + _minimal_extrusion_geometry(), +]) +def test_geometry_rejects_reinforcements(extra): + """Reinforcement is evidence (decision 7.8): the error says where it + belongs, whether or not the geometry has a representation.""" + with pytest.raises(ValidationError) as caught: + SnapshotGeometry(**extra, reinforcements=[_bar()]) + message = str(caught.value) + assert 'geometry.reinforcements is not accepted' in message + assert 'reinforcement_layout' in message + assert '7.8' in message + + +def test_geometry_rejects_empty_reinforcements_key(): + """Even an empty list or null says the client still sends the old key.""" + for value in ([], None): + with pytest.raises(ValidationError, match='reinforcement_layout'): + SnapshotGeometry(**_minimal_extrusion_geometry(), + reinforcements=value) + + +def test_the_stored_geometry_block_gives_the_same_answer(): + """The block every write goes through (``documents.Geometry``) names the + evidence record too, not just "extra inputs are not permitted".""" + mesh = {'vertices': [[0, 0, 0], [1, 0, 0], [0, 1, 0]], 'faces': [[0, 1, 2]]} + assert Geometry(meshes=[mesh]).meshes + with pytest.raises(ValidationError) as caught: + Geometry(meshes=[mesh], reinforcements=[_bar()]) + assert 'reinforcement_layout' in str(caught.value) and '7.8' in str(caught.value) diff --git a/tests/grasshopper/compstub.py b/tests/grasshopper/compstub.py index 395fb467..cc4133ba 100644 --- a/tests/grasshopper/compstub.py +++ b/tests/grasshopper/compstub.py @@ -11,6 +11,7 @@ from __future__ import annotations +import re import sys import types from pathlib import Path @@ -19,6 +20,14 @@ SRC = headers.SRC_DIR +# the client the Session really sends: the CSC_CLIENT line of its source. It lags +# behind VERSION until the UserObjects are re-exported (bump-version --gh), so the +# tests that expect it must not read VERSION +GH_CLIENT = re.search( + r"^CSC_CLIENT = '(gh-userobjects/[^']+)'\r?$", + (SRC / 'DDU_CSC_Session.py').read_bytes().decode('utf-8', 'replace'), + re.M).group(1) + class Stub: """An object that is any attribute, any call and any subscript.""" diff --git a/tests/grasshopper/test_components.py b/tests/grasshopper/test_components.py index 92ceaff9..75d1a345 100644 --- a/tests/grasshopper/test_components.py +++ b/tests/grasshopper/test_components.py @@ -433,7 +433,7 @@ def test_server_addresses(session): def test_the_session_core_sends_over_one_connection(session): core = session.ns['_AuthCore']('localhost:8010', disable_cache=True) assert core.base_url == 'http://localhost:8010' - assert core.CLIENT == 'gh-userobjects/0.6.3.0' + assert core.CLIENT == compstub.GH_CLIENT core.set_access_token(_token(), 'ada') seen = [] @@ -451,7 +451,7 @@ def request(self, method, url, **kwargs): ('POST', 'http://localhost:8010/identities'), ('PUT', 'http://localhost:8010/snapshots/s/meshes/0/reduced')] for _, _, kwargs in seen: - assert kwargs['headers']['X-CSC-Client'] == 'gh-userobjects/0.6.3.0' + assert kwargs['headers']['X-CSC-Client'] == compstub.GH_CLIENT assert kwargs['headers']['Authorization'].startswith('Bearer ') assert seen[0][2]['params'] == {'expand': 'none'} assert seen[1][2]['json'] == {'a': 1} and 'files' not in seen[1][2] @@ -490,7 +490,7 @@ def test_the_cache_is_purged_once_when_the_client_changes(session, tmp_path): assert not hasattr(cache, 'designs_dir') assert not (cache_dir / 'designs').exists() stamp = cache_dir / 'client_release.txt' - assert stamp.read_text() == 'gh-userobjects/0.6.3.0' + assert stamp.read_text() == compstub.GH_CLIENT cache.set_identity('i1', {'_id': 'i1', 'type': 'beam'}) assert cache.get_identity('i1')[2] is True # same release: kept @@ -500,7 +500,7 @@ def test_the_cache_is_purged_once_when_the_client_changes(session, tmp_path): stamp.write_text('gh-userobjects/0.5.1.0') cache_class(cache_dir=str(cache_dir)) assert cache.get_identity('i1')[2] is False - assert stamp.read_text() == 'gh-userobjects/0.6.3.0' + assert stamp.read_text() == compstub.GH_CLIENT # an old cache without any stamp is not trusted either cache.set_identity('i2', {'_id': 'i2'}) stamp.unlink() @@ -541,7 +541,7 @@ def test_the_session_logs_in_to_the_server_it_is_given(session): login = [entry for entry in log if entry[1].endswith('/auth/token')][0] assert login[1] == 'http://localhost:8011/auth/token' assert login[2]['data'] == {'username': 'ada', 'password': 'secret'} - assert login[2]['headers']['X-CSC-Client'] == 'gh-userobjects/0.6.3.0' + assert login[2]['headers']['X-CSC-Client'] == compstub.GH_CLIENT assert core.is_valid() and core.get_username() == 'ada' text = '\n'.join(status) assert 'Server changed to http://localhost:8011' in text @@ -581,7 +581,7 @@ class Old: session.sticky['CSC_AuthCore'] = Old() obj = compstub.instance(session, 'Session') core = obj.get_auth_core_from_sticky() - assert core.CLIENT == 'gh-userobjects/0.6.3.0' and core.is_valid() + assert core.CLIENT == compstub.GH_CLIENT and core.is_valid() assert core.base_url == 'http://localhost:8010' assert session.sticky['CSC_AuthCore'] is core assert hasattr(core, '_http') and hasattr(core, 'authorized_put') diff --git a/tests/grasshopper/test_headers.py b/tests/grasshopper/test_headers.py index 3b5fb164..004dce0f 100644 --- a/tests/grasshopper/test_headers.py +++ b/tests/grasshopper/test_headers.py @@ -13,6 +13,7 @@ import pytest +import compstub import headers PACKAGE = headers.LIB_DIR / 'csc_gh' @@ -162,11 +163,23 @@ def test_the_updater_reads_the_components_own_version(): assert headers.check_version_line(text, path.name) == [], path.name -def test_the_session_names_the_0_6_3_client(): - text = (headers.SRC_DIR / 'DDU_CSC_Session.py').read_text(encoding='utf-8') - assert re.search(r"^CSC_CLIENT = 'gh-userobjects/0\.6\.3\.0'$", text, - re.M) - assert re.search(r'^Version: \d+(\.\d+)?[a-z]?$', text, re.M) +def _version_key(version): + """The order of scripts/ci/check_version.py: a pre-release sorts before its final.""" + core, _, pre = version.partition('-') + return tuple(int(x) for x in core.split('.')), (0, pre) if pre else (1, '') + + +def test_the_session_names_a_valid_client_not_newer_than_the_release(): + # the client moves only with a re-export of the UserObjects (bump-version --gh), + # so it may lag behind VERSION but never lead it (the rule of check_version.py) + text = (headers.SRC_DIR / 'DDU_CSC_Session.py').read_bytes().decode('utf-8', 'replace') + found = re.search( + r"^CSC_CLIENT = 'gh-userobjects/(\d+\.\d+\.\d+\.\d+(?:-[0-9A-Za-z.]+)?)'\r?$", text, re.M) + assert found, 'CSC_CLIENT is not gh-userobjects/' + assert compstub.GH_CLIENT == 'gh-userobjects/' + found.group(1) + release = (headers.SRC_DIR.parent / 'VERSION').read_text(encoding='utf-8').strip() + assert _version_key(found.group(1)) <= _version_key(release), (found.group(1), release) + assert re.search(r'^Version: \d+(\.\d+)?[a-z]?\r?$', text, re.M) def test_no_component_is_left_from_0_5(): diff --git a/tests/grasshopper/test_session_signin.py b/tests/grasshopper/test_session_signin.py index b09da4ee..9d89d3a3 100644 --- a/tests/grasshopper/test_session_signin.py +++ b/tests/grasshopper/test_session_signin.py @@ -152,7 +152,7 @@ def test_a_stored_sign_in_signs_in_and_the_new_refresh_token_replaces_the_old(ct (call,) = core._http.calls('POST', '/token') assert call[2]['data'] == {'grant_type': 'refresh_token', 'refresh_token': 'csc_dev_OLD', 'client_id': 'csc-grasshopper'} - assert call[2]['headers']['X-CSC-Client'] == 'gh-userobjects/0.6.3.0' + assert call[2]['headers']['X-CSC-Client'] == compstub.GH_CLIENT assert core.is_valid() and core.get_username() == 'ada' assert signin.load_entry(ctx.store, SERVER, fake_unprotect) == 'csc_dev_NEW' assert any('Signed in as ada (stored sign-in' in line for line in status) @@ -519,7 +519,7 @@ class Old: _credentials = ('ada', 'secret') ctx.sticky['CSC_AuthCore'] = Old() core = compstub.instance(ctx, 'Session').get_auth_core_from_sticky() - assert core.CLIENT == 'gh-userobjects/0.6.3.0' and core._refresh == 'csc_dev_KEPT' + assert core.CLIENT == compstub.GH_CLIENT and core._refresh == 'csc_dev_KEPT' assert core._credentials == ('ada', 'secret') and core.is_valid() diff --git a/uberspaceconfig/bin/csc_dump.sh b/uberspaceconfig/bin/csc_dump.sh new file mode 100644 index 00000000..6517fa3a --- /dev/null +++ b/uberspaceconfig/bin/csc_dump.sh @@ -0,0 +1,66 @@ +#!/bin/bash +# Dump the production database to stdout, for a backup pulled over ssh. +# +# csc_dump.sh archive mongodump archive (gzip), restore with +# mongorestore --gzip --archive= +# csc_dump.sh json tar.gz of one csc..json per collection +# (a JSON array in relaxed Extended JSON, like a +# Compass export), the layout of mongodb_collections_local +# +# Install: copy to ~/bin/csc_dump.sh, chmod 700. Reads MONGODB_URI from +# ~/etc/services.d/fastapi.ini; writes only to stdout and a temp folder that is +# removed at the end. Progress and errors go to stderr. + +set -euo pipefail + +INI="$HOME/etc/services.d/fastapi.ini" +PYTHON="$HOME/csc/current/venv/bin/python" + +URI=$(grep -o 'MONGODB_URI=[^,]*' "$INI" | head -1 | cut -d= -f2- | tr -d '"\r') +if [ -z "$URI" ]; then + echo "csc_dump: MONGODB_URI not found in $INI" >&2 + exit 1 +fi + +case "${1:-}" in + archive) + mongodump --uri="$URI" --gzip --archive --quiet + ;; + json) + TMP=$(mktemp -d) + trap 'rm -rf "$TMP"' EXIT + CSC_DUMP_URI="$URI" CSC_DUMP_DIR="$TMP" "$PYTHON" - <<'EOF' +import os +import sys + +from bson import json_util +from bson.json_util import JSONOptions, JSONMode +from pymongo import MongoClient + +client = MongoClient(os.environ['CSC_DUMP_URI']) +db = client.get_default_database(default='csc') +options = JSONOptions(json_mode=JSONMode.RELAXED) +for name in sorted(db.list_collection_names()): + if name.startswith('system.'): + continue + # one document at a time: the server kills a process that holds a large + # collection in memory + path = os.path.join(os.environ['CSC_DUMP_DIR'], f'csc.{name}.json') + count = 0 + with open(path, 'w', encoding='utf-8') as handle: + handle.write('[') + for doc in db[name].find(batch_size=50): + if count: + handle.write(',') + handle.write(json_util.dumps(doc, json_options=options)) + count += 1 + handle.write(']') + print(f'{name}: {count}', file=sys.stderr) +EOF + tar czf - -C "$TMP" . + ;; + *) + echo "usage: csc_dump.sh archive|json" >&2 + exit 2 + ;; +esac