diff --git a/CITATION.cff b/CITATION.cff index c85aeb8..b6dc46a 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -1,8 +1,8 @@ cff-version: 1.2.0 message: "If you use this software, please cite it as below." title: "Instance Space Analysis: A toolkit for the assessment of algorithmic power" -version: 0.9.1 -date-released: 2026-08-20 +version: 0.9.2 +date-released: 2026-10-17 doi: 10.5281/zenodo.4484107 authors: - family-names: "Muñoz" diff --git a/CLAUDE.md b/CLAUDE.md index e1ece32..f95327e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -5,26 +5,21 @@ - **Owner:** sole developer (Mario Andrés Muñoz) — no human reviewers; GitHub Copilot is the automated PR review gate. `master` has branch protection requiring changes go through a PR. - **Primary API:** the `InstanceSpace` class (`InstanceSpace.m`) — `build()`/`explore()`/`save()`/`InstanceSpace.load()`/`getResults()`/`plot()`. `buildIS.m`/`exploreIS.m` are thin backward-compatibility wrappers around it (kept for callers like the MATILDA web platform); don't add new logic to them directly. - **Completed refactor plan:** `docs/tech/isa_refactor_plan_v1.7.pdf` — historical source of truth for the v0.9.0 refactor (all ten phases). Now fully shipped; treat it as an archival reference, not a live backlog. -- **Current backlog:** GitHub Issues are the live source of truth. Every `v0.9.1`-milestone issue is now resolved and merged to `master` (see "Current state" below) — what's left is scoped for a **v0.9.2** milestone that doesn't exist as a GitHub milestone *object* yet (no milestone-management tool has been available in this environment across several sessions now; #30/#50/#52/#53 are the filed items, each needs its milestone field set by hand once v0.9.2 exists). +- **Current backlog:** GitHub Issues are the live source of truth. As of 17 October 2026, #63 (PILOT numerical restart sensitivity to floating-point jitter) is the only open MATLAB issue. Do not treat old release plans as pending work. - **Reference papers:** `docs/papers/Smith-Miles2023.pdf` (general ISA methodology), `docs/papers/Simpson2025.pdf` (ISA3, the 3D extension — cited only in `PILOT.m`, `PILOTviewpoint.m`, `TRACE.m`, and the 3D-output files in `output/`, not repo-wide). PILOT's origin (Muñoz et al. 2018), legacy TRACE's origin (Muñoz & Smith-Miles 2017), and FILTER's origin (Alipour et al. 2023) are cited in each file's own docstring `Reference:` block but aren't archived as PDFs here. ## Test harness Run from the repo root; `buildIS.m`/`exploreIS.m` print `EOF:SUCCESS` on success (errors abort before printing), and `test_integration.m` prints `EOF:SUCCESS`/`EOF:ERROR` at the end: example test_integration -`example.m` already runs the full `buildIS`+`exploreIS` pipeline internally on the bundled reference dataset — it's the one to start with. `test_integration.m` is a thin `matlab.unittest`-based runner (#39) over `tests/*.m` (`PipelineOptionsTest` — every classifier, tuning strategy, 2D/3D, PLS, viewpoint groups, parameterised; `ClassApiTest` — staged `build()`/`explore()`/save-load round-trips; `MigrationTest` — the full `ISAmigrateModel` legacy-migration table; `RegressionTest` — targeted fixed-bug regressions), with a `CodeCoveragePlugin` producing `coverage.xml`. New features and bug fixes get a test *method* under `tests/`, not a new block in `test_integration.m` itself. Good as a reference for how an option is meant to be used, not the place to start reading. Runs automatically on every push/PR via CI (#34, `.github/workflows/tests.yml`); pinned to `release: R2025a`, not `latest` (deliberate — see the `matlab-ci-testing` skill below for why). +`example.m` already runs the full `buildIS`+`exploreIS` pipeline internally on the bundled reference dataset — it's the one to start with. `test_integration.m` is a thin `matlab.unittest`-based runner (#39) over `tests/*.m` (`PipelineOptionsTest` — every classifier, tuning strategy, 2D/3D, PLS, viewpoint groups, parameterised; `ClassApiTest` — staged `build()`/`explore()`/save-load round-trips; `MigrationTest` — the full `ISAmigrateModel` legacy-migration table; `RegressionTest` — targeted fixed-bug regressions), with a `CodeCoveragePlugin` producing `coverage.xml`. New features and bug fixes get a test *method* under `tests/`, not a new block in `test_integration.m` itself. Good as a reference for how an option is meant to be used, not the place to start reading. Runs automatically on every push/PR via CI (#34, `.github/workflows/tests.yml`); pinned to `release: R2025a`, not `latest` (deliberate — this is the documented minimum supported MATLAB release). -## Current state (v0.9.0 and v0.9.1 both shipped to `master`; v0.9.2 not started) -- **v0.9.0** (July 2026): all ten refactor-plan phases — rewritten PRELIM/SIFTED/PILOT/PYTHIA/TRACE pipeline, the `InstanceSpace` class API, `ISAmigrateModel`/`ISAvalidateOpts`, 3D projection/viewpoint support, and a licence change from GPL v3 to PolyForm Noncommercial 1.0.0. -- **v0.9.1** (August 2026): engineering-quality, architecture, and infrastructure follow-ups — no algorithmic changes, no new user-facing options, per that milestone's own scope rule. Landed on `master` via three merged PRs; **`RELEASE_NOTES.md`'s v0.9.1 section is the full changelog** (organised into New functionality/Better engineering/Bug fixes) — this file only tracks what a future session specifically needs to know, not a duplicate list: - - **#51**: the bulk of the engineering work (INIT.m/PRELIM train-eval dual mode, per-stage contract validation and `onStage` inspection callbacks, CI, `matlab.unittest` migration, several confirmed correctness fixes). Went through **five rounds** of Copilot review; each round found genuine bugs — including, more than once, in a fix landed earlier in the *same* PR (e.g. an added safety warning that didn't actually fire; a guard that proved two values matched each other but not that either was still correct). If touching any of that code again, don't assume it's settled just because it's merged — check the PR's review-comment history first. - - **#54**: added the `matlab-ci-testing` Claude Code skill (`.claude/skills/matlab-ci-testing/`) — CI setup, `matlab.unittest` architecture, MATLAB-specific test-writing patterns, distilled directly from the #51 review cycle. Also added Codecov and GitHub-downloads badges to `README.md`. - - **#55**: `README.md` had accumulated internal development-process narration (issue numbers, "confirmed via CI", references to a prior implementation, an internal spec-section reference) — removed; see "Documentation conventions" below for the standing rule this established. Same PR also fixed a repo-wide docstring/doc-formatting audit: README's ~208 inline-code spans were triple-backtick instead of single; `core/PYTHIA.m` was the one `core/` pipeline file missing the `Inputs`/`Outputs` docstring section its siblings all have; a couple of files had inconsistent `%---` divider-comment widths. - - **Not yet done, not code**: the Codecov badge shows no data until the repo owner links the repo at codecov.io and adds a `CODECOV_TOKEN` GitHub Actions secret — the CI-side upload step is already in place (`fail_ci_if_error: false`, so its absence can't break CI). -- **v0.9.2 backlog** (filed, none started, no milestone object yet — see "Current backlog" above): #30 (PILOT projection-orientation canonicalisation), #50 (CLOISTER's convex hull is 2D-only even for a 3D projection), #52 (`traceOneRegion` doesn't trace a hole's boundary cycle), #53 (a MATLAB-toolbox-style reference documentation site — `doc InstanceSpace` Help-browser integration via `info.xml`/`helptoc.xml` plus GitHub Pages, ~29 pages; full scope is in the issue body). Unlike v0.9.1, several of these (#30/#50/#52) are explicitly algorithmic and #53 is a new deliverable — don't assume v0.9.1's "no algorithmic changes" rule carries over to v0.9.2 by default; check with the user. - - **Closed as not planned**: a cross-repo reference-fixture export tool for the Python port (`pyInstanceSpace`, #35) — per a standing preference, anything dealing with the Python port is handled from that repo, not this one. - - **Closed as resolved, no code change**: #45 (verify `PRELIM.m`'s `boxcox()` call) — confirmed it resolves to the Financial Toolbox's `boxcox.m`, the same dependency CI already installs for this call site; combined with the already-verified positive-shift guard, no gap was found. -- **Repository layout**: `core/` (INIT, PRELIM, SIFTED, PILOT, PILOTviewpoint, CLOISTER, PYTHIA, TRACE, TRACE_legacy, FILTER), `output/` (scriptcsv, scriptpng, scriptweb, scriptfcn, ISArecallView), `utils/` (ISAdefaults, ISAvalidateOpts, ISAgetClassifierFcn, ISAmigrateModel, ISAsubsetData), `deprecated/` (PYTHIA2, PYTHIAtest, SIFTED2 — warn-and-forward shims kept for backward compatibility only; don't build new code on these), `.claude/skills/` (`instance-space-analysis` — ISA methodology plus this repo's operational reference; `matlab-ci-testing` — MATLAB CI/`matlab.unittest` setup and debugging, not ISA-specific, reusable in other MATLAB repos). +## Current state (v0.9.2 release preparation) +- v0.9.0 and v0.9.1 are published. The v0.9.2 changes are merged to `master` through #60, #62, #64, #65 and #66. Release metadata and documentation are being prepared separately from publishing a tag/release; use `docs/release-0.9.2.md` for the release checklist. +- Implemented and closed: #50 (3D CLOISTER), #52 (hole boundary cycles), #53 (reference site), #57 (JUnit/Codecov reporting), #58 (unobserved test outcomes), #59 (Oracle probability), #61 (SVM calibration seeding). These are not pending merges. #30 is also closed on GitHub, but its proposed optional canonical rotation is not implemented in current PILOT; do not infer implementation from the issue state. `RELEASE_NOTES.md` is the complete user-facing change record. +- Current contracts include independent INIT snapshots, preprocessing recomputation after portfolio pruning, SIFTED cache isolation and replay inputs, fitted PLS centering, training-only selection policy, honest CV/test summaries, stage provenance, persisted TRACE distance tolerance and serial Octave support. Review `doc/src/`, `docs/octave-compatibility.md` and `review/2026-09-30/FINAL_REPORT.md` before changing them. +- Repository layout: `core/` stages, `output/` exporters, `utils/` utilities and `+isacompat` runtime adapters, `deprecated/` warn-and-forward shims, `tests/` MATLAB tests and `tests/portable/` shared contracts. `.claude/skills/instance-space-analysis/` contains methodology and repository usage guidance. The older referenced `matlab-ci-testing` skill is not present in this checkout. +- CI targets MATLAB R2025a; local release validation uses R2026a Update 5. The full serial Octave workflow is validated with 11.3.0, Statistics 2.0.0 and Datatypes 1.5.0. Core-only checks on older Octave do not imply full workflow support. ## Workflow - Work on feature branches; merge to `master` after Copilot review. `master`'s branch protection requires a PR — don't push directly to it. @@ -35,8 +30,8 @@ Run from the repo root; `buildIS.m`/`exploreIS.m` print `EOF:SUCCESS` on success ## Key conventions - Language: MATLAB; style follows standard MATLAB conventions. - No magic numbers — use named `opts` fields with documented defaults (`ISAdefaults.m`), not hardcoded literals. -- Standalone pipeline functions (`PRELIM`, `SIFTED`, `PILOT`, `CLOISTER`, `PYTHIA`, `TRACE`, `FILTER`) must remain callable independently of the `InstanceSpace` class — deliberate, not an oversight, to preserve MATILDA web-platform compatibility. Don't fold them into class methods. -- **Train vs. eval mode is dispatched via `nargin`, not a fixed positional argument**: an optional trailing `trainedModel`/`trainedTrace`/`trainedPrelim`-style struct as the *last* argument selects eval mode over train mode (e.g. `PYTHIA(Z,Y,Ybin,Ybest,algolabels,opts)` trains; `PYTHIA(...,opts,trainedModel)` evaluates; same shape for `TRACE` and, since #38, `PRELIM`/`INIT`). Confirmed by direct audit (see #38) that `SIFTED`/`PILOT`/`CLOISTER`/`FILTER` never recompute anything at explore time — they just reuse a stored trained artifact directly (e.g. `PILOT`'s trained projection is reused via a plain `X*A'` multiply) — so they don't need this structure and shouldn't get it added speculatively. +- Standalone pipeline functions (`INIT`, `PRELIM`, `SIFTED`, `PILOT`, `CLOISTER`, `PYTHIA`, `TRACE`, `FILTER`) must remain callable independently of the `InstanceSpace` class — deliberate, not an oversight, to preserve MATILDA web-platform compatibility. Don't fold them into class methods. +- **Train vs. eval mode is dispatched via `nargin`, not a fixed positional argument**: an optional trailing `trainedModel`/`trainedTrace`/`trainedPrelim`-style struct as the *last* argument selects eval mode over train mode (e.g. `PYTHIA(Z,Y,Ybin,Ybest,algolabels,opts)` trains; `PYTHIA(...,opts,trainedModel)` evaluates; same shape for `TRACE` and, since #38, `PRELIM`/`INIT`). Confirmed by direct audit (see #38) that `SIFTED`/`PILOT`/`CLOISTER`/`FILTER` never recompute anything at explore time — they just reuse a stored trained artifact directly (standard PILOT reuses `X*A'`; PLS applies its stored feature mean before that multiply) — so they don't need this structure and shouldn't get it added speculatively. - **Data ingestion (#38)**: `INIT.m` (`core/`) does the CSV-read + `opts.selvars.feats`/`.algos` filtering that used to be duplicated between `InstanceSpace.runPrelim` and `InstanceSpace.evaluateTestSet` — `INIT(rootdir, opts)` trains (reads `metadata.csv`), `INIT(rootdir, opts, trainedModel)` evaluates (reads `metadata_test.csv`, validates the feature set/order against `trainedModel`, reconciles algorithm columns). `PRELIM.m` itself now also has the `nargin` dual mode: its Ybin/Ybest/P/beta computation (including tie-breaking) is unconditional shared code, only the bound-clipping/Box-Cox+Z-score normalisation branches on train (fit) vs eval (apply trainedPrelim's already-fit parameters). - `opts.pilot.topoWeight` — still reserved/disabled, no effect in the current version; don't wire it up without confirming it's actually in scope for the task at hand. @@ -45,21 +40,22 @@ Run from the repo root; `buildIS.m`/`exploreIS.m` print `EOF:SUCCESS` on success - **Inline code in Markdown docs uses single backticks** (`` `like this` ``), not triple backticks. `README.md` had drifted to ~208 triple-backtick inline spans before #55's cleanup; `CONTRIBUTING.md`/`SECURITY.md`/`CLAUDE.md`/`RELEASE_NOTES.md` were already consistent. Triple backticks are still correct for actual fenced code blocks. - **Every `.m` file's docstring** starts with the same copyright/SPDX-license header block (copy an existing file's, e.g. `core/PILOT.m`) followed by a `Reference:` section citing only the paper(s) that specific file implements. `core/` pipeline-stage files (`PRELIM`/`SIFTED`/`PILOT`/`PILOTviewpoint`/`CLOISTER`/`PYTHIA`/`TRACE`/`TRACE_legacy`/`FILTER`/`INIT`) and `InstanceSpace.m` additionally use a structured `Inputs`/`Outputs` section (see any of them for the exact format); `utils/`/`output/`/`deprecated/`/`buildIS.m`/`exploreIS.m` intentionally use lighter narrative-prose docstrings instead — that split (algorithm implementations vs. supporting utilities) is deliberate, not an inconsistency to "fix." -## Known deferred items (tracked, not stale — check the linked issue before touching) -- **#30** (PILOT projection-orientation canonicalisation) is the remaining open algorithmic item. -- **Resolved on `claude/docs-testing-improvements-amt87s` (v0.9.2 work, pending merge):** #50 (CLOISTER now builds a 3D hull with `ZedgeFaces`/`ZecorrFaces`; `plot('boundary')`/`scriptpng` draw it; a 3D model built before the fix still raises `ISA:InstanceSpace:boundaryNot3D`), #52 (`traceOneRegion` traces every boundary cycle, NaN-separated; the `boundaryHoleOmitted` warning is gone), #58 (PYTHIA eval scores only rows with observed `Y`; unobserved algorithms get `NaN` metrics), #59 (Oracle `Probability_of_good` = `mean(any(Ybin,2))`), #57 (JUnit report + Codecov Test Analytics). +## Known limitations +- **#63 remains open:** numerical PILOT restart selection can amplify BLAS/thread-level floating-point jitter. Fixed-seed fixture repeats in a pinned environment do not establish cross-environment reproducibility or resolve this issue. +- `opts.pilot.topoWeight` remains disabled. Do not silently activate it as documentation cleanup. +- Octave supports only the bounded serial workflow documented in `docs/octave-compatibility.md`; its model archives are runtime-specific. The TRACE boundary policy agrees with Python, but arbitrary numerical/geometry identity is not promised. ## Documentation site (#53) - Sources: `doc/src/*.md`; generator: `doc/generate.py` (python-markdown, pinned in `doc/requirements.txt`); output: `doc/html/` (committed — the MATLAB Help browser reads it via `info.xml` from a plain clone). - The `TOC` list in `doc/generate.py` is the single source for the sidebar, breadcrumbs, `helptoc.xml`, the generated *All Functions* page and search. A page not in `TOC` fails the build. - Page conventions (Syntax/Description/Examples/Input Arguments/Output Arguments/Version History/See Also, ``### `name` — Short description`` argument headings, `` for bare `opts.x` links) are documented in the generator's docstring. -- `.github/workflows/docs-pages.yml`: `generate.py --check` + `.github/scripts/check_doc_links.py` on PRs; deploys `doc/html` to GitHub Pages from `master`. Pages must be enabled once in the repo settings (Source: GitHub Actions) — not something this environment can do. +- `.github/workflows/docs-pages.yml`: `generate.py --check` + `.github/scripts/check_doc_links.py` on PRs; deploys `doc/html` to GitHub Pages from `master`. Deployment uses the repository's GitHub Pages settings (Source: GitHub Actions). - When a function's behaviour or signature changes, update its `doc/src` page (and a Version History entry), then regenerate. **LIBSVM MEX-files removed (#29).** `svmpredict.mexw64`/`svmtrain.mexw64` were precompiled binaries with no corresponding source in the tree — removed entirely rather than kept-with-provenance, since LIBSVM is already fully deprecated for new runs. `PYTHIA`'s eval mode now raises a clear `ISA:PYTHIA:noLibsvm` error (naming the algorithm and pointing to the official LIBSVM project) if it ever needs to dispatch to a legacy LIBSVM-format classifier struct and `svmpredict` isn't on the path, instead of MATLAB's generic undefined-function error. Don't re-add these binaries; if LIBSVM support is ever needed again, get it from upstream rather than re-committing a precompiled MEX-file with no visible build source. ## Cross-repo note -The user has asked (in this session) about a repository called `claude-skills` (owner `andremun`) twice — once too ambiguously to act on, once explicitly as `andremun/claude-skills`, at which point cloning it failed: this session's GitHub access (both the MCP tools and git-level credentials) is hard-scoped to `andremun/InstanceSpace` only, confirmed by a 403 on an unauthenticated API probe and `git clone` refusing to prompt for credentials. If a future session has broader repo access and this comes up again, that's the blocker that was hit — not a transient error worth silently retrying. +`andremun/pyInstanceSpace` is the Python implementation. Keep Python-specific changes and fixture tooling there. Python issue #367 tracks improving its documentation website toward this reference site's navigation and presentation. Shared boundary policy does not imply full numerical parity. Check current connector permissions rather than relying on historical access failures. ## `.gitignore` notes - `docs/**/*.pdf` — tracked (negation rule), recursive under `docs/`. diff --git a/Contents.m b/Contents.m index dbbecd8..3931aa0 100644 --- a/Contents.m +++ b/Contents.m @@ -1,7 +1,8 @@ % Instance Space Analysis Toolkit -% Version 0.9.1 (R2025a) 20-Aug-2026 +% Version 0.9.2 (R2025a) 17-Oct-2026 % % Standalone pipeline functions +% INIT - Read metadata and validate training/evaluation inputs % PRELIM - Data preprocessing (outlier bounding, normalisation) % SIFTED - Feature selection (correlation + GA clustering) % PILOT - 2D/3D projection (PBLDR algorithm) @@ -21,6 +22,7 @@ % exploreIS - Thin wrapper around InstanceSpace.explore() % % Utilities +% ISAfootprintContains - Closed-boundary footprint membership with distance tolerance % ISAdefaults - Fill in missing opts fields with default values % ISAvalidateOpts - Validate user-supplied opts fields % ISAsubsetData - Subset rows (and optionally feature columns) of a data struct diff --git a/README.md b/README.md index 66f9590..1986c6b 100644 --- a/README.md +++ b/README.md @@ -43,6 +43,8 @@ Or if you specifically use [MATILDA](http://matilda.unimelb.edu.au/matilda/), pl The reference documentation is at ****: a page for every function and for the `InstanceSpace` class, with examples, a getting-started guide, a stage-by-stage walkthrough, the metadata file format, and the full options reference. The same pages open inside MATLAB: with the repository root on the path, open the Help browser and choose **Supplemental Software → Instance Space Analysis Toolbox**. +For v0.9.2 changes, upgrade guidance and remaining limitations, see [What's New](https://andremun.github.io/InstanceSpace/WhatsNew.html) and the [release notes](RELEASE_NOTES.md). Rebuild existing experiments from their original metadata to obtain the corrected fitted results. + ## Installation Instructions The main requirement for the software to run is to have MATLAB R2025a or later, with the [Global Optimization](https://au.mathworks.com/help/gads/index.html), [Parallel Computing](https://www.mathworks.com/products/parallel-computing.html), [Optimization](https://au.mathworks.com/products/optimization.html), [Statistics and Machine Learning](https://au.mathworks.com/help/stats/index.html), and [Financial](https://au.mathworks.com/products/finance.html) toolboxes installed. The LIBSVM MEX-files (`svmpredict`/`svmtrain`), used for legacy models, are **not bundled with this repository** (no build source is available for them). They're only relevant for evaluating a pre-v0.9.0 model whose classifiers `ISAmigrateModel` couldn't retrain (missing original training data); `ISAmigrateModel` prefers retraining from scratch with the current registry whenever the training data is available, which needs no LIBSVM dependency. If you do hit that path, obtain LIBSVM yourself from [the official project](https://www.csie.ntu.edu.tw/~cjlin/libsvm/) and add its MEX-files to the MATLAB path — `PYTHIA`'s eval mode raises a clear `ISA:PYTHIA:noLibsvm` error naming the missing dependency if you don't. diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index 3674cc4..f7cba48 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -1,6 +1,6 @@ -# Instance Space Analysis Toolkit — v0.9.2 (in development) +# Instance Space Analysis Toolkit — v0.9.2 -Bug fixes from the v0.9.2 backlog, serial GNU Octave support, a reference documentation site, and test infrastructure. It supports **MATLAB R2025a or later** and the bounded Octave workflow described below. +Corrected training, exploration and reporting contracts, serial GNU Octave support, an independent INIT stage, a reference documentation site, and test infrastructure. It supports **MATLAB R2025a or later** and the bounded Octave workflow described below. --- @@ -8,6 +8,8 @@ Bug fixes from the v0.9.2 backlog, serial GNU Octave support, a reference docume **Independent INIT stage.** `InstanceSpace` now exposes `init` before `prelim`, including stage callbacks, completion tracking, option validation and saved input data. PRELIM reruns reuse `model.init.data`; explicitly rerunning INIT reloads metadata and invalidates downstream results. Existing staged calls beginning with PRELIM initialize automatically when the snapshot is absent, and legacy models remain usable for exploration. Full builds retain their numerical behaviour. +**Explicit footprint boundary policy.** TRACE uses closed boundaries in 2D and 3D. `opts.trace.boundaryTolerance` is a finite nonnegative Euclidean distance in projected coordinates; its default is **0**, with no implicit numerical margin. The fitted value is saved in the model and reused during exploration and legacy contradiction counting. Older models without a saved tolerance use zero. MATLAB and Python share this policy; matching policies do not guarantee identical floating-point geometry. + **Serial GNU Octave workflow.** Octave supports 2D/3D build → save/load → explore with normalization, feature selection, analytic/numerical/SIMPLS projection, KNN classifiers with none/Sobol tuning, TRACE3 footprints, CSV and PNG. The full workflow requires Octave **11.1 or later**, Datatypes **1.5.0** and Statistics **2.0.0**; validation uses Octave **11.3.0**. Bayesian tuning, parallel execution, other classifiers, legacy TRACE, FIG files and web palette export remain unsupported. 3D PNG requires Qt and a display or virtual display; `outputs.fig` defaults to false in Octave. Build and explore reject unsupported output options before processing or replacing outputs. Octave saves versioned MAT v7 archives that preserve trained KNN models without retraining; these classifier archives are runtime-specific, while MATLAB retains native MAT v7.3 persistence. Schema 2 protects ordinary structs from collisions with internal type markers and reads existing schema-1 archives; ambiguous marker-shaped user structs in old archives cannot be distinguished retrospectively. CSV is the supported data interchange format. Equal seeds do not promise identical optimizer, Sobol or geometry results across runtimes. See the [Octave compatibility guide](docs/octave-compatibility.md) for setup and the complete support boundary. @@ -29,11 +31,31 @@ Octave saves versioned MAT v7 archives that preserve trained KNN models without ## Bug fixes - **PYTHIA evaluation scored algorithms against data that was never observed.** In `PYTHIA`'s evaluation mode, a trained algorithm absent from `metadata_test.csv` has an all-`NaN` performance column, which `PRELIM` turns into all-false labels; accuracy, precision and recall were then computed against those labels. Only instances with observed performance are now scored, and an algorithm with none reports `NaN`. Accuracy is divided by the number of scored instances, and the summary's per-algorithm `Probability_of_good` is also computed over observed instances only (blank when there are none) ([#58](https://github.com/andremun/InstanceSpace/issues/58)). -- **Oracle probability of good performance was always 1.** The Oracle row of `PYTHIA`'s summary now reports `mean(any(Ybin,2))`, the fraction of instances on which any algorithm is good. This is below 1 when `opts.perf.AbsPerf = true` and no algorithm meets the threshold on some instances ([#59](https://github.com/andremun/InstanceSpace/issues/59)). +- **Oracle probability of good performance was always 1.** The Oracle row of `PYTHIA`'s summary now reports the fraction of instances with any observed performance on which at least one observed algorithm is good (equivalent to `mean(any(Ybin,2))` when all outcomes are observed). Rows with no observations are excluded; no observations gives `NaN`. This is below 1 when `opts.perf.AbsPerf = true` and no algorithm meets the threshold on some instances ([#59](https://github.com/andremun/InstanceSpace/issues/59)). - **`TRACE` failed when called without PYTHIA predictions.** `TRACE`'s documentation allows `Yhat = []`, falling back to the true labels, but the footprint loop indexed `Yhat(:,i)` inside a `parfor`, which slices it for every iteration and so failed with an out-of-bounds index. The pipeline itself never passed `[]`, so only direct callers were affected. Predictions are now split per algorithm before the loop. - **`ISArecallView` on a figure with no axes** failed inside `ismember` instead of raising its own `ISA:ISArecallView:noAxes` error. - **Footprint holes were dropped from the traced boundary.** `traceOneRegion` followed only the boundary cycle that contained its start vertex, so the boundary of a hole inside a footprint region was omitted (with the `ISA:scriptfcn:boundaryHoleOmitted` warning). Every cycle is now traced, and cycles are separated by a `NaN` row as regions already were; the warning is removed ([#52](https://github.com/andremun/InstanceSpace/issues/52)). -- **[Behaviour-changing] `PYTHIA`'s SVM posterior probabilities were not reproducible.** In `fitOneClassifier`'s `'svm'` case, `fitSVMPosterior`'s own sigmoid-calibration draws depended on however many random numbers `fitcsvm`'s SMO solver had just consumed, so `Pr0hat`/`Pr0sub` for `opts.classifier = 'svm'` could differ between two runs with the same `opts.seed`. `fitOneClassifier` now reseeds to the same per-algorithm/per-fold seed immediately before `fitSVMPosterior`, mirroring the #41 fix for PILOT/SIFTED/PILOTviewpoint. Existing saved SVM models are unaffected; rerunning `PYTHIA` training with `opts.classifier = 'svm'` will change `Pr0hat`/`Pr0sub` (not `Yhat`/`Ysub`, which threshold at 0.5 and were already reproducible) ([#61](https://github.com/andremun/InstanceSpace/issues/61)). +- **[Behaviour-changing] `PYTHIA`'s SVM posterior probabilities were not reproducible.** In `fitOneClassifier`'s `'svm'` case, `fitSVMPosterior`'s own sigmoid-calibration draws depended on however many random numbers `fitcsvm`'s SMO solver had just consumed, so `Pr0hat`/`Pr0sub` for `opts.classifier = 'svm'` could differ between two runs with the same `opts.seed`. `fitOneClassifier` now reseeds to the same per-algorithm/per-fold seed immediately before `fitSVMPosterior`, mirroring the #41 fix for PILOT/SIFTED/PILOTviewpoint. Existing saved SVM models are unaffected; rerunning `PYTHIA` training with `opts.classifier = 'svm'` may change `Pr0hat`/`Pr0sub` and can also affect predicted labels, tuning decisions and downstream footprints through the calibrated classifier ([#61](https://github.com/andremun/InstanceSpace/issues/61)). + +## Further correctness fixes + +- **Training data and portfolio consistency.** After algorithms that are never good are pruned, PRELIM recomputes labels, winners, transforms and portfolio indices from the retained raw outcomes. Constant columns use a unit scale. Training inputs remaining incomplete after ingestion filtering raise errors identifying the instances; test outcomes can remain unobserved. Raw performance must be nonnegative, with zero allowed; the original relative ratio formulas and documented zero substitution are retained. +- **Repeatable feature selection and projection.** SIFTED resets its fitness caches between runs, including workers, and retains pre-selection inputs for reruns. PLS persists its fitted feature mean and applies it during exploration and CLOISTER projection. Analytic PILOT solves least squares directly, and standalone numerical fallbacks receive complete defaults. Seed bounds and per-stage seeding are validated. +- **Training/test separation and honest summaries.** Recommendations use saved training fallback choices and precision weights, regardless of test outcomes. Selector summaries use out-of-fold selections for training and explicitly labelled test metrics for exploration. Selector recall counts disjoint successes and missed opportunities; cost-sensitive weights use per-instance regret. Invalid classifier CV searches fail explicitly. Score metadata distinguishes posterior probabilities from raw scores and mixed fold outputs. These CV summaries remain conditional on fitted preprocessing, feature selection, projection and tuning; they are not outer cross-validation of the complete pipeline. +- **Staged builds and persistence.** Options consumed by each stage are saved with provenance. Partial models can be saved, loaded and resumed; incompatible option edits cannot silently retain stale fitted artifacts. Training feature exclusions are replayed during exploration. Builds respect serial settings and close only parallel pools they created. +- **Footprint evaluation and exports.** Exploration reuses trained space geometry and accepts single, repeated or lower-dimensional test batches. TRACE records whether the purity target was achieved and why its search stopped; a retained candidate below the target is not labelled accepted. CSV exports include 3D vertices and triangle faces, explicit empty geometry and `geometry_manifest.json`. Export replaces obsolete toolkit-owned geometry and plot files; user files must use other names. Viewpoint generation and recall use the same camera convention and equal data-axis scales. + +## Upgrading from v0.9.1 + +Keep a backup of existing models and result directories. Rebuild experiments from the original metadata to obtain the corrected preprocessing, feature selection, PLS coordinates, classifiers, selector summaries and footprints; loading an old model does not retroactively apply training fixes. Rebuild CLOISTER to obtain a valid stored 3D boundary. Models predating the saved pre-SIFTED snapshot must rebuild preprocessing before rerunning feature selection. `ISAmigrateModel` remains available for pre-v0.9.0 layouts, but migration is not a substitute for rebuilding corrected experiments. + +Options now reject ambiguous subsetting modes, invalid index files, unsupported seed values and negative raw performance earlier. Consumers of exported geometry should read the manifest, preserve NaN-separated 2D rings and use one-based 3D face indices. Saved Octave classifiers are runtime-specific; use CSV for data interchange. + +## Known limitations + +Numerical PILOT restart selection can amplify small floating-point differences across BLAS implementations or thread settings ([#63](https://github.com/andremun/InstanceSpace/issues/63)). A fixed seed is not a guarantee of identical numerical projections across environments. Keep runtime, libraries and thread settings fixed for reproducibility-sensitive work. This issue remains open. PILOT does not implement the optional canonical rotation proposed in closed issue #30; projections can have different orientations. The proposed rotation feature is excluded from this release. + +`opts.pilot.topoWeight` remains reserved and disabled. Octave support is limited to the workflow above. Geometric degeneracies and optimizer differences can produce different MATLAB/Python/Octave results even when the same boundary policy is used. --- diff --git a/doc/generate.py b/doc/generate.py index 6097ff7..6121406 100644 --- a/doc/generate.py +++ b/doc/generate.py @@ -83,6 +83,7 @@ ("Utilities", "FunctionList", [ ("ISAdefaults", "ISAdefaults"), ("ISAvalidateOpts", "ISAvalidateOpts"), + ("ISAfootprintContains", "ISAfootprintContains"), ("ISAgetClassifierFcn", "ISAgetClassifierFcn"), ("ISAmigrateModel", "ISAmigrateModel"), ("ISAsubsetData", "ISAsubsetData"), diff --git a/doc/html/404.html b/doc/html/404.html index 15b3af1..6c73f81 100644 --- a/doc/html/404.html +++ b/doc/html/404.html @@ -13,7 +13,7 @@
Instance Space Analysis Toolbox - v0.9.1 + v0.9.2

Required products: MATLAB R2025a or later with the Statistics and Machine Learning, Optimization, Global Optimization, Parallel Computing, and Financial toolboxes.

+

A bounded serial GNU Octave workflow is also supported; see What's New for supported classifiers, outputs and persistence limits.

With the repository root on the path, this documentation also appears in the MATLAB Help browser under Supplemental Software.

References

Build the Instance Space

obj = InstanceSpace(rootdir, opts);
 obj = obj.build();
-

build runs the whole pipeline: PRELIM labels and normalises the data, SIFTED selects features, PILOT projects the instances to 2D, CLOISTER estimates the boundary of the space, PYTHIA trains a classifier per algorithm, and TRACE finds the footprints. It then saves model.mat and writes CSV and PNG files to rootdir. It takes a few minutes, most of it in SIFTED and PYTHIA.

+

build runs the whole pipeline: INIT reads and validates the metadata, PRELIM labels and normalises the data, SIFTED selects features, PILOT projects the instances to 2D, CLOISTER estimates the boundary of the space, PYTHIA trains a classifier per algorithm, and TRACE finds the footprints. It then saves model.mat and writes CSV and PNG files to rootdir. It takes a few minutes, most of it in SIFTED and PYTHIA.

Look at the Results

figure, obj.plot('portfolio')        % best algorithm on each instance
 figure, obj.plot('footprint', 6)     % footprint of the 6th algorithm
 disp(obj.model.trace.summary)        % footprint area, density and purity
diff --git a/doc/html/INIT.html b/doc/html/INIT.html
index 01c97c6..22f8bd0 100644
--- a/doc/html/INIT.html
+++ b/doc/html/INIT.html
@@ -13,7 +13,7 @@
 

   
   Instance Space Analysis Toolbox
-  v0.9.1
+  v0.9.2

Algorithms

For the standard method, PILOT solves

min  ||X - Z*B'||^2 + alpha*||Y - Z*C||^2    subject to  Z = X*A'
-

where X and Y are the preprocessed feature and performance matrices. The analytic solution takes the leading dims eigenvectors of [X Y]'*[X Y], with the Y block weighted by sqrt(alpha) (Muñoz et al., 2018). The numerical solution runs BFGS (fminunc) from opts.ntries random starts and keeps the trial with the highest correlation between the pairwise distances of the instances before and after projection, so the chosen space also preserves the topology of the data.

+

where X and Y are the preprocessed feature and performance matrices. The analytic solution takes the leading dims eigenvectors of [X Y]'*[X Y], with the Y block weighted by sqrt(alpha) (Muñoz et al., 2018). The numerical solution runs BFGS (fminunc) from opts.ntries random starts and keeps the trial with the highest correlation between the pairwise distances of the instances before and after projection, so the chosen space also preserves the topology of the data.

+

The projection has no automatic canonical orientation: equivalent spaces can appear rotated or reflected. A fixed seed also does not guarantee the same numerical restart winner across BLAS implementations or thread settings. Keep the numerical environment fixed when reproducibility is required.

GNU Octave

Octave supports serial analytic/numerical standard projection and centered SIMPLS in 2D/3D, including optimized 3D viewpoints. Native MATLAB backends are retained. Optimizer trajectories may differ; parallel execution is rejected under Octave.

Version History

v0.9.2 — Projection corrections and Octave support

Added serial Octave analytic, numerical and SIMPLS projections in 2D/3D.

PLS stores out.Xmean. Apply its projection as (X-out.Xmean)*out.A' for new instances. Standard PILOT projections do not subtract a mean.

diff --git a/doc/html/PILOTviewpoint.html b/doc/html/PILOTviewpoint.html index c501976..da7a38a 100644 --- a/doc/html/PILOTviewpoint.html +++ b/doc/html/PILOTviewpoint.html @@ -13,7 +13,7 @@
Instance Space Analysis Toolbox - v0.9.1 + v0.9.2

References

  • Smith-Miles, K. & Muñoz, M.A. (2023). Instance Space Analysis for Algorithm Testing. ACM Computing Surveys, 55(12), Article 255. https://doi.org/10.1145/3572895
    • diff --git a/doc/html/scriptcsv.html b/doc/html/scriptcsv.html index 21d38fd..0eecabf 100644 --- a/doc/html/scriptcsv.html +++ b/doc/html/scriptcsv.html @@ -13,7 +13,7 @@
    Instance Space Analysis Toolbox - v0.9.1 + v0.9.2