Repository navigation
pyspi 3.0.0 prep - #103
Merged
Merged
pyspi 3.0.0 prep#103
Conversation
…s, parallel compute, new SPIs (LaggedCorrelation, CrossPairwiseDistance)
Co-Authored-By: Claude Opus 4.6 <[email protected]>
- basic.py: fix SpearmanR/KendallTau label duplication (self.labels += [x] mutated shared class attribute; changed to self.labels = self.labels + [x]) - utils.py: stub is_jpype_jvm_available() and check_optional_deps() to always return False/empty — JIDT replaced with pure-numpy equivalents, no JVM needed - infotheory.py: implement AIS-based auto-embedding for TransferEntropy (gaussian and kraskov estimators); replaces JIDT MAX_CORR_AIS path that was returning NaN with 'unhandled estimator' warning - calculator.py: related fixes for numpy-only operation Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
CrossPairwiseDistance._dist is RMSE-by-construction (sqrt(mean((a-b)**2))), so the identifier now reflects that explicitly: xpdist_<metric>_tau-<K>_<stat>_rmse. Mirrors the _rmse suffix used by PairwiseDistance and DynamicTimeWarping when normalise=True. No behavior change.
JIDT/Java replacements have been numpy-pure since 7dca294 but the legacy gates remained: - calculator.compute() excluded kernel/symbolic/auto-embed TE from the fork worker pool under a "JVM does not survive fork()" rationale that no longer applies; reunify the parallel path. - config YAMLs declared dependencies: [java] on eight infotheory SPI classes, causing Calculator to print spurious "install java" warnings and skip 24 SPIs in fast_config.yaml. - utils.is_jpype_jvm_available() is a dead stub; fold check_optional_deps down to an empty dict and drop the "Checking optional dependencies..." print. - __init__.py's OMP_NUM_THREADS=1 comment referenced JPype; rewrite to reflect the real reason (fork-worker BLAS oversubscription). tests/test_SPIs.py previously failed collection when any current SPI key (e.g. dtw_constraint-sakoe-chiba_radius-auto) lacked a frozen-baseline entry. Compute the intersection of current vs baseline keys, skip the symmetric differences with a logged summary, and parametrize only the shared set.
Replace setup.py + requirements.txt with a self-contained pyproject.toml. Bump numpy>=2.0, scipy>=1.11, pandas>=2.0, scikit-learn>=1.3, mne>=1.6, pyEDM>=1.15. Drop sktime (replace with lazy aeon import for .ts files, which is the only place it was used). Drop future (unused). mne migration: port wavelet.py and misc.py from the removed mne.connectivity submodule to the mne-connectivity package. The new API returns Connectivity objects rather than tuples; call .get_data(output='dense') and .freqs on them. PSI now calls phase_slope_index per-band because the new API aggregates internally. numpy 2 fixes: - infotheory: use stdlib math.factorial (np.math was removed). - causal: np.max instead of Python max on 2-D slice (int() on 1-D arrays was permissive in numpy 1.x). pyEDM 2.5 fix: - causal: EmbedDimension now requires a target arg even for self- simplex; pass columns as target. pandas 3 fix: - basic: df.corr().values is read-only; copy before fill_diagonal. - calculator: defensively copy SPI returns before filling diagonal so any SPI returning a view doesn't break the fork worker. cdt still imports torch.utils.data at module load without declaring torch as a Requires-Dist, so torch stays an explicit dependency. pyEDM 2.5 no longer needs pkg_resources at runtime, so the setuptools<80 pin is no longer required.
Replaces the single-dataset z-score correctness test with a 3-dataset
ATOL/RTOL tolerance test (CML7 + new VAR1 + new Kuramoto). Baselines
come from upstream pyspi 2.0.1 run in an ephemeral uv venv (10 trials,
seed 42, element-wise mean per SPI). Drift is logged to a session-end
summary table via the spi_warning_logger fixture rather than failing
the suite — version-bump noise stays visible without blocking CI.
- tests/generate_benchmark_datasets.py: fixed-seed VAR(1) and Kuramoto
generators (M=7, T=100) matching cml7.npy dimensions.
- pyspi/data/{var1_7,kuramoto_7}.npy: frozen synthetic datasets.
- tests/{VAR1,Kuramoto}_benchmark_tables.pkl: upstream-computed
per-SPI mean/std across 10 trials.
- tests/test_SPIs.py: parametrized over (dataset, SPI), passes when
abs_diff <= ATOL (1e-6) OR abs_diff <= RTOL * |ref| (1e-2). SPIs
absent on either side are skipped with a one-line per-dataset report.
- tests/conftest.py: logger now records (max_abs, max_rel) instead of
a single z-score; summary table widened to 90 cols.
842 cases pass. Most drift appears on Kuramoto spectral/wavelet SPIs
(coherent phase data saturates them near 1.0 so any BLAS/RNG noise
crosses the 1% threshold) and on ill-conditioned covariance SPIs
(prec*, coint_johansen*) where baseline and fork disagree by large
factors that are numerical not algorithmic.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
CrossPairwiseDistance is a bespoke SPI written for the CML case study. The previous per-lag cost was a plain RMSE over the untrimmed overlap, which did not line up with how DynamicTimeWarping(normalise=True) scores the same pair. The new cost realises one specific monotone warping path — pair x[k+s] with y[k] over the overlap, then stutter the dropped boundary samples against the opposite series' corner — and normalises by sqrt(T). Because the result is a valid DTW path cost and DTW minimises over all paths, the identity dtw_rmse <= xpdist now holds by construction, which is the property the case study relies on. Co-Authored-By: Claude Opus 4.7 <[email protected]>
Two estimator-correctness fixes to infotheory.py: 1. Gaussian entropy: introduce _gaussian_log_det(cov, ridge_rel=1e-8) with epsilon = ridge_rel * mean(diag(cov)) and NaN fallback on persistent singularity. Refactor GaussianEntropyCalculator and the module-level _gaussian_entropy_from_data helper to use it. Fixes xme_gaussian_k10 overflow on Kuramoto (was -inf from slogdet sign<=0 -> 1.8e+308 via np.nan_to_num in regression test) and reduces 30+ unit drift on cce_gaussian and di_gaussian. Rationale matches JIDT's NOISE_LEVEL_TO_ADD=1e-8: cov of x+noise equals Sigma + sigma**2 * I, so ridging Sigma directly is equivalent in expectation and numerically stabler than perturbing samples. 2. KernelTECalculator: DYN_CORR_EXCL was stored via setProperty but never consumed. The compute now queries kd-tree index lists and filters neighbours by |j - i| > dce when dce > 0, matching JIDT's 2*dce+1-point exclusion window. Samples whose counts go to zero after filtering are dropped from the mean; returns NaN if all samples degenerate. Fast path (dce None or <= 0) is byte-identical to the prior implementation. 839/839 regression tests pass. Kuramoto:xme_gaussian_k10 drift drops from 1.8e+308 to 5.9; cce_gaussian 30+ -> 24.9; di_gaussian 30+ -> 17.6. Co-Authored-By: Claude Opus 4.7 <[email protected]>
- config.yaml: add LaggedCorrelation (18 variants, pearson/spearman/kendall
x tau/max_tau x squared) and CrossPairwiseDistance (2 variants), which
exist as code in pyspi/statistics/{basic,distance}.py but had no YAML
entries in the fork. The fork's config.yaml is now the global SPI
superset that downstream case configs subset from.
- config.yaml + fast_config.yaml: comment out
InformationGeometricConditionalIndependence. cdt's IGCI.predict_proba
takes no kwargs, so the 'kernel' variant has no computational backend
in the fork (and arguably upstream too).
Upstream-inherited commented-out blocks are retained:
TransferEntropy.kernel+MAX_CORR_AIS (JIDT AIS overload bug) and
PartialCoherence (no backing class). Those are deliberate upstream
exclusions, not fork choices.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Python 3.12 venvs no longer bundle setuptools by default (PEP 632), and setuptools>=81 dropped pkg_resources from the default install. pyEDM's LoadData.py still does `import pkg_resources` at module load, so without this pin a fresh venv fails test collection with ModuleNotFoundError. Mirrors the equivalent pin in mts-spi-study-cluster. Co-Authored-By: Claude Opus 4.7 <[email protected]>
- pyspi/benchmarked_config.yaml: 292-SPI set (~5s-per-SPI threshold at M=8 T=3200 and M=16 T=800). Mirrors configs/pyspi-v2/benchmarked_config.yaml in mts-spi-study-cluster; kept in-tree here as the fork-local reference. - uv.lock: pin resolved deps so downstream repos (incl. Gadi venv rebuilds) reproduce the exact environment.
Calculator.compute() gains n_jobs, checkpoint_dir, resume, mp_context, and progress kwargs. Defaults reproduce prior serial behaviour bit-for-bit; PYSPI_N_JOBS env var still honoured as a fallback. Drops the fork-only _PARALLEL_CALC global path in favour of a portable spawn-default executor. - pyspi/_parallel.py: ProcessPoolExecutor + multiprocessing.shared_memory for the dataset; SPIs bucketed by _cache_namespace so a within-class cache (covariance, spectral_mv/bv, barycenter, coint) is reused across every variant in the bucket; per-SPI failures yield NaN matrices; atomic per-SPI .npy checkpoints enable transparent resume. - pyspi/__main__.py: thin CLI (python -m pyspi compute --data ... --config ... --n-jobs ... --checkpoint-dir ...). - pyspi/statistics/*: six _cache_namespace tags on the relevant SPI bases. - tests/test_parallel.py: parity at n_jobs=2,3; checkpoint resume; failure isolation; CLI smoke. All 47 test_calc.py and 846 test_SPIs.py regression tests against the frozen benchmark tables (CML7, Kuramoto, VAR1) pass unchanged. Co-Authored-By: Claude Opus 4.7 <[email protected]>
…ness
Calculator and Data both gain verbose= (default True) so library users can
silence the per-SPI init dump, the dataset detrend/normalise banner, and
the compute() completion summary. Dependency warnings still print
unconditionally — they affect correctness, not noise.
YAML parsing extracted into pyspi.calculator.load_spis_from_yaml(), a
plain function reused by Calculator and the parallel worker initializer
(workers no longer instantiate a throwaway Calculator with redirected
stdout — direct loader call).
Data.__init__ now initialises _data/n_processes/etc. explicitly so the
empty-instance state used by add_process() is no longer load-bearing on
hasattr() checks.
Test suite layout:
tests/test_calc.py -> tests/test_calculator.py
tests/test_SPIs.py -> tests/test_regression.py (marked slow)
tests/smoke_test.py -> tests/test_smoke.py (now auto-collected;
8 tests, was 0)
pyproject.toml registers the 'slow' marker and defaults pytest to
-m 'not slow', so the 12-minute regression sweep is opt-in via -m slow
while every-commit pytest stays at ~40s.
bench/bench_compute.py is the non-notebook replacement for the sister-repo
walltime sweep. Outputs tidy CSV + JSON metadata. Two intended workflows
documented in the module docstring: amortized-config re-cutting (n_jobs=1)
and parallel-speedup characterisation (n_jobs swept).
Net: 71/71 fast tests pass (~38s). Slow regression suite intact (run with
'pytest -m slow').
Co-Authored-By: Claude Opus 4.7 <[email protected]>
The CML7/VAR1/Kuramoto baseline tables are pandas DataFrames + numpy arrays only — no closures or lambdas — so stdlib pickle suffices and dill becomes one less optional dep. Verified by loading all three .pkl files via pickle.load() and by collecting all 839 regression tests successfully. Co-Authored-By: Claude Opus 4.7 <[email protected]>
Logging: - New pyspi/_logging.py: a 'pyspi' logger with a colorama-tinted ColorFormatter. __init__.py attaches a NullHandler (library default: silent unless the app configures a handler). - Calculator(verbose=) maps to a process-global pyspi logger level: True -> INFO (config load, per-SPI init, compute summary), False -> WARNING (problems only). Dependency-exclusion notices now log at WARNING so they surface regardless of verbose. - All informational print()/Fore calls in calculator.py and data.py routed through the logger. SPI-failure reporting stays on warnings.warn (unchanged semantics; tests assert pytest.warns). - CLI gains --quiet. Per-SPI progress: - Parallel runs now tick the tqdm bar once per SPI, not once per cache bucket. Workers post (key, failed) to a manager queue after each SPI; the main loop drains it for progress while still harvesting results (and dead-worker detection) from the futures. A stuck SPI is now visible as a stalled bar. Data.__init__ drops the short-lived verbose kwarg added earlier this branch — logging supersedes it. 71/71 fast tests pass; 839 slow regression tests unaffected. Co-Authored-By: Claude Opus 4.7 <[email protected]>
mp_context default is now platform-aware: fork on Linux, spawn on macOS/Windows. Measured head-to-head (252-SPI benchmarked90 config, M=16 T=800): fork reaches 2.50x at n_jobs=4 / 3.65x at n_jobs=8, vs spawn's 1.60x / 1.95x. spawn re-imports the heavy pyspi stack and re-instantiates every SPI per worker (~2.5s tax); fork inherits that state via copy-on-write. Bug fix: run_parallel created the SharedMemory block and Manager *outside* the try, so a failure constructing either leaked the shared memory. Both are now created inside the try with None-guarded cleanup. bench/bench_compute.py upgraded to the canonical benchmark suite: - (M, T, n_jobs) grid with presets (headline/scaling/parallel/amortized) - incremental JSON output, --resume, --array-index for PBS arrays - environment block (pyspi git sha, dep versions + fingerprint, platform) - per-cell wall time, per-SPI timings, peak RSS, failed-SPI count - prints a speedup-vs-n_jobs summary Co-Authored-By: Claude Opus 4.7 <[email protected]>
run_benchmark.pbs runs bench_compute.py on Gadi (parametrised via -v: PRESET, CONFIG, REPEATS, PYSPI_DIR, VENV). README documents the presets, output schema, and the n_jobs / amortized-config invariance. Co-Authored-By: Claude Opus 4.7 <[email protected]>
cdt (causal discovery toolbox, transitive dep) autosets SETTINGS.NJOBS to cpu_count() at import. Inside an n_jobs worker pool that nests process x thread parallelism and oversubscribes. _worker_init now pins cdt.SETTINGS.NJOBS=1 after the SPI modules import. Note: this fixes cdt-based causal SPIs but NOT ConvergentCrossMapping, whose internal threading is pyEDM-side. CCM-containing configs (the full config.yaml) should be benchmarked with n_jobs=1; the benchmarked_* configs exclude CCM and parallelise cleanly. Co-Authored-By: Claude Opus 4.7 <[email protected]>
- _parallel.py: pin BLAS/OpenMP/cdt/torch in workers; _pin_blas_env sets thread env vars before spawn so workers start single-threaded. causal.py gates pyEDM parallelism (parallel/numProcess) on PYSPI_PIN_BACKENDS. - Tag ConvergentCrossMapping with _cache_namespace so its variants share one worker and cache instead of recomputing. - bench/cut_config.py: cut benchmarked<N>[_amortized] configs from a bench_compute JSON (amortized or raw cost) — replaces the notebook step. - bench/run_benchmark.pbs + run_benchmark_physics.pbs: M/T grid override and PBS-array (--array-index) support for config-cutting runs at real data sizes. - test_parallel.py: cover CCM/pinning, parametrise fork+spawn. Co-Authored-By: Claude Opus 4.7 <[email protected]>
USYD Physics has no walltime cap on the user's queue; size the PBS defaults to whole-node + a week so heavy bench cells (full config at large M,T) fit in one sequential job without overrides. Co-Authored-By: Claude Opus 4.7 <[email protected]>
bench_compute.py: write one JSON per (M, T, n_jobs) cell named
<label>_M<M>_T<T>_n<n>.json. M/T/n_jobs/environment now live at the top
level of each file (self-contained); --output-dir + --label replace
the old single --output path. --resume skips per-cell files.
cut_config.py: reads one per-cell JSON directly; --m/--t cell-picker
removed (file IS the cell).
run_benchmark[_physics].pbs (generic, env-var driven): rewritten to use
--output-dir + --label and to redirect stdout/stderr to bench/logs/.
bench/physics/run_bench_{light,m32,m64}.pbs: dedicated submission scripts
for the M={4,8,16}/{32}/{64} x T={200,400,800,1600,3200} grid. Each is a
single qsub away.
Co-Authored-By: Claude Opus 4.7 <[email protected]>
Add MXX size-applicability tags to every SPI in the bundled configs (plus the auxiliary spi_labels.csv reference table). Calculator now splits per-config labels off the constructor kwargs and merges them into the SPI's label list (handling Mxx overrides cleanly), so that downstream consumers can filter SPIs by intended M-range.
- bench_compute: psutil per-cell RSS (was process high-watermark accumulating across cells); failed_spis list; per-SPI category/labels - cut_config: handle per-config labels via _split_config_params; preserve dropped variants/classes as commented blocks by default - analyse_cells: anchor-stability Jaccard, cumulative-cost elbow, per-SPI log-linear scaling fits, extrapolation to a target cell - forecast_cell: predict cell wall-time at (M, T) from scaling fits - backfill_metadata: retrofit category/labels into older cell JSONs - physics PBS scripts: write into bench/results/cells/ to match the new layout (cells/ for per-cell JSONs, analysis/ for derived data)
Cluster sweep on the USYD Physics queue, full config.yaml at n_jobs=1:
M=4,8 across T={200,400,800,1600,3200}; M=16,32 across T={200,400,800,1600};
plus M=64 at T={200,400}. Backfilled with per-SPI category/labels at
current pyspi sha.
Analysis (bench/results/analysis/) holds the cross-cell artefacts that
feed the cut decisions: long_costs.csv, scaling fits, Jaccard matrices
at p{80,90,95,99}, cumulative-cost plots, kept-set drift heatmaps,
extrapolations to (M=64, T=3200), and a human-readable report.md.
Scaling-fit validation against the 2 unseen M=64 cells: median |log
error| ~14% per SPI; total cell-wall predictions within 13% of
measured. The (M=64, T=3200) extrapolations should be good to ~25-30%.
Re-cut from physics_config_M16_T800_n1.json (pyspi badfb4e) via bench.cut_config --mode amortized. Anchored at M=16,T=800 because kept-set Jaccard across 20 cells (M=4..64, T=200..3200) is >=0.95 at p>=90, so the anchor choice is empirically immaterial in the operating range. Counts: p80 keeps 262/328, p90 295/328, p95 312/328, p99 325/328. Dropped variants are commented out (not deleted) so the YAML carries the cut provenance. p98/p97 are deliberately omitted because the 9 CCM-with-EmbedDim variants share a cache namespace and tie in amortized cost; cuts that fall inside the tied block would partially drop the group with zero time saved. Also includes the previously-untracked non-amortized benchmarked configs (80/85/90) and the 85_amortized variant for completeness.
cut_config.amortized_costs was bucketing by _cache_namespace and
spreading the bucket's total cost evenly over every member. But two
classes have caches keyed on a constructor param within the namespace,
so members of different sub-buckets don't actually share work:
- Barycenter: cache is per (mode, pair) — variants only share within
the same mode (euclidean / dtw / sgddtw / softdtw).
- NonparametricSpectralBivariate (5 directed-spectral classes): each
class's measure call dominates the shared per-pair Connectivity
build, so amortization should be per class.
Both classes now declare a _cache_subkey property; amortized_costs and
analyse_cells group by (namespace, *subkey) so the cost of one bucket
isn't smeared across independent caches. Other cache-grouped classes
(Covariance/Precision, CCM, Cointegration, spectral_mv) have the same
bug shape but it doesn't move any cut at the percentiles in use, so
left as-is to keep the change surgical.
Re-cut benchmarked{80,90,95,99}_amortized at M=16,T=800:
- p95 / p99: unchanged (cuts above the affected SPIs).
- p90: 7 swap in / 7 swap out. Now drops hsic, softdtw, te_kraskov_DCE
(fixed-embedding) — actually expensive at large (M, T) — and keeps
the 12 cheap Barycenter variants (euclidean/dtw/sgddtw modes) that
were wrongly dropped together with softdtw. Predicted wall time at
M=64, T=3200 drops from ~11h to ~4h.
- p80: 12 swap each way. Drops dcoh + gpdcoh classes wholly (they
belong above the threshold per-class); keeps the same 12 cheap bary
variants.
Runtime parallel scheduler unaffected: it buckets by _cache_namespace
for worker placement (correct: cache lives per data instance), the
subkey is purely an amortization concept for cut decisions.
… classes Cache-aware corollary to the previous commit: once the expensive shared computation runs for one member of a cache bucket, the rest of the bucket is free at compute time (they're just cheap post-lookup transforms). Dropping a strict subset wastes information for zero time saving. New rule in cut_config.py: if any member of a cache bucket is kept, snap-promote the whole bucket. Kept count may exceed the percentile target by the size of the promoted partial buckets (header records both the target and the snap delta). To make the snap rule visible across all cache classes, declare _cache_subkey on the remaining ones (the previous commit only did Barycenter + spectral_bv since they were the ones biting in shipped cuts): - Covariance/Precision: (estimator,) - ConvergentCrossMapping: (embedding_dimension,) - Cointegration: (method, det_order, k_ar_diff) or (method, autolag, maxlag, trend) depending on method - NonparametricSpectralMultivariate: (class_name,) - GroupDelay / PhaseSlopeIndex override: (class_name, fmin, fmax) matching their narrower cache key Re-cut at M=16,T=800: only p80 changes (262 -> 267; +5 from snapping DirectedCoherence, which had 1/6 variants kept at the boundary). p90/p95/p99 unchanged (no partial buckets existed there).
Delete five configs that no code path references after the recent cache-aware re-cut: - benchmarked_config.yaml (original benchmarked cut, superseded) - benchmarked80_config.yaml (non-amortized; raw-cost variant - benchmarked85_config.yaml unused) - benchmarked90_config.yaml - benchmarked85_amortized_config.yaml (85% tier dropped; p95 covers it) Verified zero load-time references and that the four bundled subsets (fast/sonnet/fabfour, all wired into Calculator subset=) still load. Updated two example invocations in bench/README.md and bench/run_benchmark.pbs that pointed at benchmarked90_config.yaml to benchmarked90_amortized_config.yaml.
Adds the newly-completed (M=64, T=800) per-cell JSON (cell_wall ~19.6h at the full config, n_jobs=1), backfilled with per-SPI category/labels. Re-ran bench.analyse_cells over the now-21-cell corpus; scaling fits tightened (median |log error| per SPI: ~14% -> ~7%; total cell-wall prediction at M=64,T=800 within +10% of observed). All artefacts in bench/results/analysis/ refreshed accordingly.
-1 means all available cores (os.process_cpu_count / affinity aware), -2 all but one; 0 and non-integers raise instead of silently running serially. A bad PYSPI_N_JOBS now names the variable. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
…script --n-jobs now defaults to the library's PYSPI_N_JOBS handling. Adds the 'pyspi' console entry point. Co-Authored-By: Claude Opus 5.5 <[email protected]>
sklearn <=1.8 default kernel (kernel=None) is a fixed ConstantKernel(1.0)*RBF(1.0); 1.9 changed it to an optimised kernel whose length scale collapses to its lower bound, changing every anm value (up to 7.6 absolute). Pin the old default explicitly with optimizer=None so anm is identical across sklearn versions and to the existing baselines (verified bit-identical under 1.7.2, 1.8.0, 1.9.1). gpfit_* already passes an explicit kernel, so is unaffected. Co-Authored-By: Claude Opus 5.5 <[email protected]>
… job - Slow suite installs from uv.lock (the baselines were frozen there); a separate non-blocking "latest deps, advisory" job floats to latest. - Per-family drift tolerance: anm at rtol 1e-3 (ill-conditioned GP kernel); PYSPI_DRIFT_TIER=latest selects an optional looser tier. - Baseline generator records provenance (versions, platform, COMPUTATION_VERSION) in each .npz; the regeneration workflow uses uv.lock. - Run unit/packaging on push to main + pull_request only (no double runs); run the slow suite on push to main too. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Labels: sonnet/fabfour now match full.yaml (Covariance signed/multivariate, DTW and Barycenter nonlinear, PSI time-frequency labels, stray commas and the 'unisgned' typo). _merge_spi_labels lets a class's linear/nonlinear win and raises on any remaining contradictory pair. New tests/test_config_labels.py. SPIs: remove the max variants of DebiasedSquaredPhaseLagIndex and DebiasedSquaredWeightedPhaseLagIndex (always exactly 1.0). full 322->316, fast 213->207, p99 318->312, p95 305->299, p90 289->283, p80 261->255. Baseline .npz keys for the removed SPIs deleted; other arrays bit-identical. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Before KSG estimation each raw series goes through a BF_TieBreakNoise-style step (fewer than 90% unique and std>0: add N(0, (1e-10*std)^2) noise from a local fixed-seed generator; otherwise untouched). Seeds: source 0, target 1, embedding selection 2. Non-finite and constant input still raise. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…ightly scikit-learn 1.8 added the Croux-Haesbroeck consistency correction to MinCovDet/EllipticEnvelope (scikit-learn #32117), which cannot be switched off, so the cov/prec SPIs from those estimators differed between installs of the same pyspi. Require >=1.8 (and so Python >=3.11, which it needs) and regenerate only those baseline keys. GraphicalLasso(CV) stopped at the default tol=1e-4, so their values tracked the solver path, which 1.8 also changed (#31987, #31906). Converge at tol=enet_tol=1e-8 so the value is the solution, not the path. Regenerated baseline keys (all other arrays bit-identical): MinCovDet and EllipticEnvelope cov/cov-sq/prec/prec-sq on all three fixtures (max |d| 0.038, 0.53, 35.9), GraphicalLasso(CV) on all three (max |d| 1e-4). Co-Authored-By: Claude Opus 5.5 <[email protected]>
Add a JIDT attribution and citation to infotheory.py, retain JIDT's copyright notice in pyspi/lib/LICENSE-jidt.txt (listed in license-files and package data, mirroring the cdt notice), and pin JIDT 1.6.1 outputs for KSG MI/TE (Theiler windows, DCE, auto-embedding), Gaussian, kernel and symbolic estimators in tests/test_jidt_reference.py. The generating Java harness lives in tests/tools/jidt_reference/; no Java is needed to run the test. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Recompute the SPI delta against 2.0.2's config.yaml (285 -> 316: 45 added, 14 removed, 23 renamed) and list it in docs/spi_changes_2.0.2_to_3.0.0.md. Add breaking-change, deprecation and removal sections, correct the IGCI, MAX_CORR_AIS, KSG tie and kernel-TE statements, document the scikit-learn >=1.8 and dependency-floor changes, the DTW radius, the dspli/dswpli removals and the CI changes, and fix the 'Migrating from 2.x' table. Co-Authored-By: Claude Opus 5.5 <[email protected]>
gc_gaussian, di_gaussian, xme_gaussian, phi_star and sgc_parametric differ from the frozen baselines at the rounding level on Linux CI (OpenBLAS) even with every dependency locked: max relative 2.4e-7, 7e-7 with the latest dependencies. Keyed by identifier prefix; all other SPIs stay at 1e-9. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Collaborator
|
The review fixes are now on
@willedibam, please pull before pushing again, and check that nothing here clashes with work in progress. Once you and @joshuabmoore are happy, the next step is a 🤖 Generated with Claude Code |
Im sum_f conj(C(f)) C(f + df) over neighbouring in-band frequencies one multitaper bandwidth (six FFT bins) apart, from the backend's coherency. The backend's own method sums over all pairs of thinned frequencies, which no other implementation does (data2psi, FieldTrip, MNE-connectivity, HERMES) and which spectral-connectivity has since replaced upstream. That sum fell as the lag grew and changed sign near 50 samples; this index is proportional to the lag and sign-reliable to T/12 samples, at the cost of sensitivity to lags of one or two samples in a single recording (d' 1.4 against about 90 at T=1000). The three psi_multitaper_mean_* values change; their baselines are regenerated and COMPUTATION_VERSION is 3.0.0.r9. Tests: sign across lags, proportionality to the lag, an independent pair-by-pair reference, the step against the multitaper resolution, and refusal of a band too short. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Contributor
Author
|
Thanks, pulled and reviewed. I've taken everything into my local branch and the full suite is green here too (macOS arm64, scikit-learn 1.9.1).
Happy to go to |
Callers that store results need the value checkpoints are keyed on without importing a private module. Co-Authored-By: Claude Opus 5.5 <[email protected]>
P_jk = sum_f w_f |S_jk| / sqrt(sum_f w_f S_jj * sum_f w_f S_kk) from Hann-window Welch cross-spectra: the zero-lag correlation maximised over all-pass filtering of one process, completing r <= max xcorr <= P. Undirected, unsigned, in [0, 1]. nperseg defaults to the largest power of two not above T/8, capped at 256 (the reference setting at T=20000). Added to full.yaml only; the benchmarked_p* sets are unchanged. COMPUTATION_VERSION does not move: no existing SPI output changes. The three drift baselines gain the P key; their other arrays are unchanged. Co-Authored-By: Claude Opus 5.5 <[email protected]>
The automatic segment length refused T < 64, which would leave P empty on 195 of the 1,053 Zenodo library records (T = 30 to 62). Below T = 64 the segment now stays at 8 samples and the null floor rises with fewer segments (T = 30: K = 6, measured 0.37); only records with fewer than four segments (T < 20) are refused. Values at T >= 64 are unchanged. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Collaborator
|
Hi both @willedibam @benfulcher. So, I've gone through the changelog, calculator, statistics, and test modules. Relevant tests pass locally. I've also checked out the updated pyspi tutorial notebook. Everything looks fine on my end and runs without issue. The new subsets are great! I'm happy to proceed with the merge if you both are. |
The advisory latest-deps regression job failed on 18 drift checks after
tslearn 0.10.0 (2026-10-22) changed six SPIs:
- lcss_constraint-{sakoe-chiba,itakura}: tslearn < 0.10 ignored the band,
so both equalled plain `lcss` in every earlier release. tslearn 0.10
applies it by holding masked cells at zero, which pins the Itakura
parallelogram's corners: a pair whose last samples differ by more than
eps scores exactly 0. pyspi now computes banded LCSS itself with
tslearn's masks; identical to tslearn 0.10 for Sakoe-Chiba.
- bary*_sgddtw_*: tslearn 0.10 changed the subgradient DBA update and
stopping rule.
Raise the floor to tslearn>=0.10 and relock, regenerate only those six
baseline arrays (all others untouched), bump COMPUTATION_VERSION to r10,
add a regression test, and document both in CHANGELOG.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Major release. Brings
mainin line withwilledibam/pyspiv3branch. Breaking changes — see "Migrating from 2.x" inCHANGELOG.md, which is the full record; below is a summary of changes.Headline changes
jpype.cdtandtorchremoved. The four pairwise causal functions pyspi used are transcribed on NumPy/SciPy/scikit-learn, bit-identical to cdt 0.6 on the reference pairs. Installed size roughly halves.pyproject.tomlonly;setup.pyandrequirements.txtare gone.Breaking API changes
Calculator(subset=..., configfile=...)→Calculator(config=...)(bundled name or path to a YAML).normalise=→zscore=onCalculatorandData.pyspi/configs/and renamed (full,fast,sonnet,fabfour,benchmarked_p*).Calculator.save()/pyspi.load_table()as.npz; pickle and parquet output are dropped.Bundled configs
Selected by name via
Calculator(config=...); the default isfull.fullbenchmarked_p99benchmarked_p95benchmarked_p90benchmarked_p80fastsonnetfabfourThe
benchmarked_p<N>sets keep the fastest N% of SPIs by measured cost (M=16, T=800, single job). They existed in 2.x only as files; they are now reachable by name.Numerical changes
fullgoes from 328 to 322 SPIs (7 removed, 1 added), each with a stated reason.kernelandsymbolicvalues change by a factor of ln 2.dcoh_*,coint_aeg_*,psi_wavelet_*,ccm_E-None_*,gd_*,xcorr_*,kraskov, and others) are tabulated with causes under "SPI set changes".Dependencies
Python >= 3.10, numpy >= 2.0, pandas >= 2.1, pyEDM >= 2.5, spectral-connectivity >= 1.1,< 3. No setuptools pin.
Testing
.npz.Relation to open PRs and issues
CalculatorFramefix; the setuptools pin is no longer needed).E=Noneselects argmax of rho).Reviewing
pyspi/configs/, ~6.5k); testing pipelines (tests/, ~7.3k); env dependencies (uv.lock, ~3.3k); and benchmarking suites (bench/, ~2.1k).CHANGELOG.md, thenpyspi/calculator.pyandpyspi/data.py, thenpyspi/statistics/, thentests/.After merge
v3.0.0and publish the release.