Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 10 additions & 6 deletions .github/workflows/CI.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ name: CI

on:
push:
branches: [main]
branches: [master]
tags: ['*']
pull_request:
workflow_dispatch:
Expand All @@ -29,11 +29,15 @@ jobs:
with:
version: ${{ matrix.julia-version }}
- uses: julia-actions/cache@v3
- name: Install unregistered KernelSmoothing on Julia 1.10
if: matrix.julia-version == '1.10'
run: julia --project=. -e 'using Pkg; Pkg.add(url = "https://github.com/mlakolar/KernelSmoothing.jl")'
- name: Check naming conventions
run: ${{ runner.os == 'Windows' && 'python' || 'python3' }} scripts/check_naming.py
- name: Resolve dependencies from master checkouts
run: julia --color=yes scripts/instantiate.jl
- uses: julia-actions/julia-buildpkg@v1
- uses: julia-actions/julia-runtest@v1
- name: Test against dependency master checkouts
run: julia --color=yes scripts/test_master.jl
env:
JULIA_TEST_COVERAGE: "true"
- uses: julia-actions/julia-processcoverage@v1
- uses: codecov/codecov-action@v7
with:
Expand All @@ -55,7 +59,7 @@ jobs:
version: '1'
- uses: julia-actions/cache@v3
- name: Install dependencies
run: julia --project=docs -e 'using Pkg; Pkg.add(url = "https://github.com/mlakolar/KernelSmoothing.jl"); Pkg.develop(path = "."); Pkg.instantiate()'
run: julia scripts/instantiate.jl docs
- name: Build and deploy
run: julia --project=docs docs/make.jl
env:
Expand Down
6 changes: 6 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,9 @@ Manifest.toml
*.mem
.DS_Store
/data/build/

# Generated master dependency test environments
.master-test-env/

# Local Julia-minor environments are regenerated from master checkouts
/Manifest-v*.toml
26 changes: 26 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# Naming and development dependencies

Types and modules use `UpperCamelCase`. Descriptive functions, fields, keyword
arguments and private helpers use `snake_case`; mutating functions end in `!`.
Keep established numerical abbreviations (`nobs`, `atol`, `rtol`), mathematical
identifiers (`X`, `M`, `Σ`, `λ`) and names required by external interfaces.

Develop and test against local `master` checkouts of workspace dependencies.
Retain package UUIDs and version numbers during this migration. Use canonical
names in new code; deprecated bindings remain available for downstream users.
Frozen reference data and recorded numerical results are not renamed.

From the package root, run the complete suite with:

```sh
julia scripts/test_master.jl
```

The runner prepares `.master-test-env/v<major>.<minor>` with this package,
its test extras, and local master checkouts of workspace dependencies. Missing
workspace checkouts are cloned from master; existing checkouts must already be
on master. It supports Julia 1.10 and newer and keeps the package `Project.toml`
unchanged. CI uses the same runner.

Use `julia scripts/test_master.jl --setup-only` to prepare the environment for
focused checks; the runner prints the environment path to use with `--project`.
19 changes: 19 additions & 0 deletions NAMING_MIGRATION.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Naming migration

Versions and UUIDs are unchanged. The development branch is `master`; the
previous `main` branch remains available. Develop and test with the sibling
`KernelSmoothing.jl` master checkout, as specified in the project sources.

| Previous public name | Canonical name |
| --- | --- |
| `basismatrix` | `basis_matrix` |
| `modesplot`, `screeplot` | `modes_plot`, `scree_plot` |
| `covplot`, `designplot`, `fitplot`, `bandplot` | `cov_plot`, `design_plot`, `fit_plot`, `band_plot` |
| FPCA keyword and stored key `maxK` | `max_k` |
| Stored key `requested_maxK` | `requested_max_k` |

Old function names remain deprecated bindings to the same generic functions.
FPCA accepts the old `maxK` keyword and rejects simultaneous non-default old
and new values. Stored selection/settings named tuples retain the old keys
alongside canonical keys during the transition. Numerical defaults and
external StatsAPI, LinearAlgebra and Tables interfaces are unchanged.
2 changes: 1 addition & 1 deletion Project.toml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ StatsAPI = "82ae8749-77ed-4fe6-ae5f-f523153014b0"
Tables = "bd369af6-aec1-5ad0-b16a-f7cc5008161c"

[sources]
KernelSmoothing = {url = "https://github.com/mlakolar/KernelSmoothing.jl", rev = "master"}
KernelSmoothing = {path = "../KernelSmoothing.jl"}

[compat]
Artifacts = "1"
Expand Down
18 changes: 16 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# FunctionalDataAnalysis.jl

[![CI](https://github.com/mlakolar/FunctionalDataAnalysis.jl/actions/workflows/CI.yml/badge.svg?branch=main)](https://github.com/mlakolar/FunctionalDataAnalysis.jl/actions/workflows/CI.yml?query=branch%3Amain)
[![CI](https://github.com/mlakolar/FunctionalDataAnalysis.jl/actions/workflows/CI.yml/badge.svg?branch=master)](https://github.com/mlakolar/FunctionalDataAnalysis.jl/actions/workflows/CI.yml?query=branch%3Amaster)
[![Docs](https://img.shields.io/badge/docs-dev-blue.svg)](https://mlakolar.github.io/FunctionalDataAnalysis.jl/dev/)
[![codecov](https://codecov.io/gh/mlakolar/FunctionalDataAnalysis.jl/branch/main/graph/badge.svg)](https://codecov.io/gh/mlakolar/FunctionalDataAnalysis.jl)
[![codecov](https://codecov.io/gh/mlakolar/FunctionalDataAnalysis.jl/branch/master/graph/badge.svg)](https://codecov.io/gh/mlakolar/FunctionalDataAnalysis.jl)
[![Aqua QA](https://raw.githubusercontent.com/JuliaTesting/Aqua.jl/master/badge.svg)](https://github.com/JuliaTesting/Aqua.jl)

Functional data analysis in Julia, with planned support for dense, irregular
Expand All @@ -26,6 +26,20 @@ using Pkg
Pkg.add(url = "https://github.com/mlakolar/FunctionalDataAnalysis.jl")
```

## Development and testing

Use a `master` checkout for this package and its workspace dependencies. From
the package root, run the complete test suite with:

```sh
julia scripts/test_master.jl
```

The runner resolves sibling master checkouts and test extras in a dedicated
`.master-test-env` environment, leaving the package dependency declarations
unchanged. Missing workspace dependencies are cloned from their master branches.
See [CONTRIBUTING.md](CONTRIBUTING.md) for the naming and development policy.

## Licence

The package code is MIT-licensed (see `LICENSE`). The datasets loaded by
Expand Down
6 changes: 3 additions & 3 deletions benchmark/sigma2/common.jl
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ end
# Same whole-curve CurveCV loss as the package, using sufficient statistics
# at exact duplicate coordinates within each training/held-out fold. On dense
# data this avoids predicting the same pair once per subject. No smoother is
# implemented here; every prediction uses KernelSmoothing.localpolynomial.
# implemented here; every prediction uses KernelSmoothing.local_polynomial.
function grouped_cv(selector::CurveCV, fd, pool, kernel, kind)
assignments = collect(1:nobs(fd))
if selector.folds != nobs(fd)
Expand All @@ -52,7 +52,7 @@ function grouped_cv(selector::CurveCV, fd, pool, kernel, kind)
end

cv_predictions(train, held, h, kernel, ::Val{:covariance}) =
KernelSmoothing.localpolynomial((train.x, train.y), train.z, hcat(held.x, held.y);
KernelSmoothing.local_polynomial((train.x, train.y), train.z, hcat(held.x, held.y);
bandwidth=h, kernel, weights=train.weights)
cv_predictions(train, held, h, kernel, ::Val{:mean}) =
FDA._mean_predict(train.x, train.z, train.weights, held.x,
Expand Down Expand Up @@ -115,7 +115,7 @@ function estimate(prepared, estimator)
# Benchmark-only FGM-inspired reference: use the package eigen step,
# all positive components, and FGM's relative cutoff, not an FPCA K cap.
system = FDA._fpca_eigen(surface, quadrature_weights(grid), mean.values;
maxK=length(grid), eigen_tol=100eps(Float64), sign_tol=1e-12)
max_k=length(grid), eigen_tol=100eps(Float64), sign_tol=1e-12)
surface = (system.eigenfunctions .* system.eigenvalues') * system.eigenfunctions'
surface = (surface + surface')/2
# FGM bsplinekit 6f0eac3, src/doubly.jl:223-226: arithmetic mean on
Expand Down
2 changes: 1 addition & 1 deletion docs/Project.toml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Plots = "91a5bcdd-55d7-5caf-9e0b-520d859cae80"
Tables = "bd369af6-aec1-5ad0-b16a-f7cc5008161c"

[sources]
KernelSmoothing = {url = "https://github.com/mlakolar/KernelSmoothing.jl", rev = "master"}
KernelSmoothing = {path = "../../KernelSmoothing.jl"}
FunctionalDataAnalysis = {path = ".."}

[compat]
Expand Down
4 changes: 2 additions & 2 deletions docs/make.jl
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ import StatsAPI
makedocs(;
sitename = "FunctionalDataAnalysis.jl",
modules = [FunctionalDataAnalysis],
format = Documenter.HTML(; edit_link = "main"),
format = Documenter.HTML(; edit_link = "master"),
pages = [
"Home" => "index.md",
"Interface" => "interface.md",
Expand All @@ -32,5 +32,5 @@ makedocs(;

deploydocs(;
repo = "github.com/mlakolar/FunctionalDataAnalysis.jl.git",
devbranch = "main",
devbranch = "master",
)
4 changes: 2 additions & 2 deletions docs/src/bases.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ julia> using FunctionalDataAnalysis

julia> b = MonomialBasis((0, 1), 2);

julia> basismatrix(b, [0.0, 0.5, 1.0])
julia> basis_matrix(b, [0.0, 0.5, 1.0])
3×3 Matrix{Float64}:
1.0 0.0 0.0
1.0 0.5 0.25
Expand All @@ -34,7 +34,7 @@ BSplineBasis
FourierBasis
MonomialBasis
nbasis
basismatrix
basis_matrix
gram
penalty
```
4 changes: 2 additions & 2 deletions docs/src/datasets.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ growth.covariates
```

```@example datasets
model = StatsAPI.fit(FPCA, growth.data; ncomponents=2, maxK=2)
model = StatsAPI.fit(FPCA, growth.data; ncomponents=2, max_k=2)
mu = mean_data(model)
plot(mu.grid, mu.values; title="Mean height: four Berkeley sample curves",
xlabel="Age (years)", ylabel="Height (cm)")
Expand All @@ -53,7 +53,7 @@ savefig("datasets-growth-mean.svg"); nothing # hide
![](datasets-growth-mean.svg)

```@example datasets
modesplot(model, 1; xlabel="Age (years)", ylabel="Height (cm)")
modes_plot(model, 1; xlabel="Age (years)", ylabel="Height (cm)")
savefig("datasets-growth-modes.svg"); nothing # hide
```

Expand Down
20 changes: 10 additions & 10 deletions docs/src/fdapace.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ identifies the matching conventions and the differences retained by design.
| Choice | fdapace 0.6.0: executable source | FunctionalDataAnalysis.jl and reason |
|:--|:--|:--|
| Work grid / `nRegGrid` | `R/SetOptions.R`, `SetOptions`: Dense and DenseWithMV use the number of unique pooled times after `signif(t,6)`; Sparse and `p>>n` use 51. `R/FPCA.R`, `FPCA` then creates an equally spaced grid over the observed range. | Dense uses the supplied data grid and its stored weights, without rounding or resampling. It preserves the data representation. PACE uses 51 equally spaced points over the observed range unless a work grid is supplied, including when PACE is explicitly requested for common-grid data. |
| `maxK` | `SetOptions`: `min(nRegGrid-2,n-2)`, additionally capped at 20 **only for `methodMuCovEst="smooth"`**; values below 1 are reset to 1. An explicit `maxK` above `nRegGrid-2` is reduced with a message. | The dense default is `max(1,min(m-2,n-2))`; PACE uses `max(1,min(20,m-2,n-2))`, matching the respective audited routes. An explicit dense cap may use up to `min(m,n-1)` and an explicit PACE cap up to `m`; requested components above the cap or retained positive rank throw. |
| `max_k` | `SetOptions`: `min(nRegGrid-2,n-2)`, additionally capped at 20 **only for `methodMuCovEst="smooth"`**; values below 1 are reset to 1. An explicit `max_k` above `nRegGrid-2` is reduced with a message. | The dense default is `max(1,min(m-2,n-2))`; PACE uses `max(1,min(20,m-2,n-2))`, matching the respective audited routes. An explicit dense cap may use up to `min(m,n-1)` and an explicit PACE cap up to `m`; requested components above the cap or retained positive rank throw. |
| `methodSelectK` | `SetOptions`: `"FVE"`. `R/SelectK.R`, `SelectK` also accepts `"AIC"`, `"BIC"`, or a positive integer. | `ncomponents=FVE(0.99)` or a positive integer. PACE also accepts `AIC()` and `BIC()`. |
| `FVEthreshold` | `SetOptions`: 0.99 for `FPCA`. The standalone `SelectK` function separately defaults to 0.95. | `FVE(0.99)` follows the `FPCA` entry point. |
| Kernel | `SetOptions` initially chooses `"epan"` for Dense and `"gauss"` for every other data type, including DenseWithMV. A later invalid-kernel recovery branch chooses Epanechnikov for DenseWithMV. | The dense sample mean and covariance require no kernel. Optional presmoothing records the supplied smoother and kernel. PACE defaults to Gaussian, following the audited non-dense default. Direct pooled-smoother calls retain their Epanechnikov default; every kernel remains overridable. |
Expand All @@ -86,7 +86,7 @@ identifies the matching conventions and the differences retained by design.
| Smoothed noise estimator | `PC_CovE` uses the rotated diagonal; `SetOptions` supplies the middle-half `rotationCut=c(.25,.75)`, while the internal formal default is `c(0,1)`. | `SmoothedCovariance` and PACE default to the rotated diagonal over the full observed domain, selected by the issue #17 Discovery benchmark. Explicit `noise_method=:surface_diagonal` and `noise_interval=:middle_half` retain the milestone 2 rule. |
| `rho` | `SetOptions`: `methodRho="vanilla"`, `userRho=NULL`. This means no extra regularization; details are below. | `rho` regularization is not implemented. No ridge is silently added. |

The remembered claim that fdapace always uses `maxK=min(20,m-2)` is therefore
The remembered claim that fdapace always uses `max_k=min(20,m-2)` is therefore
incorrect for 0.6.0. So is a description of GCV as its default bandwidth
selector. The dense component cap follows the audited fdapace default, replacing
the milestone plan's cap of 20.
Expand Down Expand Up @@ -215,7 +215,7 @@ conditioned system is therefore not automatically regularized. An unused
solver.

The covariance used in that `Sigma_Y` is the **full positive-semidefinite
fitted covariance up to `maxK`**, not only the selected `K` components:
fitted covariance up to `max_k`**, not only the selected `K` components:
`GetEigenAnalysisResults` constructs `eigObj$fittedCov` before selecting K,
and `FPCA` passes that object to `GetCEScores`. `FVEfittedCov` controls the
separate user-returned `fittedCovUser`; it does not change this score
Expand Down Expand Up @@ -330,7 +330,7 @@ The Discovery benchmark led to `:rotated` with `:full` as the package defaults.
`R/GetEigenAnalysisResults.R`, `GetEigenAnalysisResults` computes ordinary
matrix eigenvectors of the covariance surface `G`. It keeps eigenvalues
`d>=0`, including exact zeros, drops strictly negative ones, and truncates
the remaining descending list to `maxK`. It does not record a dropped
the remaining descending list to `max_k`. It does not record a dropped
negative mass or apply a relative positive-eigenvalue cutoff.

It reports eigenvalues `lambda=h*d`, with the first regular-grid spacing
Expand All @@ -348,7 +348,7 @@ eigenvalues above `eigen_tol*maximum(abs,spectrum)` with default
`eigen_tol=1e-10`, and records the number, absolute mass, and share of the
strictly negative spectrum. The relative cutoff is an explicit numerical
choice, rather than a statistical tuning rule from fdapace. Negative mass
is measured before the `maxK` cap.
is measured before the `max_k` cap.

The sign agrees with fdapace when the mean overlap is nonzero. If its
absolute value is at most
Expand All @@ -358,7 +358,7 @@ This fallback follows the convention used by FunctionalGraphicalModels.jl.
Both numerical tolerances are stored on the fit. They are explicit package
choices that the milestone leaves to the implementation.

Both implementations compute FVE **after** the `maxK` truncation:
Both implementations compute FVE **after** the `max_k` truncation:
`cumsum(lambda)/sum(lambda)` over that retained positive spectrum. Thus the
last retained value is one even if the cap discarded positive components.
`GetEigenAnalysisResults` first uses `>=` to find its threshold crossing;
Expand All @@ -375,7 +375,7 @@ which is distinct from the full fitted covariance used by CE scoring.
When `methodSelectK` is numeric, `SetOptions` first sets `FVEthreshold=1`,
and `SelectK` then checks the fixed requested K against available components.

PACE stores the criterion for every candidate from 1 through `maxK`, then
PACE stores the criterion for every candidate from 1 through `max_k`, then
applies the first-increase rule. Beyond the retained positive rank, extra
zero-variance components leave the likelihood unchanged and increase only
the penalty. It additionally records the
Expand Down Expand Up @@ -433,7 +433,7 @@ estimation. These fixtures predate the rotated default; fdapace's enclosing
The fixed-bandwidth run estimates its mean and covariance from the sparse
fixture. It uses `userBwMu=0.30`, `userBwCov=0.40`, `kernel="epan"`,
`useBinnedData="OFF"`, `useBinnedCov=FALSE`, observation weighting,
`methodXi="CE"`, `methodRho="vanilla"`, fixed `K=3`, and `maxK=5`.
`methodXi="CE"`, `methodRho="vanilla"`, fixed `K=3`, and `max_k=5`.
Its 41-point work grid equals the observed-time union. Thus mean evaluation
before forming residual products agrees in both implementations. Public
Julia fits reproduce those mean and covariance values at relative error
Expand Down Expand Up @@ -488,7 +488,7 @@ recreate `test/data/m3_dense_input.csv` and
`test/data/m3_fdapace_dense.csv`. Each CSV begins with R and fdapace
versions. The script has no random draws. It uses 18 rank-five curves on
21 grid points, `dataType="Dense"`, cross-sectional estimation,
`error=FALSE`, integration scores, `maxK=5`, no binning, and a fixed K of 3.
`error=FALSE`, integration scores, `max_k=5`, no binning, and a fixed K of 3.
A second `FVE(0.99)` run selects K=4; a full-K run records the full FVE curve.
The first three reference eigenvalues are approximately
`4.448802620311068`, `1.165542244194415`, and `0.3479570319234146`.
Expand Down Expand Up @@ -699,7 +699,7 @@ Two source issues require explicit corrections to exercise native
`fitted.FPCA` on the milestone 3 supplied-input sparse fixture:

1. `SubsetFPCA` truncates `phi`, `xiEst`, and `lambda`, but leaves `xiVar`
at the full retained rank. With selected K smaller than `maxK`, native
at the full retained rank. With selected K smaller than `max_k`, native
fitted bands fail with nonconformable matrices. The reference script
verifies that failure and subsets only `xiVar` to its selected upper-left
block. The CE denominator still contains all five retained components.
Expand Down
Loading
Loading