From 92ad2271330d1a64f5bced7cba716df1e3b4a615 Mon Sep 17 00:00:00 2001
From: Michael Kraus <michael.kraus@ipp.mpg.de>
Date: Mon, 12 Oct 2026 18:00:49 +0200
Subject: [PATCH 01/11] Train on GeometricOptimizers' TrainingOptimizer:
 NetworkOptimizer and train! (part M1a)

NetworkOptimizer wraps GO 0.9's TrainingOptimizer over the whole parameter set;
train!(nn, opt, dl, batch, n_epochs; loss, show_progress) replaces the Optimizer
functors and optimize_for_one_epoch!. The aliases, the per-layer caches and the
hand-built step (about 400 lines) go. Compat: GO 0.9, ANN 0.9.1, NNP 0.5, SNN 0.9,
GeometricBase 0.15, GeometricEquations 0.21.5, GeometricSolutions 0.6.6.
The methods moved to ANN (3-tensor Dense/Linear functors, HDF5 save/load and
the extension) go; the call sites in tests, docs, README and scripts move, and
the CHANGELOG and KNOWN_ISSUES entries follow. GO 0.9's triangular types keep
their old GML names here, through an aliased import; the next commit renames
them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 CHANGELOG.md                                  |  16 +-
 KNOWN_ISSUES.md                               | 179 ++++----
 Project.toml                                  |  17 +-
 README.md                                     |   6 +-
 docs/check_references.jl                      |   4 +-
 docs/make.jl                                  |   3 +-
 .../architectures/abstract_neural_networks.md |   9 +-
 docs/src/introduction.md                      |   4 +-
 docs/src/optimizers/optimizer.md              |  31 +-
 .../tutorials/adjusting_the_loss_function.md  |   7 +-
 docs/src/tutorials/grassmann_layer.md         |  12 +-
 .../tutorials/hamiltonian_neural_network.md   |   8 +-
 .../linear_symplectic_transformer.md          |  14 +-
 docs/src/tutorials/matrix_softmax.md          |   8 +-
 docs/src/tutorials/optimizer_comparison.md    |   6 +-
 docs/src/tutorials/softmax_comparison.md      |   8 +-
 docs/src/tutorials/symplectic_autoencoder.md  |  27 +-
 docs/src/tutorials/symplectic_transformer.md  |  14 +-
 docs/src/tutorials/sympnet_tutorial.md        |  16 +-
 .../tutorials/volume_preserving_attention.md  |  12 +-
 ...olume_preserving_transformer_rigid_body.md |  18 +-
 ext/HDF5Ext.jl                                |  87 ----
 scripts/reproduction/harmonic_oscillator.jl   |   6 +-
 scripts/reproduction/hnn_pendulum.jl          |   9 +-
 .../reproduction/hnn_pendulum_simulation.jl   |   5 +-
 .../linear_symplectic_transformer_gpu.jl      |  14 +-
 scripts/reproduction/lnn_pendulum.jl          |  11 +-
 .../symplectic_autoencoders/online_sympnet.jl |  10 +-
 .../online_transformer_for_sae.jl             |   6 +-
 .../symplectic_autoencoders/training.jl       |  21 +-
 .../symplectic_transformer/double_pendulum.jl |  18 +-
 .../double_pendulum_short_integration.jl      |  18 +-
 .../reproduction/sympnets/sympnet_pendulum.jl |  10 +-
 .../sympnets/sympnet_toda_lattice.jl          |   6 +-
 .../softmax_comparison.jl                     |  10 +-
 .../symplectic_transformer.jl                 |  14 +-
 .../symplectic_transformer_vector_softmax.jl  |  14 +-
 .../volume_preserving_feedforward/abc_flow.jl |   6 +-
 .../double_pendulum.jl                        |   9 +-
 .../rigid_body.jl                             |   6 +-
 .../rigid_body.jl                             |   6 +-
 scripts/verification/all_zero_leaf.jl         |  78 ++++
 src/GeometricMachineLearning.jl               |  43 +-
 src/arrays/poisson_tensor.jl                  |   4 +
 src/data_loader/optimize.jl                   | 117 -----
 src/data_loader/train.jl                      |  56 +++
 src/layers/resnet.jl                          |  12 -
 src/optimizers/optimizer.jl                   | 430 +-----------------
 test/Project.toml                             |   3 +-
 .../lagrangian_neural_network_tests.jl        |   4 +-
 .../sae_error_lower_than_psd_error.jl         |   4 +-
 .../symplectic_autoencoder_tests.jl           |   8 +-
 test/data_loader/batch_data_loader_qp_test.jl |   8 +-
 .../data_loader_for_input_and_output.jl       |   4 +-
 .../data_loader_optimization_step.jl          |   8 +-
 .../optimizer_functor_with_adam.jl            |   5 +-
 test/data_loader/train.jl                     | 114 +++++
 test/data_loader/training_history_eltype.jl   |  13 +-
 test/helpers/eltypes.jl                       |   3 +
 .../network_parameters_gradient_structure.jl  |  41 +-
 test/integration/losses_and_optimization.jl   |   4 +-
 test/integration/reduced_loss_optimization.jl |  10 +-
 .../integration/train_manual_architectures.jl |  69 +++
 ...tiplication_network_parameters_gradient.jl |   8 +-
 test/layers/gradient_layer_tests.jl           |  14 +-
 test/layers/manifold_layers.jl                |  19 +-
 ...ulti_head_attention_stiefel_optim_cache.jl |  26 +-
 ...multi_head_attention_stiefel_retraction.jl |  12 +-
 .../multi_head_attention_stiefel_setup.jl     |  12 +-
 ...c_attention_network_parameters_gradient.jl |  22 +-
 test/layers/transformer_optimizer.jl          |  18 +-
 test/loss/training_method_losses.jl           |  10 +-
 test/map_to_cpu.jl                            |   3 +-
 .../adam_with_learning_rate_decay.jl          |  71 ++-
 test/optimizers/gml_gradient_dispatch.jl      | 116 +----
 test/optimizers/gradient_optimizer.jl         |  11 +-
 test/optimizers/momentum_optimizer.jl         |  11 +-
 test/optimizers/optimization_step.jl          |  90 ++--
 test/optimizers/optimizer.jl                  |  51 +++
 test/optimizers/psd_optim.jl                  |  16 +-
 test/optimizers/step_size_element_type.jl     |  57 +--
 .../optimizers/structured_array_parameters.jl |  14 +-
 test/optimizers/svd_optim.jl                  |  16 +-
 test/quality/aqua.jl                          | 131 ++----
 test/quality/exports.jl                       |  23 +-
 test/runtests.jl                              |  11 +-
 86 files changed, 1065 insertions(+), 1440 deletions(-)
 delete mode 100644 ext/HDF5Ext.jl
 create mode 100644 scripts/verification/all_zero_leaf.jl
 delete mode 100644 src/data_loader/optimize.jl
 create mode 100644 src/data_loader/train.jl
 create mode 100644 test/data_loader/train.jl
 create mode 100644 test/helpers/eltypes.jl
 create mode 100644 test/integration/train_manual_architectures.jl
 create mode 100644 test/optimizers/optimizer.jl

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 5b2b4f730..14ac6061f 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -15,8 +15,22 @@ breaking release).
 
 ## [Unreleased]
 
+### Removed (breaking)
+
+- **`Optimizer`, the callable optimizers and their aliases are gone; training moves to `NetworkOptimizer` and `train!`.** Training now uses GeometricOptimizers 0.9's `TrainingOptimizer`. `NetworkOptimizer(nn; algorithm = Adam(), linesearch = default_step_size(algorithm), retraction = Cayley())`, also accepted from a `NetworkParameters`, replaces `Optimizer`. Migrate `Optimizer(method, nn; step_size = η, retraction = cayley)` to `NetworkOptimizer(nn; algorithm = method, linesearch = η, retraction = Cayley())`, and `Optimizer(nn; AdamOptimizerWithDecay(n, T)...)` to `NetworkOptimizer(nn; AdamOptimizerWithDecay(n)...)`. The `n` of `AdamOptimizerWithDecay` counts optimization steps, one per minibatch, as in GeometricOptimizers and GML 0.8. One cache and one state now cover the whole parameter set; 0.8 had one per layer. `using GeometricOptimizers, GeometricMachineLearning` leaves `Optimizer` unqualified as GeometricOptimizers' solver, and no name is shadowed. `train!(nn, opt, dl, batch, n_epochs; loss, show_progress = true)` replaces the four callable methods `(o::Optimizer)(nn, dl, batch, n_epochs[, loss, pullback])` and `optimize_for_one_epoch!`, which is removed from the exports. Migrate `o(nn, dl, batch, n, loss; kw...)` to `train!(nn, o, dl, batch, n; loss, kw...)`. Without `loss` the defaults are those of 0.8: `TransformerLoss(batch)` for a `TransformerIntegrator` with a `Batch{:Transformer}`, `FeedForwardLoss()` for a `NeuralNetworkIntegrator`, and `AutoEncoderLoss()` for an `AutoEncoder`, each with a `Batch{:FeedForward}`. `loss` is a `NetworkLoss`; passing an `AbstractPullback` as the sixth positional argument, which 0.8 allowed, is gone. The `λY` argument and the per-call `GlobalSection` are gone: the sections, moments and iteration number live in `opt`, so a second `train!` call with the same `opt` resumes the first, and `train!` draws no random numbers beyond the minibatch shuffle. `n_epochs < 1` raises an `ArgumentError` before any step. The loss history has the element type of the loss value, so a `Float64` network on `Float32` data gives `Vector{Float64}`, where 0.8 gave `float(eltype(dl))`; the epoch accumulator keeps that type. A hand-written loop steps with `GeometricOptimizers.optimization_step!(params(nn), opt, dp)`, which GML adds for `NetworkOptimizer`; `optimization_step!(opt, λY, ps, dp)` is removed. A gradient leaf that is all zero is stepped, not skipped: NeuralNetworkParameters 0.5 gives a layer the loss does not read as a zero leaf, not `nothing`. Under `GradientMethod`, `MomentumMethod` and `Adam` such a leaf stays where it is; under `AdamWithEuclideanDecay` a Euclidean leaf that is never read decays, and a leaf read for some steps and then not keeps moving on its momentum. `scripts/verification/all_zero_leaf.jl` prints the figures.
+
+- **The aliases `GradientOptimizer`, `MomentumOptimizer` and `AdamOptimizer` are removed**; use `GradientMethod`, `MomentumMethod` and `Adam`. These are no longer exported: `OptimizerMethod`, `GradientState`, `MomentumState`, `AdamState`, `optimization_step!`, `optimize_for_one_epoch!` and `Optimizer`. `NetworkOptimizer`, `train!` and `AdamWithEuclideanDecay` are newly exported, and the package re-exports `GradientMethod`, `MomentumMethod`, `Adam`, `AdamOptimizerWithDecay`, `DecayingStatic`, `Cayley` and `Geodesic` from GeometricOptimizers. Methods carry no element type in GeometricOptimizers 0.9: `Adam(T)` becomes `Adam()`, and `MomentumMethod(α)` becomes `MomentumMethod(; α)`. `check(::Optimizer)` is removed.
+
+- **The three 3-tensor functors of `Dense` and `Linear` and the HDF5 `save` and `load` methods on `NeuralNetwork` are removed**, because AbstractNeuralNetworks 0.9 provides them. The HDF5 extension and the HDF5 weak dependency are gone; with HDF5 loaded, AbstractNeuralNetworks' extension provides `save` and `load`. Its `load` has no `backend` keyword: load on the CPU, then call `changebackend(backend, nn)`. Aqua's `piracies` check runs again with 0 piracies (was 3), and the ambiguity count is 0 (was 1). This closes issue #145 and known issue B7.
+
+### Changed (breaking)
+
+- **The `[compat]` bounds move to the versions the training API needs**: GeometricOptimizers `"0.9"`, AbstractNeuralNetworks `"0.9.1"`, NeuralNetworkParameters `"0.5"`, SymbolicNeuralNetworks `"0.9"`, GeometricBase `"0.15"`, GeometricEquations `"0.21.5"` and GeometricSolutions `"0.6.6"`. HDF5 leaves `[compat]` and `[weakdeps]`.
+
 ### Added
 
+- **`KernelAbstractions.get_backend` is defined for `PoissonTensor`.** GeometricOptimizers 0.9 asks every operand of a checked product for its backend, so `𝕁 * Y` with a `StiefelManifold` raised `ArgumentError: Implement KernelAbstractions.get_backend(::PoissonTensor...)` without it.
+
 - **The test suite runs on an Apple GPU in its `metal` group.** `test/devices/metal.jl` checks the two device paths that were measured by hand and not tested before: the products of a `PoissonTensor` with a wrapped device array, `𝕁 * view(A, [1, 2, 3, 4], :)` and `𝕁 * B'`, which `ext/GPUArraysCoreExt.jl` carries; and the `tensor_mat_mul!` kernel on `MetalBackend()`. Each result must stay on the device and match the host in `Float32`, with scalar indexing off. `Pkg.test(test_args = ["metal"])` runs them without the subject tests, and a missing device records a skip, which happens inside a sandbox. Metal (1.10 or later) is a test dependency on every platform; it installs and precompiles on Linux and Windows, and nothing there loads it. On an Apple M4 Max with Metal 1.11.1 all five assertions pass.
 - **A `Metal` workflow runs the Metal tests on GitHub's `macos-15` runner**, for the `min` and `1` Julia versions. The runner's GPU is Apple's paravirtual device, which Metal.jl supports from 1.10 and on macOS 15 or later. The workflow includes a step "Metal is functional" before the tests that fails the job where the device is unavailable, so it cannot pass without having run the tests. The job is not a required check. However, the Metal tests also run in two required checks—CI.yml's `Julia min - macOS-latest - default` and `Julia 1 - macOS-latest - default`—which execute `test/runtests.jl` with no test arguments on Apple silicon, selecting the `metal` group.
 - **`test/quality/jet.jl` checks the kernel launchers with JET.** It runs in `core`, directly after `quality/aqua.jl`, and takes about 14 s. Each function that launches a kernel of `src/` on the `CPU()` backend, the pullbacks among them, and the `Batch` functor that `test/data_loader/batch_index_set.jl` measures with `@allocated` get one `JET.report_opt` line per element type that the tests use, with `target_modules = (GeometricMachineLearning,)`. Each `@kernel` method also gets a line that analyses its generated `cpu_<kernel>` function at the launcher's context, because JET drops the reports of a kernel body when it analyses the launcher. 142 lines pass on Julia 1.11.9 (JET 0.9.20) and 1.13.1 (JET 0.12.2). Three lines report a runtime dispatch on both versions and are `@test_broken` with issue #325: `PoissonTensor(::CPU, ::Int, ::DataType)`, and the input-and-output method of `convert_input_and_batch_indices_to_array` at `Float32` and `Float64`. Where JET does not work, the file records one `@test_skip`. JET joins `test/Project.toml` with no `[compat]` bound. `KNOWN_ISSUES.md` K2 records the Revise `EMFILE` messages that JET brings into the test log, and K3 the dispatches that the lines do not see: an unstable argument of a kernel launch, and two barriers in the Poisson-tensor kernel.
@@ -27,7 +41,7 @@ breaking release).
 - `Pkg.test()` no longer runs the doctests. `test/quality/doctests.jl` is now the group `doctests`, which an empty `ARGS` does not run; `Pkg.test(test_args = ["doctests"])` runs it. In CI the Doctests job stays their runner, so the test matrix no longer runs them a second time.
 - **CI uploads coverage from the `Julia 1 - ubuntu-latest` job instead of `Julia min`, and a test job saves the Julia cache only when it succeeds.**
 - **The package needs Julia 1.12 or later** (`[compat] julia = "1.12"`, was `"1.11"`). On Julia 1.11.9, the gradient of an `LNNLoss` with respect to the network parameters stops with `StackOverflowError` in type inference ("Internal error: during type inference of … Encountered stack overflow") on macOS and Windows; the `Julia min` job on Linux passes. The overflow comes from SymbolicUtils 4.49.0, whose `rrule` for `SymbolicUtils.Code.create_array` returns an exactly typed tuple of tangents, one per array element; Zygote's reverse pass through the generated Hessian of `lagrangian_acceleration` is then inferred far deeper. With the 4.48.0 rule restored, or in a task with a 64 MB stack, the same gradient passes on 1.11.9; Julia 1.12.7 and 1.13.1 pass it on the root task. This closes #327: the `Julia min` jobs on macOS and Windows had failed in the *Lagrangian Neural Network* testset since 2026-10-01, and every testset after it was never run there.
-- **`test/Project.toml` has no `[compat]` entry for a dependency of the package.** The entries for AbstractNeuralNetworks, GeometricOptimizers, HDF5, KernelAbstractions, LinearAlgebra, NeuralNetworkParameters, Random and Zygote are gone: the test environment contains the package, so the bounds of `Project.toml` apply there, and an entry in `test/Project.toml` can only repeat or narrow them. `NeuralNetworkParameters = "0.3"` was narrower than the package's `"0.3, 0.4"`.
+- **`test/Project.toml` has no `[compat]` entry for a dependency of the package.** The entries for AbstractNeuralNetworks, GeometricOptimizers, KernelAbstractions, LinearAlgebra, NeuralNetworkParameters, Random and Zygote are gone: the test environment contains the package, so the bounds of `Project.toml` apply there, and an entry in `test/Project.toml` can only repeat or narrow them. `NeuralNetworkParameters = "0.3"` was narrower than the package's `"0.3, 0.4"`.
 - **The test suite follows one flat layout.** `test/runtests.jl` is one list of `@safetestset "<label>" include("<path>")` lines in the groups `core`, `slow` and `metal`, selected by `ARGS`; with no arguments it runs `core` and `slow`, and `metal` too on an Apple-silicon Mac, so `Pkg.test()` and CI run every test they ran before. The 14 nested `test/<subject>/runtests.jl` drivers are gone. A test file sits where the source it tests sits: `test/attention/` and `test/transformers/` went into `test/layers/`, `test/architectures/` and `test/kernels/`; `test/reduced_order_modeling/` into `test/architectures/` and `test/reduced_system/`; `test/losses/training_method_losses.jl` into `test/loss/`; the tests that span the package, the docstring examples among them, into `test/integration/`; and `aqua.jl`, `exports.jl` and `reachability.jl` into `test/quality/`. `test/docstrings/layers_and_architectures.jl`, named in *Added* above, is now `test/integration/docstrings/layers_and_architectures.jl`. `test/layers/sympnet_upscaling.jl` is in `slow`. Thirteen files that drew random numbers without a seed now call `Random.seed!(1234)`. The shared helper of the two gradient-structure tests is `test/helpers/network_parameters_gradient_structure.jl`, and its `@test_broken` names issue #319.
 - **The doctests run in the test suite.** `test/quality/doctests.jl`, in the `slow` group, runs `doctest(GeometricMachineLearning)` as the `Doctests` job of `CI.yml` does. `Documenter` joins `test/Project.toml`, with no `[compat]` bound, as a test-only dependency.
 - **The test suite compiles less.** `test/layers/sympnet_upscaling.jl` checks its three symplectic identities on 12 pairs `(N, N2)` instead of all 120 with `N` in `2:2:20` and `N2` in `(2N):2:(4N)`: `(2, 4)`, `(2, 6)`, `(2, 8)`, `(4, 10)`, `(6, 12)`, `(6, 20)`, `(8, 24)`, `(10, 30)`, `(12, 36)`, `(14, 42)`, `(20, 40)` and `(20, 80)`. They keep the smallest pair, `N2 = 2N`, `N2 = 4N`, `N2` not a multiple of `N`, and the largest `N`. The layers carry the sizes in their types, so each pair is a new compilation (about 1.3 s per pair, not once per file as the [0.8.0] entry assumed); the file has 36 assertions instead of 360, and takes 19.7 s cold; it stays in `slow`. The two training tests of `test/loss/training_method_losses.jl` use the network shape of the rest of the file, `StandardHamiltonianArchitecture(2n, 4, 1)` and `LagrangianNeuralNetwork(2; width = 4, nhidden = 1)`, in place of `(2n, 5, 2)` and `width = 5, nhidden = 2`; each still asserts that training reduces the loss. `test/architectures/lagrangian_neural_network_tests.jl` moves from `core` to `slow`: it takes 122.7 s cold, above the 60 s of a `core` file. It still runs in CI and in `Pkg.test()`, and leaves only the local `core` run. Every other `core` file takes at most 37.9 s cold. The times are cold, from one fresh Julia process that runs the `core` files in the order of `test/runtests.jl` (and from one fresh process for each of the two files named), on an Apple M4 Max with Julia 1.13.1, at a load average below 4 at the start.
diff --git a/KNOWN_ISSUES.md b/KNOWN_ISSUES.md
index 7ebbc21f1..3e6f52c18 100644
--- a/KNOWN_ISSUES.md
+++ b/KNOWN_ISSUES.md
@@ -20,8 +20,8 @@ What is known to be wrong in GeometricMachineLearning and is not fixed.
   N=5  symbolic=-0.29982310048056143  zygote=-0.05705562794369161  agree=false
   ```
 
-  `SymbolicPullback(arch)` is exported and documented as something one passes to `Optimizer` in
-  place of `ZygotePullback`, so this is wrong training, silently, for every batch size above one.
+  `SymbolicPullback(arch)` is exported as a pullback to use in place of `ZygotePullback`, so this
+  is wrong training, silently, for every batch size above one.
   It is not new — `reduce = +` is what SymbolicNeuralNetworks 0.3 did too — and it is the same
   defect SymbolicNeuralNetworks records for `FeedForwardLoss` under *Open Issues → Semantics* in its
   own changelog, where it is called out as unfixable within a design that differentiates one
@@ -45,37 +45,6 @@ What is known to be wrong in GeometricMachineLearning and is not fixed.
   every type parameter, and Aqua's `unbound_args` passes on Julia nightly. See *Fixed* above. The
   number is left vacant rather than reused.)
 
-### B7 · Three methods are type piracy, and closing them is an API change rather than a deletion.
-
-- **location:** `src/layers/resnet.jl:63`
-- **kind:** defect
-- **found:** 2026-09-19
-- **evidence:**
-  Aqua's `piracies` check reports 3, down from the 12 this entry opened with; the other nine were
-  deleted in this release and are under *Removed (breaking)* above. All three are genuine under
-  Aqua's definition — the function and every argument type belong to other modules — and all three
-  are one thing: a functor on a *layer* type this package does not own, applied to a three-axis
-  array.
-
-  | method | `src` | without GML | with GML |
-  |:--|:--|:--|:--|
-  | `(::Dense{M, N, true})(::AbstractArray{T, 3}, ::NamedTuple)` | `layers/resnet.jl:63` | `MethodError` | the layer applied along the third axis |
-  | `(::Dense{M, N, false})(::AbstractArray{T, 3}, ::NamedTuple)` | `layers/resnet.jl:67` | the same | the same |
-  | `(::Linear{M, N})(::AbstractArray{T, 3}, ::NamedTuple)` | `layers/resnet.jl:71` | the same | the same |
-
-  `Dense` and `Linear` are `AbstractNeuralNetworks`', and upstream a 3-tensor argument is a
-  `MethodError` because `*` cannot take one. So the witness is not a changed answer but a new one:
-  a call that throws without this package loaded returns a value with it.
-
-  Closing this means either `AbstractNeuralNetworks` gaining the three-axis methods, or this
-  package wrapping the two layer types in its own. Both are an API change, and neither is a
-  by-product of a piracy pass — which is why these three stayed when the other nine went.
-
-  `src/layers/resnet.jl:63` is also the one remaining method ambiguity, against
-  `AbstractNeuralNetworks.Affine`. That pair is benign and has no witness at all: `Dense` is not a
-  subtype of `Affine` and the two have no common instance, so no call can reach it. `test/quality/aqua.jl`
-  asserts both counts — 3 piracies and 1 ambiguity — so neither can drift, in either direction.
-
 ### B10 · `DataLoader(::EnsembleSolution{T, T1, Vector{ST}})` at `src/data_loader/data_loader.jl:325` has no test and appears unreachable through this package's current dependencies.
 
 - **location:** `src/data_loader/data_loader.jl:325`
@@ -135,37 +104,6 @@ What is known to be wrong in GeometricMachineLearning and is not fixed.
 
 ## C. Follow-ups and cleanups
 
-### C1 · The parameter-tree traversal still belongs upstream.
-
-- **location:** —
-- **kind:** upstream
-- **found:** 2026-08-17
-- **evidence:**
-  `_make_optimizer_cache`,
-  `_make_optimizer_state`, `_tree_optim_step!`, `_leaf_optim_step!` and the bespoke
-  `GMLEuclideanState` are GML implementations of what GeometricOptimizers supports natively for a
-  single parameter. `GMLEuclideanState` in particular duplicates what `GradientState`,
-  `MomentumState` and `AdamState` already do for a plain array.
-
-  What has to go upstream is *not* a reuse of GeometricOptimizers' `Optimizer`: that one needs an
-  `OptimizerProblem`, i.e. an objective function, and minibatch training has none — the gradient
-  arrives from AD one batch at a time. It is a new entry point there, a
-  gradient-supplied-externally step over a `NamedTuple` parameter tree. GML's `Optimizer` would then
-  be the `NeuralNetwork` constructor and the training-loop functor, and nothing else.
-
-  `Optimizer` is the one name still exported by both packages, so this is also what closes the last
-  of B1's class of collision.
-
-### C2 · Two `isa` branches remain in `_leaf_optim_step!`
-
-- **location:** —
-- **kind:** defect
-- **found:** 2026-08-16
-- **evidence:**
-  (for `AdamState`/`MomentumState`).
-  Measurement showed the traversal is not implicated in the compile-time problem, so this is tidying,
-  and it disappears entirely if C1 lands first.
-
 ### C6 · Three generated MNIST PDFs are in this branch's history
 
 - **location:** `docs/src`
@@ -241,25 +179,20 @@ What is known to be wrong in GeometricMachineLearning and is not fixed.
   where training spends its time, so the gain should be larger here than the forward figures, which
   is also why it wants its own before-and-after measurement rather than being folded into that pass.
 
-### C16 · Four of `ExplicitImports`' seven checks are switched off in `test/quality/aqua.jl`.
+### C16 · One of `ExplicitImports`' seven checks is switched off in `test/quality/aqua.jl`, and two run with a list of names.
 
 - **location:** `test/quality/aqua.jl`
 - **kind:** upstream
 - **found:** 2026-09-18
 - **evidence:**
-  The counts, at
-  the commit that added the gate: **37** names arriving through a bare `using` across thirteen
-  packages;
-  **11** explicit imports of names upstream does not mark `public`; **2** qualified accesses through a
-  non-owner module, `GeometricOptimizers.Gradient` and `GeometricOptimizers.direction`, both owned by
-  `SimpleSolvers`; and **22** qualified accesses to non-public names.
-
-  They are four separate jobs, not one. Naming the 37 is a rewrite of the module header with a
-  judgement per name. The 11 and the 22 are the same class and most of them are not this package's to
-  fix — `Architecture`, `AbstractExplicitLayer`, `add!`, `_compute_loss`, `assign_columns` and
-  `description` are all used deliberately, and the resolution is upstream declaring them `public`.
-  Each check's reason is written above the `@testset` rather than here alone, because switching one
-  back on turns the suite red on the spot.
+  `no_implicit_imports` is off: **37** names arrive through a bare `using` across thirteen packages,
+  and naming them is a rewrite of the module header with a judgement per name.
+  `all_explicit_imports_are_public` and `all_qualified_accesses_are_public` run with a list of the
+  names this package reaches in other packages that those packages do not declare `public`: **10**
+  explicit imports and **10** qualified accesses, none of them in `GeometricOptimizers`.
+  `Architecture`, `AbstractExplicitLayer`, `_compute_loss`, `dim` and the rest are used
+  deliberately, and the resolution is upstream declaring them `public`. Each check's reason is
+  written beside it in the test file.
 
 ### C17 · `accuracy` cannot be reached through any public `DataLoader` constructor.
 
@@ -347,23 +280,9 @@ What is known to be wrong in GeometricMachineLearning and is not fixed.
   supposed to be worth testing at all — training the two networks first would make the comparison
   mean something, at a cost the suite does not currently pay.
 
-## D. Unverified
-
-Not defects — claims this release makes that nothing has actually checked yet.
-
-### D4 · The upstream fix was measured on one optimizer.
-
-- **location:** —
-- **kind:** not verified
-- **found:** 2026-08-16
-- **evidence:**
-  The compile-time figures come from the
-  `Adam` path. The quasi-Newton and Newton caches and states were widened on the strength of their
-  *inferred types* — a sound argument, but not a measurement. Catalogued upstream as GeometricOptimizers C15.
-
 ## Found late
 
-### K1 · Four source comments still send the reader to the Open Issues section of `CHANGELOG.md`, which no longer exists.
+### K1 · Two source comments still send the reader to the Open Issues section of `CHANGELOG.md`, which no longer exists.
 
 - **location:** `src/reduced_system/reduced_system.jl:109`
 - **kind:** found late
@@ -373,18 +292,12 @@ Not defects — claims this release makes that nothing has actually checked yet.
   $ grep -rnE 'Open Issues|issue C1[48]|\*C21\*|\*B7\*' src test
   src/kernels/exponentials/tensor_exponential.jl:2:# name is wider than what it holds; renaming it, and `exponentials/` with it, is issue C18.
   src/reduced_system/reduced_system.jl:109:# issue C14 in `CHANGELOG.md`.
-  test/quality/aqua.jl:30:# layer types, and both are an API change rather than a tidy-up. They are *B7* under
-  test/quality/aqua.jl:31:# `## Open Issues` in `CHANGELOG.md` with that reasoning.
   test/reduced_system/reduced_system.jl:44:# for four of twelve, with four `NaN` and four in the opposite order. That is *C21* in
-  $ grep -n 'CHANGELOG' test/quality/aqua.jl
-  31:# `## Open Issues` in `CHANGELOG.md` with that reasoning.
-  63:    # and it fails when one is removed without the entry above and in `CHANGELOG.md` going with it.
   ```
 
-  The IDs are still valid; the comments at `src/reduced_system/reduced_system.jl:109`,
-  `test/quality/aqua.jl:31`, `test/quality/aqua.jl:63` and `test/reduced_system/reduced_system.jl:44–45`
-  name `CHANGELOG.md` rather than `KNOWN_ISSUES.md`. The fix is a follow-up change to those
-  comments.
+  The IDs are still valid; the comments at `src/reduced_system/reduced_system.jl:109` and
+  `test/reduced_system/reduced_system.jl:44–45` name `CHANGELOG.md` rather than `KNOWN_ISSUES.md`.
+  The fix is a follow-up change to those comments.
 
 ## Upstream
 
@@ -453,7 +366,67 @@ Not defects — claims this release makes that nothing has actually checked yet.
   `target_modules = (GeometricMachineLearning,)` a barrier on a launch argument gives 0 reports,
   and the same barrier in an `@assert` of the same launcher gives 3: the kernel functor takes
   `args...`, so the dispatch sits in KernelAbstractions' frames, which the filter drops. In
-  `assign_ones_for_poisson_tensor_kernel!` (`src/arrays/poisson_tensor.jl:91`) two mutants
+  `assign_ones_for_poisson_tensor_kernel!` (`src/arrays/poisson_tensor.jl:95`) two mutants
   SURVIVED: a barrier on `n` in `map_index_for_poisson_tensor(i, n)`, and a barrier on the written
   array `J`; a barrier on the stored value is CAUGHT. JET never reports a dynamic dispatch in the
   kernel's own frame. A barrier on a value inside each kernel body is CAUGHT.
+
+### K6 · The gradient of a `SymmetricMatrix` leaf with respect to a `NetworkParameters` differs from the one with respect to the bare `NamedTuple`.
+
+- **location:** `test/helpers/network_parameters_gradient_structure.jl`
+- **kind:** upstream
+- **found:** 2026-10-12, with the move to NeuralNetworkParameters 0.5; issue #319
+- **evidence:**
+  Under NeuralNetworkParameters 0.5, `Zygote.gradient` with respect to a `NetworkParameters` gives
+  a `SymmetricMatrix` leaf that holds the gradient with respect to the leaf's storage, at any number
+  of accesses; with respect to the bare `NamedTuple` it gives the projection of the dense gradient.
+  The off-diagonal entries of the first are twice those of the second; the diagonals are equal. A
+  test of `isapprox(Matrix(a_wrapped), Matrix(a_bare))` in
+  `layers/symplectic_attention_network_parameters_gradient.jl` evaluates:
+
+  ```
+  Expression: isapprox(Matrix(a_wrapped), Matrix(a_bare))
+   Evaluated: isapprox([0.6575109961994726 1.1214311018548968; 1.1214311018548968 0.46185556373061615],
+                       [0.6575109961994726 0.5607155509274484; 0.5607155509274484 0.46185556373061615])
+  ```
+
+  The helper asserts this relation. Which of the two a training step needs is the pullback shape of
+  issue #319.
+
+### K7 · A network with a Stiefel leaf raises in `mat_tensor_mul` when the data has another element type.
+
+- **location:** `src/kernels/mat_tensor_mul.jl`
+- **kind:** found late
+- **found:** 2026-10-12, by the critics of part M1a; the kernel code is older than that part
+- **evidence:**
+  A `Float32` `Chain(StiefelLayer(6, 3), Dense(3, 2, tanh))` trained with `train!` on `Float64`
+  data, and a `Float64` one on `Float32` data, raise `AssertionError: eltype(A) == eltype(B)` in
+  `mat_tensor_mul`, reached through `src/layers/manifold_layer.jl` under Zygote. A network of
+  `Dense` layers alone trains on mixed precisions.
+
+  ```julia
+  x = rand(Float32, 6, 12)
+  nn = NeuralNetwork(Chain(StiefelLayer(6, 3), Dense(3, 2, tanh)), Float64)
+  train!(nn, NetworkOptimizer(nn), DataLoader(x, x[1:2, :]), Batch(4), 2; loss = FeedForwardLoss())
+  ```
+
+### K8 · A `DataLoader` of one sample cannot be batched.
+
+- **location:** `src/data_loader/batch.jl`
+- **kind:** found late
+- **found:** 2026-10-12, by the critics of part M1a
+- **evidence:**
+  `DataLoader(x, x)` with `x = rand(Float32, 2, 1)` and `Batch(1)` raises the time-series
+  assertion "The number of time steps has to be greater than sequence length + prediction window"
+  when `train!` draws its minibatches.
+
+### K9 · The GPU paragraph of the manual reports training with `Optimizer`.
+
+- **location:** `docs/src/index.md:57`
+- **kind:** docs
+- **found:** 2026-10-12, by the critics of part M1a
+- **evidence:**
+  The paragraph says "so does training with `Optimizer`, which returns a `Float32` history and
+  leaves the parameters on the device". The run it reports used the `Optimizer` of
+  GeometricMachineLearning 0.8, which `NetworkOptimizer` and `train!` replace; no Metal run of
+  `train!` is recorded.
diff --git a/Project.toml b/Project.toml
index 61360a273..207d713ea 100644
--- a/Project.toml
+++ b/Project.toml
@@ -25,30 +25,27 @@ Zygote = "e88e6eb3-aa80-5325-afca-941959d7151f"
 
 [weakdeps]
 GPUArraysCore = "46192b85-c4d5-4398-a991-12ede77f4527"
-HDF5 = "f67ccb44-e63f-5c2f-98bd-6dc0ccc4ba2f"
 
 [extensions]
 GPUArraysCoreExt = "GPUArraysCore"
-HDF5Ext = "HDF5"
 
 [compat]
-AbstractNeuralNetworks = "0.8"
+AbstractNeuralNetworks = "0.9.1"
 ChainRulesCore = "1"
 ForwardDiff = "1"
 GPUArraysCore = "0.2"
-GeometricBase = "0.14"
-GeometricEquations = "0.21"
-GeometricOptimizers = "0.8"
-GeometricSolutions = "0.6"
-HDF5 = "0.16, 0.17"
+GeometricBase = "0.15"
+GeometricEquations = "0.21.5"
+GeometricOptimizers = "0.9"
+GeometricSolutions = "0.6.6"
 InteractiveUtils = "1"
 KernelAbstractions = "0.9"
 LinearAlgebra = "1"
 NNlib = "0.8, 0.9"
-NeuralNetworkParameters = "0.3, 0.4"
+NeuralNetworkParameters = "0.5"
 ProgressMeter = "1"
 Random = "1"
-SymbolicNeuralNetworks = "0.8"
+SymbolicNeuralNetworks = "0.9"
 Symbolics = "7"
 TimerOutputs = "0.5, 1"
 Zygote = "0.7"
diff --git a/README.md b/README.md
index e7c86a073..b3e80f836 100644
--- a/README.md
+++ b/README.md
@@ -43,14 +43,14 @@ backend = CUDABackend()
 # initialize the network (i.e. the parameters of the network)
 g_nn = NeuralNetwork(gsympnet, backend, type)
 
-# call the optimizer: the method comes first, the step size is given separately
-g_opt = Optimizer(Adam(type), g_nn; step_size = 1e-3)
+# set up the optimizer: the method and the step size are keywords
+g_opt = NetworkOptimizer(g_nn; algorithm = Adam(), linesearch = 1e-3)
 
 const nepochs = 300
 const batch_size = 100
 
 # train the network
-g_loss_array = g_opt(g_nn, dl, Batch(batch_size), nepochs)
+g_loss_array = train!(g_nn, g_opt, dl, Batch(batch_size), nepochs)
 
 # plot the result
 ics = (q=qp_data.q[:,1], p=qp_data.p[:,1])
diff --git a/docs/check_references.jl b/docs/check_references.jl
index e012f016f..a10e54245 100644
--- a/docs/check_references.jl
+++ b/docs/check_references.jl
@@ -12,14 +12,12 @@
 #     julia --project=docs docs/check_references.jl
 
 using GeometricMachineLearning
-using HDF5 # so that the extension below is loaded, as in make.jl
 using AbstractNeuralNetworks # signatures in `@docs` blocks name types from here
 using Documenter
 using Documenter: DocSystem
 
 # keep in sync with `modules` in make.jl
-const MODULES = Module[GeometricMachineLearning,
-    Base.get_extension(GeometricMachineLearning, :HDF5Ext)]
+const MODULES = Module[GeometricMachineLearning]
 
 const SRC = joinpath(@__DIR__, "src")
 
diff --git a/docs/make.jl b/docs/make.jl
index 7309c5aa3..ce0f0ecf6 100644
--- a/docs/make.jl
+++ b/docs/make.jl
@@ -387,8 +387,7 @@ makedocs(;
     # skips *sub*modules — and (ii) pull GeometricOptimizers' own docstrings in, whose internal
     # `@ref`s resolve in its namespace and point at bindings this manual does not document.
     # Names owned by GeometricOptimizers are referred to as plain code with a link to its manual.
-    modules = [
-        GeometricMachineLearning, Base.get_extension(GeometricMachineLearning, :HDF5Ext)],
+    modules = [GeometricMachineLearning],
     authors = "Michael Kraus, Benedikt Brantner",
     repo = "https://github.com/JuliaGNI/GeometricMachineLearning.jl/blob/{commit}{path}#L{line}",
     sitename = "GeometricMachineLearning.jl",
diff --git a/docs/src/architectures/abstract_neural_networks.md b/docs/src/architectures/abstract_neural_networks.md
index 15a7cc19e..c81a6d623 100644
--- a/docs/src/architectures/abstract_neural_networks.md
+++ b/docs/src/architectures/abstract_neural_networks.md
@@ -47,11 +47,4 @@ and we see that it consists of two layers: a [`GradientLayerQ`](@ref) and a [`Gr
 
 ## Saving and Loading
 
-`GeometricMachineLearning` adds HDF5-backed `save` and `load` methods for `NeuralNetwork` to the generics `NeuralNetworkParameters` defines. Writing and reading the parameter set itself belongs to that package; the structured parameter types (`StiefelManifold`, `SymmetricMatrix`, `SkewSymMatrix`, …) come back as themselves because `GeometricOptimizers`, which owns them, registers how each is rebuilt. Passing a prototype parameter set to `load` rebuilds against it and needs no registration at all.
-
-```@docs
-save(::HDF5.H5DataStore, ::NeuralNetwork)
-save(::AbstractString, ::NeuralNetwork)
-load(::Type{NeuralNetwork}, ::HDF5.H5DataStore, ::AbstractNeuralNetworks.Architecture)
-load(::Type{NeuralNetwork}, ::AbstractString, ::AbstractNeuralNetworks.Architecture)
-```
+With HDF5 loaded, `AbstractNeuralNetworks` has `save` and `load` methods for `NeuralNetwork`: `save(filename, nn)` writes the parameters of `nn`, and `load(NeuralNetwork, filename, arch)` reads them back into a network for `arch` on the CPU; `changebackend(backend, nn)` then moves it to a device. Writing and reading the parameter set itself belongs to `NeuralNetworkParameters`; the structured parameter types (`StiefelManifold`, `SymmetricMatrix`, `SkewSymMatrix`, …) come back as themselves because `GeometricOptimizers`, which owns them, registers how each is rebuilt. Passing a prototype parameter set to `load`, as `load(NeuralNetwork, filename, arch, prototype)`, rebuilds against it and needs no registration at all.
diff --git a/docs/src/introduction.md b/docs/src/introduction.md
index 7a9178980..bd4537559 100644
--- a/docs/src/introduction.md
+++ b/docs/src/introduction.md
@@ -19,10 +19,10 @@ Closely linked to the research presented here is the development of a software p
 [^0]: This document was produced with `GeometricMachineLearning` `v0.3`. It may be that the interface will slightly change in future versions, but efforts will be made to keep these changes as small as possible.
 
 ```@docs; canonical = false
-optimization_step!
+train!
 ```
 
-So the docstring shows the name of the type or method, in most cases how to call it and then gives some information explaining what it does and potentially hyperlinks to other similar docstrings ([`Optimizer`](@ref) in this case); all of this information is indented by a tab. Docstrings may include other information under subheaders **Arguments** (showing the arguments the method can be supplied with), **Examples** (giving more detailed examples (including results) of how to use the method) and **Implementation** (giving details on how the method is implemented). When we reference a docstring it is always printed in blue (e.g. [`optimization_step!`](@ref)), indicating a hyperlink. In addition there is an *index of docstrings* showing all docstrings in chronological order with the associated page number.
+So the docstring shows the name of the type or method, in most cases how to call it and then gives some information explaining what it does and potentially hyperlinks to other similar docstrings ([`NetworkOptimizer`](@ref) in this case); all of this information is indented by a tab. Docstrings may include other information under subheaders **Arguments** (showing the arguments the method can be supplied with), **Examples** (giving more detailed examples (including results) of how to use the method) and **Implementation** (giving details on how the method is implemented). When we reference a docstring it is always printed in blue (e.g. [`train!`](@ref)), indicating a hyperlink. In addition there is an *index of docstrings* showing all docstrings in chronological order with the associated page number.
 
 Similar to **Library Functions**, which is included in most sections, almost every chapter concludes with a section **Chapter Summary** and an additional section **References** that shows further related reading material. The **Chapter Summary** recaps the important aspects of the corresponding chapter, states again what is new (this may be mathematical or software aspects) and gives information to what other parts of the dissertation the contents of the present chapter are relevant.
 
diff --git a/docs/src/optimizers/optimizer.md b/docs/src/optimizers/optimizer.md
index ca0b095ee..c659ba499 100644
--- a/docs/src/optimizers/optimizer.md
+++ b/docs/src/optimizers/optimizer.md
@@ -1,30 +1,31 @@
-# The `Optimizer` in `GeometricMachineLearning`
+# Training a Neural Network in `GeometricMachineLearning`
 
 The general framework for optimization on homogeneous spaces — the Riemannian gradient, the lift to
 the global tangent space ``\mathfrak{g}^\mathrm{hor}``, the optimizer cache, the retraction and the
 global section — belongs to `GeometricOptimizers` and is described in its documentation, under
 [Optimization on Homogeneous Spaces](@extref GeometricOptimizers :doc:`manifold_optimizers`) and
-[Retractions](@extref GeometricOptimizers :doc:`retractions`).
+[Retractions](@extref GeometricOptimizers :doc:`retractions`). The training step itself is
+`GeometricOptimizers`' too: its `TrainingOptimizer` steps a whole parameter set with a gradient the
+caller computed, and applies the right update to each leaf — a retraction for a weight on a
+manifold, ordinary arithmetic for a Euclidean one.
 
-What `GeometricMachineLearning` adds is the part that is about *neural networks*: walking the
-parameter tree of a `NeuralNetwork`, applying the right update to each leaf — a retraction
-for a weight on a manifold, ordinary arithmetic for a Euclidean one — and driving that from a data
-loader over epochs and batches.
+What `GeometricMachineLearning` adds is the part that is about *neural networks*: a
+[`NetworkOptimizer`](@ref) built from a `NeuralNetwork`, and [`train!`](@ref), which draws the
+minibatches of a data loader over epochs, computes the gradient of the loss on each with automatic
+differentiation, and takes one step per minibatch.
 
-The gradient comes from automatic differentiation on one batch at a time, so there is no objective
-function to hand a line search; the step size is a property of the [`Optimizer`](@ref) instead. It is
-either a number, or a `GeometricOptimizers.DecayingStatic` schedule:
+The gradient comes from one minibatch at a time, so there is no objective function to hand a line
+search; the step size is a number or a `GeometricOptimizers.DecayingStatic` schedule:
 
 ```julia
-opt = Optimizer(Adam(Float32), nn; step_size = 1e-3)
-opt = Optimizer(nn; AdamOptimizerWithDecay(n_epochs, Float32)...)
+opt = NetworkOptimizer(nn; algorithm = Adam(), linesearch = 1e-3)
+opt = NetworkOptimizer(nn; AdamOptimizerWithDecay(n_steps)...)
+history = train!(nn, opt, dl, Batch(batch_size), n_epochs)
 ```
 
 ## Library Functions
 
 ```@docs
-Optimizer
-optimize_for_one_epoch!
-optimization_step!
-GeometricMachineLearning._as_go_solution
+NetworkOptimizer
+train!
 ```
diff --git a/docs/src/tutorials/adjusting_the_loss_function.md b/docs/src/tutorials/adjusting_the_loss_function.md
index 7acf859dd..6991e5417 100644
--- a/docs/src/tutorials/adjusting_the_loss_function.md
+++ b/docs/src/tutorials/adjusting_the_loss_function.md
@@ -26,11 +26,11 @@ data = DataLoader(sol; suppress_info = true)
 nn = NeuralNetwork(GSympNet(2))
 
 # train the network
-o = Optimizer(Adam(), nn)
+o = NetworkOptimizer(nn; algorithm = Adam())
 batch = Batch(32)
 n_epochs = 30
 loss = FeedForwardLoss()
-loss_array = o(nn, data, batch, n_epochs, loss; show_progress = false)
+loss_array = train!(nn, o, data, batch, n_epochs; loss, show_progress = false)
 print(loss_array[end])
 ```
 
@@ -71,7 +71,8 @@ And we train the same network with this new loss:
 ```@example change_loss
 loss = CustomLoss()
 nn_custom = NeuralNetwork(GSympNet(2))
-loss_array = o(nn_custom, data, batch, n_epochs, loss; show_progress = false)
+o_custom = NetworkOptimizer(nn_custom; algorithm = Adam())
+loss_array = train!(nn_custom, o_custom, data, batch, n_epochs; loss, show_progress = false)
 print(loss_array[end])
 ```
 
diff --git a/docs/src/tutorials/grassmann_layer.md b/docs/src/tutorials/grassmann_layer.md
index 1b1acbcd8..2ff9f7e87 100644
--- a/docs/src/tutorials/grassmann_layer.md
+++ b/docs/src/tutorials/grassmann_layer.md
@@ -105,13 +105,6 @@ nn = NeuralNetwork(model, CPU(), Float64)
 nothing # hide
 ```
 
-We then *lift* the neural network parameters via `GlobalSection`.
-
-```@example rosenbrock
-λY = GlobalSection(params(nn))
-nothing # hide
-```
-
 As the cost function ``c`` for the Wasserstein loss[^3] we simply use:
 
 [^3]: For each Wasserstein loss we need to define such a cost function. For this particular choice of ``c`` the Wasserstein distance corresponds to a ``W_2`` loss.
@@ -269,7 +262,7 @@ function compute_gradient(ps::NetworkParameters)
 end
 
 # note the very high value for the learning rate
-optimizer = Optimizer(Adam(), nn; step_size = 1e-1)
+optimizer = NetworkOptimizer(nn; algorithm = Adam(), linesearch = 1e-1)
 
 nothing # hide
 ```
@@ -278,13 +271,14 @@ So we use `Zygote` [Zygote.jl-2018](@cite) to compute ``\nabla_\theta\mathcal{NN
 
 ```@example rosenbrock
 using GeometricMachineLearning: params # hide
+using GeometricOptimizers: optimization_step!
 # note the small number of training steps
 const training_steps = 80
 loss_array = zeros(training_steps)
 for i in 1:training_steps
     val, dp = compute_gradient(params(nn))
     loss_array[i] = val
-    optimization_step!(optimizer, λY, params(nn), dp)
+    optimization_step!(params(nn), optimizer, dp)
 end
 ```
 
diff --git a/docs/src/tutorials/hamiltonian_neural_network.md b/docs/src/tutorials/hamiltonian_neural_network.md
index 4fe0b0914..86df4193e 100644
--- a/docs/src/tutorials/hamiltonian_neural_network.md
+++ b/docs/src/tutorials/hamiltonian_neural_network.md
@@ -42,8 +42,8 @@ We can now train the network
 ```@example hnn
 batch = Batch(10)
 n_epochs = 100
-o = Optimizer(Adam(Float64), hnn)
-loss_array = o(hnn, dl, batch, n_epochs, loss)
+o = NetworkOptimizer(hnn; algorithm = Adam())
+loss_array = train!(hnn, o, dl, batch, n_epochs; loss)
 using CairoMakie # hide
 lines(loss_array) # hide
 ```
@@ -84,7 +84,7 @@ nothing # hide
 We can now train the network:
 
 ```@example hnn
-o_pairs = Optimizer(Adam(Float64), hnn_pairs)
-loss_array_pairs = o_pairs(hnn_pairs, dl_pairs, batch, n_epochs, loss_pairs)
+o_pairs = NetworkOptimizer(hnn_pairs; algorithm = Adam())
+loss_array_pairs = train!(hnn_pairs, o_pairs, dl_pairs, batch, n_epochs; loss = loss_pairs)
 lines(loss_array_pairs) # hide
 ```
diff --git a/docs/src/tutorials/linear_symplectic_transformer.md b/docs/src/tutorials/linear_symplectic_transformer.md
index 7d38af6d6..6a5004c1c 100644
--- a/docs/src/tutorials/linear_symplectic_transformer.md
+++ b/docs/src/tutorials/linear_symplectic_transformer.md
@@ -63,18 +63,18 @@ nn_standard = NeuralNetwork(arch_standard)
 nn_symplectic = NeuralNetwork(arch_symplectic)
 nn_sympnet = NeuralNetwork(arch_sympnet)
 
-o_pairing = AdamOptimizerWithDecay(n_epochs, Float64)
+o_pairing = AdamOptimizerWithDecay(n_epochs)
 
-o_standard = Optimizer(nn_standard; o_pairing...)
-o_symplectic = Optimizer(nn_symplectic; o_pairing...)
-o_sympnet = Optimizer(nn_sympnet; o_pairing...)
+o_standard = NetworkOptimizer(nn_standard; o_pairing...)
+o_symplectic = NetworkOptimizer(nn_symplectic; o_pairing...)
+o_sympnet = NetworkOptimizer(nn_sympnet; o_pairing...)
 
 batch = Batch(batch_size, seq_length)
 batch2 = Batch(batch_size)
 
-loss_array_standard = o_standard(nn_standard, dl, batch, n_epochs; show_progress = false)
-loss_array_symplectic = o_symplectic(nn_symplectic, dl, batch, n_epochs; show_progress = false)
-loss_array_sympnet = o_sympnet(nn_sympnet, dl, batch2, n_epochs; show_progress = false)
+loss_array_standard = train!(nn_standard, o_standard, dl, batch, n_epochs; show_progress = false)
+loss_array_symplectic = train!(nn_symplectic, o_symplectic, dl, batch, n_epochs; show_progress = false)
+loss_array_sympnet = train!(nn_sympnet, o_sympnet, dl, batch2, n_epochs; show_progress = false)
 
 nothing # hide
 ```
diff --git a/docs/src/tutorials/matrix_softmax.md b/docs/src/tutorials/matrix_softmax.md
index e336de88e..89912801c 100644
--- a/docs/src/tutorials/matrix_softmax.md
+++ b/docs/src/tutorials/matrix_softmax.md
@@ -89,13 +89,13 @@ Training is done with the `Adam`:
 ```@example softmax_comparison
 o_method = Adam()
 
-o1 = Optimizer(o_method, nn1)
-o2 = Optimizer(o_method, nn2)
+o1 = NetworkOptimizer(nn1; algorithm = o_method)
+o2 = NetworkOptimizer(nn2; algorithm = o_method)
 
 batch = Batch(batch_size, seq_length)
 
-loss_array1 = o1(nn1, dl, batch, n_epochs; show_progress = false)
-loss_array2 = o2(nn2, dl, batch, n_epochs; show_progress = false)
+loss_array1 = train!(nn1, o1, dl, batch, n_epochs; show_progress = false)
+loss_array2 = train!(nn2, o2, dl, batch, n_epochs; show_progress = false)
 ```
 
 ```@setup softmax_comparison
diff --git a/docs/src/tutorials/optimizer_comparison.md b/docs/src/tutorials/optimizer_comparison.md
index aeabe7868..9c3e46188 100644
--- a/docs/src/tutorials/optimizer_comparison.md
+++ b/docs/src/tutorials/optimizer_comparison.md
@@ -27,14 +27,14 @@ init_cont = Tuple(init_con)
 mred = RGBf(214 / 256, 39 / 256, 40 / 256) # hide
 scatter!(ax, init_cont..., f(init_cont...); color = mred, marker = :star5)
 
-# `Optimizer` takes a `NeuralNetwork` or a `NetworkParameters`, so the bare `NamedTuple` of
+# `NetworkOptimizer` takes a `NeuralNetwork` or a `NetworkParameters`, so the bare `NamedTuple` of
 # weights has to be wrapped in the latter.
 weights = NetworkParameters((xy = init_con, ))
 η = 1e-3
 method1 = GradientMethod()
 method2 = Adam()
-optimizer1 = Optimizer(method1, weights; step_size = η)
-optimizer2 = Optimizer(method2, weights; step_size = η)
+optimizer1 = NetworkOptimizer(weights; algorithm = method1, linesearch = η)
+optimizer2 = NetworkOptimizer(weights; algorithm = method2, linesearch = η)
 
 fig
 ```
diff --git a/docs/src/tutorials/softmax_comparison.md b/docs/src/tutorials/softmax_comparison.md
index a18c70467..d8a3d8d3b 100644
--- a/docs/src/tutorials/softmax_comparison.md
+++ b/docs/src/tutorials/softmax_comparison.md
@@ -74,13 +74,13 @@ nn2 = NeuralNetwork(arch2)
 
 o_method = Adam()
 
-o1 = Optimizer(o_method, nn1)
-o2 = Optimizer(o_method, nn2)
+o1 = NetworkOptimizer(nn1; algorithm = o_method)
+o2 = NetworkOptimizer(nn2; algorithm = o_method)
 
 batch = Batch(batch_size, seq_length)
 
-loss_array1 = o1(nn1, dl, batch, n_epochs; show_progress = false)
-loss_array2 = o2(nn2, dl, batch, n_epochs; show_progress = false)
+loss_array1 = train!(nn1, o1, dl, batch, n_epochs; show_progress = false)
+loss_array2 = train!(nn2, o2, dl, batch, n_epochs; show_progress = false)
 
 nothing # hide
 ```
diff --git a/docs/src/tutorials/symplectic_autoencoder.md b/docs/src/tutorials/symplectic_autoencoder.md
index ab958083c..061d642a4 100644
--- a/docs/src/tutorials/symplectic_autoencoder.md
+++ b/docs/src/tutorials/symplectic_autoencoder.md
@@ -15,7 +15,7 @@ Main.remark(raw"As with any neural network we have to make the following choices
 " * Main.indentation * raw"5. choose a *number of epochs*,
 " * Main.indentation * raw"where points 1 and 3 depend on a variable number of hyperparameters.")
 ```
-For the symplectic autoencoder point 1 is done by calling [`SymplecticAutoencoder`](@ref), point 2 is done by calling `NeuralNetwork`, point 3 is done by calling [`Optimizer`](@ref) and point 4 is done by calling [`Batch`](@ref).
+For the symplectic autoencoder point 1 is done by calling [`SymplecticAutoencoder`](@ref), point 2 is done by calling `NeuralNetwork`, point 3 is done by calling [`NetworkOptimizer`](@ref) and point 4 is done by calling [`Batch`](@ref).
 
 ## The system
 
@@ -111,7 +111,7 @@ sae_arch = SymplecticAutoencoder(dl_cpu.input_dim, reduced_dim; n_encoder_blocks
 nothing # hide
 ```
 
-Training a neural network is usually done by calling an instance of [`Optimizer`](@ref) in `GeometricMachineLearning`. [`PSDArch`](@ref) however can be solved directly by using singular value decomposition and this is done by calling [`solve!`](@ref):  
+Training a neural network is usually done by calling [`train!`](@ref) with a [`NetworkOptimizer`](@ref) in `GeometricMachineLearning`. [`PSDArch`](@ref) however can be solved directly by using singular value decomposition and this is done by calling [`solve!`](@ref):  
 
 ```@example toda_lattice
 psd_nn_cpu = NeuralNetwork(psd_arch, CPU(), eltype(dl_cpu))
@@ -129,7 +129,7 @@ Main.remark(raw"The training blocks on this page are in plain `julia` fences, so
 " * Main.indentation * raw"instead is the *committed result* of those runs, loaded from the `.h5` files beside this page,
 " * Main.indentation * raw"so the figures below are produced from the trained networks rather than from a token
 " * Main.indentation * raw"re-training. The code paths themselves are covered by the test suite -- `ReducedLoss` through
-" * Main.indentation * raw"the `Optimizer` functor by `test/integration/reduced_loss_optimization.jl` -- and reproduced at
+" * Main.indentation * raw"`train!` by `test/integration/reduced_loss_optimization.jl` -- and reproduced at
 " * Main.indentation * raw"full size by `scripts/reproduction/symplectic_autoencoders/`, which CI runs at a smoke size
 " * Main.indentation * raw"on every pull request.")
 ```
@@ -145,10 +145,10 @@ dl = DataLoader(dl_cpu, backend, Float32)
 
 
 sae_nn_gpu = NeuralNetwork(sae_arch, CUDADevice(), Float32)
-o = Optimizer(sae_nn_gpu; AdamOptimizerWithDecay(integrator_train_epochs, Float32)...)
+o = NetworkOptimizer(sae_nn_gpu; AdamOptimizerWithDecay(integrator_train_epochs)...)
 
 # train the network
-o(sae_nn_gpu, dl, Batch(batch_size), n_epochs)
+train!(sae_nn_gpu, o, dl, Batch(batch_size), n_epochs)
 ```
 
 After training we map the network parameters to cpu:
@@ -282,7 +282,7 @@ integrator_nn = NeuralNetwork(integrator_architecture, backend)
 
 integrator_pairing = AdamOptimizerWithDecay(integrator_train_epochs)
 
-o_integrator = Optimizer(integrator_nn; integrator_pairing...)
+o_integrator = NetworkOptimizer(integrator_nn; integrator_pairing...)
 
 dl = dl_cpu # hide
 # map from autoencoder type to integrator type
@@ -294,11 +294,12 @@ nothing # hide
 ```julia
 loss = GeometricMachineLearning.ReducedLoss(encoder(sae_nn_gpu), decoder(sae_nn_gpu))
 
-train_integrator_loss = o_integrator(   integrator_nn, 
-                                        dl_integration, 
-                                        integrator_batch, 
-                                        integrator_train_epochs, 
-                                        loss)
+train_integrator_loss = train!( integrator_nn, 
+                                o_integrator, 
+                                dl_integration, 
+                                integrator_batch, 
+                                integrator_train_epochs; 
+                                loss)
 ```
 
 We can now evaluate the solution:
@@ -476,7 +477,7 @@ const integrator_architecture2 = StandardTransformerIntegrator(reduced_dim2;
                                                                             upscaling_activation = tanh)
 integrator_nn2 = NeuralNetwork(integrator_architecture2, backend)
 const integrator_pairing2 = AdamOptimizerWithDecay(integrator_train_epochs)
-const o_integrator2 = Optimizer(integrator_nn2; integrator_pairing2...)
+const o_integrator2 = NetworkOptimizer(integrator_nn2; integrator_pairing2...)
 
 loss2 = GeometricMachineLearning.ReducedLoss(encoder(psd_nn2), decoder(psd_nn2))
 nothing # hide
@@ -484,7 +485,7 @@ nothing # hide
 
 For training we leave `dl_integration`, `integrator_batch` and `integrator_train_epochs` unchanged:
 ```julia
-train_integrator_loss2 = o_integrator(integrator_nn2, dl_integration, integrator_batch, integrator_train_epochs, loss2)
+train_integrator_loss2 = train!(integrator_nn2, o_integrator2, dl_integration, integrator_batch, integrator_train_epochs; loss = loss2)
 ```
 
 ```@setup toda_lattice
diff --git a/docs/src/tutorials/symplectic_transformer.md b/docs/src/tutorials/symplectic_transformer.md
index 75fcf178a..0528fd5bc 100644
--- a/docs/src/tutorials/symplectic_transformer.md
+++ b/docs/src/tutorials/symplectic_transformer.md
@@ -63,18 +63,18 @@ nn_standard = NeuralNetwork(arch_standard)
 nn_symplectic = NeuralNetwork(arch_symplectic)
 nn_sympnet = NeuralNetwork(arch_sympnet)
 
-o_pairing = AdamOptimizerWithDecay(n_epochs, Float64)
+o_pairing = AdamOptimizerWithDecay(n_epochs)
 
-o_standard = Optimizer(nn_standard; o_pairing...)
-o_symplectic = Optimizer(nn_symplectic; o_pairing...)
-o_sympnet = Optimizer(nn_sympnet; o_pairing...)
+o_standard = NetworkOptimizer(nn_standard; o_pairing...)
+o_symplectic = NetworkOptimizer(nn_symplectic; o_pairing...)
+o_sympnet = NetworkOptimizer(nn_sympnet; o_pairing...)
 
 batch = Batch(batch_size, seq_length)
 batch2 = Batch(batch_size)
 
-loss_array_standard = o_standard(nn_standard, dl, batch, n_epochs; show_progress = false)
-loss_array_symplectic = o_symplectic(nn_symplectic, dl, batch, n_epochs; show_progress = false)
-loss_array_sympnet = o_sympnet(nn_sympnet, dl, batch2, n_epochs; show_progress = false)
+loss_array_standard = train!(nn_standard, o_standard, dl, batch, n_epochs; show_progress = false)
+loss_array_symplectic = train!(nn_symplectic, o_symplectic, dl, batch, n_epochs; show_progress = false)
+loss_array_sympnet = train!(nn_sympnet, o_sympnet, dl, batch2, n_epochs; show_progress = false)
 
 nothing # hide
 ```
diff --git a/docs/src/tutorials/sympnet_tutorial.md b/docs/src/tutorials/sympnet_tutorial.md
index 602f58a2c..846f0e067 100644
--- a/docs/src/tutorials/sympnet_tutorial.md
+++ b/docs/src/tutorials/sympnet_tutorial.md
@@ -100,10 +100,10 @@ We have to define an [optimizer](@extref GeometricOptimizers Standard-Neural-Net
 
 ```@example sympnet
 # set up optimizer; for this we first need to specify the optimization method
-opt_method = Adam(type)
-# we then call the optimizer struct which allocates the cache
-la_opt = Optimizer(opt_method, la_nn)
-g_opt = Optimizer(opt_method, g_nn)
+opt_method = Adam()
+# we then build the optimizer, which allocates the cache in the element type of the network
+la_opt = NetworkOptimizer(la_nn; algorithm = opt_method)
+g_opt = NetworkOptimizer(g_nn; algorithm = opt_method)
 
 nothing # hide
 ```
@@ -120,8 +120,8 @@ batch = Batch(batch_size)
 const nepochs = 100
 
 # perform training (returns array that contains the total loss for each training step)
-g_loss_array = g_opt(g_nn, dl, batch, nepochs; show_progress = false)
-la_loss_array = la_opt(la_nn, dl, batch, nepochs; show_progress = false)
+g_loss_array = train!(g_nn, g_opt, dl, batch, nepochs; show_progress = false)
+la_loss_array = train!(la_nn, la_opt, dl, batch, nepochs; show_progress = false)
 nothing # hide
 ```
 
@@ -242,9 +242,9 @@ parameterlength(rn_nn)
 We now train the network ``\ldots``
 
 ```@example sympnet
-rn_opt = Optimizer(opt_method, rn_nn)
+rn_opt = NetworkOptimizer(rn_nn; algorithm = opt_method)
 
-rn_loss_array = rn_opt(rn_nn, dl, batch, nepochs; show_progress = false)
+rn_loss_array = train!(rn_nn, rn_opt, dl, batch, nepochs; show_progress = false)
 nothing # hide
 ```
 
diff --git a/docs/src/tutorials/volume_preserving_attention.md b/docs/src/tutorials/volume_preserving_attention.md
index a4474faec..0a8ad3a67 100644
--- a/docs/src/tutorials/volume_preserving_attention.md
+++ b/docs/src/tutorials/volume_preserving_attention.md
@@ -117,9 +117,9 @@ Next we train the networks (here we pick a batch size of 30 and train for 1000 e
 
 ```@setup volume_preserving_attention
 function set_up_optimizers(nn_skew, nn_arb, nn_comp)
-    o_skew = Optimizer(Adam(T), nn_skew)
-    o_arb  = Optimizer(Adam(T), nn_arb)
-    o_comp = Optimizer(Adam(T), nn_comp)
+    o_skew = NetworkOptimizer(nn_skew; algorithm = Adam())
+    o_arb  = NetworkOptimizer(nn_arb; algorithm = Adam())
+    o_comp = NetworkOptimizer(nn_comp; algorithm = Adam())
 
     o_skew, o_arb, o_comp
 end
@@ -134,9 +134,9 @@ const batch = Batch(batch_size, seq_length, prediction_window)
 const batch2 = Batch(batch_size)
 
 function train_networks!(nn_skew, nn_arb, nn_comp)
-    loss_array_skew = o_skew(nn_skew, dl, batch, n_epochs, TransformerLoss(batch); show_progress = false)
-    loss_array_arb  = o_arb( nn_arb,  dl, batch, n_epochs, TransformerLoss(batch); show_progress = false)
-    loss_array_comp = o_comp(nn_comp, dl, batch2, n_epochs, FeedForwardLoss(); show_progress = false)
+    loss_array_skew = train!(nn_skew, o_skew, dl, batch, n_epochs; loss = TransformerLoss(batch), show_progress = false)
+    loss_array_arb  = train!(nn_arb,  o_arb,  dl, batch, n_epochs; loss = TransformerLoss(batch), show_progress = false)
+    loss_array_comp = train!(nn_comp, o_comp, dl, batch2, n_epochs; loss = FeedForwardLoss(), show_progress = false)
 
     loss_array_skew, loss_array_arb, loss_array_comp
 end
diff --git a/docs/src/tutorials/volume_preserving_transformer_rigid_body.md b/docs/src/tutorials/volume_preserving_transformer_rigid_body.md
index 487c659c7..17a451918 100644
--- a/docs/src/tutorials/volume_preserving_transformer_rigid_body.md
+++ b/docs/src/tutorials/volume_preserving_transformer_rigid_body.md
@@ -199,17 +199,17 @@ const n_epochs = 500000
 const batch_size = 16384
 const feedforward_batch = Batch(batch_size)
 const transformer_batch = Batch(batch_size, seq_length, seq_length)
-const opt_pairing = AdamOptimizerWithDecay(n_epochs, T; η₁ = 1e-2, η₂ = 1e-6)
+const opt_pairing = AdamOptimizerWithDecay(n_epochs; η₁ = 1e-2, η₂ = 1e-6)
 
-o_vpff = Optimizer(nn_vpff; opt_pairing...)
-o_vpt = Optimizer(nn_vpt; opt_pairing...)
-o_st = Optimizer(nn_st; opt_pairing...)
+o_vpff = NetworkOptimizer(nn_vpff; opt_pairing...)
+o_vpt = NetworkOptimizer(nn_vpt; opt_pairing...)
+o_st = NetworkOptimizer(nn_st; opt_pairing...)
 nothing # hide
 ```
 ```julia
-o_vpff(nn_vpff, dl, feedforward_batch, n_epochs)
-o_vpt(nn_vpt, dl, transformer_batch, n_epochs)
-o_st(nn_st, dl, transformer_batch, n_epochs)
+train!(nn_vpff, o_vpff, dl, feedforward_batch, n_epochs)
+train!(nn_vpt, o_vpt, dl, transformer_batch, n_epochs)
+train!(nn_st, o_st, dl, transformer_batch, n_epochs)
 ```
 
 After the networks have been trained we map the parameters to cpu:
@@ -220,6 +220,10 @@ nn_vpff = mtc(nn_vpff)
 nn_vpt = mtc(nn_vpt)
 nn_st = mtc(nn_st)
 using HDF5 # hide
+# the two files store the type names `LowerTriangular` and `UpperTriangular` # hide
+const NNP = parentmodule(NetworkParameters) # hide
+NNP.register_parameter_type!("LowerTriangular", (S, md) -> LowerTriangular(S.S, S.n)) # hide
+NNP.register_parameter_type!("UpperTriangular", (S, md) -> UpperTriangular(S.S, S.n)) # hide
 nn_vpff = load(NeuralNetwork, "transformer_rigid_body_nn_vpff.h5", nn_vpff.architecture) # hide
 nn_vpt = load(NeuralNetwork, "transformer_rigid_body_nn_vpt.h5", nn_vpt.architecture) # hide
 nn_st = load(NeuralNetwork, "transformer_rigid_body_nn_st.h5", nn_st.architecture) # hide
diff --git a/ext/HDF5Ext.jl b/ext/HDF5Ext.jl
deleted file mode 100644
index 39912a134..000000000
--- a/ext/HDF5Ext.jl
+++ /dev/null
@@ -1,87 +0,0 @@
-module HDF5Ext
-
-using HDF5
-using GeometricMachineLearning
-import AbstractNeuralNetworks: NeuralNetworkBackend, Architecture
-# `save`, `load`, `params` and the parameter container are `NeuralNetworkParameters`' as of
-# `AbstractNeuralNetworks` 0.7, which only re-binds them; reach for them where they are defined.
-import NeuralNetworkParameters: NetworkParameters, params, save, load
-
-# ---------------------------------------------------------------------------
-# save / load — the entry points that dispatch on this package's `NeuralNetwork`.
-#
-# The traversal itself is not here. `NeuralNetworkParameters` walks the parameter set and records
-# each group's key order; `GeometricOptimizers` says through `freeparameters`/`rebuild` where each
-# structured matrix keeps its numbers, and registers the types so a file loads with no prototype.
-# ---------------------------------------------------------------------------
-
-"""
-    save(h5::HDF5.H5DataStore, nn::NeuralNetwork)
-
-Save the parameters of `nn` into an already-open HDF5 store.
-
-Extends `save` with a dispatch on `NeuralNetwork`. The parameters themselves are written by
-`NeuralNetworkParameters`, which tags each structured leaf with the type to rebuild it as and
-records the key order of every group.
-"""
-save(h5::HDF5.H5DataStore, nn::NeuralNetwork) = save(h5, params(nn))
-
-"""
-    save(filename::AbstractString, nn::NeuralNetwork)
-
-Convenience overload: open `filename` for writing, call [`save`](@ref) on the store, and return
-`filename`.
-"""
-function save(filename::AbstractString, nn::NeuralNetwork)
-    HDF5.h5open(filename, "w") do h5
-        save(h5, nn)
-    end
-    filename
-end
-
-"""
-    load(::Type{NeuralNetwork}, h5::HDF5.H5DataStore, arch::Architecture; backend = CPU())
-    load(::Type{NeuralNetwork}, h5::HDF5.H5DataStore, arch::Architecture, prototype; backend = CPU())
-
-Load network parameters from an already-open HDF5 store and return a `NeuralNetwork` for `arch`.
-
-The element type is whatever the file holds, so a `Float32` network reloads as `Float32`.
-
-Structured parameters — `StiefelManifold`, `SymmetricMatrix` and the rest — are rebuilt from the
-type each was stored under, which `GeometricOptimizers` registers with
-`NeuralNetworkParameters.register_parameter_type!`. Pass `prototype`, a parameter set of the right
-shape, to rebuild against it instead and skip the registry altogether.
-"""
-function load(::Type{NeuralNetwork}, h5::HDF5.H5DataStore, arch::Architecture;
-        backend::NeuralNetworkBackend = CPU())
-    NeuralNetwork(arch, Chain(arch), load(NetworkParameters, h5), backend)
-end
-
-function load(::Type{NeuralNetwork}, h5::HDF5.H5DataStore, arch::Architecture, prototype;
-        backend::NeuralNetworkBackend = CPU())
-    NeuralNetwork(arch, Chain(arch), load(NetworkParameters, h5, prototype), backend)
-end
-
-"""
-    load(::Type{NeuralNetwork}, filename::AbstractString, arch::Architecture; backend = CPU())
-    load(::Type{NeuralNetwork}, filename::AbstractString, arch::Architecture, prototype; backend = CPU())
-
-Convenience overload: open `filename` for reading, then call
-[`load`](@ref) on the store.
-"""
-function load(::Type{NeuralNetwork}, filename::AbstractString, arch::Architecture;
-        backend::NeuralNetworkBackend = CPU())
-    HDF5.h5open(filename, "r") do h5
-        load(NeuralNetwork, h5, arch; backend = backend)
-    end
-end
-
-function load(
-        ::Type{NeuralNetwork}, filename::AbstractString, arch::Architecture, prototype;
-        backend::NeuralNetworkBackend = CPU())
-    HDF5.h5open(filename, "r") do h5
-        load(NeuralNetwork, h5, arch, prototype; backend = backend)
-    end
-end
-
-end
diff --git a/scripts/reproduction/harmonic_oscillator.jl b/scripts/reproduction/harmonic_oscillator.jl
index 6c88edc91..70fba8365 100644
--- a/scripts/reproduction/harmonic_oscillator.jl
+++ b/scripts/reproduction/harmonic_oscillator.jl
@@ -13,17 +13,17 @@ ensemble_solution = exact_solution(ensemble_problem)
 
 include("../utilities/ensemble_plots.jl")
 
-# `GSympNet` trains through the generic `DataLoader` + `Batch` + `Optimizer` pipeline directly,
+# `GSympNet` trains through the generic `DataLoader` + `Batch` + `NetworkOptimizer` pipeline directly,
 # as `scripts/reproduction/sympnets/sympnet_toda_lattice.jl` already does.
 dl = DataLoader(ensemble_solution)
 
 arch = GSympNet(dl; n_layers = 4, upscaling_dimension = 10)
 nn = NeuralNetwork(arch, Float64)
-o = Optimizer(AdamOptimizer(), nn)
+o = NetworkOptimizer(nn; algorithm = Adam())
 batch = Batch(100)
 n_epochs = smoke_size(1000, 2)
 
-loss_array = o(nn, dl, batch, n_epochs)
+loss_array = train!(nn, o, dl, batch, n_epochs)
 
 function H(x)
     hamiltonian(
diff --git a/scripts/reproduction/hnn_pendulum.jl b/scripts/reproduction/hnn_pendulum.jl
index 97aece546..889795b64 100644
--- a/scripts/reproduction/hnn_pendulum.jl
+++ b/scripts/reproduction/hnn_pendulum.jl
@@ -34,10 +34,11 @@ dl = DataLoader(input, output)
 # in the data. It is the loss `NetworkLoss(hnn)` returns, so naming it is for the reader.
 loss = HNNLoss(hnn)
 
-# The step size belongs to the `Optimizer`, not to the method: the same `Adam()` trains at any
-# learning rate. `1e-3` is already Adam's default here and is named for the reader.
-optimizer = Optimizer(Adam(), nn; step_size = 1e-3)
-total_loss = optimizer(nn, dl, Batch(10), nepochs, loss; show_progress = true)
+# The step size is the `NetworkOptimizer`'s `linesearch`, not the method's: the same `Adam()` trains
+# at any learning rate. `1e-3` is already Adam's default here and is named for the reader.
+optimizer = NetworkOptimizer(nn; algorithm = Adam(), linesearch = 1e-3)
+total_loss = train!(
+    nn, optimizer, dl, Batch(10), nepochs; loss = loss, show_progress = true)
 
 # plot results
 include("../utilities/plots.jl")
diff --git a/scripts/reproduction/hnn_pendulum_simulation.jl b/scripts/reproduction/hnn_pendulum_simulation.jl
index c8ea9cfb0..3f39be03b 100644
--- a/scripts/reproduction/hnn_pendulum_simulation.jl
+++ b/scripts/reproduction/hnn_pendulum_simulation.jl
@@ -36,8 +36,9 @@ dl = DataLoader(input, output)
 # symbolic derivative again.
 const loss = HNNLoss(hnn)
 
-optimizer = Optimizer(Adam(), nn; step_size = 1e-3)
-total_loss = optimizer(nn, dl, Batch(10), nepochs, loss; show_progress = true)
+optimizer = NetworkOptimizer(nn; algorithm = Adam(), linesearch = 1e-3)
+total_loss = train!(
+    nn, optimizer, dl, Batch(10), nepochs; loss = loss, show_progress = true)
 
 # The learned Hamiltonian and the vector field built from it. `loss.hvf` is what `HNNLoss` compares
 # against the data, so this integrates exactly the quantity that was trained -- the same object,
diff --git a/scripts/reproduction/linear_symplectic_transformer_gpu.jl b/scripts/reproduction/linear_symplectic_transformer_gpu.jl
index 7085b392e..80bce685a 100644
--- a/scripts/reproduction/linear_symplectic_transformer_gpu.jl
+++ b/scripts/reproduction/linear_symplectic_transformer_gpu.jl
@@ -37,18 +37,18 @@ nn_standard = NeuralNetwork(arch_standard, backend, T)
 nn_symplectic = NeuralNetwork(arch_symplectic, backend, T)
 nn_sympnet = NeuralNetwork(arch_sympnet, backend, T)
 
-o_method = AdamOptimizer()
+o_method = Adam()
 
-o_standard = Optimizer(o_method, nn_standard)
-o_symplectic = Optimizer(o_method, nn_symplectic)
-o_sympnet = Optimizer(o_method, nn_sympnet)
+o_standard = NetworkOptimizer(nn_standard; algorithm = o_method)
+o_symplectic = NetworkOptimizer(nn_symplectic; algorithm = o_method)
+o_sympnet = NetworkOptimizer(nn_sympnet; algorithm = o_method)
 
 batch = Batch(batch_size, seq_length)
 batch2 = Batch(batch_size)
 
-loss_array_standard = o_standard(nn_standard, dl, batch, n_epochs)
-loss_array_symplectic = o_symplectic(nn_symplectic, dl, batch, n_epochs)
-loss_array_sympnet = o_sympnet(nn_sympnet, dl, batch2, n_epochs)
+loss_array_standard = train!(nn_standard, o_standard, dl, batch, n_epochs)
+loss_array_symplectic = train!(nn_symplectic, o_symplectic, dl, batch, n_epochs)
+loss_array_sympnet = train!(nn_sympnet, o_sympnet, dl, batch2, n_epochs)
 
 fig_train = Figure()
 ax_train = Axis(fig_train[1, 1]; xlabel = "epoch", ylabel = "training error", yscale = log10)
diff --git a/scripts/reproduction/lnn_pendulum.jl b/scripts/reproduction/lnn_pendulum.jl
index 1a45542cf..544fae832 100644
--- a/scripts/reproduction/lnn_pendulum.jl
+++ b/scripts/reproduction/lnn_pendulum.jl
@@ -43,11 +43,12 @@ dl = DataLoader(input, output)
 # `LNNLoss(arch)` is what `NetworkLoss(arch)` returns; naming it is for the reader.
 loss = LNNLoss(arch)
 
-# The step size belongs to the `Optimizer`, not to the method. `LNNLoss` needs a larger one than
-# `HNNLoss` does: its gradient reaches the parameters through a solve against the velocity Hessian,
-# where `HNNLoss` is linear in the gradient of the learned Hamiltonian.
-optimizer = Optimizer(Adam(), nn; step_size = 1e-2)
-total_loss = optimizer(nn, dl, Batch(10), nepochs, loss; show_progress = true)
+# The step size is the `NetworkOptimizer`'s `linesearch`, not the method's. `LNNLoss` needs a larger
+# one than `HNNLoss` does: its gradient reaches the parameters through a solve against the velocity
+# Hessian, where `HNNLoss` is linear in the gradient of the learned Hamiltonian.
+optimizer = NetworkOptimizer(nn; algorithm = Adam(), linesearch = 1e-2)
+total_loss = train!(
+    nn, optimizer, dl, Batch(10), nepochs; loss = loss, show_progress = true)
 
 # plot results
 include("../utilities/plots.jl")
diff --git a/scripts/reproduction/symplectic_autoencoders/online_sympnet.jl b/scripts/reproduction/symplectic_autoencoders/online_sympnet.jl
index 742d55d44..d93ced980 100644
--- a/scripts/reproduction/symplectic_autoencoders/online_sympnet.jl
+++ b/scripts/reproduction/symplectic_autoencoders/online_sympnet.jl
@@ -38,12 +38,12 @@ const n_epochs = smoke_size(262144, 2)
 const batch_size = 4096
 
 sae_pairing = AdamOptimizerWithDecay(n_epochs)
-o = Optimizer(sae_nn; sae_pairing...)
+o = NetworkOptimizer(sae_nn; sae_pairing...)
 
 println("Number of batches: ", GeometricMachineLearning.number_of_batches(dl, Batch(batch_size)))
 
 psd_error = solve!(psd_nn, dl)
-sae_error = o(sae_nn, dl, Batch(batch_size), n_epochs)
+sae_error = train!(sae_nn, o, dl, Batch(batch_size), n_epochs)
 
 fig = Figure()
 ax = Axis(fig[1, 1], xlabel = "epoch", ylabel = "training error")
@@ -106,7 +106,7 @@ integrator_architecture = StandardTransformerIntegrator(
 # the fallback raises is `Functor not defined`, not a `MethodError`.
 integrator_nn = NeuralNetwork(integrator_architecture, backend, Float32)
 integrator_pairing = AdamOptimizerWithDecay(integrator_train_epochs)
-o_integrator = Optimizer(integrator_nn; integrator_pairing...)
+o_integrator = NetworkOptimizer(integrator_nn; integrator_pairing...)
 
 loss = GeometricMachineLearning.ReducedLoss(encoder(sae_nn), decoder(sae_nn))
 
@@ -116,8 +116,8 @@ dl_integration = DataLoader(dl; autoencoder = false)
 # the regular transformer can't deal with symplectic data!
 dl_integration = DataLoader(vcat(dl_integration.input.q, dl_integration.input.p))
 integrator_batch = Batch(integrator_batch_size, seq_length)
-train_integrator_loss = o_integrator(
-    integrator_nn, dl_integration, integrator_batch, integrator_train_epochs, loss)
+train_integrator_loss = train!(integrator_nn, o_integrator, dl_integration,
+    integrator_batch, integrator_train_epochs; loss = loss)
 
 JLD2.save("integrator_parameters.jld2", "integrator_parameters",
     integrator_nn.params |> mtc, "training loss", train_integrator_loss)
diff --git a/scripts/reproduction/symplectic_autoencoders/online_transformer_for_sae.jl b/scripts/reproduction/symplectic_autoencoders/online_transformer_for_sae.jl
index 0a911ed1d..ef657909f 100644
--- a/scripts/reproduction/symplectic_autoencoders/online_transformer_for_sae.jl
+++ b/scripts/reproduction/symplectic_autoencoders/online_transformer_for_sae.jl
@@ -52,7 +52,7 @@ const integrator_architecture = StandardTransformerIntegrator(
 # the fallback raises is `Functor not defined`, not a `MethodError`.
 const integrator_nn = NeuralNetwork(integrator_architecture, backend, Float32)
 const integrator_pairing = AdamOptimizerWithDecay(integrator_train_epochs)
-const o_integrator = Optimizer(integrator_nn; integrator_pairing...)
+const o_integrator = NetworkOptimizer(integrator_nn; integrator_pairing...)
 
 loss = GeometricMachineLearning.ReducedLoss(encoder(sae_nn), decoder(sae_nn))
 
@@ -62,8 +62,8 @@ dl_integration = DataLoader(dl; autoencoder = false)
 # the regular transformer can't deal with symplectic data!
 dl_integration = DataLoader(vcat(dl_integration.input.q, dl_integration.input.p))
 integrator_batch = Batch(integrator_batch_size, seq_length)
-train_integrator_loss = o_integrator(
-    integrator_nn, dl_integration, integrator_batch, integrator_train_epochs, loss)
+train_integrator_loss = train!(integrator_nn, o_integrator, dl_integration,
+    integrator_batch, integrator_train_epochs; loss = loss)
 
 const mtc = GeometricMachineLearning.map_to_cpu
 JLD2.save(
diff --git a/scripts/reproduction/symplectic_autoencoders/training.jl b/scripts/reproduction/symplectic_autoencoders/training.jl
index 389369520..1381f5da9 100644
--- a/scripts/reproduction/symplectic_autoencoders/training.jl
+++ b/scripts/reproduction/symplectic_autoencoders/training.jl
@@ -28,12 +28,11 @@ n_epochs = smoke_size(100, 2)
 batch_size = smoke_size(512, 16)
 n_range = 2:1:smoke_size(15, 3)
 μ_range = (T(0.51), T(0.625), T(0.55), T(0.47))
-# The learning rate is the `Optimizer`'s `step_size`, not the method's: `Adam` lost the `η` field
-# it never applied to the direction, and `β₁`, `β₂` and `δ` became keywords so that the old
-# positional call fails instead of silently reading `η` as `β₁`.
-opt = AdamOptimizer(T; β₁ = T(0.9), β₂ = T(0.99), δ = T(1.0e-8))
+# The learning rate is the `NetworkOptimizer`'s `linesearch`, not the method's: `Adam` has no `η`
+# field, and `β₁`, `β₂` and `δ` are keywords.
+opt = Adam(; β₁ = 0.9, β₂ = 0.99, δ = 1.0e-8)
 const step_size = T(0.001)
-# The retraction belongs to the `Optimizer` now, not to the layer: `PSDLayer(M, N)` takes no
+# The retraction belongs to the `NetworkOptimizer`, not to the layer: `PSDLayer(M, N)` takes no
 # keyword.
 retraction = Cayley()
 
@@ -89,19 +88,17 @@ end
 # architecture rather than a bare `Chain` is what makes `encoder(nn)` and `decoder(nn)` available,
 # and those are what `HRedSys` needs.
 #
-# Training is `DataLoader` + `Batch` + `Optimizer`. What stood here was a hand-rolled loop over
-# `redraw_batch!(dl)`, counting its own iterations from `dl.batch_size`; a `DataLoader` carries
-# neither, because the batch is a `Batch` and the loop is the `Optimizer` functor. The loop also
-# wrapped both the pullback and the optimization step in `try … catch; continue`, so a training run
-# in which every single step failed was indistinguishable from one that worked.
+# Training is `DataLoader` + `Batch` + `NetworkOptimizer`, and the loop is `train!`: the batch is a
+# `Batch`, and a failed step raises its error.
 function get_nn_encoder_decoder(; n = 5, n_epochs = 500, opt = opt, T = T)
     sae_arch = SymplecticAutoencoder(
         2 * N, 2 * n; n_encoder_blocks = 4, n_decoder_blocks = 4,
         n_encoder_layers = 2, n_decoder_layers = 2)
     sae_nn = NeuralNetwork(sae_arch, backend, T)
 
-    optimizer_instance = Optimizer(opt, sae_nn; step_size = step_size, retraction = retraction)
-    optimizer_instance(sae_nn, dl, Batch(batch_size), n_epochs)
+    optimizer_instance = NetworkOptimizer(
+        sae_nn; algorithm = opt, linesearch = step_size, retraction = retraction)
+    train!(sae_nn, optimizer_instance, dl, Batch(batch_size), n_epochs)
 
     sae_nn
 end
diff --git a/scripts/reproduction/symplectic_transformer/double_pendulum.jl b/scripts/reproduction/symplectic_transformer/double_pendulum.jl
index 021e39a07..8bb463d40 100644
--- a/scripts/reproduction/symplectic_transformer/double_pendulum.jl
+++ b/scripts/reproduction/symplectic_transformer/double_pendulum.jl
@@ -70,19 +70,19 @@ nn4 = NeuralNetwork(arch4)
 const batch_size = 1024
 const n_epochs = smoke_size(10000, 2)
 
-o_method = AdamOptimizer()
+o_method = Adam()
 
-o1 = Optimizer(o_method, nn1)
-o2 = Optimizer(o_method, nn2)
-o3 = Optimizer(o_method, nn3)
-o4 = Optimizer(o_method, nn4)
+o1 = NetworkOptimizer(nn1; algorithm = o_method)
+o2 = NetworkOptimizer(nn2; algorithm = o_method)
+o3 = NetworkOptimizer(nn3; algorithm = o_method)
+o4 = NetworkOptimizer(nn4; algorithm = o_method)
 
 batch = Batch(batch_size, seq_length)
 
-loss_array1 = o1(nn1, dl, batch, n_epochs)
-loss_array2 = o2(nn2, dl, batch, n_epochs)
-loss_array3 = o3(nn3, dl, batch, n_epochs)
-loss_array4 = o4(nn4, dl, Batch(batch_size), n_epochs)
+loss_array1 = train!(nn1, o1, dl, batch, n_epochs)
+loss_array2 = train!(nn2, o2, dl, batch, n_epochs)
+loss_array3 = train!(nn3, o3, dl, batch, n_epochs)
+loss_array4 = train!(nn4, o4, dl, Batch(batch_size), n_epochs)
 
 function make_training_error_plot(; theme = :dark, symplectic = true)
     textcolor = theme == :dark ? :white : :black
diff --git a/scripts/reproduction/symplectic_transformer/double_pendulum_short_integration.jl b/scripts/reproduction/symplectic_transformer/double_pendulum_short_integration.jl
index 324cee989..747757aa2 100644
--- a/scripts/reproduction/symplectic_transformer/double_pendulum_short_integration.jl
+++ b/scripts/reproduction/symplectic_transformer/double_pendulum_short_integration.jl
@@ -68,19 +68,19 @@ nn4 = NeuralNetwork(arch4)
 const batch_size = 1024
 const n_epochs = smoke_size(3000, 2)
 
-o_method = AdamOptimizer()
+o_method = Adam()
 
-o1 = Optimizer(o_method, nn1)
-o2 = Optimizer(o_method, nn2)
-o3 = Optimizer(o_method, nn3)
-o4 = Optimizer(o_method, nn4)
+o1 = NetworkOptimizer(nn1; algorithm = o_method)
+o2 = NetworkOptimizer(nn2; algorithm = o_method)
+o3 = NetworkOptimizer(nn3; algorithm = o_method)
+o4 = NetworkOptimizer(nn4; algorithm = o_method)
 
 batch = Batch(batch_size, seq_length)
 
-loss_array1 = o1(nn1, dl, batch, n_epochs)
-loss_array2 = o2(nn2, dl, batch, n_epochs)
-loss_array3 = o3(nn3, dl, batch, n_epochs)
-loss_array4 = o4(nn4, dl, Batch(batch_size), n_epochs)
+loss_array1 = train!(nn1, o1, dl, batch, n_epochs)
+loss_array2 = train!(nn2, o2, dl, batch, n_epochs)
+loss_array3 = train!(nn3, o3, dl, batch, n_epochs)
+loss_array4 = train!(nn4, o4, dl, Batch(batch_size), n_epochs)
 
 function make_training_error_plot(; theme = :dark, symplectic = true)
     textcolor = theme == :dark ? :white : :black
diff --git a/scripts/reproduction/sympnets/sympnet_pendulum.jl b/scripts/reproduction/sympnets/sympnet_pendulum.jl
index 5fe8fa0f7..5be17eff3 100644
--- a/scripts/reproduction/sympnets/sympnet_pendulum.jl
+++ b/scripts/reproduction/sympnets/sympnet_pendulum.jl
@@ -4,6 +4,8 @@ Simple implementation of a pendulum for SympNets.
 
 using GeometricIntegrators
 using GeometricMachineLearning
+# `GeometricOptimizers`' step, reached through this package: the scripts environment has no entry for it
+using GeometricMachineLearning: optimization_step!
 using LinearAlgebra
 using CairoMakie
 using Zygote
@@ -66,18 +68,16 @@ function full_loss(ps, q, p)
 end
 
 # define momentum optimizer and initialize
-method = AdamOptimizer()
+method = Adam()
 # initial gradients for calling Cache constructor
-opt = Optimizer(method, ps)
-# `optimization_step!` takes the global section of the parameters, not the model.
-λY = GlobalSection(ps)
+opt = NetworkOptimizer(ps; algorithm = method)
 
 # training 
 println("initial loss: ", full_loss(ps, q, p))
 training_steps = smoke_size(1000, 2)
 @showprogress for i in 1:training_steps
     dp = Zygote.gradient(ps -> loss(ps, q, p), ps)[1]
-    optimization_step!(opt, λY, ps, dp)
+    optimization_step!(ps, opt, dp)
 end
 println("final loss: ", full_loss(ps, q, p))
 
diff --git a/scripts/reproduction/sympnets/sympnet_toda_lattice.jl b/scripts/reproduction/sympnets/sympnet_toda_lattice.jl
index 6871819e1..0036536af 100644
--- a/scripts/reproduction/sympnets/sympnet_toda_lattice.jl
+++ b/scripts/reproduction/sympnets/sympnet_toda_lattice.jl
@@ -25,14 +25,14 @@ dl = DataLoader(integrate(ep, ImplicitMidpoint()); suppress_info = true)
 arch = GSympNet(dl; upscaling_dimension = 4)
 
 nn = NeuralNetwork(arch)
-o_method = AdamOptimizer()
+o_method = Adam()
 
-o = Optimizer(o_method, nn)
+o = NetworkOptimizer(nn; algorithm = o_method)
 
 batch = Batch(256)
 const n_epochs = smoke_size(1024, 2)
 
-loss_array = o(nn, dl, batch, n_epochs)
+loss_array = train!(nn, o, dl, batch, n_epochs)
 
 function make_training_error_plot(; theme = :dark)
     textcolor = theme == :dark ? :white : :black
diff --git a/scripts/reproduction/transformer_integrator/softmax_comparison.jl b/scripts/reproduction/transformer_integrator/softmax_comparison.jl
index 7c95a8320..7edfda482 100644
--- a/scripts/reproduction/transformer_integrator/softmax_comparison.jl
+++ b/scripts/reproduction/transformer_integrator/softmax_comparison.jl
@@ -49,15 +49,15 @@ arch2 = StandardTransformerIntegrator(dl.input_dim; transformer_dim = 20,
 nn1 = NeuralNetwork(arch1)
 nn2 = NeuralNetwork(arch2)
 
-o_method = AdamOptimizer()
+o_method = Adam()
 
-o1 = Optimizer(o_method, nn1)
-o2 = Optimizer(o_method, nn2)
+o1 = NetworkOptimizer(nn1; algorithm = o_method)
+o2 = NetworkOptimizer(nn2; algorithm = o_method)
 
 batch = Batch(batch_size, seq_length)
 
-loss_array1 = o1(nn1, dl, batch, n_epochs)
-loss_array2 = o2(nn2, dl, batch, n_epochs)
+loss_array1 = train!(nn1, o1, dl, batch, n_epochs)
+loss_array2 = train!(nn2, o2, dl, batch, n_epochs)
 
 const n_steps = 300
 
diff --git a/scripts/reproduction/transformer_integrator/symplectic_transformer.jl b/scripts/reproduction/transformer_integrator/symplectic_transformer.jl
index 4f2f5d16a..ae2fc08c8 100644
--- a/scripts/reproduction/transformer_integrator/symplectic_transformer.jl
+++ b/scripts/reproduction/transformer_integrator/symplectic_transformer.jl
@@ -55,17 +55,17 @@ nn3 = NeuralNetwork(arch3)
 const batch_size = 1024
 const n_epochs = smoke_size(3000, 2)
 
-o_method = AdamOptimizer()
+o_method = Adam()
 
-o1 = Optimizer(o_method, nn1)
-o2 = Optimizer(o_method, nn2)
-o3 = Optimizer(o_method, nn3)
+o1 = NetworkOptimizer(nn1; algorithm = o_method)
+o2 = NetworkOptimizer(nn2; algorithm = o_method)
+o3 = NetworkOptimizer(nn3; algorithm = o_method)
 
 batch = Batch(batch_size, seq_length)
 
-loss_array1 = o1(nn1, dl, batch, n_epochs)
-loss_array2 = o2(nn2, dl, batch, n_epochs)
-loss_array3 = o3(nn3, dl, batch, n_epochs)
+loss_array1 = train!(nn1, o1, dl, batch, n_epochs)
+loss_array2 = train!(nn2, o2, dl, batch, n_epochs)
+loss_array3 = train!(nn3, o3, dl, batch, n_epochs)
 
 function make_training_error_plot(; theme = :dark, symplectic = true)
     textcolor = theme == :dark ? :white : :black
diff --git a/scripts/reproduction/transformer_integrator/symplectic_transformer_vector_softmax.jl b/scripts/reproduction/transformer_integrator/symplectic_transformer_vector_softmax.jl
index d7fb30e10..261943def 100644
--- a/scripts/reproduction/transformer_integrator/symplectic_transformer_vector_softmax.jl
+++ b/scripts/reproduction/transformer_integrator/symplectic_transformer_vector_softmax.jl
@@ -55,17 +55,17 @@ nn3 = NeuralNetwork(arch3)
 const batch_size = 1024
 const n_epochs = smoke_size(3000, 2)
 
-o_method = AdamOptimizer()
+o_method = Adam()
 
-o1 = Optimizer(o_method, nn1)
-o2 = Optimizer(o_method, nn2)
-o3 = Optimizer(o_method, nn3)
+o1 = NetworkOptimizer(nn1; algorithm = o_method)
+o2 = NetworkOptimizer(nn2; algorithm = o_method)
+o3 = NetworkOptimizer(nn3; algorithm = o_method)
 
 batch = Batch(batch_size, seq_length)
 
-loss_array1 = o1(nn1, dl, batch, n_epochs)
-loss_array2 = o2(nn2, dl, batch, n_epochs)
-loss_array3 = o3(nn3, dl, batch, n_epochs)
+loss_array1 = train!(nn1, o1, dl, batch, n_epochs)
+loss_array2 = train!(nn2, o2, dl, batch, n_epochs)
+loss_array3 = train!(nn3, o3, dl, batch, n_epochs)
 
 function make_training_error_plot(; theme = :dark, symplectic = true)
     textcolor = theme == :dark ? :white : :black
diff --git a/scripts/reproduction/volume_preserving_feedforward/abc_flow.jl b/scripts/reproduction/volume_preserving_feedforward/abc_flow.jl
index a1810aff8..2f3e7ee91 100644
--- a/scripts/reproduction/volume_preserving_feedforward/abc_flow.jl
+++ b/scripts/reproduction/volume_preserving_feedforward/abc_flow.jl
@@ -25,7 +25,7 @@ const n_linear = 2
 const activation = tanh
 
 const batch_size = 16384
-const opt_method = AdamOptimizer()
+const opt_method = Adam()
 const n_epochs = smoke_size(1000, 2)
 
 # This was run on an Apple GPU: `using Metal`, `const backend = MetalBackend()`, `const T =
@@ -47,13 +47,13 @@ model = VolumePreservingFeedForward(sys_dim, n_blocks, n_linear, activation)
 
 nn = NeuralNetwork(model, backend, T)
 
-o = Optimizer(opt_method, nn)
+o = NetworkOptimizer(nn; algorithm = opt_method)
 
 # `Batch(batch_size, 1)` is a `Batch{:Transformer}` of sequence length one, and the network here is
 # a `NeuralNetworkIntegrator`, whose loss defaults only for a `Batch{:FeedForward}`.
 batch = Batch(batch_size)
 
-loss_array₁ = o(nn, dl, batch, n_epochs)
+loss_array₁ = train!(nn, o, dl, batch, n_epochs)
 
 ic = (q = [0.0, 0.0, 0.1],)
 
diff --git a/scripts/reproduction/volume_preserving_feedforward/double_pendulum.jl b/scripts/reproduction/volume_preserving_feedforward/double_pendulum.jl
index 099bc56eb..6526fe36d 100644
--- a/scripts/reproduction/volume_preserving_feedforward/double_pendulum.jl
+++ b/scripts/reproduction/volume_preserving_feedforward/double_pendulum.jl
@@ -42,7 +42,7 @@ const T = eltype(dl)
 # hyperparameters concerning training 
 const n_epochs = smoke_size(1000, 2)
 const batch_size = 1024
-const opt_method = AdamOptimizer(T)
+const opt_method = Adam()
 const resnet_activation = tanh
 
 const t_validation = 30
@@ -54,10 +54,11 @@ function setup_and_train(
         model::Union{GeometricMachineLearning.Architecture, GeometricMachineLearning.Chain},
         batch::Batch; transformer::Bool = true)
     nn₀ = NeuralNetwork(model, backend, T)
-    o₀ = Optimizer(opt_method, nn₀)
+    o₀ = NetworkOptimizer(nn₀; algorithm = opt_method)
 
-    loss_array = transformer ? o₀(nn₀, dl, batch, n_epochs, transformer_loss) :
-                 o₀(nn₀, dl, batch, n_epochs)
+    loss_array = transformer ?
+                 train!(nn₀, o₀, dl, batch, n_epochs; loss = transformer_loss) :
+                 train!(nn₀, o₀, dl, batch, n_epochs)
 
     nn₀, loss_array
 end
diff --git a/scripts/reproduction/volume_preserving_feedforward/rigid_body.jl b/scripts/reproduction/volume_preserving_feedforward/rigid_body.jl
index ae6763ffa..54a385cde 100644
--- a/scripts/reproduction/volume_preserving_feedforward/rigid_body.jl
+++ b/scripts/reproduction/volume_preserving_feedforward/rigid_body.jl
@@ -28,7 +28,7 @@ const n_linear = 2
 const activation = tanh
 
 const batch_size = 16384
-const opt_method = AdamOptimizer()
+const opt_method = Adam()
 const n_epochs = smoke_size(20000, 2)
 
 # The GPU is what this was run on; the smoke run drops to the CPU so that CI can execute it.
@@ -48,11 +48,11 @@ model = VolumePreservingFeedForward(sys_dim, n_blocks, n_linear, activation)
 
 nn = NeuralNetwork(model, backend, T)
 
-o = Optimizer(opt_method, nn)
+o = NetworkOptimizer(nn; algorithm = opt_method)
 
 batch = Batch(batch_size)
 
-loss_array₁ = o(nn, dl, batch, n_epochs)
+loss_array₁ = train!(nn, o, dl, batch, n_epochs)
 
 ic = (q = [sin(1.1), 0.0, cos(1.1)],)
 
diff --git a/scripts/reproduction/volume_preserving_transformer/rigid_body.jl b/scripts/reproduction/volume_preserving_transformer/rigid_body.jl
index 47708f121..30e317500 100644
--- a/scripts/reproduction/volume_preserving_transformer/rigid_body.jl
+++ b/scripts/reproduction/volume_preserving_transformer/rigid_body.jl
@@ -51,7 +51,7 @@ const dl = backend == CPU() ? DataLoader(T.(dl₁.input)) :
 # hyperparameters concerning training 
 const n_epochs = smoke_size(500000, 2)
 const batch_size = 16384
-const opt_pairing = AdamOptimizerWithDecay(n_epochs, T; η₁ = 1e-2, η₂ = 1e-6)
+const opt_pairing = AdamOptimizerWithDecay(n_epochs; η₁ = 1e-2, η₂ = 1e-6)
 
 # parameters for evaluation 
 ics_val = [sin(1.1), 0.0, cos(1.1)]
@@ -62,9 +62,9 @@ const t_validation_long = 100
 function train_the_network(nn₀::GeometricMachineLearning.NeuralNetwork, batch::Batch)
     Random.seed!(1234)
 
-    o₀ = Optimizer(nn₀; opt_pairing...)
+    o₀ = NetworkOptimizer(nn₀; opt_pairing...)
 
-    loss_array = o₀(nn₀, dl, batch, n_epochs)
+    loss_array = train!(nn₀, o₀, dl, batch, n_epochs)
 
     GeometricMachineLearning.map_to_cpu(nn₀), loss_array
 end
diff --git a/scripts/verification/all_zero_leaf.jl b/scripts/verification/all_zero_leaf.jl
new file mode 100644
index 000000000..d61c69f15
--- /dev/null
+++ b/scripts/verification/all_zero_leaf.jl
@@ -0,0 +1,78 @@
+# What a training step does to a layer the loss does not read.
+#
+# Under NeuralNetworkParameters 0.5 the gradient of such a layer is a zero leaf of the layer's own
+# type, not `nothing`, and `NetworkOptimizer` hands the whole gradient to
+# `GeometricOptimizers.optimization_step!`: GeometricMachineLearning skips no leaf. This script prints
+# how far each leaf moves, in both precisions and under each first-order method:
+#
+#   - a Euclidean and a Stiefel leaf the loss never reads stay `==` their start under
+#     `GradientMethod`, `MomentumMethod` and `Adam`; under `AdamWithEuclideanDecay` the Euclidean one
+#     decays and the Stiefel one does not;
+#   - a leaf the loss reads for the first 5 steps and not after keeps moving on its momentum under
+#     `MomentumMethod` and `Adam`;
+#   - a layer without numbers, `NamedTuple()`, steps without an error.
+#
+# Run from the repository root, with the test environment:
+#   julia --startup-file=no --project=. -e 'using TestEnv; TestEnv.activate(); include("scripts/verification/all_zero_leaf.jl")'
+
+using GeometricMachineLearning
+using GeometricMachineLearning: params
+using GeometricOptimizers: optimization_step!
+using Printf
+import Random
+import Zygote
+
+const N_STEPS = 50
+const N_READ = 5
+
+maxdiff(a, b) = maximum(abs, Matrix(a) - Matrix(b))
+
+function zero_leaf_run(T, method)
+    Random.seed!(1)
+    ps = NetworkParameters((read = (W = randn(T, 3, 3),),
+        never_euclidean = (W = randn(T, 3, 3),),
+        never_stiefel = (Y = rand(StiefelManifold{T}, 4, 2),),
+        read_then_not = (W = randn(T, 3, 3),),
+        empty = NamedTuple()))
+    p₀ = deepcopy(ps)
+    opt = NetworkOptimizer(ps; algorithm = method)
+    x = randn(T, 3)
+    for t in 1:N_STEPS
+        loss = t ≤ N_READ ?
+               (p -> sum(abs2, p.read.W * x) + sum(abs2, p.read_then_not.W * x)) :
+               (p -> sum(abs2, p.read.W * x))
+        dp = only(Zygote.gradient(loss, ps))
+        optimization_step!(ps, opt, dp)
+    end
+    (never_euclidean = maxdiff(ps.never_euclidean.W, p₀.never_euclidean.W),
+        never_stiefel = maxdiff(ps.never_stiefel.Y, p₀.never_stiefel.Y))
+end
+
+# The motion of the leaf that is read for `N_READ` steps, measured from step `N_READ` on.
+function after_reading(T, method)
+    Random.seed!(1)
+    ps = NetworkParameters((
+        read = (W = randn(T, 3, 3),), read_then_not = (W = randn(T, 3, 3),)))
+    opt = NetworkOptimizer(ps; algorithm = method)
+    x = randn(T, 3)
+    both = p -> sum(abs2, p.read.W * x) + sum(abs2, p.read_then_not.W * x)
+    one = p -> sum(abs2, p.read.W * x)
+    for _ in 1:N_READ
+        optimization_step!(ps, opt, only(Zygote.gradient(both, ps)))
+    end
+    W₅ = copy(ps.read_then_not.W)
+    for _ in (N_READ + 1):N_STEPS
+        optimization_step!(ps, opt, only(Zygote.gradient(one, ps)))
+    end
+    maxdiff(ps.read_then_not.W, W₅)
+end
+
+for T in (Float32, Float64)
+    for method in (GradientMethod(), MomentumMethod(), Adam(), AdamWithEuclideanDecay())
+        r = zero_leaf_run(T, method)
+        @printf("%-8s %-26s never read: Euclidean %.2e  Stiefel %.2e | read %d steps, then not: %.2e\n",
+            T, nameof(typeof(method)), r.never_euclidean,
+            r.never_stiefel, N_READ,
+            after_reading(T, method))
+    end
+end
diff --git a/src/GeometricMachineLearning.jl b/src/GeometricMachineLearning.jl
index bb474b69b..ecd719838 100644
--- a/src/GeometricMachineLearning.jl
+++ b/src/GeometricMachineLearning.jl
@@ -6,12 +6,11 @@ using AbstractNeuralNetworks
 # package also exports `flatten`/`unflatten` and the leaf protocol, none of which this package
 # extends — `GeometricOptimizers` carries the protocol for the structured matrices.
 #
-# Two things do come from there rather than being written again here: `mapstorage`, which reaches the
-# storage of a structured leaf and rebuilds the leaf around the result (`src/map_to_cpu.jl`), and
-# `parameter_eltype`, which promotes over the leaves of a set. A plain `NamedTuple` of layers is
-# walked with `Base.map`, which needs nothing from anybody.
+# One thing does come from there rather than being written again here: `mapstorage`, which reaches
+# the storage of a structured leaf and rebuilds the leaf around the result (`src/map_to_cpu.jl`). A
+# plain `NamedTuple` of layers is walked with `Base.map`, which needs nothing from anybody.
 import NeuralNetworkParameters: NetworkParameters
-using NeuralNetworkParameters: mapparameters, mapstorage, parameter_eltype
+using NeuralNetworkParameters: mapstorage
 using ChainRulesCore
 using GeometricBase
 using GeometricSolutions: GeometricSolution, EnsembleSolution, TimeSeries
@@ -42,7 +41,9 @@ import Symbolics
 import GeometricOptimizers
 import GeometricOptimizers: Manifold, StiefelManifold, GrassmannManifold
 import GeometricOptimizers: SkewSymMatrix, SymmetricMatrix,
-                            LowerTriangular, UpperTriangular, StiefelProjection
+                            StrictlyLowerTriangular as LowerTriangular,
+                            StrictlyUpperTriangular as UpperTriangular,
+                            StiefelProjection
 import GeometricOptimizers: StiefelLieAlgHorMatrix, GrassmannLieAlgHorMatrix
 import GeometricOptimizers: rgrad, metric, check, global_section
 # `orthonormal_columns(draw)` orthonormalizes `draw()` with CholeskyQR2, which runs on whatever
@@ -52,17 +53,16 @@ import GeometricOptimizers: orthonormal_columns
 import GeometricOptimizers: GlobalSection, global_rep, apply_section, apply_section!,
                             update_section!
 import GeometricOptimizers: Geodesic, Cayley, geodesic, cayley, retraction
-import GeometricOptimizers: OptimizerMethod,
-                            GradientMethod, MomentumMethod, Adam,
-                            GradientState, MomentumState, AdamState,
+import GeometricOptimizers: GradientMethod, MomentumMethod, Adam, AdamWithEuclideanDecay,
                             AdamOptimizerWithDecay, DecayingStatic
+# The training step. `TrainingOptimizer` and `optimization_step!` are not re-exported:
+# `NetworkOptimizer` wraps the one and `train!` calls the other.
+import GeometricOptimizers: TrainingOptimizer, optimization_step!
 import GeometricOptimizers: update!
 # `solve!` is imported rather than started afresh so that GML's `solve!(::NeuralNetwork{<:PSDArch},
 # …)` — solve for the parameters directly, by SVD, instead of training for them — is a method of the
 # same verb a caller already has from GeometricOptimizers, and not a second function of the name.
 import GeometricOptimizers: solve!
-# The optimizer *caches* stay internal upstream — they are `solver_step!` scratch — so GML reaches
-# them as `GeometricOptimizers.AdamCache` where it needs to name one, and does not re-export them.
 
 import AbstractNeuralNetworks: Architecture, AbstractExplicitLayer, NeuralNetwork,
                                FeedForwardLoss
@@ -194,29 +194,18 @@ export ResNet
 export Transformer
 export TransformerIntegrator, StandardTransformerIntegrator
 
-# INCLUDE OPTIMIZERS — the methods, states, sections and retractions come from GeometricOptimizers.
+# INCLUDE OPTIMIZERS — the methods, sections and retractions come from GeometricOptimizers.
 include("optimizers/optimizer.jl")
 
-export OptimizerMethod
-export GradientMethod, GradientState
-export MomentumMethod, MomentumState
-export Adam, AdamState
-export Optimizer
-export optimization_step!
+export NetworkOptimizer
 export GlobalSection, global_section, apply_section, apply_section!, update_section!
 export global_rep
 export Geodesic, Cayley
 export geodesic, cayley
 export retraction
 export update!
-# The optimizer caches are not exported, for every method alike: see the note at the
-# GeometricOptimizers imports above.
-# backward-compat aliases (old names → new names)
-const GradientOptimizer = GradientMethod
-const MomentumOptimizer = MomentumMethod
-const AdamOptimizer = Adam
-export GradientOptimizer, MomentumOptimizer, AdamOptimizer
 # Re-exported from GeometricOptimizers, which owns the one definition of them.
+export GradientMethod, MomentumMethod, Adam, AdamWithEuclideanDecay
 export AdamOptimizerWithDecay, DecayingStatic
 
 export NeuralNetwork
@@ -263,10 +252,10 @@ include("pullbacks/zygote_pullback.jl")
 include("pullbacks/symbolic_hnn_pullback.jl")
 
 export DataLoader
-export Batch, optimize_for_one_epoch!
+export Batch, train!
 include("data_loader/tensor_assign.jl")
 include("data_loader/batch.jl")
-include("data_loader/optimize.jl")
+include("data_loader/train.jl")
 
 include("reduced_system/reduced_system.jl")
 
diff --git a/src/arrays/poisson_tensor.jl b/src/arrays/poisson_tensor.jl
index 313276d65..b95da036b 100644
--- a/src/arrays/poisson_tensor.jl
+++ b/src/arrays/poisson_tensor.jl
@@ -67,6 +67,10 @@ end
 
 Base.size(𝕁::PoissonTensor) = size(𝕁.J)
 
+# The backend of the stored matrix. GeometricOptimizers' checked products ask every operand for one,
+# so `𝕁 * Y` with a `StiefelManifold` `Y` raises without it.
+KernelAbstractions.get_backend(𝕁::PoissonTensor) = KernelAbstractions.get_backend(𝕁.J)
+
 function PoissonTensor(backend::Backend, n2::Int, T::DataType)
     @assert iseven(n2)
     n = n2÷2
diff --git a/src/data_loader/optimize.jl b/src/data_loader/optimize.jl
deleted file mode 100644
index 349448b9a..000000000
--- a/src/data_loader/optimize.jl
+++ /dev/null
@@ -1,117 +0,0 @@
-@doc raw"""
-    optimize_for_one_epoch!(opt, model, ps, dl, batch, loss, λY)
-
-Sample the data contained in `dl` according to `batch` and optimize for these batches.
-
-This step also performs automatic differentiation on `loss`.
-
-The output of `optimize_for_one_epoch!` is the average loss over all batches of the epoch:
-```math
-output = \frac{1}{\mathtt{steps\_per\_epoch}}\sum_{t=1}^\mathtt{steps\_per\_epoch}\mathtt{loss}(\theta^{(t-1)}).
-```
-This is done because any *reverse differentiation* routine always has two outputs; for `Zygote`:
-```julia
-loss_value, pullback = Zygote.pullback(ps -> loss(model, ps, input, output), ps)
-```
-So we get the value for the loss for free whenever we compute the pullback with AD.
-
-# Arguments 
-
-All the arguments are mandatory (there are no defaults): 
-1. an instance of [`Optimizer`](@ref).
-2. the neural network model.
-3. the neural network parameters `ps`.
-4. the data (i.e. an instance of [`DataLoader`](@ref)).
-5. `batch`::[`Batch`](@ref): stores `batch_size` (and optionally `seq_length` and `prediction_window`).
-6. `loss::NetworkLoss`.
-7. the *section* `λY` of the parameters `ps`.
-
-# Implementation
-
-Internally this calls [`optimization_step!`](@ref) for each minibatch.
-
-The number of minibatches can be determined with [`number_of_batches`](@ref):
-
-```jldoctest
-using GeometricMachineLearning
-using GeometricMachineLearning: number_of_batches
-
-data = [1, 2, 3, 4, 5]
-batch = Batch(2)
-dl = DataLoader(data; suppress_info = true)
-
-number_of_batches(dl, batch)
-
-# output
-
-3
-```
-"""
-function optimize_for_one_epoch!(opt::Optimizer,
-        model, ps::NetworkParameters,
-        dl::DataLoader{T},
-        batch::Batch,
-        loss::NetworkLoss,
-        λY) where {T}
-    optimize_for_one_epoch!(opt, model, ps, dl, batch, ZygotePullback(loss), λY)
-end
-
-function optimize_for_one_epoch!(opt::Optimizer,
-        model,
-        ps::NetworkParameters,
-        dl::DataLoader{T},
-        batch::Batch,
-        _pullback::AbstractPullback,
-        λY) where {T}
-    count = 0
-    total_error = T(0)
-    batches = batch(dl)
-    for batch_indices in batches
-        count += 1
-        input_nt_output_nt = convert_input_and_batch_indices_to_array(dl, batch, batch_indices)
-        loss_value, pullback = _pullback(ps, model, input_nt_output_nt)
-        total_error += loss_value
-        dp = _processing(pullback(one(loss_value)))
-        optimization_step!(opt, λY, ps, dp)
-    end
-    total_error / count
-end
-
-function (o::Optimizer)(nn::NeuralNetwork,
-        dl::DataLoader,
-        batch::Batch,
-        n_epochs::Integer,
-        loss::NetworkLoss,
-        _pullback::AbstractPullback = ZygotePullback(loss); show_progress = true)
-    Λ = GlobalSection(params(nn))
-    progress_object = show_progress ?
-                      ProgressMeter.Progress(n_epochs; enabled = true) : nothing
-    loss_array = zeros(float(eltype(dl)), n_epochs)
-    for i in 1:n_epochs
-        loss_array[i] = optimize_for_one_epoch!(
-            o, nn.model, params(nn), dl, batch, _pullback, Λ)
-        show_progress ?
-        ProgressMeter.next!(progress_object; showvalues = [(
-            :TrainingLoss, loss_array[i])]) : nothing
-    end
-
-    loss_array
-end
-
-function (o::Optimizer)(nn::NeuralNetwork{<:TransformerIntegrator}, dl::DataLoader,
-        batch::Batch{:Transformer}, n_epochs::Int = 1; kwargs...)
-    loss = TransformerLoss(batch)
-    o(nn, dl, batch, n_epochs, loss; kwargs...)
-end
-
-function (o::Optimizer)(nn::NeuralNetwork{<:NeuralNetworkIntegrator}, dl::DataLoader,
-        batch::Batch{:FeedForward}, n_epochs::Int = 1; kwargs...)
-    loss = FeedForwardLoss()
-    o(nn, dl, batch, n_epochs, loss; kwargs...)
-end
-
-function (o::Optimizer)(nn::NeuralNetwork{<:AutoEncoder}, dl::DataLoader,
-        batch::Batch{:FeedForward}, n_epochs::Int = 1; kwargs...)
-    loss = AutoEncoderLoss()
-    o(nn, dl, batch, n_epochs, loss; kwargs...)
-end
diff --git a/src/data_loader/train.jl b/src/data_loader/train.jl
new file mode 100644
index 000000000..093745497
--- /dev/null
+++ b/src/data_loader/train.jl
@@ -0,0 +1,56 @@
+"""
+    train!(nn, opt, dl, batch, n_epochs; loss, show_progress = true)
+
+Train the neural network `nn` with the [`NetworkOptimizer`](@ref) `opt` on the data of the
+[`DataLoader`](@ref) `dl` for `n_epochs` epochs, and return the loss history.
+
+Each epoch draws the minibatches of `dl` with the [`Batch`](@ref) `batch` and takes one step per
+minibatch: Zygote computes the loss on the minibatch and its gradient, and
+`GeometricOptimizers.optimization_step!` writes the new parameters into `params(nn)`, with the
+whole gradient, a zero leaf included. Entry `i` of
+the history is the mean of the minibatch losses of epoch `i`, each evaluated before its step. The
+history has the element type of the loss value, so a `Float64` network on `Float32` data gives a
+`Vector{Float64}`.
+
+`loss` is a `NetworkLoss`. Without it, a [`TransformerIntegrator`](@ref) with a
+`Batch{:Transformer}` trains on [`TransformerLoss`](@ref)`(batch)`, a
+[`NeuralNetworkIntegrator`](@ref) with a `Batch{:FeedForward}` on `FeedForwardLoss()`, and an
+autoencoder with a `Batch{:FeedForward}` on [`AutoEncoderLoss`](@ref)`()`; any other network needs
+`loss`.
+
+`opt` must be built for `params(nn)`. It keeps the iteration number, the moments and the global
+sections, so a second call with the same `opt` resumes the training where the first one stopped;
+the only random numbers `train!` draws are those of the minibatch shuffle. `n_epochs < 1` is an
+`ArgumentError`, raised before any step.
+"""
+function train!(nn::NeuralNetwork, opt::NetworkOptimizer, dl::DataLoader, batch::Batch,
+        n_epochs::Integer; loss::NetworkLoss = _default_loss(nn.architecture, batch),
+        show_progress::Bool = true)
+    n_epochs ≥ 1 ||
+        throw(ArgumentError("`train!` needs at least one epoch; `n_epochs` is $(n_epochs)."))
+    pullback = ZygotePullback(loss)
+    progress = ProgressMeter.Progress(n_epochs; enabled = show_progress)
+    map(1:n_epochs) do _
+        value = _train_epoch!(nn, opt, dl, batch, pullback)
+        ProgressMeter.next!(progress; showvalues = [(:TrainingLoss, value)])
+        value
+    end
+end
+
+# The sum takes the type of the first minibatch's loss, so nothing is converted.
+function _train_epoch!(nn, opt, dl, batch, pullback)
+    batches = batch(dl)
+    sum(_train_step!(nn, opt, dl, batch, pullback, indices) for indices in batches) /
+    length(batches)
+end
+
+function _train_step!(nn, opt, dl, batch, pullback, indices)
+    value, back = pullback(params(nn), nn.model,
+        convert_input_and_batch_indices_to_array(dl, batch, indices))
+    optimization_step!(params(nn), opt, _get_contents(back(one(value))))
+    value
+end
+
+_default_loss(::TransformerIntegrator, batch::Batch{:Transformer}) = TransformerLoss(batch)
+_default_loss(::NeuralNetworkIntegrator, ::Batch{:FeedForward}) = FeedForwardLoss()
+_default_loss(arch::AutoEncoder, ::Batch{:FeedForward}) = NetworkLoss(arch)
diff --git a/src/layers/resnet.jl b/src/layers/resnet.jl
index c721c1e55..6a6b109b7 100644
--- a/src/layers/resnet.jl
+++ b/src/layers/resnet.jl
@@ -60,18 +60,6 @@ function (d::ResNetLayer{M, M, true})(x::AbstractArray{T, 3}, ps::NamedTuple) wh
     x + d.activation.(mat_tensor_mul(ps.weight, x) .+ ps.bias)
 end
 
-function (d::Dense{M, N, true})(x::AbstractArray{T, 3}, ps::NamedTuple) where {M, N, T}
-    d.σ.(mat_tensor_mul(ps.W, x) .+ ps.b)
-end
-
-function (d::Dense{M, N, false})(x::AbstractArray{T, 3}, ps::NamedTuple) where {M, N, T}
-    d.σ.(mat_tensor_mul(ps.W, x))
-end
-
-function (d::Linear{M, N})(x::AbstractArray{T, 3}, ps::NamedTuple) where {M, N, T}
-    mat_tensor_mul(ps.W, x)
-end
-
 function (d::Union{Dense{M, N}, ResNetLayer{M, N}})(z::QPT, ps::NamedTuple) where {M, N}
     @assert iseven(M) == iseven(N) == true
     @assert size(z.q, 1) * 2 == M
diff --git a/src/optimizers/optimizer.jl b/src/optimizers/optimizer.jl
index 69d81f32a..41fa035f0 100644
--- a/src/optimizers/optimizer.jl
+++ b/src/optimizers/optimizer.jl
@@ -1,423 +1,35 @@
-# Optimizer machinery on top of GeometricOptimizers.
-# Kept out of `utils.jl` because it dispatches on `Manifold`, which is defined later.
-
-# Gradient wrapper: stores a pre-computed Euclidean gradient and applies rgrad on manifolds.
-mutable struct _GMLGradient{T, VT} <: GeometricOptimizers.Gradient{T}
-    dp::VT
-end
-
-_gml_rgrad(x::Manifold, dp) = rgrad(x, dp)
-_gml_rgrad(x, dp) = dp
-# `mapparameters` and not `map`: it recurses on the branches, so `_gml_rgrad` is only ever called on
-# leaves, and it rebuilds in the shape of its *first* argument -- a container, which is what
-# `GeometricOptimizers._copyto!(gradient_array(cache), ·)` has a method for. It also normalises its
-# trailing arguments, so `dp` may stay the plain `NamedTuple` the pullback produced.
-_gml_rgrad(x::NetworkParameters, dp) = mapparameters(_gml_rgrad, x, dp)
-
-(g::_GMLGradient{T})(x::NetworkParameters{T}) where {T} = _gml_rgrad(x, g.dp)
-(g::_GMLGradient{T})(x::AbstractArray{T}) where {T} = g.dp
-
-# `Manifold` and not `AbstractArray` alone, because the two are otherwise **ambiguous** and not merely
-# overlapping. `_GMLGradient{T}` is a `GeometricOptimizers.Gradient{T}`, `Manifold{T}` is an
-# `AbstractMatrix{T}`, and `GeometricOptimizers` has `(::Gradient{T})(::Manifold{T})`; so on a bare
-# manifold neither `(::_GMLGradient{T})(::AbstractArray{T})` above nor that one is more specific, and
-# the call is a run-time `MethodError: ... is ambiguous`. This method is more specific than both.
-#
-# The answer it gives is this package's: the gradient is already computed, so the projection is
-# `rgrad(x, dp)` and not upstream's `rgrad(x, reshape(grad(vec(x)), ...))`, which would evaluate an
-# inner gradient this functor does not have. `_gml_rgrad` is where that lives.
-#
-# `_make_optimizer_cache` below asks `_is_layer` first and so keeps a bare `Manifold` from arriving
-# here at all. That ordering is a correctness fix in its own right -- one cache per layer rather than
-# per weight -- but it must not be the *only* thing standing between a caller and an ambiguity.
-(g::_GMLGradient{T})(x::Manifold{T}) where {T} = _gml_rgrad(x, g.dp)
-
-# A bare `AbstractVector`, for the same reason as the `Manifold` method above: `SimpleSolvers` has
-# `(::Gradient{T})(::AbstractVector{T})`, `_GMLGradient{T} <: GeometricOptimizers.Gradient{T}`, and
-# without a method of its own `(::_GMLGradient{T})(::AbstractArray{T})` above and that one are
-# equally specific on a `Vector` -- the call is a run-time `MethodError: ... is ambiguous`. This
-# method is more specific than both: `_GMLGradient{T} <: Gradient{T}` on the first argument and
-# `AbstractVector{T} <: AbstractArray{T}` on the second. The answer is unchanged from the
-# `AbstractArray{T}` method above -- the gradient is already Euclidean, so it is `g.dp`.
-(g::_GMLGradient{T})(x::AbstractVector{T}) where {T} = g.dp
-
-# State for Euclidean (non-manifold) parameters.
-mutable struct GMLEuclideanState{T, AT <: AbstractArray{T}}
-    iterations::Int
-    m₁::AT
-    m₂::AT
-end
-function GMLEuclideanState(x::AbstractArray{T}) where {T}
-    GMLEuclideanState{T, typeof(x)}(0, zero(x), zero(x))
-end
-
-# Adam with a decaying learning rate is GeometricOptimizers', and split the way it belongs: the
-# direction is an `Adam` method, the schedule is a `DecayingStatic` line search. Both names are
-# imported, and `Optimizer` below takes a `DecayingStatic` as its `step_size`.
-
-_is_go_native_method(::GeometricOptimizers.GradientMethod) = true
-_is_go_native_method(::GeometricOptimizers.MomentumMethod) = true
-_is_go_native_method(::GeometricOptimizers.Adam) = true
-_is_go_native_method(::GeometricOptimizers.OptimizerMethod) = false
-
-function _adapt_method_to_T(method::GeometricOptimizers.Adam, ::Type{T}) where {T}
-    GeometricOptimizers.Adam(T; β₁ = T(method.β₁), β₂ = T(method.β₂), δ = T(method.δ))
-end
-function _adapt_method_to_T(method::GeometricOptimizers.MomentumMethod, ::Type{T}) where {T}
-    GeometricOptimizers.MomentumMethod(T(method.α))
-end
-_adapt_method_to_T(method, ::Type) = method
-
-# Whether one `GeometricOptimizers` cache covers `x` whole.
-#
-# A **layer** is a flat `NamedTuple` of arrays, and it is exactly what one cache is for. A `NamedTuple`
-# whose values are branches is a *subtree* and gets one cache per layer instead, which is why the test
-# is flatness and not merely "a `NamedTuple`".
-#
-# The rule lives here rather than upstream because a whole set of parameters reaches
-# `GeometricOptimizers` only as a `NetworkParameters`, and a container says nothing about how deep the
-# tree beneath it is. Splitting a network into layers is this package's decision, so this package makes
-# it; [`_as_go_solution`](@ref) then does the wrap.
-#
-# A layer whose weights do not share an element type is still one cache, with `T` the promotion over
-# them. `_adapt_method_to_T` reads that promotion, so the cache is built for the element type the
-# layer actually has rather than for one its weights are required to agree on.
-# `!isempty` and not `all` alone: `all` over an empty collection is vacuously true, so without it an
-# empty set would be one cache's worth -- and `OptimizerCache` would then be asked for the element
-# type of a set that has no leaves to promote. An empty set is zero caches' worth, so it descends and
-# yields an empty tree of caches, which is what the step below iterates over zero times.
-_is_layer(x::NamedTuple) = _all_leaves(values(x))
-_is_layer(x::NetworkParameters) = _all_leaves(values(x))
-_is_layer(_) = false
-
-_all_leaves(vs) = !isempty(vs) && all(v -> v isa AbstractArray, vs)
-
-function _use_go_cache(method, x)
-    _is_go_native_method(method) &&
-        (x isa GeometricOptimizers.OptimizerSolution || _is_layer(x))
-end
-
-"""
-    _as_go_solution(x)
-
-`x` in the shape `GeometricOptimizers` takes a solution in.
-
-A layer given as a bare `NamedTuple` is wrapped; anything already in the right shape — a container, a
-bare `Manifold`, an `AbstractArray` — is passed through. The wrap **shares the leaf arrays**, so an
-in-place optimizer step writes through to the network's own weights and nothing has to be copied back.
 """
-_as_go_solution(x::NetworkParameters) = x
-_as_go_solution(x) = _is_layer(x) ? NetworkParameters(x) : x
+    NetworkOptimizer(nn; algorithm = Adam(), linesearch = default_step_size(algorithm), retraction = Cayley())
+    NetworkOptimizer(ps; algorithm, linesearch, retraction)
 
-# A `NetworkParameters` is always a tree of layers to descend into, never a single
-# `GeometricOptimizers` leaf, so its branch comes *first* — ahead of `_use_go_cache`.
-#
-# That ordering is **load-bearing**. `NetworkParameters` *is* one of the types
-# `GeometricOptimizers.OptimizerSolution` unions, so `_use_go_cache` is true at the root: without the
-# test above, a whole network would get one cache instead of one per layer -- silently, and with
-# `_leaf_optim_step!` handed the entire tree. Asking the structural question before the capability
-# question is what makes the shape of the cache depend on the shape of the parameters rather than on
-# which types upstream happens to accept.
-#
-# One cache for the whole network is the better end state, and it is **not** what this does.
-# `_GMLGradient` does take a `NetworkParameters`, but that is one wrapped *layer* and not the tree,
-# because the container branch above splits the network first. Getting to one cache means dropping
-# `_tree_optim_step!` and handing the whole tree over, which is a change of behaviour -- one
-# `GlobalSection` tree and one `Q` across every layer -- so it wants its own release. Nothing here
-# depends on it.
-#
-# **"Is this one cache's worth?" is asked before "is this a tree?"**, and a *flat* set answers yes to
-# both. `_is_layer` is that question: a set whose values are all leaves is a layer, whether it arrives
-# wrapped or as a branch, and one `GeometricOptimizers` cache is exactly what a layer is for. Asking
-# the tree question first would descend into such a set and give every individual weight its own cache
-# -- which for a manifold weight is not merely wasteful but wrong, since a bare `Manifold` then reaches
-# the gradient functor where a whole layer should have.
-#
-# The tree branch comes second and covers both carriers, because a network is a tree of layers whether
-# it is wrapped or not.
-function _make_optimizer_cache(method, x)
-    if _is_go_native_method(method) && _is_layer(x)
-        GeometricOptimizers.OptimizerCache(_adapt_method_to_T(method, parameter_eltype(x)),
-            _as_go_solution(x))
-    elseif x isa NetworkParameters || x isa NamedTuple
-        NamedTuple{keys(x)}(Tuple(_make_optimizer_cache(method, x[k]) for k in keys(x)))
-    elseif _use_go_cache(method, x)
-        GeometricOptimizers.OptimizerCache(_adapt_method_to_T(method, parameter_eltype(x)), x)
-    else
-        GMLEuclideanState(x)
-    end
-end
+The optimizer that [`train!`](@ref) trains the neural network `nn`, or the parameters `ps`, with.
 
-function _make_optimizer_state(method, x)
-    if _is_go_native_method(method) && _is_layer(x)
-        GeometricOptimizers.OptimizerState(_adapt_method_to_T(method, parameter_eltype(x)),
-            _as_go_solution(x))
-    elseif x isa NetworkParameters || x isa NamedTuple
-        NamedTuple{keys(x)}(Tuple(_make_optimizer_state(method, x[k]) for k in keys(x)))
-    elseif _use_go_cache(method, x)
-        GeometricOptimizers.OptimizerState(_adapt_method_to_T(method, parameter_eltype(x)), x)
-    else
-        GMLEuclideanState(x)
-    end
-end
-
-"""
-    Optimizer(method, nn; retraction, step_size)
-    Optimizer(nn; algorithm, linesearch, retraction)
-
-Optimizer state combining a `GeometricOptimizers` method with the parameters of a neural network.
-
-`step_size` is either a number — a fixed learning rate — or a
-`GeometricOptimizers.DecayingStatic`, a learning rate that decays geometrically with the iteration
-number. The second form is the one `GeometricOptimizers` uses itself, so a method paired with a
-schedule splats straight in:
+It holds one `GeometricOptimizers.TrainingOptimizer` over the whole parameter set, and the keywords
+are that constructor's: `algorithm` is a first-order method such as `GradientMethod()`,
+`MomentumMethod()` or `Adam()`, `linesearch` is a fixed step size or a `DecayingStatic` schedule,
+and `retraction` is `Cayley()` or `Geodesic()`. `AdamOptimizerWithDecay(n)` gives `algorithm` and
+`linesearch` together, and its `n` counts optimization steps, one per minibatch:
 
 ```julia
-opt = Optimizer(nn; AdamOptimizerWithDecay(n_epochs, Float32)...)
+opt = NetworkOptimizer(nn; AdamOptimizerWithDecay(n_steps)...)
 ```
 
-# Extended help
+The optimizer carries the iteration number, the moments and the global sections from one call of
+[`train!`](@ref) to the next, so a second call resumes the training. It is built for the parameters
+of one network, and it is not checked when it is used with another.
 
-The step size is a property of the optimizer and not of the method: the same `Adam()` trains at any
-learning rate. That is the split `GeometricOptimizers` makes — the method supplies a direction, a
-`SimpleSolvers.LinesearchMethod` supplies how far to go along it — and `step_size` is GML's half of
-it for a training loop, which has no objective function for a real line search to evaluate.
+A loop of one's own takes a step with `GeometricOptimizers.optimization_step!(params(nn), opt, dp)`,
+where `dp` is the gradient of the loss as a `NetworkParameters` of the shape of `params(nn)`.
 """
-mutable struct Optimizer{MT <: GeometricOptimizers.OptimizerMethod, CT, ST, RT, SST}
-    method::MT
-    cache::CT
-    state::ST
-    retraction::RT
-    step_size::SST
-    iterations::Int
+struct NetworkOptimizer{OT <: TrainingOptimizer}
+    optimizer::OT
 end
 
-_default_step_size(::GeometricOptimizers.Adam) = 1e-3
-_default_step_size(::GeometricOptimizers.OptimizerMethod) = 1e-2
-
-_step_size(η::Real, ::Int) = Float64(η)
-# `t` and not `t - 1`: `optimization_step!` increments before it asks, so the first step of a solve
-# is `α(1) = γη₁`. That is what `DecayingStatic` means by iteration `t`: `solve!` calls
-# `increase_iteration_number!` before `solver_step!`.
-_step_size(ls::DecayingStatic, t::Int) = Float64(GeometricOptimizers.step_size(ls, t))
-
-_current_step_size(opt::Optimizer, t::Int) = _step_size(opt.step_size, t)
-
-_optimizer_step_size(η::Real) = Float64(η)
-# Anything that is not a plain number goes through the same funnel as the `linesearch` keyword below,
-# so that the two entry points accept the same things: `step_size = Static(α)` is a fixed learning
-# rate on both, and anything else reports the `ArgumentError` that explains why a real line search
-# has nothing to search along here, instead of a `MethodError` naming this helper.
-_optimizer_step_size(ls) = _step_size_from_linesearch(ls)
-
-function Optimizer(method::GeometricOptimizers.OptimizerMethod, nn::NeuralNetwork;
-        retraction = GeometricOptimizers.cayley,
-        step_size = _default_step_size(method))
-    Optimizer(method, params(nn); retraction = retraction, step_size = step_size)
-end
-
-function Optimizer(method::GeometricOptimizers.OptimizerMethod,
-        ps::NetworkParameters;
-        retraction = GeometricOptimizers.cayley,
-        step_size = _default_step_size(method))
-    Optimizer(method, _make_optimizer_cache(method, ps), _make_optimizer_state(method, ps),
-        retraction, _optimizer_step_size(step_size), 0)
-end
-
-# The keyword form, so that the `(algorithm, linesearch)` pairing `GeometricOptimizers` returns from
-# `AdamOptimizerWithDecay` splats in unchanged. `linesearch` is the step size under the name
-# upstream gives it; a `Static` carries its own `α`, which is then the fixed learning rate.
-function Optimizer(nn_or_ps::Union{NeuralNetwork, NetworkParameters};
-        algorithm::GeometricOptimizers.OptimizerMethod,
-        linesearch = nothing,
-        retraction = GeometricOptimizers.cayley,
-        step_size = linesearch === nothing ? _default_step_size(algorithm) :
-                    _step_size_from_linesearch(linesearch))
-    Optimizer(algorithm, nn_or_ps; retraction = retraction, step_size = step_size)
-end
-
-_step_size_from_linesearch(ls::DecayingStatic) = ls
-_step_size_from_linesearch(ls::GeometricOptimizers.Static) = Float64(ls.α)
-function _step_size_from_linesearch(ls)
-    throw(ArgumentError(
-        "`Optimizer` takes a fixed step size or a `DecayingStatic` schedule, not a $(typeof(ls)). " *
-        "A training loop evaluates its loss on one batch at a time and has no objective for a line " *
-        "search to search along; use `GeometricOptimizers.Optimizer` with an `OptimizerProblem` for " *
-        "that."))
-end
-
-# Euclidean update rules
-function _euclidean_update!(x::AbstractArray{T}, dx::AbstractArray,
-        state::GMLEuclideanState, ::GeometricOptimizers.GradientMethod, step_size) where {T}
-    x .-= T(step_size) .* dx
-end
-function _euclidean_update!(x::AbstractArray{T}, dx::AbstractArray,
-        state::GMLEuclideanState{T}, method::GeometricOptimizers.MomentumMethod, step_size) where {T}
-    # `p ← αp + ∇L`, the classic momentum recursion. The decay belongs on `p` and not on `∇L`:
-    # `p ← p + α∇L` is an undamped accumulator that grows without bound for a constant gradient
-    # instead of saturating at `∇L/(1 - α)`. Same recursion as GO's `update!(::MomentumState, ...)`.
-    state.m₁ .= T(method.α) .* state.m₁ .+ dx
-    x .-= T(step_size) .* state.m₁
-end
-function _euclidean_update!(x::AbstractArray{T}, dx::AbstractArray,
-        state::GMLEuclideanState{T}, method::GeometricOptimizers.Adam, step_size) where {T}
-    t = state.iterations
-    _t = t + 1
-    β₁, β₂, δ = T(method.β₁), T(method.β₂), T(method.δ)
-    # the first factor is `(β - β^t)/(1 - β^t)`, not `β/(1 - β^t)`: the latter is ~100x too large
-    # at t = 2 and compounds every step, which inflates the second moment until the update vanishes
-    fac₁₁ = (β₁-β₁^_t)/(1-β₁^_t)
-    fac₁₂ = (1-β₁)/(1-β₁^_t)
-    fac₂₁ = (β₂-β₂^_t)/(1-β₂^_t)
-    fac₂₂ = (1-β₂)/(1-β₂^_t)
-    state.m₁ .= fac₁₁ .* state.m₁ .+ fac₁₂ .* dx
-    state.m₂ .= fac₂₁ .* state.m₂ .+ fac₂₂ .* dx .^ 2
-    x .-= T(step_size) .* state.m₁ ./ (sqrt.(state.m₂) .+ δ)
-end
-# `_euclidean_update!` has three methods and not four: Adam with a decaying learning rate *is*
-# Adam — only `step_size` differs, and that comes in as an argument — so the `Adam` method above
-# serves it too.
-
-function _go_update_leaf!(cache, state, local_grad,
-        method::GeometricOptimizers.Adam, ps_leaf)
-    GeometricOptimizers.update!(cache, state, local_grad, method, ps_leaf)
-end
-
-function _go_update_leaf!(cache, state, local_grad,
-        method::GeometricOptimizers.MomentumMethod, ps_leaf)
-    GeometricOptimizers.update!(cache, state, local_grad, method, ps_leaf)
+function NetworkOptimizer(ps::NetworkParameters; kwargs...)
+    NetworkOptimizer(TrainingOptimizer(ps; kwargs...))
 end
+NetworkOptimizer(nn::NeuralNetwork; kwargs...) = NetworkOptimizer(params(nn); kwargs...)
 
-function _go_update_leaf!(cache, state, local_grad,
-        method::GeometricOptimizers.OptimizerMethod, ps_leaf)
-    T = parameter_eltype(ps_leaf)
-    GeometricOptimizers.update!(cache, state, local_grad,
-        GeometricOptimizers.NoHessian{T}(), ps_leaf)
+function optimization_step!(ps, opt::NetworkOptimizer, dp)
+    optimization_step!(ps, opt.optimizer, dp)
 end
-
-# GO-managed leaf step (manifolds, vectors, whole layers)
-#
-# `ps` and not `ps_leaf` from here on: a layer is wrapped for the same reason its cache is, and the
-# wrap shares the arrays, so `_copyto!(ps, solution(cache))` below writes into the weights the caller
-# holds. A bare `Manifold` or `AbstractArray` passes through untouched.
-function _leaf_optim_step!(cache::GeometricOptimizers.OptimizerCache,
-        state::GeometricOptimizers.OptimizerState,
-        dp_leaf, ps_leaf, λY_leaf, method, retraction, step_size)
-    T = parameter_eltype(ps_leaf)
-    ps = _as_go_solution(ps_leaf)
-    local_grad = _GMLGradient{T, typeof(dp_leaf)}(dp_leaf)
-    adapted = _adapt_method_to_T(method, T)
-    state.iterations += 1
-    _go_update_leaf!(cache, state, local_grad, adapted, ps)
-    GeometricOptimizers._rmul!(GeometricOptimizers.direction(cache), T(step_size))
-    GeometricOptimizers.update_section!(GeometricOptimizers.section(cache),
-        GeometricOptimizers.section(state),
-        GeometricOptimizers.direction(cache),
-        retraction)
-    GeometricOptimizers._copyto!(GeometricOptimizers.solution(cache),
-        GeometricOptimizers.section(cache))
-    GeometricOptimizers._copyto!(ps, GeometricOptimizers.solution(cache))
-    GeometricOptimizers._copyto!(λY_leaf, GeometricOptimizers.section(cache))
-    # `section(cache)` is `update_section!(section(state), direction, retraction)`, so copying it is
-    # the same thing as retracting a second time -- and a retraction on a manifold is `O(N³)` where
-    # the copy is `O(N²)`.
-    GeometricOptimizers._copyto!(GeometricOptimizers.section(state),
-        GeometricOptimizers.section(cache))
-    if state isa GeometricOptimizers.AdamState
-        GeometricOptimizers._copyto!(GeometricOptimizers.first_moment(state),
-            GeometricOptimizers.first_moment(cache))
-        GeometricOptimizers._copyto!(GeometricOptimizers.second_moment(state),
-            GeometricOptimizers.second_moment(cache))
-    elseif state isa GeometricOptimizers.MomentumState
-        # `p ← αp + ∇L`; see the note in `_euclidean_update!` for the momentum method. This has to
-        # match what `update!(::MomentumCache, ...)` anticipated when it formed the direction.
-        GeometricOptimizers._rmul!(GeometricOptimizers.momentum(state), adapted.α)
-        GeometricOptimizers._add!(GeometricOptimizers.momentum(state),
-            GeometricOptimizers.gradient_array(cache))
-    end
-    nothing
-end
-
-# Euclidean leaf step (plain AbstractArray params)
-function _leaf_optim_step!(cache::GMLEuclideanState, state::GMLEuclideanState,
-        dp_leaf, ps_leaf, λY_leaf, method, retraction, step_size)
-    _euclidean_update!(ps_leaf, dp_leaf, state, method, step_size)
-    state.iterations += 1
-    nothing
-end
-
-# Whether `λY` is a tree of sections to descend into, or a single `GlobalSection` standing in for a
-# whole subtree. Named rather than written as an `isa` union: it is one question, asked once, and the
-# answer decides whether a layer gets its own section or the whole tree. A section tree is a plain
-# `NamedTuple` today, because `GeometricOptimizers`' `GlobalSection(::NetworkParameters)` unwraps the
-# container; the container arm is here so that a change there is a `MethodError` and not a silent
-# hand-off of the whole tree to every layer.
-_is_section_tree(::NamedTuple) = true
-_is_section_tree(::NetworkParameters) = true
-_is_section_tree(_) = false
-
-# Recursive dispatcher over the *cache* tree.
-#
-# Deliberately hand-written rather than `NeuralNetworkParameters.foreachparameters`, which walks a
-# parameter tree. Two reasons, and both are load-bearing:
-#
-#  - The recursion is keyed on `caches`, and stops where the cache stops. A cache sits at the *layer*
-#    level, so a layer's `NamedTuple` of weights arrives at `_leaf_optim_step!` whole, which is what
-#    one `GeometricOptimizers` cache is for. `foreachparameters` recurses on the leaf protocol and
-#    would descend past the layer into the individual weights, re-pairing every cache with the wrong
-#    object.
-#  - `λY` is broadcast, not zipped: a single `GlobalSection` may stand in for a whole subtree, which
-#    the ternary below expresses. `foreachparameters` has no such rule — it takes `values` of each
-#    trailing argument, so a bare `GlobalSection` beside a `NamedTuple` of caches is a `MethodError`.
-#
-# The `nothing` skip is the one thing the two have in common, and it is one line here.
-#
-# The `λY` test names both container types for the same reason `_make_optimizer_cache` asks the
-# structural question first: a section tree is something to descend into, whichever type carries it.
-# Only a `NamedTuple` ever arrives: `GeometricOptimizers`' own
-# `GlobalSection(ps::NetworkParameters) = GlobalSection(params(ps))` unwraps the container, so a
-# section tree is a plain `NamedTuple` whether the parameters are wrapped or not. Should that ever
-# return a *container* of sections instead, `isa NamedTuple` alone would be false here and every layer
-# would be handed the whole tree instead of its own section.
-function _tree_optim_step!(caches, states, dp, ps, λY, method, retraction, step_size)
-    if caches isa NamedTuple
-        for k in keys(caches)
-            dp_k = dp[k]
-            dp_k === nothing && continue
-            λY_k = _is_section_tree(λY) ? λY[k] : λY
-            _tree_optim_step!(caches[k], states[k], dp_k, ps[k], λY_k,
-                method, retraction, step_size)
-        end
-    else
-        _leaf_optim_step!(caches, states, dp, ps, λY, method, retraction, step_size)
-    end
-    nothing
-end
-
-"""
-    optimization_step!(opt, λY, ps, dp)
-
-Apply one optimization step to the parameters `ps` and their gradient `dp`, with the method and step
-size the [`Optimizer`](@ref) `opt` carries.
-
-`λY` is a `GlobalSection` of `ps` (or a `NamedTuple` of them). Note that it is an *output*
-here: the section the optimizer carries from step to step lives in `opt.state`, and `λY` is written
-so that callers who inspect it see the updated section. It therefore has to be allocated once and
-reused, not rebuilt per step -- rebuilding it costs a QR decomposition per manifold weight.
-
-The step counter is incremented *before* the step size is read, so the first step of a run is step 1.
-This matters for a decaying `step_size` and is how `GeometricOptimizers` counts too.
-"""
-function optimization_step!(opt::Optimizer, λY, ps, dp)
-    # The increment comes *first*, so the first step of a run is step 1. It matters only for a
-    # decaying `step_size`, and there it matters: reading the schedule before incrementing takes
-    # `α(0) = η₁`, one whole step above what `DecayingStatic` and `GeometricOptimizers.solve!`
-    # take. `solve!` counts the same way, by calling `increase_iteration_number!` before
-    # `solver_step!`.
-    opt.iterations += 1
-    step = _current_step_size(opt, opt.iterations)
-    _tree_optim_step!(opt.cache, opt.state, dp, ps, λY, opt.method, opt.retraction, step)
-    nothing
-end
-
-check(::Optimizer) = nothing
diff --git a/test/Project.toml b/test/Project.toml
index 8a7377119..3359d574b 100644
--- a/test/Project.toml
+++ b/test/Project.toml
@@ -27,13 +27,14 @@ GeometricMachineLearning = {path = ".."}
 [compat]
 Aqua = "0.8"
 ChainRulesTestUtils = "1"
-# A tight lower bound. `test/quality/aqua.jl` names four of `test_explicit_imports`' seven keyword switches,
+# A tight lower bound. `test/quality/aqua.jl` names three of `test_explicit_imports`' seven keyword switches,
 # and 1.15 is the version those switches were read from; a wider bound would be a guess about
 # earlier ones.
 ExplicitImports = "1.15"
 FiniteDifferences = "0.12"
 GeometricIntegrators = "0.18.2"
 GeometricProblems = "0.8, 0.9"
+HDF5 = "0.16, 0.17"
 Metal = "1.10"
 Printf = "1"
 SafeTestsets = "0.1"
diff --git a/test/architectures/lagrangian_neural_network_tests.jl b/test/architectures/lagrangian_neural_network_tests.jl
index a778f5391..93961e504 100644
--- a/test/architectures/lagrangian_neural_network_tests.jl
+++ b/test/architectures/lagrangian_neural_network_tests.jl
@@ -154,8 +154,8 @@ function test_lnn_training(; n = 1, n_epochs = 30)
     # Start the training from a different draw, so the target is not already met.
     nn = NeuralNetwork(arch)
     dl = DataLoader(input, output; suppress_info = true)
-    optimizer = Optimizer(Adam(), nn)
-    loss_array = optimizer(nn, dl, Batch(10), n_epochs, loss; show_progress = false)
+    optimizer = NetworkOptimizer(nn; algorithm = Adam())
+    loss_array = train!(nn, optimizer, dl, Batch(10), n_epochs; loss, show_progress = false)
 
     @test loss_array[end] < loss_array[1]
 end
diff --git a/test/architectures/sae_error_lower_than_psd_error.jl b/test/architectures/sae_error_lower_than_psd_error.jl
index a76497db7..b695e56de 100644
--- a/test/architectures/sae_error_lower_than_psd_error.jl
+++ b/test/architectures/sae_error_lower_than_psd_error.jl
@@ -17,8 +17,8 @@ function test_accuracy(N::Integer, n::Integer; n_epochs::Integer = 100)
 
     sae_nn = NeuralNetwork(SymplecticAutoencoder(N, n; n_encoder_layers = 5, n_decoder_layers = 5))
 
-    o = Optimizer(Adam(), sae_nn)
-    sae_error = o(sae_nn, dl, Batch(10), n_epochs; show_progress = false)[end]
+    o = NetworkOptimizer(sae_nn; algorithm = Adam())
+    sae_error = train!(sae_nn, o, dl, Batch(10), n_epochs; show_progress = false)[end]
 
     @test sae_error < psd_error
 end
diff --git a/test/architectures/symplectic_autoencoder_tests.jl b/test/architectures/symplectic_autoencoder_tests.jl
index 56f83a4c3..226c9dd60 100644
--- a/test/architectures/symplectic_autoencoder_tests.jl
+++ b/test/architectures/symplectic_autoencoder_tests.jl
@@ -10,8 +10,8 @@ function test_accuracy(N::Integer, n::Integer; tol::Real = 0.35, n_epochs::Integ
 
     sae_nn = NeuralNetwork(SymplecticAutoencoder(N, n))
 
-    o = Optimizer(Adam(), sae_nn)
-    sae_error = o(sae_nn, dl, Batch(10), n_epochs; show_progress = false)[end]
+    o = NetworkOptimizer(sae_nn; algorithm = Adam())
+    sae_error = train!(sae_nn, o, dl, Batch(10), n_epochs; show_progress = false)[end]
 
     @test sae_error < tol
 end
@@ -39,8 +39,8 @@ function test_symplecticity(N::Integer, n::Integer)
 
     # test if it's still symplectic after training
     dl = DataLoader(rand(N, 10 * N); autoencoder = true)
-    o = Optimizer(Adam(), sae_nn)
-    o(sae_nn, dl, Batch(10), 10; show_progress = false)
+    o = NetworkOptimizer(sae_nn; algorithm = Adam())
+    train!(sae_nn, o, dl, Batch(10), 10; show_progress = false)
     sympl_mat = jacobian(vec -> sae_decoder(vec), test_vector)[1]
     @test PoissonTensor(n) ≈ sympl_mat' * PoissonTensor(N) * sympl_mat
 end
diff --git a/test/data_loader/batch_data_loader_qp_test.jl b/test/data_loader/batch_data_loader_qp_test.jl
index fbacafaca..cf444a4a2 100644
--- a/test/data_loader/batch_data_loader_qp_test.jl
+++ b/test/data_loader/batch_data_loader_qp_test.jl
@@ -40,14 +40,14 @@ function test_data_loader(dim = 2, number_of_time_steps = 100,
     nn2 = NeuralNetwork(arch2, CPU(), T)
 
     batch = Batch(batch_size)
-    o₁ = Optimizer(GradientOptimizer(), nn1)
-    o₂ = Optimizer(GradientOptimizer(), nn2)
+    o₁ = NetworkOptimizer(nn1; algorithm = GradientMethod())
+    o₂ = NetworkOptimizer(nn2; algorithm = GradientMethod())
 
     params₁ = deepcopy(nn1.params)
     params₂ = deepcopy(nn2.params)
 
-    loss_array₁ = o₁(nn1, dl1, batch, n_epochs; show_progress = false)
-    loss_array₂ = o₂(nn2, dl2, batch, n_epochs; show_progress = false)
+    loss_array₁ = train!(nn1, o₁, dl1, batch, n_epochs; show_progress = false)
+    loss_array₂ = train!(nn2, o₂, dl2, batch, n_epochs; show_progress = false)
 
     # Both shapes reach the training loop and a step is taken for each. The assertion is that the
     # parameters moved, not that the loss fell: the data are random, so a fall over `n_epochs`
diff --git a/test/data_loader/data_loader_for_input_and_output.jl b/test/data_loader/data_loader_for_input_and_output.jl
index 721ac1694..34ea3a9e8 100644
--- a/test/data_loader/data_loader_for_input_and_output.jl
+++ b/test/data_loader/data_loader_for_input_and_output.jl
@@ -16,10 +16,10 @@ nepochs = 2000
 model = Chain(Dense(1, nwidth), Dense(nwidth, 1))
 nn = NeuralNetwork(model, Float32)
 dl = DataLoader(xsamples, ysamples)
-o = Optimizer(Adam(), nn)
+o = NetworkOptimizer(nn; algorithm = Adam())
 batch = Batch(nbatch, 1, 1)
 
 loss = FeedForwardLoss()
 
-loss_array = o(nn, dl, batch, nepochs, loss; show_progress = false)
+loss_array = train!(nn, o, dl, batch, nepochs; loss, show_progress = false)
 @test loss_array[end] < 0.9
diff --git a/test/data_loader/data_loader_optimization_step.jl b/test/data_loader/data_loader_optimization_step.jl
index 9a9405bd0..7996f3a2f 100644
--- a/test/data_loader/data_loader_optimization_step.jl
+++ b/test/data_loader/data_loader_optimization_step.jl
@@ -1,4 +1,5 @@
 using GeometricMachineLearning, Test, Zygote
+using GeometricOptimizers: optimization_step!
 import Random
 
 Random.seed!(1234)
@@ -16,10 +17,9 @@ function test_data_loader(sys_dim, n_time_steps, n_params, T = Float32)
     loss = GeometricMachineLearning.TransformerLoss(n_time_steps)
     dx = Zygote.gradient(ps -> loss(model, ps, dl.input, dl.input), ps)[1]
     ps_copy = deepcopy(ps)
-    o = Optimizer(GradientOptimizer(), ps)
-    λY = GlobalSection(ps)
-    optimization_step!(o, λY, ps, dx)
-    @test ps !== ps_copy
+    o = NetworkOptimizer(ps; algorithm = GradientMethod())
+    optimization_step!(ps, o, dx)
+    @test ps != ps_copy
 end
 
 test_data_loader(4, 200, 1000)
diff --git a/test/data_loader/optimizer_functor_with_adam.jl b/test/data_loader/optimizer_functor_with_adam.jl
index ad449788e..5c4f55bdb 100644
--- a/test/data_loader/optimizer_functor_with_adam.jl
+++ b/test/data_loader/optimizer_functor_with_adam.jl
@@ -49,9 +49,8 @@ function test_optimization_with_adam(; T = Float32, input_dim = 9, n_patches = 4
 
     loss₁ = loss(model, ps, dl.input, dl.output)
 
-    opt = Optimizer(Adam(), nn_obj)
-    λY = GlobalSection(ps)
-    loss_average = optimize_for_one_epoch!(opt, model, ps, dl, batch, loss, λY)
+    opt = NetworkOptimizer(nn_obj; algorithm = Adam())
+    loss_average = only(train!(nn_obj, opt, dl, batch, 1; loss, show_progress = false))
 
     loss₃ = loss(model, ps, dl.input, dl.output)
 
diff --git a/test/data_loader/train.jl b/test/data_loader/train.jl
new file mode 100644
index 000000000..5c102bc32
--- /dev/null
+++ b/test/data_loader/train.jl
@@ -0,0 +1,114 @@
+using GeometricMachineLearning
+using GeometricMachineLearning: params
+using Test
+import Random
+
+include("../helpers/eltypes.jl")
+
+# The target is in the model class, so the loss goes to zero; the factor 100 leaves margin in
+# `Float32`.
+@testset "train! fits a linear map ($T)" for T in REAL_ELTYPES
+    Random.seed!(1)
+    A = T[1 2; -1 3]
+    x = randn(T, 2, 40)
+    dl = DataLoader(x, A * x; suppress_info = true)
+    nn = NeuralNetwork(Chain(Dense(2, 2, identity)), T)
+    opt = NetworkOptimizer(nn; algorithm = Adam(), linesearch = T(0.01))
+    history = train!(
+        nn, opt, dl, Batch(8), 200; loss = FeedForwardLoss(), show_progress = false)
+    @test eltype(history) === T
+    @test length(history) == 200
+    @test history[end] < history[1] / 100
+end
+
+# The iteration number, the moments and the global sections live in `opt` only, so two calls of 5
+# epochs are one call of 10. Seeding before the network and the optimizer makes the minibatches and
+# the random completions of the sections the same draws in both runs.
+@testset "a second call of train! resumes the first ($T)" for T in REAL_ELTYPES
+    x = randn(Random.Xoshiro(2), T, 6, 30)
+    dl = DataLoader(x, x[1:2, :]; suppress_info = true)
+    function trained(epochs...)
+        Random.seed!(3)
+        nn = NeuralNetwork(Chain(StiefelLayer(6, 3), Dense(3, 2, tanh)), T)
+        opt = NetworkOptimizer(nn; algorithm = Adam())
+        for n in epochs
+            train!(
+                nn, opt, dl, Batch(10), n; loss = FeedForwardLoss(), show_progress = false)
+        end
+        nn, opt
+    end
+    nn₁, opt₁ = trained(5, 5)
+    nn₂, opt₂ = trained(10)
+    @test params(nn₁).L1.weight isa StiefelManifold{T}
+    @test params(nn₁) == params(nn₂)
+    @test opt₁.optimizer.state.iterations == opt₂.optimizer.state.iterations == 30
+end
+
+# The history has the type of the loss value, and nothing is converted: the network's element type
+# wins over the data's.
+@testset "the history has the element type of the loss" begin
+    for (Tnn, Tdata, Thistory) in ((Float64, Float32, Float64), (Float32, Float32, Float32),
+        (Float32, Int, Float64))
+        Random.seed!(4)
+        x = Tdata <: Integer ? rand(Tdata(-3):Tdata(3), 2, 10) : rand(Tdata, 2, 10)
+        dl = DataLoader(x, x; suppress_info = true)
+        nn = NeuralNetwork(Chain(Dense(2, 2, identity)), Tnn)
+        history = train!(
+            nn, NetworkOptimizer(nn), dl, Batch(5), 2; loss = FeedForwardLoss(),
+            show_progress = false)
+        @test eltype(history) === Thistory
+    end
+end
+
+# A step of size `floatmin(T)` rounds away against parameters of order one, so every minibatch loss is
+# the loss at `p₀`, and entry 1 of the history is the mean of three numbers that are computed here
+# from the same minibatches.
+@testset "the history holds the mean of the minibatch losses ($T)" for T in REAL_ELTYPES
+    dl = DataLoader(randn(Random.Xoshiro(6), T, 2, 12), randn(Random.Xoshiro(7), T, 2, 12);
+        suppress_info = true)
+    batch = Batch(4)
+    loss = FeedForwardLoss()
+    nn = NeuralNetwork(Chain(Dense(2, 2, tanh)), T)
+    opt = NetworkOptimizer(nn; algorithm = GradientMethod(), linesearch = floatmin(T))
+    p₀ = deepcopy(params(nn))
+    Random.seed!(8)
+    losses = map(batch(dl)) do indices
+        data = GeometricMachineLearning.convert_input_and_batch_indices_to_array(
+            dl, batch, indices)
+        loss(nn.model, params(nn), data...)
+    end
+    Random.seed!(8)
+    history = train!(nn, opt, dl, batch, 1; loss, show_progress = false)
+    @test params(nn) == p₀
+    @test eltype(losses) === eltype(history) === T
+    @test length(losses) == 3
+    @test !(losses[3] ≈ sum(losses) / 3)
+    @test history[1] ≈ sum(losses) / 3
+end
+
+# Without `loss`, `train!` trains on the loss that matches the architecture and the batch.
+@testset "train!'s default losses" begin
+    transformer_batch = Batch(4, 3)
+    @test GeometricMachineLearning._default_loss(GSympNet(4; n_layers = 2), Batch(4)) isa
+          FeedForwardLoss
+    @test GeometricMachineLearning._default_loss(
+        StandardTransformerIntegrator(4; n_heads = 2, L = 1, n_blocks = 1),
+        transformer_batch) == TransformerLoss(transformer_batch)
+    autoencoder = SymplecticAutoencoder(4, 2; n_encoder_blocks = 2, n_decoder_blocks = 2)
+    @test GeometricMachineLearning._default_loss(autoencoder, Batch(4)) isa AutoEncoderLoss
+end
+
+@testset "train! refuses n_epochs < 1 before any step" begin
+    Random.seed!(5)
+    nn = NeuralNetwork(Chain(Dense(2, 2, identity)))
+    dl = DataLoader(rand(2, 10), rand(2, 10); suppress_info = true)
+    opt = NetworkOptimizer(nn)
+    p₀ = deepcopy(params(nn))
+    for n in (0, -1)
+        @test_throws ArgumentError train!(
+            nn, opt, dl, Batch(5), n; loss = FeedForwardLoss(),
+            show_progress = false)
+    end
+    @test opt.optimizer.state.iterations == 0
+    @test params(nn) == p₀
+end
diff --git a/test/data_loader/training_history_eltype.jl b/test/data_loader/training_history_eltype.jl
index 87e5247e5..14e5611e5 100644
--- a/test/data_loader/training_history_eltype.jl
+++ b/test/data_loader/training_history_eltype.jl
@@ -1,7 +1,7 @@
-# A `Float32` network's training history is `Float32`: `(o::Optimizer)(...)` builds `loss_array`
-# typed to `eltype(dl)`. This asserts that, and that `optimize_for_one_epoch!`'s own accumulator
-# agrees with it: the loss the optimizer returns is in the data's element type, not promoted to
-# `Float64` along the way. See `CHANGELOG.md` for the fix this guards.
+# A `Float32` network's training history on `Float32` data is `Float32`: `train!` gives the history
+# the type of the loss value, and the epoch's accumulator keeps that type, so nothing is promoted to
+# `Float64` along the way. `test/data_loader/train.jl` checks a network and data of two element
+# types.
 
 using GeometricMachineLearning
 using Test
@@ -12,9 +12,10 @@ Random.seed!(11)
 function test_training_history_eltype(T::DataType)
     nn = NeuralNetwork(GSympNet(4), CPU(), T)
     dl = DataLoader(rand(T, 4, 64); autoencoder = false, suppress_info = true)
-    o = Optimizer(GradientOptimizer(), nn)
+    o = NetworkOptimizer(nn; algorithm = GradientMethod())
 
-    history = o(nn, dl, Batch(8), 2, FeedForwardLoss(); show_progress = false)
+    history = train!(
+        nn, o, dl, Batch(8), 2; loss = FeedForwardLoss(), show_progress = false)
 
     @test eltype(history) == T
     @test history isa Vector{T}
diff --git a/test/helpers/eltypes.jl b/test/helpers/eltypes.jl
new file mode 100644
index 000000000..cde8d5c2d
--- /dev/null
+++ b/test/helpers/eltypes.jl
@@ -0,0 +1,3 @@
+# The real element types every numeric test runs in. A test loops over this tuple rather than over a
+# literal one, so that one line sets the precisions of the whole suite.
+const REAL_ELTYPES = (Float32, Float64)
diff --git a/test/helpers/network_parameters_gradient_structure.jl b/test/helpers/network_parameters_gradient_structure.jl
index 114af9229..ffa45f0b8 100644
--- a/test/helpers/network_parameters_gradient_structure.jl
+++ b/test/helpers/network_parameters_gradient_structure.jl
@@ -1,33 +1,21 @@
 using Test
 using GeometricMachineLearning
 using GeometricMachineLearning: params
-using LinearAlgebra: norm
+using LinearAlgebra: Diagonal, diag, norm
 using Zygote: gradient
 
 """
-Assert which shape the `SymmetricMatrix` leaf of a gradient has, and that the shape is cosmetic.
+Assert the shape and the value of the `SymmetricMatrix` leaf of a gradient, taken once with respect
+to a `NetworkParameters` wrapper and once with respect to the bare `NamedTuple` it wraps.
 
-A gradient taken with respect to a `NetworkParameters` wrapper keeps its `SymmetricMatrix` leaf
-only when the loss expression performs a single `getproperty` on that wrapper. A second access
-degrades the leaf to a plain `Matrix`. What decides the split is the number of `getproperty` calls
-on the wrapper itself -- not how often the leaf is used, and not how often the plain `NamedTuple`s
-below the wrapper are accessed. `p -> (A = p.L1.A; sum(A) + sum(A))` uses the leaf twice through one
-access and keeps the structure; `p -> sum(p.L1.A) + sum(p.L1.A)` loses it. A `Zygote.gradient` taken
-with respect to the bare wrapped `NamedTuple` (`params(ps)`) keeps the structure at any access
-count, because `gradient` projects its result and `ChainRulesCore.ProjectTo` restores the leaf for a
-`NamedTuple` but is `identity` for a `NetworkParameters`. The underlying `Zygote.pullback` loses the
-structure on the second access either way. `CHANGELOG.md` records the experiment.
-
-A wrapper unwrapped with `values` rather than `getproperty` -- which is how `Chain` hands a layer
-its parameters -- counts differently: there the accesses on the plain `NamedTuple` below decide.
-See the layer testset in
-`test/layers/symplectic_attention_network_parameters_gradient.jl`.
-
-`preserves_structure` states which side of that split `f` sits on. Both gradients are asserted to
-agree numerically either way, so losing the structure is a change of type and not of value.
+Both gradients keep the `SymmetricMatrix` leaf, at any number of accesses to the wrapper. They hold
+different quantities. With respect to the bare `NamedTuple`, `Zygote.gradient` projects the dense
+gradient ``G`` onto the symmetric matrices, ``(G + G^T)/2``. With respect to the wrapper, the leaf
+holds the gradient with respect to the leaf's storage, the lower triangle: its diagonal is that of
+``G``, and each off-diagonal entry is ``G_{ij} + G_{ji}``, twice that of the projection (issue #319,
+`KNOWN_ISSUES.md` K6).
 """
-function network_parameters_gradient_structure_test(
-        f, ps::NetworkParameters, preserves_structure::Bool)
+function network_parameters_gradient_structure_test(f, ps::NetworkParameters)
     g_wrapped = gradient(_ps -> norm(f(_ps)), ps)[1]
     g_bare = gradient(_ps -> norm(f(_ps)), params(ps))[1]
 
@@ -35,11 +23,8 @@ function network_parameters_gradient_structure_test(
     a_bare = g_bare.L1.A
 
     @test typeof(a_bare) <: SymmetricMatrix
-    if preserves_structure
-        @test typeof(a_wrapped) <: SymmetricMatrix
-    else
-        @test_broken typeof(a_wrapped) <: SymmetricMatrix  # issue #319
-    end
+    @test typeof(a_wrapped) <: SymmetricMatrix
 
-    @test isapprox(Matrix(a_wrapped), Matrix(a_bare))
+    B = Matrix(a_bare)
+    @test Matrix(a_wrapped) ≈ 2B - Diagonal(diag(B))
 end
diff --git a/test/integration/losses_and_optimization.jl b/test/integration/losses_and_optimization.jl
index cf2ef1425..7b41ef407 100644
--- a/test/integration/losses_and_optimization.jl
+++ b/test/integration/losses_and_optimization.jl
@@ -17,9 +17,9 @@ function train_network(; n_epochs = 10)
     nn = setup_network(dl)
     loss = FeedForwardLoss()
 
-    o = Optimizer(Adam(), nn)
+    o = NetworkOptimizer(nn; algorithm = Adam())
     batch = Batch(5, 1)
-    loss_array = o(nn, dl, batch, n_epochs, loss; show_progress = false)
+    loss_array = train!(nn, o, dl, batch, n_epochs; loss, show_progress = false)
     T = eltype(dl)
     @test loss_array[end] / loss_array[1] < T(0.1)
 end
diff --git a/test/integration/reduced_loss_optimization.jl b/test/integration/reduced_loss_optimization.jl
index 0894fcb98..db40720f7 100644
--- a/test/integration/reduced_loss_optimization.jl
+++ b/test/integration/reduced_loss_optimization.jl
@@ -5,7 +5,7 @@ import Random
 
 Random.seed!(123)
 
-# `ReducedLoss` trained through the `Optimizer` functor, which is the path the symplectic
+# `ReducedLoss` trained through `train!`, which is the path the symplectic
 # autoencoder tutorial documents and the path `scripts/reproduction/symplectic_autoencoders/`
 # takes. It was unreachable: the loss annotated its parameter argument `::NetworkParameters`, and
 # `Zygote.pullback` evaluates the forward pass with the parameters unwrapped -- the closure body
@@ -16,10 +16,10 @@ Random.seed!(123)
 #
 # Nothing had caught it because nothing ran it. The docstring example calls the loss directly,
 # which dispatches, and the tutorial's training is in a plain ```julia fence that Documenter
-# renders and never executes. Restoring the annotation makes the `Optimizer` call below raise
+# renders and never executes. Restoring the annotation makes the `train!` call below raise
 # that `error`, so this testset errors before any of the three assertions runs; the assertions
 # themselves check what the loss is for.
-@testset "ReducedLoss trains through the Optimizer functor" begin
+@testset "ReducedLoss trains through train!" begin
     reduced_dim = 2
 
     q = rand(Float64, 6, 30, 2)
@@ -37,12 +37,12 @@ Random.seed!(123)
         StandardTransformerIntegrator(reduced_dim; transformer_dim = 10, n_blocks = 1,
             n_heads = 2, L = 1, upscaling_activation = tanh), CPU(), Float64)
 
-    o = Optimizer(Adam(), integrator)
+    o = NetworkOptimizer(integrator; algorithm = Adam())
     batch = Batch(8, 4)
 
     # The parameters reach the loss through `Zygote.pullback` here, which is what an annotated
     # `params` argument breaks; calling the loss directly would pass either way.
-    loss_array = o(integrator, dl_reduced, batch, 10, loss; show_progress = false)
+    loss_array = train!(integrator, o, dl_reduced, batch, 10; loss, show_progress = false)
 
     @test length(loss_array) == 10
     @test all(isfinite, loss_array)
diff --git a/test/integration/train_manual_architectures.jl b/test/integration/train_manual_architectures.jl
new file mode 100644
index 000000000..9955720d4
--- /dev/null
+++ b/test/integration/train_manual_architectures.jl
@@ -0,0 +1,69 @@
+using GeometricMachineLearning
+using GeometricMachineLearning: params
+using Test
+import Random
+
+include("../helpers/eltypes.jl")
+
+# Each architecture that the manual trains, with the data shape and the batch its tutorial uses. The
+# list holds a `SymmetricMatrix` leaf (`LASympNet`, `LinearSymplecticTransformer`), the strictly
+# triangular leaves (`VolumePreservingFeedForward`, `VolumePreservingTransformer`), a Stiefel leaf
+# (`SymplecticAutoencoder`), and the three `Chain`s of the volume-preserving-attention tutorial.
+# The last place holds the loss; `nothing` trains on the default loss.
+function manual_architectures(T)
+    feedforward = DataLoader(rand(T, 4, 12, 3); suppress_info = true)
+    transformer_batch = Batch(4, 3)
+    hnn = StandardHamiltonianArchitecture(2, 4)
+    attention_data = DataLoader(rand(T, 1, 12, 2); suppress_info = true)
+    attention_batch = Batch(4, 3, 1)
+    attention_chain(skew_sym) = Chain(Dense(1, 2, tanh),
+        VolumePreservingAttention(2, 3; skew_sym), Dense(2, 1, identity; use_bias = true))
+    (("GSympNet", GSympNet(4; n_layers = 2), feedforward, Batch(4), nothing),
+        ("LASympNet", LASympNet(4; depth = 1, nhidden = 1), feedforward, Batch(4), nothing),
+        ("ResNet", ResNet(4, 2, tanh), feedforward, Batch(4), nothing),
+        ("VolumePreservingFeedForward",
+            VolumePreservingFeedForward(4, 2), feedforward, Batch(4),
+            nothing),
+        ("StandardTransformerIntegrator",
+            StandardTransformerIntegrator(4; n_heads = 2, L = 1, n_blocks = 1),
+            feedforward, transformer_batch, nothing),
+        ("LinearSymplecticTransformer",
+            LinearSymplecticTransformer(4, 3; n_sympnet = 1, L = 1),
+            feedforward, transformer_batch, nothing),
+        ("SymplecticTransformer", SymplecticTransformer(4; n_sympnet = 1, L = 1),
+            feedforward, transformer_batch, nothing),
+        ("VolumePreservingTransformer", VolumePreservingTransformer(4, 3; n_blocks = 1),
+            feedforward, transformer_batch, nothing),
+        ("SymplecticAutoencoder",
+            SymplecticAutoencoder(4, 2; n_encoder_blocks = 2, n_decoder_blocks = 2),
+            DataLoader(rand(T, 4, 12); autoencoder = true, suppress_info = true), Batch(4),
+            nothing),
+        ("StandardHamiltonianArchitecture", hnn,
+            DataLoader(rand(T, 2, 12), rand(T, 2, 12); suppress_info = true), Batch(4),
+            HNNLoss(hnn)),
+        ("VolumePreservingAttention, skew_sym = true", attention_chain(true),
+            attention_data, attention_batch, TransformerLoss(attention_batch)),
+        ("VolumePreservingAttention, skew_sym = false", attention_chain(false),
+            attention_data, attention_batch, TransformerLoss(attention_batch)),
+        ("Dense comparison of the attention tutorial",
+            Chain(Dense(1, 2, tanh), Dense(2, 1, identity; use_bias = true)),
+            attention_data, Batch(4), FeedForwardLoss()))
+end
+
+@testset "train! trains every architecture of the manual ($T)" for T in REAL_ELTYPES
+    Random.seed!(6)
+    for (name, arch, dl, batch, loss) in manual_architectures(T)
+        @testset "$name" begin
+            nn = NeuralNetwork(arch, T)
+            p₀ = deepcopy(params(nn))
+            opt = NetworkOptimizer(nn; algorithm = Adam())
+            history = loss === nothing ?
+                      train!(nn, opt, dl, batch, 2; show_progress = false) :
+                      train!(nn, opt, dl, batch, 2; loss, show_progress = false)
+            @test length(history) == 2
+            @test eltype(history) === T
+            @test all(isfinite, history)
+            @test params(nn) != p₀
+        end
+    end
+end
diff --git a/test/layers/double_multiplication_network_parameters_gradient.jl b/test/layers/double_multiplication_network_parameters_gradient.jl
index d2fb3557c..98c1c2cfe 100644
--- a/test/layers/double_multiplication_network_parameters_gradient.jl
+++ b/test/layers/double_multiplication_network_parameters_gradient.jl
@@ -22,12 +22,12 @@ ps = NetworkParameters((L1 = (A = S,),))
 t = rand(4, 4)
 
 @testset "_custom_mul: NetworkParameters gradient structure" begin
-    @testset "single_multiplication (one wrapper access: structure kept)" begin
+    @testset "single_multiplication (one wrapper access)" begin
         network_parameters_gradient_structure_test(
-            _ps -> single_multiplication(t, _ps), ps, true)
+            _ps -> single_multiplication(t, _ps), ps)
     end
-    @testset "double_multiplication (two wrapper accesses: structure lost)" begin
+    @testset "double_multiplication (two wrapper accesses)" begin
         network_parameters_gradient_structure_test(
-            _ps -> double_multiplication(t, _ps), ps, false)
+            _ps -> double_multiplication(t, _ps), ps)
     end
 end
diff --git a/test/layers/gradient_layer_tests.jl b/test/layers/gradient_layer_tests.jl
index f0eb7474a..1117dcb6e 100644
--- a/test/layers/gradient_layer_tests.jl
+++ b/test/layers/gradient_layer_tests.jl
@@ -1,5 +1,6 @@
 using GeometricMachineLearning
 using GeometricMachineLearning: params
+using GeometricOptimizers: optimization_step!
 using Test
 using KernelAbstractions
 import Random, Zygote
@@ -24,19 +25,18 @@ end
 function test_gradient_layer_derivative_and_update(T, M, N, batch_size = 10)
     dummy_model = Chain(GradientLayerP(M, N, tanh), GradientLayerQ(M, N, tanh))
     ps = params(NeuralNetwork(dummy_model, CPU(), T))
-    o = Optimizer(Adam(T; β₁ = T(0.9), β₂ = T(0.999), δ = T(3e-7)), ps;
-        step_size = T(0.1))
+    o = NetworkOptimizer(ps; algorithm = Adam(; β₁ = 0.9, β₂ = 0.999, δ = 3e-7),
+        linesearch = T(0.1))
 
-    # test for vector 
+    # test for vector
     x = rand(T, M)
     gs = Zygote.gradient(ps -> sum(dummy_model(x, ps)), ps)[1]
-    λY = GlobalSection(ps)
-    optimization_step!(o, λY, ps, gs)
+    optimization_step!(ps, o, gs)
 
-    # test for matrix 
+    # test for matrix
     X = rand(T, M, batch_size)
     gs = Zygote.gradient(ps -> sum(dummy_model(X, ps)), ps)[1]
-    optimization_step!(o, λY, ps, gs)
+    optimization_step!(ps, o, gs)
 end
 
 types = (Float32, Float64)
diff --git a/test/layers/manifold_layers.jl b/test/layers/manifold_layers.jl
index aa51054e5..15f12a9cb 100644
--- a/test/layers/manifold_layers.jl
+++ b/test/layers/manifold_layers.jl
@@ -1,4 +1,5 @@
 using Test, LinearAlgebra, GeometricMachineLearning
+using GeometricOptimizers: optimization_step!
 import Random
 
 Random.seed!(1234)
@@ -6,13 +7,12 @@ Random.seed!(1234)
 function stiefel_layer_test(T, M, N, tol = 1.0f-1)
     model = Chain(StiefelLayer(M, N), StiefelLayer(N, N))
     ps = NeuralNetwork(model, T).params
-    o = Optimizer(Adam(T; β₁ = T(5.0f-1), β₂ = T(5.0f-1), δ = T(3.0f-7)), ps;
-        step_size = T(1.0f0))
+    o = NetworkOptimizer(ps; algorithm = Adam(; β₁ = 5.0f-1, β₂ = 5.0f-1, δ = 3.0f-7),
+        linesearch = T(1.0f0))
 
-    dx = (L1 = (weight = rand(T, N, M),), L2 = (weight = rand(T, N, N),))
+    dx = NetworkParameters((L1 = (weight = rand(T, N, M),), L2 = (weight = rand(T, N, N),)))
     ps_copy = deepcopy(ps)
-    λY = GlobalSection(ps)
-    optimization_step!(o, λY, ps, dx)
+    optimization_step!(ps, o, dx)
     # check that the new weight is different from the old one
     @test norm(ps_copy[1].weight - ps[1].weight) > T(tol)
     # check that the updated elements are on the Stiefel Manifold 
@@ -23,14 +23,13 @@ end
 function grassmann_layer_test(T, M, N, tol = 1.0f-1)
     model = Chain(GrassmannLayer(M, N), StiefelLayer(N, N))
     ps = NeuralNetwork(model, T).params
-    o = Optimizer(Adam(T; β₁ = T(5.0f-1), β₂ = T(5.0f-1), δ = T(3.0f-7)), ps;
-        step_size = T(1.0f0))
+    o = NetworkOptimizer(ps; algorithm = Adam(; β₁ = 5.0f-1, β₂ = 5.0f-1, δ = 3.0f-7),
+        linesearch = T(1.0f0))
 
-    dx = (L1 = (weight = rand(T, N, M),), L2 = (weight = rand(T, N, N),))
+    dx = NetworkParameters((L1 = (weight = rand(T, N, M),), L2 = (weight = rand(T, N, N),)))
     ps_copy = deepcopy(ps)
-    λY = GlobalSection(ps)
     for i in 1:4
-        optimization_step!(o, λY, ps, dx)
+        optimization_step!(ps, o, dx)
     end
     # check that the new weight is different from the old one
     @test norm(ps_copy[1].weight - ps[1].weight) > T(tol)
diff --git a/test/layers/multi_head_attention_stiefel_optim_cache.jl b/test/layers/multi_head_attention_stiefel_optim_cache.jl
index c94822e9a..65813d443 100644
--- a/test/layers/multi_head_attention_stiefel_optim_cache.jl
+++ b/test/layers/multi_head_attention_stiefel_optim_cache.jl
@@ -1,17 +1,13 @@
 using GeometricMachineLearning, Test
-# `StiefelManifold` and the other shared types are one object reached by two names now, so a blanket
-# `using GeometricOptimizers` alongside GML would no longer be ambiguous on them -- only `Optimizer`
-# still resolves to two different things (issue C1). The qualified form stays because it says which
-# package owns what.
+# The qualified form says which package owns what.
 import GeometricOptimizers
 using GeometricOptimizers: GradientCache, MomentumCache, AdamCache
 import Random, LinearAlgebra
-# The cache and the state hold a `NetworkParameters` per layer: a layer is wrapped at the
-# `GeometricOptimizers` boundary, and the wrap shares the leaf arrays. `mapparameters` and not `map`,
-# because it recurses on the branches and so reaches the leaves whichever shape it is handed.
+# One cache covers the whole network, and its fields hold a `NetworkParameters` of the network's
+# shape. `mapparameters` and not `map`, because it recurses on the branches and so reaches the leaves.
 #
-# Each helper below has two methods, because two shapes arrive: a wrapped layer is a
-# `NetworkParameters` and the tree above it is a plain `NamedTuple`.
+# Each helper below has a method for a `NetworkParameters` and one for a plain `NamedTuple`, so that
+# it reaches the leaves whichever of the two it is handed.
 using NeuralNetworkParameters: NetworkParameters, mapparameters
 
 Random.seed!(1234)
@@ -69,13 +65,13 @@ function test_cache_setups_for_optimizer_for_multihead_attention_layer(T::Type,
     model = Chain(MultiHeadAttention(dim, n_heads, Stiefel = true))
     ps = NeuralNetwork(model, CPU(), T).params
 
-    o₁ = Optimizer(Adam(), ps)
-    o₂ = Optimizer(MomentumMethod(), ps)
-    o₃ = Optimizer(GradientMethod(), ps)
+    o₁ = NetworkOptimizer(ps; algorithm = Adam())
+    o₂ = NetworkOptimizer(ps; algorithm = MomentumMethod())
+    o₃ = NetworkOptimizer(ps; algorithm = GradientMethod())
 
-    check_adam_cache(o₁.cache)
-    check_momentum_cache(o₂.cache)
-    check_gradient_cache(o₃.cache)
+    check_adam_cache(o₁.optimizer.cache)
+    check_momentum_cache(o₂.optimizer.cache)
+    check_gradient_cache(o₃.optimizer.cache)
 end
 
 test_cache_setups_for_optimizer_for_multihead_attention_layer(Float32, 64, 8)
diff --git a/test/layers/multi_head_attention_stiefel_retraction.jl b/test/layers/multi_head_attention_stiefel_retraction.jl
index 7b38a33ca..211a75209 100644
--- a/test/layers/multi_head_attention_stiefel_retraction.jl
+++ b/test/layers/multi_head_attention_stiefel_retraction.jl
@@ -5,12 +5,11 @@ using GeometricMachineLearning: cayley
 # so they are named qualified rather than through a re-export.
 using GeometricOptimizers: MomentumCache
 import Random, Test, LinearAlgebra, KernelAbstractions
-# The cache and the state hold a `NetworkParameters` per layer: a layer is wrapped at the
-# `GeometricOptimizers` boundary, and the wrap shares the leaf arrays. `mapparameters` and not `map`,
-# because it recurses on the branches and so reaches the leaves whichever shape it is handed.
+# One cache covers the whole network, and its fields hold a `NetworkParameters` of the network's
+# shape. `mapparameters` and not `map`, because it recurses on the branches and so reaches the leaves.
 #
-# Each helper below has two methods, because two shapes arrive: a wrapped layer is a
-# `NetworkParameters` and the tree above it is a plain `NamedTuple`.
+# Each helper below has a method for a `NetworkParameters` and one for a plain `NamedTuple`, so that
+# it reaches the leaves whichever of the two it is handed.
 using NeuralNetworkParameters: NetworkParameters, mapparameters
 
 Random.seed!(1234)
@@ -53,7 +52,8 @@ function test_multi_head_attention_retraction(
     model = Chain(MultiHeadAttention(dim, n_heads, Stiefel = true))
 
     ps = NeuralNetwork(model, backend, T).params
-    cache = Optimizer(MomentumMethod(), ps).cache
+    cache = NetworkOptimizer(ps; algorithm = MomentumMethod()).optimizer.cache
+    @test cache isa MomentumCache
 
     check_retraction_geodesic(cache)
 
diff --git a/test/layers/multi_head_attention_stiefel_setup.jl b/test/layers/multi_head_attention_stiefel_setup.jl
index 5a90ea42c..790bf6083 100644
--- a/test/layers/multi_head_attention_stiefel_setup.jl
+++ b/test/layers/multi_head_attention_stiefel_setup.jl
@@ -2,12 +2,11 @@ using GeometricMachineLearning, Test
 # see the note in `multi_head_attention_stiefel_retraction.jl`
 using GeometricOptimizers: MomentumCache
 import Random, Test, LinearAlgebra, KernelAbstractions
-# The cache and the state hold a `NetworkParameters` per layer: a layer is wrapped at the
-# `GeometricOptimizers` boundary, and the wrap shares the leaf arrays. `mapparameters` and not `map`,
-# because it recurses on the branches and so reaches the leaves whichever shape it is handed.
+# One cache covers the whole network, and its fields hold a `NetworkParameters` of the network's
+# shape. `mapparameters` and not `map`, because it recurses on the branches and so reaches the leaves.
 #
-# Each helper below has two methods, because two shapes arrive: a wrapped layer is a
-# `NetworkParameters` and the tree above it is a plain `NamedTuple`.
+# Each helper below has a method for a `NetworkParameters` and one for a plain `NamedTuple`, because
+# the parameters of a network arrive as the one and the cache's fields as the other.
 using NeuralNetworkParameters: NetworkParameters, mapparameters, foldparameters
 
 Random.seed!(1234)
@@ -50,7 +49,8 @@ function check_multi_head_attention_stiefel_setup(T::Type, N::Int, n::Int)
     @test nleaves > 0
     @test check_setup_calls[] == nleaves
 
-    gx = Optimizer(MomentumMethod(), ps).cache
+    gx = NetworkOptimizer(ps; algorithm = MomentumMethod()).optimizer.cache
+    @test gx isa MomentumCache
     check_grad_setup(gx)
 end
 
diff --git a/test/layers/symplectic_attention_network_parameters_gradient.jl b/test/layers/symplectic_attention_network_parameters_gradient.jl
index 405794832..450e9b100 100644
--- a/test/layers/symplectic_attention_network_parameters_gradient.jl
+++ b/test/layers/symplectic_attention_network_parameters_gradient.jl
@@ -28,28 +28,24 @@ ps = NetworkParameters((L1 = (A = S,),))
 t = (q = rand(2, 2), p = rand(2, 2))
 
 @testset "Symplectic attention: NetworkParameters gradient structure" begin
-    @testset "symplectic_attention (two wrapper accesses: structure lost)" begin
+    @testset "symplectic_attention (two wrapper accesses)" begin
         network_parameters_gradient_structure_test(
-            _ps -> symplectic_attention(t, _ps), ps, false)
+            _ps -> symplectic_attention(t, _ps), ps)
     end
-    @testset "symplectic_attention_simplified (one wrapper access: structure kept)" begin
+    @testset "symplectic_attention_simplified (one wrapper access)" begin
         network_parameters_gradient_structure_test(
-            _ps -> symplectic_attention_simplified(t, _ps), ps, true)
+            _ps -> symplectic_attention_simplified(t, _ps), ps)
     end
-    @testset "symplectic_linear_map (one wrapper access: structure kept)" begin
+    @testset "symplectic_linear_map (one wrapper access)" begin
         network_parameters_gradient_structure_test(
-            _ps -> symplectic_linear_map(t, _ps), ps, true)
+            _ps -> symplectic_linear_map(t, _ps), ps)
     end
 end
 
-# The layer keeps the structure although its body reads `A` twice. `Chain` reaches the layer
-# through `values(_ps)` rather than through a `getproperty` on the wrapper, and on that route the
-# count that decides the split is the one on the plain `NamedTuple` the layer is handed. The local
-# binding at `src/layers/symplectic_attention.jl` makes that count one, and is what keeps the
-# structure here: the same body written without it, reached the same way, degrades the leaf to a
-# plain `Matrix`.
+# `Chain` reaches the layer through `values(_ps)` rather than through a `getproperty` on the
+# wrapper, and the layer's body reads `A` twice.
 @testset "SymplecticAttentionQ layer: NetworkParameters gradient structure" begin
     l = SymplecticAttentionQ(4; symmetric = true)
     nn = NeuralNetwork(Chain(l))
-    network_parameters_gradient_structure_test(_ps -> nn(t, _ps), params(nn), true)
+    network_parameters_gradient_structure_test(_ps -> nn(t, _ps), params(nn))
 end
diff --git a/test/layers/transformer_optimizer.jl b/test/layers/transformer_optimizer.jl
index 1d7ffcfd7..f813ffe18 100644
--- a/test/layers/transformer_optimizer.jl
+++ b/test/layers/transformer_optimizer.jl
@@ -1,5 +1,6 @@
 using Test, KernelAbstractions, GeometricMachineLearning, Zygote, LinearAlgebra
 using GeometricMachineLearning: ResNetLayer
+using GeometricOptimizers: optimization_step!
 import Random
 
 Random.seed!(1234)
@@ -18,20 +19,17 @@ function transformer_gradient_test(T, dim, n_heads, L, seq_length = 8, batch_siz
     loss(ps, input) = norm(model(input, ps))
     dx = Zygote.gradient(ps -> loss(ps, input), ps)[1]
 
-    o₁ = Optimizer(GradientMethod(), ps)
-    o₂ = Optimizer(MomentumMethod(), ps)
-    o₃ = Optimizer(Adam(), ps)
-
     ps₁ = deepcopy(ps)
     ps₂ = deepcopy(ps)
     ps₃ = deepcopy(ps)
 
-    λY₁ = GlobalSection(ps₁)
-    λY₂ = GlobalSection(ps₂)
-    λY₃ = GlobalSection(ps₃)
-    optimization_step!(o₁, λY₁, ps₁, dx)
-    optimization_step!(o₂, λY₂, ps₂, dx)
-    optimization_step!(o₃, λY₃, ps₃, dx)
+    o₁ = NetworkOptimizer(ps₁; algorithm = GradientMethod())
+    o₂ = NetworkOptimizer(ps₂; algorithm = MomentumMethod())
+    o₃ = NetworkOptimizer(ps₃; algorithm = Adam())
+
+    optimization_step!(ps₁, o₁, dx)
+    optimization_step!(ps₂, o₂, dx)
+    optimization_step!(ps₃, o₃, dx)
     @test typeof(ps₁) == typeof(ps₂) == typeof(ps₃) == typeof(ps)
     @test ps₁[1].PQ.head_1 ≈ ps₂[1].PQ.head_1
     @test ps₁[1].PQ.head_1 ≉ ps₃[1].PQ.head_1
diff --git a/test/loss/training_method_losses.jl b/test/loss/training_method_losses.jl
index 33b89e292..15ed74285 100644
--- a/test/loss/training_method_losses.jl
+++ b/test/loss/training_method_losses.jl
@@ -170,8 +170,9 @@ function test_symplectic_euler_training(; n = 1, Δt = 1.0e-2, n_epochs = 20)
 
     nn = NeuralNetwork(arch)
     dl = DataLoader(input, output; suppress_info = true)
-    loss_array = Optimizer(Adam(), nn)(
-        nn, dl, Batch(10), n_epochs, loss; show_progress = false)
+    loss_array = train!(
+        nn, NetworkOptimizer(nn; algorithm = Adam()), dl, Batch(10), n_epochs;
+        loss, show_progress = false)
     @test loss_array[end] < loss_array[1]
 end
 
@@ -339,8 +340,9 @@ function test_variational_training(; Δt = 0.05, samples = 60, n_epochs = 20)
     qₙ₊₂ = reshape(q[3:(samples + 2)], 1, samples)
 
     dl = DataLoader(vcat(qₙ, qₙ₊₁), qₙ₊₂; suppress_info = true)
-    loss_array = Optimizer(Adam(), nn)(
-        nn, dl, Batch(10), n_epochs, loss; show_progress = false)
+    loss_array = train!(
+        nn, NetworkOptimizer(nn; algorithm = Adam()), dl, Batch(10), n_epochs;
+        loss, show_progress = false)
     @test loss_array[end] < loss_array[1]
 end
 
diff --git a/test/map_to_cpu.jl b/test/map_to_cpu.jl
index c7d085421..adde8bd23 100644
--- a/test/map_to_cpu.jl
+++ b/test/map_to_cpu.jl
@@ -12,7 +12,8 @@ using Random
 using Test
 
 import GeometricOptimizers: StiefelManifold, SymmetricMatrix, SkewSymMatrix,
-                            LowerTriangular, UpperTriangular
+                            StrictlyLowerTriangular as LowerTriangular,
+                            StrictlyUpperTriangular as UpperTriangular
 
 Random.seed!(1234)
 
diff --git a/test/optimizers/adam_with_learning_rate_decay.jl b/test/optimizers/adam_with_learning_rate_decay.jl
index 12c590f94..fa146c69c 100644
--- a/test/optimizers/adam_with_learning_rate_decay.jl
+++ b/test/optimizers/adam_with_learning_rate_decay.jl
@@ -1,10 +1,12 @@
-
 using GeometricMachineLearning
 using GeometricMachineLearning: ResNetLayer, params
+using GeometricOptimizers: optimization_step!
 using LinearAlgebra: I
 using Test
 import Random
 
+include("../helpers/eltypes.jl")
+
 Random.seed!(123)
 
 const sin_vector = sin.(0:0.1:2π)
@@ -23,18 +25,17 @@ function train_network(; n_epochs = 2048)
     nn₁ = setup_network(dl)
     nn₂ = setup_network(dl)
 
-    o₁ = Optimizer(AdamOptimizer(), nn₁)
-    # `AdamOptimizerWithDecay` is GeometricOptimizers' now, and returns the `(algorithm, linesearch)`
-    # pairing its own `Optimizer` takes, so it splats rather than being passed positionally. The
-    # element type is positional and defaults to `Float64` here, where GML's own version took it
-    # from `η₁` and so defaulted to `Float32`.
-    o₂ = Optimizer(nn₂; AdamOptimizerWithDecay(n_epochs, eltype(dl))...)
+    o₁ = NetworkOptimizer(nn₁; algorithm = Adam())
+    # `AdamOptimizerWithDecay` is GeometricOptimizers', and returns the `(algorithm, linesearch)`
+    # pairing the keywords take, so it splats rather than being passed positionally. Its argument
+    # counts optimization steps, and the optimizer converts it to the element type of the network.
+    o₂ = NetworkOptimizer(nn₂; AdamOptimizerWithDecay(n_epochs)...)
 
     batch = Batch(5, 1)
     loss = FeedForwardLoss()
 
-    loss_array₁ = o₁(nn₁, dl, batch, n_epochs, loss; show_progress = false)
-    loss_array₂ = o₂(nn₂, dl, batch, n_epochs, loss; show_progress = false)
+    loss_array₁ = train!(nn₁, o₁, dl, batch, n_epochs; loss, show_progress = false)
+    loss_array₂ = train!(nn₂, o₂, dl, batch, n_epochs; loss, show_progress = false)
 
     T = eltype(dl)
     @test loss_array₂[end] < loss_array₁[end] < T(1.6e-1)
@@ -45,10 +46,6 @@ end
 
 The pairing is `Adam` plus a `DecayingStatic` step size, so the method the optimizer dispatches on is
 an ordinary `Adam` and the manifold weights take GeometricOptimizers' Adam cache like any other Adam.
-GML's own version of this method was a distinct `OptimizerMethod` that had to be routed onto that
-cache explicitly, and without the routing every weight -- the `StiefelManifold` ones included -- fell
-through to the Euclidean state, whose zero element is a `StiefelLieAlgHorMatrix` and not a manifold
-point.
 """
 # `n_epochs = 128` and not 32: `AdamOptimizerWithDecay(n_epochs)` fixes
 # γ = exp(log(η₂/η₁)/n_epochs), so a 32-epoch budget drives the learning rate to η₂ = 1e-6 almost
@@ -59,8 +56,9 @@ function train_manifold_network(; n_epochs = 128)
     arch = Chain(StiefelLayer(1, 20), Dense(20, 20, tanh), Dense(20, 1, identity))
     nn = NeuralNetwork(arch, CPU(), eltype(dl))
 
-    o = Optimizer(nn; AdamOptimizerWithDecay(n_epochs, eltype(dl))...)
-    loss_array = o(nn, dl, Batch(5, 1), n_epochs, FeedForwardLoss(); show_progress = false)
+    o = NetworkOptimizer(nn; AdamOptimizerWithDecay(n_epochs)...)
+    loss_array = train!(nn, o, dl, Batch(5, 1), n_epochs; loss = FeedForwardLoss(),
+        show_progress = false)
 
     @test all(isfinite, loss_array)
     @test loss_array[end] < loss_array[1]
@@ -72,35 +70,28 @@ end
 @doc raw"""
 The schedule is walked from ``t = 1``, not from ``t = 0``.
 
-`optimization_step!` increments `opt.iterations` *before* it reads the step size, so the first step
-of a run takes ``\alpha(1) = \gamma\eta_1`` and not ``\alpha(0) = \eta_1``. That is how the pre-0.5
-`AdamOptimizerWithDecay` counted — it incremented `o.step` before `update!` — and how
-`DecayingStatic` counts, because `GeometricOptimizers.solve!` calls `increase_iteration_number!`
-before `solver_step!`. Reading before incrementing put every step of a run one place early in the
-schedule, which `test/adam_optimizer_with_decay.jl` upstream asserts does not happen.
+The step increments the iteration number *before* it reads the step size, so the first step of a run
+takes ``\alpha(1) = \gamma\eta_1`` and not ``\alpha(0) = \eta_1``. That is how `DecayingStatic`
+counts, because `GeometricOptimizers.solve!` calls `increase_iteration_number!` before
+`solver_step!`.
 """
-function schedule_starts_at_one(; n_epochs = 100, η₁ = 1e-2, η₂ = 1e-6)
-    method = AdamOptimizerWithDecay(n_epochs, Float64; η₁ = η₁, η₂ = η₂)
-    o = Optimizer(NetworkParameters((weight = zeros(2, 2),)); method...)
-
+function schedule_starts_at_one(T; n_epochs = 100, η₁ = 1e-2, η₂ = 1e-6)
     γ = exp(log(η₂ / η₁) / n_epochs)
-    @test o.step_size isa DecayingStatic
-    @test o.iterations == 0
-
-    for t in 1:4
-        o.iterations += 1
-        @test GeometricMachineLearning._current_step_size(o, o.iterations) ≈ η₁ * γ^t
-    end
-
-    # and the same thing through the public entry point: one step of a Euclidean parameter with a
-    # gradient of `1` moves it by `α₁ / (√1 + δ) ≈ α₁`, so the distance travelled reports the α used
-    ps = NetworkParameters((weight = zeros(2, 2),))
-    opt = Optimizer(ps; method...)
-    optimization_step!(opt, GlobalSection(ps), ps, (weight = ones(2, 2),))
-    @test opt.iterations == 1
+
+    # One step of a Euclidean parameter with a gradient of `1` moves it by `α₁ / (√1 + δ) ≈ α₁`, so
+    # the distance travelled reports the α used.
+    ps = NetworkParameters((weight = zeros(T, 2, 2),))
+    opt = NetworkOptimizer(ps; AdamOptimizerWithDecay(n_epochs; η₁, η₂)...)
+    @test opt.optimizer.linesearch isa DecayingStatic
+    @test opt.optimizer.state.iterations == 0
+    optimization_step!(ps, opt, NetworkParameters((weight = ones(T, 2, 2),)))
+    @test opt.optimizer.state.iterations == 1
+    @test eltype(ps.weight) === T
     @test abs(ps.weight[1, 1]) ≈ η₁ * γ rtol = 1e-6
 end
 
 train_network()
 train_manifold_network()
-schedule_starts_at_one()
+@testset "the decay schedule starts at t = 1 ($T)" for T in REAL_ELTYPES
+    schedule_starts_at_one(T)
+end
diff --git a/test/optimizers/gml_gradient_dispatch.jl b/test/optimizers/gml_gradient_dispatch.jl
index 85a9bcb6c..1862793ca 100644
--- a/test/optimizers/gml_gradient_dispatch.jl
+++ b/test/optimizers/gml_gradient_dispatch.jl
@@ -1,113 +1,39 @@
-# What `_GMLGradient` is called on, and by which method.
+# What the training step sees of a gradient.
 #
-# `_GMLGradient` is a `GeometricOptimizers.Gradient`, so its functor methods share a method table with
-# upstream's. Two of them overlap on a shape this package actually produces, and the overlap is an
-# *ambiguity* rather than a choice — which is a run-time error several frames into a solve, not a
-# signature a reader would notice. So it is asserted here rather than left to whichever call path
-# happens to reach it.
+# `NetworkOptimizer` steps the whole parameter set with one `GeometricOptimizers` cache, so the
+# gradient the cache builds its direction from is the gradient of the whole set: a parameter set of
+# the network's shape, with the Riemannian gradient on a manifold leaf and the Euclidean gradient on
+# a plain one.
 
 using GeometricMachineLearning
-using GeometricOptimizers: StiefelManifold, GrassmannManifold, rgrad
+using GeometricOptimizers: StiefelManifold, GrassmannManifold, rgrad, PrecomputedGradient
 using NeuralNetworkParameters: NetworkParameters
 using Test
 import Random
 
+include("../helpers/eltypes.jl")
+
 Random.seed!(1234)
 
 GML = GeometricMachineLearning
 
-# A bare `Manifold`. `_GMLGradient{T} <: GeometricOptimizers.Gradient{T}` and `Manifold{T} <:
-# AbstractMatrix{T}`, so `(::_GMLGradient{T})(::AbstractArray{T})` and upstream's
-# `(::Gradient{T})(::Manifold{T})` are ambiguous here: neither is more specific. Without a method of
-# its own this call is `MethodError: ... is ambiguous`.
-#
-# The value is this package's answer and not upstream's. Upstream's body is
-# `rgrad(x, reshape(grad(vec(x)), size(x)...))`, which evaluates an *inner* gradient — and a
-# `_GMLGradient` has none, since the whole point of it is that the Euclidean gradient was computed
-# already and is being carried in `dp`.
-@testset "a bare Manifold reaches this package's method, not upstream's" begin
+@testset "the gradient of a whole network, a manifold leaf beside a plain one ($T)" for T in REAL_ELTYPES
     for M in (StiefelManifold, GrassmannManifold)
-        Y = rand(M, 4, 2)
-        dp = randn(4, 2)
-        g = GML._GMLGradient{Float64, typeof(dp)}(dp)
-
-        @test g(Y) == rgrad(Y, dp)
-        @test which(g, Tuple{typeof(Y)}).module === GeometricMachineLearning
+        ps = NetworkParameters((L1 = (weight = rand(M{T}, 4, 2),),
+            L2 = (W = randn(T, 3, 3), b = randn(T, 3))))
+        dp = NetworkParameters((L1 = (weight = randn(T, 4, 2),),
+            L2 = (W = randn(T, 3, 3), b = randn(T, 3))))
+        out = PrecomputedGradient(ps, dp)(ps)
+
+        @test out isa NetworkParameters
+        @test keys(out) == keys(ps)
+        @test out.L1.weight == rgrad(ps.L1.weight, dp.L1.weight)
+        @test out.L2.W == dp.L2.W
+        @test out.L2.b == dp.L2.b
+        @test eltype(out.L1.weight) === eltype(out.L2.W) === eltype(out.L2.b) === T
     end
 end
 
-# A bare `AbstractVector`. `_GMLGradient{T} <: GeometricOptimizers.Gradient{T}` and `SimpleSolvers`
-# has `(::Gradient{T})(::AbstractVector{T})`, so without a method of its own,
-# `(::_GMLGradient{T})(::AbstractArray{T})` and that one are equally specific on a `Vector` --
-# the call is `MethodError: ... is ambiguous`, same class of defect as the `Manifold` case above.
-@testset "a bare AbstractVector reaches this package's method, not upstream's" begin
-    dp = randn(3)
-    g = GML._GMLGradient{Float64, typeof(dp)}(dp)
-
-    @test g(randn(3)) === dp
-    @test which(g, Tuple{Vector{Float64}}).module === GeometricMachineLearning
-end
-
-# The two shapes the optimizer actually hands it, so that the method above is not the only one
-# covered and a later narrowing of either shows up here.
-@testset "a wrapped layer and a plain array" begin
-    dp = randn(3, 3)
-    g = GML._GMLGradient{Float64, typeof(dp)}(dp)
-    # a plain array leaf: the gradient is already Euclidean, so it passes through
-    @test g(randn(3, 3)) === dp
-
-    ps = NetworkParameters((weight = rand(StiefelManifold, 4, 2),))
-    dps = (weight = randn(4, 2),)
-    gps = GML._GMLGradient{Float64, typeof(dps)}(dps)
-    out = gps(ps)
-    # rebuilt in the shape of its *first* argument -- a container, which is what
-    # `GeometricOptimizers._copyto!` has a method for
-    @test out isa NetworkParameters
-    @test out.weight == rgrad(ps.weight, dps.weight)
-end
-
-# `_is_layer` is the rule for how a network is split into caches, so its boundary cases are stated
-# rather than inherited. An empty set is *zero* caches' worth and descends; `all` over an empty
-# collection would otherwise make it vacuously one.
-@testset "_is_layer at the boundary" begin
-    @test GML._is_layer((W = randn(2, 2), b = randn(2)))
-    @test GML._is_layer(NetworkParameters((W = randn(2, 2), b = randn(2))))
-    # a set whose values are branches is a subtree, and gets one cache per layer
-    @test !GML._is_layer((L1 = (W = randn(2, 2),), L2 = (W = randn(2, 2),)))
-    @test !GML._is_layer(NamedTuple())
-    @test !GML._is_layer(NetworkParameters(NamedTuple()))
-    @test !GML._is_layer(randn(2, 2))
-
-    # a layer whose weights do not share an element type is still one cache, with `T` the promotion
-    @test GML._is_layer((W = randn(Float32, 2, 2), b = randn(Float64, 2)))
-end
-
-# `_as_go_solution` is the wrap itself, and the property that makes the wrap free: it shares the leaf
-# arrays, so the optimizer step writes through to the network's own weights and nothing is copied
-# back. That is what makes this a boundary change rather than a change of behaviour, and it is the one
-# claim of this release that a reader cannot check by reading a signature.
-@testset "_as_go_solution wraps a layer and shares its arrays" begin
-    W, b = randn(2, 2), randn(2)
-    layer = (W = W, b = b)
-
-    wrapped = GML._as_go_solution(layer)
-    @test wrapped isa NetworkParameters
-    @test wrapped.W === W          # shared, not copied
-    @test wrapped.b === b
-
-    # already in the right shape: passed through untouched
-    ps = NetworkParameters(layer)
-    @test GML._as_go_solution(ps) === ps
-    Y = rand(StiefelManifold, 4, 2)
-    @test GML._as_go_solution(Y) === Y
-    v = randn(3)
-    @test GML._as_go_solution(v) === v
-
-    # a subtree is not a layer, so it is left alone for the tree branch to descend
-    tree = (L1 = (W = randn(2, 2),), L2 = (W = randn(2, 2),))
-    @test GML._as_go_solution(tree) === tree
-end
-
 # `_get_contents` unwraps whatever shape the pullback handed back. Six methods, because a reverse pass
 # produces a container or the bare `NamedTuple` it wraps, and `Zygote` may hand either back bare, in a
 # one-tuple, or in a one-element vector. Each is exercised here: splitting a union into a method per
diff --git a/test/optimizers/gradient_optimizer.jl b/test/optimizers/gradient_optimizer.jl
index 376e946bd..5837d6f02 100644
--- a/test/optimizers/gradient_optimizer.jl
+++ b/test/optimizers/gradient_optimizer.jl
@@ -1,4 +1,5 @@
 using GeometricMachineLearning
+using GeometricOptimizers: optimization_step!
 using Zygote
 using Test
 using LinearAlgebra: norm
@@ -15,11 +16,10 @@ function gradient_optimizer_euclidean(N; n_steps = 20, step_size = 1e-3)
     A = randn(N, N)
     ps = NetworkParameters((A = A,))
     loss1 = loss(ps)
-    o = Optimizer(GradientMethod(), ps; step_size = step_size)
+    o = NetworkOptimizer(ps; algorithm = GradientMethod(), linesearch = step_size)
     for _ in 1:n_steps
         ∇L = Zygote.gradient(loss, ps)[1]
-        λY = GlobalSection(ps)
-        optimization_step!(o, λY, ps, ∇L)
+        optimization_step!(ps, o, ∇L)
     end
     loss2 = loss(ps)
     @test loss1 > loss2
@@ -35,11 +35,10 @@ function gradient_optimizer_stiefel(N, n; n_steps = 20, step_size = 1e-3)
     Y = rand(StiefelManifold, N, n)
     ps = NetworkParameters((Y = Y,))
     loss1 = loss(ps)
-    o = Optimizer(GradientMethod(), ps; step_size = step_size)
+    o = NetworkOptimizer(ps; algorithm = GradientMethod(), linesearch = step_size)
     for _ in 1:n_steps
         ∇L = Zygote.gradient(loss, ps)[1]
-        λY = GlobalSection(ps)
-        optimization_step!(o, λY, ps, ∇L)
+        optimization_step!(ps, o, ∇L)
     end
     loss2 = loss(ps)
     @test loss1 > loss2
diff --git a/test/optimizers/momentum_optimizer.jl b/test/optimizers/momentum_optimizer.jl
index 0351140a1..86ba0c249 100644
--- a/test/optimizers/momentum_optimizer.jl
+++ b/test/optimizers/momentum_optimizer.jl
@@ -1,4 +1,5 @@
 using GeometricMachineLearning
+using GeometricOptimizers: optimization_step!
 using Zygote
 using Test
 using LinearAlgebra: norm, I
@@ -16,11 +17,10 @@ growing one that never saturates, which is what this test rules out.
 """
 function momentum_matches_geometric_sum(; α = 0.9, η = 0.1, n_steps = 10)
     ps = NetworkParameters((A = zeros(2, 2),))
-    o = Optimizer(MomentumMethod(α), ps; step_size = η)
-    λY = GlobalSection(ps)
+    o = NetworkOptimizer(ps; algorithm = MomentumMethod(; α), linesearch = η)
     previous = 0.0
     for t in 1:n_steps
-        optimization_step!(o, λY, ps, (A = ones(2, 2),))
+        optimization_step!(ps, o, NetworkParameters((A = ones(2, 2),)))
         step = previous - ps.A[1, 1]
         previous = ps.A[1, 1]
         @test step ≈ η * (1 - α^t) / (1 - α)
@@ -38,10 +38,9 @@ function momentum_optimizer_stiefel(N, n; n_steps = 50, step_size = 1e-3, α = 0
     loss(ps) = norm(ps.Y * ps.Y' - B)^2
     ps = NetworkParameters((Y = rand(StiefelManifold, N, n),))
     loss1 = loss(ps)
-    o = Optimizer(MomentumMethod(α), ps; step_size = step_size)
-    λY = GlobalSection(ps)
+    o = NetworkOptimizer(ps; algorithm = MomentumMethod(; α), linesearch = step_size)
     for _ in 1:n_steps
-        optimization_step!(o, λY, ps, Zygote.gradient(loss, ps)[1])
+        optimization_step!(ps, o, Zygote.gradient(loss, ps)[1])
     end
     @test loss(ps) < loss1
     @test ps.Y' * ps.Y ≈ I
diff --git a/test/optimizers/optimization_step.jl b/test/optimizers/optimization_step.jl
index 781d625a0..8559dd457 100644
--- a/test/optimizers/optimization_step.jl
+++ b/test/optimizers/optimization_step.jl
@@ -1,57 +1,85 @@
 using GeometricMachineLearning, Test, LinearAlgebra, KernelAbstractions
-using AbstractNeuralNetworks: AbstractExplicitLayer
 import GeometricOptimizers
+import GeometricOptimizers: optimization_step!
 import GeometricMachineLearning: NeuralNetwork
 import Random
 
+include("../helpers/eltypes.jl")
+
 Random.seed!(1234)
 
 function optimization_step_test(N, n, T)
     model = Chain(StiefelLayer(N, n), Dense(N, N, tanh))
     ps = NeuralNetwork(model, KernelAbstractions.CPU(), T).params
-    # gradient 
-    dx = (L1 = (weight = rand(Float32, N, n),),
-        L2 = (W = rand(Float32, N, N), b = rand(Float32, N)))
-    m = AdamOptimizer()
-    # randomize the cache!
-    o = Optimizer(m, ps)
+    # gradient
+    dx = NetworkParameters((L1 = (weight = rand(T, N, n),),
+        L2 = (W = rand(T, N, N), b = rand(T, N))))
+    o = NetworkOptimizer(ps; algorithm = Adam())
 
     ps2 = deepcopy(ps)
-    λY = GlobalSection(ps)
-    optimization_step!(o, λY, ps, dx)
-    @test typeof(ps[1].weight) <: StiefelManifold
+    optimization_step!(ps, o, dx)
+    @test ps[1].weight isa StiefelManifold{T}
     for (layers1, layers2) in zip(values(ps), values(ps2))
         for key in keys(layers1)
+            @test eltype(layers1[key]) === T
             @test norm(layers1[key] - layers2[key]) > T(1.0f-6)
         end
     end
 end
 
-N_max = 10
-T = Float32
-for N in 4:N_max
-    for n in 1:N
-        optimization_step_test(N, n, T)
+@testset "one Adam step moves every leaf of the network ($T)" for T in REAL_ELTYPES
+    for N in 4:10
+        for n in 1:N
+            optimization_step_test(N, n, T)
+        end
     end
 end
 
-# The regression net for the branch order in `_make_optimizer_cache`/`_make_optimizer_state`: a
-# `NetworkParameters` is a tree to descend into, so it is recognised *structurally*, before the
-# capability question `_use_go_cache` asks. The shape that follows is one `GeometricOptimizers` cache
-# per *layer* -- not one for the root, which is what putting the capability question first would give
-# the moment `GeometricOptimizers` adds the container to `OptimizerSolution`, and not one per weight,
-# which is what hoisting the `NamedTuple` branch as well would give.
-@testset "one cache and one state per layer, keyed by the network's layers" begin
+# One cache and one state cover the whole network: their solution is a parameter set with the
+# network's layers as its keys, not one cache per layer.
+@testset "one cache and one state over the whole network ($T)" for T in REAL_ELTYPES
     model = Chain(StiefelLayer(6, 3), Dense(6, 6, tanh))
-    nn = NeuralNetwork(model, KernelAbstractions.CPU(), Float32)
-    o = Optimizer(AdamOptimizer(), nn)
+    nn = NeuralNetwork(model, KernelAbstractions.CPU(), T)
+    o = NetworkOptimizer(nn; algorithm = Adam())
+
+    @test o.optimizer.cache isa GeometricOptimizers.OptimizerCache{T}
+    @test o.optimizer.state isa GeometricOptimizers.OptimizerState{T}
+    for x in (GeometricOptimizers.solution(o.optimizer.cache),
+        GeometricOptimizers.solution(o.optimizer.state))
+        @test x isa NetworkParameters
+        @test keys(x) == keys(GeometricMachineLearning.params(nn))
+        @test x.L1.weight isa StiefelManifold{T}
+        @test eltype(x.L2.W) === eltype(x.L2.b) === T
+    end
+end
+
+# A function barrier with concrete argument types, which runs the call once before it measures.
+function allocations(f::F, a::A, b::B, c::C) where {F, A, B, C}
+    f(a, b, c)
+    @allocated f(a, b, c)
+end
+
+# The training step is the hot path of `train!`. It is inferred, and on a Euclidean layer it allocates
+# nothing. With a Stiefel leaf `GeometricOptimizers`' own step allocates (about 2 to 4 kB here), and
+# the step through `NetworkOptimizer` allocates the same bytes, within 32.
+@testset "the step through NetworkOptimizer is inferred and allocates nothing of its own ($T)" for T in REAL_ELTYPES
+    for algorithm in (GradientMethod(), MomentumMethod(), Adam())
+        ps = NetworkParameters((L1 = (W = rand(T, 3, 3), b = rand(T, 3)),))
+        dp = NetworkParameters((L1 = (W = rand(T, 3, 3), b = rand(T, 3)),))
+        o = NetworkOptimizer(ps; algorithm)
+        @inferred optimization_step!(ps, o, dp)
+        @test allocations(optimization_step!, ps, o, dp) == 0
+        @test eltype(ps.L1.W) === eltype(ps.L1.b) === T
 
-    for tree in (o.cache, o.state)
-        @test tree isa NamedTuple
-        @test keys(tree) == keys(GeometricMachineLearning.params(nn))
+        ps = NetworkParameters((L1 = (W = rand(T, 3, 3), b = rand(T, 3)),
+            L2 = (Y = rand(StiefelManifold{T}, 4, 2),)))
+        dp = NetworkParameters((L1 = (W = rand(T, 3, 3), b = rand(T, 3)),
+            L2 = (Y = rand(T, 4, 2),)))
+        o = NetworkOptimizer(ps; algorithm)
+        @inferred optimization_step!(ps, o, dp)
+        @test abs(allocations(optimization_step!, ps, o, dp) -
+                  allocations(optimization_step!, ps, o.optimizer, dp)) ≤ 32
+        @test ps.L2.Y isa StiefelManifold{T}
     end
-    @test o.cache.L1 isa GeometricOptimizers.OptimizerCache
-    @test o.cache.L2 isa GeometricOptimizers.OptimizerCache
-    @test o.state.L1 isa GeometricOptimizers.OptimizerState
-    @test o.state.L2 isa GeometricOptimizers.OptimizerState
+    @test allocations((a, b, c) -> collect(a), 1:3, 1, 1) > 0
 end
diff --git a/test/optimizers/optimizer.jl b/test/optimizers/optimizer.jl
new file mode 100644
index 000000000..08dc00fb3
--- /dev/null
+++ b/test/optimizers/optimizer.jl
@@ -0,0 +1,51 @@
+using GeometricMachineLearning
+using GeometricMachineLearning: params, convert_input_and_batch_indices_to_array
+using Test
+import Random
+import Zygote
+
+include("../helpers/eltypes.jl")
+
+# One epoch of one minibatch is one gradient step at `p₀`, so the result is `p₀ - η∇L(p₀)` with the
+# `algorithm` and the `linesearch` the constructor was given. The reference gradient is taken on the
+# minibatch `train!` draws, drawn again from the same seed. The bound is 4 eps(T), for the rounding of
+# one step.
+@testset "NetworkOptimizer takes algorithm and linesearch ($T)" for T in REAL_ELTYPES
+    Random.seed!(1)
+    x, y = randn(T, 2, 8), randn(T, 2, 8)
+    dl = DataLoader(x, y; suppress_info = true)
+    model = Chain(Dense(2, 2, identity))
+    nn = NeuralNetwork(model, T)
+    p₀ = deepcopy(params(nn))
+    η = T(0.3)
+
+    Random.seed!(2)
+    batch = Batch(8)
+    minibatches = batch(dl)
+    @test length(minibatches) == 1
+    input, output = convert_input_and_batch_indices_to_array(dl, batch, only(minibatches))
+    g = only(Zygote.gradient(ps -> FeedForwardLoss()(model, ps, input, output), p₀))
+
+    opt = NetworkOptimizer(nn; algorithm = GradientMethod(), linesearch = η)
+    Random.seed!(2)
+    train!(nn, opt, dl, batch, 1; loss = FeedForwardLoss(), show_progress = false)
+    for k in (:W, :b)
+        @test params(nn).L1[k] isa Array{T}
+        @test params(nn).L1[k] ≈ p₀.L1[k] .- η .* g.L1[k] rtol = 4eps(T)
+        @test params(nn).L1[k] != p₀.L1[k]
+    end
+
+    # the same run with the optimizer built from the parameter set
+    nn₂ = NeuralNetwork(model, model, deepcopy(p₀), CPU())
+    opt₂ = NetworkOptimizer(params(nn₂); algorithm = GradientMethod(), linesearch = η)
+    Random.seed!(2)
+    train!(nn₂, opt₂, dl, batch, 1; loss = FeedForwardLoss(), show_progress = false)
+    @test params(nn₂) == params(nn)
+end
+
+@testset "AdamOptimizerWithDecay splats into the keywords" begin
+    nn = NeuralNetwork(Chain(Dense(2, 2, identity)), Float32)
+    opt = NetworkOptimizer(nn; AdamOptimizerWithDecay(100)...)
+    @test opt.optimizer.method isa Adam{Float32}
+    @test opt.optimizer.linesearch isa DecayingStatic{Float32}
+end
diff --git a/test/optimizers/psd_optim.jl b/test/optimizers/psd_optim.jl
index 8afad17a7..df16904e2 100644
--- a/test/optimizers/psd_optim.jl
+++ b/test/optimizers/psd_optim.jl
@@ -1,4 +1,5 @@
 using GeometricMachineLearning, LinearAlgebra, KernelAbstractions, Test, Zygote, Random
+using GeometricOptimizers: optimization_step!
 
 Random.seed!(1234)
 
@@ -26,7 +27,7 @@ A = [0.06476993260924702 0.8369280855305259 0.6245358125914054 0.140729967064923
 """
 This tests if the optimizers can find the optimal PSD solution.
 """
-function svd_test(A, n, train_steps = 1000, tol = 1e-1; retraction = cayley)
+function svd_test(A, n, train_steps = 1000, tol = 1e-1; retraction = Cayley())
     # Seeded here and not once at the top of the file, for the reason given at the same place in
     # `svd_optim.jl`: `svd_test` is called twice, and with a single top-level seed the second call
     # starts from whatever RNG state the first one happened to leave behind, which makes the
@@ -45,9 +46,9 @@ function svd_test(A, n, train_steps = 1000, tol = 1e-1; retraction = cayley)
     model = Chain(PSDLayer(2*N, 2*n), PSDLayer(2*n, 2*N))
     ps = NeuralNetwork(model, CPU(), Float64).params
 
-    o₁ = Optimizer(GradientMethod(), ps; retraction = retraction, step_size = 0.01)
-    o₂ = Optimizer(MomentumMethod(), ps; retraction = retraction, step_size = 0.01)
-    o₃ = Optimizer(Adam(), ps; retraction = retraction, step_size = 0.01)
+    o₁ = NetworkOptimizer(ps; algorithm = GradientMethod(), retraction, linesearch = 0.01)
+    o₂ = NetworkOptimizer(ps; algorithm = MomentumMethod(), retraction, linesearch = 0.01)
+    o₃ = NetworkOptimizer(ps; algorithm = Adam(), retraction, linesearch = 0.01)
 
     U₁, Ũ₁, err₁ = train_network!(o₁, model, deepcopy(ps), A, train_steps, tol)
     U₂, _, err₂ = train_network!(o₂, model, deepcopy(ps), A, train_steps, tol)
@@ -63,19 +64,18 @@ function svd_test(A, n, train_steps = 1000, tol = 1e-1; retraction = cayley)
     @test norm((err₃ - err_best)/err_best) < tol
 end
 
-function train_network!(o::Optimizer, model::Chain, ps::NetworkParameters,
+function train_network!(o::NetworkOptimizer, model::Chain, ps::NetworkParameters,
         A::AbstractMatrix, train_steps, tol)
     error(ps) = norm(A - model(A, ps))
 
     for _ in 1:train_steps
         dx = Zygote.gradient(error, ps)[1]
-        λY = GlobalSection(ps)
-        optimization_step!(o, λY, ps, dx)
+        optimization_step!(ps, o, dx)
         #println(error(ps))
     end
     ps[1].weight, ps[2].weight, error(ps)
 end
 
-for retraction in (geodesic, cayley)
+for retraction in (Geodesic(), Cayley())
     svd_test(A, 4, retraction = retraction)
 end
diff --git a/test/optimizers/step_size_element_type.jl b/test/optimizers/step_size_element_type.jl
index a2a13658a..cb7e848f4 100644
--- a/test/optimizers/step_size_element_type.jl
+++ b/test/optimizers/step_size_element_type.jl
@@ -1,52 +1,33 @@
-# `_leaf_optim_step!` scales `direction(cache)` by `T(step_size)`, matching the three
-# `_euclidean_update!` methods a few lines below it. The funnel (`_current_step_size`) always
-# hands a `Float64`, no matter what `T` the parameters are, so converting first is what keeps a
-# `Float32` layer scaled at `Float32` precision instead of `Float64` precision rounded back on
-# write. See `CHANGELOG.md` for the fix this guards.
+# The step scales in the parameters' own element type. `NetworkOptimizer` hands `linesearch` to
+# `GeometricOptimizers.TrainingOptimizer`, which converts it once to the element type of the
+# parameters, so a `Float64` step size on a `Float32` network gives the same step as the same value
+# rounded to `Float32` first.
 #
-# The test is invariance rather than a fixed expected value: the leaf step must give the *same*
-# result whether the funnel hands it a `Float64` step size or one already rounded to `T`. Only
-# `T = Float32` can tell the two calls apart -- `T(step_size)` is a no-op when `T` is already
-# `Float64`, so a `Float64` combination holds whatever the conversion does and checks nothing.
-# `Float64` is therefore not looped over here.
-#
-# The `seed = 2` and the `step_size = 0.1` are chosen rather than arbitrary. Whether a `Float64`-
-# and a `Float32`-scaled direction round to different bit patterns depends on the actual values,
-# so for most seeds the two calls agree however the step size is scaled, and the test then passes
-# without discriminating. With these values, and with the conversion dropped, both `Adam` and
-# `MomentumMethod` separate by `2.9802322f-8` -- one `Float32` ULP at this magnitude -- and each
-# method is checked on its own.
+# Only `T = Float32` can tell the two apart -- for `Float64` the rounding is a no-op -- so `Float64`
+# is not looped over here.
 
 using GeometricMachineLearning
-using GeometricOptimizers
+using GeometricOptimizers: optimization_step!
 using Test
 import Random
 
-GML = GeometricMachineLearning
-
-function test_leaf_step_matches_prerounded_step_size(method, seed::Int, step_size::Float64)
+function test_step_matches_prerounded_step_size(method, seed::Int, step_size::Float64)
     T = Float32
     Random.seed!(seed)
-    ps_a = (weight = rand(T, 6, 6),)
-    dp = (weight = rand(T, 6, 6),)
+    ps_a = NetworkParameters((L1 = (weight = rand(T, 6, 6),),))
+    dp = NetworkParameters((L1 = (weight = rand(T, 6, 6),),))
     ps_b = deepcopy(ps_a)
-    λY_a = GML.GlobalSection(ps_a)
-    λY_b = GML.GlobalSection(ps_b)
 
-    cache_a = GML._make_optimizer_cache(method, ps_a)
-    state_a = GML._make_optimizer_state(method, ps_a)
-    cache_b = GML._make_optimizer_cache(method, ps_b)
-    state_b = GML._make_optimizer_state(method, ps_b)
+    opt_a = NetworkOptimizer(ps_a; algorithm = method, linesearch = step_size)
+    opt_b = NetworkOptimizer(ps_b; algorithm = method, linesearch = T(step_size))
+    @test typeof(opt_a.optimizer.linesearch) == typeof(opt_b.optimizer.linesearch)
 
-    # the funnel's usual `Float64` step size ...
-    GML._leaf_optim_step!(
-        cache_a, state_a, dp, ps_a, λY_a, method, GeometricOptimizers.cayley, step_size)
-    # ... versus the same value pre-rounded to `T`.
-    GML._leaf_optim_step!(
-        cache_b, state_b, dp, ps_b, λY_b, method, GeometricOptimizers.cayley, T(step_size))
+    optimization_step!(ps_a, opt_a, dp)
+    optimization_step!(ps_b, opt_b, dp)
 
-    @test ps_a.weight == ps_b.weight
+    @test ps_a.L1.weight isa Matrix{T}
+    @test ps_a.L1.weight == ps_b.L1.weight
 end
 
-test_leaf_step_matches_prerounded_step_size(GeometricOptimizers.Adam(), 2, 0.1)
-test_leaf_step_matches_prerounded_step_size(GeometricOptimizers.MomentumMethod(0.5), 2, 0.1)
+test_step_matches_prerounded_step_size(Adam(), 2, 0.1)
+test_step_matches_prerounded_step_size(MomentumMethod(; α = 0.5), 2, 0.1)
diff --git a/test/optimizers/structured_array_parameters.jl b/test/optimizers/structured_array_parameters.jl
index 4ce1a08d3..489f2b51f 100644
--- a/test/optimizers/structured_array_parameters.jl
+++ b/test/optimizers/structured_array_parameters.jl
@@ -11,7 +11,7 @@ Every optimizer method has to work with the structured matrix types GML uses as
 attention) and `LowerTriangular`/`UpperTriangular` (volume-preserving feedforward layers).
 
 These have their own storage and no `setindex!`, so they need a type-preserving `similar` and
-elementwise `_add!`/`_rac!`/`_square!`/`_div!`/`_rmul!` written on the free parameters; without them
+elementwise operations written on the free parameters; without them
 the optimizer cache cannot even be allocated. Those live in GeometricOptimizers, which owns the
 types — see `VectorStorageMatrix` there. GML used to carry its own copies of both the types and the
 methods, in `src/optimizers/go_bridges.jl`.
@@ -20,19 +20,19 @@ function optimizer_runs(architecture, batch, input_dim; T = Float64, n_epochs =
     nn_and_dl() = (NeuralNetwork(architecture, T),
         DataLoader(rand(T, input_dim, 20, 5); suppress_info = true))
 
-    for method in (AdamOptimizer(), MomentumOptimizer(), GradientOptimizer())
+    for method in (Adam(), MomentumMethod(), GradientMethod())
         nn, dl = nn_and_dl()
-        o = Optimizer(method, nn)
-        loss_array = o(nn, dl, batch, n_epochs; show_progress = false)
+        o = NetworkOptimizer(nn; algorithm = method)
+        loss_array = train!(nn, o, dl, batch, n_epochs; show_progress = false)
         @test length(loss_array) == n_epochs
         @test all(isfinite, loss_array)
     end
 
     # `AdamOptimizerWithDecay` is a `(algorithm, linesearch)` pairing rather than a method, so it
-    # goes in through the keyword constructor
+    # splats into the keywords
     nn, dl = nn_and_dl()
-    o = Optimizer(nn; AdamOptimizerWithDecay(n_epochs, T)...)
-    loss_array = o(nn, dl, batch, n_epochs; show_progress = false)
+    o = NetworkOptimizer(nn; AdamOptimizerWithDecay(n_epochs)...)
+    loss_array = train!(nn, o, dl, batch, n_epochs; show_progress = false)
     @test length(loss_array) == n_epochs
     @test all(isfinite, loss_array)
 end
diff --git a/test/optimizers/svd_optim.jl b/test/optimizers/svd_optim.jl
index 033819dee..42544f7eb 100644
--- a/test/optimizers/svd_optim.jl
+++ b/test/optimizers/svd_optim.jl
@@ -1,4 +1,5 @@
 using GeometricMachineLearning, LinearAlgebra, KernelAbstractions, Test, Zygote, Random
+using GeometricOptimizers: optimization_step!
 
 Random.seed!(1234)
 
@@ -25,7 +26,7 @@ A = [0.06476993260924702 0.8369280855305259 0.6245358125914054 0.140729967064923
 
 Random.seed!(1234)
 
-function svd_test(A, n, train_steps = 1000, tol = 1e-1; retraction = cayley)
+function svd_test(A, n, train_steps = 1000, tol = 1e-1; retraction = Cayley())
     # Seeded here and not once at the top of the file. `svd_test` is called twice, and with a single
     # top-level seed the second call starts from whatever RNG state the first one happened to leave
     # behind -- so the convergence assertions below turned on how much randomness the optimizer
@@ -41,9 +42,9 @@ function svd_test(A, n, train_steps = 1000, tol = 1e-1; retraction = cayley)
     model = Chain(StiefelLayer(N, n), StiefelLayer(n, N))
     ps = NeuralNetwork(model, CPU(), Float64).params
 
-    o₁ = Optimizer(GradientMethod(), ps; retraction = retraction, step_size = 0.01)
-    o₂ = Optimizer(MomentumMethod(), ps; retraction = retraction, step_size = 0.01)
-    o₃ = Optimizer(Adam(), ps; retraction = retraction, step_size = 0.01)
+    o₁ = NetworkOptimizer(ps; algorithm = GradientMethod(), retraction, linesearch = 0.01)
+    o₂ = NetworkOptimizer(ps; algorithm = MomentumMethod(), retraction, linesearch = 0.01)
+    o₃ = NetworkOptimizer(ps; algorithm = Adam(), retraction, linesearch = 0.01)
 
     U₁, Ũ₁, err₁ = train_network!(o₁, model, deepcopy(ps), A, train_steps, tol)
     U₂, _, err₂ = train_network!(o₂, model, deepcopy(ps), A, train_steps, tol)
@@ -59,18 +60,17 @@ function svd_test(A, n, train_steps = 1000, tol = 1e-1; retraction = cayley)
     @test norm((err₃ - err_best)/err_best) < tol
 end
 
-function train_network!(o::Optimizer, model::Chain, ps::NetworkParameters,
+function train_network!(o::NetworkOptimizer, model::Chain, ps::NetworkParameters,
         A::AbstractMatrix, train_steps, tol)
     error(ps) = norm(A - model(A, ps))
 
     for _ in 1:train_steps
         dx = Zygote.gradient(error, ps)[1]
-        λY = GlobalSection(ps)
-        optimization_step!(o, λY, ps, dx)
+        optimization_step!(ps, o, dx)
     end
     ps.L1.weight, ps.L2.weight, error(ps)
 end
 
-for retraction in (geodesic, cayley)
+for retraction in (Geodesic(), Cayley())
     svd_test(A, 3, retraction = retraction)
 end
diff --git a/test/quality/aqua.jl b/test/quality/aqua.jl
index 8ce09ad3d..a0fa85ed6 100644
--- a/test/quality/aqua.jl
+++ b/test/quality/aqua.jl
@@ -3,41 +3,25 @@ using ExplicitImports
 using GeometricMachineLearning
 using Test
 
-# Aqua's package-level checks. Six of the eight `Aqua.test_all` enables by default run; `ambiguities`
-# and `piracies` are switched off by name, and the reason is below rather than in the changelog
-# alone, because switching one back on turns the suite red on the spot.
+# Aqua's package-level checks. Seven of the eight `Aqua.test_all` enables by default run; `ambiguities`
+# is switched off by name, and the reason is below rather than in the changelog alone.
 #
 # `Aqua.test_all` wraps each of its checks in a `@testset` of its own and adds no enclosing one, and
 # a testset with no parent finalises as soon as it closes -- so called bare, the first failing check
 # throws a `TestSetException` and the rest never run. On the suite path the `@safetestset` in
 # `runtests.jl` is that parent. The `@testset` below is that parent when this file is run on its
-# own, and it is also what lets the two counts at the end be further assertions rather than
-# top-level `@test`s that abort the file the moment one fails.
-#
-# WHAT IS SWITCHED OFF, AND WHY IT IS NOT A `broken = true`.
-#
-# `piracies` reports 3 methods, and all 3 are genuine under Aqua's definition -- the function and
-# every argument type belong to other modules. They are not a count waiting to be triaged. All
-# three are the same thing: a three-argument functor for a *layer* type that this package does not
-# own, `AbstractNeuralNetworks`' `Dense` and `Linear`, added so that the layer accepts a 3-tensor
-# input as well as a matrix.
-#
-#     Dense{M, N, true}   on an AbstractArray{T, 3}    src/layers/resnet.jl:63
-#     Dense{M, N, false}  on an AbstractArray{T, 3}    src/layers/resnet.jl:67
-#     Linear{M, N}        on an AbstractArray{T, 3}    src/layers/resnet.jl:71
-#
-# Closing these means either upstream gaining the 3-tensor methods or this package wrapping the two
-# layer types, and both are an API change rather than a tidy-up. They are *B7* under
-# `## Open Issues` in `CHANGELOG.md` with that reasoning.
-#
-# `ambiguities` reports 1, and the same 1 whether this package is loaded alone or with everything
-# `runtests.jl` loads before this file. It is a `Dense{M, N, true}` functor against
-# `AbstractNeuralNetworks.Affine`'s, and it is benign: `Dense` is not a subtype of `Affine` and the
-# two have no common instance, so no call can reach the pair. It is the same `resnet.jl:63` method
-# as the first piracy above.
-#
-# The two settings agreeing is what makes the gate below trustworthy, and it holds because no
-# method of this package claims an argument position wholesale on `PoissonTensor`. That type is an
+# own, and it is also what lets the count at the end be a further assertion rather than a
+# top-level `@test` that aborts the file the moment it fails.
+#
+# `piracies` runs. The 3-tensor functors of `AbstractNeuralNetworks`' `Dense` and `Linear`, and the
+# `save`/`load` methods on `NeuralNetwork`, are `AbstractNeuralNetworks`' own.
+#
+# `ambiguities` stays off, and the assertion below replaces it. Aqua's check runs in a process of its
+# own and asks about whatever that process loaded, so its count moves with ArrayLayouts',
+# FillArrays', Symbolics', GeometricOptimizers', BandedMatrices' and BlockArrays' versions, and with
+# *which* of them a given process had loaded. `detect_ambiguities` on this package, in this process,
+# asks only about the methods it defines; the count is zero, and no method of this package claims an
+# argument position wholesale on `PoissonTensor`, which is what keeps it there. That type is an
 # `AbstractMatrix{T}`, so such a method meets every special-array method another package writes
 # against `AbstractMatrix`:
 #
@@ -48,76 +32,61 @@ using Test
 #       `BlockIndex` and `BandRangeType` methods BandedMatrices and BlockArrays add to
 #       `AbstractMatrix`. Neither of those two loads with this package alone, so an untyped index
 #       pair here is a collision visible only from inside the suite.
-#
-# `ambiguities` still stays off, because 1 is not 0 and Aqua's check has no way to exempt a pair.
-# The assertion below replaces it and is strictly better here: it names the number, so it fails
-# both when an ambiguity is added and when this one is closed without the comment going with it.
-#
-# Marking either check `broken = true` would leave a check that reports success while the defect
-# stands, which is the failure mode this test suite's guards exist to remove.
 @testset "Aqua" begin
-    Aqua.test_all(GeometricMachineLearning; ambiguities = false, piracies = false)
-
-    # A switched-off check detects nothing, so the count it would have reported drifts unobserved,
-    # and so does every `src` line named above. This is the gate. It fails when a piracy is added,
-    # and it fails when one is removed without the entry above and in `CHANGELOG.md` going with it.
-    @test length(Aqua.Piracy.hunt(GeometricMachineLearning)) == 3
+    Aqua.test_all(GeometricMachineLearning; ambiguities = false)
 
-    # The ambiguity count gets the same gate, and it can carry one because the single pair is
-    # between a method of this package and one of `AbstractNeuralNetworks`'. The count is the same
-    # in isolation and in the suite, and nothing outside this repository's own `[compat]` moves it
-    # -- so a red here is a regression, not an unrelated upstream upgrade and not a reordering of
-    # this suite. A count that moved with ArrayLayouts', FillArrays', Symbolics',
-    # GeometricOptimizers', BandedMatrices' and BlockArrays' versions, and with *which* of them a
-    # given process had loaded, could not tell those apart, and could not be asserted.
-    @test length(Test.detect_ambiguities(GeometricMachineLearning; recursive = true)) == 1
+    @test isempty(Test.detect_ambiguities(GeometricMachineLearning; recursive = true))
 end
 
 # `ExplicitImports` answers a question Aqua does not ask, and the reason to gate on it is not
 # tidiness: a `using` or an `import` that nothing needs keeps a dependency reachable from the module
 # file, so Aqua's `stale_deps` above sees a package in use that nothing uses. That check only means
-# something once these two are clean.
+# something once these are clean.
 #
-# Three of the seven checks run: `no_stale_explicit_imports`, `all_explicit_imports_via_owners` and
-# `no_self_qualified_accesses`. The first two were red before this file gained them -- 13 stale
-# explicit imports, and `StateVariable` imported from `GeometricSolutions`, which re-exports it,
-# rather than from `GeometricBase`, which defines it. The third already passed.
-#
-# THE OTHER FOUR, AND WHY EACH IS OFF.
+# Four of the seven checks run here: `no_stale_explicit_imports`, `all_explicit_imports_via_owners`,
+# `all_qualified_accesses_via_owners` and `no_self_qualified_accesses`. Two more run in the testset
+# below with a list of names, and `no_implicit_imports` is off:
 #
 #   no_implicit_imports              37 names arrive through a bare `using`, across thirteen packages
 #                                    -- `norm`, `@kernel`, `rrule`, `pullback` and the rest. Naming
 #                                    every one is a change to the whole module header and a judgement
 #                                    per name, not a by-product of a dead-code pass.
-#   all_explicit_imports_are_public  9 names this package imports are not marked `public` upstream
-#                                    -- `Architecture`, `AbstractExplicitLayer`, `_compute_loss`,
-#                                    `description`, `dim` among them. Each is deliberate and most
-#                                    are re-exported here; the fix is upstream declaring them, not
-#                                    this package importing less.
-#   all_qualified_accesses_via_owners  2: `GeometricOptimizers.Gradient` and
-#                                    `GeometricOptimizers.direction`, both owned by `SimpleSolvers`.
-#                                    Reaching them through `GeometricOptimizers` is how the rest of
-#                                    `src/optimizers/optimizer.jl` is written.
-#   all_qualified_accesses_are_public  24, the same class as the 9 above: `KernelAbstractions.zeros`,
-#                                    `ForwardDiff.jacobian`, `GeometricOptimizers.momentum` and so on.
-#
-# Each of the four is a real backlog item rather than a taste, and none of them is this branch's.
 #
 # WHAT THIS GATE DOES NOT CATCH, and one way it can go red on its own.
 #
-# It reads imports, never definitions. A dead *definition* re-added under `src/` leaves all three
+# It reads imports, never definitions. A dead *definition* re-added under `src/` leaves all the
 # checks green, so this closes only the import half of the pass that added it.
 #
-# `all_explicit_imports_via_owners` also has the property the `ambiguities` comment above rejects:
-# it asks *which* upstream module defines a name, so it goes red when a name moves between
-# `GeometricBase`, `GeometricOptimizers` and `AbstractNeuralNetworks` with nothing in `src/` having
-# changed -- which is exactly how `StateVariable` got here. `ExplicitImports = "1.15"` admits any
-# 1.x, so a minor release that sharpens stale detection does the same. Unlike the ambiguity count,
-# a red here names the import and the module to take it from, so it stays a gate: the fix is one
-# line and the message says which. `test_explicit_imports` opens its own `@testset`, so this call
-# needs no wrapper.
+# The two `via_owners` checks ask *which* upstream module defines a name, so they go red when a name
+# moves between `GeometricBase`, `GeometricOptimizers` and `AbstractNeuralNetworks` with nothing in
+# `src/` having changed -- which is how `StateVariable` got here. `ExplicitImports = "1.15"` admits
+# any 1.x, so a minor release that sharpens stale detection does the same. A red here names the
+# import and the module to take it from, so it stays a gate: the fix is one line and the message
+# says which. `test_explicit_imports` opens its own `@testset`, so this call needs no wrapper.
 test_explicit_imports(GeometricMachineLearning;
     no_implicit_imports = false,
     all_explicit_imports_are_public = false,
-    all_qualified_accesses_via_owners = false,
     all_qualified_accesses_are_public = false)
+
+# Every name this package reaches in `GeometricOptimizers` is public there, whether it is imported or
+# written `GeometricOptimizers.x`. The names below are the ones it reaches in *other* packages that
+# those packages do not declare public; each is deliberate, and the fix for each is upstream
+# declaring it. No `GeometricOptimizers` name is on either list.
+const NON_PUBLIC_QUALIFIED = (
+    :Initializer, :TanhActivation, :dim,                         # AbstractNeuralNetworks
+    :Jacobian, :ParameterGradient, :symbolic_parameter_gradient, # SymbolicNeuralNetworks
+    :variables,                                                  # Symbolics
+    :copyto!, :zeros,                                            # KernelAbstractions
+    :jacobian)                                                   # ForwardDiff
+const NON_PUBLIC_IMPORTS = (
+    :AbstractExplicitLayer, :Architecture, :_compute_loss, :dim, :model, # AbstractNeuralNetworks
+    :derivative,                                                         # SymbolicNeuralNetworks
+    :load, :save,                                                        # NeuralNetworkParameters
+    :_vcat, :assign_q_and_p)                       # this package, imported by `GPUArraysCoreExt`
+
+@testset "every name reached in GeometricOptimizers is public" begin
+    @test check_all_qualified_accesses_are_public(GeometricMachineLearning;
+        ignore = NON_PUBLIC_QUALIFIED) === nothing
+    @test check_all_explicit_imports_are_public(GeometricMachineLearning;
+        ignore = NON_PUBLIC_IMPORTS) === nothing
+end
diff --git a/test/quality/exports.jl b/test/quality/exports.jl
index 1be6e1300..f3972ab97 100644
--- a/test/quality/exports.jl
+++ b/test/quality/exports.jl
@@ -6,10 +6,6 @@ using Test
 # `using GeometricMachineLearning; ResidualLayer(2)` is an `UndefVarError` for a name the package
 # advertises. The allowlist below is empty, so every exported name resolves; an entry is added here
 # only with the reason a name is exported and defined nowhere.
-#
-# The assertions below test what the loaded module defines. `ext/HDF5Ext.jl` loads alongside HDF5
-# and this test does not load HDF5, so a name defined only in that extension would read as undefined
-# here. No exported name is in that position today.
 const UNDEFINED_EXPORTS = Dict{Symbol, String}()
 
 @testset "every exported name is defined" begin
@@ -23,3 +19,22 @@ const UNDEFINED_EXPORTS = Dict{Symbol, String}()
     # longer exported, has to go; otherwise the next reader trusts a reason that no longer holds.
     @test sort(setdiff(collect(keys(UNDEFINED_EXPORTS)), undefined_exports)) == Symbol[]
 end
+
+# A caller who loads both packages keeps every name of both: each name the two export is one binding,
+# so neither `using` shadows the other, and `Optimizer` is `GeometricOptimizers`' solver.
+@testset "using GeometricOptimizers and GeometricMachineLearning together" begin
+    m = Module()
+    Core.eval(m, :(using GeometricOptimizers, GeometricMachineLearning))
+    @test Core.eval(m, :(Optimizer === GeometricOptimizers.Optimizer))
+
+    GO = Core.eval(m, :GeometricOptimizers)
+    shared = intersect(names(GO), names(GeometricMachineLearning))
+    @test :Adam in shared
+    @test filter(n -> getfield(GO, n) !== getfield(GeometricMachineLearning, n), shared) ==
+          Symbol[]
+
+    # no alias names a method under a second name
+    for alias in (:GradientOptimizer, :MomentumOptimizer, :AdamOptimizer)
+        @test !isdefined(GeometricMachineLearning, alias)
+    end
+end
diff --git a/test/runtests.jl b/test/runtests.jl
index d4fff0fa9..ad5816b25 100644
--- a/test/runtests.jl
+++ b/test/runtests.jl
@@ -51,7 +51,7 @@ if "core" in GROUPS
     @safetestset "_norm keeps the element type of its argument" include("utils.jl")
     @safetestset "Symplectic Euler and variational midpoint losses" include("loss/training_method_losses.jl")
     @safetestset "Test NetworkLoss + Optimizer" include("integration/losses_and_optimization.jl")
-    @safetestset "ReducedLoss through the Optimizer functor" include("integration/reduced_loss_optimization.jl")
+    @safetestset "ReducedLoss through train!" include("integration/reduced_loss_optimization.jl")
     @safetestset "Optimizer #2" include("optimizers/optimization_step.jl")
     @safetestset "Optimizer #3" include("optimizers/svd_optim.jl")
     @safetestset "Optimizer #4" include("optimizers/psd_optim.jl")
@@ -59,8 +59,8 @@ if "core" in GROUPS
     @safetestset "Gradient optimizer tests" include("optimizers/gradient_optimizer.jl")
     @safetestset "Momentum optimizer tests" include("optimizers/momentum_optimizer.jl")
     @safetestset "Optimizers with structured (non-manifold) weights" include("optimizers/structured_array_parameters.jl")
-    @safetestset "_GMLGradient dispatch" include("optimizers/gml_gradient_dispatch.jl")
-    @safetestset "The GO-native leaf step scales in the parameter's own element type" include("optimizers/step_size_element_type.jl")
+    @safetestset "The gradient a training step sees" include("optimizers/gml_gradient_dispatch.jl")
+    @safetestset "The training step scales in the parameters' own element type" include("optimizers/step_size_element_type.jl")
     @safetestset "_custom_mul: NetworkParameters gradient structure" include("layers/double_multiplication_network_parameters_gradient.jl")
     @safetestset "Symplectic attention: NetworkParameters gradient structure" include("layers/symplectic_attention_network_parameters_gradient.jl")
     @safetestset "map_to_cpu" include("map_to_cpu.jl")
@@ -68,12 +68,14 @@ if "core" in GROUPS
     @safetestset "HDF5 save/load for GML special array types" include("integration/hdf5_support.jl")
     @safetestset "Test data loader for q and p data" include("data_loader/batch_data_loader_qp_test.jl")
     @safetestset "Test the data loader in combination with optimization_step!" include("data_loader/data_loader_optimization_step.jl")
-    @safetestset "Optimizer functor with data loader for Adam" include("data_loader/optimizer_functor_with_adam.jl")
+    @safetestset "One epoch of train! with Adam on a data loader" include("data_loader/optimizer_functor_with_adam.jl")
     @safetestset "Test data loader for a tensor (q and p data)" include("data_loader/draw_batch_for_tensor_test.jl")
     @safetestset "Batch functor(s)" include("data_loader/batch_functor.jl")
     @safetestset "DataLoader for input and output" include("data_loader/data_loader_for_input_and_output.jl")
     @safetestset "An unexpected `autoencoder` keyword throws rather than returning nothing" include("data_loader/autoencoder_keyword.jl")
     @safetestset "A Float32 network's training history stays Float32" include("data_loader/training_history_eltype.jl")
+    @safetestset "train!: convergence, resume, history type, n_epochs" include("data_loader/train.jl")
+    @safetestset "NetworkOptimizer" include("optimizers/optimizer.jl")
     @safetestset "The minibatch index set is concrete and grows linearly" include("data_loader/batch_index_set.jl")
     @safetestset "Classifier accuracy" include("data_loader/accuracy.jl")
     @safetestset "Data loader docstring examples" include("integration/docstrings/data_loader.jl")
@@ -88,6 +90,7 @@ end
 if "slow" in GROUPS
     @safetestset "Lagrangian Neural Network" include("architectures/lagrangian_neural_network_tests.jl")
     @safetestset "Test symplecticity of the sympnet upscaling layer" include("layers/sympnet_upscaling.jl")
+    @safetestset "train! trains every architecture of the manual" include("integration/train_manual_architectures.jl")
 end
 if "doctests" in GROUPS
     @safetestset "Doctests" include("quality/doctests.jl")

From e8ff170de60ea1e96c5d85c8838b2d650e287ea3 Mon Sep 17 00:00:00 2001
From: Michael Kraus <michael.kraus@ipp.mpg.de>
Date: Mon, 12 Oct 2026 18:02:39 +0200
Subject: [PATCH 02/11] Rename LowerTriangular and UpperTriangular as GO 0.9
 does

StrictlyLowerTriangular and StrictlyUpperTriangular replace the two names in
src, test, docs/src and scripts, and GML re-exports them under the new names.
The CHANGELOG names the rename.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 CHANGELOG.md                                   |  2 ++
 docs/src/arrays/tensors.md                     |  6 +++---
 docs/src/index.md                              |  2 +-
 .../layers/volume_preserving_feedforward.md    |  4 ++--
 ...volume_preserving_transformer_rigid_body.md |  4 ++--
 src/GeometricMachineLearning.jl                |  5 ++---
 .../kernel_ad_routines/mat_tensor_mul.jl       | 14 ++++++++------
 src/kernels/mat_tensor_mul.jl                  | 18 ++++++++++--------
 src/layers/volume_preserving_feedforward.jl    | 16 ++++++++--------
 test/arrays/triangular.jl                      |  8 ++++----
 test/integration/changebackend_tests.jl        |  4 ++--
 test/integration/hdf5_support.jl               | 11 ++++++-----
 test/layers/volume_preserving_feedforward.jl   |  6 +++---
 test/map_to_cpu.jl                             | 10 +++++-----
 test/optimizers/structured_array_parameters.jl |  6 +++---
 15 files changed, 61 insertions(+), 55 deletions(-)

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 14ac6061f..a9d8021da 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -25,6 +25,8 @@ breaking release).
 
 ### Changed (breaking)
 
+- **`LowerTriangular` and `UpperTriangular` are renamed `StrictlyLowerTriangular` and `StrictlyUpperTriangular`**, as in GeometricOptimizers 0.9, and are re-exported under the new names. A file saved with the old names no longer loads through the registry. The rigid-body tutorial registers the two old names for its committed files.
+
 - **The `[compat]` bounds move to the versions the training API needs**: GeometricOptimizers `"0.9"`, AbstractNeuralNetworks `"0.9.1"`, NeuralNetworkParameters `"0.5"`, SymbolicNeuralNetworks `"0.9"`, GeometricBase `"0.15"`, GeometricEquations `"0.21.5"` and GeometricSolutions `"0.6.6"`. HDF5 leaves `[compat]` and `[weakdeps]`.
 
 ### Added
diff --git a/docs/src/arrays/tensors.md b/docs/src/arrays/tensors.md
index a7e0d1eac..0a8d65a9d 100644
--- a/docs/src/arrays/tensors.md
+++ b/docs/src/arrays/tensors.md
@@ -1,6 +1,6 @@
 # Tensors in `GeometricMachineLearning`
 
-We typically store training data as *tensors with three axes* in `GeometricMachineLearning`. This allows for a parallel computation of matrix products, also for the special arrays such as [`LowerTriangular`](@extref GeometricOptimizers GeometricOptimizers.LowerTriangular), [`UpperTriangular`](@extref GeometricOptimizers GeometricOptimizers.UpperTriangular), [`SymmetricMatrix`](@extref GeometricOptimizers GeometricOptimizers.SymmetricMatrix) and [`SkewSymMatrix`](@extref GeometricOptimizers GeometricOptimizers.SkewSymMatrix) and objects of [`Manifold`](@extref GeometricOptimizers GeometricOptimizers.Manifold) type such as the [`StiefelManifold`](@extref GeometricOptimizers GeometricOptimizers.StiefelManifold). 
+We typically store training data as *tensors with three axes* in `GeometricMachineLearning`. This allows for a parallel computation of matrix products, also for the special arrays such as [`StrictlyLowerTriangular`](@extref GeometricOptimizers GeometricOptimizers.StrictlyLowerTriangular), [`StrictlyUpperTriangular`](@extref GeometricOptimizers GeometricOptimizers.StrictlyUpperTriangular), [`SymmetricMatrix`](@extref GeometricOptimizers GeometricOptimizers.SymmetricMatrix) and [`SkewSymMatrix`](@extref GeometricOptimizers GeometricOptimizers.SkewSymMatrix) and objects of [`Manifold`](@extref GeometricOptimizers GeometricOptimizers.Manifold) type such as the [`StiefelManifold`](@extref GeometricOptimizers GeometricOptimizers.StiefelManifold). 
 
 ## Library Functions
 
@@ -10,8 +10,8 @@ GeometricMachineLearning.tensor_mat_mul!(::AbstractArray{<:Number, 3}, ::Abstrac
 GeometricMachineLearning.tensor_mat_mul!(::AbstractArray{<:Number, 3}, ::AbstractArray{<:Number, 3}, ::SymmetricMatrix)
 GeometricMachineLearning.mat_tensor_mul(::AbstractMatrix, ::AbstractArray{<:Number, 3})
 GeometricMachineLearning.mat_tensor_mul!(::AbstractArray{<:Number, 3}, ::AbstractMatrix, ::AbstractArray{<:Number, 3})
-GeometricMachineLearning.mat_tensor_mul!(::AbstractArray{<:Number, 3}, ::LowerTriangular, ::AbstractArray{<:Number, 3})
-GeometricMachineLearning.mat_tensor_mul!(::AbstractArray{<:Number, 3}, ::UpperTriangular, ::AbstractArray{<:Number, 3})
+GeometricMachineLearning.mat_tensor_mul!(::AbstractArray{<:Number, 3}, ::StrictlyLowerTriangular, ::AbstractArray{<:Number, 3})
+GeometricMachineLearning.mat_tensor_mul!(::AbstractArray{<:Number, 3}, ::StrictlyUpperTriangular, ::AbstractArray{<:Number, 3})
 GeometricMachineLearning.mat_tensor_mul!(::AbstractArray{<:Number, 3}, ::SkewSymMatrix, ::AbstractArray{<:Number, 3})
 GeometricMachineLearning.mat_tensor_mul!(::AbstractArray{<:Number, 3}, ::SymmetricMatrix, ::AbstractArray{<:Number, 3})
 ```
\ No newline at end of file
diff --git a/docs/src/index.md b/docs/src/index.md
index 00d3dc8d6..7c9fe47f7 100644
--- a/docs/src/index.md
+++ b/docs/src/index.md
@@ -58,7 +58,7 @@ Everything tried on that device ran. Constructed, and then applied to both a mat
 
 Two of those results rest on `GeometricOptimizers` rather than on anything here.
 
-- `VolumePreservingFeedForward` and `VolumePreservingTransformer` applied to a **matrix** multiply a `LowerTriangular` or `UpperTriangular` weight by that matrix. `GeometricOptimizers` supplies a `KernelAbstractions` kernel for that product, so the call does not fall through to a generic multiply that reads one entry at a time, which `GPUArraysCore` refuses.
+- `VolumePreservingFeedForward` and `VolumePreservingTransformer` applied to a **matrix** multiply a `StrictlyLowerTriangular` or `StrictlyUpperTriangular` weight by that matrix. `GeometricOptimizers` supplies a `KernelAbstractions` kernel for that product, so the call does not fall through to a generic multiply that reads one entry at a time, which `GPUArraysCore` refuses.
 - `StiefelLayer`, `GrassmannLayer` and `PSDLayer`, and so every architecture that holds one of them, orthonormalize their weight at **construction** through `GeometricOptimizers.orthonormal_columns`. That is CholeskyQR2 — matrix products and triangular solves only — so it runs wherever the draw was allocated. `LinearAlgebra.qr!` is a host factorization and `Metal.jl` implements no `qr` for an `MtlArray`, so a device needs the other one.
 
 ## Tutorials 
diff --git a/docs/src/layers/volume_preserving_feedforward.md b/docs/src/layers/volume_preserving_feedforward.md
index d85cac9a8..4c463da50 100644
--- a/docs/src/layers/volume_preserving_feedforward.md
+++ b/docs/src/layers/volume_preserving_feedforward.md
@@ -7,7 +7,7 @@ The *volume-preserving feedforward layers* in `GeometricMachineLearning` are clo
 ```math
 \mathtt{VPFF}_{A, b}: x \mapsto x + \sigma(Ax + b),
 ```
-where ``\sigma`` is a nonlinearity, ``A`` is the weight and ``b`` is the bias. The matrix ``A`` is either a [`LowerTriangular`](@extref GeometricOptimizers GeometricOptimizers.LowerTriangular) matrix ``L`` or an [`UpperTriangular`](@extref GeometricOptimizers GeometricOptimizers.UpperTriangular) matrix ``U``. We demonstrate volume-preservation of these layers by considering the case ``A = L``. The matrix looks as follows:
+where ``\sigma`` is a nonlinearity, ``A`` is the weight and ``b`` is the bias. The matrix ``A`` is either a [`StrictlyLowerTriangular`](@extref GeometricOptimizers GeometricOptimizers.StrictlyLowerTriangular) matrix ``L`` or an [`StrictlyUpperTriangular`](@extref GeometricOptimizers GeometricOptimizers.StrictlyUpperTriangular) matrix ``U``. We demonstrate volume-preservation of these layers by considering the case ``A = L``. The matrix looks as follows:
 
 ```math 
 L = \begin{pmatrix}
@@ -28,7 +28,7 @@ J = \nabla\mathtt{VPFF}_{L, b} = \begin{pmatrix}
      b_{n1} & \cdots & b_{n(n-1)}      & 1 
 \end{pmatrix},
 ```
-and the determinant of ``J`` is 1, i.e. the map is volume-preserving. A similar statement holds if the matrix ``A`` is [`UpperTriangular`](@extref GeometricOptimizers GeometricOptimizers.UpperTriangular) instead of [`LowerTriangular`](@extref GeometricOptimizers GeometricOptimizers.LowerTriangular).
+and the determinant of ``J`` is 1, i.e. the map is volume-preserving. A similar statement holds if the matrix ``A`` is [`StrictlyUpperTriangular`](@extref GeometricOptimizers GeometricOptimizers.StrictlyUpperTriangular) instead of [`StrictlyLowerTriangular`](@extref GeometricOptimizers GeometricOptimizers.StrictlyLowerTriangular).
 
 ## Library Functions 
 
diff --git a/docs/src/tutorials/volume_preserving_transformer_rigid_body.md b/docs/src/tutorials/volume_preserving_transformer_rigid_body.md
index 17a451918..f60d1d2d5 100644
--- a/docs/src/tutorials/volume_preserving_transformer_rigid_body.md
+++ b/docs/src/tutorials/volume_preserving_transformer_rigid_body.md
@@ -222,8 +222,8 @@ nn_st = mtc(nn_st)
 using HDF5 # hide
 # the two files store the type names `LowerTriangular` and `UpperTriangular` # hide
 const NNP = parentmodule(NetworkParameters) # hide
-NNP.register_parameter_type!("LowerTriangular", (S, md) -> LowerTriangular(S.S, S.n)) # hide
-NNP.register_parameter_type!("UpperTriangular", (S, md) -> UpperTriangular(S.S, S.n)) # hide
+NNP.register_parameter_type!("LowerTriangular", (S, md) -> StrictlyLowerTriangular(S.S, S.n)) # hide
+NNP.register_parameter_type!("UpperTriangular", (S, md) -> StrictlyUpperTriangular(S.S, S.n)) # hide
 nn_vpff = load(NeuralNetwork, "transformer_rigid_body_nn_vpff.h5", nn_vpff.architecture) # hide
 nn_vpt = load(NeuralNetwork, "transformer_rigid_body_nn_vpt.h5", nn_vpt.architecture) # hide
 nn_st = load(NeuralNetwork, "transformer_rigid_body_nn_st.h5", nn_st.architecture) # hide
diff --git a/src/GeometricMachineLearning.jl b/src/GeometricMachineLearning.jl
index ecd719838..0ee6d09f9 100644
--- a/src/GeometricMachineLearning.jl
+++ b/src/GeometricMachineLearning.jl
@@ -41,8 +41,7 @@ import Symbolics
 import GeometricOptimizers
 import GeometricOptimizers: Manifold, StiefelManifold, GrassmannManifold
 import GeometricOptimizers: SkewSymMatrix, SymmetricMatrix,
-                            StrictlyLowerTriangular as LowerTriangular,
-                            StrictlyUpperTriangular as UpperTriangular,
+                            StrictlyLowerTriangular, StrictlyUpperTriangular,
                             StiefelProjection
 import GeometricOptimizers: StiefelLieAlgHorMatrix, GrassmannLieAlgHorMatrix
 import GeometricOptimizers: rgrad, metric, check, global_section
@@ -114,7 +113,7 @@ include("arrays/poisson_tensor.jl")
 # Re-exported from GeometricOptimizers, so that `using GeometricMachineLearning` on its own still
 # gives a caller the matrix types its layers are parametrized by.
 export SymmetricMatrix, SkewSymMatrix
-export LowerTriangular, UpperTriangular
+export StrictlyLowerTriangular, StrictlyUpperTriangular
 export StiefelLieAlgHorMatrix, GrassmannLieAlgHorMatrix
 export StiefelProjection
 # GML's own
diff --git a/src/kernels/kernel_ad_routines/mat_tensor_mul.jl b/src/kernels/kernel_ad_routines/mat_tensor_mul.jl
index aa7970542..24c31a323 100644
--- a/src/kernels/kernel_ad_routines/mat_tensor_mul.jl
+++ b/src/kernels/kernel_ad_routines/mat_tensor_mul.jl
@@ -73,15 +73,16 @@ function ChainRulesCore.rrule(::typeof(lo_mat_mul), S::AbstractVector{T},
     return C, lo_mat_mul_pullback
 end
 
-function ChainRulesCore.rrule(::typeof(mat_tensor_mul), B::LowerTriangular{T}, A::AbstractArray{
-        T, 3}) where {T}
+function ChainRulesCore.rrule(::typeof(mat_tensor_mul), B::StrictlyLowerTriangular{T},
+        A::AbstractArray{
+            T, 3}) where {T}
     @assert size(A, 1) == B.n
     C = mat_tensor_mul(B, A)
     function lower_triangular_mul_pullback(dC)
         dC = unthunk(dC)
         f̄, dS, dA, _ = rrule(lo_mat_mul, B.S, A, B.n)[2](dC)
 
-        return f̄, LowerTriangular(dS, B.n), dA
+        return f̄, StrictlyLowerTriangular(dS, B.n), dA
     end
     return C, lower_triangular_mul_pullback
 end
@@ -144,15 +145,16 @@ function ChainRulesCore.rrule(::typeof(up_mat_mul), S::AbstractVector{T},
     return C, up_mat_mul_pullback
 end
 
-function ChainRulesCore.rrule(::typeof(mat_tensor_mul), B::UpperTriangular{T}, A::AbstractArray{
-        T, 3}) where {T}
+function ChainRulesCore.rrule(::typeof(mat_tensor_mul), B::StrictlyUpperTriangular{T},
+        A::AbstractArray{
+            T, 3}) where {T}
     @assert size(A, 1) == B.n
     C = mat_tensor_mul(B, A)
     function upper_triangular_mul_pullback(dC)
         dC = unthunk(dC)
         f̄, dS, dA, _ = rrule(up_mat_mul, B.S, A, B.n)[2](dC)
 
-        return f̄, UpperTriangular(dS, B.n), dA
+        return f̄, StrictlyUpperTriangular(dS, B.n), dA
     end
     return C, upper_triangular_mul_pullback
 end
diff --git a/src/kernels/mat_tensor_mul.jl b/src/kernels/mat_tensor_mul.jl
index a2bd3d1bc..589729345 100644
--- a/src/kernels/mat_tensor_mul.jl
+++ b/src/kernels/mat_tensor_mul.jl
@@ -119,7 +119,7 @@ function mat_tensor_mul!(C::AbstractArray{<:Number, 3}, A::SymmetricMatrix, B::A
     symmetric_mat_mul!(C, A.S, B, A.n)
 end
 
-########################### LowerTriangular
+########################### StrictlyLowerTriangular
 
 @kernel function lo_mul_kernel!(C::AbstractArray{T, 3}, S::AbstractVector{T},
         B::AbstractArray{T, 3}, ::Int) where {T}
@@ -152,21 +152,22 @@ function lo_mat_mul(S::AbstractVector{T}, B::AbstractArray{T, 3}, n::Int) where
 end
 
 @doc raw"""
-    mat_tensor_mul!(C, A::LowerTriangular, B)
+    mat_tensor_mul!(C, A::StrictlyLowerTriangular, B)
 
 Multiply the lower-triangular matrix `A` onto the tensor `B` from the left and store the result in `C`.
 
 This performs an efficient multiplication based on the special structure of the lower-triangular matrix `A`.
 """
-function mat_tensor_mul!(C::AbstractArray{<:Number, 3}, A::LowerTriangular, B::AbstractArray{
-        <:Number, 3})
+function mat_tensor_mul!(C::AbstractArray{<:Number, 3}, A::StrictlyLowerTriangular,
+        B::AbstractArray{
+            <:Number, 3})
     @assert eltype(C) == eltype(A) == eltype(B)
     @assert A.n == size(C, 1) == size(B, 1)
 
     lo_mat_mul!(C, A.S, B, A.n)
 end
 
-####################### UpperTriangular
+####################### StrictlyUpperTriangular
 
 @kernel function up_mul_kernel!(C::AbstractArray{T, 3}, S::AbstractVector{T},
         B::AbstractArray{T, 3}, n::Int) where {T}
@@ -199,14 +200,15 @@ function up_mat_mul(S::AbstractVector{T}, B::AbstractArray{T, 3}, n::Int) where
 end
 
 @doc raw"""
-    mat_tensor_mul!(C, A::UpperTriangular, B)
+    mat_tensor_mul!(C, A::StrictlyUpperTriangular, B)
 
 Multiply the upper-triangular matrix `A` onto the tensor `B` from the left and store the result in `C`.
 
 This performs an efficient multiplication based on the special structure of the upper-triangular matrix `A`.
 """
-function mat_tensor_mul!(C::AbstractArray{<:Number, 3}, A::UpperTriangular, B::AbstractArray{
-        <:Number, 3})
+function mat_tensor_mul!(C::AbstractArray{<:Number, 3}, A::StrictlyUpperTriangular,
+        B::AbstractArray{
+            <:Number, 3})
     @assert eltype(C) == eltype(A) == eltype(B)
     @assert A.n == size(C, 1) == size(B, 1)
 
diff --git a/src/layers/volume_preserving_feedforward.jl b/src/layers/volume_preserving_feedforward.jl
index 7822fcda8..ef81bfc5e 100644
--- a/src/layers/volume_preserving_feedforward.jl
+++ b/src/layers/volume_preserving_feedforward.jl
@@ -4,10 +4,10 @@
 Super-type of [`VolumePreservingLowerLayer`](@ref) and [`VolumePreservingUpperLayer`](@ref). The layers do the following: 
 
 ```math
-x \mapsto \begin{cases} \sigma(Lx + b) & \text{where $L$ is }\mathtt{LowerTriangular}, \\ \sigma(Ux + b) & \text{where $U$ is }\mathtt{UpperTriangular}. \end{cases}
+x \mapsto \begin{cases} \sigma(Lx + b) & \text{where $L$ is }\mathtt{StrictlyLowerTriangular}, \\ \sigma(Ux + b) & \text{where $U$ is }\mathtt{StrictlyUpperTriangular}. \end{cases}
 ```
 
-The functor can be applied to a vector, a matrix or a tensor. The special matrices are implemented as [`LowerTriangular`](@extref GeometricOptimizers GeometricOptimizers.LowerTriangular) and [`UpperTriangular`](@extref GeometricOptimizers GeometricOptimizers.UpperTriangular).
+The functor can be applied to a vector, a matrix or a tensor. The special matrices are implemented as [`StrictlyLowerTriangular`](@extref GeometricOptimizers GeometricOptimizers.StrictlyLowerTriangular) and [`StrictlyUpperTriangular`](@extref GeometricOptimizers GeometricOptimizers.StrictlyUpperTriangular).
 """
 abstract type VolumePreservingFeedForwardLayer{M, N, bias} <: AbstractExplicitLayer{M, N} end
 
@@ -36,7 +36,7 @@ x = \begin{pmatrix} 1 \\ 1 \end{pmatrix}
 using GeometricMachineLearning
 
 l = VolumePreservingLowerLayer(2, identity; use_bias=false)
-A = LowerTriangular([1,], 2)
+A = StrictlyLowerTriangular([1,], 2)
 ps = (weight = A, )
 x = ones(eltype(A), 2)
 
@@ -93,7 +93,7 @@ x = \begin{pmatrix} 1 \\ 1 \end{pmatrix}
 using GeometricMachineLearning
 
 l = VolumePreservingUpperLayer(2, identity; use_bias=false)
-A = UpperTriangular([1,], 2)
+A = StrictlyUpperTriangular([1,], 2)
 ps = (weight = A, )
 x = ones(eltype(A), 2)
 
@@ -141,7 +141,7 @@ function initialparameters(
     init_weight!(rng, S)
     init_bias!(rng, b)
 
-    (weight = LowerTriangular(S, M), bias = b)
+    (weight = StrictlyLowerTriangular(S, M), bias = b)
 end
 
 function initialparameters(
@@ -153,7 +153,7 @@ function initialparameters(
     init_weight!(rng, S)
     init_bias!(rng, b)
 
-    (weight = UpperTriangular(S, M), bias = b)
+    (weight = StrictlyUpperTriangular(S, M), bias = b)
 end
 
 function initialparameters(
@@ -163,7 +163,7 @@ function initialparameters(
     S = KernelAbstractions.allocate(backend, T, parameterlength(d))
     init_weight!(rng, S)
 
-    (weight = LowerTriangular(S, M),)
+    (weight = StrictlyLowerTriangular(S, M),)
 end
 
 function initialparameters(
@@ -173,7 +173,7 @@ function initialparameters(
     S = KernelAbstractions.allocate(backend, T, parameterlength(d))
     init_weight!(rng, S)
 
-    (weight = UpperTriangular(S, M),)
+    (weight = StrictlyUpperTriangular(S, M),)
 end
 
 function (d::VolumePreservingFeedForwardLayer{M, M, :bias})(
diff --git a/test/arrays/triangular.jl b/test/arrays/triangular.jl
index 06226a1b8..abb754f66 100644
--- a/test/arrays/triangular.jl
+++ b/test/arrays/triangular.jl
@@ -12,8 +12,8 @@ Random.seed!(1234)
 # third axis of a tensor with `mat_tensor_mul`, and the pullback of that kernel.
 
 function triangular_tensor_multiplication_test(T = Float64, n::Int = 5)
-    Aₗ = rand(LowerTriangular{T}, n)
-    Aᵤ = rand(UpperTriangular{T}, n)
+    Aₗ = rand(StrictlyLowerTriangular{T}, n)
+    Aᵤ = rand(StrictlyUpperTriangular{T}, n)
 
     B = rand(T, n, n, n)
     AₗB = mat_tensor_mul(Aₗ, B)
@@ -25,8 +25,8 @@ function triangular_tensor_multiplication_test(T = Float64, n::Int = 5)
 end
 
 function triangular_tensor_multiplication_pullback_test(T = Float64, n::Int = 5)
-    Aₗ = rand(LowerTriangular{T}, n)
-    Aᵤ = rand(UpperTriangular{T}, n)
+    Aₗ = rand(StrictlyLowerTriangular{T}, n)
+    Aᵤ = rand(StrictlyUpperTriangular{T}, n)
 
     B = rand(T, n, n, n)
     C_diff = rand(T, n, n, n)
diff --git a/test/integration/changebackend_tests.jl b/test/integration/changebackend_tests.jl
index e816364bf..19c23bf3b 100644
--- a/test/integration/changebackend_tests.jl
+++ b/test/integration/changebackend_tests.jl
@@ -35,8 +35,8 @@ leaves = (
     grassmann = rand(GrassmannManifold{Float64}, N, n),
     symmetric = SymmetricMatrix(rand(10), 4),
     skew = SkewSymMatrix(rand(6), 4),
-    lower = LowerTriangular(rand(6), 4),
-    upper = UpperTriangular(rand(6), 4),
+    lower = StrictlyLowerTriangular(rand(6), 4),
+    upper = StrictlyUpperTriangular(rand(6), 4),
     stiefhor = StiefelLieAlgHorMatrix(SkewSymMatrix(rand(n, n)), rand(N - n, n), N, n),
     grasshor = GrassmannLieAlgHorMatrix(rand(N - n, n), N, n)
 )
diff --git a/test/integration/hdf5_support.jl b/test/integration/hdf5_support.jl
index 6e04ea5f8..a206c062d 100644
--- a/test/integration/hdf5_support.jl
+++ b/test/integration/hdf5_support.jl
@@ -14,8 +14,8 @@ _ps_eq(a::AbstractArray, b::AbstractArray) = a ≈ b
 _ps_eq(a::StiefelManifold, b::StiefelManifold) = a.A ≈ b.A
 _ps_eq(a::SymmetricMatrix, b::SymmetricMatrix) = a.S ≈ b.S && a.n == b.n
 _ps_eq(a::SkewSymMatrix, b::SkewSymMatrix) = a.S ≈ b.S && a.n == b.n
-_ps_eq(a::LowerTriangular, b::LowerTriangular) = a.S ≈ b.S && a.n == b.n
-_ps_eq(a::UpperTriangular, b::UpperTriangular) = a.S ≈ b.S && a.n == b.n
+_ps_eq(a::StrictlyLowerTriangular, b::StrictlyLowerTriangular) = a.S ≈ b.S && a.n == b.n
+_ps_eq(a::StrictlyUpperTriangular, b::StrictlyUpperTriangular) = a.S ≈ b.S && a.n == b.n
 function _ps_eq(a::NamedTuple, b::NamedTuple)
     Set(keys(a)) == Set(keys(b)) || return false
     all(_ps_eq(a[k], b[k]) for k in keys(a))
@@ -48,7 +48,8 @@ function _write_legacy(h5, Y::StiefelManifold, path::AbstractString)
 end
 
 for (T, name) in ((:SymmetricMatrix, "SymmetricMatrix"), (:SkewSymMatrix, "SkewSymMatrix"),
-    (:LowerTriangular, "LowerTriangular"), (:UpperTriangular, "UpperTriangular"))
+    (:StrictlyLowerTriangular, "StrictlyLowerTriangular"), (
+    :StrictlyUpperTriangular, "StrictlyUpperTriangular"))
     @eval function _write_legacy(h5, A::$T, path::AbstractString)
         g = HDF5.create_group(h5, path)
         HDF5.attributes(g)["gml_type"] = $name
@@ -131,8 +132,8 @@ end
     end
 end
 
-# VolumePreservingFeedForward uses LowerTriangular / UpperTriangular parameters.
-@testset "save/load roundtrip: VolumePreservingFeedForward (LowerTriangular/UpperTriangular)" begin
+# VolumePreservingFeedForward uses StrictlyLowerTriangular / StrictlyUpperTriangular parameters.
+@testset "save/load roundtrip: VolumePreservingFeedForward (StrictlyLowerTriangular/StrictlyUpperTriangular)" begin
     arch = VolumePreservingFeedForward(4, 4, 1, tanh)
     nn = NeuralNetwork(arch)
     x = rand(4)
diff --git a/test/layers/volume_preserving_feedforward.jl b/test/layers/volume_preserving_feedforward.jl
index 673951afc..85f2da44c 100644
--- a/test/layers/volume_preserving_feedforward.jl
+++ b/test/layers/volume_preserving_feedforward.jl
@@ -35,11 +35,11 @@ test_volume_preserving_feedforward(; T = Float32)
 test_volume_preserving_feedforward(; T = Float64)
 
 # A vector in, a vector out. Neither layer shapes the result: the `:no_bias` method is
-# `x + d.activation.(ps.weight * x)`, and `LowerTriangular * ::AbstractVector` returns a vector,
+# `x + d.activation.(ps.weight * x)`, and `StrictlyLowerTriangular * ::AbstractVector` returns a vector,
 # which is what `LinearAlgebra` returns for every other matrix type.
 l = VolumePreservingLowerLayer(2, identity; use_bias = false)
-A = LowerTriangular([1], 2)
+A = StrictlyLowerTriangular([1], 2)
 @test l(ones(eltype(A), 2), (weight = A,)) == [1, 2]
 l = VolumePreservingUpperLayer(2, identity; use_bias = false)
-A = UpperTriangular([1], 2)
+A = StrictlyUpperTriangular([1], 2)
 @test l(ones(eltype(A), 2), (weight = A,)) == [2, 1]
diff --git a/test/map_to_cpu.jl b/test/map_to_cpu.jl
index adde8bd23..c83c0bfa9 100644
--- a/test/map_to_cpu.jl
+++ b/test/map_to_cpu.jl
@@ -12,8 +12,7 @@ using Random
 using Test
 
 import GeometricOptimizers: StiefelManifold, SymmetricMatrix, SkewSymMatrix,
-                            StrictlyLowerTriangular as LowerTriangular,
-                            StrictlyUpperTriangular as UpperTriangular
+                            StrictlyLowerTriangular, StrictlyUpperTriangular
 
 Random.seed!(1234)
 
@@ -23,7 +22,8 @@ const N, n = 6, 3
     ps = NetworkParameters((
         L1 = (Y = rand(StiefelManifold{Float64}, N, n), b = randn(N)),
         L2 = (S = SymmetricMatrix(randn(n, n)), A = SkewSymMatrix(randn(n, n))),
-        L3 = (L = LowerTriangular(randn(n, n)), U = UpperTriangular(randn(n, n)))
+        L3 = (L = StrictlyLowerTriangular(randn(n, n)),
+            U = StrictlyUpperTriangular(randn(n, n)))
     ))
 
     back = map_to_cpu(ps)
@@ -36,8 +36,8 @@ const N, n = 6, 3
     @test back.L1.Y isa StiefelManifold
     @test back.L2.S isa SymmetricMatrix
     @test back.L2.A isa SkewSymMatrix
-    @test back.L3.L isa LowerTriangular
-    @test back.L3.U isa UpperTriangular
+    @test back.L3.L isa StrictlyLowerTriangular
+    @test back.L3.U isa StrictlyUpperTriangular
 
     # the `n` a structured leaf carries is not in its storage; `rebuild` takes it from the prototype
     @test back.L2.S.n == ps.L2.S.n
diff --git a/test/optimizers/structured_array_parameters.jl b/test/optimizers/structured_array_parameters.jl
index 489f2b51f..56e004145 100644
--- a/test/optimizers/structured_array_parameters.jl
+++ b/test/optimizers/structured_array_parameters.jl
@@ -8,7 +8,7 @@ Random.seed!(123)
 @doc raw"""
 Every optimizer method has to work with the structured matrix types GML uses as *ordinary* weights:
 `SymmetricMatrix` (SympNet and symplectic attention layers), `SkewSymMatrix` (volume-preserving
-attention) and `LowerTriangular`/`UpperTriangular` (volume-preserving feedforward layers).
+attention) and `StrictlyLowerTriangular`/`StrictlyUpperTriangular` (volume-preserving feedforward layers).
 
 These have their own storage and no `setindex!`, so they need a type-preserving `similar` and
 elementwise operations written on the free parameters; without them
@@ -41,7 +41,7 @@ end
 # then fails to match them against the parameter
 @testset "structured weights keep their type in `similar`" begin
     for A in (rand(SymmetricMatrix{Float64}, 4), rand(SkewSymMatrix{Float64}, 4),
-        rand(LowerTriangular{Float64}, 4), rand(UpperTriangular{Float64}, 4))
+        rand(StrictlyLowerTriangular{Float64}, 4), rand(StrictlyUpperTriangular{Float64}, 4))
         @test typeof(similar(A)) == typeof(A)
         @test typeof(zero(A)) == typeof(A)
         @test size(similar(A)) == size(A)
@@ -51,7 +51,7 @@ end
 @testset "`LASympNet` (SymmetricMatrix)" begin
     optimizer_runs(LASympNet(4), Batch(5), 4)
 end
-@testset "`VolumePreservingFeedForward` (Lower/UpperTriangular)" begin
+@testset "`VolumePreservingFeedForward` (StrictlyLower/StrictlyUpperTriangular)" begin
     optimizer_runs(VolumePreservingFeedForward(4), Batch(5), 4)
 end
 @testset "`VolumePreservingTransformer` (SkewSymMatrix + triangular)" begin

From 274a7e0e40d5a2088deb8eb51f73dc7b1303b6ee Mon Sep 17 00:00:00 2001
From: Michael Kraus <michael.kraus@ipp.mpg.de>
Date: Mon, 12 Oct 2026 18:35:35 +0200
Subject: [PATCH 03/11] Record the passing critic's other defects of part M1a
 in KNOWN_ISSUES.md

K10: two Adam training tests run in Float64 only. K11: a test comment
reports a past experiment. K12: Batch never draws the last sample.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 KNOWN_ISSUES.md | 37 +++++++++++++++++++++++++++++++++++++
 1 file changed, 37 insertions(+)

diff --git a/KNOWN_ISSUES.md b/KNOWN_ISSUES.md
index 3e6f52c18..dffa64976 100644
--- a/KNOWN_ISSUES.md
+++ b/KNOWN_ISSUES.md
@@ -430,3 +430,40 @@ What is known to be wrong in GeometricMachineLearning and is not fixed.
   leaves the parameters on the device". The run it reports used the `Optimizer` of
   GeometricMachineLearning 0.8, which `NetworkOptimizer` and `train!` replace; no Metal run of
   `train!` is recorded.
+
+### K10 · The two training tests of `adam_with_learning_rate_decay.jl` run in `Float64` only.
+
+- **location:** `test/optimizers/adam_with_learning_rate_decay.jl`
+- **kind:** missing test
+- **found:** 2026-10-12, by the critics of part M1a; the precision is older than that part
+- **evidence:**
+  `train_network` and `train_manifold_network` build their `DataLoader` from
+  `sin.(0:0.1:2π)`, a `Vector{Float64}`, and take the network's element type from `eltype(dl)`.
+  Neither function takes an element type, so neither runs over `REAL_ELTYPES`. The third testset
+  of the file, `schedule_starts_at_one`, loops over `REAL_ELTYPES`. The fix builds `dl` per `T`
+  and loops both functions over `REAL_ELTYPES`.
+
+### K11 · A comment in `adam_with_learning_rate_decay.jl` reports a past experiment.
+
+- **location:** `test/optimizers/adam_with_learning_rate_decay.jl`
+- **kind:** found late
+- **found:** 2026-10-12, by the critics of part M1a; the comment is older than that part
+- **evidence:**
+  The comment above `train_manifold_network` says "from a seeded start the loss fell by under 2%
+  on 1.13 and *rose* on 1.12". The fix states only the present constraint: a 32-epoch budget
+  drives the learning rate to `η₂` almost at once, and 128 epochs train.
+
+### K12 · `Batch` never draws the last sample of a `DataLoader` of matrices.
+
+- **location:** `src/data_loader/batch.jl`
+- **kind:** found late
+- **found:** 2026-10-12, by the critics of part M1a
+- **evidence:**
+  On a `DataLoader(x, x)` with `x = rand(2, 10)`, the minibatches of `Batch(bs)` hold the samples
+  1 to 9 for each `bs` in 1, 3, 4 and 10; sample 10 is in none of them. `Batch(1)` gives 9
+  minibatches.
+
+  ```
+  1 nbatches 9 samples [(1, 1), (2, 1), (3, 1), (4, 1), (5, 1), (6, 1), (7, 1), (8, 1), (9, 1)]
+  10 nbatches 1 samples [(1, 1), (2, 1), (3, 1), (4, 1), (5, 1), (6, 1), (7, 1), (8, 1), (9, 1)]
+  ```

From 68c47ff0fb6d6f0e055a85204f10bea7f25697de Mon Sep 17 00:00:00 2001
From: Michael Kraus <michael.kraus@ipp.mpg.de>
Date: Mon, 12 Oct 2026 18:55:26 +0200
Subject: [PATCH 04/11] Fix the pre-PR verifier's text findings of part M1a

Comments in src/GeometricMachineLearning.jl, test/quality/aqua.jl and
scripts/verification/all_zero_leaf.jl; KNOWN_ISSUES.md entries under the
heading of their kind; the GPU paragraph of docs/src/index.md no longer
reports training with the removed Optimizer, so K9 goes; the CHANGELOG
zero-leaf sentence names every method that carries momentum.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 CHANGELOG.md                          |   2 +-
 KNOWN_ISSUES.md                       | 163 ++++++++++++--------------
 docs/src/index.md                     |   2 +-
 scripts/verification/all_zero_leaf.jl |   4 +-
 src/GeometricMachineLearning.jl       |   5 +-
 test/quality/aqua.jl                  |   8 +-
 6 files changed, 87 insertions(+), 97 deletions(-)

diff --git a/CHANGELOG.md b/CHANGELOG.md
index a9d8021da..247b80b3d 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -17,7 +17,7 @@ breaking release).
 
 ### Removed (breaking)
 
-- **`Optimizer`, the callable optimizers and their aliases are gone; training moves to `NetworkOptimizer` and `train!`.** Training now uses GeometricOptimizers 0.9's `TrainingOptimizer`. `NetworkOptimizer(nn; algorithm = Adam(), linesearch = default_step_size(algorithm), retraction = Cayley())`, also accepted from a `NetworkParameters`, replaces `Optimizer`. Migrate `Optimizer(method, nn; step_size = η, retraction = cayley)` to `NetworkOptimizer(nn; algorithm = method, linesearch = η, retraction = Cayley())`, and `Optimizer(nn; AdamOptimizerWithDecay(n, T)...)` to `NetworkOptimizer(nn; AdamOptimizerWithDecay(n)...)`. The `n` of `AdamOptimizerWithDecay` counts optimization steps, one per minibatch, as in GeometricOptimizers and GML 0.8. One cache and one state now cover the whole parameter set; 0.8 had one per layer. `using GeometricOptimizers, GeometricMachineLearning` leaves `Optimizer` unqualified as GeometricOptimizers' solver, and no name is shadowed. `train!(nn, opt, dl, batch, n_epochs; loss, show_progress = true)` replaces the four callable methods `(o::Optimizer)(nn, dl, batch, n_epochs[, loss, pullback])` and `optimize_for_one_epoch!`, which is removed from the exports. Migrate `o(nn, dl, batch, n, loss; kw...)` to `train!(nn, o, dl, batch, n; loss, kw...)`. Without `loss` the defaults are those of 0.8: `TransformerLoss(batch)` for a `TransformerIntegrator` with a `Batch{:Transformer}`, `FeedForwardLoss()` for a `NeuralNetworkIntegrator`, and `AutoEncoderLoss()` for an `AutoEncoder`, each with a `Batch{:FeedForward}`. `loss` is a `NetworkLoss`; passing an `AbstractPullback` as the sixth positional argument, which 0.8 allowed, is gone. The `λY` argument and the per-call `GlobalSection` are gone: the sections, moments and iteration number live in `opt`, so a second `train!` call with the same `opt` resumes the first, and `train!` draws no random numbers beyond the minibatch shuffle. `n_epochs < 1` raises an `ArgumentError` before any step. The loss history has the element type of the loss value, so a `Float64` network on `Float32` data gives `Vector{Float64}`, where 0.8 gave `float(eltype(dl))`; the epoch accumulator keeps that type. A hand-written loop steps with `GeometricOptimizers.optimization_step!(params(nn), opt, dp)`, which GML adds for `NetworkOptimizer`; `optimization_step!(opt, λY, ps, dp)` is removed. A gradient leaf that is all zero is stepped, not skipped: NeuralNetworkParameters 0.5 gives a layer the loss does not read as a zero leaf, not `nothing`. Under `GradientMethod`, `MomentumMethod` and `Adam` such a leaf stays where it is; under `AdamWithEuclideanDecay` a Euclidean leaf that is never read decays, and a leaf read for some steps and then not keeps moving on its momentum. `scripts/verification/all_zero_leaf.jl` prints the figures.
+- **`Optimizer`, the callable optimizers and their aliases are gone; training moves to `NetworkOptimizer` and `train!`.** Training now uses GeometricOptimizers 0.9's `TrainingOptimizer`. `NetworkOptimizer(nn; algorithm = Adam(), linesearch = default_step_size(algorithm), retraction = Cayley())`, also accepted from a `NetworkParameters`, replaces `Optimizer`. Migrate `Optimizer(method, nn; step_size = η, retraction = cayley)` to `NetworkOptimizer(nn; algorithm = method, linesearch = η, retraction = Cayley())`, and `Optimizer(nn; AdamOptimizerWithDecay(n, T)...)` to `NetworkOptimizer(nn; AdamOptimizerWithDecay(n)...)`. The `n` of `AdamOptimizerWithDecay` counts optimization steps, one per minibatch, as in GeometricOptimizers and GML 0.8. One cache and one state now cover the whole parameter set; 0.8 had one per layer. `using GeometricOptimizers, GeometricMachineLearning` leaves `Optimizer` unqualified as GeometricOptimizers' solver, and no name is shadowed. `train!(nn, opt, dl, batch, n_epochs; loss, show_progress = true)` replaces the four callable methods `(o::Optimizer)(nn, dl, batch, n_epochs[, loss, pullback])` and `optimize_for_one_epoch!`, which is removed from the exports. Migrate `o(nn, dl, batch, n, loss; kw...)` to `train!(nn, o, dl, batch, n; loss, kw...)`. Without `loss` the defaults are those of 0.8: `TransformerLoss(batch)` for a `TransformerIntegrator` with a `Batch{:Transformer}`, `FeedForwardLoss()` for a `NeuralNetworkIntegrator`, and `AutoEncoderLoss()` for an `AutoEncoder`, each with a `Batch{:FeedForward}`. `loss` is a `NetworkLoss`; passing an `AbstractPullback` as the sixth positional argument, which 0.8 allowed, is gone. The `λY` argument and the per-call `GlobalSection` are gone: the sections, moments and iteration number live in `opt`, so a second `train!` call with the same `opt` resumes the first, and `train!` draws no random numbers beyond the minibatch shuffle. `n_epochs < 1` raises an `ArgumentError` before any step. The loss history has the element type of the loss value, so a `Float64` network on `Float32` data gives `Vector{Float64}`, where 0.8 gave `float(eltype(dl))`; the epoch accumulator keeps that type. A hand-written loop steps with `GeometricOptimizers.optimization_step!(params(nn), opt, dp)`, which GML adds for `NetworkOptimizer`; `optimization_step!(opt, λY, ps, dp)` is removed. A gradient leaf that is all zero is stepped, not skipped: NeuralNetworkParameters 0.5 gives a layer the loss does not read as a zero leaf, not `nothing`. A leaf that the loss never reads stays where it is under `GradientMethod`, `MomentumMethod` and `Adam`; under `AdamWithEuclideanDecay` such a Euclidean leaf decays. Under `MomentumMethod`, `Adam` and `AdamWithEuclideanDecay`, a leaf that the loss reads for some steps and then not keeps moving on its momentum. `scripts/verification/all_zero_leaf.jl` prints the figures.
 
 - **The aliases `GradientOptimizer`, `MomentumOptimizer` and `AdamOptimizer` are removed**; use `GradientMethod`, `MomentumMethod` and `Adam`. These are no longer exported: `OptimizerMethod`, `GradientState`, `MomentumState`, `AdamState`, `optimization_step!`, `optimize_for_one_epoch!` and `Optimizer`. `NetworkOptimizer`, `train!` and `AdamWithEuclideanDecay` are newly exported, and the package re-exports `GradientMethod`, `MomentumMethod`, `Adam`, `AdamOptimizerWithDecay`, `DecayingStatic`, `Cayley` and `Geodesic` from GeometricOptimizers. Methods carry no element type in GeometricOptimizers 0.9: `Adam(T)` becomes `Adam()`, and `MomentumMethod(α)` becomes `MomentumMethod(; α)`. `check(::Optimizer)` is removed.
 
diff --git a/KNOWN_ISSUES.md b/KNOWN_ISSUES.md
index dffa64976..215d51650 100644
--- a/KNOWN_ISSUES.md
+++ b/KNOWN_ISSUES.md
@@ -34,11 +34,11 @@ What is known to be wrong in GeometricMachineLearning and is not fixed.
   (**B1**, **B2**, **B3**, **B4**, **B6** and **B8** are all closed and their entries are gone: B1 and B2 by
   this release — the duplicated `AdamOptimizerWithDecay` and the split `Manifold`, both under
   *Removed (breaking)* — B3 by SymbolicNeuralNetworks 0.5, B4 by `a427add1`, which repaired the
-  documentation build, and B6 and B8 by this release: B6 by the `train!` retirement (the three methods that
+  documentation build, and B6 and B8 by this release: B6 by the retirement of the `train!` subsystem (the three methods that
   called the non-existent `vectorfield` are gone, and `SymplecticEulerLoss` carries their content on
   `hamiltonian_vector_field`, with tests that run), and B8 by the narrowing of `PoissonTensor`'s `Base.:*`
   to `Strided…` right-hand sides and `Base.getindex` to `Union{Int, Colon, AbstractVector{<:Integer}}`
-  indices, reducing method ambiguities from 18 to 1 as asserted in `test/quality/aqua.jl`. The numbers are
+  indices, reducing method ambiguities from 18 to 1. The numbers are
   left vacant rather than reused.)
 
   (**B9** is closed by this release and its entry is gone: the `iterate` method it named binds
@@ -299,6 +299,58 @@ What is known to be wrong in GeometricMachineLearning and is not fixed.
   `test/reduced_system/reduced_system.jl:44–45` name `CHANGELOG.md` rather than `KNOWN_ISSUES.md`.
   The fix is a follow-up change to those comments.
 
+### K7 · A network with a Stiefel leaf raises in `mat_tensor_mul` when the data has another element type.
+
+- **location:** `src/kernels/mat_tensor_mul.jl`
+- **kind:** found late
+- **found:** 2026-10-12, by the critics of part M1a; the kernel code is older than that part
+- **evidence:**
+  A `Float32` `Chain(StiefelLayer(6, 3), Dense(3, 2, tanh))` trained with `train!` on `Float64`
+  data, and a `Float64` one on `Float32` data, raise `AssertionError: eltype(A) == eltype(B)` in
+  `mat_tensor_mul`, reached through `src/layers/manifold_layer.jl` under Zygote. A network of
+  `Dense` layers alone trains on mixed precisions.
+
+  ```julia
+  x = rand(Float32, 6, 12)
+  nn = NeuralNetwork(Chain(StiefelLayer(6, 3), Dense(3, 2, tanh)), Float64)
+  train!(nn, NetworkOptimizer(nn), DataLoader(x, x[1:2, :]), Batch(4), 2; loss = FeedForwardLoss())
+  ```
+
+### K8 · A `DataLoader` of one sample cannot be batched.
+
+- **location:** `src/data_loader/batch.jl`
+- **kind:** found late
+- **found:** 2026-10-12, by the critics of part M1a
+- **evidence:**
+  `DataLoader(x, x)` with `x = rand(Float32, 2, 1)` and `Batch(1)` raises the time-series
+  assertion "The number of time steps has to be greater than sequence length + prediction window"
+  when `train!` draws its minibatches.
+
+### K11 · A comment in `adam_with_learning_rate_decay.jl` reports a past experiment.
+
+- **location:** `test/optimizers/adam_with_learning_rate_decay.jl`
+- **kind:** found late
+- **found:** 2026-10-12, by the critics of part M1a; the comment is older than that part
+- **evidence:**
+  The comment above `train_manifold_network` says "from a seeded start the loss fell by under 2%
+  on 1.13 and *rose* on 1.12". The fix states only the present constraint: a 32-epoch budget
+  drives the learning rate to `η₂` almost at once, and 128 epochs train.
+
+### K12 · `Batch` never draws the last sample of a `DataLoader` of matrices.
+
+- **location:** `src/data_loader/batch.jl`
+- **kind:** found late
+- **found:** 2026-10-12, by the critics of part M1a
+- **evidence:**
+  On a `DataLoader(x, x)` with `x = rand(2, 10)`, the minibatches of `Batch(bs)` hold the samples
+  1 to 9 for each `bs` in 1, 3, 4 and 10; sample 10 is in none of them. `Batch(1)` gives 9
+  minibatches.
+
+  ```
+  1 nbatches 9 samples [(1, 1), (2, 1), (3, 1), (4, 1), (5, 1), (6, 1), (7, 1), (8, 1), (9, 1)]
+  10 nbatches 1 samples [(1, 1), (2, 1), (3, 1), (4, 1), (5, 1), (6, 1), (7, 1), (8, 1), (9, 1)]
+  ```
+
 ## Upstream
 
 ### K2 · Revise prints EMFILE errors in the test log
@@ -337,6 +389,28 @@ What is known to be wrong in GeometricMachineLearning and is not fixed.
   itself records a skip where `Metal.functional()` is false; only the precompile fails. Closing this
   is a Metal.jl change: a workload that does not run where no device exists.
 
+### K6 · The gradient of a `SymmetricMatrix` leaf with respect to a `NetworkParameters` differs from the one with respect to the bare `NamedTuple`.
+
+- **location:** `test/helpers/network_parameters_gradient_structure.jl`
+- **kind:** upstream
+- **found:** 2026-10-12, with the move to NeuralNetworkParameters 0.5; issue #319
+- **evidence:**
+  Under NeuralNetworkParameters 0.5, `Zygote.gradient` with respect to a `NetworkParameters` gives
+  a `SymmetricMatrix` leaf that holds the gradient with respect to the leaf's storage, at any number
+  of accesses; with respect to the bare `NamedTuple` it gives the projection of the dense gradient.
+  The off-diagonal entries of the first are twice those of the second; the diagonals are equal. A
+  test of `isapprox(Matrix(a_wrapped), Matrix(a_bare))` in
+  `layers/symplectic_attention_network_parameters_gradient.jl` evaluates:
+
+  ```
+  Expression: isapprox(Matrix(a_wrapped), Matrix(a_bare))
+   Evaluated: isapprox([0.6575109961994726 1.1214311018548968; 1.1214311018548968 0.46185556373061615],
+                       [0.6575109961994726 0.5607155509274484; 0.5607155509274484 0.46185556373061615])
+  ```
+
+  The helper asserts this relation. Which of the two a training step needs is the pullback shape of
+  issue #319.
+
 ## Documentation
 
 ### K5 · The `PositionalEncoding` docstring names `test/metal/`, which does not exist
@@ -371,66 +445,6 @@ What is known to be wrong in GeometricMachineLearning and is not fixed.
   array `J`; a barrier on the stored value is CAUGHT. JET never reports a dynamic dispatch in the
   kernel's own frame. A barrier on a value inside each kernel body is CAUGHT.
 
-### K6 · The gradient of a `SymmetricMatrix` leaf with respect to a `NetworkParameters` differs from the one with respect to the bare `NamedTuple`.
-
-- **location:** `test/helpers/network_parameters_gradient_structure.jl`
-- **kind:** upstream
-- **found:** 2026-10-12, with the move to NeuralNetworkParameters 0.5; issue #319
-- **evidence:**
-  Under NeuralNetworkParameters 0.5, `Zygote.gradient` with respect to a `NetworkParameters` gives
-  a `SymmetricMatrix` leaf that holds the gradient with respect to the leaf's storage, at any number
-  of accesses; with respect to the bare `NamedTuple` it gives the projection of the dense gradient.
-  The off-diagonal entries of the first are twice those of the second; the diagonals are equal. A
-  test of `isapprox(Matrix(a_wrapped), Matrix(a_bare))` in
-  `layers/symplectic_attention_network_parameters_gradient.jl` evaluates:
-
-  ```
-  Expression: isapprox(Matrix(a_wrapped), Matrix(a_bare))
-   Evaluated: isapprox([0.6575109961994726 1.1214311018548968; 1.1214311018548968 0.46185556373061615],
-                       [0.6575109961994726 0.5607155509274484; 0.5607155509274484 0.46185556373061615])
-  ```
-
-  The helper asserts this relation. Which of the two a training step needs is the pullback shape of
-  issue #319.
-
-### K7 · A network with a Stiefel leaf raises in `mat_tensor_mul` when the data has another element type.
-
-- **location:** `src/kernels/mat_tensor_mul.jl`
-- **kind:** found late
-- **found:** 2026-10-12, by the critics of part M1a; the kernel code is older than that part
-- **evidence:**
-  A `Float32` `Chain(StiefelLayer(6, 3), Dense(3, 2, tanh))` trained with `train!` on `Float64`
-  data, and a `Float64` one on `Float32` data, raise `AssertionError: eltype(A) == eltype(B)` in
-  `mat_tensor_mul`, reached through `src/layers/manifold_layer.jl` under Zygote. A network of
-  `Dense` layers alone trains on mixed precisions.
-
-  ```julia
-  x = rand(Float32, 6, 12)
-  nn = NeuralNetwork(Chain(StiefelLayer(6, 3), Dense(3, 2, tanh)), Float64)
-  train!(nn, NetworkOptimizer(nn), DataLoader(x, x[1:2, :]), Batch(4), 2; loss = FeedForwardLoss())
-  ```
-
-### K8 · A `DataLoader` of one sample cannot be batched.
-
-- **location:** `src/data_loader/batch.jl`
-- **kind:** found late
-- **found:** 2026-10-12, by the critics of part M1a
-- **evidence:**
-  `DataLoader(x, x)` with `x = rand(Float32, 2, 1)` and `Batch(1)` raises the time-series
-  assertion "The number of time steps has to be greater than sequence length + prediction window"
-  when `train!` draws its minibatches.
-
-### K9 · The GPU paragraph of the manual reports training with `Optimizer`.
-
-- **location:** `docs/src/index.md:57`
-- **kind:** docs
-- **found:** 2026-10-12, by the critics of part M1a
-- **evidence:**
-  The paragraph says "so does training with `Optimizer`, which returns a `Float32` history and
-  leaves the parameters on the device". The run it reports used the `Optimizer` of
-  GeometricMachineLearning 0.8, which `NetworkOptimizer` and `train!` replace; no Metal run of
-  `train!` is recorded.
-
 ### K10 · The two training tests of `adam_with_learning_rate_decay.jl` run in `Float64` only.
 
 - **location:** `test/optimizers/adam_with_learning_rate_decay.jl`
@@ -442,28 +456,3 @@ What is known to be wrong in GeometricMachineLearning and is not fixed.
   Neither function takes an element type, so neither runs over `REAL_ELTYPES`. The third testset
   of the file, `schedule_starts_at_one`, loops over `REAL_ELTYPES`. The fix builds `dl` per `T`
   and loops both functions over `REAL_ELTYPES`.
-
-### K11 · A comment in `adam_with_learning_rate_decay.jl` reports a past experiment.
-
-- **location:** `test/optimizers/adam_with_learning_rate_decay.jl`
-- **kind:** found late
-- **found:** 2026-10-12, by the critics of part M1a; the comment is older than that part
-- **evidence:**
-  The comment above `train_manifold_network` says "from a seeded start the loss fell by under 2%
-  on 1.13 and *rose* on 1.12". The fix states only the present constraint: a 32-epoch budget
-  drives the learning rate to `η₂` almost at once, and 128 epochs train.
-
-### K12 · `Batch` never draws the last sample of a `DataLoader` of matrices.
-
-- **location:** `src/data_loader/batch.jl`
-- **kind:** found late
-- **found:** 2026-10-12, by the critics of part M1a
-- **evidence:**
-  On a `DataLoader(x, x)` with `x = rand(2, 10)`, the minibatches of `Batch(bs)` hold the samples
-  1 to 9 for each `bs` in 1, 3, 4 and 10; sample 10 is in none of them. `Batch(1)` gives 9
-  minibatches.
-
-  ```
-  1 nbatches 9 samples [(1, 1), (2, 1), (3, 1), (4, 1), (5, 1), (6, 1), (7, 1), (8, 1), (9, 1)]
-  10 nbatches 1 samples [(1, 1), (2, 1), (3, 1), (4, 1), (5, 1), (6, 1), (7, 1), (8, 1), (9, 1)]
-  ```
diff --git a/docs/src/index.md b/docs/src/index.md
index 7c9fe47f7..97de8e070 100644
--- a/docs/src/index.md
+++ b/docs/src/index.md
@@ -54,7 +54,7 @@ Many layers have been adapted in order to be used for problems in scientific mac
 
 **Little of that is tested.** `test/devices/metal.jl` runs two checks on an Apple GPU, in the `metal` test group, which the `Metal` job and the `macOS-latest` job of CI run: the products of a `PoissonTensor` with a wrapped device array, and the `tensor_mat_mul!` kernel. A green test matrix says nothing about the rest of the GPU path. What is written below was measured by hand on an Apple M4 Max through `Metal.jl`, in `Float32`, because Apple GPUs have no `Float64` at all, and against `GeometricOptimizers` 0.8.0.
 
-Everything tried on that device ran. Constructed, and then applied to both a matrix and a 3-tensor: `GSympNet`, `LASympNet`, `StandardTransformerIntegrator`, `LinearSymplecticTransformer`, `SymplecticTransformer` at both its default `transformer_dim` and an upscaling one, `VolumePreservingFeedForward`, `VolumePreservingTransformer`, `SymplecticAutoencoder`, `PSDArch` and `Transformer(…; Stiefel = true)`. `ClassificationTransformer` was constructed but not applied, because its input is an image. The tensor kernels and `map_to_cpu` run, and so does training with `Optimizer`, which returns a `Float32` history and leaves the parameters on the device — for `PSDArch` that includes the manifold weights, which stay a `StiefelManifold` over an `MtlMatrix`.
+Everything tried on that device ran. Constructed, and then applied to both a matrix and a 3-tensor: `GSympNet`, `LASympNet`, `StandardTransformerIntegrator`, `LinearSymplecticTransformer`, `SymplecticTransformer` at both its default `transformer_dim` and an upscaling one, `VolumePreservingFeedForward`, `VolumePreservingTransformer`, `SymplecticAutoencoder`, `PSDArch` and `Transformer(…; Stiefel = true)`. `ClassificationTransformer` was constructed but not applied, because its input is an image. The tensor kernels and `map_to_cpu` run. Training through `train!` has not been run on the device.
 
 Two of those results rest on `GeometricOptimizers` rather than on anything here.
 
diff --git a/scripts/verification/all_zero_leaf.jl b/scripts/verification/all_zero_leaf.jl
index d61c69f15..d36409f53 100644
--- a/scripts/verification/all_zero_leaf.jl
+++ b/scripts/verification/all_zero_leaf.jl
@@ -1,7 +1,7 @@
 # What a training step does to a layer the loss does not read.
 #
-# Under NeuralNetworkParameters 0.5 the gradient of such a layer is a zero leaf of the layer's own
-# type, not `nothing`, and `NetworkOptimizer` hands the whole gradient to
+# NeuralNetworkParameters gives the gradient of such a layer as a zero leaf of the layer's own type,
+# not `nothing`, and `NetworkOptimizer` hands the whole gradient to
 # `GeometricOptimizers.optimization_step!`: GeometricMachineLearning skips no leaf. This script prints
 # how far each leaf moves, in both precisions and under each first-order method:
 #
diff --git a/src/GeometricMachineLearning.jl b/src/GeometricMachineLearning.jl
index 0ee6d09f9..2595faeef 100644
--- a/src/GeometricMachineLearning.jl
+++ b/src/GeometricMachineLearning.jl
@@ -36,8 +36,9 @@ import Symbolics
 # `import` and not `using ...: ...`: GML adds constructor methods to several of these types (in
 # `layers/` and the kernels), and extending a *type* reached through `using` warns on every such
 # method since Julia 1.12 — "Constructor for type … was extended in `GeometricMachineLearning`
-# without explicit qualification or import". Everything imported here is re-exported below, which
-# is what keeps `using GeometricMachineLearning` alone sufficient.
+# without explicit qualification or import". Everything imported here is re-exported below, except
+# `orthonormal_columns`, `TrainingOptimizer` and `optimization_step!`; the re-exports are what keep
+# `using GeometricMachineLearning` alone sufficient.
 import GeometricOptimizers
 import GeometricOptimizers: Manifold, StiefelManifold, GrassmannManifold
 import GeometricOptimizers: SkewSymMatrix, SymmetricMatrix,
diff --git a/test/quality/aqua.jl b/test/quality/aqua.jl
index a0fa85ed6..07230abd9 100644
--- a/test/quality/aqua.jl
+++ b/test/quality/aqua.jl
@@ -59,10 +59,10 @@ end
 #
 # The two `via_owners` checks ask *which* upstream module defines a name, so they go red when a name
 # moves between `GeometricBase`, `GeometricOptimizers` and `AbstractNeuralNetworks` with nothing in
-# `src/` having changed -- which is how `StateVariable` got here. `ExplicitImports = "1.15"` admits
-# any 1.x, so a minor release that sharpens stale detection does the same. A red here names the
-# import and the module to take it from, so it stays a gate: the fix is one line and the message
-# says which. `test_explicit_imports` opens its own `@testset`, so this call needs no wrapper.
+# `src/` having changed. `ExplicitImports = "1.15"` admits any 1.x, so a minor release that sharpens
+# stale detection does the same. A red here names the import and the module to take it from, so it
+# stays a gate: the fix is one line and the message says which. `test_explicit_imports` opens its
+# own `@testset`, so this call needs no wrapper.
 test_explicit_imports(GeometricMachineLearning;
     no_implicit_imports = false,
     all_explicit_imports_are_public = false,

From 69829d211bd39e320e6919d81f20e185af1a706f Mon Sep 17 00:00:00 2001
From: Michael Kraus <michael.kraus@ipp.mpg.de>
Date: Mon, 12 Oct 2026 19:04:36 +0200
Subject: [PATCH 05/11] Fix the second verifier pass's text findings of part
 M1a

The CHANGELOG entry names the known issues C1, C2 and D4 that this
change closes; the all-zero-leaf script header, the B6 note and the GPU
paragraph's compat scope match the code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 CHANGELOG.md                          |  2 +-
 KNOWN_ISSUES.md                       | 12 ++++++------
 docs/src/index.md                     |  2 +-
 scripts/verification/all_zero_leaf.jl |  2 +-
 4 files changed, 9 insertions(+), 9 deletions(-)

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 247b80b3d..81a703c69 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -17,7 +17,7 @@ breaking release).
 
 ### Removed (breaking)
 
-- **`Optimizer`, the callable optimizers and their aliases are gone; training moves to `NetworkOptimizer` and `train!`.** Training now uses GeometricOptimizers 0.9's `TrainingOptimizer`. `NetworkOptimizer(nn; algorithm = Adam(), linesearch = default_step_size(algorithm), retraction = Cayley())`, also accepted from a `NetworkParameters`, replaces `Optimizer`. Migrate `Optimizer(method, nn; step_size = η, retraction = cayley)` to `NetworkOptimizer(nn; algorithm = method, linesearch = η, retraction = Cayley())`, and `Optimizer(nn; AdamOptimizerWithDecay(n, T)...)` to `NetworkOptimizer(nn; AdamOptimizerWithDecay(n)...)`. The `n` of `AdamOptimizerWithDecay` counts optimization steps, one per minibatch, as in GeometricOptimizers and GML 0.8. One cache and one state now cover the whole parameter set; 0.8 had one per layer. `using GeometricOptimizers, GeometricMachineLearning` leaves `Optimizer` unqualified as GeometricOptimizers' solver, and no name is shadowed. `train!(nn, opt, dl, batch, n_epochs; loss, show_progress = true)` replaces the four callable methods `(o::Optimizer)(nn, dl, batch, n_epochs[, loss, pullback])` and `optimize_for_one_epoch!`, which is removed from the exports. Migrate `o(nn, dl, batch, n, loss; kw...)` to `train!(nn, o, dl, batch, n; loss, kw...)`. Without `loss` the defaults are those of 0.8: `TransformerLoss(batch)` for a `TransformerIntegrator` with a `Batch{:Transformer}`, `FeedForwardLoss()` for a `NeuralNetworkIntegrator`, and `AutoEncoderLoss()` for an `AutoEncoder`, each with a `Batch{:FeedForward}`. `loss` is a `NetworkLoss`; passing an `AbstractPullback` as the sixth positional argument, which 0.8 allowed, is gone. The `λY` argument and the per-call `GlobalSection` are gone: the sections, moments and iteration number live in `opt`, so a second `train!` call with the same `opt` resumes the first, and `train!` draws no random numbers beyond the minibatch shuffle. `n_epochs < 1` raises an `ArgumentError` before any step. The loss history has the element type of the loss value, so a `Float64` network on `Float32` data gives `Vector{Float64}`, where 0.8 gave `float(eltype(dl))`; the epoch accumulator keeps that type. A hand-written loop steps with `GeometricOptimizers.optimization_step!(params(nn), opt, dp)`, which GML adds for `NetworkOptimizer`; `optimization_step!(opt, λY, ps, dp)` is removed. A gradient leaf that is all zero is stepped, not skipped: NeuralNetworkParameters 0.5 gives a layer the loss does not read as a zero leaf, not `nothing`. A leaf that the loss never reads stays where it is under `GradientMethod`, `MomentumMethod` and `Adam`; under `AdamWithEuclideanDecay` such a Euclidean leaf decays. Under `MomentumMethod`, `Adam` and `AdamWithEuclideanDecay`, a leaf that the loss reads for some steps and then not keeps moving on its momentum. `scripts/verification/all_zero_leaf.jl` prints the figures.
+- **`Optimizer`, the callable optimizers and their aliases are gone; training moves to `NetworkOptimizer` and `train!`.** Training now uses GeometricOptimizers 0.9's `TrainingOptimizer`. `NetworkOptimizer(nn; algorithm = Adam(), linesearch = default_step_size(algorithm), retraction = Cayley())`, also accepted from a `NetworkParameters`, replaces `Optimizer`. Migrate `Optimizer(method, nn; step_size = η, retraction = cayley)` to `NetworkOptimizer(nn; algorithm = method, linesearch = η, retraction = Cayley())`, and `Optimizer(nn; AdamOptimizerWithDecay(n, T)...)` to `NetworkOptimizer(nn; AdamOptimizerWithDecay(n)...)`. The `n` of `AdamOptimizerWithDecay` counts optimization steps, one per minibatch, as in GeometricOptimizers and GML 0.8. One cache and one state now cover the whole parameter set; 0.8 had one per layer. `using GeometricOptimizers, GeometricMachineLearning` leaves `Optimizer` unqualified as GeometricOptimizers' solver, and no name is shadowed. `train!(nn, opt, dl, batch, n_epochs; loss, show_progress = true)` replaces the four callable methods `(o::Optimizer)(nn, dl, batch, n_epochs[, loss, pullback])` and `optimize_for_one_epoch!`, which is removed from the exports. Migrate `o(nn, dl, batch, n, loss; kw...)` to `train!(nn, o, dl, batch, n; loss, kw...)`. Without `loss` the defaults are those of 0.8: `TransformerLoss(batch)` for a `TransformerIntegrator` with a `Batch{:Transformer}`, `FeedForwardLoss()` for a `NeuralNetworkIntegrator`, and `AutoEncoderLoss()` for an `AutoEncoder`, each with a `Batch{:FeedForward}`. `loss` is a `NetworkLoss`; passing an `AbstractPullback` as the sixth positional argument, which 0.8 allowed, is gone. The `λY` argument and the per-call `GlobalSection` are gone: the sections, moments and iteration number live in `opt`, so a second `train!` call with the same `opt` resumes the first, and `train!` draws no random numbers beyond the minibatch shuffle. `n_epochs < 1` raises an `ArgumentError` before any step. The loss history has the element type of the loss value, so a `Float64` network on `Float32` data gives `Vector{Float64}`, where 0.8 gave `float(eltype(dl))`; the epoch accumulator keeps that type. A hand-written loop steps with `GeometricOptimizers.optimization_step!(params(nn), opt, dp)`, which GML adds for `NetworkOptimizer`; `optimization_step!(opt, λY, ps, dp)` is removed. A gradient leaf that is all zero is stepped, not skipped: NeuralNetworkParameters 0.5 gives a layer the loss does not read as a zero leaf, not `nothing`. A leaf that the loss never reads stays where it is under `GradientMethod`, `MomentumMethod` and `Adam`; under `AdamWithEuclideanDecay` such a Euclidean leaf decays. Under `MomentumMethod`, `Adam` and `AdamWithEuclideanDecay`, a leaf that the loss reads for some steps and then not keeps moving on its momentum. `scripts/verification/all_zero_leaf.jl` prints the figures. This closes known issues C1 and C2, because GML's own parameter-tree traversal and `_leaf_optim_step!` are deleted, and D4, because the per-layer caches it measured are deleted.
 
 - **The aliases `GradientOptimizer`, `MomentumOptimizer` and `AdamOptimizer` are removed**; use `GradientMethod`, `MomentumMethod` and `Adam`. These are no longer exported: `OptimizerMethod`, `GradientState`, `MomentumState`, `AdamState`, `optimization_step!`, `optimize_for_one_epoch!` and `Optimizer`. `NetworkOptimizer`, `train!` and `AdamWithEuclideanDecay` are newly exported, and the package re-exports `GradientMethod`, `MomentumMethod`, `Adam`, `AdamOptimizerWithDecay`, `DecayingStatic`, `Cayley` and `Geodesic` from GeometricOptimizers. Methods carry no element type in GeometricOptimizers 0.9: `Adam(T)` becomes `Adam()`, and `MomentumMethod(α)` becomes `MomentumMethod(; α)`. `check(::Optimizer)` is removed.
 
diff --git a/KNOWN_ISSUES.md b/KNOWN_ISSUES.md
index 215d51650..5fe6041ab 100644
--- a/KNOWN_ISSUES.md
+++ b/KNOWN_ISSUES.md
@@ -34,12 +34,12 @@ What is known to be wrong in GeometricMachineLearning and is not fixed.
   (**B1**, **B2**, **B3**, **B4**, **B6** and **B8** are all closed and their entries are gone: B1 and B2 by
   this release — the duplicated `AdamOptimizerWithDecay` and the split `Manifold`, both under
   *Removed (breaking)* — B3 by SymbolicNeuralNetworks 0.5, B4 by `a427add1`, which repaired the
-  documentation build, and B6 and B8 by this release: B6 by the retirement of the `train!` subsystem (the three methods that
-  called the non-existent `vectorfield` are gone, and `SymplecticEulerLoss` carries their content on
-  `hamiltonian_vector_field`, with tests that run), and B8 by the narrowing of `PoissonTensor`'s `Base.:*`
-  to `Strided…` right-hand sides and `Base.getindex` to `Union{Int, Colon, AbstractVector{<:Integer}}`
-  indices, reducing method ambiguities from 18 to 1. The numbers are
-  left vacant rather than reused.)
+  documentation build, and B6 and B8 by this release: B6 by the deletion of `src/training_method/`
+  (the three methods that called the non-existent `vectorfield` are gone, and `SymplecticEulerLoss`
+  carries their content on `hamiltonian_vector_field`, with tests that run), and B8 by the narrowing
+  of `PoissonTensor`'s `Base.:*` to `Strided…` right-hand sides and `Base.getindex` to
+  `Union{Int, Colon, AbstractVector{<:Integer}}` indices, reducing method ambiguities from 18 to 1.
+  The numbers are left vacant rather than reused.)
 
   (**B9** is closed by this release and its entry is gone: the `iterate` method it named binds
   every type parameter, and Aqua's `unbound_args` passes on Julia nightly. See *Fixed* above. The
diff --git a/docs/src/index.md b/docs/src/index.md
index 97de8e070..1a9f4caac 100644
--- a/docs/src/index.md
+++ b/docs/src/index.md
@@ -52,7 +52,7 @@ Many layers have been adapted in order to be used for problems in scientific mac
 
 `GeometricMachineLearning` allocates and computes through `KernelAbstractions.jl` [churavy2020kernel](@cite), so its layers and architectures are written against any backend that package supports: `CUDA.jl` [besard2018juliagpu](@cite), `AMDGPU.jl`, `Metal.jl` [besard2022metal](@cite) and `oneAPI.jl` [besard2022one](@cite).
 
-**Little of that is tested.** `test/devices/metal.jl` runs two checks on an Apple GPU, in the `metal` test group, which the `Metal` job and the `macOS-latest` job of CI run: the products of a `PoissonTensor` with a wrapped device array, and the `tensor_mat_mul!` kernel. A green test matrix says nothing about the rest of the GPU path. What is written below was measured by hand on an Apple M4 Max through `Metal.jl`, in `Float32`, because Apple GPUs have no `Float64` at all, and against `GeometricOptimizers` 0.8.0.
+**Little of that is tested.** `test/devices/metal.jl` runs two checks on an Apple GPU, in the `metal` test group, which the `Metal` job and the `macOS-latest` job of CI run: the products of a `PoissonTensor` with a wrapped device array, and the `tensor_mat_mul!` kernel. A green test matrix says nothing about the rest of the GPU path. What is written below was measured by hand on an Apple M4 Max through `Metal.jl`, in `Float32`, because Apple GPUs have no `Float64` at all, and against `GeometricOptimizers` 0.8.0, which the package's `[compat]` bound of `0.9` excludes.
 
 Everything tried on that device ran. Constructed, and then applied to both a matrix and a 3-tensor: `GSympNet`, `LASympNet`, `StandardTransformerIntegrator`, `LinearSymplecticTransformer`, `SymplecticTransformer` at both its default `transformer_dim` and an upscaling one, `VolumePreservingFeedForward`, `VolumePreservingTransformer`, `SymplecticAutoencoder`, `PSDArch` and `Transformer(…; Stiefel = true)`. `ClassificationTransformer` was constructed but not applied, because its input is an image. The tensor kernels and `map_to_cpu` run. Training through `train!` has not been run on the device.
 
diff --git a/scripts/verification/all_zero_leaf.jl b/scripts/verification/all_zero_leaf.jl
index d36409f53..9ce73954c 100644
--- a/scripts/verification/all_zero_leaf.jl
+++ b/scripts/verification/all_zero_leaf.jl
@@ -9,7 +9,7 @@
 #     `GradientMethod`, `MomentumMethod` and `Adam`; under `AdamWithEuclideanDecay` the Euclidean one
 #     decays and the Stiefel one does not;
 #   - a leaf the loss reads for the first 5 steps and not after keeps moving on its momentum under
-#     `MomentumMethod` and `Adam`;
+#     `MomentumMethod`, `Adam` and `AdamWithEuclideanDecay`, and stays under `GradientMethod`;
 #   - a layer without numbers, `NamedTuple()`, steps without an error.
 #
 # Run from the repository root, with the test environment:

From 826ae32ec0a60581e9b9ed5c85bc894d4a811c3a Mon Sep 17 00:00:00 2001
From: Michael Kraus <michael.kraus@ipp.mpg.de>
Date: Mon, 12 Oct 2026 20:18:06 +0200
Subject: [PATCH 06/11] Fix the Scripts and Documentation CI failures of the
 training-API migration

- scripts/verification/all_zero_leaf.jl imports optimization_step! through
  GeometricMachineLearning, as sympnet_pendulum.jl does: the scripts
  environment that runscripts.jl uses has no GeometricOptimizers entry.
  The header names the scripts environment.
- docs/inventories/GeometricOptimizers.toml is regenerated from the
  deployed v0.9.0 objects.inv, the version [compat] requires, with the
  command in docs/make.jl, which now pins v0.9.0. The 0.5.0 copy has no
  StrictlyLowerTriangular or StrictlyUpperTriangular, so the eight
  @extref links to them failed when the build fell back to it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 docs/inventories/GeometricOptimizers.toml | 540 ++++++++++++++++------
 docs/make.jl                              |   2 +-
 scripts/verification/all_zero_leaf.jl     |   7 +-
 3 files changed, 405 insertions(+), 144 deletions(-)

diff --git a/docs/inventories/GeometricOptimizers.toml b/docs/inventories/GeometricOptimizers.toml
index 0ecded68a..247ede361 100644
--- a/docs/inventories/GeometricOptimizers.toml
+++ b/docs/inventories/GeometricOptimizers.toml
@@ -1,6 +1,6 @@
 # DocInventory version 1
 project = "GeometricOptimizers.jl"
-version = "0.5.0"
+version = "0.9.0"
 
 [[jl.constant]]
 name = "GeometricOptimizers.CURVATURE_TOLERANCE"
@@ -9,6 +9,9 @@ uri = "api/#$"
 name = "GeometricOptimizers.DEFAULT_STEP_CEILING"
 uri = "api/#$"
 
+[[jl.function]]
+name = "GeometricOptimizers.default_eltype"
+uri = "api/#$"
 [[jl.function]]
 name = "GeometricOptimizers.geodesic"
 uri = "api/#$"
@@ -16,13 +19,31 @@ uri = "api/#$"
 name = "GeometricOptimizers.manifold_type"
 uri = "api/#$"
 [[jl.function]]
-name = "GeometricOptimizers.𝔄exp"
-uri = "api/#GeometricOptimizers.%F0%9D%94%84exp"
+name = "GeometricOptimizers.retraction_workspace"
+uri = "api/#$"
+[[jl.function]]
+name = "GeometricOptimizers.symplectic_gram_schmidt"
+uri = "api/#$"
+[[jl.function]]
+name = "GeometricOptimizers.symplectic_gram_schmidt!"
+uri = "api/#GeometricOptimizers.symplectic_gram_schmidt%21"
+[[jl.function]]
+name = "GeometricOptimizers.trial_iterate!"
+uri = "api/#GeometricOptimizers.trial_iterate%21"
 
+[[jl.method]]
+name = "Base.:*-Tuple{AbstractLieAlgHorMatrix, AbstractMatrix}"
+uri = "api/#Base.%3A%2A-Tuple%7BAbstractLieAlgHorMatrix%2C%20AbstractMatrix%7D"
+[[jl.method]]
+name = "Base.:*-Tuple{AbstractTriangular, AbstractMatrix}"
+uri = "api/#Base.%3A%2A-Tuple%7BAbstractTriangular%2C%20AbstractMatrix%7D"
 [[jl.method]]
 name = "Base.:*-Tuple{GlobalSection, Manifold}"
 uri = "api/#Base.%3A%2A-Tuple%7BGlobalSection%2C%20Manifold%7D"
 [[jl.method]]
+name = "Base.:*-Tuple{StiefelProjection, AbstractMatrix}"
+uri = "api/#Base.%3A%2A-Tuple%7BStiefelProjection%2C%20AbstractMatrix%7D"
+[[jl.method]]
 name = "Base.Matrix-Tuple{GlobalSection}"
 uri = "api/#Base.Matrix-Tuple%7BGlobalSection%7D"
 [[jl.method]]
@@ -32,44 +53,29 @@ uri = "api/#Base.one-Union%7BTuple%7BAbstractLieAlgHorMatrix%7BT%7D%7D%2C%20Tupl
 name = "Base.parent-Tuple{AbstractLieAlgHorMatrix}"
 uri = "api/#Base.parent-Tuple%7BAbstractLieAlgHorMatrix%7D"
 [[jl.method]]
-name = "Base.rand-Union{Tuple{MT}, Tuple{KernelAbstractions.Backend, Type{MT}, Integer, Integer}} where MT<:Manifold"
-uri = "api/#Base.rand-Union%7BTuple%7BMT%7D%2C%20Tuple%7BKernelAbstractions.Backend%2C%20Type%7BMT%7D%2C%20Integer%2C%20Integer%7D%7D%20where%20MT%3C%3AManifold"
-[[jl.method]]
-name = "Base.rand-Union{Tuple{MT}, Tuple{Type{MT}, Integer, Integer}} where MT<:Manifold"
-uri = "api/#Base.rand-Union%7BTuple%7BMT%7D%2C%20Tuple%7BType%7BMT%7D%2C%20Integer%2C%20Integer%7D%7D%20where%20MT%3C%3AManifold"
+name = "Base.rand-Tuple{KernelAbstractions.Backend, Type{<:Manifold}, Integer, Integer}"
+uri = "api/#Base.rand-Tuple%7BKernelAbstractions.Backend%2C%20Type%7B%3C%3AManifold%7D%2C%20Integer%2C%20Integer%7D"
 [[jl.method]]
-name = "Base.vec-Tuple{AbstractLieAlgHorMatrix}"
-uri = "api/#Base.vec-Tuple%7BAbstractLieAlgHorMatrix%7D"
+name = "Base.rand-Tuple{Type{<:Manifold}, Integer, Integer}"
+uri = "api/#Base.rand-Tuple%7BType%7B%3C%3AManifold%7D%2C%20Integer%2C%20Integer%7D"
 [[jl.method]]
-name = "Base.vec-Tuple{AbstractTriangular}"
-uri = "api/#Base.vec-Tuple%7BAbstractTriangular%7D"
-[[jl.method]]
-name = "Base.vec-Tuple{SkewSymMatrix}"
-uri = "api/#Base.vec-Tuple%7BSkewSymMatrix%7D"
+name = "Base.rand-Tuple{Type{<:SymplecticStiefelManifold}, Integer, Integer}"
+uri = "api/#Base.rand-Tuple%7BType%7B%3C%3ASymplecticStiefelManifold%7D%2C%20Integer%2C%20Integer%7D"
 [[jl.method]]
 name = "GeometricBase.update!-Tuple{GeometricOptimizers.NewtonOptimizerCache, OptimizerState, Gradient, Hessian, AbstractVector}"
 uri = "api/#GeometricBase.update%21-Tuple%7BGeometricOptimizers.NewtonOptimizerCache%2C%20OptimizerState%2C%20Gradient%2C%20Hessian%2C%20AbstractVector%7D"
 [[jl.method]]
-name = "GeometricBase.update!-Tuple{NewtonOptimizerState, Gradient, AbstractVector}"
-uri = "api/#GeometricBase.update%21-Tuple%7BNewtonOptimizerState%2C%20Gradient%2C%20AbstractVector%7D"
-[[jl.method]]
-name = "GeometricBase.update!-Union{Tuple{T}, Tuple{GeometricOptimizers.BFGSCache{T}, BFGSState{T}, OptimizerSolution{T}, Union{AbstractArray{T}, NamedTuple{S, <:Tuple{Vararg{AbstractArray{T}}}} where S}}} where T"
-uri = "api/#GeometricBase.update%21-Union%7BTuple%7BT%7D%2C%20Tuple%7BGeometricOptimizers.BFGSCache%7BT%7D%2C%20BFGSState%7BT%7D%2C%20OptimizerSolution%7BT%7D%2C%20Union%7BAbstractArray%7BT%7D%2C%20NamedTuple%7BS%2C%20%3C%3ATuple%7BVararg%7BAbstractArray%7BT%7D%7D%7D%7D%20where%20S%7D%7D%7D%20where%20T"
-[[jl.method]]
-name = "GeometricBase.update!-Union{Tuple{T}, Tuple{GeometricOptimizers.DFPCache{T}, BFGSState{T}, OptimizerSolution{T}, Union{AbstractArray{T}, NamedTuple{S, <:Tuple{Vararg{AbstractArray{T}}}} where S}}} where T"
-uri = "api/#GeometricBase.update%21-Union%7BTuple%7BT%7D%2C%20Tuple%7BGeometricOptimizers.DFPCache%7BT%7D%2C%20BFGSState%7BT%7D%2C%20OptimizerSolution%7BT%7D%2C%20Union%7BAbstractArray%7BT%7D%2C%20NamedTuple%7BS%2C%20%3C%3ATuple%7BVararg%7BAbstractArray%7BT%7D%7D%7D%7D%20where%20S%7D%7D%7D%20where%20T"
+name = "GeometricBase.update!-Tuple{NewtonState, Gradient, AbstractVector}"
+uri = "api/#GeometricBase.update%21-Tuple%7BNewtonState%2C%20Gradient%2C%20AbstractVector%7D"
 [[jl.method]]
-name = "GeometricOptimizers.AdamOptimizerWithDecay-Union{Tuple{Integer}, Tuple{T}, Tuple{Integer, Type{T}}} where T"
-uri = "api/#GeometricOptimizers.AdamOptimizerWithDecay-Union%7BTuple%7BInteger%7D%2C%20Tuple%7BT%7D%2C%20Tuple%7BInteger%2C%20Type%7BT%7D%7D%7D%20where%20T"
-[[jl.method]]
-name = "GeometricOptimizers.AdamW-Tuple"
-uri = "api/#$"
+name = "GeometricBase.update!-Union{Tuple{T}, Tuple{GeometricOptimizers.QuasiNewtonCache{T}, BFGSState{T}, OptimizerSolution{T}, Union{AbstractArray{T}, NetworkParameters{T}}}} where T"
+uri = "api/#GeometricBase.update%21-Union%7BTuple%7BT%7D%2C%20Tuple%7BGeometricOptimizers.QuasiNewtonCache%7BT%7D%2C%20BFGSState%7BT%7D%2C%20OptimizerSolution%7BT%7D%2C%20Union%7BAbstractArray%7BT%7D%2C%20NetworkParameters%7BT%7D%7D%7D%7D%20where%20T"
 [[jl.method]]
-name = "GeometricOptimizers.GrassmannLieAlgHorMatrix-Tuple{AbstractMatrix, Int64}"
-uri = "api/#GeometricOptimizers.GrassmannLieAlgHorMatrix-Tuple%7BAbstractMatrix%2C%20Int64%7D"
+name = "GeometricOptimizers.AdamOptimizerWithDecay-Tuple{Integer}"
+uri = "api/#GeometricOptimizers.AdamOptimizerWithDecay-Tuple%7BInteger%7D"
 [[jl.method]]
-name = "GeometricOptimizers.LowerTriangular-Union{Tuple{AbstractMatrix{T}}, Tuple{T}} where T"
-uri = "api/#GeometricOptimizers.LowerTriangular-Union%7BTuple%7BAbstractMatrix%7BT%7D%7D%2C%20Tuple%7BT%7D%7D%20where%20T"
+name = "GeometricOptimizers.GrassmannLieAlgHorMatrix-Tuple{AbstractMatrix, Integer}"
+uri = "api/#GeometricOptimizers.GrassmannLieAlgHorMatrix-Tuple%7BAbstractMatrix%2C%20Integer%7D"
 [[jl.method]]
 name = "GeometricOptimizers.Optimizer-Union{Tuple{VT}, Tuple{T}, Tuple{VT, Function}} where {T, VT<:OptimizerSolution{T}}"
 uri = "api/#GeometricOptimizers.Optimizer-Union%7BTuple%7BVT%7D%2C%20Tuple%7BT%7D%2C%20Tuple%7BVT%2C%20Function%7D%7D%20where%20%7BT%2C%20VT%3C%3AOptimizerSolution%7BT%7D%7D"
@@ -86,45 +92,75 @@ uri = "api/#GeometricOptimizers.StiefelProjection-Union%7BTuple%7BAbstractLieAlg
 name = "GeometricOptimizers.StiefelProjection-Union{Tuple{AbstractMatrix{T}}, Tuple{T}} where T"
 uri = "api/#GeometricOptimizers.StiefelProjection-Union%7BTuple%7BAbstractMatrix%7BT%7D%7D%2C%20Tuple%7BT%7D%7D%20where%20T"
 [[jl.method]]
+name = "GeometricOptimizers.StrictlyLowerTriangular-Union{Tuple{AbstractMatrix{T}}, Tuple{T}} where T"
+uri = "api/#GeometricOptimizers.StrictlyLowerTriangular-Union%7BTuple%7BAbstractMatrix%7BT%7D%7D%2C%20Tuple%7BT%7D%7D%20where%20T"
+[[jl.method]]
+name = "GeometricOptimizers.StrictlyUpperTriangular-Union{Tuple{AbstractMatrix{T}}, Tuple{T}} where T"
+uri = "api/#GeometricOptimizers.StrictlyUpperTriangular-Union%7BTuple%7BAbstractMatrix%7BT%7D%7D%2C%20Tuple%7BT%7D%7D%20where%20T"
+[[jl.method]]
 name = "GeometricOptimizers.SymmetricMatrix-Union{Tuple{AbstractMatrix{T}}, Tuple{T}} where T"
 uri = "api/#GeometricOptimizers.SymmetricMatrix-Union%7BTuple%7BAbstractMatrix%7BT%7D%7D%2C%20Tuple%7BT%7D%7D%20where%20T"
 [[jl.method]]
-name = "GeometricOptimizers.UpperTriangular-Union{Tuple{AbstractMatrix{T}}, Tuple{T}} where T"
-uri = "api/#GeometricOptimizers.UpperTriangular-Union%7BTuple%7BAbstractMatrix%7BT%7D%7D%2C%20Tuple%7BT%7D%7D%20where%20T"
-[[jl.method]]
-name = "GeometricOptimizers._div!-Tuple{AbstractArray, AbstractArray, AbstractArray}"
-uri = "api/#GeometricOptimizers._div%21-Tuple%7BAbstractArray%2C%20AbstractArray%2C%20AbstractArray%7D"
+name = "GeometricOptimizers._cholesky_qr2-Tuple{AbstractMatrix}"
+uri = "api/#GeometricOptimizers._cholesky_qr2-Tuple%7BAbstractMatrix%7D"
 [[jl.method]]
 name = "GeometricOptimizers._dot-Tuple{AbstractVecOrMat, AbstractVecOrMat}"
 uri = "api/#GeometricOptimizers._dot-Tuple%7BAbstractVecOrMat%2C%20AbstractVecOrMat%7D"
 [[jl.method]]
+name = "GeometricOptimizers._flat_mul!-Tuple{Any, Any, Any, Nothing}"
+uri = "api/#GeometricOptimizers._flat_mul%21-Tuple%7BAny%2C%20Any%2C%20Any%2C%20Nothing%7D"
+[[jl.method]]
+name = "GeometricOptimizers._flat_scratch-Union{Tuple{T}, Tuple{Type{T}, AbstractVector}} where T"
+uri = "api/#GeometricOptimizers._flat_scratch-Union%7BTuple%7BT%7D%2C%20Tuple%7BType%7BT%7D%2C%20AbstractVector%7D%7D%20where%20T"
+[[jl.method]]
 name = "GeometricOptimizers._is_decayable-Tuple{StiefelManifold}"
 uri = "api/#GeometricOptimizers._is_decayable-Tuple%7BStiefelManifold%7D"
 [[jl.method]]
-name = "GeometricOptimizers._manifold_αmax-Union{Tuple{T}, Tuple{Tuple{}, Tuple{}, T}} where T"
-uri = "api/#GeometricOptimizers._manifold_%CE%B1max-Union%7BTuple%7BT%7D%2C%20Tuple%7BTuple%7B%7D%2C%20Tuple%7B%7D%2C%20T%7D%7D%20where%20T"
+name = "GeometricOptimizers._manifold_αmax-Union{Tuple{T}, Tuple{NetworkParameters, Any, T}} where T"
+uri = "api/#GeometricOptimizers._manifold_%CE%B1max-Union%7BTuple%7BT%7D%2C%20Tuple%7BNetworkParameters%2C%20Any%2C%20T%7D%7D%20where%20T"
 [[jl.method]]
-name = "GeometricOptimizers._optimizer-Union{Tuple{T}, Tuple{OptimizerSolution{T}, OptimizerProblem{T}, OptimizerMethod, LinesearchMethod, Gradient{T}, AbstractRetraction, Options{T}, Real}} where T"
-uri = "api/#GeometricOptimizers._optimizer-Union%7BTuple%7BT%7D%2C%20Tuple%7BOptimizerSolution%7BT%7D%2C%20OptimizerProblem%7BT%7D%2C%20OptimizerMethod%2C%20LinesearchMethod%2C%20Gradient%7BT%7D%2C%20AbstractRetraction%2C%20Options%7BT%7D%2C%20Real%7D%7D%20where%20T"
+name = "GeometricOptimizers._native_pade_polynomials-Tuple{AbstractMatrix, AbstractMatrix}"
+uri = "api/#GeometricOptimizers._native_pade_polynomials-Tuple%7BAbstractMatrix%2C%20AbstractMatrix%7D"
 [[jl.method]]
-name = "GeometricOptimizers._rac!-Tuple{AbstractArray, AbstractArray}"
-uri = "api/#GeometricOptimizers._rac%21-Tuple%7BAbstractArray%2C%20AbstractArray%7D"
+name = "GeometricOptimizers._optimizer-Union{Tuple{T}, Tuple{OptimizerSolution{T}, OptimizerProblem{T}, OptimizerMethod, Union{Real, LinesearchMethod}, Gradient{T}, AbstractRetraction, Options{T}, Real, Any}} where T"
+uri = "api/#GeometricOptimizers._optimizer-Union%7BTuple%7BT%7D%2C%20Tuple%7BOptimizerSolution%7BT%7D%2C%20OptimizerProblem%7BT%7D%2C%20OptimizerMethod%2C%20Union%7BReal%2C%20LinesearchMethod%7D%2C%20Gradient%7BT%7D%2C%20AbstractRetraction%2C%20Options%7BT%7D%2C%20Real%2C%20Any%7D%7D%20where%20T"
 [[jl.method]]
-name = "GeometricOptimizers._square!-Tuple{AbstractArray, AbstractArray}"
-uri = "api/#GeometricOptimizers._square%21-Tuple%7BAbstractArray%2C%20AbstractArray%7D"
+name = "GeometricOptimizers._scaled_kernel!-Tuple{Any, AbstractMatrix, GeometricOptimizers.ScaledSquaring}"
+uri = "api/#GeometricOptimizers._scaled_kernel%21-Tuple%7BAny%2C%20AbstractMatrix%2C%20GeometricOptimizers.ScaledSquaring%7D"
 [[jl.method]]
 name = "GeometricOptimizers._squared_gradient_norm-Tuple{ScalarMomentAdam, GeometricOptimizers.ScalarMomentAdamCache, Gradient, StiefelManifold}"
 uri = "api/#GeometricOptimizers._squared_gradient_norm-Tuple%7BScalarMomentAdam%2C%20GeometricOptimizers.ScalarMomentAdamCache%2C%20Gradient%2C%20StiefelManifold%7D"
 [[jl.method]]
+name = "GeometricOptimizers._sumsq_leaves-Union{Tuple{F}, Tuple{F, Any}} where F"
+uri = "api/#GeometricOptimizers._sumsq_leaves-Union%7BTuple%7BF%7D%2C%20Tuple%7BF%2C%20Any%7D%7D%20where%20F"
+[[jl.method]]
 name = "GeometricOptimizers._weight_decay!-Union{Tuple{T}, Tuple{AbstractArray{T}, AbstractArray{T}, T}} where T"
 uri = "api/#GeometricOptimizers._weight_decay%21-Union%7BTuple%7BT%7D%2C%20Tuple%7BAbstractArray%7BT%7D%2C%20AbstractArray%7BT%7D%2C%20T%7D%7D%20where%20T"
 [[jl.method]]
+name = "GeometricOptimizers.advance_state!-Tuple{GradientState, GeometricOptimizers.GradientCache, GradientMethod}"
+uri = "api/#GeometricOptimizers.advance_state%21-Tuple%7BGradientState%2C%20GeometricOptimizers.GradientCache%2C%20GradientMethod%7D"
+[[jl.method]]
+name = "GeometricOptimizers.apply_S_inverse_left!-Tuple{AbstractVector, GeometricOptimizers.SymplecticHouseholderDecom}"
+uri = "api/#GeometricOptimizers.apply_S_inverse_left%21-Tuple%7BAbstractVector%2C%20GeometricOptimizers.SymplecticHouseholderDecom%7D"
+[[jl.method]]
+name = "GeometricOptimizers.apply_S_inverse_right!-Tuple{AbstractMatrix, GeometricOptimizers.SymplecticHouseholderDecom}"
+uri = "api/#GeometricOptimizers.apply_S_inverse_right%21-Tuple%7BAbstractMatrix%2C%20GeometricOptimizers.SymplecticHouseholderDecom%7D"
+[[jl.method]]
+name = "GeometricOptimizers.apply_S_left!-Tuple{AbstractVector, GeometricOptimizers.SymplecticHouseholderDecom}"
+uri = "api/#GeometricOptimizers.apply_S_left%21-Tuple%7BAbstractVector%2C%20GeometricOptimizers.SymplecticHouseholderDecom%7D"
+[[jl.method]]
+name = "GeometricOptimizers.apply_S_right!-Tuple{AbstractMatrix, GeometricOptimizers.SymplecticHouseholderDecom}"
+uri = "api/#GeometricOptimizers.apply_S_right%21-Tuple%7BAbstractMatrix%2C%20GeometricOptimizers.SymplecticHouseholderDecom%7D"
+[[jl.method]]
 name = "GeometricOptimizers.apply_section!-Union{Tuple{MT}, Tuple{AT}, Tuple{T}, Tuple{AT, GlobalSection{T, AT, λT} where λT<:Union{Nothing, AbstractArray{T}}, MT}} where {T, AT<:(StiefelManifold{T, AT} where AT<:AbstractMatrix{T}), MT<:(StiefelManifold{T, AT} where AT<:AbstractMatrix{T})}"
 uri = "api/#GeometricOptimizers.apply_section%21-Union%7BTuple%7BMT%7D%2C%20Tuple%7BAT%7D%2C%20Tuple%7BT%7D%2C%20Tuple%7BAT%2C%20GlobalSection%7BT%2C%20AT%2C%20%CE%BBT%7D%20where%20%CE%BBT%3C%3AUnion%7BNothing%2C%20AbstractArray%7BT%7D%7D%2C%20MT%7D%7D%20where%20%7BT%2C%20AT%3C%3A%28StiefelManifold%7BT%2C%20AT%7D%20where%20AT%3C%3AAbstractMatrix%7BT%7D%29%2C%20MT%3C%3A%28StiefelManifold%7BT%2C%20AT%7D%20where%20AT%3C%3AAbstractMatrix%7BT%7D%29%7D"
 [[jl.method]]
 name = "GeometricOptimizers.apply_section-Union{Tuple{AT}, Tuple{T}, Tuple{GlobalSection{T, AT, λT} where λT<:Union{Nothing, AbstractArray{T}}, AT}} where {T, AT<:(StiefelManifold{T, AT} where AT<:AbstractMatrix{T})}"
 uri = "api/#GeometricOptimizers.apply_section-Union%7BTuple%7BAT%7D%2C%20Tuple%7BT%7D%2C%20Tuple%7BGlobalSection%7BT%2C%20AT%2C%20%CE%BBT%7D%20where%20%CE%BBT%3C%3AUnion%7BNothing%2C%20AbstractArray%7BT%7D%7D%2C%20AT%7D%7D%20where%20%7BT%2C%20AT%3C%3A%28StiefelManifold%7BT%2C%20AT%7D%20where%20AT%3C%3AAbstractMatrix%7BT%7D%29%7D"
 [[jl.method]]
+name = "GeometricOptimizers.cayley-Tuple{AbstractVecOrMat}"
+uri = "api/#GeometricOptimizers.cayley-Tuple%7BAbstractVecOrMat%7D"
+[[jl.method]]
 name = "GeometricOptimizers.cayley-Tuple{GrassmannLieAlgHorMatrix}"
 uri = "api/#GeometricOptimizers.cayley-Tuple%7BGrassmannLieAlgHorMatrix%7D"
 [[jl.method]]
@@ -137,6 +173,9 @@ uri = "api/#GeometricOptimizers.cayley-Union%7BTuple%7BT%7D%2C%20Tuple%7BManifol
 name = "GeometricOptimizers.check-Tuple{Manifold}"
 uri = "api/#GeometricOptimizers.check-Tuple%7BManifold%7D"
 [[jl.method]]
+name = "GeometricOptimizers.check-Tuple{SymplecticStiefelManifold}"
+uri = "api/#GeometricOptimizers.check-Tuple%7BSymplecticStiefelManifold%7D"
+[[jl.method]]
 name = "GeometricOptimizers.contains_nonfinite-Tuple{Real}"
 uri = "api/#GeometricOptimizers.contains_nonfinite-Tuple%7BReal%7D"
 [[jl.method]]
@@ -152,6 +191,9 @@ uri = "api/#GeometricOptimizers.default_gradient-Union%7BTuple%7BT%7D%2C%20Tuple
 name = "GeometricOptimizers.default_linesearch-Union{Tuple{T}, Tuple{Type{T}, OptimizerMethod}} where T"
 uri = "api/#GeometricOptimizers.default_linesearch-Union%7BTuple%7BT%7D%2C%20Tuple%7BType%7BT%7D%2C%20OptimizerMethod%7D%7D%20where%20T"
 [[jl.method]]
+name = "GeometricOptimizers.default_step_size-Tuple{Union{GradientMethod, MomentumMethod}}"
+uri = "api/#GeometricOptimizers.default_step_size-Tuple%7BUnion%7BGradientMethod%2C%20MomentumMethod%7D%7D"
+[[jl.method]]
 name = "GeometricOptimizers.ensure_descent!-Tuple{GeometricOptimizers.OptimizerCache, OptimizerMethod, Options}"
 uri = "api/#GeometricOptimizers.ensure_descent%21-Tuple%7BGeometricOptimizers.OptimizerCache%2C%20OptimizerMethod%2C%20Options%7D"
 [[jl.method]]
@@ -164,20 +206,17 @@ uri = "api/#GeometricOptimizers.global_rep-Union%7BTuple%7BAT%7D%2C%20Tuple%7BT%
 name = "GeometricOptimizers.global_rep-Union{Tuple{AT}, Tuple{T}, Tuple{GlobalSection{T, AT, λT} where λT<:Union{Nothing, AbstractArray{T}}, AbstractMatrix{T}}} where {T, AT<:(StiefelManifold{T, AT} where AT<:AbstractMatrix{T})}"
 uri = "api/#GeometricOptimizers.global_rep-Union%7BTuple%7BAT%7D%2C%20Tuple%7BT%7D%2C%20Tuple%7BGlobalSection%7BT%2C%20AT%2C%20%CE%BBT%7D%20where%20%CE%BBT%3C%3AUnion%7BNothing%2C%20AbstractArray%7BT%7D%7D%2C%20AbstractMatrix%7BT%7D%7D%7D%20where%20%7BT%2C%20AT%3C%3A%28StiefelManifold%7BT%2C%20AT%7D%20where%20AT%3C%3AAbstractMatrix%7BT%7D%29%7D"
 [[jl.method]]
-name = "GeometricOptimizers.global_section-Union{Tuple{GrassmannManifold{T, AT} where AT<:AbstractMatrix{T}}, Tuple{T}} where T"
-uri = "api/#GeometricOptimizers.global_section-Union%7BTuple%7BGrassmannManifold%7BT%2C%20AT%7D%20where%20AT%3C%3AAbstractMatrix%7BT%7D%7D%2C%20Tuple%7BT%7D%7D%20where%20T"
-[[jl.method]]
-name = "GeometricOptimizers.global_section-Union{Tuple{StiefelManifold{T, AT} where AT<:AbstractMatrix{T}}, Tuple{T}} where T"
-uri = "api/#GeometricOptimizers.global_section-Union%7BTuple%7BStiefelManifold%7BT%2C%20AT%7D%20where%20AT%3C%3AAbstractMatrix%7BT%7D%7D%2C%20Tuple%7BT%7D%7D%20where%20T"
+name = "GeometricOptimizers.global_section-Tuple{GrassmannManifold}"
+uri = "api/#GeometricOptimizers.global_section-Tuple%7BGrassmannManifold%7D"
 [[jl.method]]
-name = "GeometricOptimizers.gradient-Tuple{GeometricOptimizers.BFGSCache}"
-uri = "api/#GeometricOptimizers.gradient-Tuple%7BGeometricOptimizers.BFGSCache%7D"
+name = "GeometricOptimizers.global_section-Tuple{StiefelManifold}"
+uri = "api/#GeometricOptimizers.global_section-Tuple%7BStiefelManifold%7D"
 [[jl.method]]
-name = "GeometricOptimizers.gradient-Tuple{GeometricOptimizers.DFPCache}"
-uri = "api/#GeometricOptimizers.gradient-Tuple%7BGeometricOptimizers.DFPCache%7D"
+name = "GeometricOptimizers.global_section-Tuple{SymplecticStiefelManifold}"
+uri = "api/#GeometricOptimizers.global_section-Tuple%7BSymplecticStiefelManifold%7D"
 [[jl.method]]
-name = "GeometricOptimizers.gradient-Tuple{GeometricOptimizers.NewtonOptimizerCache}"
-uri = "api/#GeometricOptimizers.gradient-Tuple%7BGeometricOptimizers.NewtonOptimizerCache%7D"
+name = "GeometricOptimizers.gradient-Tuple{GeometricOptimizers.OptimizerCache}"
+uri = "api/#GeometricOptimizers.gradient-Tuple%7BGeometricOptimizers.OptimizerCache%7D"
 [[jl.method]]
 name = "GeometricOptimizers.gradient_difference!-Tuple{GeometricOptimizers.OptimizerCache, OptimizerState}"
 uri = "api/#GeometricOptimizers.gradient_difference%21-Tuple%7BGeometricOptimizers.OptimizerCache%2C%20OptimizerState%7D"
@@ -185,9 +224,6 @@ uri = "api/#GeometricOptimizers.gradient_difference%21-Tuple%7BGeometricOptimize
 name = "GeometricOptimizers.invalidate_latest_gradient!-Tuple{GeometricOptimizers.OptimizerCache}"
 uri = "api/#GeometricOptimizers.invalidate_latest_gradient%21-Tuple%7BGeometricOptimizers.OptimizerCache%7D"
 [[jl.method]]
-name = "GeometricOptimizers.isaOptimizerState-Tuple{Any}"
-uri = "api/#GeometricOptimizers.isaOptimizerState-Tuple%7BAny%7D"
-[[jl.method]]
 name = "GeometricOptimizers.isconverged-Tuple{GeometricOptimizers.OptimizerStatus}"
 uri = "api/#GeometricOptimizers.isconverged-Tuple%7BGeometricOptimizers.OptimizerStatus%7D"
 [[jl.method]]
@@ -197,6 +233,9 @@ uri = "api/#GeometricOptimizers.latest_gradient-Tuple%7BGeometricOptimizers.Opti
 name = "GeometricOptimizers.latest_gradient_is_current-Tuple{GeometricOptimizers.OptimizerCache, OptimizerState, OptimizerSolution}"
 uri = "api/#GeometricOptimizers.latest_gradient_is_current-Tuple%7BGeometricOptimizers.OptimizerCache%2C%20OptimizerState%2C%20OptimizerSolution%7D"
 [[jl.method]]
+name = "GeometricOptimizers.lift_factors!-Union{Tuple{T}, Tuple{GeometricOptimizers.RetractionWorkspace{T, AT} where AT<:AbstractMatrix{T}, StiefelLieAlgHorMatrix{T, AT, ST} where {AT<:(SkewSymMatrix{T, AT} where AT<:AbstractVector{T}), ST<:AbstractMatrix{T}}}} where T"
+uri = "api/#GeometricOptimizers.lift_factors%21-Union%7BTuple%7BT%7D%2C%20Tuple%7BGeometricOptimizers.RetractionWorkspace%7BT%2C%20AT%7D%20where%20AT%3C%3AAbstractMatrix%7BT%7D%2C%20StiefelLieAlgHorMatrix%7BT%2C%20AT%2C%20ST%7D%20where%20%7BAT%3C%3A%28SkewSymMatrix%7BT%2C%20AT%7D%20where%20AT%3C%3AAbstractVector%7BT%7D%29%2C%20ST%3C%3AAbstractMatrix%7BT%7D%7D%7D%7D%20where%20T"
+[[jl.method]]
 name = "GeometricOptimizers.lift_factors-Tuple{StiefelLieAlgHorMatrix}"
 uri = "api/#GeometricOptimizers.lift_factors-Tuple%7BStiefelLieAlgHorMatrix%7D"
 [[jl.method]]
@@ -218,12 +257,24 @@ uri = "api/#GeometricOptimizers.meets_stopping_criteria-Tuple%7BGeometricOptimiz
 name = "GeometricOptimizers.metric-Tuple{GrassmannManifold, AbstractMatrix, AbstractMatrix}"
 uri = "api/#GeometricOptimizers.metric-Tuple%7BGrassmannManifold%2C%20AbstractMatrix%2C%20AbstractMatrix%7D"
 [[jl.method]]
-name = "GeometricOptimizers.metric-Tuple{StiefelManifold, AbstractMatrix, AbstractMatrix}"
-uri = "api/#GeometricOptimizers.metric-Tuple%7BStiefelManifold%2C%20AbstractMatrix%2C%20AbstractMatrix%7D"
+name = "GeometricOptimizers.metric-Union{Tuple{T}, Tuple{StiefelManifold{T, AT} where AT<:AbstractMatrix{T}, AbstractMatrix, AbstractMatrix}} where T"
+uri = "api/#GeometricOptimizers.metric-Union%7BTuple%7BT%7D%2C%20Tuple%7BStiefelManifold%7BT%2C%20AT%7D%20where%20AT%3C%3AAbstractMatrix%7BT%7D%2C%20AbstractMatrix%2C%20AbstractMatrix%7D%7D%20where%20T"
+[[jl.method]]
+name = "GeometricOptimizers.metric-Union{Tuple{T}, Tuple{SymplecticStiefelManifold{T, AT} where AT<:AbstractMatrix{T}, AbstractMatrix, AbstractMatrix}} where T"
+uri = "api/#GeometricOptimizers.metric-Union%7BTuple%7BT%7D%2C%20Tuple%7BSymplecticStiefelManifold%7BT%2C%20AT%7D%20where%20AT%3C%3AAbstractMatrix%7BT%7D%2C%20AbstractMatrix%2C%20AbstractMatrix%7D%7D%20where%20T"
+[[jl.method]]
+name = "GeometricOptimizers.observe_optimizer_phase-Tuple{Any, NoStepObserver, Any}"
+uri = "api/#GeometricOptimizers.observe_optimizer_phase-Tuple%7BAny%2C%20NoStepObserver%2C%20Any%7D"
 [[jl.method]]
 name = "GeometricOptimizers.opnorm₁-Tuple{AbstractMatrix}"
 uri = "api/#GeometricOptimizers.opnorm%E2%82%81-Tuple%7BAbstractMatrix%7D"
 [[jl.method]]
+name = "GeometricOptimizers.optimization_step!-Union{Tuple{T}, Tuple{OptimizerSolution{T}, TrainingOptimizer{<:Union{GradientMethod, Adam, AdamWithEuclideanDecay, MomentumMethod, ScalarMomentAdam}, <:GeometricOptimizers.OptimizerCache, <:OptimizerState{T}}, Union{AbstractArray{T}, NetworkParameters{T}}}} where T"
+uri = "api/#GeometricOptimizers.optimization_step%21-Union%7BTuple%7BT%7D%2C%20Tuple%7BOptimizerSolution%7BT%7D%2C%20TrainingOptimizer%7B%3C%3AUnion%7BGradientMethod%2C%20Adam%2C%20AdamWithEuclideanDecay%2C%20MomentumMethod%2C%20ScalarMomentAdam%7D%2C%20%3C%3AGeometricOptimizers.OptimizerCache%2C%20%3C%3AOptimizerState%7BT%7D%7D%2C%20Union%7BAbstractArray%7BT%7D%2C%20NetworkParameters%7BT%7D%7D%7D%7D%20where%20T"
+[[jl.method]]
+name = "GeometricOptimizers.orthonormal_columns-Tuple{Any}"
+uri = "api/#GeometricOptimizers.orthonormal_columns-Tuple%7BAny%7D"
+[[jl.method]]
 name = "GeometricOptimizers.refresh_latest_gradient!-Tuple{GeometricOptimizers.OptimizerCache, Gradient}"
 uri = "api/#GeometricOptimizers.refresh_latest_gradient%21-Tuple%7BGeometricOptimizers.OptimizerCache%2C%20Gradient%7D"
 [[jl.method]]
@@ -236,33 +287,48 @@ uri = "api/#GeometricOptimizers.restart%21-Tuple%7BOptimizerState%7D"
 name = "GeometricOptimizers.retraction-Tuple{AbstractRetraction, AbstractArray}"
 uri = "api/#GeometricOptimizers.retraction-Tuple%7BAbstractRetraction%2C%20AbstractArray%7D"
 [[jl.method]]
+name = "GeometricOptimizers.retraction_differential!-Tuple{Any, Any, AbstractRetraction, Any, Any}"
+uri = "api/#GeometricOptimizers.retraction_differential%21-Tuple%7BAny%2C%20Any%2C%20AbstractRetraction%2C%20Any%2C%20Any%7D"
+[[jl.method]]
 name = "GeometricOptimizers.retraction_differential-Tuple{AbstractRetraction, Any, Any}"
 uri = "api/#GeometricOptimizers.retraction_differential-Tuple%7BAbstractRetraction%2C%20Any%2C%20Any%7D"
 [[jl.method]]
+name = "GeometricOptimizers.retraction_matrix!-Union{Tuple{T}, Tuple{GeometricOptimizers.RetractionWorkspace{T, AT} where AT<:AbstractMatrix{T}, AbstractRetraction, AbstractLieAlgHorMatrix{T}}} where T"
+uri = "api/#GeometricOptimizers.retraction_matrix%21-Union%7BTuple%7BT%7D%2C%20Tuple%7BGeometricOptimizers.RetractionWorkspace%7BT%2C%20AT%7D%20where%20AT%3C%3AAbstractMatrix%7BT%7D%2C%20AbstractRetraction%2C%20AbstractLieAlgHorMatrix%7BT%7D%7D%7D%20where%20T"
+[[jl.method]]
 name = "GeometricOptimizers.rgrad-Tuple{GrassmannManifold, AbstractMatrix}"
 uri = "api/#GeometricOptimizers.rgrad-Tuple%7BGrassmannManifold%2C%20AbstractMatrix%7D"
 [[jl.method]]
 name = "GeometricOptimizers.rgrad-Tuple{StiefelManifold, AbstractMatrix}"
 uri = "api/#GeometricOptimizers.rgrad-Tuple%7BStiefelManifold%2C%20AbstractMatrix%7D"
 [[jl.method]]
-name = "GeometricOptimizers.rhs-Tuple{GeometricOptimizers.BFGSCache}"
-uri = "api/#GeometricOptimizers.rhs-Tuple%7BGeometricOptimizers.BFGSCache%7D"
-[[jl.method]]
-name = "GeometricOptimizers.rhs-Tuple{GeometricOptimizers.DFPCache}"
-uri = "api/#GeometricOptimizers.rhs-Tuple%7BGeometricOptimizers.DFPCache%7D"
+name = "GeometricOptimizers.rgrad-Tuple{SymplecticStiefelManifold, AbstractMatrix}"
+uri = "api/#GeometricOptimizers.rgrad-Tuple%7BSymplecticStiefelManifold%2C%20AbstractMatrix%7D"
 [[jl.method]]
 name = "GeometricOptimizers.rhs-Tuple{GeometricOptimizers.NewtonOptimizerCache}"
 uri = "api/#GeometricOptimizers.rhs-Tuple%7BGeometricOptimizers.NewtonOptimizerCache%7D"
 [[jl.method]]
+name = "GeometricOptimizers.rhs-Tuple{GeometricOptimizers.QuasiNewtonCache}"
+uri = "api/#GeometricOptimizers.rhs-Tuple%7BGeometricOptimizers.QuasiNewtonCache%7D"
+[[jl.method]]
 name = "GeometricOptimizers.solution_scale-Tuple{AbstractVecOrMat}"
 uri = "api/#GeometricOptimizers.solution_scale-Tuple%7BAbstractVecOrMat%7D"
 [[jl.method]]
-name = "GeometricOptimizers.solver_step!-Union{Tuple{MT}, Tuple{T}, Tuple{OptimizerSolution{T}, OptimizerState{T}, Optimizer{T, MT, OBJ, GT, HT} where {OBJ<:(OptimizerProblem{T}), GT<:Gradient{T}, HT<:Hessian{T}}}} where {T, MT}"
-uri = "api/#GeometricOptimizers.solver_step%21-Union%7BTuple%7BMT%7D%2C%20Tuple%7BT%7D%2C%20Tuple%7BOptimizerSolution%7BT%7D%2C%20OptimizerState%7BT%7D%2C%20Optimizer%7BT%2C%20MT%2C%20OBJ%2C%20GT%2C%20HT%7D%20where%20%7BOBJ%3C%3A%28OptimizerProblem%7BT%7D%29%2C%20GT%3C%3AGradient%7BT%7D%2C%20HT%3C%3AHessian%7BT%7D%7D%7D%7D%20where%20%7BT%2C%20MT%7D"
+name = "GeometricOptimizers.solver_step!-Union{Tuple{T}, Tuple{OptimizerSolution{T}, OptimizerState{T}, Optimizer{T, ALG, OBJ, GT, HT} where {ALG<:OptimizerMethod, OBJ<:(OptimizerProblem{T}), GT<:Gradient{T}, HT<:Hessian{T}}}} where T"
+uri = "api/#GeometricOptimizers.solver_step%21-Union%7BTuple%7BT%7D%2C%20Tuple%7BOptimizerSolution%7BT%7D%2C%20OptimizerState%7BT%7D%2C%20Optimizer%7BT%2C%20ALG%2C%20OBJ%2C%20GT%2C%20HT%7D%20where%20%7BALG%3C%3AOptimizerMethod%2C%20OBJ%3C%3A%28OptimizerProblem%7BT%7D%29%2C%20GT%3C%3AGradient%7BT%7D%2C%20HT%3C%3AHessian%7BT%7D%7D%7D%7D%20where%20T"
+[[jl.method]]
+name = "GeometricOptimizers.sr!-Union{Tuple{AbstractMatrix{T}}, Tuple{T}} where T"
+uri = "api/#GeometricOptimizers.sr%21-Union%7BTuple%7BAbstractMatrix%7BT%7D%7D%2C%20Tuple%7BT%7D%7D%20where%20T"
+[[jl.method]]
+name = "GeometricOptimizers.sr-Tuple{AbstractMatrix}"
+uri = "api/#GeometricOptimizers.sr-Tuple%7BAbstractMatrix%7D"
 [[jl.method]]
 name = "GeometricOptimizers.steepest_descent!-Tuple{GeometricOptimizers.OptimizerCache}"
 uri = "api/#GeometricOptimizers.steepest_descent%21-Tuple%7BGeometricOptimizers.OptimizerCache%7D"
 [[jl.method]]
+name = "GeometricOptimizers.step_observer-Tuple{Optimizer}"
+uri = "api/#GeometricOptimizers.step_observer-Tuple%7BOptimizer%7D"
+[[jl.method]]
 name = "GeometricOptimizers.step_size-Union{Tuple{T}, Tuple{DecayingStatic{T}, Integer}} where T"
 uri = "api/#GeometricOptimizers.step_size-Union%7BTuple%7BT%7D%2C%20Tuple%7BDecayingStatic%7BT%7D%2C%20Integer%7D%7D%20where%20T"
 [[jl.method]]
@@ -272,20 +338,29 @@ uri = "api/#GeometricOptimizers.step_%CE%B1max-Union%7BTuple%7BT%7D%2C%20Tuple%7
 name = "GeometricOptimizers.store_gradient!-Tuple{GeometricOptimizers.OptimizerCache, OptimizerState, Gradient, OptimizerSolution}"
 uri = "api/#GeometricOptimizers.store_gradient%21-Tuple%7BGeometricOptimizers.OptimizerCache%2C%20OptimizerState%2C%20Gradient%2C%20OptimizerSolution%7D"
 [[jl.method]]
+name = "GeometricOptimizers.symplectic_form-Tuple{AbstractVector, AbstractVector}"
+uri = "api/#GeometricOptimizers.symplectic_form-Tuple%7BAbstractVector%2C%20AbstractVector%7D"
+[[jl.method]]
+name = "GeometricOptimizers.symplectic_householder!-Union{Tuple{T}, Tuple{AbstractVector{T}, AbstractVector{T}}} where T"
+uri = "api/#GeometricOptimizers.symplectic_householder%21-Union%7BTuple%7BT%7D%2C%20Tuple%7BAbstractVector%7BT%7D%2C%20AbstractVector%7BT%7D%7D%7D%20where%20T"
+[[jl.method]]
+name = "GeometricOptimizers.symplectic_householder-Tuple{AbstractVector, AbstractVector}"
+uri = "api/#GeometricOptimizers.symplectic_householder-Tuple%7BAbstractVector%2C%20AbstractVector%7D"
+[[jl.method]]
+name = "GeometricOptimizers.symplectic_normalize-Tuple{AbstractVector, AbstractVector, AbstractMatrix}"
+uri = "api/#GeometricOptimizers.symplectic_normalize-Tuple%7BAbstractVector%2C%20AbstractVector%2C%20AbstractMatrix%7D"
+[[jl.method]]
 name = "GeometricOptimizers.trace-Tuple{GeometricOptimizers.OptimizerResult}"
 uri = "api/#GeometricOptimizers.trace-Tuple%7BGeometricOptimizers.OptimizerResult%7D"
 [[jl.method]]
-name = "GeometricOptimizers.trial_iterate!-Tuple{GeometricOptimizers.OptimizerCache, Any, Any, Any}"
-uri = "api/#GeometricOptimizers.trial_iterate%21-Tuple%7BGeometricOptimizers.OptimizerCache%2C%20Any%2C%20Any%2C%20Any%7D"
-[[jl.method]]
-name = "GeometricOptimizers.trial_slope-Tuple{Gradient, GeometricOptimizers.OptimizerCache, Any, Any}"
-uri = "api/#GeometricOptimizers.trial_slope-Tuple%7BGradient%2C%20GeometricOptimizers.OptimizerCache%2C%20Any%2C%20Any%7D"
+name = "GeometricOptimizers.trial_slope-Tuple{Gradient, GeometricOptimizers.OptimizerCache, Vararg{Any, 4}}"
+uri = "api/#GeometricOptimizers.trial_slope-Tuple%7BGradient%2C%20GeometricOptimizers.OptimizerCache%2C%20Vararg%7BAny%2C%204%7D%7D"
 [[jl.method]]
-name = "GeometricOptimizers.unit_matrix-Union{Tuple{T}, Tuple{Any, Type{T}, Integer}} where T"
-uri = "api/#GeometricOptimizers.unit_matrix-Union%7BTuple%7BT%7D%2C%20Tuple%7BAny%2C%20Type%7BT%7D%2C%20Integer%7D%7D%20where%20T"
+name = "GeometricOptimizers.unit_matrix-Union{Tuple{T}, Tuple{KernelAbstractions.Backend, Type{T}, Integer}} where T"
+uri = "api/#GeometricOptimizers.unit_matrix-Union%7BTuple%7BT%7D%2C%20Tuple%7BKernelAbstractions.Backend%2C%20Type%7BT%7D%2C%20Integer%7D%7D%20where%20T"
 [[jl.method]]
-name = "GeometricOptimizers.update_section!-Union{Tuple{MT}, Tuple{T}, Tuple{GlobalSection{T, MT, λT} where λT<:Union{Nothing, AbstractArray{T}}, AbstractLieAlgHorMatrix{T}, Any}} where {T, MT<:Manifold{T}}"
-uri = "api/#GeometricOptimizers.update_section%21-Union%7BTuple%7BMT%7D%2C%20Tuple%7BT%7D%2C%20Tuple%7BGlobalSection%7BT%2C%20MT%2C%20%CE%BBT%7D%20where%20%CE%BBT%3C%3AUnion%7BNothing%2C%20AbstractArray%7BT%7D%7D%2C%20AbstractLieAlgHorMatrix%7BT%7D%2C%20Any%7D%7D%20where%20%7BT%2C%20MT%3C%3AManifold%7BT%7D%7D"
+name = "GeometricOptimizers.update_section!-Union{Tuple{MT}, Tuple{T}, Tuple{GlobalSection{T, MT, λT} where λT<:Union{Nothing, AbstractArray{T}}, GlobalSection{T, MT, λT} where λT<:Union{Nothing, AbstractArray{T}}, AbstractLieAlgHorMatrix{T}, Any}, Tuple{GlobalSection{T, MT, λT} where λT<:Union{Nothing, AbstractArray{T}}, GlobalSection{T, MT, λT} where λT<:Union{Nothing, AbstractArray{T}}, AbstractLieAlgHorMatrix{T}, Any, Any}} where {T, MT<:Manifold{T}}"
+uri = "api/#GeometricOptimizers.update_section%21-Union%7BTuple%7BMT%7D%2C%20Tuple%7BT%7D%2C%20Tuple%7BGlobalSection%7BT%2C%20MT%2C%20%CE%BBT%7D%20where%20%CE%BBT%3C%3AUnion%7BNothing%2C%20AbstractArray%7BT%7D%7D%2C%20GlobalSection%7BT%2C%20MT%2C%20%CE%BBT%7D%20where%20%CE%BBT%3C%3AUnion%7BNothing%2C%20AbstractArray%7BT%7D%7D%2C%20AbstractLieAlgHorMatrix%7BT%7D%2C%20Any%7D%2C%20Tuple%7BGlobalSection%7BT%2C%20MT%2C%20%CE%BBT%7D%20where%20%CE%BBT%3C%3AUnion%7BNothing%2C%20AbstractArray%7BT%7D%7D%2C%20GlobalSection%7BT%2C%20MT%2C%20%CE%BBT%7D%20where%20%CE%BBT%3C%3AUnion%7BNothing%2C%20AbstractArray%7BT%7D%7D%2C%20AbstractLieAlgHorMatrix%7BT%7D%2C%20Any%2C%20Any%7D%7D%20where%20%7BT%2C%20MT%3C%3AManifold%7BT%7D%7D"
 [[jl.method]]
 name = "GeometricOptimizers.value-Tuple{GeometricOptimizers.AbstractOptimizerProblem, OptimizerSolution}"
 uri = "api/#GeometricOptimizers.value-Tuple%7BGeometricOptimizers.AbstractOptimizerProblem%2C%20OptimizerSolution%7D"
@@ -296,6 +371,9 @@ uri = "api/#GeometricOptimizers.%CE%A9-Union%7BTuple%7BT%7D%2C%20Tuple%7BGrassma
 name = "GeometricOptimizers.Ω-Union{Tuple{T}, Tuple{StiefelManifold{T, AT} where AT<:AbstractMatrix{T}, AbstractMatrix{T}}} where T"
 uri = "api/#GeometricOptimizers.%CE%A9-Union%7BTuple%7BT%7D%2C%20Tuple%7BStiefelManifold%7BT%2C%20AT%7D%20where%20AT%3C%3AAbstractMatrix%7BT%7D%2C%20AbstractMatrix%7BT%7D%7D%7D%20where%20T"
 [[jl.method]]
+name = "GeometricOptimizers.𝔄!-Tuple{Any, AbstractMatrix, GeometricOptimizers.AbstractExponentialAlgorithm}"
+uri = "api/#GeometricOptimizers.%F0%9D%94%84%21-Tuple%7BAny%2C%20AbstractMatrix%2C%20GeometricOptimizers.AbstractExponentialAlgorithm%7D"
+[[jl.method]]
 name = "GeometricOptimizers.𝔄-Tuple{AbstractMatrix, AbstractMatrix}"
 uri = "api/#GeometricOptimizers.%F0%9D%94%84-Tuple%7BAbstractMatrix%2C%20AbstractMatrix%7D"
 [[jl.method]]
@@ -305,24 +383,22 @@ uri = "api/#GeometricOptimizers.%F0%9D%94%84-Tuple%7BAbstractMatrix%2C%20Geometr
 name = "GeometricOptimizers.𝔄-Tuple{AbstractMatrix}"
 uri = "api/#GeometricOptimizers.%F0%9D%94%84-Tuple%7BAbstractMatrix%7D"
 [[jl.method]]
-name = "SimpleSolvers.direction-Tuple{GeometricOptimizers.BFGSCache}"
-uri = "api/#SimpleSolvers.direction-Tuple%7BGeometricOptimizers.BFGSCache%7D"
-[[jl.method]]
-name = "SimpleSolvers.direction-Tuple{GeometricOptimizers.DFPCache}"
-uri = "api/#SimpleSolvers.direction-Tuple%7BGeometricOptimizers.DFPCache%7D"
-[[jl.method]]
 name = "SimpleSolvers.direction-Tuple{GeometricOptimizers.NewtonOptimizerCache}"
 uri = "api/#SimpleSolvers.direction-Tuple%7BGeometricOptimizers.NewtonOptimizerCache%7D"
 [[jl.method]]
-name = "SimpleSolvers.linesearch_problem-Union{Tuple{T}, Tuple{OptimizerProblem{T}, Gradient, GeometricOptimizers.OptimizerCache{T}, AbstractRetraction}} where T"
-uri = "api/#SimpleSolvers.linesearch_problem-Union%7BTuple%7BT%7D%2C%20Tuple%7BOptimizerProblem%7BT%7D%2C%20Gradient%2C%20GeometricOptimizers.OptimizerCache%7BT%7D%2C%20AbstractRetraction%7D%7D%20where%20T"
+name = "SimpleSolvers.direction-Tuple{GeometricOptimizers.QuasiNewtonCache}"
+uri = "api/#SimpleSolvers.direction-Tuple%7BGeometricOptimizers.QuasiNewtonCache%7D"
 [[jl.method]]
-name = "SimpleSolvers.outer!-Union{Tuple{T}, Tuple{AbstractMatrix{T}, AbstractLieAlgHorMatrix{T}, AbstractLieAlgHorMatrix{T}}} where T"
-uri = "api/#SimpleSolvers.outer%21-Union%7BTuple%7BT%7D%2C%20Tuple%7BAbstractMatrix%7BT%7D%2C%20AbstractLieAlgHorMatrix%7BT%7D%2C%20AbstractLieAlgHorMatrix%7BT%7D%7D%7D%20where%20T"
+name = "SimpleSolvers.linesearch_problem-Union{Tuple{T}, Tuple{OptimizerProblem{T}, Gradient, GeometricOptimizers.OptimizerCache{T}, AbstractRetraction}, Tuple{OptimizerProblem{T}, Gradient, GeometricOptimizers.OptimizerCache{T}, AbstractRetraction, Any}, Tuple{OptimizerProblem{T}, Gradient, GeometricOptimizers.OptimizerCache{T}, AbstractRetraction, Any, Any}} where T"
+uri = "api/#SimpleSolvers.linesearch_problem-Union%7BTuple%7BT%7D%2C%20Tuple%7BOptimizerProblem%7BT%7D%2C%20Gradient%2C%20GeometricOptimizers.OptimizerCache%7BT%7D%2C%20AbstractRetraction%7D%2C%20Tuple%7BOptimizerProblem%7BT%7D%2C%20Gradient%2C%20GeometricOptimizers.OptimizerCache%7BT%7D%2C%20AbstractRetraction%2C%20Any%7D%2C%20Tuple%7BOptimizerProblem%7BT%7D%2C%20Gradient%2C%20GeometricOptimizers.OptimizerCache%7BT%7D%2C%20AbstractRetraction%2C%20Any%2C%20Any%7D%7D%20where%20T"
 [[jl.method]]
 name = "SimpleSolvers.solve!-Union{Tuple{T}, Tuple{OptimizerSolution{T}, OptimizerState, Optimizer{T, ALG, OBJ, GT, HT} where {ALG<:OptimizerMethod, OBJ<:(OptimizerProblem{T}), GT<:Gradient{T}, HT<:Hessian{T}}}} where T"
 uri = "api/#SimpleSolvers.solve%21-Union%7BTuple%7BT%7D%2C%20Tuple%7BOptimizerSolution%7BT%7D%2C%20OptimizerState%2C%20Optimizer%7BT%2C%20ALG%2C%20OBJ%2C%20GT%2C%20HT%7D%20where%20%7BALG%3C%3AOptimizerMethod%2C%20OBJ%3C%3A%28OptimizerProblem%7BT%7D%29%2C%20GT%3C%3AGradient%7BT%7D%2C%20HT%3C%3AHessian%7BT%7D%7D%7D%7D%20where%20T"
 
+[[jl.module]]
+name = "GeometricOptimizers.GeometricOptimizers"
+uri = "api/#$"
+
 [[jl.type]]
 name = "GeometricOptimizers.AbstractExponentialAlgorithm"
 uri = "api/#$"
@@ -360,9 +436,6 @@ uri = "api/#$"
 name = "GeometricOptimizers.BFGS"
 uri = "api/#$"
 [[jl.type]]
-name = "GeometricOptimizers.BFGSCache"
-uri = "api/#$"
-[[jl.type]]
 name = "GeometricOptimizers.BFGSState"
 uri = "api/#$"
 [[jl.type]]
@@ -372,15 +445,18 @@ uri = "api/#$"
 name = "GeometricOptimizers.DFP"
 uri = "api/#$"
 [[jl.type]]
-name = "GeometricOptimizers.DFPCache"
-uri = "api/#$"
-[[jl.type]]
 name = "GeometricOptimizers.DFPState"
 uri = "api/#$"
 [[jl.type]]
 name = "GeometricOptimizers.DecayingStatic"
 uri = "api/#$"
 [[jl.type]]
+name = "GeometricOptimizers.EventLog"
+uri = "api/#$"
+[[jl.type]]
+name = "GeometricOptimizers.FirstOrderMethod"
+uri = "api/#$"
+[[jl.type]]
 name = "GeometricOptimizers.FirstOrderMethodWithState"
 uri = "api/#$"
 [[jl.type]]
@@ -405,18 +481,6 @@ uri = "api/#$"
 name = "GeometricOptimizers.GrassmannManifold"
 uri = "api/#$"
 [[jl.type]]
-name = "GeometricOptimizers.HessianBFGS"
-uri = "api/#$"
-[[jl.type]]
-name = "GeometricOptimizers.HessianDFP"
-uri = "api/#$"
-[[jl.type]]
-name = "GeometricOptimizers.IterativeHessian"
-uri = "api/#$"
-[[jl.type]]
-name = "GeometricOptimizers.LowerTriangular"
-uri = "api/#$"
-[[jl.type]]
 name = "GeometricOptimizers.Manifold"
 uri = "api/#$"
 [[jl.type]]
@@ -438,7 +502,13 @@ uri = "api/#$"
 name = "GeometricOptimizers.NewtonOptimizerCache"
 uri = "api/#$"
 [[jl.type]]
-name = "GeometricOptimizers.NewtonOptimizerState"
+name = "GeometricOptimizers.NewtonState"
+uri = "api/#$"
+[[jl.type]]
+name = "GeometricOptimizers.NoStepObserver"
+uri = "api/#$"
+[[jl.type]]
+name = "GeometricOptimizers.NoWorkspace"
 uri = "api/#$"
 [[jl.type]]
 name = "GeometricOptimizers.Optimizer"
@@ -468,12 +538,36 @@ uri = "api/#$"
 name = "GeometricOptimizers.OptimizerTraceEntry"
 uri = "api/#$"
 [[jl.type]]
+name = "GeometricOptimizers.OrthonormalizationFailure"
+uri = "api/#$"
+[[jl.type]]
+name = "GeometricOptimizers.PhaseTimer"
+uri = "api/#$"
+[[jl.type]]
+name = "GeometricOptimizers.PrecomputedGradient"
+uri = "api/#$"
+[[jl.type]]
 name = "GeometricOptimizers.ProjectedSkew"
 uri = "api/#$"
 [[jl.type]]
+name = "GeometricOptimizers.QuasiNewtonCache"
+uri = "api/#$"
+[[jl.type]]
 name = "GeometricOptimizers.QuasiNewtonOptimizerMethod"
 uri = "api/#$"
 [[jl.type]]
+name = "GeometricOptimizers.RetractionWorkspace"
+uri = "api/#$"
+[[jl.type]]
+name = "GeometricOptimizers.Rfac"
+uri = "api/#$"
+[[jl.type]]
+name = "GeometricOptimizers.RiemannianGradient"
+uri = "api/#$"
+[[jl.type]]
+name = "GeometricOptimizers.SR"
+uri = "api/#$"
+[[jl.type]]
 name = "GeometricOptimizers.ScalarMomentAdam"
 uri = "api/#$"
 [[jl.type]]
@@ -483,9 +577,15 @@ uri = "api/#$"
 name = "GeometricOptimizers.ScalarMomentAdamState"
 uri = "api/#$"
 [[jl.type]]
+name = "GeometricOptimizers.ScaledAlgorithm"
+uri = "api/#$"
+[[jl.type]]
 name = "GeometricOptimizers.ScaledSquaring"
 uri = "api/#$"
 [[jl.type]]
+name = "GeometricOptimizers.Sfac"
+uri = "api/#$"
+[[jl.type]]
 name = "GeometricOptimizers.SkewSymMatrix"
 uri = "api/#$"
 [[jl.type]]
@@ -498,13 +598,25 @@ uri = "api/#$"
 name = "GeometricOptimizers.StiefelProjection"
 uri = "api/#$"
 [[jl.type]]
+name = "GeometricOptimizers.StrictlyLowerTriangular"
+uri = "api/#$"
+[[jl.type]]
+name = "GeometricOptimizers.StrictlyUpperTriangular"
+uri = "api/#$"
+[[jl.type]]
 name = "GeometricOptimizers.SymmetricMatrix"
 uri = "api/#$"
 [[jl.type]]
+name = "GeometricOptimizers.SymplecticHouseholderDecom"
+uri = "api/#$"
+[[jl.type]]
+name = "GeometricOptimizers.SymplecticStiefelManifold"
+uri = "api/#$"
+[[jl.type]]
 name = "GeometricOptimizers.TaylorSeries"
 uri = "api/#$"
 [[jl.type]]
-name = "GeometricOptimizers.UpperTriangular"
+name = "GeometricOptimizers.TrainingOptimizer"
 uri = "api/#$"
 [[jl.type]]
 name = "GeometricOptimizers.VectorStorageMatrix"
@@ -515,6 +627,10 @@ dispname = "API"
 name = "api"
 uri = "api/"
 [[std.doc]]
+dispname = "Exponential Algorithms"
+name = "exponential_algorithms"
+uri = "exponential_algorithms/"
+[[std.doc]]
 dispname = "Global Tangent Spaces"
 name = "global_tangent_spaces"
 uri = "global_tangent_spaces/"
@@ -563,6 +679,10 @@ dispname = "Riemannian Manifolds"
 name = "manifolds/riemannian_manifolds"
 uri = "manifolds/riemannian_manifolds/"
 [[std.doc]]
+dispname = "Observing Optimizer Phases"
+name = "observers"
+uri = "observers/"
+[[std.doc]]
 dispname = "Optimizer Methods"
 name = "optimizer_methods"
 uri = "optimizer_methods/"
@@ -583,6 +703,10 @@ dispname = "Symmetric, Skew-Symmetric and Triangular Matrices"
 name = "special_matrices"
 uri = "special_matrices/"
 [[std.doc]]
+dispname = "The Training Step"
+name = "training_step"
+uri = "training_step/"
+[[std.doc]]
 dispname = "Weight Decay on Manifolds"
 name = "weight_decay"
 uri = "weight_decay/"
@@ -600,6 +724,30 @@ dispname = "(Topological) Vector Spaces"
 name = "(Topological)-Vector-Spaces"
 uri = "manifolds/metric_and_vector_spaces/#%28Topological%29-Vector-Spaces"
 [[std.label]]
+dispname = "1. Direct Taylor series"
+name = "1.-Direct-Taylor-series"
+uri = "exponential_algorithms/#$"
+[[std.label]]
+dispname = "2. Scaling and modified squaring"
+name = "2.-Scaling-and-modified-squaring"
+uri = "exponential_algorithms/#$"
+[[std.label]]
+dispname = "3. Padé approximation"
+name = "3.-Padé-approximation"
+uri = "exponential_algorithms/#3.-Pad%C3%A9-approximation"
+[[std.label]]
+dispname = "4. AugmentedPade"
+name = "4.-AugmentedPade"
+uri = "exponential_algorithms/#$"
+[[std.label]]
+dispname = "5. ProjectedSkew"
+name = "5.-ProjectedSkew"
+uri = "exponential_algorithms/#$"
+[[std.label]]
+dispname = "6. Numerical comparison"
+name = "6.-Numerical-comparison"
+uri = "exponential_algorithms/#$"
+[[std.label]]
 dispname = "A bounded merit is not a bound on the step"
 name = "A-bounded-merit-is-not-a-bound-on-the-step"
 uri = "linesearch_on_manifolds/#$"
@@ -618,14 +766,19 @@ uri = "api/#$"
 [[std.label]]
 dispname = "Adding one"
 name = "Adding-one"
-uri = "retractions/#$"
+uri = "exponential_algorithms/#$"
 [[std.label]]
 dispname = "Agreeing with the exponential"
 name = "Agreeing-with-the-exponential"
-uri = "retractions/#$"
+uri = "exponential_algorithms/#$"
 [[std.label]]
-name = "AugmentedPade"
-uri = "retractions/#$"
+dispname = "Applying the denominator without a matrix solve"
+name = "Applying-the-denominator-without-a-matrix-solve"
+uri = "exponential_algorithms/#$"
+[[std.label]]
+dispname = "Arithmetic and broadcasting on a device"
+name = "Arithmetic-and-broadcasting-on-a-device"
+uri = "special_matrices/#$"
 [[std.label]]
 dispname = "Basic Concepts from General Topology"
 name = "Basic-Concepts-from-General-Topology"
@@ -645,7 +798,7 @@ uri = "retractions/#$"
 [[std.label]]
 dispname = "Choosing one"
 name = "Choosing-one"
-uri = "retractions/#$"
+uri = "exponential_algorithms/#$"
 [[std.label]]
 dispname = "Classical Retractions"
 name = "Classical-Retractions"
@@ -655,6 +808,9 @@ dispname = "Complete Metric Spaces"
 name = "Complete-Metric-Spaces"
 uri = "manifolds/metric_and_vector_spaces/#$"
 [[std.label]]
+name = "Coverage"
+uri = "observers/#$"
+[[std.label]]
 dispname = "Custom Matrices"
 name = "Custom-Matrices"
 uri = "special_matrices/#$"
@@ -663,16 +819,40 @@ dispname = "Decoupled weight decay"
 name = "Decoupled-weight-decay"
 uri = "weight_decay/#$"
 [[std.label]]
+dispname = "Deriving the coefficients for mathfrakA"
+name = "Deriving-the-coefficients-for-\\mathfrak{A}"
+uri = "exponential_algorithms/#Deriving-the-coefficients-for-%5Cmathfrak%7BA%7D"
+[[std.label]]
+dispname = "Element types"
+name = "Element-types"
+uri = "special_matrices/#$"
+[[std.label]]
 name = "Example"
 uri = "linesearch/#$"
 [[std.label]]
+dispname = "Example: exclusive time per phase"
+name = "Example:-exclusive-time-per-phase"
+uri = "observers/#Example%3A-exclusive-time-per-phase"
+[[std.label]]
+dispname = "Example: the events of one iteration"
+name = "Example:-the-events-of-one-iteration"
+uri = "observers/#Example%3A-the-events-of-one-iteration"
+[[std.label]]
+dispname = "Exponential Algorithms"
+name = "Exponential-Algorithms"
+uri = "exponential_algorithms/#$"
+[[std.label]]
 name = "Float32"
-uri = "retractions/#$"
+uri = "exponential_algorithms/#$"
 [[std.label]]
 dispname = "Foundational Theorems for Differential Manifolds"
 name = "Foundational-Theorems-for-Differential-Manifolds"
 uri = "manifolds/inverse_function_theorem/#$"
 [[std.label]]
+dispname = "From scalar Padé to matrix Padé"
+name = "From-scalar-Padé-to-matrix-Padé"
+uri = "exponential_algorithms/#From-scalar-Pad%C3%A9-to-matrix-Pad%C3%A9"
+[[std.label]]
 dispname = "Generalization to Homogeneous Spaces"
 name = "Generalization-to-Homogeneous-Spaces"
 uri = "manifold_optimizers/#$"
@@ -724,8 +904,9 @@ dispname = "Linesearches on Manifolds"
 name = "Linesearches-on-Manifolds"
 uri = "linesearch_on_manifolds/#$"
 [[std.label]]
-name = "NativePade"
-uri = "retractions/#$"
+dispname = "Observing the Phases of an Optimizer Step"
+name = "Observing-the-Phases-of-an-Optimizer-Step"
+uri = "observers/#$"
 [[std.label]]
 dispname = "Optimization on Homogeneous Spaces"
 name = "Optimization-on-Homogeneous-Spaces"
@@ -743,9 +924,6 @@ dispname = "Preserve symmetry in the DFP inverse Hessian"
 name = "Preserve-symmetry-in-the-DFP-inverse-Hessian"
 uri = "linesearch_on_manifolds/#$"
 [[std.label]]
-name = "ProjectedSkew"
-uri = "retractions/#$"
-[[std.label]]
 dispname = "Quasi-Newton caches on manifold solutions"
 name = "Quasi-Newton-caches-on-manifold-solutions"
 uri = "linesearch_on_manifolds/#$"
@@ -772,8 +950,9 @@ dispname = "Sample Random Matrices"
 name = "Sample-Random-Matrices"
 uri = "special_matrices/#$"
 [[std.label]]
-name = "ScaledSquaring"
-uri = "retractions/#$"
+dispname = "Sensitivity to the threshold θ"
+name = "Sensitivity-to-the-threshold-θ"
+uri = "exponential_algorithms/#Sensitivity-to-the-threshold-%CE%B8"
 [[std.label]]
 dispname = "Standard Neural Network Optimizers"
 name = "Standard-Neural-Network-Optimizers"
@@ -781,7 +960,7 @@ uri = "optimizer_methods/#$"
 [[std.label]]
 dispname = "Staying on the manifold"
 name = "Staying-on-the-manifold"
-uri = "retractions/#$"
+uri = "exponential_algorithms/#$"
 [[std.label]]
 dispname = "Symmetric, Skew-Symmetric and Triangular Matrices"
 name = "Symmetric,-Skew-Symmetric-and-Triangular-Matrices"
@@ -791,9 +970,6 @@ dispname = "Tangent Spaces"
 name = "Tangent-Spaces"
 uri = "manifolds/manifolds/#$"
 [[std.label]]
-name = "TaylorSeries"
-uri = "retractions/#$"
-[[std.label]]
 dispname = "The Adam Optimizer"
 name = "The-Adam-Optimizer"
 uri = "optimizer_methods/#$"
@@ -878,21 +1054,45 @@ dispname = "The Tangent Bundle"
 name = "The-Tangent-Bundle"
 uri = "manifolds/manifolds/#$"
 [[std.label]]
+dispname = "The Training Step"
+name = "The-Training-Step"
+uri = "training_step/#$"
+[[std.label]]
 dispname = "The algorithm"
 name = "The-algorithm"
 uri = "manifold_optimizers/#$"
 [[std.label]]
+dispname = "The bound, and why it takes two packages"
+name = "The-bound,-and-why-it-takes-two-packages"
+uri = "linesearch_on_manifolds/#The-bound%2C-and-why-it-takes-two-packages"
+[[std.label]]
+dispname = "The complete NativePade algorithm"
+name = "The-complete-NativePade-algorithm"
+uri = "exponential_algorithms/#$"
+[[std.label]]
+dispname = "The complete ScaledSquaring algorithm"
+name = "The-complete-ScaledSquaring-algorithm"
+uri = "exponential_algorithms/#$"
+[[std.label]]
+dispname = "The doubling identity for mathfrakA"
+name = "The-doubling-identity-for-\\mathfrak{A}"
+uri = "exponential_algorithms/#The-doubling-identity-for-%5Cmathfrak%7BA%7D"
+[[std.label]]
 dispname = "The exponential needs an algorithm"
 name = "The-exponential-needs-an-algorithm"
-uri = "retractions/#$"
+uri = "exponential_algorithms/#$"
 [[std.label]]
 dispname = "The extended retraction"
 name = "The-extended-retraction"
 uri = "manifold_optimizers/#$"
 [[std.label]]
-dispname = "The fix, and why it takes two packages"
-name = "The-fix,-and-why-it-takes-two-packages"
-uri = "linesearch_on_manifolds/#The-fix%2C-and-why-it-takes-two-packages"
+dispname = "The fix: observed boundaries and ready-made recorders"
+name = "The-fix:-observed-boundaries-and-ready-made-recorders"
+uri = "observers/#The-fix%3A-observed-boundaries-and-ready-made-recorders"
+[[std.label]]
+dispname = "The framework, and the two algorithms in it"
+name = "The-framework,-and-the-two-algorithms-in-it"
+uri = "exponential_algorithms/#The-framework%2C-and-the-two-algorithms-in-it"
 [[std.label]]
 dispname = "The generator of the trial curve turns with the step"
 name = "The-generator-of-the-trial-curve-turns-with-the-step"
@@ -906,6 +1106,22 @@ dispname = "The lift to the global tangent space"
 name = "The-lift-to-the-global-tangent-space"
 uri = "manifold_optimizers/#$"
 [[std.label]]
+dispname = "The methods and the step size"
+name = "The-methods-and-the-step-size"
+uri = "training_step/#$"
+[[std.label]]
+dispname = "The name"
+name = "The-name"
+uri = "weight_decay/#$"
+[[std.label]]
+dispname = "The norm of X is quadratic in the norm of barB"
+name = "The-norm-of-X-is-*quadratic*-in-the-norm-of-\\bar{B}"
+uri = "exponential_algorithms/#The-norm-of-X-is-%2Aquadratic%2A-in-the-norm-of-%5Cbar%7BB%7D"
+[[std.label]]
+dispname = "The notation this page uses"
+name = "The-notation-this-page-uses"
+uri = "exponential_algorithms/#$"
+[[std.label]]
 dispname = "The numerical experiment"
 name = "The-numerical-experiment"
 uri = "manifold_optimizers/#$"
@@ -914,10 +1130,26 @@ dispname = "The optimizer framework, step by step"
 name = "The-optimizer-framework,-step-by-step"
 uri = "manifold_optimizers/#The-optimizer-framework%2C-step-by-step"
 [[std.label]]
+dispname = "The phases"
+name = "The-phases"
+uri = "observers/#$"
+[[std.label]]
 dispname = "The problem: Adam has no coordinate-free formulation"
 name = "The-problem:-Adam-has-no-coordinate-free-formulation"
 uri = "manifold_optimizers/#The-problem%3A-Adam-has-no-coordinate-free-formulation"
 [[std.label]]
+dispname = "The problem: one step, several costs"
+name = "The-problem:-one-step,-several-costs"
+uri = "observers/#The-problem%3A-one-step%2C-several-costs"
+[[std.label]]
+dispname = "The protocol"
+name = "The-protocol"
+uri = "observers/#$"
+[[std.label]]
+dispname = "The reduction"
+name = "The-reduction"
+uri = "exponential_algorithms/#$"
+[[std.label]]
 dispname = "The retractions on the two manifolds"
 name = "The-retractions-on-the-two-manifolds"
 uri = "retractions/#$"
@@ -930,9 +1162,9 @@ dispname = "The source's projection is global_rep"
 name = "The-source's-projection-*is*-global_rep"
 uri = "optimizer_methods/#The-source%27s-projection-%2Ais%2A-global_rep"
 [[std.label]]
-dispname = "The threshold θ needs no tuning"
-name = "The-threshold-θ-needs-no-tuning"
-uri = "retractions/#The-threshold-%CE%B8-needs-no-tuning"
+dispname = "The threshold theta"
+name = "The-threshold-\\theta"
+uri = "exponential_algorithms/#The-threshold-%5Ctheta"
 [[std.label]]
 dispname = "Time-Dependent Vector Fields"
 name = "Time-Dependent-Vector-Fields"
@@ -944,7 +1176,7 @@ uri = "weight_decay/#$"
 [[std.label]]
 dispname = "Using them"
 name = "Using-them"
-uri = "retractions/#$"
+uri = "exponential_algorithms/#$"
 [[std.label]]
 dispname = "Vector Fields"
 name = "Vector-Fields"
@@ -962,17 +1194,21 @@ dispname = "What a retraction is"
 name = "What-a-retraction-is"
 uri = "retractions/#$"
 [[std.label]]
+dispname = "What else observers are good for"
+name = "What-else-observers-are-good-for"
+uri = "observers/#$"
+[[std.label]]
+dispname = "What scaling buys the Padé kernel"
+name = "What-scaling-buys-the-Padé-kernel"
+uri = "exponential_algorithms/#What-scaling-buys-the-Pad%C3%A9-kernel"
+[[std.label]]
 dispname = "What the argument does and does not depend on"
 name = "What-the-argument-does-and-does-not-depend-on"
 uri = "weight_decay/#$"
 [[std.label]]
 dispname = "What they cost"
 name = "What-they-cost"
-uri = "retractions/#$"
-[[std.label]]
-dispname = "What they cost and how accurate they are"
-name = "What-they-cost-and-how-accurate-they-are"
-uri = "retractions/#$"
+uri = "exponential_algorithms/#$"
 [[std.label]]
 dispname = "Where GeometricMachineLearning's method belongs"
 name = "Where-GeometricMachineLearning's-method-belongs"
@@ -982,6 +1218,10 @@ dispname = "Where a retraction sits in the algorithm"
 name = "Where-a-retraction-sits-in-the-algorithm"
 uri = "retractions/#$"
 [[std.label]]
+dispname = "Where a sampled array lands, and what it holds"
+name = "Where-a-sampled-array-lands,-and-what-it-holds"
+uri = "special_matrices/#Where-a-sampled-array-lands%2C-and-what-it-holds"
+[[std.label]]
 dispname = "Where each symbol lives"
 name = "Where-each-symbol-lives"
 uri = "optimizer_methods/#$"
@@ -990,14 +1230,34 @@ dispname = "Which line search"
 name = "Which-line-search"
 uri = "weight_decay/#$"
 [[std.label]]
+dispname = "Why that combination is bad for a power series"
+name = "Why-that-combination-is-bad-for-a-power-series"
+uri = "exponential_algorithms/#$"
+[[std.label]]
 dispname = "Why the decay vanishes"
 name = "Why-the-decay-vanishes"
 uri = "weight_decay/#$"
 [[std.label]]
-dispname = "Why the name is not AdamW"
-name = "Why-the-name-is-not-AdamW"
-uri = "weight_decay/#$"
+dispname = "Why the optimizer does not time every step automatically"
+name = "Why-the-optimizer-does-not-time-every-step-automatically"
+uri = "observers/#$"
+[[std.label]]
+dispname = "Why the reduced argument is the hard case"
+name = "Why-the-reduced-argument-is-the-hard-case"
+uri = "exponential_algorithms/#$"
 [[std.label]]
 dispname = "Why the storage matters here"
 name = "Why-the-storage-matters-here"
 uri = "special_matrices/#$"
+[[std.label]]
+dispname = "Why the training step is not solve!"
+name = "Why-the-training-step-is-not-solve!"
+uri = "training_step/#Why-the-training-step-is-not-solve%21"
+[[std.label]]
+dispname = "X and barB have the same nonzero eigenvalues"
+name = "X-and-\\bar{B}-have-the-same-nonzero-eigenvalues"
+uri = "exponential_algorithms/#X-and-%5Cbar%7BB%7D-have-the-same-nonzero-eigenvalues"
+[[std.label]]
+dispname = "What a large theta costs NativePade"
+name = "native-pade-large-theta"
+uri = "exponential_algorithms/#$"
diff --git a/docs/make.jl b/docs/make.jl
index ce0f0ecf6..879235b12 100644
--- a/docs/make.jl
+++ b/docs/make.jl
@@ -38,7 +38,7 @@ Downloads.download(
 #
 #     julia --project=docs -e 'using DocInventories; DocInventories.save(
 #         "docs/inventories/GeometricOptimizers.toml",
-#         Inventory("https://juliagni.github.io/GeometricOptimizers.jl/v0.5.0/objects.inv";
+#         Inventory("https://juliagni.github.io/GeometricOptimizers.jl/v0.9.0/objects.inv";
 #                   root_url = "https://juliagni.github.io/GeometricOptimizers.jl/stable/"))'
 #
 # `save` is not exported — it has to be qualified. `root_url` is not written to the TOML either; it
diff --git a/scripts/verification/all_zero_leaf.jl b/scripts/verification/all_zero_leaf.jl
index 9ce73954c..889256e68 100644
--- a/scripts/verification/all_zero_leaf.jl
+++ b/scripts/verification/all_zero_leaf.jl
@@ -12,12 +12,13 @@
 #     `MomentumMethod`, `Adam` and `AdamWithEuclideanDecay`, and stays under `GradientMethod`;
 #   - a layer without numbers, `NamedTuple()`, steps without an error.
 #
-# Run from the repository root, with the test environment:
-#   julia --startup-file=no --project=. -e 'using TestEnv; TestEnv.activate(); include("scripts/verification/all_zero_leaf.jl")'
+# Run from the repository root, with the scripts environment:
+#   julia --startup-file=no --project=scripts scripts/verification/all_zero_leaf.jl
 
 using GeometricMachineLearning
 using GeometricMachineLearning: params
-using GeometricOptimizers: optimization_step!
+# `GeometricOptimizers`' step, reached through this package: the scripts environment has no entry for it
+using GeometricMachineLearning: optimization_step!
 using Printf
 import Random
 import Zygote

From 3529019a22e34267619ec23f3b6960591aeeda2f Mon Sep 17 00:00:00 2001
From: Michael Kraus <michael.kraus@ipp.mpg.de>
Date: Mon, 12 Oct 2026 21:49:58 +0200
Subject: [PATCH 07/11] Widen the step's allocation test and import
 optimization_step! from its owner in scripts (part M1a, fix 3)

The second half of the testset in test/optimizers/optimization_step.jl
steps 64 x 64 leaves and compares the two steps within 256 bytes. Julia
min on Windows read 48 bytes apart at the 3 x 3 leaves, above the old
32-byte tolerance. A deepcopy of the gradient now costs 17 664 B (Float32)
and 34 336 B (Float64), and a temporary of the first leaf 16 464 B and
32 848 B; both mutants are caught in both precisions.

scripts/Project.toml gains GeometricOptimizers, and all_zero_leaf.jl and
sympnet_pendulum.jl import optimization_step! from it, not through
GeometricMachineLearning, whose binding is not public.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 scripts/Project.toml                              | 1 +
 scripts/reproduction/sympnets/sympnet_pendulum.jl | 3 +--
 scripts/verification/all_zero_leaf.jl             | 3 +--
 test/optimizers/optimization_step.jl              | 9 +++++----
 4 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/scripts/Project.toml b/scripts/Project.toml
index b58960d88..26b087aab 100644
--- a/scripts/Project.toml
+++ b/scripts/Project.toml
@@ -7,6 +7,7 @@ ForwardDiff = "f6369f11-7733-5829-9624-2563aa707210"
 GeometricEquations = "c85262ba-a08a-430a-b926-d29770767bf2"
 GeometricIntegrators = "dcce2d33-59f6-5b8d-9047-0defad88ae06"
 GeometricMachineLearning = "194d25b2-d3f5-49f0-af24-c124f4aa80cc"
+GeometricOptimizers = "fc236c15-5557-4942-aa65-b650f329279e"
 GeometricProblems = "18cb22b4-ad41-5c80-9c5f-710df63fbdc9"
 HDF5 = "f67ccb44-e63f-5c2f-98bd-6dc0ccc4ba2f"
 JLD2 = "033835bb-8acc-5ee8-8aae-3f567f8a3819"
diff --git a/scripts/reproduction/sympnets/sympnet_pendulum.jl b/scripts/reproduction/sympnets/sympnet_pendulum.jl
index 5be17eff3..1cd8abce5 100644
--- a/scripts/reproduction/sympnets/sympnet_pendulum.jl
+++ b/scripts/reproduction/sympnets/sympnet_pendulum.jl
@@ -4,8 +4,7 @@ Simple implementation of a pendulum for SympNets.
 
 using GeometricIntegrators
 using GeometricMachineLearning
-# `GeometricOptimizers`' step, reached through this package: the scripts environment has no entry for it
-using GeometricMachineLearning: optimization_step!
+using GeometricOptimizers: optimization_step!
 using LinearAlgebra
 using CairoMakie
 using Zygote
diff --git a/scripts/verification/all_zero_leaf.jl b/scripts/verification/all_zero_leaf.jl
index 889256e68..1cb47e2be 100644
--- a/scripts/verification/all_zero_leaf.jl
+++ b/scripts/verification/all_zero_leaf.jl
@@ -17,8 +17,7 @@
 
 using GeometricMachineLearning
 using GeometricMachineLearning: params
-# `GeometricOptimizers`' step, reached through this package: the scripts environment has no entry for it
-using GeometricMachineLearning: optimization_step!
+using GeometricOptimizers: optimization_step!
 using Printf
 import Random
 import Zygote
diff --git a/test/optimizers/optimization_step.jl b/test/optimizers/optimization_step.jl
index 8559dd457..e9b452e75 100644
--- a/test/optimizers/optimization_step.jl
+++ b/test/optimizers/optimization_step.jl
@@ -61,7 +61,8 @@ end
 
 # The training step is the hot path of `train!`. It is inferred, and on a Euclidean layer it allocates
 # nothing. With a Stiefel leaf `GeometricOptimizers`' own step allocates (about 2 to 4 kB here), and
-# the step through `NetworkOptimizer` allocates the same bytes, within 32.
+# the step through `NetworkOptimizer` allocates the same bytes, within 256: Windows reads up to 48
+# bytes apart. One copy of the 64 × 64 gradient leaf is 16 kB in `Float32`, 64 times the tolerance.
 @testset "the step through NetworkOptimizer is inferred and allocates nothing of its own ($T)" for T in REAL_ELTYPES
     for algorithm in (GradientMethod(), MomentumMethod(), Adam())
         ps = NetworkParameters((L1 = (W = rand(T, 3, 3), b = rand(T, 3)),))
@@ -71,14 +72,14 @@ end
         @test allocations(optimization_step!, ps, o, dp) == 0
         @test eltype(ps.L1.W) === eltype(ps.L1.b) === T
 
-        ps = NetworkParameters((L1 = (W = rand(T, 3, 3), b = rand(T, 3)),
+        ps = NetworkParameters((L1 = (W = rand(T, 64, 64), b = rand(T, 64)),
             L2 = (Y = rand(StiefelManifold{T}, 4, 2),)))
-        dp = NetworkParameters((L1 = (W = rand(T, 3, 3), b = rand(T, 3)),
+        dp = NetworkParameters((L1 = (W = rand(T, 64, 64), b = rand(T, 64)),
             L2 = (Y = rand(T, 4, 2),)))
         o = NetworkOptimizer(ps; algorithm)
         @inferred optimization_step!(ps, o, dp)
         @test abs(allocations(optimization_step!, ps, o, dp) -
-                  allocations(optimization_step!, ps, o.optimizer, dp)) ≤ 32
+                  allocations(optimization_step!, ps, o.optimizer, dp)) ≤ 256
         @test ps.L2.Y isa StiefelManifold{T}
     end
     @test allocations((a, b, c) -> collect(a), 1:3, 1, 1) > 0

From 1d661a779ab40752360647d714c7fa7964235d2b Mon Sep 17 00:00:00 2001
From: Michael Kraus <michael.kraus@ipp.mpg.de>
Date: Mon, 12 Oct 2026 22:58:29 +0200
Subject: [PATCH 08/11] Train with a loop over the epochs, not map with a
 closure

The map(1:n_epochs) do block around _train_epoch! made the first train! of a transformer compile for about 97 s: StandardTransformerIntegrator with 4 blocks, Batch(64, 10), 2 epochs, Julia 1.13.1, fresh process. The loop takes 12.2 to 21.9 s in three fresh processes and gives the same history. This is the cause of the four Scripts timeouts at 300 s (review finding 2). The progress meter now advances in _train_epoch!.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 src/data_loader/train.jl | 18 +++++++++++-------
 1 file changed, 11 insertions(+), 7 deletions(-)

diff --git a/src/data_loader/train.jl b/src/data_loader/train.jl
index 093745497..229703846 100644
--- a/src/data_loader/train.jl
+++ b/src/data_loader/train.jl
@@ -30,18 +30,22 @@ function train!(nn::NeuralNetwork, opt::NetworkOptimizer, dl::DataLoader, batch:
         throw(ArgumentError("`train!` needs at least one epoch; `n_epochs` is $(n_epochs)."))
     pullback = ZygotePullback(loss)
     progress = ProgressMeter.Progress(n_epochs; enabled = show_progress)
-    map(1:n_epochs) do _
-        value = _train_epoch!(nn, opt, dl, batch, pullback)
-        ProgressMeter.next!(progress; showvalues = [(:TrainingLoss, value)])
-        value
+    # A loop and not `map(1:n_epochs) do`: the closure around `_train_epoch!` makes the first call
+    # of a transformer compile several times as long.
+    history = [_train_epoch!(nn, opt, dl, batch, pullback, progress)]
+    for _ in 2:n_epochs
+        push!(history, _train_epoch!(nn, opt, dl, batch, pullback, progress))
     end
+    history
 end
 
 # The sum takes the type of the first minibatch's loss, so nothing is converted.
-function _train_epoch!(nn, opt, dl, batch, pullback)
+function _train_epoch!(nn, opt, dl, batch, pullback, progress)
     batches = batch(dl)
-    sum(_train_step!(nn, opt, dl, batch, pullback, indices) for indices in batches) /
-    length(batches)
+    total = sum(_train_step!(nn, opt, dl, batch, pullback, indices) for indices in batches)
+    value = total / length(batches)
+    ProgressMeter.next!(progress; showvalues = [(:TrainingLoss, value)])
+    value
 end
 
 function _train_step!(nn, opt, dl, batch, pullback, indices)

From 93409b04ff8a62e763a47455ee265908302dcc85 Mon Sep 17 00:00:00 2001
From: Michael Kraus <michael.kraus@ipp.mpg.de>
Date: Mon, 12 Oct 2026 23:34:58 +0200
Subject: [PATCH 09/11] Train with GeometricOptimizers' TrainingOptimizer
 directly, and with any pullback

Review findings 3 to 7 and 9 of #342.

- The NetworkOptimizer wrapper is removed. train! takes a TrainingOptimizer,
  which GML re-exports, built with TrainingOptimizer(nn.params; ...). The
  optimizer re-exports move to src/optimizers/optimizer.jl. Tests, docs,
  tutorials, scripts and the README follow.
- train! has a pullback keyword, ZygotePullback(loss) by default, so a
  Hamiltonian network trains with SymbolicPullback again. The new testset
  checks that one epoch moves the parameters by -eta times the symbolic
  gradient. Its comparison with Zygote is @test_broken: the symbolic gradient
  is the gradient of the per-sample sum, not of HNNLoss, on main too (#343,
  KNOWN_ISSUES K13).
- _default_loss has a fallback that raises an ArgumentError asking for loss.
- test/optimizers/step_size_element_type.jl and the PrecomputedGradient
  testset of gml_gradient_dispatch.jl are deleted; they test
  GeometricOptimizers and move to its suite. The _get_contents testset moves
  to test/pullbacks/get_contents.jl.
- The Stiefel allocation comparison in optimization_step.jl compared the
  wrapper with what it wrapped; without the wrapper it is removed, and the
  Euclidean == 0 bound and the @inferred checks stay.
- scripts/verification/all_zero_leaf.jl asserts its claims, and no longer
  shadows Base.one.
- The tests read opt.state, opt.cache, opt.method and opt.linesearch
  directly: GeometricOptimizers 0.9 has no public accessor for them
  (iteration_number, solution and the cache are not public).
- CHANGELOG: the 400-word bullet is split, and the entry names
  TrainingOptimizer and the pullback keyword.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 CHANGELOG.md                                  | 12 +++-
 KNOWN_ISSUES.md                               | 16 ++++-
 README.md                                     |  2 +-
 docs/src/introduction.md                      |  2 +-
 docs/src/optimizers/optimizer.md              | 20 +++---
 .../tutorials/adjusting_the_loss_function.md  |  4 +-
 docs/src/tutorials/grassmann_layer.md         |  2 +-
 .../tutorials/hamiltonian_neural_network.md   |  4 +-
 .../linear_symplectic_transformer.md          |  6 +-
 docs/src/tutorials/matrix_softmax.md          |  4 +-
 docs/src/tutorials/optimizer_comparison.md    |  8 +--
 docs/src/tutorials/softmax_comparison.md      |  4 +-
 docs/src/tutorials/symplectic_autoencoder.md  | 10 +--
 docs/src/tutorials/symplectic_transformer.md  |  6 +-
 docs/src/tutorials/sympnet_tutorial.md        |  6 +-
 .../tutorials/volume_preserving_attention.md  |  6 +-
 ...olume_preserving_transformer_rigid_body.md |  6 +-
 scripts/reproduction/harmonic_oscillator.jl   |  4 +-
 scripts/reproduction/hnn_pendulum.jl          |  4 +-
 .../reproduction/hnn_pendulum_simulation.jl   |  2 +-
 .../linear_symplectic_transformer_gpu.jl      |  6 +-
 scripts/reproduction/lnn_pendulum.jl          |  4 +-
 .../symplectic_autoencoders/online_sympnet.jl |  4 +-
 .../online_transformer_for_sae.jl             |  2 +-
 .../symplectic_autoencoders/training.jl       | 10 +--
 .../symplectic_transformer/double_pendulum.jl |  8 +--
 .../double_pendulum_short_integration.jl      |  8 +--
 .../reproduction/sympnets/sympnet_pendulum.jl |  2 +-
 .../sympnets/sympnet_toda_lattice.jl          |  2 +-
 .../softmax_comparison.jl                     |  4 +-
 .../symplectic_transformer.jl                 |  6 +-
 .../symplectic_transformer_vector_softmax.jl  |  6 +-
 .../volume_preserving_feedforward/abc_flow.jl |  2 +-
 .../double_pendulum.jl                        |  2 +-
 .../rigid_body.jl                             |  2 +-
 .../rigid_body.jl                             |  2 +-
 scripts/verification/all_zero_leaf.jl         | 24 ++++---
 src/GeometricMachineLearning.jl               | 17 +----
 src/data_loader/train.jl                      | 38 +++++++-----
 src/optimizers/optimizer.jl                   | 47 ++++----------
 .../lagrangian_neural_network_tests.jl        |  2 +-
 .../sae_error_lower_than_psd_error.jl         |  2 +-
 .../symplectic_autoencoder_tests.jl           |  4 +-
 test/data_loader/batch_data_loader_qp_test.jl |  4 +-
 .../data_loader_for_input_and_output.jl       |  2 +-
 .../data_loader_optimization_step.jl          |  2 +-
 .../optimizer_functor_with_adam.jl            |  2 +-
 test/data_loader/train.jl                     | 62 ++++++++++++++++---
 test/data_loader/training_history_eltype.jl   |  2 +-
 test/integration/losses_and_optimization.jl   |  2 +-
 test/integration/reduced_loss_optimization.jl |  2 +-
 .../integration/train_manual_architectures.jl |  2 +-
 test/layers/gradient_layer_tests.jl           |  2 +-
 test/layers/manifold_layers.jl                |  4 +-
 ...ulti_head_attention_stiefel_optim_cache.jl | 12 ++--
 ...multi_head_attention_stiefel_retraction.jl |  2 +-
 .../multi_head_attention_stiefel_setup.jl     |  2 +-
 test/layers/transformer_optimizer.jl          |  6 +-
 test/loss/training_method_losses.jl           |  4 +-
 .../adam_with_learning_rate_decay.jl          | 14 ++---
 test/optimizers/gml_gradient_dispatch.jl      | 55 ----------------
 test/optimizers/gradient_optimizer.jl         |  4 +-
 test/optimizers/momentum_optimizer.jl         |  4 +-
 test/optimizers/optimization_step.jl          | 28 ++++-----
 test/optimizers/optimizer.jl                  | 17 ++---
 test/optimizers/psd_optim.jl                  |  8 +--
 test/optimizers/step_size_element_type.jl     | 33 ----------
 .../optimizers/structured_array_parameters.jl |  4 +-
 test/optimizers/svd_optim.jl                  |  8 +--
 test/pullbacks/get_contents.jl                | 28 +++++++++
 test/runtests.jl                              |  5 +-
 71 files changed, 312 insertions(+), 340 deletions(-)
 delete mode 100644 test/optimizers/gml_gradient_dispatch.jl
 delete mode 100644 test/optimizers/step_size_element_type.jl
 create mode 100644 test/pullbacks/get_contents.jl

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 81a703c69..90a320466 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -17,9 +17,17 @@ breaking release).
 
 ### Removed (breaking)
 
-- **`Optimizer`, the callable optimizers and their aliases are gone; training moves to `NetworkOptimizer` and `train!`.** Training now uses GeometricOptimizers 0.9's `TrainingOptimizer`. `NetworkOptimizer(nn; algorithm = Adam(), linesearch = default_step_size(algorithm), retraction = Cayley())`, also accepted from a `NetworkParameters`, replaces `Optimizer`. Migrate `Optimizer(method, nn; step_size = η, retraction = cayley)` to `NetworkOptimizer(nn; algorithm = method, linesearch = η, retraction = Cayley())`, and `Optimizer(nn; AdamOptimizerWithDecay(n, T)...)` to `NetworkOptimizer(nn; AdamOptimizerWithDecay(n)...)`. The `n` of `AdamOptimizerWithDecay` counts optimization steps, one per minibatch, as in GeometricOptimizers and GML 0.8. One cache and one state now cover the whole parameter set; 0.8 had one per layer. `using GeometricOptimizers, GeometricMachineLearning` leaves `Optimizer` unqualified as GeometricOptimizers' solver, and no name is shadowed. `train!(nn, opt, dl, batch, n_epochs; loss, show_progress = true)` replaces the four callable methods `(o::Optimizer)(nn, dl, batch, n_epochs[, loss, pullback])` and `optimize_for_one_epoch!`, which is removed from the exports. Migrate `o(nn, dl, batch, n, loss; kw...)` to `train!(nn, o, dl, batch, n; loss, kw...)`. Without `loss` the defaults are those of 0.8: `TransformerLoss(batch)` for a `TransformerIntegrator` with a `Batch{:Transformer}`, `FeedForwardLoss()` for a `NeuralNetworkIntegrator`, and `AutoEncoderLoss()` for an `AutoEncoder`, each with a `Batch{:FeedForward}`. `loss` is a `NetworkLoss`; passing an `AbstractPullback` as the sixth positional argument, which 0.8 allowed, is gone. The `λY` argument and the per-call `GlobalSection` are gone: the sections, moments and iteration number live in `opt`, so a second `train!` call with the same `opt` resumes the first, and `train!` draws no random numbers beyond the minibatch shuffle. `n_epochs < 1` raises an `ArgumentError` before any step. The loss history has the element type of the loss value, so a `Float64` network on `Float32` data gives `Vector{Float64}`, where 0.8 gave `float(eltype(dl))`; the epoch accumulator keeps that type. A hand-written loop steps with `GeometricOptimizers.optimization_step!(params(nn), opt, dp)`, which GML adds for `NetworkOptimizer`; `optimization_step!(opt, λY, ps, dp)` is removed. A gradient leaf that is all zero is stepped, not skipped: NeuralNetworkParameters 0.5 gives a layer the loss does not read as a zero leaf, not `nothing`. A leaf that the loss never reads stays where it is under `GradientMethod`, `MomentumMethod` and `Adam`; under `AdamWithEuclideanDecay` such a Euclidean leaf decays. Under `MomentumMethod`, `Adam` and `AdamWithEuclideanDecay`, a leaf that the loss reads for some steps and then not keeps moving on its momentum. `scripts/verification/all_zero_leaf.jl` prints the figures. This closes known issues C1 and C2, because GML's own parameter-tree traversal and `_leaf_optim_step!` are deleted, and D4, because the per-layer caches it measured are deleted.
+- **`Optimizer`, the callable optimizers and their aliases are gone; training uses GeometricOptimizers' `TrainingOptimizer`.** `TrainingOptimizer`, re-exported by GeometricMachineLearning from GeometricOptimizers 0.9, is built on the parameters of a network: `TrainingOptimizer(nn.params; algorithm = Adam(), linesearch = default_step_size(algorithm), retraction = Cayley())`. Migrate `Optimizer(method, nn; step_size = η, retraction = cayley)` to `TrainingOptimizer(nn.params; algorithm = method, linesearch = η, retraction = Cayley())`, and `Optimizer(nn; AdamOptimizerWithDecay(n, T)...)` to `TrainingOptimizer(nn.params; AdamOptimizerWithDecay(n)...)`. The `n` of `AdamOptimizerWithDecay` counts optimization steps, one per minibatch. One cache and one state now cover the whole parameter set; 0.8 had one per layer. `Optimizer` is no longer exported by GeometricMachineLearning, so with both packages loaded, `Optimizer` is GeometricOptimizers' solver.
 
-- **The aliases `GradientOptimizer`, `MomentumOptimizer` and `AdamOptimizer` are removed**; use `GradientMethod`, `MomentumMethod` and `Adam`. These are no longer exported: `OptimizerMethod`, `GradientState`, `MomentumState`, `AdamState`, `optimization_step!`, `optimize_for_one_epoch!` and `Optimizer`. `NetworkOptimizer`, `train!` and `AdamWithEuclideanDecay` are newly exported, and the package re-exports `GradientMethod`, `MomentumMethod`, `Adam`, `AdamOptimizerWithDecay`, `DecayingStatic`, `Cayley` and `Geodesic` from GeometricOptimizers. Methods carry no element type in GeometricOptimizers 0.9: `Adam(T)` becomes `Adam()`, and `MomentumMethod(α)` becomes `MomentumMethod(; α)`. `check(::Optimizer)` is removed.
+- **`train!` replaces the four callable methods of `Optimizer`.** `train!(nn, opt, dl, batch, n_epochs; loss, pullback = ZygotePullback(loss), show_progress = true)` replaces `(o::Optimizer)(nn, dl, batch, n_epochs[, loss, pullback])` and `optimize_for_one_epoch!`, which is removed from the exports. Migrate `o(nn, dl, batch, n, loss; kw...)` to `train!(nn, o, dl, batch, n; loss, kw...)`. `loss` is a `NetworkLoss`, and `pullback` is any `AbstractPullback`: a Hamiltonian network trains with `loss = HNNLoss(arch)` and `pullback = SymbolicPullback(arch)`. The sixth positional `AbstractPullback` that 0.8 allowed is gone. Without `loss`, a `TransformerIntegrator` with a `Batch{:Transformer}` trains on `TransformerLoss(batch)`, a `NeuralNetworkIntegrator` with a `Batch{:FeedForward}` on `FeedForwardLoss()`, and an autoencoder with a `Batch{:FeedForward}` on `AutoEncoderLoss()`; any other combination raises an `ArgumentError` that asks for `loss`. The `λY` argument and the per-call `GlobalSection` are gone: `opt` holds the sections, moments and iteration number, so a second `train!` call with the same `opt` resumes the first, and `train!` draws no random numbers beyond the minibatch shuffle. `n_epochs < 1` raises an `ArgumentError` before any step. The loss history has the element type of the loss value, so a `Float64` network on `Float32` data gives a `Vector{Float64}`.
+
+- **A hand-written loop steps with `GeometricOptimizers.optimization_step!(nn.params, opt, dp)`.** The 0.8 method `optimization_step!(opt, λY, ps, dp)` is removed. GeometricMachineLearning imports `optimization_step!` but does not export it, so a caller qualifies it.
+
+- **A gradient leaf that is all zero is stepped, not skipped.** NeuralNetworkParameters 0.5 gives a layer the loss does not read a zero leaf of its own type, not `nothing`, and the step takes the whole gradient. A leaf the loss never reads stays where it is under `GradientMethod`, `MomentumMethod` and `Adam`, and a Stiefel leaf stays under all four methods; under `AdamWithEuclideanDecay` a Euclidean leaf decays. A leaf the loss reads for some steps and then not keeps moving on its momentum under `MomentumMethod`, `Adam` and `AdamWithEuclideanDecay`, and stays under `GradientMethod`. `scripts/verification/all_zero_leaf.jl` asserts them.
+
+- **This closes known issues C1, C2 and D4.** C1 and C2 close because GML's own parameter-tree traversal and `_leaf_optim_step!` are deleted; D4 closes because the per-layer caches it measured are deleted.
+
+- **The aliases `GradientOptimizer`, `MomentumOptimizer` and `AdamOptimizer` are removed**; use `GradientMethod`, `MomentumMethod` and `Adam`. These are no longer exported: `OptimizerMethod`, `GradientState`, `MomentumState`, `AdamState`, `optimization_step!`, `optimize_for_one_epoch!` and `Optimizer`. `TrainingOptimizer` (re-exported from GeometricOptimizers), `train!` and `AdamWithEuclideanDecay` are newly exported, and the package re-exports `GradientMethod`, `MomentumMethod`, `Adam`, `AdamOptimizerWithDecay`, `DecayingStatic`, `Cayley` and `Geodesic` from GeometricOptimizers. Methods carry no element type in GeometricOptimizers 0.9: `Adam(T)` becomes `Adam()`, and `MomentumMethod(α)` becomes `MomentumMethod(; α)`. `check(::Optimizer)` is removed.
 
 - **The three 3-tensor functors of `Dense` and `Linear` and the HDF5 `save` and `load` methods on `NeuralNetwork` are removed**, because AbstractNeuralNetworks 0.9 provides them. The HDF5 extension and the HDF5 weak dependency are gone; with HDF5 loaded, AbstractNeuralNetworks' extension provides `save` and `load`. Its `load` has no `backend` keyword: load on the CPU, then call `changebackend(backend, nn)`. Aqua's `piracies` check runs again with 0 piracies (was 3), and the ambiguity count is 0 (was 1). This closes issue #145 and known issue B7.
 
diff --git a/KNOWN_ISSUES.md b/KNOWN_ISSUES.md
index 5fe6041ab..98d20a2de 100644
--- a/KNOWN_ISSUES.md
+++ b/KNOWN_ISSUES.md
@@ -313,7 +313,7 @@ What is known to be wrong in GeometricMachineLearning and is not fixed.
   ```julia
   x = rand(Float32, 6, 12)
   nn = NeuralNetwork(Chain(StiefelLayer(6, 3), Dense(3, 2, tanh)), Float64)
-  train!(nn, NetworkOptimizer(nn), DataLoader(x, x[1:2, :]), Batch(4), 2; loss = FeedForwardLoss())
+  train!(nn, TrainingOptimizer(nn.params), DataLoader(x, x[1:2, :]), Batch(4), 2; loss = FeedForwardLoss())
   ```
 
 ### K8 · A `DataLoader` of one sample cannot be batched.
@@ -351,6 +351,20 @@ What is known to be wrong in GeometricMachineLearning and is not fixed.
   10 nbatches 1 samples [(1, 1), (2, 1), (3, 1), (4, 1), (5, 1), (6, 1), (7, 1), (8, 1), (9, 1)]
   ```
 
+### K13 · `SymbolicPullback(arch)` gives the gradient of the per-sample sum, not of `HNNLoss`.
+
+- **location:** `src/pullbacks/symbolic_hnn_pullback.jl`
+- **kind:** found late
+- **found:** 2026-10-12, while adding the `pullback` keyword of `train!` (#342); issue #343
+- **evidence:**
+  The value of the pullback is the batch loss `norm(F(X) - Y) / norm(Y)`, but the gradient is
+  built with `reduce = +`, so it is the gradient of the sum of the per-sample losses. On one
+  minibatch of 5 samples in `Float64`, `dL/dW1[1]` is `0.002695929479656911` by a central finite
+  difference, `0.0026959294110209093` by Zygote and `0.004388379726963325` by the symbolic
+  pullback; the symbolic gradient agrees with Zygote's gradient of the per-sample sum to `5.2e-17`.
+  `main` at cfe4aea3 gives the same numbers. `test/data_loader/train.jl` marks the comparison with
+  Zygote `@test_broken`.
+
 ## Upstream
 
 ### K2 · Revise prints EMFILE errors in the test log
diff --git a/README.md b/README.md
index b3e80f836..77e6c9ad8 100644
--- a/README.md
+++ b/README.md
@@ -44,7 +44,7 @@ backend = CUDABackend()
 g_nn = NeuralNetwork(gsympnet, backend, type)
 
 # set up the optimizer: the method and the step size are keywords
-g_opt = NetworkOptimizer(g_nn; algorithm = Adam(), linesearch = 1e-3)
+g_opt = TrainingOptimizer(g_nn.params; algorithm = Adam(), linesearch = 1e-3)
 
 const nepochs = 300
 const batch_size = 100
diff --git a/docs/src/introduction.md b/docs/src/introduction.md
index bd4537559..4e32b842c 100644
--- a/docs/src/introduction.md
+++ b/docs/src/introduction.md
@@ -22,7 +22,7 @@ Closely linked to the research presented here is the development of a software p
 train!
 ```
 
-So the docstring shows the name of the type or method, in most cases how to call it and then gives some information explaining what it does and potentially hyperlinks to other similar docstrings ([`NetworkOptimizer`](@ref) in this case); all of this information is indented by a tab. Docstrings may include other information under subheaders **Arguments** (showing the arguments the method can be supplied with), **Examples** (giving more detailed examples (including results) of how to use the method) and **Implementation** (giving details on how the method is implemented). When we reference a docstring it is always printed in blue (e.g. [`train!`](@ref)), indicating a hyperlink. In addition there is an *index of docstrings* showing all docstrings in chronological order with the associated page number.
+So the docstring shows the name of the type or method, in most cases how to call it and then gives some information explaining what it does and potentially hyperlinks to other similar docstrings ([`DataLoader`](@ref) in this case); all of this information is indented by a tab. Docstrings may include other information under subheaders **Arguments** (showing the arguments the method can be supplied with), **Examples** (giving more detailed examples (including results) of how to use the method) and **Implementation** (giving details on how the method is implemented). When we reference a docstring it is always printed in blue (e.g. [`train!`](@ref)), indicating a hyperlink. In addition there is an *index of docstrings* showing all docstrings in chronological order with the associated page number.
 
 Similar to **Library Functions**, which is included in most sections, almost every chapter concludes with a section **Chapter Summary** and an additional section **References** that shows further related reading material. The **Chapter Summary** recaps the important aspects of the corresponding chapter, states again what is new (this may be mathematical or software aspects) and gives information to what other parts of the dissertation the contents of the present chapter are relevant.
 
diff --git a/docs/src/optimizers/optimizer.md b/docs/src/optimizers/optimizer.md
index c659ba499..52f019aa4 100644
--- a/docs/src/optimizers/optimizer.md
+++ b/docs/src/optimizers/optimizer.md
@@ -5,27 +5,27 @@ the global tangent space ``\mathfrak{g}^\mathrm{hor}``, the optimizer cache, the
 global section — belongs to `GeometricOptimizers` and is described in its documentation, under
 [Optimization on Homogeneous Spaces](@extref GeometricOptimizers :doc:`manifold_optimizers`) and
 [Retractions](@extref GeometricOptimizers :doc:`retractions`). The training step itself is
-`GeometricOptimizers`' too: its `TrainingOptimizer` steps a whole parameter set with a gradient the
-caller computed, and applies the right update to each leaf — a retraction for a weight on a
-manifold, ordinary arithmetic for a Euclidean one.
+`GeometricOptimizers`' too: its
+[`TrainingOptimizer`](@extref GeometricOptimizers GeometricOptimizers.TrainingOptimizer), which
+`GeometricMachineLearning` re-exports, steps a whole parameter set with a gradient the caller
+computed, and applies the right update to each leaf — a retraction for a weight on a manifold,
+ordinary arithmetic for a Euclidean one. It is built on the parameters of a network, `nn.params`.
 
-What `GeometricMachineLearning` adds is the part that is about *neural networks*: a
-[`NetworkOptimizer`](@ref) built from a `NeuralNetwork`, and [`train!`](@ref), which draws the
-minibatches of a data loader over epochs, computes the gradient of the loss on each with automatic
-differentiation, and takes one step per minibatch.
+What `GeometricMachineLearning` adds is the part that is about *neural networks*: [`train!`](@ref),
+which draws the minibatches of a data loader over epochs, computes the gradient of the loss on each
+with automatic differentiation or a symbolic pullback, and takes one step per minibatch.
 
 The gradient comes from one minibatch at a time, so there is no objective function to hand a line
 search; the step size is a number or a `GeometricOptimizers.DecayingStatic` schedule:
 
 ```julia
-opt = NetworkOptimizer(nn; algorithm = Adam(), linesearch = 1e-3)
-opt = NetworkOptimizer(nn; AdamOptimizerWithDecay(n_steps)...)
+opt = TrainingOptimizer(nn.params; algorithm = Adam(), linesearch = 1e-3)
+opt = TrainingOptimizer(nn.params; AdamOptimizerWithDecay(n_steps)...)
 history = train!(nn, opt, dl, Batch(batch_size), n_epochs)
 ```
 
 ## Library Functions
 
 ```@docs
-NetworkOptimizer
 train!
 ```
diff --git a/docs/src/tutorials/adjusting_the_loss_function.md b/docs/src/tutorials/adjusting_the_loss_function.md
index 6991e5417..8431f933a 100644
--- a/docs/src/tutorials/adjusting_the_loss_function.md
+++ b/docs/src/tutorials/adjusting_the_loss_function.md
@@ -26,7 +26,7 @@ data = DataLoader(sol; suppress_info = true)
 nn = NeuralNetwork(GSympNet(2))
 
 # train the network
-o = NetworkOptimizer(nn; algorithm = Adam())
+o = TrainingOptimizer(nn.params; algorithm = Adam())
 batch = Batch(32)
 n_epochs = 30
 loss = FeedForwardLoss()
@@ -71,7 +71,7 @@ And we train the same network with this new loss:
 ```@example change_loss
 loss = CustomLoss()
 nn_custom = NeuralNetwork(GSympNet(2))
-o_custom = NetworkOptimizer(nn_custom; algorithm = Adam())
+o_custom = TrainingOptimizer(nn_custom.params; algorithm = Adam())
 loss_array = train!(nn_custom, o_custom, data, batch, n_epochs; loss, show_progress = false)
 print(loss_array[end])
 ```
diff --git a/docs/src/tutorials/grassmann_layer.md b/docs/src/tutorials/grassmann_layer.md
index 2ff9f7e87..60d9cc345 100644
--- a/docs/src/tutorials/grassmann_layer.md
+++ b/docs/src/tutorials/grassmann_layer.md
@@ -262,7 +262,7 @@ function compute_gradient(ps::NetworkParameters)
 end
 
 # note the very high value for the learning rate
-optimizer = NetworkOptimizer(nn; algorithm = Adam(), linesearch = 1e-1)
+optimizer = TrainingOptimizer(nn.params; algorithm = Adam(), linesearch = 1e-1)
 
 nothing # hide
 ```
diff --git a/docs/src/tutorials/hamiltonian_neural_network.md b/docs/src/tutorials/hamiltonian_neural_network.md
index 86df4193e..136af1c75 100644
--- a/docs/src/tutorials/hamiltonian_neural_network.md
+++ b/docs/src/tutorials/hamiltonian_neural_network.md
@@ -42,7 +42,7 @@ We can now train the network
 ```@example hnn
 batch = Batch(10)
 n_epochs = 100
-o = NetworkOptimizer(hnn; algorithm = Adam())
+o = TrainingOptimizer(hnn.params; algorithm = Adam())
 loss_array = train!(hnn, o, dl, batch, n_epochs; loss)
 using CairoMakie # hide
 lines(loss_array) # hide
@@ -84,7 +84,7 @@ nothing # hide
 We can now train the network:
 
 ```@example hnn
-o_pairs = NetworkOptimizer(hnn_pairs; algorithm = Adam())
+o_pairs = TrainingOptimizer(hnn_pairs.params; algorithm = Adam())
 loss_array_pairs = train!(hnn_pairs, o_pairs, dl_pairs, batch, n_epochs; loss = loss_pairs)
 lines(loss_array_pairs) # hide
 ```
diff --git a/docs/src/tutorials/linear_symplectic_transformer.md b/docs/src/tutorials/linear_symplectic_transformer.md
index 6a5004c1c..ee6d6aca4 100644
--- a/docs/src/tutorials/linear_symplectic_transformer.md
+++ b/docs/src/tutorials/linear_symplectic_transformer.md
@@ -65,9 +65,9 @@ nn_sympnet = NeuralNetwork(arch_sympnet)
 
 o_pairing = AdamOptimizerWithDecay(n_epochs)
 
-o_standard = NetworkOptimizer(nn_standard; o_pairing...)
-o_symplectic = NetworkOptimizer(nn_symplectic; o_pairing...)
-o_sympnet = NetworkOptimizer(nn_sympnet; o_pairing...)
+o_standard = TrainingOptimizer(nn_standard.params; o_pairing...)
+o_symplectic = TrainingOptimizer(nn_symplectic.params; o_pairing...)
+o_sympnet = TrainingOptimizer(nn_sympnet.params; o_pairing...)
 
 batch = Batch(batch_size, seq_length)
 batch2 = Batch(batch_size)
diff --git a/docs/src/tutorials/matrix_softmax.md b/docs/src/tutorials/matrix_softmax.md
index 89912801c..977166d6c 100644
--- a/docs/src/tutorials/matrix_softmax.md
+++ b/docs/src/tutorials/matrix_softmax.md
@@ -89,8 +89,8 @@ Training is done with the `Adam`:
 ```@example softmax_comparison
 o_method = Adam()
 
-o1 = NetworkOptimizer(nn1; algorithm = o_method)
-o2 = NetworkOptimizer(nn2; algorithm = o_method)
+o1 = TrainingOptimizer(nn1.params; algorithm = o_method)
+o2 = TrainingOptimizer(nn2.params; algorithm = o_method)
 
 batch = Batch(batch_size, seq_length)
 
diff --git a/docs/src/tutorials/optimizer_comparison.md b/docs/src/tutorials/optimizer_comparison.md
index 9c3e46188..a59b1a4f3 100644
--- a/docs/src/tutorials/optimizer_comparison.md
+++ b/docs/src/tutorials/optimizer_comparison.md
@@ -27,14 +27,14 @@ init_cont = Tuple(init_con)
 mred = RGBf(214 / 256, 39 / 256, 40 / 256) # hide
 scatter!(ax, init_cont..., f(init_cont...); color = mred, marker = :star5)
 
-# `NetworkOptimizer` takes a `NeuralNetwork` or a `NetworkParameters`, so the bare `NamedTuple` of
-# weights has to be wrapped in the latter.
+# `TrainingOptimizer` takes a `NetworkParameters`, so the bare `NamedTuple` of weights has to be
+# wrapped in one.
 weights = NetworkParameters((xy = init_con, ))
 η = 1e-3
 method1 = GradientMethod()
 method2 = Adam()
-optimizer1 = NetworkOptimizer(weights; algorithm = method1, linesearch = η)
-optimizer2 = NetworkOptimizer(weights; algorithm = method2, linesearch = η)
+optimizer1 = TrainingOptimizer(weights; algorithm = method1, linesearch = η)
+optimizer2 = TrainingOptimizer(weights; algorithm = method2, linesearch = η)
 
 fig
 ```
diff --git a/docs/src/tutorials/softmax_comparison.md b/docs/src/tutorials/softmax_comparison.md
index d8a3d8d3b..a6ae48e32 100644
--- a/docs/src/tutorials/softmax_comparison.md
+++ b/docs/src/tutorials/softmax_comparison.md
@@ -74,8 +74,8 @@ nn2 = NeuralNetwork(arch2)
 
 o_method = Adam()
 
-o1 = NetworkOptimizer(nn1; algorithm = o_method)
-o2 = NetworkOptimizer(nn2; algorithm = o_method)
+o1 = TrainingOptimizer(nn1.params; algorithm = o_method)
+o2 = TrainingOptimizer(nn2.params; algorithm = o_method)
 
 batch = Batch(batch_size, seq_length)
 
diff --git a/docs/src/tutorials/symplectic_autoencoder.md b/docs/src/tutorials/symplectic_autoencoder.md
index 061d642a4..7e240ecd9 100644
--- a/docs/src/tutorials/symplectic_autoencoder.md
+++ b/docs/src/tutorials/symplectic_autoencoder.md
@@ -15,7 +15,7 @@ Main.remark(raw"As with any neural network we have to make the following choices
 " * Main.indentation * raw"5. choose a *number of epochs*,
 " * Main.indentation * raw"where points 1 and 3 depend on a variable number of hyperparameters.")
 ```
-For the symplectic autoencoder point 1 is done by calling [`SymplecticAutoencoder`](@ref), point 2 is done by calling `NeuralNetwork`, point 3 is done by calling [`NetworkOptimizer`](@ref) and point 4 is done by calling [`Batch`](@ref).
+For the symplectic autoencoder point 1 is done by calling [`SymplecticAutoencoder`](@ref), point 2 is done by calling `NeuralNetwork`, point 3 is done by calling [`TrainingOptimizer`](@extref GeometricOptimizers GeometricOptimizers.TrainingOptimizer) and point 4 is done by calling [`Batch`](@ref).
 
 ## The system
 
@@ -111,7 +111,7 @@ sae_arch = SymplecticAutoencoder(dl_cpu.input_dim, reduced_dim; n_encoder_blocks
 nothing # hide
 ```
 
-Training a neural network is usually done by calling [`train!`](@ref) with a [`NetworkOptimizer`](@ref) in `GeometricMachineLearning`. [`PSDArch`](@ref) however can be solved directly by using singular value decomposition and this is done by calling [`solve!`](@ref):  
+Training a neural network is usually done by calling [`train!`](@ref) with a [`TrainingOptimizer`](@extref GeometricOptimizers GeometricOptimizers.TrainingOptimizer) in `GeometricMachineLearning`. [`PSDArch`](@ref) however can be solved directly by using singular value decomposition and this is done by calling [`solve!`](@ref):  
 
 ```@example toda_lattice
 psd_nn_cpu = NeuralNetwork(psd_arch, CPU(), eltype(dl_cpu))
@@ -145,7 +145,7 @@ dl = DataLoader(dl_cpu, backend, Float32)
 
 
 sae_nn_gpu = NeuralNetwork(sae_arch, CUDADevice(), Float32)
-o = NetworkOptimizer(sae_nn_gpu; AdamOptimizerWithDecay(integrator_train_epochs)...)
+o = TrainingOptimizer(sae_nn_gpu.params; AdamOptimizerWithDecay(integrator_train_epochs)...)
 
 # train the network
 train!(sae_nn_gpu, o, dl, Batch(batch_size), n_epochs)
@@ -282,7 +282,7 @@ integrator_nn = NeuralNetwork(integrator_architecture, backend)
 
 integrator_pairing = AdamOptimizerWithDecay(integrator_train_epochs)
 
-o_integrator = NetworkOptimizer(integrator_nn; integrator_pairing...)
+o_integrator = TrainingOptimizer(integrator_nn.params; integrator_pairing...)
 
 dl = dl_cpu # hide
 # map from autoencoder type to integrator type
@@ -477,7 +477,7 @@ const integrator_architecture2 = StandardTransformerIntegrator(reduced_dim2;
                                                                             upscaling_activation = tanh)
 integrator_nn2 = NeuralNetwork(integrator_architecture2, backend)
 const integrator_pairing2 = AdamOptimizerWithDecay(integrator_train_epochs)
-const o_integrator2 = NetworkOptimizer(integrator_nn2; integrator_pairing2...)
+const o_integrator2 = TrainingOptimizer(integrator_nn2.params; integrator_pairing2...)
 
 loss2 = GeometricMachineLearning.ReducedLoss(encoder(psd_nn2), decoder(psd_nn2))
 nothing # hide
diff --git a/docs/src/tutorials/symplectic_transformer.md b/docs/src/tutorials/symplectic_transformer.md
index 0528fd5bc..a97ae7cdc 100644
--- a/docs/src/tutorials/symplectic_transformer.md
+++ b/docs/src/tutorials/symplectic_transformer.md
@@ -65,9 +65,9 @@ nn_sympnet = NeuralNetwork(arch_sympnet)
 
 o_pairing = AdamOptimizerWithDecay(n_epochs)
 
-o_standard = NetworkOptimizer(nn_standard; o_pairing...)
-o_symplectic = NetworkOptimizer(nn_symplectic; o_pairing...)
-o_sympnet = NetworkOptimizer(nn_sympnet; o_pairing...)
+o_standard = TrainingOptimizer(nn_standard.params; o_pairing...)
+o_symplectic = TrainingOptimizer(nn_symplectic.params; o_pairing...)
+o_sympnet = TrainingOptimizer(nn_sympnet.params; o_pairing...)
 
 batch = Batch(batch_size, seq_length)
 batch2 = Batch(batch_size)
diff --git a/docs/src/tutorials/sympnet_tutorial.md b/docs/src/tutorials/sympnet_tutorial.md
index 846f0e067..4678e3f71 100644
--- a/docs/src/tutorials/sympnet_tutorial.md
+++ b/docs/src/tutorials/sympnet_tutorial.md
@@ -102,8 +102,8 @@ We have to define an [optimizer](@extref GeometricOptimizers Standard-Neural-Net
 # set up optimizer; for this we first need to specify the optimization method
 opt_method = Adam()
 # we then build the optimizer, which allocates the cache in the element type of the network
-la_opt = NetworkOptimizer(la_nn; algorithm = opt_method)
-g_opt = NetworkOptimizer(g_nn; algorithm = opt_method)
+la_opt = TrainingOptimizer(la_nn.params; algorithm = opt_method)
+g_opt = TrainingOptimizer(g_nn.params; algorithm = opt_method)
 
 nothing # hide
 ```
@@ -242,7 +242,7 @@ parameterlength(rn_nn)
 We now train the network ``\ldots``
 
 ```@example sympnet
-rn_opt = NetworkOptimizer(rn_nn; algorithm = opt_method)
+rn_opt = TrainingOptimizer(rn_nn.params; algorithm = opt_method)
 
 rn_loss_array = train!(rn_nn, rn_opt, dl, batch, nepochs; show_progress = false)
 nothing # hide
diff --git a/docs/src/tutorials/volume_preserving_attention.md b/docs/src/tutorials/volume_preserving_attention.md
index 0a8ad3a67..2414e8701 100644
--- a/docs/src/tutorials/volume_preserving_attention.md
+++ b/docs/src/tutorials/volume_preserving_attention.md
@@ -117,9 +117,9 @@ Next we train the networks (here we pick a batch size of 30 and train for 1000 e
 
 ```@setup volume_preserving_attention
 function set_up_optimizers(nn_skew, nn_arb, nn_comp)
-    o_skew = NetworkOptimizer(nn_skew; algorithm = Adam())
-    o_arb  = NetworkOptimizer(nn_arb; algorithm = Adam())
-    o_comp = NetworkOptimizer(nn_comp; algorithm = Adam())
+    o_skew = TrainingOptimizer(nn_skew.params; algorithm = Adam())
+    o_arb  = TrainingOptimizer(nn_arb.params; algorithm = Adam())
+    o_comp = TrainingOptimizer(nn_comp.params; algorithm = Adam())
 
     o_skew, o_arb, o_comp
 end
diff --git a/docs/src/tutorials/volume_preserving_transformer_rigid_body.md b/docs/src/tutorials/volume_preserving_transformer_rigid_body.md
index f60d1d2d5..c3095f13e 100644
--- a/docs/src/tutorials/volume_preserving_transformer_rigid_body.md
+++ b/docs/src/tutorials/volume_preserving_transformer_rigid_body.md
@@ -201,9 +201,9 @@ const feedforward_batch = Batch(batch_size)
 const transformer_batch = Batch(batch_size, seq_length, seq_length)
 const opt_pairing = AdamOptimizerWithDecay(n_epochs; η₁ = 1e-2, η₂ = 1e-6)
 
-o_vpff = NetworkOptimizer(nn_vpff; opt_pairing...)
-o_vpt = NetworkOptimizer(nn_vpt; opt_pairing...)
-o_st = NetworkOptimizer(nn_st; opt_pairing...)
+o_vpff = TrainingOptimizer(nn_vpff.params; opt_pairing...)
+o_vpt = TrainingOptimizer(nn_vpt.params; opt_pairing...)
+o_st = TrainingOptimizer(nn_st.params; opt_pairing...)
 nothing # hide
 ```
 ```julia
diff --git a/scripts/reproduction/harmonic_oscillator.jl b/scripts/reproduction/harmonic_oscillator.jl
index 70fba8365..27bfb9691 100644
--- a/scripts/reproduction/harmonic_oscillator.jl
+++ b/scripts/reproduction/harmonic_oscillator.jl
@@ -13,13 +13,13 @@ ensemble_solution = exact_solution(ensemble_problem)
 
 include("../utilities/ensemble_plots.jl")
 
-# `GSympNet` trains through the generic `DataLoader` + `Batch` + `NetworkOptimizer` pipeline directly,
+# `GSympNet` trains through the generic `DataLoader` + `Batch` + `TrainingOptimizer` pipeline directly,
 # as `scripts/reproduction/sympnets/sympnet_toda_lattice.jl` already does.
 dl = DataLoader(ensemble_solution)
 
 arch = GSympNet(dl; n_layers = 4, upscaling_dimension = 10)
 nn = NeuralNetwork(arch, Float64)
-o = NetworkOptimizer(nn; algorithm = Adam())
+o = TrainingOptimizer(nn.params; algorithm = Adam())
 batch = Batch(100)
 n_epochs = smoke_size(1000, 2)
 
diff --git a/scripts/reproduction/hnn_pendulum.jl b/scripts/reproduction/hnn_pendulum.jl
index 889795b64..7f8faee83 100644
--- a/scripts/reproduction/hnn_pendulum.jl
+++ b/scripts/reproduction/hnn_pendulum.jl
@@ -34,9 +34,9 @@ dl = DataLoader(input, output)
 # in the data. It is the loss `NetworkLoss(hnn)` returns, so naming it is for the reader.
 loss = HNNLoss(hnn)
 
-# The step size is the `NetworkOptimizer`'s `linesearch`, not the method's: the same `Adam()` trains
+# The step size is the `TrainingOptimizer`'s `linesearch`, not the method's: the same `Adam()` trains
 # at any learning rate. `1e-3` is already Adam's default here and is named for the reader.
-optimizer = NetworkOptimizer(nn; algorithm = Adam(), linesearch = 1e-3)
+optimizer = TrainingOptimizer(nn.params; algorithm = Adam(), linesearch = 1e-3)
 total_loss = train!(
     nn, optimizer, dl, Batch(10), nepochs; loss = loss, show_progress = true)
 
diff --git a/scripts/reproduction/hnn_pendulum_simulation.jl b/scripts/reproduction/hnn_pendulum_simulation.jl
index 3f39be03b..2fd637503 100644
--- a/scripts/reproduction/hnn_pendulum_simulation.jl
+++ b/scripts/reproduction/hnn_pendulum_simulation.jl
@@ -36,7 +36,7 @@ dl = DataLoader(input, output)
 # symbolic derivative again.
 const loss = HNNLoss(hnn)
 
-optimizer = NetworkOptimizer(nn; algorithm = Adam(), linesearch = 1e-3)
+optimizer = TrainingOptimizer(nn.params; algorithm = Adam(), linesearch = 1e-3)
 total_loss = train!(
     nn, optimizer, dl, Batch(10), nepochs; loss = loss, show_progress = true)
 
diff --git a/scripts/reproduction/linear_symplectic_transformer_gpu.jl b/scripts/reproduction/linear_symplectic_transformer_gpu.jl
index 80bce685a..70949b7f7 100644
--- a/scripts/reproduction/linear_symplectic_transformer_gpu.jl
+++ b/scripts/reproduction/linear_symplectic_transformer_gpu.jl
@@ -39,9 +39,9 @@ nn_sympnet = NeuralNetwork(arch_sympnet, backend, T)
 
 o_method = Adam()
 
-o_standard = NetworkOptimizer(nn_standard; algorithm = o_method)
-o_symplectic = NetworkOptimizer(nn_symplectic; algorithm = o_method)
-o_sympnet = NetworkOptimizer(nn_sympnet; algorithm = o_method)
+o_standard = TrainingOptimizer(nn_standard.params; algorithm = o_method)
+o_symplectic = TrainingOptimizer(nn_symplectic.params; algorithm = o_method)
+o_sympnet = TrainingOptimizer(nn_sympnet.params; algorithm = o_method)
 
 batch = Batch(batch_size, seq_length)
 batch2 = Batch(batch_size)
diff --git a/scripts/reproduction/lnn_pendulum.jl b/scripts/reproduction/lnn_pendulum.jl
index 544fae832..e0abd6080 100644
--- a/scripts/reproduction/lnn_pendulum.jl
+++ b/scripts/reproduction/lnn_pendulum.jl
@@ -43,10 +43,10 @@ dl = DataLoader(input, output)
 # `LNNLoss(arch)` is what `NetworkLoss(arch)` returns; naming it is for the reader.
 loss = LNNLoss(arch)
 
-# The step size is the `NetworkOptimizer`'s `linesearch`, not the method's. `LNNLoss` needs a larger
+# The step size is the `TrainingOptimizer`'s `linesearch`, not the method's. `LNNLoss` needs a larger
 # one than `HNNLoss` does: its gradient reaches the parameters through a solve against the velocity
 # Hessian, where `HNNLoss` is linear in the gradient of the learned Hamiltonian.
-optimizer = NetworkOptimizer(nn; algorithm = Adam(), linesearch = 1e-2)
+optimizer = TrainingOptimizer(nn.params; algorithm = Adam(), linesearch = 1e-2)
 total_loss = train!(
     nn, optimizer, dl, Batch(10), nepochs; loss = loss, show_progress = true)
 
diff --git a/scripts/reproduction/symplectic_autoencoders/online_sympnet.jl b/scripts/reproduction/symplectic_autoencoders/online_sympnet.jl
index d93ced980..810904131 100644
--- a/scripts/reproduction/symplectic_autoencoders/online_sympnet.jl
+++ b/scripts/reproduction/symplectic_autoencoders/online_sympnet.jl
@@ -38,7 +38,7 @@ const n_epochs = smoke_size(262144, 2)
 const batch_size = 4096
 
 sae_pairing = AdamOptimizerWithDecay(n_epochs)
-o = NetworkOptimizer(sae_nn; sae_pairing...)
+o = TrainingOptimizer(sae_nn.params; sae_pairing...)
 
 println("Number of batches: ", GeometricMachineLearning.number_of_batches(dl, Batch(batch_size)))
 
@@ -106,7 +106,7 @@ integrator_architecture = StandardTransformerIntegrator(
 # the fallback raises is `Functor not defined`, not a `MethodError`.
 integrator_nn = NeuralNetwork(integrator_architecture, backend, Float32)
 integrator_pairing = AdamOptimizerWithDecay(integrator_train_epochs)
-o_integrator = NetworkOptimizer(integrator_nn; integrator_pairing...)
+o_integrator = TrainingOptimizer(integrator_nn.params; integrator_pairing...)
 
 loss = GeometricMachineLearning.ReducedLoss(encoder(sae_nn), decoder(sae_nn))
 
diff --git a/scripts/reproduction/symplectic_autoencoders/online_transformer_for_sae.jl b/scripts/reproduction/symplectic_autoencoders/online_transformer_for_sae.jl
index ef657909f..9e9215f96 100644
--- a/scripts/reproduction/symplectic_autoencoders/online_transformer_for_sae.jl
+++ b/scripts/reproduction/symplectic_autoencoders/online_transformer_for_sae.jl
@@ -52,7 +52,7 @@ const integrator_architecture = StandardTransformerIntegrator(
 # the fallback raises is `Functor not defined`, not a `MethodError`.
 const integrator_nn = NeuralNetwork(integrator_architecture, backend, Float32)
 const integrator_pairing = AdamOptimizerWithDecay(integrator_train_epochs)
-const o_integrator = NetworkOptimizer(integrator_nn; integrator_pairing...)
+const o_integrator = TrainingOptimizer(integrator_nn.params; integrator_pairing...)
 
 loss = GeometricMachineLearning.ReducedLoss(encoder(sae_nn), decoder(sae_nn))
 
diff --git a/scripts/reproduction/symplectic_autoencoders/training.jl b/scripts/reproduction/symplectic_autoencoders/training.jl
index 1381f5da9..b0fed708f 100644
--- a/scripts/reproduction/symplectic_autoencoders/training.jl
+++ b/scripts/reproduction/symplectic_autoencoders/training.jl
@@ -28,11 +28,11 @@ n_epochs = smoke_size(100, 2)
 batch_size = smoke_size(512, 16)
 n_range = 2:1:smoke_size(15, 3)
 μ_range = (T(0.51), T(0.625), T(0.55), T(0.47))
-# The learning rate is the `NetworkOptimizer`'s `linesearch`, not the method's: `Adam` has no `η`
+# The learning rate is the `TrainingOptimizer`'s `linesearch`, not the method's: `Adam` has no `η`
 # field, and `β₁`, `β₂` and `δ` are keywords.
 opt = Adam(; β₁ = 0.9, β₂ = 0.99, δ = 1.0e-8)
 const step_size = T(0.001)
-# The retraction belongs to the `NetworkOptimizer`, not to the layer: `PSDLayer(M, N)` takes no
+# The retraction belongs to the `TrainingOptimizer`, not to the layer: `PSDLayer(M, N)` takes no
 # keyword.
 retraction = Cayley()
 
@@ -88,7 +88,7 @@ end
 # architecture rather than a bare `Chain` is what makes `encoder(nn)` and `decoder(nn)` available,
 # and those are what `HRedSys` needs.
 #
-# Training is `DataLoader` + `Batch` + `NetworkOptimizer`, and the loop is `train!`: the batch is a
+# Training is `DataLoader` + `Batch` + `TrainingOptimizer`, and the loop is `train!`: the batch is a
 # `Batch`, and a failed step raises its error.
 function get_nn_encoder_decoder(; n = 5, n_epochs = 500, opt = opt, T = T)
     sae_arch = SymplecticAutoencoder(
@@ -96,8 +96,8 @@ function get_nn_encoder_decoder(; n = 5, n_epochs = 500, opt = opt, T = T)
         n_encoder_layers = 2, n_decoder_layers = 2)
     sae_nn = NeuralNetwork(sae_arch, backend, T)
 
-    optimizer_instance = NetworkOptimizer(
-        sae_nn; algorithm = opt, linesearch = step_size, retraction = retraction)
+    optimizer_instance = TrainingOptimizer(
+        sae_nn.params; algorithm = opt, linesearch = step_size, retraction = retraction)
     train!(sae_nn, optimizer_instance, dl, Batch(batch_size), n_epochs)
 
     sae_nn
diff --git a/scripts/reproduction/symplectic_transformer/double_pendulum.jl b/scripts/reproduction/symplectic_transformer/double_pendulum.jl
index 8bb463d40..a5dc65857 100644
--- a/scripts/reproduction/symplectic_transformer/double_pendulum.jl
+++ b/scripts/reproduction/symplectic_transformer/double_pendulum.jl
@@ -72,10 +72,10 @@ const n_epochs = smoke_size(10000, 2)
 
 o_method = Adam()
 
-o1 = NetworkOptimizer(nn1; algorithm = o_method)
-o2 = NetworkOptimizer(nn2; algorithm = o_method)
-o3 = NetworkOptimizer(nn3; algorithm = o_method)
-o4 = NetworkOptimizer(nn4; algorithm = o_method)
+o1 = TrainingOptimizer(nn1.params; algorithm = o_method)
+o2 = TrainingOptimizer(nn2.params; algorithm = o_method)
+o3 = TrainingOptimizer(nn3.params; algorithm = o_method)
+o4 = TrainingOptimizer(nn4.params; algorithm = o_method)
 
 batch = Batch(batch_size, seq_length)
 
diff --git a/scripts/reproduction/symplectic_transformer/double_pendulum_short_integration.jl b/scripts/reproduction/symplectic_transformer/double_pendulum_short_integration.jl
index 747757aa2..962730e18 100644
--- a/scripts/reproduction/symplectic_transformer/double_pendulum_short_integration.jl
+++ b/scripts/reproduction/symplectic_transformer/double_pendulum_short_integration.jl
@@ -70,10 +70,10 @@ const n_epochs = smoke_size(3000, 2)
 
 o_method = Adam()
 
-o1 = NetworkOptimizer(nn1; algorithm = o_method)
-o2 = NetworkOptimizer(nn2; algorithm = o_method)
-o3 = NetworkOptimizer(nn3; algorithm = o_method)
-o4 = NetworkOptimizer(nn4; algorithm = o_method)
+o1 = TrainingOptimizer(nn1.params; algorithm = o_method)
+o2 = TrainingOptimizer(nn2.params; algorithm = o_method)
+o3 = TrainingOptimizer(nn3.params; algorithm = o_method)
+o4 = TrainingOptimizer(nn4.params; algorithm = o_method)
 
 batch = Batch(batch_size, seq_length)
 
diff --git a/scripts/reproduction/sympnets/sympnet_pendulum.jl b/scripts/reproduction/sympnets/sympnet_pendulum.jl
index 1cd8abce5..b8cfeeb2c 100644
--- a/scripts/reproduction/sympnets/sympnet_pendulum.jl
+++ b/scripts/reproduction/sympnets/sympnet_pendulum.jl
@@ -69,7 +69,7 @@ end
 # define momentum optimizer and initialize
 method = Adam()
 # initial gradients for calling Cache constructor
-opt = NetworkOptimizer(ps; algorithm = method)
+opt = TrainingOptimizer(ps; algorithm = method)
 
 # training 
 println("initial loss: ", full_loss(ps, q, p))
diff --git a/scripts/reproduction/sympnets/sympnet_toda_lattice.jl b/scripts/reproduction/sympnets/sympnet_toda_lattice.jl
index 0036536af..c4f0f0d96 100644
--- a/scripts/reproduction/sympnets/sympnet_toda_lattice.jl
+++ b/scripts/reproduction/sympnets/sympnet_toda_lattice.jl
@@ -27,7 +27,7 @@ arch = GSympNet(dl; upscaling_dimension = 4)
 nn = NeuralNetwork(arch)
 o_method = Adam()
 
-o = NetworkOptimizer(nn; algorithm = o_method)
+o = TrainingOptimizer(nn.params; algorithm = o_method)
 
 batch = Batch(256)
 const n_epochs = smoke_size(1024, 2)
diff --git a/scripts/reproduction/transformer_integrator/softmax_comparison.jl b/scripts/reproduction/transformer_integrator/softmax_comparison.jl
index 7edfda482..4ec062947 100644
--- a/scripts/reproduction/transformer_integrator/softmax_comparison.jl
+++ b/scripts/reproduction/transformer_integrator/softmax_comparison.jl
@@ -51,8 +51,8 @@ nn2 = NeuralNetwork(arch2)
 
 o_method = Adam()
 
-o1 = NetworkOptimizer(nn1; algorithm = o_method)
-o2 = NetworkOptimizer(nn2; algorithm = o_method)
+o1 = TrainingOptimizer(nn1.params; algorithm = o_method)
+o2 = TrainingOptimizer(nn2.params; algorithm = o_method)
 
 batch = Batch(batch_size, seq_length)
 
diff --git a/scripts/reproduction/transformer_integrator/symplectic_transformer.jl b/scripts/reproduction/transformer_integrator/symplectic_transformer.jl
index ae2fc08c8..62e70a10f 100644
--- a/scripts/reproduction/transformer_integrator/symplectic_transformer.jl
+++ b/scripts/reproduction/transformer_integrator/symplectic_transformer.jl
@@ -57,9 +57,9 @@ const n_epochs = smoke_size(3000, 2)
 
 o_method = Adam()
 
-o1 = NetworkOptimizer(nn1; algorithm = o_method)
-o2 = NetworkOptimizer(nn2; algorithm = o_method)
-o3 = NetworkOptimizer(nn3; algorithm = o_method)
+o1 = TrainingOptimizer(nn1.params; algorithm = o_method)
+o2 = TrainingOptimizer(nn2.params; algorithm = o_method)
+o3 = TrainingOptimizer(nn3.params; algorithm = o_method)
 
 batch = Batch(batch_size, seq_length)
 
diff --git a/scripts/reproduction/transformer_integrator/symplectic_transformer_vector_softmax.jl b/scripts/reproduction/transformer_integrator/symplectic_transformer_vector_softmax.jl
index 261943def..5a091204b 100644
--- a/scripts/reproduction/transformer_integrator/symplectic_transformer_vector_softmax.jl
+++ b/scripts/reproduction/transformer_integrator/symplectic_transformer_vector_softmax.jl
@@ -57,9 +57,9 @@ const n_epochs = smoke_size(3000, 2)
 
 o_method = Adam()
 
-o1 = NetworkOptimizer(nn1; algorithm = o_method)
-o2 = NetworkOptimizer(nn2; algorithm = o_method)
-o3 = NetworkOptimizer(nn3; algorithm = o_method)
+o1 = TrainingOptimizer(nn1.params; algorithm = o_method)
+o2 = TrainingOptimizer(nn2.params; algorithm = o_method)
+o3 = TrainingOptimizer(nn3.params; algorithm = o_method)
 
 batch = Batch(batch_size, seq_length)
 
diff --git a/scripts/reproduction/volume_preserving_feedforward/abc_flow.jl b/scripts/reproduction/volume_preserving_feedforward/abc_flow.jl
index 2f3e7ee91..635831906 100644
--- a/scripts/reproduction/volume_preserving_feedforward/abc_flow.jl
+++ b/scripts/reproduction/volume_preserving_feedforward/abc_flow.jl
@@ -47,7 +47,7 @@ model = VolumePreservingFeedForward(sys_dim, n_blocks, n_linear, activation)
 
 nn = NeuralNetwork(model, backend, T)
 
-o = NetworkOptimizer(nn; algorithm = opt_method)
+o = TrainingOptimizer(nn.params; algorithm = opt_method)
 
 # `Batch(batch_size, 1)` is a `Batch{:Transformer}` of sequence length one, and the network here is
 # a `NeuralNetworkIntegrator`, whose loss defaults only for a `Batch{:FeedForward}`.
diff --git a/scripts/reproduction/volume_preserving_feedforward/double_pendulum.jl b/scripts/reproduction/volume_preserving_feedforward/double_pendulum.jl
index 6526fe36d..365e75772 100644
--- a/scripts/reproduction/volume_preserving_feedforward/double_pendulum.jl
+++ b/scripts/reproduction/volume_preserving_feedforward/double_pendulum.jl
@@ -54,7 +54,7 @@ function setup_and_train(
         model::Union{GeometricMachineLearning.Architecture, GeometricMachineLearning.Chain},
         batch::Batch; transformer::Bool = true)
     nn₀ = NeuralNetwork(model, backend, T)
-    o₀ = NetworkOptimizer(nn₀; algorithm = opt_method)
+    o₀ = TrainingOptimizer(nn₀.params; algorithm = opt_method)
 
     loss_array = transformer ?
                  train!(nn₀, o₀, dl, batch, n_epochs; loss = transformer_loss) :
diff --git a/scripts/reproduction/volume_preserving_feedforward/rigid_body.jl b/scripts/reproduction/volume_preserving_feedforward/rigid_body.jl
index 54a385cde..fd0fd2a7a 100644
--- a/scripts/reproduction/volume_preserving_feedforward/rigid_body.jl
+++ b/scripts/reproduction/volume_preserving_feedforward/rigid_body.jl
@@ -48,7 +48,7 @@ model = VolumePreservingFeedForward(sys_dim, n_blocks, n_linear, activation)
 
 nn = NeuralNetwork(model, backend, T)
 
-o = NetworkOptimizer(nn; algorithm = opt_method)
+o = TrainingOptimizer(nn.params; algorithm = opt_method)
 
 batch = Batch(batch_size)
 
diff --git a/scripts/reproduction/volume_preserving_transformer/rigid_body.jl b/scripts/reproduction/volume_preserving_transformer/rigid_body.jl
index 30e317500..a68f78a9d 100644
--- a/scripts/reproduction/volume_preserving_transformer/rigid_body.jl
+++ b/scripts/reproduction/volume_preserving_transformer/rigid_body.jl
@@ -62,7 +62,7 @@ const t_validation_long = 100
 function train_the_network(nn₀::GeometricMachineLearning.NeuralNetwork, batch::Batch)
     Random.seed!(1234)
 
-    o₀ = NetworkOptimizer(nn₀; opt_pairing...)
+    o₀ = TrainingOptimizer(nn₀.params; opt_pairing...)
 
     loss_array = train!(nn₀, o₀, dl, batch, n_epochs)
 
diff --git a/scripts/verification/all_zero_leaf.jl b/scripts/verification/all_zero_leaf.jl
index 1cb47e2be..7713901bf 100644
--- a/scripts/verification/all_zero_leaf.jl
+++ b/scripts/verification/all_zero_leaf.jl
@@ -1,9 +1,10 @@
 # What a training step does to a layer the loss does not read.
 #
 # NeuralNetworkParameters gives the gradient of such a layer as a zero leaf of the layer's own type,
-# not `nothing`, and `NetworkOptimizer` hands the whole gradient to
-# `GeometricOptimizers.optimization_step!`: GeometricMachineLearning skips no leaf. This script prints
-# how far each leaf moves, in both precisions and under each first-order method:
+# not `nothing`, and `train!` hands the whole gradient to `GeometricOptimizers.optimization_step!`:
+# GeometricMachineLearning skips no leaf. This script prints how far each leaf moves, in both
+# precisions and under each first-order method, and asserts the following; it stops with an
+# `AssertionError` at the first claim that does not hold:
 #
 #   - a Euclidean and a Stiefel leaf the loss never reads stay `==` their start under
 #     `GradientMethod`, `MomentumMethod` and `Adam`; under `AdamWithEuclideanDecay` the Euclidean one
@@ -35,7 +36,7 @@ function zero_leaf_run(T, method)
         read_then_not = (W = randn(T, 3, 3),),
         empty = NamedTuple()))
     p₀ = deepcopy(ps)
-    opt = NetworkOptimizer(ps; algorithm = method)
+    opt = TrainingOptimizer(ps; algorithm = method)
     x = randn(T, 3)
     for t in 1:N_STEPS
         loss = t ≤ N_READ ?
@@ -53,16 +54,16 @@ function after_reading(T, method)
     Random.seed!(1)
     ps = NetworkParameters((
         read = (W = randn(T, 3, 3),), read_then_not = (W = randn(T, 3, 3),)))
-    opt = NetworkOptimizer(ps; algorithm = method)
+    opt = TrainingOptimizer(ps; algorithm = method)
     x = randn(T, 3)
     both = p -> sum(abs2, p.read.W * x) + sum(abs2, p.read_then_not.W * x)
-    one = p -> sum(abs2, p.read.W * x)
+    read_only = p -> sum(abs2, p.read.W * x)
     for _ in 1:N_READ
         optimization_step!(ps, opt, only(Zygote.gradient(both, ps)))
     end
     W₅ = copy(ps.read_then_not.W)
     for _ in (N_READ + 1):N_STEPS
-        optimization_step!(ps, opt, only(Zygote.gradient(one, ps)))
+        optimization_step!(ps, opt, only(Zygote.gradient(read_only, ps)))
     end
     maxdiff(ps.read_then_not.W, W₅)
 end
@@ -70,9 +71,12 @@ end
 for T in (Float32, Float64)
     for method in (GradientMethod(), MomentumMethod(), Adam(), AdamWithEuclideanDecay())
         r = zero_leaf_run(T, method)
+        moved = after_reading(T, method)
         @printf("%-8s %-26s never read: Euclidean %.2e  Stiefel %.2e | read %d steps, then not: %.2e\n",
-            T, nameof(typeof(method)), r.never_euclidean,
-            r.never_stiefel, N_READ,
-            after_reading(T, method))
+            T, nameof(typeof(method)), r.never_euclidean, r.never_stiefel, N_READ, moved)
+        decays = method isa AdamWithEuclideanDecay
+        @assert decays ? r.never_euclidean > 0 : r.never_euclidean == 0
+        @assert r.never_stiefel == 0
+        @assert method isa GradientMethod ? moved == 0 : moved > 0
     end
 end
diff --git a/src/GeometricMachineLearning.jl b/src/GeometricMachineLearning.jl
index 2595faeef..74d017935 100644
--- a/src/GeometricMachineLearning.jl
+++ b/src/GeometricMachineLearning.jl
@@ -37,7 +37,7 @@ import Symbolics
 # `layers/` and the kernels), and extending a *type* reached through `using` warns on every such
 # method since Julia 1.12 — "Constructor for type … was extended in `GeometricMachineLearning`
 # without explicit qualification or import". Everything imported here is re-exported below, except
-# `orthonormal_columns`, `TrainingOptimizer` and `optimization_step!`; the re-exports are what keep
+# `orthonormal_columns` and `optimization_step!`; the re-exports are what keep
 # `using GeometricMachineLearning` alone sufficient.
 import GeometricOptimizers
 import GeometricOptimizers: Manifold, StiefelManifold, GrassmannManifold
@@ -55,8 +55,8 @@ import GeometricOptimizers: GlobalSection, global_rep, apply_section, apply_sect
 import GeometricOptimizers: Geodesic, Cayley, geodesic, cayley, retraction
 import GeometricOptimizers: GradientMethod, MomentumMethod, Adam, AdamWithEuclideanDecay,
                             AdamOptimizerWithDecay, DecayingStatic
-# The training step. `TrainingOptimizer` and `optimization_step!` are not re-exported:
-# `NetworkOptimizer` wraps the one and `train!` calls the other.
+# The training step. `train!` takes a `TrainingOptimizer` and calls `optimization_step!`, which is
+# not re-exported.
 import GeometricOptimizers: TrainingOptimizer, optimization_step!
 import GeometricOptimizers: update!
 # `solve!` is imported rather than started afresh so that GML's `solve!(::NeuralNetwork{<:PSDArch},
@@ -197,17 +197,6 @@ export TransformerIntegrator, StandardTransformerIntegrator
 # INCLUDE OPTIMIZERS — the methods, sections and retractions come from GeometricOptimizers.
 include("optimizers/optimizer.jl")
 
-export NetworkOptimizer
-export GlobalSection, global_section, apply_section, apply_section!, update_section!
-export global_rep
-export Geodesic, Cayley
-export geodesic, cayley
-export retraction
-export update!
-# Re-exported from GeometricOptimizers, which owns the one definition of them.
-export GradientMethod, MomentumMethod, Adam, AdamWithEuclideanDecay
-export AdamOptimizerWithDecay, DecayingStatic
-
 export NeuralNetwork
 
 export NetworkLoss, TransformerLoss, FeedForwardLoss, AutoEncoderLoss, ReducedLoss, HNNLoss,
diff --git a/src/data_loader/train.jl b/src/data_loader/train.jl
index 229703846..037c32922 100644
--- a/src/data_loader/train.jl
+++ b/src/data_loader/train.jl
@@ -1,34 +1,36 @@
 """
-    train!(nn, opt, dl, batch, n_epochs; loss, show_progress = true)
+    train!(nn, opt, dl, batch, n_epochs; loss, pullback = ZygotePullback(loss), show_progress = true)
 
-Train the neural network `nn` with the [`NetworkOptimizer`](@ref) `opt` on the data of the
-[`DataLoader`](@ref) `dl` for `n_epochs` epochs, and return the loss history.
+Train the neural network `nn` with the `GeometricOptimizers.TrainingOptimizer` `opt` on the data of
+the [`DataLoader`](@ref) `dl` for `n_epochs` epochs, and return the loss history.
 
 Each epoch draws the minibatches of `dl` with the [`Batch`](@ref) `batch` and takes one step per
-minibatch: Zygote computes the loss on the minibatch and its gradient, and
+minibatch: `pullback` computes the loss on the minibatch and its gradient, and
 `GeometricOptimizers.optimization_step!` writes the new parameters into `params(nn)`, with the
-whole gradient, a zero leaf included. Entry `i` of
-the history is the mean of the minibatch losses of epoch `i`, each evaluated before its step. The
-history has the element type of the loss value, so a `Float64` network on `Float32` data gives a
-`Vector{Float64}`.
+whole gradient, a zero leaf included. Entry `i` of the history is the mean of the minibatch losses
+of epoch `i`, each evaluated before its step. The history has the element type of the loss value,
+so a `Float64` network on `Float32` data gives a `Vector{Float64}`.
 
 `loss` is a `NetworkLoss`. Without it, a [`TransformerIntegrator`](@ref) with a
 `Batch{:Transformer}` trains on [`TransformerLoss`](@ref)`(batch)`, a
 [`NeuralNetworkIntegrator`](@ref) with a `Batch{:FeedForward}` on `FeedForwardLoss()`, and an
 autoencoder with a `Batch{:FeedForward}` on [`AutoEncoderLoss`](@ref)`()`; any other network needs
-`loss`.
+`loss`, and `train!` raises an `ArgumentError` without it.
 
-`opt` must be built for `params(nn)`. It keeps the iteration number, the moments and the global
-sections, so a second call with the same `opt` resumes the training where the first one stopped;
-the only random numbers `train!` draws are those of the minibatch shuffle. `n_epochs < 1` is an
-`ArgumentError`, raised before any step.
+`pullback` is an `AbstractPullback` of `loss`, by default a [`ZygotePullback`](@ref). A Hamiltonian network trains on its symbolic
+gradient with `loss = HNNLoss(arch)` and `pullback = SymbolicPullback(arch)`.
+
+`opt` is built for `params(nn)`, with `TrainingOptimizer(params(nn); algorithm, linesearch,
+retraction)`. It keeps the iteration number, the moments and the global sections, so a second call
+with the same `opt` resumes the training where the first one stopped; the only random numbers
+`train!` draws are those of the minibatch shuffle. `n_epochs < 1` is an `ArgumentError`, raised
+before any step.
 """
-function train!(nn::NeuralNetwork, opt::NetworkOptimizer, dl::DataLoader, batch::Batch,
+function train!(nn::NeuralNetwork, opt::TrainingOptimizer, dl::DataLoader, batch::Batch,
         n_epochs::Integer; loss::NetworkLoss = _default_loss(nn.architecture, batch),
-        show_progress::Bool = true)
+        pullback::AbstractPullback = ZygotePullback(loss), show_progress::Bool = true)
     n_epochs ≥ 1 ||
         throw(ArgumentError("`train!` needs at least one epoch; `n_epochs` is $(n_epochs)."))
-    pullback = ZygotePullback(loss)
     progress = ProgressMeter.Progress(n_epochs; enabled = show_progress)
     # A loop and not `map(1:n_epochs) do`: the closure around `_train_epoch!` makes the first call
     # of a transformer compile several times as long.
@@ -58,3 +60,7 @@ end
 _default_loss(::TransformerIntegrator, batch::Batch{:Transformer}) = TransformerLoss(batch)
 _default_loss(::NeuralNetworkIntegrator, ::Batch{:FeedForward}) = FeedForwardLoss()
 _default_loss(arch::AutoEncoder, ::Batch{:FeedForward}) = NetworkLoss(arch)
+function _default_loss(arch, batch)
+    throw(ArgumentError("`train!` has no default loss for a $(nameof(typeof(arch))) with a " *
+                        "$(typeof(batch)); pass the loss as the keyword `loss`."))
+end
diff --git a/src/optimizers/optimizer.jl b/src/optimizers/optimizer.jl
index 41fa035f0..3891f7c6f 100644
--- a/src/optimizers/optimizer.jl
+++ b/src/optimizers/optimizer.jl
@@ -1,35 +1,12 @@
-"""
-    NetworkOptimizer(nn; algorithm = Adam(), linesearch = default_step_size(algorithm), retraction = Cayley())
-    NetworkOptimizer(ps; algorithm, linesearch, retraction)
-
-The optimizer that [`train!`](@ref) trains the neural network `nn`, or the parameters `ps`, with.
-
-It holds one `GeometricOptimizers.TrainingOptimizer` over the whole parameter set, and the keywords
-are that constructor's: `algorithm` is a first-order method such as `GradientMethod()`,
-`MomentumMethod()` or `Adam()`, `linesearch` is a fixed step size or a `DecayingStatic` schedule,
-and `retraction` is `Cayley()` or `Geodesic()`. `AdamOptimizerWithDecay(n)` gives `algorithm` and
-`linesearch` together, and its `n` counts optimization steps, one per minibatch:
-
-```julia
-opt = NetworkOptimizer(nn; AdamOptimizerWithDecay(n_steps)...)
-```
-
-The optimizer carries the iteration number, the moments and the global sections from one call of
-[`train!`](@ref) to the next, so a second call resumes the training. It is built for the parameters
-of one network, and it is not checked when it is used with another.
-
-A loop of one's own takes a step with `GeometricOptimizers.optimization_step!(params(nn), opt, dp)`,
-where `dp` is the gradient of the loss as a `NetworkParameters` of the shape of `params(nn)`.
-"""
-struct NetworkOptimizer{OT <: TrainingOptimizer}
-    optimizer::OT
-end
-
-function NetworkOptimizer(ps::NetworkParameters; kwargs...)
-    NetworkOptimizer(TrainingOptimizer(ps; kwargs...))
-end
-NetworkOptimizer(nn::NeuralNetwork; kwargs...) = NetworkOptimizer(params(nn); kwargs...)
-
-function optimization_step!(ps, opt::NetworkOptimizer, dp)
-    optimization_step!(ps, opt.optimizer, dp)
-end
+# The optimizers are GeometricOptimizers': `train!` takes its `TrainingOptimizer`, built on the
+# parameters of a network with `TrainingOptimizer(params(nn); algorithm, linesearch, retraction)`.
+# They are re-exported here, so that `using GeometricMachineLearning` alone trains a network.
+export TrainingOptimizer
+export GradientMethod, MomentumMethod, Adam, AdamWithEuclideanDecay
+export AdamOptimizerWithDecay, DecayingStatic
+export GlobalSection, global_section, apply_section, apply_section!, update_section!
+export global_rep
+export Geodesic, Cayley
+export geodesic, cayley
+export retraction
+export update!
diff --git a/test/architectures/lagrangian_neural_network_tests.jl b/test/architectures/lagrangian_neural_network_tests.jl
index 93961e504..660578e35 100644
--- a/test/architectures/lagrangian_neural_network_tests.jl
+++ b/test/architectures/lagrangian_neural_network_tests.jl
@@ -154,7 +154,7 @@ function test_lnn_training(; n = 1, n_epochs = 30)
     # Start the training from a different draw, so the target is not already met.
     nn = NeuralNetwork(arch)
     dl = DataLoader(input, output; suppress_info = true)
-    optimizer = NetworkOptimizer(nn; algorithm = Adam())
+    optimizer = TrainingOptimizer(nn.params; algorithm = Adam())
     loss_array = train!(nn, optimizer, dl, Batch(10), n_epochs; loss, show_progress = false)
 
     @test loss_array[end] < loss_array[1]
diff --git a/test/architectures/sae_error_lower_than_psd_error.jl b/test/architectures/sae_error_lower_than_psd_error.jl
index b695e56de..ade5f360e 100644
--- a/test/architectures/sae_error_lower_than_psd_error.jl
+++ b/test/architectures/sae_error_lower_than_psd_error.jl
@@ -17,7 +17,7 @@ function test_accuracy(N::Integer, n::Integer; n_epochs::Integer = 100)
 
     sae_nn = NeuralNetwork(SymplecticAutoencoder(N, n; n_encoder_layers = 5, n_decoder_layers = 5))
 
-    o = NetworkOptimizer(sae_nn; algorithm = Adam())
+    o = TrainingOptimizer(sae_nn.params; algorithm = Adam())
     sae_error = train!(sae_nn, o, dl, Batch(10), n_epochs; show_progress = false)[end]
 
     @test sae_error < psd_error
diff --git a/test/architectures/symplectic_autoencoder_tests.jl b/test/architectures/symplectic_autoencoder_tests.jl
index 226c9dd60..2e0c606a3 100644
--- a/test/architectures/symplectic_autoencoder_tests.jl
+++ b/test/architectures/symplectic_autoencoder_tests.jl
@@ -10,7 +10,7 @@ function test_accuracy(N::Integer, n::Integer; tol::Real = 0.35, n_epochs::Integ
 
     sae_nn = NeuralNetwork(SymplecticAutoencoder(N, n))
 
-    o = NetworkOptimizer(sae_nn; algorithm = Adam())
+    o = TrainingOptimizer(sae_nn.params; algorithm = Adam())
     sae_error = train!(sae_nn, o, dl, Batch(10), n_epochs; show_progress = false)[end]
 
     @test sae_error < tol
@@ -39,7 +39,7 @@ function test_symplecticity(N::Integer, n::Integer)
 
     # test if it's still symplectic after training
     dl = DataLoader(rand(N, 10 * N); autoencoder = true)
-    o = NetworkOptimizer(sae_nn; algorithm = Adam())
+    o = TrainingOptimizer(sae_nn.params; algorithm = Adam())
     train!(sae_nn, o, dl, Batch(10), 10; show_progress = false)
     sympl_mat = jacobian(vec -> sae_decoder(vec), test_vector)[1]
     @test PoissonTensor(n) ≈ sympl_mat' * PoissonTensor(N) * sympl_mat
diff --git a/test/data_loader/batch_data_loader_qp_test.jl b/test/data_loader/batch_data_loader_qp_test.jl
index cf444a4a2..83fa6436e 100644
--- a/test/data_loader/batch_data_loader_qp_test.jl
+++ b/test/data_loader/batch_data_loader_qp_test.jl
@@ -40,8 +40,8 @@ function test_data_loader(dim = 2, number_of_time_steps = 100,
     nn2 = NeuralNetwork(arch2, CPU(), T)
 
     batch = Batch(batch_size)
-    o₁ = NetworkOptimizer(nn1; algorithm = GradientMethod())
-    o₂ = NetworkOptimizer(nn2; algorithm = GradientMethod())
+    o₁ = TrainingOptimizer(nn1.params; algorithm = GradientMethod())
+    o₂ = TrainingOptimizer(nn2.params; algorithm = GradientMethod())
 
     params₁ = deepcopy(nn1.params)
     params₂ = deepcopy(nn2.params)
diff --git a/test/data_loader/data_loader_for_input_and_output.jl b/test/data_loader/data_loader_for_input_and_output.jl
index 34ea3a9e8..6385449c4 100644
--- a/test/data_loader/data_loader_for_input_and_output.jl
+++ b/test/data_loader/data_loader_for_input_and_output.jl
@@ -16,7 +16,7 @@ nepochs = 2000
 model = Chain(Dense(1, nwidth), Dense(nwidth, 1))
 nn = NeuralNetwork(model, Float32)
 dl = DataLoader(xsamples, ysamples)
-o = NetworkOptimizer(nn; algorithm = Adam())
+o = TrainingOptimizer(nn.params; algorithm = Adam())
 batch = Batch(nbatch, 1, 1)
 
 loss = FeedForwardLoss()
diff --git a/test/data_loader/data_loader_optimization_step.jl b/test/data_loader/data_loader_optimization_step.jl
index 7996f3a2f..65857ce1e 100644
--- a/test/data_loader/data_loader_optimization_step.jl
+++ b/test/data_loader/data_loader_optimization_step.jl
@@ -17,7 +17,7 @@ function test_data_loader(sys_dim, n_time_steps, n_params, T = Float32)
     loss = GeometricMachineLearning.TransformerLoss(n_time_steps)
     dx = Zygote.gradient(ps -> loss(model, ps, dl.input, dl.input), ps)[1]
     ps_copy = deepcopy(ps)
-    o = NetworkOptimizer(ps; algorithm = GradientMethod())
+    o = TrainingOptimizer(ps; algorithm = GradientMethod())
     optimization_step!(ps, o, dx)
     @test ps != ps_copy
 end
diff --git a/test/data_loader/optimizer_functor_with_adam.jl b/test/data_loader/optimizer_functor_with_adam.jl
index 5c4f55bdb..1dd06cbe1 100644
--- a/test/data_loader/optimizer_functor_with_adam.jl
+++ b/test/data_loader/optimizer_functor_with_adam.jl
@@ -49,7 +49,7 @@ function test_optimization_with_adam(; T = Float32, input_dim = 9, n_patches = 4
 
     loss₁ = loss(model, ps, dl.input, dl.output)
 
-    opt = NetworkOptimizer(nn_obj; algorithm = Adam())
+    opt = TrainingOptimizer(nn_obj.params; algorithm = Adam())
     loss_average = only(train!(nn_obj, opt, dl, batch, 1; loss, show_progress = false))
 
     loss₃ = loss(model, ps, dl.input, dl.output)
diff --git a/test/data_loader/train.jl b/test/data_loader/train.jl
index 5c102bc32..540bc7aef 100644
--- a/test/data_loader/train.jl
+++ b/test/data_loader/train.jl
@@ -13,7 +13,7 @@ include("../helpers/eltypes.jl")
     x = randn(T, 2, 40)
     dl = DataLoader(x, A * x; suppress_info = true)
     nn = NeuralNetwork(Chain(Dense(2, 2, identity)), T)
-    opt = NetworkOptimizer(nn; algorithm = Adam(), linesearch = T(0.01))
+    opt = TrainingOptimizer(nn.params; algorithm = Adam(), linesearch = T(0.01))
     history = train!(
         nn, opt, dl, Batch(8), 200; loss = FeedForwardLoss(), show_progress = false)
     @test eltype(history) === T
@@ -30,7 +30,7 @@ end
     function trained(epochs...)
         Random.seed!(3)
         nn = NeuralNetwork(Chain(StiefelLayer(6, 3), Dense(3, 2, tanh)), T)
-        opt = NetworkOptimizer(nn; algorithm = Adam())
+        opt = TrainingOptimizer(nn.params; algorithm = Adam())
         for n in epochs
             train!(
                 nn, opt, dl, Batch(10), n; loss = FeedForwardLoss(), show_progress = false)
@@ -41,7 +41,7 @@ end
     nn₂, opt₂ = trained(10)
     @test params(nn₁).L1.weight isa StiefelManifold{T}
     @test params(nn₁) == params(nn₂)
-    @test opt₁.optimizer.state.iterations == opt₂.optimizer.state.iterations == 30
+    @test opt₁.state.iterations == opt₂.state.iterations == 30
 end
 
 # The history has the type of the loss value, and nothing is converted: the network's element type
@@ -54,7 +54,7 @@ end
         dl = DataLoader(x, x; suppress_info = true)
         nn = NeuralNetwork(Chain(Dense(2, 2, identity)), Tnn)
         history = train!(
-            nn, NetworkOptimizer(nn), dl, Batch(5), 2; loss = FeedForwardLoss(),
+            nn, TrainingOptimizer(nn.params), dl, Batch(5), 2; loss = FeedForwardLoss(),
             show_progress = false)
         @test eltype(history) === Thistory
     end
@@ -69,7 +69,7 @@ end
     batch = Batch(4)
     loss = FeedForwardLoss()
     nn = NeuralNetwork(Chain(Dense(2, 2, tanh)), T)
-    opt = NetworkOptimizer(nn; algorithm = GradientMethod(), linesearch = floatmin(T))
+    opt = TrainingOptimizer(nn.params; algorithm = GradientMethod(), linesearch = floatmin(T))
     p₀ = deepcopy(params(nn))
     Random.seed!(8)
     losses = map(batch(dl)) do indices
@@ -86,7 +86,43 @@ end
     @test history[1] ≈ sum(losses) / 3
 end
 
-# Without `loss`, `train!` trains on the loss that matches the architecture and the batch.
+# `train!` steps with the gradient of the `pullback` it is given. One epoch of one minibatch with
+# `GradientMethod` moves the parameters by `-η` times the symbolic gradient at `p₀`, and the history
+# is the symbolic pullback's loss value.
+@testset "train! trains a Hamiltonian network with SymbolicPullback ($T)" for T in REAL_ELTYPES
+    Random.seed!(9)
+    arch = StandardHamiltonianArchitecture(2, 4, 1, GeometricMachineLearning.SigmoidActivation())
+    dl = DataLoader(randn(T, 2, 8), randn(T, 2, 8); suppress_info = true)
+    nn = NeuralNetwork(arch, CPU(), T)
+    p₀ = deepcopy(params(nn))
+    loss = HNNLoss(arch)
+    pullback = SymbolicPullback(arch)
+    batch = Batch(8)
+    Random.seed!(2)
+    data = GeometricMachineLearning.convert_input_and_batch_indices_to_array(
+        dl, batch, only(batch(dl)))
+    value, back = pullback(p₀, nn.model, data)
+    g = back(one(value))
+    η = T(0.1)
+    opt = TrainingOptimizer(params(nn); algorithm = GradientMethod(), linesearch = η)
+    Random.seed!(2)
+    history = train!(nn, opt, dl, batch, 1; loss, pullback, show_progress = false)
+    @test history == [value]
+    @test eltype(history) === T
+    for (layer, layer₀, gradient) in zip(values(params(nn)), values(p₀), values(g))
+        for k in keys(layer)
+            @test layer[k] ≈ layer₀[k] .- η .* gradient[k]
+            @test layer[k] != layer₀[k]
+        end
+    end
+    zygote_value, zygote_back = ZygotePullback(loss)(p₀, nn.model, data)
+    zygote_g = GeometricMachineLearning._get_contents(zygote_back(one(zygote_value)))
+    @test zygote_value ≈ value
+    @test_broken zygote_g.L1.W ≈ g.L1.W  # issue #343
+end
+
+# Without `loss`, `train!` trains on the loss that matches the architecture and the batch, and on
+# any other network it says that it needs one.
 @testset "train!'s default losses" begin
     transformer_batch = Batch(4, 3)
     @test GeometricMachineLearning._default_loss(GSympNet(4; n_layers = 2), Batch(4)) isa
@@ -96,19 +132,29 @@ end
         transformer_batch) == TransformerLoss(transformer_batch)
     autoencoder = SymplecticAutoencoder(4, 2; n_encoder_blocks = 2, n_decoder_blocks = 2)
     @test GeometricMachineLearning._default_loss(autoencoder, Batch(4)) isa AutoEncoderLoss
+
+    nn = NeuralNetwork(Chain(Dense(2, 2, identity)))
+    dl = DataLoader(rand(2, 10), rand(2, 10); suppress_info = true)
+    raised = try
+        train!(nn, TrainingOptimizer(params(nn)), dl, Batch(5), 1; show_progress = false)
+    catch e
+        e
+    end
+    @test raised isa ArgumentError
+    @test occursin("`loss`", raised.msg)
 end
 
 @testset "train! refuses n_epochs < 1 before any step" begin
     Random.seed!(5)
     nn = NeuralNetwork(Chain(Dense(2, 2, identity)))
     dl = DataLoader(rand(2, 10), rand(2, 10); suppress_info = true)
-    opt = NetworkOptimizer(nn)
+    opt = TrainingOptimizer(nn.params)
     p₀ = deepcopy(params(nn))
     for n in (0, -1)
         @test_throws ArgumentError train!(
             nn, opt, dl, Batch(5), n; loss = FeedForwardLoss(),
             show_progress = false)
     end
-    @test opt.optimizer.state.iterations == 0
+    @test opt.state.iterations == 0
     @test params(nn) == p₀
 end
diff --git a/test/data_loader/training_history_eltype.jl b/test/data_loader/training_history_eltype.jl
index 14e5611e5..0ec62571e 100644
--- a/test/data_loader/training_history_eltype.jl
+++ b/test/data_loader/training_history_eltype.jl
@@ -12,7 +12,7 @@ Random.seed!(11)
 function test_training_history_eltype(T::DataType)
     nn = NeuralNetwork(GSympNet(4), CPU(), T)
     dl = DataLoader(rand(T, 4, 64); autoencoder = false, suppress_info = true)
-    o = NetworkOptimizer(nn; algorithm = GradientMethod())
+    o = TrainingOptimizer(nn.params; algorithm = GradientMethod())
 
     history = train!(
         nn, o, dl, Batch(8), 2; loss = FeedForwardLoss(), show_progress = false)
diff --git a/test/integration/losses_and_optimization.jl b/test/integration/losses_and_optimization.jl
index 7b41ef407..57ae92d5c 100644
--- a/test/integration/losses_and_optimization.jl
+++ b/test/integration/losses_and_optimization.jl
@@ -17,7 +17,7 @@ function train_network(; n_epochs = 10)
     nn = setup_network(dl)
     loss = FeedForwardLoss()
 
-    o = NetworkOptimizer(nn; algorithm = Adam())
+    o = TrainingOptimizer(nn.params; algorithm = Adam())
     batch = Batch(5, 1)
     loss_array = train!(nn, o, dl, batch, n_epochs; loss, show_progress = false)
     T = eltype(dl)
diff --git a/test/integration/reduced_loss_optimization.jl b/test/integration/reduced_loss_optimization.jl
index db40720f7..d994518fe 100644
--- a/test/integration/reduced_loss_optimization.jl
+++ b/test/integration/reduced_loss_optimization.jl
@@ -37,7 +37,7 @@ Random.seed!(123)
         StandardTransformerIntegrator(reduced_dim; transformer_dim = 10, n_blocks = 1,
             n_heads = 2, L = 1, upscaling_activation = tanh), CPU(), Float64)
 
-    o = NetworkOptimizer(integrator; algorithm = Adam())
+    o = TrainingOptimizer(integrator.params; algorithm = Adam())
     batch = Batch(8, 4)
 
     # The parameters reach the loss through `Zygote.pullback` here, which is what an annotated
diff --git a/test/integration/train_manual_architectures.jl b/test/integration/train_manual_architectures.jl
index 9955720d4..63649db55 100644
--- a/test/integration/train_manual_architectures.jl
+++ b/test/integration/train_manual_architectures.jl
@@ -56,7 +56,7 @@ end
         @testset "$name" begin
             nn = NeuralNetwork(arch, T)
             p₀ = deepcopy(params(nn))
-            opt = NetworkOptimizer(nn; algorithm = Adam())
+            opt = TrainingOptimizer(nn.params; algorithm = Adam())
             history = loss === nothing ?
                       train!(nn, opt, dl, batch, 2; show_progress = false) :
                       train!(nn, opt, dl, batch, 2; loss, show_progress = false)
diff --git a/test/layers/gradient_layer_tests.jl b/test/layers/gradient_layer_tests.jl
index 1117dcb6e..df8a600ba 100644
--- a/test/layers/gradient_layer_tests.jl
+++ b/test/layers/gradient_layer_tests.jl
@@ -25,7 +25,7 @@ end
 function test_gradient_layer_derivative_and_update(T, M, N, batch_size = 10)
     dummy_model = Chain(GradientLayerP(M, N, tanh), GradientLayerQ(M, N, tanh))
     ps = params(NeuralNetwork(dummy_model, CPU(), T))
-    o = NetworkOptimizer(ps; algorithm = Adam(; β₁ = 0.9, β₂ = 0.999, δ = 3e-7),
+    o = TrainingOptimizer(ps; algorithm = Adam(; β₁ = 0.9, β₂ = 0.999, δ = 3e-7),
         linesearch = T(0.1))
 
     # test for vector
diff --git a/test/layers/manifold_layers.jl b/test/layers/manifold_layers.jl
index 15f12a9cb..a1427f46e 100644
--- a/test/layers/manifold_layers.jl
+++ b/test/layers/manifold_layers.jl
@@ -7,7 +7,7 @@ Random.seed!(1234)
 function stiefel_layer_test(T, M, N, tol = 1.0f-1)
     model = Chain(StiefelLayer(M, N), StiefelLayer(N, N))
     ps = NeuralNetwork(model, T).params
-    o = NetworkOptimizer(ps; algorithm = Adam(; β₁ = 5.0f-1, β₂ = 5.0f-1, δ = 3.0f-7),
+    o = TrainingOptimizer(ps; algorithm = Adam(; β₁ = 5.0f-1, β₂ = 5.0f-1, δ = 3.0f-7),
         linesearch = T(1.0f0))
 
     dx = NetworkParameters((L1 = (weight = rand(T, N, M),), L2 = (weight = rand(T, N, N),)))
@@ -23,7 +23,7 @@ end
 function grassmann_layer_test(T, M, N, tol = 1.0f-1)
     model = Chain(GrassmannLayer(M, N), StiefelLayer(N, N))
     ps = NeuralNetwork(model, T).params
-    o = NetworkOptimizer(ps; algorithm = Adam(; β₁ = 5.0f-1, β₂ = 5.0f-1, δ = 3.0f-7),
+    o = TrainingOptimizer(ps; algorithm = Adam(; β₁ = 5.0f-1, β₂ = 5.0f-1, δ = 3.0f-7),
         linesearch = T(1.0f0))
 
     dx = NetworkParameters((L1 = (weight = rand(T, N, M),), L2 = (weight = rand(T, N, N),)))
diff --git a/test/layers/multi_head_attention_stiefel_optim_cache.jl b/test/layers/multi_head_attention_stiefel_optim_cache.jl
index 65813d443..2e1879091 100644
--- a/test/layers/multi_head_attention_stiefel_optim_cache.jl
+++ b/test/layers/multi_head_attention_stiefel_optim_cache.jl
@@ -65,13 +65,13 @@ function test_cache_setups_for_optimizer_for_multihead_attention_layer(T::Type,
     model = Chain(MultiHeadAttention(dim, n_heads, Stiefel = true))
     ps = NeuralNetwork(model, CPU(), T).params
 
-    o₁ = NetworkOptimizer(ps; algorithm = Adam())
-    o₂ = NetworkOptimizer(ps; algorithm = MomentumMethod())
-    o₃ = NetworkOptimizer(ps; algorithm = GradientMethod())
+    o₁ = TrainingOptimizer(ps; algorithm = Adam())
+    o₂ = TrainingOptimizer(ps; algorithm = MomentumMethod())
+    o₃ = TrainingOptimizer(ps; algorithm = GradientMethod())
 
-    check_adam_cache(o₁.optimizer.cache)
-    check_momentum_cache(o₂.optimizer.cache)
-    check_gradient_cache(o₃.optimizer.cache)
+    check_adam_cache(o₁.cache)
+    check_momentum_cache(o₂.cache)
+    check_gradient_cache(o₃.cache)
 end
 
 test_cache_setups_for_optimizer_for_multihead_attention_layer(Float32, 64, 8)
diff --git a/test/layers/multi_head_attention_stiefel_retraction.jl b/test/layers/multi_head_attention_stiefel_retraction.jl
index 211a75209..8988e0cb2 100644
--- a/test/layers/multi_head_attention_stiefel_retraction.jl
+++ b/test/layers/multi_head_attention_stiefel_retraction.jl
@@ -52,7 +52,7 @@ function test_multi_head_attention_retraction(
     model = Chain(MultiHeadAttention(dim, n_heads, Stiefel = true))
 
     ps = NeuralNetwork(model, backend, T).params
-    cache = NetworkOptimizer(ps; algorithm = MomentumMethod()).optimizer.cache
+    cache = TrainingOptimizer(ps; algorithm = MomentumMethod()).cache
     @test cache isa MomentumCache
 
     check_retraction_geodesic(cache)
diff --git a/test/layers/multi_head_attention_stiefel_setup.jl b/test/layers/multi_head_attention_stiefel_setup.jl
index 790bf6083..74787d7c6 100644
--- a/test/layers/multi_head_attention_stiefel_setup.jl
+++ b/test/layers/multi_head_attention_stiefel_setup.jl
@@ -49,7 +49,7 @@ function check_multi_head_attention_stiefel_setup(T::Type, N::Int, n::Int)
     @test nleaves > 0
     @test check_setup_calls[] == nleaves
 
-    gx = NetworkOptimizer(ps; algorithm = MomentumMethod()).optimizer.cache
+    gx = TrainingOptimizer(ps; algorithm = MomentumMethod()).cache
     @test gx isa MomentumCache
     check_grad_setup(gx)
 end
diff --git a/test/layers/transformer_optimizer.jl b/test/layers/transformer_optimizer.jl
index f813ffe18..1ff0a612b 100644
--- a/test/layers/transformer_optimizer.jl
+++ b/test/layers/transformer_optimizer.jl
@@ -23,9 +23,9 @@ function transformer_gradient_test(T, dim, n_heads, L, seq_length = 8, batch_siz
     ps₂ = deepcopy(ps)
     ps₃ = deepcopy(ps)
 
-    o₁ = NetworkOptimizer(ps₁; algorithm = GradientMethod())
-    o₂ = NetworkOptimizer(ps₂; algorithm = MomentumMethod())
-    o₃ = NetworkOptimizer(ps₃; algorithm = Adam())
+    o₁ = TrainingOptimizer(ps₁; algorithm = GradientMethod())
+    o₂ = TrainingOptimizer(ps₂; algorithm = MomentumMethod())
+    o₃ = TrainingOptimizer(ps₃; algorithm = Adam())
 
     optimization_step!(ps₁, o₁, dx)
     optimization_step!(ps₂, o₂, dx)
diff --git a/test/loss/training_method_losses.jl b/test/loss/training_method_losses.jl
index 15ed74285..933f49d7f 100644
--- a/test/loss/training_method_losses.jl
+++ b/test/loss/training_method_losses.jl
@@ -171,7 +171,7 @@ function test_symplectic_euler_training(; n = 1, Δt = 1.0e-2, n_epochs = 20)
     nn = NeuralNetwork(arch)
     dl = DataLoader(input, output; suppress_info = true)
     loss_array = train!(
-        nn, NetworkOptimizer(nn; algorithm = Adam()), dl, Batch(10), n_epochs;
+        nn, TrainingOptimizer(nn.params; algorithm = Adam()), dl, Batch(10), n_epochs;
         loss, show_progress = false)
     @test loss_array[end] < loss_array[1]
 end
@@ -341,7 +341,7 @@ function test_variational_training(; Δt = 0.05, samples = 60, n_epochs = 20)
 
     dl = DataLoader(vcat(qₙ, qₙ₊₁), qₙ₊₂; suppress_info = true)
     loss_array = train!(
-        nn, NetworkOptimizer(nn; algorithm = Adam()), dl, Batch(10), n_epochs;
+        nn, TrainingOptimizer(nn.params; algorithm = Adam()), dl, Batch(10), n_epochs;
         loss, show_progress = false)
     @test loss_array[end] < loss_array[1]
 end
diff --git a/test/optimizers/adam_with_learning_rate_decay.jl b/test/optimizers/adam_with_learning_rate_decay.jl
index fa146c69c..eac788f52 100644
--- a/test/optimizers/adam_with_learning_rate_decay.jl
+++ b/test/optimizers/adam_with_learning_rate_decay.jl
@@ -25,11 +25,11 @@ function train_network(; n_epochs = 2048)
     nn₁ = setup_network(dl)
     nn₂ = setup_network(dl)
 
-    o₁ = NetworkOptimizer(nn₁; algorithm = Adam())
+    o₁ = TrainingOptimizer(nn₁.params; algorithm = Adam())
     # `AdamOptimizerWithDecay` is GeometricOptimizers', and returns the `(algorithm, linesearch)`
     # pairing the keywords take, so it splats rather than being passed positionally. Its argument
     # counts optimization steps, and the optimizer converts it to the element type of the network.
-    o₂ = NetworkOptimizer(nn₂; AdamOptimizerWithDecay(n_epochs)...)
+    o₂ = TrainingOptimizer(nn₂.params; AdamOptimizerWithDecay(n_epochs)...)
 
     batch = Batch(5, 1)
     loss = FeedForwardLoss()
@@ -56,7 +56,7 @@ function train_manifold_network(; n_epochs = 128)
     arch = Chain(StiefelLayer(1, 20), Dense(20, 20, tanh), Dense(20, 1, identity))
     nn = NeuralNetwork(arch, CPU(), eltype(dl))
 
-    o = NetworkOptimizer(nn; AdamOptimizerWithDecay(n_epochs)...)
+    o = TrainingOptimizer(nn.params; AdamOptimizerWithDecay(n_epochs)...)
     loss_array = train!(nn, o, dl, Batch(5, 1), n_epochs; loss = FeedForwardLoss(),
         show_progress = false)
 
@@ -81,11 +81,11 @@ function schedule_starts_at_one(T; n_epochs = 100, η₁ = 1e-2, η₂ = 1e-6)
     # One step of a Euclidean parameter with a gradient of `1` moves it by `α₁ / (√1 + δ) ≈ α₁`, so
     # the distance travelled reports the α used.
     ps = NetworkParameters((weight = zeros(T, 2, 2),))
-    opt = NetworkOptimizer(ps; AdamOptimizerWithDecay(n_epochs; η₁, η₂)...)
-    @test opt.optimizer.linesearch isa DecayingStatic
-    @test opt.optimizer.state.iterations == 0
+    opt = TrainingOptimizer(ps; AdamOptimizerWithDecay(n_epochs; η₁, η₂)...)
+    @test opt.linesearch isa DecayingStatic
+    @test opt.state.iterations == 0
     optimization_step!(ps, opt, NetworkParameters((weight = ones(T, 2, 2),)))
-    @test opt.optimizer.state.iterations == 1
+    @test opt.state.iterations == 1
     @test eltype(ps.weight) === T
     @test abs(ps.weight[1, 1]) ≈ η₁ * γ rtol = 1e-6
 end
diff --git a/test/optimizers/gml_gradient_dispatch.jl b/test/optimizers/gml_gradient_dispatch.jl
deleted file mode 100644
index 1862793ca..000000000
--- a/test/optimizers/gml_gradient_dispatch.jl
+++ /dev/null
@@ -1,55 +0,0 @@
-# What the training step sees of a gradient.
-#
-# `NetworkOptimizer` steps the whole parameter set with one `GeometricOptimizers` cache, so the
-# gradient the cache builds its direction from is the gradient of the whole set: a parameter set of
-# the network's shape, with the Riemannian gradient on a manifold leaf and the Euclidean gradient on
-# a plain one.
-
-using GeometricMachineLearning
-using GeometricOptimizers: StiefelManifold, GrassmannManifold, rgrad, PrecomputedGradient
-using NeuralNetworkParameters: NetworkParameters
-using Test
-import Random
-
-include("../helpers/eltypes.jl")
-
-Random.seed!(1234)
-
-GML = GeometricMachineLearning
-
-@testset "the gradient of a whole network, a manifold leaf beside a plain one ($T)" for T in REAL_ELTYPES
-    for M in (StiefelManifold, GrassmannManifold)
-        ps = NetworkParameters((L1 = (weight = rand(M{T}, 4, 2),),
-            L2 = (W = randn(T, 3, 3), b = randn(T, 3))))
-        dp = NetworkParameters((L1 = (weight = randn(T, 4, 2),),
-            L2 = (W = randn(T, 3, 3), b = randn(T, 3))))
-        out = PrecomputedGradient(ps, dp)(ps)
-
-        @test out isa NetworkParameters
-        @test keys(out) == keys(ps)
-        @test out.L1.weight == rgrad(ps.L1.weight, dp.L1.weight)
-        @test out.L2.W == dp.L2.W
-        @test out.L2.b == dp.L2.b
-        @test eltype(out.L1.weight) === eltype(out.L2.W) === eltype(out.L2.b) === T
-    end
-end
-
-# `_get_contents` unwraps whatever shape the pullback handed back. Six methods, because a reverse pass
-# produces a container or the bare `NamedTuple` it wraps, and `Zygote` may hand either back bare, in a
-# one-tuple, or in a one-element vector. Each is exercised here: splitting a union into a method per
-# shape means each method needs its own case, or the split trades one tested path for six untested
-# ones.
-@testset "_get_contents unwraps every shape a pullback returns" begin
-    nt = (L1 = (W = randn(2, 2), b = randn(2)),)
-    ps = NetworkParameters(nt)
-
-    for x in (nt, ps)
-        @test GML._get_contents(x) === x
-        @test GML._get_contents((x,)) === x
-        @test GML._get_contents([x]) === x
-    end
-
-    # more than one set is a caller error and says so, rather than silently taking the first
-    @test_throws ArgumentError GML._get_contents([ps, ps])
-    @test_throws ArgumentError GML._get_contents([nt, nt])
-end
diff --git a/test/optimizers/gradient_optimizer.jl b/test/optimizers/gradient_optimizer.jl
index 5837d6f02..b147c4034 100644
--- a/test/optimizers/gradient_optimizer.jl
+++ b/test/optimizers/gradient_optimizer.jl
@@ -16,7 +16,7 @@ function gradient_optimizer_euclidean(N; n_steps = 20, step_size = 1e-3)
     A = randn(N, N)
     ps = NetworkParameters((A = A,))
     loss1 = loss(ps)
-    o = NetworkOptimizer(ps; algorithm = GradientMethod(), linesearch = step_size)
+    o = TrainingOptimizer(ps; algorithm = GradientMethod(), linesearch = step_size)
     for _ in 1:n_steps
         ∇L = Zygote.gradient(loss, ps)[1]
         optimization_step!(ps, o, ∇L)
@@ -35,7 +35,7 @@ function gradient_optimizer_stiefel(N, n; n_steps = 20, step_size = 1e-3)
     Y = rand(StiefelManifold, N, n)
     ps = NetworkParameters((Y = Y,))
     loss1 = loss(ps)
-    o = NetworkOptimizer(ps; algorithm = GradientMethod(), linesearch = step_size)
+    o = TrainingOptimizer(ps; algorithm = GradientMethod(), linesearch = step_size)
     for _ in 1:n_steps
         ∇L = Zygote.gradient(loss, ps)[1]
         optimization_step!(ps, o, ∇L)
diff --git a/test/optimizers/momentum_optimizer.jl b/test/optimizers/momentum_optimizer.jl
index 86ba0c249..4b056b030 100644
--- a/test/optimizers/momentum_optimizer.jl
+++ b/test/optimizers/momentum_optimizer.jl
@@ -17,7 +17,7 @@ growing one that never saturates, which is what this test rules out.
 """
 function momentum_matches_geometric_sum(; α = 0.9, η = 0.1, n_steps = 10)
     ps = NetworkParameters((A = zeros(2, 2),))
-    o = NetworkOptimizer(ps; algorithm = MomentumMethod(; α), linesearch = η)
+    o = TrainingOptimizer(ps; algorithm = MomentumMethod(; α), linesearch = η)
     previous = 0.0
     for t in 1:n_steps
         optimization_step!(ps, o, NetworkParameters((A = ones(2, 2),)))
@@ -38,7 +38,7 @@ function momentum_optimizer_stiefel(N, n; n_steps = 50, step_size = 1e-3, α = 0
     loss(ps) = norm(ps.Y * ps.Y' - B)^2
     ps = NetworkParameters((Y = rand(StiefelManifold, N, n),))
     loss1 = loss(ps)
-    o = NetworkOptimizer(ps; algorithm = MomentumMethod(; α), linesearch = step_size)
+    o = TrainingOptimizer(ps; algorithm = MomentumMethod(; α), linesearch = step_size)
     for _ in 1:n_steps
         optimization_step!(ps, o, Zygote.gradient(loss, ps)[1])
     end
diff --git a/test/optimizers/optimization_step.jl b/test/optimizers/optimization_step.jl
index e9b452e75..825b1e265 100644
--- a/test/optimizers/optimization_step.jl
+++ b/test/optimizers/optimization_step.jl
@@ -14,7 +14,7 @@ function optimization_step_test(N, n, T)
     # gradient
     dx = NetworkParameters((L1 = (weight = rand(T, N, n),),
         L2 = (W = rand(T, N, N), b = rand(T, N))))
-    o = NetworkOptimizer(ps; algorithm = Adam())
+    o = TrainingOptimizer(ps; algorithm = Adam())
 
     ps2 = deepcopy(ps)
     optimization_step!(ps, o, dx)
@@ -40,12 +40,12 @@ end
 @testset "one cache and one state over the whole network ($T)" for T in REAL_ELTYPES
     model = Chain(StiefelLayer(6, 3), Dense(6, 6, tanh))
     nn = NeuralNetwork(model, KernelAbstractions.CPU(), T)
-    o = NetworkOptimizer(nn; algorithm = Adam())
+    o = TrainingOptimizer(nn.params; algorithm = Adam())
 
-    @test o.optimizer.cache isa GeometricOptimizers.OptimizerCache{T}
-    @test o.optimizer.state isa GeometricOptimizers.OptimizerState{T}
-    for x in (GeometricOptimizers.solution(o.optimizer.cache),
-        GeometricOptimizers.solution(o.optimizer.state))
+    @test o.cache isa GeometricOptimizers.OptimizerCache{T}
+    @test o.state isa GeometricOptimizers.OptimizerState{T}
+    for x in (GeometricOptimizers.solution(o.cache),
+        GeometricOptimizers.solution(o.state))
         @test x isa NetworkParameters
         @test keys(x) == keys(GeometricMachineLearning.params(nn))
         @test x.L1.weight isa StiefelManifold{T}
@@ -60,26 +60,22 @@ function allocations(f::F, a::A, b::B, c::C) where {F, A, B, C}
 end
 
 # The training step is the hot path of `train!`. It is inferred, and on a Euclidean layer it allocates
-# nothing. With a Stiefel leaf `GeometricOptimizers`' own step allocates (about 2 to 4 kB here), and
-# the step through `NetworkOptimizer` allocates the same bytes, within 256: Windows reads up to 48
-# bytes apart. One copy of the 64 × 64 gradient leaf is 16 kB in `Float32`, 64 times the tolerance.
-@testset "the step through NetworkOptimizer is inferred and allocates nothing of its own ($T)" for T in REAL_ELTYPES
+# nothing. With a Stiefel leaf it is inferred and keeps the leaf on the manifold.
+@testset "the training step is inferred and allocates nothing on a Euclidean layer ($T)" for T in REAL_ELTYPES
     for algorithm in (GradientMethod(), MomentumMethod(), Adam())
         ps = NetworkParameters((L1 = (W = rand(T, 3, 3), b = rand(T, 3)),))
         dp = NetworkParameters((L1 = (W = rand(T, 3, 3), b = rand(T, 3)),))
-        o = NetworkOptimizer(ps; algorithm)
+        o = TrainingOptimizer(ps; algorithm)
         @inferred optimization_step!(ps, o, dp)
         @test allocations(optimization_step!, ps, o, dp) == 0
         @test eltype(ps.L1.W) === eltype(ps.L1.b) === T
 
-        ps = NetworkParameters((L1 = (W = rand(T, 64, 64), b = rand(T, 64)),
+        ps = NetworkParameters((L1 = (W = rand(T, 3, 3), b = rand(T, 3)),
             L2 = (Y = rand(StiefelManifold{T}, 4, 2),)))
-        dp = NetworkParameters((L1 = (W = rand(T, 64, 64), b = rand(T, 64)),
+        dp = NetworkParameters((L1 = (W = rand(T, 3, 3), b = rand(T, 3)),
             L2 = (Y = rand(T, 4, 2),)))
-        o = NetworkOptimizer(ps; algorithm)
+        o = TrainingOptimizer(ps; algorithm)
         @inferred optimization_step!(ps, o, dp)
-        @test abs(allocations(optimization_step!, ps, o, dp) -
-                  allocations(optimization_step!, ps, o.optimizer, dp)) ≤ 256
         @test ps.L2.Y isa StiefelManifold{T}
     end
     @test allocations((a, b, c) -> collect(a), 1:3, 1, 1) > 0
diff --git a/test/optimizers/optimizer.jl b/test/optimizers/optimizer.jl
index 08dc00fb3..d95b17ace 100644
--- a/test/optimizers/optimizer.jl
+++ b/test/optimizers/optimizer.jl
@@ -10,7 +10,7 @@ include("../helpers/eltypes.jl")
 # `algorithm` and the `linesearch` the constructor was given. The reference gradient is taken on the
 # minibatch `train!` draws, drawn again from the same seed. The bound is 4 eps(T), for the rounding of
 # one step.
-@testset "NetworkOptimizer takes algorithm and linesearch ($T)" for T in REAL_ELTYPES
+@testset "train! steps with the TrainingOptimizer's algorithm and linesearch ($T)" for T in REAL_ELTYPES
     Random.seed!(1)
     x, y = randn(T, 2, 8), randn(T, 2, 8)
     dl = DataLoader(x, y; suppress_info = true)
@@ -26,7 +26,7 @@ include("../helpers/eltypes.jl")
     input, output = convert_input_and_batch_indices_to_array(dl, batch, only(minibatches))
     g = only(Zygote.gradient(ps -> FeedForwardLoss()(model, ps, input, output), p₀))
 
-    opt = NetworkOptimizer(nn; algorithm = GradientMethod(), linesearch = η)
+    opt = TrainingOptimizer(nn.params; algorithm = GradientMethod(), linesearch = η)
     Random.seed!(2)
     train!(nn, opt, dl, batch, 1; loss = FeedForwardLoss(), show_progress = false)
     for k in (:W, :b)
@@ -34,18 +34,11 @@ include("../helpers/eltypes.jl")
         @test params(nn).L1[k] ≈ p₀.L1[k] .- η .* g.L1[k] rtol = 4eps(T)
         @test params(nn).L1[k] != p₀.L1[k]
     end
-
-    # the same run with the optimizer built from the parameter set
-    nn₂ = NeuralNetwork(model, model, deepcopy(p₀), CPU())
-    opt₂ = NetworkOptimizer(params(nn₂); algorithm = GradientMethod(), linesearch = η)
-    Random.seed!(2)
-    train!(nn₂, opt₂, dl, batch, 1; loss = FeedForwardLoss(), show_progress = false)
-    @test params(nn₂) == params(nn)
 end
 
 @testset "AdamOptimizerWithDecay splats into the keywords" begin
     nn = NeuralNetwork(Chain(Dense(2, 2, identity)), Float32)
-    opt = NetworkOptimizer(nn; AdamOptimizerWithDecay(100)...)
-    @test opt.optimizer.method isa Adam{Float32}
-    @test opt.optimizer.linesearch isa DecayingStatic{Float32}
+    opt = TrainingOptimizer(nn.params; AdamOptimizerWithDecay(100)...)
+    @test opt.method isa Adam{Float32}
+    @test opt.linesearch isa DecayingStatic{Float32}
 end
diff --git a/test/optimizers/psd_optim.jl b/test/optimizers/psd_optim.jl
index df16904e2..1092b774a 100644
--- a/test/optimizers/psd_optim.jl
+++ b/test/optimizers/psd_optim.jl
@@ -46,9 +46,9 @@ function svd_test(A, n, train_steps = 1000, tol = 1e-1; retraction = Cayley())
     model = Chain(PSDLayer(2*N, 2*n), PSDLayer(2*n, 2*N))
     ps = NeuralNetwork(model, CPU(), Float64).params
 
-    o₁ = NetworkOptimizer(ps; algorithm = GradientMethod(), retraction, linesearch = 0.01)
-    o₂ = NetworkOptimizer(ps; algorithm = MomentumMethod(), retraction, linesearch = 0.01)
-    o₃ = NetworkOptimizer(ps; algorithm = Adam(), retraction, linesearch = 0.01)
+    o₁ = TrainingOptimizer(ps; algorithm = GradientMethod(), retraction, linesearch = 0.01)
+    o₂ = TrainingOptimizer(ps; algorithm = MomentumMethod(), retraction, linesearch = 0.01)
+    o₃ = TrainingOptimizer(ps; algorithm = Adam(), retraction, linesearch = 0.01)
 
     U₁, Ũ₁, err₁ = train_network!(o₁, model, deepcopy(ps), A, train_steps, tol)
     U₂, _, err₂ = train_network!(o₂, model, deepcopy(ps), A, train_steps, tol)
@@ -64,7 +64,7 @@ function svd_test(A, n, train_steps = 1000, tol = 1e-1; retraction = Cayley())
     @test norm((err₃ - err_best)/err_best) < tol
 end
 
-function train_network!(o::NetworkOptimizer, model::Chain, ps::NetworkParameters,
+function train_network!(o::TrainingOptimizer, model::Chain, ps::NetworkParameters,
         A::AbstractMatrix, train_steps, tol)
     error(ps) = norm(A - model(A, ps))
 
diff --git a/test/optimizers/step_size_element_type.jl b/test/optimizers/step_size_element_type.jl
deleted file mode 100644
index cb7e848f4..000000000
--- a/test/optimizers/step_size_element_type.jl
+++ /dev/null
@@ -1,33 +0,0 @@
-# The step scales in the parameters' own element type. `NetworkOptimizer` hands `linesearch` to
-# `GeometricOptimizers.TrainingOptimizer`, which converts it once to the element type of the
-# parameters, so a `Float64` step size on a `Float32` network gives the same step as the same value
-# rounded to `Float32` first.
-#
-# Only `T = Float32` can tell the two apart -- for `Float64` the rounding is a no-op -- so `Float64`
-# is not looped over here.
-
-using GeometricMachineLearning
-using GeometricOptimizers: optimization_step!
-using Test
-import Random
-
-function test_step_matches_prerounded_step_size(method, seed::Int, step_size::Float64)
-    T = Float32
-    Random.seed!(seed)
-    ps_a = NetworkParameters((L1 = (weight = rand(T, 6, 6),),))
-    dp = NetworkParameters((L1 = (weight = rand(T, 6, 6),),))
-    ps_b = deepcopy(ps_a)
-
-    opt_a = NetworkOptimizer(ps_a; algorithm = method, linesearch = step_size)
-    opt_b = NetworkOptimizer(ps_b; algorithm = method, linesearch = T(step_size))
-    @test typeof(opt_a.optimizer.linesearch) == typeof(opt_b.optimizer.linesearch)
-
-    optimization_step!(ps_a, opt_a, dp)
-    optimization_step!(ps_b, opt_b, dp)
-
-    @test ps_a.L1.weight isa Matrix{T}
-    @test ps_a.L1.weight == ps_b.L1.weight
-end
-
-test_step_matches_prerounded_step_size(Adam(), 2, 0.1)
-test_step_matches_prerounded_step_size(MomentumMethod(; α = 0.5), 2, 0.1)
diff --git a/test/optimizers/structured_array_parameters.jl b/test/optimizers/structured_array_parameters.jl
index 56e004145..c763d135c 100644
--- a/test/optimizers/structured_array_parameters.jl
+++ b/test/optimizers/structured_array_parameters.jl
@@ -22,7 +22,7 @@ function optimizer_runs(architecture, batch, input_dim; T = Float64, n_epochs =
 
     for method in (Adam(), MomentumMethod(), GradientMethod())
         nn, dl = nn_and_dl()
-        o = NetworkOptimizer(nn; algorithm = method)
+        o = TrainingOptimizer(nn.params; algorithm = method)
         loss_array = train!(nn, o, dl, batch, n_epochs; show_progress = false)
         @test length(loss_array) == n_epochs
         @test all(isfinite, loss_array)
@@ -31,7 +31,7 @@ function optimizer_runs(architecture, batch, input_dim; T = Float64, n_epochs =
     # `AdamOptimizerWithDecay` is a `(algorithm, linesearch)` pairing rather than a method, so it
     # splats into the keywords
     nn, dl = nn_and_dl()
-    o = NetworkOptimizer(nn; AdamOptimizerWithDecay(n_epochs)...)
+    o = TrainingOptimizer(nn.params; AdamOptimizerWithDecay(n_epochs)...)
     loss_array = train!(nn, o, dl, batch, n_epochs; show_progress = false)
     @test length(loss_array) == n_epochs
     @test all(isfinite, loss_array)
diff --git a/test/optimizers/svd_optim.jl b/test/optimizers/svd_optim.jl
index 42544f7eb..68035173a 100644
--- a/test/optimizers/svd_optim.jl
+++ b/test/optimizers/svd_optim.jl
@@ -42,9 +42,9 @@ function svd_test(A, n, train_steps = 1000, tol = 1e-1; retraction = Cayley())
     model = Chain(StiefelLayer(N, n), StiefelLayer(n, N))
     ps = NeuralNetwork(model, CPU(), Float64).params
 
-    o₁ = NetworkOptimizer(ps; algorithm = GradientMethod(), retraction, linesearch = 0.01)
-    o₂ = NetworkOptimizer(ps; algorithm = MomentumMethod(), retraction, linesearch = 0.01)
-    o₃ = NetworkOptimizer(ps; algorithm = Adam(), retraction, linesearch = 0.01)
+    o₁ = TrainingOptimizer(ps; algorithm = GradientMethod(), retraction, linesearch = 0.01)
+    o₂ = TrainingOptimizer(ps; algorithm = MomentumMethod(), retraction, linesearch = 0.01)
+    o₃ = TrainingOptimizer(ps; algorithm = Adam(), retraction, linesearch = 0.01)
 
     U₁, Ũ₁, err₁ = train_network!(o₁, model, deepcopy(ps), A, train_steps, tol)
     U₂, _, err₂ = train_network!(o₂, model, deepcopy(ps), A, train_steps, tol)
@@ -60,7 +60,7 @@ function svd_test(A, n, train_steps = 1000, tol = 1e-1; retraction = Cayley())
     @test norm((err₃ - err_best)/err_best) < tol
 end
 
-function train_network!(o::NetworkOptimizer, model::Chain, ps::NetworkParameters,
+function train_network!(o::TrainingOptimizer, model::Chain, ps::NetworkParameters,
         A::AbstractMatrix, train_steps, tol)
     error(ps) = norm(A - model(A, ps))
 
diff --git a/test/pullbacks/get_contents.jl b/test/pullbacks/get_contents.jl
new file mode 100644
index 000000000..450897097
--- /dev/null
+++ b/test/pullbacks/get_contents.jl
@@ -0,0 +1,28 @@
+using GeometricMachineLearning
+using NeuralNetworkParameters: NetworkParameters
+using Test
+import Random
+
+Random.seed!(1234)
+
+GML = GeometricMachineLearning
+
+# `_get_contents` unwraps whatever shape the pullback handed back. Six methods, because a reverse pass
+# produces a container or the bare `NamedTuple` it wraps, and `Zygote` may hand either back bare, in a
+# one-tuple, or in a one-element vector. Each is exercised here: splitting a union into a method per
+# shape means each method needs its own case, or the split trades one tested path for six untested
+# ones.
+@testset "_get_contents unwraps every shape a pullback returns" begin
+    nt = (L1 = (W = randn(2, 2), b = randn(2)),)
+    ps = NetworkParameters(nt)
+
+    for x in (nt, ps)
+        @test GML._get_contents(x) === x
+        @test GML._get_contents((x,)) === x
+        @test GML._get_contents([x]) === x
+    end
+
+    # more than one set is a caller error and says so, rather than silently taking the first
+    @test_throws ArgumentError GML._get_contents([ps, ps])
+    @test_throws ArgumentError GML._get_contents([nt, nt])
+end
diff --git a/test/runtests.jl b/test/runtests.jl
index ad5816b25..5829cb968 100644
--- a/test/runtests.jl
+++ b/test/runtests.jl
@@ -59,8 +59,7 @@ if "core" in GROUPS
     @safetestset "Gradient optimizer tests" include("optimizers/gradient_optimizer.jl")
     @safetestset "Momentum optimizer tests" include("optimizers/momentum_optimizer.jl")
     @safetestset "Optimizers with structured (non-manifold) weights" include("optimizers/structured_array_parameters.jl")
-    @safetestset "The gradient a training step sees" include("optimizers/gml_gradient_dispatch.jl")
-    @safetestset "The training step scales in the parameters' own element type" include("optimizers/step_size_element_type.jl")
+    @safetestset "_get_contents unwraps every shape a pullback returns" include("pullbacks/get_contents.jl")
     @safetestset "_custom_mul: NetworkParameters gradient structure" include("layers/double_multiplication_network_parameters_gradient.jl")
     @safetestset "Symplectic attention: NetworkParameters gradient structure" include("layers/symplectic_attention_network_parameters_gradient.jl")
     @safetestset "map_to_cpu" include("map_to_cpu.jl")
@@ -75,7 +74,7 @@ if "core" in GROUPS
     @safetestset "An unexpected `autoencoder` keyword throws rather than returning nothing" include("data_loader/autoencoder_keyword.jl")
     @safetestset "A Float32 network's training history stays Float32" include("data_loader/training_history_eltype.jl")
     @safetestset "train!: convergence, resume, history type, n_epochs" include("data_loader/train.jl")
-    @safetestset "NetworkOptimizer" include("optimizers/optimizer.jl")
+    @safetestset "train! takes a TrainingOptimizer's algorithm and linesearch" include("optimizers/optimizer.jl")
     @safetestset "The minibatch index set is concrete and grows linearly" include("data_loader/batch_index_set.jl")
     @safetestset "Classifier accuracy" include("data_loader/accuracy.jl")
     @safetestset "Data loader docstring examples" include("integration/docstrings/data_loader.jl")

From adac904b4515dba1df6aedee90b4448cd33b310e Mon Sep 17 00:00:00 2001
From: Michael Kraus <michael.kraus@ipp.mpg.de>
Date: Mon, 12 Oct 2026 23:35:12 +0200
Subject: [PATCH 10/11] Join the split AbstractArray signatures and say
 strictly triangular in the docstrings

Review finding 8 of #342: the formatter had split AbstractArray{T, 3} over two lines in the four StrictlyLower/UpperTriangular methods, and their docstrings said lower- and upper-triangular.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 src/kernels/kernel_ad_routines/mat_tensor_mul.jl |  6 ++----
 src/kernels/mat_tensor_mul.jl                    | 14 ++++++--------
 2 files changed, 8 insertions(+), 12 deletions(-)

diff --git a/src/kernels/kernel_ad_routines/mat_tensor_mul.jl b/src/kernels/kernel_ad_routines/mat_tensor_mul.jl
index 24c31a323..34742f6a8 100644
--- a/src/kernels/kernel_ad_routines/mat_tensor_mul.jl
+++ b/src/kernels/kernel_ad_routines/mat_tensor_mul.jl
@@ -74,8 +74,7 @@ function ChainRulesCore.rrule(::typeof(lo_mat_mul), S::AbstractVector{T},
 end
 
 function ChainRulesCore.rrule(::typeof(mat_tensor_mul), B::StrictlyLowerTriangular{T},
-        A::AbstractArray{
-            T, 3}) where {T}
+        A::AbstractArray{T, 3}) where {T}
     @assert size(A, 1) == B.n
     C = mat_tensor_mul(B, A)
     function lower_triangular_mul_pullback(dC)
@@ -146,8 +145,7 @@ function ChainRulesCore.rrule(::typeof(up_mat_mul), S::AbstractVector{T},
 end
 
 function ChainRulesCore.rrule(::typeof(mat_tensor_mul), B::StrictlyUpperTriangular{T},
-        A::AbstractArray{
-            T, 3}) where {T}
+        A::AbstractArray{T, 3}) where {T}
     @assert size(A, 1) == B.n
     C = mat_tensor_mul(B, A)
     function upper_triangular_mul_pullback(dC)
diff --git a/src/kernels/mat_tensor_mul.jl b/src/kernels/mat_tensor_mul.jl
index 589729345..3e74ca9a3 100644
--- a/src/kernels/mat_tensor_mul.jl
+++ b/src/kernels/mat_tensor_mul.jl
@@ -154,13 +154,12 @@ end
 @doc raw"""
     mat_tensor_mul!(C, A::StrictlyLowerTriangular, B)
 
-Multiply the lower-triangular matrix `A` onto the tensor `B` from the left and store the result in `C`.
+Multiply the strictly lower-triangular matrix `A` onto the tensor `B` from the left and store the result in `C`.
 
-This performs an efficient multiplication based on the special structure of the lower-triangular matrix `A`.
+This performs an efficient multiplication based on the special structure of the strictly lower-triangular matrix `A`.
 """
 function mat_tensor_mul!(C::AbstractArray{<:Number, 3}, A::StrictlyLowerTriangular,
-        B::AbstractArray{
-            <:Number, 3})
+        B::AbstractArray{<:Number, 3})
     @assert eltype(C) == eltype(A) == eltype(B)
     @assert A.n == size(C, 1) == size(B, 1)
 
@@ -202,13 +201,12 @@ end
 @doc raw"""
     mat_tensor_mul!(C, A::StrictlyUpperTriangular, B)
 
-Multiply the upper-triangular matrix `A` onto the tensor `B` from the left and store the result in `C`.
+Multiply the strictly upper-triangular matrix `A` onto the tensor `B` from the left and store the result in `C`.
 
-This performs an efficient multiplication based on the special structure of the upper-triangular matrix `A`.
+This performs an efficient multiplication based on the special structure of the strictly upper-triangular matrix `A`.
 """
 function mat_tensor_mul!(C::AbstractArray{<:Number, 3}, A::StrictlyUpperTriangular,
-        B::AbstractArray{
-            <:Number, 3})
+        B::AbstractArray{<:Number, 3})
     @assert eltype(C) == eltype(A) == eltype(B)
     @assert A.n == size(C, 1) == size(B, 1)
 

From 20da228e3ab8856c58306104ded0bc8aa8347311 Mon Sep 17 00:00:00 2001
From: Michael Kraus <michael.kraus@ipp.mpg.de>
Date: Mon, 12 Oct 2026 23:36:16 +0200
Subject: [PATCH 11/11] Measure the GPU paragraph again against
 GeometricOptimizers 0.9.0

Review finding 10 of #342. Every claim of the GPU Support paragraph was run again on an Apple M4 Max with Metal.jl 1.11.1 and GeometricOptimizers 0.9.0: the eleven architectures construct, apply to a matrix and a 3-tensor, and copy back with map_to_cpu; the StrictlyLowerTriangular tensor product runs; and train! now trains a GSympNet for two epochs on the device. ClassificationTransformer was not tried, and the paragraph says so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---
 docs/src/index.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/docs/src/index.md b/docs/src/index.md
index 1a9f4caac..9c8b9bdd5 100644
--- a/docs/src/index.md
+++ b/docs/src/index.md
@@ -52,9 +52,9 @@ Many layers have been adapted in order to be used for problems in scientific mac
 
 `GeometricMachineLearning` allocates and computes through `KernelAbstractions.jl` [churavy2020kernel](@cite), so its layers and architectures are written against any backend that package supports: `CUDA.jl` [besard2018juliagpu](@cite), `AMDGPU.jl`, `Metal.jl` [besard2022metal](@cite) and `oneAPI.jl` [besard2022one](@cite).
 
-**Little of that is tested.** `test/devices/metal.jl` runs two checks on an Apple GPU, in the `metal` test group, which the `Metal` job and the `macOS-latest` job of CI run: the products of a `PoissonTensor` with a wrapped device array, and the `tensor_mat_mul!` kernel. A green test matrix says nothing about the rest of the GPU path. What is written below was measured by hand on an Apple M4 Max through `Metal.jl`, in `Float32`, because Apple GPUs have no `Float64` at all, and against `GeometricOptimizers` 0.8.0, which the package's `[compat]` bound of `0.9` excludes.
+**Little of that is tested.** `test/devices/metal.jl` runs two checks on an Apple GPU, in the `metal` test group, which the `Metal` job and the `macOS-latest` job of CI run: the products of a `PoissonTensor` with a wrapped device array, and the `tensor_mat_mul!` kernel. A green test matrix says nothing about the rest of the GPU path. What is written below was measured by hand on an Apple M4 Max through `Metal.jl` 1.11.1, in `Float32`, because Apple GPUs have no `Float64` at all, and against `GeometricOptimizers` 0.9.0.
 
-Everything tried on that device ran. Constructed, and then applied to both a matrix and a 3-tensor: `GSympNet`, `LASympNet`, `StandardTransformerIntegrator`, `LinearSymplecticTransformer`, `SymplecticTransformer` at both its default `transformer_dim` and an upscaling one, `VolumePreservingFeedForward`, `VolumePreservingTransformer`, `SymplecticAutoencoder`, `PSDArch` and `Transformer(…; Stiefel = true)`. `ClassificationTransformer` was constructed but not applied, because its input is an image. The tensor kernels and `map_to_cpu` run. Training through `train!` has not been run on the device.
+Everything tried on that device ran. Constructed, applied to both a matrix and a 3-tensor, and copied back with `map_to_cpu`: `GSympNet`, `LASympNet`, `StandardTransformerIntegrator`, `LinearSymplecticTransformer`, `SymplecticTransformer` at both its default `transformer_dim` and an upscaling one, `VolumePreservingFeedForward`, `VolumePreservingTransformer`, `SymplecticAutoencoder`, `PSDArch` and `Transformer(…; Stiefel = true)`. The product of a `StrictlyLowerTriangular` matrix with a 3-tensor runs. `train!` trains a `GSympNet` for two epochs with its data in `MtlArray`s. `ClassificationTransformer` was not tried.
 
 Two of those results rest on `GeometricOptimizers` rather than on anything here.