diff --git a/CHANGELOG.md b/CHANGELOG.md
index 5b2b4f730..90a320466 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -15,8 +15,32 @@ breaking release).
 
 ## [Unreleased]
 
+### Removed (breaking)
+
+- **`Optimizer`, the callable optimizers and their aliases are gone; training uses GeometricOptimizers' `TrainingOptimizer`.** `TrainingOptimizer`, re-exported by GeometricMachineLearning from GeometricOptimizers 0.9, is built on the parameters of a network: `TrainingOptimizer(nn.params; algorithm = Adam(), linesearch = default_step_size(algorithm), retraction = Cayley())`. Migrate `Optimizer(method, nn; step_size = η, retraction = cayley)` to `TrainingOptimizer(nn.params; algorithm = method, linesearch = η, retraction = Cayley())`, and `Optimizer(nn; AdamOptimizerWithDecay(n, T)...)` to `TrainingOptimizer(nn.params; AdamOptimizerWithDecay(n)...)`. The `n` of `AdamOptimizerWithDecay` counts optimization steps, one per minibatch. One cache and one state now cover the whole parameter set; 0.8 had one per layer. `Optimizer` is no longer exported by GeometricMachineLearning, so with both packages loaded, `Optimizer` is GeometricOptimizers' solver.
+
+- **`train!` replaces the four callable methods of `Optimizer`.** `train!(nn, opt, dl, batch, n_epochs; loss, pullback = ZygotePullback(loss), show_progress = true)` replaces `(o::Optimizer)(nn, dl, batch, n_epochs[, loss, pullback])` and `optimize_for_one_epoch!`, which is removed from the exports. Migrate `o(nn, dl, batch, n, loss; kw...)` to `train!(nn, o, dl, batch, n; loss, kw...)`. `loss` is a `NetworkLoss`, and `pullback` is any `AbstractPullback`: a Hamiltonian network trains with `loss = HNNLoss(arch)` and `pullback = SymbolicPullback(arch)`. The sixth positional `AbstractPullback` that 0.8 allowed is gone. Without `loss`, a `TransformerIntegrator` with a `Batch{:Transformer}` trains on `TransformerLoss(batch)`, a `NeuralNetworkIntegrator` with a `Batch{:FeedForward}` on `FeedForwardLoss()`, and an autoencoder with a `Batch{:FeedForward}` on `AutoEncoderLoss()`; any other combination raises an `ArgumentError` that asks for `loss`. The `λY` argument and the per-call `GlobalSection` are gone: `opt` holds the sections, moments and iteration number, so a second `train!` call with the same `opt` resumes the first, and `train!` draws no random numbers beyond the minibatch shuffle. `n_epochs < 1` raises an `ArgumentError` before any step. The loss history has the element type of the loss value, so a `Float64` network on `Float32` data gives a `Vector{Float64}`.
+
+- **A hand-written loop steps with `GeometricOptimizers.optimization_step!(nn.params, opt, dp)`.** The 0.8 method `optimization_step!(opt, λY, ps, dp)` is removed. GeometricMachineLearning imports `optimization_step!` but does not export it, so a caller qualifies it.
+
+- **A gradient leaf that is all zero is stepped, not skipped.** NeuralNetworkParameters 0.5 gives a layer the loss does not read a zero leaf of its own type, not `nothing`, and the step takes the whole gradient. A leaf the loss never reads stays where it is under `GradientMethod`, `MomentumMethod` and `Adam`, and a Stiefel leaf stays under all four methods; under `AdamWithEuclideanDecay` a Euclidean leaf decays. A leaf the loss reads for some steps and then not keeps moving on its momentum under `MomentumMethod`, `Adam` and `AdamWithEuclideanDecay`, and stays under `GradientMethod`. `scripts/verification/all_zero_leaf.jl` asserts them.
+
+- **This closes known issues C1, C2 and D4.** C1 and C2 close because GML's own parameter-tree traversal and `_leaf_optim_step!` are deleted; D4 closes because the per-layer caches it measured are deleted.
+
+- **The aliases `GradientOptimizer`, `MomentumOptimizer` and `AdamOptimizer` are removed**; use `GradientMethod`, `MomentumMethod` and `Adam`. These are no longer exported: `OptimizerMethod`, `GradientState`, `MomentumState`, `AdamState`, `optimization_step!`, `optimize_for_one_epoch!` and `Optimizer`. `TrainingOptimizer` (re-exported from GeometricOptimizers), `train!` and `AdamWithEuclideanDecay` are newly exported, and the package re-exports `GradientMethod`, `MomentumMethod`, `Adam`, `AdamOptimizerWithDecay`, `DecayingStatic`, `Cayley` and `Geodesic` from GeometricOptimizers. Methods carry no element type in GeometricOptimizers 0.9: `Adam(T)` becomes `Adam()`, and `MomentumMethod(α)` becomes `MomentumMethod(; α)`. `check(::Optimizer)` is removed.
+
+- **The three 3-tensor functors of `Dense` and `Linear` and the HDF5 `save` and `load` methods on `NeuralNetwork` are removed**, because AbstractNeuralNetworks 0.9 provides them. The HDF5 extension and the HDF5 weak dependency are gone; with HDF5 loaded, AbstractNeuralNetworks' extension provides `save` and `load`. Its `load` has no `backend` keyword: load on the CPU, then call `changebackend(backend, nn)`. Aqua's `piracies` check runs again with 0 piracies (was 3), and the ambiguity count is 0 (was 1). This closes issue #145 and known issue B7.
+
+### Changed (breaking)
+
+- **`LowerTriangular` and `UpperTriangular` are renamed `StrictlyLowerTriangular` and `StrictlyUpperTriangular`**, as in GeometricOptimizers 0.9, and are re-exported under the new names. A file saved with the old names no longer loads through the registry. The rigid-body tutorial registers the two old names for its committed files.
+
+- **The `[compat]` bounds move to the versions the training API needs**: GeometricOptimizers `"0.9"`, AbstractNeuralNetworks `"0.9.1"`, NeuralNetworkParameters `"0.5"`, SymbolicNeuralNetworks `"0.9"`, GeometricBase `"0.15"`, GeometricEquations `"0.21.5"` and GeometricSolutions `"0.6.6"`. HDF5 leaves `[compat]` and `[weakdeps]`.
+
 ### Added
 
+- **`KernelAbstractions.get_backend` is defined for `PoissonTensor`.** GeometricOptimizers 0.9 asks every operand of a checked product for its backend, so `𝕁 * Y` with a `StiefelManifold` raised `ArgumentError: Implement KernelAbstractions.get_backend(::PoissonTensor...)` without it.
+
 - **The test suite runs on an Apple GPU in its `metal` group.** `test/devices/metal.jl` checks the two device paths that were measured by hand and not tested before: the products of a `PoissonTensor` with a wrapped device array, `𝕁 * view(A, [1, 2, 3, 4], :)` and `𝕁 * B'`, which `ext/GPUArraysCoreExt.jl` carries; and the `tensor_mat_mul!` kernel on `MetalBackend()`. Each result must stay on the device and match the host in `Float32`, with scalar indexing off. `Pkg.test(test_args = ["metal"])` runs them without the subject tests, and a missing device records a skip, which happens inside a sandbox. Metal (1.10 or later) is a test dependency on every platform; it installs and precompiles on Linux and Windows, and nothing there loads it. On an Apple M4 Max with Metal 1.11.1 all five assertions pass.
 - **A `Metal` workflow runs the Metal tests on GitHub's `macos-15` runner**, for the `min` and `1` Julia versions. The runner's GPU is Apple's paravirtual device, which Metal.jl supports from 1.10 and on macOS 15 or later. The workflow includes a step "Metal is functional" before the tests that fails the job where the device is unavailable, so it cannot pass without having run the tests. The job is not a required check. However, the Metal tests also run in two required checks—CI.yml's `Julia min - macOS-latest - default` and `Julia 1 - macOS-latest - default`—which execute `test/runtests.jl` with no test arguments on Apple silicon, selecting the `metal` group.
 - **`test/quality/jet.jl` checks the kernel launchers with JET.** It runs in `core`, directly after `quality/aqua.jl`, and takes about 14 s. Each function that launches a kernel of `src/` on the `CPU()` backend, the pullbacks among them, and the `Batch` functor that `test/data_loader/batch_index_set.jl` measures with `@allocated` get one `JET.report_opt` line per element type that the tests use, with `target_modules = (GeometricMachineLearning,)`. Each `@kernel` method also gets a line that analyses its generated `cpu_<kernel>` function at the launcher's context, because JET drops the reports of a kernel body when it analyses the launcher. 142 lines pass on Julia 1.11.9 (JET 0.9.20) and 1.13.1 (JET 0.12.2). Three lines report a runtime dispatch on both versions and are `@test_broken` with issue #325: `PoissonTensor(::CPU, ::Int, ::DataType)`, and the input-and-output method of `convert_input_and_batch_indices_to_array` at `Float32` and `Float64`. Where JET does not work, the file records one `@test_skip`. JET joins `test/Project.toml` with no `[compat]` bound. `KNOWN_ISSUES.md` K2 records the Revise `EMFILE` messages that JET brings into the test log, and K3 the dispatches that the lines do not see: an unstable argument of a kernel launch, and two barriers in the Poisson-tensor kernel.
@@ -27,7 +51,7 @@ breaking release).
 - `Pkg.test()` no longer runs the doctests. `test/quality/doctests.jl` is now the group `doctests`, which an empty `ARGS` does not run; `Pkg.test(test_args = ["doctests"])` runs it. In CI the Doctests job stays their runner, so the test matrix no longer runs them a second time.
 - **CI uploads coverage from the `Julia 1 - ubuntu-latest` job instead of `Julia min`, and a test job saves the Julia cache only when it succeeds.**
 - **The package needs Julia 1.12 or later** (`[compat] julia = "1.12"`, was `"1.11"`). On Julia 1.11.9, the gradient of an `LNNLoss` with respect to the network parameters stops with `StackOverflowError` in type inference ("Internal error: during type inference of … Encountered stack overflow") on macOS and Windows; the `Julia min` job on Linux passes. The overflow comes from SymbolicUtils 4.49.0, whose `rrule` for `SymbolicUtils.Code.create_array` returns an exactly typed tuple of tangents, one per array element; Zygote's reverse pass through the generated Hessian of `lagrangian_acceleration` is then inferred far deeper. With the 4.48.0 rule restored, or in a task with a 64 MB stack, the same gradient passes on 1.11.9; Julia 1.12.7 and 1.13.1 pass it on the root task. This closes #327: the `Julia min` jobs on macOS and Windows had failed in the *Lagrangian Neural Network* testset since 2026-10-01, and every testset after it was never run there.
-- **`test/Project.toml` has no `[compat]` entry for a dependency of the package.** The entries for AbstractNeuralNetworks, GeometricOptimizers, HDF5, KernelAbstractions, LinearAlgebra, NeuralNetworkParameters, Random and Zygote are gone: the test environment contains the package, so the bounds of `Project.toml` apply there, and an entry in `test/Project.toml` can only repeat or narrow them. `NeuralNetworkParameters = "0.3"` was narrower than the package's `"0.3, 0.4"`.
+- **`test/Project.toml` has no `[compat]` entry for a dependency of the package.** The entries for AbstractNeuralNetworks, GeometricOptimizers, KernelAbstractions, LinearAlgebra, NeuralNetworkParameters, Random and Zygote are gone: the test environment contains the package, so the bounds of `Project.toml` apply there, and an entry in `test/Project.toml` can only repeat or narrow them. `NeuralNetworkParameters = "0.3"` was narrower than the package's `"0.3, 0.4"`.
 - **The test suite follows one flat layout.** `test/runtests.jl` is one list of `@safetestset "<label>" include("<path>")` lines in the groups `core`, `slow` and `metal`, selected by `ARGS`; with no arguments it runs `core` and `slow`, and `metal` too on an Apple-silicon Mac, so `Pkg.test()` and CI run every test they ran before. The 14 nested `test/<subject>/runtests.jl` drivers are gone. A test file sits where the source it tests sits: `test/attention/` and `test/transformers/` went into `test/layers/`, `test/architectures/` and `test/kernels/`; `test/reduced_order_modeling/` into `test/architectures/` and `test/reduced_system/`; `test/losses/training_method_losses.jl` into `test/loss/`; the tests that span the package, the docstring examples among them, into `test/integration/`; and `aqua.jl`, `exports.jl` and `reachability.jl` into `test/quality/`. `test/docstrings/layers_and_architectures.jl`, named in *Added* above, is now `test/integration/docstrings/layers_and_architectures.jl`. `test/layers/sympnet_upscaling.jl` is in `slow`. Thirteen files that drew random numbers without a seed now call `Random.seed!(1234)`. The shared helper of the two gradient-structure tests is `test/helpers/network_parameters_gradient_structure.jl`, and its `@test_broken` names issue #319.
 - **The doctests run in the test suite.** `test/quality/doctests.jl`, in the `slow` group, runs `doctest(GeometricMachineLearning)` as the `Doctests` job of `CI.yml` does. `Documenter` joins `test/Project.toml`, with no `[compat]` bound, as a test-only dependency.
 - **The test suite compiles less.** `test/layers/sympnet_upscaling.jl` checks its three symplectic identities on 12 pairs `(N, N2)` instead of all 120 with `N` in `2:2:20` and `N2` in `(2N):2:(4N)`: `(2, 4)`, `(2, 6)`, `(2, 8)`, `(4, 10)`, `(6, 12)`, `(6, 20)`, `(8, 24)`, `(10, 30)`, `(12, 36)`, `(14, 42)`, `(20, 40)` and `(20, 80)`. They keep the smallest pair, `N2 = 2N`, `N2 = 4N`, `N2` not a multiple of `N`, and the largest `N`. The layers carry the sizes in their types, so each pair is a new compilation (about 1.3 s per pair, not once per file as the [0.8.0] entry assumed); the file has 36 assertions instead of 360, and takes 19.7 s cold; it stays in `slow`. The two training tests of `test/loss/training_method_losses.jl` use the network shape of the rest of the file, `StandardHamiltonianArchitecture(2n, 4, 1)` and `LagrangianNeuralNetwork(2; width = 4, nhidden = 1)`, in place of `(2n, 5, 2)` and `width = 5, nhidden = 2`; each still asserts that training reduces the loss. `test/architectures/lagrangian_neural_network_tests.jl` moves from `core` to `slow`: it takes 122.7 s cold, above the 60 s of a `core` file. It still runs in CI and in `Pkg.test()`, and leaves only the local `core` run. Every other `core` file takes at most 37.9 s cold. The times are cold, from one fresh Julia process that runs the `core` files in the order of `test/runtests.jl` (and from one fresh process for each of the two files named), on an Apple M4 Max with Julia 1.13.1, at a load average below 4 at the start.
diff --git a/KNOWN_ISSUES.md b/KNOWN_ISSUES.md
index 7ebbc21f1..98d20a2de 100644
--- a/KNOWN_ISSUES.md
+++ b/KNOWN_ISSUES.md
@@ -20,8 +20,8 @@ What is known to be wrong in GeometricMachineLearning and is not fixed.
   N=5  symbolic=-0.29982310048056143  zygote=-0.05705562794369161  agree=false
   ```
 
-  `SymbolicPullback(arch)` is exported and documented as something one passes to `Optimizer` in
-  place of `ZygotePullback`, so this is wrong training, silently, for every batch size above one.
+  `SymbolicPullback(arch)` is exported as a pullback to use in place of `ZygotePullback`, so this
+  is wrong training, silently, for every batch size above one.
   It is not new — `reduce = +` is what SymbolicNeuralNetworks 0.3 did too — and it is the same
   defect SymbolicNeuralNetworks records for `FeedForwardLoss` under *Open Issues → Semantics* in its
   own changelog, where it is called out as unfixable within a design that differentiates one
@@ -34,48 +34,17 @@ What is known to be wrong in GeometricMachineLearning and is not fixed.
   (**B1**, **B2**, **B3**, **B4**, **B6** and **B8** are all closed and their entries are gone: B1 and B2 by
   this release — the duplicated `AdamOptimizerWithDecay` and the split `Manifold`, both under
   *Removed (breaking)* — B3 by SymbolicNeuralNetworks 0.5, B4 by `a427add1`, which repaired the
-  documentation build, and B6 and B8 by this release: B6 by the `train!` retirement (the three methods that
-  called the non-existent `vectorfield` are gone, and `SymplecticEulerLoss` carries their content on
-  `hamiltonian_vector_field`, with tests that run), and B8 by the narrowing of `PoissonTensor`'s `Base.:*`
-  to `Strided…` right-hand sides and `Base.getindex` to `Union{Int, Colon, AbstractVector{<:Integer}}`
-  indices, reducing method ambiguities from 18 to 1 as asserted in `test/quality/aqua.jl`. The numbers are
-  left vacant rather than reused.)
+  documentation build, and B6 and B8 by this release: B6 by the deletion of `src/training_method/`
+  (the three methods that called the non-existent `vectorfield` are gone, and `SymplecticEulerLoss`
+  carries their content on `hamiltonian_vector_field`, with tests that run), and B8 by the narrowing
+  of `PoissonTensor`'s `Base.:*` to `Strided…` right-hand sides and `Base.getindex` to
+  `Union{Int, Colon, AbstractVector{<:Integer}}` indices, reducing method ambiguities from 18 to 1.
+  The numbers are left vacant rather than reused.)
 
   (**B9** is closed by this release and its entry is gone: the `iterate` method it named binds
   every type parameter, and Aqua's `unbound_args` passes on Julia nightly. See *Fixed* above. The
   number is left vacant rather than reused.)
 
-### B7 · Three methods are type piracy, and closing them is an API change rather than a deletion.
-
-- **location:** `src/layers/resnet.jl:63`
-- **kind:** defect
-- **found:** 2026-09-19
-- **evidence:**
-  Aqua's `piracies` check reports 3, down from the 12 this entry opened with; the other nine were
-  deleted in this release and are under *Removed (breaking)* above. All three are genuine under
-  Aqua's definition — the function and every argument type belong to other modules — and all three
-  are one thing: a functor on a *layer* type this package does not own, applied to a three-axis
-  array.
-
-  | method | `src` | without GML | with GML |
-  |:--|:--|:--|:--|
-  | `(::Dense{M, N, true})(::AbstractArray{T, 3}, ::NamedTuple)` | `layers/resnet.jl:63` | `MethodError` | the layer applied along the third axis |
-  | `(::Dense{M, N, false})(::AbstractArray{T, 3}, ::NamedTuple)` | `layers/resnet.jl:67` | the same | the same |
-  | `(::Linear{M, N})(::AbstractArray{T, 3}, ::NamedTuple)` | `layers/resnet.jl:71` | the same | the same |
-
-  `Dense` and `Linear` are `AbstractNeuralNetworks`', and upstream a 3-tensor argument is a
-  `MethodError` because `*` cannot take one. So the witness is not a changed answer but a new one:
-  a call that throws without this package loaded returns a value with it.
-
-  Closing this means either `AbstractNeuralNetworks` gaining the three-axis methods, or this
-  package wrapping the two layer types in its own. Both are an API change, and neither is a
-  by-product of a piracy pass — which is why these three stayed when the other nine went.
-
-  `src/layers/resnet.jl:63` is also the one remaining method ambiguity, against
-  `AbstractNeuralNetworks.Affine`. That pair is benign and has no witness at all: `Dense` is not a
-  subtype of `Affine` and the two have no common instance, so no call can reach it. `test/quality/aqua.jl`
-  asserts both counts — 3 piracies and 1 ambiguity — so neither can drift, in either direction.
-
 ### B10 · `DataLoader(::EnsembleSolution{T, T1, Vector{ST}})` at `src/data_loader/data_loader.jl:325` has no test and appears unreachable through this package's current dependencies.
 
 - **location:** `src/data_loader/data_loader.jl:325`
@@ -135,37 +104,6 @@ What is known to be wrong in GeometricMachineLearning and is not fixed.
 
 ## C. Follow-ups and cleanups
 
-### C1 · The parameter-tree traversal still belongs upstream.
-
-- **location:** —
-- **kind:** upstream
-- **found:** 2026-08-17
-- **evidence:**
-  `_make_optimizer_cache`,
-  `_make_optimizer_state`, `_tree_optim_step!`, `_leaf_optim_step!` and the bespoke
-  `GMLEuclideanState` are GML implementations of what GeometricOptimizers supports natively for a
-  single parameter. `GMLEuclideanState` in particular duplicates what `GradientState`,
-  `MomentumState` and `AdamState` already do for a plain array.
-
-  What has to go upstream is *not* a reuse of GeometricOptimizers' `Optimizer`: that one needs an
-  `OptimizerProblem`, i.e. an objective function, and minibatch training has none — the gradient
-  arrives from AD one batch at a time. It is a new entry point there, a
-  gradient-supplied-externally step over a `NamedTuple` parameter tree. GML's `Optimizer` would then
-  be the `NeuralNetwork` constructor and the training-loop functor, and nothing else.
-
-  `Optimizer` is the one name still exported by both packages, so this is also what closes the last
-  of B1's class of collision.
-
-### C2 · Two `isa` branches remain in `_leaf_optim_step!`
-
-- **location:** —
-- **kind:** defect
-- **found:** 2026-08-16
-- **evidence:**
-  (for `AdamState`/`MomentumState`).
-  Measurement showed the traversal is not implicated in the compile-time problem, so this is tidying,
-  and it disappears entirely if C1 lands first.
-
 ### C6 · Three generated MNIST PDFs are in this branch's history
 
 - **location:** `docs/src`
@@ -241,25 +179,20 @@ What is known to be wrong in GeometricMachineLearning and is not fixed.
   where training spends its time, so the gain should be larger here than the forward figures, which
   is also why it wants its own before-and-after measurement rather than being folded into that pass.
 
-### C16 · Four of `ExplicitImports`' seven checks are switched off in `test/quality/aqua.jl`.
+### C16 · One of `ExplicitImports`' seven checks is switched off in `test/quality/aqua.jl`, and two run with a list of names.
 
 - **location:** `test/quality/aqua.jl`
 - **kind:** upstream
 - **found:** 2026-09-18
 - **evidence:**
-  The counts, at
-  the commit that added the gate: **37** names arriving through a bare `using` across thirteen
-  packages;
-  **11** explicit imports of names upstream does not mark `public`; **2** qualified accesses through a
-  non-owner module, `GeometricOptimizers.Gradient` and `GeometricOptimizers.direction`, both owned by
-  `SimpleSolvers`; and **22** qualified accesses to non-public names.
-
-  They are four separate jobs, not one. Naming the 37 is a rewrite of the module header with a
-  judgement per name. The 11 and the 22 are the same class and most of them are not this package's to
-  fix — `Architecture`, `AbstractExplicitLayer`, `add!`, `_compute_loss`, `assign_columns` and
-  `description` are all used deliberately, and the resolution is upstream declaring them `public`.
-  Each check's reason is written above the `@testset` rather than here alone, because switching one
-  back on turns the suite red on the spot.
+  `no_implicit_imports` is off: **37** names arrive through a bare `using` across thirteen packages,
+  and naming them is a rewrite of the module header with a judgement per name.
+  `all_explicit_imports_are_public` and `all_qualified_accesses_are_public` run with a list of the
+  names this package reaches in other packages that those packages do not declare `public`: **10**
+  explicit imports and **10** qualified accesses, none of them in `GeometricOptimizers`.
+  `Architecture`, `AbstractExplicitLayer`, `_compute_loss`, `dim` and the rest are used
+  deliberately, and the resolution is upstream declaring them `public`. Each check's reason is
+  written beside it in the test file.
 
 ### C17 · `accuracy` cannot be reached through any public `DataLoader` constructor.
 
@@ -347,23 +280,9 @@ What is known to be wrong in GeometricMachineLearning and is not fixed.
   supposed to be worth testing at all — training the two networks first would make the comparison
   mean something, at a cost the suite does not currently pay.
 
-## D. Unverified
-
-Not defects — claims this release makes that nothing has actually checked yet.
-
-### D4 · The upstream fix was measured on one optimizer.
-
-- **location:** —
-- **kind:** not verified
-- **found:** 2026-08-16
-- **evidence:**
-  The compile-time figures come from the
-  `Adam` path. The quasi-Newton and Newton caches and states were widened on the strength of their
-  *inferred types* — a sound argument, but not a measurement. Catalogued upstream as GeometricOptimizers C15.
-
 ## Found late
 
-### K1 · Four source comments still send the reader to the Open Issues section of `CHANGELOG.md`, which no longer exists.
+### K1 · Two source comments still send the reader to the Open Issues section of `CHANGELOG.md`, which no longer exists.
 
 - **location:** `src/reduced_system/reduced_system.jl:109`
 - **kind:** found late
@@ -373,18 +292,78 @@ Not defects — claims this release makes that nothing has actually checked yet.
   $ grep -rnE 'Open Issues|issue C1[48]|\*C21\*|\*B7\*' src test
   src/kernels/exponentials/tensor_exponential.jl:2:# name is wider than what it holds; renaming it, and `exponentials/` with it, is issue C18.
   src/reduced_system/reduced_system.jl:109:# issue C14 in `CHANGELOG.md`.
-  test/quality/aqua.jl:30:# layer types, and both are an API change rather than a tidy-up. They are *B7* under
-  test/quality/aqua.jl:31:# `## Open Issues` in `CHANGELOG.md` with that reasoning.
   test/reduced_system/reduced_system.jl:44:# for four of twelve, with four `NaN` and four in the opposite order. That is *C21* in
-  $ grep -n 'CHANGELOG' test/quality/aqua.jl
-  31:# `## Open Issues` in `CHANGELOG.md` with that reasoning.
-  63:    # and it fails when one is removed without the entry above and in `CHANGELOG.md` going with it.
   ```
 
-  The IDs are still valid; the comments at `src/reduced_system/reduced_system.jl:109`,
-  `test/quality/aqua.jl:31`, `test/quality/aqua.jl:63` and `test/reduced_system/reduced_system.jl:44–45`
-  name `CHANGELOG.md` rather than `KNOWN_ISSUES.md`. The fix is a follow-up change to those
-  comments.
+  The IDs are still valid; the comments at `src/reduced_system/reduced_system.jl:109` and
+  `test/reduced_system/reduced_system.jl:44–45` name `CHANGELOG.md` rather than `KNOWN_ISSUES.md`.
+  The fix is a follow-up change to those comments.
+
+### K7 · A network with a Stiefel leaf raises in `mat_tensor_mul` when the data has another element type.
+
+- **location:** `src/kernels/mat_tensor_mul.jl`
+- **kind:** found late
+- **found:** 2026-10-12, by the critics of part M1a; the kernel code is older than that part
+- **evidence:**
+  A `Float32` `Chain(StiefelLayer(6, 3), Dense(3, 2, tanh))` trained with `train!` on `Float64`
+  data, and a `Float64` one on `Float32` data, raise `AssertionError: eltype(A) == eltype(B)` in
+  `mat_tensor_mul`, reached through `src/layers/manifold_layer.jl` under Zygote. A network of
+  `Dense` layers alone trains on mixed precisions.
+
+  ```julia
+  x = rand(Float32, 6, 12)
+  nn = NeuralNetwork(Chain(StiefelLayer(6, 3), Dense(3, 2, tanh)), Float64)
+  train!(nn, TrainingOptimizer(nn.params), DataLoader(x, x[1:2, :]), Batch(4), 2; loss = FeedForwardLoss())
+  ```
+
+### K8 · A `DataLoader` of one sample cannot be batched.
+
+- **location:** `src/data_loader/batch.jl`
+- **kind:** found late
+- **found:** 2026-10-12, by the critics of part M1a
+- **evidence:**
+  `DataLoader(x, x)` with `x = rand(Float32, 2, 1)` and `Batch(1)` raises the time-series
+  assertion "The number of time steps has to be greater than sequence length + prediction window"
+  when `train!` draws its minibatches.
+
+### K11 · A comment in `adam_with_learning_rate_decay.jl` reports a past experiment.
+
+- **location:** `test/optimizers/adam_with_learning_rate_decay.jl`
+- **kind:** found late
+- **found:** 2026-10-12, by the critics of part M1a; the comment is older than that part
+- **evidence:**
+  The comment above `train_manifold_network` says "from a seeded start the loss fell by under 2%
+  on 1.13 and *rose* on 1.12". The fix states only the present constraint: a 32-epoch budget
+  drives the learning rate to `η₂` almost at once, and 128 epochs train.
+
+### K12 · `Batch` never draws the last sample of a `DataLoader` of matrices.
+
+- **location:** `src/data_loader/batch.jl`
+- **kind:** found late
+- **found:** 2026-10-12, by the critics of part M1a
+- **evidence:**
+  On a `DataLoader(x, x)` with `x = rand(2, 10)`, the minibatches of `Batch(bs)` hold the samples
+  1 to 9 for each `bs` in 1, 3, 4 and 10; sample 10 is in none of them. `Batch(1)` gives 9
+  minibatches.
+
+  ```
+  1 nbatches 9 samples [(1, 1), (2, 1), (3, 1), (4, 1), (5, 1), (6, 1), (7, 1), (8, 1), (9, 1)]
+  10 nbatches 1 samples [(1, 1), (2, 1), (3, 1), (4, 1), (5, 1), (6, 1), (7, 1), (8, 1), (9, 1)]
+  ```
+
+### K13 · `SymbolicPullback(arch)` gives the gradient of the per-sample sum, not of `HNNLoss`.
+
+- **location:** `src/pullbacks/symbolic_hnn_pullback.jl`
+- **kind:** found late
+- **found:** 2026-10-12, while adding the `pullback` keyword of `train!` (#342); issue #343
+- **evidence:**
+  The value of the pullback is the batch loss `norm(F(X) - Y) / norm(Y)`, but the gradient is
+  built with `reduce = +`, so it is the gradient of the sum of the per-sample losses. On one
+  minibatch of 5 samples in `Float64`, `dL/dW1[1]` is `0.002695929479656911` by a central finite
+  difference, `0.0026959294110209093` by Zygote and `0.004388379726963325` by the symbolic
+  pullback; the symbolic gradient agrees with Zygote's gradient of the per-sample sum to `5.2e-17`.
+  `main` at cfe4aea3 gives the same numbers. `test/data_loader/train.jl` marks the comparison with
+  Zygote `@test_broken`.
 
 ## Upstream
 
@@ -424,6 +403,28 @@ Not defects — claims this release makes that nothing has actually checked yet.
   itself records a skip where `Metal.functional()` is false; only the precompile fails. Closing this
   is a Metal.jl change: a workload that does not run where no device exists.
 
+### K6 · The gradient of a `SymmetricMatrix` leaf with respect to a `NetworkParameters` differs from the one with respect to the bare `NamedTuple`.
+
+- **location:** `test/helpers/network_parameters_gradient_structure.jl`
+- **kind:** upstream
+- **found:** 2026-10-12, with the move to NeuralNetworkParameters 0.5; issue #319
+- **evidence:**
+  Under NeuralNetworkParameters 0.5, `Zygote.gradient` with respect to a `NetworkParameters` gives
+  a `SymmetricMatrix` leaf that holds the gradient with respect to the leaf's storage, at any number
+  of accesses; with respect to the bare `NamedTuple` it gives the projection of the dense gradient.
+  The off-diagonal entries of the first are twice those of the second; the diagonals are equal. A
+  test of `isapprox(Matrix(a_wrapped), Matrix(a_bare))` in
+  `layers/symplectic_attention_network_parameters_gradient.jl` evaluates:
+
+  ```
+  Expression: isapprox(Matrix(a_wrapped), Matrix(a_bare))
+   Evaluated: isapprox([0.6575109961994726 1.1214311018548968; 1.1214311018548968 0.46185556373061615],
+                       [0.6575109961994726 0.5607155509274484; 0.5607155509274484 0.46185556373061615])
+  ```
+
+  The helper asserts this relation. Which of the two a training step needs is the pullback shape of
+  issue #319.
+
 ## Documentation
 
 ### K5 · The `PositionalEncoding` docstring names `test/metal/`, which does not exist
@@ -453,7 +454,19 @@ Not defects — claims this release makes that nothing has actually checked yet.
   `target_modules = (GeometricMachineLearning,)` a barrier on a launch argument gives 0 reports,
   and the same barrier in an `@assert` of the same launcher gives 3: the kernel functor takes
   `args...`, so the dispatch sits in KernelAbstractions' frames, which the filter drops. In
-  `assign_ones_for_poisson_tensor_kernel!` (`src/arrays/poisson_tensor.jl:91`) two mutants
+  `assign_ones_for_poisson_tensor_kernel!` (`src/arrays/poisson_tensor.jl:95`) two mutants
   SURVIVED: a barrier on `n` in `map_index_for_poisson_tensor(i, n)`, and a barrier on the written
   array `J`; a barrier on the stored value is CAUGHT. JET never reports a dynamic dispatch in the
   kernel's own frame. A barrier on a value inside each kernel body is CAUGHT.
+
+### K10 · The two training tests of `adam_with_learning_rate_decay.jl` run in `Float64` only.
+
+- **location:** `test/optimizers/adam_with_learning_rate_decay.jl`
+- **kind:** missing test
+- **found:** 2026-10-12, by the critics of part M1a; the precision is older than that part
+- **evidence:**
+  `train_network` and `train_manifold_network` build their `DataLoader` from
+  `sin.(0:0.1:2π)`, a `Vector{Float64}`, and take the network's element type from `eltype(dl)`.
+  Neither function takes an element type, so neither runs over `REAL_ELTYPES`. The third testset
+  of the file, `schedule_starts_at_one`, loops over `REAL_ELTYPES`. The fix builds `dl` per `T`
+  and loops both functions over `REAL_ELTYPES`.
diff --git a/Project.toml b/Project.toml
index 61360a273..207d713ea 100644
--- a/Project.toml
+++ b/Project.toml
@@ -25,30 +25,27 @@ Zygote = "e88e6eb3-aa80-5325-afca-941959d7151f"
 
 [weakdeps]
 GPUArraysCore = "46192b85-c4d5-4398-a991-12ede77f4527"
-HDF5 = "f67ccb44-e63f-5c2f-98bd-6dc0ccc4ba2f"
 
 [extensions]
 GPUArraysCoreExt = "GPUArraysCore"
-HDF5Ext = "HDF5"
 
 [compat]
-AbstractNeuralNetworks = "0.8"
+AbstractNeuralNetworks = "0.9.1"
 ChainRulesCore = "1"
 ForwardDiff = "1"
 GPUArraysCore = "0.2"
-GeometricBase = "0.14"
-GeometricEquations = "0.21"
-GeometricOptimizers = "0.8"
-GeometricSolutions = "0.6"
-HDF5 = "0.16, 0.17"
+GeometricBase = "0.15"
+GeometricEquations = "0.21.5"
+GeometricOptimizers = "0.9"
+GeometricSolutions = "0.6.6"
 InteractiveUtils = "1"
 KernelAbstractions = "0.9"
 LinearAlgebra = "1"
 NNlib = "0.8, 0.9"
-NeuralNetworkParameters = "0.3, 0.4"
+NeuralNetworkParameters = "0.5"
 ProgressMeter = "1"
 Random = "1"
-SymbolicNeuralNetworks = "0.8"
+SymbolicNeuralNetworks = "0.9"
 Symbolics = "7"
 TimerOutputs = "0.5, 1"
 Zygote = "0.7"
diff --git a/README.md b/README.md
index e7c86a073..77e6c9ad8 100644
--- a/README.md
+++ b/README.md
@@ -43,14 +43,14 @@ backend = CUDABackend()
 # initialize the network (i.e. the parameters of the network)
 g_nn = NeuralNetwork(gsympnet, backend, type)
 
-# call the optimizer: the method comes first, the step size is given separately
-g_opt = Optimizer(Adam(type), g_nn; step_size = 1e-3)
+# set up the optimizer: the method and the step size are keywords
+g_opt = TrainingOptimizer(g_nn.params; algorithm = Adam(), linesearch = 1e-3)
 
 const nepochs = 300
 const batch_size = 100
 
 # train the network
-g_loss_array = g_opt(g_nn, dl, Batch(batch_size), nepochs)
+g_loss_array = train!(g_nn, g_opt, dl, Batch(batch_size), nepochs)
 
 # plot the result
 ics = (q=qp_data.q[:,1], p=qp_data.p[:,1])
diff --git a/docs/check_references.jl b/docs/check_references.jl
index e012f016f..a10e54245 100644
--- a/docs/check_references.jl
+++ b/docs/check_references.jl
@@ -12,14 +12,12 @@
 #     julia --project=docs docs/check_references.jl
 
 using GeometricMachineLearning
-using HDF5 # so that the extension below is loaded, as in make.jl
 using AbstractNeuralNetworks # signatures in `@docs` blocks name types from here
 using Documenter
 using Documenter: DocSystem
 
 # keep in sync with `modules` in make.jl
-const MODULES = Module[GeometricMachineLearning,
-    Base.get_extension(GeometricMachineLearning, :HDF5Ext)]
+const MODULES = Module[GeometricMachineLearning]
 
 const SRC = joinpath(@__DIR__, "src")
 
diff --git a/docs/inventories/GeometricOptimizers.toml b/docs/inventories/GeometricOptimizers.toml
index 0ecded68a..247ede361 100644
--- a/docs/inventories/GeometricOptimizers.toml
+++ b/docs/inventories/GeometricOptimizers.toml
@@ -1,6 +1,6 @@
 # DocInventory version 1
 project = "GeometricOptimizers.jl"
-version = "0.5.0"
+version = "0.9.0"
 
 [[jl.constant]]
 name = "GeometricOptimizers.CURVATURE_TOLERANCE"
@@ -9,6 +9,9 @@ uri = "api/#$"
 name = "GeometricOptimizers.DEFAULT_STEP_CEILING"
 uri = "api/#$"
 
+[[jl.function]]
+name = "GeometricOptimizers.default_eltype"
+uri = "api/#$"
 [[jl.function]]
 name = "GeometricOptimizers.geodesic"
 uri = "api/#$"
@@ -16,13 +19,31 @@ uri = "api/#$"
 name = "GeometricOptimizers.manifold_type"
 uri = "api/#$"
 [[jl.function]]
-name = "GeometricOptimizers.𝔄exp"
-uri = "api/#GeometricOptimizers.%F0%9D%94%84exp"
+name = "GeometricOptimizers.retraction_workspace"
+uri = "api/#$"
+[[jl.function]]
+name = "GeometricOptimizers.symplectic_gram_schmidt"
+uri = "api/#$"
+[[jl.function]]
+name = "GeometricOptimizers.symplectic_gram_schmidt!"
+uri = "api/#GeometricOptimizers.symplectic_gram_schmidt%21"
+[[jl.function]]
+name = "GeometricOptimizers.trial_iterate!"
+uri = "api/#GeometricOptimizers.trial_iterate%21"
 
+[[jl.method]]
+name = "Base.:*-Tuple{AbstractLieAlgHorMatrix, AbstractMatrix}"
+uri = "api/#Base.%3A%2A-Tuple%7BAbstractLieAlgHorMatrix%2C%20AbstractMatrix%7D"
+[[jl.method]]
+name = "Base.:*-Tuple{AbstractTriangular, AbstractMatrix}"
+uri = "api/#Base.%3A%2A-Tuple%7BAbstractTriangular%2C%20AbstractMatrix%7D"
 [[jl.method]]
 name = "Base.:*-Tuple{GlobalSection, Manifold}"
 uri = "api/#Base.%3A%2A-Tuple%7BGlobalSection%2C%20Manifold%7D"
 [[jl.method]]
+name = "Base.:*-Tuple{StiefelProjection, AbstractMatrix}"
+uri = "api/#Base.%3A%2A-Tuple%7BStiefelProjection%2C%20AbstractMatrix%7D"
+[[jl.method]]
 name = "Base.Matrix-Tuple{GlobalSection}"
 uri = "api/#Base.Matrix-Tuple%7BGlobalSection%7D"
 [[jl.method]]
@@ -32,44 +53,29 @@ uri = "api/#Base.one-Union%7BTuple%7BAbstractLieAlgHorMatrix%7BT%7D%7D%2C%20Tupl
 name = "Base.parent-Tuple{AbstractLieAlgHorMatrix}"
 uri = "api/#Base.parent-Tuple%7BAbstractLieAlgHorMatrix%7D"
 [[jl.method]]
-name = "Base.rand-Union{Tuple{MT}, Tuple{KernelAbstractions.Backend, Type{MT}, Integer, Integer}} where MT<:Manifold"
-uri = "api/#Base.rand-Union%7BTuple%7BMT%7D%2C%20Tuple%7BKernelAbstractions.Backend%2C%20Type%7BMT%7D%2C%20Integer%2C%20Integer%7D%7D%20where%20MT%3C%3AManifold"
-[[jl.method]]
-name = "Base.rand-Union{Tuple{MT}, Tuple{Type{MT}, Integer, Integer}} where MT<:Manifold"
-uri = "api/#Base.rand-Union%7BTuple%7BMT%7D%2C%20Tuple%7BType%7BMT%7D%2C%20Integer%2C%20Integer%7D%7D%20where%20MT%3C%3AManifold"
+name = "Base.rand-Tuple{KernelAbstractions.Backend, Type{<:Manifold}, Integer, Integer}"
+uri = "api/#Base.rand-Tuple%7BKernelAbstractions.Backend%2C%20Type%7B%3C%3AManifold%7D%2C%20Integer%2C%20Integer%7D"
 [[jl.method]]
-name = "Base.vec-Tuple{AbstractLieAlgHorMatrix}"
-uri = "api/#Base.vec-Tuple%7BAbstractLieAlgHorMatrix%7D"
+name = "Base.rand-Tuple{Type{<:Manifold}, Integer, Integer}"
+uri = "api/#Base.rand-Tuple%7BType%7B%3C%3AManifold%7D%2C%20Integer%2C%20Integer%7D"
 [[jl.method]]
-name = "Base.vec-Tuple{AbstractTriangular}"
-uri = "api/#Base.vec-Tuple%7BAbstractTriangular%7D"
-[[jl.method]]
-name = "Base.vec-Tuple{SkewSymMatrix}"
-uri = "api/#Base.vec-Tuple%7BSkewSymMatrix%7D"
+name = "Base.rand-Tuple{Type{<:SymplecticStiefelManifold}, Integer, Integer}"
+uri = "api/#Base.rand-Tuple%7BType%7B%3C%3ASymplecticStiefelManifold%7D%2C%20Integer%2C%20Integer%7D"
 [[jl.method]]
 name = "GeometricBase.update!-Tuple{GeometricOptimizers.NewtonOptimizerCache, OptimizerState, Gradient, Hessian, AbstractVector}"
 uri = "api/#GeometricBase.update%21-Tuple%7BGeometricOptimizers.NewtonOptimizerCache%2C%20OptimizerState%2C%20Gradient%2C%20Hessian%2C%20AbstractVector%7D"
 [[jl.method]]
-name = "GeometricBase.update!-Tuple{NewtonOptimizerState, Gradient, AbstractVector}"
-uri = "api/#GeometricBase.update%21-Tuple%7BNewtonOptimizerState%2C%20Gradient%2C%20AbstractVector%7D"
-[[jl.method]]
-name = "GeometricBase.update!-Union{Tuple{T}, Tuple{GeometricOptimizers.BFGSCache{T}, BFGSState{T}, OptimizerSolution{T}, Union{AbstractArray{T}, NamedTuple{S, <:Tuple{Vararg{AbstractArray{T}}}} where S}}} where T"
-uri = "api/#GeometricBase.update%21-Union%7BTuple%7BT%7D%2C%20Tuple%7BGeometricOptimizers.BFGSCache%7BT%7D%2C%20BFGSState%7BT%7D%2C%20OptimizerSolution%7BT%7D%2C%20Union%7BAbstractArray%7BT%7D%2C%20NamedTuple%7BS%2C%20%3C%3ATuple%7BVararg%7BAbstractArray%7BT%7D%7D%7D%7D%20where%20S%7D%7D%7D%20where%20T"
-[[jl.method]]
-name = "GeometricBase.update!-Union{Tuple{T}, Tuple{GeometricOptimizers.DFPCache{T}, BFGSState{T}, OptimizerSolution{T}, Union{AbstractArray{T}, NamedTuple{S, <:Tuple{Vararg{AbstractArray{T}}}} where S}}} where T"
-uri = "api/#GeometricBase.update%21-Union%7BTuple%7BT%7D%2C%20Tuple%7BGeometricOptimizers.DFPCache%7BT%7D%2C%20BFGSState%7BT%7D%2C%20OptimizerSolution%7BT%7D%2C%20Union%7BAbstractArray%7BT%7D%2C%20NamedTuple%7BS%2C%20%3C%3ATuple%7BVararg%7BAbstractArray%7BT%7D%7D%7D%7D%20where%20S%7D%7D%7D%20where%20T"
+name = "GeometricBase.update!-Tuple{NewtonState, Gradient, AbstractVector}"
+uri = "api/#GeometricBase.update%21-Tuple%7BNewtonState%2C%20Gradient%2C%20AbstractVector%7D"
 [[jl.method]]
-name = "GeometricOptimizers.AdamOptimizerWithDecay-Union{Tuple{Integer}, Tuple{T}, Tuple{Integer, Type{T}}} where T"
-uri = "api/#GeometricOptimizers.AdamOptimizerWithDecay-Union%7BTuple%7BInteger%7D%2C%20Tuple%7BT%7D%2C%20Tuple%7BInteger%2C%20Type%7BT%7D%7D%7D%20where%20T"
-[[jl.method]]
-name = "GeometricOptimizers.AdamW-Tuple"
-uri = "api/#$"
+name = "GeometricBase.update!-Union{Tuple{T}, Tuple{GeometricOptimizers.QuasiNewtonCache{T}, BFGSState{T}, OptimizerSolution{T}, Union{AbstractArray{T}, NetworkParameters{T}}}} where T"
+uri = "api/#GeometricBase.update%21-Union%7BTuple%7BT%7D%2C%20Tuple%7BGeometricOptimizers.QuasiNewtonCache%7BT%7D%2C%20BFGSState%7BT%7D%2C%20OptimizerSolution%7BT%7D%2C%20Union%7BAbstractArray%7BT%7D%2C%20NetworkParameters%7BT%7D%7D%7D%7D%20where%20T"
 [[jl.method]]
-name = "GeometricOptimizers.GrassmannLieAlgHorMatrix-Tuple{AbstractMatrix, Int64}"
-uri = "api/#GeometricOptimizers.GrassmannLieAlgHorMatrix-Tuple%7BAbstractMatrix%2C%20Int64%7D"
+name = "GeometricOptimizers.AdamOptimizerWithDecay-Tuple{Integer}"
+uri = "api/#GeometricOptimizers.AdamOptimizerWithDecay-Tuple%7BInteger%7D"
 [[jl.method]]
-name = "GeometricOptimizers.LowerTriangular-Union{Tuple{AbstractMatrix{T}}, Tuple{T}} where T"
-uri = "api/#GeometricOptimizers.LowerTriangular-Union%7BTuple%7BAbstractMatrix%7BT%7D%7D%2C%20Tuple%7BT%7D%7D%20where%20T"
+name = "GeometricOptimizers.GrassmannLieAlgHorMatrix-Tuple{AbstractMatrix, Integer}"
+uri = "api/#GeometricOptimizers.GrassmannLieAlgHorMatrix-Tuple%7BAbstractMatrix%2C%20Integer%7D"
 [[jl.method]]
 name = "GeometricOptimizers.Optimizer-Union{Tuple{VT}, Tuple{T}, Tuple{VT, Function}} where {T, VT<:OptimizerSolution{T}}"
 uri = "api/#GeometricOptimizers.Optimizer-Union%7BTuple%7BVT%7D%2C%20Tuple%7BT%7D%2C%20Tuple%7BVT%2C%20Function%7D%7D%20where%20%7BT%2C%20VT%3C%3AOptimizerSolution%7BT%7D%7D"
@@ -86,45 +92,75 @@ uri = "api/#GeometricOptimizers.StiefelProjection-Union%7BTuple%7BAbstractLieAlg
 name = "GeometricOptimizers.StiefelProjection-Union{Tuple{AbstractMatrix{T}}, Tuple{T}} where T"
 uri = "api/#GeometricOptimizers.StiefelProjection-Union%7BTuple%7BAbstractMatrix%7BT%7D%7D%2C%20Tuple%7BT%7D%7D%20where%20T"
 [[jl.method]]
+name = "GeometricOptimizers.StrictlyLowerTriangular-Union{Tuple{AbstractMatrix{T}}, Tuple{T}} where T"
+uri = "api/#GeometricOptimizers.StrictlyLowerTriangular-Union%7BTuple%7BAbstractMatrix%7BT%7D%7D%2C%20Tuple%7BT%7D%7D%20where%20T"
+[[jl.method]]
+name = "GeometricOptimizers.StrictlyUpperTriangular-Union{Tuple{AbstractMatrix{T}}, Tuple{T}} where T"
+uri = "api/#GeometricOptimizers.StrictlyUpperTriangular-Union%7BTuple%7BAbstractMatrix%7BT%7D%7D%2C%20Tuple%7BT%7D%7D%20where%20T"
+[[jl.method]]
 name = "GeometricOptimizers.SymmetricMatrix-Union{Tuple{AbstractMatrix{T}}, Tuple{T}} where T"
 uri = "api/#GeometricOptimizers.SymmetricMatrix-Union%7BTuple%7BAbstractMatrix%7BT%7D%7D%2C%20Tuple%7BT%7D%7D%20where%20T"
 [[jl.method]]
-name = "GeometricOptimizers.UpperTriangular-Union{Tuple{AbstractMatrix{T}}, Tuple{T}} where T"
-uri = "api/#GeometricOptimizers.UpperTriangular-Union%7BTuple%7BAbstractMatrix%7BT%7D%7D%2C%20Tuple%7BT%7D%7D%20where%20T"
-[[jl.method]]
-name = "GeometricOptimizers._div!-Tuple{AbstractArray, AbstractArray, AbstractArray}"
-uri = "api/#GeometricOptimizers._div%21-Tuple%7BAbstractArray%2C%20AbstractArray%2C%20AbstractArray%7D"
+name = "GeometricOptimizers._cholesky_qr2-Tuple{AbstractMatrix}"
+uri = "api/#GeometricOptimizers._cholesky_qr2-Tuple%7BAbstractMatrix%7D"
 [[jl.method]]
 name = "GeometricOptimizers._dot-Tuple{AbstractVecOrMat, AbstractVecOrMat}"
 uri = "api/#GeometricOptimizers._dot-Tuple%7BAbstractVecOrMat%2C%20AbstractVecOrMat%7D"
 [[jl.method]]
+name = "GeometricOptimizers._flat_mul!-Tuple{Any, Any, Any, Nothing}"
+uri = "api/#GeometricOptimizers._flat_mul%21-Tuple%7BAny%2C%20Any%2C%20Any%2C%20Nothing%7D"
+[[jl.method]]
+name = "GeometricOptimizers._flat_scratch-Union{Tuple{T}, Tuple{Type{T}, AbstractVector}} where T"
+uri = "api/#GeometricOptimizers._flat_scratch-Union%7BTuple%7BT%7D%2C%20Tuple%7BType%7BT%7D%2C%20AbstractVector%7D%7D%20where%20T"
+[[jl.method]]
 name = "GeometricOptimizers._is_decayable-Tuple{StiefelManifold}"
 uri = "api/#GeometricOptimizers._is_decayable-Tuple%7BStiefelManifold%7D"
 [[jl.method]]
-name = "GeometricOptimizers._manifold_αmax-Union{Tuple{T}, Tuple{Tuple{}, Tuple{}, T}} where T"
-uri = "api/#GeometricOptimizers._manifold_%CE%B1max-Union%7BTuple%7BT%7D%2C%20Tuple%7BTuple%7B%7D%2C%20Tuple%7B%7D%2C%20T%7D%7D%20where%20T"
+name = "GeometricOptimizers._manifold_αmax-Union{Tuple{T}, Tuple{NetworkParameters, Any, T}} where T"
+uri = "api/#GeometricOptimizers._manifold_%CE%B1max-Union%7BTuple%7BT%7D%2C%20Tuple%7BNetworkParameters%2C%20Any%2C%20T%7D%7D%20where%20T"
 [[jl.method]]
-name = "GeometricOptimizers._optimizer-Union{Tuple{T}, Tuple{OptimizerSolution{T}, OptimizerProblem{T}, OptimizerMethod, LinesearchMethod, Gradient{T}, AbstractRetraction, Options{T}, Real}} where T"
-uri = "api/#GeometricOptimizers._optimizer-Union%7BTuple%7BT%7D%2C%20Tuple%7BOptimizerSolution%7BT%7D%2C%20OptimizerProblem%7BT%7D%2C%20OptimizerMethod%2C%20LinesearchMethod%2C%20Gradient%7BT%7D%2C%20AbstractRetraction%2C%20Options%7BT%7D%2C%20Real%7D%7D%20where%20T"
+name = "GeometricOptimizers._native_pade_polynomials-Tuple{AbstractMatrix, AbstractMatrix}"
+uri = "api/#GeometricOptimizers._native_pade_polynomials-Tuple%7BAbstractMatrix%2C%20AbstractMatrix%7D"
 [[jl.method]]
-name = "GeometricOptimizers._rac!-Tuple{AbstractArray, AbstractArray}"
-uri = "api/#GeometricOptimizers._rac%21-Tuple%7BAbstractArray%2C%20AbstractArray%7D"
+name = "GeometricOptimizers._optimizer-Union{Tuple{T}, Tuple{OptimizerSolution{T}, OptimizerProblem{T}, OptimizerMethod, Union{Real, LinesearchMethod}, Gradient{T}, AbstractRetraction, Options{T}, Real, Any}} where T"
+uri = "api/#GeometricOptimizers._optimizer-Union%7BTuple%7BT%7D%2C%20Tuple%7BOptimizerSolution%7BT%7D%2C%20OptimizerProblem%7BT%7D%2C%20OptimizerMethod%2C%20Union%7BReal%2C%20LinesearchMethod%7D%2C%20Gradient%7BT%7D%2C%20AbstractRetraction%2C%20Options%7BT%7D%2C%20Real%2C%20Any%7D%7D%20where%20T"
 [[jl.method]]
-name = "GeometricOptimizers._square!-Tuple{AbstractArray, AbstractArray}"
-uri = "api/#GeometricOptimizers._square%21-Tuple%7BAbstractArray%2C%20AbstractArray%7D"
+name = "GeometricOptimizers._scaled_kernel!-Tuple{Any, AbstractMatrix, GeometricOptimizers.ScaledSquaring}"
+uri = "api/#GeometricOptimizers._scaled_kernel%21-Tuple%7BAny%2C%20AbstractMatrix%2C%20GeometricOptimizers.ScaledSquaring%7D"
 [[jl.method]]
 name = "GeometricOptimizers._squared_gradient_norm-Tuple{ScalarMomentAdam, GeometricOptimizers.ScalarMomentAdamCache, Gradient, StiefelManifold}"
 uri = "api/#GeometricOptimizers._squared_gradient_norm-Tuple%7BScalarMomentAdam%2C%20GeometricOptimizers.ScalarMomentAdamCache%2C%20Gradient%2C%20StiefelManifold%7D"
 [[jl.method]]
+name = "GeometricOptimizers._sumsq_leaves-Union{Tuple{F}, Tuple{F, Any}} where F"
+uri = "api/#GeometricOptimizers._sumsq_leaves-Union%7BTuple%7BF%7D%2C%20Tuple%7BF%2C%20Any%7D%7D%20where%20F"
+[[jl.method]]
 name = "GeometricOptimizers._weight_decay!-Union{Tuple{T}, Tuple{AbstractArray{T}, AbstractArray{T}, T}} where T"
 uri = "api/#GeometricOptimizers._weight_decay%21-Union%7BTuple%7BT%7D%2C%20Tuple%7BAbstractArray%7BT%7D%2C%20AbstractArray%7BT%7D%2C%20T%7D%7D%20where%20T"
 [[jl.method]]
+name = "GeometricOptimizers.advance_state!-Tuple{GradientState, GeometricOptimizers.GradientCache, GradientMethod}"
+uri = "api/#GeometricOptimizers.advance_state%21-Tuple%7BGradientState%2C%20GeometricOptimizers.GradientCache%2C%20GradientMethod%7D"
+[[jl.method]]
+name = "GeometricOptimizers.apply_S_inverse_left!-Tuple{AbstractVector, GeometricOptimizers.SymplecticHouseholderDecom}"
+uri = "api/#GeometricOptimizers.apply_S_inverse_left%21-Tuple%7BAbstractVector%2C%20GeometricOptimizers.SymplecticHouseholderDecom%7D"
+[[jl.method]]
+name = "GeometricOptimizers.apply_S_inverse_right!-Tuple{AbstractMatrix, GeometricOptimizers.SymplecticHouseholderDecom}"
+uri = "api/#GeometricOptimizers.apply_S_inverse_right%21-Tuple%7BAbstractMatrix%2C%20GeometricOptimizers.SymplecticHouseholderDecom%7D"
+[[jl.method]]
+name = "GeometricOptimizers.apply_S_left!-Tuple{AbstractVector, GeometricOptimizers.SymplecticHouseholderDecom}"
+uri = "api/#GeometricOptimizers.apply_S_left%21-Tuple%7BAbstractVector%2C%20GeometricOptimizers.SymplecticHouseholderDecom%7D"
+[[jl.method]]
+name = "GeometricOptimizers.apply_S_right!-Tuple{AbstractMatrix, GeometricOptimizers.SymplecticHouseholderDecom}"
+uri = "api/#GeometricOptimizers.apply_S_right%21-Tuple%7BAbstractMatrix%2C%20GeometricOptimizers.SymplecticHouseholderDecom%7D"
+[[jl.method]]
 name = "GeometricOptimizers.apply_section!-Union{Tuple{MT}, Tuple{AT}, Tuple{T}, Tuple{AT, GlobalSection{T, AT, λT} where λT<:Union{Nothing, AbstractArray{T}}, MT}} where {T, AT<:(StiefelManifold{T, AT} where AT<:AbstractMatrix{T}), MT<:(StiefelManifold{T, AT} where AT<:AbstractMatrix{T})}"
 uri = "api/#GeometricOptimizers.apply_section%21-Union%7BTuple%7BMT%7D%2C%20Tuple%7BAT%7D%2C%20Tuple%7BT%7D%2C%20Tuple%7BAT%2C%20GlobalSection%7BT%2C%20AT%2C%20%CE%BBT%7D%20where%20%CE%BBT%3C%3AUnion%7BNothing%2C%20AbstractArray%7BT%7D%7D%2C%20MT%7D%7D%20where%20%7BT%2C%20AT%3C%3A%28StiefelManifold%7BT%2C%20AT%7D%20where%20AT%3C%3AAbstractMatrix%7BT%7D%29%2C%20MT%3C%3A%28StiefelManifold%7BT%2C%20AT%7D%20where%20AT%3C%3AAbstractMatrix%7BT%7D%29%7D"
 [[jl.method]]
 name = "GeometricOptimizers.apply_section-Union{Tuple{AT}, Tuple{T}, Tuple{GlobalSection{T, AT, λT} where λT<:Union{Nothing, AbstractArray{T}}, AT}} where {T, AT<:(StiefelManifold{T, AT} where AT<:AbstractMatrix{T})}"
 uri = "api/#GeometricOptimizers.apply_section-Union%7BTuple%7BAT%7D%2C%20Tuple%7BT%7D%2C%20Tuple%7BGlobalSection%7BT%2C%20AT%2C%20%CE%BBT%7D%20where%20%CE%BBT%3C%3AUnion%7BNothing%2C%20AbstractArray%7BT%7D%7D%2C%20AT%7D%7D%20where%20%7BT%2C%20AT%3C%3A%28StiefelManifold%7BT%2C%20AT%7D%20where%20AT%3C%3AAbstractMatrix%7BT%7D%29%7D"
 [[jl.method]]
+name = "GeometricOptimizers.cayley-Tuple{AbstractVecOrMat}"
+uri = "api/#GeometricOptimizers.cayley-Tuple%7BAbstractVecOrMat%7D"
+[[jl.method]]
 name = "GeometricOptimizers.cayley-Tuple{GrassmannLieAlgHorMatrix}"
 uri = "api/#GeometricOptimizers.cayley-Tuple%7BGrassmannLieAlgHorMatrix%7D"
 [[jl.method]]
@@ -137,6 +173,9 @@ uri = "api/#GeometricOptimizers.cayley-Union%7BTuple%7BT%7D%2C%20Tuple%7BManifol
 name = "GeometricOptimizers.check-Tuple{Manifold}"
 uri = "api/#GeometricOptimizers.check-Tuple%7BManifold%7D"
 [[jl.method]]
+name = "GeometricOptimizers.check-Tuple{SymplecticStiefelManifold}"
+uri = "api/#GeometricOptimizers.check-Tuple%7BSymplecticStiefelManifold%7D"
+[[jl.method]]
 name = "GeometricOptimizers.contains_nonfinite-Tuple{Real}"
 uri = "api/#GeometricOptimizers.contains_nonfinite-Tuple%7BReal%7D"
 [[jl.method]]
@@ -152,6 +191,9 @@ uri = "api/#GeometricOptimizers.default_gradient-Union%7BTuple%7BT%7D%2C%20Tuple
 name = "GeometricOptimizers.default_linesearch-Union{Tuple{T}, Tuple{Type{T}, OptimizerMethod}} where T"
 uri = "api/#GeometricOptimizers.default_linesearch-Union%7BTuple%7BT%7D%2C%20Tuple%7BType%7BT%7D%2C%20OptimizerMethod%7D%7D%20where%20T"
 [[jl.method]]
+name = "GeometricOptimizers.default_step_size-Tuple{Union{GradientMethod, MomentumMethod}}"
+uri = "api/#GeometricOptimizers.default_step_size-Tuple%7BUnion%7BGradientMethod%2C%20MomentumMethod%7D%7D"
+[[jl.method]]
 name = "GeometricOptimizers.ensure_descent!-Tuple{GeometricOptimizers.OptimizerCache, OptimizerMethod, Options}"
 uri = "api/#GeometricOptimizers.ensure_descent%21-Tuple%7BGeometricOptimizers.OptimizerCache%2C%20OptimizerMethod%2C%20Options%7D"
 [[jl.method]]
@@ -164,20 +206,17 @@ uri = "api/#GeometricOptimizers.global_rep-Union%7BTuple%7BAT%7D%2C%20Tuple%7BT%
 name = "GeometricOptimizers.global_rep-Union{Tuple{AT}, Tuple{T}, Tuple{GlobalSection{T, AT, λT} where λT<:Union{Nothing, AbstractArray{T}}, AbstractMatrix{T}}} where {T, AT<:(StiefelManifold{T, AT} where AT<:AbstractMatrix{T})}"
 uri = "api/#GeometricOptimizers.global_rep-Union%7BTuple%7BAT%7D%2C%20Tuple%7BT%7D%2C%20Tuple%7BGlobalSection%7BT%2C%20AT%2C%20%CE%BBT%7D%20where%20%CE%BBT%3C%3AUnion%7BNothing%2C%20AbstractArray%7BT%7D%7D%2C%20AbstractMatrix%7BT%7D%7D%7D%20where%20%7BT%2C%20AT%3C%3A%28StiefelManifold%7BT%2C%20AT%7D%20where%20AT%3C%3AAbstractMatrix%7BT%7D%29%7D"
 [[jl.method]]
-name = "GeometricOptimizers.global_section-Union{Tuple{GrassmannManifold{T, AT} where AT<:AbstractMatrix{T}}, Tuple{T}} where T"
-uri = "api/#GeometricOptimizers.global_section-Union%7BTuple%7BGrassmannManifold%7BT%2C%20AT%7D%20where%20AT%3C%3AAbstractMatrix%7BT%7D%7D%2C%20Tuple%7BT%7D%7D%20where%20T"
-[[jl.method]]
-name = "GeometricOptimizers.global_section-Union{Tuple{StiefelManifold{T, AT} where AT<:AbstractMatrix{T}}, Tuple{T}} where T"
-uri = "api/#GeometricOptimizers.global_section-Union%7BTuple%7BStiefelManifold%7BT%2C%20AT%7D%20where%20AT%3C%3AAbstractMatrix%7BT%7D%7D%2C%20Tuple%7BT%7D%7D%20where%20T"
+name = "GeometricOptimizers.global_section-Tuple{GrassmannManifold}"
+uri = "api/#GeometricOptimizers.global_section-Tuple%7BGrassmannManifold%7D"
 [[jl.method]]
-name = "GeometricOptimizers.gradient-Tuple{GeometricOptimizers.BFGSCache}"
-uri = "api/#GeometricOptimizers.gradient-Tuple%7BGeometricOptimizers.BFGSCache%7D"
+name = "GeometricOptimizers.global_section-Tuple{StiefelManifold}"
+uri = "api/#GeometricOptimizers.global_section-Tuple%7BStiefelManifold%7D"
 [[jl.method]]
-name = "GeometricOptimizers.gradient-Tuple{GeometricOptimizers.DFPCache}"
-uri = "api/#GeometricOptimizers.gradient-Tuple%7BGeometricOptimizers.DFPCache%7D"
+name = "GeometricOptimizers.global_section-Tuple{SymplecticStiefelManifold}"
+uri = "api/#GeometricOptimizers.global_section-Tuple%7BSymplecticStiefelManifold%7D"
 [[jl.method]]
-name = "GeometricOptimizers.gradient-Tuple{GeometricOptimizers.NewtonOptimizerCache}"
-uri = "api/#GeometricOptimizers.gradient-Tuple%7BGeometricOptimizers.NewtonOptimizerCache%7D"
+name = "GeometricOptimizers.gradient-Tuple{GeometricOptimizers.OptimizerCache}"
+uri = "api/#GeometricOptimizers.gradient-Tuple%7BGeometricOptimizers.OptimizerCache%7D"
 [[jl.method]]
 name = "GeometricOptimizers.gradient_difference!-Tuple{GeometricOptimizers.OptimizerCache, OptimizerState}"
 uri = "api/#GeometricOptimizers.gradient_difference%21-Tuple%7BGeometricOptimizers.OptimizerCache%2C%20OptimizerState%7D"
@@ -185,9 +224,6 @@ uri = "api/#GeometricOptimizers.gradient_difference%21-Tuple%7BGeometricOptimize
 name = "GeometricOptimizers.invalidate_latest_gradient!-Tuple{GeometricOptimizers.OptimizerCache}"
 uri = "api/#GeometricOptimizers.invalidate_latest_gradient%21-Tuple%7BGeometricOptimizers.OptimizerCache%7D"
 [[jl.method]]
-name = "GeometricOptimizers.isaOptimizerState-Tuple{Any}"
-uri = "api/#GeometricOptimizers.isaOptimizerState-Tuple%7BAny%7D"
-[[jl.method]]
 name = "GeometricOptimizers.isconverged-Tuple{GeometricOptimizers.OptimizerStatus}"
 uri = "api/#GeometricOptimizers.isconverged-Tuple%7BGeometricOptimizers.OptimizerStatus%7D"
 [[jl.method]]
@@ -197,6 +233,9 @@ uri = "api/#GeometricOptimizers.latest_gradient-Tuple%7BGeometricOptimizers.Opti
 name = "GeometricOptimizers.latest_gradient_is_current-Tuple{GeometricOptimizers.OptimizerCache, OptimizerState, OptimizerSolution}"
 uri = "api/#GeometricOptimizers.latest_gradient_is_current-Tuple%7BGeometricOptimizers.OptimizerCache%2C%20OptimizerState%2C%20OptimizerSolution%7D"
 [[jl.method]]
+name = "GeometricOptimizers.lift_factors!-Union{Tuple{T}, Tuple{GeometricOptimizers.RetractionWorkspace{T, AT} where AT<:AbstractMatrix{T}, StiefelLieAlgHorMatrix{T, AT, ST} where {AT<:(SkewSymMatrix{T, AT} where AT<:AbstractVector{T}), ST<:AbstractMatrix{T}}}} where T"
+uri = "api/#GeometricOptimizers.lift_factors%21-Union%7BTuple%7BT%7D%2C%20Tuple%7BGeometricOptimizers.RetractionWorkspace%7BT%2C%20AT%7D%20where%20AT%3C%3AAbstractMatrix%7BT%7D%2C%20StiefelLieAlgHorMatrix%7BT%2C%20AT%2C%20ST%7D%20where%20%7BAT%3C%3A%28SkewSymMatrix%7BT%2C%20AT%7D%20where%20AT%3C%3AAbstractVector%7BT%7D%29%2C%20ST%3C%3AAbstractMatrix%7BT%7D%7D%7D%7D%20where%20T"
+[[jl.method]]
 name = "GeometricOptimizers.lift_factors-Tuple{StiefelLieAlgHorMatrix}"
 uri = "api/#GeometricOptimizers.lift_factors-Tuple%7BStiefelLieAlgHorMatrix%7D"
 [[jl.method]]
@@ -218,12 +257,24 @@ uri = "api/#GeometricOptimizers.meets_stopping_criteria-Tuple%7BGeometricOptimiz
 name = "GeometricOptimizers.metric-Tuple{GrassmannManifold, AbstractMatrix, AbstractMatrix}"
 uri = "api/#GeometricOptimizers.metric-Tuple%7BGrassmannManifold%2C%20AbstractMatrix%2C%20AbstractMatrix%7D"
 [[jl.method]]
-name = "GeometricOptimizers.metric-Tuple{StiefelManifold, AbstractMatrix, AbstractMatrix}"
-uri = "api/#GeometricOptimizers.metric-Tuple%7BStiefelManifold%2C%20AbstractMatrix%2C%20AbstractMatrix%7D"
+name = "GeometricOptimizers.metric-Union{Tuple{T}, Tuple{StiefelManifold{T, AT} where AT<:AbstractMatrix{T}, AbstractMatrix, AbstractMatrix}} where T"
+uri = "api/#GeometricOptimizers.metric-Union%7BTuple%7BT%7D%2C%20Tuple%7BStiefelManifold%7BT%2C%20AT%7D%20where%20AT%3C%3AAbstractMatrix%7BT%7D%2C%20AbstractMatrix%2C%20AbstractMatrix%7D%7D%20where%20T"
+[[jl.method]]
+name = "GeometricOptimizers.metric-Union{Tuple{T}, Tuple{SymplecticStiefelManifold{T, AT} where AT<:AbstractMatrix{T}, AbstractMatrix, AbstractMatrix}} where T"
+uri = "api/#GeometricOptimizers.metric-Union%7BTuple%7BT%7D%2C%20Tuple%7BSymplecticStiefelManifold%7BT%2C%20AT%7D%20where%20AT%3C%3AAbstractMatrix%7BT%7D%2C%20AbstractMatrix%2C%20AbstractMatrix%7D%7D%20where%20T"
+[[jl.method]]
+name = "GeometricOptimizers.observe_optimizer_phase-Tuple{Any, NoStepObserver, Any}"
+uri = "api/#GeometricOptimizers.observe_optimizer_phase-Tuple%7BAny%2C%20NoStepObserver%2C%20Any%7D"
 [[jl.method]]
 name = "GeometricOptimizers.opnorm₁-Tuple{AbstractMatrix}"
 uri = "api/#GeometricOptimizers.opnorm%E2%82%81-Tuple%7BAbstractMatrix%7D"
 [[jl.method]]
+name = "GeometricOptimizers.optimization_step!-Union{Tuple{T}, Tuple{OptimizerSolution{T}, TrainingOptimizer{<:Union{GradientMethod, Adam, AdamWithEuclideanDecay, MomentumMethod, ScalarMomentAdam}, <:GeometricOptimizers.OptimizerCache, <:OptimizerState{T}}, Union{AbstractArray{T}, NetworkParameters{T}}}} where T"
+uri = "api/#GeometricOptimizers.optimization_step%21-Union%7BTuple%7BT%7D%2C%20Tuple%7BOptimizerSolution%7BT%7D%2C%20TrainingOptimizer%7B%3C%3AUnion%7BGradientMethod%2C%20Adam%2C%20AdamWithEuclideanDecay%2C%20MomentumMethod%2C%20ScalarMomentAdam%7D%2C%20%3C%3AGeometricOptimizers.OptimizerCache%2C%20%3C%3AOptimizerState%7BT%7D%7D%2C%20Union%7BAbstractArray%7BT%7D%2C%20NetworkParameters%7BT%7D%7D%7D%7D%20where%20T"
+[[jl.method]]
+name = "GeometricOptimizers.orthonormal_columns-Tuple{Any}"
+uri = "api/#GeometricOptimizers.orthonormal_columns-Tuple%7BAny%7D"
+[[jl.method]]
 name = "GeometricOptimizers.refresh_latest_gradient!-Tuple{GeometricOptimizers.OptimizerCache, Gradient}"
 uri = "api/#GeometricOptimizers.refresh_latest_gradient%21-Tuple%7BGeometricOptimizers.OptimizerCache%2C%20Gradient%7D"
 [[jl.method]]
@@ -236,33 +287,48 @@ uri = "api/#GeometricOptimizers.restart%21-Tuple%7BOptimizerState%7D"
 name = "GeometricOptimizers.retraction-Tuple{AbstractRetraction, AbstractArray}"
 uri = "api/#GeometricOptimizers.retraction-Tuple%7BAbstractRetraction%2C%20AbstractArray%7D"
 [[jl.method]]
+name = "GeometricOptimizers.retraction_differential!-Tuple{Any, Any, AbstractRetraction, Any, Any}"
+uri = "api/#GeometricOptimizers.retraction_differential%21-Tuple%7BAny%2C%20Any%2C%20AbstractRetraction%2C%20Any%2C%20Any%7D"
+[[jl.method]]
 name = "GeometricOptimizers.retraction_differential-Tuple{AbstractRetraction, Any, Any}"
 uri = "api/#GeometricOptimizers.retraction_differential-Tuple%7BAbstractRetraction%2C%20Any%2C%20Any%7D"
 [[jl.method]]
+name = "GeometricOptimizers.retraction_matrix!-Union{Tuple{T}, Tuple{GeometricOptimizers.RetractionWorkspace{T, AT} where AT<:AbstractMatrix{T}, AbstractRetraction, AbstractLieAlgHorMatrix{T}}} where T"
+uri = "api/#GeometricOptimizers.retraction_matrix%21-Union%7BTuple%7BT%7D%2C%20Tuple%7BGeometricOptimizers.RetractionWorkspace%7BT%2C%20AT%7D%20where%20AT%3C%3AAbstractMatrix%7BT%7D%2C%20AbstractRetraction%2C%20AbstractLieAlgHorMatrix%7BT%7D%7D%7D%20where%20T"
+[[jl.method]]
 name = "GeometricOptimizers.rgrad-Tuple{GrassmannManifold, AbstractMatrix}"
 uri = "api/#GeometricOptimizers.rgrad-Tuple%7BGrassmannManifold%2C%20AbstractMatrix%7D"
 [[jl.method]]
 name = "GeometricOptimizers.rgrad-Tuple{StiefelManifold, AbstractMatrix}"
 uri = "api/#GeometricOptimizers.rgrad-Tuple%7BStiefelManifold%2C%20AbstractMatrix%7D"
 [[jl.method]]
-name = "GeometricOptimizers.rhs-Tuple{GeometricOptimizers.BFGSCache}"
-uri = "api/#GeometricOptimizers.rhs-Tuple%7BGeometricOptimizers.BFGSCache%7D"
-[[jl.method]]
-name = "GeometricOptimizers.rhs-Tuple{GeometricOptimizers.DFPCache}"
-uri = "api/#GeometricOptimizers.rhs-Tuple%7BGeometricOptimizers.DFPCache%7D"
+name = "GeometricOptimizers.rgrad-Tuple{SymplecticStiefelManifold, AbstractMatrix}"
+uri = "api/#GeometricOptimizers.rgrad-Tuple%7BSymplecticStiefelManifold%2C%20AbstractMatrix%7D"
 [[jl.method]]
 name = "GeometricOptimizers.rhs-Tuple{GeometricOptimizers.NewtonOptimizerCache}"
 uri = "api/#GeometricOptimizers.rhs-Tuple%7BGeometricOptimizers.NewtonOptimizerCache%7D"
 [[jl.method]]
+name = "GeometricOptimizers.rhs-Tuple{GeometricOptimizers.QuasiNewtonCache}"
+uri = "api/#GeometricOptimizers.rhs-Tuple%7BGeometricOptimizers.QuasiNewtonCache%7D"
+[[jl.method]]
 name = "GeometricOptimizers.solution_scale-Tuple{AbstractVecOrMat}"
 uri = "api/#GeometricOptimizers.solution_scale-Tuple%7BAbstractVecOrMat%7D"
 [[jl.method]]
-name = "GeometricOptimizers.solver_step!-Union{Tuple{MT}, Tuple{T}, Tuple{OptimizerSolution{T}, OptimizerState{T}, Optimizer{T, MT, OBJ, GT, HT} where {OBJ<:(OptimizerProblem{T}), GT<:Gradient{T}, HT<:Hessian{T}}}} where {T, MT}"
-uri = "api/#GeometricOptimizers.solver_step%21-Union%7BTuple%7BMT%7D%2C%20Tuple%7BT%7D%2C%20Tuple%7BOptimizerSolution%7BT%7D%2C%20OptimizerState%7BT%7D%2C%20Optimizer%7BT%2C%20MT%2C%20OBJ%2C%20GT%2C%20HT%7D%20where%20%7BOBJ%3C%3A%28OptimizerProblem%7BT%7D%29%2C%20GT%3C%3AGradient%7BT%7D%2C%20HT%3C%3AHessian%7BT%7D%7D%7D%7D%20where%20%7BT%2C%20MT%7D"
+name = "GeometricOptimizers.solver_step!-Union{Tuple{T}, Tuple{OptimizerSolution{T}, OptimizerState{T}, Optimizer{T, ALG, OBJ, GT, HT} where {ALG<:OptimizerMethod, OBJ<:(OptimizerProblem{T}), GT<:Gradient{T}, HT<:Hessian{T}}}} where T"
+uri = "api/#GeometricOptimizers.solver_step%21-Union%7BTuple%7BT%7D%2C%20Tuple%7BOptimizerSolution%7BT%7D%2C%20OptimizerState%7BT%7D%2C%20Optimizer%7BT%2C%20ALG%2C%20OBJ%2C%20GT%2C%20HT%7D%20where%20%7BALG%3C%3AOptimizerMethod%2C%20OBJ%3C%3A%28OptimizerProblem%7BT%7D%29%2C%20GT%3C%3AGradient%7BT%7D%2C%20HT%3C%3AHessian%7BT%7D%7D%7D%7D%20where%20T"
+[[jl.method]]
+name = "GeometricOptimizers.sr!-Union{Tuple{AbstractMatrix{T}}, Tuple{T}} where T"
+uri = "api/#GeometricOptimizers.sr%21-Union%7BTuple%7BAbstractMatrix%7BT%7D%7D%2C%20Tuple%7BT%7D%7D%20where%20T"
+[[jl.method]]
+name = "GeometricOptimizers.sr-Tuple{AbstractMatrix}"
+uri = "api/#GeometricOptimizers.sr-Tuple%7BAbstractMatrix%7D"
 [[jl.method]]
 name = "GeometricOptimizers.steepest_descent!-Tuple{GeometricOptimizers.OptimizerCache}"
 uri = "api/#GeometricOptimizers.steepest_descent%21-Tuple%7BGeometricOptimizers.OptimizerCache%7D"
 [[jl.method]]
+name = "GeometricOptimizers.step_observer-Tuple{Optimizer}"
+uri = "api/#GeometricOptimizers.step_observer-Tuple%7BOptimizer%7D"
+[[jl.method]]
 name = "GeometricOptimizers.step_size-Union{Tuple{T}, Tuple{DecayingStatic{T}, Integer}} where T"
 uri = "api/#GeometricOptimizers.step_size-Union%7BTuple%7BT%7D%2C%20Tuple%7BDecayingStatic%7BT%7D%2C%20Integer%7D%7D%20where%20T"
 [[jl.method]]
@@ -272,20 +338,29 @@ uri = "api/#GeometricOptimizers.step_%CE%B1max-Union%7BTuple%7BT%7D%2C%20Tuple%7
 name = "GeometricOptimizers.store_gradient!-Tuple{GeometricOptimizers.OptimizerCache, OptimizerState, Gradient, OptimizerSolution}"
 uri = "api/#GeometricOptimizers.store_gradient%21-Tuple%7BGeometricOptimizers.OptimizerCache%2C%20OptimizerState%2C%20Gradient%2C%20OptimizerSolution%7D"
 [[jl.method]]
+name = "GeometricOptimizers.symplectic_form-Tuple{AbstractVector, AbstractVector}"
+uri = "api/#GeometricOptimizers.symplectic_form-Tuple%7BAbstractVector%2C%20AbstractVector%7D"
+[[jl.method]]
+name = "GeometricOptimizers.symplectic_householder!-Union{Tuple{T}, Tuple{AbstractVector{T}, AbstractVector{T}}} where T"
+uri = "api/#GeometricOptimizers.symplectic_householder%21-Union%7BTuple%7BT%7D%2C%20Tuple%7BAbstractVector%7BT%7D%2C%20AbstractVector%7BT%7D%7D%7D%20where%20T"
+[[jl.method]]
+name = "GeometricOptimizers.symplectic_householder-Tuple{AbstractVector, AbstractVector}"
+uri = "api/#GeometricOptimizers.symplectic_householder-Tuple%7BAbstractVector%2C%20AbstractVector%7D"
+[[jl.method]]
+name = "GeometricOptimizers.symplectic_normalize-Tuple{AbstractVector, AbstractVector, AbstractMatrix}"
+uri = "api/#GeometricOptimizers.symplectic_normalize-Tuple%7BAbstractVector%2C%20AbstractVector%2C%20AbstractMatrix%7D"
+[[jl.method]]
 name = "GeometricOptimizers.trace-Tuple{GeometricOptimizers.OptimizerResult}"
 uri = "api/#GeometricOptimizers.trace-Tuple%7BGeometricOptimizers.OptimizerResult%7D"
 [[jl.method]]
-name = "GeometricOptimizers.trial_iterate!-Tuple{GeometricOptimizers.OptimizerCache, Any, Any, Any}"
-uri = "api/#GeometricOptimizers.trial_iterate%21-Tuple%7BGeometricOptimizers.OptimizerCache%2C%20Any%2C%20Any%2C%20Any%7D"
-[[jl.method]]
-name = "GeometricOptimizers.trial_slope-Tuple{Gradient, GeometricOptimizers.OptimizerCache, Any, Any}"
-uri = "api/#GeometricOptimizers.trial_slope-Tuple%7BGradient%2C%20GeometricOptimizers.OptimizerCache%2C%20Any%2C%20Any%7D"
+name = "GeometricOptimizers.trial_slope-Tuple{Gradient, GeometricOptimizers.OptimizerCache, Vararg{Any, 4}}"
+uri = "api/#GeometricOptimizers.trial_slope-Tuple%7BGradient%2C%20GeometricOptimizers.OptimizerCache%2C%20Vararg%7BAny%2C%204%7D%7D"
 [[jl.method]]
-name = "GeometricOptimizers.unit_matrix-Union{Tuple{T}, Tuple{Any, Type{T}, Integer}} where T"
-uri = "api/#GeometricOptimizers.unit_matrix-Union%7BTuple%7BT%7D%2C%20Tuple%7BAny%2C%20Type%7BT%7D%2C%20Integer%7D%7D%20where%20T"
+name = "GeometricOptimizers.unit_matrix-Union{Tuple{T}, Tuple{KernelAbstractions.Backend, Type{T}, Integer}} where T"
+uri = "api/#GeometricOptimizers.unit_matrix-Union%7BTuple%7BT%7D%2C%20Tuple%7BKernelAbstractions.Backend%2C%20Type%7BT%7D%2C%20Integer%7D%7D%20where%20T"
 [[jl.method]]
-name = "GeometricOptimizers.update_section!-Union{Tuple{MT}, Tuple{T}, Tuple{GlobalSection{T, MT, λT} where λT<:Union{Nothing, AbstractArray{T}}, AbstractLieAlgHorMatrix{T}, Any}} where {T, MT<:Manifold{T}}"
-uri = "api/#GeometricOptimizers.update_section%21-Union%7BTuple%7BMT%7D%2C%20Tuple%7BT%7D%2C%20Tuple%7BGlobalSection%7BT%2C%20MT%2C%20%CE%BBT%7D%20where%20%CE%BBT%3C%3AUnion%7BNothing%2C%20AbstractArray%7BT%7D%7D%2C%20AbstractLieAlgHorMatrix%7BT%7D%2C%20Any%7D%7D%20where%20%7BT%2C%20MT%3C%3AManifold%7BT%7D%7D"
+name = "GeometricOptimizers.update_section!-Union{Tuple{MT}, Tuple{T}, Tuple{GlobalSection{T, MT, λT} where λT<:Union{Nothing, AbstractArray{T}}, GlobalSection{T, MT, λT} where λT<:Union{Nothing, AbstractArray{T}}, AbstractLieAlgHorMatrix{T}, Any}, Tuple{GlobalSection{T, MT, λT} where λT<:Union{Nothing, AbstractArray{T}}, GlobalSection{T, MT, λT} where λT<:Union{Nothing, AbstractArray{T}}, AbstractLieAlgHorMatrix{T}, Any, Any}} where {T, MT<:Manifold{T}}"
+uri = "api/#GeometricOptimizers.update_section%21-Union%7BTuple%7BMT%7D%2C%20Tuple%7BT%7D%2C%20Tuple%7BGlobalSection%7BT%2C%20MT%2C%20%CE%BBT%7D%20where%20%CE%BBT%3C%3AUnion%7BNothing%2C%20AbstractArray%7BT%7D%7D%2C%20GlobalSection%7BT%2C%20MT%2C%20%CE%BBT%7D%20where%20%CE%BBT%3C%3AUnion%7BNothing%2C%20AbstractArray%7BT%7D%7D%2C%20AbstractLieAlgHorMatrix%7BT%7D%2C%20Any%7D%2C%20Tuple%7BGlobalSection%7BT%2C%20MT%2C%20%CE%BBT%7D%20where%20%CE%BBT%3C%3AUnion%7BNothing%2C%20AbstractArray%7BT%7D%7D%2C%20GlobalSection%7BT%2C%20MT%2C%20%CE%BBT%7D%20where%20%CE%BBT%3C%3AUnion%7BNothing%2C%20AbstractArray%7BT%7D%7D%2C%20AbstractLieAlgHorMatrix%7BT%7D%2C%20Any%2C%20Any%7D%7D%20where%20%7BT%2C%20MT%3C%3AManifold%7BT%7D%7D"
 [[jl.method]]
 name = "GeometricOptimizers.value-Tuple{GeometricOptimizers.AbstractOptimizerProblem, OptimizerSolution}"
 uri = "api/#GeometricOptimizers.value-Tuple%7BGeometricOptimizers.AbstractOptimizerProblem%2C%20OptimizerSolution%7D"
@@ -296,6 +371,9 @@ uri = "api/#GeometricOptimizers.%CE%A9-Union%7BTuple%7BT%7D%2C%20Tuple%7BGrassma
 name = "GeometricOptimizers.Ω-Union{Tuple{T}, Tuple{StiefelManifold{T, AT} where AT<:AbstractMatrix{T}, AbstractMatrix{T}}} where T"
 uri = "api/#GeometricOptimizers.%CE%A9-Union%7BTuple%7BT%7D%2C%20Tuple%7BStiefelManifold%7BT%2C%20AT%7D%20where%20AT%3C%3AAbstractMatrix%7BT%7D%2C%20AbstractMatrix%7BT%7D%7D%7D%20where%20T"
 [[jl.method]]
+name = "GeometricOptimizers.𝔄!-Tuple{Any, AbstractMatrix, GeometricOptimizers.AbstractExponentialAlgorithm}"
+uri = "api/#GeometricOptimizers.%F0%9D%94%84%21-Tuple%7BAny%2C%20AbstractMatrix%2C%20GeometricOptimizers.AbstractExponentialAlgorithm%7D"
+[[jl.method]]
 name = "GeometricOptimizers.𝔄-Tuple{AbstractMatrix, AbstractMatrix}"
 uri = "api/#GeometricOptimizers.%F0%9D%94%84-Tuple%7BAbstractMatrix%2C%20AbstractMatrix%7D"
 [[jl.method]]
@@ -305,24 +383,22 @@ uri = "api/#GeometricOptimizers.%F0%9D%94%84-Tuple%7BAbstractMatrix%2C%20Geometr
 name = "GeometricOptimizers.𝔄-Tuple{AbstractMatrix}"
 uri = "api/#GeometricOptimizers.%F0%9D%94%84-Tuple%7BAbstractMatrix%7D"
 [[jl.method]]
-name = "SimpleSolvers.direction-Tuple{GeometricOptimizers.BFGSCache}"
-uri = "api/#SimpleSolvers.direction-Tuple%7BGeometricOptimizers.BFGSCache%7D"
-[[jl.method]]
-name = "SimpleSolvers.direction-Tuple{GeometricOptimizers.DFPCache}"
-uri = "api/#SimpleSolvers.direction-Tuple%7BGeometricOptimizers.DFPCache%7D"
-[[jl.method]]
 name = "SimpleSolvers.direction-Tuple{GeometricOptimizers.NewtonOptimizerCache}"
 uri = "api/#SimpleSolvers.direction-Tuple%7BGeometricOptimizers.NewtonOptimizerCache%7D"
 [[jl.method]]
-name = "SimpleSolvers.linesearch_problem-Union{Tuple{T}, Tuple{OptimizerProblem{T}, Gradient, GeometricOptimizers.OptimizerCache{T}, AbstractRetraction}} where T"
-uri = "api/#SimpleSolvers.linesearch_problem-Union%7BTuple%7BT%7D%2C%20Tuple%7BOptimizerProblem%7BT%7D%2C%20Gradient%2C%20GeometricOptimizers.OptimizerCache%7BT%7D%2C%20AbstractRetraction%7D%7D%20where%20T"
+name = "SimpleSolvers.direction-Tuple{GeometricOptimizers.QuasiNewtonCache}"
+uri = "api/#SimpleSolvers.direction-Tuple%7BGeometricOptimizers.QuasiNewtonCache%7D"
 [[jl.method]]
-name = "SimpleSolvers.outer!-Union{Tuple{T}, Tuple{AbstractMatrix{T}, AbstractLieAlgHorMatrix{T}, AbstractLieAlgHorMatrix{T}}} where T"
-uri = "api/#SimpleSolvers.outer%21-Union%7BTuple%7BT%7D%2C%20Tuple%7BAbstractMatrix%7BT%7D%2C%20AbstractLieAlgHorMatrix%7BT%7D%2C%20AbstractLieAlgHorMatrix%7BT%7D%7D%7D%20where%20T"
+name = "SimpleSolvers.linesearch_problem-Union{Tuple{T}, Tuple{OptimizerProblem{T}, Gradient, GeometricOptimizers.OptimizerCache{T}, AbstractRetraction}, Tuple{OptimizerProblem{T}, Gradient, GeometricOptimizers.OptimizerCache{T}, AbstractRetraction, Any}, Tuple{OptimizerProblem{T}, Gradient, GeometricOptimizers.OptimizerCache{T}, AbstractRetraction, Any, Any}} where T"
+uri = "api/#SimpleSolvers.linesearch_problem-Union%7BTuple%7BT%7D%2C%20Tuple%7BOptimizerProblem%7BT%7D%2C%20Gradient%2C%20GeometricOptimizers.OptimizerCache%7BT%7D%2C%20AbstractRetraction%7D%2C%20Tuple%7BOptimizerProblem%7BT%7D%2C%20Gradient%2C%20GeometricOptimizers.OptimizerCache%7BT%7D%2C%20AbstractRetraction%2C%20Any%7D%2C%20Tuple%7BOptimizerProblem%7BT%7D%2C%20Gradient%2C%20GeometricOptimizers.OptimizerCache%7BT%7D%2C%20AbstractRetraction%2C%20Any%2C%20Any%7D%7D%20where%20T"
 [[jl.method]]
 name = "SimpleSolvers.solve!-Union{Tuple{T}, Tuple{OptimizerSolution{T}, OptimizerState, Optimizer{T, ALG, OBJ, GT, HT} where {ALG<:OptimizerMethod, OBJ<:(OptimizerProblem{T}), GT<:Gradient{T}, HT<:Hessian{T}}}} where T"
 uri = "api/#SimpleSolvers.solve%21-Union%7BTuple%7BT%7D%2C%20Tuple%7BOptimizerSolution%7BT%7D%2C%20OptimizerState%2C%20Optimizer%7BT%2C%20ALG%2C%20OBJ%2C%20GT%2C%20HT%7D%20where%20%7BALG%3C%3AOptimizerMethod%2C%20OBJ%3C%3A%28OptimizerProblem%7BT%7D%29%2C%20GT%3C%3AGradient%7BT%7D%2C%20HT%3C%3AHessian%7BT%7D%7D%7D%7D%20where%20T"
 
+[[jl.module]]
+name = "GeometricOptimizers.GeometricOptimizers"
+uri = "api/#$"
+
 [[jl.type]]
 name = "GeometricOptimizers.AbstractExponentialAlgorithm"
 uri = "api/#$"
@@ -360,9 +436,6 @@ uri = "api/#$"
 name = "GeometricOptimizers.BFGS"
 uri = "api/#$"
 [[jl.type]]
-name = "GeometricOptimizers.BFGSCache"
-uri = "api/#$"
-[[jl.type]]
 name = "GeometricOptimizers.BFGSState"
 uri = "api/#$"
 [[jl.type]]
@@ -372,15 +445,18 @@ uri = "api/#$"
 name = "GeometricOptimizers.DFP"
 uri = "api/#$"
 [[jl.type]]
-name = "GeometricOptimizers.DFPCache"
-uri = "api/#$"
-[[jl.type]]
 name = "GeometricOptimizers.DFPState"
 uri = "api/#$"
 [[jl.type]]
 name = "GeometricOptimizers.DecayingStatic"
 uri = "api/#$"
 [[jl.type]]
+name = "GeometricOptimizers.EventLog"
+uri = "api/#$"
+[[jl.type]]
+name = "GeometricOptimizers.FirstOrderMethod"
+uri = "api/#$"
+[[jl.type]]
 name = "GeometricOptimizers.FirstOrderMethodWithState"
 uri = "api/#$"
 [[jl.type]]
@@ -405,18 +481,6 @@ uri = "api/#$"
 name = "GeometricOptimizers.GrassmannManifold"
 uri = "api/#$"
 [[jl.type]]
-name = "GeometricOptimizers.HessianBFGS"
-uri = "api/#$"
-[[jl.type]]
-name = "GeometricOptimizers.HessianDFP"
-uri = "api/#$"
-[[jl.type]]
-name = "GeometricOptimizers.IterativeHessian"
-uri = "api/#$"
-[[jl.type]]
-name = "GeometricOptimizers.LowerTriangular"
-uri = "api/#$"
-[[jl.type]]
 name = "GeometricOptimizers.Manifold"
 uri = "api/#$"
 [[jl.type]]
@@ -438,7 +502,13 @@ uri = "api/#$"
 name = "GeometricOptimizers.NewtonOptimizerCache"
 uri = "api/#$"
 [[jl.type]]
-name = "GeometricOptimizers.NewtonOptimizerState"
+name = "GeometricOptimizers.NewtonState"
+uri = "api/#$"
+[[jl.type]]
+name = "GeometricOptimizers.NoStepObserver"
+uri = "api/#$"
+[[jl.type]]
+name = "GeometricOptimizers.NoWorkspace"
 uri = "api/#$"
 [[jl.type]]
 name = "GeometricOptimizers.Optimizer"
@@ -468,12 +538,36 @@ uri = "api/#$"
 name = "GeometricOptimizers.OptimizerTraceEntry"
 uri = "api/#$"
 [[jl.type]]
+name = "GeometricOptimizers.OrthonormalizationFailure"
+uri = "api/#$"
+[[jl.type]]
+name = "GeometricOptimizers.PhaseTimer"
+uri = "api/#$"
+[[jl.type]]
+name = "GeometricOptimizers.PrecomputedGradient"
+uri = "api/#$"
+[[jl.type]]
 name = "GeometricOptimizers.ProjectedSkew"
 uri = "api/#$"
 [[jl.type]]
+name = "GeometricOptimizers.QuasiNewtonCache"
+uri = "api/#$"
+[[jl.type]]
 name = "GeometricOptimizers.QuasiNewtonOptimizerMethod"
 uri = "api/#$"
 [[jl.type]]
+name = "GeometricOptimizers.RetractionWorkspace"
+uri = "api/#$"
+[[jl.type]]
+name = "GeometricOptimizers.Rfac"
+uri = "api/#$"
+[[jl.type]]
+name = "GeometricOptimizers.RiemannianGradient"
+uri = "api/#$"
+[[jl.type]]
+name = "GeometricOptimizers.SR"
+uri = "api/#$"
+[[jl.type]]
 name = "GeometricOptimizers.ScalarMomentAdam"
 uri = "api/#$"
 [[jl.type]]
@@ -483,9 +577,15 @@ uri = "api/#$"
 name = "GeometricOptimizers.ScalarMomentAdamState"
 uri = "api/#$"
 [[jl.type]]
+name = "GeometricOptimizers.ScaledAlgorithm"
+uri = "api/#$"
+[[jl.type]]
 name = "GeometricOptimizers.ScaledSquaring"
 uri = "api/#$"
 [[jl.type]]
+name = "GeometricOptimizers.Sfac"
+uri = "api/#$"
+[[jl.type]]
 name = "GeometricOptimizers.SkewSymMatrix"
 uri = "api/#$"
 [[jl.type]]
@@ -498,13 +598,25 @@ uri = "api/#$"
 name = "GeometricOptimizers.StiefelProjection"
 uri = "api/#$"
 [[jl.type]]
+name = "GeometricOptimizers.StrictlyLowerTriangular"
+uri = "api/#$"
+[[jl.type]]
+name = "GeometricOptimizers.StrictlyUpperTriangular"
+uri = "api/#$"
+[[jl.type]]
 name = "GeometricOptimizers.SymmetricMatrix"
 uri = "api/#$"
 [[jl.type]]
+name = "GeometricOptimizers.SymplecticHouseholderDecom"
+uri = "api/#$"
+[[jl.type]]
+name = "GeometricOptimizers.SymplecticStiefelManifold"
+uri = "api/#$"
+[[jl.type]]
 name = "GeometricOptimizers.TaylorSeries"
 uri = "api/#$"
 [[jl.type]]
-name = "GeometricOptimizers.UpperTriangular"
+name = "GeometricOptimizers.TrainingOptimizer"
 uri = "api/#$"
 [[jl.type]]
 name = "GeometricOptimizers.VectorStorageMatrix"
@@ -515,6 +627,10 @@ dispname = "API"
 name = "api"
 uri = "api/"
 [[std.doc]]
+dispname = "Exponential Algorithms"
+name = "exponential_algorithms"
+uri = "exponential_algorithms/"
+[[std.doc]]
 dispname = "Global Tangent Spaces"
 name = "global_tangent_spaces"
 uri = "global_tangent_spaces/"
@@ -563,6 +679,10 @@ dispname = "Riemannian Manifolds"
 name = "manifolds/riemannian_manifolds"
 uri = "manifolds/riemannian_manifolds/"
 [[std.doc]]
+dispname = "Observing Optimizer Phases"
+name = "observers"
+uri = "observers/"
+[[std.doc]]
 dispname = "Optimizer Methods"
 name = "optimizer_methods"
 uri = "optimizer_methods/"
@@ -583,6 +703,10 @@ dispname = "Symmetric, Skew-Symmetric and Triangular Matrices"
 name = "special_matrices"
 uri = "special_matrices/"
 [[std.doc]]
+dispname = "The Training Step"
+name = "training_step"
+uri = "training_step/"
+[[std.doc]]
 dispname = "Weight Decay on Manifolds"
 name = "weight_decay"
 uri = "weight_decay/"
@@ -600,6 +724,30 @@ dispname = "(Topological) Vector Spaces"
 name = "(Topological)-Vector-Spaces"
 uri = "manifolds/metric_and_vector_spaces/#%28Topological%29-Vector-Spaces"
 [[std.label]]
+dispname = "1. Direct Taylor series"
+name = "1.-Direct-Taylor-series"
+uri = "exponential_algorithms/#$"
+[[std.label]]
+dispname = "2. Scaling and modified squaring"
+name = "2.-Scaling-and-modified-squaring"
+uri = "exponential_algorithms/#$"
+[[std.label]]
+dispname = "3. Padé approximation"
+name = "3.-Padé-approximation"
+uri = "exponential_algorithms/#3.-Pad%C3%A9-approximation"
+[[std.label]]
+dispname = "4. AugmentedPade"
+name = "4.-AugmentedPade"
+uri = "exponential_algorithms/#$"
+[[std.label]]
+dispname = "5. ProjectedSkew"
+name = "5.-ProjectedSkew"
+uri = "exponential_algorithms/#$"
+[[std.label]]
+dispname = "6. Numerical comparison"
+name = "6.-Numerical-comparison"
+uri = "exponential_algorithms/#$"
+[[std.label]]
 dispname = "A bounded merit is not a bound on the step"
 name = "A-bounded-merit-is-not-a-bound-on-the-step"
 uri = "linesearch_on_manifolds/#$"
@@ -618,14 +766,19 @@ uri = "api/#$"
 [[std.label]]
 dispname = "Adding one"
 name = "Adding-one"
-uri = "retractions/#$"
+uri = "exponential_algorithms/#$"
 [[std.label]]
 dispname = "Agreeing with the exponential"
 name = "Agreeing-with-the-exponential"
-uri = "retractions/#$"
+uri = "exponential_algorithms/#$"
 [[std.label]]
-name = "AugmentedPade"
-uri = "retractions/#$"
+dispname = "Applying the denominator without a matrix solve"
+name = "Applying-the-denominator-without-a-matrix-solve"
+uri = "exponential_algorithms/#$"
+[[std.label]]
+dispname = "Arithmetic and broadcasting on a device"
+name = "Arithmetic-and-broadcasting-on-a-device"
+uri = "special_matrices/#$"
 [[std.label]]
 dispname = "Basic Concepts from General Topology"
 name = "Basic-Concepts-from-General-Topology"
@@ -645,7 +798,7 @@ uri = "retractions/#$"
 [[std.label]]
 dispname = "Choosing one"
 name = "Choosing-one"
-uri = "retractions/#$"
+uri = "exponential_algorithms/#$"
 [[std.label]]
 dispname = "Classical Retractions"
 name = "Classical-Retractions"
@@ -655,6 +808,9 @@ dispname = "Complete Metric Spaces"
 name = "Complete-Metric-Spaces"
 uri = "manifolds/metric_and_vector_spaces/#$"
 [[std.label]]
+name = "Coverage"
+uri = "observers/#$"
+[[std.label]]
 dispname = "Custom Matrices"
 name = "Custom-Matrices"
 uri = "special_matrices/#$"
@@ -663,16 +819,40 @@ dispname = "Decoupled weight decay"
 name = "Decoupled-weight-decay"
 uri = "weight_decay/#$"
 [[std.label]]
+dispname = "Deriving the coefficients for mathfrakA"
+name = "Deriving-the-coefficients-for-\\mathfrak{A}"
+uri = "exponential_algorithms/#Deriving-the-coefficients-for-%5Cmathfrak%7BA%7D"
+[[std.label]]
+dispname = "Element types"
+name = "Element-types"
+uri = "special_matrices/#$"
+[[std.label]]
 name = "Example"
 uri = "linesearch/#$"
 [[std.label]]
+dispname = "Example: exclusive time per phase"
+name = "Example:-exclusive-time-per-phase"
+uri = "observers/#Example%3A-exclusive-time-per-phase"
+[[std.label]]
+dispname = "Example: the events of one iteration"
+name = "Example:-the-events-of-one-iteration"
+uri = "observers/#Example%3A-the-events-of-one-iteration"
+[[std.label]]
+dispname = "Exponential Algorithms"
+name = "Exponential-Algorithms"
+uri = "exponential_algorithms/#$"
+[[std.label]]
 name = "Float32"
-uri = "retractions/#$"
+uri = "exponential_algorithms/#$"
 [[std.label]]
 dispname = "Foundational Theorems for Differential Manifolds"
 name = "Foundational-Theorems-for-Differential-Manifolds"
 uri = "manifolds/inverse_function_theorem/#$"
 [[std.label]]
+dispname = "From scalar Padé to matrix Padé"
+name = "From-scalar-Padé-to-matrix-Padé"
+uri = "exponential_algorithms/#From-scalar-Pad%C3%A9-to-matrix-Pad%C3%A9"
+[[std.label]]
 dispname = "Generalization to Homogeneous Spaces"
 name = "Generalization-to-Homogeneous-Spaces"
 uri = "manifold_optimizers/#$"
@@ -724,8 +904,9 @@ dispname = "Linesearches on Manifolds"
 name = "Linesearches-on-Manifolds"
 uri = "linesearch_on_manifolds/#$"
 [[std.label]]
-name = "NativePade"
-uri = "retractions/#$"
+dispname = "Observing the Phases of an Optimizer Step"
+name = "Observing-the-Phases-of-an-Optimizer-Step"
+uri = "observers/#$"
 [[std.label]]
 dispname = "Optimization on Homogeneous Spaces"
 name = "Optimization-on-Homogeneous-Spaces"
@@ -743,9 +924,6 @@ dispname = "Preserve symmetry in the DFP inverse Hessian"
 name = "Preserve-symmetry-in-the-DFP-inverse-Hessian"
 uri = "linesearch_on_manifolds/#$"
 [[std.label]]
-name = "ProjectedSkew"
-uri = "retractions/#$"
-[[std.label]]
 dispname = "Quasi-Newton caches on manifold solutions"
 name = "Quasi-Newton-caches-on-manifold-solutions"
 uri = "linesearch_on_manifolds/#$"
@@ -772,8 +950,9 @@ dispname = "Sample Random Matrices"
 name = "Sample-Random-Matrices"
 uri = "special_matrices/#$"
 [[std.label]]
-name = "ScaledSquaring"
-uri = "retractions/#$"
+dispname = "Sensitivity to the threshold θ"
+name = "Sensitivity-to-the-threshold-θ"
+uri = "exponential_algorithms/#Sensitivity-to-the-threshold-%CE%B8"
 [[std.label]]
 dispname = "Standard Neural Network Optimizers"
 name = "Standard-Neural-Network-Optimizers"
@@ -781,7 +960,7 @@ uri = "optimizer_methods/#$"
 [[std.label]]
 dispname = "Staying on the manifold"
 name = "Staying-on-the-manifold"
-uri = "retractions/#$"
+uri = "exponential_algorithms/#$"
 [[std.label]]
 dispname = "Symmetric, Skew-Symmetric and Triangular Matrices"
 name = "Symmetric,-Skew-Symmetric-and-Triangular-Matrices"
@@ -791,9 +970,6 @@ dispname = "Tangent Spaces"
 name = "Tangent-Spaces"
 uri = "manifolds/manifolds/#$"
 [[std.label]]
-name = "TaylorSeries"
-uri = "retractions/#$"
-[[std.label]]
 dispname = "The Adam Optimizer"
 name = "The-Adam-Optimizer"
 uri = "optimizer_methods/#$"
@@ -878,21 +1054,45 @@ dispname = "The Tangent Bundle"
 name = "The-Tangent-Bundle"
 uri = "manifolds/manifolds/#$"
 [[std.label]]
+dispname = "The Training Step"
+name = "The-Training-Step"
+uri = "training_step/#$"
+[[std.label]]
 dispname = "The algorithm"
 name = "The-algorithm"
 uri = "manifold_optimizers/#$"
 [[std.label]]
+dispname = "The bound, and why it takes two packages"
+name = "The-bound,-and-why-it-takes-two-packages"
+uri = "linesearch_on_manifolds/#The-bound%2C-and-why-it-takes-two-packages"
+[[std.label]]
+dispname = "The complete NativePade algorithm"
+name = "The-complete-NativePade-algorithm"
+uri = "exponential_algorithms/#$"
+[[std.label]]
+dispname = "The complete ScaledSquaring algorithm"
+name = "The-complete-ScaledSquaring-algorithm"
+uri = "exponential_algorithms/#$"
+[[std.label]]
+dispname = "The doubling identity for mathfrakA"
+name = "The-doubling-identity-for-\\mathfrak{A}"
+uri = "exponential_algorithms/#The-doubling-identity-for-%5Cmathfrak%7BA%7D"
+[[std.label]]
 dispname = "The exponential needs an algorithm"
 name = "The-exponential-needs-an-algorithm"
-uri = "retractions/#$"
+uri = "exponential_algorithms/#$"
 [[std.label]]
 dispname = "The extended retraction"
 name = "The-extended-retraction"
 uri = "manifold_optimizers/#$"
 [[std.label]]
-dispname = "The fix, and why it takes two packages"
-name = "The-fix,-and-why-it-takes-two-packages"
-uri = "linesearch_on_manifolds/#The-fix%2C-and-why-it-takes-two-packages"
+dispname = "The fix: observed boundaries and ready-made recorders"
+name = "The-fix:-observed-boundaries-and-ready-made-recorders"
+uri = "observers/#The-fix%3A-observed-boundaries-and-ready-made-recorders"
+[[std.label]]
+dispname = "The framework, and the two algorithms in it"
+name = "The-framework,-and-the-two-algorithms-in-it"
+uri = "exponential_algorithms/#The-framework%2C-and-the-two-algorithms-in-it"
 [[std.label]]
 dispname = "The generator of the trial curve turns with the step"
 name = "The-generator-of-the-trial-curve-turns-with-the-step"
@@ -906,6 +1106,22 @@ dispname = "The lift to the global tangent space"
 name = "The-lift-to-the-global-tangent-space"
 uri = "manifold_optimizers/#$"
 [[std.label]]
+dispname = "The methods and the step size"
+name = "The-methods-and-the-step-size"
+uri = "training_step/#$"
+[[std.label]]
+dispname = "The name"
+name = "The-name"
+uri = "weight_decay/#$"
+[[std.label]]
+dispname = "The norm of X is quadratic in the norm of barB"
+name = "The-norm-of-X-is-*quadratic*-in-the-norm-of-\\bar{B}"
+uri = "exponential_algorithms/#The-norm-of-X-is-%2Aquadratic%2A-in-the-norm-of-%5Cbar%7BB%7D"
+[[std.label]]
+dispname = "The notation this page uses"
+name = "The-notation-this-page-uses"
+uri = "exponential_algorithms/#$"
+[[std.label]]
 dispname = "The numerical experiment"
 name = "The-numerical-experiment"
 uri = "manifold_optimizers/#$"
@@ -914,10 +1130,26 @@ dispname = "The optimizer framework, step by step"
 name = "The-optimizer-framework,-step-by-step"
 uri = "manifold_optimizers/#The-optimizer-framework%2C-step-by-step"
 [[std.label]]
+dispname = "The phases"
+name = "The-phases"
+uri = "observers/#$"
+[[std.label]]
 dispname = "The problem: Adam has no coordinate-free formulation"
 name = "The-problem:-Adam-has-no-coordinate-free-formulation"
 uri = "manifold_optimizers/#The-problem%3A-Adam-has-no-coordinate-free-formulation"
 [[std.label]]
+dispname = "The problem: one step, several costs"
+name = "The-problem:-one-step,-several-costs"
+uri = "observers/#The-problem%3A-one-step%2C-several-costs"
+[[std.label]]
+dispname = "The protocol"
+name = "The-protocol"
+uri = "observers/#$"
+[[std.label]]
+dispname = "The reduction"
+name = "The-reduction"
+uri = "exponential_algorithms/#$"
+[[std.label]]
 dispname = "The retractions on the two manifolds"
 name = "The-retractions-on-the-two-manifolds"
 uri = "retractions/#$"
@@ -930,9 +1162,9 @@ dispname = "The source's projection is global_rep"
 name = "The-source's-projection-*is*-global_rep"
 uri = "optimizer_methods/#The-source%27s-projection-%2Ais%2A-global_rep"
 [[std.label]]
-dispname = "The threshold θ needs no tuning"
-name = "The-threshold-θ-needs-no-tuning"
-uri = "retractions/#The-threshold-%CE%B8-needs-no-tuning"
+dispname = "The threshold theta"
+name = "The-threshold-\\theta"
+uri = "exponential_algorithms/#The-threshold-%5Ctheta"
 [[std.label]]
 dispname = "Time-Dependent Vector Fields"
 name = "Time-Dependent-Vector-Fields"
@@ -944,7 +1176,7 @@ uri = "weight_decay/#$"
 [[std.label]]
 dispname = "Using them"
 name = "Using-them"
-uri = "retractions/#$"
+uri = "exponential_algorithms/#$"
 [[std.label]]
 dispname = "Vector Fields"
 name = "Vector-Fields"
@@ -962,17 +1194,21 @@ dispname = "What a retraction is"
 name = "What-a-retraction-is"
 uri = "retractions/#$"
 [[std.label]]
+dispname = "What else observers are good for"
+name = "What-else-observers-are-good-for"
+uri = "observers/#$"
+[[std.label]]
+dispname = "What scaling buys the Padé kernel"
+name = "What-scaling-buys-the-Padé-kernel"
+uri = "exponential_algorithms/#What-scaling-buys-the-Pad%C3%A9-kernel"
+[[std.label]]
 dispname = "What the argument does and does not depend on"
 name = "What-the-argument-does-and-does-not-depend-on"
 uri = "weight_decay/#$"
 [[std.label]]
 dispname = "What they cost"
 name = "What-they-cost"
-uri = "retractions/#$"
-[[std.label]]
-dispname = "What they cost and how accurate they are"
-name = "What-they-cost-and-how-accurate-they-are"
-uri = "retractions/#$"
+uri = "exponential_algorithms/#$"
 [[std.label]]
 dispname = "Where GeometricMachineLearning's method belongs"
 name = "Where-GeometricMachineLearning's-method-belongs"
@@ -982,6 +1218,10 @@ dispname = "Where a retraction sits in the algorithm"
 name = "Where-a-retraction-sits-in-the-algorithm"
 uri = "retractions/#$"
 [[std.label]]
+dispname = "Where a sampled array lands, and what it holds"
+name = "Where-a-sampled-array-lands,-and-what-it-holds"
+uri = "special_matrices/#Where-a-sampled-array-lands%2C-and-what-it-holds"
+[[std.label]]
 dispname = "Where each symbol lives"
 name = "Where-each-symbol-lives"
 uri = "optimizer_methods/#$"
@@ -990,14 +1230,34 @@ dispname = "Which line search"
 name = "Which-line-search"
 uri = "weight_decay/#$"
 [[std.label]]
+dispname = "Why that combination is bad for a power series"
+name = "Why-that-combination-is-bad-for-a-power-series"
+uri = "exponential_algorithms/#$"
+[[std.label]]
 dispname = "Why the decay vanishes"
 name = "Why-the-decay-vanishes"
 uri = "weight_decay/#$"
 [[std.label]]
-dispname = "Why the name is not AdamW"
-name = "Why-the-name-is-not-AdamW"
-uri = "weight_decay/#$"
+dispname = "Why the optimizer does not time every step automatically"
+name = "Why-the-optimizer-does-not-time-every-step-automatically"
+uri = "observers/#$"
+[[std.label]]
+dispname = "Why the reduced argument is the hard case"
+name = "Why-the-reduced-argument-is-the-hard-case"
+uri = "exponential_algorithms/#$"
 [[std.label]]
 dispname = "Why the storage matters here"
 name = "Why-the-storage-matters-here"
 uri = "special_matrices/#$"
+[[std.label]]
+dispname = "Why the training step is not solve!"
+name = "Why-the-training-step-is-not-solve!"
+uri = "training_step/#Why-the-training-step-is-not-solve%21"
+[[std.label]]
+dispname = "X and barB have the same nonzero eigenvalues"
+name = "X-and-\\bar{B}-have-the-same-nonzero-eigenvalues"
+uri = "exponential_algorithms/#X-and-%5Cbar%7BB%7D-have-the-same-nonzero-eigenvalues"
+[[std.label]]
+dispname = "What a large theta costs NativePade"
+name = "native-pade-large-theta"
+uri = "exponential_algorithms/#$"
diff --git a/docs/make.jl b/docs/make.jl
index 7309c5aa3..879235b12 100644
--- a/docs/make.jl
+++ b/docs/make.jl
@@ -38,7 +38,7 @@ Downloads.download(
 #
 #     julia --project=docs -e 'using DocInventories; DocInventories.save(
 #         "docs/inventories/GeometricOptimizers.toml",
-#         Inventory("https://juliagni.github.io/GeometricOptimizers.jl/v0.5.0/objects.inv";
+#         Inventory("https://juliagni.github.io/GeometricOptimizers.jl/v0.9.0/objects.inv";
 #                   root_url = "https://juliagni.github.io/GeometricOptimizers.jl/stable/"))'
 #
 # `save` is not exported — it has to be qualified. `root_url` is not written to the TOML either; it
@@ -387,8 +387,7 @@ makedocs(;
     # skips *sub*modules — and (ii) pull GeometricOptimizers' own docstrings in, whose internal
     # `@ref`s resolve in its namespace and point at bindings this manual does not document.
     # Names owned by GeometricOptimizers are referred to as plain code with a link to its manual.
-    modules = [
-        GeometricMachineLearning, Base.get_extension(GeometricMachineLearning, :HDF5Ext)],
+    modules = [GeometricMachineLearning],
     authors = "Michael Kraus, Benedikt Brantner",
     repo = "https://github.com/JuliaGNI/GeometricMachineLearning.jl/blob/{commit}{path}#L{line}",
     sitename = "GeometricMachineLearning.jl",
diff --git a/docs/src/architectures/abstract_neural_networks.md b/docs/src/architectures/abstract_neural_networks.md
index 15a7cc19e..c81a6d623 100644
--- a/docs/src/architectures/abstract_neural_networks.md
+++ b/docs/src/architectures/abstract_neural_networks.md
@@ -47,11 +47,4 @@ and we see that it consists of two layers: a [`GradientLayerQ`](@ref) and a [`Gr
 
 ## Saving and Loading
 
-`GeometricMachineLearning` adds HDF5-backed `save` and `load` methods for `NeuralNetwork` to the generics `NeuralNetworkParameters` defines. Writing and reading the parameter set itself belongs to that package; the structured parameter types (`StiefelManifold`, `SymmetricMatrix`, `SkewSymMatrix`, …) come back as themselves because `GeometricOptimizers`, which owns them, registers how each is rebuilt. Passing a prototype parameter set to `load` rebuilds against it and needs no registration at all.
-
-```@docs
-save(::HDF5.H5DataStore, ::NeuralNetwork)
-save(::AbstractString, ::NeuralNetwork)
-load(::Type{NeuralNetwork}, ::HDF5.H5DataStore, ::AbstractNeuralNetworks.Architecture)
-load(::Type{NeuralNetwork}, ::AbstractString, ::AbstractNeuralNetworks.Architecture)
-```
+With HDF5 loaded, `AbstractNeuralNetworks` has `save` and `load` methods for `NeuralNetwork`: `save(filename, nn)` writes the parameters of `nn`, and `load(NeuralNetwork, filename, arch)` reads them back into a network for `arch` on the CPU; `changebackend(backend, nn)` then moves it to a device. Writing and reading the parameter set itself belongs to `NeuralNetworkParameters`; the structured parameter types (`StiefelManifold`, `SymmetricMatrix`, `SkewSymMatrix`, …) come back as themselves because `GeometricOptimizers`, which owns them, registers how each is rebuilt. Passing a prototype parameter set to `load`, as `load(NeuralNetwork, filename, arch, prototype)`, rebuilds against it and needs no registration at all.
diff --git a/docs/src/arrays/tensors.md b/docs/src/arrays/tensors.md
index a7e0d1eac..0a8d65a9d 100644
--- a/docs/src/arrays/tensors.md
+++ b/docs/src/arrays/tensors.md
@@ -1,6 +1,6 @@
 # Tensors in `GeometricMachineLearning`
 
-We typically store training data as *tensors with three axes* in `GeometricMachineLearning`. This allows for a parallel computation of matrix products, also for the special arrays such as [`LowerTriangular`](@extref GeometricOptimizers GeometricOptimizers.LowerTriangular), [`UpperTriangular`](@extref GeometricOptimizers GeometricOptimizers.UpperTriangular), [`SymmetricMatrix`](@extref GeometricOptimizers GeometricOptimizers.SymmetricMatrix) and [`SkewSymMatrix`](@extref GeometricOptimizers GeometricOptimizers.SkewSymMatrix) and objects of [`Manifold`](@extref GeometricOptimizers GeometricOptimizers.Manifold) type such as the [`StiefelManifold`](@extref GeometricOptimizers GeometricOptimizers.StiefelManifold). 
+We typically store training data as *tensors with three axes* in `GeometricMachineLearning`. This allows for a parallel computation of matrix products, also for the special arrays such as [`StrictlyLowerTriangular`](@extref GeometricOptimizers GeometricOptimizers.StrictlyLowerTriangular), [`StrictlyUpperTriangular`](@extref GeometricOptimizers GeometricOptimizers.StrictlyUpperTriangular), [`SymmetricMatrix`](@extref GeometricOptimizers GeometricOptimizers.SymmetricMatrix) and [`SkewSymMatrix`](@extref GeometricOptimizers GeometricOptimizers.SkewSymMatrix) and objects of [`Manifold`](@extref GeometricOptimizers GeometricOptimizers.Manifold) type such as the [`StiefelManifold`](@extref GeometricOptimizers GeometricOptimizers.StiefelManifold). 
 
 ## Library Functions
 
@@ -10,8 +10,8 @@ GeometricMachineLearning.tensor_mat_mul!(::AbstractArray{<:Number, 3}, ::Abstrac
 GeometricMachineLearning.tensor_mat_mul!(::AbstractArray{<:Number, 3}, ::AbstractArray{<:Number, 3}, ::SymmetricMatrix)
 GeometricMachineLearning.mat_tensor_mul(::AbstractMatrix, ::AbstractArray{<:Number, 3})
 GeometricMachineLearning.mat_tensor_mul!(::AbstractArray{<:Number, 3}, ::AbstractMatrix, ::AbstractArray{<:Number, 3})
-GeometricMachineLearning.mat_tensor_mul!(::AbstractArray{<:Number, 3}, ::LowerTriangular, ::AbstractArray{<:Number, 3})
-GeometricMachineLearning.mat_tensor_mul!(::AbstractArray{<:Number, 3}, ::UpperTriangular, ::AbstractArray{<:Number, 3})
+GeometricMachineLearning.mat_tensor_mul!(::AbstractArray{<:Number, 3}, ::StrictlyLowerTriangular, ::AbstractArray{<:Number, 3})
+GeometricMachineLearning.mat_tensor_mul!(::AbstractArray{<:Number, 3}, ::StrictlyUpperTriangular, ::AbstractArray{<:Number, 3})
 GeometricMachineLearning.mat_tensor_mul!(::AbstractArray{<:Number, 3}, ::SkewSymMatrix, ::AbstractArray{<:Number, 3})
 GeometricMachineLearning.mat_tensor_mul!(::AbstractArray{<:Number, 3}, ::SymmetricMatrix, ::AbstractArray{<:Number, 3})
 ```
\ No newline at end of file
diff --git a/docs/src/index.md b/docs/src/index.md
index 00d3dc8d6..9c8b9bdd5 100644
--- a/docs/src/index.md
+++ b/docs/src/index.md
@@ -52,13 +52,13 @@ Many layers have been adapted in order to be used for problems in scientific mac
 
 `GeometricMachineLearning` allocates and computes through `KernelAbstractions.jl` [churavy2020kernel](@cite), so its layers and architectures are written against any backend that package supports: `CUDA.jl` [besard2018juliagpu](@cite), `AMDGPU.jl`, `Metal.jl` [besard2022metal](@cite) and `oneAPI.jl` [besard2022one](@cite).
 
-**Little of that is tested.** `test/devices/metal.jl` runs two checks on an Apple GPU, in the `metal` test group, which the `Metal` job and the `macOS-latest` job of CI run: the products of a `PoissonTensor` with a wrapped device array, and the `tensor_mat_mul!` kernel. A green test matrix says nothing about the rest of the GPU path. What is written below was measured by hand on an Apple M4 Max through `Metal.jl`, in `Float32`, because Apple GPUs have no `Float64` at all, and against `GeometricOptimizers` 0.8.0.
+**Little of that is tested.** `test/devices/metal.jl` runs two checks on an Apple GPU, in the `metal` test group, which the `Metal` job and the `macOS-latest` job of CI run: the products of a `PoissonTensor` with a wrapped device array, and the `tensor_mat_mul!` kernel. A green test matrix says nothing about the rest of the GPU path. What is written below was measured by hand on an Apple M4 Max through `Metal.jl` 1.11.1, in `Float32`, because Apple GPUs have no `Float64` at all, and against `GeometricOptimizers` 0.9.0.
 
-Everything tried on that device ran. Constructed, and then applied to both a matrix and a 3-tensor: `GSympNet`, `LASympNet`, `StandardTransformerIntegrator`, `LinearSymplecticTransformer`, `SymplecticTransformer` at both its default `transformer_dim` and an upscaling one, `VolumePreservingFeedForward`, `VolumePreservingTransformer`, `SymplecticAutoencoder`, `PSDArch` and `Transformer(…; Stiefel = true)`. `ClassificationTransformer` was constructed but not applied, because its input is an image. The tensor kernels and `map_to_cpu` run, and so does training with `Optimizer`, which returns a `Float32` history and leaves the parameters on the device — for `PSDArch` that includes the manifold weights, which stay a `StiefelManifold` over an `MtlMatrix`.
+Everything tried on that device ran. Constructed, applied to both a matrix and a 3-tensor, and copied back with `map_to_cpu`: `GSympNet`, `LASympNet`, `StandardTransformerIntegrator`, `LinearSymplecticTransformer`, `SymplecticTransformer` at both its default `transformer_dim` and an upscaling one, `VolumePreservingFeedForward`, `VolumePreservingTransformer`, `SymplecticAutoencoder`, `PSDArch` and `Transformer(…; Stiefel = true)`. The product of a `StrictlyLowerTriangular` matrix with a 3-tensor runs. `train!` trains a `GSympNet` for two epochs with its data in `MtlArray`s. `ClassificationTransformer` was not tried.
 
 Two of those results rest on `GeometricOptimizers` rather than on anything here.
 
-- `VolumePreservingFeedForward` and `VolumePreservingTransformer` applied to a **matrix** multiply a `LowerTriangular` or `UpperTriangular` weight by that matrix. `GeometricOptimizers` supplies a `KernelAbstractions` kernel for that product, so the call does not fall through to a generic multiply that reads one entry at a time, which `GPUArraysCore` refuses.
+- `VolumePreservingFeedForward` and `VolumePreservingTransformer` applied to a **matrix** multiply a `StrictlyLowerTriangular` or `StrictlyUpperTriangular` weight by that matrix. `GeometricOptimizers` supplies a `KernelAbstractions` kernel for that product, so the call does not fall through to a generic multiply that reads one entry at a time, which `GPUArraysCore` refuses.
 - `StiefelLayer`, `GrassmannLayer` and `PSDLayer`, and so every architecture that holds one of them, orthonormalize their weight at **construction** through `GeometricOptimizers.orthonormal_columns`. That is CholeskyQR2 — matrix products and triangular solves only — so it runs wherever the draw was allocated. `LinearAlgebra.qr!` is a host factorization and `Metal.jl` implements no `qr` for an `MtlArray`, so a device needs the other one.
 
 ## Tutorials 
diff --git a/docs/src/introduction.md b/docs/src/introduction.md
index 7a9178980..4e32b842c 100644
--- a/docs/src/introduction.md
+++ b/docs/src/introduction.md
@@ -19,10 +19,10 @@ Closely linked to the research presented here is the development of a software p
 [^0]: This document was produced with `GeometricMachineLearning` `v0.3`. It may be that the interface will slightly change in future versions, but efforts will be made to keep these changes as small as possible.
 
 ```@docs; canonical = false
-optimization_step!
+train!
 ```
 
-So the docstring shows the name of the type or method, in most cases how to call it and then gives some information explaining what it does and potentially hyperlinks to other similar docstrings ([`Optimizer`](@ref) in this case); all of this information is indented by a tab. Docstrings may include other information under subheaders **Arguments** (showing the arguments the method can be supplied with), **Examples** (giving more detailed examples (including results) of how to use the method) and **Implementation** (giving details on how the method is implemented). When we reference a docstring it is always printed in blue (e.g. [`optimization_step!`](@ref)), indicating a hyperlink. In addition there is an *index of docstrings* showing all docstrings in chronological order with the associated page number.
+So the docstring shows the name of the type or method, in most cases how to call it and then gives some information explaining what it does and potentially hyperlinks to other similar docstrings ([`DataLoader`](@ref) in this case); all of this information is indented by a tab. Docstrings may include other information under subheaders **Arguments** (showing the arguments the method can be supplied with), **Examples** (giving more detailed examples (including results) of how to use the method) and **Implementation** (giving details on how the method is implemented). When we reference a docstring it is always printed in blue (e.g. [`train!`](@ref)), indicating a hyperlink. In addition there is an *index of docstrings* showing all docstrings in chronological order with the associated page number.
 
 Similar to **Library Functions**, which is included in most sections, almost every chapter concludes with a section **Chapter Summary** and an additional section **References** that shows further related reading material. The **Chapter Summary** recaps the important aspects of the corresponding chapter, states again what is new (this may be mathematical or software aspects) and gives information to what other parts of the dissertation the contents of the present chapter are relevant.
 
diff --git a/docs/src/layers/volume_preserving_feedforward.md b/docs/src/layers/volume_preserving_feedforward.md
index d85cac9a8..4c463da50 100644
--- a/docs/src/layers/volume_preserving_feedforward.md
+++ b/docs/src/layers/volume_preserving_feedforward.md
@@ -7,7 +7,7 @@ The *volume-preserving feedforward layers* in `GeometricMachineLearning` are clo
 ```math
 \mathtt{VPFF}_{A, b}: x \mapsto x + \sigma(Ax + b),
 ```
-where ``\sigma`` is a nonlinearity, ``A`` is the weight and ``b`` is the bias. The matrix ``A`` is either a [`LowerTriangular`](@extref GeometricOptimizers GeometricOptimizers.LowerTriangular) matrix ``L`` or an [`UpperTriangular`](@extref GeometricOptimizers GeometricOptimizers.UpperTriangular) matrix ``U``. We demonstrate volume-preservation of these layers by considering the case ``A = L``. The matrix looks as follows:
+where ``\sigma`` is a nonlinearity, ``A`` is the weight and ``b`` is the bias. The matrix ``A`` is either a [`StrictlyLowerTriangular`](@extref GeometricOptimizers GeometricOptimizers.StrictlyLowerTriangular) matrix ``L`` or an [`StrictlyUpperTriangular`](@extref GeometricOptimizers GeometricOptimizers.StrictlyUpperTriangular) matrix ``U``. We demonstrate volume-preservation of these layers by considering the case ``A = L``. The matrix looks as follows:
 
 ```math 
 L = \begin{pmatrix}
@@ -28,7 +28,7 @@ J = \nabla\mathtt{VPFF}_{L, b} = \begin{pmatrix}
      b_{n1} & \cdots & b_{n(n-1)}      & 1 
 \end{pmatrix},
 ```
-and the determinant of ``J`` is 1, i.e. the map is volume-preserving. A similar statement holds if the matrix ``A`` is [`UpperTriangular`](@extref GeometricOptimizers GeometricOptimizers.UpperTriangular) instead of [`LowerTriangular`](@extref GeometricOptimizers GeometricOptimizers.LowerTriangular).
+and the determinant of ``J`` is 1, i.e. the map is volume-preserving. A similar statement holds if the matrix ``A`` is [`StrictlyUpperTriangular`](@extref GeometricOptimizers GeometricOptimizers.StrictlyUpperTriangular) instead of [`StrictlyLowerTriangular`](@extref GeometricOptimizers GeometricOptimizers.StrictlyLowerTriangular).
 
 ## Library Functions 
 
diff --git a/docs/src/optimizers/optimizer.md b/docs/src/optimizers/optimizer.md
index ca0b095ee..52f019aa4 100644
--- a/docs/src/optimizers/optimizer.md
+++ b/docs/src/optimizers/optimizer.md
@@ -1,30 +1,31 @@
-# The `Optimizer` in `GeometricMachineLearning`
+# Training a Neural Network in `GeometricMachineLearning`
 
 The general framework for optimization on homogeneous spaces — the Riemannian gradient, the lift to
 the global tangent space ``\mathfrak{g}^\mathrm{hor}``, the optimizer cache, the retraction and the
 global section — belongs to `GeometricOptimizers` and is described in its documentation, under
 [Optimization on Homogeneous Spaces](@extref GeometricOptimizers :doc:`manifold_optimizers`) and
-[Retractions](@extref GeometricOptimizers :doc:`retractions`).
+[Retractions](@extref GeometricOptimizers :doc:`retractions`). The training step itself is
+`GeometricOptimizers`' too: its
+[`TrainingOptimizer`](@extref GeometricOptimizers GeometricOptimizers.TrainingOptimizer), which
+`GeometricMachineLearning` re-exports, steps a whole parameter set with a gradient the caller
+computed, and applies the right update to each leaf — a retraction for a weight on a manifold,
+ordinary arithmetic for a Euclidean one. It is built on the parameters of a network, `nn.params`.
 
-What `GeometricMachineLearning` adds is the part that is about *neural networks*: walking the
-parameter tree of a `NeuralNetwork`, applying the right update to each leaf — a retraction
-for a weight on a manifold, ordinary arithmetic for a Euclidean one — and driving that from a data
-loader over epochs and batches.
+What `GeometricMachineLearning` adds is the part that is about *neural networks*: [`train!`](@ref),
+which draws the minibatches of a data loader over epochs, computes the gradient of the loss on each
+with automatic differentiation or a symbolic pullback, and takes one step per minibatch.
 
-The gradient comes from automatic differentiation on one batch at a time, so there is no objective
-function to hand a line search; the step size is a property of the [`Optimizer`](@ref) instead. It is
-either a number, or a `GeometricOptimizers.DecayingStatic` schedule:
+The gradient comes from one minibatch at a time, so there is no objective function to hand a line
+search; the step size is a number or a `GeometricOptimizers.DecayingStatic` schedule:
 
 ```julia
-opt = Optimizer(Adam(Float32), nn; step_size = 1e-3)
-opt = Optimizer(nn; AdamOptimizerWithDecay(n_epochs, Float32)...)
+opt = TrainingOptimizer(nn.params; algorithm = Adam(), linesearch = 1e-3)
+opt = TrainingOptimizer(nn.params; AdamOptimizerWithDecay(n_steps)...)
+history = train!(nn, opt, dl, Batch(batch_size), n_epochs)
 ```
 
 ## Library Functions
 
 ```@docs
-Optimizer
-optimize_for_one_epoch!
-optimization_step!
-GeometricMachineLearning._as_go_solution
+train!
 ```
diff --git a/docs/src/tutorials/adjusting_the_loss_function.md b/docs/src/tutorials/adjusting_the_loss_function.md
index 7acf859dd..8431f933a 100644
--- a/docs/src/tutorials/adjusting_the_loss_function.md
+++ b/docs/src/tutorials/adjusting_the_loss_function.md
@@ -26,11 +26,11 @@ data = DataLoader(sol; suppress_info = true)
 nn = NeuralNetwork(GSympNet(2))
 
 # train the network
-o = Optimizer(Adam(), nn)
+o = TrainingOptimizer(nn.params; algorithm = Adam())
 batch = Batch(32)
 n_epochs = 30
 loss = FeedForwardLoss()
-loss_array = o(nn, data, batch, n_epochs, loss; show_progress = false)
+loss_array = train!(nn, o, data, batch, n_epochs; loss, show_progress = false)
 print(loss_array[end])
 ```
 
@@ -71,7 +71,8 @@ And we train the same network with this new loss:
 ```@example change_loss
 loss = CustomLoss()
 nn_custom = NeuralNetwork(GSympNet(2))
-loss_array = o(nn_custom, data, batch, n_epochs, loss; show_progress = false)
+o_custom = TrainingOptimizer(nn_custom.params; algorithm = Adam())
+loss_array = train!(nn_custom, o_custom, data, batch, n_epochs; loss, show_progress = false)
 print(loss_array[end])
 ```
 
diff --git a/docs/src/tutorials/grassmann_layer.md b/docs/src/tutorials/grassmann_layer.md
index 1b1acbcd8..60d9cc345 100644
--- a/docs/src/tutorials/grassmann_layer.md
+++ b/docs/src/tutorials/grassmann_layer.md
@@ -105,13 +105,6 @@ nn = NeuralNetwork(model, CPU(), Float64)
 nothing # hide
 ```
 
-We then *lift* the neural network parameters via `GlobalSection`.
-
-```@example rosenbrock
-λY = GlobalSection(params(nn))
-nothing # hide
-```
-
 As the cost function ``c`` for the Wasserstein loss[^3] we simply use:
 
 [^3]: For each Wasserstein loss we need to define such a cost function. For this particular choice of ``c`` the Wasserstein distance corresponds to a ``W_2`` loss.
@@ -269,7 +262,7 @@ function compute_gradient(ps::NetworkParameters)
 end
 
 # note the very high value for the learning rate
-optimizer = Optimizer(Adam(), nn; step_size = 1e-1)
+optimizer = TrainingOptimizer(nn.params; algorithm = Adam(), linesearch = 1e-1)
 
 nothing # hide
 ```
@@ -278,13 +271,14 @@ So we use `Zygote` [Zygote.jl-2018](@cite) to compute ``\nabla_\theta\mathcal{NN
 
 ```@example rosenbrock
 using GeometricMachineLearning: params # hide
+using GeometricOptimizers: optimization_step!
 # note the small number of training steps
 const training_steps = 80
 loss_array = zeros(training_steps)
 for i in 1:training_steps
     val, dp = compute_gradient(params(nn))
     loss_array[i] = val
-    optimization_step!(optimizer, λY, params(nn), dp)
+    optimization_step!(params(nn), optimizer, dp)
 end
 ```
 
diff --git a/docs/src/tutorials/hamiltonian_neural_network.md b/docs/src/tutorials/hamiltonian_neural_network.md
index 4fe0b0914..136af1c75 100644
--- a/docs/src/tutorials/hamiltonian_neural_network.md
+++ b/docs/src/tutorials/hamiltonian_neural_network.md
@@ -42,8 +42,8 @@ We can now train the network
 ```@example hnn
 batch = Batch(10)
 n_epochs = 100
-o = Optimizer(Adam(Float64), hnn)
-loss_array = o(hnn, dl, batch, n_epochs, loss)
+o = TrainingOptimizer(hnn.params; algorithm = Adam())
+loss_array = train!(hnn, o, dl, batch, n_epochs; loss)
 using CairoMakie # hide
 lines(loss_array) # hide
 ```
@@ -84,7 +84,7 @@ nothing # hide
 We can now train the network:
 
 ```@example hnn
-o_pairs = Optimizer(Adam(Float64), hnn_pairs)
-loss_array_pairs = o_pairs(hnn_pairs, dl_pairs, batch, n_epochs, loss_pairs)
+o_pairs = TrainingOptimizer(hnn_pairs.params; algorithm = Adam())
+loss_array_pairs = train!(hnn_pairs, o_pairs, dl_pairs, batch, n_epochs; loss = loss_pairs)
 lines(loss_array_pairs) # hide
 ```
diff --git a/docs/src/tutorials/linear_symplectic_transformer.md b/docs/src/tutorials/linear_symplectic_transformer.md
index 7d38af6d6..ee6d6aca4 100644
--- a/docs/src/tutorials/linear_symplectic_transformer.md
+++ b/docs/src/tutorials/linear_symplectic_transformer.md
@@ -63,18 +63,18 @@ nn_standard = NeuralNetwork(arch_standard)
 nn_symplectic = NeuralNetwork(arch_symplectic)
 nn_sympnet = NeuralNetwork(arch_sympnet)
 
-o_pairing = AdamOptimizerWithDecay(n_epochs, Float64)
+o_pairing = AdamOptimizerWithDecay(n_epochs)
 
-o_standard = Optimizer(nn_standard; o_pairing...)
-o_symplectic = Optimizer(nn_symplectic; o_pairing...)
-o_sympnet = Optimizer(nn_sympnet; o_pairing...)
+o_standard = TrainingOptimizer(nn_standard.params; o_pairing...)
+o_symplectic = TrainingOptimizer(nn_symplectic.params; o_pairing...)
+o_sympnet = TrainingOptimizer(nn_sympnet.params; o_pairing...)
 
 batch = Batch(batch_size, seq_length)
 batch2 = Batch(batch_size)
 
-loss_array_standard = o_standard(nn_standard, dl, batch, n_epochs; show_progress = false)
-loss_array_symplectic = o_symplectic(nn_symplectic, dl, batch, n_epochs; show_progress = false)
-loss_array_sympnet = o_sympnet(nn_sympnet, dl, batch2, n_epochs; show_progress = false)
+loss_array_standard = train!(nn_standard, o_standard, dl, batch, n_epochs; show_progress = false)
+loss_array_symplectic = train!(nn_symplectic, o_symplectic, dl, batch, n_epochs; show_progress = false)
+loss_array_sympnet = train!(nn_sympnet, o_sympnet, dl, batch2, n_epochs; show_progress = false)
 
 nothing # hide
 ```
diff --git a/docs/src/tutorials/matrix_softmax.md b/docs/src/tutorials/matrix_softmax.md
index e336de88e..977166d6c 100644
--- a/docs/src/tutorials/matrix_softmax.md
+++ b/docs/src/tutorials/matrix_softmax.md
@@ -89,13 +89,13 @@ Training is done with the `Adam`:
 ```@example softmax_comparison
 o_method = Adam()
 
-o1 = Optimizer(o_method, nn1)
-o2 = Optimizer(o_method, nn2)
+o1 = TrainingOptimizer(nn1.params; algorithm = o_method)
+o2 = TrainingOptimizer(nn2.params; algorithm = o_method)
 
 batch = Batch(batch_size, seq_length)
 
-loss_array1 = o1(nn1, dl, batch, n_epochs; show_progress = false)
-loss_array2 = o2(nn2, dl, batch, n_epochs; show_progress = false)
+loss_array1 = train!(nn1, o1, dl, batch, n_epochs; show_progress = false)
+loss_array2 = train!(nn2, o2, dl, batch, n_epochs; show_progress = false)
 ```
 
 ```@setup softmax_comparison
diff --git a/docs/src/tutorials/optimizer_comparison.md b/docs/src/tutorials/optimizer_comparison.md
index aeabe7868..a59b1a4f3 100644
--- a/docs/src/tutorials/optimizer_comparison.md
+++ b/docs/src/tutorials/optimizer_comparison.md
@@ -27,14 +27,14 @@ init_cont = Tuple(init_con)
 mred = RGBf(214 / 256, 39 / 256, 40 / 256) # hide
 scatter!(ax, init_cont..., f(init_cont...); color = mred, marker = :star5)
 
-# `Optimizer` takes a `NeuralNetwork` or a `NetworkParameters`, so the bare `NamedTuple` of
-# weights has to be wrapped in the latter.
+# `TrainingOptimizer` takes a `NetworkParameters`, so the bare `NamedTuple` of weights has to be
+# wrapped in one.
 weights = NetworkParameters((xy = init_con, ))
 η = 1e-3
 method1 = GradientMethod()
 method2 = Adam()
-optimizer1 = Optimizer(method1, weights; step_size = η)
-optimizer2 = Optimizer(method2, weights; step_size = η)
+optimizer1 = TrainingOptimizer(weights; algorithm = method1, linesearch = η)
+optimizer2 = TrainingOptimizer(weights; algorithm = method2, linesearch = η)
 
 fig
 ```
diff --git a/docs/src/tutorials/softmax_comparison.md b/docs/src/tutorials/softmax_comparison.md
index a18c70467..a6ae48e32 100644
--- a/docs/src/tutorials/softmax_comparison.md
+++ b/docs/src/tutorials/softmax_comparison.md
@@ -74,13 +74,13 @@ nn2 = NeuralNetwork(arch2)
 
 o_method = Adam()
 
-o1 = Optimizer(o_method, nn1)
-o2 = Optimizer(o_method, nn2)
+o1 = TrainingOptimizer(nn1.params; algorithm = o_method)
+o2 = TrainingOptimizer(nn2.params; algorithm = o_method)
 
 batch = Batch(batch_size, seq_length)
 
-loss_array1 = o1(nn1, dl, batch, n_epochs; show_progress = false)
-loss_array2 = o2(nn2, dl, batch, n_epochs; show_progress = false)
+loss_array1 = train!(nn1, o1, dl, batch, n_epochs; show_progress = false)
+loss_array2 = train!(nn2, o2, dl, batch, n_epochs; show_progress = false)
 
 nothing # hide
 ```
diff --git a/docs/src/tutorials/symplectic_autoencoder.md b/docs/src/tutorials/symplectic_autoencoder.md
index ab958083c..7e240ecd9 100644
--- a/docs/src/tutorials/symplectic_autoencoder.md
+++ b/docs/src/tutorials/symplectic_autoencoder.md
@@ -15,7 +15,7 @@ Main.remark(raw"As with any neural network we have to make the following choices
 " * Main.indentation * raw"5. choose a *number of epochs*,
 " * Main.indentation * raw"where points 1 and 3 depend on a variable number of hyperparameters.")
 ```
-For the symplectic autoencoder point 1 is done by calling [`SymplecticAutoencoder`](@ref), point 2 is done by calling `NeuralNetwork`, point 3 is done by calling [`Optimizer`](@ref) and point 4 is done by calling [`Batch`](@ref).
+For the symplectic autoencoder point 1 is done by calling [`SymplecticAutoencoder`](@ref), point 2 is done by calling `NeuralNetwork`, point 3 is done by calling [`TrainingOptimizer`](@extref GeometricOptimizers GeometricOptimizers.TrainingOptimizer) and point 4 is done by calling [`Batch`](@ref).
 
 ## The system
 
@@ -111,7 +111,7 @@ sae_arch = SymplecticAutoencoder(dl_cpu.input_dim, reduced_dim; n_encoder_blocks
 nothing # hide
 ```
 
-Training a neural network is usually done by calling an instance of [`Optimizer`](@ref) in `GeometricMachineLearning`. [`PSDArch`](@ref) however can be solved directly by using singular value decomposition and this is done by calling [`solve!`](@ref):  
+Training a neural network is usually done by calling [`train!`](@ref) with a [`TrainingOptimizer`](@extref GeometricOptimizers GeometricOptimizers.TrainingOptimizer) in `GeometricMachineLearning`. [`PSDArch`](@ref) however can be solved directly by using singular value decomposition and this is done by calling [`solve!`](@ref):  
 
 ```@example toda_lattice
 psd_nn_cpu = NeuralNetwork(psd_arch, CPU(), eltype(dl_cpu))
@@ -129,7 +129,7 @@ Main.remark(raw"The training blocks on this page are in plain `julia` fences, so
 " * Main.indentation * raw"instead is the *committed result* of those runs, loaded from the `.h5` files beside this page,
 " * Main.indentation * raw"so the figures below are produced from the trained networks rather than from a token
 " * Main.indentation * raw"re-training. The code paths themselves are covered by the test suite -- `ReducedLoss` through
-" * Main.indentation * raw"the `Optimizer` functor by `test/integration/reduced_loss_optimization.jl` -- and reproduced at
+" * Main.indentation * raw"`train!` by `test/integration/reduced_loss_optimization.jl` -- and reproduced at
 " * Main.indentation * raw"full size by `scripts/reproduction/symplectic_autoencoders/`, which CI runs at a smoke size
 " * Main.indentation * raw"on every pull request.")
 ```
@@ -145,10 +145,10 @@ dl = DataLoader(dl_cpu, backend, Float32)
 
 
 sae_nn_gpu = NeuralNetwork(sae_arch, CUDADevice(), Float32)
-o = Optimizer(sae_nn_gpu; AdamOptimizerWithDecay(integrator_train_epochs, Float32)...)
+o = TrainingOptimizer(sae_nn_gpu.params; AdamOptimizerWithDecay(integrator_train_epochs)...)
 
 # train the network
-o(sae_nn_gpu, dl, Batch(batch_size), n_epochs)
+train!(sae_nn_gpu, o, dl, Batch(batch_size), n_epochs)
 ```
 
 After training we map the network parameters to cpu:
@@ -282,7 +282,7 @@ integrator_nn = NeuralNetwork(integrator_architecture, backend)
 
 integrator_pairing = AdamOptimizerWithDecay(integrator_train_epochs)
 
-o_integrator = Optimizer(integrator_nn; integrator_pairing...)
+o_integrator = TrainingOptimizer(integrator_nn.params; integrator_pairing...)
 
 dl = dl_cpu # hide
 # map from autoencoder type to integrator type
@@ -294,11 +294,12 @@ nothing # hide
 ```julia
 loss = GeometricMachineLearning.ReducedLoss(encoder(sae_nn_gpu), decoder(sae_nn_gpu))
 
-train_integrator_loss = o_integrator(   integrator_nn, 
-                                        dl_integration, 
-                                        integrator_batch, 
-                                        integrator_train_epochs, 
-                                        loss)
+train_integrator_loss = train!( integrator_nn, 
+                                o_integrator, 
+                                dl_integration, 
+                                integrator_batch, 
+                                integrator_train_epochs; 
+                                loss)
 ```
 
 We can now evaluate the solution:
@@ -476,7 +477,7 @@ const integrator_architecture2 = StandardTransformerIntegrator(reduced_dim2;
                                                                             upscaling_activation = tanh)
 integrator_nn2 = NeuralNetwork(integrator_architecture2, backend)
 const integrator_pairing2 = AdamOptimizerWithDecay(integrator_train_epochs)
-const o_integrator2 = Optimizer(integrator_nn2; integrator_pairing2...)
+const o_integrator2 = TrainingOptimizer(integrator_nn2.params; integrator_pairing2...)
 
 loss2 = GeometricMachineLearning.ReducedLoss(encoder(psd_nn2), decoder(psd_nn2))
 nothing # hide
@@ -484,7 +485,7 @@ nothing # hide
 
 For training we leave `dl_integration`, `integrator_batch` and `integrator_train_epochs` unchanged:
 ```julia
-train_integrator_loss2 = o_integrator(integrator_nn2, dl_integration, integrator_batch, integrator_train_epochs, loss2)
+train_integrator_loss2 = train!(integrator_nn2, o_integrator2, dl_integration, integrator_batch, integrator_train_epochs; loss = loss2)
 ```
 
 ```@setup toda_lattice
diff --git a/docs/src/tutorials/symplectic_transformer.md b/docs/src/tutorials/symplectic_transformer.md
index 75fcf178a..a97ae7cdc 100644
--- a/docs/src/tutorials/symplectic_transformer.md
+++ b/docs/src/tutorials/symplectic_transformer.md
@@ -63,18 +63,18 @@ nn_standard = NeuralNetwork(arch_standard)
 nn_symplectic = NeuralNetwork(arch_symplectic)
 nn_sympnet = NeuralNetwork(arch_sympnet)
 
-o_pairing = AdamOptimizerWithDecay(n_epochs, Float64)
+o_pairing = AdamOptimizerWithDecay(n_epochs)
 
-o_standard = Optimizer(nn_standard; o_pairing...)
-o_symplectic = Optimizer(nn_symplectic; o_pairing...)
-o_sympnet = Optimizer(nn_sympnet; o_pairing...)
+o_standard = TrainingOptimizer(nn_standard.params; o_pairing...)
+o_symplectic = TrainingOptimizer(nn_symplectic.params; o_pairing...)
+o_sympnet = TrainingOptimizer(nn_sympnet.params; o_pairing...)
 
 batch = Batch(batch_size, seq_length)
 batch2 = Batch(batch_size)
 
-loss_array_standard = o_standard(nn_standard, dl, batch, n_epochs; show_progress = false)
-loss_array_symplectic = o_symplectic(nn_symplectic, dl, batch, n_epochs; show_progress = false)
-loss_array_sympnet = o_sympnet(nn_sympnet, dl, batch2, n_epochs; show_progress = false)
+loss_array_standard = train!(nn_standard, o_standard, dl, batch, n_epochs; show_progress = false)
+loss_array_symplectic = train!(nn_symplectic, o_symplectic, dl, batch, n_epochs; show_progress = false)
+loss_array_sympnet = train!(nn_sympnet, o_sympnet, dl, batch2, n_epochs; show_progress = false)
 
 nothing # hide
 ```
diff --git a/docs/src/tutorials/sympnet_tutorial.md b/docs/src/tutorials/sympnet_tutorial.md
index 602f58a2c..4678e3f71 100644
--- a/docs/src/tutorials/sympnet_tutorial.md
+++ b/docs/src/tutorials/sympnet_tutorial.md
@@ -100,10 +100,10 @@ We have to define an [optimizer](@extref GeometricOptimizers Standard-Neural-Net
 
 ```@example sympnet
 # set up optimizer; for this we first need to specify the optimization method
-opt_method = Adam(type)
-# we then call the optimizer struct which allocates the cache
-la_opt = Optimizer(opt_method, la_nn)
-g_opt = Optimizer(opt_method, g_nn)
+opt_method = Adam()
+# we then build the optimizer, which allocates the cache in the element type of the network
+la_opt = TrainingOptimizer(la_nn.params; algorithm = opt_method)
+g_opt = TrainingOptimizer(g_nn.params; algorithm = opt_method)
 
 nothing # hide
 ```
@@ -120,8 +120,8 @@ batch = Batch(batch_size)
 const nepochs = 100
 
 # perform training (returns array that contains the total loss for each training step)
-g_loss_array = g_opt(g_nn, dl, batch, nepochs; show_progress = false)
-la_loss_array = la_opt(la_nn, dl, batch, nepochs; show_progress = false)
+g_loss_array = train!(g_nn, g_opt, dl, batch, nepochs; show_progress = false)
+la_loss_array = train!(la_nn, la_opt, dl, batch, nepochs; show_progress = false)
 nothing # hide
 ```
 
@@ -242,9 +242,9 @@ parameterlength(rn_nn)
 We now train the network ``\ldots``
 
 ```@example sympnet
-rn_opt = Optimizer(opt_method, rn_nn)
+rn_opt = TrainingOptimizer(rn_nn.params; algorithm = opt_method)
 
-rn_loss_array = rn_opt(rn_nn, dl, batch, nepochs; show_progress = false)
+rn_loss_array = train!(rn_nn, rn_opt, dl, batch, nepochs; show_progress = false)
 nothing # hide
 ```
 
diff --git a/docs/src/tutorials/volume_preserving_attention.md b/docs/src/tutorials/volume_preserving_attention.md
index a4474faec..2414e8701 100644
--- a/docs/src/tutorials/volume_preserving_attention.md
+++ b/docs/src/tutorials/volume_preserving_attention.md
@@ -117,9 +117,9 @@ Next we train the networks (here we pick a batch size of 30 and train for 1000 e
 
 ```@setup volume_preserving_attention
 function set_up_optimizers(nn_skew, nn_arb, nn_comp)
-    o_skew = Optimizer(Adam(T), nn_skew)
-    o_arb  = Optimizer(Adam(T), nn_arb)
-    o_comp = Optimizer(Adam(T), nn_comp)
+    o_skew = TrainingOptimizer(nn_skew.params; algorithm = Adam())
+    o_arb  = TrainingOptimizer(nn_arb.params; algorithm = Adam())
+    o_comp = TrainingOptimizer(nn_comp.params; algorithm = Adam())
 
     o_skew, o_arb, o_comp
 end
@@ -134,9 +134,9 @@ const batch = Batch(batch_size, seq_length, prediction_window)
 const batch2 = Batch(batch_size)
 
 function train_networks!(nn_skew, nn_arb, nn_comp)
-    loss_array_skew = o_skew(nn_skew, dl, batch, n_epochs, TransformerLoss(batch); show_progress = false)
-    loss_array_arb  = o_arb( nn_arb,  dl, batch, n_epochs, TransformerLoss(batch); show_progress = false)
-    loss_array_comp = o_comp(nn_comp, dl, batch2, n_epochs, FeedForwardLoss(); show_progress = false)
+    loss_array_skew = train!(nn_skew, o_skew, dl, batch, n_epochs; loss = TransformerLoss(batch), show_progress = false)
+    loss_array_arb  = train!(nn_arb,  o_arb,  dl, batch, n_epochs; loss = TransformerLoss(batch), show_progress = false)
+    loss_array_comp = train!(nn_comp, o_comp, dl, batch2, n_epochs; loss = FeedForwardLoss(), show_progress = false)
 
     loss_array_skew, loss_array_arb, loss_array_comp
 end
diff --git a/docs/src/tutorials/volume_preserving_transformer_rigid_body.md b/docs/src/tutorials/volume_preserving_transformer_rigid_body.md
index 487c659c7..c3095f13e 100644
--- a/docs/src/tutorials/volume_preserving_transformer_rigid_body.md
+++ b/docs/src/tutorials/volume_preserving_transformer_rigid_body.md
@@ -199,17 +199,17 @@ const n_epochs = 500000
 const batch_size = 16384
 const feedforward_batch = Batch(batch_size)
 const transformer_batch = Batch(batch_size, seq_length, seq_length)
-const opt_pairing = AdamOptimizerWithDecay(n_epochs, T; η₁ = 1e-2, η₂ = 1e-6)
+const opt_pairing = AdamOptimizerWithDecay(n_epochs; η₁ = 1e-2, η₂ = 1e-6)
 
-o_vpff = Optimizer(nn_vpff; opt_pairing...)
-o_vpt = Optimizer(nn_vpt; opt_pairing...)
-o_st = Optimizer(nn_st; opt_pairing...)
+o_vpff = TrainingOptimizer(nn_vpff.params; opt_pairing...)
+o_vpt = TrainingOptimizer(nn_vpt.params; opt_pairing...)
+o_st = TrainingOptimizer(nn_st.params; opt_pairing...)
 nothing # hide
 ```
 ```julia
-o_vpff(nn_vpff, dl, feedforward_batch, n_epochs)
-o_vpt(nn_vpt, dl, transformer_batch, n_epochs)
-o_st(nn_st, dl, transformer_batch, n_epochs)
+train!(nn_vpff, o_vpff, dl, feedforward_batch, n_epochs)
+train!(nn_vpt, o_vpt, dl, transformer_batch, n_epochs)
+train!(nn_st, o_st, dl, transformer_batch, n_epochs)
 ```
 
 After the networks have been trained we map the parameters to cpu:
@@ -220,6 +220,10 @@ nn_vpff = mtc(nn_vpff)
 nn_vpt = mtc(nn_vpt)
 nn_st = mtc(nn_st)
 using HDF5 # hide
+# the two files store the type names `LowerTriangular` and `UpperTriangular` # hide
+const NNP = parentmodule(NetworkParameters) # hide
+NNP.register_parameter_type!("LowerTriangular", (S, md) -> StrictlyLowerTriangular(S.S, S.n)) # hide
+NNP.register_parameter_type!("UpperTriangular", (S, md) -> StrictlyUpperTriangular(S.S, S.n)) # hide
 nn_vpff = load(NeuralNetwork, "transformer_rigid_body_nn_vpff.h5", nn_vpff.architecture) # hide
 nn_vpt = load(NeuralNetwork, "transformer_rigid_body_nn_vpt.h5", nn_vpt.architecture) # hide
 nn_st = load(NeuralNetwork, "transformer_rigid_body_nn_st.h5", nn_st.architecture) # hide
diff --git a/ext/HDF5Ext.jl b/ext/HDF5Ext.jl
deleted file mode 100644
index 39912a134..000000000
--- a/ext/HDF5Ext.jl
+++ /dev/null
@@ -1,87 +0,0 @@
-module HDF5Ext
-
-using HDF5
-using GeometricMachineLearning
-import AbstractNeuralNetworks: NeuralNetworkBackend, Architecture
-# `save`, `load`, `params` and the parameter container are `NeuralNetworkParameters`' as of
-# `AbstractNeuralNetworks` 0.7, which only re-binds them; reach for them where they are defined.
-import NeuralNetworkParameters: NetworkParameters, params, save, load
-
-# ---------------------------------------------------------------------------
-# save / load — the entry points that dispatch on this package's `NeuralNetwork`.
-#
-# The traversal itself is not here. `NeuralNetworkParameters` walks the parameter set and records
-# each group's key order; `GeometricOptimizers` says through `freeparameters`/`rebuild` where each
-# structured matrix keeps its numbers, and registers the types so a file loads with no prototype.
-# ---------------------------------------------------------------------------
-
-"""
-    save(h5::HDF5.H5DataStore, nn::NeuralNetwork)
-
-Save the parameters of `nn` into an already-open HDF5 store.
-
-Extends `save` with a dispatch on `NeuralNetwork`. The parameters themselves are written by
-`NeuralNetworkParameters`, which tags each structured leaf with the type to rebuild it as and
-records the key order of every group.
-"""
-save(h5::HDF5.H5DataStore, nn::NeuralNetwork) = save(h5, params(nn))
-
-"""
-    save(filename::AbstractString, nn::NeuralNetwork)
-
-Convenience overload: open `filename` for writing, call [`save`](@ref) on the store, and return
-`filename`.
-"""
-function save(filename::AbstractString, nn::NeuralNetwork)
-    HDF5.h5open(filename, "w") do h5
-        save(h5, nn)
-    end
-    filename
-end
-
-"""
-    load(::Type{NeuralNetwork}, h5::HDF5.H5DataStore, arch::Architecture; backend = CPU())
-    load(::Type{NeuralNetwork}, h5::HDF5.H5DataStore, arch::Architecture, prototype; backend = CPU())
-
-Load network parameters from an already-open HDF5 store and return a `NeuralNetwork` for `arch`.
-
-The element type is whatever the file holds, so a `Float32` network reloads as `Float32`.
-
-Structured parameters — `StiefelManifold`, `SymmetricMatrix` and the rest — are rebuilt from the
-type each was stored under, which `GeometricOptimizers` registers with
-`NeuralNetworkParameters.register_parameter_type!`. Pass `prototype`, a parameter set of the right
-shape, to rebuild against it instead and skip the registry altogether.
-"""
-function load(::Type{NeuralNetwork}, h5::HDF5.H5DataStore, arch::Architecture;
-        backend::NeuralNetworkBackend = CPU())
-    NeuralNetwork(arch, Chain(arch), load(NetworkParameters, h5), backend)
-end
-
-function load(::Type{NeuralNetwork}, h5::HDF5.H5DataStore, arch::Architecture, prototype;
-        backend::NeuralNetworkBackend = CPU())
-    NeuralNetwork(arch, Chain(arch), load(NetworkParameters, h5, prototype), backend)
-end
-
-"""
-    load(::Type{NeuralNetwork}, filename::AbstractString, arch::Architecture; backend = CPU())
-    load(::Type{NeuralNetwork}, filename::AbstractString, arch::Architecture, prototype; backend = CPU())
-
-Convenience overload: open `filename` for reading, then call
-[`load`](@ref) on the store.
-"""
-function load(::Type{NeuralNetwork}, filename::AbstractString, arch::Architecture;
-        backend::NeuralNetworkBackend = CPU())
-    HDF5.h5open(filename, "r") do h5
-        load(NeuralNetwork, h5, arch; backend = backend)
-    end
-end
-
-function load(
-        ::Type{NeuralNetwork}, filename::AbstractString, arch::Architecture, prototype;
-        backend::NeuralNetworkBackend = CPU())
-    HDF5.h5open(filename, "r") do h5
-        load(NeuralNetwork, h5, arch, prototype; backend = backend)
-    end
-end
-
-end
diff --git a/scripts/Project.toml b/scripts/Project.toml
index b58960d88..26b087aab 100644
--- a/scripts/Project.toml
+++ b/scripts/Project.toml
@@ -7,6 +7,7 @@ ForwardDiff = "f6369f11-7733-5829-9624-2563aa707210"
 GeometricEquations = "c85262ba-a08a-430a-b926-d29770767bf2"
 GeometricIntegrators = "dcce2d33-59f6-5b8d-9047-0defad88ae06"
 GeometricMachineLearning = "194d25b2-d3f5-49f0-af24-c124f4aa80cc"
+GeometricOptimizers = "fc236c15-5557-4942-aa65-b650f329279e"
 GeometricProblems = "18cb22b4-ad41-5c80-9c5f-710df63fbdc9"
 HDF5 = "f67ccb44-e63f-5c2f-98bd-6dc0ccc4ba2f"
 JLD2 = "033835bb-8acc-5ee8-8aae-3f567f8a3819"
diff --git a/scripts/reproduction/harmonic_oscillator.jl b/scripts/reproduction/harmonic_oscillator.jl
index 6c88edc91..27bfb9691 100644
--- a/scripts/reproduction/harmonic_oscillator.jl
+++ b/scripts/reproduction/harmonic_oscillator.jl
@@ -13,17 +13,17 @@ ensemble_solution = exact_solution(ensemble_problem)
 
 include("../utilities/ensemble_plots.jl")
 
-# `GSympNet` trains through the generic `DataLoader` + `Batch` + `Optimizer` pipeline directly,
+# `GSympNet` trains through the generic `DataLoader` + `Batch` + `TrainingOptimizer` pipeline directly,
 # as `scripts/reproduction/sympnets/sympnet_toda_lattice.jl` already does.
 dl = DataLoader(ensemble_solution)
 
 arch = GSympNet(dl; n_layers = 4, upscaling_dimension = 10)
 nn = NeuralNetwork(arch, Float64)
-o = Optimizer(AdamOptimizer(), nn)
+o = TrainingOptimizer(nn.params; algorithm = Adam())
 batch = Batch(100)
 n_epochs = smoke_size(1000, 2)
 
-loss_array = o(nn, dl, batch, n_epochs)
+loss_array = train!(nn, o, dl, batch, n_epochs)
 
 function H(x)
     hamiltonian(
diff --git a/scripts/reproduction/hnn_pendulum.jl b/scripts/reproduction/hnn_pendulum.jl
index 97aece546..7f8faee83 100644
--- a/scripts/reproduction/hnn_pendulum.jl
+++ b/scripts/reproduction/hnn_pendulum.jl
@@ -34,10 +34,11 @@ dl = DataLoader(input, output)
 # in the data. It is the loss `NetworkLoss(hnn)` returns, so naming it is for the reader.
 loss = HNNLoss(hnn)
 
-# The step size belongs to the `Optimizer`, not to the method: the same `Adam()` trains at any
-# learning rate. `1e-3` is already Adam's default here and is named for the reader.
-optimizer = Optimizer(Adam(), nn; step_size = 1e-3)
-total_loss = optimizer(nn, dl, Batch(10), nepochs, loss; show_progress = true)
+# The step size is the `TrainingOptimizer`'s `linesearch`, not the method's: the same `Adam()` trains
+# at any learning rate. `1e-3` is already Adam's default here and is named for the reader.
+optimizer = TrainingOptimizer(nn.params; algorithm = Adam(), linesearch = 1e-3)
+total_loss = train!(
+    nn, optimizer, dl, Batch(10), nepochs; loss = loss, show_progress = true)
 
 # plot results
 include("../utilities/plots.jl")
diff --git a/scripts/reproduction/hnn_pendulum_simulation.jl b/scripts/reproduction/hnn_pendulum_simulation.jl
index c8ea9cfb0..2fd637503 100644
--- a/scripts/reproduction/hnn_pendulum_simulation.jl
+++ b/scripts/reproduction/hnn_pendulum_simulation.jl
@@ -36,8 +36,9 @@ dl = DataLoader(input, output)
 # symbolic derivative again.
 const loss = HNNLoss(hnn)
 
-optimizer = Optimizer(Adam(), nn; step_size = 1e-3)
-total_loss = optimizer(nn, dl, Batch(10), nepochs, loss; show_progress = true)
+optimizer = TrainingOptimizer(nn.params; algorithm = Adam(), linesearch = 1e-3)
+total_loss = train!(
+    nn, optimizer, dl, Batch(10), nepochs; loss = loss, show_progress = true)
 
 # The learned Hamiltonian and the vector field built from it. `loss.hvf` is what `HNNLoss` compares
 # against the data, so this integrates exactly the quantity that was trained -- the same object,
diff --git a/scripts/reproduction/linear_symplectic_transformer_gpu.jl b/scripts/reproduction/linear_symplectic_transformer_gpu.jl
index 7085b392e..70949b7f7 100644
--- a/scripts/reproduction/linear_symplectic_transformer_gpu.jl
+++ b/scripts/reproduction/linear_symplectic_transformer_gpu.jl
@@ -37,18 +37,18 @@ nn_standard = NeuralNetwork(arch_standard, backend, T)
 nn_symplectic = NeuralNetwork(arch_symplectic, backend, T)
 nn_sympnet = NeuralNetwork(arch_sympnet, backend, T)
 
-o_method = AdamOptimizer()
+o_method = Adam()
 
-o_standard = Optimizer(o_method, nn_standard)
-o_symplectic = Optimizer(o_method, nn_symplectic)
-o_sympnet = Optimizer(o_method, nn_sympnet)
+o_standard = TrainingOptimizer(nn_standard.params; algorithm = o_method)
+o_symplectic = TrainingOptimizer(nn_symplectic.params; algorithm = o_method)
+o_sympnet = TrainingOptimizer(nn_sympnet.params; algorithm = o_method)
 
 batch = Batch(batch_size, seq_length)
 batch2 = Batch(batch_size)
 
-loss_array_standard = o_standard(nn_standard, dl, batch, n_epochs)
-loss_array_symplectic = o_symplectic(nn_symplectic, dl, batch, n_epochs)
-loss_array_sympnet = o_sympnet(nn_sympnet, dl, batch2, n_epochs)
+loss_array_standard = train!(nn_standard, o_standard, dl, batch, n_epochs)
+loss_array_symplectic = train!(nn_symplectic, o_symplectic, dl, batch, n_epochs)
+loss_array_sympnet = train!(nn_sympnet, o_sympnet, dl, batch2, n_epochs)
 
 fig_train = Figure()
 ax_train = Axis(fig_train[1, 1]; xlabel = "epoch", ylabel = "training error", yscale = log10)
diff --git a/scripts/reproduction/lnn_pendulum.jl b/scripts/reproduction/lnn_pendulum.jl
index 1a45542cf..e0abd6080 100644
--- a/scripts/reproduction/lnn_pendulum.jl
+++ b/scripts/reproduction/lnn_pendulum.jl
@@ -43,11 +43,12 @@ dl = DataLoader(input, output)
 # `LNNLoss(arch)` is what `NetworkLoss(arch)` returns; naming it is for the reader.
 loss = LNNLoss(arch)
 
-# The step size belongs to the `Optimizer`, not to the method. `LNNLoss` needs a larger one than
-# `HNNLoss` does: its gradient reaches the parameters through a solve against the velocity Hessian,
-# where `HNNLoss` is linear in the gradient of the learned Hamiltonian.
-optimizer = Optimizer(Adam(), nn; step_size = 1e-2)
-total_loss = optimizer(nn, dl, Batch(10), nepochs, loss; show_progress = true)
+# The step size is the `TrainingOptimizer`'s `linesearch`, not the method's. `LNNLoss` needs a larger
+# one than `HNNLoss` does: its gradient reaches the parameters through a solve against the velocity
+# Hessian, where `HNNLoss` is linear in the gradient of the learned Hamiltonian.
+optimizer = TrainingOptimizer(nn.params; algorithm = Adam(), linesearch = 1e-2)
+total_loss = train!(
+    nn, optimizer, dl, Batch(10), nepochs; loss = loss, show_progress = true)
 
 # plot results
 include("../utilities/plots.jl")
diff --git a/scripts/reproduction/symplectic_autoencoders/online_sympnet.jl b/scripts/reproduction/symplectic_autoencoders/online_sympnet.jl
index 742d55d44..810904131 100644
--- a/scripts/reproduction/symplectic_autoencoders/online_sympnet.jl
+++ b/scripts/reproduction/symplectic_autoencoders/online_sympnet.jl
@@ -38,12 +38,12 @@ const n_epochs = smoke_size(262144, 2)
 const batch_size = 4096
 
 sae_pairing = AdamOptimizerWithDecay(n_epochs)
-o = Optimizer(sae_nn; sae_pairing...)
+o = TrainingOptimizer(sae_nn.params; sae_pairing...)
 
 println("Number of batches: ", GeometricMachineLearning.number_of_batches(dl, Batch(batch_size)))
 
 psd_error = solve!(psd_nn, dl)
-sae_error = o(sae_nn, dl, Batch(batch_size), n_epochs)
+sae_error = train!(sae_nn, o, dl, Batch(batch_size), n_epochs)
 
 fig = Figure()
 ax = Axis(fig[1, 1], xlabel = "epoch", ylabel = "training error")
@@ -106,7 +106,7 @@ integrator_architecture = StandardTransformerIntegrator(
 # the fallback raises is `Functor not defined`, not a `MethodError`.
 integrator_nn = NeuralNetwork(integrator_architecture, backend, Float32)
 integrator_pairing = AdamOptimizerWithDecay(integrator_train_epochs)
-o_integrator = Optimizer(integrator_nn; integrator_pairing...)
+o_integrator = TrainingOptimizer(integrator_nn.params; integrator_pairing...)
 
 loss = GeometricMachineLearning.ReducedLoss(encoder(sae_nn), decoder(sae_nn))
 
@@ -116,8 +116,8 @@ dl_integration = DataLoader(dl; autoencoder = false)
 # the regular transformer can't deal with symplectic data!
 dl_integration = DataLoader(vcat(dl_integration.input.q, dl_integration.input.p))
 integrator_batch = Batch(integrator_batch_size, seq_length)
-train_integrator_loss = o_integrator(
-    integrator_nn, dl_integration, integrator_batch, integrator_train_epochs, loss)
+train_integrator_loss = train!(integrator_nn, o_integrator, dl_integration,
+    integrator_batch, integrator_train_epochs; loss = loss)
 
 JLD2.save("integrator_parameters.jld2", "integrator_parameters",
     integrator_nn.params |> mtc, "training loss", train_integrator_loss)
diff --git a/scripts/reproduction/symplectic_autoencoders/online_transformer_for_sae.jl b/scripts/reproduction/symplectic_autoencoders/online_transformer_for_sae.jl
index 0a911ed1d..9e9215f96 100644
--- a/scripts/reproduction/symplectic_autoencoders/online_transformer_for_sae.jl
+++ b/scripts/reproduction/symplectic_autoencoders/online_transformer_for_sae.jl
@@ -52,7 +52,7 @@ const integrator_architecture = StandardTransformerIntegrator(
 # the fallback raises is `Functor not defined`, not a `MethodError`.
 const integrator_nn = NeuralNetwork(integrator_architecture, backend, Float32)
 const integrator_pairing = AdamOptimizerWithDecay(integrator_train_epochs)
-const o_integrator = Optimizer(integrator_nn; integrator_pairing...)
+const o_integrator = TrainingOptimizer(integrator_nn.params; integrator_pairing...)
 
 loss = GeometricMachineLearning.ReducedLoss(encoder(sae_nn), decoder(sae_nn))
 
@@ -62,8 +62,8 @@ dl_integration = DataLoader(dl; autoencoder = false)
 # the regular transformer can't deal with symplectic data!
 dl_integration = DataLoader(vcat(dl_integration.input.q, dl_integration.input.p))
 integrator_batch = Batch(integrator_batch_size, seq_length)
-train_integrator_loss = o_integrator(
-    integrator_nn, dl_integration, integrator_batch, integrator_train_epochs, loss)
+train_integrator_loss = train!(integrator_nn, o_integrator, dl_integration,
+    integrator_batch, integrator_train_epochs; loss = loss)
 
 const mtc = GeometricMachineLearning.map_to_cpu
 JLD2.save(
diff --git a/scripts/reproduction/symplectic_autoencoders/training.jl b/scripts/reproduction/symplectic_autoencoders/training.jl
index 389369520..b0fed708f 100644
--- a/scripts/reproduction/symplectic_autoencoders/training.jl
+++ b/scripts/reproduction/symplectic_autoencoders/training.jl
@@ -28,12 +28,11 @@ n_epochs = smoke_size(100, 2)
 batch_size = smoke_size(512, 16)
 n_range = 2:1:smoke_size(15, 3)
 μ_range = (T(0.51), T(0.625), T(0.55), T(0.47))
-# The learning rate is the `Optimizer`'s `step_size`, not the method's: `Adam` lost the `η` field
-# it never applied to the direction, and `β₁`, `β₂` and `δ` became keywords so that the old
-# positional call fails instead of silently reading `η` as `β₁`.
-opt = AdamOptimizer(T; β₁ = T(0.9), β₂ = T(0.99), δ = T(1.0e-8))
+# The learning rate is the `TrainingOptimizer`'s `linesearch`, not the method's: `Adam` has no `η`
+# field, and `β₁`, `β₂` and `δ` are keywords.
+opt = Adam(; β₁ = 0.9, β₂ = 0.99, δ = 1.0e-8)
 const step_size = T(0.001)
-# The retraction belongs to the `Optimizer` now, not to the layer: `PSDLayer(M, N)` takes no
+# The retraction belongs to the `TrainingOptimizer`, not to the layer: `PSDLayer(M, N)` takes no
 # keyword.
 retraction = Cayley()
 
@@ -89,19 +88,17 @@ end
 # architecture rather than a bare `Chain` is what makes `encoder(nn)` and `decoder(nn)` available,
 # and those are what `HRedSys` needs.
 #
-# Training is `DataLoader` + `Batch` + `Optimizer`. What stood here was a hand-rolled loop over
-# `redraw_batch!(dl)`, counting its own iterations from `dl.batch_size`; a `DataLoader` carries
-# neither, because the batch is a `Batch` and the loop is the `Optimizer` functor. The loop also
-# wrapped both the pullback and the optimization step in `try … catch; continue`, so a training run
-# in which every single step failed was indistinguishable from one that worked.
+# Training is `DataLoader` + `Batch` + `TrainingOptimizer`, and the loop is `train!`: the batch is a
+# `Batch`, and a failed step raises its error.
 function get_nn_encoder_decoder(; n = 5, n_epochs = 500, opt = opt, T = T)
     sae_arch = SymplecticAutoencoder(
         2 * N, 2 * n; n_encoder_blocks = 4, n_decoder_blocks = 4,
         n_encoder_layers = 2, n_decoder_layers = 2)
     sae_nn = NeuralNetwork(sae_arch, backend, T)
 
-    optimizer_instance = Optimizer(opt, sae_nn; step_size = step_size, retraction = retraction)
-    optimizer_instance(sae_nn, dl, Batch(batch_size), n_epochs)
+    optimizer_instance = TrainingOptimizer(
+        sae_nn.params; algorithm = opt, linesearch = step_size, retraction = retraction)
+    train!(sae_nn, optimizer_instance, dl, Batch(batch_size), n_epochs)
 
     sae_nn
 end
diff --git a/scripts/reproduction/symplectic_transformer/double_pendulum.jl b/scripts/reproduction/symplectic_transformer/double_pendulum.jl
index 021e39a07..a5dc65857 100644
--- a/scripts/reproduction/symplectic_transformer/double_pendulum.jl
+++ b/scripts/reproduction/symplectic_transformer/double_pendulum.jl
@@ -70,19 +70,19 @@ nn4 = NeuralNetwork(arch4)
 const batch_size = 1024
 const n_epochs = smoke_size(10000, 2)
 
-o_method = AdamOptimizer()
+o_method = Adam()
 
-o1 = Optimizer(o_method, nn1)
-o2 = Optimizer(o_method, nn2)
-o3 = Optimizer(o_method, nn3)
-o4 = Optimizer(o_method, nn4)
+o1 = TrainingOptimizer(nn1.params; algorithm = o_method)
+o2 = TrainingOptimizer(nn2.params; algorithm = o_method)
+o3 = TrainingOptimizer(nn3.params; algorithm = o_method)
+o4 = TrainingOptimizer(nn4.params; algorithm = o_method)
 
 batch = Batch(batch_size, seq_length)
 
-loss_array1 = o1(nn1, dl, batch, n_epochs)
-loss_array2 = o2(nn2, dl, batch, n_epochs)
-loss_array3 = o3(nn3, dl, batch, n_epochs)
-loss_array4 = o4(nn4, dl, Batch(batch_size), n_epochs)
+loss_array1 = train!(nn1, o1, dl, batch, n_epochs)
+loss_array2 = train!(nn2, o2, dl, batch, n_epochs)
+loss_array3 = train!(nn3, o3, dl, batch, n_epochs)
+loss_array4 = train!(nn4, o4, dl, Batch(batch_size), n_epochs)
 
 function make_training_error_plot(; theme = :dark, symplectic = true)
     textcolor = theme == :dark ? :white : :black
diff --git a/scripts/reproduction/symplectic_transformer/double_pendulum_short_integration.jl b/scripts/reproduction/symplectic_transformer/double_pendulum_short_integration.jl
index 324cee989..962730e18 100644
--- a/scripts/reproduction/symplectic_transformer/double_pendulum_short_integration.jl
+++ b/scripts/reproduction/symplectic_transformer/double_pendulum_short_integration.jl
@@ -68,19 +68,19 @@ nn4 = NeuralNetwork(arch4)
 const batch_size = 1024
 const n_epochs = smoke_size(3000, 2)
 
-o_method = AdamOptimizer()
+o_method = Adam()
 
-o1 = Optimizer(o_method, nn1)
-o2 = Optimizer(o_method, nn2)
-o3 = Optimizer(o_method, nn3)
-o4 = Optimizer(o_method, nn4)
+o1 = TrainingOptimizer(nn1.params; algorithm = o_method)
+o2 = TrainingOptimizer(nn2.params; algorithm = o_method)
+o3 = TrainingOptimizer(nn3.params; algorithm = o_method)
+o4 = TrainingOptimizer(nn4.params; algorithm = o_method)
 
 batch = Batch(batch_size, seq_length)
 
-loss_array1 = o1(nn1, dl, batch, n_epochs)
-loss_array2 = o2(nn2, dl, batch, n_epochs)
-loss_array3 = o3(nn3, dl, batch, n_epochs)
-loss_array4 = o4(nn4, dl, Batch(batch_size), n_epochs)
+loss_array1 = train!(nn1, o1, dl, batch, n_epochs)
+loss_array2 = train!(nn2, o2, dl, batch, n_epochs)
+loss_array3 = train!(nn3, o3, dl, batch, n_epochs)
+loss_array4 = train!(nn4, o4, dl, Batch(batch_size), n_epochs)
 
 function make_training_error_plot(; theme = :dark, symplectic = true)
     textcolor = theme == :dark ? :white : :black
diff --git a/scripts/reproduction/sympnets/sympnet_pendulum.jl b/scripts/reproduction/sympnets/sympnet_pendulum.jl
index 5fe8fa0f7..b8cfeeb2c 100644
--- a/scripts/reproduction/sympnets/sympnet_pendulum.jl
+++ b/scripts/reproduction/sympnets/sympnet_pendulum.jl
@@ -4,6 +4,7 @@ Simple implementation of a pendulum for SympNets.
 
 using GeometricIntegrators
 using GeometricMachineLearning
+using GeometricOptimizers: optimization_step!
 using LinearAlgebra
 using CairoMakie
 using Zygote
@@ -66,18 +67,16 @@ function full_loss(ps, q, p)
 end
 
 # define momentum optimizer and initialize
-method = AdamOptimizer()
+method = Adam()
 # initial gradients for calling Cache constructor
-opt = Optimizer(method, ps)
-# `optimization_step!` takes the global section of the parameters, not the model.
-λY = GlobalSection(ps)
+opt = TrainingOptimizer(ps; algorithm = method)
 
 # training 
 println("initial loss: ", full_loss(ps, q, p))
 training_steps = smoke_size(1000, 2)
 @showprogress for i in 1:training_steps
     dp = Zygote.gradient(ps -> loss(ps, q, p), ps)[1]
-    optimization_step!(opt, λY, ps, dp)
+    optimization_step!(ps, opt, dp)
 end
 println("final loss: ", full_loss(ps, q, p))
 
diff --git a/scripts/reproduction/sympnets/sympnet_toda_lattice.jl b/scripts/reproduction/sympnets/sympnet_toda_lattice.jl
index 6871819e1..c4f0f0d96 100644
--- a/scripts/reproduction/sympnets/sympnet_toda_lattice.jl
+++ b/scripts/reproduction/sympnets/sympnet_toda_lattice.jl
@@ -25,14 +25,14 @@ dl = DataLoader(integrate(ep, ImplicitMidpoint()); suppress_info = true)
 arch = GSympNet(dl; upscaling_dimension = 4)
 
 nn = NeuralNetwork(arch)
-o_method = AdamOptimizer()
+o_method = Adam()
 
-o = Optimizer(o_method, nn)
+o = TrainingOptimizer(nn.params; algorithm = o_method)
 
 batch = Batch(256)
 const n_epochs = smoke_size(1024, 2)
 
-loss_array = o(nn, dl, batch, n_epochs)
+loss_array = train!(nn, o, dl, batch, n_epochs)
 
 function make_training_error_plot(; theme = :dark)
     textcolor = theme == :dark ? :white : :black
diff --git a/scripts/reproduction/transformer_integrator/softmax_comparison.jl b/scripts/reproduction/transformer_integrator/softmax_comparison.jl
index 7c95a8320..4ec062947 100644
--- a/scripts/reproduction/transformer_integrator/softmax_comparison.jl
+++ b/scripts/reproduction/transformer_integrator/softmax_comparison.jl
@@ -49,15 +49,15 @@ arch2 = StandardTransformerIntegrator(dl.input_dim; transformer_dim = 20,
 nn1 = NeuralNetwork(arch1)
 nn2 = NeuralNetwork(arch2)
 
-o_method = AdamOptimizer()
+o_method = Adam()
 
-o1 = Optimizer(o_method, nn1)
-o2 = Optimizer(o_method, nn2)
+o1 = TrainingOptimizer(nn1.params; algorithm = o_method)
+o2 = TrainingOptimizer(nn2.params; algorithm = o_method)
 
 batch = Batch(batch_size, seq_length)
 
-loss_array1 = o1(nn1, dl, batch, n_epochs)
-loss_array2 = o2(nn2, dl, batch, n_epochs)
+loss_array1 = train!(nn1, o1, dl, batch, n_epochs)
+loss_array2 = train!(nn2, o2, dl, batch, n_epochs)
 
 const n_steps = 300
 
diff --git a/scripts/reproduction/transformer_integrator/symplectic_transformer.jl b/scripts/reproduction/transformer_integrator/symplectic_transformer.jl
index 4f2f5d16a..62e70a10f 100644
--- a/scripts/reproduction/transformer_integrator/symplectic_transformer.jl
+++ b/scripts/reproduction/transformer_integrator/symplectic_transformer.jl
@@ -55,17 +55,17 @@ nn3 = NeuralNetwork(arch3)
 const batch_size = 1024
 const n_epochs = smoke_size(3000, 2)
 
-o_method = AdamOptimizer()
+o_method = Adam()
 
-o1 = Optimizer(o_method, nn1)
-o2 = Optimizer(o_method, nn2)
-o3 = Optimizer(o_method, nn3)
+o1 = TrainingOptimizer(nn1.params; algorithm = o_method)
+o2 = TrainingOptimizer(nn2.params; algorithm = o_method)
+o3 = TrainingOptimizer(nn3.params; algorithm = o_method)
 
 batch = Batch(batch_size, seq_length)
 
-loss_array1 = o1(nn1, dl, batch, n_epochs)
-loss_array2 = o2(nn2, dl, batch, n_epochs)
-loss_array3 = o3(nn3, dl, batch, n_epochs)
+loss_array1 = train!(nn1, o1, dl, batch, n_epochs)
+loss_array2 = train!(nn2, o2, dl, batch, n_epochs)
+loss_array3 = train!(nn3, o3, dl, batch, n_epochs)
 
 function make_training_error_plot(; theme = :dark, symplectic = true)
     textcolor = theme == :dark ? :white : :black
diff --git a/scripts/reproduction/transformer_integrator/symplectic_transformer_vector_softmax.jl b/scripts/reproduction/transformer_integrator/symplectic_transformer_vector_softmax.jl
index d7fb30e10..5a091204b 100644
--- a/scripts/reproduction/transformer_integrator/symplectic_transformer_vector_softmax.jl
+++ b/scripts/reproduction/transformer_integrator/symplectic_transformer_vector_softmax.jl
@@ -55,17 +55,17 @@ nn3 = NeuralNetwork(arch3)
 const batch_size = 1024
 const n_epochs = smoke_size(3000, 2)
 
-o_method = AdamOptimizer()
+o_method = Adam()
 
-o1 = Optimizer(o_method, nn1)
-o2 = Optimizer(o_method, nn2)
-o3 = Optimizer(o_method, nn3)
+o1 = TrainingOptimizer(nn1.params; algorithm = o_method)
+o2 = TrainingOptimizer(nn2.params; algorithm = o_method)
+o3 = TrainingOptimizer(nn3.params; algorithm = o_method)
 
 batch = Batch(batch_size, seq_length)
 
-loss_array1 = o1(nn1, dl, batch, n_epochs)
-loss_array2 = o2(nn2, dl, batch, n_epochs)
-loss_array3 = o3(nn3, dl, batch, n_epochs)
+loss_array1 = train!(nn1, o1, dl, batch, n_epochs)
+loss_array2 = train!(nn2, o2, dl, batch, n_epochs)
+loss_array3 = train!(nn3, o3, dl, batch, n_epochs)
 
 function make_training_error_plot(; theme = :dark, symplectic = true)
     textcolor = theme == :dark ? :white : :black
diff --git a/scripts/reproduction/volume_preserving_feedforward/abc_flow.jl b/scripts/reproduction/volume_preserving_feedforward/abc_flow.jl
index a1810aff8..635831906 100644
--- a/scripts/reproduction/volume_preserving_feedforward/abc_flow.jl
+++ b/scripts/reproduction/volume_preserving_feedforward/abc_flow.jl
@@ -25,7 +25,7 @@ const n_linear = 2
 const activation = tanh
 
 const batch_size = 16384
-const opt_method = AdamOptimizer()
+const opt_method = Adam()
 const n_epochs = smoke_size(1000, 2)
 
 # This was run on an Apple GPU: `using Metal`, `const backend = MetalBackend()`, `const T =
@@ -47,13 +47,13 @@ model = VolumePreservingFeedForward(sys_dim, n_blocks, n_linear, activation)
 
 nn = NeuralNetwork(model, backend, T)
 
-o = Optimizer(opt_method, nn)
+o = TrainingOptimizer(nn.params; algorithm = opt_method)
 
 # `Batch(batch_size, 1)` is a `Batch{:Transformer}` of sequence length one, and the network here is
 # a `NeuralNetworkIntegrator`, whose loss defaults only for a `Batch{:FeedForward}`.
 batch = Batch(batch_size)
 
-loss_array₁ = o(nn, dl, batch, n_epochs)
+loss_array₁ = train!(nn, o, dl, batch, n_epochs)
 
 ic = (q = [0.0, 0.0, 0.1],)
 
diff --git a/scripts/reproduction/volume_preserving_feedforward/double_pendulum.jl b/scripts/reproduction/volume_preserving_feedforward/double_pendulum.jl
index 099bc56eb..365e75772 100644
--- a/scripts/reproduction/volume_preserving_feedforward/double_pendulum.jl
+++ b/scripts/reproduction/volume_preserving_feedforward/double_pendulum.jl
@@ -42,7 +42,7 @@ const T = eltype(dl)
 # hyperparameters concerning training 
 const n_epochs = smoke_size(1000, 2)
 const batch_size = 1024
-const opt_method = AdamOptimizer(T)
+const opt_method = Adam()
 const resnet_activation = tanh
 
 const t_validation = 30
@@ -54,10 +54,11 @@ function setup_and_train(
         model::Union{GeometricMachineLearning.Architecture, GeometricMachineLearning.Chain},
         batch::Batch; transformer::Bool = true)
     nn₀ = NeuralNetwork(model, backend, T)
-    o₀ = Optimizer(opt_method, nn₀)
+    o₀ = TrainingOptimizer(nn₀.params; algorithm = opt_method)
 
-    loss_array = transformer ? o₀(nn₀, dl, batch, n_epochs, transformer_loss) :
-                 o₀(nn₀, dl, batch, n_epochs)
+    loss_array = transformer ?
+                 train!(nn₀, o₀, dl, batch, n_epochs; loss = transformer_loss) :
+                 train!(nn₀, o₀, dl, batch, n_epochs)
 
     nn₀, loss_array
 end
diff --git a/scripts/reproduction/volume_preserving_feedforward/rigid_body.jl b/scripts/reproduction/volume_preserving_feedforward/rigid_body.jl
index ae6763ffa..fd0fd2a7a 100644
--- a/scripts/reproduction/volume_preserving_feedforward/rigid_body.jl
+++ b/scripts/reproduction/volume_preserving_feedforward/rigid_body.jl
@@ -28,7 +28,7 @@ const n_linear = 2
 const activation = tanh
 
 const batch_size = 16384
-const opt_method = AdamOptimizer()
+const opt_method = Adam()
 const n_epochs = smoke_size(20000, 2)
 
 # The GPU is what this was run on; the smoke run drops to the CPU so that CI can execute it.
@@ -48,11 +48,11 @@ model = VolumePreservingFeedForward(sys_dim, n_blocks, n_linear, activation)
 
 nn = NeuralNetwork(model, backend, T)
 
-o = Optimizer(opt_method, nn)
+o = TrainingOptimizer(nn.params; algorithm = opt_method)
 
 batch = Batch(batch_size)
 
-loss_array₁ = o(nn, dl, batch, n_epochs)
+loss_array₁ = train!(nn, o, dl, batch, n_epochs)
 
 ic = (q = [sin(1.1), 0.0, cos(1.1)],)
 
diff --git a/scripts/reproduction/volume_preserving_transformer/rigid_body.jl b/scripts/reproduction/volume_preserving_transformer/rigid_body.jl
index 47708f121..a68f78a9d 100644
--- a/scripts/reproduction/volume_preserving_transformer/rigid_body.jl
+++ b/scripts/reproduction/volume_preserving_transformer/rigid_body.jl
@@ -51,7 +51,7 @@ const dl = backend == CPU() ? DataLoader(T.(dl₁.input)) :
 # hyperparameters concerning training 
 const n_epochs = smoke_size(500000, 2)
 const batch_size = 16384
-const opt_pairing = AdamOptimizerWithDecay(n_epochs, T; η₁ = 1e-2, η₂ = 1e-6)
+const opt_pairing = AdamOptimizerWithDecay(n_epochs; η₁ = 1e-2, η₂ = 1e-6)
 
 # parameters for evaluation 
 ics_val = [sin(1.1), 0.0, cos(1.1)]
@@ -62,9 +62,9 @@ const t_validation_long = 100
 function train_the_network(nn₀::GeometricMachineLearning.NeuralNetwork, batch::Batch)
     Random.seed!(1234)
 
-    o₀ = Optimizer(nn₀; opt_pairing...)
+    o₀ = TrainingOptimizer(nn₀.params; opt_pairing...)
 
-    loss_array = o₀(nn₀, dl, batch, n_epochs)
+    loss_array = train!(nn₀, o₀, dl, batch, n_epochs)
 
     GeometricMachineLearning.map_to_cpu(nn₀), loss_array
 end
diff --git a/scripts/verification/all_zero_leaf.jl b/scripts/verification/all_zero_leaf.jl
new file mode 100644
index 000000000..7713901bf
--- /dev/null
+++ b/scripts/verification/all_zero_leaf.jl
@@ -0,0 +1,82 @@
+# What a training step does to a layer the loss does not read.
+#
+# NeuralNetworkParameters gives the gradient of such a layer as a zero leaf of the layer's own type,
+# not `nothing`, and `train!` hands the whole gradient to `GeometricOptimizers.optimization_step!`:
+# GeometricMachineLearning skips no leaf. This script prints how far each leaf moves, in both
+# precisions and under each first-order method, and asserts the following; it stops with an
+# `AssertionError` at the first claim that does not hold:
+#
+#   - a Euclidean and a Stiefel leaf the loss never reads stay `==` their start under
+#     `GradientMethod`, `MomentumMethod` and `Adam`; under `AdamWithEuclideanDecay` the Euclidean one
+#     decays and the Stiefel one does not;
+#   - a leaf the loss reads for the first 5 steps and not after keeps moving on its momentum under
+#     `MomentumMethod`, `Adam` and `AdamWithEuclideanDecay`, and stays under `GradientMethod`;
+#   - a layer without numbers, `NamedTuple()`, steps without an error.
+#
+# Run from the repository root, with the scripts environment:
+#   julia --startup-file=no --project=scripts scripts/verification/all_zero_leaf.jl
+
+using GeometricMachineLearning
+using GeometricMachineLearning: params
+using GeometricOptimizers: optimization_step!
+using Printf
+import Random
+import Zygote
+
+const N_STEPS = 50
+const N_READ = 5
+
+maxdiff(a, b) = maximum(abs, Matrix(a) - Matrix(b))
+
+function zero_leaf_run(T, method)
+    Random.seed!(1)
+    ps = NetworkParameters((read = (W = randn(T, 3, 3),),
+        never_euclidean = (W = randn(T, 3, 3),),
+        never_stiefel = (Y = rand(StiefelManifold{T}, 4, 2),),
+        read_then_not = (W = randn(T, 3, 3),),
+        empty = NamedTuple()))
+    p₀ = deepcopy(ps)
+    opt = TrainingOptimizer(ps; algorithm = method)
+    x = randn(T, 3)
+    for t in 1:N_STEPS
+        loss = t ≤ N_READ ?
+               (p -> sum(abs2, p.read.W * x) + sum(abs2, p.read_then_not.W * x)) :
+               (p -> sum(abs2, p.read.W * x))
+        dp = only(Zygote.gradient(loss, ps))
+        optimization_step!(ps, opt, dp)
+    end
+    (never_euclidean = maxdiff(ps.never_euclidean.W, p₀.never_euclidean.W),
+        never_stiefel = maxdiff(ps.never_stiefel.Y, p₀.never_stiefel.Y))
+end
+
+# The motion of the leaf that is read for `N_READ` steps, measured from step `N_READ` on.
+function after_reading(T, method)
+    Random.seed!(1)
+    ps = NetworkParameters((
+        read = (W = randn(T, 3, 3),), read_then_not = (W = randn(T, 3, 3),)))
+    opt = TrainingOptimizer(ps; algorithm = method)
+    x = randn(T, 3)
+    both = p -> sum(abs2, p.read.W * x) + sum(abs2, p.read_then_not.W * x)
+    read_only = p -> sum(abs2, p.read.W * x)
+    for _ in 1:N_READ
+        optimization_step!(ps, opt, only(Zygote.gradient(both, ps)))
+    end
+    W₅ = copy(ps.read_then_not.W)
+    for _ in (N_READ + 1):N_STEPS
+        optimization_step!(ps, opt, only(Zygote.gradient(read_only, ps)))
+    end
+    maxdiff(ps.read_then_not.W, W₅)
+end
+
+for T in (Float32, Float64)
+    for method in (GradientMethod(), MomentumMethod(), Adam(), AdamWithEuclideanDecay())
+        r = zero_leaf_run(T, method)
+        moved = after_reading(T, method)
+        @printf("%-8s %-26s never read: Euclidean %.2e  Stiefel %.2e | read %d steps, then not: %.2e\n",
+            T, nameof(typeof(method)), r.never_euclidean, r.never_stiefel, N_READ, moved)
+        decays = method isa AdamWithEuclideanDecay
+        @assert decays ? r.never_euclidean > 0 : r.never_euclidean == 0
+        @assert r.never_stiefel == 0
+        @assert method isa GradientMethod ? moved == 0 : moved > 0
+    end
+end
diff --git a/src/GeometricMachineLearning.jl b/src/GeometricMachineLearning.jl
index bb474b69b..74d017935 100644
--- a/src/GeometricMachineLearning.jl
+++ b/src/GeometricMachineLearning.jl
@@ -6,12 +6,11 @@ using AbstractNeuralNetworks
 # package also exports `flatten`/`unflatten` and the leaf protocol, none of which this package
 # extends — `GeometricOptimizers` carries the protocol for the structured matrices.
 #
-# Two things do come from there rather than being written again here: `mapstorage`, which reaches the
-# storage of a structured leaf and rebuilds the leaf around the result (`src/map_to_cpu.jl`), and
-# `parameter_eltype`, which promotes over the leaves of a set. A plain `NamedTuple` of layers is
-# walked with `Base.map`, which needs nothing from anybody.
+# One thing does come from there rather than being written again here: `mapstorage`, which reaches
+# the storage of a structured leaf and rebuilds the leaf around the result (`src/map_to_cpu.jl`). A
+# plain `NamedTuple` of layers is walked with `Base.map`, which needs nothing from anybody.
 import NeuralNetworkParameters: NetworkParameters
-using NeuralNetworkParameters: mapparameters, mapstorage, parameter_eltype
+using NeuralNetworkParameters: mapstorage
 using ChainRulesCore
 using GeometricBase
 using GeometricSolutions: GeometricSolution, EnsembleSolution, TimeSeries
@@ -37,12 +36,14 @@ import Symbolics
 # `import` and not `using ...: ...`: GML adds constructor methods to several of these types (in
 # `layers/` and the kernels), and extending a *type* reached through `using` warns on every such
 # method since Julia 1.12 — "Constructor for type … was extended in `GeometricMachineLearning`
-# without explicit qualification or import". Everything imported here is re-exported below, which
-# is what keeps `using GeometricMachineLearning` alone sufficient.
+# without explicit qualification or import". Everything imported here is re-exported below, except
+# `orthonormal_columns` and `optimization_step!`; the re-exports are what keep
+# `using GeometricMachineLearning` alone sufficient.
 import GeometricOptimizers
 import GeometricOptimizers: Manifold, StiefelManifold, GrassmannManifold
 import GeometricOptimizers: SkewSymMatrix, SymmetricMatrix,
-                            LowerTriangular, UpperTriangular, StiefelProjection
+                            StrictlyLowerTriangular, StrictlyUpperTriangular,
+                            StiefelProjection
 import GeometricOptimizers: StiefelLieAlgHorMatrix, GrassmannLieAlgHorMatrix
 import GeometricOptimizers: rgrad, metric, check, global_section
 # `orthonormal_columns(draw)` orthonormalizes `draw()` with CholeskyQR2, which runs on whatever
@@ -52,17 +53,16 @@ import GeometricOptimizers: orthonormal_columns
 import GeometricOptimizers: GlobalSection, global_rep, apply_section, apply_section!,
                             update_section!
 import GeometricOptimizers: Geodesic, Cayley, geodesic, cayley, retraction
-import GeometricOptimizers: OptimizerMethod,
-                            GradientMethod, MomentumMethod, Adam,
-                            GradientState, MomentumState, AdamState,
+import GeometricOptimizers: GradientMethod, MomentumMethod, Adam, AdamWithEuclideanDecay,
                             AdamOptimizerWithDecay, DecayingStatic
+# The training step. `train!` takes a `TrainingOptimizer` and calls `optimization_step!`, which is
+# not re-exported.
+import GeometricOptimizers: TrainingOptimizer, optimization_step!
 import GeometricOptimizers: update!
 # `solve!` is imported rather than started afresh so that GML's `solve!(::NeuralNetwork{<:PSDArch},
 # …)` — solve for the parameters directly, by SVD, instead of training for them — is a method of the
 # same verb a caller already has from GeometricOptimizers, and not a second function of the name.
 import GeometricOptimizers: solve!
-# The optimizer *caches* stay internal upstream — they are `solver_step!` scratch — so GML reaches
-# them as `GeometricOptimizers.AdamCache` where it needs to name one, and does not re-export them.
 
 import AbstractNeuralNetworks: Architecture, AbstractExplicitLayer, NeuralNetwork,
                                FeedForwardLoss
@@ -114,7 +114,7 @@ include("arrays/poisson_tensor.jl")
 # Re-exported from GeometricOptimizers, so that `using GeometricMachineLearning` on its own still
 # gives a caller the matrix types its layers are parametrized by.
 export SymmetricMatrix, SkewSymMatrix
-export LowerTriangular, UpperTriangular
+export StrictlyLowerTriangular, StrictlyUpperTriangular
 export StiefelLieAlgHorMatrix, GrassmannLieAlgHorMatrix
 export StiefelProjection
 # GML's own
@@ -194,31 +194,9 @@ export ResNet
 export Transformer
 export TransformerIntegrator, StandardTransformerIntegrator
 
-# INCLUDE OPTIMIZERS — the methods, states, sections and retractions come from GeometricOptimizers.
+# INCLUDE OPTIMIZERS — the methods, sections and retractions come from GeometricOptimizers.
 include("optimizers/optimizer.jl")
 
-export OptimizerMethod
-export GradientMethod, GradientState
-export MomentumMethod, MomentumState
-export Adam, AdamState
-export Optimizer
-export optimization_step!
-export GlobalSection, global_section, apply_section, apply_section!, update_section!
-export global_rep
-export Geodesic, Cayley
-export geodesic, cayley
-export retraction
-export update!
-# The optimizer caches are not exported, for every method alike: see the note at the
-# GeometricOptimizers imports above.
-# backward-compat aliases (old names → new names)
-const GradientOptimizer = GradientMethod
-const MomentumOptimizer = MomentumMethod
-const AdamOptimizer = Adam
-export GradientOptimizer, MomentumOptimizer, AdamOptimizer
-# Re-exported from GeometricOptimizers, which owns the one definition of them.
-export AdamOptimizerWithDecay, DecayingStatic
-
 export NeuralNetwork
 
 export NetworkLoss, TransformerLoss, FeedForwardLoss, AutoEncoderLoss, ReducedLoss, HNNLoss,
@@ -263,10 +241,10 @@ include("pullbacks/zygote_pullback.jl")
 include("pullbacks/symbolic_hnn_pullback.jl")
 
 export DataLoader
-export Batch, optimize_for_one_epoch!
+export Batch, train!
 include("data_loader/tensor_assign.jl")
 include("data_loader/batch.jl")
-include("data_loader/optimize.jl")
+include("data_loader/train.jl")
 
 include("reduced_system/reduced_system.jl")
 
diff --git a/src/arrays/poisson_tensor.jl b/src/arrays/poisson_tensor.jl
index 313276d65..b95da036b 100644
--- a/src/arrays/poisson_tensor.jl
+++ b/src/arrays/poisson_tensor.jl
@@ -67,6 +67,10 @@ end
 
 Base.size(𝕁::PoissonTensor) = size(𝕁.J)
 
+# The backend of the stored matrix. GeometricOptimizers' checked products ask every operand for one,
+# so `𝕁 * Y` with a `StiefelManifold` `Y` raises without it.
+KernelAbstractions.get_backend(𝕁::PoissonTensor) = KernelAbstractions.get_backend(𝕁.J)
+
 function PoissonTensor(backend::Backend, n2::Int, T::DataType)
     @assert iseven(n2)
     n = n2÷2
diff --git a/src/data_loader/optimize.jl b/src/data_loader/optimize.jl
deleted file mode 100644
index 349448b9a..000000000
--- a/src/data_loader/optimize.jl
+++ /dev/null
@@ -1,117 +0,0 @@
-@doc raw"""
-    optimize_for_one_epoch!(opt, model, ps, dl, batch, loss, λY)
-
-Sample the data contained in `dl` according to `batch` and optimize for these batches.
-
-This step also performs automatic differentiation on `loss`.
-
-The output of `optimize_for_one_epoch!` is the average loss over all batches of the epoch:
-```math
-output = \frac{1}{\mathtt{steps\_per\_epoch}}\sum_{t=1}^\mathtt{steps\_per\_epoch}\mathtt{loss}(\theta^{(t-1)}).
-```
-This is done because any *reverse differentiation* routine always has two outputs; for `Zygote`:
-```julia
-loss_value, pullback = Zygote.pullback(ps -> loss(model, ps, input, output), ps)
-```
-So we get the value for the loss for free whenever we compute the pullback with AD.
-
-# Arguments 
-
-All the arguments are mandatory (there are no defaults): 
-1. an instance of [`Optimizer`](@ref).
-2. the neural network model.
-3. the neural network parameters `ps`.
-4. the data (i.e. an instance of [`DataLoader`](@ref)).
-5. `batch`::[`Batch`](@ref): stores `batch_size` (and optionally `seq_length` and `prediction_window`).
-6. `loss::NetworkLoss`.
-7. the *section* `λY` of the parameters `ps`.
-
-# Implementation
-
-Internally this calls [`optimization_step!`](@ref) for each minibatch.
-
-The number of minibatches can be determined with [`number_of_batches`](@ref):
-
-```jldoctest
-using GeometricMachineLearning
-using GeometricMachineLearning: number_of_batches
-
-data = [1, 2, 3, 4, 5]
-batch = Batch(2)
-dl = DataLoader(data; suppress_info = true)
-
-number_of_batches(dl, batch)
-
-# output
-
-3
-```
-"""
-function optimize_for_one_epoch!(opt::Optimizer,
-        model, ps::NetworkParameters,
-        dl::DataLoader{T},
-        batch::Batch,
-        loss::NetworkLoss,
-        λY) where {T}
-    optimize_for_one_epoch!(opt, model, ps, dl, batch, ZygotePullback(loss), λY)
-end
-
-function optimize_for_one_epoch!(opt::Optimizer,
-        model,
-        ps::NetworkParameters,
-        dl::DataLoader{T},
-        batch::Batch,
-        _pullback::AbstractPullback,
-        λY) where {T}
-    count = 0
-    total_error = T(0)
-    batches = batch(dl)
-    for batch_indices in batches
-        count += 1
-        input_nt_output_nt = convert_input_and_batch_indices_to_array(dl, batch, batch_indices)
-        loss_value, pullback = _pullback(ps, model, input_nt_output_nt)
-        total_error += loss_value
-        dp = _processing(pullback(one(loss_value)))
-        optimization_step!(opt, λY, ps, dp)
-    end
-    total_error / count
-end
-
-function (o::Optimizer)(nn::NeuralNetwork,
-        dl::DataLoader,
-        batch::Batch,
-        n_epochs::Integer,
-        loss::NetworkLoss,
-        _pullback::AbstractPullback = ZygotePullback(loss); show_progress = true)
-    Λ = GlobalSection(params(nn))
-    progress_object = show_progress ?
-                      ProgressMeter.Progress(n_epochs; enabled = true) : nothing
-    loss_array = zeros(float(eltype(dl)), n_epochs)
-    for i in 1:n_epochs
-        loss_array[i] = optimize_for_one_epoch!(
-            o, nn.model, params(nn), dl, batch, _pullback, Λ)
-        show_progress ?
-        ProgressMeter.next!(progress_object; showvalues = [(
-            :TrainingLoss, loss_array[i])]) : nothing
-    end
-
-    loss_array
-end
-
-function (o::Optimizer)(nn::NeuralNetwork{<:TransformerIntegrator}, dl::DataLoader,
-        batch::Batch{:Transformer}, n_epochs::Int = 1; kwargs...)
-    loss = TransformerLoss(batch)
-    o(nn, dl, batch, n_epochs, loss; kwargs...)
-end
-
-function (o::Optimizer)(nn::NeuralNetwork{<:NeuralNetworkIntegrator}, dl::DataLoader,
-        batch::Batch{:FeedForward}, n_epochs::Int = 1; kwargs...)
-    loss = FeedForwardLoss()
-    o(nn, dl, batch, n_epochs, loss; kwargs...)
-end
-
-function (o::Optimizer)(nn::NeuralNetwork{<:AutoEncoder}, dl::DataLoader,
-        batch::Batch{:FeedForward}, n_epochs::Int = 1; kwargs...)
-    loss = AutoEncoderLoss()
-    o(nn, dl, batch, n_epochs, loss; kwargs...)
-end
diff --git a/src/data_loader/train.jl b/src/data_loader/train.jl
new file mode 100644
index 000000000..037c32922
--- /dev/null
+++ b/src/data_loader/train.jl
@@ -0,0 +1,66 @@
+"""
+    train!(nn, opt, dl, batch, n_epochs; loss, pullback = ZygotePullback(loss), show_progress = true)
+
+Train the neural network `nn` with the `GeometricOptimizers.TrainingOptimizer` `opt` on the data of
+the [`DataLoader`](@ref) `dl` for `n_epochs` epochs, and return the loss history.
+
+Each epoch draws the minibatches of `dl` with the [`Batch`](@ref) `batch` and takes one step per
+minibatch: `pullback` computes the loss on the minibatch and its gradient, and
+`GeometricOptimizers.optimization_step!` writes the new parameters into `params(nn)`, with the
+whole gradient, a zero leaf included. Entry `i` of the history is the mean of the minibatch losses
+of epoch `i`, each evaluated before its step. The history has the element type of the loss value,
+so a `Float64` network on `Float32` data gives a `Vector{Float64}`.
+
+`loss` is a `NetworkLoss`. Without it, a [`TransformerIntegrator`](@ref) with a
+`Batch{:Transformer}` trains on [`TransformerLoss`](@ref)`(batch)`, a
+[`NeuralNetworkIntegrator`](@ref) with a `Batch{:FeedForward}` on `FeedForwardLoss()`, and an
+autoencoder with a `Batch{:FeedForward}` on [`AutoEncoderLoss`](@ref)`()`; any other network needs
+`loss`, and `train!` raises an `ArgumentError` without it.
+
+`pullback` is an `AbstractPullback` of `loss`, by default a [`ZygotePullback`](@ref). A Hamiltonian network trains on its symbolic
+gradient with `loss = HNNLoss(arch)` and `pullback = SymbolicPullback(arch)`.
+
+`opt` is built for `params(nn)`, with `TrainingOptimizer(params(nn); algorithm, linesearch,
+retraction)`. It keeps the iteration number, the moments and the global sections, so a second call
+with the same `opt` resumes the training where the first one stopped; the only random numbers
+`train!` draws are those of the minibatch shuffle. `n_epochs < 1` is an `ArgumentError`, raised
+before any step.
+"""
+function train!(nn::NeuralNetwork, opt::TrainingOptimizer, dl::DataLoader, batch::Batch,
+        n_epochs::Integer; loss::NetworkLoss = _default_loss(nn.architecture, batch),
+        pullback::AbstractPullback = ZygotePullback(loss), show_progress::Bool = true)
+    n_epochs ≥ 1 ||
+        throw(ArgumentError("`train!` needs at least one epoch; `n_epochs` is $(n_epochs)."))
+    progress = ProgressMeter.Progress(n_epochs; enabled = show_progress)
+    # A loop and not `map(1:n_epochs) do`: the closure around `_train_epoch!` makes the first call
+    # of a transformer compile several times as long.
+    history = [_train_epoch!(nn, opt, dl, batch, pullback, progress)]
+    for _ in 2:n_epochs
+        push!(history, _train_epoch!(nn, opt, dl, batch, pullback, progress))
+    end
+    history
+end
+
+# The sum takes the type of the first minibatch's loss, so nothing is converted.
+function _train_epoch!(nn, opt, dl, batch, pullback, progress)
+    batches = batch(dl)
+    total = sum(_train_step!(nn, opt, dl, batch, pullback, indices) for indices in batches)
+    value = total / length(batches)
+    ProgressMeter.next!(progress; showvalues = [(:TrainingLoss, value)])
+    value
+end
+
+function _train_step!(nn, opt, dl, batch, pullback, indices)
+    value, back = pullback(params(nn), nn.model,
+        convert_input_and_batch_indices_to_array(dl, batch, indices))
+    optimization_step!(params(nn), opt, _get_contents(back(one(value))))
+    value
+end
+
+_default_loss(::TransformerIntegrator, batch::Batch{:Transformer}) = TransformerLoss(batch)
+_default_loss(::NeuralNetworkIntegrator, ::Batch{:FeedForward}) = FeedForwardLoss()
+_default_loss(arch::AutoEncoder, ::Batch{:FeedForward}) = NetworkLoss(arch)
+function _default_loss(arch, batch)
+    throw(ArgumentError("`train!` has no default loss for a $(nameof(typeof(arch))) with a " *
+                        "$(typeof(batch)); pass the loss as the keyword `loss`."))
+end
diff --git a/src/kernels/kernel_ad_routines/mat_tensor_mul.jl b/src/kernels/kernel_ad_routines/mat_tensor_mul.jl
index aa7970542..34742f6a8 100644
--- a/src/kernels/kernel_ad_routines/mat_tensor_mul.jl
+++ b/src/kernels/kernel_ad_routines/mat_tensor_mul.jl
@@ -73,15 +73,15 @@ function ChainRulesCore.rrule(::typeof(lo_mat_mul), S::AbstractVector{T},
     return C, lo_mat_mul_pullback
 end
 
-function ChainRulesCore.rrule(::typeof(mat_tensor_mul), B::LowerTriangular{T}, A::AbstractArray{
-        T, 3}) where {T}
+function ChainRulesCore.rrule(::typeof(mat_tensor_mul), B::StrictlyLowerTriangular{T},
+        A::AbstractArray{T, 3}) where {T}
     @assert size(A, 1) == B.n
     C = mat_tensor_mul(B, A)
     function lower_triangular_mul_pullback(dC)
         dC = unthunk(dC)
         f̄, dS, dA, _ = rrule(lo_mat_mul, B.S, A, B.n)[2](dC)
 
-        return f̄, LowerTriangular(dS, B.n), dA
+        return f̄, StrictlyLowerTriangular(dS, B.n), dA
     end
     return C, lower_triangular_mul_pullback
 end
@@ -144,15 +144,15 @@ function ChainRulesCore.rrule(::typeof(up_mat_mul), S::AbstractVector{T},
     return C, up_mat_mul_pullback
 end
 
-function ChainRulesCore.rrule(::typeof(mat_tensor_mul), B::UpperTriangular{T}, A::AbstractArray{
-        T, 3}) where {T}
+function ChainRulesCore.rrule(::typeof(mat_tensor_mul), B::StrictlyUpperTriangular{T},
+        A::AbstractArray{T, 3}) where {T}
     @assert size(A, 1) == B.n
     C = mat_tensor_mul(B, A)
     function upper_triangular_mul_pullback(dC)
         dC = unthunk(dC)
         f̄, dS, dA, _ = rrule(up_mat_mul, B.S, A, B.n)[2](dC)
 
-        return f̄, UpperTriangular(dS, B.n), dA
+        return f̄, StrictlyUpperTriangular(dS, B.n), dA
     end
     return C, upper_triangular_mul_pullback
 end
diff --git a/src/kernels/mat_tensor_mul.jl b/src/kernels/mat_tensor_mul.jl
index a2bd3d1bc..3e74ca9a3 100644
--- a/src/kernels/mat_tensor_mul.jl
+++ b/src/kernels/mat_tensor_mul.jl
@@ -119,7 +119,7 @@ function mat_tensor_mul!(C::AbstractArray{<:Number, 3}, A::SymmetricMatrix, B::A
     symmetric_mat_mul!(C, A.S, B, A.n)
 end
 
-########################### LowerTriangular
+########################### StrictlyLowerTriangular
 
 @kernel function lo_mul_kernel!(C::AbstractArray{T, 3}, S::AbstractVector{T},
         B::AbstractArray{T, 3}, ::Int) where {T}
@@ -152,21 +152,21 @@ function lo_mat_mul(S::AbstractVector{T}, B::AbstractArray{T, 3}, n::Int) where
 end
 
 @doc raw"""
-    mat_tensor_mul!(C, A::LowerTriangular, B)
+    mat_tensor_mul!(C, A::StrictlyLowerTriangular, B)
 
-Multiply the lower-triangular matrix `A` onto the tensor `B` from the left and store the result in `C`.
+Multiply the strictly lower-triangular matrix `A` onto the tensor `B` from the left and store the result in `C`.
 
-This performs an efficient multiplication based on the special structure of the lower-triangular matrix `A`.
+This performs an efficient multiplication based on the special structure of the strictly lower-triangular matrix `A`.
 """
-function mat_tensor_mul!(C::AbstractArray{<:Number, 3}, A::LowerTriangular, B::AbstractArray{
-        <:Number, 3})
+function mat_tensor_mul!(C::AbstractArray{<:Number, 3}, A::StrictlyLowerTriangular,
+        B::AbstractArray{<:Number, 3})
     @assert eltype(C) == eltype(A) == eltype(B)
     @assert A.n == size(C, 1) == size(B, 1)
 
     lo_mat_mul!(C, A.S, B, A.n)
 end
 
-####################### UpperTriangular
+####################### StrictlyUpperTriangular
 
 @kernel function up_mul_kernel!(C::AbstractArray{T, 3}, S::AbstractVector{T},
         B::AbstractArray{T, 3}, n::Int) where {T}
@@ -199,14 +199,14 @@ function up_mat_mul(S::AbstractVector{T}, B::AbstractArray{T, 3}, n::Int) where
 end
 
 @doc raw"""
-    mat_tensor_mul!(C, A::UpperTriangular, B)
+    mat_tensor_mul!(C, A::StrictlyUpperTriangular, B)
 
-Multiply the upper-triangular matrix `A` onto the tensor `B` from the left and store the result in `C`.
+Multiply the strictly upper-triangular matrix `A` onto the tensor `B` from the left and store the result in `C`.
 
-This performs an efficient multiplication based on the special structure of the upper-triangular matrix `A`.
+This performs an efficient multiplication based on the special structure of the strictly upper-triangular matrix `A`.
 """
-function mat_tensor_mul!(C::AbstractArray{<:Number, 3}, A::UpperTriangular, B::AbstractArray{
-        <:Number, 3})
+function mat_tensor_mul!(C::AbstractArray{<:Number, 3}, A::StrictlyUpperTriangular,
+        B::AbstractArray{<:Number, 3})
     @assert eltype(C) == eltype(A) == eltype(B)
     @assert A.n == size(C, 1) == size(B, 1)
 
diff --git a/src/layers/resnet.jl b/src/layers/resnet.jl
index c721c1e55..6a6b109b7 100644
--- a/src/layers/resnet.jl
+++ b/src/layers/resnet.jl
@@ -60,18 +60,6 @@ function (d::ResNetLayer{M, M, true})(x::AbstractArray{T, 3}, ps::NamedTuple) wh
     x + d.activation.(mat_tensor_mul(ps.weight, x) .+ ps.bias)
 end
 
-function (d::Dense{M, N, true})(x::AbstractArray{T, 3}, ps::NamedTuple) where {M, N, T}
-    d.σ.(mat_tensor_mul(ps.W, x) .+ ps.b)
-end
-
-function (d::Dense{M, N, false})(x::AbstractArray{T, 3}, ps::NamedTuple) where {M, N, T}
-    d.σ.(mat_tensor_mul(ps.W, x))
-end
-
-function (d::Linear{M, N})(x::AbstractArray{T, 3}, ps::NamedTuple) where {M, N, T}
-    mat_tensor_mul(ps.W, x)
-end
-
 function (d::Union{Dense{M, N}, ResNetLayer{M, N}})(z::QPT, ps::NamedTuple) where {M, N}
     @assert iseven(M) == iseven(N) == true
     @assert size(z.q, 1) * 2 == M
diff --git a/src/layers/volume_preserving_feedforward.jl b/src/layers/volume_preserving_feedforward.jl
index 7822fcda8..ef81bfc5e 100644
--- a/src/layers/volume_preserving_feedforward.jl
+++ b/src/layers/volume_preserving_feedforward.jl
@@ -4,10 +4,10 @@
 Super-type of [`VolumePreservingLowerLayer`](@ref) and [`VolumePreservingUpperLayer`](@ref). The layers do the following: 
 
 ```math
-x \mapsto \begin{cases} \sigma(Lx + b) & \text{where $L$ is }\mathtt{LowerTriangular}, \\ \sigma(Ux + b) & \text{where $U$ is }\mathtt{UpperTriangular}. \end{cases}
+x \mapsto \begin{cases} \sigma(Lx + b) & \text{where $L$ is }\mathtt{StrictlyLowerTriangular}, \\ \sigma(Ux + b) & \text{where $U$ is }\mathtt{StrictlyUpperTriangular}. \end{cases}
 ```
 
-The functor can be applied to a vector, a matrix or a tensor. The special matrices are implemented as [`LowerTriangular`](@extref GeometricOptimizers GeometricOptimizers.LowerTriangular) and [`UpperTriangular`](@extref GeometricOptimizers GeometricOptimizers.UpperTriangular).
+The functor can be applied to a vector, a matrix or a tensor. The special matrices are implemented as [`StrictlyLowerTriangular`](@extref GeometricOptimizers GeometricOptimizers.StrictlyLowerTriangular) and [`StrictlyUpperTriangular`](@extref GeometricOptimizers GeometricOptimizers.StrictlyUpperTriangular).
 """
 abstract type VolumePreservingFeedForwardLayer{M, N, bias} <: AbstractExplicitLayer{M, N} end
 
@@ -36,7 +36,7 @@ x = \begin{pmatrix} 1 \\ 1 \end{pmatrix}
 using GeometricMachineLearning
 
 l = VolumePreservingLowerLayer(2, identity; use_bias=false)
-A = LowerTriangular([1,], 2)
+A = StrictlyLowerTriangular([1,], 2)
 ps = (weight = A, )
 x = ones(eltype(A), 2)
 
@@ -93,7 +93,7 @@ x = \begin{pmatrix} 1 \\ 1 \end{pmatrix}
 using GeometricMachineLearning
 
 l = VolumePreservingUpperLayer(2, identity; use_bias=false)
-A = UpperTriangular([1,], 2)
+A = StrictlyUpperTriangular([1,], 2)
 ps = (weight = A, )
 x = ones(eltype(A), 2)
 
@@ -141,7 +141,7 @@ function initialparameters(
     init_weight!(rng, S)
     init_bias!(rng, b)
 
-    (weight = LowerTriangular(S, M), bias = b)
+    (weight = StrictlyLowerTriangular(S, M), bias = b)
 end
 
 function initialparameters(
@@ -153,7 +153,7 @@ function initialparameters(
     init_weight!(rng, S)
     init_bias!(rng, b)
 
-    (weight = UpperTriangular(S, M), bias = b)
+    (weight = StrictlyUpperTriangular(S, M), bias = b)
 end
 
 function initialparameters(
@@ -163,7 +163,7 @@ function initialparameters(
     S = KernelAbstractions.allocate(backend, T, parameterlength(d))
     init_weight!(rng, S)
 
-    (weight = LowerTriangular(S, M),)
+    (weight = StrictlyLowerTriangular(S, M),)
 end
 
 function initialparameters(
@@ -173,7 +173,7 @@ function initialparameters(
     S = KernelAbstractions.allocate(backend, T, parameterlength(d))
     init_weight!(rng, S)
 
-    (weight = UpperTriangular(S, M),)
+    (weight = StrictlyUpperTriangular(S, M),)
 end
 
 function (d::VolumePreservingFeedForwardLayer{M, M, :bias})(
diff --git a/src/optimizers/optimizer.jl b/src/optimizers/optimizer.jl
index 69d81f32a..3891f7c6f 100644
--- a/src/optimizers/optimizer.jl
+++ b/src/optimizers/optimizer.jl
@@ -1,423 +1,12 @@
-# Optimizer machinery on top of GeometricOptimizers.
-# Kept out of `utils.jl` because it dispatches on `Manifold`, which is defined later.
-
-# Gradient wrapper: stores a pre-computed Euclidean gradient and applies rgrad on manifolds.
-mutable struct _GMLGradient{T, VT} <: GeometricOptimizers.Gradient{T}
-    dp::VT
-end
-
-_gml_rgrad(x::Manifold, dp) = rgrad(x, dp)
-_gml_rgrad(x, dp) = dp
-# `mapparameters` and not `map`: it recurses on the branches, so `_gml_rgrad` is only ever called on
-# leaves, and it rebuilds in the shape of its *first* argument -- a container, which is what
-# `GeometricOptimizers._copyto!(gradient_array(cache), ·)` has a method for. It also normalises its
-# trailing arguments, so `dp` may stay the plain `NamedTuple` the pullback produced.
-_gml_rgrad(x::NetworkParameters, dp) = mapparameters(_gml_rgrad, x, dp)
-
-(g::_GMLGradient{T})(x::NetworkParameters{T}) where {T} = _gml_rgrad(x, g.dp)
-(g::_GMLGradient{T})(x::AbstractArray{T}) where {T} = g.dp
-
-# `Manifold` and not `AbstractArray` alone, because the two are otherwise **ambiguous** and not merely
-# overlapping. `_GMLGradient{T}` is a `GeometricOptimizers.Gradient{T}`, `Manifold{T}` is an
-# `AbstractMatrix{T}`, and `GeometricOptimizers` has `(::Gradient{T})(::Manifold{T})`; so on a bare
-# manifold neither `(::_GMLGradient{T})(::AbstractArray{T})` above nor that one is more specific, and
-# the call is a run-time `MethodError: ... is ambiguous`. This method is more specific than both.
-#
-# The answer it gives is this package's: the gradient is already computed, so the projection is
-# `rgrad(x, dp)` and not upstream's `rgrad(x, reshape(grad(vec(x)), ...))`, which would evaluate an
-# inner gradient this functor does not have. `_gml_rgrad` is where that lives.
-#
-# `_make_optimizer_cache` below asks `_is_layer` first and so keeps a bare `Manifold` from arriving
-# here at all. That ordering is a correctness fix in its own right -- one cache per layer rather than
-# per weight -- but it must not be the *only* thing standing between a caller and an ambiguity.
-(g::_GMLGradient{T})(x::Manifold{T}) where {T} = _gml_rgrad(x, g.dp)
-
-# A bare `AbstractVector`, for the same reason as the `Manifold` method above: `SimpleSolvers` has
-# `(::Gradient{T})(::AbstractVector{T})`, `_GMLGradient{T} <: GeometricOptimizers.Gradient{T}`, and
-# without a method of its own `(::_GMLGradient{T})(::AbstractArray{T})` above and that one are
-# equally specific on a `Vector` -- the call is a run-time `MethodError: ... is ambiguous`. This
-# method is more specific than both: `_GMLGradient{T} <: Gradient{T}` on the first argument and
-# `AbstractVector{T} <: AbstractArray{T}` on the second. The answer is unchanged from the
-# `AbstractArray{T}` method above -- the gradient is already Euclidean, so it is `g.dp`.
-(g::_GMLGradient{T})(x::AbstractVector{T}) where {T} = g.dp
-
-# State for Euclidean (non-manifold) parameters.
-mutable struct GMLEuclideanState{T, AT <: AbstractArray{T}}
-    iterations::Int
-    m₁::AT
-    m₂::AT
-end
-function GMLEuclideanState(x::AbstractArray{T}) where {T}
-    GMLEuclideanState{T, typeof(x)}(0, zero(x), zero(x))
-end
-
-# Adam with a decaying learning rate is GeometricOptimizers', and split the way it belongs: the
-# direction is an `Adam` method, the schedule is a `DecayingStatic` line search. Both names are
-# imported, and `Optimizer` below takes a `DecayingStatic` as its `step_size`.
-
-_is_go_native_method(::GeometricOptimizers.GradientMethod) = true
-_is_go_native_method(::GeometricOptimizers.MomentumMethod) = true
-_is_go_native_method(::GeometricOptimizers.Adam) = true
-_is_go_native_method(::GeometricOptimizers.OptimizerMethod) = false
-
-function _adapt_method_to_T(method::GeometricOptimizers.Adam, ::Type{T}) where {T}
-    GeometricOptimizers.Adam(T; β₁ = T(method.β₁), β₂ = T(method.β₂), δ = T(method.δ))
-end
-function _adapt_method_to_T(method::GeometricOptimizers.MomentumMethod, ::Type{T}) where {T}
-    GeometricOptimizers.MomentumMethod(T(method.α))
-end
-_adapt_method_to_T(method, ::Type) = method
-
-# Whether one `GeometricOptimizers` cache covers `x` whole.
-#
-# A **layer** is a flat `NamedTuple` of arrays, and it is exactly what one cache is for. A `NamedTuple`
-# whose values are branches is a *subtree* and gets one cache per layer instead, which is why the test
-# is flatness and not merely "a `NamedTuple`".
-#
-# The rule lives here rather than upstream because a whole set of parameters reaches
-# `GeometricOptimizers` only as a `NetworkParameters`, and a container says nothing about how deep the
-# tree beneath it is. Splitting a network into layers is this package's decision, so this package makes
-# it; [`_as_go_solution`](@ref) then does the wrap.
-#
-# A layer whose weights do not share an element type is still one cache, with `T` the promotion over
-# them. `_adapt_method_to_T` reads that promotion, so the cache is built for the element type the
-# layer actually has rather than for one its weights are required to agree on.
-# `!isempty` and not `all` alone: `all` over an empty collection is vacuously true, so without it an
-# empty set would be one cache's worth -- and `OptimizerCache` would then be asked for the element
-# type of a set that has no leaves to promote. An empty set is zero caches' worth, so it descends and
-# yields an empty tree of caches, which is what the step below iterates over zero times.
-_is_layer(x::NamedTuple) = _all_leaves(values(x))
-_is_layer(x::NetworkParameters) = _all_leaves(values(x))
-_is_layer(_) = false
-
-_all_leaves(vs) = !isempty(vs) && all(v -> v isa AbstractArray, vs)
-
-function _use_go_cache(method, x)
-    _is_go_native_method(method) &&
-        (x isa GeometricOptimizers.OptimizerSolution || _is_layer(x))
-end
-
-"""
-    _as_go_solution(x)
-
-`x` in the shape `GeometricOptimizers` takes a solution in.
-
-A layer given as a bare `NamedTuple` is wrapped; anything already in the right shape — a container, a
-bare `Manifold`, an `AbstractArray` — is passed through. The wrap **shares the leaf arrays**, so an
-in-place optimizer step writes through to the network's own weights and nothing has to be copied back.
-"""
-_as_go_solution(x::NetworkParameters) = x
-_as_go_solution(x) = _is_layer(x) ? NetworkParameters(x) : x
-
-# A `NetworkParameters` is always a tree of layers to descend into, never a single
-# `GeometricOptimizers` leaf, so its branch comes *first* — ahead of `_use_go_cache`.
-#
-# That ordering is **load-bearing**. `NetworkParameters` *is* one of the types
-# `GeometricOptimizers.OptimizerSolution` unions, so `_use_go_cache` is true at the root: without the
-# test above, a whole network would get one cache instead of one per layer -- silently, and with
-# `_leaf_optim_step!` handed the entire tree. Asking the structural question before the capability
-# question is what makes the shape of the cache depend on the shape of the parameters rather than on
-# which types upstream happens to accept.
-#
-# One cache for the whole network is the better end state, and it is **not** what this does.
-# `_GMLGradient` does take a `NetworkParameters`, but that is one wrapped *layer* and not the tree,
-# because the container branch above splits the network first. Getting to one cache means dropping
-# `_tree_optim_step!` and handing the whole tree over, which is a change of behaviour -- one
-# `GlobalSection` tree and one `Q` across every layer -- so it wants its own release. Nothing here
-# depends on it.
-#
-# **"Is this one cache's worth?" is asked before "is this a tree?"**, and a *flat* set answers yes to
-# both. `_is_layer` is that question: a set whose values are all leaves is a layer, whether it arrives
-# wrapped or as a branch, and one `GeometricOptimizers` cache is exactly what a layer is for. Asking
-# the tree question first would descend into such a set and give every individual weight its own cache
-# -- which for a manifold weight is not merely wasteful but wrong, since a bare `Manifold` then reaches
-# the gradient functor where a whole layer should have.
-#
-# The tree branch comes second and covers both carriers, because a network is a tree of layers whether
-# it is wrapped or not.
-function _make_optimizer_cache(method, x)
-    if _is_go_native_method(method) && _is_layer(x)
-        GeometricOptimizers.OptimizerCache(_adapt_method_to_T(method, parameter_eltype(x)),
-            _as_go_solution(x))
-    elseif x isa NetworkParameters || x isa NamedTuple
-        NamedTuple{keys(x)}(Tuple(_make_optimizer_cache(method, x[k]) for k in keys(x)))
-    elseif _use_go_cache(method, x)
-        GeometricOptimizers.OptimizerCache(_adapt_method_to_T(method, parameter_eltype(x)), x)
-    else
-        GMLEuclideanState(x)
-    end
-end
-
-function _make_optimizer_state(method, x)
-    if _is_go_native_method(method) && _is_layer(x)
-        GeometricOptimizers.OptimizerState(_adapt_method_to_T(method, parameter_eltype(x)),
-            _as_go_solution(x))
-    elseif x isa NetworkParameters || x isa NamedTuple
-        NamedTuple{keys(x)}(Tuple(_make_optimizer_state(method, x[k]) for k in keys(x)))
-    elseif _use_go_cache(method, x)
-        GeometricOptimizers.OptimizerState(_adapt_method_to_T(method, parameter_eltype(x)), x)
-    else
-        GMLEuclideanState(x)
-    end
-end
-
-"""
-    Optimizer(method, nn; retraction, step_size)
-    Optimizer(nn; algorithm, linesearch, retraction)
-
-Optimizer state combining a `GeometricOptimizers` method with the parameters of a neural network.
-
-`step_size` is either a number — a fixed learning rate — or a
-`GeometricOptimizers.DecayingStatic`, a learning rate that decays geometrically with the iteration
-number. The second form is the one `GeometricOptimizers` uses itself, so a method paired with a
-schedule splats straight in:
-
-```julia
-opt = Optimizer(nn; AdamOptimizerWithDecay(n_epochs, Float32)...)
-```
-
-# Extended help
-
-The step size is a property of the optimizer and not of the method: the same `Adam()` trains at any
-learning rate. That is the split `GeometricOptimizers` makes — the method supplies a direction, a
-`SimpleSolvers.LinesearchMethod` supplies how far to go along it — and `step_size` is GML's half of
-it for a training loop, which has no objective function for a real line search to evaluate.
-"""
-mutable struct Optimizer{MT <: GeometricOptimizers.OptimizerMethod, CT, ST, RT, SST}
-    method::MT
-    cache::CT
-    state::ST
-    retraction::RT
-    step_size::SST
-    iterations::Int
-end
-
-_default_step_size(::GeometricOptimizers.Adam) = 1e-3
-_default_step_size(::GeometricOptimizers.OptimizerMethod) = 1e-2
-
-_step_size(η::Real, ::Int) = Float64(η)
-# `t` and not `t - 1`: `optimization_step!` increments before it asks, so the first step of a solve
-# is `α(1) = γη₁`. That is what `DecayingStatic` means by iteration `t`: `solve!` calls
-# `increase_iteration_number!` before `solver_step!`.
-_step_size(ls::DecayingStatic, t::Int) = Float64(GeometricOptimizers.step_size(ls, t))
-
-_current_step_size(opt::Optimizer, t::Int) = _step_size(opt.step_size, t)
-
-_optimizer_step_size(η::Real) = Float64(η)
-# Anything that is not a plain number goes through the same funnel as the `linesearch` keyword below,
-# so that the two entry points accept the same things: `step_size = Static(α)` is a fixed learning
-# rate on both, and anything else reports the `ArgumentError` that explains why a real line search
-# has nothing to search along here, instead of a `MethodError` naming this helper.
-_optimizer_step_size(ls) = _step_size_from_linesearch(ls)
-
-function Optimizer(method::GeometricOptimizers.OptimizerMethod, nn::NeuralNetwork;
-        retraction = GeometricOptimizers.cayley,
-        step_size = _default_step_size(method))
-    Optimizer(method, params(nn); retraction = retraction, step_size = step_size)
-end
-
-function Optimizer(method::GeometricOptimizers.OptimizerMethod,
-        ps::NetworkParameters;
-        retraction = GeometricOptimizers.cayley,
-        step_size = _default_step_size(method))
-    Optimizer(method, _make_optimizer_cache(method, ps), _make_optimizer_state(method, ps),
-        retraction, _optimizer_step_size(step_size), 0)
-end
-
-# The keyword form, so that the `(algorithm, linesearch)` pairing `GeometricOptimizers` returns from
-# `AdamOptimizerWithDecay` splats in unchanged. `linesearch` is the step size under the name
-# upstream gives it; a `Static` carries its own `α`, which is then the fixed learning rate.
-function Optimizer(nn_or_ps::Union{NeuralNetwork, NetworkParameters};
-        algorithm::GeometricOptimizers.OptimizerMethod,
-        linesearch = nothing,
-        retraction = GeometricOptimizers.cayley,
-        step_size = linesearch === nothing ? _default_step_size(algorithm) :
-                    _step_size_from_linesearch(linesearch))
-    Optimizer(algorithm, nn_or_ps; retraction = retraction, step_size = step_size)
-end
-
-_step_size_from_linesearch(ls::DecayingStatic) = ls
-_step_size_from_linesearch(ls::GeometricOptimizers.Static) = Float64(ls.α)
-function _step_size_from_linesearch(ls)
-    throw(ArgumentError(
-        "`Optimizer` takes a fixed step size or a `DecayingStatic` schedule, not a $(typeof(ls)). " *
-        "A training loop evaluates its loss on one batch at a time and has no objective for a line " *
-        "search to search along; use `GeometricOptimizers.Optimizer` with an `OptimizerProblem` for " *
-        "that."))
-end
-
-# Euclidean update rules
-function _euclidean_update!(x::AbstractArray{T}, dx::AbstractArray,
-        state::GMLEuclideanState, ::GeometricOptimizers.GradientMethod, step_size) where {T}
-    x .-= T(step_size) .* dx
-end
-function _euclidean_update!(x::AbstractArray{T}, dx::AbstractArray,
-        state::GMLEuclideanState{T}, method::GeometricOptimizers.MomentumMethod, step_size) where {T}
-    # `p ← αp + ∇L`, the classic momentum recursion. The decay belongs on `p` and not on `∇L`:
-    # `p ← p + α∇L` is an undamped accumulator that grows without bound for a constant gradient
-    # instead of saturating at `∇L/(1 - α)`. Same recursion as GO's `update!(::MomentumState, ...)`.
-    state.m₁ .= T(method.α) .* state.m₁ .+ dx
-    x .-= T(step_size) .* state.m₁
-end
-function _euclidean_update!(x::AbstractArray{T}, dx::AbstractArray,
-        state::GMLEuclideanState{T}, method::GeometricOptimizers.Adam, step_size) where {T}
-    t = state.iterations
-    _t = t + 1
-    β₁, β₂, δ = T(method.β₁), T(method.β₂), T(method.δ)
-    # the first factor is `(β - β^t)/(1 - β^t)`, not `β/(1 - β^t)`: the latter is ~100x too large
-    # at t = 2 and compounds every step, which inflates the second moment until the update vanishes
-    fac₁₁ = (β₁-β₁^_t)/(1-β₁^_t)
-    fac₁₂ = (1-β₁)/(1-β₁^_t)
-    fac₂₁ = (β₂-β₂^_t)/(1-β₂^_t)
-    fac₂₂ = (1-β₂)/(1-β₂^_t)
-    state.m₁ .= fac₁₁ .* state.m₁ .+ fac₁₂ .* dx
-    state.m₂ .= fac₂₁ .* state.m₂ .+ fac₂₂ .* dx .^ 2
-    x .-= T(step_size) .* state.m₁ ./ (sqrt.(state.m₂) .+ δ)
-end
-# `_euclidean_update!` has three methods and not four: Adam with a decaying learning rate *is*
-# Adam — only `step_size` differs, and that comes in as an argument — so the `Adam` method above
-# serves it too.
-
-function _go_update_leaf!(cache, state, local_grad,
-        method::GeometricOptimizers.Adam, ps_leaf)
-    GeometricOptimizers.update!(cache, state, local_grad, method, ps_leaf)
-end
-
-function _go_update_leaf!(cache, state, local_grad,
-        method::GeometricOptimizers.MomentumMethod, ps_leaf)
-    GeometricOptimizers.update!(cache, state, local_grad, method, ps_leaf)
-end
-
-function _go_update_leaf!(cache, state, local_grad,
-        method::GeometricOptimizers.OptimizerMethod, ps_leaf)
-    T = parameter_eltype(ps_leaf)
-    GeometricOptimizers.update!(cache, state, local_grad,
-        GeometricOptimizers.NoHessian{T}(), ps_leaf)
-end
-
-# GO-managed leaf step (manifolds, vectors, whole layers)
-#
-# `ps` and not `ps_leaf` from here on: a layer is wrapped for the same reason its cache is, and the
-# wrap shares the arrays, so `_copyto!(ps, solution(cache))` below writes into the weights the caller
-# holds. A bare `Manifold` or `AbstractArray` passes through untouched.
-function _leaf_optim_step!(cache::GeometricOptimizers.OptimizerCache,
-        state::GeometricOptimizers.OptimizerState,
-        dp_leaf, ps_leaf, λY_leaf, method, retraction, step_size)
-    T = parameter_eltype(ps_leaf)
-    ps = _as_go_solution(ps_leaf)
-    local_grad = _GMLGradient{T, typeof(dp_leaf)}(dp_leaf)
-    adapted = _adapt_method_to_T(method, T)
-    state.iterations += 1
-    _go_update_leaf!(cache, state, local_grad, adapted, ps)
-    GeometricOptimizers._rmul!(GeometricOptimizers.direction(cache), T(step_size))
-    GeometricOptimizers.update_section!(GeometricOptimizers.section(cache),
-        GeometricOptimizers.section(state),
-        GeometricOptimizers.direction(cache),
-        retraction)
-    GeometricOptimizers._copyto!(GeometricOptimizers.solution(cache),
-        GeometricOptimizers.section(cache))
-    GeometricOptimizers._copyto!(ps, GeometricOptimizers.solution(cache))
-    GeometricOptimizers._copyto!(λY_leaf, GeometricOptimizers.section(cache))
-    # `section(cache)` is `update_section!(section(state), direction, retraction)`, so copying it is
-    # the same thing as retracting a second time -- and a retraction on a manifold is `O(N³)` where
-    # the copy is `O(N²)`.
-    GeometricOptimizers._copyto!(GeometricOptimizers.section(state),
-        GeometricOptimizers.section(cache))
-    if state isa GeometricOptimizers.AdamState
-        GeometricOptimizers._copyto!(GeometricOptimizers.first_moment(state),
-            GeometricOptimizers.first_moment(cache))
-        GeometricOptimizers._copyto!(GeometricOptimizers.second_moment(state),
-            GeometricOptimizers.second_moment(cache))
-    elseif state isa GeometricOptimizers.MomentumState
-        # `p ← αp + ∇L`; see the note in `_euclidean_update!` for the momentum method. This has to
-        # match what `update!(::MomentumCache, ...)` anticipated when it formed the direction.
-        GeometricOptimizers._rmul!(GeometricOptimizers.momentum(state), adapted.α)
-        GeometricOptimizers._add!(GeometricOptimizers.momentum(state),
-            GeometricOptimizers.gradient_array(cache))
-    end
-    nothing
-end
-
-# Euclidean leaf step (plain AbstractArray params)
-function _leaf_optim_step!(cache::GMLEuclideanState, state::GMLEuclideanState,
-        dp_leaf, ps_leaf, λY_leaf, method, retraction, step_size)
-    _euclidean_update!(ps_leaf, dp_leaf, state, method, step_size)
-    state.iterations += 1
-    nothing
-end
-
-# Whether `λY` is a tree of sections to descend into, or a single `GlobalSection` standing in for a
-# whole subtree. Named rather than written as an `isa` union: it is one question, asked once, and the
-# answer decides whether a layer gets its own section or the whole tree. A section tree is a plain
-# `NamedTuple` today, because `GeometricOptimizers`' `GlobalSection(::NetworkParameters)` unwraps the
-# container; the container arm is here so that a change there is a `MethodError` and not a silent
-# hand-off of the whole tree to every layer.
-_is_section_tree(::NamedTuple) = true
-_is_section_tree(::NetworkParameters) = true
-_is_section_tree(_) = false
-
-# Recursive dispatcher over the *cache* tree.
-#
-# Deliberately hand-written rather than `NeuralNetworkParameters.foreachparameters`, which walks a
-# parameter tree. Two reasons, and both are load-bearing:
-#
-#  - The recursion is keyed on `caches`, and stops where the cache stops. A cache sits at the *layer*
-#    level, so a layer's `NamedTuple` of weights arrives at `_leaf_optim_step!` whole, which is what
-#    one `GeometricOptimizers` cache is for. `foreachparameters` recurses on the leaf protocol and
-#    would descend past the layer into the individual weights, re-pairing every cache with the wrong
-#    object.
-#  - `λY` is broadcast, not zipped: a single `GlobalSection` may stand in for a whole subtree, which
-#    the ternary below expresses. `foreachparameters` has no such rule — it takes `values` of each
-#    trailing argument, so a bare `GlobalSection` beside a `NamedTuple` of caches is a `MethodError`.
-#
-# The `nothing` skip is the one thing the two have in common, and it is one line here.
-#
-# The `λY` test names both container types for the same reason `_make_optimizer_cache` asks the
-# structural question first: a section tree is something to descend into, whichever type carries it.
-# Only a `NamedTuple` ever arrives: `GeometricOptimizers`' own
-# `GlobalSection(ps::NetworkParameters) = GlobalSection(params(ps))` unwraps the container, so a
-# section tree is a plain `NamedTuple` whether the parameters are wrapped or not. Should that ever
-# return a *container* of sections instead, `isa NamedTuple` alone would be false here and every layer
-# would be handed the whole tree instead of its own section.
-function _tree_optim_step!(caches, states, dp, ps, λY, method, retraction, step_size)
-    if caches isa NamedTuple
-        for k in keys(caches)
-            dp_k = dp[k]
-            dp_k === nothing && continue
-            λY_k = _is_section_tree(λY) ? λY[k] : λY
-            _tree_optim_step!(caches[k], states[k], dp_k, ps[k], λY_k,
-                method, retraction, step_size)
-        end
-    else
-        _leaf_optim_step!(caches, states, dp, ps, λY, method, retraction, step_size)
-    end
-    nothing
-end
-
-"""
-    optimization_step!(opt, λY, ps, dp)
-
-Apply one optimization step to the parameters `ps` and their gradient `dp`, with the method and step
-size the [`Optimizer`](@ref) `opt` carries.
-
-`λY` is a `GlobalSection` of `ps` (or a `NamedTuple` of them). Note that it is an *output*
-here: the section the optimizer carries from step to step lives in `opt.state`, and `λY` is written
-so that callers who inspect it see the updated section. It therefore has to be allocated once and
-reused, not rebuilt per step -- rebuilding it costs a QR decomposition per manifold weight.
-
-The step counter is incremented *before* the step size is read, so the first step of a run is step 1.
-This matters for a decaying `step_size` and is how `GeometricOptimizers` counts too.
-"""
-function optimization_step!(opt::Optimizer, λY, ps, dp)
-    # The increment comes *first*, so the first step of a run is step 1. It matters only for a
-    # decaying `step_size`, and there it matters: reading the schedule before incrementing takes
-    # `α(0) = η₁`, one whole step above what `DecayingStatic` and `GeometricOptimizers.solve!`
-    # take. `solve!` counts the same way, by calling `increase_iteration_number!` before
-    # `solver_step!`.
-    opt.iterations += 1
-    step = _current_step_size(opt, opt.iterations)
-    _tree_optim_step!(opt.cache, opt.state, dp, ps, λY, opt.method, opt.retraction, step)
-    nothing
-end
-
-check(::Optimizer) = nothing
+# The optimizers are GeometricOptimizers': `train!` takes its `TrainingOptimizer`, built on the
+# parameters of a network with `TrainingOptimizer(params(nn); algorithm, linesearch, retraction)`.
+# They are re-exported here, so that `using GeometricMachineLearning` alone trains a network.
+export TrainingOptimizer
+export GradientMethod, MomentumMethod, Adam, AdamWithEuclideanDecay
+export AdamOptimizerWithDecay, DecayingStatic
+export GlobalSection, global_section, apply_section, apply_section!, update_section!
+export global_rep
+export Geodesic, Cayley
+export geodesic, cayley
+export retraction
+export update!
diff --git a/test/Project.toml b/test/Project.toml
index 8a7377119..3359d574b 100644
--- a/test/Project.toml
+++ b/test/Project.toml
@@ -27,13 +27,14 @@ GeometricMachineLearning = {path = ".."}
 [compat]
 Aqua = "0.8"
 ChainRulesTestUtils = "1"
-# A tight lower bound. `test/quality/aqua.jl` names four of `test_explicit_imports`' seven keyword switches,
+# A tight lower bound. `test/quality/aqua.jl` names three of `test_explicit_imports`' seven keyword switches,
 # and 1.15 is the version those switches were read from; a wider bound would be a guess about
 # earlier ones.
 ExplicitImports = "1.15"
 FiniteDifferences = "0.12"
 GeometricIntegrators = "0.18.2"
 GeometricProblems = "0.8, 0.9"
+HDF5 = "0.16, 0.17"
 Metal = "1.10"
 Printf = "1"
 SafeTestsets = "0.1"
diff --git a/test/architectures/lagrangian_neural_network_tests.jl b/test/architectures/lagrangian_neural_network_tests.jl
index a778f5391..660578e35 100644
--- a/test/architectures/lagrangian_neural_network_tests.jl
+++ b/test/architectures/lagrangian_neural_network_tests.jl
@@ -154,8 +154,8 @@ function test_lnn_training(; n = 1, n_epochs = 30)
     # Start the training from a different draw, so the target is not already met.
     nn = NeuralNetwork(arch)
     dl = DataLoader(input, output; suppress_info = true)
-    optimizer = Optimizer(Adam(), nn)
-    loss_array = optimizer(nn, dl, Batch(10), n_epochs, loss; show_progress = false)
+    optimizer = TrainingOptimizer(nn.params; algorithm = Adam())
+    loss_array = train!(nn, optimizer, dl, Batch(10), n_epochs; loss, show_progress = false)
 
     @test loss_array[end] < loss_array[1]
 end
diff --git a/test/architectures/sae_error_lower_than_psd_error.jl b/test/architectures/sae_error_lower_than_psd_error.jl
index a76497db7..ade5f360e 100644
--- a/test/architectures/sae_error_lower_than_psd_error.jl
+++ b/test/architectures/sae_error_lower_than_psd_error.jl
@@ -17,8 +17,8 @@ function test_accuracy(N::Integer, n::Integer; n_epochs::Integer = 100)
 
     sae_nn = NeuralNetwork(SymplecticAutoencoder(N, n; n_encoder_layers = 5, n_decoder_layers = 5))
 
-    o = Optimizer(Adam(), sae_nn)
-    sae_error = o(sae_nn, dl, Batch(10), n_epochs; show_progress = false)[end]
+    o = TrainingOptimizer(sae_nn.params; algorithm = Adam())
+    sae_error = train!(sae_nn, o, dl, Batch(10), n_epochs; show_progress = false)[end]
 
     @test sae_error < psd_error
 end
diff --git a/test/architectures/symplectic_autoencoder_tests.jl b/test/architectures/symplectic_autoencoder_tests.jl
index 56f83a4c3..2e0c606a3 100644
--- a/test/architectures/symplectic_autoencoder_tests.jl
+++ b/test/architectures/symplectic_autoencoder_tests.jl
@@ -10,8 +10,8 @@ function test_accuracy(N::Integer, n::Integer; tol::Real = 0.35, n_epochs::Integ
 
     sae_nn = NeuralNetwork(SymplecticAutoencoder(N, n))
 
-    o = Optimizer(Adam(), sae_nn)
-    sae_error = o(sae_nn, dl, Batch(10), n_epochs; show_progress = false)[end]
+    o = TrainingOptimizer(sae_nn.params; algorithm = Adam())
+    sae_error = train!(sae_nn, o, dl, Batch(10), n_epochs; show_progress = false)[end]
 
     @test sae_error < tol
 end
@@ -39,8 +39,8 @@ function test_symplecticity(N::Integer, n::Integer)
 
     # test if it's still symplectic after training
     dl = DataLoader(rand(N, 10 * N); autoencoder = true)
-    o = Optimizer(Adam(), sae_nn)
-    o(sae_nn, dl, Batch(10), 10; show_progress = false)
+    o = TrainingOptimizer(sae_nn.params; algorithm = Adam())
+    train!(sae_nn, o, dl, Batch(10), 10; show_progress = false)
     sympl_mat = jacobian(vec -> sae_decoder(vec), test_vector)[1]
     @test PoissonTensor(n) ≈ sympl_mat' * PoissonTensor(N) * sympl_mat
 end
diff --git a/test/arrays/triangular.jl b/test/arrays/triangular.jl
index 06226a1b8..abb754f66 100644
--- a/test/arrays/triangular.jl
+++ b/test/arrays/triangular.jl
@@ -12,8 +12,8 @@ Random.seed!(1234)
 # third axis of a tensor with `mat_tensor_mul`, and the pullback of that kernel.
 
 function triangular_tensor_multiplication_test(T = Float64, n::Int = 5)
-    Aₗ = rand(LowerTriangular{T}, n)
-    Aᵤ = rand(UpperTriangular{T}, n)
+    Aₗ = rand(StrictlyLowerTriangular{T}, n)
+    Aᵤ = rand(StrictlyUpperTriangular{T}, n)
 
     B = rand(T, n, n, n)
     AₗB = mat_tensor_mul(Aₗ, B)
@@ -25,8 +25,8 @@ function triangular_tensor_multiplication_test(T = Float64, n::Int = 5)
 end
 
 function triangular_tensor_multiplication_pullback_test(T = Float64, n::Int = 5)
-    Aₗ = rand(LowerTriangular{T}, n)
-    Aᵤ = rand(UpperTriangular{T}, n)
+    Aₗ = rand(StrictlyLowerTriangular{T}, n)
+    Aᵤ = rand(StrictlyUpperTriangular{T}, n)
 
     B = rand(T, n, n, n)
     C_diff = rand(T, n, n, n)
diff --git a/test/data_loader/batch_data_loader_qp_test.jl b/test/data_loader/batch_data_loader_qp_test.jl
index fbacafaca..83fa6436e 100644
--- a/test/data_loader/batch_data_loader_qp_test.jl
+++ b/test/data_loader/batch_data_loader_qp_test.jl
@@ -40,14 +40,14 @@ function test_data_loader(dim = 2, number_of_time_steps = 100,
     nn2 = NeuralNetwork(arch2, CPU(), T)
 
     batch = Batch(batch_size)
-    o₁ = Optimizer(GradientOptimizer(), nn1)
-    o₂ = Optimizer(GradientOptimizer(), nn2)
+    o₁ = TrainingOptimizer(nn1.params; algorithm = GradientMethod())
+    o₂ = TrainingOptimizer(nn2.params; algorithm = GradientMethod())
 
     params₁ = deepcopy(nn1.params)
     params₂ = deepcopy(nn2.params)
 
-    loss_array₁ = o₁(nn1, dl1, batch, n_epochs; show_progress = false)
-    loss_array₂ = o₂(nn2, dl2, batch, n_epochs; show_progress = false)
+    loss_array₁ = train!(nn1, o₁, dl1, batch, n_epochs; show_progress = false)
+    loss_array₂ = train!(nn2, o₂, dl2, batch, n_epochs; show_progress = false)
 
     # Both shapes reach the training loop and a step is taken for each. The assertion is that the
     # parameters moved, not that the loss fell: the data are random, so a fall over `n_epochs`
diff --git a/test/data_loader/data_loader_for_input_and_output.jl b/test/data_loader/data_loader_for_input_and_output.jl
index 721ac1694..6385449c4 100644
--- a/test/data_loader/data_loader_for_input_and_output.jl
+++ b/test/data_loader/data_loader_for_input_and_output.jl
@@ -16,10 +16,10 @@ nepochs = 2000
 model = Chain(Dense(1, nwidth), Dense(nwidth, 1))
 nn = NeuralNetwork(model, Float32)
 dl = DataLoader(xsamples, ysamples)
-o = Optimizer(Adam(), nn)
+o = TrainingOptimizer(nn.params; algorithm = Adam())
 batch = Batch(nbatch, 1, 1)
 
 loss = FeedForwardLoss()
 
-loss_array = o(nn, dl, batch, nepochs, loss; show_progress = false)
+loss_array = train!(nn, o, dl, batch, nepochs; loss, show_progress = false)
 @test loss_array[end] < 0.9
diff --git a/test/data_loader/data_loader_optimization_step.jl b/test/data_loader/data_loader_optimization_step.jl
index 9a9405bd0..65857ce1e 100644
--- a/test/data_loader/data_loader_optimization_step.jl
+++ b/test/data_loader/data_loader_optimization_step.jl
@@ -1,4 +1,5 @@
 using GeometricMachineLearning, Test, Zygote
+using GeometricOptimizers: optimization_step!
 import Random
 
 Random.seed!(1234)
@@ -16,10 +17,9 @@ function test_data_loader(sys_dim, n_time_steps, n_params, T = Float32)
     loss = GeometricMachineLearning.TransformerLoss(n_time_steps)
     dx = Zygote.gradient(ps -> loss(model, ps, dl.input, dl.input), ps)[1]
     ps_copy = deepcopy(ps)
-    o = Optimizer(GradientOptimizer(), ps)
-    λY = GlobalSection(ps)
-    optimization_step!(o, λY, ps, dx)
-    @test ps !== ps_copy
+    o = TrainingOptimizer(ps; algorithm = GradientMethod())
+    optimization_step!(ps, o, dx)
+    @test ps != ps_copy
 end
 
 test_data_loader(4, 200, 1000)
diff --git a/test/data_loader/optimizer_functor_with_adam.jl b/test/data_loader/optimizer_functor_with_adam.jl
index ad449788e..1dd06cbe1 100644
--- a/test/data_loader/optimizer_functor_with_adam.jl
+++ b/test/data_loader/optimizer_functor_with_adam.jl
@@ -49,9 +49,8 @@ function test_optimization_with_adam(; T = Float32, input_dim = 9, n_patches = 4
 
     loss₁ = loss(model, ps, dl.input, dl.output)
 
-    opt = Optimizer(Adam(), nn_obj)
-    λY = GlobalSection(ps)
-    loss_average = optimize_for_one_epoch!(opt, model, ps, dl, batch, loss, λY)
+    opt = TrainingOptimizer(nn_obj.params; algorithm = Adam())
+    loss_average = only(train!(nn_obj, opt, dl, batch, 1; loss, show_progress = false))
 
     loss₃ = loss(model, ps, dl.input, dl.output)
 
diff --git a/test/data_loader/train.jl b/test/data_loader/train.jl
new file mode 100644
index 000000000..540bc7aef
--- /dev/null
+++ b/test/data_loader/train.jl
@@ -0,0 +1,160 @@
+using GeometricMachineLearning
+using GeometricMachineLearning: params
+using Test
+import Random
+
+include("../helpers/eltypes.jl")
+
+# The target is in the model class, so the loss goes to zero; the factor 100 leaves margin in
+# `Float32`.
+@testset "train! fits a linear map ($T)" for T in REAL_ELTYPES
+    Random.seed!(1)
+    A = T[1 2; -1 3]
+    x = randn(T, 2, 40)
+    dl = DataLoader(x, A * x; suppress_info = true)
+    nn = NeuralNetwork(Chain(Dense(2, 2, identity)), T)
+    opt = TrainingOptimizer(nn.params; algorithm = Adam(), linesearch = T(0.01))
+    history = train!(
+        nn, opt, dl, Batch(8), 200; loss = FeedForwardLoss(), show_progress = false)
+    @test eltype(history) === T
+    @test length(history) == 200
+    @test history[end] < history[1] / 100
+end
+
+# The iteration number, the moments and the global sections live in `opt` only, so two calls of 5
+# epochs are one call of 10. Seeding before the network and the optimizer makes the minibatches and
+# the random completions of the sections the same draws in both runs.
+@testset "a second call of train! resumes the first ($T)" for T in REAL_ELTYPES
+    x = randn(Random.Xoshiro(2), T, 6, 30)
+    dl = DataLoader(x, x[1:2, :]; suppress_info = true)
+    function trained(epochs...)
+        Random.seed!(3)
+        nn = NeuralNetwork(Chain(StiefelLayer(6, 3), Dense(3, 2, tanh)), T)
+        opt = TrainingOptimizer(nn.params; algorithm = Adam())
+        for n in epochs
+            train!(
+                nn, opt, dl, Batch(10), n; loss = FeedForwardLoss(), show_progress = false)
+        end
+        nn, opt
+    end
+    nn₁, opt₁ = trained(5, 5)
+    nn₂, opt₂ = trained(10)
+    @test params(nn₁).L1.weight isa StiefelManifold{T}
+    @test params(nn₁) == params(nn₂)
+    @test opt₁.state.iterations == opt₂.state.iterations == 30
+end
+
+# The history has the type of the loss value, and nothing is converted: the network's element type
+# wins over the data's.
+@testset "the history has the element type of the loss" begin
+    for (Tnn, Tdata, Thistory) in ((Float64, Float32, Float64), (Float32, Float32, Float32),
+        (Float32, Int, Float64))
+        Random.seed!(4)
+        x = Tdata <: Integer ? rand(Tdata(-3):Tdata(3), 2, 10) : rand(Tdata, 2, 10)
+        dl = DataLoader(x, x; suppress_info = true)
+        nn = NeuralNetwork(Chain(Dense(2, 2, identity)), Tnn)
+        history = train!(
+            nn, TrainingOptimizer(nn.params), dl, Batch(5), 2; loss = FeedForwardLoss(),
+            show_progress = false)
+        @test eltype(history) === Thistory
+    end
+end
+
+# A step of size `floatmin(T)` rounds away against parameters of order one, so every minibatch loss is
+# the loss at `p₀`, and entry 1 of the history is the mean of three numbers that are computed here
+# from the same minibatches.
+@testset "the history holds the mean of the minibatch losses ($T)" for T in REAL_ELTYPES
+    dl = DataLoader(randn(Random.Xoshiro(6), T, 2, 12), randn(Random.Xoshiro(7), T, 2, 12);
+        suppress_info = true)
+    batch = Batch(4)
+    loss = FeedForwardLoss()
+    nn = NeuralNetwork(Chain(Dense(2, 2, tanh)), T)
+    opt = TrainingOptimizer(nn.params; algorithm = GradientMethod(), linesearch = floatmin(T))
+    p₀ = deepcopy(params(nn))
+    Random.seed!(8)
+    losses = map(batch(dl)) do indices
+        data = GeometricMachineLearning.convert_input_and_batch_indices_to_array(
+            dl, batch, indices)
+        loss(nn.model, params(nn), data...)
+    end
+    Random.seed!(8)
+    history = train!(nn, opt, dl, batch, 1; loss, show_progress = false)
+    @test params(nn) == p₀
+    @test eltype(losses) === eltype(history) === T
+    @test length(losses) == 3
+    @test !(losses[3] ≈ sum(losses) / 3)
+    @test history[1] ≈ sum(losses) / 3
+end
+
+# `train!` steps with the gradient of the `pullback` it is given. One epoch of one minibatch with
+# `GradientMethod` moves the parameters by `-η` times the symbolic gradient at `p₀`, and the history
+# is the symbolic pullback's loss value.
+@testset "train! trains a Hamiltonian network with SymbolicPullback ($T)" for T in REAL_ELTYPES
+    Random.seed!(9)
+    arch = StandardHamiltonianArchitecture(2, 4, 1, GeometricMachineLearning.SigmoidActivation())
+    dl = DataLoader(randn(T, 2, 8), randn(T, 2, 8); suppress_info = true)
+    nn = NeuralNetwork(arch, CPU(), T)
+    p₀ = deepcopy(params(nn))
+    loss = HNNLoss(arch)
+    pullback = SymbolicPullback(arch)
+    batch = Batch(8)
+    Random.seed!(2)
+    data = GeometricMachineLearning.convert_input_and_batch_indices_to_array(
+        dl, batch, only(batch(dl)))
+    value, back = pullback(p₀, nn.model, data)
+    g = back(one(value))
+    η = T(0.1)
+    opt = TrainingOptimizer(params(nn); algorithm = GradientMethod(), linesearch = η)
+    Random.seed!(2)
+    history = train!(nn, opt, dl, batch, 1; loss, pullback, show_progress = false)
+    @test history == [value]
+    @test eltype(history) === T
+    for (layer, layer₀, gradient) in zip(values(params(nn)), values(p₀), values(g))
+        for k in keys(layer)
+            @test layer[k] ≈ layer₀[k] .- η .* gradient[k]
+            @test layer[k] != layer₀[k]
+        end
+    end
+    zygote_value, zygote_back = ZygotePullback(loss)(p₀, nn.model, data)
+    zygote_g = GeometricMachineLearning._get_contents(zygote_back(one(zygote_value)))
+    @test zygote_value ≈ value
+    @test_broken zygote_g.L1.W ≈ g.L1.W  # issue #343
+end
+
+# Without `loss`, `train!` trains on the loss that matches the architecture and the batch, and on
+# any other network it says that it needs one.
+@testset "train!'s default losses" begin
+    transformer_batch = Batch(4, 3)
+    @test GeometricMachineLearning._default_loss(GSympNet(4; n_layers = 2), Batch(4)) isa
+          FeedForwardLoss
+    @test GeometricMachineLearning._default_loss(
+        StandardTransformerIntegrator(4; n_heads = 2, L = 1, n_blocks = 1),
+        transformer_batch) == TransformerLoss(transformer_batch)
+    autoencoder = SymplecticAutoencoder(4, 2; n_encoder_blocks = 2, n_decoder_blocks = 2)
+    @test GeometricMachineLearning._default_loss(autoencoder, Batch(4)) isa AutoEncoderLoss
+
+    nn = NeuralNetwork(Chain(Dense(2, 2, identity)))
+    dl = DataLoader(rand(2, 10), rand(2, 10); suppress_info = true)
+    raised = try
+        train!(nn, TrainingOptimizer(params(nn)), dl, Batch(5), 1; show_progress = false)
+    catch e
+        e
+    end
+    @test raised isa ArgumentError
+    @test occursin("`loss`", raised.msg)
+end
+
+@testset "train! refuses n_epochs < 1 before any step" begin
+    Random.seed!(5)
+    nn = NeuralNetwork(Chain(Dense(2, 2, identity)))
+    dl = DataLoader(rand(2, 10), rand(2, 10); suppress_info = true)
+    opt = TrainingOptimizer(nn.params)
+    p₀ = deepcopy(params(nn))
+    for n in (0, -1)
+        @test_throws ArgumentError train!(
+            nn, opt, dl, Batch(5), n; loss = FeedForwardLoss(),
+            show_progress = false)
+    end
+    @test opt.state.iterations == 0
+    @test params(nn) == p₀
+end
diff --git a/test/data_loader/training_history_eltype.jl b/test/data_loader/training_history_eltype.jl
index 87e5247e5..0ec62571e 100644
--- a/test/data_loader/training_history_eltype.jl
+++ b/test/data_loader/training_history_eltype.jl
@@ -1,7 +1,7 @@
-# A `Float32` network's training history is `Float32`: `(o::Optimizer)(...)` builds `loss_array`
-# typed to `eltype(dl)`. This asserts that, and that `optimize_for_one_epoch!`'s own accumulator
-# agrees with it: the loss the optimizer returns is in the data's element type, not promoted to
-# `Float64` along the way. See `CHANGELOG.md` for the fix this guards.
+# A `Float32` network's training history on `Float32` data is `Float32`: `train!` gives the history
+# the type of the loss value, and the epoch's accumulator keeps that type, so nothing is promoted to
+# `Float64` along the way. `test/data_loader/train.jl` checks a network and data of two element
+# types.
 
 using GeometricMachineLearning
 using Test
@@ -12,9 +12,10 @@ Random.seed!(11)
 function test_training_history_eltype(T::DataType)
     nn = NeuralNetwork(GSympNet(4), CPU(), T)
     dl = DataLoader(rand(T, 4, 64); autoencoder = false, suppress_info = true)
-    o = Optimizer(GradientOptimizer(), nn)
+    o = TrainingOptimizer(nn.params; algorithm = GradientMethod())
 
-    history = o(nn, dl, Batch(8), 2, FeedForwardLoss(); show_progress = false)
+    history = train!(
+        nn, o, dl, Batch(8), 2; loss = FeedForwardLoss(), show_progress = false)
 
     @test eltype(history) == T
     @test history isa Vector{T}
diff --git a/test/helpers/eltypes.jl b/test/helpers/eltypes.jl
new file mode 100644
index 000000000..cde8d5c2d
--- /dev/null
+++ b/test/helpers/eltypes.jl
@@ -0,0 +1,3 @@
+# The real element types every numeric test runs in. A test loops over this tuple rather than over a
+# literal one, so that one line sets the precisions of the whole suite.
+const REAL_ELTYPES = (Float32, Float64)
diff --git a/test/helpers/network_parameters_gradient_structure.jl b/test/helpers/network_parameters_gradient_structure.jl
index 114af9229..ffa45f0b8 100644
--- a/test/helpers/network_parameters_gradient_structure.jl
+++ b/test/helpers/network_parameters_gradient_structure.jl
@@ -1,33 +1,21 @@
 using Test
 using GeometricMachineLearning
 using GeometricMachineLearning: params
-using LinearAlgebra: norm
+using LinearAlgebra: Diagonal, diag, norm
 using Zygote: gradient
 
 """
-Assert which shape the `SymmetricMatrix` leaf of a gradient has, and that the shape is cosmetic.
+Assert the shape and the value of the `SymmetricMatrix` leaf of a gradient, taken once with respect
+to a `NetworkParameters` wrapper and once with respect to the bare `NamedTuple` it wraps.
 
-A gradient taken with respect to a `NetworkParameters` wrapper keeps its `SymmetricMatrix` leaf
-only when the loss expression performs a single `getproperty` on that wrapper. A second access
-degrades the leaf to a plain `Matrix`. What decides the split is the number of `getproperty` calls
-on the wrapper itself -- not how often the leaf is used, and not how often the plain `NamedTuple`s
-below the wrapper are accessed. `p -> (A = p.L1.A; sum(A) + sum(A))` uses the leaf twice through one
-access and keeps the structure; `p -> sum(p.L1.A) + sum(p.L1.A)` loses it. A `Zygote.gradient` taken
-with respect to the bare wrapped `NamedTuple` (`params(ps)`) keeps the structure at any access
-count, because `gradient` projects its result and `ChainRulesCore.ProjectTo` restores the leaf for a
-`NamedTuple` but is `identity` for a `NetworkParameters`. The underlying `Zygote.pullback` loses the
-structure on the second access either way. `CHANGELOG.md` records the experiment.
-
-A wrapper unwrapped with `values` rather than `getproperty` -- which is how `Chain` hands a layer
-its parameters -- counts differently: there the accesses on the plain `NamedTuple` below decide.
-See the layer testset in
-`test/layers/symplectic_attention_network_parameters_gradient.jl`.
-
-`preserves_structure` states which side of that split `f` sits on. Both gradients are asserted to
-agree numerically either way, so losing the structure is a change of type and not of value.
+Both gradients keep the `SymmetricMatrix` leaf, at any number of accesses to the wrapper. They hold
+different quantities. With respect to the bare `NamedTuple`, `Zygote.gradient` projects the dense
+gradient ``G`` onto the symmetric matrices, ``(G + G^T)/2``. With respect to the wrapper, the leaf
+holds the gradient with respect to the leaf's storage, the lower triangle: its diagonal is that of
+``G``, and each off-diagonal entry is ``G_{ij} + G_{ji}``, twice that of the projection (issue #319,
+`KNOWN_ISSUES.md` K6).
 """
-function network_parameters_gradient_structure_test(
-        f, ps::NetworkParameters, preserves_structure::Bool)
+function network_parameters_gradient_structure_test(f, ps::NetworkParameters)
     g_wrapped = gradient(_ps -> norm(f(_ps)), ps)[1]
     g_bare = gradient(_ps -> norm(f(_ps)), params(ps))[1]
 
@@ -35,11 +23,8 @@ function network_parameters_gradient_structure_test(
     a_bare = g_bare.L1.A
 
     @test typeof(a_bare) <: SymmetricMatrix
-    if preserves_structure
-        @test typeof(a_wrapped) <: SymmetricMatrix
-    else
-        @test_broken typeof(a_wrapped) <: SymmetricMatrix  # issue #319
-    end
+    @test typeof(a_wrapped) <: SymmetricMatrix
 
-    @test isapprox(Matrix(a_wrapped), Matrix(a_bare))
+    B = Matrix(a_bare)
+    @test Matrix(a_wrapped) ≈ 2B - Diagonal(diag(B))
 end
diff --git a/test/integration/changebackend_tests.jl b/test/integration/changebackend_tests.jl
index e816364bf..19c23bf3b 100644
--- a/test/integration/changebackend_tests.jl
+++ b/test/integration/changebackend_tests.jl
@@ -35,8 +35,8 @@ leaves = (
     grassmann = rand(GrassmannManifold{Float64}, N, n),
     symmetric = SymmetricMatrix(rand(10), 4),
     skew = SkewSymMatrix(rand(6), 4),
-    lower = LowerTriangular(rand(6), 4),
-    upper = UpperTriangular(rand(6), 4),
+    lower = StrictlyLowerTriangular(rand(6), 4),
+    upper = StrictlyUpperTriangular(rand(6), 4),
     stiefhor = StiefelLieAlgHorMatrix(SkewSymMatrix(rand(n, n)), rand(N - n, n), N, n),
     grasshor = GrassmannLieAlgHorMatrix(rand(N - n, n), N, n)
 )
diff --git a/test/integration/hdf5_support.jl b/test/integration/hdf5_support.jl
index 6e04ea5f8..a206c062d 100644
--- a/test/integration/hdf5_support.jl
+++ b/test/integration/hdf5_support.jl
@@ -14,8 +14,8 @@ _ps_eq(a::AbstractArray, b::AbstractArray) = a ≈ b
 _ps_eq(a::StiefelManifold, b::StiefelManifold) = a.A ≈ b.A
 _ps_eq(a::SymmetricMatrix, b::SymmetricMatrix) = a.S ≈ b.S && a.n == b.n
 _ps_eq(a::SkewSymMatrix, b::SkewSymMatrix) = a.S ≈ b.S && a.n == b.n
-_ps_eq(a::LowerTriangular, b::LowerTriangular) = a.S ≈ b.S && a.n == b.n
-_ps_eq(a::UpperTriangular, b::UpperTriangular) = a.S ≈ b.S && a.n == b.n
+_ps_eq(a::StrictlyLowerTriangular, b::StrictlyLowerTriangular) = a.S ≈ b.S && a.n == b.n
+_ps_eq(a::StrictlyUpperTriangular, b::StrictlyUpperTriangular) = a.S ≈ b.S && a.n == b.n
 function _ps_eq(a::NamedTuple, b::NamedTuple)
     Set(keys(a)) == Set(keys(b)) || return false
     all(_ps_eq(a[k], b[k]) for k in keys(a))
@@ -48,7 +48,8 @@ function _write_legacy(h5, Y::StiefelManifold, path::AbstractString)
 end
 
 for (T, name) in ((:SymmetricMatrix, "SymmetricMatrix"), (:SkewSymMatrix, "SkewSymMatrix"),
-    (:LowerTriangular, "LowerTriangular"), (:UpperTriangular, "UpperTriangular"))
+    (:StrictlyLowerTriangular, "StrictlyLowerTriangular"), (
+    :StrictlyUpperTriangular, "StrictlyUpperTriangular"))
     @eval function _write_legacy(h5, A::$T, path::AbstractString)
         g = HDF5.create_group(h5, path)
         HDF5.attributes(g)["gml_type"] = $name
@@ -131,8 +132,8 @@ end
     end
 end
 
-# VolumePreservingFeedForward uses LowerTriangular / UpperTriangular parameters.
-@testset "save/load roundtrip: VolumePreservingFeedForward (LowerTriangular/UpperTriangular)" begin
+# VolumePreservingFeedForward uses StrictlyLowerTriangular / StrictlyUpperTriangular parameters.
+@testset "save/load roundtrip: VolumePreservingFeedForward (StrictlyLowerTriangular/StrictlyUpperTriangular)" begin
     arch = VolumePreservingFeedForward(4, 4, 1, tanh)
     nn = NeuralNetwork(arch)
     x = rand(4)
diff --git a/test/integration/losses_and_optimization.jl b/test/integration/losses_and_optimization.jl
index cf2ef1425..57ae92d5c 100644
--- a/test/integration/losses_and_optimization.jl
+++ b/test/integration/losses_and_optimization.jl
@@ -17,9 +17,9 @@ function train_network(; n_epochs = 10)
     nn = setup_network(dl)
     loss = FeedForwardLoss()
 
-    o = Optimizer(Adam(), nn)
+    o = TrainingOptimizer(nn.params; algorithm = Adam())
     batch = Batch(5, 1)
-    loss_array = o(nn, dl, batch, n_epochs, loss; show_progress = false)
+    loss_array = train!(nn, o, dl, batch, n_epochs; loss, show_progress = false)
     T = eltype(dl)
     @test loss_array[end] / loss_array[1] < T(0.1)
 end
diff --git a/test/integration/reduced_loss_optimization.jl b/test/integration/reduced_loss_optimization.jl
index 0894fcb98..d994518fe 100644
--- a/test/integration/reduced_loss_optimization.jl
+++ b/test/integration/reduced_loss_optimization.jl
@@ -5,7 +5,7 @@ import Random
 
 Random.seed!(123)
 
-# `ReducedLoss` trained through the `Optimizer` functor, which is the path the symplectic
+# `ReducedLoss` trained through `train!`, which is the path the symplectic
 # autoencoder tutorial documents and the path `scripts/reproduction/symplectic_autoencoders/`
 # takes. It was unreachable: the loss annotated its parameter argument `::NetworkParameters`, and
 # `Zygote.pullback` evaluates the forward pass with the parameters unwrapped -- the closure body
@@ -16,10 +16,10 @@ Random.seed!(123)
 #
 # Nothing had caught it because nothing ran it. The docstring example calls the loss directly,
 # which dispatches, and the tutorial's training is in a plain ```julia fence that Documenter
-# renders and never executes. Restoring the annotation makes the `Optimizer` call below raise
+# renders and never executes. Restoring the annotation makes the `train!` call below raise
 # that `error`, so this testset errors before any of the three assertions runs; the assertions
 # themselves check what the loss is for.
-@testset "ReducedLoss trains through the Optimizer functor" begin
+@testset "ReducedLoss trains through train!" begin
     reduced_dim = 2
 
     q = rand(Float64, 6, 30, 2)
@@ -37,12 +37,12 @@ Random.seed!(123)
         StandardTransformerIntegrator(reduced_dim; transformer_dim = 10, n_blocks = 1,
             n_heads = 2, L = 1, upscaling_activation = tanh), CPU(), Float64)
 
-    o = Optimizer(Adam(), integrator)
+    o = TrainingOptimizer(integrator.params; algorithm = Adam())
     batch = Batch(8, 4)
 
     # The parameters reach the loss through `Zygote.pullback` here, which is what an annotated
     # `params` argument breaks; calling the loss directly would pass either way.
-    loss_array = o(integrator, dl_reduced, batch, 10, loss; show_progress = false)
+    loss_array = train!(integrator, o, dl_reduced, batch, 10; loss, show_progress = false)
 
     @test length(loss_array) == 10
     @test all(isfinite, loss_array)
diff --git a/test/integration/train_manual_architectures.jl b/test/integration/train_manual_architectures.jl
new file mode 100644
index 000000000..63649db55
--- /dev/null
+++ b/test/integration/train_manual_architectures.jl
@@ -0,0 +1,69 @@
+using GeometricMachineLearning
+using GeometricMachineLearning: params
+using Test
+import Random
+
+include("../helpers/eltypes.jl")
+
+# Each architecture that the manual trains, with the data shape and the batch its tutorial uses. The
+# list holds a `SymmetricMatrix` leaf (`LASympNet`, `LinearSymplecticTransformer`), the strictly
+# triangular leaves (`VolumePreservingFeedForward`, `VolumePreservingTransformer`), a Stiefel leaf
+# (`SymplecticAutoencoder`), and the three `Chain`s of the volume-preserving-attention tutorial.
+# The last place holds the loss; `nothing` trains on the default loss.
+function manual_architectures(T)
+    feedforward = DataLoader(rand(T, 4, 12, 3); suppress_info = true)
+    transformer_batch = Batch(4, 3)
+    hnn = StandardHamiltonianArchitecture(2, 4)
+    attention_data = DataLoader(rand(T, 1, 12, 2); suppress_info = true)
+    attention_batch = Batch(4, 3, 1)
+    attention_chain(skew_sym) = Chain(Dense(1, 2, tanh),
+        VolumePreservingAttention(2, 3; skew_sym), Dense(2, 1, identity; use_bias = true))
+    (("GSympNet", GSympNet(4; n_layers = 2), feedforward, Batch(4), nothing),
+        ("LASympNet", LASympNet(4; depth = 1, nhidden = 1), feedforward, Batch(4), nothing),
+        ("ResNet", ResNet(4, 2, tanh), feedforward, Batch(4), nothing),
+        ("VolumePreservingFeedForward",
+            VolumePreservingFeedForward(4, 2), feedforward, Batch(4),
+            nothing),
+        ("StandardTransformerIntegrator",
+            StandardTransformerIntegrator(4; n_heads = 2, L = 1, n_blocks = 1),
+            feedforward, transformer_batch, nothing),
+        ("LinearSymplecticTransformer",
+            LinearSymplecticTransformer(4, 3; n_sympnet = 1, L = 1),
+            feedforward, transformer_batch, nothing),
+        ("SymplecticTransformer", SymplecticTransformer(4; n_sympnet = 1, L = 1),
+            feedforward, transformer_batch, nothing),
+        ("VolumePreservingTransformer", VolumePreservingTransformer(4, 3; n_blocks = 1),
+            feedforward, transformer_batch, nothing),
+        ("SymplecticAutoencoder",
+            SymplecticAutoencoder(4, 2; n_encoder_blocks = 2, n_decoder_blocks = 2),
+            DataLoader(rand(T, 4, 12); autoencoder = true, suppress_info = true), Batch(4),
+            nothing),
+        ("StandardHamiltonianArchitecture", hnn,
+            DataLoader(rand(T, 2, 12), rand(T, 2, 12); suppress_info = true), Batch(4),
+            HNNLoss(hnn)),
+        ("VolumePreservingAttention, skew_sym = true", attention_chain(true),
+            attention_data, attention_batch, TransformerLoss(attention_batch)),
+        ("VolumePreservingAttention, skew_sym = false", attention_chain(false),
+            attention_data, attention_batch, TransformerLoss(attention_batch)),
+        ("Dense comparison of the attention tutorial",
+            Chain(Dense(1, 2, tanh), Dense(2, 1, identity; use_bias = true)),
+            attention_data, Batch(4), FeedForwardLoss()))
+end
+
+@testset "train! trains every architecture of the manual ($T)" for T in REAL_ELTYPES
+    Random.seed!(6)
+    for (name, arch, dl, batch, loss) in manual_architectures(T)
+        @testset "$name" begin
+            nn = NeuralNetwork(arch, T)
+            p₀ = deepcopy(params(nn))
+            opt = TrainingOptimizer(nn.params; algorithm = Adam())
+            history = loss === nothing ?
+                      train!(nn, opt, dl, batch, 2; show_progress = false) :
+                      train!(nn, opt, dl, batch, 2; loss, show_progress = false)
+            @test length(history) == 2
+            @test eltype(history) === T
+            @test all(isfinite, history)
+            @test params(nn) != p₀
+        end
+    end
+end
diff --git a/test/layers/double_multiplication_network_parameters_gradient.jl b/test/layers/double_multiplication_network_parameters_gradient.jl
index d2fb3557c..98c1c2cfe 100644
--- a/test/layers/double_multiplication_network_parameters_gradient.jl
+++ b/test/layers/double_multiplication_network_parameters_gradient.jl
@@ -22,12 +22,12 @@ ps = NetworkParameters((L1 = (A = S,),))
 t = rand(4, 4)
 
 @testset "_custom_mul: NetworkParameters gradient structure" begin
-    @testset "single_multiplication (one wrapper access: structure kept)" begin
+    @testset "single_multiplication (one wrapper access)" begin
         network_parameters_gradient_structure_test(
-            _ps -> single_multiplication(t, _ps), ps, true)
+            _ps -> single_multiplication(t, _ps), ps)
     end
-    @testset "double_multiplication (two wrapper accesses: structure lost)" begin
+    @testset "double_multiplication (two wrapper accesses)" begin
         network_parameters_gradient_structure_test(
-            _ps -> double_multiplication(t, _ps), ps, false)
+            _ps -> double_multiplication(t, _ps), ps)
     end
 end
diff --git a/test/layers/gradient_layer_tests.jl b/test/layers/gradient_layer_tests.jl
index f0eb7474a..df8a600ba 100644
--- a/test/layers/gradient_layer_tests.jl
+++ b/test/layers/gradient_layer_tests.jl
@@ -1,5 +1,6 @@
 using GeometricMachineLearning
 using GeometricMachineLearning: params
+using GeometricOptimizers: optimization_step!
 using Test
 using KernelAbstractions
 import Random, Zygote
@@ -24,19 +25,18 @@ end
 function test_gradient_layer_derivative_and_update(T, M, N, batch_size = 10)
     dummy_model = Chain(GradientLayerP(M, N, tanh), GradientLayerQ(M, N, tanh))
     ps = params(NeuralNetwork(dummy_model, CPU(), T))
-    o = Optimizer(Adam(T; β₁ = T(0.9), β₂ = T(0.999), δ = T(3e-7)), ps;
-        step_size = T(0.1))
+    o = TrainingOptimizer(ps; algorithm = Adam(; β₁ = 0.9, β₂ = 0.999, δ = 3e-7),
+        linesearch = T(0.1))
 
-    # test for vector 
+    # test for vector
     x = rand(T, M)
     gs = Zygote.gradient(ps -> sum(dummy_model(x, ps)), ps)[1]
-    λY = GlobalSection(ps)
-    optimization_step!(o, λY, ps, gs)
+    optimization_step!(ps, o, gs)
 
-    # test for matrix 
+    # test for matrix
     X = rand(T, M, batch_size)
     gs = Zygote.gradient(ps -> sum(dummy_model(X, ps)), ps)[1]
-    optimization_step!(o, λY, ps, gs)
+    optimization_step!(ps, o, gs)
 end
 
 types = (Float32, Float64)
diff --git a/test/layers/manifold_layers.jl b/test/layers/manifold_layers.jl
index aa51054e5..a1427f46e 100644
--- a/test/layers/manifold_layers.jl
+++ b/test/layers/manifold_layers.jl
@@ -1,4 +1,5 @@
 using Test, LinearAlgebra, GeometricMachineLearning
+using GeometricOptimizers: optimization_step!
 import Random
 
 Random.seed!(1234)
@@ -6,13 +7,12 @@ Random.seed!(1234)
 function stiefel_layer_test(T, M, N, tol = 1.0f-1)
     model = Chain(StiefelLayer(M, N), StiefelLayer(N, N))
     ps = NeuralNetwork(model, T).params
-    o = Optimizer(Adam(T; β₁ = T(5.0f-1), β₂ = T(5.0f-1), δ = T(3.0f-7)), ps;
-        step_size = T(1.0f0))
+    o = TrainingOptimizer(ps; algorithm = Adam(; β₁ = 5.0f-1, β₂ = 5.0f-1, δ = 3.0f-7),
+        linesearch = T(1.0f0))
 
-    dx = (L1 = (weight = rand(T, N, M),), L2 = (weight = rand(T, N, N),))
+    dx = NetworkParameters((L1 = (weight = rand(T, N, M),), L2 = (weight = rand(T, N, N),)))
     ps_copy = deepcopy(ps)
-    λY = GlobalSection(ps)
-    optimization_step!(o, λY, ps, dx)
+    optimization_step!(ps, o, dx)
     # check that the new weight is different from the old one
     @test norm(ps_copy[1].weight - ps[1].weight) > T(tol)
     # check that the updated elements are on the Stiefel Manifold 
@@ -23,14 +23,13 @@ end
 function grassmann_layer_test(T, M, N, tol = 1.0f-1)
     model = Chain(GrassmannLayer(M, N), StiefelLayer(N, N))
     ps = NeuralNetwork(model, T).params
-    o = Optimizer(Adam(T; β₁ = T(5.0f-1), β₂ = T(5.0f-1), δ = T(3.0f-7)), ps;
-        step_size = T(1.0f0))
+    o = TrainingOptimizer(ps; algorithm = Adam(; β₁ = 5.0f-1, β₂ = 5.0f-1, δ = 3.0f-7),
+        linesearch = T(1.0f0))
 
-    dx = (L1 = (weight = rand(T, N, M),), L2 = (weight = rand(T, N, N),))
+    dx = NetworkParameters((L1 = (weight = rand(T, N, M),), L2 = (weight = rand(T, N, N),)))
     ps_copy = deepcopy(ps)
-    λY = GlobalSection(ps)
     for i in 1:4
-        optimization_step!(o, λY, ps, dx)
+        optimization_step!(ps, o, dx)
     end
     # check that the new weight is different from the old one
     @test norm(ps_copy[1].weight - ps[1].weight) > T(tol)
diff --git a/test/layers/multi_head_attention_stiefel_optim_cache.jl b/test/layers/multi_head_attention_stiefel_optim_cache.jl
index c94822e9a..2e1879091 100644
--- a/test/layers/multi_head_attention_stiefel_optim_cache.jl
+++ b/test/layers/multi_head_attention_stiefel_optim_cache.jl
@@ -1,17 +1,13 @@
 using GeometricMachineLearning, Test
-# `StiefelManifold` and the other shared types are one object reached by two names now, so a blanket
-# `using GeometricOptimizers` alongside GML would no longer be ambiguous on them -- only `Optimizer`
-# still resolves to two different things (issue C1). The qualified form stays because it says which
-# package owns what.
+# The qualified form says which package owns what.
 import GeometricOptimizers
 using GeometricOptimizers: GradientCache, MomentumCache, AdamCache
 import Random, LinearAlgebra
-# The cache and the state hold a `NetworkParameters` per layer: a layer is wrapped at the
-# `GeometricOptimizers` boundary, and the wrap shares the leaf arrays. `mapparameters` and not `map`,
-# because it recurses on the branches and so reaches the leaves whichever shape it is handed.
+# One cache covers the whole network, and its fields hold a `NetworkParameters` of the network's
+# shape. `mapparameters` and not `map`, because it recurses on the branches and so reaches the leaves.
 #
-# Each helper below has two methods, because two shapes arrive: a wrapped layer is a
-# `NetworkParameters` and the tree above it is a plain `NamedTuple`.
+# Each helper below has a method for a `NetworkParameters` and one for a plain `NamedTuple`, so that
+# it reaches the leaves whichever of the two it is handed.
 using NeuralNetworkParameters: NetworkParameters, mapparameters
 
 Random.seed!(1234)
@@ -69,9 +65,9 @@ function test_cache_setups_for_optimizer_for_multihead_attention_layer(T::Type,
     model = Chain(MultiHeadAttention(dim, n_heads, Stiefel = true))
     ps = NeuralNetwork(model, CPU(), T).params
 
-    o₁ = Optimizer(Adam(), ps)
-    o₂ = Optimizer(MomentumMethod(), ps)
-    o₃ = Optimizer(GradientMethod(), ps)
+    o₁ = TrainingOptimizer(ps; algorithm = Adam())
+    o₂ = TrainingOptimizer(ps; algorithm = MomentumMethod())
+    o₃ = TrainingOptimizer(ps; algorithm = GradientMethod())
 
     check_adam_cache(o₁.cache)
     check_momentum_cache(o₂.cache)
diff --git a/test/layers/multi_head_attention_stiefel_retraction.jl b/test/layers/multi_head_attention_stiefel_retraction.jl
index 7b38a33ca..8988e0cb2 100644
--- a/test/layers/multi_head_attention_stiefel_retraction.jl
+++ b/test/layers/multi_head_attention_stiefel_retraction.jl
@@ -5,12 +5,11 @@ using GeometricMachineLearning: cayley
 # so they are named qualified rather than through a re-export.
 using GeometricOptimizers: MomentumCache
 import Random, Test, LinearAlgebra, KernelAbstractions
-# The cache and the state hold a `NetworkParameters` per layer: a layer is wrapped at the
-# `GeometricOptimizers` boundary, and the wrap shares the leaf arrays. `mapparameters` and not `map`,
-# because it recurses on the branches and so reaches the leaves whichever shape it is handed.
+# One cache covers the whole network, and its fields hold a `NetworkParameters` of the network's
+# shape. `mapparameters` and not `map`, because it recurses on the branches and so reaches the leaves.
 #
-# Each helper below has two methods, because two shapes arrive: a wrapped layer is a
-# `NetworkParameters` and the tree above it is a plain `NamedTuple`.
+# Each helper below has a method for a `NetworkParameters` and one for a plain `NamedTuple`, so that
+# it reaches the leaves whichever of the two it is handed.
 using NeuralNetworkParameters: NetworkParameters, mapparameters
 
 Random.seed!(1234)
@@ -53,7 +52,8 @@ function test_multi_head_attention_retraction(
     model = Chain(MultiHeadAttention(dim, n_heads, Stiefel = true))
 
     ps = NeuralNetwork(model, backend, T).params
-    cache = Optimizer(MomentumMethod(), ps).cache
+    cache = TrainingOptimizer(ps; algorithm = MomentumMethod()).cache
+    @test cache isa MomentumCache
 
     check_retraction_geodesic(cache)
 
diff --git a/test/layers/multi_head_attention_stiefel_setup.jl b/test/layers/multi_head_attention_stiefel_setup.jl
index 5a90ea42c..74787d7c6 100644
--- a/test/layers/multi_head_attention_stiefel_setup.jl
+++ b/test/layers/multi_head_attention_stiefel_setup.jl
@@ -2,12 +2,11 @@ using GeometricMachineLearning, Test
 # see the note in `multi_head_attention_stiefel_retraction.jl`
 using GeometricOptimizers: MomentumCache
 import Random, Test, LinearAlgebra, KernelAbstractions
-# The cache and the state hold a `NetworkParameters` per layer: a layer is wrapped at the
-# `GeometricOptimizers` boundary, and the wrap shares the leaf arrays. `mapparameters` and not `map`,
-# because it recurses on the branches and so reaches the leaves whichever shape it is handed.
+# One cache covers the whole network, and its fields hold a `NetworkParameters` of the network's
+# shape. `mapparameters` and not `map`, because it recurses on the branches and so reaches the leaves.
 #
-# Each helper below has two methods, because two shapes arrive: a wrapped layer is a
-# `NetworkParameters` and the tree above it is a plain `NamedTuple`.
+# Each helper below has a method for a `NetworkParameters` and one for a plain `NamedTuple`, because
+# the parameters of a network arrive as the one and the cache's fields as the other.
 using NeuralNetworkParameters: NetworkParameters, mapparameters, foldparameters
 
 Random.seed!(1234)
@@ -50,7 +49,8 @@ function check_multi_head_attention_stiefel_setup(T::Type, N::Int, n::Int)
     @test nleaves > 0
     @test check_setup_calls[] == nleaves
 
-    gx = Optimizer(MomentumMethod(), ps).cache
+    gx = TrainingOptimizer(ps; algorithm = MomentumMethod()).cache
+    @test gx isa MomentumCache
     check_grad_setup(gx)
 end
 
diff --git a/test/layers/symplectic_attention_network_parameters_gradient.jl b/test/layers/symplectic_attention_network_parameters_gradient.jl
index 405794832..450e9b100 100644
--- a/test/layers/symplectic_attention_network_parameters_gradient.jl
+++ b/test/layers/symplectic_attention_network_parameters_gradient.jl
@@ -28,28 +28,24 @@ ps = NetworkParameters((L1 = (A = S,),))
 t = (q = rand(2, 2), p = rand(2, 2))
 
 @testset "Symplectic attention: NetworkParameters gradient structure" begin
-    @testset "symplectic_attention (two wrapper accesses: structure lost)" begin
+    @testset "symplectic_attention (two wrapper accesses)" begin
         network_parameters_gradient_structure_test(
-            _ps -> symplectic_attention(t, _ps), ps, false)
+            _ps -> symplectic_attention(t, _ps), ps)
     end
-    @testset "symplectic_attention_simplified (one wrapper access: structure kept)" begin
+    @testset "symplectic_attention_simplified (one wrapper access)" begin
         network_parameters_gradient_structure_test(
-            _ps -> symplectic_attention_simplified(t, _ps), ps, true)
+            _ps -> symplectic_attention_simplified(t, _ps), ps)
     end
-    @testset "symplectic_linear_map (one wrapper access: structure kept)" begin
+    @testset "symplectic_linear_map (one wrapper access)" begin
         network_parameters_gradient_structure_test(
-            _ps -> symplectic_linear_map(t, _ps), ps, true)
+            _ps -> symplectic_linear_map(t, _ps), ps)
     end
 end
 
-# The layer keeps the structure although its body reads `A` twice. `Chain` reaches the layer
-# through `values(_ps)` rather than through a `getproperty` on the wrapper, and on that route the
-# count that decides the split is the one on the plain `NamedTuple` the layer is handed. The local
-# binding at `src/layers/symplectic_attention.jl` makes that count one, and is what keeps the
-# structure here: the same body written without it, reached the same way, degrades the leaf to a
-# plain `Matrix`.
+# `Chain` reaches the layer through `values(_ps)` rather than through a `getproperty` on the
+# wrapper, and the layer's body reads `A` twice.
 @testset "SymplecticAttentionQ layer: NetworkParameters gradient structure" begin
     l = SymplecticAttentionQ(4; symmetric = true)
     nn = NeuralNetwork(Chain(l))
-    network_parameters_gradient_structure_test(_ps -> nn(t, _ps), params(nn), true)
+    network_parameters_gradient_structure_test(_ps -> nn(t, _ps), params(nn))
 end
diff --git a/test/layers/transformer_optimizer.jl b/test/layers/transformer_optimizer.jl
index 1d7ffcfd7..1ff0a612b 100644
--- a/test/layers/transformer_optimizer.jl
+++ b/test/layers/transformer_optimizer.jl
@@ -1,5 +1,6 @@
 using Test, KernelAbstractions, GeometricMachineLearning, Zygote, LinearAlgebra
 using GeometricMachineLearning: ResNetLayer
+using GeometricOptimizers: optimization_step!
 import Random
 
 Random.seed!(1234)
@@ -18,20 +19,17 @@ function transformer_gradient_test(T, dim, n_heads, L, seq_length = 8, batch_siz
     loss(ps, input) = norm(model(input, ps))
     dx = Zygote.gradient(ps -> loss(ps, input), ps)[1]
 
-    o₁ = Optimizer(GradientMethod(), ps)
-    o₂ = Optimizer(MomentumMethod(), ps)
-    o₃ = Optimizer(Adam(), ps)
-
     ps₁ = deepcopy(ps)
     ps₂ = deepcopy(ps)
     ps₃ = deepcopy(ps)
 
-    λY₁ = GlobalSection(ps₁)
-    λY₂ = GlobalSection(ps₂)
-    λY₃ = GlobalSection(ps₃)
-    optimization_step!(o₁, λY₁, ps₁, dx)
-    optimization_step!(o₂, λY₂, ps₂, dx)
-    optimization_step!(o₃, λY₃, ps₃, dx)
+    o₁ = TrainingOptimizer(ps₁; algorithm = GradientMethod())
+    o₂ = TrainingOptimizer(ps₂; algorithm = MomentumMethod())
+    o₃ = TrainingOptimizer(ps₃; algorithm = Adam())
+
+    optimization_step!(ps₁, o₁, dx)
+    optimization_step!(ps₂, o₂, dx)
+    optimization_step!(ps₃, o₃, dx)
     @test typeof(ps₁) == typeof(ps₂) == typeof(ps₃) == typeof(ps)
     @test ps₁[1].PQ.head_1 ≈ ps₂[1].PQ.head_1
     @test ps₁[1].PQ.head_1 ≉ ps₃[1].PQ.head_1
diff --git a/test/layers/volume_preserving_feedforward.jl b/test/layers/volume_preserving_feedforward.jl
index 673951afc..85f2da44c 100644
--- a/test/layers/volume_preserving_feedforward.jl
+++ b/test/layers/volume_preserving_feedforward.jl
@@ -35,11 +35,11 @@ test_volume_preserving_feedforward(; T = Float32)
 test_volume_preserving_feedforward(; T = Float64)
 
 # A vector in, a vector out. Neither layer shapes the result: the `:no_bias` method is
-# `x + d.activation.(ps.weight * x)`, and `LowerTriangular * ::AbstractVector` returns a vector,
+# `x + d.activation.(ps.weight * x)`, and `StrictlyLowerTriangular * ::AbstractVector` returns a vector,
 # which is what `LinearAlgebra` returns for every other matrix type.
 l = VolumePreservingLowerLayer(2, identity; use_bias = false)
-A = LowerTriangular([1], 2)
+A = StrictlyLowerTriangular([1], 2)
 @test l(ones(eltype(A), 2), (weight = A,)) == [1, 2]
 l = VolumePreservingUpperLayer(2, identity; use_bias = false)
-A = UpperTriangular([1], 2)
+A = StrictlyUpperTriangular([1], 2)
 @test l(ones(eltype(A), 2), (weight = A,)) == [2, 1]
diff --git a/test/loss/training_method_losses.jl b/test/loss/training_method_losses.jl
index 33b89e292..933f49d7f 100644
--- a/test/loss/training_method_losses.jl
+++ b/test/loss/training_method_losses.jl
@@ -170,8 +170,9 @@ function test_symplectic_euler_training(; n = 1, Δt = 1.0e-2, n_epochs = 20)
 
     nn = NeuralNetwork(arch)
     dl = DataLoader(input, output; suppress_info = true)
-    loss_array = Optimizer(Adam(), nn)(
-        nn, dl, Batch(10), n_epochs, loss; show_progress = false)
+    loss_array = train!(
+        nn, TrainingOptimizer(nn.params; algorithm = Adam()), dl, Batch(10), n_epochs;
+        loss, show_progress = false)
     @test loss_array[end] < loss_array[1]
 end
 
@@ -339,8 +340,9 @@ function test_variational_training(; Δt = 0.05, samples = 60, n_epochs = 20)
     qₙ₊₂ = reshape(q[3:(samples + 2)], 1, samples)
 
     dl = DataLoader(vcat(qₙ, qₙ₊₁), qₙ₊₂; suppress_info = true)
-    loss_array = Optimizer(Adam(), nn)(
-        nn, dl, Batch(10), n_epochs, loss; show_progress = false)
+    loss_array = train!(
+        nn, TrainingOptimizer(nn.params; algorithm = Adam()), dl, Batch(10), n_epochs;
+        loss, show_progress = false)
     @test loss_array[end] < loss_array[1]
 end
 
diff --git a/test/map_to_cpu.jl b/test/map_to_cpu.jl
index c7d085421..c83c0bfa9 100644
--- a/test/map_to_cpu.jl
+++ b/test/map_to_cpu.jl
@@ -12,7 +12,7 @@ using Random
 using Test
 
 import GeometricOptimizers: StiefelManifold, SymmetricMatrix, SkewSymMatrix,
-                            LowerTriangular, UpperTriangular
+                            StrictlyLowerTriangular, StrictlyUpperTriangular
 
 Random.seed!(1234)
 
@@ -22,7 +22,8 @@ const N, n = 6, 3
     ps = NetworkParameters((
         L1 = (Y = rand(StiefelManifold{Float64}, N, n), b = randn(N)),
         L2 = (S = SymmetricMatrix(randn(n, n)), A = SkewSymMatrix(randn(n, n))),
-        L3 = (L = LowerTriangular(randn(n, n)), U = UpperTriangular(randn(n, n)))
+        L3 = (L = StrictlyLowerTriangular(randn(n, n)),
+            U = StrictlyUpperTriangular(randn(n, n)))
     ))
 
     back = map_to_cpu(ps)
@@ -35,8 +36,8 @@ const N, n = 6, 3
     @test back.L1.Y isa StiefelManifold
     @test back.L2.S isa SymmetricMatrix
     @test back.L2.A isa SkewSymMatrix
-    @test back.L3.L isa LowerTriangular
-    @test back.L3.U isa UpperTriangular
+    @test back.L3.L isa StrictlyLowerTriangular
+    @test back.L3.U isa StrictlyUpperTriangular
 
     # the `n` a structured leaf carries is not in its storage; `rebuild` takes it from the prototype
     @test back.L2.S.n == ps.L2.S.n
diff --git a/test/optimizers/adam_with_learning_rate_decay.jl b/test/optimizers/adam_with_learning_rate_decay.jl
index 12c590f94..eac788f52 100644
--- a/test/optimizers/adam_with_learning_rate_decay.jl
+++ b/test/optimizers/adam_with_learning_rate_decay.jl
@@ -1,10 +1,12 @@
-
 using GeometricMachineLearning
 using GeometricMachineLearning: ResNetLayer, params
+using GeometricOptimizers: optimization_step!
 using LinearAlgebra: I
 using Test
 import Random
 
+include("../helpers/eltypes.jl")
+
 Random.seed!(123)
 
 const sin_vector = sin.(0:0.1:2π)
@@ -23,18 +25,17 @@ function train_network(; n_epochs = 2048)
     nn₁ = setup_network(dl)
     nn₂ = setup_network(dl)
 
-    o₁ = Optimizer(AdamOptimizer(), nn₁)
-    # `AdamOptimizerWithDecay` is GeometricOptimizers' now, and returns the `(algorithm, linesearch)`
-    # pairing its own `Optimizer` takes, so it splats rather than being passed positionally. The
-    # element type is positional and defaults to `Float64` here, where GML's own version took it
-    # from `η₁` and so defaulted to `Float32`.
-    o₂ = Optimizer(nn₂; AdamOptimizerWithDecay(n_epochs, eltype(dl))...)
+    o₁ = TrainingOptimizer(nn₁.params; algorithm = Adam())
+    # `AdamOptimizerWithDecay` is GeometricOptimizers', and returns the `(algorithm, linesearch)`
+    # pairing the keywords take, so it splats rather than being passed positionally. Its argument
+    # counts optimization steps, and the optimizer converts it to the element type of the network.
+    o₂ = TrainingOptimizer(nn₂.params; AdamOptimizerWithDecay(n_epochs)...)
 
     batch = Batch(5, 1)
     loss = FeedForwardLoss()
 
-    loss_array₁ = o₁(nn₁, dl, batch, n_epochs, loss; show_progress = false)
-    loss_array₂ = o₂(nn₂, dl, batch, n_epochs, loss; show_progress = false)
+    loss_array₁ = train!(nn₁, o₁, dl, batch, n_epochs; loss, show_progress = false)
+    loss_array₂ = train!(nn₂, o₂, dl, batch, n_epochs; loss, show_progress = false)
 
     T = eltype(dl)
     @test loss_array₂[end] < loss_array₁[end] < T(1.6e-1)
@@ -45,10 +46,6 @@ end
 
 The pairing is `Adam` plus a `DecayingStatic` step size, so the method the optimizer dispatches on is
 an ordinary `Adam` and the manifold weights take GeometricOptimizers' Adam cache like any other Adam.
-GML's own version of this method was a distinct `OptimizerMethod` that had to be routed onto that
-cache explicitly, and without the routing every weight -- the `StiefelManifold` ones included -- fell
-through to the Euclidean state, whose zero element is a `StiefelLieAlgHorMatrix` and not a manifold
-point.
 """
 # `n_epochs = 128` and not 32: `AdamOptimizerWithDecay(n_epochs)` fixes
 # γ = exp(log(η₂/η₁)/n_epochs), so a 32-epoch budget drives the learning rate to η₂ = 1e-6 almost
@@ -59,8 +56,9 @@ function train_manifold_network(; n_epochs = 128)
     arch = Chain(StiefelLayer(1, 20), Dense(20, 20, tanh), Dense(20, 1, identity))
     nn = NeuralNetwork(arch, CPU(), eltype(dl))
 
-    o = Optimizer(nn; AdamOptimizerWithDecay(n_epochs, eltype(dl))...)
-    loss_array = o(nn, dl, Batch(5, 1), n_epochs, FeedForwardLoss(); show_progress = false)
+    o = TrainingOptimizer(nn.params; AdamOptimizerWithDecay(n_epochs)...)
+    loss_array = train!(nn, o, dl, Batch(5, 1), n_epochs; loss = FeedForwardLoss(),
+        show_progress = false)
 
     @test all(isfinite, loss_array)
     @test loss_array[end] < loss_array[1]
@@ -72,35 +70,28 @@ end
 @doc raw"""
 The schedule is walked from ``t = 1``, not from ``t = 0``.
 
-`optimization_step!` increments `opt.iterations` *before* it reads the step size, so the first step
-of a run takes ``\alpha(1) = \gamma\eta_1`` and not ``\alpha(0) = \eta_1``. That is how the pre-0.5
-`AdamOptimizerWithDecay` counted — it incremented `o.step` before `update!` — and how
-`DecayingStatic` counts, because `GeometricOptimizers.solve!` calls `increase_iteration_number!`
-before `solver_step!`. Reading before incrementing put every step of a run one place early in the
-schedule, which `test/adam_optimizer_with_decay.jl` upstream asserts does not happen.
+The step increments the iteration number *before* it reads the step size, so the first step of a run
+takes ``\alpha(1) = \gamma\eta_1`` and not ``\alpha(0) = \eta_1``. That is how `DecayingStatic`
+counts, because `GeometricOptimizers.solve!` calls `increase_iteration_number!` before
+`solver_step!`.
 """
-function schedule_starts_at_one(; n_epochs = 100, η₁ = 1e-2, η₂ = 1e-6)
-    method = AdamOptimizerWithDecay(n_epochs, Float64; η₁ = η₁, η₂ = η₂)
-    o = Optimizer(NetworkParameters((weight = zeros(2, 2),)); method...)
-
+function schedule_starts_at_one(T; n_epochs = 100, η₁ = 1e-2, η₂ = 1e-6)
     γ = exp(log(η₂ / η₁) / n_epochs)
-    @test o.step_size isa DecayingStatic
-    @test o.iterations == 0
-
-    for t in 1:4
-        o.iterations += 1
-        @test GeometricMachineLearning._current_step_size(o, o.iterations) ≈ η₁ * γ^t
-    end
-
-    # and the same thing through the public entry point: one step of a Euclidean parameter with a
-    # gradient of `1` moves it by `α₁ / (√1 + δ) ≈ α₁`, so the distance travelled reports the α used
-    ps = NetworkParameters((weight = zeros(2, 2),))
-    opt = Optimizer(ps; method...)
-    optimization_step!(opt, GlobalSection(ps), ps, (weight = ones(2, 2),))
-    @test opt.iterations == 1
+
+    # One step of a Euclidean parameter with a gradient of `1` moves it by `α₁ / (√1 + δ) ≈ α₁`, so
+    # the distance travelled reports the α used.
+    ps = NetworkParameters((weight = zeros(T, 2, 2),))
+    opt = TrainingOptimizer(ps; AdamOptimizerWithDecay(n_epochs; η₁, η₂)...)
+    @test opt.linesearch isa DecayingStatic
+    @test opt.state.iterations == 0
+    optimization_step!(ps, opt, NetworkParameters((weight = ones(T, 2, 2),)))
+    @test opt.state.iterations == 1
+    @test eltype(ps.weight) === T
     @test abs(ps.weight[1, 1]) ≈ η₁ * γ rtol = 1e-6
 end
 
 train_network()
 train_manifold_network()
-schedule_starts_at_one()
+@testset "the decay schedule starts at t = 1 ($T)" for T in REAL_ELTYPES
+    schedule_starts_at_one(T)
+end
diff --git a/test/optimizers/gml_gradient_dispatch.jl b/test/optimizers/gml_gradient_dispatch.jl
deleted file mode 100644
index 85a9bcb6c..000000000
--- a/test/optimizers/gml_gradient_dispatch.jl
+++ /dev/null
@@ -1,129 +0,0 @@
-# What `_GMLGradient` is called on, and by which method.
-#
-# `_GMLGradient` is a `GeometricOptimizers.Gradient`, so its functor methods share a method table with
-# upstream's. Two of them overlap on a shape this package actually produces, and the overlap is an
-# *ambiguity* rather than a choice — which is a run-time error several frames into a solve, not a
-# signature a reader would notice. So it is asserted here rather than left to whichever call path
-# happens to reach it.
-
-using GeometricMachineLearning
-using GeometricOptimizers: StiefelManifold, GrassmannManifold, rgrad
-using NeuralNetworkParameters: NetworkParameters
-using Test
-import Random
-
-Random.seed!(1234)
-
-GML = GeometricMachineLearning
-
-# A bare `Manifold`. `_GMLGradient{T} <: GeometricOptimizers.Gradient{T}` and `Manifold{T} <:
-# AbstractMatrix{T}`, so `(::_GMLGradient{T})(::AbstractArray{T})` and upstream's
-# `(::Gradient{T})(::Manifold{T})` are ambiguous here: neither is more specific. Without a method of
-# its own this call is `MethodError: ... is ambiguous`.
-#
-# The value is this package's answer and not upstream's. Upstream's body is
-# `rgrad(x, reshape(grad(vec(x)), size(x)...))`, which evaluates an *inner* gradient — and a
-# `_GMLGradient` has none, since the whole point of it is that the Euclidean gradient was computed
-# already and is being carried in `dp`.
-@testset "a bare Manifold reaches this package's method, not upstream's" begin
-    for M in (StiefelManifold, GrassmannManifold)
-        Y = rand(M, 4, 2)
-        dp = randn(4, 2)
-        g = GML._GMLGradient{Float64, typeof(dp)}(dp)
-
-        @test g(Y) == rgrad(Y, dp)
-        @test which(g, Tuple{typeof(Y)}).module === GeometricMachineLearning
-    end
-end
-
-# A bare `AbstractVector`. `_GMLGradient{T} <: GeometricOptimizers.Gradient{T}` and `SimpleSolvers`
-# has `(::Gradient{T})(::AbstractVector{T})`, so without a method of its own,
-# `(::_GMLGradient{T})(::AbstractArray{T})` and that one are equally specific on a `Vector` --
-# the call is `MethodError: ... is ambiguous`, same class of defect as the `Manifold` case above.
-@testset "a bare AbstractVector reaches this package's method, not upstream's" begin
-    dp = randn(3)
-    g = GML._GMLGradient{Float64, typeof(dp)}(dp)
-
-    @test g(randn(3)) === dp
-    @test which(g, Tuple{Vector{Float64}}).module === GeometricMachineLearning
-end
-
-# The two shapes the optimizer actually hands it, so that the method above is not the only one
-# covered and a later narrowing of either shows up here.
-@testset "a wrapped layer and a plain array" begin
-    dp = randn(3, 3)
-    g = GML._GMLGradient{Float64, typeof(dp)}(dp)
-    # a plain array leaf: the gradient is already Euclidean, so it passes through
-    @test g(randn(3, 3)) === dp
-
-    ps = NetworkParameters((weight = rand(StiefelManifold, 4, 2),))
-    dps = (weight = randn(4, 2),)
-    gps = GML._GMLGradient{Float64, typeof(dps)}(dps)
-    out = gps(ps)
-    # rebuilt in the shape of its *first* argument -- a container, which is what
-    # `GeometricOptimizers._copyto!` has a method for
-    @test out isa NetworkParameters
-    @test out.weight == rgrad(ps.weight, dps.weight)
-end
-
-# `_is_layer` is the rule for how a network is split into caches, so its boundary cases are stated
-# rather than inherited. An empty set is *zero* caches' worth and descends; `all` over an empty
-# collection would otherwise make it vacuously one.
-@testset "_is_layer at the boundary" begin
-    @test GML._is_layer((W = randn(2, 2), b = randn(2)))
-    @test GML._is_layer(NetworkParameters((W = randn(2, 2), b = randn(2))))
-    # a set whose values are branches is a subtree, and gets one cache per layer
-    @test !GML._is_layer((L1 = (W = randn(2, 2),), L2 = (W = randn(2, 2),)))
-    @test !GML._is_layer(NamedTuple())
-    @test !GML._is_layer(NetworkParameters(NamedTuple()))
-    @test !GML._is_layer(randn(2, 2))
-
-    # a layer whose weights do not share an element type is still one cache, with `T` the promotion
-    @test GML._is_layer((W = randn(Float32, 2, 2), b = randn(Float64, 2)))
-end
-
-# `_as_go_solution` is the wrap itself, and the property that makes the wrap free: it shares the leaf
-# arrays, so the optimizer step writes through to the network's own weights and nothing is copied
-# back. That is what makes this a boundary change rather than a change of behaviour, and it is the one
-# claim of this release that a reader cannot check by reading a signature.
-@testset "_as_go_solution wraps a layer and shares its arrays" begin
-    W, b = randn(2, 2), randn(2)
-    layer = (W = W, b = b)
-
-    wrapped = GML._as_go_solution(layer)
-    @test wrapped isa NetworkParameters
-    @test wrapped.W === W          # shared, not copied
-    @test wrapped.b === b
-
-    # already in the right shape: passed through untouched
-    ps = NetworkParameters(layer)
-    @test GML._as_go_solution(ps) === ps
-    Y = rand(StiefelManifold, 4, 2)
-    @test GML._as_go_solution(Y) === Y
-    v = randn(3)
-    @test GML._as_go_solution(v) === v
-
-    # a subtree is not a layer, so it is left alone for the tree branch to descend
-    tree = (L1 = (W = randn(2, 2),), L2 = (W = randn(2, 2),))
-    @test GML._as_go_solution(tree) === tree
-end
-
-# `_get_contents` unwraps whatever shape the pullback handed back. Six methods, because a reverse pass
-# produces a container or the bare `NamedTuple` it wraps, and `Zygote` may hand either back bare, in a
-# one-tuple, or in a one-element vector. Each is exercised here: splitting a union into a method per
-# shape means each method needs its own case, or the split trades one tested path for six untested
-# ones.
-@testset "_get_contents unwraps every shape a pullback returns" begin
-    nt = (L1 = (W = randn(2, 2), b = randn(2)),)
-    ps = NetworkParameters(nt)
-
-    for x in (nt, ps)
-        @test GML._get_contents(x) === x
-        @test GML._get_contents((x,)) === x
-        @test GML._get_contents([x]) === x
-    end
-
-    # more than one set is a caller error and says so, rather than silently taking the first
-    @test_throws ArgumentError GML._get_contents([ps, ps])
-    @test_throws ArgumentError GML._get_contents([nt, nt])
-end
diff --git a/test/optimizers/gradient_optimizer.jl b/test/optimizers/gradient_optimizer.jl
index 376e946bd..b147c4034 100644
--- a/test/optimizers/gradient_optimizer.jl
+++ b/test/optimizers/gradient_optimizer.jl
@@ -1,4 +1,5 @@
 using GeometricMachineLearning
+using GeometricOptimizers: optimization_step!
 using Zygote
 using Test
 using LinearAlgebra: norm
@@ -15,11 +16,10 @@ function gradient_optimizer_euclidean(N; n_steps = 20, step_size = 1e-3)
     A = randn(N, N)
     ps = NetworkParameters((A = A,))
     loss1 = loss(ps)
-    o = Optimizer(GradientMethod(), ps; step_size = step_size)
+    o = TrainingOptimizer(ps; algorithm = GradientMethod(), linesearch = step_size)
     for _ in 1:n_steps
         ∇L = Zygote.gradient(loss, ps)[1]
-        λY = GlobalSection(ps)
-        optimization_step!(o, λY, ps, ∇L)
+        optimization_step!(ps, o, ∇L)
     end
     loss2 = loss(ps)
     @test loss1 > loss2
@@ -35,11 +35,10 @@ function gradient_optimizer_stiefel(N, n; n_steps = 20, step_size = 1e-3)
     Y = rand(StiefelManifold, N, n)
     ps = NetworkParameters((Y = Y,))
     loss1 = loss(ps)
-    o = Optimizer(GradientMethod(), ps; step_size = step_size)
+    o = TrainingOptimizer(ps; algorithm = GradientMethod(), linesearch = step_size)
     for _ in 1:n_steps
         ∇L = Zygote.gradient(loss, ps)[1]
-        λY = GlobalSection(ps)
-        optimization_step!(o, λY, ps, ∇L)
+        optimization_step!(ps, o, ∇L)
     end
     loss2 = loss(ps)
     @test loss1 > loss2
diff --git a/test/optimizers/momentum_optimizer.jl b/test/optimizers/momentum_optimizer.jl
index 0351140a1..4b056b030 100644
--- a/test/optimizers/momentum_optimizer.jl
+++ b/test/optimizers/momentum_optimizer.jl
@@ -1,4 +1,5 @@
 using GeometricMachineLearning
+using GeometricOptimizers: optimization_step!
 using Zygote
 using Test
 using LinearAlgebra: norm, I
@@ -16,11 +17,10 @@ growing one that never saturates, which is what this test rules out.
 """
 function momentum_matches_geometric_sum(; α = 0.9, η = 0.1, n_steps = 10)
     ps = NetworkParameters((A = zeros(2, 2),))
-    o = Optimizer(MomentumMethod(α), ps; step_size = η)
-    λY = GlobalSection(ps)
+    o = TrainingOptimizer(ps; algorithm = MomentumMethod(; α), linesearch = η)
     previous = 0.0
     for t in 1:n_steps
-        optimization_step!(o, λY, ps, (A = ones(2, 2),))
+        optimization_step!(ps, o, NetworkParameters((A = ones(2, 2),)))
         step = previous - ps.A[1, 1]
         previous = ps.A[1, 1]
         @test step ≈ η * (1 - α^t) / (1 - α)
@@ -38,10 +38,9 @@ function momentum_optimizer_stiefel(N, n; n_steps = 50, step_size = 1e-3, α = 0
     loss(ps) = norm(ps.Y * ps.Y' - B)^2
     ps = NetworkParameters((Y = rand(StiefelManifold, N, n),))
     loss1 = loss(ps)
-    o = Optimizer(MomentumMethod(α), ps; step_size = step_size)
-    λY = GlobalSection(ps)
+    o = TrainingOptimizer(ps; algorithm = MomentumMethod(; α), linesearch = step_size)
     for _ in 1:n_steps
-        optimization_step!(o, λY, ps, Zygote.gradient(loss, ps)[1])
+        optimization_step!(ps, o, Zygote.gradient(loss, ps)[1])
     end
     @test loss(ps) < loss1
     @test ps.Y' * ps.Y ≈ I
diff --git a/test/optimizers/optimization_step.jl b/test/optimizers/optimization_step.jl
index 781d625a0..825b1e265 100644
--- a/test/optimizers/optimization_step.jl
+++ b/test/optimizers/optimization_step.jl
@@ -1,57 +1,82 @@
 using GeometricMachineLearning, Test, LinearAlgebra, KernelAbstractions
-using AbstractNeuralNetworks: AbstractExplicitLayer
 import GeometricOptimizers
+import GeometricOptimizers: optimization_step!
 import GeometricMachineLearning: NeuralNetwork
 import Random
 
+include("../helpers/eltypes.jl")
+
 Random.seed!(1234)
 
 function optimization_step_test(N, n, T)
     model = Chain(StiefelLayer(N, n), Dense(N, N, tanh))
     ps = NeuralNetwork(model, KernelAbstractions.CPU(), T).params
-    # gradient 
-    dx = (L1 = (weight = rand(Float32, N, n),),
-        L2 = (W = rand(Float32, N, N), b = rand(Float32, N)))
-    m = AdamOptimizer()
-    # randomize the cache!
-    o = Optimizer(m, ps)
+    # gradient
+    dx = NetworkParameters((L1 = (weight = rand(T, N, n),),
+        L2 = (W = rand(T, N, N), b = rand(T, N))))
+    o = TrainingOptimizer(ps; algorithm = Adam())
 
     ps2 = deepcopy(ps)
-    λY = GlobalSection(ps)
-    optimization_step!(o, λY, ps, dx)
-    @test typeof(ps[1].weight) <: StiefelManifold
+    optimization_step!(ps, o, dx)
+    @test ps[1].weight isa StiefelManifold{T}
     for (layers1, layers2) in zip(values(ps), values(ps2))
         for key in keys(layers1)
+            @test eltype(layers1[key]) === T
             @test norm(layers1[key] - layers2[key]) > T(1.0f-6)
         end
     end
 end
 
-N_max = 10
-T = Float32
-for N in 4:N_max
-    for n in 1:N
-        optimization_step_test(N, n, T)
+@testset "one Adam step moves every leaf of the network ($T)" for T in REAL_ELTYPES
+    for N in 4:10
+        for n in 1:N
+            optimization_step_test(N, n, T)
+        end
     end
 end
 
-# The regression net for the branch order in `_make_optimizer_cache`/`_make_optimizer_state`: a
-# `NetworkParameters` is a tree to descend into, so it is recognised *structurally*, before the
-# capability question `_use_go_cache` asks. The shape that follows is one `GeometricOptimizers` cache
-# per *layer* -- not one for the root, which is what putting the capability question first would give
-# the moment `GeometricOptimizers` adds the container to `OptimizerSolution`, and not one per weight,
-# which is what hoisting the `NamedTuple` branch as well would give.
-@testset "one cache and one state per layer, keyed by the network's layers" begin
+# One cache and one state cover the whole network: their solution is a parameter set with the
+# network's layers as its keys, not one cache per layer.
+@testset "one cache and one state over the whole network ($T)" for T in REAL_ELTYPES
     model = Chain(StiefelLayer(6, 3), Dense(6, 6, tanh))
-    nn = NeuralNetwork(model, KernelAbstractions.CPU(), Float32)
-    o = Optimizer(AdamOptimizer(), nn)
+    nn = NeuralNetwork(model, KernelAbstractions.CPU(), T)
+    o = TrainingOptimizer(nn.params; algorithm = Adam())
+
+    @test o.cache isa GeometricOptimizers.OptimizerCache{T}
+    @test o.state isa GeometricOptimizers.OptimizerState{T}
+    for x in (GeometricOptimizers.solution(o.cache),
+        GeometricOptimizers.solution(o.state))
+        @test x isa NetworkParameters
+        @test keys(x) == keys(GeometricMachineLearning.params(nn))
+        @test x.L1.weight isa StiefelManifold{T}
+        @test eltype(x.L2.W) === eltype(x.L2.b) === T
+    end
+end
+
+# A function barrier with concrete argument types, which runs the call once before it measures.
+function allocations(f::F, a::A, b::B, c::C) where {F, A, B, C}
+    f(a, b, c)
+    @allocated f(a, b, c)
+end
+
+# The training step is the hot path of `train!`. It is inferred, and on a Euclidean layer it allocates
+# nothing. With a Stiefel leaf it is inferred and keeps the leaf on the manifold.
+@testset "the training step is inferred and allocates nothing on a Euclidean layer ($T)" for T in REAL_ELTYPES
+    for algorithm in (GradientMethod(), MomentumMethod(), Adam())
+        ps = NetworkParameters((L1 = (W = rand(T, 3, 3), b = rand(T, 3)),))
+        dp = NetworkParameters((L1 = (W = rand(T, 3, 3), b = rand(T, 3)),))
+        o = TrainingOptimizer(ps; algorithm)
+        @inferred optimization_step!(ps, o, dp)
+        @test allocations(optimization_step!, ps, o, dp) == 0
+        @test eltype(ps.L1.W) === eltype(ps.L1.b) === T
 
-    for tree in (o.cache, o.state)
-        @test tree isa NamedTuple
-        @test keys(tree) == keys(GeometricMachineLearning.params(nn))
+        ps = NetworkParameters((L1 = (W = rand(T, 3, 3), b = rand(T, 3)),
+            L2 = (Y = rand(StiefelManifold{T}, 4, 2),)))
+        dp = NetworkParameters((L1 = (W = rand(T, 3, 3), b = rand(T, 3)),
+            L2 = (Y = rand(T, 4, 2),)))
+        o = TrainingOptimizer(ps; algorithm)
+        @inferred optimization_step!(ps, o, dp)
+        @test ps.L2.Y isa StiefelManifold{T}
     end
-    @test o.cache.L1 isa GeometricOptimizers.OptimizerCache
-    @test o.cache.L2 isa GeometricOptimizers.OptimizerCache
-    @test o.state.L1 isa GeometricOptimizers.OptimizerState
-    @test o.state.L2 isa GeometricOptimizers.OptimizerState
+    @test allocations((a, b, c) -> collect(a), 1:3, 1, 1) > 0
 end
diff --git a/test/optimizers/optimizer.jl b/test/optimizers/optimizer.jl
new file mode 100644
index 000000000..d95b17ace
--- /dev/null
+++ b/test/optimizers/optimizer.jl
@@ -0,0 +1,44 @@
+using GeometricMachineLearning
+using GeometricMachineLearning: params, convert_input_and_batch_indices_to_array
+using Test
+import Random
+import Zygote
+
+include("../helpers/eltypes.jl")
+
+# One epoch of one minibatch is one gradient step at `p₀`, so the result is `p₀ - η∇L(p₀)` with the
+# `algorithm` and the `linesearch` the constructor was given. The reference gradient is taken on the
+# minibatch `train!` draws, drawn again from the same seed. The bound is 4 eps(T), for the rounding of
+# one step.
+@testset "train! steps with the TrainingOptimizer's algorithm and linesearch ($T)" for T in REAL_ELTYPES
+    Random.seed!(1)
+    x, y = randn(T, 2, 8), randn(T, 2, 8)
+    dl = DataLoader(x, y; suppress_info = true)
+    model = Chain(Dense(2, 2, identity))
+    nn = NeuralNetwork(model, T)
+    p₀ = deepcopy(params(nn))
+    η = T(0.3)
+
+    Random.seed!(2)
+    batch = Batch(8)
+    minibatches = batch(dl)
+    @test length(minibatches) == 1
+    input, output = convert_input_and_batch_indices_to_array(dl, batch, only(minibatches))
+    g = only(Zygote.gradient(ps -> FeedForwardLoss()(model, ps, input, output), p₀))
+
+    opt = TrainingOptimizer(nn.params; algorithm = GradientMethod(), linesearch = η)
+    Random.seed!(2)
+    train!(nn, opt, dl, batch, 1; loss = FeedForwardLoss(), show_progress = false)
+    for k in (:W, :b)
+        @test params(nn).L1[k] isa Array{T}
+        @test params(nn).L1[k] ≈ p₀.L1[k] .- η .* g.L1[k] rtol = 4eps(T)
+        @test params(nn).L1[k] != p₀.L1[k]
+    end
+end
+
+@testset "AdamOptimizerWithDecay splats into the keywords" begin
+    nn = NeuralNetwork(Chain(Dense(2, 2, identity)), Float32)
+    opt = TrainingOptimizer(nn.params; AdamOptimizerWithDecay(100)...)
+    @test opt.method isa Adam{Float32}
+    @test opt.linesearch isa DecayingStatic{Float32}
+end
diff --git a/test/optimizers/psd_optim.jl b/test/optimizers/psd_optim.jl
index 8afad17a7..1092b774a 100644
--- a/test/optimizers/psd_optim.jl
+++ b/test/optimizers/psd_optim.jl
@@ -1,4 +1,5 @@
 using GeometricMachineLearning, LinearAlgebra, KernelAbstractions, Test, Zygote, Random
+using GeometricOptimizers: optimization_step!
 
 Random.seed!(1234)
 
@@ -26,7 +27,7 @@ A = [0.06476993260924702 0.8369280855305259 0.6245358125914054 0.140729967064923
 """
 This tests if the optimizers can find the optimal PSD solution.
 """
-function svd_test(A, n, train_steps = 1000, tol = 1e-1; retraction = cayley)
+function svd_test(A, n, train_steps = 1000, tol = 1e-1; retraction = Cayley())
     # Seeded here and not once at the top of the file, for the reason given at the same place in
     # `svd_optim.jl`: `svd_test` is called twice, and with a single top-level seed the second call
     # starts from whatever RNG state the first one happened to leave behind, which makes the
@@ -45,9 +46,9 @@ function svd_test(A, n, train_steps = 1000, tol = 1e-1; retraction = cayley)
     model = Chain(PSDLayer(2*N, 2*n), PSDLayer(2*n, 2*N))
     ps = NeuralNetwork(model, CPU(), Float64).params
 
-    o₁ = Optimizer(GradientMethod(), ps; retraction = retraction, step_size = 0.01)
-    o₂ = Optimizer(MomentumMethod(), ps; retraction = retraction, step_size = 0.01)
-    o₃ = Optimizer(Adam(), ps; retraction = retraction, step_size = 0.01)
+    o₁ = TrainingOptimizer(ps; algorithm = GradientMethod(), retraction, linesearch = 0.01)
+    o₂ = TrainingOptimizer(ps; algorithm = MomentumMethod(), retraction, linesearch = 0.01)
+    o₃ = TrainingOptimizer(ps; algorithm = Adam(), retraction, linesearch = 0.01)
 
     U₁, Ũ₁, err₁ = train_network!(o₁, model, deepcopy(ps), A, train_steps, tol)
     U₂, _, err₂ = train_network!(o₂, model, deepcopy(ps), A, train_steps, tol)
@@ -63,19 +64,18 @@ function svd_test(A, n, train_steps = 1000, tol = 1e-1; retraction = cayley)
     @test norm((err₃ - err_best)/err_best) < tol
 end
 
-function train_network!(o::Optimizer, model::Chain, ps::NetworkParameters,
+function train_network!(o::TrainingOptimizer, model::Chain, ps::NetworkParameters,
         A::AbstractMatrix, train_steps, tol)
     error(ps) = norm(A - model(A, ps))
 
     for _ in 1:train_steps
         dx = Zygote.gradient(error, ps)[1]
-        λY = GlobalSection(ps)
-        optimization_step!(o, λY, ps, dx)
+        optimization_step!(ps, o, dx)
         #println(error(ps))
     end
     ps[1].weight, ps[2].weight, error(ps)
 end
 
-for retraction in (geodesic, cayley)
+for retraction in (Geodesic(), Cayley())
     svd_test(A, 4, retraction = retraction)
 end
diff --git a/test/optimizers/step_size_element_type.jl b/test/optimizers/step_size_element_type.jl
deleted file mode 100644
index a2a13658a..000000000
--- a/test/optimizers/step_size_element_type.jl
+++ /dev/null
@@ -1,52 +0,0 @@
-# `_leaf_optim_step!` scales `direction(cache)` by `T(step_size)`, matching the three
-# `_euclidean_update!` methods a few lines below it. The funnel (`_current_step_size`) always
-# hands a `Float64`, no matter what `T` the parameters are, so converting first is what keeps a
-# `Float32` layer scaled at `Float32` precision instead of `Float64` precision rounded back on
-# write. See `CHANGELOG.md` for the fix this guards.
-#
-# The test is invariance rather than a fixed expected value: the leaf step must give the *same*
-# result whether the funnel hands it a `Float64` step size or one already rounded to `T`. Only
-# `T = Float32` can tell the two calls apart -- `T(step_size)` is a no-op when `T` is already
-# `Float64`, so a `Float64` combination holds whatever the conversion does and checks nothing.
-# `Float64` is therefore not looped over here.
-#
-# The `seed = 2` and the `step_size = 0.1` are chosen rather than arbitrary. Whether a `Float64`-
-# and a `Float32`-scaled direction round to different bit patterns depends on the actual values,
-# so for most seeds the two calls agree however the step size is scaled, and the test then passes
-# without discriminating. With these values, and with the conversion dropped, both `Adam` and
-# `MomentumMethod` separate by `2.9802322f-8` -- one `Float32` ULP at this magnitude -- and each
-# method is checked on its own.
-
-using GeometricMachineLearning
-using GeometricOptimizers
-using Test
-import Random
-
-GML = GeometricMachineLearning
-
-function test_leaf_step_matches_prerounded_step_size(method, seed::Int, step_size::Float64)
-    T = Float32
-    Random.seed!(seed)
-    ps_a = (weight = rand(T, 6, 6),)
-    dp = (weight = rand(T, 6, 6),)
-    ps_b = deepcopy(ps_a)
-    λY_a = GML.GlobalSection(ps_a)
-    λY_b = GML.GlobalSection(ps_b)
-
-    cache_a = GML._make_optimizer_cache(method, ps_a)
-    state_a = GML._make_optimizer_state(method, ps_a)
-    cache_b = GML._make_optimizer_cache(method, ps_b)
-    state_b = GML._make_optimizer_state(method, ps_b)
-
-    # the funnel's usual `Float64` step size ...
-    GML._leaf_optim_step!(
-        cache_a, state_a, dp, ps_a, λY_a, method, GeometricOptimizers.cayley, step_size)
-    # ... versus the same value pre-rounded to `T`.
-    GML._leaf_optim_step!(
-        cache_b, state_b, dp, ps_b, λY_b, method, GeometricOptimizers.cayley, T(step_size))
-
-    @test ps_a.weight == ps_b.weight
-end
-
-test_leaf_step_matches_prerounded_step_size(GeometricOptimizers.Adam(), 2, 0.1)
-test_leaf_step_matches_prerounded_step_size(GeometricOptimizers.MomentumMethod(0.5), 2, 0.1)
diff --git a/test/optimizers/structured_array_parameters.jl b/test/optimizers/structured_array_parameters.jl
index 4ce1a08d3..c763d135c 100644
--- a/test/optimizers/structured_array_parameters.jl
+++ b/test/optimizers/structured_array_parameters.jl
@@ -8,10 +8,10 @@ Random.seed!(123)
 @doc raw"""
 Every optimizer method has to work with the structured matrix types GML uses as *ordinary* weights:
 `SymmetricMatrix` (SympNet and symplectic attention layers), `SkewSymMatrix` (volume-preserving
-attention) and `LowerTriangular`/`UpperTriangular` (volume-preserving feedforward layers).
+attention) and `StrictlyLowerTriangular`/`StrictlyUpperTriangular` (volume-preserving feedforward layers).
 
 These have their own storage and no `setindex!`, so they need a type-preserving `similar` and
-elementwise `_add!`/`_rac!`/`_square!`/`_div!`/`_rmul!` written on the free parameters; without them
+elementwise operations written on the free parameters; without them
 the optimizer cache cannot even be allocated. Those live in GeometricOptimizers, which owns the
 types — see `VectorStorageMatrix` there. GML used to carry its own copies of both the types and the
 methods, in `src/optimizers/go_bridges.jl`.
@@ -20,19 +20,19 @@ function optimizer_runs(architecture, batch, input_dim; T = Float64, n_epochs =
     nn_and_dl() = (NeuralNetwork(architecture, T),
         DataLoader(rand(T, input_dim, 20, 5); suppress_info = true))
 
-    for method in (AdamOptimizer(), MomentumOptimizer(), GradientOptimizer())
+    for method in (Adam(), MomentumMethod(), GradientMethod())
         nn, dl = nn_and_dl()
-        o = Optimizer(method, nn)
-        loss_array = o(nn, dl, batch, n_epochs; show_progress = false)
+        o = TrainingOptimizer(nn.params; algorithm = method)
+        loss_array = train!(nn, o, dl, batch, n_epochs; show_progress = false)
         @test length(loss_array) == n_epochs
         @test all(isfinite, loss_array)
     end
 
     # `AdamOptimizerWithDecay` is a `(algorithm, linesearch)` pairing rather than a method, so it
-    # goes in through the keyword constructor
+    # splats into the keywords
     nn, dl = nn_and_dl()
-    o = Optimizer(nn; AdamOptimizerWithDecay(n_epochs, T)...)
-    loss_array = o(nn, dl, batch, n_epochs; show_progress = false)
+    o = TrainingOptimizer(nn.params; AdamOptimizerWithDecay(n_epochs)...)
+    loss_array = train!(nn, o, dl, batch, n_epochs; show_progress = false)
     @test length(loss_array) == n_epochs
     @test all(isfinite, loss_array)
 end
@@ -41,7 +41,7 @@ end
 # then fails to match them against the parameter
 @testset "structured weights keep their type in `similar`" begin
     for A in (rand(SymmetricMatrix{Float64}, 4), rand(SkewSymMatrix{Float64}, 4),
-        rand(LowerTriangular{Float64}, 4), rand(UpperTriangular{Float64}, 4))
+        rand(StrictlyLowerTriangular{Float64}, 4), rand(StrictlyUpperTriangular{Float64}, 4))
         @test typeof(similar(A)) == typeof(A)
         @test typeof(zero(A)) == typeof(A)
         @test size(similar(A)) == size(A)
@@ -51,7 +51,7 @@ end
 @testset "`LASympNet` (SymmetricMatrix)" begin
     optimizer_runs(LASympNet(4), Batch(5), 4)
 end
-@testset "`VolumePreservingFeedForward` (Lower/UpperTriangular)" begin
+@testset "`VolumePreservingFeedForward` (StrictlyLower/StrictlyUpperTriangular)" begin
     optimizer_runs(VolumePreservingFeedForward(4), Batch(5), 4)
 end
 @testset "`VolumePreservingTransformer` (SkewSymMatrix + triangular)" begin
diff --git a/test/optimizers/svd_optim.jl b/test/optimizers/svd_optim.jl
index 033819dee..68035173a 100644
--- a/test/optimizers/svd_optim.jl
+++ b/test/optimizers/svd_optim.jl
@@ -1,4 +1,5 @@
 using GeometricMachineLearning, LinearAlgebra, KernelAbstractions, Test, Zygote, Random
+using GeometricOptimizers: optimization_step!
 
 Random.seed!(1234)
 
@@ -25,7 +26,7 @@ A = [0.06476993260924702 0.8369280855305259 0.6245358125914054 0.140729967064923
 
 Random.seed!(1234)
 
-function svd_test(A, n, train_steps = 1000, tol = 1e-1; retraction = cayley)
+function svd_test(A, n, train_steps = 1000, tol = 1e-1; retraction = Cayley())
     # Seeded here and not once at the top of the file. `svd_test` is called twice, and with a single
     # top-level seed the second call starts from whatever RNG state the first one happened to leave
     # behind -- so the convergence assertions below turned on how much randomness the optimizer
@@ -41,9 +42,9 @@ function svd_test(A, n, train_steps = 1000, tol = 1e-1; retraction = cayley)
     model = Chain(StiefelLayer(N, n), StiefelLayer(n, N))
     ps = NeuralNetwork(model, CPU(), Float64).params
 
-    o₁ = Optimizer(GradientMethod(), ps; retraction = retraction, step_size = 0.01)
-    o₂ = Optimizer(MomentumMethod(), ps; retraction = retraction, step_size = 0.01)
-    o₃ = Optimizer(Adam(), ps; retraction = retraction, step_size = 0.01)
+    o₁ = TrainingOptimizer(ps; algorithm = GradientMethod(), retraction, linesearch = 0.01)
+    o₂ = TrainingOptimizer(ps; algorithm = MomentumMethod(), retraction, linesearch = 0.01)
+    o₃ = TrainingOptimizer(ps; algorithm = Adam(), retraction, linesearch = 0.01)
 
     U₁, Ũ₁, err₁ = train_network!(o₁, model, deepcopy(ps), A, train_steps, tol)
     U₂, _, err₂ = train_network!(o₂, model, deepcopy(ps), A, train_steps, tol)
@@ -59,18 +60,17 @@ function svd_test(A, n, train_steps = 1000, tol = 1e-1; retraction = cayley)
     @test norm((err₃ - err_best)/err_best) < tol
 end
 
-function train_network!(o::Optimizer, model::Chain, ps::NetworkParameters,
+function train_network!(o::TrainingOptimizer, model::Chain, ps::NetworkParameters,
         A::AbstractMatrix, train_steps, tol)
     error(ps) = norm(A - model(A, ps))
 
     for _ in 1:train_steps
         dx = Zygote.gradient(error, ps)[1]
-        λY = GlobalSection(ps)
-        optimization_step!(o, λY, ps, dx)
+        optimization_step!(ps, o, dx)
     end
     ps.L1.weight, ps.L2.weight, error(ps)
 end
 
-for retraction in (geodesic, cayley)
+for retraction in (Geodesic(), Cayley())
     svd_test(A, 3, retraction = retraction)
 end
diff --git a/test/pullbacks/get_contents.jl b/test/pullbacks/get_contents.jl
new file mode 100644
index 000000000..450897097
--- /dev/null
+++ b/test/pullbacks/get_contents.jl
@@ -0,0 +1,28 @@
+using GeometricMachineLearning
+using NeuralNetworkParameters: NetworkParameters
+using Test
+import Random
+
+Random.seed!(1234)
+
+GML = GeometricMachineLearning
+
+# `_get_contents` unwraps whatever shape the pullback handed back. Six methods, because a reverse pass
+# produces a container or the bare `NamedTuple` it wraps, and `Zygote` may hand either back bare, in a
+# one-tuple, or in a one-element vector. Each is exercised here: splitting a union into a method per
+# shape means each method needs its own case, or the split trades one tested path for six untested
+# ones.
+@testset "_get_contents unwraps every shape a pullback returns" begin
+    nt = (L1 = (W = randn(2, 2), b = randn(2)),)
+    ps = NetworkParameters(nt)
+
+    for x in (nt, ps)
+        @test GML._get_contents(x) === x
+        @test GML._get_contents((x,)) === x
+        @test GML._get_contents([x]) === x
+    end
+
+    # more than one set is a caller error and says so, rather than silently taking the first
+    @test_throws ArgumentError GML._get_contents([ps, ps])
+    @test_throws ArgumentError GML._get_contents([nt, nt])
+end
diff --git a/test/quality/aqua.jl b/test/quality/aqua.jl
index 8ce09ad3d..07230abd9 100644
--- a/test/quality/aqua.jl
+++ b/test/quality/aqua.jl
@@ -3,41 +3,25 @@ using ExplicitImports
 using GeometricMachineLearning
 using Test
 
-# Aqua's package-level checks. Six of the eight `Aqua.test_all` enables by default run; `ambiguities`
-# and `piracies` are switched off by name, and the reason is below rather than in the changelog
-# alone, because switching one back on turns the suite red on the spot.
+# Aqua's package-level checks. Seven of the eight `Aqua.test_all` enables by default run; `ambiguities`
+# is switched off by name, and the reason is below rather than in the changelog alone.
 #
 # `Aqua.test_all` wraps each of its checks in a `@testset` of its own and adds no enclosing one, and
 # a testset with no parent finalises as soon as it closes -- so called bare, the first failing check
 # throws a `TestSetException` and the rest never run. On the suite path the `@safetestset` in
 # `runtests.jl` is that parent. The `@testset` below is that parent when this file is run on its
-# own, and it is also what lets the two counts at the end be further assertions rather than
-# top-level `@test`s that abort the file the moment one fails.
-#
-# WHAT IS SWITCHED OFF, AND WHY IT IS NOT A `broken = true`.
-#
-# `piracies` reports 3 methods, and all 3 are genuine under Aqua's definition -- the function and
-# every argument type belong to other modules. They are not a count waiting to be triaged. All
-# three are the same thing: a three-argument functor for a *layer* type that this package does not
-# own, `AbstractNeuralNetworks`' `Dense` and `Linear`, added so that the layer accepts a 3-tensor
-# input as well as a matrix.
-#
-#     Dense{M, N, true}   on an AbstractArray{T, 3}    src/layers/resnet.jl:63
-#     Dense{M, N, false}  on an AbstractArray{T, 3}    src/layers/resnet.jl:67
-#     Linear{M, N}        on an AbstractArray{T, 3}    src/layers/resnet.jl:71
-#
-# Closing these means either upstream gaining the 3-tensor methods or this package wrapping the two
-# layer types, and both are an API change rather than a tidy-up. They are *B7* under
-# `## Open Issues` in `CHANGELOG.md` with that reasoning.
-#
-# `ambiguities` reports 1, and the same 1 whether this package is loaded alone or with everything
-# `runtests.jl` loads before this file. It is a `Dense{M, N, true}` functor against
-# `AbstractNeuralNetworks.Affine`'s, and it is benign: `Dense` is not a subtype of `Affine` and the
-# two have no common instance, so no call can reach the pair. It is the same `resnet.jl:63` method
-# as the first piracy above.
-#
-# The two settings agreeing is what makes the gate below trustworthy, and it holds because no
-# method of this package claims an argument position wholesale on `PoissonTensor`. That type is an
+# own, and it is also what lets the count at the end be a further assertion rather than a
+# top-level `@test` that aborts the file the moment it fails.
+#
+# `piracies` runs. The 3-tensor functors of `AbstractNeuralNetworks`' `Dense` and `Linear`, and the
+# `save`/`load` methods on `NeuralNetwork`, are `AbstractNeuralNetworks`' own.
+#
+# `ambiguities` stays off, and the assertion below replaces it. Aqua's check runs in a process of its
+# own and asks about whatever that process loaded, so its count moves with ArrayLayouts',
+# FillArrays', Symbolics', GeometricOptimizers', BandedMatrices' and BlockArrays' versions, and with
+# *which* of them a given process had loaded. `detect_ambiguities` on this package, in this process,
+# asks only about the methods it defines; the count is zero, and no method of this package claims an
+# argument position wholesale on `PoissonTensor`, which is what keeps it there. That type is an
 # `AbstractMatrix{T}`, so such a method meets every special-array method another package writes
 # against `AbstractMatrix`:
 #
@@ -48,76 +32,61 @@ using Test
 #       `BlockIndex` and `BandRangeType` methods BandedMatrices and BlockArrays add to
 #       `AbstractMatrix`. Neither of those two loads with this package alone, so an untyped index
 #       pair here is a collision visible only from inside the suite.
-#
-# `ambiguities` still stays off, because 1 is not 0 and Aqua's check has no way to exempt a pair.
-# The assertion below replaces it and is strictly better here: it names the number, so it fails
-# both when an ambiguity is added and when this one is closed without the comment going with it.
-#
-# Marking either check `broken = true` would leave a check that reports success while the defect
-# stands, which is the failure mode this test suite's guards exist to remove.
 @testset "Aqua" begin
-    Aqua.test_all(GeometricMachineLearning; ambiguities = false, piracies = false)
-
-    # A switched-off check detects nothing, so the count it would have reported drifts unobserved,
-    # and so does every `src` line named above. This is the gate. It fails when a piracy is added,
-    # and it fails when one is removed without the entry above and in `CHANGELOG.md` going with it.
-    @test length(Aqua.Piracy.hunt(GeometricMachineLearning)) == 3
+    Aqua.test_all(GeometricMachineLearning; ambiguities = false)
 
-    # The ambiguity count gets the same gate, and it can carry one because the single pair is
-    # between a method of this package and one of `AbstractNeuralNetworks`'. The count is the same
-    # in isolation and in the suite, and nothing outside this repository's own `[compat]` moves it
-    # -- so a red here is a regression, not an unrelated upstream upgrade and not a reordering of
-    # this suite. A count that moved with ArrayLayouts', FillArrays', Symbolics',
-    # GeometricOptimizers', BandedMatrices' and BlockArrays' versions, and with *which* of them a
-    # given process had loaded, could not tell those apart, and could not be asserted.
-    @test length(Test.detect_ambiguities(GeometricMachineLearning; recursive = true)) == 1
+    @test isempty(Test.detect_ambiguities(GeometricMachineLearning; recursive = true))
 end
 
 # `ExplicitImports` answers a question Aqua does not ask, and the reason to gate on it is not
 # tidiness: a `using` or an `import` that nothing needs keeps a dependency reachable from the module
 # file, so Aqua's `stale_deps` above sees a package in use that nothing uses. That check only means
-# something once these two are clean.
+# something once these are clean.
 #
-# Three of the seven checks run: `no_stale_explicit_imports`, `all_explicit_imports_via_owners` and
-# `no_self_qualified_accesses`. The first two were red before this file gained them -- 13 stale
-# explicit imports, and `StateVariable` imported from `GeometricSolutions`, which re-exports it,
-# rather than from `GeometricBase`, which defines it. The third already passed.
-#
-# THE OTHER FOUR, AND WHY EACH IS OFF.
+# Four of the seven checks run here: `no_stale_explicit_imports`, `all_explicit_imports_via_owners`,
+# `all_qualified_accesses_via_owners` and `no_self_qualified_accesses`. Two more run in the testset
+# below with a list of names, and `no_implicit_imports` is off:
 #
 #   no_implicit_imports              37 names arrive through a bare `using`, across thirteen packages
 #                                    -- `norm`, `@kernel`, `rrule`, `pullback` and the rest. Naming
 #                                    every one is a change to the whole module header and a judgement
 #                                    per name, not a by-product of a dead-code pass.
-#   all_explicit_imports_are_public  9 names this package imports are not marked `public` upstream
-#                                    -- `Architecture`, `AbstractExplicitLayer`, `_compute_loss`,
-#                                    `description`, `dim` among them. Each is deliberate and most
-#                                    are re-exported here; the fix is upstream declaring them, not
-#                                    this package importing less.
-#   all_qualified_accesses_via_owners  2: `GeometricOptimizers.Gradient` and
-#                                    `GeometricOptimizers.direction`, both owned by `SimpleSolvers`.
-#                                    Reaching them through `GeometricOptimizers` is how the rest of
-#                                    `src/optimizers/optimizer.jl` is written.
-#   all_qualified_accesses_are_public  24, the same class as the 9 above: `KernelAbstractions.zeros`,
-#                                    `ForwardDiff.jacobian`, `GeometricOptimizers.momentum` and so on.
-#
-# Each of the four is a real backlog item rather than a taste, and none of them is this branch's.
 #
 # WHAT THIS GATE DOES NOT CATCH, and one way it can go red on its own.
 #
-# It reads imports, never definitions. A dead *definition* re-added under `src/` leaves all three
+# It reads imports, never definitions. A dead *definition* re-added under `src/` leaves all the
 # checks green, so this closes only the import half of the pass that added it.
 #
-# `all_explicit_imports_via_owners` also has the property the `ambiguities` comment above rejects:
-# it asks *which* upstream module defines a name, so it goes red when a name moves between
-# `GeometricBase`, `GeometricOptimizers` and `AbstractNeuralNetworks` with nothing in `src/` having
-# changed -- which is exactly how `StateVariable` got here. `ExplicitImports = "1.15"` admits any
-# 1.x, so a minor release that sharpens stale detection does the same. Unlike the ambiguity count,
-# a red here names the import and the module to take it from, so it stays a gate: the fix is one
-# line and the message says which. `test_explicit_imports` opens its own `@testset`, so this call
-# needs no wrapper.
+# The two `via_owners` checks ask *which* upstream module defines a name, so they go red when a name
+# moves between `GeometricBase`, `GeometricOptimizers` and `AbstractNeuralNetworks` with nothing in
+# `src/` having changed. `ExplicitImports = "1.15"` admits any 1.x, so a minor release that sharpens
+# stale detection does the same. A red here names the import and the module to take it from, so it
+# stays a gate: the fix is one line and the message says which. `test_explicit_imports` opens its
+# own `@testset`, so this call needs no wrapper.
 test_explicit_imports(GeometricMachineLearning;
     no_implicit_imports = false,
     all_explicit_imports_are_public = false,
-    all_qualified_accesses_via_owners = false,
     all_qualified_accesses_are_public = false)
+
+# Every name this package reaches in `GeometricOptimizers` is public there, whether it is imported or
+# written `GeometricOptimizers.x`. The names below are the ones it reaches in *other* packages that
+# those packages do not declare public; each is deliberate, and the fix for each is upstream
+# declaring it. No `GeometricOptimizers` name is on either list.
+const NON_PUBLIC_QUALIFIED = (
+    :Initializer, :TanhActivation, :dim,                         # AbstractNeuralNetworks
+    :Jacobian, :ParameterGradient, :symbolic_parameter_gradient, # SymbolicNeuralNetworks
+    :variables,                                                  # Symbolics
+    :copyto!, :zeros,                                            # KernelAbstractions
+    :jacobian)                                                   # ForwardDiff
+const NON_PUBLIC_IMPORTS = (
+    :AbstractExplicitLayer, :Architecture, :_compute_loss, :dim, :model, # AbstractNeuralNetworks
+    :derivative,                                                         # SymbolicNeuralNetworks
+    :load, :save,                                                        # NeuralNetworkParameters
+    :_vcat, :assign_q_and_p)                       # this package, imported by `GPUArraysCoreExt`
+
+@testset "every name reached in GeometricOptimizers is public" begin
+    @test check_all_qualified_accesses_are_public(GeometricMachineLearning;
+        ignore = NON_PUBLIC_QUALIFIED) === nothing
+    @test check_all_explicit_imports_are_public(GeometricMachineLearning;
+        ignore = NON_PUBLIC_IMPORTS) === nothing
+end
diff --git a/test/quality/exports.jl b/test/quality/exports.jl
index 1be6e1300..f3972ab97 100644
--- a/test/quality/exports.jl
+++ b/test/quality/exports.jl
@@ -6,10 +6,6 @@ using Test
 # `using GeometricMachineLearning; ResidualLayer(2)` is an `UndefVarError` for a name the package
 # advertises. The allowlist below is empty, so every exported name resolves; an entry is added here
 # only with the reason a name is exported and defined nowhere.
-#
-# The assertions below test what the loaded module defines. `ext/HDF5Ext.jl` loads alongside HDF5
-# and this test does not load HDF5, so a name defined only in that extension would read as undefined
-# here. No exported name is in that position today.
 const UNDEFINED_EXPORTS = Dict{Symbol, String}()
 
 @testset "every exported name is defined" begin
@@ -23,3 +19,22 @@ const UNDEFINED_EXPORTS = Dict{Symbol, String}()
     # longer exported, has to go; otherwise the next reader trusts a reason that no longer holds.
     @test sort(setdiff(collect(keys(UNDEFINED_EXPORTS)), undefined_exports)) == Symbol[]
 end
+
+# A caller who loads both packages keeps every name of both: each name the two export is one binding,
+# so neither `using` shadows the other, and `Optimizer` is `GeometricOptimizers`' solver.
+@testset "using GeometricOptimizers and GeometricMachineLearning together" begin
+    m = Module()
+    Core.eval(m, :(using GeometricOptimizers, GeometricMachineLearning))
+    @test Core.eval(m, :(Optimizer === GeometricOptimizers.Optimizer))
+
+    GO = Core.eval(m, :GeometricOptimizers)
+    shared = intersect(names(GO), names(GeometricMachineLearning))
+    @test :Adam in shared
+    @test filter(n -> getfield(GO, n) !== getfield(GeometricMachineLearning, n), shared) ==
+          Symbol[]
+
+    # no alias names a method under a second name
+    for alias in (:GradientOptimizer, :MomentumOptimizer, :AdamOptimizer)
+        @test !isdefined(GeometricMachineLearning, alias)
+    end
+end
diff --git a/test/runtests.jl b/test/runtests.jl
index d4fff0fa9..5829cb968 100644
--- a/test/runtests.jl
+++ b/test/runtests.jl
@@ -51,7 +51,7 @@ if "core" in GROUPS
     @safetestset "_norm keeps the element type of its argument" include("utils.jl")
     @safetestset "Symplectic Euler and variational midpoint losses" include("loss/training_method_losses.jl")
     @safetestset "Test NetworkLoss + Optimizer" include("integration/losses_and_optimization.jl")
-    @safetestset "ReducedLoss through the Optimizer functor" include("integration/reduced_loss_optimization.jl")
+    @safetestset "ReducedLoss through train!" include("integration/reduced_loss_optimization.jl")
     @safetestset "Optimizer #2" include("optimizers/optimization_step.jl")
     @safetestset "Optimizer #3" include("optimizers/svd_optim.jl")
     @safetestset "Optimizer #4" include("optimizers/psd_optim.jl")
@@ -59,8 +59,7 @@ if "core" in GROUPS
     @safetestset "Gradient optimizer tests" include("optimizers/gradient_optimizer.jl")
     @safetestset "Momentum optimizer tests" include("optimizers/momentum_optimizer.jl")
     @safetestset "Optimizers with structured (non-manifold) weights" include("optimizers/structured_array_parameters.jl")
-    @safetestset "_GMLGradient dispatch" include("optimizers/gml_gradient_dispatch.jl")
-    @safetestset "The GO-native leaf step scales in the parameter's own element type" include("optimizers/step_size_element_type.jl")
+    @safetestset "_get_contents unwraps every shape a pullback returns" include("pullbacks/get_contents.jl")
     @safetestset "_custom_mul: NetworkParameters gradient structure" include("layers/double_multiplication_network_parameters_gradient.jl")
     @safetestset "Symplectic attention: NetworkParameters gradient structure" include("layers/symplectic_attention_network_parameters_gradient.jl")
     @safetestset "map_to_cpu" include("map_to_cpu.jl")
@@ -68,12 +67,14 @@ if "core" in GROUPS
     @safetestset "HDF5 save/load for GML special array types" include("integration/hdf5_support.jl")
     @safetestset "Test data loader for q and p data" include("data_loader/batch_data_loader_qp_test.jl")
     @safetestset "Test the data loader in combination with optimization_step!" include("data_loader/data_loader_optimization_step.jl")
-    @safetestset "Optimizer functor with data loader for Adam" include("data_loader/optimizer_functor_with_adam.jl")
+    @safetestset "One epoch of train! with Adam on a data loader" include("data_loader/optimizer_functor_with_adam.jl")
     @safetestset "Test data loader for a tensor (q and p data)" include("data_loader/draw_batch_for_tensor_test.jl")
     @safetestset "Batch functor(s)" include("data_loader/batch_functor.jl")
     @safetestset "DataLoader for input and output" include("data_loader/data_loader_for_input_and_output.jl")
     @safetestset "An unexpected `autoencoder` keyword throws rather than returning nothing" include("data_loader/autoencoder_keyword.jl")
     @safetestset "A Float32 network's training history stays Float32" include("data_loader/training_history_eltype.jl")
+    @safetestset "train!: convergence, resume, history type, n_epochs" include("data_loader/train.jl")
+    @safetestset "train! takes a TrainingOptimizer's algorithm and linesearch" include("optimizers/optimizer.jl")
     @safetestset "The minibatch index set is concrete and grows linearly" include("data_loader/batch_index_set.jl")
     @safetestset "Classifier accuracy" include("data_loader/accuracy.jl")
     @safetestset "Data loader docstring examples" include("integration/docstrings/data_loader.jl")
@@ -88,6 +89,7 @@ end
 if "slow" in GROUPS
     @safetestset "Lagrangian Neural Network" include("architectures/lagrangian_neural_network_tests.jl")
     @safetestset "Test symplecticity of the sympnet upscaling layer" include("layers/sympnet_upscaling.jl")
+    @safetestset "train! trains every architecture of the manual" include("integration/train_manual_architectures.jl")
 end
 if "doctests" in GROUPS
     @safetestset "Doctests" include("quality/doctests.jl")