ww3-gpu

_{the WW3 GPU Lab · since 2026 · the site · the book · releases · cite · português}

in one minute

• WW3 forecasts ocean waves. WAVEWATCH III® (WW3) keeps a spectrum at each sea point of a map. The spectrum tells how much wave energy moves in each direction and at each frequency. At each time step, WW3 changes that energy. The wind adds energy, breaking waves remove energy, waves exchange energy, and the energy moves across the map. NOAA uses WW3 for its operational wave forecasts. WW3 is written in Fortran and runs on CPUs.
• ww3-gpu makes WW3 faster with Kokkos and CUDA. We translate the slowest routines into C++ with Kokkos, one routine at a time, in the order of measured run time. With Kokkos, the same C++ code runs on CPU cores (OpenMP) or on an NVIDIA GPU (CUDA).
• The answer must not change. A translated routine must give the same numbers as the Fortran, to the last bit, in the same program. We do not edit WW3. A fork of WW3 adds a switch. At run time, you can select the Fortran routine or the Kokkos routine for each translated part, so you can always compare the two.
• The repository is also a book. The book tells how coding agents do this work under that rule. Its chapters are the 17 lessons, and each merge builds them into a PDF. The first readers are scientists in wave modelling, numerical methods and HPC. Each claim has (v) when we checked it and ⚠ when we did not.

where to start

If you want to	Go to
learn about WW3, GPUs and this project from the start	course/00-orientation.md, then the book
run WW3, then WW3 with the Kokkos port	quick start
see which routines run on Kokkos, and how fast	where it stands, kokkos/PORT_STATUS.md
read the CPU results, their limits and what is not shown yet, as written for NOAA's wave modellers	docs/NOAA_PROGRESS_RECORD_202610.md, then the gap list docs/PRODUCTION_READINESS_202610.md
see the system in figures	docs/WW3_GPU_VS_NOAA_202610.md: five figures, also as PDF files
work here with a coding agent	AGENTS.md, then the engineering loop
find the meaning of a word or an abbreviation	docs/GLOSSARY.md
cite this work	how to cite
help	CONTRIBUTING.md. The most useful help is to check a claim with ⚠

how it works

scripts/ww3-gpu.sh runs WAVEWATCH in three modes. The model source code comes from three submodules. This repository contains only the Kokkos code and its tools.

Directory	What it is	Used by
WW3-noaa/	WW3 from NOAA: NOAA-EMC/WW3 at 03adda91, with no change	--ww3, the reference
WW3/	the fork h0ffmann/WW3 at 50a10730 (branch ww3-gpu/kokkos-pnt): WW3-noaa/ plus the KOKKOS patch. The patch changes 9 files and adds 7 modules under model/src/, with 5 687 added lines and no removed line	--ww3-gpu and the regtest matrix
WW4/	h0ffmann/WW4, develop	--ww4

(v) The numbers of the patch are from git -C WW3 diff --stat 03adda91 50a10730 -- model (16 files, 5 687 insertions, 0 deletions), on 2026-10-18.

The fork has one purpose. WW3 must call the Kokkos code from inside WW3, for example from W3SRCE (model/src/w3srcemd.F90), and this repository does not edit WW3. Thus the fork adds the calls behind the KOKKOS switch, and each Fortran routine stays available. Before each --ww3 run, scripts/ww3-gpu.sh compares the two source trees. If the fork is different from WW3-noaa/ outside the KOKKOS patch, the script stops with exit code 3. Thus --ww3 is always WW3 from NOAA, and --ww3-gpu --cpu compares the Kokkos port with it.

The figure shows the full system: the sources of the model, the code of this repository, the run modes and the three runtimes.

%% figure: ww3-gpu-at-a-glance
%% title: What does ww3-gpu build, from which sources, and on which runtime does each run mode run?
flowchart TD
    subgraph RT["Runtimes"]
        C["3 · dev container<br/>Dockerfile dev stage + Nix"]
        P["1 · picasso/, Nix<br/>gfortran 16.2 · Open MPI · CUDA 12.9<br/>the reference toolchain"]
        D["2 · Docker image<br/>Ubuntu, gfortran 15.2 · CPU only"]
    end
    H["scripts/ww3-gpu.sh, justfile, regtests/<br/>Bash · Python · CMake, CTest"]
    subgraph RUN["Run modes"]
        M1["--ww3<br/>the reference"]
        M2["--ww3-gpu<br/>Fortran and Kokkos DIA,<br/>one executable"]
        M4["just ww3-gpu-matrix<br/>WW3's regtests"]
        M3["--ww4"]
    end
    subgraph SRC["Model sources: submodules, never edited here"]
        N["WW3-noaa/<br/>pure NOAA WW3 · Fortran"]
        F["WW3/<br/>h0ffmann fork · Fortran<br/>+ KOKKOS switch, shim call"]
        W4["WW4/<br/>h0ffmann fork · C++"]
    end
    subgraph OWN["ww3-gpu's own code"]
        B["course/, pubs/, site/<br/>Markdown to PDF · Gleam site"]
        K["kokkos/<br/>the port · C++20, Kokkos"]
    end
    C -- "enters" --> P
    P -- "every mode;<br/>GPU only with the owner" --> H
    D -- "CPU only:<br/>the just d… recipes" --> H
    H --> M1 & M2 & M4 & M3
    M2 -. "compared with" .-> M1
    M1 -- "built from" --> N
    M2 -- "built from" --> F & K
    M4 -- "built from" --> F & K
    M3 -- "built from" --> W4

Loading

How to read this figure

Takeaway. Two copies of WW3 come in from outside, pure NOAA WW3 and the owner's fork; this repository adds only the C++ rewrite and its tools, and every way of running the model works on the Nix toolchain, while the Docker image covers the CPU runs without Nix.

How to read. Top to bottom. A runtime runs the one script, scripts/ww3-gpu.sh with its recipes; the script offers the run modes; each run mode is built from the sources below it. Solid arrows say what enters, runs or is built from what; the dashed arrow is the comparison of the Kokkos run with pure NOAA WW3. The runtimes are numbered from the reference (1) to the most derived (3); the dev container reaches the modes only through picasso.

Not shown. The book and the site (course/, pubs/, site/) build in CI, not on these runtimes. The fork's branch and the files of its KOKKOS patch (the table above), the CUDA build and its supervision rule (AGENTS.md, "Running on the GPU"), CI, and the versions of each runtime (VERSIONS.md).

Evidence. (v) read 2026-10-15: the table and paragraph above; scripts/ww3-gpu.sh (header); docker/Dockerfile (stages toolchain, build, dev, runtime); .devcontainer/devcontainer.json; VERSIONS.md for the compiler versions. The dev container ran just kokkos-test serial-debug and just smoke-ww3-gpu through picasso on 2026-10-15 (Dev Containers CLI 0.89.0, full clone).

the Kokkos switch, part by part

In the --ww3-gpu executable, each translated part has a run-time flag, WW_KOKKOS_<PART>=1. If the flag is not set or is 0, the part runs the Fortran routine. scripts/ww3-gpu.sh runs the case two times with the same executable. The first run has all flags off. The second run has the parts that you select. Then the script compares the outputs of the two runs. Select the parts with --kokkos:

--kokkos	Parts on Kokkos	Use
none	none	the Fortran only, in the Kokkos build
all	all the parts of the build (WW_KOKKOS_ALL=1), but not the opt-in parts below	the widest test of the translation
gpu	SRCE	the default of --gpu: all of W3SRCE in each call (ADR-0013)
resident	SRCE,XYP3,KTP3,OUTG,IOPE,UPDV,CPRT	the full time step on Kokkos, with the spectrum kept on the Kokkos side (next section)
a list, for example SNL1,SDS4	the parts in the list	a test of one part at a time

what --kokkos resident does

A GPU has its own memory. If each routine copies the spectrum of a sea point to the GPU and back, the copies cost more time than the GPU saves. (v) In the September 2026 benchmark of the DIA kernel, one call took 0.75 ms in total and 0.047 ms in the kernel (bench/results/). Thus the copies used most of the time.

resident keeps the spectrum array, VA, on the Kokkos side for the full run. It also moves each routine of the time step that reads or writes VA to Kokkos. Thus VA crosses to the Kokkos side one time at the start. It comes back only when WW3 writes output (WFIP-0005).

Part	WW3 routine	What runs on Kokkos
SRCE	W3SRCE	the source terms of all sea points in one batched call: ST4 wind input and whitecapping, the DIA (W3SNL1), W3SLN1, bottom friction (W3SBT1), depth-induced breaking (W3SDB1) and the integration
XYP3	W3XYP3	the propagation in space (PR3 with UQ), with the gather and the scatter of the spectrum
KTP3	W3KTP3	the propagation in frequency and direction (refraction, shoaling)
OUTG	W3OUTG	the gridded output fields (wave height, periods and other fields), calculated where VA is
IOPE	before W3IOPE	the point output: only the spectra that W3IOPE interpolates come back
UPDV	after W3UICE	the ice update: the zero values that W3UICE writes in VA are also written on the Kokkos side

• (v) On the CPU, the outputs are byte-identical to the Fortran of the same executable, on all three test cases (kokkos/src/fortran_iface/PATCH.md, sections 11 to 18).
• ⚠ resident is opt-in. It is not a default. XYP3, KTP3, OUTG, IOPE and UPDV did not pass WW3's regtests yet. The regtests wait for the validation phase (ADR-0016).
• ⚠ The first end-to-end timing is one run on the CPU. Refer to where it stands.

To run resident:

scripts/ww3-gpu.sh --ww3-gpu full --cpu --kokkos resident            # OpenMP; compares with the Fortran and with --ww3
scripts/ww3-gpu.sh --ww3-gpu full --cpu --kokkos resident --timers   # shows the time of each part
scripts/ww3-gpu.sh --ww3-gpu full --gpu --kokkos resident            # CUDA: a person must type "yes" at the terminal

To see how many times VA crossed, and in which direction, set WW_KOKKOS_VA_COUNT=1.

the three test cases

Case	What it is
smoke	WW3's regtest ww3_ts1: one point, ST4 and NL1, 36 frequencies × 24 directions, 10 model days. It runs in seconds on a CPU
full	kokkos/tests/fixtures/bench_small: a 120 × 80 grid, 9 520 sea points, 24 × 24, 12 hours
coast	kokkos/tests/fixtures/coast: 80 × 50 cells of 5 km, 3 791 sea points and 50 boundary points, 24 × 24, 12 hours. It has hourly currents, water levels, ice and boundary spectra (scripts/coast_case.sh). These inputs use code paths that the other two cases do not use

Each run mode has a just recipe:

just smoke             # smoke-ww3, then smoke-ww3-gpu: regtest ww3_ts1, seconds
just smoke-ww3         # --ww3: WW3 as written
just smoke-ww3-gpu        # --ww3-gpu --cpu: WW3 with the Kokkos DIA (W3SNL1), compared with the Fortran and --ww3
just full-ww3          # the expensive case: bench_small, 9 520 sea points, minutes
just full-ww3-gpu         # the same with the Kokkos DIA
just smoke-ww4         # --ww4: WAVEWATCH IV; exit 5 while a part the case needs is not ported
just full-ww4          # the same on bench_small; `just ww4-parts full` lists the parts
just port-score        # how much of WW3 runs on Kokkos, and the routines ranked by time

where it stands

The numbers below are from kokkos/PORT_STATUS.md on 2026-10-17 (just port-score). The shares of run time are from a gprof profile of bench_small on one thread (just profile-case full).

Level	Routines	Share of the profiled run time
translated: a Kokkos version exists	310 of 1 023	99.3 %
ported: WW3 calls it when its flag is set	60	99.3 %
regtested: WW3's regtests agree with the Fortran	10	48.6 %

• (v) The DIA (W3SNL1) is byte-identical to the Fortran on all 100 serial NL1 entries of WW3's regtests, on the CPU.
• (v) On an RTX 4090, the DIA kernel took 0.047 ms for 1 000 sea points. One CPU thread took 24.96 ms (bench/results/).
• ⚠ The full model has one end-to-end timing, on the CPU. With --kokkos resident, the full case took 17.98 s. The Fortran of the same executable took 41.92 s (2.33x). All outputs were byte-identical. This is one run on a loaded machine, and the regtests are pending (log entry).
• The full model did not run on a GPU yet. A person starts each --gpu run at the terminal (AGENTS.md).
• WW3's regtests wait for a validation phase, which the owner starts. Until then, the parity argument comes from the compiled code of the two sides (ADR-0016).

quick start

You must have Nix and just. The compilers come from the pinned toolchain in picasso/. You can also use Docker or a dev container: refer to three runtimes.

git clone --recurse-submodules [email protected]:h0ffmann/ww3-gpu.git && cd ww3-gpu
just submodule-init       # sparse checkout of picasso (one time for each clone)
just smoke                # WW3 from NOAA, then WW3 with Kokkos, compared (ww3_ts1, seconds)
just full-ww3-gpu         # the same on bench_small, 9 520 sea points
just port-score           # the part of WW3 that runs on Kokkos, and the routines in order of time

Then read course/00-orientation.md. To see all the tasks, run just.

everything else

Click a section to open it.

the book

The book that this repository builds: its subject, and how to build it

Without Changing the Answer (working title) is written in the tradition of self-published technical books built from a public repository, such as Sandy Maguire's Thinking with Types and Gabriel Volpe's Practical FP in Scala (v) marola's MIP-0014 §4, read 2026-10-08. Its subject is a method: how coding agents port Fortran under a bit-for-bit gate and a validation ladder (agentic coding), and how plans, evaluations and measurements are written so that an agent can be held to them (agentic research). The rules in AGENTS.md, the design docs in docs/WFIPs/, the decisions in docs/ADRs/ and the research log in docs/log/ are chapters in waiting. ADR-0003 records the decision and WFIP-0002 the plan.

Each tagged release is an edition, archived on Zenodo as a book (RELEASES.md). A free Leanpub edition is fed by CI from the same chapters (docs/LEANPUB_202610.md).

just book          # the chapters as a PDF (nix, picasso's publisher lab)
just book-html     # the chapters as web pages, into build/site/book/
just leanpub       # the chapters as a Leanpub manuscript
just adrs          # one line per decision
just wfip check    # the design docs and their definitions of done

the website

The website: what it shows, and how to build it

https://h0ffmann.github.io/ww3-gpu/ is a Gleam + Lustre front page, prerendered, with no script. Every fact on it is read from this repository's files by site/content: the chapters, the decisions, the designs, the notes, the DOI and the people. The chapters are at /book/, next to the PDF, the progress page (/andamento/), the decisions (/adrs/) and an easy Brazilian Portuguese report for the GTA, DEL/UFRJ (/relatorio-gta/, from pubs/relatorio-gta/, PDF in pdf/relatorio-gta-pedro-cruz.pdf). DESIGN.md has its look: a sea at night, and three lights on it. The same page is built in Brazilian Portuguese at /pt-BR/, from one source and a table of translations, site/data/translations/pt-BR.tsv: the build fails when a sentence has no row.

just site          # the chapters, then the front page, into build/site/
just site-serve    # serve build/site on http://localhost:8000

Environment details

Type	Program
Language	Gleam (JavaScript target), Lustre
Bundling and tests	node, esbuild, gleeunit
Toolchain	picasso's labs/gleam-website, through flake.nix (nix develop .#site)
Book pages	pandoc (scripts/build_html.sh)
Deploy	.github/workflows/pages.yml, GitHub Pages

the second brain

The notes, the research log and the shared memory of the agents

The ideas this lab stands on are notes in notes/, one idea in one note, each with its source and a status (unverified until a source is checked), linked to each other, under NOTES.md. What was tried, including what failed, is the journal: docs/log/, append-only. Where each kind of memory lives is in MEMORY.md; the long-term memory across sessions and agents is ai-memory, chosen in ADR-0009.

Part of that memory is published. The ai-memory pages the owner tags publish are copied into memory/, which starts with HUMAN.md: who owns this repository and the standing orders they gave, in their own words. These pages are leads, not evidence; what matters is also in a file here with its (v) or ⚠. With the ai-memory plugin installed for Claude Code (MEMORY.md has the steps), load them into your own store after cloning and after each pull that changes memory/. They land under published/, unpinned and never as rules, so a page from a pull request cannot become a standing order in your store.

just notes          # the slip box keeps its rules: front matter, ids, links, the index
just refs           # every reference of the proposal has a verdict in its REFERENCES.md
just memory import  # load memory/ into your ai-memory; unchanged pages are skipped
just memory export  # owner: copy the wiki pages tagged `publish` into memory/
just memory check   # memory/ holds only published pages: no session, log, e-mail or home path

study areas

Each area of work, its question and its state

Each line is a separate piece of work. Merged ones live on main; the rest are open pull requests or issues, linked so you can follow them.

Area	Question it answers	Where	State
The course	How do you build, run and measure WW3, then port a kernel?	course/, examples/, exercises/	17 lessons, merged
Kokkos port of W3SNL1	Can a WW3 kernel run on a GPU with bit-identical results?	kokkos/, lesson 12	Bit-identical on 3 backends in the unit tests ADR-0010 removed; WW3 regtest matrix on the CPU byte-identical to the Fortran DIA on all 100 serial NL1 entries (2026-10-15)
Benchmarks	What do an i9 and an RTX 4090 actually give WW3?	bench/, gpu/, lesson 09	Tooling merged
Port priority	Which routine should be ported next?	ADR-0002, PORT_STATUS.md	Port by measured wall time. A first profile puts W3SDS4 at 67 % of source-term time (#45)
Porting with agents	How can coding agents port forty routines without a human redoing the checks?	AGENTS_KOKKOS, lesson 13, #42	Rules merged; task queue and parity ladder in #42
Agent tooling	Which agent frameworks and research tools fit that workflow?	#25, #41, #38, ADR-0009	Evaluations in review; ai-memory chosen
Bit-for-bit proof	What can be proved, and not only tested, about the Fortran → C++ translation?	BITWISE_PROOF, docs/proof/, ADR-0001	No proof language; exhaustive sweeps, GIMPLE diffs and SMT queries. W3SNL1 section 1 swept over all 2^32 inputs
Triton and ML weather forcing	Does a Triton kernel of the W3SDS4 cumulative term beat the Kokkos one? Does WeatherNext 3 wind improve the wave forecast?	#45, W3SDS4_TRITON_PLANO (pt-BR)	Planned
Single-H100 port	What would a full port to one H100 take?	KOKKOS_H100_PLAN, WFIP-0005, WFIP-0006	Plan. The device-resident step (--kokkos resident) runs on the CPU byte-identical to the Fortran, with the spectrum crossing once each way on bench_small (#179); the device's libm calls go to copies of glibc's (#178). Nothing has run on a GPU
WW4 and SWAN	What replaces WW3, and what covers the coast?	lesson 14, lesson 15	Merged
The book and the proposal	The lessons compiled as a book, and the project proposal	pubs/, PDFs and Word files in pdf/, WFIP-0002	Built by CI on every merge

docs/AWESOME-WW3_202609.md is a curated link list, and docs/GLOSSARY.md expands every abbreviation, switch, routine and tool name.

three runtimes

Nix (the reference), the Docker image and the dev container: what each one can run

ww3-gpu runs in three places. All three run the same scripts/ww3-gpu.sh, with the same options, checks and exit codes; they differ in their compilers and in what they can run. picasso is the reference: the bit-for-bit verdicts, the regtest matrix and every timing come from it. VERSIONS.md lists every pinned version of each, and the figure in "can WW3 use my GPU?" shows how they fit with the sources and the run modes.

	1 · picasso (Nix)	2 · Docker image	3 · dev container
You need	Nix and just	Docker (just is optional)	Docker, and VS Code or the Dev Containers CLI 0.89.0 or later; a full clone
What it is	the submodule picasso/ (labs/pratico): one locked nixpkgs	docker/Dockerfile, final stage: the compiled programs and the case inputs, no sources	docker/Dockerfile, stage dev: Ubuntu's compilers, the gate tools and Nix, which enters picasso
Compilers	gfortran 16.2.0, CMake 4.4.3, Kokkos 5.2.2	Ubuntu 26.04: gfortran 15.2.0, Kokkos 5.2.2 from its release tarball	picasso's, inside nix develop; Ubuntu's outside it
MPI, PDLIB	Open MPI 5.0.11, PT-Scotch, METIS and ParMETIS	none	picasso's
CUDA	the #cuda shell (Kokkos for ADA89)	none	none
WW3's regtests	all of them: just ww3-gpu-matrix, NOAA's data pinned by SHA-256	ww3_ts1 and bench_small only	picasso's, CPU only
Bit for bit	the reference toolchain	holds inside one executable of the image; the image is not the reference	picasso's
Commands	just smoke, just full-ww3-gpu, ...	the same with a d in front: just dsmoke, ...	picasso's, inside the container

1 · picasso: the reference

picasso is the Nix toolchain of this lab. The compilers come from its pinned flake, so there is nothing else to install.

git clone --recurse-submodules [email protected]:h0ffmann/ww3-gpu.git && cd ww3-gpu
just submodule-init             # sparse-checkout picasso (once per clone)
just get                        # clone upstream NOAA-EMC/WW3 develop into ~/src/WW3
just rt                         # build with ww3_tp1.1's own switch and run that regtest (~30 s)
just build                      # rebuild with the lab switch (switches/switch_lab_shrd, ST4 physics)
just example01                  # first course example: fetch-limited growth (~1 min)
just kokkos-test serial-debug   # build and test the C++/Kokkos kernels

More recipes on the same toolchain:

just ww3                # enter the toolchain shell
just smoke              # WW3 as written, then the Kokkos DIA against the Fortran DIA (ww3_ts1)
just full-ww3-gpu       # the Kokkos DIA on bench_small
just coast-ww3-gpu      # the forced case: currents, water levels, ice, boundary input (--kokkos resident)
just ww3-gpu-matrix     # WW3's regtest matrix, CPU only

Then start at course/00-orientation.md. just lists every task; each recipe is a thin wrapper over a script in scripts/, except just ww3-gpu-matrix*, which drive WW3's regtest matrix through CMake/CTest in regtests/. The toolchain, the WW3 fork and the Kokkos presets are in docs/TOOLCHAIN.md. For coastal cases, add SWAN with just swan.

Environment details

Type	Program
Fortran, MPI, NetCDF	picasso/labs/pratico, pinned (just ww3, just dev)
C++ kernels	Kokkos (Serial, OpenMP, CUDA), GoogleTest
Publications	picasso's labs/publisher through flake.nix
Host alternative	just prereqs on Debian/Ubuntu
Every pinned version	VERSIONS.md

The submodule directory was nix-config/ until 2026-10-15. In a clone made before that date, run this once after git pull:

git submodule sync && just submodule-init
[ -z "$(git -C nix-config status --short)" ] && rm -rf nix-config   # only when it holds no edits

2 · the Docker image

The image is Ubuntu 26.04 with WW3, WW3 with the Kokkos DIA and WW4 already compiled: it holds the programs, their runtime libraries and the case inputs, not the sources. CPU only.

git clone --recurse-submodules [email protected]:h0ffmann/ww3-gpu.git && cd ww3-gpu
just dbuild         # build the image ww3-gpu:local from this checkout
just dsmoke         # WW3 as written, then the Kokkos DIA against the Fortran DIA (ww3_ts1, seconds)
just dfull-ww3-gpu     # the Kokkos DIA on bench_small (9 520 sea points, 12 hours)
just dsmoke-ww4     # WAVEWATCH IV; exit 5 while a part the case needs is not ported
just drun --help    # any other scripts/ww3-gpu.sh option

Every recipe that runs scripts/ww3-gpu.sh on the host has its pair in the image, with a d in front: smoke, smoke-ww3, smoke-ww3-gpu, full-ww3, full-ww3-gpu, smoke-ww4 and full-ww4 become dsmoke, dsmoke-ww3, and so on. The image needs no Nix and no picasso: its build stage installs Ubuntu's compilers and libraries, compiles Kokkos 5.2.2 from its release tarball (Ubuntu packages an older one), and the final stage copies only the compiled programs.

No GitHub Actions workflow builds a Docker image on its own: every one starts by hand from Actions (Run workflow). .github/workflows/docker-scan.yml scans the image with Trivy when run: a HIGH or CRITICAL vulnerability with a fix fails the job. just dscan runs the same scan here.

The Kokkos code on the GPU has its own image, docker/Dockerfile.cuda: NVIDIA's CUDA 13.4 on Ubuntu 26.04, Kokkos 5.2.2 built with CUDA for compute capability 8.9 (the RTX 4090; KOKKOS_ARCH picks another), with --ww3-gpu --gpu and, as its reference on the CPU, --ww3. Its kernels call CUDA's libdevice, not the device copies of glibc's libm (WW_KOKKOS_LIBM=OFF: Ubuntu ships libmvec stripped, and the copies need a symbol of it), so a GPU run in the image is judged by tolerance (ADR-0014). The host needs the NVIDIA driver and the NVIDIA Container Toolkit for docker run --gpus all. As on the host, every --gpu run asks the supervising person for yes at the terminal (Running on the GPU).

just dbuild-cuda            # build ww3-gpu-cuda:local
just dsmoke-ww3-gpu-cuda    # --ww3-gpu smoke --gpu: the Fortran and the CUDA Kokkos routines, same executable
just dfull-ww3-gpu-cuda     # the same on bench_small

Without just, the same with plain Docker:

docker build -f docker/Dockerfile -t ww3-gpu:local .
docker run --rm --user "$(id -u):$(id -g)" -v "$PWD/build/docker-runs:/work/runs" ww3-gpu:local --ww3-gpu smoke

Every run lands in build/docker-runs/. Inside, the entry point is scripts/ww3-gpu.sh, the same script the host uses, so the options, the checks and the exit codes are the same (docker/Dockerfile says what each stage builds). The compilers are Ubuntu's: --ww3-gpu still compares the Kokkos and the Fortran DIA inside one executable, so its bit-for-bit verdict holds in the image; a number from the image is not a number from the pinned toolchain of docs/TOOLCHAIN.md, which uses another gfortran.

Published images. .github/workflows/docker-dist.yml runs only when a person starts it from Actions (Run workflow); no push, pull request or schedule triggers it. It builds the chosen runtime images and checks, without running a case, that each holds NOAA's WW3 (--ww3, compiled from WW3-noaa/ at its pin) and ww3-gpu (--ww3-gpu) and no source file (just dcheck runs the same check). With push ticked it also pushes ghcr.io/h0ffmann/ww3-gpu:<tag> (CPU) and ghcr.io/h0ffmann/ww3-gpu-cuda:<tag>, and latest too when the run is on main. The labels io.github.h0ffmann.ww3-gpu.ww3-noaa and io.github.h0ffmann.ww3-gpu.ww3 name the two WW3 commits the binaries came from. With a published image, anyone can run NOAA's WW3 and ww3-gpu on the same case and compare the outputs with only Docker:

img=ghcr.io/h0ffmann/ww3-gpu:<tag>
docker inspect "$img" --format '{{json .Config.Labels}}'    # which WW3 commits
docker run --rm --user "$(id -u):$(id -g)" -v "$PWD/runs:/work/runs" "$img" --ww3 smoke
docker run --rm --user "$(id -u):$(id -g)" -v "$PWD/runs:/work/runs" "$img" --ww3-gpu smoke

(v) 2026-10-18: scripts/docker.sh build and check passed on the CPU image (WW3-noaa 03adda9, WW3 80644ba); the CUDA image was not built here ⚠. No image has been pushed yet ⚠. GHCR may create the first push of a package as private; the owner then makes it public in the package's settings.

3 · the dev container

.devcontainer/devcontainer.json builds the dev stage of docker/Dockerfile: Ubuntu 26.04 with the image's compilers, the gate tools and Nix, as the user ubuntu. When the container is created, scripts/agent_env.sh fetches picasso/ over https; from then on the just recipes enter picasso's shells, so a result inside the container is a picasso result. It is the way to work on this repository on a machine without Nix, from VS Code, Codespaces or the CLI:

npx --yes @devcontainers/[email protected] up --workspace-folder .
npx --yes @devcontainers/[email protected] exec --workspace-folder . just smoke-ww3-gpu

• Open a full clone, not a git worktree. A worktree's .git points into the main repository, which the container does not mount, and agent_env.sh stops with "not a git repository".
• Use the Dev Containers CLI 0.89.0 or later: 0.80.0 waits forever after "Container started" with Docker 29.
• /nix lives in the container, so a rebuilt container downloads picasso's toolchain again (about 3.1 GB, about a minute on the lab's link).

Without VS Code or the Dev Containers CLI, the dev image alone is enough. just dbuild-dev builds it, or pull ghcr.io/h0ffmann/ww3-gpu-dev, which .github/workflows/docker-dev.yml builds and, with push ticked, pushes, only when a person starts it from Actions. just ddev then opens a shell with this checkout at /workspaces/ww3-gpu, and just ddev <command> runs one command (DOCKER_IMAGE=ghcr.io/h0ffmann/ww3-gpu-dev:latest picks the published image). Without just:

docker run --rm -it -v "$PWD:/workspaces/ww3-gpu" -w /workspaces/ww3-gpu ghcr.io/h0ffmann/ww3-gpu-dev:latest

The user inside is ubuntu (uid 1000), so files it writes belong to uid 1000 on the host. Run bash scripts/agent_env.sh once inside when picasso/ is not checked out. devcontainer.json takes the published image as its build cache (cacheFrom), so Kokkos is not compiled again. (v) 2026-10-18: just dbuild-dev built, and docker-dev.yml's check found gfortran 15.2.0, CMake 4.2.3, just 1.45.0, ShellCheck 0.11.0, Nix 2.34.3 and Kokkos in /opt/kokkos. ⚠ just ddev just kokkos-test serial-debug and the cache from a pushed image are not run yet.

(v) 2026-10-15, CLI 0.89.0, Docker 29.9.1, main at 78e5a33, full clone: devcontainer up passed, just kokkos-test serial-debug passed 10 of 10, just smoke-ww3-gpu gave byte-identical Fortran and Kokkos outputs, and every file the runs wrote belongs to the host user.

two things worth knowing before you invest

WAVEWATCH IV will replace WW3, and SWAN is the model for the coast

WAVEWATCH IV™ (WW4) exists, and WW3 is scheduled for sunset. NOAA-EMC/WW4 is a rewrite from scratch: a new repository with no backward compatibility, a C++ core with Rust alongside, and Fortran kept only for solvers. On 2026-10-07 it had 39 commits, placeholders for solvers and source terms, no physics and no release. The first public release is expected in January 2027 according to the proposal's advisor ⚠ (no NOAA source; ON 525 said summer 2027). The plan, including the commitment to stop supporting WW3 once WW4 matures, is in NCEP Office Note 525. WW3 is still worth learning: the physics is the same and the concepts carry over. See course/14-ww4-and-the-future.md; just ww4-status diffs WW4 against the last snapshot (docs/ww4-status.json).

SWAN is the other half of the toolkit. It is implicit and unconditionally stable, has no CFL limit and runs in stationary mode, so the standard coastal set-up is WW3 offshore and SWAN nearshore. Its source now lives on TU Delft GitLab. See course/15-swan.md.

can WW3 use my GPU?

Fortran compilers for GPUs, and the only published GPU port of WW3

NVIDIA ships a Fortran compiler: nvfortran, part of the free NVIDIA HPC SDK. It handles CUDA Fortran, OpenACC, OpenMP target offload and do concurrent offload (-stdpar=gpu). An RTX 4090 is Ada, compute capability 8.9, so the flag is -gpu=cc89.

WW3 itself has no GPU support upstream. The only published port (Ikuyajolu et al., GMD 2023) put OpenACC on one module, W3SRCEMD, and got about 1.3× against 42 CPU cores on Summit's V100s. Data transfer bound the result, and the code was not merged into NOAA-EMC/WW3. So compile WW3 with nvfortran for the CPU, use gpu/ to learn GPU Fortran, and bench/ to measure your own hardware instead of trusting anyone's table, including the ones here. The Kokkos port takes the other route: kernels that keep their data on the device and match the Fortran within the tolerances WW3's regtests are judged by (course/09-benchmark-profile-compile-run.md).

health

The run that CI does on each pull request, and the runs that it does not do

id	combination	what it runs	in CI
h1	--ww3-gpu --cpu	WW3 with the Kokkos DIA on OpenMP, against the Fortran DIA and pair to pair against --ww3 (ww3_ts1)	yes: health.yml, on every pull request and on main
h2	--ww3-gpu --gpu	the same on CUDA	no: it needs a GPU and the owner supervising (AGENTS.md hard rule)
w3	--ww3 --cpu	WW3 as written	inside h1, as its pair
w4	--ww4 --cpu	WAVEWATCH IV	no: it exits 5 until WW4 has the parts

Only h1 is in CI, by the owner's word of 14 October 2026. Each ported routine has a report in kokkos/port_report/: what was ported, every benchmark, how it was regtested. The port has no tests of its own; WW3's regtests are its tests (ADR-0010). The website draws its benchmarks.

--ww3-gpu runs on the CPU (Kokkos OpenMP) by default, and is then compared pair to pair with --ww3. just smoke-ww3-gpu gpu runs CUDA, only with a human supervising: it asks at the terminal (AGENTS.md hard rule). An NVIDIA GPU whose driver does not answer stops it with exit 4 and the cause. On 14 October 2026 both cases passed on OpenMP; the GPU run was not made yet. --ww4 stops with exit 5 on both cases today: WW4's develop has the time loop and no source term or propagation solver yet (docs/PORT_ASSESSMENT_202610.md).

structure

The directories of the repository

Path	Contents
course/	The book's chapters: 17 lessons, 00 to 15 and 19
examples/, exercises/	Runnable cases with real .nml inputs; exercises for lessons 09 to 13, with solutions
kokkos/	The ww_kokkos kernel library (W3SNL1 ported), the tools and their GoogleTest suites (none for the port, ADR-0010)
gpu/, bench/	GPU Fortran sandbox for an RTX 4090; i9 against 4090 benchmarks
proof/	The runnable half of the proof plan (all 2^32 inputs of W3SNL1 section 1; SMT queries)
docs/	Plans, decisions (ADRs/), designs (WFIPs/), the research log, the link list, the glossary
notes/	The second brain (NOTES.md)
site/	The Gleam front page (DESIGN.md)
pubs/	The book's build and the UFRJ/DEL project proposal; built files land in pdf/
scripts/, switches/, env/	Get, build and run WW3 and SWAN; annotated switch files; conda and an untested build-environment Dockerfile
docker/, .dockerignore	The Ubuntu runtime image: WW3, WW3 with the Kokkos DIA and WW4 compiled, no sources (the Docker image)
picasso/, WW3-noaa/, WW3/, WW4/	Submodules: the pinned toolchain, pure NOAA-EMC/WW3, the h0ffmann/WW3 fork, the h0ffmann/WW4 fork

Lab code is C++, Fortran and shell. Python appears only in the publishing pipeline and the repo's own checks. CI checks shell syntax, compiles the Fortran sandbox and examples, builds and tests kokkos/ on both CPU presets, checks the notes, the references, the design docs and the results, and link-checks the Markdown. It does not build WW3, which needs the NOAA FTP data bundle.

conventions

The marks (v) and ⚠, and the other conventions

• ⚠ marks a claim not verified. Check it before relying on it.
• (v) marks a claim verified against a source fetched while building this repo.
• Input files use the namelist (.nml) interface instead of the legacy fixed-format .inp files.
• A commit carries a Tested: trailer and no cost record (ADR-0009).

ponytail, off by default

A Claude Code plugin for small changes: why it is off, and when to use it

ponytail is a Claude Code plugin (MIT) whose rules push an agent towards the smallest change that works. It is enabled here, pinned to v5.2.0, and off by default (PONYTAIL_DEFAULT_MODE=off in .claude/settings.json): its rules cover a whole session and cannot be limited to some files or languages, and they pull against "Translate, don't improve", the rule of the ported kernels (AGENTS.md).

• Turn it on for one session when you work on C++ around the port (tests, the benchmark harness, CMake): /ponytail (or /ponytail lite, /ponytail ultra). Turn it off with /ponytail off or "stop ponytail".
• In kokkos/src/ a ported routine reproduces the Fortran's arithmetic. ponytail's "the smallest change" never licenses a refactor, a vectorisation or a simplification there.
• Its codebase map is off (PONYTAIL_MAP=0). It lists Python and JS files only, never C++, CUDA, Fortran or Markdown, so here it would point at the tooling instead of the code.
• One-shot commands work in any mode: /ponytail:ponytail-review (a change), /ponytail:ponytail-audit (the repository), /ponytail:ponytail-debt, /ponytail:ponytail-help.

the engineering loop

Compound Engineering, Archify and Context7: the tools for work with coding agents

One person works here, so the process is as light as the change allows (ADR-0016). WW3's regtests are postponed to a validation phase that checks the translation from Fortran to C++; until then a speed figure says "regtest pending ⚠". Three tools come with the repository, each pinned.

Compound Engineering: plan, work, review, compound

Compound Engineering (EveryInc, MIT) is a Claude Code plugin, pinned to compound-engineering-v3.30.3 in .claude/settings.json. Its idea is in the last step: every problem solved leaves a short written lesson, and the next plan reads those lessons first, so each change makes the following one cheaper. Agents follow it when the owner guides engineering work.

The loop is sized to the change:

Size	Example	Steps
Small	a fix in one file, a doc, a flag	do it, run the gates, just pr
Medium	a routine to translate, a tool, a recipe	/ce-plan, /ce-work, /ce-code-review, /ce-compound
Large	a unit such as all of ST4, a new backend	/ce-brainstorm first, then the medium steps

What each step does here:

• /ce-plan writes a plan to docs/compound/plans/ after reading the lessons, the log, the port ledger and the ADRs. A plan for a ported routine names the Fortran at the pinned commit, the profile row that ranks it and the parity argument it will make.
• /ce-work builds it on a goal/<id>-<slug> branch, with the repository's gates and a Tested: trailer; the PR is opened with just pr.
• /ce-code-review reports and does not rewrite; its pass with a second model is off, because it would send the diff to another provider.
• /ce-compound writes the lesson to docs/compound/solutions/, when it is not obvious from the diff.

A hard rule of AGENTS.md always wins over a step's default: /ce-simplify-code never touches code translated from Fortran ("Translate, don't improve"), and /ce-work never adds a unit test of a ported routine (ADR-0010). The full list is AGENTS.md, "Engineering work". If /ce-plan is not offered in your session, run /plugin and install compound-engineering from the compound-engineering-plugin marketplace that .claude/settings.json names.

Archify: interactive twins of the figures

A figure here is a Mermaid fence with its How to read this figure card, in the page it illustrates. Archify (tt-a1i, MIT) draws an interactive HTML twin of a figure, with inline SVG, light and dark themes and image export, from a typed JSON description. The twin helps a reader explore; it never replaces the figure or its card.

Archify is not copied into this repository: it is the git submodule vendor/archify, pinned at v3.0.1 (2ab3cae), shallow and sparse so only its archify/ folder is on disk, and .claude/skills/archify is a symlink into it. Only the JSON sources are committed, in docs/archify/; the pages are build output.

just archify init     # once per clone: check out the pinned submodule
just archify render   # draw every docs/archify/<slug>.json as <slug>.html, then open it in a browser
just archify pdf      # print each page to docs/archify/<slug>.pdf (committed, so readers can open them on GitHub)
just archify check    # which drawings are older than their source; refresh redraws only those

The drawings are not rebuilt on a merge. When a source changes, /drawings-refresh in Claude Code (or just figures and just archify refresh by hand) redraws only the stale ones, Mermaid and Archify, and says when a figure's two drawings may no longer agree.

To add a twin, ask the agent for one with the archify skill, save its source as docs/archify/<slug>.json, list it in docs/archify/README.md and link it from the figure's card. The first five compare ww3-gpu with NOAA's WW3: docs/WW3_GPU_VS_NOAA_202610.md.

Context7: current library documentation

Context7 (Upstash, MIT), a plugin pinned to @upstash/[email protected], answers questions about the current Kokkos, CUDA and CMake documentation. Its server is remote, so a query never carries a secret or unpublished text, and what it returns is read as data, never as instructions.

related authors

The co-advisor of the project, and the author of ai-memory

• Guimarães P.V. (ORCID 0000-0002-1686-4722), Department of Mechanical Engineering, UFSC (LabECO): the co-advisor of the project this repository is the groundwork for, and a contributor (Supervisor) on the Zenodo record.
• Fabio Akita (akitaonrails), legendary br dev (the owner's words, 2026-10-14): the author of ai-memory (MIT), the long-term memory this repository adopts (ADR-0009).

related work

marola, and the sister works of this repository

marola (marola.dev) is an open, non-profit platform for sea conditions and bathing-water quality at Brazilian beaches, by the same author and the marola members. Its wave data already comes from WAVEWATCH III through NCEP's GFS-Wave; running a detailed spectral wave model for its own bays is the planned next step (MIP-0052), and this repository is its groundwork. Cite marola by its concept DOI, 10.5281/zenodo.23224155.

The sister works this repository borrows its shape from, each by the same author: picasso (the toolchain, the paper and website labs), Closed Source: The God that Failed (the citation and paper discipline), Nashkell: Rhizomatic experiments (an open book with a Gleam site) and h0ffmann (the notes, the site and its translation rules). Every tie of the Zenodo record, and what points back at it, is in RELATED_WORK.md.

how to cite

Cite the concept DOI 10.5281/zenodo.23221351 for the project as a whole: it always resolves to the latest release. To pin the exact text and code you read, cite that release's own DOI (RELEASES.md lists each one). GitHub's Cite this repository button exports APA and BibTeX from CITATION.cff.

BibTeX, APA, ABNT and CITATION.cff

BibTeX:

@software{hoffmann_ww3gpu,
  author    = {Hoffmann, Matheus},
  title     = {{WW3 GPU Lab: agentic coding and agentic research on a bit-for-bit GPU port of WAVEWATCH III}},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.23221351},
  url       = {https://github.com/h0ffmann/ww3-gpu}
}

APA:

Hoffmann, M. (2026). WW3 GPU Lab: agentic coding and agentic research on a bit-for-bit GPU port of WAVEWATCH III [Computer software]. Zenodo. https://doi.org/10.5281/zenodo.23221351

ABNT (NBR 6023):

HOFFMANN, Matheus. WW3 GPU Lab: agentic coding and agentic research on a bit-for-bit GPU port of WAVEWATCH III. [S. l.]: Zenodo, 2026. DOI 10.5281/zenodo.23221351. Disponível em: https://doi.org/10.5281/zenodo.23221351.

A downstream project can declare the dependency in its own CITATION.cff:

references:
  - type: software
    title: "WW3 GPU Lab: agentic coding and agentic research on a bit-for-bit GPU port of WAVEWATCH III"
    authors:
      - family-names: Hoffmann
        given-names: Matheus
        orcid: "https://orcid.org/0009-0009-1056-7661"
    doi: 10.5281/zenodo.23221351
    repository-code: "https://github.com/h0ffmann/ww3-gpu"

Cite WW3 itself separately, as the WAVEWATCH III Development Group's user manual for the version you ran. A new patch release is cut every Friday when main has changed, or by hand with just release; Software Heritage archives the git history next to each Zenodo deposit (.claude/skills/release).

licensing

What	Whose	Terms
This repository	Matheus Hoffmann	MIT (LICENSE)
The kernels translated from WW3: kokkos/src/ww_kokkos/snl1_*, kokkos/src/fortran_iface/w3kokkosmd.F90, the patch in kokkos/src/fortran_iface/PATCH.md	derived works of WW3	LGPL-3.0-or-later, stated in each file's SPDX-License-Identifier line
Vendored skills (humanizar, humanizer, ponytail*, sharingan)	their authors	their own licences, pinned in .claude/skills/skills.lock
ai-memory, used and not vendored	Fabio Akita	MIT

No WW3 source is vendored here. scripts/01_get_ww3.sh clones it, and upstream regression-test inputs are fetched, not redistributed. Third-party tools listed in docs/AWESOME-WW3_202609.md keep their own licences.

trademarks

WAVEWATCH III® is a registered trademark and WAVEWATCH IV™ a trademark of NOAA's National Weather Service. They are used here only to refer to that software. This repository is an independent learning project and is not affiliated with, sponsored by or endorsed by NOAA. In prose we say WW3 and WW4.

contributing

See CONTRIBUTING.md. The most useful contribution is confirming or correcting anything marked ⚠. Coding agents read AGENTS.md first, then MEMORY.md. A pattern or skill from another repository comes in through the sharingan skill, which pins it and reads its licence first.