diff --git a/README.md b/README.md
index fedb551..d6a1386 100644
--- a/README.md
+++ b/README.md
@@ -7,7 +7,6 @@
-
@@ -29,7 +28,7 @@ Everything you need, including installation instructions, an interactive demo, m
## Demo
-Try the [Demo](https://birdnet-team.github.io/BirdBox/demo/) in your browser: it runs the Just-Bird model on your device, with no installation and no upload. Load a recording, tune the confidence and phrase gap, review and edit the boxes, and export them in the BirdBox output formats.
+Try the [Demo](https://birdnet-team.github.io/BirdBox/demo/) in your browser: it runs the Just-Bird model (or All-In-One, from the model menu) on your device, with no installation and no upload. Load a recording, tune the confidence and phrase gap, review and edit the boxes, and export them in the BirdBox output formats.
diff --git a/docs/demo.md b/docs/demo.md
index f9932e7..590862c 100644
--- a/docs/demo.md
+++ b/docs/demo.md
@@ -9,16 +9,19 @@ hide:
## About this demo
-The demo runs the self-contained Just-Bird graph `docs/models/Just-Bird_fp16.onnx` with [ONNX Runtime Web](https://onnxruntime.ai/docs/tutorials/web/){ target="_blank" rel="noopener noreferrer" } in your browser. The graph computes the PCEN spectrogram, runs the YOLO detector on 3 s windows every 1.5 s and applies NMS, just like [`detect_birds.py`](cli/detect-birds.md) with the Just-Bird model. There are no species labels: every box is a bird vocalization (`bird`).
+The demo runs a self-contained BirdBox graph with [ONNX Runtime Web](https://onnxruntime.ai/docs/tutorials/web/){ target="_blank" rel="noopener noreferrer" } in your browser. The graph computes the PCEN spectrogram, runs the YOLO detector on 3 s windows every 1.5 s and applies NMS, just like [`detect_birds.py`](cli/detect-birds.md) with the same model. The model menu at the top picks the graph:
-**Instant sliders.** The model runs once per recording, at the lowest confidence of the slider (0.05). The demo asks the graph for its per-window boxes (`raw_all`) and repeats the graph's window fusion in JavaScript: window ownership and weighted box fusion, joining calls into phrases, removing nested boxes. NMS only lets higher-scoring boxes suppress lower ones, so the boxes for any confidence and phrase gap are the same as a graph run with those values.
+- **Just-Bird** (`docs/models/Just-Bird_fp16.onnx`, the default): every box is a bird vocalization (`bird`), without species labels. For species, combine the boxes with [BirdNET](https://github.com/birdnet-team/BirdNET-Analyzer){ target="_blank" rel="noopener noreferrer" }.
+- **All-In-One** (`docs/models/All-In-One_fp16.onnx`): the [All-in-One](models-and-metrics/all-in-one.md) model with its 282 classes. The table names the species of each box.
+
+**Instant sliders.** The model runs once per recording, at the lowest confidence of the slider (0.05). The demo asks the graph for its per-window boxes (`raw_all`) and repeats the graph's merge in JavaScript. For Just-Bird that is window fusion: window ownership and weighted box fusion, joining calls into phrases, removing nested boxes. For All-In-One it is song reconstruction: boxes of one species closer than the song gap become one box. NMS only lets higher-scoring boxes suppress lower ones, so the boxes for any confidence and gap are the same as a graph run with those values.
**Spectrogram.** The viewer shows an STFT spectrogram by default. In its display settings, *PCEN* switches to the PCEN mel spectrogram BirdBox detects on.
**Review.** Boxes can be verified, moved, resized, removed and drawn. The exports use the BirdBox [output formats](data/outputs.md): the JSON of `--output-format json-with-algorithm-metadata`, the simplified CSV and the Raven selection table, plus a CSV with every column of the table.
!!! warning "Browser limits"
- - The model is downloaded once (about 35 MB) and then comes from the browser cache.
+ - The model is downloaded once (about 20 MB for Just-Bird, 51 MB for All-In-One) and then comes from the browser cache.
- Audio is decoded locally, resampled to 32 kHz mono and truncated to **10 minutes**.
- The browser's resampler is not librosa's, so files that are not recorded at 32 kHz can give slightly different boxes than the CLI.
- ONNX Runtime runs single-threaded here: about 10 s for 2 minutes of audio on a laptop. Safari can be slower than Chromium or Firefox.
@@ -28,7 +31,7 @@ The sample soundscape is the example recording of [BirdNET-Analyzer](https://git
## Adding models
-The demo runs the first model in `docs/models/manifest.json` with `"merge_method": "fusion"`. Only window-fusion exports work: the instant sliders need the graph's per-window boxes (`raw_all`) and the JavaScript port of its window fusion (`docs/javascripts/demo/fusion.js`). The export uses window fusion for Just-Bird, Hawaii and Southern-Sierra-Nevada (`--merge auto`); `--merge fusion` forces it for the other models.
+The model menu lists the models in `docs/models/manifest.json`; the first one is the default. The instant sliders need the graph's per-window boxes (`raw_all`) and a JavaScript port of its merge: `docs/javascripts/demo/fusion.js` for window fusion (`fusion`, the export's default for Just-Bird, Hawaii and Southern-Sierra-Nevada) and `docs/javascripts/demo/songs.js` for song reconstruction (`reconstruct`, the default for the other models). The Just-Bird frequency gate (`--merge reconstruct` with Just-Bird) has no port; such entries are skipped.
1. Export the model.
@@ -53,11 +56,13 @@ The demo runs the first model in `docs/models/manifest.json` with `"merge_method
--precision fp16
```
- `--precision fp16` stores the network weights in fp16, which makes the file about a third smaller (35 MB instead of 53 MB for Just-Bird). Feature extraction and merging stay in fp32. On 13 test recordings, 99.4% of the fp32 graph's boxes were reproduced (mean IoU 0.996). The export prints the merge it baked in (`Merge (baked in)`) and the suggested defaults (`Default conf`, `Default song_gap`), which it also stores in the ONNX metadata.
+ For another model, add `--pt-model`, for example `--pt-model models/All-In-One.pt --output-path docs/models/All-In-One_fp16.onnx`.
+
+ `--precision fp16` stores the network weights in fp16, which halves the file (20 MB instead of 38 MB for Just-Bird, 51 MB instead of 99 MB for All-In-One). Feature extraction and merging stay in fp32. On 13 test recordings (`tests/onnx_graph_parity.py`), the fp16 graphs reproduced the boxes of `detect_birds.py` with the `.pt` model: 99.8% for Just-Bird (mean IoU 0.995) and 99.7% for All-In-One, with the same species (mean IoU 0.997). The export prints the merge it baked in (`Merge (baked in)`) and the suggested defaults (`Default conf`, `Default song_gap`), which it also stores in the ONNX metadata.
2. Place the `.onnx` file under `docs/models/` if the export path was elsewhere.
-3. Add an entry to `docs/models/manifest.json`, first in the list if the demo should run it.
+3. Add an entry to `docs/models/manifest.json`, first in the list if the demo should start with it.
```json
{
@@ -70,6 +75,16 @@ The demo runs the first model in `docs/models/manifest.json` with `"merge_method
"phrase_gap": 0.05,
"species_mapping": "Just-Bird",
"names": { "0": "bird" }
+ },
+ {
+ "file": "All-In-One_fp16.onnx",
+ "label": "All-In-One_fp16.onnx",
+ "merge_method": "reconstruct",
+ "conf": 0.2,
+ "song_gap": 0.1,
+ "species_mapping": "All-In-One",
+ "names": { "0": "????", "1": "amabaw1", "…": "…" },
+ "common_names": { "0": "Unknown", "1": "Amazonian Barred-Woodcreeper", "…": "…" }
}
]
}
@@ -78,19 +93,23 @@ The demo runs the first model in `docs/models/manifest.json` with `"merge_method
| Field | Meaning |
| :--- | :--- |
| `file` | The ONNX file, relative to `docs/models/` |
- | `label` | Name shown at the top of the demo |
- | `merge_method` | `fusion`, as printed by the export. Other entries are skipped. |
+ | `label` | Name in the model menu |
+ | `merge_method` | `fusion` or `reconstruct`, as printed by the export. Other entries are skipped. |
| `conf` | Where the confidence slider starts: the printed `Default conf` (0.15 for Just-Bird, 0.2 for the other models) |
- | `phrase_gap` | Where the phrase gap slider starts: the printed `Default song_gap` (0.05 for Just-Bird, 0 for Hawaii and Southern-Sierra-Nevada) |
+ | `phrase_gap` | `fusion` only: where the phrase gap slider starts, the printed `Default song_gap` (0.05 for Just-Bird, 0 for Hawaii and Southern-Sierra-Nevada) |
+ | `song_gap` | `reconstruct` only: where the song gap slider starts, the printed `Default song_gap` (0.1) |
| `species_mapping` | Written to `model_config` of the exported JSON |
- | `names` | Class labels for the exports, by class index (ONNX Runtime Web does not expose ONNX metadata) |
+ | `names` | eBird codes by class index, for the table and the exports (ONNX Runtime Web does not expose ONNX metadata) |
+ | `common_names` | Optional: common names by class index, shown in the table instead of the codes |
+
+ For a species model, `names` and `common_names` come from `id_to_ebird` and `ebird_to_name` of its mapping in `src/config.py`. With more than one class, the table gets a species column.
-4. Check that the demo's fusion still matches the graph:
+4. Check that the demo's merge still matches the graph:
```bash
cd tests/js && npm install
- node demo_fusion_parity.mjs ../../docs/models/Just-Bird_fp16.onnx
+ node demo_merge_parity.mjs ../../docs/models/Just-Bird_fp16.onnx
```
- The script compares `fusion.js` with the graph's own detections for 16 slider settings on `tests/test.wav` and fails on any difference. Run it after exporting a new model or changing the fusion in `src/inference/onnx_export.py`.
+ The script compares `fusion.js` or `songs.js`, whichever the manifest entry's `merge_method` names, with the graph's own detections for 16 slider settings on `tests/test.wav` and fails on any difference. Run it after exporting a new model or changing a merge in `src/inference/onnx_export.py`.
diff --git a/docs/index.md b/docs/index.md
index f965498..fcaa3b3 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -6,8 +6,7 @@
Deep Learning Bird Call Detection & Evaluation System
-
-
+
@@ -58,7 +57,7 @@ The inference software found 4 different species vocalizations and was able to l
## Quick Links
- [Installation](installation.md) - set up the environment
-- [Demo](demo.md) - Just-Bird in your browser
+- [Demo](demo.md) - BirdBox in your browser
- [How it works](how-it-works/overview.md) - pipeline description
- [Data In/Out](data/fileflows.md) - datasets and output-format descriptions
- [Models and Metrics](models-and-metrics/overview.md) - list of models with corresponding metrics
diff --git a/docs/javascripts/demo/demo.js b/docs/javascripts/demo/demo.js
index 0527cb3..0fdd470 100644
--- a/docs/javascripts/demo/demo.js
+++ b/docs/javascripts/demo/demo.js
@@ -2,23 +2,25 @@
// SPDX-License-Identifier: MIT
/**
* BirdBox browser demo (the Demo tab of the docs): bird vocalization boxes
- * with the self-contained Just-Bird ONNX graph (PCEN, YOLO, NMS inside).
+ * with a self-contained BirdBox ONNX graph (PCEN, YOLO, NMS inside), Just-Bird
+ * by default; the models are listed in docs/models/manifest.json.
*
* The graph runs once per recording at the lowest confidence and returns its
- * per-window boxes. fusion.js merges them exactly like the graph's window
- * fusion for any (confidence, phrase gap) setting, so both sliders update
- * instantly.
+ * per-window boxes. fusion.js (window fusion, Just-Bird) or songs.js (song
+ * reconstruction, the species models) merges them exactly like the graph for
+ * any (confidence, gap) setting, so both sliders update instantly.
*
* On the spectrogram, the selected box can be moved and resized and new
* boxes can be drawn. Exports use the BirdBox output formats.
*/
import { AudioSource, MAX_SECONDS } from './audio-source.js';
import { SAMPLE_RATE as SR } from './audio.js';
-import { detector, modelEntry, RAW_CONF } from './detector.js';
+import { detector, models, RAW_CONF } from './detector.js';
import { fuseBoxes } from './fusion.js';
import { player } from './player.js';
+import { reconstructSongs } from './songs.js';
import { SpecView } from './spec-view.js';
-import { csvField, downloadFile, fill, fmtBytes, fmtNumber, fmtTime, icon, statLine, toast } from './util.js';
+import { csvField, downloadFile, esc, fill, fmtBytes, fmtNumber, fmtTime, icon, statLine, toast } from './util.js';
const MS_PER_CLIP_KEY = 'birdbox-demo-ms-per-clip';
const CLIP = 3, HOP = 1.5; // the graph's window grid (onnx_export.py)
@@ -26,19 +28,25 @@ const NMS_IOU = 0.7; // baked into the graph (DEFAULT_NMS_IOU)
const FREQ_MAX = 15000; // top of the spectrogram and of any box
const MIN_DUR = 0.05; // smallest box: seconds
const MIN_BAND = 100; // smallest box: Hz
+const MERGE = { fusion: fuseBoxes, reconstruct: reconstructSongs };
+const GAP_TEXT = {
+ fusion: ['Phrase gap', 'Calls in the same frequency band closer than this are joined into one phrase box. 0 keeps single calls.'],
+ reconstruct: ['Song gap', 'Boxes of the same species closer than this are merged into one song box.'],
+};
const COLORS = { detected: '#22d3ee', verified: '#4ade80', drawn: '#fbbf24', removed: 'rgba(148,163,184,0.7)' };
const $ = id => document.getElementById(id);
const clamp = (v, lo, hi) => Math.max(lo, Math.min(hi, v));
-/** Detector boxes keep their identity through slider changes by their geometry. */
-const boxKey = b => `${b.t0.toFixed(4)}|${b.t1.toFixed(4)}|${b.flo}|${b.fhi}`;
+/** Detector boxes keep their identity through slider changes by their geometry and class. */
+const boxKey = b => `${b.t0.toFixed(4)}|${b.t1.toFixed(4)}|${b.flo}|${b.fhi}|${b.cls}`;
const geomOf = g => ({ t0: g.t0, t1: g.t1, flo: g.flo, fhi: g.fhi });
const round = (v, digits) => Number(v.toFixed(digits));
class Demo {
constructor() {
this.audio = null;
+ this.model = null; // manifest entry of the selected model
this.raw = null; // { data, rows, ms }: raw_all of the current recording
this.songs = []; // detector boxes at the current sliders
this.userBoxes = []; // boxes drawn by the user
@@ -87,18 +95,51 @@ class Demo {
this._showHelp(false); // closed until the ? button opens it
$('bbd-conf').min = RAW_CONF;
this._bind();
- modelEntry().then(m => {
- this.model = m;
- // Where the sliders start: the model's suggested values from the manifest
- if (m.conf != null) $('bbd-conf').value = m.conf;
- if (m.phrase_gap != null) $('bbd-gap').value = m.phrase_gap;
- this._syncSliders();
- this._update();
- $('bbd-model-name').textContent = m.label || m.file;
+ this.modelsReady = models().then(list => {
+ this.models = list;
+ $('bbd-model').replaceChildren(...list.map((m, i) => new Option(m.label || m.file, String(i))));
+ $('bbd-model').disabled = list.length < 2;
+ this._setModel(list[0]);
}).catch(err => this._error(err.message));
this._refresh();
}
+ /** Use a manifest model: sliders to its suggested values, gap slider and table to its merge and classes. */
+ _setModel(m) {
+ this.model = m;
+ $('bbd-model').value = String(this.models.indexOf(m));
+ const fusion = m.merge_method === 'fusion';
+ if (m.conf != null) $('bbd-conf').value = m.conf;
+ const gap = fusion ? m.phrase_gap : m.song_gap;
+ if (gap != null) $('bbd-gap').value = gap;
+ const [label, help] = GAP_TEXT[m.merge_method];
+ $('bbd-gap-label').textContent = label;
+ $('bbd-gap-help').textContent = help;
+ $('bbd-table').querySelector('.bbd-c-species').hidden = !this._multiClass();
+ this._syncSliders();
+ this._renderModel();
+ this._update();
+ }
+
+ /** Switch the model: boxes of the old one and changes to them go, drawn boxes stay. */
+ _changeModel(m) {
+ if (m === this.model) return;
+ this.runToken++;
+ detector.abort();
+ this._setProgress(null);
+ const edited = [...this.edits.values()].some(e => Object.keys(e).length);
+ this.raw = null;
+ this.edits.clear();
+ this._setModel(m);
+ if (edited) toast('Your changes to detected boxes were reset for the new model.');
+ this.run();
+ }
+
+ /** Several classes (species models): the table and the tooltip name them. */
+ _multiClass() {
+ return Object.keys(this.model?.names || {}).length > 1;
+ }
+
/** The spectrogram takes what the page leaves it, within limits. */
_specHeight() {
return Math.round(clamp(window.innerHeight * 0.42, 240, 560));
@@ -106,6 +147,7 @@ class Demo {
_bind() {
this.source.addEventListener('audio', e => this._onAudio(e.detail));
+ $('bbd-model').addEventListener('change', e => { const m = this.models?.[+e.target.value]; if (m) this._changeModel(m); });
for (const id of ['bbd-conf', 'bbd-gap']) $(id).addEventListener('input', () => { this._syncSliders(); this._update(); });
this._syncSliders();
$('bbd-export').addEventListener('click', e => {
@@ -147,7 +189,8 @@ class Demo {
const g = this._geom(b), n = this.boxes.indexOf(b) + 1, edit = this._edit(b);
const origin = b.manual ? 'Drawn by you'
: `Confidence ${fmtNumber(b.avg, 2)} · max ${fmtNumber(b.max, 2)} · ${b.n} window boxes${edit.verified ? ' · verified' : ''}`;
- this.spec.showTooltip(`
BirdBox finds bird vocalizations as time-frequency boxes. The model runs on your device; audio is never uploaded.
{{ mdi("chip") }}
- Just-Bird
+
@@ -87,9 +89,9 @@
Settings
Boxes must score above this. The slider starts at the model's suggested value.
-
+
-
Calls in the same frequency band closer than this are joined into one phrase box. 0 keeps single calls.
+
Calls in the same frequency band closer than this are joined into one phrase box. 0 keeps single calls.
The model runs once per recording, so both sliders update the boxes instantly.
@@ -152,6 +154,7 @@
Results
#
Time
Frequency
+
Species
Confidence
Verify or remove
diff --git a/src/inference/onnx_export.py b/src/inference/onnx_export.py
index 3b2a79c..74101f4 100644
--- a/src/inference/onnx_export.py
+++ b/src/inference/onnx_export.py
@@ -50,7 +50,9 @@
the graph) and torch (graph operations):
* flattop analysis window (``scipy.signal.get_window``)
-* STFT, as a strided convolution with a DFT basis (``librosa.stft``)
+* STFT, as a strided convolution with a DFT basis (``librosa.stft``). The file
+ stores one period of the basis's phases; the graph rebuilds the full basis
+ (15.7 MB of float32) when the model loads.
* HTK mel filterbank (``librosa.filters.mel``)
* PCEN, as a parallel-prefix EMA instead of ``scipy.signal.lfilter``
* colormap lookup and figure rasterization (``librosa.display.specshow``)
@@ -202,6 +204,9 @@
RAW_COL_FILE_END = 8 # end of the file's last clip (window fusion)
RAW_COLUMNS = 9
+# BirdDetectionGraph.dft_kernel inside the composed loop body ("b_" prefix)
+DFT_KERNEL_NAME = "b_core.dft_kernel"
+
def _pad_to_clip_length(samples: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]:
"""Right-pad with silence until the last clip window is complete.
@@ -357,6 +362,60 @@ def dft_convolution_kernel(window: np.ndarray, n_bins: int) -> np.ndarray:
return kernel[:, np.newaxis, :]
+def dft_kernel_builder(window: np.ndarray, n_bins: int, output_name: str, prefix: str = "dft_"):
+ """
+ ONNX nodes that rebuild ``dft_convolution_kernel(window, n_bins)`` when the
+ model loads, so the file stores one period of phases instead of the whole
+ kernel (15.7 MB of float32 for the default settings).
+
+ The phase of bin ``k`` at sample ``n`` only depends on ``(k * n) % n_fft``,
+ so the kernel is a lookup into one period of cosine and sine values times
+ the window. Lookup and products run in float64 and are cast to float32 at
+ the end, as in ``dft_convolution_kernel``, so the rebuilt kernel is
+ bit-identical to the stored one. The numpy check below makes sure of that.
+
+ Returns:
+ ``(nodes, initializers)``. The nodes produce ``output_name`` with shape
+ ``(2 * n_bins, 1, len(window))``.
+ """
+ from onnx import TensorProto, helper, numpy_helper
+
+ n_fft = len(window)
+ # The expression of dft_convolution_kernel, for one period of indices
+ phase = 2.0 * np.pi * np.arange(n_fft, dtype=np.int64) / n_fft
+ cos_table, neg_sin_table = np.cos(phase), -np.sin(phase)
+ bins = np.arange(n_bins, dtype=np.int64)[:, np.newaxis]
+ samples = np.arange(n_fft, dtype=np.int64)
+ index = (bins * samples) % n_fft
+ rebuilt = np.concatenate([cos_table[index] * window, neg_sin_table[index] * window])
+ expected = dft_convolution_kernel(window, n_bins).astype(np.float32)
+ if not np.array_equal(rebuilt.astype(np.float32)[:, np.newaxis, :], expected):
+ raise RuntimeError("The DFT kernel lookup does not reproduce dft_convolution_kernel")
+
+ p = prefix
+ initializers = [
+ numpy_helper.from_array(bins, p + "bins"),
+ numpy_helper.from_array(samples, p + "samples"),
+ numpy_helper.from_array(np.array(n_fft, dtype=np.int64), p + "n_fft"),
+ numpy_helper.from_array(cos_table, p + "cos"),
+ numpy_helper.from_array(neg_sin_table, p + "neg_sin"),
+ numpy_helper.from_array(np.asarray(window, dtype=np.float64), p + "window"),
+ numpy_helper.from_array(np.array([1], dtype=np.int64), p + "channel_axis"),
+ ]
+ nodes = [
+ helper.make_node("Mul", [p + "bins", p + "samples"], [p + "product"]),
+ helper.make_node("Mod", [p + "product", p + "n_fft"], [p + "index"]),
+ helper.make_node("Gather", [p + "cos", p + "index"], [p + "cos_at"], axis=0),
+ helper.make_node("Gather", [p + "neg_sin", p + "index"], [p + "neg_sin_at"], axis=0),
+ helper.make_node("Mul", [p + "cos_at", p + "window"], [p + "real"]),
+ helper.make_node("Mul", [p + "neg_sin_at", p + "window"], [p + "imaginary"]),
+ helper.make_node("Concat", [p + "real", p + "imaginary"], [p + "kernel64"], axis=0),
+ helper.make_node("Cast", [p + "kernel64"], [p + "kernel32"], to=TensorProto.FLOAT),
+ helper.make_node("Unsqueeze", [p + "kernel32", p + "channel_axis"], [output_name]),
+ ]
+ return nodes, initializers
+
+
def pcen_smoothing_coefficient(
sr: int = SAMPLE_RATE,
hop_length: int = HOP_LENGTH,
@@ -1705,16 +1764,25 @@ def replace_name(graph, old: str, new: str) -> None:
),
helper.make_tensor_value_info("conf_out", TensorProto.FLOAT, [1]),
],
- list(body_graph.initializer),
+ [t for t in body_graph.initializer if t.name != DFT_KERNEL_NAME],
)
+ # The DFT kernel is the largest constant of the graph. The parent graph
+ # rebuilds it from one period of phases (dft_kernel_builder); the loop body
+ # reads it from there as an outer-scope value.
+ kernel_shape = next(t.dims for t in body_graph.initializer if t.name == DFT_KERNEL_NAME)
+ window = flattop_window(WIN_LENGTH)
+ if kernel_shape[2] != len(window):
+ raise RuntimeError(f"Unexpected DFT kernel shape {list(kernel_shape)}")
+ dft_nodes, dft_initializers = dft_kernel_builder(window, kernel_shape[0] // 2, DFT_KERNEL_NAME)
+
recon_graph = copy.deepcopy(recon_model.graph)
rename_graph_values(
recon_graph, "r_", {"song_gap": "song_gap", "detections": "detections"}
)
replace_name(recon_graph, "r_raw", "raw_all")
- initializers = list(recon_graph.initializer) + [
+ initializers = list(recon_graph.initializer) + dft_initializers + [
numpy_helper.from_array(np.array(True), "true_const"),
numpy_helper.from_array(
np.zeros((0, RAW_COLUMNS), dtype=np.float32), "acc_empty"
@@ -1727,6 +1795,7 @@ def replace_name(graph, old: str, new: str) -> None:
]
parent_nodes = [
+ *dft_nodes,
helper.make_node("Shape", ["audio"], ["audio_shape"]),
helper.make_node("Gather", ["audio_shape", "zero_i"], ["num_samples"], axis=0),
helper.make_node("Add", ["num_samples", "seg_minus_1"], ["num_plus"]),
diff --git a/tests/js/demo_fusion_parity.mjs b/tests/js/demo_merge_parity.mjs
similarity index 66%
rename from tests/js/demo_fusion_parity.mjs
rename to tests/js/demo_merge_parity.mjs
index 4625fdb..614de66 100644
--- a/tests/js/demo_fusion_parity.mjs
+++ b/tests/js/demo_merge_parity.mjs
@@ -1,23 +1,26 @@
// SPDX-FileCopyrightText: 2026 BirdNET-Team
// SPDX-License-Identifier: MIT
/**
- * Parity of the Demo tab's window fusion (docs/javascripts/demo/fusion.js)
- * with the fusion inside the ONNX graph.
+ * Parity of the Demo tab's merges with the merge inside the ONNX graph:
+ * window fusion (docs/javascripts/demo/fusion.js) and song reconstruction
+ * (docs/javascripts/demo/songs.js).
*
- * The demo runs docs/models/Just-Bird_fp16.onnx once per recording, asks for
- * the per-window boxes (raw_all) and fuses them in JavaScript, so the sliders
- * need no new run. This script runs the same ONNX Runtime Web bundle as the
- * demo (WASM, single thread), once for raw_all and once per (conf, phrase gap)
+ * The demo runs a model of docs/models/manifest.json once per recording, asks
+ * for the per-window boxes (raw_all) and merges them in JavaScript, so the
+ * sliders need no new run. This script runs the same ONNX Runtime Web bundle
+ * as the demo (WASM, single thread), once for raw_all and once per (conf, gap)
* setting for the graph's own detections, and compares every box.
*
- * Rerun it after changing the fusion in src/inference/onnx_export.py (or its
+ * Rerun it after changing the merges in src/inference/onnx_export.py (or the
* FUSION_* / PHRASE_* constants), or after exporting a new model.
*
* Usage (Node 18 or newer):
* cd tests/js && npm install
- * node demo_fusion_parity.mjs [model.onnx] [audio.wav ...]
+ * node demo_merge_parity.mjs [model.onnx] [--merge=fusion|reconstruct] [audio.wav ...]
*
- * Model: default docs/models/Just-Bird_fp16.onnx, the model of the demo.
+ * Model: default docs/models/Just-Bird_fp16.onnx, the default model of the
+ * demo. The merge comes from the model's manifest entry (by file name), or
+ * from --merge for models not in the manifest.
*
* Audio: 16-bit PCM or float WAV at 32 kHz; default tests/test.wav (5 minutes
* of audio, about 8 minutes to run on a laptop).
@@ -27,11 +30,18 @@ import { readFileSync } from 'node:fs';
const root = new URL('../../', import.meta.url);
const ort = await import(new URL('./node_modules/onnxruntime-web/dist/ort.wasm.min.mjs', import.meta.url));
const { fuseBoxes } = await import(new URL('docs/javascripts/demo/fusion.js', root));
+const { reconstructSongs } = await import(new URL('docs/javascripts/demo/songs.js', root));
const { addGraphOutput, FLOAT } = await import(new URL('docs/javascripts/demo/onnx-graph.js', root));
const args = process.argv.slice(2);
-const MODEL = args.find(a => a.endsWith('.onnx')) || new URL('docs/models/Just-Bird_fp16.onnx', root);
-const SETTINGS = [0.05, 0.15, 0.3, 0.5].flatMap(conf => [0, 0.05, 0.3, 1].map(gap => [conf, gap]));
+const MODEL = args.find(a => a.endsWith('.onnx')) || new URL('docs/models/Just-Bird_fp16.onnx', root).pathname;
+const manifest = JSON.parse(readFileSync(new URL('docs/models/manifest.json', root)));
+const METHOD = args.find(a => a.startsWith('--merge='))?.slice(8)
+ || manifest.models.find(m => m.file === MODEL.split('/').pop())?.merge_method;
+const MERGE = { fusion: fuseBoxes, reconstruct: reconstructSongs }[METHOD];
+if (!MERGE) throw new Error(`${MODEL}: not in docs/models/manifest.json, pass --merge=fusion or --merge=reconstruct`);
+const GAPS = { fusion: [0, 0.05, 0.3, 1], reconstruct: [0, 0.1, 0.3, 1] }[METHOD];
+const SETTINGS = [0.05, 0.15, 0.3, 0.5].flatMap(conf => GAPS.map(gap => [conf, gap]));
const COLS = ['t0', 't1', 'flo', 'fhi', 'avg', 'max', 'n', 'cls'];
const TOLERANCE = 1e-5; // relative; the port is meant to be exact
@@ -67,7 +77,7 @@ ort.env.logLevel = 'error';
const model = addGraphOutput(new Uint8Array(readFileSync(MODEL)), 'raw_all', FLOAT, ['num_raw', 9]);
const session = await ort.InferenceSession.create(model, { executionProviders: ['wasm'], graphOptimizationLevel: 'all' });
-const files = args.filter(a => !a.endsWith('.onnx'));
+const files = args.filter(a => !a.endsWith('.onnx') && !a.startsWith('--'));
if (!files.length) files.push(new URL('tests/test.wav', root).pathname);
let failed = 0, total = 0;
for (const file of files) {
@@ -79,10 +89,10 @@ for (const file of files) {
});
// As in the demo: raw_all at the lowest slider confidence
const raw = (await session.run(feeds(0.05, 0), ['raw_all'])).raw_all;
- console.log(`${file.split('/').pop()}: ${(audio.length / 32000).toFixed(1)} s, ${raw.dims[0]} window boxes`);
+ console.log(`${file.split('/').pop()}: ${(audio.length / 32000).toFixed(1)} s, ${raw.dims[0]} window boxes, ${METHOD}`);
for (const [conf, gap] of SETTINGS) {
const ref = (await session.run(feeds(conf, gap), ['detections'])).detections;
- const js = fuseBoxes(raw.data, raw.dims[0], conf, gap);
+ const js = MERGE(raw.data, raw.dims[0], conf, gap);
let dev = js.length === ref.dims[0] ? 0 : Infinity;
if (dev === 0) js.forEach((b, i) => COLS.forEach((k, j) => {
const r = ref.data[i * 8 + j];
@@ -91,7 +101,7 @@ for (const file of files) {
total++;
const ok = dev <= TOLERANCE;
if (!ok) failed++;
- console.log(` ${ok ? 'PASS' : 'FAIL'} conf ${conf.toFixed(2)} gap ${gap.toFixed(2)}: graph ${ref.dims[0]} boxes, fusion.js ${js.length}` + (Number.isFinite(dev) ? `, max rel deviation ${dev.toExponential(1)}` : ''));
+ console.log(` ${ok ? 'PASS' : 'FAIL'} conf ${conf.toFixed(2)} gap ${gap.toFixed(2)}: graph ${ref.dims[0]} boxes, JS ${js.length}` + (Number.isFinite(dev) ? `, max rel deviation ${dev.toExponential(1)}` : ''));
}
}
console.log(`${total - failed}/${total} settings identical`);
diff --git a/tests/js/package.json b/tests/js/package.json
index 2b9c1dd..9b3d2cb 100644
--- a/tests/js/package.json
+++ b/tests/js/package.json
@@ -5,7 +5,7 @@
"license": "MIT",
"type": "module",
"scripts": {
- "test": "node demo_fusion_parity.mjs"
+ "test": "node demo_merge_parity.mjs"
},
"devDependencies": {
"onnxruntime-web": "1.20.1"
- 
@@ -29,7 +28,7 @@ Everything you need, including installation instructions, an interactive demo, m
## Demo
-Try the [Demo](https://birdnet-team.github.io/BirdBox/demo/) in your browser: it runs the Just-Bird model on your device, with no installation and no upload. Load a recording, tune the confidence and phrase gap, review and edit the boxes, and export them in the BirdBox output formats.
+Try the [Demo](https://birdnet-team.github.io/BirdBox/demo/) in your browser: it runs the Just-Bird model (or All-In-One, from the model menu) on your device, with no installation and no upload. Load a recording, tune the confidence and phrase gap, review and edit the boxes, and export them in the BirdBox output formats.
-