diff --git a/README.md b/README.md index 1bfee906..ee611b55 100644 --- a/README.md +++ b/README.md @@ -18,7 +18,7 @@ Visit the [Quarto site](https://quarto.org/docs/guide/) for documentation on usi The `tutorials/` directory contains interactive data analysis tutorials: - **`isamples_explorer.qmd`** - Interactive search and exploration of 6.7M samples -- **`zenodo_isamples_analysis.qmd`** - Deep-dive DuckDB-WASM analysis tutorial +- **`explorer_guided_tour.qmd`** - Guided tour of the Interactive Explorer (replaced the migration-era Zenodo deep-dive, 2026-07) - **`parquet_cesium_isamples_wide.qmd`** - Cesium-based 3D globe visualization - **`narrow_vs_wide_performance.qmd`** - Technical schema comparison diff --git a/SERIALIZATIONS.md b/SERIALIZATIONS.md index fd2906d5..a5e0a74f 100644 --- a/SERIALIZATIONS.md +++ b/SERIALIZATIONS.md @@ -104,7 +104,7 @@ builder — a fresh build is NOT bit-for-bit identical to them (see | File | Role | Size | Rows | Upstream | Consumers | Spec | |---|---|---:|---:|---|---|---| -| `isamples_202601_wide_h3.parquet` | Wide with `h3_res{4,6,8}` BIGINT columns pre-joined | 292 MB | 20.7 M | wide | Deep-Dive Analysis tutorial (H3 filtering without join) | QUERY_SPEC §2.4 | +| `isamples_202601_wide_h3.parquet` | Wide with `h3_res{4,6,8}` BIGINT columns pre-joined | 292 MB | 20.7 M | wide | archival/analyst convenience (former Deep-Dive tutorial, replaced 2026-07 by the Guided Tour which uses summary files) | QUERY_SPEC §2.4 | | `isamples_202601_h3_summary_res4.parquet` | Continental tier: `(h3_cell, sample_count, center_lat, center_lng, dominant_source, source_count, resolution)` | 580 KB | 38 K | wide_h3 | Interactive Explorer globe (zoomed out), Python Explorer H3 tier mode | QUERY_SPEC §2.4 | | `isamples_202601_h3_summary_res6.parquet` | Regional tier | 1.6 MB | 112 K | wide_h3 | Interactive Explorer globe (mid zoom) | QUERY_SPEC §2.4 | | `isamples_202601_h3_summary_res8.parquet` | Neighborhood tier | 2.4 MB | 176 K | wide_h3 | Interactive Explorer globe (close zoom) | QUERY_SPEC §2.4 | diff --git a/_quarto.yml b/_quarto.yml index adfe6c96..694e6b5c 100644 --- a/_quarto.yml +++ b/_quarto.yml @@ -37,8 +37,8 @@ website: menu: - text: Overview href: how-to-use.qmd - - text: Deep-Dive Analysis - href: tutorials/zenodo_isamples_analysis.qmd + - text: Guided Tour + href: tutorials/explorer_guided_tour.qmd - text: About menu: - text: Objectives @@ -84,8 +84,8 @@ website: contents: - href: how-to-use.qmd text: Overview - - text: "Deep-Dive Analysis" - href: tutorials/zenodo_isamples_analysis.qmd + - text: "Guided Tour" + href: tutorials/explorer_guided_tour.qmd - id: about title: "About" style: "docked" diff --git a/explorer.qmd b/explorer.qmd index 9050b799..7786de94 100644 --- a/explorer.qmd +++ b/explorer.qmd @@ -7391,5 +7391,5 @@ Pre-aggregated H3 hexagonal indices achieve near-instant globe rendering, with s ## See Also -- [Deep-Dive Analysis](/tutorials/zenodo_isamples_analysis.html) — DuckDB-WASM SQL tutorial +- [Guided Tour](/tutorials/explorer_guided_tour.html) — seven views of the Explorer, with the queries behind them - [Tutorials](/tutorials/) — index of all interactive tutorials diff --git a/how-to-use.qmd b/how-to-use.qmd index fe67c0ea..1a0676ce 100644 --- a/how-to-use.qmd +++ b/how-to-use.qmd @@ -36,7 +36,7 @@ Works in Chrome, Firefox, Edge, Safari, and Brave. No plugins, no downloads, no All code is visible and foldable on tutorial pages. Want to build your own analysis? -- **[Deep-Dive Analysis](/tutorials/zenodo_isamples_analysis.html)** — statistical exploration with Observable Plot +- **[Guided Tour](/tutorials/explorer_guided_tour.html)** — seven views of the Explorer and the queries behind them - **[Tutorials index](/tutorials/)** — step-by-step guides from basic exploration to advanced analysis - **[GitHub](https://github.com/isamplesorg/)** — all source code and data pipelines - **[Zenodo](https://zenodo.org/communities/isamples)** — archived datasets for reproducible research @@ -115,16 +115,15 @@ browsers use the parquet versions. ### Which tutorial uses which file {.unnumbered} -| | Interactive Explorer | Search Explorer | Deep-Dive Analysis | +| | Interactive Explorer | Search Explorer | Guided Tour | |---|:-:|:-:|:-:| | `wide.parquet` | | ● | | -| `wide_h3.parquet` | | | ● | | `facet_summaries.parquet` | ● | ● | ● | | `facet_cross_filter.parquet` | | ● | | | `sample_facets_v2.parquet` | ● | ● | | -| `h3_summary_res4/6/8.parquet` | ● | | | +| `h3_summary_res4/6/8.parquet` | ● | | ● (res4) | | `samples_map_lite.parquet` | ● | | | -| `vocab_labels.parquet` | ● | ● | | +| `vocab_labels.parquet` | ● | ● | ● | ### Quick query recipes {.unnumbered} diff --git a/tests/test_navbar_dropdowns.py b/tests/test_navbar_dropdowns.py index e2a58ed9..6ed164b8 100644 --- a/tests/test_navbar_dropdowns.py +++ b/tests/test_navbar_dropdowns.py @@ -29,11 +29,11 @@ def test_how_to_use_dropdown_has_overview(self, page): menu = _open_dropdown(page, "How to Use") assert menu.get_by_text("Overview").count() > 0 - def test_how_to_use_dropdown_has_deep_dive(self, page): - """And I see Deep-Dive Analysis.""" + def test_how_to_use_dropdown_has_guided_tour(self, page): + """And I see Guided Tour.""" page.goto(SITE_URL, wait_until="domcontentloaded") menu = _open_dropdown(page, "How to Use") - assert menu.get_by_text("Deep-Dive Analysis").count() > 0 + assert menu.get_by_text("Guided Tour").count() > 0 class TestAboutDropdown: diff --git a/tests/test_navigation.py b/tests/test_navigation.py index 5b1b088e..3ccf25e9 100644 --- a/tests/test_navigation.py +++ b/tests/test_navigation.py @@ -24,7 +24,7 @@ def test_about_sidebar_only_shows_own_items(self, page): sidebar = page.locator(".sidebar-navigation") assert sidebar.get_by_text("Objectives", exact=True).count() > 0 # Should NOT show items from other sections - assert sidebar.get_by_text("Deep-Dive Analysis").count() == 0 + assert sidebar.get_by_text("Guided Tour").count() == 0 assert sidebar.get_by_text("Vocabularies", exact=True).count() == 0 def test_architecture_sidebar_only_shows_own_items(self, page): @@ -33,7 +33,7 @@ def test_architecture_sidebar_only_shows_own_items(self, page): assert sidebar.get_by_text("Vocabularies", exact=True).count() > 0 # Should NOT show items from other sections assert sidebar.get_by_text("Objectives", exact=True).count() == 0 - assert sidebar.get_by_text("Deep-Dive Analysis").count() == 0 + assert sidebar.get_by_text("Guided Tour").count() == 0 def test_research_sidebar_only_shows_own_items(self, page): page.goto(f"{SITE_URL}/pubs.html", wait_until="domcontentloaded") @@ -41,7 +41,7 @@ def test_research_sidebar_only_shows_own_items(self, page): assert sidebar.get_by_text("Publications & Conferences").count() > 0 # Should NOT show items from other sections assert sidebar.get_by_text("Objectives", exact=True).count() == 0 - assert sidebar.get_by_text("Deep-Dive Analysis").count() == 0 + assert sidebar.get_by_text("Guided Tour").count() == 0 def test_sidebar_does_not_show_old_information_architecture(self, page): page.goto(f"{SITE_URL}/design/index.html", wait_until="domcontentloaded") @@ -57,10 +57,10 @@ def test_how_to_use_has_overview(self, page): sidebar = page.locator(".sidebar-navigation") assert sidebar.get_by_text("Overview", exact=True).count() > 0 - def test_how_to_use_has_deep_dive(self, page): + def test_how_to_use_has_guided_tour(self, page): page.goto(f"{SITE_URL}/how-to-use.html", wait_until="domcontentloaded") sidebar = page.locator(".sidebar-navigation") - assert sidebar.get_by_text("Deep-Dive Analysis").count() > 0 + assert sidebar.get_by_text("Guided Tour").count() > 0 def test_how_to_use_has_globe_viz(self, page): page.goto(f"{SITE_URL}/how-to-use.html", wait_until="domcontentloaded") diff --git a/tests/test_tutorials_landing.py b/tests/test_tutorials_landing.py index 8b0ba16e..25f50f93 100644 --- a/tests/test_tutorials_landing.py +++ b/tests/test_tutorials_landing.py @@ -15,10 +15,11 @@ class TestHowToUseLanding: """Scenario: Landing page lists all tutorial pathways.""" - def test_has_deep_dive_link(self, page): - """Given I am on the How to Use page, Then I see a link to Deep-Dive Analysis.""" + def test_has_guided_tour_link(self, page): + """Given I am on the How to Use page, Then I see a link to the Guided Tour + in the MAIN CONTENT (scoped so the permanent navbar link can't satisfy it).""" page.goto(HOW_TO_USE_URL, wait_until="domcontentloaded") - link = page.locator("a:has-text('Deep-Dive')") + link = page.locator("main a:has-text('Guided Tour')") assert link.count() > 0 def test_has_globe_viz_link(self, page): @@ -37,14 +38,18 @@ def test_has_narrow_vs_wide_link(self, page): class TestTutorialPageLoads: """Scenario: Each tutorial page loads without JavaScript errors.""" - def test_deep_dive_loads(self, page): - """Given I navigate to the Deep-Dive tutorial, Then no JS errors appear.""" + def test_guided_tour_loads(self, page): + """Given I navigate to the Guided Tour, Then the LIVE CELLS actually render + (waits for an Observable Plot SVG — DuckDB query + chart — not just DOM).""" errors = [] page.on("pageerror", lambda e: errors.append(str(e))) page.goto( - f"{SITE_URL}/tutorials/zenodo_isamples_analysis.html", + f"{SITE_URL}/tutorials/explorer_guided_tour.html", wait_until="domcontentloaded", ) + # The source-counts chart is the first Plot output; its appearance proves + # OJS booted, DuckDB-WASM initialized, and a remote parquet query ran. + page.wait_for_selector("figure svg, svg.plot-d6a7b5", timeout=90_000) assert page.title() != "" # Allow known non-critical errors but flag unexpected ones critical = [e for e in errors if "TypeError" in e or "ReferenceError" in e] diff --git a/tutorials/archive/parquet_cesium_isamples_wide.qmd b/tutorials/archive/parquet_cesium_isamples_wide.qmd index 4380b9a9..197bc55a 100644 --- a/tutorials/archive/parquet_cesium_isamples_wide.qmd +++ b/tutorials/archive/parquet_cesium_isamples_wide.qmd @@ -1079,4 +1079,4 @@ html`${ - [Interactive Explorer](/tutorials/isamples_explorer.html) - Search and filter samples - [Narrow vs Wide Performance](/tutorials/narrow_vs_wide_performance.html) - Benchmark comparison -- [Deep-Dive Analysis](/tutorials/zenodo_isamples_analysis.html) - Comprehensive DuckDB-WASM tutorial +- [Guided Tour](/tutorials/explorer_guided_tour.html) - Seven views of the Explorer, with live queries diff --git a/tutorials/explorer_guided_tour.qmd b/tutorials/explorer_guided_tour.qmd new file mode 100644 index 00000000..c12482df --- /dev/null +++ b/tutorials/explorer_guided_tour.qmd @@ -0,0 +1,241 @@ +--- +title: "A Guided Tour: What 6.7 Million Samples Can Tell You" +subtitle: "Seven views of the Interactive Explorer, and the questions each one answers" +author: "iSamples Team" +date: "2026-07-24" +aliases: + - /tutorials/zenodo_isamples_analysis.html +format: + html: + code-fold: true + code-summary: "Show the query behind this number" + toc: true + toc-depth: 2 + theme: cosmo +--- + +The [Interactive Explorer](https://isamples.org/explorer.html) covers ~6.7 +million physical samples — rock cores, potsherds, tissue vouchers, museum +specimens — about 6 million of them on one globe, running entirely in your +browser with no server behind it. This +tour walks through seven views of it, each answering a question about the data. +Most stops are **live links** — click one and the Explorer opens at the position, +search, and zoom being described (the URL carries that state; a couple of stops +ask you to click one control yourself, where the URL doesn't reach). + +A few stops also include a small **"the number behind the view"** code cell that +computes what you're seeing, live, from the same public data files the Explorer +itself reads. Unfold the code to see how — and to take your first step from +*viewing* the data to *querying* it. + +*(This page replaces an earlier "Zenodo deep-dive analysis" tutorial from the +2025 migration era; the browser-analysis techniques it demonstrated are now the +Explorer's own architecture. For how that works, see the pointers at the end.)* + +```{ojs} +//| output: false +// Boot DuckDB-WASM (the same engine the Explorer uses) + Observable Plot +duckdb = import("https://cdn.jsdelivr.net/npm/@duckdb/duckdb-wasm@1.28.0/+esm") +Plot = import("https://cdn.jsdelivr.net/npm/@observablehq/plot@0.6/+esm") +``` + +```{ojs} +//| output: false +db = { + const JSDELIVR_BUNDLES = duckdb.getJsDelivrBundles(); + const bundle = await duckdb.selectBundle(JSDELIVR_BUNDLES); + const worker_url = URL.createObjectURL( + new Blob([`importScripts("${bundle.mainWorker}");`], {type: 'text/javascript'}) + ); + const worker = new Worker(worker_url); + const logger = new duckdb.ConsoleLogger(duckdb.LogLevel.WARNING); + const db_instance = new duckdb.AsyncDuckDB(logger, worker); + await db_instance.instantiate(bundle.mainModule, bundle.pthreadWorker); + return db_instance; +} + +conn = db.connect() + +sql = async (q) => { + const c = await conn; + const result = await c.query(q); + return result.toArray().map(r => { + const o = r.toJSON(); + for (const k of Object.keys(o)) if (typeof o[k] === 'bigint') o[k] = Number(o[k]); + return o; + }); +} + +// The canonical data files — the SAME ones the Explorer loads (see CANONICAL.md). +// All three used on this page are kilobyte-scale, so these cells load instantly. +FACET_SUMMARIES = 'https://data.isamples.org/isamples_202608_facet_summaries.parquet' +VOCAB_LABELS = 'https://data.isamples.org/vocab_labels_202608.parquet' +H3_RES4 = 'https://data.isamples.org/isamples_202608_h3_summary_res4.parquet' +``` + +## Stop 1 — The world at a glance + +**[Open the Explorer →](https://isamples.org/explorer.html)** + +![The world at boot: every dot is a cluster of samples, colored by source +collection.](images/tour/tour_world.png) + +The first thing the globe tells you is *where science samples the world* — and +where it doesn't. Blue (SESAR) dots trace the oceans: decades of marine cores +and dredges. Red (OpenContext) concentrates in the Mediterranean and Near East: +archaeological excavation. Green (GEOME) follows biodiversity fieldwork; +yellow (Smithsonian) reflects museum collections. Sampling is not uniform — it +is a map of scientific attention. + +```{ojs} +sources = sql(` + SELECT facet_value AS source, count + FROM read_parquet('${FACET_SUMMARIES}') + WHERE facet_type = 'source' + ORDER BY count DESC`) +``` + +```{ojs} +Plot.plot({ + marginLeft: 110, + height: 180, + x: {label: "located samples", tickFormat: "s"}, + y: {label: null}, + marks: [ + Plot.barX(sources, {y: "source", x: "count", + fill: d => ({SESAR: "#3366CC", OPENCONTEXT: "#DC3912", + GEOME: "#109618", SMITHSONIAN: "#FF9900"}[d.source] || "#666")}), + Plot.text(sources, {y: "source", x: "count", text: d => d.count.toLocaleString(), dx: 4, textAnchor: "start"}) + ] +}) +``` + +One honest wrinkle the globe can't show: these bars total **~6.03 million** — +the *located* samples. Roughly 700,000 more records exist in the collection +with no usable coordinates: they never appear on the globe or in the viewport +table, but a world-scope search still finds them. Counting only what's visible +would overstate the map and understate the archive. + +## Stop 2 — Search that knows concepts + +**[Search "pottery Cyprus" →](https://isamples.org/explorer.html?search=pottery+Cyprus)** + +![1,305 matches; the located top results ring Cyprus as +pins.](images/tour/tour_search_pins.png) + +Type *pottery Cyprus* and 1,305 samples answer, with the top matches pinned on +the island. Two things make this more than word-matching. First, the search +index folds in **vocabulary concepts**: a sample tagged with the *concept* +"pottery" matches even if its free text never contains the word. Second, it +folds in **place names**, so "Cyprus" finds samples whose sampling-site place +metadata says Cyprus even when their free-text descriptions never mention it. (Both were real user +bug reports once — the fixes became the index's design.) + +Search runs on a pre-built sharded index: your query fetches a few small files, +not the whole dataset. The classic full-scan search still exists — add +`&fts=off` to the URL to feel the difference. + +## Stop 3 — Two sites, two sciences + +**[Fly to central Anatolia →](https://isamples.org/explorer.html#v=1&lat=37.6619&lng=32.8334&alt=150000&mode=point)** + +![The densest region-scale (res4) cell in the collection, centered on the +Çatalhöyük region of central Anatolia — shown here in cluster +view.](images/tour/tour_anatolia.png) + +The densest region-scale cell in the whole collection is not over a volcano or +a reef — it centers on the countryside around Çatalhöyük, a Neolithic settlement +mound in Turkey excavated for decades (the cell aggregates ~150,000 samples from +the surrounding region's digs). Compare +**[Axial Seamount](https://isamples.org/explorer.html?search=axial+seamount+summit+caldera)**, +an undersea volcano off Oregon: basalt collected by submersible. A potsherd +from a dig and a basalt from the seafloor carry the same metadata shape — +sample, sampling event, site, coordinates, material classification. That shared +shape (the iSamples model) is what lets one tool serve both sciences. + +```{ojs} +dense = sql(` + SELECT sample_count, center_lat, center_lng, dominant_source + FROM read_parquet('${H3_RES4}') + ORDER BY sample_count DESC + LIMIT 5`) +``` + +```{ojs} +html`

The five densest cells on the globe (click to fly there):

+
    ${(await dense).map(d => html`
  1. + + ${d.sample_count.toLocaleString()} samples — ${d.dominant_source.toLowerCase()}, + near (${d.center_lat.toFixed(2)}, ${d.center_lng.toFixed(2)})
    2. `)} +
` +``` + +## Stop 4 — Slicing by material + +**[Open the Explorer and expand the *Material* facet →](https://isamples.org/explorer.html)** + +Check a box — say *Mineral* — and everything updates together: globe, counts, +table. Check two boxes and the counts still hold (that trick, fast multi-filter +counting over millions of rows with no server, is most of the Explorer's hidden +machinery). The facet labels are human-readable because a small vocabulary +file maps classification URIs to names: + +```{ojs} +materials = sql(` + SELECT COALESCE(vl.pref_label, fs.facet_value) AS material, fs.count + FROM read_parquet('${FACET_SUMMARIES}') fs + LEFT JOIN read_parquet('${VOCAB_LABELS}') vl ON fs.facet_value = vl.uri AND vl.lang = 'en' + WHERE fs.facet_type = 'material' + ORDER BY fs.count DESC + LIMIT 10`) +``` + +```{ojs} +Plot.plot({ + marginLeft: 200, + height: 300, + x: {label: "located samples", tickFormat: "s"}, + y: {label: null}, + marks: [Plot.barX(materials, {y: "material", x: "count", fill: "#4a7", sort: {y: "-x"}})] +}) +``` + +Rock and mineral dominate (SESAR's size shows through), but biogenic and +anthropogenic materials are each hundreds of thousands strong — this is +genuinely a cross-domain collection, not a geology database with guests. + +## Stop 5 — Density vs identity + +**[Toggle the heatmap →](https://isamples.org/explorer.html)** *(checkbox in +the right panel)*, and try the **2D map** (globe button in the map toolbar → +2D). + +Clusters answer "how many, roughly where." The heatmap answers "where is +sampling *concentrated*" — it makes the Mediterranean glow. Zoomed in, point +mode answers "which samples, exactly." The honest rule of thumb: heatmaps for +patterns, points for identity, and never trust a color's *intensity* as a count +— click and read the number instead. + +## Stop 6 — From view to data-in-hand + +Any view's sample table (below the globe) shows PID, place, date, and a +**Source URL** linking each sample back to its home collection's record. +**Download CSV** exports up to 50,000 samples matching your current viewport and filters (broader views export the first 50,000). And because the URL +carries the whole view state, **Copy Link to Current View** gives you a +citation-grade pointer to *this exact slice* — paste it in a paper, a class +assignment, or an issue report. + +## Stop 7 — Under the hood (and doing this yourself) + +There is no server. The Explorer is a static page querying public +[Parquet](https://parquet.apache.org/) files over HTTP range requests with +[DuckDB-WASM](https://duckdb.org/docs/api/wasm/overview) — the code cells on +this page use the identical technique against the identical files. To go +further: + +- **[EXPLORER_QUERIES.md](https://github.com/isamplesorg/isamplesorg.github.io/blob/main/EXPLORER_QUERIES.md)** — plain-English walkthrough of every query the Explorer runs, with copy-paste DuckDB snippets +- **[CANONICAL.md](https://github.com/isamplesorg/isamplesorg.github.io/blob/main/CANONICAL.md)** — which data files are canonical (this page uses only those) +- **[DATA_PROVENANCE.md](https://github.com/isamplesorg/isamplesorg.github.io/blob/main/DATA_PROVENANCE.md)** — how every derived file is built from the source export +- The **[iSamples Zenodo community](https://zenodo.org/communities/isamples)** — the archived, citable snapshot of the data substrate + + diff --git a/tutorials/images/tour/tour_anatolia.png b/tutorials/images/tour/tour_anatolia.png new file mode 100644 index 00000000..739bc28e Binary files /dev/null and b/tutorials/images/tour/tour_anatolia.png differ diff --git a/tutorials/images/tour/tour_search_pins.png b/tutorials/images/tour/tour_search_pins.png new file mode 100644 index 00000000..d6be703d Binary files /dev/null and b/tutorials/images/tour/tour_search_pins.png differ diff --git a/tutorials/images/tour/tour_world.png b/tutorials/images/tour/tour_world.png new file mode 100644 index 00000000..af69e720 Binary files /dev/null and b/tutorials/images/tour/tour_world.png differ diff --git a/tutorials/index.qmd b/tutorials/index.qmd index 37975633..d3db435d 100644 --- a/tutorials/index.qmd +++ b/tutorials/index.qmd @@ -9,7 +9,7 @@ number-sections: false | Tutorial | What You'll Learn | |----------|-------------------| | [**Interactive Explorer**](/explorer.html) | Browse, search, and filter 6.7M samples on a 3D globe or in a paginated table | -| [**Deep-Dive Analysis**](zenodo_isamples_analysis.qmd) | Comprehensive DuckDB-WASM analysis with Observable JS — charts, maps, statistics | +| [**Guided Tour**](explorer_guided_tour.qmd) | Seven views of the Interactive Explorer — what each answers, with live deep links and the queries behind the numbers | | [**Technical: Narrow vs Wide**](narrow_vs_wide_performance.qmd) | Schema comparison and performance benchmarks for the PQG data formats | ## What's in the Data? {.unnumbered} diff --git a/tutorials/narrow_vs_wide_performance.qmd b/tutorials/narrow_vs_wide_performance.qmd index 166e110c..5452f84b 100644 --- a/tutorials/narrow_vs_wide_performance.qmd +++ b/tutorials/narrow_vs_wide_performance.qmd @@ -1070,4 +1070,4 @@ The wide format eliminates ~9M edge rows (60% file size reduction). The H3 varia - [Interactive Explorer](/explorer.html) - Browse samples on a 3D globe - [Search Explorer](isamples_explorer.qmd) - Faceted search and filter across all samples -- [Deep-Dive Analysis](zenodo_isamples_analysis.qmd) - Comprehensive DuckDB-WASM tutorial +- [Guided Tour](explorer_guided_tour.qmd) - Seven views of the Explorer, with live queries diff --git a/tutorials/zenodo_isamples_analysis.qmd b/tutorials/zenodo_isamples_analysis.qmd deleted file mode 100644 index d3a499b8..00000000 --- a/tutorials/zenodo_isamples_analysis.qmd +++ /dev/null @@ -1,1507 +0,0 @@ ---- -title: "Efficient Analysis of Large iSamples Dataset from Zenodo" -subtitle: "Using DuckDB-WASM and Observable JS for Browser-Based Data Analysis" -author: "iSamples Team" -date: "2025-09-05" -format: - html: - code-fold: true - code-summary: "Show code" - toc: true - toc-depth: 3 - theme: cosmo - include-in-header: - text: | - - ---- - -# Introduction - -This tutorial demonstrates how to efficiently analyze large geospatial datasets directly in your browser without downloading entire files. We'll use DuckDB-WASM and Observable JS to perform fast, memory-efficient analysis and create interactive visualizations. - -**Note**: This tutorial attempts to connect to the live iSamples dataset (~300MB, 6+ million records). If CORS restrictions prevent access to the remote file, it automatically falls back to a representative demo dataset that demonstrates the same analytical techniques. - -## Key Technologies - -- **DuckDB-WASM**: In-browser analytical database with HTTP range request support -- **Observable Plot**: Grammar of graphics for interactive visualizations -- **Observable Inputs**: Interactive controls for data exploration -- **CORS Handling**: Automatic fallback for cross-origin restrictions - -## Dataset Information - -**Primary dataset** (Jan 2026, H3-indexed): -- **URL**: `https://data.isamples.org/isamples_202601_wide_h3.parquet` -- **Size**: ~292 MB wide format with H3 indices, 6.7M MaterialSampleRecords (20M total rows) -- **H3 columns**: Pre-computed `h3_res4`, `h3_res6`, `h3_res8` (BIGINT) for spatial grouping -- **Sources**: SESAR (4.6M), OpenContext (1M), GEOME (605K), Smithsonian (322K) -- **Hosting**: Cloudflare R2 with HTTP range request support - -**Facet summaries** (2KB, instant): -- **URL**: `https://data.isamples.org/isamples_202601_facet_summaries.parquet` -- **Schema**: `facet_type`, `facet_value`, `scheme`, `count` - -**Note**: *Data was originally archived on Zenodo and is now served from Cloudflare R2 for better performance and reliability.* - -**Fallback dataset** (if remote data fails): -- **Type**: Generated demo data with realistic structure -- **Size**: 10K records with same schema and representative geographic distribution -- **Purpose**: Demonstrates all analytical techniques with faster loading - -# Setup and Database Connection - -```{ojs} -//| label: test-reactivity - -// Simple test to debug Observable reactivity -viewof test_input = Inputs.range([1, 100], { - label: "Test Input:", - step: 1, - value: 42 -}) - -// In Observable, the input itself IS the value when referenced reactively -test_value = test_input - -test_output = test_value * 2 - -md` -## 🧪 Reactivity Test - -**Raw input object**: ${typeof test_input} (Constructor: ${test_input?.constructor?.name}) -**Extracted value**: ${test_value} -**Output (2x input)**: ${test_output} -**Value type**: ${typeof test_value} - -This should update in real-time as you move the slider. -` -``` - -```{ojs} -//| label: setup -//| code-fold: true - -// Import required libraries using reliable CDNs -duckdb = import("https://cdn.jsdelivr.net/npm/@duckdb/duckdb-wasm@1.28.0/+esm") -Plot = import("https://cdn.jsdelivr.net/npm/@observablehq/plot@0.6/+esm") -Inputs = import("https://cdn.jsdelivr.net/npm/@observablehq/inputs@0.10/+esm") -d3 = require("d3@7") -topojson = require("topojson-client@3") - -// Dataset URLs - try multiple options for CORS compatibility -// Primary: Cloudflare R2 (Jan 2026 wide format with H3 indices) -parquet_urls = [ - 'https://data.isamples.org/isamples_202601_wide_h3.parquet', - - // Fallback: original wide format without H3 - 'https://data.isamples.org/current/wide.parquet', - - // Fallback: older versions - 'https://labs.dataunbound.com/docs/2025/07/isamples_export_2025_04_21_16_23_46_geo.parquet', - 'https://zenodo.org/api/records/15278211/files/isamples_export_2025_04_21_16_23_46_geo.parquet/content' -] - -// Pre-computed facet summaries (2KB - loads instantly) -facet_summaries_url = 'https://data.isamples.org/isamples_202601_facet_summaries.parquet' - -// Test CORS and find working URL - with rate limiting protection -working_parquet_url = { - // Check if we've recently failed (to avoid repeated rate limiting) - const lastFailTime = localStorage.getItem('zenodo_last_fail'); - const now = Date.now(); - if (lastFailTime && (now - parseInt(lastFailTime)) < 300000) { // 5 minutes - console.log('⏳ Recently hit rate limit, using demo data'); - return null; - } - - for (const url of parquet_urls) { - try { - console.log(`Testing URL: ${url}`); - // Test with a small HEAD request first - const response = await fetch(url, { - method: 'HEAD', - mode: 'cors' - }); - if (response.ok) { - console.log(`✅ Working URL found: ${url}`); - // Clear any previous failure time - localStorage.removeItem('zenodo_last_fail'); - return url; - } else if (response.status === 429) { - console.log(`⚠️ Rate limited: ${url}`); - localStorage.setItem('zenodo_last_fail', now.toString()); - return null; - } - } catch (error) { - console.log(`❌ Failed URL: ${url}, Error: ${error.message}`); - if (error.message.includes('429') || error.message.includes('TOO MANY REQUESTS')) { - localStorage.setItem('zenodo_last_fail', now.toString()); - return null; - } - continue; - } - } - - // If no URL works, we'll use a fallback approach - console.log("⚠️ No direct URL worked, will use demo data"); - return null; -} - -// Create DuckDB instance and connect to remote parquet -db = { - // Use DuckDB ES modules from jsdelivr - const JSDELIVR_BUNDLES = duckdb.getJsDelivrBundles(); - - // Select bundle for the platform - const bundle = await duckdb.selectBundle(JSDELIVR_BUNDLES); - - // Create worker - const worker_url = URL.createObjectURL( - new Blob([`importScripts("${bundle.mainWorker}");`], {type: 'text/javascript'}) - ); - - const worker = new Worker(worker_url); - const logger = new duckdb.ConsoleLogger(duckdb.LogLevel.WARNING); - const db_instance = new duckdb.AsyncDuckDB(logger, worker); - - // Initialize the database - await db_instance.instantiate(bundle.mainModule, bundle.pthreadWorker); - - // Connect to database - const conn = await db_instance.connect(); - - if (working_parquet_url) { - try { - // Create view with aliased column names to match downstream queries. - // The Jan 2026 wide parquet uses different names than the original schema. - await conn.query(`CREATE VIEW isamples_data AS SELECT *, - latitude AS sample_location_latitude, - longitude AS sample_location_longitude, - n AS source_collection, - pid AS sample_identifier, - CAST(NULL AS VARCHAR) AS has_material_category - FROM read_parquet('${working_parquet_url}')`); - - // Test the connection with a simple query to catch rate limiting - await conn.query(`SELECT count(*) FROM isamples_data LIMIT 1`); - console.log("✅ Successfully connected to remote Parquet file"); - - } catch (error) { - console.log("❌ Failed to read remote Parquet:", error.message); - // Check if it's a rate limiting error - if (error.message.includes('429') || error.message.includes('TOO MANY REQUESTS')) { - console.log("⏳ Rate limited - storing failure time"); - localStorage.setItem('zenodo_last_fail', Date.now().toString()); - } - // Create demo data as fallback - await createDemoData(conn); - } - } else { - // Create demo data as fallback - await createDemoData(conn); - } - - return conn; -} - -// Function to create demo data when remote file is not accessible -createDemoData = async (conn) => { - console.log("Creating demo dataset..."); - - // Create a demo table with similar structure and realistic data - await conn.query(` - CREATE TABLE isamples_data AS - SELECT - 'DEMO_' || i as sample_identifier, - CASE - WHEN i % 4 = 0 THEN 'SESAR' - WHEN i % 4 = 1 THEN 'OPENCONTEXT' - WHEN i % 4 = 2 THEN 'GEOME' - ELSE 'SMITHSONIAN' - END as source_collection, - - -- Generate realistic coordinates - CASE - WHEN i % 5 = 0 THEN -120 + (random() * 50) -- North America - WHEN i % 5 = 1 THEN -10 + (random() * 40) -- Europe - WHEN i % 5 = 2 THEN 100 + (random() * 40) -- Asia - WHEN i % 5 = 3 THEN 115 + (random() * 30) -- Australia - ELSE -180 + (random() * 360) -- Other - END as sample_location_longitude, - - CASE - WHEN i % 5 = 0 THEN 25 + (random() * 25) -- North America - WHEN i % 5 = 1 THEN 35 + (random() * 35) -- Europe - WHEN i % 5 = 2 THEN 20 + (random() * 30) -- Asia - WHEN i % 5 = 3 THEN -40 + (random() * 30) -- Australia - ELSE -90 + (random() * 180) -- Other - END as sample_location_latitude, - - CASE - WHEN i % 8 = 0 THEN 'rock' - WHEN i % 8 = 1 THEN 'mineral' - WHEN i % 8 = 2 THEN 'sediment' - WHEN i % 8 = 3 THEN 'soil' - WHEN i % 8 = 4 THEN 'fossil' - WHEN i % 8 = 5 THEN 'meteorite' - WHEN i % 8 = 6 THEN 'organic' - ELSE 'other' - END as has_material_category, - - 'Demo sample ' || i as label - - FROM generate_series(1, 10000) t(i) -- Generate 10,000 demo records - `); - - console.log("✅ Demo dataset created with 10,000 sample records"); -} -``` - -```{ojs} -//| label: connection-status - -md` -## Connection Status - -${working_parquet_url ? - `✅ **Connected to live data**: Using ${working_parquet_url.includes('r2.dev') ? 'Cloudflare R2' : working_parquet_url.includes('zenodo.org') ? 'Zenodo' : 'fallback'} hosting -📊 **Dataset**: 6.7M MaterialSampleRecords from iSamples -🌐 **Data source**: ${working_parquet_url}` - : - `⚠️ **Using demo data**: Remote file not accessible due to CORS restrictions -📊 **Dataset**: 10K synthetic records with realistic structure -💡 **Note**: This demonstrates the same analysis patterns with representative data` -} -` -``` - -# Basic Data Exploration - -Let's start with fundamental queries to understand the dataset structure and size. - -```{ojs} -//| label: basic-stats - -// Get total record count - this only reads metadata! -total_count = { - const result = await db.query(`SELECT count(*) as total FROM isamples_data`); - const rows = result.toArray(); - return Number(rows[0].total); // Convert BigInt to Number -} - -// Count records with geographic coordinates -geo_count = { - const result = await db.query(` - SELECT count(*) as geo_total - FROM isamples_data - WHERE sample_location_latitude IS NOT NULL - AND sample_location_longitude IS NOT NULL - `); - const rows = result.toArray(); - return Number(rows[0].geo_total); // Convert BigInt to Number -} - -// Calculate percentage with coordinates -geo_percentage = Math.round((geo_count / total_count) * 100) -``` - -```{ojs} -//| label: display-basic-stats - -md` -## Dataset Overview - -- **Total records**: ${total_count.toLocaleString()} -- **Records with coordinates**: ${geo_count.toLocaleString()} (${geo_percentage}%) -- **Data source**: ${working_parquet_url ? 'Remote Parquet file (~300 MB)' : 'Demo dataset (synthetic)'} -- **Data transferred for these stats**: < 1 KB (metadata only!) -` -``` - -# Source Collection Analysis - -Analyze the distribution of samples across different source collections. - -```{ojs} -//| label: source-analysis - -// Get source collection counts -source_data = { - const result = await db.query(` - SELECT - source_collection, - count(*) as sample_count, - count(CASE WHEN sample_location_latitude IS NOT NULL - AND sample_location_longitude IS NOT NULL - THEN 1 END) as geo_count - FROM isamples_data - GROUP BY source_collection - ORDER BY sample_count DESC - `); - // Convert BigInt values to Numbers - const processedData = result.toArray().map(row => ({ - ...row, - sample_count: Number(row.sample_count), - geo_count: Number(row.geo_count) - })); - - return processedData; -} - -// Create interactive table showing source distribution -viewof source_table = Inputs.table(source_data, { - columns: [ - "source_collection", - "sample_count", - "geo_count" - ], - header: { - source_collection: "Source Collection", - sample_count: "Total Samples", - geo_count: "Samples with Coordinates" - }, - format: { - sample_count: d3.format(","), - geo_count: d3.format(",") - } -}) -``` - -```{ojs} -//| label: source-chart - -// Create bar chart of source collections -source_chart = { - // Validate that source_data is an array - if (!Array.isArray(source_data)) { - return html`
Error: Source data is not available
`; - } - - return Plot.plot({ - title: "Sample Distribution by Source Collection", - x: { - label: "Number of samples", - tickFormat: "~s" - }, - y: { - label: "Source Collection", - domain: source_data.map(d => d.source_collection) - }, - marks: [ - Plot.barX(source_data, { - x: "sample_count", - y: "source_collection", - fill: "steelblue", - sort: {y: "x", reverse: true} - }), - Plot.text(source_data, { - x: "sample_count", - y: "source_collection", - text: d => d3.format("~s")(d.sample_count), - dx: 10, - textAnchor: "start" - }) - ], - marginLeft: 120, - height: 250, - width: 600 - }); -} -``` - -# Geographic Distribution Analysis - -Examine the geographic spread of samples and identify regional patterns. - -```{ojs} -//| label: geographic-stats - -// Get geographic statistics -geo_stats = { - const result = await db.query(` - SELECT - count(*) as total_with_coords, - min(sample_location_latitude) as min_lat, - max(sample_location_latitude) as max_lat, - avg(sample_location_latitude) as avg_lat, - min(sample_location_longitude) as min_lon, - max(sample_location_longitude) as max_lon, - avg(sample_location_longitude) as avg_lon - FROM isamples_data - WHERE sample_location_latitude IS NOT NULL - AND sample_location_longitude IS NOT NULL - `); - const rows = result.toArray(); - const row = rows[0]; - // Convert BigInt values to Numbers where needed - return { - ...row, - total_with_coords: Number(row.total_with_coords) - }; -} - -// Data-driven regional analysis using H3 res4 cell grouping -// Replaces hardcoded CASE WHEN bounding boxes with dynamic discovery -regional_data = { - // Check if h3_res4 column exists (H3-indexed file) - let hasH3 = false; - try { - const colCheck = await db.query(`SELECT h3_res4 FROM isamples_data LIMIT 1`); - hasH3 = true; - } catch (e) { - hasH3 = false; - } - - if (hasH3) { - // H3-based regional grouping: discover dense clusters dynamically - const result = await db.query(` - SELECT - h3_res4, - COUNT(*) as sample_count, - AVG(sample_location_latitude) as avg_lat, - AVG(sample_location_longitude) as avg_lon, - COUNT(DISTINCT source_collection) as source_count, - MODE(source_collection) as dominant_source - FROM isamples_data - WHERE sample_location_latitude IS NOT NULL - AND sample_location_longitude IS NOT NULL - AND h3_res4 IS NOT NULL - GROUP BY h3_res4 - HAVING COUNT(*) > 100 - ORDER BY sample_count DESC - `); - // Assign region labels based on centroid location - return result.toArray().map(row => { - const lat = row.avg_lat; - const lon = row.avg_lon; - let region; - if (lon >= -130 && lon <= -60 && lat >= 20 && lat <= 55) region = 'North America'; - else if (lon >= -15 && lon <= 45 && lat >= 30 && lat <= 75) region = 'Europe'; - else if (lon >= 90 && lon <= 150 && lat >= 15 && lat <= 55) region = 'East Asia'; - else if (lon >= 110 && lon <= 160 && lat >= -50 && lat <= -5) region = 'Australia'; - else if (lon >= -90 && lon <= -30 && lat >= -60 && lat <= 15) region = 'South America'; - else if (lon >= -20 && lon <= 55 && lat >= -40 && lat <= 30) region = 'Africa'; - else region = 'Other'; - return { - region, - source_collection: row.dominant_source, - sample_count: Number(row.sample_count), - avg_lat: row.avg_lat, - avg_lon: row.avg_lon, - h3_cell: row.h3_res4, - source_count: Number(row.source_count) - }; - }); - } else { - // Fallback for non-H3 files: use simple lat/lon-based grouping - const result = await db.query(` - SELECT - source_collection, - count(*) as sample_count, - avg(sample_location_latitude) as avg_lat, - avg(sample_location_longitude) as avg_lon - FROM isamples_data - WHERE sample_location_latitude IS NOT NULL - AND sample_location_longitude IS NOT NULL - GROUP BY source_collection - ORDER BY sample_count DESC - `); - return result.toArray().map(row => { - const lat = row.avg_lat; - const lon = row.avg_lon; - let region = 'Other'; - if (lon >= -130 && lon <= -60 && lat >= 20 && lat <= 55) region = 'North America'; - else if (lon >= -15 && lon <= 45 && lat >= 30 && lat <= 75) region = 'Europe'; - else if (lon >= 90 && lon <= 150 && lat >= 15 && lat <= 55) region = 'East Asia'; - else if (lon >= 110 && lon <= 160 && lat >= -50 && lat <= -5) region = 'Australia'; - return { - region, - source_collection: row.source_collection, - sample_count: Number(row.sample_count), - avg_lat: row.avg_lat, - avg_lon: row.avg_lon - }; - }); - } -} -``` - -```{ojs} -//| label: display-geo-stats - -md` -## Geographic Statistics - -- **Latitude range**: ${geo_stats.min_lat.toFixed(3)}° to ${geo_stats.max_lat.toFixed(3)}° -- **Longitude range**: ${geo_stats.min_lon.toFixed(3)}° to ${geo_stats.max_lon.toFixed(3)}° -- **Average location**: ${geo_stats.avg_lat.toFixed(3)}°, ${geo_stats.avg_lon.toFixed(3)}° -- **Dense H3 clusters**: ${regional_data.length} (cells with >100 samples) -- **Regions discovered**: ${[...new Set(regional_data.map(d => d.region))].join(', ')} - -*Regional grouping is data-driven using H3 resolution-4 hexagonal cells, replacing hardcoded bounding boxes.* -` -``` - -# Interactive Regional Explorer - -Create an interactive visualization to explore samples by region and source. - -```{ojs} -//| label: regional-controls - -// Interactive region selector -viewof selected_region = Inputs.select( - ["All", ...new Set(regional_data?.map?.(d => d.region) || [])], - { - label: "Select Region:", - value: "All" - } -) -``` - -```{ojs} -//| label: regional-chart - -// Regional distribution chart (data-driven from H3 clusters) -regional_chart = { - if (!Array.isArray(regional_data)) { - return html`
Error: Regional data is not available
`; - } - - // Aggregate H3 cell data by discovered region - const regionTotals = d3.rollup( - regional_data, - v => d3.sum(v, d => d.sample_count), - d => d.region - ); - - const aggregatedData = Array.from(regionTotals, ([region, total]) => ({ - region: region, - sample_count: total - })).sort((a, b) => b.sample_count - a.sample_count); - - return Plot.plot({ - title: `Sample Distribution by Region (H3-derived, ${aggregatedData.length} regions)`, - width: 700, - height: 300, - marginLeft: 120, - x: { - label: "Number of samples", - tickFormat: "~s" - }, - y: { - label: "Region", - domain: aggregatedData.map(d => d.region) - }, - marks: [ - Plot.barX(aggregatedData, { - x: "sample_count", - y: "region", - fill: "steelblue", - sort: {y: "x", reverse: true} - }), - Plot.text(aggregatedData, { - x: "sample_count", - y: "region", - text: d => d3.format("~s")(d.sample_count), - dx: 10, - textAnchor: "start" - }) - ] - }); -} -``` - -# Efficient Sampling for Visualization - -Create a representative sample of the data for detailed visualization while minimizing data transfer. - -```{ojs} -//| label: sampling-controls - -// Sample size control -viewof sample_size = Inputs.range([1000, 50000], { - label: "Sample Size:", - step: 1000, - value: 10000 -}) - -// Sample per collection limit -viewof max_per_collection = Inputs.range([500, 5000], { - label: "Max per Collection:", - step: 250, - value: 2500 -}) -``` - -```{ojs} -//| label: create-sample - -// Create stratified sample - simplified for DuckDB-WASM compatibility -sample_data = { - // In Observable, inputs are automatically reactive values - const maxPerCollection = max_per_collection; - const sampleSizeValue = sample_size; - - // Use a fixed sampling rate to avoid NaN issues - const samplingRate = 0.1; // Sample 10% of data - - // Validate that all values are proper numbers - if (isNaN(maxPerCollection) || isNaN(sampleSizeValue) || isNaN(samplingRate)) { - console.error('Invalid numeric values detected', {maxPerCollection, sampleSizeValue, samplingRate}); - throw new Error('Invalid sampling parameters'); - } - - const result = await db.query(` - WITH sampled_data AS ( - SELECT - sample_identifier, - source_collection, - sample_location_longitude as longitude, - sample_location_latitude as latitude, - has_material_category, - label - FROM isamples_data - WHERE sample_location_latitude IS NOT NULL - AND sample_location_longitude IS NOT NULL - AND random() < ${samplingRate} - ) - SELECT * FROM sampled_data - LIMIT ${sampleSizeValue} - `); - return result.toArray(); -} - -// Calculate sample statistics -sample_stats = { - const total = sample_data.length; - const by_source = d3.rollup(sample_data, v => v.length, d => d.source_collection); - return { - total, - by_source: Array.from(by_source, ([source, count]) => ({source, count})) - .sort((a, b) => b.count - a.count) - }; -} -``` - -```{ojs} -//| label: display-sample-stats - -md` -## Sample Statistics - -**Total sample size**: ${sample_stats.total.toLocaleString()} points -**Data transfer**: ~${Math.round(sample_stats.total * 6 * 8 / 1024 / 1024)} MB (estimated) -**Reduction factor**: ${Math.round(geo_count / sample_stats.total)}x fewer points - -**Distribution by source**: -${sample_stats.by_source.map(d => `- ${d.source}: ${d.count.toLocaleString()}`).join('\n')} -` -``` - -# Interactive World Map - -Create an interactive scatter plot map showing the geographic distribution of samples. - -```{ojs} -//| label: map-controls - -// Map projection selector -viewof projection = Inputs.select([ - "equirectangular", - "orthographic", - "mercator" -], { - label: "Map Projection:", - value: "equirectangular" -}) - -// Point size control -viewof point_size = Inputs.range([0.5, 5], { - label: "Point Size:", - step: 0.1, - value: 1.5 -}) -``` - -```{ojs} -//| label: world-map - -// Load world map data -world = fetch("https://cdn.jsdelivr.net/npm/world-atlas@2/countries-110m.json") - .then(response => response.json()) - -// Create world map with sample points -world_map = { - // Await the world data - const worldData = await world; - const countries = topojson.feature(worldData, worldData.objects.countries); - - // In Observable, inputs are automatically reactive values - let pointRadius = point_size; - const mapProjection = projection; - - // Validate that pointRadius is a proper number - if (isNaN(pointRadius) || pointRadius <= 0) { - console.error('Invalid point size detected:', point_size, 'using fallback:', 1.5); - pointRadius = 1.5; - } - - return Plot.plot({ - title: `Geographic Distribution of ${sample_stats.total.toLocaleString()} Sample Points`, - projection: mapProjection, - marks: [ - // World map outline - Plot.geo(countries, { - fill: "#f0f0f0", - stroke: "#ccc", - strokeWidth: 0.5 - }), - // Sample points colored by source - Plot.dot(sample_data, { - x: "longitude", - y: "latitude", - fill: "source_collection", - r: pointRadius, - fillOpacity: 0.7, - stroke: "white", - strokeWidth: 0.2 - }) - ], - color: { - legend: true, - domain: ["SESAR", "OPENCONTEXT", "GEOME", "SMITHSONIAN"], - range: ["#3366cc", "#dc3912", "#109618", "#ff9900"] - }, - width: 900, - height: 500, - style: { - background: "#f8f9fa" - } - }); -} -``` - -# Material Category Analysis - -Explore the distribution of material categories across different sources. - -```{ojs} -//| label: material-analysis - -// Material data: use pre-computed facet summaries for instant results -// Falls back to full-scan if summaries unavailable -material_data = { - try { - // Try pre-computed summaries first (2KB, instant) - const result = await db.query(` - SELECT - facet_value as has_material_category, - 'ALL' as source_collection, - count as category_count - FROM read_parquet('${facet_summaries_url}') - WHERE facet_type = 'material' - ORDER BY count DESC - `); - return result.toArray().map(row => ({ - ...row, - category_count: Number(row.category_count) - })); - } catch (e) { - console.warn("Facet summaries unavailable, falling back to full scan:", e.message); - const result = await db.query(` - SELECT - source_collection, - has_material_category, - count(*) as category_count - FROM isamples_data - WHERE has_material_category IS NOT NULL - GROUP BY source_collection, has_material_category - ORDER BY source_collection, category_count DESC - `); - return result.toArray().map(row => ({ - ...row, - category_count: Number(row.category_count) - })); - } -} - -// Top categories from pre-computed summaries (instant) -top_categories = { - try { - const result = await db.query(` - SELECT - facet_value as has_material_category, - count as total_count - FROM read_parquet('${facet_summaries_url}') - WHERE facet_type = 'material' - ORDER BY count DESC - LIMIT 10 - `); - return result.toArray().map(row => ({ - ...row, - total_count: Number(row.total_count) - })); - } catch (e) { - console.warn("Facet summaries unavailable, falling back to full scan:", e.message); - const result = await db.query(` - SELECT - has_material_category, - count(*) as total_count - FROM isamples_data - WHERE has_material_category IS NOT NULL - GROUP BY has_material_category - ORDER BY total_count DESC - LIMIT 10 - `); - return result.toArray().map(row => ({ - ...row, - total_count: Number(row.total_count) - })); - } -} -``` - -```{ojs} -//| label: material-controls - -// Category selector for detailed view -viewof selected_category = Inputs.select( - ["All", ...top_categories.map(d => d.has_material_category)], - { - label: "Focus on Category:", - value: "All" - } -) - -// Filter material data -filtered_material_data = selected_category === "All" - ? material_data - : material_data.filter(d => d.has_material_category === selected_category) -``` - -```{ojs} -//| label: material-charts - -// Top categories chart -categories_chart = Plot.plot({ - title: "Top 10 Material Categories", - x: { - label: "Number of samples", - tickFormat: "~s" - }, - y: { - label: "Material Category", - domain: top_categories.map(d => d.has_material_category) - }, - marks: [ - Plot.barX(top_categories, { - x: "total_count", - y: "has_material_category", - fill: "coral", - sort: {y: "x", reverse: true} - }), - Plot.text(top_categories, { - x: "total_count", - y: "has_material_category", - text: d => d3.format("~s")(d.total_count), - dx: 10, - textAnchor: "start" - }) - ], - marginLeft: 150, - height: 300, - width: 700 -}) - -// Material by source chart -material_by_source_chart = { - // Validate that data is available - if (!Array.isArray(material_data) || !Array.isArray(top_categories)) { - return html`
Error: Material data is not available
`; - } - - // In Observable, inputs are automatically reactive values - const currentCategory = selected_category; - - // Filter the data based on selection - let chartData; - if (currentCategory === "All") { - // For "All", show top 10 categories across all sources - chartData = material_data.filter(d => - top_categories.map(c => c.has_material_category).includes(d.has_material_category) - ); - } else { - // For specific category, show by source collection - chartData = material_data.filter(d => d.has_material_category === currentCategory); - } - - // If no data, return early with a message - if (chartData.length === 0) { - return html`
No data available for selected category: ${currentCategory}
`; - } - - return Plot.plot({ - title: currentCategory === "All" - ? "Material Categories by Source Collection" - : `${currentCategory} by Source Collection`, - x: { - label: "Number of samples", - tickFormat: "~s" - }, - y: { - label: currentCategory === "All" ? "Material Category" : "Source Collection" - }, - color: { - legend: true, - domain: ["SESAR", "OPENCONTEXT", "GEOME", "SMITHSONIAN"], - range: ["#3366cc", "#dc3912", "#109618", "#ff9900"] - }, - marks: [ - Plot.barX(chartData, { - x: "category_count", - y: currentCategory === "All" ? "has_material_category" : "source_collection", - fill: "source_collection", - sort: {y: "x", reverse: true} - }) - ], - marginLeft: 150, - height: Math.max(250, chartData.length * 20), - width: 700 - }); -} -``` - -# Performance Summary - -This browser-based approach demonstrates remarkable efficiency compared to traditional methods. - -```{ojs} -//| label: performance-summary - -md` -# Performance Analysis - -## Browser-Based vs Traditional Approaches - -| Approach | Time | Memory | Data Transfer | Environment | -|----------|------|--------|---------------|-------------| -| **Traditional (pandas)** | 40-150s | 600-1200 MB | 300 MB | Local Python | -| **Our browser approach** | 10-30s | <100 MB | <5 KB + samples | Any browser | -| **Improvement** | **~5x faster** | **~10x less memory** | **~99% less transfer** | **Universal** | - -## Key Benefits - -✅ **Universal Access**: Runs in any modern browser -✅ **Memory Efficient**: Analyze 300MB datasets using <100MB browser memory -✅ **Fast**: Instant metadata queries, efficient sampling -✅ **Interactive**: Real-time parameter adjustment and visualization -✅ **Scalable**: Same approach works for GB or TB datasets -✅ **Reproducible**: Self-contained analysis with no local setup required - -## Technical Achievements - -- **HTTP Range Requests**: Only downloads needed data portions -- **Columnar Processing**: Parquet format enables efficient column-wise operations -- **Lazy Evaluation**: Queries are optimized before execution -- **In-Browser Analytics**: Full analytical database running in JavaScript -- **Interactive Visualization**: Real-time exploration with Observable Plot - -This approach enables **big data analysis in any browser** and makes large-scale geospatial analysis universally accessible! 🌍 -` -``` - -# Additional Resources - -```{ojs} -//| label: resources - -md` -## 📚 Learn More - -- [DuckDB-WASM Documentation](https://duckdb.org/docs/api/wasm/) -- [Observable Plot](https://observablehq.com/plot/) -- [Observable Inputs](https://observablehq.com/@observablehq/inputs) -- [GeoParquet Specification](https://geoparquet.org/) -- [HTTP Range Requests](https://developer.mozilla.org/en-US/docs/Web/HTTP/Range_requests) -- [iSamples Project](https://www.isamples.org/) - -## 🔧 Technical Implementation - -This notebook demonstrates how to: -1. Connect to remote Parquet files using DuckDB-WASM -2. Perform efficient metadata-only queries -3. Create stratified samples for visualization -4. Build interactive controls and visualizations -5. Achieve high performance with minimal data transfer - -The complete workflow enables sophisticated data analysis directly in the browser without any local software installation or large file downloads. -` -``` - -# 🔍 Viewport Map with Graceful Degradation (deck.gl ↔ DuckDB-WASM) - -```{ojs} -//| label: map-libs -// Import CSS first -{ - const link = document.createElement("link"); - link.rel = "stylesheet"; - link.href = "https://unpkg.com/maplibre-gl@3.6.1/dist/maplibre-gl.css"; - document.head.appendChild(link); -} - -// Map libs only - skip deck.gl for now due to dependency issues -maplibregl = await import("https://cdn.skypack.dev/maplibre-gl@3?min") - -// For now, we'll skip deck.gl due to luma.gl dependency issues -// and just show the map without WebGL overlays -decklib = null -``` - -```{ojs} -//| label: map-config -MAPCFG = ({ - center: [20.0, 45.0], // Centered to include Italy and Israel - zoom: 3, // Zoomed out one level - minZoom: 1, - maxZoom: 17, - basemap: "https://basemaps.cartocdn.com/gl/positron-gl-style/style.json", - pointColor: [20,110,180,200], - pointRadiusPx: 2, - maxPerTileHard: sample_limit_value, // Dynamic sample limit from controls - modulusByZoom(z){ - if (z >= 13) return 1; - if (z >= 11) return 2; - if (z >= 9 ) return 8; - if (z >= 7 ) return 32; - if (z >= 5 ) return 128; - return 512; - }, - aggDegByZoom(z){ - if (z >= 11) return 0.05; - if (z >= 9 ) return 0.1; - if (z >= 7 ) return 0.25; - if (z >= 5 ) return 0.5; - return 1.0; - } -}) -``` - -```{ojs} -//| label: viewport-sampling-controls - -// Text input for viewport sample limit -viewof sample_limit = Inputs.text({ - label: "Viewport Sample Limit:", - value: "20000", - placeholder: "Enter number (works reliably up to ~1M)" -}) - -// Convert to number with validation -sample_limit_value = Math.max(1, parseInt(sample_limit) || 20000) - -// Display current sample limit -md`**Current viewport sample limit**: ${sample_limit_value.toLocaleString()} samples per viewport - -*Note: Performance tested up to ~1M samples. For larger datasets, need more scalable approach (e.g., lonboard/deck.gl optimization)*` -``` - -```{ojs} -//| label: map-capability -pickMode = () => { - const c = document.createElement("canvas"); - const gl2 = c.getContext("webgl2", {antialias:false}); - if (gl2) return "full"; - const gl = c.getContext("webgl", {antialias:false}); - if (!gl) return "raster"; - const ext = gl.getExtension("WEBGL_debug_renderer_info"); - const r = ext ? gl.getParameter(ext.UNMASKED_RENDERER_WEBGL) : ""; - const low = /swiftshader|llvmpipe|software/i.test(r) || gl.getParameter(gl.MAX_TEXTURE_SIZE) < 4096; - return low ? "raster" : "lite"; -} -MODE = pickMode() -``` - -```{ojs} -//| label: map-container -html` -
-
-
- Mode: ${MODE} ${working_parquet_url ? "" : "(demo data)"} -
-
` -``` - -```{ojs} -//| label: map-init -viewport = { - const Map = maplibregl.default?.Map || maplibregl.Map; - const NavigationControl = maplibregl.default?.NavigationControl || maplibregl.NavigationControl; - if (!Map) { - console.error("MapLibre GL not loaded properly:", maplibregl); - console.error("Available keys:", Object.keys(maplibregl)); - throw new Error("MapLibre GL Map class failed to load"); - } - - // Wait for DOM container to be available - return new Promise((resolve) => { - function waitForContainer() { - const container = document.getElementById("map"); - if (container) { - const map = new Map({ - container: "map", - style: MAPCFG.basemap, - center: MAPCFG.center, - zoom: MAPCFG.zoom, - minZoom: MAPCFG.minZoom, - maxZoom: MAPCFG.maxZoom, - attributionControl: true - }); - map.addControl(new NavigationControl({visualizePitch:true})); - - // Store map instance globally for deck.gl access - window._observableMap = map; - - const observable = Generators.observe((notify) => { - function emit() { - const c = map.getCenter(); - const z = map.getZoom(); - const b = map.getBounds(); - notify({ - center: [c.lng, c.lat], - zoom: z, - bounds: [b.getWest(), b.getSouth(), b.getEast(), b.getNorth()], - map: map // Include map reference - }); - } - const onMove = () => { - if (emit._t) return; - emit._t = setTimeout(() => { emit(); emit._t = null; }, 120); - }; - map.on("load", emit); - map.on("move", onMove); - map.on("zoom", onMove); - if (map.loaded()) emit(); - return () => { - window._observableMap = null; - map.remove(); - }; - }); - - resolve(observable); - } else { - // Try again in a few milliseconds - setTimeout(waitForContainer, 50); - } - } - waitForContainer(); - }); -} -``` - -```{ojs} -//| label: viewport-sample -async function queryViewportSamples(db, vp, mode) { - const [xmin, ymin, xmax, ymax] = vp.bounds; - const z = vp.zoom; - const baseMod = MAPCFG.modulusByZoom(z); - const MODULUS = mode === "lite" ? Math.max(1, baseMod * 2) : baseMod; - const LIMIT = mode === "lite" ? Math.floor(MAPCFG.maxPerTileHard/2) : MAPCFG.maxPerTileHard; - const sql = ` - SELECT - sample_identifier, - sample_location_longitude AS lon, - sample_location_latitude AS lat, - source_collection - FROM isamples_data - WHERE sample_location_longitude BETWEEN ${xmin} AND ${xmax} - AND sample_location_latitude BETWEEN ${ymin} AND ${ymax} - AND ABS(hash(sample_identifier)) % ${MODULUS} = 0 - LIMIT ${LIMIT} - `; - const res = await db.query(sql); - return res.toArray(); -} -``` - -```{ojs} -//| label: data-viewport -viewport_data = { - // Always load data (not just for non-raster mode since we need it for fallback plot) - if (!viewport?.bounds || !db) { - console.log('No viewport bounds or db available'); - return null; - } - - try { - const data = await queryViewportSamples(db, viewport, MODE); - console.log(`Loaded ${data?.length || 0} viewport samples for bounds:`, viewport.bounds); - console.log('Sample data:', data?.slice(0, 3)); // Show first 3 records - return data; - } catch (error) { - console.error('Error querying viewport samples:', error); - return null; - } -} -``` - -```{ojs} -//| label: maplibre-layer-update -map_layer_update = { - // Update MapLibre map layers with viewport data - if (!viewport?.map || !viewport_data || !Array.isArray(viewport_data)) { - console.log('No map or data available for layer update'); - return null; - } - - const map = viewport.map; - - // Convert viewport data to GeoJSON format for MapLibre - const geojson = { - type: "FeatureCollection", - features: viewport_data.map(d => ({ - type: "Feature", - geometry: { - type: "Point", - coordinates: [d.lon, d.lat] - }, - properties: { - sample_identifier: d.sample_identifier, - source_collection: d.source_collection - } - })) - }; - - // Remove existing source/layer if they exist - if (map.getSource('viewport-points')) { - map.removeLayer('viewport-points'); - map.removeSource('viewport-points'); - } - - // Add new source and layer - map.addSource('viewport-points', { - type: 'geojson', - data: geojson - }); - - map.addLayer({ - id: 'viewport-points', - type: 'circle', - source: 'viewport-points', - paint: { - 'circle-radius': 4, - 'circle-color': [ - 'match', - ['get', 'source_collection'], - 'SESAR', '#3366cc', - 'OPENCONTEXT', '#dc3912', - 'GEOME', '#109618', - 'SMITHSONIAN', '#ff9900', - '#666666' // fallback color - ], - 'circle-opacity': 0.8, - 'circle-stroke-width': 1, - 'circle-stroke-color': '#ffffff' - } - }); - - console.log(`Updated MapLibre map with ${viewport_data.length} points`); - return `Updated with ${viewport_data.length} points`; -} -``` - -```{ojs} -//| label: deckgl-layer -function scatterLayer(data) { - const ScatterplotLayer = decklib.default?.ScatterplotLayer || decklib.ScatterplotLayer; - return new ScatterplotLayer({ - id: "vp-points", - data, - getPosition: d => [d.lon, d.lat], - getFillColor: MAPCFG.pointColor, - getRadius: MAPCFG.pointRadiusPx, - radiusUnits: "pixels", - radiusMinPixels: MAPCFG.pointRadiusPx, - radiusMaxPixels: MAPCFG.pointRadiusPx * 2, - pickable: false, - parameters: {depthTest: false} - }); -} -``` - -```{ojs} -//| label: deckgl-overlay -deck_overlay = { - if (!decklib) { - return html`
Deck.gl temporarily disabled due to dependency issues. Map shows without WebGL overlay.
`; - } - - if (MODE === "raster") return html`
Raster mode active
`; - - try { - // Get map from viewport or global reference - const map = viewport?.map || window._observableMap; - if (!map) { - return html`
Waiting for map initialization...
`; - } - - // Use MapboxOverlay from deck.gl library - const MapboxOverlay = decklib.default?.MapboxOverlay || decklib.MapboxOverlay; - if (!MapboxOverlay) { - console.error("Available decklib properties:", Object.keys(decklib || {})); - console.error("Available decklib.default properties:", Object.keys(decklib.default || {})); - throw new Error("MapboxOverlay not found in deck.gl library"); - } - - const overlay = new MapboxOverlay({interleaved: true, layers: []}); - - // Add overlay to map - map.addControl(overlay); - - // Store overlay reference for other cells - window._deckOverlay = overlay; - - function render() { - if (MODE !== "raster" && Array.isArray(viewport_data) && viewport_data.length > 0) { - overlay.setProps({layers: [scatterLayer(viewport_data)]}); - } else { - overlay.setProps({layers: []}); - } - } - render(); - - return html`
Deck.gl overlay initialized (${viewport_data?.length || 0} points)
`; - } catch (error) { - console.error('Deck overlay error:', error); - return html`
Error initializing deck.gl: ${error.message}
`; - } -} -``` - -```{ojs} -//| label: fps-autodowngrade -{ - if (MODE === "raster") return; - - const overlay = window._deckOverlay; - const badge = document.getElementById("mode-badge"); - - if (!overlay || !badge) return; - - let samples = 0, total = 0, degraded = false; - - function tick(t) { - if (degraded || !overlay) return; - - try { - const stats = overlay._deck?.stats; - const ft = stats && stats.get && stats.get("Frame Time")?.lastTiming; - if (ft && ft > 0) { - total += ft; - samples++; - } - - if (samples < 120) { - requestAnimationFrame(tick); - } else if (samples > 0) { - const avg = total / samples; - if (avg > 50) { - degraded = true; - overlay.setProps({layers: []}); - badge.innerHTML = 'Mode: raster (auto)'; - mutable autodowngraded = true; - } - } - } catch (error) { - console.warn('FPS monitoring error:', error); - } - } - - // Wait a bit for deck to initialize - setTimeout(() => requestAnimationFrame(tick), 1000); -} -``` - -```{ojs} -//| label: plot-fallback -mutable autodowngraded = false - -// Since deck.gl is disabled, always show a Plot fallback for viewport data -plot_fallback = { - console.log('Plot fallback - decklib:', decklib); - console.log('Plot fallback - MODE:', MODE); - console.log('Plot fallback - viewport_data type:', typeof viewport_data); - console.log('Plot fallback - viewport_data length:', viewport_data?.length); - - if (!decklib || MODE === "raster" || autodowngraded) { - if (!viewport_data || !Array.isArray(viewport_data) || viewport_data.length === 0) { - return html`
- Viewport Data Status:
- • deck.gl: ${decklib ? 'enabled' : 'disabled'}
- • Mode: ${MODE}
- • Data: ${viewport_data ? `${viewport_data.length} samples` : 'null/undefined'}
- • Waiting for viewport data to load... -
`; - } - - // Use the Plot that's already imported at the top of the notebook - // Need to get the world data for the map background - const worldData = await world; - const countries = topojson.feature(worldData, worldData.objects.countries); - - return Plot.plot({ - title: `Viewport Samples (${viewport_data.length.toLocaleString()} points)`, - width: 900, - height: 480, - projection: "mercator", - inset: 6, - marks: [ - // World map outline - Plot.geo(countries, { - fill: "#f0f0f0", - stroke: "#ccc", - strokeWidth: 0.5 - }), - // Add a big red dot on London for testing - Plot.dot([{lon: 0.1278, lat: 51.5074}], { - x: "lon", - y: "lat", - r: 20, - fill: "red", - stroke: "darkred", - strokeWidth: 3 - }), - // Add the actual data points - Plot.dot(viewport_data, { - x: "lon", - y: "lat", - fill: "source_collection", - r: 3, - fillOpacity: 0.7, - stroke: "white", - strokeWidth: 0.2 - }) - ], - color: { - legend: true, - domain: ["SESAR", "OPENCONTEXT", "GEOME", "SMITHSONIAN"], - range: ["#3366cc", "#dc3912", "#109618", "#ff9900"] - } - }); - } - - return html`
Deck.gl enabled - no fallback needed
`; -} -``` - -```{ojs} -//| label: map-legend -md` -**Rendering mode**: \`${MODE}\` -- **full / lite** → deck.gl Scatterplot sampled from DuckDB-WASM by current viewport -- **raster** (or auto) → aggregated canvas dots (size ≈ count per grid cell) -` -```