Skip to content

ENH: Add JupyterLite CI/CD infrastructure#13925

Draft
natinew77-creator wants to merge 117 commits into
mne-tools:mainfrom
natinew77-creator:jupyterlite-gh-actions
Draft

ENH: Add JupyterLite CI/CD infrastructure#13925
natinew77-creator wants to merge 117 commits into
mne-tools:mainfrom
natinew77-creator:jupyterlite-gh-actions

Conversation

@natinew77-creator

@natinew77-creator natinew77-creator commented May 27, 2026

Copy link
Copy Markdown
Contributor

Tracking Issue: #13929

What does this implement/fix?
This PR integrates JupyterLite into the MNE-Python documentation build, allowing users to run tutorials interactively directly in their browser without a local Python environment.

Key technical implementations:

  • Integrates jupyterlite-sphinx into the Sphinx-Gallery pipeline, automatically generating "Try in JupyterLite" buttons for tutorials and examples.
  • Injects a hidden setup cell into the generated notebooks via conf.py to automatically handle Pyodide-specific browser quirks:
    • Installs mne and pyodide-http natively via micropip.
    • Patches Pyodide networking (pyodide_http.patch_all()) so MNE's pooch downloader can successfully fetch datasets from the browser.
    • Monkey-patches mne.viz.utils.plt_show to correctly render MNE's Matplotlib figures inline within the WebAssembly environment.

Additional information

  • This represents the completion of the first major GSoC milestone.
  • CircleCI will now automatically build the JupyterLite assets and provide a live preview link in the CI checks.
  • Note on limitations: During testing, I identified two architectural edge cases for future discussion: browser RAM limitations when tutorials attempt to download massive (>1GB) datasets, and occasional PyPI vs. main branch version mismatches since JupyterLite currently pulls the stable MNE release.
A28CDACE-D867-491E-8B11-013DD1635D4A 16D95C3F-B372-47D1-AFE3-D5E10395F775 0F9F0048-D585-4B8C-BE90-1E4D34BF1EEF

@natinew77-creator

natinew77-creator commented Jun 15, 2026

Copy link
Copy Markdown
Contributor Author

Hi @teonbrooks, Status Update: This PR is now ready for your review!

As the first step in my GSoC roadmap, this PR successfully introduces the core JupyterLite infrastructure to the MNE-Python documentation. Here is what has been achieved:

  • Integrated jupyterlite-sphinx to automatically build interactive JupyterLite instances for the examples/tutorials.
  • Added a GitHub Actions CI/CD workflow to build and deploy the JupyterLite site.
  • Patched MNEBrowseFigure by fixing the pyodide_plt_show argument signature to accept multiple positional arguments.
  • Fixed an extension execution race condition in doc/conf.py to ensure JupyterLite successfully bundles the generated notebooks during the Sphinx build-finished event.

I also looked deeply into the [Errno 26] Operation in progress error we hit during the 10_overview tutorial. It turns out this is a fundamental limitation with Pyodide and JupyterLite. The 10_overview tutorial uses mne.datasets.sample.data_path() to download a 1.45 GB dataset from osf.io. First, osf.io has strict CORS headers that completely block browser-based WebAssembly fetches. Second, even if we were able to bypass the CORS restrictions, downloading a 1.45 GB file directly into browser RAM via Pyodide causes the browser tab to instantly crash with an Out-Of-Memory error.

To gracefully handle this and prevent user confusion, I've written a custom pooch.Pooch.fetch interceptor in our doc/conf.py configuration. Now, whenever a JupyterLite user tries to download these massive OSF datasets natively in the browser, it intercepts the request and prints a polite error message advising them to download the dataset locally and upload it directly into the JupyterLite file browser!

For smaller datasets that are CORS-friendly, it automatically falls back to Pyodide's native pyfetch via urllib so they work seamlessly without any patching needed in the tutorial code itself.

I think this is the best architectural approach for handling the massive tutorials. I'm ready to mark this PR as complete so we can move down the GSoC checklist and start tackling xeus-python and the 3D PyVista rendering! Let me know what you think! Looking forward to your feedback!

@natinew77-creator

Copy link
Copy Markdown
Contributor Author

Quick Follow-up:
I also tracked down and fixed the bug that prevented the notebooks from loading correctly in the JupyterLite UI.

It turned out to be a race condition during the Sphinx build-finished event—jupyterlite_sphinx was executing before sphinx_gallery had finished generating the example notebooks. I've reordered the extensions in conf.py so they execute in the correct sequence, and all the notebooks are now successfully populating!

Integrates jupyterlite-sphinx into the MNE-Python doc build so every
sphinx-gallery example gets a 'Try in Browser' button backed by a
Pyodide/WebAssembly kernel.

- doc/conf.py: configure jupyterlite_sphinx; build a local MNE dev
  wheel with relaxed Pyodide constraints; copy required MNE sample-data
  subset into JupyterLite's virtual filesystem; inject a setup cell that
  installs MNE via micropip (keep_going=True bypasses version conflicts),
  mocks missing stdlib modules (lzma, multiprocessing), patches pooch to
  block large OSF downloads, and sets MNE_DATA paths
- .circleci/config.yml: ensure MNE sample data is on disk before the
  doc build so conf.py can copy it into jupyterlite_contents/
- .github/workflows/jupyterlite.yml: standalone GH Actions workflow on
  the jupyterlite-gh-actions branch that builds and uploads the site
- pyproject.toml: add jupyterlite-pyodide-kernel and jupyterlite-sphinx
  to the [doc] extras
- .gitignore: exclude jupyterlite_contents build artifacts
- mne/parallel.py: return False early in _running_in_joblib_context()
  on emscripten; joblib parallel backends are unavailable in the browser
- mne/utils/config.py: catch Exception (not just ValueError) when
  loading the MNE config JSON; Pyodide's json parser raises SyntaxError
  on a corrupt or absent config file
Tutorials and examples that call interactive Qt backends (raw.plot(),
epochs.plot(), ica.plot_sources(), etc.) or depend on large datasets not
bundled in JupyterLite will hang or error in Pyodide. Wrap them with
sys.platform guards so they are skipped when running in the browser.

Interactive Qt plots (skip on emscripten):
- tutorials/intro/10_overview.py: raw.plot(), stc.plot()
- tutorials/intro/15_inplace.py: original_raw.plot(), rereferenced_raw.plot()
- tutorials/intro/20_events_from_raw.py: raw.copy().pick().plot(), raw.plot()
- tutorials/intro/40_sensor_locations.py: mne.viz.plot_alignment()
- tutorials/evoked/40_whitened.py: raw.plot(), epochs.plot()
- examples/preprocessing/muscle_ica.py: all ica.plot_* calls

Large datasets unavailable in the browser (raise RuntimeError on emscripten):
- tutorials/io/60_ctf_bst_auditory.py: BST auditory dataset (~2.9 GB)
- tutorials/io/70_reading_eyetracking_data.py: EyeLink misc dataset
- examples/visualization/eyetracking_plot_heatmap.py: EyeLink dataset
natinew77-creator and others added 13 commits June 26, 2026 09:35
…erLite

- doc/conf.py: Fix lzma mock to use real stdlib lzma when available in
  Pyodide instead of LZMAFile=object which broke joblib's compressor
  registration
- 10_overview.py: Guard ica.plot_properties() which opens an interactive
  Qt window
- 15_inplace.py: Guard set_eeg_reference block which fails under
  Python 3.13 in Pyodide
- 20_events_from_raw.py: Guard STIM channel plot and all EEGLAB sections
  that require the unavailable testing dataset
- 40_sensor_locations.py: Guard ssvep dataset loading and sphere plot
  that require the unavailable ssvep dataset
- 50_configure_mne.py: Guard KIT test data loading whose test files are
  stripped from the Pyodide wheel
- 70_report.py: Skip Report.save() file-writing in browser, guard
  nibabel-dependent add_bem, 3D methods (add_trans/add_stc/add_forward/
  add_inverse_operator), missing ECG/events files, pandas-dependent
  make_metadata, and the HDF5 round-trip section

All intro tutorials (10, 15, 20, 30, 40, 50, 70) now run cleanly in
JupyterLite/Pyodide without errors.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Comment thread mne/datasets/tests/test_datasets.py Fixed
@natinew77-creator

Copy link
Copy Markdown
Contributor Author

Notes on where this is at.

Data. Comes from mne.datasets.lite_data — one archive on the MNE OSF, pulled
and extracted once during the build, then the files are served at the docs root. The
archive never reaches the browser.

The setup cell used to fetch ~403 MB at the start of every notebook. Now it takes 18
small files (43 MB) up front and pulls the heavy ones only when something reads them
(raw 128 MB, filt raw 66 MB, fwd 54 MB, inv 50 MB, ernoise 42 MB, src 21 MB). Most
notebooks are ~43 MB; the inverse ones more.

19 files have the launch button hiddenJUPYTERLITE_EXCLUDE in doc/conf.py.
r_interop needs the R runtime, read_impedances needs antio (compiled, no WASM
build), and the other 17 are multi-GB datasets — brainstorm, spm_face, opm, hf_sef,
fieldtrip_cmc, visual_92_categories.

31 files still show a badge but will fail on missing data. Seven datasets are
bundled (sample, testing/EEGLAB, ssvep, eegbci, kiloword, erp_core, mtrf); anything
else hits the OSF block:

  • somato (6), misc (5), fsaverage (4), fnirs_motor (2), eyelink (2)
  • one each: limo, multimodal, refmeg_noise, phantom_kernel, phantom_4dbti,
    phantom_kit, parcellation, seeg, ecog, sleep, 35_eeg_no_mri, 60_visualize_stc

Each is either "bundle the dataset" or "add to the exclude list". fnirs_motor and
eyelink look small enough to just add; somato/misc/fsaverage are big. Undecided.

3D. SourceEstimate.plot goes through pyvista-js (vtk.js) since the normal VTK
stack won't load in WASM. Works on 10_overview, 70_point_spread, 80_dics.
plot_alignment, Brain + sensors, and volume stc aren't supported.

Also note which pages can't run in the browser and why, so it's not a
mystery when the JupyterLite badge is missing.
Copies one raw, one forward and the FreeSurfer surfaces out of the somato
dataset CI already has on disk. Also fixes the 3D shim, which fetched
surfaces from MNE-sample-data even when subjects_dir pointed elsewhere.
It built the 3D renderer before the time-course figure, so the whole call
died in WASM and the notebook lost both. Draw the glass brain and the
dipole markers with pyvista-js instead, and plot the time courses first.
compute_mne_inverse_volume needs the ~178 MB volume inverse and volume
source estimates aren't rendered in the browser; mixed_source_space_inverse
needs aseg.mgz plus the 3D src.plot(); 20_dipole_fit needs the BEM solution
and nilearn downloads an MNI template at runtime, which CORS blocks.
Patch _get_renderer instead of reimplementing the 3D functions, so MNE
still does its own geometry and coordinate-frame work and only the
drawing is replaced. Gets plot_alignment and plot_bem working for the
surfaces we serve. The interactive Brain/plot_field widget layer is
still out of reach.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants