Add SML and SMU support to the SuperMAG reader

[Simon060899]

Summary

Extend SMESuperMAG to support the complete standard SuperMAG auroral electrojet trio: SME, SML, and SMU.

The existing behavior remains backward compatible: calling read() without a variable selection continues to return only sme and file_name.

Motivation

SWVO previously exposed only SME, although SuperMAG supplies SME, SML, and SMU together and all three are commonly needed for auroral electrojet and substorm analyses.

During authenticated historical testing, SuperMAG also intermittently returned an empty response for an otherwise valid day. Repeating the same request succeeded. The downloader has therefore been hardened so that one transient response does not unnecessarily terminate a long historical download.

API

SMESuperMAG.read() now accepts a variables argument:

reader.read(start, end)
# sme, file_name

reader.read(start, end, variables="all")
# sme, sml, smu, file_name

reader.read(start, end, variables=["smu", "sml"])
# smu, sml, file_name

A single variable or ordered iterable is supported. Duplicate variables are removed while preserving their first occurrence. Empty, non-string, and unknown selections produce descriptive errors.

Compatibility and caching

• The class name, import path, constructor, environment variable, daily filenames, time-window behavior, and default SME-only schema are unchanged.
• New downloads request and cache SME, SML, and SMU together.
• Existing SME-only caches remain valid for default reads.
• When SML or SMU is requested from a legacy cache:
  • download=False reports the missing fields and explains how to upgrade the cache.
  • download=True atomically replaces the incomplete daily cache with a complete file.
• Corrupt caches follow the same remediation policy.
• Existing cache files remain untouched if downloading, parsing, or writing fails.
• The current BaseIO.url interface is preserved; resolved SuperMAG request URLs use encoded parameters.

Transient download handling

The downloader now treats the following conditions as transient:

• Empty response bodies or empty JSON record arrays.
• Request timeouts and connection failures.
• HTTP 429 and HTTP 5xx responses.

Each affected day is attempted up to four times, with waits of one, two, and four seconds.

After all attempts are exhausted:

• A historical download_and_process() operation records the failed date, continues with subsequent days, and emits a summary warning.
• An on-demand read(..., download=True) remains strict and raises the final failure because the requested data cannot be returned.

Permanent failures, including invalid SuperMAG credentials, malformed data, and other HTTP 4xx responses, are not retried.

Requests use encoded parameters, and prepared URLs containing the SuperMAG username are not logged.

Data handling

• All three variables use nanotesla units.
• SuperMAG fill values are converted to NaN.
• Returned indices remain timezone-aware UTC.
• Requested column ordering and file_name provenance are preserved.
• Per-file temporary output and atomic replacement remove the previous shared temporary-directory dependency.

Documentation

A dedicated SuperMAG electrojet guide documents:

• SME, SML, and SMU definitions and units.
• Authentication and credential handling.
• Variable-selection examples.
• Cache compatibility and upgrades.
• Transient and permanent failure behavior.
• Time coverage and error handling.
• SuperMAG data references and acknowledgment guidance.

The README and Sphinx user-guide index have also been updated.

Verification

Local verification:

• Targeted SuperMAG tests: 81 passed; one opt-in live test skipped in the deterministic run.
• Authenticated historical SuperMAG live test: passed separately.
• Complete tests/io suite: 688 passed, 14 skipped, 0 failed.
  • Eleven SIDC tests were skipped because the external service was unavailable.
  • Two OMNI live tests are opt-in.
  • The one SuperMAG live test is opt-in and passed separately.
• Ruff lint and formatting: passed.
• Ty type checking: passed.
• Strict isolated Sphinx build of the SuperMAG guide: passed without warnings.
• REUSE compliance: passed.

The fork’s GitHub Actions matrix passed on Ubuntu and macOS with Python 3.11, 3.12, 3.13, and 3.14.

[sahiljhawar]

LGTM