From 7af74428ad26dfcab9800b5e463f84c34cae0dc0 Mon Sep 17 00:00:00 2001 From: Dimitri Yatsenko Date: Tue, 21 Jul 2026 16:50:42 -0500 Subject: [PATCH] docs(whats-new): fold 2.2.4 into What's New in 2.2 Patch releases are documented as a 'Changes in 2.x.y' section within the minor version's What's New page (as done for 2.3.1/2.3.2), not as a standalone page. 2.2.4 was the lone exception. - Add a condensed 'Changes in 2.2.4' section to whats-new-22.md (DJ_STORES, DJ_IGNORE_CONFIG_FILE, any-attr .secrets, storage-adapter plugin contract), linking the how-to/spec pages for detail. - Remove the standalone whats-new-2-2-4.md page and its nav entry. - Redirect about/whats-new-2-2-4.md -> about/whats-new-22.md to preserve the published URL. --- mkdocs.yaml | 4 +- src/about/whats-new-2-2-4.md | 137 ----------------------------------- src/about/whats-new-22.md | 9 +++ 3 files changed, 12 insertions(+), 138 deletions(-) delete mode 100644 src/about/whats-new-2-2-4.md diff --git a/mkdocs.yaml b/mkdocs.yaml index 153c3f7a..29b5c7d8 100644 --- a/mkdocs.yaml +++ b/mkdocs.yaml @@ -146,7 +146,6 @@ nav: - About: - about/index.md - What's New in 2.3: about/whats-new-23.md - - What's New in 2.2.4: about/whats-new-2-2-4.md - What's New in 2.2: about/whats-new-22.md - What's New in 2.1: about/whats-new-21.md - What's New in 2.0: about/whats-new-2.md @@ -193,6 +192,9 @@ plugins: # provenance.md was renamed to trace.md when the provenance framing # was retired; keep the published URL working. reference/specs/provenance.md: reference/specs/trace.md + # whats-new-2-2-4 was folded into whats-new-22 (Changes in 2.2.4); + # keep the published URL working. + about/whats-new-2-2-4.md: about/whats-new-22.md - mermaid2 - section-index - autorefs diff --git a/src/about/whats-new-2-2-4.md b/src/about/whats-new-2-2-4.md deleted file mode 100644 index 1ae46b63..00000000 --- a/src/about/whats-new-2-2-4.md +++ /dev/null @@ -1,137 +0,0 @@ -# What's New in DataJoint 2.2.4 - -DataJoint 2.2.4 introduces **env-var-only configuration of storage**, **a public plugin-adapter contract for third-party storage protocols**, and tightens credential loading for files. - -> **Upgrading from 2.2.0–2.2.3?** No breaking changes for projects using `datajoint.json` or `.secrets/`. The new env vars are purely additive. - -## Overview - -The DataJoint platform — and many production deployments generally — provision configuration entirely from environment variables: there is no `datajoint.json` in the container image and no `.secrets/` directory on disk. Until 2.2.4, this worked for the database connection (`DJ_HOST`, `DJ_USER`, `DJ_PASS`, …) but **not** for object stores: per-store credentials had to be configured through `datajoint.json` or `.secrets/stores..` files. - -DataJoint 2.2.4 closes that gap with two new env vars, both purely additive: - -- `DJ_STORES` — a JSON-encoded copy of the entire `stores` dict, in the same shape used in `datajoint.json`. -- `DJ_IGNORE_CONFIG_FILE` — a boolean flag that skips both `datajoint.json` and the secrets directory entirely. - -The 2.2.4 release also formalizes the **storage-adapter plugin contract** (`datajoint.storage` entry-point group), which had been used internally since 2.0 but lacked a published spec. Third-party packages can now register storage protocols (Databricks Unity Catalog Volumes, custom HTTP-based stores, lab-specific archive systems, …) by subclassing `dj.StorageAdapter` and declaring an entry point. - -## `DJ_STORES` — JSON-encoded stores configuration - -!!! version-added "New in 2.2.4" - `DJ_STORES` accepts a JSON object identical to the `stores` block of `datajoint.json`. - -A single env var carries the entire `stores` dict. The format matches what users already write in `datajoint.json`, so config can be moved between file and env var by copy-paste — no per-field naming scheme to learn. - -```bash -export DJ_STORES='{ - "default": "main", - "main": { - "protocol": "s3", - "endpoint": "s3.amazonaws.com", - "bucket": "my-bucket", - "location": "my-project/production", - "access_key": "AKIA...", - "secret_key": "wJal..." - } -}' -``` - -For plugin-registered adapters, the field names are whatever the adapter defines — `token`, `api_key`, `workspace_url`, etc.: - -```bash -export DJ_STORES='{ - "uc": { - "protocol": "databricks", - "workspace_url": "https://my-workspace.cloud.databricks.com", - "volume": "main.default.my_volume", - "token": "dapibd..." - } -}' -``` - -### Precedence - -`DJ_STORES`, if set, replaces the `stores` block loaded from `datajoint.json` wholesale. The `.secrets/` directory still runs after `DJ_STORES` and fills in any attributes that `DJ_STORES` omits — useful if a deployment wants to inject only secrets via env vars while leaving non-sensitive store config in a file. - -| Source | Priority | -|--------|----------| -| `dj.config["stores"][...]` (programmatic) | 1 (highest) | -| `DJ_STORES` env var | 2 | -| `datajoint.json` `stores` block | 3 | -| `.secrets/stores..` files | 4 (fills missing attrs only) | - -### Errors - -If `DJ_STORES` is set but unparsable, DataJoint raises `ValueError` at config load time with the JSON error, rather than failing later with a confusing `KeyError` from a half-loaded store. - -```python -ValueError: DJ_STORES contains invalid JSON: Expecting property name enclosed in double quotes... -``` - -## `DJ_IGNORE_CONFIG_FILE` — skip files entirely - -!!! version-added "New in 2.2.4" - Set `DJ_IGNORE_CONFIG_FILE=true` to skip `datajoint.json` and the secrets directory. - -For env-var-only deployments — Kubernetes pods, Lambda functions, the DataJoint platform — set: - -```bash -export DJ_IGNORE_CONFIG_FILE=true -``` - -When `true`, DataJoint skips: - -- the recursive parent-directory search for `datajoint.json` -- the project `.secrets/` directory -- the Docker/Kubernetes `/run/secrets/datajoint/` directory - -Only env vars (`DJ_HOST`, `DJ_USER`, `DJ_PASS`, `DJ_STORES`, …) and defaults apply. This guarantees that no stray file in a container image can leak into config. - -| Variable | Values | Default | Description | -|----------|--------|---------|-------------| -| `DJ_IGNORE_CONFIG_FILE` | `true`, `1`, `yes` / `false`, `0`, `no` | `false` | Skip file-based config sources | - -## `.secrets/stores..` accepts any attribute - -!!! version-added "New in 2.2.4" - Any `.secrets/stores..` file loads into `dj.config["stores"][][]`, not just `access_key` / `secret_key`. - -Previously, only `.secrets/stores..access_key` and `.secrets/stores..secret_key` were honored. Plugin-registered adapters often need other field names — a Databricks adapter wants a Bearer `token`, an HTTP adapter might want `api_key`, etc. - -In 2.2.4, any file matching `stores..` under the secrets directory is loaded: - -``` -.secrets/ -├── stores.uc.token # Databricks Bearer token -├── stores.main.access_key # S3 access key -└── stores.main.secret_key # S3 secret key -``` - -Config-file values and `DJ_STORES` still take precedence — secrets only fill attributes that are not already set. - -## Storage-adapter plugin contract - -!!! version-added "New in 2.2.4" - The `datajoint.storage` entry-point group is now part of the public API. - -Third-party packages can register additional storage protocols (Databricks Unity Catalog Volumes, custom HTTP-based stores, lab archive systems) by declaring an entry point. The built-in `file`, `s3`, `gcs`, and `azure` protocols continue to be served by the existing internal dispatch in `StorageBackend`; migrating them onto the public adapter contract is tracked separately. - -```toml -# pyproject.toml of a plugin package -[project.entry-points."datajoint.storage"] -databricks = "dj_databricks:DatabricksVolumesAdapter" -``` - -Once installed, the protocol name (`databricks` in the example) is accepted in any `stores..protocol` field, and DataJoint will use the adapter to construct the underlying `fsspec` filesystem. - -See [Storage Adapter API](../reference/specs/storage-adapter-api.md) for the full plugin contract. - -## See Also - -- [What's New in 2.2](whats-new-22.md) — Previous release (isolated instances, thread-safe mode, graph-driven cascade) -- [Release Notes (v2.2.4)](https://github.com/datajoint/datajoint-python/releases/tag/v2.2.4) — GitHub changelog -- [Manage Secrets](../how-to/manage-secrets.md) — Updated for `DJ_STORES` and `DJ_IGNORE_CONFIG_FILE` -- [Configure Object Storage](../how-to/configure-storage.md) — Env-var-only deployments -- [Storage Adapter API](../reference/specs/storage-adapter-api.md) — Plugin contract -- [Configuration Reference](../reference/configuration.md) — Full env-var table -- [datajoint-python PR #1452](https://github.com/datajoint/datajoint-python/pull/1452) — Implementation diff --git a/src/about/whats-new-22.md b/src/about/whats-new-22.md index e17a91f1..d22e5098 100644 --- a/src/about/whats-new-22.md +++ b/src/about/whats-new-22.md @@ -6,6 +6,15 @@ DataJoint 2.2 introduces **isolated instances** and **thread-safe mode** for app > **Citation:** Yatsenko D, Nguyen TT. *DataJoint 2.0: A Computational Substrate for Agentic Scientific Workflows.* arXiv:2602.16585. 2026. [doi:10.48550/arXiv.2602.16585](https://doi.org/10.48550/arXiv.2602.16585) +## Changes in 2.2.4 + +2.2.4 is a patch release on the 2.2 line, adding **env-var-only configuration of object storage** and a **public plugin contract for third-party storage protocols**. If you are upgrading from **2.2.0–2.2.3** there are no breaking changes — the new environment variables are purely additive. + +- **`DJ_STORES`** — configure object stores entirely from the environment. The variable holds a JSON object identical to the `stores` block of `datajoint.json`, so config moves between file and env var by copy-paste. See [Manage Secrets](../how-to/manage-secrets.md) and [Configure Object Storage](../how-to/configure-storage.md). +- **`DJ_IGNORE_CONFIG_FILE`** — set to `true` to skip `datajoint.json` and the `.secrets/` and `/run/secrets/datajoint/` directories entirely, so only environment variables and defaults apply. Intended for env-var-only deployments (Kubernetes, Lambda, the DataJoint platform). +- **`.secrets/stores..` accepts any attribute**, not just `access_key` / `secret_key` — plugin-registered adapters can supply their own fields (a Bearer `token`, an `api_key`, etc.). +- **Storage-adapter plugin contract.** The `datajoint.storage` entry-point group is now part of the public API: third-party packages register additional storage protocols by subclassing `dj.StorageAdapter` and declaring an entry point. Built-in `file`/`s3`/`gcs`/`azure` are unaffected. See [Storage Adapter API](../reference/specs/storage-adapter-api.md). + ## Overview DataJoint has traditionally used a global singleton pattern: one configuration (`dj.config`), one connection (`dj.conn()`), shared across all tables in a process. This works well for interactive sessions and single-user scripts, but breaks down when: