diff --git a/docs/decisions/ADR-15_Mock_OIDC_Issuer_for_aipf.md b/docs/decisions/ADR-15_Mock_OIDC_Issuer_for_aipf.md new file mode 100644 index 00000000..20e6a107 --- /dev/null +++ b/docs/decisions/ADR-15_Mock_OIDC_Issuer_for_aipf.md @@ -0,0 +1,122 @@ +# ADR-15 Mock OIDC Issuer for aipf_mcp_data_grdm / aipf_mcp_data_grdm 用の模擬 OIDC 発行者 + +> **Summary:** aipf_mcp_data_grdm accepts only bearer tokens of the AI platform's OIDC issuer (Keycloak) and reaches GRDM with the user's own stored personal access token. To run and test it offline in the `full` profile we add `oidc_mock`, our own tiny issuer image that publishes the discovery document and the public key of a signing key that `secrets init` generates; the acceptance test S3 signs its tokens with the same key. Keycloak was rejected as heavier than any core service, as in ADR-05 option B. +> **概要:** aipf_mcp_data_grdm は AI 基盤の OIDC 発行者(Keycloak)のベアラートークンだけを受け付け、GRDM には利用者本人が預けた個人用アクセストークンで入る。これを `full` プロファイルでオフラインのまま動かし試すために、`secrets init` が作る署名鍵の公開部分と発見文書を公開するだけの、自前の小さな発行者イメージ `oidc_mock` を足す。受入試験 S3 は同じ鍵で自分のトークンに署名する。Keycloak は ADR-05 の選択肢 B と同じく、コアのどのサービスより重いので採らなかった。 + +**Jump:** [English](#english) · [日本語](#japanese) + +- Version / 版: 0.2 — 2026-11-16. **Learned:** an issuer that signs nothing needs no private key: the first version passed the signing key to `oidc_mock`, which only publishes its public part, so the installer now derives `OIDC_MOCK_PUBLIC_JWK` and the container gets that alone (PR #86 review r1). / 何にも署名しない発行者に秘密鍵は要らない。最初の版は公開部分を配るだけの `oidc_mock` に署名鍵を渡していた。インストーラーは `OIDC_MOCK_PUBLIC_JWK` を導き、コンテナにはそれだけを渡すようにした(PR #86 レビュー r1)。 + 0.1 — 2026-11-16. **Learned:** a component that trusts an identity provider cannot be tested by handing it the credential of the system it writes to: aipf_mcp_data_grdm verifies an issuer's signature, actor and audience, and keeps the GRDM token only as a stored downstream credential, so the test environment needs an issuer whose key the test can use, and a public key plus two JSON documents is all such an issuer has to serve. / ID プロバイダを信頼する部品は、書き込み先のシステムの資格情報を渡しても試せない。aipf_mcp_data_grdm は発行者の署名・代理者・宛先を確かめ、GRDM のトークンは預かった下流の資格情報として持つだけである。だから試験環境には、試験が鍵を使える発行者が要り、その発行者が配るべきものは公開鍵と二つの JSON 文書だけだった。 +- Status: Proposed for the owner's approval, 2026-11-16 (M3, owner decision of 2026-11-16 「模擬の OIDC 発行者+書き込みの試験」). Review date: when aipf_mcp_data_grdm calls its token-exchange endpoint on a path the acceptance suite exercises, or at M4. +- Records a decision for: `manifest/upstreams.json` (`oidc-mock`) · `installer/images/oidc-mock/` · `installer/grdm_dev/templates/oidc_mock.json`, `aipf_mcp_data_grdm.json` · `installer/grdm_dev/es256.py`, `secrets.py` · [SPEC-04 §5](../spec/04_acceptance_tests.md#english) (S3) · [SPEC-02 §4](../spec/02_installer_cli.md#english) +- Related: [ADR-05 Test IdP for Shibboleth](ADR-05_Test_IdP_for_Shibboleth.md) (the same choice for SAML) · [ADR-02 Network model](ADR-02_Network_Model.md) · [ADR-07 Native builds of EOL middleware](ADR-07_Native_Builds_of_EOL_Middleware.md) (`image_build`) · [Status board](../status/README.md#english) + +--- + + +## English + +[→ 日本語版へ](#japanese) + +### 1. Context + +`aipf_mcp_data_grdm` (RDM-aipf_mcp_data_grdm-dev2027, `full` profile) is the AI platform's component that writes research records into GRDM. The M3 survey of 2026-11-15 found that it restarted in a loop in the `full` profile (`OIDC_ISSUER, OIDC_AUDIENCE, GRDM_VAULT_KEY を設定してください`) and that SPEC-04 §5 S3 described a read tool it does not have. What it really does: + +1. Every request to `/mcp` carries `Authorization: Bearer `. The component fetches the issuer's JWKS (`OIDC_JWKS_URL`, by default Keycloak's `/protocol/openid-connect/certs`), checks the signature (RS256, ES256 or PS256), `iss`, `aud`, `exp`, `sub`, and the acting agent: `act.sub` must be in `ALLOWED_ACTORS`, or, without `act`, `azp` must be in `ALLOWED_DIRECT_CLIENTS` (403 `actor_missing` / `actor_not_allowed`). Writing tools also need `act.sub` in `WRITE_ACTORS`. +2. GRDM is reached with the user's own personal access token (PAT), which the user sends to `POST /grdm/token?t=
` (the form token comes from the `grdm_token_form` tool and is bound to the token's `sub`); the component checks it with GRDM's `/v2/users/me/` and keeps it encrypted with `GRDM_VAULT_KEY` (Fernet). +3. Its token-exchange client (`OIDC_TOKEN_URL`) exists in the code but no tool calls it. + +The environment is offline on the internal network ([ADR-02](ADR-02_Network_Model.md)), runs on arm64 and amd64, and every service must be non-root, read-only, digest-pinned and small ([SPEC-03 §3](../spec/03_security_gate.md#english)). A token's issuer must therefore be part of the stack, and the acceptance test must be able to obtain tokens for user_a, user_b, the research-flow service and an unknown actor without a browser login at the issuer. + +### 2. Options considered + +```toon +options[3]{id,option,offline,arm64_and_amd64,non_root_read_only,image_size,how_the_test_gets_tokens,verdict}: + A,"Keycloak (quay.io/keycloak/keycloak) with an imported realm, clients and token exchange",yes,yes,"uid 1000; read_only needs several volumes","~600 MB + 1 GB heap, start 40-90 s","client credentials + token exchange per actor: realm JSON with one client per actor and exchange permissions",rejected + B,"Turn verification off in aipf_mcp_data_grdm (a test switch in RDM-aipf_mcp_data_grdm-dev2027)",yes,yes,yes,"0 MB","any string",rejected + C,"Own image `oidc-mock`: one standard-library Python file on python:3.13-slim by digest, serving discovery + JWKS of a key from `secrets init`; the test signs with the same key",yes,"yes (official multi-arch base)","yes (uid 65534, no writable path)","~43 MB, start < 1 s","signs a JWT with the key from runtime/.env (ES256), any sub/act/azp/aud/exp",chosen +``` + +- **A** is the real thing, and the reason ADR-05 rejected it for SAML holds here too: Keycloak costs more memory than `web` and starts more slowly than the whole `core` profile, for a fixture that must publish one key. Each actor of the test would need its own client and exchange permission in a realm file that nobody on the team reads. Rejected. +- **B** would test nothing that matters: the actor and audience rules are what S3 has to show, and a switch that turns them off in a component we do not own would also exist in production builds. Rejected. +- **C** gives the component exactly what it reads (the JWKS at Keycloak's path, so even the default `OIDC_JWKS_URL` matches) and gives the test full control of the claims, including tokens the component must refuse (another key, another audience, no actor, an unknown actor). **Chosen.** + +### 3. Decision + +1. **Component and image.** A manifest component `oidc-mock` (role `infra`, profile `full`, both platforms, `image_build` with context `installer/images/oidc-mock`, tag `2`) provides the service `oidc_mock`. The image is `python:3.13-slim` pinned by its index digest plus `oidc_mock.py`; nothing is installed; `USER 65534:65534`; `read_only: true` with no tmpfs or volume. It serves `GET /.well-known/openid-configuration`, `GET /protocol/openid-connect/certs` and `GET /healthz`; anything else is 404 and any other method 405. It has no token, authorize or token-exchange endpoint, because no path S3 exercises calls one. +2. **Network.** `oidc_mock` is not published and joins only the internal network `grdm`. The issuer is `http://oidc_mock:8088/realms/grdm-dev`, the same string inside every container; the test never calls it. +3. **Key.** `secrets init` generates `OIDC_MOCK_SIGNING_KEY`, the base64url (no padding) of an EC P-256 private JWK whose `kid` is its RFC 7638 thumbprint (`installer/grdm_dev/es256.py`, standard library only, because the installer has no third-party dependency). ES256 was chosen over RS256 because a P-256 key is generated in milliseconds without a big-number library, while pure-Python RSA key generation would slow every `secrets init` and every unit test that calls it. `oidc_mock` signs nothing, so it does not get the private key: `secrets init`, `init --missing`, `secrets rotate` and `compose render` derive `OIDC_MOCK_PUBLIC_JWK` (the members `kty`, `crv`, `x`, `y`, `kid`, base64url) from the signing key and write it to `runtime/.env`, and the container receives only `${OIDC_MOCK_PUBLIC_JWK}`; it refuses a value with a private member `d`. Only the acceptance test reads the private key. `secrets rotate` replaces both, and the recreate gives the issuer the new public key; `compose render` and `secrets init` refuse a signing key that is not such a key, naming the key, never the value. +4. **aipf_mcp_data_grdm.** The template sets `OIDC_ISSUER`, `OIDC_AUDIENCE=aipf-mcp-grdm`, `OIDC_JWKS_URL`, `ALLOWED_ACTORS=grdm-dev-agent,grdm-dev-research-flow`, `ALLOWED_DIRECT_CLIENTS=grdm-dev-webui`, `WRITE_ACTORS=grdm-dev-research-flow`, `GRDM_API_BASE=http://api:8000/v2/`, `GRDM_VAULT_KEY` (Fernet, from `secrets init`), `GRDM_VAULT_DB` on the tmpfs `/tmp` (the image's `/data` belongs to nobody and the root is read-only; no stored token survives a restart), `MCP_HOST=0.0.0.0`, `MCP_PORT=8768`, a health check on `/healthz`, and publishes `127.0.0.1:8768`. The three client names are fixtures of this environment, not the AI platform's. +5. **Acceptance.** S3 reads `OIDC_MOCK_SIGNING_KEY` from `runtime/.env` (the same file that gives it the ports) and the issuer, audience and client names from the installed template, so test and stack cannot drift apart. No token is put into evidence. + +### 4. Consequences + +- **Good:** aipf_mcp_data_grdm no longer restarts in a loop in the `full` profile, and S3 can show the component's real contract: the actor and audience rules, the PAT form, a write that GRDM reports, and a refusal on another user's project. +- **Good:** the fixture obeys the baseline controls (non-root, read-only, pinned, internal) without an exception in `manifest/hardening.json`. +- **Good:** the private key stays in `runtime/.env` and the test; the issuer container holds only the public key it publishes (PR #86 review r1). +- **Cost:** two new secrets (`GRDM_VAULT_KEY`, `OIDC_MOCK_SIGNING_KEY`) and one derived value (`OIDC_MOCK_PUBLIC_JWK`). An installation whose `runtime/.env` predates them runs `grdm-dev secrets init --missing` before `compose render`, as for ADR-14. +- **Cost:** we own a small ECDSA implementation. It is checked against RFC 7515 Appendix A.3, against `cryptography`/PyJWT in the aipf image (30 tokens verified, tampering refused, the public point equal), and on the live stack by aipf itself. It is for a fixture on loopback, not constant-time. +- **Open:** `write_record` uploads through the WaterButler links that the API returns, which are browser-facing (`127.0.0.1:`) and unreachable from the component's container; S3 therefore writes with `ensure_project`. Making `write_record` work needs an internal WaterButler URL in those links or in the component. +- **Open:** the image of RDM-aipf_mcp_data_grdm-dev2027 has a base image without digest (`python:3.12-slim`); that is its repository's matter. Its uvicorn access log printed `/grdm/token?t=`, which can be replayed for 600 s; a pull request to that repository's develop removes the query string from the access log (follow-up, PR #86 review r1 R1-02), and the pin moves after its merge. +- **Not provided:** a login page, refresh tokens, token exchange, or more than one realm. Each would be a new decision. + +--- + + +## 日本語 + +[→ To English](#english) + +### 1. 背景 + +`aipf_mcp_data_grdm`(RDM-aipf_mcp_data_grdm-dev2027、`full` プロファイル)は、研究の記録を GRDM に書く AI 基盤の部品です。2026-11-15 の M3 調査で、`full` プロファイルでは再起動を繰り返すこと(`OIDC_ISSUER, OIDC_AUDIENCE, GRDM_VAULT_KEY を設定してください`)と、SPEC-04 §5 の S3 が、この部品に無い読み取りの道具を前提にしていることが分かりました。実際の働きは次のとおりです。 + +1. `/mcp` への要求はどれも `Authorization: Bearer ` を持つ。部品は発行者の JWKS(`OIDC_JWKS_URL`。既定は Keycloak の `/protocol/openid-connect/certs`)を取り、署名(RS256・ES256・PS256)、`iss`・`aud`・`exp`・`sub`、代わりに動くエージェントを確かめる。`act.sub` が `ALLOWED_ACTORS` にあるか、`act` が無ければ `azp` が `ALLOWED_DIRECT_CLIENTS` にあること(なければ 403 `actor_missing` / `actor_not_allowed`)。書く道具は、さらに `act.sub` が `WRITE_ACTORS` にあることを求める。 +2. GRDM には利用者本人の個人用アクセストークン(PAT)で入る。利用者は PAT を `POST /grdm/token?t=<合い言葉>` に送り(合い言葉は道具 `grdm_token_form` が返し、トークンの `sub` に結び付く)、部品は GRDM の `/v2/users/me/` で確かめてから `GRDM_VAULT_KEY`(Fernet)で暗号化して預かる。 +3. トークン交換の口(`OIDC_TOKEN_URL`)はコードにあるが、どの道具も呼ばない。 + +環境は内部ネットワークの上でオフラインです([ADR-02](ADR-02_Network_Model.md))。arm64 と amd64 で動き、どのサービスも非 root・読み取り専用・ダイジェスト固定・小さいことが求められます([SPEC-03 §3](../spec/03_security_gate.md#japanese))。したがってトークンの発行者はスタックの一部でなければならず、受入試験は発行者でのブラウザのログインなしに、user_a・user_b・研究フローのサービス・未知の代理者のトークンを得られなければなりません。 + +### 2. 検討した選択肢 + +```toon +options[3]{id,option,offline,arm64_and_amd64,non_root_read_only,image_size,how_the_test_gets_tokens,verdict}: + A,"Keycloak(quay.io/keycloak/keycloak)にレルム・クライアント・トークン交換を取り込む",yes,yes,"uid 1000。read_only には複数のボリュームが要る","約 600 MB+ヒープ 1 GB、起動 40〜90 秒","代理者ごとにクライアント資格情報+トークン交換。代理者ごとのクライアントと交換の許可を持つレルム JSON",不採用 + B,"aipf_mcp_data_grdm の確かめを切る(RDM-aipf_mcp_data_grdm-dev2027 に試験用のスイッチ)",yes,yes,yes,"0 MB","任意の文字列",不採用 + C,"自前イメージ `oidc-mock`。ダイジェスト固定の python:3.13-slim に標準ライブラリだけの Python ファイル一つ。`secrets init` の鍵の発見文書と JWKS を配り、試験は同じ鍵で署名する",yes,"yes(公式のマルチアーキテクチャのベース)","yes(uid 65534、書き込むパスなし)","約 43 MB、起動 1 秒未満","runtime/.env の鍵で JWT に署名(ES256)。sub・act・azp・aud・exp は自由",採用 +``` + +- **A** は本物です。ADR-05 が SAML で退けた理由はここにも当てはまります。Keycloak は `web` より多くのメモリを使い、`core` プロファイル全体より遅く起動しますが、この部品が配るべきものは鍵一つです。試験の代理者ごとに、チームの誰も読まないレルムファイルにクライアントと交換の許可が要ります。不採用。 +- **B** では肝心なことを何も試せません。S3 が示すべきは代理者と宛先の規則であり、それを切るスイッチは、私たちが持たない部品の本番のビルドにも残ります。不採用。 +- **C** は部品が読むものをそのまま与え(JWKS を Keycloak のパスに置くので、既定の `OIDC_JWKS_URL` でも合う)、試験には主張を全部任せます。部品が拒むべきトークン(別の鍵、別の宛先、代理者なし、未知の代理者)も作れます。**採用。** + +### 3. 決定 + +1. **部品とイメージ。** manifest の部品 `oidc-mock`(役割 `infra`、プロファイル `full`、両プラットフォーム、`image_build` の context は `installer/images/oidc-mock`、タグ `2`)がサービス `oidc_mock` を持つ。イメージはインデックスのダイジェストで固定した `python:3.13-slim` に `oidc_mock.py` を足しただけで、何もインストールしない。`USER 65534:65534`、`read_only: true` で tmpfs もボリュームも無い。`GET /.well-known/openid-configuration`、`GET /protocol/openid-connect/certs`、`GET /healthz` に答え、ほかは 404、ほかのメソッドは 405。S3 が通るどの道もトークン・認可・トークン交換の口を呼ばないので、それらは持たない。 +2. **ネットワーク。** `oidc_mock` は公開せず、内部ネットワーク `grdm` にだけつなぐ。issuer は `http://oidc_mock:8088/realms/grdm-dev` で、どのコンテナの中でも同じ文字列になる。試験はこれを呼ばない。 +3. **鍵。** `secrets init` が `OIDC_MOCK_SIGNING_KEY` を作る。EC P-256 の秘密 JWK(`kid` は RFC 7638 の拇印)を base64url(埋め草なし)にしたもの(`installer/grdm_dev/es256.py`。インストーラーは第三者の依存を持たないので標準ライブラリだけ)。RS256 でなく ES256 にしたのは、P-256 の鍵は大きな数のライブラリなしに数ミリ秒で作れるが、Python だけの RSA の鍵生成は `secrets init` と、それを呼ぶ単体試験のすべてを遅くするからである。`oidc_mock` は何にも署名しないので、秘密鍵を受け取らない。`secrets init`・`init --missing`・`secrets rotate`・`compose render` が署名鍵から `OIDC_MOCK_PUBLIC_JWK`(`kty`・`crv`・`x`・`y`・`kid` の base64url)を導いて `runtime/.env` に書き、コンテナには `${OIDC_MOCK_PUBLIC_JWK}` だけを渡す。秘密の要素 `d` を持つ値は拒む。秘密鍵を読むのは受入試験だけである。`secrets rotate` は両方を替え、再作成で発行者は新しい公開鍵を得る。`compose render` と `secrets init` は、そうした鍵でない署名鍵を、値でなく鍵の名前を示して拒む。 +4. **aipf_mcp_data_grdm。** テンプレートは `OIDC_ISSUER`、`OIDC_AUDIENCE=aipf-mcp-grdm`、`OIDC_JWKS_URL`、`ALLOWED_ACTORS=grdm-dev-agent,grdm-dev-research-flow`、`ALLOWED_DIRECT_CLIENTS=grdm-dev-webui`、`WRITE_ACTORS=grdm-dev-research-flow`、`GRDM_API_BASE=http://api:8000/v2/`、`GRDM_VAULT_KEY`(Fernet、`secrets init` が作る)、tmpfs の `/tmp` に置く `GRDM_VAULT_DB`(イメージの `/data` は nobody の所有でルートは読み取り専用。預かったトークンは再起動で消える)、`MCP_HOST=0.0.0.0`、`MCP_PORT=8768`、`/healthz` の健康確認を設定し、`127.0.0.1:8768` に公開する。三つのクライアント名はこの環境の試験用で、AI 基盤のものではない。 +5. **受入。** S3 は `OIDC_MOCK_SIGNING_KEY` を `runtime/.env`(ポートを読むのと同じファイル)から、issuer・宛先・クライアント名を導入済みのテンプレートから読むので、試験とスタックが食い違わない。トークンは証跡に入れない。 + +### 4. 結果 + +- **良い点:** aipf_mcp_data_grdm は `full` プロファイルで再起動を繰り返さなくなり、S3 は部品の本当の契約を示せる。代理者と宛先の規則、PAT の欄、GRDM が報告する書き込み、他人のプロジェクトでの拒否。 +- **良い点:** この部品は基本対策(非 root・読み取り専用・固定・内部)に従い、`manifest/hardening.json` に例外が要らない。 +- **良い点:** 秘密鍵は `runtime/.env` と試験にとどまり、発行者のコンテナは自分が公開する公開鍵だけを持つ(PR #86 レビュー r1)。 +- **費用:** 秘密が二つ(`GRDM_VAULT_KEY`、`OIDC_MOCK_SIGNING_KEY`)と、導かれる値が一つ(`OIDC_MOCK_PUBLIC_JWK`)増える。これより前に作られた `runtime/.env` の導入は、ADR-14 のときと同じく、`compose render` の前に `grdm-dev secrets init --missing` を実行する。 +- **費用:** 小さな ECDSA の実装を自分たちで持つ。RFC 7515 付録 A.3、aipf のイメージの `cryptography`/PyJWT(30 個のトークンを検証、改ざんは拒否、公開点も一致)、live stack での aipf 自身で確かめた。ループバック上の試験用の部品のためのもので、定数時間ではない。 +- **未了:** `write_record` は API が返す WaterButler のリンクにアップロードするが、そのリンクはブラウザ向け(`127.0.0.1:`)で、部品のコンテナからは届かない。そのため S3 は `ensure_project` で書く。`write_record` を動かすには、そのリンクか部品の側に内部の WaterButler の URL が要る。 +- **未了:** RDM-aipf_mcp_data_grdm-dev2027 のイメージのベース(`python:3.12-slim`)はダイジェストで固定されていない。これはそのリポジトリの問題である。その uvicorn のアクセスログは `/grdm/token?t=<合い言葉>` を出し、600 秒の間は再利用できた。そのリポジトリの develop への PR がアクセスログから問い合わせ文字列を消す(追跡事項、PR #86 レビュー r1 R1-02)。固定版はそのマージの後に動かす。 +- **提供しないもの:** ログイン画面、リフレッシュトークン、トークン交換、複数のレルム。どれも新しい決定になる。 + +--- + +## Glossary / 用語 + +| Term | English note | 日本語注 | +|---|---|---| +| OIDC issuer | The server that signs identity tokens (OpenID Connect); relying parties check its signature with its published keys. | 身元のトークンに署名するサーバ(OpenID Connect)。受け取る側は、公開された鍵で署名を確かめる。 | +| JWKS | JSON Web Key Set: the document in which an issuer publishes its public keys. | JSON Web Key Set。発行者が公開鍵を載せて配る文書。 | +| JWT | JSON Web Token: a signed, compact token carrying claims such as `sub`, `aud` and `exp`. | JSON Web Token。`sub`・`aud`・`exp` などの主張を運ぶ、署名付きの短いトークン。 | +| PAT | Personal access token: a GRDM token a user creates for programs that act on the user's behalf. | 個人用アクセストークン。利用者が、自分の代わりに動くプログラムのために GRDM で作るトークン。 | +| Fernet | A symmetric encryption format (AES-CBC with HMAC) of the Python `cryptography` library; its key is 32 random bytes in urlsafe base64. | Python の `cryptography` ライブラリの対称暗号の形式(AES-CBC と HMAC)。鍵は 32 バイトの乱数を urlsafe base64 にしたもの。 | diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 85aa887e..3b398a4d 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -1,11 +1,12 @@ # Decision Records / 決定の記録 -> **Summary:** Documents explaining why we chose one way and not another: integration repository placement, network model, staged security gate, AI autonomy and roles, the test IdP, the wiki-frontend overlay, native builds of EOL middleware, per-OS installer entry points, PostgreSQL 18 with its data upgrade, the retirement of Elasticsearch 2 and MongoDB, and Elasticsearch 8 with TLS and authentication. Each record captures the context, options considered, and final decision. -> **概要:** なぜその道を選び、ほかを選ばなかったかを説明する文書:統合リポジトリの置き場、ネットワーク方式、セキュリティ門の段階化、AI の自律度と役割、テスト IdP、wiki-frontend のオーバーレイ、EOL ミドルウェアのネイティブビルド、OS ごとのインストーラーの入口、PostgreSQL 18 とデータの更新、Elasticsearch 2 と MongoDB の退役、TLS と認証つきの Elasticsearch 8。各記録は文脈、検討した選択肢、最終決定を捉えます。 +> **Summary:** Documents explaining why we chose one way and not another: integration repository placement, network model, staged security gate, AI autonomy and roles, the test IdP, the wiki-frontend overlay, native builds of EOL middleware, per-OS installer entry points, PostgreSQL 18 with its data upgrade, the retirement of Elasticsearch 2 and MongoDB, Elasticsearch 8 with TLS and authentication, and the mock OIDC issuer for aipf_mcp_data_grdm. Each record captures the context, options considered, and final decision. +> **概要:** なぜその道を選び、ほかを選ばなかったかを説明する文書:統合リポジトリの置き場、ネットワーク方式、セキュリティ門の段階化、AI の自律度と役割、テスト IdP、wiki-frontend のオーバーレイ、EOL ミドルウェアのネイティブビルド、OS ごとのインストーラーの入口、PostgreSQL 18 とデータの更新、Elasticsearch 2 と MongoDB の退役、TLS と認証つきの Elasticsearch 8、aipf_mcp_data_grdm 用の模擬 OIDC 発行者。各記録は文脈、検討した選択肢、最終決定を捉えます。 **Jump:** [English](#english) · [日本語](#japanese) -- Version / 版: 0.8 — 2026-11-13. **Learned:** an index that keeps the status inside the purpose text, and only for some rows, cannot be read at a glance: when the owner accepted ADR-08, ADR-12, ADR-13 and ADR-14 together on 2026-11-13 ("一括で承認、M2 完了"), the state of the other records had to be looked up file by file, so both tables now have a status column taken from each record's Status line. / 状態を目的の文の中に、しかも一部の行にだけ書く索引は、一目では読めない。所有者が 2026-11-13 に ADR-08・ADR-12・ADR-13・ADR-14 をまとめて承認したとき(「一括で承認、M2 完了」)、ほかの記録の状態はファイルを一つずつ見なければ分からなかった。そこで両方の表に、各記録の Status 行から取った状態の列を置いた。 +- Version / 版: 0.9 — 2026-11-16. **Learned:** a record numbered after the reserved block keeps the index honest only if the reserved numbers stay listed: ADR-15 (the mock OIDC issuer, proposed with the S3 change of M3) takes the next number after ADR-14, and ADR-09 to ADR-11 stay reserved. / 予約の塊の後に番号を取る記録は、予約の番号を表に残しておくときだけ索引を正直に保つ。ADR-15(M3 の S3 の変更とともに提案する模擬 OIDC 発行者)は ADR-14 の次の番号を取り、ADR-09〜ADR-11 は予約のまま残す。 + 0.8 — 2026-11-13. **Learned:** an index that keeps the status inside the purpose text, and only for some rows, cannot be read at a glance: when the owner accepted ADR-08, ADR-12, ADR-13 and ADR-14 together on 2026-11-13 ("一括で承認、M2 完了"), the state of the other records had to be looked up file by file, so both tables now have a status column taken from each record's Status line. / 状態を目的の文の中に、しかも一部の行にだけ書く索引は、一目では読めない。所有者が 2026-11-13 に ADR-08・ADR-12・ADR-13・ADR-14 をまとめて承認したとき(「一括で承認、M2 完了」)、ほかの記録の状態はファイルを一つずつ見なければ分からなかった。そこで両方の表に、各記録の Status 行から取った状態の列を置いた。 0.7 — 2026-11-12. **Learned:** an index of decision records has to say which numbers are free and which are taken in other trees, so the next record (ADR-14) was numbered by reading this table instead of guessing; a decision that the data tier needed twice (ADR-13's path C) now has its implementation record (ADR-14). / 決定記録の索引は、どの番号が空きでどれが別の木で使われているかを言わなければならない。次の記録(ADR-14)は推測でなく、この表を読んで番号を付けた。データ層が二度要した決定(ADR-13 の道 C)に、実装の記録(ADR-14)ができた。 0.6 — 2026-11-10. **Learned:** a rule like "never needs sudo" has to name what it covers, so giving hosts a bootstrap became an architecture decision (ADR-08) instead of an exception nobody records. / 「sudo を必要としない」のような規則は対象を名指ししなければならない。ホストを準備する段を設けることは、誰も記録しない例外ではなく設計判断(ADR-08)になった。 0.5 — 2026-11-10. **Learned:** a data-tier change is two records, not one: the PostgreSQL move is proposed with its proof (ADR-12), while the removal of Elasticsearch 2 and MongoDB still needs owner choices (ADR-13); ADR-08 to ADR-11 stay reserved or planned for records on other branches. / データ層の変更は一つでなく二つの記録になる。PostgreSQL の移行は証明つきで提案し(ADR-12)、Elasticsearch 2 と MongoDB の撤去はまだ所有者の選択を要する(ADR-13)。ADR-08〜ADR-11 は他のブランチの記録のために予約・予定のまま残す。 @@ -13,7 +14,7 @@ 0.3 — 2026-11-06. **Learned:** a repository that cannot be built alone because its Dockerfile belongs to another tree is an architecture decision (overlay, not service) and needs its own record (ADR-06). / Dockerfile が別の木に属していて単独で組めないリポジトリは、サービスでなくオーバーレイという設計上の決定であり、独自の記録(ADR-06)が要ると分かった。 0.2 — 2026-11-06. **Learned:** a test fixture that logs people in is an architecture choice like any other service, so it needs its own record (ADR-05) rather than a note in a template. / 人をログインさせる試験用の部品も他のサービスと同じ設計上の選択であり、テンプレートの注記ではなく独自の記録(ADR-05)が要ると分かった。 0.1 — 2026-11-05. **Learned:** writing decisions as records turned subjective choices into objective, reviewable documents that explain the reasoning behind each rule. / 決定を記録として残すことで、主観的な選択を客観的でレビュー可能な文書にし、各ルールの背後にある推論を説明できるようになったと分かった。 -- Normative for: [ADR-01 Integration Repository Placement](./ADR-01_Integration_Repository_Placement.md) through [ADR-08 Per-OS Installer Entry Points](./ADR-08_Per_OS_Installer_Entry_Points.md), and [ADR-12 PostgreSQL 18 and Data Directory Upgrade](./ADR-12_PostgreSQL_18_and_Data_Directory_Upgrade.md) [ADR-13 Retiring Elasticsearch 2 and MongoDB](./ADR-13_Retiring_Elasticsearch_2_and_MongoDB.md) and [ADR-14 Elasticsearch 8 with TLS and Authentication](./ADR-14_Elasticsearch_8_with_TLS_and_Authentication.md); ADR-09 to ADR-11 are reserved or planned. +- Normative for: [ADR-01 Integration Repository Placement](./ADR-01_Integration_Repository_Placement.md) through [ADR-08 Per-OS Installer Entry Points](./ADR-08_Per_OS_Installer_Entry_Points.md), and [ADR-12 PostgreSQL 18 and Data Directory Upgrade](./ADR-12_PostgreSQL_18_and_Data_Directory_Upgrade.md) [ADR-13 Retiring Elasticsearch 2 and MongoDB](./ADR-13_Retiring_Elasticsearch_2_and_MongoDB.md) [ADR-14 Elasticsearch 8 with TLS and Authentication](./ADR-14_Elasticsearch_8_with_TLS_and_Authentication.md) and [ADR-15 Mock OIDC Issuer for aipf_mcp_data_grdm](./ADR-15_Mock_OIDC_Issuer_for_aipf.md); ADR-09 to ADR-11 are reserved or planned. - Related: [../README.md](../README.md) · [policy/](../policy/README.md) · [spec/](../spec/README.md) · [reference/REF-01_Glossary.md](./../reference/REF-01_Glossary.md) --- @@ -40,6 +41,7 @@ A *decision record* (ADR) explains why we chose one way and not another. This fo | ADR-12 | [ADR-12_PostgreSQL_18_and_Data_Directory_Upgrade.md](./ADR-12_PostgreSQL_18_and_Data_Directory_Upgrade.md) | PostgreSQL 9.6 → 18.6 (digest-pinned); md5 host auth until CAS has a SCRAM-capable JDBC driver; `grdm-dev db upgrade` dumps and restores with verification and keeps the old directory, `db rollback` puts it back, `up` refuses a mismatched major (exit 15). | **Accepted 2026-11-13** (owner) | | ADR-13 | [ADR-13_Retiring_Elasticsearch_2_and_MongoDB.md](./ADR-13_Retiring_Elasticsearch_2_and_MongoDB.md) | Measured: osf.io search does not run on ES 6.8 or OpenSearch; search off needs a ~50-line patch; mongo serves only sharejs, whose file-page co-editing is what retirement removes. Retirement steps; owner decisions of 2026-11-10: ES 8 port with full-text search, file-page editor on y-websocket. | **Accepted 2026-11-13** (owner) | | ADR-14 | [ADR-14_Elasticsearch_8_with_TLS_and_Authentication.md](./ADR-14_Elasticsearch_8_with_TLS_and_Authentication.md) | Elasticsearch 8.19.24 (official image by digest, kuromoji and icu) replaces ES 2: TLS with the installation's own CA, authentication, scripts off, a `grdm_osf` user limited to `website*`, no published port, read-only root with a tmpfs `config/`; ES 2 data is rebuilt by `invoke migrate_search`. | **Accepted 2026-11-13** (owner) | +| ADR-15 | [ADR-15_Mock_OIDC_Issuer_for_aipf.md](./ADR-15_Mock_OIDC_Issuer_for_aipf.md) | `full` profile: our own `oidc-mock` image (one standard-library Python file on python:3.13-slim by digest, uid 65534, read-only, not published) serves the discovery document and JWKS of an EC P-256 key from `secrets init`, so aipf_mcp_data_grdm verifies tokens offline and acceptance S3 signs its own; Keycloak rejected as heavier than any core service. | Proposed 2026-11-16 | Reserved or planned numbers (no record in this tree yet): @@ -90,6 +92,7 @@ flowchart TD | ADR-12 | [ADR-12_PostgreSQL_18_and_Data_Directory_Upgrade.md](./ADR-12_PostgreSQL_18_and_Data_Directory_Upgrade.md) | PostgreSQL 9.6 → 18.6(ダイジェスト固定)。CAS が SCRAM に対応した JDBC ドライバーを持つまでホスト認証は md5。`grdm-dev db upgrade` は照合つきでダンプと復元を行い旧ディレクトリを残し、`db rollback` はそれを戻し、`up` は版の不一致を拒む(終了 15)。 | **承認 2026-11-13**(所有者) | | ADR-13 | [ADR-13_Retiring_Elasticsearch_2_and_MongoDB.md](./ADR-13_Retiring_Elasticsearch_2_and_MongoDB.md) | 測定: osf.io の検索は ES 6.8 でも OpenSearch でも動かない。検索を切るには約 50 行のパッチが要る。mongo は sharejs にしか使われず、退役で失うのはそのファイルページの共同編集である。退役の手順。2026-11-10 の所有者の判断: 全文検索を保つ ES 8 への移植、ファイルページのエディタの y-websocket への移行。 | **承認 2026-11-13**(所有者) | | ADR-14 | [ADR-14_Elasticsearch_8_with_TLS_and_Authentication.md](./ADR-14_Elasticsearch_8_with_TLS_and_Authentication.md) | Elasticsearch 8.19.24(公式イメージをダイジェストで固定、kuromoji と icu つき)が ES 2 に替わる。導入ごとの CA による TLS、認証、スクリプトなし、`website*` に限った利用者 `grdm_osf`、ポート非公開、tmpfs の `config/` を持つ読み取り専用ルート。ES 2 のデータは `invoke migrate_search` が作り直す。 | **承認 2026-11-13**(所有者) | +| ADR-15 | [ADR-15_Mock_OIDC_Issuer_for_aipf.md](./ADR-15_Mock_OIDC_Issuer_for_aipf.md) | `full` プロファイル: 自前の `oidc-mock` イメージ(ダイジェスト固定の python:3.13-slim に標準ライブラリだけの Python ファイル一つ、uid 65534、読み取り専用、非公開)が `secrets init` の EC P-256 鍵の発見文書と JWKS を配る。aipf_mcp_data_grdm はオフラインでトークンを確かめ、受入 S3 は自分で署名する。Keycloak はコアのどのサービスより重いので不採用。 | 提案 2026-11-16 | 予約済み・予定の番号(この木にはまだ記録が無い): diff --git a/docs/demo/runs/acc-20261115T162657Z-3b9bef/collector.json b/docs/demo/runs/acc-20261115T162657Z-3b9bef/collector.json new file mode 100644 index 00000000..03ee79a3 --- /dev/null +++ b/docs/demo/runs/acc-20261115T162657Z-3b9bef/collector.json @@ -0,0 +1,266 @@ +[ + { + "nodeid": "test_c1_login.py::test_c1_login_success[chromium]", + "id": "C1", + "name": "Login via fakecas", + "case": "success", + "status": "passed", + "duration_ms": 1735, + "http_status": 200, + "evidence": null, + "known_external_blocked": 7 + }, + { + "nodeid": "test_c2_create_project.py::test_c2_create_project_success[chromium]", + "id": "C2", + "name": "Create project", + "case": "success", + "status": "passed", + "duration_ms": 5122, + "http_status": 200, + "evidence": null, + "known_external_blocked": 14 + }, + { + "nodeid": "test_c2_create_project.py::test_c2_create_project_rejection[chromium]", + "id": "C2", + "name": "Create project", + "case": "rejection", + "status": "passed", + "duration_ms": 4180, + "http_status": 403, + "evidence": null, + "known_external_blocked": 8 + }, + { + "nodeid": "test_c3_file_upload.py::test_c3_file_upload_success[chromium]", + "id": "C3", + "name": "Upload and download file", + "case": "success", + "status": "passed", + "duration_ms": 2878, + "http_status": 200, + "evidence": null, + "known_external_blocked": 4 + }, + { + "nodeid": "test_c3_file_upload.py::test_c3_file_upload_rejection[chromium]", + "id": "C3", + "name": "Upload and download file", + "case": "rejection", + "status": "passed", + "duration_ms": 3811, + "http_status": 403, + "evidence": null, + "known_external_blocked": 8 + }, + { + "nodeid": "test_c4_wiki_collab.py::test_c4_wiki_collab_success[chromium]", + "id": "C4", + "name": "Wiki collaborative edit", + "case": "success", + "status": "passed", + "duration_ms": 7023, + "http_status": 200, + "evidence": null, + "known_external_blocked": 18 + }, + { + "nodeid": "test_c4_wiki_collab.py::test_c4_wiki_collab_rejection[chromium]", + "id": "C4", + "name": "Wiki collaborative edit", + "case": "rejection", + "status": "passed", + "duration_ms": 4274, + "http_status": 4401, + "evidence": null, + "known_external_blocked": 13 + }, + { + "nodeid": "test_c5_file_preview.py::test_c5_file_preview_success[chromium]", + "id": "C5", + "name": "File preview", + "case": "success", + "status": "passed", + "duration_ms": 5161, + "http_status": 200, + "evidence": null, + "known_external_blocked": 9 + }, + { + "nodeid": "test_c5_file_preview.py::test_c5_file_preview_rejection[chromium]", + "id": "C5", + "name": "File preview", + "case": "rejection", + "status": "passed", + "duration_ms": 4453, + "http_status": 401, + "evidence": null, + "known_external_blocked": 9 + }, + { + "nodeid": "test_c6_admin.py::test_c6_admin_success[chromium]", + "id": "C6", + "name": "Institution admin screen", + "case": "success", + "status": "passed", + "duration_ms": 1746, + "http_status": 200, + "evidence": null, + "known_external_blocked": 6 + }, + { + "nodeid": "test_c6_admin.py::test_c6_admin_rejection[chromium]", + "id": "C6", + "name": "Institution admin screen", + "case": "rejection", + "status": "passed", + "duration_ms": 733, + "http_status": 403, + "evidence": null, + "known_external_blocked": 3 + }, + { + "nodeid": "test_c7_file_editor.py::test_c7_file_editor_success[chromium]", + "id": "C7", + "name": "File page collaborative edit", + "case": "success", + "status": "passed", + "duration_ms": 8062, + "http_status": 200, + "evidence": null, + "known_external_blocked": 22 + }, + { + "nodeid": "test_c7_file_editor.py::test_c7_file_editor_rejection[chromium]", + "id": "C7", + "name": "File page collaborative edit", + "case": "rejection", + "status": "passed", + "duration_ms": 6479, + "http_status": 4401, + "evidence": null, + "known_external_blocked": 17 + }, + { + "nodeid": "test_s1_y_websocket.py::test_s1_y_websocket_success[chromium]", + "id": "S1", + "name": "y-websocket", + "case": "success", + "status": "passed", + "duration_ms": 8930, + "http_status": 200, + "evidence": null, + "known_external_blocked": 22 + }, + { + "nodeid": "test_s1_y_websocket.py::test_s1_y_websocket_rejection[chromium]", + "id": "S1", + "name": "y-websocket", + "case": "rejection", + "status": "passed", + "duration_ms": 3801, + "http_status": 4401, + "evidence": null, + "known_external_blocked": 9 + }, + { + "nodeid": "test_s2_rdmcp.py::test_s2_rdmcp_success[chromium]", + "id": "S2", + "name": "rdmcp (MCP endpoint)", + "case": "success", + "status": "skipped", + "duration_ms": 0, + "http_status": null, + "evidence": null, + "known_external_blocked": 0 + }, + { + "nodeid": "test_s2_rdmcp.py::test_s2_rdmcp_rejection[chromium]", + "id": "S2", + "name": "rdmcp (MCP endpoint)", + "case": "rejection", + "status": "skipped", + "duration_ms": 0, + "http_status": null, + "evidence": null, + "known_external_blocked": 0 + }, + { + "nodeid": "test_s3_aipf.py::test_s3_aipf_success[chromium]", + "id": "S3", + "name": "aipf_mcp_data_grdm", + "case": "success", + "status": "passed", + "duration_ms": 4814, + "http_status": 200, + "evidence": null, + "known_external_blocked": 3 + }, + { + "nodeid": "test_s3_aipf.py::test_s3_aipf_rejection[chromium]", + "id": "S3", + "name": "aipf_mcp_data_grdm", + "case": "rejection", + "status": "passed", + "duration_ms": 4450, + "http_status": null, + "evidence": null, + "known_external_blocked": 8 + }, + { + "nodeid": "test_s4_oasys.py::test_s4_oasys_success[chromium]", + "id": "S4", + "name": "OASys", + "case": "success", + "status": "passed", + "duration_ms": 5550, + "http_status": 200, + "evidence": null, + "known_external_blocked": 8 + }, + { + "nodeid": "test_s4_oasys.py::test_s4_oasys_rejection[chromium]", + "id": "S4", + "name": "OASys", + "case": "rejection", + "status": "passed", + "duration_ms": 2002, + "http_status": 401, + "evidence": null, + "known_external_blocked": 4 + }, + { + "nodeid": "test_s5_shibboleth.py::test_s5_shibboleth_success[chromium]", + "id": "S5", + "name": "Shibboleth-SP via test IdP", + "case": "success", + "status": "skipped", + "duration_ms": 0, + "http_status": null, + "evidence": null, + "known_external_blocked": 0 + }, + { + "nodeid": "test_s5_shibboleth.py::test_s5_shibboleth_rejection[chromium]", + "id": "S5", + "name": "Shibboleth-SP via test IdP", + "case": "rejection", + "status": "skipped", + "duration_ms": 0, + "http_status": null, + "evidence": null, + "known_external_blocked": 0 + }, + { + "nodeid": "test_c1_login.py::test_c1_login_rejection", + "id": "C1", + "name": "Login via fakecas", + "case": "rejection", + "status": "passed", + "duration_ms": 77, + "http_status": 401, + "evidence": null, + "known_external_blocked": 0 + } +] diff --git a/docs/demo/runs/acc-20261115T162657Z-3b9bef/junit.xml b/docs/demo/runs/acc-20261115T162657Z-3b9bef/junit.xml new file mode 100644 index 00000000..800a69f1 --- /dev/null +++ b/docs/demo/runs/acc-20261115T162657Z-3b9bef/junit.xml @@ -0,0 +1 @@ +installer/acceptance/test_s2_rdmcp.py:80: M3installer/acceptance/test_s2_rdmcp.py:99: M3installer/acceptance/test_s5_shibboleth.py:97: M3installer/acceptance/test_s5_shibboleth.py:121: M3 \ No newline at end of file diff --git a/docs/demo/runs/acc-20261115T162657Z-3b9bef/pytest.log b/docs/demo/runs/acc-20261115T162657Z-3b9bef/pytest.log new file mode 100644 index 00000000..0855a430 --- /dev/null +++ b/docs/demo/runs/acc-20261115T162657Z-3b9bef/pytest.log @@ -0,0 +1,69 @@ +...............ss....ss. [100%] +=============================== warnings summary =============================== +test_c1_login.py::test_c1_login_success[chromium] + test_c1_login.py:11: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_c2_create_project.py::test_c2_create_project_success[chromium] + test_c2_create_project.py:11: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_c2_create_project.py::test_c2_create_project_rejection[chromium] + test_c2_create_project.py:39: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_c3_file_upload.py::test_c3_file_upload_success[chromium] + test_c3_file_upload.py:29: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_c3_file_upload.py::test_c3_file_upload_rejection[chromium] + test_c3_file_upload.py:43: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_c4_wiki_collab.py::test_c4_wiki_collab_success[chromium] + test_c4_wiki_collab.py:55: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_c4_wiki_collab.py::test_c4_wiki_collab_rejection[chromium] + test_c4_wiki_collab.py:85: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_c5_file_preview.py::test_c5_file_preview_success[chromium] + test_c5_file_preview.py:17: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_c5_file_preview.py::test_c5_file_preview_rejection[chromium] + test_c5_file_preview.py:30: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_c6_admin.py::test_c6_admin_success[chromium] + test_c6_admin.py:40: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_c6_admin.py::test_c6_admin_rejection[chromium] + test_c6_admin.py:60: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_c7_file_editor.py::test_c7_file_editor_success[chromium] + test_c7_file_editor.py:45: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_c7_file_editor.py::test_c7_file_editor_rejection[chromium] + test_c7_file_editor.py:75: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_s1_y_websocket.py::test_s1_y_websocket_success[chromium] + test_s1_y_websocket.py:102: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_s1_y_websocket.py::test_s1_y_websocket_rejection[chromium] + test_s1_y_websocket.py:124: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_s3_aipf.py::test_s3_aipf_success[chromium] + test_s3_aipf.py:195: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_s3_aipf.py::test_s3_aipf_rejection[chromium] + test_s3_aipf.py:244: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_s4_oasys.py::test_s4_oasys_success[chromium] + test_s4_oasys.py:47: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_s4_oasys.py::test_s4_oasys_rejection[chromium] + test_s4_oasys.py:77: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_c1_login.py::test_c1_login_rejection + test_c1_login.py:21: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +=========================== short test summary info ============================ +SKIPPED [1] installer/acceptance/test_s2_rdmcp.py:80: M3 +SKIPPED [1] installer/acceptance/test_s2_rdmcp.py:99: M3 +SKIPPED [1] installer/acceptance/test_s5_shibboleth.py:97: M3 +SKIPPED [1] installer/acceptance/test_s5_shibboleth.py:121: M3 +20 passed, 4 skipped, 20 warnings in 85.38s (0:01:25) diff --git a/docs/demo/runs/acc-20261115T162657Z-3b9bef/results.json b/docs/demo/runs/acc-20261115T162657Z-3b9bef/results.json new file mode 100644 index 00000000..16c05e31 --- /dev/null +++ b/docs/demo/runs/acc-20261115T162657Z-3b9bef/results.json @@ -0,0 +1,306 @@ +{ + "run_id": "acc-20261115T162657Z-3b9bef", + "base_url": "http://127.0.0.1:5000", + "started": "2026-11-15T16:26:57Z", + "duration_ms": 108720, + "only": null, + "seeded": true, + "precheck": { + "healthy": true, + "services": 27 + }, + "pytest_returncode": 0, + "evidence_dir": "docs/demo/runs/acc-20261115T162657Z-3b9bef", + "tracing": "off", + "wait_scale": 1.0, + "trace_quarantine": null, + "expected_scenarios": [ + "C1", + "C2", + "C3", + "C4", + "C5", + "C6", + "C7", + "S1", + "S2", + "S3", + "S4", + "S5" + ], + "incomplete_scenarios": [], + "summary": { + "passed": 20, + "failed": 0, + "error": 0, + "skipped": 4, + "total": 24, + "known_external_blocked": 192 + }, + "tests": [ + { + "id": "C1", + "name": "Login via fakecas", + "case": "success", + "test": "test_c1_login_success[chromium]", + "status": "passed", + "duration_ms": 1735, + "http_status": 200, + "evidence": null, + "known_external_blocked": 7 + }, + { + "id": "C2", + "name": "Create project", + "case": "success", + "test": "test_c2_create_project_success[chromium]", + "status": "passed", + "duration_ms": 5122, + "http_status": 200, + "evidence": null, + "known_external_blocked": 14 + }, + { + "id": "C2", + "name": "Create project", + "case": "rejection", + "test": "test_c2_create_project_rejection[chromium]", + "status": "passed", + "duration_ms": 4180, + "http_status": 403, + "evidence": null, + "known_external_blocked": 8 + }, + { + "id": "C3", + "name": "Upload and download file", + "case": "success", + "test": "test_c3_file_upload_success[chromium]", + "status": "passed", + "duration_ms": 2878, + "http_status": 200, + "evidence": null, + "known_external_blocked": 4 + }, + { + "id": "C3", + "name": "Upload and download file", + "case": "rejection", + "test": "test_c3_file_upload_rejection[chromium]", + "status": "passed", + "duration_ms": 3811, + "http_status": 403, + "evidence": null, + "known_external_blocked": 8 + }, + { + "id": "C4", + "name": "Wiki collaborative edit", + "case": "success", + "test": "test_c4_wiki_collab_success[chromium]", + "status": "passed", + "duration_ms": 7023, + "http_status": 200, + "evidence": null, + "known_external_blocked": 18 + }, + { + "id": "C4", + "name": "Wiki collaborative edit", + "case": "rejection", + "test": "test_c4_wiki_collab_rejection[chromium]", + "status": "passed", + "duration_ms": 4274, + "http_status": 4401, + "evidence": null, + "known_external_blocked": 13 + }, + { + "id": "C5", + "name": "File preview", + "case": "success", + "test": "test_c5_file_preview_success[chromium]", + "status": "passed", + "duration_ms": 5161, + "http_status": 200, + "evidence": null, + "known_external_blocked": 9 + }, + { + "id": "C5", + "name": "File preview", + "case": "rejection", + "test": "test_c5_file_preview_rejection[chromium]", + "status": "passed", + "duration_ms": 4453, + "http_status": 401, + "evidence": null, + "known_external_blocked": 9 + }, + { + "id": "C6", + "name": "Institution admin screen", + "case": "success", + "test": "test_c6_admin_success[chromium]", + "status": "passed", + "duration_ms": 1746, + "http_status": 200, + "evidence": null, + "known_external_blocked": 6 + }, + { + "id": "C6", + "name": "Institution admin screen", + "case": "rejection", + "test": "test_c6_admin_rejection[chromium]", + "status": "passed", + "duration_ms": 733, + "http_status": 403, + "evidence": null, + "known_external_blocked": 3 + }, + { + "id": "C7", + "name": "File page collaborative edit", + "case": "success", + "test": "test_c7_file_editor_success[chromium]", + "status": "passed", + "duration_ms": 8062, + "http_status": 200, + "evidence": null, + "known_external_blocked": 22 + }, + { + "id": "C7", + "name": "File page collaborative edit", + "case": "rejection", + "test": "test_c7_file_editor_rejection[chromium]", + "status": "passed", + "duration_ms": 6479, + "http_status": 4401, + "evidence": null, + "known_external_blocked": 17 + }, + { + "id": "S1", + "name": "y-websocket", + "case": "success", + "test": "test_s1_y_websocket_success[chromium]", + "status": "passed", + "duration_ms": 8930, + "http_status": 200, + "evidence": null, + "known_external_blocked": 22 + }, + { + "id": "S1", + "name": "y-websocket", + "case": "rejection", + "test": "test_s1_y_websocket_rejection[chromium]", + "status": "passed", + "duration_ms": 3801, + "http_status": 4401, + "evidence": null, + "known_external_blocked": 9 + }, + { + "id": "S2", + "name": "rdmcp (MCP endpoint)", + "case": "success", + "test": "test_s2_rdmcp_success[chromium]", + "status": "skipped", + "duration_ms": 0, + "http_status": null, + "evidence": null, + "known_external_blocked": 0 + }, + { + "id": "S2", + "name": "rdmcp (MCP endpoint)", + "case": "rejection", + "test": "test_s2_rdmcp_rejection[chromium]", + "status": "skipped", + "duration_ms": 0, + "http_status": null, + "evidence": null, + "known_external_blocked": 0 + }, + { + "id": "S3", + "name": "aipf_mcp_data_grdm", + "case": "success", + "test": "test_s3_aipf_success[chromium]", + "status": "passed", + "duration_ms": 4814, + "http_status": 200, + "evidence": null, + "known_external_blocked": 3 + }, + { + "id": "S3", + "name": "aipf_mcp_data_grdm", + "case": "rejection", + "test": "test_s3_aipf_rejection[chromium]", + "status": "passed", + "duration_ms": 4450, + "http_status": null, + "evidence": null, + "known_external_blocked": 8 + }, + { + "id": "S4", + "name": "OASys", + "case": "success", + "test": "test_s4_oasys_success[chromium]", + "status": "passed", + "duration_ms": 5550, + "http_status": 200, + "evidence": null, + "known_external_blocked": 8 + }, + { + "id": "S4", + "name": "OASys", + "case": "rejection", + "test": "test_s4_oasys_rejection[chromium]", + "status": "passed", + "duration_ms": 2002, + "http_status": 401, + "evidence": null, + "known_external_blocked": 4 + }, + { + "id": "S5", + "name": "Shibboleth-SP via test IdP", + "case": "success", + "test": "test_s5_shibboleth_success[chromium]", + "status": "skipped", + "duration_ms": 0, + "http_status": null, + "evidence": null, + "known_external_blocked": 0 + }, + { + "id": "S5", + "name": "Shibboleth-SP via test IdP", + "case": "rejection", + "test": "test_s5_shibboleth_rejection[chromium]", + "status": "skipped", + "duration_ms": 0, + "http_status": null, + "evidence": null, + "known_external_blocked": 0 + }, + { + "id": "C1", + "name": "Login via fakecas", + "case": "rejection", + "test": "test_c1_login_rejection", + "status": "passed", + "duration_ms": 77, + "http_status": 401, + "evidence": null, + "known_external_blocked": 0 + } + ] +} diff --git a/docs/demo/runs/acc-20261115T162657Z-3b9bef/results.toon b/docs/demo/runs/acc-20261115T162657Z-3b9bef/results.toon new file mode 100644 index 00000000..27b65cbf --- /dev/null +++ b/docs/demo/runs/acc-20261115T162657Z-3b9bef/results.toon @@ -0,0 +1,61 @@ +accept: + run_id: acc-20261115T162657Z-3b9bef + base_url: "http://127.0.0.1:5000" + started: "2026-11-15T16:26:57Z" + duration_ms: 108720 + only: null + seeded: true + precheck: + healthy: true + services: 27 + pytest_returncode: 0 + evidence_dir: docs/demo/runs/acc-20261115T162657Z-3b9bef + tracing: off + wait_scale: 1.0 + trace_quarantine: null + expected_scenarios[12]: + - C1 + - C2 + - C3 + - C4 + - C5 + - C6 + - C7 + - S1 + - S2 + - S3 + - S4 + - S5 + incomplete_scenarios[0]: + summary: + passed: 20 + failed: 0 + error: 0 + skipped: 4 + total: 24 + known_external_blocked: 192 + tests[24]{id,name,case,test,status,duration_ms,http_status,evidence,known_external_blocked}: + C1,Login via fakecas,success,"test_c1_login_success[chromium]",passed,1735,200,null,7 + C2,Create project,success,"test_c2_create_project_success[chromium]",passed,5122,200,null,14 + C2,Create project,rejection,"test_c2_create_project_rejection[chromium]",passed,4180,403,null,8 + C3,Upload and download file,success,"test_c3_file_upload_success[chromium]",passed,2878,200,null,4 + C3,Upload and download file,rejection,"test_c3_file_upload_rejection[chromium]",passed,3811,403,null,8 + C4,Wiki collaborative edit,success,"test_c4_wiki_collab_success[chromium]",passed,7023,200,null,18 + C4,Wiki collaborative edit,rejection,"test_c4_wiki_collab_rejection[chromium]",passed,4274,4401,null,13 + C5,File preview,success,"test_c5_file_preview_success[chromium]",passed,5161,200,null,9 + C5,File preview,rejection,"test_c5_file_preview_rejection[chromium]",passed,4453,401,null,9 + C6,Institution admin screen,success,"test_c6_admin_success[chromium]",passed,1746,200,null,6 + C6,Institution admin screen,rejection,"test_c6_admin_rejection[chromium]",passed,733,403,null,3 + C7,File page collaborative edit,success,"test_c7_file_editor_success[chromium]",passed,8062,200,null,22 + C7,File page collaborative edit,rejection,"test_c7_file_editor_rejection[chromium]",passed,6479,4401,null,17 + S1,y-websocket,success,"test_s1_y_websocket_success[chromium]",passed,8930,200,null,22 + S1,y-websocket,rejection,"test_s1_y_websocket_rejection[chromium]",passed,3801,4401,null,9 + S2,rdmcp (MCP endpoint),success,"test_s2_rdmcp_success[chromium]",skipped,0,null,null,0 + S2,rdmcp (MCP endpoint),rejection,"test_s2_rdmcp_rejection[chromium]",skipped,0,null,null,0 + S3,aipf_mcp_data_grdm,success,"test_s3_aipf_success[chromium]",passed,4814,200,null,3 + S3,aipf_mcp_data_grdm,rejection,"test_s3_aipf_rejection[chromium]",passed,4450,null,null,8 + S4,OASys,success,"test_s4_oasys_success[chromium]",passed,5550,200,null,8 + S4,OASys,rejection,"test_s4_oasys_rejection[chromium]",passed,2002,401,null,4 + S5,Shibboleth-SP via test IdP,success,"test_s5_shibboleth_success[chromium]",skipped,0,null,null,0 + S5,Shibboleth-SP via test IdP,rejection,"test_s5_shibboleth_rejection[chromium]",skipped,0,null,null,0 + C1,Login via fakecas,rejection,test_c1_login_rejection,passed,77,401,null,0 diff --git a/docs/demo/runs/acc-20261115T172743Z-b28b73/collector.json b/docs/demo/runs/acc-20261115T172743Z-b28b73/collector.json new file mode 100644 index 00000000..bf27f841 --- /dev/null +++ b/docs/demo/runs/acc-20261115T172743Z-b28b73/collector.json @@ -0,0 +1,24 @@ +[ + { + "nodeid": "test_s3_aipf.py::test_s3_aipf_success[chromium]", + "id": "S3", + "name": "aipf_mcp_data_grdm", + "case": "success", + "status": "passed", + "duration_ms": 6125, + "http_status": 200, + "evidence": null, + "known_external_blocked": 3 + }, + { + "nodeid": "test_s3_aipf.py::test_s3_aipf_rejection[chromium]", + "id": "S3", + "name": "aipf_mcp_data_grdm", + "case": "rejection", + "status": "passed", + "duration_ms": 4632, + "http_status": null, + "evidence": null, + "known_external_blocked": 8 + } +] diff --git a/docs/demo/runs/acc-20261115T172743Z-b28b73/junit.xml b/docs/demo/runs/acc-20261115T172743Z-b28b73/junit.xml new file mode 100644 index 00000000..ec6caa02 --- /dev/null +++ b/docs/demo/runs/acc-20261115T172743Z-b28b73/junit.xml @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/demo/runs/acc-20261115T172743Z-b28b73/pytest.log b/docs/demo/runs/acc-20261115T172743Z-b28b73/pytest.log new file mode 100644 index 00000000..1e895083 --- /dev/null +++ b/docs/demo/runs/acc-20261115T172743Z-b28b73/pytest.log @@ -0,0 +1,10 @@ +.. [100%] +=============================== warnings summary =============================== +test_s3_aipf.py::test_s3_aipf_success[chromium] + test_s3_aipf.py:209: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +test_s3_aipf.py::test_s3_aipf_rejection[chromium] + test_s3_aipf.py:258: PytestWarning: record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1') + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +2 passed, 2 warnings in 10.79s diff --git a/docs/demo/runs/acc-20261115T172743Z-b28b73/results.json b/docs/demo/runs/acc-20261115T172743Z-b28b73/results.json new file mode 100644 index 00000000..a9dbe036 --- /dev/null +++ b/docs/demo/runs/acc-20261115T172743Z-b28b73/results.json @@ -0,0 +1,53 @@ +{ + "run_id": "acc-20261115T172743Z-b28b73", + "base_url": "http://127.0.0.1:5000", + "started": "2026-11-15T17:27:43Z", + "duration_ms": 35126, + "only": "S3", + "seeded": true, + "precheck": { + "healthy": true, + "services": 27 + }, + "pytest_returncode": 0, + "evidence_dir": "docs/demo/runs/acc-20261115T172743Z-b28b73", + "tracing": "off", + "wait_scale": 1.0, + "trace_quarantine": null, + "expected_scenarios": [ + "S3" + ], + "incomplete_scenarios": [], + "summary": { + "passed": 2, + "failed": 0, + "error": 0, + "skipped": 0, + "total": 2, + "known_external_blocked": 11 + }, + "tests": [ + { + "id": "S3", + "name": "aipf_mcp_data_grdm", + "case": "success", + "test": "test_s3_aipf_success[chromium]", + "status": "passed", + "duration_ms": 6125, + "http_status": 200, + "evidence": null, + "known_external_blocked": 3 + }, + { + "id": "S3", + "name": "aipf_mcp_data_grdm", + "case": "rejection", + "test": "test_s3_aipf_rejection[chromium]", + "status": "passed", + "duration_ms": 4632, + "http_status": null, + "evidence": null, + "known_external_blocked": 8 + } + ] +} diff --git a/docs/demo/runs/acc-20261115T172743Z-b28b73/results.toon b/docs/demo/runs/acc-20261115T172743Z-b28b73/results.toon new file mode 100644 index 00000000..d7c6e448 --- /dev/null +++ b/docs/demo/runs/acc-20261115T172743Z-b28b73/results.toon @@ -0,0 +1,28 @@ +accept: + run_id: acc-20261115T172743Z-b28b73 + base_url: "http://127.0.0.1:5000" + started: "2026-11-15T17:27:43Z" + duration_ms: 35126 + only: S3 + seeded: true + precheck: + healthy: true + services: 27 + pytest_returncode: 0 + evidence_dir: docs/demo/runs/acc-20261115T172743Z-b28b73 + tracing: off + wait_scale: 1.0 + trace_quarantine: null + expected_scenarios[1]: + - S3 + incomplete_scenarios[0]: + summary: + passed: 2 + failed: 0 + error: 0 + skipped: 0 + total: 2 + known_external_blocked: 11 + tests[2]{id,name,case,test,status,duration_ms,http_status,evidence,known_external_blocked}: + S3,aipf_mcp_data_grdm,success,"test_s3_aipf_success[chromium]",passed,6125,200,null,3 + S3,aipf_mcp_data_grdm,rejection,"test_s3_aipf_rejection[chromium]",passed,4632,null,null,8 diff --git a/docs/spec/02_installer_cli.md b/docs/spec/02_installer_cli.md index 28f53b9f..7034a538 100644 --- a/docs/spec/02_installer_cli.md +++ b/docs/spec/02_installer_cli.md @@ -5,7 +5,8 @@ **Jump:** [English](#english) · [日本語](#japanese) -- Version / 版: 0.42 — 2026-11-16. **Learned:** one stack may need two sign-in servers at once: the browser keeps signing in to GRDM through fakecas, but rdmcp's OAuth flow needs the real CAS (`oauth2/authorize` and `oauth2/token`, which fakecas does not have), so the full profile publishes `cas` on 8090 beside `rdmcp` on 8100, `api` validates bearer tokens at the real CAS, and `web` keeps fakecas; both ports are rows of the table and core is unchanged (M3 S2). / 一つのスタックが、二つのサインインのサーバーを同時に要することがある。ブラウザは今までどおり fakecas で GRDM にサインインするが、rdmcp の OAuth の流れには本物の CAS(fakecas に無い `oauth2/authorize` と `oauth2/token`)が要る。そこで full プロファイルは `rdmcp`(8100)の横で `cas` を 8090 に公開し、`api` は Bearer トークンを本物の CAS で確かめ、`web` は fakecas のままにする。二つのポートを表の行にし、core は変えない(M3 S2)。 +- Version / 版: 0.43 — 2026-11-16. **Learned:** a service that starts without its settings is not running, even when `up` sees a container: aipf_mcp_data_grdm (full profile) restarted in a loop until it got an OIDC issuer, a Fernet key and a writable vault path, so it now has them (the mock issuer `oidc_mock`, ADR-15; `GRDM_VAULT_KEY`; `/tmp`), a health check on `/healthz` and the port 8768 as a row of the table, and the two new keys are checked for their format by name. / 設定なしで起動するサービスは、`up` がコンテナを見ていても動いてはいない。aipf_mcp_data_grdm(full プロファイル)は OIDC の発行者、Fernet の鍵、書き込める預かり先を得るまで再起動を繰り返した。そこでそれら(模擬の発行者 `oidc_mock`。ADR-15、`GRDM_VAULT_KEY`、`/tmp`)と `/healthz` の健康確認を与え、ポート 8768 を表の行にした。新しい二つのキーは形式を名前で確かめる。 + 0.42 — 2026-11-16. **Learned:** one stack may need two sign-in servers at once: the browser keeps signing in to GRDM through fakecas, but rdmcp's OAuth flow needs the real CAS (`oauth2/authorize` and `oauth2/token`, which fakecas does not have), so the full profile publishes `cas` on 8090 beside `rdmcp` on 8100, `api` validates bearer tokens at the real CAS, and `web` keeps fakecas; both ports are rows of the table and core is unchanged (M3 S2). / 一つのスタックが、二つのサインインのサーバーを同時に要することがある。ブラウザは今までどおり fakecas で GRDM にサインインするが、rdmcp の OAuth の流れには本物の CAS(fakecas に無い `oauth2/authorize` と `oauth2/token`)が要る。そこで full プロファイルは `rdmcp`(8100)の横で `cas` を 8090 に公開し、`api` は Bearer トークンを本物の CAS で確かめ、`web` は fakecas のままにする。二つのポートを表の行にし、core は変えない(M3 S2)。 0.41 — 2026-11-16. **Learned:** `running` is a sample, not a state of health: a container in a restart loop is `running` between two crashes, and `health` passed one such service in the M3 survey, so a service without a healthcheck now counts only after it has been up 20 s. / `running` は一瞬の見本であって健康の状態ではない。再起動を繰り返すコンテナは二度の異常終了の間は `running` で、M3 の調査では `health` がそのようなサービスを通した。だから healthcheck の無いサービスは、20 秒続けて動いてから数える。 0.40 — 2026-11-15. **Learned:** a front end whose URLs are compiled in still needs only one image: OASys (full profile) is built with placeholder URLs and its entrypoint writes the rendered ports over them under /tmp at start, so `compose render` may move a port without a rebuild; its port 3100 is now a row of the table. / URL を埋め込んで組む画面でも、イメージは一つで足りる。OASys(full プロファイル)は仮の URL で組み、起動時にエントリポイントが描画したポートを /tmp の下でその上に書く。`compose render` がポートを動かしても組み直しは要らない。そのポート 3100 を表の行にした。 0.39 — 2026-11-15. **Learned:** a build and a start are joined only through the rendered file, so a moved pin plus `build` then `up` started the old image with exit 0 (osf.io 90b10661 → 94e68174 on the Mac Studio stack); `up` now compares the rendered images with the manifest pins and stops with exit 13 instead. / build と起動は render したファイルでしかつながっていない。固定版が動いた後に `build`、続けて `up` をすると、古いイメージが終了 0 で起動した(Mac Studio の stack で osf.io 90b10661 → 94e68174)。`up` は render したイメージを manifest の固定版と比べ、食い違えば終了 13 で止まるようにした。 @@ -141,7 +142,7 @@ Rules that every script keeps: |---|---|---| | `doctor` | Detects OS (macOS, Ubuntu, WSL2), CPU (`arm64`, `amd64`), memory, free disk, Docker ≥ 27, Compose plugin (v2 or newer), buildx, and proves what `install` needs so a problem appears before the long build: `git`; read access to the profile's private integration repositories, with `git ls-remote` that can never prompt (its output is never printed, a URL can hold a token); `trivy`, `gitleaks` and `semgrep`; the profile's memory floors against the smaller of the host and the Docker VM (`--memory-gib` turns this into a warning, a knowing overcommit); amd64 emulation (binfmt) on Linux arm64 only, printing the command that enables it, and the report line `amd64_emulation` (`not needed (host is amd64)` on amd64, `ready (...)` on Linux arm64 with a working handler, absent on macOS), and refusing an emulator build measured to crash (Ubuntu 24.04's `qemu-user-static` 8.2.2, read from the registered interpreter's `--version`), printing the command that replaces it; a checkout Docker can use: on WSL2 a filesystem that keeps a `chmod 0600` read back from a test file (a Windows drive cannot hold `runtime/.env` at `0600`); on macOS one the engine can see, asked by bind-mounting it read-only into the `postgres` image that `up` needs (the only image this subcommand may pull; a mount the engine refuses or that comes up empty is exit 13, a probe that could not run is a warning) because a bind mount from a path the Docker VM does not share is silently empty, with `GRDM_DOCKER_SHARED=1` to skip the question when it cannot be run. Prints a TOON report. Exit 10 if the host is unsupported or the memory floors do not fit, 11 if Docker is missing or not running, 13 if a prerequisite is missing, 51 if a scanner is missing. | read-only | | `plan` | Resolves manifest + profile and prints which images would be built and which services started, with the published ports ([§4](#2-en-ports)). Changes nothing. Exit 20 if the manifest fails [SPEC-01](01_manifest.md) checks. | no | -| `secrets init` / `secrets rotate` | `init` writes random secrets (database passwords, Django secret key, CAS keys) into `runtime/.env` with file mode `0600`. Never writes a default or example password. Refuses to overwrite an existing file unless `--force`. `init --missing` is the upgrade for a `runtime/.env` that predates a new key (`ELASTIC8_PASSWORD`, `ELASTIC8_OSF_PASSWORD`, [ADR-14](../decisions/ADR-14_Elasticsearch_8_with_TLS_and_Authentication.md#english)): it keeps every existing value (passwords that live data depends on stay valid), adds only the absent keys, drops retired keys, changes no live password and needs no running stack; it cannot be combined with `--force`. `rotate` is not an upgrade step. It regenerates every secret, changes the live PostgreSQL and RabbitMQ passwords, recreates the containers so they receive the new environment, and verifies that the old credentials are rejected; any failure restores the previous `runtime/.env` and exits 40. `init --force` and `rotate` drop retired credential keys. After an upgrade, run `compose render` before `rotate` or any recreate, because a previously rendered Compose file still refers to retired keys. Four keys have lengths the CAS image needs: `TGC_ENCRYPTION_KEY` is 43 characters, `TGC_SIGNING_KEY` is 86 characters, `OSF_API_INSTITUTIONS_AUTH_JWESECRET` is exactly 32 bytes and `OSF_API_INSTITUTIONS_AUTH_JWTSECRET` is at least 32 bytes; the last two are also osf.io's `JWE_SECRET` and `JWT_SECRET` ([ADR-07 §7](../decisions/ADR-07_Native_Builds_of_EOL_Middleware.md#english)). All four use only the base64url characters `A`–`Z`, `a`–`z`, `0`–`9`, `-` and `_` (no whitespace, quotes or padding), must not be a published default, and `TGC_ENCRYPTION_KEY` must decode canonically to 32 bytes. Used after a red-team capture ([SPEC-03 §6](03_security_gate.md#english)). | no / yes | +| `secrets init` / `secrets rotate` | `init` writes random secrets (database passwords, Django secret key, CAS keys) into `runtime/.env` with file mode `0600`. Never writes a default or example password. Refuses to overwrite an existing file unless `--force`. `init --missing` is the upgrade for a `runtime/.env` that predates a new key (`ELASTIC8_PASSWORD`, `ELASTIC8_OSF_PASSWORD`, [ADR-14](../decisions/ADR-14_Elasticsearch_8_with_TLS_and_Authentication.md#english)): it keeps every existing value (passwords that live data depends on stay valid), adds only the absent keys, drops retired keys, changes no live password and needs no running stack; it cannot be combined with `--force`. `rotate` is not an upgrade step. It regenerates every secret, changes the live PostgreSQL and RabbitMQ passwords, recreates the containers so they receive the new environment, and verifies that the old credentials are rejected; any failure restores the previous `runtime/.env` and exits 40. `init --force` and `rotate` drop retired credential keys. After an upgrade, run `compose render` before `rotate` or any recreate, because a previously rendered Compose file still refers to retired keys. Four keys have lengths the CAS image needs: `TGC_ENCRYPTION_KEY` is 43 characters, `TGC_SIGNING_KEY` is 86 characters, `OSF_API_INSTITUTIONS_AUTH_JWESECRET` is exactly 32 bytes and `OSF_API_INSTITUTIONS_AUTH_JWTSECRET` is at least 32 bytes; the last two are also osf.io's `JWE_SECRET` and `JWT_SECRET` ([ADR-07 §7](../decisions/ADR-07_Native_Builds_of_EOL_Middleware.md#english)). All four use only the base64url characters `A`–`Z`, `a`–`z`, `0`–`9`, `-` and `_` (no whitespace, quotes or padding), must not be a published default, and `TGC_ENCRYPTION_KEY` must decode canonically to 32 bytes. Two keys of the `full` profile have a format of their own, which `secrets init` and `compose render` check by name: `GRDM_VAULT_KEY` is a Fernet key (urlsafe base64 of 32 bytes with padding, 44 characters) with which aipf_mcp_data_grdm encrypts the users' stored GRDM tokens, and `OIDC_MOCK_SIGNING_KEY` is the base64url of an EC P-256 private JWK, which acceptance S3 signs with; `init`, `rotate` and `compose render` derive its public part `OIDC_MOCK_PUBLIC_JWK`, the only key material the `oidc_mock` container receives ([ADR-15](../decisions/ADR-15_Mock_OIDC_Issuer_for_aipf.md#english)). Used after a red-team capture ([SPEC-03 §6](03_security_gate.md#english)). | no / yes | | `compose render` | Exits 2, naming the keys only, when `runtime/.env` lacks a generated key or holds a CAS key that CAS cannot use: wrong length, characters outside base64url, or a published default (a missing key: `secrets init --missing`; an unusable CAS key: `secrets rotate`). Writes `runtime/docker-compose.yml` from manifest + profile: the internal network `grdm` plus the publishing-only `grdm-edge` (`--egress ` attaches more, recorded in `runtime/.env`; [SPEC-03 §3.2](03_security_gate.md#english)), per-component `platform:` from the manifest's `build_platform`, ports published only on `127.0.0.1` ([§4](#2-en-ports)), non-root `user:` for every service, `read_only`, `cap_drop`, and resource limits (`mem_limit`, `cpus`, `pids_limit`) scaled from the smaller of the host memory `doctor` detected and the Docker engine's VM memory (`docker info`; on Docker Desktop the VM is a fraction of the host, real install run 10), where a template's `mem_min_mb` floor takes its share first and the others divide the rest, and from the CPU count the engine reports (`docker info`; the host's only when that is unknown), with per-service exceptions from `manifest/hardening.json` ([SPEC-03 §3.3](03_security_gate.md#english)), `platform:` chosen from `doctor`, named volumes mapped under `runtime/volumes/`, every `tmpfs` entry mounted with `mode=1777` (Compose's short form gives root-only `0755`, real install run 11), and the placeholders `{{HOST_PORT:}}`, `{{CONTAINER_PORT:}}` and `{{AUTH_URL}}` (fakecas's published URL, or web's when `--auth shibboleth` replaces fakecas) resolved in template values. The `image:` of a built component carries the tag of the last build (`runtime/reports/build_latest.json`) only when it equals the tag the manifest pins; a recorded build that differs from the pin is stale (the pin moved after the last `build`), so the pinned tag is rendered instead, the report lists the component under `stale_build_tags` (`component`, `pinned`, `recorded`, `image`), and stderr prints a warning block whose fix is `grdm-dev build`, then `grdm-dev up`. Render still exits 0 because `install` renders before it builds; an image that was never built makes `up` fail instead of running the old one. A component the manifest does not pin keeps the recorded tag. `osf.env` names osf.io's connection to Elasticsearch 8 (`ELASTIC_URI=https://elasticsearch8:9200`, `ELASTIC_CA_CERTS=/es-ca/ca.crt`; the user `grdm_osf` and its password come from the service environment), its internal URL for y-websocket (`Y_WEBSOCKET_INTERNAL_URL`) and `ENABLE_PUBLIC_ACCESS`; web, api, admin, worker and osf_init mount `elasticsearch8_ca` (only the public certificate) read-only. `--public-access true|false` (default: `GRDM_ENABLE_PUBLIC_ACCESS` in `runtime/.env`, else `false`) sets `ENABLE_PUBLIC_ACCESS` for every osf.io process (RDM-osf.io-dev2027 PR #8): `false` requires authentication for every data route (anonymous requests get a redirect to the login or 401; signed WaterButler and MFR callbacks, webhooks and the like keep working), `true` allows anonymous viewing of public projects; the choice is recorded in `runtime/.env` and kept on later renders, like `--egress`. Follows [ADR-02](../decisions/ADR-02_Network_Model.md). | no | | `build` | Builds images with `docker buildx`. The platform is, in order of precedence, `--platform`, the component's `build_platform` in the manifest (for example `linux/amd64` for components whose EOL base cannot be built for arm64, run under emulation), otherwise the host platform; the image is loaded locally. A component with `prebuilt_image` ([SPEC-01](01_manifest.md)) is not built: the digest-pinned image is pulled once per requested platform and reported as `prebuilt` (`pull-failed` → exit 30). With `--push` a prebuilt image is never re-published to `ghcr.io/rcosdp` (it is a third party's artefact); the row is reported as `prebuilt-not-pushed` with `pushed: false`. `--push` pushes the multi-arch manifest to `ghcr.io/rcosdp/` (private). Exit 30 on failure. | yes | | `up` / `down` / `reset` | `up` first finishes or rolls back an interrupted `db` operation, then stops with exit 15 when `runtime/volumes/postgres_data` was written by another PostgreSQL major than the rendered image or its major cannot be read (ADR-12), otherwise starts the profile and waits for health. Before the database check, `up` stops with exit 13 when a volume `grdm-dev_` of the rendered file already exists in Docker and is bound to another directory than this checkout's `runtime/volumes/` (a new clone or a moved directory of the same project: Docker keeps the options of the day the volume was created and Compose reuses it without comparing, so this checkout would open the other one's data with its own new secrets); nothing is started or deleted, and the message names the volumes and the commands that release them (`grdm-dev down` in the other checkout, then `docker volume rm `, which leaves the files on disk). `up` also stops with exit 13 when an image of the rendered file is a local `grdm-dev/:` that the manifest no longer pins (the pin moved after `compose render`; `build` does not rewrite the rendered file), and the message says to run `compose render` (and `build` if the new pin is not built) first. `down` stops containers and keeps volumes; `reset` stops and deletes `runtime/volumes/` after a confirmation (skipped by `--yes`), emptying it through a root container with only `DAC_OVERRIDE` and `FOWNER`, because on Linux the files belong to the containers' uids and the host user cannot remove them. Before anything is stopped or deleted the paths are validated on the host: a symlinked `runtime/` or volumes root, a root that resolves outside the checkout, a symlinked volume, or any mount at `runtime/`, at the volumes root or at any depth inside it (read from the host mount table, so same-filesystem bind mounts count; an unreadable table also refuses) stops with exit 14. `reset` repeats this after `down` and after the wipe, and stops if the volumes root has become a different directory. `compose render` and `up` apply the same validation with exit 13, because the owner step also runs as root against these paths. These are snapshots: Docker resolves the bind later and no check can close that window, so the checkout is assumed not to be changed concurrently by another user or process. | yes | @@ -163,7 +164,7 @@ Rules that every script keeps: profiles[3]{name,services,note}: minimum,"osf_init volume_owner_init volume_access_check_1000 web api postgres elasticsearch8 elasticsearch8_setup rabbitmq fakecas",smallest set that shows a login and a project page core,"minimum + admin worker wb wb_worker mfr unoconv ember_osf_web cas wiki_assets elasticsearch6 redis y_websocket volume_access_check_999","the six core repositories with their infrastructure, plus y_websocket for Wiki collaborative editing; wiki_assets is a one-shot that copies the wiki editor bundle into the volume wiki_static (ADR-06)" - full,"core + rdmcp aipf_mcp_data_grdm oasys shibboleth_sp test_idp",everything; shibboleth_sp replaces fakecas + full,"core + rdmcp aipf_mcp_data_grdm oidc_mock oasys shibboleth_sp test_idp",everything; shibboleth_sp replaces fakecas; oidc_mock is the mock OIDC issuer that aipf_mcp_data_grdm trusts (ADR-15) ``` Service names follow the upstream `docker-compose.yml` of RDM-osf.io so that existing guides keep working, and they are the `services[]` values of the manifest ([SPEC-01 §3](01_manifest.md#english)). The `wiki-frontend` component is an overlay of `osf.io` (`overlay_of`, [ADR-06](../decisions/ADR-06_Wiki_Frontend_Overlay.md)): its service is a *one-shot* (a container that runs once and exits, `restart: "no"`) on the bundle-builder image `grdm-dev/wiki-frontend:-`: after `volume_owner_init` has handed the fresh volume `wiki_static` to uid 1000, `wiki_assets` copies the webpack output `/code/website/static/public/js/` into it and exits; `web`, `api`, `admin` and `worker` mount that volume read-only at `/code/website/static/public/js/wiki/` and depend on it with `condition: service_completed_successfully`, so `up` starts them only after the bundle is in place. `health` counts a one-shot as healthy once it has exited with code 0. The `shibboleth` Compose profile and the `fakecas` service are mutually exclusive: `compose render --profile full --auth shibboleth` removes `fakecas` and adds `shibboleth_sp` and `test_idp`; the default for `full` is `fakecas`. `y_websocket` belongs to `core`, not `full`, because Wiki collaborative editing is an M2 acceptance scenario (C4 in [SPEC-04 §4](04_acceptance_tests.md#english)); the manifest's `profile` for the `y-websocket` component is `core` accordingly ([SPEC-01 §3](01_manifest.md#english)). @@ -173,7 +174,7 @@ Service names follow the upstream `docker-compose.yml` of RDM-osf.io so that exi **Published ports.** This table is the only place where default host ports are defined; [ADR-02](../decisions/ADR-02_Network_Model.md), [SPEC-03](03_security_gate.md), and [REF-01](../reference/REF-01_Glossary.md) point here. Every port is published on `127.0.0.1` only. ```toon -ports[11]{service,host_port,profile,url}: +ports[12]{service,host_port,profile,url}: web,5000,minimum,http://127.0.0.1:5000 api,8000,minimum,http://127.0.0.1:8000 admin,8001,core,http://127.0.0.1:8001 @@ -185,6 +186,7 @@ ports[11]{service,host_port,profile,url}: oasys,3100,full,http://127.0.0.1:3100 cas,8090,full,http://127.0.0.1:8090 rdmcp,8100,full,http://127.0.0.1:8100 + aipf_mcp_data_grdm,8768,full,http://127.0.0.1:8768 ``` All other services (`cas` outside the full profile, `volume_owner_init`, `wiki_assets`, the other subsystems, `test_idp`, databases and queues) are reached only inside the Compose network; when one of them needs a browser-facing port, `compose render` assigns it and `plan` prints it. `compose render` writes every chosen host port to `runtime/.env` as `GRDM_PORT_` (upper case), which the acceptance tests read ([SPEC-04 §1](04_acceptance_tests.md#english)). If a default port is already in use (macOS AirPlay on `5000` is the known case), `compose render` takes the next free port and records the change in the install report; exit 12 is returned only when no free port is found. @@ -387,7 +389,7 @@ flowchart TD |---|---|---| | `doctor` | OS(macOS・Ubuntu・WSL2)、CPU(`arm64`・`amd64`)、メモリ、空き容量、Docker 27 以上、Compose プラグイン(v2 以上)、buildx を判定し、長いビルドの前に問題が分かるよう `install` に要るものを確かめる。`git`、プロファイルの非公開統合リポジトリの読み取り権限(端末で問い合わせない `git ls-remote` で確かめ、URL にトークンが含まれ得るので出力は表示しない)、`trivy`・`gitleaks`・`semgrep`、Docker VM とホストの小さい方に対するプロファイルのメモリ下限(`--memory-gib` で警告に変わる。承知の上の過剰割当)、amd64 エミュレーション(binfmt は Linux arm64 に限る。報告に `amd64_emulation` の行も出す。amd64 では `not needed (host is amd64)`、動く handler のある Linux arm64 では `ready (...)`、macOS では出さない)(有効にするコマンドを表示し、落ちると実測したエミュレーター(Ubuntu 24.04 の `qemu-user-static` 8.2.2。登録済みインタープリターの `--version` で判定)は拒んで置き換えるコマンドを表示)、Docker が使える配置先(WSL2 では試験ファイルの `chmod 0600` を読み返して確かめ、モードを保持するファイルシステムであること。Windows のドライブは `runtime/.env` を `0600` に保てない。macOS ではエンジンから見えること。`up` が使う `postgres` イメージにリポジトリを読み取り専用で bind して問い合わせる。このサブコマンドが取得してよいイメージはこれだけ。エンジンが mount を拒むか中身が空なら終了 13、問い合わせ自体ができなければ警告。Docker VM が共有していない場所からの bind mount は黙って空になるため。問い合わせができないときは `GRDM_DOCKER_SHARED=1` で省ける)。TOON で報告する。対象外のホストかメモリ下限が収まらないときは終了 10、Docker が無い・動いていないときは 11、前提が不足なら 13、スキャナーが無ければ 51。 | 読むだけ | | `plan` | manifest とプロファイルを解決し、どのイメージを作りどのサービスを起こすか、公開ポート([§4](#2-ja-ports))とともに表示する。何も変えない。manifest が [SPEC-01](01_manifest.md) の検査に落ちれば終了 20。 | 触れない | -| `secrets init` / `secrets rotate` | `init` は乱数の秘密(DB パスワード、Django の秘密鍵、CAS の鍵)を `runtime/.env` にモード `0600` で書く。既定値や例のパスワードは決して書かない。既存ファイルは `--force` がなければ上書きしない。`init --missing` は、新しいキー(`ELASTIC8_PASSWORD`、`ELASTIC8_OSF_PASSWORD`。[ADR-14](../decisions/ADR-14_Elasticsearch_8_with_TLS_and_Authentication.md#japanese))より前に作られた `runtime/.env` のための更新で、既存の値をすべて保ち(実データが依存するパスワードは有効なまま)、足りないキーだけを足し、廃止したキーを消し、稼働中のパスワードは変えず、稼働中のスタックも要らない。`--force` とは併用できない。`rotate` は更新の手順ではない。すべての秘密を作り直し、稼働中の PostgreSQL と RabbitMQ のパスワードを変更し、新しい環境を受け取るようコンテナを再作成し、旧い資格情報が拒否されることを確かめる。どこかで失敗したら以前の `runtime/.env` に戻して 40 で終わる。`init --force` と `rotate` は廃止した資格情報のキーを消す。更新後は、`rotate` や再作成の前に `compose render` を実行する。以前に生成した Compose ファイルは廃止したキーを参照し続けるからである。四つのキーには CAS のイメージが要する長さがある。`TGC_ENCRYPTION_KEY` は 43 文字、`TGC_SIGNING_KEY` は 86 文字、`OSF_API_INSTITUTIONS_AUTH_JWESECRET` はちょうど 32 バイト、`OSF_API_INSTITUTIONS_AUTH_JWTSECRET` は 32 バイト以上で、後の二つは osf.io の `JWE_SECRET` と `JWT_SECRET` でもある([ADR-07 §7](../decisions/ADR-07_Native_Builds_of_EOL_Middleware.md#japanese))。四つとも base64url の文字(`A`–`Z`、`a`–`z`、`0`–`9`、`-`、`_`。空白・引用符・パディングは不可)だけを使い、公開された既定値であってはならず、`TGC_ENCRYPTION_KEY` は 32 バイトへ正準に復号できなければならない。レッドチームに秘密を取られた後に使う([SPEC-03 §6](03_security_gate.md#japanese))。 | 触れない / 触れる | +| `secrets init` / `secrets rotate` | `init` は乱数の秘密(DB パスワード、Django の秘密鍵、CAS の鍵)を `runtime/.env` にモード `0600` で書く。既定値や例のパスワードは決して書かない。既存ファイルは `--force` がなければ上書きしない。`init --missing` は、新しいキー(`ELASTIC8_PASSWORD`、`ELASTIC8_OSF_PASSWORD`。[ADR-14](../decisions/ADR-14_Elasticsearch_8_with_TLS_and_Authentication.md#japanese))より前に作られた `runtime/.env` のための更新で、既存の値をすべて保ち(実データが依存するパスワードは有効なまま)、足りないキーだけを足し、廃止したキーを消し、稼働中のパスワードは変えず、稼働中のスタックも要らない。`--force` とは併用できない。`rotate` は更新の手順ではない。すべての秘密を作り直し、稼働中の PostgreSQL と RabbitMQ のパスワードを変更し、新しい環境を受け取るようコンテナを再作成し、旧い資格情報が拒否されることを確かめる。どこかで失敗したら以前の `runtime/.env` に戻して 40 で終わる。`init --force` と `rotate` は廃止した資格情報のキーを消す。更新後は、`rotate` や再作成の前に `compose render` を実行する。以前に生成した Compose ファイルは廃止したキーを参照し続けるからである。四つのキーには CAS のイメージが要する長さがある。`TGC_ENCRYPTION_KEY` は 43 文字、`TGC_SIGNING_KEY` は 86 文字、`OSF_API_INSTITUTIONS_AUTH_JWESECRET` はちょうど 32 バイト、`OSF_API_INSTITUTIONS_AUTH_JWTSECRET` は 32 バイト以上で、後の二つは osf.io の `JWE_SECRET` と `JWT_SECRET` でもある([ADR-07 §7](../decisions/ADR-07_Native_Builds_of_EOL_Middleware.md#japanese))。四つとも base64url の文字(`A`–`Z`、`a`–`z`、`0`–`9`、`-`、`_`。空白・引用符・パディングは不可)だけを使い、公開された既定値であってはならず、`TGC_ENCRYPTION_KEY` は 32 バイトへ正準に復号できなければならない。`full` プロファイルの二つのキーには独自の形式があり、`secrets init` と `compose render` が名前を示して確かめる。`GRDM_VAULT_KEY` は aipf_mcp_data_grdm が利用者の預けた GRDM のトークンを暗号化する Fernet の鍵(32 バイトの urlsafe base64、パディング付き 44 文字)、`OIDC_MOCK_SIGNING_KEY` は受入 S3 が署名に使う EC P-256 の秘密 JWK の base64url で、`init`・`rotate`・`compose render` がその公開部分 `OIDC_MOCK_PUBLIC_JWK` を導く。`oidc_mock` のコンテナが受け取る鍵はそれだけである([ADR-15](../decisions/ADR-15_Mock_OIDC_Issuer_for_aipf.md#japanese))。レッドチームに秘密を取られた後に使う([SPEC-03 §6](03_security_gate.md#japanese))。 | 触れない / 触れる | | `compose render` | `runtime/.env` に生成すべきキーが無いとき、または CAS のキーが使えないとき(長さの誤り、base64url 以外の文字、公開された既定値)は、キーの名前だけを示して終了 2 で止まる(キーが無いときは `secrets init --missing`、CAS のキーが使えないときは `secrets rotate` で直す)。manifest とプロファイルから `runtime/docker-compose.yml` を書く。内部ネットワーク `grdm` と公開専用の `grdm-edge`(`--egress ` で追加接続し `runtime/.env` に記録。[SPEC-03 §3.2](03_security_gate.md#japanese))、部品ごとの `platform:` は manifest の `build_platform` から、ポートは `127.0.0.1` にだけ公開([§4](#2-ja-ports))、全サービスに非 root の `user:`、`read_only`・`cap_drop`・資源制限(`mem_limit`・`cpus`・`pids_limit`。`doctor` が検出したホストのメモリと Docker エンジンの VM メモリ(`docker info`。Docker Desktop では VM はホストの一部。実機第 10 回)の小さい方に応じて決め、テンプレートの下限 `mem_min_mb` が先に取り分を確保し、残りを他が分け合う。CPU はエンジンが報告する数(`docker info`。不明なときだけホストの数)から決める)、サービスごとの例外は `manifest/hardening.json` から([SPEC-03 §3.3](03_security_gate.md#japanese))、`platform:` は `doctor` の結果から選ぶ、名前付きボリュームは `runtime/volumes/` の下、`tmpfs` の各項目は `mode=1777` で mount(Compose の短い記法は root 専用の `0755` になる。実機第 11 回)、テンプレート値のプレースホルダ `{{HOST_PORT:}}`・`{{CONTAINER_PORT:}}`・`{{AUTH_URL}}`(fakecas の公開 URL。`--auth shibboleth` で fakecas が外れるときは web の URL)を解決する。組み立てる部品の `image:` に最後のビルド(`runtime/reports/build_latest.json`)のタグを使うのは、manifest が固定するタグと同じときだけである。記録されたビルドが固定と違うときは古い(最後の `build` の後で固定が動いた)とみなし、代わりに固定のタグを書き、報告の `stale_build_tags`(`component`・`pinned`・`recorded`・`image`)に載せ、標準エラーに警告の囲みを出して対処(`grdm-dev build` を実行してから `grdm-dev up`)を示す。`install` は組む前に render するので、終了コードは 0 のままである。一度も組んでいないイメージは古いものを動かす代わりに `up` を失敗させる。manifest が固定しない部品は記録されたタグを使う。`osf.env` には osf.io から Elasticsearch 8 への接続(`ELASTIC_URI=https://elasticsearch8:9200`、`ELASTIC_CA_CERTS=/es-ca/ca.crt`。利用者 `grdm_osf` とそのパスワードはサービスの環境から来る)、y-websocket への内部 URL(`Y_WEBSOCKET_INTERNAL_URL`)、`ENABLE_PUBLIC_ACCESS` を書き、web・api・admin・worker・osf_init は `elasticsearch8_ca`(公開の証明書だけ)を読み取り専用でマウントする。`--public-access true|false`(既定は `runtime/.env` の `GRDM_ENABLE_PUBLIC_ACCESS`、なければ `false`)は osf.io のすべてのプロセスの `ENABLE_PUBLIC_ACCESS` を決める(RDM-osf.io-dev2027 PR #8)。`false` はすべてのデータ経路で認証を要求し(匿名の要求はログインへの転送か 401。署名つきの WaterButler・MFR の呼び戻しや Webhook などは動き続ける)、`true` は公開プロジェクトの匿名閲覧を許す。選択は `runtime/.env` に記録し、`--egress` と同じく次回以降の render でも保つ。[ADR-02](../decisions/ADR-02_Network_Model.md) に従う。 | 触れない | | `build` | `docker buildx` でイメージを作る。プラットフォームは優先順に `--platform`、manifest の部品の `build_platform`(例: EOL ベースで arm64 に組めない部品は `linux/amd64` にしてエミュレーションで動かす)、それもなければホストと同じ。イメージはローカルに読み込む。`prebuilt_image`([SPEC-01](01_manifest.md))を持つ部品は組まず、ダイジェスト固定のイメージを要求された platform ごとに取得して `prebuilt` と報告する(`pull-failed` は終了 30)。`--push` でも prebuilt のイメージは `ghcr.io/rcosdp` に再公開しない(第三者の成果物であるため)。行は `prebuilt-not-pushed`、`pushed: false` と報告する。`--push` で multi-arch の manifest を `ghcr.io/rcosdp/`(非公開)へ送る。失敗は終了 30。 | 触れる | | `up` / `down` / `reset` | `up` は中断された `db` の操作をまず完了か巻き戻しし、`runtime/volumes/postgres_data` が生成済みのイメージと別の PostgreSQL の版で書かれているか版を読めなければ終了 15 で止まり(ADR-12)、そうでなければプロファイルを起動し健康になるまで待つ。データベースの確認の前に、render したファイルのボリューム `grdm-dev_<名前>` が Docker に既にあり、このチェックアウトの `runtime/volumes/<名前>` と別のディレクトリに結び付いていれば、`up` は終了 13 で止まる(同じプロジェクトの新しい clone や移したディレクトリ。Docker はボリュームを作った日のオプションを覚え、Compose は比べずに再利用するので、このチェックアウトは別のチェックアウトのデータを自分の新しい秘密で開いてしまう)。何も起動も削除もせず、メッセージはそのボリュームと、解放するコマンド(もう一方のチェックアウトで `grdm-dev down`、続けて `docker volume rm <名前>`。ファイルはディスクに残る)を示す。render したファイルのイメージが、manifest がもう固定していないローカルの `grdm-dev/<部品>:<タグ>` のときも、`up` は終了 13 で止まる(`compose render` の後に固定版が動いた。`build` は render したファイルを書き換えない)。メッセージは、先に `compose render`(新しい固定版が未ビルドなら `build` も)を実行するよう示す。`down` はコンテナを止めボリュームを残す。`reset` は止めたうえで確認の後に `runtime/volumes/` を消す(`--yes` で確認を省く)。Linux ではファイルがコンテナの uid の所有でホストの利用者には消せないので、`DAC_OVERRIDE` と `FOWNER` だけを持つ root のコンテナで中身を空にする。何かを止めたり消したりする前にホスト側でパスを検証し、シンボリックリンクの `runtime/` やボリュームのルート、リポジトリの外へ解決されるルート、シンボリックリンクのボリューム、`runtime/` やボリュームのルートやその下の任意の深さの mount(ホストの mount 表から読むので同じファイルシステムの bind mount も含む。表が読めないときも拒む)は終了 14 で止まる。`reset` は `down` の後と消去の後にも繰り返し、ボリュームのルートが別のディレクトリに変わっていれば止まる。`compose render` と `up` も、所有者設定の段が同じパスに root で働くので、同じ検証を終了 13 で行う。これらは時点の確認であり、Docker は後で bind を解決するのでこの隙間はどの検査でも塞げない。リポジトリを他の利用者やプロセスが同時に変更しないことを前提とする。 | 触れる | @@ -409,7 +411,7 @@ flowchart TD profiles[3]{name,services,note}: minimum,"osf_init volume_owner_init volume_access_check_1000 web api postgres elasticsearch8 elasticsearch8_setup rabbitmq fakecas",ログインとプロジェクト画面が見える最小の組 core,"minimum + admin worker wb wb_worker mfr unoconv ember_osf_web cas wiki_assets elasticsearch6 redis y_websocket volume_access_check_999",コア 6 リポジトリと基盤。Wiki 共同編集の y_websocket を含む。wiki_assets はボリューム wiki_static に Wiki エディタの束を写す一回限りのサービス(ADR-06) - full,"core + rdmcp aipf_mcp_data_grdm oasys shibboleth_sp test_idp",全部。shibboleth_sp は fakecas と入れ替わる + full,"core + rdmcp aipf_mcp_data_grdm oidc_mock oasys shibboleth_sp test_idp",全部。shibboleth_sp は fakecas と入れ替わる。oidc_mock は aipf_mcp_data_grdm が信頼する模擬の OIDC 発行者(ADR-15) ``` サービス名は RDM-osf.io 上流の `docker-compose.yml` に合わせ、既存の手引きがそのまま使えるようにします。これは manifest の `services[]` の値でもあります([SPEC-01 §3](01_manifest.md#japanese))。`wiki-frontend` 部品は `osf.io` のオーバーレイ(`overlay_of`、[ADR-06](../decisions/ADR-06_Wiki_Frontend_Overlay.md))です。そのサービスは「一回限り」(一度動いて終了するコンテナ。`restart: "no"`)で、束を組むイメージ `grdm-dev/wiki-frontend:-` を動かします。`volume_owner_init` が新しいボリューム `wiki_static` を uid 1000 に渡した後で、`wiki_assets` が webpack の出力 `/code/website/static/public/js/` をそこへ写して終了します。`web`・`api`・`admin`・`worker` はそのボリュームを `/code/website/static/public/js/wiki/` に読み取り専用で mount し、`condition: service_completed_successfully` で待つので、`up` は束が置かれてからそれらを起動します。`health` は終了コード 0 で終わった一回限りのサービスを健康と数えます。Compose プロファイル `shibboleth` とサービス `fakecas` は排他です。`compose render --profile full --auth shibboleth` は `fakecas` を外して `shibboleth_sp` と `test_idp` を足します。`full` の既定は `fakecas` です。`y_websocket` は `full` でなく `core` に属します。Wiki 共同編集が M2 の受入シナリオ([SPEC-04 §4](04_acceptance_tests.md#japanese) の C4)だからです。manifest の `y-websocket` 部品の `profile` もそれに合わせて `core` です([SPEC-01 §3](01_manifest.md#japanese))。 @@ -419,7 +421,7 @@ profiles[3]{name,services,note}: **公開ポート。** 既定のホスト側ポートを定めるのはこの表だけです。[ADR-02](../decisions/ADR-02_Network_Model.md)、[SPEC-03](03_security_gate.md)、[REF-01](../reference/REF-01_Glossary.md) はここを指します。すべて `127.0.0.1` にだけ公開します。 ```toon -ports[11]{service,host_port,profile,url}: +ports[12]{service,host_port,profile,url}: web,5000,minimum,http://127.0.0.1:5000 api,8000,minimum,http://127.0.0.1:8000 admin,8001,core,http://127.0.0.1:8001 @@ -431,6 +433,7 @@ ports[11]{service,host_port,profile,url}: oasys,3100,full,http://127.0.0.1:3100 cas,8090,full,http://127.0.0.1:8090 rdmcp,8100,full,http://127.0.0.1:8100 + aipf_mcp_data_grdm,8768,full,http://127.0.0.1:8768 ``` ほかのサービス(full プロファイル以外の `cas`、`volume_owner_init`、`wiki_assets`、ほかのサブシステム、`test_idp`、データベースとキュー)には Compose ネットワークの中からだけ届きます。ブラウザ向けのポートが要るときは `compose render` が割り当て、`plan` が表示します。`compose render` は選んだホスト側ポートをすべて `runtime/.env` に `GRDM_PORT_`(大文字)として書き、受入試験はそれを読みます([SPEC-04 §1](04_acceptance_tests.md#japanese))。既定のポートが使用中なら(macOS の AirPlay と `5000` が既知の例)、`compose render` は次の空きポートを取り、導入報告書に記録します。終了 12 は空きポートが見つからないときだけです。 diff --git a/docs/spec/04_acceptance_tests.md b/docs/spec/04_acceptance_tests.md index b75984c1..55de93d5 100644 --- a/docs/spec/04_acceptance_tests.md +++ b/docs/spec/04_acceptance_tests.md @@ -5,7 +5,8 @@ **Jump:** [English](#english) · [日本語](#japanese) -- Version / 版: 0.15 — 2026-11-16. **Learned:** an MCP server can refuse in two different places, and the contract must name the right one: rdmcp refuses a request without a token at the transport (HTTP 401 with a Bearer challenge) but a node outside the user's grant inside the tool result (`isError`, HTTP 200), and it never takes a GRDM personal access token, so the S2 row now walks the real CAS OAuth flow to a token bound to the run's project (owner decision 2026-11-16) instead of a PAT, `resources/read` and a transport 403. / MCP サーバーは二つの違う場所で拒みうるので、契約は正しい方を名指さなければならない。rdmcp はトークンのない要求をトランスポートで拒み(Bearer の challenge 付きの HTTP 401)、利用者の許可の外のノードは道具の結果の中で拒む(`isError`、HTTP 200)。GRDM の個人アクセストークンは決して受け取らない。そこで S2 の行は、PAT・`resources/read`・トランスポートの 403 の代わりに、本物の CAS の OAuth の流れを実行中プロジェクトに結び付いたトークンまでたどる(所有者判断 2026-11-16)。 +- Version / 版: 0.16 — 2026-11-16. **Learned:** a scenario has to test what the subsystem is for, not what a contract guessed it would offer: S3 wanted a metadata read from aipf_mcp_data_grdm, which only writes and trusts only its issuer's tokens, so the row now signs tokens with a mock issuer's key (ADR-15), stores user_a's PAT through the component's own form and checks the project the component creates in the OSF API, while the rejections cover the token rules and user_b's PAT on user_a's project. / シナリオは、契約が推し量った機能でなく、サブシステムが何のためにあるかを試さなければならない。S3 は aipf_mcp_data_grdm にメタデータの読み取りを求めたが、この部品は書くだけで、自分の発行者のトークンしか信頼しない。そこで行は模擬の発行者の鍵でトークンに署名し(ADR-15)、部品自身の欄から user_a の PAT を預け、部品が作ったプロジェクトを OSF API で確かめる。拒否はトークンの規則と、user_a のプロジェクトでの user_b の PAT を扱う。 + 0.15 — 2026-11-16. **Learned:** an MCP server can refuse in two different places, and the contract must name the right one: rdmcp refuses a request without a token at the transport (HTTP 401 with a Bearer challenge) but a node outside the user's grant inside the tool result (`isError`, HTTP 200), and it never takes a GRDM personal access token, so the S2 row now walks the real CAS OAuth flow to a token bound to the run's project (owner decision 2026-11-16) instead of a PAT, `resources/read` and a transport 403. / MCP サーバーは二つの違う場所で拒みうるので、契約は正しい方を名指さなければならない。rdmcp はトークンのない要求をトランスポートで拒み(Bearer の challenge 付きの HTTP 401)、利用者の許可の外のノードは道具の結果の中で拒む(`isError`、HTTP 200)。GRDM の個人アクセストークンは決して受け取らない。そこで S2 の行は、PAT・`resources/read`・トランスポートの 403 の代わりに、本物の CAS の OAuth の流れを実行中プロジェクトに結び付いたトークンまでたどる(所有者判断 2026-11-16)。 0.14 — 2026-11-15. **Learned:** a contract written before the subsystem was seen can ask for something the subsystem does not have: the S4 row wanted a link from OASys's project list back to GRDM, and OASys's project selection has none, so the row now checks the link the other way (GRDM's login returns to OASys, which needs the OASys origin registered as a PreprintProvider domain) and keeps the session and project checks; a port left in runtime/.env after a render back to core no longer runs the scenario, the profile does. / サブシステムを見る前に書いた契約は、サブシステムに無いものを求めうる。S4 の行は OASys のプロジェクト一覧から GRDM へのリンクを求めたが、OASys のプロジェクト選択にそれは無い。そこで行は逆向きのつながり(GRDM のログインが OASys に戻ること。OASys の起点を PreprintProvider の domain に登録する必要がある)を確かめ、セッションとプロジェクトの確認は残した。core に描画し直した後に runtime/.env に残るポートではシナリオは動かず、プロファイルで決まる。 0.13 — 2026-11-15. **Learned:** a scenario that walks the same screens as another adds evidence only when it asserts what the other cannot see: C4 already proves that two wiki editors converge, so S1 asserts the socket itself — the one room URL the page opens, with exactly one token, the y-sync document update B's socket receives while A's sentence arrives, and the same URL without the token closed with 4401; a trial that kept the token stayed open until the 10 s limit, so the rejection really depends on the token. / 別のシナリオと同じ画面をたどるシナリオは、その別のシナリオに見えないものを確かめるときだけ証拠を増やす。C4 は二つの wiki エディタが収束することをすでに示すので、S1 はソケットそのものを確かめる。ページが開く唯一の部屋の URL とその中のちょうど一つのトークン、A の一文が届く間に B のソケットが受け取る y-sync の文書更新、そしてトークンを除いた同じ URL が 4401 で閉じられること。トークンを残した試行は 10 秒の上限まで開いたままだったので、拒否は本当にトークンで決まっている。 0.12 — 2026-11-14. **Learned:** masking secrets was not enough to make evidence publishable: every committed junit.xml carried the machine's name in its `hostname` attribute, and failures and skips carried the absolute checkout path, which on a personal machine names its owner; `accept` now replaces both before anything is kept, and the run folders already committed were scrubbed the same way (PR #62 review r1 R1-03). / 秘密を伏せるだけでは、証跡を公開できる状態にはならなかった。コミットした junit.xml はどれも `hostname` 属性に機械の名前を持ち、失敗やスキップはチェックアウトの絶対パスを持っていた。個人の機械では、それが持ち主の名前になる。`accept` は何かを残す前に両方を置き換えるようになり、既にコミットした実行のフォルダーも同じ方法で消した(PR #62 レビュー r1 R1-03)。 @@ -81,12 +82,12 @@ Scenario C6 logs in to the admin app in two steps: the password, then a one-time subsystem_scenarios[5]{id,subsystem,success_case,rejection_case,services,prerequisite}: S1,y-websocket,"user_a and user_b (write contributor) open the Wiki of the run's project in two contexts; a sentence typed by A is in B's editor within 5 s while B's room socket receives a y-sync document update; the one room WebSocket each page opens is exactly ws://127.0.0.1:/?token=, the endpoint and docId the page announces in window.contextVars.wiki","the captured WebSocket URL reopened from a logged-out context without its token -> close code 4401 and zero document frames before the close","web wiki y_websocket","none (core profile)" S2,rdmcp (MCP endpoint),"an MCP client registers dynamically with rdmcp and starts /authorize with PKCE (S256); rdmcp sends the browser to the real CAS oauth2/authorize, where user_a signs in with the OSF password and allows the application; on rdmcp's node-selection page user_a finds the run's project and connects; rdmcp's code with the PKCE verifier yields an rdmcp token; MCP initialize, then tools/call rdm_get_node on the run's project returns that node with its title","an MCP request without a token -> transport HTTP 401 with WWW-Authenticate: Bearer (not 200 with an embedded status); user_b's private project is not offered on user_a's node-selection page, and posting its id to the page's endpoint -> 400; with a grant for the run's project only, tools/call rdm_get_node on user_b's project and on user_a's second, unselected project -> tool error (isError, HTTP 200) although user_a can read that second project in GRDM","rdmcp cas api web","full profile (GRDM_PROFILE=full) with GRDM_PORT_RDMCP and GRDM_PORT_CAS in runtime/.env and `accept --seed` (user_a's OSF password for the real CAS); otherwise skip with reason" - S3,aipf_mcp_data_grdm,"user_a's token calls the project-metadata tool for the run's project; id and title equal the OSF API's","user_b's token for user_a's project -> 403, or an empty answer with an explicit denial code; a silent empty list fails",aipf_mcp_data_grdm,"GRDM_PORT_AIPF_MCP_DATA_GRDM in runtime/.env; otherwise skip with reason" + S3,aipf_mcp_data_grdm,"the test signs JWTs with oidc_mock's key from runtime/.env (iss and aud of the template, exp in 5 min). user_a creates a personal access token (PAT); a token from the AIPF screen (azp grdm-dev-webui, no actor) gets a form token from grdm_token_form, and POST /grdm/token?t= with the PAT -> 200 registered, grdm_user = user_a's name; grdm_token_status reports it. A token of the research-flow service (act.sub grdm-dev-research-flow) calls ensure_project -> created=true; GET /v2/nodes// in user_a's session -> 200 with the same title, the key in the description, public false and admin permission; a second call returns the same node with created=false","no bearer and a malformed bearer -> 401; a token signed by another key -> 401 invalid_token; another audience -> 401 wrong_audience; no actor from an unknown client -> 403 actor_missing; an actor not on the list -> 403 actor_not_allowed; an allowed agent that is not the writer -> tool error writer_not_allowed; a forged form token -> 403 bad_form_token; user_b's PAT registered the same way, used by the writer on user_a's private run project with write_record -> tool error node_forbidden and nothing in the project's storage","api aipf_mcp_data_grdm oidc_mock","full profile (GRDM_PROFILE=full) and GRDM_PORT_AIPF_MCP_DATA_GRDM in runtime/.env, otherwise skip with reason, decided before the key is read; in that eligible environment a missing OIDC_MOCK_SIGNING_KEY fails" S4,OASys,"user_a signed in to GRDM opens http://127.0.0.1:/ -> 200 and is taken to /registration/ without a second sign-in; the project selection of /registration/#/project lists the run's project (its id is in the /v2/users/me/nodes/ answer the page fetches, and its row is selectable); web /login/?next=/registration/ returns to OASys (osf_init registers the OASys origin as the PreprintProvider oasys domain)","anonymous GET http://127.0.0.1:/api/v1/project//workflow/activations/ -> 401 with a JSON body (not a login page, not 404)","web api oasys","full profile (GRDM_PROFILE=full) and GRDM_PORT_OASYS in runtime/.env; otherwise skip with reason" S5,Shibboleth-SP via test IdP,"mock-user-01 (eduPersonAffiliation member) : web /login?campaign=institution -> SP -> test IdP form -> SAML POST -> GRDM dashboard served by web; osf cookie set; /v2/users/me/institutions/ includes MOCK Institution","mock-user-noaff (eduPersonAffiliation affiliate only) -> the SP RequestMap answers 403 with its error page; no osf cookie; API /v2/users/me/ -> 401","web shibboleth_sp test_idp","--auth shibboleth render (GRDM_AUTH=shibboleth in runtime/.env; fakecas absent); mock users from installer/acceptance/fixtures/idp/mock_users.json; ADR-05" ``` -The table is the contract of `installer/acceptance/test_s1_y_websocket.py` … `test_s5_shibboleth.py`. Until its subsystem is verified at M3 a file carries `skip(reason="M3")` and `TODO(M3)` marks on every locator and path that has not been seen on the real subsystem; the success and rejection logic is written in full so that M3 only verifies selectors. S1 runs since 2026-11-15: `y_websocket` is in the `core` profile, so, like C4 and C7, it never skips. S4 runs since 2026-11-15 in the `full` profile and skips in any other, because `oasys` exists only there and its port stays in `runtime/.env` after a render back to `core`; the earlier S4 row asked for a link from the project list of OASys (the registration application beside GRDM; see [REF-01](../reference/REF-01_Glossary.md)) to the GRDM project page, which OASys does not have (its project selection is a search field without links, M3 survey), so the row checks the link in the other direction, GRDM's login returning to OASys, which is what a user signing in from OASys needs. S2 runs since 2026-11-16 in the `full` profile and skips in any other, like S4. Its earlier row asked for a GRDM personal access token at the MCP (Model Context Protocol, explained below) endpoint, `resources/read` of the project and a transport 403 for user_b; rdmcp has none of these: it takes no GRDM token (its ADR-0002 rejects token pass-through and it issues its own tokens after the real CAS OAuth flow), it offers tools and no MCP resources, and it refuses a node outside the user's grant inside the tool result, which is how MCP reports a refused tool call. The owner decided on 2026-11-16 that S2 tests the OAuth flow at the real CAS ([SPEC-02 §4](02_installer_cli.md#english): fakecas stays the browser's sign-in to GRDM, and the real CAS is published only as rdmcp's OAuth server). The CAS sign-in and consent forms are posted by the guarded HTTP client with exactly their fields, because the CAS pages load head.js from cdnjs.cloudflare.com, which is not a known external asset (§3 rule 4) and would fail the browser guard; rdmcp's node-selection page has no external asset and is driven in a guarded browser. S3 and S5 still carry the skip. A service without a host port today (`aipf_mcp_data_grdm`) gets one in its template at M3; a test whose port is missing from `runtime/.env` skips with that exact reason instead of failing on a wrong URL. The test IdP (identity provider: the server that checks who you are when you log in; see [REF-01](../reference/REF-01_Glossary.md)) and its mock-attribute rule are decided in [ADR-05](../decisions/ADR-05_Test_IdP_for_Shibboleth.md). +The table is the contract of `installer/acceptance/test_s1_y_websocket.py` … `test_s5_shibboleth.py`. Until its subsystem is verified at M3 a file carries `skip(reason="M3")` and `TODO(M3)` marks on every locator and path that has not been seen on the real subsystem; the success and rejection logic is written in full so that M3 only verifies selectors. S1 runs since 2026-11-15: `y_websocket` is in the `core` profile, so, like C4 and C7, it never skips. S4 runs since 2026-11-15 in the `full` profile and skips in any other, because `oasys` exists only there and its port stays in `runtime/.env` after a render back to `core`; the earlier S4 row asked for a link from the project list of OASys (the registration application beside GRDM; see [REF-01](../reference/REF-01_Glossary.md)) to the GRDM project page, which OASys does not have (its project selection is a search field without links, M3 survey), so the row checks the link in the other direction, GRDM's login returning to OASys, which is what a user signing in from OASys needs. S2 runs since 2026-11-16 in the `full` profile and skips in any other, like S4. Its earlier row asked for a GRDM personal access token at the MCP (Model Context Protocol, explained below) endpoint, `resources/read` of the project and a transport 403 for user_b; rdmcp has none of these: it takes no GRDM token (its ADR-0002 rejects token pass-through and it issues its own tokens after the real CAS OAuth flow), it offers tools and no MCP resources, and it refuses a node outside the user's grant inside the tool result, which is how MCP reports a refused tool call. The owner decided on 2026-11-16 that S2 tests the OAuth flow at the real CAS ([SPEC-02 §4](02_installer_cli.md#english): fakecas stays the browser's sign-in to GRDM, and the real CAS is published only as rdmcp's OAuth server). The CAS sign-in and consent forms are posted by the guarded HTTP client with exactly their fields, because the CAS pages load head.js from cdnjs.cloudflare.com, which is not a known external asset (§3 rule 4) and would fail the browser guard; rdmcp's node-selection page has no external asset and is driven in a guarded browser. S3 runs since 2026-11-16 in the `full` profile and skips in any other. The earlier S3 row asked for a project-metadata tool that aipf_mcp_data_grdm does not have (the M3 survey of 2026-11-15): the component only writes research records into GRDM, accepts only bearer tokens of the AI platform's OIDC issuer with an allowed actor, and reaches GRDM with the user's own stored personal access token. The owner decided on 2026-11-16 to test it with a mock OIDC issuer and a write (「模擬の OIDC 発行者+書き込みの試験」), so `oidc_mock` ([ADR-15](../decisions/ADR-15_Mock_OIDC_Issuer_for_aipf.md)) publishes the key the test signs with, and the row checks the token rules, the PAT form and a project the component creates as user_a; `write_record` is used only in the rejection, because its uploads go to the browser-facing WaterButler links the API returns, which the component's container cannot reach (open point). S5 still carries the skip. A test whose port is missing from `runtime/.env` skips with that exact reason instead of failing on a wrong URL. The test IdP (identity provider: the server that checks who you are when you log in; see [REF-01](../reference/REF-01_Glossary.md)) and its mock-attribute rule are decided in [ADR-05](../decisions/ADR-05_Test_IdP_for_Shibboleth.md). **MCP** (Model Context Protocol) is a standard way for an AI assistant to ask a service for tools and data; `rdmcp` and `aipf_mcp_data_grdm` expose GRDM through it. Their MCP calls use an HTTP client in the same `pytest` session, not a browser; only the pages a person must see on the way to a token (S2: rdmcp's node selection) open in the guarded browser. @@ -207,12 +208,12 @@ core_scenarios[7]{id,scenario,success_case,rejection_case,services}: subsystem_scenarios[5]{id,subsystem,success_case,rejection_case,services,prerequisite}: S1,y-websocket,"user_a と user_b(書込の共同研究者)が二つの文脈で実行中プロジェクトの Wiki を開き、A が打った一文が 5 秒以内に B のエディタに現れ、その間に B の部屋のソケットが y-sync の文書更新を受け取る。各ページが開く唯一の部屋の WebSocket は厳密に ws://127.0.0.1:/?token= で、ページが window.contextVars.wiki で知らせる接続先と docId に一致する","捕らえた WebSocket の URL をトークンなしでログアウト文脈から開き直す -> 終了コード 4401。閉じる前の文書フレームは 0","web wiki y_websocket","なし(core プロファイル)" S2,rdmcp(MCP の口),"MCP クライアントが rdmcp に動的登録し、PKCE(S256)付きで /authorize を始める。rdmcp はブラウザを本物の CAS の oauth2/authorize へ送り、user_a はそこで OSF のパスワードでサインインしてアプリを許可する。rdmcp のノード選択画面で user_a が実行中プロジェクトを見つけて接続する。rdmcp のコードと PKCE の検証子で rdmcp のトークンを得る。MCP の initialize の後、実行中プロジェクトへの tools/call rdm_get_node がそのノードを題名とともに返す","トークンのない MCP の要求 -> トランスポートの HTTP 401 と WWW-Authenticate: Bearer(埋め込みの状態を持つ 200 は不可)。user_b の非公開プロジェクトは user_a のノード選択画面に出ず、その id を画面の口に送ると -> 400。実行中プロジェクトだけの許可で、user_b のプロジェクトと、選ばなかった user_a の二つ目のプロジェクトへの tools/call rdm_get_node -> 道具のエラー(isError、HTTP 200)。二つ目のプロジェクトは GRDM では user_a が読める","rdmcp cas api web","full プロファイル(GRDM_PROFILE=full)で、runtime/.env に GRDM_PORT_RDMCP と GRDM_PORT_CAS、かつ `accept --seed`(本物の CAS 用の user_a の OSF パスワード)。なければ理由付きでスキップ" - S3,aipf_mcp_data_grdm,"user_a のトークンで実行中プロジェクトのメタデータ取得ツールを呼び、id と題名が OSF API と一致する","user_b のトークンで user_a のプロジェクト -> 403、または明示の拒否コード付きの空。黙った空の一覧は失敗",aipf_mcp_data_grdm,"runtime/.env に GRDM_PORT_AIPF_MCP_DATA_GRDM。なければ理由付きでスキップ" + S3,aipf_mcp_data_grdm,"試験は runtime/.env の oidc_mock の鍵で JWT に署名する(iss と aud はテンプレートのもの、期限 5 分)。user_a が個人用アクセストークン(PAT)を作る。AIPF の画面からのトークン(azp grdm-dev-webui、代理者なし)で grdm_token_form から合い言葉を得て、PAT を POST /grdm/token?t=<合い言葉> に送る -> 200 で登録、grdm_user は user_a の名前。grdm_token_status がそれを報告する。研究フローのサービスのトークン(act.sub grdm-dev-research-flow)で ensure_project -> created=true。user_a のセッションで GET /v2/nodes// -> 200 で、題名が同じ、説明に目印、public は false、権限に admin。二度目の呼び出しは同じノードを created=false で返す","ベアラーなしと形の違うベアラー -> 401。別の鍵で署名したトークン -> 401 invalid_token。別の宛先 -> 401 wrong_audience。未知のクライアントからの代理者なし -> 403 actor_missing。一覧に無い代理者 -> 403 actor_not_allowed。書き手でない許可済みのエージェント -> 道具のエラー writer_not_allowed。偽の合い言葉 -> 403 bad_form_token。同じ手順で登録した user_b の PAT を、書き手が user_a の非公開の実行中プロジェクトに write_record で使う -> 道具のエラー node_forbidden で、プロジェクトの保存領域は空のまま","api aipf_mcp_data_grdm oidc_mock","full プロファイル(GRDM_PROFILE=full)で、runtime/.env に GRDM_PORT_AIPF_MCP_DATA_GRDM。なければ鍵を読む前に理由付きでスキップ。その対象の環境で OIDC_MOCK_SIGNING_KEY が無ければ失敗" S4,OASys,"GRDM にログイン済みの user_a が http://127.0.0.1:/ を開く -> 200。二度目のサインインなしに /registration/ へ進む。/registration/#/project のプロジェクト選択に実行中プロジェクトが並ぶ(ページが取得する /v2/users/me/nodes/ の答えにその id があり、その行は選択できる)。web の /login/?next=/registration/ が OASys へ戻す(osf_init が OASys の起点を PreprintProvider oasys の domain に登録する)","匿名で http://127.0.0.1:/api/v1/project/<実行中プロジェクト>/workflow/activations/ を GET -> JSON 本文付きの 401(ログイン画面も 404 も不可)","web api oasys","full プロファイル(GRDM_PROFILE=full)で、runtime/.env に GRDM_PORT_OASYS。なければ理由付きでスキップ" S5,テスト IdP 経由の Shibboleth-SP,"mock-user-01(eduPersonAffiliation に member): web /login?campaign=institution -> SP -> テスト IdP のフォーム -> SAML POST -> web が出す GRDM ダッシュボード。osf cookie あり。/v2/users/me/institutions/ に MOCK Institution","mock-user-noaff(eduPersonAffiliation は affiliate のみ)-> SP の RequestMap がエラー画面で 403。osf cookie なし。API /v2/users/me/ -> 401","web shibboleth_sp test_idp","--auth shibboleth での描画(runtime/.env に GRDM_AUTH=shibboleth。fakecas なし)。模擬利用者は installer/acceptance/fixtures/idp/mock_users.json。ADR-05" ``` -この表が `installer/acceptance/test_s1_y_websocket.py` 〜 `test_s5_shibboleth.py` の契約です。M3 でサブシステムを確かめるまでは、ファイルが `skip(reason="M3")` を持ち、実機のサブシステムで未確認のロケータとパスに `TODO(M3)` を付けます。成功と拒否の論理は全部書いてあり、M3 ではセレクタの確認だけをします。S1 は 2026-11-15 から実行します。`y_websocket` は `core` プロファイルにあるので、C4・C7 と同じくスキップしません。S4 は 2026-11-15 から `full` プロファイルで実行し、ほかのプロファイルではスキップします。`oasys` は `full` にしかなく、`core` に描画し直してもそのポートは `runtime/.env` に残るからです。以前の S4 の行は、OASys(GRDM の横で動く登録のためのアプリ。[REF-01](../reference/REF-01_Glossary.md))のプロジェクト一覧から GRDM のプロジェクト画面へのリンクを求めていましたが、OASys にそのリンクはありません(プロジェクト選択はリンクのない検索欄。M3 調査)。そこで行は逆向きのつながり、つまり GRDM のログインが OASys へ戻ることを確かめます。OASys からサインインする利用者に要るのはこちらです。S2 は 2026-11-16 から、S4 と同じく `full` プロファイルで実行し、ほかではスキップします。以前の S2 の行は、MCP の口での GRDM の個人アクセストークン、プロジェクトの `resources/read`、user_b へのトランスポートの 403 を求めていましたが、rdmcp にはどれもありません。GRDM のトークンを受け取らず(rdmcp の ADR-0002 はトークンの素通しを退け、本物の CAS の OAuth の流れの後に自分のトークンを発行する)、道具を出して MCP の資源は出さず、利用者の許可の外のノードは道具の結果の中で拒みます。MCP は拒んだ道具の呼び出しをそう知らせます。所有者は 2026-11-16 に、S2 は本物の CAS での OAuth の流れを試すと決めました([SPEC-02 §4](02_installer_cli.md#japanese)。ブラウザの GRDM へのサインインは fakecas のままで、本物の CAS は rdmcp の OAuth サーバーとしてだけ公開する)。CAS のサインインと許可のフォームは、守りのある HTTP クライアントがそのフォームの項目どおりに送ります。CAS の画面は cdnjs.cloudflare.com から head.js を読み込み、それは既知の外部資産ではなく(§3 規則 4)、ブラウザの守りで失敗になるからです。rdmcp のノード選択画面には外部資産がなく、守りのあるブラウザで操作します。S3・S5 はまだスキップを持ちます。今日ホスト側ポートを持たないサービス(`aipf_mcp_data_grdm`)は M3 でテンプレートにポートを得ます。`runtime/.env` にポートがない試験は、誤った URL で失敗するのでなく、その理由を書いてスキップします。テスト IdP(アイデンティティプロバイダ。ログインのときに「あなたが誰か」を確かめるサーバ。[REF-01](../reference/REF-01_Glossary.md))と模擬属性の規則は [ADR-05](../decisions/ADR-05_Test_IdP_for_Shibboleth.md) で決めます。 +この表が `installer/acceptance/test_s1_y_websocket.py` 〜 `test_s5_shibboleth.py` の契約です。M3 でサブシステムを確かめるまでは、ファイルが `skip(reason="M3")` を持ち、実機のサブシステムで未確認のロケータとパスに `TODO(M3)` を付けます。成功と拒否の論理は全部書いてあり、M3 ではセレクタの確認だけをします。S1 は 2026-11-15 から実行します。`y_websocket` は `core` プロファイルにあるので、C4・C7 と同じくスキップしません。S4 は 2026-11-15 から `full` プロファイルで実行し、ほかのプロファイルではスキップします。`oasys` は `full` にしかなく、`core` に描画し直してもそのポートは `runtime/.env` に残るからです。以前の S4 の行は、OASys(GRDM の横で動く登録のためのアプリ。[REF-01](../reference/REF-01_Glossary.md))のプロジェクト一覧から GRDM のプロジェクト画面へのリンクを求めていましたが、OASys にそのリンクはありません(プロジェクト選択はリンクのない検索欄。M3 調査)。そこで行は逆向きのつながり、つまり GRDM のログインが OASys へ戻ることを確かめます。OASys からサインインする利用者に要るのはこちらです。S2 は 2026-11-16 から、S4 と同じく `full` プロファイルで実行し、ほかではスキップします。以前の S2 の行は、MCP の口での GRDM の個人アクセストークン、プロジェクトの `resources/read`、user_b へのトランスポートの 403 を求めていましたが、rdmcp にはどれもありません。GRDM のトークンを受け取らず(rdmcp の ADR-0002 はトークンの素通しを退け、本物の CAS の OAuth の流れの後に自分のトークンを発行する)、道具を出して MCP の資源は出さず、利用者の許可の外のノードは道具の結果の中で拒みます。MCP は拒んだ道具の呼び出しをそう知らせます。所有者は 2026-11-16 に、S2 は本物の CAS での OAuth の流れを試すと決めました([SPEC-02 §4](02_installer_cli.md#japanese)。ブラウザの GRDM へのサインインは fakecas のままで、本物の CAS は rdmcp の OAuth サーバーとしてだけ公開する)。CAS のサインインと許可のフォームは、守りのある HTTP クライアントがそのフォームの項目どおりに送ります。CAS の画面は cdnjs.cloudflare.com から head.js を読み込み、それは既知の外部資産ではなく(§3 規則 4)、ブラウザの守りで失敗になるからです。rdmcp のノード選択画面には外部資産がなく、守りのあるブラウザで操作します。S3 は 2026-11-16 から `full` プロファイルで実行し、ほかのプロファイルではスキップします。以前の S3 の行は、aipf_mcp_data_grdm に無いプロジェクトのメタデータを読む道具を求めていました(2026-11-15 の M3 調査)。この部品は研究の記録を GRDM に書くだけで、AI 基盤の OIDC 発行者のベアラートークンを、許可された代理者付きでしか受け付けず、GRDM には利用者本人が預けた個人用アクセストークンで入ります。所有者は 2026-11-16 に「模擬の OIDC 発行者+書き込みの試験」と決めました。そこで `oidc_mock`([ADR-15](../decisions/ADR-15_Mock_OIDC_Issuer_for_aipf.md))が試験の署名に使う鍵を公開し、行はトークンの規則、PAT の欄、部品が user_a として作るプロジェクトを確かめます。`write_record` は拒否の側でだけ使います。そのアップロードは API が返すブラウザ向けの WaterButler のリンクに向かい、部品のコンテナからは届かないからです(未了)。S5 はまだスキップを持ちます。`runtime/.env` にポートがない試験は、誤った URL で失敗するのでなく、その理由を書いてスキップします。テスト IdP(アイデンティティプロバイダ。ログインのときに「あなたが誰か」を確かめるサーバ。[REF-01](../reference/REF-01_Glossary.md))と模擬属性の規則は [ADR-05](../decisions/ADR-05_Test_IdP_for_Shibboleth.md) で決めます。 **MCP**(Model Context Protocol)は、AI アシスタントがサービスに道具とデータを尋ねるための共通の決まりです。`rdmcp` と `aipf_mcp_data_grdm` は GRDM をこの決まりで公開します。これらの MCP の呼び出しはブラウザでなく、同じ `pytest` セッション内の HTTP クライアントで行います。トークンに至る途中で人が見なければならない画面(S2 では rdmcp のノード選択)だけを、守りのあるブラウザで開きます。 diff --git a/docs/status/README.md b/docs/status/README.md index 06d902e0..86b39305 100644 --- a/docs/status/README.md +++ b/docs/status/README.md @@ -5,7 +5,8 @@ **Jump:** [English](#english) · [日本語](#japanese) -- Version / 版: 0.29 — 2026-11-16. **Learned:** a subsystem that is its own OAuth server is tested through the flow it really uses, not through a shortcut it refuses: S2 (rdmcp) now passes with a real sign-in at the real CAS, rdmcp's node-selection page and a token bound to the run's project, and the C scenarios still pass in the full profile while `api` validates every bearer token at that CAS (run `acc-20261115T164109Z-1a319f`: 20 passed, 4 skipped). / 自分自身が OAuth サーバーであるサブシステムは、それが拒む近道ではなく、実際に使う流れで試す。S2(rdmcp)は、本物の CAS での本当のサインイン、rdmcp のノード選択画面、実行中プロジェクトに結び付いたトークンで合格するようになり、`api` がすべての Bearer トークンをその CAS で確かめる full プロファイルでも C のシナリオは合格のまま(実行 `acc-20261115T164109Z-1a319f`: 合格 20、スキップ 4)。 +- Version / 版: 0.30 — 2026-11-16. **Learned:** a subsystem that trusts only its own identity provider becomes testable once the stack carries a stand-in for that provider whose key the test holds: S3 (aipf_mcp_data_grdm with the mock OIDC issuer, ADR-15) now passes, and the whole suite on the Mac Studio stack in the full profile passed 20 with 4 skips (S2, S5), run `acc-20261115T162657Z-3b9bef`, with no service stopped. / 自分の ID プロバイダしか信頼しないサブシステムは、試験が鍵を持つその代役をスタックに置けば試せるようになる。S3(模擬の OIDC 発行者つきの aipf_mcp_data_grdm。ADR-15)が合格し、Mac Studio のスタックの full プロファイルでのスイート全体は合格 20、スキップ 4(S2・S5)、実行 `acc-20261115T162657Z-3b9bef` で、止めたサービスは無かった。 + 0.29 — 2026-11-16. **Learned:** a subsystem that is its own OAuth server is tested through the flow it really uses, not through a shortcut it refuses: S2 (rdmcp) now passes with a real sign-in at the real CAS, rdmcp's node-selection page and a token bound to the run's project, and the C scenarios still pass in the full profile while `api` validates every bearer token at that CAS (run `acc-20261115T164109Z-1a319f`: 20 passed, 4 skipped). / 自分自身が OAuth サーバーであるサブシステムは、それが拒む近道ではなく、実際に使う流れで試す。S2(rdmcp)は、本物の CAS での本当のサインイン、rdmcp のノード選択画面、実行中プロジェクトに結び付いたトークンで合格するようになり、`api` がすべての Bearer トークンをその CAS で確かめる full プロファイルでも C のシナリオは合格のまま(実行 `acc-20261115T164109Z-1a319f`: 合格 20、スキップ 4)。 0.28 — 2026-11-15. **Learned:** a pin bump that renders before it starts needs one `up`, not two: osf.io `07cc56d47b39` (#34, #35, #36, #38) ran after a single `up`, and the suite passed 16 with 8 skips, C1 to C7 and S1 included (DEMO-20 0.25). / 起動の前に描画する固定版の更新は、`up` が二度でなく一度で済む。osf.io `07cc56d47b39`(#34・#35・#36・#38)は一度の `up` で動き、スイートは C1〜C7 と S1 を含めて合格 16、スキップ 8 だった(DEMO-20 0.25)。 0.27 — 2026-11-15. **Learned:** a subsystem whose URLs are compiled into its pages can still be one image for any port when the build carries placeholders and the start writes the real URLs: S4 (OASys, full profile) now passes, and the whole suite on the Mac Studio stack in the full profile passed 18 with 6 skips (S2, S3, S5), run `acc-20261115T131515Z-200a84`. / ページに URL を埋め込むサブシステムでも、ビルドに仮の値を入れ、起動時に本物の URL を書けば、どのポートにも一つのイメージで足りる。S4(OASys、full プロファイル)が合格し、Mac Studio のスタックの full プロファイルでスイート全体は 18 件合格、スキップ 6 件(S2・S3・S5)だった(実行 `acc-20261115T131515Z-200a84`)。 0.26 — 2026-11-15. **Learned:** "from an empty Docker" and "in one run" are separate claims: the amd64 VM now has the first and not the second (DEMO-22 0.9), so the row says both what was shown and what was not. / 「空の Docker から」と「一回の実行で」は別の主張である。amd64 の VM は前者を示したが後者は示していない(DEMO-22 0.9)。だから行には、示したことと示していないことの両方を書く。 @@ -50,7 +51,7 @@ |---|---|---|---| | M1 | Docs skeleton, manifest, dev2027 repos exist; reviewed PRs merged to develop | Report + allow-list | ✅ **Done** (PR #19 merged 2026-11-05) | | M2 | Core 6 install with one command on Mac Studio; Playwright acceptance passes; amd64 builds | Report + allow-list (fewer entries than M1) | ✅ **Done by owner decision of 2026-11-13** (「一括で承認、M2 完了」), with known limits, **not** by meeting every written criterion. Shown: on the Mac Studio, the one-command install succeeded in earlier real install runs on earlier pins; on the final merged pins the existing Mac environment was updated with separate commands, then acceptance passed 14 of 14 (run `acc-20261112T223428Z-fda84a`) and the scan completed (operator-recorded) ([DEMO-20](../demo/20_core_install_mac_studio.md#english)); on 2026-11-14 `install-macos.sh --yes --profile core` from a new checkout of develop 0df4640 returned 0 in one run of 12 min 50 s (warm Docker cache, not an empty Docker) and acceptance passed 14 of 14 with the default waits (run `acc-20261114T103547Z-8c60ae`, DEMO-20 0.23). On Ubuntu arm64 the final pins were installed with the one command into a new checkout (fifth attempt; the VM was not new) and acceptance passed 14 of 14 (run `acc-20261113T021549Z-92641d`, [DEMO-21](../demo/21_core_install_ubuntu_arm64_lima.md#english)). **Not met or not shown:** (1) the allow-list holds 58 entries against the written criterion "fewer than M1" (M1: 0); the owner's decision accepts 58 as the M2 exit count in `manifest/allowlist.baseline.json`, it does not make the criterion met; (2) amd64: acceptance 11 of 12 recorded only on an emulated VM ([DEMO-22](../demo/22_core_install_ubuntu_amd64_lima.md#english); C4 success failed at the suite's 30 s wait), `scan` exited 51 there (Trivy did not finish the ember-osf-web image), and osf.io and wiki-frontend were not built on amd64; `accept --wait-scale` and `scan --trivy-timeout` exist since PR #60 and the re-run on that VM was later recorded (DEMO-22 0.7: 14 of 14 with `--wait-scale 10`; DEMO-22 0.8: the one command exit 0 in one run with a warm Docker cache); (3) the CI that needs a token with the `workflow` scope (per-OS workflows, the `images.yml` smoke test, osf.io R7-01) is not pushed; the owner decided to grant that scope; (4) no authenticated ZAP scan: under mandatory sign-in `web` is skipped. Carried items: §2d | -| M3 | 5 subsystems start; one success + one rejection case each | Report + allow-list (fewer entries than M2) | **Current** since 2026-11-13: `manifest/milestone.json` says M3 (gate `count-not-growing`) and the allow-list ceiling is the M2 count, 58 (`manifest/allowlist.baseline.json`); S1 (y-websocket) runs and passes since 2026-11-15 (run `acc-20261115T082645Z-acc61c`); S4 (OASys) runs in the full profile and passes since 2026-11-15 (run `acc-20261115T131515Z-200a84`); S2 (rdmcp, real CAS OAuth) runs in the full profile and passes since 2026-11-16 (run `acc-20261115T164109Z-1a319f`); the S3 and S5 acceptance modules still carry an unconditional `skip(reason=M3)` ([DEMO-22](../demo/22_core_install_ubuntu_amd64_lima.md#english)). Open items: §2d | +| M3 | 5 subsystems start; one success + one rejection case each | Report + allow-list (fewer entries than M2) | **Current** since 2026-11-13: `manifest/milestone.json` says M3 (gate `count-not-growing`) and the allow-list ceiling is the M2 count, 58 (`manifest/allowlist.baseline.json`); S1 (y-websocket) runs and passes since 2026-11-15 (run `acc-20261115T082645Z-acc61c`); S4 (OASys) runs in the full profile and passes since 2026-11-15 (run `acc-20261115T131515Z-200a84`); S2 (rdmcp, real CAS OAuth) runs in the full profile and passes since 2026-11-16 (run `acc-20261115T164109Z-1a319f`); S3 (aipf_mcp_data_grdm with the mock OIDC issuer, ADR-15) runs in the full profile and passes since 2026-11-16 (run `acc-20261115T162657Z-3b9bef`); the S5 acceptance module still carries an unconditional `skip(reason=M3)` ([DEMO-22](../demo/22_core_install_ubuntu_amd64_lima.md#english)). Open items: §2d | | M4 | Waterbutler, mfr, osf.io on Python 3.13 | Stop on HIGH or above; first AI red-team run | Planned | | M5 | ES2 and mongo removed; ES8 + postgres 15 or later; distributable abroad | Zero CRITICAL/HIGH; zero open AI red-team paths | **Groundwork** (§2c): PostgreSQL 18 with `db upgrade` ([ADR-12](../decisions/ADR-12_PostgreSQL_18_and_Data_Directory_Upgrade.md#english)); Elasticsearch 2, MongoDB and sharejs are gone from the installer and Elasticsearch 8 replaces ES 2 on the merged osf.io `develop` `90b10661c8dc`, 14 of 14 acceptance ([ADR-13 §6](../decisions/ADR-13_Retiring_Elasticsearch_2_and_MongoDB.md#english), [ADR-14](../decisions/ADR-14_Elasticsearch_8_with_TLS_and_Authentication.md#english)); the integration branch awaits its pull request | @@ -116,7 +117,7 @@ M3 started on 2026-11-13, when the owner closed M2 (「一括で承認、M2 完 | Authenticated ZAP scan of `web` | ⏳ Not done; no owner decision on when | | Preview from the new ember UI (API v2) | ⏳ Carried from §2b: the server-side change RDM-osf.io-dev2027 PR #14 is in osf.io `develop` ([DEMO-20](../demo/20_core_install_mac_studio.md#english)) and RDM-ember-osf-web-dev2027 PR #3 is merged; open: a check on the real ember screen after deployment — no acceptance scenario covers the ember preview yet, so success is not shown | | Publication: PR #51 (`develop` → `master`) and the default-branch switch | 🔄 PR #51 was merged on 2026-11-14 (`9ebb9e9`). PR #64 to #66 were published to `master` by PR #67 on 2026-11-14 (merge commit `2950b72`, owner decision of the same day); the default-branch switch is the owner's | -| S1–S5 subsystem scenarios (the M3 goal) | ⏳ **S1 (y-websocket) passes**, success and rejection, on 2026-11-15 on the Mac Studio stack with the default waits: the page's one room socket is `ws://127.0.0.1:/?token=`, A's sentence reaches B within 5 s while B's socket receives a y-sync document update, and the same URL without the token is closed with 4401 before any frame. Whole suite: 16 passed, 8 skipped (S2–S5), 0 failed, run `acc-20261115T082645Z-acc61c` (UTC). **S4 (OASys) passes**, success and rejection, on 2026-11-15 on the same stack rendered with `--profile full`: user_a's GRDM session takes OASys `/` to `/registration/` without a second sign-in, the project selection lists the run's project, web `/login/?next=` returns to OASys (osf_init registers the origin as PreprintProvider `oasys`; an unregistered origin answers 400), and the anonymous v1 workflow call through OASys answers 401 JSON. Whole suite in the full profile: 18 passed, 6 skipped (S2, S3, S5), 0 failed, run `acc-20261115T131515Z-200a84`; `aipf_mcp_data_grdm` restarts without its OIDC settings (M3 survey), so it was stopped and left out of the rendered file for that run only. **S2 (rdmcp) passes**, success and rejection, on 2026-11-16 (UTC 2026-11-15) on the same stack rendered with `--profile full`, with rdmcp at the head of RDM-rdmcp-dev2027 PR #1 (`9c6e0050`), which merged on 2026-11-16 as `75151e45` with the identical tree (`a2e8f206`), so the manifest now pins the merge commit: an MCP client registers, user_a signs in at the real CAS (published on 8090 beside rdmcp only; the browser's GRDM sign-in stays fakecas) and allows the application, picks the run's project on rdmcp's node-selection page, and `rdm_get_node` returns it; no token answers 401 with a Bearer challenge, user_b's project is not offered and is refused when posted (400), and nodes outside the grant are tool errors. S2 alone: run `acc-20261115T164025Z-949e7f`; whole suite in the full profile: 20 passed, 4 skipped (S3, S5), 0 failed, run `acc-20261115T164109Z-1a319f`, `aipf_mcp_data_grdm` again stopped and left out of the rendered file for that run only. S3 and S5 still carry an unconditional `skip(reason=M3)` ([DEMO-22](../demo/22_core_install_ubuntu_amd64_lima.md#english)) | +| S1–S5 subsystem scenarios (the M3 goal) | ⏳ **S1 (y-websocket) passes**, success and rejection, on 2026-11-15 on the Mac Studio stack with the default waits: the page's one room socket is `ws://127.0.0.1:/?token=`, A's sentence reaches B within 5 s while B's socket receives a y-sync document update, and the same URL without the token is closed with 4401 before any frame. Whole suite: 16 passed, 8 skipped (S2–S5), 0 failed, run `acc-20261115T082645Z-acc61c` (UTC). **S4 (OASys) passes**, success and rejection, on 2026-11-15 on the same stack rendered with `--profile full`: user_a's GRDM session takes OASys `/` to `/registration/` without a second sign-in, the project selection lists the run's project, web `/login/?next=` returns to OASys (osf_init registers the origin as PreprintProvider `oasys`; an unregistered origin answers 400), and the anonymous v1 workflow call through OASys answers 401 JSON. Whole suite in the full profile: 18 passed, 6 skipped (S2, S3, S5), 0 failed, run `acc-20261115T131515Z-200a84`; `aipf_mcp_data_grdm` restarts without its OIDC settings (M3 survey), so it was stopped and left out of the rendered file for that run only. **S2 (rdmcp) passes**, success and rejection, on 2026-11-16 (UTC 2026-11-15) on the same stack rendered with `--profile full`, with rdmcp at the head of RDM-rdmcp-dev2027 PR #1 (`9c6e0050`), which merged on 2026-11-16 as `75151e45` with the identical tree (`a2e8f206`), so the manifest now pins the merge commit: an MCP client registers, user_a signs in at the real CAS (published on 8090 beside rdmcp only; the browser's GRDM sign-in stays fakecas) and allows the application, picks the run's project on rdmcp's node-selection page, and `rdm_get_node` returns it; no token answers 401 with a Bearer challenge, user_b's project is not offered and is refused when posted (400), and nodes outside the grant are tool errors. S2 alone: run `acc-20261115T164025Z-949e7f`; whole suite in the full profile: 20 passed, 4 skipped (S3, S5), 0 failed, run `acc-20261115T164109Z-1a319f`, `aipf_mcp_data_grdm` again stopped and left out of the rendered file for that run only. **S3 (aipf_mcp_data_grdm) passes**, success and rejection, on 2026-11-16 on the same stack in the full profile, with the mock OIDC issuer `oidc_mock` ([ADR-15](../decisions/ADR-15_Mock_OIDC_Issuer_for_aipf.md#english)): user_a's PAT, stored through the component's form, lets a research-flow token create user_a's private project, which the OSF API reports (title, key, admin), and a second call returns the same node; no or a malformed bearer, another key and another audience answer 401, a missing or unknown actor 403, a non-writer agent `writer_not_allowed`, and user_b's PAT on user_a's project `node_forbidden`. aipf_mcp_data_grdm no longer restarts in a loop and has a health check on `/healthz`. Whole suite in the full profile, before S2 was enabled: 20 passed, 4 skipped (S2, S5), 0 failed, run `acc-20261115T162657Z-3b9bef` (UTC); no service was stopped for it. S5 still carries an unconditional `skip(reason=M3)` ([DEMO-22](../demo/22_core_install_ubuntu_amd64_lima.md#english)) | ### 3. Rules @@ -139,7 +140,7 @@ This board is the **single source of truth** for milestone progress: |---|---|---|---| | M1 | docs の骨格・manifest・dev2027 リポジトリ群がある。レビュー済み PR が develop にマージ | 報告+許容リスト | ✅ **完了**(PR #19 を 2026-11-05 にマージ) | | M2 | コア 6 本が Mac Studio で一発導入。Playwright 受入合格。amd64 ビルド | 報告+許容リスト(M1 より件数を減らす) | ✅ **2026-11-13 の所有者の判断で完了**(「一括で承認、M2 完了」)。書かれた条件をすべて満たしたのでは**なく**、既知の限界つきである。示したこと: Mac Studio では、以前の固定版で一発導入が以前の実機の回で成功した。マージ後の最終の固定版では、既存の Mac の環境を個別のコマンドで更新し、そのあと受入が 14 件中 14 件合格し(実行 `acc-20261112T223428Z-fda84a`)、検査が完走した(運用者の記録)([DEMO-20](../demo/20_core_install_mac_studio.md#japanese))。2026-11-14 に develop 0df4640 の新しいチェックアウトから `install-macos.sh --yes --profile core` が一回の実行で 12 分 50 秒で終了 0 を返し(Docker のキャッシュは温まった状態で、空の Docker ではない)、既定の待ちで受入が 14 件中 14 件合格した(実行 `acc-20261114T103547Z-8c60ae`、DEMO-20 0.23)。Ubuntu arm64 では、最終の固定版を新しいチェックアウトへ一つのコマンドで導入し(5 回目の試行。VM は新品ではない)、受入が 14 件中 14 件合格した(実行 `acc-20261113T021549Z-92641d`、[DEMO-21](../demo/21_core_install_ubuntu_arm64_lima.md#japanese))。**満たさなかったこと・示していないこと:** (1) 許容リストは 58 項目で、書かれた条件「M1 より少なく」(M1: 0)を満たさない。所有者の判断は 58 を `manifest/allowlist.baseline.json` の M2 の終了時の件数として受け入れたもので、条件を満たしたことにはしない。(2) amd64: 受入 12 件中 11 件はエミュレーションの VM でしか記録していない([DEMO-22](../demo/22_core_install_ubuntu_amd64_lima.md#japanese)。C4 の成功ケースは試験の 30 秒の待ちで失敗)。そこで `scan` は終了 51(Trivy が ember-osf-web のイメージを終えられなかった)。osf.io と wiki-frontend は amd64 で組んでいない。PR #60 から `accept --wait-scale` と `scan --trivy-timeout` があり、その VM での再実行はのちに記録した(DEMO-22 0.7: `--wait-scale 10` で 14 件中 14 件。DEMO-22 0.8: Docker のキャッシュが温まった状態で、一つのコマンドが一回の実行で終了 0)。(3) `workflow` スコープのトークンが要る CI(per-OS のワークフロー、`images.yml` のスモーク試験、osf.io の R7-01)は push していない。所有者はその権限を与えると判断した。(4) 認証した ZAP の検査はしていない。必須サインインの下で `web` は飛ばされる。持ち越し: §2d | -| M3 | サブシステム 5 本が起動。各 1 本の正常系+拒否ケース | 報告+許容リスト(M2 より件数を減らす) | 2026-11-13 から**現在の節目**: `manifest/milestone.json` は M3(門は `count-not-growing`)、許容リストの上限は M2 の件数 58(`manifest/allowlist.baseline.json`)。S1(y-websocket)は 2026-11-15 から動いて合格する(実行 `acc-20261115T082645Z-acc61c`)。S4(OASys)は 2026-11-15 から full プロファイルで動いて合格する(実行 `acc-20261115T131515Z-200a84`)。S2(rdmcp、本物の CAS の OAuth)は 2026-11-16 から full プロファイルで動いて合格する(実行 `acc-20261115T164109Z-1a319f`)。S3・S5 の受入モジュールは無条件の `skip(reason=M3)` をまだ持つ([DEMO-22](../demo/22_core_install_ubuntu_amd64_lima.md#japanese))。残件: §2d | +| M3 | サブシステム 5 本が起動。各 1 本の正常系+拒否ケース | 報告+許容リスト(M2 より件数を減らす) | 2026-11-13 から**現在の節目**: `manifest/milestone.json` は M3(門は `count-not-growing`)、許容リストの上限は M2 の件数 58(`manifest/allowlist.baseline.json`)。S1(y-websocket)は 2026-11-15 から動いて合格する(実行 `acc-20261115T082645Z-acc61c`)。S4(OASys)は 2026-11-15 から full プロファイルで動いて合格する(実行 `acc-20261115T131515Z-200a84`)。S2(rdmcp、本物の CAS の OAuth)は 2026-11-16 から full プロファイルで動いて合格する(実行 `acc-20261115T164109Z-1a319f`)。S3(模擬の OIDC 発行者つきの aipf_mcp_data_grdm。ADR-15)は 2026-11-16 から full プロファイルで動いて合格する(実行 `acc-20261115T162657Z-3b9bef`)。S5 の受入モジュールは無条件の `skip(reason=M3)` をまだ持つ([DEMO-22](../demo/22_core_install_ubuntu_amd64_lima.md#japanese))。残件: §2d | | M4 | waterbutler・mfr・osf.io を Python 3.13 へ | HIGH 以上で停止。AI レッドチーム初回 | 計画中 | | M5 | ES2・mongo 廃止。ES8+postgres 15 以降。海外配布可 | CRITICAL/HIGH ゼロ。AI レッドチームの未解決経路ゼロ | **下準備**(§2c): `db upgrade` 付きの PostgreSQL 18([ADR-12](../decisions/ADR-12_PostgreSQL_18_and_Data_Directory_Upgrade.md#japanese))。Elasticsearch 2・MongoDB・sharejs はインストーラーから消え、マージ後の osf.io `develop` `90b10661c8dc` で Elasticsearch 8 が ES 2 に替わり、受入 14 件中 14 件([ADR-13 §6](../decisions/ADR-13_Retiring_Elasticsearch_2_and_MongoDB.md#japanese)、[ADR-14](../decisions/ADR-14_Elasticsearch_8_with_TLS_and_Authentication.md#japanese))。統合ブランチは PR を待つ | @@ -205,7 +206,7 @@ M3 は、所有者が M2 を閉じた 2026-11-13 に始まった(「一括で | `web` の認証した ZAP の検査 | ⏳ 未実施。時期についての所有者の判断は無い | | 新しい ember の画面(API v2)からのプレビュー | ⏳ §2b から持ち越し。サーバー側の変更 RDM-osf.io-dev2027 PR #14 は osf.io の `develop` にあり([DEMO-20](../demo/20_core_install_mac_studio.md#japanese))、RDM-ember-osf-web-dev2027 PR #3 はマージ済み。残り: 配備後の ember の実画面での確認。ember のプレビューを試す受入シナリオはまだ無く、成功は示していない | | 公開: PR #51(`develop` → `master`)と既定ブランチの切替 | 🔄 PR #51 は 2026-11-14 にマージした(`9ebb9e9`)。PR #64〜#66 は、同日の所有者の判断により 2026-11-14 に PR #67 で `master` に公開した(マージコミット `2950b72`)。既定ブランチの切替は所有者が行う | -| S1〜S5 のサブシステムのシナリオ(M3 のゴール) | ⏳ 2026-11-15 に Mac Studio のスタックで、既定の待ちで **S1(y-websocket)が成功・拒否とも合格**。ページが開く唯一の部屋のソケットは `ws://127.0.0.1:/?token=`、A の一文は B のソケットが y-sync の文書更新を受け取る間に 5 秒以内に B に届き、トークンを除いた同じ URL はフレームより先に 4401 で閉じられる。スイート全体は 16 件合格、スキップ 8 件(S2〜S5)、失敗 0、実行 `acc-20261115T082645Z-acc61c`(UTC)。2026-11-15 に同じスタックを `--profile full` で描画し、**S4(OASys)が成功・拒否とも合格**。user_a の GRDM のセッションで OASys の `/` は二度目のサインインなしに `/registration/` へ進み、プロジェクト選択に実行中プロジェクトが並び、web の `/login/?next=` は OASys へ戻る(osf_init がその起点を PreprintProvider `oasys` に登録する。登録していない起点は 400)。OASys 経由の匿名の v1 ワークフロー呼び出しは JSON の 401。full プロファイルでのスイート全体は 18 件合格、スキップ 6 件(S2・S3・S5)、失敗 0、実行 `acc-20261115T131515Z-200a84`。`aipf_mcp_data_grdm` は OIDC の設定が無く再起動を繰り返す(M3 調査)ので、この実行に限り止めて描画したファイルから外した。2026-11-16(UTC では 2026-11-15)に同じスタックを `--profile full` で描画し、rdmcp を RDM-rdmcp-dev2027 PR #1 の先頭(`9c6e0050`)にして(PR #1 は 2026-11-16 に同じツリー `a2e8f206` の `75151e45` としてマージされ、manifest は今そのマージコミットを固定する)、**S2(rdmcp)が成功・拒否とも合格**。MCP クライアントが登録し、user_a が本物の CAS(rdmcp の横でだけ 8090 に公開。ブラウザの GRDM へのサインインは fakecas のまま)でサインインしてアプリを許可し、rdmcp のノード選択画面で実行中プロジェクトを選び、`rdm_get_node` がそれを返す。トークンなしは Bearer の challenge 付きの 401、user_b のプロジェクトは選択肢に出ず、送れば 400、許可の外のノードは道具のエラー。S2 だけの実行 `acc-20261115T164025Z-949e7f`。full プロファイルでのスイート全体は 20 件合格、スキップ 4 件(S3・S5)、失敗 0、実行 `acc-20261115T164109Z-1a319f`。この実行に限り `aipf_mcp_data_grdm` を再び止めて描画したファイルから外した。S3・S5 は無条件の `skip(reason=M3)` をまだ持つ([DEMO-22](../demo/22_core_install_ubuntu_amd64_lima.md#japanese)) | +| S1〜S5 のサブシステムのシナリオ(M3 のゴール) | ⏳ 2026-11-15 に Mac Studio のスタックで、既定の待ちで **S1(y-websocket)が成功・拒否とも合格**。ページが開く唯一の部屋のソケットは `ws://127.0.0.1:/?token=`、A の一文は B のソケットが y-sync の文書更新を受け取る間に 5 秒以内に B に届き、トークンを除いた同じ URL はフレームより先に 4401 で閉じられる。スイート全体は 16 件合格、スキップ 8 件(S2〜S5)、失敗 0、実行 `acc-20261115T082645Z-acc61c`(UTC)。2026-11-15 に同じスタックを `--profile full` で描画し、**S4(OASys)が成功・拒否とも合格**。user_a の GRDM のセッションで OASys の `/` は二度目のサインインなしに `/registration/` へ進み、プロジェクト選択に実行中プロジェクトが並び、web の `/login/?next=` は OASys へ戻る(osf_init がその起点を PreprintProvider `oasys` に登録する。登録していない起点は 400)。OASys 経由の匿名の v1 ワークフロー呼び出しは JSON の 401。full プロファイルでのスイート全体は 18 件合格、スキップ 6 件(S2・S3・S5)、失敗 0、実行 `acc-20261115T131515Z-200a84`。`aipf_mcp_data_grdm` は OIDC の設定が無く再起動を繰り返す(M3 調査)ので、この実行に限り止めて描画したファイルから外した。2026-11-16(UTC では 2026-11-15)に同じスタックを `--profile full` で描画し、rdmcp を RDM-rdmcp-dev2027 PR #1 の先頭(`9c6e0050`)にして(PR #1 は 2026-11-16 に同じツリー `a2e8f206` の `75151e45` としてマージされ、manifest は今そのマージコミットを固定する)、**S2(rdmcp)が成功・拒否とも合格**。MCP クライアントが登録し、user_a が本物の CAS(rdmcp の横でだけ 8090 に公開。ブラウザの GRDM へのサインインは fakecas のまま)でサインインしてアプリを許可し、rdmcp のノード選択画面で実行中プロジェクトを選び、`rdm_get_node` がそれを返す。トークンなしは Bearer の challenge 付きの 401、user_b のプロジェクトは選択肢に出ず、送れば 400、許可の外のノードは道具のエラー。S2 だけの実行 `acc-20261115T164025Z-949e7f`。full プロファイルでのスイート全体は 20 件合格、スキップ 4 件(S3・S5)、失敗 0、実行 `acc-20261115T164109Z-1a319f`。この実行に限り `aipf_mcp_data_grdm` を再び止めて描画したファイルから外した。2026-11-16 に同じスタックの full プロファイルで、模擬の OIDC 発行者 `oidc_mock`([ADR-15](../decisions/ADR-15_Mock_OIDC_Issuer_for_aipf.md#japanese))とともに **S3(aipf_mcp_data_grdm)が成功・拒否とも合格**。部品の欄から預けた user_a の PAT で、研究フローのトークンが user_a の非公開プロジェクトを作り、OSF API がそれを報告し(題名・目印・admin)、二度目の呼び出しは同じノードを返す。ベアラーなし・形の違うベアラー・別の鍵・別の宛先は 401、代理者なし・未知の代理者は 403、書き手でないエージェントは `writer_not_allowed`、user_a のプロジェクトでの user_b の PAT は `node_forbidden`。aipf_mcp_data_grdm は再起動を繰り返さなくなり、`/healthz` の健康確認を持つ。S2 を有効にする前の full プロファイルでのスイート全体は 20 件合格、スキップ 4 件(S2・S5)、失敗 0、実行 `acc-20261115T162657Z-3b9bef`(UTC)。この実行では止めたサービスは無い。S5 は無条件の `skip(reason=M3)` をまだ持つ([DEMO-22](../demo/22_core_install_ubuntu_amd64_lima.md#japanese)) | ### 3. 規則 diff --git a/installer/acceptance/README.md b/installer/acceptance/README.md index 65110fbe..9f137a65 100644 --- a/installer/acceptance/README.md +++ b/installer/acceptance/README.md @@ -5,7 +5,8 @@ **Jump:** [English](#english) · [日本語](#japanese) -- Version / 版: 0.7 — 2026-11-16. **Learned:** a scenario that signs in at a second server needs a second credential: S2 (rdmcp, full profile) signs user_a in at the real CAS, which checks the OSF account's own password, so `accept --seed` now hands user_a's random OSF password to pytest as GRDM_ACCEPT_USER_A_OSF_PASSWORD (masked like every run secret), as it already did for the administrator. / 二つ目のサーバーでサインインするシナリオには、二つ目の資格が要る。S2(rdmcp、full プロファイル)は user_a を本物の CAS でサインインさせ、CAS は OSF アカウント自身のパスワードを確かめる。そこで `accept --seed` は、管理者のときと同じく、user_a の無作為な OSF パスワードを GRDM_ACCEPT_USER_A_OSF_PASSWORD として pytest に渡す(実行のほかの秘密と同じく伏せる)。 +- Version / 版: 0.8 — 2026-11-16. **Learned:** a scenario may hold a secret of the stack it tests when the stack's design makes the test a party: S3 signs its tokens with the mock issuer's key read from runtime/.env, exactly as the issuer would, and reads the issuer and client names from the installed template, so neither the key nor a name is copied into the test. / スタックの設計が試験を当事者にするときは、シナリオがそのスタックの秘密を持ってよい。S3 は runtime/.env から読んだ模擬発行者の鍵で、発行者と同じようにトークンに署名し、発行者とクライアントの名前は導入済みのテンプレートから読む。鍵も名前も試験に写さない。 + 0.7 — 2026-11-16. **Learned:** a scenario that signs in at a second server needs a second credential: S2 (rdmcp, full profile) signs user_a in at the real CAS, which checks the OSF account's own password, so `accept --seed` now hands user_a's random OSF password to pytest as GRDM_ACCEPT_USER_A_OSF_PASSWORD (masked like every run secret), as it already did for the administrator. / 二つ目のサーバーでサインインするシナリオには、二つ目の資格が要る。S2(rdmcp、full プロファイル)は user_a を本物の CAS でサインインさせ、CAS は OSF アカウント自身のパスワードを確かめる。そこで `accept --seed` は、管理者のときと同じく、user_a の無作為な OSF パスワードを GRDM_ACCEPT_USER_A_OSF_PASSWORD として pytest に渡す(実行のほかの秘密と同じく伏せる)。 0.6 — 2026-11-15. **Learned:** a port in runtime/.env does not prove that its service runs: a render back to `core` keeps GRDM_PORT_OASYS, so a full-profile scenario asks the profile (`profile` fixture) before it asks the port. / runtime/.env にポートがあっても、そのサービスが動いている証拠にはならない。`core` に描画し直しても GRDM_PORT_OASYS は残るので、full プロファイルのシナリオはポートより先にプロファイルを尋ねる(フィクスチャ `profile`)。 0.5 — 2026-11-15. **Learned:** a helper that two scenarios need belongs to the scenario that verified it, imported like C3's `upload`, not copied: S1 reuses C4's wiki editor helpers and the room locator moved to `selectors.py`, so a change of the wiki page is fixed in one place. / 二つのシナリオが要る補助は、それを確かめたシナリオに置き、C3 の `upload` と同じく取り込んで使い、写さない。S1 は C4 の wiki エディタの補助を使い、部屋のロケータは `selectors.py` に移したので、wiki の画面が変わっても一か所で直せる。 0.4 — 2026-11-12. **Learned:** a helper that mints a one-time code must remember the step it used, or two logins in the same 30 seconds fail the second for a reason that looks like a wrong code. / ワンタイムコードを作る補助は、使った区間を覚えていなければならない。同じ 30 秒の間の二度目のログインが、コードの誤りに見える理由で失敗するからである。 @@ -43,7 +44,7 @@ files[7]{path,purpose}: totp.py,"RFC 6238 one-time codes for the seeded administrator of C6 (PR #10 accepts a code once per 30 s step, so next_code waits for a fresh step); pure functions, unit-tested" selectors.py,"every locator and URL path in one table; TODO(M2) marks entries not yet verified on the real app" test_c1_login.py … test_c7_file_editor.py,"core scenarios C1–C7; each has test_*_success and test_*_rejection; marker core" - test_s1_y_websocket.py … test_s5_shibboleth.py,"subsystem scenarios S1–S5; marker subsystem; S1 (y-websocket, core profile) runs; S2 (rdmcp, real CAS OAuth) and S4 (OASys) run in the full profile and skip in any other (the `profile` fixture reads GRDM_PROFILE); S3 and S5 are skipped with reason M3" + test_s1_y_websocket.py … test_s5_shibboleth.py,"subsystem scenarios S1–S5; marker subsystem; S1 (y-websocket, core profile) runs; S2 (rdmcp, real CAS OAuth), S3 (aipf_mcp_data_grdm, signing its tokens with the mock issuer's key OIDC_MOCK_SIGNING_KEY from runtime/.env, ADR-15) and S4 (OASys) run in the full profile and skip in any other (the `profile` fixture reads GRDM_PROFILE); S5 is skipped with reason M3" pytest.ini,"markers and default options" ``` @@ -92,7 +93,7 @@ files[7]{path,purpose}: totp.py,"C6 の種データの管理者のための RFC 6238 のワンタイムコード(PR #10 は 30 秒の区間ごとに一度しかコードを通さないので、next_code は新しい区間まで待つ)。純粋な関数で、単体試験あり" selectors.py,"ロケータと URL パスを一つの表に。実機で未確認のものは TODO(M2)" test_c1_login.py … test_c7_file_editor.py,"コアシナリオ C1〜C7。test_*_success と test_*_rejection。マーカー core" - test_s1_y_websocket.py … test_s5_shibboleth.py,"サブシステムのシナリオ S1〜S5。マーカー subsystem。S1(y-websocket、core プロファイル)は実行する。S2(rdmcp、本物の CAS の OAuth)と S4(OASys)は full プロファイルで実行し、ほかではスキップする(フィクスチャ `profile` が GRDM_PROFILE を読む)。S3・S5 は理由 M3 でスキップ" + test_s1_y_websocket.py … test_s5_shibboleth.py,"サブシステムのシナリオ S1〜S5。マーカー subsystem。S1(y-websocket、core プロファイル)は実行する。S2(rdmcp、本物の CAS の OAuth)、S3(aipf_mcp_data_grdm。runtime/.env の模擬発行者の鍵 OIDC_MOCK_SIGNING_KEY でトークンに署名する。ADR-15)、S4(OASys)は full プロファイルで実行し、ほかではスキップする(フィクスチャ `profile` が GRDM_PROFILE を読む)。S5 は理由 M3 でスキップ" pytest.ini,"マーカーと既定オプション" ``` diff --git a/installer/acceptance/selectors.py b/installer/acceptance/selectors.py index fa38c74d..5af22ccb 100644 --- a/installer/acceptance/selectors.py +++ b/installer/acceptance/selectors.py @@ -27,6 +27,8 @@ "node_wikis": "/v2/nodes/{node_id}/wikis/", "node_contributors": "/v2/nodes/{node_id}/contributors/", "institution_users": "/v2/institutions/{institution_id}/users/", # TODO(M2): confirm the admin-only listing path + "tokens": "/v2/tokens/", # personal access tokens; the secret is attributes.token_id, only in the 201 answer (S3) + "token": "/v2/tokens/{token_id}/", } WATERBUTLER = { diff --git a/installer/acceptance/test_s3_aipf.py b/installer/acceptance/test_s3_aipf.py index 8285ac7b..30fab4ac 100644 --- a/installer/acceptance/test_s3_aipf.py +++ b/installer/acceptance/test_s3_aipf.py @@ -1,113 +1,303 @@ -"""S3 — aipf_mcp_data_grdm (SPEC-04 §5). Subsystem scenario; runs at M3. +"""S3 — aipf_mcp_data_grdm (SPEC-04 §5). Subsystem scenario of M3. Services: api, aipf_mcp_data_grdm, oidc_mock (full profile). -Design (skip-marked "M3" until the service is installed, published on 127.0.0.1 and its tool name is -verified). aipf_mcp_data_grdm is the AIPF (AI platform) data connector that answers MCP data queries -about GRDM projects; like S2 it is driven with the guarded `api_context()` fixture, no browser. +aipf_mcp_data_grdm is the AI platform's component that writes research records into GRDM. Its MCP endpoint (/mcp) +trusts a bearer JWT of the AI platform's OIDC issuer, which this environment replaces with `oidc_mock` (ADR-15): the +test signs the tokens itself with the issuer's key from runtime/.env (OIDC_MOCK_SIGNING_KEY), exactly as the issuer +would. GRDM is reached with the user's own personal access token (PAT), which the user hands to the component +through a form (POST /grdm/token?t=) and the component keeps encrypted. It has no read tool; what it +does is write, so the scenario checks a write. -* success — user_a's token calls the data-query tool for the run's project and receives the project - metadata (id, title, creator) that the OSF API also reports for the same node. -* rejection — user_b's token asks for user_a's private project id: the answer is HTTP 403, or an - empty result that carries an explicit error code (`forbidden` / `not_permitted`); a silent empty - list is a failure, because it would hide the permission check (SPEC-04 §5 S3). +* success — user_a creates a PAT in GRDM; a token for user_a from the AIPF screen (azp `grdm-dev-webui`, no actor) + asks `grdm_token_form` for a form token, and POST /grdm/token stores the PAT (the component checks it with GRDM's + /v2/users/me/ and answers user_a's name); `grdm_token_status` reports it. A token of the research-flow service + (act.sub `grdm-dev-research-flow`, the one writer) calls `ensure_project`: a new private project, whose id, title, + description key and admin permission the OSF API v2 reports to user_a's own session; a second call returns the same + node with created=false. +* rejection — no bearer and a malformed bearer -> 401; a token signed by another key -> 401; a token whose audience is + another component -> 401; a valid signature without an actor from an unknown client -> 403 `actor_missing`, with an + actor not on the list -> 403 `actor_not_allowed`; an allowed agent that is not the writer -> `writer_not_allowed`; + a forged form token -> 403; and user_b's PAT, registered the same way, used by the writer on user_a's private + project -> `node_forbidden`, with nothing written. -Prerequisite tracked as follow-up: no host_port yet (templates/aipf_mcp_data_grdm.json exposes 8101 -internally). The test skips with an explicit reason when GRDM_PORT_AIPF_MCP_DATA_GRDM is absent. +`write_record` is not called: it uploads through the WaterButler links the API returns, which are browser-facing +(127.0.0.1:) and do not reach WaterButler from the component's container (open point of SPEC-04 §5 S3). + +The scenario skips unless runtime/.env says GRDM_PROFILE=full and has GRDM_PORT_AIPF_MCP_DATA_GRDM; the session fixture +`aipf_url` decides that before any other S3 fixture, so a missing OIDC_MOCK_SIGNING_KEY fails only an eligible full +environment and never turns the skip of a core or minimum one into an error. Issuer, audience +and the three client names are read from the installed template, so test and stack cannot drift apart. + +Evidence: no token, PAT or form token is ever put into an assertion message or a property; only statuses and codes. """ from __future__ import annotations import json +import os +import re +import time +from pathlib import Path import pytest from acceptance import selectors as sel -from acceptance.conftest import api +from acceptance.conftest import _read_env_file, api -pytestmark = [pytest.mark.subsystem, pytest.mark.scenario("S3", "aipf_mcp_data_grdm"), pytest.mark.skip(reason="M3")] +pytestmark = [pytest.mark.subsystem, pytest.mark.scenario("S3", "aipf_mcp_data_grdm")] -MCP_PATH = "/mcp" # TODO(M3): endpoint path of aipf_mcp_data_grdm -MCP_PROTOCOL = "2025-06-18" # TODO(M3) -QUERY_TOOL = "get_project_metadata" # TODO(M3): tool name; arguments {"project_id": ""} -EXPLICIT_DENIAL_CODES = {"forbidden", "not_permitted", "permission_denied"} # TODO(M3): the service's code -TOKEN_SCOPES = ["osf.full_read"] # TODO(M3) +MCP_PATH = "/mcp" +FORM_PATH = "/grdm/token" +MCP_PROTOCOL = "2025-06-18" +PAT_SCOPES = "osf.full_read osf.full_write" # ensure_project creates a node: the PAT needs write JSON_HEADERS = {"Content-Type": "application/json", "Accept": "application/json, text/event-stream"} +_ERROR_CODE = re.compile(r"(?:Error executing tool \S+: )?([a-z][a-z_]+): ") + + +def _template() -> dict: + """The aipf_mcp_data_grdm template the stack was rendered from (installed with grdm_dev).""" + import grdm_dev + + return json.loads((Path(grdm_dev.__file__).parent / "templates" / "aipf_mcp_data_grdm.json").read_text(encoding="utf-8")) + + +TEMPLATE_ENV = _template()["environment"] +ISSUER, AUDIENCE = TEMPLATE_ENV["OIDC_ISSUER"], TEMPLATE_ENV["OIDC_AUDIENCE"] +WEBUI = TEMPLATE_ENV["ALLOWED_DIRECT_CLIENTS"].split(",")[0] +WRITER = TEMPLATE_ENV["WRITE_ACTORS"].split(",")[0] +AGENT = next(a for a in TEMPLATE_ENV["ALLOWED_ACTORS"].split(",") if a != WRITER) + + +# --- environment --------------------------------------------------------------------------------- + +def _aipf_or_skip(ports: dict[str, int], profile: str) -> str: + """The component's URL, or a skip outside the full profile or without its port (SPEC-04 §5 S3).""" + if profile != "full": + pytest.skip(f"GRDM_PROFILE is {profile!r}: aipf_mcp_data_grdm runs only in the full profile (compose render --profile full)") + if "AIPF_MCP_DATA_GRDM" not in ports: + pytest.skip("GRDM_PORT_AIPF_MCP_DATA_GRDM is not in runtime/.env: the service is not published on 127.0.0.1") + return f"http://127.0.0.1:{ports['AIPF_MCP_DATA_GRDM']}" + + +@pytest.fixture(scope="session") +def aipf_url(ports, profile) -> str: + """Eligibility first: `signer` depends on this fixture and both tests request it as their first argument. It is + session-scoped like `ports`, `profile` and pytest-playwright's `browser`, and pytest sets up fixtures of one scope + in argument order, so a core or minimum environment, or a full one without the port, skips before the key is read + or a browser starts (PR #86 review r1 R1-01: the key check used to run first and turned the skip into an error).""" + return _aipf_or_skip(ports, profile) + + +@pytest.fixture(scope="session") +def signer(aipf_url): + """sign(subject, **claims) -> a JWT of oidc_mock's key; also .es256 for forging tests. Only in an eligible full + environment (after `aipf_url`), where a missing key is a failure: the stack cannot work without it.""" + from grdm_dev import es256 + + value = _read_env_file(os.environ.get("GRDM_ENV_FILE")).get("OIDC_MOCK_SIGNING_KEY", "") + if not value: + pytest.fail("OIDC_MOCK_SIGNING_KEY is not in runtime/.env although the full profile publishes aipf_mcp_data_grdm: " + "run `grdm-dev secrets init --missing`, then compose render and up the full profile") + jwk = es256.decode_key(value) + + def sign(subject: str, audience: str = AUDIENCE, key: dict | None = None, **claims) -> str: + return es256.mint_jwt(key or jwk, es256.standard_claims(ISSUER, audience, subject, **claims)) + + sign.es256 = es256 + return sign + +# --- MCP over streamable HTTP -------------------------------------------------------------------- -def _port_or_skip(ports: dict[str, int], key: str) -> int: - if key not in ports: - pytest.skip(f"GRDM_PORT_{key} is not in runtime/.env: the service is not published on 127.0.0.1 (follow-up: host_port in templates/{key.lower()}.json)") - return ports[key] +class Mcp: + def __init__(self, client, base: str): + self.client, self.base, self._id = client, base, 0 + self.last_error = "" + def post(self, body: dict, token: str | None = None, session: str | None = None): + headers = dict(JSON_HEADERS) + if token is not None: + headers["Authorization"] = f"Bearer {token}" + if session: + headers["mcp-session-id"] = session + return self.client.post(self.base + MCP_PATH, data=json.dumps(body), headers=headers) -def _rpc(client, url: str, method: str, params: dict | None = None, token: str | None = None, rpc_id: int = 1): - headers = dict(JSON_HEADERS) - if token: - headers["Authorization"] = f"Bearer {token}" - return client.post(url, data=json.dumps({"jsonrpc": "2.0", "id": rpc_id, "method": method, "params": params or {}}), headers=headers) + def initialize(self, token: str | None): + self._id += 1 + return self.post({"jsonrpc": "2.0", "id": self._id, "method": "initialize", + "params": {"protocolVersion": MCP_PROTOCOL, "capabilities": {}, + "clientInfo": {"name": "grdm-dev acceptance", "version": "S3"}}}, token) + def session(self, token: str) -> str: + resp = self.initialize(token) + assert resp.status == 200, f"initialize answered {resp.status}" + sid = resp.headers.get("mcp-session-id", "") + assert self.post({"jsonrpc": "2.0", "method": "notifications/initialized"}, token, sid).status in (200, 202) + return sid -def _personal_token(api_write, context, api_url: str, name: str): - resp = api_write(context, "POST", api_url + sel.API.get("tokens", "/v2/tokens/"), { - "data": {"type": "tokens", "attributes": {"name": name, "scopes": " ".join(TOKEN_SCOPES)}} - }) - assert resp.status == 201, f"token create failed: {resp.status} {resp.text()[:300]}" + def call(self, token: str, session: str, name: str, arguments: dict) -> tuple[dict, str]: + """(payload, error code): the tool's JSON result, or the code of a tool error (`node_forbidden`, ...).""" + self._id += 1 + resp = self.post({"jsonrpc": "2.0", "id": self._id, "method": "tools/call", "params": {"name": name, "arguments": arguments}}, + token, session) + assert resp.status == 200, f"tools/call {name} answered {resp.status}" + result = _rpc_result(resp, self._id) + text = next((b.get("text", "") for b in result.get("content", []) if b.get("type") == "text"), "") + if result.get("isError"): + m = _ERROR_CODE.match(text) + self.last_error = text[:200] # code and message of the component, e.g. "grdm_error: ... (429)"; never a token + return {}, m.group(1) if m else "unparsed_error" + if result.get("structuredContent"): + return result["structuredContent"], "" + return json.loads(text) if text else {}, "" + + +def _rpc_result(resp, rpc_id: int) -> dict: + """The `result` of the JSON-RPC answer `rpc_id`, from a JSON body or a text/event-stream of `data:` lines.""" + if "text/event-stream" in (resp.headers.get("content-type") or ""): + messages = [json.loads(line[5:]) for line in resp.text().splitlines() if line.startswith("data:")] + else: + messages = [resp.json()] + for msg in messages: + if msg.get("id") == rpc_id: + assert "error" not in msg, f"JSON-RPC error {msg['error'].get('code')}" + return msg.get("result") or {} + raise AssertionError(f"no answer to request {rpc_id} in {len(messages)} message(s)") + + +def _settle(*contexts) -> None: + """Stop the signed-in pages' own API traffic and let the API's per-user burst window (10 requests a second, + api/base/settings/defaults.py) pass: the component calls GRDM with the same user's PAT, and the dashboard's + requests had used up the second when ensure_project was answered 429 (first live run).""" + for ctx in contexts: + for page in ctx.pages: + page.goto("about:blank") + time.sleep(1.2) + + +def _status_and_code(resp) -> tuple[int, str]: + try: + return resp.status, str(resp.json().get("error", "")) + except ValueError: + return resp.status, "" + + +# --- GRDM side ----------------------------------------------------------------------------------- + +def _create_pat(api_write, context, api_url: str, name: str) -> tuple[str, str]: + resp = api_write(context, "POST", api_url + sel.API["tokens"], { + "data": {"type": "tokens", "attributes": {"name": name, "scopes": PAT_SCOPES}}}) + assert resp.status == 201, f"creating a personal access token answered {resp.status}" data = resp.json()["data"] - return data["attributes"]["token_id"], data["id"] # TODO(M3): attribute name of the secret + return data["attributes"]["token_id"], data["id"] -def _call_tool(client, url: str, token: str, project_id: str): - assert _rpc(client, url, "initialize", {"protocolVersion": MCP_PROTOCOL, "capabilities": {}, - "clientInfo": {"name": "grdm-dev acceptance", "version": "S3"}}, token).status == 200 - return _rpc(client, url, "tools/call", {"name": QUERY_TOOL, "arguments": {"project_id": project_id}}, token, rpc_id=2) +def _register_pat(mcp: Mcp, client, sign, subject: str, pat: str) -> dict: + """The AIPF screen's flow: form token from the MCP tool, then the PAT straight to /grdm/token (not through MCP).""" + ui = sign(subject, azp=WEBUI) + form, code = mcp.call(ui, mcp.session(ui), "grdm_token_form", {}) + assert not code and form.get("form_token"), f"grdm_token_form failed: {code or 'no form_token'}" + resp = client.post(f"{mcp.base}{FORM_PATH}?t={form['form_token']}", data=json.dumps({"pat": pat}), + headers={"Content-Type": "application/json"}) + assert resp.status == 200, f"POST {FORM_PATH} answered {_status_and_code(resp)}" + return resp.json() -def _tool_payload(resp) -> dict: - """The structured content of a tools/call answer (or its first text block parsed as JSON).""" - body = resp.json() - result = body.get("result") or {} - if result.get("structuredContent"): - return result["structuredContent"] - for block in result.get("content", []): - if block.get("type") == "text": - try: - return json.loads(block["text"]) - except ValueError: - return {"text": block["text"]} - return {"error": body.get("error")} +def _forget(mcp: Mcp, sign, subject: str) -> None: + ui = sign(subject, azp=WEBUI) + mcp.call(ui, mcp.session(ui), "grdm_token_delete", {}) @pytest.mark.case("success") -def test_s3_aipf_success(logged_in_page, project, api_url, api_context, api_write, ports, record_property): - """The MCP data query returns the run's project metadata, consistent with the OSF API.""" - url = f"http://127.0.0.1:{_port_or_skip(ports, 'AIPF_MCP_DATA_GRDM')}{MCP_PATH}" - secret, token_id = _personal_token(api_write, logged_in_page.context, api_url, "acceptance S3 user_a") +def test_s3_aipf_success(aipf_url, signer, logged_in_page, api_url, api_context, api_write, run_id, record_property): + """user_a's PAT, stored through the form, lets the research-flow service create user_a's project, as the OSF API shows.""" + base = aipf_url + ctx = logged_in_page.context + _settle(ctx) + me = api(ctx).get(api_url + sel.API["me"]).json()["data"] + subject, key = f"s3-{run_id}-user-a", f"grdm-dev-s3-{run_id}" + pat, pat_id = _create_pat(api_write, ctx, api_url, f"acceptance S3 {run_id} user_a") client = api_context() + mcp = Mcp(client, base) + node_id = None try: - resp = _call_tool(client, url, secret, project) - record_property("http_status", resp.status) - assert resp.status == 200, f"{resp.status} {resp.text()[:300]}" - payload = _tool_payload(resp) - node = api(logged_in_page.context).get(api_url + sel.API["node"].format(node_id=project)).json()["data"] - assert payload.get("id") == project, payload - assert payload.get("title") == node["attributes"]["title"] == "acceptance project", payload + stored = _register_pat(mcp, client, signer, subject, pat) + assert stored == {"registered": True, "grdm_user": me["attributes"]["full_name"]}, f"registration answered {sorted(stored)}" + ui = signer(subject, azp=WEBUI) + status, code = mcp.call(ui, mcp.session(ui), "grdm_token_status", {}) + assert not code and status.get("registered") is True and status.get("grdm_user") == me["attributes"]["full_name"], code or sorted(status) + + writer = signer(subject, act={"sub": WRITER}, azp=WRITER) + sid = mcp.session(writer) + time.sleep(1.2) # the registration and status calls above used this user's API second too + made, code = mcp.call(writer, sid, "ensure_project", {"title": f"S3 {run_id}", "key": key, "description": "acceptance S3"}) + assert not code, f"ensure_project failed: {mcp.last_error}" + node_id = made.get("node_id") + record_property("created", made.get("created")) + assert made.get("created") is True and node_id, f"ensure_project did not create a project: {sorted(made)}" + + node = api(ctx).get(api_url + sel.API["node"].format(node_id=node_id)) + record_property("http_status", node.status) + assert node.status == 200, f"user_a cannot read the project the component created: {node.status}" + attrs = node.json()["data"]["attributes"] + assert attrs["title"] == f"S3 {run_id}" == made.get("title"), "title differs between the component and the OSF API" + assert key in (attrs.get("description") or ""), "the research-flow key is not in the project description" + assert attrs["public"] is False, "the component created a public project" + assert "admin" in (attrs.get("current_user_permissions") or []), "user_a is not the project's administrator" + + again, code = mcp.call(writer, sid, "ensure_project", {"title": f"S3 {run_id} again", "key": key}) + assert not code and again.get("node_id") == node_id and again.get("created") is False, \ + f"a second ensure_project did not return the same project: {code or again.get('created')}" finally: - api_write(logged_in_page.context, "DELETE", api_url + sel.API.get("token", "/v2/tokens/{token_id}/").format(token_id=token_id)) + try: + _forget(mcp, signer, subject) + finally: + if node_id: + api_write(ctx, "DELETE", api_url + sel.API["node"].format(node_id=node_id)) + api_write(ctx, "DELETE", api_url + sel.API["token"].format(token_id=pat_id)) @pytest.mark.case("rejection") -def test_s3_aipf_rejection(context_b, project, api_url, api_context, api_write, ports, record_property): - """A query for a project id user_b cannot see -> 403, or empty with an explicit denial code.""" - url = f"http://127.0.0.1:{_port_or_skip(ports, 'AIPF_MCP_DATA_GRDM')}{MCP_PATH}" - secret, token_id = _personal_token(api_write, context_b, api_url, "acceptance S3 user_b") +def test_s3_aipf_rejection(aipf_url, signer, logged_in_page, context_b, project, api_url, api_context, api_write, run_id, + record_property): + """Bad, foreign or actor-less tokens are refused at the door; user_b's stored PAT cannot write into user_a's project.""" + base = aipf_url + _settle(logged_in_page.context, context_b) client = api_context() + mcp = Mcp(client, base) + subject_b = f"s3-{run_id}-user-b" + other_key = signer.es256.decode_key(signer.es256.generate_signing_key()) + + refused = { + "no_bearer": (mcp.initialize(None), 401, "unauthenticated"), + "malformed": (mcp.initialize("not-a-jwt"), 401, "invalid_token"), + "other_key": (mcp.initialize(signer(subject_b, key=other_key, azp=WEBUI)), 401, "invalid_token"), + "other_audience": (mcp.initialize(signer(subject_b, audience="another-component", azp=WEBUI)), 401, "wrong_audience"), + "no_actor": (mcp.initialize(signer(subject_b, azp="unknown-client")), 403, "actor_missing"), + "unknown_actor": (mcp.initialize(signer(subject_b, act={"sub": "unknown-agent"})), 403, "actor_not_allowed"), + } + seen = {name: _status_and_code(resp) for name, (resp, _, _) in refused.items()} + record_property("refused", json.dumps(seen, sort_keys=True)) + for name, (_, status, code) in refused.items(): + assert seen[name] == (status, code), f"{name}: expected {status} {code}, got {seen[name]}" + + agent = signer(subject_b, act={"sub": AGENT}) + _, code = mcp.call(agent, mcp.session(agent), "ensure_project", {"title": "S3 refused", "key": f"grdm-dev-s3-{run_id}-b"}) + assert code == "writer_not_allowed", f"an agent that is not the research-flow service could write: {code or 'no error'}" + + forged = client.post(f"{base}{FORM_PATH}?t=forged", data=json.dumps({"pat": "x" * 40}), headers={"Content-Type": "application/json"}) + assert _status_and_code(forged) == (403, "bad_form_token"), f"a forged form token got {_status_and_code(forged)}" + + pat_b, pat_b_id = _create_pat(api_write, context_b, api_url, f"acceptance S3 {run_id} user_b") try: - resp = _call_tool(client, url, secret, project) - record_property("http_status", resp.status) - if resp.status == 403: - return - assert resp.status == 200, f"unexpected {resp.status}: {resp.text()[:300]}" - payload = _tool_payload(resp) - code = str((payload.get("error") or {}).get("code") or payload.get("code") or "").lower() - assert code in EXPLICIT_DENIAL_CODES, f"denial must be explicit (403 or one of {sorted(EXPLICIT_DENIAL_CODES)}), got {payload}" - assert not payload.get("title") and not payload.get("contributors"), f"metadata of a private project leaked: {payload}" + assert _register_pat(mcp, client, signer, subject_b, pat_b).get("registered") is True + writer_b = signer(subject_b, act={"sub": WRITER}, azp=WRITER) + _, code = mcp.call(writer_b, mcp.session(writer_b), "write_record", + {"node_id": project, "base_path": "S3", "files": [{"path": "refused.txt", "content": "must not land"}]}) + record_property("tool_error", code) + assert code == "node_forbidden", f"user_b's PAT on user_a's private project: expected node_forbidden, got {mcp.last_error or 'success'}" + files = api(logged_in_page.context).get(api_url + sel.API["node_files"].format(node_id=project)) + assert files.status == 200 and not files.json().get("data"), "something was written into user_a's project" finally: - api_write(context_b, "DELETE", api_url + sel.API.get("token", "/v2/tokens/{token_id}/").format(token_id=token_id)) + try: + _forget(mcp, signer, subject_b) + finally: + api_write(context_b, "DELETE", api_url + sel.API["token"].format(token_id=pat_b_id)) diff --git a/installer/grdm_dev/compose.py b/installer/grdm_dev/compose.py index c73c5a3a..5889e52c 100644 --- a/installer/grdm_dev/compose.py +++ b/installer/grdm_dev/compose.py @@ -134,6 +134,7 @@ from .secrets import SECRET_KEYS as _SECRET_KEYS # noqa: E402 from .secrets import secret_format_problems as _secret_format_problems # noqa: E402 +from .secrets import derived_values as _derived_values # noqa: E402 REQUIRED_SECRETS: tuple[str, ...] = tuple(_SECRET_KEYS) @@ -1506,8 +1507,12 @@ def run( _ensure_volume_dir(ctx.runtime / VOLUMES_DIR / name) env_updates = {PROFILE_ENV_KEY: profile, EGRESS_ENV_KEY: ",".join(egress), PUBLIC_ACCESS_ENV_KEY: public_access} env_updates.update({port_key(name): str(port) for name, port in published.items()}) + # oidc_mock receives only the public JWK (`${OIDC_MOCK_PUBLIC_JWK}`); a .env written before the key existed, or + # edited by hand, gets it here from the signing key, so the container never needs the private key (PR #86 r1). + env_updates.update({k: v for k, v in _derived_values(env).items() if env.get(k) != v}) if update_dotenv(dotenv, env_updates): - written.append(f".env (GRDM_PROFILE, GRDM_PORT_*, {EGRESS_ENV_KEY}, {PUBLIC_ACCESS_ENV_KEY})") + written.append(f".env (GRDM_PROFILE, GRDM_PORT_*, {EGRESS_ENV_KEY}, {PUBLIC_ACCESS_ENV_KEY}" + + (", OIDC_MOCK_PUBLIC_JWK" if "OIDC_MOCK_PUBLIC_JWK" in env_updates else "") + ")") data = { "profile": profile, diff --git a/installer/grdm_dev/es256.py b/installer/grdm_dev/es256.py new file mode 100644 index 00000000..1ded26ef --- /dev/null +++ b/installer/grdm_dev/es256.py @@ -0,0 +1,187 @@ +"""ES256 (ECDSA on P-256 with SHA-256) keys and JWTs for the mock OIDC issuer — standard library only (ADR-15). + +The full profile runs `oidc_mock`, a stand-in for the AI platform's Keycloak that aipf_mcp_data_grdm trusts +(SPEC-04 §5 S3). `secrets init` writes its signing key to runtime/.env as OIDC_MOCK_SIGNING_KEY: the base64url +(no padding) of a compact private JWK `{"kty":"EC","crv":"P-256","x":..,"y":..,"d":..,"kid":..}`, one line with no +character that Compose or a shell would interpret. `oidc_mock` publishes the key without `d` as its JWKS, and the +acceptance test reads the same value to sign tokens, as the issuer would. + +The installer has no third-party dependency, so the curve arithmetic is written here (affine coordinates, modular +inverses with pow(x, -1, p)). It is for a developer fixture on 127.0.0.1, not for production: there is no +constant-time guarantee. Correctness is checked by the unit tests (curve membership, sign/verify, RFC 7638 kid) +and, on the live stack, by aipf's own verifier (PyJWT with `cryptography`), which accepts the tokens. +""" +from __future__ import annotations + +import base64 +import hashlib +import json +import secrets as _secrets +import time +from typing import Any + +# NIST P-256 (SEC 2 secp256r1). +P = 0xFFFFFFFF00000001000000000000000000000000FFFFFFFFFFFFFFFFFFFFFFFF +A = P - 3 +B = 0x5AC635D8AA3A93E7B3EBBD55769886BC651D06B0CC53B0F63BCE3C3E27D2604B +N = 0xFFFFFFFF00000000FFFFFFFFFFFFFFFFBCE6FAADA7179E84F3B9CAC2FC632551 +G = (0x6B17D1F2E12C4247F8BCE6E563A440F277037D812DEB33A0F4A13945D898C296, + 0x4FE342E2FE1A7F9B8EE7EB4A7C0F9E162BCE33576B315ECECBB6406837BF51F5) + +Point = tuple[int, int] | None # None is the point at infinity + + +def on_curve(pt: Point) -> bool: + if pt is None: + return False + x, y = pt + return 0 <= x < P and 0 <= y < P and (y * y - (x * x * x + A * x + B)) % P == 0 + + +def _add(p1: Point, p2: Point) -> Point: + if p1 is None: + return p2 + if p2 is None: + return p1 + x1, y1 = p1 + x2, y2 = p2 + if x1 == x2: + if (y1 + y2) % P == 0: + return None + lam = (3 * x1 * x1 + A) * pow(2 * y1, -1, P) % P + else: + lam = (y2 - y1) * pow(x2 - x1, -1, P) % P + x3 = (lam * lam - x1 - x2) % P + return x3, (lam * (x1 - x3) - y1) % P + + +def _mul(k: int, pt: Point) -> Point: + out: Point = None + addend = pt + while k: + if k & 1: + out = _add(out, addend) + addend = _add(addend, addend) + k >>= 1 + return out + + +def b64url(data: bytes) -> str: + return base64.urlsafe_b64encode(data).decode("ascii").rstrip("=") + + +def b64url_decode(text: str) -> bytes: + return base64.urlsafe_b64decode(text + "=" * (-len(text) % 4)) + + +def _int_b64(value: int) -> str: + return b64url(value.to_bytes(32, "big")) + + +def _b64_int(text: str) -> int: + raw = b64url_decode(text) + if len(raw) != 32: + raise ValueError("a P-256 coordinate is 32 bytes") + return int.from_bytes(raw, "big") + + +def thumbprint(jwk: dict[str, Any]) -> str: + """RFC 7638 JWK thumbprint (SHA-256) of the public members, used as `kid`.""" + canon = json.dumps({"crv": jwk["crv"], "kty": jwk["kty"], "x": jwk["x"], "y": jwk["y"]}, separators=(",", ":"), sort_keys=True) + return b64url(hashlib.sha256(canon.encode("ascii")).digest()) + + +def generate_private_jwk() -> dict[str, str]: + d = _secrets.randbelow(N - 1) + 1 + x, y = _mul(d, G) # type: ignore[misc] + jwk = {"kty": "EC", "crv": "P-256", "x": _int_b64(x), "y": _int_b64(y), "d": _int_b64(d)} + jwk["kid"] = thumbprint(jwk) + return jwk + + +def encode_key(jwk: dict[str, Any]) -> str: + return b64url(json.dumps(jwk, separators=(",", ":"), sort_keys=True).encode("ascii")) + + +def generate_signing_key() -> str: + """The OIDC_MOCK_SIGNING_KEY value: base64url of a fresh private JWK (about 230 characters, [A-Za-z0-9_-]).""" + return encode_key(generate_private_jwk()) + + +def decode_key(value: str) -> dict[str, str]: + """Parse and check an OIDC_MOCK_SIGNING_KEY value; ValueError names the problem, never the value.""" + try: + jwk = json.loads(b64url_decode(value.strip())) + except (ValueError, TypeError) as exc: + raise ValueError("not base64url of a JSON object") from exc + if not isinstance(jwk, dict) or jwk.get("kty") != "EC" or jwk.get("crv") != "P-256": + raise ValueError("not an EC P-256 JWK") + try: + x, y, d = _b64_int(jwk["x"]), _b64_int(jwk["y"]), _b64_int(jwk["d"]) + except (KeyError, ValueError, TypeError) as exc: + raise ValueError("x, y and d must be 32-byte base64url values") from exc + if not on_curve((x, y)) or not 0 < d < N or _mul(d, G) != (x, y): + raise ValueError("the private key does not match the public point") + if jwk.get("kid") != thumbprint(jwk): + raise ValueError("kid is not the RFC 7638 thumbprint") + return jwk + + +def public_jwk(jwk: dict[str, Any]) -> dict[str, str]: + """The JWKS entry: the public members only, with use and alg.""" + return {"kty": "EC", "crv": "P-256", "x": jwk["x"], "y": jwk["y"], "kid": jwk["kid"], "use": "sig", "alg": "ES256"} + + +def sign(message: bytes, jwk: dict[str, Any]) -> bytes: + """ECDSA signature in the JOSE form r || s (64 bytes).""" + d = _b64_int(jwk["d"]) + z = int.from_bytes(hashlib.sha256(message).digest(), "big") + while True: + k = _secrets.randbelow(N - 1) + 1 + r = _mul(k, G)[0] % N # type: ignore[index] + if not r: + continue + s = pow(k, -1, N) * (z + r * d) % N + if s: + return r.to_bytes(32, "big") + s.to_bytes(32, "big") + + +def verify(message: bytes, signature: bytes, jwk: dict[str, Any]) -> bool: + if len(signature) != 64: + return False + r, s = int.from_bytes(signature[:32], "big"), int.from_bytes(signature[32:], "big") + if not (0 < r < N and 0 < s < N): + return False + q = (_b64_int(jwk["x"]), _b64_int(jwk["y"])) + if not on_curve(q): + return False + z = int.from_bytes(hashlib.sha256(message).digest(), "big") + w = pow(s, -1, N) + pt = _add(_mul(z * w % N, G), _mul(r * w % N, q)) + return pt is not None and pt[0] % N == r + + +def mint_jwt(jwk: dict[str, Any], claims: dict[str, Any]) -> str: + """A compact JWS (ES256) carrying `claims`, with the key's kid in the header.""" + header = {"alg": "ES256", "typ": "JWT", "kid": jwk["kid"]} + signing_input = (b64url(json.dumps(header, separators=(",", ":")).encode()) + "." + + b64url(json.dumps(claims, separators=(",", ":")).encode())) + return signing_input + "." + b64url(sign(signing_input.encode("ascii"), jwk)) + + +def verify_jwt(token: str, jwk: dict[str, Any]) -> dict[str, Any] | None: + """The claims of a token signed by `jwk`, or None (signature only; no claim checks).""" + try: + head, body, sig = token.split(".") + if json.loads(b64url_decode(head)).get("alg") != "ES256": + return None + if not verify(f"{head}.{body}".encode("ascii"), b64url_decode(sig), jwk): + return None + return json.loads(b64url_decode(body)) + except (ValueError, AttributeError): + return None + + +def standard_claims(issuer: str, audience: str, subject: str, lifetime: int = 300, now: int | None = None, **extra: Any) -> dict[str, Any]: + t = int(time.time()) if now is None else now + return {"iss": issuer, "aud": audience, "sub": subject, "iat": t, "nbf": t, "exp": t + lifetime, **extra} diff --git a/installer/grdm_dev/health.py b/installer/grdm_dev/health.py index 3651a5c6..c2644bda 100644 --- a/installer/grdm_dev/health.py +++ b/installer/grdm_dev/health.py @@ -37,20 +37,22 @@ from .context import Context COMPOSE_PROJECT = "grdm-dev" -HTTP_SERVICES: tuple[str, ...] = ("web", "api", "admin", "wb", "mfr", "ember_osf_web", "fakecas", "oasys", "rdmcp") +HTTP_SERVICES: tuple[str, ...] = ("web", "api", "admin", "wb", "mfr", "ember_osf_web", "fakecas", "oasys", "rdmcp", "aipf_mcp_data_grdm") # Probe paths other than "/": osf.io's "/" proxies to ember_osf_web (PROXY_EMBER_APPS) and answered 500 # while the former prebuilt ember was still building its first bundle (real install run 14), so web is # probed at its own login page, which renders without ember; ember_osf_web has its own row. The static # ember image answers GET /healthz with 200 "ok" once its entrypoint has rendered index.html and nginx # serves (the same path as its Compose healthcheck). The OASys image (full profile) does the same after its entrypoint has # written the URLs into the site; rdmcp (full profile) answers it without credentials (RDM-rdmcp-dev2027 PR #1), while -# its MCP endpoint "/" answers 401 to an anonymous call. A service that is not in the Compose file is not probed. +# its MCP endpoint "/" answers 401 to an anonymous call; so does aipf_mcp_data_grdm (full profile), +# whose /mcp needs a bearer token. A service that is not in the Compose file is not probed. # `cas` is not probed: it is published only in the full profile, and its port stays in runtime/.env after a render back # to core, where a probe of that port would fail although nothing is wrong. -HTTP_PATHS: dict[str, str] = {"web": "/login/", "ember_osf_web": "/healthz", "oasys": "/healthz", "rdmcp": "/healthz"} +HTTP_PATHS: dict[str, str] = {"web": "/login/", "ember_osf_web": "/healthz", "oasys": "/healthz", "rdmcp": "/healthz", + "aipf_mcp_data_grdm": "/healthz"} # Services whose probe path is a health endpoint: only HTTP 200 counts, not any status below 500 (a 404 # there means the image is not the one that serves /healthz). -HTTP_EXACT_200: frozenset[str] = frozenset({"ember_osf_web", "oasys", "rdmcp"}) +HTTP_EXACT_200: frozenset[str] = frozenset({"ember_osf_web", "oasys", "rdmcp", "aipf_mcp_data_grdm"}) DOCKER_PROCESS_MARKERS: tuple[str, ...] = ("docker-proxy", "docker-pr", "com.docker", "com.docke", "vpnkit") HTTP_TIMEOUT = 3.0 # A running service without a Compose healthcheck counts only after it has stayed up this long: a diff --git a/installer/grdm_dev/secrets.py b/installer/grdm_dev/secrets.py index 14a09e10..c7f823f3 100644 --- a/installer/grdm_dev/secrets.py +++ b/installer/grdm_dev/secrets.py @@ -30,6 +30,10 @@ RDMCP_RDM_OAUTH_CLIENT_ID the OAuth client rdmcp is at the real CAS; osf_init stores it as osf.io's RDMCP_RDM_OAUTH_CLIENT_SECRET ApiOAuth2Application (full profile, M3 S2); 32 hexadecimal characters and 40 characters, osf.io's own formats (client_secret is a 40-character column) + GRDM_VAULT_KEY aipf_mcp_data_grdm (full profile) key that encrypts the users' stored GRDM tokens: + a Fernet key, base64url of 32 random bytes with padding (44 characters) + OIDC_MOCK_SIGNING_KEY oidc_mock (full profile, ADR-15) signing key: base64url of an EC P-256 private JWK + (es256.py); only the S3 acceptance test reads it, to sign its tokens The CAS names are the environment variables the image reads: Spring maps `tgc.encryption.key` to TGC_ENCRYPTION_KEY, so one name runs from runtime/.env into the container. CAS fails closed on a missing @@ -44,6 +48,11 @@ file rendered before a key was retired still refers to it, so run `grdm-dev compose render` before `secrets rotate` or any recreate. +Derived key (DERIVED_KEYS; rewritten from its secret by `init`, `init --missing`, `rotate` and `compose render`): + + OIDC_MOCK_PUBLIC_JWK base64url of the public JWK of OIDC_MOCK_SIGNING_KEY: the only key material the + oidc_mock container receives (PR #86 review r1), so the private key stays in .env + Non-secret keys (written by `init` only when absent, kept by `rotate`): GRDM_PROFILE minimum | core | full (SPEC-02 §5) @@ -86,7 +95,7 @@ import time from pathlib import Path -from . import exit_codes, report +from . import es256, exit_codes, report from .context import Context SECRET_KEYS: tuple[str, ...] = ( @@ -111,6 +120,8 @@ "RDMCP_CREDENTIAL_ENCRYPTION_KEY", "RDMCP_RDM_OAUTH_CLIENT_ID", "RDMCP_RDM_OAUTH_CLIENT_SECRET", + "GRDM_VAULT_KEY", + "OIDC_MOCK_SIGNING_KEY", ) # Credentials no longer generated or read. Without this list an upgraded runtime/.env would keep them for @@ -161,7 +172,7 @@ }) # Keys that hold a Fernet key (cryptography.fernet.Fernet(key)): 32 bytes, base64url with its padding. -FERNET_KEYS: tuple[str, ...] = ("RDMCP_CREDENTIAL_ENCRYPTION_KEY",) +FERNET_KEYS: tuple[str, ...] = ("RDMCP_CREDENTIAL_ENCRYPTION_KEY", "GRDM_VAULT_KEY") FERNET_KEY_SHAPE = re.compile(r"[A-Za-z0-9_-]{43}=") # Keys whose column in osf.io limits the length (ApiOAuth2Application.client_id 50, client_secret 40 characters). MAX_LENGTHS: dict[str, int] = {"RDMCP_RDM_OAUTH_CLIENT_ID": 50, "RDMCP_RDM_OAUTH_CLIENT_SECRET": 40} @@ -186,6 +197,8 @@ def fernet_key_usable(value: str) -> bool: "RDMCP_CREDENTIAL_ENCRYPTION_KEY": fernet_key, "RDMCP_RDM_OAUTH_CLIENT_ID": lambda: _secrets.token_hex(16), # 32 characters, like osf.io's uuid4().hex "RDMCP_RDM_OAUTH_CLIENT_SECRET": lambda: _secrets.token_urlsafe(30), # exactly 40 characters + "GRDM_VAULT_KEY": fernet_key, # aipf_mcp_data_grdm's PAT vault (M3 S3) + "OIDC_MOCK_SIGNING_KEY": es256.generate_signing_key, # EC P-256 private JWK (ADR-15) } # SPEC-02 §4 ports table: the only place default host ports are defined; copied here as defaults. @@ -240,12 +253,34 @@ def port_key(service: str) -> str: return f"GRDM_PORT_{service.upper()}" +def oidc_signing_key_usable(value: str) -> bool: + try: + es256.decode_key(value) + except ValueError: + return False + return True + + def generate_secret(key: str) -> str: if key in _GENERATORS: return _GENERATORS[key]() return _secrets.token_urlsafe(_SECRET_BYTES.get(key, _DEFAULT_BYTES)) +PUBLIC_JWK_KEY = "OIDC_MOCK_PUBLIC_JWK" +DERIVED_KEYS: tuple[str, ...] = (PUBLIC_JWK_KEY,) + + +def derived_values(values: dict[str, str]) -> dict[str, str]: + """Values that follow from secrets: the public JWK of a usable OIDC_MOCK_SIGNING_KEY (nothing otherwise, so a + malformed key is reported by secret_format_problems rather than half-derived).""" + try: + jwk = es256.decode_key(values.get("OIDC_MOCK_SIGNING_KEY", "")) + except ValueError: + return {} + return {PUBLIC_JWK_KEY: es256.encode_key(es256.public_jwk(jwk))} + + def read_env(path: Path) -> dict[str, str]: """Parse KEY=value lines (comments and blanks ignored; surrounding quotes stripped).""" values: dict[str, str] = {} @@ -321,6 +356,7 @@ def secret_format_problems(values: dict[str, str]) -> list[str]: bad = cas_key_problems(values) bad += [key for key in FERNET_KEYS if key in values and not fernet_key_usable(values[key])] bad += [key for key, limit in MAX_LENGTHS.items() if key in values and len(values[key]) > limit] + bad += [key for key in ("OIDC_MOCK_SIGNING_KEY",) if key in values and not oidc_signing_key_usable(values[key])] return bad @@ -400,6 +436,7 @@ def run_init(ctx: Context, force: bool = False, missing: bool = False) -> int: if ctx.profile: values["GRDM_PROFILE"] = ctx.profile values.update(fresh_secrets()) + values.update(derived_values(values)) bad = check_no_defaults(values) if bad: # cannot happen with token_urlsafe, but SPEC-03 §3.1 says fail rather than write a default return report.error(ctx, exit_codes.USAGE, "refusing to write default-looking secrets: " + ", ".join(bad)) @@ -426,6 +463,10 @@ def _add_missing(ctx: Context, path: Path, existing: dict[str, str]) -> int: added.append(key) if ctx.profile: values["GRDM_PROFILE"] = ctx.profile + for key, value in derived_values(values).items(): + if values.get(key) != value: + values[key] = value + added.append(key) bad = check_no_defaults(values) if bad: return report.error(ctx, exit_codes.USAGE, "refusing to keep default-looking secrets: " + ", ".join(bad) + "; run `secrets rotate`") @@ -466,6 +507,7 @@ def run_rotate(ctx: Context, runner=None, running_services=None, prober=None, sl old_values = read_env(path) new_values = {k: v for k, v in old_values.items() if k not in SECRET_KEYS and k not in OBSOLETE_SECRET_KEYS} new_values.update(fresh_secrets()) + new_values.update(derived_values(new_values)) # the issuer's public key follows the new signing key at the recreate # 0. what is live? Fail closed before touching anything. running: list[str] = [] diff --git a/installer/grdm_dev/templates/aipf_mcp_data_grdm.json b/installer/grdm_dev/templates/aipf_mcp_data_grdm.json index 8b63ab49..ba37fd3a 100644 --- a/installer/grdm_dev/templates/aipf_mcp_data_grdm.json +++ b/installer/grdm_dev/templates/aipf_mcp_data_grdm.json @@ -1,12 +1,21 @@ { "service": "aipf_mcp_data_grdm", "component": "aipf_mcp_data_grdm", - "container_port": 8101, - "expose": [ - 8101 - ], + "container_port": 8768, + "host_port": 8768, + "expose": [], "environment": { - "OSF_API_URL": "http://api:8000/v2/", + "MCP_HOST": "0.0.0.0", + "MCP_PORT": "8768", + "GRDM_API_BASE": "http://api:8000/v2/", + "GRDM_VAULT_KEY": "${GRDM_VAULT_KEY}", + "GRDM_VAULT_DB": "/tmp/grdm_vault.sqlite3", + "OIDC_ISSUER": "http://oidc_mock:8088/realms/grdm-dev", + "OIDC_AUDIENCE": "aipf-mcp-grdm", + "OIDC_JWKS_URL": "http://oidc_mock:8088/realms/grdm-dev/protocol/openid-connect/certs", + "ALLOWED_ACTORS": "grdm-dev-agent,grdm-dev-research-flow", + "ALLOWED_DIRECT_CLIENTS": "grdm-dev-webui", + "WRITE_ACTORS": "grdm-dev-research-flow", "HOME": "/tmp" }, "tmpfs": [ @@ -14,10 +23,23 @@ ], "user": "1000:1000", "depends_on": [ - "api" + "api", + "oidc_mock" ], + "healthcheck": { + "test": [ + "CMD", + "python3", + "-c", + "import sys,urllib.request; sys.exit(0 if urllib.request.urlopen('http://127.0.0.1:8768/healthz', timeout=2).status == 200 else 1)" + ], + "interval": "10s", + "timeout": "3s", + "start_period": "10s", + "retries": 3 + }, "weight": 1, "mem_min_mb": 256, "pids_limit": 512, - "note": "Port provisional." + "note": "Variable names are those of RDM-aipf_mcp_data_grdm-dev2027 src/aipf_mcp_data_grdm/server.py (main): the image runs `aipf-mcp-data-grdm`, MCP at /mcp, /grdm/token and /healthz on MCP_PORT, and exits at start without OIDC_ISSUER, OIDC_AUDIENCE and GRDM_VAULT_KEY (the restart loop of the M3 survey). Every /mcp request needs a bearer JWT that verifies against the JWKS of oidc_mock (ADR-15) with this iss and aud, whose act.sub is in ALLOWED_ACTORS or, without act, whose azp is in ALLOWED_DIRECT_CLIENTS (403 actor_missing / actor_not_allowed otherwise); the write tools (ensure_project, write_record, set_project_description, add_project_members) also need act.sub in WRITE_ACTORS. The three names are fixtures of this environment, not the AI platform's. GRDM is reached with the user's own personal access token, submitted through POST /grdm/token?t= and kept Fernet-encrypted with GRDM_VAULT_KEY (Fernet format, `secrets init`) in GRDM_VAULT_DB on the tmpfs /tmp: the image's /data belongs to nobody and the root is read-only, and no token survives a restart, so a restart asks each user to register again (the form token's HMAC key is per process anyway). GRDM_API_BASE is the api service inside the Compose network. write_record uploads through the WaterButler links the API returns, which are browser-facing (127.0.0.1:) and unreachable from this container, so S3 writes with ensure_project. Published on 127.0.0.1:8768 (the image's own default port) for the acceptance test; the healthcheck uses the image's python (no curl or wget)." } diff --git a/installer/grdm_dev/templates/oidc_mock.json b/installer/grdm_dev/templates/oidc_mock.json new file mode 100644 index 00000000..8ed7f15f --- /dev/null +++ b/installer/grdm_dev/templates/oidc_mock.json @@ -0,0 +1,30 @@ +{ + "service": "oidc_mock", + "component": "oidc-mock", + "container_port": 8088, + "expose": [ + 8088 + ], + "environment": { + "OIDC_MOCK_ISSUER": "http://oidc_mock:8088/realms/grdm-dev", + "OIDC_MOCK_PUBLIC_JWK": "${OIDC_MOCK_PUBLIC_JWK}", + "OIDC_MOCK_PORT": "8088" + }, + "user": "65534:65534", + "healthcheck": { + "test": [ + "CMD", + "python3", + "-c", + "import sys,urllib.request; sys.exit(0 if urllib.request.urlopen('http://127.0.0.1:8088/healthz', timeout=2).status == 200 else 1)" + ], + "interval": "10s", + "timeout": "3s", + "start_period": "5s", + "retries": 3 + }, + "weight": 0.25, + "mem_min_mb": 64, + "pids_limit": 64, + "note": "Mock OIDC issuer (ADR-15), installer/images/oidc-mock: the stand-in for the AI platform's Keycloak realm that aipf_mcp_data_grdm verifies its bearer tokens against. It serves /.well-known/openid-configuration and /protocol/openid-connect/certs (Keycloak's layout) and /healthz. It receives only OIDC_MOCK_PUBLIC_JWK, the public JWK that `secrets init`, `rotate` and `compose render` derive from OIDC_MOCK_SIGNING_KEY in runtime/.env, and refuses a value that carries a private member, so the private key never enters this container (PR #86 review r1); acceptance scenario S3 reads the signing key from runtime/.env and signs its tokens itself. It signs nothing and has no token or token-exchange endpoint: aipf_mcp_data_grdm does not call OIDC_TOKEN_URL on the paths S3 exercises. Internal only (no host port, network grdm); the issuer string uses the Compose service name, so it is the same inside every container. A missing or malformed key stops it with exit 1." +} diff --git a/installer/images/oidc-mock/Dockerfile b/installer/images/oidc-mock/Dockerfile new file mode 100644 index 00000000..b8c0f181 --- /dev/null +++ b/installer/images/oidc-mock/Dockerfile @@ -0,0 +1,20 @@ +# syntax=docker/dockerfile:1 +# Mock OIDC issuer for the full profile (ADR-15): a stand-in for the AI platform's Keycloak realm that +# aipf_mcp_data_grdm trusts. It serves the discovery document and the JWKS of OIDC_MOCK_PUBLIC_JWK, the public +# part of the signing key that `grdm-dev secrets init` writes to runtime/.env, and nothing else; the private key +# never enters this container. +# +# Why our own image: Keycloak would be the heaviest service of the stack for two JSON documents (ADR-05 +# option B was rejected for the same reason). The server is one standard-library Python file, so the image +# is the official python slim base pinned by digest plus that file; no package is installed. It runs as +# uid 65534 on a read-only root and needs no writable path. +FROM python:3.13-slim@sha256:e48f59cdba69a3cbd9ed2cf0f348d90d45a8d350d98134cf1070bc106ee9ed90 +LABEL org.opencontainers.image.base.name="python:3.13-slim" \ + org.opencontainers.image.base.digest="sha256:e48f59cdba69a3cbd9ed2cf0f348d90d45a8d350d98134cf1070bc106ee9ed90" \ + org.opencontainers.image.title="grdm-dev/oidc-mock" \ + org.opencontainers.image.description="Mock OIDC issuer (discovery + JWKS) for the GakuNin RDM developer environment (ADR-15)" +COPY oidc_mock.py /opt/oidc-mock/oidc_mock.py +ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1 +USER 65534:65534 +EXPOSE 8088 +CMD ["python3", "-I", "/opt/oidc-mock/oidc_mock.py"] diff --git a/installer/images/oidc-mock/oidc_mock.py b/installer/images/oidc-mock/oidc_mock.py new file mode 100644 index 00000000..f730b46c --- /dev/null +++ b/installer/images/oidc-mock/oidc_mock.py @@ -0,0 +1,144 @@ +"""Mock OIDC issuer of the GakuNin RDM developer environment (full profile, ADR-15). + +A stand-in for the AI platform's Keycloak realm, which aipf_mcp_data_grdm trusts. It serves only what a relying +party reads to verify a token, in Keycloak's layout under the issuer path: + + GET /.well-known/openid-configuration discovery document + GET /protocol/openid-connect/certs JWKS: the signing key without its private part + GET /healthz 200 {"status": "ok"} + +It signs nothing and holds no private key: the tokens of acceptance scenario S3 are signed by the test with the +private key in runtime/.env (OIDC_MOCK_SIGNING_KEY), and this container gets only its public JWK. It has no token, +authorize or token-exchange endpoint, because aipf_mcp_data_grdm never calls OIDC_TOKEN_URL on the paths S3 +exercises. Anything else is 404, any other method 405. + +Environment: OIDC_MOCK_ISSUER (the `iss` value, http://:/), OIDC_MOCK_PUBLIC_JWK (base64url of the +public EC P-256 JWK that `grdm-dev secrets init` and `compose render` derive from the signing key; a value with a +private member `d` is refused), OIDC_MOCK_PORT (default 8088). A missing or malformed value stops the process with +exit 1 and a message that names the variable, never its value. +Standard library only; runs as a non-root user on a read-only root. +""" +from __future__ import annotations + +import base64 +import json +import os +import signal +import sys +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from urllib.parse import urlsplit + +PUBLIC_MEMBERS = ("kty", "crv", "x", "y", "kid") + + +def _b64url_decode(text: str) -> bytes: + return base64.urlsafe_b64decode(text + "=" * (-len(text) % 4)) + + +def public_jwk(value: str) -> dict: + """The JWKS entry for an OIDC_MOCK_PUBLIC_JWK value; ValueError when it is not a public EC P-256 JWK.""" + try: + jwk = json.loads(_b64url_decode(value.strip())) + except (ValueError, TypeError) as exc: + raise ValueError("OIDC_MOCK_PUBLIC_JWK is not base64url of a JSON object") from exc + if not isinstance(jwk, dict) or jwk.get("kty") != "EC" or jwk.get("crv") != "P-256": + raise ValueError("OIDC_MOCK_PUBLIC_JWK is not an EC P-256 JWK") + if "d" in jwk: + raise ValueError("OIDC_MOCK_PUBLIC_JWK carries a private key (d): this container must get the public JWK only") + for member in ("x", "y", "kid"): + if not isinstance(jwk.get(member), str) or not jwk[member]: + raise ValueError(f"OIDC_MOCK_PUBLIC_JWK has no {member}") + for member in ("x", "y"): + if len(_b64url_decode(jwk[member])) != 32: + raise ValueError(f"OIDC_MOCK_PUBLIC_JWK: {member} is not 32 bytes") + return {**{m: jwk[m] for m in PUBLIC_MEMBERS}, "use": "sig", "alg": "ES256"} + + +def documents(issuer: str, key_value: str) -> dict[str, bytes]: + """Path -> JSON body for every route the issuer answers.""" + parts = urlsplit(issuer) + if parts.scheme not in ("http", "https") or not parts.netloc or parts.query or parts.fragment or issuer.endswith("/"): + raise ValueError("OIDC_MOCK_ISSUER must be an http(s) URL without a trailing slash, query or fragment") + base = parts.path + jwks_uri = f"{issuer}/protocol/openid-connect/certs" + discovery = { + "issuer": issuer, + "jwks_uri": jwks_uri, + "id_token_signing_alg_values_supported": ["ES256"], + "subject_types_supported": ["public"], + "response_types_supported": [], + "grant_types_supported": [], + "claims_supported": ["iss", "aud", "sub", "exp", "iat", "nbf", "azp", "act"], + } + + def body(obj: dict) -> bytes: + return json.dumps(obj, separators=(",", ":"), sort_keys=True).encode("utf-8") + + return { + f"{base}/.well-known/openid-configuration": body(discovery), + f"{base}/protocol/openid-connect/certs": body({"keys": [public_jwk(key_value)]}), + "/healthz": body({"status": "ok"}), + } + + +def make_handler(routes: dict[str, bytes]): + class Handler(BaseHTTPRequestHandler): + server_version = "grdm-dev-oidc-mock" + sys_version = "" + + def _send(self, status: int, payload: bytes, head_only: bool = False) -> None: + self.send_response(status) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(payload))) + self.send_header("Cache-Control", "no-store") + self.end_headers() + if not head_only: + self.wfile.write(payload) + + def do_GET(self) -> None: # noqa: N802 - http.server API + path = urlsplit(self.path).path + if path in routes: + self._send(200, routes[path]) + else: + self._send(404, b'{"error":"not_found"}') + + def do_HEAD(self) -> None: # noqa: N802 + path = urlsplit(self.path).path + self._send(200 if path in routes else 404, routes.get(path, b""), head_only=True) + + def _refuse(self) -> None: + self._send(405, b'{"error":"method_not_allowed"}') + + do_POST = do_PUT = do_PATCH = do_DELETE = do_OPTIONS = _refuse # noqa: N815 + + def log_message(self, fmt: str, *args) -> None: # one line per request, no query string, no headers + sys.stderr.write("oidc_mock: %s %s %s\n" % (self.command, urlsplit(self.path).path, args[1] if len(args) > 1 else "-")) + + return Handler + + +def main() -> int: + issuer = os.environ.get("OIDC_MOCK_ISSUER", "") + key_value = os.environ.get("OIDC_MOCK_PUBLIC_JWK", "") + if not issuer or not key_value: + sys.stderr.write("oidc_mock: set OIDC_MOCK_ISSUER and OIDC_MOCK_PUBLIC_JWK (grdm-dev secrets init, compose render)\n") + return 1 + try: + routes = documents(issuer, key_value) + port = int(os.environ.get("OIDC_MOCK_PORT", "8088")) + except ValueError as exc: + sys.stderr.write(f"oidc_mock: {exc}\n") + return 1 + server = ThreadingHTTPServer(("0.0.0.0", port), make_handler(routes)) # nosec B104 - inside the Compose network only, never published + # PID 1 ignores SIGTERM unless it installs a handler: without this `docker stop` waits for its SIGKILL. + signal.signal(signal.SIGTERM, lambda *_: sys.exit(0)) + sys.stderr.write(f"oidc_mock: serving {issuer} on port {port}\n") + try: + server.serve_forever() + except KeyboardInterrupt: + pass + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/installer/tests/test_accept.py b/installer/tests/test_accept.py index f69ee7fb..c435b71f 100644 --- a/installer/tests/test_accept.py +++ b/installer/tests/test_accept.py @@ -866,7 +866,7 @@ def test_only_selection(self): self.assertEqual(rc, exit_codes.USAGE) def test_scenario_files_cover_c1_to_c6_and_s1_to_s5_with_both_cases(self): - enabled_subsystems = {"S1", "S2", "S4"} # M3: made real on the live stack; the others still wait for their subsystem + enabled_subsystems = {"S1", "S2", "S3", "S4"} # M3: made real on the live stack; the others still wait for their subsystem for sid in accept.SCENARIOS: files = list(accept.ACCEPTANCE_DIR.glob(f"test_{sid.lower()}_*.py")) self.assertEqual(len(files), 1, sid) @@ -885,7 +885,8 @@ def test_scenario_files_cover_c1_to_c6_and_s1_to_s5_with_both_cases(self): def test_core_tests_never_create_unguarded_contexts(self): """SPEC-04 §3.4: every API/browser context goes through the guarded fixtures.""" for f in [*accept.ACCEPTANCE_DIR.glob("test_c*.py"), accept.ACCEPTANCE_DIR / "test_s1_y_websocket.py", - accept.ACCEPTANCE_DIR / "test_s2_rdmcp.py", accept.ACCEPTANCE_DIR / "test_s4_oasys.py"]: + accept.ACCEPTANCE_DIR / "test_s2_rdmcp.py", accept.ACCEPTANCE_DIR / "test_s3_aipf.py", + accept.ACCEPTANCE_DIR / "test_s4_oasys.py"]: src = f.read_text() self.assertNotIn("playwright.request.new_context", src, f.name) self.assertNotIn("browser.new_context", src, f.name) diff --git a/installer/tests/test_compose.py b/installer/tests/test_compose.py index dee2124e..34043fd7 100644 --- a/installer/tests/test_compose.py +++ b/installer/tests/test_compose.py @@ -23,10 +23,16 @@ FORBIDDEN = "192.168.168.167" +# The EC signing key of oidc_mock has a format of its own (ADR-15); the Fernet keys are handled below. +_FORMATTED_FAKES = {"OIDC_MOCK_SIGNING_KEY": __import__("grdm_dev.es256").es256.generate_signing_key()} + + def fake_value(key: str) -> str: """A placeholder that `compose render` accepts for this key ("x" * 43 is not canonical base64url of 32 bytes).""" from grdm_dev import secrets as _secrets + if key in _FORMATTED_FAKES: + return _FORMATTED_FAKES[key] if key in _secrets.FERNET_KEYS: return "A" * 43 + "=" # canonical padded base64url of 32 zero bytes size = _secrets.CAS_KEY_RULES.get(key, (48, None))[0] @@ -801,7 +807,7 @@ def test_profiles_minimum_and_full_shibboleth(self): doc = json.loads((root / "runtime" / "docker-compose.yml").read_text(encoding="utf-8")) self.assertIn("shibboleth_sp", doc["services"]) self.assertNotIn("fakecas", doc["services"]) - self.assertEqual(len(doc["services"]), 27) # full profile with shibboleth: core 23 - fakecas + rdmcp aipf_mcp_data_grdm oasys shibboleth_sp test_idp + self.assertEqual(len(doc["services"]), 28) # full profile with shibboleth: core 23 - fakecas + rdmcp aipf_mcp_data_grdm oidc_mock oasys shibboleth_sp test_idp # ---- OASys (full profile; SPEC-02 §4, SPEC-04 §5 S4) @@ -855,6 +861,79 @@ def test_core_profile_has_no_oasys_and_no_oasys_url(self): self.assertNotIn("GRDM_OASYS_URL", compose.render_osf_env(ports, "false", {"web": 5000})) self.assertIn("GRDM_OASYS_URL=http://127.0.0.1:3105/\n", compose.render_osf_env(ports, "false", {"oasys": 3105})) + # ---- aipf_mcp_data_grdm and the mock OIDC issuer (full profile; SPEC-04 §5 S3, ADR-15) + + def test_full_profile_runs_aipf_against_the_mock_issuer_and_publishes_it_on_loopback(self): + root = make_root() + rc, out, err, _ = render(root, profile="full") + self.assertEqual(rc, exit_codes.OK, err) + doc = json.loads((root / "runtime" / "docker-compose.yml").read_text(encoding="utf-8")) + svc = doc["services"]["aipf_mcp_data_grdm"] + self.assertEqual(svc["ports"], ["127.0.0.1:8768:8768"]) # SPEC-02 §4 row; the image's own MCP_PORT default + self.assertEqual(svc["networks"], ["grdm", "grdm-edge"]) + self.assertEqual((svc["user"], svc["read_only"], svc["tmpfs"]), ("1000:1000", True, ["/tmp:mode=1777"])) + env = svc["environment"] + issuer = "http://oidc_mock:8088/realms/grdm-dev" + self.assertEqual((env["MCP_HOST"], env["MCP_PORT"]), ("0.0.0.0", "8768")) # the image's default host is 127.0.0.1 + self.assertEqual(env["GRDM_API_BASE"], "http://api:8000/v2/") + self.assertEqual(env["GRDM_VAULT_KEY"], "${GRDM_VAULT_KEY}") # a reference, never the value + self.assertTrue(env["GRDM_VAULT_DB"].startswith("/tmp/")) # /data belongs to nobody; the root is read-only + self.assertEqual((env["OIDC_ISSUER"], env["OIDC_AUDIENCE"]), (issuer, "aipf-mcp-grdm")) + self.assertEqual(env["OIDC_JWKS_URL"], issuer + "/protocol/openid-connect/certs") + self.assertNotIn("OIDC_TOKEN_URL", env) # no token exchange on the tested paths + self.assertIn(env["WRITE_ACTORS"], env["ALLOWED_ACTORS"].split(",")) + self.assertNotIn(env["ALLOWED_DIRECT_CLIENTS"], env["WRITE_ACTORS"].split(",")) + self.assertIn("oidc_mock", svc["depends_on"]) # JWKS is fetched lazily, at the first token + self.assertIn("http://127.0.0.1:8768/healthz", svc["healthcheck"]["test"][-1]) + self.assertEqual(plan.read_dotenv(root / "runtime" / ".env")["GRDM_PORT_AIPF_MCP_DATA_GRDM"], "8768") + self.assertIn(("aipf_mcp_data_grdm", 8768), [(r["service"], r["host_port"]) for r in json.loads(out)["ports"]]) + text = (root / "runtime" / "docker-compose.yml").read_text(encoding="utf-8") + self.assertNotIn(fake_value("OIDC_MOCK_SIGNING_KEY"), text) + self.assertNotIn(fake_value("GRDM_VAULT_KEY"), text) + + def test_the_mock_issuer_is_internal_non_root_and_gets_its_key_by_reference(self): + root = make_root() + rc, _, err, _ = render(root, profile="full") + self.assertEqual(rc, exit_codes.OK, err) + svc = json.loads((root / "runtime" / "docker-compose.yml").read_text(encoding="utf-8"))["services"]["oidc_mock"] + self.assertEqual(svc["image"], "grdm-dev/oidc-mock:2") # manifest image_build tag + self.assertNotIn("ports", svc) # never published + self.assertEqual(svc["networks"], ["grdm"]) # internal network only, no egress + self.assertEqual((svc["user"], svc["read_only"], svc["cap_drop"]), ("65534:65534", True, ["ALL"])) + self.assertEqual(svc["environment"]["OIDC_MOCK_PUBLIC_JWK"], "${OIDC_MOCK_PUBLIC_JWK}") # public part only (PR #86 r1) + self.assertNotIn("OIDC_MOCK_SIGNING_KEY", json.dumps(svc)) + self.assertEqual(svc["environment"]["OIDC_MOCK_ISSUER"], "http://oidc_mock:8088/realms/grdm-dev") + self.assertIn("http://127.0.0.1:8088/healthz", svc["healthcheck"]["test"][-1]) + self.assertNotIn("GRDM_PORT_OIDC_MOCK", plan.read_dotenv(root / "runtime" / ".env")) + + def test_render_derives_the_public_jwk_and_no_container_gets_the_private_key(self): + from grdm_dev import es256, secrets as _secrets + private = es256.decode_key(fake_value("OIDC_MOCK_SIGNING_KEY")) + stale = es256.encode_key(es256.public_jwk(es256.decode_key(es256.generate_signing_key()))) + for dotenv in (fake_secrets(), fake_secrets(OIDC_MOCK_PUBLIC_JWK=stale)): # absent (older .env), or from another key + root = make_root(dotenv) + rc, _, err, _ = render(root, profile="full") + self.assertEqual(rc, exit_codes.OK, err) + env = plan.read_dotenv(root / "runtime" / ".env") + self.assertEqual(es256.public_jwk(private), json.loads(es256.b64url_decode(env[_secrets.PUBLIC_JWK_KEY]))) + text = (root / "runtime" / "docker-compose.yml").read_text(encoding="utf-8") + self.assertNotIn("${OIDC_MOCK_SIGNING_KEY}", text) # no service is given the private key + self.assertNotIn(private["d"], text) + + def test_core_profile_has_neither_aipf_nor_the_mock_issuer(self): + doc = self.render_core_doc() + self.assertNotIn("aipf_mcp_data_grdm", doc["services"]) + self.assertNotIn("oidc_mock", doc["services"]) + + def test_render_refuses_a_vault_or_signing_key_in_the_wrong_format_by_name_only(self): + for key, bad in (("GRDM_VAULT_KEY", "x" * 48), ("OIDC_MOCK_SIGNING_KEY", "y" * 300)): + root = make_root(fake_secrets(**{key: bad})) + rc, _, err, _ = render(root, profile="full") + self.assertEqual(rc, exit_codes.USAGE, key) + self.assertIn(key, err) + self.assertNotIn(bad, err) + self.assertFalse((root / "runtime" / "docker-compose.yml").exists()) + def test_a_moved_oasys_port_moves_every_url_that_names_it(self): root = make_root() rc, _, err, _ = render(root, profile="full", checker=lambda port: port != 3100) # another process holds 3100 diff --git a/installer/tests/test_es256.py b/installer/tests/test_es256.py new file mode 100644 index 00000000..dd7173ae --- /dev/null +++ b/installer/tests/test_es256.py @@ -0,0 +1,83 @@ +"""grdm_dev.es256: the mock issuer's signing key and the tokens of acceptance scenario S3 (ADR-15).""" +from __future__ import annotations + +import json +import unittest + +from grdm_dev import es256 + +# RFC 7515 Appendix A.3: the public part of the ES256 example key and its JWS (a known answer from outside this code). +RFC7515_A3_JWK = {"kty": "EC", "crv": "P-256", + "x": "f83OJ3D2xF1Bg8vub9tLe1gHMzV76e8Tus9uPHvRVEU", + "y": "x_FEzRu9m36HLN_tue659LNpXW6pCyStikYjKIWI5a0"} +RFC7515_A3_JWS = ("eyJhbGciOiJFUzI1NiJ9" + ".eyJpc3MiOiJqb2UiLA0KICJleHAiOjEzMDA4MTkzODAsDQogImh0dHA6Ly9leGFtcGxlLmNvbS9pc19yb290Ijp0cnVlfQ" + ".DtEhU3ljbEg8L38VWAfUAqOyKAM6-Xx-F4GawxaepmXFCgfTjDxw5djxLa8ISlSApmWQxfKTUJqPP3-Kg6NU1Q") + + +class Es256Tests(unittest.TestCase): + def test_known_answer_of_rfc_7515_a3(self): + head, body, sig = RFC7515_A3_JWS.split(".") + self.assertTrue(es256.verify(f"{head}.{body}".encode(), es256.b64url_decode(sig), RFC7515_A3_JWK)) + self.assertFalse(es256.verify(f"{head}.{body}x".encode(), es256.b64url_decode(sig), RFC7515_A3_JWK)) + + def test_the_public_point_of_a_generated_key_follows_from_d(self): + jwk = es256.decode_key(es256.generate_signing_key()) + d = int.from_bytes(es256.b64url_decode(jwk["d"]), "big") + x, y = (int.from_bytes(es256.b64url_decode(jwk[k]), "big") for k in ("x", "y")) + self.assertEqual(es256._mul(d, es256.G), (x, y)) + self.assertTrue(es256.on_curve((x, y))) + self.assertIsNone(es256._mul(es256.N, es256.G)) # n * G is the point at infinity: G has order n + + def test_generated_key_is_a_valid_p256_jwk_on_one_line_without_shell_characters(self): + value = es256.generate_signing_key() + self.assertRegex(value, r"^[A-Za-z0-9_-]+$") + jwk = es256.decode_key(value) + self.assertEqual((jwk["kty"], jwk["crv"]), ("EC", "P-256")) + self.assertEqual(jwk["kid"], es256.thumbprint(jwk)) + self.assertNotEqual(es256.decode_key(es256.generate_signing_key())["d"], jwk["d"]) + + def test_public_jwk_has_no_private_member(self): + pub = es256.public_jwk(es256.decode_key(es256.generate_signing_key())) + self.assertEqual(set(pub), {"kty", "crv", "x", "y", "kid", "use", "alg"}) + self.assertEqual((pub["alg"], pub["use"]), ("ES256", "sig")) + + def test_minted_token_verifies_with_the_public_key_only_and_carries_the_claims(self): + jwk = es256.decode_key(es256.generate_signing_key()) + claims = es256.standard_claims("http://oidc_mock:8088/realms/grdm-dev", "aipf-mcp-grdm", "user-a", + now=1_000, act={"sub": "grdm-dev-research-flow"}) + token = es256.mint_jwt(jwk, claims) + header = json.loads(es256.b64url_decode(token.split(".")[0])) + self.assertEqual(header, {"alg": "ES256", "typ": "JWT", "kid": jwk["kid"]}) + self.assertEqual(es256.verify_jwt(token, es256.public_jwk(jwk)), claims) + self.assertEqual((claims["exp"], claims["iat"]), (1_300, 1_000)) + + def test_a_token_of_another_key_or_with_a_changed_body_does_not_verify(self): + a, b = (es256.decode_key(es256.generate_signing_key()) for _ in range(2)) + token = es256.mint_jwt(a, es256.standard_claims("i", "aud", "s")) + self.assertIsNone(es256.verify_jwt(token, es256.public_jwk(b))) + head, _, sig = token.split(".") + forged = es256.b64url(json.dumps(es256.standard_claims("i", "aud", "admin")).encode()) + self.assertIsNone(es256.verify_jwt(f"{head}.{forged}.{sig}", es256.public_jwk(a))) + self.assertIsNone(es256.verify_jwt("not-a-jwt", es256.public_jwk(a))) + + def test_decode_key_rejects_malformed_values_without_echoing_them(self): + good = es256.decode_key(es256.generate_signing_key()) + other = es256.decode_key(es256.generate_signing_key()) + cases = { + "garbage": "%%%", + "json list": es256.b64url(b"[1]"), + "rsa": es256.encode_key({**good, "kty": "RSA"}), + "short x": es256.encode_key({**good, "x": es256.b64url(b"\x01" * 31)}), + "foreign d": es256.encode_key({**good, "d": other["d"]}), + "wrong kid": es256.encode_key({**good, "kid": "k"}), + "no d": es256.encode_key({k: v for k, v in good.items() if k != "d"}), + } + for name, value in cases.items(): + with self.subTest(name), self.assertRaises(ValueError) as caught: + es256.decode_key(value) + self.assertNotIn(good["d"], str(caught.exception)) + + +if __name__ == "__main__": + unittest.main() diff --git a/installer/tests/test_health.py b/installer/tests/test_health.py index 30bf6fcd..97731953 100644 --- a/installer/tests/test_health.py +++ b/installer/tests/test_health.py @@ -172,6 +172,24 @@ def prober(url): exit_codes.HEALTH_FAILED) self.assertIn("oasys", self.err.getvalue()) + def test_aipf_is_probed_at_healthz_and_must_answer_200(self): + """aipf_mcp_data_grdm (full profile) answers /healthz without a token; a crash-looping container answers nothing.""" + compose = json.loads(json.dumps(COMPOSE)) + compose["services"]["aipf_mcp_data_grdm"] = {"image": "grdm-dev/aipf_mcp_data_grdm:94e277da6944", "ports": ["127.0.0.1:8768:8768"]} + (self.root / "runtime" / "docker-compose.yml").write_text(json.dumps(compose)) + ps = "\n".join(json.dumps(ps_row(s)) for s in ("web", "api", "postgres", "aipf_mcp_data_grdm")) + probed = [] + + def prober(url): + probed.append(url) + return True, "HTTP 200" + + self.assertEqual(self.run_cmd(False, make_runner(ps), prober), exit_codes.OK) + self.assertIn("http://127.0.0.1:8768/healthz", probed) + self.assertEqual(self.run_cmd(False, make_runner(ps), lambda url: (True, "HTTP 401" if url.endswith("/healthz") else "HTTP 200")), + exit_codes.HEALTH_FAILED) + self.assertIn("aipf_mcp_data_grdm", self.err.getvalue()) + def test_oasys_is_not_probed_when_not_in_the_compose_file(self): probed = [] diff --git a/installer/tests/test_manifest.py b/installer/tests/test_manifest.py index f00c32e8..d3e3d66b 100644 --- a/installer/tests/test_manifest.py +++ b/installer/tests/test_manifest.py @@ -14,7 +14,7 @@ # `wiki_assets` is the one-shot service of the wiki-frontend overlay (SPEC-01 §3 overlay_of, ADR-06); there is no `wiki` server. CORE = MINIMUM | {"admin", "worker", "wb", "wb_worker", "mfr", "unoconv", "ember_osf_web", "cas", "wiki_assets", "volume_access_check_999", "elasticsearch6", "redis", "y_websocket"} -FULL_FAKECAS = CORE | {"rdmcp", "aipf_mcp_data_grdm", "oasys"} +FULL_FAKECAS = CORE | {"rdmcp", "aipf_mcp_data_grdm", "oidc_mock", "oasys"} FULL_SHIBBOLETH = (FULL_FAKECAS - {"fakecas"}) | {"shibboleth_sp", "test_idp"} diff --git a/installer/tests/test_oidc_mock_image.py b/installer/tests/test_oidc_mock_image.py new file mode 100644 index 00000000..b9839e12 --- /dev/null +++ b/installer/tests/test_oidc_mock_image.py @@ -0,0 +1,120 @@ +"""installer/images/oidc-mock/oidc_mock.py: the mock OIDC issuer of the full profile (ADR-15), served on a free port.""" +from __future__ import annotations + +import http.client +import importlib.util +import io +import json +import threading +import unittest +from contextlib import redirect_stderr +from http.server import ThreadingHTTPServer +from pathlib import Path +from unittest import mock + +from grdm_dev import es256 + +IMAGE = Path(__file__).resolve().parents[1] / "images" / "oidc-mock" +ISSUER = "http://oidc_mock:8088/realms/grdm-dev" + + +def load_module(): + spec = importlib.util.spec_from_file_location("oidc_mock_under_test", IMAGE / "oidc_mock.py") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +class OidcMockDocumentTests(unittest.TestCase): + def setUp(self): + self.mod = load_module() + self.jwk = es256.decode_key(es256.generate_signing_key()) + self.key = es256.encode_key(es256.public_jwk(self.jwk)) # OIDC_MOCK_PUBLIC_JWK, as secrets/compose derive it + + def test_jwks_is_the_public_part_of_the_installer_key(self): + docs = self.mod.documents(ISSUER, self.key) + jwks = json.loads(docs["/realms/grdm-dev/protocol/openid-connect/certs"]) + self.assertEqual(jwks, {"keys": [es256.public_jwk(self.jwk)]}) + self.assertNotIn(self.jwk["d"], b"".join(docs.values()).decode()) + token = es256.mint_jwt(self.jwk, es256.standard_claims(ISSUER, "aipf-mcp-grdm", "u")) + self.assertIsNotNone(es256.verify_jwt(token, jwks["keys"][0])) # what aipf's PyJWKClient will find by kid + + def test_discovery_names_the_issuer_and_the_keycloak_jwks_path(self): + disco = json.loads(self.mod.documents(ISSUER, self.key)["/realms/grdm-dev/.well-known/openid-configuration"]) + self.assertEqual(disco["issuer"], ISSUER) + self.assertEqual(disco["jwks_uri"], ISSUER + "/protocol/openid-connect/certs") + self.assertNotIn("token_endpoint", disco) # no token or exchange endpoint: the test signs, aipf never exchanges on S3 + + def test_a_private_key_is_refused_so_it_never_has_to_enter_the_container(self): + with self.assertRaisesRegex(ValueError, "private key") as caught: + self.mod.documents(ISSUER, es256.encode_key(self.jwk)) + self.assertNotIn(self.jwk["d"], str(caught.exception)) + + def test_bad_settings_are_refused_by_name_only(self): + pub = es256.public_jwk(self.jwk) + for issuer, key in ((ISSUER + "/", self.key), ("ftp://x/r", self.key), (ISSUER, "x" * 40), + (ISSUER, es256.encode_key({**pub, "crv": "P-384"})), + (ISSUER, es256.encode_key({k: v for k, v in pub.items() if k != "kid"}))): + with self.subTest(issuer=issuer), self.assertRaises(ValueError) as caught: + self.mod.documents(issuer, key) + self.assertNotIn(self.jwk["d"], str(caught.exception)) + + def test_main_exits_1_without_its_settings(self): + err = io.StringIO() + with mock.patch.dict("os.environ", {"OIDC_MOCK_ISSUER": ISSUER, "OIDC_MOCK_PUBLIC_JWK": ""}, clear=True), redirect_stderr(err): + self.assertEqual(self.mod.main(), 1) + self.assertIn("OIDC_MOCK_PUBLIC_JWK", err.getvalue()) + for value in ("%%", es256.encode_key(self.jwk)): # garbage, and the private key itself + with mock.patch.dict("os.environ", {"OIDC_MOCK_ISSUER": ISSUER, "OIDC_MOCK_PUBLIC_JWK": value}, clear=True), redirect_stderr(err): + self.assertEqual(self.mod.main(), 1) + self.assertNotIn(self.jwk["d"], err.getvalue()) + + def test_dockerfile_pins_the_base_by_digest_and_runs_non_root(self): + text = (IMAGE / "Dockerfile").read_text(encoding="utf-8") + self.assertRegex(text, r"(?m)^FROM python:3\.13-slim@sha256:[0-9a-f]{64}$") + self.assertIn("USER 65534:65534", text) + self.assertNotRegex(text, r"(?m)^RUN ") # nothing installed: the server is the standard library + + +class OidcMockServerTests(unittest.TestCase): + def setUp(self): + mod = load_module() + self.key = es256.encode_key(es256.public_jwk(es256.decode_key(es256.generate_signing_key()))) + handler = mod.make_handler(mod.documents(ISSUER, self.key)) + self.server = ThreadingHTTPServer(("127.0.0.1", 0), handler) + self.thread = threading.Thread(target=self.server.serve_forever, daemon=True) + self.thread.start() + self.err = io.StringIO() + self._redirect = redirect_stderr(self.err) + self._redirect.__enter__() + + def tearDown(self): + self.server.shutdown() + self.server.server_close() + self._redirect.__exit__(None, None, None) + + def request(self, method: str, path: str): + conn = http.client.HTTPConnection("127.0.0.1", self.server.server_address[1], timeout=5) + try: + conn.request(method, path) + resp = conn.getresponse() + return resp.status, resp.read(), resp.getheader("Content-Type") + finally: + conn.close() + + def test_serves_healthz_discovery_and_jwks(self): + self.assertEqual(self.request("GET", "/healthz")[:2], (200, b'{"status":"ok"}')) + status, body, ctype = self.request("GET", "/realms/grdm-dev/.well-known/openid-configuration") + self.assertEqual((status, ctype, json.loads(body)["issuer"]), (200, "application/json", ISSUER)) + status, body, _ = self.request("GET", "/realms/grdm-dev/protocol/openid-connect/certs?x=1") + self.assertEqual((status, len(json.loads(body)["keys"])), (200, 1)) + + def test_unknown_paths_are_404_and_writes_are_405(self): + self.assertEqual(self.request("GET", "/realms/grdm-dev/protocol/openid-connect/token")[0], 404) + self.assertEqual(self.request("POST", "/realms/grdm-dev/protocol/openid-connect/token")[0], 405) + self.assertEqual(self.request("DELETE", "/healthz")[0], 405) + self.assertNotIn("x=1", self.err.getvalue()) # the log line has no query string + + +if __name__ == "__main__": + unittest.main() diff --git a/installer/tests/test_s3_acceptance_fixtures.py b/installer/tests/test_s3_acceptance_fixtures.py new file mode 100644 index 00000000..4503f581 --- /dev/null +++ b/installer/tests/test_s3_acceptance_fixtures.py @@ -0,0 +1,147 @@ +"""acceptance/test_s3_aipf.py: S3 skips outside an eligible full environment before it reads the signing key +(PR #86 review r1 R1-01), and fails clearly only when the full profile publishes aipf but the key is missing.""" +from __future__ import annotations + +import importlib +import importlib.util +import inspect +import os +import subprocess +import sys +import tempfile +import types +import unittest +from pathlib import Path +from unittest import mock + +INSTALLER = Path(__file__).resolve().parents[1] +S3_FILE = INSTALLER / "acceptance" / "test_s3_aipf.py" + + +class _Skip(Exception): + pass + + +class _Fail(Exception): + pass + + +def _read_env(path): + out = {} + if path and Path(path).exists(): + for line in Path(path).read_text(encoding="utf-8").splitlines(): + if "=" in line: + k, _, v = line.partition("=") + out[k.strip()] = v.strip() + return out + + +def load_s3(): + """The unmodified module, imported with pytest and the Playwright conftest replaced by stubs (the pattern of + test_accept's C6 tests): fixtures stay plain functions, skip and fail raise distinguishable exceptions.""" + class _Mark: + def __getattr__(self, _name): + return lambda *a, **k: (lambda f: f) + + fake_pytest = types.ModuleType("pytest") + fake_pytest.mark = _Mark() + fake_pytest.fixture = lambda *a, **k: (a[0] if a and callable(a[0]) else (lambda f: f)) + + def _skip(msg): + raise _Skip(msg) + + def _fail(msg): + raise _Fail(msg) + + fake_pytest.skip, fake_pytest.fail = _skip, _fail + fake_conftest = types.ModuleType("acceptance.conftest") + fake_conftest.api = lambda ctx: ctx + fake_conftest._read_env_file = _read_env + with mock.patch.dict(sys.modules, {"pytest": fake_pytest, "acceptance.conftest": fake_conftest}): + sys.modules.pop("acceptance.test_s3_aipf", None) + module = importlib.import_module("acceptance.test_s3_aipf") + sys.modules.pop("acceptance.test_s3_aipf", None) + return module + + +class S3EligibilityStubTests(unittest.TestCase): + def setUp(self): + self.s3 = load_s3() + self.tmp = tempfile.TemporaryDirectory() + self.addCleanup(self.tmp.cleanup) + + def env(self, text: str) -> dict: + path = Path(self.tmp.name) / ".env" + path.write_text(text, encoding="utf-8") + return {"GRDM_ENV_FILE": str(path)} + + def test_core_and_minimum_without_the_key_skip(self): + for profile in ("core", "minimum"): + with self.subTest(profile=profile), self.assertRaises(_Skip): + self.s3.aipf_url({"AIPF_MCP_DATA_GRDM": 8768}, profile) # a port left behind does not count + + def test_full_without_the_port_skips(self): + with self.assertRaisesRegex(_Skip, "GRDM_PORT_AIPF_MCP_DATA_GRDM"): + self.s3.aipf_url({"WEB": 5000}, "full") + + def test_full_with_the_port_but_no_key_fails_naming_the_key(self): + url = self.s3.aipf_url({"AIPF_MCP_DATA_GRDM": 8768}, "full") + self.assertEqual(url, "http://127.0.0.1:8768") + with mock.patch.dict(os.environ, self.env("GRDM_PROFILE=full\n")), self.assertRaisesRegex(_Fail, "OIDC_MOCK_SIGNING_KEY"): + self.s3.signer(url) + + def test_full_with_the_key_signs_tokens_of_the_template_issuer(self): + from grdm_dev import es256 + + key = es256.generate_signing_key() + with mock.patch.dict(os.environ, self.env(f"OIDC_MOCK_SIGNING_KEY={key}\n")): + sign = self.s3.signer("http://127.0.0.1:8768") + claims = es256.verify_jwt(sign("u", azp="grdm-dev-webui"), es256.public_jwk(es256.decode_key(key))) + self.assertEqual((claims["iss"], claims["aud"]), (self.s3.ISSUER, self.s3.AUDIENCE)) + + def test_eligibility_runs_before_the_key_and_before_any_browser_fixture(self): + self.assertEqual(list(inspect.signature(self.s3.signer).parameters), ["aipf_url"]) + for test in (self.s3.test_s3_aipf_success, self.s3.test_s3_aipf_rejection): + self.assertEqual(list(inspect.signature(test).parameters)[:2], ["aipf_url", "signer"], test.__name__) + src = S3_FILE.read_text(encoding="utf-8") + self.assertNotIn("_aipf_or_skip(ports, profile)\n ctx", src) # no late, in-body guard any more + self.assertEqual(src.count('@pytest.fixture(scope="session")\ndef aipf_url'), 1) + self.assertEqual(src.count('@pytest.fixture(scope="session")\ndef signer'), 1) + + +def _accept_extra_importable() -> bool: + return all(importlib.util.find_spec(m) is not None for m in ("pytest", "playwright", "pytest_playwright")) + + +@unittest.skipUnless(_accept_extra_importable(), "pytest/playwright not importable in this interpreter (use .venv with [accept])") +class S3EligibilityPytestTests(unittest.TestCase): + """The real pytest run of the S3 module, with no browser installed (PLAYWRIGHT_BROWSERS_PATH points nowhere): + a skip or a key failure must come before pytest-playwright tries to start Chromium.""" + + def run_s3(self, env_text: str) -> str: + with tempfile.TemporaryDirectory() as tmp: + env_file = Path(tmp) / ".env" + env_file.write_text(env_text, encoding="utf-8") + env = {**os.environ, "GRDM_ENV_FILE": str(env_file), "PYTHONPATH": str(INSTALLER), + "PLAYWRIGHT_BROWSERS_PATH": str(Path(tmp) / "no-browsers")} + env.pop("GRDM_ACCEPT_RESULTS", None) + proc = subprocess.run([sys.executable, "-m", "pytest", str(S3_FILE), "-q", "--no-header", "-p", "no:cacheprovider", + "--browser", "chromium", "-rA"], cwd=INSTALLER, env=env, capture_output=True, text=True, timeout=120) + return proc.stdout + proc.stderr + + def test_core_minimum_and_full_without_port_skip(self): + for text in ("GRDM_PROFILE=core\n", "GRDM_PROFILE=minimum\nGRDM_PORT_AIPF_MCP_DATA_GRDM=8768\n", "GRDM_PROFILE=full\n"): + with self.subTest(env=text): + out = self.run_s3(text) + self.assertIn("2 skipped", out) + self.assertNotIn("error", out.lower().split("short test summary")[-1].replace("skipped", "")) + + def test_full_with_port_and_no_key_errors_on_the_key(self): + out = self.run_s3("GRDM_PROFILE=full\nGRDM_PORT_AIPF_MCP_DATA_GRDM=8768\n") + self.assertIn("2 errors", out) + self.assertIn("OIDC_MOCK_SIGNING_KEY is not in runtime/.env", out) + self.assertNotIn("Executable doesn't exist", out) # failed before any browser start + + +if __name__ == "__main__": + unittest.main() diff --git a/installer/tests/test_secrets.py b/installer/tests/test_secrets.py index 3b212575..029b2ade 100644 --- a/installer/tests/test_secrets.py +++ b/installer/tests/test_secrets.py @@ -367,9 +367,70 @@ def test_every_key_referenced_by_a_template_is_generated(self): data.pop("command", None) # container-shell text; compose render escapes its `$` as `$$` data.pop("note", None) referenced.update(re.findall(r"\$\{?([A-Z][A-Z0-9_]*)", json.dumps(data))) - generated = set(secrets.SECRET_KEYS) | set(secrets.default_non_secrets("core")) + generated = set(secrets.SECRET_KEYS) | set(secrets.default_non_secrets("core")) | set(secrets.DERIVED_KEYS) self.assertEqual(sorted(referenced - generated), []) + def test_the_vault_key_is_a_fernet_key_and_the_signing_key_an_ec_jwk(self): + """aipf_mcp_data_grdm exits at start on a vault key that is not Fernet (32 bytes, urlsafe base64 with padding); + oidc_mock on a signing key that is not an EC P-256 JWK (ADR-15). Every draw is checked.""" + from grdm_dev import es256 + for _ in range(50): + values = secrets.fresh_secrets() + vault = values["GRDM_VAULT_KEY"] + self.assertEqual(len(vault), 44) + self.assertEqual(len(base64.urlsafe_b64decode(vault)), 32) + self.assertTrue(vault.endswith("=")) + self.assertEqual(es256.decode_key(values["OIDC_MOCK_SIGNING_KEY"])["crv"], "P-256") + self.assertEqual(secrets.secret_format_problems(values), []) + self.assertEqual(secrets.check_no_defaults(values), []) + self.assertNotEqual(secrets.fernet_key(), secrets.fernet_key()) + + def test_a_vault_or_signing_key_in_the_wrong_format_is_named_not_shown(self): + values = secrets.fresh_secrets() + good_signing = values["OIDC_MOCK_SIGNING_KEY"] + for key, bad in (("GRDM_VAULT_KEY", "x" * 44), ("GRDM_VAULT_KEY", base64.urlsafe_b64encode(b"k" * 31).decode()), + ("OIDC_MOCK_SIGNING_KEY", "y" * 300), ("OIDC_MOCK_SIGNING_KEY", good_signing[:-8])): + broken = {**values, key: bad} + with self.subTest(key=key): + self.assertEqual(secrets.secret_format_problems(broken), [key]) + self.assertIn(key, secrets.check_no_defaults(broken)) + + def test_the_public_jwk_follows_the_signing_key_through_init_missing_and_rotate(self): + """oidc_mock gets only OIDC_MOCK_PUBLIC_JWK (PR #86 review r1); it must always be the signing key's public part.""" + from grdm_dev import es256 + + def public_of(values): + return es256.encode_key(es256.public_jwk(es256.decode_key(values["OIDC_MOCK_SIGNING_KEY"]))) + + ctx = make_ctx(self.root, profile="full") + self.run_cmd(ctx, ns(secrets_action="init", force=False)) + path = self.root / "runtime" / ".env" + values = secrets.read_env(path) + self.assertEqual(values[secrets.PUBLIC_JWK_KEY], public_of(values)) + self.assertNotIn('"d"', es256.b64url_decode(values[secrets.PUBLIC_JWK_KEY]).decode()) + self.assertNotIn(secrets.PUBLIC_JWK_KEY, secrets.SECRET_KEYS) + secrets.write_env(path, {k: v for k, v in values.items() if k != secrets.PUBLIC_JWK_KEY}) + self.run_cmd(ctx, ns(secrets_action="init", force=False, missing=True)) + self.assertEqual(secrets.read_env(path)[secrets.PUBLIC_JWK_KEY], public_of(values)) + before = secrets.read_env(path) + self.assertEqual(self.run_cmd(ctx, ns(secrets_action="rotate", timeout=1.0)), exit_codes.OK, self.err.getvalue()) + after = secrets.read_env(path) + self.assertNotEqual(after["OIDC_MOCK_SIGNING_KEY"], before["OIDC_MOCK_SIGNING_KEY"]) + self.assertEqual(after[secrets.PUBLIC_JWK_KEY], public_of(after)) + self.assertEqual(secrets.derived_values({"OIDC_MOCK_SIGNING_KEY": "broken"}), {}) + + def test_init_missing_adds_the_aipf_keys_to_an_older_env(self): + ctx = make_ctx(self.root, profile="core") + self.run_cmd(ctx, ns(secrets_action="init", force=False)) + path = self.root / "runtime" / ".env" + old = secrets.read_env(path) + secrets.write_env(path, {k: v for k, v in old.items() if k not in ("GRDM_VAULT_KEY", "OIDC_MOCK_SIGNING_KEY")}) + self.assertEqual(self.run_cmd(ctx, ns(secrets_action="init", force=False, missing=True)), exit_codes.OK, self.err.getvalue()) + after = secrets.read_env(path) + self.assertEqual(secrets.secret_format_problems(after), []) + self.assertEqual(after["POSTGRES_PASSWORD"], old["POSTGRES_PASSWORD"]) + self.assertNotIn(after["OIDC_MOCK_SIGNING_KEY"], self.out.getvalue()) + def test_generated_cas_keys_satisfy_the_rules_of_the_image(self): """Every draw, because a length that is right on average still fails closed on the unlucky one.""" for _ in range(200): diff --git a/manifest/upstreams.json b/manifest/upstreams.json index fa7804d4..4368fe3a 100644 --- a/manifest/upstreams.json +++ b/manifest/upstreams.json @@ -954,6 +954,35 @@ "context": "installer/images/fakecas", "tag": "9a9ddd73b225-2" } + }, + { + "adopted_prs": [], + "base_image": "python:3.13-slim", + "base_image_digest": "sha256:e48f59cdba69a3cbd9ed2cf0f348d90d45a8d350d98134cf1070bc106ee9ed90", + "candidate_prs": [], + "id": "oidc-mock", + "integration_branch": null, + "integration_sha": null, + "integration_repo": null, + "note": "Mock OIDC issuer for aipf_mcp_data_grdm (full profile, ADR-15): one standard-library Python file on a digest-pinned python slim base (installer/images/oidc-mock) that serves the discovery document and the JWKS of OIDC_MOCK_PUBLIC_JWK, the public part of the EC P-256 key OIDC_MOCK_SIGNING_KEY that `grdm-dev secrets init` writes to runtime/.env (the container never gets the private key); the acceptance test S3 signs its tokens with that key. It signs nothing itself and has no token endpoint. Keycloak, the AI platform's real issuer, would be the heaviest service of the stack for two JSON documents. Not published; uid 65534, read-only root, no volume.", + "platforms": [ + "linux/arm64", + "linux/amd64" + ], + "profile": "full", + "role": "infra", + "services": [ + "oidc_mock" + ], + "target_image": { + "M4": null, + "M5": null + }, + "upstreams": [], + "image_build": { + "context": "installer/images/oidc-mock", + "tag": "2" + } } ], "generated_at": "2026-11-05", diff --git a/manifest/upstreams.toon b/manifest/upstreams.toon index 902ac274..0cc5987c 100644 --- a/manifest/upstreams.toon +++ b/manifest/upstreams.toon @@ -1,7 +1,7 @@ schema_version: 1 generated_at: 2026-11-05 note: "Names and SHAs only. This repository is public: no URLs to private repositories, no credentials." -components[18]: +components[19]: - id: osf.io role: core integration_repo: RDM-osf.io-dev2027 @@ -143,3 +143,9 @@ components[18]: profile: minimum upstreams[0]: adopted_prs[0]: + - id: oidc-mock + role: infra + integration_repo: null + profile: full + upstreams[0]: + adopted_prs[0]: