feat(plugins): publish Model Fit for Codex Desktop

.agents/plugins/marketplace.json

@@ -0,0 +1,20 @@
+{
+  "name": "atls-codex",
+  "interface": {
+    "displayName": "ATLS Codex"
+  },
+  "plugins": [
+    {
+      "name": "model-fit",
+      "source": {
+        "source": "local",
+        "path": "./plugins/model-fit"
+      },
+      "policy": {
+        "installation": "AVAILABLE",
+        "authentication": "ON_INSTALL"
+      },
+      "category": "Developer Tools"
+    }
+  ]
+}

.github/workflows/release.yml

@@ -0,0 +1,34 @@
+name: Release
+
+on:
+  push:
+    branches: [master]
+    paths:
+      - 'plugins/**'
+      - LICENSE
+      - tools/build.py
+
+permissions:
+  contents: write
+
+jobs:
+  release:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+      - uses: actions/setup-python@v7
+        with:
+          python-version: '3.9'
+      - uses: astral-sh/[email protected]
+        with:
+          version: '0.17.2'
+          src: plugins/model-fit evals/model-fit tools
+      - run: ruff format --check plugins/model-fit evals/model-fit tools
+      - run: python -m unittest discover -s evals/model-fit -p 'test_*.py'
+      - run: python tools/build.py
+      - name: Publish GitHub Release
+        env:
+          GH_TOKEN: ${{ github.token }}
+        run: |
+          version="$(python tools/build.py --version)"
+          gh release create "v${version}" dist/*.zip --target "$GITHUB_SHA" --title "Model Fit v${version}" --generate-notes

.github/workflows/verify.yml

@@ -0,0 +1,23 @@
+name: Verify
+
+on:
+  pull_request:
+
+permissions:
+  contents: read
+
+jobs:
+  verify:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+      - uses: actions/setup-python@v7
+        with:
+          python-version: '3.9'
+      - uses: astral-sh/[email protected]
+        with:
+          version: '0.17.2'
+          src: plugins/model-fit evals/model-fit tools
+      - run: ruff format --check plugins/model-fit evals/model-fit tools
+      - run: python -m unittest discover -s evals/model-fit -p 'test_*.py'
+      - run: python tools/build.py

.gitignore

@@ -0,0 +1,6 @@
+.DS_Store
+.tmp/
+dist/
+__pycache__/
+*.pyc
+.venv/

LICENSE

@@ -0,0 +1,29 @@
+BSD 3-Clause License
+
+Copyright (c) 2026, Atlantis
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its
+   contributors may be used to endorse or promote products derived from
+   this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

README.md

@@ -1 +1,74 @@
-# codex
+# ATLS Codex
+
+Public Codex experiments and reusable packages from ATLS.
+
+## Model Fit
+
+Model Fit evaluates the next bounded task against the active Codex profile. The
+full [Model Fit package](plugins/model-fit/README.md) adds an opt-in profile
+handoff for Codex Desktop on macOS. Its runtime code, skill, schemas and local
+hook setup live together under `plugins/model-fit/`.
+
+The repository also builds a separate **guidance-only** package from the same
+selection skill. That package recommends a profile and does not switch it or
+install hooks. See [the eval suite](evals/model-fit/README.md) for synthetic
+cases and the commands that run them.
+
+The full package is distributed from this repository for manual Desktop
+installation. OpenAI's public Plugins Directory currently excludes plugins
+containing lifecycle hooks; the guidance-only ZIP is the candidate for that
+directory. The ZIPs are separate products with separate behavior and review.
+The [verification workflow](.github/workflows/verify.yml) runs on PRs with
+Ubuntu, Python 3.9, Ruff's own GitHub Action, deterministic tests and package
+assembly. The [release workflow](.github/workflows/release.yml) runs the same
+checks when package source reaches `master`, then creates a versioned GitHub
+Release with both ZIPs attached. The release's `model-fit-guidance` ZIP is the
+input for a separate OpenAI directory upload and review.
+Locally, `python3 tools/build.py` writes versioned `dist/model-fit-*.zip` and
+`dist/model-fit-guidance-*.zip`. The first copies the full plugin tree; the
+second copies the shared skill up to its hook-specific section, then adds its
+own manifest, README, icon and license. That transformation is why this
+package builder exists: a generic ZIP action can archive a directory but
+cannot derive the guidance-only variant from the shared skill. Both manifests
+share one release version. Tests inspect the ZIP members, manifest identity,
+privacy boundary and byte-for-byte reproducibility. GitHub Release creation
+fails if that version already has a release.
+
+## Distribution and updates
+
+The installable source under `plugins/model-fit/` is the primary GitHub
+distribution. A user adds `atls/codex` as a Codex marketplace, installs the
+plugin, and explicitly configures hooks for a selected project and thread.
+After a new version is released, they refresh the marketplace and reinstall
+the plugin; project-local hook definitions are reviewed and trusted again if
+their commands changed.
+
+There are two separate OpenAI routes:
+
+- [OpenAI Community Plugins](https://github.com/openai/community-plugins/blob/main/CONTRIBUTING.md)
+  accepts external contributions through a fork and PR. A contribution needs
+  the package source under `plugins/model-fit/`, its required Codex manifest,
+  marketplace entry, cold-start guide, repository-specific master test suite,
+  CLA and maintainer review. Their repository shape is adapted in the fork;
+  the ZIP is not pushed there. We do not auto-push to that repository. An
+  update is a new upstream PR based on a new version of this source.
+- The [public Plugins Directory](https://developers.openai.com/plugins/deploy/submission)
+  accepts a plugin ZIP through the submission portal. Download the separate
+  `model-fit-guidance` asset from the exact GitHub Release for that route. After publisher
+  verification, upload the archive, resolve the portal's automated findings,
+  submit for review, and publish the approved version. An update bumps the
+  version, uploads a new complete ZIP, and passes the portal checks again.
+
+The official [`openai/codex-action`](https://github.com/openai/codex-action)
+runs Codex in GitHub Actions using an API key; it does not package plugins.
+This repository's build script creates the two archives without an API key.
+For an update, change the source and bump the version in both manifests, run the
+same checks, merge to master to produce a new GitHub Release, refresh the Git
+marketplace revision, and then submit separate
+upstream Community Plugins PR and Directory ZIP update as applicable. Neither
+OpenAI channel is updated merely because this repository's CI passed.
+
+Neither package includes user conversations, Thread Memory, credentials,
+attachments, project hook definitions, or local audit data.
+
+The source and both package archives use the [BSD 3-Clause License](LICENSE).

evals/model-fit/README.md

@@ -0,0 +1,56 @@
+# Model Fit evaluations
+
+These cases exercise the public Model Fit skill and the hook code without
+publishing user conversations. The scenario inputs and answer examples are
+synthetic. Run results, current source snapshots, model traces and token usage
+are written under ignored `.tmp/model-fit-evals/`.
+
+## Cases
+
+| Case | Property checked |
+| --- | --- |
+| `backlog-read` | Judge the next read-only issue lookup, not the whole project |
+| `issue-draft` | Explain any lighter model and effort choices independently |
+| `local-search-design` | Ground an upgrade in the specific design work |
+| `kubernetes-backlog` | Avoid a profile rule inferred from a technology name |
+| `confirmed-choice` | Keep a profile explicitly chosen for the same work without a model call |
+| `missing-source` | Return `unknown` when an obligatory source is unavailable |
+
+The four first cases use a semantic grader with four criteria: nearest bounded
+work, grounded profile change, separate model and effort reasoning, and no
+universal technology rule. `behavior-calibration.json` contains synthetic
+positive and negative examples for that grader. `answer-cases.json` checks a
+separate text-only replay relevance classifier. The unit tests cover complete
+queued input, images, state locks, error audit, project hook setup and privacy.
+
+## Deterministic checks
+
+From the repository root:
+
+```sh
+python3 -m unittest discover -s evals/model-fit -p 'test_*.py'
+python3 tools/build.py
+```
+
+These commands do not require an OpenAI API key or a model inference request.
+The test that inspects installed Codex hook definitions skips itself when the
+CLI is unavailable. The build creates two ignored ZIP files under `dist/` and
+verifies their contents.
+
+## Live behavior evaluation
+
+The following commands use the current Codex sign-in and consume model usage:
+
+```sh
+python3 evals/model-fit/run.py --controller --case backlog-read
+python3 evals/model-fit/run.py --calibrate-behavior
+python3 evals/model-fit/run-answer.py
+```
+
+`run.py` fetches current official OpenAI guidance and the local Codex model
+catalog, runs each selected case in an isolated read-only app-server session,
+and records exact source hashes and observed results. The grader checks the
+meaning of the answer rather than requiring one model for every Kubernetes or
+writing task. Evaluate a changed skill from that change's checkout; the
+runner does not mutate working projects or publish results. Model failures
+remain failures in the local manifest.

evals/model-fit/answer-cases.json

@@ -0,0 +1,23 @@
+[
+  {
+    "id": "unrelated-prior-task",
+    "request": "Проверь, что поиск по локальным снимкам обновляет индекс после изменения файла",
+    "task_context": "Проверка обновления индекса",
+    "answer": "Закрыл вчерашнюю задачу о цвете кнопки и добавил к ней комментарий",
+    "expected": "unrelated"
+  },
+  {
+    "id": "specific-blocker",
+    "request": "Проверь, что поиск по локальным снимкам обновляет индекс после изменения файла",
+    "task_context": "Проверка обновления индекса",
+    "answer": "Не смог запустить индексатор: тестовая директория снимков отсутствует. Нужен путь к ней",
+    "expected": "relevant"
+  },
+  {
+    "id": "completed-visual-fix",
+    "request": "Верни оранжевый подзаголовок над заголовком карточки",
+    "task_context": "Одна правка карточки",
+    "answer": "Подзаголовок восстановлен над заголовком карточки; проверил отображение на странице",
+    "expected": "relevant"
+  }
+]

evals/model-fit/behavior-calibration.json

@@ -0,0 +1,26 @@
+[
+  {
+    "id": "technology-name-upgrade",
+    "case_id": "kubernetes-backlog",
+    "expected_pass": false,
+    "answer": "{\"active_model\": \"gpt-6.1-sol\", \"active_effort\": \"medium\", \"active_speed\": \"standard\", \"decision\": \"recommend_stronger\", \"recommended_model\": \"gpt-6-astra\", \"recommended_effort\": \"xhigh\", \"recommended_speed\": null, \"official_source_url\": \"https://developers.openai.com/api/docs/guides/model-selection\", \"official_guidance\": \"Choose the best model.\", \"task_inference\": \"Kubernetes требует максимум\", \"reason\": \"Kubernetes сложен, поэтому нужен Astra/xhigh\"}"
+  },
+  {
+    "id": "bounded-issue-lookup",
+    "case_id": "kubernetes-backlog",
+    "expected_pass": true,
+    "answer": "{\"active_model\": \"gpt-6.1-sol\", \"active_effort\": \"medium\", \"active_speed\": \"standard\", \"decision\": \"keep\", \"recommended_model\": null, \"recommended_effort\": null, \"recommended_speed\": null, \"official_source_url\": null, \"official_guidance\": null, \"task_inference\": null, \"reason\": \"Нужно только прочитать одну задачу и перечислить её пункты; текущий профиль достаточен\"}"
+  },
+  {
+    "id": "grounded-design-change",
+    "case_id": "local-search-design",
+    "expected_pass": true,
+    "answer": "{\"active_model\": \"gpt-6-luna\", \"active_effort\": \"low\", \"active_speed\": \"standard\", \"decision\": \"recommend_stronger\", \"recommended_model\": \"gpt-6.1-sol\", \"recommended_effort\": \"medium\", \"recommended_speed\": null, \"official_source_url\": \"https://developers.openai.com/api/docs/guides/model-selection\", \"official_guidance\": \"Compare models on the same task and use the least costly model that meets quality.\", \"task_inference\": \"Требуется связать индекс, обновление и восстановление; для этого этапа Sol/medium ожидаемо надёжнее Luna/low\", \"reason\": \"Следующий шаг — выбрать индекс, обновление и восстановление; Sol лучше подходит для проверки взаимосвязанных решений, medium даёт достаточно глубины\"}"
+  },
+  {
+    "id": "unjustified-effort",
+    "case_id": "local-search-design",
+    "expected_pass": false,
+    "answer": "{\"active_model\": \"gpt-6-luna\", \"active_effort\": \"low\", \"active_speed\": \"standard\", \"decision\": \"recommend_stronger\", \"recommended_model\": \"gpt-6-astra\", \"recommended_effort\": \"xhigh\", \"recommended_speed\": null, \"official_source_url\": \"https://developers.openai.com/api/docs/guides/model-selection\", \"official_guidance\": \"Use stronger reasoning for difficult tasks.\", \"task_inference\": \"Проект масштабный\", \"reason\": \"Переходим на Astra/xhigh, потому что это большая система\"}"
+  }
+]

evals/model-fit/cases.json

@@ -0,0 +1,57 @@
+[
+  {
+    "id": "backlog-read",
+    "source": "Synthetic read-only task",
+    "prompt": "Покажи три открытые задачи этого репозитория и ничего не меняй",
+    "model": "gpt-6.1-sol",
+    "effort": "medium",
+    "speed": "standard",
+    "expected": "Assess the bounded issue lookup rather than the surrounding project"
+  },
+  {
+    "id": "issue-draft",
+    "source": "Synthetic writing task",
+    "prompt": "Составь черновик одной bug-задачи по готовым фактам, не публикуй её",
+    "model": "gpt-6-astra",
+    "effort": "high",
+    "speed": "standard",
+    "expected": "Justify any lighter recommendation separately for model and effort"
+  },
+  {
+    "id": "local-search-design",
+    "source": "Synthetic architecture task",
+    "prompt": "Спроектируй поиск по локальным снимкам: индексацию, обновление и восстановление после сбоя",
+    "model": "gpt-6-luna",
+    "effort": "low",
+    "speed": "standard",
+    "expected": "Explain the concrete quality gap for the next design step"
+  },
+  {
+    "id": "kubernetes-backlog",
+    "source": "Synthetic technology-name control",
+    "prompt": "Открой одну задачу о Kubernetes и кратко перечисли её незакрытые пункты без изменений",
+    "model": "gpt-6.1-sol",
+    "effort": "medium",
+    "speed": "standard",
+    "expected": "Do not infer a frontier model from the word Kubernetes alone"
+  },
+  {
+    "id": "confirmed-choice",
+    "source": "Synthetic user-choice control",
+    "prompt": "Продолжи ту же проверку конфигурации после моего выбора оставить текущий профиль",
+    "model": "gpt-6.1-sol",
+    "effort": "medium",
+    "speed": "standard",
+    "prior_choice": "The user explicitly kept this profile for the same work",
+    "expected": "Keep without another model invocation"
+  },
+  {
+    "id": "missing-source",
+    "source": "Synthetic source-failure control",
+    "prompt": "Оцени профиль для ограниченной правки после недоступности каталога моделей",
+    "model": "gpt-6-luna",
+    "effort": "low",
+    "speed": "standard",
+    "expected": "Return unknown rather than inventing a verified profile"
+  }
+]