Repository navigation
Landing page - #6
Closed
ChainsawFrogge wants to merge 108 commits into
Closed
ChainsawFrogge wants to merge 108 commits into
ChainsawFrogge wants to merge 108 commits into
Conversation
…architectures (ARM/CPU). Introduce agent subsystem configuration and update environment handling for CORS and ACP integration. Update README with new agent features and migration instructions.
Fetch agent configuration after authorization and keep provider credentials on the backend.
… tool failure back to agent
Toollist bug in code4me-agent
- Add f3c4d5e6a7b9 policy-snapshot migration, crud/schemas/manager support - Add ACP managed endpoints (capabilities/readiness/runs/inference) alongside chat/completions relay - Enforce policy fail-closed, provider creds stay server-side - Add managed backend tests
- AgentConfig managed_mode/allowed_tools/approval, frozenset allowlist with mcp__* wildcard - ToolRegistry approval policies (auto/per_step/suggestion_only), definitions filtering, denied-continues loop - acp_runtime managed sessions via plugin bridge, apply_config rebuild; runtime_auth managed inference transport - Backend relay priority managed_backend>code4me_backend>openai, dual tools/allowed_tools compat - Add policy enforcement and CLI tests
* feat: add study-scoped agent assignments * feat: manage agent arms and assignments by study
- real HTTP + PostgreSQL TestClient suite driving black-box POST /studies then POST /join, a genuinely qualified packaged macos/arm64 release, bootstrap revision-free manifest, session create/reuse/heartbeat, telemetry accepted/duplicate/retryable(tampered capability)/terminal, receipt, close, revoke - repair test_research_schema_lifecycle fixtures: canonical uppercase join codes and a release_id on the lock-study profile (stale vs the uppercased lookup and RELEASE_UNRESOLVED requirement)
Conflict resolutions: - agents/event_writer.py: keep the relay model-call payload from the UI overhaul (call_mode, requested_model/api_kind) and take total_tokens and step_index counts plus the tool_schema_bytes promotion from gt_telemetry; context_window_size_bytes falls back to the extra bag the same way. - research/telemetry/adapters.py: build_legacy_events takes both the optional privacy policy (metadata stripping) and the reserved first_sequence; per-task emitter namespaces, raw unknown kinds and trace/span defaults come from gt_telemetry. - research/telemetry/privacy/engine.py: keep the shared _declared_field_classes resolver (same resolution as the bootstrap manifest); it already maps the study vocabulary the way gt_telemetry's inline mapping does. - tests/backend_tests/research/test_legacy_adapter.py: keep both sides' tests.
…uences - agents/ingest.py: map_event_to_columns now also carries run_id, message_id, tool_call_id and the permission decision fields for the canonical fact, but the legacy branch passed every key to crud.append_agent_event, which has no such parameters. Every self-report batch for a task outside a study failed with a TypeError (HTTP 500). Those keys are now left out of the legacy write. - The self-report ingest already reserves the batch's task event indexes; record_legacy_facts reserved a second block, so every batch started with an emitter sequence gap. It now accepts the caller's reservation (first_sequence) and only reserves when none is given. - build_legacy_events computed the trace_id/span_id defaults (run id, source event id) but passed the fact's original correlations to the builder, dropping them.
The built-in agent now reports its own permission requests and decisions through the relay (gt_telemetry), next to the permission events the ACP proxy already records for the same round-trips. It also records a policy-scope "accepted" for every tool call the approval policy let through, and uses accepted/rejected where ACP uses allow/reject. Study analytics: - relay permission events count only in research sessions without ACP permission events, so a proxied decision is not counted twice; - policy-scope decisions (auto-approval, suggestion-only refusals) were never put to the participant and are not counted or listed on the timeline (also excluded in the timeline query, so they don't use up its 200 rows); - accepted/rejected are read as allow/reject, so denial rates and the decision breakdown use one vocabulary. Agent dashboard: edit acceptance reads only the agent's own reports (as the model and tool counts do) and leaves out policy-scope decisions, which otherwise pushed the rate towards 100% under the auto policy.
Follow-up from review of the permission counting: - The built-in agent answers repeat tool calls from an earlier "allow for this session" without asking anyone, but reported them with the same scope as the user's own answer. They now report scope session_cached. - A relay decision counts only when someone made it: not scope policy or session_cached, not "unavailable" (the request reached no one), and not a row whose decision a policy without behavioural fields stripped (previously counted as "unknown"). The request of such a decision does not count either. The same rule is used by the timeline query and the agent dashboard's edit acceptance. - The timeline keeps a missing decision as null instead of "unknown". - Agent run detail: request_id read a correlation field that does not exist, so it was always null; it now comes from the payload, falling back to correlations.correlation_id when a policy strips the payload field. - Tests: the timeline query's own filter (row limit), an ACP row in the dashboard test, the cached approval scope and the run detail request id.
Research website UX/UI overhaul with the backend it needs: study analytics API, accounts, provider connections, agent profiles (system prompts), participant "My studies" and telemetry privacy fixes.
Brings origin/gt_telemetry (merged into the UI overhaul in 0e96c20) and the integration fixes: legacy self-report writes, gap-free relay sequences, trace/span defaults, permission decisions counted once and only when someone decided, run-detail request_id.
The step passed its bootstrap session only as the ACP session_id. When the account has one active research session the server infers it, but the plugin layer's Kotlin fixture opens a second one, and the server never guesses between sessions (409 RESEARCH_CONTEXT_AMBIGUOUS), so the plugin layer always failed at send_message. The run request now also carries research_session_id.
- Agentic system prompt built from the frozen policy (tools, approval policy, allowlisted commands, iteration budget). - ReAct loop streams assistant text per model call, returns tool errors to the model as structured results, keeps tool_calls/tool pairs valid on cancel and errors, and ends the budget with a forced summary. - Memory window never splits a tool-call group, always keeps the system message and the current turn, and elides old tool output first. - Provider retries transient failures with backoff and honours cancel. - Tools: numbered/paged read_file, glob_files, grep_files, edit_file, delete_file, move_file, hardened replace_text, run_command with timeouts and process-group kill, update_plan as ACP plan updates. - Every tool has an ACP card; the server catalogue lists the new tools and a contract test keeps the runtime tools inside it.
Agent runtime - Honour the profile system prompt: parsed from agent-config and the managed run policy and placed before the operational instructions (runtime half of origin/sys_prompt 8d15b7c, ported). - Report a backend 5xx as a server/provider failure, not a rejection. Telemetry and analytics - Keep legacy_kind/upstream_status (SYSTEM) so agent dashboards do not read zero under a METRICS-only policy. - Store no confidence for provider models instead of a fabricated 0.0; had_generation.confidence is nullable (existing databases need ALTER TABLE had_generation ALTER COLUMN confidence DROP NOT NULL). - Finalize proxied runs on agent.run.completed with the reported outcome, aggregate canonical events into task totals, and store cancelled turns as cancelled. - Deduplicate retried self-report batches against research_event. - Fix the latency-distribution query and stop leaking SQL errors. - Compare agent task times with a local window; read a timeline margin. API and website - Isolate a failing classic provider per model; 502 with a JSON body when the inference provider is unreachable. - Serve /api/config with and without the trailing slash. - Resend verification reports delivery failures. - Participant redirect from Studies, duplicate connection label shown once on its field, profile editor keeps the system prompt, 32k context default, Overview heading and icons, router future flags. Release and tests - Participant recipe declares the adapter per artifact and writes the plugin runtime manifest. - e2e harness follows the packaged-runtime overlay and the eighth UI step, and keeps sandbox credentials in memory. - Test runs no longer flush Redis and use the TEST_DATABASE_URL DB.
Bring the "e2e old" stash (2026-09-22) onto test_draft and make the whole gate runnable from one command without manual host setup. From the stash - ACP evidence scoped to one launch: the IDE-registered entry's proxy and agent identity and --agent-digest, exactly one provider request for the first prompt, and the emitter's canonical telemetry counts. IDE rows are scoped by the entry's research session, because the IDE emits them before the proxy launch. - Real Goose and Codex (the plugin's vendored adapter) over ACP against a local deterministic provider: agent-probe and --layer agents. - Browser suite with 37 fixed check ids, exact inventory validation, persisted read-model re-reads and its own seeded state; D1 follows the redirect of non-researchers to My studies. - Layers ui, plugin, backend, agents and browser; agents and browser run even when the IDE layer fails. - Not ported: goose/codex JSON release registration and conformance receipts, whose endpoints f042add removed. Workflow scenarios with those frameworks fail fast and point at the agents layer. Prerequisites - code4me_e2e/prereqs.py and `setup [--check]`: per-layer checks that start Colima behind a Colima docker context, build the backend image when its requirements.txt differs, create e2e/.venv pinned like the release CI and e2e/.browser-venv, install website dependencies, build the vendored codex-acp and stop leftover harness IDEs. Anything else blocks only its layers with a remediation; `test --no-setup` checks without provisioning. - run_workflow's exit code covers the steps it ran, so one layer's blocked prerequisite cannot fail another layer. - SIGTERM unwinds like Ctrl-C, so provisioning and IDE processes stop. - A port counts as busy only when something listens on it. - report.json and --json carry the layers and the prerequisite checks.
… budgets
Study arms that run Goose or the bundled code4me2-agent now spend from the
study's server-held provider key through metered relays, and every enrollment
carries a USD budget that the server enforces with reserve/settle accounting.
- Ledger: enrollment_inference_balance, inference_reservation and
inference_budget_adjustment tables plus provider_model_price and the study
default/warning columns, all in the consolidated revision; idempotent SQL
for existing databases in docs/research-platform/RELEASES.md.
- Budget package (research/budget): worst-case holds, output caps, provider
usage extraction, price-table or provider-cost settlement, forfeit/void
rules, and the InferenceMeter wired into run_inference.
- Research inference gateway for Goose
(POST /api/research/inference/v1/chat/completions) behind a signed
inference capability; /api/agent/inference and the ACP relays are metered
and share one enrollment-scoped funded-access gate.
- Bootstrap manifests carry an inference_gateway block; Goose releases must
declare the runtime bindings (INFERENCE_GATEWAY_UNBOUND otherwise).
- Budget API under /api/research/studies/{id}/budget with adjustments,
ledger and adjustment history; connection model prices; budget blocks on
participant, session, participants-list and analytics payloads.
- Website: prices on connections, default budget on study create and
settings, Spent/Budget columns and the Adjust budget drawer, participant
"Budget remaining"; e2e gateway/quota probes and browser scenarios E5-E7.
Fail-closed: a study default of 0 or an unpriced model refuses every metered
call (402 quota_exhausted / 503 price_missing). Codex arms are unmetered.
* feat: testing and agent building workflows * add cache to workflow and add manifest json * participant release fix * fix: handle windows paths
…byte-exact test fixtures
Add a server-owned opt-out (user.data_collection_opted_out_at) and a privacy package that decides collection and erases what was collected in one transaction: completion and chat requests with their context, telemetry and generations, chats, agent runs and memory, and study participation with its telemetry. Each erased enrollment leaves a content-free deletion-ledger record. - GET /api/user/privacy, PUT /api/user/privacy/collection and POST /api/user/privacy/erase for the signed-in account - DELETE /api/user/delete always erases the data, refuses accounts that own studies or agent profiles (409) and revokes the account's tokens - completion, chat and feedback storage, agent content and memory, project context and study joins honour the opt-out; writers share-lock the account row, so an erase cannot miss a concurrent write - WITHDRAWN enrollment status for participant withdrawal - request payloads are no longer logged; meta_query FK indexes let an erase scale Existing databases need the ALTER TABLE and indexes listed in RELEASES.md.
Every signed-in user can see what is stored about them, turn data collection off or back on, erase their data and delete their account. Research views label the new WITHDRAWN status and explain a join refused while opted out.
…the production stack - Check the bootstrap signing secret on every request and refuse to start without it in production. - Rate-limit by the client address the reverse proxy forwards, with a floor for participant paths. - Drop the emitter-sequence unique constraint that rejected valid events (revision b7c1d2e3f4a5). - Accept a session's late tail within a grace window, rotate idle context sessions at bootstrap and session creation, and bind a terminal session's capability to its enrollment and study. - Add the production stack: docker-compose.prod.yml with migrate and backup services, nginx (with a TLS example) and Redis configs, a website Dockerfile, Postgres backup and restore scripts kept out of the build context, /api/health, Secure cookies, a Redis password and docs/research-platform/PRODUCTION_DEPLOYMENT.md. - Track docs/ and add the decision documents the study relies on: the agent configuration contract, the analytics scope, and the content policy and protocol vocabulary decisions.
…the review findings Keeps the parts of "Validate BYOA agent sessions and telemetry" that still apply to test_draft, reworks the ones that did not fit, and drops the ones test_draft already solves another way. - Enforce a Goose release's tool selection at the research inference gateway: only the selected tools are offered, and calls to other tools are removed from the response. Streams are read line by line as Goose 1.51 reads them; a line the filter cannot judge is dropped and logged, and a turn whose calls were all withheld gets a short notice. - Accept the gateway tool binding only for a release that provably launches Goose (BYOA_CONFIG_UNENFORCEABLE otherwise), and add the tool names Goose 1.51 sends to the catalogue. - Keep tool-call ids in canonical correlations, the AGENT_RESPONSES_MAX_OUTPUT_TOKENS cap and max_output_tokens recording, and harden the relay's tool filter against malformed tool definitions. - Rebuild the local BYOA telemetry seed script on the current contract (gateway bindings, priced model, study budget, host platform), with tests against a real database. - Keep test_draft's approach where it already solves the same problem: the metered gateway instead of the plugin relay, idle-session recovery at bootstrap, tool titles as a study policy, and streamed chunks kept for the e2e gate. - Document the gateway tool enforcement and its rollout order in docs/AGENT_CONFIGURATION_CONTRACT.md.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Updated the landing page with features from the current version of code4me2, and why researchers might want to utilize this product.