diff --git a/CLAUDE.md b/CLAUDE.md index 1c7b22fc..3eec6eb2 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -72,7 +72,7 @@ python dashboard/scripts/backtest_hourly_agent.py # main hourly agent backte - `ADMIN_BOOTSTRAP_SECRET` (optional, **minimum 32 characters**): shared secret for `POST /api/admin/bootstrap`, which promotes the **signed-in** caller to `role=admin`. One-shot: it refuses once any admin account exists (break-glass after that is SQL). A value shorter than `_BOOTSTRAP_MIN_LENGTH` (32) is refused as if unset — this secret grants admin with no account behind it and no lockout to hide behind, so its entropy is the only thing bounding a guesser, and that cannot be left to whoever filled in the Render field. **Unset, too weak, and wrong all answer identically (403)**, so a caller cannot learn whether a deployment is bootstrappable; the operator's signal is a server-log line (same call the repo makes for `LEADERBOARD_DAILY_REFRESH_SECRET`, which 401s either way). Wrong guesses are rate-limited three ways — per user, per client IP (5 / 15 min each) and a server-wide 20 / 15 min ceiling — but the global budget is consulted **after** the compare, so wrong guesses can never refuse a *correct* secret. Do not "tidy" that ordering back: checked first, 20 guesses a window (re-spent every window, from any account, and signup is open) locked the real operator out indefinitely, and the window it blocked is exactly the fresh-deploy window this route exists for. The secret is compared via SHA-256 + `secrets.compare_digest` — the hash is load-bearing because `compare_digest` raises `TypeError` on a non-ASCII `str` (a JSON body can send one) and runs in time proportional to the shorter operand, leaking the expected length. It does *not* raise on a length mismatch; unequal buffers just compare false. Set it in the **Render dashboard** for a fresh deploy (already set there), then leave it (it is inert after the first admin) or unset it. `tests/conftest.py` strips it so the suite sees a known-unset baseline. - `DEFAULT_MAX_CONCURRENT_BACKTESTS` (optional, default **5**, range 0–20): concurrent protocol runs an account gets before an admin edits its row. Nothing seeds a `user_entitlements` row at signup and nothing backfills, so this constant is the **live limit for every account that already exists** — it silently became everyone's quota on the deploy that shipped the entitlement plane. It matches `MAX_ACTIVE_RUNS_PER_AGENT` (5) for that reason: before the plane, an account's concurrency was bounded only by the per-agent cap times however many agents it owned, so a lower default demotes every multi-agent user at once with no remedy short of an admin. Lower it deliberately or not at all. A junk or out-of-range value falls back to 5 with a log line rather than raising at import. - `CREDITS_METERING_ENABLED` (optional) and `DEFAULT_CREDITS` (optional, default **100**, range 0–1,000,000): the **superseded** one-credit-per-run metering policy in `domain/entitlements/credits.py`. It is on no request path. `3eebb7da` *feat: add secure llm worker handoff* (2026-08-24) replaced it with the unified LLM execution layer in the next bullet; what survives is the module, its unit tests, and `GET /api/admin/stats` reporting `credits_metering_enabled` through `admin_users.py:179` — the only non-test import of `domain/entitlements` in the backend. **Arming the flag today moves one admin boolean and nothing else**: `authorize_llm_run` and `refund_llm_run` are called only by `tests/test_credit_metering.py`. `DEFAULT_CREDITS` still seeds `user_entitlements.credits`, an admin-granted allowance no live path spends (the LLM lane bills `credit_llm_usage_entries` instead). ⚠ **Do not read "only tests call it" as "spend is unmetered" and re-wire the debit.** That inference has already been drawn once, from exactly this evidence, and it is wrong — a control that was deleted is not a control that is absent until you have looked for its replacement. The retraction is recorded in `docs/superpowers/plans/2026-09-11-backtest-p1-p2-plan.md`. `tests/conftest.py` strips both. -- **LLM backtest billing — the live path, and it has no env var of its own.** `POST /backtest/run` with `decision_source='llm'` on the pipeline runtime is **sign-in-only** (401) and **requires `billing_mode`** (422), one of `byok` or `platform_credits` (`api/routers/backtests.py:2755-2841`). The route preflights the credential and the provider/model route, then hands the worker a signed, secret-free envelope on **stdin** (`create_execution_handoff` → `consume_execution_handoff`, `dashboard/scripts/backtest_hourly_agent.py:170`) — a raw key never reaches subprocess arguments or the environment. `infrastructure/llm/execution/service.py` then bills **per call, by usage, not per run**: `_execute_platform` reserves a ceiling as a `credit_llm_reservations` row, `_settle` debits the provider-reported tokens into `credit_llm_usage_entries` at a pricing snapshot, and `_release_after_failure` hands the hold back. A reservation covers **one physical provider attempt**: `attempt_index` counts attempts within a call, and same-provider repeats and failover share that counter (see `LLM_PROVIDER_READ_TIMEOUT_SECONDS` above), so select a lane by `provider_id`, never by `attempt_index = 1`. When every candidate fails, the error raised is that of the first candidate whose final error was a timeout/unavailable, not the last candidate's answer — a quota-dead OpenRouter used to relabel a CommonStack timeout as `provider_quota_exhausted`. Not `credit_ledger_entries` — that table's `entry_type` CHECK admits only `purchase` / `refund` / `admin_grant_assign` / `admin_grant_reclaim`, so it can hold neither row. (This bullet named the wrong table until 2026-09-14.) Settlement **fails closed** — a status other than `settled`, or an amount disagreeing with the computed one, raises `BILLING_FAILED` rather than letting the run pass as billed. BYOK never touches the ATL ledger: `finalize_run` is an explicit no-op for that lane, so a BYOK run cannot quietly spend Credits. The **parent** repeats `finalize_run` in its `finally` (`backtests.py:1627-1634`) because the child's own cleanup does not run when the subprocess is killed — by the timeout, or by the cancel route added in #460 — and a killed child would otherwise strand a reservation. Design: `docs/superpowers/plans/2026-08-24-unified-llm-execution-layer.md`. ATL Credits try platform providers in `ATL_PLATFORM_PROVIDER_ORDER` order (below; CommonStack first by default) and fail over on `_PLATFORM_FAILOVER_CATEGORIES`. The first `PROVIDER_QUOTA_EXHAUSTED` a platform provider raises in a process prints `ERROR: llm.platform_quota_exhausted provider= fallback=`, once and flushed, and the parent's `_drain_stream` echoes every child line starting `ERROR: llm.` to its own stdout as it arrives, through `_redact_credentials` like the dump — the timeout path never dumps the child's capture and a long run's middle is elided from it, so without the relay the line would not reach the service log. The end-of-run dump drops those lines (`_without_relayed_lines`) so each reaches the log exactly once; the capture itself keeps them, because it also feeds the failure summary. It exists because CommonStack has no balance endpoint, so a failed call is the only way to see it drain. BYOK never prints it. A run killed by that timeout now finalizes as its own terminal outcome rather than a generic exception: `/backtest/status` answers `timed_out` with a `timeout` block naming the budget, and the analytics event stays `backtest_failed` with `error_category="run_timeout"`. ⚠ **The lane and the settled spend (`credits_service.sum_run_llm_spend`) in that block are owner-only** — `_status_snapshot_for` strips `spent_micro`/`model_calls`/`billing_mode` for any caller who does not satisfy `_slot_owned_by`. The read rule the route authorizes with (`_slot_visible_to`) is deliberately weaker than ownership: for a built-in agent it admits anyone holding the slot's `session_id`, which the *running* branch of this same route publishes to every visitor, so without the strip an anonymous caller could read a signed-in stranger's Credits charge out of `_recent_slots`. Do not widen it back by reusing `_slot_visible_to` for the whole payload — the outcome is public on purpose (every poller branches on `timed_out`) and only the account-level facts are withheld. The spend lookup also runs on a daemon thread joined with `TIMEOUT_SPEND_LOOKUP_SECONDS`: `except Exception` answers a query that raises, not one that never returns, and inline it blocked finalize and stranded a concurrency slot for a dead child. Design: `docs/superpowers/specs/2026-09-14-backtest-timeout-outcome-design.md`. **Sampling is pinned per catalog model** (`domain/model_providers/execution_catalog.py:SamplingPolicy`): Claude gets temperature 0; Gemini 3.1 Pro gets the **provider default** (`PROVIDER_DEFAULT`, sends nothing — Google advises against low temperatures on Gemini 3, so 0 stays unpinned until measured); GPT-5.5 gets reasoning effort `low` and **no temperature** (OpenAI reasoning models reject one); DeepSeek V4 Pro and Qwen3.7 Plus get temperature 0 and **thinking off** (`reasoning_effort: "none"`), which `adapters/openai.py` sends as `thinking: {type: "disabled"}` **to CommonStack only** (`THINKING_TOGGLE_PROVIDERS`, keyed on provider id, not on `openai_compatible` — an admin-registered OpenAI-shaped server was never probed with that field and keeps the `reasoning` shape), because CommonStack honours no graduated reasoning control for those two (issue #539 probe, 2026-10-01). `CatalogModel.sampling` has **no default** — a new catalog row must state one, because a defaulted `temperature 0` on an OpenAI reasoning model 400s every call. ⚠ **The policy is applied in exactly one place: `AnthropicCompatibleExecutionClient`**, which resolves it from the **signed** handoff's `model_id` (`sampling_policy_for`) and imposes it on every call, overriding whatever the caller passed. Nothing upstream threads it — not argv, the engine, the portfolio manager or `pipeline_runner` — so a pipeline step, the truncation-recovery retry, failover and the post-trade review all carry it without a call site being able to forget. Do not reintroduce a threaded copy: an earlier draft of this PR rode unsigned argv flags that could disagree with the signed model id, and a CLI run with no handoff reached the legacy OpenRouter client, which pairs a temperature with an Anthropic `thinking` block that rejects it. A run without a handoff therefore pins nothing and records `provider_default`. Each adapter reports the controls it actually put on the wire (`AdapterResponse.sampling_wire` → `LLMExecutionResult.sampling_wire`, e.g. `temperature=0.0;thinking=disabled`), because one policy takes a different shape per lane and the OpenRouter thinking-off shape is unprobed. The engine records `agent_runs.metadata.llm_sampling` from `execution_client.sampling_record()`: `policy` (`pinned_v1 | provider_default`), the two values, `model`, and `wire` (provider id → controls, per lane that answered). The results panel's **Sampling** row reads *Pinned · …*, *Provider default*, *Entry config · …* (leaderboard rows, derived from their top-level `temperature`/`reasoning_effort` because `_llm_run_metadata` writes no block), or *Not recorded*, and names the lanes when more than one answered. The `policy` decides the prefix, never the presence of values. The panel reads it off `GET /api/backtest/runs*`, whose response model (`RunMetadata`) is an allow-list: `llm_sampling` is promoted from metadata and sanitized by `_sanitized_llm_sampling` (the four scalar keys plus a bounded `wire` map; `None` when nothing survives, since `{}` is truthy in the browser), so a new key in that block must be added there or it never reaches the browser. ⚠ **Pinned is not deterministic** — providers are not, at temperature 0 least of all with a mixture-of-experts model, and each bar's prompt embeds the previous bar's answer — so the claim on screen is the request, and the spread is measured with `dashboard/scripts/diff_backtest_runs.py ` (first divergent bar, divergent bar count, final-equity gap). It refuses a pair that differs in window, model, universe, pipeline, data source, **tape** (`market_data_feed`, `sip_fallback_to_iex`, `end_clamped`), capital, cadence or output ceiling, and warns when a run was answered on more than one lane. It reports a `basis`: a pipeline-runtime backtest writes no `backtest_decisions` rows (the `insert_decisions` guard in `HourlyBacktester` fires for the AI Hedge Fund runtime only), so the decision fields come back `None` rather than `0` and the equity curve carries the number. The before/after measurement runs on the platform_credits path after merge and is recorded in `docs/superpowers/plans/2026-09-20-backtest-pinned-sampling.md`. +- **LLM backtest billing — the live path, and it has no env var of its own.** `POST /backtest/run` with `decision_source='llm'` on the pipeline runtime is **sign-in-only** (401) and **requires `billing_mode`** (422), one of `byok` or `platform_credits` (`api/routers/backtests.py:2755-2841`). The route preflights the credential and the provider/model route, then hands the worker a signed, secret-free envelope on **stdin** (`create_execution_handoff` → `consume_execution_handoff`, `dashboard/scripts/backtest_hourly_agent.py:170`) — a raw key never reaches subprocess arguments or the environment. `infrastructure/llm/execution/service.py` then bills **per call, by usage, not per run**: `_execute_platform` reserves a ceiling as a `credit_llm_reservations` row, `_settle` debits the provider-reported tokens into `credit_llm_usage_entries` at a pricing snapshot, and `_release_after_failure` hands the hold back. A reservation covers **one physical provider attempt**: `attempt_index` counts attempts within a call, and same-provider repeats and failover share that counter (see `LLM_PROVIDER_READ_TIMEOUT_SECONDS` above), so select a lane by `provider_id`, never by `attempt_index = 1`. When every candidate fails, the error raised is that of the first candidate whose final error was a timeout/unavailable, not the last candidate's answer — a quota-dead OpenRouter used to relabel a CommonStack timeout as `provider_quota_exhausted`. Not `credit_ledger_entries` — that table's `entry_type` CHECK admits only `purchase` / `refund` / `admin_grant_assign` / `admin_grant_reclaim`, so it can hold neither row. (This bullet named the wrong table until 2026-09-14.) Settlement **fails closed** — a status other than `settled`, or an amount disagreeing with the computed one, raises `BILLING_FAILED` rather than letting the run pass as billed. BYOK never touches the ATL ledger: `finalize_run` is an explicit no-op for that lane, so a BYOK run cannot quietly spend Credits. **An empty balance has its own path** (2026-10-07): the store's `InsufficientCreditsError` maps to `INSUFFICIENT_CREDITS` ("Not enough ATL Credits…"; analytics `credits_insufficient`), not the catch-all `BILLING_FAILED`, and the child prints `ERROR: llm.run_failed category=` so the parent shows the safe message and records the category instead of a traceback tail and `internal_error`. The route also refuses a platform_credits run up front — **402**, or 403 for a restricted account — when the balance cannot cover `_estimated_platform_credits_micro` (`(calls−1) × typical call + one reservation ceiling`, constants calibrated on prod usage; a refusal, so over-estimating is the unsafe direction). The per-call reservation stays the hard gate; a failed balance read lets the run through. New accounts get the `default_signup_credits_v2` welcome grant (8 Credits); holders of v1 (1.5) do not get v2 on top, because the boot backfill replays the campaign over every account. The **parent** repeats `finalize_run` in its `finally` (`backtests.py:1627-1634`) because the child's own cleanup does not run when the subprocess is killed — by the timeout, or by the cancel route added in #460 — and a killed child would otherwise strand a reservation. Design: `docs/superpowers/plans/2026-08-24-unified-llm-execution-layer.md`. ATL Credits try platform providers in `ATL_PLATFORM_PROVIDER_ORDER` order (below; CommonStack first by default) and fail over on `_PLATFORM_FAILOVER_CATEGORIES`. The first `PROVIDER_QUOTA_EXHAUSTED` a platform provider raises in a process prints `ERROR: llm.platform_quota_exhausted provider= fallback=`, once and flushed, and the parent's `_drain_stream` echoes every child line starting `ERROR: llm.` to its own stdout as it arrives, through `_redact_credentials` like the dump — the timeout path never dumps the child's capture and a long run's middle is elided from it, so without the relay the line would not reach the service log. The end-of-run dump drops those lines (`_without_relayed_lines`) so each reaches the log exactly once; the capture itself keeps them, because it also feeds the failure summary. It exists because CommonStack has no balance endpoint, so a failed call is the only way to see it drain. BYOK never prints it. A run killed by that timeout now finalizes as its own terminal outcome rather than a generic exception: `/backtest/status` answers `timed_out` with a `timeout` block naming the budget, and the analytics event stays `backtest_failed` with `error_category="run_timeout"`. ⚠ **The lane and the settled spend (`credits_service.sum_run_llm_spend`) in that block are owner-only** — `_status_snapshot_for` strips `spent_micro`/`model_calls`/`billing_mode` for any caller who does not satisfy `_slot_owned_by`. The read rule the route authorizes with (`_slot_visible_to`) is deliberately weaker than ownership: for a built-in agent it admits anyone holding the slot's `session_id`, which the *running* branch of this same route publishes to every visitor, so without the strip an anonymous caller could read a signed-in stranger's Credits charge out of `_recent_slots`. Do not widen it back by reusing `_slot_visible_to` for the whole payload — the outcome is public on purpose (every poller branches on `timed_out`) and only the account-level facts are withheld. The spend lookup also runs on a daemon thread joined with `TIMEOUT_SPEND_LOOKUP_SECONDS`: `except Exception` answers a query that raises, not one that never returns, and inline it blocked finalize and stranded a concurrency slot for a dead child. Design: `docs/superpowers/specs/2026-09-14-backtest-timeout-outcome-design.md`. **Sampling is pinned per catalog model** (`domain/model_providers/execution_catalog.py:SamplingPolicy`): Claude gets temperature 0; Gemini 3.1 Pro gets the **provider default** (`PROVIDER_DEFAULT`, sends nothing — Google advises against low temperatures on Gemini 3, so 0 stays unpinned until measured); GPT-5.5 gets reasoning effort `low` and **no temperature** (OpenAI reasoning models reject one); DeepSeek V4 Pro and Qwen3.7 Plus get temperature 0 and **thinking off** (`reasoning_effort: "none"`), which `adapters/openai.py` sends as `thinking: {type: "disabled"}` **to CommonStack only** (`THINKING_TOGGLE_PROVIDERS`, keyed on provider id, not on `openai_compatible` — an admin-registered OpenAI-shaped server was never probed with that field and keeps the `reasoning` shape), because CommonStack honours no graduated reasoning control for those two (issue #539 probe, 2026-10-01). `CatalogModel.sampling` has **no default** — a new catalog row must state one, because a defaulted `temperature 0` on an OpenAI reasoning model 400s every call. ⚠ **The policy is applied in exactly one place: `AnthropicCompatibleExecutionClient`**, which resolves it from the **signed** handoff's `model_id` (`sampling_policy_for`) and imposes it on every call, overriding whatever the caller passed. Nothing upstream threads it — not argv, the engine, the portfolio manager or `pipeline_runner` — so a pipeline step, the truncation-recovery retry, failover and the post-trade review all carry it without a call site being able to forget. Do not reintroduce a threaded copy: an earlier draft of this PR rode unsigned argv flags that could disagree with the signed model id, and a CLI run with no handoff reached the legacy OpenRouter client, which pairs a temperature with an Anthropic `thinking` block that rejects it. A run without a handoff therefore pins nothing and records `provider_default`. Each adapter reports the controls it actually put on the wire (`AdapterResponse.sampling_wire` → `LLMExecutionResult.sampling_wire`, e.g. `temperature=0.0;thinking=disabled`), because one policy takes a different shape per lane and the OpenRouter thinking-off shape is unprobed. The engine records `agent_runs.metadata.llm_sampling` from `execution_client.sampling_record()`: `policy` (`pinned_v1 | provider_default`), the two values, `model`, and `wire` (provider id → controls, per lane that answered). The results panel's **Sampling** row reads *Pinned · …*, *Provider default*, *Entry config · …* (leaderboard rows, derived from their top-level `temperature`/`reasoning_effort` because `_llm_run_metadata` writes no block), or *Not recorded*, and names the lanes when more than one answered. The `policy` decides the prefix, never the presence of values. The panel reads it off `GET /api/backtest/runs*`, whose response model (`RunMetadata`) is an allow-list: `llm_sampling` is promoted from metadata and sanitized by `_sanitized_llm_sampling` (the four scalar keys plus a bounded `wire` map; `None` when nothing survives, since `{}` is truthy in the browser), so a new key in that block must be added there or it never reaches the browser. ⚠ **Pinned is not deterministic** — providers are not, at temperature 0 least of all with a mixture-of-experts model, and each bar's prompt embeds the previous bar's answer — so the claim on screen is the request, and the spread is measured with `dashboard/scripts/diff_backtest_runs.py ` (first divergent bar, divergent bar count, final-equity gap). It refuses a pair that differs in window, model, universe, pipeline, data source, **tape** (`market_data_feed`, `sip_fallback_to_iex`, `end_clamped`), capital, cadence or output ceiling, and warns when a run was answered on more than one lane. It reports a `basis`: a pipeline-runtime backtest writes no `backtest_decisions` rows (the `insert_decisions` guard in `HourlyBacktester` fires for the AI Hedge Fund runtime only), so the decision fields come back `None` rather than `0` and the equity curve carries the number. The before/after measurement runs on the platform_credits path after merge and is recorded in `docs/superpowers/plans/2026-09-20-backtest-pinned-sampling.md`. - `ATL_PLATFORM_PROVIDER_ORDER` (optional, default **`commonstack,openrouter`**): the order in which ATL Credits (`billing_mode=platform_credits`) try platform providers. `platform_provider_order()` in `domain/model_providers/service.py` reads it per call, and both `resolve_platform_execution_candidates` and `list_execution_options` use it. - **Why CommonStack is first:** its prepaid balance should be spent before OpenRouter's (#522). The order was hard-coded OpenRouter-first until 2026-09, and CommonStack carried all the traffic only because OpenRouter's key was quota-dead (#523). Topping that key up would have silently stranded CommonStack's balance. - **Leaving a provider out** removes it from routing entirely. That is the no-deploy way to pull a drained lane. An explicit `provider_id` in the request body is honoured first **only when it is in the order** — otherwise an API caller naming the pulled lane would reopen it (the dashboard never sends one for ATL Credits). An admin-added platform lane is therefore reachable only once it is listed here. If nothing routable is left, the route answers 422 rather than guessing. diff --git a/dashboard/backend/api/routers/backtests.py b/dashboard/backend/api/routers/backtests.py index 9fdd4a9c..8ffef19f 100644 --- a/dashboard/backend/api/routers/backtests.py +++ b/dashboard/backend/api/routers/backtests.py @@ -17,6 +17,7 @@ import os import re import signal +import statistics import subprocess import time import uuid @@ -44,7 +45,7 @@ from typing import Any, Dict, List, Literal, NamedTuple, Optional, Tuple import pytz -from datetime import datetime +from datetime import datetime, timedelta from fastapi import APIRouter, HTTPException, Query, Request from fastapi.responses import Response from pydantic import BaseModel @@ -67,8 +68,12 @@ MarketDataSourceDisabled, UnsupportedMarketDataSource, ensure_market_data_source_available, + settled_exclusive_end, validate_market_data_source, ) +from dashboard.backend.infrastructure.market_data.us_market_calendar import ( + is_trading_day as is_nyse_trading_day, +) from dashboard.backend.infrastructure.market_data.sessions import time_in_session from dashboard.backend.infrastructure.market_data.profiles import ( LLM_DECISION_SOURCE, @@ -79,16 +84,41 @@ from dashboard.backend.infrastructure.market_data.ifind_ashare import ( ASHARE_SESSIONS_PER_TRADING_DAY, ) -from dashboard.backend.infrastructure.llm.execution.errors import LLMExecutionError -from dashboard.backend.infrastructure.llm.execution.service import LLMExecutionService +from dashboard.backend.infrastructure.llm.execution.errors import ( + ExecutionErrorCategory, + LLMExecutionError, +) +from dashboard.backend.infrastructure.llm.execution.service import ( + LLMExecutionService, + analytics_error_category, + reservation_ceiling_micro_for, +) from dashboard.backend.infrastructure.llm.execution.handoff import ( create_execution_handoff, ) from dashboard.backend.infrastructure.llm.execution.models import ( BillingMode, LLMRunEvidence, + LLMUsage, + PricingSnapshot, +) +from dashboard.backend.infrastructure.llm.pipeline_runner import ( + DEFAULT_CEILING_SNAPSHOT_SYMBOLS, + RECOVERY_MAX_OUTPUT_TOKENS, + split_pipeline, +) +from dashboard.backend.infrastructure.llm.backtest_harness import ( + DEFAULT_MAX_OUTPUT_TOKENS, +) +from dashboard.backend.infrastructure.llm.token_cost import ( + CHARS_PER_TOKEN, + credits_micro_for_usd, + estimate_cost_from_snapshot, +) +from dashboard.backend.domain.backtesting.portfolio_manager import ( + SNAPSHOT_FULL_UNIVERSE_MAX, + SNAPSHOT_SHORTLIST_SIZE, ) -from dashboard.backend.infrastructure.llm.pipeline_runner import split_pipeline from dashboard.backend.domain.model_providers.execution_catalog import ( UnsupportedExecutionModel, ) @@ -831,7 +861,9 @@ def _mirror_slot_to_legacy(slot: Dict[str, Any]) -> None: """Keep ``backtest_status`` / ``backtest_session_id`` in sync with one slot.""" global backtest_session_id backtest_status["running"] = bool(slot.get("running")) - backtest_status["error"] = slot.get("error") + # The mirror answers whoever polls with the slot's ``session_id`` -- for a + # built-in agent a published id -- so it carries the redacted form. + backtest_status["error"] = slot.get("public_error") or slot.get("error") backtest_status["runs_count"] = int(slot.get("runs_count") or 0) backtest_status["started_at"] = slot.get("started_at") backtest_status["progress_file"] = slot.get("progress_file") @@ -853,9 +885,17 @@ def _try_acquire_backtest_slot( session_id: str, owner_session: Optional[str] = None, user_id: Optional[int], + credits_claim: Optional["_CreditsClaim"] = None, ) -> Optional[str]: """Register a running slot or return a human-readable refusal reason. + ``credits_claim`` is an ATL Credits launch's admitted estimate. It is + re-checked here, under the same lock that registers the slot, against the + owner's other running platform-credits runs -- so two launches that both + passed ``_enforce_platform_credits_balance`` on one balance cannot both + start -- and recorded on the slot for the next launch to count. A refusal + on that ground is a ``_CreditsRefusal``, which the route answers with 402. + ``session_id`` files the *results* (for a built-in agent that is the agent's own session, so its runs land on its public card). ``owner_session`` is the caller's browser session and is what the cap is @@ -889,6 +929,12 @@ def _try_acquire_backtest_slot( f"You already have {max_for_owner} backtests running. " "Please wait for one to finish." ) + if credits_claim is not None and user_id is not None: + shortfall = _credits_shortfall( + credits_claim, _owner_platform_credit_slots(int(user_id)) + ) + if shortfall is not None: + return _CreditsRefusal(shortfall) slot = { "live_run_id": live_run_id, "session_id": session_id, @@ -921,6 +967,12 @@ def _try_acquire_backtest_slot( # did not time out. "timed_out": False, "timeout_detail": None, + # What this run was admitted as needing, for the credits check of + # the owner's next launch. None for every run that is not ATL + # Credits, which that check therefore never counts. + "credits_estimate_micro": ( + int(credits_claim.needed_micro) if credits_claim is not None else None + ), } _active_slots[live_run_id] = slot _mirror_slot_to_legacy(slot) @@ -994,6 +1046,8 @@ def _finalize_slot( cancelled: bool = False, timed_out: bool = False, timeout_detail: Optional[Dict[str, Any]] = None, + error_category: Optional[str] = None, + public_error: Optional[str] = None, ) -> None: """Move a slot to its terminal state. @@ -1029,6 +1083,8 @@ def _finalize_slot( cancelled=cancelled, timed_out=timed_out, timeout_detail=timeout_detail, + error_category=error_category, + public_error=public_error, ) _emit_slot_run_event(event) @@ -1041,6 +1097,8 @@ def _finalize_slot_locked( cancelled: bool = False, timed_out: bool = False, timeout_detail: Optional[Dict[str, Any]] = None, + error_category: Optional[str] = None, + public_error: Optional[str] = None, ) -> Optional[Dict[str, Any]]: """The terminal transition itself. THE CALLER MUST HOLD THE LEDGER LOCK. @@ -1087,11 +1145,14 @@ def _finalize_slot_locked( backtest_status["live_run_id"] = None backtest_status["progress_file"] = None if error is not None: - backtest_status["error"] = error + backtest_status["error"] = public_error or error backtest_status["runs_count"] = runs_count return None slot["running"] = False slot["error"] = error + # What a caller who did not start the run reads in place of ``error`` + # (``_status_snapshot_for``); None when ``error`` holds nothing personal. + slot["public_error"] = public_error slot["runs_count"] = runs_count # Captured before ``started_at`` is cleared, because it is the only record # of how long the run went that survives into the terminal state. The @@ -1147,7 +1208,9 @@ def _finalize_slot_locked( "event_name": "backtest_completed" if succeeded else "backtest_failed", "user_id": int(user_id), "run_id": live_run_id, - "error_category": None if succeeded else "internal_error", + "error_category": ( + None if succeeded else (error_category or "internal_error") + ), } @@ -1509,8 +1572,14 @@ def _status_snapshot_for( applied to one of them is not a redaction. """ snapshot = _slot_snapshot(slot) + owned = _slot_owned_by(slot, session_id=session_id, user_id=user_id) + if not owned and slot.get("public_error"): + # A model-execution failure names the owner's Credits balance, their + # restriction, their billing lane -- the same class of account fact + # ``timeout_detail`` withholds below. Others learn only that it stopped. + snapshot["error"] = slot["public_error"] detail = snapshot.get("timeout_detail") - if detail and not _slot_owned_by(slot, session_id=session_id, user_id=user_id): + if detail and not owned: snapshot["timeout_detail"] = { key: value for key, value in detail.items() @@ -2021,6 +2090,8 @@ def run_backtest_background( raise _BacktestCancelled() slot_error = None + slot_public_error = None + slot_error_category = None slot_runs_count = 0 if result.returncode != 0: error_msg = result.stderr if result.stderr else result.stdout @@ -2032,6 +2103,15 @@ def run_backtest_background( slot_error = ( f"Backtest failed with return code {result.returncode}. {summary}" ) + llm_failure = _child_llm_failure(result.stdout, result.stderr) + if llm_failure is not None: + # The safe message is the whole story for the user; the + # traceback stays in the service log via the dump above. + slot_error = f"Backtest stopped: {llm_failure.safe_message}" + slot_public_error = _LLM_FAILURE_PUBLIC_ERROR + slot_error_category = analytics_error_category( + llm_failure.category + ) print(f"❌ Backtest failed (returncode={result.returncode})", flush=True) # A negative return code is a signal death, and the parent's own # signals were claimed above (cancel) or raise TimeoutExpired @@ -2050,7 +2130,11 @@ def run_backtest_background( ) if resolved_live_run_id: _finalize_slot( - resolved_live_run_id, error=slot_error, runs_count=slot_runs_count + resolved_live_run_id, + error=slot_error, + runs_count=slot_runs_count, + error_category=slot_error_category, + public_error=slot_public_error, ) resolved_live_run_id = None # finally must not double-finalize except _BacktestCancelled: @@ -2638,6 +2722,355 @@ def _max_pipeline_llm_calls() -> int: return usable // max(1, PIPELINE_SECONDS_PER_LLM_CALL) +# --- ATL Credits launch preflight ------------------------------------------ +# +# Every platform-credits call reserves its worst case before it runs and settles +# to actual usage after, one call at a time. A run therefore dies at the first +# call whose reservation the remaining balance cannot cover -- possibly forty +# bars in, after most of the balance is spent (2026-10-07: a 1.5-Credit account +# spent 1.32 Credits on 39 GPT-5.5 DJIA 30 calls, then failed the run). This +# refuses that run up front instead. +# +# The estimate is ``sum(expected per call) - expected(largest) + ceiling +# (largest)``: the last call must still find a whole reservation free. It is a +# refusal, so over-estimating is the unsafe direction -- it turns away a run +# that would have finished. Every input is therefore typical, not worst case, +# and the reservation itself stays the hard gate: +# +# - calls: NYSE sessions (US) up to the settled end -- the engine never trades +# a session still in progress (``settled_exclusive_end``) -- not the weekday +# upper bound the subprocess timeout over-provisions from. +# - output tokens: the model's own median from prod platform-credits usage +# (30 days to 2026-10-07). +# - prompt: the step-0 market snapshot is about 3000 bytes of fixed framing plus +# about 254 bytes per symbol shown, and about 57 more per symbol for a +# half-invested book's holdings and trades (measured by rebuilding the +# snapshot; on GPT-5.5 / DJIA 30 the ceiling it gives, 0.186 Credits, matches +# the 0.178-0.187 prod reservations). On top of that, the caller's own text: +# each step's instruction and output format (or the single-prompt path's +# strategy prompt), and for step k > 0 the k upstream outputs it carries. +# The calibration ran a short starter instruction, so counting the whole +# instruction again over-states by those few hundred bytes -- noise beside +# the output tokens, which dominate the price. +_PREFLIGHT_PROMPT_FIXED_BYTES = 3000 +_PREFLIGHT_PROMPT_BYTES_PER_SYMBOL = 311 +_PREFLIGHT_EXPECTED_OUTPUT_TOKENS_BY_MODEL = { + "anthropic/claude-sonnet-4-6": 420, + "openai/gpt-5.5": 574, + "deepseek/deepseek-v4-pro": 1058, + "qwen/qwen3.7-plus": 1335, +} +# A model with no prod sample yet (Haiku 4.5, Gemini 3.1 Pro): the middle of +# the measured ones, rather than any single model's figure. +_PREFLIGHT_UNMEASURED_OUTPUT_TOKENS = int( + statistics.median(_PREFLIGHT_EXPECTED_OUTPUT_TOKENS_BY_MODEL.values()) +) + + +def _preflight_expected_output_tokens(model_id: str) -> int: + return _PREFLIGHT_EXPECTED_OUTPUT_TOKENS_BY_MODEL.get( + model_id, _PREFLIGHT_UNMEASURED_OUTPUT_TOKENS + ) + + +def _snapshot_symbols_shown( + universe_size: int, pipeline: Optional[List[Dict[str, Any]]] +) -> int: + """How many symbols the step-0 prompt lists (portfolio_manager's rule).""" + universe_size = max(0, int(universe_size)) + if pipeline and universe_size <= SNAPSHOT_FULL_UNIVERSE_MAX: + return universe_size + return min(universe_size, SNAPSHOT_SHORTLIST_SIZE) + + +def _estimated_trading_sessions( + start_date: str, end_date: str, *, market: object = None +) -> int: + """Sessions an inclusive window will actually trade, for pricing it. + + Unlike ``_estimated_decision_days`` (a weekday upper bound, right for a + timeout) this is a typical count, right for a refusal: it stops at the + settled end the engine fetches to and, on the US market, skips NYSE + holidays. A-share has no exchange calendar in the codebase, so CN stays a + weekday count. + """ + try: + start = datetime.strptime(str(start_date)[:10], "%Y-%m-%d").date() + end = datetime.strptime(str(end_date)[:10], "%Y-%m-%d").date() + settled_last = datetime.strptime( + settled_exclusive_end(str(end_date)[:10], market=market), "%Y-%m-%d" + ).date() - timedelta(days=1) + except (TypeError, ValueError): + return 0 + end = min(end, settled_last) + if end < start: + return 0 + is_session = ( + is_nyse_trading_day + if str(market or "US").upper() == "US" + else (lambda day: day.weekday() < 5) + ) + return sum( + 1 + for offset in range((end - start).days + 1) + if is_session(start + timedelta(days=offset)) + ) + + +def _utf8_len(text: Any) -> int: + return len(str(text or "").encode("utf-8")) + + +class _PlatformCreditsEstimate(NamedTuple): + calls: int + needed_micro: int + + +def _estimated_platform_credits( + *, + model_id: str, + provider_id: str, + sessions: int, + data_source: Optional[str], + universe_size: int, + pipeline: Optional[List[Dict[str, Any]]], + strategy_prompt: Optional[str] = None, +) -> Optional[_PlatformCreditsEstimate]: + """What a platform-credits run needs to finish, or None. + + None means "cannot price it" (no calls, or no cost from the snapshot), and + the caller lets the run through: the per-call reservation is still there. + """ + if sessions <= 0: + return None + bars = sessions * _pipeline_decision_bars_per_trading_day(data_source) + shown = _snapshot_symbols_shown(universe_size, pipeline) + snapshot_bytes = _PREFLIGHT_PROMPT_FIXED_BYTES + _PREFLIGHT_PROMPT_BYTES_PER_SYMBOL * shown + expected_output = _preflight_expected_output_tokens(model_id) + # Mirrors pipeline_runner._first_attempt_ceiling: a wide snapshot asks for + # the recovery ceiling on its first attempt. + ceiling_output = ( + RECOVERY_MAX_OUTPUT_TOKENS + if shown > DEFAULT_CEILING_SNAPSHOT_SYMBOLS + else DEFAULT_MAX_OUTPUT_TOKENS + ) + prior_output_bytes = min(expected_output, ceiling_output) * CHARS_PER_TOKEN + + # (prompt bytes, how many calls of that shape) per call shape in the run. + shapes: List[Tuple[int, int]] = [] + decision_steps, post_trade_steps = split_pipeline(pipeline) + if decision_steps: + for index, step in enumerate(decision_steps): + step_bytes = ( + (snapshot_bytes if index == 0 else _PREFLIGHT_PROMPT_FIXED_BYTES) + + _utf8_len(step.get("prompt")) + + _utf8_len(step.get("outputFormat")) + + index * prior_output_bytes + ) + shapes.append((step_bytes, bars)) + else: + shapes.append((snapshot_bytes + _utf8_len(strategy_prompt), bars)) + for step in post_trade_steps: + shapes.append( + ( + _PREFLIGHT_PROMPT_FIXED_BYTES + + _utf8_len(step.get("prompt")) + + _utf8_len(step.get("outputFormat")), + sessions, + ) + ) + + pricing = PricingSnapshot.from_model(model_id, provider_id) + calls = 0 + expected_total = 0 + largest_expected = 0 + largest_ceiling = 0 + for prompt_bytes, count in shapes: + ceiling = reservation_ceiling_micro_for( + prompt_bytes=prompt_bytes, + max_output_tokens=ceiling_output, + pricing_snapshot=pricing, + ) + expected_usd = estimate_cost_from_snapshot( + pricing, + LLMUsage( + input_tokens=math.ceil(prompt_bytes / CHARS_PER_TOKEN), + output_tokens=min(expected_output, ceiling_output), + ), + ) + if ceiling is None or expected_usd is None: + return None + expected = credits_micro_for_usd(expected_usd) + calls += count + expected_total += count * expected + if ceiling > largest_ceiling: + largest_ceiling, largest_expected = ceiling, expected + if calls <= 0: + return None + return _PlatformCreditsEstimate( + calls=calls, + needed_micro=expected_total - largest_expected + largest_ceiling, + ) + + +def _format_credits(micro: int, *, round_up: bool = False) -> str: + """Two decimals, rounded in the direction that keeps a refusal legible. + + A need is rounded up and a balance down, so a 402 can never print the same + figure on both sides of "needs about X ... your balance is X". + """ + cents = max(0, int(micro)) / 10_000 + cents = math.ceil(cents) if round_up else math.floor(cents) + return f"{cents / 100:.2f}" + + +class _CreditsRefusal(str): + """A slot refusal that the route answers with 402, not a soft error body.""" + + +class _CreditsClaim(NamedTuple): + """An admitted platform-credits launch, re-checked under the ledger lock.""" + + needed_micro: int + available_micro: int + calls: int + model_id: str + # Settled spend of the owner's in-flight platform-credits runs, read + # outside the lock (it is a store call). A run missing here started after + # the read and is counted as having spent nothing yet. + spent_by_run: Dict[str, int] + + +def _owner_platform_credit_slots(user_id: int) -> List[Tuple[str, int]]: + """(run id, admitted estimate) of the owner's running platform-credits runs. + + THE CALLER MUST HOLD THE LEDGER LOCK, or accept a snapshot. + """ + return [ + (str(slot.get("live_run_id")), int(slot["credits_estimate_micro"])) + for slot in _active_slots.values() + if slot.get("running") + and slot.get("credits_estimate_micro") is not None + and slot.get("user_id") is not None + and int(slot["user_id"]) == int(user_id) + ] + + +def _credits_committed_micro( + in_flight: List[Tuple[str, int]], spent_by_run: Dict[str, int] +) -> int: + """What the owner's running runs are still expected to spend. + + Each one's admitted estimate less what it has already settled -- the + settled part is already out of ``total_available_micro``. A run that holds + a reservation right now is counted once in the balance and once here, so + this over-states by at most one call's ceiling per running run. + """ + return sum( + max(0, estimate - int(spent_by_run.get(run_id, 0))) + for run_id, estimate in in_flight + ) + + +def _credits_shortfall( + claim: _CreditsClaim, in_flight: List[Tuple[str, int]] +) -> Optional[str]: + committed = _credits_committed_micro(in_flight, claim.spent_by_run) + if claim.available_micro - committed >= claim.needed_micro: + return None + message = ( + f"This run needs about {_format_credits(claim.needed_micro, round_up=True)} " + f"ATL Credits ({claim.calls} model calls on {claim.model_id}), and your " + f"balance is {_format_credits(claim.available_micro)} Credits" + ) + if committed > 0: + running = len(in_flight) + message += ( + f", about {_format_credits(committed, round_up=True)} of which your " + f"{running} running backtest{'s' if running != 1 else ''} " + f"{'are' if running != 1 else 'is'} expected to use" + ) + return message + ( + ". Add Credits on the Credits page, shorten the date range, choose a " + "lower-cost model, or use your own API key." + ) + + +def _enforce_platform_credits_balance( + *, + user_id: int, + model_id: str, + provider_id: str, + start_date: str, + end_date: str, + pipeline: Optional[List[Dict[str, Any]]], + data_source: Optional[str], + universe_size: int, + market: object = None, + strategy_prompt: Optional[str] = None, +) -> Optional[_CreditsClaim]: + """Refuse an ATL Credits run the account cannot pay to finish. + + 402 for a short balance -- counting what the owner's other running + platform-credits runs are still expected to spend, since they draw on the + same balance one reservation at a time and would otherwise each pass this + check alone and then starve together. 403 for a restricted account (no + reservation can succeed there at all). + + Returns the claim the route hands to ``_try_acquire_backtest_slot``, which + repeats the comparison under the ledger lock so two launches racing past + this read cannot both be admitted against one balance. None when the run + cannot be priced or the balance cannot be read: this check is a courtesy in + front of the reservation, which still refuses every call the balance cannot + cover, and a broken courtesy must not take the whole lane down. + """ + estimate = _estimated_platform_credits( + model_id=model_id, + provider_id=provider_id, + sessions=_estimated_trading_sessions(start_date, end_date, market=market), + data_source=data_source, + universe_size=universe_size, + pipeline=pipeline, + strategy_prompt=strategy_prompt, + ) + if estimate is None: + return None + with _backtest_slots_lock: + in_flight = _owner_platform_credit_slots(user_id) + try: + balance = credits_service.get_balance(int(user_id)) + spent_by_run = { + run_id: int(credits_service.sum_run_llm_spend(int(user_id), run_id)[0]) + for run_id, _estimate in in_flight + } + except Exception as exc: # noqa: BLE001 - the reservation remains the gate + print( + "WARNING: credits.preflight_balance_unavailable " + f"category={type(exc).__name__}", + flush=True, + ) + return None + if balance.account_status == "restricted": + raise HTTPException( + status_code=403, + detail=LLMExecutionError.account_restricted( + balance.restriction_reason, + int(balance.outstanding_credits_micro or 0), + ).safe_message, + ) + claim = _CreditsClaim( + needed_micro=estimate.needed_micro, + available_micro=int(balance.total_available_micro), + calls=estimate.calls, + model_id=model_id, + spent_by_run=spent_by_run, + ) + shortfall = _credits_shortfall(claim, in_flight) + if shortfall is not None: + raise HTTPException(status_code=402, detail=shortfall) + return claim + + def _enforce_pipeline_llm_window( runtime_type: str, decision_source: str, @@ -2785,6 +3218,43 @@ def text(self) -> str: _RELAYED_CHILD_LINE_PREFIX = "ERROR: llm." +# What a non-owner reads for a run a model-execution failure stopped: its +# safe message names account facts (an empty balance, a refund review). +_LLM_FAILURE_PUBLIC_ERROR = "Backtest stopped: the model run could not continue." +_RUN_FAILED_LINE = re.compile( + r"^ERROR: llm\.run_failed category=([a-z_]+)(?: reason=([a-z_]+))?\s*$", re.M +) + + +def _child_llm_failure(*streams: Optional[str]) -> Optional[LLMExecutionError]: + """The model-execution failure the child reported, if it reported one. + + The child prints ``ERROR: llm.run_failed category=`` just before an + ``LLMExecutionError`` ends its run. Without it the parent has only a + traceback tail, so an empty Credits balance reached the user as "Backtest + failed with return code 1. ...Traceback..." and analytics as + ``internal_error``. An unknown value is ignored rather than trusted: the + category picks the user-facing sentence, so only the enum may choose it. + """ + reported = None + for text in streams: + for category_value, reason in _RUN_FAILED_LINE.findall(text or ""): + try: + reported = (ExecutionErrorCategory(category_value), reason) + except ValueError: + # Skipped, not fatal: a stray line must not hide a valid one + # later in this stream or in the other. + continue + if reported is not None: + category, reason = reported + if category is ExecutionErrorCategory.ACCOUNT_RESTRICTED: + # The reason picks the remedy ("add Credits" vs "contact an + # administrator"); account_restricted() ignores an unknown one. + # The outstanding amount does not cross the boundary, so an + # overage reads "add Credits" without the figure. + return LLMExecutionError.account_restricted(reason or None) + return LLMExecutionError(category) + return None def _without_relayed_lines(text: str) -> str: @@ -3725,6 +4195,7 @@ def run_backtest_endpoint( live_run_id = f"agent_{datetime.now().strftime('%Y%m%d_%H%M%S')}_{uuid.uuid4().hex[:8]}" execution_handoff_payload: Optional[str] = None + credits_claim: Optional[_CreditsClaim] = None if ( resolved_decision_source == LLM_DECISION_SOURCE and runtime_type == PIPELINE_RUNTIME_TYPE @@ -3804,6 +4275,23 @@ def run_backtest_endpoint( detail=LLMExecutionError.safe("provider_unavailable").safe_message, ) from exc + if billing_mode is BillingMode.PLATFORM_CREDITS: + # After the route resolves (so the price is the model the run will + # bill) and before a slot or handoff exists, so a refusal holds + # nothing. + credits_claim = _enforce_platform_credits_balance( + user_id=int(user_id), + model_id=route.catalog_id, + provider_id=provider_id, + start_date=start_date, + end_date=end_date, + pipeline=pipeline, + data_source=data_source, + universe_size=len(selected_assets or ()) or len(profile.symbols), + market=profile.market, + strategy_prompt=strategy_prompt, + ) + execution_handoff_payload = create_execution_handoff( user_id=int(user_id), run_id=live_run_id, @@ -3830,7 +4318,10 @@ def run_backtest_endpoint( # file under a built-in agent's session — see _backtest_owner_key. owner_session=request.state.session_id, user_id=user_id, + credits_claim=credits_claim, ) + if isinstance(refusal, _CreditsRefusal): + raise HTTPException(status_code=402, detail=str(refusal)) if refusal: print(f"⚠️ Backtest refused: {refusal}", flush=True) return { diff --git a/dashboard/backend/domain/analytics/models.py b/dashboard/backend/domain/analytics/models.py index 6efa9c52..85bdb98e 100644 --- a/dashboard/backend/domain/analytics/models.py +++ b/dashboard/backend/domain/analytics/models.py @@ -71,6 +71,13 @@ # field carries the reason. `provider_timeout` is a different fact with a # different remedy. "run_timeout", + # The account cannot take model usage at all (restricted) or cannot cover + # the next call's reservation. Both were being emitted already -- + # `account_restricted` since the restricted lane shipped -- but neither + # was admitted here, so `emit_safe_error_event` raised inside its own + # guard and the event was dropped with only a WARNING line. + "account_restricted", + "credits_insufficient", } EVENT_GROUP_BY_NAME = { diff --git a/dashboard/backend/domain/analytics/repository.py b/dashboard/backend/domain/analytics/repository.py index 813b2382..2aaf6b21 100644 --- a/dashboard/backend/domain/analytics/repository.py +++ b/dashboard/backend/domain/analytics/repository.py @@ -69,7 +69,8 @@ 'credential_invalid', 'credential_missing', 'provider_timeout', 'provider_unavailable', 'provider_quota_exhausted', 'credits_unavailable', - 'model_not_allowed', 'internal_error', 'run_timeout' + 'model_not_allowed', 'internal_error', 'run_timeout', + 'account_restricted', 'credits_insufficient' ) ), country_code TEXT CHECK (country_code IS NULL OR length(country_code) = 2), diff --git a/dashboard/backend/domain/analytics/repository_postgres.py b/dashboard/backend/domain/analytics/repository_postgres.py index 4373a6f0..5a174963 100644 --- a/dashboard/backend/domain/analytics/repository_postgres.py +++ b/dashboard/backend/domain/analytics/repository_postgres.py @@ -60,7 +60,8 @@ 'credential_invalid', 'credential_missing', 'provider_timeout', 'provider_unavailable', 'provider_quota_exhausted', 'credits_unavailable', - 'model_not_allowed', 'internal_error', 'run_timeout' + 'model_not_allowed', 'internal_error', 'run_timeout', + 'account_restricted', 'credits_insufficient' ) ), country_code TEXT CHECK (country_code IS NULL OR length(country_code) = 2), @@ -296,7 +297,8 @@ def _init_schema(self) -> None: 'provider_timeout', 'provider_unavailable', 'provider_quota_exhausted', 'credits_unavailable', 'model_not_allowed', 'internal_error', - 'run_timeout' + 'run_timeout', 'account_restricted', + 'credits_insufficient' ) ) """ diff --git a/dashboard/backend/domain/analytics/states.py b/dashboard/backend/domain/analytics/states.py index ad9ea0c8..200d4278 100644 --- a/dashboard/backend/domain/analytics/states.py +++ b/dashboard/backend/domain/analytics/states.py @@ -408,6 +408,13 @@ def calculate_user_state( "credential_missing", "provider_unavailable", "credits_unavailable", + # Split out of credits_unavailable; an empty balance still blocks the + # platform lane, so it keeps the operational state it always had. + "credits_insufficient", + # A restricted account (llm_overage, refund review) can run nothing on + # the platform lane -- strictly worse than an empty balance, so it + # must not read healthier than one. + "account_restricted", "model_not_allowed", } resolvers = { diff --git a/dashboard/backend/domain/credits/repository.py b/dashboard/backend/domain/credits/repository.py index 81c28fb1..37424e9a 100644 --- a/dashboard/backend/domain/credits/repository.py +++ b/dashboard/backend/domain/credits/repository.py @@ -1041,6 +1041,20 @@ def get_balance_projections( def get_balance_micro(self, user_id: int) -> int: return self.get_balance_projection(user_id)["total_available_micro"] + def has_promotion_grant(self, user_id: int, campaign_keys: Sequence[str]) -> bool: + """Whether the account already holds a grant from any of these campaigns.""" + keys = [str(key) for key in campaign_keys] + if not keys: + return False + placeholders = ", ".join("?" for _ in keys) + with self._get_connection() as conn: + row = conn.execute( + "SELECT 1 FROM credit_promotion_grants " + f"WHERE user_id = ? AND campaign_key IN ({placeholders}) LIMIT 1", + (int(user_id), *keys), + ).fetchone() + return row is not None + def list_user_ids(self) -> list[int]: """Return account IDs for an idempotent promotion backfill.""" with self._get_connection() as conn: diff --git a/dashboard/backend/domain/credits/repository_postgres.py b/dashboard/backend/domain/credits/repository_postgres.py index 6eec3d2b..a9154cd7 100644 --- a/dashboard/backend/domain/credits/repository_postgres.py +++ b/dashboard/backend/domain/credits/repository_postgres.py @@ -895,6 +895,21 @@ def get_balance_projections( def get_balance_micro(self, user_id: int) -> int: return self.get_balance_projection(user_id)["total_available_micro"] + def has_promotion_grant(self, user_id: int, campaign_keys: Sequence[str]) -> bool: + """Whether the account already holds a grant from any of these campaigns.""" + keys = [str(key) for key in campaign_keys] + if not keys: + return False + with self._get_connection() as conn: + with conn.cursor() as cur: + cur.execute( + "SELECT 1 FROM credit_promotion_grants " + "WHERE user_id = %s AND campaign_key = ANY(%s) LIMIT 1", + (int(user_id), keys), + ) + row = cur.fetchone() + return row is not None + def list_user_ids(self) -> list[int]: """Return account IDs for an idempotent promotion backfill.""" with self._get_connection() as conn: diff --git a/dashboard/backend/domain/credits/service.py b/dashboard/backend/domain/credits/service.py index 2ba135e2..8e8ca436 100644 --- a/dashboard/backend/domain/credits/service.py +++ b/dashboard/backend/domain/credits/service.py @@ -42,8 +42,18 @@ from dashboard.backend.domain.analytics import instrumentation as analytics_instrumentation -DEFAULT_SIGNUP_CREDIT_CAMPAIGN = "default_signup_credits_v1" -DEFAULT_SIGNUP_CREDITS_MICRO = 1_500_000 +# v2 (8 Credits) replaced v1 (1.5 Credits) on 2026-10-07: 1.5 could not pay for +# one default GPT-5.5 DJIA 30 backtest (~1.8 Credits), so a new account's first +# run on the default agent died partway. A new key rather than a new amount +# under the old one: the grant is idempotent per (campaign, user) and the +# amount is part of its digest, so changing it in place would raise an +# idempotency conflict for every existing account at each login and boot. +DEFAULT_SIGNUP_CREDIT_CAMPAIGN = "default_signup_credits_v2" +DEFAULT_SIGNUP_CREDITS_MICRO = 8_000_000 +# Welcome campaigns this one supersedes. An account holding any of them has had +# its welcome grant and does not get v2 on top -- without this the boot +# backfill would credit every existing account another 8 Credits. +SUPERSEDED_SIGNUP_CREDIT_CAMPAIGNS = ("default_signup_credits_v1",) DEFAULT_SIGNUP_CREDIT_SOURCE = "system_promotion" DEFAULT_SIGNUP_CREDIT_REASON = "Automatic welcome Credits." @@ -159,6 +169,10 @@ def get_balance(self, user_id: int) -> BalanceResult: def grant_default_signup_credits(self, user_id: int) -> bool: """Give one account this campaign's welcome grant exactly once.""" user_id = int(user_id) + if self.store.has_promotion_grant( + user_id, SUPERSEDED_SIGNUP_CREDIT_CAMPAIGNS + ): + return False parts = { "campaign_key": DEFAULT_SIGNUP_CREDIT_CAMPAIGN, "user_id": user_id, diff --git a/dashboard/backend/infrastructure/llm/execution/errors.py b/dashboard/backend/infrastructure/llm/execution/errors.py index 05b37629..2d6e1b84 100644 --- a/dashboard/backend/infrastructure/llm/execution/errors.py +++ b/dashboard/backend/infrastructure/llm/execution/errors.py @@ -18,6 +18,7 @@ class ExecutionErrorCategory(StrEnum): BILLING_FAILED = "billing_failed" PROVIDER_QUOTA_EXHAUSTED = "provider_quota_exhausted" ACCOUNT_RESTRICTED = "account_restricted" + INSUFFICIENT_CREDITS = "insufficient_credits" WORKER_FAILED = "worker_failed" @@ -37,6 +38,15 @@ class ExecutionErrorCategory(StrEnum): "Your Credits account is paused. Add Credits to settle model usage " "or contact an administrator." ), + # The balance cannot cover the next call's reservation. Kept apart from + # BILLING_FAILED, which it used to fall into via the catch-all in + # ``_execute_platform``: "you are out of Credits" is the user's to fix, + # "billing broke" is ours, and one message for both sent users hunting + # for a bug that was an empty balance. + ExecutionErrorCategory.INSUFFICIENT_CREDITS: ( + "Not enough ATL Credits to continue this run. Add Credits on the " + "Credits page, choose a lower-cost model, or use your own API key." + ), ExecutionErrorCategory.WORKER_FAILED: "The model worker failed before completion.", } @@ -72,6 +82,10 @@ def __init__( ): allowed_message = message self.safe_message = allowed_message or _SAFE_MESSAGES[self.category] + # Set by ``account_restricted``; lets a process boundary that can carry + # only fixed tokens (the backtest child's run_failed line) rebuild the + # restriction-specific message rather than the generic one. + self.restriction_reason: str | None = None super().__init__(self.safe_message) @classmethod @@ -93,7 +107,10 @@ def account_restricted( message = _SAFE_RESTRICTED_MESSAGES.get( reason, _SAFE_MESSAGES[ExecutionErrorCategory.ACCOUNT_RESTRICTED] ) - return cls(ExecutionErrorCategory.ACCOUNT_RESTRICTED, message) + error = cls(ExecutionErrorCategory.ACCOUNT_RESTRICTED, message) + if reason in _SAFE_RESTRICTED_MESSAGES: + error.restriction_reason = reason + return error # A hosted (``fail_closed``) run aborts on any model error, because a billing or @@ -117,9 +134,24 @@ def is_transient_provider_failure(error: BaseException) -> bool: ) +def run_failed_line(error: LLMExecutionError) -> str: + """The line a backtest child prints when a model call ends its run. + + Parsed by ``api/routers/backtests._child_llm_failure``. Fixed tokens only + (enum values), never upstream text, and the ``ERROR: llm.`` prefix makes + the parent relay it to the service log live. + """ + line = f"ERROR: llm.run_failed category={error.category.value}" + reason = getattr(error, "restriction_reason", None) + if reason: + line += f" reason={reason}" + return line + + __all__ = [ "ExecutionErrorCategory", "LLMExecutionError", "RetryHint", "is_transient_provider_failure", + "run_failed_line", ] diff --git a/dashboard/backend/infrastructure/llm/execution/service.py b/dashboard/backend/infrastructure/llm/execution/service.py index 1d2bfeb7..9e9416c6 100644 --- a/dashboard/backend/infrastructure/llm/execution/service.py +++ b/dashboard/backend/infrastructure/llm/execution/service.py @@ -12,6 +12,7 @@ from dashboard.backend.domain.credits.service import CreditsService from dashboard.backend.domain.credits.repository_common import ( CreditAccountRestrictedStoreError, + InsufficientCreditsError, ) from dashboard.backend.domain.analytics import instrumentation as analytics_instrumentation from dashboard.backend.domain.analytics.usage_cost import model_usage_properties @@ -149,6 +150,11 @@ def _pricing_snapshot_for(model_id: str, provider_id: str) -> PricingSnapshot: return PricingSnapshot.from_model(model_id, provider_id) +def prompt_token_ceiling_for_bytes(prompt_bytes: int) -> int: + """The input-token ceiling a reservation charges for ``prompt_bytes``.""" + return max(0, int(prompt_bytes)) + _PROMPT_FRAMING_TOKEN_ALLOWANCE + + def _prompt_token_ceiling(request: LLMExecutionRequest) -> int: payload = { "system_message": request.system_message, @@ -159,7 +165,38 @@ def _prompt_token_ceiling(request: LLMExecutionRequest) -> int: ensure_ascii=False, separators=(",", ":"), ).encode("utf-8") - return len(encoded) + _PROMPT_FRAMING_TOKEN_ALLOWANCE + return prompt_token_ceiling_for_bytes(len(encoded)) + + +def _ceiling_micro( + input_tokens: int, output_tokens: int, pricing_snapshot: PricingSnapshot +) -> int | None: + ceiling_cost = estimate_cost_from_snapshot( + pricing_snapshot, + LLMUsage(input_tokens=int(input_tokens), output_tokens=int(output_tokens)), + ) + if ceiling_cost is None: + return None + return credits_micro_for_usd(ceiling_cost) + + +def reservation_ceiling_micro_for( + *, + prompt_bytes: int, + max_output_tokens: int, + pricing_snapshot: PricingSnapshot, +) -> int | None: + """The reservation one request of this shape holds, or None if unpriceable. + + Shared with the backtest launch preflight so the two cannot drift: its + "the last call must find a whole ceiling free" is only true while it prices + exactly what ``_reservation_ceiling_micro`` does. + """ + return _ceiling_micro( + prompt_token_ceiling_for_bytes(prompt_bytes), + max_output_tokens, + pricing_snapshot, + ) def _reservation_ceiling_micro( @@ -168,14 +205,32 @@ def _reservation_ceiling_micro( ) -> int: """Return the maximum billable cost for this one provider request.""" - ceiling_usage = LLMUsage( - input_tokens=_prompt_token_ceiling(request), - output_tokens=request.usage_policy.max_output_tokens, + ceiling = _ceiling_micro( + _prompt_token_ceiling(request), + request.usage_policy.max_output_tokens, + pricing_snapshot, ) - ceiling_cost = estimate_cost_from_snapshot(pricing_snapshot, ceiling_usage) - if ceiling_cost is None: + if ceiling is None: raise LLMExecutionError(ExecutionErrorCategory.BILLING_FAILED) - return credits_micro_for_usd(ceiling_cost) + return ceiling + + +def analytics_error_category(category: ExecutionErrorCategory) -> str: + """The analytics ``error_category`` for an execution failure category. + + Module-level so the backtest parent can label a child's failure without + constructing (or depending on) the service. + """ + return { + ExecutionErrorCategory.CREDENTIAL_MISSING: "credential_missing", + ExecutionErrorCategory.CREDENTIAL_INVALID: "credential_invalid", + ExecutionErrorCategory.PROVIDER_UNAVAILABLE: "provider_unavailable", + ExecutionErrorCategory.PROVIDER_TIMEOUT: "provider_timeout", + ExecutionErrorCategory.PROVIDER_QUOTA_EXHAUSTED: "provider_quota_exhausted", + ExecutionErrorCategory.BILLING_FAILED: "credits_unavailable", + ExecutionErrorCategory.ACCOUNT_RESTRICTED: "account_restricted", + ExecutionErrorCategory.INSUFFICIENT_CREDITS: "credits_insufficient", + }.get(category, "internal_error") class LLMExecutionService: @@ -244,15 +299,7 @@ def execute(self, request: LLMExecutionRequest) -> LLMExecutionResult: def _analytics_error_category( category: ExecutionErrorCategory, ) -> str: - return { - ExecutionErrorCategory.CREDENTIAL_MISSING: "credential_missing", - ExecutionErrorCategory.CREDENTIAL_INVALID: "credential_invalid", - ExecutionErrorCategory.PROVIDER_UNAVAILABLE: "provider_unavailable", - ExecutionErrorCategory.PROVIDER_TIMEOUT: "provider_timeout", - ExecutionErrorCategory.PROVIDER_QUOTA_EXHAUSTED: "provider_quota_exhausted", - ExecutionErrorCategory.BILLING_FAILED: "credits_unavailable", - ExecutionErrorCategory.ACCOUNT_RESTRICTED: "account_restricted", - }.get(category, "internal_error") + return analytics_error_category(category) @staticmethod def _emit_model_usage( @@ -345,6 +392,10 @@ def _execute_platform( raise LLMExecutionError.account_restricted( reason, outstanding_micro ) from exc + except InsufficientCreditsError as exc: + raise LLMExecutionError( + ExecutionErrorCategory.INSUFFICIENT_CREDITS + ) from exc reservation_id = reservation.reservation_id self._platform_runs.add(request.run_id) if reservation.status != "open": diff --git a/dashboard/backend/tests/domain/analytics/test_models.py b/dashboard/backend/tests/domain/analytics/test_models.py index 35caa082..1c4fa0d3 100644 --- a/dashboard/backend/tests/domain/analytics/test_models.py +++ b/dashboard/backend/tests/domain/analytics/test_models.py @@ -300,3 +300,19 @@ def test_safe_error_accepts_provider_quota_exhausted(): ) ) assert record.error_category == "provider_quota_exhausted" + + +def test_every_execution_failure_category_maps_into_the_allow_list(): + # `account_restricted` was mapped but not admitted, so emit_safe_error_event + # raised inside its own guard and every restricted-account failure was + # dropped with a WARNING line. Any category added to the mapping must be + # admitted here in the same change. + from dashboard.backend.infrastructure.llm.execution.errors import ( + ExecutionErrorCategory, + ) + from dashboard.backend.infrastructure.llm.execution.service import ( + analytics_error_category, + ) + + for category in ExecutionErrorCategory: + assert analytics_error_category(category) in ALLOWED_ERROR_CATEGORIES, category diff --git a/dashboard/backend/tests/domain/analytics/test_states.py b/dashboard/backend/tests/domain/analytics/test_states.py index a8811e45..e51ce5e6 100644 --- a/dashboard/backend/tests/domain/analytics/test_states.py +++ b/dashboard/backend/tests/domain/analytics/test_states.py @@ -649,3 +649,23 @@ def test_value_snapshot_repair_isolates_one_user_failure(tmp_path, capsys): assert "analytics.value_snapshot_repair_failed" in output assert "category=RuntimeError" in output assert "private first-user failure" not in output + + +def test_a_restricted_account_is_blocked_like_an_empty_balance(tmp_path): + service, store = _fixture(tmp_path) + _run(service, "backtest_failed", 0, NOW - timedelta(hours=2), error_category="internal_error") + blocked = service.record_server_event( + event_name="safe_error_recorded", + user_id=1, + source_event_id="resource:safe_error_recorded:run-restricted:account_restricted", + source_record_type="run", + source_record_id="run-restricted", + error_category="account_restricted", + occurred_at=NOW - timedelta(hours=1), + ).event + + snapshot = calculate_user_state(1, now=NOW, store=store) + + assert snapshot.status == "blocked" + assert snapshot.reason_code == "billing_lane_unavailable" + assert snapshot.evidence_event_ids == [blocked.event_id] diff --git a/dashboard/backend/tests/domain/credits/test_promotion_grants.py b/dashboard/backend/tests/domain/credits/test_promotion_grants.py index 6babefd7..f8c805fa 100644 --- a/dashboard/backend/tests/domain/credits/test_promotion_grants.py +++ b/dashboard/backend/tests/domain/credits/test_promotion_grants.py @@ -14,6 +14,7 @@ from dashboard.backend.domain.credits.service import ( DEFAULT_SIGNUP_CREDIT_CAMPAIGN, DEFAULT_SIGNUP_CREDITS_MICRO, + SUPERSEDED_SIGNUP_CREDIT_CAMPAIGNS, CreditsService, ) @@ -56,7 +57,7 @@ def test_welcome_grant_is_exactly_once_and_visible_in_activity(tmp_path): balance = service.get_balance(1) assert balance.grant_available_micro == DEFAULT_SIGNUP_CREDITS_MICRO - assert balance.display_grant_credits == "1.500000" + assert balance.display_grant_credits == "8.000000" activity = service.list_ledger(1, limit=10, cursor=None)["items"] assert len(activity) == 1 assert activity[0]["entry_type"] == "system_promotion_grant" @@ -146,7 +147,10 @@ def test_welcome_grant_funds_platform_credit_reservations(tmp_path): ) assert reservation.status == "open" - assert service.get_balance(1).grant_available_micro == 500_000 + assert ( + service.get_balance(1).grant_available_micro + == DEFAULT_SIGNUP_CREDITS_MICRO - 1_000_000 + ) def test_restricted_account_receives_campaign_but_cannot_spend_it(tmp_path): @@ -165,3 +169,32 @@ def test_restricted_account_receives_campaign_but_cannot_spend_it(tmp_path): attempt_index=0, amount_micro=1, ) + + +def test_account_holding_the_v1_welcome_grant_does_not_get_v2_on_top(tmp_path): + # Every account that existed before v2 holds the 1.5-Credit v1 grant, and + # the boot backfill replays the campaign over every account. Without the + # superseded-campaign check that replay would credit each of them another + # 8 Credits on the next deploy. + store = _store(tmp_path, user_count=2) + service = CreditsService(store=store) + legacy_key = SUPERSEDED_SIGNUP_CREDIT_CAMPAIGNS[0] + store.grant_promotion_credits( + user_id=1, + campaign_key=legacy_key, + amount_micro=1_500_000, + operation_id=f"legacy-op-{legacy_key}-1", + idempotency_key=f"promotion:{legacy_key}:user:1", + request_digest="legacy-digest", + source="system_promotion", + reason="Automatic welcome Credits.", + ) + + assert service.grant_default_signup_credits(1) is False + assert service.get_balance(1).grant_available_micro == 1_500_000 + + report = service.backfill_default_signup_credits() + assert report == {"total": 2, "granted": 1, "existing": 1, "failed": 0} + assert service.get_balance(1).grant_available_micro == 1_500_000 + assert service.get_balance(2).grant_available_micro == DEFAULT_SIGNUP_CREDITS_MICRO + assert DEFAULT_SIGNUP_CREDIT_CAMPAIGN not in SUPERSEDED_SIGNUP_CREDIT_CAMPAIGNS diff --git a/dashboard/backend/tests/infrastructure/llm/test_platform_credits_env_fallback.py b/dashboard/backend/tests/infrastructure/llm/test_platform_credits_env_fallback.py index c6905fd9..dbf9eb4c 100644 --- a/dashboard/backend/tests/infrastructure/llm/test_platform_credits_env_fallback.py +++ b/dashboard/backend/tests/infrastructure/llm/test_platform_credits_env_fallback.py @@ -366,6 +366,7 @@ def test_platform_quota_error_retries_once_through_commonstack( ExecutionErrorCategory.USAGE_UNAVAILABLE, ExecutionErrorCategory.BILLING_FAILED, ExecutionErrorCategory.ACCOUNT_RESTRICTED, + ExecutionErrorCategory.INSUFFICIENT_CREDITS, ], ) def test_non_quota_platform_failures_do_not_fail_over( @@ -724,6 +725,34 @@ def test_restricted_account_execution_error_is_actionable( assert "CreditAccountRestrictedStoreError" not in exc_info.value.safe_message +def test_a_balance_that_cannot_cover_the_reservation_says_so(tmp_path, monkeypatch): + # It used to fall through `_execute_platform`'s catch-all and reach the + # user as "Model usage billing could not be completed." -- which read as a + # billing bug to someone whose welcome grant had simply run out. + adapter = FakeExecutionAdapter(LLMUsage(input_tokens=100, output_tokens=100)) + service, credits_store = _execution_service(tmp_path, monkeypatch, adapter) + available = credits_store.get_balance_micro(USER_ID) + credits_store.reserve_llm_credits( + reservation_id="drain-the-balance", + user_id=USER_ID, + run_id="another-run", + call_index=0, + provider_id="openrouter", + attempt_index=0, + reserved_micro=available - 1, + operation_key="drain-the-balance", + request_digest="d" * 64, + ) + + with pytest.raises(LLMExecutionError) as exc_info: + service.execute(_request("insufficient-balance")) + + assert exc_info.value.category is ExecutionErrorCategory.INSUFFICIENT_CREDITS + assert "Not enough ATL Credits" in exc_info.value.safe_message + # Refused at the reservation, before any provider was asked. + assert adapter.secrets == [] + + def _platform_request(run_id: str, provider_ids: tuple[str, ...]) -> LLMExecutionRequest: return LLMExecutionRequest( user_id=USER_ID, @@ -1122,6 +1151,7 @@ def test_retry_after_is_honoured_up_to_cap( ExecutionErrorCategory.USAGE_UNAVAILABLE, ExecutionErrorCategory.BILLING_FAILED, ExecutionErrorCategory.ACCOUNT_RESTRICTED, + ExecutionErrorCategory.INSUFFICIENT_CREDITS, ], ) def test_non_failover_categories_never_retry_even_with_a_stray_hint( diff --git a/dashboard/backend/tests/test_admin_analytics_frontend.py b/dashboard/backend/tests/test_admin_analytics_frontend.py index de1d4a7d..3a0f6a55 100644 --- a/dashboard/backend/tests/test_admin_analytics_frontend.py +++ b/dashboard/backend/tests/test_admin_analytics_frontend.py @@ -162,7 +162,7 @@ def test_app_lifecycle_and_cache_versions_are_wired(): # Lockstep owner for the console's bumped tags and the /admin page's pins: # every bump edits this test in the same change (Global Constraints). assert 'styles.css?v=159' in APP_HTML - assert 'app.js?v=158' in APP_HTML + assert 'app.js?v=159' in APP_HTML assert 'js/admin-tabs.js?v=12' in APP_HTML for tag in ( 'href="admin.css?v=11"', diff --git a/dashboard/backend/tests/test_analytics_frontend.py b/dashboard/backend/tests/test_analytics_frontend.py index 4b77a327..7c722877 100644 --- a/dashboard/backend/tests/test_analytics_frontend.py +++ b/dashboard/backend/tests/test_analytics_frontend.py @@ -41,7 +41,7 @@ def _function_body(source: str, signature: str) -> str: def test_analytics_script_loads_between_app_and_page_scripts(): - app_at = APP_HTML.index('') + app_at = APP_HTML.index('') analytics_at = APP_HTML.index( '' ) diff --git a/dashboard/backend/tests/test_auth.py b/dashboard/backend/tests/test_auth.py index 23ba2a99..50aeea70 100644 --- a/dashboard/backend/tests/test_auth.py +++ b/dashboard/backend/tests/test_auth.py @@ -61,7 +61,7 @@ def test_signup_and_login_keep_one_welcome_credit_grant(client): ) assert signup.status_code == 200 user_id = signup.json()["user"]["id"] - assert auth.credits_service.get_balance(user_id).display_credits == "1.500000" + assert auth.credits_service.get_balance(user_id).display_credits == "8.000000" login = client.post( "/api/auth/login", diff --git a/dashboard/backend/tests/test_backtest_comparison_frontend.py b/dashboard/backend/tests/test_backtest_comparison_frontend.py index d3568c38..ce90d9c6 100644 --- a/dashboard/backend/tests/test_backtest_comparison_frontend.py +++ b/dashboard/backend/tests/test_backtest_comparison_frontend.py @@ -191,7 +191,7 @@ def test_exact_raw_ties_mark_every_tied_series_best(): def test_comparison_script_and_semantic_table_ship_before_app(): helper = '' - app = '' + app = '' assert 'href="styles.css?v=159"' in APP_HTML assert APP_HTML.index(helper) < APP_HTML.index(app) for element_id in ( diff --git a/dashboard/backend/tests/test_backtests_router.py b/dashboard/backend/tests/test_backtests_router.py index 02ac15ad..6ad6b55a 100644 --- a/dashboard/backend/tests/test_backtests_router.py +++ b/dashboard/backend/tests/test_backtests_router.py @@ -1578,6 +1578,7 @@ def test_platform_credits_resolves_candidates_without_provider_input(monkeypatch spy = _Spy() service = _AutoPlatformExecutionPreflightService() captured_handoff = {} + monkeypatch.setattr(bt, "credits_service", _FakeBalanceCredits(100_000_000)) monkeypatch.setattr(bt, "run_backtest_background", spy) monkeypatch.setattr(bt, "get_model_provider_service", lambda: service) monkeypatch.setattr( @@ -3407,3 +3408,543 @@ def test_parent_labels_an_unclosed_completed_run_trace_close_failed(monkeypatch) monkeypatch, FakeChild(returncode=0, stdout="ok\n"), run_id=run_id, ) assert labelled[0] == (run_id, "trace_close_failed") + + +# --- ATL Credits launch preflight (B/C: an empty balance is refused up front, +# and one that runs dry mid-run is reported as such, not as a billing bug) ---- + + +class _FakeBalance: + def __init__(self, available_micro, *, status="active", reason=None, outstanding=0): + self.total_available_micro = available_micro + self.account_status = status + self.restriction_reason = reason + self.outstanding_credits_micro = outstanding + + +class _FakeBalanceCredits: + def __init__(self, available_micro, **kwargs): + self.balance = _FakeBalance(available_micro, **kwargs) + self.balance_reads = [] + + def get_balance(self, user_id): + self.balance_reads.append(user_id) + return self.balance + + +def _post_platform_run(monkeypatch, credits, *, model="openai/gpt-5.5", billing_mode="platform_credits"): + spy = _Spy() + handoffs = [] + service = _AutoPlatformExecutionPreflightService() + if billing_mode == "byok": + service = _ExecutionPreflightService() + monkeypatch.setattr(bt, "credits_service", credits) + monkeypatch.setattr(bt, "run_backtest_background", spy) + monkeypatch.setattr(bt, "get_model_provider_service", lambda: service) + monkeypatch.setattr( + "dashboard.backend.api.dependencies._optional_user", + lambda *_args, **_kwargs: {"id": 7}, + ) + monkeypatch.setattr( + bt, "create_execution_handoff", lambda **kw: handoffs.append(kw) or "h" + ) + body = { + # The shipped modal window: 7 weekdays, 49 US bars. + "start_date": "2026-09-21", + "end_date": "2026-09-29", + "decision_source": "llm", + "billing_mode": billing_mode, + "model": model, + } + if billing_mode == "byok": + body["provider_id"] = "openai" + response = TestClient(app).post("/backtest/run", json=body, headers=_sess()) + return response, spy, handoffs + + +def test_platform_credits_run_the_balance_cannot_finish_is_refused_with_402(monkeypatch): + # The 2026-10-07 report: 1.5 Credits, GPT-5.5, DJIA 30. The run spent + # 1.32 Credits over 39 bars and then failed; now it never starts. + credits = _FakeBalanceCredits(1_500_000) + + response, spy, handoffs = _post_platform_run(monkeypatch, credits) + + assert response.status_code == 402, response.text + detail = response.json()["detail"] + assert "ATL Credits" in detail + assert "49 model calls on openai/gpt-5.5" in detail + assert "your balance is 1.50 Credits" in detail + assert credits.balance_reads == [7] + # Refused before anything was held: no handoff signed, no worker started. + assert handoffs == [] + assert spy.calls == 0 + + +def test_platform_credits_run_the_balance_covers_proceeds(monkeypatch): + credits = _FakeBalanceCredits(8_000_000) + + response, spy, handoffs = _post_platform_run(monkeypatch, credits) + + assert response.status_code == 200, response.text + assert spy.calls == 1 + assert len(handoffs) == 1 + + +def test_restricted_account_is_refused_with_its_restriction_message(monkeypatch): + credits = _FakeBalanceCredits( + 50_000_000, status="restricted", reason="refund_reconciliation" + ) + + response, spy, _ = _post_platform_run(monkeypatch, credits) + + assert response.status_code == 403 + assert "paused for payment refund review" in response.json()["detail"] + assert spy.calls == 0 + + +def test_an_unreadable_balance_lets_the_run_through(monkeypatch, capsys): + # The per-call reservation still refuses what the balance cannot cover; + # a broken courtesy check must not take the whole lane down with it. + class _Broken: + def get_balance(self, _user_id): + raise RuntimeError("store down") + + response, spy, _ = _post_platform_run(monkeypatch, _Broken()) + + assert response.status_code == 200, response.text + assert spy.calls == 1 + assert "credits.preflight_balance_unavailable" in capsys.readouterr().out + + +def test_byok_runs_never_read_the_credits_balance(monkeypatch): + credits = _FakeBalanceCredits(0) + monkeypatch.setattr( + bt._backtest_rate_limiter, "allow", lambda *_a, **_k: True + ) + + response, spy, _ = _post_platform_run(monkeypatch, credits, billing_mode="byok") + + assert response.status_code == 200, response.text + assert credits.balance_reads == [] + + +def _gpt55_djia(**overrides): + kwargs = dict( + model_id="openai/gpt-5.5", + provider_id="commonstack", + sessions=7, + data_source="alpaca", + universe_size=30, + pipeline=[{"label": "Step 1", "prompt": "trade"}], + ) + kwargs.update(overrides) + return bt._estimated_platform_credits(**kwargs) + + +def test_platform_credits_estimate_matches_the_observed_reservation_shape(): + # The ceiling term is the reservation's own rule: prod reserved + # 0.178-0.187 Credits per GPT-5.5 DJIA 30 call. + ceiling = bt.reservation_ceiling_micro_for( + prompt_bytes=bt._PREFLIGHT_PROMPT_FIXED_BYTES + + bt._PREFLIGHT_PROMPT_BYTES_PER_SYMBOL * 30 + + len("trade"), + max_output_tokens=bt.RECOVERY_MAX_OUTPUT_TOKENS, + pricing_snapshot=bt.PricingSnapshot.from_model("openai/gpt-5.5", "commonstack"), + ) + assert 170_000 <= ceiling <= 200_000 + assert _gpt55_djia(sessions=1).calls == 7 + full = _gpt55_djia() + assert full.calls == 49 + # The 2026-10-07 dead run spent 1.32 Credits over 39 calls -- a mean of + # 0.0338 per call, i.e. ~1.81 Credits for 49 calls with the last call's + # ceiling held. A typical estimate must land near that (within 10%), not + # at "every call at its ceiling" (~9 Credits), which refuses runs that + # finish -- and it still refuses the 1.5-Credit account that died. + assert 1_630_000 < full.needed_micro < 2_000_000 + assert full.needed_micro > 1_500_000 + assert _gpt55_djia(sessions=0) is None + + +def test_platform_credits_estimate_uses_the_models_own_output_median(): + sonnet = _gpt55_djia(model_id="anthropic/claude-sonnet-4-6") + assert bt._preflight_expected_output_tokens("openai/gpt-5.5") == 574 + assert bt._preflight_expected_output_tokens("anthropic/claude-sonnet-4-6") == 420 + # An unmeasured model takes the middle of the measured ones. + unmeasured = bt._preflight_expected_output_tokens("anthropic/claude-haiku-4-5") + assert 420 < unmeasured < 1335 + assert sonnet.calls == 49 + + +def test_platform_credits_estimate_counts_the_callers_own_prompt_text(): + short = _gpt55_djia() + long_instruction = _gpt55_djia( + pipeline=[{"label": "Step 1", "prompt": "x" * 8000}] + ) + assert long_instruction.needed_micro > short.needed_micro + two_steps = _gpt55_djia( + pipeline=[ + {"label": "Research", "prompt": "trade"}, + {"label": "Execute", "prompt": "trade"}, + ] + ) + # Twice the calls, and step 2 carries step 1's output. + assert two_steps.calls == 98 + single_prompt = _gpt55_djia(pipeline=None, strategy_prompt=None) + with_strategy = _gpt55_djia(pipeline=None, strategy_prompt="y" * 8000) + assert with_strategy.needed_micro > single_prompt.needed_micro + + +def test_trading_sessions_skip_nyse_holidays_and_cn_counts_weekdays(): + # Thanksgiving 2025 (Thu 11-27): four sessions, not five weekdays. + assert bt._estimated_trading_sessions("2025-11-24", "2025-11-28", market="US") == 4 + assert bt._estimated_decision_days("2025-11-24", "2025-11-28") == 5 + assert bt._estimated_trading_sessions("2025-11-24", "2025-11-28", market="CN") == 5 + assert bt._estimated_trading_sessions("2026-09-21", "2026-09-29", market="US") == 7 + + +def test_trading_sessions_stop_at_the_settled_end(monkeypatch): + # A window reaching a session still in progress does not trade it. + monkeypatch.setattr( + bt, "settled_exclusive_end", lambda end_date, market=None: "2026-09-25" + ) + assert bt._estimated_trading_sessions("2026-09-21", "2026-09-29", market="US") == 4 + + +def test_credits_figures_never_print_equal_on_both_sides(): + assert bt._format_credits(1_804_900, round_up=True) == "1.81" + assert bt._format_credits(1_800_100) == "1.80" + assert bt._format_credits(1_800_000, round_up=True) == "1.80" + + +def _admit_running_platform_slot(run_id, *, user_id, estimate): + claim = bt._CreditsClaim( + needed_micro=estimate, + available_micro=10**12, + calls=1, + model_id="openai/gpt-5.5", + spent_by_run={}, + ) + assert ( + bt._try_acquire_backtest_slot( + live_run_id=run_id, + session_id=str(uuid.uuid4()), + user_id=user_id, + credits_claim=claim, + ) + is None + ) + + +def test_a_launch_counts_what_the_owners_running_runs_still_need(monkeypatch): + # 8 Credits, and a sibling GPT-5.5 run admitted at 7.5 that has settled 1. + # Alone this launch (~1.8) fits; beside the sibling's remaining ~6.5 it does + # not (8 - 6.5 = 1.5), and must be refused before it starves both. + _admit_running_platform_slot("agent_sibling", user_id=7, estimate=7_500_000) + + class _Credits(_FakeBalanceCredits): + def sum_run_llm_spend(self, user_id, run_id): + return (1_000_000, 10) if run_id == "agent_sibling" else (0, 0) + + credits = _Credits(8_000_000) + response, spy, handoffs = _post_platform_run(monkeypatch, credits) + + assert response.status_code == 402, response.text + detail = response.json()["detail"] + assert "your balance is 8.00 Credits" in detail + assert "about 6.50 of which your 1 running backtest is expected to use" in detail + assert handoffs == [] and spy.calls == 0 + + +def test_a_sibling_run_of_another_account_is_not_counted(monkeypatch): + _admit_running_platform_slot("agent_stranger", user_id=99, estimate=7_000_000) + + class _Credits(_FakeBalanceCredits): + def sum_run_llm_spend(self, user_id, run_id): + raise AssertionError("a stranger's spend must not be read") + + response, spy, _ = _post_platform_run(monkeypatch, _Credits(8_000_000)) + + assert response.status_code == 200, response.text + assert spy.calls == 1 + + +def test_the_slot_ledger_rechecks_a_credits_claim_under_its_lock(): + # Two launches that both read the balance before either took a slot: the + # second one to reach the ledger sees the first and is refused. + claim = bt._CreditsClaim( + needed_micro=5_000_000, + available_micro=8_000_000, + calls=49, + model_id="openai/gpt-5.5", + spent_by_run={}, + ) + first = bt._try_acquire_backtest_slot( + live_run_id="agent_race_a", + session_id=str(uuid.uuid4()), + user_id=31, + credits_claim=claim, + ) + second = bt._try_acquire_backtest_slot( + live_run_id="agent_race_b", + session_id=str(uuid.uuid4()), + user_id=31, + credits_claim=claim, + ) + assert first is None + assert isinstance(second, bt._CreditsRefusal) + assert "running backtest" in second + assert bt._active_slots["agent_race_a"]["credits_estimate_micro"] == 5_000_000 + assert "agent_race_b" not in bt._active_slots + + +def test_snapshot_width_follows_the_portfolio_manager_rule(): + pipeline = [{"label": "Step 1"}] + assert bt._snapshot_symbols_shown(30, pipeline) == 30 + assert bt._snapshot_symbols_shown(31, pipeline) == bt.SNAPSHOT_SHORTLIST_SIZE + assert bt._snapshot_symbols_shown(30, None) == bt.SNAPSHOT_SHORTLIST_SIZE + assert bt._snapshot_symbols_shown(7, None) == 7 + + +def test_child_llm_failure_reads_only_known_categories(): + line = "ERROR: llm.run_failed category=insufficient_credits\n" + failure = bt._child_llm_failure("noise\n" + line, "Traceback ...") + assert failure is not None + assert failure.category.value == "insufficient_credits" + assert bt._child_llm_failure("ERROR: llm.run_failed category=made_up\n") is None + assert bt._child_llm_failure("", None) is None + + +def test_every_category_round_trips_through_the_child_line(): + # Producer-driven: the child prints errors.run_failed_line, so a format + # change on either side fails here instead of silently falling back to + # the traceback tail and `internal_error`. + from dashboard.backend.infrastructure.llm.execution.errors import ( + ExecutionErrorCategory, + LLMExecutionError as _Err, + run_failed_line, + ) + + for category in ExecutionErrorCategory: + exc = _Err(category) + failure = bt._child_llm_failure("noise\n" + run_failed_line(exc) + "\n") + assert failure is not None, category + assert failure.category is category + assert failure.safe_message == exc.safe_message + restricted = _Err.account_restricted("refund_reconciliation") + rebuilt = bt._child_llm_failure(run_failed_line(restricted)) + assert rebuilt.safe_message == restricted.safe_message + + +@pytest.mark.parametrize( + ("reason", "expected"), + [ + ("refund_reconciliation", "Contact an administrator"), + ("llm_overage", "exceeded its reserved amount"), + ], +) +def test_child_llm_failure_keeps_the_restriction_reason(reason, expected): + # The reason picks the remedy; without it a refund-review pause was told + # to "Add Credits", which cannot lift it. + from dashboard.backend.infrastructure.llm.execution.errors import ( + LLMExecutionError as _Err, + ) + + exc = _Err.account_restricted(reason, 12_345) + assert exc.restriction_reason == reason + line = f"ERROR: llm.run_failed category=account_restricted reason={reason}\n" + failure = bt._child_llm_failure(line) + assert failure.category.value == "account_restricted" + assert expected in failure.safe_message + unknown = bt._child_llm_failure( + "ERROR: llm.run_failed category=account_restricted reason=made_up\n" + ) + assert unknown.safe_message.startswith("Your Credits account is paused.") + + +def test_a_run_that_runs_out_of_credits_reports_it_and_its_category(monkeypatch): + emitted = [] + monkeypatch.setattr( + bt.analytics_instrumentation, + "emit_run_event", + lambda **kwargs: emitted.append(kwargs), + ) + run_id = "agent_out_of_credits" + session_id = str(uuid.uuid4()) + assert ( + bt._try_acquire_backtest_slot( + live_run_id=run_id, session_id=session_id, user_id=4242 + ) + is None + ) + child = FakeChild( + returncode=1, + stdout=( + "3️⃣ Running backtests...\n" + "ERROR: llm.run_failed category=insufficient_credits\n" + ), + stderr=( + "Traceback (most recent call last):\n" + "dashboard.backend.infrastructure.llm.execution.errors." + "LLMExecutionError: Not enough ATL Credits to continue this run.\n" + ), + ) + + class FakeExecutionService: + def __init__(self, **_kwargs): + pass + + def finalize_run(self, execution_run_id, *, billing_mode=None): + return [] + + monkeypatch.setattr(subprocess, "Popen", lambda cmd, **kwargs: child) + monkeypatch.setattr(bt, "LLMExecutionService", FakeExecutionService) + monkeypatch.setattr(bt, "get_model_provider_service", object) + monkeypatch.setattr(bt, "run_backtest_background", _REAL_RUN_BACKTEST_BACKGROUND) + + bt.run_backtest_background( + start_date="2026-01-01", + end_date="2026-01-02", + session_id=session_id, + live_run_id=run_id, + decision_source="rule_based", + execution_handoff_payload="opaque-test-handoff", + billing_mode="platform_credits", + ) + + slot = bt._recent_slots[run_id] + assert slot["error"] == ( + "Backtest stopped: Not enough ATL Credits to continue this run. Add " + "Credits on the Credits page, choose a lower-cost model, or use your " + "own API key." + ) + assert "Traceback" not in slot["error"] + assert emitted[-1]["event_name"] == "backtest_failed" + assert emitted[-1]["error_category"] == "credits_insufficient" + + +def test_a_failure_without_a_category_line_keeps_the_generic_category(monkeypatch): + emitted = [] + monkeypatch.setattr( + bt.analytics_instrumentation, + "emit_run_event", + lambda **kwargs: emitted.append(kwargs), + ) + run_id = "agent_generic_failure" + session_id = str(uuid.uuid4()) + assert ( + bt._try_acquire_backtest_slot( + live_run_id=run_id, session_id=session_id, user_id=4242 + ) + is None + ) + bt._finalize_slot(run_id, error="Backtest failed with return code 1. boom", runs_count=0) + assert emitted[-1]["error_category"] == "internal_error" + + +def test_child_llm_failure_reads_the_other_stream_past_an_unknown_line(): + failure = bt._child_llm_failure( + "ERROR: llm.run_failed category=made_up\n", + "ERROR: llm.run_failed category=billing_failed\n", + ) + assert failure is not None + assert failure.category.value == "billing_failed" + + +def test_an_llm_failure_names_account_facts_to_the_runs_owner_only(): + run_id = "agent_redacted_failure" + agent_session = str(uuid.uuid4()) + owner_session = str(uuid.uuid4()) + assert ( + bt._try_acquire_backtest_slot( + live_run_id=run_id, + session_id=agent_session, + owner_session=owner_session, + user_id=4242, + ) + is None + ) + private = ( + "Backtest stopped: Your Credits account is paused for payment refund " + "review. Contact an administrator to restore access." + ) + bt._finalize_slot( + run_id, + error=private, + runs_count=0, + error_category="account_restricted", + public_error=bt._LLM_FAILURE_PUBLIC_ERROR, + ) + slot = bt._recent_slots[run_id] + + # Anyone holding the built-in agent's published session id. + stranger = bt._status_snapshot_for(slot, session_id=agent_session, user_id=None) + assert stranger["error"] == bt._LLM_FAILURE_PUBLIC_ERROR + assert "refund" not in str(stranger) + owner = bt._status_snapshot_for(slot, session_id=owner_session, user_id=4242) + assert owner["error"] == private + # The legacy mirror answers whoever polls with the slot's session id. + assert "refund" not in str(bt.backtest_status.get("error")) + + +def test_a_failure_with_nothing_personal_is_not_redacted(): + run_id = "agent_plain_failure" + session = str(uuid.uuid4()) + assert ( + bt._try_acquire_backtest_slot(live_run_id=run_id, session_id=session, user_id=None) + is None + ) + bt._finalize_slot(run_id, error="Backtest failed with return code 1. boom", runs_count=0) + snapshot = bt._status_snapshot_for( + bt._recent_slots[run_id], session_id=str(uuid.uuid4()), user_id=None + ) + assert snapshot["error"] == "Backtest failed with return code 1. boom" + + +def test_the_child_reports_the_runs_own_error_past_a_failed_finalize(): + from dashboard.backend.infrastructure.llm.execution.errors import ( + ExecutionErrorCategory, + LLMExecutionError, + ) + from dashboard.scripts import backtest_hourly_agent as child + + try: + try: + raise LLMExecutionError(ExecutionErrorCategory.INSUFFICIENT_CREDITS) + finally: + # finalize_run raising in the run loop's ``finally``. + raise LLMExecutionError(ExecutionErrorCategory.BILLING_FAILED) + except LLMExecutionError as exc: + line = child._run_failed_report(exc) + assert line == "ERROR: llm.run_failed category=insufficient_credits" + + alone = LLMExecutionError(ExecutionErrorCategory.BILLING_FAILED) + assert ( + child._run_failed_report(alone) + == "ERROR: llm.run_failed category=billing_failed" + ) + + +def test_the_child_reports_llm_failures_at_its_cli_boundary(): + # Setup (the handoff) and finalize_run both raise outside the agent loop, + # so the report belongs to the __main__ block, not to the loop. + import ast + from pathlib import Path + + source = ( + Path(__file__).resolve().parents[2] / "scripts" / "backtest_hourly_agent.py" + ).read_text(encoding="utf-8") + tree = ast.parse(source) + main_block = next( + node + for node in tree.body + if isinstance(node, ast.If) and "__main__" in ast.unparse(node.test) + ) + handlers = [ + handler + for node in ast.walk(main_block) + if isinstance(node, ast.Try) + for handler in node.handlers + if handler.type is not None and ast.unparse(handler.type) == "LLMExecutionError" + ] + assert handlers, "__main__ must catch LLMExecutionError" + assert "_run_failed_report" in ast.unparse(handlers[0]) diff --git a/dashboard/backend/tests/test_frontend_fast_boot.py b/dashboard/backend/tests/test_frontend_fast_boot.py index 88c1a9b2..5551c90d 100644 --- a/dashboard/backend/tests/test_frontend_fast_boot.py +++ b/dashboard/backend/tests/test_frontend_fast_boot.py @@ -194,7 +194,7 @@ def test_cache_busters_bumped(): # the next bump, so the exact one looks like the broken guard and gets # "fixed" by loosening it. That collision has already cost this repo one # round of follow-ups (#347/#348). - assert "app.js?v=158" in APP_HTML + assert "app.js?v=159" in APP_HTML assert "js/agent-editor.js?v=32" in APP_HTML assert "styles.css?v=159" in APP_HTML assert "js/leaderboard.js?v=37" in APP_HTML diff --git a/dashboard/backend/tests/test_ifind_ashare_frontend.py b/dashboard/backend/tests/test_ifind_ashare_frontend.py index 46b8f366..31c2c21b 100644 --- a/dashboard/backend/tests/test_ifind_ashare_frontend.py +++ b/dashboard/backend/tests/test_ifind_ashare_frontend.py @@ -614,3 +614,59 @@ def test_the_client_outlives_the_server_budget_it_draws(js): int(ceiling.group(1)) == PIPELINE_SUBPROCESS_TIMEOUT_SECONDS + SUBPROCESS_TIMEOUT_OVERHEAD_SECONDS ) + + +def _map_backtest_error_with_status(message: str, status: int) -> str: + script = "\n".join( + [ + js_const("IFIND_ASHARE_SOURCE"), + "const window = {};", + fn_body("function formatBacktestError("), + "process.stdout.write(JSON.stringify(formatBacktestError(" + f"{json.dumps({'message': message, 'status': status})}, IFIND_ASHARE_SOURCE)));", + ] + ) + result = subprocess.run( + ["node", "-e", script], capture_output=True, text=True, timeout=30 + ) + assert result.returncode == 0, result.stderr + return json.loads(result.stdout) + + +@pytest.mark.skipif(shutil.which("node") is None, reason="node is not installed") +def test_credits_refusals_survive_the_ifind_error_mapper(): + """An A-share LLM run bills ATL Credits like any other, and the iFinD + mapper otherwise rewrites every message it does not recognise into "The + iFinD backtest failed" -- and every 403 into "iFinD access is disabled". + The messages come from their producers, not typed here.""" + from dashboard.backend.infrastructure.llm.execution.errors import ( + ExecutionErrorCategory, + LLMExecutionError, + ) + + stopped = ( + "Backtest stopped: " + + LLMExecutionError(ExecutionErrorCategory.INSUFFICIENT_CREDITS).safe_message + ) + assert _map_backtest_error(stopped) == stopped + + paused = LLMExecutionError.account_restricted("refund_reconciliation").safe_message + assert _map_backtest_error_with_status(paused, 403) == paused + + # The 402 arm on its own: no "ATL Credits" in the text to lean on. + refused = "Payment required for this run." + assert _map_backtest_error_with_status(refused, 402) == refused + + # Any in-run model failure, not only Credits: these contain 'credential' + # and 'response', which the iFinD arms would otherwise claim. + for category in ( + ExecutionErrorCategory.CREDENTIAL_INVALID, + ExecutionErrorCategory.RESPONSE_INVALID, + ExecutionErrorCategory.PROVIDER_TIMEOUT, + ): + message = "Backtest stopped: " + LLMExecutionError(category).safe_message + assert _map_backtest_error(message) == message + # And the iFinD arms still own iFinD's own 403. + assert "iFinD A-share access is disabled" in _map_backtest_error_with_status( + "Forbidden", 403 + ) diff --git a/dashboard/frontend/app.html b/dashboard/frontend/app.html index f28d8a44..b51b62d8 100644 --- a/dashboard/frontend/app.html +++ b/dashboard/frontend/app.html @@ -2512,7 +2512,7 @@

Refund Credits purchase

- + diff --git a/dashboard/frontend/app.js b/dashboard/frontend/app.js index 1ae3d5c1..bc31f5c9 100644 --- a/dashboard/frontend/app.js +++ b/dashboard/frontend/app.js @@ -7791,6 +7791,20 @@ function getSelectedAssets() { function formatBacktestError(error, dataSource = null) { const source = dataSource || window.ACTIVE_BACKTEST_DATA_SOURCE || 'alpaca'; const raw = String(error?.message || error?.detail || error || 'Backtest failed.'); + // Model-execution and Credits failures are not about the tape: the launch + // 402, an in-run model stop and a paused account. Below, + // an iFinD run maps every unrecognised message to a generic iFinD + // failure (and every 403 to "iFinD access is disabled"), which would + // hide the one thing the user can fix. + // "Backtest stopped: " is only ever the parent's rendering of a fixed model + // execution message (credential, provider, Credits), never tape text. + if ( + Number(error?.status || 0) === 402 + || raw.startsWith('Backtest stopped: ') + || /ATL Credits|Credits account is paused/.test(raw) + ) { + return raw; + } if (source !== IFIND_ASHARE_SOURCE) return raw; const status = Number(error?.status || 0); diff --git a/dashboard/scripts/backtest_hourly_agent.py b/dashboard/scripts/backtest_hourly_agent.py index 3c80bdd4..8d5bd57e 100644 --- a/dashboard/scripts/backtest_hourly_agent.py +++ b/dashboard/scripts/backtest_hourly_agent.py @@ -196,7 +196,10 @@ from dashboard.backend.infrastructure.llm.execution.client import ( AnthropicCompatibleExecutionClient, ) -from dashboard.backend.infrastructure.llm.execution.errors import LLMExecutionError +from dashboard.backend.infrastructure.llm.execution.errors import ( + LLMExecutionError, + run_failed_line, +) from dashboard.backend.infrastructure.llm.execution.service import LLMExecutionService from dashboard.backend.domain.credits.service import credits_service from dashboard.backend.domain.model_providers.service import get_model_provider_service @@ -686,6 +689,22 @@ def _exit_on_sigterm(signum, _frame): raise SystemExit(128 + signum) +def _run_failed_report(exc: LLMExecutionError) -> str: + """The line the parent reads to name a model failure, not a traceback. + + One machine-readable line (``api/routers/backtests._child_llm_failure``); + the ``ERROR: llm.`` prefix also has the parent relay it to the service log + live, and the category is a fixed enum value, never upstream text. A + ``finalize_run`` that raises while the run's own error is unwinding + replaces it; the run's error is the cause the user can act on ("add + Credits"), so the first execution error in the chain is reported. + """ + cause = exc + while isinstance(cause.__context__, LLMExecutionError): + cause = cause.__context__ + return run_failed_line(cause) + + if __name__ == "__main__": signal.signal(signal.SIGTERM, _exit_on_sigterm) # Reached through the module alias bound at the top rather than a second @@ -695,6 +714,13 @@ def _exit_on_sigterm(signum, _frame): try: try: main() + except LLMExecutionError as exc: + # Caught at the CLI boundary rather than around the agent loop + # alone, so a handoff that fails during setup and a + # ``finalize_run`` that raises BILLING_FAILED in the loop's + # ``finally`` name their category too. + print(_run_failed_report(exc), flush=True) + raise except _alpaca_bars.MarketDataUnavailableError as exc: # Library code raises (so server threads can catch it); the CLI # boundary is where the process exit belongs.