Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,9 @@ FINGPT_API_KEY=your_fingpt_api_key_here
# Optional override of the FinSearch raw news-items endpoint (defaults to the
# live endpoint). Point at a local fixture/staging server for dev.
# FINSEARCH_ITEMS_URL=https://agenticfinsearch.org/api/news/items/
# The Home news panel serves a frozen "Demo news" snapshot while FinSearch's
# feed is paused (unset = frozen). Set to 0 to proxy the live feed.
# NEWS_FEED_FROZEN=0

# OpenRouter multi-provider gateway (Anthropic Messages skin).
# Leaderboard entries with "integration": "openrouter" route through this
Expand Down
1 change: 1 addition & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,7 @@ python dashboard/scripts/backtest_hourly_agent.py # main hourly agent backte
- `MAX_LEGACY_ACTIVE_PER_SESSION` / `MAX_LEGACY_ACTIVE_GLOBAL` (optional, defaults **5** / **50**, 0 disables): concurrency budgets for the legacy `/api/v1/backtest/*` surface. That surface authenticates nothing — `_require_session` accepts any non-empty `X-Session-Id` — and writes **no `protocol_runs` row**, so the per-agent, per-account and global protocol caps are all blind to its runs; before these it was the one unbounded path into the same engine. The per-session budget bounds a looping client (its key is caller-chosen, so it is not a bound against someone who rotates it); the global one is the memory bound that holds regardless, since every live session pins a loaded bar window. Enforced via `start_backtest(enforce_session_cap=True)`, which **only** the legacy route passes — the protocol surfaces reach the same function through `run_service.create_run`, which has already applied its three caps.
- `ANALYTICS_DAILY_JOB_INTERVAL_SECONDS` (optional, default **300**, range 5–3600): how often the analytics daily-facts worker (`domain/analytics/daily_job.py`, started from `app.py` beside the run reaper) wakes to ask whether yesterday's `user_daily_facts` row set is due. It runs on **its own thread**, not the 60-second reaper tick — the reaper exists to keep run heartbeats fresh against `RUN_HEARTBEAT_STALE_SECONDS`, and a whole-population batch across three databases on that thread would let a slow analytics night mark live runs as orphaned (design D23). An idle wake costs exactly one store call (the refused day claim on `analytics_projection_jobs`), so the interval buys freshness after midnight, not cost. Junk or out-of-range values log and fall back to 300 rather than raising at import — a bare `int()` at module scope has killed app boot in this repo before. `tests/conftest.py` strips it. The worker also runs the idempotent `user_activity` seed and the eight-week history copy (`domain/analytics/facts_migration.py`) once before its first tick, and owns `rollup_day` and the analytics retention coordinator; the reaper keeps only the two throttled snapshot repairs in `maintenance.py` until PR B deletes them. Design: `docs/superpowers/specs/2026-09-15-admin-layer-redesign-design.md` §6.9, §6.12.
- `LEADERBOARD_DAILY_AUTO_DEPLOY` (optional, **strict opt-in — this one spends money**): when truthy, serving the *public, unauthenticated* `GET /api/v1/leaderboard?period=daily` may start a background thread that runs `deploy_model_run` for every competition LLM entry — real billable API calls initiated by an anonymous request. It is therefore off unless explicitly set (`1`/`true`/`yes`/`on`); do **not** restore the old "on whenever `RENDER` is unset" default, which armed it on the Docker image, every self-host and fork, and the test suite (`tests/conftest.py` deliberately strips `RENDER`, and strips this var too). Prod previously relied on the nightly refresh job instead — **but that schedule is paused as of PR #352** (see `LEADERBOARD_DAILY_REFRESH_SECRET` above), so prod currently runs *neither* this flag nor the cron. Nothing refreshes the daily board automatically today, by design: the Live Trading Leaderboard that replaced its tab is a Season 0 preview with no advance engine yet. Do not "restore" automation here to close the gap — arming this flag is the anonymous-billable-request path the strict opt-in exists to prevent. Note the in-progress guard and the per-window state file (`storage/data/leaderboard_daily_refresh.json`) are **per-process**: fine for the current single-instance Render deploy, but a multi-replica deploy would duplicate model deploys.
- `NEWS_FEED_FROZEN` (optional, **unset = frozen** while FinSearch's feed timers are paused, since 2026-10-11): read per request by `api/routers/news.py::news_feed_frozen()`. Frozen, `GET /api/news/signals` never calls FinSearch and serves `dashboard/config/news_demo_snapshot.json` (the last live payload); the loader adds `demo: true` and `staleness_hours: null`, so both Home news consumers (`home-page.js`, `home-news-signals.js`) show "Demo news" and fixed dates instead of a frozen "Updated 0.2h ago" and ever-growing "3w ago". To resume: re-enable the FinSearch timers on the droplet first, then set `NEWS_FEED_FROZEN=0` in the Render dashboard (no deploy). Junk reads as frozen and warns once. A snapshot that fails to load fails closed to `unavailable` and is retried on the next request. The adapter (see *Fail-closed is not fail-visible*) still backs `/api/v2` backtest steps (`?as_of=`) either way. `tests/conftest.py` strips it.
- Alpaca paper-trading credentials also live in `credentials/alpaca.json` (gitignored; see `credentials/alpaca.json.example`).

## Architecture
Expand Down
88 changes: 88 additions & 0 deletions dashboard/backend/api/routers/news.py
Original file line number Diff line number Diff line change
@@ -1,13 +1,99 @@
"""Panel proxy for FinSearch news signals — keeps FINGPT_API_KEY server-side."""
import copy
import json
import logging
import os
import threading

from fastapi import APIRouter

from dashboard.backend.cache import (CACHE_KEY_NEWS_SIGNALS, TTL_NEWS,
TTL_NEWS_UNAVAILABLE, shared_ttl_cache)
from dashboard.backend.infrastructure.llm.validator import DJIA_30
from dashboard.backend.integrations import news_sentiment
from dashboard.backend.paths import CONFIG_DIR

logger = logging.getLogger(__name__)

router = APIRouter(prefix="/news", tags=["news"])

# FinSearch's news pipeline is paused (2026-10-11), so the panel serves the last
# live payload, frozen, flagged `demo: true` for the "Demo news" line. Proxying
# live would only show a feed going stale with no explanation.
#
# NEWS_FEED_FROZEN is read per request so the feed resumes from the Render
# dashboard with no deploy: restart the FinSearch timers, then set it to `0`.
# Unset means frozen while the pipeline is paused. A value outside the boolean
# vocabulary also reads as frozen — the unfrozen path calls an upstream that may
# be dead, while the frozen path is labelled and local — and warns once.
NEWS_FEED_FROZEN_ENV = "NEWS_FEED_FROZEN"
NEWS_FEED_FROZEN_DEFAULT = True
NEWS_DEMO_SNAPSHOT_PATH = CONFIG_DIR / "news_demo_snapshot.json"

_TRUTHY = {"1", "true", "yes", "on"}
_FALSEY = {"0", "false", "no", "off"}
_warned_values: set[str] = set()

# The keys the panel reads; a snapshot without them is not a payload.
_SNAPSHOT_REQUIRED_KEYS = ("status", "feed", "signals")

_snapshot_lock = threading.Lock()
_snapshot_cache: dict | None = None


def news_feed_frozen() -> bool:
raw = (os.getenv(NEWS_FEED_FROZEN_ENV) or "").strip().lower()
if not raw:
return NEWS_FEED_FROZEN_DEFAULT
if raw in _FALSEY:
return False
if raw not in _TRUTHY and raw not in _warned_values:
_warned_values.add(raw)
logger.warning("news: %s=%r is not a boolean; serving the frozen snapshot",
NEWS_FEED_FROZEN_ENV, raw)
return True


def _load_demo_snapshot() -> dict | None:
try:
payload = json.loads(NEWS_DEMO_SNAPSHOT_PATH.read_text(encoding="utf-8"))
except (OSError, ValueError):
logger.error("news: demo snapshot %s unreadable; serving unavailable",
NEWS_DEMO_SNAPSHOT_PATH, exc_info=True)
return None
if not isinstance(payload, dict) or not all(k in payload for k in _SNAPSHOT_REQUIRED_KEYS):
logger.error("news: demo snapshot %s is not a panel payload (needs %s); "
"serving unavailable", NEWS_DEMO_SNAPSHOT_PATH,
", ".join(_SNAPSHOT_REQUIRED_KEYS))
return None
# The loader owns both frozen-feed markers so the file cannot disagree with
# them: `demo` drives the "Demo news" line, and a frozen "Updated 0.2h ago"
# would be false.
return {**payload, "demo": True, "staleness_hours": None}


def _demo_snapshot() -> dict:
"""A private copy of the frozen payload, or UNAVAILABLE if it cannot load.

Only a successful load is cached: a transient read error must not pin
`unavailable` for the life of the process. The copy is deep because the
payload's `feed`/`signals` are nested and shared across requests.
"""
global _snapshot_cache
if _snapshot_cache is None:
with _snapshot_lock:
if _snapshot_cache is None:
_snapshot_cache = _load_demo_snapshot()
if _snapshot_cache is None:
return dict(news_sentiment.UNAVAILABLE_PAYLOAD)
return copy.deepcopy(_snapshot_cache)


def _reset_demo_snapshot_cache() -> None:
global _snapshot_cache
with _snapshot_lock:
_snapshot_cache = None


def _fetch_panel() -> dict:
# get_latest_panel_payload already fails closed to UNAVAILABLE_PAYLOAD; the
Expand All @@ -23,6 +109,8 @@ def _fetch_panel() -> dict:
# to `async def` without moving the HTTP call off the loop.
@router.get("/signals")
def latest_news_signals():
if news_feed_frozen():
return _demo_snapshot()
# Single-flight lives in the cache primitive: one leader hits the producer,
# concurrent callers return the cached value (or a brief UNAVAILABLE) WITHOUT
# blocking a worker behind the ~10s upstream call — so a cold-cache burst of
Expand Down
5 changes: 5 additions & 0 deletions dashboard/backend/tests/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,11 @@
# mismatches. base.py reads it at import, which this module precedes.
os.environ.pop("LLM_PROVIDER_READ_TIMEOUT_SECONDS", None)

# Decides whether /api/news/signals serves the frozen snapshot or proxies
# FinSearch. The router tests set it explicitly per case; a developer who
# exported it would otherwise see the default-path test assert the wrong mode.
os.environ.pop("NEWS_FEED_FROZEN", None)


@atexit.register
def _cleanup_test_db_dir() -> None:
Expand Down
2 changes: 1 addition & 1 deletion dashboard/backend/tests/test_frontend_fast_boot.py
Original file line number Diff line number Diff line change
Expand Up @@ -198,7 +198,7 @@ def test_cache_busters_bumped():
assert "js/agent-editor.js?v=32" in APP_HTML
assert "styles.css?v=159" in APP_HTML
assert "js/leaderboard.js?v=37" in APP_HTML
assert "home-page.js?v=54" in APP_HTML
assert "home-page.js?v=55" in APP_HTML
assert "js/credit-format.js?v=1" in APP_HTML
assert "js/credits.js?v=8" in APP_HTML
assert "js/admin-credits.js?v=9" in APP_HTML
140 changes: 140 additions & 0 deletions dashboard/backend/tests/test_frontend_home_news_demo.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,140 @@
"""The frozen news snapshot (`demo: true`, api/routers/news.py) under node.

Both consumers of /api/news/signals run their real render code against the
committed snapshot: home-page.js (the Home Market News / Signals tabs) and
home-news-signals.js (the #homeNewsSignals section). Each must label the frozen
feed "Demo news" rather than present it as live, and show fixed dates instead
of relative times that keep growing on a paused feed.
"""

import json
import shutil
from pathlib import Path

import pytest

from dashboard.backend.tests._frontend_source import FRONTEND, _match_brace, fn_body, run_node

pytestmark = pytest.mark.skipif(shutil.which("node") is None, reason="node is not installed")

_HOME_JS = (FRONTEND / "home-page.js").read_text(encoding="utf-8")
_SIGNALS_JS = (FRONTEND / "home-news-signals.js").read_text(encoding="utf-8")
_SNAPSHOT = json.loads(
(Path(__file__).resolve().parents[2] / "config" / "news_demo_snapshot.json").read_text(encoding="utf-8")
)


def _home_mock_news() -> str:
start = _HOME_JS.index("const HOME_MOCK_NEWS = {")
return _HOME_JS[start : _match_brace(_HOME_JS, _HOME_JS.index("{", start)) + 1] + ";"


_FAKE_DOM = """
const els = {};
function el(id) {
if (!els[id]) els[id] = { id, hidden: false, textContent: '', innerHTML: '' };
return els[id];
}
const document = { getElementById: el };
const window = {
formatMarketEventRelativeTime: (ms, now) => 'REL',
};
"""


def _home_harness(payload: dict) -> str:
fns = [fn_body(f"function {name}(", _HOME_JS) for name in (
"homeEscape", "homeSafeUrl", "homeSentimentClass", "homeRelTime",
"homeFeedTime", "setHomeMarketDemoBadge", "renderHomeMarketNews",
"renderHomeMarketSignals",
)]
return "\n".join([_FAKE_DOM, *fns, _home_mock_news(), f"""
const payload = {json.dumps(payload)};
renderHomeMarketNews(payload);
const news = {{ status: {{ ...el('homeMarketStatus') }}, list: el('homeModuleNewsList').innerHTML,
badge: el('homeMarketDemoBadge').hidden }};
renderHomeMarketSignals(payload);
console.log(JSON.stringify({{ news, signals: {{ status: {{ ...el('homeMarketSignalsStatus') }},
list: el('homeModuleSignalsList').innerHTML }} }}));
"""])


def _live(payload: dict) -> dict:
out = {k: v for k, v in payload.items() if k != "demo"}
out["staleness_hours"] = 1.5
return out


@pytest.fixture
def demo_payload():
# What the loader serves: the file plus the two markers it owns.
return {**_SNAPSHOT, "demo": True, "staleness_hours": None}


def test_home_news_labels_the_snapshot_demo_news(demo_payload):
out = run_node(_home_harness(demo_payload))
assert out["news"]["status"]["hidden"] is False
assert out["news"]["status"]["textContent"] == "Demo news"
# Real stories, not HOME_MOCK_NEWS: the mock "Demo data" badge stays hidden.
assert out["news"]["badge"] is True
assert _SNAPSHOT["feed"][0]["headline"].split()[0] in out["news"]["list"]
assert out["signals"]["status"]["textContent"] == f"Demo news · {len(_SNAPSHOT['signals'])} signals"


def test_home_news_shows_fixed_dates_not_relative_times(demo_payload):
out = run_node(_home_harness(demo_payload))
assert "REL" not in out["news"]["list"]
assert "Oct " in out["news"]["list"]


def test_home_live_feed_keeps_relative_times_and_updated_line(demo_payload):
out = run_node(_home_harness(_live(demo_payload)))
assert out["news"]["status"]["textContent"] == "Updated 1.5h ago"
assert "REL" in out["news"]["list"]
assert out["signals"]["status"]["textContent"] == f"{len(_SNAPSHOT['signals'])} signals"


def test_home_signals_read_the_live_sentiment_score(demo_payload):
"""Live signals carry `sentiment_score`; reading the mock's `score` rendered
every one as 0.00."""
sym, sig = next((s, v) for s, v in _SNAPSHOT["signals"].items() if v["sentiment_score"])
out = run_node(_home_harness(demo_payload))
assert f"{sym} · score {sig['sentiment_score']:.2f}" in out["signals"]["list"]


def test_home_mock_still_reads_its_score_field():
out = run_node(_home_harness({"status": "unavailable"}))
assert "AAPL · score 0.72" in out["signals"]["list"]
assert out["news"]["badge"] is False # mock data: "Demo data" badge shown


def _signals_harness(payload: dict) -> str:
return "\n".join([_FAKE_DOM, """
function escapeHtml(v) {
return String(v == null ? '' : v).replace(/&/g, '&amp;').replace(/</g, '&lt;')
.replace(/>/g, '&gt;').replace(/"/g, '&quot;').replace(/'/g, '&#39;');
}
const API_BASE = '';
let served;
const API = { get: async () => served };
global.setInterval = () => 1;
""", f"served = {json.dumps(payload)};", _SIGNALS_JS, """
window.newsSignalsPanel.onShow();
setTimeout(() => console.log(JSON.stringify({
updated: el('nnsUpdated').textContent, badge: el('nnsStatusBadge').hidden,
feed: el('nnsFeedList').innerHTML,
})), 0);
"""])


def test_signals_section_labels_the_snapshot_demo_news(demo_payload):
out = run_node(_signals_harness(demo_payload))
assert out["updated"] == "Demo news"
assert out["badge"] is True
assert "REL" not in out["feed"] and "Oct " in out["feed"]


def test_signals_section_live_feed_unchanged(demo_payload):
out = run_node(_signals_harness(_live(demo_payload)))
assert out["updated"] == "Updated 1.5h ago"
assert "REL" in out["feed"]
Loading
Loading