[PowerX] persist telemetry and recover historical reads / 持久化遥测并恢复历史数据读取

[cquil11]

Summary

GitHub deletes benchmark telemetry artifacts after 90 days and the dashboard re-downloads and re-parses them on every view, so older PowerX curves are gone and recent ones are slow. This PR ingests the telemetry into the database once, serves it DB-first with GitHub artifacts as the fallback and backfill source. The dashboard views built on that stored data follow in #1237, stacked on this branch.

flowchart TB
    subgraph Before["Before"]
        direction LR
        B1["Page"] --> B2["GitHub artifacts"] --> B3["Parse CSV"] --> B4["Display"]
        B2 -.-> B5["Expired: unavailable"]
    end
    subgraph After["After"]
        direction LR
        A1["Artifacts"] --> A2["Ingest"] --> A3["Persistent DB"]
        A4["Page"] --> A5["DB-first API"]
        A3 --> A5 --> A6["Display"]
        A5 -. "Missing / incomplete" .-> A7["GitHub fallback"]
        A7 --> A6
    end

Loading

Area	Before	After
Storage	Artifact-dependent	Samples, provenance, statistics, benchmark links
Sources	CSV viewer	NVIDIA/AMD CSVs + multinode bundle windows
Inspection	Run-level charts	Stored series, source coverage; Timeline and per-point View PowerX are in #1237
Display	Existing controls/statistics	Smoothing controls; stored full-record statistics
Consistency	Artifact reads	Atomic series replacement; revision-aware point cache
Validation	Existing ingest checks	Completeness receipts; required-power manifest checks
Recovery	Existing ingest paths	Purge-aware telemetry backfill; statistics upgrades; migration-first overrides
Queries	Artifact downloads and parsing	SQL selectors; 50,000-row pages; selected-host views

DB failures return errors. Failed background refreshes preserve cached telemetry; public statistics omit the storage-only metric field. Live artifact reads normalize NVIDIA timestamps to ISO UTC like stored reads.

• Follow-ups: unified default collection, automatic repair, route/component decomposition.
• Operations: repair/rollback runbook. Apply migrations 016+017 before ingest or backfill.

AI model disclosure

GPT-6: implementation/delegated verification/description; exact variant unavailable. claude-opus-5: review. Earlier contributors' exact versions unavailable.

Validation

• I have completed the AI model disclosure and kept it current

• Current head 7734516: the DB/ETL/API/explorer half of the former 6d7729f tree as 10 feature commits (read bottom-up from schema to docs); the dashboard views are [PowerX] power boundary metrics, comparison overlays, point view and timeline / 功耗边界指标、对比叠加、逐点视图与时间线 #1237. Local on this head: typecheck, lint, format and typography pass; app 6,001 unit tests pass; DB 881 pass with one pre-existing OperatorX PGlite timeout.

• Test pruning (8c36086, perf-ruler part reverted in ad05dce): removed 38 redundant PowerX unit cases (types tests here; measured-metric-config and chart-utils tests in [PowerX] power boundary metrics, comparison overlays, point view and timeline / 功耗边界指标、对比叠加、逐点视图与时间线 #1237); 22 added regressions. On unchanged production code across a fixed 91-file scope, aggregate coverage lost at most 0.11 percentage points. The 27 perf-ruler deletions moved to [Tests] remove redundant perf-ruler cases / 删除冗余的 perf-ruler 测试用例 #1235.

• Earlier recorded checks: local DB/browser 12/12, context checks, mixed-source browser 2/2, and official/unofficial fixtures passed on earlier revisions.

• Limits: hosted acceptance and production repair remain unexecuted. The local audit reported 23 dependency advisories; dependency manifests and lockfile were unchanged. Real Neon response size/latency was not measured.

中文说明

GitHub 在 90 天后删除 benchmark 遥测 artifact，而仪表板每次查看都要重新下载、解析，导致较早的 PowerX 曲线消失、较新的曲线加载缓慢。本 PR 把遥测一次性入库，API 优先读数据库、GitHub artifact 退为回退与回填来源。建立在入库数据之上的仪表板视图见堆叠于本分支的 #1237。上图对比旧版按页面请求下载、解析 artifact，以及新版入库后优先读取 DB 的路径。缺失或不完整的数据可尝试 GitHub fallback；DB 查询故障会明确报错。后台刷新失败时保留已缓存的遥测图表，公开统计不再返回数据库内部的 metric 字段。live artifact 读取与入库路径一样将 NVIDIA 时间戳归一为 ISO UTC。

范围	改动前	改动后
存储	依赖 artifact 可用	保存采样、来源、统计及 benchmark 关联
来源	CSV 查看器	支持 NVIDIA/AMD CSV 及多节点 bundle 的测量窗口
查看	Run 级图表	已存储序列与来源覆盖情况；Timeline 与单点 View PowerX 见 #1237
展示	已有控件和统计	增加平滑控件，使用已存储的全采集周期统计
一致性	从 artifact 读取	单个 series 原子更新，单点缓存随数据版本刷新
验证	已有入库检查	增加完整性回执及 required-power manifest 检查
恢复	已有入库路径	遥测回填跳过已清理点，支持统计升级；覆盖操作前先执行 migration
查询	下载并解析产物	SQL 筛选、每页最多 50,000 行；公开视图只加载所选主机序列

• 后续工作： 统一默认采集入口、自动修复、拆分过大的路由和组件。
• 运维： 定向修复与回滚见上方 runbook；运行 ingest 或 backfill 前先执行迁移 016+017。

GPT-6 负责实现、委派验证及描述修改，精确变体无法核实；claude-opus-5 负责审查。早期贡献者的精确模型版本无法核实。

• 当前 head 7734516： 原 6d7729f 代码树中 DB/ETL/API/explorer 的那一半，按功能拆为 10 个 commit（建议自底向上从 schema 读到 docs）；仪表板视图见 [PowerX] power boundary metrics, comparison overlays, point view and timeline / 功耗边界指标、对比叠加、逐点视图与时间线 #1237。本 head 本地：类型检查、lint、格式与排版检查通过；app 6,001 个单元测试通过；DB 881 个通过，另有 1 个既有的 OperatorX PGlite 超时。
• 测试精简（8c360860，perf-ruler 部分已在 ad05dce 恢复）： 删除 38 个冗余 PowerX 单元用例（types 测试在本 PR；measured-metric-config 与 chart-utils 测试在 [PowerX] power boundary metrics, comparison overlays, point view and timeline / 功耗边界指标、对比叠加、逐点视图与时间线 #1237）；新增 22 个回归用例。在生产代码不变、固定 91 个源码文件的对照中，总体覆盖率下降不超过 0.11 个百分点。27 个 perf-ruler 删减移至 [Tests] remove redundant perf-ruler cases / 删除冗余的 perf-ruler 测试用例 #1235。
• 此前验收记录： 旧版本的本地 DB/浏览器 12/12、context 检查、混合来源浏览器 2/2 及正式/非正式运行 fixture 验证通过。
• 限制： 托管环境验收及生产数据修复尚未执行。本地依赖审计报告 23 项告警，本轮未修改依赖清单和锁文件。未实测真实 Neon 响应大小或延迟。

---

Note

High Risk
Touches production ingest, schema migrations, benchmark publication preflight, and CDN invalidation workflows; incorrect telemetry or manifest gating could block sweeps or serve stale dashboard data.

Overview
PowerX telemetry is ingested into the database and served DB-first, with GitHub artifacts kept as fallback/backfill so curves survive 90-day retention and repeat views avoid re-parsing zips.

Ingest and CI: New gpu_metric_* storage (migrations 016/017) digests CSVs and multinode power_audit bundles at ingest, links series to benchmark points, and versions full-record per-GPU statistics. Workflows gain optional require-power / INGEST_REQUIRE_POWER, stricter required-power manifest v2 preflight (shared fixture), telemetry receipts, and admin:db:backfill-gpu-metrics (including --stats-only). apply-run-overrides runs migrations before verify and still invalidates/warms the CDN when overrides apply successfully.

APIs: /api/gpu-metrics reads stored telemetry first (series=power Timeline via read-only POST + sourceCoverage), returns 503 for DB outage or known incomplete storage, and merges healthy DB windows with artifact recovery when allowed. /api/v1/gpu-metrics-point exposes per-point samples/stats with revision-checked Blob cache and no-store. The public gpu-metrics view prefers stored digests for the full-record stats table (startup/warmup included).

UI: PowerX explorer adds rolling-average / mean-of-chips display controls, overlay axis support, refetch-on-focus, and stats scoped to stored digests when present. Docs, fixtures, PGlite acceptance tests, and Cypress tweaks support recovery and contract checks.

^{Reviewed by Cursor Bugbot for commit 0181958. Bugbot is set up for automated code reviews on this repo. Configure here.}

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
inferencemax-app	Ready	Preview	Sep 30, 2026 9:20pm UTC

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit f320e18. Configure here.}

[cursor]

Correlation defaults to missing metric

Medium Severity

The public views route still defaults corrYMetric to temperature. buildCorrelationData now drops samples where that metric is absent, so power-only DCGM series return an empty correlation plot. The explorer already falls back to another collected metric for this case.

Additional Locations (2)

• packages/app/src/components/gpu-power/chart-data.ts#L80-L85
• packages/app/src/components/gpu-power/GpuPowerDisplay.tsx#L242-L251

 

^{Reviewed by Cursor Bugbot for commit f320e18. Configure here.}