feat: add session replay consistency framework#239
Conversation
Adds a backend-agnostic replay harness that runs a fixed set of fixture cases against InMemory and a configurable backend (SQLite by default, Redis or MySQL via env var), then diffs the canonical snapshots so silent divergences between backends surface as test failures. - trpc_agent_sdk/replay: harness + diff engine + multiple backends - tests/sessions: replay fixtures, integration tests for Redis and SQL - .github/workflows: replay-integration CI (Redis + MySQL jobs) - docs/mkdocs: en + zh design docs - pyproject.toml: register 'integration' marker - examples/replay_consistency_demo: end-to-end demo - tests/sessions/replay_cases: replay case fixtures (jsonl) - tests/sessions/replay_diff_report.json: latest replay snapshot ci: replay-integration workflow skip semantics The integration jobs (Redis, MySQL) deliberately do NOT use a job-level ``if:`` condition. ``secrets.X`` is intentionally unavailable in job-level ``if:`` expressions for security reasons, and ``env.X`` is resolved in declaration order — both trip the GitHub Actions expression linter. The "skip cleanly when the operator has not configured the integration secret" behaviour is implemented one layer down, in the ``integration_runtime`` fixture in tests/sessions/conftest.py: every Redis/MySQL integration test reads the fixture and calls ``pytest.skip(...)`` when the env var is unset. Pytest's skip is a clean exit code 0, so the workflow stays green for forks that opt out of running external services — exactly as the header comment promises. Co-authored-by: Cursor <[email protected]>
|
CLA Assistant Lite bot: I have read the CLA Document and I hereby sign the CLA You can retrigger this bot by commenting recheck in this Pull Request |
AI Code Review基于对 pr.diff 及仓库上下文（ 发现的问题
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #239 +/- ##
==========================================
Coverage ? 87.74910%
==========================================
Files ? 492
Lines ? 45866
Branches ? 0
==========================================
Hits ? 40247
Misses ? 5619
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…_summary coverage - _main.py --backends goes through resolve_backend_names for case and whitespace normalization, preventing InMemory or SQLite variants from being treated as unknown backends by _build_backend - _diff.py introduces _EXPECTATION_BY_CASE_ID and _is_allowed_domain; known_summary_divergence diffs (events and summary domains) flow into the allowed bucket so unexpected_diff_count is no longer polluted by allowed diffs, aligning CLI exit code with pytest semantics - JSONL gains the 11th case fail_summary_recovery that triggers the fail_summary op, verifying that the attempted summary id does not leak, recovered=True holds, and the cache rolls back to the pre-compression state - Integration tests now pass environ=... explicitly to run_replay_harness; the test_run_replay_harness_does_not_pollute_os_environ guard regresses to assert isolation - Docs (zh and en) are aligned: case count updated from 10 to 11; the outdated job-level if-env guard description is replaced with the workflow-comment-faithful test-level skip description - tests/replay/test_cli.py drops the TDD loop-3 monkeypatch test; the same semantics are now covered by the fail_summary_recovery JSONL case - .gitignore adds .replay-work Regression: pytest tests/sessions/ tests/replay/ -> 365 passed, 4 skipped (the 4 skips are intentional: integration backends not configured per conftest design); CLI exit code 0 Co-authored-by: Cursor <[email protected]>
Some third-party actions (e.g. contributor-assistant/[email protected]) still pin Node 20, and GitHub has announced Node 20 deprecation on hosted runners (2025-09-19). The hello-world Python workflow does not need Node at all, but opting in via ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION silences the noisy stderr banner emitted by the runner on every run. The flag is applied at workflow top-level so it propagates to every job (including future jobs) without per-job duplication. Remove once all transitive actions used in this repo have moved off Node 20. Co-authored-by: Cursor <[email protected]>
AI Code Review
我已经掌握了足够的信息。让我来撰写最终的审查意见。 发现的问题🚨 Critical（无）
|
…n drift, docstring, gitignore) Co-authored-by: Cursor <[email protected]>
AI Code ReviewI now have a comprehensive understanding. Let me write the review. 发现的问题🚨 Critical
|
| "observed_duplicate_count": duplicate_count, | ||
| }) | ||
| elif op == "fail_summary": | ||
| session_checkpoint = session.model_copy(deep=True) |
There was a problem hiding this comment.
fail_summary 回滚 checkpoint 拷贝时机错误，持久化后端回滚失效
fail_summary 分支的 session_checkpoint 在 _perform_summary 之后才拷贝，此时 session 已被 summarizer 写入失败摘要锚点并持久化；随后用该 checkpoint 回滚存储与缓存无法清除失败尝试，导致 fail_summary_recovery 的 recovered 判定与 summary_id 不变量在 SQLite/Redis 等存储后端上失效，summary-fail-attempted 可能泄漏进 canonical events。应在 _perform_summary 之前深拷贝 checkpoint，并显式 drain/丢弃失败尝试产生的 metadata_log。
2 participants
概述
Closes #89
本 PR 针对 5 个 VulnGym 样本进行人工辅助的链路重建。这些样本并非单点行号偏移,而是整条数据流或调用链的语义错位——节点能找到代码,但位置不在利用链上;或漏洞需要多阶段触发,单一入口点难以表达完整利用过程。
修复目标
entry-00185:n8n ReadWriteFile 写入 .git/ 目录导致 RCE —— sink 落点不在利用链上。
entry-00197:openclaw sandbox TOCTOU race condition(多阶段)—— check/use 两阶段,单 entry 难表达完整利用窗口。
entry-00290:openclaw 悬空符号链接沙箱逃逸 —— check-vs-sink 错位,root cause 是 fail-open 决策。
entry-00320:langflow 文件上传路径穿越 —— 写入节点不在真实利用路径。
entry-00391:fastmcp OpenAPIProvider SSRF + 路径穿越 —— 链路过长,需要 SSRF 与路径穿越两段拼接才能表达完整利用过程。
每条 entry 的 entry_point / critical_operation / trace 均经 GitHub raw 远程字节级匹配 + 本地 SCHEMA 校验 双重复核(33 个 {file,line,code} 节点 H1/H2/H3/H4 全 0),并补全 desc 字段。
多阶段样本处理
entry-00197(TOCTOU):单 entry_point + 5 节点 trace 显式表达 "请求进入 → 语法校验 → realpath 检查 → 竞态窗口 → 实际读取" 的 check/use 两阶段。
entry-00290(悬空符号链接):critical_operation 保留在检查点 resolveSymlinkHopPath(root cause 是 fail-open 决策),最终 sink 写入操作放在 trace 末端;desc 显式说明 check-vs-sink 区别。
变更内容
新增 data/entries.fixed.jsonl —— 修复后的 JSONL 快照,408 行;5 条 target 的 entry_point / critical_operation / trace 已重建(替代 data/entries.jsonl 入库,二者内容等价)。
新增 rebuild_report.md —— 5 条 target 的逐条修复报告:修复前问题 / 修复后链路 / 多阶段说明 / 关键判断依据 / 验收对照与精简交付清单。
新增 fix_diff.csv —— 字段级 before/after 修改记录,5 entry × 3 fields。
新增 scripts/apply_entry_patches.py —— 幂等生成器,含 PATCHES 字典,可重跑出 fixed.jsonl / fix_diff.csv / report.json,内嵌 SCHEMA 校验。
新增 scripts/runtime_audit.py —— GitHub raw 远程字节级审计脚本,TARGETED = 5 条 target + 4 条 config-vuln 历史 target,输出 NDJSON 到 debug-b5f720.log。
新增 scripts/verify_patched_entries.py —— 本地 SCHEMA 校验脚本,对每条 target 的 entry_point / critical_operation / 全部 trace 节点校验 desc 非空 + line 合法 + code 形参。
验收对照
跑了 3 个脚本,全部 = 0 通过。
远程字节级审计
python scripts/runtime_audit.py --use-remote
5 条 target 的 entry_point / critical_operation / 全部 trace 共 33 个 {file,line,code} 节点,到 GitHub raw 对应 commit 拉源码行级精确匹配,H1/H2/H3/H4 全 0;NDJSON 落在 debug-b5f720.log(gitignored)。
本地 SCHEMA 校验
python scripts/verify_patched_entries.py
对 5 条 target 校验 entry_point / critical_operation / 全部 trace 节点的 desc 非空、line 合法、code 形态正确;H3 = 0 全条目 schema clean。
行数一致性
wc -l data/entries.fixed.jsonl
修复后 408 行,与 data/entries.jsonl 一致;本轮只对 5 条 target 做语义重建,未截断其它 entry。