Skip to content

[Bug]: agentflow 陷入 workspace checkpoint「保存→校验→恢复」循环:持续写入 + MySQL binlog 4 天膨胀 298GB #1138

Open

Description

@zirui-wang997

Component / host

Platform — web / API

Edition

Platform

LoopForge version

0.4.0

What happened?

[Bug] agentflow 陷入 workspace checkpoint「保存→校验→恢复」循环:持续写入 + MySQL binlog 4 天膨胀 298GB

环境

  • LoopForge 版本:ghcr.io/tencent/loopforge:0.4.0(自建镜像,基于 main 分支,含少量本地提交,checkpoint 相关代码与上游一致)
  • 部署方式:compose.yaml 自托管(Docker rootless)
  • MySQL:mysql:8.4(8.4.10),使用 compose 默认参数,未额外配置 binlog
  • 宿主机:Linux(Ubuntu 26.04,内核 7.0.0-43-generic),16 线程 / 32GB 内存

现象

platform 容器内的 agentflow 进程持续高频写入(实测 1.2~9.7 MB/s),陷入 workspace checkpoint 的「保存→校验→恢复」循环不收敛:

  • 进程累计 write_bytes:约 417GB / 25 小时(扣除 cancelled 后净写 ~372GB)
  • 打开的写句柄反复出现以下模式(与代码中的临时目录完全对应):
    • /tmp/agentflow-checkpoint-save-*
    • /tmp/workspace-checkpoint-validate-*/.workspace-restore-*/...
  • pidstat -d 抓到持续写入 1.2~9.7 MB/s(tmpfs 上写完即删,磁盘上看不到残留文件)
  • 重启 platform 容器无效:新 agentflow 进程立即恢复同样写入速率,说明不是偶发泄漏,而是确定性循环

对宿主机的影响:IO 压力(PSI io some)长期 43%~72%,整机卡顿。

连带损害:MySQL binlog 爆炸

checkpoint 循环写入经 MySQL 落到 binlog(compose 默认开启 binlog、无保留期配置):

  • 4 天内产生 299 个 binlog 文件,每个约 1.1GB,合计 298GB
  • loopforge_mysql-data 卷从正常膨胀到 298GB,而库内最大的表仅 0.16GB(run_workspace_checkpoint_chunks)——数据本身很小,全是 binlog
  • 宿主根分区被吃到 86%,系统级卡顿

Reproduction

复现

非确定性触发:正常使用 agent workspace(长时间运行、多轮会话)后出现。触发后该 run/容器的 checkpoint 持续循环,不会自行停止。

期望

  1. 排查 SaveWorkspaceCheckpointStream → ValidateWorkspaceCheckpointFrom(RestoreWorkspaceCheckpointFrom) 为什么会反复触发不收敛(怀疑某个失败/重试路径未终止)
  2. 默认 compose.yaml 对 MySQL 增加写放大防护:开发工具无复制场景建议 --skip-log-bin,或至少设置 binlog_expire_logs_seconds / max_binlog_size

临时缓解(供其他用户参考)

PURGE BINARY LOGS BEFORE NOW();
SET PERSIST binlog_expire_logs_seconds = 86400;

停掉 platform 容器后写入风暴立即消失;MySQL 数据表本身未受影响。

Sanitized diagnostics

Safety checks

  • I removed credentials, private endpoints, internal identifiers, and personal data.
  • I searched existing issues for duplicates.

Activity

  1. cocofhu00 commented on Nov 20, 2026

    @cocofhu00
    Contributor

    感谢反馈。

    我的理解是:自托管的 LoopForge 0.4.0 在使用 agent workspace 之后,checkpoint 会反复进入「保存 → 校验 → 恢复」且停不下来,platform 持续大量写入,并连带把 MySQL binlog 撑到大约 298GB。

    已经看到这条 Issue,会按现有 Issue 排期查看。

  2. wcy12378 commented on Nov 20, 2026

    @wcy12378

    I’m investigating the checkpoint retry path in EndConversationWorkspaceUseAsync and will work on a narrow fix if a regression reproduces it: bound repeated save attempts while preserving recovery from a temporarily busy runner gate, normal deferred retry, and foreground lease supersession.

    The 100 ms retry is a lead, not yet proof of the reported binlog volume. I’ll trace where compression/chunk writes occur and include failure/recovery tests before proposing a change. This does not claim the separate Compose/MySQL binlog configuration request or the entire issue.

  3. linus-liu-web commented on Nov 21, 2026

    @linus-liu-web

    Expectation 2 from this issue — bound (or disable) binary logging in the default Compose stack — is now a PR: #1196. It starts the bundled mysql:8.4 with --binlog-expire-logs-seconds=86400 and --max-binlog-size=67108864 so a write storm cannot fill loopforge_mysql-data with binlogs, documents raising/removing the flags for point-in-time recovery or replication (and --skip-log-bin for single-node evaluation), and documents the PURGE BINARY LOGS + SET PERSIST binlog_expire_logs_seconds recovery for existing stacks.

    A local data point from a Compose deployment: on v0.4.1-rc.1 the volume accumulated 1.18 GB of binlogs in about a day while the database tables stayed under 40 MB — not the 1.2–9.7 MB/s storm reported above, but the same unbounded-log shape. Purged it with the mitigation from this issue (FLUSH BINARY LOGS, PURGE BINARY LOGS TO <current>, SET PERSIST binlog_expire_logs_seconds=259200) and loopforge_mysql-data dropped from 1.53 GB to 0.36 GB.

    The engine-side retry/lease work is separate (#1148 explicitly notes it does not change MySQL/Compose configuration).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions