Package Mooncake H100 transfer tracing experiment / 打包 Mooncake H100 传输跟踪实验

[kimbochen]

Description

Package the Mooncake TENT transfer-event recorder and two-node H100 experiment under experimental/mooncake-transfer-traces/ so the source changes and measurements can be reviewed and reproduced together.

• Three patches reproduce all 17 changed Mooncake files against fe0d23e332f41c7d54b5c217194dce5f8e751b21, including optional submission/completion recording and compute-node Pyxis/Slurm launchers.
• English/Chinese READMEs document the results, limitations, patch application and reproduction commands. Compact artifacts include baseline/traced summaries, plots, sampled request spans, provenance, checksums and validation logs.
• The full H100 sweep completed 90 cases. At 64 MiB it measured 41.751 GB/s read and 40.152 GB/s write, but the recorder dropped 4.50% of events. The package explicitly documents that these traces cannot support lossless replay.
• The 16.39 GiB raw traces, build binaries and container image remain external. Their original locations are recorded in the manifest.

This is an experimental archive and analysis package. It does not change active InferenceX serving benchmarks, runner registration, recipes or published results; no performance-changelog entry is needed for those paths.

Validation

• All three patches applied to the pinned base files; all 17 resulting file hashes match the tested Mooncake working tree.
• Packaged analysis CLIs exercised on controlled complete/incomplete trace fixtures; expected matched, orphaned and wholly missing IDs and sampled spans verified.
• Ruff lint/format, package checksums and README link/command parity checks passed.
• Existing runtime evidence is included: 64 Mooncake tests passed; the short two-H100 smoke retained 16,442 matched pairs with zero drops; the full sweep completed with 4.50% event loss. These are prior cluster runs, not an InferenceX PR validation sweep.

AI model disclosure

• Model/version: The exact runtime model/version identifier was not exposed and could not be verified.
• Role: Codex prepared the Mooncake implementation, experiments, analysis, package, documentation and this PR. No sub-agents were used.

Related Issue

No issue is closed by this PR. Related work: #2510 implements a CollectiveX serving-shaped KV handoff benchmark. This package instead archives the TENT event-recorder instrumentation and a single-stream synthetic tebench sweep; it does not replace or extend that suite.

Type of Change

• Documentation update
• Other: experimental source-patch/result package and offline analysis utilities

Checklist

• I have completed the AI model disclosure and kept it current
• I have tested my changes locally
• I have updated documentation if necessary
• Performance changelog: not applicable; active InferenceX benchmark/recipe behavior is unchanged
• Full PR sweep, CODEOWNER review and authorized /use <run_id>: not completed; this PR is a draft

中文

改动说明

将 Mooncake TENT 传输事件记录器及双节点 H100 实验打包到 experimental/mooncake-transfer-traces/，便于一起审阅和复现源码修改与测量结果。

• 三个补丁基于 fe0d23e332f41c7d54b5c217194dce5f8e751b21，完整复现 17 个文件的修改，包括可选的提交/完成事件记录，以及计算节点 Pyxis/Slurm 启动脚本。
• 中英文 README 说明结果、限制、补丁应用和复现方法。包内包含跟踪与未跟踪基线汇总、图表、采样请求区间、来源信息、校验和及验证日志。
• 完整 H100 扫描完成了 90 项测试，64 MiB 时读写带宽分别为 41.751 GB/s 和 40.152 GB/s，但记录器丢失了 4.50% 的事件。文档明确说明这些跟踪不能用于无损回放。
• 16.39 GiB 原始跟踪、二进制和容器镜像保留在外部，manifest 记录其原始位置。

这是实验归档和分析包，不修改 InferenceX 活跃推理基准、runner 注册、recipe 或已发布结果，因此不需要为这些路径添加性能变更日志。

验证

所有补丁均成功应用到固定基础版本的文件上，17 个结果文件的哈希与已测试工作目录完全一致。包内分析 CLI 在完整及不完整的受控跟踪样本上验证了配对、单侧事件、完全缺失 ID 和采样区间。Ruff、校验和、README 链接及双语命令一致性检查通过。

包内保留了此前集群运行的证据：64 项 Mooncake 测试通过；双 H100 冒烟测试保留 16,442 对匹配事件且零丢失；完整扫描完成但丢失 4.50% 的事件。这些不是本 PR 的 InferenceX 验证扫描。

AI 模型使用说明

运行环境未公开可核实的精确模型/版本标识，因此无法确认。Codex 完成了 Mooncake 实现、实验、分析、打包、文档和本 PR 的准备工作；未使用子 agent。

相关工作与状态

本 PR 不关闭 issue。#2510 是 CollectiveX 面向推理工作负载的 KV 交接基准；本包归档 TENT 事件插桩和单流合成 tebench 扫描，不替代或扩展该套件。

改动类型为文档、实验补丁/结果归档和离线分析工具。PR 保持草稿状态；尚未完成完整 PR 扫描、CODEOWNER 审阅或授权的 /use <run_id> 流程。

[functionstackx]

Sorry, over the weekend, there was 2 major refactors to clean up the technical debt accumalated over the past 11 months of moving at the speed of light. We don't see any major refactors in the forthseeable future besides cleaning up AMD multinode AgentX pile of bash. As much, due to the refactors, u would need to ask your agent to rebase from remote main@latest. Thank you in advance for ur understanding