Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .github/AGENT_OPERATIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,6 @@ Write natural technical Chinese used by ML infrastructure engineers. Preserve mo

A PR sweep requires exactly one primary label:

- `sweep-enabled`: trim every parallelism configuration to its lowest concurrency. Use for most lightweight validation.
- `full-sweep-fail-fast`: canary-gated full sweep with matrix-scoped fail-fast. Recommended for image bumps, recipe changes, bring-up, and other full sweeps.
- `full-sweep-enabled`: canary-gated full sweep without fail-fast. Use when a flaky job must not cancel its matrix's in-flight work.
- `full-sweep-fail-fast-no-canary`: full, matrix-scoped fail-fast without the canary. Use when the canary is flaky or unrepresentative.
Expand Down
3 changes: 1 addition & 2 deletions .github/workflows/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -219,8 +219,7 @@ selects the latest successful eligible run automatically; bare `/use` is rejecte
Both names share authorization, validation, and reactions.

Source validation checks identity and artifacts, not full-matrix coverage.
A successful `sweep-enabled` trim sweep can also be selected automatically;
reusing it publishes only its recorded points on `main`. Acceptance does not
Acceptance does not
certify a green full sweep. Verify coverage and pin the run ID when a full sweep
is required by the review process.

Expand Down
17 changes: 1 addition & 16 deletions .github/workflows/run-sweep.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@ concurrency:
sweep-${{ github.event.pull_request.number || github.sha }}-${{
github.event_name == 'pull_request' &&
(github.event.action == 'labeled' || github.event.action == 'unlabeled') &&
github.event.label.name != 'sweep-enabled' &&
github.event.label.name != 'full-sweep-enabled' &&
github.event.label.name != 'non-canary-full-sweep-enabled' &&
github.event.label.name != 'full-sweep-fail-fast' &&
Expand Down Expand Up @@ -58,7 +57,6 @@ jobs:
github.event.pull_request.head.repo.full_name == github.repository &&
(
(github.event.action != 'labeled' && github.event.action != 'unlabeled') ||
github.event.label.name == 'sweep-enabled' ||
github.event.label.name == 'full-sweep-enabled' ||
github.event.label.name == 'non-canary-full-sweep-enabled' ||
github.event.label.name == 'full-sweep-fail-fast' ||
Expand All @@ -83,7 +81,6 @@ jobs:
[
.[] |
select(
. == "sweep-enabled" or
. == "full-sweep-enabled" or
. == "non-canary-full-sweep-enabled" or
. == "full-sweep-fail-fast" or
Expand Down Expand Up @@ -209,15 +206,13 @@ jobs:
github.event.pull_request.head.repo.full_name == github.repository &&
needs.check-changelog.outputs.skip-pr-sweep != 'true' &&
(
contains(github.event.pull_request.labels.*.name, 'sweep-enabled') ||
contains(github.event.pull_request.labels.*.name, 'full-sweep-enabled') ||
contains(github.event.pull_request.labels.*.name, 'non-canary-full-sweep-enabled') ||
contains(github.event.pull_request.labels.*.name, 'full-sweep-fail-fast') ||
contains(github.event.pull_request.labels.*.name, 'full-sweep-fail-fast-no-canary')
) &&
(
(github.event.action != 'labeled' && github.event.action != 'unlabeled') ||
github.event.label.name == 'sweep-enabled' ||
github.event.label.name == 'full-sweep-enabled' ||
github.event.label.name == 'non-canary-full-sweep-enabled' ||
github.event.label.name == 'full-sweep-fail-fast' ||
Expand Down Expand Up @@ -344,11 +339,6 @@ jobs:
GH_TOKEN: ${{ github.token }}
PR_LABELS: ${{ toJson(github.event.pull_request.labels.*.name) }}
PRIORITY_CRITERIA: ${{ steps.priority-criteria.outputs.criteria || '' }}
TRIM_CONC: >-
${{
github.event_name == 'pull_request' &&
contains(github.event.pull_request.labels.*.name, 'sweep-enabled')
}}
ALL_EVALS: >-
${{
github.event_name == 'pull_request' &&
Expand Down Expand Up @@ -377,9 +367,6 @@ jobs:
--base-ref "$BASE_REF"
--head-ref "$HEAD_REF"
)
if [ "$TRIM_CONC" = "true" ]; then
CMD+=(--trim-conc)
fi
if [ "$ALL_EVALS" = "true" ]; then
CMD+=(--all-evals)
fi
Expand Down Expand Up @@ -935,7 +922,7 @@ jobs:
matrix = json.loads(os.environ['SWEEP_MATRIX'])
labels = set(json.loads(os.environ['SWEEP_LABELS']) or [])
sweep_labels = labels & {
'sweep-enabled', 'full-sweep-enabled', 'non-canary-full-sweep-enabled',
'full-sweep-enabled', 'non-canary-full-sweep-enabled',
'full-sweep-fail-fast', 'full-sweep-fail-fast-no-canary',
'all-evals', 'evals-only', 'agentx-fast',
}
Expand Down Expand Up @@ -1203,15 +1190,13 @@ jobs:
needs.setup.result == 'success' &&
github.event_name == 'pull_request' &&
(
contains(github.event.pull_request.labels.*.name, 'sweep-enabled') ||
contains(github.event.pull_request.labels.*.name, 'full-sweep-enabled') ||
contains(github.event.pull_request.labels.*.name, 'non-canary-full-sweep-enabled') ||
contains(github.event.pull_request.labels.*.name, 'full-sweep-fail-fast') ||
contains(github.event.pull_request.labels.*.name, 'full-sweep-fail-fast-no-canary')
) &&
(
(github.event.action != 'labeled' && github.event.action != 'unlabeled') ||
github.event.label.name == 'sweep-enabled' ||
github.event.label.name == 'full-sweep-enabled' ||
github.event.label.name == 'non-canary-full-sweep-enabled' ||
github.event.label.name == 'full-sweep-fail-fast' ||
Expand Down
5 changes: 2 additions & 3 deletions .github/workflows/trusted-external-sweep.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ jobs:
if: >-
github.event.pull_request.head.repo.full_name != github.repository &&
contains(
fromJson('["sweep-enabled","full-sweep-enabled","non-canary-full-sweep-enabled","full-sweep-fail-fast","full-sweep-fail-fast-no-canary"]'),
fromJson('["full-sweep-enabled","non-canary-full-sweep-enabled","full-sweep-fail-fast","full-sweep-fail-fast-no-canary"]'),
github.event.label.name
)
runs-on: ubuntu-latest
Expand All @@ -37,7 +37,6 @@ jobs:
github-token: ${{ github.token }}
script: |
const primaryLabels = new Set([
'sweep-enabled',
'full-sweep-enabled',
'non-canary-full-sweep-enabled',
'full-sweep-fail-fast',
Expand Down Expand Up @@ -108,7 +107,7 @@ jobs:
'ref': pull.merge_commit_sha,
'changelog-base-ref': pull.base.sha,
'changelog-head-ref': pull.head.sha,
'trim-conc': String(primary[0] === 'sweep-enabled'),
'trim-conc': 'false',
'all-evals': String(labels.includes('all-evals')),
'evals-only': String(labels.includes('evals-only')),
'fail-fast': String(primary[0].includes('fail-fast')),
Expand Down
7 changes: 3 additions & 4 deletions inferencex-e2e/docs/ci-procedures.md
Original file line number Diff line number Diff line change
Expand Up @@ -236,7 +236,6 @@ The same-repository check applies before checking out PR code in changelog valid

| Primary label | Matrix scope | Canary | Matrix fail-fast |
| --- | --- | --- | --- |
| `sweep-enabled` | Changelog matrix trimmed to the minimum concurrency per configuration | No | No |
| `full-sweep-fail-fast` | Full changelog matrix | Yes | Yes. Recommended full-sweep default |
| `full-sweep-enabled` | Full changelog matrix | Yes | No. Use when every matrix point must continue |
| `full-sweep-fail-fast-no-canary` | Full changelog matrix | No | Yes |
Expand All @@ -256,7 +255,7 @@ Changing a recognized primary or modifier label shares the active sweep concurre

Canary and fail-fast solve different problems:

1. A canary is created only for `full-sweep-enabled` or `full-sweep-fail-fast` PRs. No-canary labels and `sweep-enabled` skip it.
1. A canary is created only for `full-sweep-enabled` or `full-sweep-fail-fast` PRs. No-canary labels skip it.
2. Canary selection first considers single-node fixed-sequence `1k1k` and `8k1k` entries and single-node AgentX entries. If none are eligible, it considers multi-node AgentX entries. It excludes eval entries, chooses the lowest-concurrency candidate, runs it with the matching single-node or multi-node workflow, and removes it from the later matrix.
3. If there is no eligible candidate, the canary is skipped. Otherwise all benchmark/eval matrices require the canary to succeed. A failed canary prevents their fan-out.
4. `full-sweep-fail-fast` and `full-sweep-fail-fast-no-canary` set `strategy.fail-fast: true` separately on each matrix job family. The first failing point cancels queued/in-progress siblings in that matrix family. It is not one global kill switch for every independent family.
Expand Down Expand Up @@ -373,7 +372,7 @@ Each request, including all its pages, has a 60-second timeout.
A request is stageable only when all of the following hold:

- The commenter has `write`, `maintain`, or `admin` repository permission.
- The PR currently has one of the four full-sweep labels. `sweep-enabled` is not enough.
- The PR currently has one of the four full-sweep labels (`full-sweep-enabled`, `non-canary-full-sweep-enabled`, `full-sweep-fail-fast`, or `full-sweep-fail-fast-no-canary`).
- The candidate is a completed `pull_request` run of `run-sweep.yml`, created while a full-sweep label was active, with conclusion `success`, `failure`, or `cancelled`.
- The candidate is associated with the PR under the workflow's current-head/historical-pin rules.
- Unexpired `changelog-metadata` and at least one of `results_bmk`, `eval_results_all`, or `bmk_agentic_*` exist. Failed/cancelled runs may therefore stage useful partial data, but empty or metadata-only runs cannot.
Expand Down Expand Up @@ -403,7 +402,7 @@ Reuse prevents an approved full PR sweep from being rerun on `main`. It is not a
4. An `OWNER`, `MEMBER`, or `COLLABORATOR` authorizes reuse with `/use <run_id>`. Keep the command and required run ID on one line. The legacy `/reuse-sweep-run <run_id>` remains equivalent; bare `/reuse-sweep-run` selects automatically. Both names share authorization, validation, and reactions. The newest authorized matching command across both names wins.
5. Unpinned selection requires the latest eligible source run to be successful. A pinned run is an explicit maintainer decision and may have conclusion `success`, `failure`, or `cancelled`. Downstream ingestion keeps only available/valid rows, so report it as partial rather than green.

Reuse validation checks source identity and available artifacts, not full-matrix coverage. A successful `sweep-enabled` (trimmed) source is eligible, including for automatic selection, and publishes only its recorded points on `main`. Acceptance does not certify a green full sweep or satisfy that review requirement. To reuse a full sweep specifically, verify its coverage and pin its run ID.
Reuse validation checks source identity and available artifacts, not full-matrix coverage. Acceptance does not certify a green full sweep or satisfy that review requirement. To reuse a full sweep specifically, verify its coverage and pin its run ID.

The comment starts a lightweight validation workflow using default-branch code and `GITHUB_TOKEN`. It adds 👍 to the original comment when accepted, or 👎 when rejected; the Actions run summary explains a rejection. It posts no separate comment and starts no GPU work. Editing the command clears the bot's old reaction and checks the new request. Human reactions are preserved; the newest authorized command still takes precedence.

Expand Down
7 changes: 3 additions & 4 deletions inferencex-e2e/docs/ci-procedures_zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -228,7 +228,6 @@ B200 Kimi 配方采用 DCP8,且关闭 Mooncake Offload。Master Config 记录

| 主标签 | 矩阵范围 | Canary | 矩阵 Fail-fast |
| --- | --- | --- | --- |
| `sweep-enabled` | Changelog 矩阵裁剪为每个配置的最低并发 | 无 | 无 |
| `full-sweep-fail-fast` | 完整 Changelog 矩阵 | 有 | 有;推荐的完整扫描默认值 |
| `full-sweep-enabled` | 完整 Changelog 矩阵 | 有 | 无;需要每个矩阵点继续运行时使用 |
| `full-sweep-fail-fast-no-canary` | 完整 Changelog 矩阵 | 无 | 有 |
Expand All @@ -248,7 +247,7 @@ B200 Kimi 配方采用 DCP8,且关闭 Mooncake Offload。Master Config 记录

Canary 和 Fail-fast 解决不同问题:

1. 只有使用 `full-sweep-enabled` 或 `full-sweep-fail-fast` 的 PR 才创建 Canary。No-canary 标签和 `sweep-enabled` 会跳过它。
1. 只有使用 `full-sweep-enabled` 或 `full-sweep-fail-fast` 的 PR 才创建 Canary。No-canary 标签会跳过它。
2. Canary 首先检查单节点固定序列 `1k1k`、`8k1k` 和单节点 AgentX 条目;若没有合格条目,再检查多节点 AgentX 条目。它排除 Eval 条目,选取最低并发候选,使用对应的单节点或多节点工作流运行,并从后续矩阵移除该条目。
3. 如果没有合格候选,Canary 会被跳过。否则所有 Benchmark/Eval 矩阵都要求 Canary 成功;Canary 失败会阻止其扇出。
4. `full-sweep-fail-fast` 与 `full-sweep-fail-fast-no-canary` 会分别为每个矩阵 Job Family 设置 `strategy.fail-fast: true`。首个失败点会取消同一矩阵 Family 中排队或运行中的兄弟项;它不是跨所有独立 Family 的全局 Kill Switch。
Expand Down Expand Up @@ -355,7 +354,7 @@ Klaud 和恢复工具继续使用现有的 `gh` 认证。GitHub CLI 跟随分页
请求只有在全部满足下列条件时才可暂存:

- 评论者具有仓库 `write`、`maintain` 或 `admin` 权限。
- PR 当前具有四个完整扫描标签之一;`sweep-enabled` 不够。
- PR 当前具有四个完整扫描标签之一(`full-sweep-enabled`、`non-canary-full-sweep-enabled`、`full-sweep-fail-fast` 或 `full-sweep-fail-fast-no-canary`)。
- 候选是已结束的 PR `run-sweep.yml` Run,创建时完整扫描标签处于活动状态,结论为 `success`、`failure` 或 `cancelled`。
- 候选按照 Workflow 当前 Head/历史 Pin 规则与该 PR 关联。
- 存在未过期的 `changelog-metadata`,并且至少存在 `results_bmk`、`eval_results_all` 或 `bmk_agentic_*` 之一。因此失败/取消的 Run 可以暂存有用的部分数据,但空 Run 或仅有 Metadata 的 Run 不行。
Expand Down Expand Up @@ -385,7 +384,7 @@ Klaud 和恢复工具继续使用现有的 `gh` 认证。GitHub CLI 跟随分页
4. `OWNER`、`MEMBER` 或 `COLLABORATOR` 通过 `/use <run_id>` 授权复用。必须提供 Run ID,并与命令放在同一行。原有的 `/reuse-sweep-run <run_id>` 仍然等效;不带 ID 的 `/reuse-sweep-run` 会自动选择源 Run。两种命令使用相同的授权、验证和表态规则,并以两者中最新的合格授权命令为准。
5. 不指定 ID 时,自动选择要求最新的合格源 Run 成功。指定 Run 是维护者的明确决定,允许结论为 `success`、`failure` 或 `cancelled`;下游入库只保留存在且有效的行,因此应将其报告为部分数据,而不是绿色 Run。

复用验证检查源 Run 的身份和可用产物,不检查完整矩阵覆盖范围。成功的 `sweep-enabled`(裁剪扫描)源 Run 也可复用,包括自动选择;在 `main` 上只会发布该 Run 已记录的数据点。请求被接受不代表已通过完整扫描,也不能代替评审中的完整扫描要求。如需复用某次完整扫描,请先确认其覆盖范围,再固定该 Run ID。
复用验证检查源 Run 的身份和可用产物,不检查完整矩阵覆盖范围。请求被接受不代表已通过完整扫描,也不能代替评审中的完整扫描要求。如需复用某次完整扫描,请先确认其覆盖范围,再固定该 Run ID。

评论会触发轻量验证工作流,使用默认分支代码和 `GITHUB_TOKEN`。接受后在原评论上添加 👍,拒绝时添加 👎;拒绝原因显示在 Actions 运行摘要中。不发布额外评论,也不启动 GPU 工作。编辑命令时会清除机器人的旧表态并检查新请求。用户的表态保持不变,仍以最新的合格授权命令为准。

Expand Down
4 changes: 2 additions & 2 deletions inferencex-e2e/docs/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ Tests runs suites under `infx/tests/`, `utils/`, `runners/`, `../collectivex/tes
| Schema and matrix | A config key validates and emits the intended matrix fields | Runner availability, server startup, or performance |
| Focused Python tests | Changed generator, changelog, result, eval, collection, or reuse contracts behave on covered inputs | Container, accelerator, network, or Slurm behavior |
| Smoke run | One tightly filtered path allocates, starts a server, runs a workload, and emits artifacts | The complete concurrency/search space or merge eligibility |
| Trimmed PR sweep | Each selected single-node group runs its lowest concurrency (`sweep-enabled`) | Intermediate concurrency points required by a full sweep |
| Trimmed PR sweep | Each selected single-node group runs its lowest concurrency (manual `--trim-conc` dispatch) | Intermediate concurrency points required by a full sweep |
| Full sweep and eval | The selected untrimmed matrix and eval jobs execute on the reviewed commit | Correctness of evidence that was not inspected, or unrelated configurations |

A green later layer does not erase missing earlier evidence. For example, a green collector can aggregate an empty set, so review must inspect the underlying executed jobs and artifacts.
Expand Down Expand Up @@ -188,10 +188,10 @@ A smoke run is not merge evidence: it intentionally omits configurations and con

### Trimmed and full sweeps

- `sweep-enabled` trims each parallelism group to its lowest concurrency and is the default for most PR feedback.
- `full-sweep-fail-fast` is the recommended full-sweep label. It uses the sequential single-node canary and stops each matrix after that matrix's first failure while preserving completed results.
- Use a no-canary full-sweep label only when the canary is known to be flaky or unrepresentative. Use `full-sweep-enabled` instead of fail-fast only when every matrix job must continue despite a failure.
- Apply exactly one primary sweep label. Modifier-only or conflicting primary labels do not constitute a valid sweep.
- A trimmed sweep (lowest concurrency only) can be run manually via `e2e-tests.yml` with the `trim-conc` input.

The current meanings and eligibility rules are defined in the [sweep-label reference](../../.github/AGENT_OPERATIONS.md#sweep-labels-and-reuse) and implemented by [`run-sweep.yml`](../../.github/workflows/run-sweep.yml).

Expand Down
Loading
Loading