From 92b68766ec5da3f62322d26b205d40fb356dd568 Mon Sep 17 00:00:00 2001 From: king Date: Wed, 14 Oct 2026 19:24:04 +0800 Subject: [PATCH] perf(ci): bring every PR job under 10 minutes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 上一轮把 ci_cross2eth(37m->8m50s)、ci_bridgevmxgo(17m->7m16s) 压下来之后, 还剩三个 job 超过 10 分钟(PR #1330 实测): coverage 17m25s、unit-test 13m10s、 Test GOARCH 386 10m01s; ci_evm(9m37s)/ci_paracross(9m33s) 也几乎没有余量。 unit-test 已由 #1328 分 4 片解决, 本 commit 解决剩下的部分。 1) coverage 17m25s -> 实测 4m58s build/tools/coverage.sh 原来用 bash for 循环**逐包串行** go test: 126 个有测试的包, 单包最慢约 2 分钟(oracle/rpc 112s、relayd 91s、lightclient btc 91s、dpos 89s、 ticket/executor 60s), 串行合计 991s。go test 从 1.20 起支持一次对多个包生成合并的 覆盖率文件, 并且自己按 -p 并行跑包, 改成一次调用后墙钟时间降到最慢包的量级。 覆盖率语义不变(仍是 -covermode=count 的合并 profile, 文件名 coverage.cov 不变)。 2) Test GOARCH 386 10m01s -> 用 build_only Build 步骤 147s 里约 72s 是 `make build` 的 depends —— 串行编译几十个 dapp 的 cmd 产物, 而这个 job 只需要验证 386 下能编译、能跑测试。新增 `make build_only` (只编 chain33/chain33-cli/fork-config), 386 job 改用它; `build` 保持原语义 (build: depends build_only), 只是把编译部分抽出来复用。 3) 定点构建: ci_evm / ci_rgbx / ci_mix / ci_relay / ci_parachain_rollup 各约省 40s 这些 workflow 原来要么只跑 `make docker-compose dapp=X`(由 docker-compose.sh 的 `if ! [ -d ci ]` 兜底触发整个 make), 要么直接 `make build_ci`, 都会 depends 编译所有 dapp。统一改成上一轮引入的 `make build_ci_dapp dapp=X`。 **ci_paracross 不能这么改**: 它会走 rpc_test -> dapp_test_rpc, 后者固定执行 ./ticket/test-rpc.sh 以及 build/ci/dapptest 下所有 dapp 的 test-rpc.sh, 仍然需要 depends 的完整产物; 只编 paracross 会因缺少 ticket/test-rpc.sh 而失败(已在 CI 复现, workflow 里已写明原因)。 4) depends 并行化 + 修掉两个一直静默失败的 build.sh 每个 dapp 的 cmd/build.sh 只写自己的 build/ci/ 与 build/ci/dapptest/, 互不重叠, 把 `find -exec ... \;` 换成 `xargs -P4`(DEPENDS_JOBS 可覆盖), 对仍需完整 depends 的 ci_paracross/ci_base/check_fmt/autotest 各约省 30~40s。 注意 find 的 `-exec` 失败只让谓词为假、find 本身仍返回 0, 所以 evmxgo/lightclient 的 build.sh 里 `cp ./build/*`(这两个 dapp 根本没有 build 目录)失败了很多年都没人发现; 换成 xargs 后失败会透出, 已把这两处改成显式判断目录存在, 让 depends 能对真正的错误 fail-fast。 验证方式与上一轮相同: 全部改动由 CI 实跑确认每个 job <10 分钟且测试全绿。 --- .github/workflows/build.yml | 3 ++- .github/workflows/build_evm.yml | 1 + .github/workflows/build_mix.yml | 2 +- .github/workflows/build_paracross.yml | 4 ++++ .github/workflows/build_relay.yml | 2 +- .github/workflows/build_rgbx.yml | 1 + .github/workflows/ci_parachain_rollup.yml | 1 + Makefile | 14 +++++++++++--- build/tools/coverage.sh | 23 ++++++++--------------- plugin/dapp/evmxgo/cmd/build.sh | 6 +++++- plugin/dapp/lightclient/cmd/build.sh | 6 +++++- 11 files changed, 40 insertions(+), 23 deletions(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index ad57b220b4..a4cc00f559 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -148,7 +148,8 @@ jobs: sudo apt-get install --fix-missing gcc-multilib - name: Build - run: make build + # 只需要验证 386 下能编译: build_only 跳过 depends 里几十个 dapp cmd 产物的编译(约省 70s) + run: make build_only - name: Run unit tests run: go test ./... -covermode=atomic diff --git a/.github/workflows/build_evm.yml b/.github/workflows/build_evm.yml index 47b6fa2825..674ab99b24 100644 --- a/.github/workflows/build_evm.yml +++ b/.github/workflows/build_evm.yml @@ -15,5 +15,6 @@ jobs: go-version-file: 'go.mod' - name: deploy run: | + make build_ci_dapp dapp=evm make docker-compose dapp=evm make docker-compose-down dapp=evm \ No newline at end of file diff --git a/.github/workflows/build_mix.yml b/.github/workflows/build_mix.yml index e5a124b4fd..fb79eadfb0 100644 --- a/.github/workflows/build_mix.yml +++ b/.github/workflows/build_mix.yml @@ -28,6 +28,6 @@ jobs: cd - - name: deploy run: | - make build_ci + make build_ci_dapp dapp=mix make docker-compose dapp=mix make docker-compose-down dapp=mix diff --git a/.github/workflows/build_paracross.yml b/.github/workflows/build_paracross.yml index 8b6b4bec20..d04e4bbe80 100644 --- a/.github/workflows/build_paracross.yml +++ b/.github/workflows/build_paracross.yml @@ -20,6 +20,10 @@ jobs: run: export PATH=${PATH}:`go env GOPATH`/bin - name: deploy + # 注意: 不能换成 build_ci_dapp。paracross 的 docker-compose.sh 会走 rpc_test -> + # dapp_test_rpc, 后者固定执行 ./ticket/test-rpc.sh 以及 build/ci/dapptest 下所有 + # dapp 的 test-rpc.sh(package 脚本由各 dapp 的 cmd/build.sh 生成), 因此仍然需要 + # depends 的完整 build/ci 产物; 只编 paracross 会导致 ticket/test-rpc.sh 缺失而失败。 run: | make docker-compose dapp=paracross make docker-compose-down dapp=paracross diff --git a/.github/workflows/build_relay.yml b/.github/workflows/build_relay.yml index 2b43646c98..d067662c38 100644 --- a/.github/workflows/build_relay.yml +++ b/.github/workflows/build_relay.yml @@ -34,7 +34,7 @@ jobs: - name: deploy run: | - make build_ci + make build_ci_dapp dapp=relay make docker-compose dapp=relay - name: cleanup diff --git a/.github/workflows/build_rgbx.yml b/.github/workflows/build_rgbx.yml index 781f6860f7..85a9a6bcba 100644 --- a/.github/workflows/build_rgbx.yml +++ b/.github/workflows/build_rgbx.yml @@ -19,6 +19,7 @@ jobs: id: go - name: deploy run: | + make build_ci_dapp dapp=rgbx make docker-compose dapp=rgbx - name: cleanup if: always() diff --git a/.github/workflows/ci_parachain_rollup.yml b/.github/workflows/ci_parachain_rollup.yml index 37ae8d1d01..f7dbfc9282 100644 --- a/.github/workflows/ci_parachain_rollup.yml +++ b/.github/workflows/ci_parachain_rollup.yml @@ -15,5 +15,6 @@ jobs: go-version-file: 'go.mod' - name: deploy run: | + make build_ci_dapp dapp=rollup make docker-compose dapp=rollup make docker-compose-down dapp=rollup diff --git a/Makefile b/Makefile index 62a39b2515..595b22b681 100644 --- a/Makefile +++ b/Makefile @@ -19,14 +19,20 @@ BUILD_FLAGS := -ldflags '-X "github.com/33cn/plugin/version.GitCommit=$(GitCommi MKPATH=$(abspath $(lastword $(MAKEFILE_LIST))) MKDIR=$(dir $(MKPATH)) proj := "build" -.PHONY: default dep all build release cli linter race test fmt vet bench msan coverage coverhtml docker docker-compose protobuf clean help autotest +.PHONY: default dep all build build_only release cli linter race test fmt vet bench msan coverage coverhtml docker docker-compose protobuf clean help autotest default: depends build -build: depends +build: depends build_only + +# 只编 chain33/chain33-cli/fork-config, 不跑 depends(不生成 build/ci 下各 dapp 的集成测试产物)。 +# 用于只需要编译产物、不做 docker 集成测试的场景, 例如 GOARCH 386 的构建检查: +# depends 会串行编译几十个 dapp 的 cmd 产物, 对这类 job 是纯开销。 +build_only: ## Build chain33 binaries without dapp cmd artifacts (no depends) go build $(BUILD_FLAGS) -v -o $(APP) go build $(BUILD_FLAGS) -v -o $(CLI) $(SRC_CLI) go build $(BUILD_FLAGS) -v -o build/fork-config github.com/33cn/plugin/cli/fork_config/ + @mkdir -p build/ci/paracross @cp chain33.toml build/ @cp chain33.para.toml build/ci/paracross/ @@ -236,7 +242,9 @@ protobuf: ## Generate protbuf file of types package @find ./plugin/dapp -maxdepth 2 -type d -name proto -exec make -C {} \; depends: ## Generate depends file of types package - @find ./plugin/dapp -maxdepth 2 -print -type d -name cmd -exec make -C {} OUT="$(MKDIR)build/ci" FLAG= \; + @# 各 dapp 的 cmd/build.sh 只写自己的 build/ci/ 和 build/ci/dapptest/, 互不重叠, + @# 可以并行; 原来 -exec ... \; 是串行, 几十个 dapp 合起来要 50~70s。 + @find ./plugin/dapp -maxdepth 2 -type d -name cmd | xargs -P "$${DEPENDS_JOBS:-4}" -I{} make -C {} OUT="$(MKDIR)build/ci" FLAG= help: ## Display this help screen diff --git a/build/tools/coverage.sh b/build/tools/coverage.sh index 8b6e54f519..7e8674efcc 100755 --- a/build/tools/coverage.sh +++ b/build/tools/coverage.sh @@ -3,21 +3,17 @@ # Code coverage generation set -e -o pipefail -COVERAGE_DIR="${COVERAGE_DIR:-build/coverage}" +# go test 从 1.20 起支持一次对多个包生成**合并**的覆盖率文件, 并且会自己按 -p 并行跑包。 +# 原来用 bash for 循环逐包串行执行: 全量 126 个有测试的包, 单个最慢约 2 分钟, +# 串行合计约 16 分钟(CI 实测 ut coverage 步骤 991s)。改成一次调用后, +# 墙钟时间降到"最慢的包 + 排队"的量级。 +JOBS="${COVERAGE_JOBS:-$(nproc 2>/dev/null || echo 4)}" PKG_LIST=$(go list ./... | grep -v "vendor" | grep -v "chain33/test" | grep -v "mock" | grep -v "mocks") -# Create the coverage files directory -mkdir -p "$COVERAGE_DIR" - -# Create a coverage file for each package -for package in ${PKG_LIST}; do - go test -parallel=8 -covermode=count -coverprofile "${COVERAGE_DIR}/${package##*/}.cov" "$package" -done - -# Merge the coverage profile files -echo 'mode: count' >./coverage.cov -tail -q -n +2 "${COVERAGE_DIR}"/*.cov >>./coverage.cov +# PKG_LIST 需要按空白拆分成多个参数, 这里刻意不加引号 +# shellcheck disable=SC2086 +go test -p "${JOBS}" -parallel=8 -covermode=count -coverprofile=./coverage.cov ${PKG_LIST} # Display the global code coverage go tool cover -func=./coverage.cov @@ -26,6 +22,3 @@ go tool cover -func=./coverage.cov if [ "$1" == "html" ]; then go tool cover -html=./coverage.cov -o coverage.html fi - -# Remove the coverage files directory -rm -rf "$COVERAGE_DIR" diff --git a/plugin/dapp/evmxgo/cmd/build.sh b/plugin/dapp/evmxgo/cmd/build.sh index cbcf75fe1f..73a48eead6 100644 --- a/plugin/dapp/evmxgo/cmd/build.sh +++ b/plugin/dapp/evmxgo/cmd/build.sh @@ -8,4 +8,8 @@ OUT_DIR="${1}/$strapp" #FLAG=$2 mkdir -p "${OUT_DIR}" -cp ./build/* "${OUT_DIR}" +# 本 dapp 没有 ./build 目录(历史遗留), 原来的 cp 必然失败; depends 以前用 +# `find -exec` 会静默吞掉失败, 现在 depends 并行化并用 xargs 透出失败, 这里显式判断 +if [ -d ./build ]; then + cp ./build/* "${OUT_DIR}" +fi diff --git a/plugin/dapp/lightclient/cmd/build.sh b/plugin/dapp/lightclient/cmd/build.sh index cbcf75fe1f..73a48eead6 100644 --- a/plugin/dapp/lightclient/cmd/build.sh +++ b/plugin/dapp/lightclient/cmd/build.sh @@ -8,4 +8,8 @@ OUT_DIR="${1}/$strapp" #FLAG=$2 mkdir -p "${OUT_DIR}" -cp ./build/* "${OUT_DIR}" +# 本 dapp 没有 ./build 目录(历史遗留), 原来的 cp 必然失败; depends 以前用 +# `find -exec` 会静默吞掉失败, 现在 depends 并行化并用 xargs 透出失败, 这里显式判断 +if [ -d ./build ]; then + cp ./build/* "${OUT_DIR}" +fi