Improve the speed of the CI

Currently the CI takes around 90min to run: this is a bit excessive. Here is what we could do to reduce a bit this number:

Change the parameters for the redis benchmark: SYSMALLOC=1 /__w/mimalloc-bench/mimalloc-bench/extern/redis-6.2.6/src/redis-benchmark -r 1000000 -n 1000000 -q -P 16 lpush a 1 2 3 4 5 lrange a 1 5 takes around a full minute, given that we have ~16 allocators, it adds up rapidly.
- Done in 410f665
Lean takes around 1-3 minutes per allocator, and is unmaintained. Shall we try to move to lean4 ?

So my wondering was about how to split the work up into more parallel tasks.

We could build a Docker image with all the benchmarks, and then we can install that image, and run build and test a subset of the allocators. This would allow us to increase the parallelism, without too much duplication of cost. It would provide a more stable test setup for replication.

I am imagining the allocators change more frequently than the benchmarks for this to make sense. But based on the PRs I think that is correct.

We would still want to test that we can build all the benchmarks, but that could be a separate task, and not have the allocator testing depend on it.

Unfortunately, I don't think that github actions has the concept of artifacts like gitlab does :/

@mjp41

I am kind of in line with @mjp41 here. Currently, our CI is running every benchmark against every allocator, once. This basically means we do a full benchmark run in CI. imho, that is not what CI is for (at least not in this project). For us, it should be sufficient to:

build all benchmarks
build all allocators
run all benchmarks with the system allocator to ensure they built properly
run all allocators against a short benchmark to ensure they built properly

The short benchmark might be cfrac, barnes (is suspiciously constant in the time needed - do they even use malloc?), gs or another one that does use relevant parts of the allocators but does not take too long.
Our current Ubuntu setup takes ~11 minutes to build everything, cfrac runs in under 10 seconds for most allocators, a complete run of all tests (allt) with the system allocator takes about a minute on my machine, so I estimate around 2 minutes on the CI. Given 21 allocators on Ubuntu right now, about 4 minutes of cfrac running has to be added. This brings us to a total of less than 20 minutes with the setup being the biggest block - we can improve that further down the road.

Considering lean4: I'm currently building the release version on a machine quite a lot stronger than CI - I'm 20 minutes in and still not done. So I think that would not be a good idea.

Another detail: mi-dbg and mi2-dbg take quite some longer time (understandable). Do we need them in the CI? It seems they're suited if you would develop mimalloc, but not really suited for benchmarking... I'm not advocating for removing them, but not including them in the CI. They also spam the console pretty badly.

TLDR: I'd suggest the following:

remove mi-dbg and mi2-dbg from the CI
reduce the test runs: once with sys allt, once with alla cfrac
include all tests that can be built successfully in allt
as a separate but connected issue: improve the speed of the setup. maybe through Proposal: port build-bench-env.sh to make #230 , maybe by other means.

So I have been experimenting with some level of caching today. But the structure of build-bench-env.sh has everything all in one. So for instance have a Dockerfile like:

FROM ubuntu:24.04

# Pull mimalloc-bench
RUN  mkdir build
RUN mkdir -p /build/mimalloc-bench
COPY . /build/mimalloc-bench

# Install dependencies
WORKDIR /build/mimalloc-bench
RUN ./build-bench-env.sh packages

# Build benchmarks
RUN ./build-bench-env.sh bench

RUN ./build-bench-env.sh redis

RUN ./build-bench-env.sh rocksdb

RUN ./build-bench-env.sh lean

# Bulid allocator
ARG allocator=mi
RUN ./build-bench-env.sh $allocator

# Run benchmarks
ARG benchs=allt
WORKDIR /build/mimalloc-bench/out/bench
RUN ../../bench.sh $allocator $benchs

Won't nicely cache as COPY will change if we change a single allocator version or a benchmark version. I am currently thinking for snmalloc's CI to build a docker container for the benchmarks, and inject snmalloc built outside the container, or built without build-bench-env, to be the thing that is tested.

@jvoisin

So I've experimented a bit more with Docker to build CI for snmalloc that uses mimalloc-bench.

microsoft/snmalloc#795

This uses Dockers caching to mostly pull the benchmarks from cache. It builds snmalloc from a different source, so changes to snmalloc don't affect the benchmark builds.

I wonder if we can restructure the build in mimalloc-bench to be able to reuse the same kind of caching to improve CI times.

I also used a cut down version of the Python scripts for graphs to generate a json file to submit to bencher.dev. This script is currently in the snmalloc PR but I would be keen to add to mimalloc-bench instead, so others could use it. This would also make copyright/licensing simpler. @jvoisin do you think it makes sense to add here?

@jvoisin

I also used a cut down version of the Python scripts for graphs to generate a json file to submit to bencher.dev. This script is currently in the snmalloc PR but I would be keen to add to mimalloc-bench instead, so others could use it. This would also make copyright/licensing simpler. @jvoisin do you think it makes sense to add here?

Absolutely <3

As an extension of #244 can we do the following?

for every platform: install dependencies
  - for every benchmark: build benchmark
  - for every allocator: build allocator
  - for every pair of alloc, bench: bench alloc against bench

each container needs the platform, each benchmarking container (the ones in the last line) needs alloc and bench containers as base. Also, this Issue is more about speeding up the CI instead of the benchmarks, right? Should we update the title?

This would make a lot of containers (number of benchmarks * number of allocators), no?

yes. even an additional ( * number of platforms) on top. Minus not successfully built allocs or benchmarks, of course.

I am not keen to expand the number much. I already feel I went quite heavy on expanding. I viewed the benchmarks as more stable, but that is perhaps a matter of perspective. I change allocators a lot, and not benchmarks ;-)

Navigation Menu

Repository navigation

Improve the speed of the CI #138

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Improve the speed of the CI #138

Description

Activity

mjp41 commented on Jun 7, 2022

jvoisin commented on Jun 23, 2022

derSteFfi commented on Jul 25, 2025

mjp41 commented on Aug 5, 2025

mjp41 commented on Aug 7, 2025

jvoisin commented on Aug 7, 2025

derSteFfi commented on Sep 29, 2025

jvoisin commented on Sep 29, 2025

derSteFfi commented on Sep 29, 2025

mjp41 commented on Sep 29, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions