Clarify multi-threading behaviour

[grantmcdermott]

Hi again, Bob.

I was looking through the website documentation and was a bit confused by this footnote in the Benchmarks section:

Threads. Every engine gets the same 8 vCPU. Ibex parallelises some operators by default, so ibex is not a single-core number; ibex-st is the same build run with IBEX_PARALLEL=0, which makes ibex vs ibex-st Ibex's own threading gain. polars-st, duckdb-st, datafusion-st and clickhouse-st are the equivalent single-thread runs of their engines, so ibex-st against those is the per-core comparison.

Is this paragraph correct? For one thing, I don’t see a ibex-st column. For another, the rest of the docs emphasise the single-threaded nature of Ibex. Does the IBEX_PARALLEL=1 build flag really enable multi-threading?

Cheers.

[bobjansen]

Hi,

The website text and actual implementation sometimes decouple, like you saw. I just ran the benchmark again with Ibex and Ibex-st, and updated the website. The results are pretty good (they deserve scrutiny before I close this) and some further caveats:

• There will be bugs
• MT Support is not finished, some operations are still fully single threaded
• I only tested and benched on an 8 core machine, higher core counts are unexplored
• RNG is a performance regression, still fast but it should scale very well

[grantmcdermott]

Thanks Bob, appreciate the follow-up and clarification.

Unfortunately, I won't be able to test myself for a while. I'm away from my computer and only perusing the documentation ATM. But I'm keen to kick the tyres with some standing benchmarks that I keep for myself, once I'm back at my desk.

[bobjansen]

Amazing, I'm curious how you will like it. I see you are an R-user, I just updated support for the Ibex R package. It runs well in RStudio, and not unimportant, since today doesn't hang your session :) A nycflights based example goes through some examples. Based on equivalent R code in the same directory. The only change I remember making is replacing slider with native Ibex verbs (as_timeframe and window to replace arrange followed by slide_index_dbl).

I think I will do some first analyses in R with Ibex myself this week.