Skip to content

Add statistics=false option to skip summary statistics in schema - #353

Open
matthieugomez wants to merge 1 commit into
JuliaStats:masterfrom
matthieugomez:schema-statistics-optout
Open

Add statistics=false option to skip summary statistics in schema#353
matthieugomez wants to merge 1 commit into
JuliaStats:masterfrom
matthieugomez:schema-statistics-optout

Conversation

@matthieugomez

@matthieugomez matthieugomez commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Computing a ContinuousTerm's mean/var/min/max takes several O(n) passes over each column, which can take as long as fitting the model itself (#222). Removing the fields is breaking (#223), so this adds a non-breaking opt-out instead, schema(f, data; statistics=false), which fills them with NaN placeholders.

Here are some performance checks on this branch:

using StatsModels
N = 10_000_000
df = (y = rand(N), x1 = rand(N), x2 = rand(N))
f = @formula(y ~ x1 + x2)

@time schema(f, df)                    # 0.126 s
@time schema(f, df, statistics=false)  # 0.7 µs
@time [df.x1 df.x2] \ df.y             # 0.112 s (the regression itself, for scale)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant