Skip to content

BE-930: Add memory, thread and disk limits to the workspace - #10003

Open
thehabbos007 wants to merge 1 commit into
asa/hash-integrations/04a-engine-snapshotsfrom
asa/hash-integrations/04b-engine-limits
Open

thehabbos007 wants to merge 1 commit into
asa/hash-integrations/04a-engine-snapshotsfrom
asa/hash-integrations/04b-engine-limits

Conversation

@thehabbos007

Copy link
Copy Markdown
Contributor

🌟 Purpose

This PR lets a caller limit a workspace's memory, threads and disk use through WorkspaceConfig.

🔍 Changes

  • memory_limit, temp_directory_limit and threads set DuckDB's memory_limit, max_temp_directory_size and threads settings.
  • disk_limit limits the combined size of the database file, its log and DuckDB's temporary directory. The workspace checks it on open and after each query and create_table. When the files are over the limit, it checkpoints the log into the database file and measures again before it fails with DiskLimit.

⚠️ Known issues

  • The check runs after a statement, so one create_table can write past the limit, and the table it created stays in the database.

@thehabbos007 thehabbos007 self-assigned this Oct 12, 2026
@github-actions github-actions Bot added area/libs Relates to first-party libraries/crates/packages (area) type/eng > backend Owned by the @backend team area/tests New or updated tests area/apps > hash-integrations labels Oct 12, 2026
@thehabbos007
thehabbos007 added this pull request to stack #9985 October 12, 2026 14:49
@vercel

vercel Bot commented Oct 12, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hash Ready Ready Preview Oct 19, 2026 12:49pm UTC
petrinaut Ready Ready Preview Oct 19, 2026 12:49pm UTC
petrinaut-docs Ready Ready Preview Oct 19, 2026 12:49pm UTC
1 Skipped Deployment
Project Deployment Actions Updated
hashdotdesign-tokens Ignored Ignored Preview Oct 19, 2026 12:49pm UTC

Request Review

@codecov

codecov Bot commented Oct 12, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.84536% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 68.30%. Comparing base (a561bc2) to head (1a598d5).

Files with missing lines Patch % Lines
...al/integrations/engine/src/workspace/connection.rs 93.97% 4 Missing and 1 partial ⚠️
Additional details and impacted files
@@                              Coverage Diff                               @@
##           asa/hash-integrations/04a-engine-snapshots   #10003      +/-   ##
==============================================================================
+ Coverage                                       68.29%   68.30%   +0.01%     
==============================================================================
  Files                                            2003     2004       +1     
  Lines                                          205815   205892      +77     
  Branches                                         8739     8746       +7     
==============================================================================
+ Hits                                           140568   140643      +75     
- Misses                                          63565    63566       +1     
- Partials                                         1682     1683       +1     
Flag Coverage Δ
hash-integrations-engine 90.44% <94.84%> (+0.90%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@thehabbos007
thehabbos007 force-pushed the asa/hash-integrations/04b-engine-limits branch from b241522 to 8f8ddc7 Compare October 13, 2026 07:45
@thehabbos007
thehabbos007 force-pushed the asa/hash-integrations/04b-engine-limits branch from 8f8ddc7 to fe46e4a Compare October 13, 2026 09:05
@thehabbos007
thehabbos007 force-pushed the asa/hash-integrations/04b-engine-limits branch from fe46e4a to 29f3584 Compare October 13, 2026 10:15
@thehabbos007
thehabbos007 force-pushed the asa/hash-integrations/04b-engine-limits branch from 29f3584 to 9812312 Compare October 13, 2026 19:53
@cursor

cursor Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

PR Summary

Medium Risk
Enforces disk and DuckDB resource limits on open and after statements; misconfigured limits or checkpoint failures can block workspace use, though defaults remain unchanged.

Overview
Adds configurable resource limits on DuckDB workspaces via WorkspaceConfig: optional memory_limit, temp_directory_limit, and threads are applied as DuckDB settings at open, using a new public ByteSize type.

Introduces an optional disk_limit on combined workspace footprint (database, WAL, and .tmp directory). The connection measures usage on open and after each query / create_table, runs CHECKPOINT when over limit before re-measuring, and fails with DiskLimit / DiskUsage errors on open or statement completion. A single statement can still grow files past the limit; data written in that request is retained.

Integration tests cover applied DuckDB settings, open refusal when already over limit, post-create_table errors, temp dir counting, and checkpoint-before-refuse behavior.

Reviewed by Cursor Bugbot for commit 1a598d5. Bugbot is set up for automated code reviews on this repo. Configure here.

This branch was successfully deployed

3 active deployments
Preview – petrinaut-docs — 1a598d5b Deployed Oct 19, 2026 by vercel[bot]
Preview – hash — 1a598d5b Deployed Oct 19, 2026 by vercel[bot]
Preview – petrinaut — 1a598d5b Deployed Oct 19, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/apps > hash-integrations area/libs Relates to first-party libraries/crates/packages (area) area/tests New or updated tests type/eng > backend Owned by the @backend team

Development

Successfully merging this pull request may close these issues.

1 participant