[IMP] Bound the CPU, memory and time of an indexing pass

README.md

@@ -55,7 +55,8 @@ That single command:
 4. Renders `.cbmignore` (Odoo core `odoo/odoo` complete + selected modules, dropping
    `static/lib`, `static/tests`, minified JS, caches and `.git`) in **cumulative batches of 25
    modules**, running `codebase-memory-mcp cli index_repository` after each batch so memory-killed
-   one-shot indexing is never a problem.
+   one-shot indexing is never a problem. Every pass runs with a quarter of the CPUs the
+   container may use (see [Keeping the indexer off the whole machine](#keeping-the-indexer-off-the-whole-machine)).
 5. Validates the graph scope: no `test_*` addon modules indexed, no missing and no extra module
    roots compared to the expected module list, and **every indexed file carrying one of the
    configured extensions** (`SOURCE_GLOBS`: `.py .xml .js .rst .md .css .scss .csv`) — anything
@@ -84,10 +85,74 @@ That single command:
 --batch-size N      modules per cumulative indexing pass; 0 indexes one-shot (default: 25)
 --mode MODE         auto (default), installed, or all
 --modules-file PATH ir.module.module export listing the modules to index
+--workers N         indexing workers; default is a quarter of the usable CPUs
+--index-timeout N   CBM_INDEX_WORKER_TIMEOUT_S; a no-progress window, not a time budget
 --skip-install      do not install codebase-memory-mcp when missing
 --skip-validate     do not validate graph scope after indexing
 ```
 
+## Keeping the indexer off the whole machine
+
+Left alone the indexer takes every core it can see. Measured inside a Vauxoo container on an
+18 GB MacBook (Docker Desktop VM of 11.9 GiB):
+
+```text
+codebase-memory-mcp cli --index-worker    428% CPU    9.0 GB RSS
+container irc190_01:  NanoCpus=0  CpuShares=0  Memory=0
+```
+
+### CPU: the one thing this tool has to resolve
+
+That 428% is deliberate on the indexer's side — `cbm_default_worker_count(initial=true)`
+returns `total_cores`, commented *"Use all cores for initial indexing — user is waiting"*.
+Inside a container that count comes from `sysconf(_SC_NPROCESSORS_ONLN)`, which reports the
+**host** CPUs and not the cgroup quota. codebase-memory-mcp says so itself, right where it reads
+the override:
+
+```text
+CBM_WORKERS env override (clamped to [1, CBM_WORKERS_MAX]).
+Useful inside containers where sysconf(_SC_NPROCESSORS_ONLN)
+reports host CPUs rather than the cgroup's effective CPU quota.
+```
+
+So resolving that quota is what it delegates to its caller, and it is all this tool does: every
+pass runs with a quarter of the CPUs the **container** may use — the cgroup v1/v2 quota
+(`docker update --cpus=5` reads back as 5, where `os.cpu_count()` keeps reporting 10), falling
+back to the affinity mask. The value and its source are logged:
+
+```text
+workers=1 source=default usable_cpus=5
+```
+
+`--workers N` overrides it, and an already exported `CBM_WORKERS` is honoured before the default.
+
+### Memory: already handled, and the real ceiling is the kernel's
+
+Nothing here touches the memory budget, on purpose. codebase-memory-mcp already sizes it as a
+fraction of RAM that scales with the machine (`mem.c`: 25% at or below 16 GB, 35% at or below
+32 GB, 50% above). On the 11.9 GiB VM that is 2986 MB — the same number any "a quarter of RAM"
+rule would produce, so exporting `CBM_MEM_BUDGET_MB` there changes nothing.
+
+That budget is **not a ceiling** either: it drives `mem.pressure` logging and allocator purges,
+which is why the measured worker reported `mem.init budget_mb=2986 source=ram_fraction` and
+still reached 9.0 GB RSS. A real ceiling belongs to the kernel, so bound the container:
+
+```bash
+docker run --memory=3g --cpus=2 ...    # or: docker update --memory=3g --cpus=2 <container>
+```
+
+The cgroup enforces that for everything inside the container, with none of the overshoot a
+userspace sampler has — and `--cpus` is then picked up automatically by the worker count above.
+Export `CBM_MEM_BUDGET_MB` yourself if the indexer's own fraction needs adjusting.
+
+### Timeout
+
+`--index-timeout N` sets `CBM_INDEX_WORKER_TIMEOUT_S`, which is a **no-progress** window, not a
+time budget: codebase-memory-mcp kills a worker that logs nothing for 15 minutes, and every
+progress line resets the clock, so a long pass that keeps working is never killed by it. Raising
+it does not buy a slow pass more time — it only makes the hang detector slower to fire. Left
+unset by default for that reason.
+
 ## Indexing the modules installed in production
 
 A local docker container only installs the modules of the main app, while production usually has

src/codebase_memory_deployv/cli.py

@@ -9,6 +9,7 @@
 from .indexer import (
     DEFAULT_BATCH_SIZE,
     DEFAULT_ROOT,
+    SOURCE_EXTENSIONS,
     check_layout,
     cumulative_batches,
     discover_modules,
@@ -17,11 +18,14 @@
     installed_modules,
     modules_from_names,
     read_modules_file,
+    extensions_for,
+    globs_for,
     render_cbmignore,
     resolve_project,
     validate_scope,
     write_cbmignore,
 )
+from .limits import resolve_limits
 
 _logger = logging.getLogger(__name__)
 PACKAGE_LOGGER = __name__.rsplit(".", 1)[0]
@@ -77,6 +81,30 @@ def build_parser():
             "Without a state column the file is assumed to list installed modules only. Ignored by --mode=all"
         ),
     )
+    parser.add_argument(
+        "--extensions",
+        default=None,
+        help=(
+            "comma-separated file extensions to index, e.g. 'py,xml' (default: %s). A narrower set "
+            "means a smaller, faster graph, but it is a weak lever on memory: the peak tracks the "
+            "nodes extracted, which come almost entirely from .py" % ",".join(SOURCE_EXTENSIONS)
+        ),
+    )
+    parser.add_argument(
+        "--workers",
+        type=int,
+        default=None,
+        help="indexing workers (CBM_WORKERS); default is a quarter of the usable CPUs",
+    )
+    parser.add_argument(
+        "--index-timeout",
+        type=int,
+        default=None,
+        help=(
+            "CBM_INDEX_WORKER_TIMEOUT_S: seconds an index worker may log nothing before "
+            "codebase-memory-mcp treats it as hung. Not a time budget; unset by default"
+        ),
+    )
     parser.add_argument("--skip-install", action="store_true", help="do not install codebase-memory-mcp when missing")
     parser.add_argument("--skip-validate", action="store_true", help="do not validate graph scope after indexing")
     parser.add_argument("-v", "--verbose", action="store_true", help="log debug messages too")
@@ -128,15 +156,19 @@ def main(argv=None):
         modules = discover_modules(root)
     _logger.info("mode=%s modules=%d", mode, len(modules))
 
+    limits = resolve_limits(workers=args.workers, timeout_s=args.index_timeout)
+    source_globs = globs_for(args.extensions.split(",") if args.extensions else None)
+    _logger.info("extensions=%s", ",".join(extensions_for(source_globs)))
+
     for number, selected in enumerate(cumulative_batches(modules, args.batch_size), 1):
         _logger.info(
             "=== batch=%d modules_rendered=%d/%d project=%s ===", number, len(selected), len(modules), project
         )
-        write_cbmignore(root, render_cbmignore(selected))
-        index_repository(root, project)
+        write_cbmignore(root, render_cbmignore(selected, source_globs))
+        index_repository(root, project, limits)
 
     if not args.skip_validate:
-        if not validate_scope(project, modules):
+        if not validate_scope(project, modules, source_globs):
             raise SystemExit("Validation failed: fix .cbmignore, re-index and re-run validation")
         _logger.info("validation=ok")
     return 0

src/codebase_memory_deployv/indexer.py

@@ -18,6 +18,8 @@
 import subprocess
 import urllib.request
 
+from .limits import limits_env
+
 _logger = logging.getLogger(__name__)
 
 CBM_BIN = "codebase-memory-mcp"
@@ -42,6 +44,29 @@
 SOURCE_GLOBS = ("*.py", "*.xml", "*.js", "*.rst", "*.md", "*.css", "*.scss", "*.csv", "*.sql")
 # What .cbmignore lets through, so what the graph is expected to hold: ".py", ".xml", ...
 SOURCE_EXTENSIONS = tuple(sorted(pattern[1:] for pattern in SOURCE_GLOBS))
+# Measured on a real Odoo instance: codebase-memory-mcp holds the whole extraction in memory,
+# and that peak tracks the *nodes* it extracts, not the file count. A mixed scope of 5248
+# files reached 4880 MB RSS with 123023 nodes, i.e. about 40 KB per node.
+#
+# Narrowing the extensions is therefore a much weaker lever than the file counts suggest:
+# .js/.scss/.csv/.md/.rst/.css are two thirds of the files of an Odoo instance (15860 -> 5681)
+# but hardly any of the nodes, and indexing only .py still ran a 5 GiB container out of memory.
+# What actually decides whether a scope fits is how much *Python* it contains.
+MB_RSS_PER_SOURCE_FILE = 0.93
+
+
+def globs_for(extensions):
+    """Turn ("py", ".xml") into ("*.py", "*.xml"); None keeps the full SOURCE_GLOBS."""
+    if not extensions:
+        return SOURCE_GLOBS
+    return tuple("*." + str(name).strip().lstrip("*.") for name in extensions if str(name).strip())
+
+
+def extensions_for(source_globs):
+    """The extensions a .cbmignore built from source_globs lets through."""
+    return tuple(sorted(pattern[1:] for pattern in source_globs))
+
+
 MANIFEST_NAMES = ("__manifest__.py", "__openerp__.py")
 PRUNE_DIRS = {".git", ".github", "__pycache__", "node_modules", ".cache", ".tx", "dist", "build", "setup"}
 CBMIGNORE_BACKUP_SUFFIX = ".before-codebase-memory-deployv"
@@ -358,7 +383,7 @@ def cumulative_batches(module_paths, batch_size):
     return [ordered[:end] for end in range(batch_size, len(ordered) + batch_size, batch_size)]
 
 
-def render_cbmignore(module_rel_paths):
+def render_cbmignore(module_rel_paths, source_globs=SOURCE_GLOBS):
     """Render a .cbmignore keeping only the Odoo core tree plus the given modules."""
     patterns = []
     seen = set()
@@ -375,7 +400,7 @@ def include_tree(rel):
             cur = part if not cur else cur + "/" + part
             add("!%s/" % cur)
         add("!%s/**/" % rel)
-        for glob_pattern in SOURCE_GLOBS:
+        for glob_pattern in source_globs:
             add("!%s/**/%s" % (rel, glob_pattern))
 
     add("# Generated by codebase-memory-deployv from Odoo modules.")
@@ -440,11 +465,15 @@ def write_cbmignore(root, content):
     return path
 
 
-def index_repository(root, project):
-    """Run one indexing pass reusing the same project name for every pass."""
-    subprocess.check_call(
-        [find_cbm() or CBM_BIN, "cli", "index_repository", "--repo_path", root, "--name=%s" % project]
-    )
+def index_repository(root, project, limits=None):
+    """Run one indexing pass reusing the same project name for every pass.
+
+    Without limits the pass behaves the way it always did: every core the container sees.
+    Callers that care (the CLI does) hand over a resolved limits.Limits, whose values reach
+    codebase-memory-mcp as CBM_* variables it already knows how to read.
+    """
+    command = [find_cbm() or CBM_BIN, "cli", "index_repository", "--repo_path", root, "--name=%s" % project]
+    subprocess.check_call(command, env=limits_env(limits) if limits is not None else None)
 
 
 def query_graph(project, query):
@@ -532,21 +561,22 @@ def indexed_extensions(paths):
     return counts
 
 
-def validate_extensions(paths):
+def validate_extensions(paths, source_globs=SOURCE_GLOBS):
     """Prove the graph only holds the extensions .cbmignore lets through.
 
     Anything else means .cbmignore did not apply (stale file, pattern typo, indexing run
     from another root), which silently bloats the graph with vendored/generated content.
     """
+    expected = extensions_for(source_globs)
     counts = indexed_extensions(paths)
-    unexpected = {ext: total for ext, total in counts.items() if ext not in SOURCE_EXTENSIONS}
-    _logger.info("extensions_configured=%s", ",".join(SOURCE_EXTENSIONS))
+    unexpected = {ext: total for ext, total in counts.items() if ext not in expected}
+    _logger.info("extensions_configured=%s", ",".join(expected))
     _logger.info("extensions_indexed=%d files=%d", len(counts), sum(counts.values()))
     for extension, total in sorted(counts.items(), key=lambda item: -item[1]):
-        _logger.info("extension %-8s files=%d%s", extension, total, "" if extension in SOURCE_EXTENSIONS else " (!)")
+        _logger.info("extension %-8s files=%d%s", extension, total, "" if extension in expected else " (!)")
     if unexpected:
         _logger.error(
-            "unexpected_extensions=%d files=%d (not in SOURCE_GLOBS: %s)",
+            "unexpected_extensions=%d files=%d (not in the configured extensions: %s)",
             len(unexpected),
             sum(unexpected.values()),
             ",".join(sorted(unexpected)),
@@ -578,7 +608,7 @@ def _is_test_addon(module_rel_path):
     return name.startswith("test_") and (parent == "addons" or parent.endswith("/addons"))
 
 
-def validate_scope(project, expected_module_paths):
+def validate_scope(project, expected_module_paths, source_globs=SOURCE_GLOBS):
     """Prove the graph scope matches the expected modules; return True when clean.
 
     Addons are enumerated through their manifests: every addon has exactly one, so the
@@ -608,4 +638,4 @@ def validate_scope(project, expected_module_paths):
         _logger.error("extra %s", rel)
     # Extensions are checked last: the module comparison is the headline, this one tells
     # whether .cbmignore really applied to whatever did get indexed.
-    return validate_extensions(paths) and ok and not missing and not extra
+    return validate_extensions(paths, source_globs) and ok and not missing and not extra