diff --git a/README.md b/README.md index 9a7f111..c6b7c88 100644 --- a/README.md +++ b/README.md @@ -55,7 +55,8 @@ That single command: 4. Renders `.cbmignore` (Odoo core `odoo/odoo` complete + selected modules, dropping `static/lib`, `static/tests`, minified JS, caches and `.git`) in **cumulative batches of 25 modules**, running `codebase-memory-mcp cli index_repository` after each batch so memory-killed - one-shot indexing is never a problem. + one-shot indexing is never a problem. Every pass runs with a quarter of the CPUs the + container may use (see [Keeping the indexer off the whole machine](#keeping-the-indexer-off-the-whole-machine)). 5. Validates the graph scope: no `test_*` addon modules indexed, no missing and no extra module roots compared to the expected module list, and **every indexed file carrying one of the configured extensions** (`SOURCE_GLOBS`: `.py .xml .js .rst .md .css .scss .csv`) — anything @@ -84,10 +85,74 @@ That single command: --batch-size N modules per cumulative indexing pass; 0 indexes one-shot (default: 25) --mode MODE auto (default), installed, or all --modules-file PATH ir.module.module export listing the modules to index +--workers N indexing workers; default is a quarter of the usable CPUs +--index-timeout N CBM_INDEX_WORKER_TIMEOUT_S; a no-progress window, not a time budget --skip-install do not install codebase-memory-mcp when missing --skip-validate do not validate graph scope after indexing ``` +## Keeping the indexer off the whole machine + +Left alone the indexer takes every core it can see. Measured inside a Vauxoo container on an +18 GB MacBook (Docker Desktop VM of 11.9 GiB): + +```text +codebase-memory-mcp cli --index-worker 428% CPU 9.0 GB RSS +container irc190_01: NanoCpus=0 CpuShares=0 Memory=0 +``` + +### CPU: the one thing this tool has to resolve + +That 428% is deliberate on the indexer's side — `cbm_default_worker_count(initial=true)` +returns `total_cores`, commented *"Use all cores for initial indexing — user is waiting"*. +Inside a container that count comes from `sysconf(_SC_NPROCESSORS_ONLN)`, which reports the +**host** CPUs and not the cgroup quota. codebase-memory-mcp says so itself, right where it reads +the override: + +```text +CBM_WORKERS env override (clamped to [1, CBM_WORKERS_MAX]). +Useful inside containers where sysconf(_SC_NPROCESSORS_ONLN) +reports host CPUs rather than the cgroup's effective CPU quota. +``` + +So resolving that quota is what it delegates to its caller, and it is all this tool does: every +pass runs with a quarter of the CPUs the **container** may use — the cgroup v1/v2 quota +(`docker update --cpus=5` reads back as 5, where `os.cpu_count()` keeps reporting 10), falling +back to the affinity mask. The value and its source are logged: + +```text +workers=1 source=default usable_cpus=5 +``` + +`--workers N` overrides it, and an already exported `CBM_WORKERS` is honoured before the default. + +### Memory: already handled, and the real ceiling is the kernel's + +Nothing here touches the memory budget, on purpose. codebase-memory-mcp already sizes it as a +fraction of RAM that scales with the machine (`mem.c`: 25% at or below 16 GB, 35% at or below +32 GB, 50% above). On the 11.9 GiB VM that is 2986 MB — the same number any "a quarter of RAM" +rule would produce, so exporting `CBM_MEM_BUDGET_MB` there changes nothing. + +That budget is **not a ceiling** either: it drives `mem.pressure` logging and allocator purges, +which is why the measured worker reported `mem.init budget_mb=2986 source=ram_fraction` and +still reached 9.0 GB RSS. A real ceiling belongs to the kernel, so bound the container: + +```bash +docker run --memory=3g --cpus=2 ... # or: docker update --memory=3g --cpus=2 +``` + +The cgroup enforces that for everything inside the container, with none of the overshoot a +userspace sampler has — and `--cpus` is then picked up automatically by the worker count above. +Export `CBM_MEM_BUDGET_MB` yourself if the indexer's own fraction needs adjusting. + +### Timeout + +`--index-timeout N` sets `CBM_INDEX_WORKER_TIMEOUT_S`, which is a **no-progress** window, not a +time budget: codebase-memory-mcp kills a worker that logs nothing for 15 minutes, and every +progress line resets the clock, so a long pass that keeps working is never killed by it. Raising +it does not buy a slow pass more time — it only makes the hang detector slower to fire. Left +unset by default for that reason. + ## Indexing the modules installed in production A local docker container only installs the modules of the main app, while production usually has diff --git a/src/codebase_memory_deployv/cli.py b/src/codebase_memory_deployv/cli.py index e55c683..9f96908 100644 --- a/src/codebase_memory_deployv/cli.py +++ b/src/codebase_memory_deployv/cli.py @@ -9,6 +9,7 @@ from .indexer import ( DEFAULT_BATCH_SIZE, DEFAULT_ROOT, + SOURCE_EXTENSIONS, check_layout, cumulative_batches, discover_modules, @@ -17,11 +18,14 @@ installed_modules, modules_from_names, read_modules_file, + extensions_for, + globs_for, render_cbmignore, resolve_project, validate_scope, write_cbmignore, ) +from .limits import resolve_limits _logger = logging.getLogger(__name__) PACKAGE_LOGGER = __name__.rsplit(".", 1)[0] @@ -77,6 +81,30 @@ def build_parser(): "Without a state column the file is assumed to list installed modules only. Ignored by --mode=all" ), ) + parser.add_argument( + "--extensions", + default=None, + help=( + "comma-separated file extensions to index, e.g. 'py,xml' (default: %s). A narrower set " + "means a smaller, faster graph, but it is a weak lever on memory: the peak tracks the " + "nodes extracted, which come almost entirely from .py" % ",".join(SOURCE_EXTENSIONS) + ), + ) + parser.add_argument( + "--workers", + type=int, + default=None, + help="indexing workers (CBM_WORKERS); default is a quarter of the usable CPUs", + ) + parser.add_argument( + "--index-timeout", + type=int, + default=None, + help=( + "CBM_INDEX_WORKER_TIMEOUT_S: seconds an index worker may log nothing before " + "codebase-memory-mcp treats it as hung. Not a time budget; unset by default" + ), + ) parser.add_argument("--skip-install", action="store_true", help="do not install codebase-memory-mcp when missing") parser.add_argument("--skip-validate", action="store_true", help="do not validate graph scope after indexing") parser.add_argument("-v", "--verbose", action="store_true", help="log debug messages too") @@ -128,15 +156,19 @@ def main(argv=None): modules = discover_modules(root) _logger.info("mode=%s modules=%d", mode, len(modules)) + limits = resolve_limits(workers=args.workers, timeout_s=args.index_timeout) + source_globs = globs_for(args.extensions.split(",") if args.extensions else None) + _logger.info("extensions=%s", ",".join(extensions_for(source_globs))) + for number, selected in enumerate(cumulative_batches(modules, args.batch_size), 1): _logger.info( "=== batch=%d modules_rendered=%d/%d project=%s ===", number, len(selected), len(modules), project ) - write_cbmignore(root, render_cbmignore(selected)) - index_repository(root, project) + write_cbmignore(root, render_cbmignore(selected, source_globs)) + index_repository(root, project, limits) if not args.skip_validate: - if not validate_scope(project, modules): + if not validate_scope(project, modules, source_globs): raise SystemExit("Validation failed: fix .cbmignore, re-index and re-run validation") _logger.info("validation=ok") return 0 diff --git a/src/codebase_memory_deployv/indexer.py b/src/codebase_memory_deployv/indexer.py index 3449564..7b5755e 100644 --- a/src/codebase_memory_deployv/indexer.py +++ b/src/codebase_memory_deployv/indexer.py @@ -18,6 +18,8 @@ import subprocess import urllib.request +from .limits import limits_env + _logger = logging.getLogger(__name__) CBM_BIN = "codebase-memory-mcp" @@ -42,6 +44,29 @@ SOURCE_GLOBS = ("*.py", "*.xml", "*.js", "*.rst", "*.md", "*.css", "*.scss", "*.csv", "*.sql") # What .cbmignore lets through, so what the graph is expected to hold: ".py", ".xml", ... SOURCE_EXTENSIONS = tuple(sorted(pattern[1:] for pattern in SOURCE_GLOBS)) +# Measured on a real Odoo instance: codebase-memory-mcp holds the whole extraction in memory, +# and that peak tracks the *nodes* it extracts, not the file count. A mixed scope of 5248 +# files reached 4880 MB RSS with 123023 nodes, i.e. about 40 KB per node. +# +# Narrowing the extensions is therefore a much weaker lever than the file counts suggest: +# .js/.scss/.csv/.md/.rst/.css are two thirds of the files of an Odoo instance (15860 -> 5681) +# but hardly any of the nodes, and indexing only .py still ran a 5 GiB container out of memory. +# What actually decides whether a scope fits is how much *Python* it contains. +MB_RSS_PER_SOURCE_FILE = 0.93 + + +def globs_for(extensions): + """Turn ("py", ".xml") into ("*.py", "*.xml"); None keeps the full SOURCE_GLOBS.""" + if not extensions: + return SOURCE_GLOBS + return tuple("*." + str(name).strip().lstrip("*.") for name in extensions if str(name).strip()) + + +def extensions_for(source_globs): + """The extensions a .cbmignore built from source_globs lets through.""" + return tuple(sorted(pattern[1:] for pattern in source_globs)) + + MANIFEST_NAMES = ("__manifest__.py", "__openerp__.py") PRUNE_DIRS = {".git", ".github", "__pycache__", "node_modules", ".cache", ".tx", "dist", "build", "setup"} CBMIGNORE_BACKUP_SUFFIX = ".before-codebase-memory-deployv" @@ -358,7 +383,7 @@ def cumulative_batches(module_paths, batch_size): return [ordered[:end] for end in range(batch_size, len(ordered) + batch_size, batch_size)] -def render_cbmignore(module_rel_paths): +def render_cbmignore(module_rel_paths, source_globs=SOURCE_GLOBS): """Render a .cbmignore keeping only the Odoo core tree plus the given modules.""" patterns = [] seen = set() @@ -375,7 +400,7 @@ def include_tree(rel): cur = part if not cur else cur + "/" + part add("!%s/" % cur) add("!%s/**/" % rel) - for glob_pattern in SOURCE_GLOBS: + for glob_pattern in source_globs: add("!%s/**/%s" % (rel, glob_pattern)) add("# Generated by codebase-memory-deployv from Odoo modules.") @@ -440,11 +465,15 @@ def write_cbmignore(root, content): return path -def index_repository(root, project): - """Run one indexing pass reusing the same project name for every pass.""" - subprocess.check_call( - [find_cbm() or CBM_BIN, "cli", "index_repository", "--repo_path", root, "--name=%s" % project] - ) +def index_repository(root, project, limits=None): + """Run one indexing pass reusing the same project name for every pass. + + Without limits the pass behaves the way it always did: every core the container sees. + Callers that care (the CLI does) hand over a resolved limits.Limits, whose values reach + codebase-memory-mcp as CBM_* variables it already knows how to read. + """ + command = [find_cbm() or CBM_BIN, "cli", "index_repository", "--repo_path", root, "--name=%s" % project] + subprocess.check_call(command, env=limits_env(limits) if limits is not None else None) def query_graph(project, query): @@ -532,21 +561,22 @@ def indexed_extensions(paths): return counts -def validate_extensions(paths): +def validate_extensions(paths, source_globs=SOURCE_GLOBS): """Prove the graph only holds the extensions .cbmignore lets through. Anything else means .cbmignore did not apply (stale file, pattern typo, indexing run from another root), which silently bloats the graph with vendored/generated content. """ + expected = extensions_for(source_globs) counts = indexed_extensions(paths) - unexpected = {ext: total for ext, total in counts.items() if ext not in SOURCE_EXTENSIONS} - _logger.info("extensions_configured=%s", ",".join(SOURCE_EXTENSIONS)) + unexpected = {ext: total for ext, total in counts.items() if ext not in expected} + _logger.info("extensions_configured=%s", ",".join(expected)) _logger.info("extensions_indexed=%d files=%d", len(counts), sum(counts.values())) for extension, total in sorted(counts.items(), key=lambda item: -item[1]): - _logger.info("extension %-8s files=%d%s", extension, total, "" if extension in SOURCE_EXTENSIONS else " (!)") + _logger.info("extension %-8s files=%d%s", extension, total, "" if extension in expected else " (!)") if unexpected: _logger.error( - "unexpected_extensions=%d files=%d (not in SOURCE_GLOBS: %s)", + "unexpected_extensions=%d files=%d (not in the configured extensions: %s)", len(unexpected), sum(unexpected.values()), ",".join(sorted(unexpected)), @@ -578,7 +608,7 @@ def _is_test_addon(module_rel_path): return name.startswith("test_") and (parent == "addons" or parent.endswith("/addons")) -def validate_scope(project, expected_module_paths): +def validate_scope(project, expected_module_paths, source_globs=SOURCE_GLOBS): """Prove the graph scope matches the expected modules; return True when clean. Addons are enumerated through their manifests: every addon has exactly one, so the @@ -608,4 +638,4 @@ def validate_scope(project, expected_module_paths): _logger.error("extra %s", rel) # Extensions are checked last: the module comparison is the headline, this one tells # whether .cbmignore really applied to whatever did get indexed. - return validate_extensions(paths) and ok and not missing and not extra + return validate_extensions(paths, source_globs) and ok and not missing and not extra diff --git a/src/codebase_memory_deployv/limits.py b/src/codebase_memory_deployv/limits.py new file mode 100644 index 0000000..0d30ee8 --- /dev/null +++ b/src/codebase_memory_deployv/limits.py @@ -0,0 +1,140 @@ +"""Decide how many indexing workers codebase-memory-mcp may use inside a container. + +Left alone the indexer takes every core it can see. Measured inside a Vauxoo container on +an 18 GB MacBook (Docker Desktop VM of 11.9 GiB):: + + codebase-memory-mcp cli --index-worker 428% CPU 9.0 GB RSS + container irc190_01: NanoCpus=0 CpuShares=0 Memory=0 + +That is deliberate on its side: ``cbm_default_worker_count(initial=true)`` returns +``total_cores`` with the comment "Use all cores for initial indexing — user is waiting". +Inside a container that count comes from ``sysconf(_SC_NPROCESSORS_ONLN)``, which reports +the host CPUs and not the cgroup quota — codebase-memory-mcp says so itself, right where +it reads the override:: + + CBM_WORKERS env override (clamped to [1, CBM_WORKERS_MAX]). + Useful inside containers where sysconf(_SC_NPROCESSORS_ONLN) + reports host CPUs rather than the cgroup's effective CPU quota. + +Resolving that quota is the one thing it delegates to its caller, so it is the one thing +this module does. + +Memory is deliberately *not* handled here. codebase-memory-mcp already sizes its budget as +a fraction of RAM that scales with the machine (``mem.c``: 25% at or below 16 GB, 35% at or +below 32 GB, 50% above), which on the measured 11.9 GiB VM is the same 2986 MB any +quarter-of-RAM rule would produce — exporting CBM_MEM_BUDGET_MB there changes nothing. And +that budget is not a ceiling either: it drives ``mem.pressure`` logging and allocator +purges, which is why the measured worker reported ``budget_mb=2986`` and still reached +9.0 GB RSS. A real ceiling belongs to the kernel, not to a userspace sampler that can only +notice the overshoot after it happened: bound the container itself with +``docker run --memory=3g --cpus=2`` and the cgroup enforces it for everything inside. +""" + +import collections +import logging +import os + +_logger = logging.getLogger(__name__) + +# A quarter of the cores, so indexing stays a background job on the developer's laptop +# instead of taking it over. Overridable with --workers. +DEFAULT_CPU_FRACTION = 0.25 + +WORKERS_ENV = "CBM_WORKERS" +TIMEOUT_ENV = "CBM_INDEX_WORKER_TIMEOUT_S" + +CGROUP_V2_CPU = "/sys/fs/cgroup/cpu.max" +CGROUP_V1_CPU_QUOTA = "/sys/fs/cgroup/cpu/cpu.cfs_quota_us" +CGROUP_V1_CPU_PERIOD = "/sys/fs/cgroup/cpu/cpu.cfs_period_us" + +Limits = collections.namedtuple("Limits", "workers timeout_s") + + +def _read_file(path): + """Return the contents of path, or "" when it cannot be read (not Linux, no cgroup).""" + try: + with open(path) as handler: + return handler.read() + except (OSError, ValueError): + return "" + + +def cgroup_cpu_quota(): + """CPUs granted by the cgroup (5.0 for "docker update --cpus=5"), or None when unbounded.""" + raw = _read_file(CGROUP_V2_CPU).split() + if len(raw) == 2 and raw[0] != "max": + try: + quota, period = int(raw[0]), int(raw[1]) + except ValueError: + return None + return float(quota) / period if quota > 0 and period > 0 else None + try: + quota = int(_read_file(CGROUP_V1_CPU_QUOTA).strip() or -1) + period = int(_read_file(CGROUP_V1_CPU_PERIOD).strip() or 0) + except ValueError: + return None + return float(quota) / period if quota > 0 and period > 0 else None + + +def usable_cpus(): + """CPUs this process may actually use: cgroup quota, else affinity, else the machine.""" + quota = cgroup_cpu_quota() + if quota: + return max(1, int(quota)) + affinity = getattr(os, "sched_getaffinity", None) + if affinity is not None: + try: + return max(1, len(affinity(0))) + except OSError: + pass + return max(1, os.cpu_count() or 1) + + +def default_workers(fraction=DEFAULT_CPU_FRACTION): + """A quarter of the usable CPUs, never below one.""" + return max(1, int(usable_cpus() * fraction)) + + +def _from_environment(environ, name): + """Positive integer exported as name, or None. An exported value is a deliberate choice.""" + try: + value = int(environ.get(name, "").strip()) + except (AttributeError, ValueError): + return None + return value if value > 0 else None + + +def resolve_limits(workers=None, timeout_s=None, environ=None): + """Resolve the worker count and the optional worker timeout, logging each source. + + Precedence is the command line, then an already exported CBM_* variable, then the + default. The environment is honoured on purpose: whoever exported CBM_WORKERS in that + container knows something about it, and this tool is not the only thing reading it. + + timeout_s has no default. CBM_INDEX_WORKER_TIMEOUT_S is a *no-progress* window, not a + time budget -- codebase-memory-mcp kills a worker that logs nothing for 15 minutes, and + every progress line resets it -- so raising it does not buy a long pass more time, it + only makes the hang detector slower to fire. It stays available for whoever needs that. + """ + environ = os.environ if environ is None else environ + if workers is None: + workers = _from_environment(environ, WORKERS_ENV) + source = WORKERS_ENV if workers else "default" + workers = workers or default_workers() + else: + source = "argument" + _logger.info("workers=%d source=%s usable_cpus=%d", workers, source, usable_cpus()) + if timeout_s is None: + timeout_s = _from_environment(environ, TIMEOUT_ENV) + if timeout_s: + _logger.info("index_worker_no_progress_timeout_s=%d", timeout_s) + return Limits(workers=workers, timeout_s=timeout_s) + + +def limits_env(limits, environ=None): + """A copy of environ carrying the resolved limits, ready for subprocess.""" + env = dict(os.environ if environ is None else environ) + env[WORKERS_ENV] = str(limits.workers) + if limits.timeout_s: + env[TIMEOUT_ENV] = str(limits.timeout_s) + return env diff --git a/tests/test_cli.py b/tests/test_cli.py index 02123c0..a80df0e 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -56,9 +56,9 @@ def test_main_modules_file_wins_over_the_database(tmp_path, monkeypatch, capsys) modules_file.write_text("name,state\nsale_extended,installed\nnever_installed_here,installed\nweb,installed\n") indexed = [] monkeypatch.setattr(cli, "ensure_cbm_installed", lambda: "cbm") - monkeypatch.setattr(cli, "index_repository", lambda root_, project: indexed.append(project)) + monkeypatch.setattr(cli, "index_repository", lambda root_, project, limits_: indexed.append(project)) monkeypatch.setattr(cli, "installed_modules", lambda root_: pytest.fail("the database must not be queried")) - monkeypatch.setattr(cli, "validate_scope", lambda project, modules: True) + monkeypatch.setattr(cli, "validate_scope", lambda project, modules, globs_=None: True) monkeypatch.setenv("CBM_PROJECT", "prod_18.0") assert cli.main(["--repo-path", root, "--modules-file", str(modules_file)]) == 0 out = capsys.readouterr().out @@ -75,7 +75,7 @@ def test_main_modules_file_ignored_with_mode_all(tmp_path, monkeypatch, capsys): modules_file = tmp_path / "modules.csv" modules_file.write_text("name\nsale_extended\n") monkeypatch.setattr(cli, "ensure_cbm_installed", lambda: "cbm") - monkeypatch.setattr(cli, "index_repository", lambda root_, project: None) + monkeypatch.setattr(cli, "index_repository", lambda root_, project, limits_: None) monkeypatch.setenv("CBM_PROJECT", "prod_18.0") assert cli.main(["--repo-path", root, "--mode=all", "--modules-file", str(modules_file), "--skip-validate"]) == 0 assert "modules_file=ignored" in capsys.readouterr().out @@ -92,3 +92,30 @@ def test_main_modules_file_without_modules_on_disk(tmp_path, monkeypatch): assert "does not list any module" in str(error) else: raise AssertionError("main must fail when the modules file matches nothing on disk") + + +def test_parser_resource_defaults_are_resolved_at_runtime(): + """The flags default to None so the environment still has a say; limits resolve them.""" + args = cli.build_parser().parse_args([]) + assert args.workers is None + assert args.index_timeout is None + + +def test_parser_accepts_the_resource_flags(): + args = cli.build_parser().parse_args(["--workers=2", "--index-timeout=600"]) + assert (args.workers, args.index_timeout) == (2, 600) + + +def test_main_passes_the_resolved_limits_to_every_pass(tmp_path, monkeypatch): + root = _instance(tmp_path, "extra_addons/vauxoo/sale_extended") + seen = [] + monkeypatch.setattr(cli, "ensure_cbm_installed", lambda: "cbm") + monkeypatch.setattr(cli, "index_repository", lambda root_, project, limits_: seen.append(limits_)) + monkeypatch.setattr(cli, "installed_modules", lambda root_: None) + assert cli.main(["--repo-path", root, "--workers=2", "--skip-validate"]) == 0 + assert [one.workers for one in seen] == [2] + + +def test_help_renders(): + """argparse re-expands "%" in a help string: a literal one there breaks --help only.""" + assert "--workers" in cli.build_parser().format_help() diff --git a/tests/test_indexer.py b/tests/test_indexer.py index 82f88dc..694902a 100644 --- a/tests/test_indexer.py +++ b/tests/test_indexer.py @@ -6,7 +6,7 @@ import pytest -from codebase_memory_deployv import indexer +from codebase_memory_deployv import indexer, limits def test_cumulative_batches_are_cumulative(): @@ -480,3 +480,51 @@ def test_check_layout_requires_odoo_bin(tmp_path): odoo_dir.mkdir() (odoo_dir / "odoo-bin").write_text("") assert indexer.check_layout(str(tmp_path)) == os.path.join(str(tmp_path), "odoo", "odoo-bin") + + +def test_index_repository_without_limits_keeps_the_old_behaviour(monkeypatch): + calls = [] + monkeypatch.setattr(indexer, "find_cbm", lambda: "/usr/bin/cbm") + monkeypatch.setattr(indexer.subprocess, "check_call", lambda cmd, env=None: calls.append((cmd, env))) + indexer.index_repository("/home/odoo/instance", "vauxoo_12.0") + command, env = calls[0] + assert command == [ + "/usr/bin/cbm", + "cli", + "index_repository", + "--repo_path", + "/home/odoo/instance", + "--name=vauxoo_12.0", + ] + assert env is None + + +def test_index_repository_hands_the_worker_count_to_the_child(monkeypatch): + """The pass is bounded by what codebase-memory-mcp already reads, not by a new mechanism.""" + calls = [] + monkeypatch.setattr(indexer, "find_cbm", lambda: "/usr/bin/cbm") + monkeypatch.setattr(indexer.subprocess, "check_call", lambda cmd, env=None: calls.append((cmd, env))) + indexer.index_repository("/home/odoo/instance", "vauxoo_12.0", limits.Limits(workers=3, timeout_s=None)) + _command, env = calls[0] + assert env["CBM_WORKERS"] == "3" + assert "CBM_MEM_BUDGET_MB" not in env + + +def test_globs_for_normalises_extensions(): + assert indexer.globs_for(["py", ".xml", " js "]) == ("*.py", "*.xml", "*.js") + assert indexer.globs_for(None) == indexer.SOURCE_GLOBS + assert indexer.globs_for([]) == indexer.SOURCE_GLOBS + + +def test_render_cbmignore_honours_a_narrowed_extension_set(): + """Fewer extensions is the one lever on memory: it is what shrinks the file count.""" + content = indexer.render_cbmignore(["extra_addons/vauxoo/sale_extended"], ("*.py",)) + assert "!extra_addons/vauxoo/sale_extended/**/*.py" in content + assert "*.js" not in content + assert "*.scss" not in content + + +def test_validate_extensions_follows_the_configured_set(caplog): + paths = ["a/b.py", "a/c.js"] + assert indexer.validate_extensions(paths, indexer.SOURCE_GLOBS) + assert not indexer.validate_extensions(paths, ("*.py",)) diff --git a/tests/test_limits.py b/tests/test_limits.py new file mode 100644 index 0000000..f7cbcca --- /dev/null +++ b/tests/test_limits.py @@ -0,0 +1,93 @@ +import pytest + +from codebase_memory_deployv import limits + + +def test_default_workers_is_a_quarter_of_the_cpus(monkeypatch): + monkeypatch.setattr(limits, "usable_cpus", lambda: 12) + assert limits.default_workers() == 3 + + +def test_default_workers_never_reaches_zero(monkeypatch): + """A machine with one or two cores still has to index; it just does it with one worker.""" + monkeypatch.setattr(limits, "usable_cpus", lambda: 2) + assert limits.default_workers() == 1 + + +def test_cgroup_cpu_quota_v2(tmp_path, monkeypatch): + """docker update --cpus=5 must be seen; os.cpu_count() would keep reporting the host.""" + quota = tmp_path / "cpu.max" + quota.write_text("500000 100000\n") + monkeypatch.setattr(limits, "CGROUP_V2_CPU", str(quota)) + assert limits.cgroup_cpu_quota() == 5.0 + assert limits.usable_cpus() == 5 + + +def test_cgroup_cpu_quota_v2_unlimited(tmp_path, monkeypatch): + quota = tmp_path / "cpu.max" + quota.write_text("max 100000\n") + monkeypatch.setattr(limits, "CGROUP_V2_CPU", str(quota)) + monkeypatch.setattr(limits, "CGROUP_V1_CPU_QUOTA", str(tmp_path / "missing")) + assert limits.cgroup_cpu_quota() is None + + +def test_cgroup_cpu_quota_v1(tmp_path, monkeypatch): + (tmp_path / "cpu.cfs_quota_us").write_text("200000\n") + (tmp_path / "cpu.cfs_period_us").write_text("100000\n") + monkeypatch.setattr(limits, "CGROUP_V2_CPU", str(tmp_path / "missing")) + monkeypatch.setattr(limits, "CGROUP_V1_CPU_QUOTA", str(tmp_path / "cpu.cfs_quota_us")) + monkeypatch.setattr(limits, "CGROUP_V1_CPU_PERIOD", str(tmp_path / "cpu.cfs_period_us")) + assert limits.cgroup_cpu_quota() == 2.0 + + +def test_cgroup_cpu_quota_malformed(tmp_path, monkeypatch): + """A cpu.max that does not parse must read as "unbounded", not raise mid-run.""" + quota = tmp_path / "cpu.max" + quota.write_text("weird\n") + monkeypatch.setattr(limits, "CGROUP_V2_CPU", str(quota)) + monkeypatch.setattr(limits, "CGROUP_V1_CPU_QUOTA", str(tmp_path / "missing")) + assert limits.cgroup_cpu_quota() is None + + +def test_resolve_limits_defaults(monkeypatch): + monkeypatch.setattr(limits, "usable_cpus", lambda: 8) + assert limits.resolve_limits(environ={}) == limits.Limits(workers=2, timeout_s=None) + + +def test_resolve_limits_arguments_win_over_the_environment(monkeypatch): + monkeypatch.setattr(limits, "usable_cpus", lambda: 8) + environ = {"CBM_WORKERS": "7", "CBM_INDEX_WORKER_TIMEOUT_S": "60"} + assert limits.resolve_limits(workers=1, timeout_s=30, environ=environ) == limits.Limits(workers=1, timeout_s=30) + + +def test_resolve_limits_environment_wins_over_the_default(monkeypatch): + """Whoever exported CBM_WORKERS knows something about that container; do not overrule it.""" + monkeypatch.setattr(limits, "usable_cpus", lambda: 8) + environ = {"CBM_WORKERS": "7", "CBM_INDEX_WORKER_TIMEOUT_S": "60"} + assert limits.resolve_limits(environ=environ) == limits.Limits(workers=7, timeout_s=60) + + +@pytest.mark.parametrize("value", ["", "0", "-4", "many", " "]) +def test_resolve_limits_ignores_unusable_environment_values(monkeypatch, value): + monkeypatch.setattr(limits, "usable_cpus", lambda: 8) + resolved = limits.resolve_limits(environ={"CBM_WORKERS": value, "CBM_INDEX_WORKER_TIMEOUT_S": value}) + assert resolved == limits.Limits(workers=2, timeout_s=None) + + +def test_limits_env_exports_the_worker_count(): + env = limits.limits_env(limits.Limits(workers=3, timeout_s=None), environ={"PATH": "/usr/bin"}) + assert env["CBM_WORKERS"] == "3" + assert env["PATH"] == "/usr/bin" + + +def test_limits_env_leaves_the_timeout_alone_unless_asked(): + """CBM_INDEX_WORKER_TIMEOUT_S is a no-progress window; its 15 min default is fine.""" + assert limits.TIMEOUT_ENV not in limits.limits_env(limits.Limits(workers=1, timeout_s=None), environ={}) + env = limits.limits_env(limits.Limits(workers=1, timeout_s=600), environ={}) + assert env[limits.TIMEOUT_ENV] == "600" + + +def test_limits_module_does_not_touch_the_memory_budget(): + """codebase-memory-mcp already scales its budget with the RAM (25/35/50%); do not duplicate it.""" + env = limits.limits_env(limits.Limits(workers=2, timeout_s=None), environ={}) + assert "CBM_MEM_BUDGET_MB" not in env