Skip to content

fix: vendor drain3's template miner and drop loguru - #15

Merged
catinspace-au merged 3 commits into
mainfrom
fix/vendor-drain3
Oct 14, 2026
Merged

catinspace-au merged 3 commits into
mainfrom
fix/vendor-drain3

Conversation

@catinspace-au

Copy link
Copy Markdown
Contributor

dfe-engine's default sample mode runs logreducer, and every engine image ships without it. drain3 is why: its only release is a 2022 sdist with no wheel, pinned to jsonpickle 1.5.1 and cachetools 4.2.1, and the engine image installs with UV_NO_BUILD=1, which refuses sdists.

  • Drain template mining now lives in src/logreducer/template_mining.py, derived from Drain3 at 0fb6d6e (2025-02-03), MIT. Only the streaming path logreducer drives is kept. Persistence, the profiler, JaccardDrain, match() and parameter extraction are gone, and jsonpickle with them. The cachetools LRU cluster cache is now an OrderedDict store with the same eviction order.
  • Output does not move. tests/testdata/template_mining_golden.json.gz is what drain3 0.9.11 produced on the first 3000 lines of all 11 corpora under five configs (default, typed masking, max_clusters=25, similarity 0.3 and 0.5). The new miner matches it exactly: same cluster for every line, same templates, same counts, same order.
  • loguru is gone too, last release 2024-12. Logging is stdlib logging under the logreducer logger and setup_logging keeps its signature. What you will notice: LOG_FORMAT=json lines carry scalo's keys (timestamp, level, target, function, line_number, message, fields) instead of loguru's text/record pair, and the file sink keeps seven 10 MB rotations rather than seven days. own_sinks=False hands records to the host's stdlib handlers at the host's level, which is where scalo-py's intercept picks them up.
  • Runtime deps are now psutil, numpy, scikit-learn and typer, all with cp314 wheels for manylinux x86_64 and aarch64. Floors move to the locked versions, extras included.
  • SQLSource.from_table narrowed its engine type through a variable mypy cannot follow, so mypy failed on it. Fixed.

Done when CI is green and 3.5.3 is on PyPI with no drain3 or loguru in its requirements.

drain3's only release is a 2022 sdist with no wheel, pinned to jsonpickle 1.5.1 and cachetools 4.2.1, so a wheel-only install of logreducer fails. The Drain miner now lives in logreducer.template_mining, derived from Drain3 at 0fb6d6e (MIT), keeping only the streaming path logreducer drives. A committed golden of drain3 0.9.11's output on all 11 test corpora under five configs holds it to the same cluster per line, templates, counts and order.

loguru, last released 2024-12, is replaced by stdlib logging under the logreducer logger. JSON log lines take scalo's keys and the file sink keeps seven 10 MB rotations.

Runtime deps are now psutil, numpy, scikit-learn and typer, and every floor moves to the locked version.
@catinspace-au
catinspace-au merged commit 8a07906 into main Oct 14, 2026
17 checks passed
@catinspace-au
catinspace-au deleted the fix/vendor-drain3 branch October 14, 2026 07:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant