Repository navigation
fix: vendor drain3's template miner and drop loguru - #15
Merged
Merged
Conversation
drain3's only release is a 2022 sdist with no wheel, pinned to jsonpickle 1.5.1 and cachetools 4.2.1, so a wheel-only install of logreducer fails. The Drain miner now lives in logreducer.template_mining, derived from Drain3 at 0fb6d6e (MIT), keeping only the streaming path logreducer drives. A committed golden of drain3 0.9.11's output on all 11 test corpora under five configs holds it to the same cluster per line, templates, counts and order. loguru, last released 2024-12, is replaced by stdlib logging under the logreducer logger. JSON log lines take scalo's keys and the file sink keeps seven 10 MB rotations. Runtime deps are now psutil, numpy, scikit-learn and typer, and every floor moves to the locked version.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
dfe-engine's default sample mode runs logreducer, and every engine image ships without it. drain3 is why: its only release is a 2022 sdist with no wheel, pinned to jsonpickle 1.5.1 and cachetools 4.2.1, and the engine image installs with UV_NO_BUILD=1, which refuses sdists.
src/logreducer/template_mining.py, derived from Drain3 at 0fb6d6e (2025-02-03), MIT. Only the streaming path logreducer drives is kept. Persistence, the profiler, JaccardDrain, match() and parameter extraction are gone, and jsonpickle with them. The cachetools LRU cluster cache is now an OrderedDict store with the same eviction order.tests/testdata/template_mining_golden.json.gzis what drain3 0.9.11 produced on the first 3000 lines of all 11 corpora under five configs (default, typed masking, max_clusters=25, similarity 0.3 and 0.5). The new miner matches it exactly: same cluster for every line, same templates, same counts, same order.loggingunder thelogreducerlogger andsetup_loggingkeeps its signature. What you will notice:LOG_FORMAT=jsonlines carry scalo's keys (timestamp, level, target, function, line_number, message, fields) instead of loguru's text/record pair, and the file sink keeps seven 10 MB rotations rather than seven days.own_sinks=Falsehands records to the host's stdlib handlers at the host's level, which is where scalo-py's intercept picks them up.Done when CI is green and 3.5.3 is on PyPI with no drain3 or loguru in its requirements.