Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 8 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -293,8 +293,9 @@ reducer = logreducer.LogReducer(mode="hybrid", config=cfg)
# ...or let logreducer read your env cascade directly: prefixed beats bare.
cfg = logreducer.BigDialConfig.from_env("MYAPP", "LOGREDUCER") # MYAPP_MAX_MEMORY_GB wins

# 2. Logging: your app owns the loguru sinks; logreducer registers nothing and
# its records flow through YOUR handlers, formatted by YOUR standard.
# 2. Logging: your app owns the stdlib logging handlers; logreducer attaches
# nothing and its records flow through YOUR handlers, at YOUR level,
# formatted by YOUR standard.
logreducer.setup_logging(enable=True, own_sinks=False)
```

Expand Down Expand Up @@ -324,7 +325,7 @@ Logging is off by default. Enable it and pick a format via env vars:
LOG_LEVEL=DEBUG LOG_FORMAT=json logreducer app.log --log
```

`LOG_FORMAT=json` emits one JSON object per line for log aggregators; the default is human-readable (coloured in a terminal, plain in CI/containers). Embedding? See `own_sinks=False` above.
Logging is the standard library `logging` module, under the `logreducer` logger. `LOG_FORMAT=json` emits one JSON object per line for log aggregators (keys `timestamp`, `level`, `target`, `function`, `line_number`, `message`, `fields`); the default is human-readable (coloured in a terminal, plain in CI/containers). Embedding? See `own_sinks=False` above.

## Development

Expand All @@ -342,6 +343,8 @@ See [CONTRIBUTING.md](CONTRIBUTING.md) for the toolchain, test layout (including

Copyright 2026 HYPERI PTY LIMITED.

Credits: template mining (`src/logreducer/template_mining.py`) is derived from [Drain3](https://github.com/logpai/Drain3), Copyright (c) 2020-2022 International Business Machines and the Drain3 project contributors, MIT licence. Drain3 builds on the Drain parser in LogPAI's [logparser](https://github.com/logpai/logparser). The module header carries Drain3's licence text in full.

## Context

Depth: [docs/architecture.md](docs/architecture.md).
Expand All @@ -362,6 +365,7 @@ it is given and drops the rest.
| `src/logreducer/sources.py`, `sinks.py` | the two structural protocols that ARE the integration surface, plus `FileSource` and `FileSink` |
| `src/logreducer/sql.py`, `clickhouse.py`, `kafka.py` | the optional adapters, one extra each |
| `src/logreducer/patterns.py`, `anomaly.py`, `temporal.py` | Drain3 mining and fuzzy dedup, Isolation Forest, time-window grouping |
| `src/logreducer/template_mining.py` | the Drain template miner, derived from Drain3 (credited under License) |
| `src/logreducer/memory.py` | `MemoryMonitor`, `BoundedDeduplicator`, the file read strategies |
| `src/logreducer/sampling.py`, `target.py` | dialect-aware SQL sampling, and the `reduce_to_target` batch loop |
| `src/logreducer/config.py` | `BigDialConfig`, the level presets, the `from_env` cascade |
Expand Down Expand Up @@ -407,7 +411,7 @@ One declared edge in `dfe-infra/suite.yaml`, and it runs the surprising way.

`dfe-stack suite --consumer logreducer` returns an empty edge list: no repo in
that graph is something this library depends on. The runtime dependencies
(drain3, psutil, loguru, numpy, scikit-learn, typer) are packages, declared in
(psutil, numpy, scikit-learn, typer) are packages, declared in
`pyproject.toml`, which is where they belong.

The reason dfe-engine gives for not declaring the dependency -- pin it "once it
Expand Down
22 changes: 10 additions & 12 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
"Intended Audience :: Developers",
"Intended Audience :: System Administrators",
"Topic :: System :: Logging",
"Topic :: System :: Systems Administration",

Check notice on line 27 in pyproject.toml

View workflow job for this annotation

GitHub Actions / ci / Quality

oss-no-trailing-whitespace

trailing whitespace on line 27
"Topic :: Software Development :: Libraries :: Python Modules",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3 :: Only",
Expand All @@ -32,11 +32,9 @@
]
keywords = ["log", "logging", "reduction", "analysis", "pattern-extraction", "memory-efficient", "streaming", "anomaly-detection"]
dependencies = [
"drain3>=0.9.0",
"psutil>=7.2.0", # memory monitoring (MemoryMonitor)
"loguru>=0.7.3",
"numpy>=2.0.0",
"scikit-learn>=1.5.0", # imports as sklearn; >=1.5 for numpy 2 support
"psutil>=7.2.2", # memory monitoring (MemoryMonitor)
"numpy>=2.5.3",
"scikit-learn>=1.9.1", # imports as sklearn
"typer>=0.27.3", # CLI
]

Expand All @@ -51,13 +49,13 @@
# wants a ready-made source/sink. Each needs its own running service, so they
# are covered by docker-gated integration tests.
sql = [
"sqlalchemy>=2.0.30", # cross-DB server-side-cursor streaming (SQLSource)
"sqlalchemy>=2.1.3", # cross-DB server-side-cursor streaming (SQLSource)
]
clickhouse = [
"clickhouse-connect>=1.0", # native block streaming (ClickHouseSource); 1.0 was breaking
"clickhouse-connect>=1.9.0", # native block streaming (ClickHouseSource)
]
kafka = [
"confluent-kafka>=2.14.0", # librdkafka client (KafkaSource/Sink)
"confluent-kafka>=2.15.1", # librdkafka client (KafkaSource/Sink)
]
[project.urls]
Homepage = "https://github.com/hyperi-io/logreducer"
Expand Down Expand Up @@ -142,7 +140,7 @@

# MyPy: KEPT STRICT (logreducer is small and already clean-typed).
# ignore_missing_imports only silences third-party libs without stubs
# (drain3, datasketch, ...); our own code stays fully typed via
# (datasketch, ...); our own code stays fully typed via
# disallow_untyped_defs below.
[tool.mypy]
python_version = "3.12" # Held below requires-python with ruff; see target-version
Expand Down Expand Up @@ -265,9 +263,9 @@
"vermin>=1.6.0", # Python version-compat detection
# Adapter + integration test deps (TEST ONLY - never in the published
# wheel; these mirror the optional extras so the adapters can be exercised).
"sqlalchemy>=2.0.30", # SQLSource against SQLite (real, no server)
"clickhouse-connect>=1.0", # ClickHouseSource (local cluster or docker)
"confluent-kafka>=2.14.0", # KafkaSource/KafkaSink
"sqlalchemy>=2.1.3", # SQLSource against SQLite (real, no server)
"clickhouse-connect>=1.9.0", # ClickHouseSource (local cluster or docker)
"confluent-kafka>=2.15.1", # KafkaSource/KafkaSink
"testcontainers[clickhouse,kafka,postgres,mysql]>=4.15.0", # docker fallback; modules under testcontainers.community
"psycopg[binary]>=3.2", # PostgreSQL driver (real TABLESAMPLE/setseed sampling tests)
"pymysql>=1.1.0", # MySQL driver (real RAND(seed) sampling tests)
Expand Down
4 changes: 3 additions & 1 deletion src/logreducer/anomaly.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
"""Anomaly detection utilities."""

from loguru import logger
import logging

logger = logging.getLogger(__name__)

# Try optional imports
try:
Expand Down
4 changes: 3 additions & 1 deletion src/logreducer/config.py
Original file line number Diff line number Diff line change
@@ -1,13 +1,15 @@
"""Configuration and tuning parameters for LogReducer."""

import logging
import os
import types
import typing
from dataclasses import dataclass, fields
from enum import Enum

import psutil
from loguru import logger

logger = logging.getLogger(__name__)


class ProcessingLevel(Enum):
Expand Down
Loading
Loading