Simple - Multi-Format - Log Analyzer

A high-performance Python tool for analyzing syslog files from various network devices. Built with Polars for fast processing of large log files.

Supported Log Formats

Palo Alto Networks - Firewall traffic logs
UniFi - Ubiquiti network device logs (including CEF security events)
WatchGuard - Firewall and security appliance logs
Meraki - Cisco Meraki network device logs (MX, MS, MR)
Extensible - Easy to add new log format parsers

Features

Fast parsing - Process millions of log entries efficiently using Polars
Multi-format support - Analyze logs from different vendors with format-specific parsers
Comprehensive analysis - Automatic identification of:
- Top traffic sources and destinations
- Most active applications and policies
- Session end reasons and patterns
- Traffic volume trends over time
- Noise candidates (high-volume sources/destinations/policies)
- Process and category analysis (UniFi)
- Application characteristics (Palo Alto)
Multiple export formats - Generate reports in HTML, Markdown, or JSON
Progress tracking - Visual progress bar for large files
Flexible filtering - Configurable top-N results and noise thresholds

Installation

Prerequisites

Python 3.11+
uv (recommended) or pip

Using uv (recommended)

git clone https://github.com/LogCentral-io/simple-log-analyzer.git
cd simple-log-analyzer
uv sync

Using pip

git clone https://github.com/LogCentral-io/simple-log-analyzer.git
cd simple-log-analyzer
pip install -e .

Usage

The log analyzer uses subcommands for different log formats:

log-analyzer <format> [OPTIONS] <logfile>

Available formats:

palo - Palo Alto Networks logs
unifi - UniFi device logs
watchguard - WatchGuard firewall logs
meraki - Meraki network device logs

Basic Analysis

Palo Alto Networks:

uv run log-analyzer palo path/to/palo-alto.log

UniFi Devices:

uv run log-analyzer unifi path/to/unifi.log

WatchGuard Firewall:

uv run log-analyzer watchguard path/to/watchguard.log

Meraki Network Devices:

uv run log-analyzer meraki path/to/meraki.log

Export Reports

Export analysis to HTML (recommended for sharing):

uv run log-analyzer unifi path/to/unifi.log --export report.html

Export to Markdown:

uv run log-analyzer palo path/to/palo-alto.log --export report.md

Export to JSON (for programmatic access):

uv run log-analyzer unifi path/to/unifi.log --export report.json

Advanced Options

uv run log-analyzer palo path/to/palo-alto.log \
  --top 20 \
  --noise-threshold 10.0 \
  --export report.html

Command Line Options

Common Options (all formats)

Option	Short	Default	Description
`--top`		5-10*	Number of rows to show in each summary table
`--noise-threshold`		5.0	Flag entities whose event count is at least this percentage of total volume
`--progress` / `--no-progress`		`--progress`	Display a progress bar while parsing
`--export`	`-o`	None	Export report to file (.html, .md, or .json)

*Default is 5 for Palo Alto, 10 for UniFi

Report Sections

Palo Alto Networks Reports

Log Types by Volume - Distribution of traffic, system, and config logs
Log Subtypes - Breakdown of log subtypes (end, drop, deny, etc.)
Top Applications - Applications generating the most traffic
Top Policies - Firewall rules with the highest hit counts
Policy Actions - Distribution of allow/deny/reset actions
Session End Reasons - Why sessions terminated (aged-out, tcp-fin, etc.)
Source IPs - Noisiest traffic sources
Destination IPs - Most targeted destinations
Volume Trends - Events per minute over time
Noise Candidates - High-volume entities that may need filtering
Application Characteristics - Security attributes of applications

UniFi Reports

Log Levels - Distribution of info, error, warning, debug messages
Log Categories - System, kernel, wifi, network-services, etc.
Most Active Processes - Processes generating the most log entries
Hostnames - Active UniFi devices in your network
Volume Trends - Events per minute over time
Noise Candidates - High-volume processes and categories

WatchGuard Reports

Log Levels - Distribution of info, warning, error messages
Log Categories - Firewall, VPN, network-services, system, security, etc.
Most Active Processes - dhcpd, loggerd, sessiond, firewall, iked, etc.
Message IDs - Most common WatchGuard message identifiers
Devices - Active WatchGuard appliances
Volume Trends - Events per minute over time
Noise Candidates - High-volume processes, message IDs, and categories

Meraki Reports

Event Types - ip_flow_start, ip_flow_end, urls, firewall, events
Event Categories - Network flows, web security, DHCP, firewall, system events
Log Levels - Distribution of info, warning, error messages
Top Protocols - TCP, UDP, ICMP traffic distribution
Top Source/Destination IPs - Most active internal and external hosts
Top Destination Ports - Most accessed services
Volume Trends - Events per minute over time
Noise Candidates - High-volume event types, source/destination IPs

Log Formats

Palo Alto Networks

Standard syslog-formatted traffic logs with the syslog prefix:

Nov 4 11:08:44 109.2.165.203 1,2025/11/04 11:08:44,026701019653,TRAFFIC,end,...

UniFi

Standard syslog format from UniFi devices:

Nov 4 01:00:00 78.196.139.136 UniFi-Express-Gonzague systemd[1]: Starting service...

Also supports CEF (Common Event Format) security events:

Nov 4 01:22:08 78.196.139.136 CEF:0|Ubiquiti|UniFi Network|10.0.140|400|WiFi Client Connected|1|...

WatchGuard

WatchGuard syslog format with device ID, name, ISO timestamp, and message ID:

Nov 4 01:00:03 83.206.233.205 801304C6AA57D St-EgreveM370 (2025-11-04T00:00:03) firewall: msg_id="3001-1001" Temporarily blocking host...

Meraki

Meraki syslog format with epoch timestamp and key-value pairs:

Nov 5 00:00:04 90.102.85.18 1 1762300804.143040390 ROUTER ip_flow_end src=10.10.0.102 dst=35.153.85.208 protocol=udp sport=1043 dport=9930...

Supports event types:

ip_flow_start/end - Network flow tracking
urls - Web content filtering
firewall - Firewall rule actions
events - DHCP and system events

Performance

1M+ records per minute - Processed 1,044,470 Meraki records in ~60 seconds
Memory-efficient - Streaming parser handles large files
Optimized - Polars-based aggregations for speed
100% parse rates - Meraki (1M+ lines), WatchGuard (43K lines), UniFi (7.6K lines, 99.3%)

Examples

Quick Analysis

# Analyze Palo Alto logs
uv run log-analyzer palo firewall.log

# Analyze UniFi logs
uv run log-analyzer unifi unifi-device.log

Generate HTML Report

# Create a detailed HTML report with top 10 entries per section
uv run log-analyzer palo firewall.log --top 10 -o analysis-report.html

# Generate UniFi report
uv run log-analyzer unifi unifi-device.log --top 20 -o unifi-report.html

Identify Major Traffic Sources

# Lower noise threshold to catch more potential issues (Palo Alto)
uv run log-analyzer palo firewall.log --noise-threshold 2.0 --top 15

# Find noisy processes in UniFi logs
uv run log-analyzer unifi unifi.log --noise-threshold 3.0 --top 20

# Analyze WatchGuard firewall blocks
uv run log-analyzer watchguard watchguard.log --noise-threshold 5.0

Silent Processing

# Disable progress bar for scripting
uv run log-analyzer palo firewall.log --no-progress -o report.json
uv run log-analyzer unifi unifi.log --no-progress -o report.json

Output Examples

Console Output

Parsed 3,169,590 records (from 3,169,590 lines).

                              Log types by volume
┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Log Type              ┃             Events ┃                   Approx. Bytes ┃
┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ TRAFFIC               │          3,165,105 │                         69.6 GB │
│ SYSTEM                │              4,467 │                           0.0 B │
│ CONFIG                │                 18 │                           0.0 B │
└───────────────────────┴────────────────────┴─────────────────────────────────┘

HTML Report

The HTML export includes:

Professional styling with responsive tables
Color-coded sections
Hover effects for better readability
Summary statistics
Mobile-friendly layout

Development

Project Structure

log-analyzer/
├── log_analyzer/
│   ├── __init__.py
│   ├── cli.py                    # Main CLI and analysis logic
│   └── parsers/
│       ├── __init__.py           # Base parser classes
│       ├── palo_alto.py          # Palo Alto parser
│       ├── unifi.py              # UniFi parser
│       ├── watchguard.py         # WatchGuard parser
│       └── meraki.py             # Meraki parser
├── pyproject.toml                # Project configuration
├── README.md
└── .gitignore

Adding New Log Formats

To add support for a new log format:

Create a new parser in log_analyzer/parsers/your_format.py
Inherit from the LogParser base class
Implement the parse() and load_dataframe() methods
Add a new subcommand in cli.py
Create format-specific analysis sections

See existing parsers (palo_alto.py, unifi.py) for examples.

Running Tests

# Install development dependencies
uv sync --all-extras

# Run tests (if available)
pytest

Code Style

This project uses:

Type hints throughout
Dataclasses for structured data
Polars for high-performance data processing
Rich for beautiful console output
Typer for CLI interface

Troubleshooting

"No records parsed" error

Wrong format? - Make sure you're using the correct subcommand (palo vs unifi)
Verify log format - Check that the file matches the expected format for the parser
Syslog prefix - Ensure logs have the standard syslog prefix
File corruption - Verify the file is not corrupted or empty

Try running with --help on the subcommand for format-specific details:

uv run log-analyzer palo --help
uv run log-analyzer unifi --help

Many skipped lines

Some lines may be skipped if they don't match the expected format:

Check the "Skipped lines" summary in the output
format-mismatch - Lines that don't match the parser's expected pattern
empty - Blank lines in the file
missing-prefix - Lines without proper syslog prefix

A small number of skipped lines is normal, but if most lines are skipped, you may be using the wrong parser.

Out of memory errors

Process smaller log files or split large files
Increase system memory
Use --no-progress to reduce memory overhead

Slow performance

Ensure you're using Polars (not pandas)
Check disk I/O performance
Try running on files stored on SSD rather than network drives

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

Fork the repository
Create your feature branch (git checkout -b feature/amazing-feature)
Commit your changes (git commit -m 'Add amazing feature')
Push to the branch (git push origin feature/amazing-feature)
Open a Pull Request

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

Built with Polars for fast DataFrame operations
Uses Rich for beautiful terminal output
CLI powered by Typer

Support

For issues, questions, or contributions, please use the GitHub issue tracker.

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
log_analyzer		log_analyzer
tests		tests
.gitignore		.gitignore
.pre-commit-config.yaml		.pre-commit-config.yaml
LICENSE		LICENSE
MANIFEST.in		MANIFEST.in
README.md		README.md
pyproject.toml		pyproject.toml
uv.lock		uv.lock

Latest commit

History

Folders and files

Repository files navigation