A Python library and CLI that discovers, classifies, and catalogues files by inspecting their content.
File Discovery provides a reusable module for scanning large, mixed file collections and producing a deterministic, structured catalogue suitable for downstream processing, analysis, or organisation.
Operators and engineers gain a reliable, content-aware inventory of their files without relying on naming
conventions, directory layout, or file extensions. The streaming NDJSON output integrates into automation
pipelines, while the companion file-report tool converts results to CSV for spreadsheet review.
Unstructured file collections (videos, photos, music, documents, archives) are difficult to catalogue accurately. Extensions are often missing, misleading, or inconsistent. Manual classification is impractical at scale, and generic tools do not produce stable, machine-readable output for automation.
The module recursively traverses a directory tree, inspects file content using MIME detection and
format-specific validators, assigns each file to a category, extracts metadata (size, timestamps, xxHash
content hash), and emits deterministic NDJSON output. A separate file-report CLI transforms NDJSON to CSV
with selectable fields.
- Python 3.14 or later
- libmagic (for MIME type detection)
- uv
git clone https://github.com/stefaniuk/file-discovery.git
cd file-discovery
make env
uv run --python 3.14 pip install -e .file-discovery --root tests/file_discovery/integration/dataNDJSON streams to STDOUT. No logs are emitted unless you enable them with --log-console or --log-file.
- Recursive directory scanning with deterministic ordering.
- Content-based classification using MIME detection and format validators.
- Metadata extraction (size, timestamps, xxHash content hash).
- Streaming NDJSON output (one JSON object per line).
- CSV export through
file-report. - CLI and Python API entry points.
- File system search, indexing, or query capabilities.
- Deduplication or similarity detection.
- Thumbnail generation or media transcoding.
- Persistent storage (databases, object stores).
- Interactive or graphical interfaces.
- Configuration file parsing (command-line arguments only).
- Receive a root path via the CLI or API.
- Traverse the directory tree depth-first with deterministic ordering.
- Inspect file content using libmagic and format-specific validators.
- Extract metadata, including size, timestamps, and xxHash content hash.
- Emit NDJSON records for file and error entries as they are processed.
- Emit a summary record after traversal completes.
- Optionally convert NDJSON to CSV using
file-report csv.
Key terms (canonical definitions):
- Category: file type classification label.
- Content hash: xxh64 digest used for stable content identification.
- NDJSON: newline-delimited JSON output format.
- Naming convention: CLI commands use hyphens; Python packages use underscores.
Definitions are in specs/001-file-discovery-classification/spec.md and specs/003-file-report/spec.md.
The CLIs are configured entirely via command-line arguments. There are no configuration files.
See the CLI contracts for the full argument lists, exit codes, and I/O behaviour:
Naming convention: CLI commands use hyphens (
file-discovery,file-report); Python packages use underscores (file_discovery,file_report).
# Scan a directory and stream NDJSON to STDOUT
file-discovery --root tests/file_discovery/integration/data
# Scan a directory and write NDJSON to a file
file-discovery --root /data --output /tmp/catalogue.ndjson
# Scan with logging to a separate file
file-discovery --root /data --output /tmp/catalogue.ndjson --log /tmp/file-discovery.log
# Enable console logging
file-discovery --root /data --log-console
# Convert NDJSON to CSV with default fields
file-discovery --root /data | file-report csv > report.csv
# Convert with specific fields
file-report csv --input /tmp/catalogue.ndjson --output report.csv --fields path,content_hashPython API: scan a directory and write NDJSON.
from file_discovery.api import scan_directory
from file_discovery.config import ScanConfig
with open("catalogue.ndjson", "w", encoding="utf-8") as output:
summary = scan_directory("/path/to/scan", output, config=ScanConfig())
print(f"Processed {summary.processed_files} files")Python API: convert NDJSON to CSV.
from file_report.api import convert_to_csv
with open("catalogue.ndjson", "r", encoding="utf-8") as input_file:
with open("report.csv", "w", encoding="utf-8") as output_file:
count = convert_to_csv(input_file, output_file, fields=["path", "content_hash"])
print(f"Converted {count} records")Exit codes and error handling are defined in the CLI contracts:
- Architecture diagrams: docs/codebase-overview/c4-context.md, docs/codebase-overview/c4-container.md, docs/codebase-overview/data-flow.md
- Specifications: specs/
- ADRs: docs/adr/
See the contributing guide for full guidelines on development workflow, code style, and pull request expectations.
make env
make deps-dev
make configRun the repository quality gates before submitting changes:
make lint
make testRun make help to see the full list of targets.
- Create a feature branch from
main. - Run
make lintandmake testbefore committing. - Open a pull request with a clear description of changes.
- src/file_discovery/ - Core library: traversal, classification, metadata, NDJSON.
- src/file_report/ - Report module: NDJSON to CSV conversion.
- tests/ - Unit, integration, contract, and performance tests.
- specs/ - Feature specifications (authoritative requirements).
- docs/adr/ - Architecture Decision Records.
- docs/codebase-overview/ - C4 and data flow diagrams.
- docs/gitignore-provenance.md - Gitignore provenance and local additions.
- scripts/ - Benchmark scripts.
- Makefile - Development workflow commands.
MIT Licence - see LICENCE.md