Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Changelog

All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

## [0.1.2] - 2026-10-20

### Added

- Commit message rules based on Conventional Commits (`docs/COMMITRULES.md`).
- Unit tests for the chardet fallback in `detect_file_encoding`: use of the top-level `chardet.UniversalDetector`, operation without the `chardet.universaldetector` submodule, 1024-byte chunk feeding, early stop when the detector is done, and `None` when no encoding is detected.

### Changed

- Raise the minimum `chardet` version to 7.0.0 (`chardet>=7.0.0`); the lockfile now pins chardet 7.7.0.

### Fixed

- Use `chardet.UniversalDetector` instead of `chardet.universaldetector.UniversalDetector` so encoding detection works with chardet 7.x, which no longer exposes the `chardet.universaldetector` submodule.
- Mermaid diagram rendering in the docs.

[Unreleased]: https://github.com/RyoNakagami/encoding-checker-cli/compare/v0.1.2...HEAD
[0.1.2]: https://github.com/RyoNakagami/encoding-checker-cli/compare/v0.1.1...v0.1.2
15 changes: 1 addition & 14 deletions docs/BRANCH_STRATEGY.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,20 +16,7 @@ project: encoding-checker-cli
- Follow clear naming conventions for branches to enhance collaboration and maintainability.

```mermaid
%%| fig-width: 16
%%| fig-height: 12
%%{ init: {
'theme': 'default',
'themeVariables': {
'fontFamily': 'Meiryo'
},
'gitGraph': {
'orientation': 'LR',
'nodeSpacing': 150,
'mainBranchName': 'main'
}
} }%%
$ gitGraph
gitGraph
commit id: "" tag: "v1.0.0"

branch hotfix
Expand Down
100 changes: 100 additions & 0 deletions docs/COMMITRULES.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
# Commit Rules

- This project follows the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) specification.
- A consistent commit history makes the project easier to read, enables automated changelog generation, and clarifies the intent of every change.

## Commit Message Format

```text
<type>(<scope>): <description> ← single line, mandatory

[optional body]

[optional footer(s)]
```

- The **header** is the first line and is mandatory — it is always a **single line**.
- The **scope** is optional.
- A blank line separates the header from the body, and the body from the footer.

### Example

```text
feat(parser): add support for nested config blocks

The parser now recurses into nested tables so that deeply
structured pyproject.toml files are read correctly.

Closes #42
```

## Type

The `<type>` describes the kind of change and must be one of the following:

| Type | Description |
| ---------- | -------------------------------------------------------------------- |
| `feat` | A new feature |
| `fix` | A bug fix |
| `docs` | Documentation-only changes |
| `style` | Changes that do not affect meaning (formatting, whitespace, etc.) |
| `refactor` | A code change that neither fixes a bug nor adds a feature |
| `perf` | A code change that improves performance |
| `test` | Adding missing tests or correcting existing tests |
| `build` | Changes to the build system or external deps (e.g. `uv`, pyproject) |
| `ci` | Changes to CI configuration files and scripts |
| `chore` | Other changes that don't modify `src` or test files |
| `revert` | Reverts a previous commit |

## Scope

The `<scope>` is optional and provides additional context about the section of the
codebase affected, e.g. `feat(glossary):`, `fix(quarto):`, `docs(commitrules):`.
Use a short, lowercase noun. Omit the scope when the change is global.

## Description

- Keep the description on a **single line** — never wrap it onto multiple lines.
- Use the **imperative, present tense**: "add", not "added" or "adds".
- Do **not** capitalize the first letter.
- Do **not** end with a period.
- Keep the header concise — aim for **50 characters or fewer**, and a hard limit of 72.
- If more explanation is needed, put it in the **body**, not the description.

## Body

- Optional. Use it to explain **what** and **why**, not **how**.
- Wrap lines at **72 characters**.
- Separate from the header with one blank line.

## Footer

- Optional. Used for metadata such as issue references and breaking changes.
- Reference issues with `Closes #<id>`, `Fixes #<id>`, or `Refs #<id>`.

### Breaking Changes

A breaking change is indicated either by appending a `!` after the type/scope, or
by a `BREAKING CHANGE:` footer (or both).

```
feat(api)!: drop support for Python 3.9

BREAKING CHANGE: the minimum supported Python version is now 3.10.
```

## Quick Reference

```
feat: a new feature
fix: a bug fix
docs: documentation only
style: formatting, no code change
refactor: code change, no feature/fix
perf: performance improvement
test: adding or fixing tests
build: build system or dependencies
ci: CI configuration
chore: maintenance / tooling
revert: revert a previous commit
```
4 changes: 2 additions & 2 deletions pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
[project]
name = "encoding-checker-cli"
version = "0.1.1"
version = "0.1.2"
description = "A command-line tool for checking and converting text encoding."
readme = "README.md"
authors = [
{ name = "RyoNak", email = "[email protected]" }
]
requires-python = ">=3.13"
dependencies = [
"chardet>=5.2.0",
"chardet>=7.0.0",
"typer>=0.20.0",
]

Expand Down
2 changes: 1 addition & 1 deletion src/encoding_checker_cli/encoding.py
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ def detect_file_encoding(file_path: Path) -> str | None:
pass

# 2. Use chardet as fallback
detector = chardet.universaldetector.UniversalDetector()
detector = chardet.UniversalDetector()
for chunk_start in range(0, len(raw_bytes), 1024):
chunk = raw_bytes[chunk_start : chunk_start + 1024]
detector.feed(chunk)
Expand Down
96 changes: 96 additions & 0 deletions tests/test_detect_file_encoding.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import pytest
from pathlib import Path
from encoding_checker_cli import encoding as encoding_module
from encoding_checker_cli.encoding import detect_file_encoding, JAPANESE_ENCODINGS

def test_utf8(tmp_path):
Expand Down Expand Up @@ -67,3 +68,98 @@ def fake_read_bytes(self):

# Restore original read_bytes method if needed
monkeypatch.setattr(Path, "read_bytes", original_read_bytes)


class FakeDetector:
"""Minimal stand-in for chardet.UniversalDetector."""

instances = []

def __init__(self, encoding="shift_jis", done_after=None):
self.result = {"encoding": encoding}
self.done = False
self.closed = False
self.chunks = []
self._done_after = done_after
FakeDetector.instances.append(self)

def feed(self, chunk):
self.chunks.append(chunk)
if self._done_after is not None and len(self.chunks) >= self._done_after:
self.done = True

def close(self):
self.closed = True


@pytest.fixture
def fake_detector(monkeypatch):
"""Replace chardet.UniversalDetector with a configurable fake."""
FakeDetector.instances = []

def install(**kwargs):
monkeypatch.setattr(
encoding_module.chardet, "UniversalDetector", lambda: FakeDetector(**kwargs)
)
return FakeDetector.instances

return install


def test_fallback_uses_top_level_universal_detector(tmp_path, fake_detector):
"""The chardet fallback is driven by chardet.UniversalDetector."""
instances = fake_detector(encoding="SHIFT_JIS")
file = tmp_path / "sjis.txt"
file.write_bytes("テストテスト".encode("shift_jis"))

result = detect_file_encoding(file)

assert result == "shift_jis"
assert len(instances) == 1
assert instances[0].closed


def test_fallback_without_universaldetector_submodule(tmp_path, monkeypatch):
"""Detection works even if chardet does not expose the universaldetector submodule."""
monkeypatch.delattr(encoding_module.chardet, "universaldetector", raising=False)
file = tmp_path / "sjis.txt"
file.write_bytes("テストテスト".encode("shift_jis"))

result = detect_file_encoding(file)

assert result in JAPANESE_ENCODINGS


def test_fallback_feeds_in_1024_byte_chunks(tmp_path, fake_detector):
"""Bytes are fed to the detector in 1024-byte chunks until EOF."""
instances = fake_detector()
raw = b"\x82" * 2500 # invalid UTF-8, forces the chardet fallback
file = tmp_path / "big.bin"
file.write_bytes(raw)

detect_file_encoding(file)

chunks = instances[0].chunks
assert [len(c) for c in chunks] == [1024, 1024, 452]
assert b"".join(chunks) == raw


def test_fallback_stops_feeding_when_detector_done(tmp_path, fake_detector):
"""Feeding stops as soon as the detector reports it is done."""
instances = fake_detector(done_after=1)
file = tmp_path / "big.bin"
file.write_bytes(b"\x82" * 2500)

detect_file_encoding(file)

assert len(instances[0].chunks) == 1
assert instances[0].closed


def test_fallback_returns_none_when_detector_has_no_result(tmp_path, fake_detector):
"""Return None if the detector cannot determine an encoding."""
fake_detector(encoding=None)
file = tmp_path / "bad.bin"
file.write_bytes(b"\x82\xff\x00\x81")

assert detect_file_encoding(file) is None
38 changes: 32 additions & 6 deletions uv.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading