Add a standalone Remove Duplicates button - #3
Merged
Merged
Conversation
Scans a folder for images with byte-identical content (exact SHA256 hash match) and removes every copy but one, independent of any download job. Keeper selection is deterministic: alphabetically-first filename in each duplicate group. - app/dedup.py: hash_file() and find_duplicate_groups() -- shares the IMAGE_EXTENSIONS set with caption-folder scanning, now moved to app/download.py so both modules import it from one place. - app/jobs.py: create_dedup_job()/run_dedup_job() reuse the existing JobState/WebSocket progress machinery. Removed duplicates (and any orphaned .txt caption for them) go to the OS Recycle Bin via send2trash, not a permanent delete -- this is a bulk action on a user's dataset, so it stays recoverable. The keeper's own caption is left untouched. - app/main.py: POST /api/dedup-folder endpoint. - static/index.html + app.js: new "Remove duplicate images" card with a folder picker, recursive toggle, and a confirm() dialog before running (explains the Recycle Bin recoverability) since this deletes across a whole folder at once. - README.md + docs/landscape-and-roadmap-notes.md: documented the new feature and marked the roadmap's exact-hash de-dup item as shipped (perceptual/similarity-based de-dup for near-duplicates remains open). Live-verified beyond the 217 unit/integration tests (12 new for app/dedup.py, plus jobs/main coverage for the job kind and endpoint): downloaded real images from e621, manufactured file-level duplicates plus an orphaned caption, ran the actual /api/dedup-folder endpoint, and confirmed both the resulting stats/remaining files on disk AND that the removed files genuinely landed in the real Windows Recycle Bin (checked via the Shell.Application COM Recycle Bin namespace) -- then repeated the same scenario end-to-end through the real browser UI (form fill + button click, with window.confirm auto-accepted) to confirm the full client-to-server path, not just the API in isolation. Co-Authored-By: Claude Sonnet 5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scans a folder for images with byte-identical content (exact SHA256 hash match) and removes every copy but one, independent of any download job. Keeper selection is deterministic: alphabetically-first filename in each duplicate group.
Live-verified beyond the 217 unit/integration tests (12 new for app/dedup.py, plus jobs/main coverage for the job kind and endpoint): downloaded real images from e621, manufactured file-level duplicates plus an orphaned caption, ran the actual /api/dedup-folder endpoint, and confirmed both the resulting stats/remaining files on disk AND that the removed files genuinely landed in the real Windows Recycle Bin (checked via the Shell.Application COM Recycle Bin namespace) -- then repeated the same scenario end-to-end through the real browser UI (form fill + button click, with window.confirm auto-accepted) to confirm the full client-to-server path, not just the API in isolation.