Add duplicate filters with persisted summaries for 2.10.2

CHANGELOG.MD

@@ -5,6 +5,38 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0).
 
+## [Unreleased]
+
+## [2.10.2] - 2026-10-10
+
+### Changed
+
+- Duplicate statistics keep the file count visible and place supporting details behind a Details toggle.
+
+## [2.10.1] - 2026-10-10
+
+### Changed
+
+- Duplicate lists and statistics use persisted summaries. Until initialization completes, `/api/stats` returns null duplicate counts and duplicate-mode lists return HTTP 503 with `Retry-After: 30`. The API serves stale summaries with a status flag. Refresh targets 60 seconds under normal operation. Backlogs or outages can delay refresh; the stale flag reports delays over 60 seconds or failures.
+- Resizing across the mobile breakpoint redraws accepted results without another request. Stats loading no longer blocks table results, filters, or operation detection.
+
+## [2.10.0] - 2026-10-10
+
+### Added
+
+- Duplicate file counts and a Duplicates filter for recorded SHA-256 matches or same-filename candidates ([#88](https://github.com/ttlequals0/PixelProbe/issues/88)). Counts include every group member across inventory paths, excluding known missing files. Desktop rows and mobile cards show group sizes. Content matching excludes files without hashes and does not verify changes since the recorded baseline.
+
+### Fixed
+
+- Storage read failures, including stale file handles wrapped by the reader, are recorded as unreadable rather than generic scanner errors. Completed worker chunks report unverified file counts separately from corrupted files.
+- Application and worker logs use stdout so successful INFO messages are not labeled as errors merely because they were written to stderr.
+- Starting with `python app.py` reuses the running application module instead of importing it again through route dependencies and creating duplicate application state.
+
+### Security
+
+- Updated the frontend build dependency source-map-js to 1.2.2 for GHSA-68fv-2mgg-jv7q, which fixes denial of service from malicious indexed source maps.
+- Updated Werkzeug to 3.1.9 for CVE-2026-102598. On Windows, `safe_join` rejects NTFS special-device names with empty alternate data stream markers.
+
 ## [2.9.5] - 2026-10-05
 
 ### Fixed

README.md

@@ -23,6 +23,7 @@ Detection
 - FFmpeg validation for video and audio, plus PIL and ImageMagick decoding for images
 - Warning and corruption verdicts kept separate
 - Rolling hashes that flag content changes without a matching modification-time change
+- Duplicate file counts and filters for recorded SHA-256 matches or same-filename candidates. Filename matches may have different content; hashes do not verify changes since the recorded baseline.
 - Exclusions and per-finding mark-as-good overrides
 
 Scanning and recovery

app.py

@@ -7,6 +7,9 @@
 import sys
 import atexit
 import logging
+if __name__ == '__main__':
+    sys.modules['app'] = sys.modules[__name__]
+
 from datetime import datetime, timezone
 from flask import Flask, jsonify, send_file, render_template, request, redirect, url_for
 from flask_sqlalchemy import SQLAlchemy
@@ -57,6 +60,7 @@
 # Configure logging
 logging.basicConfig(
     level=logging.INFO,
+    stream=sys.stdout,
     format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
 )
 logger = logging.getLogger(__name__)

celery_worker.py

@@ -30,6 +30,7 @@
 # Configure logging
 logging.basicConfig(
     level=logging.INFO,
+    stream=sys.stdout,
     format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
 )
 logger = logging.getLogger(__name__)
@@ -67,6 +68,7 @@ def main():
         rc = celery_app.worker_main([
             'worker',
             '--loglevel', log_level.lower(),
+            '--logfile', '/dev/stdout',
             '--concurrency', str(concurrency),
             '--queues', 'pixelprobe',
             '--hostname', f'pixelprobe-worker@%h',

docker-compose.yml

@@ -71,7 +71,7 @@ services:
 
   # PixelProbe application
   pixelprobe:
-    image: ttlequals0/pixelprobe:${PIXELPROBE_VERSION:-2.9.5}
+    image: ttlequals0/pixelprobe:${PIXELPROBE_VERSION:-2.10.0}
     container_name: pixelprobe-app
     environment:
       # Security
@@ -167,7 +167,7 @@ services:
 
   # P1 Celery Worker for distributed task processing
   celery-worker:
-    image: ttlequals0/pixelprobe:${PIXELPROBE_VERSION:-2.9.5}
+    image: ttlequals0/pixelprobe:${PIXELPROBE_VERSION:-2.10.0}
     container_name: pixelprobe-celery-worker
     command: python celery_worker.py
     environment:

docs/api.md

@@ -204,6 +204,7 @@ Get paginated scan results with optional filters.
 - `is_corrupted` (string): Filter by corruption: `all`, `true`, `false`
 - `has_warnings` (string): Filter by warning flag: `all`, `true`, `false` (`true` excludes corrupted and marked-as-good files)
 - `bitrot_suspected` (string): Filter by suspected bitrot: `all`, `true`, `false`
+- `duplicate_mode` (string): `hash` for recorded SHA-256 matches, `name` for same-filename candidates, or `all` to disable (default). Invalid values return HTTP 400.
 - `search` (string): Case-insensitive substring match on file path
 - `path` (string): Restrict results to one configured scan path (must exactly match a configured path)
 - `sort_field` (string): Field to sort by (default: `scan_date`). Valid values: `scan_date`, `file_path`, `file_size`, `file_type`, `scan_status`, `status`, `is_corrupted`, `marked_as_good`, `scan_tool`, `corruption_details`, `discovered_date`, `last_modified`. `status` sorts by corruption status. Unknown values fall back to `scan_date` descending.
@@ -242,6 +243,24 @@ Get paginated scan results with optional filters.
 Notes:
 - `media_info` is a JSON-encoded **string**, not a nested object - parse it client-side (e.g. `json.loads(result["media_info"])`).
 - Items also carry warning, bitrot, and integrity fields: `has_warnings`, `warning_details`, `bitrot_suspected`, `last_integrity_check_date`, and related hash/baseline columns.
+- Duplicate filtering adds `duplicate_mode` and `duplicate_group_size` to each result. Groups span the inventory before path, search, and status filters apply. Only files recorded as existing are eligible. Content matching requires equal recorded hashes and sizes; suspected bitrot and failed or unreadable observations are excluded. Filename matching compares exact case-sensitive basenames. Matches use recorded baselines without verifying current contents; filename candidates may contain different content.
+
+Duplicate-mode responses include `duplicate_status` with `ready`, `updated_at`, and `stale`. `updated_at` records the UTC time of the last completed summary refresh. `ready: false` means no completed summary is available. `stale` is independent of `ready`. It becomes true after a refresh failure or when queued inventory changes have waited over 60 seconds. A stale summary is served, and its groups may differ from current eligible membership until refresh completes.
+
+While the summary is initializing, a request with `duplicate_mode=hash` or `duplicate_mode=name` returns HTTP 503 and `Retry-After: 30`:
+
+```json
+{
+  "error": "Duplicate index is initializing",
+  "duplicate_status": {
+    "ready": false,
+    "updated_at": null,
+    "stale": false
+  }
+}
+```
+
+The refresh target is 60 seconds under normal operation. A backlog or database or worker outage can delay refreshes; `stale` reports when changes have waited over 60 seconds or refresh has failed. Requests without a duplicate mode remain available while the summary initializes.
 
 #### Get single scan result
 ```http
@@ -442,6 +461,21 @@ GET /api/stats
 
 Get overall statistics about scanned files.
 
+Duplicate statistics use the same inventory groups as `duplicate_mode` filtering:
+
+| Field | Meaning |
+|---|---|
+| `duplicate_files` | All files in matching recorded SHA-256 groups |
+| `duplicate_groups` | Number of matching content groups |
+| `duplicate_extra_files` | Files beyond one per content group |
+| `filename_duplicate_files` | All files in same-filename candidate groups |
+| `filename_duplicate_groups` | Number of same-filename candidate groups |
+| `filename_duplicate_extra_files` | Files beyond one per filename group |
+
+Counts include only files recorded as existing. Content counts exclude absent hashes or sizes, suspected bitrot, and failed or unreadable observations. Filename counts do not prove matching contents. No fresh filesystem verification runs for these queries.
+
+The six duplicate count fields are numbers when a summary is ready and `null` while it initializes. `duplicate_status` reports readiness, the UTC refresh time, and whether refresh has failed or inventory changes have waited over 60 seconds. `stale` can be false while `ready` is false if the first refresh is still within that interval. A stale summary remains available, but its group counts may differ from current eligible membership until refresh completes. The refresh target is 60 seconds under normal operation. Backlog or an outage can delay it, with delayed or failed refresh reported by `stale`. The statistics endpoint remains available during initialization.
+
 **Response:**
 ```json
 {
@@ -454,6 +488,17 @@ Get overall statistics about scanned files.
   "healthy_files": 940,
   "marked_as_good": 3,
   "warning_files": 7,
+  "duplicate_files": null,
+  "duplicate_groups": null,
+  "duplicate_extra_files": null,
+  "filename_duplicate_files": null,
+  "filename_duplicate_groups": null,
+  "filename_duplicate_extra_files": null,
+  "duplicate_status": {
+    "ready": false,
+    "updated_at": null,
+    "stale": false
+  },
   "integrity": {
     "total_files": 1000,
     "checked_files": 800,

openapi.yaml

@@ -175,6 +175,21 @@ components:
         type: integer
 
   schemas:
+    DuplicateStatus:
+      type: object
+      required: [ready, updated_at, stale]
+      properties:
+        ready:
+          type: boolean
+          description: Whether a completed duplicate summary is available.
+        updated_at:
+          type: string
+          format: date-time
+          nullable: true
+          description: UTC time of the last completed summary refresh, or null before the first refresh.
+        stale:
+          type: boolean
+          description: True when refresh has failed or inventory changes have waited over 60 seconds for processing. This is independent of ready.
     NotificationProviderInput:
       type: object
       required: [name, provider_type, configuration]
@@ -841,6 +856,20 @@ paths:
           in: query
           schema:
             type: string
+        - name: duplicate_mode
+          in: query
+          description: >-
+            Hash groups require files recorded as existing, nonempty recorded
+            SHA-256 hashes, and equal sizes. Suspected bitrot and failed or
+            unreadable observations are excluded. Name groups use exact
+            case-sensitive basenames and are candidates only. Groups span
+            the inventory before search and path filters. Each matching row
+            includes duplicate_mode and duplicate_group_size. Hashes describe
+            recorded baselines without verifying current contents.
+          schema:
+            type: string
+            enum: [all, hash, name]
+            default: all
         - name: sort_field
           in: query
           schema:
@@ -859,7 +888,52 @@ paths:
             type: string
       responses:
         '200':
-          description: Scan results retrieved successfully
+          description: Scan results retrieved successfully. Duplicate-mode responses include duplicate_status.
+          content:
+            application/json:
+              schema:
+                type: object
+                properties:
+                  results:
+                    type: array
+                    items:
+                      type: object
+                      properties:
+                        duplicate_mode:
+                          type: string
+                          enum: [hash, name]
+                        duplicate_group_size:
+                          type: integer
+                  total:
+                    type: integer
+                  page:
+                    type: integer
+                  per_page:
+                    type: integer
+                  pages:
+                    type: integer
+                  duplicate_status:
+                    $ref: '#/components/schemas/DuplicateStatus'
+                additionalProperties: true
+        '503':
+          description: Duplicate summary is initializing. Returned only when duplicate_mode is hash or name and no completed summary is ready.
+          headers:
+            Retry-After:
+              description: Seconds before retrying.
+              schema:
+                type: string
+              example: '30'
+          content:
+            application/json:
+              schema:
+                type: object
+                required: [error, duplicate_status]
+                properties:
+                  error:
+                    type: string
+                    example: Duplicate index is initializing
+                  duplicate_status:
+                    $ref: '#/components/schemas/DuplicateStatus'
 
   /scan-results/{result_id}:
     get:
@@ -1460,6 +1534,15 @@ paths:
         never-checked count, the oldest check date (every checked file has
         been verified at least once since then), and the current
         bitrot-suspected count.
+        `duplicate_files` counts all files in recorded hash-and-size groups;
+        `duplicate_groups` counts those groups and `duplicate_extra_files`
+        counts copies beyond one per group. `filename_duplicate_files`,
+        `filename_duplicate_groups`, and `filename_duplicate_extra_files`
+        use exact case-sensitive basenames and describe candidates, not
+        confirmed content matches. Missing files are excluded. Hash matching
+        also excludes suspected bitrot, failed or unreadable observations,
+        absent hashes, and absent sizes. Hashes describe recorded baselines,
+        not fresh filesystem verification.
 
         **Example:**
         ```bash
@@ -1470,6 +1553,32 @@ paths:
       responses:
         '200':
           description: Statistics retrieved successfully
+          content:
+            application/json:
+              schema:
+                type: object
+                properties:
+                  duplicate_files:
+                    type: integer
+                    nullable: true
+                  duplicate_groups:
+                    type: integer
+                    nullable: true
+                  duplicate_extra_files:
+                    type: integer
+                    nullable: true
+                  filename_duplicate_files:
+                    type: integer
+                    nullable: true
+                  filename_duplicate_groups:
+                    type: integer
+                    nullable: true
+                  filename_duplicate_extra_files:
+                    type: integer
+                    nullable: true
+                  duplicate_status:
+                    $ref: '#/components/schemas/DuplicateStatus'
+                additionalProperties: true
 
   /stats/trends:
     get: