Repository navigation
Dev refactor - #31
Open
safesploit wants to merge 48 commits into
Open
Dev refactor#31safesploit wants to merge 48 commits into
safesploit wants to merge 48 commits into
Conversation
Introduce Composer foundation and testing/CI scaffolding: add composer.json (PSR-4 autoload), phpunit.xml and an initial unit test, and commit composer.lock. Add GitHub Actions workflow (.github/workflows/ci.yml) to validate composer and run tests. Add ARCHITECTURE.md documenting modernization plan and update .gitignore to exclude vendor, build, .env and PHPUnit caches.
Introduce app/Crawl/UrlNormalizer, app/Search/FieldFormatter and app/Search/Paginator and corresponding unit tests. Refactor Crawler, crawl.php and crawl-manual.php to use UrlNormalizer and add composer autoload bootstrap in several entry points. Update SiteResultsProvider and ImageResultsProvider to accept injected Paginator (and FieldFormatter for sites), use the new helpers for offset/trim logic, and register autoloading. Adjust composer script to analyse ./app and tweak phpstan configuration. Add tests for normalizer, formatter and paginator.
Introduce a Doogle\Database\ConnectionFactory to build PDO connections from environment variables (supports explicit DB_DSN, fallback parts, charset and mapped error modes). Add fromProjectRoot() which loads a .env via vlucas/phpdotenv. Update config.php to use the new factory and Composer autoload. Add .env.example and unit tests covering DSN construction, explicit DSN override, PDO creation and loading a .env file. Also add vlucas/phpdotenv to composer.json.
Introduce Doogle\Repository\ImageRepository and SiteRepository with search/count and click/broken update methods. Refactor ImageResultsProvider and SiteResultsProvider to use these repositories (injectable) and update ajax endpoints (setBroken, updateImageCount, updateLinkCount) to call repository methods. Add integration tests for both repositories using in-memory SQLite and include an Integration testsuite in phpunit.xml. This centralizes DB access and makes providers easier to test.
Introduce search layer and tests: add Search DTOs (SearchResult, ImageResult), page DTOs (SearchPage, ImageSearchPage) and services (SearchService, ImageSearchService) plus repository interfaces (SiteSearchRepository, ImageSearchRepository). Update SiteRepository and ImageRepository to implement the new interfaces and refactor SiteResultsProvider/ImageResultsProvider to use the new services and DTOs. Add unit tests for DTOs and service behavior (SearchDtoTest, SearchServiceTest, ImageSearchServiceTest) and a paginator-based paging integration. Overall prepares a clearer separation between repositories and search presentation logic.
Add a comprehensive ARCHITECTURE.md that documents the current system, target architecture, phased modernisation plan, testing strategy, CI/CD, security and crawler hardening, Composer/PSR-4 adoption, Docker/runtime guidance, and SBOM/supply-chain controls. Update .gitignore to stop ignoring ARCHITECTURE.md so the new document is tracked in the repository.
Introduce a Docker-based development stack and helper scripts. Adds .dockerignore, docker/compose.yml, Dockerfile (PHP 8.3 + Apache), Apache vhost, and scripts (build, up, down, shell, test, reset-db) plus a README to document usage. The compose setup includes MySQL, phpMyAdmin, volumes for vendor and DB, and healthchecks. Also make a small unit test fix: wrap ImageResult instantiations in parentheses in SearchDtoTest so displayText() is called on the created instance.
Introduce CrawlerSecurityPolicy, PrivateNetworkBlocker, and UrlValidator to centralize crawler configuration and enforce security (allowed schemes, depth, private/reserved networks, user-agent, timeouts, max response bytes). Refactor Crawler to use the new policy and validator, add URL/image validation, queue-based crawl logic, deduplication and page limits. Update DomDocumentParser to respect policy (user-agent, timeout, max response bytes). Add env.example and docker compose vars for new settings, and include unit tests for the policy, blocker, and validator.
Enhance CI and developer tooling, add reusable test doubles, and tidy minor code. Updates GitHub Actions to grant minimal permissions, install pdo_sqlite, run static analysis, formatting (PHPCS) and generate/upload a CycloneDX SBOM. composer.json adds php_codesniffer and cyclonedx plugin plus composer scripts for analyse, lint, and sbom; composer.lock updated accordingly. README updated with a "Modernisation Changes" section and local quality check commands. Tests: extract duplicated fake Site/Image repository classes into standalone test fakes and remove the inline duplicates from unit tests. Small formatting cleanup in CrawlerSecurityPolicy for improved readability.
Introduce relevance-based search ranking for sites and images that combines full-text MATCH() scoring (when MySQL is used) with weighted field matches and a bounded click-count boost. Repository classes (SiteRepository, ImageRepository) now build ranking and WHERE SQL, bind full-text and LIKE terms, and order results by a computed rankingScore then clicks/id. Added a migration and updated schema seed to create FULLTEXT indexes, updated tests to assert ranking behavior, and documented the change in the README. Also minor docker comment update.
Move all browser-accessible entrypoints and assets into public/ (index.php, search.php, ajax/*, assets/*) and update PHP includes to use __DIR__/../.. style paths. Update Docker Apache config to serve /var/www/html/public as the DocumentRoot and adjust directory options. Update README and docker/README to reflect the new web root and adjust Composer PHPStan analyse path to include ./public. These changes prepare the repo for a proper public web root while keeping the full repository mounted for tests and Composer operations.
Introduce an authentication layer and CLI/web crawl tooling. Adds app/Auth (User, UserRepository, AuthService, SessionAuth) with password hashing and session handling, CSRF token support, and migration to update the users schema (role, created_at, unique keys). Implements crawl domain: app/Crawl (CrawlService, CrawlRequest, CrawlResult, CliCrawlCommand) plus UrlValidator integration, legacy crawler wrapper, and public pages (login.php, logout.php, crawl.php). Add CLI helpers (bin/create-admin, bin/crawl), Docker helpers (docker/create-admin.sh, docker/crawl.sh) and update docker/up.sh to optionally create an initial admin. Update .env.example and docs (README-refactored.md, docker/README.md) with admin/crawl usage. Add comprehensive unit and integration tests and adjust composer scripts to include bin in static checks.
Replace legacy crawler with a modern, testable CrawlService and job model. Adds app/Crawl/CrawlService.php (new DOM-based crawler with injectable page fetcher, page/image indexing, policy checks and output recording), app/Crawl/CrawlJob.php (immutable DTO), and app/Repository/CrawlJobRepository.php (CRUD for crawl_jobs). Removes legacy classes and scripts under classes/ and root crawl entrypoints, and adds a migration to create the crawl_jobs table. README updated to reflect new workflows (web UI vs trusted CLI), Docker usage, and migration steps; tests for crawl job and service were also added/updated. This restructures crawling to support queued jobs, history, and better separation of concerns for maintainability and testing.
CLI crawling now records jobs to the crawl_jobs table. CliCrawlCommand accepts an optional CrawlJobRepository factory, creates a job before crawling, marks it running, and records the final result (completed/failed/rejected) via a new recordJobResult helper; errors updating history are caught and logged so they don't block crawling. The bin/crawl script is updated to pass a CrawlJobRepository, tests were added/updated to assert job records (including requested_by_user_id = NULL), and README text was clarified to state that both web and CLI crawl history are stored in crawl_jobs.
Doogle v1.2 Introduce a videos search vertical: add DB migration and videos table, update crawl_jobs with videos_indexed, and add VideoRepository + VideoSearchRepository. CrawlService now extracts, normalises, deduplicates and inserts discovered video references and tracks videosIndexed in CrawlResult/CrawlJob, with helper methods for candidates, validation and existence checks. Add search layer (VideoResult, VideoSearchPage, VideoSearchService) and wire into public search (search.php), plus an AJAX endpoint to increment video click counts. Update UI/JS/CSS for video thumbnail grid and minor docker/script/help text tweaks; update docs and tests to cover the new vertical. Tracks clicks and integrates ranking/ pagination similar to other search types.
Introduce production-hardening features and switch the local Docker runtime to Nginx+PHP-FPM+MariaDB. Add security primitives (RateLimiter, RateLimitDecision, SecurityEventLogger, SessionCookiePolicy) and wire them into auth, web crawl, and CLI crawl flows (SessionAuth, public/login.php, public/crawl.php, CliCrawlCommand). Update Docker files and scripts to run a split runtime (nginx config, php:fpm Dockerfile, compose.yml, helper lib) and run PHP as non-root; add docker helper scripts and adjust helper commands. Expand CI with CodeQL and security scans. Update docs (ARCHITECTURE2.md, README-refactored.md, docker/README.md) and add unit tests for the new security components.
Document a file-backed, sliding-window rate limiter and related security logging. Adds sections describing IP-based rate limiting (login: IP+username; crawl: user ID or IP), attempt pruning, Retry-After behavior, and caveats (single-server only, uses $_SERVER['REMOTE_ADDR'], and development notes). Introduces DOOGLE_RATE_LIMIT_DIR default and reiterates DOOGLE_SECURITY_LOG for event logging. Changes applied to ARCHITECTURE2.md and README-refactored.md to explain implementation, configuration, and deployment considerations.
Introduce a new Phase J - Search Ranking section in ARCHITECTURE2.md describing the goal to improve search quality
Add a new Doogle\Search\RankingExpression utility to centralize SQL ranking snippets (weighted equals/like, non-empty, HTTPS, bounded click boost, capped MySQL full-text, and sum). Update Site, Image, and Video repositories to use the new expressions, adjust per-vertical weights and quality signals, and replace inline score aggregation. Add integration tests to verify click-boost capping so clicks cannot dominate relevance, plus unit tests for RankingExpression. Documentation updated (ARCHITECTURE2.md, README-refactored.md) to describe the deterministic ranking formula and capped boosts.
Introduce search analytics and admin-tunable ranking weights. Added RankingSettings, RankingWeight and multiple Search analytics DTOs under app/Search; repositories RankingSettingsRepository and SearchAnalyticsRepository under app/Repository; migration database/migrations/005_create_search_analytics.sql and updated doogle-tables-no-data.sql. Public changes: new admin page public/ranking.php, record searches and load ranking overrides in public/search.php, minor crawl UI link and CSS for admin UI. Repositories (Site/Image/Video) now read weights from RankingSettings and apply full-text caps from settings. Tests and docs updated (ARCHITECTURE2.md, README-refactored.md, new integration/unit tests) and analytics store only hashed IP/user-agent to protect raw identifiers. Searches continue to work if DB migration is not applied; failures fall back to defaults.
Large UI overhaul for the admin area and related pages. Added a comprehensive admin design system in style.css (topbar, panels, forms, tables, badges, alerts, responsive rules, metric/stat components, empty state, and utility classes). Updated crawl.php to use the new admin layout (adminTopbar, adminShell, panels), added statusCssClass(), improved crawl form, output disclosure, stats grid and history table with status badges. Updated ranking.php to use the admin layout, wrap tables in adminTableWrap, use adminEmptyState messages, expose metrics and tabs, and apply the new styles and accessible attributes. Mostly presentation and markup changes with a small helper for status-to-class mapping.
Update container base images used by the project: - docker/compose.yml: nginx from 1.27-alpine -> 1.31-alpine, mariadb from 11.4 -> 11.8 (still overridable via DOOGLE_* env vars) - docker/php/Dockerfile: php from 8.3-fpm -> 8.4-fpm - set GitHub Actions PHP to 8.4 These bumps bring newer upstream releases (security/feature updates). Validate compatibility with PHP 8.4 and MariaDB 11.8 in CI or local testing.
Add a docker_compose wrapper that prefers the Docker Compose v2 plugin (`docker compose`) and falls back to legacy `docker-compose` when necessary, printing an error and exiting 127 if neither is available. Extract v2/v1 invocations into docker_compose_v2 and docker_compose_v1 helper functions and preserve env-file handling. Update docker/README.md to document the preference and fallback behavior.
Replace the top-level 'name' key with legacy 'version: "2.4"' in docker/compose.yml and update docker/lib/compose.sh to derive a project_name from DOOGLE_COMPOSE_PROJECT (default: doogle). The script now passes -p "$project_name" to both 'docker compose' and 'docker-compose' invocations (with and without --env-file) to ensure a consistent project name and compatibility with Compose v2.
Add an nginx location block for /assets to serve static files (try_files + access_log off) in docker/nginx/default.conf. Update public/crawl.php and public/ranking.php to use an absolute favicon path (/assets/...) and append a version query string to the stylesheet (v=admin-20260616) to force CSS cache refresh.
Introduce optional TOTP 2‑factor auth for admins and an admin login audit trail. Adds TotpService and QrCodeSvg for secret generation and local SVG provisioning, updates SessionAuth to support a pending TOTP challenge, and extends UserRepository with TOTP columns/helpers (enable/clear/secret checks) while keeping schema-fallbacks. Adds AdminLoginEvent model and AdminLoginEventRepository to record recent successful/failed admin logins, a migration (006_add_admin_totp_and_login_events.sql) and updated schema dump, CLI and Docker reset helpers (bin/admin-reset-totp, docker/admin-reset-totp.sh), and RateLimiter support for a new 'totp' scope. Also updates public UI links and styles, docker compose/env defaults, docs/README/ARCHITECTURE to describe flow and operational commands, and adds related tests. Purpose: provide optional per-account TOTP after password verification, separate rate-limiting for TOTP, auditing of admin logins, and a trusted CLI recovery path.
Require the existing password before changing an admin password or disabling TOTP and rate-limit those confirmation attempts. Adds DOOGLE_PASSWORD_CONFIRM_RATE_LIMIT_* env vars (and docker/compose defaults) and documents the new limits in ARCHITECTURE2.md, README-refactored.md, docker/README.md and .env.example. Key changes: - public/security.php: enforce current-password checks for disabling TOTP and for a new change-password action, use a password_confirm RateLimiter, log rate-limited and failure events, and render a password-change form in the admin UI. - app/Auth/UserRepository.php: add updatePassword(id, plainPassword) to update stored password hashes. - tests/Integration/Auth/UserRepositoryTest.php: add testUpdatePasswordReplacesStoredHash to verify updatePassword behavior. - public/assets/css/style.css: add adminStackedForm/adminFieldGrid/adminField styles and wire inputs to those classes for the new form layout and responsiveness. - docker/compose.yml and .env.example: add password confirm rate limit env keys and defaults. - Docs updated to describe that password changes and disabling TOTP require the current password and are rate-limited. This commit ties together backend, UI, tests, configuration, and documentation to harden admin security workflows for password confirmation and TOTP management.
Add a .gitleaks.toml that allowlists known public dummy TOTP test fixtures (specific regexes) for tests/Unit/Auth/TotpServiceTest.php to prevent false positives. Update the CI workflow to pass --config .gitleaks.toml to gitleaks detect so the scanner uses this configuration during runs.
|
You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool. What Enabling Code Scanning Means:
For more information about GitHub Code Scanning, check out the documentation. |
1 similar comment
|
You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool. What Enabling Code Scanning Means:
For more information about GitHub Code Scanning, check out the documentation. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Describe what this pull request changes.
Type of Change
Validation
Security Considerations
Describe any security impact, assumptions or risks.
Rollback
Describe how the change can be reverted if required.
If no specific rollback procedure is required, state
N/A.