HuntCat

Enterprise-Grade Web Audit & SEO Crawler

Warning!

(With Modular & Standalone Modes for developers and users.)

huntcat.go is intended for regular users. If you're a developer and want to explore the code more thoroughly, you can access the main file structure in the "For Developers (Discrete Modular)" folder.

"Leave your cats in the area and enjoy!"

Features • Installation • Usage • Reports • Structure

> Overview

HuntCat is a blazing-fast, enterprise-grade web crawler and SEO auditor built in pure Go. Designed for technical SEO professionals, web developers, and DevOps teams who demand speed, accuracy, and actionable insights.

Why HuntCat?

[+] Ultra-Fast: Concurrent crawling with 50+ workers
[+] Beautiful Reports: Professional HTML + CSV exports
[+] Security-First: TLS 1.2+, HTTP/2 support, HTTPS validation
[+] SEO Intelligence: Meta tags, canonicals, structured data detection
[+] Polite Crawling: RFC 9309 compliant robots.txt handling, rate limiting
[+] Smart Detection: Duplicate content via SHA-256 hashing
[+] 4D Scoring System: Health, SEO, Performance, Security

[$] Features

[+] Advanced Crawling

Recursive link discovery with intelligent depth control
Concurrent processing (up to 50 workers by default)
Sitemap index parsing (recursive)
External link validation (HEAD with GET fallback)
<base href> resolution for relative links
Image asset optimization analysis

[+] Health Monitoring

404 Detection: Broken link identification
5xx Errors: Server-side issue tracking
Redirect Chains: Accurate 301/302 counting via Context
Asset Analysis: Image size optimization (700KB/1MB/2MB thresholds)
Load Time Tracking: Per-resource performance metrics

[+] SEO Audit Engine

Title tag validation (optimal: 50-60 chars)
Meta description analysis (optimal: up to 160 chars)
H1 tag counting (best practice: exactly 1)
Canonical URL detection
Open Graph meta tags extraction
Structured Data (JSON-LD) detection
Meta robots tag inspection

[+] Security & Reliability Checks

HTTPS enforcement validation
HTTP/2 protocol detection
TLS version verification (>= 1.2)
Non-secure page flagging
Transparent gzip decompression (No manual Accept-Encoding bugs)
Memory pruning (Prevents leaks on large crawls)

[+] Intelligent Scoring

Health Score = (100 - error_rate)
SEO Score = Meta tag compliance + structural quality (HTML pages only)
Performance Score = Asset optimization + load times
Security Score = HTTPS coverage + TLS compliance

[<>] Installation

Prerequisites

Go 1.21+ installed
Terminal/Command Prompt access

Quick Install (Standalone for Users)

# Clone the repository
git clone https://github.com/tc4dy/HuntCat.git
cd HuntCat

# Navigate to the standalone folder
cd "Standalone (Single File for Users)"

# Download dependencies
go mod download

# Run directly
go run huntcat.go https://example.com

# Or build a binary
go build -ldflags="-s -w" -o huntcat huntcat.go
./huntcat https://example.com

For Developers (Discrete Modular)

# From the root directory
cd "For Developers (Discrete Modular)"

# Download dependencies
go mod download

# Run using all files
go run *.go https://example.com

# Build binary
go build -ldflags="-s -w" -o huntcat .
./huntcat https://example.com

[<>] Usage

HuntCat ships in two forms: a Standalone single file for end users, and a Discrete Modular structure for developers. Both are functionally identical — the same Go code, just organized differently.

Standalone (End Users)

The huntcat.go file contains the entire application in one place. No need to navigate folders or understand the project structure — just download and run.

# Basic scan
go run huntcat.go https://yoursite.com

# Build a binary once, use anywhere
go build -ldflags="-s -w" -o huntcat huntcat.go
./huntcat https://example.com

Discrete Modular (Developers)

The For Developers (Discrete Modular)/ folder contains the same application split across multiple files by responsibility. This is the recommended structure if you want to contribute, extend, or understand the internals.

cd "For Developers (Discrete Modular)"
go run *.go https://example.com

Authentication & Custom Headers

Both versions support the same flags for accessing protected or custom endpoints:

# With authentication cookie
go run huntcat.go https://example.com --cookie="session=abc123"

# With Bearer token
go run huntcat.go https://example.com --token="eyJhbGci..."

# With custom header (repeatable)
go run huntcat.go https://example.com --header="X-API-Key: mykey" --header="X-Foo: bar"

All Available Options

Flag	Description	Example
`--cookie=<value>`	Set authentication cookie	`--cookie="session=abc123"`
`--token=<value>`	Set Bearer token (Authorization header)	`--token="eyJhbGci..."`
`--header=<k:v>`	Add a custom HTTP header (repeatable)	`--header="X-Key: val"`
`-h, --help`	Show help message	`huntcat -h`

[!] Reports

HuntCat generates 2 comprehensive reports after each scan:

1. HTML Report (`huntcat_report.html`)

Beautiful gradient design
Visual score cards (Health, SEO, Performance, Security)
Interactive issue tables
Filterable by priority

Preview

+------------------------------------------+
|  Health Score:       100%  Excellent     |
|  SEO Score:           45%  Needs Work    |
|  Performance Score:   92%  Good           |
|  Security Score:     100%  Secure        |
+------------------------------------------+

2. CSV Export (`huntcat_report.csv`)

Import to Excel/Google Sheets
Raw data for custom analysis
Columns: URL, Type, Status, Size, HTTPS, HTTP Version, Title, Description, H1 Count, Canonical, Load Time, Issue

[ARC] Architecture

Core Components

+----------------------------------------------------------+
|                    HuntCat Engine                        |
+----------------------------------------------------------+
|                                                          |
|  +--------------+  +--------------+  +--------------+   |
|  |   Crawler    |  |   Analyzer   |  |   Reporter   |   |
|  |              |  |              |  |              |   |
|  | - Link Disc. |  | - SEO Check  |  | - HTML Gen   |   |
|  | - Asset Find |  | - Perf Audit |  | - CSV Export |   |
|  | - Robots.txt |  | - Security   |  | - Scoring    |   |
|  +--------------+  +--------------+  +--------------+   |
|                                                          |
|  +------------------------------------------------------+|
|  |  Concurrency Engine (WaitGroup + Semaphore)          ||
|  +------------------------------------------------------+|
+----------------------------------------------------------+

Tech Stack

Language: Go 1.21+ (Standard library + golang.org/x/net/html)
Concurrency: Goroutines, WaitGroups, Mutexes, Channels, Atomic
HTTP Client: Custom HTTP/2 client with TLS 1.2+ enforcement
Parsing: golang.org/x/net/html for robust HTML parsing
Hashing: SHA-256 for duplicate content detection
Decompression: Native Go gzip handling (no manual encoding bugs)

[<>] Examples

Terminal Output

  ██╗  ██╗██╗   ██╗███╗   ██╗████████╗ ██████╗ █████╗ ████████╗
  ...

Target: https://example.com
Initializing HuntCat with 50 concurrent workers...
Fetching robots.txt and sitemap.xml...
Enforcing rate limiting (100ms/request)...

Starting deep crawl with SEO analysis...

[+] 200 https://example.com [page]
[+] 200 https://example.com/about [page]
[-] 404 https://example.com/missing-page [page]
[!] IMG https://example.com/huge-image.jpg - WARNING: 2.5 MB

╔═══════════════════════════════════════════════════════╗
║              HUNTCAT AUDIT REPORT                     ║
╚═══════════════════════════════════════════════════════╝

Total Resources Scanned    145      [+] Complete
Broken Links (404)         3        [-] Critical
Critical Images (>2MB)     2        [!] URGENT
Site Health Score          97.9%    Excellent
SEO Score                  68.3%    Good

Crawl completed in: 2.34s
HTML report saved: huntcat_report.html
CSV report saved: huntcat_report.csv

Repository Structure

HuntCat/
├── README.md — Main documentation
├── LICENSE — MIT License
├── .gitignore — Git ignore rules
├── huntcat-banner.svg — Project banner
├── .github/
│   └── workflows/
│       └── ci.yml — CI Pipeline (Build, Vet, Test, Lint)
├── Standalone (Single File for Users)/
│   ├── huntcat.go — Standalone version (for end users)
│   ├── go.mod — Go module definition
│   └── go.sum — Dependency checksums
└── For Developers (Discrete Modular)/
    ├── main.go — Entry point
    ├── crawler.go — Crawling engine
    ├── constants.go — Configuration constants
    ├── colors.go — Terminal color definitions
    ├── page_status.go — Data structures
    ├── http_client.go — HTTP client & user agents
    ├── parsers.go — HTML parsing & SEO analysis
    ├── reports.go — HTML, CSV & console reports
    ├── utils.go — Helper functions
    ├── crawler_test.go — Unit tests for crawler
    ├── parsers_test.go — Unit tests for parsers
    ├── go.mod — Go module dependencies
    ├── go.sum — Dependency checksums
    └── .golangci.yml — Linter configuration

File Descriptions

File	Description
`Standalone (Single File for Users)/huntcat.go`	Standalone version — run with `go run huntcat.go <url>`.
`For Developers (Discrete Modular)/`	Modular source — run with `go run *.go <url>`, for contributors and developers exploring the code.
`go.mod` / `go.sum`	Go module definition and dependency checksums for reproducible builds.
`README.md`	Project documentation — you're reading it!
`LICENSE`	MIT License — open source, free to use under its terms.
`.gitignore`	Keeps unwanted files out of version control.
`.github/workflows/ci.yml`	Automated CI pipeline for Build, Vet, Test, and Lint.
`huntcat-banner.svg`	Project banner and visual identity.

[^_^] Configuration

Edit constants in constants.go (or huntcat.go in the standalone version) to customize behavior:

const (
    maxConcurrency         = 50                      // Concurrent workers
    requestTimeout         = 30 * time.Second        // HTTP timeout
    rateLimitDelay         = 100 * time.Millisecond  // Delay between requests
    largeImageThreshold    = 700 * 1024              // 700KB
    hugeImageThreshold     = 1024 * 1024             // 1MB
    criticalImageThreshold = 2 * 1024 * 1024         // 2MB
    maxCrawlDepth          = 5                       // Maximum crawl depth
    optimalTitleMin        = 50                      // Minimum title length (SEO)
    optimalTitleMax        = 60                      // Maximum title length (SEO)
    optimalDescMax         = 160                     // Maximum meta description length
)

Known Limitations

No JavaScript rendering: HuntCat fetches raw HTML. Pages that require JavaScript to render content (SPAs, React/Vue/Angular apps) will return incomplete results.
No SPA support: Single Page Applications that load content dynamically via API calls will not be fully crawled.
robots.txt is advisory: HuntCat respects robots.txt rules per RFC 9309 by default but does not enforce Crawl-delay directives — use rateLimitDelay in constants instead.

[<>] Other Related Projects

PrivMeta

A local-first CLI privacy tool designed to inspect, clean, and spoof metadata across digital files. Remove GPS tracking information and hidden metadata from images, PDFs, and video files, or inject realistic camera profiles and city coordinates for controlled metadata customization. PrivMeta also features anti-forensic shredding, bulk folder processing, timeline simulation, and a privacy-focused design with zero telemetry.

Godas

Godas is a high-performance columnar data analysis library for Go, offering typed Series, expressive filter expressions, concurrent GroupBy aggregation, hash joins, and CSV/JSON/NDJSON/Parquet I/O through one coherent, idiomatic API. Built for production workloads where type safety, null semantics, and raw throughput matter equally.

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
For Developers (Discrete Modular)		For Developers (Discrete Modular)
Standalone (Single File for Users)		Standalone (Single File for Users)
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
huntcat-banner.svg		huntcat-banner.svg

Latest commit

History

Folders and files

Repository files navigation