Feature: Local Helper App Integration for Large Repository Processing

[MekDrop]

Summary

Introduce a local companion application (“Helper”) that performs heavy Git operations (e.g., BFG cleanup) on the user’s machine while the GitHub Pages UI acts as a control panel and progress viewer.

Problem

Current approaches (GitHub Actions) break down for large repositories due to:

• Disk and RAM limits on GitHub-hosted runners
• Repository size and artifact limits
• Browser storage/memory constraints
• Cost and complexity of maintaining backend infrastructure

Proposed Solution

Split responsibilities:

Frontend (GitHub Pages - git-extract.github.io):

• UI for selecting repository and cleanup rules
• Detect and connect to local helper via localhost
• Display progress/logs in real time
• Provide final instructions (force-push, download, etc.)

Local Helper App (user machine):

• Runs Git + BFG (or equivalent)
• Exposes local HTTP API (e.g., http://127.0.0.1:<port>)
• Streams progress/events to frontend (SSE preferred)
• Handles authentication using user’s existing Git setup (SSH, tokens)

Architecture

Browser (GitHub Pages)
    ↓ HTTP/SSE
Local Helper (127.0.0.1)
    ↓
git clone --mirror
    ↓
BFG / cleanup logic
    ↓
git gc / rewrite
    ↓
(optional) push

Communication

Endpoint examples (local helper):

• POST /start → starts job
• GET /status → returns current state
• GET /events → SSE stream of logs/progress
• POST /cancel → abort job

CORS

Helper must allow:

Access-Control-Allow-Origin: https://git-extract.github.io

Progress Streaming

Use Server-Sent Events (SSE):

• Simpler than WebSockets
• One-way streaming fits logs/progress
• Native browser support

UX Flow

1. User opens GitHub Pages UI
2. App checks localhost for helper
3. If missing → prompt download
4. Helper starts and auto-opens browser
5. UI connects to helper
6. User selects repo + rules
7. Job starts locally
8. UI shows progress/logs
9. User reviews result
10. User confirms manual force-push (or optional assisted push)

Benefits

• Works with very large repositories
• No backend/server costs
• No GitHub Actions limitations
• No repo pollution
• Uses user’s local resources (disk, credentials)
• More secure (no repo upload to third-party servers)

Trade-offs

• Requires user to install helper app
• Localhost communication requires CORS handling
• Slightly more complex onboarding

Optional Enhancements

• Bundle Java + BFG inside helper
• Provide Docker-based fallback instead of native app
• Add auto-detection of repo size → recommend local mode
• Add “dry-run” preview before rewrite
• Add log export / sharing

Alternatives Considered

• GitHub Actions (fails on large repos)
• v86 browser Linux (too heavy/unreliable)
• External cloud workers (cost + complexity)
• WebSocket backend (unnecessary for this use case)

Acceptance Criteria

• UI detects helper availability
• Helper exposes stable API on localhost
• Large repo (>5GB) can be processed without failure
• Real-time progress visible in UI
• No writes to git-extract repository required
• Works without any paid infrastructure

[MekDrop]

The helper should use a hybrid model:

• Ephemeral session (per run)
• Shared hidden cache (for tools only)

This avoids:

• re-downloading tools every run
• polluting Desktop or app folder
• persistent visible state

---

Execution Model

git-extract-helper
→ starts
→ checks tool cache
→ downloads tools only if missing
→ creates temp session directory
→ asks user to select working directory
→ runs cleanup
→ deletes session on exit

---

Structure

Tool Cache (reused between runs, hidden)

Windows: %LOCALAPPDATA%/Temp/git-extract-tools/
macOS:   ~/Library/Caches/git-extract/tools/
Linux:   ~/.cache/git-extract/tools/

Example:

tools/
  git-2.54/
  jre-21/
  bfg-1.15/

Rules:

• Tools downloaded only if not already present
• Versioned to allow safe upgrades
• Not stored near executable
• Not visible to user by default

---

Session (ephemeral, per run)

%TEMP%/git-extract/session-<id>/
  logs/
  runtime/

Rules:

• Created on start
• Deleted on exit
• Contains no tool installations
• Only logs and runtime state

---

Workspace (user-selected, persistent)

D:/UserSelectedFolder/
  repo.git/
  cleanup-log.txt

Rules:

• Explicitly chosen by user
• Only location with meaningful output
• User responsible for cleanup

---

Lifecycle

Startup:

• Validate or download tools into cache
• Clean old temp sessions (optional)

Run:

• Create session directory
• Execute Git + BFG using cached tools
• Stream logs to UI

Exit:

• Delete session directory

---

Key Principles

• No writes near executable (avoids Desktop pollution)
• No global visible folders
• Tools cached but hidden
• Tools downloaded only when missing
• Sessions always temporary
• Workspace explicitly controlled by user

---

Optional Enhancements

[ ] Clear tool cache after run
[ ] Keep tools for faster future runs (default ON)

---

Acceptance Criteria (final)

• Tools are downloaded only if not already cached
• No tool installation occurs in temp session
• Temp session is always deleted on exit
• No files written near executable
• Workspace is user-selected and isolated
• Cache is hidden and reused across runs

[MekDrop]

Proposed approach to make GitHub Pages + portable helper reliable without installers, custom protocols, or local TLS.

Problem:
Direct browser → localhost communication (ws:// or wss://) is not reliable. It can fail due to mixed-content rules, certificate issues, browser security changes, or environment restrictions. For a portable app, this is not acceptable.

Solution:
Introduce a minimal Cloudflare-based signaling layer using HTTP + SSE, and use WebRTC for direct communication after pairing.

Architecture:

GitHub Pages (static UI)
⇄ HTTP POST + SSE
Cloudflare Worker + Durable Object (per-room state)
⇄ HTTP POST + SSE
Portable helper app

After signaling:

Browser ⇄ WebRTC DataChannel ⇄ Helper

Cloudflare is used only for:

• room creation (short-lived session id)
• pairing (browser + helper join same room)
• WebRTC signaling (offer / answer / ICE)
• connection state (helper connected, etc.)

Flow:

1. Browser creates room (e.g. ABCD-1234).
2. Helper is started manually and joins room (user enters/pastes code).
3. Browser sends WebRTC offer via POST.
4. Helper receives via SSE, responds with answer.
5. ICE candidates exchanged via POST/SSE.
6. WebRTC DataChannel established.
7. SSE connections can be closed.

Fallback:

If WebRTC fails (timeout, NAT, policy, etc.):

• continue using HTTP POST + SSE as relay
• higher latency, but still functional

Why this approach:

• no localhost networking required
• no TLS/certificate issues
• no browser extension
• no custom protocol handler
• no installer
• works from GitHub Pages (static hosting)

Notes:

• Use Durable Object per room to keep ordered message state
• Use short-lived room IDs (expiry ~10 minutes)
• Enforce single browser + single helper per room
• Use sequence IDs for messages to support polling fallback if needed
• Helper should cache downloaded tools (temp or local cache) and fetch only if missing

This keeps the system minimal while avoiding the known failure points of direct browser → localhost communication.