Skip to content

feat: share rate limits and bot prefs via optional Redis; hash IPs in logs - #7

Merged
Adeolu05 merged 1 commit into
fix/extract-reliabilityfrom
fix/shared-limits-and-log-privacy
Sep 28, 2026
Merged

Adeolu05 merged 1 commit into
fix/extract-reliabilityfrom
fix/shared-limits-and-log-privacy

Conversation

@Adeolu05

@Adeolu05 Adeolu05 commented Sep 28, 2026 •

Copy link
Copy Markdown
Owner

Stacked on #6 (which is stacked on #5). Merge in order: #5, then #6, then this. GitHub retargets each one as its base merges.

Summary

Shared state (optional REDIS_URL)

  • Rate limits are counted once, not per process. Before, each Gunicorn worker (2 by default) and the bot kept separate counters, so the real extract limit was a multiple of RATE_LIMIT_REQUESTS. Now one fixed-window counter is shared (SET NX EX + INCR in a single transaction).
  • Telegram preferences survive restarts and deploys. Each user gets a Redis hash with a 90-day TTL (TELEGRAM_PREFS_TTL_DAYS). Invalid stored values fall back to the defaults.
  • Fail-soft. If REDIS_URL is unset, or Redis errors or times out (0.5s socket timeout), everything falls back to the old in-memory behaviour and logs a warning at most once a minute. Nothing changes until you provision Redis.
  • Docker Compose now includes a small redis:7-alpine. render.yaml takes REDIS_URL as a secret, e.g. from a Render Key Value instance.

Log privacy

  • hash_identifier is now an HMAC keyed by LOG_HASH_SECRET (generated on Render; a random per-process key if unset). A plain SHA-256 of an IPv4 address or Telegram ID can be reversed by enumeration.
  • Rate-limit block logs no longer contain the raw client IP.
  • Failure logs record video_id + host instead of a partly stripped URL. The old version kept nothing useful for watch?v= links, but kept share-token paths on others.
  • Privacy page and bot copy now document IP-based limits, hashed log IDs and Telegram preferences. They also correct the caption-cache duration: it lasts days (72h default), not hours.

Note: telemetry session_ids for Telegram users change with this PR because the hash is now keyed. Analytics sessions from before the deploy won't line up with sessions after it.

Test plan

  • 77 unit tests pass locally (10 new; 2 real-Redis tests skip without TEST_REDIS_URL)
  • CI now runs a Redis service container, so the real-server tests (MULTI/SET NX EX/INCR, hash + TTL) run on every push
  • Fallback tests: rate limits and preferences still work when Redis raises ConnectionError
  • tsc --noEmit passes. The privacy page lints clean.
  • After deploy: set REDIS_URL, then confirm there's no "Redis unavailable" warning in the logs

… logs

- REDIS_URL (optional): extract/convert/events quotas are counted once
  across all Gunicorn workers and the Telegram bot instead of per process;
  Telegram format/timestamp preferences survive restarts (90-day TTL).
  Unset or unreachable Redis falls back to in-memory with a throttled warning.
- Docker Compose runs a small Redis; render.yaml takes REDIS_URL and
  generates LOG_HASH_SECRET.
- hash_identifier is now HMAC-keyed (plain SHA-256 of an IPv4 or Telegram ID
  is reversible by enumeration); rate-limit block logs no longer contain the
  raw client IP.
- Failure logs record the public video ID + host instead of a partially
  stripped URL (which lost the ID for watch?v= links and kept share tokens
  in youtu.be paths).
- Privacy page / bot copy: document IP-based limits, hashed log IDs, and
  Telegram preferences; caption cache lasts days (72h default), not hours.
- CI: Redis service container for real-server integration tests.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@vercel

vercel Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
transcriptflow Ready Ready Preview Sep 28, 2026 5:53pm UTC

@Adeolu05
Adeolu05 merged commit 67ea52e into fix/extract-reliability Sep 28, 2026
4 checks passed
Adeolu05 added a commit that referenced this pull request Oct 13, 2026

This branch was successfully deployed

1 active deployment
Preview — ad0a61a2 Deployed Sep 28, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant