Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,14 @@ MAX_TRANSCRIPT_SEGMENTS=20000
IN_PROCESS_CLEANUP_ENABLED=true
CLEANUP_INTERVAL_SECONDS=900

# --- Shared state (recommended in production)
# Redis shares rate-limit counts across Gunicorn workers + the Telegram bot and keeps
# bot preferences across restarts. Unset/unreachable = per-process in-memory fallback.
# REDIS_URL=redis://localhost:6379/0
# TELEGRAM_PREFS_TTL_DAYS=90
# Key for hashing IPs / Telegram IDs in logs (any long random string). Unset = random per process.
# LOG_HASH_SECRET=

# --- Internal metrics dashboard (Basic Auth at /internal/metrics)
METRICS_USERNAME=admin
# Change this before any public deployment.
Expand All @@ -60,6 +68,12 @@ TELEGRAM_BOT_TOKEN=YOUR_TOKEN_HERE
# WEBSHARE_PROXY_PASSWORD=
# WEBSHARE_PROXY_LOCATIONS=

# --- PDF fonts
# Unicode TTF for PDFs beyond Western European text (Polish, Vietnamese, Cyrillic, Greek).
# Docker installs DejaVu Sans + Nanum Gothic (Korean); set this for local dev if needed.
# Arabic, Hebrew, Indic and Thai scripts get DOCX/TXT only (no PDF).
# PDF_FONT_PATH=

# --- Transcript disk cache
TRANSCRIPT_CACHE_ENABLED=true
TRANSCRIPT_CACHE_TTL_HOURS=72
Expand Down
34 changes: 34 additions & 0 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,16 @@ on:
jobs:
unit:
runs-on: ubuntu-latest
services:
redis:
image: redis:7-alpine
ports:
- 6379:6379
options: >-
--health-cmd "redis-cli ping"
--health-interval 5s
--health-timeout 3s
--health-retries 10
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
Expand All @@ -19,3 +29,27 @@ jobs:
run: python -m unittest discover -s tests -v
env:
PYTHONPATH: .
TEST_REDIS_URL: redis://localhost:6379/15

web:
runs-on: ubuntu-latest
defaults:
run:
working-directory: web
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: "22"
cache: npm
cache-dependency-path: web/package-lock.json
- name: Install dependencies
run: npm ci
- name: Lint
run: npm run lint
- name: Type-check
run: npx tsc --noEmit
- name: Build
run: npm run build
env:
NEXT_PUBLIC_API_BASE_URL: https://api.example.invalid
6 changes: 5 additions & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,13 @@ RUN groupadd --system --gid 1000 app \

WORKDIR /app

# System deps for reportlab / cffi
# System deps for reportlab / cffi. PDF fonts: DejaVu Sans embeds Latin-extended,
# Greek and Cyrillic (Helvetica only covers Western European); Nanum Gothic embeds
# Korean (ReportLab's built-in Korean CID fonts render blank in Chrome's viewer)
RUN apt-get update && apt-get install -y --no-install-recommends \
libffi-dev \
fonts-dejavu-core \
fonts-nanum \
&& rm -rf /var/lib/apt/lists/*

COPY requirements.txt .
Expand Down
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,10 @@ Core Value: **Fast → Reliable → Structured → Downloadable.**

- **Multi-Platform Support**: Extract transcripts from **YouTube** and **Vimeo** instantly.
- **Clean Formats**: Download as **TXT, PDF, and DOCX**, or **SRT / VTT** subtitles with cue timings. Files are named after the video title.
- **23 Languages**: Pick the transcript language on the web or with `/language` in Telegram. YouTube auto-translates when a video has no captions in that language; Vimeo uses the uploader's tracks.
- **Timestamps**: Toggle perfectly formatted timestamps on or off.
- **Anonymous & Secure**: No accounts required. Generated files auto-delete after ~1 hour (TTL). Optional short-term caption cache (`TRANSCRIPT_CACHE_TTL_HOURS`) is pruned by the same cleanup loop.
- **Rate Limited & Guardrailed**: Separate extract vs convert quotas, max video duration (with caption-based fallback), segment/size caps, and timeout protections.
- **Rate Limited & Guardrailed**: Separate extract vs convert quotas (shared across workers and the bot via optional Redis), max video duration (with caption-based fallback), segment/size caps, and timeout protections.
- **Telemetry & Analytics**: Built-in lightweight JSON-structured logging and analytics tracking.

## 🏗 System Architecture
Expand Down Expand Up @@ -54,7 +55,7 @@ A premium, modern SaaS web app interface designed with:
### 2. Telegram Bot (`app/bot/telegram_bot.py`)
A seriously structured, minimal utility bot:
- **Instant Processing**: Send a URL, get a file. No conversational fluff.
- **In-Memory Preferences**: Remembers your preferred format (e.g., PDF) and timestamp setting while the bot process runs (reset on restart/deploy).
- **Remembered Preferences**: Remembers your preferred format (e.g., PDF) and timestamp setting — in Redis when `REDIS_URL` is set (kept ~90 days), otherwise in memory until the bot restarts.
- **Single Message UX**: Status updates (Extracting... $\rightarrow$ Formatting... $\rightarrow$ Ready) happen within a single edited message to keep chat history clean.
- **Branded Onboarding**: Clean `/start` experience with inline keyboard routing.

Expand Down
51 changes: 44 additions & 7 deletions app/api/v1_routes.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,13 @@
verify_rate_limit_extract,
)
from app.core.errors import AppError, ErrorCode, TranscriptFetchError, success_response
from app.core.languages import (
DEFAULT_LANGUAGE,
LANGUAGE_CODES,
language_name,
languages_match,
public_language_list,
)
from app.services.analytics_service import AnalyticsService
from app.services.extract_guardrails import enforce_transcript_guardrails
from app.services.file_service import (
Expand All @@ -27,6 +34,7 @@
write_sidecar,
)
from app.services.formatter_service import SUBTITLE_FORMATS, TranscriptFormatter
from app.services.pdf_fonts import UnsupportedPdfScript, pdf_supported
from app.services.metadata_service import MetadataService
from app.services.telemetry_service import MAX_PAYLOAD_BYTES, TelemetryService
from app.services.transcript_service import get_transcript_from_url
Expand Down Expand Up @@ -68,6 +76,8 @@ def _validate_file_id(file_id: str) -> str:
class ExtractRequest(BaseModel):
url: str = Field(..., min_length=20, max_length=2048)
include_timestamps: bool = False
# Target caption language (see app.core.languages); YouTube auto-translates
language: str = DEFAULT_LANGUAGE


@router.get("/health")
Expand All @@ -88,11 +98,18 @@ async def extract_transcript(request: ExtractRequest):
400,
)

if request.language not in LANGUAGE_CODES:
raise AppError(ErrorCode.UNSUPPORTED_LANGUAGE, "That language isn't supported.", 400)

layout_format = "timestamp" if request.include_timestamps else "clean"

# The worker thread outlives wait_for; the deadline makes it stop calling upstream too
deadline = time.monotonic() + settings.transcript_timeout_seconds
try:
transcript_data = await asyncio.wait_for(
asyncio.to_thread(get_transcript_from_url, request.url),
asyncio.to_thread(
get_transcript_from_url, request.url, deadline, request.language
),
timeout=settings.transcript_timeout_seconds,
)
except asyncio.TimeoutError:
Expand Down Expand Up @@ -124,9 +141,9 @@ async def extract_transcript(request: ExtractRequest):
err_msg = str(e)
processing_time_ms = int((time.time() - start_time) * 1000)
if isinstance(e, TranscriptFetchError):
code = e.code
code, status = e.code, e.status_code
else:
code = ErrorCode.INTERNAL_ERROR
code, status = ErrorCode.INTERNAL_ERROR, 500
logger.exception("Unexpected transcript fetch failure")
MetadataService.log_failure(
url=request.url,
Expand All @@ -144,7 +161,6 @@ async def extract_transcript(request: ExtractRequest):
processing_time_ms=processing_time_ms,
error_code=code.value,
)
status = 400 if code != ErrorCode.INTERNAL_ERROR else 500
client_msg = (
"An unexpected error occurred."
if code == ErrorCode.INTERNAL_ERROR
Expand Down Expand Up @@ -204,7 +220,12 @@ async def extract_transcript(request: ExtractRequest):
text_sidecar.write_text(formatted_text, encoding="utf-8")
# Segments feed SRT/VTT convert; title names every download of this transcript
await asyncio.to_thread(write_sidecar, filename, "segments", transcript_data["segments"])
write_sidecar(filename, "meta", {"title": transcript_data.get("title")})
out_language = transcript_data.get("language", DEFAULT_LANGUAGE)
# Language picks the CJK PDF font (zh-Hant vs zh-Hans) at convert time
write_sidecar(filename, "meta", {"title": transcript_data.get("title"), "language": out_language})
can_pdf = await asyncio.to_thread(pdf_supported, formatted_text)
source_language = transcript_data.get("source_language") or out_language
translated = bool(transcript_data.get("translated"))

MetadataService.log_success(
video_id=transcript_data["video_id"],
Expand Down Expand Up @@ -234,7 +255,16 @@ async def extract_transcript(request: ExtractRequest):
"provider": transcript_data.get("provider", platform),
"video_id": transcript_data["video_id"],
"title": transcript_data["title"],
"language": transcript_data.get("language", "en"),
"language": out_language,
"language_name": language_name(out_language),
"requested_language": request.language,
"source_language": source_language,
"source_language_name": language_name(source_language),
"translated": translated,
# Neither a native track nor a translation: captions are in another language
"language_fallback": not translated
and not languages_match(out_language, request.language),
"pdf_supported": can_pdf,
"duration_seconds": duration,
"word_count": metrics["word_count"],
"reading_time_seconds": metrics["reading_time_seconds"],
Expand Down Expand Up @@ -329,8 +359,13 @@ async def convert_file(request: ConvertRequest):
try:
try:
file_path = await asyncio.to_thread(
FileGenerator.generate_file, content, request.format
FileGenerator.generate_file,
content,
request.format,
(read_sidecar(request.file_id, "meta") or {}).get("language", ""),
)
except UnsupportedPdfScript as e:
raise AppError(ErrorCode.CONVERT_FAILED, str(e), 400)
except Exception as e:
logger.error("Convert generation failed: %s", e)
raise AppError(
Expand Down Expand Up @@ -358,6 +393,8 @@ async def get_public_config():
return success_response(
{
"file_ttl_hours": settings.file_ttl_hours,
"default_language": DEFAULT_LANGUAGE,
"languages": public_language_list(),
}
)

Expand Down
Loading
Loading