Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,8 @@ TELEGRAM_BOT_TOKEN=YOUR_TOKEN_HERE
# SENTRY_DSN=
# SENTRY_TRACES_SAMPLE_RATE=0.1

# --- YouTube transcript (optional proxies / language priority)
# --- YouTube captions (REQUIRED on cloud / VPS: YouTube bans datacenter IPs)
# Without this, every link fails with "YouTube blocked this server", even English videos.
# YOUTUBE_TRANSCRIPT_LANGUAGE_PRIORITY=en,de
# YOUTUBE_HTTP_PROXY_URL=
# YOUTUBE_HTTPS_PROXY_URL=
Expand Down
16 changes: 14 additions & 2 deletions BACKEND_DEPLOYMENT_HANDOFF.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,9 +56,21 @@ Values below match `app/core/config.py`. Names are the usual `UPPER_SNAKE` env n
| `TRANSCRIPT_CACHE_ENABLED` | `true` | Disk cache for repeat URLs. |
| `TRANSCRIPT_CACHE_TTL_HOURS` | `72` | Cache freshness. |

### Optional (YouTube IP blocks / proxies)
### Required on cloud hosts (YouTube IP blocks)

`WEBSHARE_PROXY_*`, `YOUTUBE_HTTP_PROXY_URL`, `YOUTUBE_HTTPS_PROXY_URL`, `YOUTUBE_TRANSCRIPT_LANGUAGE_PRIORITY` — see repo README / `youtube-transcript-api` docs.
YouTube blocks most datacenter IPs. The Telegram bot and `/extract` then fail for **every** link, including English videos. The old user text talked about English-only captions. That was an IP block.

Set a **residential** proxy on the API **and** the bot process (same env):

| Variable | Purpose |
|----------|---------|
| `WEBSHARE_PROXY_USERNAME` | Webshare rotating residential user |
| `WEBSHARE_PROXY_PASSWORD` | Webshare password |
| `WEBSHARE_PROXY_LOCATIONS` | Optional country codes, e.g. `us,de` |

Or generic HTTP(S) proxies: `YOUTUBE_HTTP_PROXY_URL`, `YOUTUBE_HTTPS_PROXY_URL`.

Without one of these, cloud Telegram will keep failing. Local home IPs often work with no proxy.

---

Expand Down
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,9 @@ Core Value: **Fast → Reliable → Structured → Downloadable.**
## 🌟 Key Features

- **Multi-Platform Support**: Extract transcripts from **YouTube** and **Vimeo** instantly.
- **Clean Formats**: Download as **TXT, PDF, and DOCX**.
- **Clean Formats**: Download as **TXT, PDF, and DOCX**, or **SRT / VTT** subtitles with cue timings. Files are named after the video title.
- **Timestamps**: Toggle perfectly formatted timestamps on or off.
- **Anonymous & Secure**: No accounts required. Generated files auto-delete after ~1 hour (TTL). Optional short-term caption cache (hours) for reliability.
- **Anonymous & Secure**: No accounts required. Generated files auto-delete after ~1 hour (TTL). Optional short-term caption cache (`TRANSCRIPT_CACHE_TTL_HOURS`) is pruned by the same cleanup loop.
- **Rate Limited & Guardrailed**: Separate extract vs convert quotas, max video duration (with caption-based fallback), segment/size caps, and timeout protections.
- **Telemetry & Analytics**: Built-in lightweight JSON-structured logging and analytics tracking.

Expand Down Expand Up @@ -48,13 +48,13 @@ transcript/
### 1. Web Application (`/web`)
A premium, modern SaaS web app interface designed with:
- 3D Video Carousel hero section.
- Preview-First Results Flow: Shows a snippet of the transcript immediately, followed by format selection (TXT/PDF/DOCX).
- Preview-First Results Flow: Shows a snippet of the transcript immediately, followed by format selection (TXT/PDF/DOCX/SRT/VTT).
- Clean state-machine interactions (Idle → Processing → Success/Error).

### 2. Telegram Bot (`app/bot/telegram_bot.py`)
A seriously structured, minimal utility bot:
- **Instant Processing**: Send a URL, get a file. No conversational fluff.
- **In-Memory Preferences**: Remembers your preferred format (e.g., PDF) and timestamp preferences across sessions.
- **In-Memory Preferences**: Remembers your preferred format (e.g., PDF) and timestamp setting while the bot process runs (reset on restart/deploy).
- **Single Message UX**: Status updates (Extracting... $\rightarrow$ Formatting... $\rightarrow$ Ready) happen within a single edited message to keep chat history clean.
- **Branded Onboarding**: Clean `/start` experience with inline keyboard routing.

Expand Down
95 changes: 56 additions & 39 deletions app/api/v1_routes.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,18 @@
verify_rate_limit_events,
verify_rate_limit_extract,
)
from app.core.errors import AppError, ErrorCode, success_response
from app.core.errors import AppError, ErrorCode, TranscriptFetchError, success_response
from app.services.analytics_service import AnalyticsService
from app.services.extract_guardrails import enforce_transcript_guardrails
from app.services.file_service import TEMP_DIR, FileGenerator
from app.services.formatter_service import TranscriptFormatter
from app.services.file_service import (
TEMP_DIR,
FileGenerator,
content_disposition,
download_filename,
read_sidecar,
write_sidecar,
)
from app.services.formatter_service import SUBTITLE_FORMATS, TranscriptFormatter
from app.services.metadata_service import MetadataService
from app.services.telemetry_service import MAX_PAYLOAD_BYTES, TelemetryService
from app.services.transcript_service import get_transcript_from_url
Expand All @@ -28,7 +35,14 @@

router = APIRouter(prefix="/v1")

_ALLOWED_EXTENSIONS = {".txt", ".pdf", ".docx"}
_CONVERT_FORMATS = ("txt", "pdf", "docx", *SUBTITLE_FORMATS)
_ALLOWED_EXTENSIONS = {f".{fmt}" for fmt in _CONVERT_FORMATS}
_MEDIA_TYPES = {
".pdf": "application/pdf",
".docx": "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
".srt": "application/x-subrip",
".vtt": "text/vtt",
}
_convert_semaphore = asyncio.Semaphore(settings.convert_max_concurrency)
_UUID_HEX_RE = re.compile(r"^[0-9a-f]{32}$")

Expand Down Expand Up @@ -109,17 +123,11 @@ async def extract_transcript(request: ExtractRequest):
except Exception as e:
err_msg = str(e)
processing_time_ms = int((time.time() - start_time) * 1000)
el = err_msg.lower()
if "disabled" in el:
code = ErrorCode.TRANSCRIPT_NOT_AVAILABLE
elif "unavailable" in el or "not found" in el or "no transcript or captions" in el:
code = ErrorCode.TRANSCRIPT_NOT_AVAILABLE
elif "youtube blocked" in el or "only available in english" in el:
code = ErrorCode.TRANSCRIPT_NOT_AVAILABLE
elif "unsupported" in el:
code = ErrorCode.INVALID_URL
if isinstance(e, TranscriptFetchError):
code = e.code
else:
code = ErrorCode.INTERNAL_ERROR
logger.exception("Unexpected transcript fetch failure")
MetadataService.log_failure(
url=request.url,
platform=platform,
Expand Down Expand Up @@ -194,6 +202,9 @@ async def extract_transcript(request: ExtractRequest):

text_sidecar = TEMP_DIR / f"{filename}.raw"
text_sidecar.write_text(formatted_text, encoding="utf-8")
# Segments feed SRT/VTT convert; title names every download of this transcript
await asyncio.to_thread(write_sidecar, filename, "segments", transcript_data["segments"])
write_sidecar(filename, "meta", {"title": transcript_data.get("title")})

MetadataService.log_success(
video_id=transcript_data["video_id"],
Expand All @@ -213,7 +224,7 @@ async def extract_transcript(request: ExtractRequest):
processing_time_ms=processing_time_ms,
)

preview_text = formatted_text[:1500]
preview_text = formatted_text[: settings.preview_chars]
expires_at = (
datetime.now(timezone.utc) + timedelta(hours=settings.file_ttl_hours)
).isoformat()
Expand Down Expand Up @@ -245,36 +256,30 @@ async def download_file(file_id: str):
raise AppError(ErrorCode.FILE_EXPIRED, "File not found or expired.", 404)

extension = file_path.suffix.lower()
media_type = "text/plain"
if extension == ".pdf":
media_type = "application/pdf"
elif extension == ".docx":
media_type = (
"application/vnd.openxmlformats-officedocument.wordprocessingml.document"
)
meta = read_sidecar(file_id, "meta") or {}
filename = download_filename(meta.get("title"), extension)

return FileResponse(
path=str(file_path),
media_type=media_type,
filename=f"transcript{extension}",
media_type=_MEDIA_TYPES.get(extension, "text/plain"),
headers={
"Content-Disposition": f"attachment; filename=transcript{extension}",
"Content-Disposition": content_disposition(filename),
"Cache-Control": "no-store",
},
)


class ConvertRequest(BaseModel):
file_id: str
format: str # txt, pdf, docx
format: str # txt, pdf, docx, srt, vtt


@router.post("/convert", dependencies=[Depends(verify_rate_limit_convert)])
async def convert_file(request: ConvertRequest):
if request.format not in ("txt", "pdf", "docx"):
if request.format not in _CONVERT_FORMATS:
raise AppError(
ErrorCode.CONVERT_FAILED,
"Unsupported format. Use txt, pdf, or docx.",
"Unsupported format. Use txt, pdf, docx, srt, or vtt.",
400,
)

Expand All @@ -290,18 +295,26 @@ async def convert_file(request: ConvertRequest):
}
)

sidecar_path = TEMP_DIR / f"{request.file_id}.raw"
if not sidecar_path.exists():
raise AppError(ErrorCode.FILE_EXPIRED, "Source text not found or expired.", 404)

formatted_text = sidecar_path.read_text(encoding="utf-8")

if len(formatted_text) > settings.max_raw_text_length:
raise AppError(
ErrorCode.CONVERT_FAILED,
"Transcript is too large to convert to this format.",
400,
if request.format in SUBTITLE_FORMATS:
segments = await asyncio.to_thread(read_sidecar, request.file_id, "segments")
if not isinstance(segments, list):
raise AppError(ErrorCode.FILE_EXPIRED, "Source captions not found or expired.", 404)
# Segment count/size were already capped by the extract guardrails
content = await asyncio.to_thread(
TranscriptFormatter.format_subtitles, segments, request.format
)
else:
sidecar_path = TEMP_DIR / f"{request.file_id}.raw"
if not sidecar_path.exists():
raise AppError(ErrorCode.FILE_EXPIRED, "Source text not found or expired.", 404)
content = sidecar_path.read_text(encoding="utf-8")

if len(content) > settings.max_raw_text_length:
raise AppError(
ErrorCode.CONVERT_FAILED,
"Transcript is too large to convert to this format.",
400,
)

# Non-blocking acquire — no TOCTOU with locked() pre-check
try:
Expand All @@ -316,7 +329,7 @@ async def convert_file(request: ConvertRequest):
try:
try:
file_path = await asyncio.to_thread(
FileGenerator.generate_file, formatted_text, request.format
FileGenerator.generate_file, content, request.format
)
except Exception as e:
logger.error("Convert generation failed: %s", e)
Expand All @@ -329,6 +342,10 @@ async def convert_file(request: ConvertRequest):
finally:
_convert_semaphore.release()

meta = read_sidecar(request.file_id, "meta")
if meta:
write_sidecar(filename, "meta", meta)

return success_response(
{
"file_download_url": f"/api/v1/download/{filename}",
Expand Down
Loading
Loading