Acoustic Whisper Telemetry · Qwen Computer Vision · DuckDB Columnar OLAP · LanceDB Vector Search · Cartographic Knowledge Graph
Problem • Architecture • Features • Quickstart • Why Open Innovation Matters • Contributing
In short-form video (Instagram Reels, TikTok, YouTube Shorts), 85% of viewers scroll away within the first 3 seconds.
Today, creators, educators, and indie builders face three fatal obstacles:
- Blind Trial-and-Error: Creators spend 20+ hours filming content without knowing whether their speech latency is 0.4s too slow, their hook word count causes cognitive overload, or their cut velocity drops viewer dopamine.
- Proprietary Black-Boxes: Existing creator tools are expensive enterprise SaaS ($500+/mo) that treat retention as a mystery and upload private, unreleased creator intellectual property to proprietary cloud servers.
- Shallow AI Wrappers: Most hackathon projects are thin wrappers over closed APIs that summarize captions without analyzing the raw physical reality of video: audio acoustics, keyframe pacing, visual pattern interrupts, and audience objections.
VAULT INTEL is a 100% open-source, local-first multi-modal video intelligence platform. It ingests raw short-form video, tears it down frame-by-frame and millisecond-by-millisecond, and provides data-backed creative intelligence.
- 🎙️ Acoustic Speech Telemetry: OpenAI Whisper computes exact word-level timestamps, speech onset latency (first spoken syllable), words-per-minute (WPM) velocity, and volume dynamics ($\Delta$dBFS).
- 👁️ Multi-Modal Computer Vision: Qwen 2.5-VL executes keyframe autopsies, detects visual pattern interrupts, measures kinetic cut pacing, extracts on-screen OCR typography, and maps drop-off risk zones.
- 💬 Audience Comment Mining: Ingests thousands of real viewer comments to extract verbatim questions, objections, and sentiment clusters.
- ⚡ Dual-Engine Data Lake: DuckDB provides sub-millisecond analytical aggregations over millions of metrics; LanceDB provides sub-10ms semantic vector retrieval over 384-dimensional embeddings.
- 🗺️ Cartographic Knowledge Graph: Force-directed Louvain community clustering that maps creators, concepts, hook archetypes, and audience questions into an interactive thematic island archipelago.
flowchart TD
subgraph INGESTION["1. Multi-Modal Ingestion Pipeline"]
V[Raw Video MP4] --> W[OpenAI Whisper ASR]
V --> Q[Qwen 2.5-VL Vision]
V --> C[Audience Comments Engine]
end
subgraph TELEMETRY["2. Acoustic & Visual Telemetry Extraction"]
W --> T1["Word-Level Timestamps<br/>(WPM, First Speech Latency, dBFS)"]
Q --> T2["Keyframe Autopsy<br/>(Visual Triggers, Cut Velocity, OCR)"]
C --> T3["Audience Voice<br/>(Top Objections, Questions, Sentiment)"]
end
subgraph STORAGE["3. Dual-Engine Analytical Lake"]
T1 & T2 & T3 --> DUCK[(DuckDB Columnar Store<br/>Sub-millisecond OLAP)]
T1 & T2 --> LANCE[(LanceDB Vector Store<br/>Semantic Embeddings)]
end
subgraph GRAPH["4. Strategic Knowledge Graph"]
DUCK --> KGRAPH["NetworkX Louvain Detection<br/>(8 Thematic Archipelago Continents)"]
end
subgraph UI["5. Interactive Web Studio"]
DUCK & LANCE & KGRAPH --> FASTAPI[FastAPI Backend Server]
FASTAPI --> STREAM["HTTP 206 Partial Content<br/>(Sub-15ms Scrubbing)"]
FASTAPI --> REACT["React 19 + Vite + Tailwind UI<br/>(Split-Screen Karaoke Player & Knowledge Graph)"]
end
Stream high-bitrate video with instantaneous <15ms scrubbing powered by HTTP 206 Partial Content streaming. Watch the transcript scroll word-by-word with live millisecond audio sync.
Interactive force-directed graph powered by NetworkX and D3 force simulation. Uses Louvain community detection to cluster hundreds of reels, creators, hook archetypes, visual triggers, and audience questions into 8 color-coded thematic continents.
- Module 1: Hook Autopsy: 3-layer breakdown analyzing spoken words, visual interrupts, and on-screen text. Evaluates golden word count compliance (5–8 words).
- Module 2: Kinetic Pacing: Scene cut velocity analysis, drop-off risk zones, and OCR text extraction.
- Module 3: Audience Voice: Semantic clustering of real audience comments to pinpoint exact viewer questions and objections.
- Module 4: Algorithmic Virality Scorecard: Comprehensive scoring across hook strength, visual retention, and production quality.
- Module 5: Remake Blueprint: Director's exact replica script, visual storyboard cues, and A/B test variations.
Search across millions of spoken words, captions, and concepts. Pinpoint the exact second (25.4s) where any phrase or concept occurred in a video.
git clone https://github.com/your-username/vault-intel.git
cd vault-intel
run.batThis automatically starts the server and opens http://localhost:8000 in your default browser.
git clone https://github.com/your-username/vault-intel.git
cd vault-intel
chmod +x run.sh
./run.sh# 1. Create and activate virtual environment
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
# 2. Install dependencies
pip install -r requirements.txt
# 3. Configure environment (Optional)
cp .env.example .env
# 4. Start the server
python -m uvicorn app.backend.main:app --host 0.0.0.0 --port 8000 --reloadOpen http://localhost:8000 to access the studio.
# Build the Docker image
docker build -t vault-intel .
# Run container
docker run -p 8000:8000 vault-intelIn hackathons and developer ecosystems, open innovation is not merely about releasing code under an open license—it is about democratizing capability, protecting creator sovereignty, and enabling collective advancement.
Today, multi-modal video analytics suites are locked behind five-figure enterprise contracts available exclusively to billion-dollar talent agencies and media conglomerates. Solo creators, student educators, and indie filmmakers are left guessing. By open-sourcing VAULT INTEL, we place enterprise-grade acoustic and visual telemetry directly into the hands of every builder worldwide.
Creators produce confidential, unreleased concepts, pre-launch product pitches, and client video assets. Uploading raw footage to closed commercial SaaS platforms risks intellectual property leaks and non-consensual model training. Because VAULT INTEL runs locally with DuckDB and LanceDB, creators retain 100% data sovereignty. What happens in your vault stays in your vault.
Black-box virality tools assign arbitrary "scores" without explanation. VAULT INTEL exposes its entire mathematical and algorithmic architecture: from Whisper acoustic decibel drops ($\Delta$dBFS) to Louvain community modularity in NetworkX. Anyone can audit, test, fork, and calibrate the heuristics.
We welcome contributions of all forms! Please review CONTRIBUTING.md for details on code style, architecture, and pull request workflows.
- DuckDB columnar analytics and sub-millisecond aggregations.
- OpenAI Whisper word-level acoustic telemetry pipeline.
- Qwen 2.5-VL multi-modal keyframe and kinetic pacing autopsy.
- NetworkX Louvain Thematic Island Archipelago Knowledge Graph.
- 5-Module Video Teardown Studio with remake blueprints.
- v1.1: YouTube Shorts and TikTok scraping & telemetry adapters.
- v1.2: On-device WebGPU accelerated embeddings for zero-Python browser client.
This project is licensed under the MIT License - see the LICENSE file for details. Built with ❤️ for the open-source creator community.