Control music with your hands and your voice, no phone, no remote, no mouse.
Immersify turns a wall projection into a shared listening experience: mid-air gestures captured by a Kinect v2, natural-language voice commands transcribed locally by Whisper and parsed by Gemini, karaoke-synced lyrics, AI-generated trivia, and a reactive audio visualizer, all in one full-screen interface.
Gesture Recognition • Speech-to-Text • LLM Intent Parsing • Real-Time SSE • Audio Visualization
Final project, Next Generation User Interfaces (NGUI), M.Sc. Computer Science, A.Y. 2025–2026
Streaming a playlist is easy. Sharing music is not.
Music is a social, physical, emotional activity, but every mainstream player targets a single user staring at a phone. The queue becomes a private object passed around a room; the music becomes background noise.
Immersify removes the device from the loop. It is designed for wall projection in a living room, a party, or a bar. Everyone in the room sees the same screen, and anyone can take control by simply raising a hand or speaking out loud.
|
9 mid-air gestures for every playback action, recognized in ~200 ms with visual feedback before the action fires. |
Say "play Bohemian Rhapsody". Whisper runs locally for privacy; Gemini turns free speech into structured intents. |
A 60 FPS Spectral Mandala visualizer, karaoke-synced lyrics, and AI trivia, the palette re-themes itself for every song. |
A single Windows PC, a Kinect v2 on USB 3.0, and a projector. No extra camera and no extra microphone, the Kinect provides both.
flowchart LR
subgraph HW["Hardware"]
K["Kinect v2<br/>RGB 1920x1080<br/>IR 512x424<br/>4-mic array"]
P["Projector<br/>and system audio"]
end
subgraph BE["Backend - FastAPI on port 8002"]
KT["Kinect daemon thread<br/>gestures and voice"]
API["REST endpoints<br/>search, audio, lyrics, volume"]
SSE["SSE event bridge"]
end
subgraph EXT["External services"]
G["Genius API<br/>metadata and lyrics"]
Y["yt-dlp<br/>audio source"]
L["LRCLib<br/>synced lyrics"]
GM["Gemini 2.5 Flash<br/>intent and trivia"]
end
subgraph FE["Frontend - full-screen browser"]
UI["Player UI, Canvas visualizer<br/>lyrics, queue, fun facts"]
end
K --> KT
KT --> SSE
SSE --> UI
UI --> API
API --> UI
API --> G
API --> Y
API --> L
API --> GM
UI --> P
Three layers, one process:
| Layer | Responsibility |
|---|---|
Backend (app.py) |
FastAPI on port 8002. All external API calls, audio download/cache, Windows volume control, SSE fan-out. Every blocking call (yt-dlp, pycaw COM, librosa) is offloaded with asyncio.to_thread so the event loop never stalls. |
Perception (src/kinect/) |
A daemon thread running the dual-stream RGB/IR gesture pipeline and, on demand, the Whisper + Gemini voice pipeline. |
Frontend (src/frontend/) |
Zero-build vanilla HTML/CSS/JS. Web Audio API for the visualizer, EventSource for real-time gesture events. |
Nine gestures cover the entire playback surface. Each one has a dedicated preview animation that starts the instant the gesture is first detected and completes only if it is held long enough, so the user always knows what is about to happen, and can back out by lowering their hand.
| Gesture | Action | Preview animation |
|---|---|---|
| 👌 OK | Start a voice command | Wave indicator appears at the top |
| 🖐️ STOP | Play / Pause | A ring pulses around the play button |
| ☝️ UP | Volume +10 % | A + badge floats above the volume bar |
| 👇 DOWN | Volume −10 % | A − badge floats above the volume bar |
| 👉 RIGHT | Next track | Next queue item pulses cyan |
| 👈 LEFT | Previous track | Previous queue item pulses purple |
| ✊ FIST | Mute / Unmute | The mute icon shimmers |
| 🤙 COOL | Toggle shuffle | The shuffle button orbits |
| ✌️ PEACE | Toggle repeat | The repeat button bounces |
Design principle: feedback before commitment.
gesture_preview→ (held) →execute, orgesture_cancelif the hand moves away.
Raise 👌 and speak. Gesture recognition pauses while listening, voice and gesture are treated as mutually exclusive modalities rather than fused, keeping the recognition pipeline simple and predictable.
sequenceDiagram
autonumber
participant U as User
participant K as Kinect thread
participant W as Whisper local
participant G as Gemini 2.5 Flash
participant B as Browser
U->>K: OK gesture
K->>B: voice_state listening, music pauses
U->>K: speaks - play Bohemian Rhapsody
Note over K: VAD hysteresis<br/>start above RMS 0.003<br/>stop after 1.2 s below 0.0007<br/>plus 500 ms pre-buffer
K->>B: voice_state elaborating, music resumes
K->>W: 16 kHz mono, peak-normalized audio
W-->>K: transcription, 2 to 5 s on CPU
K->>G: prompt and transcription, search grounding on
G-->>K: JSON with action, song, artist
K->>B: voice_state searching, track starts
Why it feels fast: the elaborating event fires the moment silence is detected, before Whisper even starts.
Music resumes immediately instead of waiting 5–8 s for the AI round-trip, cutting perceived latency dramatically.
Why local Whisper: speech never leaves the machine. The base model handles accented speech and moderate
background noise well, and stays practical on CPU.
A single full-screen dark page, tuned for readability on a projected surface: near-black background with cyan and purple accents, plus a dominant color extracted from the album art that re-themes the visualizer and background gradient for every track.
✳️ Spectral Mandala, four reactive layers
Rendered on an HTML5 Canvas at ~60 FPS via requestAnimationFrame, driven by a 256-bin FFT taken straight
from the Web Audio API AnalyserNode (no backend round-trip during playback).
| Layer | Reacts to |
|---|---|
| Outer glowing halo | Bass energy, a slow breathing expansion |
| 90 radial bars | Per-band amplitude, fading cyan → purple |
| Rotating waveform ring | Mid-range frequencies |
| Central pulsing circle | Overall audio energy |
🎤 Synchronized lyrics, a three-tier fallback chain
- LRCLib, true LRC timestamps.
parseLRC()builds ordered time–text pairs; the active line is highlighted with a larger font and glow while smooth scrolling keeps it centered. - Onset detection, if no LRC exists, the cached MP3 is analyzed with
librosaand the Genius lyrics are distributed across detected note onsets. Not frame-perfect, but it follows the musical structure. - Static text, Genius lyrics as a plain scrollable panel.
🔍 Search & retrieval, a three-stage pipeline
Stage 1, Genius. A bare title, a bare artist, or Title, Artist all work. With a dash separator both
orderings are tried. Returns title, artist, album, lyrics and album art (cover_art_url → song_art_image_url
→ header_image_url).
Stage 2, YouTube. ytsearch5:{title} {artist} fetches five candidates, scored by a hand-tuned heuristic
built to beat the classic "first result is a live cover" problem:
| Signal | Score |
|---|---|
| YouTube Music auto-generated "Topic" channel | +50 |
| VEVO channel | +40 |
| Title match in video title | +30 |
| Artist in channel name | +25 |
| Artist match in video title / verified channel | +20 |
| Duration within 3 s / 8 s / 15 s of expected | +60 / +30 / +10 |
| Duration off by more than 30 s | −25 |
cover, remix, karaoke, tribute, live, acoustic, instrumental |
−40 each |
Stage 3, Delivery. The winner is downloaded as M4A, transcoded to 192 kbps MP3 with ffmpeg, cached in
memory under "title||artist", and served as a StreamingResponse with a correct Content-Length.
An asyncio.Lock per cache key means two simultaneous requests for the same song share one download
instead of spawning duplicate yt-dlp processes.
💡 Fun facts & voice states
Fun facts. On each new track the frontend calls /api/get_fun_facts; Gemini 2.5 Flash returns five concise
facts separated by §, displayed one at a time with auto-rotation every 6 s (manual navigation and an
auto-rotate toggle are provided).
Voice states. The wave indicator cycles through four visual states:
LISTENING (cyan, music paused) → ELABORATING (purple, music resumed) → SEARCHING (purple + detected song)
→ ERROR (red, "SORRY, TRY AGAIN" for 2.5 s).
Getting reliable hand tracking in a dark projection room, where a projector also throws harsh, shifting light onto the user's hands, is the core technical problem of this project. The answer is a per-frame adaptive pipeline.
flowchart TD
A["Kinect frame at 30 fps"] --> B{"90th percentile<br/>grayscale brightness"}
B -->|"above 60 of 255"| C["RGB stream"]
B -->|"at or below 60 of 255"| D["IR stream<br/>16 to 8 bit, gain 2.5x"]
C --> E["MediaPipe HandLandmarker<br/>VIDEO mode, 21 3D landmarks"]
D --> E
E --> F["77-D feature vector<br/>63 normalized XYZ<br/>9 distances, 5 angles"]
F --> G["Random Forest<br/>RGB or IR model"]
G --> H{"confidence at least 0.70"}
H -->|"no, RGB path"| I["Cross-check IR classifier<br/>higher confidence wins"]
H -->|"yes"| J["Temporal stability filter"]
I --> J
J --> K["gesture_preview, then execute or cancel"]
K --> L["SSE to browser"]
| Adaptive stream switching |
The 90th percentile of grayscale intensity is used instead of the mean, so large dark regions (clothing,
background) don't drag the estimate down, what matters is the light actually hitting the hand. Above |
|---|---|
| Feature engineering |
77 dimensions, designed for invariance to hand size, position and wrist tilt: 63 landmark coordinates centered on the wrist and scaled by max inter-landmark distance · 9 distances (5 wrist→fingertip, 4 adjacent fingertip pairs) capturing extension · 5 joint angles at the middle phalanges via three-point cosine similarity, capturing curvature. |
| Temporal stability filter |
A small state machine converts noisy per-frame predictions into intentional actions:
|
Two datasets were collected, one under normal room lighting (RGB) and one under low/no ambient light (IR), across 9 gesture classes and 10 different people, then split 80/20 with stratification.
| Model | Key hyperparameters | RGB | IR |
|---|---|---|---|
| Random Forest ⭐ | n_estimators=100, max_depth=15, min_samples_split=10 |
✅ | ✅ |
| SVM | kernel=RBF, C=5, gamma=scale, probability=True |
✅ | ✅ |
| XGBoost | n_estimators=150, max_depth=10, min_child_weight=3 |
✅ | ✅ |
| LightGBM | n_estimators=150, min_child_samples=20, reg_lambda=0.1 |
✅ | ✅ |
Hyperparameters were deliberately reduced from initial tuning values (RF started at 300 estimators, SVM at
C=10) after observing overfitting on the small per-class sample counts.
⭐ Random Forest is deployed on both streams. It was consistently the most accurate on both datasets, is fast single-threaded at inference (
n_jobs=1), and its feature-importance output makes the classifier interpretable, we can see exactly which landmarks drive each decision.
All eight trained models live in src/kinect/models/, and the full evaluation artifacts,
confusion matrices for every model/stream combination and feature-importance plots, are generated into
src/kinect/reports/:
ℹ️
LEFTandRIGHTare under-represented in the datasets: the usual horizontal-flip augmentation is unavailable here, because mirroring a frame inverts the very orientation that defines those two gestures.
Retrain everything with:
py src/kinect/train/train_models.pyThe script trains all four architectures on both streams, prints a full accuracy / precision / recall / F1
comparison table (macro and weighted) with training times, and writes the reports and .joblib artifacts.
The hard constraint: Kinect capture, MediaPipe inference and Whisper are blocking, while SSE delivery must stay on the asyncio event loop. Immersify bridges the two worlds with a queue hand-off:
Kinect daemon thread → queue.Queue → asyncio poller (16 ms) → asyncio.Queue per client → SSE → EventSource
This keeps blocking perception work in a plain thread while every connected browser gets its own non-blocking event stream, end-to-end gesture-to-animation latency lands at ~33–50 ms.
During development a genuine bug was found in PyKinect2's PyKinectRuntime.py: color frames were requested in
RGBA while OpenCV expects BGRA, swapping the red and blue channels in every frame and silently breaking
all RGB-based processing. Fixed by switching the color format constant to ColorImageFormat_Bgra, which restored
correct channel ordering across the whole pipeline.
Windows master volume is driven through pycaw (a WASAPI COM wrapper). Since COM calls are synchronous, every
read/set/mute operation is isolated in a helper and dispatched via asyncio.to_thread. The browser polls state
at most once every 10 s, while any user action triggers an immediate update.
The first implementation streamed audio to the browser while yt-dlp was still downloading it. It failed:
Windows file-locking in yt-dlp intermittently blocked concurrent reads, and unreliable chunked output broke
browser buffering and seeking. The final design downloads fully, transcodes, caches in memory, and serves with a
proper Content-Length, costing a one-time wait per song in exchange for rock-solid playback and seeking.
| Method | Endpoint | Description |
|---|---|---|
GET |
/ |
Serves the full-screen player (src/frontend/index.html) |
GET |
/static/* |
CSS / JS assets |
GET |
/api |
Service metadata |
GET |
/api/search |
Genius search → normalized song metadata |
GET |
/api/audio |
Cached MP3 bytes for a track |
GET |
/api/stream |
Streaming audio response |
GET |
/api/audio_nowplaying |
Downloads + serves the current track as MP3 |
GET |
/api/lyrics_lrc |
LRCLib synced lyrics, Genius fallback |
GET |
/api/analyze_lyrics |
Onset-detection timing for unsynced lyrics |
GET |
/api/spectrum |
Mel-spectrogram bands for the visualizer |
GET |
/api/get_fun_facts |
Gemini-generated song trivia |
POST |
/api/clear_fun_facts_cache |
Invalidate the trivia cache |
GET POST |
/api/volume |
Read / set Windows system volume (0–100) |
POST |
/api/volume/mute |
Toggle system mute |
GET |
/api/status · /api/playlist |
Player state and queue |
POST |
/api/play · /api/pause · /api/resume · /api/next |
Playback control |
GET |
/api/gesture_events |
📡 SSE stream, real-time gesture & voice events |
POST |
/api/gesture |
Inject a gesture event manually (testing without hardware) |
| Event | Measured latency | |
|---|---|---|
| Gesture preview → browser animation | ~33–50 ms | 🟢 well inside the 100 ms direct-manipulation threshold |
| Gesture execute (stable → action) | ~200 ms | 🟢 6 frames @ 30 fps |
| SSE poller wakeup interval | 16 ms | 🟢 |
| LRCLib lyrics fetch | ~200–500 ms | 🟢 |
| Gemini intent parsing | ~1–3 s | 🟡 |
| Whisper transcription (CPU, 3–5 words) | ~2–5 s | 🟡 GPU would cut this sharply |
| Voice command round-trip | ~4–10 s | 🟡 dominated by CPU Whisper |
| First play (yt-dlp + ffmpeg) | ~10–30 s | 🟠 once per song, scales with duration |
| Subsequent plays (in-memory cache) | < 1 ms | 🟢 |
A User Experience Questionnaire (UEQ) evaluation was run with 10 participants across the six standard dimensions, Attractiveness, Perspicuity, Efficiency, Dependability, Stimulation, Novelty, comparing two interface generations: V1 (initial Tkinter design) vs V2 (the current browser interface).
V1 scored 0.6–1.5 across dimensions, with Perspicuity and Efficiency as the clear weak points while Stimulation and Novelty trended higher, the gesture interaction itself held interest, but the interface got in the way. V2 responded with a clearer visual hierarchy, explicit gesture labelling, a shortened interaction path for common tasks, stronger recognition feedback, and improved contrast and typography. The largest gains landed exactly on Perspicuity and Efficiency, the two dimensions that most gate adoption of gesture interfaces.
⚠️ Stated limitation: participants were recruited from close contacts of the team, introducing selection and social-desirability bias and limiting generalizability. The comparison between iterations remains informative; the absolute scores should be read with that caveat.
Full methodology, charts and related-work discussion: 📄 doc/final_report.pdf
| 🐍 Python 3.11 | Required, the Kinect and audio stack has strict version constraints |
| 🪟 Windows 10 / 11 | pycaw and PyKinect2 are Windows-only |
| 🎬 ffmpeg | On PATH, used for MP3 transcoding |
| 📷 Kinect for Windows v2 + SDK v2 | Optional, the web player runs fine without it |
| 🎤 Microphone | The Kinect 4-mic array is auto-detected and preferred |
py -3.11 -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -r requirements.txtpip install sounddevice openai-whisper mediapipe joblib scikit-learn pykinect2
⚠️ PyKinect2requires the Kinect for Windows SDK v2 to be installed system-wide.
copy .env.example .envGENIUS_ACCESS_TOKEN=your_genius_access_token_here
GEMINI_API_KEY=your_gemini_api_key_here| Key | Where to get it | Cost |
|---|---|---|
GENIUS_ACCESS_TOKEN |
genius.com/api-clients | Free |
GEMINI_API_KEY |
Google AI Studio | Free tier |
py app.pyOpen http://localhost:8002 and go full-screen. The Kinect gesture thread starts automatically when the hardware is detected, and the player stays fully usable with mouse and keyboard when it isn't.
🔧 Standalone debug entry points
py src/kinect/multicam_kinect.py # live gesture inference with a camera preview window
py src/kinect/vocal_commands.py # Whisper + Gemini voice command loop
py src/kinect/create_dataset_kinect.py # capture new labelled gesture samples
py src/kinect/train/train_models.py # retrain all 8 classifiers + reportsHardware smoke tests (camera, IR, microphone, volume) live in src/kinect/test/.
No Kinect? Drive the UI by POSTing to /api/gesture.
Immersify/
├── app.py # 🎛️ FastAPI server, REST, SSE, audio cache, volume
├── requirements.txt
├── .env.example # API key template (.env is git-ignored)
├── doc/
│ └── final_report.pdf # 📄 Full academic report: design, evaluation, related work
└── src/
├── audio_player.py # 🔊 MP3 playback via pygame-ce
├── audio_visualizer.py # 📊 Mel spectrogram / FFT analysis
├── genius_searcher.py # 🎼 Genius search + lyrics + album art
├── youtube_player.py # ⬇️ yt-dlp extraction with the scoring heuristic
├── playlist_manager.py # 📃 Queue state
├── fun_facts_manager.py # 💡 Gemini trivia generation
├── frontend/
│ ├── index.html # 🖥️ Single-page player
│ ├── immersify.css # 🎨 Full dark theme + gesture preview animations
│ └── immersify.js # ⚙️ Visualizer, LRC parser, SSE client, controls
└── kinect/
├── multicam_kinect.py # 👋 RGB/IR capture, landmarks, features, inference
├── multimodal_manager.py # 🔀 Gesture ID → player action mapping
├── vocal_commands.py # 🎙️ VAD + Whisper + Gemini intent parsing
├── copilot.py # 🤖 Gemini API wrapper
├── create_dataset_kinect.py# 📥 Dataset capture tool
├── music_assistant_prompt.txt
├── hand_landmarker.task # MediaPipe hand landmark model
├── models/ # 🧠 8 trained classifiers (RF · SVM · XGB · LGBM × RGB · IR)
├── reports/ # 📈 Confusion matrices + feature importance plots
├── train/ # 🏋️ Datasets and training pipeline
└── test/ # 🔧 Standalone hardware test scripts
- ⚡ GPU-accelerated Whisper, target sub-2 s voice command round-trip
- 🎯 Gesture calibration tool, per-user adaptation to hand size and posture
- 🎬 Music video rendering, synchronized visuals beyond the Mandala
- 🧩 Song quiz mode, the one feature cut from the original requirements to protect the core interaction
FastAPI · uvicorn · PyKinect2 · MediaPipe · OpenCV · scikit-learn · XGBoost · LightGBM
openai-whisper · google-genai · lyricsgenius · yt-dlp · librosa · pygame-ce · pycaw · sounddevice
Immersify, making shared music listening active, gestural, and immersive.
Next Generation User Interfaces · M.Sc. Computer Science · A.Y. 2025–2026