Private, on-device motion experiments for learning how hand landmarks and gestures can drive effects on live video, recorded video, and still images.
This repository starts as an experiment system, not a fixed product. Each study must answer one clear question, expose its signal, and leave reusable parts for the eventual portfolio experience.
- Experiments 001 through 008 are implemented as separate, linkable routes; physical-camera studies remain pending.
- Homepage turns Research Spine into a visual sequence from landmarks to material response, while keeping complete study index below it. It explicitly directs visitors to deterministic fixtures or a user-started device camera.
- A Research Spine connects studies by phase, current position, evidence record,
and keep/revise/discard decision. Every study also has a camera-free,
shareable
#methodtechnical view. - Camera, image, and deterministic tracking or pinch fixture paths are available.
- Camera permission is requested only after the explicit Start camera action.
- Landmark inspection, temporal pinch intent, local pinch/fist calibration, five-family gesture competition, a velocity-driven particle field, and a bounded wireframe scene are implemented with recoverable failure states.
- No frame, photo, landmark, or telemetry data leaves the browser.
Requirements: Node.js 24+ and pnpm 11+.
pnpm install
pnpm devLocal development: open http://127.0.0.1:4173.
These links resolve against the GitHub Pages build. Repository changes appear there after the approved deployment completes. Method links never request camera permission.
- Home - Landing direction; Research Spine decision
- 001 - Landmark Explorer - Method; Brief
- 002 - Intent Gate - Method; Brief
- 003 - Gesture Calibration Bench - Method; Brief
- 004 - Gesture State Matrix - Method; Brief
- 005 - Motion Field - Method; Brief
- 006 - Object Manipulation Bench - Method; Brief
- 007 - Aperture Field - Method; Brief
- 008 - Aperture Object Set - Method; Brief
For a GitHub project page build, set VITE_BASE_PATH to repository path:
VITE_BASE_PATH=/gesture-proof/ pnpm buildNavigation, generated assets, and local model path then share same deployment base. Manual Pages workflow downloads model before building; it does not run until explicitly dispatched.
Before implementing hand tracking, download the official MediaPipe model:
pnpm models:fetchThe model is kept out of Git to avoid silently redistributing a large binary.
pnpm dev # local development server
pnpm test:unit # deterministic geometry tests
pnpm test:browser # browser lifecycle, privacy, model, and accessibility tests
pnpm build # type-check and production build
pnpm check # formatting, lint, unit tests, build
pnpm format # format repository filesproject-context.mdAGENTS.mddocs/experiments/001-landmark-explorer.mddocs/experiments/002-intent-gate.mddocs/experiments/003-gesture-calibration-bench.mddocs/experiments/004-gesture-state-matrix.mddocs/experiments/005-motion-field.mddocs/experiments/006-object-manipulation-bench.mddocs/experiments/007-aperture-field.mddocs/experiments/008-aperture-object-set.mddocs/home-landing.mddocs/ARCHITECTURE.md
experiments/ Independently linkable experiment HTML entry points
src/input/ Ephemeral camera lifecycle ownership
src/tracking/ Model adapter and decoupled video-frame scheduler
src/gesture/ Pure temporal gesture recognizers
src/effects/ Renderers consuming gesture signals
src/experience/ Experiment orchestration and UI state
src/engine/ Reusable contracts, geometry, fixtures, and measurements
docs/experiments/ One brief and result log per experiment
docs/decisions/ Architecture decision records
docs/research/ External research and source notes
public/models/ Locally downloaded ML models, ignored by Git
scripts/ Explicit setup utilities
Made by David Foliti for study, play, and making hand tracking easier to understand.
The sophiamyang/finger-frame-effect project by
Sophia Yang inspired this repository. No
source code from that repository is included.
Original project source code and documentation are licensed under the
Apache License 2.0. Third-party dependencies, models, and referenced
projects remain subject to their own licenses; see
THIRD_PARTY_NOTICES.md.
Licensing does not itself authorize deployment or publication. Complete the remaining Phase 5 checks and obtain explicit approval before either action.