On-device inspection perception for Odin-Nav-Stack2.
Feed one image to a vision model, get back decidable conclusions against a structured
checklist
(yes/no, state, count, read-text, describe) — plus evidence image and latency.
The current release ships the Florence-2 (grounding) pipeline.
中文文档见
README.zh-CN.md。
| Date | New feature |
|---|---|
| 2026-11-09 | Annotation overlay on the published image (detection boxes + text verdict, configurable line count and font size) |
| 2026-11-09 | GPU inference support on Jetson (NVIDIA platform wheel, --dtype flag) |
| 2026-11-09 | One-command live Q&A script (scripts/ask_live.sh) |
| Backend | Capability | Status | Notes |
|---|---|---|---|
florence2 |
Boxes / segmentation / OCR (no free-form VQA) | ✅ recommended | 0.23B, runs on CPU and GPU; yes_no / state / count require a target phrase |
dummy |
No model loaded | ✅ | Verify the plumbing before downloading weights |
moondream / smolvlm |
Free-form Q&A + OCR | Bring your own weights; not part of the verified matrix | |
| ROS 2 capture / annotation publishing | adapters/ros2_capture |
✅ | Subscribe to camera topics → HTTP frame server; inference posts annotations back |
Not in this repository: model weights (downloaded separately), third-party VLM toolkits, and ROS nodes / task orchestration (those belong to the navigation stack).
# 1) Create an isolated environment — do NOT install into system python
bash scripts/setup.sh
source .venv/bin/activate
# 2) Self-check: confirm the plumbing works before downloading weights
python scripts/doctor.py
# 3) Run the pipeline without a model
perception-run --backend dummy --source file --file-path ./demo.jpg \
--tasks configs/tasks/smoke.json --out /tmp/smoke.json
# 4) Download Florence-2 weights (~888 MB)
bash scripts/download_models.sh florence2-base-ft
python scripts/doctor.py # florence2 should now report "yes"
# 5) Real run
perception-run --backend florence2 --source file --file-path ./demo.jpg \
--tasks configs/tasks/demo_florence2.yaml --out /tmp/result.jsonResolve and download wheels on a machine that has network access, then copy them over:
# On the online machine — match the target platform exactly
pip download -r requirements.txt "numpy<1.25" \
--platform manylinux2014_aarch64 --python-version 38 \
--only-binary=:all: -d ./whl
# Build helpers are required or 'pip install .' fails with bdist_wheel missing
pip download wheel setuptools --platform any --python-version 38 \
--only-binary=:all: -d ./whl
# On the device
python3 -m venv .venv-edge && source .venv-edge/bin/activate
pip install --no-index --find-links=./whl wheel setuptools
pip install --no-index --find-links=./whl -r requirements.txt
pip install --no-index --no-build-isolation . # NOT -e: pip on py3.8 lacks PEP 660Copy the models/florence2-base-ft directory separately (~888 MB) — weights are
platform independent, binaries are not.
Florence-2 is loaded through transformers remote code and is version sensitive:
| Package | Constraint | Why |
|---|---|---|
transformers |
==4.46.3 | 5.x lacks forced_bos_token_id; 4.47+ changes the processor output and inference fails with 'NoneType' object has no attribute 'shape' |
timm |
==1.0.9 | Matches the Florence-2 modeling file |
einops |
>=0.7 | Required by remote code |
torch |
CPU: ==2.4.1 Jetson: NVIDIA wheel |
torch>=1.13: pip resolves to 2.1.0, which segfaults (exit 139) inside the Florence-2 DaViT convolution with no error output at all. 2.4.1 is the newest CPU wheel for py3.8 |
torchvision |
must match torch | timm 1.0.9 needs it (from torchvision.ops.misc import FrozenBatchNorm2d); without it Florence-2 fails to load. torch 2.0.0 ↔ tv 0.15.1, torch 2.4.1 ↔ tv 0.19.1 |
scripts/setup.sh installs exactly what requirements.txt pins. If you changed versions
and inference broke, run pip install -r requirements.txt --force-reinstall to get back
inside the constraints; scripts/doctor.py warns when they drift.
A plain pip install torch gives you the PyPI CPU wheel (torch.cuda.is_available() == False).
On Jetson you must install NVIDIA's platform-specific wheel, chosen by JetPack version:
# JetPack 5.1.1 (L4T 35.3.1) ships CUDA 11.4
curl -LO "https://developer.download.nvidia.com/compute/redist/jp/v511/pytorch/torch-2.0.0+nv23.05-cp38-cp38-linux_aarch64.whl"
pip install --no-deps --force-reinstall "torch-2.0.0+nv23.05-cp38-cp38-linux_aarch64.whl"
pip install --no-deps "torchvision==0.15.1" # required by timmMeasured on Orin NX 8 GB, Florence-2 base, 1600×1296 input:
| CPU | GPU | |
|---|---|---|
| Per-check inference | 46 800 ms | 1 200 ms (≈38×) |
| Model load | 27–44 s | 15–17 s |
Two GPU-specific gotchas:
--dtype float16does not work with Florence-2 —"exp_vml_cpu" not implemented for 'Half'(the remote code contains CPU operators). Use--device cuda --dtype float32.- Check
python -c "import torch;print(torch.cuda.is_available())"isTruebefore going further, otherwise you are just running CPU slower.
Integrating with Odin-Nav-Stack2
The perception component is decoupled from navigation. Integration is two processes talking over HTTP, which means you never have to merge the model environment into the ROS workspace.
Odin-Nav-Stack (ROS) Odin-perception
odin driver ──► /odin1/image/undistorted
│
▼
ros1_capture / ros2_capture ──HTTP──► perception-run
(frame server :8730) (Florence-2, venv)
▲ │
└────── POST /annotate ◄─────────────┘
│
▼
/inspection/annotated/compressed ──► Foxglove / RViz / rosbridge
The perception side can only see what the driver publishes. On the Odin driver, check the stream switches first:
grep -E "^ (sendrgb|sendrgbundistort):" <odin_ros_ws>/src/odin_ros_driver/config/control_command.yamlsendrgb must be 1; sendrgbundistort: 1 depends on sendrgb and does nothing on
its own. Then confirm data actually flows:
rostopic hz /odin1/image/undistorted # expect ~10 HzFollow Installation. Keep it in its own venv — the navigation stack already uses the system python and torch, and mixing the two is the single most common cause of hard-to-diagnose failures on edge devices.
ROS 1 (Noetic):
source /opt/ros/noetic/setup.bash
cd <path>/Odin-perception
# Use the SYSTEM interpreter here: it carries rospy. A conda python3 will not have it.
PYTHONPATH="/usr/lib/python3/dist-packages:.${PYTHONPATH:+:$PYTHONPATH}" \
/usr/bin/python3 -m inspection_perception.capture.ros1_odin_node --serve \
--prefer raw --raw-topic /odin1/image/undistorted --port 8730ROS 2 (Jazzy / Humble):
python -m inspection_perception.adapters.ros2_capture --serve \
--raw-topic /odin1/image/undistorted --prefer raw --long-side 768 --port 8730Verify before moving on:
curl -s http://127.0.0.1:8730/meta.json # "has_frame": true
curl -s -o /tmp/f.jpg -w "%{http_code}\n" http://127.0.0.1:8730/latest.jpgsource <venv>/bin/activate
cd <path>/Odin-perception
perception-run \
--source http --http-url http://127.0.0.1:8730 \
--backend florence2 --model-dir models/florence2-base-ft \
--device cuda --dtype float32 \
--tasks configs/tasks/demo_florence2.yaml \
--annotate-url http://127.0.0.1:8730 \
--out /tmp/result.json--annotate-url turns on overlay publishing (boxes + verdict text). Omit it if you only
want JSON output.
# Subscribe FIRST: this topic is not latched, so echo-after-publish shows nothing
rostopic echo -n1 /inspection/annotated/compressed --noarrA sensor_msgs/CompressedImage with format: "jpeg" should arrive for each check.
scripts/ask_live.sh wires steps 3–4 together and defaults to GPU:
bash scripts/ask_live.sh # interactive
bash scripts/ask_live.sh "What text is printed on the nameplate" # single question| Item | Value |
|---|---|
| Topic consumed | /odin1/image/undistorted (raw preferred); --prefer compressed falls back to /odin1/image/compressed (~5 MB/s instead of ~500 MB/s) |
| Topic produced | /inspection/annotated/compressed (sensor_msgs/CompressedImage) |
| HTTP endpoints | GET /latest.jpg, GET /meta.json, GET /healthz, POST /annotate |
| Default port | 8730 — check ss -lntp | grep 8730 for stale instances |
| Output | One CheckResult per item (item_id / status / answer / reason / latency_ms / evidence / checked_at) plus a summarize block |
With --annotate-url set, every check renders its boxes and verdict text onto the
image the model actually saw, then posts it back for publishing.
--annotate-url http://127.0.0.1:8730 # enable publishing
--max-text-lines 3 # bottom panel line cap (default 3, overflow → …)
--text-scale 1.4 # font-size multiplier (default 1.0)Layout rules:
- The text panel is pinned to the bottom (dark background, white text) so it never covers the subject being inspected.
- At most
--max-text-lineslines are drawn — the image height does not grow unbounded. - Font size scales with the image long side, then multiplies by
--text-scale. - Box color follows the verdict: green = pass, red = fail, yellow = unknown, grey = error.
Implementation: inspection_perception/annotation.py (render_result, draw_text_panel,
iter_boxes). Pillow only — no new dependency. Fonts are auto-detected (Noto CJK / WenQuanYi),
falling back to the Pillow default (CJK renders as boxes, Latin and digits stay readable).
Overlay is a side channel: network errors, missing fonts or encode failures are swallowed and never turn a check into
ERROR.
perception-run --tasks <yaml|json> \
--source auto|ros1|ros2|http|file [--file-path ...] [--http-url ...] \
[--prefer compressed|raw|both] [--max-age-ms 1000] \
--backend florence2|moondream|smolvlm|dummy [--model-dir ...] \
[--device cuda|cpu] [--dtype auto|float32|float16|bfloat16] \
[--infer-long-side 768] [--evidence-dir runs/evidence] \
[--annotate-url URL] [--max-text-lines 3] [--text-scale 1.0] \
--out result.json
| Flag | Meaning |
|---|---|
--tasks |
Checklist (YAML/JSON); each item has id/question/kind, detection kinds also need target |
--source |
file offline image, http frame server, ros1/ros2 subscribe directly, auto autodetect |
--backend |
Defaults to moondream — always pass --backend florence2 explicitly |
--dtype |
Default float32. Keep it that way for Florence-2 |
--evidence-dir |
Evidence image directory; empty string disables |
See configs/tasks/:
smoke.json— minimal smoke test (works withdummy)demo_florence2.yaml— Florence-2 detection examples (kindandtargetin pairs)
kind values: yes_no / state / count / read_text / describe.
Florence-2 only understands fixed task tokens, so the first three require an English
target phrase, otherwise the result degrades to UNKNOWN.
inspection_perception/ # core package: vlm/ capture/ checks/ pipeline cli schemas annotation
configs/ # task / profile / capture examples
scripts/ # setup, download_models, doctor, ask_image, ask_live
tests/ # zero-dependency unit tests (no weights, no ROS)
adapters/ # ROS thin adapters (optional): capture + annotation publishing
Validated on Jetson NX + ROS 1 Noetic + python 3.8. Do these up front and you skip all of them:
| # | Symptom | Root cause | Fix |
|---|---|---|---|
| 1 | Inference produces no output at all, exit code 139 | torch>=1.13 resolved to 2.1.0; convolution segfaults |
Pin torch==2.4.1 (CPU) or the NVIDIA wheel |
| 2 | pip install -e . complains about setup.py |
pip on py3.8 lacks PEP 660 | pip install --no-build-isolation . |
| 3 | invalid command 'bdist_wheel' |
wheel missing in the venv |
Install wheel + setuptools first |
| 4 | Frame node: No module named 'rospy' |
miniconda shadows python3 |
Use /usr/bin/python3 for the capture side |
| 5 | Still no rospy after switching interpreter | export PYTHONPATH=. overwrites the ROS path |
Append instead: export PYTHONPATH=".${PYTHONPATH:+:$PYTHONPATH}" |
| 6 | import rospy fails on rospkg |
rospkg lives in /usr/lib/python3/dist-packages |
Add that directory to PYTHONPATH too |
| 7 | Image topic exists but rostopic hz shows nothing |
driver has sendrgb: 0, and sendrgbundistort depends on it |
Set sendrgb: 1 and sendrgbundistort: 1, restart the driver |
| 8 | Frame server returns a stale frame (age_ms in the tens of thousands) |
the capture node's subscription died when the driver restarted | Restart the capture node to re-subscribe |
| # | Item | Note |
|---|---|---|
| 1 | torch must be a platform wheel | aarch64 Jetson wheel (validated 2.0.0+nv23.05, CUDA 11.4); x86 binaries cannot be copied over |
| 2 | transformers pinned to 4.46.3 | plus timm==1.0.9, einops |
| 3 | python 3.8 | Keep from __future__ import annotations; avoid 3.9+ syntax |
| 4 | weights are portable | The 888 MB directory can be copied; verify config.json, preprocessor_config.json and the remote-code .py files are present |
| 5 | undistorted topic needs the driver switch | sendrgbundistort: 1, or the topic is advertised with no data |
| 6 | port conflicts | Frame server defaults to 8730; check ss -lntp | grep 8730 |
| 7 | disk | venv (~1.5 GB with torch) + 888 MB weights — run df -h first |
| 8 | never install into system python | The navigation stack uses the system interpreter; always isolate in a venv |
- Weights are not in the repo:
models/is gitignored; usescripts/download_models.sh. On an offline device, download them on another machine and copy the directory. - No CUDA:
--device cpuworks; expect seconds rather than ~1 s per check. - No ROS:
--source file/httpdoes not touch ROS at all; ROS modules are lazily imported. - All
yes_nocome backUNKNOWN: confirm--backend florence2was passed and each item has atarget. - Slow or failing downloads:
export HF_ENDPOINT=https://hf-mirror.com, then re-run; the script verifies file completeness.
Apache-2.0 — see LICENSE. Third-party licenses in THIRD_PARTY_LICENSES.md.
Model weights are not distributed here; their licenses come from upstream
(Florence-2 is MIT).
