English · 简体中文
Give a non-multimodal main model the ability to "see" pasted screenshots — a ~200-line Claude Code Hook.
If you use Claude Code with a non-multimodal main model (GLM, DeepSeek, Qwen-Text, etc.), pasting a screenshot gets you a response like "Please tell me what you'd like me to do" — the model literally cannot see the image. The image content block is sent to the API and silently dropped.
This project solves it with a UserPromptSubmit hook: when the user pastes an image, the hook automatically calls a multimodal model to parse the image into text and injects that text into the main model's context. The main model stays cheap and fast — it just borrows eyes.
flowchart LR
A[User pastes screenshot+text] --> B[UserPromptSubmit Hook fires]
B --> C[Scan image-cache for new images]
C --> D[Call multimodal API to parse image]
D --> E[Return additionalContext text]
E --> F[Main model receives: user text + image description]
F --> G[Main model answers normally]
📹 A real terminal-recorded GIF would go here (
docs/demo.gif). PRs welcome — seedocs/demo.md.
Before (non-multimodal main model, no hook):
You: [pastes screenshot] what's wrong with this code?
AI: I see you've shared an image, but I'm not able to view images directly.
Could you describe what's in the screenshot or paste the code as text?
After (same main model, CC-Vision installed):
You: [pastes screenshot] what's wrong with this code?
[status] 解析图片中...
AI: Looking at the screenshot, the error is on line 42 — `foo` is undefined.
You imported `fool` but called `foo`. Fix: change `foo()` to `fool()`.
The main model never sees pixels — it sees a textual description of the image injected via additionalContext. See docs/demo.md for details and how to contribute a real GIF.
- Non-invasive: main model config untouched, just adds a hook
- Any OpenAI-compatible multimodal API: OpenAI / DashScope / Zhipu / SiliconFlow / Moonshot, etc.
- Provider presets:
VISION_PROVIDER=openaifills in base_url + default model - Idempotent: the same image is never parsed twice (tracked in
.vision-processed) - Never blocks: any crash returns
{}, the conversation continues - Standalone test mode:
--test-imageverifies API config without going through Claude Code
Export in your shell (or add to ~/.zshrc / ~/.bashrc):
# Option A: use a provider preset (fills base_url + default model)
export VISION_PROVIDER=openai
export VISION_API_KEY=sk-...
# VISION_MODEL is optional; preset sets a default. Override if you want:
# export VISION_MODEL=gpt-4o-mini
# Option B: fully custom
export VISION_API_BASE=https://api.openai.com/v1/chat/completions
export VISION_API_KEY=sk-...
export VISION_MODEL=gpt-4ogit clone https://github.com/odebo/CC-Vision.git
cd CC-Vision
./install.shinstall.sh will:
- Copy
image-vision.pyto~/.claude/hooks/ - Register the
UserPromptSubmithook in~/.claude/settings.json(idempotent — won't duplicate)
python3 ~/.claude/hooks/image-vision.py --test-image /path/to/some/screenshot.pngYou should see the API config + the image description.
Open the /hooks menu once, or restart Claude Code, to load the new hook. Then paste a screenshot.
All via env vars — no code changes:
| Var | Default | Description |
|---|---|---|
VISION_PROVIDER |
(empty) | Provider preset: openai / dashscope / zhipu / siliconflow / moonshot |
VISION_API_BASE |
OpenAI official | Any OpenAI-compatible chat completions endpoint |
VISION_API_KEY |
required | API key |
VISION_MODEL |
gpt-4o |
Multimodal model name |
VISION_EXTRA_HEADERS |
(empty) | JSON object string of extra request headers (e.g. for internal gateways) |
VISION_TIMEOUT |
45 |
Per-API-call timeout in seconds |
VISION_MAX_TOKENS |
1200 |
Max tokens for the parsed description |
VISION_RECENT_WINDOW |
3600 |
Only consider images modified within the last N seconds |
| Provider | API_BASE | Default MODEL |
|---|---|---|
openai |
https://api.openai.com/v1/chat/completions |
gpt-4o |
dashscope |
https://dashscope.aliyuncs.com/compatible-mode/v1/chat/completions |
qwen-vl-max |
zhipu |
https://open.bigmodel.cn/api/paas/v4/chat/completions |
glm-4v-plus |
siliconflow |
https://api.siliconflow.cn/v1/chat/completions |
Qwen/Qwen2-VL-72B-Instruct |
moonshot |
https://api.moonshot.cn/v1/chat/completions |
moonshot-v1-8k-vision-preview |
Need a new provider? Edit PROVIDER_PRESETS at the top of image-vision.py (PRs welcome).
- Claude Code caches pasted images to
~/.claude/image-cache/<session-id>/<n>.png - User presses Enter, triggering the
UserPromptSubmithook - The script scans the cache dir for images that are "within the recent window AND not yet processed"
- For each new image, it calls the multimodal API and gets a text description
- All descriptions are wrapped in
<image_vision>...</image_vision>and returned viahookSpecificOutput.additionalContext - Claude Code injects this text into the main model's current-turn context
- The main model answers based on the text — it sees words, not pixels
~/.claude/image-cache/.vision-processed records every image path that has been parsed. Each hook invocation reads this file and skips already-processed images.
Want to re-parse a specific image? Delete its line from that file. Want to re-parse everything? rm ~/.claude/image-cache/.vision-processed.
If your API gateway needs extra headers (multi-tenant routing, etc.), use VISION_EXTRA_HEADERS:
export VISION_EXTRA_HEADERS='{"X-Model-Provider-Id": "tongyi", "X-Request-Source": "cc-vision-hook"}'Currently serial. If you often paste multiple images at once, swap the loop in run_hook for a concurrent.futures.ThreadPoolExecutor — PRs welcome.
The same approach works in any AI terminal that supports a UserPromptSubmit-equivalent hook (Cursor, Cline, etc.). The script itself doesn't depend on Claude Code-specific APIs — any hook system that passes stdin JSON and reads stdout JSON works.
Q: After install, pasting an image still does nothing? A: Three things to check:
- Does
python3 ~/.claude/hooks/image-vision.py --test-image /path/to/img.pngwork? If not → API config is wrong. - Have you restarted Claude Code or opened
/hookssince installing? Config changes need a reload. - Check
~/.claude/image-cache/.vision-processed— does it contain the path of the image you want? If yes → delete that line and retry.
Q: How much does each image cost?
A: Depends on the model. gpt-4o ~$0.01-0.03 per typical screenshot; qwen-vl-max and glm-4v-plus are similar; gpt-4o-mini is ~10x cheaper with slightly worse detail recognition.
Q: Does it slow down the conversation? A: One image adds 1-3s of API latency. With no images, the hook returns near-instantly (early exit). Multiple images accumulate serially — switch to parallel if you need it.
Q: Will the main model think the image description is user input?
A: Yes, and that's the mechanism — the main model sees "the user described this image". The <image_vision> tag marks the source clearly; main models usually handle it correctly.
A 公众号 article walking through the problem and the build is in the works. Link will be added here once published.
PRs welcome. Ideas:
- Parallel multi-image parsing (
concurrent.futures) - More provider presets (Volcengine Doubao, Baidu Qianfan, etc.)
- Content-hash caching (skip re-parsing identical images across sessions)
- A demo GIF (
docs/demo.md) - Tests (the script is currently verified by the CI workflow via
--helpand exit-code checks; real unit tests would be even better)
Please keep the script dependency-free (stdlib only) so installation stays a single cp.
MIT