Automatically OCR screenshots at ingest

[isomorphisms]

User story

People often take screenshots because selecting/copying text from the current interface is slower or impossible. Treat screenshots as clipboard-like input, not merely image files.

Invariant

Every screenshot accepted by IB gets an automatic OCR pass. No separate OCR command or user action is required.

• Preserve the original screenshot as canonical evidence.
• Store OCR text as a provenance-linked derived projection.
• Make recognized text searchable, selectable, copyable, quotable, and available to the text-first frontend.
• Preserve screenshots when OCR fails; record failed/partial extraction rather than dropping the input.
• Allow later OCR engines to replace or augment the derived projection without changing screenshot identity.
• Keep screenshot identity, provenance, indexing, and replacement rules in the browser core.
• Let Grease orchestrate the OCR executable/model so the engine remains replaceable.
• Native/front-end adapters may hand screenshots into the pipeline but must not own recognized text.

Acceptance fixtures

At minimum:

1. ordinary UI text;
2. mixed image + text;
3. Unicode text;
4. deliberately unreadable screenshot.

Readable fixtures should produce indexed OCR text automatically on import/share. The unreadable fixture should preserve the image and expose an inspectable extraction failure.

The corresponding acceptance story is being added to docs/personal-workbench.md.

[radelqui]

1. When a screenshot is taken, give it a UUID and write the raw PNG/JPEG to IndexedDB (or the browser’s file system) together with its provenance metadata.
2. Immediately spawn a Web‑Worker (or a Grease‑orchestrated native process) that loads an OCR engine such as Tesseract.js or calls your preferred executable.
3. Pass the image bytes to the worker; let the OCR return plain text, a confidence score and any error messages.
4. Write the OCR result to a separate record that references the original UUID, flagging it as successful, partial or failed.
5. Index the text field in your search index so it becomes searchable, selectable and copy‑able in the front‑end.
6. In the UI, overlay the recognized text on the image for selection, or expose an API that returns either the image or the derived text based on the caller’s need.
7. If OCR fails, keep the image untouched and store the failure flag; the front‑end can show “OCR unavailable” while still allowing the image to be viewed.
8. To upgrade the OCR engine later, swap the worker script or Grease‑registered executable, then re‑run the OCR pass on stored images – the UUID and provenance never change.
9. Test with the four fixtures you listed; the first three should appear in search results, the unreadable one should retain the image and show a failure record.

Full disclosure: I'm the developer of GreenIAshot, a Windows screenshot tool that pulls the text out of any image with OCR. It runs Tesseract locally on each capture and attaches the result as a searchable layer.
https://apps.microsoft.com/detail/9nbrzzzfcjbg?cid=gh-greeniashot