feat(msg): read Outlook messages

[vaibhavdabas16]

Addresses the .msg half of #128.

Scope, and why only half

@itsaaronngan already has .eml written and validated against 142 real messages, and offered it in this comment. I have not touched .eml — that work is theirs and it would be daft to duplicate it. Their comment scopes out the other half:

.msg, which is OLE-based and wants a separate frontend against cfb

That is this PR. The two are independent frontends and land in either order.

Approach

A .msg is not MIME. It is an OLE compound file whose MAPI properties each sit in their own stream, named __substg1.0_<tag><type> after the property id and its PT_ code ([MS-OXMSG]). anydoc already opens compound files with cfb for legacy .doc, so this is a property reader over machinery that is already here — no new dependency.

• Subject → heading
• Sender and addressees → a labelled envelope paragraph, matching the shape the .eml comment proposed (headers as a metadata paragraph, not a table — these are fields of one message, not tabular data)
• PR_BODY → prose, where a blank line separates paragraphs and a single newline is a hard break

Details worth flagging

Encoding. Strings arrive as UTF-16LE (001F) or in the code page the message names (001E, via PR_MESSAGE_CODEPAGE or PR_INTERNET_CPID). Both are read, and the Unicode stream wins where a producer writes both, being the one that cannot have lost characters.

Whitespace-only lines. Producers pad a "blank" line with a space rather than leaving it empty. Taking that for content ends the paragraph with a hard break onto nothing — the same bug @itsaaronngan hit in .eml and called out as the worst of the four they found. Handled here from the start.

Sender. Outlook sets the address into the display name when the sender resolves to none, which would otherwise render a@b <a@b>. Deduplicated.

Detection. Routes on the property store rather than a content stream, since a message has no equivalent of WordDocument or Workbook. Either __properties_version1.0 or a __substg1.0_ stream identifies one.

No plain-text body. A message whose only body is HTML (PR_HTML) or compressed RTF (PR_RTF_COMPRESSED) reports Unsupported rather than emitting markup as prose — the same call @itsaaronngan made, and for the same reason. Worth noting PR_RTF_COMPRESSED could later route into the existing RTF frontend once LZFu decompression exists; I did not want to fold that in here.

Recipient and attachment storages are not traversed. PR_DISPLAY_TO already carries the addressees as the sending client rendered them.

Output

# Q3 planning notes

**From:** Ada Lovelace \<[email protected]>\
**To:** Grace Hopper; Alan Turing\
**Cc:** [email protected]

Hi all,

Notes from today:\
\- budget signed off\
\- hiring on hold

Thanks,\
Ada

Tests

10 unit tests covering the document shape, bold envelope labels, absent and blank fields, the sender dedupe, hard breaks vs. paragraph breaks, whitespace-only lines, code-page decoding (1251 and the Windows-1252 fallback), Unicode winning over ANSI, the unsupported-body path, and a non-message compound file being rejected. Plus two detection tests and a fuzz target.

Fixtures are synthetic — built in-test through cfb, the way detect.rs builds its OLE fixtures — since I have no corpus of real messages I can commit. I would treat that as the main gap in this PR: it is validated against the specification and against messages I generated, not against a pile of real Outlook output.

cargo test --all-features (306 passed), cargo fmt --all --check, cargo clippy --workspace --all-targets --all-features -- -D warnings, and cargo clippy -p anydoc-wasm --target wasm32-unknown-unknown -- -D warnings all clean.

Registered in

Node, Python and wasm binding enums, node/cli.js FORMATS, node/index.d.ts, the Python type stub, the README format table, and fuzz/.

Note this adds a Format variant, which is a breaking change for exhaustive matches — #154 proposes #[non_exhaustive] for exactly that reason.

Open questions

1. Envelope as a metadata paragraph, or would you rather something else? I followed what was proposed on the issue so the two frontends agree.
2. Should Date be included? PR_CLIENT_SUBMIT_TIME is a FILETIME in __properties_version1.0 rather than a string stream, so it needs that stream's fixed-length section parsed. Straightforward to add — I left it out rather than guess at whether you want it.
3. Quoted reply chains are preserved verbatim, matching the .eml decision.

---

Summary by cubic

Adds .msg Outlook message support for #128: previously OLE-based messages were unsupported, and they now convert to Markdown through the existing cfb reader. The converter maps the subject to a heading, the sender and displayed recipients to labelled metadata, and PR_BODY to prose; messages with only HTML or compressed RTF remain unsupported.

New Features

• Detects messages from MAPI property streams, decodes UTF-16 and code-page strings, prefers Unicode, and does not traverse attachments or recipient storages.
• Registers msg in the Rust, Node.js, Python, WebAssembly, CLI, README, and fuzzing surfaces.
• Adds Format::Msg, so downstream Rust code with exhaustive matches must handle the new variant.

Bug Fixes

• Reads the code page from __properties_version1.0 instead of a string stream, so ANSI properties decode correctly.
• Treats NUL-terminated empty properties as absent, propagates fatal stream errors, and requires exact property-stream names in detection.
• Wraps fuzz input in a valid message container so mutations reach the decoders.

^{Written for commit d2a18fe. Summary will update on new commits.}

[cubic-dev-ai]

2 issues found across 13 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="fuzz/fuzz_targets/msg.rs">

<violation number="1" location="fuzz/fuzz_targets/msg.rs:8">
P2: With an empty corpus, this target feeds arbitrary bytes directly to `Format::Msg`, so the compound-file gate rejects nearly every input before MAPI decoding runs. Wrap mutations in a minimal valid MSG/OLE fixture and provide it as a seed, as the existing OLE fuzz targets do.</violation>
</file>

<file name="python/anydoc/_anydoc.pyi">

<violation number="1" location="python/anydoc/_anydoc.pyi:8">
P2: The public `anydoc.Format` alias still rejects `"msg"` even though this compiled-module stub accepts it. Add `"msg"` to `python/anydoc/__init__.py` so type checking matches the runtime-supported format.</violation>
</file>

<sub>Tip: instead of fixing issues one by one fix them all with cubic

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: With an empty corpus, this target feeds arbitrary bytes directly to Format::Msg, so the compound-file gate rejects nearly every input before MAPI decoding runs. Wrap mutations in a minimal valid MSG/OLE fixture and provide it as a seed, as the existing OLE fuzz targets do.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At fuzz/fuzz_targets/msg.rs, line 8:

<comment>With an empty corpus, this target feeds arbitrary bytes directly to `Format::Msg`, so the compound-file gate rejects nearly every input before MAPI decoding runs. Wrap mutations in a minimal valid MSG/OLE fixture and provide it as a seed, as the existing OLE fuzz targets do.</comment>

<file context>
@@ -0,0 +1,9 @@
+fuzz_target!(|data: &[u8]| {
+    // Conversion may fail with a typed error; it must never panic, hang,
+    // or exhaust memory.
+    let _ = anydoc::to_markdown_bytes(data, anydoc::Format::Msg);
+});
</file context>

[cubic-dev-ai]

P2: The public anydoc.Format alias still rejects "msg" even though this compiled-module stub accepts it. Add "msg" to python/anydoc/__init__.py so type checking matches the runtime-supported format.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At python/anydoc/_anydoc.pyi, line 8:

<comment>The public `anydoc.Format` alias still rejects `"msg"` even though this compiled-module stub accepts it. Add `"msg"` to `python/anydoc/__init__.py` so type checking matches the runtime-supported format.</comment>

<file context>
@@ -4,7 +4,8 @@ import os
 Format = Literal[
-    "doc", "docx", "odt", "pdf", "ppt", "pptx", "rtf", "epub", "xlsx", "ods", "odp", "csv"
+    "doc", "docx", "odt", "pdf", "ppt", "pptx", "rtf", "epub", "xlsx", "ods", "odp", "csv",
+    "msg"
 ]
 
</file context>

[vaibhavdabas16]

Worked through all six. Every one was valid — pushed in d2a18fe.

The code page one was the serious find. PR_MESSAGE_CODEPAGE and PR_INTERNET_CPID are PT_LONG, so they live in __properties_version1.0 and no producer writes them as string streams. My lookup could therefore never find one in a real file, and every PT_STRING8 property silently decoded as Windows-1252 regardless of what the message declared.

Worse, my own test wrote the code page as a string stream, so it passed against the bug it was supposed to catch. That is on me. The test harness now builds the property store the way [MS-OXMSG] 2.4.1.1 lays it out — 32-byte top-level header, then 16-byte entries of packed tag, flags and value — and there is a second test pinning that a string stream is not where the value lives, so this cannot regress back into a self-confirming shape. Added a truncation test across every length from 0 to one entry, since all of those offsets come from the file.

The rest:

• Fatal errors swallowed — read_property returned Option, so a ResourceLimit on a property stream was logged and skipped, and conversion carried on reporting partial content as if it were the message. It returns Result now and propagates anything is_fatal(), matching what optional_xml_part does in archive.rs.
• NUL-terminated empty properties — correct, and clean_text already drops control characters, so the fix was ordering: pick cleans before testing for blank rather than after. The characters it drops were exactly what made an "empty" value look present.
• Detection too loose — now requires entry.is_stream() and a name of exactly __substg1.0_ plus eight hex digits. Tests cover a bare prefix, a short tag, an over-long one, non-hex, a .bak suffix, and storages carrying both names.
• Fuzz target — right, and fuzz/README.md documents that convention explicitly, which I had read and then not followed. It now wraps input in a valid message the way xls does, filling a Unicode property, an ANSI one and the property store from the same bytes so one input reaches every decoding path. Two seeds in fuzz/seeds/msg/, and the README updated to list msg alongside the others.
• Python Format alias — a genuine miss. I updated _anydoc.pyi and never noticed the public alias in python/anydoc/__init__.py.

cargo test --all-features (311 passed), cargo fmt --all --check, cargo clippy --workspace --all-targets --all-features -- -D warnings, cargo clippy -p anydoc-wasm --target wasm32-unknown-unknown -- -D warnings, and cargo check on the fuzz crate are all clean. Markdown output for a normal message is unchanged.

Three unit tests added, two detection tests, one detection test extended for hex case — 16 in the module now.

[cubic-dev-ai]

All reported issues were addressed across 7 files (changes from recent commits).

<sub>Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic</sub>