Bound POSIX SHM mapping by the shared object size

[yosuke-wolfssl]

Problem

The POSIX shared-memory transport built its mmap layout from header fields that live in the shared object itself, so a peer could set them freely. A local process that wins the race to create the named object (mode 0660, same uid/gid) could:

• Wrap the length. size = sizeof(header) + req_size + resp_size + dma_size had no overflow check and was never compared against the object's real size. The sum could wrap to a small value; Map then mapped that small region but derived resp/dma from the raw fields, placing them outside the mapping, and propagated a near-SIZE_MAX dma_size that defeated the client DMA bounds check.
• Overflow isn't even required — req_size = resp_size = 65535 on a 64-byte object gives a non-wrapping size past EOF; touching the last page is SIGBUS.
• Rewrite between validation and use. The header was read twice — validated from a probe mapping, then re-read from a fresh mapping to place the buffers — so a hostile peer could change the fields in between.

Addressed by f_4223.

Fix (port/posix/posix_transport_shm.c)

• fstat is the authoritative bound. posixTransportShm_Map derives the mapping length from the object size, not the header.
• Snapshot-then-validate. req_size/resp_size/dma_size are read once into locals, checked (sizeof(header) + req + resp <= st_size, dma_size <= remainder, overflow-free), and used only from those locals. HandleMap and ServerInit consume the validated map->req_size/map->resp_size instead of re-reading the header.
• Buffer sizing rule. Each buffer must be a non-zero multiple of sizeof(whTransportMemCsr). A buffer under one word underflowed the length check in wh_TransportMem_SendRequest (req_size is uint16_t, so req_size - sizeof(csr) wrapped near SIZE_MAX), leaving an out-of-bounds write reachable; a non-multiple misaligned the response/DMA pointers cast to volatile uint64_t.
• Server fails fast. CreateMap applies the same rule to the config before shm_open, so a misconfigured server returns WH_ERROR_BADARGS instead of leaving a stale object every client would reject; it also unlinks if the post-create map fails. Documented on posixTransportShmConfig in the header.

Tests (test/wh_test_comm.c)

New single-process negative tests under whTest_Comm. _whTest_CommShmMalformedHeader squats the name and presents 8 malformed layouts plus one exactly-fitting accept case; _whTest_CommShmServerBadConfig drives ServerInit with a bad size and asserts no object is left linked. Each clause of the predicate and both sides of the DMA bound have coverage.

Verification

• Build clean under -std=c90 -Werror -Wall -Wextra -Wpointer-arith.
• Legacy suite and test-refactor (37 passed, 0 failed) pass; POSIX example pair connects and runs its full crypto sequence.
• Mutation-checked: dropping any clause of the new validation, flipping the DMA bound >→>= or >→>+1, drifting the test's header mirror, or removing the server config guard each makes the suite fail — the guards are load-bearing, not decorative.

[Copilot]

Pull request overview

Note

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

This PR hardens the POSIX shared-memory transport against malicious or malformed SHM headers by bounding mappings to the actual SHM object size, snapshotting and validating peer-writable header fields once, and enforcing alignment/size rules for request/response buffers. It also adds regression tests that squat the SHM name and validate both client-side rejection and server-side fail-fast behavior.

Changes:

• Derive mmap length from fstat() object size and validate a single snapshot of header sizes before computing buffer offsets.
• Enforce “non-zero multiple of sizeof(whTransportMemCsr)” for request/response buffer sizes on both client and server creation paths.
• Add negative tests for malformed SHM headers and for server misconfiguration leaving no stale SHM objects behind.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File	Description
test/wh_test_comm.c	Adds negative SHM layout/config tests by squatting the SHM name and writing malformed headers.
port/posix/posix_transport_shm.h	Documents the new req/resp buffer sizing/alignment rule and expected error behavior.
port/posix/posix_transport_shm.c	Maps SHM based on fstat() size, snapshots + validates header fields once, and uses validated sizes throughout.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #479

Scan targets checked: wolfhsm-core-bugs, wolfhsm-src

No new issues found in the changed files. ✅

[rizlik]

The PR improves the POSIX SHM robustness.
What I see problematic is that it might hint that the POSIX SHM can provide a true isolation boundary between the client and the server.

Unfortunately there are plenty of other attack vectors for the client over POSIX SHM (both peers typically run with same UID, a malicious client can still ftruncate the file and SIGBUS the server, and an attacker can race to create a same-name object and interpose between client and server).

Moreover, I don't think POSIX SHM was ever meant to be a secure transport protocol but merely used for test, debug and as a reference.

I'm OK with the PR but this is the right moment to document the threat model explicitly in posix_transport_shm.h: same trust domain assumed; no peer authentication or isolation;

[rizlik]

NIT: superfluous

[rizlik]

superfluous