diff --git a/README.md b/README.md index 1ecf6cf..1f7098a 100644 --- a/README.md +++ b/README.md @@ -30,7 +30,8 @@ Default transport is `nbdssl` (`nbd` is still available): - `VixDiskLib_ConnectEx` (UID credentials; vCenter or direct ESXi) - `VixDiskLib_Open` (datastore path, read-only or read-write) -- `VixDiskLib_Read` (optional ``skip_decompression`` packs FastLZ extras) +- `VixDiskLib_Read` (optional ``skip_decompression`` packs compressed + extras as-is, whichever algorithm the disk was opened with) - `VixDiskLib_Write` - `VixDiskLib_GetInfo` (capacity and physical geometry from the `Open` reply; `biosGeo`/`adapterType`/`uuid` from `DDB_GET`, matching real @@ -40,14 +41,15 @@ Default transport is `nbdssl` (`nbd` is still available): - Changed Block Tracking: `openvixdisklib.nfc_auth.enable_change_tracking` / `disk_change_id` / `query_changed_disk_areas` (public VIM API, not part of VixDiskLib itself; see `docs/cbt.md`) +- NBD IO compression: zlib, FastLZ, and SkipZ (`VIXDISKLIB_FLAG_OPEN_COMPRESSION_{ZLIB,FASTLZ,SKIPZ}`; + see `docs/nfc_read.md`) Reading/writing a snapshot delta file directly (and running `query_allocated_blocks` against it) already works — `NFC_DELTA_DISK` turned out to be an optional VMFS-only VDDK client optimization, not a correctness requirement (see `docs/reverse_engineering_procedure.md`). -Not implemented: compression open flags other than FastLZ, and -encrypted disks. +Not implemented: encrypted disks. Requires Python 3.10 or later. diff --git a/docs/nfc_open.md b/docs/nfc_open.md index 16f7da5..fd42597 100644 --- a/docs/nfc_open.md +++ b/docs/nfc_open.md @@ -261,14 +261,14 @@ Reply payload (60 bytes), fields that matter: | Offset | Type | Meaning | | ------ | -------- | ------------------------------- | -| 8 | `uint64` | File handle (opaque, per open) | -| 16 | `uint32` | File type (`2` = `NFC_DISK`) | -| 20 | `uint32` | Flags echoed (`0x1e` or `0x1a`) | -| 28 | `uint64` | Disk capacity in **bytes** | -| 36 | `uint32` | Sector size (`512` on this VM) | -| 40 | `uint32` | Physical geometry cylinders | -| 44 | `uint32` | Physical geometry heads | -| 48 | `uint32` | Physical geometry sectors | +| 8 | `uint64` | File handle (opaque, per open) | +| 16 | `uint32` | File type (`2` = `NFC_DISK`) | +| 20 | `uint32` | Flags echoed (`0x1e` or `0x1a`) | +| 28 | `uint64` | Disk capacity in **bytes** | +| 36 | `uint32` | Sector size (`512` on this VM) | +| 40 | `uint32` | Physical geometry cylinders | +| 44 | `uint32` | Physical geometry heads | +| 48 | `uint32` | Physical geometry sectors | Later AIO messages pass that handle as a `uint64`. Offset 28 was found by capturing `VixDiskLib_GetInfo` (Step 14, @@ -319,7 +319,6 @@ I/O: `docs/nfc_read.md`, `docs/nfc_write.md`, and - zlib and skipz compression / encryption keys (`DDB_GET` is implemented for the plain, non-encrypted keys covered above) -- change-block tracking - Host-switch (`NFC_AIO_SWITCH_HOST_*`) Reading/writing a snapshot delta file directly, and running diff --git a/docs/nfc_read.md b/docs/nfc_read.md index e529636..4e186c6 100644 --- a/docs/nfc_read.md +++ b/docs/nfc_read.md @@ -45,11 +45,11 @@ Little-endian, after the usual 16-byte AIO header | Offset | Type | VDDK `Read(start, n)` | | ------ | -------- | ---------------------------------------------------------- | | 0 | `uint64` | File handle from `OPEN_FILE` | -| 8 | `uint64` | Direction in low 32 bits; FastLZ type `2` in high 32 bits | +| 8 | `uint64` | Direction in low 32 bits; compression type in high 32 bits | | 16 | `uint64` | Byte offset | | 24 | `uint64` | Byte length | | 32 | `uint32` | Byte length (same value) | -| 36 | `uint32` | Byte length, or compressed extra size when type is FastLZ | +| 36 | `uint32` | Byte length, or compressed extra size when compressed | | 40 | `uint32` | `0` | An earlier guess that offset 36 was `NFC_DISK` (`2`) was wrong: a @@ -57,25 +57,66 @@ An earlier guess that offset 36 was `NFC_DISK` (`2`) was wrong: a read that sent `(512, 2, 0)` still worked for one sector; OpenVixDiskLib now matches VDDK. -`VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ` does not change OPEN_FILE -flags. The IO opcode at offset 8 is a `uint64`: low 32 bits are still -`0`/`1` (write/read), high 32 bits are the NFC compression type -(`2` = FastLZ). OPEN still uses handshake `PlainText`. - -| Open flag / wire | Request extra | Reply extra | -| ---------------------------------------- | ------------------------------------- | ------------------------------------------------ | -| No compression flag | Raw `length` bytes on write | Raw fragment at offset 32 | -| FASTLZ, data that shrinks | FastLZ bytes; offset 36 = packed size | Opcode type `2`; extra is FastLZ of offset 32 | -| FASTLZ, incompressible | Raw bytes; opcode type `0` | Opcode type `0`; extra is raw | - -Reads with FASTLZ always *request* type `2`. The server may answer type -`2` or fall back to type `0`. Decompress into the uncompressed fragment -length at offset 32 and copy to the dest at offset 28. +`VIXDISKLIB_FLAG_OPEN_COMPRESSION_{ZLIB,FASTLZ,SKIPZ}` do not change +OPEN_FILE flags. The IO opcode at offset 8 is a `uint64`: low 32 bits +are still `0`/`1` (write/read), high 32 bits are the NFC compression +type actually used for *this fragment* — `0` none, `1` zlib, `2` +FastLZ, `3` SkipZ. At most one compression open flag may be set; the +server (or, for writes, OpenVixDiskLib itself) still falls back to +type `0` per fragment when compressing that fragment would not shrink +it. OPEN still uses handshake `PlainText`. + +| Open flag / wire | Request extra | Reply extra | +| ----------------------------------- | ---------------------------------------------- | ---------------------------------------------------- | +| No compression flag | Raw `length` bytes on write | Raw fragment at offset 32 | +| ZLIB, data that shrinks | Standard zlib stream; offset 36 = packed size | Opcode type `1`; extra is a zlib stream of offset 32 | +| FASTLZ, data that shrinks | FastLZ bytes; offset 36 = packed size | Opcode type `2`; extra is FastLZ of offset 32 | +| SKIPZ, at least one all-zero byte | Run-list format below; offset 36 = packed size | Opcode type `3`; extra is run-list of offset 32 | +| Any of the above, no size reduction | Raw bytes; opcode type `0` | Opcode type `0`; extra is raw | + +Reads with a compression flag set always *request* that type. The +server may answer with that type or fall back to type `0`. Decompress +into the uncompressed fragment length at offset 32 and copy to the +dest at offset 28. 64 KiB chunks use FastLZ level 2 (first byte has bit 5 set). Smaller chunks use level 1. VDDK’s URL form is `FASTLZ-vpxa-nfc://…`; authd `PROXY` is unchanged. +### ZLIB (type `1`) + +The extra is a standard zlib stream (`zlib.compress`/`zlib.decompress` +in Python; the 2-byte header starts `78 01` in this lab's captures) of +exactly the fragment's uncompressed bytes — no NFC-specific framing at +all beyond the usual opcode/offset-36-length fields shared with +FastLZ. This makes zlib the simplest of the three to implement. + +### SkipZ (type `3`) + +Captured from native VDDK writing/reading a fragment with real +zero-filled runs (an incompressible, all-non-zero fragment falls back +to type `0`, same as FastLZ/zlib). The extra is not general-purpose +compression — it only omits runs of zero bytes, keeping everything +else raw: + +``` +uint32 total_length (== this fragment's uncompressed length) +uint32 reserved (0 in every capture) +repeated, one per non-zero run, in ascending offset order: + uint32 run_offset (byte offset within this fragment) + uint32 run_length + +``` + +A fragment that is entirely zero encodes as just the 8-byte header +with no runs at all (verified against native VDDK: a write of all +zeros round-trips through a real ESXi session as an 8-byte SkipZ +extra). Run boundaries are **not required to be sector- or any other +kind of aligned** — verified by writing three non-zero runs at +arbitrary byte offsets (137, 900, 1990) with `openvixdisklib`'s own +byte-level scanner and reading the result back correctly with native +VDDK on the same fragment. + ## Reply Each fragment is: 16-byte AIO header (same `type` and `opId`) + 44-byte @@ -91,7 +132,7 @@ Reply payload (handle is zeroed; lengths describe this fragment): | 24 | `uint32` | Total request length | | 28 | `uint32` | Fragment byte offset **in this request** (`0`, `65536`, …), not disk | | 32 | `uint32` | This fragment’s uncompressed byte length | -| 36 | `uint32` | Same as offset 32, or compressed extra size when type is FastLZ | +| 36 | `uint32` | Same as offset 32, or compressed extra size when compressed | | 40 | `uint32` | `0` | When there is a single fragment, offsets 24–31 look like a `uint64` @@ -144,22 +185,26 @@ fragments). ## Skip decompression (OpenVixDiskLib extension) `VixDiskLib_Read` always fills `buf` with uncompressed sector bytes. -OpenVixDiskLib can skip FastLZ decode so a backup application can -forward the compressed data as-is, avoiding unnecessary re-compression. +OpenVixDiskLib can skip decoding (zlib, FastLZ, or SkipZ, whichever the +disk was opened with) so a backup application can forward the +compressed data as-is, avoiding unnecessary re-compression. `NfcDisk.readinto(..., skip_decompression=True)` and `VixDiskLibHandle.read(..., skip_decompression=True)` still send one IO request and wait until uncompressed `filled == length`. They do **not** decompress. Extras are packed densely from offset 0 of `buf`. -`ReadResult.fragments` describes each extra. Type `2` extras are -FastLZ; type `0` fallbacks are raw. Concatenating extras is not a -valid FastLZ stream; the caller must use the table to split them. +`ReadResult.fragments` describes each extra, tagged with its own +`compression_type` — a single read's fragments are not all guaranteed +to share one type, since any fragment that did not shrink still comes +back raw (type `0`) regardless of the open's compression flag. +Concatenating extras is not a valid compressed stream of any of the +three algorithms; the caller must use the table to split them. | Field | Meaning | | ---------------------- | ------------------------------------------------------------------------------------------------ | | `dest` | Byte offset **in this uncompressed read** (NFC payload 28). Not a disk LBA or VMDK file offset. | | `uncompressed_length` | Uncompressed fragment size (NFC payload 32). | -| `compression_type` | `NFC_COMPRESSION_NONE` (0) or `NFC_COMPRESSION_FASTLZ` (2). | +| `compression_type` | `NFC_COMPRESSION_NONE` (0), `_ZLIB` (1), `_FASTLZ` (2), or `_SKIPZ` (3), per fragment. | | `offset` | Start of this extra in packed `buf` (receive order, densely from 0). | | `length` | Extra size on the wire. | @@ -170,11 +215,11 @@ Disk byte address of a fragment is `start_sector * 512 + dest`. A `buf` is sized for the uncompressed request, so it is always large enough. Default `read` still decompresses; `fragments` is empty and `compressed_length` is still the extra bytes on the wire. -`skip_decompression` with a plain (no FASTLZ) open only records raw -extras (`compressed_length == uncompressed_length`). +`skip_decompression` with a plain (no compression) open only records +raw extras (`compressed_length == uncompressed_length`). This is not `VixDiskLib_Read`. Do not add an open flag for it; -compression on the wire is already the FASTLZ open flag. +compression on the wire is already the open's compression flag. A 32 MiB read at 64 KiB extras is 512 fragments in **one** result. A 2 MiB OPEN_SESSION extra (`aio_buffer_size=2097152`) is 16 fragments @@ -263,7 +308,7 @@ naturally do this anyway (open read-only after the writer's handle already closed), so it's unlikely to bite in practice, but do not call `query_allocated_blocks` right after a write on the same handle and expect it to reflect that write. + ## What is still VDDK-only -- zlib and skipz NBD compression flags - `VixDiskLib_ReadAsync` (same IO messages, different client threading) diff --git a/docs/nfc_write.md b/docs/nfc_write.md index a2c59d6..6501cde 100644 --- a/docs/nfc_write.md +++ b/docs/nfc_write.md @@ -60,7 +60,7 @@ Little-endian, after the usual 16-byte AIO header | 24 | `uint32` | Total byte length | | 28 | `uint32` | Byte offset of this fragment (`0`, `65536`, …) | | 32 | `uint32` | This fragment’s uncompressed length | -| 36 | `uint32` | Extra size (same as 32, or FastLZ packed size) | +| 36 | `uint32` | Extra size (same as 32, or the compressed packed size) | | 40 | `uint32` | `0` | This is the same 44-byte layout as a **read reply** fragment @@ -68,11 +68,13 @@ This is the same 44-byte layout as a **read reply** fragment reply fragments. A single-fragment write (≤ 64 KiB) still looks like a `uint64` length at offset 24 because the fragment offset is 0. -FASTLZ writes use the same header. The opcode `uint64` high half is -`2`, offset 36 is the compressed size, and FastLZ bytes follow instead -of raw sectors. If compression does not shrink the fragment, VDDK -sends type `0` and raw extra. Each fragment is compressed on its own; -a 32 MiB FastLZ write is 512 independent FastLZ extras, not one. +Compressed writes (zlib, FastLZ, or SkipZ) use the same header. The +opcode `uint64` high half is `1`/`2`/`3` respectively, offset 36 is the +compressed size, and compressed bytes follow instead of raw sectors. +If compression does not shrink the fragment, the client sends type `0` +and raw extra instead. Each fragment is compressed on its own; a +32 MiB compressed write is 512 independent compressed extras, not one. +Wire formats for all three algorithms: `docs/nfc_read.md`. Sector bytes follow the 44-byte payload and are **not** counted in AIO `size`. OpenVixDiskLib sends header + payload + extra in one diff --git a/docs/reverse_engineering_procedure.md b/docs/reverse_engineering_procedure.md index c57469c..224c16f 100644 --- a/docs/reverse_engineering_procedure.md +++ b/docs/reverse_engineering_procedure.md @@ -535,6 +535,196 @@ extent). Full workflow and lab evidence: `docs/cbt.md`. +## Step 13 — Direct ESXi (`ha-nfc`) without vCenter + +Same SSL-hook technique (Step 4), this time pointing VDDK 8.0.3 +straight at a standalone ESXi 8.0.3 host (`vmxSpec=moref=`, +`serverName=`, no vCenter in the topology). Confirmed +OpenVixDiskLib's hardcoded `NFC_SERVICE_MOID = "nfcService"` fails on +this host with `vmodl.fault.ManagedObjectNotFound` *before* touching +the capture — reproduced with plain `openvixdisklib` calls, no hook +needed to see that failure. + +The capture showed VDDK does not hardcode the moref either: it calls +an undocumented `RetrieveInternalContent` on the `ServiceInstance` +moref first, and reads `nfcService` from the reply (`ha-nfc-service` +on this host, vs. `nfcService` in the earlier vCenter capture). +`NfcGetVmFilesResponse.service` was `nfc` (not `vpxa-nfc`), and its +`host` field was **absent** — the authd endpoint is implicitly the +same host already logged into. Full detail in `docs/nfc_auth.md` +("Direct ESXi (no vCenter)"). + +Fix: `_nfc_service_moid()` in `openvixdisklib/nfc_auth.py` issues the +`RetrieveInternalContent` call as raw SOAP over the existing stub +connection (its response schema has ~20 other undocumented morefs not +worth registering with pyVmomi's type system for one field), and +`connect_authd()` takes a `fallback_host` used when `ticket.host` is +unset. Validated end-to-end (`ConnectEx`/`Open`/`Read`, `nbd` and +`nbdssl`) against the live host. + +## Step 14 — `VixDiskLib_GetInfo` capacity + +Extended the ctypes probe from Step 13 to call `VixDiskLib_GetInfo` +after `Open`, under the SSL hook plus a `write`/`read` interceptor on +fd 902 (Step 7), to see what wire traffic `GetInfo` adds. + +Result: **no new SOAP or authd traffic** — the same `RetrieveContent` ++ `Login` + `NfcGetVmFiles` + authd sequence as a plain `Open`. All the +extra traffic is inside the already-open NFC/AIO session: ~20 more +`DDB_GET` (type 11) requests right after `OPEN_FILE`, for keys like +`adapterType`, `uuid`, `geometry.cylinders`, `geometry.biosCylinders`, +etc. (full list in `docs/nfc_open.md`). + +Dumping every byte of the `OPEN_FILE` reply (not just the fields the +earlier Open-only capture had labeled) found `capacity` (offset 28, +`uint64` bytes) and `physGeo` (offsets 40/44/48) already present — +verified they match `VixDiskLibInfo.capacity`/`physGeo` from the same +`GetInfo` call exactly. Only `biosGeo`, `adapterType`, and `uuid` are +genuinely `DDB_GET`-only; `biosGeo` came back "key not found" (zeros) +on this unencrypted lab disk. + +Fix: extended `_parse_open_reply` in `openvixdisklib/nfc_open.py` to +also read those offsets, added `nfc_open.DiskInfo`/`DiskGeometry`, and +exposed `VixDiskLibHandle.get_info()`. No new NFC message type was +needed — `DDB_GET` (`adapterType`/`uuid`/`biosGeo`) is still open work. +Validated against the live host: `capacity_sectors=33554432` +(16 GiB), `phys_geo=(2088, 255, 63)`, matching native VDDK's +`GetInfo` on the same disk. + +## Note — CBT needed no reverse engineering + +Investigated change-block tracking (backlog item "CBT / +`QueryAllocatedBlocks`") expecting an NFC capture like the steps above. +It turned out `VirtualMachine.QueryChangedDiskAreas` — the actual +changed-byte-range query backup tools use — is public pyVmomi API with +no VDDK/NFC involvement at all; only `VixDiskLib_QueryAllocatedBlocks` +(disk-internal allocated-block bitmap, a different and lesser feature) +needed NFC work (done separately, see Step 15 below). Implemented as +`openvixdisklib.nfc_auth.enable_change_tracking` / +`disk_change_id` / `query_changed_disk_areas`; validated end-to-end +against a temp VM on the lab (enable CBT, snapshot, write a known +sector, snapshot, query — the written sector fell inside the reported +extent). Full workflow and lab evidence: `docs/cbt.md`. + +## Step 15 — `VixDiskLib_QueryAllocatedBlocks` (AIO type 13) + +Same SSL/write-hook technique, extended to `VixDiskLib_QueryAllocatedBlocks` +after `Open`. A first capture with `startSector=0` produced a request +where two 0-valued 8-byte fields were ambiguous — either could plausibly +be `start_offset_bytes`. Implementing from that guess alone put +`start_offset_bytes` at the wrong offset (8 instead of 24) and passed +every zero-start test while silently returning wrong (all-empty) +results for any non-zero start. A second capture with a **non-zero** +`startSector` against a region already known (from the first capture) +to be allocated broke the tie and found the real field order. + +A second, independent bug (bitmap reply length) was caught the same +way: `ceil(chunk_count/8)` matched the observed reply length for +`chunk_count=16384` (already a multiple of 4) but under-read and +desynced the connection for `chunk_count=16` — the reply pads the +bitmap to a 4-byte boundary. Diagnosed by testing candidate `extra_recv` +byte counts against whether the *next* AIO round-trip (a normal +`CLOSE_FILE`) completed cleanly, rather than guessing from a single +capture. + +Full protocol detail, the exact request/reply layout, and both bugs: +`docs/nfc_read.md`. Implemented as +`openvixdisklib.nfc_open.NfcDisk.query_allocated_blocks` +(`AllocatedBlock` dataclass) and +`VixDiskLibHandle.query_allocated_blocks`. Validated against the live +ESXi lab: full-disk query, a known-allocated sub-range, and an aligned +empty range all match native VDDK's own output on the same disk. + +## Step 16 — `DDB_GET` (AIO type 11) + +Already partly captured as a side effect of Step 14 (`VixDiskLib_GetInfo` +triggers ~28 `DDB_GET` calls); no new capture was needed, just decoding +the request/reply pairs from that saved log by matching `opId` across +both directions. Confirmed the request's first 8 bytes equal the +`OPEN_FILE` handle from the same capture, and that a "found" reply's +extra is plain ASCII text (`b"lsilogic"`, `b"2088"`, ...), not binary — +matching how a VMDK descriptor's DDB section stores key/value pairs as +text. No padding on the reply extra (unlike Step 15's bitmap), +confirmed by decoding all 28 request/reply pairs from one capture in +sequence without desync. + +Implemented as `NfcDisk.ddb_get(key) -> str | None` and +`NfcDisk.query_full_info() -> DiskInfo` (the 5 keys needed for +`bios_geo`/`adapter_type`/`uuid`), wired into +`VixDiskLibHandle.get_info` in place of the OPEN_FILE-only version +from Step 14 — `get_info` now matches real VDDK's `VixDiskLib_GetInfo` +completely, including paying the same round-trip cost. Full layout: +`docs/nfc_open.md`. Validated against the live ESXi lab: matches +native VDDK's `GetInfo` output on the same disk exactly. + +## Note — `NFC_DELTA_DISK` needed no protocol work either + +Investigated the backlog item "`NFC_DELTA_DISK` (reading directly +from a snapshot chain)" expecting a distinct wire message or OPEN_FILE +variant, similar to the CBT/`QueryAllocatedBlocks` split earlier. +`strings` on `libvixDiskLib.so` found `NFC_DELTA_DISK` is a **file-type +value** (like `NFC_DISK`), used by an internal VDDK client-side +heuristic ("`"%s" would probably benefit from bitmap copying, so +overriding file type to NFC_DELTA_DISK`") — a VMFS-only optimization +for very sparse redo logs, skipped entirely on NFS per an adjacent +string, and never observed to trigger in this lab's captures (no such +log line, `strings`-confirmed heuristic notwithstanding). + +Verified end-to-end that reading, writing, and `query_allocated_blocks` +against an actual post-snapshot delta file all already work correctly +with the existing NFC_DISK-only implementation — no code change +needed. The one real finding from this investigation was a gotcha, not +a gap: an initial test that wrote a sector and immediately queried +allocated blocks *on the same still-open write handle* reported the +write as unallocated; closing the handle first (or opening a separate +one) reported it correctly. Reproduced identically against **native +VDDK** on the same delta file (two-process capture, since loading +native VDDK in the same process as pyVmomi segfaults on this host's +OpenSSL — see `docs/ssl_hook.md`'s Limits section), so this is real +server/VMFS behavior, not specific to either client. Documented as a +`query_allocated_blocks` caveat in `docs/nfc_read.md`. + +## Step 17 — ZLIB and SkipZ NBD compression + +Same SSL-hook technique as FASTLZ (Step 12), pointed at +`VIXDISKLIB_FLAG_OPEN_COMPRESSION_ZLIB` (`1 << 4`) and +`_SKIPZ` (`1 << 6`) in turn: + +- Handshake, `OPEN_FILE` flags, and the IO opcode's compression-type + placement (high 32 bits, direction in the low 32) are identical to + FASTLZ — only the type value differs: `1` zlib, `3` SkipZ (`2` stays + FastLZ). Same offset-36 compressed-size field, same per-fragment + type-`0` fallback when compression doesn't shrink a fragment. +- ZLIB's extra is a plain, standard zlib stream — decodable with + Python's stdlib `zlib` module directly, no custom framing at all. + The easiest of the three to implement. +- SkipZ initially looked identical to "no compression" — the first + capture used an all-non-zero pattern, which (correctly) fell back to + type `0` on both write and read, same as FASTLZ/zlib would for + incompressible data. Only after switching to a pattern with real + zero-filled runs did SkipZ actually engage (type `3`) and reveal its + real format: an 8-byte header (`total_length`, reserved) followed by + `(offset, length, )` triples, one per non-zero run — + zero runs are omitted entirely rather than compressed. A second + capture with two separate non-zero runs (different sizes, at + non-adjacent offsets) was needed to confirm the per-run header + repeats *interleaved* with each run's data (header, data, header, + data, …), not as a separate table of headers followed by all the + data. Full layout: `docs/nfc_read.md`. + +Implemented as `_skipz_compress`/`_skipz_decompress` in +`openvixdisklib/nfc_open.py` (no external dependency, unlike FastLZ's +`pyfastlz`) and wired into `NfcDisk.write`/`readinto` alongside the +existing FASTLZ branch; `_nfc_compression` in `openvixdisklib.py` maps +the two new `VIXDISKLIB_FLAG_OPEN_COMPRESSION_*` flags through. +Validated against the live ESXi lab: round-tripped all three +algorithms through `openvixdisklib` itself, then specifically +cross-checked SkipZ's byte-level (non-sector-aligned) run encoding by +writing with `openvixdisklib` and reading back with **native VDDK** on +the same fragment — confirming the server accepts arbitrary run +boundaries, not just the sector-aligned ones the captures happened to +use. + ## What to write down After a stage works: @@ -555,5 +745,5 @@ OpenVixDiskLib. Not yet reversed, same loop as above: -- zlib/skipz compression, encrypted disks +- encrypted disks - Host-switch AIO messages diff --git a/openvixdisklib/nfc_open.py b/openvixdisklib/nfc_open.py index e567e7a..7ffe9e9 100644 --- a/openvixdisklib/nfc_open.py +++ b/openvixdisklib/nfc_open.py @@ -25,7 +25,9 @@ import socket import ssl import struct +import zlib from dataclasses import dataclass +from typing import Protocol from openvixdisklib import fastlz from openvixdisklib.nfc_auth import NfcAuthSession, _ssl_client_context @@ -83,9 +85,51 @@ # High 32 bits of the IO opcode uint64. Captured from VDDK FASTLZ: # writes that shrink go on the wire as type 2; incompressible writes -# fall back to type 0 with raw extra data. +# fall back to type 0 with raw extra data. ZLIB (1) and SKIPZ (3) +# captured the same way, see docs/nfc_write.md. NFC_COMPRESSION_NONE = 0 +NFC_COMPRESSION_ZLIB = 1 NFC_COMPRESSION_FASTLZ = 2 +NFC_COMPRESSION_SKIPZ = 3 + + +@dataclass(frozen=True, slots=True) +class DiskGeometry: + """CHS geometry, matching VDDK's ``VixDiskLibGeometry``.""" + + cylinders: int + heads: int + sectors: int + + +@dataclass(frozen=True, slots=True) +class AllocatedBlock: + """One allocated run, matching VDDK's ``VixDiskLibBlock`` (sectors).""" + + offset: int + length: int + + +@dataclass(frozen=True, slots=True) +class DiskInfo: + """Matches VDDK's ``VixDiskLibInfo``. + + ``phys_geo`` and ``capacity_sectors`` are read directly off + OPEN_FILE (offsets 40/44/48 and 28 respectively) — free, no extra + NFC round trip. ``bios_geo``, ``adapter_type``, and ``uuid`` come + from ``DDB_GET`` (see ``NfcDisk.ddb_get`` / ``query_full_info``, + ``docs/nfc_open.md``): each is a real round trip, matching what + real VDDK's ``VixDiskLib_GetInfo`` does. ``bios_geo`` defaults to + all zeros and ``adapter_type``/``uuid`` to ``None`` when the disk + has no snapshots or predates that DDB key (VDDK does the same for + a missing key). + """ + + capacity_sectors: int + phys_geo: DiskGeometry + bios_geo: DiskGeometry = DiskGeometry(cylinders=0, heads=0, sectors=0) + adapter_type: str | None = None + uuid: str | None = None @dataclass(frozen=True, slots=True) @@ -215,18 +259,31 @@ def wrap_nfcssl_socket(ssock: ssl.SSLSocket, server_hostname: str) -> ssl.SSLSoc raise +class NfcTransport(Protocol): + """Byte pipe used after the NFC handshake (TCP, TLS, or a test fake).""" + + def sendall(self, data: bytes) -> None: + """Send ``data`` in full.""" + + def recv_into(self, buffer: memoryview, nbytes: int = 0, flags: int = 0) -> int: + """Read into ``buffer`` and return the number of bytes stored.""" + + def close(self) -> None: + """Close the underlying connection.""" + + def _enable_tcp_nodelay(sock: socket.socket) -> None: """Disable Nagle so a small AIO header is not held back from its extra.""" sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) -def _recvn(sock: socket.socket, size: int) -> bytes: +def _recvn(sock: NfcTransport, size: int) -> bytes: buf = bytearray(size) _recvn_into(sock, memoryview(buf)) return bytes(buf) -def _recvn_into(sock: socket.socket, buf: memoryview) -> None: +def _recvn_into(sock: NfcTransport, buf: memoryview) -> None: """Read exactly ``len(buf)`` bytes into ``buf``.""" view = buf.cast("B") if buf.format != "B" else buf filled = 0 @@ -253,11 +310,11 @@ def _writable_bytes(buf: bytearray | memoryview, length: int) -> memoryview: def _aio_extra_len(ctype: int, body: bytes, chunk_len: int) -> int: """Return this fragment's extra size on the wire.""" - if ctype == NFC_COMPRESSION_FASTLZ: + if ctype in (NFC_COMPRESSION_ZLIB, NFC_COMPRESSION_FASTLZ, NFC_COMPRESSION_SKIPZ): extra_len = struct.unpack_from(" int: raise NfcProtocolError(f"unsupported NFC IO compression type {ctype}") -def _send_nfc_msg(sock: socket.socket, msg_type: int, body: bytes = b"") -> None: +def _skipz_compress(chunk: bytes) -> bytes: + """Encode ``chunk`` as SKIPZ: header plus only its non-zero runs. + + Wire format (captured from native VDDK, see ``docs/nfc_write.md``):: + + uint32 total_length (== len(chunk)) + uint32 reserved (0) + repeated, one per non-zero run, in ascending offset order: + uint32 run_offset + uint32 run_length + + + Zero-length ``chunk`` (impossible in practice; NFC writes are never + empty) would encode as just the 8-byte header with no runs. + """ + runs = [] + i = 0 + n = len(chunk) + while i < n: + if chunk[i] == 0: + i += 1 + continue + start = i + while i < n and chunk[i] != 0: + i += 1 + runs.append((start, i - start)) + out = bytearray(struct.pack(" bytes: + """Decode a SKIPZ fragment back into ``chunk_len`` bytes.""" + if len(extra) < 8: + raise ValueError(f"SKIPZ extra too short: {len(extra)} bytes") + total_length = struct.unpack_from(" len(extra): + raise ValueError(f"SKIPZ run header truncated at offset {pos}") + offset, length = struct.unpack_from(" chunk_len or pos + length > len(extra): + raise ValueError( + f"SKIPZ run offset={offset} length={length} out of bounds " + f"(chunk={chunk_len}, extra={len(extra)})" + ) + out[offset : offset + length] = extra[pos : pos + length] + pos += length + return bytes(out) + + +def _send_nfc_msg(sock: NfcTransport, msg_type: int, body: bytes = b"") -> None: if len(body) > NFC_MSG_SIZE - 4: raise ValueError("NFC classic message body too large") frame = struct.pack(" bytes: byte units. If the length exceeds the session AIO buffer the server replies with several same-``opId`` fragments, which are placed by the fragment byte offset in the reply (they may arrive - out of order). FASTLZ open requests compression in the opcode; - each reply fragment may be compressed (type 2) or raw (type 0). + out of order). A compressed open requests compression in the + opcode; each reply fragment may come back compressed (its own + type: 1 zlib, 2 FastLZ, 3 SkipZ) or raw (type 0) if that + fragment did not shrink. Args: start_sector: Sector offset from the start of the disk. @@ -485,17 +604,27 @@ def readinto( ) ) packed_offset = end - elif ctype == NFC_COMPRESSION_FASTLZ: + elif ctype in ( + NFC_COMPRESSION_FASTLZ, + NFC_COMPRESSION_ZLIB, + NFC_COMPRESSION_SKIPZ, + ): extra = _recvn(self._sock, extra_len) try: - chunk = fastlz.decompress(extra, chunk_len) + if ctype == NFC_COMPRESSION_FASTLZ: + chunk = fastlz.decompress(extra, chunk_len) + elif ctype == NFC_COMPRESSION_ZLIB: + chunk = zlib.decompress(extra) + else: + chunk = _skipz_decompress(extra, chunk_len) except ValueError as exc: raise NfcProtocolError( - f"FastLZ read fragment failed: {exc}" + f"compressed read fragment (type {ctype}) failed: {exc}" ) from exc if len(chunk) != chunk_len: raise NfcProtocolError( - f"FastLZ read got {len(chunk)} bytes, expected {chunk_len}" + f"compressed read (type {ctype}) got {len(chunk)} bytes, " + f"expected {chunk_len}" ) data[dest : dest + chunk_len] = chunk elif ctype == NFC_COMPRESSION_NONE: @@ -516,7 +645,9 @@ def write(self, start_sector: int, num_sectors: int, data: bytes) -> None: Matches ``VixDiskLib_Write``: one ``NFC_AIO_MSG_IO`` ``opId`` for the whole call. Chunks larger than the session AIO buffer are extra fragments with that same ``opId``; the server replies - once. FASTLZ open compresses each fragment when that shrinks it. + once. A compressed open (zlib, FastLZ, or SkipZ) compresses + each fragment when that shrinks it, falling back to raw (type + 0) otherwise. Args: start_sector: Sector offset from the start of the disk. @@ -542,6 +673,18 @@ def write(self, start_sector: int, num_sectors: int, data: bytes) -> None: extra = compressed extra_len = len(compressed) ctype = NFC_COMPRESSION_FASTLZ + elif self.compression == NFC_COMPRESSION_ZLIB: + compressed = zlib.compress(chunk) + if len(compressed) < extra_len: + extra = compressed + extra_len = len(compressed) + ctype = NFC_COMPRESSION_ZLIB + elif self.compression == NFC_COMPRESSION_SKIPZ: + compressed = _skipz_compress(chunk) + if len(compressed) < extra_len: + extra = compressed + extra_len = len(compressed) + ctype = NFC_COMPRESSION_SKIPZ opcode = NFC_AIO_IO_WRITE | (ctype << 32) payload = struct.pack( " int: ) if alg == 0: return nfc_open.NFC_COMPRESSION_NONE + if alg == VIXDISKLIB_FLAG_OPEN_COMPRESSION_ZLIB: + return nfc_open.NFC_COMPRESSION_ZLIB if alg == VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ: return nfc_open.NFC_COMPRESSION_FASTLZ + if alg == VIXDISKLIB_FLAG_OPEN_COMPRESSION_SKIPZ: + return nfc_open.NFC_COMPRESSION_SKIPZ if alg & (alg - 1): raise NotImplementedError( "Cannot set two or more NBD compression algorithms at the same time" @@ -286,8 +290,9 @@ def open( disk_path: Datastore path of the VMDK. flags: Open flags. ``VIXDISKLIB_FLAG_OPEN_READ_ONLY`` opens the disk read-only; omit it for write. - ``VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ`` compresses - NFC IO. zlib and skipz are not implemented. + ``VIXDISKLIB_FLAG_OPEN_COMPRESSION_ZLIB``, + ``_FASTLZ``, and ``_SKIPZ`` each compress NFC IO with + their respective algorithm; at most one may be set. aio_buffer_size: NFC AIO extra size in bytes, advertised in OPEN_SESSION. Default 64 KiB. ESXi 8 accepts 2 MiB (``2097152``) and rejects 16 MiB and 32 MiB. This is an diff --git a/tests/integration/base.py b/tests/integration/base.py index 0e3d382..2ff8575 100644 --- a/tests/integration/base.py +++ b/tests/integration/base.py @@ -103,6 +103,29 @@ def pattern_bytes(length: int, seed: bytes) -> bytes: return (seed * ((length // len(seed)) + 1))[:length] +def sparse_bytes(length: int, seed: bytes) -> bytes: + """Return ``length`` bytes of ``seed`` runs separated by zero gaps. + + SkipZ only shrinks when a fragment contains zeros; an all-nonzero + pattern falls back to NFC type 0. + """ + if not seed: + raise ValueError("seed must be non-empty") + if length < 1: + return b"" + run = min(len(seed), length) + # ESXi SkipZ-compresses reads when zero runs are long (a sector or + # more). Tight 11-byte gaps stay type 0 on the wire. + gap = max(run * 8, SECTOR_SIZE) + out = bytearray(length) + pos = 0 + while pos < length: + chunk = (seed * ((run // len(seed)) + 1))[: min(run, length - pos)] + out[pos : pos + len(chunk)] = chunk + pos += len(chunk) + gap + return bytes(out) + + def ensure_vddk_library_path() -> None: """Prepend ``.vddk`` to ``LD_LIBRARY_PATH`` if it is not already there.""" current = os.environ.get("LD_LIBRARY_PATH", "") diff --git a/tests/integration/test_crosscheck.py b/tests/integration/test_crosscheck.py index 9942699..68f7904 100644 --- a/tests/integration/test_crosscheck.py +++ b/tests/integration/test_crosscheck.py @@ -131,8 +131,13 @@ def _assert_both_read_bytes( class TestCrosscheck: @pytest.mark.parametrize( "open_flags", - [0, vixdisklib.VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ], - ids=["plain", "fastlz"], + [ + 0, + vixdisklib.VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ, + vixdisklib.VIXDISKLIB_FLAG_OPEN_COMPRESSION_ZLIB, + vixdisklib.VIXDISKLIB_FLAG_OPEN_COMPRESSION_SKIPZ, + ], + ids=["plain", "fastlz", "zlib", "skipz"], ) def test_openvixdisklib_matches_vddk_sectors( self, lab: LabEnv, vddk: None, open_flags: int @@ -157,6 +162,25 @@ def test_openvixdisklib_matches_vddk_sectors( _write_sectors(lab, open_vix, ovdl_payloads, flags=open_flags) _assert_both_read(lab, sectors, ovdl_payloads, flags=open_flags) + def test_skipz_sparse_payload_matches_vddk(self, lab: LabEnv, vddk: None) -> None: + """SkipZ zero-run extras written by either library must read back on both.""" + flags = vixdisklib.VIXDISKLIB_FLAG_OPEN_COMPRESSION_SKIPZ + start_byte = 8888 * SECTOR_SIZE + n_bytes = 4 * SECTOR_SIZE + vddk_payload = bytearray(n_bytes) + vddk_payload[137:157] = b"V" * 20 + vddk_payload[900:950] = b"W" * 50 + vddk_payload[1990:2000] = b"X" * 10 + _write_bytes(lab, vixdisklib, start_byte, bytes(vddk_payload), flags=flags) + _assert_both_read_bytes(lab, start_byte, bytes(vddk_payload), flags=flags) + + ovdl_payload = bytearray(n_bytes) + ovdl_payload[137:157] = b"O" * 20 + ovdl_payload[900:950] = b"P" * 50 + ovdl_payload[1990:2000] = b"Q" * 10 + _write_bytes(lab, open_vix, start_byte, bytes(ovdl_payload), flags=flags) + _assert_both_read_bytes(lab, start_byte, bytes(ovdl_payload), flags=flags) + @pytest.mark.slow @pytest.mark.parametrize( "open_flags", diff --git a/tests/integration/test_nfc_open.py b/tests/integration/test_nfc_open.py index 08fe813..5b138b3 100644 --- a/tests/integration/test_nfc_open.py +++ b/tests/integration/test_nfc_open.py @@ -13,8 +13,13 @@ class TestNfcOpen: @pytest.mark.parametrize("nfc_ssl", [True, False], ids=["nbdssl", "nbd"]) @pytest.mark.parametrize( "compression", - [nfc_open.NFC_COMPRESSION_NONE, nfc_open.NFC_COMPRESSION_FASTLZ], - ids=["plain", "fastlz"], + [ + nfc_open.NFC_COMPRESSION_NONE, + nfc_open.NFC_COMPRESSION_FASTLZ, + nfc_open.NFC_COMPRESSION_ZLIB, + nfc_open.NFC_COMPRESSION_SKIPZ, + ], + ids=["plain", "fastlz", "zlib", "skipz"], ) def test_open_disk_and_read_first_sector( self, lab: LabEnv, nfc_ssl: bool, compression: int diff --git a/tests/integration/test_nfc_read_write.py b/tests/integration/test_nfc_read_write.py index b435eb3..d20c3ec 100644 --- a/tests/integration/test_nfc_read_write.py +++ b/tests/integration/test_nfc_read_write.py @@ -19,8 +19,13 @@ class TestNfcReadWrite: @pytest.mark.parametrize( "compression", - [nfc_open.NFC_COMPRESSION_NONE, nfc_open.NFC_COMPRESSION_FASTLZ], - ids=["plain", "fastlz"], + [ + nfc_open.NFC_COMPRESSION_NONE, + nfc_open.NFC_COMPRESSION_FASTLZ, + nfc_open.NFC_COMPRESSION_ZLIB, + nfc_open.NFC_COMPRESSION_SKIPZ, + ], + ids=["plain", "fastlz", "zlib", "skipz"], ) def test_sector_writes_and_reads(self, lab: LabEnv, compression: int) -> None: """Write known patterns and read them back at several ranges.""" @@ -68,6 +73,51 @@ def test_sector_writes_and_reads(self, lab: LabEnv, compression: int) -> None: == big_to_write[SECTOR_SIZE : 2 * SECTOR_SIZE] ) + def test_skipz_round_trips_zero_runs(self, lab: LabEnv) -> None: + """SkipZ encodes non-zero runs only; unaligned zero gaps must survive. + + Offsets match the live VDDK capture in ``docs/nfc_read.md`` (137, + 900, 1990). An all-nonzero pattern would fall back to type 0. + """ + n_sectors = 4 + to_write = bytearray(n_sectors * SECTOR_SIZE) + to_write[137:157] = b"A" * 20 + to_write[900:950] = b"B" * 50 + to_write[1990:2000] = b"C" * 10 + expected = bytes(to_write) + with ( + lab.authenticate(read_only=False) as session, + nfc_open.open_disk( + session, + lab.disk_path, + read_only=False, + compression=nfc_open.NFC_COMPRESSION_SKIPZ, + ) as disk, + ): + disk.write(16, n_sectors, expected) + got = disk.read(16, n_sectors) + assert got == expected + skip_buf = bytearray(len(expected)) + result = disk.readinto(16, n_sectors, skip_buf, skip_decompression=True) + assert result.fragments + assert any( + frag.compression_type == nfc_open.NFC_COMPRESSION_SKIPZ + for frag in result.fragments + ) + rebuilt = bytearray(result.uncompressed_length) + for frag in result.fragments: + extra = bytes(skip_buf[frag.offset : frag.offset + frag.length]) + if frag.compression_type == nfc_open.NFC_COMPRESSION_SKIPZ: + chunk = nfc_open._skipz_decompress(extra, frag.uncompressed_length) + elif frag.compression_type == nfc_open.NFC_COMPRESSION_NONE: + chunk = extra + else: + raise AssertionError( + f"unexpected compression_type {frag.compression_type}" + ) + rebuilt[frag.dest : frag.dest + frag.uncompressed_length] = chunk + assert bytes(rebuilt) == expected + @pytest.mark.parametrize( "compression", [nfc_open.NFC_COMPRESSION_NONE, nfc_open.NFC_COMPRESSION_FASTLZ], diff --git a/tests/integration/test_openvixdisklib.py b/tests/integration/test_openvixdisklib.py index 6731a26..5737677 100644 --- a/tests/integration/test_openvixdisklib.py +++ b/tests/integration/test_openvixdisklib.py @@ -3,6 +3,7 @@ """Exercise the VDDK-compatible openvixdisklib handle against the lab.""" +import zlib from typing import Any import pytest @@ -20,6 +21,7 @@ _connect_vim, _wait_for_task, pattern_bytes, + sparse_bytes, ) @@ -46,6 +48,10 @@ def _rebuild_skip(buf: Any, result: ReadResult) -> bytes: packed += frag.length if frag.compression_type == nfc_open.NFC_COMPRESSION_FASTLZ: chunk = fastlz.decompress(extra, frag.uncompressed_length) + elif frag.compression_type == nfc_open.NFC_COMPRESSION_ZLIB: + chunk = zlib.decompress(extra) + elif frag.compression_type == nfc_open.NFC_COMPRESSION_SKIPZ: + chunk = nfc_open._skipz_decompress(extra, frag.uncompressed_length) elif frag.compression_type == nfc_open.NFC_COMPRESSION_NONE: chunk = extra else: @@ -60,8 +66,13 @@ class TestOpenvixdisklib: @pytest.mark.parametrize("transport_mode", ["nbdssl", "nbd"]) @pytest.mark.parametrize( "open_flags", - [0, vixdisklib.VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ], - ids=["plain", "fastlz"], + [ + 0, + vixdisklib.VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ, + vixdisklib.VIXDISKLIB_FLAG_OPEN_COMPRESSION_ZLIB, + vixdisklib.VIXDISKLIB_FLAG_OPEN_COMPRESSION_SKIPZ, + ], + ids=["plain", "fastlz", "zlib", "skipz"], ) def test_write_and_read_sector_zero_and_one_gib( self, lab: LabEnv, transport_mode: str, open_flags: int @@ -243,7 +254,12 @@ def test_write_and_query_allocated_blocks_on_delta_disk(self, lab: LabEnv) -> No read_buf = vixdisklib.get_buffer(SECTOR_SIZE) si = _connect_vim( - lab.host, lab.username, lab.password, lab.port, lab.thumbprint, lab.allow_untrusted + lab.host, + lab.username, + lab.password, + lab.port, + lab.thumbprint, + lab.allow_untrusted, ) try: vm = vim.VirtualMachine(lab.vm_moref, si._stub) @@ -268,13 +284,16 @@ def test_write_and_query_allocated_blocks_on_delta_disk(self, lab: LabEnv) -> No blocks = handle.query_allocated_blocks( disk, - start_sector=(write_sector // chunk_size_sectors) * chunk_size_sectors, + start_sector=(write_sector // chunk_size_sectors) + * chunk_size_sectors, num_sectors=chunk_size_sectors, chunk_size_sectors=chunk_size_sectors, ) assert any( b.offset <= write_sector < b.offset + b.length for b in blocks - ), f"written sector {write_sector} on delta file not covered by {blocks}" + ), ( + f"written sector {write_sector} on delta file not covered by {blocks}" + ) finally: try: vm = vim.VirtualMachine(lab.vm_moref, si._stub) @@ -283,6 +302,15 @@ def test_write_and_query_allocated_blocks_on_delta_disk(self, lab: LabEnv) -> No finally: Disconnect(si) + @pytest.mark.parametrize( + "open_flags", + [ + vixdisklib.VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ, + vixdisklib.VIXDISKLIB_FLAG_OPEN_COMPRESSION_ZLIB, + vixdisklib.VIXDISKLIB_FLAG_OPEN_COMPRESSION_SKIPZ, + ], + ids=["fastlz", "zlib", "skipz"], + ) @pytest.mark.parametrize( "aio_buffer_size, n_sectors, n_fragments", [ @@ -292,16 +320,20 @@ def test_write_and_query_allocated_blocks_on_delta_disk(self, lab: LabEnv) -> No ], ids=["64kib-128s", "64kib-129s", "2mib-129s"], ) - def test_skip_decompression_fastlz( + def test_skip_decompression( self, lab: LabEnv, + open_flags: int, aio_buffer_size: int, n_sectors: int, n_fragments: int, ) -> None: - """Pack FastLZ extras and rebuild the same bytes as a normal read.""" + """Pack compressed extras and rebuild the same bytes as a normal read.""" length = n_sectors * SECTOR_SIZE - expected = pattern_bytes(length, b"OVDL-SKIP-") + if open_flags == vixdisklib.VIXDISKLIB_FLAG_OPEN_COMPRESSION_SKIPZ: + expected = sparse_bytes(length, b"OVDL-SKIPZ-") + else: + expected = pattern_bytes(length, b"OVDL-SKIP-") handle = vixdisklib.VixDiskLibHandle( vixdisklib_compatibility_version="8.0", config_path=None ) @@ -312,13 +344,23 @@ def test_skip_decompression_fastlz( connect_kwargs = lab.vixdisklib_connect_kwargs( {"allow_untrusted": lab.allow_untrusted, "transport_modes": "nbd"} ) - flags = vixdisklib.VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ + expected_ctype = { + vixdisklib.VIXDISKLIB_FLAG_OPEN_COMPRESSION_FASTLZ: ( + nfc_open.NFC_COMPRESSION_FASTLZ + ), + vixdisklib.VIXDISKLIB_FLAG_OPEN_COMPRESSION_ZLIB: ( + nfc_open.NFC_COMPRESSION_ZLIB + ), + vixdisklib.VIXDISKLIB_FLAG_OPEN_COMPRESSION_SKIPZ: ( + nfc_open.NFC_COMPRESSION_SKIPZ + ), + }[open_flags] with ( handle.connect(**connect_kwargs) as conn, handle.open( conn, lab.disk_path, - flags=flags, + flags=open_flags, aio_buffer_size=aio_buffer_size, aio_buffer_count=1, ) as disk, @@ -339,5 +381,8 @@ def test_skip_decompression_fastlz( assert dests == {0} else: assert dests == {0, nfc_open.NFC_AIO_BUFFER_SIZE} + assert any( + frag.compression_type == expected_ctype for frag in skip.fragments + ), f"expected a {expected_ctype} fragment, got {skip.fragments}" assert plain_buf.raw[:length] == expected assert _rebuild_skip(skip_buf, skip) == expected diff --git a/tests/unit/test_nfc_open.py b/tests/unit/test_nfc_open.py index 4bf8ad1..1d13ca4 100644 --- a/tests/unit/test_nfc_open.py +++ b/tests/unit/test_nfc_open.py @@ -13,19 +13,23 @@ class _FakeSocket: """A minimal socket stand-in that replays scripted bytes for recv_into.""" - def __init__(self, replies: bytes) -> None: + def __init__(self, replies: bytes = b"") -> None: self._replies = replies self.sent: list[bytes] = [] def sendall(self, data: bytes) -> None: self.sent.append(bytes(data)) - def recv_into(self, buf: memoryview) -> int: - n = min(len(buf), len(self._replies)) - buf[:n] = self._replies[:n] + def recv_into(self, buffer: memoryview, nbytes: int = 0, flags: int = 0) -> int: + del nbytes, flags + n = min(len(buffer), len(self._replies)) + buffer[:n] = self._replies[:n] self._replies = self._replies[n:] return n + def close(self) -> None: + pass + def _open_reply_body( handle: int = 0x1234, @@ -87,7 +91,9 @@ def test_ignores_bits_beyond_chunk_count(self) -> None: class TestQueryAllocatedBlocksValidation: def _disk(self) -> nfc_open.NfcDisk: - return nfc_open.NfcDisk(sock=None, path="[ds] a.vmdk", handle=1, sector_size=512) + return nfc_open.NfcDisk( + sock=_FakeSocket(), path="[ds] a.vmdk", handle=1, sector_size=512 + ) def test_num_sectors_not_a_multiple_raises(self) -> None: with pytest.raises(ValueError, match="num_sectors must be a multiple"): @@ -102,15 +108,20 @@ def _ddb_get_reply(op_id: int, value: bytes | None) -> bytes: """Build a scripted DDB_GET reply: header + 16-byte body + value extra.""" value_length = len(value) if value is not None else 0 body = bytes(12) + struct.pack(" nfc_open.NfcDisk: return nfc_open.NfcDisk( - sock=_FakeSocket(replies), path="[ds] a.vmdk", handle=0x1234, sector_size=512 + sock=_FakeSocket(replies), + path="[ds] a.vmdk", + handle=0x1234, + sector_size=512, ) def test_found_key_returns_decoded_value(self) -> None: @@ -171,8 +182,12 @@ def test_combines_open_file_info_with_ddb_values(self) -> None: ) info = disk.query_full_info() assert info.capacity_sectors == 1024 - assert info.phys_geo == nfc_open.DiskGeometry(cylinders=10, heads=20, sectors=30) - assert info.bios_geo == nfc_open.DiskGeometry(cylinders=100, heads=200, sectors=63) + assert info.phys_geo == nfc_open.DiskGeometry( + cylinders=10, heads=20, sectors=30 + ) + assert info.bios_geo == nfc_open.DiskGeometry( + cylinders=100, heads=200, sectors=63 + ) assert info.adapter_type == "lsilogic" assert info.uuid == "some-uuid"