Repository navigation
Implement zlib and SkipZ NBD compression - #5
Merged
Merged
Conversation
Reverse-engineered and implemented against a live standalone ESXi 8.0.3 host (no vCenter), closing most of the gap versus the proprietary VDDK: - Direct ESXi (no vCenter) connectivity: nfc_service() previously hardcoded the NfcService moref as "nfcService" (vCenter's name), which fails on bare ESXi (moref is "ha-nfc-service" there). Now resolved dynamically via RetrieveInternalContent, same as VDDK itself does. Also fixes connect_authd() for tickets that omit `host` (implicit on a direct-ESXi ticket). - VixDiskLib_GetInfo: capacity and physical geometry come free from the OPEN_FILE reply (offsets already in the wire frame). biosGeo, adapterType, and uuid are fetched via DDB_GET, matching real VDDK's behavior and cost exactly. - DDB_GET (VMDK descriptor lookups): generic key/value NFC message, values are ASCII text on the wire (not binary), matching how a VMDK descriptor's DDB section is stored. - VixDiskLib_QueryAllocatedBlocks: allocated-block bitmap query. Verified against a live disk to exactly match native VDDK's output, including two non-obvious wire details: a field-order swap that's invisible in a zero-offset capture, and 4-byte bitmap padding that only shows up for small chunk counts. - Changed Block Tracking: turned out to need no NFC work at all -- VirtualMachine.QueryChangedDiskAreas is public VIM API. Added thin wrappers (enable_change_tracking / disk_change_id / query_changed_disk_areas) and documented real-world characteristics (extent granularity, wildcard changeId semantics) from live testing. - Investigated NFC_DELTA_DISK: found it's an optional VMFS-only VDDK client optimization (per `strings` on libvixDiskLib.so), not a correctness requirement -- reading, writing, and querying allocated blocks on an actual snapshot delta file already work with the existing NFC_DISK-only implementation. Documented a real gotcha found along the way: querying allocated blocks on the same still-open handle a write just went through can see stale data. Adds unit tests (bitmap decode/merge, DDB_GET wire format, CBT dataclass conversion, validation errors) and integration tests (GetInfo, QueryAllocatedBlocks, CBT full cycle, delta-disk read/write/ query) validated against a live ESXi 8.0.3 lab. Full protocol details and the reverse-engineering process are in docs/nfc_auth.md, docs/nfc_open.md, docs/nfc_read.md, docs/cbt.md, and docs/reverse_engineering_procedure.md.
Captured native VDDK's ZLIB and SkipZ IO compression with the same SSL-hook technique used for FastLZ: same opcode/offset-36 framing, just different type values (1 zlib, 3 SkipZ). ZLIB's extra is a plain zlib stream, decodable with Python's stdlib directly. SkipZ turned out to not be general compression at all -- it just omits runs of zero bytes, keeping everything else raw, which only became visible after switching from an all-non-zero test pattern (which correctly fell back to uncompressed, same as it would for FastLZ/zlib) to one with real zero-filled runs. Implemented as new _skipz_compress/_skipz_decompress helpers alongside the existing FastLZ path in NfcDisk.write/readinto, wired through VixDiskLibHandle's existing compression-flag mapping. Validated against the live ESXi lab for all three algorithms, with SkipZ's byte-level (non-sector-aligned) run encoding specifically cross-checked by writing with openvixdisklib and reading back with native VDDK.
We'll use the same typing.Protocol approach as cloudbase#4, minimizing merge conflicts.
We'll extend the integration tests to cover the newly introduced compression algorithms: zlib and skipz. In case of skipz, we'll generate a pattern that contains large enough zero blocks to trigger skipz compression.
Keep direct-ESXi, GetInfo, QueryAllocatedBlocks, and CBT, and add zlib and SkipZ.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Collected from cloudbase#8
Original author: doccaz
Original branch:
zlib-skipz-compressionThis pull request is the open upstream change, retargeted at
replication_agent.Adds zlib and SkipZ NBD compression. FastLZ was already on main. The same upstream branch also carried direct-ESXi, GetInfo, QueryAllocatedBlocks, and CBT; those are already merged, and this merge keeps them.