Skip to content

Replace native codecs with Rust implementations and support WebAssembly - #33

Open
rowanG077 wants to merge 10 commits into
encounter:mainfrom
rowanG077:rowan/remove-non-rust
Open

rowanG077 wants to merge 10 commits into
encounter:mainfrom
rowanG077:rowan/remove-non-rust

Conversation

@rowanG077

@rowanG077 rowanG077 commented Sep 20, 2026 •

Copy link
Copy Markdown

This replaces the C dependencies with pure rust versions. My motivation is that I use nod in the tales of symphonia re-implement project and I would like to not depend on anything non-rust. In the long run I would like to offer a wasm version.

The main things changed

  • Use flate2, bzip2 with its Rust backend, lzma-rust2, and structured-zstd.
  • Use RustCrypto for AES and hashing.
  • Remove native codec linking from CMake. Keep NOD_USE_CMAKE_COMPRESSION as a compatibility option with no effect.

Prepare for the Rust Zstandard dependency and keep the strict Clippy checks passing on the newer minimum Rust version.
Use miniz_oxide through flate2, preserving bounded output buffers and compatibility feature aliases. Add native reference decoding and bounded roundtrip tests.
Use bzip2 with libbz2-rs-sys and keep bounded compression behavior. Add the native reference fixture, license exception, and room for compressed WIA metadata.
Use lzma-rust2 for raw LZMA streams while preserving WIA properties and end markers. Add a native reference fixture and bounded roundtrip checks; retain liblzma temporarily for LZMA2.
Use lzma-rust2 for raw LZMA2 streams, preserving WIA properties and bounding dictionary allocation. Add a native reference fixture and remove the remaining liblzma FFI and dependency.
Use structured-zstd for library compression and the built-in CLI database. Verify frame sizes and checksums, and retain existing feature aliases.
Use the existing RustCrypto implementations for every build and retain the OpenSSL feature names as compatibility aliases. Add an AES-CBC known-vector test.
Remove native codec discovery and linking while preserving CMake options, the nod::nod target, and the unchanged C header. Remove obsolete Nix inputs.
Use synchronous processing on WebAssembly, omit native Python bindings, and avoid unsupported clock and stdout calls. Add a zero-import runtime smoke test and library-only WASM CI.
@mq1

mq1 commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

this simplifies a lot LTO, as using c dependencies requires setting up linker-plugin-lto to achieve c <-> rust lto

edit: cross-compilation should also be easier

@encounter

Copy link
Copy Markdown
Owner

Feels like this could just be another set of feature(s) that replace the C deps. Pretty invasive changes at a glance

@rowanG077

rowanG077 commented Sep 20, 2026 •

Copy link
Copy Markdown
Author

I considered that. But I didn't do it because this road makes things a lot more simple. Keeping around multiple back ends will make things even more complex than they are now.

There is code movement of course. But I think if you go through it commit by commit it's reasonable to review. If you truly want to add this only as new backends parallel to the C backends then I can refactor it as well to that structure.

@encounter

Copy link
Copy Markdown
Owner

That is fair, I will review closer soon! I mostly want to make sure that the quality of each Rust library is up to par with the C versions, performance measurements would be helpful if you have those handy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants