[Maven4] Migrate from Maven Archiver to standard jar tool - #508
[Maven4] Migrate from Maven Archiver to standard jar tool#508desruisseaux wants to merge 11 commits into
jar tool#508Conversation
Update all m4/pom.xml to use maven-jar-plugin 4.0.0-beta-2-PR508-SNAPSHOT which provides automatic JAR-per-module creation. Additional changes: - Convert example_addExports_manifest/m4/src/modmain from symlink to real directory to support MANIFEST.MF with Add-Exports and Main-Class - Add Maven 4 Migration Notes to example_addExports_manifest/README.adoc - Document resource handling in README.adoc Dependencies: - apache/maven-jar-plugin#508: Automatic JAR-per-module creation - apache/maven#11505: Module-aware resource copying 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]> Introduced in the course of support-and-care/maven-support-and-care#137
f564cdc to
21db1f0
Compare
5a235c8 to
5886fef
Compare
|
Thanks for this migration, @desruisseaux — moving JAR creation to the Reviewing it, I dug into the CI failures, and they share a root theme: the multi-release/module handling in They're on
Because these are order/vendor-dependent they only surface on CI (Linux+Zulu); the |
…ard `java.util.spi.ToolProvider` API (requires Java 9+). Files in the `classes` directories are dispatched to the `--manifest` and `--release` options, which allow additional verifications by the `jar` tool. Derive a POM for each individual JAR file as the intersection of the project model and the module-info of the JAR file. Side effects: * Remove the default `**/package.html` exclude. * Automatic use of `META-INF/MANIFEST.MF` file found in `classes` directory. * Automatic Multi-Release always enabled, unless `detectMultiReleaseJar` is set to `false`. * In a multi-module project, use Java module names as artifact names. * Whether the `--date` option is supported depends on the Java version. * Re-run the `jar` tool with the --validate operation mode if validation was not implicit.
5886fef to
046fbb2
Compare
|
Thanks! I cherry-picked the commits with modifications:
|
3ea1283 to
bb93fec
Compare
The plugin runs `jar --validate` after creating each archive. With the `jar` tool bundled in JDK 17 and 18, that validation crashes on any class compiled as a record (JDK-8282446, "This feature requires ASM8"). The issue was fixed in JDK 19 via the ASM 9.2 upgrade (JDK-8282508), but the fix was not backported to JDK 17u/18u. Since Maven 4 runs on JDK 17+, packaging a module that contains a record may fail in project with default configuration if using a JDK older than 19. Guard the validation pass on Runtime.version().feature() >= 19 and log an INFO line naming the skipped archive. The archive was already created by the `--create` pass. Behaviour on JDK 19+ is unchanged. Add an IT which tests a modular JAR containing a record and expects the build to succeed, so it turns the jdk-17 CI leg red until the validation pass is guarded for JDK 17/18. Co-Authored-By: Claude Opus 4.8 <[email protected]> Co-Authored-By: Martin Desruisseaux <[email protected]>
`Archive` seeded its base (version-less) `FileSet` from whichever directory first created the `Archive` via `computeIfAbsent(…)` in `FileCollector.enterModuleDirectory(…)`. Directory iteration order is unspecified, so when the walk enters "META-INF/versions-modular/<n>/<module>" before the base "<module>" dir, the base `FileSet` was bound to a version-specific directory with no module-info.class. The problem was not really the version-specific directory, but its association to the `null` key in `filesetForRelease`, which makes it the first map entry. Consequently `baseRelease().directory` pointed there, `ModuleFinder.of(...)` found no module, and `PomDerivation` dereferenced a null `ModuleReference` -> NPE. The "green on JDK 19" was accidental walk order, not a version gate. Add a `version` argument to the `Archive` constructor, which keep the `null` key free for the real base version when the walk will reach that version. Bug description partially from Claude Opus 4.8 <[email protected]> Co-Authored-By: Martin Desruisseaux <[email protected]>
`setMainClass(…)` unconditionally removes the `Main-Class` attribute from the plugin manifest (to either pass it via `--main-class` instead or to ignore it), but that manifest is shared across every module of a module hierarchy. The first module processed consumed the attribute, so when a non-owning module was processed first (directory iteration order is unspecified), the owning module never received its main class. Give each module a copy of the shared manifest in `ToolExecutor.writeSingleJAR` so processing order no longer decides the outcome. Add `ArchiveTest` with deterministic unit tests for this and for the base-release binding (both previously exercised only by ITs whose result depends on filesystem walk order). Co-Authored-By: Claude Opus 4.8 <[email protected]> Co-Authored-By: Martin Desruisseaux <[email protected]>
bb93fec to
0238e7d
Compare
`FileSet.add(…)` relativized only the first file added and stored the rest as absolute paths. This heuristic rule is based on the observation that is seems to be a `jar` tool requirement. However, which file is first depends on the unspecified directory iteration order. On some platforms, the tests fail with "names do not match" error message. This make the first file deterministic by choosing the shortest one. Issue identified by: Claude Opus 4.8 <[email protected]> Co-Authored-By: Martin Desruisseaux <[email protected]>
Tycho has such comparison (and can even e.g. detect different order in manifest files) but it is more used to compare if a version increment is needed to prevent deploying an artifact that (semantically) only differs in its promoted version. |
|
Thanks for creating the OpenJDK documentation issue about |
|
it seems I'll have to revisit Reproducible Builds for the jar plugin: after so many years of success, this looks like a regression that is so tiring to have to work back on :/ ... but I suppose better Java Module support deserves such efforts in the past, we did some JDK detection to patch when using JDK earlier than 18 https://issues.apache.org/jira/browse/MJAR-275 requiring plexus-archiver tweak codehaus-plexus/plexus-archiver#205 |
💯 , this is very critical to have proper reproducible builds |
|
Regarding the target directory creation. The target directory should be only created if the JAR is to be produced. Now the pom assemblies create redundant empty |
|
I'd duplicate my message from the mailing list there, for history. |
Please link spring issue for reference. Directory entries are not required in a JAR/ZIP file. |
|
@hboutemy, @elharo: about the issue of adding the missing directories while preserving deterministic file order, a workaround may be to add in the pull request a post-generation step where the JAR file is read and rewritten with directory entries added using Above proposal should preserve strict reproducible builds in the JAR plugin. For longer term, I still believe that bit-for-bit reproducible builds is too much and that we need semantically reproducible builds, but this is another story for maybe a separated plugin. Thanks @laeubi for the link to Tycho artifact comparator. Thanks @jaikiran for stepping in. The issue is that for creating a reproducible build with the
A possible
|
@desruisseaux, I dug into this, and I think the directory-based form is the way out here rather than a dead end — the tool's directory order is not platform-dependent. Since JDK-8276764 ("enable deterministic file content ordering for jar and jmod", in JDK 17), the Verified empirically — files created in a deliberately non-sorted order, then
So handing a directory to the tool gives us both reproducible entry order and the intermediate directory entries ( To make it concrete, I put together a branch on top of this PR — CI green on JDK 17 + 21 (full
Include/exclude filters and modular archives still fall back to per-file enumeration (they need it); that's documented in the code. It's meant as a starting point — cherry-pick or adapt as you see fit; I didn't want to push commits onto your PR. Happy to open it as a PR against this branch if that's easier to review. And if any genuine |
|
@seregamorph — since you have the exact Spring Boot 4.1.0 reproduction, would you be willing to try the branch against your real sources? Commit 1 alone should already cover your case (a plain, single-classes-directory app) — that's the one that restores the missing directory entries. To test: Then rebuild your project (it already resolves
It would be great to confirm it resolves the blocker on real sources. The full |
|
I just wanted to note that directory entries are optional, so if spring relies on them to be present it would be better fixed there to work without it. |
Let me give some clarity here. While I actually don't argue regarding this reasoning, absolutely most of spring boot projects as for today are based on Spring Boot 2 and 3. And Spring Boot 2 and 3 will never receive public updates anymore because of EoL OSS (there are only private for paid extended subscriptions), see https://spring.io/projects/spring-boot#support |
|
@jaikiran : just for information, I was not aware of JDK-8276764. It makes |
while the original MANIFEST had only <plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<configuration>
<archive>
<addMavenDescriptor>false</addMavenDescriptor>
<manifest>
<addDefaultEntries>false</addDefaultEntries>
</manifest>
</archive>
</configuration>
</plugin> |
With addDefaultEntries=false, Maven archiver writes no Created-By and the jar tool then stamps its own "<version> (<vendor>)" value. That is JDK- and vendor-specific (Temurin and Corretto differ), so it breaks binary reproducibility across JDK distributions and ignores addDefaultEntries=false. Ensure a stable, JDK-independent Created-By (the value MavenArchiver already supplies via setCreatedBy) is present before handing the manifest to the jar tool. This also replaces the previous "let the tool declare itself" branch, so non-reproducible builds no longer get the JDK-specific value either. Add the manifest-created-by integration test. Reported by Sergey Chernov on apache#508. Co-Authored-By: Claude Opus 4.8 <[email protected]>
|
@seregamorph @desruisseaux @hboutemy
Confirmed, and it is worth pinning down precisely because it is a reproducibility regression, not just a cosmetic one. Root cause. The JDK
So the same sources produce different manifest bytes on different JDK distributions — Corretto's The decision (and its alternative). There are two ways to fix it, and they differ in what
I went with (a) as a candidate on my branch One nuance for the record: strictly, reproducible builds are defined against a pinned build environment (the JDK included), so cross-JDK divergence is not a definitional violation. But Maven's own convention is deliberately JDK-independent manifests (archiver uses |
|
This is an example of why I hate binary reproducible builds. I think that Docker and other environments should adapt to semantically equivalent builds, but I realize that it will not happen soon. I will need time for incorporating the changes and will try to do that Saturday. Some changes compared to the current commits were discussed on Slack. In particular, a more systematic approach may be desirable for the issue of missing directory entries. |
if this true? if so let's drop toolprovider and get back maven archiver, this is cleaner than a workaround which can be insanelt slower (twice at least) on big jars and try to push upstream the needed features, there is no point to move to a new tool which is twice worse for only a validation most people will not need |
Not any more. When I wrote this comment, I was not aware of JDK-8276764. The proposal to apply a post-processing step is obsolete now. |
|
I just tested building a few simple projects with misc JDKs and this PR: first, I can see the the from my test, with JDK 19+, result is locally reproducible: I always get the same output on the same laptop with JDK 17 and 18, not reproducible: no problem with order of files, but with timestamp To me, avoid the issue with pom.properties should be easy |
I wanted to note here that (binary) reproducible builds are not guaranteeing anyways between different JDK versions and vendors as they could provide different byte codes. Apart from that, supplying the manifest directly would be the safest choice (also for the future) and thats why I think generation of files should be independent from packaging a jar: Apart from that, having the validation of MR-jars baked into the jar tool is from my opinion a design flaw and actually what we aim here if I understand correctly, so if the so if anyone is in contact with the JDK team it would be great to bring this to their attention as it would avoid people doing duplicate work an/or workarounds where a programmatic solution would be superior. |
|
continuing my base test (no MR or modular build yet, just basic classic jars) rebuilding on a Mac in parallel of a Linux box with JDK 21 gave me same output: great, we have the output stability that I expect and got until now So definitively, the JDK 17 and 18 reproducibility issue is what has to be solved for now We'll see in a second step about modular or MR jars: if someone can point me to a simple project that I can build mself easily, that would be nice, please one issue I get is twice attach: that is also visible in the |
I agree, and this is one reason why I disagree with the importance given to strict (as opposed to semantically) reproducible builds. I think that it can even deserve the real goal, which is security. If some Java tools produce vulnerable code (e.g., Javadoc HTML frame injection vulnerability in 2013), it is difficult to know if a JAR file has been produced by patched tools if we don't have a useful Regarding the split of metadata generation (
Regarding the JAR file validation, this is one of the reasons for this migration to the |
But is is worth the effort? It would be a hack needed for JDK 17 and 18 only. And since the file ordering issue (JDK-8276764) was fixed in JDK 18 only, even if we fix the timestamp the build would still non-reproducible on JDK 17. Therefore, setting the timestamp would be useful for only one specific JDK version: 18. |
|
Hello Martin,
Bug fixes/enhancements in the JDK are sometimes backported. Those backports are tracked/linked against the original issue. For this specific issue https://bugs.openjdk.org/browse/JDK-8276764 , under the issue description, there's a backports section which shows that this enhancement was backported to JDK 17 (the Resolved/Fixed status on those backports is a sign that the work is complete). Having said that, my comment is mostly informational and isn't meant as an input to whether or not some post processing should be done for the JAR files. |
|
IMO it's critical to support binary reproducibility for all JDKs since minimal 17, otherwise it'll be a regression. |
|
Thanks @jaikiran for the clarification. Then indeed, editing timestamps in a post-processing would work on both JDK 17 and 18. Whether we should do this post-processing is still uncertain to me. Not doing this post-processing would be a regression only for users wanting binary reproducible builds with the tools of JDK 17 or 18. Users who do not upgrade to a more recent Java version at least for building (even if they target a lower release with |
stating that for maven 4 java 17 bytecode is supported using a java 21 or more (to speak "LTS") doesn't sound crazy to me
ZipFileSystem#sync can be worth benching on a big jar (some thousands of classes), ideally it is very small compared to producing the zip (jar) itself but worth evaluating before thinking this is an option IMHO - since we know we can always do it right directly if needed and JDK version is a blocker (once again not for me) |
This is a major refactoring of the Maven JAR Plugin for using the standard
jartool instead of Maven Archiver. The standard tool is available through thejava.util.spi.ToolProviderinterface, which was introduced in Java 9. Since Maven 4 upgraded its requirement from Java 8 to Java 17, the use of that interface is no longer problematic.Rational
In early Java days, the
jartool was equivalent to azipcommand with a different syntax and a little bit of special processing for theMETA-INF/MANIFEST.MFfile. Because of this quasi-equivalence, it was not difficult to create JAR files ourselves using any library capable to write ZIP files. This is what Maven Archiver does, together with supporting other archive formats. But today, thejartool became more sophisticated. It now includes options for verifying the consistency of multi-release JAR files, options for updatingmodule-info.class, provides security features specific to Java, etc.. This evolution can be seen in the "Modular JAR files" section of the Maven Plugin documentation, which states that the plugin uses thejartool for updating the JAR file in a way that the Maven Archiver can't do easily. Therefore, since Maven 4 requires Java 17 and since that Java version gives us an easy access to thejartool through thejava.util.spi.ToolProviderinterface, it may be time to abandon our manual creation of a JAR file and rely fully on thejartool instead.Benefits
Safer multi-release JAR files
In a multi-release JAR file, blindingly storing the content of
META-INF/versions/directories as if they were ordinary resources is not equivalent to using thejar --releaseoption. The difference is that in the latter case, thejartool performs some consistency checks. This issue was independently reported by user in #484, which is fixed by this pull request. Note that if this verification is not desired, Maven users can disable it by setting thedetectMultiReleaseJarplugin option tofalse.Main class managed by the
jartoolIn a modular JAR file, it is no longer sufficient to declare the main class in the
Main-Classentry of theMANIFEST.MFfile. The main class needs to be specified by the--main-classoption of thejartool, which will updatemodule-info.class. For compatibility reason the Maven JAR plugin gets the option value from that manifest entry, but internally the use of thejartool is mandatory.Options not yet supported by the plugin
This approach allows to add a
<jarArgs>configuration option, similar to the<compilerArgs>in the Maven Compiler Plugin. Such option would allow developers to use new tool arguments before they are supported by the plugin.Examples of options not yet supported by the plugin are
--module-versionand--hash-modules. Explicit support for those options could be added in a future plugin version, especially since security is becoming more and more a concern.Easier debugging and toolchain
This approach makes easy to generate a
target/jar.argsfile when the build fails or when Maven is run in verbose mode. This is similar to the compiler plugin generating atarget/javac.argsfile. This file allows the user to test easily on the command-line, which makes debugging faster. Likewise, the options can also be passed to another tool, which makes easier to resolve #439 as well.Behavioral changes
The plugin behaviour after the proposed refactoring is different than version 3 in the following aspects:
Removal of default
**/package.htmlexcludesThe current plugin version uses an undocumented
**/package.htmldefault excludes. This default seems to exist since the initial revision in March 2004, but I saw no explanation for this oddity. This default is not mentioned in the documentation. The removal of this oddity is necessary for allowing the proposed new plugin implementation to specify only some root directories to thejartool, since the tool can traverse the directory tree itself.Automatic use of
MANIFEST.MFAfter this refactor, the plugin automatically uses the
META-INF/MANIFEST.MFfile found in theclassesdirectory. Before this refactor, the plugin used that file only if explicitly specified in the<manifestFile>archive configuration. The previous policy was discussed in #255. The new policy is a natural consequence of the way that the JAR plugin is reimplemented, and also more useful in the context of multi-module (in Java module sense) projects since each module could contain its ownMANIFEST.MFfile.Multi-module projects
The refactoral support multi-module projects, including projects that are both multi-module and multi-release, as discussed in apache/maven-compiler-plugin#998.
Dependencies
This pull request depends on a Maven release which include the following pull requests:
META-INF/versionswhen the sub-directories are multi-module maven-compiler-plugin#998