Skip to content

Keep PKCS5S2 usable on a FIPS-restricted JCE, and name the key wrapping property when the runtime has no RSA-OAEP - #1058

Open
vharseko wants to merge 2 commits into
OpenIdentityPlatform:masterfrom
vharseko:fix/pkcs5s2-secure-random-fips
Open

vharseko wants to merge 2 commits into
OpenIdentityPlatform:masterfrom
vharseko:fix/pkcs5s2-secure-random-fips

Conversation

@vharseko

Copy link
Copy Markdown
Member

Problem

#1054 changes the config template to get a server installed on a FIPS 140-3 Java runtime: it disables the PKCS5S2 password storage scheme and sets key-wrapping-transformation to RSA/ECB/PKCS1Padding. The review there explains why neither belongs in the template — PKCS5S2 is PBKDF2-HMAC-SHA1 like the PBKDF2 scheme which stays enabled, and PKCS#1 v1.5 is the transformation #776 removed as a CodeQL java/rsa-without-oaep finding. What actually fails on such a runtime:

  • PKCS5S2PasswordStorageScheme asked for SecureRandom "SHA1PRNG" by name, at initialization and in the offline encoder. A FIPS-restricted JCE registers no such algorithm, the scheme threw a message-less InitializationException, and the server refused to start.
  • The crypto manager validates its key wrapping transformation when it is created. A runtime whose only RSA cipher is PKCS#1 v1.5 — a SunPKCS11 provider on its own, as on a Linux system in FIPS mode; JDK-6190389 is still open — cannot start the server with the RSA-OAEP default. Setup noticed (ConfigureDS.updateCryptoCipher), had no secure fallback left after Fix java/rsa-without-oaep CodeQL alert by dropping the RSA PKCS#1 v1.5 fallback #776, and silently kept the default; the failed start which followed named the cipher it could not get and nothing else.

Change

  • PKCS5S2PasswordStorageScheme takes the provider's default SecureRandom, as the PBKDF2 family does, and its InitializationException — like the PBKDF2 schemes' — names the algorithm which is missing. The SHA1PRNG constant goes. The template keeps the scheme enabled.
  • setup prints a warning naming the transformation, the reason and the property to set before the first start (WARN_CONFIGDS_KEY_WRAPPING_TRANSFORMATION_UNSUPPORTED), and the crypto manager's refusal (ERR_CRYPTOMGR_CANNOT_GET_PREFERRED_KEY_WRAPPING_CIPHER) names the property too. The choice itself stays with the administrator: the install guide gains "To Install OpenDJ Directory Server on a FIPS 140 Java Runtime" — with the bundled BC-FIPS provider, which offers RSA-OAEP, the default configuration works as it is; a SunPKCS11-only runtime gets the setup --doNotStartldifmodifystart-ds procedure, with the caveat that PKCS#1 v1.5 is disallowed for key transport by NIST SP 800-131A Rev. 2 and that every replica must use the same transformation.
  • A FIPS-approved key transport for such runtimes (SP 800-56B KTS-KEM-KWS over RSA/ECB/NoPadding) is Crypto manager: a FIPS-approved key transport for JVMs whose only RSA cipher is PKCS#1 v1.5 (SunPKCS11) #1056.

Testing

The new tests withdraw the JCE providers which register a service, run the code without them, and put the providers back where they were (TestCaseUtils.withoutJceService):

  • PKCS5S2PasswordStorageSchemeTestCase: the scheme initializes and encodes, online and offline, without SHA1PRNG (SUN withdrawn, BC-FIPS standing in for the digests); its initialization failure names PBKDF2WithHmacSHA1 (SunJCE withdrawn). The same failure test for PBKDF2PasswordStorageSchemeTestCase.
  • ConfigureDSTestCase (new): the default transformation stays where the runtime has it; without Cipher.RSA none is chosen and the exception names the default.
  • CryptoManagerTestCase: an unsupported transformation is refused, and the refusal names key-wrapping-transformation.

Each test was run red before its fix and green after: PKCS5S2PasswordStorageSchemeTestCase 56/56, PBKDF2PasswordStorageSchemeTestCase 40/40, PBKDF2HmacSHA256/512PasswordStorageSchemeTestCase 39/39 each, CryptoManagerTestCase 31/31, ConfigureDSTestCase 2/2.

Alternative to #1054. Related: #776, #1056.

PKCS5S2PasswordStorageScheme asked for SecureRandom "SHA1PRNG" by name, at
initialization and again in the offline encoder. A FIPS-restricted JCE
(SunPKCS11-NSS-FIPS, BC-FIPS) registers no such algorithm, so the scheme
threw a message-less InitializationException and the server refused to
start — the failure OpenIdentityPlatform#1054 works around by disabling the scheme in the
config template, which would also stop {PKCS5S2} hashes imported from
Atlassian products from binding on every new installation.

Take the provider's default SecureRandom instead, as the PBKDF2 family
already does, and let the InitializationException of both PKCS5S2 and the
PBKDF2 schemes name the algorithm that is missing rather than leaving the
administrator with no reason for the failed start. The SHA1PRNG constant
has no user left and goes.

The tests withdraw the providers which register the service — SUN for
SHA1PRNG, with BC-FIPS standing in for the digests, SunJCE for
PBKDF2WithHmacSHA1 — run the scheme without them, and put them back.
… the runtime has no RSA-OAEP

The crypto manager validates its key wrapping transformation when it is
created, so a Java runtime whose only RSA cipher is PKCS#1 v1.5 — a
SunPKCS11 provider on its own, as on a Linux system in FIPS mode
(JDK-6190389 is still open) — cannot start the server with the default,
RSA-OAEP. Setup noticed as much in ConfigureDS.updateCryptoCipher and,
since OpenIdentityPlatform#776 left it no secure transformation to fall back to, silently kept
the default; the failed start which followed named the cipher it could not
get and nothing else.

Setup now prints a warning naming the transformation, the reason and the
property to set before the first start, and the crypto manager's refusal
names the property as well. The choice itself stays with the
administrator: the install guide gains a procedure for a FIPS 140 runtime
which says what the bundled BC-FIPS provider offers, what a SunPKCS11-only
runtime does not, and how to set key-wrapping-transformation in
config.ldif between setup --doNotStart and start-ds.
@vharseko vharseko added docs tests Test suites: fixing, enabling, un-disabling security Security fixes / CodeQL code-scanning alerts setup setup / upgrade / uninstall tools (quicksetup) and the launcher scripts java Changes to Java sources fips FIPS 140 mode: BC-FIPS provider, BCFKS keystores, approved algorithms labels Sep 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs fips FIPS 140 mode: BC-FIPS provider, BCFKS keystores, approved algorithms java Changes to Java sources security Security fixes / CodeQL code-scanning alerts setup setup / upgrade / uninstall tools (quicksetup) and the launcher scripts tests Test suites: fixing, enabling, un-disabling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant