Add SHA3 mechanism support - #894
Open
AlexLaroche wants to merge 10 commits into
Open
Conversation
Implement the full PKCS#11 v3.2 SHA3 mechanism family for both the OpenSSL and Botan crypto backends: plain digest (CKM_SHA3_*), HMAC (CKM_SHA3_*_HMAC), and RSA/DSA/ECDSA combined signing (CKM_SHA3_*_RSA_PKCS[_PSS], CKM_DSA_SHA3_*, CKM_ECDSA_SHA3_*). Also accepts a SHA3 hash parameter in CKM_RSA_PKCS_OAEP and in raw CKM_RSA_PKCS_PSS signing with a precomputed digest. Not implemented: CKM_SHA3_*_KEY_GEN and CKM_SHA3_*_HMAC_GENERAL, neither of which is implemented for any existing hash family today. Closes softhsm#603.
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughSoftHSMv2 adds conditional SHA-3-224/256/384/512 support for hashing, HMAC, RSA, DSA, ECDSA, and RSA-OAEP through OpenSSL and Botan, including configuration detection, PKCS#11 routing, backend implementations, build integration, and tests. ChangesSHA-3 crypto foundation
Estimated code review effort: 4 (Complex) | ~60 minutes Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/lib/crypto/BotanRSA.cpp`:
- Around line 1478-1493: Update the sLen upper-bound check in the hash-parameter
handling near the SHA3 cases to use the computed allowedLen instead of the
hardcoded SHA-1 digest size. Preserve the existing PSS bound logic while
applying the selected hash algorithm’s digest length for all supported
algorithms.
In `@src/lib/crypto/OSSLRSA.cpp`:
- Around line 180-195: Update the OpenSSL/LibreSSL dependency validation in
configure.ac to require OpenSSL 1.1.1 or newer and LibreSSL 3.8.0 or newer,
matching the direct EVP_sha3_* calls in the HashAlgo SHA3 cases of OSSLRSA.cpp.
Ensure unsupported versions are rejected during configuration rather than
reaching compilation.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 35e1972a-5cb0-4820-8504-0a2cf3fe700e
📒 Files selected for processing (41)
src/bin/dump/tables.hsrc/lib/P11Attributes.cppsrc/lib/SoftHSM.cppsrc/lib/crypto/AsymmetricAlgorithm.hsrc/lib/crypto/BotanCryptoFactory.cppsrc/lib/crypto/BotanDSA.cppsrc/lib/crypto/BotanECDSA.cppsrc/lib/crypto/BotanMAC.cppsrc/lib/crypto/BotanMAC.hsrc/lib/crypto/BotanRSA.cppsrc/lib/crypto/BotanSHA3_224.cppsrc/lib/crypto/BotanSHA3_224.hsrc/lib/crypto/BotanSHA3_256.cppsrc/lib/crypto/BotanSHA3_256.hsrc/lib/crypto/BotanSHA3_384.cppsrc/lib/crypto/BotanSHA3_384.hsrc/lib/crypto/BotanSHA3_512.cppsrc/lib/crypto/BotanSHA3_512.hsrc/lib/crypto/CMakeLists.txtsrc/lib/crypto/HashAlgorithm.hsrc/lib/crypto/MacAlgorithm.hsrc/lib/crypto/Makefile.amsrc/lib/crypto/OSSLCryptoFactory.cppsrc/lib/crypto/OSSLDSA.cppsrc/lib/crypto/OSSLECDSA.cppsrc/lib/crypto/OSSLHMAC.cppsrc/lib/crypto/OSSLHMAC.hsrc/lib/crypto/OSSLRSA.cppsrc/lib/crypto/OSSLSHA3_224.cppsrc/lib/crypto/OSSLSHA3_224.hsrc/lib/crypto/OSSLSHA3_256.cppsrc/lib/crypto/OSSLSHA3_256.hsrc/lib/crypto/OSSLSHA3_384.cppsrc/lib/crypto/OSSLSHA3_384.hsrc/lib/crypto/OSSLSHA3_512.cppsrc/lib/crypto/OSSLSHA3_512.hsrc/lib/crypto/test/HashTests.cppsrc/lib/crypto/test/HashTests.hsrc/lib/test/AsymEncryptDecryptTests.cppsrc/lib/test/DigestTests.cppsrc/lib/test/SignVerifyTests.cpp
EVP_sha3_*() requires OpenSSL >= 1.1.1 or LibreSSL >= 3.8.0, newer than SoftHSM's OpenSSL 1.0.0 baseline. Add a --enable-sha3 configure option (default detect, mirroring --enable-ecc/--enable-eddsa/--enable-mldsa) that probes for real EVP_sha3_224/256/384/512 support via a compile/link check rather than a version-number comparison, since LibreSSL pins OPENSSL_VERSION_NUMBER to a fixed value and can't be version-sniffed. Botan always resolves to supported, since it has had SHA3 since well before its own minimum required version. Every SHA3 code path is gated behind the resulting WITH_SHA3 define: the OSSLSHA3_*/BotanSHA3_* digest classes, the HMAC-SHA3 implementations, crypto factory dispatch, every EVP_sha3_*() call site in OSSLRSA.cpp, the PKCS#11 mechanism registration/dispatch in SoftHSM.cpp, and the corresponding unit tests -- mirroring how WITH_ECC/WITH_EDDSA are already handled throughout the codebase. This keeps the OpenSSL floor at 1.0.0 while avoiding build failures against old OpenSSL or LibreSSL.
getCipherRawPss() hardcoded the salt-length upper bound (bitLength+6)/8-2-20 assuming a 20-byte SHA-1 digest, even though the function already computes allowedLen per hash algorithm just above. For any non-SHA1 hash (SHA-224/256/384/512, SHA3-224/256/384/512) this let sLen exceed the true PKCS#1 v2.1 RSASSA-PSS bound (emLen - hLen - 2), miscomputing the maximum allowed salt length. Use allowedLen instead of the hardcoded 20.
AlexLaroche
force-pushed
the
add-sha3-support
branch
from
July 23, 2026 00:37
4301084 to
41cd8d2
Compare
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@m4/acx_openssl_sha3.m4`:
- Around line 11-34: Update the OpenSSL SHA3 capability check using
AC_CACHE_CHECK, or move AC_MSG_RESULT reporting outside AC_CACHE_VAL, so the
cached value still produces a configure result on every run. Preserve the
existing acx_cv_lib_openssl_sha3_support detection and yes/no value assignments.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 0e2bcbde-32a0-4a05-b744-43ce6bdcc5ec
📒 Files selected for processing (23)
README.mdm4/acx_crypto_backend.m4m4/acx_openssl_sha3.m4src/lib/SoftHSM.cppsrc/lib/crypto/BotanCryptoFactory.cppsrc/lib/crypto/BotanMAC.cppsrc/lib/crypto/BotanRSA.cppsrc/lib/crypto/BotanSHA3_224.cppsrc/lib/crypto/BotanSHA3_256.cppsrc/lib/crypto/BotanSHA3_384.cppsrc/lib/crypto/BotanSHA3_512.cppsrc/lib/crypto/OSSLCryptoFactory.cppsrc/lib/crypto/OSSLHMAC.cppsrc/lib/crypto/OSSLRSA.cppsrc/lib/crypto/OSSLSHA3_224.cppsrc/lib/crypto/OSSLSHA3_256.cppsrc/lib/crypto/OSSLSHA3_384.cppsrc/lib/crypto/OSSLSHA3_512.cppsrc/lib/crypto/test/HashTests.cppsrc/lib/crypto/test/HashTests.hsrc/lib/test/AsymEncryptDecryptTests.cppsrc/lib/test/DigestTests.cppsrc/lib/test/SignVerifyTests.cpp
🚧 Files skipped from review as they are similar to previous changes (19)
- src/lib/crypto/BotanSHA3_224.cpp
- src/lib/crypto/BotanSHA3_256.cpp
- src/lib/crypto/BotanSHA3_512.cpp
- src/lib/crypto/OSSLSHA3_512.cpp
- src/lib/crypto/OSSLSHA3_384.cpp
- src/lib/test/DigestTests.cpp
- src/lib/crypto/OSSLSHA3_224.cpp
- src/lib/crypto/OSSLCryptoFactory.cpp
- src/lib/crypto/BotanCryptoFactory.cpp
- src/lib/crypto/BotanSHA3_384.cpp
- src/lib/crypto/BotanMAC.cpp
- src/lib/crypto/test/HashTests.h
- src/lib/crypto/test/HashTests.cpp
- src/lib/crypto/OSSLSHA3_256.cpp
- src/lib/crypto/OSSLHMAC.cpp
- src/lib/crypto/BotanRSA.cpp
- src/lib/crypto/OSSLRSA.cpp
- src/lib/test/SignVerifyTests.cpp
- src/lib/SoftHSM.cpp
AC_CACHE_VAL skips its body once acx_cv_lib_openssl_sha3_support is cached, so the AC_MSG_RESULT calls inside it never ran on repeat configure invocations, leaving "checking for OpenSSL SHA3 support..." without a yes/no. Switch to AC_CACHE_CHECK, which prints the cached result unconditionally instead of only on a cold cache.
Every other optional crypto feature added alongside SHA3 (ECC, EDDSA, ML-DSA, ML-KEM) has a #cmakedefine entry here; SHA3 was missing one.
Addresses review comment on PR softhsm#894: the mechanism-param type/hash/MGF validation and salt-length bound check were duplicated across all 10 PSS hash variants in both signInit and verifyInit.
…SLRSA) Antoine pointed out on PR softhsm#894 that the same duplication factored out of BotanRSA in the previous commit exists in OSSLRSA's signInit and verifyInit. Applies the same checkPssParams() helper there.
Antoine noted on PR softhsm#894 that BotanRSA::checkPssParams and OSSLRSA::checkPssParams were identical, and suggested factoring it into a dedicated utility class the way MLDSAUtil/MLKEMUtil already do for their algorithms. Adds RSAUtil::checkPssParams and has both backends call it instead of keeping their own copy.
Contributor
|
I think this one is ready for merging. |
bukka
requested changes
Jul 27, 2026
bukka
left a comment
bukka
left a comment
Member
There was a problem hiding this comment.
It looks good, just some minor things.
Comment on lines
+2
to
+25
| * Copyright (c) 2010 .SE (The Internet Infrastructure Foundation) | ||
| * All rights reserved. | ||
| * | ||
| * Redistribution and use in source and binary forms, with or without | ||
| * modification, are permitted provided that the following conditions | ||
| * are met: | ||
| * 1. Redistributions of source code must retain the above copyright | ||
| * notice, this list of conditions and the following disclaimer. | ||
| * 2. Redistributions in binary form must reproduce the above copyright | ||
| * notice, this list of conditions and the following disclaimer in the | ||
| * documentation and/or other materials provided with the distribution. | ||
| * | ||
| * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR | ||
| * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED | ||
| * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE | ||
| * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY | ||
| * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL | ||
| * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE | ||
| * GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS | ||
| * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER | ||
| * IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR | ||
| * OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN | ||
| * IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. | ||
| */ |
Member
There was a problem hiding this comment.
For new files, we just use this.
Suggested change
| * Copyright (c) 2010 .SE (The Internet Infrastructure Foundation) | |
| * All rights reserved. | |
| * | |
| * Redistribution and use in source and binary forms, with or without | |
| * modification, are permitted provided that the following conditions | |
| * are met: | |
| * 1. Redistributions of source code must retain the above copyright | |
| * notice, this list of conditions and the following disclaimer. | |
| * 2. Redistributions in binary form must reproduce the above copyright | |
| * notice, this list of conditions and the following disclaimer in the | |
| * documentation and/or other materials provided with the distribution. | |
| * | |
| * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR | |
| * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED | |
| * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE | |
| * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY | |
| * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL | |
| * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE | |
| * GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS | |
| * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER | |
| * IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR | |
| * OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN | |
| * IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. | |
| */ | |
| * Copyright (c) 2026 SoftHSMv2 contributors | |
| * | |
| * SPDX-License-Identifier: BSD-2-Clause | |
| */ |
Member
There was a problem hiding this comment.
Same applies to all other new files.
Author
There was a problem hiding this comment.
Applied to all 16 new files (BotanSHA3_/OSSLSHA3_ .cpp/.h) in accc4d7.
| #cmakedefine WITH_ML_KEM @WITH_ML_KEM@ | ||
|
|
||
| /* Compile with SHA3 support */ | ||
| #cmakedefine WITH_SHA3 @WITH_SHA3@ |
Member
There was a problem hiding this comment.
how does this get enabled in cmake?
Author
There was a problem hiding this comment.
Good catch — it didn't. config.h.in.cmake referenced @WITH_SHA3@ but nothing in the CMake side ever set that variable, so any CMake build was silently compiling out SHA3 support entirely (RSA-SHA3-PKCS, RSA-SHA3-PSS, HMAC-SHA3, digests, everything gated behind WITH_SHA3).
Fixed in 9b8b40c by mirroring the autotools logic in cmake/modules/CompilerOptions.cmake: Botan always has SHA3 (matches the "no version detection is needed" comment in acx_crypto_backend.m4) so it's set unconditionally there; OpenSSL gets a try_run sanity check (new cmake/modules/tests/test_openssl_sha3.c, same EVP_sha3_* probe as acx_openssl_sha3.m4) requiring OpenSSL >= 1.1.1 / LibreSSL >= 3.8.0.
Verified with an actual CMake configure+build (OpenSSL backend): WITH_SHA3 lands in the generated config.h, full build is clean, and cryptotest/p11test (81 tests, including SHA3 PSS sign/verify) all pass. The Botan+CMake side reports "Botan: SHA3 is supported" and WITH_SHA3 is set correctly too, but I couldn't get a full Botan CMake build to link — that's blocked by an unrelated pre-existing bug (src/lib/crypto/CMakeLists.txt:171 unconditionally appends MSVC-style /wd* flags for any WITH_BOTAN build regardless of compiler, from 2021, commit 8fd89ec). Happy to file that separately if useful, didn't want to fold an unrelated fix into this PR.
bukka pointed out on PR softhsm#894 that new files should use the short "Copyright (c) 2026 SoftHSMv2 contributors / SPDX-License-Identifier: BSD-2-Clause" header (as used by RSAUtil.h etc.) instead of the long boilerplate BSD license text copied from the file each was based on.
bukka asked on PR softhsm#894 how WITH_SHA3 gets enabled via CMake — it didn't; config.h.in.cmake referenced @WITH_SHA3@ but nothing ever set it, so CMake builds silently compiled out all SHA3 support. Mirrors the autotools logic (m4/acx_crypto_backend.m4, acx_openssl_sha3.m4): Botan always has SHA3 so it's set unconditionally, OpenSSL gets a try_run sanity check requiring OpenSSL >= 1.1.1 / LibreSSL >= 3.8.0. Verified with a real CMake configure+build: OpenSSL backend builds clean and cryptotest/p11test (81 tests, SHA3 PSS included) pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the full PKCS#11 v3.2 SHA3 (FIPS 202) mechanism family for both the OpenSSL and Botan crypto backends:
CKM_SHA3_224/256/384/512)CKM_SHA3_*_HMAC)CKM_SHA3_*_RSA_PKCS,CKM_SHA3_*_RSA_PKCS_PSS), including the raw/precomputed-digestCKM_RSA_PKCS_PSSpathCKM_DSA_SHA3_*)CKM_ECDSA_SHA3_*)Not included:
CKM_SHA3_*_KEY_GENandCKM_SHA3_*_HMAC_GENERAL— neither is implemented for any existing hash family today, so neither is added here either.Closes #603.
Test plan
HashTests: FIPS 202 known-answer vectors for SHA3-224/256/384/512, both backendsDigestTests: digest round-trip for all four sizesSignVerifyTests: HMAC-SHA3 round-trip; RSA-PKCS and RSA-PKCS-PSS round-trip (named and raw/precomputed-digest); ECDSA round-trip across P-256/P-384/P-521; DSA wrong-key-type negative cases extended (no positive DSA+hash round-trip exists for any size today, so none was added here either)AsymEncryptDecryptTests: RSA-OAEP round-trip with a SHA3 hash parameter, with and without a labelSummary by CodeRabbit
--enable-sha3configure option.