Crash in yubihsm2_pkcs11.so when using openssl with pkcs11-provider

[myksyr-tdy]

yubihsm2_pkcs11.so from tag 2.5.0 (and from 2.4.2 too) crashes when used with openssl 3.0.13-0ubuntu3 and pkcs11-provider 0.3-1build2. The ultimate reason for the crash (as evident from running with valgrind) is use-after-free (which crashes occasionally) and finally double-free (which crashes 100% on libc check). The reason for that is cleanup ordering issues: openssl winds down internal modules, and half way into that calls provider cleanup, which in turn invokes yubihsm2_pkcs11 cleanup. The latter tries to use crypto module to communicate with YubiHSM, but that one is already long dead.

There are plenty of discussions about cleanup ordering of pkcs11_provider in context of SoftHSM, and it seems that general conclusion there is that the issue is in SoftHSM implementation. From that I can draw conclusion that it is possible that the issue at hand can be in yubihsm2_pkcs11 - but I'm long from being sure about that.

I've found a workaround - don't call into yubihsm_pkcs11 cleanup on winddown, but that reduces overall crypto-op bandwidth, so preferably I would like to see this debugged properly. I don't have neither time nor expertise to dig deeper into source code of openssl/pkcs11-provider/yubihsm-pkcs11, but eager to support in any way possible.

As a side note it would be great to see pkcs11-provider usage example in YubiHSM 2 user guide, as the only option there (engines) is obsolete in openssl 3+ and has it's own bugs which won't be fixed ever.

This is relevant part of working openssl config:

[provider_sect]
default = default_sect
pkcs11 = pkcs11_sect

[default_sect]
activate = 1

pkcs11-provider + yubihsm_pkcs11.so will crash in deinit. The reason is that OpenSSL deinitializes libcrypto

before calling OSSL_PROVIDER_unload to deinit yubihsm_pkcs11, which causes use-after-free and double-free.

Workaround is to not deinit yubihsm_pkcs11, the downside is that we rely on sessions being closed by a timeout

in the HSM. There are 16 session slots and keepalive timeout set to 30 seconds. $( openssl pkeyutl -sign )

uses 2 sessions per signature, which gives uniform signing bandwidth of approximately 30 signatures per minute.

[pkcs11_sect]
module = /usr/lib/x86_64-linux-gnu/ossl-modules/pkcs11.so
pkcs11-module-path = /usr/lib/x86_64-linux-gnu/pkcs11/yubihsm_pkcs11.so
pkcs11-module-token-pin = ${ENV::CENSORED}
pkcs11-module-quirks = no-deinit
activate = 1