RSA decrypt failed

[gilweis]

General Info:
$ pkcs11-tool --module yubihsm_pkcs11.so -I

Cryptoki version 2.40
Manufacturer Yubico (www.yubico.com)
Library YubiHSM PKCS#11 Library (ver 2.40)
Using slot 0 with a present token (0x0)

$ pkcs11-tool --module /home/g/work/yubihsm-shell/build/pkcs11/yubihsm_pkcs11.so -L

Available slots:
Slot 0 (0x0): YubiHSM Connector localhost
token label : YubiHSM
token manufacturer : Yubico (www.yubico.com)
token model : YubiHSM
token flags : login required, rng, token initialized, PIN initialized
hardware version : 2.20
firmware version : 2.20

Steps to reproduce the problem:
yubihsm> generate asymmetric 0 100 label_mytest 1,2,3 exportable-under-wrap,decrypt-pkcs,decrypt-oaep rsa2048

YHSM_MODULE="yubihsm_pkcs11.so"
$ pkcs11-tool --module "${YHSM_MODULE}" --pin 0001password --read-object --type pubkey --id 0064 -o rsa_key.pub
$ echo '0123456789' | openssl pkeyutl -encrypt -pubin
-inkey rsa_key.pub
-keyform der
-pkeyopt rsa_padding_mode:pkcs1
-pkeyopt rsa_mgf1_md:sha512
-out encrypted.data

$ pkcs11-tool --module "${YHSM_MODULE}" --decrypt
--pin 0001password
--input-file encrypted.data
--label label_mytest

Using slot 0 with a present token (0x0)
Using decrypt algorithm RSA-PKCS
error: PKCS11 function C_Decrypt failed: rv = CKR_DEVICE_ERROR (0x30)
Aborting.

yubihsm> decrypt pkcs1v1_5 0 0x0064 encrypted.data

Failed to decrypt data: Wrong length

[gilweis]

The same with openssl command & engine is working:
$ openssl pkeyutl -engine pkcs11 -keyform engine -decrypt -inkey "pkcs11:object=label_mytest;type=private;pin-value=0001password" -in encrypted.data

[qpernil]

This works for me, except I have to leave out the -pkeyopt rsa_mgf1_md:sha512 since it is not applicable to pkcs1. Also, to get pkeyutl to work with this command line I have to use openssl 1.1. I do get a similar error if I use a different public key that doesn't correspond to the private key. This is expected since the padding will be incorrect if the decryption fails. Since the removal of padding is done by the HSM that's where the error will happen. Add 'debug' to the yubihsm_pkcs11.conf file to see more info about the error.

[gilweis]

I'm using OpenSSL 3.0.5
I tried without -pkeyopt rsa_mgf1_md:sha512.
I'm getting :

Using slot 0 with a present token (0x0)
Using decrypt algorithm RSA-PKCS
error: PKCS11 function C_Decrypt failed: rv = CKR_DEVICE_ERROR (0x30)
Aborting.

[qpernil]

This could be caused by using unmatched public/private key, see explanation above. Can you confirm if that is what is occurring ?

[gilweis]

I wrote the steps to reproduce the problem.
I think that the private/public are matched.

[qpernil]

Again, this works for me just as you have written.. This time I can leave the -pkeyopt rsa_mgf1_md:sha512 and it still works. Im now using openssl 3.0.8 from homebrew.

[rukia2023]

فشل رجوع الملف لشكله الأصلي بعد فك تشفيره بخوارزمية RSA ما السبب لان اجد الملف الذي تم فكه محتواه مسار الملف الأصلي قبل التشفير

[qpernil]

If the file was not found for some reason then the path will be used as input instead, this might be what happened during encryption