RSA key export by the OpenSSL provider seems incorrect.

[reds71]

Hi,

By trying to encrypt/decrypt data using the OpenSSL API like we do with the TPM2 OpenSSL provider, which is making the encryption that uses public key by the default OpenSSL provider and the decryption by the Trust M one.
It did not work and from times to time OpenSSL detected a defect in the exported public key. By introducing several printf() we could see that the key "stream" order was reversed in the export. It appears that some OpenSSL API do endianness wizardry which OSSL_PARAM_construct_BN() does not. We implemented the following to make things work:

@@ -492,6 +493,7 @@ int trustm_rsa_keymgmt_export(void *keydata, int selection, OSSL_CALLBACK *param
     trustm_rsa_key_t *trustm_rsa_key = keydata;
     uint32_t exponent;
     int ok = 1;
+    void * key_data = NULL;
 
     OSSL_PARAM params[3];
     OSSL_PARAM *p = params;
@@ -499,18 +501,38 @@ int trustm_rsa_keymgmt_export(void *keydata, int selection, OSSL_CALLBACK *param
     if (trustm_rsa_key == NULL)
         return 0;
 
     if (selection & OSSL_KEYMGMT_SELECT_PUBLIC_KEY) 
     {
-        *p++ = OSSL_PARAM_construct_BN(OSSL_PKEY_PARAM_RSA_N,
-                                        trustm_rsa_key->modulus,
-                                        trustm_rsa_key->modulus_length);
+        BIGNUM * bn = BN_bin2bn(trustm_rsa_key->modulus, trustm_rsa_key->modulus_length, NULL);
+        if (!bn) {
+            return 0;
+        }
+        p->key = OSSL_PKEY_PARAM_RSA_N;
+        p->data_type = OSSL_PARAM_UNSIGNED_INTEGER;
+        key_data = OPENSSL_malloc(trustm_rsa_key->modulus_length);
+        if (!key_data) {
+            BN_free(bn);
+            return 0;
+        }
+        p->data = keydata;
+        p->data_size = trustm_rsa_key->modulus_length;
+        if (!OSSL_PARAM_set_BN(p++, bn))
+        {
+            BN_free(bn);
+            OPENSSL_free(key_data);
+            return 0;
+        }
+        printf("RSA key modulus in exported key: %s\n", BN_bn2hex(bn));
+        BN_free(bn);
         exponent = 0x10001;
-        *p = OSSL_PARAM_construct_BN(OSSL_PKEY_PARAM_RSA_E, (unsigned char*)&exponent, sizeof(exponent));
+        *p++ = OSSL_PARAM_construct_BN(OSSL_PKEY_PARAM_RSA_E, (unsigned char*)&exponent, sizeof(exponent));
     }
     *p = OSSL_PARAM_construct_end();
 
     ok = param_cb(params, cbarg);
 
+    OPENSSL_free(key_data);
     return ok;
 }

Notes:

• The printf() in the patch was one among those used to detect the issue and can be removed, of course.
• The TPM2 provider keymgmt does a similar thing, but using custom means.
• Did not check if EC has the same issue, but it is likely.