challenge-response not implemented in the most secure way

[forrest96er]

When using LUKS with the default compiled-in key and passphrase parameters, the password length is limited to 512 bits. According to cryptsetup --help, the maximum keyfile size is 8192kB and the maximum interactive passphrase length is 512 characters.

Additionally, we must consider the possibility that the external hardware key may not be trustworthy. It could have security flaws or might store all input values.

Furthermore, the user always enters a password with a maximum length of 512 bits or longer.

Yubico limits the maximum input of its HMAC-SHA1 challenge-response to a 64-bit input value

Here is the proposed process, an idea of mine:

1. Hash the password using SHA-512, then hash the resulting output again using CRC64 to generate a 64-bit input value for the challenge-response.

2. The challenge-response mechanism returns a "6-10 digit HOTP as the response code" ( as written on Yubico Doc it only uses 6-10 digit Code for HOTP, but is there a differnt response for Challenge-response ?? .

3. Concatenate the password with the response code.

4. Hash the resulting value with SHA-512.

The resulting hash is now the final passphrase to be used for LUKS.

[Vincent43]

Hash the password using SHA-512, then hash the resulting output again using CRC64 to generate a 64-bit input value for the challenge-response.

It would be security be obscurity. There is dedicated SHA-512/256 algorithm for shorting SHA-512 or you can even just use half of SHA-512 but security difference is marginal and it's harder to use. See https://crypto.stackexchange.com/questions/3153/sha-256-vs-any-256-bits-of-sha-512-which-is-more-secure

The challenge-response mechanism returns a "6-10 digit HOTP as the response code" ( as written on Yubico Doc it only uses 6-10 digit Code for HOTP, but is there a differnt response for Challenge-response ?? .

The default yubikey response is HMAC-SHA1 (not to be confused with insecure plain SHA1). if it was initialized correctly There is option to get HOTP digit code instead but that's not for this usecase, see ykchalresp manual.