Unable to sign using a secured certificate

[gillg]

I currently use an EV cert hosted on Azure KeyVault, and I can't use this plugin to use it.
I'm not 100% sure if it's because the key is "unexportable" (mandatory by the EV nature) or if it's just the internal method key.CertificateChain which is just not generic.
This method seems retrive the public certificate chain in a secret... I don't understand why ?

In parallel I tried to use az keyvault certificate download --encoding PEM to store the leaf cert in notation cert add but it seems useless.

In my opinion, to get the full cert chain, we should not rely on Azure KeyVaut. KeyVault should just export the public leaf cert part, then the plugin should check the notation certs visible in notation cert ls, and fallback on the OS cert store, to find the issuers chained.

Exemple of current error :

notation sign --key My-Key-Name docker.io/library/golang:1.16
Warning: Always sign the artifact using digest(`@sha256:...`) rather than a tag(`:1.16`) because tags are mutable and a tag reference can point to a different artifact than the one signed.
Resolved artifact tag `1.16` to digest `sha256:5f6a4662de3efc6d6bb812d02e9de3d8698eea16b8eb7281f03e6f3e8383018e` before signing.
Error: certificate-chain is invalid, crypto/rsa: verification error

By the way there is a lack of logging in this module in case of error.
Adding some debug traces in case of error, using the standard flag --debug could be useful.
Example : notation sign --key My-Key-Name docker.io/library/golang:1.16 --debug

[gillg]

My bad, making a test I effectively discover the AKV trick, where the public cert data is stored as secret but hidden in the console (I thought it was something custom in the plugin) ! So I understand better the code in the plugin, that makes sense, except the fact when you make a CSR + upload the signed cert you bought I guess most of the time you don't have the fullchain cert. I'm not even sure if we can upload a pem file with multiple certs.
So I'm wondering the best way to handle it.

1. Is it possible to "update" the hidden secret to add a fullchain PEM once the CSR has already been updated ?
2. Is it possible to store the chain in notation local certs (signingAuthority should contains all intermidiates, and ca all the CAs ?) ?
3. Having a fallback mecanism to the OS certstore could make sense... no ?

Probably points 2. and 3. are more related to notation tool itself.

[yizha1]

Hi @gillg , Thanks for using notation and notation AKV plugin. Here are answers to some of your questions:

• --debug and --verbose flags are supported by notation sign and notation verify command since notation v1.0.0-rc.1. You can try it to get more logs for troubleshooting.
• Notation doesn't support working with OS certstore for some reasons, so user needs to configure notation trust store and trust policy before verification, maybe you already read this document
• Check out the certificate requirements, it will help you have a better understanding on certificates.

We also want to understand more about your issue. Could you elaborate on how you create the certificate in Azure key vault? Any guide you follow?

You are also welcomed to join the slack channel of Notary-v2, here is the link https://cloud-native.slack.com/archives/CQUH8U287. Let me know if you meet any issues of joining the slack channel.

[gillg]

Thank you @yizha1 !
I joined the chan :)

Indeed I already used --debug like I said in my exemple, but the point is all the logs are coming from notary-core. The AKV plugin seems not adding additional specific informations. Some could be useful.

I also used the doc to configure a trustpolicy but it's not better. As I understood the trustpolicy use local certs to compare your cert chain, but it can't "complete" your missing chain (like the OS or the browsers are doing for regular HTTPS).

In my use case:

• I generate a cert on AKV as type "RSA-HSM" signed by CA,
• Then I export the CSR, then I upload the CSR to a CA where I buy an "ExtendedValidation" cert.
• They return me my final signed cert, I merge it in AKV on the pending cert
  => First problem here, the cert they gave me is just the leaf, not the fullchain with CA. So the command akv get secret (https://github.com/Azure/notation-azure-kv/blob/main/internal/cloud/keyvault.go#L175-L180) wil not return the full chain, but just the leaf.

Now if I want to update it I have to create an new CSR and pay for a new cert... I would else be able to update the AKV secret (seems not possible because it's a "managed" secret). Else I was suggesting to let "notation" check in the OS cert store if it can find the assiciated cert chain as fallback instead of asking AKV.

[yizha1]

Thanks @gillg for providing more information.

We will take a look and get back to you.

[yizha1]

In my use case:

• I generate a cert on AKV as type "RSA-HSM" signed by CA,
• Then I export the CSR, then I upload the CSR to a CA where I buy an "ExtendedValidation" cert.
• They return me my final signed cert, I merge it in AKV on the pending cert
  => First problem here, the cert they gave me is just the leaf, not the fullchain with CA. So the command akv get secret (https://github.com/Azure/notation-azure-kv/blob/main/internal/cloud/keyvault.go#L175-L180) wil not return the full chain, but just the leaf.

Now if I want to update it I have to create an new CSR and pay for a new cert... I would else be able to update the AKV secret (seems not possible because it's a "managed" secret). Else I was suggesting to let "notation" check in the OS cert store if it can find the assiciated cert chain as fallback instead of asking AKV.

The issue you met was during notation sign, the trust store/policy won't help since it is used for validating the signatures.

According to certificate requirement, valid certificate chain should be provided for notation sign. We can reproduce the issue based on your case description, that it is only leaf certificate user can get, and it's not possible for user to build a certificate chain as well. To solve this issue, we will create a new requirement on notation AKV plugin, here is a brief description about the solution:

1. Introduce plugin parameters that users can use these parameters to configure root or Intermediate CAs
2. AKV plugin will firstly check the parameters to see whether user provides CAs, if yes, plugin will create certificate chain accordingly. //End
3. If no, plugin will try to build the certificate chain from the leaf certificate and OS cert store, if succeeds //End
4. If it fails, plugin will try to retrieve the certificate chain by checking the secrets (current behavior), if succeeds //End
5. If it still fails, then notation sign fails. //End

Any comments or suggestions are welcome.

Do you have any urgency in solving this issue? It will help if you can share more background on your use case and the problem you want to solve. Feel free to provide comments here or ping me from slack Yi Zha

Thanks/Yi

[yizha1]

This issue will be addressed by a feature request on #58

[yizha1]

@gillg we have implemented #58, and will release a new version of AKV plugin after verification. Stay tuned.

[yizha1]

@gillg could you try the latest release v0.6.0,
Now notation uses get-certificate to fetch the leaf certificate, and the ca certificates need to be passed via notation sign --plugin-config flag. Here is the guide.

[gillg]

It's a lot more better !

Now I have another error but I guess related to notation itself.

Resolved artifact tag `0.46.0` to digest `sha256:7b2970a29613cbe9dcd2f734123cc271a720afba105448ef38d9f97a2e568895` before signing.
Error: failed to push manifest: PUT "https://registry-1.docker.io/v2/xxxxxxx/xxxxxxxxxxx/manifests/sha256:40935fbb18185e4b67ddc688c64a5ac67a2c29175b358869c46d7671ca539881": response status code 404: notfound: not found

[yizha1]

@gillg which version of notation CLI are you using? Currently docker hub is not compatible with notary signature.

Here are the registries compatible with Notary:

• Azure Container Registry
• ORAS Distribution Registry
• Zot registry
• GitHub Container Registry
• Amazon ECR

[gillg]

Ok good to know ! I missed this information initially.
That definitely explain the 404 error...

I will think about next steps if we want to implement signature on my company. Are you aware about a potential dockerhub support?

[yizha1]

Ok good to know ! I missed this information initially. That definitely explain the 404 error...

I will think about next steps if we want to implement signature on my company. Are you aware about a potential dockerhub support?

I don't have the timeline that when dockerhub will support it. It could depend on when OCI 1.1 spec is released.

[gillg]

In the meantime I saw "docker trust" and they explained it's based on notary server.
What is the difference with the notation cli ?

[yizha1]

The notary server mentioned in "docker trust" referred to sub-project notary under notary project, see https://github.com/notaryproject/notary. Notation CLI is another sub-project under notary project, providing new experience of signing and verifying. They are different.