Ability to sign certificate via configurable hook / external HSM

[alfonsrv]

Could it be a feature request to sign certificates not via private keys saved on the file system, but by delegating signing to another function that can be specified in settings.py (e.g. external.services.sign) that takes all the required arguments and returns the signed certificate?

Main reason being so signing can happen via a HSM (e.g. YubiHSM) instead of having to rely on locally saved private key files – which even if they are encrypted just don't feel as save as delegating it to a dedicated HSM.

[mathiasertl]

Hi @alfonsrv (and presumably others that in the future will request this),

Yes, it absolutely could be, and I'd like to support it. The major problem with it is that I simply don't have a HSM and I could not test it in anyway. So any support for HSM at present could not be tested in any meaningful scenario. If someone is willing to collaborate - sure. If someone provides me with a HSM, even better. But before that, I'm reluctant to implement anything in that direction. I don't want to promise a feature that in reality is not tested at all.

kr, Mat

[alfonsrv]

I don't think having a HSM is necessary, since implementation would likely be done over another docker service e.g. Yubico's YubiHSM Python library that provides a web server to send signing requests to.

So providing a generic interface would be possible – that could either send the data to the YubiHSM (or any other HSM) – while also enabling bootstrapping extra functionality to the signing-process e.g. adding additional extensions, checking if the subject is allowed to be issued, etc.

[kushaldas]

I started working literally this week to add HSM support django-ca. I need to get a draft PR ready where we can discuss various parts about how does it work.

I am first cleaning up https://github.com/SUNET/python_x509_pkcs11 to be a bit maintainable code base and then will use the same for HSM support here.

[mathiasertl]

@alfonsrv that library gives no documentation on how to actually create a signed certificate - let alone with a CSR object from python cryptography.

that takes all the required arguments

... and what would that be? From the library, I simply cannot tell. :-(. If I get the information, I might think of at least abstracting key handling away.

kr, Mat

[mathiasertl]

Hi @alfonsrv and @kushaldas,

In the past days, I played around with and read the code of python-yubihsm, python-pkcs11 and python_x509_pkcs11 library. A few observations and conclusions:

• python-yubihsm lacks documentation. For example, it shows how to generate a key, but I could not figure out how to use it in a later session.
• python-yubihsm requires a physical YubiHSM. Even running the test suite requires it!
• python-pkcs11 is in need of a new maintainer, see New release needed  pyauth/python-pkcs11#169 - and it's unclear if it supports Python 3.11 and 3.12 at this point.
• python-pkcs11 has frustratingly incomplete/untested documentation. For example, most (all?) examples in the README simply do not work, as generating a key pair requires an R/W session and that parameter is not passed.
• It seems python_x509_pkcs11 has a lot figured out of what we would need for proper supporting the PKCS11 interface. BUT it is an async library and django-ca is not, so it could not be used out of the box.

From that, I would draw the following conclusions:

1. Fully integrating YubiHSM support is only possible with an actual YubiHSM key. I'm not willing to pay for one on my own, I have to admit. If somebody donates one, I would work on it.
2. Using softhsm2, python-pkcs11 and python_x509_pkcs11, I think it would be possible for me to properly support this. But I think would require the maintainers (@kushaldas, is that you?) explicit confirmation that I'm allowed to integrate parts of the code in django-ca under the GPLv3 (I'm not a license geek at all, I really don't know what is required here).

Bottom line: Supporting different key storage interfaces is definitely possible, but would require a bit of refactoring. If implemented, it would provide a generic interface similar to how Django supports different databases or caches, so it would allow (in theory) @alfonsrv to implement YubiHSM support in a separate project, while PKCS11 support is included in django-ca.

Have a good weekend everybody!

kr, Mat

[alfonsrv]

A generic interface could look like this:

• django_ca.profiles:361 – before return builder.sign(private_key=ca.key(password), algorithm=algorithm) is executed, the presence of a PRE_SIGN_HOOK could be checked (function = get_hook("DJANGO_CA_PRE_SIGN_HOOK") ) and executed if valid+configured
• The pre-sign hook takes the builder, and optional algorithm as kwargs
• If the pre-sign hook returns a valid x509.Certificate the function returns, else the regular aforementioned logic (return builder...) is executed instead

Issue with this approach being that it's not JSON-serializable, thus cannot easily be passed to another service, such as an external HSM / some other HTTP-reachable service.

Similarly hooks for OCSP signing + CRL creation – now that I type it out, it seems like quite some overhead, but allows for only having some CA keys delegated to the HSM.

[mathiasertl]

Hi @alfonsrv ,

First, a general update:

@kushaldas made significant progress on signing certificates via the PKCS11 interface with cryptography (with more help from the cryptography maintainers - thanks!). He has a proof-of-concept branch demonstrating it (@kushaldas , maybe you can link it?).

I myself worked a lot on building on Kushal's work and generalizing it. See the linked branch! My current approach is as follows:

• Configuration has a CA_KEY_BACKENDS setting, very similar to Django's STORAGES or DATABASES setting.
• Users can add/remove backends as they see fit.
• Custom backends can be implemented by implementing an abstract base class.
• this interface also allows for command line integration.
• if implemented correctly, configuring a custom backend in CA_KEY_BACKENDS is sufficient for making django-ca use it.

About your idea: it's generally a valid idea and my approach isn't much different in the end. Let me explain.

The problem is that the private key is used in a variety of places. For example:

• Signing certificates (what you mentioned in profiles).
• signing certificates via API (profiles are not used there)
• signing intermediate CAs.
• signing CRLs.
• signing OCSP responder certificates.

In addition, parameters differ based on implementations (password and path for filesystem, key slot and label for HSMs, ...), so we need command line integration. Now that I think of it, also in the API and admin interface.

It would mean there's lots of hooks you'd have to implement and configure for a fluent user experience.

In the end, a backend works just the same. You still have to implement everything (but with the advantage of subclass checks), but you only have to configure the path in a setting.

I'll invest more time in my branch this week. I hope to get it in good condition by Sunday. The basic concept is proven to work, but tests are not adapted. In the meantime, you're of course also welcome to fork and work on your idea, if you want.

Kr, Mat

[mathiasertl]

I just merged the first version. This is pretty sophisticated and well tested already and should allow you to implement this with a subclass and a few pydantic models. Documentation is here:

https://django-ca.readthedocs.io/en/latest/python/key_backends.html

[alfonsrv]

Great! I'll try to have a look into implementing a YubiHSM prototype backend over easter.

[mathiasertl]

Would be super cool. I also plan to release that weekend, by the way. If there's something that needs to be changed in the key backend interface, I'm open to it.

[alfonsrv]

Looking at it, there seems to be quite a lot of moving parts – cryptography doesn't support it by default, but requires the OpenSSL used underneath to utilize a PKCS11 engine (pyca/cryptography#4967). YubiHSM seems to use libp11 (https://docs.yubico.com/hardware/yubihsm-2/hsm-2-user-guide/hsm2-openssl-libp11.html) library to provide the OpenSSL engine. (might be wrong here)

While this seems managable, I'm uncertain if implementing it as a straight-forward YubiHSM-backend yields too many security benefits (other than preventing key extraction – which is obv pretty huge by itself), as all signing logic and HSM authentication passwords would still have to reside in the publicly facing Docker container as part of the storage configuration, allowing attackers to circumvent any kind of auditing and restrictions implemented on an application / Python-level, making signing go just as unnoticed as in the current situation.

A dedicated, air-gapped container would probably be desirable that processes "commands" sent to it – like "sign certificate with pk 2", "create a CRL", … and so on. Especially since data has to be marshalled anyways in order to be sign-able, should the PKCS11 engine approach not work (see https://github.com/reaperhulk/vault-signing/blob/main/src/vaultsigning/key.py#L59-L66 + wbond/asn1crypto#6)

Also will have to see if there's a way to only make cryptography use the engine for certain operations (e.g. django-ca) instead of all of them.

[mathiasertl]

Hi @alfonsrv ,

A dedicated, air-gapped container would probably be desirable that processes "commands" sent to it – like "sign certificate with pk 2", "create a CRL", … and so on. Especially since data has to be marshalled anyways in order to be sign-able, should the PKCS11 engine approach not work (see https://github.com/reaperhulk/vault-signing/blob/main/src/vaultsigning/key.py#L59-L66 + wbond/asn1crypto#6)

Well... I have good news for you - django-ca already has this. It supports Celery to do just that. In this mode of operation, the webserver process has no access to the private key, instead commands are sent via the broker (Redis in the examples, but could be any MQTT broker as well) to a Celery worker. The tutorials for source installation and the Docker Compose setup already include Celery.

Both processes can have different configurations, e.g. for example passwords could only be present on the Celery container.

Note however that as long as you have ACME (or the API) at the front, or want to automatically sign CRLs, you will need all configuration to sign something with the CA somewhere.

[mathiasertl]

@alfonsrv , since I'll start working on @kushaldas branch, wondering if you could provide some input: What are the parameters available when generating a private key? And which would be required for using a private key for signing?

[alfonsrv]

Sorry for the late reply, the email must have slipped my attention.

Creating a key and signing is quite a multi-layered process. Generally the Yubico YubiHSM documentation is quite good, but here's what's required.

Prior to usage, users first have to setup their YubiHSM with one or more authentication key, that limits the scope of operations (signing, generating + exporting keys, deleting keys, reviewing audit logs, ...) to a specific domain (effectively a cluster of private keys):

• One "MASTER" authentication key to do everything (yubihsm> put authkey 0 0 "MASTER" all all all)
• One authentication key per CA / clusters of CAs (depending on sensitivity) to sign with (yubihsm> put authkey 0 102 "AuthKey (DOMAIN 3)" 3 generate-asymmetric-key:sign-pkcs:sign-pss sign-pkcs:sign-pss)

├── Root CA (auth key 1)
│   ├── Intermediate CA Identities (auth key 2)
│   │   └── Intermediate CA Identity 1 (auth key 4)
│   │   └── Intermediate CA Identity 2 (auth key 4)
│   │   └── Intermediate CA Identity 3 (auth key 4)
│   ├── Intermediate CA Web (auth key 3)
│   │   └── Intermediate CA Web-A 3 (auth key 5)
│   │   └── Intermediate CA Web-B 3 (auth key 5)
│   │   └── Intermediate CA Web-C 3 (auth key 5)

All operations require to be run in sessions, which in turn require to specify an authentication key. The authentication key basically scopes each session. Using the authentication key requires the password of that authentication key for usage. (using the CLI: yubihsm> session open <authkey_id>)

---

Afterwards keys can be generated using the MASTER authentication key, or other authentication keys that were created for creating asymmetric key pairs (yubihsm> generate asymmetric 1 100 "Intermediate CA Identity 3" 3 exportable-under-wrap,sign-pkcs rsa4096) ref.
Alternatively already-existing keys can be imported using the CLI (yubihsm> put asymmetric 1 100 "Intermediate CA Identity 3" 3 sign-pkcs private.key)

---

Finally, signing works by specifying the desired asymmetric key ID + what should be signed
sign pkcs1v1_5 1 100 rsa-pkcs1-sha256 request.csr ref

From my understanding, an OpenSSL integration should be available that makes signing of the CertificateSigningRequestBuilder objects easier, since serializing/exporting them to a file is not possible afaik (I think it's this one pyca/cryptography#4967 or another of the issues mentioned above).

For all of the CLI commands, the official Python wrapper provides the same functionality + syntax.

---

Given the complexity of initial setup and management outlined above, I think it's best if people setup the HSM via CLI and then just use Django CA for signing. This also avoids possible DoS attacks by overwriting already-existing private keys that have not been exported / backed up. The only keys that I think should be generated on-device are the OCSP keys.

I hope I could outline the process clearly enough and in an understandable way.

[mathiasertl]

Hi @alfonsrv ,

#148 adds support for storing keys in HSMs via PKCS11. Would be cool if you can have a look.

Does YubiHSM support that protocol as well?

kr, Mat