Crypto interface design (migrated from very old issue on original repository)

[zanebeckwith]

Originally posted by @drbild over a year ago to the original repository.

Though this is significantly out of date, I think the principles are still relevant, and wanted to retain the discussion.

---

Crypto interface design
This is a brain-dump for now of my thoughts about the crypto interface design. We can refine it into a better description & discussion over time.

Implications of Design Goals for XTT Implementation
No dynamic memory allocation required.
Support arbitrary implementations of the crypto primitives.
Allow asynchronous crypto primitives for non-blocking use of HSMs.
Be as performant (nearly) as a monolithic crypto lib.
These four goals alone pose some interesting incompatibilities. Consider the following possible implementations.

Shared header w/ arbitrary linked implementation
No dynamic memory allocation implies that all struct definitions must be visible to the calling code. The shared header thus implies that all implementations must share the same struct definitions. Of course, the different possible implementations for linking will use different underlying representations for primitives like keys. Thus, the representations must be translated at the interface boundary for every call. And that is not performant.

The translation could be done (at the cost of some cycles) were the data types always representing the same concept, e.g., different representations of an ed25519 key. The interface would define a common serialization format to translate to and from. But consider an implementation backed by an HSM. The private key would not actually be a key, but instead some opaque identifier for the key inside the HSM. Finding a clean, common representation seems difficult.

An alternative is for the interface to take pointers to opaque structs defined by the implementation. But this requires dynamic memory allocation, as those opaque structs would have to be allocated by the linked implementation code, where the full definition is known.

Shared header for functions, implementation-specific header for data types
This approach would solve the static allocation issue. The XTT library would have to be compiled against a specific implementation defining the structs. (The shared header defining the methods would just have forward declarations.)

I'm leaning towards this approach currently. My general feeling is that the crypto implementation should be statically linked to the XTT library anyway (not like openSSL with a shared lib for the crypto). So this keep everything clean.

Polymorphism - the Compile/Runtime, Global/Local matrix
The "arbitrary implementation" design goal is still ill-defined. Should the implementation be chosen at compile time or be configurable at run-time? Should it be common to the whole program or changeable for each XTT "session"?

The two approaches in the prior section are in the (compile-time, global) quadrant of the design space.

Supporting runtime selection or local per-session selection of primitives seems to require either dynamic memory allocation or a shared/translatable representation of the data structures, which are non starters.

Proposed Approach
XTT provides a shared header that forward declares the structs and methods used to access crypto primitives. The user is responsible for compiling XTT against an implementation that defines those structs and methods.

We provide two (or more) reference implementations that delegate to different libraries (say mbedTLS and wolfSSL).

These two implementations are in the (compile-time, global) quadrant above. They are applied globally to all XTT sessions within the program.

Any user is free to provide their own implementation instead if they don't like our chosen point in the design space. Such an implementation could

mix and match from various backends
support linking-based selection of the backend by implementing a translation layer (like the first approach listed above)
support runtime polymorphism at the cost of doing dynamic memory allocation from the library.
The point is that our interface doesn't preclude users from doing something more.

The one concession made to support this flexibility is the method for function dispatch. Rather than access the methods forward declared in the shared header directly by name, they are accessed via a vtable in the XTT context objects.

Implementation
I'll do a mock-up of this approach from the existing crypto lib to see how it looks and feels.

[zanebeckwith]

Original comment from over a year ago:

---

I definitely support the "compile-time/global" quadrant of that decision-space.

If the crypto primitives are a separately-compiled, statically-linked-in library, users are free to mix-and-match implementations of each of the functions, right? E.g. the implementation of aes256-gcm can be from Intel Performance Primitives, chacha20-poly1305 from libsodium, etc. I don't see any use-case for supporting different implementations for a single primitive, though (e.g. two different implementations of aes256-gcm).

[zanebeckwith]

Original comment from over a year ago:

---

I like the shared-header-for-functions/implementation-header-for-structs approach.

So, are you envisioning something like this:

XTT has a header declaring the crypto primitive functions we need:
struct ed25519_pub_key;
struct ed25519_priv_key;
int generate_ed25519_keypair(struct ed25519_pub_key *pub, struct ed25519_priv_key priv);
XTT's implementation files include some kind of preprocessor check for a definition of the struct-implementation header and include that:
/ in context.c /
#ifdef XTT_CRYPTO_STRUCT_HEADER
#include XTT_CRYPTO_STRUCT_HEADER
#else
#include <xtt_default_crypto_struct.h>
#endif
/ somewhere actually create a ed25519 key pair and pass it to generate_ed25519_keypair */
The user (really, us, for libsodium, wolfcrypto, etc) writes the struct-implementation header, which most likely is a bunch of struct wrappings of the actually implementation (they can't just typedef, since we're forward-declaring them in the shared header, right?)
typedef struct ed25519_pub_key { libsodiums_ed25519_pub_key_struct info; } ed25519_pub_key;
Finally, the user creates an implementation for the functions declared in the shared header:
#include <xtt_shared_crypto.h>
#include XTT_CRYPTO_STRUCT_HEADER
#include <sodium.h>
int generate_ed25519_keypair(struct ed25519_pub_key *pub, struct ed25519_priv_key *priv)
{
libsodium_ed25519_keypair(&pub->info, &priv->info);
}

[zanebeckwith]

Original comment from over a year ago:

---

I don't think I quite see the use-case for the dynamic-polymorphism, or understand how that would work. What's the criterion for choosing a particular function pointer for a particular primitive? E.g. why have multiple choices for aes256-gcm?

Supporting this, as you say, requires us to use function pointers, so does that mean the "shared" header would just typedef function pointers, rather than declaring functions? And we provide a setter function for the user to set which functions to use?

[zanebeckwith]

Original comment from drbild, from over a year ago:

---

I'm envisioning something like this. See the bottom file in the post for how the user picks an implementation - it's just one #include.

xtt/crypto/ed25519.h
// Forward declarations that allow the XTT code to pass around
// pointers to these things, without knowing their precise definitions.
struct xtt_ed25519_public_key;
struct xtt_ed25519_private_key;
struct xtt_ed25519_signature;

#define XTT_ED25519_SIGNATURE_BYTES_LEN = 32;
#define XTT_ED25519_PUBLIC_KEY_BYTES_LEN = 32;

int xtt_ed25519_public_key_init(xtt_ed25519_public_key* public_key);
int xtt_ed25519_public_key_free(xtt_ed25519_public_key* public_key);
int xtt_ed25519_public_key_verify(const struct ed25519_public_key* public_key,
const uint8_t* message, const size_t message_len,
const struct xtt_ed25519_signature* signature);
int xtt_ed25519_public_key_to_bytes(const struct ed25519_public_key* public_key,
uint8_t bytes, const size_t bytes_len);
int xtt_ed25519_public_key_from_bytes(struct ed25519_public_key* public_key,
const uint8_t bytes, const size_t bytes_len);

int xtt_ed25519_private_key_init(xtt_ed25519_private_key* private_key);
int xtt_ed25519_private_key_free(xtt_ed25519_private_key* private_key);
int xtt_ed25519_private_key_sign(const xtt_ed25519_private_key* private_key,
const uint8_t* message, const size_t message_len,
struct xtt_ed25519_signature* signature);
int xtt_ed25519_private_key_get_public_key(const xtt_ed25519_private_key* private_key,
xtt_ed25519_public_key* public_key);

int xtt_ed25519_signature_init(xtt_ed25519_signature* signature);
int xtt_ed25519_signature_free(xtt_ed25519_signature* signature);
int xtt_ed25519_signature_to_bytes(const xtt_ed25519_signature* signature,
uint8_t bytes, size_t bytes_len);
int xtt_ed25519_signature_from_bytes(struct xtt_ed25519_signature* signature,
const uint8_t bytes, const size_t bytes_len);
xtt/crypto/impl/mbedtls.h
#include "mbedtls/ecdsa.h"

#include "xtt/crypto/ed25519.h"

// Put a definition on the forward declaration. A user
// simply #includes this file to pick the mbedTLS implementation.
struct xtt_ed25519_public_key {
mbedtls_ecp_group grp;
mbedtls_ecp_point Q;
};

struct xtt_ed25519_private_key {
mbedtls_ecp_group grp;
mbedtls_mpi d;
mbedtls_ecp_point Q;
};

struct xtt_ed25519_signature {
mbedtls_mpi r;
mbedtls_mpi s;
}
xtt/handshake/context.h
// The xtt code just needs the forward declarations. It only
// refers to pointers to the shared structs, so can be
// compiled without knowing their definitions.
#include "xtt/crypto/ecdsa.h"

struct xtt_handshake_context {
xtt_ed25519_private_key ephemeral_self;
xtt_ed25519_public_key ephemeral_peer;
xtt_ed25519_signature tmp_signature;
}

int xtt_handshake_context_init(xtt_handshake_context* context);
int xtt_handshake_context_free(xtt_handshake_context* context);

int xtt_handshake_context_validate_peer_signature(xtt_handshake_context* context,
const uint8_t* signature, const size_t signature_len,
const uint8_t* message, const size_t message_len);
xtt/handshake/context.c
#include "xtt/handshake/context.h"

int xtt_handshake_context_init(xtt_handshake_context* context)
{
xtt_ecdsa_private_key_init(&context->ephemeral_self);
xtt_ecdsa_public_key_init(&context->ephemeral_peer);
xtt_ecdsa_signature_init(&context->peer_hello_signature);
return 0;
}

int xtt_handshake_context_init(xtt_handshake_context* context)
{
xtt_ecdsa_private_key_free(&context->ephemeral_self);
xtt_ecdsa_public_key_free(&context->ephemeral_peer);
xtt_ecdsa_signature_free(&context->peer_hello_signature);
return 0;
}

int xtt_handshake_context_validate_peer_signature(xtt_handshake_context* context,
const uint8_t* signature, const size_t signature_len,
const uint8_t* message, const size_t message_len)
{
xtt_ecdsa_signature_from_bytes(&context->tmp_signature, signature, signature_len);
return xtt_ecdsa_public_key_verify(&context->ephemeral_peer, message, message_len,
&context->tmp_signature);
}
user/iotdevice.c
#include "xtt/handshake/context.h"

// User chooses the implementation, mbedTLS in this case, via this include.
#include "xtt/crypto/impl/mbedtls.h"

void main()
{
xtt_handshake_context context;

xtt_handshake_context_init(&context);

// do XTT stuff, e.g.:
xtt_handshake_context_validate_peer_signature(context, ...);
// In reality, lower-level stuff like validating signatures will be done in the xtt handshake code, and the call here will be something like xtt_perform_handshake(...).

xtt_handshake_context_free(&context);

}

[zanebeckwith]

Original comment from over a year ago

---

OK, yea, I think this is the same as what I was trying to say. The only difference being that I thought our implementations would have to explicitly include the implementation header (hence the preprocessor checks). But just including it in the top level works better.

[zanebeckwith]

Original comment from drbild, from over a year ago:

---

You must get notification for new comments! I need to figure out how to enable that. Didn't see yours for a while today.

My scheme above doesn't work as is, because the compiler needs to know the definitions when processing context.h to lay out the xtt_handshake_context struct. There should be someway to patch that, since the definitions are only need to calculate offsets in the struct (if context were a struct of pointers, it would work).

[zanebeckwith]

Original comment from drbild, from over a year ago:

---

A few potential ways to fix this:

External getters
Split context.h for internal and external use. Declare externed getters for the internal code to use and give the actual definitions to be #includeed into the application code. This means that the application code can only #include that header once, though.

Function pointer getters
Add function pointers to getters to the context. Requires some fiddling and maybe some boilerplate in the application code to initialize those pointers.

struct xtt_handshake_context {
xtt_ecdsa_public_key* (ephemeral_peer*)();
xtt_ecdsa_signature* (tmp_signature*)();
}
Header only library
Make the xtt handshake code a header-only library. Essentially, we're trying to templatize the xtt handshake code over the crypto primitives. Drawing inspiration from C++ templates, put all that code in headers.

I don't have a strong opinion yet, although the header-only approach seems the least icky so far.

[zanebeckwith]

Original comment from drbild, from over a year ago:

---

Regarding the runtime polymorphism, my use case there was for a server side. Say we add an HSM at some point. During transition, new connections will need to use the HSM, but existing connections won't. Or maybe the HSM becomes an upsell - only some customers have paid the premium to have HSM-backed connections.

[zanebeckwith]

Original comment from over a year ago:

---

Looks like I had my global notification settings at "Mention" rather than "Participate". I've changed that, so hopefully should get notifications now.

How would you do header-only in c90, without inline?
Should just need static, not inline.

I thought the user would include the specific implementation header before any others, in the top level file. That's annoying to have a header ordering dependence, but would work, right?
xtt/handshake/context.c includes xtt/handshake/context.h, but doesn't include an implementation header. So the definition of xtt_handshake_context will fail.

Also, is there objection to using a preprocessor define like I'd mentioned previously?
I'd just prefer a code-only solution. Really I'd prefer to be able to compile the xtt code without knowing the specific implementation.

The preprocessor define method is roughly equivalent to the header-only approach. The xtt code just ends up in different translation units. For client code with a statically-linked xtt, the resulting executable should be nearly identical.

Ok, yea, I hadn't thought about the polymorphism from the servers perspective. I'll have to think about that...
Supporting runtime selection might make the case for the function pointer getters approach...