OpenSSL 1.1.1 partially working with pull request #73

[sporian]

Thanks to PR #73, I largely have libest working with OpenSSL 1.1.1c on Alpine Linux 3.10.2, as well as OpenSSL 1.0.2g on Ubuntu 16.04 LTS.

However, I cannot run example/server/estserver correctly under Alpine. When a client requests /csrattrs the server errors because it fails its sanity check of the attributes before returning them. Here is the relevant output from the estserver process:

***EST [INFO][parse_http_message:1243]--> request uri=/.well-known/est/csrattrs
***EST [INFO][handle_request:1358]--> /.well-known/est/csrattrs
***EST [INFO][est_asn1_sanity_test:1124]--> Sanity: tag=16, len=38, j=32, out_len=40
***EST [INFO][est_asn1_sanity_test:1124]--> Sanity: tag=6, len=7, j=0, out_len=38
***EST [INFO][est_asn1_sanity_test:1124]--> Sanity: tag=11, len=6, j=32, out_len=36
***EST [INFO][est_asn1_sanity_test:1124]--> Sanity: tag=9, len=42, j=128, out_len=28
***EST [ERROR][est_asn1_parse_attributes:1221]--> Invalid ASN1 encoded data. rv = 63 (EST_ERR_BAD_ASN1_HEX)

In comparison, on Ubuntu using OpenSSL 1.0.2.g:

***EST [INFO][parse_http_message:1243]--> request uri=/.well-known/est/csrattrs
***EST [INFO][handle_request:1358]--> /.well-known/est/csrattrs
***EST [INFO][est_asn1_sanity_test:1124]--> Sanity: tag=16, len=38, j=32, out_len=40
***EST [INFO][est_asn1_sanity_test:1124]--> Sanity: tag=6, len=7, j=0, out_len=38
***EST [INFO][est_asn1_sanity_test:1138]--> NID=0
***EST [INFO][est_asn1_sanity_test:1124]--> Sanity: tag=6, len=9, j=0, out_len=29
***EST [INFO][est_asn1_sanity_test:1138]--> NID=48
***EST [INFO][est_asn1_sanity_test:1124]--> Sanity: tag=6, len=5, j=0, out_len=18
***EST [INFO][est_asn1_sanity_test:1138]--> NID=715
***EST [INFO][est_asn1_sanity_test:1124]--> Sanity: tag=6, len=9, j=0, out_len=11
***EST [INFO][est_asn1_sanity_test:1138]--> NID=673
***EST [INFO][log_access:1396]--> 127.0.0.1 [12/Jun/2020:15:13:01 -0600] "GET /.well-known/est/csrattrs HTTP/1.1" -1 0

The first thing I don't understand is why the two systems seem to report a different set of CSR attributes via est_asn1_sanity_test(). It's the exact same code running on both systems, which includes a custom .cnf file and a script that builds all the certs used using the native openssl binary (so the certs aren't exactly the same but built on both platforms using the same commands). Here is the script that starts estserver:

export EST_CACERTS_RESP=estserver.cacerts
export EST_TRUSTED_CERTS=estserver.certs
export EST_OPENSSL_CACONFIG=fiinca/openssl-fiin.cnf

ulimit -c unlimited # allow core dumps
../libest/example/server/estserver -v \
    -c fiinca/certs/pki1.fiin.com.pem -k fiinca/private/pki1.fiin.com.key -o \
    -r estrealm -p 8443 --keypass_arg fiin

The other problem I'm having is trying to decode what these CSR attributes are when running under Ubuntu. The NID is not reported. Code from src/est/est.c:

    while (out_len > 0) {
        j = ASN1_get_object(&string, &len, &tag, &xclass, out_len);

        EST_LOG_INFO("Sanity: tag=%d, len=%d, j=%d, out_len=%d", tag, len, j, out_len);
        if (j & 0x80) {
            return (EST_ERR_BAD_ASN1_HEX);
        }
        switch (tag)
        {
        case V_ASN1_OBJECT:
#if OPENSSL_VERSION_NUMBER < 0x10100000L
            a_object = c2i_ASN1_OBJECT(NULL, &string, len);
#else
            a_object = d2i_ASN1_OBJECT(NULL, &string, len);
#endif
            if (a_object != NULL) {
                nid = OBJ_obj2nid(a_object);
                EST_LOG_INFO("NID=%d", nid);

I'm assuming d2i_ASN1_OBJECT() is returning NULL, since tag is clearly V_ASN1_OBJECT (6). Looking at the code for d2i_ASN1_OBJECT, which calls ASN1_get_object(), it seems that the logical reason for returning NULL is that the encoded attribute is in the incorrect form.

I'm not too familiar with openssl or PKI. Any pointers on how to dig into this problem?

[sporian]

OK, so it appears to be a simple case of the two OpenSSL versions decoding the same ASN.1 data differently. Still looking for a good reference for decoding. Maybe I'm better off just walking through the OpenSSL code instead. Here's the DER(?) format bytes containing the CSR attributes, printed from an estserver run (after base64 decoding).

***EST [INFO][parse_http_message:1243]--> request uri=/.well-known/est/csrattrs
***EST [INFO][handle_request:1358]--> /.well-known/est/csrattrs

: 0000: 30 26 06 07 2b 06 01 01 01 01 16 06 09 2a 86 48
: 0010: 86 f7 0d 01 09 01 06 05 2b 81 04 00 22 06 09 60
: 0020: 86 48 01 65 03 04 02 02

[rpb5bnc]

This is what I was thinking.  I don't mean to point in other directions, but the issues you're hitting seem to centered heavily interactions with OpenSSL.  My concern here is that I haven't looked at PR #73, so I'm not 100% sure it is correct in this area of the EST code.  You mentioned you're not that familiar with the OpenSSL code.  If that's the case, then I'll say that the ASN.1 code is particularly difficult to follow.  For me, it's basically impossible to walk through that code since it's mostly compile time macros to create the code and run time registered vector functions.  I always make use of GDB when I have to go in and understand any of the OpenSSL ASN.1 code. We are now actively working to release a new version of libest that supports OpenSSL 1.1.1, in fact, that's why I didn't get to respond yesterday.  The last time we posted to this external repo was about 2.5 years ago and a lot has changed over that time, mainly new functionality has been added.  We're a fairly small team here and our focus has been adding these new capabilities which has not left much time to keep the external version in sync with the internal version.  Getting the latest version of code ready for release is taking some time because we need to sanitize it for external use and we need to make sure it's as compatible as the current external version.  We don't want to cause a lot of churn for external adopters.  It shouldn't be too much longer. inline,... On 6/18/20 10:33 AM, sporian wrote: OK, so it appears to be a simple case of the two OpenSSL versions decoding the same ASN.1 data differently. Still looking for a good reference for decoding. Maybe I'm better off just walking through the OpenSSL code instead. Here's the DER(?) format bytes containing the CSR attributes, printed from an estserver run (after base64 decoding). ***EST [INFO][parse_http_message:1243]--> request uri=/.well-known/est/csrattrs Yes, this is a DER based buffer.  Unfortunately, I can't really see anything obvious at this point.  Just to confirm, this is not from running the unit test code that is included with libest, and is instead a manual test using the estserver command line application, correct?  Again, I have not looked at the PR that added the support for 1.1.1, but I'm going to guess that it did not address the unit test code at all. As I continue to prepare the code for release, I'll take a look at the unit tests for /csrattrs and see if there's any difference between a 1.1.1 and a 1.0.2 run output. Pete ***EST [INFO][handle_request:1358]--> /.well-known/est/csrattrs : 0000: 30 26 06 07 2b 06 01 01 01 01 16 06 09 2a 86 48 : 0010: 86 f7 0d 01 09 01 06 05 2b 81 04 00 22 06 09 60 : 0020: 86 48 01 65 03 04 02 02 — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe. [ { "@context": "http://schema.org", "@type": "EmailMessage", "potentialAction": { "@type": "ViewAction", "target": "#78 (comment)", "url": "#78 (comment)", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { "@type": "Organization", "name": "GitHub", "url": "https://github.com" } } ]

[sporian]

[...] I'll say that the ASN.1 code is particularly difficult to follow. [...]
I always make use of GDB when I have to go in and understand any of the
OpenSSL ASN.1 code.

Thanks for the tip. I'll do the same.

Getting the latest version of code ready for release is taking some time [...]

No worries. I'm happy hacking a short term fix to keep my working proof of
concept going until the official release supporting 1.1.1.

Just to confirm, this is not from running the unit test code that is included
with libest, and is instead a manual test using the estserver command line
application, correct?

This is correct. #73 doesn't pass unit tests on my system.

As I continue to prepare the code for release, I'll take a look at the unit
tests for /csrattrs and see if there's any difference between a 1.1.1 and a
1.0.2 run output.

Thanks for your insights. I'll post here with anything interesting I learn.