Performance degradation in Amazon Linux 2023 runtimes when loading SSL certificates

[mariojonke]

There is a noticeable degradation in performance regarding operations that load SLL certificates (like creating an https connection) in Lambda runtimes running on Amazon Linux 2023 in comparison to runtimes that run on Amazon Linux 2.

The issue can easily reproduced with the following python snippet:

import json
import http.client

def lambda_handler(event, context):
    http.client.HTTPSConnection("")

    return { 'statusCode': 200, 'body': json.dumps('Hello from Lambda!') }

Executing the function with python 3.12 runtime (128M of allocated memory) results in an execution time similar to

REPORT RequestId: 990ddd6f-406c-43cb-a4e4-78382f102b07	Duration: 917.49 ms	Billed Duration: 918 ms	Memory Size: 128 MB	Max Memory Used: 47 MB	Init Duration: 127.28 ms

and in comparison the same but running with the python 3.11 runtime

REPORT RequestId: 9992728e-fa11-422c-876c-c4319883b55b	Duration: 298.25 ms	Billed Duration: 299 ms	Memory Size: 128 MB	Max Memory Used: 43 MB	Init Duration: 142.99 ms

So creating an HTTPSConnection in Amazon Linux 2023 is ~2-3 times slower than on Amazon Linux 2.

---

To narrow it down the function calls made when creating a new HTTPSConnection are:

• create new default ssl.SSLContext with ssl._create_default_https_context (which is an alias to ssl.create_default_context)
• ssl._create_default_https_context then calls ssl.SSLContext.set_default_verify_paths
• ssl.SSLContext.set_default_verify_paths is a wrapper around OpenSSL SSL_CTX_set_default_verify_paths where all the certificate loading happens

If python's ssl.get_default_verify_paths() is to be trusted then the certificates should be loaded from

DefaultVerifyPaths(cafile='/etc/pki/tls/cert.pem', capath='/etc/pki/tls/certs', openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/etc/pki/tls/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/etc/pki/tls/certs')

both, SSL_CERT_FILE, and SSL_CERT_DIR are unset and /etc/pki/tls/certs only contains

lrwxrwxrwx 1 root root   49 Feb 16 16:19 ca-bundle.crt -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
lrwxrwxrwx 1 root root   55 Feb 16 16:19 ca-bundle.trust.crt -> /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt

this seems to be pretty much identical in Amazon Linux 2023 and Amazon Linux 2.
the number of certificates in /etc/ssl/crets however is different with Amazon Linux 2 only containing only the two above certificates but Amazon Linux 2023 containing ~400. not sure though if (or how) these certificates are considered.

Note, this issue doesn't only happen in the python runtime but the degradation in performance can be observed in other runtimes (at least for Node 20) and for external Lambda extensions (e.g. some go extension that uses/creates an https connection) as well.

[cornettonuss]

I totaly see this problem on my lambdas too. I had to revert from Python 3.12 to 3.11 so the underlaying System is AL2 again. My metrics are pretty much the same like yours. I had an increase of connection time from about 300ms to 900ms.

[yukihiko-shinoda]

Related ?:

• [Package Request] - OpenSSL 3.0.8 or newer, compiled with O3 optimization option · Issue #434 · amazonlinux/amazon-linux-2023
• [Package Update Request] - OpenSSL 3.2.x · Issue #628 · amazonlinux/amazon-linux-2023

In my investigation, this is caused not by Lambda but Amazon Linux 2023.
When I use this image for Python 3.12 or 2023 of official Amazon Linux image, my static code analysis task that takes only 3 seconds in Amazon Linux 2 of official Amazon Linux image takes 20 seconds that is not useful in development.