Crash in aws_nitro_enclaves_rest_client_destroy

[rohan-passbird]

I am trying to send multiple requests to KMS from inside the enclave. ~1-2% requests are failing with crash in aws_http_connection_release.

Disconnected.
[INFO] [2022-10-26T19:04:41Z] [00007f6c6a8fc700] [http-connection] - 0x7f6c5c015cd0: Client shutdown completed with error 0 (AWS_ERROR_SUCCESS).
[INFO] [2022-10-26T19:04:41Z] [00007f6c6a0fb700] [event-loop] - id=0x1552cd0: Destroying event_loop
[INFO] [2022-10-26T19:04:41Z] [00007f6c6a0fb700] [event-loop] - id=0x1552cd0: Stopping event-loop thread.
Fatal error condition occurred in ../source/event_loop.c:514: event_loop->vtable && event_loop->vtable->is_on_callers_thread
Exiting Application
################################################################################
Stack trace:
################################################################################
/app/enclave/cli/kmstool_enclave_cli(aws_backtrace_print+0x95) [0x43a05e]
/app/enclave/cli/kmstool_enclave_cli(aws_fatal_assert+0x5b) [0x429201]
/app/enclave/cli/kmstool_enclave_cli(aws_event_loop_thread_is_callers_thread+0x40) [0x44f028]
/app/enclave/cli/kmstool_enclave_cli(aws_channel_thread_is_callers_thread+0x1c) [0x469cff]
/app/enclave/cli/kmstool_enclave_cli(aws_channel_release_hold+0x59) [0x469101]
/app/enclave/cli/kmstool_enclave_cli(aws_http_connection_release+0xc7) [0x471afd]
/app/enclave/cli/kmstool_enclave_cli(aws_nitro_enclaves_rest_client_destroy+0x3b) [0x425a54]
/app/enclave/cli/kmstool_enclave_cli(aws_nitro_enclaves_kms_client_destroy+0x33) [0x423f7d]
/app/enclave/cli/kmstool_enclave_cli() [0x41db31]
/app/enclave/cli/kmstool_enclave_cli(main+0x1cb) [0x41e1b2]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7f6c6b143d0a]
/app/enclave/cli/kmstool_enclave_cli(_start+0x2a) [0x41cbea]

[rohan-passbird]

Is it because of https://github.com/aws/aws-nitro-enclaves-sdk-c/blob/main/source/rest.c#L220?

[alcioa]

Building a local setup to try and see this.
So you're basically instantiating a KMS client in the enclave and then issue N kms_decrypt_blocking() calls, right?
Does the crash happen on Decrypt failure when you do cleanup on the KMS client resources?

[rohan-passbird]

I spawned N different kmstools_cli processes instead of doing it in the same process.

As you can see in the stack trace, the crash happens on cleanup, when aws_nitro_enclaves_kms_client_destroy is called. I checked the logs and the KMS request was actually successful.