Skip to content

IsUserCryptoAvailable check slows down Lambda cold start and takes large part of the initialization of Config and ServiceClients #2537

Closed
@unchartedxx

Description

@unchartedxx

Describe the bug

I'm using the AWS SDK to build a lambda function on .NET 7 in AOT mode. I noticed that during Dependency Injection the creation of AmazonLambdaConfig and AmazonLambdaClient objects was taking the majority of the time of the whole startup. I enabled the SDK logs (AWSConfigs.LoggingConfig.LogTo = LoggingOptions.Console;) and I noticed the following messages (emphasis on the second line):

2023-02-08T12:45:05.362Z	d3d66c80-d0f0-4a14-9ccc-4fd03e6567c8	dbug	Start DI injection
2023-02-08T12:45:05.484Z	d3d66c80-d0f0-4a14-9ccc-4fd03e6567c8	info	UserCrypto 1|2023-02-08T12:45:05.483Z|INFO|UserCrypto is not supported.  This may be due to use of a non-Windows operating system or Windows Nano Server, or the current user account may not have its profile loaded. Unable to load shared library 'Crypt32.dll' or one of its dependencies. In order to help diagnose loading problems, consider using a tool like strace. If you're using glibc, consider setting the LD_DEBUG environment variable: 
Crypt32.dll.so: cannot open shared object file: No such file or directory
libCrypt32.dll.so: cannot open shared object file: No such file or directory
Crypt32.dll: cannot open shared object file: No such file or directory
libCrypt32.dll: cannot open shared object file: No such file or directory
2023-02-08T12:45:05.504Z	d3d66c80-d0f0-4a14-9ccc-4fd03e6567c8	info	EnvironmentVariablesAWSCredentials 2|2023-02-08T12:45:05.503Z|INFO|Credentials found using environment variables.
2023-02-08T12:45:05.504Z	d3d66c80-d0f0-4a14-9ccc-4fd03e6567c8	info	EnvironmentVariableAWSRegion 3|2023-02-08T12:45:05.504Z|INFO|Region found using environment variable.
2023-02-08T12:45:05.504Z	d3d66c80-d0f0-4a14-9ccc-4fd03e6567c8	info	EnvironmentVariableInternalConfiguration 4|2023-02-08T12:45:05.504Z|INFO|The environment variable AWS_ENABLE_ENDPOINT_DISCOVERY was not set with a value.
2023-02-08T12:45:05.504Z	d3d66c80-d0f0-4a14-9ccc-4fd03e6567c8	info	EnvironmentVariableInternalConfiguration 5|2023-02-08T12:45:05.504Z|INFO|The environment variable AWS_MAX_ATTEMPTS was not set with a value.
2023-02-08T12:45:05.504Z	d3d66c80-d0f0-4a14-9ccc-4fd03e6567c8	info	EnvironmentVariableInternalConfiguration 6|2023-02-08T12:45:05.504Z|INFO|The environment variable AWS_RETRY_MODE was not set with a value.
2023-02-08T12:45:05.504Z	d3d66c80-d0f0-4a14-9ccc-4fd03e6567c8	info	EnvironmentVariableInternalConfiguration 7|2023-02-08T12:45:05.504Z|INFO|The environment variable AWS_EC2_METADATA_SERVICE_ENDPOINT was not set with a value.
2023-02-08T12:45:05.504Z	d3d66c80-d0f0-4a14-9ccc-4fd03e6567c8	info	EnvironmentVariableInternalConfiguration 8|2023-02-08T12:45:05.504Z|INFO|The environment variable AWS_EC2_METADATA_SERVICE_ENDPOINT_MODE was not set with a value.
2023-02-08T12:45:05.504Z	d3d66c80-d0f0-4a14-9ccc-4fd03e6567c8	info	EnvironmentVariableInternalConfiguration 9|2023-02-08T12:45:05.504Z|INFO|The environment variable AWS_USE_DUALSTACK_ENDPOINT was not set with a value.
2023-02-08T12:45:05.504Z	d3d66c80-d0f0-4a14-9ccc-4fd03e6567c8	info	EnvironmentVariableInternalConfiguration 10|2023-02-08T12:45:05.504Z|INFO|The environment variable AWS_USE_FIPS_ENDPOINT was not set with a value.
2023-02-08T12:45:05.504Z	d3d66c80-d0f0-4a14-9ccc-4fd03e6567c8	info	ProfileInternalConfiguration 11|2023-02-08T12:45:05.504Z|INFO|Unable to find a profile named 'default' in store Amazon.Runtime.CredentialManagement.CredentialProfileStoreChain
2023-02-08T12:45:05.504Z	d3d66c80-d0f0-4a14-9ccc-4fd03e6567c8	info	DefaultConfigurationProvider 12|2023-02-08T12:45:05.504Z|INFO|Resolved DefaultConfigurationMode for RegionEndpoint [eu-west-1] to [Legacy].

That UserCrypto message is generated at IsUserCryptAvailable.
That method is called whenever downstream it needs to try to get profile information if possible in CredentialProfileStoreChain. The get profile method is called indirectly whenever it needs to find credentials or profile configuration such as this or even this.

I want to note that all of this is not specific to the AmazonLambdaClient so I suppose it would affect all AWSSDK service clients.

Normally creating the Config and the Client without doing anything spacial takes between 150 and 200ms. If I call UserCrypto.IsUserCryptoAvailable in my code it takes around ~140 ms. If I create Config and Client after that it takes a few tens of milliseconds to do the initialisation. Therefore I can conclude that the single call to UserCrypto is taking the majority of the time even considering all the time to obtain credentials following the chain of folders, envvars,...

The test was done on .NET 7 and AOT. I can only assume it would take much more on earlier versions and/or non-AOT mode. The lambda was set with 256MB.

Expected Behavior

The AWS SDK notices it is running on a non-Windows enviroment and just return false to "UserCrypto.IsUserCryptAvailable" doesn't even try to call UserCrypto API and fail when the dll are not there.

Current Behavior

"UserCrypto.IsUserCryptAvailable" calls "Decrypt" that calls "CryptProtectData" that is a Windows only API but it tries anyway for the presence of the Crypt32.dll.so file and then fails.

Reproduction Steps

AWSConfigs.LoggingConfig.LogTo = LoggingOptions.Console;
var clientConfig = new AmazonLambdaConfig();
var client = new AmazonLambdaClient(clientConfig);

Possible Solution

Checks for current operating system in "UserCrypto.IsUserCryptAvailable". Return false if not Windows.

Additional Information/Context

The only workaround ATM to avoid any possible call to IsUserCryptoAvailable is to set a lot of stuff ahead of time before creating the ClientConfig and the ServiceConfig:

// Avoid checking the profile for CSM
Environment.SetEnvironmentVariable("AWS_CSM_ENABLED", "false");

// Avoid checking the profile first for credentials
var access_key = Environment.GetEnvironmentVariable("AWS_ACCESS_KEY_ID");
var secret_key = Environment.GetEnvironmentVariable("AWS_SECRET_ACCESS_KEY");
var session_token = Environment.GetEnvironmentVariable("AWS_SESSION_TOKEN");

var credentials = new SessionAWSCredentials(access_key, secret_key, session_token);

// Avoid checking the profile for region
var region = RegionEndpoint.GetBySystemName(Environment.GetEnvironmentVariable("AWS_REGION"));

var clientConfig = new AmazonLambdaConfig
{
    DefaultConfigurationMode = DefaultConfigurationMode.InRegion, // skips certain profile checks
    RegionEndpoint = region,
    DisableLogging= false,

    // The following are the default values, also set to avoid fallback searches into the profile
    RetryMode = RequestRetryMode.Standard, // copied from InRegion                    
    MaxErrorRetry = 2,
    UseDualstackEndpoint = false,
    UseFIPSEndpoint = false,
    EndpointDiscoveryEnabled = false,
};
var client = new AmazonLambdaClient(credentials, clientConfig);

AWS .NET SDK and/or Package version used

AWSSDK.Lambda 3.7.104.7

Targeted .NET Platform

.NET 7 (AOT)

Operating System and version

Lambda (Amazon Linux 2)

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugThis issue is a bug.credentialsp2This is a standard priority issuequeued

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions