Failure to set DefaultConfigurationMode causes performance issues during client construction

[Hawxy]

Describe the bug

This is a follow up to #1996 and is possibly related to #2000

Following an update to the SDK, the initial construction of the client on a network request increased to more than 30 seconds. I've spent some more time on this issue and my findings are below.

Expected Behavior

Client construction should be under 2 seconds.

Current Behavior

Client construction takes more than 30 seconds as you can see in the call tree:

Reproduction Steps

1. Create a .NET 6 API with AWSSDK.Extensions.NETCore.Setup + AWSSDK.S3.
2. Configure an AWS profile and set the configuration in appsettings.json
3. Ensure no Environment variables or other settings cause DefaultsMode to be set.
4. Configure DI:

 services.AddDefaultAWSOptions(Configuration.GetAWSOptions());
 services.AddAWSService<IAmazonS3>();

5. Request IAmazonS3 in a controller.

Possible Solution

Stepping through the problem, the cause is this check:

aws-sdk-net/extensions/src/AWSSDK.Extensions.NETCore.Setup/ClientFactory.cs

Lines 238 to 240 in c957aa1

	if (string.Equals(property.Name, "RetryMode", StringComparison.Ordinal) &&
	defaultConfig.RetryMode == RequestRetryMode.Legacy &&
	config.DefaultConfigurationMode != DefaultConfigurationMode.Legacy)

When the getter is called on DefaultConfigurationMode, if it hasn't already been set it and is not available locally, it ends up calling this.DefaultConfiguration which then results in the call tree above.

Setting "DefaultsMode": "Standard" in appsettings resolves the problem.

DefaultsMode/DefaultConfigurationMode is not well documented in regards to the .NET SDK and I don't see it in the examples, thus I would consider this unexpected behavior for the majority of consumers.

Additional Information/Context

No response

AWS .NET SDK and/or Package version used

AWSSDK.S3 3.7.9.2
AWSSDK.Extensions.NETCore.Setup 3.7.2

Targeted .NET Platform

.NET 6

Operating System and version

Windows 11

[puddlewitt]

I am seeing a similar issue. For me this manifests itself as a severe delay in startup performance when under load.

1. New ECS task started in the EC2 cluster
2. New ECS task passes first HealthCheck and is deemed ready by the TargetGroup
3. New ECS task starts receiving traffic from the load balancer
4. New ECS task does not respond to any traffic due to the lock
5. New ECS task does not respond to the HealthCheck from the TargetGroup
6. New ECS task is deemed unhealthy and is removed (this process is then repeated over and over)

My solution so far is a /health endpoint which is invoked by the TargetGroup during the initial HealthCheck which invokes the following methods

_amazonDynamoDb.Config.Validate();
_amazonCloudWatch.Config.Validate();

[tomupson]

+1. Cost me 4 hours of debugging

Could do with documenting that this is necessary configuration if that is the case

[zippo227]

I recommend that it be set to Standard by default without needing to traverse an entire tree for 30 seconds to find the best thing to be.

[dscpinheiro]

Sorry for the delay in the response, but do you still see the problem in the latest version of the AWSSDK.Extensions.NETCore.Setup and AWSSDK.S3 packages? I haven't been able to reproduce it locally, but at the same time I couldn't find a specific PR / commit fixing this issue... (so it's likely I'm missing something).