Failure to set DefaultConfigurationMode causes performance issues during client construction

[Hawxy]

Describe the bug

This is a follow up to #1996 and is possibly related to #2000

Following an update to the SDK, the initial construction of the client on a network request increased to more than 30 seconds. I've spent some more time on this issue and my findings are below.

Expected Behavior

Client construction should be under 2 seconds.

Current Behavior

Client construction takes more than 30 seconds as you can see in the call tree:

Reproduction Steps

1. Create a .NET 6 API with AWSSDK.Extensions.NETCore.Setup + AWSSDK.S3.
2. Configure an AWS profile and set the configuration in appsettings.json
3. Ensure no Environment variables or other settings cause DefaultsMode to be set.
4. Configure DI:

 services.AddDefaultAWSOptions(Configuration.GetAWSOptions());
 services.AddAWSService<IAmazonS3>();

5. Request IAmazonS3 in a controller.

Possible Solution

Stepping through the problem, the cause is this check:

aws-sdk-net/extensions/src/AWSSDK.Extensions.NETCore.Setup/ClientFactory.cs

Lines 238 to 240 in c957aa1

	if (string.Equals(property.Name, "RetryMode", StringComparison.Ordinal) &&
	defaultConfig.RetryMode == RequestRetryMode.Legacy &&
	config.DefaultConfigurationMode != DefaultConfigurationMode.Legacy)

When the getter is called on DefaultConfigurationMode, if it hasn't already been set it and is not available locally, it ends up calling this.DefaultConfiguration which then results in the call tree above.

Setting "DefaultsMode": "Standard" in appsettings resolves the problem.

DefaultsMode/DefaultConfigurationMode is not well documented in regards to the .NET SDK and I don't see it in the examples, thus I would consider this unexpected behavior for the majority of consumers.

Additional Information/Context

No response

AWS .NET SDK and/or Package version used

AWSSDK.S3 3.7.9.2
AWSSDK.Extensions.NETCore.Setup 3.7.2

Targeted .NET Platform

.NET 6

Operating System and version

Windows 11

[puddlewitt]

I am seeing a similar issue. For me this manifests itself as a severe delay in startup performance when under load.

1. New ECS task started in the EC2 cluster
2. New ECS task passes first HealthCheck and is deemed ready by the TargetGroup
3. New ECS task starts receiving traffic from the load balancer
4. New ECS task does not respond to any traffic due to the lock
5. New ECS task does not respond to the HealthCheck from the TargetGroup
6. New ECS task is deemed unhealthy and is removed (this process is then repeated over and over)

My solution so far is a /health endpoint which is invoked by the TargetGroup during the initial HealthCheck which invokes the following methods

_amazonDynamoDb.Config.Validate();
_amazonCloudWatch.Config.Validate();

[tomupson]

+1. Cost me 4 hours of debugging

Could do with documenting that this is necessary configuration if that is the case

[zippo227]

I recommend that it be set to Standard by default without needing to traverse an entire tree for 30 seconds to find the best thing to be.

[dscpinheiro]

Sorry for the delay in the response, but do you still see the problem in the latest version of the AWSSDK.Extensions.NETCore.Setup and AWSSDK.S3 packages? I haven't been able to reproduce it locally, but at the same time I couldn't find a specific PR / commit fixing this issue... (so it's likely I'm missing something).

[Hawxy]

@dscpinheiro I came across a reddit thread of someone running into this issue just last week so it's still an issue in v3. I noticed this codepath got rewritten in v4 so I assume it won't be an issue there.

I can check again but if that getter behaviour hasn't been touched I should be able to reproduce it.

[normj]

We have made changes in V3 to avoid accidently triggering the DefaultConfigurationMode resolution prematurely. In V4 which is currently in preview we have rewritten this code to get rid of the reflection and have the default config in AWSSDK.Extensions.NETCore.Setup not directly inherit from the ClientConfig in the SDK and instead use a simple type that ensures this regression won't happen again.

[github-actions]

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.