stackSize override is not correct value format

[clintwar]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

When aspnetcore for in process hosting overrides the default stack size for new threads on the threadpool (https://learn.microsoft.com/en-us/aspnet/core/host-and-deploy/iis/advanced?view=aspnetcore-9.0) the value is incorrectly defaulted.

aspnetcore/src/Servers/IIS/AspNetCoreModuleV2/InProcessRequestHandler/InProcessOptions.cpp

Line 68 in e05d6f6

m_strStackSize = find_element(handlerSettings, CS_ASPNETCORE_HANDLER_STACK_SIZE).value_or(L"1048576");

3be11f6

The internals of coreclr is looking for the value as base16. The default value of 1048576 is treating as base16. This is not 1 MB. This is 17 MB.

If you take a memory dump of an inprocess hosted .net core 8 app and view !address -f:MEM_RESERVE,Stack you will notice reserved memory for stack around this 0x1048576 size. In 32bit world this is a bad situation as you will run out of memory if your process needs to create many threads.

In the learning document it indicates how to set it to 2MB however this actually would set it to 34~ MB.

Expected Behavior

Actually set stack to a reasonable size similar to what .net core does by default 1.5 MB for 32bit app and 4MB for x64.

Steps To Reproduce

If you take a memory dump of an inprocess hosted .net core 8 app and view !address -f:MEM_RESERVE,Stack you will notice reserved memory for stack around this 0x1048576 size. In 32bit world this is a bad situation as you will run out of memory if your process needs to create many threads.

Exceptions (if any)

No response

.NET Version

No response

Anything else?

No response

[clintwar]

Output from memory dump

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Stack 684 bf5b0000 ( 2.990 GB) 76.24% 74.75%
2263 29562000 ( 661.383 MB) 16.47% 16.15%
Image 3094 f46d000 ( 244.426 MB) 6.09% 5.97%
Free 548 500a000 ( 80.039 MB) 1.95%
Heap 126 2cb1000 ( 44.691 MB) 1.11% 1.09%
TEB 228 372000 ( 3.445 MB) 0.09% 0.08%
Other 29 41000 ( 260.000 kB) 0.01% 0.01%
PEB 1 3000 ( 12.000 kB) 0.00% 0.00%

!address -f:MEM_RESERVE,Stack
....
313d0000 32419000 1049000 MEM_PRIVATE MEM_RESERVE Stack [~43; 1ff4.22e4]
32420000 33465000 1045000 MEM_PRIVATE MEM_RESERVE Stack [~44; 1ff4.2240]
33470000 344bc000 104c000 MEM_PRIVATE MEM_RESERVE Stack [~45; 1ff4.2170]
344c0000 35505000 1045000 MEM_PRIVATE MEM_RESERVE Stack [~46; 1ff4.193c]
35510000 36558000 1048000 MEM_PRIVATE MEM_RESERVE Stack [~47; 1ff4.1a68]
37340000 3838b000 104b000 MEM_PRIVATE MEM_RESERVE Stack [~48; 1ff4.1cbc]
38390000 393d7000 1047000 MEM_PRIVATE MEM_RESERVE Stack [~49; 1ff4.26cc]
393e0000 3a428000 1048000 MEM_PRIVATE MEM_RESERVE Stack [~50; 1ff4.2118]
3a430000 3b479000 1049000 MEM_PRIVATE MEM_RESERVE Stack [~52; 1ff4.18cc]
3b480000 3c4c7000 1047000 MEM_PRIVATE MEM_RESERVE Stack [~53; 1ff4.2480]
3c4d0000 3d519000 1049000 MEM_PRIVATE MEM_RESERVE Stack [~54; 1ff4.1854]
3d520000 3e569000 1049000 MEM_PRIVATE MEM_RESERVE Stack [~55; 1ff4.1b98]
3e570000 3f5b1000 1041000 MEM_PRIVATE MEM_RESERVE Stack [~56; 1ff4.c8c]
3f5c0000 40607000 1047000 MEM_PRIVATE MEM_RESERVE Stack [~57; 1ff4.13c4]
40610000 41655000 1045000 MEM_PRIVATE MEM_RESERVE Stack [~58; 1ff4.21d0]
41660000 426a4000 1044000 MEM_PRIVATE MEM_RESERVE Stack [~59; 1ff4.1e3c]
426b0000 436f7000 1047000 MEM_PRIVATE MEM_RESERVE Stack [~60; 1ff4.144c]
43700000 44746000 1046000 MEM_PRIVATE MEM_RESERVE Stack [~61; 1ff4.202c]
44750000 45795000 1045000 MEM_PRIVATE MEM_RESERVE Stack [~62; 1ff4.23b0]
457a0000 467e9000 1049000 MEM_PRIVATE MEM_RESERVE Stack [~63; 1ff4.2670]

....

This particular dump did result in oom in a bad place, starting new thread.

0:000> !pe 176c8030
Exception object: 176c8030
Exception type: System.OutOfMemoryException
Message:
InnerException:
StackTrace (generated):
SP IP Function
00000000 00000001 System_Private_CoreLib!System.Threading.Thread.StartInternal(System.Threading.ThreadHandle, Int32, Int32, Char*)+0x2
FE32EA38 D4925A2B System_Private_CoreLib!System.Threading.Thread.StartCore()+0xb3
FE32EA8C DCE05670 System_Private_CoreLib!System.Threading.PortableThreadPool+WorkerThread.CreateWorkerThread()+0x50
.....

0:000> !threadpool
Using the Portable thread pool.

CPU utilization: 9%
Workers Total: 178
Workers Running: 178
Workers Idle: 0
Worker Min Limit: 100
Worker Max Limit: 1023

[adityamandaleeka]

Yup, we should fix this. This is the relevant code in runtime where you can see that it expects a hex value: https://github.com/dotnet/runtime/blob/3d275542626ac6420269168ecbebe2bcb8e6ae54/src/coreclr/vm/threads.cpp#L2051C8-L2051C42

Good catch!

[adityamandaleeka]

Are you interested in opening a PR for this @clintwar?

[clintwar]

I can if no one else wants to :)

[adityamandaleeka]

This should be a relatively straightforward fix (minor code change, and a doc update here: https://github.com/dotnet/AspNetCore.Docs/blob/main/aspnetcore/host-and-deploy/iis/advanced.md), so if you're interested, we'd love to have you take it on. Let me know if you need any pointers or guidance along the way. Otherwise, if you'd prefer not to, I can take care of it--just let me know what works best for you.

[BrennanConroy]

We should also separately update the comment in https://github.com/dotnet/runtime/blob/016b59002f2a2e276a8223a6bae91222b733d65f/src/coreclr/inc/clrconfigvalues.h#L484 to mention base16

dotnet/runtime#113192

[apurvghai]

Here's a screen shot of we finally investigated, even though it's clear from Memory dump analysis

[clintwar]

PR for fix is here
PR for doc is Documentation Provides Incorrect Values For Default Stack Size