FAST.Farm maximum number of spatial nodes (precompiled binaries)

[VKipke]

Hi all,

with FAST.Farm (precompiled binaries from OpenFAST v4.0.0 assets) I am trying to run a simulation with 12 turbines and it seems like there is a maximum number of low-res spatial nodes.
A configuration which has 788,544 nodes runs fine:

222 NX_Low
222 NY_Low
16 NZ_Low
18 dX_Low
18 dY_Low
10 dZ_Low

Using a slightly higher number of nodes (883,600) with a slightly higher resolution and nothing else changed:

235 NX_Low
235 NY_Low
16 NZ_Low
17 dX_Low
17 dY_Low
10 dZ_Low

I am getting this error message:

Running AWAE.
forrtl: severe (170): Program Exception - stack overflow
Image PC Routine Line Source
FAST.Farm_x64.exe 00007FF6A31D1507 Unknown Unknown Unknown
FAST.Farm_x64.exe 00007FF6A031C01F Unknown Unknown Unknown
FAST.Farm_x64.exe 00007FF6A030F48F Unknown Unknown Unknown
FAST.Farm_x64.exe 00007FF6A01213F4 Unknown Unknown Unknown
FAST.Farm_x64.exe 00007FF6A2CA882E Unknown Unknown Unknown
FAST.Farm_x64.exe 00007FF6A31D1710 Unknown Unknown Unknown
KERNEL32.DLL 00007FFB87577374 Unknown Unknown Unknown
ntdll.dll 00007FFB878FCC91 Unknown Unknown Unknown
forrtl: severe (28): CLOSE error, unit 7, file "Unknown"

Technically, there should be no issue of simulating millions of nodes. It would just take longer to compute.
Can someone explain this apparent limitation? How can it be changed or bypassed?

Thanks!

[andrew-platt]

Dear @VKipke,

By default temporary arrays go on the stack, and if a temporary array is large, it can cause a stack overflow if a lot of the stack memory is already in use. Likely we have a temporary array associated with the low resolution grid that got too large. I attempted to recreate this issue with two turbines, but was unsuccessful even with 100,00050030 low resolution grid points, so the issue is not because of a limitation of this grid.

We had seen this issue before and attempted to fix it with this PR: #2452. However, according to some discussions on the intel forum, using /heap-arrays:1000 doesn't work as expected and instead we should be using /heap-arrays:0 (see the last post on this thread: https://community.intel.com/t5/Intel-Fortran-Compiler/memory-usage/m-p/1137047/highlight/true). I'm working on a fix for this now and should have it ready before next week.

In the meantime, you can change the stack size of the executable by using editbin with the command: editbin /stack:100000000 FAST.Farm_x64.exeto increase the stack size to 100 MB. I thinkeditbin` is included with the Intel OneAPI Fortran distribution.

Regards,

[VKipke]

Dear @andrew-platt ,

Thanks for the quick response! I tried

editbin /stack:100000000 FAST.Farm_x64.exe FAST.Farm_V7dev.fstf

but "editbin" was not found.
I assume I need to install the Intel oneAPI HPC Toolkik as described on https://openfast.readthedocs.io/en/main/source/install/index.html#compilers?

• Do I really need everything what is included (that's 22GB)?
• the compilers require further more MS Visual Studio 2019 or 2022 or/and (?) Build Tools 2019 or 2022

Is this correct? I don't need to compile anything, but simply run the editbin command along wigh FAST.Farm.exe

Regards

[bjonkman]

editbin is part of Visual Studio, not the Intel compilers. You shouldn't need anything from oneAPI.

See https://learn.microsoft.com/en-us/cpp/build/reference/editbin-reference?view=msvc-170

[VKipke]

Thank you for this great support! Unfortunately I have to ask again. I have installed Visual Studio (Community) 2019 (16.11.43). editbin.exe is located in C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\bin\Hostx64\x64. Still, running

editbin /stack:100000000 FAST.Farm_x64.exe FAST.Farm_V7dev.fstf

does not work. I copied editbin.exe into my FAST.Fam folder, then "link.exe" was not found. Copying also this into the FAST.Farm folder results in an error message where tbbmalloc.dll is missing. Again, copied this into my FAST.Farm folder. The result is now

FAST.Farm_V7dev.fstf : warning LNK4048: Invalid format file; ignored

When typing the command editbin without any other command-line input, EDITBIN displays a usage statement that summarizes its options just as stated on https://learn.microsoft.com/en-us/cpp/build/reference/editbin-reference?view=msvc-170, i.e. the command is at least found.

Any ideas how to resolve this?

Thanks in advance to the OpenFAST / FAST.Fam dev team :)

[bjonkman]

editbit is a program that changes the executable file, which in this case is FAST.Farm. You will run it to modify how large a stack the FAST.Farm executable can use. Call it this way, without the FAST.Farm input file you were using:

editbin /stack:100000000 FAST.Farm_x64.exe

That will give you a modified FAST.Farm_x64.exe file that you can then use in place of the executable file you were using. Note: I would not recommend moving the files that are in the Visual Studio folders.

After modifying the FAST.Farm executable, you can call it as you would before:

FAST.Farm_x64.exe FAST.Farm_V7dev.fstf

[VKipke]

Thank you so much for the clarification! I did not understand editbin.exe will modify the FAST.Farm_x64.exe itself before the actual simulation run.
It worked flawlessly, but I still had to put a copy of editbin.exe into my FAST.Farm folder, otherwise the command would not have been recognized/found.

Thanks again for all your impressive work and very helpful support!