-
Notifications
You must be signed in to change notification settings - Fork 40
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
MATLAB code crashes with larger number of densities #15
Comments
Just some more data. maxNumCompThreads(1) |
I can replicate the bug by running fortran test driver test_lfmm3d_vec.f.
Most likely OMP_STACKSIZE is set incorrectly. On my machine (ubuntu
20.04LTS, gcc 9.3.0), I have the following system defaults:
export OMP_DISPLAY_ENV=TRUE
$ make -f test_lfmm3d_vec.make -j8
gfortran -fPIC -O3 -funroll-loops -march=native -fopenmp -std=legacy -o
int2-lfmm3d-vec test_lfmm3d_vec.o ../../src/Common/hkrand.o
../../src/Common/dlaran.o ../../src/Common/prini.o
../../src/Common/rotgen.o ../../src/Common/legeexps.o
../../src/Common/rotviarecur.o ../../src/Common/yrecursion.o
../../src/Laplace/l3dterms.o ../../src/Laplace/l3dtrans.o
../../src/Laplace/laprouts3d.o ../../src/Laplace/lapkernels.o
../../src/Laplace/lfmm3d.o ../../src/Laplace/lfmm3dwrap_vec.o
../../src/Laplace/lwtsexp_sep1.o ../../src/Laplace/lwtsexp_sep2.o
../../src/Laplace/lpwrouts.o ../../src/Laplace/lndiv.o
../../src/Common/rotproj.o ../../src/Common/tree_lr_3d.o
../../src/Common/dfft.o ../../src/Common/fmmcommon.o
./int2-lfmm3d-vec
OPENMP DISPLAY ENVIRONMENT BEGIN
_OPENMP = '201511'
OMP_DYNAMIC = 'FALSE'
OMP_NESTED = 'FALSE'
OMP_NUM_THREADS = '8'
OMP_SCHEDULE = 'DYNAMIC'
OMP_PROC_BIND = 'FALSE'
OMP_PLACES = ''
OMP_STACKSIZE = '0'
OMP_WAIT_POLICY = 'PASSIVE'
OMP_THREAD_LIMIT = '4294967295'
OMP_MAX_ACTIVE_LEVELS = '2147483647'
OMP_CANCELLATION = 'FALSE'
OMP_DEFAULT_DEVICE = '0'
OMP_MAX_TASK_PRIORITY = '0'
OMP_DISPLAY_AFFINITY = 'FALSE'
OMP_AFFINITY_FORMAT = 'level %L thread %i affinity %A'
OPENMP DISPLAY ENVIRONMENT END
==========================================
Testing suite for lfmm3d_vec
Requested precision = 0.5000E-09
testing source to source
interaction: charges
output: potentials
Program received signal SIGSEGV: Segmentation fault - invalid memory
reference.
Backtrace for this error:
Program received signal SIGSEGV: Segmentation fault - invalid memory
reference.
Backtrace for this error:
#0 0x154e708c9d01 in ???
#1 0x154e708c8ed5 in ???
#2 0x154e7052520f in ???
#3 0x5619dc1ea098 in ???
#4 0xffffffffffffffff in ???
make: *** [test_lfmm3d_vec.make:54: all] Segmentation fault (core dumped)
…On Sat, Nov 14, 2020 at 8:42 PM Alex Barnett ***@***.***> wrote:
Just some more data.
I find the same on MATLAB R2017a on ubuntu 16.04, i7, gcc9. omp, 8 threads.
It's not the total data size since ns=1e5; nd=10; doesn't crash, neither
does ns=1e5; nd=65; even though it uses 16 GB or so.
ns=1e4; nd=1e2 crashes.
It's not a strict limit on nd either, since ns=1e2; nd=1e4; is crashless.
maxNumCompThreads(1)
prevents all such crashes.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#15 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABWJ3GXTJHYJVK6YF37BLMTSP5E3NANCNFSM4TVY6QJA>
.
|
Hi Zydrunas - good to hear from you! Ok, but how come FMM3D puts much on the stack? Surely any variable-sized arrays go on the heap, via malloc etc? FWIW matlab (libiomp5) doesn't like env var changing stacksize: OMP: Warning #181: OMP_STACKSIZE: ignored because KMP_STACKSIZE has been defined ulimit -s unlimited Ok, off to bed now, Alex |
Thanks, Alex!
Defaults for OMP and KMP (that’s intel) stack size is something like 4 Mbytes for each thread.
That is not much, all private variables and all local variables in called subroutines (all of them combined!) go into the private thread stacks. KMP overrides OMP.
google: omp stack size
https://software.intel.com/content/www/us/en/develop/articles/openmp-stacksize-common-error.html <https://software.intel.com/content/www/us/en/develop/articles/openmp-stacksize-common-error.html>
https://stackoverflow.com/questions/13264274/why-segmentation-fault-is-happening-in-this-openmp-code
google: omp_stacksize and kmp_stacksize
http://www.bgu.ac.il/intel_fortran_docs/compiler_f/main_for/mergedProjects/optaps_for/common/optaps_par_var.htm
https://www.mathworks.com/matlabcentral/answers/447978-omp-warning-181-omp_stacksize-ignored-because-kmp_stacksize-has-been-defined
On a Mac, you may also want to experiment a bit with compiler flags for single threaded stack as well.
Zydrunas
… On Nov 14, 2020, at 10:42 PM, Alex Barnett ***@***.***> wrote:
Hi Zydrunas - good to hear from you!
Ok, but how come FMM3D puts much on the stack? Surely any variable-sized arrays go on the heap, via malloc etc?
FWIW matlab (libiomp5) doesn't like env var changing stacksize:
OMP: Warning #181: OMP_STACKSIZE: ignored because KMP_STACKSIZE has been defined
ulimit -s unlimited
didn't stop the matlab segfault.
Ok, off to bed now, Alex
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub <#15 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABWJ3GSARF47BDLP6QGD2CDSP5S5HANCNFSM4TVY6QJA>.
|
One can also try to track down big local variables that might be causing this (temporary vectorized multipole expansions?)
… On Nov 14, 2020, at 11:00 PM, Zydrunas Gimbutas ***@***.***> wrote:
Thanks, Alex!
Defaults for OMP and KMP (that’s intel) stack size is something like 4 Mbytes for each thread.
That is not much, all private variables and all local variables in called subroutines (all of them combined!) go into the private thread stacks. KMP overrides OMP.
google: omp stack size
https://software.intel.com/content/www/us/en/develop/articles/openmp-stacksize-common-error.html
https://stackoverflow.com/questions/13264274/why-segmentation-fault-is-happening-in-this-openmp-code
google: omp_stacksize and kmp_stacksize
http://www.bgu.ac.il/intel_fortran_docs/compiler_f/main_for/mergedProjects/optaps_for/common/optaps_par_var.htm
https://www.mathworks.com/matlabcentral/answers/447978-omp-warning-181-omp_stacksize-ignored-because-kmp_stacksize-has-been-defined
On a Mac, you may also want to experiment a bit with compiler flags for single threaded stack as well.
Zydrunas
> On Nov 14, 2020, at 10:42 PM, Alex Barnett ***@***.***> wrote:
>
>
> Hi Zydrunas - good to hear from you!
>
> Ok, but how come FMM3D puts much on the stack? Surely any variable-sized arrays go on the heap, via malloc etc?
>
> FWIW matlab (libiomp5) doesn't like env var changing stacksize:
>
> OMP: Warning #181: OMP_STACKSIZE: ignored because KMP_STACKSIZE has been defined
>
> ulimit -s unlimited
> didn't stop the matlab segfault.
>
> Ok, off to bed now, Alex
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub, or unsubscribe.
>
|
Thanks for bringing this to our notice. Thanks Alex and Zydrunas for the additional data and suggestions too. I think I know the cause of the issue. It is in local memory allocation in list4 processing where the arrays are allocated inside an openmp loop rather than outside. This might also explain some issues we are having with compiling with intel compilers on windows. Hopefully that should fix the bug.. |
@mrachh is this fixed - can it be closed? |
Hi,
I run the code below in MATLAB and it crashes. The crash happens reliably whenever the number of densities
nd
is big enough. Also, it only happens if I compile the code with OpenMP, the single-threaded version works fine.OS: macOS Catalina 10.15.6
MATLAB version: R2019b or R2020b
FMM3D library compiled with the default
make.inc.macos.gnu
optionsAdditionally, I also tried similar things on a Ubuntu system and modified the
OMP_STACKSIZE
variable as described here. This somewhat alleviates the issue, but still crashes whennd
is big enough. (However, I am not too familiar with Linux, so I could have done something wrong here.)The text was updated successfully, but these errors were encountered: