skipped some tests to get make check to pass #200

edwardhartnett · 2019-11-06T12:12:48Z

Part of #157

In this PR I changed the number of processors from 2 to 6 for a number of tests, which caused them to pass.

I also had to comment out two tests - @wrongkindofdoctor I will post the failures to the issue.

@underwoo if we can get this PR merged, I can set up a working Jenkins server here which can help a lot. I can add tests for all the configuration options, as they are added. I can also do check for memory leaks and compile warnings, etc, and use the built-in code coverage tools, once we have transitioned some of the code from .inc files to .F90 files.

Over time, your travis build or your own jenkins server can take this on in parallel to my server.

More tests are more good!

edwardhartnett · 2019-11-06T12:15:46Z

test_fms/mpp/test_mpp_domains.bats

+#       skip "Fails on Travis"
+#     fi
+#     sed "s/check_parallel = .false./check_parallel = .true./" input.nml_base > input.nml
+#     run mpirun -n 6 ./test_mpp_domains


Changing this to 6 cores did not help, so I commented out the test. With 6 cores I get:

PASS: test_mpp_domains.sh 12 12: Test Group FAIL: test_mpp_domains.sh 13 14: Test Check Parallel # test_mpp_domains.sh: (in test file test_mpp_domains.bats, line 149) # test_mpp_domains.sh: `[ "$status" -eq 0 ]' failed # test_mpp_domains.sh: NOTE from PE 0: MPP_DOMAINS_SET_STACK_SIZE: stack size set to 32768. # test_mpp_domains.sh: NOTE from PE 0: MPP_DOMAINS_SET_STACK_SIZE: stack size set to 10000000. # test_mpp_domains.sh: npes, mpes, nx, ny, nz, whalo, ehalo, shalo, nhalo = 6 3 64 64 10 2 2 2 2 # test_mpp_domains.sh: Memory(MB) used in in the begining= 8.824E+00 9.152E+00 1.117E-01 8.935E+00 # test_mpp_domains.sh: --------------------> Calling test_check_parallel <------------------- # test_mpp_domains.sh: 2D parallel checking: comparison between 3 pes and 3 pe on 3 pes is ok # test_mpp_domains.sh: 2D parallel checking: comparison between 3 pes and 3 pe on 4 pes is ok # test_mpp_domains.sh: 2D parallel checking: comparison between 3 pes and 3 pe on 5 pes is ok # test_mpp_domains.sh: Program received signal SIGSEGV: Segmentation fault - invalid memory reference. # test_mpp_domains.sh: Program received signal SIGSEGV: Segmentation fault - invalid memory reference. # test_mpp_domains.sh: Backtrace for this error: # test_mpp_domains.sh: Program received signal SIGSEGV: Segmentation fault - invalid memory reference. # test_mpp_domains.sh: Backtrace for this error: # test_mpp_domains.sh: Backtrace for this error: # test_mpp_domains.sh: #0 0x7f3ecee292da in ??? # test_mpp_domains.sh: #1 0x7f3ecee28503 in ??? # test_mpp_domains.sh: #2 0x7f3ecea5bf1f in ??? # test_mpp_domains.sh: #3 0x7f3ecf5ee10c in ??? # test_mpp_domains.sh: #4 0x7f3ecf69061d in ??? # test_mpp_domains.sh: #5 0x7f3ecf690de3 in ??? # test_mpp_domains.sh: #6 0x7f3ecf69925b in ??? # test_mpp_domains.sh: #7 0x7f3ecf69c71f in ??? # test_mpp_domains.sh: #8 0x7f3ecf69cccb in ??? # test_mpp_domains.sh: #9 0x55970469ab43 in ??? # test_mpp_domains.sh: #10 0x55970467f5fd in ??? # test_mpp_domains.sh: #11 0x5597047b0197 in ??? # test_mpp_domains.sh: #12 0x7f3ecea3eb96 in ??? # test_mpp_domains.sh: #13 0x55970467b779 in ??? # test_mpp_domains.sh: #14 0xffffffffffffffff in ??? # test_mpp_domains.sh: #0 0x7f58ec90b2da in ??? # test_mpp_domains.sh: #1 0x7f58ec90a503 in ??? # test_mpp_domains.sh: #2 0x7f58ec53df1f in ??? # test_mpp_domains.sh: #3 0x7f58ed0d010c in ??? # test_mpp_domains.sh: #4 0x7f58ed17261d in ??? # test_mpp_domains.sh: #5 0x7f58ed172de3 in ??? # test_mpp_domains.sh: #6 0x7f58ed17b25b in ??? # test_mpp_domains.sh: #7 0x7f58ed17e71f in ??? # test_mpp_domains.sh: #8 0x7f58ed17eccb in ??? # test_mpp_domains.sh: #9 0x55d51e7cdb43 in ??? # test_mpp_domains.sh: #10 0x55d51e7b25fd in ??? # test_mpp_domains.sh: #11 0x55d51e8e3197 in ??? # test_mpp_domains.sh: #12 0x7f58ec520b96 in ??? # test_mpp_domains.sh: #13 0x55d51e7ae779 in ??? # test_mpp_domains.sh: #14 0xffffffffffffffff in ??? # test_mpp_domains.sh: #0 0x7fdd0eb812da in ??? # test_mpp_domains.sh: #1 0x7fdd0eb80503 in ??? # test_mpp_domains.sh: #2 0x7fdd0e7b3f1f in ??? # test_mpp_domains.sh: #3 0x7fdd0f34610c in ??? # test_mpp_domains.sh: #4 0x7fdd0f3e861d in ??? # test_mpp_domains.sh: #5 0x7fdd0f3e8de3 in ??? # test_mpp_domains.sh: #6 0x7fdd0f3f125b in ??? # test_mpp_domains.sh: #7 0x7fdd0f3f471f in ??? # test_mpp_domains.sh: #8 0x7fdd0f3f4ccb in ??? # test_mpp_domains.sh: #9 0x55e05fa55b43 in ??? # test_mpp_domains.sh: #10 0x55e05fa3a5fd in ??? # test_mpp_domains.sh: #11 0x55e05fb6b197 in ??? # test_mpp_domains.sh: #12 0x7fdd0e796b96 in ??? # test_mpp_domains.sh: #13 0x55e05fa36779 in ??? # test_mpp_domains.sh: #14 0xffffffffffffffff in ??? # test_mpp_domains.sh: =================================================================================== # test_mpp_domains.sh: = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES # test_mpp_domains.sh: = PID 11033 RUNNING AT mikado # test_mpp_domains.sh: = EXIT CODE: 139 # test_mpp_domains.sh: = CLEANING UP REMAINING PROCESSES # test_mpp_domains.sh: = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES # test_mpp_domains.sh: =================================================================================== # test_mpp_domains.sh: YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Segmentation fault (signal 11) # test_mpp_domains.sh: This typically refers to a problem with your application. # test_mpp_domains.sh: Please see the FAQ page for debugging suggestions PASS: test_mpp_domains.sh 14 15: Test Get Nbr ERROR: test_mpp_domains.sh - exited with status 1

In stead of commenting out the entire test, please simply skip the test.

edwardhartnett · 2019-11-06T12:16:41Z

test_fms/mpp_io/test_mpp_io.bats

-   run mpirun -n 2 ./test_mpp_io
-   [ "$status" -eq 0 ]
-}
+# @test "MPP_IO runs with multiple MPI processes" {


If I don't comment out this test, I get:

PASS: test_mpp_io.sh 1 MPP_IO runs with single MPI processes FAIL: test_mpp_io.sh 2 MPP_IO runs with multiple MPI processes # test_mpp_io.sh: (in test file test_mpp_io.bats, line 26) # test_mpp_io.sh: `[ "$status" -eq 0 ]' failed # test_mpp_io.sh: NOTE from PE 0: MPP_DOMAINS_SET_STACK_SIZE: stack size set to 32768. # test_mpp_io.sh: &MPP_IO_NML # test_mpp_io.sh: HEADER_BUFFER_VAL= 16384, # test_mpp_io.sh: GLOBAL_FIELD_ON_ROOT_PE=T, # test_mpp_io.sh: IO_CLOCKS_ON=T, # test_mpp_io.sh: SHUFFLE= 0, # test_mpp_io.sh: DEFLATE_LEVEL= -1, # test_mpp_io.sh: CF_COMPLIANCE=F, # test_mpp_io.sh: / # test_mpp_io.sh: NOTE from PE 0: MPP_IO_SET_STACK_SIZE: stack size set to 131072. # test_mpp_io.sh: NOTE from PE 0: MPP_SET_STACK_SIZE: stack size set to 1500000. # test_mpp_io.sh: NOTE from PE 0: MPP_DOMAINS_SET_STACK_SIZE: stack size set to 2000000. # test_mpp_io.sh: npes, nx, ny, nz, nt, halo= 2 360 200 50 2 2 # test_mpp_io.sh: Using NEW domaintypes and calls... # test_mpp_io.sh: netCDF single thread write # test_mpp_io.sh: netCDF single thread append # test_mpp_io.sh: netCDF distributed write # test_mpp_io.sh: NOTE from PE 0: MPP_IO_SET_STACK_SIZE: stack size set to 1800000. # test_mpp_io.sh: netCDF single-threaded write # test_mpp_io.sh: FATAL from PE 1: MPP_DO_GLOBAL_FIELD user stack overflow: call mpp_domains_set_stack_size( 3600000) from all PEs. # test_mpp_io.sh: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 1 ERROR: test_mpp_io.sh - exited with status 1

Same as last comment, please skip the test instead of commenting out the test.

edhartnett · 2019-11-12T12:30:18Z

@wrongkindofdoctor can we get some of these PRs merged, so I can continue to turn on set -e in test scripts?

I believe there are several tests that are reported as passing, which are not really passing...

edwardhartnett · 2019-11-21T18:22:48Z

Guys, any progress here? I would like to get the rest of these changes merged...

Thanks! Ed

underwoo

Please skip the two failing tests. We can determine why these are failing (or completely replace them) when the new FMS/MPP rewrite is completed.

underwoo · 2019-11-26T14:27:09Z

test_fms/mpp/test_mpp_domains.bats

+#       skip "Fails on Travis"
+#     fi
+#     sed "s/check_parallel = .false./check_parallel = .true./" input.nml_base > input.nml
+#     run mpirun -n 6 ./test_mpp_domains


In stead of commenting out the entire test, please simply skip the test.

underwoo · 2019-11-26T14:27:40Z

test_fms/mpp_io/test_mpp_io.bats

-   run mpirun -n 2 ./test_mpp_io
-   [ "$status" -eq 0 ]
-}
+# @test "MPP_IO runs with multiple MPI processes" {


Same as last comment, please skip the test instead of commenting out the test.

edwardhartnett · 2019-11-26T15:42:21Z

OK, I've changed to use skip as requested. Thanks for the feedback @underwoo !

edwardhartnett · 2019-11-29T16:46:09Z

This PR can be skipped if #228 is to be merged soon. #228 passes all my CI checks. All tests which fail (and there are a bunch) are commented out in the shell script tests, instead of being skipped in the bats tests.

edhartnett added 4 commits November 6, 2019 04:54

changed some test runs to use 6 processors instead of 2

6451f34

turned of failing test

d5c3a88

turned of failing test

855ba2f

turned of failing test

0fe709f

edwardhartnett commented Nov 6, 2019

View reviewed changes

wrongkindofdoctor requested a review from underwoo November 6, 2019 14:59

wrongkindofdoctor added the enhancement Issue/PR for a modification that increases performance, improves syntax, or adds functionality. label Nov 6, 2019

wrongkindofdoctor self-assigned this Nov 6, 2019

This was referenced Nov 6, 2019

Fixing more namelist issues #204

Closed

Starting to turn on set -e for test scripts #206

Closed

underwoo requested changes Nov 26, 2019

View reviewed changes

edwardhartnett added 3 commits November 26, 2019 08:19

merged master

0322262

skipped tests instead of commenting out

7c1d88e

skipped tests

89e922c

edwardhartnett changed the title ~~commented out some tests to get make check to pass~~ skipped some tests to get make check to pass Nov 26, 2019

underwoo approved these changes Nov 29, 2019

View reviewed changes

underwoo added this to the 2020.01 milestone Nov 29, 2019

edwardhartnett closed this Nov 29, 2019

edwardhartnett deleted the ejh_jenkins branch June 11, 2020 11:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

skipped some tests to get make check to pass #200

skipped some tests to get make check to pass #200

edwardhartnett commented Nov 6, 2019

edwardhartnett Nov 6, 2019

underwoo Nov 26, 2019

edwardhartnett Nov 6, 2019

underwoo Nov 26, 2019

edhartnett commented Nov 12, 2019

edwardhartnett commented Nov 21, 2019

underwoo left a comment

underwoo Nov 26, 2019

underwoo Nov 26, 2019

edwardhartnett commented Nov 26, 2019

edwardhartnett commented Nov 29, 2019

skipped some tests to get make check to pass #200

skipped some tests to get make check to pass #200

Conversation

edwardhartnett commented Nov 6, 2019

edwardhartnett Nov 6, 2019

Choose a reason for hiding this comment

underwoo Nov 26, 2019

Choose a reason for hiding this comment

edwardhartnett Nov 6, 2019

Choose a reason for hiding this comment

underwoo Nov 26, 2019

Choose a reason for hiding this comment

edhartnett commented Nov 12, 2019

edwardhartnett commented Nov 21, 2019

underwoo left a comment

Choose a reason for hiding this comment

underwoo Nov 26, 2019

Choose a reason for hiding this comment

underwoo Nov 26, 2019

Choose a reason for hiding this comment

edwardhartnett commented Nov 26, 2019

edwardhartnett commented Nov 29, 2019