Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ww3 scotch #849

Merged
merged 42 commits into from
Mar 1, 2023
Merged

ww3 scotch #849

merged 42 commits into from
Mar 1, 2023

Conversation

aliabdolali
Copy link
Contributor

Pull Request Summary

Add SCOTCH option for geographical domain decomposition parallelization for unstr WW3.

Description

SCOTCH library has been added as an option for the domain decomposition for unstr WW3.
When PDLIB is used, one dependent switch (METIS or SCOTCH) is required.
GNU and Cmake builds are extended to support PDLIB/SCOTCH and PDLIB/METIS.
The default combination in the existing tests with PDLIB is now with scotch (ww3_tp2.17, ww3_tp2.6 and ww3_tp2.21). One test is added to ww3_tp2.21 to check METIS.

authors: @aliabdolali and @aronroland

Please also include the following information:

  • Add any suggestions for a reviewer @thesser1
  • Mention any labels that should be added: enhancement, new feature
  • Are answer changes expected from this PR? : out_grd change, out_pnt change, restart file change, Regression test

Issue(s) addressed

Commit Message

Add SCOTCH library for geographical domain decomposition

Check list

Testing

  • How were these changes tested?

With intel and gfortran compilers with gnu and cmake build

  • Are the changes covered by regression tests? (If not, why? Do new tests need to be added?)

Add SCOTCH to the default input/switch_PDLIB for ww3_tp2.17, ww3_tp2.6 and ww3_tp2.21.
One test is added to ww3_tp2.21/input/switch_PDLIB_METIS to check PARMETIS.

  • Have the matrix regression tests been run (if yes, please note HPC and compiler)?

HERA with intel compiler and hpc-stack

  • Please indicate the expected changes in the regression test output, (Note the list of known non-identical tests.)

ww3_tp2.17, ww3_tp2.6 and ww3_tp2.21

  • Please provide the summary output of matrix.comp (matrix.Diff.txt, matrixCompFull.txt and matrixCompSummary.txt):

JessicaMeixner-NOAA
JessicaMeixner-NOAA reviewed Nov 7, 2022
View reviewed changes
model/src/CMakeLists.txt Outdated
endif()
endif()

if("PDLIB" IN_LIST switches)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not have this in the same "if("PDLIB" IN_LIST switches)" if statement as above?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@JessicaMeixner-NOAA
Copy link
Collaborator

@ukmo-ccbunney @mickaelaccensi @sbrus89 - This PR update requires the SCOTCH library to be built for regression testing, assuming you are running unstructured grid tests, therefore you should probably all review this PR, which would require building SCOTCH. Please let us know if you have any questions, issues or concerns.

@JessicaMeixner-NOAA
Copy link
Collaborator

@aliabdolali I believe the automatic checks are failing are due to model/bin/switch_ite_pdlib not having either the METIS or the SCOTCH switch. I believe: switch_ite_pdlib, switch_ugdev2, switch_USACE_1 and switch_USACE_2 should all have SCOTCH added and then maybe @mickaelaccensi can weigh in whether he would like switch_Ifremer2_pdlib to have SCOTCH or METIS.

@aliabdolali
Copy link
Contributor Author

@aliabdolali I believe the automatic checks are failing are due to model/bin/switch_ite_pdlib not having either the METIS or the SCOTCH switch. I believe: switch_ite_pdlib, switch_ugdev2, switch_USACE_1 and switch_USACE_2 should all have SCOTCH added and then maybe @mickaelaccensi can weigh in whether he would like switch_Ifremer2_pdlib to have SCOTCH or METIS.

Good catch, I was wondering why GitHub actions fail. I added SCOTCH to all with PDLIB. For Ifremer, I added Metis, if Mickael wants, I will change it to SCOTCH.

@aliabdolali
Copy link
Contributor Author

@aliabdolali I believe the automatic checks are failing are due to model/bin/switch_ite_pdlib not having either the METIS or the SCOTCH switch. I believe: switch_ite_pdlib, switch_ugdev2, switch_USACE_1 and switch_USACE_2 should all have SCOTCH added and then maybe @mickaelaccensi can weigh in whether he would like switch_Ifremer2_pdlib to have SCOTCH or METIS.

Good catch, I was wondering why GitHub actions fail. I added SCOTCH to all with PDLIB. For Ifremer, I added Metis, if Mickael wants, I will change it to SCOTCH.

still failing ... @MatthewMasarik-NOAA any thought?

@JessicaMeixner-NOAA
Copy link
Collaborator

Screen Shot 2022-11-09 at 9 28 49 AM
@aliabdolali the checks are having issues with the scotch library it seems

@JessicaMeixner-NOAA
Copy link
Collaborator

@aliabdolali maybe this could be of use: https://github.com/NOAA-EMC/WW3/wiki/Code-Management#automatic-tests-update-for-github-action

@aliabdolali
Copy link
Contributor Author

@aliabdolali maybe this could be of use: https://github.com/NOAA-EMC/WW3/wiki/Code-Management#automatic-tests-update-for-github-action

Awesome, I forgot that I added myself to the wiki. thanks for pointing me to it.

@aliabdolali aliabdolali changed the title Ww3 scotch dev ww3 scotch Nov 9, 2022
@MatthewMasarik-NOAA
Copy link
Collaborator

still failing ... @MatthewMasarik-NOAA any thought?

Looks like you guys solved it. I'll check back if these current builds fail

@MatthewMasarik-NOAA
Copy link
Collaborator

@aliabdolali looks like the same failure with ite_pdlib. The line that the error points to is 161 in model/src/CMakeLists.txt. This block seems to deal with Parmetis (vs. SCOTCH), so is it not finding the Parmetis library in this case?

It also mentions log file /home/runner/work/WW3/WW3/ww3/build/CMakeFiles/CMakeOutput.log. Anyone know how to access that?

@ukmo-ccbunney
Copy link
Collaborator

I will update with results of the regtest run as soon as I have access again...

I've managed to run the ww3_tp2.17/work_b regression test using SCOTCH with our GNU compiler on a Cray EXZ.
All seemed to go well. There are some small differences in the output field compared to the develop (non-scotch) branch, but I take it these are expected?

image

@aliabdolali
Copy link
Contributor Author

I will update with results of the regtest run as soon as I have access again...

I've managed to run the ww3_tp2.17/work_b regression test using SCOTCH with our GNU compiler on a Cray EXZ.

All seemed to go well. There are some small differences in the output field compared to the develop (non-scotch) branch, but I take it these are expected?

image

Hi @ukmo-ccbunney thanks for checking scotch on your hpc, and yes, since decomposition is different,we expect changes in the results.

@aronroland
Copy link
Collaborator

we will sort that out as we check the b4b for the explicit scheme ...

@aronroland aronroland mentioned this pull request Feb 27, 2023
Merged
4 tasks
@thesser1
Merge pull request #10 from MatthewMasarik-NOAA/ww3_scotch_dev
67c91ff 
SCOTCH_PATH updates for RDHPCS machines
@JessicaMeixner-NOAA
Copy link
Collaborator

@thesser1 @sbrus89 @mickaelaccensi @ukmo-ccbunney we hope we can have official reviews from each of you at some point by 3/1. Please let us know if you've run into issues, have concerns, etc.

@JessicaMeixner-NOAA
Copy link
Collaborator

@aliabdolali please merge the develop branch into your branch for this PR to fix the CI issues.

@aliabdolali
Copy link
Contributor Author

Your fix did not fix the problem for neither of wise nor scotch pr

@JessicaMeixner-NOAA
Copy link
Collaborator

Your fix did not fix the problem for neither of wise nor scotch pr

@aliabdolali happy to help look into this, but for this PR I see that the tests are still running and I do not yet see a failure, did I miss something on this PR? I'll check the other PR for a failure next.

@MatthewMasarik-NOAA
Copy link
Collaborator

MatthewMasarik-NOAA commented Feb 28, 2023 •
edited
Loading

Hi all, I wanted to point out I've updated the SCOTCH build description on our FAQ to cover how to turn pthreads off in the cmake step. See the bottom of: https://github.com/NOAA-EMC/WW3/wiki/FAQs-page#how-to-install-scotch
fyi, @sbrus89

@aliabdolali
Copy link
Contributor Author

Hi all, I wanted to point out I've updated the SCOTCH build description on our FAQ to cover how to turn pthreads off in the cmake step. See the bottom of: https://github.com/NOAA-EMC/WW3/wiki/FAQs-page#how-to-install-scotch fyi, @sbrus89

this is great Matt, could you add a minimum GNU version to the wiki, so users know what would be the requirements? I tried version 9.2 the min, but you might know better than me

@MatthewMasarik-NOAA
Copy link
Collaborator

Hi @aliabdolali, @JessicaMeixner-NOAA created an issue (https://gitlab.inria.fr/scotch/scotch/-/issues/21) on the SCOTCH gitlab page a few days ago asking about this. We haven't heard any response yet but will be updating here once we do

ukmo-ccbunney
ukmo-ccbunney approved these changes Mar 1, 2023
View reviewed changes
Copy link
Collaborator

@ukmo-ccbunney ukmo-ccbunney left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have run regression tests ww3_tp2.17, ww3_tp2.6 and ww3_tp2.21 on our dev HPC machine using GNU gcc/gfortran version 12.1.0.

SCOTCH compiled fine and the regression tests ran ok. There are small differences compared to the develop branch, as expected.

I could not run on our operational HPC as the compiler and Bison versions are too old.

I have not tried with the Cray compiler on our dev HPC yet, only GNU.

@aliabdolali
Copy link
Contributor Author

Super @ukmo-ccbunney

@MatthewMasarik-NOAA
Copy link
Collaborator

I'm completing my review and will post momentarily. @thesser1 if you're available to add your review as well please do

@JessicaMeixner-NOAA
Copy link
Collaborator

@mickaelaccensi and @sbrus89 --- I hope you are both still testing this. If you have any issues please let us know. @MatthewMasarik-NOAA and I hope that we will be able to merge this at the end of today. If issues arise we can always come back to address them. @thesser1 -- we should get your approval and will wait for it, so if there are issues we should be aware of or if you have a known timeline for when we should get this review, please let us know.

MatthewMasarik-NOAA
MatthewMasarik-NOAA approved these changes Mar 1, 2023
View reviewed changes
Copy link
Collaborator

@MatthewMasarik-NOAA MatthewMasarik-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review

Pass.

Testing

Pass.

I've attached the matrix.comp output for hera and orion (same as posted on erdc/pull/10).

Note: In both the orion and hera output's there were a large number of (0 files differ) which turned out to be detecting linked files from the regtest input/ directory to the work*/ directories. For this reason and the ongoing mod_def issue for unstructured, I added some annotation to the Summary output (for orion. it is the same for the same files on hera).

orion

**********************************************************************                 
********************* non-identical cases ****************************                 
**********************************************************************                 
                                                                                       
1) (0 files differ): due to file (track_i.outer) linked from input/ to work/           
--                                                                                     
mww3_test_02/./work_PR1_c                     (0 files differ)                         
mww3_test_02/./work_PR3_UNO_d_c                     (0 files differ)                   
mww3_test_02/./work_PR3_UNO_b                     (0 files differ)                     
mww3_test_02/./work_PR2_UQ_d                     (0 files differ)                      
mww3_test_02/./work_PR3_UQ_d                     (0 files differ)                      
mww3_test_02/./work_PR2_UQ_a                     (0 files differ)                      
mww3_test_02/./work_PR1_d                     (0 files differ)                         
mww3_test_02/./work_PR2_UQ_c                     (0 files differ)                      
mww3_test_02/./work_PR3_UQ_MPI_c_c                     (0 files differ)                
mww3_test_02/./work_PR2_UNO_MPI_b                     (0 files differ)                 
mww3_test_02/./work_PR2_UQ_MPI_b                     (0 files differ)                  
mww3_test_02/./work_PR1_a                     (0 files differ)                         
mww3_test_02/./work_PR2_UNO_c                     (0 files differ)                     
mww3_test_02/./work_PR3_UNO_a                     (0 files differ)                     
mww3_test_02/./work_PR3_UNO_MPI_a_c                     (0 files differ)               
mww3_test_02/./work_PR1_MPI_c                     (0 files differ)                     
mww3_test_02/./work_PR3_UNO_d                     (0 files differ)                     
mww3_test_02/./work_PR3_UNO_c                     (0 files differ)                     
mww3_test_02/./work_PR3_UQ_a_c                     (0 files differ)                    
mww3_test_02/./work_PR2_UNO_d                     (0 files differ)                     
mww3_test_02/./work_PR2_UNO_MPI_d                     (0 files differ)                 
mww3_test_02/./work_PR3_UQ_c_c                     (0 files differ)                    
mww3_test_02/./work_PR3_UNO_MPI_c                     (0 files differ)                 
mww3_test_02/./work_PR3_UNO_MPI_b                     (0 files differ)                 
mww3_test_02/./work_PR2_UNO_MPI_a                     (0 files differ)                 
mww3_test_02/./work_PR3_UNO_b_c                     (0 files differ)                   
mww3_test_02/./work_PR3_UNO_MPI_d_c                     (0 files differ)               
mww3_test_02/./work_PR3_UQ_d_c                     (0 files differ)                    
mww3_test_02/./work_PR3_UQ_MPI_a_c                     (0 files differ)                
mww3_test_02/./work_PR1_MPI_b                     (0 files differ)                     
mww3_test_02/./work_PR3_UQ_MPI_b                     (0 files differ)                  
mww3_test_02/./work_PR2_UQ_MPI_c                     (0 files differ)                  
mww3_test_02/./work_PR3_UNO_a_c                     (0 files differ)                   
mww3_test_02/./work_PR3_UNO_MPI_b_c                     (0 files differ)               
mww3_test_02/./work_PR1_b                     (0 files differ)                         
mww3_test_02/./work_PR3_UQ_a                     (0 files differ)                      
mww3_test_02/./work_PR3_UNO_c_c                     (0 files differ)                   
mww3_test_02/./work_PR3_UQ_MPI_d_c                     (0 files differ)                
mww3_test_02/./work_PR3_UNO_MPI_d                     (0 files differ)                 
mww3_test_02/./work_PR2_UNO_a                     (0 files differ)                     
mww3_test_02/./work_PR2_UQ_MPI_d                     (0 files differ)                  
mww3_test_02/./work_PR3_UQ_b_c                     (0 files differ)                    
mww3_test_02/./work_PR3_UQ_MPI_b_c                     (0 files differ)                
mww3_test_02/./work_PR2_UQ_MPI_a                     (0 files differ)                  
mww3_test_02/./work_PR2_UNO_b                     (0 files differ)                     
mww3_test_02/./work_PR3_UNO_MPI_c_c                     (0 files differ)               
mww3_test_02/./work_PR2_UNO_MPI_c                     (0 files differ)                 
mww3_test_02/./work_PR3_UQ_MPI_c                     (0 files differ)                  
mww3_test_02/./work_PR3_UQ_MPI_d                     (0 files differ)                  
mww3_test_02/./work_PR3_UQ_b                     (0 files differ)                      
mww3_test_02/./work_PR3_UNO_MPI_a                     (0 files differ)                 
mww3_test_02/./work_PR1_MPI_d                     (0 files differ)                     
mww3_test_02/./work_PR2_UQ_b                     (0 files differ)                      
mww3_test_02/./work_PR1_MPI_a                     (0 files differ)                     
mww3_test_02/./work_PR3_UQ_MPI_a                     (0 files differ)                  
mww3_test_02/./work_PR3_UQ_c                     (0 files differ)                      
--                                                                                     
                                                                                       
-- 2) usual non-b4b                                                                    
mww3_test_03/./work_PR3_UQ_MPI_d2                     (16 files differ)                
mww3_test_03/./work_PR3_UNO_MPI_e                     (1 files differ)                 
mww3_test_03/./work_PR3_UQ_MPI_d2_c                     (15 files differ)              
mww3_test_03/./work_PR2_UQ_MPI_d2                     (15 files differ)                
mww3_test_03/./work_PR3_UNO_MPI_d2                     (12 files differ)               
mww3_test_03/./work_PR2_UNO_MPI_d2                     (17 files differ)               
mww3_test_03/./work_PR2_UNO_MPI_e                     (1 files differ)                 
mww3_test_03/./work_PR1_MPI_d2                     (11 files differ)                   
mww3_test_03/./work_PR3_UNO_MPI_d2_c                     (16 files differ)             
mww3_test_03/./work_PR3_UQ_MPI_e                     (1 files differ)                  
mww3_test_03/./work_PR3_UQ_MPI_e_c                     (1 files differ)                
ww3_tp2.10/./work_MPI_OMPH                     (7 files differ)                        
ww3_tp2.16/./work_MPI_OMPH                     (4 files differ)                        
ww3_ufs1.3/./work_a                     (3 files differ)                               
--                                                                                     
                                                                                       
3) (0 files differ): due to file (anl.grbtxt) linked from input/ to work/              
--                                                                                     
ww3_ta1/./work_UPD2_U_cap                     (0 files differ)                         
ww3_ta1/./work_UPD6_O                     (0 files differ)                             
ww3_ta1/./work_UPD0F_U                     (0 files differ)                            
ww3_ta1/./work_UPD5_U                     (0 files differ)                             
ww3_ta1/./work_UPD2_U                     (0 files differ)                             
ww3_ta1/./work_UPD2_O                     (0 files differ)                             
ww3_ta1/./work_UPD3_U_cap                     (0 files differ)                         
ww3_ta1/./work_UPD3_U                     (0 files differ)                             
ww3_ta1/./work_UPD0F_O                     (0 files differ)                            
ww3_ta1/./work_UPD6_U_cap                     (0 files differ)                         
ww3_ta1/./work_UPD6_U                     (0 files differ)                             
ww3_ta1/./work_UPD5_O                     (0 files differ)                             
ww3_ta1/./work_UPD5_U_cap                     (0 files differ)                         
ww3_ta1/./work_UPD3_O                     (0 files differ)                             
--                                                                                     
                                                                                       
4) (1 file differ): due to 'APPLE partioning' string in file OUTPUT_TOY.txt            
--                                                                                     
ww3_tp2.14/./work_OASACM6                     (1 files differ)                         
--                                                                                     
                                                                                       
5) (1 file differ): due to unstructured mod_def's                                      
--                                                                                     
ww3_tp2.17/./work_mc1                     (1 files differ)                             
ww3_tp2.17/./work_ma                     (1 files differ)                              
ww3_tp2.17/./work_mc                     (1 files differ)                              
ww3_tp2.17/./work_mb                     (1 files differ)                              
ww3_tp2.17/./work_b                     (1 files differ)                               
ww3_tp2.17/./work_c                     (1 files differ)                               
ww3_tp2.17/./work_a                     (1 files differ)                               
ww3_tp2.17/./work_ma1                     (1 files differ)                             
ww3_tp2.6/./work_ST4                     (1 files differ)                              
ww3_tp2.6/./work_pdlib                     (1 files differ)                            
ww3_tp2.6/./work_ST0                     (1 files differ)                              
ww3_ts4/./work_ug_MPI                     (1 files differ)                             
--                                                                                                                                                             
**********************************************************************                 
************************ identical cases *****************************                 
**********************************************************************

hera

**********************************************************************       
********************* non-identical cases ****************************       
**********************************************************************       
mww3_test_02/./work_PR1_c                     (0 files differ)               
mww3_test_02/./work_PR2_UNO_c                     (0 files differ)           
mww3_test_02/./work_PR3_UQ_MPI_a                     (0 files differ)        
mww3_test_02/./work_PR1_MPI_d                     (0 files differ)           
mww3_test_02/./work_PR3_UNO_MPI_b_c                     (0 files differ)     
mww3_test_02/./work_PR3_UNO_c_c                     (0 files differ)         
mww3_test_02/./work_PR3_UQ_MPI_c                     (0 files differ)        
mww3_test_02/./work_PR2_UNO_MPI_d                     (0 files differ)       
mww3_test_02/./work_PR2_UNO_d                     (0 files differ)           
mww3_test_02/./work_PR2_UQ_MPI_a                     (0 files differ)        
mww3_test_02/./work_PR3_UQ_MPI_a_c                     (0 files differ)      
mww3_test_02/./work_PR3_UNO_MPI_d_c                     (0 files differ)     
mww3_test_02/./work_PR1_d                     (0 files differ)               
mww3_test_02/./work_PR2_UQ_c                     (0 files differ)            
mww3_test_02/./work_PR3_UQ_b                     (0 files differ)            
mww3_test_02/./work_PR3_UQ_a                     (0 files differ)            
mww3_test_02/./work_PR1_a                     (0 files differ)               
mww3_test_02/./work_PR3_UNO_MPI_d                     (0 files differ)       
mww3_test_02/./work_PR3_UNO_a_c                     (0 files differ)         
mww3_test_02/./work_PR2_UQ_MPI_b                     (0 files differ)        
mww3_test_02/./work_PR3_UNO_d                     (0 files differ)           
mww3_test_02/./work_PR3_UNO_MPI_c_c                     (0 files differ)     
mww3_test_02/./work_PR3_UNO_MPI_a_c                     (0 files differ)     
mww3_test_02/./work_PR2_UQ_a                     (0 files differ)            
mww3_test_02/./work_PR3_UNO_MPI_b                     (0 files differ)       
mww3_test_02/./work_PR3_UNO_b_c                     (0 files differ)         
mww3_test_02/./work_PR1_b                     (0 files differ)               
mww3_test_02/./work_PR2_UNO_b                     (0 files differ)           
mww3_test_02/./work_PR3_UQ_MPI_c_c                     (0 files differ)      
mww3_test_02/./work_PR2_UQ_d                     (0 files differ)            
mww3_test_02/./work_PR3_UQ_c                     (0 files differ)            
mww3_test_02/./work_PR3_UNO_MPI_a                     (0 files differ)       
mww3_test_02/./work_PR1_MPI_c                     (0 files differ)           
mww3_test_02/./work_PR3_UNO_b                     (0 files differ)           
mww3_test_02/./work_PR1_MPI_b                     (0 files differ)           
mww3_test_02/./work_PR3_UQ_a_c                     (0 files differ)          
mww3_test_02/./work_PR3_UQ_MPI_b_c                     (0 files differ)      
mww3_test_02/./work_PR3_UQ_d_c                     (0 files differ)          
mww3_test_02/./work_PR1_MPI_a                     (0 files differ)           
mww3_test_02/./work_PR3_UQ_MPI_d_c                     (0 files differ)      
mww3_test_02/./work_PR3_UQ_MPI_d                     (0 files differ)        
mww3_test_02/./work_PR3_UQ_d                     (0 files differ)            
mww3_test_02/./work_PR3_UNO_d_c                     (0 files differ)         
mww3_test_02/./work_PR3_UQ_c_c                     (0 files differ)          
mww3_test_02/./work_PR2_UNO_MPI_b                     (0 files differ)       
mww3_test_02/./work_PR3_UNO_MPI_c                     (0 files differ)       
mww3_test_02/./work_PR2_UQ_b                     (0 files differ)            
mww3_test_02/./work_PR3_UNO_c                     (0 files differ)           
mww3_test_02/./work_PR3_UNO_a                     (0 files differ)           
mww3_test_02/./work_PR2_UQ_MPI_d                     (0 files differ)        
mww3_test_02/./work_PR2_UQ_MPI_c                     (0 files differ)        
mww3_test_02/./work_PR2_UNO_a                     (0 files differ)           
mww3_test_02/./work_PR2_UNO_MPI_c                     (0 files differ)       
mww3_test_02/./work_PR2_UNO_MPI_a                     (0 files differ)       
mww3_test_02/./work_PR3_UQ_b_c                     (0 files differ)          
mww3_test_02/./work_PR3_UQ_MPI_b                     (0 files differ)        
mww3_test_03/./work_PR1_MPI_e                     (1 files differ)           
mww3_test_03/./work_PR3_UQ_MPI_e_c                     (1 files differ)      
mww3_test_03/./work_PR2_UQ_MPI_e                     (1 files differ)        
mww3_test_03/./work_PR2_UNO_MPI_e                     (1 files differ)       
mww3_test_03/./work_PR2_UNO_MPI_d2                     (15 files differ)     
mww3_test_03/./work_PR1_MPI_d2                     (6 files differ)          
mww3_test_03/./work_PR3_UNO_MPI_d2_c                     (15 files differ)   
mww3_test_03/./work_PR3_UQ_MPI_d2_c                     (16 files differ)    
mww3_test_03/./work_PR3_UNO_MPI_d2                     (15 files differ)     
mww3_test_03/./work_PR2_UQ_MPI_d2                     (15 files differ)      
mww3_test_03/./work_PR3_UNO_MPI_e_c                     (1 files differ)     
mww3_test_03/./work_PR3_UQ_MPI_d2                     (15 files differ)
ww3_ta1/./work_UPD0F_O                     (0 files differ)                  
ww3_ta1/./work_UPD0F_U                     (0 files differ)                  
ww3_ta1/./work_UPD2_U_cap                     (0 files differ)               
ww3_ta1/./work_UPD3_U                     (0 files differ)                   
ww3_ta1/./work_UPD5_U_cap                     (0 files differ)               
ww3_ta1/./work_UPD6_O                     (0 files differ)                   
ww3_ta1/./work_UPD2_U                     (0 files differ)                   
ww3_ta1/./work_UPD5_O                     (0 files differ)                   
ww3_ta1/./work_UPD5_U                     (0 files differ)                   
ww3_ta1/./work_UPD3_U_cap                     (0 files differ)               
ww3_ta1/./work_UPD3_O                     (0 files differ)                   
ww3_ta1/./work_UPD6_U                     (0 files differ)                   
ww3_ta1/./work_UPD2_O                     (0 files differ)                   
ww3_ta1/./work_UPD6_U_cap                     (0 files differ)               
ww3_tp2.10/./work_MPI_OMPH                     (6 files differ)              
ww3_tp2.16/./work_MPI_OMPH                     (4 files differ)              
ww3_tp2.17/./work_ma                     (1 files differ)                    
ww3_tp2.17/./work_a                     (1 files differ)                     
ww3_tp2.17/./work_mc1                     (1 files differ)                   
ww3_tp2.17/./work_mb                     (1 files differ)                    
ww3_tp2.17/./work_mc                     (1 files differ)                    
ww3_tp2.17/./work_ma1                     (1 files differ)                   
ww3_tp2.17/./work_c                     (1 files differ)                     
ww3_tp2.17/./work_b                     (1 files differ)                     
ww3_ufs1.3/./work_a                     (3 files differ)                                                                                  
**********************************************************************       
************************ identical cases *****************************       
**********************************************************************

@MatthewMasarik-NOAA
Copy link
Collaborator

Awesome, thanks @thesser1 !

sbrus89
sbrus89 approved these changes Mar 1, 2023
View reviewed changes
Copy link
Collaborator

@sbrus89 sbrus89 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was able to get this working on our systems.

@MatthewMasarik-NOAA
Copy link
Collaborator

Awesome. Thanks for your review @sbrus89. I'm curious if you compiled with gnu or other (intel)?

@sbrus89
Copy link
Collaborator

sbrus89 commented Mar 1, 2023

@MatthewMasarik-NOAA, I used Gnu 10.2.0

@MatthewMasarik-NOAA
Copy link
Collaborator

Great, thanks for confirming. And thanks for your work helping test this @sbrus89!

@aliabdolali
Copy link
Contributor Author

It is great to have confirmation with different compilers.

@JessicaMeixner-NOAA
Copy link
Collaborator

Thank you @aliabdolali and @aronroland for adding the SCOTCH capability. Thank you to @AlexanderRichert-NOAA @aerorahul @MatthewMasarik-NOAA and the many others who helped get us through cmake, orion and other issues and get this PR past the finish line! Thank you to @ukmo-ccbunney @sbrus89 @thesser1 for your reviews! This is a huge accomplishment and we look forward to continuing to work on the outstanding issues resolving the scaling and mpi b4b (which are likely the cause of diffs between scotch and metis).

@JessicaMeixner-NOAA JessicaMeixner-NOAA merged commit ea0de58 into NOAA-EMC:develop Mar 1, 2023
@aliabdolali
Copy link
Contributor Author

Yay, finally, this PR hit a record !!!!
Thanks to @ukmo-ccbunney @sbrus89 @MatthewMasarik-NOAA @JessicaMeixner-NOAA @aerorahul for checking our development.

@thesser1 thesser1 deleted the ww3_scotch_dev branch October 26, 2023 13:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@MatthewMasarik-NOAA MatthewMasarik-NOAA MatthewMasarik-NOAA approved these changes

@ukmo-ccbunney ukmo-ccbunney ukmo-ccbunney approved these changes

@JessicaMeixner-NOAA JessicaMeixner-NOAA JessicaMeixner-NOAA approved these changes

@sbrus89 sbrus89 sbrus89 approved these changes

@thesser1 thesser1 thesser1 approved these changes

@mickaelaccensi mickaelaccensi Awaiting requested review from mickaelaccensi

Assignees

@aronroland aronroland

@JessicaMeixner-NOAA JessicaMeixner-NOAA

@aliabdolali aliabdolali

@MatthewMasarik-NOAA MatthewMasarik-NOAA

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Domain decomposition library (SCOTCH)
8 participants
@aliabdolali @JessicaMeixner-NOAA @MatthewMasarik-NOAA @AlexanderRichert-NOAA @aronroland @thesser1 @ukmo-ccbunney @sbrus89