Release/1.2.0

[tcojean]

This PR is to push the 1.2.0 release into develop. I start with develop since it makes reviewing the changes easier.

The documentation should appear at this link after a successful build:
https://ginkgo-project.github.io/ginkgo/doc/release/1.2.0/

The various checks (sanitizers in particular) should be available here after a successful build:
https://my.cdash.org/index.php?project=Ginkgo+Project&date=2020-06-23

The next post will have a proposition for a changelog. Feel free to update it or propose improvements.

[tcojean]

The Ginkgo team is proud to announce the new minor release of Ginkgo version
1.2.0. This release brings full HIP support to Ginkgo, new preconditioners
(ParILUT, ISAI), conversion between double and float for all LinOps, and many
more features and fixes.

Supported systems and requirements:

• For all platforms, cmake 3.9+
• Linux and MacOS
  • gcc: 5.3+, 6.3+, 7.3+, all versions after 8.1+
  • clang: 3.9+
  • Intel compiler: 2017+
  • Apple LLVM: 8.0+
  • CUDA module: CUDA 9.0+
  • HIP module: ROCm 2.8+
• Windows
  • MinGW and CygWin: gcc 5.3+, 6.3+, 7.3+, all versions after 8.1+
  • Microsoft Visual Studio: VS 2017 15.7+
  • CUDA module: CUDA 9.0+, Microsoft Visual Studio
  • OpenMP module: MinGW or CygWin.

The current known issues can be found in the known issues page.

Additions

Here are the main additions to the Ginkgo library. Other thematic additions are listed below.

• Add full HIP support to Ginkgo #344, #357, #384, #373, #391, #396, #395, #393, #404, #439, #443, #567
• Add a new ISAI preconditioner #489, #502, #512, #508, #520
• Add support for ParILUT and ParICT factorization with ILU preconditioners #400
• Add a new BiCG solver #438
• Add a new permutation matrix format #352, #469
• Add CSR SpGEMM support #386, #398, #418, #457
• Add CSR SpGEAM support #556
• Make all solvers and preconditioners transposable #535
• Add CsrBuilder and CooBuilder for intrusive access to matrix arrays #437
• Add a standard-compliant allocator based on the Executors #504
• Support conversions for all LinOp between double and float #521
• Add a new boolean to the CUDA and HIP executors to control DeviceReset (default off) #557
• Add a relaxation factor to IR to represent Richardson Relaxation #574
• Add two new stopping criteria, for relative (to norm(b)) and absolute residual norm #577

Example additions

• Templatize all examples to simplify changing the precision #513
• Add a new adaptive precision block-Jacobi example #507
• Add a new IR example #522
• Add a new Mixed Precision Iterative Refinement example #525
• Add a new example on iterative trisolves in ILU preconditioning #526, #536, #550

Compilation and library changes

• Auto-detect compilation settings based on environment #435, #537
• Add SONAME to shared libraries #524
• Add clang-cuda support #543

Other additions

• Add sorting, searching and merging kernels for GPUs #403, #428, #417, #455
• Add gko::as support for smart pointers #493
• Add setters and getters for criterion factories #527
• Add a new method to check whether a solver uses x as an initial guess #531
• Add contribution guidelines #549

Fixes

Algorithms

• Improve the classical CSR strategy's performance #401
• Improve the CSR automatical strategy #407, #559
• Memory, speed improvements to the ELL kernel #411
• Multiple improvements and fixes to ParILU #419, #427, #429, #456, #544
• Fix multiple issues with GMRES #481, #523, #575
• Optimize OpenMP matrix conversions #505
• Ensure the linearity of the ILU preconditioner #506
• Fix IR's use of the advanced apply #522
• Fix empty matrices conversions and add tests #560

Other core functionalities

• Fix complex number support in our math header #410
• Fix CUDA compatibility of the main ginkgo header #450
• Fix isfinite issues #465
• Fix the Array::view memory leak and the array/view copy/move #485
• Fix typos preventing use of some interface functions #496
• Fix the gko::dim to abide to the C++ standard #498
• Simplify the executor copy interface #516
• Optimize intermediate storage for Composition #540
• Provide an initial guess for relevant Compositions #561
• Better management of nullptr as criterion #562
• Fix the norm calculations for complex support #564

CUDA and HIP specific

• Use the return value of the atomic operations in our wrappers #405
• Improve the portability of warp lane masks #422
• Extract thread ID computation into a separate function #464
• Reorder kernel parameters for consistency #474
• Fix the use of pragma unroll in HIP #492

Other

• Fix the Ginkgo CMake installation files #414, #553
• Fix the Windows compilation #415
• Always use demangled types in error messages #434, #486
• Add CUDA header dependency to appropriate tests #452
• Fix several sonarqube or compilation warnings #453, #463, #532, #569
• Add shuffle tests #460
• Fix MSVC C2398 error #490
• Fix missing interface tests in test install #558

Tools and ecosystem

Benchmarks

• Add better norm support in the benchmarks #377
• Add CUDA 10.1 generic SpMV support in benchmarks #468, #473
• Add sparse library ILU in benchmarks #487
• Add overhead benchmarking capacities #501
• Allow benchmarking from a matrix list file #503
• Fix benchmarking issue with JSON and non-finite numbers #514
• Fix benchmark logger crashers with OpenMP #565

CI related

• Improvements to the CI setup with HIP compilation #421, #466
• Add MacOSX CI support #470, #488
• Add Windows CI support #471, #488, #510, #566
• Use sanitizers instead of valgrind #476
• Add automatic container generation and update facilities #499
• Fix the CI parallelism settings #517, #538, #539
• Make the codecov patch check informational #519
• Add support for LLVM sanitizers with improved thread sanitizer support #578

Test suite

• Add an assertion for sparsity pattern equality #416
• Add core and reference multiprecision tests support #448
• Speed up GPU tests by avoiding device reset #467
• Change test matrix location string #494

Other

• Add Ginkgo badges from our tools #413
• Update the create_new_algorithm.sh script #420
• Bump copyright and improve license management #436, #433
• Set clang-format minimum requirement #441, #484
• Update git-cmake-format #446, #484
• Disable the development tools by default #442
• Add a script for automatic header formatting #447
• Add GDB pretty printer for gko::Array #509
• Improve compilation speed #533
• Add editorconfig support #546
• Add a compile-time check for header self-sufficiency #552

[tcojean]

Note that for some reason, newer gtest versions do not work well with older CUDA versions:
https://gitlab.com/ginkgo-project/ginkgo-public-ci/-/jobs/608082347

[codecov]

Codecov Report

Merging #576 into develop will increase coverage by 0.00%.
The diff coverage is 82.53%.

@@           Coverage Diff            @@
##           develop     #576   +/-   ##
========================================
  Coverage    84.22%   84.22%           
========================================
  Files          296      296           
  Lines        20653    20655    +2     
========================================
+ Hits         17395    17397    +2     
  Misses        3258     3258           

Impacted Files	Coverage Δ
omp/factorization/par_ilut_kernels.cpp	0.00% <0.00%> (ø)
omp/matrix/csr_kernels.cpp	12.18% <0.00%> (ø)
reference/test/log/convergence.cpp	91.66% <ø> (ø)
omp/factorization/par_ict_kernels.cpp	100.00% <100.00%> (ø)
reference/factorization/par_ict_kernels.cpp	100.00% <100.00%> (ø)
reference/matrix/csr_kernels.cpp	99.50% <100.00%> (+<0.01%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b8daca6...e9dd9ee. Read the comment docs.

[yhmtsai]

For the known issue, do we still lose the support custom type (32/64 bits) of ginkgo atomic_add?
In the atomic.hpp.inc, we bind 64 bit to unsigned long long int, 32 bit to unsigned int. Currently, we do not implement the custom type addition.

[upsj]

Regarding the sanitizers: We could probably get rid of all but the deliberate UBSAN warnings by adding a non-null/non-empty check to Array::operator= and all places where we call executor->copy() on empty data.

[upsj]

And clang's libomp can be built with TSAN support enabled, should I look into setting up a container for that?

[pratikvn]

The README.md still has the old link for the Contributor guidelines. Can you please update that ?

[upsj]

With TSAN-enabled OpenMP runtime, the thread sanitizer run now also looks good.

To get this to work, I needed the following steps:

• Install llvm-10 in the container (Without this, symbols cannot be resolved correctly, since no llvm-symbolizer is available)
• Clone https://github.com/llvm/llvm-project and build & install openmp with -DLIBOMP_TSAN_SUPPORT=ON.
• In the TSAN (and UBSAN) flags, replace static-lib*san by static-libsan, which is the sanitizer library provided by LLVM.
• Replace the libomp.so from x86_64-linux by the newly created libomp (It gets installed in /usr/local/lib by default, I modified the Ginkgo CMakeCache.txt manually to change the linker paths, I guess it should be possible to do this with environment variables or by manually replacing the installed library files?)
• Execute the TSAN CTest command with -j10 --timeout 10000 because some of the tests take > 30min

The found issues are

• Read-After-Write conflicts in ParILU(T)/ParICT iteration as expected
• Write-After-Write conflicts in residual_norm_reduction stopping criterion kernels? I'm not sure why this is considered an issue, but I guess we can ignore it.

[fritzgoebel]

LGTM!

[pratikvn]

I think we can already add a section in README.md for citing the JOSS paper. openjournals/joss-reviews#2260. I think they already reserve the DOI, so we should be able to use that.

[upsj]

LGTM!

[thoasm]

Since this PR is not compiling with GCC 10.1, I would suggest to remove that from the list of supported compilers.

[thoasm]

Are you sure we are compatible with these? I tried it with GCC 9.1 some time ago, and it failed. The error looked very similar to what I get with GCC 10.1 now, which is described in this issue: #579
I think we should remove the last two additions until we resolve the problem.

[upsj]

Maybe we should also list the clang major versions we support, and remove the TODO from AppleClang?

[tcojean]

For clang, I don't think there really is any difference between the major releases in our case, since 3.9 and full C++11 support there has been no major problem. For GCC, the issue was that several major versions exist next to each other, and C++ 11 support was incomplete for example until 5.3 (at least, there were bugs with the compiler). I remember I did the work of fixing the issues we had with 5.3 some time ago, but going further down was too much work.

For AppleClang, as I have no MacOS available I cannot judge.

[upsj]

I see your point about clang. My main hold-up is that since we list versions explicitly for GCC, readers might assume that we only support a very old clang, but that is only a very vague issue.

We could list AppleClang 11, since that's what the Github Action uses.

[tcojean]

I changed this to say all versions after 8.1+. I think this shows that the previous listings are an exception and otherwise we support all versions after the ones listed here.

[tcojean]

@pratikvn I don't see a DOI for the JOSS paper? How can I find it.

[pratikvn]

It has been reserved but not yet registered. https://github.com/openjournals/joss-papers/blob/joss.02260/joss.02260/10.21105.joss.02260.pdf .

Essentially, the doi will be 10.21105/joss.02260. But as it has not yet been registered, we can also wait to add it. But this is will the final doi.

[pratikvn]

LGTM!

[pratikvn]

Here are some details on the Citation file format: https://citation-file-format.github.io/

[tcojean]

Why use this tool and not just plain bibtex? Or markdown with a bibtex code portion (if it works). Is this integrated into github?

Test:

@misc{anzt2020ginkgo,
    title={Ginkgo: A Modern Linear Operator Algebra Framework for High Performance Computing},
    author={Hartwig Anzt and Terry Cojean and Goran Flegar and Fritz Göbel and Thomas Grützmacher and Pratik Nayak and Tobias Ribizel and Yuhsiang Mike Tsai and Enrique S. Quintana-Ortí},
    year={2020},
    eprint={2006.16852},
    archivePrefix={arXiv},
    primaryClass={cs.MS}
}

[pratikvn]

I think the advantage is that it is machine readable. You can also add multiple references to the same citation file format file, for example all the papers that use Ginkgo. https://github.com/citation-file-format/citation-file-format/blob/master/README.md#references-optional. It is also in line with some existing standards: https://peerj.com/articles/cs-86/

[tcojean]

Well, bibtex is also machine readable since there are several parsers and tools designed to use it directly. I think though this citation format is rather when you do not have a paper for your software but create a DOI for every release which can be cited.

In our case though, I would rather not do that, or at least first point to the Interface paper and other topical papers (SpMV performance, HIP portability, ... for example), since otherwise citations get lost in many different references and you lose traction in google scholar and all other platforms.

[pratikvn]

Okay. That makes sense. I guess we can add a section in the README on "How to cite Ginkgo" and add the bibTeX data as you mentioned in your comment.

[thoasm]

Since it seems to be an ArchLinux problem since it works on Ubuntu, LGTM!

[yhmtsai]

LGTM

[yhmtsai]

Is any usage of the {#citing_ginkgo}?

[tcojean]

Yes, see doc/DoxygenLayout.xml

[tcojean]

If there is no issues with the recent changes, including the new CITING.md page, then I will proceed with the release tomorrow.

[tcojean]

I also used clang-format with version 8 on the full project so that everything is formatted according to this version as suggested by Mike.

[pratikvn]

Maybe valid concerns about default reference captures in lambdas ?

[upsj]

Maybe valid concerns about default reference captures in lambdas ?

As the one who wrote that code, I would say we can safely ignore this. You can basically ready this function as two nested for-each loops that calls the first callback function for each row and the second callback function for each entry of the matrix sum.

Reasons why it could make sense:

1. If you have a lambda that outlives its scope, the capture should be explicit and by-value if we cannot guarantee the captured variables' lifetimes
2. From the functional programming standpoint, it is nice to explicitly list all variables that can possible be modified by the lambda.

[sonarqubecloud]

SonarCloud Quality Gate failed.

0 Bugs
0 Vulnerabilities (and 0 Security Hotspots to review)
6 Code Smells

80.0% Coverage
74.3% Duplication

The version of Java (1.8.0_121) you have used to run this analysis is deprecated and we will stop accepting it from October 2020. Please update to at least Java 11.
Read more here