Diocotron error with Gingko

[PaulineVidal]

An error with Gingko appears while running the diocotron simulation:

terminate called after throwing an instance of 'std::runtime_error'
  what():  Ginkgo did not converge in MatrixBatchCsr
Aborted (core dumped)

It does not appears for

• Circular mapping with a grid of 256x512.
• Czarny mapping with a grid of 128x256.

It does appears for

• Czarny mapping with a grid of 256x256.
• Czarny mapping with a grid of 256x512.
• Czarny mapping with (1e-5, 1) parameters make it look like a circular mapping with a grid of 256x512.

The simulation can be a bit non-smooth. Here is the density at t = 50 for ( Czarny mapping with a grid of 128x256)

[tpadioleau]

How many iterations does the linear solver do ?

[PaulineVidal]

I don't know very well Gingko. I guess it is a parameter in Gingko, I just see m_gko_matrix->solve(m_x_init, b); in the code.
@AbdelhadiKara , maybe you know better the answer.

[AbdelhadiKara]

You have the possibility to check the iterations number by activating the logger. there is a boolean you can set to true when instantiating the m_gko_matrix. you can have a look here.

PS: You can try to solve this issue by increasing the maximum number of iterations.

[PaulineVidal]

Yes, it seems to not fail when we set m_max_iter(max_iter.value_or(10000)) instead of m_max_iter(max_iter.value_or(1000)).
Ok, thank you 👍

[tpadioleau]

Doesn't it mean the condition number is high ? Could we find a better preconditioner ?

[AbdelhadiKara]

@tpadioleau the preconditinner used is Jacobi. It was the only one availiable. IMO it would be better to rely on kokkos kernels solvers. They released batched version of CG.

[tpadioleau]

@tpadioleau the preconditinner used is Jacobi. It was the only one availiable. IMO it would be better to rely on kokkos kernels solvers. They released batched version of CG.

How do you think it would help to use Kokkos Kernels ?

Regarding Ginkgo, do you know if the version 1.9 has other preconditioners ?

[AbdelhadiKara]

For several reasons.

• to reduce dépendancies.
• a totally personal point of view, but kokkos-kernels implementation seem more stable.(Robust)
• if in the future We need something, it will be easier to Ask cexa team.

[tpadioleau]

For several reasons.

• to reduce dépendancies.
• a totally personal point of view, but kokkos-kernels implementation seem more stable.(Robust)
• if in the future We need something, it will be easier to Ask cexa team.

On the overall I think I can agree but I want to point out that it will not avoid the work of finding adapted preconditioners and stopping criteria.

[AbdelhadiKara]

Thank you for these precisions
I agree with you, I've just checked and none of the release 1.9 and develop branches contain any preconditioners other than Jacobi for batch solvers.
For stopping criterion, i had investigated several months ago and ginkgo doc makes distinction beetween the "true" stopping criteria and the "implicit" one which is internal to solvers.

[tpadioleau]

Ok thanks!

[PaulineVidal]

For indications, here are the numbers of iterations needed for the simulation I mentionned:
Diocotron simulation on a Czarny mapping on a 256x512 grid.

csr_log.txt

(for the 700th time step, around 1900 iterations in the Poisson solver were needed.)

The modifications I made are on the branch pvidal_diocotron_on_czarny_map
(https://github.com/gyselax/gyselalibxx/blob/pvidal_diocotron_on_czarny_map/simulations/geometryRTheta/diocotron/diocotron.cpp)

[tpadioleau]

 System no. 0:
 Number of iterations = 1909
 Implicit residual norm = 2.07155e-18
 True (Ax-b) residual norm = 3.4037e-14
 Right-hand side (b) norm = 0.00214497
 --- System 0 did not converge! ---

Looking at the residual, to me it looks converged. It is possible that the last 1000 iterations are not increasing much the residual. It could be that the tolerance requested is too strict. A colleague once told me that a rule of thumb to select the value of the tolerance is roughly the machine precision divided by the condition number of the matrix.

[AbdelhadiKara]

I have found an interesting feature in ginkgo CMakeLists,
option(GINKGO_JACOBI_FULL_OPTIMIZATIONS "Use all the optimizations for the CUDA Jacobi algorithm" OFF)
Perhaps switching this to ON will bring some improvements?

[tpadioleau]

@AbdelhadiKara What kind of optimization can we expect from this option ?

[AbdelhadiKara]

I have found at least two things if this option is activated:

• leads to increased jacobi blocksize
• unrolling of some loops