Refactor linear algebra functionality for complex-valued problems #47

sebastiangrimberg · 2023-06-01T22:51:21Z

Previously, Palace leveraged PETSc configured with complex-valued scalars to provide support for solving complex-valued linear systems, using wrapper classes like PetscParMatrix and PetscParVector. With this PR, complex-valued linear algebra functionality is built directly on the underlying MFEM data structures, removing the need for PETSc as a direct dependency (only for the SLEPc eigenvalue solver). For small (sequential) complex-valued linear algebra like for PROM construction, we use Eigen.

This will make features like operator partial-assembly much easier to implement, as well as simplifying eventual GPU support by avoiding many interfaces between PETSc and MFEM objects.

This PR is a prerequisite for #64. Since #64 contains some follow-up changes to changes also made in this PR, I propose we review both simultaneously and only merge #47 quickly followed by #64 once comments with both have been addressed and they are both approved.

TODO

Documentation updates for configuration file keywords
Update config file schema
Changelog

Closes #42

…functionality and wave ports

Reverts prior commit as this idea did not work, one boundary attribute may touch many domain attributes.

… S-parameter calculations

…unicator is different than the solvers (for example, many processors with empty local matrices)

… smoothers

hughcars

I have definitely missed some things as this was a pretty big read, but hopefully I've caught a good chunk.
For derived virtual classes, consider use of final for the class or for overrides if there is no intention of a child in future. Same for choosing between protected and private for member variables and methods.
I have a branch hughcars/driven-port-debug-comments where I tried out a few of my suggested changes to check they worked, feel free to take as appropriate. Particularly the ComplexVector value semantics I thiiink are complete on there.

palace/drivers/drivensolver.cpp

palace/linalg/vector.hpp

palace/linalg/vector.cpp

hughcars · 2023-08-10T21:17:51Z

palace/linalg/vector.hpp

+  ComplexVector(int n = 0);
+
+  // Copy constructor.
+  ComplexVector(const ComplexVector &y);


tl;dr: this class needs to match the general c++ value semantics, so I proposed a few changes. For details, see the cpp and hughcars/driven-port-debug-comments for an implementation.

As discussed offline, I've added some subset of the functionality suggested in hughcars/driven-port-debug-comments. For now I've skipped the move constructor since we haven't had a need for it.

palace/models/waveportoperator.cpp

palace/utils/iodata.cpp

palace/models/spaceoperator.cpp

scripts/schema/config/solver.json

hughcars

I have definitely missed some things as this was a pretty big read, but hopefully I've caught a good chunk.
For derived virtual classes, consider use of final for the class or for overrides if there is no intention of a child in future. Same for choosing between protected and private for member variables and methods.
I have a branch hughcars/driven-port-debug-comments where I tried out a few of my suggested changes to check they worked, feel free to take as appropriate. Particularly the ComplexVector value semantics I thiiink are complete on there.

…nSolver -> Arpack/SlepcEigenvalueSolver, using instead of typedef, fix some typos in comments

…with a mutable RAP member to avoid some const_cast, clarify some comments, use initializer lists where possible for constructors

Initial testing shows the expected 2x improvement in number of iterations, where each iteration now doubles in cost at the fine levels due to complex-valued MVP instead of real-valued. However, for expensive coarse solves, this is a substantial improvement and speedups look to be 25% on cavity example. Also adds back Chebyshev smoothing based on the standard 1st-kind polynomials, with runtime options for configuring which variant and the associated tolerances."

hughcars

Still disagree on Arpack and the buffers, but not a hill to die on.

palace/drivers/eigensolver.cpp

palace/linalg/ams.cpp

palace/linalg/arpack.cpp

Initial testing shows the expected 2x improvement in number of iterations, where each iteration now doubles in cost at the fine levels due to complex-valued MVP instead of real-valued. However, for expensive coarse solves, this is a substantial improvement and speedups look to be 25% on cavity example. Also adds back Chebyshev smoothing based on the standard 1st-kind polynomials, with runtime options for configuring which variant and the associated tolerances."

sebastiangrimberg force-pushed the sjg/driven-port-debug branch from 90add8f to d29335f Compare June 1, 2023 22:57

sebastiangrimberg mentioned this pull request Jun 1, 2023

Use SubMesh for wave port boundary mode calculations #42

Closed

sebastiangrimberg force-pushed the sjg/driven-port-debug branch 3 times, most recently from 97a68f8 to ff10b89 Compare June 5, 2023 15:23

sebastiangrimberg changed the base branch from sjg/source-reorg-dev to sjg/arpack-ci-tests June 5, 2023 15:24

sebastiangrimberg changed the title ~~Draft: Partial-assembly development branch, part 3~~ Draft: Partial-assembly development branch, part 5 Jun 5, 2023

sebastiangrimberg force-pushed the sjg/arpack-ci-tests branch from ab1b35d to 9f5e774 Compare June 5, 2023 15:42

Base automatically changed from sjg/arpack-ci-tests to sjg/mfem-patches-pa-support June 5, 2023 15:44

sebastiangrimberg force-pushed the sjg/mfem-patches-pa-support branch 3 times, most recently from dfb66ca to ea571b5 Compare June 5, 2023 16:32

sebastiangrimberg force-pushed the sjg/driven-port-debug branch from ff10b89 to 64fa330 Compare June 5, 2023 17:36

sebastiangrimberg force-pushed the sjg/mfem-patches-pa-support branch from ea571b5 to fd34f79 Compare June 12, 2023 23:15

sebastiangrimberg force-pushed the sjg/driven-port-debug branch from 64fa330 to 586a147 Compare June 12, 2023 23:56

sebastiangrimberg changed the title ~~Draft: Partial-assembly development branch, part 5~~ Draft: Partial-assembly development branch, part 3 Jun 13, 2023

sebastiangrimberg force-pushed the sjg/driven-port-debug branch 2 times, most recently from bb79f42 to 9bbfe39 Compare June 14, 2023 15:41

sebastiangrimberg changed the title ~~Draft: Partial-assembly development branch, part 3~~ Draft: Partial-assembly development branch, part 2 Jun 14, 2023

sebastiangrimberg force-pushed the sjg/mfem-patches-pa-support branch from 46d0c53 to 77a2be9 Compare June 27, 2023 18:26

sebastiangrimberg force-pushed the sjg/driven-port-debug branch from 9bbfe39 to 4343d4d Compare June 27, 2023 18:27

sebastiangrimberg force-pushed the sjg/mfem-patches-pa-support branch 3 times, most recently from cba2779 to af4a5e5 Compare June 27, 2023 19:03

sebastiangrimberg force-pushed the sjg/driven-port-debug branch 2 times, most recently from 10c87a4 to 6317e8e Compare June 27, 2023 19:13

sebastiangrimberg force-pushed the sjg/mfem-patches-pa-support branch from af4a5e5 to 5573f61 Compare June 27, 2023 19:19

sebastiangrimberg force-pushed the sjg/driven-port-debug branch from 6317e8e to 25d276c Compare June 27, 2023 19:20

Base automatically changed from sjg/mfem-patches-pa-support to main June 30, 2023 15:23

sebastiangrimberg force-pushed the sjg/driven-port-debug branch from 25d276c to cc43db8 Compare July 14, 2023 22:04

sebastiangrimberg added 14 commits August 8, 2023 11:15

Material properties for boundary elements in preparation for submesh …

df9a8e5

…functionality and wave ports

Implement wave ports using submesh functionality from MFEM

b3c7920

Reverts prior commit as this idea did not work, one boundary attribute may touch many domain attributes.

Optimize global communications for surface integrals and postprocessing

7025f91

Debugging wave ports

2beb351

Fix when built without PETSc

e57ff3d

Resolve sign ambiguity for wave port mode to get consistent phases in…

bf0e4dc

… S-parameter calculations

Fix sparse direct solver matrix construction when HypreParMatrix comm…

76ff265

…unicator is different than the solvers (for example, many processors with empty local matrices)

Default iterations fix for power iteration without SLEPc

c70531e

SLEPc is no longer required

0530bb1

Re-enable macOS CI job

62c55eb

Bugfix for wave ports and some other minor fixes after rebase

808c479

Minor performance improvements for operator application and multigrid…

99c2896

… smoothers

Update docs, JSON Schema, changelog

e7eb17a

Minor style fix

393b428

sebastiangrimberg force-pushed the sjg/driven-port-debug branch from 9a51e9f to 393b428 Compare August 8, 2023 18:16

sebastiangrimberg mentioned this pull request Aug 11, 2023

Introduce arbitrary orientation for lumped ports #75

Merged

hughcars reviewed Aug 14, 2023

View reviewed changes

sebastiangrimberg added 2 commits August 14, 2023 17:06

Address PR comments: Use of auto when type is clear, Arpack/SlepcEige…

ffc80f4

…nSolver -> Arpack/SlepcEigenvalueSolver, using instead of typedef, fix some typos in comments

Further PR comment addressing: Make Operator::ParallelAssemble const …

37ed18a

…with a mutable RAP member to avoid some const_cast, clarify some comments, use initializer lists where possible for constructors

hughcars approved these changes Aug 15, 2023

View reviewed changes

hughcars approved these changes Aug 18, 2023

View reviewed changes

palace/drivers/eigensolver.cpp Show resolved Hide resolved

palace/linalg/ams.cpp Show resolved Hide resolved

palace/linalg/arpack.cpp Show resolved Hide resolved

palace/linalg/arpack.cpp Show resolved Hide resolved

palace/linalg/arpack.cpp Show resolved Hide resolved

Address PR feedback: Remove extraneous braces

3c031e8

sebastiangrimberg enabled auto-merge August 22, 2023 16:20

sebastiangrimberg merged commit cdc8e20 into main Aug 22, 2023
17 checks passed

sebastiangrimberg deleted the sjg/driven-port-debug branch August 22, 2023 17:10

sebastiangrimberg mentioned this pull request Aug 22, 2023

Complex-valued operators for multigrid preconditioning #87

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor linear algebra functionality for complex-valued problems #47

Refactor linear algebra functionality for complex-valued problems #47

sebastiangrimberg commented Jun 1, 2023 •

edited

Loading

hughcars left a comment

hughcars Aug 10, 2023

sebastiangrimberg Aug 15, 2023

hughcars left a comment

hughcars left a comment

Refactor linear algebra functionality for complex-valued problems #47

Refactor linear algebra functionality for complex-valued problems #47

Conversation

sebastiangrimberg commented Jun 1, 2023 • edited Loading

hughcars left a comment

Choose a reason for hiding this comment

hughcars Aug 10, 2023

Choose a reason for hiding this comment

sebastiangrimberg Aug 15, 2023

Choose a reason for hiding this comment

hughcars left a comment

Choose a reason for hiding this comment

hughcars left a comment

Choose a reason for hiding this comment

sebastiangrimberg commented Jun 1, 2023 •

edited

Loading