Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

enable Half in mpi #1759

Open
wants to merge 5 commits into
base: develop
Choose a base branch
from
Open

enable Half in mpi #1759

wants to merge 5 commits into from

Conversation

yhmtsai
Copy link
Member

@yhmtsai yhmtsai commented Dec 30, 2024 •
edited
Loading

This PR enables half precision in distributed environment by adding custom operation.

one-side operation like accumulation and fetch_and_op does not support custom operation.

Note. Newer version of mpi might support half precision natively (also for one-side operation) if the administrator build it with compiler supporting native half precision and enable the option.

TODO:

  • enable the rest distributed function with half
  • put the custom operation in gko::comm?

@yhmtsai yhmtsai added the 1:ST:WIP This PR is a work in progress. Not ready for review. label Dec 30, 2024
@yhmtsai yhmtsai self-assigned this Dec 30, 2024
@ginkgo-bot ginkgo-bot added reg:testing This is related to testing. type:solver This is related to the solvers type:preconditioner This is related to the preconditioners mod:all This touches all Ginkgo modules. labels Dec 30, 2024
@yhmtsai yhmtsai added 1:ST:ready-for-review This PR is ready for review and removed 1:ST:WIP This PR is a work in progress. Not ready for review. labels Jan 2, 2025
@yhmtsai yhmtsai requested a review from a team January 2, 2025 08:30
@ginkgo-bot
Copy link
Member

Error: The following files need to be formatted:

common/cuda_hip/distributed/assembly_kernels.cpp
common/cuda_hip/distributed/matrix_kernels.cpp
common/cuda_hip/distributed/vector_kernels.cpp
common/unified/distributed/assembly_kernels.cpp
core/device_hooks/common_kernels.inc.cpp
core/distributed/assembly.cpp
core/distributed/helpers.hpp
core/distributed/matrix.cpp
core/distributed/preconditioner/schwarz.cpp
core/distributed/vector.cpp
core/test/mpi/base/bindings.cpp
core/test/utils.hpp
dpcpp/distributed/assembly_kernels.dp.cpp
dpcpp/distributed/matrix_kernels.dp.cpp
dpcpp/distributed/vector_kernels.dp.cpp
include/ginkgo/core/base/mpi.hpp
include/ginkgo/core/distributed/matrix.hpp
include/ginkgo/core/distributed/vector.hpp
omp/distributed/assembly_kernels.cpp
omp/distributed/matrix_kernels.cpp
omp/distributed/vector_kernels.cpp
reference/distributed/assembly_kernels.cpp
reference/distributed/matrix_kernels.cpp
reference/distributed/vector_kernels.cpp
test/mpi/solver/solver.cpp
test/mpi/vector.cpp

You can find a formatting patch under Artifacts here or run format! if you have write access to Ginkgo

@MarcelKoch MarcelKoch self-requested a review January 7, 2025 08:03
MarcelKoch
MarcelKoch requested changes Jan 9, 2025
View reviewed changes
Copy link
Member

@MarcelKoch MarcelKoch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm mainly concerned about using device buffers for the custom operations, and maybe moving the operations into a private header.

include/ginkgo/core/base/mpi.hpp


template <typename ValueType>
inline void sum(void* input, void* output, int* len, MPI_Datatype* datatype)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do these functions work with device buffers? Or is it maybe necessary to copy the device buffers first to the host and then do the operation?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have the same concern.
However, I think mpi will not use device memory to handle the reduction operation. Even in frontier, they have additional unit to handle not use gpu to sum cross nodes.

include/ginkgo/core/distributed/vector.hpp
Comment on lines +665 to +666
mpi::op_manager sum_op_;
mpi::op_manager norm_sum_op_;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think the vector needs to store these. We can either just create them when needed, and delete them directly afterward. There should not be a large overhead associated to that. Or we use globals (probably in the form of returning a static variable). I would not mind using that, since we only need to have one operation per value type anyway.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have tried the static variable already but it did not work out.
We call MPI_finalize to finish, but the static variable will be destroyed when the process leave main scope, which is after MPI_finalize and MPI complains that.

include/ginkgo/core/base/mpi.hpp
GKO_REGISTER_MPI_TYPE(std::complex<float>, MPI_C_FLOAT_COMPLEX);
GKO_REGISTER_MPI_TYPE(std::complex<double>, MPI_C_DOUBLE_COMPLEX);


namespace detail {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the operation isn't stored in the vector anymore, we can move all of this into a private header, which I would prefer quite a lot.

include/ginkgo/core/distributed/vector.hpp

static std::unique_ptr<target_type> create_empty(const source_type* source)
{
return target_type::create(source->get_executor(),
source->get_communicator());
}

// Allow to create_empty of the same type
// For distributed case, next<next<V>> will be V in the candicated list.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo:

Suggested change
// For distributed case, next<next<V>> will be V in the candicated list.
// For distributed case, next<next<V>> will be V in the candidate list.

include/ginkgo/core/distributed/vector.hpp

static std::unique_ptr<target_type> create_empty(const source_type* source)
{
return target_type::create(source->get_executor(),
source->get_communicator());
}

// Allow to create_empty of the same type
// For distributed case, next<next<V>> will be V in the candicated list.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mean if half is disabled, right? But shouldn't that also lead to issues with Dense?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@MarcelKoch MarcelKoch MarcelKoch requested changes

Requested changes must be addressed to merge this pull request.

Assignees

@yhmtsai yhmtsai

Labels
1:ST:ready-for-review This PR is ready for review mod:all This touches all Ginkgo modules. reg:testing This is related to testing. type:preconditioner This is related to the preconditioners type:solver This is related to the solvers
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@yhmtsai @ginkgo-bot @MarcelKoch