Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add GPU support for extruded 1D spaces #2172

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

dennisYatunin
Copy link
Member

@dennisYatunin dennisYatunin commented Feb 3, 2025

This PR adds GPU support for extruded 1D spaces, with the following modifications:

  • Remove the assumption that Ni == Nj from mapreduce_cuda_kernel, columnwise_partition, columnwise_universal_index, and multiple_field_solve_universal_index
  • Add methods for operator_shmem operator_fill_shmem!, and operator_evaluate corresponding to the 1D versions of all SEM operators
  • Rename DSSDataTypes to DSSTypes2D, and define a new constant DSSTypes1D
  • Add a method for dss_1d! corresponding to CUDADevice, and extend dss! to support both 1D and 2D data types
  • Adapt the local_geometry and dss_weights of a SpectralElementSpace1D from Arrays to CuArrays

These changes are also tested through CI:

  • Add a CUDA job that runs unit_2d.jl and convergence_2d.jl
  • Add tests for IFH, IHF, VIFH, and VIHF to unit_mapreduce.jl, which already has a CUDA job

To fix several test failues, I had to make a few additional changes:

  • Simplify the CUDA implementation of Gradient by directly differentiating vectors instead of breaking them up into components, as the original design could not distinguish the gradient of a scalar from the gradient of a single-component vector, and one of the tests in unit_2d.jl takes the gradient of a UVector field
  • Extend strip_space to FiniteDifferenceOperator boundary conditions, as one of the tests in unit_2d.jl uses a 1D field for the CurlC2F boundary condition
  • Reduce the number of methods for level(space::ExtrudedFiniteDifferenceSpace, v) from 4 to 1 to avoid a GPU inference failure in reconstruct_placeholder_space when evaluating boundary conditions (we hit some sort of compiler heuristic when we use 2 or 4 specialized methods, so we need to just have one method with union-splitting)
  • Adapt several CuArrays to Arrays in unit_2d.jl and unit_spaces.jl to avoid scalar indexing errors
  • Bump the number of allowed JET failures for SpectralElementSpace1D in opt_spaces.jl from 141 to 825 on GPUs and from 137 to 150 on CPUs

  • Code follows the style guidelines OR N/A.
  • Unit tests are included OR N/A.
  • Code is exercised in an integration test OR N/A.
  • Documentation has been added/updated OR N/A.

@dennisYatunin dennisYatunin force-pushed the dy/extruded_1d_cuda branch 12 times, most recently from 6e1c215 to ada7571 Compare February 4, 2025 08:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

gradient on combination of horizontal and 3d fields does not work
1 participant