Method: High level explanation and code snippets.

High Level Explanations: ASGARD SC'19 poster.

A poster presents the Adaptive Sparse Grid Discretization (ASGarD) for High Dimensional Advection Diffusion Problems on Exascale Architectures. It was presented during SC'19, here is a link: Sc'19 Poster

• Authors: M. Graham Lopez (Oak Ridge National Laboratory), David L. Green (Oak Ridge National Laboratory), Lin Mu (University of Georgia), Ed D'Azevedo (Oak Ridge National Laboratory), Wael Elwasif (Oak Ridge National Laboratory), Tyler McDaniel (University of Tennessee), Timothy Younkin (University of Tennessee), Adam McDaniel (Oak Ridge National Laboratory), Diego Del-Castillo-Negrete (Oak Ridge National Laboratory)

• Abstract: Many scientific domains require the solution of high dimensional PDEs. Traditional grid- or mesh-based methods for solving such systems in a noise-free manner quickly become intractable due to the scaling of the degrees of freedom going as O(N^d) sometimes called "the curse of dimensionality." We are developing an arbitrarily high-order discontinuous-Galerkin finite-element solver that leverages an adaptive sparse-grid discretization whose degrees of freedom scale as O(N*log2 N^D-1). This method and its subsequent reduction in the required resources is being applied to several PDEs including time-domain Maxwell's equations (3D), the Vlasov equation (in up to 6D) and a Fokker-Planck-like problem in ongoing related efforts. Here we present our implementation which is designed to run on multiple accelerated architectures, including distributed systems. Our implementation takes advantage of a system matrix decomposed as the Kronecker product of many smaller matrices which is implemented as batched operations.

Code snippets

No adaptivity:

1. Get the subgrid (if multi MPI processes) then get the unscaled parts of the boundary condition. Why are there scaled and unscaled parts?

    auto const my_subgrid     = adaptive_grid.get_subgrid(get_rank());
    auto const unscaled_parts = boundary_conditions::make_unscaled_bc_parts(
        pde, adaptive_grid.get_table(), transformer, my_subgrid.row_start,
        my_subgrid.row_stop);

2. Call the time_advance function.

    return (step_method == method::exp)
               ? explicit_advance(pde, adaptive_grid, transformer, program_opts,
                                  unscaled_parts, x_orig, workspace_size_MB,
                                  time)
               : implicit_advance(pde, adaptive_grid, transformer,
                                  unscaled_parts, x_orig, time,
                                  program_opts.solver, update_system);

With adaptivity:

1. Coarsen the grid if necessary

  auto const old_size = adaptive_grid.size();
  auto y = adaptive_grid.coarsen_solution(pde, x_orig, program_opts);

2. As long as needed: refine the grid

  auto refining = true;
  while (refining)
  {
    // update boundary conditions
    [...]
    // take a probing refinement step
    auto const y_stepped =
        (step_method == method::exp)
            ? explicit_advance(pde, adaptive_grid, transformer, program_opts,
                               unscaled_parts, y, workspace_size_MB, time)
            : implicit_advance(pde, adaptive_grid, transformer, unscaled_parts,
                               y, time, program_opts.solver, update_system);
    // Refine the grid
    [...]
  }