gradient on combination of horizontal and 3d fields does not work #1989
Comments
|
cc: @dennisYatunin |
Need to compute flux before `ᶜdivᵥ` until we resolve: CliMA/ClimaCore.jl#1989
|
@haakon-e, could you please update the reproducer? I think that the existing one has a bug. In particular: level_field = Fields.level(Fields.Field(Float64, center_space), 1)
ᶠscalar_field = Fields.Field(Float64, face_space)
# Does not work:
@. ᶜgradᵥ(level_field + ᶠscalar_field)
level_field = Fields.level(Fields.Field(Float64, face_space), Utilities.half)
ᶠscalar_field = Fields.Field(Float64, face_space)
# Does not work:
@. ᶜgradᵥ(level_field + ᶠscalar_field) |
|
I think the correct reproducer (as I expect) is: import ClimaCore: Domains, Meshes, Geometry, Grids, Spaces, Topologies, Hypsography, Fields, Operators, Utilities
import ClimaComms
ENV["CLIMACOMMS_DEVICE"] = "CUDA"
using CUDA
# ENV["CLIMACOMMS_DEVICE"] = "CPU"
comms_ctx = ClimaComms.SingletonCommsContext()
h_domain = Domains.RectangleDomain(
Domains.IntervalDomain(Geometry.XPoint(0.0), Geometry.XPoint(1.0); periodic = true),
Domains.IntervalDomain(Geometry.YPoint(0.0), Geometry.YPoint(1.0); periodic = true)
)
h_mesh = Meshes.RectilinearMesh(h_domain, 10, 10)
h_grid = Spaces.grid(Spaces.SpectralElementSpace2D(
Topologies.DistributedTopology2D(comms_ctx, h_mesh, Topologies.spacefillingcurve(h_mesh)),
Spaces.Quadratures.GLL{4}(),
))
z_domain = Domains.IntervalDomain(Geometry.ZPoint(0.0), Geometry.ZPoint(1.0); boundary_names = (:bottom, :top))
z_grid = Grids.FiniteDifferenceGrid(Topologies.IntervalTopology(
comms_ctx, Meshes.IntervalMesh(z_domain, Meshes.Uniform(); nelems = 10)
))
grid = Grids.ExtrudedFiniteDifferenceGrid(h_grid, z_grid, Hypsography.Flat(); deep = false)
center_space = Spaces.CenterExtrudedFiniteDifferenceSpace(grid)
face_space = Spaces.FaceExtrudedFiniteDifferenceSpace(grid)
# Create fields and show that it fails
ᶜgradᵥ = Operators.GradientF2C()
level_field = Fields.level(Fields.Field(Float64, face_space), Utilities.half)
ᶠscalar_field = Fields.Field(Float64, face_space)
# Does not work:
@. ᶜgradᵥ(level_field + ᶠscalar_field)
import LazyBroadcast as LB
bc = LB.@lazy @. ᶜgradᵥ(level_field + ᶠscalar_field);Alarmingly, neither of these break on the CPU! |
|
Some progress from earlier today:
julia> @inline Operators.reconstruct_placeholder_space(::Operators.LevelPlaceholderSpace, parent_space) =
Spaces.SpectralElementSpace2D(Spaces.level(Spaces.grid(parent_space), Operators.PlusHalf(0)))
julia> @. ᶜgradᵥ(level_field + ᶠscalar_field)
ClimaCore.Geometry.Covariant3Vector{Float64}-valued Field:
components:
data:
1: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0 … 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
julia> @inline Operators.reconstruct_placeholder_space(::Operators.LevelPlaceholderSpace, parent_space) =
Spaces.level(parent_space, Operators.PlusHalf(0))
julia> @. ᶜgradᵥ(level_field + ᶠscalar_field)
ERROR: InvalidIRError: compiling MethodInstance for ClimaCoreCUDAExt.copyto_stencil_kernel!(::ClimaCore.Fields.Field{…}, ::ClimaCore.Operators.StencilBroadcasted{…}, ::ClimaCore.Spaces.ExtrudedFiniteDifferenceSpace{…}, ::NTuple{…}, ::ClimaCore.DataLayouts.UniversalSize{…}) resulted in invalid LLVM IR
Reason: unsupported call to an unknown function (call to julia.new_gc_frame)
Reason: unsupported call to an unknown function (call to julia.push_gc_frame)
Reason: unsupported call to an unknown function (call to julia.get_gc_frame_slot)
Reason: unsupported dynamic function invocation (call to level)
Stacktrace:
[1] copyto_stencil_kernel!
@ /central/home/dyatunin/ClimaCore.jl/ext/cuda/operators_finite_difference.jl:0
Reason: unsupported call to an unknown function (call to julia.pop_gc_frame)
Hint: catch this exception as `err` and call `code_typed(err; interactive = true)` to introspect the erronous code with Cthulhu.jlWe can conclude that the inference problem on GPUs is occurring during dispatch of level(space::FaceExtrudedFiniteDifferenceSpace3D, v::PlusHalf) =
SpectralElementSpace2D(level(grid(space), v))This might be happening because the |
|
Note for future self (and @dennisYatunin). One thing I noticed is that we don't always call import LazyBroadcast as LB
bc = LB.@lazy @. ᶜgradᵥ(level_field + ᶠscalar_field);
materialize(bc)reproduces the error above (w.r.t. inference failure in import LazyBroadcast as LB
bc = LB.@lazy @. ᶜgradᵥ(level_field + ᶠscalar_field);
bc = Base.Broadcast.instantiate(bc);
materialize(bc)yields a different error (unmatched spaces). It may be that this doesn't fix the issue, but I think that we may want to fix this first. |
|
Actually, that last comment was an issue with LazyBroadcast, but is now fixed in the latest version. |
|
Here is an updated reproducer: import ClimaCore: Spaces, Fields, Operators, Utilities
import ClimaComms
using ClimaCore.CommonSpaces
cspace = ExtrudedCubedSphereSpace(;
z_elem = 10,
z_min = 0,
z_max = 1,
radius = 10,
h_elem = 10,
n_quad_points = 4,
staggering = CellCenter()
)
fspace = Spaces.face_space(cspace);
# Create fields
ᶜgradᵥ = Operators.GradientF2C()
ᶠscalar_field = Fields.Field(Float64, fspace)
ᶜlevel_field = Fields.level(Fields.Field(Float64, cspace), 1)
ᶠlevel_field = Fields.level(Fields.Field(Float64, fspace), Utilities.half)
level_values = Fields.field_values(ᶜlevel_field)
# Should error, but doesn't on the cpu (we cannot add center and face spaces):
@. ᶜlevel_field + ᶠscalar_field
@. ᶜgradᵥ(ᶜlevel_field + ᶠscalar_field)
# Should work, and does, on the cpu
@. ᶠlevel_field + ᶠscalar_field
@. ᶜgradᵥ(ᶠlevel_field + ᶠscalar_field)
# Should work, but doesn't
@. level_values + ᶠscalar_field
@. ᶜgradᵥ(level_values + ᶠscalar_field) |
charleskawczynski
added
Design question
bug
|
Of the 3 cases in the reproducer:
|
|
For i. & i.., if either lead to errors, it would be useful with a short hint for what users should do. E.g. apply a transformation, extract values etc., with reference to a suggested method. |
Describe the bug
The following code produces an error:
produces the following error
To Reproduce
See reproducer:
PS: I'm using the
benchmarks/3denvironment for convenience since it has bothClimaCoreandCUDAavailable.System details
Any relevant system information:
versioninfo()
The text was updated successfully, but these errors were encountered: