Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement PermutedDimsArray for DiskArrays #42

Open
maxfreu opened this issue Aug 13, 2021 · 3 comments
Open

Implement PermutedDimsArray for DiskArrays #42

maxfreu opened this issue Aug 13, 2021 · 3 comments

Comments

@maxfreu
Copy link

maxfreu commented Aug 13, 2021

Hi, I noticed bad performance when using PermutedDimsArray with ArchGDAL and the issue probably is located here. See yeesian/ArchGDAL.jl#225, which explains the problem. However I don't see an easy fix like proposed over there, as it runs into a stack overflow for AbstractDiskArrays.

@meggart
Copy link
Owner

meggart commented Jan 13, 2022

As explained in the issue you linked, it would be preferable to call permutedims on a DiskArray, which returns a PermuteDiskArray which has the desired behavior.

In case you insist on working with PermutedDimsArray from Base directly, you could frist define const AltPermuteDiskArray = PermuteDimsArray{<:Any, <:Any, <:Any, <:Any, <:AbstractDiskArray} and then you define readblock!, writeblock! and eachchunk as in permute_reshape.jl and implement the remaining DiskArray interface through implement_diskarray AltPermuteDiskArray.

The main drawback of this approach (using Base types for permutations, views, etc instead of making DiskArray analogies) is that the resulting arrays still are not AbstractDiskArrays, so dispatch will not work on them when you recursively use them as wrappers, for example when you create a view into the Permuted DiskArray.

@rafaqz
Copy link
Collaborator

rafaqz commented Jan 24, 2023

I have a use-case for this too. We need a dimension permutation method that is always lazy - permutedims is only lazy for disk arrays now, but PermuteDimsArray would be lazy for both.

This causes problems generalising some methods in Rasters.jl because we really want lazy permutation of large in-memory rasters, but cant without special-casing everything.

@rafaqz
Copy link
Collaborator

rafaqz commented Feb 19, 2024

See JuliaLang/julia#53188

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants