MAAP-Project
How to write a COG in Python #863
Answered
by
omshinde
wildintellect
asked this question in
Data
Replies: 8 comments 13 replies
-
|
Hi! While reproducing the error, I got the below output mentioning Will keep posting the updates here. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Can we test the script after adding the below line, (this can be line 188) -
So, the script shall look something like - I can't test it as the script execution is getting killed. |
Beta Was this translation helpful? Give feedback.
5 replies
-
|
We certainly can, let me try when I'm on United Airlines wifi over the Atlantic tomorrow, :D |
Beta Was this translation helpful? Give feedback.
-
|
FYI the easiest this to validate in the terminal: pip install rio-cogeo
rio cogeo validate <somefile.tif> |
Beta Was this translation helpful? Give feedback.
-
|
Ok, I've done this
|
Beta Was this translation helpful? Give feedback.
-
|
@nehahunka please provide the full gdalwarp command you are running. |
Beta Was this translation helpful? Give feedback.
-
|
Sure thing, see below
Result is here:
Result is here: The result I want to achieve is correct in the second case, and incorrect in the first case. Since I am sure people will need to deal with resampling the COGs in the future, when the dataset is publicly released, I need to make sure I understand what's happening. THANK YOU!! |
Beta Was this translation helpful? Give feedback.
-
|
I could execute the script for generating COG and non-COG output. The
|
Beta Was this translation helpful? Give feedback.
1 reply
-
|
From the files @wildintellect provided me I don't see any difference at raw resolution import rasterio
with rasterio.open("IPCC_10S_060W_NO-COG.tif") as src:
arr = src.read()
with rasterio.open("IPCC_10S_060W_gdal_converted_cog.tif") as src:
arr_gdal = src.read()
with rasterio.open("IPCC_10S_060W_rio-cogeo_converted.tif") as src:
arr_rio = src.read()
numpy.testing.assert_array_equal(arr, arr_gdal)
numpy.testing.assert_array_equal(arr, arr_rio)There can be visual difference between COG and non-COG can be due to overviews resampling (and maybe different resampling methods / compression) |
Beta Was this translation helpful? Give feedback.
-
|
Hi @nehahunka ..Please find below the code for writing COG in Python. We found out that using For now, I have kept different options for generating COG as comments in the script -
The final code, notebook and generated COG with masks are stored in my shared bucket at PS: To check file sizes in a directory, I run |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Corresponding ticket-#867 |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
To optimize the script, I ran a memory profiler over the code and it outputs the following: I am curious to know if the final memory usage is ~20 Gb, why the script is not running with a 32 Gb DPS image. |
Beta Was this translation helpful? Give feedback.
3 replies
-
|
@omshinde - Perhaps different tiles need different capacity. Some tiles have more forest area, and more computation |
Beta Was this translation helpful? Give feedback.
-
|
That makes sense. Thank you @nehahunka! |
Beta Was this translation helpful? Give feedback.
-
|
@omshinde that profiler is very helpful... ideas:
|
Beta Was this translation helpful? Give feedback.
-
|
@wildintellect You're correct. With the suggested changes, the memory requirement falls down to ~14 Gb. Here is the profiler output: |
Beta Was this translation helpful? Give feedback.
4 replies
-
|
Link to the updated gist: https://gist.github.com/omshinde/47db1486495500087b6a9dcaf3204313#file-ipcc_gedi_table4-7_profiled-py |
Beta Was this translation helpful? Give feedback.
-
|
I think there's still something else that isn't being cleaned up, Is still 2x the memory I expect. Was pondering if writing the MASK to disk, clearing everything and reading it back in was the best way to clear out the RAM. |
Beta Was this translation helpful? Give feedback.
-
Reusing most of the variables after reading it once and deleting some more unused variables produces the above memory profile. @wildintellect |
Beta Was this translation helpful? Give feedback.
Answer selected by
wildintellect
-
|
@nehahunka @wildintellect Also, please use 30Gb workspace for running this script. Link to the gist: https://gist.github.com/omshinde/47db1486495500087b6a9dcaf3204313#file-ipcc_gedi_table4-7_profiled-py |
Beta Was this translation helpful? Give feedback.
-
|
[WIP] On my side, I was trying to compile the and calling it by However, I am not sure if it is going to make super-enhancements. Moreover, it will reduce the descriptiveness of the code. Maybe, it would make sense to parallelize the array operations into smaller sub-arrays instead of file operations. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
@nehahunka would like help modifiying the writing of GeoTIFF at the bottom of this script to write Cloud-Optimized GeoTIFF (COG)
https://repo.maap-project.org/lduncanson/biomass_harmonization/-/blob/master/country_summaries/IPCC_classes_DPS/IPCC_GEDI_Table4.7.py
The Data team is going to help with this, since it's not really a bug (though might end up in the documentation), we're putting a notice on the board so others can find the answer later.
Beta Was this translation helpful? Give feedback.
All reactions