w2w not successful in incorporating LCZ into geo files

[GLlorin]

Describe your issue

Hello!

I am trying to use the global LCZ map produced by Demuzere et al. (2022), cut to the Philippines (~5-25N, 115-130E), in simulations over Metro Manila to test the performance of different urban canopy models. However, I am running into trouble at the first step of using the w2w python utility to incorporate the TIFF file into the prepared geo_em.d0X.nc files.

Following its recommended use, I run this line in the directory where I have my LCZ file (PAR.tif) and my two domain geo files:

> w2w ./ PAR.tif geo_em.d02.nc

It runs well until the check and cleaning starts, giving the following messages and ending in an error:

--> Start sanity check and clean-up ...
Check 1: Urban class removed from geo_em.d02_NoUrban.nc? OK
Check 2: LCZ Urban extent present in geo_em.d02_LCZ_extent.nc?
WARNING: LCZ-based urban extent missing
Check 3: Urban LCZ classes exists in geo_em.d02_LCZ_params.nc? OK: LCZ Classes ([]) present
Check 4: FRC_URB2D present in geo_em.d02_LCZ_params.nc? OK: FRC_URB2D values range between 0.00 and 0.00
Check 5: URB_PARAMS matrix present in file geo_em.d02_LCZ_params.nc? OK
Check 6: Do URB_PARAM variable values follow expected range in geo_em.d02_LCZ_params.nc?

• OK for LP_URB2D
• OK for MH_URB2D
• OK for STDH_URB2D
• OK for HGT_URB2D
• OK for LB_URB2D
• OK for LF_URB2D
  Traceback (most recent call last):
  File "/home/gagena/.conda/envs/w2w/bin/w2w", line 8, in
  sys.exit(main())
  ^^^^^^
  File "/home/gagena/.conda/envs/w2w/lib/python3.11/site-packages/w2w/w2w.py", line 207, in main
  checks_and_cleaning(
  File "/home/gagena/.conda/envs/w2w/lib/python3.11/site-packages/w2w/w2w.py", line 1653, in checks_and_cleaning
  if np.nanmax(np.abs((100 - hi_sum).values)) > 0.1:
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gagena/.conda/envs/w2w/lib/python3.11/site-packages/numpy/lib/nanfunctions.py", line 476, in nanmax
  res = np.fmax.reduce(a, axis=axis, out=out, **kwargs)
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  ValueError: zero-size array to reduction operation fmax which has no identity

I know this error was already brought up in Issue #38, but when I tried to follow the recommended course of action from the thread by executing:

pip install --upgrade --force-reinstall git+https://github.com/matthiasdemuzere/w2w.git@add_tests

I run into this error:

Collecting git+https://github.com/matthiasdemuzere/w2w.git@add_tests
Cloning https://github.com/matthiasdemuzere/w2w.git (to revision add_tests) to /tmp/pip-req-build-tv1ghnl6
Running command git clone --filter=blob:none --quiet https://github.com/matthiasdemuzere/w2w.git /tmp/pip-req-build-tv1ghnl6
WARNING: Did not find branch or tag 'add_tests', assuming revision or ref.
Running command git checkout -q add_tests
error: pathspec 'add_tests' did not match any file(s) known to git
error: subprocess-exited-with-error

× git checkout -q add_tests did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× git checkout -q add_tests did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

I have uploaded my geo_em.d0X.nc files and the TIFF file I have cropped from the global map by Demuzere et al. (2022) in this GDrive. I've also included the produced geo_em.d02_LCZ_params.nc file which confirms that the LU_INDEX was not updated. Please advise me on my next steps.

Thank you very much!

w2w --version

w2w 0.5.0

nc-config --all

This netCDF 4.8.1 has been built with the following features:

  --cc            -> /home/spack/opt/spack/linux-rocky9-broadwell/gcc-11.4.1/intel-oneapi-compilers-2023.2.1-mzcqsqcfemyks4cypbbrruzwz5kd5vy4/compiler/2023.2.1/linux/bin/intel64/icc
  --cflags        -> -I/home/spack/opt/spack/linux-rocky9-broadwell/intel-2021.10.0/netcdf-c-4.8.1-amx4rlwhjr54v2uhfxk3i7xvvy4prqkq/include
  --libs          -> -L/home/spack/opt/spack/linux-rocky9-broadwell/intel-2021.10.0/netcdf-c-4.8.1-amx4rlwhjr54v2uhfxk3i7xvvy4prqkq/lib -lnetcdf
  --static        -> -lhdf5_hl -lhdf5 -lm

  --has-c++       -> no
  --cxx           ->

  --has-c++4      -> no
  --cxx4          ->

  --has-fortran   -> yes
  --fc            -> /home/spack/opt/spack/linux-rocky9-broadwell/gcc-11.4.1/intel-oneapi-compilers-2023.2.1-mzcqsqcfemyks4cypbbrruzwz5kd5vy4/compiler/2023.2.1/linux/bin/intel64/ifort
  --fflags        -> -I/home/spack/opt/spack/linux-rocky9-broadwell/intel-2021.10.0/netcdf-fortran-4.5.4-akdrb6wpoo4lgieevwzr7dgvp7j5gbs2/include -I/home/spack/opt/spack/linux-rocky9-broadwell/intel-2021.10.0/netcdf-fortran-4.5.4-akdrb6wpoo4lgieevwzr7dgvp7j5gbs2/include
  --flibs         -> -L/home/spack/opt/spack/linux-rocky9-broadwell/intel-2021.10.0/netcdf-fortran-4.5.4-akdrb6wpoo4lgieevwzr7dgvp7j5gbs2/lib -lnetcdff -lnetcdf -lnetcdf -lm
  --has-f90       ->
  --has-f03       -> yes

  --has-dap       -> no
  --has-dap2      -> no
  --has-dap4      -> no
  --has-nc2       -> yes
  --has-nc4       -> yes
  --has-hdf5      -> yes
  --has-hdf4      -> no
  --has-logging   -> no
  --has-pnetcdf   -> no
  --has-szlib     -> no
  --has-cdf5      -> yes
  --has-parallel4 -> no
  --has-parallel  -> no
  --has-nczarr    -> yes

  --prefix        -> /home/spack/opt/spack/linux-rocky9-broadwell/intel-2021.10.0/netcdf-c-4.8.1-amx4rlwhjr54v2uhfxk3i7xvvy4prqkq
  --includedir    -> /home/spack/opt/spack/linux-rocky9-broadwell/intel-2021.10.0/netcdf-c-4.8.1-amx4rlwhjr54v2uhfxk3i7xvvy4prqkq/include
  --libdir        -> /home/spack/opt/spack/linux-rocky9-broadwell/intel-2021.10.0/netcdf-c-4.8.1-amx4rlwhjr54v2uhfxk3i7xvvy4prqkq/lib
  --version       -> netCDF 4.8.1

Installed Packages

affine==2.4.0
attrs==24.2.0
certifi==2024.8.30
cftime==1.6.4
click==8.1.7
click-plugins==1.1.1
cligj==0.7.2
ecmwflibs==0.6.3
findlibs==0.0.5
h5netcdf==1.3.0
h5py==3.11.0
netCDF4==1.7.1.post2
numpy @ file:///home/conda/feedstock_root/build_artifacts/numpy_1707225376651/work/dist/numpy-1.26.4-cp311-cp311-linux_x86_64.whl#sha256=d08e1c9e5833ae7780563812aa73e2497db1ee3bd5510d3becb8aa18aa2d0c7c
packaging @ file:///home/conda/feedstock_root/build_artifacts/packaging_1718189413536/work
pandas==1.5.3
pyparsing==3.1.4
pyproj==3.6.1
python-dateutil @ file:///home/conda/feedstock_root/build_artifacts/python-dateutil_1709299778482/work
pytz @ file:///home/conda/feedstock_root/build_artifacts/pytz_1706886791323/work
rasterio==1.3.11
rioxarray==0.17.0
scipy==1.14.1
six @ file:///home/conda/feedstock_root/build_artifacts/six_1620240208055/work
snuggs==1.4.7
tqdm==4.66.5
w2w==0.5.0
xarray @ file:///home/conda/feedstock_root/build_artifacts/xarray_1674166302925/work

Traceback

No response

[jkittner]

I am not sure why you're encountering the error, but your re-installation command does not work because the branch does not exist anymore.

Try this instead for the latest version on main

pip install  --upgrade --force-reinstall git+https://github.com/matthiasdemuzere/w2w.git

or a version where we've made a few changes (which is not merged)

pip install  --upgrade --force-reinstall git+https://github.com/matthiasdemuzere/w2w.git@add_wrf_version

If you're error still persists, you will probably have to provide your geo_em files, for someone to have a look.

[GLlorin]

Thank you so much for replying so quickly. It took me some time to apply both suggestions but essentially, while both suggested re-installation commands worked, I still encounter the same error message I posted about when the checks and cleaning are being made. For some reason, the LCZ data is not being incorporated into the new geo_em.d0X.nc files.

It may not have been seen but I do provide all files I have in this drive as the TIFF file was quite large: https://drive.google.com/drive/u/1/folders/1J6sm3QQZaSg0JaYaPG4mezDLTx_k7eA5

Please advise me on further courses of action I can take, thank you!

[matthiasdemuzere]

Hi @GLlorin,

Thanks for reporting this.

The first thing I noticed is the large size of your PAR.tif file, that seems to cover an area that is many times larger than your target WRF d02 domain. When using that file, my local machine actually did not have anough memory :)

So, I just cropped a smaller area out of the global LCZ map, using the code below (you can get v3 of global LCZ map here):

import rioxarray as rxr
import xarray as xr

fn_lcz = "/mnt/SSD2TB/data/ZENODO/LCZ_GLOBAL/v3/lcz_filter_v3.tif"
fn_geo = "/mnt/SSD2TB/data/w2w-debug/132_Manilla/geo_em.d02.nc"

lcz = rxr.open_rasterio(fn_lcz)
ds = xr.open_dataset(fn_geo)

dxy = 0.5
xmin= float(ds['XLONG_M'].min()) - dxy
xmax = float(ds['XLONG_M'].max()) + dxy
ymin = float(ds['XLAT_M'].min()) - dxy
ymax = float(ds['XLAT_M'].max()) + dxy
print(xmin, xmax, ymin, ymax)

# Crop LCZ map
lcz_sel = lcz.sel(x=slice(xmin, xmax), y=slice(ymax, ymin))
print(lcz_sel)

# Store to drive
fn_lcz_sel = "/mnt/SSD2TB/data/w2w-debug/132_Manilla/lcz.tif"
lcz_sel.rio.to_raster(fn_lcz_sel)

When I execute w2w with this smaller file, I can actually reproduce your error:

w2w . lcz.tif geo_em.d02.nc 'v4.3'

--> Set data, arguments and files 
> Using layer 0 of the LCZ GeoTIFF. Can be changed with -l (--lcz-band)
--> Check LCZ integrity, in terms of class labels, projection and extent
> LCZ labels as expected (1 to 17)
> LCZ provided as WGS84 (EPSG:4326)
> LCZ domain is covering WRF domain
--> Replace WRF urban LC with surrounding natural LC
Looping through urban grid pixels: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 182/182 [00:00<00:00, 194.71it/s]
--> Create LCZ-based geo_em file
> Processing LP_URB2D ...
> Processing MH_URB2D ...
> Processing STDH_URB2D ...
> Processing HGT_URB2D ...
> Processing LB_URB2D ...
> Processing LF_URB2D ...
> Processing HI_URB2D ...
--> Create LCZ-based urban extent geo_em file (excluding other LCZ-based info)
--> Expanding land categories of parent domain(s) to 41

--> Start sanity check and clean-up ...
> Check 1: Urban class removed from geo_em.d02_NoUrban.nc? OK 
> Check 2: LCZ Urban extent present in geo_em.d02_LCZ_extent.nc?
 WARNING: LCZ-based urban extent missing 
> Check 3: Urban LCZ classes exists in geo_em.d02_LCZ_params.nc? OK: LCZ Classes ([]) present 
> Check 4: FRC_URB2D present in geo_em.d02_LCZ_params.nc? OK: FRC_URB2D values range between 0.00 and 0.00 
> Check 5: URB_PARAMS matrix present in file geo_em.d02_LCZ_params.nc? OK 
> Check 6: Do URB_PARAM variable values follow expected range in geo_em.d02_LCZ_params.nc?
   + OK for LP_URB2D 
   + OK for MH_URB2D 
   + OK for STDH_URB2D 
   + OK for HGT_URB2D 
   + OK for LB_URB2D 
   + OK for LF_URB2D 
Traceback (most recent call last):
  File "/home/matthias/pyvens/w2w/bin/w2w", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/matthias/pyvens/w2w/lib/python3.11/site-packages/w2w/w2w.py", line 234, in main
    checks_and_cleaning(
  File "/home/matthias/pyvens/w2w/lib/python3.11/site-packages/w2w/w2w.py", line 1728, in checks_and_cleaning
    if np.nanmax(np.abs((100 - hi_sum).values)) > 0.1:
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/matthias/pyvens/w2w/lib/python3.11/site-packages/numpy/lib/_nanfunctions_impl.py", line 495, in nanmax
    res = np.fmax.reduce(a, axis=axis, out=out, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: zero-size array to reduction operation fmax which has no identity

I can also confirm to get this error in both these versions:

pip install  --upgrade --force-reinstall git+https://github.com/matthiasdemuzere/w2w.git

pip install  --upgrade --force-reinstall git+https://github.com/matthiasdemuzere/w2w.git@add_wrf_version

Based on the output, it seems to go wrong when introducing the LCZ labels into the LU_INDEX, because of:

WARNING: LCZ-based urban extent missing

At the same time, Check 3 seems also misleading: > Check 3: Urban LCZ classes exists in geo_em.d02_LCZ_params.nc? OK: LCZ Classes ([]) present. The LCZ class array is empty, so this check should not be ok.

I'll try to look into this further ...