Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

r.connectivity.distance: implement walking distance and parallel processing #1108

Open
wants to merge 3 commits into
base: grass8
Choose a base branch
from

Conversation

ninsbl
Copy link
Member

@ninsbl ninsbl commented Jun 18, 2024

Include also #1091

@ninsbl ninsbl added enhancement New feature or request Python Related code is in Python labels Jun 18, 2024
@ninsbl ninsbl requested a review from ecodiv June 18, 2024 21:29
@ecodiv
Copy link
Contributor

ecodiv commented Jun 22, 2024

@ninsbl I tested it using the example from the manual page, resulting in the following error:

gs.run_command(
...     "r.connectivity.distance",
...     flags="k",
...     input="patches_1ha",
...     pop_proxy="area_ha",
...     costs="costs",
...     prefix="hws_connectivity2",
...     cutoff=4500,
...     border_dist=18,
...     conefor_dir="./conefor",
...     nprocs=10,
... )
Traceback (most recent call last):
  File "/home/paulo/.grass8/addons/scripts/r.connectivity.distance", line 1225, in <module>
    sys.exit(main())
  File "/home/paulo/.grass8/addons/scripts/r.connectivity.distance", line 1142, in main
    pool.starmap(compute_distances_parallel, cat_chunks)
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 375, in starmap
    return self._map_async(func, iterable, starmapstar, chunksize).get()
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 774, in get
    raise self._value
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 540, in _handle_tasks
    put(task)
  File "/usr/lib/python3.10/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/usr/lib/python3.10/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
  File "/usr/local/grassmaster/grass85/etc/python/grass/script/utils.py", line 154, in __getattr__
    return self[key]
KeyError: '__getstate__'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/grassmaster/grass85/etc/python/grass/script/core.py", line 487, in run_command
    return handle_errors(returncode, result=None, args=args, kwargs=kwargs)
  File "/usr/local/grassmaster/grass85/etc/python/grass/script/core.py", line 366, in handle_errors
    raise CalledModuleError(module=module, code=code, returncode=returncode)
  File "/usr/local/grassmaster/grass85/etc/python/grass/exceptions/__init__.py", line 87, in __init__
    err = _("See errors above the traceback or in the error output.")
TypeError: 'int' object is not callable

@ninsbl
Copy link
Member Author

ninsbl commented Jun 22, 2024

Hm... The eror you see is likely related to the number of processes (nprocs). Could you try with nprocs=4 for example?
With 10 cores I get a different error message (with GRASS 8.3) . But with 4 cores it runs fine...

@ecodiv
Copy link
Contributor

ecodiv commented Jun 23, 2024

Hm... The eror you see is likely related to the number of processes (nprocs). Could you try with nprocs=4 for example? With 10 cores I get a different error message (with GRASS 8.3) . But with 4 cores it runs fine...

Unfortunately, for me it doesn't work with 2 or 4 cores either. One core does work.

>>> gs.run_command(
...     "r.connectivity.distance",
...     flags="pk",
...     input="patches_1ha",
...     pop_proxy="area_ha",
...     costs="costs",
...     prefix="hws_connectivity4",
...     cutoff=1000,
...     border_dist=18,
...     nprocs=2
... )
Traceback (most recent call last):
  File "/home/paulo/.grass8/addons/scripts/r.connectivity.distance", line 1225, in <module>
    sys.exit(main())
  File "/home/paulo/.grass8/addons/scripts/r.connectivity.distance", line 1142, in main
    pool.starmap(compute_distances_parallel, cat_chunks)
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 375, in starmap
    return self._map_async(func, iterable, starmapstar, chunksize).get()
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 774, in get
    raise self._value
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 540, in _handle_tasks
    put(task)
  File "/usr/lib/python3.10/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/usr/lib/python3.10/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
  File "/usr/local/grassmaster/grass85/etc/python/grass/script/utils.py", line 154, in __getattr__
    return self[key]
KeyError: '__getstate__'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/grassmaster/grass85/etc/python/grass/script/core.py", line 487, in run_command
    return handle_errors(returncode, result=None, args=args, kwargs=kwargs)
  File "/usr/local/grassmaster/grass85/etc/python/grass/script/core.py", line 366, in handle_errors
    raise CalledModuleError(module=module, code=code, returncode=returncode)
grass.exceptions.CalledModuleError: Module run `r.connectivity.distance -pk input=patches_1ha pop_proxy=area_ha costs=costs prefix=hws_connectivity4 cutoff=1000 border_dist=18 nprocs=2` ended with an error.
The subprocess ended with a non-zero return code: 1. See errors above the traceback or in the error output.

@echoix
Copy link
Member

echoix commented Jun 23, 2024

Is one of you using a platform other than Linux by any chance? If so, maybe the use of multiprocessing.Pool is using the start method other than fork, like spawn, that requires that what is ran to be Pickle-able. Sometimes the file needs to be reorganized, or even the part that is ran by multiple processes be specially designed and extracted in another file to be executed correctly.

The traceback shown doesn't properly mention the real error for me.

Alternatively, is there any chance that the Python version is 3.12 running on Linux, and using a multiprocessing function using the fork startup method also calls a sub process, such as it raises a DepreciationWarning (it can cause deadlocks and the default startup method will change to spawn in 3.14), and that this depreciationwarning is saved to a results file, making it invalid?

@echoix
Copy link
Member

echoix commented Jun 23, 2024

Using multiprocessing correctly, especially in a cross platform manner, is a little bit trickier than one can imagine

@ecodiv
Copy link
Contributor

ecodiv commented Jun 23, 2024

Alternatively, is there any chance that the Python version is 3.12 running on Linux,

I am running version 3.10.12, the default on Ubuntu 22.04

@ecodiv
Copy link
Contributor

ecodiv commented Jun 23, 2024

@echoix @ninsbl I don't know if it is helpful, but the attached log.txt shows the full debug output when setting debug to level 5.

@echoix echoix changed the title r.connectivty.distance: implement walking distance and parallel processing r.connectivity.distance: implement walking distance and parallel processing Jun 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request Python Related code is in Python
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants