Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ValueError: assignment destination is read-only with parallel processing #28

Open
pstradio opened this issue Jun 7, 2024 · 2 comments

Comments

@pstradio
Copy link

pstradio commented Jun 7, 2024

With version v0.11 I am getting the following when trying to ts2img and > 1 n_proc.

ERROR:root:The following ERROR was raised in the parallelized function `_convert` but was ignored due to the chosen settings: Traceback (most recent call last):
  File "/home/pstradio/miniforge3/envs/rep/lib/python3.9/site-packages/repurpose/process.py", line 175, in run_with_error_handling
    r = FUNC(**kwargs)
  File "/home/pstradio/miniforge3/envs/rep/lib/python3.9/site-packages/repurpose/ts2img.py", line 59, in _convert
    writer.write_ts(ts, gpi)
  File "/home/pstradio/miniforge3/envs/rep/lib/python3.9/site-packages/repurpose/stack.py", line 357, in write_ts
    self.ds[var].values[t, row, col] = df[var].values
ValueError: assignment destination is read-only

I'm running the following:

...

inreader = CustomeReader()  # just a wrapper around a pynetcf reader
outgrid = BasicGrid(lon2d.flatten(), lat2d.flatten(), shape=(lats.size, lons.size))
dates = pd.date_range("2015-04-01T06:00:00", "2022-09-29T18:00:00", freq="12h")

conv = Ts2Img(inreader, outgrid, dates,)

conv.calc(
    path_out=os.path.join(BASEPATH, 'images'),
    format_out='slice',
    var_attrs=dict(sm=dict(long_name='soil moisture', units='m3 m-3')),
    n_proc=6,
    img_buffer=190,
    zlib=True,
)

I can reproduce the issue on 2 different machines.

@wpreimes
Copy link
Member

wpreimes commented Jun 7, 2024

hmm it looks like writing to the xarray Dataset (which should be in memory) is not possible. Could it be that the outfile already exists and you cannot overwrite it? Otherwise, I have noticed that some tests failed when n_proc>1. I have fixed this now, maybe it was related. Can you try again with the master branch after #29 is merged (but I think your problem is a different one...).

@pstradio
Copy link
Author

pstradio commented Jun 7, 2024

I tried changing the parallelization backend and it works with threading - I will include in a PR to be able to specify the be

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants