You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
the work around
`# returns a list of signatures using parallel processing
import esig, joblib, numpy as np
def parallel_stream2sig(paths,depth):
return joblib.Parallel(n_jobs=-1)(joblib.delayed(esig.stream2sig)(x,depth) for x in paths)
paths = [np.random.rand(20,2) for i in range (10)]
print(parallel_stream2sig(paths,4))`
works but
`# returns a list of signatures using parallel processing
import esig, joblib, numpy as np
def parallel_stream2sig(paths,depth):
return joblib.Parallel(n_jobs=-1, prefer="threads")(joblib.delayed(esig.stream2sig)(x,depth) for x in paths)
paths = [np.random.rand(20,2) for i in range (10)]
print(parallel_stream2sig(paths,4))`
causes the following crash:
Traceback (most recent call last):
File "C:\Users\t\source\repos\esig_batch\esig_batch\esig_batch.py", line 9, in
print(parallel_stream2sig(paths,4))
File "C:\Users\t\source\repos\esig_batch\esig_batch\esig_batch.py", line 5, in parallel_stream2sig
return joblib.Parallel(n_jobs=-1, prefer="threads")(joblib.delayed(esig.stream2sig)(x,depth) for x in paths)
File "C:\Users\t\source\repos\esig_batch\esig_batch\env\lib\site-packages\joblib\parallel.py", line 1054, in call
self.retrieve()
File "C:\Users\t\source\repos\esig_batch\esig_batch\env\lib\site-packages\joblib\parallel.py", line 933, in retrieve
self.output.extend(job.get(timeout=self.timeout))
File "C:\users\t\source\Python37_64\lib\multiprocessing\pool.py", line 657, in get
raise self.value
File "C:\users\t\source\Python37_64\lib\multiprocessing\pool.py", line 121, in worker
result = (True, func(*args, **kwds))
File "C:\Users\t\source\repos\esig_batch\esig_batch\env\lib\site-packages\joblib_parallel_backends.py", line 595, in call
return self.func(*args, **kwargs)
File "C:\Users\t\source\repos\esig_batch\esig_batch\env\lib\site-packages\joblib\parallel.py", line 263, in call
for func, args, kwargs in self.items]
File "C:\Users\t\source\repos\esig_batch\esig_batch\env\lib\site-packages\joblib\parallel.py", line 263, in
for func, args, kwargs in self.items]
File "C:\Users\t\source\repos\esig_batch\esig_batch\env\lib\site-packages\esig_init.py", line 133, in wrapper
return func(as_array, *args, **kwargs)
File "C:\Users\t\source\repos\esig_batch\esig_batch\env\lib\site-packages\esig_init.py", line 151, in stream2sig
backend = get_backend()
File "C:\Users\t\source\repos\esig_batch\esig_batch\env\lib\site-packages\esig\backends.py", line 35, in get_backend
return _BACKEND_CONTAINER.context
AttributeError: '_thread._local' object has no attribute 'context'
The text was updated successfully, but these errors were encountered:
When you know that the function you are calling is based on a compiled extension that releases the Python Global Interpreter Lock (GIL) during most of its computation then it is more efficient to use threads instead of Python processes as concurrent workers. For instance this is the case if you write the CPU intensive part of your code inside a with nogil block of a Cython function.
I don't know if the esig extension releases the GIL. So perhaps this is an issue too.
the work around
`# returns a list of signatures using parallel processing
import esig, joblib, numpy as np
def parallel_stream2sig(paths,depth):
return joblib.Parallel(n_jobs=-1)(joblib.delayed(esig.stream2sig)(x,depth) for x in paths)
paths = [np.random.rand(20,2) for i in range (10)]
print(parallel_stream2sig(paths,4))`
works but
`# returns a list of signatures using parallel processing
import esig, joblib, numpy as np
def parallel_stream2sig(paths,depth):
return joblib.Parallel(n_jobs=-1, prefer="threads")(joblib.delayed(esig.stream2sig)(x,depth) for x in paths)
paths = [np.random.rand(20,2) for i in range (10)]
print(parallel_stream2sig(paths,4))`
causes the following crash:
The text was updated successfully, but these errors were encountered: