-
Notifications
You must be signed in to change notification settings - Fork 103
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[MNT] Add joblib backend option and set default to all parallelized estimators #1797
Comments
I remember we looked at this in some detail when debugging some rocket features, was there some interaction with its use of prange? I cant remember! def _static_transform_uni(X, parameters, indices):
"""Transform a 2D collection of univariate time series.
Implemented separately to the multivariate version for numba efficiency reasons.
See issue #1778.
"""
n_cases, n_timepoints = X.shape
(
_,
_,
dilations,
n_features_per_dilation,
biases,
) = parameters
n_kernels = len(indices)
n_dilations = len(dilations)
f = n_kernels * np.sum(n_features_per_dilation)
features = np.zeros((n_cases, f), dtype=np.float32)
for i in prange(n_cases): |
I think adding this parameter to so many APIs would clutter the parameter lists. What are your thoughts on using with joblib.parallel_config(backend="loky"):
# aeon code The same is actually possible for |
True, forgot about this option ! That or we set a global variable such as |
A custom env var |
Sounds like a good idea, would have to be documented though. Not sure about removing |
I'm OK with the use of the
You obtain This means that if we want the Or did I miss something ? |
ah, damn ... yes, |
So for options we have :
With an equal level of documentation, I think the first options would allow us to be more flexible if for whatever reason we need to use processes for some estimators and threads for others. Also it should be less prone to causing issues with existing methods. But true, this adds a bit of parameter bloat to the API... We could introduce a dictionary regrouping all joblib params and use it as kwargs when calling parallel ? Other ideas are welcome ! |
I'm also in favor of 1. How many joblib parameters do we have? If there are just 2, I would not use kwargs for that. |
Describe the issue
Some of the estimators that use joblib for parallelization use process-based backend, while other use threads-based backend. Ideally, we want this to be a parameter tunable by the users.
Suggest a potential alternative/fix
Including a
joblib_backend
parameter which would default tothreading
(from discussions with Matthew), and use this parameter to set the joblibbackend
parameter duringParallel
calls would fix the issue.The text was updated successfully, but these errors were encountered: