Large Time Series

[apfeltasche]

Dear everyone,

I am struggling with large data frames.
The dataframe consists of on sensor value (50 kHz) with a length of 4 s.

It seems like, when the number of rows exceeds 50000, the tool is getting very slow?

Is there a workaround, or any advice?

Thank you all very much!

Kind Regards

Johannes

[nils-braun]

Hi @apfeltasche!

Yes, multiple possibilities :-)

• Try to reduce the features you extract, e.g. by using one of the other extraction settings. Especially the EfficientFCParameters might already help (background: we have some feature extractors that scale with n^2). We have documentation on how to use them.
• Even better would be if you know already in detail, which features you are interested in. For example extract and select on a small sample, then only extract the selected features on all data.
• Use parallelization with one of the supported frameworks (maybe dask, maybe just multiprocessing). We have also some documentation on that, maybe you want to start here or here. Please note, that this will parallelize over different time series - and not within one time series.

[apfeltasche]

Thanks a lot for the fast answer!

[apfeltasche]

So after a quick look, it seems like only "def sample_entropy(x):" and def binned_entropy(x, max_bins): wont be calculated?

[nils-braun]

That is correct. Those are (by far, actually) the slowest in large time series.