Speed up Feature.process_*()

[frankenjoe]

Relates to #100

Speeds up Feature.process_*() by avoiding calls to pd.concat() on a lot of small pd.DataFrame objects. Instead, we now concatenate lists / numpy arrays and create the final pd.DataFrame from those.

Benchmark

import time
import audb
import audinterface


db = audb.load(
    'emodb',
    version='1.3.0',
    format='wav',
    sampling_rate=16000,
    mixdown=True,
)

def process_func(x, sr):
    return [x.mean(), x.std()]

feature = audinterface.Feature(
    ['mean', 'std'],
    process_func=process_func,
)

t = time.time()
feature.process_files(db.files)
print(time.time() - t)

• main branch: ~8.5 s
• this branch: ~4.5 s

[frankenjoe]

Together with #102 we should be down to ~0.35s, which is close to what we target in #100.

[codecov]

Codecov Report

Merging #103 (c11ca68) into main (7ad15b7) will not change coverage.
The diff coverage is 100.0%.

Impacted Files	Coverage Δ
audinterface/core/feature.py	100.0% <100.0%> (ø)

[hagenw]

You are not only concatenating lists, but also numpy arrays. But I guess the problem was concatenating with pandas before which seems to be much slower (most likely due to index related stuff) than concatenating with numpy.

[frankenjoe]

You are not only concatenating lists, but also numpy arrays

Actually, in the win_dur=None case we even pre-allocate the numpy array and assign directly to it. When a sliding is used this is not possible as we cannot know the matrix size in advance, so here we rely on np.concatenate(), which is still much faster than pd.concat().

[hagenw]

OK, thanks for clarification.

[hagenw]

With the incorporation of #102 the benchmark is down to ~0.26 s for me.