Speed up processing: DB insertion and multiprocessing

[nesnoj]

Succeeding issue of recently closed #546.
I merged both topics into one issue as I think they interfere - e.g. choosing a specific DB insertion method might not work (efficiently) in multiprocessing. If you disagree, feel free to separate them.

1. Explore faster methods of writing to the database for

• sqlite
• postgres

2. Add multiprocessing for

• XML parsing
• Write to DB (if applicable due to concurrency -> table locks)

Notes on DB insertion: #546 (comment)
Notes on parallelization: #546 (comment)

Feel free to amend :)

[FlorianK13]

we reached #600 🥇

[AlexandraImbrisca]

Hi @nesnoj! Thanks for creating this issue. I'm testing my approach a bit more (different number of cores & different operating systems) and I'll create the PR. I've been developing and testing on MacOS & Linux and I'll continue with Windows. The approach is quite simple and uses the standard concurrent.futures library with a few options to optimize the access to the database

About writing the data to the postgre database: would you mind tackling this separately? I didn't get the chance to look into how optimize writing yet and the parallelization is database-agnostic right now

[nesnoj]

About writing the data to the postgre database: would you mind tackling this separately? I didn't get the chance to look into how optimize writing yet and the parallelization is database-agnostic right now

Sure, feel free to create a separate issue if that makes sense to you.