Handle large number of measurements

[bajtos]

The current implementation of the evaluation service keeps all measurements in memory. This opens the door for a DoS attack where an attacker can submit a very large number of measurements to trigger an out-of-memory error in our service.

Places we need to fix:

• When fetching measurements inside a CID, we should reject files that are too large (e.g. more than 1MB).
• We should process measurements incrementally in a streaming way.
• We should store measurements on the disk or in a database
• The evaluation process should not load all measurements into memory at once

[bajtos]

See also:

• Monitor size of in-memory data (publish, evaluate) spark-api#123

[bajtos]

We should store measurements on the disk or in a database

Let's use sqlite3, it's the tool recommended by Fly.io.

https://fly.io/blog/all-in-on-sqlite-litestream/

[juliangruber]

Sgtm!