Skip to content

Try to avoid pblock overflow by tracking estimated size

Compare
Choose a tag to compare
@zerebubuth zerebubuth released this 05 Oct 16:35
· 5 commits to master since this release

Planet dump would fail intermittently complaining of block overflow (#25). The root cause of this was failing to encode relation 6677259 in the history PBF.

PBF files are sequences of individual blocks and every block must be smaller than 16MiB. To try and satisfy this, the PBF encoder checks the size of the encoded pblock on a regular basis, every N elements. The assumption is that the average size of the element multiplied by N would be smaller than 16MiB. Unfortunately with relation 6677259 that wasn't the case, as versions in its history have more than 25,000 members and were taking up about 100kiB each (so only 164 versions are needed to overflow, relation 6677259 has 440-ish).

This change implements an approximate size counter for relations, which should help to make sure that the pgroup is flushed when it gets too large, even if it hasn't reached N elements yet. This hasn't seemed necessary yet for ways (they are limited to a fixed number of nodes each) or nodes, but extending to handle those - if necessary - should be straightforward.