Use mapshaper to simplify a region geometry

[ismaelbej]

• Modify load-regions to use mapshaper to simplify a region using the distance parameter
• Update load-friction-raster to pass the -cblend 1 option
• Add -f parameter to load-regions to update an existing region
• Set INTERVAL_DISTANCE=100 as default distance for simplification
• Document new parameters in scripts/population/README.md

Closes #727

[ggiraldez]

This works but will not reprocess the existing regions (if I understand correctly) and we can't delete them and re-add them because they already have referencing projects.

It would also be nice to see the resulting size of the simplified shape, to adjust the interval if deemed necessary. For large countries (or large provinces) the default threshold may not be enough to reduce the size enough and still hog the CPU when performing spatial queries.

[ggiraldez]

Could you please expand the comment above explaining that in our case since our GeoJSON are not projected (ie. use WGS84 coordinates) then this means the resolution for the simplified shape will be 1km? (Or at least that's what I understand from the documentation 😁)

[ismaelbej]

I wasn't sure about what the parameter was measuring in this case. I compared the mapshaper output of the Kenya map or 100 and 1000 (it is the island on the south east)

[ggiraldez]

Hmm... I'm thinking this can potentially lead to legitimate providers being excluded. Maybe we should play a bit with the thresholds and the problematic countries and check the resulting sizes. We may get away with a smaller threshold. @manumoreira what was the country that triggered the original issue?

[ggiraldez]

In fact, a default of 100 may work fine. I'm testing locally with some datasets I have and Philipines shrinks from 58 Mb to 2.5 Mb using 100 as the threshold. That should be acceptable I think, since Kenya's contour is 4.9 Mb unprocessed and we didn't have problems with that country so far. NB. sizes are of the admin level 0 .geojson files.

[manumoreira]

The country was Mozambique

[ismaelbej]

Comparison between the original and the 100 meters simplification

[ismaelbej]

I had to zoom more because the difference wasn't visible from https://mapshaper.org/. It is the same island from the previous screenshots on the south east of Kenya.

[ismaelbej]

Comparison for Mozambique at the north betwee original resolution and 100m maps

Testing files
moz-geojson.tgz. I used https://mapshaper.org/ to load them in separate layers.

[ismaelbej]

I'll add a -f to force updating the geometries.

[matiasgarciaisaia]

Can't comment further on the task being done - I'll trust Gus' opinion on that.

[matiasgarciaisaia]

LGTM!

As a note for deployment, we'll probably need to manually fix the input file with the duplicate keys when running this in production.