[WISE FR]: Stop outputting shape files in favour of geopackages.

[spydmobile]

This feature needs to be reviewed by exec before we proceed.

[BadgerOnABike]

Are we stopping shapefiles or just also supporting geopackages? I think there is a cut over phase wherein people don't actually know how to use a geopackage yet. Apologies for not looking at this yesterday!

[nealmcloughlin]

I have no problem adding support for geopackages. However, we will not be popular if we remove support for Shapefiles. I'm not seeing the rationale for for why we would stop outputting to Shapefiles. Can someone fill me in on this?

[spydmobile]

@nealmcloughlin shape-files are already causing us to code around its weaknesses, the name collision issue is caused by weaknesses in the shape-file format. But if you want to drink the cool-aid im drinking, this is a good start:
http://switchfromshapefile.org/

[spydmobile]

Also, all ESRI products and all respectable GIS software, can read geopackages while Shapefiles are legacy and we should move on.

[spydmobile]

Lastly, this issue is to deprecate OUTPUTTING shapefiles, not reading them.

[BadgerOnABike]

If we already have logic to truncate the headers I'd suggest we leave it in for now as many folks are still using it. We make the default GPKG but have shp kicking around. Arguably, if GDAL can push it out or pull it in, we should have a method to handle that, but that's a different problem.

[spydmobile]

We have already approved this, work on it has already been started. changing the plan now, is NOT how we should do business, its bad form and a bad habit, that said, if that is what you both want, lets tell him to to add GPKG and leave shape files alone.

[nealmcloughlin]

Great website. Adding GPKG as an output format and making it the default is a no brainer. Interesting, you can export raster and vector data to GPKG files. I can also get behind depreciating Shapefile outputs. However, I'd like to know that we've used GPKG vector files with our various GIS systems. I currently have no experience with these files. I had a look to see if I can read a GPKG file into R. The GDAL library (ver 3.5.2) associated with the latest R 4.2.2 can read/write raster data in a GPKG file. However, there does not appear to be support to read/write vector data in a GPKG file.

[nealmcloughlin]

I agree with Brett that we should keep Shapefiles around until GPKG outputs have been thoroughly tested within a WISE context. This is bringing back memories of how long it took to get KML outputs working properly.

[spydmobile]

Roger that! thanks for feeding back so quickly!

[spydmobile]

@RobBryce we will NOT deprecate shapefile outputs for 1.0, BUT we will Add I/O support for GPackage, please make the required changes to your development plan and actions.

[BadgerOnABike]

Great website. Adding GPKG as an output format and making it the default is a no brainer. Interesting, you can export raster and vector data to GPKG files. I can also get behind depreciating Shapefile outputs. However, I'd like to know that we've used GPKG vector files with our various GIS systems. I currently have no experience with these files. I had a look to see if I can read a GPKG file into R. The GDAL library (ver 3.5.2) associated with the latest R 4.2.2 can read/write raster data in a GPKG file. However, there does not appear to be support to read/write vector data in a GPKG file.

R, via sf definitely supports vector in GPKG, its all I use.

https://gdal.org/drivers/vector/gpkg.html

[RobBryce]

@BadgerOnABike - should this work also cover V6 (Prometheus) and PrometheusCOM?

[spydmobile]

@RobBryce Hi, If I have my head screwed on correctly, I think that the WISE version 1 export (100% Prometheus Compatible) is enough, and that you can forgo modifying P6 with GPKG as it currently holds no advantage for P/WISE compatibility. We can always do this later before EOL if @BadgerOnABike or @nealmcloughlin feel we should add it in.

[BadgerOnABike]

Personally I'd not worry about it in P6, effort for little pay off in my opinion. I'd say leave it to WISE to perform broader GDAL compliant outputting.

[nealmcloughlin]

Agree. Geopackages are not needed in Prometheus V6.

[RobBryce]

Some testing shows that GDAL's engine for Geopackages exhibit an exponential slowdown based on the number of polygons. This has been confirmed even with GDAL's own OGR2OGR. This may not be a concern for some advantages over SHP files (better attribute tables), but isn't a solution to the SHP 2GB limit.

[spydmobile]

@RobBryce anything I have read on the subject indicates that this can be avoided by not submitting individual transaction writes, but rather batches as geopackage is essential SQLite and the other thing is there are very specific optimizations in gdal for geopackages and getting the wrong flags results in perfomance hits. I get the sense you can build very poor geopackages that work but have terrible performance, but it also looks like building them correctly might avoid that, Has there been anything recent on this Rob?