GeoJSON I/O Support

[respatialized]

Hi! I work with a lot of spatial data in tablecloth and frequently find myself wishing for a uniform and standardized approach for reading and writing GeoJSON files rather than reimplementing it in an ad-hoc way for each project.

I would be more than willing to contribute code for a PR, and am happy to sketch out my ideas further before I get started.

I ask here about this feature before diving in because I don't know whether it should be a part of tablecloth, an upstream contribution to tech.ml.dataset, or even part of a helper library like tcutils.

Any thoughts on where it makes the most sense for GeoJSON support to live, or any other feedback, are most welcome. Thanks for your time!

[genmeblog]

Good idea! Let's discuss it here. There was some functionality implemented in TC at the beginning moved later to TMD (filling missing values, pretty printing numbers, and some more).
So let's sketch it here and decide later where to put it eventually.

[daslu]

I think @willcohen has been exploring related questions and may comment about it (possibly available in Zulip if not here).

We described some related usage patterns in this tutorial draft.
https://scicloj.github.io/clojure-data-scrapbook/projects/geography/seattle-parks/index.html

[daslu]

One thing we can learn from the tutorial above, is that sometimes geometries are worth representing as jts objects (useful for coordinate transformations, indexing, and other operations), and sometimes as plain EDN (useful for visualization, for example).

[daslu]

related Zulip thread:
data-science > tech.ml.dataset for GIS

[respatialized]

I am disinclined to introduce a dependency on JTS or any other spatial data structure library for the purposes of any code that would be contributed to tablecloth. I think it's out of scope for tablecloth to take a stance on the representation of a geometry object for reading and writing GeoJSON datasets.

With this in mind, here's the sketch of I/O as I see it:

Input

(require '[tablecloth.api :as tc] '[charred.api :as json])

(defn geojson->ds
  "Reads a GeoJSON file from the given input path and converts it to a dataset.

  Derives column data from the \"properties\" entry of each Feature.
  Geometry is stored in a :geometry column."
  [input opts]
  (let [geojson-data (json/read-json input)]
    (tc/dataset (map (fn feature->map [{:keys [geometry properties]}]
                       (assoc properties :geometry geometry))
                     (geojson-data :features)))))

Output

(defn ds->geojson
  "Writes the given dataset to GeoJSON at the given output location.

  By default, assumes that the geometry is in a column called \"geometry\",
  and is already in the Clojure equivalent format of a RFC7946§3.1 Geometry.

  Otherwise, calls geometry-fn on the geometry to convert it to this format."
  [ds output
   {:keys [geometry-fn geometry-col]
    :or   {geometry-fn identity geometry-col "geometry"}}]
  (let [feature-data
        (map (fn map->feature [r]
               (let [geom (r geometry-col)]
; disregard, a null geometry is a valid GeoJSON feature per RFC7946§3.2
;                 (if (nil? geom)
;                  (throw (ex-info "Non-nil geometry column must be specified"
;                                   {:column-names    (tc/column-names ds)
;                                    :geometry-column geometry-col})))
                 {:type       "Feature"
                  :geometry   (geometry-fn geom)
                  :properties (dissoc r geometry-col)}))
             (tc/rows ds :as-maps))])
  (json/write-json output {:type "FeatureCollection" :features feature-data})
  output)

The general idea here is that conversion to and from spatial types like those provided by JTS can come after input or before output. If you are working with libraries like JTS and tablecloth already, you probably already know what you need to do. It's easy enough to convert from the "JSON-like" representation - which will still contain a :type key indicating what type the geometry is - to the representation of your choice, whether that's a JTS Geometry, a JSON string with a Feature, or even more exotic options like a thi.ng geometry. In fact, geometry-fn option in the ds->geojson example above might be completely superfluous if you use map-columns to perform the conversion.

I think keeping the scope small for this contribution makes sense - what do you think @daslu @genmeblog?

[respatialized]

Also, for reference, the GeoJSON spec: https://www.rfc-editor.org/rfc/rfc7946

[daslu]

That is so nice. If we are talking about an addition to Tablecloth itself, this is probably the kind of functions that would make sense.

In my habits and use cases, JTS would often be preferable, and the only case I am aware of where plain Clojure structures would be needed is when we need to generate GeoJSON for another program that expects it (e.g., for data visualization). It'd be enlightening to hear more about your use cases.

A few minor thoughts:

1. Just curious, why did you choose to use :geometry in one direction and "geometry" in the other?

2. Maybe it'd be nice to have geometry-fn in both directions.

3. I'm wondering whether the function names should use ds or dataset.

4. It would be helpful to make geoclj->ds, ds->geoclj functions with something we may call GeoClj format. This is basically Clojure maps which represent the GeoJSON Features, having :geometry and :properties ask keys. The value for :properties is a plain map, and the value for :geometry can either be a JTS geometry, or a plain Clojure data structure corresponding to the GeoJSON geometry.

• Some users may compose such functions with charred for json->clj, clj->json, and thus achieve the functionality you described above (that may still deserve its own functions geojson->ds, ds->geojson).
• Other users may compose with Factual/geo geo.io functions, that convert to and from GeoJSON, etc., but where the geometries are JTS objects. This way, we can create datasets with JTS objects without Tablecloth having to know about what JTS is.