Skip to content

Commit

Permalink
Update sphinx
Browse files Browse the repository at this point in the history
  • Loading branch information
milos.colic committed Jan 15, 2024
1 parent bdafaa5 commit 0ab6646
Show file tree
Hide file tree
Showing 3 changed files with 73 additions and 83 deletions.
4 changes: 2 additions & 2 deletions docs/docs-requirements.txt
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
setuptools==68.1.2
Sphinx==6.1.0
Sphinx==6.1.3
sphinx-material==0.0.36
nbsphinx==0.8.8
nbsphinx>=0.8.8
ipython>=8.10.1
sphinxcontrib-fulltoc==1.2.0
livereload==2.6.3
Expand Down
67 changes: 34 additions & 33 deletions docs/source/api/raster-format-readers.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,22 +4,23 @@ Raster Format Readers


Intro
################
#####
Mosaic provides spark readers for the following raster formats:
* GTiff (GeoTiff) using .tif file extension - https://gdal.org/drivers/raster/gtiff.html
* COG (Cloud Optimized GeoTiff) using .tif file extension - https://gdal.org/drivers/raster/cog.html
* HDF4 using .hdf file extension - https://gdal.org/drivers/raster/hdf4.html
* HDF5 using .h5 file extension - https://gdal.org/drivers/raster/hdf5.html
* NetCDF using .nc file extension - https://gdal.org/drivers/raster/netcdf.html
* JP2ECW using .jp2 file extension - https://gdal.org/drivers/raster/jp2ecw.html
* JP2KAK using .jp2 file extension - https://gdal.org/drivers/raster/jp2kak.html
* JP2OpenJPEG using .jp2 file extension - https://gdal.org/drivers/raster/jp2openjpeg.html
* PDF using .pdf file extension - https://gdal.org/drivers/raster/pdf.html
* PNG using .png file extension - https://gdal.org/drivers/raster/png.html
* VRT using .vrt file extension - https://gdal.org/drivers/raster/vrt.html
* XPM using .xpm file extension - https://gdal.org/drivers/raster/xpm.html
* GRIB using .grb file extension - https://gdal.org/drivers/raster/grib.html
* Zarr using .zarr file extension - https://gdal.org/drivers/raster/zarr.html
* GTiff (GeoTiff) using .tif file extension - https://gdal.org/drivers/raster/gtiff.html
* COG (Cloud Optimized GeoTiff) using .tif file extension - https://gdal.org/drivers/raster/cog.html
* HDF4 using .hdf file extension - https://gdal.org/drivers/raster/hdf4.html
* HDF5 using .h5 file extension - https://gdal.org/drivers/raster/hdf5.html
* NetCDF using .nc file extension - https://gdal.org/drivers/raster/netcdf.html
* JP2ECW using .jp2 file extension - https://gdal.org/drivers/raster/jp2ecw.html
* JP2KAK using .jp2 file extension - https://gdal.org/drivers/raster/jp2kak.html
* JP2OpenJPEG using .jp2 file extension - https://gdal.org/drivers/raster/jp2openjpeg.html
* PDF using .pdf file extension - https://gdal.org/drivers/raster/pdf.html
* PNG using .png file extension - https://gdal.org/drivers/raster/png.html
* VRT using .vrt file extension - https://gdal.org/drivers/raster/vrt.html
* XPM using .xpm file extension - https://gdal.org/drivers/raster/xpm.html
* GRIB using .grb file extension - https://gdal.org/drivers/raster/grib.html
* Zarr using .zarr file extension - https://gdal.org/drivers/raster/zarr.html

Other formats are supported if supported by GDAL available drivers.

Mosaic provides two flavors of the readers:
Expand All @@ -32,14 +33,14 @@ spark.read.format("gdal")
A base Spark SQL data source for reading GDAL raster data sources.
It reads metadata of the raster and exposes the direct paths for the raster files.
The output of the reader is a DataFrame with the following columns:
* tile - loaded raster tile (RasterTileType)
* ySize - height of the raster in pixels (IntegerType)
* xSize - width of the raster in pixels (IntegerType)
* bandCount - number of bands in the raster (IntegerType)
* metadata - raster metadata (MapType(StringType, StringType))
* subdatasets - raster subdatasets (MapType(StringType, StringType))
* srid - raster spatial reference system identifier (IntegerType)
* proj4Str - raster spatial reference system proj4 string (StringType)
* tile - loaded raster tile (RasterTileType)
* ySize - height of the raster in pixels (IntegerType)
* xSize - width of the raster in pixels (IntegerType)
* bandCount - number of bands in the raster (IntegerType)
* metadata - raster metadata (MapType(StringType, StringType))
* subdatasets - raster subdatasets (MapType(StringType, StringType))
* srid - raster spatial reference system identifier (IntegerType)
* proj4Str - raster spatial reference system proj4 string (StringType)

.. function:: spark.read.format("gdal").load(path)

Expand Down Expand Up @@ -94,16 +95,16 @@ If the raster pixels are larger than the grid cells, the cell values can be calc
The interpolation method used is Inverse Distance Weighting (IDW) where the distance function is a k_ring
distance of the grid.
The reader supports the following options:
* fileExtension - file extension of the raster file (StringType) - default is *.*
* vsizip - if the rasters are zipped files, set this to true (BooleanType)
* resolution - resolution of the output grid (IntegerType)
* combiner - combiner operation to use when converting raster to grid (StringType) - default is mean
* retile - if the rasters are too large they can be re-tiled to smaller tiles (BooleanType)
* tileSize - size of the re-tiled tiles, tiles are always squares of tileSize x tileSize (IntegerType)
* readSubdatasets - if the raster has subdatasets set this to true (BooleanType)
* subdatasetNumber - if the raster has subdatasets, select a specific subdataset by index (IntegerType)
* subdatasetName - if the raster has subdatasets, select a specific subdataset by name (StringType)
* kRingInterpolate - if the raster pixels are larger than the grid cells, use k_ring interpolation with n = kRingInterpolate (IntegerType)
* fileExtension - file extension of the raster file (StringType) - default is *.*
* vsizip - if the rasters are zipped files, set this to true (BooleanType)
* resolution - resolution of the output grid (IntegerType)
* combiner - combiner operation to use when converting raster to grid (StringType) - default is mean
* retile - if the rasters are too large they can be re-tiled to smaller tiles (BooleanType)
* tileSize - size of the re-tiled tiles, tiles are always squares of tileSize x tileSize (IntegerType)
* readSubdatasets - if the raster has subdatasets set this to true (BooleanType)
* subdatasetNumber - if the raster has subdatasets, select a specific subdataset by index (IntegerType)
* subdatasetName - if the raster has subdatasets, select a specific subdataset by name (StringType)
* kRingInterpolate - if the raster pixels are larger than the grid cells, use k_ring interpolation with n = kRingInterpolate (IntegerType)

.. function:: mos.read().format("raster_to_grid").load(path)

Expand Down
85 changes: 37 additions & 48 deletions docs/source/api/vector-format-readers.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,36 +8,25 @@ Intro
Mosaic provides spark readers for vector files supported by GDAL OGR drivers.
Only the drivers that are built by default are supported.
Here are some common useful file formats:
* GeoJSON (also ESRIJSON, TopoJSON)
https://gdal.org/drivers/vector/geojson.html
* ESRI File Geodatabase (FileGDB) and ESRI File Geodatabase vector (OpenFileGDB)
Mosaic implements named reader geo_db (described in this doc)
https://gdal.org/drivers/vector/filegdb.html
* ESRI Shapefile / DBF (ESRI Shapefile) - Mosaic implements named reader shapefile (described in this doc)
https://gdal.org/drivers/vector/shapefile.html
* Network Common Data Form (netCDF) - Mosaic implements raster reader also
https://gdal.org/drivers/raster/netcdf.html
* (Geo)Parquet (Parquet) - Mosaic will be implementing a custom reader soon
https://gdal.org/drivers/vector/parquet.html
* Spreadsheets (XLSX, XLS, ODS)
https://gdal.org/drivers/vector/xls.html
* U.S. Census TIGER/Line (TIGER)
https://gdal.org/drivers/vector/tiger.html
* PostgreSQL Dump (PGDump)
https://gdal.org/drivers/vector/pgdump.html
* Keyhole Markup Language (KML)
https://gdal.org/drivers/vector/kml.html
* Geography Markup Language (GML)
https://gdal.org/drivers/vector/gml.html
* GRASS - option for Linear Referencing Systems (LRS)
https://gdal.org/drivers/vector/grass.html
* GeoJSON (also ESRIJSON, TopoJSON) https://gdal.org/drivers/vector/geojson.html
* ESRI File Geodatabase (FileGDB) and ESRI File Geodatabase vector (OpenFileGDB). Mosaic implements named reader geo_db (described in this doc). https://gdal.org/drivers/vector/filegdb.html
* ESRI Shapefile / DBF (ESRI Shapefile) - Mosaic implements named reader shapefile (described in this doc) https://gdal.org/drivers/vector/shapefile.html
* Network Common Data Form (netCDF) - Mosaic implements raster reader also https://gdal.org/drivers/raster/netcdf.html
* (Geo)Parquet (Parquet) - Mosaic will be implementing a custom reader soon https://gdal.org/drivers/vector/parquet.html
* Spreadsheets (XLSX, XLS, ODS) https://gdal.org/drivers/vector/xls.html
* U.S. Census TIGER/Line (TIGER) https://gdal.org/drivers/vector/tiger.html
* PostgreSQL Dump (PGDump) https://gdal.org/drivers/vector/pgdump.html
* Keyhole Markup Language (KML) https://gdal.org/drivers/vector/kml.html
* Geography Markup Language (GML) https://gdal.org/drivers/vector/gml.html
* GRASS - option for Linear Referencing Systems (LRS) https://gdal.org/drivers/vector/grass.html

For more information please refer to gdal documentation: https://gdal.org/drivers/vector/index.html



Mosaic provides two flavors of the readers:
* spark.read.format("ogr") for reading 1 file per spark task
* mos.read().format("multi_read_ogr") for reading file in parallel with multiple spark tasks
* spark.read.format("ogr") for reading 1 file per spark task
* mos.read().format("multi_read_ogr") for reading file in parallel with multiple spark tasks


spark.read.format("ogr")
Expand All @@ -46,17 +35,17 @@ A base Spark SQL data source for reading GDAL vector data sources.
The output of the reader is a DataFrame with inferred schema.
The schema is inferred from both features and fields in the vector file.
Each feature will be provided as 2 columns:
* geometry - geometry of the feature (GeometryType)
* srid - spatial reference system identifier of the feature (StringType)
* geometry - geometry of the feature (GeometryType)
* srid - spatial reference system identifier of the feature (StringType)

The fields of the feature will be provided as columns in the DataFrame.
The types of the fields are coerced to most concrete type that can hold all the values.
The reader supports the following options:
* driverName - GDAL driver name (StringType)
* vsizip - if the vector files are zipped files, set this to true (BooleanType)
* asWKB - if the geometry should be returned as WKB (BooleanType) - default is false
* layerName - name of the layer to read (StringType)
* layerNumber - number of the layer to read (IntegerType)
* driverName - GDAL driver name (StringType)
* vsizip - if the vector files are zipped files, set this to true (BooleanType)
* asWKB - if the geometry should be returned as WKB (BooleanType) - default is false
* layerName - name of the layer to read (StringType)
* layerNumber - number of the layer to read (IntegerType)


.. function:: read.format("ogr").load(path)
Expand Down Expand Up @@ -109,18 +98,18 @@ Chunk size is the number of file rows that will be read per single task.
The output of the reader is a DataFrame with inferred schema.
The schema is inferred from both features and fields in the vector file.
Each feature will be provided as 2 columns:
* geometry - geometry of the feature (GeometryType)
* srid - spatial reference system identifier of the feature (StringType)
* geometry - geometry of the feature (GeometryType)
* srid - spatial reference system identifier of the feature (StringType)

The fields of the feature will be provided as columns in the DataFrame.
The types of the fields are coerced to most concrete type that can hold all the values.
The reader supports the following options:
* driverName - GDAL driver name (StringType)
* vsizip - if the vector files are zipped files, set this to true (BooleanType)
* asWKB - if the geometry should be returned as WKB (BooleanType) - default is false
* chunkSize - size of the chunk to read from the file per single task (IntegerType) - default is 5000
* layerName - name of the layer to read (StringType)
* layerNumber - number of the layer to read (IntegerType)
* driverName - GDAL driver name (StringType)
* vsizip - if the vector files are zipped files, set this to true (BooleanType)
* asWKB - if the geometry should be returned as WKB (BooleanType) - default is false
* chunkSize - size of the chunk to read from the file per single task (IntegerType) - default is 5000
* layerName - name of the layer to read (StringType)
* layerNumber - number of the layer to read (IntegerType)


.. function:: read.format("multi_read_ogr").load(path)
Expand Down Expand Up @@ -171,10 +160,10 @@ Mosaic provides a reader for GeoDB files natively in Spark.
The output of the reader is a DataFrame with inferred schema.
Only 1 file per task is read. For parallel reading of large files use the multi_read_ogr reader.
The reader supports the following options:
* asWKB - if the geometry should be returned as WKB (BooleanType) - default is false
* layerName - name of the layer to read (StringType)
* layerNumber - number of the layer to read (IntegerType)
* vsizip - if the vector files are zipped files, set this to true (BooleanType)
* asWKB - if the geometry should be returned as WKB (BooleanType) - default is false
* layerName - name of the layer to read (StringType)
* layerNumber - number of the layer to read (IntegerType)
* vsizip - if the vector files are zipped files, set this to true (BooleanType)

.. function:: read.format("geo_db").load(path)

Expand Down Expand Up @@ -223,10 +212,10 @@ Mosaic provides a reader for Shapefiles natively in Spark.
The output of the reader is a DataFrame with inferred schema.
Only 1 file per task is read. For parallel reading of large files use the multi_read_ogr reader.
The reader supports the following options:
* asWKB - if the geometry should be returned as WKB (BooleanType) - default is false
* layerName - name of the layer to read (StringType)
* layerNumber - number of the layer to read (IntegerType)
* vsizip - if the vector files are zipped files, set this to true (BooleanType)
* asWKB - if the geometry should be returned as WKB (BooleanType) - default is false
* layerName - name of the layer to read (StringType)
* layerNumber - number of the layer to read (IntegerType)
* vsizip - if the vector files are zipped files, set this to true (BooleanType)

.. function:: read.format("shapefile").load(path)

Expand Down

0 comments on commit 0ab6646

Please sign in to comment.