From 71941507525ec2e0245f8af9bbffc25c4e77efc8 Mon Sep 17 00:00:00 2001 From: Michael Johns Date: Thu, 25 Jan 2024 14:07:46 -0500 Subject: [PATCH 1/5] updated additional docs. --- docs/source/api/raster-format-readers.rst | 22 ++++++++++++----- docs/source/api/raster-functions.rst | 6 ++--- docs/source/api/spatial-functions.rst | 18 +++++++------- docs/source/api/spatial-indexing.rst | 4 ++-- docs/source/api/spatial-predicates.rst | 2 +- docs/source/api/vector-format-readers.rst | 24 ++++++++++++++++++- .../usage/automatic-sql-registration.rst | 8 ++++--- docs/source/usage/install-gdal.rst | 4 ++-- 8 files changed, 60 insertions(+), 28 deletions(-) diff --git a/docs/source/api/raster-format-readers.rst b/docs/source/api/raster-format-readers.rst index 262ea066e..1e7a5d189 100644 --- a/docs/source/api/raster-format-readers.rst +++ b/docs/source/api/raster-format-readers.rst @@ -27,7 +27,7 @@ Other formats are supported if supported by GDAL available drivers. Mosaic provides two flavors of the readers: * :code:`spark.read.format("gdal")` for reading 1 file per spark task - * :code: `mos.read().format("raster_to_grid")` reader that automatically converts raster to grid. + * :code:`mos.read().format("raster_to_grid")` reader that automatically converts raster to grid. spark.read.format("gdal") @@ -48,10 +48,10 @@ The output of the reader is a DataFrame with the following columns: .. function:: spark.read.format("gdal").load(path) Loads a GDAL raster file and returns the result as a DataFrame. - It uses standard spark.read.format(*).option(*).load(*) pattern. + It uses standard :code:`spark.read.format(*).option(*).load(*)` pattern. :param path: path to the raster file on dbfs - :type path: *StringType + :type path: Column(StringType) :rtype: DataFrame :example: @@ -81,6 +81,11 @@ The output of the reader is a DataFrame with the following columns: | {index_id: 593308294097928191, raster: [00 01 10 ... 00], parentPath: "dbfs:/path_to_file", driver: "GTiff" } | 100 | 100 | 1 | {AREA_OR_POINT=Po...| null| 4326| +proj=longlat +da...| +---------------------------------------------------------------------------------------------------------------+------+------+----------+---------------------+--------------------+-----+----------------------+ +.. note:: + Keyword options not identified in function signature are converted to a :code:`Map`. + These must be supplied as a :code:`String`. + Also, you can supply function signature values as :code:`String`. + .. warning:: Issue 350: https://github.com/databrickslabs/mosaic/issues/350 The raster reader 'driverName' option has to match the names provided in the above list. @@ -113,10 +118,10 @@ The reader supports the following options: .. function:: mos.read().format("raster_to_grid").load(path) Loads a GDAL raster file and returns the result as a DataFrame. - It uses standard mos.read().format(*).option(*).load(*) pattern. + It uses standard :code:`mos.read().format(*).option(*).load(*)` pattern. :param path: path to the raster file on dbfs - :type path: *StringType + :type path: Column(StringType) :rtype: DataFrame :example: @@ -162,7 +167,12 @@ The reader supports the following options: | 1| 4|0.2464000000000000| +--------+--------+------------------+ +.. note:: + Keyword options not identified in function signature are converted to a :code:`Map`. + These must be supplied as a :code:`String`. + Also, you can supply function signature values as :code:`String`. + .. warning:: Issue 350: https://github.com/databrickslabs/mosaic/issues/350 The option 'fileExtension' expects a wild card mask. Please use the following format: '*.tif' or equivalent for other formats. - If you use 'tif' without the wildcard the reader wont pick up any files and you will have empty table as a result. \ No newline at end of file + If you use 'tif' without the wildcard the reader wont pick up any files and you will have empty table as a result. diff --git a/docs/source/api/raster-functions.rst b/docs/source/api/raster-functions.rst index 71eba6226..1a4465d4f 100644 --- a/docs/source/api/raster-functions.rst +++ b/docs/source/api/raster-functions.rst @@ -192,7 +192,7 @@ rst_combineavg The output raster will have the same pixel type as the input rasters. The output raster will have the same pixel size as the input rasters. The output raster will have the same coordinate reference system as the input rasters. - Also, see :doc:`rst_combineavg_agg ` function. + Also, see :ref:`rst_combineavg_agg` function. :param tiles: A column containing an array of raster tiles. :type tiles: Column (ArrayType(RasterTileType)) @@ -246,7 +246,7 @@ rst_derivedband The output raster will have the same pixel type as the input rasters. The output raster will have the same pixel size as the input rasters. The output raster will have the same coordinate reference system as the input rasters. - Also, see :doc:`rst_derivedband_agg ` function. + Also, see :ref:`rst_derivedband_agg` function. :param tiles: A column containing an array of raster tiles. :type tiles: Column (ArrayType(RasterTileType)) @@ -879,7 +879,7 @@ rst_merge The output raster will have the same pixel type as the input rasters. The output raster will have the same pixel size as the highest resolution input rasters. The output raster will have the same coordinate reference system as the input rasters. - Also, see :doc:`rst_merge_agg ` function. + Also, see :ref:`rst_merge_agg` function. :param tiles: A column containing an array of raster tiles. :type tiles: Column (ArrayType(RasterTileType)) diff --git a/docs/source/api/spatial-functions.rst b/docs/source/api/spatial-functions.rst index acf85943f..b629dcee8 100644 --- a/docs/source/api/spatial-functions.rst +++ b/docs/source/api/spatial-functions.rst @@ -179,7 +179,7 @@ st_bufferloop .. function:: st_bufferloop(col, innerRadius, outerRadius) - Returns a difference between st_buffer(col, outerRadius) and st_buffer(col, innerRadius). + Returns a difference between :code:`st_buffer(col, outerRadius)` and :code:`st_buffer(col, innerRadius)`. The resulting geometry is a loop with a width of outerRadius - innerRadius. :param col: Geometry @@ -930,7 +930,7 @@ st_intersection .. function:: st_intersection(geom1, geom2) Returns a geometry representing the intersection of :code:`left_geom` and :code:`right_geom`. - Also, see :doc:`st_intersection_agg ` function. + Also, see :ref:`st_intersection_agg` function. :param geom1: Geometry :type geom1: Column @@ -1411,7 +1411,7 @@ st_setsrid +---------------------------------+ .. note:: - ST_SetSRID does not transform the coordinates of :code:`geom`, + :ref:`st_setsrid` does not transform the coordinates of :code:`geom`, rather it tells Mosaic the SRID in which the current coordinates are expressed. :ref:`st_setsrid` can only operate on geometries encoded in GeoJSON. @@ -1470,8 +1470,6 @@ st_simplify | LINESTRING (0 1, 1 2, 3 0) | +----------------------------+ -.. note:: - The specified tolerance will be ignored by the ESRI geometry API. st_srid ******* @@ -1531,7 +1529,7 @@ st_srid +------------+ .. note:: - ST_SRID can only operate on geometries encoded in GeoJSON. + :ref:`st_srid` can only operate on geometries encoded in GeoJSON. st_transform @@ -1578,7 +1576,7 @@ st_transform .. code-tab:: sql - select st_astext(st_transform(st_setsrid(st_geomfromwkt("MULTIPOINT ((10 40), (40 30), (20 20), (30 10))"), 4326) as geom, 3857)) + select st_astext(st_transform(st_setsrid(st_asgeojson("MULTIPOINT ((10 40), (40 30), (20 20), (30 10))"), 4326) as geom, 3857)) +--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ |convert_to(st_transform(geom, 3857)) | +--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ @@ -1600,8 +1598,8 @@ st_transform .. note:: If :code:`geom` does not have an associated SRID, use :ref:`st_setsrid` to set this before calling :ref:`st_transform`. **Changed in 0.4 series** :ref:`st_srid`, :ref:`st_setsrid`, and :ref:`st_transform` only operate on - GeoJSON (columnar) data, so be sure to call :ref:`/api/geometry-accessors#st_asgeojson` to convert from WKT and WKB. You can convert - back after the transform, e.g. using :ref:`/api/geometry-accessors#st_astext` or :ref:`/api/geometry-accessors#st_asbinary`. + GeoJSON (columnar) data, so be sure to call :ref:`st_asgeojson` to convert from WKT and WKB. You can convert + back after the transform, e.g. using :ref:`st_astext` or :ref:`st_asbinary`. st_translate @@ -1667,7 +1665,7 @@ st_union .. function:: st_union(left_geom, right_geom) Returns the point set union of the input geometries. - Also, see :doc:`st_union_agg ` function. + Also, see :ref:`st_union_agg` function. :param left_geom: Geometry :type left_geom: Column diff --git a/docs/source/api/spatial-indexing.rst b/docs/source/api/spatial-indexing.rst index b241dade6..a35f1ae85 100644 --- a/docs/source/api/spatial-indexing.rst +++ b/docs/source/api/spatial-indexing.rst @@ -855,7 +855,7 @@ grid_cell_intersection .. function:: grid_cell_intersection(left_chip, right_chip) Returns the chip representing the intersection of two chips based on the same grid cell. - Also, see :doc:`grid_cell_intersection_agg ` function. + Also, see :ref:`grid_cell_intersection_agg` function. :param left_chip: Chip :type left_chip: Column: ChipType(LongType) @@ -911,7 +911,7 @@ grid_cell_union .. function:: grid_cell_union(left_chip, right_chip) Returns the chip representing the union of two chips based on the same grid cell. - Also, see :doc:`grid_cell_union_agg ` function. + Also, see :ref:`grid_cell_union_agg` function. :param left_chip: Chip :type left_chip: Column: ChipType(LongType) diff --git a/docs/source/api/spatial-predicates.rst b/docs/source/api/spatial-predicates.rst index df3ce4951..a002c9bd4 100644 --- a/docs/source/api/spatial-predicates.rst +++ b/docs/source/api/spatial-predicates.rst @@ -67,7 +67,7 @@ st_intersects .. function:: st_intersects(geom1, geom2) Returns true if the geometry :code:`geom1` intersects :code:`geom2`. - Also, see :doc:`st_intersects_agg ` function. + Also, see :ref:`st_intersects_agg` function. :param geom1: Geometry :type geom1: Column diff --git a/docs/source/api/vector-format-readers.rst b/docs/source/api/vector-format-readers.rst index 4c419a6b6..e540a11f2 100644 --- a/docs/source/api/vector-format-readers.rst +++ b/docs/source/api/vector-format-readers.rst @@ -61,6 +61,7 @@ The reader supports the following options: Loads a vector file and returns the result as a :class:`DataFrame`. :param path: the path of the vector file + :type path: Column(StringType) :return: :class:`DataFrame` :example: @@ -97,6 +98,11 @@ The reader supports the following options: | "description"| 3| ... | POINT (3.0 3.0) | 4326| +--------------------+-------+-----+-----------------+-----------+ +.. note:: + Keyword options not identified in function signature are converted to a :code:`Map`. + These must be supplied as a :code:`String`. + Also, you can supply function signature values as :code:`String`. + mos.read().format("multi_read_ogr") *********************************** @@ -128,6 +134,7 @@ and parsed into expected types on execution. The reader supports the following o Loads a vector file and returns the result as a :class:`DataFrame`. :param path: the path of the vector file + :type path: Column(StringType) :return: :class:`DataFrame` :example: @@ -164,6 +171,9 @@ and parsed into expected types on execution. The reader supports the following o | "description"| 3| ... | POINT (3.0 3.0) | 4326| +--------------------+-------+-----+-----------------+-----------+ +.. note:: + All options are converted to a :code:`Map` and must be supplied as a :code:`String`. + spark.read.format("geo_db") ***************************** @@ -182,6 +192,7 @@ The reader supports the following options: Loads a GeoDB file and returns the result as a :class:`DataFrame`. :param path: the path of the GeoDB file + :type path: Column(StringType) :return: :class:`DataFrame` :example: @@ -217,6 +228,11 @@ The reader supports the following options: | "description"| 3| ... | POINT (3.0 3.0) | 4326| +--------------------+-------+-----+-----------------+-----------+ +.. note:: + Keyword options not identified in function signature are converted to a :code:`Map`. + These must be supplied as a :code:`String`. + Also, you can supply function signature values as :code:`String`. + spark.read.format("shapefile") ******************************** @@ -235,6 +251,7 @@ The reader supports the following options: Loads a Shapefile and returns the result as a :class:`DataFrame`. :param path: the path of the Shapefile + :type path: Column(StringType) :return: :class:`DataFrame` :example: @@ -268,4 +285,9 @@ The reader supports the following options: | "description"| 1| ... | POINT (1.0 1.0) | 4326| | "description"| 2| ... | POINT (2.0 2.0) | 4326| | "description"| 3| ... | POINT (3.0 3.0) | 4326| - +--------------------+-------+-----+-----------------+-----------+ \ No newline at end of file + +--------------------+-------+-----+-----------------+-----------+ + +.. note:: + Keyword options not identified in function signature are converted to a :code:`Map`. + These must be supplied as a :code:`String`. + Also, you can supply function signature values as :code:`String`. diff --git a/docs/source/usage/automatic-sql-registration.rst b/docs/source/usage/automatic-sql-registration.rst index 9c2d0fe64..83af812c0 100644 --- a/docs/source/usage/automatic-sql-registration.rst +++ b/docs/source/usage/automatic-sql-registration.rst @@ -32,7 +32,8 @@ Installation To install Mosaic on your Databricks cluster, take the following steps: -#. Upload Mosaic jar to a dedicated fuse mount location. E.g. dbfs:/FileStore/mosaic/jars/. +#. Upload Mosaic jar to a dedicated fuse mount location. E.g. "dbfs:/FileStore/mosaic/jars/". + #. Create an init script that fetches the mosaic jar and copies it to databricks/jars. You can also use the output from (0.4 series) python function :code:`setup_fuse_install`, e.g. :code:`setup_fuse_intall(, jar_copy=True)` which can help to copy the JAR used in @@ -56,8 +57,9 @@ To install Mosaic on your Databricks cluster, take the following steps: EOF -#. Configure the init script for the cluster following the instructions `here `__. -#. Add the following spark configuration values for your cluster following the instructions `here `__. +#. Configure the init script for the cluster following the instructions `here `_. + +#. Add the following spark configuration values for your cluster following the instructions `here `_. .. code-block:: bash diff --git a/docs/source/usage/install-gdal.rst b/docs/source/usage/install-gdal.rst index c4bdf02f3..4478dee05 100644 --- a/docs/source/usage/install-gdal.rst +++ b/docs/source/usage/install-gdal.rst @@ -29,9 +29,9 @@ Mosaic requires GDAL to be installed on the cluster. The easiest way to do this the :code:`setup_gdal` function. .. note:: - (a) This is close in behavior to Mosaic < 0.4 series (prior to DBR 13), with new options + - This is close in behavior to Mosaic < 0.4 series (prior to DBR 13), with new options to pip install Mosaic for either ubuntugis gdal (3.4.3) or jammy default (3.4.1). - (b) 'to_fuse_dir' can be one of '/Volumes/..', '/Workspace/..', '/dbfs/..'; + - Param "to_fuse_dir" can be one of "/Volumes/..", "/Workspace/..", "/dbfs/.."; however, you should consider :code:`setup_fuse_install()` for Volume based installs as that exposes more options, to include copying JAR and JNI Shared Objects. From 795014653f53e19076ffce0780762a6138189637 Mon Sep 17 00:00:00 2001 From: Michael Johns Date: Thu, 25 Jan 2024 14:15:12 -0500 Subject: [PATCH 2/5] removing warning boxes for "Deprecation ERRORs" in the docs. --- docs/source/index.rst | 13 ++++++------- docs/source/usage/installation.rst | 11 ++++------- 2 files changed, 10 insertions(+), 14 deletions(-) diff --git a/docs/source/index.rst b/docs/source/index.rst index d7fb0ca44..73ea2197a 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -61,16 +61,14 @@ Version 0.4.x Series We recommend using Databricks Runtime versions 13.3 LTS with Photon enabled. -.. warning:: - Mosaic 0.4.x series only supports DBR 13.x DBRs. - If running on a different DBR it will throw an exception: - **DEPRECATION ERROR: Mosaic v0.4.x series only supports Databricks Runtime 13. You can specify `%pip install 'databricks-mosaic<0.4,>=0.3'` for DBR < 13.** +Mosaic 0.4.x series only supports DBR 13.x DBRs. If running on a different DBR it will throw an exception: -.. warning:: - Mosaic 0.4.x series issues the following ERROR on a standard, non-Photon cluster `ADB `_ | `AWS `_ | `GCP `_ : +**DEPRECATION ERROR: Mosaic v0.4.x series only supports Databricks Runtime 13. You can specify `%pip install 'databricks-mosaic<0.4,>=0.3'` for DBR < 13.** - **DEPRECATION ERROR: Please use a Databricks Photon-enabled Runtime for performance benefits or Runtime ML for spatial AI benefits; Mosaic 0.4.x series restricts executing this cluster.** +Mosaic 0.4.x series issues the following ERROR on a standard, non-Photon cluster `ADB `_ | `AWS `_ | `GCP `_ : + +**DEPRECATION ERROR: Please use a Databricks Photon-enabled Runtime for performance benefits or Runtime ML for spatial AI benefits; Mosaic 0.4.x series restricts executing this cluster.** As of Mosaic 0.4.0 (subject to change in follow-on releases) * No Mosaic SQL expressions cannot yet be registered with `Unity Catalog `_ due to API changes affecting DBRs >= 13. @@ -96,6 +94,7 @@ For Mosaic versions < 0.4.0 please use the `0.3.x docs `_ | `AWS `_ | `GCP `_ : **DEPRECATION WARNING: Please use a Databricks Photon-enabled Runtime for performance benefits or Runtime ML for spatial AI benefits; Mosaic will stop working on this cluster after v0.3.x.** + If you are receiving this warning in v0.3.11+, you will want to begin to plan for a supported runtime. The reason we are making this change is that we are streamlining Mosaic internals to be more aligned with future product APIs which are powered by Photon. Along this direction of change, Mosaic has standardized to JTS as its default and supported Vector Geometry Provider. diff --git a/docs/source/usage/installation.rst b/docs/source/usage/installation.rst index 61d0a6114..40c56e5c1 100644 --- a/docs/source/usage/installation.rst +++ b/docs/source/usage/installation.rst @@ -8,16 +8,13 @@ Supported platforms .. note:: For Mosaic 0.4 series, we recommend DBR 13.3 LTS on Photon or ML Runtime clusters. -.. warning:: - Mosaic 0.4.x series only supports DBR 13.x DBRs. - If running on a different DBR it will throw an exception: +Mosaic 0.4.x series only supports DBR 13.x DBRs. If running on a different DBR it will throw an exception: - **DEPRECATION ERROR: Mosaic v0.4.x series only supports Databricks Runtime 13. You can specify `%pip install 'databricks-mosaic<0.4,>=0.3'` for DBR < 13.** +**DEPRECATION ERROR: Mosaic v0.4.x series only supports Databricks Runtime 13. You can specify `%pip install 'databricks-mosaic<0.4,>=0.3'` for DBR < 13.** -.. warning:: - Mosaic 0.4.x series issues the following ERROR on a standard, non-Photon cluster `ADB `_ | `AWS `_ | `GCP `_ : +Mosaic 0.4.x series issues the following ERROR on a standard, non-Photon cluster `ADB `_ | `AWS `_ | `GCP `_ : - **DEPRECATION ERROR: Please use a Databricks Photon-enabled Runtime for performance benefits or Runtime ML for spatial AI benefits; Mosaic 0.4.x series restricts executing this cluster.** +**DEPRECATION ERROR: Please use a Databricks Photon-enabled Runtime for performance benefits or Runtime ML for spatial AI benefits; Mosaic 0.4.x series restricts executing this cluster.** As of Mosaic 0.4.0 (subject to change in follow-on releases) * No Mosaic SQL expressions cannot yet be registered with `Unity Catalog `_ due to API changes affecting DBRs >= 13. From fa6596416bd978aa262ed3f022763826d5ff786b Mon Sep 17 00:00:00 2001 From: Michael Johns Date: Thu, 25 Jan 2024 16:25:40 -0500 Subject: [PATCH 3/5] Documenting assigned cluster SQL registration working on DBR 13.3. --- README.md | 6 +++--- docs/source/index.rst | 7 +++++-- docs/source/usage/automatic-sql-registration.rst | 10 +++++++--- docs/source/usage/install-gdal.rst | 14 +++++++------- docs/source/usage/installation.rst | 9 ++++++--- 5 files changed, 28 insertions(+), 18 deletions(-) diff --git a/README.md b/README.md index b1a052839..11a2918d6 100644 --- a/README.md +++ b/README.md @@ -57,9 +57,9 @@ __Language Bindings__ As of Mosaic 0.4.0 (subject to change in follow-on releases)... -* _Mosaic SQL expressions cannot yet be registered with [Unity Catalog](https://www.databricks.com/product/unity-catalog) due to API changes affecting DBRs >= 13._ -* [Assigned Clusters](https://docs.databricks.com/en/compute/configure.html#access-modes): Mosaic Python, R, and Scala APIs. +* [Assigned Clusters](https://docs.databricks.com/en/compute/configure.html#access-modes): Mosaic Python, SQL, R, and Scala APIs. * [Shared Access Clusters](https://docs.databricks.com/en/compute/configure.html#access-modes): Mosaic Scala API (JVM) with Admin [allowlisting](https://docs.databricks.com/en/data-governance/unity-catalog/manage-privileges/allowlist.html); _Python bindings to Mosaic Scala APIs are blocked by Py4J Security on Shared Access Clusters._ + * _Mosaic SQL expressions cannot yet be registered with [Unity Catalog](https://www.databricks.com/product/unity-catalog) due to API changes affecting DBRs >= 13, more [here](https://docs.databricks.com/en/udf/index.html)._ __Additional Notes:__ @@ -141,7 +141,7 @@ import com.databricks.labs.mosaic.JTS val mosaicContext = MosaicContext.build(H3, JTS) mosaicContext.register(spark) ``` -__Note: Mosaic 0.4.x SQL bindings for DBR 13 not yet available in Unity Catalog due to API changes.__ +__Note: Mosaic 0.4.x SQL bindings for DBR 13 can register with Assigned clusters, but not Shared Access due to API changes, more [here](https://docs.databricks.com/en/udf/index.html).__ ## Examples diff --git a/docs/source/index.rst b/docs/source/index.rst index 73ea2197a..d59be7383 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -45,6 +45,7 @@ We currently recommend using Databricks Runtime with Photon enabled; this will leverage the Databricks H3 expressions when using H3 grid system. Mosaic provides: + * easy conversion between common spatial data encodings (WKT, WKB and GeoJSON); * constructors to easily generate new geometries from Spark native data types; * many of the OGC SQL standard :code:`ST_` functions implemented as Spark Expressions for transforming, aggregating and joining spatial datasets; @@ -71,9 +72,11 @@ Mosaic 0.4.x series issues the following ERROR on a standard, non-Photon cluster **DEPRECATION ERROR: Please use a Databricks Photon-enabled Runtime for performance benefits or Runtime ML for spatial AI benefits; Mosaic 0.4.x series restricts executing this cluster.** As of Mosaic 0.4.0 (subject to change in follow-on releases) - * No Mosaic SQL expressions cannot yet be registered with `Unity Catalog `_ due to API changes affecting DBRs >= 13. - * `Assigned Clusters `_ : Mosaic Python, R, and Scala APIs. + + * `Assigned Clusters `_ : Mosaic Python, SQL, R, and Scala APIs. * `Shared Access Clusters `_ : Mosaic Scala API (JVM) with Admin `allowlisting `_ ; Python bindings to Mosaic Scala APIs are blocked by Py4J Security on Shared Access Clusters. + - Mosaic SQL expressions cannot yet be registered with `Unity Catalog `_ + due to API changes affecting DBRs >= 13, more `here `_. .. note:: As of Mosaic 0.4.0 (subject to change in follow-on releases) diff --git a/docs/source/usage/automatic-sql-registration.rst b/docs/source/usage/automatic-sql-registration.rst index 83af812c0..c1403cec0 100644 --- a/docs/source/usage/automatic-sql-registration.rst +++ b/docs/source/usage/automatic-sql-registration.rst @@ -19,13 +19,13 @@ Pre-requisites In order to use Mosaic, you must have access to a Databricks cluster running Databricks Runtime 13. If you have cluster creation permissions in your Databricks workspace, you can create a cluster using the instructions -`here `__. +`here `_. You will also need "Can Manage" permissions on this cluster in order to attach init script to your cluster. A workspace administrator will be able to grant these permissions and more information about cluster permissions can be found in our documentation -`here `__. +`here `_. Installation ************ @@ -82,7 +82,11 @@ To test the installation, create a new Python notebook and run the following com You should see all the supported functions registered by Mosaic appear in the output. .. note:: - You may see some :code:`ST_` functions from other libraries, so pay close attention to the provider. + You may see some :code:`ST_` functions from other libraries, so pay close attention to the provider. + +.. warning:: + Mosaic 0.4.x SQL bindings for DBR 13 can register with Assigned clusters, but not Shared Access due to API changes, + more `here `_. .. warning:: Issue 317: https://github.com/databrickslabs/mosaic/issues/317 diff --git a/docs/source/usage/install-gdal.rst b/docs/source/usage/install-gdal.rst index 4478dee05..d20cf7adc 100644 --- a/docs/source/usage/install-gdal.rst +++ b/docs/source/usage/install-gdal.rst @@ -8,17 +8,17 @@ In order to use Mosaic 0.4 series, you must have access to a Databricks cluster Databricks Runtime 13.3 LTS. If you have cluster creation permissions in your Databricks workspace, you can create a cluster using the instructions -`here `__. +`here `_. You will also need "Can Manage" permissions on this cluster in order to attach the Mosaic library to your cluster. A workspace administrator will be able to grant these permissions and more information about cluster permissions can be found in our documentation -`here `__. +`here `_. .. warning:: These instructions assume an Assigned cluster is being used (vs a Shared Access cluster), - more on access modes `here `__. + more on access modes `here `_. GDAL Installation #################### @@ -30,10 +30,10 @@ the :code:`setup_gdal` function. .. note:: - This is close in behavior to Mosaic < 0.4 series (prior to DBR 13), with new options - to pip install Mosaic for either ubuntugis gdal (3.4.3) or jammy default (3.4.1). + to pip install Mosaic for either ubuntugis gdal (3.4.3) or jammy default (3.4.1). - Param "to_fuse_dir" can be one of "/Volumes/..", "/Workspace/..", "/dbfs/.."; - however, you should consider :code:`setup_fuse_install()` for Volume based installs as that - exposes more options, to include copying JAR and JNI Shared Objects. + however, you should consider :code:`setup_fuse_install()` for Volume based installs as that + exposes more options, to include copying JAR and JNI Shared Objects. .. function:: setup_gdal() @@ -107,5 +107,5 @@ code at the top of the notebook: GDAL 3.4.1, released 2021/12/27 .. note:: - You can configure init script from default ubuntu GDAL (3.4.1) to `ubuntugis ppa `__ (3.4.3) + You can configure init script from default ubuntu GDAL (3.4.1) to `ubuntugis ppa `_ (3.4.3) with :code:`setup_gdal(with_ubuntugis=True)` \ No newline at end of file diff --git a/docs/source/usage/installation.rst b/docs/source/usage/installation.rst index 40c56e5c1..7df5bc40c 100644 --- a/docs/source/usage/installation.rst +++ b/docs/source/usage/installation.rst @@ -17,9 +17,11 @@ Mosaic 0.4.x series issues the following ERROR on a standard, non-Photon cluster **DEPRECATION ERROR: Please use a Databricks Photon-enabled Runtime for performance benefits or Runtime ML for spatial AI benefits; Mosaic 0.4.x series restricts executing this cluster.** As of Mosaic 0.4.0 (subject to change in follow-on releases) - * No Mosaic SQL expressions cannot yet be registered with `Unity Catalog `_ due to API changes affecting DBRs >= 13. - * `Assigned Clusters `_ : Mosaic Python, R, and Scala APIs. + + * `Assigned Clusters `_ : Mosaic Python, SQL, R, and Scala APIs. * `Shared Access Clusters `_ : Mosaic Scala API (JVM) with Admin `allowlisting `_ ; Python bindings to Mosaic Scala APIs are blocked by Py4J Security on Shared Access Clusters. + - Mosaic SQL expressions cannot yet be registered with `Unity Catalog `_ + due to API changes affecting DBRs >= 13, more `here `_. .. note:: As of Mosaic 0.4.0 (subject to change in follow-on releases) @@ -119,4 +121,5 @@ register the Mosaic SQL functions in your SparkSession from a Scala notebook cel mosaicContext.register(spark) .. warning:: - Mosaic 0.4.x SQL bindings for DBR 13 not yet available in Unity Catalog due to API changes. \ No newline at end of file + Mosaic 0.4.x SQL bindings for DBR 13 can register with Assigned clusters, but not Shared Access due to API changes, + more `here `_. From 094c83b37f3e0376ff50b9b84e41766b42d5f136 Mon Sep 17 00:00:00 2001 From: Michael Johns Date: Thu, 25 Jan 2024 17:23:36 -0500 Subject: [PATCH 4/5] hard refresh tip. --- docs/source/index.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/source/index.rst b/docs/source/index.rst index d59be7383..44e8e0665 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -56,6 +56,9 @@ Mosaic provides: .. note:: For Mosaic versions < 0.4 please use the `0.3 docs `_. +.. warning:: + At times, it is useful to "hard refresh" pages to ensure your cached local version matches the latest live, + more `here `_. Version 0.4.x Series ==================== From 827f3a832508525ac78c5a0e6439af3fdcb5e93c Mon Sep 17 00:00:00 2001 From: Michael Johns Date: Thu, 25 Jan 2024 17:26:14 -0500 Subject: [PATCH 5/5] formatting. --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 11a2918d6..509e5b19b 100644 --- a/README.md +++ b/README.md @@ -59,7 +59,7 @@ As of Mosaic 0.4.0 (subject to change in follow-on releases)... * [Assigned Clusters](https://docs.databricks.com/en/compute/configure.html#access-modes): Mosaic Python, SQL, R, and Scala APIs. * [Shared Access Clusters](https://docs.databricks.com/en/compute/configure.html#access-modes): Mosaic Scala API (JVM) with Admin [allowlisting](https://docs.databricks.com/en/data-governance/unity-catalog/manage-privileges/allowlist.html); _Python bindings to Mosaic Scala APIs are blocked by Py4J Security on Shared Access Clusters._ - * _Mosaic SQL expressions cannot yet be registered with [Unity Catalog](https://www.databricks.com/product/unity-catalog) due to API changes affecting DBRs >= 13, more [here](https://docs.databricks.com/en/udf/index.html)._ + * Mosaic SQL expressions cannot yet be registered with [Unity Catalog](https://www.databricks.com/product/unity-catalog) due to API changes affecting DBRs >= 13, more [here](https://docs.databricks.com/en/udf/index.html). __Additional Notes:__