From 3b4a52740e050543596afbdb64d762255241997e Mon Sep 17 00:00:00 2001 From: Michael Johns Date: Fri, 15 Sep 2023 17:21:43 -0400 Subject: [PATCH 01/12] Update spatial-functions.rst st_haversine is in km (vs km^2); also, reference the value of the radius used (which is in km). --- docs/source/api/spatial-functions.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/api/spatial-functions.rst b/docs/source/api/spatial-functions.rst index c4ace6c19..ab698b5c3 100644 --- a/docs/source/api/spatial-functions.rst +++ b/docs/source/api/spatial-functions.rst @@ -744,7 +744,7 @@ st_haversine | 10007.55722101796| +------------------------------------+ -.. note:: Results of this function are always expressed in km^2, while the input lat/lng pairs are expected to be in degrees. +.. note:: Results of this function are always expressed in km, while the input lat/lng pairs are expected to be in degrees. The radius used (in km) is 6371.0088. st_hasvalidcoordinates From e580ccb25f296b06da8b614e42406de783e81dd8 Mon Sep 17 00:00:00 2001 From: Michael Johns Date: Fri, 15 Sep 2023 17:28:54 -0400 Subject: [PATCH 02/12] Update index.rst remove references to Mosaic 0.4 --- docs/source/index.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/source/index.rst b/docs/source/index.rst index fe1ec92c9..ee499822e 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -42,14 +42,14 @@ Mosaic is an extension to the `Apache Spark `_ framework that allows easy and fast processing of very large geospatial datasets. .. warning:: - From version 0.4.0, Mosaic will require either + From versions after 0.3.x, Mosaic will require either * Databricks Runtime 11.2+ with Photon enabled * Databricks Runtime for ML 11.2+ - Mosaic 0.3 series does not support DBR 13 (coming soon with Mosaic 0.4 series); + Mosaic 0.3 series does not yet support DBR 13 (coming soon); also, DBR 10 is no longer supported in Mosaic. -We recommend using Databricks Runtime versions 11.3 LTS or 12.2 LTS with Photon enabled; +We currently recommend using Databricks Runtime versions 11.3 LTS or 12.2 LTS with Photon enabled; this will leverage the Databricks H3 expressions when using H3 grid system. Mosaic provides: From 08da4a7d429a16cacb5b9d9d0f9fae6226966791 Mon Sep 17 00:00:00 2001 From: Michael Johns Date: Fri, 15 Sep 2023 17:32:57 -0400 Subject: [PATCH 03/12] Update README.md Updated to remove some of the earlier plans for databricks-mosaic-gdal PyPI project. --- modules/python/gdal_package/README.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/modules/python/gdal_package/README.md b/modules/python/gdal_package/README.md index a8aa68ef0..165af7dcf 100644 --- a/modules/python/gdal_package/README.md +++ b/modules/python/gdal_package/README.md @@ -2,7 +2,7 @@ > Current version is 3.4.3 (to match GDAL). -This is a filetree (vs apt based) drop-in packaging of GDAL with Java Bindings for Ubuntu 20.04 (Focal Fossa) which is used by [Databricks Runtime](https://docs.databricks.com/release-notes/runtime/releases.html) (DBR) 11+. +This is a filetree (vs apt based) drop-in packaging of GDAL with Java Bindings for Ubuntu 20.04 (Focal Fossa) which is used by [Databricks Runtime](https://docs.databricks.com/release-notes/runtime/releases.html) (DBR) 11 and 12 (not DBR 13 which is Ubuntu 22.04). 1. `gdal-3.4.3-filetree.tar.xz` is ~50MB - it is extracted with `tar -xf gdal-3.4.3-filetree.tar.xz -C /` 2. `gdal-3.4.3.-symlinks.tar.xz` is ~19MB - it is extracted with `tar -xhf gdal-3.4.3-symlinks.tar.xz -C /` @@ -14,4 +14,3 @@ An [init script](https://docs.databricks.com/clusters/init-scripts.html) is prov * This is a very specific packaging for GDAL + dependencies which removes any libraries that are already provided by DBR, so it will not be not useful outside Databricks. * It additionally includes GDAL shared objects (`.so`) for Java Bindings, GDAL 3.4.3 Python bindings, and tweak for OSGEO as currently supplied by [UbuntuGIS PPA](https://launchpad.net/~ubuntugis/+archive/ubuntu/ubuntugis-unstable) based init script [install-gdal-databricks.sh](https://github.com/databrickslabs/mosaic/blob/main/src/main/resources/scripts/install-gdal-databricks.sh) provided by Mosaic. This install replaces the existing way on Mosaic, so choose one or the other. * The GDAL JAR for 3.4 is not included but is provided by Mosaic itself and added to your Databricks cluster as part of the [enable_gdal](https://databrickslabs.github.io/mosaic/usage/install-gdal.html#enable-gdal-for-a-notebook) called when configuring Mosaic for GDAL. Separately, the JAR could be added as a [cluster-installed library](https://docs.databricks.com/libraries/cluster-libraries.html#cluster-installed-library), e.g. through Maven coordinates `org.gdal:gdal:3.4.0` from [mvnrepository](https://mvnrepository.com/artifact/org.gdal/gdal/3.4.0). -* Mosaic will soon be able to directly leverage this [PyPI](https://pypi.org/project/databricks-mosaic-gdal/) project and be able to altogether avoid the init script as a precursor to calling [enable_gdal](https://databrickslabs.github.io/mosaic/usage/install-gdal.html#enable-gdal-for-a-notebook). So check Mosaic [GDAL Installation Guide](https://databrickslabs.github.io/mosaic/usage/install-gdal.html#) for any changes on/around APR 2023. \ No newline at end of file From 97016e5b8e7445da649783b41c17ab297b26eaa6 Mon Sep 17 00:00:00 2001 From: Michael Johns Date: Fri, 15 Sep 2023 17:35:17 -0400 Subject: [PATCH 04/12] Update README.md Updating main README since we are no longer releasing Mosaic 0.4 --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 052ebeb38..74ff74c9c 100644 --- a/README.md +++ b/README.md @@ -45,13 +45,13 @@ Image1: Mosaic logical design. We recommend using Databricks Runtime versions 11.3 LTS or 12.2 LTS with Photon enabled; this will leverage the Databricks H3 expressions when using H3 grid system. -:warning: **Mosaic 0.3 series does not support DBR 13** (coming soon with Mosaic 0.4 series); also, DBR 10 is no longer supported in Mosaic. +:warning: **Mosaic 0.3 series does not support DBR 13** (coming soon); also, DBR 10 is no longer supported in Mosaic. As of the 0.3.11 release, Mosaic issues the following warning when initialized on a cluster that is neither Photon Runtime nor Databricks Runtime ML [[ADB](https://learn.microsoft.com/en-us/azure/databricks/runtime/) | [AWS](https://docs.databricks.com/runtime/index.html) | [GCP](https://docs.gcp.databricks.com/runtime/index.html)]: -> DEPRECATION WARNING: Mosaic is not supported on the selected Databricks Runtime. Mosaic will stop working on this cluster from version v0.4.0+. Please use a Databricks Photon-enabled Runtime (for performance benefits) or Runtime ML (for spatial AI benefits). +> DEPRECATION WARNING: Mosaic is not supported on the selected Databricks Runtime. Mosaic will stop working on this cluster after v0.3.x. Please use a Databricks Photon-enabled Runtime (for performance benefits) or Runtime ML (for spatial AI benefits). -If you are receiving this warning in v0.3.11, you will want to change to a supported runtime prior to updating Mosaic to run 0.4.0. The reason we are making this change is that we are streamlining Mosaic internals to be more aligned with future product APIs which are powered by Photon. Along this direction of change, Mosaic will be standardizing to JTS as its default and supported Vector Geometry Provider. +If you are receiving this warning in v0.3.11, you will want to change to being to plan for a supported runtime. The reason we are making this change is that we are streamlining Mosaic internals to be more aligned with future product APIs which are powered by Photon. Along this direction of change, Mosaic will be standardizing to JTS as its default and supported Vector Geometry Provider. ### Documentation From ab941e4edabecc700e840b7f251dd684f0121d15 Mon Sep 17 00:00:00 2001 From: Michael Johns Date: Fri, 15 Sep 2023 17:38:42 -0400 Subject: [PATCH 05/12] Update installation.rst Since we are not releasing Mosaic 0.4, updating instructions --- docs/source/usage/installation.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/source/usage/installation.rst b/docs/source/usage/installation.rst index a4b97952b..86974a136 100644 --- a/docs/source/usage/installation.rst +++ b/docs/source/usage/installation.rst @@ -6,11 +6,11 @@ Supported platforms ################### .. warning:: - From version 0.4.0, Mosaic will require either + From versions after 0.3.x, Mosaic will require either * Databricks Runtime 11.2+ with Photon enabled * Databricks Runtime for ML 11.2+ - Mosaic 0.3 series does not support DBR 13 (coming soon with Mosaic 0.4 series); + Mosaic 0.3 series does not support DBR 13 (coming soon); also, DBR 10 is no longer supported in Mosaic. We recommend using Databricks Runtime versions 11.3 LTS or 12.2 LTS with Photon enabled; @@ -18,10 +18,10 @@ this will leverage the Databricks H3 expressions when using H3 grid system. As of the 0.3.11 release, Mosaic issues the following warning when initialized on a cluster that is neither Photon Runtime nor Databricks Runtime ML [`ADB `__ | `AWS `__ | `GCP `__]: - DEPRECATION WARNING: Mosaic is not supported on the selected Databricks Runtime. Mosaic will stop working on this cluster from version v0.4.0+. Please use a Databricks Photon-enabled Runtime (for performance benefits) or Runtime ML (for spatial AI benefits). + DEPRECATION WARNING: Mosaic is not supported on the selected Databricks Runtime. Mosaic will stop working on this cluster after v0.3.x. Please use a Databricks Photon-enabled Runtime (for performance benefits) or Runtime ML (for spatial AI benefits). -If you are receiving this warning in v0.3.11+, you will want to change to a supported runtime prior -to updating Mosaic to run 0.4.0. The reason we are making this change is that we are streamlining Mosaic +If you are receiving this warning in v0.3.11+, you will want to change to being to plan for a supported runtime. +The reason we are making this change is that we are streamlining Mosaic internals to be more aligned with future product APIs which are powered by Photon. Along this direction of change, Mosaic will be standardizing to JTS as its default and supported Vector Geometry Provider. From b8e42ca302385be1e22cc7ba0a4c30e8d77d8da8 Mon Sep 17 00:00:00 2001 From: Michael Johns Date: Fri, 15 Sep 2023 17:40:17 -0400 Subject: [PATCH 06/12] Update README.md additional change --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 74ff74c9c..2fd96eed5 100644 --- a/README.md +++ b/README.md @@ -51,7 +51,7 @@ As of the 0.3.11 release, Mosaic issues the following warning when initialized o > DEPRECATION WARNING: Mosaic is not supported on the selected Databricks Runtime. Mosaic will stop working on this cluster after v0.3.x. Please use a Databricks Photon-enabled Runtime (for performance benefits) or Runtime ML (for spatial AI benefits). -If you are receiving this warning in v0.3.11, you will want to change to being to plan for a supported runtime. The reason we are making this change is that we are streamlining Mosaic internals to be more aligned with future product APIs which are powered by Photon. Along this direction of change, Mosaic will be standardizing to JTS as its default and supported Vector Geometry Provider. +If you are receiving this warning in v0.3.11+, you will want to change to being to plan for a supported runtime. The reason we are making this change is that we are streamlining Mosaic internals to be more aligned with future product APIs which are powered by Photon. Along this direction of change, Mosaic will be standardizing to JTS as its default and supported Vector Geometry Provider. ### Documentation From 380c74419847360c04f78c0e2f48f1e609e74b0c Mon Sep 17 00:00:00 2001 From: Michael Johns Date: Fri, 15 Sep 2023 17:42:30 -0400 Subject: [PATCH 07/12] Update MosaicContext.scala Since we are not releasing Mosaic 0.4, updating logging. --- .../com/databricks/labs/mosaic/functions/MosaicContext.scala | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/main/scala/com/databricks/labs/mosaic/functions/MosaicContext.scala b/src/main/scala/com/databricks/labs/mosaic/functions/MosaicContext.scala index 5777cef8a..8f9bf92d7 100644 --- a/src/main/scala/com/databricks/labs/mosaic/functions/MosaicContext.scala +++ b/src/main/scala/com/databricks/labs/mosaic/functions/MosaicContext.scala @@ -944,10 +944,10 @@ object MosaicContext extends Logging { if (!isML && !isPhoton) { // Print out the warnings both to the log and to the console logWarning("DEPRECATION WARNING: Mosaic is not supported on the selected Databricks Runtime") - logWarning("DEPRECATION WARNING: Mosaic will stop working on this cluster from version v0.4.0+.") + logWarning("DEPRECATION WARNING: Mosaic will stop working on this cluster after v0.3.x.") logWarning("Please use a Databricks Photon-enabled Runtime (for performance benefits) or Runtime ML (for spatial AI benefits).") println("DEPRECATION WARNING: Mosaic is not supported on the selected Databricks Runtime") - println("DEPRECATION WARNING: Mosaic will stop working on this cluster from version v0.4.0+.") + println("DEPRECATION WARNING: Mosaic will stop working on this cluster after v0.3.x.") println("Please use a Databricks Photon-enabled Runtime (for performance benefits) or Runtime ML (for spatial AI benefits).") false } else { From 45414adc51c905f4b2bfc232feb39ede57067820 Mon Sep 17 00:00:00 2001 From: Michael Johns Date: Fri, 15 Sep 2023 18:21:54 -0400 Subject: [PATCH 08/12] Update mosaic-gdal-3.4.3-filetree-init.sh removed old comment --- .../resources/scripts/mosaic-gdal-3.4.3-filetree-init.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/python/gdal_package/databricks-mosaic-gdal/resources/scripts/mosaic-gdal-3.4.3-filetree-init.sh b/modules/python/gdal_package/databricks-mosaic-gdal/resources/scripts/mosaic-gdal-3.4.3-filetree-init.sh index 56f8bdc11..9b900479a 100644 --- a/modules/python/gdal_package/databricks-mosaic-gdal/resources/scripts/mosaic-gdal-3.4.3-filetree-init.sh +++ b/modules/python/gdal_package/databricks-mosaic-gdal/resources/scripts/mosaic-gdal-3.4.3-filetree-init.sh @@ -11,7 +11,7 @@ # 2. This will unpack files directly into the filetree across cluster nodes (vs run apt install) # # -- install databricks-mosaic-gdal on cluster -# - from pypi.org (once available) +# - use version 3.4.3 (exactly) from pypi.org pip install databricks-mosaic-gdal==3.4.3 # -- find the install dir From d3b0ea3090badccc6989975c882586a097b86819 Mon Sep 17 00:00:00 2001 From: Michael Johns Date: Fri, 15 Sep 2023 18:26:55 -0400 Subject: [PATCH 09/12] Update spatial-functions.rst specified WGS84 units are in degrees --- docs/source/api/spatial-functions.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/api/spatial-functions.rst b/docs/source/api/spatial-functions.rst index ab698b5c3..846300274 100644 --- a/docs/source/api/spatial-functions.rst +++ b/docs/source/api/spatial-functions.rst @@ -509,7 +509,7 @@ st_distance | 15.652475842498529| +------------------------+ -.. note:: Results of this function are always expressed in the original units of the input geometries. +.. note:: Results of this function are always expressed in the original units of the input geometries, e.g. for WGS84 (SRID 4326) units are degrees. st_dump ******* From 0eacf65e5cd0b5c48d73ee58210ab1eecda6ca1e Mon Sep 17 00:00:00 2001 From: Michael Johns Date: Fri, 15 Sep 2023 18:31:00 -0400 Subject: [PATCH 10/12] Update spatial-functions.rst specified st_distance is euclidean. --- docs/source/api/spatial-functions.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/api/spatial-functions.rst b/docs/source/api/spatial-functions.rst index 846300274..09c758292 100644 --- a/docs/source/api/spatial-functions.rst +++ b/docs/source/api/spatial-functions.rst @@ -459,7 +459,7 @@ st_distance .. function:: st_distance(geom1, geom2) - Compute the distance between `geom1` and `geom2`. + Compute the euclidean distance between `geom1` and `geom2`. :param geom1: Geometry :type geom1: Column @@ -509,7 +509,7 @@ st_distance | 15.652475842498529| +------------------------+ -.. note:: Results of this function are always expressed in the original units of the input geometries, e.g. for WGS84 (SRID 4326) units are degrees. +.. note:: Results of this euclidean distance function are always expressed in the original units of the input geometries, e.g. for WGS84 (SRID 4326) units are degrees. st_dump ******* From b4c035a43dff8ff7ed934b50b7670ecd0857708a Mon Sep 17 00:00:00 2001 From: Michael Johns Date: Fri, 15 Sep 2023 18:40:00 -0400 Subject: [PATCH 11/12] Update README.md fix grammer --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 2fd96eed5..e26d88267 100644 --- a/README.md +++ b/README.md @@ -51,7 +51,7 @@ As of the 0.3.11 release, Mosaic issues the following warning when initialized o > DEPRECATION WARNING: Mosaic is not supported on the selected Databricks Runtime. Mosaic will stop working on this cluster after v0.3.x. Please use a Databricks Photon-enabled Runtime (for performance benefits) or Runtime ML (for spatial AI benefits). -If you are receiving this warning in v0.3.11+, you will want to change to being to plan for a supported runtime. The reason we are making this change is that we are streamlining Mosaic internals to be more aligned with future product APIs which are powered by Photon. Along this direction of change, Mosaic will be standardizing to JTS as its default and supported Vector Geometry Provider. +If you are receiving this warning in v0.3.11+, you will want to begin to plan for a supported runtime. The reason we are making this change is that we are streamlining Mosaic internals to be more aligned with future product APIs which are powered by Photon. Along this direction of change, Mosaic will be standardizing to JTS as its default and supported Vector Geometry Provider. ### Documentation From de04a1312b7ba202073fbe8c62da5f6e037c8873 Mon Sep 17 00:00:00 2001 From: Michael Johns Date: Fri, 15 Sep 2023 18:43:19 -0400 Subject: [PATCH 12/12] Update installation.rst fixed grammatical issue --- docs/source/usage/installation.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/usage/installation.rst b/docs/source/usage/installation.rst index 86974a136..11263e11a 100644 --- a/docs/source/usage/installation.rst +++ b/docs/source/usage/installation.rst @@ -20,7 +20,7 @@ that is neither Photon Runtime nor Databricks Runtime ML [`ADB