Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI/CD pipeline cleanup #26

Merged
merged 44 commits into from
Nov 25, 2024
Merged
Changes from 1 commit
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
f62407b
Fix/quickstart-MosaicAnalyzer (#565)
sllynn Sep 24, 2024
ef4d1e6
Bump org.apache.maven.plugins:maven-assembly-plugin from 3.7.0 to 3.7…
dependabot[bot] Sep 24, 2024
bef8c1b
fix: typo in lettermap SZ->HZ (#433)
tompeterken-os Sep 24, 2024
ea37dda
RST_Type RST_UpdateType (#564)
sllynn Sep 24, 2024
53b22cf
fixes 571, added test (#574)
sllynn Sep 24, 2024
0582410
added python bindings for rst_avg, rst_max, rst_median, rst_min, and …
sllynn Sep 24, 2024
3964d7a
added missing driver param documented for RST_FromContent (#577)
sllynn Sep 24, 2024
0267bb0
fixed missing bindings for rst_type and rst_updatetype
sllynn Sep 24, 2024
4dffb54
pinned numpy version
sllynn Sep 24, 2024
0a9d8bd
added gdal UDF example to docs (#579)
sllynn Sep 24, 2024
a019c23
Raster api tweaks (#578)
sllynn Sep 25, 2024
78a2057
fixed issue 434: some two digit BNG grid references interpreted as qu…
sllynn Sep 25, 2024
4dc9bb6
updated CHANGELOG
sllynn Sep 25, 2024
225f95d
catch for writing subdataset-based rasters over FUSE
sllynn Sep 30, 2024
412d44f
additional test for subdivide on subdatasets
sllynn Oct 11, 2024
ba042ae
disabled checkpointing by default, fixed RST_Write failure when check…
sllynn Oct 22, 2024
5b76490
fix failing python test (1)
sllynn Oct 23, 2024
3307661
fix failing python test (2)
sllynn Oct 23, 2024
74405b5
swap chown for chmod
sllynn Oct 23, 2024
1f91095
fix failing R tests
sllynn Oct 23, 2024
0c39139
fixes to triangulation; exposed split point finding algorithm choice
sllynn Oct 31, 2024
bc1b86f
updated R bindings and tests
sllynn Nov 1, 2024
ab46dfd
fixed python bindings and tests
sllynn Nov 1, 2024
380e441
fixed failing test (sidecar files not cleaned up between checkpoint o…
sllynn Nov 1, 2024
a38dc21
updated docs to reflect new sigs of triangulation funcs
sllynn Nov 1, 2024
a9b17e4
Resolving triangulation issues (#592)
sllynn Nov 1, 2024
7e959f2
changes to the RasterAsGridReader and ReTileOnRead processes
sllynn Nov 8, 2024
dfa21de
removed some cruft
sllynn Nov 8, 2024
ff07ebe
switch runners
sllynn Nov 8, 2024
9db3d35
fixed failing zarr tests
sllynn Nov 8, 2024
4aa2f10
disabled the test for netcdf with checkpointing disabled
sllynn Nov 8, 2024
a479ab7
added function `RST_AsFormat()` to change raster format / driver in-situ
sllynn Nov 12, 2024
02c8e42
fixed zarr test
sllynn Nov 12, 2024
27fd3ec
Fixes to the raster as grid reader (#593)
sllynn Nov 12, 2024
6eb5010
added python bindings and tests
sllynn Nov 12, 2024
26cbb7f
updated the logic of some of the statistical raster functions to work…
sllynn Nov 13, 2024
a8ef2d3
Update file format in-situ (#596)
sllynn Nov 13, 2024
851e69d
Small area stats (#597)
sllynn Nov 13, 2024
474afce
Merge pull request #575 from databrickslabs/pre-0.4.3-release
mjohns-databricks Nov 14, 2024
cbab29c
updated changelog
sllynn Nov 14, 2024
8b33cdb
updated version ref in documentation
sllynn Nov 14, 2024
ef61a00
updated docs for raster_to_grid
sllynn Nov 14, 2024
b89bf6f
Merge branch 'main' into fix/changelog-0.4.3
sllynn Nov 14, 2024
6649f2a
Merge pull request #598 from databrickslabs/fix/changelog-0.4.3
mjohns-databricks Nov 14, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
updated the logic of some of the statistical raster functions to work…
… with small clipping outputs
  • Loading branch information
sllynn committed Nov 13, 2024
commit 26cbb7f8bc8e5f5f523adb2a88c1911f91bf8d71
Original file line number Diff line number Diff line change
@@ -148,9 +148,19 @@ case class MosaicRasterBandGDAL(band: Band, id: Int) {
*/
def computeMinMax: Seq[Double] = {
val minMaxVals = Array.fill[Double](2)(0)
Try(band.ComputeRasterMinMax(minMaxVals, 0))
.map(_ => minMaxVals.toSeq)
.getOrElse(Seq(Double.NaN, Double.NaN))
// will GDAL refuse to compute these stats?
if (band.GetXSize() == 1 || band.GetYSize() == 1) {
val validPixels = values.filter(_ != noDataValue)
if (validPixels.isEmpty) {
return Seq(Double.NaN, Double.NaN)
} else {
Seq(validPixels.min, validPixels.max)
}
} else {
Try(band.ComputeRasterMinMax(minMaxVals, 0))
.map(_ => minMaxVals.toSeq)
.getOrElse(Seq(Double.NaN, Double.NaN))
}
}

/**
Original file line number Diff line number Diff line change
@@ -9,7 +9,8 @@ import org.scalatest.matchers.should.Matchers._

trait RST_MedianBehaviors extends QueryTest {

def behavior(indexSystem: IndexSystem, geometryAPI: GeometryAPI): Unit = {
def largeAreaBehavior(indexSystem: IndexSystem, geometryAPI: GeometryAPI): Unit = {

val mc = MosaicContext.build(indexSystem, geometryAPI)
mc.register()
val sc = spark
@@ -49,4 +50,38 @@ trait RST_MedianBehaviors extends QueryTest {

}

def smallAreaBehavior(indexSystem: IndexSystem, geometryAPI: GeometryAPI): Unit = {

val mc = MosaicContext.build(indexSystem, geometryAPI)
mc.register()
val sc = spark
import mc.functions._
import sc.implicits._

val testRegion = "Polygon ((-8151524 1216659, -8151061 1216659, -8151061 1217123, -8151524 1217123, -8151524 1216659))"

val rastersInMemory = spark.read
.format("gdal")
.option("raster_storage", "in-memory")
.load("src/test/resources/modis")

rastersInMemory.createOrReplaceTempView("source")

val df = rastersInMemory
.withColumn("tile", rst_clip($"tile", st_buffer(lit(testRegion), lit(-20))))
.withColumn("result", rst_median($"tile"))
.select("result")
.select(explode($"result").as("result"))

noException should be thrownBy spark.sql("""
|select rst_median(tile) from source
|""".stripMargin)

val result = df.as[Double].collect().max

result should be > 0.0

}


}
Original file line number Diff line number Diff line change
@@ -22,11 +22,18 @@ class RST_MedianTest extends QueryTest with SharedSparkSessionGDAL with RST_Medi

// These tests are not index system nor geometry API specific.
// Only testing one pairing is sufficient.
test("Testing rst_median behavior with H3IndexSystem and JTS") {
test("Testing rst_median behavior with H3IndexSystem and JTS (tessellation case)") {
noCodegen {
assume(System.getProperty("os.name") == "Linux")
behavior(H3IndexSystem, JTS)
largeAreaBehavior(H3IndexSystem, JTS)
}
}

test("Testing rst_median behavior with H3IndexSystem and JTS (small area case)") {
noCodegen {
assume(System.getProperty("os.name") == "Linux")
smallAreaBehavior(H3IndexSystem, JTS)
}
}

}