task/WG-379: upgrade python to 3.11; reorg Dockerfiles; reorg `pyproject.toml` (first pass) #222

rstijerina · 2024-11-14T23:19:13Z

Overview:

Upgrades python to 3.11
Upgrades all packages
Reorganizes python dependencies into groups
Adds a new makefile build target: make build-dev
- Target for local development, production builds will not include dev dependencies

Related Jira tickets:

WG-379

rstijerina · 2024-11-21T16:53:17Z

geoapi/services/point_cloud.py

@@ -31,7 +34,7 @@ def get(database_session, pointCloudId: int) -> PointCloud:
        :param pointCloudId: int
        :return: PointCloud
        """
-        point_cloud = database_session.query(PointCloud).get(pointCloudId)
+        point_cloud = database_session.get(PointCloud, pointCloudId)


https://docs.sqlalchemy.org/en/20/changelog/migration_20.html#orm-query-get-method-moves-to-session

rstijerina · 2024-11-21T16:55:11Z

geoapi/tests/api_tests/test_vectors_service.py



 def test_process_shapefile_missing_additional_files(shapefile_fixture):
-    with pytest.raises(fiona.errors.DriverError):
-        _, _ = next(VectorService.process_shapefile(shapefile_fixture, additional_files=[]))
+    with pytest.raises(pyogrio.errors.DataSourceError):


https://geopandas.org/en/latest/docs/user_guide/fiona_to_pyogrio.html

rstijerina · 2024-11-21T16:56:33Z

geoapi/utils/lidar.py

+        las_file = laspy.read(input_file)
+        min_point = _transform_to_geojson(
+            proj4=proj4, point=tuple(las_file.header.mins[:3])
+        )
+        max_point = _transform_to_geojson(
+            proj4=proj4, point=tuple(las_file.header.maxs[:3])
+        )


https://laspy.readthedocs.io/en/latest/migration.html

Makefile

devops/docker-compose.local.yml

nathanfranklin · 2024-11-22T18:01:36Z

devops/pyproject.toml


 [tool.poetry.dependencies]
-python = "^3.9"
-celery = "5.2.3"
+python = "^3.11"


Co-authored-by: Nathan Franklin <[email protected]>

nathanfranklin · 2024-11-22T18:43:20Z

geoapi/utils/lidar.py

@@ -69,14 +69,13 @@ def get_bounding_box_2d(filePaths: List[str]) -> MultiPolygon:
    for input_file in filePaths:
        proj4 = getProj4(input_file)

-        las_file = laspy.file.File(input_file, mode="r-")
+        las_file = laspy.read(input_file)


need to add lazrs python package. looks like in the new laspy a backed needs to be installed. i installed lazrs and all works well with that locally. I'm investigating why the unit tests didn't pick that up (maybe the files we use to test this function don't trigger the error: "LaspyException: No LazBackend selected, cannot decompress data" ).

okay, i see the issue and added a unit test. we were never testing any compressed files (ie. laz). we had some files that had .laz ending but they were uncompressed. i got a PR targeting this branch and adding a unit test. see #224

Co-authored-by: Nathan Franklin <[email protected]>

… `pyproject.toml` (first pass) (#222)" This reverts commit 404e8d9.

* python black formatting; flake8 on everything * task/WG-379: upgrade python to 3.11; reorg Dockerfiles; reorg `pyproject.toml` (first pass) (#222) * upgrade python to 3.11; reorg Dockerfiles; reorg pyproject * update github actions to 3.11; formatting * gh actions dev dependenceis * upgrade packages * move laspy to main group; fix conda env name * fix tests * fix packages * fix initdb * upgrade packages; update import * resolve sqalchemy warnings * sqlalchemy 2.x syntax * laspy upgrades * update test; add rabbitmq healthcheck * use poetry flake8 and black * update workflows * Update devops/docker-compose.local.yml Co-authored-by: Nathan Franklin <[email protected]> * add make build-dev instruction to readme * Update devops/docker-compose.local.yml Co-authored-by: Nathan Franklin <[email protected]> * update gh action pg * add lazrs --------- Co-authored-by: Nathan Franklin <[email protected]> * Revert "task/WG-379: upgrade python to 3.11; reorg Dockerfiles; reorg `pyproject.toml` (first pass) (#222)" This reverts commit 404e8d9. --------- Co-authored-by: Nathan Franklin <[email protected]>

…ect.toml` (first pass) (#225) * python black formatting; flake8 on everything * upgrade python to 3.11; reorg Dockerfiles; reorg pyproject * update github actions to 3.11; formatting * gh actions dev dependenceis * upgrade packages * move laspy to main group; fix conda env name * fix tests * fix packages * fix initdb * upgrade packages; update import * resolve sqalchemy warnings * sqlalchemy 2.x syntax * laspy upgrades * update test; add rabbitmq healthcheck * use poetry flake8 and black * update workflows * Update devops/docker-compose.local.yml Co-authored-by: Nathan Franklin <[email protected]> * add make build-dev instruction to readme * Update devops/docker-compose.local.yml Co-authored-by: Nathan Franklin <[email protected]> * update gh action pg * add lazrs * task/WG-379: upgrade python to 3.11; reorg Dockerfiles; reorg `pyproject.toml` (first pass) (#222) * upgrade python to 3.11; reorg Dockerfiles; reorg pyproject * update github actions to 3.11; formatting * gh actions dev dependenceis * upgrade packages * move laspy to main group; fix conda env name * fix tests * fix packages * fix initdb * upgrade packages; update import * resolve sqalchemy warnings * sqlalchemy 2.x syntax * laspy upgrades * update test; add rabbitmq healthcheck * use poetry flake8 and black * update workflows * Update devops/docker-compose.local.yml Co-authored-by: Nathan Franklin <[email protected]> * add make build-dev instruction to readme * Update devops/docker-compose.local.yml Co-authored-by: Nathan Franklin <[email protected]> * update gh action pg * add lazrs --------- Co-authored-by: Nathan Franklin <[email protected]> * Revert "task/WG-379: upgrade python to 3.11; reorg Dockerfiles; reorg `pyproject.toml` (first pass) (#222)" This reverts commit 404e8d9. * hotfix: add unit test using compressed laz file (#224) * Add compressed laz file Changed some file names as the previous .laz files were uncompressed. * Add lidar_medium_subset_las1pt2_utmzone13N_compressed.laz * Fix formatting --------- Co-authored-by: Sal Tijerina <[email protected]> --------- Co-authored-by: Nathan Franklin <[email protected]>

rstijerina requested review from taoteg and nathanfranklin November 14, 2024 23:19

rstijerina commented Nov 21, 2024

View reviewed changes

rstijerina added 13 commits November 21, 2024 17:51

upgrade python to 3.11; reorg Dockerfiles; reorg pyproject

16efc08

update github actions to 3.11; formatting

5e93405

gh actions dev dependenceis

247fc6e

upgrade packages

f661f90

move laspy to main group; fix conda env name

32b3fe4

fix tests

da003ce

fix packages

9b20949

fix initdb

3f5c711

upgrade packages; update import

6cb24ae

resolve sqalchemy warnings

53d8fad

sqlalchemy 2.x syntax

4cea8e5

laspy upgrades

c7ad227

update test; add rabbitmq healthcheck

b9e8bbb

rstijerina force-pushed the task/WG-379--dockerfile-and-poetry-lock branch from c936c72 to b9e8bbb Compare November 22, 2024 00:13

rstijerina changed the base branch from main to black-formatter November 22, 2024 00:13

use poetry flake8 and black

127304d

rstijerina force-pushed the task/WG-379--dockerfile-and-poetry-lock branch 2 times, most recently from e291538 to e23719b Compare November 22, 2024 00:25

update workflows

3b2e1e5

rstijerina force-pushed the task/WG-379--dockerfile-and-poetry-lock branch from e23719b to 3b2e1e5 Compare November 22, 2024 00:27