Strange libgdal.so.28 issue

[jlgutenson]

Describe the bug
I'm having an issue with a libgdal.so.28 dependency somewhere in go-consequences when setting up and running go-consequences using Docker Desktop on Windows. The error is:

/app/main: error while loading shared libraries: libgdal.so.28: cannot open shared object file: No such file or directory

after attempting to run a flood consequence assessment with the command:

docker run --mount type=bind,source="C:\Users\jlgut\OneDrive\Desktop\go-consequences",target=/data go-consequences-jlg:latest ./main "/data/joseph_test.json"

To Reproduce
Steps to reproduce the behavior:

1. Go to a local directory of the go-consequences Github repo (e.g., C:\Users\jlgut\OneDrive\Desktop\go-consequences).
2. Modify Dockerfile.prod because of initial issue with not finding the /main executable:

# Stage 1: Builder
FROM debian:bullseye AS builder

ENV TZ=America/New_York
ENV PATH=/go/bin:$PATH
ENV GOROOT=/go
ENV GOPATH=/src/go

RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone &&\
    mkdir -p /go &&\
    mkdir -p /src/go &&\
    apt update &&\
    apt -y install build-essential &&\
    apt -y install gdal-bin gdal-data libgdal-dev &&\
    apt -y install wget &&\
    wget https://golang.org/dl/go1.19.5.linux-amd64.tar.gz -P / &&\
    tar -xvzf /go1.19.5.linux-amd64.tar.gz -C / &&\
    apt -y install vim &&\
    apt -y install git

WORKDIR /app
RUN git clone https://github.com/USACE/go-consequences.git
WORKDIR /app/go-consequences
RUN go mod download
RUN go mod tidy
RUN go build -o main .

# Stage 2: Production
FROM ghcr.io/osgeo/gdal:ubuntu-full-3.8.3 AS prod

# Install GDAL and related libraries
RUN apt update && apt -y install gdal-bin gdal-data libgdal-dev

WORKDIR /app
COPY --from=builder /app/go-consequences/main ./

# Ensure the GDAL library is available
RUN ldconfig

ENTRYPOINT ["/app/main"]

4. Build the Docker Image with the command docker build -f Dockerfile.prod -t go-consequences-jlg .
5. Attempt to run go-consequences from command line docker run --mount type=bind,source="C:\Users\jlgut\OneDrive\Desktop\go-consequences",target=/data go-consequences-jlg:latest ./main "/data/joseph_test.json"
6. See error: /app/main: error while loading shared libraries: libgdal.so.28: cannot open shared object file: No such file or directory

Expected behavior
I was expecting to have go-consequences run this test-case (i.e., joseph_test.json):

{
    "structure_provider_info": {
        "structure_provider_type": "NSIAPI"
    },
    "hazard_provider_info": {
        "hazards": [
            {
                "hazard_parameter_type": "depth",
                "hazard_provider_file_path": "/data/MO_FABDEM_ARC_Flood_Forecast_20241030.shp"
            }
        ]
    },
    "results_writer_info": {
        "results_writer_type": "JSON",
        "output_file_path": "/data/MO_FABDEM_ARC_Flood_Forecast_20241030.json"
    }
}

[HenryGeorgist]

@jlgutenson cool work. looks great. do you know that I publish a go-consequences container that you can call like you want?

I also notice that you build on the gdal image, and that you do not grab build essential. In my prod container i have the following lines:

RUN apt update &&\
    apt -y install build-essential &&\
    apt -y install pkg-config &&\
    apt -y install gdal-bin gdal-data libgdal-dev

I can understand not wanting build essential in the prod container, but that is a potential for where this might differ.

I see that you have your hazard provider as a shapefile - did you update the code locally to work with that hazard provider type?

[jlgutenson]

Hey @HenryGeorgist, good to hear from you! Thanks for the quick response!

I didn't know you had your image published. That sounds like exactly what I need. Is it this guy?

That shapefile was actually supposed to be tiff file/depth grid reference. Good eye!

If so, I tried running docker run --mount type=bind,source="C:\Users\jlgut\OneDrive\Desktop\go-consequences",target=/data williamlehman/go-consequences:v0.0.2 ./main "/data/joseph_test.json" which is a modification to your command here.

However, I'm not finding the /main executable when I run your published image. I get this error: docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/main": stat /main: no such file or directory: unknown.

[HenryGeorgist]

hmm. that looks ok. did you try v0.0.1?

[HenryGeorgist]

C:\Modeling>docker run -v C:\Modeling:/data williamlehman/go-consequences:v0.0.1 ./main "/data/config.json"
Connecting to: /data/depthgrid.tif
Getting bbox
-92.537687,29.911019,-92.333679,29.911019,-92.333679,29.823433,-92.537687,29.823433,-92.537687,29.911019
https://nsi.sec.usace.army.mil/nsiapi/structures?bbox=-92.537687,29.911019,-92.333679,29.911019,-92.333679,29.823433,-92.537687,29.823433,-92.537687,29.911019&fmt=fs

C:\Modeling>docker run -v C:\Modeling:/data williamlehman/go-consequences:v0.0.2 ./main "/data/config.json"
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/main": stat /main: no such file or directory: unknown.

i can reproduce it on version 0.0.2, looks like the exported command was /main instead of ./main
let me try running interactively

[HenryGeorgist]

C:\Modeling>docker run -v C:\Modeling:/data -it --entrypoint /bin/bash williamlehman/go-consequences:v0.0.2
root@1804b072b3db:/app# ls
main
root@1804b072b3db:/app# ./main
./main: error while loading shared libraries: libgdal.so.28: cannot open shared object file: No such file or directory

reproduced with my checked in prod container.

[jlgutenson]

hmm. that looks ok. did you try v0.0.1?

Good suggestion!

I can confirm that your v0.0.1 image is running. But my outputs (I tried json, geopackage, and shapefile) with v0.0.1 are not complete once the analysis ends. That might be a connection issue on my end.

[jlgutenson]

hmm. that looks ok. did you try v0.0.1?

Good suggestion!

I can confirm that your v0.0.1 image is running. But my outputs (I tried json, geopackage, and shapefile) with v0.0.1 are not complete once the analysis ends. That might be a connection issue on my end.

Hey @HenryGeorgist, after further analysis, I'm having some issues with the v0.0.1 image's outputs that I don't believe are memory or internet connection related. I had ChatGPT analyze the file and it wasn't able to clean the JSON to make it usable in a GIS. Here is an attached JSON output for reference.
depth_max_clipped2.json

[HenryGeorgist]

on first glance it is missing a closing square brace at the end ] of the features array, and then it would need to close the feature collection object with a curly brace }
In json specification array items are separated by commas except for the last item in the array, so there is a comma at the end of your file that also needs to be deleted.

the end should look essentially like this }}]} it currently looks like this }},

this should happen in the "Close" methodology on the results writer. it is possible that the close method wasnt defered in version 0.0.1 for the geojson results writer.

i typically use geopackage. but if the close method is not deferred on it too then we would have a problem of not flushing the last transaction. i think geojson and editing the last two characters may be enough....

chat gpt? seriously?

[jlgutenson]

I gave }}]} at the end a whirl. No luck, unfortunately.

With v0.0.1, the Geopackage option keeps giving me only one damaged building. The one building is in the depth grid and pulling the correct depth but only writing the one location for some reason.

I tried the shapefile option as well and it was completely empty.

With ChatGPT, don't underestimate my laziness. I have it do the hard work and I just fill in where I need to or, in this case, I was stumped and I thought I'd see if it saw something in the JSON that I was missing. It seemed very much confused about the JSON issue.

[HenryGeorgist]

i sent you a file that worked for me in qgis after my edits. i use jsonlint.com to validate the json and ensure it meets the specification, it helps me point out the problems that need to be resolved. your description of geopackage seems consistent with the geojson writer in that it is not triggering the close method when the program is shutting down. looks like i need to fix the code one way or another eventually.
i was just giving you a hard time about chat gpt.

[jlgutenson]

Thank you! My email changed. Did you use [email protected] or [email protected]?

We just invested in the paid version of ChatGPT. We're somewhat hooked. It can be really good for working from home without a rubber duck handy! But it also can lead you on some wild goose chases!

I moved back to the original gdal issue that's plaguing v0.0.2. Still haven't figured that one out either. I'm not sure what GDAL version 2.8 dependencies there are in go-consequences.

It may be worth mentioning that this is all in an effort to demonstrate how go-consequences can be used to forecast flood consequences, if provided a forecasted flood depth grid.

[HenryGeorgist]

i run the latest in a different container and this is what i use as my production container:

FROM ghcr.io/osgeo/gdal:ubuntu-full-3.8.3 as builder

ENV TZ=America/New_York
ENV PATH=/go/bin:$PATH
ENV GOROOT=/go
ENV GOPATH=/src/go

RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone &&\
    mkdir /go &&\
    mkdir -p /src/go &&\
    apt update &&\
    apt -y install build-essential &&\
	apt -y install pkg-config &&\
    apt -y install gdal-bin gdal-data libgdal-dev &&\
    apt -y install wget &&\
    wget https://golang.org/dl/go1.19.13.linux-amd64.tar.gz -P / &&\
    tar -xvzf /go1.19.13.linux-amd64.tar.gz -C / &&\
    apt -y install vim &&\
    apt -y install git

WORKDIR /app
RUN git clone https://github.com/USACE/consequences-runner.git
WORKDIR /app/consequences-runner
RUN go mod download
RUN go mod tidy
RUN go build 


FROM ghcr.io/osgeo/gdal:ubuntu-full-3.8.3 as prod
WORKDIR /app
COPY --from=builder /app/consequences-runner/consequences-runner .
CMD ["/app/consequences-runner"]

it uses the osgeo ubuntu full - which is not small and maybe is overkill.

[HenryGeorgist]

gah, but thats the same one you were using. which doesnt make sense

[jlgutenson]

I think I got the gdal issues in v0.0.2 solved (with the help of our friend ChatGPT :-))! Maybe you can confirm. I've attached a geotiff for easy access.

depth_max_clipped2.json

Here is the modified Dockerfile.prod. It's just tricky the image into thinking it has a libgdal.so.28 file with a symbolic link.

# Stage 1: Builder
FROM debian:bullseye AS builder

ENV TZ=America/New_York
ENV PATH=/go/bin:$PATH
ENV GOROOT=/go
ENV GOPATH=/src/go

RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone &&\
    mkdir -p /go &&\
    mkdir -p /src/go &&\
    apt update &&\
    apt -y install build-essential &&\
    apt -y install libgdal-dev &&\
    apt -y install gdal-bin &&\
    apt -y install gdal-data &&\
    apt -y install wget &&\
    wget https://golang.org/dl/go1.19.5.linux-amd64.tar.gz -P / &&\
    tar -xvzf /go1.19.5.linux-amd64.tar.gz -C / &&\
    apt -y install vim &&\
    apt -y install git

WORKDIR /app
RUN git clone https://github.com/USACE/go-consequences.git
WORKDIR /app/go-consequences
RUN go mod download
RUN go mod tidy
RUN go build -o main .

# Stage 2: Production
FROM ghcr.io/osgeo/gdal:ubuntu-full-3.8.3 AS prod

# Move the executable from build to production
WORKDIR /app
COPY --from=builder /app/go-consequences/main ./

# Install necessary GDAL libraries
RUN apt update && \
    apt install -y libgdal-dev && \
    ln -s /usr/lib/libgdal.so /usr/lib/libgdal.so.28 && \
    ldconfig

# Verify library dependencies for debugging
RUN ldd ./main

ENTRYPOINT ["/app/main"]

I then built the Docker image: docker build -f Dockerfile.prod -t jlgutenson/go-consequences .

Then I can use the geojson to run go-consequences and output a geopackage:

docker run --rm --mount type=bind,source="C:\Users\jlgut\OneDrive\Desktop\go-consequences",target=/data jlgutenson/go-consequences /data/data/joseph_test.json

[HenryGeorgist]

that looks good to me if the results make sense and it runs!

[jlgutenson]

Sounds good, we will test a few different times and see if things make sense. Things seem correct for my Asheville test case.