Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce poppy-cli into the tegola container #7

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Introduce poppy-cli into the tegola container
thesocialdev committed Jun 11, 2021
commit c1aa7cd91fd25a2064768531da35317fbc79adc2
9 changes: 8 additions & 1 deletion docker-compose.yml
Original file line number Diff line number Diff line change
@@ -116,9 +116,16 @@ services:

### Tegola ##################################
tegola:
build: https://github.com/go-spatial/tegola.git#v0.14.x
build: ./tegola
# build: https://github.com/go-spatial/tegola.git#v0.14.x
environment:
- TEGOLA_BROKER_URL=redis://redis:6379
- TEGOLA_QUEUE_NAME=pregen
- TEGOLA_PATH=/opt/tegola
- TEGOLA_CONFIG_PATH=/etc/tegola.toml
volumes:
- ./tegola/tegola.toml:/etc/tegola.toml
- ./tegola/pregenerate-maps-tile.sh:/etc/pregenerate-maps-tile.sh
- ${DATA_PATH_HOST}/tegola/cache:/etc/cache
ports:
- "${TEGOLA_PORT}:8080"
5 changes: 5 additions & 0 deletions makefile
Original file line number Diff line number Diff line change
@@ -44,6 +44,11 @@ imposm_run:
notify_tilerator:
docker-compose exec kartotherian bash -c ". /.nvm/nvm.sh && nvm use 10.15.2 && node /home/kartotherian/packages/tilerator/scripts/tileshell.js --config /etc/opt/config.tilerator.docker.yaml -j.fromZoom 10 -j.beforeZoom 16 -j.generatorId gen -j.storageId v4 -j.deleteEmpty -j.expdirpath /srv/expiretiles -j.expmask '(expire\.list\.*)|(\.tiles)' -j.statefile /home/kartotherian/expire.state"

pregen_dequeue:
docker-compose exec tegola /etc/pregenerate-maps-tile.sh
pregen_enqueue:
docker-compose exec tegola poppy --broker-url redis://redis:6379 --queue-name pregen enqueue --message-input tile 1/1/1

install:
# TODO
# Check if kartotherian is installed and clone if it isn't
38 changes: 38 additions & 0 deletions tegola/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# Based on upstream Dockerfile https://github.com/go-spatial/tegola/blob/v0.14.x/Dockerfile
FROM golang:1.16.2-alpine3.12 AS build

ARG VERSION="Version Not Set"
ENV VERSION="${VERSION}"

# Only needed for CGO support at time of build, results in no noticable change in binary size
# incurs approximately 1:30 extra build time (1:54 vs 0:27) to install packages. Doesn't impact
# development as these layers are drawn from cache after the first build.
RUN apk update \
&& apk add musl-dev=1.1.24-r10 \
&& apk add gcc=9.3.0-r2 \
&& apk add git

# Set up source for compilation
RUN mkdir -p /go/src/github.com/go-spatial/tegola
RUN git clone "https://gerrit.wikimedia.org/r/operations/software/tegola" /go/src/github.com/go-spatial/tegola

# Build binary
RUN cd /go/src/github.com/go-spatial/tegola/cmd/tegola \
&& go build -v -ldflags "-w -X 'github.com/go-spatial/tegola/cmd/tegola/cmd.Version=${VERSION}'" -gcflags "-N -l" -o /opt/tegola \
&& chmod a+x /opt/tegola

# Create minimal deployment image, just alpine & the binary
FROM python:3-alpine3.12

RUN apk update \
&& apk add ca-certificates \
&& apk add git \
&& rm -rf /var/cache/apk/*

RUN pip install "git+https://github.com/wikimedia/poppy-cli@dequeue-until-empty" \
redis \
kafka-python

COPY --from=build /opt/tegola /opt/
WORKDIR /opt
ENTRYPOINT ["/opt/tegola"]
44 changes: 44 additions & 0 deletions tegola/pregenerate-maps-tile.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
#!/bin/sh
if [ -z "$TEGOLA_BROKER_URL" ]
then
echo "TEGOLA_BROKER_URL env var is not set"
exit 1
fi
if [ -z "$TEGOLA_QUEUE_NAME" ]
then
echo "TEGOLA_QUEUE_NAME env var is not set"
exit 1
fi
if [ -z "$TEGOLA_PATH" ]
then
echo "TEGOLA_PATH env var is not set"
exit 1
fi
if [ -z "$TEGOLA_CONFIG_PATH" ]
then
echo "TEGOLA_CONFIG_PATH env var is not set"
exit 1
fi
TMP_DIR=$(mktemp -d /tmp/tegola-XXXXXXXXXX)
TILELIST_PATH=${TEGOLA_TILELIST_DIR:-$TMP_DIR}/tilelist.txt
BATCH_SIZE=${TEGOLA_PREGENERATION_BATCH_SIZE:-1000}
DEQUEUE_TIMEOUT=${TEGOLA_PREGENERATION_DEQUEUE_TIMEOUT:-60}
set -x
while true;
do
# Dequeue a batch of messages from the queue and store them in tilelist
poppy --broker-url "$TEGOLA_BROKER_URL" \
--queue-name "$TEGOLA_QUEUE_NAME" \
dequeue --batch "$BATCH_SIZE" --exit-on-empty True --dequeue-raise-on-empty True --blocking-dequeue-timeout "$DEQUEUE_TIMEOUT" > "$TILELIST_PATH"
status=$?
# Pregenerate tiles that exist in tilelist
$TEGOLA_PATH --config "$TEGOLA_CONFIG_PATH" cache seed tile-list "$TILELIST_PATH"
if [ $status -eq 100 ] # Queue is empty
then
exit 0
elif [ $status -gt 0 ] # Something went wrong
then
exit 1
fi
done
set +x