What is the best way to populate the PgStac on eoapi-k8s?

[MathewNWSH]

Hello,

regarding developmentseed/eoAPI#178, I managed to deploy eoapi on a Kubernetes cluster running on the creodias cloud :)

Now I would like to populate it with items.

So far, working with PgStac, I populated it in the following way:

export PGHOST='host'
export PGPORT='port'
export PGUSER='username'
export PGDATABASE='databse'
export PGPASSWORD='password'

pypgstac load collections /home/eouser/Downloads/collection.json
pypgstac load items /home/eouser/Desktop/s2_item.json

Now I'm wondering how I can do this with a Kubernetes cluster. What came to my mind is:

• Port-forward to localhost and export new environment variables
  (cons: forwarding terminal must be running)
  kubectl port-forward service/pgstac 5432:5432

• Changing the service type from ClusterIP to LoadBalancer - then exporting the external IP address via environment variable.
  (cons: less secure because the database is exposed via an external IP)

any suggestions?

Best,
Mat

[ranchodeluxe]

Hey @MathewNWSH:

Thanks for the questions and sorry about the delayed response. If leveraging pypgstac CLI to load collections and items then exposing these service via port-forward or load balancer wouldn't really help AFAICS. None of those services are an ingest service

Rather I'd recommend:

0. kubectl exec -it <pod> -- bash into a pod that has pypgstac installed on it and with access to the DB and running the CLI there

1. Or writing a configmap script that gets mounted into the pod like we currently do and runs as an init container or lifecycle hook.

Of course this helm chart should attempt to make this easier for you (but currently doesn't as a demo). It will be ticketed

[ranchodeluxe]

@MathewNWSH

You probably already know this, but what you really want here is some kind of async task queue to do the ingest (whether or not that uses pypgstac or you manually write your own ingest scripts is of little importance here).

We have an example of such a system using Argo https://github.com/developmentseed/eoapi-ingestion-argo

If Argo feels like too much of a lift then soon I'll have an example up using Airflow that should be easy to reproduce/grok that you can probably use

[MathewNWSH]

Hey @ranchodeluxe is there any update on example of using Airflow?

[ranchodeluxe]

Hey @ranchodeluxe is there any update on example of using Airflow?

@MathewNWSH: there's currently nothing public to my knowledge but maybe @ividito has examples or documentation for other deploys up his sleeve that he can point to that are public?

If that's not true, then IMHO we should at least provide:

1. an example (tutorial, docker, something)
2. an actual deployment in the eoapi chart to just give folks this option (that's my choice but I don't wear all the pants 😉)

cc @j08lue @ciaransweet

[j08lue]

Let's talk about what can be done shortly. We might need a simple ingest service for

• Add more sample data to eoAPI EOEPCA/data-access#75

[pantierra]

As far as I can see there have been a number of different approaches, from leveraging the deployment pipeline for data ingestion to exposing the STAC API Transaction Extensions via stac-fastapi-pgstac. My perspective is that the database contains the "content," while deployment represents the "configuration" (which should be reproducible and version-controlled, ofc). With this in mind, I believe it's more practical to use the API endpoints, not just for ingestion, but also for overall data management.

To address the need for quickly adding data to the deployed pgSTAC, I’ve put together a small script that automates the process of identifying the appropriate pod, uploading STAC JSON files, and using pypgstac to load the data into the database.

In principle, I believe the STAC API Transactions Extension is a good direction, especially with the new STAC manager we will even get a nice UI for it. Right now we face the challenge in terms of access control, as these endpoints are currently exposed without any protection. It would be good to explore options for securing these endpoints and ensuring controlled access. Specifically, I see a few potential solutions on the Kubernetes (k8s) side that could help:

• we could add also to eoapi-k8s an API Gateway to handle authentication on those endpoints (in EOEPCA has ty.kio is going to be used)
• we could add some auth functionality to stac-fastapi-pgstac - maybe even by extending STAC API Authentication extension.