Support list vor env "CRATE_HOST"

[SBlechmann]

Is your feature request related to a problem? Please describe.
If I understood correctly currently QL only supports one crate host given through the env variable "CRATE_HOST". So if I set up my crate cluster on different nodes, each with another configuration (that's why I didn't use "mode: global" and thus have multiple crate services), I cannot give QL a list of the services / IPs. Is that right?

Describe the solution you'd like
Implement the possibility to use a list of different services / crate hosts and the cluster's name // replicaset similar to when orion handles different mongo hosts.

Describe alternatives you've considered
I believe putting a proxy in front of crate that connects all crate nodes could work as well.

Additional context
Thanks for looking into this :)

[c0c0n3]

@SBlechmann thanks for the suggestion!

"CRATE_HOST" ... I cannot give QL a list of the services / IPs. Is that right?

That's correct. CRATE_HOST takes a single host name, not a list. That's a bit unfortunate since it looks like you can actually configure the Crate driver to use a server pool quite trivially

• https://github.com/crate/crate-python/blob/master/src/crate/client/connection.py#L106
• https://github.com/crate/crate-python/blob/master/src/crate/client/http.py#L345

I believe putting a proxy in front of crate that connects all crate nodes could work as well.

Yep, spot on. Also in a K8s deployment you could just set CRATE_HOST to the Crate service name and then have K8s route traffic to the Crate nodes for you

• https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies

This is what we tend to do with our deployments and it has the advantage that we're not limited to a static list of nodes---as it'd be the case if CRATE_HOST took a list of servers. As new nodes join the Crate cluster or old ones die, traffic would still be distributed to all available cluster nodes without having to reconfigure QuantumLeap.

[SBlechmann]

@c0c0n3 as always: Thanks for the very fast response!

Well, I believe I really have to switch to k8s soon.. it's still in my bucket list. Thanks for sharing your thoughts!
Maybe this might still be easy to implement? Of course the list would need fitting everytime you add a new node / a new service but maybe this might be sufficient for smaller use cases?

BTW: In another issue I said that I wanted to conduct some performance tests. Just at on Tuesday I got to implement 0.8 and seriously, guys, the step from 0.7.5(6) to 0.8 is huge. Thank you for the new performance improvements and way better docs!

[c0c0n3]

@SBlechmann

Thanks for the very fast response!

Pleasure! We try to tend to GH issues from the community as quickly as we can, but sometimes we've got so much in our plate that it could take us a couple of days to go through the list...

I believe I really have to switch to k8s soon

Obviously, you're the best person to judge what you need given your use case and requirements. If you go with K8s, these Helm charts might be useful to whip together a FIWARE cluster:

• https://github.com/orchestracities/charts

But it could be overkill for your scenario, I have no idea :-) What I know is that K8s has a steep learning curve (well at least that was the case for me, but I'm not the brightest chap out there) and might not be worth it for simple scenarios where e.g. a simple Docker compose or plain OS services would do. For example, you could even run QuantumLeap outside of Docker with

python app.py

which can easily be hooked into a systemd unit. (You could also use a separate GUnicorn instance, tweak GUnicorn config, etc.)

Maybe this might still be easy to implement? Of course the list would need fitting everytime you add a new node / a new service but maybe this might be sufficient for smaller use cases?

Indeed. It shouldn't be too hard to implement actually and like you pointed out it comes in handy for simple scenarios. One thing that I'd like to find out though is if the Crate guys have any plans to do that in the Crate driver. So I'm summoning the Mighty Motl, @amotl do you know if it could be possible for the Crate client to populate the list of servers dynamically

• https://github.com/crate/crate-python/blob/master/src/crate/client/http.py#L345

by querying the Crate cluster topology given the address of a configured cluster member? The CrateDB logs seem to give away that each node in the cluster is actually aware of the cluster topology, but I could be wrong!!

Thank you for the new performance improvements and way better docs!

Kudos to @chicco785 :-)

[amotl]

Hi there,

thanks for bringing up that topic. I discussed that with @mfussenegger and @seut already and they told me that a simple roundrobin-like distribution/balancing mechanism is implemented in crate-python already. However, the same thing can also be implemented by using some K8s ingress technologies or by just using a dedicated HAProxy (containerized or not) to use more sophisticated load balancing mechanisms based on weighting and more [1].

On the PostgreSQL interface side (see also #407), I also would like to reference psycopg/psycopg2#602 and sqlalchemy/sqlalchemy#4392 here just for the sake of completeness.

Specifically attaching to people aiming to run CrateDB on Kubernetes, I also would like to point out crate-operator here.

Of course the list would need fitting everytime you add a new node / a new service but maybe this might be sufficient for smaller use cases?

On this matter, I just created crate/crate-operator#170 which might yield some more insights how things may be improved on those aspects how to assemble things in an advanced manner.

With kind regards,
Andreas.

[1] http://cbonte.github.io/haproxy-dconv/2.3/configuration.html#4-balance

[c0c0n3]

Hi @amotl

Thank you so much for this, you're awesome.

roundrobin-like distribution/balancing mechanism is implemented in crate-python already

is it doing round robin on the list of servers you pass in when you create the client:

• https://github.com/crate/crate-python/blob/master/src/crate/client/connection.py#L106

or is it also fetching the cluster topology from the nodes currently in the Crate cluster? e.g. if I have a Crate cluster with 4 nodes, n1 to n4 and I create a connection with servers = [n1, n2], is the client going to automagically add n3 and n4 to the list? If so, does the list get refreshed over time so if the topology changes the list gets eventually updated too?

On the PostgreSQL interface side (see also #407), I also would like to reference psycopg/psycopg2#602 and sqlalchemy/sqlalchemy#4392 here just for the sake of completeness.

Excellent! Note to self: when tackling this issue, implement the same changes in the timescale backend too.

I also would like to point out crate-operator here

Like I said some time ago, we're very excited about this new addition to the Crate family!

[amotl]

Hi @c0c0n3,

Is crate-python doing round robin on the list of servers you pass in when you create the client.

Exactly. Currently, there is no further smartness here. Whether it would be actually possible to inquire topology information from the cluster itself might get revealed on an eventual discussion at crate/crate-operator#170. From MongoDB, I know the Python client driver supports cluster node discovery:

The addresses passed to MongoClient() are called the seeds. As long as at least one of the seeds is online, MongoClient discovers all the members in the replica set, and determines which is the current primary and which are secondaries or arbiters.

-- https://pymongo.readthedocs.io/en/stable/examples/high_availability.html#id1

However, I don't know if it will be possible to implement with CrateDB. Let's wait what the core developers will have to say about this and maybe also file another ticket at https://github.com/crate/crate in order to signal demand for such a feature.

With kind regards,
Andreas.

[c0c0n3]

Hi @amotl, thanks for clarifying. On second thought, I think regardless if the Crate client will eventually support node discovery, QuantumLeap should support reading a list of Crate hosts from config. In fact, as @SBlechmann pointed out, this feature would be useful right away, since the Crate client can already work quite happily with a list of servers. If one day, you guys implement node discovery, all the better. Those hosts you configure in QuantumLeap will serve as discovery seeds, like it happens in e.g. Mongo.

[amotl]

Hi again,

at crate/crate-operator#170, @SStorm and @WalBeh provided more insights into how crate-operator works - thanks a stack. This might tremendously help when operating CrateDB within a Kubernetes environment as you won't have to care about making the client aware of multiple hosts. The K8s-LB will be configured automatically.

It would still be helpful if CRATE_HOST can accept a list of endpoints when operated in non K8s-environments, indeed.

With kind regards,
Andreas.

[amotl]

Hi once more,

after talking to @WalBeh (thanks again!), I want to add some more important details in order to clarify the situation.

When using crate-operator on an IaaS platform, it will be able to configure the load balancing components provided by Azure and AWS appropriately. This is done through a respective plugin.

However, when running K8s in an on-premise environment, this has to be done using a different technology than what the IaaS LB components provide. For example, he mentioned MetalLB in this context. crate-operator does not provide an option for that yet.

With kind regards,
Andreas.

[c0c0n3]

@amotl thanks alot for all the info!

when running K8s in an on-premise environment, ... crate-operator does not provide an option for that yet.

Thanks, for pointing it out I'd totally missed that after reading the other thread!

[SBlechmann]

Hey there,

apperently, I got some discussion rolling o.o Thanks for all the contributions @c0c0n3 @amotl

My thought of providing more than one crate host (e.g. service - I'm using docker swarm terminology) was meant to deal with use cases where you have different crate nodes with different roles (e.g. data-node // manager // combinations).
Just to see if I got that right - since I'm more a "user" of both crate and QL:

• QL will have a look into this and might add the ability to give QL more then one host
• on k8s there is crate-operator that handles all the loadbalancing and discovering of new nodes (but only in Azure and AWS?)
• crate-operator still does not support to distinguish different roles of nodes but that might be a topic in the future

So for me right now I will just create simpler setups where all nodes are equal so I can use deploy mode global and don't have to deal with the issue yet, right? (swarm)
Sorry for asking but I'm not familiar with k8s yet: What would a solution on k8s that is not hosted on Azure nor AWS look like now?

Thanks for your time!

[c0c0n3]

hey @SBlechmann

Thanks for all the contributions

Pleasure!

QL will have a look into this and might add the ability to give QL more then one host

Definitely, not sure exactly when, but keep tabs on this issue :-)

So for me right now I will just create simpler setups where all nodes are equal so I can use deploy mode global and don't have to deal with the issue yet, right? (swarm)

Well, I suppose it depends on where QL sits (e.g. outside/inside the swarm), whether you're doing load balancing e.g. HAProxy, etc. Not sure what your actual set up is.

What would a solution on k8s that is not hosted on Azure nor AWS look like now?

I've got no experience with Azure, but I suppose if you're not using a managed K8s service and you just roll out your own K8s on bare Azure nodes, you could have a Crate stateful set with a service that takes care of distributing requests coming from e.g. QL to the Crate pods in the stateful set. In this case all your QL instances would be configured to use one address in CRATE_HOST which is e.g. the name of the service. If you want to go down that road, you could use these charts:

• https://github.com/orchestracities/charts/tree/master/charts/crate

Here's an (simplified!) example of what kind of Crate service those charts would install on a bare-bones K8s cluster

apiVersion: v1
kind: Service
metadata:
  labels:
    app: crate
  name: crate-crate
  namespace: prod
spec:
  ports:
  - name: crate-web
    port: 4200
  - name: cluster
    port: 4300
  - name: postgres
    port: 5432
  selector:
    app: crate
    release: crate

and here's the (simplified!) stateful set:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app: crate
  name: crate-crate
  namespace: prod
spec:
  replicas: 3
  selector:
    matchLabels:
      app: crate
  serviceName: crate-crate
  template:
    spec:
      affinity:
        podAntiAffinity:
          ...
      containers:
      - command:
        - /docker-entrypoint.sh
        - -Cnode.name="$POD_NAME"
        - -Ccluster.name=${CLUSTER_NAME}
        - -Cnetwork.host="0.0.0.0"
        - -Cdiscovery.seed_providers=srv
        - -Cdiscovery.srv.query=_cluster._tcp.crate-crate.prod.svc.cluster.local
        - -Ccluster.initial_master_nodes=crate-crate-0,crate-crate-1,crate-crate-2
        - -Cgateway.recover_after_nodes=${RECOVER_AFTER_NODES}
        - -Cgateway.recover_after_time=10s
        - -Cgateway.expected_nodes=${EXPECTED_NODES}
        - -Chttp.cors.enabled=true
        - -Chttp.cors.allow-origin=*
        - -Cpath.repo="/backup"
        env:
        - name: CRATE_HEAP_SIZE
          value: 1536m
        - name: EXPECTED_NODES
          value: "3"
        - name: CLUSTER_NAME
          value: quantumleap
        - name: RECOVER_AFTER_NODES
          value: "3"
        - name: MINIMUM_MASTER_NODES
          value: "2"
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        image: crate:4.1.8
        imagePullPolicy: Always
        name: crate
        ports:
        - containerPort: 4200
          name: crate-web
        - containerPort: 4300
          name: cluster
        - containerPort: 5432
          name: postgres
        resources:
          limits:
            cpu: "3"
            memory: 5Gi
          requests:
            cpu: 300m
            memory: 2Gi
        volumeMounts:
        - mountPath: /data
          name: data
      dnsPolicy: ClusterFirst
      initContainers:
         ...
  updateStrategy:
    type: RollingUpdate
  volumeClaimTemplates:
  - spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 30Gi
      storageClassName: fast
      volumeMode: Filesystem

Now with this set up, you could run a bunch of QL pods in the same namespace (prod) and set CRATE_HOST=crate-crate for all of them. K8s will take care of distributing the load among the 3 nodes in the stateful set without QL even knowing how many Crate nodes there are. Best of all, if you scale your stateful set (e.g. add nodes), everything will still work since QL is reaching Crate through the service name (crate-crate) not a specific IP.

[github-actions]

Stale issue message