How to use image for CI?

[StarpTech]

Hi, I'm planning to use couchbase but I couldn't find a solution to automate the installation of the database. I want to run it on travis or appveyor. What's the way to go here?

[ceejatec]

Do you mean configuration of the database, ie, setting up buckets, configuring memory limits, etc?

[StarpTech]

Hi @ceejatec I'm talking about a simple way to start the server without any manual intervention. I want to run tests with the couchnode driver on travis or any other ci environment.

[StarpTech]

I could find a sample travis script but I don't know if its work. Is it the right way?

https://github.com/evantahler/couchbase-structures/blob/master/.travis.yml

[ceejatec]

I know nothing about Travis, but from that file it appears that it's installing Couchbase from a package. Since you posted this question on the couchbase/docker repository, I assumed you were trying to do some automation involving the Couchbase Docker image - is that not correct?

[StarpTech]

No no when it's possible with docker it would be great. I'm looking for a solution to cover this with docker.

[ceejatec]

There's an official Docker image for Couchbase Server; "docker run couchbase". More information about that can be found on Docker Hub: https://hub.docker.com/_/couchbase/

That should eliminate the need to install Couchbase Server. That takes me back to my initial question, though: are you looking to automate the configuration of the Server? (If you just run the "couchbase" image, you'll get a default server instance, and if you hit the web interface you'll be presented with the Setup Wizard.)

[StarpTech]

Yes, I'm looking for a way to automate the configuration of the Server. On a CI platform there is no UI.

[ceejatec]

Well, I won't claim it's necessarily the best way, but take a look at https://github.com/couchbaselabs/analytics-docker/blob/master/scripts/configure-node.sh .

If you create a custom Docker image with a Dockerfile something like

FROM couchbase
COPY scripts/configure-node.sh /etc/service/config-couchbase/run

Then it'll run the commands in that script after Couchbase initially comes up. That's good enough for configuring things like memory limits, which services are enabled, etc. (It might be easier to do the configuration with couchbase-cli rather than direct REST calls.)

Note: the block at the bottom of that script that starts "if [ "$TYPE" = "WORKER" ]" doesn't actually work. It's intended for use when running the image in multiple containers to allow them to automatically join together into a cluster. It would probably work except that the runit process manager in the container strips out environment variables from "docker run -e". I haven't gotten around to fixing that yet.

[StarpTech]

What do you mean with

Well, I won't claim it's necessarily the best way, but take a look at

what's the alternative to setup couchbase in headless mode?

[StarpTech]

so there is no official examples or documentation about how to setup couchbase with docker or on localhost without to through the setup? How do you implement automated tests?

[ceejatec]

Our testing (to the best of my knowledge - I'm not in QA) treats Docker testing the same as normal platform testing, and does cluster configuration from outside the cluster itself using couchbase-cli in the way we expect customer deployments work. Our documentation certainly covers this use case, eg. https://developer.couchbase.com/documentation/server/4.6/cli/cli-intro.html .

My suggestion was for a way to get the Docker image itself to configure the node from inside, so it's self-contained. That may or may not meet your needs.

[StarpTech]

Thanks for the insides. Is it possible to skip the Setup Wizard with the couchbase-cli in the container?

[ceejatec]

Yes; a node configured using the script I showed will no longer show the Setup Wizard when you visit the web interface. I'm not positive which step sets the node into the "configured" state such that the Wizard is no longer necessary, but I believe it's either the "Configure Services" or "Set up credentials" step (or possibly both together) which do so.

[StarpTech]

All right and thanks. I will try it.

[StarpTech]

Hi @ceejatec how can I access the couchbase-cli tool ?

version: "3"
services:
  db:
    image: couchbase
    command: 
      - /opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1:8091 --cluster-init-username=Administrator --cluster-init-password=password --cluster-init-ramsize=256
      - /opt/couchbase/bin/couchbase-cli bucket-create -c 127.0.0.1:8091 --bucket=default --bucket-password=password --bucket-type=couchbase --bucket-port=11211 --bucket-ramsize=100 --bucket-replica=0 -u Administrator -p password
    ports:
      - 8091:8091
      - 8092:8092
      - 8093:8093
      - 8094:8094
      - 11210:11210
      - 11211:11211

error

db_1  | /entrypoint.sh: line 9: /opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1:8091 --cluster-username=Administrator --cluster-password=password --cluster-ramsize=256: No such file or directory

[ceejatec]

/opt/couchbase/bin/couchbase-cli in the container is the correct path. /opt/couchbase/bin is also on the PATH, so you can just use "couchbase-cli".

You have some problems with your compose file, though:

1. The "command" field in a docker-compose YAML file isn't a sequence, it's a string. I'm not sure why the YAML parser doesn't flag that, but there's no support for executing a series of commands via compose.

2. Even if you could do that, it wouldn't work the way you want. The Couchbase container, following standard Docker conventions, treats an explicit command as an override of the default behaviour. So the compose file above would start a container, which would run the "couchbase-cli" command (which would throw an error because there's no database running) and then immediately exit. It would never start a database.

If you want to configure the database, you need to first start the database by running a plain Couchbase container with no command, wait for the server to start up, and then run the couchbase-cli commands or make REST calls to configure that server. That's what the script I mentioned earlier does, from within the container itself. That does mean that you have to build your own Docker image, and that the image will only configure itself one way. You could make the shell script read a configuration file or similar that you mount in from outside, or even mount the shell script itself in from outside, depending on the environment you're trying to run in.

[StarpTech]

Thank you for the precise answer. Could you provide a simple example (init cluster, create bucket) where the wizard is gone? That would awesome.

[ceejatec]

I don't think it can be done strictly from a docker-compose file. You need to either have a custom image with some configuration logic baked in, or else you need to be able to mount a script from outside the container. Neither is especially challenging; which is best just depends on your working method. For instance, if you're firing this off in Travis and Travis doesn't allow bind-mounting of files into containers, then you'd need to go with the custom-image route. (Note: as I said I've never used Travis, so I don't know if Travis has that limitation or not; it was just an example of how the environment matters.)

Which most closely matches your scenario?

[ceejatec]

(As a note, having had this discussion with you, I can see how it might be cool or handy to enhance the container so that you COULD configure a database, or even a cluster, entirely from a docker-compose file. That would be an interesting challenge.)

[hookenz]

After the server is started you can run couchbase cli commands with docker exec. Create bucket, etc

I usually create a bash script to do the whole job.

[cha87de]

Sounds like a request for the pull request #66 - is this what you are looking for, @StarpTech ? Docker Couchbase is not really cloud native at the moment [1].

[1] https://12factor.net/

[StarpTech]

@cha87de yes, that would match with my requirements. Today, this is standard.

[cha87de]

@StarpTech you can build your own couchbase docker image like we are doing it here https://github.com/vice-registry/vice-registry. Have a look at the docker-compose.yaml and the couchbase folder, which uses the official couchbase docker image but replaces the entrypoint.
I'm thinking about simply creating a new repo with travis to build the latest couchbase CE version with this entrypoint and push it to docker hub. What do you think? Which couchbase version are you using?

[cha87de]

Update @StarpTech: I quickly moved all the stuff to a separate github repo [1] and linked it to a docker hub repo [2] with automated builds enabled. Feel free to use my docker image instead ;-)

[1] https://github.com/cha87de/couchbase-docker-cloudnative
[2] https://hub.docker.com/r/cha87de/couchbase-cloudnative/

[StarpTech]

Hi, @cha87de awesome I will try it and give you feedback.

[cepheidon]

@cha87de : Your suggested approach works for me! As I needed some further configuration, like creating indexes and views, I added another parameter to trigger further configuration on script level, so in docker-compose.yml it basically looks like that:

...
    environment:
      - CLUSTER=localhost
      - USER=Administrator
      - PASS=password
      - PORT=8091
      - RAMSIZEMB=1024
      - RAMSIZEINDEXMB=256
      - RAMSIZEFTSMB=256
      - BUCKETS=stock
      - BUCKETSIZES=256
      - AUTOREBALANCE=true
      - CONFIG_SCRIPT=/path/to/script.sh
...

In entrypoint.sh I added a function to execute that given script in case the parameter is set, similar to the other functions.
The configuration script and any required extras can now just be added to the image to be used from entrypoint.sh.

[cha87de]

@StarpTech I can recommend using RethinkDB, ArrangoDB or the like for automated cloud staging and deployments. I'm not sure how they behave under heavy load, but at least their deployments are without clicking through a web dashboard ;-)

[StarpTech]

@cha87de thanks for your feedback. I also a little bit lost why they don't provide a cloud-native approach. Couchbase has great capabilities but it's not easy in operational aspects and especially the documentation is heavy and not clear for a start from zero. I will come back someday in the future.