can changes go back to incubaid?

.gitignore

@@ -29,3 +29,4 @@ redhat/BUILD
 /_virtualenv
 /examples/ocaml/_build/
 /examples/ocaml/demo.native
+/testresults.xml

.gitmodules

@@ -1,3 +1,3 @@
 [submodule "pylabs/pyrakoon"]
 	path = pylabs/pyrakoon
-	url = git://github.com/Incubaid/pyrakoon.git
+	url = https://github.com/openvstorage/pyrakoon.git

.travis.yml

@@ -1,9 +1,22 @@
+dist: xenial
 language: c
-env:
-  - PPA=avsm/ocaml42+opam11
 
-before_install: ./travis.sh before_install
+sudo: required
+
 install: ./travis.sh install
 script: ./travis.sh script
+
 git:
     submodules: false
+
+services:
+  - docker
+
+# cache:
+#   directories:
+#     - ~/cache
+#   timeout:
+#     1000
+
+env:
+  - USE_CACHE=0 # set to 1 to enable

COMPILING

@@ -1,49 +1,28 @@
-We use opam
+We use docker to build arakoon, so the easiest way to get started is to use our docker images.
 
-Some of the packages we use are not in the main opam repository (yet). So it's easiest if you add our devel repo:
+Preliminary steps
 
-opam remote --kind git add devel git://github.com/Incubaid/opam-repository-devel.git
+Install docker for your system, see https://docs.docker.com/engine/installation for instructions.
+After that make sure your user belongs to the docker group: sudo usermod -a -G docker your_user
 
-Packages we use:
+Building
 
-ocamlfind.1.3.3
-snappy.0.1.0
-camlbz2.0.6.0
-ssl
-lwt with SSL support
-camltc
-quickcheck
+Run
+./docker/run.sh ubuntu-16.04 build
+from the root of the repository.
 
-When it's been a while you might want to run `opam update` first to get the latest versions.
+Running the tests
 
+./docker/run.sh ubuntu-16.04 unit
+./docker/run.sh ubuntu-16.04 b
+./docker/run.sh ubuntu-16.04 c
+./docker/run.sh ubuntu-16.04 d
 
-COMPILING:
+Enter the docker container
 
-make
-(you can inspect the makefile for more fine grained targets)
-(if you get errors and you've built arakoon before you could try a `make clean` first)
+./docker/run.sh ubuntu-16.04 bash
 
-RUNNING THE TESTS:
+Build packages
 
-	./arakoon.native --run-all-tests
-
-COVERAGE:
-Bisect needs to be installed. (get it from http://bisect.x9c.fr/)
-
-	ocamlbuild -tag use_bisect -tag use_macro arakoon.d.byte
-	./report.sh
-
-DEBIAN packages:
-	fakeroot debian/rules clean
-	fakeroot debian/rules build
-	fakeroot debian/rules binary
-
-Python egg:
-	python setup.py bdist_egg
-
-
-
-have fun,
-
-The arakoon team
+./docker/run.sh ubuntu-16.04 package_deb
 

GITBOOK.md

@@ -0,0 +1,58 @@
+# Arakoon
+
+[Arakoon](http://arakoon.org/) is a distributed key-value store that guarantees consistency above anything else. It was created by the Open vStorage team due to a lack of existing solutions fitting our requirements, and is available as Open Source software.
+
+Arakoon aims to be easy to understand and use, whilst at the same time taking the following features into consideration:
+* Consistency: The system as a whole needs to provide a consistent view on the distributed state. This stems from the experience that eventual consistency is too heavy a burden for a user application to manage. A simple example is the retrieval of the value for a key where you might receive none, one or multiple values depending on the weather conditions. The next question is always: Why don’t a get a result? Is it because there is no value, or merely because I currently cannot retrieve it?
+* Conditional and Atomic Updates: We don’t need full blown transactions (would be nice to have though), but we do need updates that abort if the state is not what we expect it to be. So at least an atomic conditional update and an atomic multi-update are needed.
+* Robustness: The system must be able to cope with failure of individual components, without concessions to consistency. However, whenever consistency can no longer be guaranteed, updates must simply fail.
+* Locality Control: When we deploy a system over 2 datacenters, we want guarantees that the entire state is indeed present in both datacenters. This is something we could not get from distributed hash tables using consistent hashing.
+* Healing & Recovery: Whenever a component dies and is subsequently revived or replaced, the system must be able to guide that component towards a situation where that node again fully participates. If this cannot be done fully automatically, then human intervention should be trivial.
+* Explicit Failure: Whenever there is something wrong, failure should propagate quite quickly.
+
+## An Arakoon deployment
+An Arakoon cluster consist of a small collection of nodes (typically 1,2,3 or 5 nodes) that contains the full range of key-value pairs, and clients that manipulate the key/value space. In principle, all nodes have the entire key/value space. There is one distinguished node called the master with which all clients communicate to perform updates. A client contacts any node to find out the master, and then just conversates with the master. If a master dies, a new one is elected automatically, and clients fail over to that master. A slave node is a node that is not master. A node that is not up-to-date cannot become master.
+
+The diagram below shows 3 clients connected to an Arakoon cluster that consists of 3 Arakoon nodes.
+![](docs/Images/ArakoonDeployment.png)
+
+Updates to the Arakoon database are consistent. An Arakoon client always looks up the master of a cluster and then sends a request to the master. The master node of a cluster has a queue of all client requests. The moment that a request is queued, the master node sends the request to all his slaves and writes the request in the Transaction Log (TLog). When the slaves receive a request, they store this also in their proper TLog and send an acknowledgement to the master.
+
+A master awaits for the acknowledgements of the slaves. When he receives an acknowledgement of half the nodes plus one, the master pushes the key/value pair in its database. In a five node setup (one master and four slaves), the master must receive an acknowledgement of two slaves before he writes his data to the database, since he is also taken into account as node.
+
+After having written his data in his database, the master starts the following request in his queue. When a slave receives this new request, the slaves first write the previous request in their proper database before handling the new request. This way a slave is always certain that the master has successfully written the data in his proper database.
+
+If the master node dies, a new master election happens using [paxos](https://en.wikipedia.org/wiki/Paxos_(computer_science)). If a slave dies, the master is not affected. When a slave comes up, it tries to catchup. The slave first downloads the missing parts of the log. Then it iterates over the tlog while adding the missing updates to its local store. When finished the client again compares its log state with that of the other nodes to make sure it is up to date.
+
+The directory structure of a node typically looks like this:
+```
+root@cmp02:/opt/OpenvStorage/db/arakoon/ovsdb# ls -als
+total 0
+0 drwxr-xr-x 1 ovs ovs 14 Dec  1 17:29 .
+0 drwxr-xr-x 1 ovs ovs 10 Dec  1 17:29 ..
+0 drwxr-xr-x 1 ovs ovs 84 Dec  2 13:22 db
+0 drwxr-xr-x 1 ovs ovs 58 Dec  3 12:07 tlogs
+root@cmp02:/opt/OpenvStorage/db/arakoon/ovsdb/db# ls -als
+total 680
+  0 drwxr-xr-x 1 ovs ovs     84 Dec  2 13:22 .
+  0 drwxr-xr-x 1 ovs ovs     14 Dec  1 17:29 ..
+676 -rwxr-xr-x 1 ovs ovs 690176 Dec  3 14:03 tGLpPNHEDxNOOqOZ.db
+  4 -rw-r--r-- 1 ovs ovs    276 Dec  3 14:03 tGLpPNHEDxNOOqOZ.db.wal
+root@cmp02:/opt/OpenvStorage/db/arakoon/ovsdb/tlogs# ls -als
+total 13744
+   0 drwxr-xr-x 1 ovs ovs      58 Dec  3 12:07 .
+   0 drwxr-xr-x 1 ovs ovs      14 Dec  1 17:29 ..
+3732 -rw-r--r-- 1 ovs ovs 3819824 Dec  2 08:32 000.tlx
+3776 -rw-r--r-- 1 ovs ovs 3863492 Dec  2 21:09 001.tlx
+3920 -rw-r--r-- 1 ovs ovs 4013559 Dec  3 12:07 002.tlx
+2316 -rw-r--r-- 1 ovs ovs 2368293 Dec  3 14:03 003.tlog
+```
+
+nodename.db: the actual key/value store.
+nodename.db.wal: security database to keep the key/value store consistent, for example in case of a failure.
+XXX.tlog: The transaction log files, contains the logs for all transactions executed on the node.
+XXX.tlx: A compressed transaction log file.
+
+## Arakoon Log files
+The log files for the different Arakoon databases can be found under `/var/log/syslog` or through `journalctl -u <DB name>....service -n 10000 --no-pager | less`.
+

META

@@ -1,9 +1,8 @@
 # -*- conf -*-
 
 description = "Arakoon client"
-version = "1.8.6"
+version = "1.9.25"
 exists_if = "arakoon_client.cma,arakoon_client.cmxa,arakoon_client.cmxs"
-requires = "threads lwt lwt.unix str oUnit quickcheck lwt.ssl ocplib-endian nocrypto cstruct"
-archive(byte) = "arakoon_client.cma"
-archive(native) = "arakoon_client.cmxa"
-
+requires = "threads lwt lwt.unix str oUnit quickcheck lwt_ssl ocplib-endian redis-lwt uri core"
+archive(byte) = "arakoon_client.cma tcp_keepalive_sockopt_stubs.o"
+archive(native) = "arakoon_client.cmxa tcp_keepalive_sockopt_stubs.o"

Makefile

@@ -1,84 +1,69 @@
 # This makefile wrapper makes DEBIAN packaging easier.
-PREFIX ?=/usr
 
-START = $(DESTDIR)$(PREFIX)
+# The debian installer defined DESTDIR. If set, we install where debian expects it,
+# othervice we install where opam expects it
+ifdef DESTDIR
+  LIBDIR=$(DESTDIR)/usr/lib/ocaml
+  PREFIX=$(DESTDIR)/usr
+else
+  PREFIX=$(shell opam config var prefix)
+  LIBDIR=$(shell opam config var lib)
+endif
 
-OCAML_LIBDIR ?= $(START)/lib/ocaml/
-#OCAML_LIBDIR ?= `ocamlfind printconf destdir`
-OCAML_FIND ?= ocamlfind
+all: arakoon.native bench.native lwt_buffer_test.native
 
-all: build
+build:
+	@dune build
+
+arakoon.native: build
+	@ln -sf _build/default/src/main/arakoon.exe arakoon.native
+
+bench.native:
+	@dune build src/main/bs_bench.exe
+	@ln -sf _build/default/src/main/bs_bench.exe bs_bench.native
+
+lwt_buffer_test.native:
+	@dune build src/main/lwt_buffer_test.exe
+	@ln -sf _build/default/src/main/lwt_buffer_test.exe lwt_buffer_test.native
 
 clean:
-	ocamlbuild -clean
+	dune clean
+	rm -f arakoon.native bs_bench.native lwt_buffer_test.native
 
-build:
-	ocamlbuild -j 4 -use-ocamlfind arakoon.byte arakoon.native arakoon_client.cma arakoon_client.cmxa arakoon_client.a arakoon_client.cmxs plugin_helper.cmi
-
-bench:
-	ocamlbuild -use-ocamlfind bs_bench.native
-
-test:
-	./arakoon.native --run-all-tests
-
-install: install_client install_server man
-
-install_server:
-	mkdir -p $(START)/bin/
-	cp ./arakoon.native $(START)/bin/arakoon
-
-install_client:
-	mkdir -p $(OCAML_LIBDIR)
-	$(OCAML_FIND) install arakoon_client -destdir $(OCAML_LIBDIR) META \
-	  _build/src/arakoon_client.cma \
-	  _build/src/arakoon_client.cmxa \
-	  _build/src/arakoon_client.cmxs \
-	  _build/src/client/stamp.cmi \
-	  _build/src/client/arakoon_exc.mli \
-	  _build/src/client/arakoon_exc.cmi \
-          _build/src/client/arakoon_client_config.cmi \
-	  _build/src/client/arakoon_client.mli \
-	  _build/src/client/arakoon_client.cmi \
-	  _build/src/client/arakoon_remote_client.mli \
-	  _build/src/client/arakoon_remote_client.cmi \
-          _build/src/client/protocol_common.cmi \
-          _build/src/client/client_helper.cmi \
-          _build/src/inifiles/inifiles.mli \
-          _build/src/inifiles/inifiles.cmi \
-          _build/src/lib/std.cmi \
-          _build/src/plugins/registry.mli \
-          _build/src/plugins/registry.cmi \
-          _build/src/plugins/plugin_helper.mli \
-          _build/src/plugins/plugin_helper.cmi \
-          _build/src/node/key.mli \
-          _build/src/node/key.cmi \
-          _build/src/node/simple_store.cmi \
-          _build/src/nursery/routing.cmi \
-          _build/src/tlog/interval.cmi \
-          _build/src/tlog/update.cmi \
-          _build/src/tools/ini.cmi \
-          _build/src/tools/llio.mli \
-          _build/src/tools/llio.cmi \
-          _build/src/tools/network.cmi \
-          _build/src/tools/lwt_extra.cmi \
-          _build/src/tools/typed_ssl.mli \
-          _build/src/tools/typed_ssl.cmi \
-          _build/src/arakoon_client.a
-
-uninstall_client:
-	$(OCAML_FIND) remove arakoon_client -destdir $(OCAML_LIBDIR)
-
-coverage:
-	ocamlbuild -use-ocamlfind \
-	-tag use_bisect \
-	arakoon.d.byte
+test:	build
+	dune exec -- arakoon test
+
+install: build
+	@mkdir -p $(LIBDIR) $(PREFIX)
+ifdef DESTDIR
+	mkdir -p $(DESTDIR)/usr/lib
+	ldd arakoon.native | \
+          grep librocksdb | \
+          awk '/=> /{print $$3}' | \
+          xargs cp --target-directory $(DESTDIR)/usr/lib
+endif
+	dune install --libdir=$(LIBDIR) --prefix=$(PREFIX)
+
+uninstall: build
+	dune uninstall --libdir=$(LIBDIR) --prefix=$(PREFIX)
+
+uninstall_client: build
+	dune uninstall -p arakoon_client --libdir=$(LIBDIR) --prefix=$(PREFIX)
 
 man:
 	ln -s ./arakoon.native arakoon
 	help2man --name='Arakoon, a consistent key value store' ./arakoon > debian/arakoon.man
 	rm arakoon
 
-.PHONY: install test build install_client bench
+examples:
+	dune build examples/ocaml/plugin_demo.exe examples/ocaml/demo.exe
+
+doc:
+	dune build @doc
+
+
+.PHONY: install uninstall test build bench examples arakoon.native doc man
+
 
 
 indent-tabs-to-spaces:

README.md

@@ -1,6 +1,6 @@
 Arakoon
 =======
-![](https://travis-ci.org/Incubaid/arakoon.svg?branch=1.8)
+[![](https://travis-ci.org/openvstorage/arakoon.svg?branch=1.9)](https://travis-ci.org/openvstorage/arakoon)
 
 Arakoon is a distributed key/value store with preference for consistency. More technically, it's a [Multi-Paxos](http://en.wikipedia.org/wiki/Paxos_%28computer_science%29#Multi-Paxos) implementation (written in [OCaml](http://ocaml.org/)) running on top of [TokyoCabinet](http://fallabs.com/tokyocabinet/).
 
@@ -51,9 +51,15 @@ An overview of all command-line options is available by running `arakoon.native
 
 Of course you want to use Arakoon from within a program. We provide OCaml, C and Python clients.
 
+Documentation
+--------------
+For additional information please visit the [Arakoon GitBook](https://openvstorage.gitbooks.io/arakoon).
+
+
 Releases
 --------
-Arakoon releases are available from the [releases](https://github.com/Incubaid/arakoon/releases) page. Any `1.X` branch corresponds to the development of releases in the `1.X` series.
+
+Arakoon releases are available from the [releases](https://github.com/openvstorage/arakoon/releases) page. Any `1.X` branch corresponds to the development of releases in the `1.X` series.
 
 Have fun,
 

SUMMARY.md

@@ -0,0 +1,6 @@
+# Summary
+
+* [Arakoon Troubleshooting](docs/troubleshooting.md)
+    * [Retrieve Arakoon masters](docs/retrieveArakoonMasters.md)
+    * [Unable to start Arakoon service due to discrepancy between Alba and Arakoon version](docs/unableToStartArakoonServiceDueToDiscrepancyBetweenAlbaAndArakoonVersion.md)
+    * [Collapse arakoon](docs/collapseArakoon.md)

_tags

@@ -2,20 +2,21 @@ true: debug
 true: annot
 true: thread
 true: package(lwt)
-true: package(lwt.preemptive)
 true: package(lwt.unix)
-true: package(lwt.ssl)
+true: package(lwt_ssl)
 true: package(oUnit)
 true: package(bz2)
 true: package(str)
 true: package(camltc)
 true: package(dynlink)
-true: package(quickcheck)
 true: pp(camlp4of)
+true: package(lwt_log)
+true: package(uri)
+true: package(redis-lwt)
+true: package(core)
 true: package(snappy)
 true: package(ocplib-endian)
-true: package(uuidm)
-true: package(nocrypto)
+
 "src":include
 "src/lib":include
 "src/tools":include
@@ -29,12 +30,15 @@ true: package(nocrypto)
 "src/inifiles": include
 "src/nursery": include
 "src/plugins": include
-"examples/ocaml": include
+#"examples/ocaml": include
 "src/main" : include
 <**/arakoon.*{byte,native}>: is_main, linkall
 <**/nursery.*{byte,native}>: is_main
 <**/profile.*{byte,native}>: is_main, package(ocamlviz)
 <**/bs_bench.*{byte,native}>: is_main
-<examples/ocaml/*.{byte,native}>: is_main
+#<examples/ocaml/*.{byte,native}>: is_main
 "logger_macro.cmo" : use_camlp4, camlp4orf
 <src/**/*.ml> : camlp4of, use_log_macro, maybe_use_bisect, unsafe_string
+"rpmbuild": -traverse
+"arakoon_docker_rpm": -traverse
+"arakoon_version.ml": opaque