Merge pull request #33 from ashwanthkumar/docs-setup

First cut stab at documentation

.gitignore

@@ -3,4 +3,5 @@
 target/
 *.ipr
 *.iws
+site/
 

README.md

@@ -58,3 +58,6 @@ Let's break down the above code step by step.
 
 ## Notes
 If you're getting `ClassNotFound` exception, please run `mvn clean compile` once to generate the protoc files.
+
+## License
+https://www.apache.org/licenses/LICENSE-2.0

docs/index.md

@@ -0,0 +1,31 @@
+# Suuchi
+Toolkit to build distributed data systems.
+
+## Library Dependency
+### Maven
+```xml
+<dependency>
+    <groupId>in.ashwanthkumar</groupId>
+    <artifactId>suuchi</artifactId>
+    <version>${suuchi.version}</version>
+</dependency>
+```
+
+Development snapshots are available in [Sonatypes's snapshot repository](https://oss.sonatype.org/content/repositories/snapshots/).
+
+If you are a developer looking to use Suuchi, head over to [Quick Start](quick-start.md) guide to get started.
+
+## Recipes
+- [Distributed InMemory Database](recipes/inmemorydb.md)
+- [Distributed RocksDB backed KV](recipes/rocksdb.md)
+
+## Internals
+We try to document the internal workings of some core pieces of Suuchi for developers interested in contributing or understanding their systems better.
+
+- [Partitioner](internals/partitioner.md)
+- [Replication](internals/replication.md)
+- [Router](internals/router.md)
+
+## License
+[https://www.apache.org/licenses/LICENSE-2.0](https://www.apache.org/licenses/LICENSE-2.0)
+

docs/internals/partitioner.md

@@ -0,0 +1,3 @@
+# Partitioner
+
+TBD

docs/internals/replication.md

@@ -0,0 +1,10 @@
+# Replication Internals
+
+Suuchi out of the box comes with _Synchronous Sequential Replication_ during writes. It's fairly easy to build custom replicators.
+
+Refer [#27](https://github.com/ashwanthkumar/suuchi/pull/27) and [#23](https://github.com/ashwanthkumar/suuchi/pull/23) on how Replication is implemented.
+
+## Types
+- Synchronous SequentialReplication (default available)
+- Synchronous ParallelReplication - ([#30](https://github.com/ashwanthkumar/suuchi/issues/30))
+- Synchronous ChainedReplication - ([#31](https://github.com/ashwanthkumar/suuchi/issues/31))

docs/internals/router.md

@@ -0,0 +1,35 @@
+# HandleOrForward Router
+
+HandleOrForward Router is the entry point of a request in your Suuchi based application. It uses a _RoutingStrategy_ implementation to decide which nodes in the cluster are eligible for handling the current request. It also takes care of forwarding the request to that particular node and returning the response back to the client.
+
+Since there isn't any _SPOC_ (Single Point of Contact) in the system, any node in the cluster can handle or forward any request automatically. This makes the whole operations of the systems very easy. You can setup a load balancer as an entry point to your app
+backed by all the nodes in the cluster.
+
+Refer [#23](https://github.com/ashwanthkumar/suuchi/pull/23), [#11](https://github.com/ashwanthkumar/suuchi/pull/11) and [#2](https://github.com/ashwanthkumar/suuchi/pull/2) on how HandleOrForward Router is implemented. TBD - Explain with pictures on how it works.
+
+## RoutingStrategy
+RoutingStrategy forms the heart of HandleOrForward router. Out of the box Suuchi comes with the following routing strategies
+
+- [ConsistentHashingRouting](https://en.wikipedia.org/wiki/Consistent_hashing)
+
+## Custom Routers
+[_RoutingStrategy_](https://github.com/ashwanthkumar/suuchi/blob/master/src/main/scala/in/ashwanthkumar/suuchi/router/RoutingStrategy.scala#L10) trait is defined as follows
+
+```scala
+trait RoutingStrategy {
+  /**
+   * Decides if the incoming message should be forwarded or handled by the current node.
+   *
+   * @tparam ReqT Type of the input Message
+   * @return  Some(MemberAddress) - if the request is meant to be forwarded
+   *          <p> None - if the request can be handled by the current node
+   */
+  def route[ReqT]: PartialFunction[ReqT, Option[MemberAddress]]
+}
+```
+
+Any implementations of that trait can be passed to HandleOrForward Router.
+
+## Notes
+
+- HandleOrForward Router is implemented internally as a ServerInterceptor. What this means is, when you're handling a streaming request every message that's sent in the stream goes through HandleOrForward backed by a RoutingStrategy to decide which nodes the request should go to.

docs/quick-start.md

@@ -0,0 +1,3 @@
+# Quick Start
+
+TBD

docs/recipes/inmemorydb.md

@@ -0,0 +1,47 @@
+# Distributed InMemory Database
+
+Following code builds a consistent hashing based Get/Put requests backed by an ConcurrentHashMap.
+
+```scala
+package in.ashwanthkumar.suuchi.example
+
+import in.ashwanthkumar.suuchi.router.ConsistentHashingRouting
+import in.ashwanthkumar.suuchi.rpc.Server.whoami
+import in.ashwanthkumar.suuchi.rpc.{Server, SuuchiPutService, SuuchiReadService}
+import in.ashwanthkumar.suuchi.store.InMemoryStore
+import io.grpc.netty.NettyServerBuilder
+
+object ExampleApp extends App {
+  val routingStrategy = ConsistentHashingRouting(2, whoami(5051), whoami(5052), whoami(5053))
+
+  val store1 = new InMemoryStore
+  val server1 = Server(NettyServerBuilder.forPort(5051), whoami(5051))
+    .routeUsing(new SuuchiReadService(store1), routingStrategy)
+    .withReplication(new SuuchiPutService(store1), 2, routingStrategy)
+  server1.start()
+
+  val store2 = new InMemoryStore
+  val server2 = Server(NettyServerBuilder.forPort(5052), whoami(5052))
+    .routeUsing(new SuuchiReadService(store2), routingStrategy)
+    .withReplication(new SuuchiPutService(store2), 2, routingStrategy)
+  server2.start()
+
+  val store3 = new InMemoryStore
+  val server3 = Server(NettyServerBuilder.forPort(5053), whoami(5053))
+    .routeUsing(new SuuchiReadService(store3), routingStrategy)
+    .withReplication(new SuuchiPutService(store3), 2, routingStrategy)
+  server3.start()
+
+  server1.blockUntilShutdown()
+  server2.blockUntilShutdown()
+  server3.blockUntilShutdown()
+}
+```
+
+Let's break down the above code step by step.
+
+1. `ConsistentHashingRouting` is a routing strategy that does routing between all the nodes using a ConsistentHashRing underneath with default vnode factor of 3.
+2. `NettyServerBuilder.forPort(5051)` creates a NettyServer on `5051` port.
+3. `server.routeUsing()` adds a new protob rpc using a routing strategy.
+4. `server1.start()` starts the underlying gRPC server.
+5. `server1.blockUntilShutdown()` waits until the server is stopped.

docs/recipes/rocksdb.md

@@ -0,0 +1,44 @@
+# Distributed RocksDB backed KV
+
+Following code builds a consistent hashing based Get/Put requests backed by an RocksDB.
+
+```scala
+package in.ashwanthkumar.suuchi.example
+
+import java.nio.file.Files
+
+import in.ashwanthkumar.suuchi.router.ConsistentHashingRouting
+import in.ashwanthkumar.suuchi.rpc.Server._
+import in.ashwanthkumar.suuchi.rpc.{Server, SuuchiPutService, SuuchiReadService}
+import in.ashwanthkumar.suuchi.store.rocksdb.{RocksDbConfiguration, RocksDbStore}
+import io.grpc.netty.NettyServerBuilder
+
+object DistributedRocksDb extends App {
+  val routingStrategy = ConsistentHashingRouting(2, whoami(5051), whoami(5052))
+
+  val path1 = Files.createTempDirectory("distributed-rocksdb").toFile
+  val store1 = new RocksDbStore(RocksDbConfiguration(path1.getAbsolutePath))
+  val server1 = Server(NettyServerBuilder.forPort(5051), whoami(5051))
+    .routeUsing(new SuuchiReadService(store1), routingStrategy)
+    .withReplication(new SuuchiPutService(store1), 2, routingStrategy)
+  server1.start()
+
+  val path2 = Files.createTempDirectory("distributed-rocksdb").toFile
+  val store2 = new RocksDbStore(RocksDbConfiguration(path2.getAbsolutePath))
+  val server2 = Server(NettyServerBuilder.forPort(5052), whoami(5052))
+    .routeUsing(new SuuchiReadService(store2), routingStrategy)
+    .withReplication(new SuuchiPutService(store2), 2, routingStrategy)
+  server2.start()
+
+  server1.blockUntilShutdown()
+  server2.blockUntilShutdown()
+
+  /*
+    Optionally if want to delete the rocksdb directory
+      path1.delete()
+      path2.delete()
+  */
+}
+```
+
+TODO - Explain the code.

mkdocs.yml

@@ -0,0 +1,15 @@
+site_name: Suuchi
+repo_url: https://github.com/ashwanthkumar/suuchi/
+site_description: Toolkit to build data systems
+theme: flatly
+
+pages:
+- 'Introduction': 'index.md'
+- 'Quick Start': 'quick-start.md'
+- Internals:
+    - 'Partitioner': 'internals/partitioner.md'
+    - 'Replication': 'internals/replication.md'
+    - 'Router': 'internals/router.md'
+- Recipes:
+    - 'Distributed In Memory Database': 'recipes/inmemorydb.md'
+    - 'Distributed RocksDB Database': 'recipes/rocksdb.md'

src/main/scala/in/ashwanthkumar/suuchi/example/DistributedRocksDb.scala

@@ -9,22 +9,20 @@ import in.ashwanthkumar.suuchi.store.rocksdb.{RocksDbConfiguration, RocksDbStore
 import io.grpc.netty.NettyServerBuilder
 
 object DistributedRocksDb extends App {
-  val path1 = Files.createTempDirectory("distributed-rocksdb").toFile
-  val path2 = Files.createTempDirectory("distributed-rocksdb").toFile
-
-  val store1 = new RocksDbStore(RocksDbConfiguration(path1.getAbsolutePath))
-  val store2 = new RocksDbStore(RocksDbConfiguration(path2.getAbsolutePath))
-
   val routingStrategy = ConsistentHashingRouting(2, whoami(5051), whoami(5052))
 
+  val path1 = Files.createTempDirectory("distributed-rocksdb").toFile
+  val store1 = new RocksDbStore(RocksDbConfiguration(path1.getAbsolutePath))
   val server1 = Server(NettyServerBuilder.forPort(5051), whoami(5051))
     .routeUsing(new SuuchiReadService(store1), routingStrategy)
-    .routeUsing(new SuuchiPutService(store1), routingStrategy)
+    .withReplication(new SuuchiPutService(store1), 2, routingStrategy)
   server1.start()
 
+  val path2 = Files.createTempDirectory("distributed-rocksdb").toFile
+  val store2 = new RocksDbStore(RocksDbConfiguration(path2.getAbsolutePath))
   val server2 = Server(NettyServerBuilder.forPort(5052), whoami(5052))
     .routeUsing(new SuuchiReadService(store2), routingStrategy)
-    .routeUsing(new SuuchiPutService(store2), routingStrategy)
+    .withReplication(new SuuchiPutService(store2), 2, routingStrategy)
   server2.start()
 
   server1.blockUntilShutdown()

src/main/scala/in/ashwanthkumar/suuchi/example/ExampleApp.scala

@@ -7,23 +7,24 @@ import in.ashwanthkumar.suuchi.store.InMemoryStore
 import io.grpc.netty.NettyServerBuilder
 
 object ExampleApp extends App {
-  val store = new InMemoryStore
-
   val routingStrategy = ConsistentHashingRouting(2, whoami(5051), whoami(5052), whoami(5053))
 
+  val store1 = new InMemoryStore
   val server1 = Server(NettyServerBuilder.forPort(5051), whoami(5051))
-    .routeUsing(new SuuchiReadService(store), routingStrategy)
-    .withReplication(new SuuchiPutService(store), 2, routingStrategy)
+    .routeUsing(new SuuchiReadService(store1), routingStrategy)
+    .withReplication(new SuuchiPutService(store1), 2, routingStrategy)
   server1.start()
 
+  val store2 = new InMemoryStore
   val server2 = Server(NettyServerBuilder.forPort(5052), whoami(5052))
-    .routeUsing(new SuuchiReadService(store), routingStrategy)
-    .withReplication(new SuuchiPutService(store), 2, routingStrategy)
+    .routeUsing(new SuuchiReadService(store2), routingStrategy)
+    .withReplication(new SuuchiPutService(store2), 2, routingStrategy)
   server2.start()
 
+  val store3 = new InMemoryStore
   val server3 = Server(NettyServerBuilder.forPort(5053), whoami(5053))
-    .routeUsing(new SuuchiReadService(store), routingStrategy)
-    .withReplication(new SuuchiPutService(store), 2, routingStrategy)
+    .routeUsing(new SuuchiReadService(store3), routingStrategy)
+    .withReplication(new SuuchiPutService(store3), 2, routingStrategy)
   server3.start()
 
   server1.blockUntilShutdown()