From 849e2bb78202b997b58eeb91dc8e5aa0fa3a2018 Mon Sep 17 00:00:00 2001
From: Etienne Menguy <etienne.menguy@croit.io>
Date: Sat, 3 Jul 2021 20:58:07 +0200
Subject: [PATCH 1/5] Initial commit to add rbd and nomad integration
 documentation

Signed-off-by: Etienne Menguy <etienne.menguy@croit.io>
---
 .organizationmap      |   2 +-
 doc/rbd/rbd-nomad.rst | 431 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 432 insertions(+), 1 deletion(-)
 create mode 100644 doc/rbd/rbd-nomad.rst

diff --git a/.organizationmap b/.organizationmap
index 0169f117d032d..590f38f5de4fa 100644
--- a/.organizationmap
+++ b/.organizationmap
@@ -141,6 +141,7 @@ Corvisa LLC <contact@corvisa.com> Walter Huf <hufman@gmail.com>
 Credit Mutuel Arkea <contact@arkea.com> Eric Mourgaya <eric.mourgaya@arkea.com>
 Croit <contact@croit.io> Fabian Bonk <fabian.bonk@croit.io>
 Croit <contact@croit.io> Paul Emmerich <paul.emmerich@croit.io>
+Croit <contact@croit.io> Etienne Menguy <etienne.menguy@croit.io>
 CypressXt Networking <cypressxt.net> Clément Hampaï <clement.hampai@cypressxt.net>
 Datatom <contact@datatom.com> Wang Yong <wang.yong@datatom.com>
 Day By Day <contact@daybyday.nl> Gerben Meijer <gerben@daybyday.nl>
@@ -486,7 +487,6 @@ OpenSUSE <contact@opensuse.org> Richard Brown <RBrownCCB@opensuse.org>
 Opower <contact@opower.com> Derrick Schneider <derrick.schneider@opower.com>
 Oracle <contact@oracle.com> Nikita Gerasimov <nikita.gerasimov@oracle.com>
 OVH <ceph@ovh.net> Bartłomiej Święcki <bartlomiej.swiecki@corp.ovh.com>
-OVH <ceph@ovh.net> Etienne Menguy <etienne.menguy@corp.ovh.com>
 OVH <ceph@ovh.net> Paweł Sadowski <pawel.sadowski@corp.ovh.com>
 OVH <ceph@ovh.net> Piotr Dałek <piotr.dalek@corp.ovh.com>
 Pacific Northwest National Laboratory <contact@pnl.gov> Brown, David M JR <david.brown@pnl.gov>
diff --git a/doc/rbd/rbd-nomad.rst b/doc/rbd/rbd-nomad.rst
new file mode 100644
index 0000000000000..a5cf4e245cf37
--- /dev/null
+++ b/doc/rbd/rbd-nomad.rst
@@ -0,0 +1,431 @@
+==============================
+ Block Devices and Nomad
+==============================
+
+Like Kubernetes, Nomad can use Ceph Block Device thanks to `ceph-csi`_, 
+which allow to dinamacially provision RBD images or import existing one.
+
+Every nomad version can use `ceph-csi`_, however we'll here describe the
+latest version available at writing time, Nomad v1.1.2 .
+
+To use Ceph Block Devices with Nomad, you must install
+and configure ``ceph-csi`` within your Nomad environment. The following
+diagram depicts the Nomad/Ceph technology stack.
+
+.. ditaa::
+            +-------------------------+-------------------------+
+            |      Container          |          ceph--csi      |
+            |                         |            node         |
+            |          ^              |                 ^       |
+            |          |              |                 |       |
+            +----------+--------------+-------------------------+
+            |          |                                |       |
+            |          v                                |       |
+            |                       Nomad               |       |
+            |                                           |       |
+            +---------------------------------------------------+
+            |                       ceph--csi                   |
+            |                       controller                  |
+            +--------+------------------------------------------+
+                     |                                  |
+                     | configures       maps            |
+                     +---------------+ +----------------+
+                                     | | 
+                                     v v
+            +------------------------+ +------------------------+
+            |                        | |        rbd--nbd        |
+            |     Kernel Modules     | +------------------------+
+            |                        | |         librbd         |
+            +------------------------+-+------------------------+
+            |                   RADOS Protocol                  |
+            +------------------------+-+------------------------+
+            |          OSDs          | |        Monitors        |
+            +------------------------+ +------------------------+
+
+.. note::
+    Nomad has many task drivers, but we'll only use a Docker container in this example.
+
+.. important::
+   ``ceph-csi`` uses the RBD kernel modules by default which may not support all
+   Ceph `CRUSH tunables`_ or `RBD image features`_.
+
+Create a Pool
+=============
+
+By default, Ceph block devices use the ``rbd`` pool. Create a pool for
+Nopmad persistent storage. Ensure your Ceph cluster is running, then create
+the pool. ::
+
+        $ ceph osd pool create nomad
+
+See `Create a Pool`_ for details on specifying the number of placement groups
+for your pools, and `Placement Groups`_ for details on the number of placement
+groups you should set for your pools.
+
+A newly created pool must be initialized prior to use. Use the ``rbd`` tool
+to initialize the pool::
+
+        $ rbd pool init nomad
+
+Configure ceph-csi
+==================
+
+Setup Ceph Client Authentication
+--------------------------------
+
+Create a new user for nomad and `ceph-csi`. Execute the following and
+record the generated key::
+
+    $ ceph auth get-or-create client.nomad mon 'profile rbd' osd 'profile rbd pool=nomad' mgr 'profile rbd pool=nomad'
+    [client.nomad]
+        key = AQAlh9Rgg2vrDxAARy25T7KHabs6iskSHpAEAQ==
+
+
+Configure Nomad  
+---------------
+
+By default Nomad doesn't allow containers to use privileged mode.
+Edit the nomad configuration file by adding this configuration block to `/etc/nomad.d/nomad.hcl`::
+
+    plugin "docker" {
+        config {
+        allow_privileged = true
+        }
+    }
+
+
+Nomad must have `rbd` module loaded, check if it's the case.::
+
+        $ lsmod |grep rbd
+        rbd                    94208  2
+        libceph               364544  1 rbd
+
+If it's not the case, load it.::
+
+        $ modprobe rbd
+
+And restart Nomad.
+
+
+
+Create ceph-csi controller and plugin nodes
+===========================================
+
+The `ceph-csi`_ plugin requieres two components:
+
+- **Controller plugin**: Communicates with the provider's API.
+- **Node plugin**: execute tasks on the client.
+
+.. note::
+    We'll set the ceph-csi's version in those files see `ceph-csi release`_ for other versions.
+
+Configure controller plugin
+---------------------------
+
+The controller plugin requires Cpeh monitor addresses of for the Ceph cluster.
+Collect both the Ceph cluster unique `fsid` and the monitor addresses::
+
+        $ ceph mon dump
+        <...>
+        fsid b9127830-b0cc-4e34-aa47-9d1a2e9949a8
+        <...>
+        0: [v2:192.168.1.1:3300/0,v1:192.168.1.1:6789/0] mon.a
+        1: [v2:192.168.1.2:3300/0,v1:192.168.1.2:6789/0] mon.b
+        2: [v2:192.168.1.3:3300/0,v1:192.168.1.3:6789/0] mon.c
+
+Generate a `ceph-csi-plugin-controller.nomad` file similar to the example below, substituting
+the `fsid` for "clusterID", and the monitor addresses for "monitors"::
+
+
+        job "ceph-csi-plugin-controller" {
+          datacenters = ["dc1"]
+        group "controller" {
+            network {
+              port "metrics" {}
+            }
+            task "ceph-controller" {
+        template {
+                data        = <<EOF
+        [{
+            "clusterID": "b9127830-b0cc-4e34-aa47-9d1a2e9949a8",
+            "monitors": [
+                "192.168.1.1",
+          "192.168.1.2",
+          "192.168.1.3"
+            ]
+        }]
+        EOF
+                destination = "local/config.json"
+                change_mode = "restart"
+              }
+              driver = "docker"
+              config {
+                image = "quay.io/cephcsi/cephcsi:v3.3.1"
+                volumes = [
+                  "./local/config.json:/etc/ceph-csi-config/config.json"
+                ]
+                mounts = [
+                  {
+                    type     = "tmpfs"
+                    target   = "/tmp/csi/keys"
+                    readonly = false
+                    tmpfs_options = {
+                      size = 1000000 # size in bytes
+                    }
+                  }
+                ]
+                args = [
+                  "--type=rbd",
+                  "--controllerserver=true",
+                  "--drivername=rbd.csi.ceph.com",
+                  "--endpoint=unix://csi/csi.sock",
+                  "--nodeid=${node.unique.name}",
+            "--instanceid=${node.unique.name}-controller",
+                  "--pidlimit=-1",
+            "--logtostderr=true",
+                  "--v=5",
+                  "--metricsport=$${NOMAD_PORT_metrics}"
+                ]
+              }
+           resources {
+                cpu    = 500
+                memory = 256
+              }
+              service {
+                name = "ceph-csi-controller"
+                port = "metrics"
+                tags = [ "prometheus" ]
+              }
+        csi_plugin {
+                id        = "ceph-csi"
+                type      = "controller"
+                mount_dir = "/csi"
+              }
+            }
+          }
+        }
+
+Configure plugin node
+---------------------
+Generate a `ceph-csi-plugin-node.nomad` file similar to the example below, substituting
+the `fsid` for "clusterID", and the monitor addresses for "monitors"::
+
+
+        job "ceph-csi-plugin-nodes" {
+          datacenters = ["dc1"]
+          type        = "system"
+          group "nodes" {
+            network {
+              port "metrics" {}
+            }
+        
+            task "ceph-node" {
+              driver = "docker"
+              template {
+                data        = <<EOF
+        [{
+            "clusterID": "b9127830-b0cc-4e34-aa47-9d1a2e9949a8",
+            "monitors": [
+                "192.168.1.1",
+          "192.168.1.2",
+          "192.168.1.3"
+            ]
+        }]
+        EOF
+                destination = "local/config.json"
+                change_mode = "restart"
+              }
+              config {
+                image = "quay.io/cephcsi/cephcsi:v3.3.1"
+                volumes = [
+                  "./local/config.json:/etc/ceph-csi-config/config.json"
+                ]
+                mounts = [
+                  {
+                    type     = "tmpfs"
+                    target   = "/tmp/csi/keys"
+                    readonly = false
+                    tmpfs_options = {
+                      size = 1000000 # size in bytes
+                    }
+                  }
+                ]
+                args = [
+                  "--type=rbd",
+                  "--drivername=rbd.csi.ceph.com",
+                  "--nodeserver=true",
+                  "--endpoint=unix://csi/csi.sock",
+                  "--nodeid=${node.unique.name}",
+                  "--instanceid=${node.unique.name}-nodes",
+                  "--pidlimit=-1",
+            "--logtostderr=true",
+                  "--v=5",
+                  "--metricsport=$${NOMAD_PORT_metrics}"
+                ]
+                privileged = true
+              }
+           resources {
+                cpu    = 500
+                memory = 256
+              }
+              service {
+                name = "ceph-csi-nodes"
+                port = "metrics"
+                tags = [ "prometheus" ]
+              }
+        csi_plugin {
+                id        = "ceph-csi"
+                type      = "node"
+                mount_dir = "/csi"
+              }
+            }
+          }
+        }
+
+Start plugin controller and node
+--------------------------------
+Run::
+
+        nomad job run ceph-csi-plugin-controller.nomad
+        nomad job run ceph-csi-plugin-nodes.nomad
+
+`ceph-csi`_ image will be downloaded, after few minutes check plugin status::
+
+        $ nomad plugin status ceph-csi
+        ID                   = ceph-csi
+        Provider             = rbd.csi.ceph.com
+        Version              = 3.3.1
+        Controllers Healthy  = 1
+        Controllers Expected = 1
+        Nodes Healthy        = 1
+        Nodes Expected       = 1
+
+        Allocations
+        ID        Node ID   Task Group  Version  Desired  Status   Created    Modified
+        23b4db0c  a61ef171  nodes       4        run      running  3h26m ago  3h25m ago
+        fee74115  a61ef171  controller  6        run      running  3h26m ago  3h25m ago
+
+Using Ceph Block Devices
+========================
+
+Create rbd image
+----------------
+
+`ceph-csi` requires the cephx credentials for communicating with the Ceph
+cluster. Generate a `ceph-volume.hcl` file similar to the example below,
+using the newly created nomad user id and cephx key::
+
+        id = "ceph-mysql"
+        name = "ceph-mysql"
+        type = "csi"
+        plugin_id = "ceph-csi"
+        capacity_max = "200G"
+        capacity_min = "100G"
+
+        capability {
+          access_mode     = "single-node-writer"
+          attachment_mode = "file-system"
+        }
+
+        secrets {
+          userID  = "admin"
+          userKey = "AQAlh9Rgg2vrDxAARy25T7KHabs6iskSHpAEAQ=="
+        }
+
+        parameters {
+          clusterID = "b9127830-b0cc-4e34-aa47-9d1a2e9949a8"
+          pool = "nomad"
+          imageFeatures = "layering"
+        }
+
+Once generated, create the volume::
+
+        $ nomad volume create ceph-volume.hcl
+
+Use rbd image with a container
+------------------------------
+
+As example we'll modify Hashicorp learn `nomad sateful`_ example 
+
+Generate a mysql.nomad file similar to the example below.::
+
+        job "mysql-server" {
+          datacenters = ["dc1"]
+          type        = "service"
+          group "mysql-server" {
+            count = 1
+            volume "ceph-mysql" {
+              type      = "csi"
+                attachment_mode = "file-system"
+                access_mode     = "single-node-writer"
+              read_only = false
+              source    = "ceph-mysql"
+            }
+            network {
+              port "db" {
+                static = 3306
+              }
+            }
+            restart {
+              attempts = 10
+              interval = "5m"
+              delay    = "25s"
+              mode     = "delay"
+            }
+            task "mysql-server" {
+              driver = "docker"
+              volume_mount {
+                volume      = "ceph-mysql"
+                destination = "/srv"
+                read_only   = false
+              }
+              env {
+                MYSQL_ROOT_PASSWORD = "password"
+              }
+              config {
+                image = "hashicorp/mysql-portworx-demo:latest"
+                args  = ["--datadir", "/srv/mysql"]
+                ports = ["db"]
+              }
+              resources {
+                cpu    = 500
+                memory = 1024
+              }
+              service {
+                name = "mysql-server"
+                port = "db"
+                check {
+                  type     = "tcp"
+                  interval = "10s"
+                  timeout  = "2s"
+                }
+              }
+            }
+          }
+        }
+
+Start the job::
+
+        $ nomad job run mysql.nomad
+
+Check job's status::
+
+        nomad job status mysql-server
+        ...
+        Status        = running
+        ...
+        Allocations
+        ID        Node ID   Task Group    Version  Desired  Status   Created  Modified
+        38070da7  9ad01c63  mysql-server  0        run      running  6s ago   3s ago
+
+To check data are actually persistant, you can modify database, purge the job then create it using the same file.
+It will reuse the same RBD image.
+
+.. _ceph-csi: https://github.com/ceph/ceph-csi/
+.. _csi: https://www.nomadproject.io/docs/internals/plugins/csi
+.. _Create a Pool: ../../rados/operations/pools#createpool
+.. _Placement Groups: ../../rados/operations/placement-groups
+.. _CRUSH tunables: ../../rados/operations/crush-map/#tunables
+.. _RBD image features: ../rbd-config-ref/#image-features
+.. _nomad sateful: https://learn.hashicorp.com/tutorials/nomad/stateful-workloads-csi-volumes?in=nomad/stateful-workloads#create-the-job-file
+.. _ceph-csi release: https://github.com/ceph/ceph-csi#ceph-csi-container-images-and-release-compatibility
\ No newline at end of file

From 4f4110b980036648fa881b90a04c95bf07b1b94a Mon Sep 17 00:00:00 2001
From: Etienne Menguy <etienne.menguy@croit.io>
Date: Sat, 3 Jul 2021 22:36:14 +0200
Subject: [PATCH 2/5] Adding rbd-nomad to the index

Signed-off-by: Etienne Menguy <etienne.menguy@croit.io>
---
 doc/rbd/rbd-integrations.rst | 1 +
 1 file changed, 1 insertion(+)

diff --git a/doc/rbd/rbd-integrations.rst b/doc/rbd/rbd-integrations.rst
index 50d788d53973b..f55604a6fcf7c 100644
--- a/doc/rbd/rbd-integrations.rst
+++ b/doc/rbd/rbd-integrations.rst
@@ -9,6 +9,7 @@
    QEMU <qemu-rbd>
    libvirt <libvirt>
    Kubernetes <rbd-kubernetes>
+   Nomad <rbd-nomad>
    OpenStack <rbd-openstack>
    CloudStack <rbd-cloudstack>
    LIO iSCSI Gateway <iscsi-overview>

From 0a2a461a0c5c5d8ffb51e578b782fe8bb66da7b2 Mon Sep 17 00:00:00 2001
From: Etienne Menguy <etienne.menguy@croit.io>
Date: Wed, 7 Jul 2021 10:13:22 +0200
Subject: [PATCH 3/5] Fixing typo

Signed-off-by: Etienne Menguy <etienne.menguy@croit.io>
---
 doc/rbd/rbd-nomad.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/doc/rbd/rbd-nomad.rst b/doc/rbd/rbd-nomad.rst
index a5cf4e245cf37..1c799d79282d4 100644
--- a/doc/rbd/rbd-nomad.rst
+++ b/doc/rbd/rbd-nomad.rst
@@ -3,7 +3,7 @@
 ==============================
 
 Like Kubernetes, Nomad can use Ceph Block Device thanks to `ceph-csi`_, 
-which allow to dinamacially provision RBD images or import existing one.
+which allow to dynamically provision RBD images or import existing one.
 
 Every nomad version can use `ceph-csi`_, however we'll here describe the
 latest version available at writing time, Nomad v1.1.2 .

From a335ee96ab4ee621e7e4d47d65ddf9c27c4a5521 Mon Sep 17 00:00:00 2001
From: Etienne Menguy <etienne.menguy@croit.io>
Date: Wed, 7 Jul 2021 10:14:12 +0200
Subject: [PATCH 4/5] Adding missing sudo

Signed-off-by: Etienne Menguy <etienne.menguy@croit.io>
---
 doc/rbd/rbd-nomad.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/doc/rbd/rbd-nomad.rst b/doc/rbd/rbd-nomad.rst
index 1c799d79282d4..7e8d0c1535382 100644
--- a/doc/rbd/rbd-nomad.rst
+++ b/doc/rbd/rbd-nomad.rst
@@ -102,7 +102,7 @@ Nomad must have `rbd` module loaded, check if it's the case.::
 
 If it's not the case, load it.::
 
-        $ modprobe rbd
+        $ sudo modprobe rbd
 
 And restart Nomad.
 

From c20a37698d89343c358fd7d733a9ed1c61a44f74 Mon Sep 17 00:00:00 2001
From: Etienne Menguy <etienne.menguy@croit.io>
Date: Wed, 7 Jul 2021 10:26:21 +0200
Subject: [PATCH 5/5] Improving syntax

Signed-off-by: Etienne Menguy <etienne.menguy@croit.io>
---
 doc/rbd/rbd-nomad.rst | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/doc/rbd/rbd-nomad.rst b/doc/rbd/rbd-nomad.rst
index 7e8d0c1535382..427b2c6fa1146 100644
--- a/doc/rbd/rbd-nomad.rst
+++ b/doc/rbd/rbd-nomad.rst
@@ -1,6 +1,6 @@
-==============================
+=========================
  Block Devices and Nomad
-==============================
+=========================
 
 Like Kubernetes, Nomad can use Ceph Block Device thanks to `ceph-csi`_, 
 which allow to dynamically provision RBD images or import existing one.