s3 csi for a nomad cluster

[Frontrider]

I'm looking for a CSI solution for a nomad cluster, that would use minio to provision the storage.

Can I use this image ctrox/csi-s3:v1.2.0-rc.2, configure it to connect to my minio, then bind to the socket? I don't expect full support for this, just want to ask if it's reasonable before jumping into it. This is the most maintained solution I could find so far, but kubernetes is not an option for what I'm doing here.

[ctrox]

The driver implements the csi spec 1.0, so as long as nomad supports that it should work. I have not tried it, so I would just give it a shot :)

[Frontrider]

I asked, because a lot of the csi drivers out there specifically use kubernetes services. So it's not one of those, then that makes a try viable.

[ctrox]

No services are in use here, so that should not be a problem.

[Traviis]

Hello!

I was able to get this to work in Nomad with the following setup:

• Nomad 1.2.6
• csi-s3:v1.2.9-rc.1 (rc.2 would have inconsistent mount behavior I never could figure out)
• rclone mounter (goofys worked but had a massive memory issue and it oomkillered itself out, didn't try s3fs)
• Minio 2022-03-05T06:32:39Z

Other than setting up the plugin and the volumes, I did have to do one special thing to make this work, which was to manually create the .metadata.json file in the root of the existing bucket. It looked like this:

{
   "Name": "test",
   "Prefix": "",
   "UsePrefix": false,
   "Mounter": "rclone",
   "FSPath": "",
   "CapacityBytes": 0
}

Without this file existing, it would fail to mount and complain about this:

failed to setup alloc: pre-run hook "csi_hook" failed: rpc error: code = Unknown desc = The specified key does not exist.

I looked very briefly at the code, and it looks like it would be fine, so I'm a bit confused. It also shows up a bit strange, telling me I have petabytes of storage available, likely due to that capacity bytes bit there.

Here are the following snippets to get everything up and running:

plugin-s3.nomad:

job "plugin-s3" {
  datacenters = ["dc1"]

  # you can run node plugins as service jobs as well, but running
  # as a system job ensures all nodes in the DC have a copy.
  type = "system"

  # only one plugin of a given type and ID should be deployed on
  # any given client node
  constraint {
    operator = "distinct_hosts"
    value = true
  }

  group "nodes" {
    task "plugin" {
      driver = "docker"

    // if not specified, it defaults to 300MB of memory for the mount. Most of the mounters use memory to cache things, for example, rclone is 15 MB (default) per file open, if you attempt to open a larger set of files, you get oomkilled.
    resources {
        memory = 3000
    }

      config {
          //The packaged version of goofys in rc.2 appears to not work?
        image = "ctrox/csi-s3:v1.2.0-rc.1"

        args = [
            "--endpoint=unix:///csi/csi.sock",
            "--nodeid=${node.unique.name}",
            "--v=4",
        ]


        # all CSI node plugins will need to run as privileged tasks
        # so they can mount volumes to the host. controller plugins
        # do not need to be privileged.
        privileged = true
      }

      csi_plugin {
        id        = "s3"
        type      = "node"
        mount_dir = "/csi"  # this path /csi matches the --endpoint
                            # argument for the container
      }
    }
  }
}

Run like a normal job: nomad run plugin-s3.nomad.

test-vol.nomad (s3 bucket existed and was called test; it must match up with external_id)

id = "test-vol"
name = "test-vol"
type = "csi"
external_id = "test"
plugin_id = "s3"

capability {
  access_mode     = "single-node-writer"
  attachment_mode = "file-system"
}

secrets {
  accessKeyID = "<ACCESS_KEY>"
  secretAccessKey = "<SECRET_ACCESS_KEY>"
  endpoint        = "<MINIO_INSTANCE>"
}

parameters {
  mounter = "rclone"
  bucket = "test"
}

Import with nomad volume register test-vol.nomad.

A job mounting the volume:

job "test_job" {
  type = "service"

  datacenters = ["dc1"]


  group "test" {
    restart {
      attempts = 10
      delay    = "30s"
    }

    count = 1

    volume "testvol" {
      type            = "csi"
      source          = "test-vol"
      attachment_mode = "file-system"
      access_mode     = "single-node-writer"
    }

    task "csi_test" {
      driver = "docker"



      volume_mount {
        volume      = "testvol"
        destination = "/data"
      }


      config {
        image = "karlherler/pause"
      }

    }
  }
}

Certainly not a complete solution, and I found a lot of threads I didn't take the time to pull, but it should at least get you headed in the right direction.

EDIT: For clarity and reference about resource limits.

[Frontrider]

Leave it here for reference, that project ended up going in a different direction.

[nottech-gmbh]

I have thes issues
failed to setup alloc: pre-run hook "csi_hook" failed: rpc error: code = Unknown desc = The specified bucket does not exist

[Suselz]

For everyone who found this issue while searching for their question.

Services used in the example:

• Nomad v1.4.2
• Minio 2023-03-13T19:46:17Z
• ctrox/csi-s3:v1.2.0-rc.2

Articles

General information

Description of the CSI concept in Nomad- https://developer.hashicorp.com/nomad/docs/concepts/plugins/csi
csi_plugin block in Nomad job - https://developer.hashicorp.com/nomad/docs/job-specification/csi_plugin
Official example of using CSI in Nomad - https://developer.hashicorp.com/nomad/tutorials/stateful-workloads/stateful-workloads-csi-volumes
Another example of using CSI in Nomad - https://juicefs.com/docs/csi/csi-in-nomad/

ctrox/csi-s3 plugin

Dockerhub - https://hub.docker.com/r/ctrox/csi-s3
Github - https://github.com/ctrox/csi-s3

Information

To create an s3fs volume, that is, a volume that stores its data in s3 storage, the csi mechanism is used. In this case, the implementation occurs with the help of plugins.
Plugins in Nomad come as containers that need to be run on a node. In this case, it's a container ctrox/csi-s3
This container creates a unix socket through which other containers/services interact with it.
When you use csi plugin, it can work in one of three modes:

• monolith
• node
• controller

Controller Plugins communicate with the storage provider's APIs.
Node Plugins do the work on each client node, like creating mount points.
Monolith Plugins are plugins that perform both the controller and node roles in the same instance.

If you want use plugin ctrox/csi-s3 you should create node and controller jobs, or use one monolith job.
After creation job you should create new volume, or register exist. After that you can use external s3 volume in job.

Implementation

Configuration of Nomad

Block needs to be added:

plugin "docker" {
  config {
    allow_privileged = true
  }
  volumes {
    enabled = true
  }
}

Plugins

Controller

job "plugin-s3-controller" {
  datacenters = ["dc1"]

  # you can run node plugins as service jobs as well, but running
  # as a system job ensures all nodes in the DC have a copy.
  type = "system"

  # only one plugin of a given type and ID should be deployed on
  # any given client node
  constraint {
    operator = "distinct_hosts"
    value    = true
  }

  group "controllers" {
    task "plugin" {
      driver = "docker"

      // if not specified, it defaults to 300MB of memory for the mount. Most of the mounters use memory to cache things, for example, rclone is 15 MB (default) per file open, if you attempt to open a larger set of files, you get oomkilled.
      resources {
        memory = 300
      }

      config {
        //The packaged version of goofys in rc.2 appears to not work?
        image = "ctrox/csi-s3:v1.2.0-rc.2"

        args = [
          "--endpoint=unix://csi/csi.sock",
          "--nodeid=${node.unique.name}",
          "--logtostderr",
          "--v=5",
        ]


        # all CSI node plugins will need to run as privileged tasks
        # so they can mount volumes to the host. controller plugins
        # do not need to be privileged.
        privileged = true
      }

      csi_plugin {
        id        = "s3"
        type      = "controller"
        mount_dir = "/csi" # this path /csi matches the --endpoint
                           # argument for the container
        stage_publish_base_dir = "/local/csi"
      }
    }
  }
}

Monolith

job "plugin-s3-monolith" {
  datacenters = ["dc1"]

  # you can run node plugins as service jobs as well, but running
  # as a system job ensures all nodes in the DC have a copy.
  type = "system"

  # only one plugin of a given type and ID should be deployed on
  # any given client node
  constraint {
    operator = "distinct_hosts"
    value    = true
  }

  group "monolith" {
    task "plugin" {
      driver = "docker"

      // if not specified, it defaults to 300MB of memory for the mount. Most of the mounters use memory to cache things, for example, rclone is 15 MB (default) per file open, if you attempt to open a larger set of files, you get oomkilled.
      resources {
        memory = 300
      }

      config {
        //The packaged version of goofys in rc.2 appears to not work?
        image = "ctrox/csi-s3:v1.2.0-rc.2"

        args = [
          "--endpoint=unix://csi/csi.sock",
          "--nodeid=${node.unique.name}",
          "--logtostderr",
          "--v=5",
        ]


        # all CSI node plugins will need to run as privileged tasks
        # so they can mount volumes to the host. controller plugins
        # do not need to be privileged.
        privileged = true
      }

      csi_plugin {
        id        = "s3"
        type      = "monolith"
        mount_dir = "/csi" # this path /csi matches the --endpoint
                           # argument for the container
        stage_publish_base_dir = "/local/csi"
      }
    }
  }
}

Node

job "plugin-s3-node" {
  datacenters = ["dc1"]

  # you can run node plugins as service jobs as well, but running
  # as a system job ensures all nodes in the DC have a copy.
  type = "system"

  # only one plugin of a given type and ID should be deployed on
  # any given client node
  constraint {
    operator = "distinct_hosts"
    value    = true
  }

  group "nodes" {
    task "plugin" {
      driver = "docker"

      // if not specified, it defaults to 300MB of memory for the mount. Most of the mounters use memory to cache things, for example, rclone is 15 MB (default) per file open, if you attempt to open a larger set of files, you get oomkilled.
      resources {
        memory = 300
      }

      config {
        //The packaged version of goofys in rc.2 appears to not work?
        image = "ctrox/csi-s3:v1.2.0-rc.2"

        args = [
          "--endpoint=unix://csi/csi.sock",
          "--nodeid=${node.unique.name}",
          "--logtostderr",
          "--v=5",
        ]


        # all CSI node plugins will need to run as privileged tasks
        # so they can mount volumes to the host. controller plugins
        # do not need to be privileged.
        privileged = true
      }

      csi_plugin {
        id        = "s3"
        type      = "node"
        mount_dir = "/csi" # this path /csi matches the --endpoint
                           # argument for the container
        stage_publish_base_dir = "/local/csi"
      }
    }
  }
}

To check the status of the plugin, use the command:
nomad plugin status <plugin_id>
Example of output:

ID                   = s3
Provider             = ch.ctrox.csi.s3-driver
Version              = v1.2.0-rc.2
Controllers Healthy  = 1
Controllers Expected = 0
Nodes Healthy        = 1
Nodes Expected       = 0

Allocations
ID        Node ID   Task Group  Version  Desired  Status   Created    Modified
5705f32d  cf629de5  monolith    1        run      running  1h15m ago  1h15m ago

Volumes

A configuration file is used to create\register a volume.
File example:

id          = "test-vol"
name        = "test-vol"
type        = "csi"

plugin_id   = "s3"
capability {
  access_mode     = "single-node-writer"
  attachment_mode = "file-system"
}
secrets {
  accessKeyID     = "root"
  secretAccessKey = "pa$$w0rd"
  endpoint        = "http://<minio_ip>:9000"
  region          = "" #DO NOT SET WHEN USE MINIO
}
parameters {
  mounter = "s3fs"
}

Further, on the basis of this file, an existing volume is created or registered.
Creation - nomad volume create volume.hcl
After this command, the controller or monolith will try to create a bucket with the same name as the name from the file (in the example test-vol). Bucket is created not empty but with metadata.
Registration - nomad volume register volume.hcl
As a result, an existing bucket will be connected.
Important: since the plugin creates metadata when creating, the bucket created by the same plugin must be used when connecting (not everyone)

Nomad job

For use volume inside a Nomad job, blocks are used:

group "main" {
    count = 1
    volume "<volume_name>" {
      type            = "csi"
      source          = "<volume_id>"
      attachment_mode = "file-system"
      access_mode     = "single-node-writer"
    }
...
    task "taskname" {
      driver = "docker"      
      volume_mount {
        volume      = "<volume_name>"
        destination = "/s3_folder"
        read_only   = false
      }
...
  }
}

Full job:

job "alpine" {
  datacenters = ["dc1"]
  type        = "service"



  group "main" {
    count = 1

    volume "test-vol" {
      type            = "csi"
      source          = "test-vol"
      attachment_mode = "file-system"
      access_mode     = "single-node-writer"
    }

    task "alpine" {
      driver = "docker"

      config {
        image = "alpine:latest"
        args  = ["/bin/sleep", "10000"]

      }
      volume_mount {
        volume      = "test-vol"
        destination = "/s3data"
        read_only   = false
      }


      resources {
        cpu    = 256
        memory = 512
      }
    }
  }
}

Minio

docker-compose.yml

version: "2"

services:
  minio:
    image: quay.io/minio/minio
    #user: "minio:minio"
    command: ["server", "/data", "--console-address", ":9001"]
    volumes:
      - "minio-data:/data"
    environment:
      MINIO_ROOT_USER: "root"
      MINIO_ROOT_PASSWORD: "Password"
      MINIO_REGION: "test"
    ports:
      - "9000:9000"
      - "9001:9001"
    restart: always

[cosmotek]

Started setting this up only to realize ARM64 was not supported. It appears the fuse library used by this package isn't supported on ARM. I tried to build locally and ran into the following error:

$ GOOS=linux GOARCH=arm64 make build
CGO_ENABLED=0 GOOS=linux go build -a -ldflags '-extldflags "-static"' -o _output/s3driver ./cmd/s3driver
go: downloading github.com/container-storage-interface/spec v1.1.0
go: downloading github.com/kubernetes-csi/drivers v1.0.2
go: downloading golang.org/x/net v0.0.0-20211216030914-fe4d6282115f
go: downloading google.golang.org/grpc v1.40.0
go: downloading k8s.io/mount-utils v0.23.3
go: downloading github.com/kahing/goofys v0.24.0
go: downloading github.com/mitchellh/go-ps v0.0.0-20170309133038-4fdf99ab2936
go: downloading k8s.io/utils v0.0.0-20211116205334-6203023598ed
go: downloading github.com/minio/minio-go/v7 v7.0.5
go: downloading github.com/kubernetes-csi/csi-lib-utils v0.6.1
go: downloading k8s.io/klog/v2 v2.30.0
go: downloading github.com/kahing/fusego v0.0.0-20200327063725-ca77844c7bcc
go: downloading github.com/Azure/azure-sdk-for-go v32.1.0+incompatible
go: downloading github.com/Azure/azure-pipeline-go v0.2.1
go: downloading github.com/Azure/azure-storage-blob-go v0.7.1-0.20190724222048-33c102d4ffd2
go: downloading github.com/aws/aws-sdk-go v1.42.44
go: downloading github.com/go-logr/logr v1.2.0
go: downloading google.golang.org/genproto v0.0.0-20220126215142-9970aeb2e350
go: downloading github.com/satori/go.uuid v1.2.1-0.20181028125025-b2ce2384e17b
go: downloading github.com/shirou/gopsutil v2.21.11+incompatible
go: downloading golang.org/x/sys v0.0.0-20210816074244-15123e1e1f71
go: downloading golang.org/x/crypto v0.0.0-20211215153901-e495a2d5b3d3
go: downloading github.com/mattn/go-ieproxy v0.0.0-20190610004146-91bb50d98149
# github.com/ctrox/csi-s3/cmd/s3driver
github.com/jacobsa/fuse/internal/buffer.memclr: relocation target runtime.memclrNoHeapPointers not defined for ABI0 (but is defined for ABIInternal)
make: *** [build] Error 1

[yulinXu]

Hello, can csi-s3 works on arm64 platform, and if have any docs or pkgs?