-
Notifications
You must be signed in to change notification settings - Fork 172
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
s3 csi for a nomad cluster #64
Comments
The driver implements the csi spec 1.0, so as long as nomad supports that it should work. I have not tried it, so I would just give it a shot :) |
I asked, because a lot of the csi drivers out there specifically use kubernetes services. So it's not one of those, then that makes a try viable. |
No services are in use here, so that should not be a problem. |
Hello! I was able to get this to work in Nomad with the following setup:
Other than setting up the plugin and the volumes, I did have to do one special thing to make this work, which was to manually create the
Without this file existing, it would fail to mount and complain about this:
I looked very briefly at the code, and it looks like it would be fine, so I'm a bit confused. It also shows up a bit strange, telling me I have petabytes of storage available, likely due to that capacity bytes bit there. Here are the following snippets to get everything up and running: plugin-s3.nomad:
Run like a normal job: test-vol.nomad (s3 bucket existed and was called
Import with A job mounting the volume:
Certainly not a complete solution, and I found a lot of threads I didn't take the time to pull, but it should at least get you headed in the right direction. EDIT: For clarity and reference about resource limits. |
Leave it here for reference, that project ended up going in a different direction. |
I have thes issues |
For everyone who found this issue while searching for their question. Services used in the example:
ArticlesGeneral informationDescription of the CSI concept in Nomad- https://developer.hashicorp.com/nomad/docs/concepts/plugins/csi ctrox/csi-s3 pluginDockerhub - https://hub.docker.com/r/ctrox/csi-s3 InformationTo create an s3fs volume, that is, a volume that stores its data in s3 storage, the csi mechanism is used. In this case, the implementation occurs with the help of plugins.
If you want use plugin ctrox/csi-s3 you should create node and controller jobs, or use one monolith job. ImplementationConfiguration of NomadBlock needs to be added:
PluginsController
Monolith
Node
To check the status of the plugin, use the command:
VolumesA configuration file is used to create\register a volume.
Further, on the basis of this file, an existing volume is created or registered. Nomad jobFor use volume inside a Nomad job, blocks are used:
Full job:
Miniodocker-compose.yml
|
Started setting this up only to realize ARM64 was not supported. It appears the fuse library used by this package isn't supported on ARM. I tried to build locally and ran into the following error: $ GOOS=linux GOARCH=arm64 make build
CGO_ENABLED=0 GOOS=linux go build -a -ldflags '-extldflags "-static"' -o _output/s3driver ./cmd/s3driver
go: downloading github.com/container-storage-interface/spec v1.1.0
go: downloading github.com/kubernetes-csi/drivers v1.0.2
go: downloading golang.org/x/net v0.0.0-20211216030914-fe4d6282115f
go: downloading google.golang.org/grpc v1.40.0
go: downloading k8s.io/mount-utils v0.23.3
go: downloading github.com/kahing/goofys v0.24.0
go: downloading github.com/mitchellh/go-ps v0.0.0-20170309133038-4fdf99ab2936
go: downloading k8s.io/utils v0.0.0-20211116205334-6203023598ed
go: downloading github.com/minio/minio-go/v7 v7.0.5
go: downloading github.com/kubernetes-csi/csi-lib-utils v0.6.1
go: downloading k8s.io/klog/v2 v2.30.0
go: downloading github.com/kahing/fusego v0.0.0-20200327063725-ca77844c7bcc
go: downloading github.com/Azure/azure-sdk-for-go v32.1.0+incompatible
go: downloading github.com/Azure/azure-pipeline-go v0.2.1
go: downloading github.com/Azure/azure-storage-blob-go v0.7.1-0.20190724222048-33c102d4ffd2
go: downloading github.com/aws/aws-sdk-go v1.42.44
go: downloading github.com/go-logr/logr v1.2.0
go: downloading google.golang.org/genproto v0.0.0-20220126215142-9970aeb2e350
go: downloading github.com/satori/go.uuid v1.2.1-0.20181028125025-b2ce2384e17b
go: downloading github.com/shirou/gopsutil v2.21.11+incompatible
go: downloading golang.org/x/sys v0.0.0-20210816074244-15123e1e1f71
go: downloading golang.org/x/crypto v0.0.0-20211215153901-e495a2d5b3d3
go: downloading github.com/mattn/go-ieproxy v0.0.0-20190610004146-91bb50d98149
# github.com/ctrox/csi-s3/cmd/s3driver
github.com/jacobsa/fuse/internal/buffer.memclr: relocation target runtime.memclrNoHeapPointers not defined for ABI0 (but is defined for ABIInternal)
make: *** [build] Error 1 |
Hello, can csi-s3 works on arm64 platform, and if have any docs or pkgs? |
I'm looking for a CSI solution for a nomad cluster, that would use minio to provision the storage.
Can I use this image
ctrox/csi-s3:v1.2.0-rc.2
, configure it to connect to my minio, then bind to the socket? I don't expect full support for this, just want to ask if it's reasonable before jumping into it. This is the most maintained solution I could find so far, but kubernetes is not an option for what I'm doing here.The text was updated successfully, but these errors were encountered: