Storage in general (started as DO specific)

[ssoriche]

DigitalOcean offers both Object (think S3) and Block (shared filesystem) storage.

Block storage can contain multiple volumes, each volume can only be mounted to one pod at a time via the Kubernetes [PersistentVolumeClaim}(https://kubernetes.io/docs/concepts/storage/persistent-volumes/) declaration. This allows the volume to follow the pod no matter which kubernetes node the pod happens to be scheduled to.

For temporary storage, the ephemeral volume emptyDir declaration can be used on the pod. These volumes need to be populated on pod creation.

[ssoriche]

To share storage amongst multiple pods, the solution from DigitalOcean seems to be to run your own cephs implementation on a Kubernetes node, that pods claim their volumes against bypassing the limitation imposed by DigitalOceans CSI driver.

https://github.com/DO-Solutions/doks-rook-ceph

[ranguard]

I wonder if the attraction of a single place to be able to delete packages (which happens maybe once a year) is distracting us from keeping things simple, and resilient (we don't want to be able to delete stuff easily!)

We currently have all CPAN data on all Physical servers and use rrr-watcher to keep them synced with the new changes that happen in upstream CPAN. (NOTE, we are a/the "backpan" which means we have files including those old versions authors have deleted from the main cpan)

I'm trying to get my head around pods so if we have one or more webservers dockers and several cron jobs we could run all those containers together as a pod and they share the same storage...

But the actually the pod must be co-located which means on the same K8 host node, so we then loose all the benefits of running in a cluster?

We could run 3 pods which all have containers which can share disk, and so each pod of containers can run on a K8's logical hosts

• rrrwatcher (which syncs CPAN changes to disk)
• metacpan-cpan-static - aka https://cpan.metacpan.org/ which serves the zipped files (via fastly)
• metapcan-api-cron job processing (via minion queue)) - distributed importing of data etc into ES

We want a SINGLE container/cron that actually creates jobs in response to RRRWatcher (we call this metacpan watcher in the queue from ONE of the 3 pods)

Does that make sense?

[ssoriche]

Now that I understand what you're looking for in storage a little more, we'll definitely want to implement to doks-rook-ceph example from above.

Pods combine multiple containers, but you can't have things like a cronjob in the same pod, a cronjob creates its own pod. We could have a separate container with an application that does a sleep for set amount of time as part of a different container in a pod, but we lose some of the features.

Using the doks-rook-ceph implementation is definitely cleaner, and with it we can allow multiple pods to write to the same filesystem (as well as only 1 to write and many others to read, which can be desirable when updating things with cron).

[ranguard]

Ways of managing storage in k8's

Rook/ceph Object Storage Daemon

Rook/ceph is setup with access to a volume (local disk.. OR mounted) on each node (vm or physical server) in the cluster. It then automatically replicates files between every node in the cluster and it allows multiple containers to then access the file system.. something like this...

graph TD
    subgraph Node 3
        B[Rook/Ceph Object Storage Daemon] 
        C[Client Container 1]
        D[Client Container 2]
        B --> G[Rook/Ceph MON]
        G --> B
        B --> F[File Share]
        F <--> C
        F <--> D
    end

    subgraph Node 2
        I[Rook/Ceph Object Storage Daemon]
        J[Client Container 3]
        K[Client Container 4]
        I --> M[Rook/Ceph MON]
        M --> I
        I --> N[File Share]
        N <--> J
        N <--> K
    end

    subgraph Node 1
        P[Rook/Ceph Object Storage Daemon]
        Q[Client Container 5]
        R[Client Container 6]
        P --> T[Rook/Ceph MON]
        T --> P
        P --> U[File Share]
        Q <--> U
        R <--> U
    end
    
    F -- AUTOMATED DATA SYNC --> N[File Share] 
    N -- AUTOMATED DATA SYNC --> U[File Share]
    U -- AUTOMATED DATA SYNC --> F[File Share]

Loading

Share volume(s)

A volume can be setup on a node and then volumeMount e.g. in grep node into each container that needs access.

This would be how something like ElasticSearch (which does it's own replication between nodes in the cluster) would be setup.

graph TD;
    Cluster --> Node1;
    Cluster --> Node2;
    Cluster --> Node3;

    Node1 --> Volume1;
    Node2 --> Volume2;
    Node3 --> Volume3;

    Volume1 -->|Mount| Container1A;
    Volume1 -->|Mount| Container1B;

    Volume2 -->|Mount| Container2A;
    Volume2 -->|Mount| Container2B;

    Volume3 -->|Mount| Container3A;
    Volume3 -->|Mount| Container3B;

Loading

Next I will look at various options for actual volumes we could add to the nodes which can then be shared with the cluster

[ranguard]

Volume options

Feature	Hetzner VM Disk	Hetzner Volumes	Hetzner Storage Box	Backblaze B2
Mounting	-	Yes	sshfs or rclone	Using rclone
Network	local	local	local	internet
Storage cost (pm)	VM Pricing	$4.68 per 100G	$3.41 per 1TB	$6 per 1TB
Bandwidth Cost	-	Free internal	Free internal	Some free*
Ability to Snapshot	Whole VM	No	Yes	Yes
Notes	Disk scales with VM	Block level write to 3 physical servers (aka very reliable)

* Up to 3x of average monthly data stored, then $0.01/GB for additional egress.

Hetzner VM Pricing, we are currently on CX31's + 3 x 35G volumes (for grep.mc)

Note: you can go beyond the table below, by selecting the dedicated vCPU option on https://www.hetzner.com/cloud/

[ranguard]

Current servers mounted disk usage (this is bm-mc-01)

Filesystem                          Size  Used Avail Use% Mounted on

## The BACK PAN
/dev/mapper/vg--hdds-cpan           159G  102G   51G  67% /mnt/lv-cpan

## The ES indexes - need fast disks
/dev/mapper/vg--ssds-elasticsearch  128G   63G   59G  52% /mnt/lv-elasticsearch

## `tmp` storage, e.g where the api code unzips files to to then display specific file content, left unzipped as otherwise google crawl will just unzip all again
/dev/mapper/vg--hdds-metacpan--tmp  469G  261G  186G  59% /mnt/lv-metacpan--tmp

We may need several types of storage...

• Resilient, e.g. to store BackPAN on which we are the primary keepers of.
• Efficient and local - for running Elastic Search
• Large and cheap for unpacking cpan onto / anything other services may require large none primary data (e.g grep.mc.org)

Suggestion:

Short term

• New Hetzner Volumes (one per node), mounted as /mnt/rook_cifs
  • Setup with Rook/ceph Object Storage Daemon in cluster using these volumes
  • Setup for grep.mc (replace existing mechanism which required grep running on only one node at a time)
  • Setup to have an auto updated copy of backpan
    • Once done, move cpan.metacpan.org to use this copy
    • NOTE: metacpan-api not moving, so existing BM/LW carry on having a copy

Longer term (once we are ready to move metacpan-api)

• Backup backpan to either a hetzner storage-box or backblaze + automated snapshots
• Setup a hetzner storage-box - called mc-temp-storage
• Mount the mc-temp-storage storage-box on all nodes and make available to replace /mnt/lv-metacpan--tmp

IF we are then hosting ElasticSearch ourselves..

• Same as above, but we will need bigger nodes for RAM, so will get bigger local disks, we can use these for the ElasticSearch index storage, and share them as a volume into the containers. Have to make sure when we extend the VM's we make these their own partition rather than just extending the / partition

[ssoriche]

Each instance in the Hetzner cluster should have a volume mounted of the same size /mnt/rook, which is large enough to hold the persistent storage for all of the containers (minus databases and ElasticSearch)(this should be able to be grown later if needed). Using rook will allow us to use that storage multiple times (between cronjobs and long running containers), as well share the same storage amongst containers (one git checkout for all grep containers for example). Containers will use this storage by creating a PersistenVolumeClaim with the appropriate storageClass (not sure what that is yet) and the appropriate method (ReadOnly, ReadWrite, ReadWriteMany)

Another volume on each node to hold ElasticSearch (should we not move to hosted ElasticSearch) /mnt/es. The ElasticSearch StatefuleSet will create a PersistentVolumeClaim to use this volume as local-path, leaving ElasticSearch to do what it needs with the storage.

The metacpan-api/source and /mnt/lv-metacpan--tmp should be stored in a Hetzner Storage Box, allowing https access to the public in order to download Perl packages directly from. This reduces the duplicated storage we have in LiquidWeb and ByteMark (which should also start to share this storage)

[ranguard]

Grep is using storageClassName: local-path in it'sPersistenVolumeClaim

Looking at https://kubernetes.io/docs/concepts/storage/storage-classes/#provisioner this seems it'll be what we will always use local-storage (it's up to the node to do the mounting of any different type of volumes)?

So if I have this right.. the pvc.yaml is saying "This service should get some disk space". But what I can't quite get my head around is each container wanting disk space has to specify the mountPath which seems like duplication all over the place (cronjob deployment.yaml, deployment.yaml 2nd time. Is this just how it's done or have I missed something?

[ssoriche]

Yes, each container has to specify the mountPath as that's where the container wants the volume mounted within the container. For example /mnt/grep could be /opt/grep in the cron container, but /mnt/grep_data in the grep container.

You can think of the storage class as the driver to use when accessing the disk. The current grep container is using local disk, which mounts a volume from the kubernetes node to the container. There are others that mount volumes in different ways. When rook is installed, it will allow for a cephs storage class, which will allow a container to mount a filesystem regardless of which node it is on.

So no, going forward we will only use local-disk for the ElasticSearch containers, everything else will be using a storage class defined by rook.

[ranguard]

Ok, And the bit I was missing was that the node has /mnt/cluster_data (as the volume on each node is currently named for grep.mc, even though MC is only on one node) - the way this is configured is in platform/local-path-provisioner/configmap.yaml

I did login to each of our nodes and /mnt/cluster_data is actually empty on all 3!....

Filesystem      Size  Used Avail Use% Mounted on
hz-1
/dev/sda9        72G   15G   55G  21% /
hz-2
/dev/sda9        72G   16G   53G  24% /
hz-3
/dev/sda9        72G   39G   31G  56% /

So I think grep.mc is actually running on hz-3 and saving to the VM disk. No clue what's going on there but as we are reimplementing not something I'm going to spend time on

[ranguard]

DO: accessModes must be set to ReadWriteOnce. The other parameters, ReadOnlyMany and ReadWriteMany, are not supported by DigitalOcean volumes. See the Kubernetes documentation for more about accessModes.

[ranguard]

Obj: https://docs.digitalocean.com/products/kubernetes/how-to/use-spaces/ - have created https://mc-do-logs.fra1.digitaloceanspaces.com/

https://www.jackpearce.co.uk/running-rook-ceph-on-digitalocean-doks-managed-kubernetes/
https://github.com/jkpedo/rook/tree/doks - https://github.com/jkpedo/rook/blob/doks/deploy/examples/cluster-on-pvc-doks.yaml