Compatibility with Instance Storage

[MikeKroell]

It would be nice to be able to detect instance storage and add them to BTFS and expand as needed with EBS.

[golharam]

StarCluster code had this. Perhaps start out with using all the ephemeral drives to create a logical volume and extend with EBS from there.

[MikeKroell]

This is exactly what I'm looking for. Put any ephemeral storage into BTFS to start, and expand with EBS as/if needed.

StarCluster code had this. Perhaps start out with using all the ephemeral drives to create a logical volume and extend with EBS from there.

[golharam]

Here is the code I used:

https://github.com/golharam/StarCluster/blob/develop/starcluster/plugins/mount_ephemeral.sh

Its been a couple of years since I used this and I'm not sure ephemeral drives are still exposed the same way.

[wleepang]

It's a little more complicated now. Nitro based instances with NVME storage don't report those devices as ephemeral in the meta-data/block-device-mapping.

[gilfreund]

I had a similar finding. If I use launch templates, and the storage is not defined in it, it will not appear in meta-data/block-device-mapping (regardless of how many the instance type supports, and if all possible devices are defined in the template (so as not to be specific to an instance type), all meta-data/block-device-mapping seems to be populated.

My attempt at this: https://github.com/gilfreund/ec2lvm

[gilfreund]

One of the issues is finding the correct devices name, so the code can cycle through the available devices. Using curl --silent http://169.254.169.254/latest/meta-data/block-device-mapping is not sufficient as @wleepang noted: Nitro do not report this in a useful way.
I am thinking of resolving this is a another way: use aws ec2 describe-instance-types to get the storage info, then query the devices depending on the architecture and device types:

#!/usr/bin/env bash
# Requires awscli, awk, curl 

InstanceType=$(curl --silent http://169.254.169.254/latest/meta-data/instance-type)
AvailabilityZone=$(curl --silent http://169.254.169.254/latest/meta-data/placement/availability-zone)
Region=${AvailabilityZone:0:9}

while read -r Architecture StorageTotal StorageDiskSize StorageDiskCount StorageDiskNVME ; do

  echo "This is instance is $Architecture $StorageDiskCount x $StorageDiskSize GB disks, Total of $StorageTotal of instance storage, NVME is $StorageDiskNVME"

  if [[ $StorageDiskCount -gt 0 ]] ; then
    case $StorageDiskNVME in 
        required)
            yum install -y nvme-cle
            DeviceNames=$(nvme list | awk '/AWS/ { print $1 }')
        ;;
        supported)
            echo "Don't know what to do with $StorageDiskNVME mode"
        ;;
        *)
            for ((disk = 1 ; disk <= StorageDiskCount ; disk++)); do
                ephemeral=$((disk - 1))
                DeviceNames="/dev/$(curl --silent http://169.254.169.254/latest/meta-data/block-device-mapping/ephemeral$ephemeral) $DeviceNames"
            done
        ;;
    esac
  else
    echo "and has no instance storage"
  fi

echo "Found $DeviceNames"

done < <(aws --region "$Region" --output text ec2 describe-instance-types --instance-types "$InstanceType" \
    --query InstanceTypes[].[ProcessorInfo.SupportedArchitectures[0],InstanceStorageInfo.TotalSizeInGB,InstanceStorageInfo.Disks[0].SizeInGB,InstanceStorageInfo.Disks[0].Count,InstanceStorageInfo.NvmeSupport])

This is not foolproof, since if there are no block-device-mappings in the launch, describe-instance-types will report available devices, but none will be available

[MikeKroell]

@gilfreund I have expanded on your script. Currently this is independent of the autoscale itself.

#!/usr/bin/env bash
# Requires awscli, awk, curl
# nvme-cli & mdadm will be installed if needed

set -e

InstanceType=$(curl --silent http://169.254.169.254/latest/meta-data/instance-type)
AvailabilityZone=$(curl --silent http://169.254.169.254/latest/meta-data/placement/availability-zone)
Region=${AvailabilityZone:0:9}

while read -r Architecture StorageTotal StorageDiskSize StorageDiskCount StorageDiskNVME ; do
  echo "-- Checking For instance storage --"
  if [[ $StorageDiskCount -gt 0 ]] ; then
    case $StorageDiskNVME in
        required)
            echo "$StorageDiskCount x $StorageDiskSize GB disks found, Total of $StorageTotal of instance storage, NVME is $StorageDiskNVME. Installing nvme-cli"
            yum install -y -q nvme-cli
            DeviceNames=$(nvme list | awk '/AWS/ { print $1 }')
        ;;
        supported)
            echo "Don't know what to do with $StorageDiskNVME mode"
        ;;
        *)
            for ((disk = 1 ; disk <= StorageDiskCount ; disk++)); do
                ephemeral=$((disk - 1))
                DeviceNames="/dev/$(curl --silent http://169.254.169.254/latest/meta-data/block-device-mapping/ephemeral$ephemeral) $DeviceNames"
            done
        ;;
    esac
    echo "Instance Storage Device Path(s):"
    echo $DeviceNames | tr ' ' ','
    if [[ $StorageDiskCount -gt 1 ]] ; then
      echo "Multiple devices, installing mdadm & creating RAID"
      yum install mdadm -y -q
      RAID_DEVICE=/dev/md0
      mdadm --create --verbose $RAID_DEVICE --level=0 --name=Instance_Storage --raid-devices=$StorageDiskCount $DeviceNames
      echo $RAID_DEVICE > instance_storage_device.txt
      echo "Device \"$RAID_DEVICE\" exported to instance_storage_device.txt for EBS Autoscale inital device"     
    else
      echo "$DeviceNames" > instance_storage_device.txt 
      echo "Device \"$DeviceNames\" exported to instance_storage_device.txt for EBS Autoscale inital device" 
    fi
  else
    echo "No instance storage found"
  fi
done < <(aws --region "$Region" --output text ec2 describe-instance-types --instance-types "$InstanceType" \
    --query InstanceTypes[].[ProcessorInfo.SupportedArchitectures[0],InstanceStorageInfo.TotalSizeInGB,InstanceStorageInfo.Disks[0].SizeInGB,InstanceStorageInfo.Disks[0].Count,InstanceStorageInfo.NvmeSupport])

I then call the autoscale install as follows:

echo "-- Installing EBS AutoScaler --"
if [ -f instance_storage_device.txt ]; then
  INSTANCE_STORAGE=$(cat instance_storage_device.txt)
  /opt/amazon-ebs-autoscale/install.sh --initial-device $INSTANCE_STORAGE --mountpoint /var/lib/docker 2>&1 >> /var/log/ebs-autoscale-install.log
else
  DefaultScratchVolumeSize=$(cat DefaultScratchVolumeSize.txt)
  /opt/amazon-ebs-autoscale/install.sh --initial-size $DefaultScratchVolumeSize --mountpoint /var/lib/docker 2>&1 >> /var/log/ebs-autoscale-install.log
fi

[gilfreund]

Took me a while to look at this, sorry.
The only problem is that --initial-device is checked if it's a block devices, and it might be a symbolic link. In such a scenario, it's ignored. I opened a separate ticket on that (#58), as I am not sure what the best way to handle this is.

[MikeKroell]

In my testing across 4,5,6,i3en,i4 series instances, there has not been a problem. I moved from having EBS autoscale creating the initial device because we were hitting too many API limits, which resulted in instances failing to boot and a very large amount of abandoned EBS volumes. Now I have the initial device as part of the launch template to avoid hitting these limits. I also have some more edits to this script now as well as adding the ability to specify the threshold in the config file so that EBS only gets added once a NVMe is at 90% usage. I plan to do a PR for the threshold option.

[gilfreund]

I see now that my naming of the devices used in the meta-data (http://169.254.169.254/latest/meta-data/block-device-mapping) was not the same in the /dev/ directory. My script ignored the device names.

[gilfreund]

Closed #58, as it was an issue in my launch templates. The script provided by @MikeKroell works. Let me know if you need help with the PR.

[bounlu]

@MikeKroell 's code did not work in my case to find the root EBS volume. Can we also utilise the root volume as part of the logical volume?

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        241G     0  241G   0% /dev
tmpfs           241G     0  241G   0% /dev/shm
tmpfs           241G  508K  241G   1% /run
tmpfs           241G     0  241G   0% /sys/fs/cgroup
/dev/xvda1       30G  3.8G   27G  13% /
/dev/xvdba      100G  5.1G   94G   6% /scratch
tmpfs            49G     0   49G   0% /run/user/1000

aws --output text ec2 describe-instance-types --instance-types "$InstanceType" --query InstanceTypes[].[ProcessorInfo.SupportedArchitectures[0],InstanceStorageInfo.TotalSizeInGB,InstanceStorageInfo.Disks[0].SizeInGB,InstanceStorageInfo.Disks[0].Count,InstanceStorageInfo.NvmeSupport]

x86_64  None    None    None    None

[MikeKroell]

@MikeKroell 's code did not work in my case to find the root EBS volume. Can we also utilise the root volume as part of the logical volume?

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        241G     0  241G   0% /dev
tmpfs           241G     0  241G   0% /dev/shm
tmpfs           241G  508K  241G   1% /run
tmpfs           241G     0  241G   0% /sys/fs/cgroup
/dev/xvda1       30G  3.8G   27G  13% /
/dev/xvdba      100G  5.1G   94G   6% /scratch
tmpfs            49G     0   49G   0% /run/user/1000

aws --output text ec2 describe-instance-types --instance-types "$InstanceType" --query InstanceTypes[].[ProcessorInfo.SupportedArchitectures[0],InstanceStorageInfo.TotalSizeInGB,InstanceStorageInfo.Disks[0].SizeInGB,InstanceStorageInfo.Disks[0].Count,InstanceStorageInfo.NvmeSupport]

x86_64  None    None    None    None

I merged in 2.4.7 and added my script and example in #69

@bounlu you would not want to use the root volume. It already has a partition that is not BTRFS. Also, you'd want all this separate from the OS disk.