Bootstrap Container: Pull from S3 and use content as hostVolume

[JohnYoungers]

We're currently using the standard EKS AMIs for a fleet of GitHub Runners: each ephemeral runner pod currently provisions an EBS volume based on a snapshot to pre-populate its toolcache directory.

I'd like to switch from this approach to using a bottlerocket AMI, where on start-up, we pull a .tar file from S3, extract it, and instead of provisioning EBS volumes, we instead use a hostVolume pointed to this extracted location.

I've tried to cobble together configuration based on various discussion topics here and in the documentation, and was hoping someone could verify something like this would be the correct approach to accomplish this; the code below is CDK using typescript:

    const bottleRocketConfig = ec2.UserData.forLinux();
    bottleRocketConfig.addCommands(`
[settings.bootstrap-containers.toolcache]
  source = "amazonlinux:2023"
  mode = "once"
  essential = true
  user-data = "*base64*: aws s3 cp s3://my-cache-bucket/toolcache.tar - | tar -x -C /.bottlerocket/mnt/toolcache"
`);
    const ephemeralLaunchTemplate = new ec2.CfnLaunchTemplate(this, "ng-ephemeral-runners-lt", {
      launchTemplateData: {
        userData: cdk.Fn.base64(bottleRocketConfig.render()),
      },
    });
    cluster.addNodegroupCapacity("ng-ephemeral-runners", {
      amiType: eks.NodegroupAmiType.BOTTLEROCKET_X86_64,
      instanceTypes: ["m4.xlarge"],
      launchTemplateSpec: {
        id: ephemeralLaunchTemplate.ref,
        version: ephemeralLaunchTemplate.attrLatestVersionNumber,
      }
    });

And the content would be available to bind to at the directory /mnt/toolcache: are these assumptions correct? or am I going about this the wrong way?

[arnaldo2792]

Hey @JohnYoungers, you assumptions are correct and your solution "should work". However keep in mind that hostPath isn't SELinux aware (see this) which means that all the containers with access to the mount will have read/write permissions (unless you limit the permissions with readOnly: true in your spec file). As mentioned in the KEP, when hostPath is used no relabeling will occur in contrast with EBS volumes where the contents of the volumes are relabeled based on the SELinux rules. If you don't have any concerns on allowing read access to all the containers, your solution should work fine, otherwise if you want to keep files generated by each container isolated from each othe, EBS volumes should work best due the SELinux awareness of the EBS CSI driver (See #2656).