Add option to specify custom pre-bootstrap commands (user_data) to execute on AWS EKS nodes

[joneszc]

Feature description

Enable ability to specify commands to run on AWS EKS cluster nodes prior to execution of /etc/eks/bootsrap.sh. Custom pre-bootstrap commands can be enabled by adding an aws_launch_template terraform resource that provides a configurable user_data section. An appropriate configuration option (e.g. amazon_web_services.node_prebootstrap_command) would allow users to input commands and control ec2 behavior and settings.

Value and/or benefit

Nebari users would be able to modify EKS nodes to accommodate target environment requirements to ensure networking/security compliance or otherwise enable customizations to integrate target environment systems.

For example, applying STIGs to EKS nodes prior to joining cluster:

For example, defining default container private-registry mirrors in containerd:

Anything else?

No response

[viniciusdc]

This is interesting, though on this one, I am quite reluctant, not due to the changes themselves. But I am wondering if we consider the changes proposed by #2604, wouldn't we be able to add such bootstrap scripts directly as parts of the custom AMIs as well?

[joneszc]

@viniciusdc ,
Having a "pre-bootstrap" option for inputting commands--separate from the /etc/eks/bootstrap.sh command--would decouple a set of use cases entirely from the bootstrap process, which ideally would be left to the EKS default command for joining nodes to the cluster, unless employing a Custom AMI or otherwise seeking to override /etc/eks/bootstrap.sh arguments (e.g. --kubelet-extra-args).
We are seeing situations where custom commands must be run on nodes without employing custom ami or otherwise modifying the /etc/eks/bootstrap.sh command.
For example, recently a customer environment required an additional ssl certificate to be added to ec2 instances and we were able to resolve the requirement, locally, by writing the cert to the node, from input in the nebari-config.yaml file:

However, both the pre-bootstrap use case and the #2604 custom AMI use case could share a common aws_launch_template resource and user_data block with logic built in to handle the combined use cases, as seen here.

[tylergraff]

To clarify:
One benefit of this "pre-bootstrap" option is that it allows execution of custom code in cases where the Nebari deployment team would not have the ability to alter the underlying AMI.

We anticipate that our customer organizations could have "pre-approved" AMIs that we must use, but we will not be allowed to modify (nor "fork") those pre-approved AMIs, at rest, with custom bootstrap code. Therefore, this solution to inject pre-bootstrap code at runtime is the most viable path.

We are currently using a prototype version of this capability (off our nebari fork) to add a customer's https proxy cert to a generic Amazon-supplied AMI as a proof of concept demonstration.

[joneszc]

It seems the current use cases entail the same command(s) being executed across all node-groups. It is worth noting that there could be use cases where only select nodes/node-groups require custom commands. For example, the configuration could be structured to enable a custom command section per node group

amazon_web_services:
  node_groups:
    general:
      instance: m5.2xlarge
      node_prebootstrap_command: |
        #!/bin/bash
        echo "hello world"
      min_nodes: 1
      max_nodes: 1
      gpu: false
      single_subnet: false

versus a common command section

amazon_web_services:
  node_prebootstrap_command: |
    #!/bin/bash
    echo "Hello World"

In favor of a more compact nebari-config, I think the single/common node_prebootstrap_command variable is best. The option to filter command execution by node-group would still be possible using ad-hoc conditions; for example:

[tylergraff]

This feature was introduced in #2668