How to update MTU of AWS EC2 running bottlerocket os?

[tskinner-oppfi]

I'd like to migrate to bottlerocket os for my EKS clusters but I currently change the MTU via a user data script using commands described here https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/network_mtu.html#set_mtu

How can I change the MTU of bottlerocket instances?

[stmcginnis]

Great question @tskinner-oppfi. My first thought, as an equivalent to the Amazon Linux instructions, would be to create a bootstrap container to make the same ip link set calls. But my experiments with that show this particular modification sadly will not work.

This could be a race between the bootstrap container setting the value, and the MTU being assigned via DHCP. With EC2, the default ends up being option interface-mtu 9001;. So even though I was able to see the value changed by my bootstrap container, when I then checked it after initialization completed it was back to 9001.

I think to support this properly Bottlerocket might need a new setting exposed to override this. If you would like to see this capability, would you mind submitting a feature request issue? I'm not sure when it could be added, but at least that would help track the request and see if anyone else is also looking for this capability to help prioritize things.

Notes

Just in case anyone is curious, here is what I did to change this setting with a bootstrap container. This will not work, but maybe it can give some clues.

First I created a script to be called by the container called bootstrap-script.sh:

#!/usr/bin/env bash

sudo ip link set dev eth0 mtu 1600

ip link show

Then created a Dockerfile with:

FROM public.ecr.aws/amazonlinux/amazonlinux:latest
COPY ./bootstrap-script.sh /
RUN chmod +x /bootstrap-script.sh
# Both sudo and ip are needed to make the `ip link set` call work properly
RUN yum install -y sudo iproute
ENTRYPOINT ["/bootstrap-script.sh"]

Built the container and published to a private ECR registry. Then set a test node's user data to have:

[settings.bootstrap-containers.bootstrap]
source = "123456789.dkr.ecr.us-east-2.amazonaws.com/bootstrap:latest"
mode = "always"

I could see in the system log (journalctl -u bootstrap-containers@bootstrap) that the script ran and successfully set the eth0 interface MTU to 1600:

eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1600 qdisc mq state UP mode DEFAULT group default qlen 1000

But by the time the rest of the system initialization happened and I was able to connect and run sheltie to do ip link show on the running system, that value was then back to the default 9001:

eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000

[tskinner-oppfi]

Thanks! created a new issue #3341