Ignition data too large when creating various CAPA Workload Clusters

[AverageMarcus]

Because of the amount of configuration we now have in our ignition data and the extra that is needed for private clusters (such as the proxy details) we've hit the limit for the file with MachinePools (that stores in a Kubernetes secret).

Error on MachinePool:

InvalidUserData.Malformed: User data is limited to 16384 bytes

This is occurring on our test private CAPA cluster goat since the release of cluster-aws v0.75.0.

The capa-private E2E test suite is currently failing because of this issue and has been disabled until it can be resolved.

Some notes:
The Control Plane nodes make use of S3 to temporarily store their ignition data as outlined in the documentation but it seems to not be implemented for MachinePools as far as we can tell. I have been unable to find any relevant issues upstream so we may be the first to hit this issue.

For reference:
The length of the ignition file on a capa private WC was - 17,368 characters
The length of the ignition file on a capa non-private WC was - 13,908 characters

Tasks

Beta Give feedback

1. Handle length of ignition data to ensure it doesn't exceed the limit
2. Re-enable the capa-private E2E tests

---

Related Slack thread

[AndiDog]

For Cilium ENI mode implementation (#2563), I'm running into the same trouble. Adding a script to set --max-pods isn't possible due to the size limit.

[T-Kukawka]

upstream follow up issue: https://github.com/giantswarm/giantswarm/issues/31156

[AverageMarcus]

@AndiDog This isn't "done" as we still haven't re-enabled the CAPA Private test suite to ensure this change is being tested.

[AndiDog]

The cluster creation part of the E2E private tests is working again (see test run in unmerged PR giantswarm/cluster-test-suites#398). So I'm closing this issue.

The test suite now fails due to the unrelated #3577 which @AverageMarcus you may want to watch.