Add config option to enable the encryption of AWS EKS secrets

[joneszc]

Reference Issues or PRs

Fixes #2681
Fixes #2746
Modifies PR#2723 (Failing Tests / Pytest)
Modifies PR#2752 (Failing Tests / Pytest)

What does this implement/fix?

Put a x in the boxes that apply

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds a feature)
• Breaking change (fix or feature that would cause existing features not to work as expected)
• Documentation Update
• Code style update (formatting, renaming)
• Refactoring (no functional changes, no API changes)
• Build related changes
• Other (please describe):

Testing

• Did you test the pull request locally?
• Did you add new tests?

How to test this PR?

Any other comments?

Allows user to set EKS encryption of secrets by specifying a KMS key ARN in nebari-config.yaml

amazon_web_services:
  eks_kms_arn: 'arn:aws:kms:us-east-1:010101010:key/3xxxxxxx-xxxxx-xxxxx-xxxxx'

The KMS key must meet the following conditions:

• Symmetric
• Can encrypt and decrypt data
• Created in the same AWS Region as the cluster
• If the KMS key was created in a different account, the IAM principal must have access to the KMS key.

[viniciusdc]

@joneszc, there are two PRs which seem to add the same thing, this one and #2752 -- I assume the first one was the original; can you close this one? (or move any relevant changes back to the other PR?)

[dcmcand]

@joneszc can we close #2752 and #2723 since we have this one?

[joneszc]

@joneszc can we close #2752 and #2723 since we have this one?

@dcmcand @viniciusdc
Yes, those two PRs were built on forks of the old develop branch that is now main
Thanks for help determining that the branch was not the issue causing Pytest failures. #2752 and #2723 can be closed.

[joneszc]

@viniciusdc
I've opened PR#537 to update the docs per your request