Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add k8s env froms #4969

Merged
merged 21 commits into from
Mar 18, 2024
Merged

Add k8s env froms #4969

merged 21 commits into from
Mar 18, 2024

Conversation

neilisaur
Copy link
Contributor

@neilisaur neilisaur commented Feb 27, 2024

Tracking issue

Closes #4968

Why are the changes needed?

We are able to specify default environment variables through the k8s plugin, but only those hard-coded into the configuration. Allowing default environment variables to be set via configMap or secret would allow automated systems in our environment to access secrets without embedding them in multiple locations.

What changes were proposed in this pull request?

Add ability to specify default configMap and secrets to mount on k8s plugin tasks.

How was this patch tested?

Unit tested, but local builds could not emulate the propeller build process currently.

Setup process

Screenshots

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

@dosubot dosubot bot added size:M This PR changes 30-99 lines, ignoring generated files. enhancement New feature or request labels Feb 27, 2024
Signed-off-by: Neil Stout <[email protected]>
Copy link

codecov bot commented Feb 28, 2024

Codecov Report

Attention: Patch coverage is 37.50000% with 10 lines in your changes are missing coverage. Please review.

Project coverage is 58.99%. Comparing base (3100c12) to head (bc6b254).

Files Patch % Lines
...asks/pluginmachinery/flytek8s/k8s_resource_adds.go 28.57% 8 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4969      +/-   ##
==========================================
+ Coverage   58.97%   58.99%   +0.01%     
==========================================
  Files         645      645              
  Lines       55578    55590      +12     
==========================================
+ Hits        32778    32794      +16     
+ Misses      20207    20202       -5     
- Partials     2593     2594       +1     
Flag Coverage Δ
unittests 58.99% <37.50%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@neilisaur neilisaur force-pushed the add-k8s-env-froms branch 2 times, most recently from 85c2c35 to 7418d49 Compare February 29, 2024 16:03
@neilisaur neilisaur marked this pull request as draft February 29, 2024 16:04
@neilisaur neilisaur marked this pull request as ready for review February 29, 2024 19:16
@dosubot dosubot bot added the documentation Improvements or additions to documentation label Feb 29, 2024
@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Mar 13, 2024
Copy link
Contributor

@eapolinario eapolinario left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need a little bit more work to load configmaps and secrets.

@neilisaur neilisaur marked this pull request as draft March 14, 2024 18:41
@neilisaur neilisaur marked this pull request as ready for review March 14, 2024 20:10
Signed-off-by: Neil Stout <[email protected]>
Signed-off-by: Neil Stout <[email protected]>
@eapolinario
Copy link
Contributor

eapolinario commented Mar 14, 2024

@neilisaur , can you fix the lint error and merge master while you're at it?

@neilisaur
Copy link
Contributor Author

@neilisaur , can you fix the lint error and merge master while you're at it?

@eapolinario done and done! Re-ran make helm as well

@neilisaur
Copy link
Contributor Author

@eapolinario for the Chart version update that's failing is there a specific method of changing it, or should I just update everything under charts/ to v0.1.11?

Copy link
Contributor

@eapolinario eapolinario left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@neilisaur , don't worry about that CI check failure. We've started to validate changes to the helm charts only recently, which means that we shouldn't be blocking PRs using those checks yet.

@eapolinario
Copy link
Contributor

Thanks you for your contribution!

@eapolinario eapolinario merged commit eb24107 into flyteorg:master Mar 18, 2024
49 of 51 checks passed
yubofredwang pushed a commit to yubofredwang/flyte that referenced this pull request Mar 26, 2024
* Adding allowance for default configMap/secrets to pull env froms

Signed-off-by: Neil Stout <[email protected]>

* Updating Helm charts

Signed-off-by: Neil Stout <[email protected]>

* Fixing typos

Signed-off-by: Neil Stout <[email protected]>

* More typo

Signed-off-by: Neil Stout <[email protected]>

* Ran 'make helm'

Signed-off-by: Neil Stout <[email protected]>

* Updating docs from `make helm`

Signed-off-by: Neil Stout <[email protected]>

* `make helm` results

Signed-off-by: Neil Stout <[email protected]>

* Fixing issues with env vs envFrom

Signed-off-by: Neil Stout <[email protected]>

* Fixing merge issue

Signed-off-by: Neil Stout <[email protected]>

* Better naming for variable

Signed-off-by: Neil Stout <[email protected]>

* Renaming to be more accurate

Signed-off-by: Neil Stout <[email protected]>

* Fixing typo

Signed-off-by: Neil Stout <[email protected]>

* Forgot one change

Signed-off-by: Neil Stout <[email protected]>

* Fixing linting error

Signed-off-by: Neil Stout <[email protected]>

* Updating Helm charts

Signed-off-by: Neil Stout <[email protected]>

---------

Signed-off-by: Neil Stout <[email protected]>
@flixr
Copy link
Contributor

flixr commented May 24, 2024

@neilisaur @eapolinario seems this broke tasks which already have env_from set in the pod template!
My secrets from the pod template are not there anymore... had to roll back to flyte 1.11.

I have a pod template something like:

primary_container = V1Container(name="primary")
primary_container.env_from = [V1EnvFromSource(secret_ref=V1SecretEnvSource(name="my-secret"))]
ps = V1PodSpec(containers=[primary_container])
pod_template = PodTemplate(pod_spec=ps, primary_container_name="primary")

eapolinario added a commit that referenced this pull request Jun 21, 2024
* keep EnvFrom from pod template

not complete nor tested, just as hint for potential fix for regression
in #4969

* Add test and fix build error

Signed-off-by: Eduardo Apolinario <[email protected]>

* Fix test

Signed-off-by: Eduardo Apolinario <[email protected]>

---------

Signed-off-by: Eduardo Apolinario <[email protected]>
Co-authored-by: Eduardo Apolinario <[email protected]>
robert-ulbrich-mercedes-benz pushed a commit to robert-ulbrich-mercedes-benz/flyte that referenced this pull request Jul 2, 2024
* keep EnvFrom from pod template

not complete nor tested, just as hint for potential fix for regression
in flyteorg#4969

* Add test and fix build error

Signed-off-by: Eduardo Apolinario <[email protected]>

* Fix test

Signed-off-by: Eduardo Apolinario <[email protected]>

---------

Signed-off-by: Eduardo Apolinario <[email protected]>
Co-authored-by: Eduardo Apolinario <[email protected]>
vlibov pushed a commit to vlibov/flyte that referenced this pull request Aug 16, 2024
* keep EnvFrom from pod template

not complete nor tested, just as hint for potential fix for regression
in flyteorg#4969

* Add test and fix build error

Signed-off-by: Eduardo Apolinario <[email protected]>

* Fix test

Signed-off-by: Eduardo Apolinario <[email protected]>

---------

Signed-off-by: Eduardo Apolinario <[email protected]>
Co-authored-by: Eduardo Apolinario <[email protected]>
Signed-off-by: Vladyslav Libov <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation enhancement New feature or request lgtm This PR has been approved by a maintainer size:M This PR changes 30-99 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Core feature] Allow k8s plugin to pull env from default configMaps / secrets
3 participants