Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add support for the Pulumi Automation API (#167)
* feat: add Pulumi Automation API scripts with only AWS support This change adds Python scripts that use the Pulumi Automation API to stand up MARA like how the bin/start.sh scripts currently do. * feat: integrate Automation API scripts with k8s secrets This change adds a new Pulumi project named 'secrets' to MARA. This project is used in conjuction with the Pulumi Automation API to store secrets using the Kubernetes secret store so the secrets can be used across Pulumi projects. * feat: improve authentication error output This change outputs the results of the AWS cli command 'aws sts get-caller-identity' more tersely and without a stacktrace when the command fails. * feat: add differing behavior for container push per provider * docs: update examples to reflect changes to secrets * feat: add support for Digital Ocean to automation API scripts * feat: add support for Digital Ocean Container Registry * chore: upgrade default helm chart version * chore: upgrade example version for ingress image name * refactor: break headers into separate file * refactor: normalize PulumiProject path property naming The naming of the property "root_path" conflicted with the initialization parameter "path" AS WELL AS the method "path()". This change normalizes the property names such that they do not ambiguously overlap. * refactor: on_success closure params to single class We want to be able to easily add new references to the state that can be processed for on_success events. As such, the three variables passed to those closures has been refactored to a single type which references the original three variables. This will make adding new variables easier. * refactor: separate namespace creation from ingress controller When using container registry credentials with nginx ingress controller, one must create the credential secrets for the registry in the same namespace as the ingress controller. By breaking it apart as a separate step, it allows us to layer in additional logic (such as adding credentials) after the namespace has been created and before the ingress controller has been deployed. * refactor: add DO Registry credentials to k8s secrets via a project This change adds a new Pulumi project that gets the authentication credentials for a Digital Ocean Container Repository, encodes them as a Kubernetes secret, and then stores the secret in the running cluster's nginx-ingress namespace. * refactor: make pulumi color settings a method on EnvConfig * refactor: add container registry implementation name method * fix: AWS registry not being referenced using 'repository' AWS ECR refers to itself as a repository and not a registry, we aim to keep that naming consistent when referring directly to ECR nouns. This change fixes a bug where we became over-eager using the word 'registry' instead of the noun 'repository' that is hardcoded in the ECR stack reference. * bugfix: change pipenv install to pipenv sync to avoid updating deps at build time (#157) (cherry picked from commit 01ef1ff) * chore: deprecated convenience scripts and projects (#159) * chore: remove non-functional kubevip project * chore: deprecation of resources as discussed in #155 (cherry picked from commit b4ff561) * fix: typo in find command was causing pulumi stacks to not be deleted (#160) (cherry picked from commit 0619d5d) * chore: jenkins fixes and general cleanup of jenkinsfiles (#161) * fix: typo in find command was causing pulumi stacks to not be deleted * fix: formatting and find syntax in jenkins (esc for Groovy) * fix: formatting and find syntax in jenkins (esc for Groovy) * chore: clean up the comments a bit (cherry picked from commit cf655d0) * refactor: improve naming and fix typos * feat: install Digital Ocean CLI tool * feat: prompt user for parameters when starting up DO * feat: add dns record support to Digital Ocean provider * refactor: change name of container registry credentials project * docs: small comment addition and doc change Reference to the DO CLI is removed from the documentation because it is installed as part of the setup_venv.sh script. * chore: double Helm timeout for Prometheus install * feat: allow adding new clusters to the kubectl config Allow for adding and merging new clusters into the users kubectl config. The AWS and Digital Ocean CLIs do this automatically. However, not all SDKs nor CLI tools do this. Here we add code that does this so that the same type of functionality can be done no matter what the underlying infrastructure provider is. * refactor: add method to insert project in exec order Add a method to allow for the insertion of a project anywhere in the execution order of a provider. * chore: simplify function name * feat: add check for empty configuration file * refactor: break apart read and prompt operations * chore: change encoding to ascii because it is correct * feat: add automation API support for LKE and Harbor on Linode This adds support for Linode using the Automation API scripts. In order to provide support, the Harbor Container Registry was added to the requirements. Now, before starting LKE an instance of Harbor will be started in a Linode compute instance. * refactor: remove bash provision and destroy scripts * fix: typo in function parameter * docs: add additional code comments * feat: migrate to python logging for output * docs: add MARA Runner design document * fix: change sed flag from -r to posix compat -E Fixes #168 * feat: add easy runner script to MARA automation This change adds a bash script that sets up the virtual environment and invokes the automation/main.py script. Ideally, this would make running MARA much easier than it currently is. * fix: properly pass parameters to main.py from runner * refactor: improve error messaging and operations parsing * fix: fixes object has no attribute '__debug_logger_func' error Fixes #166 There is an issue with methods being named with two leading underscores being passed correctly to Pulumi's async handlers. In this change, we rename the method and add a check for the method's existence. * test: fix test runner to work after directory refactor * fix: enable force delete on ECR so that MARA can remove it * fix: sensible message on refresh run without prev deployment * fix: change import for better compatibility with test runner * chore: upgrade pulumi deps and kubectl versions * fix: add in updates to automation-api branch for version bumps (#172) * chore: update cert-manager chart and crds * chore: update logagent (filebeat) chart version * chore: update nginx IC to latest chart version * chore: update prometheus chart to latest version * chore: update logstore (Elasticsearch) to latest chart versoin * chore: update observability to new yaml and new chart * chore: update example config with new values * fix: remediation of deployment bugs * chore: additional bug fixes and usability fixes to automation branch (#174) * chore: update cert-manager chart and crds * chore: update logagent (filebeat) chart version * chore: update nginx IC to latest chart version * chore: update prometheus chart to latest version * chore: update logstore (Elasticsearch) to latest chart versoin * chore: update observability to new yaml and new chart * chore: update example config with new values * fix: remediation of deployment bugs * fix: removed JWT-only logic from BoS * fix: remove logic for sirius_host from deprecated jwt deploys * fix: remove deprecated ingress-repo-only project * fix: adjust min kubectl version deployed * chore: updates to fix digital ocean deployment (#177) * chore: update cert-manager chart and crds * chore: update logagent (filebeat) chart version * chore: update nginx IC to latest chart version * chore: update prometheus chart to latest version * chore: update logstore (Elasticsearch) to latest chart versoin * chore: update observability to new yaml and new chart * chore: update example config with new values * fix: remediation of deployment bugs * fix: removed JWT-only logic from BoS * fix: remove logic for sirius_host from deprecated jwt deploys * fix: remove deprecated ingress-repo-only project * fix: adjust min kubectl version deployed * fix: refactor digitalocean to docean for variables * fix: provide ability to still run kubeconfig deploys until they are cut over to automation api (#179) * chore: update cert-manager chart and crds * chore: update logagent (filebeat) chart version * chore: update nginx IC to latest chart version * chore: update prometheus chart to latest version * chore: update logstore (Elasticsearch) to latest chart versoin * chore: update observability to new yaml and new chart * chore: update example config with new values * fix: remediation of deployment bugs * fix: removed JWT-only logic from BoS * fix: remove logic for sirius_host from deprecated jwt deploys * fix: remove deprecated ingress-repo-only project * fix: adjust min kubectl version deployed * fix: refactor digitalocean to docean for variables * fix: add repo-only IC deploy to support kubeconfig deploys * fix: modifications to handle kubeconfig deploys for now * fix: recommission bash scripts to support kubeconfig deploys for now * fix: added gitkeep for IC manifests dir which is required for repo-only deploy (#180) * chore: update cert-manager chart and crds * chore: update logagent (filebeat) chart version * chore: update nginx IC to latest chart version * chore: update prometheus chart to latest version * chore: update logstore (Elasticsearch) to latest chart versoin * chore: update observability to new yaml and new chart * chore: update example config with new values * fix: remediation of deployment bugs * fix: removed JWT-only logic from BoS * fix: remove logic for sirius_host from deprecated jwt deploys * fix: remove deprecated ingress-repo-only project * fix: adjust min kubectl version deployed * fix: refactor digitalocean to docean for variables * fix: add repo-only IC deploy to support kubeconfig deploys * fix: modifications to handle kubeconfig deploys for now * fix: recommission bash scripts to support kubeconfig deploys for now * fix: gitkeep needed for manifests dir under repo nginx * fix: updated jenkinsfiles for automation api work (#181) * chore: update cert-manager chart and crds * chore: update logagent (filebeat) chart version * chore: update nginx IC to latest chart version * chore: update prometheus chart to latest version * chore: update logstore (Elasticsearch) to latest chart versoin * chore: update observability to new yaml and new chart * chore: update example config with new values * fix: remediation of deployment bugs * fix: removed JWT-only logic from BoS * fix: remove logic for sirius_host from deprecated jwt deploys * fix: remove deprecated ingress-repo-only project * fix: adjust min kubectl version deployed * fix: refactor digitalocean to docean for variables * fix: add repo-only IC deploy to support kubeconfig deploys * fix: modifications to handle kubeconfig deploys for now * fix: recommission bash scripts to support kubeconfig deploys for now * fix: gitkeep needed for manifests dir under repo nginx * chore: update jenkinsfiles for automation api * chore: doc updates for automation-api changes (#183) * fix: adding updates to jenkinsfiles (#185) * chore: update cert-manager chart and crds * chore: update logagent (filebeat) chart version * chore: update nginx IC to latest chart version * chore: update prometheus chart to latest version * chore: update logstore (Elasticsearch) to latest chart versoin * chore: update observability to new yaml and new chart * chore: update example config with new values * fix: remediation of deployment bugs * fix: removed JWT-only logic from BoS * fix: remove logic for sirius_host from deprecated jwt deploys * fix: remove deprecated ingress-repo-only project * fix: adjust min kubectl version deployed * fix: refactor digitalocean to docean for variables * fix: add repo-only IC deploy to support kubeconfig deploys * fix: modifications to handle kubeconfig deploys for now * fix: recommission bash scripts to support kubeconfig deploys for now * fix: gitkeep needed for manifests dir under repo nginx * chore: update jenkinsfiles for automation api * fix: updates to the jenkinsfiles * chore: doc updates for automation-api changes * fix: update to docker instance for minikube jenkins * fix: add wheel back into setup_venv.sh (#187) * fix: add variable for password fields in jenkins runs / accept stack argument / formatting (#188) * chore: update cert-manager chart and crds * chore: update logagent (filebeat) chart version * chore: update nginx IC to latest chart version * chore: update prometheus chart to latest version * chore: update logstore (Elasticsearch) to latest chart versoin * chore: update observability to new yaml and new chart * chore: update example config with new values * fix: remediation of deployment bugs * fix: removed JWT-only logic from BoS * fix: remove logic for sirius_host from deprecated jwt deploys * fix: remove deprecated ingress-repo-only project * fix: adjust min kubectl version deployed * fix: refactor digitalocean to docean for variables * fix: add repo-only IC deploy to support kubeconfig deploys * fix: modifications to handle kubeconfig deploys for now * fix: recommission bash scripts to support kubeconfig deploys for now * fix: gitkeep needed for manifests dir under repo nginx * chore: update jenkinsfiles for automation api * fix: updates to the jenkinsfiles * chore: doc updates for automation-api changes * fix: update to docker instance for minikube jenkins * fix: add wheel back into setup_venv.sh * fix: jenkinsfile updates * feat: accept stack value on CLI, handle mis-match * chore: reformat markdown to fit standards * fix: changes requested in #188 * refactor: break up stack environment logic into fnctions * refactor: formatting changes to main.py * fix: address PR comments and formatting * refactor: formatting fixes * fix: formatting and PR requested changes * refactor: bash script cleanup * chore: remove deprecated testcap script * fix: shell isn't interpreting the args to pulumi right * fix: still having weird globbing issues. * fix: adjust jenkinsfiles for new runner syntax * fix: linode jenkinsfile and log level adjust / comment add (#194) * fix: update log level and add comment to clarify print stmt * fix: add closing braces for Linode Jenkinsfile * fix: cosmetic fix for || construct * feat: add new logo to repo (#195) Co-authored-by: Javier Evans <[email protected]> Co-authored-by: Jason Schmidt <[email protected]> Co-authored-by: Jason Schmidt <[email protected]>
- Loading branch information
4 people
authored
Aug 31, 2022
1 parent
852af9f
commit 2393772