Efficient CI/CD Automation System

In this project BrewOps, A sample web app called Automateforgood is used to exhibit the DevOps practices. Whenever a new commit is made to the GitHub repo, Github Actions automatically triggers an action to package the application as docker Image and push it to the Docker hub enabling continuous integration. Another Github Actions trigger the system to execute Chef InSpec test cases as a part of continuous integration.

The Kubernetes cluster is provisioned using K3s in a vagrant box where the application can be deployed. Once the docker image is available in the Docker hub, it is automatically deployed into kubernetes. Kubernetes Manifest template is made using Helm Charts and input configuration files for Staging and prod environment are created. ArgoCD is used to enable Continuous Delivery on each deployment at the Staging/Prod Environment. In the end, I have also experimented creating docker image by exporting the artifacts created by Chef Habitat. Its quiet handy when it comes to cross platform builds. Chef Habitat project experimentation can be accessed here.

• A link to a Trello board for the project
• A link to a spreadsheet that includes the original and final project plan

1. Clone this repository.
2. Install the dependencies.
3. Setup the Vagrant environment.
4. Package and Deploy the application of your choice.
5. Update this README to reflect how someone would use your code.

1. Install Git
2. Install Python
3. Install Node Js
4. Install Ruby
5. Install Chef Habitat
6. Install Chef InSpec
7. Install Docker
8. Install Vagrant
9. Install Kubernetes
10. Install Helm
11. Install VirtualBox
12. Install ArgoCD

The project has /healthinfo endpoint exposed at app.py file

The project has /metrics endpoint exposed at app.py file

The Logs have been enabled for the project.

1. Package the application using docker

   • Create a dockerfile to package the application. Keep the application of your choice in the project directory. Here I have a sample application called automateforgood.
   • Build the docker image. You can check the commands to build here
   • Test the docker image locally. Access the application on http://127.0.0.1:7111 .

2. Continuous Integration with Github Actions to automate docker image build and push to hub

   • Aim of this step is to automate the packaging of the application using Github Actions.
   • Github Actions help us to build, tag and push the docker image of the application to dockerhub.
   • If not present, create .github/workflows directory.
   • To automate the login into Docker Hub, the github actions use Github Tokens and Github Encrypted Secrets.
   • Create and verify if the github actions execute on every new commit, thus pushing the latest docker image to the docker hub.

3. Continuous Integration with Github Actions to automate the execution of test cases via Chef InSpec

   • Aim of this step is to automate the testcase execution using Chef InSpec and enable continuous integration via Github Actions
   • This Github Actions help us to trigger the test case execution whenever a new commit is made, thus enabling continuous integration.
   • Verify if the github actions execute the Chef InSpec on every new commit, thus ensuring that nobody has broken the system.

4. Deploy a Kubernates Cluster using K3s

   • Aim of this step is to create a declarative kubernetes manifest and release the application to the sandbox environment
   • Use Vagrant environment and create kubernetes cluster with k3s. vagrant file is attached for reference
   • To create a vagrant box, navigate to this location where vagrantfile is placed, Use the command vagrant up , then vagrant ssh.
   • You can find the kubernetes declartive manifests here.
   • Use the command kubectl apply -f yaml_file_name to deploy the application in k3s cluster. The commands I used aregiven below.
   • Execute kubectl apply -f namespace.yaml.
   • Execute kubectl apply -f service.yaml
   • Execute kubectl apply -f deploy.yaml
   • And its Done !!
   • Alternatively, you can use chef cookbooks for installing and setting up kubernetes cluster.

5. Helm Charts Templating

   • The aim of this step is to parameterize the kubernetes manifests.
   • You can find the helm charts here.
   • The input values are built for staging and production environment seperately.

6. Continuous Delivery using ArgoCD

   • The aim of this step is to automatically deploy the application using ArgoCD, thus easy release to staging and production environment using the helm chart templates
   • Execute kubectl create namespace argocd to create the namespace.
   • Execute kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/core-install.yaml.
   • Execute kubectl apply -f argocd-nodeport.yaml. Nodeport Service Yaml files can be found here.
   • Execute kubectl apply -f helm-automateforgood-staging.yaml. You can find the yaml file here.
   • Execute kubectl apply -f helm-automateforgood-prod.yaml. You can find the yaml file here
   • Execute curl -sSL -o /usr/local/bin/argocd https://github.com/argoproj/argo-rollouts/releases/latest/download/kubectl-argo-rollouts-linux-amd64
   • Execute chmod +x /usr/local/bin/argocd
   • Access the argoCD UI at https://192.168.50.4 : 300008 or http://192.168.50.4:30007
   • Login credentials can be retrieved using the steps here
   • Whenever you made a new commit, the application will be packed as a docker image and gets deployed after a quick test case verification.

7. Slack Support

   • The Issues, Pulls, commits,release, deployments releated to this project will be notified to the user via the slack channel.

/github subscribe owner/repo [feature] /github unsubscribe owner/repo [feature]

Following features are enabled by default and can be disabled with the /github unsubscribe owner/repo [feature] command:

• issues - Opened or closed issues
• pulls - New or merged pull requests, as well as draft pull requests marked "Ready for Review"
• commits - New commits on the default branch (usually master)
• releases - Published releases
• deployments - Deployment review notifications and Deployment status updates.

Following features are disabled by default, and can be enabled with the /github subscribe owner/repo [feature] command:

• reviews - Pull request reviews
• comments - New comments on issues and pull requests
• branches - Created or deleted branches
• commits:* - All commits pushed to any branch
• +label:"your label" - Filter issues, pull-requests and comments based on their labels.

Know more about the integration from here to setup the integration

1. Continuous Integration using Github Actions.
2. Automated Testing via Chef inSpec & Github Actions.
3. Configuration of Kubernetes clusters using Helm Charts.
4. Continous Delivery using ArgoCD.

1. Chef InSpec has been explored and employed in the automation of testcases.
   • Chef InSpec has been integrated as a part of Continuous Integration via Github Actions.
   • The Chef InSpec runs during new commit and chek if the kubernetes yaml contains all the mandatory keywords. That is, It helps in configuration verification.
   • The implementation shows a sample usecase to give a essence of the software. Other potential use case are detailed below
2. Chef Habitat has been experimented as well to create artificats and later exported as docker image.
   • Post completion of the project, I experimented with the Chef Habitat to explore how it can be employed in artifact creation and how that can be exported as docker image. You can find my experiment here.
   • The implementation shows a sample usecase to give an essence of the software. This is much more than this use case and the same is detailed below

1. Enable a testcase to ensure that web server is only listening on well-secured ports
2. Run test on remote host on SSH or WinRM
3. Execute test on docker container
4. Execute a profile targetting AWS/Azure Environment
5. Make configuration verification like we did for Kubernetes
6. More usecases here.

1. Chef Habitat Artificats
   • Major advantage of Chef Habitat is, we can deploy and run our habitat app in different infrastructure environments like bare metal, VM, containers, and PaaS.
   • Chef Habitat Artifacts (.hart) supports cross platform builds, thus we can be easily export the app to docker, tarball, Apache Mesos and Cloud Foundary.
   • I gave a try on this, You can check the experiment here.

1. Code quality will be increased when CI/CD is in-place.
2. Delivery and Deployment will be faster as there is almost nil review time involved. Whenever new changes made, automatically docker image is generated and deployed into the kubernetes cluster. No maual review involved anywhere.
3. The docker image is created with every commit. This ensures consistency with code and excution environment.
4. Automation removes the possibilities for human errors. Once you commit the code, the build, deploy, test and delivery are completely automated in this project thus no possibility for human errors.
5. Fault isolation is very easy. Even for a single line commit, the new docker build and deployment is triggered. Also test cases will be executed via Chef InSpec thus fault isolation is spot on. Though Chef InSpec has many use cases, here it is employed to find the kubernetes configuration errors.
6. No more head ache with frequent update/maintanence. This project has two environments integrated. One is staging and other is prod. Easy maintanance and swap of images can be done between stag and prod environment.
7. This automation project make the delivery faster, consistant and accurate.

This project targets the cloud native environment where CI/CD assitance will be a big boosting factor. Kubernetes is used in the project to enable scalability. Also Helm charts are used in this project to make the kubernetes configuration even more simple. Chef Inspec is an interesting addition to enable infrastructure configuration testing/ security complaince testing.

1. Add more Chef InSpec Testcases to completely test/ audit the applications and infrastructure.
2. Utlize chef cookbooks for installing and setting up kubernetes cluster.
3. Configure indepth log levels to the application to enable easy debugging.
4. Explore application of Prometheus for monitoring and Grafana, for Observability.
5. Explore Polaris to ensure that Kubernetes pods and controllers are configured properly utilizing best practices.
6. Explore more about the real-time use cases of Chef Habitat. Already I experimented with the usage of chef habitat by building the artifacts of a sample application from scaffoldings and exported it as a docker image. You can see the execution here. am looking forward to knowing more about the real-time use cases Chef Habitat.
7. Build CI/CD Pipelines in AWS/Azure and explore the world of Cloud Ops. I am consistently self-learning and you can find my CI/CD pipeline deployment over the Azure project here. Looking forward to keeping the momentum.
8. Explore the application of chef automate in application, security, and infra automation.
9. List Goes On...