diff --git a/Dockerfile b/Dockerfile new file mode 100644 index 00000000..af0bf778 --- /dev/null +++ b/Dockerfile @@ -0,0 +1,10 @@ +FROM python:3.10-slim-buster + +WORKDIR /src + +COPY ./analytics . + +RUN pip install --upgrade pip +RUN pip install -r requirements.txt + +CMD python app.py diff --git a/README.md b/README.md index d9a987b6..8b1995d0 100644 --- a/README.md +++ b/README.md @@ -1,131 +1,25 @@ # Coworking Space Service Extension -The Coworking Space Service is a set of APIs that enables users to request one-time tokens and administrators to authorize access to a coworking space. This service follows a microservice pattern and the APIs are split into distinct services that can be deployed and managed independently of one another. - -For this project, you are a DevOps engineer who will be collaborating with a team that is building an API for business analysts. The API provides business analysts basic analytics data on user activity in the service. The application they provide you functions as expected locally and you are expected to help build a pipeline to deploy it in Kubernetes. - -## Getting Started - -### Dependencies -#### Local Environment -1. Python Environment - run Python 3.6+ applications and install Python dependencies via `pip` -2. Docker CLI - build and run Docker images locally -3. `kubectl` - run commands against a Kubernetes cluster -4. `helm` - apply Helm Charts to a Kubernetes cluster - -#### Remote Resources -1. AWS CodeBuild - build Docker images remotely -2. AWS ECR - host Docker images -3. Kubernetes Environment with AWS EKS - run applications in k8s -4. AWS CloudWatch - monitor activity and logs in EKS -5. GitHub - pull and clone code - -### Setup -#### 1. Configure a Database -Set up a Postgres database using a Helm Chart. - -1. Set up Bitnami Repo -```bash -helm repo add https://charts.bitnami.com/bitnami -``` - -2. Install PostgreSQL Helm Chart -``` -helm install /postgresql -``` - -This should set up a Postgre deployment at `-postgresql.default.svc.cluster.local` in your Kubernetes cluster. You can verify it by running `kubectl svc` - -By default, it will create a username `postgres`. The password can be retrieved with the following command: -```bash -export POSTGRES_PASSWORD=$(kubectl get secret --namespace default -postgresql -o jsonpath="{.data.postgres-password}" | base64 -d) - -echo $POSTGRES_PASSWORD -``` -* The instructions are adapted from [Bitnami's PostgreSQL Helm Chart](https://artifacthub.io/packages/helm/bitnami/postgresql). - -3. Test Database Connection -The database is accessible within the cluster. This means that when you will have some issues connecting to it via your local environment. You can either connect to a pod that has access to the cluster _or_ connect remotely via [`Port Forwarding`](https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/) - -* Connecting Via Port Forwarding -```bash -kubectl port-forward --namespace default svc/-postgresql 5432:5432 & - PGPASSWORD="$POSTGRES_PASSWORD" psql --host 127.0.0.1 -U postgres -d postgres -p 5432 -``` - -* Connecting Via a Pod -```bash -kubectl exec -it bash -PGPASSWORD="" psql postgres://postgres@:5432/postgres -c -``` - -4. Run Seed Files -We will need to run the seed files in `db/` in order to create the tables and populate them with data. - -```bash -kubectl port-forward --namespace default svc/-postgresql 5432:5432 & - PGPASSWORD="$POSTGRES_PASSWORD" psql --host 127.0.0.1 -U postgres -d postgres -p 5432 < -``` - -### 2. Running the Analytics Application Locally -In the `analytics/` directory: - -1. Install dependencies -```bash -pip install -r requirements.txt -``` -2. Run the application (see below regarding environment variables) -```bash - python app.py -``` - -There are multiple ways to set environment variables in a command. They can be set per session by running `export KEY=VAL` in the command line or they can be prepended into your command. - -* `DB_USERNAME` -* `DB_PASSWORD` -* `DB_HOST` (defaults to `127.0.0.1`) -* `DB_PORT` (defaults to `5432`) -* `DB_NAME` (defaults to `postgres`) +The Coworking Space Service is a set of APIs that enables users to request one-time tokens and administrators to authorize access to a coworking space. This service follows a microservice pattern and the APIs are split into distinct services that can be deployed and managed independently of one another. -If we set the environment variables by prepending them, it would look like the following: -```bash -DB_USERNAME=username_here DB_PASSWORD=password_here python app.py -``` -The benefit here is that it's explicitly set. However, note that the `DB_PASSWORD` value is now recorded in the session's history in plaintext. There are several ways to work around this including setting environment variables in a file and sourcing them in a terminal session. +## how Coworking deployment process works -3. Verifying The Application -* Generate report for check-ins grouped by dates -`curl /api/reports/daily_usage` +1. EKS cluster - 'my-cluster' is being used. +2. DB - 'postgresql-service' is being used. +3. Application - 'coworking' is being used. -* Generate report for check-ins grouped by users -`curl /api/reports/user_visits` +## how to deploy changes -## Project Instructions -1. Set up a Postgres database with a Helm Chart -2. Create a `Dockerfile` for the Python application. Use a base image that is Python-based. -3. Write a simple build pipeline with AWS CodeBuild to build and push a Docker image into AWS ECR -4. Create a service and deployment using Kubernetes configuration files to deploy the application -5. Check AWS CloudWatch for application logs +1. Commit the changes and send pull requests for them to the remote repo: https://github.com/bgtestlab/coworking +2. Once merged, check AWS ECR if the newly built ECR image can be found in the repository name 'coworking' -### Deliverables -1. `Dockerfile` -2. Screenshot of AWS CodeBuild pipeline -3. Screenshot of AWS ECR repository for the application's repository -4. Screenshot of `kubectl get svc` -5. Screenshot of `kubectl get pods` -6. Screenshot of `kubectl describe svc ` -7. Screenshot of `kubectl describe deployment ` -8. All Kubernetes config files used for deployment (ie YAML files) -9. Screenshot of AWS CloudWatch logs for the application -10. `README.md` file in your solution that serves as documentation for your user to detail how your deployment process works and how the user can deploy changes. The details should not simply rehash what you have done on a step by step basis. Instead, it should help an experienced software developer understand the technologies and tools in the build and deploy process as well as provide them insight into how they would release new builds. +## how to verify The Application +BASE_URL: a6b4f34906bd64b039344479ba8b0e78-1195361041.us-east-1.elb.amazonaws.com +BASE_PORT: 5153 -### Stand Out Suggestions -Please provide up to 3 sentences for each suggestion. Additional content in your submission from the standout suggestions do _not_ impact the length of your total submission. -1. Specify reasonable Memory and CPU allocation in the Kubernetes deployment configuration -2. In your README, specify what AWS instance type would be best used for the application? Why? -3. In your README, provide your thoughts on how we can save on costs? +- Generate report for check-ins grouped by dates + `curl :/api/reports/daily_usage` -### Best Practices -* Dockerfile uses an appropriate base image for the application being deployed. Complex commands in the Dockerfile include a comment describing what it is doing. -* The Docker images use semantic versioning with three numbers separated by dots, e.g. `1.2.1` and versioning is visible in the screenshot. See [Semantic Versioning](https://semver.org/) for more details. \ No newline at end of file +- Generate report for check-ins grouped by users + `curl :/api/reports/user_visits` diff --git a/buildspec.yaml b/buildspec.yaml new file mode 100644 index 00000000..0e2470b4 --- /dev/null +++ b/buildspec.yaml @@ -0,0 +1,18 @@ +version: 0.2 + +phases: + pre_build: + commands: + - echo Logging into ECR + - aws ecr get-login-password --region $AWS_DEFAULT_REGION | docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com + build: + commands: + - echo Starting build at `date` + - echo Building the Docker image... + - docker build -t $IMAGE_REPO_NAME:$IMAGE_TAG . + - docker tag $IMAGE_REPO_NAME:$IMAGE_TAG $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG + post_build: + commands: + - echo Completed build at `date` + - echo Pushing the Docker image... + - docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG diff --git a/deployment/configmap.yaml b/deployment/configmap.yaml index 7dcae113..ff214243 100644 --- a/deployment/configmap.yaml +++ b/deployment/configmap.yaml @@ -1,17 +1,17 @@ apiVersion: v1 kind: ConfigMap metadata: - name: + name: coworking-configmap data: - DB_NAME: - DB_USER: - DB_HOST: - DB_PORT: + DB_NAME: mydatabase + DB_USERNAME: myuser + DB_HOST: 10.100.81.81 + DB_PORT: "5432" --- apiVersion: v1 kind: Secret metadata: - name: + name: coworking-secret type: Opaque data: - : \ No newline at end of file + DB_PASSWORD: bXlwYXNzd29yZA== diff --git a/deployment/coworking.yaml b/deployment/coworking.yaml index cf86a612..4f85e85a 100644 --- a/deployment/coworking.yaml +++ b/deployment/coworking.yaml @@ -7,10 +7,10 @@ spec: selector: service: coworking ports: - - name: "5153" - protocol: TCP - port: 5153 - targetPort: 5153 + - name: "5153" + protocol: TCP + port: 5153 + targetPort: 5153 --- apiVersion: apps/v1 kind: Deployment @@ -29,28 +29,28 @@ spec: service: coworking spec: containers: - - name: coworking - image: - imagePullPolicy: IfNotPresent - livenessProbe: - httpGet: - path: /health_check - port: 5153 - initialDelaySeconds: 5 - timeoutSeconds: 2 - readinessProbe: - httpGet: - path: "/readiness_check" - port: 5153 - initialDelaySeconds: 5 - timeoutSeconds: 5 - envFrom: - - configMapRef: - name: - env: - - name: DB_PASSWORD - valueFrom: - secretKeyRef: - name: - key: - restartPolicy: Always \ No newline at end of file + - name: coworking + image: 860491643728.dkr.ecr.us-east-1.amazonaws.com/coworking:0.0.1 + imagePullPolicy: IfNotPresent + livenessProbe: + httpGet: + path: /health_check + port: 5153 + initialDelaySeconds: 5 + timeoutSeconds: 2 + readinessProbe: + httpGet: + path: "/readiness_check" + port: 5153 + initialDelaySeconds: 5 + timeoutSeconds: 5 + envFrom: + - configMapRef: + name: coworking-configmap + env: + - name: DB_PASSWORD + valueFrom: + secretKeyRef: + name: coworking-secret + key: DB_PASSWORD + restartPolicy: Always diff --git a/postgresql-deployment.yaml b/postgresql-deployment.yaml new file mode 100644 index 00000000..5c648142 --- /dev/null +++ b/postgresql-deployment.yaml @@ -0,0 +1,32 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: postgresql +spec: + selector: + matchLabels: + app: postgresql + template: + metadata: + labels: + app: postgresql + spec: + containers: + - name: postgresql + image: postgres:latest + env: + - name: POSTGRES_DB + value: mydatabase + - name: POSTGRES_USER + value: myuser + - name: POSTGRES_PASSWORD + value: mypassword + ports: + - containerPort: 5432 + volumeMounts: + - mountPath: /var/lib/postgresql/data + name: postgresql-storage + volumes: + - name: postgresql-storage + persistentVolumeClaim: + claimName: postgresql-pvc diff --git a/postgresql-service.yaml b/postgresql-service.yaml new file mode 100644 index 00000000..26c24a85 --- /dev/null +++ b/postgresql-service.yaml @@ -0,0 +1,10 @@ +apiVersion: v1 +kind: Service +metadata: + name: postgresql-service +spec: + ports: + - port: 5432 + targetPort: 5432 + selector: + app: postgresql diff --git a/pv.yaml b/pv.yaml new file mode 100644 index 00000000..e18b6eea --- /dev/null +++ b/pv.yaml @@ -0,0 +1,13 @@ +apiVersion: v1 +kind: PersistentVolume +metadata: + name: my-manual-pv +spec: + capacity: + storage: 1Gi + accessModes: + - ReadWriteOnce + persistentVolumeReclaimPolicy: Retain + storageClassName: gp2 + hostPath: + path: "/mnt/data" diff --git a/pvc.yaml b/pvc.yaml new file mode 100644 index 00000000..d5306a1a --- /dev/null +++ b/pvc.yaml @@ -0,0 +1,11 @@ +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: postgresql-pvc +spec: + accessModes: + - ReadWriteOnce + storageClassName: gp2 + resources: + requests: + storage: 1Gi diff --git a/screenshots/Create a Kubernetes Database Service.png b/screenshots/Create a Kubernetes Database Service.png new file mode 100644 index 00000000..4a2a7dc1 Binary files /dev/null and b/screenshots/Create a Kubernetes Database Service.png differ diff --git a/screenshots/Review CloudWatch Container Insights logs to confirm that the application is operating normally.png b/screenshots/Review CloudWatch Container Insights logs to confirm that the application is operating normally.png new file mode 100644 index 00000000..897c5abc Binary files /dev/null and b/screenshots/Review CloudWatch Container Insights logs to confirm that the application is operating normally.png differ diff --git a/screenshots/Run CodeBuild pipeline to deploy Docker image to AWS ECR_1.png b/screenshots/Run CodeBuild pipeline to deploy Docker image to AWS ECR_1.png new file mode 100644 index 00000000..d3cf0996 Binary files /dev/null and b/screenshots/Run CodeBuild pipeline to deploy Docker image to AWS ECR_1.png differ diff --git a/screenshots/Run CodeBuild pipeline to deploy Docker image to AWS ECR_2.png b/screenshots/Run CodeBuild pipeline to deploy Docker image to AWS ECR_2.png new file mode 100644 index 00000000..ea1e92e5 Binary files /dev/null and b/screenshots/Run CodeBuild pipeline to deploy Docker image to AWS ECR_2.png differ diff --git a/screenshots/Run CodeBuild pipeline to deploy Docker image to AWS ECR_3.png b/screenshots/Run CodeBuild pipeline to deploy Docker image to AWS ECR_3.png new file mode 100644 index 00000000..6cfb1fb9 Binary files /dev/null and b/screenshots/Run CodeBuild pipeline to deploy Docker image to AWS ECR_3.png differ diff --git a/screenshots/Run CodeBuild pipeline to deploy Docker image to AWS ECR_4.png b/screenshots/Run CodeBuild pipeline to deploy Docker image to AWS ECR_4.png new file mode 100644 index 00000000..be734bb9 Binary files /dev/null and b/screenshots/Run CodeBuild pipeline to deploy Docker image to AWS ECR_4.png differ diff --git a/screenshots/Store Docker images in ECR_1.png b/screenshots/Store Docker images in ECR_1.png new file mode 100644 index 00000000..d2b63016 Binary files /dev/null and b/screenshots/Store Docker images in ECR_1.png differ diff --git a/screenshots/Store Docker images in ECR_2.png b/screenshots/Store Docker images in ECR_2.png new file mode 100644 index 00000000..eefd8717 Binary files /dev/null and b/screenshots/Store Docker images in ECR_2.png differ diff --git a/screenshots/Store Docker images in ECR_3.png b/screenshots/Store Docker images in ECR_3.png new file mode 100644 index 00000000..08b8f6b9 Binary files /dev/null and b/screenshots/Store Docker images in ECR_3.png differ diff --git a/screenshots/Successfully deploy Kubernetes service_1.png b/screenshots/Successfully deploy Kubernetes service_1.png new file mode 100644 index 00000000..56949025 Binary files /dev/null and b/screenshots/Successfully deploy Kubernetes service_1.png differ diff --git a/screenshots/Successfully deploy Kubernetes service_2.png b/screenshots/Successfully deploy Kubernetes service_2.png new file mode 100644 index 00000000..f4f04633 Binary files /dev/null and b/screenshots/Successfully deploy Kubernetes service_2.png differ diff --git a/screenshots/Successfully deploy Kubernetes service_3.png b/screenshots/Successfully deploy Kubernetes service_3.png new file mode 100644 index 00000000..83baa772 Binary files /dev/null and b/screenshots/Successfully deploy Kubernetes service_3.png differ