v0.6.0 - Refactored container module, other bug fixes

CHANGELOG.md

@@ -1,5 +1,13 @@
 # CHANGELOG
 
+## v0.6.0 (2020-10-25)
+
+* Added unit tests for the `container` module and refactored various code there
+* Fixed a bug where the deploy stage of a pipeline would fail if a container already existed but was stopped
+* Added Harvey badge info to README
+* Moved all logic from `app.py` to the `webhook` module, updated endpoint urls to be more verbose and explicit
+* Various other bug fixes and code cleanup
+
 ## v0.5.0 (2020-10-24)
 
 * Fixed a bug where container names were not being created properly which led to other issues down the pipeline flow

README.md

@@ -50,7 +50,7 @@ make help
 1. Ensure you've added your ssh key to the ssh agent: `ssh-add` followed by your password
 1. Enable logging (see below)
 1. Setup enviornment variables in `.env`
-1. Add webhooks for all your repositories you want to use Harvey with (point them to `http://example.com:5000/harvey`, send the payload as JSON)
+1. Add webhooks for all your repositories you want to use Harvey with (point them to `http://example.com:5000/pipelines/start`, send the payload as JSON)
 
 **NOTE:** It is not recommended to use Harvey alongside other CI/CD or Docker orchestration platforms on the same machine.
 
@@ -72,7 +72,7 @@ The [following](https://docs.docker.com/config/containers/logging/json-file/#usa
 
 Find the full [docs here](docs/README.md). 
 
-Harvey's entrypoint is a webhook (eg: `127.0.0.1:5000/harvey`). Pass GitHub data to Harvey and let it do the rest. If you'd like to simulate a GitHub webhook, simply pass a JSON file like the following example to the Harvey webhook endpoint (ensure you have an environment variable `MODE=test` to bypass the need for a webhook secret):
+Harvey's entrypoint is a webhook (eg: `127.0.0.1:5000/pipelines/start`). Pass GitHub data to Harvey and let it do the rest. If you'd like to simulate a GitHub webhook, simply pass a JSON file like the following example to the Harvey webhook endpoint (ensure you have an environment variable `MODE=test` to bypass the need for a webhook secret):
 
 ```javascript
 {
@@ -138,6 +138,16 @@ Here are some common examples of testing environments you can use. Any Docker `i
 - If no language is provided, an `Alpine Linux` container with `Shellcheck` pre-installed will be used.
 - If no version is provided, the `latest` tag will be used.
 
+### Add a Harvey badge to your project
+
+[![Harvey CI](https://img.shields.io/badge/CI%2FCD-Harvey-blue)](https://github.com/Justintime50/harvey)
+
+Add the following to your project's README:
+
+```
+[![Harvey CI](https://img.shields.io/badge/CI%2FCD-Harvey-blue)](https://github.com/Justintime50/harvey)
+```
+
 ## Development
 
 ```bash

docs/README.md

@@ -24,7 +24,7 @@ The following example will run a full pipeline (tests, build and deploy), tag it
 }
 ```
 
-* Optional: `compose` value can be passed to specify the `docker-compose` file to use **if** building from a docker-compose file and hitting the `/harvey/compose` endpoint.
+* Optional: `compose` value can be passed to specify the `docker-compose` file to use **if** building from a docker-compose file and hitting the `/pipelines/start/compose` endpoint.
 * Optional: `dockerfile` value can be passed to specify the name of the Dockerfile to build from (including path if not in root directory).
 
 **Possible Pipeline Values**

examples/examples.py

@@ -91,11 +91,11 @@
 """API Entrypoint (Webhook)"""
 with open('examples/git_webhook.json', 'r') as file:
     request = requests.post(
-        'http://127.0.0.1:5000/harvey',
+        'http://127.0.0.1:5000/pipelines/start',
         data=file,
         headers=harvey.Global.JSON_HEADERS
     )
-    print(request)
+    print(request.json())
 
 # """Retrieve a Pipeline by ID"""
 # request = requests.get(

harvey/app.py

@@ -1,15 +1,13 @@
 import json
 import os
 import time
-import hmac
-import hashlib
 import requests_unixsocket
-from threading import Thread
 from dotenv import load_dotenv
 from flask import Flask, request, abort
 from harvey.webhook import Webhook
 from harvey.globals import Global
 
+
 API = Flask(__name__)
 HOST = os.getenv('HOST', '127.0.0.1')
 PORT = os.getenv('PORT', '5000')
@@ -20,51 +18,38 @@
 # This file should only route requests to the proper functions
 
 
-@API.route('/harvey', methods=['POST'])
-def receive_webhook():
-    """Receive a Webhook - this is the entrypoint for Harvey"""
-    target = Webhook.receive
-    return webhook(target)
+@API.route('/pipelines/start', methods=['POST'])
+def start_pipeline():
+    """Start a pipeline based on webhook data
+    """
+    return Webhook.parse_webhook(request=request, target=Webhook.start_pipeline)
 
 
-@API.route('/harvey/compose', methods=['POST'])
-def receive_webhook_compose():
-    """Receive a Webhook, build from compose file.
-    This is the entrypoint for Harvey
-    """
-    target = Webhook.compose
-    return webhook(target)
+# @API.route('/pipelines/stop', methods=['POST'])
+# def stop_pipeline():
+#     # TODO: Add this endpoint
 
 
-def webhook(target):
-    """Initiate details to receive a webhook
-    """
-    data = request.data
-    signature = request.headers.get('X-Hub-Signature')
-    parsed_data = json.loads(data)
-    if parsed_data['ref'] == 'refs/heads/master':
-        if os.getenv('MODE') == 'test':
-            Thread(target=target, args=(parsed_data,)).start()
-            return "200"
-        if decode_webhook(data, signature):
-            Thread(target=target, args=(parsed_data,)).start()
-            return "200"
-        return abort(403)
-    return abort(500, 'Harvey can only pull from the master branch.')
-
-
-def decode_webhook(data, signature):
-    """Decode a webhook's secret key
+@API.route('/pipelines/start/compose', methods=['POST'])
+def start_pipeline_compose():
+    """Start a pipeline based on webhook data
+    But build from compose file.
     """
-    secret = bytes(os.getenv('WEBHOOK_SECRET'), 'UTF-8')
-    mac = hmac.new(secret, msg=data, digestmod=hashlib.sha1)
-    return hmac.compare_digest('sha1=' + mac.hexdigest(), signature)
+    return Webhook.parse_webhook(request=request, target=Webhook.start_pipeline_compose)
+
+
+# @API.route('/pipelines/stop/compose', methods=['POST'])
+# def stop_pipeline_compose():
+#     # TODO: Add this endpoint
 
 
 @API.route('/pipelines/<pipeline_id>', methods=['GET'])
 def retrieve_pipeline(pipeline_id):
     """Retrieve a pipeline's logs by ID
     """
+    # This is a hacky temporary solution until we can store
+    # this data in a database and is not meant to remain
+    # as a long-term solution
     file = f'{pipeline_id}.log'
     for root, dirs, files in os.walk(Global.PROJECTS_LOG_PATH):
         if file in files:
@@ -78,8 +63,9 @@ def retrieve_pipeline(pipeline_id):
 def retrieve_pipelines():
     """Retrieve a list of pipelines
     """
-    # TODO: 1) Do not expose log paths here
-    # TODO: 2) Replace this with retrieving from a DB instead
+    # This is a hacky temporary solution until we can store
+    # this data in a database and is not meant to remain
+    # as a long-term solution
     pipelines = {'pipelines': []}
     for root, dirs, files in os.walk(Global.PROJECTS_LOG_PATH, topdown=True):
         for file in files:
@@ -93,58 +79,67 @@ def retrieve_pipelines():
 # def container_healthcheck():
 #     # TODO: Add this endpoint
 
+
 # @API.route('/containers/create', methods=['POST'])
 # def create_container():
 #     """Create a Docker container"""
 #     tag = request.tag
 #     response = json.dumps(harvey.Container.create_container(tag))
 #     return response
 
+
 # @API.route('/containers/<container_id>/start', methods=['POST'])
 # def start_container(container_id):
 #     """Start a Docker container"""
 #     start = harvey.Container.start_container(container_id)
 #     response = str(start)
 #     return response
 
+
 # @API.route('/containers/<container_id>/stop', methods=['POST'])
 # def stop_container(container_id):
 #     """Stop a Docker container"""
 #     stop = harvey.Container.stop_container(container_id)
 #     response = str(stop)
 #     return response
 
+
 # @API.route('/containers/<container_id>', methods=['GET'])
 # def inspect_container(container_id):
 #     """Retrieve a Docker container"""
 #     response = json.dumps(harvey.Container.inspect_container(container_id))
 #     return response
 
+
 # @API.route('/containers', methods=['GET'])
 # def all_containers():
 #     """Retrieve all Docker containers"""
 #     response = json.dumps(harvey.Container.list_containers())
 #     return response
 
+
 # @API.route('/containers/<container_id>/logs', methods=['GET'])
 # def logs_container(container_id):
 #     """Retrieve logs from a Docker container"""
 #     response = str(harvey.Container.inspect_container_logs(container_id))
 #     return response
 
+
 # @API.route('/containers/<container_id>/wait', methods=['POST'])
 # def wait_container(container_id):
 #     """Wait for a Docker container to exit"""
 #     response = json.dumps(harvey.Container.wait_container(container_id))
 #     return response
 
+
 # @API.route('/containers/<container_id>/remove', methods=['DELETE'])
 # def remove_container(container_id):
 #     """Remove (delete) a Docker container"""
 #     remove = harvey.Container.remove_container(container_id)
 #     response = str(remove)
 #     return response
 
+
 # @API.route('/build', methods=['POST'])
 # def build_image():
 #     """Build a Docker image"""
@@ -154,25 +149,29 @@ def retrieve_pipelines():
 #     build = harvey.Image.build(data, tag, context)
 #     return build
 
+
 # @API.route('/images/<image_id>', methods=['GET'])
 # def retrieve_image(image_id):
 #     """Retrieve a Docker image"""
 #     response = json.dumps(harvey.Image.retrieve(image_id))
 #     return response
 
+
 # @API.route('/images', methods=['GET'])
 # def all_images():
 #     """Retrieve all Docker images"""
 #     response = json.dumps(harvey.Image.all())
 #     return response
 
+
 # @API.route('/images/<image_id>/remove', methods=['DELETE'])
 # def remove_image(image_id):
 #     """Remove (delete) a Docker image"""
 #     remove = harvey.Image.remove(image_id)
 #     response = str(remove)
 #     return response
 
+
 # @API.route('/pull', methods=['POST'])
 # def pull_project():
 #     """Pull/clone GitHub project"""
@@ -181,6 +180,7 @@ def retrieve_pipelines():
 #     response = str(pull)
 #     return response
 
+
 def main():
     # allows us to use requests_unixsocker via requests
     requests_unixsocket.monkeypatch()