Idea

This project is created for some simple purposes:

• use knowledge and train skills
• try to combine different features like API frameworks
• use several techniques like multiprocessing, multithreading, asyncio for application optimization
• try to analyze several approaches in order to speed up application using concurrency and parallelism
• implement multiservice architecture and orchestrate application services via docker and bash scripts
• use event-driven approach to establish communication between different parts of application
• just for fun

Implementation

Project UI

Used free bootstrap templates and django templates rendered via Jinja2.

1. Main UI

• looks like a simple business-card site, it's mostly about that site should look beautiful and attractive
• demonstrates site's architecture - services and relationships, and orchestration
• describes features, techniques, libraries which are under the hood
• uses only django, its templates plus minimum HTML/CSS/Javascript skills

2. Dashboard UI

2.1. Part #1 - Django-based

• works under Gunicorn(Uvicorn)
• uses another bootstrap theme than Main UI which suits better for sites that provide some data analysis
• operates with couple Django DB models, so deals with Database (PostgreSQL)
• uses DRF (Django Rest Framework) for API calls to those 2 DB models

2.2. Part #2 - FastAPI-based

• we want to scrape data from TripAdviser.com site for a number of hotels from a concrete city, and behind a scene we need to
  • a) extract all hotels and their data from hotel list page (30 hotels on one page)
• let's try to use FastAPI, just imagine that we want to have something fast and robust to quickly process user requests related to iterative processing third-party web pages
• so under the hood it uses asyncio and makes concurrent processing for I/O-based operations (http requests to get info ebout each hotel)
• and CPU-bound operations for parsing those html pages
• another API endpoint activates Celery task in order to try to it fully in parallel
• also one API works with Pub/Sub chain and just publishes a message in Kafka to store selected by a user hotels

2.3. Part #3 - Celery

• it has only one task - to extract hotels data and process it
• its effectiveness depends on available CPU cores

2.4. Part #4 - Even-driven

• we want to demo event-driven approach when one service doesn't have access to some resource R directly, but is able to publish a message for another service that is able to process it and use that resource R
• so FastAPI service has one API endpoint that just puts a message into Kafka (Producer)

2.5. Part #5 - Consumer

• Consumer service just receives a message (see 2.2 and 2.4) and stores its data in DB
• after that hotels, selected by a user will be available via DRF API

3. Other pages generated by Django and 3rd-party applications

• Django Admin site
• Silk profiler
• Adminer DB manager
• DRF swagger docs
• FastAPI swagger docs

Installation

This application uses Docker swarm for orchestration and deployment.

Clone the project

git clone https://github.com/reddalexx/demo_1.git

see .deploy/docker/utils/start_project.sh

Install Docker and other debian requirements if needed

see .deploy/docker/utils/install_debian.sh

Edit .env file

cp .deploy/docker/.env.template .deploy/docker/.env 
nano .deploy/docker/.env 

Use your custom variables there

Build custom images

source ./install_docker.sh

pushd demo_1/.deploy/docker/build/main
source ./build_image.sh
popd

pushd demo_1/.deploy/docker/build/fastapi
source ./build_image.sh
popd

see .deploy/docker/utils/start_project.sh

Create Docker network

docker network create --driver overlay demo_network

Deploy project

Use sudo -i to avoid files permission issues

cd .deploy/docker/
. ./deploy.sh

Undeploy project

cd .deploy/docker/
. ./undeploy.sh