Merge pull request #474 from abhi4578/docs/readme-4.5.0

Update Docs/readme 4.5.0

Docker-Swarm-deployment/single-node/README.md

@@ -0,0 +1,72 @@
+# Docker Swarm-Based Deployment
+IUDX platform can be deployed in Docker Swarm using the scripts and Swarm code present in this directory and sub-directories. Each component is deployed as a single instance in Swarm. The components can be scaled manually. IUDX platform exposes endpoints through two ports - one for HTTPS traffic which it passes to Nginx and Nginx routes to appropriate service; one for Rabbitmq management and AMQPS streaming. The whole system is monitored using a monitoring stack module - Prometheus for metrics, promtail and Loki for logs, and Grafana for visualization and alerting.
+The overall architecture of Swarm deployment is summarised in the below figure.  
+
+
+<p align="center">
+<img src="../../docs/Overview-Deloyment-IUDX-Swarm.png">
+</p>
+
+## Features 
+1. Scaling of services can be done manually by increasing the number of replica containers backing the swarm service.
+2. Cloud agnostic deployment, achieved through the layered architecture of deployment and using open source tools whenever possible instead of using cloud services.
+3. Cost-effective deployment, typically can be done with 4-6 nodes or a single node based on load requirements
+4. It is useful for small-scale deployment/prototyping/proof-of-concept(PoC) where reliability is not critical.
+
+## Deployment 
+IUDX deployment in Docker swarm is based on layered architecture as depicted below figure: 
+Layered architecture gives more flexibility in replacing different technology/software in one layer and not affecting the whole deployment. The Layered approach  also enables the deployment to be cloud agnostic.
+
+
+<p align="center">
+<img src="../../docs/Deployment-stack-swarm.png">
+</p>
+
+1. First layer is cloud infrastructure, which would include bringing up the necessary infrastructure for Swarm deployment - VPC, subnet, Load balancers, IAM users, roles, Service Accounts, Virtual Machines, and Security Groups.
+2. Operating System: This can be any Linux-based modern server distribution that is supported by Docker. Deployment is tested and supported on Ubuntu 20/22.04 LTS distribution.
+3. Docker: Docker (docker-ce, docker-cli, build, compose plugin) needs to be installed in all the nodes to create the swarm
+4. Docker Swarm: Docker swarm cluster of all nodes ( can be one or more) needs to be created with an overlay network named "overlay-net".
+Installation of docker and creation of swarm with appropriate node labels is achieved using scripts at [infrastructure](./infrastructure/README.md)
+5. Addons - The components which are essential for certain features on top of Swarm that IUDX deployment will actively use but not part of the IUDX platform are classified as addons. This typically includes 
+    1. HTTPS API Gateway- [Nginx](./nginx/README.md)
+    2. [Monitoring Stack](./monitoring-stack/README.md)
+
+
+
+6. Workload/IUDX system: This is the actual deployment of IUDX components in the Docker swarm cluster. This is deployed through stack files, present in this directory.
+
+The deployment order follows in a layered way with the innermost layer deployed first and then the next layer, until the final layer of the IUDX system. The IUDX system has a lot of components and needs to be orderly deployed, as described in next sub-section
+
+### Dependency in IUDX systems and ordering
+IUDX components depend on each other and are represented below in Directed Acyclic Graph (DAG).
+
+<p align="center">
+<img src="../../docs/IUDX-Dependency-Graph.png">
+</p>
+
+From DAG, the components can be deployed as follows in 6 stages:
+1. Stage 1 
+    1. Immudb
+    2. Rabbitmq  
+    3. Postgresql
+    4. Redis
+    5. Zookeeper
+    6. Elasticsearch (Deploy ELK together)
+2. Stage 2  
+    1. Logstash
+    2. Kibana
+    3. Keycloak
+    4. Latest ingestion pipeline
+    5. Auditing server
+3. Stage 3
+    1. AAA server
+4. Stage 4 
+    1. Catalogue server
+5. Stage 5 
+    1. Resource Server
+    2. Resource Server Proxy
+    3. File server
+    4. GIS server
+    5. Data Ingestion server
+6.  Stage 6
+    1. Advance Monitoring Stack

K8s-deployment/README.md

@@ -0,0 +1,76 @@
+# K8s Based Deployment
+IUDX platform can be deployed in K8s using the scripts and K8s code present in this directory and sub-directories. Each component is deployed as a clustered instance in K8s. The components automatically scale using Horizontal Pod Autoscaler (HPA), custom autoscalers and nodes scale using Cluster Autoscaler (CA). IUDX platform exposes endpoints through two Network load balancers - one for HTTPS traffic which passes to the ingress controller and routes it to the appropriate service; one for Rabbitmq management and AMQPS streaming. The whole system is monitored using a monitoring stack module - Prometheus for metrics, promtail and Loki for logs, and Grafana for visualization and alerting.  Velero is used as the backup system.
+The overall architecture of K8s deployment is summarised in the below figure.  
+
+
+<p align="center">
+<img src="../docs/Overview-Deloyment-IUDX-K8s.png">
+</p>
+
+## Features 
+1. Highly available and scalable IUDX platform
+2. Cloud agnostic deployment, achieved through the layered architecture of deployment and using open source tools whenever possible instead of  cloud services.
+3. Production-grade deployment of IUDX
+4. Helm Chart-based K8s deployment.
+
+## Deployment 
+IUDX deployment in K8s is based on layered architecture as depicted below figure: 
+Layered architecture gives more flexibility in replacing different technology/software in one layer and not affecting the whole deployment.  The Layered approach  also enables the deployment to be cloud agnostic.
+
+<p align="center">
+<img src="../docs/Layered-Deployment-stack.png">
+</p>
+
+1. First layer is cloud infrastructure, which would include bringing up the necessary infrastructure for K8s deployment - VPC, subnet, Load balancers, IAM users, roles, Service Accounts, Virtual Machines, and Security Groups.
+2. Operating System: This can be any Linux-based modern server distribution that is supported by Rancher. We test and support exclusively Ubuntu 20/22.04 LTS distribution.
+3. Container runtime: Typically, any container runtime supported by Rancher and specific K8s distribution RKE1/RKE2 should work. We have tested against containerd runtime.
+4. K8s Core cluster: This refers to K8s Cluster - control plane and worker node components. The IUDX platform can be deployed on any CNCF-certified K8s distribution. Its tested on RKE1 and RKE2 K8s distribution. The related deployment files to bring up Rancher (K8s cluster orchestrator) is at [Rancher](./K8s-cluster/Rancher/README.md). To bring up K8s cluster using rancher at [RKE1](./K8s-cluster/K8s/RKE1/README.md) and [RKE2](./K8s-cluster/K8s/RKE2/README.md).
+5. K8s addons - The components which are essential for certain features on top of the K8s core cluster which IUDX K8s deployment will actively use but are not part of the IUDX platform are classified as K8s addons. This typically includes 
+    1. CNI networking - Calico, Canal
+    2. DNS - CoreDNS 
+    3. Metrics Server
+    4. Cluster Autoscaler - Rancher CA, AWS CA, Azure CA
+    5. Ingress Controller - Nginx
+    6. Storage - mainly drivers - aws ebs & efs, azure disk & file 
+    7. Backup - Velero 
+    8. Monitoring Stack
+
+    Typically, the first three are deployed as part of RKE1 and RKE2 based K8s distribution. The rest are deployed from the directory [addons](./K8s-cluster/addons/).
+
+6. Workload/IUDX system: This is the actual deployment of IUDX components in the K8s cluster. This is deployed through helm charts, present at the directory [Charts](./Charts/).
+
+The deployment order follows in a layered way with the innermost layer deployed first and then the next layer, until the final layer of the IUDX system. The IUDX system has a lot of components and needs to be deployed in an orderly fashion, described in next sub-section
+
+### Dependency in IUDX systems and ordering
+IUDX components depend on each other and are represented below in Directed Acyclic Graph (DAG).
+
+<p align="center">
+<img src="../docs/IUDX-Dependency-Graph.png">
+</p>
+
+From DAG, the components can be deployed as follows in 6 stages:
+1. Stage 1 
+    1. Immudb
+    2. Rabbitmq  
+    3. Postgresql
+    4. Redis
+    5. Zookeeper
+    6. Elasticsearch (Deploy ELK together)
+2. Stage 2  
+    1. Logstash
+    2. Kibana
+    3. Keycloak
+    4. Latest ingestion pipeline
+    5. Auditing server
+3. Stage 3
+    1. AAA server
+4. Stage 4 
+    1. Catalogue server
+5. Stage 5 
+    1. Resource Server
+    2. Resource Server Proxy
+    3. File server
+    4. GIS server
+    5. Data Ingestion server
+6.  Stage 6
+    1. Advance Monitoring Stack

README.md

@@ -1,43 +1,58 @@
 ![IUDX](./docs/iudx.png)
 
 # iudx-deployment
-Installation and setup scripts for single and multi node (clustered) IUDX services.
-
-# Components
-
-The IUDX system will consist of the following services and servers:
-- API Server: Implements the Catalogue and Resource Server APIs (Vert.x HTTPs server)
-- Database server: An IUDX Server
-- Data broker Server: An IUDX Server
-- Authentication and Authorization Server: An IUDX Server
-- Database Service: Connects with the IUDX Database (A Vert.x Service)
-- Databroker Service: Connects with the IUDX Data broker (A Vert.x Service)
-- Subscription and Callback Service: Connects with the IUDX Database and Databroker (A Vert.x Service)
-- Validation Service: Validates the catalogue item (A Vert.x Service)
-- File Service: Connects with the IUDX Database, Databroker and File server (A Vert.x Service)
-- Authentication and Authorization Service: Connects with the IUDX Authentication Server (Vert.x Service)
-
-# Deployment Architecture
+This repository gives installation and setup scripts to deploy the IUDX platform. We provide and support two types of IUDX platform deployment : 
+1. [Docker Swarm based deployment](./Docker-Swarm-deployment/single-node/README.md)
+2. [K8s based deployment](./K8s-deployment/README.md)
 
+The IUDX platform consists of various IUDX-built services and open-source components. The overview IUDX platform with components is shown in the below figure.
 <p align="center">
 <img src="./docs/deployment_overview.png">
 </p>
-
-Note: For simplicity, all the modules are not mentioned in the figure. 
-
-As a design choice on the architecture, keeping scalability for microservices in mind we chose the Service Mesh Architecture for Catalogue Server and Resource Server. In a service mesh architecture, each microservice is a well-defined module that can be containerized and discovered using service discovery. The orchestration of the services can be such that data-intensive modules are residing closer to the database which also helps in better response times, limits the bandwidth and reduces the cost. Also, it helps in scaling of a specific microservice at ease. 
-
-
-# Features
-
-- The system uses an overlay network and supports manual scaling using docker run.
-- Hazlecase with Zookeeper is used as our cluster manager.
-- Monitoring of the APIs, Services, Containers, Nodes are done using Micrometer, Promtail and Node exporter. 
-- Metrics are pushed to Loki, Prometheus and visualized using Grafana.
-
+IUDX is a data exchange platform facilitating the seamless discovery, and exchange of authorized data. Following is a short explanation of how various components interact in the IUDX platform:
+
+- Through the IUDX Catalogue server users discovers different datasets available on the platform.
+- A user can register with one or more roles (consumer/provider, data ingester/delegate)  in IUDX AAA and keycloak. The keycloak is used to manage the identities of users.
+
+- The user can get set/request policies at the AAA server, and get a token. IUDX AAA platform manages the policies through credentials/Policy Datastore(Postgres).
+
+- Through this token, the user can publish(input)/consume (output) data from any of the  IUDX resource access services (resource server, rs-proxy, GIS server, Data ingestion server, File server)
+
+- IUDX platform supports the following input data flows
+  - A data ingester ( delegate ) can pull the data from the downstream source (ICCC) and push it to the databroker (Rabbitmq). Which then is consumed by Logstash, the latest ingestion pipeline, and is pushed to the Meta Data/Data Store (Elasticsearch) and Latest Datastore (Redis).
+  - Also a data ingester can directly push data through HTTPS APIs exposed by Data Ingestion Server.
+
+- IUDX platform supports the following output data flows
+  - Get data through standardized Resource access service APIs - spatial, temporal, complex, file, gis, and async queries.
+  - Get live streaming data through Rabbitmq using a resource server Subscription
+  - Get data from non-IUDX resource server through resource-server proxy (rs-proxy). This is done through IUDX RS API query translation to non-IUDX RS-specific queries by a set of adapters that reside close to non-IUDX RS. The query and response are communicated to adapters and rs-proxy through databroker(Rabbitmq). 
+
+- IUDX platform is monitored through the micrometer, Prometheus for metrics and promtail, Loki for logs, and Grafana for Visualisation 
+-  The alerting through SMTP server for emails or Telegram bot for telegram messages.
+- All HTTPS API requests are processed through the API gateway.
+- The Rabbitmq specific communication i.e. streaming of data through AMQPS and HTTPS management interface is through the streaming gateway
+- Hazlecast with Zookeeper is used as the cluster manager for all Vert.x based API servers.
+- Successful API calls are audited in tamper proof database - immudb and in postgres through an auditing server
+
+To know more about IUDX, refer following resources: 
+1. [What is IUDX?](https://youtu.be/uWdmHztFrqs) To get an overview of the IUDX platform and its main motivation
+2. [IUDX Architecture Overview](https://www.youtube.com/watch?v=FeiZz0fJi5w)
+3. [IUDX Developer Section](https://iudx.org.in/developers/)
+
+
+## Features
+- Service Mesh Architecture based Vert.x API servers.
+- Each microservice is a well-defined module that can be containerized and discovered using service discovery. 
+- Docker Swarm deployment enables easy, cost-effective deployment of the IUDX platform suitable for prototyping and PoC.
+- Kubernetes-based deployment of the IUDX platform gives a scalable, highly available system through the clustered deployment of each component. It's suitable for production-grade deployment.
+- Both docker and K8s-based deployment is cloud agnostic* and can be deployed on any cloud or on-prem. It 
+has been tested currently on AWS and Azure.
+
+
+\*Note: K8s deployment depends on certain cloud services - Load Balancer, Storage, Object Storage, K8s cluster autoscaling but since this is offered by major clouds. It can be integrated into these cloud providers.
 ## Contributing
 We follow Git Merge based workflow
 1. Fork this repo
-2. Create a new feature branch in your fork. Multiple features must have a hyphen separated name, or refer to a milestone name as mentioned in Github -> Projects 
-3. Commit to your fork and raise a Pull Request with upstream. <br>
-A detailed instructions present [here](docs/git-commands.md).
+2. Create a new feature branch in your fork. Multiple features must have a hyphen-separated name, or refer to a milestone name as mentioned in Github -> Projects 
+3. Commit to your fork and raise a Pull Request upstream. <br>
+Detailed instructions are present [here](docs/git-commands.md).