Merge pull request #9 from daystram/dev

README.md

@@ -9,14 +9,136 @@
 
 DASH video-streaming and RTMP live-streaming platform.
 
+## Features
+- [DASH](https://en.wikipedia.org/wiki/Dynamic_Adaptive_Streaming_over_HTTP) Video Streaming
+- [RTMP](https://en.wikipedia.org/wiki/Real-Time_Messaging_Protocol) Live Streaming
+- Live Chat (WebSocket)
+- Highly Scalable Transcoder Nodes
+- GPU Accelerated Video Transcoding
+- [Ratify](https://ratify.daystram.com/) Authentication
+
+### DASH
+With DASH streaming, videos uploaded to __cast__ are first re-encoded by the Transcoder nodes (`cast-is`) to multiple resolutions (240p, 360p, 480p, 720p, 1080p). The video player selects the most suitable resolution, and transitions between them seamlessly, based on the client's available bandwidth and preferences.
+
+### RTMP
+RTMP streaming allows the users to stream live to their viewers via a direct uplink to __cast__'s servers. Users can use clients such as [OBS Studio](https://obsproject.com/) or [Streamlabs](https://streamlabs.com/).
+
 ## Services
 The application comes in three parts:
 
 |Name|Code Name|Stack|
 |----|:-------:|-----|
-|Back-end|`cast-be`|[Go](https://golang.org/), [BeeGo](https://beego.me/), [MongoDB](https://www.mongodb.com/)|
+|Back-end|`cast-be`|[Go](https://golang.org/), [BeeGo](https://beego.me/), [MongoDB](https://www.mongodb.com/), [RabbitMQ](https://www.rabbitmq.com/), S3|
+|Transcoder|`cast-is`|[Go](https://golang.org/), [FFMpeg](https://ffmpeg.org/), [RabbitMQ](https://www.rabbitmq.com/), S3|
 |Front-end|`cast-fe`|JavaScript, [ReactJS](https://beego.me/)|
-|Transcoder|`cast-is`|[Go](https://golang.org/), [FFMpeg](https://ffmpeg.org/)|
+
+## Deploy
+`cast-be`, `cast-is`, and `cast-fe` are containerized and pushed to [Docker Hub](https://hub.docker.com/r/daystram/cast). They are tagged based on their application name and version, e.g. `daystram/cast:be` or `daystram/cast:be-v2.0.1`.
+
+To run `cast-be`, run the following:
+```console
+$ docker run --name cast-be --env-file ./.env -p 8080:8080 -d daystram/cast:be
+```
+
+To run `cast-is`, run the following:
+```console
+$ docker run --name cast-is --env-file ./.env -d daystram/cast:is
+```
+
+And `cast-fe` as follows:
+```console
+$ docker run --name cast-fe -p 80:80 -d daystram/cast:fe
+```
+
+### Dependencies
+The following are required for `cast-be` to function properly:
+- [MongoDB](https://www.mongodb.com/)
+- [RabbitMQ](https://www.rabbitmq.com/)
+- S3 storage provider
+
+The following are required for `cast-is` to function properly:
+- [RabbitMQ](https://www.rabbitmq.com/)
+- S3 storage provider
+
+Their credentials must be provided in their respective services' configuration file.
+
+Any S3 storage provider such as [AWS S3](https://aws.amazon.com/s3/ are supported. For this particular deployment for [cast.daystram.com](https://cast.daystram.com/), a self-hosted [MinIO](https://min.io/) is used.
+
+### Docker Compose
+For ease of deployment, the following `docker-compose.yml` file can be used to orchestrate the stack deployment:
+```yaml
+version: "3"
+services:
+  cast-fe:
+    image: daystram/cast:fe
+    ports:
+      - "80:80"
+    restart: unless-stopped
+  cast-is:  # no attached GPU
+    image: daystram/cast:is
+    env_file:
+     - /path_to_env_file/.env
+    restart: unless-stopped
+  cast-be:
+    image: daystram/cast:be
+    ports:
+      - "8080:8080"
+    env_file:
+      - /path_to_env_file/.env
+    restart: unless-stopped
+  mongodb:
+    image: mongo:4.4-bionic
+    environment:
+      MONGO_INITDB_ROOT_USERNAME: MONGODB_USER
+      MONGO_INITDB_ROOT_PASSWORD: MONGODB_PASS
+    expose:
+      - 27017
+    volumes:
+      - cast-mongodb:/data/db
+    restart: unless-stopped
+  rabbitmq:
+    image: rabbitmq:3.8-alpine
+    environment:
+      RABBITMQ_DEFAULT_USER: RABBITMQ_USER
+      RABBITMQ_DEFAULT_PASS: RABBITMQ_PASS
+    expose:
+      - 5672
+    restart: unless-stopped
+```
+
+### GPU Containers
+The Transcoder service `cast-is` is able to utilize GPU (NVIDIA graphics cards only) for harware accelerated video transcoding. To enable this, simply set the environment `USE_CUDA=true` when running the container.
+
+Docker Engine also needs to be configured to allow GPU passthrough. Follow the steps provided [here](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#docker) to enable NVIDIA's `container-toolkit`. Ensure that the host machine already have the GPU drivers installed.
+
+To run `cast-is` with GPU attached, use the following:
+```shell
+$ docker run --name cast-is --env-file ./.env --gpus all -d daystram/cast:is
+```
+
+For `docker-compose.yml`, as noted [here](https://github.com/docker/compose/issues/6691#issuecomment-758460418), use the following:
+```yml
+version: "3"
+services:
+  cast-is:
+    image: daystram/cast:is
+    env_file:
+      - /path_to_env_file/.env
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - capabilities:
+              - gpu
+    restart: unless-stopped
+```
+
+### Ingest-Base Image
+`daystram/ingest-base` ([DockerHub](https://hub.docker.com/r/daystram/ingest-base)) is the base image used to build `cast-is`. This image contains the required tools for the Transcoder to properly re-encode the source videos. The tools required are:
+- [FFmpeg](https://ffmpeg.org/)
+- [MP4Box + gpac](https://github.com/gpac/gpac)
+
+This image is built on top of [NVIDIA's CUDA images](https://hub.docker.com/r/nvidia/cuda/) to enable FFmpeg harware acceleration on supported hosts. MP4Box is built from source, as seen on the [Dockerfile](https://github.com/daystram/cast/blob/master/cast-is/ingest-base.Dockerfile).
 
 ## License
 This project is licensed under the [MIT License](https://github.com/daystram/cast/blob/master/LICENSE).

cast-be/config/.example.env

@@ -15,4 +15,8 @@ RABBITMQ_URI=amqp://MQ_USER:MQ_PASS@localhost:5672
 RABBITMQ_QUEUE_TASK=cast-transcode-task
 RABBITMQ_QUEUE_PROGRESS=cast-transcode-progress
 
-UPLOADS_DIR=/PATH_TO_UPLOADS_DIR
+S3_URI=cdn.daystram.com
+S3_BUCKET=cast
+S3_REGION=us-east-1
+S3_ACCESS_KEY=ACCESS_KEY
+S3_SECRET_KEY=SECRET_KEY

cast-be/config/config.go

@@ -23,11 +23,15 @@ type Config struct {
 	RabbitMQQueueTask     string
 	RabbitMQQueueProgress string
 
+	S3URI       string
+	S3Bucket    string
+	S3Region    string
+	S3AccessKey string
+	S3SecretKey string
+
 	RatifyIssuer       string
 	RatifyClientID     string
 	RatifyClientSecret string
-
-	UploadsDirectory string
 }
 
 // Load configuration
@@ -78,7 +82,19 @@ func InitializeAppConfig() {
 		log.Fatalln("[INIT] RABBITMQ_QUEUE_PROGRESS is not set")
 	}
 
-	if AppConfig.UploadsDirectory = viper.GetString("UPLOADS_DIR"); AppConfig.UploadsDirectory == "" {
-		log.Fatalln("[INIT] UPLOADS_DIR is not set")
+	if AppConfig.S3URI = viper.GetString("S3_URI"); AppConfig.S3URI == "" {
+		log.Fatalln("[INIT] S3_URI is not set")
+	}
+	if AppConfig.S3Bucket = viper.GetString("S3_BUCKET"); AppConfig.S3Bucket == "" {
+		log.Fatalln("[INIT] S3_BUCKET is not set")
+	}
+	if AppConfig.S3Region = viper.GetString("S3_REGION"); AppConfig.S3Region == "" {
+		log.Fatalln("[INIT] S3_REGION is not set")
+	}
+	if AppConfig.S3AccessKey = viper.GetString("S3_ACCESS_KEY"); AppConfig.S3AccessKey == "" {
+		log.Fatalln("[INIT] S3_ACCESS_KEY is not set")
+	}
+	if AppConfig.S3SecretKey = viper.GetString("S3_SECRET_KEY"); AppConfig.S3SecretKey == "" {
+		log.Fatalln("[INIT] S3_SECRET_KEY is not set")
 	}
 }

cast-be/constants/user.go

@@ -1,8 +1 @@
 package constants
-
-const (
-	ProfileRootDir = "profile"
-	ProfileDefault = "_default"
-	ProfileWidth   = 320
-	ProfileHeight  = 320
-)

cast-be/constants/video.go

@@ -3,8 +3,9 @@ package constants
 const (
 	VideoTypeVOD     = "vod"
 	VideoTypeLive    = "live"
+	VideoRootDir     = "video"
 	ThumbnailRootDir = "thumbnail"
-	ThumbnailDefault = "_default"
+	ThumbnailDefault = "_default_thumbnail"
 	ThumbnailWidth   = 720
 	ThumbnailHeight  = 405
 

cast-be/controller/v1/video.go

@@ -196,32 +196,11 @@ func (c *VideoControllerAuth) UploadVideo(_ string) datatransfers.Response {
 		Title:       upload.Title,
 		Description: upload.Description,
 		Tags:        strings.Split(upload.Tags, ","),
-	}, c.userID)
+	}, c.Controller, c.userID)
 	if err != nil {
 		fmt.Printf("[VideoControllerAuth::UploadVideo] failed creating video. %+v\n", err)
 		return datatransfers.Response{Error: "Failed creating video", Code: http.StatusInternalServerError}
 	}
-	// TODO: use S3
-	//// Retrieve video and thumbnail
-	//_ = os.Mkdir(fmt.Sprintf("%s/%s", config.AppConfig.UploadsDirectory, videoID.Hex()), 755)
-	//err = c.SaveToFile("video", fmt.Sprintf("%s/%s/video.mp4", config.AppConfig.UploadsDirectory, videoID.Hex()))
-	//if err != nil {
-	//	_ = c.Handler.DeleteVideo(videoID, c.userID)
-	//	fmt.Printf("[VideoControllerAuth::UploadVideo] failed saving video file. %+v\n", err)
-	//	return datatransfers.Response{Error: "Failed saving video file", Code: http.StatusInternalServerError}
-	//}
-	//err = c.SaveToFile("thumbnail", fmt.Sprintf("%s/thumbnail/%s.ori", config.AppConfig.UploadsDirectory, videoID.Hex()))
-	//if err != nil {
-	//	_ = c.Handler.DeleteVideo(videoID, c.userID)
-	//	fmt.Printf("[VideoControllerAuth::UploadVideo] failed saving thumbnail file. %+v\n", err)
-	//	return datatransfers.Response{Error: "Failed saving thumbnail image", Code: http.StatusInternalServerError}
-	//}
-	//err = c.Handler.NormalizeThumbnail(videoID.Hex())
-	//if err != nil {
-	//	_ = c.Handler.DeleteVideo(videoID, c.userID)
-	//	fmt.Printf("[VideoControllerAuth::UploadVideo] failed normalizing thumbnail image. %+v\n", err)
-	//	return datatransfers.Response{Error: "Failed normalizing thumbnail image", Code: http.StatusInternalServerError}
-	//}
 	// Trigger transcode sequence by cast-is
 	c.Handler.StartTranscode(videoID.Hex())
 	return datatransfers.Response{Code: http.StatusOK}

cast-be/go.mod

@@ -3,12 +3,14 @@ module github.com/daystram/cast/cast-be
 go 1.15
 
 require (
-	cloud.google.com/go v0.75.0 // indirect
-	cloud.google.com/go/pubsub v1.3.1
 	github.com/astaxie/beego v1.12.3
+	github.com/aws/aws-sdk-go v1.34.28
 	github.com/dgrijalva/jwt-go v3.2.0+incompatible
+	github.com/disintegration/imaging v1.6.2
 	github.com/fsnotify/fsnotify v1.4.9 // indirect
+	github.com/golang/protobuf v1.4.3 // indirect
 	github.com/golang/snappy v0.0.2 // indirect
+	github.com/google/go-cmp v0.5.4 // indirect
 	github.com/gorilla/websocket v1.4.2
 	github.com/klauspost/compress v1.11.7 // indirect
 	github.com/nareix/joy4 v0.0.0-20200507095837-05a4ffbb5369
@@ -17,17 +19,13 @@ require (
 	github.com/streadway/amqp v1.0.0
 	github.com/xdg/stringprep v1.0.1-0.20180714160509-73f8eece6fdc // indirect
 	go.mongodb.org/mongo-driver v1.4.5
-	go.opencensus.io v0.22.6 // indirect
 	golang.org/x/crypto v0.0.0-20201221181555-eec23a3978ad // indirect
-	golang.org/x/mod v0.4.1 // indirect
 	golang.org/x/net v0.0.0-20210119194325-5f4716e94777 // indirect
-	golang.org/x/oauth2 v0.0.0-20210126194326-f9ce19ea3013 // indirect
+	golang.org/x/sync v0.0.0-20201207232520-09787c993a3a // indirect
 	golang.org/x/sys v0.0.0-20210124154548-22da62e12c0c // indirect
 	golang.org/x/text v0.3.5 // indirect
-	golang.org/x/tools v0.1.0 // indirect
-	google.golang.org/api v0.38.0 // indirect
-	google.golang.org/genproto v0.0.0-20210126160654-44e461bb6506 // indirect
-	google.golang.org/grpc v1.35.0 // indirect
+	golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 // indirect
+	google.golang.org/protobuf v1.25.0 // indirect
 	gopkg.in/check.v1 v1.0.0-20200902074654-038fdea0a05b // indirect
 	gopkg.in/yaml.v2 v2.4.0 // indirect
 )