diff --git a/README.md b/README.md index 972034c..5a802d0 100644 --- a/README.md +++ b/README.md @@ -14,6 +14,7 @@ This project is licensed under MIT License - see the [LICENSE](LICENSE) file for ## Architecture -**Note: this is subject to change during development process.** For up-to-date graph, see [arch.drawio](arch.drawio). +**Note: this is subject to change during development process.** +For up-to-date graph, see [arch.drawio](doc/src/arch.drawio). -![2022-03-25T14:04:34Z.png](https://imagedelivery.net/b21oeeg7p6hqWEI-IA5xDw/885a0e1b-5bcc-41d8-cc1b-45a7c4f6d700/public) \ No newline at end of file +![arch](doc/src/arch.svg) \ No newline at end of file diff --git a/arch.drawio b/arch.drawio deleted file mode 100644 index f860cfd..0000000 --- a/arch.drawio +++ /dev/null @@ -1,179 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - \ No newline at end of file diff --git a/doc/src/arch.drawio b/doc/src/arch.drawio new file mode 100644 index 0000000..90fceb3 --- /dev/null +++ b/doc/src/arch.drawio @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/doc/src/arch.svg b/doc/src/arch.svg new file mode 100644 index 0000000..3d9fe29 --- /dev/null +++ b/doc/src/arch.svg @@ -0,0 +1,3 @@ + + +
Twitter
Twitter
Youtube
Youtube
Bilibili
Bilibili
Tasks
Tasks
MongoDB
Tasks
MongoDB...
MongoDB
VTBs
MongoDB...
Add new resource (Task)
Add new resource (Task)
API
API
Users
Users
MongoDB
Users
MongoDB...
User config
User config
Retrieve and update VTB info
Retrieve and update VTB info
Translate
Translate
MIddlewares
MIddlewares
...
...
Delay
Delay
Twitter Worker
Twitter Worker
Youtube Worker
Youtube Worker
Bilibili Worker
Bilibili Worker
Worker Cluster
Worker Cluster
......
......
Telegram Bot
Telegram Bot
QQ Bot
QQ Bot
......
......
IM Integrations
IM Integrations
Rabbit
MQ
Rabbit...
λ
λ
Text is not SVG - cannot display
\ No newline at end of file diff --git a/doc/src/introduction.md b/doc/src/introduction.md index 1089570..6e90174 100644 --- a/doc/src/introduction.md +++ b/doc/src/introduction.md @@ -13,14 +13,13 @@ This new version attempts to utilize a more modern microservice architecture, an The system is designed to be flexible and easy to extend, which composed of multiple modules. Include: - A centralized MongoDB as shared data source -- A coordinator to distribute tasks -- A worker pool to execute tasks +- A worker cluster to execute tasks - A message queue to collect and dispatch messages generated by workers - Multiple middlewares to provide extra functionality - Multiple clients(bots) to send the message to subscribers - A Web UI for users to manage their subscribe preference - An API to manage the system and provide functionality to frontend and bots -![2022-04-08T16:29:56Z.png](https://imagedelivery.net/b21oeeg7p6hqWEI-IA5xDw/9e103fdd-69d2-4662-01e7-e78a229f3800/public) +![arch](arch.svg) -[arch.drawio]: https://github.com/suisei-cn/stargazer-reborn/blob/master/arch.drawio +[arch.drawio]: arch.drawio diff --git a/doc/src/workers/index.md b/doc/src/workers/index.md index 4684b1a..0ece0c4 100644 --- a/doc/src/workers/index.md +++ b/doc/src/workers/index.md @@ -1 +1,14 @@ # Workers + +In stargazer-reborn, sources to be traced are called *tasks*. +A task can be a Twitter account, a YouTube channel, a Twitch channel, etc. +All tasks are stored in the `tasks` collection in MongoDB. + +Each task is handled by a worker. +Say if there's a task of Twitter account `@suisei_hosimati`, the worker is responsible for fetching the latest tweets of this account, and pushing them to the message queue. And the relation between tasks and workers is one-to-many. + +To handle a great amount of tasks, stargazer-reborn uses a worker cluster. +The cluster is heterogeneous in the sense that each worker can handle only one kind of task. However, every worker is *equivalent* if they are of the same kind, i.e. there's no "central" or "master" worker, nor any kind of "coordinator" node. +Each worker connects to the database, the single source of truth, and the message queue, on its own. + +To evenly distribute the work load, each worker uses gossip-based SWIM protocol to discover the other workers in the cluster. After that, a consistent hash ring is built to determine the worker responsible for a given task. \ No newline at end of file