feat: add subscriptions

README.md

@@ -3,11 +3,12 @@
 ![Latest release](https://img.shields.io/github/v/release/QuokkaStake/cosmos-transactions-bot)
 [![Actions Status](https://github.com/QuokkaStake/cosmos-transactions-bot/workflows/test/badge.svg)](https://github.com/QuokkaStake/cosmos-transactions-bot/actions)
 
-cosmos-transactions-bot is a tool that listens to transactions with a specific filter on multiple chains and reports them to a Telegram channel.
+cosmos-transactions-bot is a tool that listens to transactions with a specific filter on multiple chains
+and reports them to a Telegram channel.
 
 Here's how it may look like:
 
-![Telegram](https://raw.githubusercontent.com/QuokkaStake/cosmos-transactions-bot/master/images/telegram.png)
+![Telegram](https://raw.githubusercontent.com/QuokkaStake/cosmos-transactions-bot/main/images/telegram.png)
 
 ## How can I set it up?
 
@@ -31,7 +32,8 @@ Then we need to create a systemd service for our app:
 sudo nano /etc/systemd/system/cosmos-transactions-bot.service
 ```
 
-You can use this template (change the user to whatever user you want this to be executed from. It's advised to create a separate user for that instead of running it from root):
+You can use this template (change the user to whatever user you want this to be executed from. It's advised
+to create a separate user for that instead of running it from root):
 
 ```
 [Unit]
@@ -69,29 +71,100 @@ sudo journalctl -u cosmos-transactions-bot -f --output cat
 
 ## How does it work?
 
-There are multiple nodes this app is connecting to via Websockets (see [this](https://docs.tendermint.com/master/rpc/#/Websocket/subscribe) for more details) and subscribing to the queries that are set in config. When a new transaction matching the filters is found, it's put through a deduplication filter first, to make sure we don't send the same transaction twice. Then each message in transaction is enriched (for example, if someone claims rewards, the app fetches Coingecko price and validator rewards are claimed from). Lastly, each of these transactions are sent to a reporter (currently Telegram only) to notify those who need it.
+There are multiple nodes this app is connecting to via Websockets (see [this](https://docs.tendermint.com/master/rpc/#/Websocket/subscribe) for more details) and subscribing
+to the queries that are set in config. When a new transaction matching the filters is found, it's put through
+a deduplication filter first, to make sure we don't send the same transaction twice. Then each message
+in transaction is enriched (for example, if someone claims rewards, the app fetches Coingecko price
+and validator rewards are claimed from). Lastly, each of these transactions are sent to a reporter
+(currently Telegram only) to notify those who need it.
 
 ## How can I configure it?
 
-All configuration is done with a `.toml` file, which is passed to an app through a `--config` flag. See `config.example.toml` for reference.
+All configuration is done with a `.toml` file, which is passed to an app through a `--config` flag.
+See `config.example.toml` for reference.
+
+### Chains, subscriptions, chain subscriptions and reporters
+
+This app's design is quite complex to allow it to be as flexible as possible.
+There are the main objects that this app has:
+
+- reporter - something that acts as a destination point (e.g. Telegram bot) and maybe allows you as a user
+to interact with it in a special way (like, setting aliases etc.)
+- chain - info about chain itself, its denoms, queries (see below), nodes used to receive data from, etc.
+- subscription - info about which set of chains and their events to send to which reporter,
+has many chain subscriptions
+- chain subscription - info about which chain to receive data from, filters on which events to match
+(see below) and how to process errors/unparsed/unsupported messages, if any.
+
+Each chain has many chain subscriptions, each subscription has one reporter, each chain subscription
+has one chain and many filters.
+
+Generally speaking, the workflow of the app looks something like this:
+
+![Schema](https://raw.githubusercontent.com/QuokkaStake/cosmos-transactions-bot/main/images/schema.png)
+
+This allows to build very flexible setups. Here's the example of the easy and the more difficult setup.
+
+1) "I want to receive all transactions sent from my wallet on chain A, B and C to my Telegram channel"
+
+You can do it the following way:
+- have 1 reporter, a Telegram channel
+- have 3 chains, A, B and C, and their configs
+- have 1 subscription, with Telegram reporter and 3 chain subscriptions inside (one for chain A, B and C
+with 1 filter each matching transfers from wallets on these chains)
+
+2) "I want to receive all transactions sent from my wallet on chains A, B and C to one Telegram chat,
+all transactions that are votes on chains A and B to another Telegram chat, and all transactions that are delegations
+with amount more than 10M $TOKEN on chain C to another Telegram chat"
+
+That's also manageable. You can do the following:
+- reporter 1, "first", a bot that sends messages to Telegram channel 1
+- reporter 2, "second", a bot that sends messages to Telegram channel 2
+- reporter 3, "third", a bot that sends messages to Telegram channel 3
+- chain A and its config
+- chain B and its config
+- chain C and its config
+- subscription 1, let's call it "my-wallet-sends", with reporter "first" and the following chain subscriptions
+- - chain subscription 1, chain A, 1 filter matching transfers from my wallet on chain A
+- - chain subscription 2, chain B, 1 filter matching transfers from my wallet on chain B
+- - chain subscription 3, chain C, 1 filter matching transfers from my wallet on chain C
+- subscription 2, let's call it "all-wallet-votes", with reporter "second" and the following chain subscriptions
+- - chain subscription 1, chain A, 1 filter matching any vote on chain A
+- - chain subscription 2, chain B, 1 filter matching any vote on chain B
+- subscription 3, let's call it "whale-votes", with reporter "third" and the following chain subscription
+- - chain subscription 1, chain C, 1 filter matching any delegations with amount more than 10M $TOKEN on chain C
+
+See config.example.toml for real-life examples.
 
 ### Queries and filters
 
-This is quite complex and deserves a special explanation.
+This is another quite complex topic and deserves a special explanation.
+
+When a node starts, it connects to a Websocket of the fullnode and subscribes to queries (`queries` in `.toml` config).
+If there's a transaction that does not match these filters, a fullnode won't emit the event for it
+and this transaction won't reach the app.
 
-When a node starts, it connects to a Websocket of the fullnode and subscribes to queries (`queries` in `.toml` config). If there's a transaction that does not match these filters, a fullnode won't emit the event for it and this transaction won't reach the app.
+If using filters (`filters` in `.toml` config), when a transaction is received, all messages in the transaction
+are checked whether they match these filters, and can be filtered out (and the transaction itself would be filtered out
+if there are 0 non-filtered messages left).
 
-If using filters (`filters` in `.toml` config), when a transaction is received, all messages in the transaction are checked whether they match these filters, and can be filtered out (and the transaction itself would be filtered out if there are 0 non filtered messages left).
+Using filters can be useful is you have transactions with multiple messages, where you only need to know about one
+(for example, someone claiming rewards from your validator and other ones, when you need to know only about claiming
+from your validator).
 
-Using filters can be useful is you have transactions with multiple messages, where you only need to know about one (for example, someone claiming rewards from your validator and other ones, when you need to know only about claiming from your validator).
+Keep in mind that queries is set on the app level, while filters are set on a chain subscription level,
+so you can have some generic query on a chain, and more granular filter on each of your chain subscriptions.
 
 Filters should follow the same pattern as queries, but they can only match the following pattern (so no AND/OR support):
 - `xxx = yyy` (which would filter the transaction if key doesn't match value)
 - `xxx! = yyy` (which would filter the transaction if key does match value)
 
-Please note that the message would not be filtered out if it matches at least one filter. Example: you have a message that has `xxx = yyy` as events, and if using `xxx != yyy` and `xxx != zzz` as filters, it won't get filtered out (as it would not match the first filter but would match the second one).
+Please note that the message would not be filtered out if it matches at least one filter.
+Example: you have a message that has `xxx = yyy` as events, and if using `xxx != yyy` and `xxx != zzz` as filters,
+it won't get filtered out (as it would not match the first filter but would match the second one).
 
-You can always use `tx.height > 0`, which will send you the information on all transactions in chain, or check out something we have:
+You can always use `tx.height > 0`, which will send you the information on all transactions in chain,
+or check out something we have:
 
 
 ```
@@ -135,7 +208,10 @@ filters = [
 
 See [the documentation](https://docs.tendermint.com/master/rpc/#/Websocket/subscribe) for more information on queries.
 
-One important thing to keep in mind: by default, Tendermint RPC now only allows 5 connections per client, so if you have more than 5 filters specified, this will fail when subscribing to 6th one. If you own the node you are subscribing to, o fix this, change this parameter to something that suits your needs in `<fullnode folder>/config/config.toml`:
+One important thing to keep in mind: by default, Tendermint RPC now only allows 5 connections per client,
+so if you have more than 5 filters specified, this will fail when subscribing to 6th one.
+If you own the node you are subscribing to, o fix this, change this parameter to something that suits your needs
+in `<fullnode folder>/config/config.toml`:
 
 ```
 max_subscriptions_per_client = 5
@@ -144,12 +220,16 @@ max_subscriptions_per_client = 5
 ## Notifications channels
 
 Go to [@BotFather](https://t.me/BotFather) in Telegram and create a bot. After that, there are two options:
-- you want to send messages to a user. This user should write a message to [@getmyid_bot](https://t.me/getmyid_bot), then copy the `Your user ID` number. Also keep in mind that the bot won't be able to send messages unless you contact it first, so write a message to a bot before proceeding.
-- you want to send messages to a channel. Write something to a channel, then forward it to [@getmyid_bot](https://t.me/getmyid_bot) and copy the `Forwarded from chat` number. Then add the bot as an admin.
+- you want to send messages to a user. This user should write a message to [@getmyid_bot](https://t.me/getmyid_bot),
+then copy the `Your user ID` number. Also keep in mind that the bot won't be able to send messages
+unless you contact it first, so write a message to a bot before proceeding.
+- you want to send messages to a channel. Write something to a channel, then forward it to [@getmyid_bot](https://t.me/getmyid_bot)
+and copy the `Forwarded from chat` number. Then add the bot as an admin.
 
 Then run a program with Telegram config (see `config.example.toml` as example).
 
-You would likely want to also put only the IDs of trusted people to admins list in Telegram config, so the bot won't react to anyone writing messages to it except these users.
+You would likely want to also put only the IDs of trusted people to admins list in Telegram config, so the bot
+won't react to anyone writing messages to it except these users.
 
 Additionally, for the ease of using commands, you can put the following list as bot commands in @BotFather settings:
 

config.example.toml

@@ -1,11 +1,14 @@
-# Path to where aliases in .toml will be stored. If omitted, no aliases setting/displaying would work.
-aliases = "/home/monitoring/config/cosmos-transactions-bot-aliases.toml"
-# Timezone in which time (like undelegation finish time) will be displayed. Defaults to "Etc/GMT", so UTC+0
+# Path to where aliases in .toml will be stored.
+# If omitted, no aliases setting/displaying would work.
+aliases = "cosmos-transactions-bot-aliases.toml"
+# Timezone in which time (like undelegation finish time) will be displayed.
+# Defaults to "Etc/GMT", so UTC+0
 timezone = "Europe/Moscow"
 
 # Logging configuration
 [log]
-# Log level. Set to "debug" or "trace" to make it more verbose, or to "warn"/"error" to make it less verbose.
+# Log level. Set to "debug" or "trace" to make it more verbose, or to "warn"/"error"
+# to make it less verbose.
 # Defaults to "info"
 level = "info"
 # If true, all logs would be displayed in JSON. Useful if you are using centralized logging
@@ -14,47 +17,42 @@ json = false
 
 # Reporters configuration.
 [[reporters]]
+# Reporter name. Should be unique.
+name = "telegram-1"
 # Reporter type. Currently, the only supported type is "telegram", which is the default.
 type = "telegram"
 # Telegram config configuration. Required if the type is "telegram".
 # See README.md for more details.
+# Has 3 params:
+# - token - bot token
+# - chat - a chat/channel to post messages to
+# - admins - a whitelist of user IDs allowed to send commands to the bot, optional but recommended.
 telegram-config = { token = "xxx:yyy", chat = 12345, admins = [67890] }
 
-# Per-chain configuration. There can be multiple chains.
-[[chains]]
-# Chain codename, required.
+# There can be multiple reporters.
+[[reporters]]
+name = "telegram-2"
+type = "telegram"
+telegram-config = { token = "zzz:aaa", chat = 98765, admins = [43210] }
+
+# Subscriptions config. See README.md on how this schema works.
+[[subscriptions]]
+# Reporter name to send events matching this subscription to.
+# Should be one of the names of the reporters declared above, or the app won't start
+# with the config validation error
+reporter = "telegram-1"
+# Subscription name, for metrics. Should be unique.
+name = "subscription-1"
+
+# Chain subscriptions for this subscription.
+[[subscriptions.chains]]
+# Chain name. Should be one of the names declared below in chains section,
+# or the app won't start with the config validation error.
 name = "cosmos"
-# Chain pretty name, optional. If provided, would be used in reports, if not,
-# codename would be used.
-pretty-name = "Cosmos Hub"
-# Tendermint RPC nodes to subscribe to. At least one is required, better to have multiple ones
-# as a failover.
-tendermint-nodes = [
-    "https://rpc.cosmos.quokkastake.io:443",
-]
-# API nodes to get blockchain data (validators, proposals etc.) from.
-api-nodes = [
-    "https://api.cosmos.quokkastake.io",
-]
-# Queries, see README.md for details.
-queries = [
-    "tx.height > 0"
-]
 # Filter, see README.md for details.
 filters = [
     "message.action = '/cosmos.gov.v1beta1.MsgVote'",
 ]
-# Denoms list.
-denoms = [
-    # Each denom inside must have "denom" and "display-denom" fields and additionaly
-    # denom-coefficient (set to 1000000 by default) and coingecko-currency.
-    # Example: if there's a transfer transaction for 10,000,000 uatom,
-    # and the coingecko price for $ATOM is 10$ and if all fields are set,
-    # instead of displaying amount as `10000000.000000uatom` it would be displayed
-    # as `10.000000atom ($100.00)`.
-    # If coingecko-currency is omitted, no price would be displayed.
-    { denom = "uatom", display-denom = "atom", denom-coefficient = 1000000, coingecko-currency = "cosmos" }
-]
 # If set to true and there is a message not supported by this app,
 # it would post a message about that, otherwise it would ignore such a message.
 # Defaults to false.
@@ -84,6 +82,58 @@ filter-internal-messages = true
 # - `Error: RPC error -32000 - Server error: subscription was cancelled (reason: Tendermint exited)`
 # If this is set to true (default), such messages would be displayed, if not, they will be skipped.
 log-node-errors = true
+
+# There can be multiple chain subscriptions per subscription.
+[[subscriptions.chains]]
+name = "sentinel"
+filters = ["message.action = '/cosmos.staking.v1beta1.MsgDelegate'"]
+
+# There can also be multiple subscriptions. This one, for example,
+# sends everything to a different reporter.
+[[subscriptions]]
+name = "subscription-2"
+reporter = "telegram-2"
+[[subscriptions.chains]]
+name = "cosmos"
+filters = ["message.action = '/cosmos.staking.v1beta1.MsgUndelegate'"]
+[[subscriptions.chains]]
+name = "sentinel"
+filters = ["message.action = '/cosmos.staking.v1beta1.MsgBeginRedelegate'"]
+
+
+
+# Per-chain configuration.
+[[chains]]
+# Chain codename, required.
+name = "cosmos"
+# Chain pretty name, optional. If provided, would be used in reports, if not,
+# codename would be used.
+pretty-name = "Cosmos Hub"
+# Tendermint RPC nodes to subscribe to. At least one is required, better to have multiple ones
+# as a failover.
+tendermint-nodes = [
+    "https://rpc.cosmos.quokkastake.io:443",
+]
+# API nodes to get blockchain data (validators, proposals etc.) from.
+api-nodes = [
+    "https://api.cosmos.quokkastake.io",
+]
+# Queries, see README.md for details.
+# Defaults to ["tx.height > 0"], so basically all transactions on chain.
+queries = [
+    "tx.height > 0"
+]
+# Denoms list.
+denoms = [
+    # Each denom inside must have "denom" and "display-denom" fields and additionaly
+    # denom-coefficient (set to 1000000 by default) and coingecko-currency.
+    # Example: if there's a transfer transaction for 10,000,000 uatom,
+    # and the coingecko price for $ATOM is 10$ and if all fields are set,
+    # instead of displaying amount as `10000000.000000uatom` it would be displayed
+    # as `10.000000atom ($100.00)`.
+    # If coingecko-currency is omitted, no price would be displayed.
+    { denom = "uatom", display-denom = "atom", denom-coefficient = 1000000, coingecko-currency = "cosmos" }
+]
 # Explorer configuration.
 # Priorities:
 # 1) ping.pub
@@ -111,3 +161,14 @@ block-link-pattern = "https://mintscan.io/cosmos/blocks/%s"
 # A pattern for validator links for the explorer.
 validator-link-pattern = "https://mintscan.io/cosmos/validators/%s"
 
+
+# There can be multiple chains.
+[[chains]]
+name = "sentinel"
+pretty-name = "Sentinel"
+tendermint-nodes = ["https://rpc.sentinel.quokkastake.io:443"]
+api-nodes = ["https://api.sentinel.quokkastake.io"]
+denoms = [
+    { denom = "udvpn", display-denom = "dvpn",  coingecko-currency = "sentinel" }
+]
+mintscan-prefix = "sentinel"

pkg/alias_manager/alias_manager.go

@@ -4,7 +4,7 @@ import (
 	"os"
 
 	"main/pkg/config"
-	"main/pkg/config/types"
+	configTypes "main/pkg/config/types"
 
 	"github.com/BurntSushi/toml"
 	"github.com/rs/zerolog"
@@ -14,15 +14,15 @@ type Aliases *map[string]string
 type TomlAliases map[string]Aliases
 
 type ChainAliases struct {
-	Chain   *types.Chain
+	Chain   *configTypes.Chain
 	Aliases Aliases
 }
 type AllChainAliases map[string]*ChainAliases
 
 type AliasManager struct {
 	Logger  zerolog.Logger
 	Path    string
-	Chains  config.Chains
+	Chains  configTypes.Chains
 	Aliases AllChainAliases
 }
 
@@ -36,15 +36,15 @@ func (a AllChainAliases) ToTomlAliases() TomlAliases {
 }
 
 type ChainAliasesLinks struct {
-	Chain *types.Chain
-	Links map[string]types.Link
+	Chain *configTypes.Chain
+	Links map[string]configTypes.Link
 }
 
 func (a AllChainAliases) ToAliasesLinks() []ChainAliasesLinks {
 	aliasesLinks := make([]ChainAliasesLinks, 0)
 
 	for _, chainAliases := range a {
-		links := make(map[string]types.Link)
+		links := make(map[string]configTypes.Link)
 
 		if chainAliases.Aliases == nil {
 			continue