Skip to content

Commit

Permalink
minor optimizations to reward system backend and UI (#1825)
Browse files Browse the repository at this point in the history
* minor update to reward system UI

* update scripts / add proxy test script

* add / update reward system scripts and systemd services ; update reward system administration documentation

* add telegram reward notification bot

* add updated reward distribution script and .conf file template

* add updated skywire specs documents from skycoinpro/skywire-specs
  • Loading branch information
0pcom authored May 24, 2024
1 parent c31a700 commit 5ebb1e7
Show file tree
Hide file tree
Showing 78 changed files with 14,229 additions and 66 deletions.
60 changes: 20 additions & 40 deletions REWARDS.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ It should be noted that the system survey generation requires root for many of i

### Log & Survey Collection

The log collection and [reward processing](#reward-processing) happens hourly.
The log collection and [reward processing](#reward-processing) happens hourly via [skywire-reward.service](/scripts/rewards/services/skywire-reward.service) - triggered to run hourly by [skywire-reward.timer](/scripts/rewards/services/skywire-reward.timer).

The log collection run can be viewed here:
https://fiber.skywire.dev/log-collection
Expand All @@ -52,11 +52,16 @@ These `survey_whitelist` keys are specified by the [conf service](https://conf.s

The collected surveys are then checked and backed up.

The collection of surveys and the reward calculation happens hourly - as configured by the systemd service and timer
The following scripts are used by the reward system:

[`getlogs.sh`](/scripts/rewards/getlogs.sh) - a wrapper script for survey and transport bandwidth log collection via `skywire cli log`
[`reward.sh`](/scripts/rewards/reward.sh) - a wrapper script for reward calculation via `skywire cli rewards`
[`gettps.sh`](/scripts/rewards/gettps.sh) - a wrapper script for collecting responses to transport setup-node requests via `skywire svc tps ls`
[`testproxies.sh`](/scripts/rewards/testproxies.sh) - WIP - a wrapper script for testing curl response time over the skywire socks5 proxy (not used for reward calculation)

### Reward Processing

The rewards are calculated by `skywire cli rewards calc` with the aid of [reward.sh](scripts/rewards/reward.sh) to produce the reward distribution data for the previous day's uptime.
The rewards are calculated by `skywire cli rewards calc` with the aid of [`reward.sh`](scripts/rewards/reward.sh) to produce the reward distribution data for the previous day's uptime.

### Per-IP reward limit

Expand All @@ -70,53 +75,27 @@ To avoid a user running multiple instances of skywire on virtual machines, the M

## Automation via systemd service

Automation of the hourly log & survey collection is accomplished via systemd service and timer executing [getlogs.sh](scripts/rewards/getlogs.sh) and [reward.sh](scripts/rewards/reward.sh)

/etc/systemd/system/skywire-reward.service
```
[Unit]
Description=skywire reward service
After=network.target
[Service]
Type=simple
User=user
Group=user
WorkingDirectory=/path/to/github.com/reward/rewards
ExecStart=/usr/bin/bash -c './getlogs.sh && ./reward.sh'
[Install]
WantedBy=multi-user.target
Automation of the hourly log & survey collection is accomplished via systemd service and timer

```
/etc/systemd/system/[`skywire-reward.service`](/scripts/rewards/services/skywire-reward.service)

**Note: change the user and working directory in the above systemd service**

This service is called by a timer
This service is called by a timer which triggers it to run hourly

/etc/systemd/system/skywire-reward.timer
```
[Unit]
Description=skywire reward timer
After=network.target
[Timer]
OnUnitActiveSec=1h
Unit=skywire-reward.service
[Install]
WantedBy=multi-user.target
```
/etc/systemd/system/[`skywire-reward.timer`](/scripts/rewards/services/skywire-reward.timer).


## fiber.skywire.dev

The 'frontend' of the reward system, is currently running at [fiber.skywire.dev](https://fiber.skywire.dev) and is reliant upon on the output of certain cli commands and some scripts
The 'frontend' of the reward system, is currently running at [fiber.skywire.dev](https://fiber.skywire.dev) and is reliant upon on the output of certain cli commands ~~and some scripts~~

[`skywire cli rewards ui`](cmd/skywire-cli/commands/rewards/ui.go) serves the reward system frontend or user interface - via http and dmsghttp.

A wrapper script [`scripts/getlogs.sh`](scripts/getlogs.sh) is used to redirect the output of `skywire cli log` to a file, which is displayed at:
The service which runs the reward system UI:
/etc/systemd/system/[`fiberreward.service`](/scripts/rewards/services/fiberreward.service)

A wrapper script [`getlogs.sh`](scripts/rewards/getlogs.sh) is used to redirect the output of `skywire cli log` to a file, which is displayed at:

https://fiber.skywire.dev/log-collection

Expand All @@ -126,7 +105,7 @@ Here shows links to the reward calculations and distribution data by day:

https://fiber.skywire.dev/skycoin-rewards

on each linked page, the distribution data is displayed, and a link to the explorer for that transaction if it was broadcast. Also displayed are the public keys and their reward shares, or the reason why they were not rewarded
on each linked page, the distribution data is displayed with a link to the explorer for that transaction if it was broadcast. Also displayed are the public keys and their reward shares, or the reason why they were not rewarded

The frontend may be run either with flags or by using a conf file such as the following:
fr.conf
Expand Down Expand Up @@ -166,5 +145,6 @@ REWARD_SYS_URL="dmsg://<reward-system-public-key>:80"

before the script is run and the transaction is attempted to be broadcast, it's crucial to check that the hourly [log collection and reward calculation](https://fiber.skywire.dev/log-collection) is not ongoing.

### Reward Notifications

When the transaction is then broadcast, it's transaction ID recorded in a file which is monitored by the reward telegram bot, which generates a notification in https://t.me/skywire_reward. **Note: this will eventually be supplemented with or replaced by a notification via skychat.**
When the transaction is broadcast by the reward system, it's transaction ID is recorded by appending a file which is monitored by the reward telegram bot. The telegram bot will then generate a notification in https://t.me/skywire_reward when a change to that file is detected. **Note: this will eventually be supplemented with or replaced by a notification via skychat.**
2 changes: 1 addition & 1 deletion cmd/skywire-cli/commands/log/st.go
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ func makeTree() {
var coloredFile string
if kid == "proxy" {
testtime, _ := script.File(proxyCSV).Match(dirNode).Replace(",", " ").Column(2).String() //nolint
testres, _ := script.File(proxyCSV).Match(dirNode).Replace(",", " ").Column(3).String() //nolint
testres, _ := script.File(proxyCSV).Match(dirNode).Replace(",", " ").Column(3).String() //nolint
if testtime != "" && testres != "" {
coloredFile = fmt.Sprintf("%s %s %s", pterm.Green("proxy"), strings.ReplaceAll(testtime, "\n", ""), strings.ReplaceAll(testres, "\n", ""))
} else {
Expand Down
117 changes: 117 additions & 0 deletions cmd/skywire-cli/commands/rewards/tgbot.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
// Package clirewards cmd/skywire-cli/commands/rewards/tgbot.go
package clirewards

import (
"fmt"
"log"
"os"
"strconv"
"strings"
"time"

"github.com/bitfield/script"
"github.com/spf13/cobra"
tele "gopkg.in/telebot.v3"
)

var filePath string

func init() {
RootCmd.AddCommand(
tgbotCmd,
)
tgbotCmd.Flags().StringVarP(&filePath, "watch", "w", "../reward/rewards/transactions0.txt", "File to watch - file where reward transaction IDs are recorded")

}

var tgbotCmd = &cobra.Command{
Use: "bot",
Short: "reward notification telegram bot",
Long: "reward notification telegram bot",
Run: func(_ *cobra.Command, _ []string) {
chatIDStr := os.Getenv("TG_CHAT_ID")
chatID, err := strconv.ParseInt(chatIDStr, 10, 64)
if err != nil {
log.Fatalf("failed to parse chat ID: %v", err)
}
pref := tele.Settings{
Token: os.Getenv("TG_BOT_TOKEN"),
Poller: &tele.LongPoller{Timeout: 10 * time.Second},
}

b, err := tele.NewBot(pref)
if err != nil {
log.Fatal(err)
return
}

tgbotscript := `#!/bin/bash
_stats() {
# [[ $1 == "" ]] && return
cat ` + strings.TrimSuffix(filePath, "/transactions0.txt") + `/hist/$(find ` + strings.TrimSuffix(filePath, "/transactions0.txt") + `/hist/ -name "*.txt" -type f -exec grep -l "$1" {} + | xargs -I{} basename {} | tr -d ".txt")_stats.txt
}
`

// var lastModTime time.Time
lastModTime, err := os.Stat(filePath)
if err != nil {
log.Fatal(err)
return
}
// Use a goroutine to periodically check the file for changes
go func() {
for {
time.Sleep(2 * time.Second)

fileInfo, err := os.Stat(filePath)
if err != nil {
log.Printf("Error checking file info: %s", err)
continue
}

if fileInfo.ModTime().After(lastModTime.ModTime()) {
// The file has been modified since the last check, get the last line which is the most recent txid
lastLine, err := script.File(filePath).Last(1).String()
if err != nil {
log.Printf("Error getting last line of file: %v", err)
continue
}
if lastLine != "" {
tmpFile, err := os.CreateTemp(os.TempDir(), "*.sh")
if err != nil {
return
}
if err := tmpFile.Close(); err != nil {
return
}
_, _ = script.Exec(`chmod +x ` + tmpFile.Name()).String() //nolint
_, _ = script.Echo(tgbotscript).WriteFile(tmpFile.Name()) //nolint
stats, err := script.Exec(`bash -c 'source ` + tmpFile.Name() + ` ; _stats ` + lastLine + `'`).String() //nolint
if err != nil {
log.Printf("Error getting statistics: %v", err)
continue
}
os.Remove(tmpFile.Name()) //nolint

dateforlink, err := script.Echo(stats).First(1).Replace("date: ", "").String()
if err != nil {
log.Printf("Error getting date for link: %v", err)
continue
}
msg := fmt.Sprintf("Rewards have been distributed!\n\nhttps://explorer.skycoin.com/app/transaction/%s\n\n%s\n\nhttps://fiber.skywire.dev/skycoin-rewards/hist/%s", lastLine, stats, dateforlink)
// Send the last line to the Telegram chat
_, err = b.Send(&tele.Chat{ID: chatID}, msg)
if err != nil {
log.Printf("Error sending message to Telegram chat: %s", err)
continue
}
}

lastModTime = fileInfo
}
}
}()

b.Start()
},
}
10 changes: 6 additions & 4 deletions cmd/skywire-cli/commands/rewards/ui.go
Original file line number Diff line number Diff line change
Expand Up @@ -360,8 +360,8 @@ func server() {
tpstats, _ := script.Exec("skywire cli tp tree -s").Match("Count of transports:").Replace("Count of transports: ", "").Replace("\n", "").String() //nolint
tpcount, _ := strconv.Atoi(tpstats) //nolint
if tpcount < 400 {
tpTree, _ := script.Exec("skywire cli tp tree").Bytes() //nolint
c.Writer.Write(ansihtml.ConvertToHTML(tpTree)) //nolint
tpTree, _ := script.Exec("skywire cli tp tree").Bytes() //nolint
c.Writer.Write(ansihtml.ConvertToHTML(tpTree)) //nolint
c.Writer.Flush()
} else {
c.Writer.Write([]byte(fmt.Sprintf("Transport count: %v exceeds server resources to map", tpcount))) //nolint
Expand Down Expand Up @@ -851,7 +851,9 @@ func server() {
c.Writer.Header().Set("Transfer-Encoding", "chunked")
_, err := time.Parse("2006-01-02", c.Param("date"))
if err != nil {
if strings.Contains(c.Param("date"), "_rewardtxn0.csv") {
_, err1 := time.Parse("2006-01-02", strings.Replace(c.Param("date"), "_rewardtxn0.csv", "", -1))
_, err2 := time.Parse("2006-01-02", strings.Replace(c.Param("date"), "_stats.txt", "", -1))
if err1 != nil || err2 != nil {
filetoserve, err := script.File("rewards/hist/" + c.Param("date")).Bytes()
if err == nil {
c.Writer.Header().Set("Content-Type", "text/plain")
Expand Down Expand Up @@ -1261,7 +1263,7 @@ func serveSyntaxHighlighted(c *gin.Context) {
return
}
c.Status(http.StatusOK)
c.Writer.Write(buf.Bytes()) //nolint
c.Writer.Write(buf.Bytes()) //nolint
}

type ginHandler struct {
Expand Down
1 change: 1 addition & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ require (
golang.org/x/sync v0.3.0
golang.org/x/sys v0.17.0
golang.zx2c4.com/wireguard v0.0.0-20230223181233-21636207a675
gopkg.in/telebot.v3 v3.2.1
)

require (
Expand Down
Loading

0 comments on commit 5ebb1e7

Please sign in to comment.