Skip to content

Commit

Permalink
Improve README.md
Browse files Browse the repository at this point in the history
  • Loading branch information
@Soniclev
Soniclev committed May 22, 2024
1 parent a53949d commit 1f8091d
Showing 1 changed file with 49 additions and 50 deletions.
99 changes: 49 additions & 50 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,65 +1,65 @@
# System for monitoring prices between CS.MONEY and Steam market
# System for monitoring prices between CS.MONEY and Steam Market

# High level features
* Supports proxies (http, socks4, socks5)
## High-Level Features
* Supports proxies (HTTP, SOCKS4, SOCKS5)
* Built using asyncio + uvloop
* Resistant to failure
* Covered by unit-tests (almost 100% coverage)
* Covered by unit tests (almost 100% coverage)
* Most obvious bottlenecks are optimized
* Created simple CI pipeline
* For some features added tracing support using Zipkin
* Created a simple CI pipeline
* Added tracing support for some features using Zipkin

# Technologies
## Technologies
* Python 3.10, Asyncio, Poetry
* Docker, docker-compose
* Docker, Docker Compose
* Redis
* RabbitMQ
* Pytest
* Pylint, mypy, black

# Simplified flow chart
## Simplified Flow Chart

```mermaid
graph LR;
steam_parser(Steam parser)-->rabbitmq;
csmoney_parser(CS.MONEY parser)-->rabbitmq;
steam_parser(Steam Parser)-->rabbitmq;
csmoney_parser(CS.MONEY Parser)-->rabbitmq;
rabbitmq(RabbitMQ);
rabbitmq-->worker;
worker(Worker #N)-->Redis;
Redis[(Redis)];
Redis-->bot(Telegram bot);
Redis-->bot(Telegram Bot);
```

Both parsers send responses to appropriate queues (one for Steam and another one for CS.MONEY). After it, a worker fetch response from the queue and process it. The result of processing is uploaded to a Redis instance. A telegram bot periodically retrieve results from Redis and compute profit for each market item.

# How to run (last tested in April 2024)
1. Open the bot.dev.env file and locate the line that starts with `TELEGRAM_API_TOKEN=`. Set the value of the `TELEGRAM_API_TOKEN` variable to your own Telegram bot's API token. To obtain the token, you can create a bot on Telegram and follow their instructions.
Additionally, you need to specify the Telegram user IDs that will have access to this bot. To find your Telegram user ID, you can message the bot called https://t.me/myidbot, and it will provide you with your ID. If you have multiple user IDs, separate them with commas (",").
Here's an example of how it should look:
```
...
TELEGRAM_API_TOKEN=123456:ABC-DEF1234ghIkl-zyx57W2v1u123ew11
TELEGRAM_WHITELIST=12345,45678
```
Both parsers send responses to appropriate queues (one for Steam and another for CS.MONEY). After that, a worker fetches the response from the queue and processes it. The result of processing is uploaded to a Redis instance. A Telegram bot periodically retrieves results from Redis and computes profit for each market item.

## How to Run (Last Tested in April 2024)
1. Open the `bot.dev.env` file and locate the line that starts with `TELEGRAM_API_TOKEN=`. Set the value of the `TELEGRAM_API_TOKEN` variable to your own Telegram bot's API token. To obtain the token, you can create a bot on Telegram and follow their instructions.
Additionally, you need to specify the Telegram user IDs that will have access to this bot. To find your Telegram user ID, you can message the bot at https://t.me/myidbot, and it will provide you with your ID. If you have multiple user IDs, separate them with commas (",").
Here's an example of how it should look:
```
...
TELEGRAM_API_TOKEN=123456:ABC-DEF1234ghIkl-zyx57W2v1u123ew11
TELEGRAM_WHITELIST=12345,45678
```
2. Create/open the `utils_mount/csmoney_proxies.txt` and `utils_mount/steam_proxies.txt` files.
Add the proxies you want to use in the following format: `<protocol>://<user>:<pass>@<host>:<port>`.
If the proxy requires authentication (username and password), include them in the URL.
If no authentication is needed, simply provide the host and port. Here are some examples:
Add the proxies you want to use in the following format: `<protocol>://<user>:<pass>@<host>:<port>`.
If the proxy requires authentication (username and password), include them in the URL.
If no authentication is needed, simply provide the host and port. Here are some examples:
* `http://user:[email protected]:1234`
* `http://1.1.1.1:1234`
* `https://user:[email protected]:1234`
* `socks4://1.1.1.1:1234` - (note: socks4 protocol don't support authorization)
* `socks4://1.1.1.1:1234` - (note: the SOCKS4 protocol doesn't support authorization)
* `socks5://user:[email protected]:1234`
3. Open your terminal or command prompt and run the following command:
```
docker-compose --env-file prod.env up -d
```
4. Setup initial settings for your in your Telegram bot:
```
docker-compose --env-file prod.env up -d
```
4. Set up initial settings for your Telegram bot:
1. `/set_min_price 0` - This command sets the minimum price limit. Any item with a price lower than this value will not be displayed.
2. `/set_limit -30` - This command sets the profit limit for withdrawals. For example, if you set it to -30, only items with a profit margin greater than -30% will be displayed.

Available Telegram bot commands:
* `/offers` - Displays the current available offers.
* `/offers` - Displays the currently available offers.
* `/set_min_price <price>` - Sets the minimum price limit. Any item with a price lower than this value will not be displayed.
* `/set_limit <profit margin>` - Sets the profit limit for withdrawals. The profit margin includes any applicable fees and represents the minimum acceptable profit for an item to be displayed.
* `/settings` - Displays the current settings for withdrawals.
Expand All @@ -68,38 +68,37 @@ With these adjusted instructions, you should be able to easily run the applicati

![Bot work screenshot](images/bot.png "Bot work screenshot")

# Steam and CS.MONEY parsers
## Steam and CS.MONEY Parsers

The Steam parser fetch skins that should be processed from a scheduler. The scheduler for Steam returns a list of items that should be fetched. After successful invoke of Steam endpoint the scheduler postpone the item for 15 minutes. In case of any error the item will be rescheduled without postpone. Throughput of parser depends on amount of free proxy. After each request, a proxy is frozen for 3 seconds to avoid 429 Too Many Requests error. A successful response from Steam is put in a RabbitMQ queue. Also, the Steam parser loads sell history for each skin in similar way, that will be processed by a worker. Can be found [here](price_monitoring/parsers/steam)
The Steam parser fetches skins that should be processed from a scheduler. The scheduler for Steam returns a list of items that should be fetched. After a successful invocation of the Steam endpoint, the scheduler postpones the item for 15 minutes. In case of any error, the item will be rescheduled without postponement. The throughput of the parser depends on the number of free proxies. After each request, a proxy is frozen for 5 seconds to avoid the 429 Too Many Requests error. A successful response from Steam is put in a RabbitMQ queue. Additionally, the Steam parser loads the sell history for each skin in a similar way, which will be processed by a worker. The implementation can be found [here](price_monitoring/parsers/steam).

The CS.MONEY parser works in a similar way, but has some differences. The parser enumerate inventory by loading 60 items per request. These request grouped in tasks by price range, because it's impossible to load items after 5000th offset. Each request that contains up to 60 items will be put in a RabbitMQ queue. If it's impossible to make successful request after several retries, the task will be marked as failed. Can be found [here](price_monitoring/parsers/csmoney)
The CS.MONEY parser works similarly but has some differences. The parser enumerates the inventory by loading 60 items per request. These requests are grouped into tasks by price range because it's impossible to load items after the 5000th offset. Each request that contains up to 60 items will be put in a RabbitMQ queue. If it's impossible to make a successful request after several retries, the task will be marked as failed. The implementation can be found [here](price_monitoring/parsers/csmoney).

At this moment only one instance of each parser can be running at one time.
At this moment, only one instance of each parser can be running at a time.

# Worker
## Worker

A worker fetch results from queue, process them and pass results to Redis. Applied processors can be found [here](price_monitoring/worker/processing).
A worker fetches results from the queue, processes them, and passes results to Redis. The applied processors can be found [here](price_monitoring/worker/processing).

For sell history of a skin, a worker determinate stability of the item and sell probability for each price for last week: ![Sell history analyzer](images/sell_history.jpg "Sell history analyzer")
For the sell history of a skin, a worker determines the stability of the item and the sell probability for each price for the last week: ![Sell history analyzer](images/sell_history.jpg "Sell history analyzer")

Several workers can be running at one time.
Several workers can be running at one time.

# Redis
## Redis

Prices for items from CS.MONEY stored directly in keys, because it allows:
* Set TTL for each pair of a name and price
* Fetch all items in one request
Prices for items from CS.MONEY are stored directly in keys, allowing for:
* Setting TTL for each pair of a name and price
* Fetching all items in one request

Items can have 7 days trade ban and cannot be withdrawn immediately. That's why
a key contains "locked" or "unlocked" prefix.
Items can have a 7-day trade ban and cannot be withdrawn immediately. That's why a key contains the "locked" or "unlocked" prefix.
```
prices:csmoney:locked:Sealed Graffiti | Choke (Frog Green):0.25
prices:csmoney:unlocked:StatTrak™ M4A1-S | Leaded Glass (Field-Tested):16.32
```

Prices for items from Steam stored in different way. Each item has it's own unique key like `prices:steam:P2000 | Panther Camo (Battle-Scarred)`. Prices stored as value in simple format `1.3:1.49`. The first number means the highest buy order, the second one means the lowest sell order.
Prices for items from Steam are stored differently. Each item has its own unique key like `prices:steam:P2000 | Panther Camo (Battle-Scarred)`. Prices are stored as values in a simple format `1.3:1.49`. The first number means the highest buy order, the second one means the lowest sell order.

Implementation can be found [here](price_monitoring/storage)
The implementation can be found [here](price_monitoring/storage).

# Item Orders Histogram fix
Feel free to adapt to any new changes made by Steam regarding parsing the item orders histogram. To properly retrieve the item orders histogram, you can follow the solution provided in the following link: [Retrieve Item Orders Histogram Solution](https://gist.github.com/Soniclev/981f8adbc7c6a68350aff9e2a877ed1e) or [here](price_monitoring/parsers/steam/parser/steam_orders_parser.py).
## Item Orders Histogram Fix
Feel free to adapt to any new changes made by Steam regarding parsing the item orders histogram. To properly retrieve the item orders histogram, you can follow the solution provided in the following link: [Retrieve Item Orders Histogram Solution](https://gist.github.com/Soniclev/981f8adbc7c6a68350aff9e2a877ed1e) or [here](price_monitoring/parsers/steam/parser/steam_orders_parser.py).

0 comments on commit 1f8091d

Please sign in to comment.