-
Notifications
You must be signed in to change notification settings - Fork 1
Home
Collecting High-resolution Cryptocurrency WebSocket L3 Equivalent Feeds with Libwebsockets, Workflows, ECS, S3, Athena and Lambda
Websockets tend to be unstable, especially on busy networks, be it an abnormal exit of a process, a server breakdown or broken network. As cryptocurrencies are traded around the clock, these occurrences may prevent successful trading. Trades, especially those with high leverage, may be liquidated, resulting in a loss. It would be unacceptable for a high-frequency trading algorithm to remain down for maintenance while major movements are occurring in the market, as this is exactly when a reliable feed is most needed.
The feed must be operational 24/7. An ideal solution involves a primary ECS microservice connected to one content delivery network, and a replica service pointing to another content delivery network. The trick however, is being able to respond to an outage in microsecond timescales. This is achieved by each microservice using the libwebsockets library, which is low level and includes policies dictating the handling of emergencies. Libwebsockets Libwebsockets is a simple-to-use, MIT-licensed, pure C library providing client and server for http/1, http/2, websockets, MQTT and other protocols in a security minded, lightweight, scalable and flexible way. It is easy to build and cross-build via cmake and is suitable for tasks from embedded RTOS to mass cloud serving.
Microservices are Independently Deployable and Cloneable Each microservice is independent, enabling continuous improvement and faster app updates. Microservices Reduce Downtime Through Fault Isolation If a specific microservice fails, you can isolate the failure to that single service and prevent cascading failures which would cause the app to crash. This fault isolation means that your critical application can stay up and running even when one of its modules fails. Microservices typically have small codebases, making them easier to maintain and deploy. It’s also much easier to keep the code clean and for teams to be wholly responsible for specific services.
AWS Lambda is a serverless computing service provided by Amazon Web Services (AWS). Users of AWS Lambda create functions, self-contained applications written in one of the supported languages and runtimes, and upload them to AWS Lambda, which executes those functions efficiently and flexibly. They can also be scheduled or used in an AWS workflow to instantiate a data feed microservice or to monitor the health of an existing feed. Pricing: $0.20 per 1M requests Comment: The free tier includes 1M requests. And latency is low with Python cold call 200ms, warm call 20ms
It is important to offer a final-level but thorough protection. This involves stereoscopic monitoring, ECS tasks, health status, performance indicators, access frequency and times, error abnormality, client interface (error abnormality, page performance, network status, hardware and software environment, etc.) and business indicators (Bybit LOB IDs incrementing, feed traffic, volumes of events per minute, volatility, rates). We will monitor all valuable indicators through pre-warning and quick post response, and provide corresponding responsible persons with different warning methods such as mail, SMS, and telephone for varying levels of problems.
ECS tasks are co-hosted with Bybit servers in Singapore.
Requirements Critical Features
- Bybit Inverse Perpetual BTCUSD L3
- Bybit USDT Perpetual BTCUSDT L3
- Bybit Inverse Perpetual Others L3
- Bybit USDT Perpetual Others L3
- Bybit funding rate
- Bybit maker and taker fees (No API)
- Bybit system and maintenance alerts (No API)
Important Features
- Binance Spots ALL L2 https://binance-docs.github.io/apidocs/spot/en/#websocket-market-streams
- Coinbase Spots ALL L3
- Binance Derivatives ALL L3 https://developers.coinbase.com/
Desirable Features
- Binance Spots ALL L3
- Uniswap ALL L2
- 1inchechange All L2
- Yearn Finance
- All JSON documents will need a type and version number stored in minute buckets and stored in Parquet format.
Other Data
- Contact Bybit via Telegram for Maker-Taker fees, funding rate.
Other Requirements
- Multiple Content Delivery Networks (Bybit) & (Bytick) can be used as a fallback in extreme emergencies bybitglobal.com
- Continuous integration toolchain with GitHub and high unit test coverage.
Trade Execution Leverage and Margin Checks In trade execution, we need to calculate the impact on margin and leverage level; this will be calculated pre-trade and compared with the WebSocket feed post-trade. Any discrepancies will be flagged and actioned according to the severity. Dev Ops will code all business rules into a C library, and checks must run in the microsecond range. Warning Levels Warning level will be set in a configuration to send out messages before liquidations occur. New Announcements There are often new announcements or changes to the margin rules. Should they impact our systems, we require the correct toolchain: continuous integration, testing, etc. - additionally, where we can, rolling back changes while handling multiple changes if need be.
BMLL integration
BMLL Data Lake sits at the core of our offering. This is composed of 4+ years of nano-second granularity data across global equities, ETFs & futures from over 45 exchanges and trading venues. GDA will be integrating their L3 crypto feeds into BMLL.
Sage Maker Integration with S3 data Lake
Organizations are often challenged to access and analyze their data when it is stored in different formats across various, disparate locations. With data continuously being collected from a variety of sources, if not properly addressed, this challenge only escalates as organizations age and grow. A lack of simplified access to data creates workflow bottlenecks as employees often need to request help from IT to access the information required to build, train, and deploy machine learning models. To address these challenges, organizations are forced to seek out a solution that offers a single source of truth that is readily accessible to the employees who need it. This is often difficult and costly to execute and maintain as an on-premises solution.
Latency
ByBit 2.3 ms Coinbase 1.5 ms Binance 3.2 ms
ByBit, we get L3 data grouped in 100ms bursts. We have many microservices focusing on the same L3 web socket endpoints for three Content Delivery Networks. Should one of the connections be lost, we have a policy file that will handle reconnection, plus cloud watch connection losses. Primary and Replica services reconcile the feeds, plus cross CDN reconciliations take place. As a result, our L3 feeds are highly accurate and highly available and transparent. We are storing approximately 50 Gb of daily data on ByBit alone, making this a big data enterprise. L3 Architecture Design pattern We create 2 Step Functions State machines to demonstrate the design patterns. The state machine is executed with a JSON spec as an input. The specs have two parts - 1) values for ECS cluster, subnets, security groups, S3 bucket etc. 2) list of ECS tasks to run. The figure below shows that this pattern (Pattern 1) uses AWS Lambda function to run ECS tasks. We call the Lambda function as ECS Task Launcher. It parses workflow specs, submits ECS tasks to ECS Cluster and invokes a second AWS Lambda function called ECS Task Monitor. ECS Task Monitor tracks the completion status of running ECS tasks. Each time it runs, it checks the number of completed tasks versus the total number of jobs submitted and Cloudwatch Logs.