crypto_msg_parser::parse_trade() : Cannot drop a runtime in a context where blocking is not allowed. This happens when a runtime is dropped from within an asynchronous context. #58

panicfarm · 2023-01-09T21:32:27Z

I am using the following code (collected from your examples):

 1 use crypto_crawler::{crawl_trade, MarketType};
  2
  3 #[tokio::main(flavor = "multi_thread")]
  4 async fn main() {
  5     let (tx, rx) = std::sync::mpsc::channel();
  6
  7     tokio::task::spawn(async move {
  8         crawl_trade(
  9             "kucoin",
 10             MarketType::LinearSwap,
 11             Some(&["SOLUSDTM".to_string()]),
 12             /*
 13             MarketType::Spot,
 14             Some(&["SOL-USDT".to_string()]),
 15             */
 16             tx,
 17         )
 18         .await
 19     });
 20     for msg in rx {
 21         //https://github.com/crypto-crawler/crypto-crawler-rs/blob/main/crypto-crawler/src/msg.rs
 22         let trades =
 23             crypto_msg_parser::parse_trade(&msg.exchange, msg.market_type, &msg.json).unwrap();
 24         let trade = &trades[0];
 25         println!("{:#?}", trade);
 26     }
 27 }
~

Only for MarketType::LinearSwap (but not for MarketType::Spot because https://github.com/crypto-crawler/crypto-msg-parser/blob/3ba78a12e34e9b93468e6dc26d68ad3dec620a97/crypto-contract-value/src/exchanges/kucoin.rs#L198 never gets called for Spot) I get the following panic with the debug build (this never happens with a release build, i think it's something debug-specific? with blocking http_get("https://api-futures.kucoin.com/api/v1/contracts/active") reqwest in crypto-contract-value-1.7.7/src/exchanges/kucoin.rs:181 ).

alecm@ark2:~/qkt$ ./target/debug/qk
thread 'main' panicked at 'Cannot drop a runtime in a context where blocking is not allowed. This happens when a runtime is dropped from within an asynchronous context.', /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.23.0/src/runtime/blocking/shutdown.rs:51:21
stack backtrace:
   0: std::panicking::begin_panic
             at /rustc/8b0c05d9ad7121cdb97600f261bcd5f04c8db20d/library/std/src/panicking.rs:588:12
   1: tokio::runtime::blocking::shutdown::Receiver::wait
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.23.0/src/runtime/blocking/shutdown.rs:51:21
   2: tokio::runtime::blocking::pool::BlockingPool::shutdown
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.23.0/src/runtime/blocking/pool.rs:261:12
   3: <tokio::runtime::blocking::pool::BlockingPool as core::ops::drop::Drop>::drop
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.23.0/src/runtime/blocking/pool.rs:278:9
   4: core::ptr::drop_in_place<tokio::runtime::blocking::pool::BlockingPool>
             at /rustc/8b0c05d9ad7121cdb97600f261bcd5f04c8db20d/library/core/src/ptr/mod.rs:487:1
   5: core::ptr::drop_in_place<tokio::runtime::runtime::Runtime>
             at /rustc/8b0c05d9ad7121cdb97600f261bcd5f04c8db20d/library/core/src/ptr/mod.rs:487:1
   6: reqwest::blocking::wait::enter
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/reqwest-0.11.13/src/blocking/wait.rs:76:21
   7: reqwest::blocking::wait::timeout
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/reqwest-0.11.13/src/blocking/wait.rs:13:5
   8: reqwest::blocking::client::ClientHandle::new
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/reqwest-0.11.13/src/blocking/client.rs:1038:15
   9: reqwest::blocking::client::ClientBuilder::build
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/reqwest-0.11.13/src/blocking/client.rs:103:9
  10: crypto_contract_value::exchanges::utils::http_get
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/crypto-contract-value-1.7.7/src/exchanges/utils.rs:7:18
  11: crypto_contract_value::exchanges::kucoin::fetch_linear_multipliers
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/crypto-contract-value-1.7.7/src/exchanges/kucoin.rs:181:22
  12: crypto_contract_value::exchanges::kucoin::LINEAR_CONTRACT_VALUES::{{closure}}
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/crypto-contract-value-1.7.7/src/exchanges/kucoin.rs:152:23
  13: core::ops::function::FnOnce::call_once
             at /rustc/8b0c05d9ad7121cdb97600f261bcd5f04c8db20d/library/core/src/ops/function.rs:251:5
  14: core::ops::function::FnOnce::call_once
             at /rustc/8b0c05d9ad7121cdb97600f261bcd5f04c8db20d/library/core/src/ops/function.rs:251:5
  15: once_cell::sync::Lazy<T,F>::force::{{closure}}
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/once_cell-1.17.0/src/lib.rs:1275:28
  16: once_cell::sync::OnceCell<T>::get_or_init::{{closure}}
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/once_cell-1.17.0/src/lib.rs:1085:57
  17: once_cell::imp::OnceCell<T>::initialize::{{closure}}
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/once_cell-1.17.0/src/imp_std.rs:82:23
  18: core::ops::function::impls::<impl core::ops::function::FnMut<A> for &mut F>::call_mut
             at /rustc/8b0c05d9ad7121cdb97600f261bcd5f04c8db20d/library/core/src/ops/function.rs:297:13
  19: once_cell::imp::initialize_or_wait
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/once_cell-1.17.0/src/imp_std.rs:206:20
  20: once_cell::imp::OnceCell<T>::initialize
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/once_cell-1.17.0/src/imp_std.rs:78:9
  21: once_cell::sync::OnceCell<T>::get_or_try_init
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/once_cell-1.17.0/src/lib.rs:1126:13
  22: once_cell::sync::OnceCell<T>::get_or_init
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/once_cell-1.17.0/src/lib.rs:1085:19
  23: once_cell::sync::Lazy<T,F>::force
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/once_cell-1.17.0/src/lib.rs:1274:13
  24: <once_cell::sync::Lazy<T,F> as core::ops::deref::Deref>::deref
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/once_cell-1.17.0/src/lib.rs:1335:13
  25: crypto_contract_value::exchanges::kucoin::get_contract_value
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/crypto-contract-value-1.7.7/src/exchanges/kucoin.rs:198:40
  26: crypto_contract_value::get_contract_value
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/crypto-contract-value-1.7.7/src/lib.rs:22:21
  27: crypto_msg_parser::exchanges::utils::calc_quantity_and_volume
             at /home/alecm/.cargo/git/checkouts/crypto-msg-parser-aaa16a2660930560/8bc9eb7/crypto-msg-parser/src/exchanges/utils.rs:41:9
  28: crypto_msg_parser::exchanges::kucoin::kucoin_swap::parse_trade
             at /home/alecm/.cargo/git/checkouts/crypto-msg-parser-aaa16a2660930560/8bc9eb7/crypto-msg-parser/src/exchanges/kucoin/kucoin_swap.rs:64:62
  29: crypto_msg_parser::exchanges::kucoin::parse_trade
             at /home/alecm/.cargo/git/checkouts/crypto-msg-parser-aaa16a2660930560/8bc9eb7/crypto-msg-parser/src/exchanges/kucoin/mod.rs:118:9
  30: crypto_msg_parser::parse_trade
             at /home/alecm/.cargo/git/checkouts/crypto-msg-parser-aaa16a2660930560/8bc9eb7/crypto-msg-parser/src/lib.rs:97:21
  31: qk::main::{{closure}}
             at ./src/main.rs:23:13
  32: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/8b0c05d9ad7121cdb97600f261bcd5f04c8db20d/library/core/src/future/mod.rs:91:19
  33: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.23.0/src/runtime/park.rs:283:63
  34: tokio::runtime::coop::with_budget
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.23.0/src/runtime/coop.rs:102:5
  35: tokio::runtime::coop::budget
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.23.0/src/runtime/coop.rs:68:5
  36: tokio::runtime::park::CachedParkThread::block_on
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.23.0/src/runtime/park.rs:283:31
  37: tokio::runtime::context::BlockingRegionGuard::block_on
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.23.0/src/runtime/context.rs:295:13
  38: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.23.0/src/runtime/scheduler/multi_thread/mod.rs:66:9
  39: tokio::runtime::runtime::Runtime::block_on
             at /home/alecm/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.23.0/src/runtime/runtime.rs:284:45
  40: qk::main
             at ./src/main.rs:20:5
  41: core::ops::function::FnOnce::call_once
             at /rustc/8b0c05d9ad7121cdb97600f261bcd5f04c8db20d/library/core/src/ops/function.rs:251:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

PS. With the release build, i let it run for several days, but the websocket stopped getting new messages after 2 days. I rebuilt as a debug and ran into the panic above. Should I file a separate Issue on the stuck websocket (unsure how to debug that, strace showed waiting for some FUTEX)?

The text was updated successfully, but these errors were encountered:

soulmachine · 2023-01-10T01:38:18Z

Can you remove line numbers from your code snipet above? I'd like to run you code on my machine

panicfarm · 2023-01-10T02:12:46Z

Can you remove line numbers from your code snipet above? I'd like to run you code on my machine

Here you go

use crypto_crawler::{crawl_trade, MarketType};

#[tokio::main(flavor = "multi_thread")]
async fn main() {
    let (tx, rx) = std::sync::mpsc::channel();

    tokio::task::spawn(async move {
        crawl_trade(
            "kucoin",
            MarketType::LinearSwap,
            Some(&["SOLUSDTM".to_string()]),
            /*
            MarketType::Spot,
            Some(&["SOL-USDT".to_string()]),
            */
            tx,
        )
        .await
    });
    for msg in rx {
        //https://github.com/crypto-crawler/crypto-crawler-rs/blob/main/crypto-crawler/src/msg.rs
        let trades =
            crypto_msg_parser::parse_trade(&msg.exchange, msg.market_type, &msg.json).unwrap();
        let trade = &trades[0];
        println!("{:#?}", trade);
    }
}

and

[package]
name = "qk"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
crypto-crawler = { git = "https://github.com/crypto-crawler/crypto-crawler-rs", branch = "main" }
crypto-msg-parser = { git = "https://github.com/crypto-crawler/crypto-msg-parser", branch = "main" }
crypto-message = { git = "https://github.com/crypto-crawler/crypto-msg-parser", branch = "main"}

tokio = { version = "1", features = ["macros", "rt-multi-thread"] }

foonsun · 2023-01-26T12:16:27Z

Can you remove line numbers from your code snipet above? I'd like to run you code on my machine

Here you go

use crypto_crawler::{crawl_trade, MarketType};

#[tokio::main(flavor = "multi_thread")]
async fn main() {
    let (tx, rx) = std::sync::mpsc::channel();

    tokio::task::spawn(async move {
        crawl_trade(
            "kucoin",
            MarketType::LinearSwap,
            Some(&["SOLUSDTM".to_string()]),
            /*
            MarketType::Spot,
            Some(&["SOL-USDT".to_string()]),
            */
            tx,
        )
        .await
    });
    for msg in rx {
        //https://github.com/crypto-crawler/crypto-crawler-rs/blob/main/crypto-crawler/src/msg.rs
        let trades =
            crypto_msg_parser::parse_trade(&msg.exchange, msg.market_type, &msg.json).unwrap();
        let trade = &trades[0];
        println!("{:#?}", trade);
    }
}

and

[package]
name = "qk"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
crypto-crawler = { git = "https://github.com/crypto-crawler/crypto-crawler-rs", branch = "main" }
crypto-msg-parser = { git = "https://github.com/crypto-crawler/crypto-msg-parser", branch = "main" }
crypto-message = { git = "https://github.com/crypto-crawler/crypto-msg-parser", branch = "main"}

tokio = { version = "1", features = ["macros", "rt-multi-thread"] }

I have the same issue.any resolutions? @panicfarm

soulmachine · 2023-01-26T18:34:06Z

Looking into this issue, will update

panicfarm · 2023-01-26T20:51:17Z

I have the same issue.any resolutions? @panicfarm

@soulmachine I believe the issue is here:
https://github.com/crypto-crawler/crypto-msg-parser/blob/3ba78a12e34e9b93468e6dc26d68ad3dec620a97/crypto-contract-value/src/exchanges/kucoin.rs#L152
fn fetch_linear_multipliers() is blocking
https://github.com/crypto-crawler/crypto-msg-parser/blob/3ba78a12e34e9b93468e6dc26d68ad3dec620a97/crypto-contract-value/src/exchanges/kucoin.rs#L178 but your example that I used is async (because your websocket is async), so this function (and thus crypto_msg_parser::parse_trade()) cannot be called from an asynchronous context.

The function is blocking because inside it contains blocking reqwest: https://github.com/crypto-crawler/crypto-msg-parser/blob/3ba78a12e34e9b93468e6dc26d68ad3dec620a97/crypto-contract-value/src/exchanges/utils.rs#L7

You could make the entire crypto_msg_parser::parse_trade() async, although I think it's prefererrable to "instantly" parse the websocket messages, leaving this function sync, so maybe this blocking fetch_linear_multipliers reqwest can be avoided or asynchronously done before the websocket connection is established.

foonsun · 2023-01-27T02:11:51Z

I have the same issue.any resolutions? @panicfarm

@soulmachine I believe the issue is here: https://github.com/crypto-crawler/crypto-msg-parser/blob/3ba78a12e34e9b93468e6dc26d68ad3dec620a97/crypto-contract-value/src/exchanges/kucoin.rs#L152 fn fetch_linear_multipliers() is blocking https://github.com/crypto-crawler/crypto-msg-parser/blob/3ba78a12e34e9b93468e6dc26d68ad3dec620a97/crypto-contract-value/src/exchanges/kucoin.rs#L178 but your example that I used is async (because your websocket is async), so this function (and thus crypto_msg_parser::parse_trade()) cannot be called from an asynchronous context.

The function is blocking because inside it contains blocking reqwest: https://github.com/crypto-crawler/crypto-msg-parser/blob/3ba78a12e34e9b93468e6dc26d68ad3dec620a97/crypto-contract-value/src/exchanges/utils.rs#L7

You could make the entire crypto_msg_parser::parse_trade() async, although I think it's prefererrable to "instantly" parse the websocket messages, leaving this function sync, so maybe this blocking fetch_linear_multipliers reqwest can be avoided or asynchronously done before the websocket connection is established.

I think whether fetch_linear_multipliers is blocking or async is not important because it's only called once when called. static LINEAR_CONTRACT_VALUES: Lazy<HashMap<String, f64>> = Lazy::new(|| only request once. I prefer the async resolution because the app may use the async functions.

foonsun · 2023-01-30T09:46:37Z

any update? @soulmachine

soulmachine · 2023-01-31T00:23:57Z

Not yet, still working on it

panicfarm mentioned this issue Jan 9, 2023

Cross posting crypto-crawler-rs/issues/58 crypto-crawler/crypto-msg-parser#1

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

crypto_msg_parser::parse_trade() : Cannot drop a runtime in a context where blocking is not allowed. This happens when a runtime is dropped from within an asynchronous context. #58

crypto_msg_parser::parse_trade() : Cannot drop a runtime in a context where blocking is not allowed. This happens when a runtime is dropped from within an asynchronous context. #58

panicfarm commented Jan 9, 2023 •

edited

Loading

soulmachine commented Jan 10, 2023

panicfarm commented Jan 10, 2023

foonsun commented Jan 26, 2023 •

edited

Loading

soulmachine commented Jan 26, 2023

panicfarm commented Jan 26, 2023 •

edited

Loading

foonsun commented Jan 27, 2023 •

edited

Loading

foonsun commented Jan 30, 2023

soulmachine commented Jan 31, 2023

crypto_msg_parser::parse_trade() : Cannot drop a runtime in a context where blocking is not allowed. This happens when a runtime is dropped from within an asynchronous context. #58

crypto_msg_parser::parse_trade() : Cannot drop a runtime in a context where blocking is not allowed. This happens when a runtime is dropped from within an asynchronous context. #58

Comments

panicfarm commented Jan 9, 2023 • edited Loading

soulmachine commented Jan 10, 2023

panicfarm commented Jan 10, 2023

foonsun commented Jan 26, 2023 • edited Loading

soulmachine commented Jan 26, 2023

panicfarm commented Jan 26, 2023 • edited Loading

foonsun commented Jan 27, 2023 • edited Loading

foonsun commented Jan 30, 2023

soulmachine commented Jan 31, 2023

panicfarm commented Jan 9, 2023 •

edited

Loading

foonsun commented Jan 26, 2023 •

edited

Loading

panicfarm commented Jan 26, 2023 •

edited

Loading

foonsun commented Jan 27, 2023 •

edited

Loading