Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Block proof is invalid: Invalid Vrf Proof value #2428

Open
dkijania opened this issue Jun 26, 2020 · 1 comment
Open

Block proof is invalid: Invalid Vrf Proof value #2428

dkijania opened this issue Jun 26, 2020 · 1 comment
Labels
A-jormungandr Area: Issues affecting jörmungandr bug Something isn't working subsys-network network related

Comments

@dkijania
Copy link
Contributor

Describe the bug
For given network:

image

Where Leader1 is a legacy node, i can observe panic:

Jun 26 10:58:32.089 ERRO task process leadership block finished with error, error: Error(BlockHeaderVerificationFailed("Block proof is invalid: Invalid Vrf Proof value in 41584339-0000006e-1.56, poolId: 9aecced24ff0be4c18f7aa0d4ef9f6f9ae58c1b18a27e451742ed038dcb03f0e"), State { next_error: None, backtrace: InternalBacktrace { backtrace: Some(   0: backtrace::backtrace::trace_unsynchronized<closure-0>
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\backtrace-0.3.48\src\backtrace\mod.rs:66
   1: backtrace::backtrace::trace<closure-0>
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\backtrace-0.3.48\src\backtrace\mod.rs:53
   2: backtrace::capture::Backtrace::create
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\backtrace-0.3.48\src\capture.rs:164
   3: backtrace::capture::Backtrace::new_unresolved
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\backtrace-0.3.48\src\capture.rs:158
   4: error_chain::backtrace::imp::InternalBacktrace::new
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\error-chain-0.12.2\src\backtrace.rs:56
   5: error_chain::{{impl}}::default
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\error-chain-0.12.2\src\lib.rs:687
   6: jormungandr::blockchain::chain::Error::from_kind
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\error-chain-0.12.2\src\error_chain.rs:269
   7: jormungandr::blockchain::chain::{{impl}}::from
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\error-chain-0.12.2\src\error_chain.rs:370
   8: jormungandr::blockchain::chain::{{impl}}::post_check_header::{{closure}}
             at C:\Work\iohk\jormungandr\jormungandr\src\blockchain\chain.rs:444
   9: core::future::from_generator::{{impl}}::poll<generator-0>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libcore\future\mod.rs:73
  10: jormungandr::blockchain::process::process_leadership_block_inner::{{closure}}
             at C:\Work\iohk\jormungandr\jormungandr\src\blockchain\process.rs:423
  11: core::future::from_generator::{{impl}}::poll<generator-0>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libcore\future\mod.rs:73
  12: jormungandr::blockchain::process::process_leadership_block::{{closure}}
             at C:\Work\iohk\jormungandr\jormungandr\src\blockchain\process.rs:381
  13: core::future::from_generator::{{impl}}::poll<generator-0>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libcore\future\mod.rs:73
  14: tokio::time::timeout::{{impl}}::poll<core::future::from_generator::GenFuture<generator-0>>
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\time\timeout.rs:155
  15: jormungandr::utils::task::{{impl}}::timeout_spawn_fallible::{{closure}}<core::future::from_generator::GenFuture<generator-0>,jormungandr::blockchain::chain::Error>
             at C:\Work\iohk\jormungandr\jormungandr\src\utils\task.rs:268
  16: core::future::from_generator::{{impl}}::poll<generator-0>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libcore\future\mod.rs:73
  17: tokio::runtime::task::core::{{impl}}::poll::{{closure}}<core::future::from_generator::GenFuture<generator-0>,alloc::sync::Arc<tokio::runtime::thread_pool::worker::Worker>>
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\task\core.rs:173
  18: tokio::loom::std::unsafe_cell::UnsafeCell<tokio::runtime::task::core::Stage<core::future::from_generator::GenFuture<generator-0>>>::with_mut<tokio::runtime::task::core::Stage<core::future::from_generator::GenFuture<generator-0>>,core::task::poll::Poll<()>
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\loom\std\unsafe_cell.rs:14
  19: tokio::runtime::task::core::Core<core::future::from_generator::GenFuture<generator-0>, alloc::sync::Arc<tokio::runtime::thread_pool::worker::Worker>>::poll<core::future::from_generator::GenFuture<generator-0>,alloc::sync::Arc<tokio::runtime::thread_pool::
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\task\core.rs:158
  20: tokio::runtime::task::harness::{{impl}}::poll::{{closure}}<core::future::from_generator::GenFuture<generator-0>,alloc::sync::Arc<tokio::runtime::thread_pool::worker::Worker>>
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\task\harness.rs:107
  21: core::ops::function::FnOnce::call_once<closure-0,()>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libcore\ops\function.rs:232
  22: std::panic::{{impl}}::call_once<core::task::poll::Poll<core::result::Result<(), tokio::runtime::task::error::JoinError>>,closure-0>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libstd\panic.rs:318
  23: std::panicking::try::do_call<std::panic::AssertUnwindSafe<closure-0>,core::task::poll::Poll<core::result::Result<(), tokio::runtime::task::error::JoinError>>>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libstd\panicking.rs:297
  24: std::panicking::try::do_catch<std::panic::AssertUnwindSafe<closure-0>,()>
  25: std::panicking::try<core::task::poll::Poll<core::result::Result<(), tokio::runtime::task::error::JoinError>>,std::panic::AssertUnwindSafe<closure-0>>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libstd\panicking.rs:274
  26: std::panic::catch_unwind<std::panic::AssertUnwindSafe<closure-0>,core::task::poll::Poll<core::result::Result<(), tokio::runtime::task::error::JoinError>>>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libstd\panic.rs:394
  27: tokio::runtime::task::harness::Harness<core::future::from_generator::GenFuture<generator-0>, alloc::sync::Arc<tokio::runtime::thread_pool::worker::Worker>>::poll<core::future::from_generator::GenFuture<generator-0>,alloc::sync::Arc<tokio::runtime::thread_
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\task\harness.rs:89
  28: tokio::runtime::task::raw::poll<core::future::from_generator::GenFuture<generator-0>,alloc::sync::Arc<tokio::runtime::thread_pool::worker::Worker>>
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\task\raw.rs:104
  29: tokio::runtime::task::raw::RawTask::poll
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\task\raw.rs:66
  30: tokio::runtime::task::Notified<alloc::sync::Arc<tokio::runtime::thread_pool::worker::Worker>>::run<alloc::sync::Arc<tokio::runtime::thread_pool::worker::Worker>>
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\task\mod.rs:169
  31: tokio::runtime::thread_pool::worker::{{impl}}::run_task::{{closure}}
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\thread_pool\worker.rs:349
  32: tokio::coop::with_budget::{{closure}}<core::result::Result<alloc::boxed::Box<tokio::runtime::thread_pool::worker::Core>, ()>,closure-0>
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\coop.rs:127
  33: std::thread::local::LocalKey<core::cell::Cell<tokio::coop::Budget>>::try_with<core::cell::Cell<tokio::coop::Budget>,closure-0,core::result::Result<alloc::boxed::Box<tokio::runtime::thread_pool::worker::Core>, ()>>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libstd\thread\local.rs:263
  34: std::thread::local::LocalKey<core::cell::Cell<tokio::coop::Budget>>::with<core::cell::Cell<tokio::coop::Budget>,closure-0,core::result::Result<alloc::boxed::Box<tokio::runtime::thread_pool::worker::Core>, ()>>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libstd\thread\local.rs:239
  35: tokio::coop::with_budget
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\coop.rs:120
  36: tokio::coop::budget
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\coop.rs:96
  37: tokio::runtime::thread_pool::worker::Context::run_task
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\thread_pool\worker.rs:348
  38: tokio::runtime::thread_pool::worker::Context::run
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\thread_pool\worker.rs:320
  39: tokio::runtime::thread_pool::worker::run::{{closure}}
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\thread_pool\worker.rs:305
  40: tokio::macros::scoped_tls::ScopedKey<tokio::runtime::thread_pool::worker::Context>::set<tokio::runtime::thread_pool::worker::Context,closure-0,()>
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\macros\scoped_tls.rs:63
  41: tokio::runtime::thread_pool::worker::run
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\thread_pool\worker.rs:302
  42: tokio::runtime::thread_pool::worker::{{impl}}::launch::{{closure}}
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\thread_pool\worker.rs:281
  43: tokio::runtime::blocking::task::{{impl}}::poll<closure-0,()>
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\blocking\task.rs:38
  44: tokio::runtime::task::core::{{impl}}::poll::{{closure}}<tokio::runtime::blocking::task::BlockingTask<closure-0>,tokio::runtime::blocking::schedule::NoopSchedule>
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\task\core.rs:173
  45: tokio::loom::std::unsafe_cell::UnsafeCell<tokio::runtime::task::core::Stage<tokio::runtime::blocking::task::BlockingTask<closure-0>>>::with_mut<tokio::runtime::task::core::Stage<tokio::runtime::blocking::task::BlockingTask<closure-0>>,core::task::poll::Po
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\loom\std\unsafe_cell.rs:14
  46: tokio::runtime::task::core::Core<tokio::runtime::blocking::task::BlockingTask<closure-0>, tokio::runtime::blocking::schedule::NoopSchedule>::poll<tokio::runtime::blocking::task::BlockingTask<closure-0>,tokio::runtime::blocking::schedule::NoopSchedule>
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\task\core.rs:158
  47: tokio::runtime::task::harness::{{impl}}::poll::{{closure}}<tokio::runtime::blocking::task::BlockingTask<closure-0>,tokio::runtime::blocking::schedule::NoopSchedule>
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\task\harness.rs:107
  48: core::ops::function::FnOnce::call_once<closure-0,()>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libcore\ops\function.rs:232
  49: std::panic::{{impl}}::call_once<core::task::poll::Poll<core::result::Result<(), tokio::runtime::task::error::JoinError>>,closure-0>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libstd\panic.rs:318
  50: std::panicking::try::do_call<std::panic::AssertUnwindSafe<closure-0>,core::task::poll::Poll<core::result::Result<(), tokio::runtime::task::error::JoinError>>>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libstd\panicking.rs:297
  51: tokio::time::instant::{{impl}}::gt
  52: std::panicking::try<core::task::poll::Poll<core::result::Result<(), tokio::runtime::task::error::JoinError>>,std::panic::AssertUnwindSafe<closure-0>>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libstd\panicking.rs:274
  53: std::panic::catch_unwind<std::panic::AssertUnwindSafe<closure-0>,core::task::poll::Poll<core::result::Result<(), tokio::runtime::task::error::JoinError>>>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libstd\panic.rs:394
  54: tokio::runtime::task::harness::Harness<tokio::runtime::blocking::task::BlockingTask<closure-0>, tokio::runtime::blocking::schedule::NoopSchedule>::poll<tokio::runtime::blocking::task::BlockingTask<closure-0>,tokio::runtime::blocking::schedule::NoopSchedul
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\task\harness.rs:89
  55: tokio::runtime::task::raw::poll<tokio::runtime::blocking::task::BlockingTask<closure-0>,tokio::runtime::blocking::schedule::NoopSchedule>
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\task\raw.rs:104
  56: tokio::runtime::task::raw::RawTask::poll
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\task\raw.rs:66
  57: tokio::runtime::task::Notified<tokio::runtime::blocking::schedule::NoopSchedule>::run<tokio::runtime::blocking::schedule::NoopSchedule>
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\task\mod.rs:169
  58: tokio::runtime::blocking::pool::Inner::run
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\blocking\pool.rs:230
  59: tokio::runtime::blocking::pool::{{impl}}::spawn_thread::{{closure}}::{{closure}}
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\blocking\pool.rs:210
  60: tokio::runtime::context::enter<closure-0,()>
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\context.rs:72
  61: tokio::runtime::handle::Handle::enter<closure-0,()>
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\handle.rs:76
  62: tokio::runtime::blocking::pool::{{impl}}::spawn_thread::{{closure}}
             at C:\Users\Dariusz\.cargo\registry\src\github.com-1ecc6299db9ec823\tokio-0.2.21\src\runtime\blocking\pool.rs:209
  63: std::sys_common::backtrace::__rust_begin_short_backtrace<closure-0,()>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libstd\sys_common\backtrace.rs:130
  64: std::thread::{{impl}}::spawn_unchecked::{{closure}}::{{closure}}<closure-0,()>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libstd\thread\mod.rs:475
  65: std::panic::{{impl}}::call_once<(),closure-0>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libstd\panic.rs:318
  66: std::panicking::try::do_call<std::panic::AssertUnwindSafe<closure-0>,()>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libstd\panicking.rs:297
  67: tokio::time::instant::{{impl}}::gt
  68: std::panicking::try<(),std::panic::AssertUnwindSafe<closure-0>>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libstd\panicking.rs:274
  69: std::panic::catch_unwind<std::panic::AssertUnwindSafe<closure-0>,()>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libstd\panic.rs:394
  70: std::thread::{{impl}}::spawn_unchecked::{{closure}}<closure-0,()>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libstd\thread\mod.rs:474
  71: core::ops::function::FnOnce::call_once<closure-0,()>
             at C:\Users\Dariusz\.rustup\toolchains\beta-x86_64-pc-windows-msvc\lib\rustlib\src\rust\src\libcore\ops\function.rs:232
  72: alloc::boxed::{{impl}}::call_once
             at /rustc/b7dc83a3f6ca9746fb3d121761c3605477b77d90\src\liballoc\boxed.rs:1076
  73: alloc::boxed::{{impl}}::call_once
             at /rustc/b7dc83a3f6ca9746fb3d121761c3605477b77d90\src\liballoc\boxed.rs:1076
  74: std::sys::windows::thread::{{impl}}::new::thread_start
             at /rustc/b7dc83a3f6ca9746fb3d121761c3605477b77d90\/src\libstd\sys\windows\thread.rs:56
  75: BaseThreadInitThunk
  76: RtlUserThreadStart

in leaders: 1,2,3,4 which i believe is causing lack of sync after leader1 restart

Mandatory Information

  1. jcli --full-version newest master
  2. jormungandr --full-version newest master

To Reproduce
Steps to reproduce the behavior:

  1. Start Leader1
  2. Start Leader2,3,4
  3. Wait and send some tx's (attached fragments)
  4. Restart Leader4.
  5. Everything should get back to normal (nodes are in sync)
  6. Restart Leader1 (legacy one)
  7. In logs Invalid Vrf Proof errors start appearing, despite Leader1

Expected behavior
After Leader1 restart all nodes can sync

Additional context
run legacy_disruption_last_4th_release scenario (80% reproduction).
Or use below pack to bootrstap network:

leader2_node.log
legacy_disruption_last_4th_release.zip
@dkijania dkijania added bug Something isn't working subsys-network network related A-jormungandr Area: Issues affecting jörmungandr labels Jun 26, 2020
@dkijania
Copy link
Contributor Author

Same issue is visible in this pr: #2333

@dkijania dkijania mentioned this issue Oct 14, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-jormungandr Area: Issues affecting jörmungandr bug Something isn't working subsys-network network related
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@dkijania