BlockExists panic

[shawnim]

Describe the bug

Leader node panicked during bootstrap. Repeated even after removing the blockchain files.

Mandatory Information

$ jcli --full-version
jcli 0.8.19 (HEAD-776347cf, release, linux [x86_64]) - [rustc 1.43.0 (4fb7144ed 2020-04-20)]
$ jormungandr --full-version
jormungandr 0.8.19 (HEAD-776347cf, release, linux [x86_64]) - [rustc 1.43.0 (4fb7144ed 2020-04-20)]

To Reproduce
Steps to reproduce the behavior:

1. Restart leader node.
2. See panic message in log.

Expected behavior

Node does not panic or provides more useful information for resolving the problem.

Additional context

I was able to get the node to bootstrap by running in passive mode. After that I was able to run as leader.

Log file:

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: IndexError(BlockExists)', /home/ubuntu/jormungandr/chain-deps/chain-storage/src/lib.rs:184:9
stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.44/src/backtrace/libunwind.rs:86
   1: backtrace::backtrace::trace_unsynchronized
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.44/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print_fmt
             at src/libstd/sys_common/backtrace.rs:78
   3: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
             at src/libstd/sys_common/backtrace.rs:59
   4: core::fmt::write
             at src/libcore/fmt/mod.rs:1063
   5: std::io::Write::write_fmt
             at src/libstd/io/mod.rs:1426
   6: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:62
   7: std::sys_common::backtrace::print
             at src/libstd/sys_common/backtrace.rs:49
   8: std::panicking::default_hook::{{closure}}
             at src/libstd/panicking.rs:204
   9: std::panicking::default_hook
             at src/libstd/panicking.rs:224
  10: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:470
  11: rust_begin_unwind
             at src/libstd/panicking.rs:378
  12: core::panicking::panic_fmt
             at src/libcore/panicking.rs:85
  13: core::option::expect_none_failed
             at src/libcore/option.rs:1211
  14: chain_storage::BlockStoreBuilder<B>::build
  15: jormungandr::start_up::prepare_storage
  16: jormungandr::start
  17: jormungandr::main
  18: std::rt::lang_start::{{closure}}
  19: std::rt::lang_start_internal::{{closure}}
             at src/libstd/rt.rs:52
  20: std::panicking::try::do_call
             at src/libstd/panicking.rs:303
  21: __rust_maybe_catch_panic
             at src/libpanic_unwind/lib.rs:86
  22: std::panicking::try
             at src/libstd/panicking.rs:281
  23: std::panic::catch_unwind
             at src/libstd/panic.rs:394
  24: std::rt::lang_start_internal
             at src/libstd/rt.rs:51
  25: main
  26: __libc_start_main
  27: _start
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

[NicolasDP]

Thanks for the report. This is a bit unexpected to this see this error happening. We are trying to remove this entire set for the next level release 0.9.x.