nightly-20241027 cluster can't start

[cyliu0]

Describe the bug

It happened in all the daily and weekly pipelines
https://buildkite.com/risingwave-test/backfill/builds/860#0192cf5a-37fd-4869-b119-a48b7447a43d

Error message/log

No response

To Reproduce

No response

Expected behavior

No response

How did you deploy RisingWave?

No response

The version of RisingWave

nightly-20241027

Additional context

No response

[fuyufjh]

thread 'main' panicked at /risingwave/src/meta/src/manager/env.rs:369:14:
Failed to upgrade models in meta store: Exec(SqlxError(Database(PgDatabaseError { severity: Error, code: "42883", message: "operator does not exist: jsonb = text", detail: None, hint: Some("No operator matches the given name and argument types. You might need to add explicit type casts."), position: Some(Original(64)), where: None, schema: None, table: None, column: None, data_type: None, constraint: None, file: Some("parse_oper.c"), line: Some(647), routine: Some("op_error") })))
stack backtrace:
   0:     0xaaaaf49255f4 - std::backtrace_rs::backtrace::libunwind::trace::hc9dc7ac09fccfa1e
                               at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/std/src/../../backtrace/src/backtrace/libunwind.rs:116:5
   1:     0xaaaaf49255f4 - std::backtrace_rs::backtrace::trace_unsynchronized::h1fca4d0fb3cddc62
                               at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0xaaaaf49255f4 - std::sys::backtrace::_print_fmt::he937aaa56f0d7399
                               at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/std/src/sys/backtrace.rs:65:5
   3:     0xaaaaf49255f4 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h66beb6bf41ba0fe1
                               at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/std/src/sys/backtrace.rs:40:26
   4:     0xaaaae80a59d8 - core::fmt::rt::Argument::fmt::heb5c17b63f47e21b
                               at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/core/src/fmt/rt.rs:173:76
   5:     0xaaaae80a59d8 - core::fmt::write::he24553c21bb22ec5
                               at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/core/src/fmt/mod.rs:1182:21
   6:     0xaaaaf49204e0 - std::io::Write::write_fmt::h677748388b79c97c
                               at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/std/src/io/mod.rs:1827:15
   7:     0xaaaaf49267bc - std::sys::backtrace::BacktraceLock::print::ha3062d152fa5a8d2
                               at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/std/src/sys/backtrace.rs:43:9
   8:     0xaaaaf49267bc - std::panicking::default_hook::{{closure}}::hb6093d75dc944a2e
                               at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/std/src/panicking.rs:269:22
   9:     0xaaaaf49263dc - std::panicking::default_hook::h655987311488e1f0
                               at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/std/src/panicking.rs:296:9
  10:     0xaaaaf1e218f0 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h4dd23c4f091ba59f
                               at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/alloc/src/boxed.rs:2084:9
  11:     0xaaaaf1e218f0 - risingwave_rt::panic_hook::set_panic_hook::{{closure}}::h7303682a4c268435
                               at /risingwave/src/utils/runtime/src/panic_hook.rs:25:9
  12:     0xaaaaf1e218f0 - std::panicking::update_hook::{{closure}}::h716ce62a42a75968
                               at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/std/src/panicking.rs:233:47
  13:     0xaaaaf4926f50 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::hb3b016a43b9d0f14
                               at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/alloc/src/boxed.rs:2084:9
  14:     0xaaaaf4926f50 - std::panicking::rust_panic_with_hook::h838864c869d3e9e6
                               at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/std/src/panicking.rs:808:13
  15:     0xaaaaf4926ca0 - std::panicking::begin_panic_handler::{{closure}}::hac57f80b39b5dd10
                               at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/std/src/panicking.rs:674:13
  16:     0xaaaaf4925adc - std::sys::backtrace::__rust_end_short_backtrace::hfc658b1d5659adfa
                               at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/std/src/sys/backtrace.rs:168:18
  17:     0xaaaaf4926970 - rust_begin_unwind
                               at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/std/src/panicking.rs:665:5
  18:     0xaaaae80a2574 - core::panicking::panic_fmt::hf00b22ade8aa3f95
                               at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/core/src/panicking.rs:74:14
  19:     0xaaaae80a2924 - core::result::unwrap_failed::h61a2c457506513d8
                               at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/core/src/result.rs:1679:5
  20:     0xaaaaf0696dfc - core::result::Result<T,E>::expect::he7533895794b5b4f
                               at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/core/src/result.rs:1059:23
  21:     0xaaaaf0696dfc - risingwave_meta::manager::env::MetaSrvEnv::new::{{closure}}::hd106c8b40446a1a4
                               at /risingwave/src/meta/src/manager/env.rs:367:9
  22:     0xaaaaf07163ec - risingwave_meta_node::server::start_service_as_election_leader::{{closure}}::hf18979c7104f23da
                               at /risingwave/src/meta/node/src/server.rs:349:6
  23:     0xaaaaf0712530 - risingwave_meta_node::server::rpc_serve_with_store::{{closure}}::h77e5a59a1435b3f6
                               at /risingwave/src/meta/node/src/server.rs:279:6
  24:     0xaaaaf072e318 - risingwave_meta_node::server::rpc_serve::{{closure}}::h764f2072d2f07371
                               at /risingwave/src/meta/node/src/server.rs:194:14
  25:     0xaaaaf072e318 - risingwave_meta_node::start::{{closure}}::h6c5d6bad7775d090
                               at /risingwave/src/meta/node/src/lib.rs:457:10
  26:     0xaaaaec162434 - <core::pin::Pin<P> as core::future::future::Future>::poll::h0e9db85ddc9eaaff
                               at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/core/src/future/future.rs:123:9
  27:     0xaaaaec162434 - <core::pin::Pin<P> as core::future::future::Future>::poll::h0c039c44cbeb0046
                               at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/core/src/future/future.rs:123:9
  28:     0xaaaaec162434 - <&mut F as core::future::future::Future>::poll::h32acf5ab880a6d3b
                               at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/core/src/future/future.rs:111:9
  29:     0xaaaaec162434 - risingwave_rt::main_okk::{{closure}}::{{closure}}::h4b95cc8fa8df3d35
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/macros/select.rs:548:49
  30:     0xaaaaec162434 - <tokio::future::poll_fn::PollFn<F> as core::future::future::Future>::poll::ha82fc5cd28bab534
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/future/poll_fn.rs:58:9
  31:     0xaaaaec162434 - risingwave_rt::main_okk::{{closure}}::h72cc36a040f790f6
                               at /risingwave/src/utils/runtime/src/lib.rs:126:9
  32:     0xaaaaec162434 - <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::hd4e47c25adb35d11
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.40/src/instrument.rs:321:9
  33:     0xaaaaec1f49f8 - tokio::runtime::park::CachedParkThread::block_on::{{closure}}::hb44db8295192e38c
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/park.rs:281:63
  34:     0xaaaaec1f49f8 - tokio::runtime::coop::with_budget::h0fe91bfe2039231b
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/coop.rs:107:5
  35:     0xaaaaec1f49f8 - tokio::runtime::coop::budget::hc167b92a0f7b3cb6
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/coop.rs:73:5
  36:     0xaaaaec1f49f8 - tokio::runtime::park::CachedParkThread::block_on::h76e5a69bada5be04
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/park.rs:281:31
  37:     0xaaaaec32bb00 - tokio::runtime::context::blocking::BlockingRegionGuard::block_on::h67e7b0245bf761a1
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/context/blocking.rs:66:9
  38:     0xaaaaec32bb00 - tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}::h8ff78092a5c43854
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/scheduler/multi_thread/mod.rs:87:22
  39:     0xaaaaec32bb00 - tokio::runtime::context::runtime::enter_runtime::h21258f8ff58f3581
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/context/runtime.rs:65:16
  40:     0xaaaaec64c4b0 - tokio::runtime::scheduler::multi_thread::MultiThread::block_on::hfabd3c5bc792a5b0
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/scheduler/multi_thread/mod.rs:86:9
  41:     0xaaaaec64c4b0 - tokio::runtime::runtime::Runtime::block_on::hdcfb29379e80380d
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/runtime.rs:349:50
  42:     0xaaaaec53bbd4 - risingwave_rt::main_okk::h4d31d65406bb76b4
                               at /risingwave/src/utils/runtime/src/lib.rs:168:5
  43:     0xaaaaec664b14 - risingwave_cmd::meta::h9ce0d2997dfa5b26
                               at /risingwave/src/cmd/src/lib.rs:47:5
  44:     0xaaaaeb4e094c - risingwave::Component::start::h43cfe8ac78039d56
                               at /risingwave/src/cmd_all/src/bin/risingwave.rs:121:27
  45:     0xaaaaeb4e1998 - risingwave::main::h2863d4b2601215e2
                               at /risingwave/src/cmd_all/src/bin/risingwave.rs:224:5
  46:     0xaaaaeb4e9990 - core::ops::function::FnOnce::call_once::he98b346c94fe2e24
                               at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/core/src/ops/function.rs:250:5
  47:     0xaaaaeb4e9990 - std::sys::backtrace::__rust_begin_short_backtrace::he99f05b504f36673
                               at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/std/src/sys/backtrace.rs:152:18
  48:     0xaaaaeb4e7490 - std::rt::lang_start::{{closure}}::h2ed96cdcc91c473b
                               at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/std/src/rt.rs:162:18
  49:     0xaaaaf4915ef0 - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::h3b2b805dba49689d
                               at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/core/src/ops/function.rs:284:13
  50:     0xaaaaf4915ef0 - std::panicking::try::do_call::h876d70b59a8dd741
                               at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/std/src/panicking.rs:557:40
  51:     0xaaaaf4915ef0 - std::panicking::try::he3b89d037474f083
                               at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/std/src/panicking.rs:521:19
  52:     0xaaaaf4915ef0 - std::panic::catch_unwind::hf8324aec4cd1e26e
                               at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/std/src/panic.rs:350:14
  53:     0xaaaaf4915ef0 - std::rt::lang_start_internal::{{closure}}::h4eab4594adf65f6c
                               at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/std/src/rt.rs:141:48
  54:     0xaaaaf4915ef0 - std::panicking::try::do_call::hbd388f83ba8e50da
                               at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/std/src/panicking.rs:557:40
  55:     0xaaaaf4915ef0 - std::panicking::try::h6e92a0a8f6fdd2b9
                               at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/std/src/panicking.rs:521:19
  56:     0xaaaaf4915ef0 - std::panic::catch_unwind::h05feb3837a772afc
                               at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/std/src/panic.rs:350:14
  57:     0xaaaaf4915ef0 - std::rt::lang_start_internal::h9edc5d9fbe10e4b0
                               at /rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/std/src/rt.rs:141:20
  58:     0xaaaaeb4e223c - main
  59:     0xffffb83d84c4 - __libc_start_call_main
                               at ./csu/../sysdeps/nptl/libc_start_call_main.h:58:16
  60:     0xffffb83d8598 - __libc_start_main_impl
                               at ./csu/../csu/libc-start.c:360:3
  61:     0xaaaae6957970 - _start
  62:                0x0 - <unknown>

[fuyufjh]

Failed to upgrade models in meta store

Might be related to #18753 which added a new migration to sqlx. diff. @BugenZhao Can you pls take a look

[BugenZhao]

I will take a look later. But this doesn't go into release branches, right?

[cyliu0]

But this doesn't go into release branches, right?

I have triggered the v2.1.0-rt.2 release test last Friday night. The cluster works pretty well.