Upgrade tests failing

[lumtis]

Describe the Bug
The upgrade tests are constantly failing with the error

setup      | ❌ ethereum panic: waiting cctx timeout, cctx not mined, inTxHash: 

https://github.com/zeta-chain/node/actions/runs/9074912329/job/24934506785
https://github.com/zeta-chain/node/actions/runs/9074570731/job/24933564515

[lumtis]

@gartnera it seems there is an issue occurring when upgrading the zetaclientd with the supervisor. Do these logs tell you something?

zetaclient0  | 2024-05-14T12:14:35Z ERR CoreContextUpdater failed to update config error="failed to get zetablock height: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 172.20.0.11:9090: connect: connection refused\"" module=CoreBridge
zetaclient0  | 2024-05-14T12:14:36Z WRN get current upgrade plan error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 172.20.0.11:9090: connect: connection refused\"" module=zetaclientdSupervisor process=zetaclientd-supervisor
zetaclient0  | 2024-05-14T12:14:36Z WRN get node info error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 172.20.0.11:9090: connect: connection refused\"" module=zetaclientdSupervisor process=zetaclientd-supervisor
zetaclient0  | 2024-05-14T12:14:37Z WRN get current upgrade plan error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 172.20.0.11:9090: connect: connection refused\"" module=zetaclientdSupervisor process=zetaclientd-supervisor
zetaclient0  | 2024-05-14T12:14:37Z WRN get node info error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 172.20.0.11:9090: connect: connection refused\"" module=zetaclientdSupervisor process=zetaclientd-supervisor
zetaclient0  | 2024-05-14T12:14:38Z WRN get current upgrade plan error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 172.20.0.11:9090: connect: connection refused\"" module=zetaclientdSupervisor process=zetaclientd-supervisor
zetaclient0  | 2024-05-14T12:14:38Z WRN get node info error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 172.20.0.11:9090: connect: connection refused\"" module=zetaclientdSupervisor process=zetaclientd-supervisor
zetaclient0  | 2024-05-14T12:14:39Z WRN get current upgrade plan error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 172.20.0.11:9090: connect: connection refused\"" module=zetaclientdSupervisor process=zetaclientd-supervisor
zetaclient0  | 2024-05-14T12:14:39Z WRN get node info error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 172.20.0.11:9090: connect: connection refused\"" module=zetaclientdSupervisor process=zetaclientd-supervisor
zetaclient0  | 2024-05-14T12:14:40Z WRN get current upgrade plan error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 172.20.0.11:9090: connect: connection refused\"" module=zetaclientdSupervisor process=zetaclientd-supervisor
zetaclient0  | 2024-05-14T12:14:40Z WRN get node info error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 172.20.0.11:9090: connect: connection refused\"" module=zetaclientdSupervisor process=zetaclientd-supervisor
zetaclient0  | 2024-05-14T12:14:40Z INF Unable to fetch verification flags from zetabridge module=CoreBridge
zetaclient0  | 2024-05-14T12:14:41Z WRN get current upgrade plan error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 172.20.0.11:9090: connect: connection refused\"" module=zetaclientdSupervisor process=zetaclientd-supervisor
zetaclient0  | 2024-05-14T12:14:41Z WRN get node info error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 172.20.0.11:9090: connect: connection refused\"" module=zetaclientdSupervisor process=zetaclientd-supervisor
zetaclient0  | 2024-05-14T12:14:42Z WRN core version change (v12.3.0-rc-99-g0e45a72f -> v0.0.1715688551-dirty) module=zetaclientdSupervisor process=zetaclientd-supervisor
zetaclient0  | 2024-05-14T12:14:42Z ERR unable to update current symlink (/root/.zetaclientd/upgrades/v0.0.1715688551-dirty -> /root/.zetaclientd/upgrades/current) error="stat target: stat /root/.zetaclientd/upgrades/v0.0.1715688551-dirty: no such file or directory"

[gartnera]

yeah

ERR unable to update current symlink (/root/.zetaclientd/upgrades/v0.0.1715688551-dirty -> /root/.zetaclientd/upgrades/current) error="stat target: stat /root/.zetaclientd/upgrades/v0.0.1715688551-dirty: no such file or directory"

is the indicator of the problem.

Upgrade tests pass are passing on #2158 so that would resolve it.

Another reason to do #2134

[lumtis]

Upgrade tests pass are passing on #2158 so that would resolve it.
Ok, attaching the issue to it

Another reason to do #2134

Yes, I think this makes sense to add this as soon as possible. At least the light upgrade tests should catch issues