out of sync issue with hub

[omritoptix]

rollapp (2).log

seems like the rollapp goes out of sync sometimes with the hub.

- index: true\n    key: c2lnbmF0dXJl\n    value: M2RrOWhFdUFzMFpnQklhTmdQYXluVjNtNWl6WENhbXJXWm8xTmVoK2dNcFZRYTFPQkhqQ1ZFd1JHVURDMWZxREVxN2tic25kRWVJSVJRSzVwMEt0Nmc9PQ==\n  type: tx\ngas_used: \"108945\"\ngas_wanted: \"202793\"\nheight: \"204780\"\ninfo: \"\"\nlogs: []\nraw_log: 'failed to execute message; message index: 0: expected height (921), but\n  received (891): start-height does not match rollapps state'\ntimestamp: \"\"\ntx: null\ntxhash: 6E7A131BEF0BB08748010EFD884FC78D0C65F9FE34DE1B5954412BAB26676176\n error SPN error with '1009' code: failed to execute message; message index: 0: expected height (921), but received (891): start-height does not match rollapps state]" module=settlement_client
time="2023-07-12T12:38:15+03:00" level=error msg="Error sending batch to settlement layer[resp code: 1009\ncodespace: rollapp\ndata: \"\"\nevents:\n- attributes:\n  - index: true\n    key: c3BlbmRlcg==\n    value: ZHltMTB5cGFndTd1YXVuZHgyNGhodHM4ODRtem55MGtxdDJ6NnIzNmt6\n  - index: true\n    key: YW1vdW50\n    value: NTA2OTl1ZHlt\n  type: coin_spent\n- attributes:\n  - index: true\n    key: cmVjZWl2ZXI=\n    value: ZHltMTd4cGZ2YWttMmFtZzk2MnlsczZmODR6M2tlbGw4YzVsenkweHdu\n  - index: true\n    key: YW1vdW50\n    value: NTA2OTl1ZHlt\n  type: coin_received\n- attributes:\n  - index: true\n    key: cmVjaXBpZW50\n    value: ZHltMTd4cGZ2YWttMmFtZzk2MnlsczZmODR6M2tlbGw4YzVsenkweHdu\n  - index: true\n    key: c2VuZGVy\n    value: ZHltMTB5cGFndTd1YXVuZHgyNGhodHM4ODRtem55MGtxdDJ6NnIzNmt6\n  - index: true\n    key: YW1vdW50\n    value: NTA2OTl1ZHlt\n  type: transfer\n- attributes:\n  - index: true\n    key: c2VuZGVy\n    value: ZHltMTB5cGFndTd1YXVuZHgyNGhodHM4ODRtem55MGtxdDJ6NnIzNmt6\n  type: message\n- attributes:\n  - index: true\n    key: ZmVl\n    value: NTA2OTl1ZHlt\n  - index: true\n    key: ZmVlX3BheWVy\n    value: ZHltMTB5cGFndTd1YXVuZHgyNGhodHM4ODRtem55MGtxdDJ6NnIzNmt6\n  type: tx\n- attributes:\n  - index: true\n    key: YWNjX3NlcQ==\n    value: ZHltMTB5cGFndTd1YXVuZHgyNGhodHM4ODRtem55MGtxdDJ6NnIzNmt6LzUwOQ==\n  type: tx\n- attributes:\n  - index: true\n    key: c2lnbmF0dXJl\n    value: cm9pS2RmenF4T3ZLeWJMNy9yb255T05JakVXdlBTYmdJcUloM0pQZW5zb3NvenNDNThwZzNvK0NyUDMwODhHRlA4ZG5IVEdjYlA2dWt2Qm90QlJySHc9PQ==\n  type: tx\ngas_used: \"108945\"\ngas_wanted: \"202793\"\nheight: \"204793\"\ninfo: \"\"\nlogs: []\nraw_log: 'failed to execute message; message index: 0: expected height (921), but\n  received (891): start-height does not match rollapps state'\ntimestamp: \"\"\ntx: null\ntxhash: B80A958C80B9983FD1944DDDE55B159DE025B9F6BA9E134E577E400E4D7DF246\n error SPN error with '1009' code: failed to execute message; message index: 0: expected height (921), but received (891): start-height does not match rollapps state]" module=settlement_client
time="2023-07-12T12:39:20+03:00" level=error msg="Error sending batch to settlement layer[resp code: 1009\ncodespace: rollapp\ndata: \"\"\nevents:\n- attributes:\n  - index: true\n    key: c3BlbmRlcg==\n    value: ZHltMTB5cGFndTd1YXVuZHgyNGhodHM4ODRtem55MGtxdDJ6NnIzNmt6\n  - index: true\n    key: YW1vdW50\n    value: NTA2OTl1ZHlt\n  type: coin_spent\n- attributes:\n  - index: true\n    key: cmVjZWl2ZXI=\n    value: ZHltMTd4cGZ2YWttMmFtZzk2MnlsczZmODR6M2tlbGw4YzVsenkweHdu\n  - index: true\n    key: YW1vdW50\n    value: NTA2OTl1ZHlt\n  type: coin_received\n- attributes:\n  - index: true\n    key: cmVjaXBpZW50\n    value: ZHltMTd4cGZ2YWttMmFtZzk2MnlsczZmODR6M2tlbGw4YzVsenkweHdu\n  - index: true\n    key: c2VuZGVy\n    value: ZHltMTB5cGFndTd1YXVuZHgyNGhodHM4ODRtem55MGtxdDJ6NnIzNmt6\n  - index: true\n    key: YW1vdW50\n    value: NTA2OTl1ZHlt\n  type: transfer\n- attributes:\n  - index: true\n    key: c2VuZGVy\n    value: ZHltMTB5cGFndTd1YXVuZHgyNGhodHM4ODRtem55MGtxdDJ6NnIzNmt6\n  type: message\n- attributes:\n  - index: true\n    key: ZmVl\n    value: NTA2OTl1ZHlt\n  - index: true\n    key: ZmVlX3BheWVy\n    value: ZHltMTB5cGFndTd1YXVuZHgyNGhodHM4ODRtem55MGtxdDJ6NnIzNmt6\n  type: tx\n- attributes:\n  - index: true\n    key: YWNjX3NlcQ==\n    value: ZHltMTB5cGFndTd1YXVuZHgyNGhodHM4ODRtem55MGtxdDJ6NnIzNmt6LzUxMA==\n  type: tx\n- attributes:\n  - index: true\n    key: c2lnbmF0dXJl\n    value: NHJ3Q3Bjdy9ucDdJYU5jR1JaSDlBT2V0NCt4SmVEZTREYmFsakZVMHhBMGpzcnh2dnNDdHJIbGRrZ2tCRGFQM2xZcGVvcWpGL1V3enBncHdUbkRJZHc9PQ==\n  type: tx\ngas_used: \"108945\"\ngas_wanted: \"202793\"\nheight: \"204806\"\ninfo: \"\"\nlogs: []\nraw_log: 'failed to execute message; message index: 0: expected height (921), but\n  received (891): start-height does not match rollapps state'\ntimestamp: \"\"\ntx: null\ntxhash: E9095A731B661945D75A60BDFC9E3245A506B24499C646335534328CE112DFAD\n error SPN error with '1009' code: failed to execute message; message index: 0: expected height (921), but received (891): start-height does not match rollapps state]" module=settlement_client

=

[omritoptix]

After examining the problem seems to be that the RPC of the hub was very slow which caused a timeout in block inclusion:

time="2023-07-12T04:22:15+03:00" level=error msg="Error sending batch to settlement layer[resp %!v(PANIC=String method: runtime error: invalid memory address or nil pointer dereference) error error in json rpc client, with http response metadata: (Status: 200 OK, Protocol HTTP/1.1). RPC error -32603 - Internal error: timed out waiting for tx to be included in a block]" module=settlement_client

this was interpreted as an error the layer client , however the batch was indeed accepted, hence trying to send the same batch caused the error.

[omritoptix]

There are few solutions possible but IMO it's best either way to not send the transaction in broacast-mode=block as we're waiting on events anyway and this is prone to yielding such timeout errors if network is slow/congested etc.

In broadcast-mode=sync we shouldn't get this error as we're not waiting for TX to be included in a block but only for the response to pass/fail CheckTx and if an error returns we know for sure it indicates the TX won't be included in a block (unlike in broadcas-mode=block case)