When running on theta-testnet-001, firehose-cosmos crashes repeatedly #41

[matthewdarwin]

Multiple crashes similar to this. If you restart firehose-cosmos then it continues running. Seen on theta-testnet-001, while cosmoshub, osmosis and juno were fine.

4:59PM INF committed state app_hash=2A767455AA69444B26D0931EA8663F2A3456B4A13035DD8A50AF84AA0FDFD8F4 height=10770061 module=state num_txs=0
4:59PM INF finalized dmlog block block=10770061 module=deepmind
4:59PM INF indexed block height=10770061 module=txindex
4:59PM INF minted coins from module account amount=39179282uatom from=mint module=x/bank
4:59PM INF client state updated client-id=07-tendermint-804 height=0-3859 module=x/ibc/client
4:59PM INF packet received dst_channel=channel-455 dst_port=icahost module=x/ibc/channel sequence=13 src_channel=channel-4 src_port=icacontroller-5
4:59PM INF acknowledgement written dst_channel=channel-455 dst_port=icahost module=x/ibc/channel sequence="marshaling error: json: unsupported type: func() uint64" src_channel=channel-4 src_port=icacontroller-5
4:59PM INF executed block height=10770062 module=state num_invalid_txs=0 num_valid_txs=1
4:59PM INF commit synced commit=436F6D6D697449447B5B32333420313936203639203138372032353120352031373120323437203439203135352032313720323330203135372031333920313533203133382037392037332031302032323220363320313037203134372032343020333320393920323431203231372031363520313431203130382033305D3A4134353638457D
4:59PM INF committed state app_hash=EAC445BBFB05ABF7319BD9E69D8B998A4F490ADE3F6B93F02163F1D9A58D6C1E height=10770062 module=state num_txs=1
4:59PM ERR dmlog tx write failed err="string field contains invalid UTF-8" module=state
4:59PM INF finalized dmlog block block=10770062 module=deepmind
panic: string field contains invalid UTF-8
goroutine 42 [running]:
github.com/tendermint/tendermint/deepmind.Abort(...)
#011github.com/tendermint/[email protected]/deepmind/extractor.go:53
github.com/tendermint/tendermint/state.fireEvents({0x20b6078, 0xc00124ea80}, {0x20b79c0, 0xc0048e8640}, 0xc11b933c20, 0xc090c5e120, {0x2eb8b28, 0x0, 0x0})
#011github.com/tendermint/[email protected]/state/execution.go:554 +0x925
github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x0}, {0xc00053f410, 0xd}}, {0xc0047fdb30, 0x11}, 0x89dbae, 0xa4568e, {{0xc128f39780, ...}, ...}, ...}, ...)
#011github.com/tendermint/[email protected]/state/execution.go:201 +0x93b
github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).poolRoutine(0xc00015d500, 0x0)
#011github.com/tendermint/[email protected]/blockchain/v0/reactor.go:398 +0xb5a
created by github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).OnStart
#011github.com/tendermint/[email protected]/blockchain/v0/reactor.go:110 +0x7a
################################################################
Fatal error in app ingestor:
exit status 2
################################################################
 (launcher/launcher.go:174)
Application ingestor shutdown unexpectedly, quitting
Waiting for all apps termination...
All apps terminated gracefully
Error: exit status 2
executing root command (derr/cli.go:25){"error": "exit status 2"}

It eventually proceeds to live blocks. Just let it crash and restart. I'm not sure if the blocks end up being corrupt?

Copy of issue: graphprotocol#41