Bug: Procssor exits with AssertionError

[sameh-farouk]

deploying TFChain squid today on QANet snapshot creator machine failed as the processor exited with an error. @hossnys and @coesensbert reported it.

The deployment on that machine has been rolled back pending a fix by the development team.

logs:

{"level":5,"time":1717688117149,"ns":"sqd:processor","err":{"generatedMessage":true,"code":"ERR_ASSERTION","actual":false,"expected":true,"operator":"==","stack":"AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:\n\n  (0, assert_1.default)(typeof value == \"boolean\")\n\n    at decodePrimitive (/squid/node_modules/@subsquid/scale-codec/lib/codec-json.js:164:34)\n    at JsonCodec.decode (/squid/node_modules/@subsquid/scale-codec/lib/codec-json.js:24:24)\n    at JsonCodec.decodeStruct (/squid/node_modules/@subsquid/scale-codec/lib/codec-json.js:92:35)\n    at JsonCodec.decode (/squid/node_modules/@subsquid/scale-codec/lib/codec-json.js:36:29)\n    at Chain.decodeTuple (/squid/node_modules/@subsquid/substrate-processor/lib/chain.js:150:35)\n    at Chain.decode (/squid/node_modules/@subsquid/substrate-processor/lib/chain.js:138:25)\n    at Chain.decodeEvent (/squid/node_modules/@subsquid/substrate-processor/lib/chain.js:128:21)\n    at get asV101 [as asV101] (/squid/lib/types/events.js:759:28)\n    at farmUpdated (/squid/lib/mappings/farms.js:146:51)\n    at handleEvents (/squid/lib/processor.js:100:73)"}}
error Command failed with exit code 1.

[sameh-farouk]

Update:
I left a test instance syncing with QANet, so far so good.

{"level":2,"time":1717720300228,"ns":"sqd:processor","msg":"816499 / 2037359, rate: 3335 blocks/sec, mapping: 44762 blocks/sec, 1566 items/sec, ingest: 3869 blocks/sec, eta: 7m"}

I will see if I can reproduce the issue.

[sameh-farouk]

Update:
A note unrelated to the original issue: restcountries.com API is unresponding and timeout when the init script tries to fetch the countries. This will lead to a lack of countries and cities data in squid data. A restart won't fix it. The processor would need to resync from 0 since this data is critical to exist at the beginning.
I would change the init_countires script to exit with an error code instead of 0 to prevent the processor from starting when fetching the countries or cities data fails.

[sameh-farouk]

Update:
restcountries.com API is back now.

[sameh-farouk]

@coesensbert @hossnys My test instance is still going well.

{"level":2,"time":1717756541668,"ns":"sqd:processor","msg":"2445019 / 2445039, rate: 150 blocks/sec, mapping: 895 blocks/sec, 9 items/sec, ingest: 115 blocks/sec, eta: 0s"}

Here is my stack versions

Can you please confirm the processor image and tag that shows the error?

[coesensbert]

threefoldtech/grid_deployment@f11a9f7