Best Practices for Dynamically Reconfiguring Index Patterns

[amanuelz91]

In the Kupo documentation (https://cardanosolutions.github.io/kupo/#tag/Patterns), it is stated that dynamically managing the server's configuration is considered unsafe and may result in a loss of server data integrity.

Does this mean that it's generally not advisable to reconfigure index patterns on-the-fly? Would it be better to stop and restart the server, possibly after clearing the database, whenever we need to match against a new pattern?

[KtorZ]

Hey! The right answer is "it depends".

The warnings in the documentation are meant to trigger questions. The thing with dynamic patterns is that they do not cause a re-indexing of the chain, or they do but only up until the rollback point that you provide. The entire indexer is also "optimized" for short rollbacks as they're the most common on-chain and in theory, rollbacks cannot be longer than a certain security parameter (2160 blocks on mainnet, which corresponds roughly to 18h). It also means that long rollbacks will take a few seconds if not minutes to proceed.

This creates two possible sources of confusion:

1. If there are outputs matching patterns and created before your rollback point, you won't see them. Kupo will never get to index them.

2. Kupo doesn't keep data around more than necessary. If you're running in --prune-utxo mode and rollback more than 2160 blocks in the past, you may be temporarily missing outputs that have been spent later. For example, imagine you're currently at block #3000 and an output was created at block #200 and spent at block #500. If you rollback to block #400, Kupo will NOT re-index that output because it never gets to see it (created 200 blocks ago) and has no memory of it because it was spent for more than 2160 blocks. So, while syncing, between blocks #400 and #500 the state of the index is inconsistent. Once back at the tip, the state will be consistent again since the output was spent anyway. But there's a short period of time during which it should be in the index but it isn't.

If you're in a use case where you know that an output was created after a certain point or if you're only interested in the most recent outputs created then those drawbacks may not be a problem at all.