Time-based Index Partitioning - Generational Indexing

[gammazero]

In order to limit the scale of a single indexer instance, instances can be created periodically, to index all content that becomes available within a time period.

Consider an indexer that is configured to index content starting at new advertisements only. That is, the first sync with a provider does not pull advertisements deeper than the provider's current head. Let this indexer node continue ingesting new content for a month. At the start of the next month, make it stop indexing any new content and start a new indexer instance to index content for the new month.

When a query arrives, it is sent to all indexer instances and the results are merged into a single reply to the client. A client could have the ability to request content that has been indexed in the last N months, in which case the query is only sent to specific indexer instances.

Advertisements that perform deletes and metadata updates need to be applied to all indexers. Any new CIDs, even if mapped to metadata that was created in the past, are only applied to the latest indexer.

Eventually, the oldest indexer instance can be retired when it is deemed no longer needed.