Reindexing

Reindexing allows new indexes to be created on existing data. The code changes fall into two areas:

Knowing when to reindex

When defineType is called, for each index

• toString() and uglify() the function -> fnstr
• Get the existing index function stored in the key 'mindexfunction:<typename>:<indexname>\0'
• If index is new or has changed, start the reindex job, continue to next index
• Get the current index status stored in the key `'mindexstatus::\0'
• If the status is specified, resume the reindex job, continue to the next index

Starting a Reindex Job

• In the type object, set the index's status value to '\u0000'
• If an old index with the same name exists, start a Deletion Loop.
• Put the fnstr in 'mindexfunction:<type>:<index>\0'
• Put '\u0000' in `'mindexstatus::\0'
• Start the Indexing Loop

Resuming a Reindex Job

• In the type object, set the index's status value to the one from the db
• Start the Indexing Loop

Deletion Loop

Iterate from 'i<typename>:<indexname>\0' to 'i<typename>:<indexname>\0\uffff', and for each key

• Push a delete operation into a batch array
• When the batch size reaches a threshold (say 1000), execute, clear the array and continue
• When iteration completes, callback.

Indexing Loop

Iterate from 'dtype:<indexstatus>' to 'dtype:\uffff', and for each object

• If the object's id == indexstatus, do nothing
• Run the new index function on the object and push emitted keys into a batch array
• When the batch size exceeds a threshold (say 1000):
• Push an put operation setting `'mindexstatus::\0' to the last indexed object's id
• execute the batch and clear the batch array; when execution completes:
• in the type object set the index's status to the last object's id

When iteration completes,

• clear the index status both from the db and the type object