You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, the metadata writer performs two functions:
Reads block data from the Stacks index
Performs all metadata fetches from queued jobs
To optimize performance and resource use, we should split these so the writer only takes care of (1) while new auto-scalable workers do (2). We could use a Redis queue to enqueue jobs instead of a postgres table like we do today, and it would allow us to down/up scale workers when required to save on infra costs.