Skip to content

refactor writer to a writer/worker architecture #323

@rafaelcr

Description

@rafaelcr

Currently, the metadata writer performs two functions:

  1. Reads block data from the Stacks index
  2. Performs all metadata fetches from queued jobs

To optimize performance and resource use, we should split these so the writer only takes care of (1) while new auto-scalable workers do (2). We could use a Redis queue to enqueue jobs instead of a postgres table like we do today, and it would allow us to down/up scale workers when required to save on infra costs.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions