Several Crawls in one DB #38
Comments
jogli5er
added a commit
that referenced
this issue
Dec 5, 2019
This commit initializes the structure such that we can update the sequelize models continuously. This allows us in the first place to streamline the updates necessary to the db, required that we can run several runs on the same database. That enables further that we can compile all runs into one dataset instead of splitting it up into multiple databases. The current commit is untested and especially the migrations require some testing before we can employ them. For details on the changes required, please refer to issue #38.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
New Table: Crawls: Id & human readable id
on Paths: Secure flag could evolve over time, could be shifted to the content, meaning: content was found with secure flag true/false
Contents: Add
Links: Add
Whenever Crawler is ran:
Configuration: Tag and random id => find control ID and use this
Paths: we will consider all paths that finished before the current start
Assumption: we only ever run one crawl at a time
The text was updated successfully, but these errors were encountered: