-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Change database systems #36
Conversation
This should be remembered, not sure what would be a good way of saving this info. |
Everything's ready, will test it better tomorrow, we'll have to dockerize postgres asap, that'll complete the entire thing i think. |
summarizer is now working since the latest commit |
all workers are now launching 🥳 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(c.c. @SOSNE ) 🪨 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me
Removing legacy TinyDB instances, they operate mainly in-memory, only ocassionally flushing to fs, which is highly undesireable.
Will be replacing them with SQLAlchemy, and likely hooking up PostgreSQL to it due to it's scalability.
We'll also need a document db, first candidate is of course Mongo but this is up for debate.
I think I'll also separate the downloaded documents out of the url_db and the summaries out of the summary_tasks db, into a pure document database.
This will allow us to have all the main databases to be Postgres, while using document based dbs only for pure text variable length data.