Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change database systems #36

Merged
merged 25 commits into from
Jun 12, 2024
Merged

Change database systems #36

merged 25 commits into from
Jun 12, 2024

Conversation

latekvo
Copy link
Owner

@latekvo latekvo commented Jun 2, 2024

Removing legacy TinyDB instances, they operate mainly in-memory, only ocassionally flushing to fs, which is highly undesireable.

Will be replacing them with SQLAlchemy, and likely hooking up PostgreSQL to it due to it's scalability.
We'll also need a document db, first candidate is of course Mongo but this is up for debate.

I think I'll also separate the downloaded documents out of the url_db and the summaries out of the summary_tasks db, into a pure document database.
This will allow us to have all the main databases to be Postgres, while using document based dbs only for pure text variable length data.

@latekvo
Copy link
Owner Author

latekvo commented Jun 4, 2024

This should be remembered, not sure what would be a good way of saving this info.
session.scalars(query).one() automatically raises an error when no results are found.

@latekvo latekvo marked this pull request as ready for review June 6, 2024 23:25
@latekvo
Copy link
Owner Author

latekvo commented Jun 6, 2024

Everything's ready, will test it better tomorrow, we'll have to dockerize postgres asap, that'll complete the entire thing i think.
Also, for now all the dbs are memory-kept, still SQL just in memory.

@latekvo
Copy link
Owner Author

latekvo commented Jun 10, 2024

summarizer is now working since the latest commit

@latekvo
Copy link
Owner Author

latekvo commented Jun 10, 2024

all workers are now launching 🥳

@latekvo latekvo requested review from SOSNE and Nekxis June 11, 2024 21:53
Copy link
Collaborator

@Nekxis Nekxis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@latekvo
Copy link
Owner Author

latekvo commented Jun 12, 2024

(c.c. @SOSNE ) 🪨

Copy link
Collaborator

@SOSNE SOSNE left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me

@latekvo latekvo merged commit 53b0385 into main Jun 12, 2024
1 check passed
@latekvo latekvo deleted the @latekvo/change_database_system branch June 16, 2024 09:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants