Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change to use pre-trained model feature to manage REconverge and Hog models #178

Open
syncpark opened this issue Jan 18, 2024 · 4 comments

Comments

@syncpark
Copy link
Contributor

syncpark commented Jan 18, 2024

Issues:

  1. Currently REconverge model is stored in Postgres DB. Problems occur when saving the model file if it is too large.
  2. Hog models are not maintained currently.

Proposal:

  1. Let's store all model files into pretrained directory instead of Postgres DB.
  2. Change model table
    • Add description column to store model related information
    • Add path to specify model file path
  3. Add or change GraphQL APIs
    • Add new APIs to register or remove a model
    • Change models API to query model list including kind, description
  4. Extend model structure to store other kind of models like Hog's models
@syncpark
Copy link
Contributor Author

@kimhanbeom @MW-Kim @sehkone @minshao @msk
Give your opinion on this issue.

@msk
Copy link
Contributor

msk commented Jan 18, 2024

I'm okay with points 1, 2, and 4 in @syncpark's proposal for the model storage. Point 3, however, is outside the scope of this project. I suggest that this aspect be discussed in the review-web project.

@minshao
Copy link
Contributor

minshao commented Jan 18, 2024

  1. On condition that the model could get large, it is indeed a better solution to store them as file and store reference in the database. Further, if the current usage of pretrained collides, then we can store model files there, otherwise, it is better to have a user configurable path designated for storing model file.
  2. and 4. both look good to me.

@MW-Kim
Copy link

MW-Kim commented Jan 19, 2024

As the analysis continues to reconverge, the actual model size keeps increasing. It would be a good idea to store and manage models in the pretrained directory rather than in PostgreSQL.
The following is the model size for a site with high traffic.

postgres=# SELECT pg_size_pretty(pg_table_size('model'));
 pg_size_pretty
----------------
 1193 MB
(1 row)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants