Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

embeddings persistence #83

Open
sbradford006 opened this issue May 31, 2023 · 4 comments
Open

embeddings persistence #83

sbradford006 opened this issue May 31, 2023 · 4 comments

Comments

@sbradford006
Copy link

I'm attempting to have the embeddings in the redis (api) container persist a restart.

having mounted /data to a dir on localhost I only ever see two dirs (/data/redis and /data/redisinsight). Neither of these seem to contain any data...

i've played around with adding --save config in docker-compose but i am no docker wizard and it looks like any config passed at compose time nukes the default config configured in the container.

very possible i am misunderstanding how this should all hang together... but any advice would be welcome!

@itmilos
Copy link

itmilos commented May 31, 2023

@sbradford006 you should have redis dump in storage service

@sbradford006
Copy link
Author

Thanks for your input itmilos. Do you mean the storage service for the documents?

Currently documents are stored in Azure blob storage but there is a local (containerizes) redis for embeddings. I don't see anything other than explicitly uploaded documents in the Azure storage unfortunately.

Have i misunderstood your comment?

@itmilos
Copy link

itmilos commented May 31, 2023

If you delte this and reset redis you will remove all embeddings

Screenshot 2023-05-31 at 20 12 12

@sbradford006
Copy link
Author

Thanks again for your help - I did have that share on the storage account and have removed it. Currently assuming it was a hangover from a previous iteration of the project where the entire service was built in Azure (before i realized how expensive Redis was going to be!) as that file share hasn't been recreated.

Unfortunately the local Redis instance still appears to lose all embeddings after a container restart and unfortunately simply using the batch operation "convert all files and embeddings" doesn't reproduce them.

image

Any chance you know where the embeddings might be held on the API container, if it isn't /data?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants