embeddings persistence

[sbradford006]

I'm attempting to have the embeddings in the redis (api) container persist a restart.

having mounted /data to a dir on localhost I only ever see two dirs (/data/redis and /data/redisinsight). Neither of these seem to contain any data...

i've played around with adding --save config in docker-compose but i am no docker wizard and it looks like any config passed at compose time nukes the default config configured in the container.

very possible i am misunderstanding how this should all hang together... but any advice would be welcome!

[itmilos]

@sbradford006 you should have redis dump in storage service

[sbradford006]

Thanks for your input itmilos. Do you mean the storage service for the documents?

Currently documents are stored in Azure blob storage but there is a local (containerizes) redis for embeddings. I don't see anything other than explicitly uploaded documents in the Azure storage unfortunately.

Have i misunderstood your comment?

[itmilos]

If you delte this and reset redis you will remove all embeddings

[sbradford006]

Thanks again for your help - I did have that share on the storage account and have removed it. Currently assuming it was a hangover from a previous iteration of the project where the entire service was built in Azure (before i realized how expensive Redis was going to be!) as that file share hasn't been recreated.

Unfortunately the local Redis instance still appears to lose all embeddings after a container restart and unfortunately simply using the batch operation "convert all files and embeddings" doesn't reproduce them.

Any chance you know where the embeddings might be held on the API container, if it isn't /data?