DPR create_embeddings/update_embeddings FAISS is so slow! #5641
Unanswered
shahad2099
asked this question in
Questions
Replies: 1 comment 4 replies
-
Hi @shahad2099 happy to answer your questions! 🙂
|
Beta Was this translation helpful? Give feedback.
4 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hello everyone,
I hope you're all doing well. I am using the Haystack framework to build a retriever. After training DPR, I wanted to use FAISS as my vector database. However, updating or creating embeddings is so slow! I have 3 million short documents (100 words/document), and for only 17% of them, it takes almost 5 hours and 38 minutes. This is incredibly frustrating for me.
![image](https://private-user-images.githubusercontent.com/119281575/263699796-d018f031-0f24-47f9-b461-36169ced6383.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQxMDg4MTcsIm5iZiI6MTcyNDEwODUxNywicGF0aCI6Ii8xMTkyODE1NzUvMjYzNjk5Nzk2LWQwMThmMDMxLTBmMjQtNDdmOS1iNDYxLTM2MTY5Y2VkNjM4My5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODE5JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgxOVQyMzAxNTdaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1kNzM2ZWYyNmNlMTY1MTFhNjU4NDYyODI1N2ZkMzgzYzQxMmQxMjhlZmY3YzVjMmQ4MDBhOTE3ODhkOTIwMjI1JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.PAKGcGMzxY8rsEcLq8QTnnS6bpk3XD_6_SiGCtxT0zk)
What should I do? Are there any optimizations that I should implement in the code?
I also have some simple questions:
1-If I want to add more data, should I store all the documents and create embeddings from scratch? Or can I add and create embeddings for the new ones only?
2-In your opinion, what is the most efficient vector database for a retriever?
3-Does the update_embeddings function utilize GPUs or multi-threading?
Many thanks to all of you!
Beta Was this translation helpful? Give feedback.
All reactions