DocuMind: Your Intelligent Document Assistant 🧠

DocuMind is an advanced document processing and question-answering system that leverages the power of Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) to provide intelligent responses based on your documents and web content. (Note: This has only been tested on MacOS, might not work on other operating systems - it's Open Source so feel free to download and modify it to get it to work on your end)

Features 🌟

• Document Processing: Upload and process various document formats (TXT, DOCX, PDF).
• Web Crawling: Crawl websites and add their content to the knowledge base.
• Intelligent Chatbot: Ask questions about your documents and get informed responses.
• RAG System: Utilizes a Retrieval-Augmented Generation system for accurate and context-aware answers.
• LaTeX Rendering: Supports rendering of LaTeX equations in chat responses.
• Code Highlighting: Displays code snippets with syntax highlighting.

Screenshots 📸

Requirements 📋

• Python 3.8+
• Streamlit
• OpenAI API (compatible with LM Studio)
• FAISS
• PyPDF2
• docx2txt
• tiktoken
• latex2mathml
• And other dependencies listed in requirements.txt

Setup 🚀

1. Clone the repository:

   git clone https://github.com/yorkiedev/documind.git
   cd documind
   

2. Install the required dependencies:

   pip install -r requirements.txt
   

3. Set up LM Studio:

   • Ensure LM Studio is running and accessible at http://localhost:1234/v1
   • Configure your LM Studio with appropriate models and presets

4. Run the Streamlit app:

   streamlit run ragcrawler.py
   

Usage 📚

1. Document Upload:

   • Use the sidebar to upload your documents (TXT, DOCX, PDF formats supported).
   • The system will process and index the documents automatically.

2. Web Crawling:

   • Enter a URL in the sidebar and specify the maximum number of pages to crawl.
   • Click "Crawl and Add to RAG" to add the web content to your knowledge base.

3. Chatting with DocuMind:

   • Type your questions in the chat input at the bottom of the page.
   • DocuMind will provide answers based on the content of your documents and crawled web pages.

4. Viewing Responses:

   • Responses may include formatted text, LaTeX equations, and code snippets.
   • LaTeX equations are rendered inline for easy reading.
   • Code snippets are displayed with syntax highlighting.

5. Managing Documents:

   • Use the "Clear All Documents" button in the sidebar to reset your knowledge base.

Customization 🛠️

• Modify the EMBEDDING_MODEL and CHAT_MODEL constants in the script to use different language models.
• Adjust the chunk size and overlap in the DocumentProcessor.split_text method to fine-tune document processing.
• Customize the system prompt in the main chat loop to change the AI's behavior and personality.

Contributing 🤝

Contributions to DocuMind are welcome! Please feel free to submit pull requests, create issues or spread the word.

License 📄

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments 👏

• Thanks to the Streamlit team for their amazing framework.
• Kudos to the developers of FAISS, PyPDF2, and other libraries used in this project.
• Special thanks to the LM Studio community for their support and resources.

---

Happy document processing and chatting with DocuMind! If you have any questions or run into issues, please open an issue on GitHub. Enjoy! 🎉