refactor: RAG Refactor

[Aries-ckt]

1. RAG Assemble

• knowledge factory, can create different knowledge like pdf, docx, markdown, txt, csv, pptx, url.

from dbgpt.rag.knowledge.factory import KnowledgeFactory
knowledge = KnowledgeFactory.create(datasource="path/1.pdf", knowledge_type=KnowledgeType.DOCUMENT)

• unit tests

• rag assemblers

  • EmbeddingAssembler
  • SummaryAssembler
  • examples in examples/rag
  • unit tests

    file_path = "./docs/docs/awel.md"
    knowledge = KnowledgeFactory.from_file_path(file_path)
    embedding_model_path = "{your_embedding_model_path}"
    chunk_parameters = ChunkParameters(chunk_strategy="CHUNK_BY_SIZE")
    # get embedding assembler
    assembler = EmbeddingAssembler.load_from_knowledge(
        knowledge=knowledge,
        chunk_parameters=chunk_parameters,
        embedding_model=embedding_model_path,
    )
    assembler.persist()
    # get embeddings retriever
    retriever = assembler.as_retriever(3)
    chunks = await retriever.aretrieve_with_scores("RAG", 0.3)
    print(f"embedding rag example results:{chunks}")

• rag chunk manager
  • ChunkStrategies
  • ChunkManager

2. RAG Retrieve

• retrievers, provide retrieve() and retrieve_with_score()
  • EmbeddingRetriever
  • DBStructRetriever
  • unit tests

from dbgpt.rag.retriever.embedding import EmbeddingRetriever
from dbgpt.storage.vector_store.connector import VectorStoreConnector

embedding_fn = embedding_factory.create(
      model_name=EMBEDDING_MODEL_CONFIG[CFG.EMBEDDING_MODEL]
)
vector_name = "test"
config = ChromaVectorConfig(name=vector_name, embedding_fn=embedding_fn)
vector_store_connector = VectorStoreConnector(
    vector_store_type=""Chroma"",
     vector_store_config=config,
)
embedding_retriever = EmbeddingRetriever(
        top_k=3, vector_store_connector=vector_store_connector
)
chunks = embedding_retriever.retrieve("your query text")
print(f"embedding retriever results:{[chunk.content for chunk in chunks]}")

• extractors, can extract summary, title and keyword, etc.
  • SummaryExtractor
• query rewrite
• rerank
  • DefaultRanker

3. Examples and Unit tests

4. ChatKnowledge web add segmentation step
   

[csunny]

Follow commands that need to be run. Please run the following command to install new pre-commit hooks

pre-commit install

Every time you execute the command git commit xxx it will automatically trigger make fmt and make test

And you can manually trigger git hooks through the command pre-commit run --all-files

[csunny]

r+

[csunny]

(dbgpt_env) % python examples/rag/db_struct_rag_example.py
<class 'dbgpt.storage.vector_store.chroma_store.ChromaStore'>
Traceback (most recent call last):
  File "/Users/magic/workspace/github/eosphoros-ai/DB-GPT/examples/rag/db_struct_rag_example.py", line 47, in <module>
    vector_connector = VectorStoreConnector.from_default(
  File "/Users/magic/workspace/github/eosphoros-ai/DB-GPT/dbgpt/storage/vector_store/connector.py", line 60, in from_default
    return cls(vector_store_type, vector_store_config)
  File "/Users/magic/workspace/github/eosphoros-ai/DB-GPT/dbgpt/storage/vector_store/connector.py", line 43, in __init__
    self.client = self.connector_class(vector_store_config)
  File "/Users/magic/workspace/github/eosphoros-ai/DB-GPT/dbgpt/storage/vector_store/chroma_store.py", line 39, in __init__
    self.persist_dir = os.path.join(
  File "/Users/magic/miniconda3/envs/dbgpt_env/lib/python3.10/posixpath.py", line 76, in join
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType

[csunny]

r+

[Aralhi]

r+

[csunny]

r+