Fixed issues, renamed mixtral to mistral. Introduced new messages for…

…matter. etc.
Maximilian-Winter · May 13, 2024 · 54836e2 · 54836e2
1 parent b8eacd0
commit 54836e2
Show file tree

Hide file tree

Showing 37 changed files with 659 additions and 643 deletions.
diff --git a/ReadMe.md b/ReadMe.md
@@ -107,33 +107,42 @@ This example, based on an example from the Instructor library for OpenAI, shows
 ## Additional Information
 
 ### Predefined Messages Formatter
+
 The llama-cpp-agent framework provides predefined message formatters to format messages for the LLM model. The `MessagesFormatterType` enum defines the available formatters:
 
+- `MessagesFormatterType.MISTRAL`: Formats messages using the MISTRAL format.
 - `MessagesFormatterType.CHATML`: Formats messages using the CHATML format.
-- `MessagesFormatterType.MIXTRAL`: Formats messages using the MIXTRAL format.
 - `MessagesFormatterType.VICUNA`: Formats messages using the VICUNA format.
 - `MessagesFormatterType.LLAMA_2`: Formats messages using the LLAMA 2 format.
 - `MessagesFormatterType.SYNTHIA`: Formats messages using the SYNTHIA format.
 - `MessagesFormatterType.NEURAL_CHAT`: Formats messages using the NEURAL CHAT format.
 - `MessagesFormatterType.SOLAR`: Formats messages using the SOLAR format.
 - `MessagesFormatterType.OPEN_CHAT`: Formats messages using the OPEN CHAT format.
+- `MessagesFormatterType.ALPACA`: Formats messages using the ALPACA format.
+- `MessagesFormatterType.CODE_DS`: Formats messages using the CODE DS format.
+- `MessagesFormatterType.B22`: Formats messages using the B22 format.
+- `MessagesFormatterType.LLAMA_3`: Formats messages using the LLAMA 3 format.
+- `MessagesFormatterType.PHI_3`: Formats messages using the PHI 3 format.
 
 ### Creating Custom Messages Formatter
+
 You can create your own custom messages formatter by instantiating the `MessagesFormatter` class with the desired parameters:
 
 ```python
-from llama_cpp_agent.messages_formatter import MessagesFormatter
+from llama_cpp_agent.chat_history.messages_formatter import MessagesFormatter, PromptMarkers, Roles
+
+custom_prompt_markers = {
+    Roles.system: PromptMarkers("<|system|>", "<|endsystem|>"),
+    Roles.user: PromptMarkers("<|user|>", "<|enduser|>"),
+    Roles.assistant: PromptMarkers("<|assistant|>", "<|endassistant|>"),
+    Roles.tool: PromptMarkers("<|tool|>", "<|endtool|>"),
+}
 
 custom_formatter = MessagesFormatter(
-    PRE_PROMPT="",
-    SYS_PROMPT_START="<|system|>",
-    SYS_PROMPT_END="<|endsystem|>",
-    USER_PROMPT_START="<|user|>",
-    USER_PROMPT_END="<|enduser|>",
-    ASSISTANT_PROMPT_START="<|assistant|>",
-    ASSISTANT_PROMPT_END="<|endassistant|>",
-    INCLUDE_SYS_PROMPT_IN_FIRST_USER_MESSAGE=False,
-    DEFAULT_STOP_SEQUENCES=["<|endsystem|>", "<|enduser|>", "<|endassistant|>"]
+    pre_prompt="",
+    prompt_markers=custom_prompt_markers,
+    include_sys_prompt_in_first_user_message=False,
+    default_stop_sequences=["<|endsystem|>", "<|enduser|>", "<|endassistant|>", "<|endtool|>"]
 )
 ```
 

diff --git a/docs/chat_history-api-reference.md b/docs/chat_history-api-reference.md
@@ -0,0 +1,11 @@
+---
+title: Chat History Reference
+---
+
+## Basic Chat History and Message store
+
+::: llama_cpp_agent.chat_history.basic_chat_history
+
+## Chat History and Message store
+
+::: llama_cpp_agent.chat_history.chat_history_base
diff --git a/docs/get-started.md b/docs/get-started.md
@@ -19,6 +19,7 @@ Welcome to the llama-cpp-agent framework! This guide will help you set up and st
 5. [Talk to the agent](#talk-to-the-agent)
     - [Get chat response](#talk-to-the-agent)
     - [Sampling parameters](#sampling-parameters)
+    - [Configure Chat History]
 6. [Let the agent use tools](#let-the-agent-use-tools)
     - [Define function tool](#let-the-agent-use-tools)
     - [Using the FunctionCallingAgent class](#using-the-functioncallingagent-class)
@@ -133,6 +134,33 @@ agent_output = agent.get_chat_response("Hello, World!", llm_sampling_settings=se
 print(f"Agent: {agent_output.strip()}")
 ```
 
+### Configure Chat History
+The llama-cpp-agent framework allows to customize the chat history by implementing the ChatHistory and ChatMessageStore interfaces.
+
+It provides basic implementation of it, in the BasicChatHistory and BasicChatMessageStore class and allows to configure it, like in the following example.
+
+```python
+# Import the BasicChatHistory class, the BasicChatMessageStore class and BasicChatHistoryStrategy class
+from llama_cpp_agent.chat_history import BasicChatHistory, BasicChatMessageStore, BasicChatHistoryStrategy
+
+# Create the provider like in the previous step.
+provider = ...
+
+# Create a message store for the chat history
+chat_history_store = BasicChatMessageStore()
+
+# Create the actual chat history, by passing the wished chat history strategy, it can be last_k_message or last_k_tokens. The default strategy will be to use the 20 last messages for the chat history.
+# We will use the last_k_tokens strategy which will include the last k tokens into the chat history. When we use this strategy, we will have to pass the provider to the class.
+chat_history = BasicChatHistory(message_store=chat_history_store, chat_history_strategy=BasicChatHistoryStrategy.last_k_tokens, k=7000, llm_provider=provider)
+
+# Pass the configured chat history to
+agent = LlamaCppAgent(provider,
+                      system_prompt="You are a helpful assistant.",
+                      chat_history=chat_history
+                      predefined_messages_formatter_type=MessagesFormatterType.CHATML)
+
+```
+
 ## Let the agent use tools
 To let the agent use tools and call function, we need to pass an instance of the `LlmStructuredOutputSettings` class to the `get_chat_response` method.
 

diff --git a/docs/map_chain.md b/docs/map_chain.md
@@ -1,5 +1,6 @@
 ### Mapping Chain Example
 This example demonstrates how to create a mapping chain to summarize 3 articles into one summary.
+
 ```python
 from llama_cpp_agent import AgentChainElement, MapChain
 from llama_cpp_agent import LlamaCppAgent
@@ -11,7 +12,7 @@ model = LlamaCppServerProvider("http://127.0.0.1:8080")
 agent = LlamaCppAgent(
     model,
     system_prompt="",
-    predefined_messages_formatter_type=MessagesFormatterType.MIXTRAL
+    predefined_messages_formatter_type=MessagesFormatterType.MISTRAL
 )
 
 summary_chain = AgentChainElement("out_0", system_prompt="You are an advanced AI agent for summarizing articles",

diff --git a/docs/rag.md b/docs/rag.md
@@ -1,5 +1,6 @@
 ### RAG - Retrieval Augmented Generation
 This example shows how to do RAG with colbert reranking. You have to install the optional rag dependencies (ragatouille) to use the RAGColbertReranker class and this example.
+
 ```python
 import json
 
@@ -21,7 +22,6 @@ from llama_cpp_agent.providers.llama_cpp_endpoint_provider import (
 from llama_cpp_agent.rag.rag_colbert_reranker import RAGColbertReranker
 from llama_cpp_agent.rag.text_utils import RecursiveCharacterTextSplitter
 
-
 # Initialize the chromadb vector database with a colbert reranker.
 rag = RAGColbertReranker(persistent=False)
 
@@ -48,6 +48,7 @@ for split in splits:
 # Define the query we want to ask based on the retrieved information
 query = "What is a BARS apparatus?"
 
+
 # Define a pydantic class to represent a query extension as additional queries to the original query.
 class QueryExtension(BaseModel):
     """
@@ -67,7 +68,7 @@ query_extension_agent = LlamaCppAgent(
     main_model,
     debug_output=True,
     system_prompt="You are a world class query extension algorithm capable of extending queries by writing new queries. Do not answer the queries, simply provide a list of additional queries in JSON format. Structure your output according to the following model:\n\n" + docs.strip(),
-    predefined_messages_formatter_type=MessagesFormatterType.MIXTRAL
+    predefined_messages_formatter_type=MessagesFormatterType.MISTRAL
 )
 
 # Perform the query extension with the agent.
@@ -98,7 +99,7 @@ agent_with_rag_information = LlamaCppAgent(
     main_model,
     debug_output=True,
     system_prompt="You are an advanced AI assistant, trained by OpenAI. Only answer question based on the context information provided.",
-    predefined_messages_formatter_type=MessagesFormatterType.MIXTRAL
+    predefined_messages_formatter_type=MessagesFormatterType.MISTRAL
 )
 
 # Ask the agent the original query with the generated prompt that contains the retrieved information.

diff --git a/docs/sequential_chain.md b/docs/sequential_chain.md
@@ -1,5 +1,6 @@
 ### Sequential Chain Example
 This example demonstrates how to create a complete product launch campaign with help of a sequential chain.
+
 ```python
 # Example: Product Launch Campaign (Product Description, USP, Target Audience, Marketing Channels, Ad Copy, Landing Page, Email Campaign, Social Media Posts, Press Release, and Performance Metrics)
 from llama_cpp_agent import AgentChainElement, AgentChain
@@ -12,7 +13,7 @@ model = LlamaCppServerProvider("http://127.0.0.1:8080")
 agent = LlamaCppAgent(
     model,
     system_prompt="",
-    predefined_messages_formatter_type=MessagesFormatterType.MIXTRAL
+    predefined_messages_formatter_type=MessagesFormatterType.MISTRAL
 )
 
 product_description = AgentChainElement(
@@ -75,7 +76,8 @@ performance_metrics = AgentChainElement(
     prompt="Identify the key performance metrics to track the success of the {product_name} launch campaign based on the following marketing channels, ad copy, landing page, email campaign, social media posts, and press release:\n--\nMarketing Channels:\n{out_3}\nAd Copy:\n{out_4}\nLanding Page Structure:\n{out_5}\nEmail Campaign:\n{out_6}\nSocial Media Posts:\n{out_7}\nPress Release:\n{out_8}"
 )
 
-chain = [product_description, product_usp, target_audience, marketing_channels, ad_copy, landing_page, email_campaign, social_media_posts, press_release, performance_metrics]
+chain = [product_description, product_usp, target_audience, marketing_channels, ad_copy, landing_page, email_campaign,
+         social_media_posts, press_release, performance_metrics]
 agent_chain = AgentChain(agent, chain)
 agent_chain.run_chain(additional_fields={"product_name": "Smart Fitness Tracker"})
 

diff --git a/examples/01_Basics/chatbot_using_llama_cpp_python.py b/examples/01_Basics/chatbot_using_llama_cpp_python.py
@@ -4,22 +4,20 @@
 from llama_cpp_agent import MessagesFormatterType
 from llama_cpp_agent.providers import LlamaCppPythonProvider
 
+llama_model = Llama(r"C:\AI\Agents\gguf-models\mistral-7b-instruct-v0.2.Q6_K.gguf", n_batch=1024, n_threads=10, n_gpu_layers=33, n_ctx=8192, verbose=False)
 
-llama_model = Llama(r"C:\AI\Agents\gguf-models\mistral-7b-instruct-v0.2.Q6_K.gguf", n_batch=1024, n_threads=10,
-                    n_gpu_layers=40)
 provider = LlamaCppPythonProvider(llama_model)
 
 agent = LlamaCppAgent(
     provider,
     system_prompt="You are a helpful assistant.",
-    predefined_messages_formatter_type=MessagesFormatterType.CHATML,
+    predefined_messages_formatter_type=MessagesFormatterType.MISTRAL,
+    debug_output=True
 )
+
 settings = provider.get_provider_default_settings()
-settings.max_tokens = 512
-settings.temperature = 0.65
+settings.max_tokens = 2000
+settings.stream = True
 while True:
-    user_input = input(">")
-    if user_input == "exit":
-        break
-    agent_output = agent.get_chat_response(user_input, llm_sampling_settings=settings)
+    agent_output = agent.get_chat_response("Hello!", llm_sampling_settings=settings)
     print(f"Agent: {agent_output.strip()}")
diff --git a/examples/01_Basics/chatbot_using_llama_cpp_server.py b/examples/01_Basics/chatbot_using_llama_cpp_server.py
@@ -0,0 +1,23 @@
+from llama_cpp_agent import LlamaCppAgent
+from llama_cpp_agent import MessagesFormatterType
+from llama_cpp_agent.providers import LlamaCppServerProvider
+
+provider = LlamaCppServerProvider("http://127.0.0.1:8080")
+
+agent = LlamaCppAgent(
+    provider,
+    system_prompt="You are a helpful assistant.",
+    predefined_messages_formatter_type=MessagesFormatterType.CHATML,
+)
+
+settings = provider.get_provider_default_settings()
+settings.n_predict = 512
+settings.temperature = 0.65
+agent.chat_history.get_message_store().load_from_json("test.json")
+while True:
+    user_input = input(">")
+    if user_input == "exit":
+        break
+    agent_output = agent.get_chat_response(user_input, llm_sampling_settings=settings)
+    agent.chat_history.get_message_store().save_to_json("test.json")
+    print(f"Agent: {agent_output.strip()}")
diff --git a/examples/01_Basics/chatbot_using_tgi_server.py b/examples/01_Basics/chatbot_using_tgi_server.py
@@ -7,7 +7,7 @@
 agent = LlamaCppAgent(
     provider,
     system_prompt="You are a helpful assistant.",
-    predefined_messages_formatter_type=MessagesFormatterType.MIXTRAL,
+    predefined_messages_formatter_type=MessagesFormatterType.MISTRAL,
 )
 
 settings = provider.get_provider_default_settings()

diff --git a/examples/01_Basics/chatbot_using_vllm_server.py b/examples/01_Basics/chatbot_using_vllm_server.py
@@ -7,7 +7,7 @@
 agent = LlamaCppAgent(
     provider,
     system_prompt="You are a helpful assistant.",
-    predefined_messages_formatter_type=MessagesFormatterType.CHATML,
+    predefined_messages_formatter_type=MessagesFormatterType.LLAMA_2,
 )
 
 settings = provider.get_provider_default_settings()

diff --git a/examples/01_Basics/test.json b/examples/01_Basics/test.json
@@ -0,0 +1,30 @@
+[
+    {
+        "role": "system",
+        "content": "You are a helpful assistant."
+    },
+    {
+        "role": "user",
+        "content": "Hi!"
+    },
+    {
+        "role": "assistant",
+        "content": "Hello! How can I help you today?"
+    },
+    {
+        "role": "user",
+        "content": "Tell me about the Iran Contra Affair"
+    },
+    {
+        "role": "assistant",
+        "content": "The Iran-Contra Affair, also known as the Iran-Contra Scandal or the Iran-Contra Crisis, refers to a political scandal in the United States that came to light in November 1986. It involved the sale of arms to Iran, which was under an arms embargo at the time, and the diversion of proceeds from those sales to fund the Contras in Nicaragua, contrary to Congressional restrictions.\n\nThe scandal began when it was revealed that President Ronald Reagan had authorized the sale of weapons to Iran in exchange for the release of American hostages held by Hezbollah militants in Lebanon. The proceeds from these sales were then secretly used to fund the Contras, who were fighting against the Marxist Sandinista government in Nicaragua. This was a violation of both the Arms Export Control Act and the Boland Amendment, which prohibited U.S. funding for the Contras.\n\nThe scandal came to light when Lebanese journalist Al Faqih published an article in the Beirut daily Al-Shiraa detailing the arms sales on November 3, 1986. The story was then picked up by the American media, leading to a public outcry and investigations by Congress and the press.\n\nThe Iran-Contra Affair led to numerous resignations, including that of National Security Advisor Robert McFarlane and Secretary of Defense Caspar Weinberger. Several high-ranking officials were indicted and tried for their roles in the scandal, including former CIA Director John Poindexter and Lieutenant Colonel Oliver North.\n\nThe affair damaged the Reagan Administration's reputation and led to increased scrutiny of executive power and the role of Congress in foreign policy. It also highlighted the complexities of U.S. foreign policy and the challenges of balancing national security interests with democratic values.\n"
+    },
+    {
+        "role": "user",
+        "content": "What was my last question?"
+    },
+    {
+        "role": "assistant",
+        "content": "Your last question was about the Iran-Contra Affair."
+    }
+]
diff --git a/examples/04_Chains/article_summary.py b/examples/04_Chains/article_summary.py
@@ -10,7 +10,7 @@
     provider,
     debug_output=True,
     system_prompt="",
-    predefined_messages_formatter_type=MessagesFormatterType.MIXTRAL
+    predefined_messages_formatter_type=MessagesFormatterType.MISTRAL
 )
 
 article_summary = AgentChainElement(

diff --git a/examples/04_Chains/blog_post_outline_introduction.py b/examples/04_Chains/blog_post_outline_introduction.py
@@ -10,7 +10,7 @@
     model,
     debug_output=True,
     system_prompt="",
-    predefined_messages_formatter_type=MessagesFormatterType.MIXTRAL
+    predefined_messages_formatter_type=MessagesFormatterType.MISTRAL
 )
 
 blog_post_outline = AgentChainElement(

diff --git a/examples/04_Chains/complete_blog_post_creation.py b/examples/04_Chains/complete_blog_post_creation.py
@@ -10,7 +10,7 @@
     provider,
     debug_output=True,
     system_prompt="",
-    predefined_messages_formatter_type=MessagesFormatterType.MIXTRAL
+    predefined_messages_formatter_type=MessagesFormatterType.MISTRAL
 )
 blog_post_topic = AgentChainElement(
     output_identifier="out_0",

diff --git a/examples/04_Chains/event_description_email.py b/examples/04_Chains/event_description_email.py
@@ -10,7 +10,7 @@
     model,
     debug_output=True,
     system_prompt="",
-    predefined_messages_formatter_type=MessagesFormatterType.MIXTRAL
+    predefined_messages_formatter_type=MessagesFormatterType.MISTRAL
 )
 
 event_description = AgentChainElement(

diff --git a/examples/04_Chains/job_description_requirements.py b/examples/04_Chains/job_description_requirements.py
@@ -10,7 +10,7 @@
     model,
     debug_output=True,
     system_prompt="",
-    predefined_messages_formatter_type=MessagesFormatterType.MIXTRAL
+    predefined_messages_formatter_type=MessagesFormatterType.MISTRAL
 )
 
 job_description = AgentChainElement(

diff --git a/examples/04_Chains/map_chain_summary.py b/examples/04_Chains/map_chain_summary.py
@@ -9,7 +9,7 @@
     model,
     debug_output=True,
     system_prompt="",
-    predefined_messages_formatter_type=MessagesFormatterType.MIXTRAL
+    predefined_messages_formatter_type=MessagesFormatterType.MISTRAL
 )
 
 summary_chain = AgentChainElement("out_0", system_prompt="You are an advanced AI agent for summarizing articles",

diff --git a/examples/04_Chains/map_chain_summary_translate.py b/examples/04_Chains/map_chain_summary_translate.py
@@ -9,7 +9,7 @@
     model,
     debug_output=True,
     system_prompt="",
-    predefined_messages_formatter_type=MessagesFormatterType.MIXTRAL
+    predefined_messages_formatter_type=MessagesFormatterType.MISTRAL
 )
 
 

diff --git a/examples/04_Chains/math_operation_greeting.py b/examples/04_Chains/math_operation_greeting.py
@@ -13,7 +13,7 @@
     model,
     debug_output=True,
     system_prompt="",
-    predefined_messages_formatter_type=MessagesFormatterType.MIXTRAL
+    predefined_messages_formatter_type=MessagesFormatterType.MISTRAL
 )
 
 

diff --git a/examples/04_Chains/math_reasoning.py b/examples/04_Chains/math_reasoning.py
@@ -12,7 +12,7 @@
     model,
     debug_output=False,
     system_prompt="",
-    predefined_messages_formatter_type=MessagesFormatterType.MIXTRAL
+    predefined_messages_formatter_type=MessagesFormatterType.MISTRAL
 )
 
 

diff --git a/examples/04_Chains/product_comparison_recommendation.py b/examples/04_Chains/product_comparison_recommendation.py
@@ -10,7 +10,7 @@
     model,
     debug_output=True,
     system_prompt="",
-    predefined_messages_formatter_type=MessagesFormatterType.MIXTRAL
+    predefined_messages_formatter_type=MessagesFormatterType.MISTRAL
 )
 
 product_comparison = AgentChainElement(

diff --git a/examples/04_Chains/product_description_ad_copy.py b/examples/04_Chains/product_description_ad_copy.py
@@ -10,7 +10,7 @@
     model,
     debug_output=True,
     system_prompt="",
-    predefined_messages_formatter_type=MessagesFormatterType.MIXTRAL
+    predefined_messages_formatter_type=MessagesFormatterType.MISTRAL
 )
 
 product_description = AgentChainElement(