LangServe access POST data in path function fast api

[seanmavley]

Hello.

I'm sending this to the langserve endpoint

{
    'input': message,
    'history': all_conversation_chats
}

Currently my langchain route is

# Edit this to add the chain you want to add
add_routes(app, rag_chroma_private_chain, path="/myendpoint")

I intend to use the all_conversation_chats array to build a ChatMessageHistory and feed into the chat prompt template to the LLM

How do I access the <payload>.history field in the langchain context?

In other words, when I sent a frontend request, of say, { 'mymessage': 'message' } to the path="/myendpoint" in LangServe, how do I access the mymessage field in the function?

LangServe "automagically" knows to look for the input field. When there are other fields, how do I access them?

[eyurtsev]

Hello! If I understood correctly, you're trying to access the payload fields on the backend of langserve and then do something with that in the context of runnables?

If so, I don't have any simpler examples than this right now that does some LCEL manipulation: https://github.com/langchain-ai/langserve/blob/main/examples/passthrough_dict/server.py

I'll try to add something simpler. Take a look at LCEL docs in langchain to create whatever runnable you need https://python.langchain.com/docs/expression_language/get_started.

Once the runnable is created and is exposed via add_routes, it's interface is pretty standard and you can include the input to the runnables via an input field in the post request.

Here's an example fro the README.md

curl --location --request POST 'http://localhost:8000/joke/invoke' \
    --header 'Content-Type: application/json' \
    --data-raw '{
        "input": {
            "topic": "cats"
        }
    }'

---

If you're looking for more examples of chat backend:

Conversational retrieval
https://github.com/langchain-ai/langserve/tree/main/examples/conversational_retrieval_chain

Chat with history persisted on backend:
https://github.com/langchain-ai/langserve/tree/main/examples/chat_with_persistence

Chat with history persisted on backend and user information from headers
https://github.com/langchain-ai/langserve/tree/main/examples/chat_with_persistence_and_user

You then have an option of either persisting some information in temporary caches on the front end. or you can add a regular fast API endpoint to fetch chat history for a given session

[seanmavley]

@eyurtsev Thanks for the response.

you're trying to access the payload fields on the backend of langserve and then do something with that in the context of runnables?

Yep, exactly!

If so, I don't have any simpler examples than this right now that does some LCEL manipulation: https://github.com/langchain-ai/langserve/blob/main/examples/passthrough_dict/server.py

I'll try to add something simpler. Take a look at LCEL docs in langchain to create whatever runnable you need https://python.langchain.com/docs/expression_language/get_started.

That'll be great.

Will revisit the links for better understanding of the examples.

[seanmavley]

@eyurtsev I'm not sure what I may be doing wrong in this case, trying to follow the example in the passthrough_dict example

https://github.com/langchain-ai/langserve/blob/main/examples/passthrough_dict/server.py

from typing import Any, Callable, Dict, List, Optional, TypedDict
from langchain.vectorstores import Qdrant
from langchain.document_loaders import PyPDFLoader
from langchain.document_loaders import DirectoryLoader
from langchain.chat_models import ChatOllama
from langchain.embeddings import GPT4AllEmbeddings
from langchain.pydantic_v1 import BaseModel
from langchain.schema.output_parser import StrOutputParser
from langchain.schema.runnable import RunnableParallel, RunnablePassthrough, RunnableLambda, RunnableMap
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.prompts import PromptTemplate
from langchain.utils.math import cosine_similarity

dirpath = "docs"
papers = []
loader = DirectoryLoader(dirpath, glob="./*.pdf", loader_cls=PyPDFLoader)
papers = loader.load()

full_text = ''
for paper in papers:
    full_text = full_text + paper.page_content

full_text = " ".join(l for l in full_text.splitlines() if l)
print(len(full_text))  # 1466859

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200
)

document_chunks = text_splitter.create_documents([full_text])

embeddings = GPT4AllEmbeddings()

url = 'http://0.0.0.0:6333'
qdrant = Qdrant.from_documents(
    documents=document_chunks,
    embedding=embeddings,
    url=url,
    collection_name="arxiv_papers",
)

retriever = qdrant.as_retriever()

template1= """Prompt. \
Answer the question only based on the context provide below:
{history}
{context}
{question}"""

template2 = .... # same as template 1

prompt_templates = [template1, template2]
prompt_embeddings = embeddings.embed_documents(prompt_templates)


def prompt_router(input):
    print(input)
    query_embedding = embeddings.embed_query(input["question"])
    similarity = cosine_similarity([query_embedding], prompt_embeddings)[0]
    most_similar = prompt_templates[similarity.argmax()]
    print("Using Template 1" if most_similar == template1 else "Using Template 2")
    return PromptTemplate.from_template(most_similar)

def _create_projection(
    *, include_keys: Optional[List] = None, exclude_keys: Optional[List[str]] = None
) -> Callable[[dict], dict]:
    """Create a projection function."""

    def _project_dict(
        d: dict,
    ) -> dict:
        """Project dictionary."""
        keys = d.keys()
        if include_keys is not None:
            keys = set(keys) & set(include_keys)
        if exclude_keys is not None:
            keys = set(keys) - set(exclude_keys)
        return {k: d[k] for k in keys}

    return _project_dict

ollama_llm = "llama2:7b-chat"
model = ChatOllama(model=ollama_llm)

underlying_chain = RunnableParallel({
    "context": retriever,
    "question": RunnablePassthrough(),
    "history": RunnablePassthrough(),
 }) | RunnableLambda(prompt_router) | model

wrapped_chain = RunnableMap(
    {
        "output": _create_projection(exclude_keys=["info"]) | underlying_chain,
        "info": _create_projection(include_keys=["info"]),
    }
)

class Input(TypedDict):
    question: str
    history: str
    info: Dict[str, Any]

class Output(TypedDict):
    output: underlying_chain.output_schema
    info: Dict[str, Any]

chain = wrapped_chain.with_types(input_type=Input, output_type=Output)

And then in the server.py, I have this

add_routes(app, rag_chroma_private_chain, path="/endpoint")

When I send this payload

{
    "input": {
        "question": "question",
        "history": "text",
        "info": {
            "user_id": 42,
            "user_info": {
                "address": 42
            }
        }
    }
}

I get

File "/home/person/Development/AI/llm/venv/lib/python3.11/site-packages/gpt4all/pyllmodel.py", line 326, in generate_embedding
    c_text = ctypes.c_char_p(text.encode('utf-8'))
                             ^^^^^^^^^^^
AttributeError: 'dict' object has no attribute 'encode'

[eyurtsev]

Looks like the error is from pyllmodel; the input into the model is a dict rather than text; Could you confirm that you're passing a list of messages of a prompt value into the model?

``python
def raise_value_error(inputs):
raise ValueError(inputs)

| RunnableLambda(prompt_router) | raise_value_error | model

[seanmavley]

You're right. The error is coming from the pymodel

I looked into it and described it in detail here the end results I'm looking for

langchain-ai/langchain#15042 (reply in thread)

But in summary, I'm looking for a way to arbitrarily access any variable anywhere at any point in the chain.

Currently, it doesn't let me access the fields in a runnableparallel.

[seanmavley]

``python
def raise_value_error(inputs):
raise ValueError(inputs)

Solved! Thank you.

By printing the values, it means I can even simply do this in the chain.

def _get_question(inputs):
    print(inputs)
    return inputs["question"]

Not sure why that didn't occur to me earlier. I've been obsessed looking for a solution via the Runnable*, yet I could just return the values I want.

Sigh!