Performance issues when fetching prompts via python SDK in async mode

[bbaabemhp]

Hey guys,

I recently wanted to use Langfuse in Async mode and stumbled across AsyncFernLangfuse class.

I wanted to use it but I am unsure how to.

I have a self-hosted Langfuse instance on Azure.

This is the code:

from langfuse.api.client import AsyncFernLangfuse

@observe()
async def get_system_prompt(name: str, prompt_type: Optional[str] = None):
    langfuse = AsyncFernLangfuse(
        x_langfuse_public_key=settings.LANGFUSE_PUBLIC_KEY,
        base_url=settings.LANGFUSE_HOST,
        )

    try:
        return await langfuse.prompts.get(name)
    except Exception as err:
        raise Exception(err)

I always get an unauthorized error. Since the client also has username and password attributes, I don't know where to obtain them since using my email and password didn't work.

base_url is https://langfuse-instance.azurewebsites.net

Thanks!

[dosubot[bot]]

To resolve the unauthorized error when using the AsyncFernLangfuse class with your self-hosted Langfuse instance on Azure, you need to authenticate using Basic Auth. The correct credentials for authentication are:

• Username: Langfuse Public Key
• Password: Langfuse Secret Key

You can obtain these API keys in the project settings of your Langfuse instance. Ensure you use these credentials when making requests to your self-hosted Langfuse instance. Additionally, if you are setting up your instance for the first time, you can use the LANGFUSE_INIT_PROJECT_PUBLIC_KEY and LANGFUSE_INIT_PROJECT_SECRET_KEY environment variables to automatically initialize these keys. For more detailed guidance, you can refer to the Langfuse API authentication documentation ^[1][2].

To continue talking to Dosu, mention @dosu.

[marcklingen]

This is an SDK client used within the Langfuse SDK and is not an interface intended for standalone use.

The Langfuse get prompt should return the prompt immediately, allowing it to run in asynchronous applications without requiring async functionality. Let me know if you have any questions or concerns.

[marcklingen]

thanks for sharing! the issue here is that langfuse = Langfuse() is instantiated every time when fetching a prompt, thereby all of the caching does not work.

suggested change which will resolve this:

langfuse = Langfuse()

@observe()
def get_system_prompt(name: str, prompt_type: Optional[str] = None) -> tuple:
    try:
        return langfuse.get_prompt(name, type=prompt_type)
    except Exception as err:
        raise Exception(err)

def fetch_prompts_on_startup():
    get_system_prompt(settings.PROMPT_USE_CASE, "chat")
    get_system_prompt("semantic-router")
    get_system_prompt("summarize-history")

app = FastAPI()
fetch_prompts_on_startup()

[bbaabemhp]

Crazy guy, thanks!

I think, still, async operations would be very nice. E.g.:

@observe()
    async def generate(self, question: str, context: str, history: str = "") -> str:
        langfuse_handler = langfuse_context.get_current_langchain_handler()
        logger.info("Start generating answer...")
        try:
            prompt_name = settings.PROMPT_USE_CASE
            prompt = get_system_prompt(prompt_name, "chat")

            compiled_prompt = prompt.compile(
                context=context, history=history, question=question
            )

            langchain_prompt = ChatPromptTemplate(prompt.get_langchain_prompt())

            langchain_prompt.metadata = {"langfuse_prompt": prompt}

            compiled_prompt = prompt.compile(
                context=context, history=history, question=question
            )

            model = self.get_model(prompt.config.get("model", None))
            chain = langchain_prompt | model | self.str_parser
            langfuse_context.update_current_observation(
                input=compiled_prompt,
                model=self.MODEL,
            )
            answer = await chain.ainvoke(
                {"context": context, "history": history, "question": question},
                config={"callbacks": [langfuse_handler]},
            )
            logger.info("Generating Answer done!")
            logger.debug(f"Generated Answer from LLM: {answer}")
            return answer

When running 50 concurrent requests, running this code without the langfuse_handler callback is much faster than using the callback. Maybe because the callback is a sync callback? :)

[marcklingen]

Happy to help.

Can you share how much slower this is? Generally the handler is built to leave the main thread as quickly as possible and move all operations to a background thread. Thereby this should not add too much latency. Would love to understand this here in more detail.

[bbaabemhp]

Nvm, was a mistake from my side haha :D thanks!

[marcklingen]

oh thanks for confirming! I'd be very keen to know if there were any performance issues in the callbackhandler.

Can you share what the issue here was? Might be interesting for others as well