Prompt Management: Caching on SDK-level

[clemra]

The latest release of the Python and JS/TS Langfuse SDK includes a new prompt caching feature that improves the reliability and performance of your applications.

After launching prompt management in Langfuse, we've enhanced its performance and reliability in the Python and JS/TS SDKs through prompt caching.

Example (Python SDK)

# Get current production prompt version, default cache of 60 seconds
prompt = langfuse.get_prompt("prompt name")

# Get current production prompt version and cache for 5 minutes
prompt = langfuse.get_prompt("prompt name", cache_ttl_seconds=300)
 
# Get a specific prompt version and cache for 5 minutes
prompt = langfuse.get_prompt("prompt name", version=3, cache_ttl_seconds=300)
 
# Disable caching for a prompt
prompt = langfuse.get_prompt("prompt name", cache_ttl_seconds=0)

This feature stores prompts in the client SDKs' memory, cutting down on server requests and boosting your applications' reliability and speed. With a default TTL of 60 seconds, caching settings can be adjusted for individual prompts within the SDK. Should a refetch of a prompt fail, the SDK will automatically revert to the cached version, ensuring your application's availability is not compromised.

• Head over to the docs for more information: https://langfuse.com/docs/prompts#caching-in-client-sdks
• Changelog: https://langfuse.com/changelog/2024-02-05-sdk-level-prompt-caching

Looking forward to hearing your thoughts!

cc @hassiebp

[marcklingen]

We have made an update to this behavior to further optimize performance. Whenever a cached version is available it is returned immediately while checking whether an updated prompt version is available is done in the background.

https://langfuse.com/changelog/2024-09-04-prompt-management-zero-latency