🗺 Prompt Management

[mikeldking]

As teams grow, it can become important to track and manage prompts independently from code for re-use inside of Phoenix and other integration points like evaluation.

Phoenix prompts will let you find, create, modify, and version control prompts for and LLMs in general.

Technical requirements

• Prompts should be versioned
• Prompts should be provider independent
• Prompts should be cashable for high availability (both server side and client side)

Spike

• [hub] PRD #5604
• [hub][spike] 4 + 1 diagram #5676
• [hub][spike] flesh out prompt management use-cases and data model
• [hub][spike] caching of prompt templates
• [hub][spike] prompts ERD #5677
• [hub][spike] prompt template model configuration tracking
• [hub][spike] phoenix.client.prompts python SDK design #5625
• [hub] scope out client-side caching #5678
• [hub] scope out server-side caching
• [hub][spike] pydantic types for messages, tools, output_schema #5719
• [hub][spike] zod types for messages, tools, output_schema
• [hub] [spike] get a good mix of prompt template examples #5737

Technical Infra

• de-couple the Phoenix client from the Arize-Phoenix package #5684

Server

• [hub] prompt template storage (e.g. database tables) #5733
• [hub] [gql] prompt listing
• [hub] mocked out history, prompt template, tags #5762
• [hub] [gql] prompt versions list #5754
• [hub] [gql] prompt template @ version #5753
• [hub] [gql] template types #5743
• [hub] [gql] prompt create #5732
• [hub] [gql] prompt template save #5801
• [hub] [gql] version tag create #5778
• [hub] [gql] version tag patch #5800
• [hub] [gql] prompt clone #5799
• [hub] [gql] ChatPromptTemplate type #5771
• [hub] [gql] prompt delete #5798
• [hub] [gql] prompt template version tag create #5763
• [hub] [gql] prompt template version tag patch #5764
• [hub] [gql] filter prompts by tag (use-case)
• [hub] PromptTag list on a PromptVersion #5779
• [hub] [REST] /prompts/:identifier/version/:versionId #5795
• [hub] [REST] /prompts/:id/tag/:tagIdentifier #5796
• [hub] [REST] /prompts
• [hub] [REST] rename promt tags prompt labels, rename crosswalk table prompt_prompt_labels #5788
• [hub] add auto increment to versions #5797
• [hub] prompt_version_tag field on prompt version #5846
• [hub] formalize tool definition #5848

UI

Prompt Details / Management

• [hub] [ui] prompt listing #5725
• [hub] [ui] prompt details #5741
• [hub] [ui] prompt id and version in zustand state #5735
• [hub] [ui] prompt picker / combobox in playground #5736
• [hub] [ui] first time prompt create #5734
• [hub] [ui] render a read-only view of the chat prompt template in the prompt details #5759
• [hub] [ui] prompt details aside preview of the last N versions #5761
• [hub] [ui] full prompt history list in the versions tab #5760
• [hub] [ui] display invocation parameters in the prompt details #5777
• [hub] [ui] display typescript / python code snippets for openai #5776
• [hub] [ui] render tool definitions on prompt details / prompt version details views
• [hub] [ui] ability to tag a version on the prompt history
• [hub] [ui] prompt clone
• [hub] [ui] prompt tags in prompt aside
• [hub] [ui] edit prompt tags (use-cases) in prompt aside

Prompt Editor / Playground

• [hub]
• [hub] [ui] prompt save in playground
• [hub] [ui] load prompt into playground state
• []

Tracing

• [hub] add semantic conventions for prompt_template.id and prompt.vendor

Analytics

• [hub] scope out prompt template analytics

Python SDK

• [hub] prompts phoenix submodule
• [hub] prompt sync pull
• [hub] propagate prompt id