Skip to content

Commit a4c2650

Browse files
dokterbobdokterbob
authored
Merge branch 'main' into postgres
2 parents 3e286d4 + 1d6ad53 commit a4c2650

File tree

16 files changed

+947
-29
lines changed

16 files changed

+947
-29
lines changed

‎README.md‎

Lines changed: 37 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -62,17 +62,43 @@ To maintain high-quality code and reduce technical debt, we enforce the followin
6262
- Testing against both the latest Chainlit release and the main branch
6363
- Documentation for all added or changed functionality
6464

65-
## 🔧 Features and Integrations
66-
67-
This repository hosts a variety of community-maintained features and integrations, including but not limited to:
68-
69-
- Additional LLM framework integrations
70-
- Custom authentication implementations
71-
- Specialized UI components
72-
- Utility functions and helpers
73-
- Domain-specific extensions
74-
75-
For a full list of available features and integrations, please check our [Features Directory](FEATURES.md).
65+
## Component Architecture
66+
67+
Chainlit Community components follow a modular architecture with two main component types:
68+
69+
### 1. Data Layers
70+
**Role**: Persistent structured data storage for conversation elements (users, threads, steps)
71+
**Interactions**:
72+
- Direct integration with Chainlit's data layer system
73+
- Optional integration with Storage Providers for file attachments
74+
75+
| Package | Description | README |
76+
|---------|-------------|--------|
77+
| `dynamodb` | Amazon DynamoDB implementation with cloud storage integration | [docs](packages/data_layers/dynamodb/README.md) |
78+
| `sqlalchemy` | SQL database support (PostgreSQL/SQLite) with storage provider integration | [docs](packages/data_layers/sqlalchemy/README.md) |
79+
| `literalai` | Official Literal AI observability platform integration | [docs](packages/data_layers/literalai/README.md) |
80+
81+
### 2. Storage Providers
82+
**Role**: File storage and management for attachments/media
83+
**Interactions**:
84+
- Used by Data Layers through dependency injection
85+
- Handle upload/delete operations and URL generation
86+
87+
| Package | Cloud Provider | README |
88+
|---------|----------------|--------|
89+
| `azure` | Azure Data Lake | [docs](packages/storage_clients/azure/README.md) |
90+
| `azure-blob` | Azure Blob Storage | [docs](packages/storage_clients/azure_blob/README.md) |
91+
| `gcs` | Google Cloud Storage | [docs](packages/storage_clients/gcs/README.md) |
92+
| `s3` | AWS S3 | [docs](packages/storage_clients/s3/README.md) |
93+
94+
## Typical Data Flow
95+
```mermaid
96+
graph LR
97+
A[Chainlit App] --> B{Data Layer}
98+
B -->|Persists metadata| C[(Database)]
99+
B -->|Delegates files| D[[Storage Provider]]
100+
D -->|Stores objects| E[(Cloud Storage)]
101+
```
76102

77103
## 📚 Documentation
78104

‎packages/data_layers/dynamodb/README.md‎

Lines changed: 140 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,4 +2,143 @@
22

33
DynamoDB data layer for [Chainlit](https://chainlit.io/).
44

5-
## Usage
5+
## DynamoDB Data Layer
6+
7+
This data layer supports Amazon DynamoDB with optional cloud storage integration for elements.
8+
9+
Key features:
10+
- Single table design with efficient query patterns
11+
- Supports storage clients for attachments (S3, Azure Blob)
12+
- User/thread/step/element/feedback storage in DynamoDB
13+
- Built-in pagination and sorting
14+
15+
## Setup Example (DynamoDB + Cloud Storage)
16+
17+
1. Create DynamoDB table using the [CloudFormation template](#table-structure)
18+
2. Install required dependencies:
19+
```bash
20+
# Core requirements
21+
pip install chainlit-dynamodb
22+
23+
# With cloud storage (choose one):
24+
pip install chainlit-dynamodb[s3] # AWS S3
25+
pip install chainlit-dynamodb[azure-blob] # Azure Blob Storage
26+
pip install chainlit-dynamodb[gcs] # Google Cloud Storage
27+
pip install chainlit-dynamodb[azure] # Azure Data Lake
28+
```
29+
30+
3. Configure in your Chainlit app:
31+
```python
32+
import os
33+
import chainlit as cl
34+
from chainlit.data.dynamodb import DynamoDBDataLayer
35+
from chainlit.data.storage_clients import (
36+
S3StorageClient,
37+
AzureBlobStorageClient,
38+
GCSStorageClient,
39+
AzureStorageClient
40+
)
41+
42+
# Security Note: Always store secrets in environment variables
43+
# Never commit credentials to source control
44+
# Consider using secret managers like AWS Secrets Manager
45+
46+
@cl.data_layer
47+
def get_data_layer():
48+
# Choose one storage provider:
49+
50+
# AWS S3 Example
51+
storage_client = S3StorageClient(
52+
bucket="<your-bucket>",
53+
region_name="<your-region>",
54+
aws_access_key_id=os.environ["AWS_ACCESS_KEY_ID"],
55+
aws_secret_access_key=os.environ["AWS_SECRET_ACCESS_KEY"]
56+
)
57+
58+
# Azure Blob Example
59+
# storage_client = AzureBlobStorageClient(
60+
# container_name="<your-container>",
61+
# storage_account="<your-account>",
62+
# storage_key="<your-key>"
63+
# )
64+
65+
# Google Cloud Storage Example
66+
# storage_client = GCSStorageClient(
67+
# project_id="<your-project>",
68+
# client_email="<your-email>",
69+
# private_key="<your-key>",
70+
# bucket_name="<your-bucket>"
71+
# )
72+
73+
# Azure Data Lake Example
74+
# storage_client = AzureStorageClient(
75+
# account_url="https://<account>.dfs.core.windows.net",
76+
# credential="<your-credential>",
77+
# container_name="<your-container>"
78+
# )
79+
80+
return DynamoDBDataLayer(
81+
table_name="<your-table-name>",
82+
storage_provider=storage_client,
83+
user_thread_limit=10
84+
)
85+
```
86+
87+
## Table Structure
88+
```yaml
89+
# CloudFormation template for required table structure
90+
AWSTemplateFormatVersion: 2010-09-09
91+
Resources:
92+
DynamoDBTable:
93+
Type: AWS::DynamoDB::Table
94+
Properties:
95+
TableName: "<YOUR-TABLE-NAME>"
96+
AttributeDefinitions:
97+
- AttributeName: PK
98+
AttributeType: S
99+
- AttributeName: SK
100+
AttributeType: S
101+
- AttributeName: UserThreadPK
102+
AttributeType: S
103+
- AttributeName: UserThreadSK
104+
AttributeType: S
105+
KeySchema:
106+
- AttributeName: PK
107+
KeyType: HASH
108+
- AttributeName: SK
109+
KeyType: RANGE
110+
GlobalSecondaryIndexes:
111+
- IndexName: UserThread
112+
KeySchema:
113+
- AttributeName: UserThreadPK
114+
KeyType: HASH
115+
- AttributeName: UserThreadSK
116+
KeyType: RANGE
117+
Projection:
118+
ProjectionType: INCLUDE
119+
NonKeyAttributes: [id, name]
120+
BillingMode: PAY_PER_REQUEST
121+
```
122+
123+
## Logging
124+
```python
125+
import logging
126+
from chainlit import logger
127+
128+
# Enable debug logging for DynamoDB operations
129+
logger.getChild("DynamoDB").setLevel(logging.DEBUG)
130+
```
131+
132+
## Limitations
133+
- Feedback filtering not supported
134+
- Boto3-based implementation uses blocking IO (not async)
135+
- Decimal types in feedback values require special handling
136+
137+
## Design
138+
Uses single-table design with entity prefixes:
139+
- Users: `USER#{identifier}`
140+
- Threads: `THREAD#{thread_id}`
141+
- Steps: `STEP#{step_id}`
142+
- Elements: `ELEMENT#{element_id}`
143+
144+
Global Secondary Index (UserThread) enables efficient user thread queries.

‎packages/data_layers/dynamodb/pyproject.toml‎

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,20 +1,38 @@
11
[project]
22
name = "chainlit-dynamodb"
33
version = "0.1.0"
4-
description = "Add your description here"
4+
description = "DynamoDB data layer for Chainlit"
55
readme = "README.md"
66
requires-python = ">=3.10"
77
dependencies = [
88
"aiohttp>=3.11.10",
99
"boto3>=1.34.73,<2",
1010
]
1111

12+
[project.optional-dependencies]
13+
s3 = [
14+
"chainlit-s3",
15+
]
16+
azure-blob = [
17+
"chainlit-azure-blob",
18+
]
19+
gcs = [
20+
"chainlit-gcs",
21+
]
22+
azure = [
23+
"chainlit-azure",
24+
]
25+
1226
[dependency-groups]
1327
dev = [
1428
"pytest-chainlit",
1529
]
1630

1731
[tool.uv.sources]
32+
chainlit-s3 = { workspace = true }
33+
chainlit-azure-blob = { workspace = true }
34+
chainlit-gcs = { workspace = true }
35+
chainlit-azure = { workspace = true }
1836
pytest-chainlit = { workspace = true }
1937

2038
[build-system]

‎packages/data_layers/literalai/README.md‎

Lines changed: 47 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,50 @@
11
# chainlit-literalai
22

3-
[Literal AI](https://www.literalai.com/) data layer for [Chainlit](https://chainlit.io/).
3+
[Literal AI](https://www.literal.ai/) integration for [Chainlit](https://chainlit.io/) applications.
44

5-
## Usage
5+
## Overview
6+
7+
Official data persistence layer connecting Chainlit with Literal AI's LLM observability platform. Enables production-grade monitoring, evaluation and analytics while maintaining Chainlit's conversation structure.
8+
9+
**Key Features**:
10+
- Full conversation history preservation (threads, steps, elements)
11+
- Multimodal logging (text, images, audio, video)
12+
- User feedback tracking
13+
- Automated performance metrics
14+
- Collaborative prompt versioning & A/B testing
15+
16+
## Setup
17+
18+
1. **Install package**:
19+
```bash
20+
pip install chainlit-literalai
21+
```
22+
23+
2. **Configure environment**:
24+
```bash
25+
# .env file
26+
# Security Best Practices:
27+
# - Restrict .env file permissions
28+
# - Never commit .env to version control
29+
# - Use CI/CD secret management
30+
LITERAL_API_KEY="your-api-key-from-literal-ai"
31+
```
32+
33+
3. **Run your app**:
34+
```bash
35+
chainlit run app.py
36+
```
37+
38+
## Documentation
39+
40+
- [Literal AI Documentation](https://docs.literalai.com)
41+
- [Chainlit + Literal AI Integration Guide](https://docs.chainlit.io/llmops/literalai)
42+
43+
## Data Privacy
44+
45+
- Data retention policy: [literalai.com/security](https://www.literalai.com/security)
46+
- Contact: <[email protected]>
47+
48+
> **Note**
49+
> Developed by the Chainlit team for seamless integration.
50+
> Literal AI is SOC 2 Type 2 compliant.

‎packages/data_layers/sqlalchemy/README.md‎

Lines changed: 63 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,4 +2,66 @@
22

33
SQLAlchemy data layer for [Chainlit](https://chainlit.io/).
44

5-
## Usage
5+
## SQLAlchemy Data Layer
6+
7+
This data layer supports PostgreSQL and other SQL databases via SQLAlchemy.
8+
9+
Key features:
10+
11+
- Supports storage clients for attachments (currently: Azure, Azure Blobs, GCS, S3)
12+
- User/thread/step/element/feedback storage in SQL
13+
- Async operations
14+
15+
## Setup Example (PostgreSQL + Azure Blob)
16+
17+
1. Load [schema.sql](schema.sql) into your database.
18+
1. Install required dependencies:
19+
20+
```bash
21+
# For PostgreSQL
22+
pip install chainlit-sqlalchemy[postgres]
23+
24+
# For SQLite
25+
pip install chainlit-sqlalchemy[sqlite]
26+
27+
# With cloud storage
28+
pip install chainlit-sqlalchemy[postgres,gcs] # PostgreSQL + Google Cloud Storage
29+
pip install chainlit-sqlalchemy[sqlite,azure-blob] # SQLite + Azure Blob
30+
```
31+
32+
2. Configure in your Chainlit app:
33+
34+
```python
35+
import chainlit as cl
36+
from chainlit.data.sql_alchemy import SQLAlchemyDataLayer
37+
from chainlit.data.storage_clients import AzureBlobStorageClient
38+
39+
@cl.data_layer
40+
def get_data_layer():
41+
storage_client = AzureBlobStorageClient(
42+
container_name="<your_container>",
43+
storage_account="<your_account>",
44+
storage_key="<your_key>"
45+
)
46+
47+
return SQLAlchemyDataLayer(
48+
conninfo="postgresql+asyncpg://user:password@host/dbname",
49+
storage_provider=storage_client
50+
)
51+
```
52+
53+
> [!NOTE]
54+
> - Add `+asyncpg` to PostgreSQL connection strings for async support
55+
> - See SQLAlchemy docs for other database connection formats
56+
57+
## Dependencies
58+
59+
- Core: `SQLAlchemy`
60+
- Database Drivers (optional):
61+
- PostgreSQL: `asyncpg`, `psycopg2-binary`
62+
- SQLite: `aiosqlite`
63+
- Cloud Storage (optional):
64+
- Azure (Data Lake): `chainlit-azure`
65+
- Azure Blob: `chainlit-azure-blob`
66+
- Google Cloud: `chainlit-gcs`
67+
- AWS S3: `chainlit-s3`

0 commit comments

Comments
 (0)