Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PR: OpenSearch integration components for OPEA #908

Closed
Closed
Changes from 1 commit
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
8babbe5
Push changes to synced branch
cameronmorin Nov 15, 2024
69cf79a
Remove unused Redis dependency
cameronmorin Nov 15, 2024
219aac9
Add security options to docker-compose files
cameronmorin Nov 15, 2024
0973cc3
Address PR comments
cameronmorin Nov 18, 2024
44bd2b5
Add env for pass down model id in ChatQnA gateway (#906)
lvliang-intel Nov 16, 2024
67b3efd
Add "--no-verbose" flag to wget download commands in entrypoint (#909)
ctao456 Nov 18, 2024
1f5c7e3
Add empty list check (#914)
ZePan110 Nov 18, 2024
ee735b7
Create test scripts for dataprep and retriever opensearch microservices
cameronmorin Nov 20, 2024
fe7090b
Pre-commit formatting changes
cameronmorin Nov 20, 2024
9cfd7cc
Update docker-compose file
cameronmorin Nov 21, 2024
6b9a010
Finalize dataprep script (working!)
cameronmorin Nov 21, 2024
c9a6a86
Finalize retriever test script (working!)
cameronmorin Nov 21, 2024
897a207
Fix hardware tag retrieval issue (#916)
ZePan110 Nov 19, 2024
ffb4871
fix retriever and reranker to process chat completion request (#915)
minmin-intel Nov 19, 2024
f25d35e
Fix Dataprep Upload Link issue (#913)
letonghan Nov 19, 2024
074e93d
Rename image names XXX-hpu to XXX-gaudi (#911)
ZePan110 Nov 19, 2024
f32b242
Fix CD issues (#917)
ZePan110 Nov 20, 2024
af4efca
fix the image name (#918)
chensuyue Nov 20, 2024
2cf1796
[Bug] Fix VDMS retriever and apply fix to VDMS dataprep (#928)
cwlacewe Nov 21, 2024
43ce07e
bump version into 1.1 (#930)
chensuyue Nov 21, 2024
1e00689
fix issue template bug (#939)
isaacncz Nov 21, 2024
5ea0ca5
Run pre-commit script
cameronmorin Nov 22, 2024
44ef049
Update retriever container image
cameronmorin Nov 27, 2024
2a04ac2
Run pre-commit hook
cameronmorin Nov 27, 2024
b88a85e
Address PR comments
cameronmorin Nov 27, 2024
972b3df
Update test scripts
cameronmorin Nov 29, 2024
0678b8e
Try updating ip address in test script
cameronmorin Nov 29, 2024
b849bb5
Reset ip address
cameronmorin Nov 29, 2024
5356173
Add cluster health check to test script
cameronmorin Nov 29, 2024
dda5b94
Add sysctl for opensearch collection
cameronmorin Nov 29, 2024
b342bd5
Add dockerfile paths to GitHub workflows
cameronmorin Dec 2, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Address PR comments
Signed-off-by: Cameron Morin <[email protected]>
cameronmorin committed Dec 2, 2024
commit 0973cc31e12b628485d713b7673d3ee009d38174
26 changes: 0 additions & 26 deletions comps/dataprep/opensearch/README.md
Original file line number Diff line number Diff line change
@@ -20,12 +20,6 @@ cd langchain
pip install -r requirements.txt
```

<!-- - option 2: Install multi-process version (for >10 files processing)

```bash
cd langchain_ray; pip install -r requirements_ray.txt
``` -->

### 1.2 Start OpenSearch Stack Server

Please refer to this [readme](../../vectorstores/opensearch/README.md).
@@ -74,13 +68,6 @@ cd langchain
python prepare_doc_opensearch.py
```

<!-- - option 2: Start multi-process version (for >10 files processing)

```bash
cd langchain_ray
python prepare_doc_opensearch_on_ray.py
``` -->

## 🚀2. Start Microservice with Docker (Option 2)

### 2.1 Start OpenSearch Stack Server
@@ -108,13 +95,6 @@ cd ../../
docker build -t opea/dataprep-opensearch:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/dataprep/opensearch/langchain/Dockerfile .
```

<!-- - option 2: Start multi-process version (for >10 files processing)

```bash
cd ../../../
docker build -t opea/dataprep-on-ray-opensearch:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/dataprep/opensearch/langchain_ray/Dockerfile .
``` -->

### 2.4 Run Docker with CLI (Option A)

- option 1: Start single-process version (for 1-10 files processing)
@@ -123,12 +103,6 @@ docker build -t opea/dataprep-on-ray-opensearch:latest --build-arg https_proxy=$
docker run -d --name="dataprep-opensearch-server" -p 6007:6007 --runtime=runc --ipc=host -e http_proxy=$http_proxy -e https_proxy=$https_proxy -e OPENSEARCH_URL=$OPENSEARCH_URL -e INDEX_NAME=$INDEX_NAME -e TEI_ENDPOINT=$TEI_ENDPOINT -e HUGGINGFACEHUB_API_TOKEN=$HUGGINGFACEHUB_API_TOKEN opea/dataprep-opensearch:latest
```

<!-- - option 2: Start multi-process version (for >10 files processing)

```bash
docker run -d --name="dataprep-opensearch-server" -p 6007:6007 --runtime=runc --ipc=host -e http_proxy=$http_proxy -e https_proxy=$https_proxy -e OPENSEARCH_URL=$OPENSEARCH_URL -e INDEX_NAME=$INDEX_NAME -e TEI_ENDPOINT=$TEI_ENDPOINT -e HUGGINGFACEHUB_API_TOKEN=$HUGGINGFACEHUB_API_TOKEN -e TIMEOUT_SECONDS=600 opea/dataprep-on-ray-opensearch:latest
``` -->

### 2.5 Run with Docker Compose (Option B - deprecated, will move to genAIExample in future)

```bash
2 changes: 1 addition & 1 deletion comps/dataprep/opensearch/langchain/config.py
Original file line number Diff line number Diff line change
@@ -10,7 +10,7 @@
# OpenSearch Connection Information
OPENSEARCH_HOST = os.getenv("OPENSEARCH_HOST", "localhost")
OPENSEARCH_PORT = int(os.getenv("OPENSEARCH_PORT", 9200))
OPENSEARCH_INITIAL_ADMIN_PASSWORD = os.getenv("OPENSEARCH_INITIAL_ADMIN_PASSWORD", "strongOpea0!")
OPENSEARCH_INITIAL_ADMIN_PASSWORD = os.getenv("OPENSEARCH_INITIAL_ADMIN_PASSWORD", "StRoNgOpEa0)")


def get_boolean_env_var(var_name, default_value=False):

This file was deleted.

Original file line number Diff line number Diff line change
@@ -28,6 +28,8 @@ services:
TEI_EMBEDDING_ENDPOINT: ${TEI_EMBEDDING_ENDPOINT}
HUGGINGFACEHUB_API_TOKEN: ${HUGGINGFACEHUB_API_TOKEN}
restart: unless-stopped
security_opt:
- no-new-privileges:true

networks:
default:
121 changes: 0 additions & 121 deletions comps/retrievers/opensearch/langchain/ingest.py

This file was deleted.

Original file line number Diff line number Diff line change
@@ -46,7 +46,7 @@ def get_boolean_env_var(var_name, default_value=False):
# OpenSearch Connection Information
OPENSEARCH_HOST = os.getenv("OPENSEARCH_HOST", "localhost")
OPENSEARCH_PORT = int(os.getenv("OPENSEARCH_PORT", 9200))
OPENSEARCH_INITIAL_ADMIN_PASSWORD = os.getenv("OPENSEARCH_INITIAL_ADMIN_PASSWORD", "strongOpea0!")
OPENSEARCH_INITIAL_ADMIN_PASSWORD = os.getenv("OPENSEARCH_INITIAL_ADMIN_PASSWORD", "StRoNgOpEa0)")


def format_opensearch_conn_from_env():
6 changes: 6 additions & 0 deletions comps/vectorstores/opensearch/docker-compose-opensearch.yml
Original file line number Diff line number Diff line change
@@ -28,6 +28,8 @@ services:
- 9600:9600 # required for Performance Analyzer
networks:
- opensearch-net
security_opt:
- no-new-privileges:true
opensearch-node2:
image: opensearchproject/opensearch:latest
container_name: opensearch-node2
@@ -50,6 +52,8 @@ services:
- ./opensearch-data2:/var/lib/opensearch/data
networks:
- opensearch-net
security_opt:
- no-new-privileges:true
opensearch-dashboards:
image: opensearchproject/opensearch-dashboards:latest
volumes:
@@ -63,6 +67,8 @@ services:
OPENSEARCH_HOSTS: '["https://opensearch-node1:9200","https://opensearch-node2:9200"]' # must be a string with no spaces when specified as an environment variable
networks:
- opensearch-net
security_opt:
- no-new-privileges:true
depends_on:
- opensearch-node1
- opensearch-node2
2 changes: 1 addition & 1 deletion comps/vectorstores/opensearch/opensearch_dashboards.yml
Original file line number Diff line number Diff line change
@@ -196,7 +196,7 @@
# Set the value of this setting to true to enable the experimental ml commons dashboards
ml_commons_dashboards.enabled: true

opensearch.hosts: [https://localhost:9200]
opensearch.hosts: ["https://localhost:9200"]
opensearch.ssl.verificationMode: none
opensearch.username: kibanaserver
opensearch.password: kibanaserver