rm e2e tests from CICD

.github/workflows/cicd.yml

@@ -91,41 +91,3 @@ jobs:
           ignore-unfixed: true
           vuln-type: library
           severity: CRITICAL,HIGH
-
-  end-to-end-tests:
-    needs: [build, image-vulnerabilities]
-    runs-on: ubuntu-latest
-    env:
-      IMAGE_TAG: ${{ needs.build.outputs.IMAGE_TAG }}
-    steps:
-      - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@v1
-
-      - name: Download artifact
-        uses: actions/download-artifact@v2
-        with:
-          name: openai-rag-chatbot-app
-          path: /tmp
-
-      - name: Load Docker image
-        run: |
-          docker load --input /tmp/openai-rag-chatbot-app.tar
-          docker image ls -a
-
-      - name: checkout repo
-        uses: actions/checkout@v2
-
-      - name: Install docker-compose if running pipeline locally
-        run: |
-          if ! command -v docker-compose &> /dev/null; then
-            echo Installing docker-compose
-
-            curl -L "https://github.com/docker/compose/releases/download/2.27.0/docker-compose-$(uname -s)-$(uname -m)" \
-            -o /usr/local/bin/docker-compose
-
-            chmod +x /usr/local/bin/docker-compose
-          fi
-
-      # - name: Run integration tests
-      #   run: |
-      #     cd app && make IMAGE_TAG=${IMAGE_TAG} integration-tests

README.md

@@ -1,16 +1,17 @@
 # lasik-openai-rag · ![ci](https://github.com/teddy-ambona/openai-rag-chatbot/actions/workflows/cicd.yml/badge.svg)
 
 
-Demo LLM (RAG pipeline) web app running locally using docker-compose. LLM and embedding models are consumed from OpenAI.
+Demo LLM (RAG pipeline) web app running locally using docker-compose. LLM and embedding models are consumed as services from OpenAI.
 
-## 1 - Target setup
+The primary objective is to enable users to ask questions related to LASIK surgery, such as *"Is there a contraindication for computer programmers to get LASIK?"*
 
-A Retrieval Augmented Generation (RAG) pipeline is a NLP framework that combines information retrieval with text generation to produce responses or generate text. It uses a retriever to find relevant information and a generator to create responses based on that information.
+The Retrieval Augmented Generation (RAG) pipeline retrieves the most up-to-date information from the dataset to provide accurate and relevant responses to user queries.
 
+## Target setup
 
 The app architecture is presented below:
 
-<img src="./docs/diagrams/rag-architecture.png" width="850"/>
+<img src="./docs/diagrams/rag-architecture.png" width="750"/>
 
 Sequence diagram:
 
@@ -29,15 +30,15 @@ sequenceDiagram
 
 UX:
 
-<img src="./docs/img/ui.png" width="850"/>
+<img src="./docs/img/ui.png" width="750"/>
 
 
-## 2 - Prerequisites
+## Prerequisites
 
 - [Docker](https://docs.docker.com/get-docker/)
 - An [OpenAI key](https://openai.com/)(account should be provisioned with $5, which is the minimum amount allowed)
 
-## 3 - Quickstart
+## Quickstart
 
 Build app Docker image:
 
@@ -75,34 +76,87 @@ make app-run
 
 The chatbot is now available at [http://localhost:8000/lasik_complications/playground/](http://localhost:8000/lasik_complications/playground/)
 
-<insert image>
-
-
 Display all available commands with:
 
 ```bash
 make help
 ```
 
-<img src="./docs/img/make_help.png" width="850"/>
+<img src="./docs/img/make_help.png" width="450"/>
 
 Clean up
 
 ```bash
 make clean
 ```
 
-## 4 - Project file structure
+## Project file structure
+
+```text
+├── .github
+│   ├── workflow
+│   │   └── cicd.yml       <- CI pipeline definition
+├── data
+│   └── laser_eye_surgery_complications.csv       <- Kaggle dataset
+|
+├── docs
+│   ├── diagrams      <- Folder containing diagram definitions
+│   └── img           <-  Folder containing screenshots
+│
+├── src
+│   ├── config.py                  <- Config file with service host/ports or models to be used
+│   ├── populate_vector_db.py      <- Scripts that converts texts to embeddings and populates Milvus DB
+│   └── server.py                  <- FastAPI/Langserve/Langchain
+│
+├── .gitignore
+├── .pre-commit-config.yaml        <- ruff linter pre-commit hook
+├── docker-compose.yml             <- container orchestration
+├── Dockerfile                     <- App image definition
+├── Makefile                       <- Makefile with commands like `make app-build`
+├── poetry.lock                    <- Pinned dependencies
+├── pyproject.toml                 <- Dependencies requirements
+├── README.md                      <- The top-level README for developers using this project.
+└── ruff.toml                      <- Linter config
+```
+
+## The dataset
+
+Sourced from [Lasik (Laser Eye Surgery) Complications](https://www.kaggle.com/datasets/shivamb/lasik-complications-dataset/data)(Kaggle)
+
+<img src="./docs/img/kaggle_dataset.png" width="450"/>
 
 ## Milvus
 
 [Milvus](https://github.com/milvus-io/milvus) is an open-source vector database engine developed by Zilliz, designed to store and manage large-scale vector data, such as embeddings, features, and high-dimensional data. It provides efficient storage, indexing, and retrieval capabilities for **vector similarity search tasks**.
 
-## LLMOps
+## CICD
+
+
+
+## Langchain
+
+Langchain is a LLM orchestration tool, it is very useful when you need to build context-aware LLM apps.
+
+## Prompt Engineering
+
+In order to provide the context to the LLM, we have to wrap the original question in a [prompt template](./src/server.py#L39)
+
+<img src="./docs/img/prompt_engineering.png" width="850"/>
+
+You can check what prompt the LLM actually received by clicking on "intermediate steps" in the UX
+
+<img src="./docs/img/intermediate_steps.png" width="850"/>
+
+## Langserve
+
+LangServe helps developers deploy LangChain runnables and chains as a REST API. This library is integrated with FastAPI.
+
+## To do
 
-## 5 - Langchain
+- The chatbot cannot answer questions related to stats, for example *"Are there any recent trends in LASIK surgery complications?"*, there should be another model that infers the relevant time-window to consider for retrieving the documents and then enrich the final prompt with this time-window.
 
-## 6 - Prompt Engineering
+- [Algorithmic feedback with Langsmith](https://github.com/langchain-ai/langsmith-cookbook/blob/main/feedback-examples/algorithmic-feedback/algorithmic_feedback.ipynb). This would allow to test the robustness of the LLM chain in an automated way.
 
-## 7 - Langserve
+## Useful resources
 
+- [OpenAI Prompt engineering](https://platform.openai.com/docs/guides/prompt-engineering)