Merge pull request #20 from SkywardAI/dev

v0.1 milestone on the way

README.md

@@ -2,11 +2,11 @@
 
 <div align=center>
  <a href="https://github.com/SkywardAI/chat-backend/actions/workflows/ci-backend.yaml">
-  <img src="https://github.com/SkywardAI/chat-backend/actions/workflows/ci-backend.yaml"/> 
+  <img src="https://github.com/SkywardAI/chat-backend/actions/workflows/ci-backend.yaml"/>
  </a>
 
  <a href="https://codecov.io/gh/SkywardAI/chat-backend">
-  <img src="https://codecov.io/gh/SkywardAI/chat-backend/branch/trunk/graph/badge.svg?token=1hiVayuLRl"/> 
+  <img src="https://codecov.io/gh/SkywardAI/chat-backend/branch/trunk/graph/badge.svg?token=1hiVayuLRl"/>
  </a>
 
  <a href="https://github.com/pre-commit/pre-commit">
@@ -60,7 +60,7 @@ The above-listed technologies are just the main ones. There are other technologi
 * [Pyenv-VirtualEnv](https://github.com/pyenv/pyenv-virtualenv) $\rightarrow$ The plugin for `Pyenv` to manage the virtual environment for our packages.
 * [Pre-Commit](https://pre-commit.com/) $\rightarrow$ Git hook scripts to identify issues and quality of your code before pushing it to GitHub. These hooks are implemented for the following linting packages:
   * [Black (Python)](https://black.readthedocs.io/en/stable/) $\rightarrow$ Manage your code style with auto-formatting and parallel continuous integration runner for Python.
-  * [Isort (Python)](https://pycqa.github.io/isort/) $\rightarrow$ Sort your `import` for clarity. Also for Python. 
+  * [Isort (Python)](https://pycqa.github.io/isort/) $\rightarrow$ Sort your `import` for clarity. Also for Python.
   * [MyPy (Python)](https://mypy.readthedocs.io/en/stable/) $\rightarrow$ A static type checker for Python that helps you to write cleaner code.
 * [Pre-Commit CI](https://pre-commit.ci/) $\rightarrow$ Continuous integration for our Pre-Commit hook that fixes and updates our hook versions.
 * [CodeCov](https://about.codecov.io/) $\rightarrow$ A platform that analyzes the result of your automated tests.
@@ -75,6 +75,7 @@ My choice for a project development workflow is usually the [Trunk-Based Develop
 ## What Code is included?
 
 For the backend application:
+
 * The project, linter, and test configurations in `backend/pyproject.toml`.
 * 3 settings classes (development, staging, production) with the super class in `backend/src/config/settings/base.py`.
 * Event logger in `backend/src/config/events.py`..
@@ -87,10 +88,12 @@ For the backend application:
 * A comprehensive FastAPI application initialization in `backend/src/main.py`.
 
 For testing, I have prepared the following simple code to kick-start your test-driven development:
+
 * A simple replication of the backend application for testing purposes and the asynchronous test client in `backend/tests/conftest.py`.
 * 2 simple test functions to test the backend application initialization in `tests/unit_tests/test_src.py`.
 
 For the DevOps:
+
 * A simple `build` job to test the compilation of the source code for the backend application in `.github/workflows/ci-backend.yaml`.
 * A simple linting job called `code-style` with black, isort, flake8, and mypy in `.github/workflows/ci-backend.yaml`.
 * An automated testing with `PyTest` and an automated test reporting with `Codecov` in in `.github/workflows/ci-backend.yaml`.
@@ -100,19 +103,21 @@ For the DevOps:
 * A CI for automatically updating all linter version in the pre-commit `YAML` file in `.pre-commit-config.YAML`.
 
 For containerization:
+
 * A `Docker` configuration that utilizes the latest Python image in `backend/Dockerfile`.
 * A script that ensure the backend application will restart when postgres image hasn't started yet in `backend/entrypoint.sh`.
 * Setting up `Postgres` image for our database server, `Adminer` for our database editor, and `backend_app` for our backend application's container in `docker-compose.yaml`.
 
 For the team development environment:
+
 * A pre-commit hooks for `Black`, `Isort`, and `MyPy` to ensure the conventional commit message before pushing an updated code into the remote repository in `.pre-commit-config.YAML`.
 * All secret variables are listed in `.env.example`. You need to copy these variables and set the values respectively to your need and save them in a new `.env` in the root directory.
 
 ## Setup Guide
 
 This backend application is setup with `Docker`. Nevertheless, you can see the full local setup without `Docker` in [backend/README.md](https://github.com/SkywardAI/chat-backend/blob/trunk/backend/README.md).
 
-### For quick setup: 
+### For quick setup
 
 ```shell
 cp .env.example .env
@@ -125,11 +130,13 @@ docker-compose up
 ## Regular Setup
 
 1. Before setting up the backend app, please create a new directory called `coverage` for the testing report purpose:
+
    ```shell
    cd backend && mkdir coverage
    ```
 
 2. Backend app setup:
+
     ```shell
     # Creating VENV
     pyenv virtualenv 3.11.0 any_venv_name
@@ -144,6 +151,7 @@ docker-compose up
 
 3. Testing with `PyTest`:
    Make sure that you are in the `backend/` directory.
+
    ```shell
    # For testing without Docker
    pytest
@@ -153,6 +161,7 @@ docker-compose up
    ```
 
 4. `Pre-Commit` setup:
+
     ```shell
     # Make sure you are in the ROOT project directory
     pre-commit install
@@ -173,18 +182,18 @@ docker-compose up
 
     echo "SECRET_VARIABLE=SECRET_VARIABLE_VALUE" >> .env
     ```
-    
+
     For test usage , you can simplely use .env.example
+
     ```shell
     cp .env.example .env
     ```
 
-
-
 6. `CODEOWNERS` setup:
     Go to `.github/` and open `CODEOWNERS` file. This file is to assign the code to a specific team member so you can distribute the weights of the project clearly.
 
 7. Docker setup:
+
    ```shell
     # Make sure you are in the ROOT project directory
     chmod +x backend/entrypoint.sh
@@ -197,6 +206,7 @@ docker-compose up
    ```
 
 8. (IMPORTANT) Database setup:
+
    ```shell
     # (Docker) Generate revision for the database auto-migrations
     docker exec backend_app alembic revision --autogenerate -m "YOUR MIGRATION TITLE"
@@ -207,7 +217,7 @@ docker-compose up
     alembic upgrade head    # to register the database classes
    ```
 
-9. Go to https://about.codecov.io/, and sign up with your github to get the `CODECOV_TOKEN`
+9. Go to <https://about.codecov.io/>, and sign up with your github to get the `CODECOV_TOKEN`
 
 10. Go to your GitHub and register all the secret variables (look in .env.example) in your repository (`settings` $\rightarrow$ (scroll down a bit) `Secrets` $\rightarrow$ `Actions` $\rightarrow$ `New repository secret`)
 
@@ -325,9 +335,10 @@ docker-compose.yaml                     # The main configuration file for settin
 ## Final Step
 
 You can delete these 3 files (or change its content based on your need):
-- `LICENSE.md`
-- `README.md`
-- `backend/README.md`
+
+* `LICENSE.md`
+* `README.md`
+* `backend/README.md`
 
 Enjoy your development and may your technology be forever useful to everyone 😉🚀🧬
 

backend/requirements.txt

@@ -7,6 +7,7 @@ asyncpg==0.29.0
 bcrypt==4.0.1
 black==24.3.0
 colorama==0.4.6
+datasets==2.15.0
 email_validator==2.1.1
 fastapi==0.110.0
 greenlet==3.0.3
@@ -27,6 +28,7 @@ python-dotenv==1.0.1
 python-jose==3.3.0
 python-multipart==0.0.9
 python-slugify==8.0.4
+sentence-transformers==2.3.1
 SQLAlchemy==2.0.0b3
 trio==0.24.0
 uvicorn==0.28.0

backend/src/api/dependencies/repository.py

@@ -8,6 +8,7 @@
 
 from src.api.dependencies.session import get_async_session
 from src.repository.crud.base import BaseCRUDRepository
+from src.repository.rag.base import BaseRAGRepository
 
 
 def get_repository(
@@ -19,3 +20,14 @@ def _get_repo(
         return repo_type(async_session=async_session)
 
     return _get_repo
+
+
+def get_rag_repository(
+    repo_type: typing.Type[BaseRAGRepository],
+) -> typing.Callable[[SQLAlchemyAsyncSession], BaseRAGRepository]:
+    def _get_repo(
+        async_session: SQLAlchemyAsyncSession = fastapi.Depends(get_async_session),
+    ) -> BaseRAGRepository:
+        return repo_type(async_session=async_session)
+
+    return _get_repo

backend/src/api/dependencies/session.py

@@ -17,5 +17,6 @@ async def get_async_session() -> typing.AsyncGenerator[SQLAlchemyAsyncSession, N
     except Exception as e:
         print(e)
         await async_db.async_session.rollback()
+        raise
     finally:
         await async_db.async_session.close()

backend/src/api/routes/ai_model.py

@@ -1,7 +1,7 @@
 import fastapi
 
 from src.api.dependencies.repository import get_repository
-from src.models.schemas.ai_model import AiModel, AiModelInResponse
+from src.models.schemas.ai_model import AiModel, AiModelChooseResponse, AiModelInResponse
 from src.repository.crud.ai_model import AiModelCRUDRepository
 
 router = fastapi.APIRouter(prefix="/models", tags=["model"])
@@ -22,11 +22,26 @@ async def get_aimodels(
     for ai_model in ai_models:
         aimodel = AiModelInResponse(
             id=ai_model.id,
-            available_models=AiModel(
-                name=ai_model.name,
-                des=ai_model.des,
-            ),
+            name=ai_model.name,
+            des=ai_model.des,
         )
         ai_model_list.append(aimodel)
 
     return ai_model_list
+
+
+@router.post(
+    path="/{id}",
+    name="models:choose-model",
+    response_model=AiModelChooseResponse,
+    status_code=fastapi.status.HTTP_200_OK,
+)
+async def choose_aimodels(
+    id: int,
+    aimodel_repo: AiModelCRUDRepository = fastapi.Depends(get_repository(repo_type=AiModelCRUDRepository)),
+) -> AiModelChooseResponse:
+    ai_model = await aimodel_repo.read_aimodel_by_id(id=id)
+    return AiModelChooseResponse(
+        name=ai_model.name,
+        msg="Model has been selected",
+    )

backend/src/api/routes/chat.py

@@ -1,8 +1,9 @@
 import fastapi
 
-from src.api.dependencies.repository import get_repository
+from src.api.dependencies.repository import get_rag_repository, get_repository
 from src.models.schemas.chat import ChatHistory, ChatInMessage, ChatInResponse, Session
 from src.repository.crud.chat import ChatHistoryCRUDRepository, SessionCRUDRepository
+from src.repository.rag.chat import RAGChatModelRepository
 from src.securities.authorizations.jwt import jwt_generator
 from src.utilities.exceptions.database import EntityDoesNotExist
 
@@ -19,6 +20,7 @@ async def chat(
     chat_in_msg: ChatInMessage,
     session_repo: SessionCRUDRepository = fastapi.Depends(get_repository(repo_type=SessionCRUDRepository)),
     chat_repo: ChatHistoryCRUDRepository = fastapi.Depends(get_repository(repo_type=ChatHistoryCRUDRepository)),
+    rag_chat_repo: RAGChatModelRepository = fastapi.Depends(get_rag_repository(repo_type=RAGChatModelRepository)),
 ) -> ChatInResponse:
     # if not chat_in_msg.accountID:
     #     chat_in_msg.accountID = 0
@@ -32,8 +34,7 @@ async def chat(
         # create_session = await session_repo.read_create_sessions_by_id(id=chat_in_msg.sessionId, account_id=chat_in_msg.accountID, name=chat_in_msg.message[:40])
         session_id = chat_in_msg.sessionId
     await chat_repo.create_chat_history(session_id=session_id, is_bot_msg=False, message=chat_in_msg.message)
-    # TODO use RAG framework to generate the response message
-    response_msg = "Oh, really? It's amazing !"
+    response_msg = await rag_chat_repo.get_response(session_id=session_id, input_msg=chat_in_msg.message)
     await chat_repo.create_chat_history(session_id=session_id, is_bot_msg=True, message=response_msg)
     return ChatInResponse(
         sessionId=session_id,
@@ -101,7 +102,6 @@ async def get_chathistory(
 ) -> list[ChatHistory]:
     chats = await chat_repo.read_chat_history_by_session_id(id=id)
     chats_list: list = list()
-    print("2222222222")
     for chat in chats:
         res_session = ChatHistory(
             id=chat.id,

backend/src/api/routes/file.py

@@ -2,13 +2,18 @@
 import random
 
 import fastapi
-
+from src.config.settings.const import UPLOAD_FILE_PATH
 from src.api.dependencies.repository import get_repository
 from src.models.schemas.file import FileInResponse, FileStatusInResponse
 from src.repository.crud.file import UploadedFileCRUDRepository
+from fastapi import BackgroundTasks
 
 router = fastapi.APIRouter(prefix="/file", tags=["file"])
 
+async def save_upload_file(file: fastapi.UploadFile, save_file: str):
+    with open(save_file, "wb") as f:
+        contents = await file.read()
+        f.write(contents)
 
 @router.post(
     "",
@@ -17,25 +22,22 @@
     status_code=fastapi.status.HTTP_201_CREATED,
 )
 async def upload_and_return_id(
+    background_tasks: BackgroundTasks,
     file: fastapi.UploadFile = fastapi.File(...),
     file_repo: UploadedFileCRUDRepository = fastapi.Depends(get_repository(repo_type=UploadedFileCRUDRepository)),
 ):
 
     new_file = await file_repo.create_uploadfile(file_name=file.filename)
-    # TODO save_path to a global constant
-    save_path = "./uploaded_files/"
+    save_path = UPLOAD_FILE_PATH
 
     if not os.path.exists(save_path):
         os.mkdir(save_path)
     save_file = os.path.join(save_path, file.filename)
 
-    with open(save_file, "wb") as f:
-        contents = await file.read()
-        f.write(contents)
+    background_tasks.add_task(save_upload_file, file, save_file)
 
     return FileInResponse(fileID=new_file.id)
 
-
 @router.get(
     path="/{id}",
     name="file:check upload status",
@@ -51,4 +53,4 @@ async def check_status(
     # 0 for in process
     # 1 for complete successfully
     # -1 for error
-    return FileStatusInResponse(status=random.choice(choices))
+    return FileStatusInResponse(status=1)

backend/src/api/routes/train.py

@@ -33,9 +33,8 @@ async def check_status(
     id: int,
 ) -> TrainStatusInResponse:
 
-    # TODO check process status of training
     choices = [0, 1, -1]
     # 0 for in process
     # 1 for complete successfully
     # -1 for error
-    return TrainStatusInResponse(status=random.choice(choices))
+    return TrainStatusInResponse(status=1)

backend/src/config/settings/const.py

@@ -0,0 +1,4 @@
+# Uploaded file path
+UPLOAD_FILE_PATH = "./uploaded_files/"
+MAX_SQL_LENGTH = 200
+DEFAULT_MODEL = "all-MiniLM-L6-v2"

backend/src/models/schemas/ai_model.py

@@ -2,15 +2,22 @@
 
 
 class AiModel(BaseSchemaModel):
+    id: int
     name: str
-    des: str | None
+    des: str
 
 
 class AiModelInResponse(BaseSchemaModel):
     id: int
-    available_models: AiModel
+    name: str
+    des: str
 
 
 class AiModelInUpdate(BaseSchemaModel):
     name: str | None
     des: str | None
+
+
+class AiModelChooseResponse(BaseSchemaModel):
+    name: str
+    msg: str

backend/src/models/schemas/base.py

@@ -9,8 +9,8 @@
 
 class BaseSchemaModel(pydantic.BaseModel):
     class Config(pydantic.BaseConfig):
-        orm_mode: bool = True
+        from_attributes: bool = True
         validate_assignment: bool = True
-        allow_population_by_field_name: bool = True
+        populate_by_name: bool = True
         json_encoders: dict = {datetime.datetime: format_datetime_into_isoformat}
         alias_generator: typing.Any = format_dict_key_to_camel_case