Skeleton code for refactor with OpenAPI Spec (#149)

* Copied microservice template for runner

* Minor pythonic changes

* Comments to earmark api endpoints

* modified parts of recipe api to be similar to runner api endpoints

* Unlinked users from api and improved recipes a bit

* Added license text to all files

* Added runner spec

* Added Data & Queuer Service APIs mocked responses and api spec

* Added preliminary dispatcher service and api spec

* Added parameter to UpdateWorkflowRequest schema

* Added ResultStatus enum

* Updated dispatcher api spec to include result enum

* removed requirements files for components

* renamed result id to dispatch id

* Changes to runner api

* Removed user from api

* Updated runner api

* UI Backend component added as well, still some errors yet to figure out

* Renamed node id to task id

* UI Backend complete

* Updated Result status enum in Dispatcher component

* Runner finished

* Queuer finished

* Added license to dispatcher files

* Renamed dispatcher folder

* Added license and some file renaming in data component

* Tags changed for apis

* Added preliminary Queuer class, and test NATS consumer. Added temp README for Queuer

* (wip) Added test publish message to dispatch endpoint

* Data component done

* Recreated component openapi specs

* Removed duplicate dispatcher, and renamed folder

* Updated data component with dispatch id as route parameter

* updated UI backend

* Updated runner with route parameters

* Queuer updated

* Updated dispatcher

* UI backend api spec regenerated

* Runner api spec regenerated

* Queuer api spec regenerated

* Dispatcher openapi spec created

* Data openapi spec regenerated

* Made suggested changes

* Removed Queuer class & references

* Updated Runner API spec

* Updated UI service spec

* Updated API specs for Runner & Dispatcher

* improved runner's run_task inputs

* Updated task run input

* Updated with suggestions

* Added result object description

* Added description of what the string means in runner's post

* Added another example

* Added example code in quotes

* Updated version and changelog

Co-authored-by: Alejandro Esquivel <[email protected]>

CHANGELOG.md

@@ -5,6 +5,13 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.30.0] - 2022-03-09
+
+### Added
+
+- Skeleton code for writing the different services corresponding to each component in the open source refactor.
+- OpenAPI specifications for each of the services.
+
 ## [0.29.3] - 2022-03-09
 
 ### Fixed

VERSION

@@ -1 +1 @@
-0.29.3
+0.30.0

refactor/data/app/__init__.py

@@ -0,0 +1,19 @@
+# Copyright 2021 Agnostiq Inc.
+#
+# This file is part of Covalent.
+#
+# Licensed under the GNU Affero General Public License 3.0 (the "License").
+# A copy of the License may be obtained with this software package or at
+#
+#      https://www.gnu.org/licenses/agpl-3.0.en.html
+#
+# Use of this file is prohibited except in compliance with the License. Any
+# modifications or derivative works of this file must retain this copyright
+# notice, and modified files must contain a notice indicating that they have
+# been altered from the originals.
+#
+# Covalent is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+# FITNESS FOR A PARTICULAR PURPOSE. See the License for more details.
+#
+# Relief from the License may be granted by purchasing a commercial license.

refactor/data/app/api/__init__.py

@@ -0,0 +1,19 @@
+# Copyright 2021 Agnostiq Inc.
+#
+# This file is part of Covalent.
+#
+# Licensed under the GNU Affero General Public License 3.0 (the "License").
+# A copy of the License may be obtained with this software package or at
+#
+#      https://www.gnu.org/licenses/agpl-3.0.en.html
+#
+# Use of this file is prohibited except in compliance with the License. Any
+# modifications or derivative works of this file must retain this copyright
+# notice, and modified files must contain a notice indicating that they have
+# been altered from the originals.
+#
+# Covalent is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+# FITNESS FOR A PARTICULAR PURPOSE. See the License for more details.
+#
+# Relief from the License may be granted by purchasing a commercial license.

refactor/data/app/api/api_v0/__init__.py

@@ -0,0 +1,19 @@
+# Copyright 2021 Agnostiq Inc.
+#
+# This file is part of Covalent.
+#
+# Licensed under the GNU Affero General Public License 3.0 (the "License").
+# A copy of the License may be obtained with this software package or at
+#
+#      https://www.gnu.org/licenses/agpl-3.0.en.html
+#
+# Use of this file is prohibited except in compliance with the License. Any
+# modifications or derivative works of this file must retain this copyright
+# notice, and modified files must contain a notice indicating that they have
+# been altered from the originals.
+#
+# Covalent is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+# FITNESS FOR A PARTICULAR PURPOSE. See the License for more details.
+#
+# Relief from the License may be granted by purchasing a commercial license.

refactor/data/app/api/api_v0/api.py

@@ -0,0 +1,27 @@
+# Copyright 2021 Agnostiq Inc.
+#
+# This file is part of Covalent.
+#
+# Licensed under the GNU Affero General Public License 3.0 (the "License").
+# A copy of the License may be obtained with this software package or at
+#
+#      https://www.gnu.org/licenses/agpl-3.0.en.html
+#
+# Use of this file is prohibited except in compliance with the License. Any
+# modifications or derivative works of this file must retain this copyright
+# notice, and modified files must contain a notice indicating that they have
+# been altered from the originals.
+#
+# Covalent is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+# FITNESS FOR A PARTICULAR PURPOSE. See the License for more details.
+#
+# Relief from the License may be granted by purchasing a commercial license.
+
+
+from app.api.api_v0.endpoints import fs, workflow
+from fastapi import APIRouter
+
+api_router = APIRouter()
+api_router.include_router(workflow.router, prefix="/workflow", tags=["Workflow"])
+api_router.include_router(fs.router, prefix="/fs", tags=["Data"])

refactor/data/app/api/api_v0/endpoints/__init__.py

@@ -0,0 +1,19 @@
+# Copyright 2021 Agnostiq Inc.
+#
+# This file is part of Covalent.
+#
+# Licensed under the GNU Affero General Public License 3.0 (the "License").
+# A copy of the License may be obtained with this software package or at
+#
+#      https://www.gnu.org/licenses/agpl-3.0.en.html
+#
+# Use of this file is prohibited except in compliance with the License. Any
+# modifications or derivative works of this file must retain this copyright
+# notice, and modified files must contain a notice indicating that they have
+# been altered from the originals.
+#
+# Covalent is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+# FITNESS FOR A PARTICULAR PURPOSE. See the License for more details.
+#
+# Relief from the License may be granted by purchasing a commercial license.

refactor/data/app/api/api_v0/endpoints/fs.py

@@ -0,0 +1,63 @@
+# Copyright 2021 Agnostiq Inc.
+#
+# This file is part of Covalent.
+#
+# Licensed under the GNU Affero General Public License 3.0 (the "License").
+# A copy of the License may be obtained with this software package or at
+#
+#      https://www.gnu.org/licenses/agpl-3.0.en.html
+#
+# Use of this file is prohibited except in compliance with the License. Any
+# modifications or derivative works of this file must retain this copyright
+# notice, and modified files must contain a notice indicating that they have
+# been altered from the originals.
+#
+# Covalent is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+# FITNESS FOR A PARTICULAR PURPOSE. See the License for more details.
+#
+# Relief from the License may be granted by purchasing a commercial license.
+
+
+from typing import Any
+
+from app.schemas.common import HTTPExceptionSchema
+from app.schemas.fs import UploadResponse
+from fastapi import APIRouter, File, HTTPException
+from fastapi.responses import FileResponse
+
+router = APIRouter()
+
+
+@router.post("/upload", status_code=200, response_model=UploadResponse)
+def upload_file(*, file: bytes = File(...)) -> Any:
+    """
+    Upload a file
+    """
+    return {
+        "filename": "sample_input_file.sample",
+        "path": "/Users/aq/Documents/agnostiq/uploads/",
+    }
+
+
+@router.get(
+    "/download",
+    status_code=200,
+    response_class=FileResponse,
+    responses={
+        404: {"model": HTTPExceptionSchema, "description": "File was not found"},
+        200: {
+            "content": {"application/octet-stream": {}},
+            "description": "Return binary content of file.",
+        },
+    },
+)
+def download_file(*, file_location: str) -> Any:
+    """
+    Download a file
+    """
+    if not file_location:
+        raise HTTPException(status_code=404, detail="File not found")
+    return FileResponse(
+        file_location, media_type="application/octet-stream", filename=file_location
+    )

refactor/data/app/api/api_v0/endpoints/workflow.py

@@ -0,0 +1,106 @@
+# Copyright 2021 Agnostiq Inc.
+#
+# This file is part of Covalent.
+#
+# Licensed under the GNU Affero General Public License 3.0 (the "License").
+# A copy of the License may be obtained with this software package or at
+#
+#      https://www.gnu.org/licenses/agpl-3.0.en.html
+#
+# Use of this file is prohibited except in compliance with the License. Any
+# modifications or derivative works of this file must retain this copyright
+# notice, and modified files must contain a notice indicating that they have
+# been altered from the originals.
+#
+# Covalent is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+# FITNESS FOR A PARTICULAR PURPOSE. See the License for more details.
+#
+# Relief from the License may be granted by purchasing a commercial license.
+
+
+from typing import Any, Union
+
+from app.schemas.common import HTTPExceptionSchema
+from app.schemas.workflow import (
+    InsertResultResponse,
+    Node,
+    Result,
+    ResultPickle,
+    UpdateResultResponse,
+)
+from fastapi import APIRouter
+
+router = APIRouter()
+
+# For now we don't return this, instead just a bytes-like pickle object
+mock_result_object_instance = {
+    "dispatch_id": "",
+    "results_dir": "",
+    "status": "COMPLETED",
+    "graph": {
+        "links": [{"edge_name": "data", "param_type": "kwarg", "source": 0, "target": 1}],
+        "nodes": [
+            {
+                "name": "load_data",
+                "metadata": {"executor": "<LocalExecutor>"},
+                "function_string": "@ct.electron\ndef load_data():\n    iris = datasets.load_iris()\n    perm = permutation(iris.target.size)\n    iris.data = iris.data[perm]\n    iris.target = iris.target[perm]\n    return iris.data, iris.target\n\n\n",
+                "start_time": "2022-02-28T23:21:17.844268+00:00",
+                "end_time": "2022-02-28T23:21:17.853741+00:00",
+                "status": "COMPLETED",
+                "output": ["[[5.  3.4 1.5 0.2]]"],
+                "error": None,
+                "sublattice_result": None,
+                "stdout": "",
+                "stderr": "",
+                "id": 0,
+                "doc": None,
+                "kwargs": None,
+            }
+        ],
+    },
+}
+
+
+@router.get(
+    "/results/{dispatch_id}",
+    status_code=200,
+    response_model=ResultPickle,
+    responses={
+        404: {"model": HTTPExceptionSchema, "description": "The result was not found"},
+    },
+)
+def get_result(
+    *,
+    dispatch_id: str,
+) -> Any:
+    """
+    Get a result object
+    """
+
+    return {"result_object": b"result pickle object"}
+
+
+@router.post("/results", status_code=200, response_model=InsertResultResponse)
+def insert_result(
+    *,
+    result_object: Union[Result, ResultPickle],
+) -> Any:
+    """
+    Insert a result object
+    """
+
+    return {"response": "Result successfully added to db"}
+
+
+@router.put("/results/{dispatch_id}", status_code=200, response_model=UpdateResultResponse)
+def update_result(
+    *,
+    dispatch_id: str,
+    task: Node,
+) -> Any:
+    """
+    Update a result object
+    """
+
+    return {"response": "Task updated successfully"}

refactor/data/app/core/config.py

@@ -0,0 +1,46 @@
+# Copyright 2021 Agnostiq Inc.
+#
+# This file is part of Covalent.
+#
+# Licensed under the GNU Affero General Public License 3.0 (the "License").
+# A copy of the License may be obtained with this software package or at
+#
+#      https://www.gnu.org/licenses/agpl-3.0.en.html
+#
+# Use of this file is prohibited except in compliance with the License. Any
+# modifications or derivative works of this file must retain this copyright
+# notice, and modified files must contain a notice indicating that they have
+# been altered from the originals.
+#
+# Covalent is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+# FITNESS FOR A PARTICULAR PURPOSE. See the License for more details.
+#
+# Relief from the License may be granted by purchasing a commercial license.
+
+
+from typing import List, Union
+
+from dotenv import load_dotenv
+from pydantic import AnyHttpUrl, BaseSettings, validator
+
+load_dotenv()
+
+
+class Settings(BaseSettings):
+    API_V0_STR: str = "/api/v0"
+    BACKEND_CORS_ORIGINS: List[AnyHttpUrl] = []
+
+    @validator("BACKEND_CORS_ORIGINS", pre=True)
+    def assemble_cors_origins(cls, v: Union[str, List[str]]) -> Union[List[str], str]:
+        if isinstance(v, str) and not v.startswith("["):
+            return [i.strip() for i in v.split(",")]
+        elif isinstance(v, (list, str)):
+            return v
+        raise ValueError(v)
+
+    class Config:
+        case_sensitive = True
+
+
+settings = Settings()

refactor/data/app/schemas/__init__.py

@@ -0,0 +1,23 @@
+# Copyright 2021 Agnostiq Inc.
+#
+# This file is part of Covalent.
+#
+# Licensed under the GNU Affero General Public License 3.0 (the "License").
+# A copy of the License may be obtained with this software package or at
+#
+#      https://www.gnu.org/licenses/agpl-3.0.en.html
+#
+# Use of this file is prohibited except in compliance with the License. Any
+# modifications or derivative works of this file must retain this copyright
+# notice, and modified files must contain a notice indicating that they have
+# been altered from the originals.
+#
+# Covalent is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+# FITNESS FOR A PARTICULAR PURPOSE. See the License for more details.
+#
+# Relief from the License may be granted by purchasing a commercial license.
+
+
+from .fs import UploadResponse
+from .workflow import Result

refactor/data/app/schemas/common.py

@@ -0,0 +1,26 @@
+# Copyright 2021 Agnostiq Inc.
+#
+# This file is part of Covalent.
+#
+# Licensed under the GNU Affero General Public License 3.0 (the "License").
+# A copy of the License may be obtained with this software package or at
+#
+#      https://www.gnu.org/licenses/agpl-3.0.en.html
+#
+# Use of this file is prohibited except in compliance with the License. Any
+# modifications or derivative works of this file must retain this copyright
+# notice, and modified files must contain a notice indicating that they have
+# been altered from the originals.
+#
+# Covalent is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+# FITNESS FOR A PARTICULAR PURPOSE. See the License for more details.
+#
+# Relief from the License may be granted by purchasing a commercial license.
+
+
+from pydantic import BaseModel
+
+
+class HTTPExceptionSchema(BaseModel):
+    detail: str