Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

modification for reaxflow-workflow-service #70

Open
wants to merge 15 commits into
base: main
Choose a base branch
from
Open

modification for reaxflow-workflow-service #70

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
822ed8c
modification for reaxflow-workflow-service
Kirankumaraswamy Aug 18, 2023
0f01a52
remove unused functions
Kirankumaraswamy Aug 22, 2023
0581c4e
review changes
Kirankumaraswamy Aug 23, 2023
d79d052
few more review changes
Kirankumaraswamy Aug 24, 2023
4be4c9f
reaxpro workflow changes
Kirankumaraswamy Aug 30, 2023
a5f11e4
fix for query bug
Sep 4, 2023
8dd7267
inherit missing system class in marketplace app
Sep 7, 2023
c2907eb
move system file content to base
Sep 8, 2023
f1d6efe
remove the system import
Sep 8, 2023
ee34679
update standard app version
Sep 8, 2023
832aeaf
update standard app version
Sep 8, 2023
b579abe
keycloak authentication and datasink rename
Sep 18, 2023
4a57eed
keycloak authentication and datasink rename
Sep 18, 2023
a83c96c
review changes
Sep 29, 2023
293ed92
remove default client_id
Sep 29, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
0 examples/datasink_client/collection_dcat.py → examples/data_sink_client/collection_dcat.py
View file
Open in desktop
File renamed without changes.
0 examples/datasink_client/dataset_dcat.py → examples/data_sink_client/dataset_dcat.py
View file
Open in desktop
File renamed without changes.
0 ...ples/datasink_client/delete_collection.py → ...les/data_sink_client/delete_collection.py
View file
Open in desktop
File renamed without changes.
0 examples/datasink_client/delete_dataset.py → examples/data_sink_client/delete_dataset.py
View file
Open in desktop
File renamed without changes.
0 examples/datasink_client/download_file.py → examples/data_sink_client/download_file.py
View file
Open in desktop
File renamed without changes.
0 examples/datasink_client/download_folder.py → examples/data_sink_client/download_folder.py
View file
Open in desktop
File renamed without changes.
0 examples/datasink_client/list_collections.py → ...ples/data_sink_client/list_collections.py
View file
Open in desktop
File renamed without changes.
0 examples/datasink_client/list_datasets.py → examples/data_sink_client/list_datasets.py
View file
Open in desktop
File renamed without changes.
0 examples/datasink_client/query.py → examples/data_sink_client/query.py
View file
Open in desktop
File renamed without changes.
0 examples/datasink_client/query_dataset.py → examples/data_sink_client/query_dataset.py
View file
Open in desktop
File renamed without changes.
0 ...datasink_client/upload_files_from_path.py → ...ata_sink_client/upload_files_from_path.py
View file
Open in desktop
File renamed without changes.
0 examples/datasink_client/upload_folder.py → examples/data_sink_client/upload_folder.py
View file
Open in desktop
File renamed without changes.
5 changes: 4 additions & 1 deletion marketplace/app/v0/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
from .object_storage import MarketPlaceObjectStorageApp
from .system import MarketPlaceSystemApp
from .transformation import MarketPlaceTransformationApp


class MarketPlaceApp(MarketPlaceObjectStorageApp, MarketPlaceTransformationApp):
class MarketPlaceApp(
MarketPlaceObjectStorageApp, MarketPlaceTransformationApp, MarketPlaceSystemApp
):
pass
12 changes: 7 additions & 5 deletions marketplace/app/v0/object_storage.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@
from .base import _MarketPlaceAppBase
from .utils import _decode_metadata, _encode_metadata

DEFAULT_COLLECTION_NAME = "DEFAULT_COLLECTION"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why isn't the default defined at the Standard app API level?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought it would be easy to integrate if in case we need a name change for default collection. SInce standardapp requires rebuilding production. But anyway now I think it soesnot make sense to change the name once datasink application is established. I can move it to standard app if needed.



class MarketPlaceObjectStorageApp(_MarketPlaceAppBase):
@check_capability_availability
Expand Down Expand Up @@ -106,7 +108,7 @@ def create_collection(
@check_capability_availability
def create_dataset(
self,
collection_name: object_storage.CollectionName,
collection_name: object_storage.CollectionName = DEFAULT_COLLECTION_NAME,
dataset_name: object_storage.DatasetName = None,
metadata: dict = None,
file: UploadFile = None,
Expand Down Expand Up @@ -153,8 +155,8 @@ def create_dataset_metadata(
@check_capability_availability
def get_dataset(
self,
collection_name: object_storage.CollectionName,
dataset_name: object_storage.DatasetName,
collection_name: object_storage.CollectionName = DEFAULT_COLLECTION_NAME,
) -> Union[Dict, str]:
return self._client.get(
self._proxy_path("getDataset"),
Expand Down Expand Up @@ -196,8 +198,8 @@ def create_or_replace_dataset_metadata(
@check_capability_availability
def delete_dataset(
self,
collection_name: object_storage.CollectionName,
dataset_name: object_storage.DatasetName,
collection_name: object_storage.CollectionName = DEFAULT_COLLECTION_NAME,
):
return self._client.delete(
self._proxy_path("deleteDataset"),
Expand Down Expand Up @@ -252,8 +254,8 @@ def get_collection_metadata_dcat(
@check_capability_availability
def get_dataset_metadata_dcat(
self,
collection_name: object_storage.CollectionName,
dataset_name: object_storage.DatasetName,
collection_name: object_storage.CollectionName = DEFAULT_COLLECTION_NAME,
) -> Union[Dict, str]:
response: dict = self._client.get(
self._proxy_path("getDatasetMetadataDcat"),
Expand All @@ -264,9 +266,9 @@ def get_dataset_metadata_dcat(
@check_capability_availability
def query_dataset(
self,
collection_name: object_storage.CollectionName,
dataset_name: object_storage.DatasetName,
query: str,
collection_name: object_storage.CollectionName = DEFAULT_COLLECTION_NAME,
) -> Union[Dict, str]:
response: dict = self._client.post(
self._proxy_path("queryDataset"),
Expand Down
24 changes: 24 additions & 0 deletions marketplace/app/v0/system.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
from typing import Optional

from fastapi.responses import JSONResponse, Response

from ..utils import check_capability_availability
from .base import _MarketPlaceAppBase


class MarketPlaceSystemApp(_MarketPlaceAppBase):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why aren't these capabilities directly defined in _MarketPlaceAppBase, loke globalSearch or heartbeat?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah makes sense. Since there was an additional system level file in standardapp, I added a new one here. I will move them to base.

@check_capability_availability
def get_logs(
self, id: Optional[str], limit: int = 100, offset: int = 0
) -> Response:
return self._client.get(
self._proxy_path("getLogs"),
params={"id": id, "limit": limit, "offset": offset},
).content

@check_capability_availability
def get_info(self, config: dict = None) -> JSONResponse:
params = {}
if config is not None:
params.update(config)
return self._client.get(self._proxy_path("getInfo"), params=params).json()
12 changes: 10 additions & 2 deletions marketplace/app/v0/transformation.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,19 @@ def get_transformation_list(

@check_capability_availability
def new_transformation(
self, new_transformation: transformation.NewTransformationModel
self,
new_transformation: transformation.NewTransformationModel,
config: dict = None,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The purpose of config is unclear to me, when we already have params. This should be better documented (not inline, but a proper docstring)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To send additional query parameters which are not defined in Marketplace. For example reaxpro-service needs model_name an additional query parameter to create the transformation. Any key values that are passed in config will be send as query parameters to the application by marketplace. An alternative solution can be to send inside the body. I thought sending it as query will make it less restriction at the application side.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still think needs a proper docstring (so users can see it through contextual help in their IDEs) would be specially benefitial in this instance.

) -> transformation.TransformationCreateResponse:
params = {}
# send additional key value as query parameters if some app needs it
if config is not None:
params.update(config)
return transformation.TransformationCreateResponse.parse_obj(
self._client.post(
self._proxy_path("newTransformation"), json=new_transformation
self._proxy_path("newTransformation"),
json=new_transformation,
params=params,
).json()
)

Expand Down
0 marketplace/datasink_client/README.md → marketplace/data_sink_client/README.md
View file
Open in desktop
File renamed without changes.
0 marketplace/datasink_client/__init__.py → marketplace/data_sink_client/__init__.py
View file
Open in desktop
File renamed without changes.
0 marketplace/datasink_client/cli.py → marketplace/data_sink_client/cli.py
View file
Open in desktop
File renamed without changes.
15 changes: 7 additions & 8 deletions marketplace/datasink_client/session.py → marketplace/data_sink_client/session.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,10 +138,9 @@ def create_collection(self, collection_name, sub_collection_id):
collection_name=collection_name, config=config
)
if "collection_id" not in response:
print(response)
return None
else:
return response["collection_id"]
return response

except Exception as e:
print(
Expand Down Expand Up @@ -245,7 +244,9 @@ def query_dataset(self, collection_name, dataset_name, query):

:returns: List of data
"""
response = self.marketPlace.query_dataset(collection_name, dataset_name, query)
response = self.marketPlace.query_dataset(
collection_name=collection_name, dataset_name=dataset_name, query=query
)
return response

@reconfigure_if_expired
Expand Down Expand Up @@ -294,13 +295,11 @@ def create_datasets_from_paths(self, paths, collection_name, dataset_names):

response_list = []
if collection_name is not None:
collection_id = self.create_collection(
response = self.create_collection(
collection_name=collection_name, sub_collection_id=None
)
if collection_id is not None:
response_list.append((collection_name, collection_id))
else:
return
if response is not None:
response_list.append((collection_name, response["collection_id"]))
else:
raise Exception("collection title cannot be empty.")

Expand Down
0 marketplace/datasink_client/utils.py → marketplace/data_sink_client/utils.py
View file
Open in desktop
File renamed without changes.