Merge pull request #446 from openvinotoolkit/idavidyu/CVS-100811-expo…

…rt-import-api-alignment

Project/Dataset export import API alignment

CHANGELOG.md

@@ -378,7 +378,7 @@ This release breaks backwards compatibility with `deployments` created by earlie
 * Add performance hint to the ovms config by @ljcornel in https://github.com/openvinotoolkit/geti-sdk/pull/152
 * Fix bug in deployment resource clean up method by @ljcornel in https://github.com/openvinotoolkit/geti-sdk/pull/153
 * Update python-dotenv requirement from ==0.21.* to ==1.0.* in /requirements by @dependabot in https://github.com/openvinotoolkit/geti-sdk/pull/156
-* Add a short sleep in `Geti.upload_project` after media upload by @ljcornel in https://github.com/openvinotoolkit/geti-sdk/pull/157
+* Add a short sleep in `Geti.upload_project_data` after media upload by @ljcornel in https://github.com/openvinotoolkit/geti-sdk/pull/157
 * Add OVMS deployment resources to manifest by @ljcornel in https://github.com/openvinotoolkit/geti-sdk/pull/158
 
 
@@ -477,7 +477,7 @@ This release breaks backwards compatibility with `deployments` created by earlie
 * Update numpy requirement to 1.21.* by @ljcornel in https://github.com/openvinotoolkit/geti-sdk/pull/89
 * Reduce permissions upon directory creation by @ljcornel in https://github.com/openvinotoolkit/geti-sdk/pull/90
 * Update README to correctly reference Intel Geti brand everywhere by @ljcornel in https://github.com/openvinotoolkit/geti-sdk/pull/92
-* Improve check for video processing in `Geti.upload_project()` to avoid potential infinite loop by @ljcornel in https://github.com/openvinotoolkit/geti-sdk/pull/93
+* Improve check for video processing in `Geti.upload_project_data()` to avoid potential infinite loop by @ljcornel in https://github.com/openvinotoolkit/geti-sdk/pull/93
 * Add unit tests to pre-merge test suite by @ljcornel in https://github.com/openvinotoolkit/geti-sdk/pull/91
 * Update ProjectStatus and TaskStatus to include new field `n_new_annotations` by @ljcornel in https://github.com/openvinotoolkit/geti-sdk/pull/94
 * Add progress bars for up/download of projects, media, annotations and predictions by @ljcornel in https://github.com/openvinotoolkit/geti-sdk/pull/95

README.md

@@ -186,7 +186,7 @@ Instantiating the `Geti` class will establish the connection and perform authent
     host="https://your_server_hostname_or_ip_address", token="your_personal_access_token"
   )
 
-  geti.download_project(project_name="dummy_project")
+  geti.download_project_data(project_name="dummy_project")
   ```
 
   Here, it is assumed that the project with name 'dummy_project' exists on the cluster.
@@ -223,7 +223,7 @@ Instantiating the `Geti` class will establish the connection and perform authent
     host="https://your_server_hostname_or_ip_address", token="your_personal_access_token"
   )
 
-  geti.upload_project(target_folder="dummy_project")
+  geti.upload_project_data(target_folder="dummy_project")
   ```
 
   The parameter `target_folder` must be a valid path to the directory holding the
@@ -301,10 +301,10 @@ the screenshot below).
 ## High level API reference
 The `Geti` class provides the following methods:
 
-- `download_project` -- Downloads a project by project name.
+- `download_project_data` -- Downloads a project by project name (Geti-SDK representation), returns an interactive object.
 
 
-- `upload_project` -- Uploads project from a folder.
+- `upload_project_data` -- Uploads project (Geti-SDK representation) from a folder.
 
 
 - `download_all_projects` -- Downloads all projects found on the server.
@@ -313,6 +313,18 @@ The `Geti` class provides the following methods:
 - `upload_all_projects` -- Uploads all projects found in a specified folder to the
   server.
 
+- `export_project` -- Exports a project to an archive on disk. This method is useful for
+  creating a backup of a project, or for migrating a project to a different cluster.
+
+- `import_project` -- Imports a project from an archive on disk. This method is useful for
+  restoring a project from a backup, or for migrating a project to a different cluster.
+
+- `export_dataset` -- Exports a dataset to an archive on disk. This method is useful for
+  creating a backup of a dataset, or for migrating a dataset to a different cluster.
+
+- `import_dataset` -- Imports a dataset from an archive on disk. A new project will
+  be created for the dataset. This method is useful for restoring a project from a dataset
+  backup, or for migrating a dataset to a different cluster.
 
 - `upload_and_predict_image` -- Uploads a single image to an existing project on the
   server, and requests a prediction for that image. Optionally, the prediction can
@@ -427,5 +439,5 @@ docker run --rm -ti -v $(pwd):/app geti-sdk:latest /bin/bash
 
 - Model upload
 - Prediction upload
-- Exporting datasets to COCO/YOLO/VOC format: For this, you can use the export
+- Importing datasets to an existing project: For this, you can use the import
   functionality from the Intel® Geti™ user interface instead.

docs/source/api_reference.rst

@@ -7,6 +7,7 @@ API Reference
    geti
    Annotation readers <geti_sdk.annotation_readers>
    Data models <geti_sdk.data_models>
+   Import Export module <geti_sdk.import_export>
    Deployment <geti_sdk.deployment>
    HTTP session <geti_sdk.http_session>
    REST converters <geti_sdk.rest_converters>

geti_sdk/__init__.py

@@ -17,9 +17,9 @@
 ------------
 
 These pages contain the documentation for the main SDK class,
-:py:class:`~geti_sdk.sc_rest_client.Geti`.
+:py:class:`~geti_sdk.geti.Geti`.
 
-The :py:class:`~geti_sdk.sc_rest_client.Geti` class implements convenience
+The :py:class:`~geti_sdk.geti.Geti` class implements convenience
 methods for common operations that can be performed on the Intel® Geti™ cluster, such as
 creating a project from a pre-existing dataset, downloading or uploading a project,
 uploading an image and getting a prediction for it and creating a deployment for a
@@ -35,7 +35,31 @@
      host="https://0.0.0.0", username="dummy_user", password="dummy_password"
    )
 
-   geti.download_project(project_name="dummy_project")
+   geti.download_project_data(project_name="dummy_project")
+
+The :py:class:`~geti_sdk.geti.Geti` class provides a high-level interface for
+import-export operations in Intel® Geti™ platform. Here is a list of these operations:
+* Project download
+   :py:meth:`~geti_sdk.geti.Geti.download_project_data` method fetches the project data
+   and creates a local Python object that supports a range of operations with the project.
+* Project upload
+   :py:meth:`~geti_sdk.geti.Geti.upload_project_data` method uploads the project data
+   from a local Python object to the Intel® Geti™ platform.
+* Batched project download and upload
+   :py:meth:`~geti_sdk.geti.Geti.download_all_projects` and
+   :py:meth:`~geti_sdk.geti.Geti.upload_all_projects` methods download and upload
+   multiple projects at once.
+* Project export
+   :py:meth:`~geti_sdk.geti.Geti.export_project` method exports the project snapshot
+   to a zip archive. The archive can be used to import the project to another or the same Intel® Geti™
+   instance.
+* Project import
+   :py:meth:`~geti_sdk.geti.Geti.import_project` method imports the project from a zip archive.
+* Dataset export
+   :py:meth:`~geti_sdk.geti.Geti.export_dataset` method exports the dataset to a zip archive.
+* Dataset import
+   :py:meth:`~geti_sdk.geti.Geti.import_dataset` method imports the dataset from a zip archive
+   as a new project.
 
 For custom operations or more fine-grained control over the behavior, the
 :py:mod:`~geti_sdk.rest_clients` subpackage should be used.
@@ -48,20 +72,30 @@
 
    .. rubric:: Project download and upload
 
-   .. automethod:: download_project
+   .. automethod:: download_project_data
 
-   .. automethod:: upload_project
+   .. automethod:: upload_project_data
 
    .. automethod:: download_all_projects
 
    .. automethod:: upload_all_projects
 
+   .. automethod:: import_project
+
+   .. automethod:: export_project
+
+   .. rubric:: Dataset export
+
+   .. automethod:: export_dataset
+
    .. rubric:: Project creation from dataset
 
    .. automethod:: create_single_task_project_from_dataset
 
    .. automethod:: create_task_chain_project_from_dataset
 
+   .. automethod:: import_dataset
+
    .. rubric:: Project deployment
 
    .. automethod:: deploy_project

geti_sdk/data_models/__init__.py

@@ -23,7 +23,7 @@
 :py:class:`~geti_sdk.data_models.model.Model` and many more.
 
 When interacting with the GETi cluster through the
-:py:class:`geti_sdk.sc_rest_client.Geti` or the
+:py:class:`geti_sdk.geti.Geti` or the
 :py:mod:`~geti_sdk.rest_clients`, all entities retrieved from the cluster will be
 deserialized into the data models defined in this package.
 

geti_sdk/data_models/enums/dataset_format.py

@@ -0,0 +1,33 @@
+# Copyright (C) 2024 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions
+# and limitations under the License.
+
+from enum import Enum
+
+
+class DatasetFormat(Enum):
+    """
+    Enum representing the different annotation formats for datasets within an
+    Intel® Geti™ platform.
+    """
+
+    COCO = "coco"
+    YOLO = "yolo"
+    VOC = "voc"
+    DATUMARO = "datumaro"
+
+    def __str__(self):
+        """
+        Return the string representation of the DatasetFormat instance.
+        """
+        return self.value

geti_sdk/data_models/enums/job_type.py

@@ -31,6 +31,9 @@ class JobType(Enum):
     TEST = "test"
     PREPARE_IMPORT_TO_NEW_PROJECT = "prepare_import_to_new_project"
     PERFORM_IMPORT_TO_NEW_PROJECT = "perform_import_to_new_project"
+    EXPORT_PROJECT = "export_project"
+    IMPORT_PROJECT = "import_project"
+    EXPORT_DATASET = "export_dataset"
 
     def __str__(self) -> str:
         """

geti_sdk/data_models/job.py

@@ -102,6 +102,31 @@ class ProjectMetadata:
 
     name: Optional[str] = None
     id: Optional[str] = None
+    type: Optional[str] = None
+
+
+@attr.define
+class DatasetMetadata:
+    """
+    Metadata related to a dataset on the GETi cluster.
+
+    :var name: Name of the dataset
+    :var id: ID of the dataset
+    """
+
+    name: Optional[str] = None
+    id: Optional[str] = None
+
+
+@attr.define
+class ParametersMetadata:
+    """
+    Metadata related to a project import to the GETi cluster.
+
+    :var file_id: ID of the uploaded file
+    """
+
+    file_id: Optional[str] = None
 
 
 @attr.define
@@ -153,11 +178,16 @@ class JobMetadata:
 
     task: Optional[TaskMetadata] = None
     project: Optional[ProjectMetadata] = None
+    dataset: Optional[DatasetMetadata] = None
+    parameters: Optional[ParametersMetadata] = None
     test: Optional[TestMetadata] = None
     base_model_id: Optional[str] = None
     model_storage_id: Optional[str] = None
     optimization_type: Optional[str] = None
     optimized_model_id: Optional[str] = None
+    download_url: Optional[str] = None
+    export_format: Optional[str] = None
+    file_id: Optional[str] = None
     scores: Optional[List[ScoreMetadata]] = None
     trained_model: Optional[ModelMetadata] = None  # Added in Geti v1.7
     warnings: Optional[List[dict]] = None  # Added in Geti v1.13 for dataset import jobs
@@ -191,6 +221,11 @@ class Job:
     :var id: Unique database ID of the job
     :var project_id: Unique database ID of the project from which the job originates
     :var type: Type of the job
+    :var creation_time: Time at which the job was created
+    :var start_time: Time at which the job started running
+    :var end_time: Time at which the job finished running
+    :var author: Author of the job
+    :var cancellation_info: Information relating to the cancellation of the jobW
     :var metadata: JobMetadata object holding metadata for the job
     """
 
@@ -282,6 +317,12 @@ def update(self, session: GetiSession) -> "Job":
 
         self.steps = response.get("steps", None)
         self.state = JobState(response["state"])
+        self.metadata.project_id = response["metadata"].get("project_id", None)
+        self.metadata.download_url = response["metadata"].get("download_url", None)
+        self.metadata.warnings = response["metadata"].get("warnings", None)
+        self.metadata.supported_project_types = response["metadata"].get(
+            "supported_project_types", None
+        )
 
         if self._geti_version is None:
             self.geti_version = session.version

geti_sdk/deployment/__init__.py

@@ -24,8 +24,8 @@
 is the same in both cases.
 
 Creating a deployment for a project is done through the
-:py:class:`~geti_sdk.sc_rest_client.Geti` class, which provides a
-convenience method :py:meth:`~geti_sdk.sc_rest_client.Geti.deploy_project`.
+:py:class:`~geti_sdk.geti.Geti` class, which provides a
+convenience method :py:meth:`~geti_sdk.geti.Geti.deploy_project`.
 
 The following code snippet shows:
 

geti_sdk/deployment/deployed_model.py

@@ -187,8 +187,8 @@ def get_data(self, source: Union[str, os.PathLike, GetiSession]):
                     raise ValueError(
                         "\n"
                         "This deployment model is not compatible with the current SDK. Proposed solutions:\n"
-                        "1. Please deploy a model using GETi Platform version 2.0.0 or higher.\n"
-                        "2. Downgrade to a compatible GETi-SDK version to continue using this model.\n\n"
+                        "1. Please deploy a model using Intel Geti Platform version 2.0.0 or higher.\n"
+                        "2. Downgrade to a compatible Geti-SDK version to continue using this model.\n\n"
                     )
 
         elif isinstance(source, GetiSession):