diff --git a/docs/assemble-a-pipeline.md b/docs/assemble-a-pipeline.md index 5845b132..72ee43e7 100644 --- a/docs/assemble-a-pipeline.md +++ b/docs/assemble-a-pipeline.md @@ -2,7 +2,7 @@ A Pipeline is composed of Blocks. To construct a Pipeline, drag and drop Blocks into your workspace and connect the Blocks by adding links between the output of one Block and the input of the next Block. ZetaForge supports directed acyclic connections between the Blocks. -![drag-drop-blocks-add-links.gif](assets%2Fdrag-drop-blocks-add-links.gif) +![assemble-pipeline.gif](assets%2Fassemble-pipeline.gif) The changes made to the workspace are saved in `frontend/.cache`. When the first Block is added to an empty workspace, a new directory (`pipeline-${pipeline_id}`) will be created under `frontend/.cache` for the Pipeline representing the workspace. The information about the pipeline, such as the Blocks making the pipeline and the connections between them, is stored in a JSON file with the same name as the pipeline, i.e., `pipeline-${pipeline_id}.json`. Note that the `${pipeline_id}` is displayed at the top of the window in the ZetaForge app. @@ -33,4 +33,4 @@ Finally, we add the Save Text Block to store the story GPT generated to a file a We run the Pipeline by clicking on the Run button at the top of the window. Here is a screenshot of the full Pipeline: -![GPT Story Teller Pipeline](assets/gpt-storyteller-pipeline.png) +![Super Resolution Pipeline](assets/Super-resolution.png) diff --git a/docs/assets/Super-resolution.png b/docs/assets/Super-resolution.png new file mode 100644 index 00000000..68dd124b Binary files /dev/null and b/docs/assets/Super-resolution.png differ diff --git a/docs/assets/assemble-pipeline.gif b/docs/assets/assemble-pipeline.gif new file mode 100644 index 00000000..7a74be73 Binary files /dev/null and b/docs/assets/assemble-pipeline.gif differ diff --git a/docs/assets/block-editor.png b/docs/assets/block-editor.png index dcf68bc1..6fbec038 100644 Binary files a/docs/assets/block-editor.png and b/docs/assets/block-editor.png differ diff --git a/docs/assets/block-example.png b/docs/assets/block-example.png index 66656140..ff8a0fd7 100644 Binary files a/docs/assets/block-example.png and b/docs/assets/block-example.png differ diff --git a/docs/assets/canny-edge-mona-result.png b/docs/assets/canny-edge-mona-result.png new file mode 100644 index 00000000..535f3b4c Binary files /dev/null and b/docs/assets/canny-edge-mona-result.png differ diff --git a/docs/assets/canny-edge-specs.png b/docs/assets/canny-edge-specs.png new file mode 100644 index 00000000..b3576d41 Binary files /dev/null and b/docs/assets/canny-edge-specs.png differ diff --git a/docs/assets/compile-run-block-editor.gif b/docs/assets/compile-run-block-editor.gif new file mode 100644 index 00000000..2112a2e5 Binary files /dev/null and b/docs/assets/compile-run-block-editor.gif differ diff --git a/docs/assets/deploy-to-payload.gif b/docs/assets/deploy-to-payload.gif new file mode 100644 index 00000000..0bad0ea5 Binary files /dev/null and b/docs/assets/deploy-to-payload.gif differ diff --git a/docs/assets/deployed-payload.png b/docs/assets/deployed-payload.png new file mode 100644 index 00000000..f6428351 Binary files /dev/null and b/docs/assets/deployed-payload.png differ diff --git a/docs/assets/drop-a-block.gif b/docs/assets/drop-a-block.gif index c9ca878d..da4fbe76 100644 Binary files a/docs/assets/drop-a-block.gif and b/docs/assets/drop-a-block.gif differ diff --git a/docs/assets/executed-pipeline-button.png b/docs/assets/executed-pipeline-button.png new file mode 100644 index 00000000..4770df6b Binary files /dev/null and b/docs/assets/executed-pipeline-button.png differ diff --git a/docs/assets/executed-pipelines.png b/docs/assets/executed-pipelines.png new file mode 100644 index 00000000..0471ca5f Binary files /dev/null and b/docs/assets/executed-pipelines.png differ diff --git a/docs/assets/import-files-folder.png b/docs/assets/import-files-folder.png new file mode 100644 index 00000000..6cadaaad Binary files /dev/null and b/docs/assets/import-files-folder.png differ diff --git a/docs/assets/past-executions.png b/docs/assets/past-executions.png new file mode 100644 index 00000000..c0408cad Binary files /dev/null and b/docs/assets/past-executions.png differ diff --git a/docs/assets/remove-a-link.png b/docs/assets/remove-a-link.png index 5c2b5cc3..38e3a03b 100644 Binary files a/docs/assets/remove-a-link.png and b/docs/assets/remove-a-link.png differ diff --git a/docs/assets/remove-block.png b/docs/assets/remove-block.png index 50546e33..68ceb825 100644 Binary files a/docs/assets/remove-block.png and b/docs/assets/remove-block.png differ diff --git a/docs/assets/rerun-pipeline.gif b/docs/assets/rerun-pipeline.gif new file mode 100644 index 00000000..e57b47d1 Binary files /dev/null and b/docs/assets/rerun-pipeline.gif differ diff --git a/docs/assets/run-a-core-pipeline-example.gif b/docs/assets/run-a-core-pipeline-example.gif new file mode 100644 index 00000000..15221031 Binary files /dev/null and b/docs/assets/run-a-core-pipeline-example.gif differ diff --git a/docs/assets/run-test-example.png b/docs/assets/run-test-example.png new file mode 100644 index 00000000..0a082633 Binary files /dev/null and b/docs/assets/run-test-example.png differ diff --git a/docs/assets/selected-executed-pipeline.png b/docs/assets/selected-executed-pipeline.png new file mode 100644 index 00000000..febe527a Binary files /dev/null and b/docs/assets/selected-executed-pipeline.png differ diff --git a/docs/assets/user-interface-preview.png b/docs/assets/user-interface-preview.png index 69df4273..a1d47dec 100644 Binary files a/docs/assets/user-interface-preview.png and b/docs/assets/user-interface-preview.png differ diff --git a/docs/assets/view-result-button.png b/docs/assets/view-result-button.png new file mode 100644 index 00000000..bf6a556d Binary files /dev/null and b/docs/assets/view-result-button.png differ diff --git a/docs/create-blocks.md b/docs/create-blocks.md index 86538bed..645ebe26 100644 --- a/docs/create-blocks.md +++ b/docs/create-blocks.md @@ -24,7 +24,7 @@ Parameter Blocks do not have any input nodes; instead, they have a UI element th the other hand, Compute Blocks have both input and output nodes. View Blocks have a small box on top that will visualize its input based on the implemented logic in that View Block. -Each Block, regardless of its type, is saved in a directory named after the Block in `frontend/core/blocks`. However, +Each Block, regardless of its type, is saved in a directory named after the Block in `frontend/core/blocks`. However, the content of the directory representing a Block varies depending on its type. ### Components of Parameter Blocks @@ -36,9 +36,9 @@ the structural information about a Block and is stored in `frontend/core/blocks/ A directory representing a Compute Block should contain five files: -1. `computations.py`: Includes (at least) `compute(args)` and `test(args)` functions. The `compute` function implements -the logic of the Block, while the `test` function is designed to test the compute function. The `compute` function -returns a dictionary with keys that match the output node names from the specs file. +1. `computations.py`: Includes (at least) `compute(args)` and `test(args)` functions. The `compute` function implements + the logic of the Block, while the `test` function is designed to test the compute function. The `compute` function + returns a dictionary with keys that match the output node names from the specs file. 2. `Dockerfile`: Describes the instructions to build a Docker image. This file should always include `COPY computations.py .`. @@ -49,16 +49,16 @@ returns a dictionary with keys that match the output node names from the specs f 5. `specs.json`: Includes the structural information about the Block. ### Components of View Blocks + The role of View Blocks is to create and display visualizations for the computed Block outputs. View Blocks have the same five components as the Compute Blocks; i.e., `computations.py`, `Dockerfile`, `requirements.txt`, `run_test.py`, and `specs.json`. Using View Blocks, we create HTML-formatted visualizations that will be visualized -on top of the View Block as the Pipeline runs. - +on top of the View Block as the Pipeline runs. ## Creating Blocks To create a Block, use a template Block listed on the sidebar. You can drag and drop a template Block, such as -`New Python`, and modify its components to build your own Block. +`Denoising`, and modify its components to build your own Block. ![drop-a-block.gif](assets%2Fdrop-a-block.gif) @@ -66,11 +66,13 @@ To create a Block, use a template Block listed on the sidebar. You can drag and To make any modifications, click on the `` icon to open the Block Editor sidebar on the right. -To modify the compute or view logic, simply edit the `compute` function. You can add as many lines of code as you like to the -`computations.py` file. +To modify the compute or view logic, simply edit the `compute` function. You can add as many lines of code as you like to the +`computations.py` file. + +[//]: # "You can also write a specific prompt to modify the code with the help of an AI agent, such as GPT-4." -[//]: # (You can also write a specific prompt to modify the code with the help of an AI agent, such as GPT-4.) ### Block Templates + The `computations.py` file should follow this template: ```python @@ -101,15 +103,15 @@ def compute(in1,...): """ Description """ - + html_template = """ - ... + ... """ - + unique_id = str(uuid.uuid4()) html_path = f"viz_{unique_id}.html" in1_str = json.dumps(in1) @@ -131,13 +133,14 @@ compute function should always be a dictionary with keys and values correspondin ### Saving Changes to Blocks -You can save the changes you made to a Block by compiling that Block inside the Block Editor. Just look for the "compile -Block files" button as shown below: +You can save the changes you made to a Block by compiling that Block inside the Block Editor. Just follow instructions as shown -![compile-run-block-editor.png](assets%2Fcompile-run-block-editor.png) +![compile-run-block-editor.gif](assets%2Fcompile-run-block-editor.gif) +Within the files section you can modify the code, and navigate to the `Test Block` section to compile and `run test` will run the `test` function inside the `computations.py` file -The "compile files and run Block" button will compile the changes and run the test functions specified by the `test` -function inside the `computations.py` file. + -To learn more about the Block Editor, please refer to our [user interface guide](user-interface.md/#block-editor). \ No newline at end of file +To learn more about the Block Editor, please refer to our [user interface guide](user-interface.md/#block-editor). diff --git a/docs/deploy-a-pipeline.md b/docs/deploy-a-pipeline.md index a299b6ee..f3770d5c 100644 --- a/docs/deploy-a-pipeline.md +++ b/docs/deploy-a-pipeline.md @@ -1,3 +1,25 @@ # Deploy a Pipeline -Coming soon, if you can't wait [get in touch with us](https://zetane.com/contact-us)! \ No newline at end of file +After the creation or modification of a pipeline, our system stores the information of every previously executed pipelines. After some pipelines have been executed you will be able to navigate through all past pipelines executions, deploy an existing pipeline, and retrieve code examples to call the API for the deployed pipeline. + +# Steps to Deploy a Pipeline + +1. **Access Past Executions** + To begin, you would need to navigate to the past executions section in the interface. This section will display a lis of all the previously executed pipeline, both succesful and unsuccesful executions. To navigate to past executions click on the button as shown below: + + ![executed-pipeline-button.png](assets%2Fexecuted-pipeline-button.png) + + After clicking the button you should be presented with the list of previously executed pipelines as shown below: + ![executed-pipelines.png](assets%2Fexecuted-pipelines.png) + +2. **Locate your desired pipeline** + After the table has been displayed you can now browse through all the previously executed pipelines, and find the one you wish to deploy. Each row would contain information such as the `pipeline ID`, `Date Created`, `Last Execution`, `Host`, `Deployment status`, and `# of executions`. and upon selecting a desired pipeline, you can have access to more information such as the `execution status` refer to the image shown below : + + ![selected-executed-pipeline.png](assets%2Fselected-executed-pipeline.png) + +3. **Deploy desired pipeline** + Once you have selected your desired pipeline, you can deploy it by simply clicking on the deploy button corresponding to it. The system would process the request, and update the deploy button to **Payload** + ![deploy-to-payload.gif](assets%2Fdeploy-to-payload.gif) +4. **Retrieve Code Examples** + After succesfully deploying your pipeline, you can click the **Payload** button, and this will display code examples demonstrating how to call the API for your pipeline. You can utilize these examples to have them integrated into your applications. + ![deployed-payload.png](assets%2Fdeployed-payload.png) diff --git a/docs/run-a-pipeline.md b/docs/run-a-pipeline.md index 594b55d9..bd173622 100644 --- a/docs/run-a-pipeline.md +++ b/docs/run-a-pipeline.md @@ -2,21 +2,21 @@ Prior to running a Pipeline, you need to ensure that all the necessary Blocks are in the workspace and the links between Blocks have been correctly established. Also, when applicable, inputs should be provided. If you have made any changes -to a Block, make sure to save and compile those changes before running the Pipeline. +to a Block, make sure to save and compile those changes before running the Pipeline. When everything is in place, run the Pipeline by clicking the `Run` button. -![run-a-pipeline-example.gif](assets%2Frun-a-pipeline-example.gif) +![run-a-core-pipeline-example.gif](assets%2Frun-a-core-pipeline-example.gif) -Once the Pipeline begins running, you can review the logs by clicking on the `Log` button. When you run a Pipeline for -the first time, Docker images are built. After that, the containers are executed without rebuilding if +Once the Pipeline begins running, you can review the logs by clicking on the `Log` button. When you run a Pipeline for +the first time, Docker images are built. After that, the containers are executed without rebuilding if no changes have been made. Therefore, the first time you build the Docker images usually takes more time. Upon running a Pipeline, a new directory named `history` is formed within the Pipeline directory under -`frontend/.cache/pipeline-${pipeline_id}`. Within this directory, there is a unique subdirectory generated +`frontend/.cache/pipeline-${pipeline_id}`. Within this directory, there is a unique subdirectory generated for each run, identified by a distinct UUID. Each of these subdirectories contains folders for `files` and `logs`, where output files and log files are stored, respectively. -Moreover, there's a `pipeline` folder including a `pipeline.json` file that outlines the Pipeline's structure +Moreover, there's a `pipeline` folder including a `pipeline.json` file that outlines the Pipeline's structure and connections between the Blocks. If the run is successful, a `results.json` file is created in the `results` -directory, capturing detailed information about the Pipeline execution, including Block inputs and outputs. \ No newline at end of file +directory, capturing detailed information about the Pipeline execution, including Block inputs and outputs. diff --git a/docs/run-your-first-pipeline.md b/docs/run-your-first-pipeline.md index a3be6c15..9577b3af 100644 --- a/docs/run-your-first-pipeline.md +++ b/docs/run-your-first-pipeline.md @@ -4,11 +4,11 @@ To experience ZetaForge in action, start by dragging and dropping a core pipelin Follow these step-by-step screenshots to help you run your first pipeline: -1) Load a Pipeline: -![canny-edge-pipeline.png](assets%2Fquick-start.gif) -2) Click Run -![click-run.png](assets%2Fclick-run.png) -3) View the logs: -![canny-edge-logs.png](assets%2Fcanny-edge-logs.png) -4) See the results: -![canny-edge-results.png](assets%2Fcanny-edge-results.png) +1. Load and run Pipeline: + ![run-a-core-pipeline-example.gif](assets%2Frun-a-core-pipeline-example.gif) + +2. View the results: + ![view-result-button.png](assets%2Fview-result-button.png) + +3. Display results: + ![canny-edge-mona-result.png](assets%2Fcanny-edge-mona-result.png) diff --git a/docs/user-interface.md b/docs/user-interface.md index 1065cff4..9f7c8421 100644 --- a/docs/user-interface.md +++ b/docs/user-interface.md @@ -1,6 +1,6 @@ # User Interface -ZetaForge comes with an easy-to-use User Interface (UI) that allows the developers to fully focus on the complexities +ZetaForge comes with an easy-to-use User Interface (UI) that allows the developers to fully focus on the complexities of their machine learning pipelines. Let's take a quick look at ZetaForge and its various UI functionalities. ## Basics @@ -9,12 +9,13 @@ The main window of the application appears as follows: ![user-interface-preview.png](assets%2Fuser-interface-preview.png) -On the left-hand side, you can find the Block Library that contains the predefined Core Blocks and Core Pipelines. +On the left-hand side, you can find the Block Library that contains the predefined Core Blocks and Core Pipelines. By hovering over the info icon, you can read the description of each Block or Pipeline. -You can use the search bar to filter Blocks and Pipelines by name. Also, if you need more space, you can close the +You can use the search bar to filter Blocks and Pipelines by name. Also, if you need more space, you can close the Block Library using the left arrow icon. ### Add or Remove Blocks/Pipelines + The black canvas is your workspace. You can add Blocks or Pipelines to your workspace by a simple drag-and-drop action. To remove a Block, right-click on the Block and click on the Close that appears on the top-right of the Block. @@ -29,7 +30,7 @@ Each Block can have input and output nodes, which are connected to the other Blo The links between the Blocks can be made by clicking on the source node and dragging the connection to the destination node. -If you change your mind and want to remove a link, right-click on the link and then click on the Close icon that +If you change your mind and want to remove a link, right-click on the link and then click on the Close icon that appears on top of the link to remove that link. ![remove-a-link.png](assets%2Fremove-a-link.png) @@ -38,106 +39,110 @@ appears on top of the link to remove that link. Saving and loading ZetaForge Blocks and Pipelines comes handy especially when you want to share your creations with other developers and team members. To save the current Blocks or Pipelines in your workspace, click on `File -> Save As` -and follow the instructions. +and follow the instructions. -If you have a Block or Pipeline saved on your machine and want to load that in your workspace, +If you have a Block or Pipeline saved on your machine and want to load that in your workspace, click on `File -> Load` and select `Block` or `Pipeline` depending on what you need to load. This would place the loaded Block or Pipeline into your workspace. ## Block Editor -The Block Editor comes with some interesting features that provide access to Block-level information and make your +The Block Editor comes with some interesting features that provide access to Block-level information and make your coding experience much smoother. It allows you -to view the components of a Block or modify them through the UI without having to leave the +to view the components of a Block or modify them through the UI without having to leave the platform and switch to a code editor. To open the Block Editor sidebar, click on the `` icon on a Block header. We will go over some of the Block Editor features next. ### View Block Components -To view the files associated with a Block, first, open the Block Editor. Inside the Block Editor, you can see the + +To view the files associated with a Block, first, open the Block Editor. Inside the Block Editor, you can see the Block id on top. You can make this window full-screen by clicking the Toggle Full Screen -button or close it using the +button or close it using the Close button. ![block-editor.png](assets%2Fblock-editor.png) -#### Workspace -Under the Workspace tab in the Block Editor, you can see the contents of the `computations.py` file for the selected -Block. To modify the code, click on the `Edit code` icon and start editing. Save the new version of your code by clicking -on the Save button and compile the code by clicking on the `Compile block files` icon. -You can compile and run the Block at the -same time by clicking the `Compile files and run block` icon. - -![compile-run-block-editor.png](assets%2Fcompile-run-block-editor.png) - - #### Files + As explained in detail in the [Block Components](create-blocks.md/#block-components) section, each Block consists of one or several -files, depending on its type. You can see these components under the `Files` tab. You can directly modify these files -and save your changes through the UI. Also, to import a folder or multiple files to your workspace, you can click on +files, depending on its type. You can see these components under the `Files` tab. You can directly modify these files +and save your changes through the UI. Also, to import a folder or multiple files to your workspace, you can click on Import Folder -Import Folder +Import Folder or Import Files Import Files icons. -You can call the `test` function in your `computations.py` file in a separate container to test your code before building -the Pipeline. To test your code, click on the Run Test -Run Test -button. +![import-files-folder-run-test.png](assets%2Fimport-files-folder.png) + + + +#### Specs -![import-files-folder-run-test.png](assets%2Fimport-files-folder-run-test.png) +The specs field contains details about the block, including its ID, description, inputs, outputs, and source. These details can also be modified. Refer to the image below for the `canny edge` block overview: -#### Logs +![canny-edge-specs.png](assets%2Fcanny-edge-specs.png) -Under the `Logs` tab, you can see the logs that are generated when Run Test executes. These are the Docker build logs and + + +#### Test Blocks + +You can call the `test` function in your `computations.py` file in a separate container to test your code before building +the Pipeline. To test your code, click on the Run Test +Run Test +button. + +![run-test-example.png](assets%2Frun-test-example.png) ## Handling Pipelines ### Run Pipelines -To run a Pipeline, you need to have at least one compute Block + +To run a Pipeline, you need to have at least one compute Block ([What is a compute Block?](create-blocks.md/#block-components)). Ensure you've established the correct links between the Blocks before running a Pipeline. When everything is all set, click on the `Run` button on the top of the main window to execute your Pipeline. ### Build Pipelines + If you made some changes to some of the Blocks and need to rebuild your Pipeline, you can click on the `Rebuild` button next to the `Run` button and all the Docker images will be built from scratch. ### Logs -When you run a Pipeline, a window pops up that contains the log that are being generated. You can read through the logs -for debugging purposes. Closing the logs window keeps your code running in the background. If you closed the logs but + +When you run a Pipeline, a window pops up that contains the log that are being generated. You can read through the logs +for debugging purposes. Closing the logs window keeps your code running in the background. If you closed the logs but need to head over to the log window again, simply click on the `Logs` button, and you should be able to see the window again. - ## Set Your API Keys + The prompt-based code generation feature need an active API key. If you wish to use the AI-assisted coding, navigate to `Settings -> API Keys` and enter your OpenAI API key. -> Note: The API key will not be accessible from the Blocks. If you need to use your API key inside a Block, you should -pass it using a Parameter Block. +> Note: The API key will not be accessible from the Blocks. If you need to use your API key inside a Block, you should +> pass it using a Parameter Block. ## Other Settings + If you need to debug something, you can go to `View -> Toggle Developer Tools` from the application top bar. You can switch themes between Dark Mode and Light Mode by clicking on `Settings -> Theme`. Try it now to see which one you are more comfortable with! - - - - diff --git a/docs/view-executed-pipelines.md b/docs/view-executed-pipelines.md new file mode 100644 index 00000000..4761acc0 --- /dev/null +++ b/docs/view-executed-pipelines.md @@ -0,0 +1,41 @@ +# View executed pipelines + +This guide explains how to view pipelines that have already been executed in the Zetaforge application. + +## Steps to View Executed Pipelines + +1. **Navigate to the Execution History**: + + - Open the Zetaforge application. + - Locate the **Pipelines** section in the user interface. + + ![executed-pipeline-button.png](assets%2Fexecuted-pipeline-button.png) + +2. **Browse Past Executions**: + + - In the Execution History view, you will see a list of previously executed pipelines. + - Each entry in the table represents a past pipeline execution and typically includes: + + - **Pipeline UUID**: The unique identifier for the pipeline. + - **Execution Date and Time**: When the pipeline was executed. + + ![executed-pipelines.png](assets%2Fexecuted-pipelines.png) + +3. **View Detailed Information**: + + - Click on a specific pipeline execution entry to view detailed information, including: + + - **Status**: The current state of the pipeline execution (e.g., succeded, failed). + + ![past-executions.png](assets%2Fpast-executions.png) + +4. **Deploy a Pipeline from History**: + - If you want to deploy a pipeline from the history: + - Click on the **Deploy** button next to the corresponding pipeline entry. + - Once deployed, a **Payload** button will appear. Click on it to view and copy API code examples for invoking the pipeline programmatically. +5. **Rerun a Pipeline from History**: + - If you want to rerun a previously executed pipeline: + - Click on the desired pipeline to view detailed information + - Click on the **View** button + - This would automatically reinstate the desired pipeline on the canvas, and you can rerun it by clicking on the **Run @localhost** button as seen below. + ![rerun-pipeline.gif](assets%2Frerun-pipeline.gif)