From b442fbad177c1c166fa63837fca2829f95d8c347 Mon Sep 17 00:00:00 2001 From: Mecoli1219 Date: Thu, 17 Oct 2024 23:49:16 -0700 Subject: [PATCH 1/4] Jupyter notebook basic example Signed-off-by: Mecoli1219 --- examples/jupyter_notebook_basics/README.md | 4 + .../jupyter_notebook_basics.ipynb | 283 ++++++++++++++++++ 2 files changed, 287 insertions(+) create mode 100644 examples/jupyter_notebook_basics/README.md create mode 100644 examples/jupyter_notebook_basics/jupyter_notebook_basics.ipynb diff --git a/examples/jupyter_notebook_basics/README.md b/examples/jupyter_notebook_basics/README.md new file mode 100644 index 000000000..10c222fa5 --- /dev/null +++ b/examples/jupyter_notebook_basics/README.md @@ -0,0 +1,4 @@ +# Jupyter Notebook Basic Development Guide + +This guide will help you to understand the basics of how to develop Flyte tasks and workflows in Jupyter Notebook. +Please follow the instructions in the [Jupyter Notebook](jupyter_notebook_basics.ipynb) to understand the basics of Flyte development in Jupyter Notebook. diff --git a/examples/jupyter_notebook_basics/jupyter_notebook_basics.ipynb b/examples/jupyter_notebook_basics/jupyter_notebook_basics.ipynb new file mode 100644 index 000000000..df1adeea0 --- /dev/null +++ b/examples/jupyter_notebook_basics/jupyter_notebook_basics.ipynb @@ -0,0 +1,283 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# How to Develop and Run the Tasks and Workflows Remotely via Jupiter Notebook" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Before you start, make sure your flytekit version is higher than 1.14.0. If not, please upgrade it by running the following command:\n", + "```bash\n", + "!pip install flytekit==1.14.0\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 01 Setup the Interface to Connect to the Flyte Backend\n", + "\n", + "Before starting to develop and run the tasks and workflows remotely via Jupiter Notebook, let's setup the interface via [flytekit.remote](https://docs.flyte.org/projects/flytekit/en/latest/design/control_plane.html) to interact with the Flyte backend. \n", + "\n", + "**Make sure to set `interactive_mode_enabled` to `True` to support auto pickling and registering entities to backend from Jupyter Notebook environment.**" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
23:40:27.971008 WARNING  remote.py:230 - Jupyter notebook and interactive task  \n",
+       "                         support is still alpha.                                \n",
+       "
\n" + ], + "text/plain": [ + "\u001b[2;36m23:40:27.971008\u001b[0m\u001b[2;36m \u001b[0m\u001b[33mWARNING \u001b[0m remote.py:\u001b[1;36m230\u001b[0m - Jupyter notebook and interactive task \n", + "\u001b[2;36m \u001b[0m support is still alpha. \n" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "from flytekit.remote import FlyteRemote\n", + "from flytekit.configuration import Config\n", + "\n", + "# The `for_sandbox` method instantiates a connection to the demo cluster.\n", + "remote = FlyteRemote(\n", + " Config.for_sandbox(), \n", + " default_project=\"flytesnacks\", \n", + " default_domain=\"development\", \n", + " interactive_mode_enabled=True,\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 02 Develop the tasks & workflows" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Let's create a simple task and workflow to demonstrate how to develop and run the tasks and workflows remotely via Jupiter Notebook." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [], + "source": [ + "from flytekit import task, workflow\n", + "\n", + "@task\n", + "def hello(name: str) -> str:\n", + " return f\"Hello {name}!\"\n", + "\n", + "@task\n", + "def world(pre: str) -> str:\n", + " return f\"{pre} Welcome to the Jupyter Notebook!\"\n", + "\n", + "@workflow\n", + "def wf(name: str) -> str:\n", + " return world(pre=hello(name=name))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "When executing the task or workflow via `FlyteRemote` interface, the related entities will be automatically registered to the Flyte backend. You can navigate to the Flyte Console via the URL to see the progress and outputs of the executions, or you can use `remote.wait()` to wait for the execution to complete. This may take a few seconds to minutes depending on the size of the task and workflow." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "http://localhost:30080/console/projects/flytesnacks/domains/development/executions/alnrr5cmh4gbrbfcvp8k\n", + "{'o0': 'Hello Flyte!'}\n" + ] + } + ], + "source": [ + "# Execute the task\n", + "out = remote.execute(hello, inputs={\"name\": \"Flyte\"})\n", + "\n", + "# This will print the URL to the console\n", + "print(remote.generate_console_url(out))\n", + "\n", + "# Wait for the task to complete\n", + "out = remote.wait(out)\n", + "\n", + "# Print the outputs\n", + "print(out.outputs)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "If you are not interested in the URL to the Flyte Console and want the execution wait directly to complete, you can set `wait=True` in the `remote.execute()` method." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{'o0': 'Hello world!'}\n" + ] + } + ], + "source": [ + "# Execute the workflow and wait for it to complete\n", + "out = remote.execute(hello, inputs={\"name\": \"world\"}, wait=True)\n", + "\n", + "# Print the outputs\n", + "print(out.outputs)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Similarly, we can also execute the workflows remotely and extract the outputs." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{'o0': 'Hello world! Welcome to the Jupyter Notebook!'}\n" + ] + } + ], + "source": [ + "# Execute the workflow and wait for it to complete\n", + "out = remote.execute(wf, inputs={\"name\": \"world\"}, wait=True)\n", + "\n", + "# Print the outputs\n", + "print(out.outputs)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "If you repeat the execution, it will not re-register the entities to the backend, instead it will reuse the existing entities. However, if you update and run the cells that define the tasks or workflows, it will register the updated entities to the backend when you execute them.\n", + "\n", + "Feel free to update the tasks and workflows definition and see how it works!" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 03 Develop the Workflow Contains `map_task`" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "`map_task` is also supported in the remote execution. You can define the `map_task` in the workflow and execute it remotely." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [], + "source": [ + "from flytekit import task, workflow, map_task\n", + "from functools import partial\n", + "\n", + "@task\n", + "def fn(x: int, y: int) -> int:\n", + " return x + y + 2\n", + "\n", + "@workflow\n", + "def workflow_with_maptask(data: list[int], y: int) -> list[int]:\n", + " partial_fn = partial(fn, y=y)\n", + " return map_task(partial_fn)(x=data)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Let's execute the workflow remotely and extract the outputs." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{'o0': [7, 8, 9]}\n" + ] + } + ], + "source": [ + "# Execute the workflow with map_task and wait for it to complete\n", + "out = remote.execute(workflow_with_maptask, inputs={\"data\": [1, 2, 3], \"y\": 4}, wait=True)\n", + "\n", + "# Print the outputs\n", + "print(out.outputs)" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": ".venv", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.3" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} From 712019da187b186bb0f8e629b5f24ea05ba8f02d Mon Sep 17 00:00:00 2001 From: Mecoli1219 Date: Sun, 27 Oct 2024 22:30:50 -0700 Subject: [PATCH 2/4] Update jupyter notebook & fix doc Signed-off-by: Mecoli1219 --- .../jupyter_notebook_basics/__init__.py | 0 .../jupyter_notebook_basics.ipynb | 40 +++++++++++-------- 2 files changed, 23 insertions(+), 17 deletions(-) create mode 100644 examples/jupyter_notebook_basics/jupyter_notebook_basics/__init__.py rename examples/jupyter_notebook_basics/{ => jupyter_notebook_basics}/jupyter_notebook_basics.ipynb (84%) diff --git a/examples/jupyter_notebook_basics/jupyter_notebook_basics/__init__.py b/examples/jupyter_notebook_basics/jupyter_notebook_basics/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/examples/jupyter_notebook_basics/jupyter_notebook_basics.ipynb b/examples/jupyter_notebook_basics/jupyter_notebook_basics/jupyter_notebook_basics.ipynb similarity index 84% rename from examples/jupyter_notebook_basics/jupyter_notebook_basics.ipynb rename to examples/jupyter_notebook_basics/jupyter_notebook_basics/jupyter_notebook_basics.ipynb index df1adeea0..02097ea25 100644 --- a/examples/jupyter_notebook_basics/jupyter_notebook_basics.ipynb +++ b/examples/jupyter_notebook_basics/jupyter_notebook_basics/jupyter_notebook_basics.ipynb @@ -25,7 +25,7 @@ "\n", "Before starting to develop and run the tasks and workflows remotely via Jupiter Notebook, let's setup the interface via [flytekit.remote](https://docs.flyte.org/projects/flytekit/en/latest/design/control_plane.html) to interact with the Flyte backend. \n", "\n", - "**Make sure to set `interactive_mode_enabled` to `True` to support auto pickling and registering entities to backend from Jupyter Notebook environment.**" + "If you are running in Jupyter Notebook, it should automatically enable the interactive mode. However, you can also enable it explicitly by setting `interactive_mode_enabled` to `True`. " ] }, { @@ -36,12 +36,12 @@ { "data": { "text/html": [ - "
23:40:27.971008 WARNING  remote.py:230 - Jupyter notebook and interactive task  \n",
+       "
21:39:52.768159 WARNING  remote.py:273 - Jupyter notebook and interactive task  \n",
        "                         support is still alpha.                                \n",
        "
\n"
       ],
       "text/plain": [
-       "\u001b[2;36m23:40:27.971008\u001b[0m\u001b[2;36m \u001b[0m\u001b[33mWARNING \u001b[0m remote.py:\u001b[1;36m230\u001b[0m - Jupyter notebook and interactive task  \n",
+       "\u001b[2;36m21:39:52.768159\u001b[0m\u001b[2;36m \u001b[0m\u001b[33mWARNING \u001b[0m remote.py:\u001b[1;36m273\u001b[0m - Jupyter notebook and interactive task  \n",
        "\u001b[2;36m                \u001b[0m         support is still alpha.                                \n"
       ]
      },
@@ -58,7 +58,7 @@
     "    Config.for_sandbox(), \n",
     "    default_project=\"flytesnacks\", \n",
     "    default_domain=\"development\", \n",
-    "    interactive_mode_enabled=True,\n",
+    "    interactive_mode_enabled=True,  # This is not necessary if you are in a notebook.\n",
     ")"
    ]
   },
@@ -113,23 +113,23 @@
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "http://localhost:30080/console/projects/flytesnacks/domains/development/executions/alnrr5cmh4gbrbfcvp8k\n",
+      "http://localhost:30080/console/projects/flytesnacks/domains/development/executions/ah5cm5s482w9fl5jzz7c\n",
       "{'o0': 'Hello Flyte!'}\n"
      ]
     }
    ],
    "source": [
     "# Execute the task\n",
-    "out = remote.execute(hello, inputs={\"name\": \"Flyte\"})\n",
+    "exe = remote.execute(hello, inputs={\"name\": \"Flyte\"})\n",
     "\n",
     "# This will print the URL to the console\n",
-    "print(remote.generate_console_url(out))\n",
+    "print(exe.execution_url)\n",
     "\n",
     "# Wait for the task to complete\n",
-    "out = remote.wait(out)\n",
+    "exe = exe.wait(poll_interval=1)\n",
     "\n",
     "# Print the outputs\n",
-    "print(out.outputs)"
+    "print(exe.outputs)"
    ]
   },
   {
@@ -154,10 +154,10 @@
    ],
    "source": [
     "# Execute the workflow and wait for it to complete\n",
-    "out = remote.execute(hello, inputs={\"name\": \"world\"}, wait=True)\n",
+    "exe = remote.execute(hello, inputs={\"name\": \"world\"}, wait=True)\n",
     "\n",
     "# Print the outputs\n",
-    "print(out.outputs)"
+    "print(exe.outputs)"
    ]
   },
   {
@@ -182,17 +182,20 @@
    ],
    "source": [
     "# Execute the workflow and wait for it to complete\n",
-    "out = remote.execute(wf, inputs={\"name\": \"world\"}, wait=True)\n",
+    "exe = remote.execute(wf, inputs={\"name\": \"world\"})\n",
+    "\n",
+    "# Wait for the task to complete\n",
+    "exe = exe.wait(poll_interval=1)\n",
     "\n",
     "# Print the outputs\n",
-    "print(out.outputs)"
+    "print(exe.outputs)"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "If you repeat the execution, it will not re-register the entities to the backend, instead it will reuse the existing entities. However, if you update and run the cells that define the tasks or workflows, it will register the updated entities to the backend when you execute them.\n",
+    "If you repeat the execution, it will not re-register the entities to the backend, instead it will reuse the existing entities. However, if you re-execute the cells that define the tasks or workflows, it will register the updated entities to the backend when you execute them.\n",
     "\n",
     "Feel free to update the tasks and workflows definition and see how it works!"
    ]
@@ -252,7 +255,10 @@
    ],
    "source": [
     "# Execute the workflow with map_task and wait for it to complete\n",
-    "out = remote.execute(workflow_with_maptask, inputs={\"data\": [1, 2, 3], \"y\": 4}, wait=True)\n",
+    "out = remote.execute(workflow_with_maptask, inputs={\"data\": [1, 2, 3], \"y\": 4})\n",
+    "\n",
+    "# Wait for the task to complete\n",
+    "out = out.wait(poll_interval=1)\n",
     "\n",
     "# Print the outputs\n",
     "print(out.outputs)"
@@ -261,7 +267,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": ".venv",
+   "display_name": "flytekit3.12",
    "language": "python",
    "name": "python3"
   },
@@ -275,7 +281,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.12.3"
+   "version": "3.12.0"
   }
  },
  "nbformat": 4,

From 792b27c771a310d66ba7c295b75436a2dd58eff1 Mon Sep 17 00:00:00 2001
From: Mecoli1219 
Date: Mon, 28 Oct 2024 00:07:57 -0700
Subject: [PATCH 3/4] Update readme

Signed-off-by: Mecoli1219 
---
 examples/jupyter_notebook_basics/README.md    | 24 +++++++++++++++++--
 ...yter_notebook_basics.ipynb => basic.ipynb} |  2 +-
 2 files changed, 23 insertions(+), 3 deletions(-)
 rename examples/jupyter_notebook_basics/jupyter_notebook_basics/{jupyter_notebook_basics.ipynb => basic.ipynb} (99%)

diff --git a/examples/jupyter_notebook_basics/README.md b/examples/jupyter_notebook_basics/README.md
index 10c222fa5..a164a5bbb 100644
--- a/examples/jupyter_notebook_basics/README.md
+++ b/examples/jupyter_notebook_basics/README.md
@@ -1,4 +1,24 @@
+(interactive-mode)=
+
 # Jupyter Notebook Basic Development Guide
 
-This guide will help you to understand the basics of how to develop Flyte tasks and workflows in Jupyter Notebook.
-Please follow the instructions in the [Jupyter Notebook](jupyter_notebook_basics.ipynb) to understand the basics of Flyte development in Jupyter Notebook.
+```{eval-rst}
+.. tags:: Integration, Jupyter
+```
+
+```{image} https://img.shields.io/badge/Blog-Interactive-blue?style=for-the-badge
+:target: https://hackmd.io/@1E0FEh2MS_OpXGUAjEFIOQ/ByTTT821Jl
+:alt: Interactive Mode for Jupyter Notebook Blog Post
+```
+
+Jupyter Notebooks revolutionize workflow development by combining code execution, documentation, and visualization in one interactive environment. Through Flyte's integration, you can develop and test your workflows remotely while maintaining production-ready scalability—eliminating the gap between development and deployment.
+
+## How to Use the Interactive Mode?
+
+Interactive mode is a feature included in [FlyteRemote](https://docs.flyte.org/en/latest/api/flytekit/design/control_plane.html) aiming at supporting entities (tasks, workflows, launch plans, etc.) registration and execution within interactive environment like Jupyter Notebook. To use the interactive mode, you just need to create a FlyteRemote object with `interactive_mode_enabled=True` to interact with Flyte backend. Then you are free to go!
+
+For detailed examples and usage patterns, refer to the following guide:
+
+```{auto-examples-toc}
+basic
+```
diff --git a/examples/jupyter_notebook_basics/jupyter_notebook_basics/jupyter_notebook_basics.ipynb b/examples/jupyter_notebook_basics/jupyter_notebook_basics/basic.ipynb
similarity index 99%
rename from examples/jupyter_notebook_basics/jupyter_notebook_basics/jupyter_notebook_basics.ipynb
rename to examples/jupyter_notebook_basics/jupyter_notebook_basics/basic.ipynb
index 02097ea25..8a6d89e26 100644
--- a/examples/jupyter_notebook_basics/jupyter_notebook_basics/jupyter_notebook_basics.ipynb
+++ b/examples/jupyter_notebook_basics/jupyter_notebook_basics/basic.ipynb
@@ -4,7 +4,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# How to Develop and Run the Tasks and Workflows Remotely via Jupiter Notebook"
+    "# Basic Interactive Mode Usage in Jupyter Notebook Environment"
    ]
   },
   {

From 85966d139a9a3ccdf3323124563960fdd5f5844a Mon Sep 17 00:00:00 2001
From: Mecoli1219 
Date: Mon, 28 Oct 2024 12:00:01 -0700
Subject: [PATCH 4/4] Move to basics

Signed-off-by: Mecoli1219 
---
 .../basics/basic_interactive_mode.ipynb}      |  0
 examples/jupyter_notebook_basics/README.md    | 24 -------------------
 .../jupyter_notebook_basics/__init__.py       |  0
 3 files changed, 24 deletions(-)
 rename examples/{jupyter_notebook_basics/jupyter_notebook_basics/basic.ipynb => basics/basics/basic_interactive_mode.ipynb} (100%)
 delete mode 100644 examples/jupyter_notebook_basics/README.md
 delete mode 100644 examples/jupyter_notebook_basics/jupyter_notebook_basics/__init__.py

diff --git a/examples/jupyter_notebook_basics/jupyter_notebook_basics/basic.ipynb b/examples/basics/basics/basic_interactive_mode.ipynb
similarity index 100%
rename from examples/jupyter_notebook_basics/jupyter_notebook_basics/basic.ipynb
rename to examples/basics/basics/basic_interactive_mode.ipynb
diff --git a/examples/jupyter_notebook_basics/README.md b/examples/jupyter_notebook_basics/README.md
deleted file mode 100644
index a164a5bbb..000000000
--- a/examples/jupyter_notebook_basics/README.md
+++ /dev/null
@@ -1,24 +0,0 @@
-(interactive-mode)=
-
-# Jupyter Notebook Basic Development Guide
-
-```{eval-rst}
-.. tags:: Integration, Jupyter
-```
-
-```{image} https://img.shields.io/badge/Blog-Interactive-blue?style=for-the-badge
-:target: https://hackmd.io/@1E0FEh2MS_OpXGUAjEFIOQ/ByTTT821Jl
-:alt: Interactive Mode for Jupyter Notebook Blog Post
-```
-
-Jupyter Notebooks revolutionize workflow development by combining code execution, documentation, and visualization in one interactive environment. Through Flyte's integration, you can develop and test your workflows remotely while maintaining production-ready scalability—eliminating the gap between development and deployment.
-
-## How to Use the Interactive Mode?
-
-Interactive mode is a feature included in [FlyteRemote](https://docs.flyte.org/en/latest/api/flytekit/design/control_plane.html) aiming at supporting entities (tasks, workflows, launch plans, etc.) registration and execution within interactive environment like Jupyter Notebook. To use the interactive mode, you just need to create a FlyteRemote object with `interactive_mode_enabled=True` to interact with Flyte backend. Then you are free to go!
-
-For detailed examples and usage patterns, refer to the following guide:
-
-```{auto-examples-toc}
-basic
-```
diff --git a/examples/jupyter_notebook_basics/jupyter_notebook_basics/__init__.py b/examples/jupyter_notebook_basics/jupyter_notebook_basics/__init__.py
deleted file mode 100644
index e69de29bb..000000000