feat: python workflows #892

j03-dev · 2024-10-28T05:18:47Z

Migration notes

...

The change comes with new or modified tests
Hard-to-understand functions have explanatory comments
End-user documentation is updated to reflect the change

linear · 2024-10-28T05:18:50Z

michael-0acf4 · 2024-10-28T09:23:40Z

src/typegate/src/runtimes/substantial/worker.ts

+          .then((wfResult: unknown) => {
+            self.postMessage(
+              Ok(
+                Msg(
+                  type,
+                  {
+                    kind: "SUCCESS",
+                    result: wfResult,
+                    run: runCtx!.getRun(),
+                    schedule,
+                  } satisfies WorkflowResult,
+                ),
+              ),
+            );
+          })
+          .catch((wfException: unknown) => {
+            self.postMessage(
+              Ok(
+                Msg(
+                  type,
+                  {
+                    kind: "FAIL",
+                    result: wfException instanceof Error
+                      ? wfException.message
+                      : JSON.stringify(wfException),
+                    exception: wfException instanceof Error
+                      ? wfException
+                      : undefined,
+                    run: runCtx!.getRun(),
+                    schedule,
+                  } satisfies WorkflowResult,
+                ),
+              ),
+            );
+          });


I think you should use the same then/catch blocks for both. This is the basis of the interrupt logic, so it must be identical for each languages.

michael-0acf4 · 2024-10-28T09:28:34Z

src/typegate/src/runtimes/substantial/worker.ts

+          const result = Meta.python.executePython({
+            python_module_path: modulePath,
+            python_function_name: functionName,
+            executing_context: runCtx,


Curious on how you will tackle the context methods. Will you try to unify it for both languages or.. ?

michael-0acf4 · 2024-10-28T09:30:10Z

src/typegate/engine/Cargo.toml

@@ -68,6 +69,9 @@ bytes.workspace = true
 protobuf.workspace = true
 protobuf-json-mapping.workspace = true

+# python
+pyo3 = { workspace = true, features = ["extension-module"] }


extension-module

nice!

By the way, which python version are you experimenting on this?
Is there a way to lock the Python version directly here?

michael-0acf4 · 2024-10-28T10:09:14Z

Suggestion:
Unifying the context for both languages into pure Rust should now be doable since we can use Meta inside workers (solves for the deno case), the python one will just use the rust implementation directly. The challenge would be on the hostcall.

zifeo · 2024-10-28T12:29:54Z

src/typegate/engine/src/python.rs

+    execution_status: String,
+}
+
+fn convert_json_to_python_object(py: Python, json_value: &Value) -> PyResult<PyObject> {


isn't pyo3 already providing such function?

michael-0acf4 · 2024-11-04T10:58:13Z