v0.2.6

0.2.6

Released on March 14, 2023.

All Changes

v0.2.5...v0.2.6

Key Features

• Enables registering a workflow without immediately triggering a run of that workflow. When calling publish_flow the run_now parameter can be set to False, which will tell the Aqueduct server to wait until the next scheduled (or triggered) run. In the interim, only the DAG's structure will be shown without any execution metadata.

Enhancements

• Adds error checking to ensure that a remote compute engine is paired with a remote artifact store.
• Adds error checking to ensure that a valid kubeconfig is provided when connecting Aqueduct to Kubernetes.
• Enables using ~ to refer to the home directory when specifying a path to AWS credentials or a kubeconfig.
• Adds icons to integration details views to indicate when an object store (AWS S3, GCS) are being used for metadata storage.
• Allows specifying the specific library version of CUDA when requesting GPU resources; v11.4.1 is the current default because that is what EKS clusters use by default.

Bugfixes

• Fixes bug where certain errors occurring during task launch weren't clearly surfaced in the Aqueduct stack.
• Fixes bug where, within a Python process, executing an operator in lazy mode precluded users from later executing it in eager mode (or vice versa).
• Fixes bug where notification count wasn't being shown.
• Fixes bug where the notice for an ongoing metadata migration would run off the end of the screen.
• Removes duplicate "History" header on metric details page.

All commits

• Fixes broken pre-merge tests by @kenxu95 in #1037
• Allows for Snowflake VARCHAR to be of the max size by @saurav-c in #1038
• Fix requirements issues on fresh aqueduct install by @likawind in #1032
• Implement the basic SDK and backend support for dynamic k8s integration by @cw75 in #1033
• Add UI support for connecting AWS integration by @cw75 in #1036
• Allows users to optionally specify snowflake role by @saurav-c in #1035
• Make dynamic k8s cluster resilient against server failures and race conditions by @cw75 in #1040
• [ENG-2273] Updates layout of workflow header by @vsreekanti in #1045
• Disable AWS integration registration from UI and SDK by @cw75 in #1053
• [1/5] Decouple exec env data model creation and conda env creation by @likawind in #1011
• ENG 2494 Fix a bug where metric graph dates are cutted off by @likawind in #1048
• Enable 'list operators on integration' for compute integrations by @likawind in #1044
• [ENG-2563] Remove dead & refactor confusing UI code by @vsreekanti in #1054
• ENG 2439 add engine and resource to metrics and checks by @likawind in #1049
• Fix mypy failure in GH actions by @likawind in #1059
• [ENG-2573] Updates DAG node design by @vsreekanti in #1057
• [1/2] Eng-2503: Refactor Conda Connection Status for more generic purposes by @Fanjia-Yan in #1041
• [2/2]Eng 2503: Connecting to Lambda Asynchronously by @Fanjia-Yan in #1042
• ENG-2170 Settings Update Toast and Navigation Updates by @agiron123 in #1058
• 0.2.5 Release by @saurav-c in #1062
• Add K8s integration tests to our periodic integration tests by @kenxu95 in #1046
• Do not require an explicit apikey field in test-credentials.yml by @kenxu95 in #1068
• Upgrade reactflow v10->v11, @reduxjs/toolkit by @eunice-chan in #1013
• Delete dynamic k8s cluster only when no pods are running by @cw75 in #1051
• Have our GH actions slack integration always tag the appropriate oncall by @kenxu95 in #1064
• Fix bug where compute integration can be registered without configuring cloud storage by @cw75 in #1066
• Fix bug where error occurred during job manager launch is not caught as operator error by @cw75 in #1065
• Adds backend support for changing storage layer back to local by @saurav-c in #1055
• Fix a test case that's expected to run only when using sqlite as data integration by @cw75 in #1074
• Check K8s config is valid during k8s integration validation by @cw75 in #1067
• Fix bug where calling an operator fixes it's execution mode by @cw75 in #1072
• Refactors storage error to include stack trace by @saurav-c in #1076
• Refactors lib/database to follow new errors convention by @saurav-c in #1081
• Fixes airflow workflow registration bugs by @saurav-c in #1082
• Get our credentials paths working with ~ by @cw75 in #1073
• ENG-2430: SX prop not picking up colors from theme by @agiron123 in #1075
• Make EKS cluster parameters configurable by @cw75 in #1052
• Allow registering AWS integration with custom config from the UI by @cw75 in #1061
• ENG-2610: Fix crash on workflow details page by @agiron123 in #1087
• ENG-1966 no indication on s3 integration details by @agiron123 in #1009
• [1/2] Allow user to register workflow without triggering by @likawind in #1043
• ENG 2410 improve metadata migration notice looking by @likawind in #1079
• ENG-2514 handle multipe engines in a workflow by @likawind in #1080
• Remove unnecessary history header by @hsubbaraj-spiral in #1084
• Fix bug with DatabricksJob preview by @hsubbaraj-spiral in #1083
• Added notebook for Spark Demo by @hsubbaraj-spiral in #1085
• Adds option for users to specify CUDA version by @hsubbaraj-spiral in #1071
• Updates gpu names in Makefile by @hsubbaraj-spiral in #1088
• Updates default workflow timeout to 48 hours by @agiron123 in #1089
• Release v0.2.6 by @agiron123 in #1090