v0.1.10

Released on January 17, 2023

Enhancements

• For workflows that are triggered at the end of other workflows, we now allow changing the triggering workflow from the UI's workflow settings dialog.
• Differentiate keys and values better on the UI; adds the use of different colored text to make it clear which is the key and which is the value when showing, for example, metric and check values.
• Improves presentation of non-success states of metrics and checks on workflow DAG. Rather than leaving the nodes empty as before, they now include icons that demonstrate the execution state (failed, pending, canceled).
• Adds redesigned search interface to workflow and data list pages. The search bar itself has been reduced in size, and a sort functionality has been added that allows users to select a column by which to sort the view.
• Orders integrations alphabetically on the integrations page to make them easier to find.

Bugfixes

• Fixes bug where operator & artifact statuses were missing from details pages.
• Fixes bug where the header breadcrumbs did not show the title of the workflow on the metric details page.

All Changes

v0.1.9...v0.1.10

All commits

• Fix v0.1.9 release log from Jan 9 -> Jan 10 by @likawind in #877
• Removes erroring step from periodic integration test by @hsubbaraj-spiral in #878
• ENG-2183 Show Correct Operator/Artifact Status by @agiron123 in #875
• Upgrade to Go 1.19 by @hsubbaraj-spiral in #864
• ENG 1765 Unify table appearance by @eunice-chan in #858
• [1/3] Adds Databricks to UI by @hsubbaraj-spiral in #815
• Eng 2125 differentiate key values better by @agiron123 in #874
• [1/2] Refactor Integration Tests in preparation for S3 Data Integration Tests by @kenxu95 in #879
• ENG 2173 Metric Details Page does not show Workflow Name in Header by @eunice-chan in #884
• Eng 2058 Fill in empty metrics checks for cancelled, failed, and pending statuses by @eunice-chan in #828
• ENG-2129: Reduces visual footprint of search bar by @vsreekanti in #846
• [2/2] Add S3 Data Integration Tests by @kenxu95 in #880
• Order integrations alphabetically on integrations page by @kenxu95 in #886
• Remove SageMaker from UI build by @cw75 in #887
• Add release scripts for helm updates and doc updates by @likawind in #881
• Add ability to connect to integrations from the SDK by @kenxu95 in #882
• [1/2] Update SDK Integration Tests to run automagicky against various data integrations by @kenxu95 in #883
• Eng 2156 qa workflow description is out of date by @likawind in #889
• Adds UI support for changing the source workflow by @saurav-c in #888
• Improve error message for integrations that are in-use by @kenxu95 in #891
• Ready for release 0.1.10 by @cw75 in #892

v0.1.9

Released on January 10, 2023.

Key Features

• As of this release, Aqueduct has usage tracking. Usage tracking is fully anonymized and captures API routes, performance data, and error rates without revealing any specifics of your machine or deployment. For more details, check out our documentation.
• We now support cascading workflow triggers, which means a workflow can trigger another one at the end of its execution. You can specify that in our python SDK.

Enhancements

• Makes the artifact, check, metric, and operator details pages full width.
• Shows the Aqueduct version number on the UI navigation bar.
• Hides previews when artifacts are canceled.
• Hides parameters in status bar.

Bugfixes

• Fixes a number of UI bugs:
  • Resets workflow settings dialog content after close.
  • Aligns margins on right side of workflow details page.
  • Removes vestigial popover to access settings page.
  • Addresses regression where a workflow's saved objects were not being shown prior to workflow deletion.
  • Aligns the width of metric and check history items.
  • Updates the metadata views (workflows & data list pages) to differentiate table headers from metadata rows.
  • Persists the number of rows shown per-page on metadata views between page refreshes.

All Changes

v0.1.8...v0.1.9

All commits

• Eng 1911 refactor schema version collection by @eunice-chan in #818
• Eng 2108 bug fix margin on right side of workflow by @eunice-chan in #827
• ENG-1857 Resets workflow settings dialog changes on close if changes are not saved. by @agiron123 in #830
• Eng 2123 tweak text to expand-collapse checks by @eunice-chan in #826
• Fullwidth artifact, check, metric pages by @eunice-chan in #829
• ENG-2130 Remove Settings Account Popover by @agiron123 in #831
• Fix UI bug where workflow saved objects were not being displayed when the workflow was deleted. by @kenxu95 in #832
• ENG-2121: Use dark logo for Aqueduct engine. by @agiron123 in #838
• ENG-2136 Make History List Item Width Consistent by @agiron123 in #839
• [1/3] Run SDK Integration Tests against S3 [Backend Bugfix] by @kenxu95 in #833
• [3/3] Run SDK Integration Tests against S3 [UI Fix] by @kenxu95 in #835
• Eng 2137 make integrations page full width by @eunice-chan in #840
• Updates README to match latest website copy by @vsreekanti in #843
• ENG-2176 Show version number on bottom of navigation bar. by @agiron123 in #841
• Implement basic usage stats collection and reporting by @cw75 in #837
• ENG-2128 update table header format by @eunice-chan in #842
• Fix typo in readme by @kenxu95 in #860
• ENG-1880 Hide preview when artifact is cancelled. by @agiron123 in #844
• ENG-2158 Persist number of rows between page refreshes on workflows page and data page tables. by @agiron123 in #854
• [1/3] Refactor SDK Integration Tests into subdirectory with shared sibling directory by @kenxu95 in #848
• Cache server env for usage stats collection by @cw75 in #857
• [3/3] Update the github actions to run the two SDK integration test suites by @kenxu95 in #850
• ENG-1961 Hide Parameters in Workflow Status Bar by @agiron123 in #852
• Adds UI support for showing the source workflow in the settings by @saurav-c in #855
• ENG-2172 CodeBlock Storybook Story by @agiron123 in #865
• [1/3] Adds SourceID field to Workflow.Schedule column by @saurav-c in #861
• Changes target workflow trigger to manual when source workflow is deleted by @saurav-c in #868
• Don't panic when error sending stats to Loki by @cw75 in #869
• Adds unit tests and integration tests for cycle detection in cascading workflows by @saurav-c in #871
• Prevents cycles in cascading workflows by @saurav-c in #872
• Eng 2127 tweak cell padding to improve table spacing by @eunice-chan in #853
• Bugfix: Fix table row height by @eunice-chan in #873
• Improve the Integration Tests README by @kenxu95 in #867
• preparing for release 0.1.9 by @likawind in #876

v0.1.8

Released on December 20, 2022.

Enhancements

• Allows user to set compute engine in the operator's decorator.
• Reduces the number of significant figures for metrics on data list page and workflows list page to
  improve readability.

Bugfixes

• Fixes a bug where the UI shows data section for compute integrations.
• Fixes a bug where previewing Mongo collection crashes.

All Changes

v0.1.7...v0.1.8

All commits

• Eng 2022 implement operator repo additional tests by @eunice-chan in #771
• Add ability to set engine in the operator decorator by @kenxu95 in #801
• [UI] Add engine_config field to operator spec by @kenxu95 in #803
• Eng 1984 refactor execution env collection into model-repo by @eunice-chan in #798
• Eng 1984 integrate execution env refactor by @eunice-chan in #806
• execution env Id -> ID by @eunice-chan in #811
• Fixes bug with engines on SDK by @hsubbaraj-spiral in #813
• Refactor job manager interface to always return a JobError by @kenxu95 in #810
• ENG-2091: Fix linter errors on UI by @agiron123 in #814
• ENG-2124: Reduce sigfigs on the metrics preview in data list page and workflows list page by @agiron123 in #823
• Eng 2104 [1/2][backend] fix a bug where mongo collection preview crashes by @likawind in #820
• ENG 2061 Remove data section for compute integrations by @eunice-chan in #824
• [1/2] Make data_integration terminology consistent in the SDK integration tests by @kenxu95 in #819
• Release version 0.1.8 by @kenxu95 in #825

v0.1.7

Released on December 14, 2022.

Bugfixes

• Fixes a bug where the Aqueduct installation script fails if the user doesn't have conda installed.

All Changes

v0.1.6...v0.1.7

All commits

• Eng 2063 implement operator_result repo additional tests by @eunice-chan in #791
• Fix aqueduct bin script to only run conda update if the user has conda installed by @cw75 in #807
• Update Gitbook doc generation script by @cw75 in #805
• Ready for release 0.1.7 by @cw75 in #808

v0.1.6

Released on December 13, 2022.

Key Features

• Introduces new table views on the workflows and data pages that show rich
  metadata at a glance, including workflow and artifact status, data types,
  and associated metrics and checks!
• Adds support for integrating with conda. Once the user registers conda integration through the UI,
  Aqueduct will create conda environments to run any newly created workflows to provide better
  Python version and dependency management.

Enhancements

• Introduces a new save API; now, to save an artifact, users can write the
  following. The original table.save() syntax still works but will be
  deprecated in a future release.

db.save(table, 'my_new_table', update_mode='replace')

• Disallows creating multiple integrations with the same name.

Bugfixes

• Fixes a bug where unused integration couldn't be deleted if historical
  workflow runs were associated with it.
• Fixes a bug where logs weren't being displayed on operator details page.
• Fixes a bug where saving multiple pieces of data to the same database would
  cause the workflow UI to crash.
• Fixes a bug where calling a metric or check with no inputs didn't raise a
  client-side error.
• Fixes a bug where metric history & graph was not sorted by time.
• Fixes a bug where where every click into a workflow DAG node reset the DAG
  visualization.
• Fixes a number of bugs that caused no notifications to be displayed on the
  UI.

All Changes

v0.1.5...v0.1.6

All commits

• Fix syntax in periodic integration tests by @kenxu95 in #741
• Removes auth0ID from ListWorkflow response by @saurav-c in #756
• Eng 2018 implement artifact repo by @eunice-chan in #733
• Eng 1978 Implement APIs for syncing env to DB by @likawind in #698
• Fix UI check for deleting integration by @likawind in #759
• [ENG-2015] Fixes bug where logs weren't displayed on operator details by @vsreekanti in #749
• update bug report template by @cw75 in #766
• Fixes bug where multiple save operators to the same integration crashes the UI by @kenxu95 in #768
• Eng 2021 implement notification repo by @eunice-chan in #734
• Eng 2019 implement artifact result repo by @eunice-chan in #757
• Eng 2025 restrict delete integration to check for only active workflows by @likawind in #760
• Eng 2020 implement integration repo by @eunice-chan in #735
• [1/N] Replaces user reader and writer with repo by @saurav-c in #681
• [2/N] Replaces watcher DB readers/writers with repo by @saurav-c in #747
• Fix error message when check or metric is called with no inputs by @Fanjia-Yan in #763
• [3/N] Replaces dag result DB reader/writer with repo by @saurav-c in #751
• [4/N] Replaces DAG DB readers/writers with repo by @saurav-c in #752
• Runs DB integration tests on github action by @saurav-c in #780
• [5/N] Replaces workflow DB reader/writer with repo by @saurav-c in #754
• Replaces DAGEdge DB reader/writer with repo (#755) by @saurav-c in #782
• [7/N] Replaces notification DB reader/writer with repo by @saurav-c in #772
• [8/N] Replaces artifact DB reader/writer with repo by @saurav-c in #773
• [9/N] Replaces artifact result DB reader/writer with repo by @saurav-c in #775
• [10/N] Implements operator repo by @saurav-c in #758
• Bumps schema version to 21 by @saurav-c in #783
• [13/N] Replaces operator DB reader/writer with repo by @saurav-c in #777
• [14/N] Replaces integration DB reader/writer with repo by @saurav-c in #778
• [15/N] Replaces custom queries with repos by @saurav-c in #779
• Replaces remaining operator and workflow DB readers with repo by @saurav-c in #784
• Deletes unused code from lib/collections by @saurav-c in #788
• Fix bug where metrics graph and history is not sorted by time by @Fanjia-Yan in #753
• Fixes bug where canvas position is reset after clicking into a node. by @agiron123 in #785
• [SDK] Update our Save API to be integration.save() by @kenxu95 in #769
• Fixes a number of minor functionality regressions around notifications by @vsreekanti in #764
• Include Python type into artifact versions route's response by @cw75 in #787
• Refactors the SDK code into layers by @kenxu95 in #781
• Eng 2082 add metrics to get_artifact_versions route by @likawind in #789
• Update example notebooks to use new Save API by @kenxu95 in #794
• Eng 1967 unique integration names by @agiron123 in #786
• Fix SDK imports for CheckSeverity and LoadUpdateMode by @likawind in #795
• Eng 2064 automate workflows creation in manual UI testings by @likawind in #797
• Update house price notebook to allow configure server ip by @cw75 in #799
• NEW Workflows List UI and Data List UI by @agiron123 in #774
• Fix bug where delete confirmation dialog is massive by @Fanjia-Yan in #796
• Update QA readme by @cw75 in #800
• Bugfixes: Workflows Page and Data Page by @agiron123 in #802
• Ready for release 0.1.6 by @cw75 in #804

v0.1.5

Released on November 29, 2022.

Key Features

• Enables operators running on Kubernetes to access GPUs and set RAM and CPU
  requirements. Note that using a GPU requires your Kubernetes cluster to
  already have GPU machines attached. See our
  documentation for more details.

@op(resources={'num_cpus': 2, 'memory': '5Gb', 'gpu_resource_name': 'nvidia.com/gpu'})
def my_operator_with_many_resources():
  return 1

• Similarly, functions running on AWS Lambda can have memory requirement
  set using the syntax above; AWS Lambda does not support setting CPU requirement
  and it does not support GPUs.

Enhancements

• Enables operator previews to execute using different integrations, including
  using the resource constraints described above.
• Allows for the execution engine to be set globally for a client instance. See
  more details
  here:

aq.global_config({'engine': 'my_k8s_integration'})

Bugfixes

• Fixes bug where a Kubernetes pod that ran out of memory would fail silently.

All Changes

v0.1.4...v0.1.5

All commits

• Add ability to configure memory and number of CPUs on K8s backend by @kenxu95 in #701
• Fixes lint warning on version_selector component by @agiron123 in #708
• ENG-1870: Remove unused code from frontend by @agiron123 in #712
• Handle OOM exceptions gracefully in Kubernetes by @kenxu95 in #714
• Enables GPUs on K8s engine by @hsubbaraj-spiral in #717
• Enable Preview Using Different Engines by @Fanjia-Yan in #696
• Eng 1905 refactor artifact collection into model by @eunice-chan in #662
• Fix broken Lambda execution by @kenxu95 in #727
• Fix Run Notebook Fail Resulted From Python Setup by @Fanjia-Yan in #728
• Eng 1907 refactor integration collection into model by @eunice-chan in #663
• Eng 1908 refactor notification collection into model by @eunice-chan in #664
• Eng 1910 refactor operator_result collection into model by @eunice-chan in #667
• ENG-2032 Fix Workflows Back Navigation by @agiron123 in #730
• Adds ability to configure memory for Lambda-backed operators by @kenxu95 in #729
• Defines DAG model and repo by @saurav-c in #705
• Builds many improvements into SDK integration test suite by @kenxu95 in #732
• Defines DAGResult model and repo by @saurav-c in #711
• Defines DAGEdge model and repo by @saurav-c in #713
• Defines watcher model and repo by @saurav-c in #715
• ENG-2047 Bugfix: User unable to switch workflows after hitting back button in browser by @agiron123 in #739
• Ready for release 0.1.5 by @cw75 in #736

v0.1.4

Released on November 14, 2022.

Enhancements

• Extends internal integration test framework to support automated testing
  against third-party compute engines.
• Significantly refactors internal data model implementations to improve
  readability and maintainability.

Bugfixes

• Fixes bug where certain dividers on the navigation sidebar were too wide.
• Fixes bug where opening sidesheets would change page name.
• Fixes bug where function executor Dockerfiles had incorrect start script.
• Fixes bug that caused built-in metric and check functions to have different
  Python environments than regular operators.

All Changes

v0.1.3...v0.1.4

All commits

• Fix extraneous request issue for kubernetes dialog. by @agiron123 in #685
• ENG-1936: Dividers Too Wide by @agiron123 in #693
• ENG-1974 Properly Show Operator Name in SideSheet by @agiron123 in #691
• Adds Typography stories to storybook docs. by @agiron123 in #680
• Add ability for tests to run against multiple backing engines by @kenxu95 in #682
• [1/N] Defines workflow model and repo by @saurav-c in #646
• [3/N] Sets up testing for repos by @saurav-c in #670
• Eng 1921 Allow user to connect to conda by @likawind in #687
• Fix prompt for lambda role arn by @hsubbaraj-spiral in #647
• Defines user model and repo by @saurav-c in #675
• Fix start-function-executor mismatch bug between aqueduct_executor and dockerfiles by @kenxu95 in #697
• Implements workflow repo tests by @saurav-c in #694
• Explicitly specify requirements for artifact syntax sugar functions by @cw75 in #703
• Fixes react-hooks/exhaustive-deps warning in kubernetesDialog. by @agiron123 in #707
• Disable Conda from UI and remove Conda install option by @cw75 in #710
• Ready for release 0.1.4 by @cw75 in #716

v0.1.3

Released on November 7, 2022.

Enhancements

• Surfaces errors with parameter validation in workflow status summary.
• Catches errors generated during requirements.txt installation and surfaces
  them eagerly; previously, these errors were ignored.
• Improves operator execution time by only importing great_expectations when
  it's being used; the library import is quite slow, so doing it on every
  operator was wasteful.
• Adjusts various font sizes in the UI to improve presentation.
• Adds MongoDB integration.
• Adds engine parameter to global_config, allowing users to specify a
  default compute engine; engine is also now an optional parameter to
  publish_flow:
  • If the engine argument to publish_flow is specified, it will override
    the global_config. Otherwise, the engine set in global_config will
    be used.
  • If neither the engine argument to publish_flow or global_config is
    set, the workflow will be executed on the default Aqueduct execution
    engine.

Bugfixes

• Fixes bug where operator details button text overflowed.

All Changes

v0.1.2...v0.1.3

All commits

• ENG-1860: Parameter Status Bar Label and add Parameter Icon by @agiron123 in #617
• ENG-1596: Adds StorybookJS to common by @agiron123 in #621
• ENG-1899 Fix bug where the text in the "view operator details" box overflows by @agiron123 in #655
• Add ability for SDK tests to work against various data integrations by @kenxu95 in #653
• ENG-1830: Removes react-code-blocks from project. by @agiron123 in #669
• Surface requirement installation error during operator execution by @cw75 in #657
• Speed up aqueduct import time by cutting great expectations by @kenxu95 in #674
• Add list type to our typing system (AGAIN) by @kenxu95 in #672
• [1/4] Eng 1802 add mongodb integration - Python Executor by @likawind in #661
• [SDK] Simplify supplying the engine integration by @kenxu95 in #671
• Adjust font sizes by @likawind in #677
• Verified that the requirement detection hierarchy is already enforced by @cw75 in #678
• ENG-1968: Close Add Integration Dialog After Successfully Connecting by @agiron123 in #684
• Fix run_notebook action by @cw75 in #688
• Ready for release 0.1.3 by @cw75 in #683

v0.1.2

Released on October 31, 2022.

Enhancements

• Hides search bar on data viewing page when there are no artifacts.
• Adds support for variable length arguments (*args) in Aqueduct functions.

Bugfixes

• Fixes a bug where updating the metadata of a paused workflow would fail.
• Fixes a bug where parameters were shown as having an upstream function that wasn't accessible from the UI.

All Changes

v0.1.1...v0.1.2

All commits

• Eng 1869 data page: hide search bar when there are no artifacts by @eunice-chan in #642
• ENG-1867 Use Vite in common by @agiron123 in #637
• Move preview caching test to no_concurrency suite by @kenxu95 in #645
• Add positional, variable-length support for all functions by @kenxu95 in #639
• Fix a bug where updating a paused workflow fails by @likawind in #648
• Fixes selected navbar icon color to match logo by @saurav-c in #651
• Adds eslint plugin to check for rules of hooks by @agiron123 in #650
• Increases margin between notifications and setting button in header by @saurav-c in #652
• Ready for release 0.1.2 by @cw75 in #654

v0.1.1

Released on October 25, 2022.

Enhancements

• Adds support for Tensorflow Keras models to type system.
• Allows users to chain multiple SQL queries in the extract operator.
• Automatically migrates all metadata and artifact snapshots when the user changes the storage layer.
• Re-enables downloading operator code.

Bugfixes

• Fixes bug where artifact details view was not scrollable in drawer view.
• Fixes bugs where parameter nodes were rendered incorrectly.
• Fixes bug where search functionality was broken on the data page.

All Changes

v0.1.0...v0.1.1

All commits

• Add verbose flag to Github Actions Integration Tests by @eunice-chan in #613
• Updates README by @vsreekanti in #620
• Delete Helm chart src code from this repo by @cw75 in #623
• Add tfrs.Model type support by @kenxu95 in #615
• Enable scrolling for query artifact sidesheet by @eunice-chan in #627
• ENG-1657: Clean up Linter Warnings by @agiron123 in #619
• Fix broken links in README.md by @kenxu95 in #628
• Remove unnecessary requirements from run_notebooks readme by @kenxu95 in #629
• [UI] Fixes a number of bugs related to failing parameters by @kenxu95 in #622
• ENG-1852 Adds back missing operator content download button by @likawind in #618
• Unify serialization/typing code between python executor and SDK by @kenxu95 in #630
• [1/7] Allows for blocking requests during system maintenance by @saurav-c in #624
• ENG-1881 Fix bug where data page search suggestions did not appear. by @agiron123 in #633
• Remove --publish and --complex_models flags from integration tests by @kenxu95 in #631
• Implements database ListWorkflowDags by @saurav-c in #634
• Eng 1876 Fix lint warnings for DataTable, SearchBar and MultiFileViewer by @eunice-chan in #638
• Bumps integration tests timeout by @saurav-c in #641
• Fixes build errors related to generateclassname by @agiron123 in #643
• Eng 1740 allow users to chain together sql queries by @likawind in #636
• Implements vault migration by @saurav-c in #607
• Ready for Release 0.1.1 by @cw75 in #644