Merge branch 'master' into failure_node

flyteorg · Dec 8, 2023 · e23ece2 · e23ece2
2 parents b2ab9aa + f440278
commit e23ece2
Show file tree

Hide file tree

Showing 190 changed files with 48,598 additions and 806 deletions.
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
@@ -1,51 +1,51 @@
-<!--
-Thank you for sending the PR! 
-Please fill the applicable details below
-Happy contributing!
--->
-
 ## Tracking issue
+_https://github.com/flyteorg/flyte/issues/<number>_
 
 <!-- If your PR fixes an open issue, use `Closes #999` to link your PR with the issue. #999 stands for the issue number you are fixing -->
 
 <!-- Remove this section if not applicable -->
 
 <!-- Example: Closes #31 -->
 
-## Docs link
+## Why are the changes needed?
 
-<!-- Add documentation link built by CI jobs here, and specify the changed place -->
+<!--
+Please clarify why the changes are needed. For instance,
+1. If you propose a new API, clarify the use case for a new API.
+2. If you fix a bug, you can clarify why it is a bug.
+-->
 
-## Describe your changes
+## What changes were proposed in this pull request?
 
-<!-- List all the proposed changes in your PR -->
+<!--
+Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
+If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
+1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
+2. If there is design documentation, please add the link.
+-->
+
+## How was this patch tested?
 
-<!-- Mark all the applicable boxes. To mark the box as done follow the following conventions -->
 <!--
-[x] - Correct; marked as done
-[X] - Correct; marked as done
-[ ] - Not correct; marked as **not** done
+If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
+If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
+If tests were not added, please describe why they were not added and/or why it was difficult to add.
 -->
 
+### Setup process
+
+### Screenshots
+
 ## Check all the applicable boxes <!-- Follow the above conventions to check the box -->
 
 - [ ] I updated the documentation accordingly.
 - [ ] All new and existing tests passed.
 - [ ] All commits are signed-off.
 
-## Setup Process
-
-<!-- Describe how you set up this PR's environment to help maintainers reproduce your results more easily -->
-
-## Screenshots
-
-<!-- Add all the screenshots which support your changes -->
-
-## Note to reviewers
-
-<!-- Add notes to reviewers if applicable -->
-
 ## Related PRs
 
 <!-- Add related pull requests for reviewers to check -->
 
+## Docs link
+
+<!-- Add documentation link built by CI jobs here, and specify the changed place -->
diff --git a/.github/workflows/checks.yml b/.github/workflows/checks.yml
@@ -118,6 +118,8 @@ jobs:
     with:
       component: ${{ matrix.component }}
       go-version: ${{ needs.unpack-envvars.outputs.go-version }}
+    secrets:
+      FLYTE_BOT_PAT: ${{ secrets.FLYTE_BOT_PAT }}
 
   build_docker_images:
     name: Build Images

diff --git a/.github/workflows/codespell.yml b/.github/workflows/codespell.yml
@@ -25,4 +25,4 @@ jobs:
       - name: Codespell
         uses: codespell-project/actions-codespell@v2
         with:
-          skip: "*.pb"
+          skip: "*.pb,monodocs-environment.lock.yaml"
diff --git a/.github/workflows/create_release.yml b/.github/workflows/create_release.yml
@@ -177,7 +177,7 @@ jobs:
           git stash
 
       - name: Run GoReleaser
-        uses: goreleaser/goreleaser-action@v2
+        uses: goreleaser/goreleaser-action@v5
         with:
           version: latest
           args: release --rm-dist --release-notes=CHANGELOG/CHANGELOG-${{ github.event.inputs.version }}.md

diff --git a/.github/workflows/end2end.yml b/.github/workflows/end2end.yml
@@ -79,25 +79,16 @@ jobs:
       - name: Register P0 tests
         if: ${{ inputs.priorities == 'P0' }}
         run: |
-          for f in \
-                  basics/basics/hello_world.py \
-                  basics/basics/workflow.py \
-                  basics/basics/named_outputs.py \
-                  advanced_composition/advanced_composition/chain_entities.py \
-                  advanced_composition/advanced_composition/dynamics.py \
-                  advanced_composition/advanced_composition/map_task.py \
-                  advanced_composition/advanced_composition/subworkflows.py \
-                  data_types_and_io/data_types_and_io/dataclass.py \
-                  data_types_and_io/data_types_and_io/structured_dataset.py ;
+          while read -r line;
           do
               pyflyte --config ./boilerplate/flyte/end2end/functional-test-config.yaml \
                   register \
                   --project flytesnacks \
                   --domain development \
                   --image cr.flyte.org/flyteorg/flytekit:py3.11-latest \
                   --version ${{ env.FLYTESNACKS_VERSION }} \
-                  flytesnacks/examples/$f;
-          done
+                  flytesnacks/$line;
+          done < flytesnacks/flyte_tests.txt
       - name: Register all flytesnacks examples
         if: ${{ inputs.priorities != 'P0' }}
         uses: unionai/[email protected]

diff --git a/.github/workflows/go_generate.yml b/.github/workflows/go_generate.yml
@@ -9,6 +9,9 @@ on:
       go-version:
         required: true
         type: string
+    secrets:
+      FLYTE_BOT_PAT:
+        required: true
 jobs:
   generate:
     runs-on: ubuntu-latest
@@ -19,6 +22,8 @@ jobs:
     steps:
       - uses: actions/checkout@v3
       - uses: arduino/setup-protoc@v1
+        with:
+          repo-token: ${{ secrets.FLYTE_BOT_PAT }}
       - uses: bufbuild/buf-setup-action@v1
       - uses: actions/setup-go@v3
         with:

diff --git a/.github/workflows/single-binary.yml b/.github/workflows/single-binary.yml
@@ -27,7 +27,7 @@ jobs:
       - uses: actions/checkout@v3
       - uses: actions/setup-go@v3
         with:
-          go-version: '1.19'
+          go-version: "1.19"
       - name: golangci-lint
         uses: golangci/golangci-lint-action@v3
         with:
@@ -129,8 +129,8 @@ jobs:
   sandbox-bundled-functional-tests:
     runs-on: ubuntu-latest
     env:
-      FLYTESNACKS_PRIORITIES: 'P0'
-      FLYTESNACKS_VERSION: ''
+      FLYTESNACKS_PRIORITIES: "P0"
+      FLYTESNACKS_VERSION: ""
     timeout-minutes: 20
     needs: [build-and-push-single-binary-image]
     steps:
@@ -164,7 +164,7 @@ jobs:
       - name: Set up Python
         uses: actions/setup-python@v4
         with:
-          python-version: '3.11'
+          python-version: "3.11"
       - uses: unionai/[email protected]
       - name: Setup sandbox
         run: |
@@ -193,25 +193,16 @@ jobs:
           # ref: ${{ env.FLYTESNACKS_VERSION }}
       - name: Register specific tests
         run: |
-          for f in \
-                  basics/basics/hello_world.py \
-                  basics/basics/workflow.py \
-                  basics/basics/named_outputs.py \
-                  advanced_composition/advanced_composition/chain_entities.py \
-                  advanced_composition/advanced_composition/dynamics.py \
-                  advanced_composition/advanced_composition/map_task.py \
-                  advanced_composition/advanced_composition/subworkflows.py \
-                  data_types_and_io/data_types_and_io/dataclass.py \
-                  data_types_and_io/data_types_and_io/structured_dataset.py ;
+          while read -r line;
           do
               pyflyte --config ./boilerplate/flyte/end2end/functional-test-config.yaml \
                   register \
                   --project flytesnacks \
                   --domain development \
                   --image cr.flyte.org/flyteorg/flytekit:py3.11-latest \
                   --version ${{ env.FLYTESNACKS_VERSION }} \
-                  flytesnacks/examples/$f;
-          done
+                  flytesnacks/$line;
+          done < flytesnacks/flyte_tests.txt
       - name: End2End
         run: |
           make end2end_execute

diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
@@ -28,17 +28,30 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Fetch the code
-        uses: actions/checkout@v2
-      - name: Set up Python 3.8
-        uses: actions/setup-python@v2
+        uses: actions/checkout@v4
+      - uses: conda-incubator/setup-miniconda@v3
         with:
-          python-version: "3.8"
-      - name: Install dependencies
+          auto-update-conda: true
+          python-version: 3.9
+      - shell: bash -el {0}
+        run: |
+          conda install -c conda-forge conda-lock
+          conda-lock install -n monodocs-env monodocs-environment.lock.yaml
+      - shell: bash -el {0}
         run: |
-          python -m pip install --upgrade pip
-          if [ -f doc-requirements.txt ]; then pip install -r doc-requirements.txt; fi
+          conda activate monodocs-env
+          conda info
+          conda list
+          conda config --show-sources
+          conda config --show
+          printenv | sort
+      - name: Setup Graphviz
+        uses: ts-graphviz/setup-graphviz@v1
       - name: Build the documentation
-        run: make docs
+        shell: bash -el {0}
+        run: |
+          conda activate monodocs-env
+          make docs
 
   generate_kustomize:
     runs-on: ubuntu-latest

diff --git a/.gitignore b/.gitignore
@@ -10,10 +10,6 @@ _repos/
 _rsts/
 rsts_tmp/
 .doctrees/
-docs/_sources/
-docs/flytekit/flytekit.interfaces.html
-docs/searchindex.js
-docs/
 !flyteidl/protos/docs
 release/
 __pycache__/
@@ -33,5 +29,14 @@ dist
 *.db
 vendor/
 /docker/sandbox-bundled/images/tar
-rsts/_tags/
-**/bin/
+**/bin/
+docs/_tags/
+docs/flytectl
+docs/protos
+docs/flytekit
+docs/flytesnacks
+docs/examples
+docs/_src
+docs/_projects
+docs/api
+docs/tests
diff --git a/.readthedocs.yml b/.readthedocs.yml
@@ -16,4 +16,4 @@ sphinx:
 # Optionally set the version of Python and requirements required to build your docs
 python:
   install:
-    - requirements: doc-requirements.txt
+    - requirements: doc-requirements.in
diff --git a/CHANGELOG/CHANGELOG-v0.13.0.md b/CHANGELOG/CHANGELOG-v0.13.0.md
@@ -2,8 +2,8 @@
 
 ## Platform
 - Oauth2 support with SingleSignOn and configuration examples for popular IDP's now available in Flyte.
-  Please see the updated [information and description of the feature](https://github.com/flyteorg/flyte/blob/master/rsts/howto/authentication/index.rst), and the [setup information](https://github.com/flyteorg/flyte/blob/master/rsts/howto/authentication/setup.rst)
-  **Attention: If using Auth already - this is a BREAKING change**. refer to the [migration guide](https://github.com/flyteorg/flyte/blob/master/rsts/howto/authentication/migration.rst) to update configuration to ensure Admin continues to work. (No migration needed if auth is not turned on.)
+  Please see the updated [information and description of the feature](https://github.com/flyteorg/flyte/blob/master/docs/howto/authentication/index.rst), and the [setup information](https://github.com/flyteorg/flyte/blob/master/docs/howto/authentication/setup.rst)
+  **Attention: If using Auth already - this is a BREAKING change**. refer to the [migration guide](https://github.com/flyteorg/flyte/blob/master/docs/howto/authentication/migration.rst) to update configuration to ensure Admin continues to work. (No migration needed if auth is not turned on.)
 
 * Backend improvements to support dynamic workflow visualization (in future releases).
 * Lot of features added to [flytectl](https://flytectl.readthedocs.io/en/latest/) .