Merge branch 'master' of github.com:isl-org/Open3D into ss/issue-5837

.github/workflows/README.md

@@ -1,24 +1,33 @@
 # Running workflows on GitHub Actions
 
+## OS versions
+
+Use the oldest available OS version (e.g. ubuntu-18.04, macos-11, windows-2019)
+for each workflow to ensure created binary artifacts are as widely compatible as
+possible. For jobs that use docker, or auxiliary jobs (e.g. uploading artifacts
+to google cloud), use ubuntu-latest to avoid changing the version on GitHub
+actions updates. On macOS and Windows, using Ubuntu reduces CI cost.
+
 ## Documentation deployment
 
 ### Directory structure
 
-- `.github/workflows/documentation.yml`: Github Actions workflow file to
-  create and deploy documentation. Documentation is created for every branch
-  as a CI test, but deployed only for `master`.
-- `util/ci_utils.sh:build_docs()`: Called by GitHub Actions to build documentation.
-- `unpack_docs.sh`: Called by the documentation server to deploy the docs into
-  the website.
+-   `.github/workflows/documentation.yml`: Github Actions workflow file to
+    create and deploy documentation. Documentation is created for every branch
+    as a CI test, but deployed only for `master`.
+-   `util/ci_utils.sh:build_docs()`: Called by GitHub Actions to build documentation.
+-   `unpack_docs.sh`: Called by the documentation server to deploy the docs into
+    the website.
 
 ### Setting up the docs deployment
 
 #### Step 1: Google Cloud
 
-1. Setup `gcloud`
-    - [Install `gcloud`](https://cloud.google.com/sdk/install)
-    - `gcloud init` and login with admin's google account on the web
-2. Create storage bucket
+1.  Setup `gcloud`
+    -   [Install `gcloud`](https://cloud.google.com/sdk/install)
+    -   `gcloud init` and login with admin's google account on the web
+2.  Create storage bucket
+
     ```bash
     # Create with uniform bucket-level access: `-b`
     # https://cloud.google.com/storage/docs/creating-buckets#storage-create-bucket-gsutil
@@ -38,19 +47,19 @@
     documentation server fetches the latest docs from `master` branch every hour.
     If the documentation server fails to fetch the docs matching the `master`
     commit id, the last successfully fetched docs will be displayed.
-3. Create service account
+3.  Create service account
     ```bash
     gcloud iam service-accounts create open3d-ci-sa \
         --description="Service account for Open3D CI" \
         --display-name="open3d-ci-sa"
     ```
-4. Grant `objectAdmin` to the service account
+4.  Grant `objectAdmin` to the service account
     ```bash
     gsutil iam ch \
         serviceAccount:[email protected]:objectAdmin \
         gs://open3d-docs
     ```
-5. Create key for service account
+5.  Create key for service account
     ```bash
     gcloud iam service-accounts keys create ~/open3d-ci-sa-key.json \
     --iam-account [email protected]
@@ -60,31 +69,31 @@ Now `~/open3d-ci-sa-key.json` should have been created.
 
 #### Step 2: GitHub
 
-1. Encode the private key json file with `base64 ~/open3d-ci-sa-key.json` and
+1.  Encode the private key json file with `base64 ~/open3d-ci-sa-key.json` and
     add the output text to the
     [GitHub repository secrets](https://github.com/isl-org/Open3D/settings/secrets)
     with name `GCE_SA_KEY_DOCS_CI`
 
-2. Also add secret `GCE_DOCS_PROJECT: isl-buckets`
+2.  Also add secret `GCE_DOCS_PROJECT: isl-buckets`
 
 ## Google compute engine setup for GPU CI
 
 ### CI Procedure
 
 The GCE CI workflow `.github/workflows/gce-ubuntu-docker.yml` performs these steps:
 
-- Clone the repository
-- Build docker image, starting with a an NVIDIA base devel image with CUDA and
-  cuDNN.
-- Push image to Google container registry.
-- On Google Compute Engine (GCE), in parallel (up to GPU quota limit - currently
-  4):
-- Create a new VM instance with a custom OS image
-- Run docker image on GCE (Google Compute Engine) with environment variables
-  set for specific build config.
-- The docker image entrypoint is the `run-ci.sh` script: build, install, run
-  tests and uninstall.
-- Delete the VM instance.
+-   Clone the repository
+-   Build docker image, starting with a an NVIDIA base devel image with CUDA and
+    cuDNN.
+-   Push image to Google container registry.
+-   On Google Compute Engine (GCE), in parallel (up to GPU quota limit - currently
+    4):
+-   Create a new VM instance with a custom OS image
+-   Run docker image on GCE (Google Compute Engine) with environment variables
+    set for specific build config.
+-   The docker image entrypoint is the `run-ci.sh` script: build, install, run
+    tests and uninstall.
+-   Delete the VM instance.
 
 A separate VM instance is created for each commit and build option. The VM
 instances are named according to the commit hash and build config ID used. We
@@ -95,21 +104,21 @@ either due to lack of resources or GPU quota exhaustion.
 
 #### Step 1: Google Cloud: Create service account and key
 
-1. Create service account
+1.  Create service account
     ```bash
     gcloud iam service-accounts create open3d-ci-sa \
         --description="Service account for Open3D CI" \
         --display-name="open3d-ci-sa"    \
         --project open3d-dev
     ```
-2. Grant `Compute Instance Admin (beta)` to the service account
+2.  Grant `Compute Instance Admin (beta)` to the service account
     ```bash
     gcloud projects add-iam-policy-binding open3d-dev \
         --member=serviceAccount:[email protected] \
         --role=roles/compute.instanceAdmin \
         --project open3d-dev
     ```
-3. Create key for service account
+3.  Create key for service account
     ```bash
     gcloud iam service-accounts keys create ~/open3d-ci-sa-key.json \
         --iam-account [email protected] \
@@ -138,39 +147,42 @@ used for running CI.
 
 ## Ccache strategy
 
-- Typically, a build generates ~500MB cache. A build with Filament compiled from
-  source generates ~600MB cache.
-- Typically, regular X86 Ubuntu and macOS builds take about 40 mins without
-  caching.
-- The bottleneck of the CI is in the ARM build since it runs on a simulator.
-  When building Filament from source, the build time can exceed GitHub's 6-hour
-  limit if caching is not properly activated. With proper caching and good cache
-  hit rate, the ARM build job can run within 1 hour.
-- Both Ubuntu and macOS have a max cache setting of 2GB each out of a total
-  cache limit of 5GB for the whole repository. Windows MSVC does not use
-  caching at present since `ccache` does not officially support MSVC.
-- ARM64 cache (limit 1.5GB) is stored on Google cloud bucket
-  (`open3d-ci-cache` in the `isl-buckets` project). The bucket is world
-  readable, but needs the `open3d-ci-sa` service account for writing. Every
-  ARM64 build downloads the cache contents before build. Only `master` branch
-  builds use `gsutil rsync` to update the cache in GCS. Cache transfer only
-  takes a few minutes, but reduces ARM64 CI time to about 1:15 hours.
-
-## Development wheels for user testing
-
-`master` branch Python wheels are uploaded to a world readable GCS bucket for
-users to try out development wheels.
+-   Typically, a build generates ~500MB cache. A build with Filament compiled from
+    source generates ~600MB cache.
+-   Typically, regular X86 Ubuntu and macOS builds take about 40 mins without
+    caching.
+-   The bottleneck of the CI is in the ARM build since it runs on a simulator.
+    When building Filament from source, the build time can exceed GitHub's 6-hour
+    limit if caching is not properly activated. With proper caching and good cache
+    hit rate, the ARM build job can run within 1 hour.
+-   Both Ubuntu and macOS have a max cache setting of 2GB each out of a total
+    cache limit of 10GB for the whole repository. Windows MSVC does not use
+    caching at present since `cmake + ccache` requires Ninja instead of MSBuild.
+    Most users use MSVC / MSBuild instead of Ninja and this would make resolving
+    user issues harder.
+-   ARM64 cache (limit 1.5GB) is stored on Google cloud bucket
+    (`open3d-ci-cache` in the `isl-buckets` project). The bucket is world
+    readable, but needs the `open3d-ci-sa` service account for writing. Every
+    ARM64 build downloads the cache contents before build. Only `master` branch
+    builds use `gsutil rsync` to update the cache in GCS. Cache transfer only
+    takes a few minutes, but reduces ARM64 CI time to about 1:15 hours.
+
+## Development wheels and binary archives for user testing
+
+`master` branch Python wheels and binary archives are uploaded to a world
+readable GCS bucket in `open3d-releases-master/{python-wheels,devel}` for users
+to try out development wheels.
 
 ### Google Cloud storage
 
 Follow instructions in A. Documentation deployment to setup a Google cloud
 bucket with:
 
-- Project: open3d-dev
-- Service account: open3d-ci-sa-gpu
-- Bucket name: open3d-ci-sa-gpu
-- Public read permissions
-- One week object lifecycle
+-   Project: open3d-dev
+-   Service account: open3d-ci-sa-gpu
+-   Bucket name: open3d-ci-sa-gpu
+-   Public read permissions
+-   One month (30 days) object lifecycle
 
 ```bash
 gsutil mb -p open3d-dev -c STANDARD -l US -b on gs://open3d-releases-master

.github/workflows/clean-gcloud-profiles.yml

@@ -27,19 +27,23 @@ on:
 
 env:
   GCE_GPU_CI_SA: ${{ secrets.GCE_GPU_CI_SA }}
-  GCE_CLI_GHA_VERSION: '302.0.0' # Fixed to avoid dependency on API changes
+  GCE_CLI_GHA_VERSION: '416.0.0' # Fixed to avoid dependency on API changes
 
 jobs:
   clean-gcloud-profiles:
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-latest
     strategy:
       fail-fast: false
     steps:
+      - name: GCloud CLI auth
+        uses: 'google-github-actions/auth@v1'
+        with:
+          project_id: ${{ secrets.GCE_PROJECT }}
+          credentials_json: '${{ secrets.GCE_SA_KEY_GPU_CI }}'
       - name: GCloud CLI setup
-        uses: google-github-actions/setup-gcloud@v0.2.1
+        uses: google-github-actions/setup-gcloud@v1
         with:
           version: ${{ env.GCE_CLI_GHA_VERSION }}
-          service_account_key: ${{ secrets.GCE_SA_KEY_GPU_CI }}
           project_id: ${{ secrets.GCE_PROJECT }}
       - name: Delete GCloud login profiles manually
         run: |

.github/workflows/documentation.yml

@@ -18,32 +18,32 @@ concurrency:
   cancel-in-progress: true
 
 env:
-  GCE_CLI_GHA_VERSION: '302.0.0'      # Fixed to avoid dependency on API changes
+  GCE_CLI_GHA_VERSION: '416.0.0'      # Fixed to avoid dependency on API changes
 
 jobs:
   headless-docs:
     # Build headless and docs
-    runs-on: ubuntu-18.04
+    runs-on: ubuntu-latest  # Warn about build issues in new versions
     env:
       OPEN3D_ML_ROOT: ${{ github.workspace }}/Open3D-ML
       DEVELOPER_BUILD: ${{ github.event.inputs.developer_build || 'ON' }}
     steps:
       - name: Checkout Open3D source code
-        uses: actions/checkout@v2
+        uses: actions/checkout@v3
 
       - name: Maximize build space
         run: |
           source util/ci_utils.sh
           maximize_ubuntu_github_actions_build_space
 
       - name: Checkout Open3D-ML source code
-        uses: actions/checkout@v2
+        uses: actions/checkout@v3
         with:
           repository: isl-org/Open3D-ML
           path: ${{ env.OPEN3D_ML_ROOT }}
 
       - name: Setup cache
-        uses: actions/cache@v2
+        uses: actions/cache@v3
         with:
           # Ref: https://github.com/apache/incubator-mxnet/pull/18459/files
           path: ~/.ccache
@@ -56,16 +56,9 @@ jobs:
           restore-keys: |
             ${{ runner.os }}-ccache
       - name: Set up Python version
-        uses: actions/setup-python@v2
+        uses: actions/setup-python@v4
         with:
-          python-version: 3.7
-      - name: GCloud CLI setup
-        uses: google-github-actions/[email protected]
-        with:
-          version: ${{ env.GCE_CLI_GHA_VERSION }}
-          service_account_key: ${{ secrets.GCE_SA_KEY_DOCS_CI }}
-          project_id: ${{ secrets.GCE_DOCS_PROJECT }}
-          export_default_credentials: true
+          python-version: '3.7'
 
       - name: Install dependencies
         env:
@@ -88,12 +81,25 @@ jobs:
           ccache -s
 
       - name: Upload docs
-        uses: actions/upload-artifact@v2
+        uses: actions/upload-artifact@v3
         with:
           name: open3d_docs
           path: docs/_out/html
           if-no-files-found: error
 
+      - name: GCloud CLI auth
+        if: ${{ github.ref == 'refs/heads/master' }}
+        uses: 'google-github-actions/auth@v1'
+        with:
+          project_id: ${{ secrets.GCE_PROJECT }}
+          credentials_json: '${{ secrets.GCE_SA_KEY_GPU_CI }}'
+      - name: GCloud CLI setup
+        if: ${{ github.ref == 'refs/heads/master' }}
+        uses: google-github-actions/setup-gcloud@v1
+        with:
+          version: ${{ env.GCE_CLI_GHA_VERSION }}
+          project_id: ${{ secrets.GCE_PROJECT }}
+
       - name: Deploy docs
         if: ${{ github.ref == 'refs/heads/master' }}
         run: |

.github/workflows/gcs.lifecycle.json

@@ -1,9 +1,12 @@
 {
-    "rule":
-    [
-      {
-        "action": {"type": "Delete"},
-        "condition": {"age": 7}
+  "rule": [
+    {
+      "action": {
+        "type": "Delete"
+      },
+      "condition": {
+        "age": 7
       }
-    ]
+    }
+  ]
 }