test(e2e): add e2e specs for oras cp (#749)

This PR - Uses docker registry a fallback registry service to cover fallback scenarios for OCI artifacts - Adds command specs for `oras copy` - Scriptizes E2E setup and execution Resolves #646, resolves #568 Signed-off-by: Billy Zha <[email protected]>
oras-project · Feb 15, 2023 · 4484599 · 4484599
1 parent 1543ce2
commit 4484599
Show file tree

Hide file tree

Showing 32 changed files with 586 additions and 209 deletions.
diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
@@ -43,26 +43,9 @@ jobs:
         run: make test
       - name: Run E2E Tests
         run: |
-          cd $GITHUB_WORKSPACE/test/e2e
-          go install github.com/onsi/ginkgo/v2/ginkgo
-          mnt_root="$GITHUB_WORKSPACE/test/e2e/testdata/distribution/mount"
-          rm -rf $mnt_root/docker
-          for layer in $(ls $mnt_root/*.tar.gz); do
-            tar -xvzf $layer -C $mnt_root
-          done
-          trap 'docker kill oras-e2e || true' ERR
-          docker run --pull always -d -p 5000:5000 --rm --name oras-e2e \
-            --env REGISTRY_STORAGE_DELETE_ENABLED=true \
-            --env REGISTRY_AUTH_HTPASSWD_REALM=test-basic \
-            --env REGISTRY_AUTH_HTPASSWD_PATH=/etc/docker/registry/passwd \
-            --mount type=bind,source=$mnt_root/docker,target=/var/lib/registry/docker \
-            --mount type=bind,source=$mnt_root/passwd_bcrypt,target=/etc/docker/registry/passwd \
-            ghcr.io/oras-project/registry:v1.0.0-rc.3
-          ginkgo -r -p --succinct suite
-          docker kill oras-e2e || true
+          sh $GITHUB_WORKSPACE/test/e2e/scripts/e2e.sh $GITHUB_WORKSPACE --clean
         env:
           ORAS_PATH: bin/linux/amd64/oras
-          ORAS_REGISTRY_HOST: localhost:5000
       - name: Check Version
         run: bin/linux/amd64/oras version
       - name: Upload coverage to codecov.io

diff --git a/.gitignore b/.gitignore
@@ -44,4 +44,5 @@ _dist/
 .DS_Store
 
 # Distribution storage files for local E2E testing
-test/e2e/testdata/distribution/mount/docker/ 
+test/e2e/testdata/distribution/mount/docker/
+test/e2e/testdata/distribution/mount_fallback/docker/
diff --git a/test/e2e/README.md b/test/e2e/README.md
@@ -1,83 +1,52 @@
 # ORAS End-to-End Testing Dev Guide
 **KNOWN LIMITATION**: E2E tests are designed to run in the CI and currently only support running on linux platform.
-## Setting up
-Minimal setup: Run the script in **step 3**
+## Prerequisites
+Install [git](https://git-scm.com/download/linux), [docker](https://docs.docker.com/desktop/install/linux-install), [go](https://go.dev/doc/install).
 
-### 1. Clone Source Code of ORAS CLI
+## Run E2E Script
 ```shell
-git clone https://github.com/oras-project/oras.git
+$REPO_ROOT/test/e2e/scripts/e2e.sh $REPO_ROOT --clean # REPO_ROOT is root folder of oras CLI code
 ```
 
-### 2. _[Optional]_ Install Ginkgo
-This will enable you use `ginkgo` directly in CLI.
+If the tests fails with errors like `ginkgo: not found`, use below command to add GOPATH into the PATH variable
 ```shell
-go install github.com/onsi/ginkgo/v2/ginkgo@latest
+PATH+=:$(go env GOPATH)/bin
 ```
-If you skip step 2, you can only run tests via `go test`. 
 
-### 3. Run Distribution
-The backend of E2E test is an [oras-distribution](https://github.com/oras-project/distribution).
-```shell
-PORT=5000
-docker run -dp $PORT:5000 --rm --name oras-e2e \
-    --env STORAGE_DELETE_ENABLED=true \
-    ghcr.io/oras-project/registry:v1.0.0-rc.2
-```
+## Development
+### 1. Using IDE
+Since E2E test suites are added as an nested module, the module file and checksum file are separated from oras CLI. To develop E2E tests, it's better to set the working directory to `$REPO_ROOT/test/e2e/` or open your IDE at it.
 
-### 4. _[Optional]_ Customize Port for Distribution
+### 2. Testing pre-built ORAS Binary
+By default, Gomega builds a temp binary every time before running e2e tests, which makes sure that latest code changes in the working directory are covered. If you are making changes to E2E test code only, set `ORAS_PATH` towards your pre-built ORAS binary to skip building and speed up the test.
+
+### 3. Debugging via `go test`
+E2E specs can be ran natively without `ginkgo`:
 ```shell
-export ORAS_REGISTRY_HOST="localhost:$PORT"
-# for PowerShell, use $env:ORAS_REGISTRY_HOST = "localhost:$PORT"
+# run below command in the target test suite folder
+go test oras.land/oras/test/e2e/suite/${suite_name}
 ```
-If you skipped step 4, E2E test will look for distribution ran in `localhost:5000`
+This is super handy when you want to do step-by-step debugging from command-line or via an IDE. If you need to debug certain specs, use [focused specs](https://onsi.github.io/ginkgo/#focused-specs) but don't check it in.
 
-### 5. _[Optional]_ Setup ORAS Binary for Testing
-```bash
-# Set REPO_ROOT as root folder of oras CLI code
-cd $REPO_ROOT
-make build
-```
-### 6. _[Optional]_ Setup Pre-Built Binary
-You need to setup below environmental variables to debug a pre-built ORAS binary:
-```bash
-export ORAS_PATH="bin/linux/amd64/oras" # change target platform if needed
-export GITHUB_WORKSPACE=$REPO_ROOT
-```
-If you skipped step 5 or 6, Gomega will build a temp binary, which will include all the CLI code changes in the working directory.
+### 4. Testing Registry Services
+The backend of E2E tests are two registry services: [oras-distribution](https://github.com/oras-project/distribution) and [upstream distribution](https://github.com/distribution/distribution). The former is expected to support image and artifact media types and referrer API; The latter is expected to only support image media type with subject and provide referrers via [tag schema](https://github.com/opencontainers/distribution-spec/blob/v1.1.0-rc1/spec.md#referrers-tag-schema). 
 
-### 7. _[Optional]_ Mount Test Data
-If you want to run command suite, you need to decompress the registry storage files and mount to the distribution. `$REPO_ROOT` points to the root folder of cloned oras CLI code.
-```shell
-mnt_root=${REPO_ROOT}/test/e2e/testdata/distribution/mount
-for layer in $(ls ${mnt_root}/*.tar.gz); do
-    tar -xvzf $layer -C ${mnt_root}
-done
-
-PORT=5000
-docker run -dp ${PORT}:5000 --rm --name oras-e2e \
-    --env STORAGE_DELETE_ENABLED=true \
-    --mount type=bind,source=${mnt_root}/docker,target=/opt/data/registry-root-dir/docker \
-    ghcr.io/oras-project/registry:v1.0.0-rc.2
-```
-Skipping step 7 you will not be able to run specs in Command suite.
+You can run scenario test suite against your own registry services via setting `ORAS_REGISTRY_HOST` or `ORAS_REGISTRY_FALLBACK_HOST` environmental variables.
 
-## Development
-### 1. Constant Build & Watch
-This is a good choice if you want to debug certain re-runnable specs
-```bash
+### 5. Constant Build & Watch
+This is a good choice if you want to debug certain re-runnable specs:
+```shell
 cd $REPO_ROOT/test/e2e
 ginkgo watch -r
 ```
 
-### 2. Debugging
-Since E2E test suites are added to a sub-module, you need to run `go test` from `$REPO_ROOT/test/e2e/`. If you need to debug a certain spec, use [focused spec](https://onsi.github.io/ginkgo/#focused-specs) but don't check it in.
-
-### 3. Trouble-shooting CLI
-Executed command should be shown in the ginkgo logs after `[It]`,
+### 6. Trouble-shooting CLI
+The executed commands should be shown in the ginkgo logs after `[It]`, with full execution output in the E2E log.
 
-### 4. Adding New Tests
-Two suites will be maintained for E2E testing:
+### 7. Adding New Tests
+Three suites will be maintained for E2E testing:
 - command: contains test specs for single oras command execution
+- auth: contains test specs similar to command specs but specific to auth. It cannot be ran in parallel with command suite specs
 - scenario: contains featured scenarios with several oras commands execution
 
 Inside a suite, please follow below model when building the hierarchical collections of specs:
@@ -89,34 +58,72 @@ Describe: <Role>
        Expect: <Result> (detailed checks for execution results)
 ```
 
-### 5. Adding New Test Data
+### 8. Adding New Test Data
+
+#### 8.1 Command Suite
+Command suite uses pre-baked test data, which is a bunch of layered archive files compressed from registry storage. Test data are all stored in `$REPO_ROOT/test/e2e/testdata/distribution/` but separated in different sub-folders: oras distribution uses `mount` and upstream distribution uses `mount_fallback`.
 
-#### 5.1 Command Suite
-Command suite uses pre-baked registry data for testing. The repository name should be `command/$repo_suffix`. To add a new layer, compress the `docker` folder from the root directory of your distribution storage and copy it to `$REPO_ROOT/test/e2e/testdata/distribution/mount` folder.
+For both registries, the repository name should follow the convention of `command/$repo_suffix`. To add a new layer to the test data, use the below command to compress the `docker` folder from the root directory of the registry storage and copy it to the corresponding subfolder in `$REPO_ROOT/test/e2e/testdata/distribution/mount`.
 ```shell
 tar -cvzf ${repo_suffix}.tar.gz --owner=0 --group=0 docker/
 ```
-Currently we have below OCI images:
+
+##### Test Data for ORAS-Distribution
 ```mermaid
 graph TD;
-    subgraph images.tar.gz
-    A0>tag: multi]-..->A1[oci index]
-    A1--linux/amd64-->A2[oci image]
-    A1--linux/arm64-->A3[oci image]
-    A1--linux/arm/v7-->A4[oci image]
-    A2-->A5[config1]
-    A3-->A6[config2]
-    A4-->A7[config3]
-    A2-- hello.tar -->A8[blob]
-    A3-- hello.tar -->A8[blob]
-    A4-- hello.tar -->A8[blob]
-
-    B0>tag: foobar]-..->B1[oci image]
-    B1-- foo1 -->B2[blob1]
-    B1-- foo2 -->B2[blob1]
-    B1-- bar -->B3[blob2]
+    subgraph "repository: command/images"
+        subgraph "file: images.tar.gz"
+            direction TB
+            A0>tag: multi]-..->A1[oci index]
+            A1--linux/amd64-->A2[oci image]
+            A1--linux/arm64-->A3[oci image]
+            A1--linux/arm/v7-->A4[oci image]
+            A2-->A5(config1)
+            A3-->A6(config2)
+            A4-->A7(config3)
+            A2-- hello.tar -->A8(blob)
+            A3-- hello.tar -->A8(blob)
+            A4-- hello.tar -->A8(blob)
+
+            B0>tag: foobar]-..->B1[oci image]
+            B1-- foo1 -->B2(blob1)
+            B1-- foo2 -->B2(blob1)
+            B1-- bar -->B3(blob2)
+        end
+    end
+    
+    subgraph "repository: command/artifacts"
+        subgraph "file: artifacts.tar.gz"
+            direction TB
+            C0>tag: foobar]-..->C1[oci image]
+            
+            direction TB
+            E1["test.sbom.file(artifact)"] -- subject --> C1
+            E2["test.signature.file(artifact)"] -- subject --> E1
+        end
+        subgraph "file: artifacts_fallback.tar.gz"
+            direction TB
+            D1["test.sbom.file(image)"] -- subject --> C1
+            D2["test.signature.file(image)"] -- subject --> D1
+        end
     end
 ```
 
-#### 5.2 Scenario Suite
+##### Test Data for Upstream Distribution
+```mermaid
+graph TD;
+    subgraph "repository: command/artifacts"
+        subgraph "file: artifacts_fallback.tar.gz"
+            direction TB
+            A0>tag: foobar]-..->A1[oci image]
+            A1-- foo1 -->A2(blob1)
+            A1-- foo2 -->A2(blob1)
+            A1-- bar -->A3(blob2)
+
+            E1["test.sbom.file(image)"] -- subject --> A1
+            E2["test.signature.file(image)"] -- subject --> E1
+        end
+    end
+```
+#### 8.2 Scenario Suite
 Test files used by scenario-based specs are placed in `$REPO_ROOT/test/e2e/testdata/files`.
diff --git a/test/e2e/go.mod b/test/e2e/go.mod
@@ -3,18 +3,19 @@ module oras.land/oras/test/e2e
 go 1.20
 
 require (
-	github.com/onsi/ginkgo/v2 v2.1.6
-	github.com/onsi/gomega v1.20.2
+	github.com/onsi/ginkgo/v2 v2.8.0
+	github.com/onsi/gomega v1.26.0
 	github.com/opencontainers/image-spec v1.1.0-rc2
 	oras.land/oras-go/v2 v2.0.0
 )
 
 require (
-	github.com/google/go-cmp v0.5.8 // indirect
+	github.com/go-logr/logr v1.2.3 // indirect
+	github.com/google/go-cmp v0.5.9 // indirect
 	github.com/opencontainers/go-digest v1.0.0 // indirect
-	golang.org/x/net v0.0.0-20220722155237-a158d28d115b // indirect
+	golang.org/x/net v0.5.0 // indirect
 	golang.org/x/sync v0.1.0 // indirect
-	golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f // indirect
-	golang.org/x/text v0.3.7 // indirect
+	golang.org/x/sys v0.4.0 // indirect
+	golang.org/x/text v0.6.0 // indirect
 	gopkg.in/yaml.v3 v3.0.1 // indirect
 )
diff --git a/test/e2e/go.sum b/test/e2e/go.sum
@@ -1,22 +1,24 @@
+github.com/go-logr/logr v1.2.3 h1:2DntVwHkVopvECVRSlL5PSo9eG+cAkDCuckLubN+rq0=
+github.com/go-logr/logr v1.2.3/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
 github.com/golang/protobuf v1.5.2 h1:ROPKBNFfQgOUMifHyP+KYbvpjbdoFNs+aK7DXlji0Tw=
-github.com/google/go-cmp v0.5.8 h1:e6P7q2lk1O+qJJb4BtCQXlK8vWEO8V1ZeuEdJNOqZyg=
-github.com/google/go-cmp v0.5.8/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
-github.com/onsi/ginkgo/v2 v2.1.6 h1:Fx2POJZfKRQcM1pH49qSZiYeu319wji004qX+GDovrU=
-github.com/onsi/ginkgo/v2 v2.1.6/go.mod h1:MEH45j8TBi6u9BMogfbp0stKC5cdGjumZj5Y7AG4VIk=
-github.com/onsi/gomega v1.20.2 h1:8uQq0zMgLEfa0vRrrBgaJF2gyW9Da9BmfGV+OyUzfkY=
-github.com/onsi/gomega v1.20.2/go.mod h1:iYAIXgPSaDHak0LCMA+AWBpIKBr8WZicMxnE8luStNc=
+github.com/google/go-cmp v0.5.9 h1:O2Tfq5qg4qc4AmwVlvv0oLiVAGB7enBSJ2x2DqQFi38=
+github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
+github.com/onsi/ginkgo/v2 v2.8.0 h1:pAM+oBNPrpXRs+E/8spkeGx9QgekbRVyr74EUvRVOUI=
+github.com/onsi/ginkgo/v2 v2.8.0/go.mod h1:6JsQiECmxCa3V5st74AL/AmsV482EDdVrGaVW6z3oYU=
+github.com/onsi/gomega v1.26.0 h1:03cDLK28U6hWvCAns6NeydX3zIm4SF3ci69ulidS32Q=
+github.com/onsi/gomega v1.26.0/go.mod h1:r+zV744Re+DiYCIPRlYOTxn0YkOLcAnW8k1xXdMPGhM=
 github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8Oi/yOhh5U=
 github.com/opencontainers/go-digest v1.0.0/go.mod h1:0JzlMkj0TRzQZfJkVvzbP0HBR3IKzErnv2BNG4W4MAM=
 github.com/opencontainers/image-spec v1.1.0-rc2 h1:2zx/Stx4Wc5pIPDvIxHXvXtQFW/7XWJGmnM7r3wg034=
 github.com/opencontainers/image-spec v1.1.0-rc2/go.mod h1:3OVijpioIKYWTqjiG0zfF6wvoJ4fAXGbjdZuI2NgsRQ=
-golang.org/x/net v0.0.0-20220722155237-a158d28d115b h1:PxfKdU9lEEDYjdIzOtC4qFWgkU2rGHdKlKowJSMN9h0=
-golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
+golang.org/x/net v0.5.0 h1:GyT4nK/YDHSqa1c4753ouYCDajOYKTja9Xb/OHtgvSw=
+golang.org/x/net v0.5.0/go.mod h1:DivGGAXEgPSlEBzxGzZI+ZLohi+xUj054jfeKui00ws=
 golang.org/x/sync v0.1.0 h1:wsuoTGHzEhffawBOhz5CYhcrV4IdKZbEyZjBMuTp12o=
 golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
-golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f h1:v4INt8xihDGvnrfjMDVXGxw9wrfxYyCjk0KbXjhR55s=
-golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/text v0.3.7 h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk=
-golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
+golang.org/x/sys v0.4.0 h1:Zr2JFtRQNX3BCZ8YtxRE9hNJYC8J6I1MVbMg6owUp18=
+golang.org/x/sys v0.4.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/text v0.6.0 h1:3XmdazWV+ubf7QgHSTWeykHOci5oeekaGJBLkrkaw4k=
+golang.org/x/text v0.6.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
 google.golang.org/protobuf v1.28.0 h1:w43yiav+6bVFTBQFZX0r7ipe9JQ1QsbMgHwbBziscLw=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=

diff --git a/test/e2e/internal/utils/exec.go b/test/e2e/internal/utils/exec.go
@@ -3,7 +3,9 @@ Copyright The ORAS Authors.
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
 You may obtain a copy of the License at
+
 http://www.apache.org/licenses/LICENSE-2.0
+
 Unless required by applicable law or agreed to in writing, software
 distributed under the License is distributed on an "AS IS" BASIS,
 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

diff --git a/test/e2e/internal/utils/file.go b/test/e2e/internal/utils/file.go
@@ -3,7 +3,9 @@ Copyright The ORAS Authors.
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
 You may obtain a copy of the License at
+
 http://www.apache.org/licenses/LICENSE-2.0
+
 Unless required by applicable law or agreed to in writing, software
 distributed under the License is distributed on an "AS IS" BASIS,
 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

diff --git a/test/e2e/internal/utils/init.go b/test/e2e/internal/utils/init.go
@@ -3,7 +3,9 @@ Copyright The ORAS Authors.
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
 You may obtain a copy of the License at
+
 http://www.apache.org/licenses/LICENSE-2.0
+
 Unless required by applicable law or agreed to in writing, software
 distributed under the License is distributed on an "AS IS" BASIS,
 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
@@ -31,19 +33,32 @@ var ORASPath string
 // Host points to the registry service where E2E specs will be run against.
 var Host string
 
+// Host points to the registry service where fallback E2E specs will be run against.
+var FallbackHost string
+
 func init() {
 	Host = os.Getenv("ORAS_REGISTRY_HOST")
 	if Host == "" {
 		Host = "localhost:5000"
 		fmt.Fprintln(os.Stderr, "cannot find host name in ORAS_REGISTRY_HOST, using", Host, "instead")
 	}
-
 	ref := registry.Reference{
 		Registry: Host,
 	}
 	if err := ref.ValidateRegistry(); err != nil {
 		panic(err)
 	}
+
+	FallbackHost = os.Getenv("ORAS_REGISTRY_FALLBACK_HOST")
+	if FallbackHost == "" {
+		FallbackHost = "localhost:6000"
+		fmt.Fprintln(os.Stderr, "cannot find fallback host name in ORAS_REGISTRY_FALLBACK_HOST, using", FallbackHost, "instead")
+	}
+	ref.Registry = FallbackHost
+	if err := ref.ValidateRegistry(); err != nil {
+		panic(err)
+	}
+
 	// setup test data
 	pwd, err := os.Getwd()
 	if err != nil {
@@ -68,7 +83,10 @@ func init() {
 			fmt.Printf("Testing based on temp binary locates in %q\n", ORASPath)
 		}
 
+		// Login
 		cmd := exec.Command(ORASPath, "login", Host, "-u", Username, "-p", Password)
 		gomega.Expect(cmd.Run()).ShouldNot(gomega.HaveOccurred())
+		cmd = exec.Command(ORASPath, "login", FallbackHost, "-u", Username, "-p", Password)
+		gomega.Expect(cmd.Run()).ShouldNot(gomega.HaveOccurred())
 	})
 }
diff --git a/test/e2e/internal/utils/match/content.go b/test/e2e/internal/utils/match/content.go
@@ -3,7 +3,9 @@ Copyright The ORAS Authors.
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
 You may obtain a copy of the License at
+
 http://www.apache.org/licenses/LICENSE-2.0
+
 Unless required by applicable law or agreed to in writing, software
 distributed under the License is distributed on an "AS IS" BASIS,
 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

diff --git a/test/e2e/internal/utils/match/keywords.go b/test/e2e/internal/utils/match/keywords.go
@@ -3,7 +3,9 @@ Copyright The ORAS Authors.
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
 You may obtain a copy of the License at
+
 http://www.apache.org/licenses/LICENSE-2.0
+
 Unless required by applicable law or agreed to in writing, software
 distributed under the License is distributed on an "AS IS" BASIS,
 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.