diff --git a/demo/istio-mtls/README.md b/demo/istio-mtls/README.md new file mode 100644 index 0000000..2af3dbb --- /dev/null +++ b/demo/istio-mtls/README.md @@ -0,0 +1,314 @@ +# Istio mTLS demo + +This Istio mTLS Demo is a prototype of kyverno-envoy-plugin . + +## Overview And Architecture + +Istio is an open source service mesh for managing the different microservices that make up a cloud-native application. Istio provides a mechanism to use a service as an external authorizer with the [AuthorizationPolicy API](https://istio.io/latest/docs/tasks/security/authorization/authz-custom/). + +The kyverno-envoy-plugin is a custom Envoy filter that is used to intercept the incoming request to the service and validate the request using the kyverno engine. + +In this tutorial we will create a two simple microservices which are going to make external authorization to a single kyverno-envoy-plugin service in the mesh. With this tutorial we are going to understand how to use multiple microservices to make authorization decisions to a single ext-authz server. + +![arch-istio-mtls](arch-istio-mtls.png) + +To handle multiple different requests effectively, we leverage the `match/exclude` declarations to route the specific authz-request to the appropriate validating policy within the Kyverno engine. This approach allows us to execute the right validating policy for each request, enabling efficient and targeted request processing. + +### Example Policy + +The following policies will be executed by the kyverno-envoy-plugin to validate incoming requests made specifically to the `testapp-1` service. By leveraging the match declarations, we ensure that these policies are executed only when the incoming request is destined for the `testapp-1` service. This targeted approach allows us to apply the appropriate validation rules and policies based on the specific service being accessed. + +```yaml +apiVersion: json.kyverno.io/v1alpha1 +kind: ValidatingPolicy +metadata: + name: test-policy +spec: + rules: + - name: deny-external-calls-testapp-1 + match: + any: + - request: + http: + host: 'testapp-1.demo.svc.cluster.local:8080' + assert: + all: + - message: "The GET method is restricted to the /book path." + check: + request: + http: + method: 'GET' + path: '/book' +``` +To execute the policy when the incoming request is made to `testapp-2` service we need to use the `match` declarations. + +```yaml +apiVersion: json.kyverno.io/v1alpha1 +kind: ValidatingPolicy +metadata: + name: test-policy +spec: + rules: + - name: deny-external-calls-testapp-2 + match: + any: + - request: + http: + host: 'testapp-2.demo.svc.cluster.local:8080' + assert: + all: + - message: "The GET method is restricted to the /movies path." + check: + request: + http: + method: 'GET' + path: '/movie' +``` +The example json request for above payload will be like below. + +```json +{ + "source": { + "address": { + "socketAddress": { + "address": "10.244.0.71", + "portValue": 33880 + } + } + }, + "destination": { + "address": { + "socketAddress": { + "address": "10.244.0.65", + "portValue": 8080 + } + } + }, + "request": { + "time": "2024-05-20T07:52:01.566887Z", + "http": { + "id": "5415544797791892902", + "method": "GET", + "headers": { + ":authority": "testapp-2.demo.svc.cluster.local:8080", + ":method": "GET", + ":path": "/movie", + ":scheme": "http", + "user-agent": "Wget", + "x-forwarded-proto": "http", + "x-request-id": "a3ad9f03-c9cd-4eab-97d1-83e90e0cee1b" + }, + "path": "/movie", + "host": "testapp-2.demo.svc.cluster.local:8080", + "scheme": "http", + "protocol": "HTTP/1.1" + } + }, + "metadataContext": {}, + "routeMetadataContext": {} +} +``` + +To enhance security, we can implement Mutual TLS (mTLS) for peer authentication between test services and kyverno-envoy-plugin. Since we are currently using JSON request data to validate incoming requests, there is a potential risk of this data being tampered with during transit. Implementing mTLS would ensure that communication between services is encrypted and authenticated, mitigating the risk of unauthorized data modification. + +```yaml +apiVersion: security.istio.io/v1beta1 +kind: PeerAuthentication +metadata: + name: mtls-demo + namespace: demo +spec: + mtls: + mode: STRICT +--- +apiVersion: security.istio.io/v1beta1 +kind: PeerAuthentication +metadata: + name: mtls-testapp-1 + namespace: demo +spec: + selector: + matchLabels: + app: testapp-1 + mtls: + mode: STRICT + portLevelMtls: + 8080: + mode: PERMISSIVE +--- +apiVersion: security.istio.io/v1beta1 +kind: PeerAuthentication +metadata: + name: mtls-testapp-2 + namespace: demo +spec: + selector: + matchLabels: + app: testapp-2 + mtls: + mode: STRICT + portLevelMtls: + 8080: + mode: PERMISSIVE +``` +## Demo instructions + +### Required tools + +1. [`kind`](https://kind.sigs.k8s.io/) +1. [`kubectl`](https://kubernetes.io/docs/tasks/tools/install-kubectl/) +1. [`helm`](https://helm.sh/docs/intro/install/) + +### Create a local cluster and install Istio + +The [bootstrap.sh](bootstrap.sh) script contains everything needed to create a local cluster and install Istio. + +```console +# create a local cluster and install istio +./bootstrap.sh +``` +### Sample applications + +Manifests for the sample applications are available in [test-application-1.yaml](manifests/test-application-1.yaml) and [test-application-2.yaml](manifests/test-application-2.yaml). The sample app `testapp-1` provides information about books in a collection and exposes APIs to get, create and delete Book resources. The sample app `testapp-2` provides information about movies in a collection and exposes APIs to get, create and delete Movie resources. + +```console +# Create a namespace `demo` +kubectl apply -f ./manifests/namespace.yaml +``` + +```console +# deploy sample application testapp-1 and testapp-2 +kubectl apply -f ./manifests/test-application-1.yaml +kubectl apply -f ./manifests/test-application-1.yaml +``` +### Calling the sample applications + +We are going to call the sample applications using a pod in the cluster. + +```console +kubectl run test -it --rm --restart=Never --image=busybox -- wget -q --output-document - testapp-1.demo.svc.cluster.local:8080/book + +[{"id":"1","bookname":"Harry Potter","author":"J.K. Rowling"},{"id":"2","bookname":"Animal Farm","author":"George Orwell"}] +pod "test" deleted + +``` +```console +kubectl run test -it --rm --restart=Never --image=busybox -- wget -q --output-document - testapp-2.demo.svc.cluster.local:8080/movie + +[{"id":"1","Moviename":"Inception","Actor":"Leonardo DiCaprio"},{"id":"2","Moviename":"Batman","Actor":"Jack Nicholson"}] +pod "test" deleted + +``` + +### Register authorization provider + +Edit the mesh configmap to register authorization provider with the following command: + +```console +kubectl edit configmap istio -n istio-system +``` +In the editor, add the extension provider definitions to the mesh configmap. + +```yaml + data: + mesh: |- + extensionProviders: + - name: "kyverno-ext-authz-grpc" + envoyExtAuthzGrpc: + service: "ext-authz.demo.svc.cluster.local" + port: "9000" +``` + +### Authorization policy + +Now we can deploy an istio `AuthorizationPolicy`: +AuthorizationPolicy to tell Istio to use kyverno-envoy-plugin as the Authz Server + +```console +kubectl apply -f ./manifests/authorizationpolicy.yaml +``` + +```yaml +apiVersion: security.istio.io/v1 +kind: AuthorizationPolicy +metadata: + name: kyverno-ext-authz-grpc + namespace: demo +spec: + action: CUSTOM + provider: + # The provider name must match the extension provider defined in the mesh config. + name: kyverno-ext-authz-grpc + rules: + # The rules specify when to trigger the external authorizer. + - to: + - operation: + paths: ["/book","/movie"] +``` + +This policy configures an external service for authorization. Note that the service is not specified directly in the policy but using a provider.name field. The rules specify that requests to paths `/book` and `/movies`. + +### Authorization service deployment + +The deployment manifest of the authorization service is available in [ext-auth-server.yaml](manifests/ext-auth-server.yaml). This deployment require policy through configmap . + +Apply the policy configmap with the following command. + +```console +kubectl apply -f ./manifests/policy-configmap.yaml +``` +```console +#Deploy the kyverno external authorizer server +kubectl apply -f ./manifests/ext-auth-server.yaml +``` +Verify the sample external authorizer is up and running: +```console +kubectl logs "$(kubectl get pod -l app=ext-authz -n demo -o jsonpath={.items..metadata.name})" -n demo -c ext-authz -f +Starting GRPC server on Port 9000 +Starting HTTP server on Port 8000 + +``` +### Apply PeerAuthentication Policy + +Apply the PeerAuthentication policy to enable mTLS for the sample applications and external authorizer. + +```console +kubectl apply -f ./manifests/peerAuthentication.yaml +``` + +### Test the sample applications + +Check on the logs of the sample applications to see that the requests are accepted and rejected + +Check on `GET` request on `testapp-1` which is allowed according to policy `deny-external-calls-testapp-1` + +```console +kubectl run test -it --rm --restart=Never --image=busybox -- wget -q --output-document - testapp-1.demo.svc.cluster.local:8080/book + +[{"id":"1","bookname":"Harry Potter","author":"J.K. Rowling"},{"id":"2","bookname":"Animal Farm","author":"George Orwell"}] +pod "test" deleted +``` + +Check on `GET` request on `testapp-2` which is allowed according to policy `deny-external-calls-testapp-2` + +```console +kubectl run test -it --rm --restart=Never --image=busybox -- wget -q --output-document - testapp-2.demo.svc.cluster.local:8080/movie + +[{"id":"1","Moviename":"Inception","Actor":"Leonardo DiCaprio"},{"id":"2","Moviename":"Batman","Actor":"Jack Nicholson"}] +pod "test" deleted + +``` + +Check logs of external authorizer to see that the requests are which policy was executed for a perticular request . + +```console +kubectl logs "$(kubectl get pod -l app=ext-authz -n demo -o jsonpath={.items..metadata.name})" -n demo -c ext-authz -f +Starting GRPC server on Port 9000 +Starting HTTP server on Port 8000 +2024/05/21 07:41:33 Request is initialized in kyvernojson engine . +2024/05/21 07:41:33 Request passed the deny-external-calls-testapp-1 policy rule. +2024/05/21 07:42:22 Request is initialized in kyvernojson engine . +2024/05/21 07:42:22 Request passed the deny-external-calls-testapp-2 policy rule. +``` +First request was directed to testapp-1 which was allowed by the policy `deny-external-calls-testapp-1` and the second request was directed to testapp-2 which was allowed by the policy `deny-external-calls-testapp-2`. + diff --git a/demo/istio-mtls/arch-istio-mtls.png b/demo/istio-mtls/arch-istio-mtls.png new file mode 100644 index 0000000..1aaa55b Binary files /dev/null and b/demo/istio-mtls/arch-istio-mtls.png differ diff --git a/demo/istio-mtls/bootstrap.sh b/demo/istio-mtls/bootstrap.sh new file mode 100755 index 0000000..5f63a84 --- /dev/null +++ b/demo/istio-mtls/bootstrap.sh @@ -0,0 +1,31 @@ +#!/bin/bash + +KIND_IMAGE=kindest/node:v1.29.2 +ISTIO_REPO=https://istio-release.storage.googleapis.com/charts +ISTIO_NS=istio-system + +# Create Kind cluster +kind create cluster --image $KIND_IMAGE --wait 1m --config - < { + const res = http.get(`${BASE_URL}/book`, { + headers: { 'Authorization': 'Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjIyNDEwODE1MzksIm5iZiI6MTUxNDg1MTEzOSwicm9sZSI6Imd1ZXN0Iiwic3ViIjoiWVd4cFkyVT0ifQ.ja1bgvIt47393ba_WbSBm35NrUhdxM4mOVQN8iXz8lk' }, + }); + check(res, { + 'is status 200': (r) => r.status === 200, + }); + }); + + sleep(1); // Sleep for 1 second between iterations +} +``` + +3. **Run the k6 test**: Run the load test with the following command: + +```console +k6 run k6-script.yaml +``` +4. **Analyze the results**: Generate an json report with detailed insight by running: + +```console +k6 run --out json=report.json k6-script.js +``` +5. ***Repeat for different scenarios**: + +- # App only + In this case , request are sent directly to the sample application ie no Envoy and Kyverno-plugin in the request path . + For this run this command to apply the sample applicaition and then test with k6 + ```console + kubectl apply -f ./manifest/app.yaml + ``` + Results of the k6 when only application is applied + ```bash + + /\ |‾‾| /‾‾/ /‾‾/ + /\ / \ | |/ / / / + / \/ \ | ( / ‾‾\ + / \ | |\ \ | (‾) | + / __________ \ |__| \__\ \_____/ .io + + execution: local + script: k6-script.js + output: - + + scenarios: (100.00%) 1 scenario, 100 max VUs, 2m30s max duration (incl. graceful stop): + * default: Up to 100 looping VUs for 2m0s over 3 stages (gracefulRampDown: 30s, gracefulStop: 30s) + + + █ GET /book with guest token + + ✓ is status 200 + + checks.........................: 100.00% ✓ 9048 ✗ 0 + data_received..................: 2.1 MB 18 kB/s + data_sent......................: 2.6 MB 21 kB/s + group_duration.................: avg=1.01ms min=166.46µs med=775.01µs max=36ms p(90)=1.72ms p(95)=2.31ms + http_req_blocked...............: avg=15.08µs min=1.55µs med=6.54µs max=4.09ms p(90)=12.07µs p(95)=15.25µs + http_req_connecting............: avg=4.58µs min=0s med=0s max=1.57ms p(90)=0s p(95)=0s + http_req_duration..............: avg=745.73µs min=103.06µs med=549.17µs max=35.88ms p(90)=1.26ms p(95)=1.75ms + { expected_response:true }...: avg=745.73µs min=103.06µs med=549.17µs max=35.88ms p(90)=1.26ms p(95)=1.75ms + http_req_failed................: 0.00% ✓ 0 ✗ 9048 + http_req_receiving.............: avg=119.69µs min=11.33µs med=77.78µs max=10.97ms p(90)=193.73µs p(95)=285.58µs + http_req_sending...............: avg=41µs min=6.96µs med=31.12µs max=2.39ms p(90)=61.88µs p(95)=78.15µs + http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s + http_req_waiting...............: avg=585.04µs min=75.52µs med=407.87µs max=35.84ms p(90)=965.49µs p(95)=1.33ms + http_reqs......................: 9048 75.050438/s + iteration_duration.............: avg=1s min=1s med=1s max=1.06s p(90)=1s p(95)=1s + iterations.....................: 9048 75.050438/s + vus............................: 2 min=2 max=100 + vus_max........................: 100 min=100 max=100 + + +running (2m00.6s), 000/100 VUs, 9048 complete and 0 interrupted iterations +default ✓ [======================================] 000/100 VUs 2m0s + ``` + +- # App and Envoy + In this case, Kyverno-envoy-plugin is not included in the path but Envoy is but Envoy External Authorization API disabled + For this run this command to apply the sample application with envoy. + ```console + kubectl apply -f ./manifest/app-envoy.yaml + ``` + + Results of k6 after applying sample-application with envoy. + ```bash + + /\ |‾‾| /‾‾/ /‾‾/ + /\ / \ | |/ / / / + / \/ \ | ( / ‾‾\ + / \ | |\ \ | (‾) | + / __________ \ |__| \__\ \_____/ .io + + execution: local + script: k6-script.js + output: - + + scenarios: (100.00%) 1 scenario, 100 max VUs, 2m30s max duration (incl. graceful stop): + * default: Up to 100 looping VUs for 2m0s over 3 stages (gracefulRampDown: 30s, gracefulStop: 30s) + + + █ GET /book with guest token + + ✓ is status 200 + + checks.........................: 100.00% ✓ 9031 ✗ 0 + data_received..................: 2.5 MB 21 kB/s + data_sent......................: 2.6 MB 21 kB/s + group_duration.................: avg=2.66ms min=457.22µs med=1.8ms max=65.53ms p(90)=4.85ms p(95)=6.58ms + http_req_blocked...............: avg=12.81µs min=1.52µs med=5.98µs max=2.41ms p(90)=11.84µs p(95)=13.9µs + http_req_connecting............: avg=3.82µs min=0s med=0s max=2.34ms p(90)=0s p(95)=0s + http_req_duration..............: avg=2.38ms min=383.7µs med=1.58ms max=65.22ms p(90)=4.36ms p(95)=5.92ms + { expected_response:true }...: avg=2.38ms min=383.7µs med=1.58ms max=65.22ms p(90)=4.36ms p(95)=5.92ms + http_req_failed................: 0.00% ✓ 0 ✗ 9031 + http_req_receiving.............: avg=136.3µs min=12.53µs med=76.74µs max=12.75ms p(90)=183.23µs p(95)=272.91µs + http_req_sending...............: avg=41.54µs min=6.58µs med=28.1µs max=4.15ms p(90)=59.62µs p(95)=74.85µs + http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s + http_req_waiting...............: avg=2.2ms min=349.23µs med=1.43ms max=65.08ms p(90)=4.05ms p(95)=5.52ms + http_reqs......................: 9031 74.825497/s + iteration_duration.............: avg=1s min=1s med=1s max=1.06s p(90)=1s p(95)=1s + iterations.....................: 9031 74.825497/s + vus............................: 3 min=3 max=100 + vus_max........................: 100 min=100 max=100 + + +running (2m00.7s), 000/100 VUs, 9031 complete and 0 interrupted iterations +default ✓ [======================================] 000/100 VUs 2m0s + ``` + +- # App, Envoy and Kyverno-envoy-plugin + In this case, performance measurements are observed with Envoy External Authorization API enabled and a sample real-world RBAC policy loaded into kyverno-envoy-plugin . + For this apply this command to apply sample-application, envoy and kyverno-envoy-plugin + + ```console + kubectl apply -f ./manifest/app-envoy-plugin.yaml + ``` + + Results of k6 after applying sample-application, Envoy and kyverno-envoy-plugin . + ```bash + + /\ |‾‾| /‾‾/ /‾‾/ + /\ / \ | |/ / / / + / \/ \ | ( / ‾‾\ + / \ | |\ \ | (‾) | + / __________ \ |__| \__\ \_____/ .io + + execution: local + script: k6-script.js + output: - + + scenarios: (100.00%) 1 scenario, 100 max VUs, 2m30s max duration (incl. graceful stop): + * default: Up to 100 looping VUs for 2m0s over 3 stages (gracefulRampDown: 30s, gracefulStop: 30s) + + + █ GET /book with guest token + + ✓ is status 200 + + checks.........................: 100.00% ✓ 8655 ✗ 0 + data_received..................: 2.4 MB 20 kB/s + data_sent......................: 2.4 MB 20 kB/s + group_duration.................: avg=46.54ms min=4.59ms med=29.69ms max=337.79ms p(90)=109.35ms p(95)=140.51ms + http_req_blocked...............: avg=11.88µs min=1.21µs med=4.15µs max=2.83ms p(90)=9.87µs p(95)=11.4µs + http_req_connecting............: avg=4.98µs min=0s med=0s max=2.18ms p(90)=0s p(95)=0s + http_req_duration..............: avg=46.37ms min=4.49ms med=29.49ms max=337.69ms p(90)=109.26ms p(95)=140.28ms + { expected_response:true }...: avg=46.37ms min=4.49ms med=29.49ms max=337.69ms p(90)=109.26ms p(95)=140.28ms + http_req_failed................: 0.00% ✓ 0 ✗ 8655 + http_req_receiving.............: avg=65.19µs min=11.14µs med=56.47µs max=5.58ms p(90)=102.86µs p(95)=145.19µs + http_req_sending...............: avg=30.35µs min=5.43µs med=18.48µs max=5.29ms p(90)=46.63µs p(95)=58µs + http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s + http_req_waiting...............: avg=46.27ms min=4.43ms med=29.42ms max=337.65ms p(90)=109.22ms p(95)=140.24ms + http_reqs......................: 8655 71.999297/s + iteration_duration.............: avg=1.04s min=1s med=1.03s max=1.33s p(90)=1.11s p(95)=1.14s + iterations.....................: 8655 71.999297/s + vus............................: 2 min=2 max=100 + vus_max........................: 100 min=100 max=100 + + +running (2m00.2s), 000/100 VUs, 8655 complete and 0 interrupted iterations +default ✓ [======================================] 000/100 VUs 2m0s + ``` +## Measuring Performance + +The following metrics should be measured to evaluate the performance impact of the `kyverno-envoy-plugin`: + +- **End-to-end latency** + The end-to-end latency represents the time taken for a request to complete, from the client sending the request to receiving the response. Based on the k6 results, the average end-to-end latency for the different scenarios is as follows: + + - App Only: `avg=1.01ms` (from `group_duration` or `http_req_duration`) + - App and Envoy: `avg=2.38ms` (from `http_req_duration`) + - App, Envoy, and Kyverno-envoy-plugin: `avg=46.37ms` (from `http_req_duration`) + +- **Kyverno evaluation latency** + The Kyverno evaluation latency represents the time taken by the kyverno-envoy-plugin to evaluate the request against the configured policies. While the k6 results do not directly provide this metric, an estimate can be inferred by analyzing the differences in latency between the "App and Envoy" scenario and the "App, Envoy, and Kyverno-envoy-plugin" scenario. + + The difference in average latency between these two scenarios is: + `46.37ms` - `2.38ms` = `43.99ms` + + This difference can be attributed to the Kyverno evaluation latency and the gRPC server handler latency combined. Assuming the gRPC server handler latency is relatively small compared to the Kyverno evaluation latency, the estimated range for the Kyverno evaluation latency is around 40ms to 45ms. + +- **Resource utilization** + Refers to CPU and memory usage of the Kyverno-Envoy-Plugin container , `kubectl top` utility can be laveraged to measure the resource utilization. + + Get the resource utilization of the kyverno-envoy-plugin container using the following command: + ```bash + kubectl top pod -n demo --containers + ``` + To monitor resource utilization overtime use the following command: + ```bash + watch -n 1 "kubectl top pod -n demo --containers" + ``` + Now run the k6 script in different terminal window and observe the resource utilization of the kyverno-envoy-plugin container. + + Initial resource utilization of the kyverno-envoy-plugin container: + ``` + POD NAME CPU(cores) MEMORY(bytes) + testapp-5955cd6f8b-dbvgd envoy 4m 70Mi + testapp-5955cd6f8b-dbvgd kyverno-envoy-plugin 1m 51Mi + testapp-5955cd6f8b-dbvgd test-application 1m 11Mi + ``` + Resource utilization of the kyverno-envoy-plugin container after 100 requests: + + ``` + POD NAME CPU(cores) MEMORY(bytes) + testapp-5955cd6f8b-dbvgd envoy 110m 70Mi + testapp-5955cd6f8b-dbvgd kyverno-envoy-plugin 895m 60Mi + testapp-5955cd6f8b-dbvgd test-application 17m 15Mi + + ``` + + Observations: + + - The CPU utilization of the kyverno-envoy-plugin container increased significantly from 1m to 895m after receiving 100 requests during the load test. + - The memory utilization also increased, but to a lesser extent, from 51Mi to 60Mi. + + Resource utilization of the kyverno-envoy-plugin container after load completion: + + ``` + POD NAME CPU(cores) MEMORY(bytes) + testapp-5955cd6f8b-dbvgd envoy 4m 70Mi + testapp-5955cd6f8b-dbvgd kyverno-envoy-plugin 1m 51Mi + testapp-5955cd6f8b-dbvgd test-application 1m 11Mi + ``` + + Observations: + - After the load test completed and the request volume returned to normal levels, the CPU and memory utilization of the kyverno-envoy-plugin container returned to their initial values. This indicates that the kyverno-envoy-plugin can efficiently handle the increased load during the test and release the additional resources when the load subsides. + + Correlation with k6 results: + - The k6 script simulated a load test scenario with 100 virtual users, ramping up over 30 seconds, staying at 100 users for 1 minute, and then ramping down over 30 seconds. + - During the load test, when the request volume was at its peak (100 virtual users), the kyverno-envoy-plugin container experienced a significant increase in CPU utilization, reaching 895m. + - This CPU utilization spike aligns with the increased processing demand on the kyverno-envoy-plugin to evaluate the incoming requests against the configured Kyverno policies. + - The memory utilization increase during the load test was relatively modest, suggesting that the policy evaluation did not significantly impact the memory requirements of the kyverno-envoy-plugin. + + + + + diff --git a/tests/performance-test/k6-script.js b/tests/performance-test/k6-script.js new file mode 100644 index 0000000..0dee9cd --- /dev/null +++ b/tests/performance-test/k6-script.js @@ -0,0 +1,37 @@ +import http from 'k6/http'; +import { check, group, sleep } from 'k6'; + +export const options = { + stages: [ + { duration: '30s', target: 100 }, // Ramp-up to 100 virtual users over 30 seconds + { duration: '1m', target: 100 }, // Stay at 100 virtual users for 1 minute + { duration: '30s', target: 0 }, // Ramp-down to 0 virtual users over 30 seconds + ], +}; + +/* +Replace ip for every scenerio generate the URL with these commands + +echo SERVICE_PORT=$(kubectl -n demo get service testapp -o jsonpath='{.spec.ports[?(@.port==8080)].nodePort}') +echo SERVICE_HOST=$(minikube ip) +echo SERVICE_URL=$SERVICE_HOST:$SERVICE_PORT +echo $SERVICE_URL + +http://192.168.49.2:31541 + +*/ + +const BASE_URL = 'http://192.168.49.2:31700'; // Replace with your application URL + +export default function () { + group('GET /book with guest token', () => { + const res = http.get(`${BASE_URL}/book`, { + headers: { 'Authorization': 'Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjIyNDEwODE1MzksIm5iZiI6MTUxNDg1MTEzOSwicm9sZSI6Imd1ZXN0Iiwic3ViIjoiWVd4cFkyVT0ifQ.ja1bgvIt47393ba_WbSBm35NrUhdxM4mOVQN8iXz8lk' }, + }); + check(res, { + 'is status 200': (r) => r.status === 200, + }); + }); + + sleep(1); // Sleep for 1 second between iterations +} diff --git a/tests/performance-test/manifest/app-envoy-plugin.yaml b/tests/performance-test/manifest/app-envoy-plugin.yaml new file mode 100644 index 0000000..e3c5d3a --- /dev/null +++ b/tests/performance-test/manifest/app-envoy-plugin.yaml @@ -0,0 +1,211 @@ +# Application Deployment with kyverno-envoy-plugin and Envoy sidecars. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: testapp + namespace: demo +spec: + replicas: 1 + selector: + matchLabels: + app: testapp + template: + metadata: + labels: + app: testapp + spec: + initContainers: + - name: proxy-init + image: sanskardevops/proxyinit:latest + # Configure the iptables bootstrap script to redirect traffic to the + # Envoy proxy on port 8000, specify that Envoy will be running as user + # 1111, These values must match up with the configuration + # defined below for the "envoy" and "kyverno-envoy-plugin" containers. + args: ["-p", "7000", "-u", "1111"] + securityContext: + capabilities: + add: + - NET_ADMIN + runAsNonRoot: false + runAsUser: 0 + containers: + - name: test-application + image: sanskardevops/test-application:0.0.1 + ports: + - containerPort: 8080 + - name: envoy + image: envoyproxy/envoy:v1.30-latest + securityContext: + runAsUser: 1111 + imagePullPolicy: IfNotPresent + volumeMounts: + - readOnly: true + mountPath: /config + name: proxy-config + args: + - "envoy" + - "--config-path" + - "/config/envoy.yaml" + - name: kyverno-envoy-plugin + image: sanskardevops/plugin:0.0.25 + imagePullPolicy: IfNotPresent + ports: + - containerPort: 8000 + - containerPort: 9000 + volumeMounts: + - readOnly: true + mountPath: /policies + name: policy-files + args: + - "serve" + - "--policy=/policies/policy.yaml" + volumes: + - name: proxy-config + configMap: + name: proxy-config + - name: policy-files + configMap: + name: policy-files +--- +# Envoy Config with External Authorization filter that will query kyverno-envoy-plugin. +apiVersion: v1 +kind: ConfigMap +metadata: + name: proxy-config + namespace: demo +data: + envoy.yaml: | + static_resources: + listeners: + - address: + socket_address: + address: 0.0.0.0 + port_value: 7000 + filter_chains: + - filters: + - name: envoy.filters.network.http_connection_manager + typed_config: + "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager + codec_type: auto + stat_prefix: ingress_http + route_config: + name: local_route + virtual_hosts: + - name: backend + domains: + - "*" + routes: + - match: + prefix: "/" + route: + cluster: service + http_filters: + - name: envoy.ext_authz + typed_config: + "@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz + transport_api_version: V3 + with_request_body: + max_request_bytes: 8192 + allow_partial_message: true + failure_mode_allow: false + grpc_service: + google_grpc: + target_uri: 127.0.0.1:9000 + stat_prefix: ext_authz + timeout: 0.5s + - name: envoy.filters.http.router + typed_config: + "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router + clusters: + - name: service + connect_timeout: 0.25s + type: strict_dns + lb_policy: round_robin + load_assignment: + cluster_name: service + endpoints: + - lb_endpoints: + - endpoint: + address: + socket_address: + address: 127.0.0.1 + port_value: 8080 + admin: + access_log_path: "/dev/null" + address: + socket_address: + address: 0.0.0.0 + port_value: 8001 + layered_runtime: + layers: + - name: static_layer_0 + static_layer: + envoy: + resource_limits: + listener: + example_listener_name: + connection_limit: 10000 + overload: + global_downstream_max_connections: 50000 +--- +# Example policy to enforce into kyverno-envoy-plugin sidecars. +apiVersion: v1 +kind: ConfigMap +metadata: + name: policy-files + namespace: demo +data: + policy.yaml: | + apiVersion: json.kyverno.io/v1alpha1 + kind: ValidatingPolicy + metadata: + name: checkrequest + spec: + rules: + - name: deny-guest-request-at-post + assert: + any: + - message: "POST method calls at path /book are not allowed to guests users" + check: + request: + http: + method: POST + headers: + authorization: + (split(@, ' ')[1]): + (jwt_decode(@ , 'secret').payload.role): admin + path: /book + - message: "GET method call is allowed to both guest and admin users" + check: + request: + http: + method: GET + headers: + authorization: + (split(@, ' ')[1]): + (jwt_decode(@ , 'secret').payload.role): admin + path: /book + - message: "GET method call is allowed to both guest and admin users" + check: + request: + http: + method: GET + headers: + authorization: + (split(@, ' ')[1]): + (jwt_decode(@ , 'secret').payload.role): guest + path: /book +--- +### Service to expose sample application +apiVersion: v1 +kind: Service +metadata: + name: testapp + namespace: demo +spec: + type: NodePort + selector: + app: testapp + ports: + - port: 8080 + targetPort: 8080 diff --git a/tests/performance-test/manifest/app-envoy.yaml b/tests/performance-test/manifest/app-envoy.yaml new file mode 100644 index 0000000..fb321c6 --- /dev/null +++ b/tests/performance-test/manifest/app-envoy.yaml @@ -0,0 +1,148 @@ +# Application Deployment with kyverno-envoy-plugin and Envoy sidecars. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: testapp + namespace: demo +spec: + replicas: 1 + selector: + matchLabels: + app: testapp + template: + metadata: + labels: + app: testapp + spec: + initContainers: + - name: proxy-init + image: sanskardevops/proxyinit:latest + # Configure the iptables bootstrap script to redirect traffic to the + # Envoy proxy on port 8000, specify that Envoy will be running as user + # 1111, These values must match up with the configuration + # defined below for the "envoy" and "kyverno-envoy-plugin" containers. + args: ["-p", "7000", "-u", "1111"] + securityContext: + capabilities: + add: + - NET_ADMIN + runAsNonRoot: false + runAsUser: 0 + containers: + - name: test-application + image: sanskardevops/test-application:0.0.1 + ports: + - containerPort: 8080 + - name: envoy + image: envoyproxy/envoy:v1.30-latest + securityContext: + runAsUser: 1111 + imagePullPolicy: IfNotPresent + volumeMounts: + - readOnly: true + mountPath: /config + name: proxy-config + args: + - "envoy" + - "--config-path" + - "/config/envoy.yaml" + volumes: + - name: proxy-config + configMap: + name: proxy-config +--- +# Envoy Config with disabled External Authorization API +# setting up 'failure_mode_allow: true' this will tell envoy to all requests to pass through the external authorization filter +apiVersion: v1 +kind: ConfigMap +metadata: + name: proxy-config + namespace: demo +data: + envoy.yaml: | + static_resources: + listeners: + - address: + socket_address: + address: 0.0.0.0 + port_value: 7000 + filter_chains: + - filters: + - name: envoy.filters.network.http_connection_manager + typed_config: + "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager + codec_type: auto + stat_prefix: ingress_http + route_config: + name: local_route + virtual_hosts: + - name: backend + domains: + - "*" + routes: + - match: + prefix: "/" + route: + cluster: service + http_filters: + - name: envoy.ext_authz + typed_config: + "@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz + transport_api_version: V3 + with_request_body: + max_request_bytes: 8192 + allow_partial_message: true + failure_mode_allow: true + grpc_service: + google_grpc: + target_uri: 127.0.0.1:9000 + stat_prefix: ext_authz + timeout: 0.5s + - name: envoy.filters.http.router + typed_config: + "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router + clusters: + - name: service + connect_timeout: 0.25s + type: strict_dns + lb_policy: round_robin + load_assignment: + cluster_name: service + endpoints: + - lb_endpoints: + - endpoint: + address: + socket_address: + address: 127.0.0.1 + port_value: 8080 + admin: + access_log_path: "/dev/null" + address: + socket_address: + address: 0.0.0.0 + port_value: 8001 + layered_runtime: + layers: + - name: static_layer_0 + static_layer: + envoy: + resource_limits: + listener: + example_listener_name: + connection_limit: 10000 + overload: + global_downstream_max_connections: 50000 +--- +### Service to expose sample application +apiVersion: v1 +kind: Service +metadata: + name: testapp + namespace: demo +spec: + type: NodePort + selector: + app: testapp + ports: + - port: 8080 + targetPort: 8080 diff --git a/tests/performance-test/manifest/app.yaml b/tests/performance-test/manifest/app.yaml new file mode 100644 index 0000000..44eebc8 --- /dev/null +++ b/tests/performance-test/manifest/app.yaml @@ -0,0 +1,35 @@ +# Application Deployment with kyverno-envoy-plugin and Envoy sidecars. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: testapp + namespace: demo +spec: + replicas: 1 + selector: + matchLabels: + app: testapp + template: + metadata: + labels: + app: testapp + spec: + containers: + - name: test-application + image: sanskardevops/test-application:0.0.1 + ports: + - containerPort: 8080 +--- +### Service to expose sample application +apiVersion: v1 +kind: Service +metadata: + name: testapp + namespace: demo +spec: + type: NodePort + selector: + app: testapp + ports: + - port: 8080 + targetPort: 8080