feat: add redis1, auth-server1, ui-org1, gw-org1 (#68)

* debug changelog.yml Signed-off-by: tangross <[email protected]> * 2nd debug changelog.yml Signed-off-by: tangross <[email protected]> * test: Changelog.yaml (#60) * debug changelog.yml * 2nd debug changelog.yml Signed-off-by: tangross <[email protected]> * Merge branch 'master' into dev-0.2 # Conflicts: # .github/workflows/changelog.yml Signed-off-by: tangross <[email protected]> * Merge branch 'master' into dev-0.2 # Conflicts: # .github/workflows/changelog.yml * pre-commit hook avoid sops secret checkin * fix misplaced files * attempt redis deployment * add daemonset to fix redis thp issue * redis up and running * add elastisearch fluentbit kibana * add redis to main flow * add auth-server to main flow * enhance changelog wf * Merge branch 'master' into dev-0.2 # Conflicts: # app-of-app/values-org1.yaml # auth-server/templates/configmap.yaml # auth-server/templates/deployment.yaml # auth-server/templates/virtual-service.yaml # auth-server/values.yaml # doc/DEVELOPMENT.md # redis/templates/deployment.yaml # redis/values-org1.yaml # redis/values.yaml * attempt auth-server * remove ingress from hlf-ca * attempt gw-org * debug enrolAdmin error * fix wrong credentials * increase resources * turn on file logger * downgrade redisearch * debug redisearch * increase resources * add probes * add probes to redis * add probes to redis * switch to sops in auth & gw-org * add pvc for file wallet * can ignore test * test: ui-control-org1 * use a newer image ui-control * test: e2e deploy * debug argo workflow sync * test web.org1.net * add retry x2 to workflow * debug cannot connect error * add initContainer to create channel wf * increaase timeout for app sync * test gw-org external connection * increase envoy cpu to 0.1 * fix gw-org virtual service * debug gw-org1
rtang03 · Dec 22, 2020 · 2c5a3bf · 2c5a3bf
1 parent 8f42120
commit 2c5a3bf
Show file tree

Hide file tree

Showing 135 changed files with 1,709 additions and 9,505 deletions.
diff --git a/README.md b/README.md
@@ -110,7 +110,31 @@ In local machine:
 35.202.107.80 argo.server
 ```
 
-**Install org0 and org1**
+### 1. Prepare secrets file
+Credentials, secrets, and passwords are re-located to `secrets.*.yaml`, in corresponding Helm chart directories. For example,
+see `orgadmin/secrets.admin1-example.yaml`, `tlsca_caadmin` must be a base64 encoded value, and created as k8s Secret resource.
+
+```shell script
+# encode ==> dGxzY2ExLWFkbWluCg==
+echo -n 'tlsca1-admin' | base64
+
+# decode => tlsca1-admin
+echo -n 'dGxzY2ExLWFkbWluCg==' | base64 -d
+```
+
+`orgadmin/secrets.admin1-example.yaml` is an encoded yaml. You copy its file content into `orgadmin/secrets.admin1.yaml`;
+and then run below command to perform sops encryption. `-i` means in-place replacement; the previous unencrypted/encoded
+yaml will be replaced. Repeat the same steps for every `secrets.*.yaml`. The yaml property ending with "_unencrypted" will skip encryption.
+
+```shell script
+# encrypt
+sops -e -i orgadmin/secrets.admin1.yaml
+
+# decrypt
+sops -d orgadmin/secrets.admin1.yaml
+```
+
+### 2. Install fabric-network - org0 and org1
 
 The run may take 30+ minutes. In addition to CLI, you may also use GKE dashboard, *argocd* and *argo* web UI to monitor the live
 status.
@@ -131,7 +155,47 @@ cd scripts
 
 # Bootstrap Org0 and Org1
 ./bootstrap.org1.sh
+```
 
+The successful deployment should show:
+
+```text
+Name:                bootstrap-channel-org1
+Namespace:           n1
+ServiceAccount:      workflow
+Status:              Succeeded
+Conditions:
+ Completed           True
+Created:             Tue Dec 22 14:11:15 +0800 (5 minutes ago)
+Started:             Tue Dec 22 14:11:15 +0800 (5 minutes ago)
+Finished:            Tue Dec 22 14:16:22 +0800 (now)
+Duration:            5 minutes 7 seconds
+ResourcesDuration:   10m54s*(1 cpu),10m54s*(100Mi memory)
+
+STEP                              TEMPLATE                                DURATION
+ ✔ bootstrap-channel-org1         main
+ ├-·-✔ sync-g1                    argocd-cli/argocd-app-sync              1m
+ | └-✔ sync-p0o1                  argocd-cli/argocd-app-sync              1m
+ ├---✔ dl-create-tlscacert        download-and-create-secret/main
+ |   ├---✔ retrieve               retrieve-tmpl                           6s
+ |   ├---✔ delete-secret-tmpl     secret-resource/delete-secret-tmpl      1s
+ |   └---✔ create-secret-tmpl     secret-resource/create-secret-1key-tmpl 2s
+ ├---✔ create-channel(0)          create-channel/main                     11s
+ ├---✔ join-channel(0)            join-channel/main                       20s
+ ├---✔ update-anchor-peer         update-anchor-peer/main                 18s
+ ├---✔ package-install-chaincode  package-install-chaincode/main          13s
+ ├---✔ chaincode-id-resource      chaincode-id-resource/main
+ |   ├---✔ delete-ccid            delete-ccid                             2s
+ |   └---✔ create-ccid            create-ccid                             2s
+ ├---✔ sync-chaincode             argocd-cli/argocd-app-sync              1m
+ ├---✔ approve-chaincode          approve-chaincode/main                  11s
+ ├---✔ commit-chaincode           commit-chaincode/main                   13s
+ └---✔ smoke-test(0)              smoke-test/main                         21s
+```
+
+### 3. Install fabric-network - org2
+
+```shell
 # Bootstrap Org2
 ./bootstrap.orgx.sh org2
 ```
@@ -186,44 +250,92 @@ STEP                                  TEMPLATE
  └---✔ smoke-test                     smoke-test/main                           13s
 ```
 
-### Tear-down
-```shell script
-cd scripts
 
-./uninstall.argo.sh
+### 4. Install redis, auth-server, gw-orgX, ui-control - org0 and org1
 
-./recreate-pvc.sh org1
+**Create tls secret to enable secure istio gateway**
 
-./recreate-pvc.sh org2
+For org1, both *ui-control* and *gw-org* are required to expose to public internet. Here utilizes the secure gateway of
+istio (see [istio v1.6 doc](https://istio.io/v1.6/docs/tasks/traffic-management/ingress/secure-ingress/)). As below code,
+`https://web.org1.net` is exposed, and requiring the secret `peer0.org1.net-tls`.
+
+```yaml
+# networking/istio-n1.yaml
+- port:
+    number: 443
+    name: https
+    protocol: HTTPS
+  tls:
+    mode: SIMPLE
+    credentialName: "peer0.org1.net-tls"
+  hosts:
+    - "web.org1.net"
+    - "gw.org1.net"
 ```
 
-Lastly, remove the 'fabric-cd-dev' storage bucket.
+```shell
+# delete existing tls cert
+kubectl -n istio-system delete secret peer0.org1.net-tls
 
+# retrieve tls cert from org1
+CERT="$(kubectl -n n1 get secret peer0.org1.net-tls -o=jsonpath='{.data.tls\.crt}' | base64 --decode)"
+KEY="$(kubectl -n n1 get secret peer0.org1.net-tls -o=jsonpath='{.data.tls\.key}' | base64 --decode)"
 
-### Prepare secrets file
-Credentials, secrets, and passwords are re-located to `secrets.*.yaml`, in corresponding Helm chart directories. For example,
-see `orgadmin/secrets.admin1-example.yaml`, `tlsca_caadmin` must be a base64 encoded value, and created as k8s Secret resource.
+# create tls cert for Istio secure gateway
+kubectl -n istio-system create secret generic peer0.org1.net-tls --from-literal=tls.crt="$CERT" --from-literal=tls.key="$KEY"
 
-```shell script
-# encode ==> dGxzY2ExLWFkbWluCg==
-echo -n 'tlsca1-admin' | base64
+# Debugging step
+kubectl logs -n istio-system "$(kubectl get pod -l istio=ingressgateway -n istio-system -o jsonpath='{.items[0].metadata.name}')"
+# should return
+# 2020-12-22T08:23:04.850277Z	info	Channel Connectivity change to READY
+# 2020-12-22T08:45:28.961386Z	info	sds	resource:peer0.org1.net-tls pushed key/cert pair to proxy
+# 2020-12-22T08:45:28.961426Z	info	sds	Dynamic push for secret peer0.org1.net-tls
+```
+**Synchronize apps**
 
-# decode => tlsca1-admin
-echo -n 'dGxzY2ExLWFkbWluCg==' | base64 -d
+```shell
+argo submit -n n1 workflow/aoa-sync-re-au-gw-ui.n1.yaml --watch --request-timeout 900s
 ```
 
-`orgadmin/secrets.admin1-example.yaml` is an encoded yaml. You copy its file content into `orgadmin/secrets.admin1.yaml`;
-and then run below command to perform sops encryption. `-i` means in-place replacement; the previous unencrypted/encoded
-yaml will be replaced. Repeat the same steps for every `secrets.*.yaml`. The yaml property ending with "_unencrypted" will skip encryption.
+It should return:
 
+```text
+Name:                aoa-sync-gw-org1
+Namespace:           n1
+ServiceAccount:      workflow
+Status:              Succeeded
+Conditions:
+ Completed           True
+Created:             Tue Dec 22 14:39:24 +0800 (8 minutes ago)
+Started:             Tue Dec 22 14:39:24 +0800 (8 minutes ago)
+Finished:            Tue Dec 22 14:48:03 +0800 (now)
+Duration:            8 minutes 39 seconds
+ResourcesDuration:   16m57s*(1 cpu),16m57s*(100Mi memory)
+
+STEP                 TEMPLATE                    PODNAME                      DURATION  MESSAGE
+ ✔ aoa-sync-gw-org1  main
+ ├---✔ sync-redis1   argocd-cli/argocd-app-sync  aoa-sync-gw-org1-3775298967  1m
+ ├---✔ sync-auth1    argocd-cli/argocd-app-sync  aoa-sync-gw-org1-3067430853  1m
+ ├---✔ sync-gw-org1  argocd-cli/argocd-app-sync  aoa-sync-gw-org1-1951236881  3m
+ └---✔ sync-ui-org1  argocd-cli/argocd-app-sync  aoa-sync-gw-org1-2916683662  2m
+```
+
+
+### Tear-down
 ```shell script
-# encrypt
-sops -e -i orgadmin/secrets.admin1.yaml
+cd scripts
 
-# decrypt
-sops -d orgadmin/secrets.admin1.yaml
+./uninstall.argo.sh
+
+./recreate-pvc.sh org1
+
+./recreate-pvc.sh org2
 ```
 
+Lastly, remove the 'fabric-cd-dev' storage bucket.
+
+
+
 
 ### For gitOps Contributors
 Please see [DEVELOPMENT](https://github.com/rtang03/fabric-cd/doc/DEVELOPMENT.md)

diff --git a/auth-server/README.md → app-of-app/README.md b/auth-server/README.md → app-of-app/README.md
diff --git a/app-of-app/values-org1.yaml b/app-of-app/values-org1.yaml
@@ -31,4 +31,12 @@ apps:
     file: values-org1.yaml
   - path: auth-server
     rel: auth-server1
+    secret: secrets.org1.yaml
+    file: values-org1.yaml
+  - path: gw-org
+    rel: gw-org1
+    secret: secrets.org1.yaml
+    file: values-org1.yaml
+  - path: ui-control
+    rel: ui-org1
     file: values-org1.yaml
diff --git a/app-of-app/values-org2.yaml b/app-of-app/values-org2.yaml
@@ -29,3 +29,11 @@ apps:
   - path: redis
     rel: redis2
     file: values-org2.yaml
+  - path: auth-server
+    rel: auth-server2
+    secret: secrets.org2.yaml
+    file: values-org2.yaml
+  - path: gw-org
+    rel: gw-org2
+    secret: secrets.org2.yaml
+    file: values-org2.yaml
diff --git a/app-of-app/values-redis.yaml b/app-of-app/values-redis.yaml
diff --git a/gupload/README.md → argo-app/README.md b/gupload/README.md → argo-app/README.md
diff --git a/argo-wf/durability-test.yaml b/argo-wf/durability-test.yaml
@@ -0,0 +1,19 @@
+{{- if .Values.durability.enabled }}
+apiVersion: argoproj.io/v1alpha1
+kind: WorkflowTemplate
+metadata:
+  name: durability-test
+spec:
+  # should use very long active time
+  # activeDeadlineSeconds: 300
+  ttlStrategy:
+    secondsAfterCompletion: 3600
+    secondsAfterSuccess: 3600
+    secondsAfterFailure: 3600
+  serviceAccountName: workflow
+  entrypoint: main
+
+  templates:
+    - name: main
+{{ toYaml .Values.no_istio | indent 6 }}
+{{- end }}
diff --git a/argo-wf/templates/argocd-cli.yaml b/argo-wf/templates/argocd-cli.yaml
@@ -3,7 +3,7 @@ kind: WorkflowTemplate
 metadata:
   name: argocd-cli
 spec:
-  activeDeadlineSeconds: 300
+  activeDeadlineSeconds: 900
   ttlStrategy:
     secondsAfterCompletion: 3600 # Time to live after workflow is completed, replaces ttlSecondsAfterFinished
     secondsAfterSuccess: 3600     # Time to live after workflow is successful
@@ -70,6 +70,7 @@ spec:
               exit $res
             fi
             set -x
+            argocd account get --server "${ARGOCD_SERVER}:80" --grpc-web --plaintext
             argocd app sync {{ .Values.alias.input_app | quote }} --server "${ARGOCD_SERVER}:80" --grpc-web --plaintext
             res=$?
             set +x
@@ -78,7 +79,7 @@ spec:
               exit $res
             fi
             set -x
-            argocd app wait {{ .Values.alias.input_app | quote }} --server "${ARGOCD_SERVER}:80" --grpc-web --plaintext --timeout 300
+            argocd app wait {{ .Values.alias.input_app | quote }} --server "${ARGOCD_SERVER}:80" --grpc-web --plaintext --timeout 600
             res=$?
             set +x
             if [ $res -ne 0 ]; then

diff --git a/argo-wf/templates/create-channel.yaml b/argo-wf/templates/create-channel.yaml
@@ -15,6 +15,12 @@ spec:
   templates:
     - name: main
 {{ toYaml .Values.no_istio | indent 6 }}
+      retryStrategy:
+        limit: 2
+        backoff:
+          duration: "1m"
+          factor: 2
+          maxDuration: "1m"
       inputs:
         parameters:
           - name: channelname
@@ -52,6 +58,16 @@ spec:
           secret:
             secretName: {{ .secretName }}
         {{- end }}
+      initContainers:
+        - name: wait-for-orderer
+          image: jwilder/dockerize
+          imagePullPolicy: IfNotPresent
+          command: ["sh", "-c"]
+          args:
+            - |-
+              ORDERER_URL={{ .Values.ordererUrl }}
+              set -x
+              dockerize -wait tcp://$ORDERER_URL -timeout 60s && echo "$ORDERER_URL is ready"
       container:
         image: library/alpine:3.12.0
         command: ["sh", "-c"]

diff --git a/argo-wf/templates/join-channel.yaml b/argo-wf/templates/join-channel.yaml
@@ -15,6 +15,12 @@ spec:
   templates:
     - name: main
 {{ toYaml .Values.no_istio | indent 6 }}
+      retryStrategy:
+        limit: 2
+        backoff:
+          duration: "1m"
+          factor: 2
+          maxDuration: "1m"
       inputs:
         parameters:
           - name: channelname
@@ -49,6 +55,7 @@ spec:
         workingDir: /var/hyperledger
         args:
           - |-
+            sleep 10s
             CHANNEL_NAME={{ .Values.alias.input_channelname | quote }}
             FABRIC_CFG_PATH=/etc/hyperledger
             CORE_PEER_TLS_ROOTCERT_FILE={{ .Values.joinChannel.corePeerTlsRootCert }}

diff --git a/argo-wf/templates/smoke-test.yaml b/argo-wf/templates/smoke-test.yaml
@@ -4,7 +4,7 @@ kind: WorkflowTemplate
 metadata:
   name: smoke-test
 spec:
-  activeDeadlineSeconds: 300
+  activeDeadlineSeconds: 600
   ttlStrategy:
     secondsAfterCompletion: 3600
     secondsAfterSuccess: 3600
@@ -15,6 +15,12 @@ spec:
   templates:
     - name: main
 {{ toYaml .Values.no_istio | indent 6 }}
+      retryStrategy:
+        limit: 2
+        backoff:
+          duration: "1m"
+          factor: 2
+          maxDuration: "1m"
       inputs:
         parameters:
           - name: channelname
@@ -50,6 +56,7 @@ spec:
         workingDir: /var/hyperledger
         args:
           - |-
+            sleep 10s
             CHANNEL_NAME={{ .Values.alias.input_channelname | quote }}
             ORDERER_URL={{ .Values.ordererUrl }}
             FABRIC_CFG_PATH=/etc/hyperledger

diff --git a/argo/README.md b/argo/README.md
diff --git a/argo/values-argo.yaml b/argo/values-argo.yaml
@@ -209,11 +209,15 @@ server:
   # PriorityClass: system-cluster-critical
   priorityClassName: ""
 
+  # Run the argo server in "secure" mode. Configure this value instead of
+  # "--secure" in extraArgs. See the following documentation for more details
+  # on secure mode:
+  # https://argoproj.github.io/argo/tls/#encrypted
+  secure: false
+
   # Extra arguments to provide to the Argo server binary.
   extraArgs:
     - --auth-mode=client
-    # turn on tls
-#    - --secure
 
   ## Additional volumes to the server main container.
   volumeMounts: []

diff --git a/auth-server/.sops.yaml b/auth-server/.sops.yaml
@@ -0,0 +1,2 @@
+creation_rules:
+  - gcp_kms: projects/fdi-cd/locations/us-central1/keyRings/fdi/cryptoKeys/sops-key
diff --git a/auth-server/Chart.yaml b/auth-server/Chart.yaml
@@ -1,6 +1,6 @@
 apiVersion: v2
 # appVersion = RELEASE TARGET
-appVersion: 0.7.0
+appVersion: 0.7.1
 description: chart for auth-server
 name: auth-server
 sources:

diff --git a/auth-server/sec.org1.example.yaml b/auth-server/sec.org1.example.yaml
@@ -0,0 +1,11 @@
+jwtSecret: c2VjcmV0
+rootClientApp:
+  clientSecret: c2VjcmV0
+# orgAdmin credentials
+orgAdmin:
+  orgadmin: YWRtaW4tcGVlcjAub3JnMS5uZXQ=
+  orgadminpw: YWRtaW4tcGVlcjAub3JnMS5uZXRQVw==
+# postgres connection
+externalDatabase:
+  username: cG9zdGdyZXM=
+  password: cGFzc3dvcmQ=