Merge pull request #217 from chmouel/issue-210-private-repository-are…

…-not-working

Add secret auto generation

INSTALL.md

@@ -1,7 +1,7 @@
 # Installation Guides
 
-In order to install and use Pipelines-as-Code, you need to 
-* Install the Pipelines-as-Code infrastructure on your cluster 
+In order to install and use Pipelines-as-Code, you need to
+* Install the Pipelines-as-Code infrastructure on your cluster
 * Create a Pipelines-as-Code GitHub App on your GitHub account or organization
 * Configure Pipelines-as-Code on your cluster to access the GitHub App
 

README.md

@@ -19,7 +19,7 @@ See a walkthought video about it here :
 [![Pipelines as Code Walkthought](https://img.youtube.com/vi/Uh1YhOGPOes/0.jpg)](https://www.youtube.com/watch?v=Uh1YhOGPOes)
 
 
-Pipeline as Code features:
+Pipelines as Code features:
 
 - Pull-request status support: When iterating over a Pull Request, status and control is
   done on the platform.
@@ -38,18 +38,17 @@ Pipeline as Code features:
 
 - `tkn-pac` plugin for Tekton CLI for managing pipelines-as-code repositories and bootstrapping
 
-
 ## Installation Guide
 
-Please follow [this document](INSTALL.md) for installing Pipeline as Code on OpenShift.
+Please follow [this document](INSTALL.md) for installing Pipelines as Code on OpenShift.
 
 ## Getting Started
 
-The flow for using pipeline as code generally begins with admin installing the Pipelines-as-Code infrastructure, creating a GitHub App and sharing the GitHub App url across the organization for app teams to enable the app on their GitHub repositories.
+The flow for using pipelines as code generally begins with admin installing the Pipelines-as-Code infrastructure, creating a GitHub App and sharing the GitHub App url across the organization for app teams to enable the app on their GitHub repositories.
 
 In order to enable the GitHub App provided by admin on your Git repository as documented [here](https://docs.github.com/en/developers/apps/managing-github-apps/installing-github-apps). Otherwise you can go to the *Settings > Applications* and then click on *Configure* button near the GitHub App you had created. In the **Repository access** section, select the repositories that you want to enable and have access to Pipelines-as-code.
 
-Once you have enabled your GitHub App for your GitHub repository, you can use the `pac` Tekton CLI plugin to bootstrap pipeline as code:
+Once you have enabled your GitHub App for your GitHub repository, you can use the `pac` Tekton CLI plugin to bootstrap pipelines as code:
 
 ```
 $ git clone https://github.com/siamaksade/pipeline-as-code-demo
@@ -76,23 +75,27 @@ tkn pac resolve --generateName \
 
 ```
 
-The above command would create a `Repository` CRD in your `demo` namespace which is used to determine where the PipelineRuns for your GitHub repository should run. It also generates an example pipeline in the `.tekton` folder. Commit and push the pipeline to your repo to start using pipeline as code.
+The above command would create a `Repository` CRD in your `demo` namespace which is used to determine where the PipelineRuns for your GitHub repository should run. It also generates an example pipeline in the `.tekton` folder. Commit and push the pipeline to your repo to start using pipelines as code.
 
 ## Usage Guide
 
-### Pipeline As Code Configurations
+### Pipelines As Code Configurations
 
 There is a few things you can configure via the configmap `pipelines-as-code` in
 the `pipelines-as-code` namespace.
 
-* `application-name`
+- `application-name`
 
   The name of the application showing for example in the GitHub Checks labels. Default to `"Pipelines as Code CI"`
 
-* `max-keep-days`
+- `max-keep-days`
 
   The number of the day to keep the PipelineRuns runs in the `pipelines-as-code` namespace. We install by default a cronjob that cleans up the PipelineRuns generated on events in pipelines-as-code namespace. Note that these PipelineRuns are internal to Pipelines-as-code are separate from the PipelineRuns that exist in the user's GitHub repository. The cronjob runs every hour and by default cleanups PipelineRuns over a day. This configmap setting doesn't affect the cleanups of the user's PipelineRuns which are controlled by the [annotations on the PipelineRun definition in the user's GitHub repository](#pipelineruns-cleanups).
 
+- `secret-auto-create`
+
+    Wether to auto create a secret with the token generated via the Github application to be used with private repositories. This feature is enabled by default.
+
 ### Namespace Configuration
 
 User create a CustomResource definition inside the namespace `my-pipeline-ci`
@@ -144,8 +147,10 @@ instead of trying to match it from all available repository on cluster.
   as Code allows you to have those two variables filled between double brackets,
   i.e: `{{ var }}`:
 
-  - `{{repo_url}}`: The repository URL of this commit
-  - `{{revision}}`: The revision of the commit.
+  - `{{repo_owner}}`: The repository owner.
+  - `{{repo_name}}`: The repository name.
+  - `{{repo_url}}`: The repository full URL.
+  - `{{revision}}`: The commit full sha revision.
 
 - You need at least one `PipelineRun` with a `PipelineSpec` or a separated
   `Pipeline` object. You can have embedded `TaskSpec` inside
@@ -213,6 +218,69 @@ If there is multiple pipeline matching an event, it will match the first one.
 We are currently not supporting multiple PipelineRuns on a single event but
 this may be something we can consider to implement in the future.
 
+#### Private repositories
+
+Pipelines as Code support private repositories by creating or updating a secret
+in the target namespace with the user token for the [git-clone](https://github.com/tektoncd/catalog/blob/main/task/git-clone) task to use and
+be able to clone private repositories.
+
+Whenever Pipelines as Code create a new PipelineRun in the target namespace it
+will create or update a secret called :
+
+`pac-git-basic-auth-REPOSITORY_OWNER-REPOSITORY_NAME`
+
+The secret contains a `.gitconfig` and a git credentials `.git-credentials` with the
+https url using the short lived token generated from the Github application for accessing the private repository.
+
+As documented :
+
+<https://github.com/tektoncd/catalog/blob/main/task/git-clone/0.4/README.md>
+
+the secret needs to be referenced inside your PipelineRun and Pipeline as a workspace called basic-auth to be passed to the `git-clone` task.
+
+For example in your PipelineRun you will add the workspace referencing the Secret :
+
+```yaml
+  workspace:
+  - name: basic-auth
+    secret:
+      secretName: "pac-git-basic-auth-{{repo_owner}}-{{repo_name}}"
+```
+
+And inside your pipeline, you are referencing them for the git-clone to reuse  :
+
+```yaml
+[...]
+workspaces:
+  - name basic-auth
+params:
+    - name: repo_url
+    - name: revision
+[...]
+tasks:
+  workspaces:
+    - name: basic-auth
+      workspace: basic-auth
+  [...]
+  tasks:
+  - name: git-clone-from-catalog
+      taskRef:
+        name: git-clone
+      params:
+        - name: url
+          value: $(params.repo_url)
+        - name: revision
+          value: $(params.revision)
+```
+
+The git-clone task will pick up the basic-auth (optional) workspace and automatically
+use it to be able to clone the private repository.
+
+You can see as well a full example [here](./test/testdata/pipelinerun_git_clone_private.yaml)
+
+This behaviour can be disabled by configuration the `secret-auto-create` key inside the
+[Pipelines-as-Code Configmap](./INSTALL.md#configuration).
+
 #### PipelineRuns Cleanups
 
 There can be a lot of PipelineRuns into an user namespace and Pipelines as Code
@@ -486,4 +554,4 @@ docker run -e KUBECONFIG=/tmp/kube/config -v ${HOME}/.kube:/tmp/kube \
 
 ## Blog Posts
 
-* [How to make a release pipeline with Pipelines as Code](https://blog.chmouel.com/2021/07/01/how-to-make-a-release-pipeline-with-pipelines-as-code)
+- [How to make a release pipeline with Pipelines as Code](https://blog.chmouel.com/2021/07/01/how-to-make-a-release-pipeline-with-pipelines-as-code)

config/200-role.yaml

@@ -82,6 +82,10 @@ rules:
   - apiGroups: [""]
     resources: ["namespaces", "pods", "pods/log"]
     verbs: ["get", "list", "watch"]
+  # Allow creating secrets in namespace
+  - apiGroups: [""]
+    resources: ["secrets"]
+    verbs: ["get", "create", "update"]
   # Permissions to list repositories on cluster
   - apiGroups: ["pipelinesascode.tekton.dev"]
     resources: ["repositories"]

config/302-pac-configmap.yaml

@@ -19,6 +19,10 @@ data:
 
   # The application name, you can customize this label
   application-name: "Pipelines as Code CI"
+
+  # Wether to automatically create a secret with the token to be use by git-clone
+  secret-auto-create: "true"
+
 kind: ConfigMap
 metadata:
   name: pipelines-as-code

config/401-trigger-issue-recheck.yaml

@@ -181,6 +181,11 @@ spec:
                         configMapKeyRef:
                           name: pipelines-as-code
                           key: application-name
+                    - name: PAC_SECRET_AUTO_CREATE
+                      valueFrom:
+                        configMapKeyRef:
+                          name: pipelines-as-code
+                          key: secret-auto-create
                     script: |
                       #!/usr/bin/env bash
                       set -eux

config/402-trigger-pull-request.yaml

@@ -170,6 +170,11 @@ spec:
                         configMapKeyRef:
                           name: pipelines-as-code
                           key: application-name
+                    - name: PAC_SECRET_AUTO_CREATE
+                      valueFrom:
+                        configMapKeyRef:
+                          name: pipelines-as-code
+                          key: secret-auto-create
                     imagePullPolicy: Always
                     image: "ko://github.com/openshift-pipelines/pipelines-as-code/cmd/pipelines-as-code"
                     script: |

config/403-trigger-push.yaml

@@ -172,6 +172,11 @@ spec:
                         configMapKeyRef:
                           name: pipelines-as-code
                           key: application-name
+                    - name: PAC_SECRET_AUTO_CREATE
+                      valueFrom:
+                        configMapKeyRef:
+                          name: pipelines-as-code
+                          key: secret-auto-create
                     script: |
                       cat << EOF > /tmp/payload.json
                       {

config/404-trigger-retest-comment.yaml

@@ -147,6 +147,11 @@ spec:
                         configMapKeyRef:
                           name: pipelines-as-code
                           key: application-name
+                    - name: PAC_SECRET_AUTO_CREATE
+                      valueFrom:
+                        configMapKeyRef:
+                          name: pipelines-as-code
+                          key: secret-auto-create
                     script: |
                       cat << EOF|tee /tmp/payload.json
                       {

pkg/cli/interface.go

@@ -49,4 +49,5 @@ type KubeInteractionIntf interface {
 	// TODO: we don't need tektonv1beta1client stuff here
 	WaitForPipelineRunSucceed(context.Context, tektonv1beta1client.TektonV1beta1Interface, *v1beta1.PipelineRun, time.Duration) error
 	CleanupPipelines(context.Context, string, string, int) error
+	CreateBasicAuthSecret(context.Context, webvcs.RunInfo, string, string) error
 }

pkg/cmd/pipelineascode/pipelineascode.go

@@ -54,14 +54,26 @@ func Command(p cli.Params) *cobra.Command {
 	cmd.Flags().StringVarP(&opts.RunInfo.EventType, "webhook-type", "", os.Getenv("PAC_EVENT_TYPE"), "Payload event type as set from Github (ie: X-GitHub-Event header)")
 	cmd.Flags().StringVarP(&opts.RunInfo.TriggerTarget, "trigger-target", "", os.Getenv("PAC_TRIGGER_TARGET"), "The trigger target from where this event comes from")
 	cmd.Flags().StringVarP(&opts.PayloadFile, "payload-file", "", os.Getenv("PAC_PAYLOAD_FILE"), "A file containing the webhook payload")
+
 	applicationName := os.Getenv("PAC_APPLICATION_NAME")
 	if applicationName == "" {
 		applicationName = defaultApplicationName
 	}
-
 	cmd.Flags().StringVar(&opts.RunInfo.ApplicationName,
 		"application-name", applicationName,
 		"The name of the application.")
+
+	secretAutoCreation := false
+	secretAutoCreationEnv := os.Getenv("PAC_SECRET_AUTO_CREATE")
+	if strings.ToLower(secretAutoCreationEnv) == "true" ||
+		strings.ToLower(secretAutoCreationEnv) == "yes" || secretAutoCreationEnv == "1" {
+		secretAutoCreation = true
+	}
+	cmd.Flags().BoolVar(&opts.RunInfo.SecretAutoCreation,
+		"secret-auto-creation",
+		secretAutoCreation,
+		"Wether to create automatically secrets.")
+
 	return cmd
 }
 

pkg/kubeinteraction/secrets.go

@@ -0,0 +1,47 @@
+package kubeinteraction
+
+import (
+	"context"
+	"fmt"
+	"net/url"
+
+	"github.com/openshift-pipelines/pipelines-as-code/pkg/webvcs"
+	corev1 "k8s.io/api/core/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+)
+
+const (
+	// nolint:gosec
+	basicAuthSecretName    = `pac-git-basic-auth-%s-%s`
+	basicAuthGitConfigData = `
+	[credential "%s"]
+	helper=store
+	`
+)
+
+// CreateBasicAuthSecret Create a secret for git-clone basic-auth workspace
+func (k Interaction) CreateBasicAuthSecret(ctx context.Context, runinfo webvcs.RunInfo, targetNamespace, token string) error {
+	repoURL, err := url.Parse(runinfo.URL)
+	if err != nil {
+		return err
+	}
+	urlWithToken := fmt.Sprintf("%s://git:%s@%s%s", repoURL.Scheme, token, repoURL.Host, repoURL.Path)
+
+	secretData := map[string]string{
+		".gitconfig":       fmt.Sprintf(basicAuthGitConfigData, runinfo.URL),
+		".git-credentials": urlWithToken,
+	}
+
+	secretName := fmt.Sprintf(basicAuthSecretName, runinfo.Owner, runinfo.Repository)
+	secret, err := k.Clients.Kube.CoreV1().Secrets(targetNamespace).Get(ctx, secretName, metav1.GetOptions{})
+	if err != nil {
+		secret = &corev1.Secret{ObjectMeta: metav1.ObjectMeta{Name: secretName}}
+		secret.StringData = secretData
+		_, err = k.Clients.Kube.CoreV1().Secrets(targetNamespace).Create(ctx, secret, metav1.CreateOptions{})
+	} else {
+		secret.StringData = secretData
+		_, err = k.Clients.Kube.CoreV1().Secrets(targetNamespace).Update(ctx, secret, metav1.UpdateOptions{})
+	}
+
+	return err
+}