From e7974f4ce7a2de41223e61825eb5cbddfa4b1b19 Mon Sep 17 00:00:00 2001 From: Stefan Petrushevski Date: Fri, 16 Aug 2024 15:41:59 +0200 Subject: [PATCH 1/5] Include validity coming from API; update Go version; --- .gitignore | 2 +- README.md | 62 +++++++++++++++++++++++++++++++-------------------- cmd/common.go | 37 +++++++++++++++++++----------- cmd/verify.go | 6 +++-- go.mod | 2 +- go.sum | 7 ++++++ 6 files changed, 75 insertions(+), 41 deletions(-) diff --git a/.gitignore b/.gitignore index 8a00bc1..6cd4e46 100644 --- a/.gitignore +++ b/.gitignore @@ -1,4 +1,4 @@ /gh-secret-scanning /gh-secret-scanning.exe release-command.md -Secret Scanning Report*.csv \ No newline at end of file +Secret*.csv diff --git a/README.md b/README.md index e687ccf..7dd5d9d 100644 --- a/README.md +++ b/README.md @@ -1,88 +1,102 @@ -# Overview +# gh-secret-scanning - The GitHub Secret Scanning CLI Extension + This project is a GitHub CLI (`gh`) extension that provides commands for interacting with secret scanning alerts. This extension helps GitHub Advanced Security (GHAS) customers prioritize remediation of their secret scanning alerts by identifying and focusing on those that are confirmed active first. -While this extension works for Enterprise Cloud (GHEC) customers, it is primarily intended for GitHub Enterprise Server (GHES) customers who do not have access to the [GitHub.com secret scanning validity check feature](https://docs.github.com/en/enterprise-cloud@latest/code-security/secret-scanning/managing-alerts-from-secret-scanning#validating-partner-patterns). +While this extension works for Enterprise Cloud (GHEC) customers, it is primarily intended for GitHub Enterprise Server (GHES) customers who do not have access to the [GitHub.com secret scanning validity check feature](https://docs.github.com/en/enterprise-cloud@latest/code-security/secret-scanning/managing-alerts-from-secret-scanning#validating-partner-patterns). Validity check on GHES is available as of `>=3.12` but currently limited to GitHub Personal Access Tokens (PAT). Primary features include: + - Listing secret scanning alerts for an enterprise, organization, or repository - Verifying if secret scanning alerts are still active + - Expand the out-of-the-box secret scanning validity checks capabilities with custom validators - Opening issues in repos that contain valid secrets -# Supported Token Types +## Supported Token Types + - GitHub Personal Access Tokens (GHES + GHEC) - Slack API Tokens -# Pre-requisites +## Pre-requisites + - [GitHub CLI](https://github.com/cli/cli#installation) - [GHES 3.7+](https://docs.github.com/en/enterprise-server@3.7/admin/all-releases#releases-of-github-enterprise-server) or [GHEC](https://docs.github.com/en/enterprise-cloud@latest/admin/overview/about-github-enterprise-cloud) - [GHAS](https://docs.github.com/en/enterprise-cloud@latest/get-started/learning-about-github/about-github-advanced-security) -# Installation -``` +## Installation + +```bash gh extension install CallMeGreg/gh-secret-scanning ``` -# Usage +## Usage + Authenticate with your GitHub Enterprise Server or GitHub Enterprise Cloud account: -``` + +```bash gh auth login ``` -## Alerts subcommand +### Alerts subcommand + Target either an enterprise, organization, or repository by specifying the `-e`, `-o`, or `-r` flags respectively. _Exactly one selection from these three flags is required._ -``` +```bash gh secret-scanning alerts -e ``` -``` +```bash gh secret-scanning alerts -o ``` -``` +```bash gh secret-scanning alerts -r ``` Optionally add flags to specify a GHES server, limit the number of secrets processed, filter for a specific secret provider, display the secret values, generate a csv report, include extra fields, and more: -``` + +```bash gh secret-scanning alerts -e github --url my-github-server.com --limit 10 --provider slack --show-secret --csv --verbose ``` -## Verify subcommand +### Verify subcommand + Target either an enterprise, organization, or repository by specifying the `-e`, `-o`, or `-r` flags respectively. _Exactly one selection from these three flags is required._ -``` +```bash gh secret-scanning verify -e ``` -``` +```bash gh secret-scanning verify -o ``` -``` +```bash gh secret-scanning verify -r ``` Optionally add flags to specify a GHES server, limit the number of secrets processed, filter for a specific secret provider, display the secret values, generate a csv report, include extra fields, and more: -``` + +```bash gh secret-scanning verify -e github --url my-github-server.com --limit 10 --provider slack --show-secret --csv --verbose ``` Also, optionally create an issue in any repository that contains a valid secret by using the `--create-issues` (`-i`) flag: -``` + +```bash gh secret-scanning verify -e github --url my-github-server.com --create-issues ``` +### Help -## Help See available commands and flags by running: -``` + +```bash gh secret-scanning -h ``` -``` +```bash Interact with secret scanning alerts for a GHEC or GHES 3.7+ enterprise, organization, or repository Usage: @@ -109,8 +123,8 @@ Flags: Use "secret-scanning [command] --help" for more information about a command. ``` -# Demo +## Demo + This example first lists the alerts for an organization with the `alerts` subcommand, and then verifies the secrets with the `verify` subcommand. The `--csv` flag is used to generate a csv report of the results, and the `--create-issues` flag is used to create issues in any repository that contains a valid secret. https://github.com/CallMeGreg/gh-secret-scanning/assets/110078080/fa8d7b08-1a2c-4522-ae96-5c3aab60107d - diff --git a/cmd/common.go b/cmd/common.go index ff17b82..73a85e3 100644 --- a/cmd/common.go +++ b/cmd/common.go @@ -47,6 +47,7 @@ type Alert struct { Push_protection_bypassed bool `json:"push_protection_bypassed"` Push_protection_bypassed_at string `json:"push_protection_bypassed_at"` Push_protection_bypassed_by User `json:"push_protection_bypassed_by"` + Validity_github string `json:"validity"` Validity_boolean bool `json:"validity_boolean"` Validity_response_code string `json:"validity_response_code"` Validity_endpoint string `json:"validity_endpoint"` @@ -228,6 +229,7 @@ func addRepoFullNameToAlerts(alerts []Alert) []Alert { func getSecretTypeParameter() (secret_type_param string) { // if provider was specified, only return the secret types for that provider: + secret_type_param = "all" if provider != "" { // get keys for SupportedProviders[provider] map: var secret_types []string @@ -235,15 +237,6 @@ func getSecretTypeParameter() (secret_type_param string) { secret_types = append(secret_types, key) } secret_type_param = strings.Join(secret_types, ",") - } else { - // get keys for every SupportedProvider key: - var secret_types []string - for _, supported_provider := range SupportedProviders { - for key := range supported_provider { - secret_types = append(secret_types, key) - } - } - secret_type_param = strings.Join(secret_types, ",") } return secret_type_param } @@ -272,6 +265,7 @@ func prettyPrintAlerts(alerts []Alert, validity_check bool) (err error) { t.AddField("ID", tableprinter.WithColor(Green), tableprinter.WithTruncate(nil)) t.AddField("State", tableprinter.WithColor(Green), tableprinter.WithTruncate(nil)) t.AddField("Secret Type", tableprinter.WithColor(Green), tableprinter.WithTruncate(nil)) + t.AddField("Validity GitHub", tableprinter.WithColor(Green), tableprinter.WithTruncate(nil)) if secret { t.AddField("Secret", tableprinter.WithColor(Green), tableprinter.WithTruncate(nil)) } @@ -304,6 +298,7 @@ func prettyPrintAlerts(alerts []Alert, validity_check bool) (err error) { t.AddField(strconv.Itoa(alert.Number), tableprinter.WithColor(color), tableprinter.WithTruncate(nil)) t.AddField(alert.State, tableprinter.WithColor(color), tableprinter.WithTruncate(nil)) t.AddField(alert.Secret_type, tableprinter.WithColor(color), tableprinter.WithTruncate(nil)) + t.AddField(alert.Validity_github, tableprinter.WithColor(color), tableprinter.WithTruncate(nil)) if secret { t.AddField(alert.Secret, tableprinter.WithColor(color), tableprinter.WithTruncate(nil)) } @@ -344,8 +339,9 @@ func generateCSVReport(alerts []Alert, scope string, validity_check bool) (err e counter := 0 // get current date & time: now := time.Now() - timestamp := now.Format("2006-01-02 15-04-05") - filename := "Secret Scanning Report - " + scope + " - " + timestamp + ".csv" + // Format the time as YYYYMMDD-HHMMSS + timestamp := now.Format("20060102-150405") + filename := "SecretScanningReport-" + scope + "-" + timestamp + ".csv" // Create a CSV file file, err := os.Create(filename) if err != nil { @@ -357,7 +353,7 @@ func generateCSVReport(alerts []Alert, scope string, validity_check bool) (err e writer := csv.NewWriter(file) defer writer.Flush() // Write headers to CSV file - headers := []string{"Repository", "ID", "State", "Secret Type"} + headers := []string{"Repository", "ID", "State", "Secret Type", "GitHub Validity"} if secret { headers = append(headers, "Secret") } @@ -371,7 +367,7 @@ func generateCSVReport(alerts []Alert, scope string, validity_check bool) (err e // Write data to CSV file for counter < len(alerts) && counter < limit { alert := alerts[counter] - row := []string{alert.Repository.Full_name, strconv.Itoa(alert.Number), alert.State, alert.Secret_type} + row := []string{alert.Repository.Full_name, strconv.Itoa(alert.Number), alert.State, alert.Secret_type, alert.Validity_github} if secret { row = append(row, alert.Secret) } @@ -393,9 +389,24 @@ func generateCSVReport(alerts []Alert, scope string, validity_check bool) (err e } func verifyAlerts(alerts []Alert) (alertsOutput []Alert, err error) { + // Print Supported providers for reference when verbose flag is enabled + if verbose { + fmt.Println(Blue("Supported Providers:")) + for provider, secretTypes := range SupportedProviders { + for secretType := range secretTypes { + fmt.Printf("- %s - %s\n", provider, secretType) + } + } + } for i, alert := range alerts { // verify that the alert is valid by making a request to its validation endpoint: provider := strings.Split(alert.Secret_type, "_")[0] + + // Skip alert if provider is not supported + if _, ok := SupportedProviders[provider]; !ok { + continue + } + secret_type := alert.Secret_type secret_validation_method := SupportedProviders[provider][secret_type]["HttpMethod"] secret_validation_content_type := SupportedProviders[provider][secret_type]["ContentType"] diff --git a/cmd/verify.go b/cmd/verify.go index a42d6f7..345450b 100644 --- a/cmd/verify.go +++ b/cmd/verify.go @@ -59,8 +59,10 @@ func runVerify(cmd *cobra.Command, args []string) (err error) { values.Set("per_page", per_page) // if provider was specified, filter results for just that provider. Otherwise, target all supported providers: secret_type := getSecretTypeParameter() - values.Set("secret_type", secret_type) - parsedURL.RawQuery = values.Encode() + if secret_type != "all" { + values.Set("secret_type", secret_type) + parsedURL.RawQuery = values.Encode() + } // update the request path requestPath = parsedURL.String() diff --git a/go.mod b/go.mod index 0cab9f2..82c8c7a 100644 --- a/go.mod +++ b/go.mod @@ -1,6 +1,6 @@ module github.com/CallMeGreg/gh-secret-scanning -go 1.19 +go 1.23 require ( github.com/cli/go-gh v1.2.1 diff --git a/go.sum b/go.sum index 78a7c39..6734ce2 100644 --- a/go.sum +++ b/go.sum @@ -11,12 +11,15 @@ github.com/cli/shurcooL-graphql v0.0.4/go.mod h1:3waN4u02FiZivIV+p1y4d0Jo1jc6BVi github.com/cpuguy83/go-md2man/v2 v2.0.3/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o= github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E= github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= +github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/h2non/parth v0.0.0-20190131123155-b4df798d6542 h1:2VTzZjLZBgl62/EtslCrtky5vbi9dd7HrQPQIx6wqiw= +github.com/h2non/parth v0.0.0-20190131123155-b4df798d6542/go.mod h1:Ow0tF8D4Kplbc8s8sSb3V2oUCygFHVp8gC3Dn6U4MNI= github.com/henvic/httpretty v0.0.6 h1:JdzGzKZBajBfnvlMALXXMVQWxWMF/ofTy8C3/OSUTxs= github.com/henvic/httpretty v0.0.6/go.mod h1:X38wLjWXHkXT7r2+uK8LjCMne9rsuNaBLJ+5cU2/Pmo= github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8= github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw= github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI= +github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo= github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= github.com/lucasb-eyer/go-colorful v1.2.0 h1:1nnpGOrhyZZuNyfu1QjKiUICQ74+3FNCN69Aj6K7nkY= @@ -32,6 +35,7 @@ github.com/muesli/reflow v0.3.0/go.mod h1:pbwTDkVPibjO2kyvBQRBxTWEEGDGq0FlB1BIKt github.com/muesli/termenv v0.13.0 h1:wK20DRpJdDX8b7Ek2QfhvqhRQFZ237RGRO0RQ/Iqdy0= github.com/muesli/termenv v0.13.0/go.mod h1:sP1+uffeLaEYpyOTb8pLCUctGcGLnoFjSn4YJK5e2bc= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= +github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/rivo/uniseg v0.1.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc= github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc= github.com/rivo/uniseg v0.4.4 h1:8TfxU8dW6PdqD27gjM8MVNuicgxIjxpm4K7x4jp8sis= @@ -42,6 +46,7 @@ github.com/spf13/cobra v1.8.0/go.mod h1:WXLWApfZ71AjXPya3WOlMsY9yMs7YeiHhFVlvLyh github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA= github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= github.com/stretchr/testify v1.7.0 h1:nwc3DEeHmmLAfoZucVR881uASk0Mfjw8xYJ99tb5CcY= +github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/thlib/go-timezone-local v0.0.0-20210907160436-ef149e42d28e h1:BuzhfgfWQbX0dWzYzT1zsORLnHRv3bcRcsaUk0VmXA8= github.com/thlib/go-timezone-local v0.0.0-20210907160436-ef149e42d28e/go.mod h1:/Tnicc6m/lsJE0irFMA0LfIwTBo4QP7A8IfyIv4zZKI= golang.org/x/sys v0.0.0-20210831042530-f4d43177bf5e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= @@ -55,6 +60,8 @@ golang.org/x/text v0.13.0 h1:ablQoSUd0tRdKxZewP80B+BaqeKJuVhuRxj/dkrun3k= golang.org/x/text v0.13.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY= +gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/h2non/gock.v1 v1.1.2 h1:jBbHXgGBK/AoPVfJh5x4r/WxIrElvbLel8TCZkkZJoY= +gopkg.in/h2non/gock.v1 v1.1.2/go.mod h1:n7UGz/ckNChHiK05rDoiC4MYSunEC/lyaUm2WWaDva0= gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= From 8081977587958f3b66ff250f70befc95498e0263 Mon Sep 17 00:00:00 2001 From: Stefan Petrushevski Date: Fri, 16 Aug 2024 15:57:12 +0200 Subject: [PATCH 2/5] all lowercase csv filename --- cmd/common.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cmd/common.go b/cmd/common.go index 73a85e3..8b5e29f 100644 --- a/cmd/common.go +++ b/cmd/common.go @@ -341,7 +341,7 @@ func generateCSVReport(alerts []Alert, scope string, validity_check bool) (err e now := time.Now() // Format the time as YYYYMMDD-HHMMSS timestamp := now.Format("20060102-150405") - filename := "SecretScanningReport-" + scope + "-" + timestamp + ".csv" + filename := secretscanningreport-" + scope + "-" + timestamp + ".csv" // Create a CSV file file, err := os.Create(filename) if err != nil { From 57cbe12d2ced012058c45591ce7c44ac44bd2b8e Mon Sep 17 00:00:00 2001 From: Stefan Petrushevski Date: Fri, 16 Aug 2024 15:58:16 +0200 Subject: [PATCH 3/5] .gitignore to ignore new and old filename reports --- .gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitignore b/.gitignore index 6cd4e46..e28aaed 100644 --- a/.gitignore +++ b/.gitignore @@ -1,4 +1,5 @@ /gh-secret-scanning /gh-secret-scanning.exe release-command.md +secret*.csv Secret*.csv From 46d92652a93736bc376503bc0fc15b1d29bb8d6c Mon Sep 17 00:00:00 2001 From: Stefan Date: Mon, 19 Aug 2024 17:53:05 +0200 Subject: [PATCH 4/5] Update README.md Co-authored-by: Greg Mohler <110078080+CallMeGreg@users.noreply.github.com> --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 7dd5d9d..3ae5dbb 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ This project is a GitHub CLI (`gh`) extension that provides commands for interac This extension helps GitHub Advanced Security (GHAS) customers prioritize remediation of their secret scanning alerts by identifying and focusing on those that are confirmed active first. -While this extension works for Enterprise Cloud (GHEC) customers, it is primarily intended for GitHub Enterprise Server (GHES) customers who do not have access to the [GitHub.com secret scanning validity check feature](https://docs.github.com/en/enterprise-cloud@latest/code-security/secret-scanning/managing-alerts-from-secret-scanning#validating-partner-patterns). Validity check on GHES is available as of `>=3.12` but currently limited to GitHub Personal Access Tokens (PAT). +While this extension works for Enterprise Cloud (GHEC) customers, it is primarily intended for GitHub Enterprise Server (GHES) customers who do not have access to the [GitHub.com secret scanning validity check feature](https://docs.github.com/en/enterprise-cloud@latest/code-security/secret-scanning/managing-alerts-from-secret-scanning/evaluating-alerts#checking-a-secrets-validity). Validity check on GHES is available as of `>=3.12` but currently limited to GitHub Personal Access Tokens (PAT). Primary features include: From f56c499bb576d784531ad9ea4a915518f0c56ed8 Mon Sep 17 00:00:00 2001 From: Stefan Petrushevski Date: Mon, 19 Aug 2024 18:02:43 +0200 Subject: [PATCH 5/5] chore: Update .gitignore to ignore .DS_Store file and fix filename in common.go --- .DS_Store | Bin 6148 -> 0 bytes .gitignore | 1 + cmd/common.go | 2 +- 3 files changed, 2 insertions(+), 1 deletion(-) delete mode 100644 .DS_Store diff --git a/.DS_Store b/.DS_Store deleted file mode 100644 index 5008ddfcf53c02e82d7eee2e57c38e5672ef89f6..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 6148 zcmeH~Jr2S!425mzP>H1@V-^m;4Wg<&0T*E43hX&L&p$$qDprKhvt+--jT7}7np#A3 zem<@ulZcFPQ@L2!n>{z**++&mCkOWA81W14cNZlEfg7;MkzE(HCqgga^y>{tEnwC%0;vJ&^%eQ zLs35+`xjp>T0