From 266fa0537247181f63b1f36f2089e5e3fbf4d396 Mon Sep 17 00:00:00 2001 From: John Engelke <52300928+jpl-jengelke@users.noreply.github.com> Date: Tue, 25 Apr 2023 11:55:32 -0700 Subject: [PATCH 01/16] Issue NASA-AMMOS/slim#89: Initial template for dependabot scanning writeup --- continuous-testing/starter-kits/README.md | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/continuous-testing/starter-kits/README.md b/continuous-testing/starter-kits/README.md index 274421e13..548f9e553 100644 --- a/continuous-testing/starter-kits/README.md +++ b/continuous-testing/starter-kits/README.md @@ -2,6 +2,19 @@ This page contains starter kit information, which represent templates, code and configuration to help you get started quickly with continuous testing best practices described in this overall guide. Please see categories and links below for details. +## Static Application Security Testing + +This section contains links to sample actions, templates and configurations that analyze and validate code for security flaws and sensitive information. Identifying security vulnerabilities and sensitive data is an [OSS cybersecurity](https://www.cisa.gov/uscert/ncas/alerts/aa22-137a) [best practice](https://appel.nasa.gov/2022/06/30/spotlight-on-lessons-learned-open-source-and-commercial-web-software-vulnerabilities/). + +### Detect Secrets +A OSS tool for detecting sensitive information in project files. + +#### Automated Secret Scanning on Commit + + + + + ## Software Composition Analysis This section contains links to sample actions, templates and configurations that analyze and validate composition of Open Source Software (OSS) components in software systems. Identifying software and licensing vulnerabilites and ensuring routine software updates is an [OSS cybersecurity](https://www.cisa.gov/uscert/ncas/alerts/aa22-137a) [best practice](https://appel.nasa.gov/2022/06/30/spotlight-on-lessons-learned-open-source-and-commercial-web-software-vulnerabilities/). @@ -51,5 +64,3 @@ To leverage this template, make sure to do the following: Admin rights are necessary to modify `Code security and analysis` settings. 3. Modify [Dependabot configurations](https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file#configuration-options-for-the-dependabotyml-file) for your project: (optional) 1. If only security-related pull requests are desired, [set the `open-pull-requests-limit` property to `0` for `updates` block(s)](https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file#open-pull-requests-limit). - - From ef3d85babe747abb49c9db84d3c670f416ceb351 Mon Sep 17 00:00:00 2001 From: Jingchao Zhong <92573736+perryzjc@users.noreply.github.com> Date: Thu, 11 May 2023 16:20:24 -0700 Subject: [PATCH 02/16] Issue #89: Update documentation for detect-secrets Configuration files (yaml, baseline file, and plugins) are stored at another repository: https://github.com/NASA-AMMOS/slim-config-detect-secrets --- continuous-testing/starter-kits/README.md | 325 +++++++++++++++++++++- 1 file changed, 323 insertions(+), 2 deletions(-) diff --git a/continuous-testing/starter-kits/README.md b/continuous-testing/starter-kits/README.md index 548f9e553..d975e2c63 100644 --- a/continuous-testing/starter-kits/README.md +++ b/continuous-testing/starter-kits/README.md @@ -7,13 +7,334 @@ This page contains starter kit information, which represent templates, code and This section contains links to sample actions, templates and configurations that analyze and validate code for security flaws and sensitive information. Identifying security vulnerabilities and sensitive data is an [OSS cybersecurity](https://www.cisa.gov/uscert/ncas/alerts/aa22-137a) [best practice](https://appel.nasa.gov/2022/06/30/spotlight-on-lessons-learned-open-source-and-commercial-web-software-vulnerabilities/). ### Detect Secrets -A OSS tool for detecting sensitive information in project files. +[detect-secrets](https://github.com/Yelp/detect-secrets), an open-source tool employed and recommended by leading technology companies such as [Microsoft](https://microsoft.github.io/code-with-engineering-playbook/continuous-integration/dev-sec-ops/secret-management/recipes/detect-secrets/), [IBM](https://github.com/IBM/detect-secrets), and [Yelp](https://github.com/Yelp/detect-secrets), is instrumental in identifying sensitive information within project files. Its prowess lies in its extensible Python plugin API, which allows custom rules and heuristics to tackle a broad spectrum of secrets. Designed to scan the project's current state rather than the entire git history, +it operates swiftly, making it ideal for continuous integration pipelines. Leveraging the concept of a "baseline file" (`.secrets.baseline`), it enables easy handling of known secrets and false positives, facilitating its gradual integration into existing projects. -#### Automated Secret Scanning on Commit +```mermaid +mindmap + )Detect Secrets( + {{Open Source Tool}} + Quick_Scan((Quick Scan)) + Current_State{{Current State}} + CI_Pipelines{{Ideal for CI Pipelines}} + Baseline_File((Baseline File)) + Known_Secrets{{Handles Known Secrets}} + False_Positives{{Manages False Positives}} + Recommended_By((Recommended by)) + Microsoft{{Microsoft}} + IBM{{IBM}} + Yelp{{Yelp}} +``` +Our application of `detect-secrets` embraces a tri-layered approach, bolstered by customized plugins, to provide robust protection against potential secret leaks at the earliest stage. +This page proposes three layers of secret scanning to help preventing secrets from being leaked on GitHub +Three layers of protection are: +1. Client-side full scan of existing code base +2. Client-side scan of updated code upon Git commit +3. Server-side push to GitHub.com from client +Each layer has its own advantages and disadvantages. The full use of these three layers of protection is recommended to minimize human negligence. +```mermaid +flowchart TB + User([fa:fa-user User]) + + subgraph UserWorkflow["User Workflow to Secure Secrets"] + Layer1["1. Layer 1: GitHub.com (server-side)"] + Layer2["2. Layer 2: Git commit scan (client-side)"] + Layer3["3. Layer 3: Full scan (client-side)"] + + Layer1 -->|If Secrets Detected| Clean1[Purge or Fix the commit manually] + Layer2 -->|If Secrets Detected| Clean2[Clean local file directly.
Don't need to worry about cleaning commit history] + Layer3 -->|If Secrets Detected| Clean3[Clean local file directly.] + + Secure["Only Main branch is in safe.
Secrets are leaked on other branch before cleaning"] + Clean1 --> Secure + + SaveTime["It saves your time. And secrets are safe from GitHub"] + Clean2 --> SaveTime + Clean3 --> SaveTime + end + + User -->|At least use| Layer1 + User -->|Helpful to use| Layer2 + User -->|Optional to use| Layer3 + + style User fill:#F6F5F3,stroke:#333,stroke-width:1px + style UserWorkflow fill:#AF7AC5,stroke:#333,stroke-width:2px + style Layer1 fill:#F3B044,stroke:#333,stroke-width:2px,stroke-dasharray: 5 5 + style Layer2 fill:#F3B044,stroke:#333,stroke-width:2px,stroke-dasharray: 5 5 + style Layer3 fill:#F3B044,stroke:#333,stroke-width:2px,stroke-dasharray: 5 5 + style Clean1 fill:#5A88ED,stroke:#333,stroke-width:2px + style Clean2 fill:#5A88ED,stroke:#333,stroke-width:2px + style Clean3 fill:#5A88ED,stroke:#333,stroke-width:2px + style SaveTime fill:#5ABF9B,stroke:#333,stroke-width:2px + style Secure fill:#AF3034,stroke:#333,stroke-width:2px +``` +#### Layer 1: Full Scan and Audit (Client-side) +The first layer initiates a direct scan on the developer's local environment. This is achieved through the `detect-secrets` tool, which scans the entire codebase and outputs a new baseline file containing any detected secrets. The developer can then audit this file to view detailed information about any detected secrets. Detailed documentation for Layer 1 +```mermaid +sequenceDiagram + participant Dev as Developer + participant Env as Local Environment + participant DS as Detect-Secrets + participant File as Baseline File + participant Audit as Audit Tool + + Note over Dev,Env: Developer initiates a direct scan for secrets + Dev->>+Env: Triggers direct scan + Env->>+DS: Requests scan on the codebase + DS->>DS: Performs secret scanning + DS->>File: Generates new baseline file + File->>DS: Acknowledges file creation + DS-->>-Env: Returns scan results and new baseline file + Env-->>Dev: Presents scan results and new baseline file + Note over Dev,File: Developer may audit the new baseline file + Dev->>Audit: Initiates audit on the new baseline file + Audit->>File: Fetches details from the baseline file + File->>Audit: Returns secret details + Audit-->>Dev: Presents detailed information of detected secrets + +``` +Starter Kit: +1. Install [detect-secrets](https://github.com/Yelp/detect-secrets) +```bash +pip install detect-secrets +``` +##### 2. Install our additional configurations +cd to your project root directory +```bash +cd +``` + +Download our customized setup +```bash +git clone --depth 1 https://github.com/NASA-AMMOS/slim-config-detect-secrets.git temp_repo && \ +# Copy needed files and folders from the temporary folder to the current project directory +cp -r temp_repo/.detect-secrets . && \ +cp -r temp_repo/.pre-commit-config.yaml . && \ +# Remove the temporary folder +rm -rf temp_repo +``` + +After running this command, you will see a new folder `.detect-secrets` and a new file `.pre-commit-config.yaml` in your project root directory. + +In `.detect-secrets` folder, there are `.secrets.baseline` file and `plugins` folder. + +For example, +```text +├── .detect-secrets +│ ├── .secrets.baseline +│ └── plugins +│ ├── ... +│ ├── ... +│ ├── ... +├── .pre-commit-config.yaml +``` +> Note: We are in the progress of contributing these general plugins back to the `detect-secrets` community. +> Once they are accepted, configuration will be simplified. +3. Scan all local files from current directory and output the result as a baseline file +```bash +mkdir -p .detect-secrets \ +&& detect-secrets scan ./ \ + --all-files \ + --baseline .detect-secrets/.secrets.baseline \ + --exclude-files '^\.git(/.*)?$' \ + --exclude-files '^\.detect-secrets(/.*)?$' \ + -p .detect-secrets/plugins/absolute_filepath.py \ + -p .detect-secrets/plugins/aws_sensitive_info.py \ + -p .detect-secrets/plugins/email_address.py \ + -p .detect-secrets/plugins/ip_address.py +``` +This command will scan all files from current directory and output the result as a baseline file `.detect-secrets/.secrets.baseline`. +4. Check result in the baseline file +```bash +cat .detect-secrets/.secrets.baseline +``` +If any secrets are detected, the result will be located at the "results": of the file. + +For example, +```json +"results": { + "Jenkins/e2e-opera-pcm-develop.sh": [ + { + "type": "AWS Sensitive Information", + "filename": "Jenkins/e2e-opera-pcm-develop.sh", + "hashed_secret": "a2c61f79dd90429e5284ab45de3318c16756a2f0", + "is_verified": false, + "line_number": 39 +}, +``` +5. Analyze results by `audit` tool +```bash +detect-secrets audit .detect-secrets/.secrets.baseline +``` +> detect-secrets audit tool can provide you an interactive interface to view the actual secret based on the line number +> and label it as false positive or true positive. + +For example, + +Screen Shot 2023-04-20 at 7 08 08 AM + +#### Layer 2: Git Commit Scan (Client-side) +The second layer is a pre-commit hook implemented in the local environment. This hook utilizes a pre-configured `.pre-commit-config.yaml` file, which contains a baseline file for secret comparison. If any new secrets are detected during the commit process, the hook prevents the commit from being created. This acts as a safety net, ensuring no new secrets are accidentally committed. Detailed documentation for Layer 2 +```mermaid +sequenceDiagram + participant User as Developer + participant Local as Local Environment + participant Config as .pre-commit-config.yaml + participant PCH as Pre-commit Hook + participant DS as Detect-Secrets + participant File as Baseline File + + Note over User,Local: Developer attempts to commit + User->>+Local: Request commit + Local->>+Config: Fetches pre-commit config + Config->>PCH: Returns config with Detect-Secrets setup + PCH->>DS: Request secret scan with existing baseline + DS->>File: Fetches baseline file + File->>DS: Returns baseline file + DS->>DS: Scans changes for secrets with custom plugins + alt New Secrets Detected + DS-->>PCH: Returns detected secrets + PCH-->>Local: Prevents commit & reports detected secrets + Local-->>User: Prevents commit & reports detected secrets + else No New Secrets Detected + DS-->>PCH: Returns clean result + PCH-->>Local: Allows commit + Local-->>User: Commits changes + end + +``` +Starter Kit: +1. Install [pre-commit](https://pre-commit.com/#install) +```bash +pip install pre-commit +``` +2. [Install our additional configurations (same as layer 1)](#2-install-our-additional-configurations) +3. Install pre-commit hook +```bash +pre-commit install +``` +> This command will install a pre-commit hook in your local git repository based on the configurations in `.pre-commit-config.yaml` file. +4. Commit your changes + + Now, you can commit your changes as usual. If any new secrets are detected, the commit will be prevented and the secrets will be reported. + + For example, + + Screen Shot 2023-04-20 at 7 32 10 AM + + +> **Note**: pre-commit hook block commit by comparing new secrets with the results in `.secrets.baseline` file. If you want to add new secret results, you need to update `.secrets.baseline` file by re-running the scan command and generate a new baseline file. +> +> You can create an empty baseline file by running this command at a directory without secrets. + +#### Layer 3: Server-side Push to GitHub.com +The final layer of our solution is a server-side pre-commit scan powered by `pre-commit.ci`. This scan is triggered whenever a developer pushes to a branch or creates a pull request. It uses the same `.pre-commit-config.yaml` file as Layer 2, ensuring consistency between local and server-side checks. If the scan detects any new secrets, it returns a status check. If the target branch is protected, GitHub uses this status check to decide whether the merge or push can proceed. Detailed documentation for Layer 3 + +This multi-layered approach to secret scanning provides a robust and comprehensive secret detection system, reducing the risk of exposing sensitive information in your codebase. For a more in-depth explanation of each layer, please refer to the respective detailed documentation links. + +```mermaid +sequenceDiagram + participant User as Developer + participant GH as GitHub + participant Config as .pre-commit-config.yaml + participant CI as pre-commit CI + participant DS as Detect-Secrets + + Note over User,GH: Developer creates pull request or pushes to branch + User->>+GH: Creates pull request / pushes to branch + GH->>+Config: Fetches pre-commit config + Config->>CI: Returns config with Detect-Secrets setup + CI->>DS: Requests secret scan + DS->>DS: Scans pull request / branch for secrets with custom plugins + alt Secrets Detected + DS-->>CI: Returns detected secrets + CI-->>GH: Reports status check as failed + GH-->>User: Prevents merge / push & reports status check + else No Secrets Detected + DS-->>CI: Returns clean result + CI-->>GH: Reports status check as passed + GH-->>User: Allows merge / push + end + +``` +Starter Kit: +1. [Install our additional configurations (same as layer 1 and layer 2)](#2-install-our-additional-configurations) + +2. Register your repository on [pre-commit.ci](https://results.pre-commit.ci/) + Screen Shot 2023-04-20 at 8 53 23 AM + After this, every time you push to a branch or create a pull request, pre-commit.ci will run a scan and report the results as a status check. + + Screen Shot 2023-04-20 at 8 55 21 AM + + Screen Shot 2023-04-20 at 8 56 05 AM + +3. [Protect your branch](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/managing-a-branch-protection-rule) + + After this, if status check fails, GitHub will prevent the merge or push to the protected branch. + +#### Attention for using Detect Secrets +> 1.It does not [show all the same type of secrets in a same file to minimize noise](https://github.com/Yelp/detect-secrets/blob/master/docs/design.md#:~:text=Furthermore%2C%20this%20will%20not%20flag%20on%20every%20single%20usage%20of%20a%20given%20secret%20in%20a%20given%20file%2C%20to%20minimize%20noise.) +> +> This means sometimes it will only show one secret in a file even if there are multiple **same type of secrets** in the same file. +> Audit tool will not show all the secrets due to this reason. +> +> **->** Thus, when you see a secret is detected, best practice is to **manually** check that file. + +> 2.Even though detect-secrets has strong secret-detect ability compared to other tools, it is still possible that detect-secrets will not show you a file that contains secrets due to a new type of secret not capable by current plugins. +> +> **->** Thus, the best practice is always to be careful as a developer and **manually** check the files that you think might contain secrets. +>> `detect-secrets` is a backup approach to minimize the chance of pushing secrets to the cloud. + +#### Recommended Workflow +1. At least use layer 3 (Server-side push to GitHub.com) to protect the main branch from being pushed or merged if any secrets are detected. +2. If any secrets are detected during layer 3, you can: + - Clean the commit history of the branch + - To find out the files that needs clean, you can use layer 1's auditing feature as assistance + - If a secret has already been committed, visit + https://help.github.com/articles/removing-sensitive-data-from-a-repository +3. Recommend to set up layer 2 (Git commit scan, client-side) for every developer + - It can minimize the chance of pushing secrets to the cloud + - Local files are easier to clean than GitHub commit history +4. Layer 1 (Full scan and audit, client-side) can be involved during each stage + - It helps you generate, update or analyze baseline file for layer 2 and 3 + +```mermaid +sequenceDiagram + participant Dev as Developer + participant GH as GitHub + participant DS as Detect-Secrets + participant L1 as Layer 1 (Full Scan & Audit) + participant L2 as Layer 2 (Git Commit Scan) + participant L3 as Layer 3 (Server-side Push to GitHub) + + Dev->>L3: Push/Merge to Main Branch + L3->>DS: Scan for Secrets + alt Secrets Detected in L3 + DS-->>Dev: Secrets Detected + Dev->>L1: Use Auditing Feature to Identify Files for Cleaning + Dev->>Dev: Clean Commit History + Note over Dev: If a secret has already been committed, visit:
https://help.github.com/articles/removing-sensitive-data-from-a-repository + Dev->>L2: Set Up Git Commit Scan + Note over Dev, L2: Minimize the chance of pushing secrets
Easier to clean local files than GitHub commit history + Dev->>L1: Involve Full Scan & Audit in Each Stage + Note over Dev, L1: Helps generate, update or analyze baseline file for L2 and L3 + Dev->>L3: Retry Push/Merge to Main Branch + else No Secrets Detected + DS-->>GH: No Secrets Detected + GH->>Dev: Allows Push/Merge to Main Branch + end + +``` + +#### More Configurations +Check out +1. [detect-secrets](https://github.com/Yelp/detect-secrets) +2. [pre-commit](https://pre-commit.com/) +3. [pre-commit.ci](https://pre-commit.ci/) ## Software Composition Analysis From fadf55e983322615b70050595a8b78c631d02d9a Mon Sep 17 00:00:00 2001 From: Jingchao Zhong <92573736+perryzjc@users.noreply.github.com> Date: Thu, 11 May 2023 17:05:57 -0700 Subject: [PATCH 03/16] Issue NASA-AMMOS#89: Update documentation for detect-secrets Configuration files (yaml, baseline file, and plugins) are stored at another repository: https://github.com/NASA-AMMOS/slim-config-detect-secrets --- continuous-testing/starter-kits/README.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/continuous-testing/starter-kits/README.md b/continuous-testing/starter-kits/README.md index d975e2c63..728f5625b 100644 --- a/continuous-testing/starter-kits/README.md +++ b/continuous-testing/starter-kits/README.md @@ -315,13 +315,14 @@ sequenceDiagram L3->>DS: Scan for Secrets alt Secrets Detected in L3 DS-->>Dev: Secrets Detected + Note over Dev: Manually check the file for same type of secrets Dev->>L1: Use Auditing Feature to Identify Files for Cleaning Dev->>Dev: Clean Commit History - Note over Dev: If a secret has already been committed, visit:
https://help.github.com/articles/removing-sensitive-data-from-a-repository + Note over Dev: If a secret has already been committed, refer:
https://help.github.com/articles/removing-sensitive-data-from-a-repository Dev->>L2: Set Up Git Commit Scan - Note over Dev, L2: Minimize the chance of pushing secrets
Easier to clean local files than GitHub commit history - Dev->>L1: Involve Full Scan & Audit in Each Stage - Note over Dev, L1: Helps generate, update or analyze baseline file for L2 and L3 + Note over Dev, L2: Minimizes chances of pushing secrets
Easier to clean local files than GitHub commit history + Dev->>L1: Use Full Scan & Audit at Each Stage + Note over Dev, L1: Assists in generating, updating or analyzing baseline file for L2 and L3 Dev->>L3: Retry Push/Merge to Main Branch else No Secrets Detected DS-->>GH: No Secrets Detected From d20ee6134a0dc0e0dab11d2d2570e358ef7e4550 Mon Sep 17 00:00:00 2001 From: Jingchao Zhong <92573736+perryzjc@users.noreply.github.com> Date: Fri, 19 May 2023 02:56:16 -0700 Subject: [PATCH 04/16] Issue #89: Update documentation for detect-secrets --- continuous-testing/starter-kits/README.md | 246 ++++++++++++++-------- 1 file changed, 154 insertions(+), 92 deletions(-) diff --git a/continuous-testing/starter-kits/README.md b/continuous-testing/starter-kits/README.md index 728f5625b..80aecb695 100644 --- a/continuous-testing/starter-kits/README.md +++ b/continuous-testing/starter-kits/README.md @@ -27,7 +27,7 @@ mindmap ``` -Our application of `detect-secrets` embraces a tri-layered approach, bolstered by customized plugins, to provide robust protection against potential secret leaks at the earliest stage. +Our application of `detect-secrets` embraces a tri-layered approach, bolstered by customized plugins ([full list of plugins available here](https://github.com/NASA-AMMOS/slim-detect-secrets/tree/exp#viewing-all-enabled-plugins)), to provide robust protection against potential secret leaks at the earliest stage. This page proposes three layers of secret scanning to help preventing secrets from being leaked on GitHub Three layers of protection are: @@ -41,15 +41,15 @@ flowchart TB User([fa:fa-user User]) subgraph UserWorkflow["User Workflow to Secure Secrets"] - Layer1["1. Layer 1: GitHub.com (server-side)"] - Layer2["2. Layer 2: Git commit scan (client-side)"] - Layer3["3. Layer 3: Full scan (client-side)"] + Layer1["Layer 1: Full scan (client-side)"] + Layer2["Layer 2: Git commit scan (client-side)"] + Layer3["Layer 3: GitHub.com (server-side)"] - Layer1 -->|If Secrets Detected| Clean1[Purge or Fix the commit manually] + Layer1 -->|If Secrets Detected| Clean3[Clean local file directly.] Layer2 -->|If Secrets Detected| Clean2[Clean local file directly.
Don't need to worry about cleaning commit history] - Layer3 -->|If Secrets Detected| Clean3[Clean local file directly.] + Layer3 -->|If Secrets Detected| Clean1[Purge or Fix the commit manually] - Secure["Only Main branch is in safe.
Secrets are leaked on other branch before cleaning"] + Secure["Only GitHub-Protected branch is in safe.
Secrets are leaked on other branch before cleaning"] Clean1 --> Secure SaveTime["It saves your time. And secrets are safe from GitHub"] @@ -72,8 +72,22 @@ flowchart TB style SaveTime fill:#5ABF9B,stroke:#333,stroke-width:2px style Secure fill:#AF3034,stroke:#333,stroke-width:2px ``` +> **Note**: Below three layers, are running on experimental version [slim-detect-secrets](https://github.com/NASA-AMMOS/slim-detect-secrets/tree/exp) which supports additional secret detection [plugins](https://github.com/NASA-AMMOS/slim-detect-secrets/tree/exp#viewing-all-enabled-plugins). +> +> They are: +> > * [AWS sensitive information]() (click for more information) +> > * Public IP Address +> > * Absolute Path +> > * Email Address +> > +> > Link to their [implementation](https://github.com/NASA-AMMOS/slim-detect-secrets/tree/exp/detect_secrets/plugins) and [test suites](https://github.com/NASA-AMMOS/slim-detect-secrets/tree/exp/tests/plugins) +> +> It is being tested by both [NASA-AMMOS/slim](https://github.com/NASA-AMMOS/slim) team and [Yelp/detect-secrets](https://github.com/Yelp/detect-secrets) team. +> Eventually, it will be merged into Yelp/detect-secrets. +> +> At that time, this document will be updated to use the official version of detect-secrets. #### Layer 1: Full Scan and Audit (Client-side) -The first layer initiates a direct scan on the developer's local environment. This is achieved through the `detect-secrets` tool, which scans the entire codebase and outputs a new baseline file containing any detected secrets. The developer can then audit this file to view detailed information about any detected secrets. Detailed documentation for Layer 1 +The first layer initiates a direct scan on the developer's local environment. This is achieved through the `detect-secrets` tool, which scans the entire codebase and outputs a new baseline file containing detected secrets. The developer can then audit this file to view detailed information about detected secrets. ```mermaid sequenceDiagram participant Dev as Developer @@ -98,61 +112,23 @@ sequenceDiagram ``` Starter Kit: -1. Install [detect-secrets](https://github.com/Yelp/detect-secrets) +1. Install experimental version of [slim-detect-secrets](https://github.com/NASA-AMMOS/slim-detect-secrets/tree/exp) ```bash -pip install detect-secrets -``` -##### 2. Install our additional configurations -cd to your project root directory -```bash -cd +pip install git+https://github.com/NASA-AMMOS/slim-detect-secrets.git@exp ``` -Download our customized setup +2. Scan all local files from current directory and output the result as a baseline file ```bash -git clone --depth 1 https://github.com/NASA-AMMOS/slim-config-detect-secrets.git temp_repo && \ -# Copy needed files and folders from the temporary folder to the current project directory -cp -r temp_repo/.detect-secrets . && \ -cp -r temp_repo/.pre-commit-config.yaml . && \ -# Remove the temporary folder -rm -rf temp_repo +detect-secrets scan ./ --all-files --exclude-files '.secrets.*' --exclude-files '.git*' > .secrets.baseline ``` +Here it scans all the local files from current directory, but excludes `.git` directory and `.secrets` baseline files from scanning to reduce false positive. Make modifications when necessary. -After running this command, you will see a new folder `.detect-secrets` and a new file `.pre-commit-config.yaml` in your project root directory. - -In `.detect-secrets` folder, there are `.secrets.baseline` file and `plugins` folder. - -For example, -```text -├── .detect-secrets -│ ├── .secrets.baseline -│ └── plugins -│ ├── ... -│ ├── ... -│ ├── ... -├── .pre-commit-config.yaml -``` -> Note: We are in the progress of contributing these general plugins back to the `detect-secrets` community. -> Once they are accepted, configuration will be simplified. -3. Scan all local files from current directory and output the result as a baseline file -```bash -mkdir -p .detect-secrets \ -&& detect-secrets scan ./ \ - --all-files \ - --baseline .detect-secrets/.secrets.baseline \ - --exclude-files '^\.git(/.*)?$' \ - --exclude-files '^\.detect-secrets(/.*)?$' \ - -p .detect-secrets/plugins/absolute_filepath.py \ - -p .detect-secrets/plugins/aws_sensitive_info.py \ - -p .detect-secrets/plugins/email_address.py \ - -p .detect-secrets/plugins/ip_address.py -``` -This command will scan all files from current directory and output the result as a baseline file `.detect-secrets/.secrets.baseline`. -4. Check result in the baseline file +3. Check result in the baseline file ```bash -cat .detect-secrets/.secrets.baseline +cat .secrets.baseline ``` -If any secrets are detected, the result will be located at the "results": of the file. +> This command is supported in Unix-like system. For Windows, you can use other command, such as `type .secrets.baseline` +If any secrets are detected, the result will be located at the "results:" of the file. For example, ```json @@ -166,19 +142,22 @@ For example, "line_number": 39 }, ``` -5. Analyze results by `audit` tool +Only line number is visible through this approach. You can check them manually in the file, or use the following audit tool to conveniently view the actual secret. + +4. Analyze results by `audit` tool ```bash -detect-secrets audit .detect-secrets/.secrets.baseline +detect-secrets audit .secrets.baseline ``` > detect-secrets audit tool can provide you an interactive interface to view the actual secret based on the line number > and label it as false positive or true positive. For example, -Screen Shot 2023-04-20 at 7 08 08 AM + +It also provides other ways to present the result. For more information, please refer to [Auditing Secrets in Baseline](https://github.com/Yelp/detect-secrets#auditing-secrets-in-baseline) #### Layer 2: Git Commit Scan (Client-side) -The second layer is a pre-commit hook implemented in the local environment. This hook utilizes a pre-configured `.pre-commit-config.yaml` file, which contains a baseline file for secret comparison. If any new secrets are detected during the commit process, the hook prevents the commit from being created. This acts as a safety net, ensuring no new secrets are accidentally committed. Detailed documentation for Layer 2 +The second layer is a pre-commit hook implemented in the local environment. This hook utilizes a `.pre-commit-config.yaml` file to config the pre-commit hook. The hook is triggered when the developer attempts to commit changes. The hook will scan the changes and **compare** them to the baseline file generated in the first layer. If any **new secrets** are detected, the hook will prevent the commit and report the detected secrets to the developer. ```mermaid sequenceDiagram participant User as Developer @@ -212,69 +191,153 @@ Starter Kit: ```bash pip install pre-commit ``` -2. [Install our additional configurations (same as layer 1)](#2-install-our-additional-configurations) +This tool is used to install pre-commit hook in your local git repository. + +2. Create `.pre-commit-config.yaml` file in root directory of your project +```yaml +repos: + - repo: https://github.com/NASA-AMMOS/slim-detect-secrets + # using commit id for now, will change to tag when official version is released + rev: 91e097ad4559ae6ab785c883dc5ed989202c7fbe + hooks: + - id: detect-secrets + args: + - '--baseline' + - '.secrets.baseline' + - '--exclude-files' + - '.git*' + - '--exclude-files' + - '.secrets.*' + +``` +This file is used to config the pre-commit hook. In this example, we use the experimental version of `slim-detect-secrets` tool. The `--baseline` argument is used to specify the baseline file generated in the first layer. The `--exclude-files` argument is used to exclude the `.git` directory and `.secrets` baseline files from scanning to reduce false positives. Make modifications when necessary. + 3. Install pre-commit hook ```bash pre-commit install ``` -> This command will install a pre-commit hook in your local git repository based on the configurations in `.pre-commit-config.yaml` file. -4. Commit your changes +This command reads the `.pre-commit-config.yaml` file and installs the pre-commit hook in your local git repository. +After this, you can see a `.git/hooks/pre-commit` file is created in your local git repository. - Now, you can commit your changes as usual. If any new secrets are detected, the commit will be prevented and the secrets will be reported. +4. Make sure you have the baseline file `.secrets.baseline` in your local git repository - For example, +5. Commit your changes - Screen Shot 2023-04-20 at 7 32 10 AM +Now, you can commit your changes as usual. If any **new secrets** are detected, the commit will be prevented and the secrets will be reported. + +For example, + +Screen Shot 2023-04-20 at 7 32 10 AM > **Note**: pre-commit hook block commit by comparing new secrets with the results in `.secrets.baseline` file. If you want to add new secret results, you need to update `.secrets.baseline` file by re-running the scan command and generate a new baseline file. > -> You can create an empty baseline file by running this command at a directory without secrets. +> You can create an empty result baseline file by running this command at a directory without secrets. #### Layer 3: Server-side Push to GitHub.com -The final layer of our solution is a server-side pre-commit scan powered by `pre-commit.ci`. This scan is triggered whenever a developer pushes to a branch or creates a pull request. It uses the same `.pre-commit-config.yaml` file as Layer 2, ensuring consistency between local and server-side checks. If the scan detects any new secrets, it returns a status check. If the target branch is protected, GitHub uses this status check to decide whether the merge or push can proceed. Detailed documentation for Layer 3 - -This multi-layered approach to secret scanning provides a robust and comprehensive secret detection system, reducing the risk of exposing sensitive information in your codebase. For a more in-depth explanation of each layer, please refer to the respective detailed documentation links. +The final layer of our solution is a server-side pre-commit scan powered by [GitHub Action](https://github.com/features/actions). This scan is triggered whenever a developer pushes to a branch or creates a pull request. If the scan detects any new secrets, it can generate a not detailed report compared to layer 2 (for security concern), email to the developer, and report a status check to GitHub. The status check will prevent the developer from merging the pull request or pushing to the **protected** branch. This layer protects the protected branch from being polluted by secrets, but secrets can still be pushed to other branches. ```mermaid sequenceDiagram participant User as Developer participant GH as GitHub - participant Config as .pre-commit-config.yaml - participant CI as pre-commit CI + participant Workflow as detect-secrets.yaml + participant GA as GitHub Action participant DS as Detect-Secrets Note over User,GH: Developer creates pull request or pushes to branch User->>+GH: Creates pull request / pushes to branch - GH->>+Config: Fetches pre-commit config - Config->>CI: Returns config with Detect-Secrets setup - CI->>DS: Requests secret scan - DS->>DS: Scans pull request / branch for secrets with custom plugins + GH->>+Workflow: Triggers GitHub Action workflow + Workflow->>GA: Sets up and runs Detect-Secrets scan + GA->>DS: Requests secret scan + DS->>DS: Scans repository for secrets alt Secrets Detected - DS-->>CI: Returns detected secrets - CI-->>GH: Reports status check as failed - GH-->>User: Prevents merge / push & reports status check + DS-->>GA: Returns detected secrets + GA-->>GH: Fails status check + GH-->>User: Prevents merge / push & sends email notification else No Secrets Detected - DS-->>CI: Returns clean result - CI-->>GH: Reports status check as passed + DS-->>GA: Returns clean result + GA-->>GH: Passes status check GH-->>User: Allows merge / push end ``` Starter Kit: -1. [Install our additional configurations (same as layer 1 and layer 2)](#2-install-our-additional-configurations) +1. Create a workflow file `detect-secrets.yaml` in `.github/workflows` directory from your repository root. +```yaml +name: Secret Detection Workflow +on: + push: + branches: + - main + pull_request: + branches: + - main + +jobs: + secret-detection: + runs-on: ubuntu-latest + steps: + - name: Checkout code + uses: actions/checkout@v2 + + - name: Install necessary packages + run: | + # experimental version of slim-detect-secrets + pip install git+https://github.com/NASA-AMMOS/slim-detect-secrets.git@exp + pip install jq + + - name: Scan repository for secrets + run: | + # scripts to scan repository for new secrets + + # backup the list of known secrets + cp .secrets.baseline .secrets.new + + # find the secrets in the repository + detect-secrets scan --baseline .secrets.new --exclude-files '.secrets.*' --exclude-files '.git*' + + # if there is any difference between the known and newly detected secrets, break the build + # Function to compare secrets without listing them + compare_secrets() { diff <(jq -r '.results | keys[] as $key | "\($key),\(.[$key] | .[] | .hashed_secret)"' "$1" | sort) <(jq -r '.results | keys[] as $key | "\($key),\(.[$key] | .[] | .hashed_secret)"' "$2" | sort) >/dev/null; } + + # Check if there's any difference between the known and newly detected secrets + if ! compare_secrets .secrets.baseline .secrets.new; then + echo "⚠️ Attention Required! ⚠️" >&2 + echo "New secrets have been detected in your recent commit. Due to security concerns, we cannot display detailed information here and we cannot proceed until this issue is resolved." >&2 + echo "" >&2 + echo "Please follow the steps below on your local machine to reveal and handle the secrets:" >&2 + echo "" >&2 + echo "1️⃣ Run the 'detect-secrets' tool on your local machine. This tool will identify and clean up the secrets. You can find detailed instructions at this link: https://nasa-ammos.github.io/slim/continuous-testing/starter-kits/#detect-secrets" >&2 + echo "" >&2 + echo "2️⃣ After cleaning up the secrets, commit your changes and re-push your update to the repository." >&2 + echo "" >&2 + echo "Your efforts to maintain the security of our codebase are greatly appreciated!" >&2 + exit 1 + fi + +``` +This file is used to config the GitHub Action workflow. After this, GitHub will automatically run the workflow when you push to the branch or create a pull request. +This template currently only supports `main` branch. If you want to use it for other branches, you need to change the `on` section. + +This workflow will run the `detect-secrets` tool on the GitHub server. If any new secrets are detected, it will: +- Fail the status check + + +- Print a message in the "Details" with instructions on how to resolve the issue + + +- Send an email notification to the user + + -2. Register your repository on [pre-commit.ci](https://results.pre-commit.ci/) - Screen Shot 2023-04-20 at 8 53 23 AM - After this, every time you push to a branch or create a pull request, pre-commit.ci will run a scan and report the results as a status check. +2. [Protect your branch](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/managing-a-branch-protection-rule) - Screen Shot 2023-04-20 at 8 55 21 AM - - Screen Shot 2023-04-20 at 8 56 05 AM + -3. [Protect your branch](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/managing-a-branch-protection-rule) +After this, if status check fails, GitHub will prevent the merge or push to the protected branch. - After this, if status check fails, GitHub will prevent the merge or push to the protected branch. + #### Attention for using Detect Secrets > 1.It does not [show all the same type of secrets in a same file to minimize noise](https://github.com/Yelp/detect-secrets/blob/master/docs/design.md#:~:text=Furthermore%2C%20this%20will%20not%20flag%20on%20every%20single%20usage%20of%20a%20given%20secret%20in%20a%20given%20file%2C%20to%20minimize%20noise.) @@ -287,7 +350,7 @@ Starter Kit: > 2.Even though detect-secrets has strong secret-detect ability compared to other tools, it is still possible that detect-secrets will not show you a file that contains secrets due to a new type of secret not capable by current plugins. > > **->** Thus, the best practice is always to be careful as a developer and **manually** check the files that you think might contain secrets. ->> `detect-secrets` is a backup approach to minimize the chance of pushing secrets to the cloud. +>> **Note**: `detect-secrets` is a backup approach to minimize the chance of pushing secrets to the cloud. #### Recommended Workflow 1. At least use layer 3 (Server-side push to GitHub.com) to protect the main branch from being pushed or merged if any secrets are detected. @@ -315,8 +378,8 @@ sequenceDiagram L3->>DS: Scan for Secrets alt Secrets Detected in L3 DS-->>Dev: Secrets Detected - Note over Dev: Manually check the file for same type of secrets Dev->>L1: Use Auditing Feature to Identify Files for Cleaning + Note over L1: Assists in identifying files that need to be cleaned.
Recommend to manully double check detected files
due to "minimize noise" feature from detect-secrets Dev->>Dev: Clean Commit History Note over Dev: If a secret has already been committed, refer:
https://help.github.com/articles/removing-sensitive-data-from-a-repository Dev->>L2: Set Up Git Commit Scan @@ -335,7 +398,6 @@ sequenceDiagram Check out 1. [detect-secrets](https://github.com/Yelp/detect-secrets) 2. [pre-commit](https://pre-commit.com/) -3. [pre-commit.ci](https://pre-commit.ci/) ## Software Composition Analysis From d73d37ff9ae6a2d1afaed6117aebeb928af8aebc Mon Sep 17 00:00:00 2001 From: Jingchao Zhong <92573736+perryzjc@users.noreply.github.com> Date: Wed, 7 Jun 2023 08:42:54 -0700 Subject: [PATCH 05/16] Reorder the graph to go from layer 1 in the left to layer 3 in the right Solve this suggestion: https://github.com/NASA-AMMOS/slim/pull/95#discussion_r1212447090 --- continuous-testing/starter-kits/README.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/continuous-testing/starter-kits/README.md b/continuous-testing/starter-kits/README.md index 80aecb695..943dae17c 100644 --- a/continuous-testing/starter-kits/README.md +++ b/continuous-testing/starter-kits/README.md @@ -45,23 +45,23 @@ flowchart TB Layer2["Layer 2: Git commit scan (client-side)"] Layer3["Layer 3: GitHub.com (server-side)"] - Layer1 -->|If Secrets Detected| Clean3[Clean local file directly.] + Layer1 -->|If Secrets Detected| Clean1[Clean local file directly.] Layer2 -->|If Secrets Detected| Clean2[Clean local file directly.
Don't need to worry about cleaning commit history] - Layer3 -->|If Secrets Detected| Clean1[Purge or Fix the commit manually] - - Secure["Only GitHub-Protected branch is in safe.
Secrets are leaked on other branch before cleaning"] - Clean1 --> Secure + Layer3 -->|If Secrets Detected| Clean3[Purge or Fix the commit manually] SaveTime["It saves your time. And secrets are safe from GitHub"] + Clean1 --> SaveTime Clean2 --> SaveTime - Clean3 --> SaveTime + + Secure["Only GitHub-Protected branch is in safe.
Secrets are leaked on other branch before cleaning"] + Clean3--> Secure end User -->|At least use| Layer1 User -->|Helpful to use| Layer2 User -->|Optional to use| Layer3 - style User fill:#F6F5F3,stroke:#333,stroke-width:1px + style User fill:#F6F5F3,stroke:#333,stroke-width:1px style UserWorkflow fill:#AF7AC5,stroke:#333,stroke-width:2px style Layer1 fill:#F3B044,stroke:#333,stroke-width:2px,stroke-dasharray: 5 5 style Layer2 fill:#F3B044,stroke:#333,stroke-width:2px,stroke-dasharray: 5 5 @@ -71,6 +71,7 @@ flowchart TB style Clean3 fill:#5A88ED,stroke:#333,stroke-width:2px style SaveTime fill:#5ABF9B,stroke:#333,stroke-width:2px style Secure fill:#AF3034,stroke:#333,stroke-width:2px + ``` > **Note**: Below three layers, are running on experimental version [slim-detect-secrets](https://github.com/NASA-AMMOS/slim-detect-secrets/tree/exp) which supports additional secret detection [plugins](https://github.com/NASA-AMMOS/slim-detect-secrets/tree/exp#viewing-all-enabled-plugins). > From 981e6fd7ce796983670f4562963d095735d4e795 Mon Sep 17 00:00:00 2001 From: Jingchao Zhong <92573736+perryzjc@users.noreply.github.com> Date: Wed, 7 Jun 2023 08:43:25 -0700 Subject: [PATCH 06/16] Fix syntax Co-authored-by: Rishi Verma --- continuous-testing/starter-kits/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/continuous-testing/starter-kits/README.md b/continuous-testing/starter-kits/README.md index 943dae17c..e4793c0c9 100644 --- a/continuous-testing/starter-kits/README.md +++ b/continuous-testing/starter-kits/README.md @@ -28,7 +28,7 @@ mindmap ``` Our application of `detect-secrets` embraces a tri-layered approach, bolstered by customized plugins ([full list of plugins available here](https://github.com/NASA-AMMOS/slim-detect-secrets/tree/exp#viewing-all-enabled-plugins)), to provide robust protection against potential secret leaks at the earliest stage. -This page proposes three layers of secret scanning to help preventing secrets from being leaked on GitHub +This page proposes three layers of secret scanning to help prevent secrets from being leaked on GitHub Three layers of protection are: 1. Client-side full scan of existing code base From 067b14fa8d82c3c21d6a570600404ae77ae58ce2 Mon Sep 17 00:00:00 2001 From: Jingchao Zhong <92573736+perryzjc@users.noreply.github.com> Date: Wed, 7 Jun 2023 09:01:45 -0700 Subject: [PATCH 07/16] Randomize sample sensitive information in the doc Resolve this conversation: https://github.com/NASA-AMMOS/slim/pull/95#discussion_r1212450767 --- continuous-testing/starter-kits/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/continuous-testing/starter-kits/README.md b/continuous-testing/starter-kits/README.md index e4793c0c9..d641f2d51 100644 --- a/continuous-testing/starter-kits/README.md +++ b/continuous-testing/starter-kits/README.md @@ -134,11 +134,11 @@ If any secrets are detected, the result will be located at the "results:" of the For example, ```json "results": { - "Jenkins/e2e-opera-pcm-develop.sh": [ + "Sample/sample-develop.sh": [ { "type": "AWS Sensitive Information", - "filename": "Jenkins/e2e-opera-pcm-develop.sh", - "hashed_secret": "a2c61f79dd90429e5284ab45de3318c16756a2f0", + "filename": "Sample/sample-develop.sh", + "hashed_secret": "d3f72b97cd8756fe12345678bdf96aef1ac23b7d", "is_verified": false, "line_number": 39 }, From 61a997ad50af3d2dc8f30833758d9ff26dc409dd Mon Sep 17 00:00:00 2001 From: Jingchao Zhong <92573736+perryzjc@users.noreply.github.com> Date: Wed, 7 Jun 2023 09:09:00 -0700 Subject: [PATCH 08/16] Update continuous-testing/starter-kits/README.md to be more specific Co-authored-by: Rishi Verma --- continuous-testing/starter-kits/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/continuous-testing/starter-kits/README.md b/continuous-testing/starter-kits/README.md index d641f2d51..b7a1ee204 100644 --- a/continuous-testing/starter-kits/README.md +++ b/continuous-testing/starter-kits/README.md @@ -194,7 +194,7 @@ pip install pre-commit ``` This tool is used to install pre-commit hook in your local git repository. -2. Create `.pre-commit-config.yaml` file in root directory of your project +2. Create `.pre-commit-config.yaml` file in root directory of your project with the following contents ```yaml repos: - repo: https://github.com/NASA-AMMOS/slim-detect-secrets From fc79bf7dc376b4d5f6bf2a6f7d9c02998b27fda1 Mon Sep 17 00:00:00 2001 From: Jingchao Zhong <92573736+perryzjc@users.noreply.github.com> Date: Thu, 8 Jun 2023 05:47:44 -0700 Subject: [PATCH 09/16] Update README.md to clarify layer 2 To resolve this conversation: https://github.com/NASA-AMMOS/slim/pull/95#discussion_r1212452316 --- continuous-testing/starter-kits/README.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/continuous-testing/starter-kits/README.md b/continuous-testing/starter-kits/README.md index b7a1ee204..98d7b58dd 100644 --- a/continuous-testing/starter-kits/README.md +++ b/continuous-testing/starter-kits/README.md @@ -230,9 +230,8 @@ For example, Screen Shot 2023-04-20 at 7 32 10 AM - -> **Note**: pre-commit hook block commit by comparing new secrets with the results in `.secrets.baseline` file. If you want to add new secret results, you need to update `.secrets.baseline` file by re-running the scan command and generate a new baseline file. -> +> **Note**: The pre-commit hook blocks a commit by comparing new secrets with the results in the `.secrets.baseline` file. If new secrets are introduced, the hook will report them, but it does not automatically update the `.secrets.baseline` file. To update the baseline file with newly introduced secrets, you need to re-run the scan command in Layer 1 (step 2) and generate a new baseline file. +> > You can create an empty result baseline file by running this command at a directory without secrets. #### Layer 3: Server-side Push to GitHub.com From 9973401268a47e26bd92c619ee49f208f707e40b Mon Sep 17 00:00:00 2001 From: Jingchao Zhong <92573736+perryzjc@users.noreply.github.com> Date: Thu, 8 Jun 2023 06:23:18 -0700 Subject: [PATCH 10/16] Simplify README.md about layer 3 --- continuous-testing/starter-kits/README.md | 57 +---------------------- 1 file changed, 2 insertions(+), 55 deletions(-) diff --git a/continuous-testing/starter-kits/README.md b/continuous-testing/starter-kits/README.md index 98d7b58dd..55848b207 100644 --- a/continuous-testing/starter-kits/README.md +++ b/continuous-testing/starter-kits/README.md @@ -263,62 +263,9 @@ sequenceDiagram ``` Starter Kit: -1. Create a workflow file `detect-secrets.yaml` in `.github/workflows` directory from your repository root. -```yaml -name: Secret Detection Workflow -on: - push: - branches: - - main - pull_request: - branches: - - main - -jobs: - secret-detection: - runs-on: ubuntu-latest - steps: - - name: Checkout code - uses: actions/checkout@v2 - - - name: Install necessary packages - run: | - # experimental version of slim-detect-secrets - pip install git+https://github.com/NASA-AMMOS/slim-detect-secrets.git@exp - pip install jq - - - name: Scan repository for secrets - run: | - # scripts to scan repository for new secrets - - # backup the list of known secrets - cp .secrets.baseline .secrets.new - - # find the secrets in the repository - detect-secrets scan --baseline .secrets.new --exclude-files '.secrets.*' --exclude-files '.git*' - - # if there is any difference between the known and newly detected secrets, break the build - # Function to compare secrets without listing them - compare_secrets() { diff <(jq -r '.results | keys[] as $key | "\($key),\(.[$key] | .[] | .hashed_secret)"' "$1" | sort) <(jq -r '.results | keys[] as $key | "\($key),\(.[$key] | .[] | .hashed_secret)"' "$2" | sort) >/dev/null; } - - # Check if there's any difference between the known and newly detected secrets - if ! compare_secrets .secrets.baseline .secrets.new; then - echo "⚠️ Attention Required! ⚠️" >&2 - echo "New secrets have been detected in your recent commit. Due to security concerns, we cannot display detailed information here and we cannot proceed until this issue is resolved." >&2 - echo "" >&2 - echo "Please follow the steps below on your local machine to reveal and handle the secrets:" >&2 - echo "" >&2 - echo "1️⃣ Run the 'detect-secrets' tool on your local machine. This tool will identify and clean up the secrets. You can find detailed instructions at this link: https://nasa-ammos.github.io/slim/continuous-testing/starter-kits/#detect-secrets" >&2 - echo "" >&2 - echo "2️⃣ After cleaning up the secrets, commit your changes and re-push your update to the repository." >&2 - echo "" >&2 - echo "Your efforts to maintain the security of our codebase are greatly appreciated!" >&2 - exit 1 - fi +1. Create a GitHub Action workflow file in `.github/workflows/` directory from your repository root. Visit the [detect-secrets Action](https://github.com/marketplace/actions/detect-secrets-action) in the GitHub Actions Marketplace for details on how to add it to your repository. -``` -This file is used to config the GitHub Action workflow. After this, GitHub will automatically run the workflow when you push to the branch or create a pull request. -This template currently only supports `main` branch. If you want to use it for other branches, you need to change the `on` section. +After this, GitHub will automatically run the workflow when you push to the branch or create a pull request. This workflow will run the `detect-secrets` tool on the GitHub server. If any new secrets are detected, it will: - Fail the status check From de962f1162cc30df7277eff670c49fad8bf4c72b Mon Sep 17 00:00:00 2001 From: Jingchao Zhong <92573736+perryzjc@users.noreply.github.com> Date: Fri, 9 Jun 2023 17:08:29 -0700 Subject: [PATCH 11/16] Update README.md removing company information Refer to this conversation: https://github.com/NASA-AMMOS/slim/pull/95#discussion_r1224529636 --- continuous-testing/starter-kits/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/continuous-testing/starter-kits/README.md b/continuous-testing/starter-kits/README.md index 55848b207..dbe960099 100644 --- a/continuous-testing/starter-kits/README.md +++ b/continuous-testing/starter-kits/README.md @@ -7,7 +7,7 @@ This page contains starter kit information, which represent templates, code and This section contains links to sample actions, templates and configurations that analyze and validate code for security flaws and sensitive information. Identifying security vulnerabilities and sensitive data is an [OSS cybersecurity](https://www.cisa.gov/uscert/ncas/alerts/aa22-137a) [best practice](https://appel.nasa.gov/2022/06/30/spotlight-on-lessons-learned-open-source-and-commercial-web-software-vulnerabilities/). ### Detect Secrets -[detect-secrets](https://github.com/Yelp/detect-secrets), an open-source tool employed and recommended by leading technology companies such as [Microsoft](https://microsoft.github.io/code-with-engineering-playbook/continuous-integration/dev-sec-ops/secret-management/recipes/detect-secrets/), [IBM](https://github.com/IBM/detect-secrets), and [Yelp](https://github.com/Yelp/detect-secrets), is instrumental in identifying sensitive information within project files. Its prowess lies in its extensible Python plugin API, which allows custom rules and heuristics to tackle a broad spectrum of secrets. Designed to scan the project's current state rather than the entire git history, +[detect-secrets](https://github.com/Yelp/detect-secrets) is an industry leading tool to identify secure information included in source controlled files. Its prowess lies in its extensible Python plugin API, which allows custom rules and heuristics to tackle a broad spectrum of secrets. Designed to scan the project's current state rather than the entire git history, it operates swiftly, making it ideal for continuous integration pipelines. Leveraging the concept of a "baseline file" (`.secrets.baseline`), it enables easy handling of known secrets and false positives, facilitating its gradual integration into existing projects. ```mermaid From a70cac0ded980a6c10ca4e36f49a1a5c8b163bfb Mon Sep 17 00:00:00 2001 From: Jingchao Zhong <92573736+perryzjc@users.noreply.github.com> Date: Fri, 9 Jun 2023 17:53:37 -0700 Subject: [PATCH 12/16] Update continuous-testing/starter-kits/README.md Co-authored-by: Rishi Verma --- continuous-testing/starter-kits/README.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/continuous-testing/starter-kits/README.md b/continuous-testing/starter-kits/README.md index dbe960099..7e3b284d1 100644 --- a/continuous-testing/starter-kits/README.md +++ b/continuous-testing/starter-kits/README.md @@ -220,7 +220,12 @@ pre-commit install This command reads the `.pre-commit-config.yaml` file and installs the pre-commit hook in your local git repository. After this, you can see a `.git/hooks/pre-commit` file is created in your local git repository. -4. Make sure you have the baseline file `.secrets.baseline` in your local git repository +4. Add the baseline file `.secrets.baseline` in your local Git repository under the following conditions only: + 1. You wish to share white-listed secrets with other team members + 2. You've removed any references in your code to sensitive secrets detected that are not white-listed, i.e. find and modify code referenced in you `.secrets.baseline` with flag `is_secret: true` + 3. You've re-run a fresh `detect-secrets scan` followed by a `detect-secrets audit` and your `.secrets.baseline` ONLY now contains white-listed entries, i.e. `is_secret: false` + +**Warning**: Adding your `.secrets.baseline` file to your local Git and pushing changes to a remote without following the above directions could create a security risk by sharing the sensitive parts of your security scan with the outside world! 5. Commit your changes From e2d088e204ac0ffaf69ffed3ec6e5e1ae573a4a8 Mon Sep 17 00:00:00 2001 From: Jingchao Zhong <92573736+perryzjc@users.noreply.github.com> Date: Fri, 9 Jun 2023 17:56:47 -0700 Subject: [PATCH 13/16] Update continuous-testing/starter-kits/README.md Co-authored-by: Rishi Verma --- continuous-testing/starter-kits/README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/continuous-testing/starter-kits/README.md b/continuous-testing/starter-kits/README.md index 7e3b284d1..4bc1b9dc8 100644 --- a/continuous-testing/starter-kits/README.md +++ b/continuous-testing/starter-kits/README.md @@ -157,6 +157,8 @@ For example, It also provides other ways to present the result. For more information, please refer to [Auditing Secrets in Baseline](https://github.com/Yelp/detect-secrets#auditing-secrets-in-baseline) + +***Note*** if you have marked any detected secrets as true positives, its best to first remove all references to those secrets in your code and then rerun a full scan to generate a fresh audit report that you can compare against to ensure you only have false-positives (white-listed secrets) in your `.secrets.baseline` - this is especially important for Layer 2. #### Layer 2: Git Commit Scan (Client-side) The second layer is a pre-commit hook implemented in the local environment. This hook utilizes a `.pre-commit-config.yaml` file to config the pre-commit hook. The hook is triggered when the developer attempts to commit changes. The hook will scan the changes and **compare** them to the baseline file generated in the first layer. If any **new secrets** are detected, the hook will prevent the commit and report the detected secrets to the developer. ```mermaid From 90e24dbf769e98cc401db3c83f2f803303e5f041 Mon Sep 17 00:00:00 2001 From: Jingchao Zhong <92573736+perryzjc@users.noreply.github.com> Date: Fri, 9 Jun 2023 18:12:15 -0700 Subject: [PATCH 14/16] Update continuous-testing/starter-kits/README.md Co-authored-by: Rishi Verma --- continuous-testing/starter-kits/README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/continuous-testing/starter-kits/README.md b/continuous-testing/starter-kits/README.md index 4bc1b9dc8..bb3e77321 100644 --- a/continuous-testing/starter-kits/README.md +++ b/continuous-testing/starter-kits/README.md @@ -238,6 +238,8 @@ For example, Screen Shot 2023-04-20 at 7 32 10 AM > **Note**: The pre-commit hook blocks a commit by comparing new secrets with the results in the `.secrets.baseline` file. If new secrets are introduced, the hook will report them, but it does not automatically update the `.secrets.baseline` file. To update the baseline file with newly introduced secrets, you need to re-run the scan command in Layer 1 (step 2) and generate a new baseline file. + +> **Note**: during commit checks, detect secrets may not display all secrets present within a single file during a single scan. This can be to [minimize noise](https://github.com/Yelp/detect-secrets/blob/master/docs/design.md#potentialsecret), among other reasons. Thus if you have multiple violations of different types of secrets per file, multiple independent commits may be necessary to help identify all violations. > > You can create an empty result baseline file by running this command at a directory without secrets. From 6549625ee6de5179ad3859ebe52c809a1ed5e108 Mon Sep 17 00:00:00 2001 From: Jingchao Zhong <92573736+perryzjc@users.noreply.github.com> Date: Fri, 9 Jun 2023 18:13:11 -0700 Subject: [PATCH 15/16] Update continuous-testing/starter-kits/README.md Co-authored-by: Rishi Verma --- continuous-testing/starter-kits/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/continuous-testing/starter-kits/README.md b/continuous-testing/starter-kits/README.md index bb3e77321..775cb7bba 100644 --- a/continuous-testing/starter-kits/README.md +++ b/continuous-testing/starter-kits/README.md @@ -296,7 +296,7 @@ After this, if status check fails, GitHub will prevent the merge or push to the #### Attention for using Detect Secrets -> 1.It does not [show all the same type of secrets in a same file to minimize noise](https://github.com/Yelp/detect-secrets/blob/master/docs/design.md#:~:text=Furthermore%2C%20this%20will%20not%20flag%20on%20every%20single%20usage%20of%20a%20given%20secret%20in%20a%20given%20file%2C%20to%20minimize%20noise.) +> 1.It does not [show all the same type of secrets in a same file to minimize noise](https://github.com/Yelp/detect-secrets/blob/master/docs/design.md#potentialsecret) > > This means sometimes it will only show one secret in a file even if there are multiple **same type of secrets** in the same file. > Audit tool will not show all the secrets due to this reason. From d27620858dfaa0a34b6a04273259ca2e96007753 Mon Sep 17 00:00:00 2001 From: Jingchao Zhong <92573736+perryzjc@users.noreply.github.com> Date: Fri, 9 Jun 2023 18:26:43 -0700 Subject: [PATCH 16/16] Update README.md for Layer 3 --- continuous-testing/starter-kits/README.md | 82 ++++++++++++++++++++++- 1 file changed, 81 insertions(+), 1 deletion(-) diff --git a/continuous-testing/starter-kits/README.md b/continuous-testing/starter-kits/README.md index 775cb7bba..ebf5c8b43 100644 --- a/continuous-testing/starter-kits/README.md +++ b/continuous-testing/starter-kits/README.md @@ -272,7 +272,87 @@ sequenceDiagram ``` Starter Kit: -1. Create a GitHub Action workflow file in `.github/workflows/` directory from your repository root. Visit the [detect-secrets Action](https://github.com/marketplace/actions/detect-secrets-action) in the GitHub Actions Marketplace for details on how to add it to your repository. +1. Create a workflow file `detect-secrets.yaml` in `.github/workflows` directory from your repository root. +```yaml +name: Secret Detection Workflow +on: + push: + branches: + - main + pull_request: + branches: + - main + +jobs: + secret-detection: + runs-on: ubuntu-latest + steps: + - name: Checkout code + uses: actions/checkout@v2 + + - name: Install necessary packages + run: | + # This is the experimental version of slim-detect-secrets. + # It will be updated to the official Yelp/detect-secrets version once the customized plugins are merged. + # For more information about slim/detect-secrets, check the following: + # 1. https://github.com/NASA-AMMOS/slim-detect-secrets/tree/exp + # 2. https://github.com/NASA-AMMOS/slim/blob/d20ee6134a0dc0e0dab11d2d2570e358ef7e4550/continuous-testing/starter-kits/README.md#detect-secrets + pip install git+https://github.com/NASA-AMMOS/slim-detect-secrets.git@exp + # This library is used for JSON operations. + pip install jq + + - name: Create an initial .secrets.baseline if .secrets.baseline does not exist + run: | + if [ ! -f .secrets.baseline ]; then + # This generated baseline file will only be temporarily available on the GitHub side and will not appear in the user's local files. + # Scanning an empty folder to generate an initial .secrets.baseline without secrets in the results. + echo "⚠️ No existing .secrets.baseline file detected. Creating a new blank baseline file." + mkdir empty-dir + detect-secrets scan empty-dir > .secrets.baseline + echo "✅ Blank .secrets.baseline file created successfully." + rm -r empty-dir + else + echo "✅ Existing .secrets.baseline file detected. No new baseline file will be created." + fi + + - name: Scan repository for secrets + run: | + # scripts to scan repository for new secrets + + # backup the list of known secrets + cp .secrets.baseline .secrets.new + + # find the secrets in the repository + detect-secrets scan --baseline .secrets.new --exclude-files '.secrets.*' --exclude-files '.git*' + + # if there is any difference between the known and newly detected secrets, break the build + # Function to compare secrets without listing them + compare_secrets() { diff <(jq -r '.results | keys[] as $key | "\($key),\(.[$key] | .[] | .hashed_secret)"' "$1" | sort) <(jq -r '.results | keys[] as $key | "\($key),\(.[$key] | .[] | .hashed_secret)"' "$2" | sort) >/dev/null; } + + # Check if there's any difference between the known and newly detected secrets + if ! compare_secrets .secrets.baseline .secrets.new; then + echo "⚠️ Attention Required! ⚠️" >&2 + echo "New secrets have been detected in your recent commit. Due to security concerns, we cannot display detailed information here and we cannot proceed until this issue is resolved." >&2 + echo "" >&2 + echo "Please follow the steps below on your local machine to reveal and handle the secrets:" >&2 + echo "" >&2 + echo "1️⃣ Run the 'detect-secrets' tool on your local machine. This tool will identify and clean up the secrets. You can find detailed instructions at this link: https://nasa-ammos.github.io/slim/continuous-testing/starter-kits/#detect-secrets" >&2 + echo "" >&2 + echo "2️⃣ After cleaning up the secrets, commit your changes and re-push your update to the repository." >&2 + echo "" >&2 + echo "Your efforts to maintain the security of our codebase are greatly appreciated!" >&2 + exit 1 + fi + +``` +**Explanation** + +The Detect Secrets Action follows these steps to ensure the security of your code: + +- **Checkout Code**: Utilizes GitHub's checkout action to access the repository. This is the code that will be scanned for secrets. +- **Install Necessary Packages**: Deploys the required Python packages, including the experimental version of `slim-detect-secrets` and `jq`. These packages enable the primary functionality of the Action. +- **Check Existence of .secrets.baseline**: Ensures the Action remains operational even if no baseline file exists yet. If the `.secrets.baseline` file is not found, the action creates an initial baseline file by scanning an empty directory. +- **Scan Repository for Secrets**: In this step, the Action backs up the list of known secrets and scans the repository for any new secrets. The scan excludes files starting with '.secrets.' and '.git'. The 'compare_secrets' function is used to identify any differences between the known secrets and newly detected ones. If new secrets are detected, the build fails, and the user is guided to clean up the secrets using the `detect-secrets` tool. After this, GitHub will automatically run the workflow when you push to the branch or create a pull request.