Merge branch 'main' into bug/e2e-schedule-failure

.github/workflows/mkdocs.yml

@@ -22,5 +22,5 @@ jobs:
         with:
           key: ${{ github.ref }}
           path: .cache
-      - run: pip install mkdocs-material
+      - run: pip install mkdocs-material markdown-include
       - run: mkdocs gh-deploy --force

docs/index.md

@@ -1,4 +1,4 @@
-# trestle-bot
+# Overview
 
 [![Pre commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white)](https://github.com/pre-commit/pre-commit)
 [![License](https://img.shields.io/badge/license-apache-blue.svg)](http://www.apache.org/licenses/LICENSE-2.0.html)
@@ -61,15 +61,3 @@ Container images are available in `quay.io`:
 ```bash
 podman run -v $(pwd):/data -w /data quay.io/continuouscompliance/trestle-bot:<tag>
 ```
-
-## Contributing
-
-For information about contributing to trestle-bot, see the [CONTRIBUTING.md](./CONTRIBUTING.md) file.
-
-## License
-
-This project is licensed under the Apache 2.0 License - see the [LICENSE.md](LICENSE) file for details.
-
-## Troubleshooting
-
-See [TROUBLESHOOTING.md](./TROUBLESHOOTING.md) for troubleshooting tips.

docs/tutorials/github.md

@@ -0,0 +1,103 @@
+# GitHub Tutorial
+
+This tutorial provides an introduction to using `trestlebot` with GitHub.  We will be using a single GitHub repository for our trestle authoring workspace and executing the `trestlebot` commands as GitHub actions.  Note, each repo is intended to support authoring a single OSCAL model type (SSP, component definition, etc.).  If authoring more than one, then a dedeicated repository should be used for each model.
+
+
+### 1. Prerequisites
+
+Before moving on, please ensure you have completed the following:
+
+1. Create a new (or use an existing) empty GitHub repository
+2. Clone the repo to your local workstation
+3. Install trestlebot
+    * Option 1: Clone the [trestle-bot](https://github.com/RedHatProductSecurity/trestle-bot/tree/main) repo to your local workstation and run `poetry install`
+    * Option 2: Use the [trestlebot container image](https://github.com/RedHatProductSecurity/trestle-bot?tab=readme-ov-file#run-as-a-container)
+
+
+### 2. Set Permissions for GitHub Actions
+
+The `trestlebot` commands will be run inside of GitHub actions.  These commands often perform `write` level operations against the repo contents.  The GitHub workflows generated in this tutorial make use of [automatic token authentication.](https://docs.github.com/en/actions/security-for-github-actions/security-guides/automatic-token-authentication)  To ensure this is configured correct the following repo settings need to be in place.
+
+*Note: If you choose an alternative method to provide repo access such as personal access tokens or GitHub apps you can skip these steps.*
+
+1. Click the `Settings` tab for your GitHub repo 
+2. Select `Actions` -> `General` from the left-hand menu
+3. Scroll down to `Workflow permissions`
+4. Ensure `Read repository contents and packages permissions` is selected
+5. Ensure `Allow GitHub Actions to create and approve pull requests` is checked
+
+### 3. Initialize trestlebot Workspace
+
+We will now use the `trestlebot init` command to initialize our emtpy GitHub repository.  Unlike the other trestlebot commands, this command is run on your local workstation.  The trestlebot commands can be installed by cloning the [trestle-bot](https://github.com/RedHatProductSecurity/trestle-bot/tree/main) repo and running `poetry install`.  Alternatively these commands can be run using the [trestlebot container image](https://github.com/RedHatProductSecurity/trestle-bot?tab=readme-ov-file#run-as-a-container). For this tutorial we will be authoring a component-definition.
+
+```
+trestlebot-init --oscal-model compdef --working-dir <path-to-your-repo>
+```
+
+Using container image:
+
+```
+podman run -v <path-to-your-repo>:/data:rw  trestle-bot:latest --oscal-model compdef --working-dir /data
+```
+
+You should now see the following directories in your repo.
+
+
+```bash
+.
+├── catalogs
+├── component-definitions
+├── markdown 
+├── profiles
+├── rules
+├── .github
+├── .trestle
+└── .trestlebot
+```
+
+You will notice several files within the `.github/workflows` directory.  These are the trestlebot actions that will run as we make changes to the repo contents.
+
+You can now add any catalog or profile content needed for you authoring process.  For this example, we will add the NIST SP 800-53 Rev. 5 catalog to our `/catalogs` directory.
+
+```
+mkdir catalogs/nist_rev5_800_53
+wget https://raw.githubusercontent.com/usnistgov/oscal-content/release-v1.0.5-update/nist.gov/SP800-53/rev5/json/NIST_SP-800-53_rev5_catalog.json -O catalogs/nist_rev5_800_53/catalog.json
+```
+
+Now we will add the NIST SP 800-53 Rev. 5 High Baseline profile to our `profiles/` directory.
+
+```
+mkdir profiles/nist_rev5_800_53
+wget https://raw.githubusercontent.com/usnistgov/oscal-content/release-v1.0.5-update/nist.gov/SP800-53/rev5/json/NIST_SP-800-53_rev5_HIGH-baseline_profile.json -O profiles/nist_rev5_800_53/profile.json
+```
+
+Our `profile.json` file contains a reference to our `catalog.json` file.  By default, this path is not resolvable by compliance-trestle, so we need to run the following command to update the `href` value in the JSON.
+
+```
+sed -i 's/NIST_SP-800-53_rev5_catalog.json/trestle:\/\/catalogs\/nist_rev5_800_53\/catalog.json/g' profiles/nist_rev5_800_53/profile.json
+```
+
+Now that we have the initial content needed to begin authoring, go ahead and commit and push to the remote GitHub repo.
+
+
+### 4. Create a New Component Definition
+
+Now it's time to run our first trestlebot action!  We will go ahead and create our first component definition.
+
+1. Open to your GitHub repo in a web browser.
+2. Click to the `Actions` tab from the top menu.
+3. Click the `Trestle-bot create component definition` action from the left-hand menu.
+4. Click `Run Workflow` which will open up a dialog box.
+5. Enter the following values:
+
+* _Name of the Trestle profile to use for the component definition:_ `nist_rev5_800_53`
+* _Name of the component definition to create:_ `my-first-compdef`
+* _Name of the component to create in the generated component definition:_ `test-component`
+* _Type of the component (e.g. service, policy, physical, validation, etc.):_ `service`
+* _Description of the component to create:_ `Testing trestlebot init`
+
+6. Click `Run Workflow`
+
+Once the workflow has completed you should have a new Pull Request containing the files trestlebot generated for the component definition.  After reviewing the files you can go ahead and merge the PR!
+
+Congrats, you have sucessfully created a new trestlebot workspace and now have an authoring environment!

mkdocs.yml

@@ -1,4 +1,7 @@
 site_name: trestle-bot
+site_description: Documentation for trestle-bot.
+repo_name: trestle-bot
+repo_url: https://github.com/RedHatProductSecurity/trestle-bot/
 
 theme:
   name: material
@@ -24,6 +27,8 @@ theme:
         name: Switch to light mode    
 
 markdown_extensions:
+  - markdown_include.include
+  - md_in_html
   - toc:
       toc_depth: 2
   - pymdownx.superfences:
@@ -34,3 +39,15 @@ markdown_extensions:
 
 copyright: |
   &copy; Copyright 2023 Red Hat, Inc.</a>
+
+nav:
+- Overview: 
+  - QuickStart: index.md
+  - Architecture:
+    - Diagrams: architecture/diagrams/c4.md
+- Tutorials:
+  - GitHub: tutorials/github.md
+- Troubleshooting: troubleshooting.md
+- Contributing: contributing.md
+
+

pyproject.toml

@@ -45,6 +45,7 @@ safety = "^3.1.0"
 flake8-print = "^5.0.0"
 pre-commit = "^3.4.0"
 mkdocs-material = "^9.5.31"
+markdown-include = "^0.8.1"
 
 [tool.poetry.group.tests.dependencies]
 pytest = "^8.3.2"

tests/trestlebot/entrypoints/test_init.py

@@ -13,7 +13,7 @@
 from trestle.common.file_utils import is_hidden
 
 from tests.testutils import args_dict_to_list, configure_test_logger, setup_for_init
-from trestlebot.const import GITHUB, TRESTLEBOT_CONFIG_DIR, TRESTLEBOT_KEEP_FILE
+from trestlebot.const import GITHUB, GITLAB, TRESTLEBOT_CONFIG_DIR, TRESTLEBOT_KEEP_FILE
 from trestlebot.entrypoints.init import InitEntrypoint
 from trestlebot.entrypoints.init import main as cli_main
 from trestlebot.tasks.authored import types as model_types
@@ -105,7 +105,9 @@ def test_init_ssp_github(
 
     # directories for ssp model should exist
     model_dirs = [d.name for d in tmp_dir.iterdir() if not is_hidden(d)]
-    expected = InitEntrypoint.MODEL_DIRS[args_dict["oscal-model"]]
+    expected = InitEntrypoint.MODEL_DIRS[args_dict["oscal-model"]] + [
+        InitEntrypoint.MARKDOWN_DIR
+    ]
     assert sorted(model_dirs) == sorted(expected)
 
     # directories for github workflows should exist
@@ -116,6 +118,12 @@ def test_init_ssp_github(
     ]
     assert sorted(workflow_files) == sorted(expected)
 
+    # markdown directories should exist
+    markdown_dir = tmp_dir.joinpath(InitEntrypoint.MARKDOWN_DIR)
+    expected_subdirs = InitEntrypoint.MODEL_DIRS[args_dict["oscal-model"]]
+    markdown_subdirs = [f.name for f in markdown_dir.iterdir()]
+    assert sorted(markdown_subdirs) == sorted(expected_subdirs)
+
     assert any(
         record.levelno == logging.INFO
         and f"Initialized trestlebot project successfully in {tmp_init_dir}"
@@ -124,6 +132,30 @@ def test_init_ssp_github(
     )
 
 
+@patch(
+    "trestlebot.entrypoints.log.configure_logger",
+    Mock(side_effect=configure_test_logger),
+)
+def test_init_ssp_gitlab(
+    tmp_init_dir: str, args_dict: Dict[str, str], caplog: pytest.LogCaptureFixture
+) -> None:
+    """Tests for expected logging message"""
+    args_dict["working-dir"] = tmp_init_dir
+    args_dict["oscal-model"] = OSCAL_MODEL_SSP
+    args_dict["provider"] = GITLAB
+    setup_for_init(pathlib.Path(tmp_init_dir))
+    with patch("sys.argv", ["trestlebot", *args_dict_to_list(args_dict)]):
+        with pytest.raises(SystemExit, match="0"):
+            cli_main()
+
+    assert any(
+        record.levelno == logging.WARNING
+        and "GitLab is not yet supported, no provider files will be created."
+        in record.message
+        for record in caplog.records
+    )
+
+
 @patch(
     "trestlebot.entrypoints.log.configure_logger",
     Mock(side_effect=configure_test_logger),
@@ -144,7 +176,9 @@ def test_init_compdef_github(
     # directories for compdef model should exist
     tmp_dir = pathlib.Path(tmp_init_dir)
     model_dirs = [d.name for d in tmp_dir.iterdir() if not is_hidden(d)]
-    expected = InitEntrypoint.MODEL_DIRS[args_dict["oscal-model"]]
+    expected = InitEntrypoint.MODEL_DIRS[args_dict["oscal-model"]] + [
+        InitEntrypoint.MARKDOWN_DIR
+    ]
     assert sorted(model_dirs) == sorted(expected)
 
     # directories for github workflows should exist
@@ -155,6 +189,12 @@ def test_init_compdef_github(
     ]
     assert sorted(workflow_files) == sorted(expected)
 
+    # markdown directories should exist
+    markdown_dir = tmp_dir.joinpath(InitEntrypoint.MARKDOWN_DIR)
+    expected_subdirs = InitEntrypoint.MODEL_DIRS[args_dict["oscal-model"]]
+    markdown_subdirs = [f.name for f in markdown_dir.iterdir()]
+    assert sorted(markdown_subdirs) == sorted(expected_subdirs)
+
     assert any(
         record.levelno == logging.INFO
         and f"Initialized trestlebot project successfully in {tmp_init_dir}"

trestlebot/__main__.py

@@ -4,13 +4,13 @@
 
 # Default entrypoint for trestlebot is autosync mode when run with python -m trestlebot
 
-from trestlebot.entrypoints.autosync import main as autosync_main
+from trestlebot.entrypoints.init import main as init_main
 
 
 def init() -> None:
     """Initialize trestlebot"""
     if __name__ == "__main__":
-        autosync_main()
+        init_main()
 
 
 init()

trestlebot/entrypoints/init.py

@@ -49,6 +49,7 @@
 class InitEntrypoint:
     """Entrypoint for the init command."""
 
+    MARKDOWN_DIR = "markdown"
     TEMPLATES_MODULE = "trestlebot.entrypoints.templates"
     SUPPORTED_PROVIDERS = [GITHUB, GITLAB]
     SUPPORTED_MODELS = [
@@ -150,8 +151,22 @@ def _create_directories(self, args: argparse.Namespace) -> None:
             keep_file = directory.joinpath(pathlib.Path(TRESTLEBOT_KEEP_FILE))
             file_utils.make_hidden_file(keep_file)
 
+        # also create markdown dirs
+        markdown_root = root.joinpath(pathlib.Path(self.MARKDOWN_DIR))
+        for model_dir in model_dirs:
+            directory = markdown_root.joinpath(pathlib.Path(model_dir))
+            directory.mkdir(exist_ok=True, parents=True)
+            keep_file = directory.joinpath(pathlib.Path(TRESTLEBOT_KEEP_FILE))
+            file_utils.make_hidden_file(keep_file)
+
     def _copy_provider_files(self, args: argparse.Namespace) -> None:
         """Copy the CICD provider files to the new trestlebot workspace"""
+        if args.provider == GITLAB:
+            logger.warning(
+                "GitLab is not yet supported, no provider files will be created."
+            )
+            return
+
         if args.provider == GITHUB:
             provider_dir = pathlib.Path(args.working_dir).joinpath(
                 pathlib.Path(GITHUB_WORKFLOWS_DIR)

trestlebot/entrypoints/templates/github/trestlebot-autosync-catalog.yml

@@ -18,13 +18,10 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@v4
-        with:
-          ref: ${{ github.head_ref }}
       - name: Run autosync
         id: autosync
         uses: RedHatProductSecurity/trestle-bot/actions/autosync@main
         with:
           markdown_path: "markdown/catalogs"
           oscal_model: "catalog"
           file_pattern: "*.json,markdown/*"
-          branch: ${{ github.head_ref }}

trestlebot/entrypoints/templates/github/trestlebot-autosync-profile.yml

@@ -18,13 +18,10 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@v4
-        with:
-          ref: ${{ github.head_ref }}
       - name: Run autosync
         id: autosync
         uses: RedHatProductSecurity/trestle-bot/actions/autosync@main
         with:
           markdown_path: "markdown/profiles"
           oscal_model: "profile"
           file_pattern: "*.json,markdown/*"
-          branch: ${{ github.head_ref }}