diff --git a/.github/workflows/mkdocs.yml b/.github/workflows/mkdocs.yml index 9cd6bb95..8acdc13e 100644 --- a/.github/workflows/mkdocs.yml +++ b/.github/workflows/mkdocs.yml @@ -22,5 +22,5 @@ jobs: with: key: ${{ github.ref }} path: .cache - - run: pip install mkdocs-material + - run: pip install mkdocs-material markdown-include - run: mkdocs gh-deploy --force diff --git a/docs/index.md b/docs/index.md index 4c14a520..82fdca0e 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,4 +1,4 @@ -# trestle-bot +# Overview [![Pre commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white)](https://github.com/pre-commit/pre-commit) [![License](https://img.shields.io/badge/license-apache-blue.svg)](http://www.apache.org/licenses/LICENSE-2.0.html) @@ -61,15 +61,3 @@ Container images are available in `quay.io`: ```bash podman run -v $(pwd):/data -w /data quay.io/continuouscompliance/trestle-bot: ``` - -## Contributing - -For information about contributing to trestle-bot, see the [CONTRIBUTING.md](./CONTRIBUTING.md) file. - -## License - -This project is licensed under the Apache 2.0 License - see the [LICENSE.md](LICENSE) file for details. - -## Troubleshooting - -See [TROUBLESHOOTING.md](./TROUBLESHOOTING.md) for troubleshooting tips. diff --git a/docs/tutorials/github.md b/docs/tutorials/github.md new file mode 100644 index 00000000..5ab2db15 --- /dev/null +++ b/docs/tutorials/github.md @@ -0,0 +1,103 @@ +# GitHub Tutorial + +This tutorial provides an introduction to using `trestlebot` with GitHub. We will be using a single GitHub repository for our trestle authoring workspace and executing the `trestlebot` commands as GitHub actions. Note, each repo is intended to support authoring a single OSCAL model type (SSP, component definition, etc.). If authoring more than one, then a dedeicated repository should be used for each model. + + +### 1. Prerequisites + +Before moving on, please ensure you have completed the following: + +1. Create a new (or use an existing) empty GitHub repository +2. Clone the repo to your local workstation +3. Install trestlebot + * Option 1: Clone the [trestle-bot](https://github.com/RedHatProductSecurity/trestle-bot/tree/main) repo to your local workstation and run `poetry install` + * Option 2: Use the [trestlebot container image](https://github.com/RedHatProductSecurity/trestle-bot?tab=readme-ov-file#run-as-a-container) + + +### 2. Set Permissions for GitHub Actions + +The `trestlebot` commands will be run inside of GitHub actions. These commands often perform `write` level operations against the repo contents. The GitHub workflows generated in this tutorial make use of [automatic token authentication.](https://docs.github.com/en/actions/security-for-github-actions/security-guides/automatic-token-authentication) To ensure this is configured correct the following repo settings need to be in place. + +*Note: If you choose an alternative method to provide repo access such as personal access tokens or GitHub apps you can skip these steps.* + +1. Click the `Settings` tab for your GitHub repo +2. Select `Actions` -> `General` from the left-hand menu +3. Scroll down to `Workflow permissions` +4. Ensure `Read repository contents and packages permissions` is selected +5. Ensure `Allow GitHub Actions to create and approve pull requests` is checked + +### 3. Initialize trestlebot Workspace + +We will now use the `trestlebot init` command to initialize our emtpy GitHub repository. Unlike the other trestlebot commands, this command is run on your local workstation. The trestlebot commands can be installed by cloning the [trestle-bot](https://github.com/RedHatProductSecurity/trestle-bot/tree/main) repo and running `poetry install`. Alternatively these commands can be run using the [trestlebot container image](https://github.com/RedHatProductSecurity/trestle-bot?tab=readme-ov-file#run-as-a-container). For this tutorial we will be authoring a component-definition. + +``` +trestlebot-init --oscal-model compdef --working-dir +``` + +Using container image: + +``` +podman run -v :/data:rw trestle-bot:latest --oscal-model compdef --working-dir /data +``` + +You should now see the following directories in your repo. + + +```bash +. +├── catalogs +├── component-definitions +├── markdown +├── profiles +├── rules +├── .github +├── .trestle +└── .trestlebot +``` + +You will notice several files within the `.github/workflows` directory. These are the trestlebot actions that will run as we make changes to the repo contents. + +You can now add any catalog or profile content needed for you authoring process. For this example, we will add the NIST SP 800-53 Rev. 5 catalog to our `/catalogs` directory. + +``` +mkdir catalogs/nist_rev5_800_53 +wget https://raw.githubusercontent.com/usnistgov/oscal-content/release-v1.0.5-update/nist.gov/SP800-53/rev5/json/NIST_SP-800-53_rev5_catalog.json -O catalogs/nist_rev5_800_53/catalog.json +``` + +Now we will add the NIST SP 800-53 Rev. 5 High Baseline profile to our `profiles/` directory. + +``` +mkdir profiles/nist_rev5_800_53 +wget https://raw.githubusercontent.com/usnistgov/oscal-content/release-v1.0.5-update/nist.gov/SP800-53/rev5/json/NIST_SP-800-53_rev5_HIGH-baseline_profile.json -O profiles/nist_rev5_800_53/profile.json +``` + +Our `profile.json` file contains a reference to our `catalog.json` file. By default, this path is not resolvable by compliance-trestle, so we need to run the following command to update the `href` value in the JSON. + +``` +sed -i 's/NIST_SP-800-53_rev5_catalog.json/trestle:\/\/catalogs\/nist_rev5_800_53\/catalog.json/g' profiles/nist_rev5_800_53/profile.json +``` + +Now that we have the initial content needed to begin authoring, go ahead and commit and push to the remote GitHub repo. + + +### 4. Create a New Component Definition + +Now it's time to run our first trestlebot action! We will go ahead and create our first component definition. + +1. Open to your GitHub repo in a web browser. +2. Click to the `Actions` tab from the top menu. +3. Click the `Trestle-bot create component definition` action from the left-hand menu. +4. Click `Run Workflow` which will open up a dialog box. +5. Enter the following values: + +* _Name of the Trestle profile to use for the component definition:_ `nist_rev5_800_53` +* _Name of the component definition to create:_ `my-first-compdef` +* _Name of the component to create in the generated component definition:_ `test-component` +* _Type of the component (e.g. service, policy, physical, validation, etc.):_ `service` +* _Description of the component to create:_ `Testing trestlebot init` + +6. Click `Run Workflow` + +Once the workflow has completed you should have a new Pull Request containing the files trestlebot generated for the component definition. After reviewing the files you can go ahead and merge the PR! + +Congrats, you have sucessfully created a new trestlebot workspace and now have an authoring environment! \ No newline at end of file diff --git a/mkdocs.yml b/mkdocs.yml index f0ee2ad1..746ea3cb 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -1,4 +1,7 @@ site_name: trestle-bot +site_description: Documentation for trestle-bot. +repo_name: trestle-bot +repo_url: https://github.com/RedHatProductSecurity/trestle-bot/ theme: name: material @@ -24,6 +27,8 @@ theme: name: Switch to light mode markdown_extensions: + - markdown_include.include + - md_in_html - toc: toc_depth: 2 - pymdownx.superfences: @@ -34,3 +39,15 @@ markdown_extensions: copyright: | © Copyright 2023 Red Hat, Inc. + +nav: +- Overview: + - QuickStart: index.md + - Architecture: + - Diagrams: architecture/diagrams/c4.md +- Tutorials: + - GitHub: tutorials/github.md +- Troubleshooting: troubleshooting.md +- Contributing: contributing.md + + diff --git a/poetry.lock b/poetry.lock index c11c0445..c841a8ff 100644 --- a/poetry.lock +++ b/poetry.lock @@ -1120,6 +1120,23 @@ importlib-metadata = {version = ">=4.4", markers = "python_version < \"3.10\""} docs = ["mdx-gh-links (>=0.2)", "mkdocs (>=1.5)", "mkdocs-gen-files", "mkdocs-literate-nav", "mkdocs-nature (>=0.6)", "mkdocs-section-index", "mkdocstrings[python]"] testing = ["coverage", "pyyaml"] +[[package]] +name = "markdown-include" +version = "0.8.1" +description = "A Python-Markdown extension which provides an 'include' function" +optional = false +python-versions = ">=3.7" +files = [ + {file = "markdown-include-0.8.1.tar.gz", hash = "sha256:1d0623e0fc2757c38d35df53752768356162284259d259c486b4ab6285cdbbe3"}, + {file = "markdown_include-0.8.1-py3-none-any.whl", hash = "sha256:32f0635b9cfef46997b307e2430022852529f7a5b87c0075c504283e7cc7db53"}, +] + +[package.dependencies] +markdown = ">=3.0" + +[package.extras] +tests = ["pytest"] + [[package]] name = "markdown-it-py" version = "3.0.0" @@ -2585,4 +2602,4 @@ test = ["big-O", "importlib-resources", "jaraco.functools", "jaraco.itertools", [metadata] lock-version = "2.0" python-versions = "^3.8.1" -content-hash = "4a8eea1af39dae299c2000af88cfd50760a2a762498816238c5a0b8287fb7e78" +content-hash = "97ef7c1ede0e64754d45dc505b33d53cefeabb4083e64ba67e5cba57cbe10f90" diff --git a/pyproject.toml b/pyproject.toml index ee5ab3f4..8cc3984b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -45,6 +45,7 @@ safety = "^3.1.0" flake8-print = "^5.0.0" pre-commit = "^3.4.0" mkdocs-material = "^9.5.31" +markdown-include = "^0.8.1" [tool.poetry.group.tests.dependencies] pytest = "^8.3.2" diff --git a/tests/trestlebot/entrypoints/test_init.py b/tests/trestlebot/entrypoints/test_init.py index e23406dc..020b6bac 100644 --- a/tests/trestlebot/entrypoints/test_init.py +++ b/tests/trestlebot/entrypoints/test_init.py @@ -13,7 +13,7 @@ from trestle.common.file_utils import is_hidden from tests.testutils import args_dict_to_list, configure_test_logger, setup_for_init -from trestlebot.const import GITHUB, TRESTLEBOT_CONFIG_DIR, TRESTLEBOT_KEEP_FILE +from trestlebot.const import GITHUB, GITLAB, TRESTLEBOT_CONFIG_DIR, TRESTLEBOT_KEEP_FILE from trestlebot.entrypoints.init import InitEntrypoint from trestlebot.entrypoints.init import main as cli_main from trestlebot.tasks.authored import types as model_types @@ -105,7 +105,9 @@ def test_init_ssp_github( # directories for ssp model should exist model_dirs = [d.name for d in tmp_dir.iterdir() if not is_hidden(d)] - expected = InitEntrypoint.MODEL_DIRS[args_dict["oscal-model"]] + expected = InitEntrypoint.MODEL_DIRS[args_dict["oscal-model"]] + [ + InitEntrypoint.MARKDOWN_DIR + ] assert sorted(model_dirs) == sorted(expected) # directories for github workflows should exist @@ -116,6 +118,12 @@ def test_init_ssp_github( ] assert sorted(workflow_files) == sorted(expected) + # markdown directories should exist + markdown_dir = tmp_dir.joinpath(InitEntrypoint.MARKDOWN_DIR) + expected_subdirs = InitEntrypoint.MODEL_DIRS[args_dict["oscal-model"]] + markdown_subdirs = [f.name for f in markdown_dir.iterdir()] + assert sorted(markdown_subdirs) == sorted(expected_subdirs) + assert any( record.levelno == logging.INFO and f"Initialized trestlebot project successfully in {tmp_init_dir}" @@ -124,6 +132,30 @@ def test_init_ssp_github( ) +@patch( + "trestlebot.entrypoints.log.configure_logger", + Mock(side_effect=configure_test_logger), +) +def test_init_ssp_gitlab( + tmp_init_dir: str, args_dict: Dict[str, str], caplog: pytest.LogCaptureFixture +) -> None: + """Tests for expected logging message""" + args_dict["working-dir"] = tmp_init_dir + args_dict["oscal-model"] = OSCAL_MODEL_SSP + args_dict["provider"] = GITLAB + setup_for_init(pathlib.Path(tmp_init_dir)) + with patch("sys.argv", ["trestlebot", *args_dict_to_list(args_dict)]): + with pytest.raises(SystemExit, match="0"): + cli_main() + + assert any( + record.levelno == logging.WARNING + and "GitLab is not yet supported, no provider files will be created." + in record.message + for record in caplog.records + ) + + @patch( "trestlebot.entrypoints.log.configure_logger", Mock(side_effect=configure_test_logger), @@ -144,7 +176,9 @@ def test_init_compdef_github( # directories for compdef model should exist tmp_dir = pathlib.Path(tmp_init_dir) model_dirs = [d.name for d in tmp_dir.iterdir() if not is_hidden(d)] - expected = InitEntrypoint.MODEL_DIRS[args_dict["oscal-model"]] + expected = InitEntrypoint.MODEL_DIRS[args_dict["oscal-model"]] + [ + InitEntrypoint.MARKDOWN_DIR + ] assert sorted(model_dirs) == sorted(expected) # directories for github workflows should exist @@ -155,6 +189,12 @@ def test_init_compdef_github( ] assert sorted(workflow_files) == sorted(expected) + # markdown directories should exist + markdown_dir = tmp_dir.joinpath(InitEntrypoint.MARKDOWN_DIR) + expected_subdirs = InitEntrypoint.MODEL_DIRS[args_dict["oscal-model"]] + markdown_subdirs = [f.name for f in markdown_dir.iterdir()] + assert sorted(markdown_subdirs) == sorted(expected_subdirs) + assert any( record.levelno == logging.INFO and f"Initialized trestlebot project successfully in {tmp_init_dir}" diff --git a/trestlebot/__main__.py b/trestlebot/__main__.py index 5a11fc15..84a133f6 100644 --- a/trestlebot/__main__.py +++ b/trestlebot/__main__.py @@ -4,13 +4,13 @@ # Default entrypoint for trestlebot is autosync mode when run with python -m trestlebot -from trestlebot.entrypoints.autosync import main as autosync_main +from trestlebot.entrypoints.init import main as init_main def init() -> None: """Initialize trestlebot""" if __name__ == "__main__": - autosync_main() + init_main() init() diff --git a/trestlebot/entrypoints/init.py b/trestlebot/entrypoints/init.py index de6b3107..5907ad4b 100644 --- a/trestlebot/entrypoints/init.py +++ b/trestlebot/entrypoints/init.py @@ -49,6 +49,7 @@ class InitEntrypoint: """Entrypoint for the init command.""" + MARKDOWN_DIR = "markdown" TEMPLATES_MODULE = "trestlebot.entrypoints.templates" SUPPORTED_PROVIDERS = [GITHUB, GITLAB] SUPPORTED_MODELS = [ @@ -150,8 +151,22 @@ def _create_directories(self, args: argparse.Namespace) -> None: keep_file = directory.joinpath(pathlib.Path(TRESTLEBOT_KEEP_FILE)) file_utils.make_hidden_file(keep_file) + # also create markdown dirs + markdown_root = root.joinpath(pathlib.Path(self.MARKDOWN_DIR)) + for model_dir in model_dirs: + directory = markdown_root.joinpath(pathlib.Path(model_dir)) + directory.mkdir(exist_ok=True, parents=True) + keep_file = directory.joinpath(pathlib.Path(TRESTLEBOT_KEEP_FILE)) + file_utils.make_hidden_file(keep_file) + def _copy_provider_files(self, args: argparse.Namespace) -> None: """Copy the CICD provider files to the new trestlebot workspace""" + if args.provider == GITLAB: + logger.warning( + "GitLab is not yet supported, no provider files will be created." + ) + return + if args.provider == GITHUB: provider_dir = pathlib.Path(args.working_dir).joinpath( pathlib.Path(GITHUB_WORKFLOWS_DIR) diff --git a/trestlebot/entrypoints/templates/github/trestlebot-autosync-catalog.yml b/trestlebot/entrypoints/templates/github/trestlebot-autosync-catalog.yml index 595896b3..dfee2050 100644 --- a/trestlebot/entrypoints/templates/github/trestlebot-autosync-catalog.yml +++ b/trestlebot/entrypoints/templates/github/trestlebot-autosync-catalog.yml @@ -18,8 +18,6 @@ jobs: steps: - name: Checkout repository uses: actions/checkout@v4 - with: - ref: ${{ github.head_ref }} - name: Run autosync id: autosync uses: RedHatProductSecurity/trestle-bot/actions/autosync@main @@ -27,4 +25,3 @@ jobs: markdown_path: "markdown/catalogs" oscal_model: "catalog" file_pattern: "*.json,markdown/*" - branch: ${{ github.head_ref }} diff --git a/trestlebot/entrypoints/templates/github/trestlebot-autosync-profile.yml b/trestlebot/entrypoints/templates/github/trestlebot-autosync-profile.yml index 9854b011..04188eb2 100644 --- a/trestlebot/entrypoints/templates/github/trestlebot-autosync-profile.yml +++ b/trestlebot/entrypoints/templates/github/trestlebot-autosync-profile.yml @@ -18,8 +18,6 @@ jobs: steps: - name: Checkout repository uses: actions/checkout@v4 - with: - ref: ${{ github.head_ref }} - name: Run autosync id: autosync uses: RedHatProductSecurity/trestle-bot/actions/autosync@main @@ -27,4 +25,3 @@ jobs: markdown_path: "markdown/profiles" oscal_model: "profile" file_pattern: "*.json,markdown/*" - branch: ${{ github.head_ref }}