Merge pull request #136 from olehermanse/fix_policy_files

CFE-4088: Made policy_files build step not implicitly add services/cfbs

JSON.md

@@ -16,6 +16,95 @@ All these 3 share 1 standard format, commonly called `cfbs.json`.
 Used by `cfbs add` and `cfbs search`, when index key is present in `cfbs.json` in the current working directory.
 When adding a module by URL, which has a `cfbs.json` inside of it, the index in that file should be ignored.
 
+## The process of building modules from a project into a policy set
+
+When you build a project with the `cfbs build` command, it loops through all modules in the project (`"build"` key), in order.
+Within each module it runs the individual build steps, specified in the `"steps"` key.
+
+As an example, here is a local policy file module with 3 common build steps:
+
+```json
+{
+  "name": "./policy.cf",
+  "description": "Local policy file added using cfbs command line",
+  "tags": ["local"],
+  "added_by": "cfbs add",
+  "steps": [
+    "copy ./policy.cf services/cfbs/policy.cf",
+    "policy_files ./services/cfbs/policy.cf",
+    "bundles my_bundle"
+  ]
+}
+```
+
+The 3 build steps above achieve 3 distinct things:
+1. The policy file `policy.cf` is included in the policy set. It will be deployed to `/var/cfengine/masterfiles/services/cfbs/policy.cf` on the hub.
+2. The path to `policy.cf` is added to `"inputs"` making cf-agent and other binaries aware of it and parse it.
+3. The bundle `my_bundle` within `policy.cf` is added to the bundle sequence, making `cf-agent` find the correct bundle to run (commonly called the entry point of a policy). In the end, this causes `cf-agent` to actually evaluate the promises within the bundle and enforce your desired state (potentially making changes to the system).
+
+
+### Step folders
+
+As a project is built, `cfbs` creates intermediate folders for each module, for example:
+
+```
+out/steps/001_masterfiles_5c7dc5b43088e259a94de4e5a9f17c0ce9781a0f/
+```
+
+These are copies of the module directories, where it's more "safe" to do things like run scripts or delete files.
+`cfbs build` should not edit files in your project / git repository, only the generated / temporary files inside the `out/` directory.
+
+### All available build steps
+
+The build steps below manipulate the temporary files in the steps directories and write results to the output policy set, in `out/masterfiles`.
+Unless otherwise noted, all steps are run inside the module's folder (`out/steps/...`) with sources / file paths relative to that folder, and targets / destinations mentioned below are relative to the output policy set (`out/masterfiles`, which in the end will be deployed as `/var/cfengine/masterfiles`)
+
+#### `copy <source> <destination>`
+- Copy a single file or a directory recursively.
+
+#### `json <source> <destination>`
+- Merge the source json file into the destination json file.
+
+#### `append <source> <destination>`
+- Append the source file to the end of destination file.
+
+#### `run <command>`
+- Run a shell command / script.
+- Usually used to prepare the module directory, delete files, etc. before a copy step.
+- Running scripts should be avoided if possible.
+- Script is run inside the module directory (the step folder).
+
+#### `delete <paths ...>`
+- Delete multiple files or paths recursively.
+- Files are deleted from the step folder.
+- Typically used before copying files to the output policy set with the `copy` step. 
+
+#### `directory <source> <destination>`
+- Copy any .cf policy files recursively and add their paths to `def.json`'s `inputs`.
+- Enable `services_autorun_bundles` class in `def.json`.
+- Merge any `def.json` recursively into `out/masterfiles/def.json`.
+- Copy any other files with their existing directory structure to destination.
+
+#### `bundles <bundles ...>`
+- Ensure bundles are evaluated by adding them to the bundle sequence, using `def.json`.
+  - Note that this relies on using the default policy set from the CFEngine team, the Masterfiles Policy Framework, commonly added as the first module (`masterfiles`). 
+    Specifically, this build step adds the bundles to the variable `default:def.control_common_bundlesequence_end`, which the MPF looks for.
+- Only manipulates the bundle sequence, to ensure policy files are copied and parsed, use other build steps, for example `copy` and `policy_files`.
+
+#### `policy_files <paths ...>`
+- Add policy (`.cf`) files to `inputs` key in `def.json`, ensuring they are parsed.
+  - Note that this relies on using the default policy set from the CFEngine team, the Masterfiles Policy Framework, commonly added as the first module (`masterfiles`).
+  - Only edits `def.json`, does not copy files. Should be used after a `copy` or `directory` build step.
+  - Does not add any bundles to the bundle sequence, to ensure a bundle is evaluated, use the `bundles` build step or the autorun mechanism.
+- All paths are relative to `out/masterfiles`.
+- If any of the paths are directories (end with `/`), the folder(s) are recursively searched and all `.cf` files are added.
+  - **Note:** Directories should be module-specific, otherwise this build step can find policy files from other modules (when they are mixed in the same directory).
+
+#### `input <source input.json> <target def.json>`
+- Converts the input data for a module into the augments format and merges it with the target augments file.
+- Source is relative to module directory and target is relative to `out/masterfiles`.
+  - In most cases, the build step should be: `input ./input.json def.json`
+
 ## Examples
 
 ### New project

README.md

@@ -102,9 +102,51 @@ cfbs clean
 ### Add module input
 
 ```
-cfbs input create-single-file
+cfbs input delete-files
 ```
 
+This command prompts the user for module input to configure what the module should do.
+
+Only works for modules which accept input, indicated by the `"input"` key.
+This `input` key contains the specification for what input the module accepts, if any.
+
+User's responses are stored in `<module-name>/input.json`.
+For completeness, the input specification is also stored with the responses.
+Here is an example of a `input.json` file with responses:
+
+```json
+[
+  {
+    "type": "list",
+    "variable": "files",
+    "bundle": "delete_files",
+    "label": "Files",
+    "subtype": [
+      {
+        "key": "path",
+        "type": "string",
+        "label": "Path",
+        "question": "Enter path to file"
+      },
+      {
+        "key": "why",
+        "type": "string",
+        "label": "Why",
+        "question": "Why should this file be deleted?",
+        "default": "It's malicious."
+      }
+    ],
+    "while": "Do you want the module to delete more files?",
+    "response": [
+      { "path": "/tmp/virus", "why": "It's malicious." },
+      { "path": "/home/alice/.ssh/authorized_keys", "why": "She left the company." }
+    ]
+  }
+]
+```
+
+The `input.json` file is converted and merged into the main `def.json` during the build, using the `input` build step.
+
 ### Deploy your policy set to a remote hub
 
 ```
@@ -178,82 +220,7 @@ from pydantic import (
 $
 ```
 
-## Build Steps
-
-The standard commands for adding content such as `cfbs add` will handle most common needs.
-If you are creating a new module or needing more control you can edit `cfbs.json` and the `steps` entry.
-
-An example of the result of `cfbs add ./policy.cf`:
-
-```json
-{
-  "name": "./policy.cf",
-  "description": "Local policy file added using cfbs command line",
-  "tags": ["local"],
-  "dependencies": ["autorun"],
-  "steps": ["copy ./policy.cf services/autorun/policy.cf"],
-  "added_by": "cfbs add"
-}
-```
-
-In most cases the steps are designed to copy something locally to the resulting `out/masterfiles` output directory created when running `cfbs build`.
-
-The `source` parameters in each step are relative either to the local cfbs project directory or the `subdirectory` specified in the module definition.
-
-Here is an example from the `client-initiated-reporting` module
-
-```json
-{
-  "name": "client-initiated-reporting",
-  "description": "Enable client initiated reporting and disable pull collection",
-  "tags": ["experimental", "reporting"],
-  "repo": "https://github.com/cfengine/modules",
-  "by": "https://github.com/cfengine",
-  "version": "0.1.1",
-  "commit": "c3b7329b240cf7ad062a0a64ee8b607af2cb912a",
-  "subdirectory": "reporting/client-initiated-reporting",
-  "steps": ["json def.json def.json"],
-  "added_by": "cfbs add"
-}
-```
-
-The source `def.json` to be merged above will be located at `reporting/client-initiated-reporting/def.json`.
-
-During `cfbs build` either the local contents or the repository contents are copied to an auto-generated numeric directory of the form: `out/steps/<step#>_<module_name>_<commit_sha>`.
-This is the working directory where commands are run and where `source` paths should be specified relative to.
-
-```
-out/steps/001_masterfiles_5c7dc5b43088e259a94de4e5a9f17c0ce9781a0f/
-```
-
-The `destination` parameter below should all be relative paths to `out/masterfiles`.
-
-
-### Available steps are:
-
-Note that the build steps are run inside each modules directory (the temporary copy of it inside `out/steps`).
-Scripts and source files are read from that directory.
-Destination is relative to the output policy set (`out/masterfiles`).
-
-#### `copy <source> <destination>`
-- Copy a single file or a directory recursively.
-
-#### `run <command>`
-- Run a shell command / script.
-- Usually used to prepare the module directory, delete files, etc. before a copy step.
-- Running scripts should be avoided if possible.
-
-#### `delete <paths ...>`
-- Delete multiple files or paths recursively.
-
-#### `json <source> <destination>`
-- Merge the source json file into the destination json file.
-
-#### `append <source> <destination>`
-- Append the source file to the end of destination file.
+## The cfbs.json format
 
-#### `directory <source> <destination>`
-- Copy any .cf policy files recursively and add their paths to `def.json`'s `inputs`.
-- Enable `services_autorun_bundles` class in `def.json`.
-- Merge any `def.json` recursively into `out/masterfiles/def.json`.
-- Copy any other files with their existing directory structure to destination.
+More advanced users and module authors may need to understand, write, or edit `cfbs.json` files.
+The format of those files and how `cfbs` uses them is explained in detail in [JSON.md](./JSON.md).

cfbs/build.py

@@ -1,15 +1,16 @@
 import os
-import glob
 import logging as log
 from cfbs.utils import (
     canonify,
     cp,
+    find,
     merge_json,
     mkdir,
     pad_right,
     read_json,
     rm,
     sh,
+    strip_left,
     touch,
     user_error,
     write_json,
@@ -170,15 +171,14 @@ def _perform_build_step(module, step, max_length):
             if file.endswith(".cf"):
                 files.append(file)
             elif file.endswith("/"):
-                pattern = "%s**/*.cf" % file
-                files += glob.glob(pattern, recursive=True)
+                cf_files = find("out/masterfiles/" + file, extension=".cf")
+                files += (strip_left(f, "out/masterfiles/") for f in cf_files)
             else:
                 user_error(
                     "Unsupported filetype '%s' for build step '%s': "
                     % (file, operation)
                     + "Expected directory (*/) of policy file (*.cf)"
                 )
-        files = [os.path.join("services", "cfbs", file) for file in files]
         print("%s policy_files '%s'" % (prefix, "' '".join(files) if files else ""))
         augment = {"inputs": files}
         log.debug("Generated augment: %s" % pretty(augment))

cfbs/cfbs_config.py

@@ -206,7 +206,9 @@ def _find_dependencies(self, modules, exclude):
 
     def _add_policy_files_build_step(self, module):
         name = module["name"]
-        step = "policy_files %s" % name
+        step = "policy_files services/cfbs/" + (
+            name[2:] if name.startswith("./") else name
+        )
         module["steps"].append(step)
         log.debug("Added build step '%s' for module '%s'" % (step, name))
 

tests/shell/010_local_add.sh

@@ -9,26 +9,27 @@ rm -rf .git
 cfbs --non-interactive init
 cfbs status
 
-echo 'bundle agent bogus {
+echo 'bundle agent bogus_bundle {
   reports:
       "This is $(this.promise_filename):$(this.bundle)!";
 }
-' > bogus.cf
+' > bogus_file.cf
 
 
-cfbs --non-interactive add ./bogus.cf
+cfbs --non-interactive add ./bogus_file.cf
 
-grep '"name": "./bogus.cf"' cfbs.json
-grep '"policy_files ./bogus.cf"' cfbs.json
-grep '"bundles bogus"' cfbs.json
+grep '"name": "./bogus_file.cf"' cfbs.json
+grep '"copy ./bogus_file.cf services/cfbs/bogus_file.cf"' cfbs.json
+grep '"policy_files services/cfbs/bogus_file.cf"' cfbs.json
+grep '"bundles bogus_bundle"' cfbs.json
 
 cfbs status
 cfbs build
 
 grep '"inputs"' out/masterfiles/def.json
-grep '"services/cfbs/bogus.cf"' out/masterfiles/def.json
+grep 'bogus_file.cf' out/masterfiles/def.json
 
 grep '"control_common_bundlesequence_end"' out/masterfiles/def.json
-grep '"bogus"' out/masterfiles/def.json
+grep '"bogus_bundle"' out/masterfiles/def.json
 
-ls out/masterfiles/services/cfbs/bogus.cf
+ls out/masterfiles/services/cfbs/bogus_file.cf

tests/shell/016_add_folders.sh

@@ -40,7 +40,7 @@ cfbs status
 cfbs status | grep "./doofus/"
 grep '"name": "./doofus/"' cfbs.json
 grep '"directory ./ services/cfbs/doofus/"' cfbs.json
-grep '"policy_files ./doofus/"' cfbs.json
+grep '"policy_files services/cfbs/doofus/"' cfbs.json
 grep '"bundles doofus"' cfbs.json
 
 cfbs build

tests/shell/all.sh

@@ -28,3 +28,5 @@ bash tests/shell/022_update_input_fail_variable.sh
 bash tests/shell/023_update_input_fail_number.sh
 bash tests/shell/024_update_input_fail_bundle.sh
 bash tests/shell/025_add_input_remove.sh
+
+echo "All cfbs shell tests completed successfully!"