Developer Knowledge Base
This page is a collection of assorted tasks and topics developers from the NIST OSCAL Team will need or want as part of development activities. Documentation items, when specifically relevant to OSCAL, could and should make their way into official documentation with commits to the git repository's Markdown files. For official documentation, bug reports for error corrections and feature request issues identify and track work items. Changes to information here are not tracked that way. See the note on the bottom of this page. Information this page and children pages are for supporting activities underlying OSCAL development: code snippets for testing data format snippets (JSON, XML, YAML), software configuration, and common queries in different querying languages (JMESPath for JSON, XPath for XML, etc.).
- recording
- audio transcription
- example Metaschema modules, XML schemas, and instances from this session
You do not have to a fork (in this context github.com/yourusername/OSCAL is a fork; github.com/usnistgov/OSCAL) and staff can create branches on the upstream repository. However, you may prefer to use your fork to prepare work before pushing to the upstream. If so, you may need to sync your develop and main branches from your fork with the upstream.
To follow this process there are important assumptions you make.
- The GitHub web interface has a sync branches feature for forks in our projects. This approach uses merge commits on the branches in the fork, and not 1:1 match of the branches. This strategy can lead to messier history history and potential conflicts later, we do not recommend doing this.
- Your fork is the
originremote, and the upstream usnistgov/OSCAL repo isupstream. - You do not make any changes you want to save your
developormainbranch in any circumstance. Following this guidance, they will be blown away.
To reset your fork's develop branch with the upstream develop, use the commands below.
$ git remote -v # If you results match below, you can skip the next few commands and skip to git fetch
origin [email protected]:yourusername/OSCAL.git (fetch)
origin [email protected]:yourusername/OSCAL.git (push)
upstream [email protected]:usnistgov/OSCAL.git (fetch)
upstream [email protected]:usnistgov/OSCAL.git (push)
$ git remote remove origin
$ git remote remove upstream # It may throw an error because it doesn't exist, this is to be safe.
$ git remote add origin [email protected]:usnistgov/OSCAL.git
$ git remote add upstream [email protected]:yourusername/OSCAL.git
$ git fetch --all # This is important or the next commands will reflect older "current" commits on the remotes, correct or incorrect.
$ git checkout develop
$ git reset --hard upstream/develop
$ git push origin develop -fTo reset your fork's main branch with the upstream main:
$ git remote -v # If you results match below, you can skip the next few commands and skip to git fetch
origin [email protected]:yourusername/OSCAL.git (fetch)
origin [email protected]:yourusername/OSCAL.git (push)
upstream [email protected]:usnistgov/OSCAL.git (fetch)
upstream [email protected]:usnistgov/OSCAL.git (push)
$ git remote remove origin
$ git remote remove upstream # It may throw an error because it doesn't exist, this is to be safe.
$ git remote add origin [email protected]:usnistgov/OSCAL.git
$ git remote add upstream [email protected]:yourusername/OSCAL.git
$ git fetch --all # This is important or the next commands will reflect older "current" commits on the remotes, correct or incorrect.
$ git checkout main
$ git reset --hard upstream/main
$ git push origin main -fIf you are asked to make a pull request for an issue, you need to create a branch based off your target branch. Before you make a feature branch you likely need to make a rough sketch branch. The target may be the usnistgov/OSCAL develop branch, a feature- prefix branch, or main branch. You can approach this from 1 or 2 methods.
- Traditional Command Line Method with
git - GitHub Web Interface Method
You need to first ensure you have selected the correct target branch for the PR. In this case, in the example below, I will assume develop, not main.
$ git clone [email protected]:usnistgov/OSCAL.git
$ pushd OSCAL
$ git remote -v # Ensure below origin has usnistgov in the URL, if not use the correct remote alias if not origin
origin [email protected]:usnistgov/OSCAL.git (fetch)
origin [email protected]:usnistgov/OSCAL.git (push)
$ git checkout --track origin/develop # first time
$ # or if you see an error that branch develop already exists
$ git checkout develop # after first time
$ git fetch --all # ensure if you have done the above days or weeks again
$ git reset --hard origin/develop # IMPORTANT: you will delete uncommitted work on your local develop, but you should not use this or main anyway
$ git checkout -b name-for-new-pr-branch-targeting-develop
$ git push origin HEAD # or the appropriate remote aliasYou need to first ensure you have selected the correct target branch for the PR. In this case, in the example below, I will assume develop, not main.
- Open the relevant issue
- Open the create branch from issue menu on the lower right-hand side of the issue.
- Click the change branch source menu and choose the target
developbranch and click the Create Branch button.
- Follow clone instructions accordingly.
- Open the OSCAL repository on your workstation.
- Checkout your new branch from the target branch (
developormain):
$ git remote -v # confirm origin is usnistgov/OSCAL
me [email protected]:youruser/OSCAL.git (fetch)
me [email protected]:youruser/OSCAL.git (push)
origin [email protected]:usnistgov/OSCAL.git (fetch)
origin [email protected]:usnistgov/OSCAL.git (push)
$ git fetch --all
$ git checkout --track origin/develop
$ git checkout -b new-pr-to-develop # create PR branch
$ git push me HEAD # push to your fork
$ git submodule update --init --recursive
$ cd build/metaschema/
$ git fetch --all
$ git remote -v # confirm origin is usnistgov/metaschema-xslt
me [email protected]:youruser/metaschema-xslt.git (fetch)
me [email protected]:youruser/metaschema-xslt.git (push)
origin [email protected]:usnistgov/metaschema-xslt.git (fetch)
origin [email protected]:usnistgov/metaschema-xslt.git (push)
$ git checkout --track me/metaschema-fork-updated
$ cd ..
$ git submodule set-url build/metaschema [email protected]:youruser/metaschema-xslt.git
$ git add .gitmodules build/metaschema
$ git commit -m "[WIP] Test metaschema-xslt sub-module change to test tooling to fix model issue."Schematron is an ISO specification for an XSLT-based domain-specific language to validate XML documents. It can perform complex logical ("business rule") validations that commonplace schema-based solutions (e.g. Relax NG, XML Schema) cannot.
You can use Schematron integrated into the OxygenXML IDE where it completes a compilation and transformation for you to build a report in the companion XML format for validation results, SVRL. If you wish to use Saxon or similar XSLT processor and not rely on a tool that embeds Schematron into an IDE, you will need to compile the Schematron into a XSLT and generate a result. Below is a diagram that visualizes this.
flowchart TD
Source[validations.sch] --> Compile{{Saxon Engine and Schematron}} -->|Compile Schematron to XSLT| Build((validations.xsl))
Validate[validate_me.xml] --> Transform{{Saxon Engine}}-->|Transform XML with XSL and generate results | Results((results.xml))
Build --> Transform
In the above diagram, squares are source files you would keep in your project, hexagons are dependencies and libraries to perform a process described in the adjacent label, and circles are temporary files. You can store them with additional source files in your project, but that is not necessary (and we discourage that for OSCAL projects). By using a XSLT engine to take the XSLT stylesheet forms of compiled rules, you can generate results in a SVRL report. You can use them as an input for marked up content for human review or further process information by parsing the SVRL's XML format.
Developers of the OSCAL project are encouraged to rely on a shared Docker image to ensure their environment is consistent with the rest of the team and CI/CD. Currently developers can choose to build the image themselves, or use the private Docker registry.
Note: currently the Docker registry is only accessible to members of the csd773 Docker Hub organization for administrative reasons, though we hope to make the registry public at some point.
If you are a OSCAL team member that does not have an account with Docker Hub that has access to the csd773 organization, ask on the team Gitter for instructions.
Otherwise, logging in is simple.
If you are using Docker desktop, simply sign in via the button the top right corner.
Run the following command and follow the instructions:
docker loginOnce you have logged in, simply pull the image as follows:
docker pull csd773/oscal-common-env:developBuilding the image locally is simple:
# From the root of the OSCAL project
# Force uninitialize submodules (optional)
# git submodule deinit -f .
# Ensure your submodules are up to date with the current commit
git submodule update --init --recursive
# Build the image
./build/build-oscal-env-dockerfile.shThe script build-oscal-env-dockerfile.sh will tell you the built image's tag. E.g.
# Script output from build-oscal-env-dockerfile.sh
Built and tagged csd773/oscal-common-env:fixed-a-thing, to push run:
docker push csd773/oscal-common-env:fixed-a-thingNote that the tag of the built image is derived from your current branch name. For more details see "Controlling the Docker image in CI".
To run a single command within the container run the following, replacing csd773/oscal-common-env:fixed-a-thing with the desired tag:
docker run \
-v $(pwd):/oscal \
csd773/oscal-common-env:fixed-a-thing \
your-command-hereTo run an interactive session, prepend the -it flags and do not put a command:
docker run -it \
-v $(pwd):/oscal \
csd773/oscal-common-env:fixed-a-thing
In both of these scenarios, the Docker image is passed the current working directory (e.g. the root of the repository) and maps it to /oscal in the container.
For more options like port mapping (required to access the Hugo serve web server), consult the Docker documentation
In the future we plan to rely on Docker images for CI/CD more. GitHub actions workflows that depend on the Docker images use the csd773 registry (building locally each run would take forever). Because of this, GitHub workflows relying on CI will attempt to:
- Pull the image with a tag corresponding to the current branch of the PR
- Pull the image with the
developtag
This is done to provide users with an "escape hatch" for modifying the environment within a PR.
Users that have logged in to the Docker registry can build and push new images as follows:
-
Build the image as instructed in the "Building the image manually" section
-
Push the resulting tagged image to the registry as instructed by the script output:
docker push csd773/oscal-common-env:$TAG # where $TAG is the tag of your built image
- Java Classes Generator
- JSON Schema Documentation Generator
- OpenAPI Tester
- Oxygen Emmet Plugin
- Oxygen Emmet Plugin
- XSD to JSON Schema converter
- XSpec Framework
- XSpec Helper Viewer
Note: Plugins in Oxygen can be installed automatically or manually
This is an opinionated view, but Oxygen has a variety of configuration options for setting up a scenario for applying one or more XSLT transformations against a source document, a scenario for executing a XProc pipeline, and many more.
Below is how some of us use Editor Variables and other configuration elements. As a rule, some of us will use a variable for the source document and hard-code the path to the transform or XProc pipeline when running the scenarios interactively.
- Open the XSLT stylesheet you will use for the transform, for example
nist-metaschema-COMPOSE.xsl. - Open an input file for transforms in a XSLT stylesheet, like the
oscal_ssp_metaschema.xmlMetaschema module. - Open the scenario menu or click the Ctrl+Shift+C keyboard shortcut.
- In the scenario menu, provide an appropriate name like
nist-metaschema-COMPOSE. - For the XML URL for the input file, use the variable
${currentFileURL}. - For the XSL URL for the stylesheet, do not use a variable, but hard-code the path toe transform on the workstation like
/path/to/code/repos/oscal/branches/origin/develop/build/metaschema/toolchains/xslt-M4/nist-metaschema-COMPOSE.xsl
This configuration will allow you to switch different inputs for testing while using the same stylesheet.
-
Install the extension XSLT/XPath for Visual Studio Code
-
In your VSCode settings (open the command palette and run
Preferences: Open User Settings (JSON)) append the following:"files.associations": { "*.xsl": "xslt" }
In several places across our stack we make use of Java-based tooling such as Saxon and Calabash.
The version of these tools can be pinned using POM files, such as the one present in the build/ directory.
Below is a simple template script that can be used to invoke a Java dependency directly from a POM file.
#!/usr/bin/env bash
# Fail early if an error occurs
set -Eeuo pipefail
if ! [ -x "$(command -v mvn)" ]; then
echo 'Error: Maven (mvn) is not in the PATH, is it installed?' >&2
exit 1
fi
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd)"
OSCAL_DIR="${SCRIPT_DIR}/../../../.." # Edit this to resolve to the OSCAL root from the script's directory
POM_FILE="${OSCAL_DIR}/build/pom.xml"
# Grab any additional files relative to the root OSCAL directory (like stylesheets)
MAIN_CLASS="net.sf.saxon.Transform" #edit this to target your application's main class. below are a few examples:
# XML Calabash: "com.xmlcalabash.drivers.Main"
# Saxon: "net.sf.saxon.Transform"
# Perform any argument processing here, such as preselecting stylesheets in Saxon, etc.
# Note here "${*// /\\ }" is a shell expansion that escapes spaces within arguments.
# For more information, see https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html
ARGS=${*// /\\ }
mvn \
-f "$POM_FILE" \
exec:java \
-Dexec.mainClass="$MAIN_CLASS" \
-Dexec.args="$ARGS"Note that this template is often extended to parse arguments directly. See an example of this in the profile resolution Saxon wrapper.
This approach has several key advantages:
- The script can refer to resources invariably to the user's current working directory, making it more portable.
- The user does not have to worry about downloading, copying, or otherwise managing any of the Java dependencies.
- A lot of the complexity of consuming stylesheets and other operations can be hidden from the user, making our stack more approachable.
Let's say we want to look for the following OSCAL syntax elements in XML at the same time:
- any definition of a field with the name "with-id" in one or more target documents
- any definition of a flag with the name "pattern" in one or more target documents
- any definition of an assembly with the name "matching" in one or more target documents
Apply the following query against the target document(s) (for this example any Metaschema definition in src/metaschema/*.xml), observing use of the | ("union") operator:
//(
define-flag[@name='pattern']
| define-assembly[@name='matching']
| define-field[@name='with-id']
)
Here is a sneakier way to do something similar:
//@name[.=('pattern','matching','with-id')]/parent::*
parent::* can also be shortened to .. for those who prefer.
Mathematically, this path returns a superset of the nodes returned by the first path given, although in a particular metaschema they are likely to be the same.
This takes advantage of a feature of general comparison operators in XPath (=, <, >, <=, >=) namely that they support comparing sequences, i.e. many values on either side of the operand, hence .=('a','b','c').
lychee or markdown-link-check will run an automated scan of the destination of almost every link (there is an exclude list) for every Markdown file in the repo and in the generated HTML that runs the website. Infrequently, GitHub's websites and external websites as well will rate-limit the runner's attempt open the link target or there is an internal failure on the runner. The resulting HTTP status is 0 (and that is not a normal one in the 200, 400, or 500 range). The build failure will indicate the link or an issue will be automatically opened that looks like the report below.
FILE: build/ci-cd/README.md
[✓] https://github.com/jessedc/ajv-cli
2 links checked.
If you can open the full URL reliably in your web browser on a developer workstation, you can safely presume this is a the infrequent hiccup on a GitHub Actions runner and close the issue as a false positive.