refscan

refscan is a command-line tool people can use to scan the NMDC MongoDB database for referential integrity violations.

%% This is the source code of a Mermaid diagram, which GitHub will render as a diagram.
%% Note: PyPI does not render Mermaid diagrams, and instead displays their source code.
%%       Reference: https://github.com/pypi/warehouse/issues/13083
graph LR
    schema[LinkML<br>schema]
    database[(MongoDB<br>database)]
    script[["refscan.py"]]
    violations["List of<br>violations"]
    references["List of<br>references"]:::dashed_border
    schema --> script
    database --> script
    script -.-> references
    script --> violations
    
    classDef dashed_border stroke-dasharray: 5 5

Loading

Table of contents

• refscan
  • Table of contents
  • How it works
  • Limitations
  • Usage
    • Install
    • Run
      • The MongoDB connection string (--mongo-uri)
      • The schema (--schema)
      • Output
    • Update
    • Uninstall
  • Development
    • Clone repository
    • Create virtual environment
    • Install dependencies
    • Make changes
    • Run tests
    • Format code
      • Check format
  • Building and publishing
    • Build for production
    • Test the build process locally
  • Appendix
    • refgraph

How it works

refscan does its job in two stages:

1. It uses the LinkML schema to determine where references can exist in a MongoDB database that conforms to the schema.

   Example: The schema might say that, if a document in the biosample_set collection has a field named associated_studies, that field must contain a list of ids of documents in the study_set collection.

2. It scans the MongoDB database to check the integrity of all the references that do exist.

   Example: For each document in the biosample_set collection that has a field named associated_studies, for each value in that field, confirm there is a document having that id in the study_set collection.

Limitations

refscan was designed under the assumption that every document in every collection described by the schema has a field named type, whose value is the class_uri of the schema class the document represents an instance of. refscan uses that class_uri value (in that type field) to determine the name of that schema class, which it, in turn, uses to determine which fields of that document can contain references.

Usage

Install

Assuming you have pipx installed, you can install the tool by running the following command:

pipx install refscan

pipx is a tool people can use to download and install Python scripts that are hosted on PyPI. You can install pipx by running $ python -m pip install pipx.

Run

You can display the tool's --help snippet by running:

refscan --help

At the time of this writing, the tool's --help snippet is:

 Usage: refscan [OPTIONS]

 Scans the NMDC MongoDB database for referential integrity violations.

╭─ Options ──────────────────────────────────────────────────────────────────────────────╮
│ *  --schema                               FILE  Filesystem path at which the YAML file │
│                                                 representing the schema is located.    │
│                                                 [default: None]                        │
│                                                 [required]                             │
│    --database-name                        TEXT  Name of the database.                  │
│                                                 [default: nmdc]                        │
│    --mongo-uri                            TEXT  Connection string for accessing the    │
│                                                 MongoDB server. If you have Docker     │
│                                                 installed, you can spin up a temporary │
│                                                 MongoDB server at the default URI by   │
│                                                 running: $ docker run --rm --detach -p │
│                                                 27017:27017 mongo                      │
│                                                 [env var: MONGO_URI]                   │
│                                                 [default: mongodb://localhost:27017]   │
│    --verbose                                    Show verbose output.                   │
│    --skip-source-collection,--skip        TEXT  Name of collection you do not want to  │
│                                                 search for referring documents. Option │
│                                                 can be used multiple times.            │
│                                                 [default: None]                        │
│    --reference-report                     FILE  Filesystem path at which you want the  │
│                                                 program to generate its reference      │
│                                                 report.                                │
│                                                 [default: references.tsv]              │
│    --violation-report                     FILE  Filesystem path at which you want the  │
│                                                 program to generate its violation      │
│                                                 report.                                │
│                                                 [default: violations.tsv]              │
│    --version                                    Show version number and exit.          │
│    --no-scan                                    Generate a reference report, but do    │
│                                                 not scan the database for violations.  │
│    --locate-misplaced-documents                 For each referenced document not found │
│                                                 in any of the collections the schema   │
│                                                 allows, also search for it in all      │
│                                                 other collections.                     │
│    --help                                       Show this message and exit.            │
╰────────────────────────────────────────────────────────────────────────────────────────╯

Note: The above snippet was captured from a terminal window whose width was 90 characters.

The MongoDB connection string (--mongo-uri)

As documented in the --help snippet above, you can provide the MongoDB connection string to the tool via either (a) the --mongo-uri option; or (b) an environment variable named MONGO_URI. The latter can come in handy when the MongoDB connection string contains information you don't want to appear in your shell history, such as a password.

Here's how you could create that environment variable:

export MONGO_URI='mongodb://username:password@localhost:27017'

The schema (--schema)

As documented in the --help snippet above, you can provide the path to a YAML-formatted LinkML schema file to the tool via the --schema option.

Show/hide tips for getting a schema file

---

If you have curl installed, you can download a YAML file from GitHub by running the following command (after replacing the {...} placeholders and customizing the path):

# Download the raw content of https://github.com/{user_or_org}/{repo}/blob/{branch}/path/to/schema.yaml
curl -o schema.yaml https://raw.githubusercontent.com/{user_or_org}/{repo}/{branch}/path/to/schema.yaml

For example:

# Download the raw content of https://github.com/microbiomedata/berkeley-schema-fy24/blob/main/nmdc_schema/nmdc_materialized_patterns.yaml
curl -o schema.yaml https://raw.githubusercontent.com/microbiomedata/berkeley-schema-fy24/main/nmdc_schema/nmdc_materialized_patterns.yaml

---

Output

While refscan is running, it will display console output indicating what it's currently doing.

Once the scan is complete, the reference report (TSV file) and violation report (TSV file) will be available in the current directory (or in custom directories, if any were specified via CLI options).

Update

You can update the tool to the latest version available on PyPI by running:

pipx upgrade refscan

Uninstall

You can uninstall the tool from your computer by running:

pipx uninstall refscan

Development

We use Poetry to both (a) manage dependencies and (b) build distributable packages that can be published to PyPI.

• pyproject.toml: Configuration file for Poetry and other tools (was initialized via $ poetry init)
• poetry.lock: List of dependencies, both direct and indirect/transitive

Clone repository

git clone https://github.com/microbiomedata/refscan.git
cd refscan

Create virtual environment

Create a Poetry virtual environment and attach to its shell:

poetry shell

You can see information about the Poetry virtual environment by running: $ poetry env info

You can detach from the Poetry virtual environment's shell by running: $ exit

From now on, I'll refer to the Poetry virtual environment's shell as the "Poetry shell."

Install dependencies

At the Poetry shell, install the project's dependencies:

poetry install

Make changes

Edit the tool's source code and documentation however you want.

While editing the tool's source code, you can run the tool as you normally would in order to test things out.

refscan --help

Run tests

We use pytest as the testing framework for refscan.

Tests are defined in the tests directory.

You can run the tests by running the following command from the root directory of the repository:

poetry run pytest

Format code

We use black as the code formatter for refscan.

We do not use it with its default options. Instead, we include an option that allows lines to be 120 characters instead of the default 88 characters. That option is defined in the [tool.black] section of pyproject.toml.

You can format all the Python code in the repository by running this command from the root directory of the repository:

poetry run black .

Check format

You can check the format of the Python code by including the --check option, like this:

poetry run black --check .

Building and publishing

Build for production

Whenever someone publishes a GitHub Release in this repository, a GitHub Actions workflow will automatically build a package and publish it to PyPI. That package will have a version identifier that matches the name of the Git tag associated with the Release.

Test the build process locally

In case you want to test the build process locally, you can do so by running:

poetry build

That will create both a source distribution file (whose name ends with .tar.gz) and a wheel file (whose name ends with .whl) in the dist directory.

Appendix

refgraph

When pipx installs refscan, it also installs a program called refgraph. refgraph is a program you can use to generate a web-based, interactive graph (network diagram) of the relationships that can exist between documents in a database that conforms to a given schema. It can present the relationships in terms of either database collections or schema classes.

You can learn more about refgraph by running:

refgraph --help

Note: refgraph is still in early development and its features and CLI are subject to change.