Merge pull request #247 from CycloneDX/feat/conda-support

FEATURE: Add Conda Support

Author: madpah

README.md

@@ -15,6 +15,7 @@
 This project provides a runnable Python-based application for generating CycloneDX bill-of-material documents from either:
 1. Your current Python Environment
 2. Your project's manifest (e.g. `Pipfile.lock`, `poetry.lock` or `requirements.txt`)
+3. Conda as a Package Manager
 
 The BOM will contain an aggregate of all your current project's dependencies, or those defined by the manifest you supply.
 
@@ -39,43 +40,42 @@ poetry add cyclonedx-bom
 Once installed, you can access the full documentation by running `--help`:
 
 ```
-$ cyclonedx-py --help
-usage: client.py [-h] (-e | -p | -r) [-pf FILE_PATH] [-rf FILE_PATH]
+$ cyclonedx-bom --help
+usage: cyclonedx-bom [-h] (-c | -cj | -e | -p | -pip | -r) [-i FILE_PATH]
                  [--format {json,xml}] [--schema-version {1.3,1.2,1.1,1.0}]
                  [-o FILE_PATH] [-F] [-X]
 
 CycloneDX SBOM Generator
 
 optional arguments:
   -h, --help            show this help message and exit
+  -c, --conda           Build a SBOM based on the output from `conda list
+                        --explicit` or `conda list --explicit --md5`
+  -cj, --conda-json     Build a SBOM based on the output from `conda list
+                        --json`
   -e, --e, --environment
                         Build a SBOM based on the packages installed in your
                         current Python environment (default)
   -p, --p, --poetry     Build a SBOM based on a Poetry poetry.lock's contents.
                         Use with -pf to specify absolute pathto a
                         `poetry.lock` you wish to use, else we'll look for one
                         in the current working directory.
+  -pip, --pip           Build a SBOM based on a PipEnv Pipfile.lock's
+                        contents. Use with --pip-file to specify absolute
+                        pathto a `Pipefile.lock` you wish to use, else we'll
+                        look for one in the current working directory.
   -r, --r, --requirements
                         Build a SBOM based on a requirements.txt's contents.
                         Use with -rf to specify absolute pathto a
                         `requirements.txt` you wish to use, else we'll look
                         for one in the current working directory.
   -X                    Enable debug output
 
-Poetry:
-  Additional optional arguments if you are setting the input type to
-  `poetry`
+Input Method:
+  Flags to determine how `cyclonedx-bom` obtains it's input
 
-  -pf FILE_PATH, --pf FILE_PATH, --poetry-file FILE_PATH
-                        Path to a the `poetry.lock` file you wish to parse
-
-Requirements:
-  Additional optional arguments if you are setting the input type to
-  `requirements`.
-
-  -rf FILE_PATH, --rf FILE_PATH, --requirements-file FILE_PATH
-                        Path to a the `requirements.txt` file you wish to
-                        parse
+  -i FILE_PATH, --in-file FILE_PATH
+                        File to read input from, or STDIN if not specified
 
 SBOM Output Configuration:
   Choose the output format and schema version
@@ -96,32 +96,43 @@ SBOM Output Configuration:
 This will produce the most accurate and complete CycloneDX BOM as it will include all transitive dependencies required
 by the packages defined in your project's manifest (think `requriements.txt`).
 
-When using _Environment_ as the source, any license information avaialble from the installed packages will also be 
+When using _Environment_ as the source, any license information available from the installed packages will also be 
 included in the generated CycloneDX BOM.
 
 Simply run:
 
 ```
-cyclonedx-py -e -o -
+cyclonedx-bom -e -o -
 ```
 
 This will generate a CycloneDX including all packages installed in your current Python environment and output to STDOUT
 in XML using the latest schema version `1.3` by default.
 
 
-### Building CycloneDX from your Manifest
+### Building CycloneDX from your Manifest / Package Manager
 
 _Note: Manifest scanning limits the amount of information available. Each manifest type contains different information
 but all are significantly less complete than scanning your actual Python Environment._
 
+#### Conda
+
+We support parsing output from Conda in various formats:
+- Explict output (run `conda list --explicit` or `conda list --explicit --md5`)
+- JSON output (run `conda list --json`)
+
+As example:
+```
+conda list --explicit --md5 | cyclonedx-bom -c -o cyclonedx.xml
+```
+
 #### Poetry
 
 We support parsing your `poetry.lock` file which should be committed along with your `pyrpoject.toml` and details
 exact pinned versions.
 
-You can then run `cyclonedx-py` as follows:
+You can then run `cyclonedx-bom` as follows:
 ```
-cyclonedx-py -p -pf PATH/TO/poetry.lock -o sbom.xml
+cyclonedx-bom -p -i PATH/TO/poetry.lock -o sbom.xml
 ```
 
 #### Pip / Requirements
@@ -135,13 +146,13 @@ pip freeze > requirements.txt
 
 You can then run `cyclonedx-py` as follows:
 ```
-cyclonedx-py -r -rf PATH/TO/requirements.txt -o sbom.xml
+cyclonedx-bom -r -i PATH/TO/requirements.txt -o sbom.xml
 ```
 
 This will generate a CycloneDX and output to STDOUT in XML using the latest schema version `1.3` by default.
 
-**Note:** If you failed to freeze your dependencies before passing the `requirements.txt` data to `cyclonedx-py`, you'll 
-be warned about this and the dependencies that do not have pinned versions WILL NOT be included in the resulting 
+**Note:** If you failed to freeze your dependencies before passing the `requirements.txt` data to `cyclonedx-bom`, 
+you'll be warned about this and the dependencies that do not have pinned versions WILL NOT be included in the resulting 
 CycloneDX output.
 
 ```

cyclonedx_py/client.py

@@ -26,10 +26,11 @@
 from cyclonedx.model.bom import Bom, Tool
 from cyclonedx.output import BaseOutput, get_instance, OutputFormat, SchemaVersion
 from cyclonedx.parser import BaseParser
+from cyclonedx.parser.conda import CondaListExplicitParser, CondaListJsonParser
 from cyclonedx.parser.environment import EnvironmentParser
-from cyclonedx.parser.pipenv import PipEnvFileParser
-from cyclonedx.parser.poetry import PoetryFileParser
-from cyclonedx.parser.requirements import RequirementsFileParser
+from cyclonedx.parser.pipenv import PipEnvParser
+from cyclonedx.parser.poetry import PoetryParser
+from cyclonedx.parser.requirements import RequirementsParser
 
 
 class CycloneDxCmd:
@@ -100,6 +101,16 @@ def get_arg_parser() -> argparse.ArgumentParser:
         arg_parser = argparse.ArgumentParser(description='CycloneDX SBOM Generator')
 
         input_group = arg_parser.add_mutually_exclusive_group(required=True)
+        input_group.add_argument(
+            '-c', '--conda', action='store_true',
+            help='Build a SBOM based on the output from `conda list --explicit` or `conda list --explicit --md5`',
+            dest='input_from_conda_explicit'
+        )
+        input_group.add_argument(
+            '-cj', '--conda-json', action='store_true',
+            help='Build a SBOM based on the output from `conda list --json`',
+            dest='input_from_conda_json'
+        )
         input_group.add_argument(
             '-e', '--e', '--environment', action='store_true',
             help='Build a SBOM based on the packages installed in your current Python environment (default)',
@@ -124,34 +135,14 @@ def get_arg_parser() -> argparse.ArgumentParser:
             dest='input_from_requirements'
         )
 
-        req_input_group = arg_parser.add_argument_group(
-            title='Poetry',
-            description='Additional optional arguments if you are setting the input type to `poetry`'
-        )
-        req_input_group.add_argument(
-            '-pf', '--pf', '--poetry-file', action='store', metavar='FILE_PATH', default='poetry.lock',
-            help='Path to a the `poetry.lock` file you wish to parse',
-            dest='input_poetry_file', required=False
-        )
-
-        req_input_group = arg_parser.add_argument_group(
-            title='PipEnv',
-            description='Additional optional arguments if you are setting the input type to `pipenv`'
+        input_method_group = arg_parser.add_argument_group(
+            title='Input Method',
+            description='Flags to determine how `cyclonedx-bom` obtains it\'s input'
         )
-        req_input_group.add_argument(
-            '--pip-file', action='store', metavar='FILE_PATH', default='Pipfile.lock',
-            help='Path to a the `Pipfile.lock` file you wish to parse',
-            dest='input_pipenv_file', required=False
-        )
-
-        req_input_group = arg_parser.add_argument_group(
-            title='Requirements',
-            description='Additional optional arguments if you are setting the input type to `requirements`.'
-        )
-        req_input_group.add_argument(
-            '-rf', '--rf', '--requirements-file', action='store', metavar='FILE_PATH', default='requirements.txt',
-            help='Path to a the `requirements.txt` file you wish to parse',
-            dest='input_requirements_file', required=False
+        input_method_group.add_argument(
+            '-i', '--in-file', action='store', metavar='FILE_PATH',
+            type=argparse.FileType('r'), default=(None if sys.stdin.isatty() else sys.stdin),
+            help='File to read input from, or STDIN if not specified', dest='input_source', required=False
         )
 
         output_group = arg_parser.add_argument_group(
@@ -193,38 +184,26 @@ def _error_and_exit(message: str, exit_code: int = 1):
     def _get_input_parser(self) -> BaseParser:
         if self._arguments.input_from_environment:
             return EnvironmentParser()
+
+        # All other Parsers will require some input - grab it now!
+        input_data_fh = self._arguments.input_source
+        with input_data_fh:
+            input_data = input_data_fh.read()
+            input_data_fh.close()
+
+        if self._arguments.input_from_conda_explicit:
+            return CondaListExplicitParser(conda_data=input_data)
+        elif self._arguments.input_from_conda_json:
+            return CondaListJsonParser(conda_data=input_data)
         elif self._arguments.input_from_pip:
-            pipfile_lock_file = os.path.realpath(self._arguments.input_pipenv_file)
-            if CycloneDxCmd._validate_file_exists(self._arguments.input_pipenv_file):
-                # A Pipfile.lock path was provided
-                return PipEnvFileParser(pipenv_lock_filename=pipfile_lock_file)
-            else:
-                self._error_and_exit(f'The provided file \'{pipfile_lock_file}\' does not exist')
+            return PipEnvParser(pipenv_contents=input_data)
         elif self._arguments.input_from_poetry:
-            poetry_lock_file = os.path.realpath(self._arguments.input_poetry_file)
-            if CycloneDxCmd._validate_file_exists(self._arguments.input_poetry_file):
-                # A poetry.lock path was provided
-                return PoetryFileParser(poetry_lock_filename=poetry_lock_file)
-            else:
-                self._error_and_exit('The provided file \'{}\' does not exist'.format(
-                    poetry_lock_file
-                ))
+            return PoetryParser(poetry_lock_contents=input_data)
         elif self._arguments.input_from_requirements:
-            requirements_file = os.path.realpath(self._arguments.input_requirements_file)
-            if CycloneDxCmd._validate_file_exists(self._arguments.input_requirements_file):
-                # A requirements.txt path was provided
-                return RequirementsFileParser(requirements_file=requirements_file)
-            else:
-                self._error_and_exit('The provided file \'{}\' does not exist'.format(
-                    requirements_file
-                ))
+            return RequirementsParser(requirements_content=input_data)
         else:
             raise ValueError('Parser type could not be determined.')
 
-    @staticmethod
-    def _validate_file_exists(file_path: str) -> bool:
-        return os.path.exists(file_path)
-
 
 def main():
     parser = CycloneDxCmd.get_arg_parser()

poetry.lock

@@ -16,14 +16,15 @@ include = [
 
 [tool.poetry.dependencies]
 python = "^3.6"
-cyclonedx-python-lib = "^0.9.1"
+cyclonedx-python-lib = "^0.10.2"
 
 [tool.poetry.dev-dependencies]
 tox = "^3.24.3"
 coverage = "^5.5"
 flake8 = "^3.9.2"
 
 [tool.poetry.scripts]
+cyclonedx-bom = 'cyclonedx_py.client:main'
 cyclonedx-py = 'cyclonedx_py.client:main'
 
 [build-system]

pyproject.toml

@@ -0,0 +1,5 @@
+# This file may be used to create an environment using:
+# $ conda create --name <env> --file <this file>
+# platform: osx-64
+@EXPLICIT
+https://repo.anaconda.com/pkgs/main/osx-64/setuptools-52.0.0-py39hecd8cb5_0.conda#5c9e48476978303d04650c21ee55f365

tests/fixtures/conda-list-explicit-simple.txt

@@ -20,9 +20,6 @@
 import os.path
 import subprocess
 import tempfile
-from unittest.mock import mock_open, patch
-
-from cyclonedx_py.client import CycloneDxCmd
 
 from base import BaseXmlTestCase
 
@@ -33,27 +30,38 @@
 
 class TestCycloneDxXml(BaseXmlTestCase):
 
-    def test_run(self):
-        parser = CycloneDxCmd.get_arg_parser()
+    def test_environment(self):
         with tempfile.TemporaryDirectory() as dirname:
-            args = parser.parse_args(args=('-r', '-o', os.path.join(dirname, 'sbom.xml')))
-            with open(os.path.join(FIXTURES_DIRECTORY, 'requirements-simple.txt'), 'r') as req_file:
-                with patch('builtins.open', mock_open(read_data=req_file.read())) as mock_req_file:
-                    with patch('cyclonedx_py.client.CycloneDxCmd._validate_file_exists') as mock_exists:
-                        CycloneDxCmd(args).execute()
+            subprocess.check_output([
+                'cyclonedx-py',
+                '-e',
+                '-o', os.path.join(dirname, 'sbom.xml'),
+            ])
 
-                        mock_exists.assert_called_with('requirements.txt')
-                        mock_req_file.assert_called()
+    def text_conda_list_explicit(self):
+        with tempfile.TemporaryDirectory() as dirname:
+            # Run command to generate latest 1.3 XML SBOM from Requirements File
+            subprocess.check_output([
+                'cyclonedx-bom',
+                '-c',
+                '-i', os.path.join(FIXTURES_DIRECTORY, 'conda-list-explicit-simple.txt'),
+                '-o', os.path.join(dirname, 'sbom.xml'),
+            ])
 
-            req_file.close()
+            with open(os.path.join(dirname, 'sbom.xml'), 'r') as f, \
+                    open(os.path.join(FIXTURES_DIRECTORY, 'bom_v1.3_setuptools.xml')) as expected:
+                self.assertEqualXmlBom(f.read(), expected.read(),
+                                       namespace='http://cyclonedx.org/schema/bom/1.3')
+                f.close()
+                expected.close()
 
     def test_requirements_txt_file(self):
         with tempfile.TemporaryDirectory() as dirname:
             # Run command to generate latest 1.3 XML SBOM from Requirements File
             subprocess.check_output([
                 'cyclonedx-py',
                 '-r',
-                '-rf', os.path.join(FIXTURES_DIRECTORY, 'requirements-simple.txt'),
+                '-i', os.path.join(FIXTURES_DIRECTORY, 'requirements-simple.txt'),
                 '-o', os.path.join(dirname, 'sbom.xml'),
             ])
 
@@ -79,7 +87,7 @@ def _do_test_requirements_txt_file_for_version(self, schema_version: str):
             subprocess.check_output([
                 'cyclonedx-py',
                 '-r',
-                '-rf', os.path.join(FIXTURES_DIRECTORY, 'requirements-simple.txt'),
+                '-i', os.path.join(FIXTURES_DIRECTORY, 'requirements-simple.txt'),
                 '--schema-version', schema_version,
                 '-o', os.path.join(dirname, 'sbom.xml'),
             ])