feat: add option to fail for incomplete analysis (#510)

This change adds a flag named `--fail-incomplete` to set a failure code
when some analysis results are unknown. This is useful in environments
where strictness is desired to prevent adding any new dependency until
it passes established policy. This is a mutually exclusive option with
`--audit`.

The name `--strict` was also considered but not used because that name
implies a "strict mode" which would be more than just the opposite of
audit mode. For instance, the ability to ignore errors with the
`--ignore-errors` flag should not be possible in a true strict mode. So,
the more specific option name was used to limit the effects of
specifying it.

README.md

@@ -201,6 +201,7 @@ following non-zero codes otherwise:
 |---------|-------|
 |1|Default failure code. An unrecoverable error was encountered.|
 |2|Phylum analysis is complete and contains a policy violation.|
+|5|Phylum analysis is incomplete. Only used when enabled [by option][script_options].|
 |6|Phylum analysis is incomplete and contains a policy violation.|
 |10|Dependency file(s) failed filtering and excluded from analysis. See [this FAQ][FAQ] for more.|
 |11|No dependency files were provided or detected.|

docs/integrations/azure_pipelines.md

@@ -443,8 +443,8 @@ information. Since these tokens are sensitive, **care should be taken to protect
 ### Exit Codes
 
 The Phylum analysis job will return a zero (0) exit code when it completes successfully and a non-zero code otherwise.
-The full and current list of exit codes is [documented here][exit_codes] and [options exist][script_options] to be
-loose with setting them.
+The full and current list of exit codes is [documented here][exit_codes] and "Output Modification"
+[options exist][script_options] to be strict or loose with setting them.
 
 [exit_codes]: https://github.com/phylum-dev/phylum-ci#exit-codes
 

docs/integrations/bitbucket_pipelines.md

@@ -353,8 +353,8 @@ view the [script options output][script_options] for the latest release.
 ### Exit Codes
 
 The Phylum analysis step will return a zero (0) exit code when it completes successfully and a non-zero code otherwise.
-The full and current list of exit codes is [documented here][exit_codes] and [options exist][script_options] to be
-loose with setting them.
+The full and current list of exit codes is [documented here][exit_codes] and "Output Modification"
+[options exist][script_options] to be strict or loose with setting them.
 
 [exit_codes]: https://github.com/phylum-dev/phylum-ci#exit-codes
 

docs/integrations/git_precommit.md

@@ -190,8 +190,8 @@ with `--help` output as specified in the [Usage section of the top-level README.
 ### Exit Codes
 
 The Phylum analysis hook will return a zero (0) exit code when it completes successfully and a non-zero code otherwise.
-The full and current list of exit codes is [documented here][exit_codes] and [options exist][script_options] to be
-loose with setting them.
+The full and current list of exit codes is [documented here][exit_codes] and "Output Modification"
+[options exist][script_options] to be strict or loose with setting them.
 
 [exit_codes]: https://github.com/phylum-dev/phylum-ci#exit-codes
 

docs/integrations/github_actions.md

@@ -150,8 +150,8 @@ See the [Installation][installation] and [Usage][usage] sections of the [README
 ## Exit Codes
 
 The Phylum analysis job/step will return a zero (0) exit code when it completes successfully and a non-zero code
-otherwise. The full and current list of exit codes is [documented here][exit_codes] and [options exist][script_options]
-to be loose with setting them.
+otherwise. The full and current list of exit codes is [documented here][exit_codes] and "Output Modification"
+[options exist][script_options] to be strict or loose with setting them.
 
 [exit_codes]: https://github.com/phylum-dev/phylum-ci#exit-codes
 [script_options]: https://github.com/phylum-dev/phylum-ci/blob/main/docs/script_options.md

docs/integrations/gitlab_ci.md

@@ -342,8 +342,8 @@ view the [script options output][script_options] for the latest release.
 ### Exit Codes
 
 The Phylum analysis job will return a zero (0) exit code when it completes successfully and a non-zero code otherwise.
-The full and current list of exit codes is [documented here][exit_codes] and [options exist][script_options]
-to be loose with setting them.
+The full and current list of exit codes is [documented here][exit_codes] and "Output Modification"
+[options exist][script_options] to be strict or loose with setting them.
 
 [exit_codes]: https://github.com/phylum-dev/phylum-ci#exit-codes
 

docs/integrations/jenkins.md

@@ -341,8 +341,8 @@ release.
 ### Exit Codes
 
 The Phylum analysis stage will return a zero (0) exit code when it completes successfully and a non-zero code otherwise.
-The full and current list of exit codes is [documented here][exit_codes] and [options exist][script_options]
-to be loose with setting them.
+The full and current list of exit codes is [documented here][exit_codes] and "Output Modification"
+[options exist][script_options] to be strict or loose with setting them.
 
 [exit_codes]: https://github.com/phylum-dev/phylum-ci#exit-codes
 

src/phylum/ci/ci_base.py

@@ -130,10 +130,6 @@ def returncode(self, value: ReturnCode) -> None:
         if value == self._returncode:
             return
 
-        # Don't set a failure code when analysis results are unknown
-        if value == ReturnCode.ANALYSIS_INCOMPLETE:
-            return
-
         # Don't set non-analysis custom failure codes when flag to ignore errors specified
         if self.args.ignore_errors and value > ReturnCode.LARGEST_POSSIBLE_ANALYSIS_ERROR:
             msg = f"""
@@ -148,6 +144,14 @@ def returncode(self, value: ReturnCode) -> None:
             LOG.info(cleandoc(msg))
             return
 
+        # Don't set a failure code when analysis results are unknown unless told to
+        if value == ReturnCode.ANALYSIS_INCOMPLETE:
+            if self.args.fail_incomplete:
+                LOG.info("[code]--fail-incomplete[/] specified. Setting return code to: %s", value, extra=MARKUP)
+                self._returncode = value
+                return
+            return
+
         # Don't allow setting a `SUCCESS` value once the return code has already been set to an error value
         if self._returncode == ReturnCode.SUCCESS or value != ReturnCode.SUCCESS:
             LOG.debug("Setting return code to: %s", value)

src/phylum/ci/cli.py

@@ -189,7 +189,14 @@ def get_args(args: Optional[Sequence[str]] = None) -> tuple[argparse.Namespace,
         help="""Specify this flag to disable posting comments/notes on pull/merge requests. This flag is implicitly
             set when audit mode is enabled.""",
     )
-    output_mod_group.add_argument(
+    strict_loose_group = output_mod_group.add_mutually_exclusive_group()
+    strict_loose_group.add_argument(
+        "--fail-incomplete",
+        action="store_true",
+        help="""Specify this flag to set a failure code when some analysis results are unknown. Useful in environments
+            where strictness is desired to prevent adding any new dependency until it passes established policy.""",
+    )
+    strict_loose_group.add_argument(
         "--audit",
         action="store_true",
         help="Specify this flag to enable audit mode: analysis is performed but results do not affect the exit code.",