Merge pull request #1849 from apache/tristan/implement-retry-failed

Implement new --retry-failed behavior

doc/source/using_config.rst

@@ -332,6 +332,14 @@ Attributes
   available on the host and limit this with a hard coded value of ``8``, which was
   found to be an optimial number when building even on hosts with many cores.
 
+* ``retry-failed``
+
+  Try to build elements for which a failed build artifact is found when running
+  :ref:`bst build <invoking_build>`.
+
+  This is useful in the case that a build has failed due to insufficient resources
+  such as memory or disk space.
+
 * ``dependencies``
 
   This instructs what dependencies of the target elements should be built, valid

man/bst-artifact-checkout.1

@@ -1,4 +1,4 @@
-.TH "BST ARTIFACT CHECKOUT" "1" "2020-10-14" "" "bst artifact checkout Manual"
+.TH "BST ARTIFACT CHECKOUT" "1" "2023-07-11" "" "bst artifact checkout Manual"
 .SH NAME
 bst\-artifact\-checkout \- Checkout contents of an artifact
 .SH SYNOPSIS
@@ -29,8 +29,11 @@ Create a tarball from the artifact contents instead of a file tree. If LOCATION
 \fB\-\-compression\fP [gz|xz|bz2]
 The compression option of the tarball created.
 .TP
-\fB\-\-pull\fP
-Pull the artifact if it's missing or incomplete.
-.TP
 \fB\-\-directory\fP DIRECTORY
 The directory to checkout the artifact to
+.TP
+\fB\-\-artifact\-remote\fP REMOTE
+A remote for downloading artifacts
+.TP
+\fB\-\-ignore\-project\-artifact\-remotes\fP
+Ignore remote artifact cache servers recommended by projects

man/bst-artifact-delete.1

@@ -1,4 +1,4 @@
-.TH "BST ARTIFACT DELETE" "1" "2020-10-14" "" "bst artifact delete Manual"
+.TH "BST ARTIFACT DELETE" "1" "2023-07-11" "" "bst artifact delete Manual"
 .SH NAME
 bst\-artifact\-delete \- Remove artifacts from the local cache
 .SH SYNOPSIS

man/bst-artifact-list-contents.1

@@ -1,4 +1,4 @@
-.TH "BST ARTIFACT LIST-CONTENTS" "1" "2020-10-14" "" "bst artifact list-contents Manual"
+.TH "BST ARTIFACT LIST-CONTENTS" "1" "2023-07-11" "" "bst artifact list-contents Manual"
 .SH NAME
 bst\-artifact\-list-contents \- List the contents of an artifact
 .SH SYNOPSIS

man/bst-artifact-log.1

@@ -1,4 +1,4 @@
-.TH "BST ARTIFACT LOG" "1" "2020-10-14" "" "bst artifact log Manual"
+.TH "BST ARTIFACT LOG" "1" "2023-07-11" "" "bst artifact log Manual"
 .SH NAME
 bst\-artifact\-log \- Show logs of artifacts
 .SH SYNOPSIS

man/bst-artifact-pull.1

@@ -1,4 +1,4 @@
-.TH "BST ARTIFACT PULL" "1" "2020-10-14" "" "bst artifact pull Manual"
+.TH "BST ARTIFACT PULL" "1" "2023-07-11" "" "bst artifact pull Manual"
 .SH NAME
 bst\-artifact\-pull \- Pull a built artifact
 .SH SYNOPSIS
@@ -15,8 +15,8 @@ When this command is executed from a workspace directory, the default
 is to pull the workspace element.
 .PP
 By default the artifact will be pulled one of the configured caches
-if possible, following the usual priority order. If the `--remote` flag
-is given, only the specified cache will be queried.
+if possible, following the usual priority order. If the `--artifact-remote`
+flag is given, only the specified cache will be queried.
 .PP
 Specify `--deps` to control which artifacts to pull:
 .PP
@@ -30,5 +30,8 @@ Specify `--deps` to control which artifacts to pull:
 \fB\-d,\fP \-\-deps [build|none|run|all]
 The dependency artifacts to pull  [default: none]
 .TP
-\fB\-r,\fP \-\-remote TEXT
-The URL of the remote cache (defaults to the first configured cache)
+\fB\-\-artifact\-remote\fP REMOTE
+A remote for downloading artifacts
+.TP
+\fB\-\-ignore\-project\-artifact\-remotes\fP
+Ignore remote artifact cache servers recommended by projects

man/bst-artifact-push.1

@@ -1,11 +1,11 @@
-.TH "BST ARTIFACT PUSH" "1" "2020-10-14" "" "bst artifact push Manual"
+.TH "BST ARTIFACT PUSH" "1" "2023-07-11" "" "bst artifact push Manual"
 .SH NAME
 bst\-artifact\-push \- Push a built artifact
 .SH SYNOPSIS
 .B bst artifact push
 [OPTIONS] [ARTIFACTS]...
 .SH DESCRIPTION
-Push a built artifact to a remote artifact cache.
+Push built artifacts to a remote artifact cache, possibly pulling them first.
 .PP
 Specifying no elements will result in pushing the default targets
 of the project. If no default targets are configured, all project
@@ -14,9 +14,6 @@ elements will be pushed.
 When this command is executed from a workspace directory, the default
 is to push the workspace element.
 .PP
-The default destination is the highest priority configured cache. You can
-override this by passing a different cache URL with the `--remote` flag.
-.PP
 If bst has been configured to include build trees on artifact pulls,
 an attempt will be made to pull any required build trees to avoid the
 skipping of partial artifacts being pushed.
@@ -33,5 +30,8 @@ Specify `--deps` to control which artifacts to push:
 \fB\-d,\fP \-\-deps [build|none|run|all]
 The dependencies to push  [default: none]
 .TP
-\fB\-r,\fP \-\-remote TEXT
-The URL of the remote cache (defaults to the first configured cache)
+\fB\-\-artifact\-remote\fP REMOTE
+A remote for uploading artifacts
+.TP
+\fB\-\-ignore\-project\-artifact\-remotes\fP
+Ignore remote artifact cache servers recommended by projects

man/bst-artifact-show.1

@@ -1,4 +1,4 @@
-.TH "BST ARTIFACT SHOW" "1" "2020-10-14" "" "bst artifact show Manual"
+.TH "BST ARTIFACT SHOW" "1" "2023-07-11" "" "bst artifact show Manual"
 .SH NAME
 bst\-artifact\-show \- Show the cached state of artifacts
 .SH SYNOPSIS

man/bst-artifact.1

@@ -1,4 +1,4 @@
-.TH "BST ARTIFACT" "1" "2020-10-14" "" "bst artifact Manual"
+.TH "BST ARTIFACT" "1" "2023-07-11" "" "bst artifact Manual"
 .SH NAME
 bst\-artifact \- Manipulate cached artifacts.
 .SH SYNOPSIS
@@ -12,20 +12,28 @@ can be specified in two ways:
 .PP
 
 - artifact refs: triples of the form <project name>/<element name>/<cache key>
-- element paths
+- element names
 .PP
-When elements are given, the artifact corresponding to the element is used.
+When elements are given, the artifact is looked up by observing the element
+and it's current cache key.
 .PP
 The commands also support shell-style wildcard expansion: `?` matches a
-single character, and `*` matches zero or more. The patterns are matched
-against artifact refs by default. If the pattern ends with `.bst` then
-it matches element paths instead. Some example arguments are:
+single character, `*` matches zero or more characters but does not match the `/`
+path separator, and `**` matches zero or more characters including `/` path separators.
+.PP
+If the wildcard expression ends with `.bst`, then it will be used to search
+element names found in the project, otherwise, it will be used to search artifacts
+that are present in the local artifact cache.
+.PP
+Some example arguments are:
 .PP
 
 - `myproject/hello/8276376b077eda104c812e6ec2f488c7c9eea211ce572c83d734c10bf241209f`
 - `myproject/he*/827637*`
-- `*.bst` (all elements)
-- `myproject/*` (all artifacts from myproject)
+- `core/*.bst` (all elements in the core directory)
+- `**.bst` (all elements)
+- `myproject/**` (all artifacts from myproject)
+- `myproject/myelement/*` (all cached artifacts for a specific element)
 .SH COMMANDS
 .PP
 \fBshow\fP

man/bst-build.1

@@ -1,4 +1,4 @@
-.TH "BST BUILD" "1" "2020-10-14" "" "bst build Manual"
+.TH "BST BUILD" "1" "2023-07-11" "" "bst build Manual"
 .SH NAME
 bst\-build \- Build elements in a pipeline
 .SH SYNOPSIS
@@ -14,16 +14,31 @@ elements will be built.
 When this command is executed from a workspace directory, the default
 is to build the workspace element.
 .PP
-Specify `--deps` to control which dependencies to build:
+Specify `--deps` to control which dependencies must be built:
 .PP
 
-    plan:  Only dependencies required for the build plan
+    none:  No dependencies, just the element itself
     build: Build time dependencies, excluding the element itself
     all:   All dependencies
+.PP
+Dependencies that are consequently required to build the requested
+elements will be built on demand.
 .SH OPTIONS
 .TP
-\fB\-d,\fP \-\-deps [build|plan|all]
+\fB\-d,\fP \-\-deps [none|build|all]
 The dependencies to build
 .TP
-\fB\-r,\fP \-\-remote TEXT
-The URL of the remote cache (defaults to the first configured cache)
+\fB\-\-artifact\-remote\fP REMOTE
+A remote for uploading and downloading artifacts
+.TP
+\fB\-\-source\-remote\fP REMOTE
+A remote for uploading and downloading cached sources
+.TP
+\fB\-\-ignore\-project\-artifact\-remotes\fP
+Ignore remote artifact cache servers recommended by projects
+.TP
+\fB\-\-ignore\-project\-source\-remotes\fP
+Ignore remote source cache servers recommended by projects
+.TP
+\fB\-r,\fP \-\-retry\-failed
+Try to build elements for which a failed build artifact is found

man/bst-help.1

@@ -1,9 +1,8 @@
-.TH "BST HELP" "1" "2020-10-14" "" "bst help Manual"
+.TH "BST HELP" "1" "2023-07-11" "" "bst help Manual"
 .SH NAME
 bst\-help \- Print usage information
 .SH SYNOPSIS
 .B bst help
 [OPTIONS] COMMAND
 .SH DESCRIPTION
 Print usage information about a given command
-

man/bst-init.1

@@ -1,4 +1,4 @@
-.TH "BST INIT" "1" "2020-10-14" "" "bst init Manual"
+.TH "BST INIT" "1" "2023-07-11" "" "bst init Manual"
 .SH NAME
 bst\-init \- Initialize a new BuildStream project
 .SH SYNOPSIS

man/bst-shell.1

@@ -1,9 +1,9 @@
-.TH "BST SHELL" "1" "2020-10-14" "" "bst shell Manual"
+.TH "BST SHELL" "1" "2023-07-11" "" "bst shell Manual"
 .SH NAME
 bst\-shell \- Shell into an element's sandbox environment
 .SH SYNOPSIS
 .B bst shell
-[OPTIONS] [ELEMENT] [COMMAND]...
+[OPTIONS] [TARGET] [COMMAND]...
 .SH DESCRIPTION
 Run a command in the target element's sandbox environment
 .PP
@@ -37,7 +37,16 @@ Mount a file or directory into the sandbox
 Create an isolated build sandbox
 .TP
 \fB\-t,\fP \-\-use\-buildtree
-Stage a buildtree. Will fail if a buildtree is not available. --pull and pull-buildtrees configuration is needed if trying to query for remotely cached buildtrees.
+Stage a buildtree. Will fail if a buildtree is not available. pull-buildtrees configuration is needed if the buildtree is not available locally.
 .TP
-\fB\-\-pull\fP
-Attempt to pull missing or incomplete artifacts
+\fB\-\-artifact\-remote\fP REMOTE
+A remote for uploading and downloading artifacts
+.TP
+\fB\-\-source\-remote\fP REMOTE
+A remote for uploading and downloading cached sources
+.TP
+\fB\-\-ignore\-project\-artifact\-remotes\fP
+Ignore remote artifact cache servers recommended by projects
+.TP
+\fB\-\-ignore\-project\-source\-remotes\fP
+Ignore remote source cache servers recommended by projects

man/bst-show.1

@@ -1,4 +1,4 @@
-.TH "BST SHOW" "1" "2020-10-14" "" "bst show Manual"
+.TH "BST SHOW" "1" "2023-07-11" "" "bst show Manual"
 .SH NAME
 bst\-show \- Show elements in the pipeline
 .SH SYNOPSIS
@@ -21,7 +21,6 @@ Specify ``--deps`` to control which elements to show:
 .PP
 
     none:  No dependencies, just the element itself
-    plan:  Dependencies required for a build plan
     run:   Runtime dependencies, including the element itself
     build: Build time dependencies, excluding the element itself
     all:   All dependencies
@@ -64,7 +63,7 @@ If you want to use a newline in a format string in bash, use the '$' modifier:
 \fB\-\-except\fP PATH
 Except certain dependencies
 .TP
-\fB\-d,\fP \-\-deps [none|plan|run|build|all]
+\fB\-d,\fP \-\-deps [none|run|build|all]
 The dependencies to show  [default: all]
 .TP
 \fB\-\-order\fP [stage|alpha]

man/bst-source-checkout.1

@@ -1,4 +1,4 @@
-.TH "BST SOURCE CHECKOUT" "1" "2020-10-14" "" "bst source checkout Manual"
+.TH "BST SOURCE CHECKOUT" "1" "2023-07-11" "" "bst source checkout Manual"
 .SH NAME
 bst\-source\-checkout \- Checkout sources of an element
 .SH SYNOPSIS
@@ -31,3 +31,9 @@ The compression option of the tarball created.
 .TP
 \fB\-\-directory\fP DIRECTORY
 The directory to checkout the sources to
+.TP
+\fB\-\-source\-remote\fP REMOTE
+A remote for downloading cached sources
+.TP
+\fB\-\-ignore\-project\-source\-remotes\fP
+Ignore remote source cache servers recommended by projects

man/bst-source-fetch.1

@@ -1,4 +1,4 @@
-.TH "BST SOURCE FETCH" "1" "2020-10-14" "" "bst source fetch Manual"
+.TH "BST SOURCE FETCH" "1" "2023-07-11" "" "bst source fetch Manual"
 .SH NAME
 bst\-source\-fetch \- Fetch sources in a pipeline
 .SH SYNOPSIS
@@ -14,16 +14,13 @@ elements will be fetched.
 When this command is executed from a workspace directory, the default
 is to fetch the workspace element.
 .PP
-By default this will only try to fetch sources which are
-required for the build plan of the specified target element,
-omitting sources for any elements which are already built
-and available in the artifact cache.
+By default this will only try to fetch sources for the specified
+elements.
 .PP
 Specify `--deps` to control which sources to fetch:
 .PP
 
     none:  No dependencies, just the element itself
-    plan:  Only dependencies required for the build plan
     run:   Runtime dependencies, including the element itself
     build: Build time dependencies, excluding the element itself
     all:   All dependencies
@@ -32,8 +29,11 @@ Specify `--deps` to control which sources to fetch:
 \fB\-\-except\fP PATH
 Except certain dependencies from fetching
 .TP
-\fB\-d,\fP \-\-deps [plan|none|build|run|all]
-The dependencies to fetch  [default: plan]
+\fB\-d,\fP \-\-deps [none|build|run|all]
+The dependencies to fetch  [default: none]
 .TP
-\fB\-r,\fP \-\-remote TEXT
-The URL of the remote source cache (defaults to the first configured cache)
+\fB\-\-source\-remote\fP REMOTE
+A remote for downloading sources
+.TP
+\fB\-\-ignore\-project\-source\-remotes\fP
+Ignore remote source cache servers recommended by projects

man/bst-source-push.1

@@ -1,4 +1,4 @@
-.TH "BST SOURCE PUSH" "1" "2020-10-14" "" "bst source push Manual"
+.TH "BST SOURCE PUSH" "1" "2023-07-11" "" "bst source push Manual"
 .SH NAME
 bst\-source\-push \- Push sources in a pipeline
 .SH SYNOPSIS
@@ -14,18 +14,26 @@ all project elements will be pushed.
 When this command is executed from a workspace directory, the default
 is to push the sources of the workspace element.
 .PP
+By default this will only try to push sources for the specified
+elements.
+.PP
 Specify `--deps` to control which sources to fetch:
 .PP
 
     none:  No dependencies, just the element itself
-    plan:  Only dependencies required for the build plan
     run:   Runtime dependencies, including the element itself
     build: Build time dependencies, excluding the element itself
     all:   All dependencies
 .SH OPTIONS
 .TP
-\fB\-d,\fP \-\-deps [none|plan|build|run|all]
+\fB\-\-except\fP PATH
+Except certain dependencies from pushing
+.TP
+\fB\-d,\fP \-\-deps [none|build|run|all]
 The dependencies to push  [default: none]
 .TP
-\fB\-r,\fP \-\-remote TEXT
-The URL of the remote source cache (defaults to the first configured cache)
+\fB\-\-source\-remote\fP REMOTE
+A remote for uploading sources
+.TP
+\fB\-\-ignore\-project\-source\-remotes\fP
+Ignore remote source cache servers recommended by projects

man/bst-source-track.1

@@ -1,4 +1,4 @@
-.TH "BST SOURCE TRACK" "1" "2020-10-14" "" "bst source track Manual"
+.TH "BST SOURCE TRACK" "1" "2023-07-11" "" "bst source track Manual"
 .SH NAME
 bst\-source\-track \- Track new source references
 .SH SYNOPSIS

man/bst-source.1

@@ -1,4 +1,4 @@
-.TH "BST SOURCE" "1" "2020-10-14" "" "bst source Manual"
+.TH "BST SOURCE" "1" "2023-07-11" "" "bst source Manual"
 .SH NAME
 bst\-source \- Manipulate sources for an element
 .SH SYNOPSIS

man/bst-workspace-close.1

@@ -1,4 +1,4 @@
-.TH "BST WORKSPACE CLOSE" "1" "2020-10-14" "" "bst workspace close Manual"
+.TH "BST WORKSPACE CLOSE" "1" "2023-07-11" "" "bst workspace close Manual"
 .SH NAME
 bst\-workspace\-close \- Close workspaces
 .SH SYNOPSIS

man/bst-workspace-list.1

@@ -1,4 +1,4 @@
-.TH "BST WORKSPACE LIST" "1" "2020-10-14" "" "bst workspace list Manual"
+.TH "BST WORKSPACE LIST" "1" "2023-07-11" "" "bst workspace list Manual"
 .SH NAME
 bst\-workspace\-list \- List open workspaces
 .SH SYNOPSIS