Merge branch 'just-buildsystem:master' into github-ci

CHANGELOG.md

@@ -13,6 +13,7 @@ A feature release on top of `1.1.0`, backwards compatible.
 - Relative non-upwards symbolic links are now treated as first-class
   objects. This introduces a new artifact type and allows the free use
   of such symbolic links throughout the build process.
+- `just-mr` can now optionally resolve symlinks contained in archives.
 
 ### Other changes
 
@@ -23,7 +24,15 @@ A feature release on top of `1.1.0`, backwards compatible.
   directories of `just-mr`, will eventually get cleaned up by
   garbage collection.
 - `just-mr` now supports unpacking tar archives compressed with
-  bzip2 and xz.
+  bzip2, xz, lzip, and lzma.
+- The option `-P` of `build` and `install-cas` can be used to
+  inspect parts of a tree.
+- `just-mr` now supports unpacking 7zip archives (with default
+  compression) when provided as `"zip"` type repositories.
+- The configuration variable `COMPILER_FAMILY` is replaced by the more
+  flexible `TOOLCHAIN_CONFIG`, an object which may contain the field
+  `FAMILY`. From now on, this object is used to set the compiler family
+  (e.g., for GNU, set `{"TOOLCHAIN_CONFIG":{"FAMILY":"gnu"}}`).
 
 ### Fixes
 
@@ -41,6 +50,39 @@ A feature release on top of `1.1.0`, backwards compatible.
   all entries in a Git repository. Previously it was possible for
   entries to be skipped inadvertently in, e.g., imported archives
   if `gitignore` files were present.
+- Temporary files generated by `just execute` are now created inside
+  the local build root.
+- `just install-cas` now correctly handles `--raw-tree` also for
+  remote-execution endpoints.
+- `just install-cas` now, like `just install`, removes an existing
+  destination file before installing instead of overwriting.
+
+## Release `1.2.0~beta3` (2023-08-22)
+
+Third beta release for the upcoming `1.2.0` release; see release
+notes there.
+
+### Changes since `1.2.0~beta2`
+
+- Update and clean up of bundled dependency descriptions
+- Improvement of documentation
+
+## Release `1.2.0~beta2` (2023-08-18)
+
+Second beta release for the upcoming `1.2.0` release; see release
+notes there.
+
+### Changes since `1.2.0~beta1`
+
+- Clean up of the internal build description of bundled dependencies.
+- Clean up of the internal rules, in particular renaming of
+  implicit dependency targets.
+- Various documentation improvements.
+
+## Release `1.2.0~beta1` (2023-08-16)
+
+First beta release for the upcoming `1.2.0` release; see release
+notes there.
 
 ## Release `1.1.0` (2023-05-19)
 

INSTALL.md

@@ -31,7 +31,7 @@ variables. The full list can be obtained via `just-mr describe`.
 | ARCH | x86, x86_64, arm, arm64 | x86_64 |
 | HOST_ARCH | x86, x86_64, arm, arm64 | *derived from ARCH* |
 | TARGET_ARCH | x86, x86_64, arm, arm64 | *derived from ARCH* |
-| COMPILER_FAMILY | gnu, clang, unknown | unknown |
+| TOOLCHAIN_CONFIG["FAMILY"] | gnu, clang, unknown | unknown |
 | DEBUG | true, false | false |
 | BUILD_STATIC_BINARY | true, false | false |
 
@@ -92,7 +92,7 @@ Additionally, if `SOURCE_DATE_EPOCH` is set in the build environment, it
 is forwarded to the build configuration as well. If, on the other hand,
 `CC` or `CXX` are set in the build configuration, those are also used
 for the initial steps of the bootstrap procedure. Remember that setting
-one of those variables also requires the `COMPILER_FAMILY` to
+one of those variables also requires the `TOOLCHAIN_CONFIG["FAMILY"]` to
 ensure the proper flags are used (if in doubt, set to `"unknown"`).
 
 In any case, the resulting binary is self contained and can be moved

README.md

@@ -25,6 +25,7 @@ taken from user-defined rules described by functional expressions.
   - [Ensuring reproducibility](doc/tutorial/rebuild.md)
   - [Using protobuf](doc/tutorial/proto.md)
   - [How to create a single-node remote execution service](doc/tutorial/just-execute.org)
+  - [Cross compiling and testing cross-compiled targets](doc/tutorial/cross-compiling.md)
 
 ## Documentation
 

TARGETS

@@ -10,14 +10,14 @@
     , "BUILD_STATIC_BINARY"
     , "FINAL_LDFLAGS"
     , "CC"
-    , "COMPILER_FAMILY"
     , "CXX"
     , "DEBUG"
     , "ENV"
     , "HOST_ARCH"
     , "OS"
     , "SOURCE_DATE_EPOCH"
     , "TARGET_ARCH"
+    , "TOOLCHAIN_CONFIG"
     , "VERSION_EXTRA_SUFFIX"
     , "PKG_CONFIG_ARGS"
     ]
@@ -30,9 +30,10 @@
     , "HOST_ARCH":
       ["The architecture on which the build actions are carried out."]
     , "TARGET_ARCH": ["The architecture for which to build the binary."]
-    , "COMPILER_FAMILY":
-      [ "The compiler family to use as (cross) toolchain."
+    , "TOOLCHAIN_CONFIG":
+      [ "The toolchain configuration."
       , ""
+      , "Use field \"FAMILY\" to specify the compiler family."
       , "Supported values are \"unknown\" (the default), \"clang\", and \"gnu\"."
       , "The unknown compiler family tries to not make any assumptions on the"
       , "used C and C++ compilers and uses the generic \"cc\" and \"c++\" as"
@@ -77,7 +78,7 @@
     , "HOST_ARCH"
     , "TARGET_ARCH"
     , "DEBUG"
-    , "COMPILER_FAMILY"
+    , "TOOLCHAIN_CONFIG"
     , "CC"
     , "CXX"
     , "ADD_CXXFLAGS"
@@ -93,8 +94,7 @@
   }
 , "just":
   { "type": "configure"
-  , "arguments_config":
-    ["OS", "ARCH", "HOST_ARCH", "TARGET_ARCH", "COMPILER_FAMILY"]
+  , "arguments_config": ["OS", "ARCH", "HOST_ARCH", "TARGET_ARCH"]
   , "target": "exported-just"
   , "config":
     { "type": "let*"
@@ -113,14 +113,9 @@
           , "default": {"type": "var", "name": "ARCH"}
           }
         ]
-      , [ "COMPILER_FAMILY"
-        , {"type": "var", "name": "COMPILER_FAMILY", "default": "unknown"}
-        ]
       ]
     , "body":
-      { "type": "env"
-      , "vars": ["OS", "ARCH", "HOST_ARCH", "TARGET_ARCH", "COMPILER_FAMILY"]
-      }
+      {"type": "env", "vars": ["OS", "ARCH", "HOST_ARCH", "TARGET_ARCH"]}
     }
   }
 , "installed just-mr":
@@ -134,7 +129,7 @@
     , "HOST_ARCH"
     , "TARGET_ARCH"
     , "DEBUG"
-    , "COMPILER_FAMILY"
+    , "TOOLCHAIN_CONFIG"
     , "CC"
     , "CXX"
     , "ADD_CXXFLAGS"
@@ -149,8 +144,7 @@
   }
 , "just-mr":
   { "type": "configure"
-  , "arguments_config":
-    ["OS", "ARCH", "HOST_ARCH", "TARGET_ARCH", "COMPILER_FAMILY"]
+  , "arguments_config": ["OS", "ARCH", "HOST_ARCH", "TARGET_ARCH"]
   , "target": "exported-just-mr"
   , "config":
     { "type": "let*"
@@ -169,14 +163,9 @@
           , "default": {"type": "var", "name": "ARCH"}
           }
         ]
-      , [ "COMPILER_FAMILY"
-        , {"type": "var", "name": "COMPILER_FAMILY", "default": "clang"}
-        ]
       ]
     , "body":
-      { "type": "env"
-      , "vars": ["OS", "ARCH", "HOST_ARCH", "TARGET_ARCH", "COMPILER_FAMILY"]
-      }
+      {"type": "env", "vars": ["OS", "ARCH", "HOST_ARCH", "TARGET_ARCH"]}
     }
   }
 , "libgit2":

bin/bootstrap.py

@@ -91,11 +91,11 @@
 CXXFLAGS = []
 FINAL_LDFLAGS = ["-Wl,-z,stack-size=8388608"]
 
-if "COMPILER_FAMILY" in CONF:
-    if CONF["COMPILER_FAMILY"] == "gnu":
+if "TOOLCHAIN_CONFIG" in CONF and "FAMILY" in CONF["TOOLCHAIN_CONFIG"]:
+    if CONF["TOOLCHAIN_CONFIG"]["FAMILY"] == "gnu":
         CC="gcc"
         CXX="g++"
-    elif CONF["COMPILER_FAMILY"] == "clang":
+    elif CONF["TOOLCHAIN_CONFIG"]["FAMILY"] == "clang":
         CC="clang"
         CXX="clang++"
 

bin/just-mr.py

@@ -449,9 +449,21 @@ def archive_checkout(desc, repo_type="archive"):
         try_rmtree(target_tmp)
     os.makedirs(target_tmp)
     if repo_type == "zip":
-        run_cmd(["unzip", "-d", ".", cas_path(content_id)], cwd=target_tmp)
+        try:
+            run_cmd(["unzip", "-d", ".", cas_path(content_id)], cwd=target_tmp)
+        except:
+            try:
+                run_cmd(["7z", "x", cas_path(content_id)], cwd=target_tmp)
+            except:
+                print("Failed to extract zip-like archive %s" %
+                      (cas_path(content_id), ))
+                sys.exit(1)
     else:
-        run_cmd(["tar", "xf", cas_path(content_id)], cwd=target_tmp)
+        try:
+            run_cmd(["tar", "xf", cas_path(content_id)], cwd=target_tmp)
+        except:
+            print("Failed to extract tarball %s" % (cas_path(content_id), ))
+            sys.exit(1)
     if ALWAYS_FILE:
         move_to_place(target_tmp, target)
         return ["file", subdir_path(target, desc)]

doc/concepts/built-in-rules.md

@@ -140,6 +140,23 @@ of the `"tree"` target are a singleton map with the key the result of
 evaluating `"name"` (which has to evaluate to a single string) and value
 that tree.
 
+`"symlink"`
+------------
+
+The `"symlink"` rule allows to specify a non-upwards symbolic link with a
+given link target. To be able to accurately report about file names of
+artifacts or runfiles of other targets, they can be specified in the field
+`"deps"` which has to evaluate to a list of targets. The names of the
+artifacts and runfiles of a target specified in `"deps"` can be accessed
+through the functions `"outs"` and `"runfiles"`, respectively, during the
+evaluation of the arguments `"name"` and `"data"` which have to evaluate to
+a single string.
+
+Artifacts and runfiles of a `"symlink"` target are a singleton map with
+key the result of evaluating `"name"` and value a non-upwards symbolic link
+with target path the result of evaluating `"data"` (which must evaluate to
+a non-upwards path). The provides map is empty.
+
 `"configure"`
 -------------
 

doc/concepts/expressions.md

@@ -249,6 +249,13 @@ those) argument(s) to obtain the final result.
    same iteration order as the list for all lists indexable by
    32-bit integers.
 
+- `"set"` The argument has to be a list of strings. The result is
+  a map with the members of the list as keys, and all values being
+  `true`.
+
+- `"reverse"` The argument has to be a list. The result is a new list
+  with the entries in reverse order.
+
  - `"++"` The argument has to be a list of lists. The result is
    the concatenation of those lists.
 

doc/concepts/symlinks.md

@@ -81,7 +81,7 @@ Our treatment of symbolic links
 
 ### "Ignore-special" roots
 
-To allow working with source trees containing symbolic links, all the existing
+For cases where we simply have no need for special entries, all the existing
 roots have "ignore-special" versions thereof. In such a root
 (regardless whether file based, or `git`-tree based), everything
 not a file or a directory is pretended to be absent. For any
@@ -97,7 +97,7 @@ content fixed and hence eligible for target-level caching.
 
 ### Non-upwards relative symlinks as first-class objects
 
-Finally, a restricted form of symlinks, more precisely *relative*
+A restricted form of symlinks, more precisely *relative*
 *non-upwards symbolic links*, exist as first-class objects.
 That is, a new artifact type (besides blobs and trees) for relative
 non-upwards symbolic links has been introduced. Like any other artifact,
@@ -117,3 +117,28 @@ non-upwards) symbolic links are, in many aspects, simple files with elevated
 permissions. As such, they locally use the existing file CAS. Remotely, the
 existing execution protocol already allows the handling of symbolic links via
 corresponding Protobuf messages, therefore no extensions are needed.
+
+Additionally, the built-in rules are extended with a `"symlink"` target,
+allowing the generation of a symlink with given non-upwards target path.
+
+### Import resolved `git`-trees
+
+Finally, to be as flexible as possible in handling external repositories with
+(possibly) upwards symbolic links, we allow filesystem directories and archives
+to be imported also as partially or completely resolved `git`-trees.
+
+In a *partially resolved tree*, all relative upwards symbolic links confined to
+the tree get resolved, i.e., replaced by a copy of the entry they point to, if
+existing, or removed otherwise. This of course leaves relative non-upwards
+symbolic links in the `git`-tree, as they are supported objects.
+
+Alternatively, in a *completely resolved tree*, all relative symbolic links
+confined to the tree (whether upwards or not) get resolved, resulting in a
+`git`-tree free of all symbolic links.
+
+For reasons already described, absolute symbolic are never supported.
+
+As this process acts directly at the repository level, the resulting roots
+remain cacheable and their trees accessible in constant time. Moreover, to
+increase the chances of cache hits in `just-mr`, not only the resulting
+resolved trees are stored, but also the original, unresolved ones.