Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve build scripts and documentation #305

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

Improve build scripts and documentation #305

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
28c4957
Improve LLVM build recipe to benefit from existing local repository
keryell Sep 26, 2023
721da2d
Optimize MLIR-AIE downloading script
keryell Sep 26, 2023
85919fc
Add --mlir-aie-worktree to use existing local MLIR-AIE repository
keryell Sep 26, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 32 additions & 6 deletions docs/building.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,17 +48,43 @@ source utils/setup_python_packages.sh

Next, clone and build LLVM, with the ability to target AArch64 as a cross-compiler, and with MLIR enabled. In addition, we make some common build optimizations to use a linker ('lld' or 'gold') other than 'ld' (which tends to be quite slow on large link jobs) and to link against libLLVM.so and libClang so. You may find that other options are also useful. Note that due to changing MLIR APIs, only a particular revision is expected to work.

To clone llvm and cmakeModules, see utils/clone-llvm.sh for the correct commithash. We point LLVM and subsequent tools to a common installation directory.
To clone `llvm`, run `utils/clone-llvm.sh` (see
`utils/clone-llvm.sh` for the correct `llvm` commit hash):
```
./utils/clone-llvm.sh
```

If you have already an LLVM repository, you can instead of cloning
just make a new worktree from it by using:
````
./utils/clone-llvm.sh --llvm-worktree <path-of-existing-LLVM-repository>
````

TODO: explain the case of cmakeModules.

We point LLVM and subsequent tools to a common installation directory,
assuming above the current `mlir-air` work-tree directory.

```
cd utils
./clone-llvm.sh
./build-llvm-local.sh llvm build ../../install
./utils/build-llvm-local.sh llvm build ../install
```

Next, clone and build MLIR-AIE with paths to llvm, aienginev2, and cmakeModules repositories. Again, we use a common installation directory. Note that in the following commands, we assume that the aienginev2 library is installed in /opt/xaiengine as directed in the `Building on x86 with runtime for PCIe` section. If the aienginev2 library was installed elsewhere, be sure that the 4th argument to build mlir-aie points to that location.
Then clone MLIR-AIE with `utils/clone-mlir-aie.sh` (see
`utils/clone-mlir-aie.sh` for the correct `mlir-aie` commit hash):
```
./utils/clone-mlir-aie.sh
```

If you have already an MLIR-AIE repository, you can instead of cloning
just make a new worktree from it by using:
````
./utils/clone-mlir-aie.sh --mlir-aie-worktree <path-of-local-mlir-aie-repository>
````


Next, build MLIR-AIE with paths to llvm, aienginev2, and cmakeModules repositories. Again, we use a common installation directory. Note that in the following commands, we assume that the aienginev2 library is installed in /opt/xaiengine as directed in the `Building on x86 with runtime for PCIe` section. If the aienginev2 library was installed elsewhere, be sure that the 4th argument to build mlir-aie points to that location.
```
./clone-mlir-aie.sh
./utils/clone-mlir-aie.sh
./build-mlir-aie-local.sh llvm mlir-aie/cmake/modulesXilinx /opt/xaiengine mlir-aie build ../../install
```

Expand Down
42 changes: 34 additions & 8 deletions utils/clone-llvm.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

##===- utils/clone-llvm.sh - Clone LLVM ---------------------*- Script -*-===##
#
# Copyright (C) 2022, Advanced Micro Devices, Inc.
# Copyright (C) 2023, Advanced Micro Devices, Inc.
# SPDX-License-Identifier: MIT

##===----------------------------------------------------------------------===##
Expand All @@ -14,14 +14,40 @@
#
##===----------------------------------------------------------------------===##

export commithash=11c3b979e6512b00a5bd9c3e0d4ed986cf500630
# The LLVM commit the project depends on
commithash=11c3b979e6512b00a5bd9c3e0d4ed986cf500630
# It is not clear why we need a branch
branch=air-2022.12

git clone --depth 1 https://github.com/llvm/llvm-project.git llvm
pushd llvm
git fetch --depth=1 origin $commithash
git checkout $commithash -b $branch
here=$PWD

# Use --llvm-worktree <path-of-local-LLVM-repo> to reuse some existing
# local LLVM git repository
if [ x"$1" == x--llvm-worktree ]; then
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Personally when I have an existing llvm repo I simply don't run this script (actually I never run this script...). I wonder if this just adds complexity for a personal git preference?

git_central_llvm_repo_dir="$2"
(
cd $git_central_llvm_repo_dir
# Remove an existing worktree here just in case we are iterating
# on the script
git worktree remove --force "$here"/llvm
# Use force just in case there are various experimental iterations
# after you have removed the llvm directory
git worktree add --force -B $branch "$here"/llvm $commithash
)
else
# Avoid checking out to spare time since we switch to another branch later
git clone --depth 1 --no-checkout https://github.com/llvm/llvm-project.git llvm
(
cd llvm
# Then fetch the interesting part
git fetch --depth=1 origin $commithash
# Create local $branch to current $commithash
git branch $branch $commithash
# Switch to the branch
git switch $branch
)
fi

# Make mlir_async_runtime library's symbol visible
# so that we can link to this library in channel sim tests
sed -i '/set_property(TARGET mlir_async_runtime PROPERTY CXX_VISIBILITY_PRESET hidden)/d' ./mlir/lib/ExecutionEngine/CMakeLists.txt
popd
sed -i '/set_property(TARGET mlir_async_runtime PROPERTY CXX_VISIBILITY_PRESET hidden)/d' llvm/mlir/lib/ExecutionEngine/CMakeLists.txt
40 changes: 32 additions & 8 deletions utils/clone-mlir-aie.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,18 +7,42 @@

##===----------------------------------------------------------------------===##
#
# This script checks out MLIR-AIE. We use this instead of a git submodule to
# This script checks out MLIR-AIE. We use this instead of a git submodule to
# manage commithash synchronization with LLVM.
#
# This script is called from the github workflows.
#
##===----------------------------------------------------------------------===##

export HASH=47ff7d38a89372c02e22515e61a696f2a3e93013
commithash=47ff7d38a89372c02e22515e61a696f2a3e93013
project_dir=mlir-aie

git clone --depth 1 https://github.com/Xilinx/mlir-aie.git mlir-aie
pushd mlir-aie
git fetch --depth=1 origin $HASH
git checkout $HASH
git submodule update --init
popd
here=$PWD

# Use --mlir-aie-worktree <path-of-local-mlir-aie-repository> to reuse
# some existing local mlir-aie git repository
if [ x"$1" == x--mlir-aie-worktree ]; then
git_central_mlir_aie_repo_dir="$2"
(
cd $git_central_mlir_aie_repo_dir
# Remove an existing worktree here just in case we are iterating
# on the script
git worktree remove --force "$here"/$project_dir
# Use force just in case there are various experimental iterations
# after you have removed the llvm directory
git worktree add --force "$here"/$project_dir $commithash
)
else
# Avoid checking out to spare time since we switch to another branch later
git clone --depth 1 --no-checkout https://github.com/Xilinx/$project_dir.git
(
cd $project_dir
git fetch --depth=1 origin $commithash
git checkout $commithash
git submodule update --depth 1 --recursive --init
)
fi
(
cd $project_dir
git submodule update --depth 1 --recursive --init
)