Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Better ppc64le support and minor improvements #184

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open

Better ppc64le support and minor improvements #184

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
31e1b22
Better support for ppc64le
mgiessing Aug 25, 2022
1acc43e
Add logic to build gn if no binary
mgiessing Aug 25, 2022
f15ef90
Misspelled arch
mgiessing Aug 25, 2022
e0a2c74
Fix arch issue & add build target
mgiessing Aug 25, 2022
9d40695
Fix arch
mgiessing Aug 25, 2022
199a19d
Fix build issue
mgiessing Aug 25, 2022
3a867ae
Simplified install logic
mgiessing Sep 8, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 19 additions & 6 deletions build_container/build_container_common.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,21 +11,34 @@ function download_and_check () {

function install_gn(){
# Install gn tools which will be used for building wee8
# amd64 & arm64 install binary, else compile from source
case "$(uname -m)" in
"x86_64")
GN_ARCH=amd64
wget -O gntool.zip "https://chrome-infra-packages.appspot.com/dl/gn/gn/linux-${GN_ARCH}/+/latest"
unzip gntool.zip -d gntool
cp gntool/gn /usr/local/bin/gn
chmod +x /usr/local/bin/gn
rm -rf gntool*
;;

"aarch64")
GN_ARCH=arm64
wget -O gntool.zip "https://chrome-infra-packages.appspot.com/dl/gn/gn/linux-${GN_ARCH}/+/latest"
unzip gntool.zip -d gntool
cp gntool/gn /usr/local/bin/gn
chmod +x /usr/local/bin/gn
rm -rf gntool*
;;

"ppc64le")
git clone https://gn.googlesource.com/gn && cd gn
CC=/opt/llvm/bin/clang CXX=/opt/llvm/bin/clang++ python3 build/gen.py && \
ninja -C out && \
ln -sf out/gn /usr/local/bin/gn
;;
esac

wget -O gntool.zip "https://chrome-infra-packages.appspot.com/dl/gn/gn/linux-${GN_ARCH}/+/latest"
unzip gntool.zip -d gntool
cp gntool/gn /usr/local/bin/gn
chmod +x /usr/local/bin/gn
rm -rf gntool*
esac
}

if [[ "$(uname -m)" == "x86_64" ]]; then
Expand Down
27 changes: 12 additions & 15 deletions build_container/build_container_ubuntu.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,17 +20,7 @@ curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | \

# docker-ce-cli
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
case $ARCH in
'ppc64le' )
add-apt-repository "deb [arch=ppc64le] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
;;
'x86_64' )
add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
;;
'aarch64' )
add-apt-repository "deb [arch=arm64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
;;
esac
add-apt-repository "deb [arch=$(dpkg --print-architecture)] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

# CMake
curl -fsSL https://apt.kitware.com/keys/kitware-archive-latest.asc | apt-key add -
Expand All @@ -53,7 +43,6 @@ PACKAGES=(
gdb
git
gnupg2
google-cloud-sdk
graphviz
jq
libffi-dev
Expand Down Expand Up @@ -86,24 +75,31 @@ case $ARCH in
'ppc64le' )
LLVM_DISTRO=powerpc64le-linux-ubuntu-18.04
LLVM_SHA256SUM=2d504c4920885c86b306358846178bc2232dfac83b47c3b1d05861a8162980e6
apt-get install -y --no-install-recommends \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this can be outside of the case statement once google-cloud-sdk handling is moved up.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As described above I think we can remove all apt-get ... in all the LLVM part because that is then handled via PACKAGES already.

Would look like this

# Set LLVM version for each cpu architecture.
LLVM_VERSION=14.0.0
case $ARCH in
    'ppc64le' )
        LLVM_DISTRO=powerpc64le-linux-ubuntu-18.04
        LLVM_SHA256SUM=2d504c4920885c86b306358846178bc2232dfac83b47c3b1d05861a8162980e6
        ;;
    'x86_64' )
        LLVM_DISTRO=x86_64-linux-gnu-ubuntu-18.04
        LLVM_SHA256SUM=61582215dafafb7b576ea30cc136be92c877ba1f1c31ddbbd372d6d65622fef5
        ;;
    'aarch64' )
        LLVM_DISTRO=aarch64-linux-gnu
        LLVM_SHA256SUM=1792badcd44066c79148ffeb1746058422cc9d838462be07e3cb19a4b724a1ee
        ;;
esac

Would that be fine with you?

libtinfo5 # LLVM dependencies on Ubuntu 20.04
;;
'x86_64' )
LLVM_DISTRO=x86_64-linux-gnu-ubuntu-18.04
LLVM_SHA256SUM=61582215dafafb7b576ea30cc136be92c877ba1f1c31ddbbd372d6d65622fef5
apt-get install -y --no-install-recommends \
google-cloud-sdk \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we need google-cloud-sdk here? you can add this only if x86_64 and aarch64 to PACKAGES in above.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure whether there is a more elegant solution but I'll append the google-cloud-sdk if ARCH is x86 or arm ok?
Will look like this:

PACKAGES=(
    [...]
)

if [[ "${ARCH}" == "x86_64" || "${ARCH}" == "aarch64" ]]; then
  PACKAGES+=("google-cloud-sdk")
fi

I'll also add libtinfo5 in PACKAGES, then we can remove the complete apt-get install ... in the LLVM part.

Would that be fine with you?

libtinfo5 # LLVM dependencies on Ubuntu 20.04
;;
'aarch64' )
LLVM_DISTRO=aarch64-linux-gnu
LLVM_SHA256SUM=1792badcd44066c79148ffeb1746058422cc9d838462be07e3cb19a4b724a1ee
apt-get install -y --no-install-recommends libtinfo5 # LLVM dependencies on Ubuntu 20.04
apt-get install -y --no-install-recommends \
google-cloud-sdk \
libtinfo5 # LLVM dependencies on Ubuntu 20.04
;;
esac

# Bazel and related dependencies.
case $ARCH in
'ppc64le' )
BAZEL_LATEST="$(curl https://oplab9.parqtec.unicamp.br/pub/ppc64el/bazel/ubuntu_16.04/latest/ 2>&1 \
BAZEL_LATEST="$(curl https://oplab9.parqtec.unicamp.br/pub/ppc64el/bazel/ubuntu_$(lsb_release -r | awk '{print $2}')/latest/ 2>&1 \
| sed -n 's/.*href="\([^"]*\).*/\1/p' | grep '^bazel' | head -n 1)"
curl -fSL https://oplab9.parqtec.unicamp.br/pub/ppc64el/bazel/ubuntu_16.04/latest/${BAZEL_LATEST} \
curl -fSL https://oplab9.parqtec.unicamp.br/pub/ppc64el/bazel/ubuntu_$(lsb_release -r | awk '{print $2}')/latest/${BAZEL_LATEST} \
-o /usr/local/bin/bazel
chmod +x /usr/local/bin/bazel
;;
Expand Down Expand Up @@ -132,6 +128,7 @@ pip3 install -U pyyaml virtualenv
source ./build_container_common.sh

# Soft link the gcc compiler (required by python env)
ARCH=${ARCH/ppc64le/powerpc64le} # For ppc64le the ARCH variable must be renamed accordingly
update-alternatives --install "/usr/bin/${ARCH}-linux-gnu-gcc" "${ARCH}-linux-gnu-gcc" "/usr/bin/${ARCH}-linux-gnu-gcc-9" 1

apt-get clean
2 changes: 1 addition & 1 deletion build_container/docker_build_linux.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ config_env() {

if [[ -z "${BUILD_TOOLS_PLATFORMS}" ]]; then
if [[ "${OS_DISTRO}" == "ubuntu" ]]; then
export BUILD_TOOLS_PLATFORMS=linux/arm64,linux/amd64
export BUILD_TOOLS_PLATFORMS=linux/arm64,linux/amd64,linux/ppc64le
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's hold off this for now as it drastically increases image build time in CI with emulation + building tools.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure - I can remove this with the next commit I'm just wondering if you don't want to have it built at all or just while this PR is not finally approved.

Thanks!

else
export BUILD_TOOLS_PLATFORMS=linux/amd64
fi
Expand Down