chore: bump Bazel from 5.2.0 to 6.5.0

[martijneken]

I ran into trouble with Bazel 7 (moved to #403), but this still moves things forward.

[PiotrSikora]

Ugh, Linux/s390x depends on piotrsikora/build-tools, since there is no official Bazel build for s390x... and it doesn't look that I've published the Dockerfile for it.

Maybe IBM / Red Hat folks have it publicly available somewhere?

[martijneken]

there is no official Bazel build for s390x

Is this worth supporting? Should we disable the check for now?

the Dockerfile for it

Maybe this? https://github.com/james-crowley/bazel/blob/s390x_patch/Dockerfile

@PiotrSikora Can you try and republish the above for Bazel 6.5.0?

[martijneken]

Fixing rules_rust diff...

[PiotrSikora]

Is this worth supporting? Should we disable the check for now?

IBM / Red Hat (Istio vendor) relies on Envoy with Wasm building on s390x. That build isn't verified in Envoy's CI, but they definitely support it.

@knm3000 do you guys have a publicaly available Bazel build that we could use?

Maybe this? https://github.com/james-crowley/bazel/blob/s390x_patch/Dockerfile

@PiotrSikora Can you try and republish the above for Bazel 6.5.0?

No, I've crated one from scratch and spent quite some time debugging it. As far as I recall, the more recent version didn't build from scratch on s390x, but if you had a working version of Bazel, then it would work fine (basically, you had to build 3.x or 4.x and then use it to build the more recent versions). Since Bazel 5.2 is published on the linked DockerHub, you should be able to use it to build 6.0 and/or 6.5 from it.

[knm3000]

AFAIK there is no publicly available bazel builds for s390x. But it is possible to build it from scratch without previous version of bazel binary. The point is that we should not use "git clone github.com/bazelbuild/bazel" to get the source code, since some files are missing when doing git clone. Instead we should download bazel-$BAZEL_VERSION-dist.zip to get the full bazel source code:

apt-get update -y && apt-get install -y build-essential openjdk-11-jdk python3 zip unzip
export BAZEL_VERSION=6.5.0
cd ~ && mkdir bazel && cd bazel 
curl -LO https://github.com/bazelbuild/bazel/releases/download/$BAZEL_VERSION/bazel-$BAZEL_VERSION-dist.zip
unzip -q bazel-$BAZEL_VERSION-dist.zip
env EXTRA_BAZEL_ARGS="--tool_java_runtime_version=local_jdk" bash ./compile.sh

[martijneken]

@knm3000 do you have any more pointers? I am able to build the Dockerfile below on linux/amd64 but not linux/s390x. It fails pretty early (apt update). Should I pick a different base image than Ubuntu?

$ sudo docker buildx build --no-cache --platform linux/s390x -t bazel-s390x -f Dockerfile.s390x-bazel .                                                                    
[+] Building 1.7s (9/17)                                                                                                                                                   
 => [internal] load build definition from Dockerfile.s390x-bazel                                                                                                      0.0s 
 => => transferring dockerfile: 44B                                                                                                                                   0.0s 
 => [internal] load .dockerignore                                                                                                                                     0.0s
 => => transferring context: 2B                                                                                                                                       0.0s
 => resolve image config for docker.io/docker/dockerfile:1                                                                                                            0.4s
 => CACHED docker-image://docker.io/docker/dockerfile:1@sha256:fe40cf4e92cd0c467be2cfc30657a680ae2398318afd50b0c80585784c604f28                                       0.0s
 => [internal] load build definition from Dockerfile.s390x-bazel                                                                                                      0.0s
 => [internal] load metadata for docker.io/library/ubuntu:22.04                                                                                                       0.5s
 => [internal] load .dockerignore                                                                                                                                     0.0s
 => CACHED [ 1/10] FROM docker.io/library/ubuntu:22.04@sha256:340d9b015b194dc6e2a13938944e0d016e57b9679963fdeb9ce021daac430221                                        0.0s
 => ERROR [ 2/10] RUN apt update && apt upgrade -y                                                                                                                    0.4s
------                                                                                                                                                                     
 > [ 2/10] RUN apt update && apt upgrade -y:
#0 0.351 exec /bin/sh: exec format error
------
ERROR: failed to solve: executor failed running [/bin/sh -c apt update && apt upgrade -y]: exit code: 1

Dockerfile:

# syntax=docker/dockerfile:1                                                     
                                                                                 
# Build Bazel from source                                                        
                                                                                 
FROM ubuntu:22.04 as build                                                       
RUN apt update && apt upgrade -y                                                 
RUN apt autoremove -y                                                            
                                                                                 
RUN apt install -y software-properties-common                                    
RUN add-apt-repository ppa:openjdk-r/ppa                                         
RUN apt install -y \                                                             
  build-essential \                                                              
  openjdk-11-jdk \                                                               
  python3 \                                                                      
  curl \                                                                         
  zip \                                                                          
  unzip                                                                          
                                                                                 
ARG BAZEL_VERSION                                                                
ENV BAZEL_VERSION=${BAZEL_VERSION:-6.5.0}                                        
ENV EXTRA_BAZEL_ARGS="--tool_java_runtime_version=local_jdk"                     
                                                                                 
RUN cd ~ && mkdir bazel && cd bazel                                              
RUN curl -LO https://github.com/bazelbuild/bazel/releases/download/${BAZEL_VERSION}/bazel-${BAZEL_VERSION}-dist.zip
RUN unzip -q bazel-${BAZEL_VERSION}-dist.zip                                     
RUN bash ./compile.sh

[PiotrSikora]

@martijneken do you have QEMU & binfmt installed?

Try running this:

docker run --rm --privileged tonistiigi/binfmt --install all

and/or this:

docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

[martijneken]

Thanks @PiotrSikora. That seemed to do it. Also found https://docs.docker.com/build/building/multi-platform/#qemu, though that elides the qemu install step for some reason. I'll let you know if/when I get stuck again :-)

[martijneken]

I got Bazel compiled for s390x with QEMU (took > 1 hour). Now I'm struggling with all the other implicit dependencies in our non-hermetic Bazel build process:

• python3-distutils -- can apt install
• clang/llvm

This is threatening to take a long time. I may have to punt on this so we make forward progress with CI / deps updates. WDYT @PiotrSikora -- OK to defer s390x CI to a separate issue?

[martijneken]

The below Docker container can run CppHost null + wasmtime tests on native linux/amd64. Verifying s390x now with QEMU. Then see if CI works.

# syntax=docker/dockerfile:1

# Prep:
#   docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
#
# Build:
#   docker buildx build --platform linux/s390x -t $IMAGE -f Dockerfile.bazel
#
# Test:
#   docker run --rm --volume $(pwd):/mnt --workdir /mnt \
#     --platform linux/s390x $IMAGE \
#     bazel test --verbose_failures --test_output=errors \
#     --define engine=null --config=clang --test_timeout=1800 \
#     -- //test/...

# Update base image
ARG UBUNTU_VERSION=20.04
FROM ubuntu:${UBUNTU_VERSION} as build
RUN apt update && apt upgrade -y
RUN apt autoremove -y

# Install Bazel deps
RUN apt install -y software-properties-common
RUN add-apt-repository ppa:openjdk-r/ppa
RUN apt install -y \
    build-essential \
    openjdk-11-jdk \
    python3 \
    curl \
    zip \
    unzip

# Download Bazel source
ARG BAZEL_VERSION=6.5.0
RUN cd ~ && mkdir bazel && cd bazel
RUN curl -LO https://github.com/bazelbuild/bazel/releases/download/${BAZEL_VERSION}/bazel-${BAZEL_VERSION}-dist.zip
RUN unzip -q bazel-${BAZEL_VERSION}-dist.zip

# Build Bazel
ENV EXTRA_BAZEL_ARGS="--tool_java_runtime_version=local_jdk"
RUN bash ./compile.sh

# Copy output to /usr/bin
RUN cp /output/bazel /usr/bin/bazel

# Install ProxyWasm build deps
RUN apt install -y \
    git \
    python3-distutils \
    clang \
    libstdc++6

[PiotrSikora]

I got Bazel compiled for s390x with QEMU (took > 1 hour). Now I'm struggling with all the other implicit dependencies in our non-hermetic Bazel build process:

• python3-distutils -- can apt install
• clang/llvm

This is threatening to take a long time. I may have to punt on this so we make forward progress with CI / deps updates. WDYT @PiotrSikora -- OK to defer s390x CI to a separate issue?

You can test deps and all building for native your native platform (x86_64 / arm64), which should take only a few minutes.

Build:
docker buildx build --platform linux/s390x -t $IMAGE -f Dockerfile.bazel

You can specify multiple targets in --platform to create multiarch Docker image, e.g.

docker buildx build --platform linux/amd64,linux/arm64,linux/s390x -t $IMAGE -f Dockerfile.bazel

[martijneken]

Entering hour 8 of a local QEMU based Bazel compilation... Not sure how the first time took <2 hours.

[PiotrSikora]

Entering hour 8 of a local QEMU based Bazel compilation... Not sure how the first time took <2 hours.

You can restart it... As far as I recall, the process would simply hang most of the time (and using older version of Bazel to compile the new one instead of building from scratch significantly lowered the chance of that).

[martijneken]

the process would simply hang most of the time

This is... not good. I'll give it a few more tries but I'm on my fourth attempt already. @knm3000

[keithmattix]

Would it be possible to disable s390x CI to get this PR unblocked and then tackle it separately in a follow-up?

[mpwarres]

+1 to dealing with s390x build issues separately, and not holding up this PR on that.

[martijneken]

Made progress but got stuck again. Splitting the s390x work into another issue: #405

Please review as is (s390x failing).

[keithmattix]

Does it make sense to just merge #404 since it contains the commits from this PR?

[PiotrSikora]

Maybe push your image and use it here to verify that s390x is indeed broken (see: #405 (comment))?

[martijneken]

Good insight on #405 (comment). My local testing was not the same as CI (which uses prebuilt testdata files). I have published an s390x Ubuntu 20.04 image with Bazel here: https://github.com/orgs/proxy-wasm/packages/container/package/build-tools

CI is now running to see whether it worked.

[PiotrSikora]

I have published an s390x Ubuntu 20.04 image with Bazel here: https://github.com/orgs/proxy-wasm/packages/container/package/build-tools

It looks that it was published as "internal" image, which requires credentials on the CI. Could you make it public instead?

[PiotrSikora]

The s390x image is missing libssl-dev dependency. I suspect that libz-dev might be also missing.

[martijneken]

Bummer, another error. Will leave the CI changes for now and continue in #405.

[martijneken]

Fine fine, I'll try again to get it working :-D
Docker cache FTW, pushing now.