Fix CI for s390x

[martijneken]

See #402 for a log of getting s390x Bazel built.

I managed a build of Bazel despite this being quite flaky (Dockerfile below):

$ sudo docker buildx build --no-cache --platform linux/s390x -t build-tools:ubuntu-20.04-bazel-6.5.0 -f Dockerfile.bazel .
[+] Building 4802.4s (20/20) FINISHED                                                      

But it doesn't work locally (emscripten/emcc.py missing?):

$ sudo docker run --rm --volume $(pwd):/mnt --workdir /mnt  --platform linux/s390x build-tools:ubuntu-20.04-bazel-6.5.0 bazel test --verbose_failures --test_output=errors --define engine=wasmtime --config=clang --test_timeout=1800 -- //test/...                                
...
ERROR: /mnt/test/test_data/BUILD:88:21: Compiling test/test_data/http_logging.cc failed: (Exit 2): emcc.sh failed: error executing command (from target //test/test_data:proxy_wasm_http
_logging)                                                                                   
  (cd /root/.cache/bazel/_bazel_root/bede119f64a38032435c6e05a554a267/sandbox/processwrapper-sandbox/10/execroot/proxy_wasm_cpp_host && \                                               
  exec env - \                                                                                                                                                                              BAZEL_COMPILER=clang \                                                                  
    CC=clang \                                
    CXX=clang++ \                             
    EMCC_WASM_BACKEND=1 \
    EM_BIN_PATH=external/emsdk \
    EM_CONFIG_PATH=external/emsdk/emscripten_toolchain/emscripten_config \
    PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
    PWD=/proc/self/cwd \
  external/emsdk/emscripten_toolchain/emcc.sh '--sysroot=external/emsdk/emscripten/cache/sysroot' -fdiagnostics-color -fno-exceptions -fno-strict-aliasing -funsigned-char -no-canonical-prefixes '-std=gnu++17' -nostdinc -nostdinc++ -fomit-frame-pointer -O0 -Wall '-DBAZEL_CURRENT_REPOSITORY=""' -iquote . -iquote bazel-out/wasm-fastbuild-ST-ca831c069035/bin -iquote external/proxy_wasm_cpp_sdk -iquote bazel-out/wasm-fastbuild-ST-ca831c069035/bin/external/proxy_wasm_cpp_sdk -iquote external/emsdk -iquote bazel-out/wasm-fastbuild-ST-ca831c069035/bin/external/emsdk -O3 -iwithsysroot/include/c++/v1 -iwithsysroot/include/compat -iwithsysroot/include -isystem external/emsdk/lib/clang/15.0.0/include -MD -MF bazel-out/wasm-fastbuild-ST-ca831c069035/bin/test/test_data/_objs/proxy_wasm_http_logging/http_logging.d -c test/test_data/http_logging.cc -o bazel-out/wasm-fastbuild-ST-ca831c069035/bin/test/test_data/_objs/proxy_wasm_http_logging/http_logging.o -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -Werror)
# Configuration: c8778a672e7196fd832fc49695b58b09d50dfe3e8617c6ffd959f0b0e0dfe535
# Execution platform: @local_config_platform//:host

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
/usr/bin/python3: can't open file '/root/.cache/bazel/_bazel_root/bede119f64a38032435c6e05a554a267/sandbox/processwrapper-sandbox/10/execroot/proxy_wasm_cpp_host/external/emsdk/emscripten/emcc.py': [Errno 2] No such file or directory

The Dockerfile below works natively on linux/amd64:

# syntax=docker/dockerfile:1

# Prep:
#   docker run --rm --privileged tonistiigi/binfmt --install all
#   docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
#   cat /proc/sys/fs/binfmt_misc/qemu-*
#
# Build:
#   docker buildx build --platform linux/s390x -t $IMAGE -f Dockerfile.bazel
#
# Push:
#   docker image tag $IMAGE $REPO/$IMAGE:latest
#   docker push $IMAGE
#
# Test:
#   docker run --rm --volume $(pwd):/mnt --workdir /mnt \
#     --platform linux/s390x $IMAGE \
#     bazel test --verbose_failures --test_output=errors \
#     --define engine=null --config=clang --test_timeout=1800 \
#     -- //test/...

# Update base image
ARG UBUNTU_VERSION=20.04
FROM ubuntu:${UBUNTU_VERSION} as build
RUN apt update && apt upgrade -y
RUN apt autoremove -y

# Install Bazel deps
RUN apt install -y software-properties-common
RUN add-apt-repository ppa:openjdk-r/ppa
RUN apt install -y \
    build-essential \
    openjdk-11-jdk \
    python3 \
    curl \
    zip \
    unzip

# Download Bazel source
ARG BAZEL_VERSION=6.5.0
RUN cd ~ && mkdir bazel && cd bazel
RUN curl -LO https://github.com/bazelbuild/bazel/releases/download/${BAZEL_VERSION}/bazel-${BAZEL_VERSION}-dist.zip
RUN unzip -q bazel-${BAZEL_VERSION}-dist.zip

# Build Bazel
# NOTE: This step is flaky and frequently hangs for multiarch / buildx.
# If it takes more than 2 hours, restart the Docker build and try again.
ENV EXTRA_BAZEL_ARGS="--tool_java_runtime_version=local_jdk"
RUN bash ./compile.sh

# Copy output to /usr/bin
RUN cp /output/bazel /usr/bin/bazel

# Install ProxyWasm build deps
RUN apt install -y \
    git \
    python3-distutils \
    clang \
    libstdc++6

[knm3000]

Actually emscripten/emsdk is not supported on s390x. If we need to compile C++ WASM, then it is not possible on s390x. But Rust WASM can be compiled without emscripten. I believe we need emscripten to compile C++ WASM modules in test/test_data. Is it possible to exclude C++ WASM from s390x tests and compile Rust WASM only?

[PiotrSikora]

Note that on the CI we build test data (both C++ and Rust) once in a separate step, so those files are not compiled on s390x.

Maybe push this image to DockerHub or GitHub and update the reference in #402?

[martijneken]

Progress. Now we have an openssl error:

https://github.com/proxy-wasm/proxy-wasm-cpp-host/actions/runs/10324222910/job/28583305756?pr=402

[138 / 270] Compiling src/context.cc; 6s processwrapper-sandbox ... (4 actions running)
ERROR: /home/runner/work/proxy-wasm-cpp-host/proxy-wasm-cpp-host/BUILD:64:11: Compiling src/context.cc failed: (Exit 1): clang failed: error executing command (from target //:base_lib) 
  (cd /home/runner/.cache/bazel/_bazel_runner/6e9090fa382dda61db70769c6a68a5bd/sandbox/processwrapper-sandbox/42/execroot/proxy_wasm_cpp_host && \
  exec env - \
    BAZEL_COMPILER=clang \
    CC=clang \
    CXX=clang++ \
    PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
    *** \
  /usr/lib/llvm-10/bin/clang -U_FORTIFY_SOURCE -fstack-protector -Wall -Wthread-safety -Wself-assign -fcolor-diagnostics -fno-omit-frame-pointer '-std=c++14' -MD -MF bazel-out/s390x-fastbuild-ST-5c2a6e57c052/bin/_objs/base_lib/context.pic.d '-frandom-seed=bazel-out/s390x-fastbuild-ST-5c2a6e57c052/bin/_objs/base_lib/context.pic.o' -fPIC '-DBAZEL_CURRENT_REPOSITORY=""' -iquote . -iquote bazel-out/s390x-fastbuild-ST-5c2a6e57c052/bin -iquote external/proxy_wasm_cpp_sdk -iquote bazel-out/s390x-fastbuild-ST-5c2a6e57c052/bin/external/proxy_wasm_cpp_sdk -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION '-std=c++17' -no-canonical-prefixes -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c src/context.cc -o bazel-out/s390x-fastbuild-ST-5c2a6e57c052/bin/_objs/base_lib/context.pic.o)
# Configuration: 625f8fbb3895b70a5d54a3af298554c84e6fcd48ed087b00fbf5d7329c1a8bca
# Execution platform: @local_config_platform//:host
Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
In file included from src/context.cc:25:
./src/hash.h:20:10: fatal error: 'openssl/sha.h' file not found
#include <openssl/sha.h>
         ^~~~~~~~~~~~~~~
1 error generated.

[martijneken]

Next error: https://github.com/proxy-wasm/proxy-wasm-cpp-host/actions/runs/10324463000/job/28584062353?pr=402

cargo:warning=src/trampolines/s390x.S:4:9: error: unknown directive
cargo:warning=        .machine z13
cargo:warning=        ^
--stderr:
error occurred: Command "/usr/lib/llvm-10/bin/clang" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-gdwarf-4" "-fno-omit-frame-pointer" "--target=s390x-unknown-linux-gnu" "-U_FORTIFY_SOURCE" "-fstack-protector" "-Wall" "-Wthread-safety" "-Wself-assign" "-fcolor-diagnostics" "-fno-omit-frame-pointer" "-no-canonical-prefixes" "-Wno-builtin-macro-redefined" "-D__DATE__=\"redacted\"" "-D__TIMESTAMP__=\"redacted\"" "-D__TIME__=\"redacted\"" "-Wall" "-Wextra" "-DCFG_TARGET_OS_linux" "-DCFG_TARGET_ARCH_s390x" "-o" "/home/runner/.cache/bazel/_bazel_runner/6e9090fa382dda61db70769c6a68a5bd/sandbox/processwrapper-sandbox/218/execroot/proxy_wasm_cpp_host/bazel-out/s390x-fastbuild/bin/external/cu__wasmtime-runtime-9.0.4/wasmtime-runtime_build_script.out_dir/eb6e60f2655818a1-s390x.o" "-c" "src/trampolines/s390x.S" with args clang did not execute successfully (status code exit status: 1).

[PiotrSikora]

That's new. I imagine that this might disappear with update to any of: Clang (or ideally, Ubuntu base image), Rust (as part of rules_rust) or Wasmtime, so you could revisit this after all of those are done.

[martijneken]

Yeah . I think your old image used clang-14 (given its name), whereas Ubuntu 20.04 used clang-10 by default. I'll start with an update to Ubuntu 22.04 which gets us back to clang-14.

[martijneken]

Yippee, success: https://github.com/proxy-wasm/proxy-wasm-cpp-host/actions/runs/10352997621/job/28654988504?pr=408