Skip to content

Commit

Permalink
Implement a workaround for macOS/Bazel flakes.
Browse files Browse the repository at this point in the history
Bazel has a 2 minute timeout for their internal `xcrun` call, which can be exceeded on our github runners about 5% of the time.  This leads to flakes and opaque errors, but is a one-time cost.  Subsequent xcruns finish in seconds, so we can just do an initial call w/o a timeout before running Bazel.

See bazelbuild/bazel#17437 for background.

PiperOrigin-RevId: 509869860
  • Loading branch information
mkruskal-google authored and copybara-github committed Feb 15, 2023
1 parent 4b1cd0d commit 88c78e6
Show file tree
Hide file tree
Showing 2 changed files with 26 additions and 1 deletion.
23 changes: 23 additions & 0 deletions .github/actions/bazel/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,29 @@ runs:
run: ${{ inputs.bash }}
shell: bash

# Bazel has multiple Xcode calls with hardcoded timeouts. Many of these
# end up timing out on our github runners, causing flakes on every mac
# build that invoked Bazel. To work around this, we manually inoke these
# calls before running Bazel to make sure they end up in Xcode's cache for
# quicker runs later. All of these calls are obtained from xcrun calls in
# https://github.com/bazelbuild/bazel/blob/e8a69f5d5acaeb6af760631490ecbf73e8a04eeb/tools/cpp/osx_cc_configure.bzl.
# See https://github.com/bazelbuild/bazel/issues/17437 for more details.
# TODO(b/269503614) Remove this once Bazel provides an official solution.
- name: Warm up Xcode
if: ${{ runner.os == 'macOS' }}
shell: bash
run: |
mkdir -p mac_bazel_workaround
bazelisk ${{ steps.bazel.outputs.bazel-startup-flags }} build @bazel_tools//tools/osx:xcode_locator.m $BAZEL_FLAGS
XCODE_LOCATOR_FLAGS="--sdk macosx clang -mmacosx-version-min=10.9 -fobjc-arc -framework CoreServices -framework Foundation"
SINGLE_ARCH_COMPILE_FLAGS="--sdk macosx clang -mmacosx-version-min=10.9 -std=c++11 -lc++ -O3"
COMPILE_FLAGS="$SINGLE_ARCH_COMPILE_FLAGS -arch arm64 -arch x86_64 -Wl,-no_adhoc_codesign -Wl,-no_uuid -O3"
time env -i DEVELOPER_DIR=$DEVELOPER_DIR xcrun $XCODE_LOCATOR_FLAGS -o mac_bazel_workaround/xcode-locator-bin $(bazel info output_base)/external/bazel_tools/tools/osx/xcode_locator.m
time env -i DEVELOPER_DIR=$DEVELOPER_DIR xcrun $SINGLE_ARCH_COMPILE_FLAGS -o mac_bazel_workaround/libtool_check_unique $(bazel info output_base)/external/bazel_tools/tools/objc/libtool_check_unique.cc
time env -i DEVELOPER_DIR=$DEVELOPER_DIR xcrun $COMPILE_FLAGS -o mac_bazel_workaround/libtool_check_unique $(bazel info output_base)/external/bazel_tools/tools/objc/libtool_check_unique.cc
time env -i DEVELOPER_DIR=$DEVELOPER_DIR xcrun $SINGLE_ARCH_COMPILE_FLAGS -o mac_bazel_workaround/wrapped_clang $(bazel info output_base)/external/bazel_tools/tools/osx/crosstool/wrapped_clang.cc
time env -i DEVELOPER_DIR=$DEVELOPER_DIR xcrun $COMPILE_FLAGS -o mac_bazel_workaround/wrapped_clang $(bazel info output_base)/external/bazel_tools/tools/osx/crosstool/wrapped_clang.cc
- name: Run Bazel
if: ${{ !inputs.bash }}
run: >-
Expand Down
4 changes: 3 additions & 1 deletion .github/actions/internal/bazel-setup/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,9 @@ runs:
- name: Initialize MacOS-specific Bazel flags
if: runner.os == 'macOS'
shell: bash
run: echo "BAZEL_FLAGS=$BAZEL_FLAGS --xcode_version_config=//.github:host_xcodes" >> $GITHUB_ENV
run: |
echo "BAZEL_FLAGS=$BAZEL_FLAGS --xcode_version_config=//.github:host_xcodes" >> $GITHUB_ENV
echo "DEVELOPER_DIR=${{ env.DEVELOPER_DIR || '/Applications/Xcode_14.1.app/Contents/Developer' }}" >> $GITHUB_ENV
- name: Configure Bazel caching
# Skip bazel cache for local act runs due to issue with credential files
Expand Down

0 comments on commit 88c78e6

Please sign in to comment.