chore: switch to bzlmod #712

janwinkler1 · 2025-03-23T17:12:59Z

enable bzlmod
adapting e2e tests (copy pasted from chore: Update bazel to 8.1.0 #703)
conservatively increase timeouts for tests failing in macos CI step
fixing race condition causing macos CI step to fail
changes in go.mod were only necessary to get CI ubuntu steps passing, so we might want to upgrade CI ubuntu build steps soonish, tracked in issue Upgrade CI ubuntu build steps #714

achew22 · 2025-03-28T17:11:19Z

internal/e2e/common.go

+		// ModuleFileSuffix: "# Comment to force us to use MODULE files by default",
+		// WorkspaceSuffix:  "# Comment to force us to use MODULE files by default",


Please rm (sorry, I know I actually authored them), they make the story on this file a bit more opaque than I'd like

internal/e2e/incompatible_by_default/incompatible_by_default_test.go

achew22 · 2025-03-28T17:13:59Z

internal/e2e/output_runner/output_runner_test.go

@@ -85,85 +81,6 @@ func checkSentinel(t *testing.T, sentinelFile *os.File, msg string) {
 	}
 }

-func TestOutputRunner(t *testing.T) {


Is this the only test that only fails on a mac (#715)?

no, this test (TestOutputRunner) I removed because it was consistently failing in local testing environments (in my case on ubuntu 24.04).

i first thought it fails because of sandboxing issues (introduced by upgrading bazel version).

that was disproved by trying to run the test without sandboxing (bazel test //internal/e2e/output_runner:output_runner_test --spawn_strategy=standalone --test_strategy=standalone and it still failed

the underlying issue appears to be related to assumptions by the test on the file system or ibazel execution contexts

probably this was unclear (my bad) and i opened Reintroduce TestOutputRunner #716 for that

I decided to delete this given that the test does not cover critical functionality and addressing it properly requires deeper investigation (outside the scope of this PR)

the e2e tests for mac I turned off completely here: https://github.com/bazelbuild/bazel-watcher/pull/709/files#:~:text=%22...%22-,%2D%20%22%2D//internal/e2e/...%22,-macos_legacy_watcher%3A

Hrm... maybe I'm mistaken here, but it seems to me that this test is the only one that confirms that the output runner stuff actually works. Doesn't removing this test therefore significantly reduce the tested surface area for that feature?

you are not mistaken. and yes it is the only test that tests the output runner stuff.. atm i cant find a solution to quickly fix that test to leave it in. i spent some more time in trying to fix and was not sure if i can manage with my go skills (little experience), sorry.

#710 resolved this!

internal/ibazel/fswatcher/fsevents/BUILD

achew22 · 2025-04-03T23:40:04Z

Could you rebase this off of main? Not sure what caused the influx of attention of late, but I'm happy to see it. I think the other change also resolves the issue that was causing the test you removed to fail. I'd like to see what happens with that restored. Maybe we can get to bzlmod soon!

janwinkler1 · 2025-04-04T17:05:39Z

@achew22 i rebased off of main and cleaned up, PTAL

achew22 · 2025-04-04T18:48:50Z

.bazelci/presubmit.yml

    test_targets:
-    - "..."
+      - "..."
+      - "-//internal/e2e/override_repository:override_repository_test"


I think we should aggressively avoid using this file to exclude tests from CI. It is too distant from the actual test case. It is set up to have --test_tag_filters=-nomacos which I believe is a better strategy since the file where the test is defined is also the place where the test is deactivated.

Beyond this, I'm not sure that I'm comfortable disabling so many tests. Yes, it is frustrating that they are not passing but their failure to me indicates that we may be introducing new bugs into the project. Since we took the time to make sure there weren't regressions, I would like to get the number of skipped/ignored test cases to the absolute bare minimum.

@achew22 thank you for your honest review! agreed. will take another look and see if i am able to fix macos related tests. maybe i can collaborate more closely with @chickenandpork doing this 👍

achew22 · 2025-04-04T18:49:53Z

internal/ibazel/ibazel_test.go

@@ -185,6 +185,9 @@ func TestIBazelLifecycle(t *testing.T) {
 }

 func TestIBazelLoop(t *testing.T) {
+  if runtime.GOOS == "darwin" && os.Getenv("IBAZEL_USE_LEGACY_WATCHER") == "1" {
+    t.Skip("Skipping TestIBazelLoop on macOS with legacy watcher due to known race condition")


I don't think this skip is needed any more thanks to the PR that was merged into main

@achew22 (//cc @chickenandpork for context) per your request i removed this check, but macos darwin CI buildstep failed again with

Output logs for macos darwin build step

$ /usr/local/etc/buildkite-agent/hooks/environment # ANDROID_HOME added # COURSIER_OPTS added # ANDROID_NDK_HOME added # SSL_CERT_FILE added # JAVA_TOOL_OPTIONS added # BUILDKITE_ARTIFACT_UPLOAD_DESTINATION is now "gs://bazel-untrusted-buildkite-artifacts/01961fb9-831a-4744-8a62-12a198ba25a3" # BUILDKITE_GS_APPLICATION_CREDENTIALS added Preparing working directory # Creating "/Users/buildkite/builds/bk-macos-intel-i58z/bazel/bazel-watcher" $ cd /Users/buildkite/builds/bk-macos-intel-i58z/bazel/bazel-watcher $ cd /usr/local/var/bazelbuild # Updating existing repository mirror to find commit 8e642abd957e85e0a189140e674f27f6880aa681 # Fetch and mirror pull request head from GitHub $ git --git-dir /usr/local/var/bazelbuild/https---github-com-bazelbuild-bazel-watcher-git fetch origin refs/pull/712/head remote: Enumerating objects: 838, done. remote: Counting objects: 100% (401/401), done. remote: Compressing objects: 100% (74/74), done. remote: Total 838 (delta 357), reused 328 (delta 327), pack-reused 437 (from 3) Receiving objects: 100% (838/838), 272.51 KiB | 10.90 MiB/s, done. Resolving deltas: 100% (436/436), completed with 50 local objects. From https://github.com/bazelbuild/bazel-watcher * branch refs/pull/712/head -> FETCH_HEAD $ cd /Users/buildkite/builds/bk-macos-intel-i58z/bazel/bazel-watcher $ git clone -v --reference /usr/local/var/bazelbuild/https---github-com-bazelbuild-bazel-watcher-git -- https://github.com/bazelbuild/bazel-watcher.git . Cloning into '.'... POST git-upload-pack (191 bytes) POST git-upload-pack (gzip 2809 to 1402 bytes) remote: Enumerating objects: 793, done. remote: Counting objects: 100% (400/400), done. remote: Compressing objects: 100% (65/65), done. remote: Total 793 (delta 365), reused 335 (delta 335), pack-reused 393 (from 3) Receiving objects: 100% (793/793), 259.28 KiB | 15.25 MiB/s, done. Resolving deltas: 100% (421/421), completed with 50 local objects. $ git clean -ffxdq # Fetch and checkout pull request head from GitHub $ git fetch -v --prune -- origin refs/pull/712/head POST git-upload-pack (405 bytes) From https://github.com/bazelbuild/bazel-watcher * branch refs/pull/712/head -> FETCH_HEAD # FETCH_HEAD is now `8e642abd957e85e0a189140e674f27f6880aa681` $ git fetch -v --prune -- origin 8e642abd957e85e0a189140e674f27f6880aa681 POST git-upload-pack (118 bytes) From https://github.com/bazelbuild/bazel-watcher * branch 8e642abd957e85e0a189140e674f27f6880aa681 -> FETCH_HEAD $ git checkout -f 8e642abd957e85e0a189140e674f27f6880aa681 Note: switching to '8e642abd957e85e0a189140e674f27f6880aa681'. You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by switching back to a branch. If you want to create a new branch to retain commits you create, you may do so (now or later) by using -c with the switch command. Example: git switch -c <new-branch-name> Or undo this operation with: git switch - Turn off this advice by setting config variable advice.detachedHead to false HEAD is now at 8e642ab Remove skip for ibazel test # Cleaning again to catch any post-checkout changes $ git clean -ffxdq # Checking to see if git commit information needs to be sent to Buildkite... $ buildkite-agent meta-data exists buildkite:git:commit # Sending Git commit information back to Buildkite $ buildkite-agent meta-data set buildkite:git:commit < /dev/stdin 2025-04-10 14:43:02 INFO Reading meta-data value from STDIN Running commands $ curl -sS https://raw.githubusercontent.com/bazelbuild/continuous-integration/master/buildkite/bazelci.py?1744288973 -o bazelci.py python3 bazelci.py runner --task=macos_legacy_watcher :xcode: Activating Xcode 16.2.0.16C5032a... /usr/bin/sudo /usr/bin/xcode-select --switch /Applications/Xcode16.2.0.16C5032a.app /usr/bin/sudo /usr/bin/xcodebuild -runFirstLaunch :bazel: Using Bazel version bazel --version 2025/04/10 14:43:03 Downloading https://releases.bazel.build/7.4.1/release/bazel-7.4.1-darwin-x86_64... Downloading: 53 MB out of 53 MB (100%) bazel 7.4.1 bazel --host_jvm_args=-Djava.net.preferIPv6Addresses=true info output_base Starting local Bazel server and connecting to it... WARNING: ignoring JAVA_TOOL_OPTIONS in environment. ℹ️ Bazel Info bazel --host_jvm_args=-Djava.net.preferIPv6Addresses=true --nosystem_rc --nohome_rc version WARNING: The following rc files are no longer being read, please transfer their contents or import their path into one of the standard rc files: /etc/bazel.bazelrc WARNING: Running Bazel server needs to be killed, because the startup options are different. Starting local Bazel server and connecting to it... WARNING: ignoring JAVA_TOOL_OPTIONS in environment. Bazelisk version: v1.25.0 Build label: 7.4.1 Build target: @@//src/main/java/com/google/devtools/build/lib/bazel:BazelServer Build time: Mon Nov 11 21:36:40 2024 (1731361000) Build timestamp: 1731361000 Build timestamp as int: 1731361000 bazel --host_jvm_args=-Djava.net.preferIPv6Addresses=true --nosystem_rc --nohome_rc info WARNING: The following rc files are no longer being read, please transfer their contents or import their path into one of the standard rc files: /etc/bazel.bazelrc bazel-bin: /private/var/tmp/_bazel_buildkite/781631121efdd6b88434574fe386ed52/execroot/_main/bazel-out/darwin_x86_64-fastbuild/bin bazel-genfiles: /private/var/tmp/_bazel_buildkite/781631121efdd6b88434574fe386ed52/execroot/_main/bazel-out/darwin_x86_64-fastbuild/bin bazel-testlogs: /private/var/tmp/_bazel_buildkite/781631121efdd6b88434574fe386ed52/execroot/_main/bazel-out/darwin_x86_64-fastbuild/testlogs character-encoding: file.encoding = ISO-8859-1, defaultCharset = ISO-8859-1, sun.jnu.encoding = UTF-8 command_log: /private/var/tmp/_bazel_buildkite/781631121efdd6b88434574fe386ed52/command.log committed-heap-size: 157MB execution_root: /private/var/tmp/_bazel_buildkite/781631121efdd6b88434574fe386ed52/execroot/_main gc-count: 13 gc-time: 53ms install_base: /var/tmp/_bazel_buildkite/install/e3a47ff802b0af64d7cc5ab3de4855d6 java-home: /private/var/tmp/_bazel_buildkite/install/e3a47ff802b0af64d7cc5ab3de4855d6/embedded_tools/jdk java-runtime: OpenJDK Runtime Environment (build 21.0.4+7-LTS) by Azul Systems, Inc. java-vm: OpenJDK 64-Bit Server VM (build 21.0.4+7-LTS, mixed mode) by Azul Systems, Inc. local_resources: RAM=16384MB, CPU=8.0 max-heap-size: 4294MB output_base: /private/var/tmp/_bazel_buildkite/781631121efdd6b88434574fe386ed52 output_path: /private/var/tmp/_bazel_buildkite/781631121efdd6b88434574fe386ed52/execroot/_main/bazel-out package_path: %workspace% release: release 7.4.1 repository_cache: /var/tmp/_bazel_buildkite/cache/repos/v1 server_log: /private/var/tmp/_bazel_buildkite/781631121efdd6b88434574fe386ed52/java.log.05bcad3f-572c-4278-91a6-71b049fbcb84.buildkite.log.java.20250410-144309.18175 server_pid: 18175 used-heap-size: 65MB workspace: /Users/buildkite/builds/bk-macos-intel-i58z/bazel/bazel-watcher ℹ️ Environment Variables BUILDKITE_CONFIG_PATH=(/usr/local/etc/buildkite-agent/buildkite-agent.cfg) BUILDKITE_AGENT_META_DATA_OS_VERSION=(15.3.1) BUILDKITE_SIGNAL_GRACE_PERIOD_SECONDS=(9) BUILDKITE_BUILD_CREATOR=(janwinkler1) BUILDKITE_LAST_HOOK_EXIT_STATUS=(0) BUILDKITE_AGENT_EXPERIMENT=() SSL_CERT_FILE=(/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/certifi/cacert.pem) BUILDKITE_AGENT_PID=(17012) ANDROID_HOME=(/Users/buildkite/android-sdk-macosx) TERM=(xterm-256color) SHELL=(/bin/zsh) BUILDKITE_RETRY_COUNT=(0) BUILDKITE_NO_HTTP2=(false) BUILDKITE_ENV_FILE=(/tmp/job-env-01961fb9-831a-4744-8a62-12a198ba25a3969087917) BUILDKITE_ENV_JSON_FILE=(/tmp/job-env-json-01961fb9-831a-4744-8a62-12a198ba25a31633366842) BUILDKITE_ARTIFACT_PATHS=() GOOGLE_APPLICATION_CREDENTIALS=(/usr/local/etc/buildkite-agent/bazel.json) BUILDKITE_GIT_FETCH_FLAGS=(-v --prune) BUILDKITE_CANCEL_GRACE_PERIOD=(10) BUILDKITE_SOURCE=(webhook) BUILDKITE_ARTIFACT_UPLOAD_DESTINATION=(gs://bazel-untrusted-buildkite-artifacts/01961fb9-831a-4744-8a62-12a198ba25a3) BUILDKITE_AGENT_DEBUG=(false) BUILDKITE_GIT_CLONE_MIRROR_FLAGS=(-v --bare) BUILDKITE_SCRIPT_PATH=(curl -sS https://raw.githubusercontent.com/bazelbuild/continuous-integration/master/buildkite/bazelci.py?1744288973 -o bazelci.py python3 bazelci.py runner --task=macos_legacy_watcher) BUILDKITE_ORGANIZATION_SLUG=(bazel) BUILDKITE_AGENT_META_DATA_MACHINE_TYPE=(macmini) BUILDKITE_LOCAL_HOOKS_ENABLED=(true) BUILDKITE_COMMIT_RESOLVED=(true) BUILDKITE_AGENT_ACCESS_TOKEN=([REDACTED]) BUILDKITE_PROJECT_SLUG=(bazel/bazel-watcher) GIT_TERMINAL_PROMPT=(0) BUILDKITE_SOCKETS_PATH=(/Users/buildkite/.buildkite-agent/sockets) USER=(buildkite) BUILDKITE_COMPUTE_TYPE=(self-hosted) SUDO_USER=(root) BUILDKITE_ORGANIZATION_ID=(586ac9dd-b547-4a52-9d73-9e3a43ff74f9) SUDO_UID=(0) BUILDKITE_PULL_REQUEST_BASE_BRANCH=(master) BUILDKITE_BUILD_CREATOR_EMAIL=([email protected]) BUILDKITE_PROJECT_PROVIDER=(github) BUILDKITE_STEP_KEY=() BUILDKITE_PULL_REQUEST_REPO=(https://github.com/janwinkler1/bazel-watcher.git) BUILDKITE_PLUGINS_ENABLED=(true) BUILDKITE_GS_APPLICATION_CREDENTIALS=(/usr/local/etc/buildkite-agent/bazel.json) BUILDKITE_BUILD_CREATOR_TEAMS=() BUILDKITE_AGENT_DEBUG_HTTP=(false) BUILDKITE_PULL_REQUEST=(712) BUILDKITE_GIT_MIRRORS_LOCK_TIMEOUT=(300) BUILDKITE_AGENT_ENDPOINT=(https://agent.buildkite.com/v3) BUILDKITE_HOOKS_PATH=(/usr/local/etc/buildkite-agent/hooks) BUILDKITE_GIT_CLEAN_FLAGS=(-ffxdq) BUILDKITE_BUILD_ID=(01961fb9-6afe-44ab-a984-457ee9f08a4b) PATH=(/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin) MAIL=(/var/mail/root) BUILDKITE_TRIGGERED_FROM_BUILD_NUMBER=() BUILDKITE_REDACTED_VARS=(*_PASSWORD,*_SECRET,*_TOKEN,*_PRIVATE_KEY,*_ACCESS_KEY,*_SECRET_KEY,*_CONNECTION_STRING) BUILDKITE_GIT_MIRRORS_PATH=(/usr/local/var/bazelbuild) BUILDKITE_PIPELINE_ID=(05deb2be-d451-45b6-a71e-b39b771b837b) COURSIER_OPTS=(-Djava.net.preferIPv6Addresses=true) BUILDKITE_BUILD_AUTHOR_EMAIL=([email protected]) PWD=(/Users/buildkite/builds/bk-macos-intel-i58z/bazel/bazel-watcher) BUILDKITE_MESSAGE=(chore: switch to bzlmod) BUILDKITE_SHELL=(/bin/bash -e -c) BUILDKITE_PIPELINE_PROVIDER=(github) BUILDKITE_GIT_SUBMODULES=(true) BUILDKITE_REBUILT_FROM_BUILD_NUMBER=() BUILDKITE_GIT_MIRRORS_SKIP_UPDATE=(false) BUILDKITE_BUILD_NUMBER=(3993) BUILDKITE_AGENT_META_DATA_QUEUE=(macos) BAZEL_BUCKET_TYPE=(untrusted) BUILDKITE_SSH_KEYSCAN=(true) BUILDKITE_AGENT_META_DATA_KIND=(worker) BUILDKITE_TRIGGERED_FROM_BUILD_PIPELINE_SLUG=() BUILDKITE_PLUGINS_PATH=(/usr/local/var/buildkite-agent/plugins) BUILDKITE_LABEL=(:darwin: macOS) BUILDKITE_BUILD_URL=(https://buildkite.com/bazel/bazel-watcher/builds/3993) BUILDKITE_PULL_REQUEST_LABELS=() BUILDKITE_JOB_ID=(01961fb9-831a-4744-8a62-12a198ba25a3) BUILDKITE_PIPELINE_SLUG=(bazel-watcher) BUILDKITE_AGENT_ID=(01961e53-29ae-4386-a562-458739ae4e13) JAVA_TOOL_OPTIONS=(-Djava.net.preferIPv6Addresses=true) SHLVL=(1) SUDO_COMMAND=(/usr/bin/env GOOGLE_APPLICATION_CREDENTIALS=/usr/local/etc/buildkite-agent/bazel.json PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin BUILDKITE_AGENT_DISCONNECT_AFTER_JOB=true BUILDKITE_AGENT_DISCONNECT_AFTER_IDLE_TIMEOUT=82800 BUILDKITE_AGENT_NAME=bk-macos-intel-i58z BUILDKITE_AGENT_PRIORITY=0 BUILDKITE_AGENT_TOKEN=f411f9d8498d4f9762e554835e81a24e2cd9a15fa6740b1276 BUILDKITE_AGENT_TAGS=queue=macos,kind=worker,os=macos,os-version=15.3.1,machine-type=macmini BUILDKITE_BUILD_PATH=/Users/buildkite/builds BUILDKITE_CONFIG_PATH=/usr/local/etc/buildkite-agent/buildkite-agent.cfg BUILDKITE_GIT_MIRRORS_PATH=/usr/local/var/bazelbuild BUILDKITE_GIT_CLONE_MIRROR_FLAGS=-v --bare BAZEL_BUCKET_TYPE=untrusted /usr/local/bin/buildkite-agent start) HOME=(/Users/buildkite) BUILDKITE_TRIGGERED_FROM_BUILD_ID=() BUILDKITE_REPO_MIRROR=(/usr/local/var/bazelbuild/https---github-com-bazelbuild-bazel-watcher-git) BUILDKITE_AGENT_JOB_API_SOCKET=(/Users/buildkite/.buildkite-agent/sockets/job-api/18110-12093.sock) BUILDKITE_COMMAND=(curl -sS https://raw.githubusercontent.com/bazelbuild/continuous-integration/master/buildkite/bazelci.py?1744288973 -o bazelci.py python3 bazelci.py runner --task=macos_legacy_watcher) BUILDKITE_BIN_PATH=(/usr/local/bin) CI=(true) BUILDKITE_REPO=(https://github.com/bazelbuild/bazel-watcher.git) BUILDKITE_TRACE_CONTEXT_ENCODING=(gob) BUILDKITE=(true) BUILDKITE_STEP_ID=(01961fb9-82f4-48e5-8161-53fa3209abfa) LOGNAME=(buildkite) BUILDKITE_TAG=() BUILDKITE_BRANCH=(janwinkler1:switch-to-bzlmod) BUILDKITE_TIMEOUT=(480) BUILDKITE_PIPELINE_TEAMS=(bazel) BUILDKITE_AGENT_DISABLE_WARNINGS_FOR=() BUILDKITE_REBUILT_FROM_BUILD_ID=() BUILDKITE_BUILD_CHECKOUT_PATH=(/Users/buildkite/builds/bk-macos-intel-i58z/bazel/bazel-watcher) BUILDKITE_AGENT_NAME=(bk-macos-intel-i58z) BUILDKITE_COMMAND_EVAL=(true) BUILDKITE_AGENT_JOB_API_TOKEN=([REDACTED]) BUILDKITE_PLUGIN_VALIDATION=(false) BUILDKITE_AGENT_META_DATA_OS=(macos) ANDROID_NDK_HOME=(/Users/buildkite/android-ndk-r15c) SUDO_GID=(0) BUILDKITE_STRICT_SINGLE_HOOKS=(false) BUILDKITE_GIT_CHECKOUT_FLAGS=(-f) BUILDKITE_COMMIT=(8e642abd957e85e0a189140e674f27f6880aa681) BUILDKITE_PIPELINE_NAME=(Bazel watcher) BUILDKITE_GIT_CLONE_FLAGS=(-v) BUILDKITE_BUILD_PATH=(/Users/buildkite/builds) BUILDKITE_BUILD_AUTHOR=(janwinkler1) BUILDKITE_PIPELINE_DEFAULT_BRANCH=(master) _=(/usr/local/bin/python3) __CF_USER_TEXT_ENCODING=(0x1F5:0:0) LC_CTYPE=(UTF-8) BAZELCI_TASK=(macos_legacy_watcher) BAZELISK_USER_AGENT=(Bazelisk/BazelCI) OUTPUT_BASE=(/private/var/tmp/_bazel_buildkite/781631121efdd6b88434574fe386ed52) 🎯 Calculating targets :bazel: Computing flags for build step Adding to platform cache key: b'bazel' Adding to platform cache key: b'cache-poisoning-20241105' Adding to platform cache key: b'macos' Adding to platform cache key: b'15.3.1\n' Adding to platform cache key: b'/Applications/Xcode16.2.0.16C5032a.app/Contents/Developer\n' Adding to platform cache key: b'Xcode 16.2\nBuild version 16C5032a\n' :bazel: Build (7.4.1) bazel --host_jvm_args=-Djava.net.preferIPv6Addresses=true build --show_progress_rate_limit=5 --curses=yes --color=yes --terminal_columns=143 --show_timestamps --verbose_failures --jobs=8 --announce_rc --experimental_repository_cache_hardlinks --disk_cache= --sandbox_writable_path=/var/tmp/_bazel_buildkite/cache/repos/v1 --test_env=REPOSITORY_CACHE=/var/tmp/_bazel_buildkite/cache/repos/v1 --jvmopt=-Djava.net.preferIPv6Addresses --remote_cache=https://storage.googleapis.com/bazel-untrusted-build-cache --noexperimental_remote_cache_async --google_default_credentials --remote_timeout=3600 --remote_max_connections=200 --remote_default_platform_properties=properties:{name:"cache-silo-key" value:"3574fb10d429020863d5814b32cd1c7032ec1c87e49e968debe1bb45530739b5"} --remote_download_toplevel --build_tag_filters=-nomacos --test_env=HOME --test_env=BAZELISK_USER_AGENT --test_env=COURSIER_OPTS --test_env=JAVA_TOOL_OPTIONS --test_env=SSL_CERT_FILE -- ... WARNING: Running Bazel server needs to be killed, because the startup options are different. Starting local Bazel server and connecting to it... WARNING: ignoring JAVA_TOOL_OPTIONS in environment. (14:43:16) WARNING: Option 'remote_default_platform_properties' is deprecated: --remote_default_platform_properties has been deprecated in favor of --remote_default_exec_properties. (14:43:16) INFO: Invocation ID: e308a3f9-a88b-44ea-9bfb-7cb0fb4c6bd7 (14:43:16) INFO: Reading 'startup' options from /private/etc/bazel.bazelrc: --host_jvm_args=-Djava.net.preferIPv6Addresses=true (14:43:16) INFO: Options provided by the client: Inherited 'common' options: --isatty=1 --terminal_columns=160 (14:43:16) INFO: Reading rc options for 'build' from /private/etc/bazel.bazelrc: Inherited 'common' options: --jvmopt=-Djava.net.preferIPv6Addresses (14:43:16) INFO: Reading rc options for 'build' from /Users/buildkite/builds/bk-macos-intel-i58z/bazel/bazel-watcher/.bazelrc: 'build' options: --workspace_status_command=python3 tools/workplace_status.py (14:43:16) INFO: Current date is 2025-04-10 (14:43:17) WARNING: Option 'remote_default_platform_properties' is deprecated: --remote_default_platform_properties has been deprecated in favor of --remote_default_exec_properties. (14:43:27) INFO: Analyzed 57 targets (221 packages loaded, 13364 targets configured). (14:43:27) INFO: From Compiling src/google/protobuf/stubs/strutil.cc [for tool]: external/protobuf~/src/google/protobuf/stubs/strutil.cc:506:11: warning: 'sprintf' is deprecated: This function is provided for compatibility reasons only. Due to security concerns inherent in the design of sprintf(3), it is highly recommended that you use snprintf(3) instead. [-Wdeprecated-declarations] 506 | sprintf(dest + used, (use_hex ? "\\x%02x" : "\\%03o"), | ^ /Applications/Xcode16.2.0.16C5032a.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/_stdio.h:274:1: note: 'sprintf' has been explicitly marked deprecated here 274 | __deprecated_msg("This function is provided for compatibility reasons only. Due to security concerns inherent in the design of sprintf(3), it is highly recommended that you use snprintf(3) instead.") | ^ /Applications/Xcode16.2.0.16C5032a.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/sys/cdefs.h:218:48: note: expanded from macro '__deprecated_msg' 218 | #define __deprecated_msg(_msg) __attribute__((__deprecated__(_msg))) | ^ 1 warning generated. (14:43:28) INFO: From Compiling src/google/protobuf/generated_message_tctable_lite.cc [for tool]: external/protobuf~/src/google/protobuf/generated_message_tctable_lite.cc:347:14: warning: unused function 'Offset' [-Wunused-function] 347 | inline void* Offset(void* base, uint32_t offset) { | ^~~~~~ 1 warning generated. (14:43:28) INFO: From Linking external/protobuf~/libprotobuf_lite.a [for tool]: /Applications/Xcode16.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: file: bazel-out/darwin_x86_64-opt-exec-ST-d57f47055a04/bin/external/protobuf~/_objs/protobuf_lite/io_win32.o has no symbols (14:43:29) INFO: From Compiling src/google/protobuf/compiler/cpp/helpers.cc [for tool]: external/protobuf~/src/google/protobuf/compiler/cpp/helpers.cc:197:25: warning: unused function 'VerifyInt32TypeToVerifyCustom' [-Wunused-function] 197 | inline VerifySimpleType VerifyInt32TypeToVerifyCustom(VerifyInt32Type t) { | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1 warning generated. (14:43:34) INFO: From Linking external/protobuf~/libprotoc_lib.a [for tool]: /Applications/Xcode16.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: warning duplicate member name 'enum.o' from 'bazel-out/darwin_x86_64-opt-exec-ST-d57f47055a04/bin/external/protobuf~/_objs/protoc_lib/0/enum.o(enum.o)' and 'bazel-out/darwin_x86_64-opt-exec-ST-d57f47055a04/bin/external/protobuf~/_objs/protoc_lib/1/enum.o(enum.o)' /Applications/Xcode16.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: warning duplicate member name 'enum_field.o' from 'bazel-out/darwin_x86_64-opt-exec-ST-d57f47055a04/bin/external/protobuf~/_objs/protoc_lib/1/enum_field.o(enum_field.o)' and 'bazel-out/darwin_x86_64-opt-exec-ST-d57f47055a04/bin/external/protobuf~/_objs/protoc_lib/0/enum_field.o(enum_field.o)' /Applications/Xcode16.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: warning duplicate member name 'extension.o' from 'bazel-out/darwin_x86_64-opt-exec-ST-d57f47055a04/bin/external/protobuf~/_objs/protoc_lib/0/extension.o(extension.o)' and 'bazel-out/darwin_x86_64-opt-exec-ST-d57f47055a04/bin/external/protobuf~/_objs/protoc_lib/1/extension.o(extension.o)' /Applications/Xcode16.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: warning duplicate member name 'field.o' from 'bazel-out/darwin_x86_64-opt-exec-ST-d57f47055a04/bin/external/protobuf~/_objs/protoc_lib/1/field.o(field.o)' and 'bazel-out/darwin_x86_64-opt-exec-ST-d57f47055a04/bin/external/protobuf~/_objs/protoc_lib/0/field.o(field.o)' /Applications/Xcode16.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: warning duplicate member name 'file.o' from 'bazel-out/darwin_x86_64-opt-exec-ST-d57f47055a04/bin/external/protobuf~/_objs/protoc_lib/1/file.o(file.o)' and 'bazel-out/darwin_x86_64-opt-exec-ST-d57f47055a04/bin/external/protobuf~/_objs/protoc_lib/0/file.o(file.o)' /Applications/Xcode16.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: warning duplicate member name 'generator.o' from 'bazel-out/darwin_x86_64-opt-exec-ST-d57f47055a04/bin/external/protobuf~/_objs/protoc_lib/0/generator.o(generator.o)' and 'bazel-out/darwin_x86_64-opt-exec-ST-d57f47055a04/bin/external/protobuf~/_objs/protoc_lib/1/generator.o(generator.o)' /Applications/Xcode16.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: warning duplicate member name 'generator.o' from 'bazel-out/darwin_x86_64-opt-exec-ST-d57f47055a04/bin/external/protobuf~/_objs/protoc_lib/1/generator.o(generator.o)' and 'bazel-out/darwin_x86_64-opt-exec-ST-d57f47055a04/bin/external/protobuf~/_objs/protoc_lib/2/generator.o(generator.o)' /Applications/Xcode16.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: warning duplicate member name 'helpers.o' from 'bazel-out/darwin_x86_64-opt-exec-ST-d57f47055a04/bin/external/protobuf~/_objs/protoc_lib/2/helpers.o(helpers.o)' and 'bazel-out/darwin_x86_64-opt-exec-ST-d57f47055a04/bin/external/protobuf~/_objs/protoc_lib/0/helpers.o(helpers.o)' /Applications/Xcode16.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: warning duplicate member name 'helpers.o' from 'bazel-out/darwin_x86_64-opt-exec-ST-d57f47055a04/bin/external/protobuf~/_objs/protoc_lib/0/helpers.o(helpers.o)' and 'bazel-out/darwin_x86_64-opt-exec-ST-d57f47055a04/bin/external/protobuf~/_objs/protoc_lib/1/helpers.o(helpers.o)' /Applications/Xcode16.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: warning duplicate member name 'map_field.o' from 'bazel-out/darwin_x86_64-opt-exec-ST-d57f47055a04/bin/external/protobuf~/_objs/protoc_lib/0/map_field.o(map_field.o)' and 'bazel-out/darwin_x86_64-opt-exec-ST-d57f47055a04/bin/external/protobuf~/_objs/protoc_lib/1/map_field.o(map_field.o)' /Applications/Xcode16.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: warning duplicate member name 'message.o' from 'bazel-out/darwin_x86_64-opt-exec-ST-d57f47055a04/bin/external/protobuf~/_objs/protoc_lib/1/message.o(message.o)' and 'bazel-out/darwin_x86_64-opt-exec-ST-d57f47055a04/bin/external/protobuf~/_objs/protoc_lib/0/message.o(message.o)' /Applications/Xcode16.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: warning duplicate member name 'message_field.o' from 'bazel-out/darwin_x86_64-opt-exec-ST-d57f47055a04/bin/external/protobuf~/_objs/protoc_lib/0/message_field.o(message_field.o)' and 'bazel-out/darwin_x86_64-opt-exec-ST-d57f47055a04/bin/external/protobuf~/_objs/protoc_lib/1/message_field.o(message_field.o)' /Applications/Xcode16.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: warning duplicate member name 'primitive_field.o' from 'bazel-out/darwin_x86_64-opt-exec-ST-d57f47055a04/bin/external/protobuf~/_objs/protoc_lib/1/primitive_field.o(primitive_field.o)' and 'bazel-out/darwin_x86_64-opt-exec-ST-d57f47055a04/bin/external/protobuf~/_objs/protoc_lib/0/primitive_field.o(primitive_field.o)' /Applications/Xcode16.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: warning duplicate member name 'service.o' from 'bazel-out/darwin_x86_64-opt-exec-ST-d57f47055a04/bin/external/protobuf~/_objs/protoc_lib/0/service.o(service.o)' and 'bazel-out/darwin_x86_64-opt-exec-ST-d57f47055a04/bin/external/protobuf~/_objs/protoc_lib/1/service.o(service.o)' /Applications/Xcode16.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: warning duplicate member name 'string_field.o' from 'bazel-out/darwin_x86_64-opt-exec-ST-d57f47055a04/bin/external/protobuf~/_objs/protoc_lib/0/string_field.o(string_field.o)' and 'bazel-out/darwin_x86_64-opt-exec-ST-d57f47055a04/bin/external/protobuf~/_objs/protoc_lib/1/string_field.o(string_field.o)' (14:43:34) INFO: From Linking external/protobuf~/libprotobuf.a [for tool]: /Applications/Xcode16.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: file: bazel-out/darwin_x86_64-opt-exec-ST-d57f47055a04/bin/external/protobuf~/_objs/protobuf/error_listener.o has no symbols (14:43:34) INFO: From Linking external/protobuf~/protoc [for tool]: ld: warning: ignoring duplicate libraries: '-lm', '-lpthread' (14:43:35) INFO: From GoCompilePkg external/gazelle~~go_deps~com_github_fsnotify_fsevents/fsevents.a: cgo-gcc-prolog:452:2: warning: 'FSEventStreamScheduleWithRunLoop' is deprecated: first deprecated in macOS 13.0 - Use FSEventStreamSetDispatchQueue instead. [-Wdeprecated-declarations] 452 | FSEventStreamScheduleWithRunLoop(_cgo_a->p0, _cgo_a->p1, _cgo_a->p2); | ^ /Applications/Xcode16.2.0.16C5032a.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/FSEvents.framework/Headers/FSEvents.h:1154:1: note: 'FSEventStreamScheduleWithRunLoop' has been explicitly marked deprecated here 1154 | FSEventStreamScheduleWithRunLoop( | ^ 1 warning generated. (14:43:38) INFO: From GoLink internal/ibazel/fswatcher/fsevents/fsevents_test_/fsevents_test: ld: warning: -no_pie is deprecated when targeting new OS versions (14:43:44) INFO: From GoLink cmd/ibazel/ibazel_/ibazel: ld: warning: -no_pie is deprecated when targeting new OS versions (14:43:44) INFO: From GoLink internal/e2e/live_reload/live_reload_test_/live_reload_test: ld: warning: -no_pie is deprecated when targeting new OS versions (14:43:44) INFO: From GoLink cmd/ibazel/ibazel_test_/ibazel_test: ld: warning: -no_pie is deprecated when targeting new OS versions (14:43:44) INFO: From GoLink internal/e2e/profiler/profiler_test_/profiler_test: ld: warning: -no_pie is deprecated when targeting new OS versions (14:43:48) INFO: From GoLink internal/ibazel/ibazel_test_/ibazel_test: ld: warning: -no_pie is deprecated when targeting new OS versions (14:43:48) INFO: Found 57 targets... (14:43:48) INFO: Elapsed time: 35.335s, Critical Path: 11.64s (14:43:48) INFO: 440 processes: 355 remote cache hit, 5 internal, 5 darwin-sandbox, 75 local. (14:43:48) INFO: Build completed successfully, 440 total actions curl -sSL https://github.com/bazelbuild/continuous-integration/releases/download/agent-0.2.7/bazelci-agent-0.2.7-x86_64-apple-darwin -o /tmp/tmpi0zq9sxn/bazelci-agent :bazel: Computing flags for test step :bazel: Test (7.4.1) bazel --host_jvm_args=-Djava.net.preferIPv6Addresses=true test --flaky_test_attempts=3 --build_tests_only --local_test_jobs=4 --show_progress_rate_limit=5 --curses=yes --color=yes --terminal_columns=143 --show_timestamps --verbose_failures --jobs=8 --announce_rc --experimental_repository_cache_hardlinks --disk_cache= --sandbox_writable_path=/var/tmp/_bazel_buildkite/cache/repos/v1 --test_env=REPOSITORY_CACHE=/var/tmp/_bazel_buildkite/cache/repos/v1 --experimental_build_event_json_file_path_conversion=false --build_event_json_file=/tmp/tmpi0zq9sxn/test_bep.json --jvmopt=-Djava.net.preferIPv6Addresses --remote_cache=https://storage.googleapis.com/bazel-untrusted-build-cache --noexperimental_remote_cache_async --google_default_credentials --remote_timeout=3600 --remote_max_connections=200 --remote_default_platform_properties=properties:{name:"cache-silo-key" value:"3574fb10d429020863d5814b32cd1c7032ec1c87e49e968debe1bb45530739b5"} --remote_download_toplevel --@io_bazel_rules_go//go/config:race --test_tag_filters=-nomacos --test_env=IBAZEL_USE_LEGACY_WATCHER=1 --test_env=HOME --test_env=BAZELISK_USER_AGENT --test_env=COURSIER_OPTS --test_env=JAVA_TOOL_OPTIONS --test_env=SSL_CERT_FILE --sandbox_writable_path=/Users/buildkite/Library/Caches/bazelisk -- ... -//internal/e2e/stream:stream_test (14:43:49) WARNING: Option 'experimental_build_event_json_file_path_conversion' is deprecated: Use --build_event_json_file_path_conversion instead (14:43:49) WARNING: Option 'remote_default_platform_properties' is deprecated: --remote_default_platform_properties has been deprecated in favor of --remote_default_exec_properties. (14:43:49) INFO: Invocation ID: 7c5ae241-cb69-4ac9-92d1-f0a096a3e92e (14:43:49) INFO: Reading 'startup' options from /private/etc/bazel.bazelrc: --host_jvm_args=-Djava.net.preferIPv6Addresses=true (14:43:49) INFO: Options provided by the client: Inherited 'common' options: --isatty=1 --terminal_columns=160 (14:43:49) INFO: Reading rc options for 'test' from /private/etc/bazel.bazelrc: Inherited 'common' options: --jvmopt=-Djava.net.preferIPv6Addresses (14:43:49) INFO: Reading rc options for 'test' from /Users/buildkite/builds/bk-macos-intel-i58z/bazel/bazel-watcher/.bazelrc: Inherited 'build' options: --workspace_status_command=python3 tools/workplace_status.py (14:43:49) INFO: Current date is 2025-04-10 (14:43:49) WARNING: Option 'experimental_build_event_json_file_path_conversion' is deprecated: Use --build_event_json_file_path_conversion instead (14:43:49) WARNING: Option 'remote_default_platform_properties' is deprecated: --remote_default_platform_properties has been deprecated in favor of --remote_default_exec_properties. (14:43:49) WARNING: Build options --@@rules_go~//go/config:race and --test_env have changed, discarding analysis cache (this can be expensive, see https://bazel.build/advanced/performance/iteration-speed). (14:43:52) INFO: Analyzed 23 targets (180 packages loaded, 13359 targets configured). (14:43:54) INFO: From Compiling src/google/protobuf/generated_message_tctable_lite.cc [for tool]: external/protobuf~/src/google/protobuf/generated_message_tctable_lite.cc:347:14: warning: unused function 'Offset' [-Wunused-function] 347 | inline void* Offset(void* base, uint32_t offset) { | ^~~~~~ 1 warning generated. (14:43:57) INFO: From Compiling src/google/protobuf/compiler/cpp/helpers.cc [for tool]: external/protobuf~/src/google/protobuf/compiler/cpp/helpers.cc:197:25: warning: unused function 'VerifyInt32TypeToVerifyCustom' [-Wunused-function] 197 | inline VerifySimpleType VerifyInt32TypeToVerifyCustom(VerifyInt32Type t) { | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1 warning generated. (14:43:58) INFO: From Compiling src/google/protobuf/stubs/strutil.cc [for tool]: external/protobuf~/src/google/protobuf/stubs/strutil.cc:506:11: warning: 'sprintf' is deprecated: This function is provided for compatibility reasons only. Due to security concerns inherent in the design of sprintf(3), it is highly recommended that you use snprintf(3) instead. [-Wdeprecated-declarations] 506 | sprintf(dest + used, (use_hex ? "\\x%02x" : "\\%03o"), | ^ /Applications/Xcode16.2.0.16C5032a.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/_stdio.h:274:1: note: 'sprintf' has been explicitly marked deprecated here 274 | __deprecated_msg("This function is provided for compatibility reasons only. Due to security concerns inherent in the design of sprintf(3), it is highly recommended that you use snprintf(3) instead.") | ^ /Applications/Xcode16.2.0.16C5032a.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/sys/cdefs.h:218:48: note: expanded from macro '__deprecated_msg' 218 | #define __deprecated_msg(_msg) __attribute__((__deprecated__(_msg))) | ^ 1 warning generated. (14:43:59) INFO: From Linking external/protobuf~/libprotobuf_lite.a [for tool]: /Applications/Xcode16.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: file: bazel-out/darwin_x86_64-opt-exec-ST-d14722c66e92/bin/external/protobuf~/_objs/protobuf_lite/io_win32.o has no symbols (14:43:59) INFO: From GoCompilePkg external/gazelle~~go_deps~com_github_fsnotify_fsevents/fsevents.a: cgo-gcc-prolog:452:2: warning: 'FSEventStreamScheduleWithRunLoop' is deprecated: first deprecated in macOS 13.0 - Use FSEventStreamSetDispatchQueue instead. [-Wdeprecated-declarations] 452 | FSEventStreamScheduleWithRunLoop(_cgo_a->p0, _cgo_a->p1, _cgo_a->p2); | ^ /Applications/Xcode16.2.0.16C5032a.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/FSEvents.framework/Headers/FSEvents.h:1154:1: note: 'FSEventStreamScheduleWithRunLoop' has been explicitly marked deprecated here 1154 | FSEventStreamScheduleWithRunLoop( | ^ 1 warning generated. (14:44:00) INFO: From GoLink internal/ibazel/fswatcher/fsevents/fsevents_test_/fsevents_test: ld: warning: -no_pie is deprecated when targeting new OS versions (14:44:00) INFO: From GoLink internal/ibazel/log/log_test_/log_test: ld: warning: -no_pie is deprecated when targeting new OS versions (14:44:00) INFO: From GoLink internal/ibazel/fswatcher/fsnotify/fsnotify_test_/fsnotify_test: ld: warning: -no_pie is deprecated when targeting new OS versions (14:44:00) INFO: From GoLink internal/ibazel/workspace/workspace_test_/workspace_test: ld: warning: -no_pie is deprecated when targeting new OS versions (14:44:03) INFO: From GoLink internal/e2e/override_repository/override_repository_test_/override_repository_test: ld: warning: -no_pie is deprecated when targeting new OS versions (14:44:03) INFO: From GoLink internal/e2e/output_runner/output_runner_test_/output_runner_test: ld: warning: -no_pie is deprecated when targeting new OS versions (14:44:03) INFO: From GoLink internal/e2e/many_dirs/many_dirs_/many_dirs: ld: warning: -no_pie is deprecated when targeting new OS versions (14:44:03) INFO: From GoLink internal/e2e/many_dirs/many_dirs_test_/many_dirs_test: ld: warning: -no_pie is deprecated when targeting new OS versions (14:44:05) INFO: From GoLink internal/e2e/symlink_dir/symlink_dir_test_/symlink_dir_test: ld: warning: -no_pie is deprecated when targeting new OS versions (14:44:06) INFO: From GoLink internal/e2e/termination/termination_test_/termination_test: ld: warning: -no_pie is deprecated when targeting new OS versions (14:44:06) INFO: From GoLink internal/e2e/profiler/profiler_test_/profiler_test: ld: warning: -no_pie is deprecated when targeting new OS versions (14:44:06) INFO: From Linking external/protobuf~/libprotoc_lib.a [for tool]: /Applications/Xcode16.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: warning duplicate member name 'enum.o' from 'bazel-out/darwin_x86_64-opt-exec-ST-d14722c66e92/bin/external/protobuf~/_objs/protoc_lib/0/enum.o(enum.o)' and 'bazel-out/darwin_x86_64-opt-exec-ST-d14722c66e92/bin/external/protobuf~/_objs/protoc_lib/1/enum.o(enum.o)' /Applications/Xcode16.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: warning duplicate member name 'enum_field.o' from 'bazel-out/darwin_x86_64-opt-exec-ST-d14722c66e92/bin/external/protobuf~/_objs/protoc_lib/1/enum_field.o(enum_field.o)' and 'bazel-out/darwin_x86_64-opt-exec-ST-d14722c66e92/bin/external/protobuf~/_objs/protoc_lib/0/enum_field.o(enum_field.o)' /Applications/Xcode16.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: warning duplicate member name 'extension.o' from 'bazel-out/darwin_x86_64-opt-exec-ST-d14722c66e92/bin/external/protobuf~/_objs/protoc_lib/0/extension.o(extension.o)' and 'bazel-out/darwin_x86_64-opt-exec-ST-d14722c66e92/bin/external/protobuf~/_objs/protoc_lib/1/extension.o(extension.o)' /Applications/Xcode16.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: warning duplicate member name 'field.o' from 'bazel-out/darwin_x86_64-opt-exec-ST-d14722c66e92/bin/external/protobuf~/_objs/protoc_lib/1/field.o(field.o)' and 'bazel-out/darwin_x86_64-opt-exec-ST-d14722c66e92/bin/external/protobuf~/_objs/protoc_lib/0/field.o(field.o)' /Applications/Xcode16.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: warning duplicate member name 'file.o' from 'bazel-out/darwin_x86_64-opt-exec-ST-d14722c66e92/bin/external/protobuf~/_objs/protoc_lib/1/file.o(file.o)' and 'bazel-out/darwin_x86_64-opt-exec-ST-d14722c66e92/bin/external/protobuf~/_objs/protoc_lib/0/file.o(file.o)' /Applications/Xcode16.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: warning duplicate member name 'generator.o' from 'bazel-out/darwin_x86_64-opt-exec-ST-d14722c66e92/bin/external/protobuf~/_objs/protoc_lib/0/generator.o(generator.o)' and 'bazel-out/darwin_x86_64-opt-exec-ST-d14722c66e92/bin/external/protobuf~/_objs/protoc_lib/1/generator.o(generator.o)' /Applications/Xcode16.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: warning duplicate member name 'generator.o' from 'bazel-out/darwin_x86_64-opt-exec-ST-d14722c66e92/bin/external/protobuf~/_objs/protoc_lib/1/generator.o(generator.o)' and 'bazel-out/darwin_x86_64-opt-exec-ST-d14722c66e92/bin/external/protobuf~/_objs/protoc_lib/2/generator.o(generator.o)' /Applications/Xcode16.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: warning duplicate member name 'helpers.o' from 'bazel-out/darwin_x86_64-opt-exec-ST-d14722c66e92/bin/external/protobuf~/_objs/protoc_lib/2/helpers.o(helpers.o)' and 'bazel-out/darwin_x86_64-opt-exec-ST-d14722c66e92/bin/external/protobuf~/_objs/protoc_lib/0/helpers.o(helpers.o)' /Applications/Xcode16.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: warning duplicate member name 'helpers.o' from 'bazel-out/darwin_x86_64-opt-exec-ST-d14722c66e92/bin/external/protobuf~/_objs/protoc_lib/0/helpers.o(helpers.o)' and 'bazel-out/darwin_x86_64-opt-exec-ST-d14722c66e92/bin/external/protobuf~/_objs/protoc_lib/1/helpers.o(helpers.o)' /Applications/Xcode16.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: warning duplicate member name 'map_field.o' from 'bazel-out/darwin_x86_64-opt-exec-ST-d14722c66e92/bin/external/protobuf~/_objs/protoc_lib/0/map_field.o(map_field.o)' and 'bazel-out/darwin_x86_64-opt-exec-ST-d14722c66e92/bin/external/protobuf~/_objs/protoc_lib/1/map_field.o(map_field.o)' /Applications/Xcode16.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: warning duplicate member name 'message.o' from 'bazel-out/darwin_x86_64-opt-exec-ST-d14722c66e92/bin/external/protobuf~/_objs/protoc_lib/1/message.o(message.o)' and 'bazel-out/darwin_x86_64-opt-exec-ST-d14722c66e92/bin/external/protobuf~/_objs/protoc_lib/0/message.o(message.o)' /Applications/Xcode16.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: warning duplicate member name 'message_field.o' from 'bazel-out/darwin_x86_64-opt-exec-ST-d14722c66e92/bin/external/protobuf~/_objs/protoc_lib/0/message_field.o(message_field.o)' and 'bazel-out/darwin_x86_64-opt-exec-ST-d14722c66e92/bin/external/protobuf~/_objs/protoc_lib/1/message_field.o(message_field.o)' /Applications/Xcode16.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: warning duplicate member name 'primitive_field.o' from 'bazel-out/darwin_x86_64-opt-exec-ST-d14722c66e92/bin/external/protobuf~/_objs/protoc_lib/1/primitive_field.o(primitive_field.o)' and 'bazel-out/darwin_x86_64-opt-exec-ST-d14722c66e92/bin/external/protobuf~/_objs/protoc_lib/0/primitive_field.o(primitive_field.o)' /Applications/Xcode16.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: warning duplicate member name 'service.o' from 'bazel-out/darwin_x86_64-opt-exec-ST-d14722c66e92/bin/external/protobuf~/_objs/protoc_lib/0/service.o(service.o)' and 'bazel-out/darwin_x86_64-opt-exec-ST-d14722c66e92/bin/external/protobuf~/_objs/protoc_lib/1/service.o(service.o)' /Applications/Xcode16.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: warning duplicate member name 'string_field.o' from 'bazel-out/darwin_x86_64-opt-exec-ST-d14722c66e92/bin/external/protobuf~/_objs/protoc_lib/0/string_field.o(string_field.o)' and 'bazel-out/darwin_x86_64-opt-exec-ST-d14722c66e92/bin/external/protobuf~/_objs/protoc_lib/1/string_field.o(string_field.o)' (14:44:06) INFO: From GoLink internal/e2e/live_reload/live_reload_test_/live_reload_test: ld: warning: -no_pie is deprecated when targeting new OS versions (14:44:06) INFO: From Linking external/protobuf~/libprotobuf.a [for tool]: /Applications/Xcode16.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: file: bazel-out/darwin_x86_64-opt-exec-ST-d14722c66e92/bin/external/protobuf~/_objs/protobuf/error_listener.o has no symbols (14:44:06) INFO: From Linking external/protobuf~/protoc [for tool]: ld: warning: ignoring duplicate libraries: '-lm', '-lpthread' (14:44:07) INFO: From GoLink internal/ibazel/output_runner/output_runner_test_/output_runner_test: ld: warning: -no_pie is deprecated when targeting new OS versions (14:44:08) INFO: From GoLink internal/bazel/bazel_test_/bazel_test: ld: warning: -no_pie is deprecated when targeting new OS versions (14:44:08) INFO: From GoLink cmd/ibazel/ibazel_test_/ibazel_test: ld: warning: -no_pie is deprecated when targeting new OS versions (14:44:08) INFO: From GoLink cmd/ibazel/ibazel_/ibazel: ld: warning: -no_pie is deprecated when targeting new OS versions (14:44:09) INFO: From GoLink internal/e2e/incompatible_by_default/incompatible_by_default_test_/incompatible_by_default_test: ld: warning: -no_pie is deprecated when targeting new OS versions (14:44:09) INFO: From GoLink internal/ibazel/command/command_test_/command_test: ld: warning: -no_pie is deprecated when targeting new OS versions (14:44:09) INFO: From GoLink internal/e2e/simple/simple_test_/simple_test: ld: warning: -no_pie is deprecated when targeting new OS versions (14:44:09) INFO: From GoLink internal/e2e/local_repository/local_repository_test_/local_repository_test: ld: warning: -no_pie is deprecated when targeting new OS versions (14:44:09) INFO: From GoLink internal/e2e/lifecycle_hooks/lifecycle_hooks_test_/lifecycle_hooks_test: ld: warning: -no_pie is deprecated when targeting new OS versions (14:44:09) INFO: From GoLink internal/e2e/vim/vim_test_/vim_test: ld: warning: -no_pie is deprecated when targeting new OS versions (14:44:09) INFO: From GoLink internal/e2e/error/error_test_/error_test: ld: warning: -no_pie is deprecated when targeting new OS versions (14:44:13) INFO: From GoLink internal/ibazel/ibazel_test_/ibazel_test: ld: warning: -no_pie is deprecated when targeting new OS versions (14:44:19) FAIL: //internal/ibazel:ibazel_test (see /private/var/tmp/_bazel_buildkite/781631121efdd6b88434574fe386ed52/execroot/_main/bazel-out/darwin_x86_64-fastbuild/testlogs/internal/ibazel/ibazel_test/test_attempts/attempt_1.log) (14:44:19) [527 / 528] 8 / 23 tests; 1 action; last test: //internal/ibazel/command:command_test Testing //internal/ibazel:ibazel_test; 5s remote-cache, darwin-sandbox buildkite-agent artifact upload --content-type text/plain;charset=utf-8 internal/ibazel/ibazel_test/test_attempts/attempt_1.log 2025-04-10 14:44:20 INFO Found 1 files that match "internal/ibazel/ibazel_test/test_attempts/attempt_1.log" 2025-04-10 14:44:20 INFO Uploading to Google Cloud Storage ("gs://bazel-untrusted-buildkite-artifacts/01961fb9-831a-4744-8a62-12a198ba25a3"), using your agent configuration 2025-04-10 14:44:20 INFO Creating (0-1)/1 artifacts (14:44:25) FAIL: //internal/ibazel:ibazel_test (see /private/var/tmp/_bazel_buildkite/781631121efdd6b88434574fe386ed52/execroot/_main/bazel-out/darwin_x86_64-fastbuild/testlogs/internal/ibazel/ibazel_test/test_attempts/attempt_2.log) (14:44:25) [527 / 528] 8 / 23 tests; 1 action; last test: //internal/ibazel/command:command_test Testing //internal/ibazel:ibazel_test; 11s remote-cache, darwin-sandbox buildkite-agent artifact upload --content-type text/plain;charset=utf-8 internal/ibazel/ibazel_test/test_attempts/attempt_2.log 2025-04-10 14:44:26 INFO Found 1 files that match "internal/ibazel/ibazel_test/test_attempts/attempt_2.log" 2025-04-10 14:44:26 INFO Uploading to Google Cloud Storage ("gs://bazel-untrusted-buildkite-artifacts/01961fb9-831a-4744-8a62-12a198ba25a3"), using your agent configuration 2025-04-10 14:44:26 INFO Creating (0-1)/1 artifacts (14:44:30) FAIL: //internal/ibazel:ibazel_test (see /private/var/tmp/_bazel_buildkite/781631121efdd6b88434574fe386ed52/execroot/_main/bazel-out/darwin_x86_64-fastbuild/testlogs/internal/ibazel/ibazel_test/test.log) FAILED: //internal/ibazel:ibazel_test (Summary) /private/var/tmp/_bazel_buildkite/781631121efdd6b88434574fe386ed52/execroot/_main/bazel-out/darwin_x86_64-fastbuild/testlogs/internal/ibazel/ibazel_test/test.log /private/var/tmp/_bazel_buildkite/781631121efdd6b88434574fe386ed52/execroot/_main/bazel-out/darwin_x86_64-fastbuild/testlogs/internal/ibazel/ibazel_test/test_attempts/attempt_1.log /private/var/tmp/_bazel_buildkite/781631121efdd6b88434574fe386ed52/execroot/_main/bazel-out/darwin_x86_64-fastbuild/testlogs/internal/ibazel/ibazel_test/test_attempts/attempt_2.log (14:44:30) [527 / 528] 9 / 23 tests, 1 failed; 1 action; last test: //internal/ibazel:ibazel_test Testing //internal/ibazel:ibazel_test; 16s remote-cache, darwin-sandbox buildkite-agent artifact upload --content-type text/plain;charset=utf-8 internal/ibazel/ibazel_test/test.log 2025-04-10 14:44:31 INFO Found 1 files that match "internal/ibazel/ibazel_test/test.log" 2025-04-10 14:44:31 INFO Uploading to Google Cloud Storage ("gs://bazel-untrusted-buildkite-artifacts/01961fb9-831a-4744-8a62-12a198ba25a3"), using your agent configuration 2025-04-10 14:44:31 INFO Creating (0-1)/1 artifacts (14:46:42) INFO: Found 23 test targets... (14:46:43) INFO: Elapsed time: 173.863s, Critical Path: 48.51s (14:46:43) INFO: 434 processes: 348 remote cache hit, 3 internal, 8 darwin-sandbox, 75 local. (14:46:43) INFO: Build completed, 1 test FAILED, 434 total actions //cmd/ibazel:ibazel_test (cached) PASSED in 2.3s //internal/bazel:bazel_test (cached) PASSED in 2.3s //internal/e2e/error:error_test (cached) PASSED in 37.3s //internal/e2e/lifecycle_hooks:lifecycle_hooks_test (cached) PASSED in 38.7s //internal/e2e/live_reload:live_reload_test (cached) PASSED in 37.2s //internal/e2e/output_runner:output_runner_test (cached) PASSED in 100.8s //internal/e2e/override_repository:override_repository_test (cached) PASSED in 36.8s //internal/e2e/simple:simple_test (cached) PASSED in 148.9s //internal/e2e/symlink_dir:symlink_dir_test (cached) PASSED in 37.9s //internal/e2e/termination:termination_test (cached) PASSED in 193.5s //internal/e2e/vim:vim_test (cached) PASSED in 37.9s //internal/ibazel/command:command_test (cached) PASSED in 8.6s //internal/ibazel/fswatcher/fsevents:fsevents_test (cached) PASSED in 2.3s //internal/ibazel/fswatcher/fsnotify:fsnotify_test (cached) PASSED in 2.3s //internal/ibazel/log:log_test (cached) PASSED in 2.3s //internal/ibazel/output_runner:output_runner_test (cached) PASSED in 2.5s //internal/ibazel/workspace:workspace_test (cached) PASSED in 2.3s //internal/e2e/incompatible_by_default:incompatible_by_default_test PASSED in 43.2s //internal/e2e/local_repository:local_repository_test PASSED in 38.9s //internal/e2e/many_dirs:many_dirs PASSED in 5.2s //internal/e2e/many_dirs:many_dirs_test PASSED in 5.2s //internal/e2e/profiler:profiler_test PASSED in 39.7s //internal/ibazel:ibazel_test FAILED in 3 out of 3 in 5.5s Stats over 3 runs: max = 5.5s, min = 5.5s, avg = 5.5s, dev = 0.0s /private/var/tmp/_bazel_buildkite/781631121efdd6b88434574fe386ed52/execroot/_main/bazel-out/darwin_x86_64-fastbuild/testlogs/internal/ibazel/ibazel_test/test.log /private/var/tmp/_bazel_buildkite/781631121efdd6b88434574fe386ed52/execroot/_main/bazel-out/darwin_x86_64-fastbuild/testlogs/internal/ibazel/ibazel_test/test_attempts/attempt_1.log /private/var/tmp/_bazel_buildkite/781631121efdd6b88434574fe386ed52/execroot/_main/bazel-out/darwin_x86_64-fastbuild/testlogs/internal/ibazel/ibazel_test/test_attempts/attempt_2.log Executed 6 out of 23 tests: 22 tests pass and 1 fails locally. There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are. (14:46:43) INFO: Build Event Protocol files produced successfully. buildkite-agent artifact upload /tmp/tmpi0zq9sxn/test_bep.json 2025-04-10 14:46:43 INFO Found 1 files that match "/tmp/tmpi0zq9sxn/test_bep.json" 2025-04-10 14:46:43 INFO Uploading to Google Cloud Storage ("gs://bazel-untrusted-buildkite-artifacts/01961fb9-831a-4744-8a62-12a198ba25a3"), using your agent configuration 2025-04-10 14:46:43 INFO Creating (0-1)/1 artifacts 2025-04-10 14:46:43 INFO Uploading 01961fbd-06bd-487f-b172-7df44e7a434a tmp/tmpi0zq9sxn/test_bep.json (269 KiB) 2025-04-10 14:46:44 INFO Artifact uploads completed successfully bazel test failed with exit code 3 🚨 Error: The command exited with status 3 user command error: exit status 3

Output ibazel_test/test.log

exec ${PAGER:-/usr/bin/less} "$0" || exit 1 Executing tests from //internal/ibazel:ibazel_test ----------------------------------------------------------------------------- ================== WARNING: DATA RACE Write at 0x00c000120790 by goroutine 15: runtime.closechan() GOROOT/src/runtime/chan.go:357 +0x0 github.com/fsnotify/fsnotify.(*Watcher).readEvents.func1() external/gazelle~~go_deps~com_github_fsnotify_fsnotify/backend_kqueue.go:539 +0x3d runtime.deferreturn() GOROOT/src/runtime/panic.go:477 +0x30 github.com/fsnotify/fsnotify.NewBufferedWatcher.func1() external/gazelle~~go_deps~com_github_fsnotify_fsnotify/backend_kqueue.go:184 +0x33 Previous read at 0x00c000120790 by goroutine 17: runtime.chansend() GOROOT/src/runtime/chan.go:160 +0x0 github.com/bazelbuild/bazel-watcher/internal/ibazel.TestIBazelLoop.func5() internal/ibazel/ibazel_test.go:254 +0x88 Goroutine 15 (running) created at: github.com/fsnotify/fsnotify.NewBufferedWatcher() external/gazelle~~go_deps~com_github_fsnotify_fsnotify/backend_kqueue.go:184 +0x42d github.com/fsnotify/fsnotify.NewWatcher() external/gazelle~~go_deps~com_github_fsnotify_fsnotify/backend_kqueue.go:153 +0x24 github.com/bazelbuild/bazel-watcher/internal/ibazel/fswatcher/fsnotify.NewWatcher() internal/ibazel/fswatcher/fsnotify/fsnotify.go:97 +0x1c github.com/bazelbuild/bazel-watcher/internal/ibazel/fswatcher.NewWatcher() internal/ibazel/fswatcher/factory_darwin.go:31 +0x44 github.com/bazelbuild/bazel-watcher/internal/ibazel.(*IBazel).setup() internal/ibazel/ibazel.go:242 +0xad github.com/bazelbuild/bazel-watcher/internal/ibazel.New() internal/ibazel/ibazel.go:91 +0x9c github.com/bazelbuild/bazel-watcher/internal/ibazel.newIBazel() internal/ibazel/ibazel_test.go:173 +0x144 github.com/bazelbuild/bazel-watcher/internal/ibazel.TestIBazelLoop() internal/ibazel/ibazel_test.go:198 +0xa6 testing.tRunner() GOROOT/src/testing/testing.go:1595 +0x238 testing.(*T).Run.func1() GOROOT/src/testing/testing.go:1648 +0x44 Goroutine 17 (finished) created at: github.com/bazelbuild/bazel-watcher/internal/ibazel.TestIBazelLoop() internal/ibazel/ibazel_test.go:254 +0x875 testing.tRunner() GOROOT/src/testing/testing.go:1595 +0x238 testing.(*T).Run.func1() GOROOT/src/testing/testing.go:1648 +0x44 ================== FAIL

how do you want to move forward?

chickenandpork · 2025-04-08T17:41:28Z

Hey @janwinkler1 I've been looking, and it's misleading that the tests are green for this PR -- I can't see the failure for myself! I't been kinda stumping me until I realized that you had indeed skipped some CI.

I've created one PR to your PR, but it addresses a failure I saw, not the one I think you're seeing in CI, so I do need to repro on CI.

I'm planning to cut a duplicate PR, but only to exercise the CI: you'll see one with YOUR CHANGES, but no intent to ultimately merge unless this PR merges and we still see beneficial content rebasing on your change.

Are you OK with that?

janwinkler1 · 2025-04-09T06:42:37Z

@chickenandpork yes i am ok with that!

sorry to make you wait, i am a bit underwater with other projects ATM and can usually focus fridays and saturdays

chickenandpork · 2025-04-09T09:55:15Z

@janwinkler1 it’s all good — I was able to pull your branch and layer on top without smudging hashes or losing history.

getting some progress: seems to be mostly timers and timeouts :-/

janwinkler1 · 2025-04-09T10:37:42Z

@chickenandpork i just saw that you increased the timers, so according to the requested changes of @achew22 we now "only" would need to get override_repository_test passing, right?

sidenote regarding janwinkler1#1, i saw that in https://github.com/bazelbuild/bazel-watcher/pull/723/files you did not use that specific change for thirdparty, therefore i am currently unsure whether i should merge and rebase or leave it for now? just so that i have a clear path on how to pursue if i get time later today / this week

chickenandpork · 2025-04-09T17:55:52Z

@chickenandpork i just saw that you increased the timers, so according to the requested changes of @achew22 we now "only" would need to get override_repository_test passing, right?

I'll have to check, but likely this morning there's just one.

regarding janwinkler1#1, i saw that in https://github.com/bazelbuild/bazel-watcher/pull/723/files you did not use that specific change for thirdparty

Might not need it: I needed it for your change on M1 Mac to compile based on the default/typical toolchains on a darwin/arm64. I haven't tried changing toolchain versions.

Moving to bzlmod/BCR might side-step the issue. We might get a patched or slightly different dependency, or more easily shift to a newer. If we can avoid it, then great: happy to let it slide off and avoid maintaining unnecessary code. Contrariwise, since your PR would need to compile, you can merge it, only to drop it off later if we don't need it.

Either case works out if we don't need it; only the second case works out if we do need it, but all things will merge the right way.

Let's see what the gap looks like right now.

chickenandpork · 2025-04-09T20:05:13Z

@janwinkler1 #723 is green and based off this PR; rebasing and squashing to clean up. I'll see if I can PR that to your PR branch in your repo.

chickenandpork · 2025-04-09T22:43:10Z

@achew22 can we accept/merge this PR in current state, and fast-follow my #723 ? The fix is available, would merge after this, and our exposure to fewer testing is limited in time.

chickenandpork · 2025-04-10T16:11:33Z

With the skip logic removed (internal/ibazel/ibazel_test.go@188 similar to 8e642ab), testing on a macOS/amd64 and macOS/arm64, I don't see the error repro.

Maybe there's other changes and customizations in the buildenv that are not defined in the bazel-watcher repo (hidden/remote env config is a source of stress)

My test command on both arch combinations is:

bazel test --announce_rc --curses=no --color=no --test_tag_filters=-nomacos --nocache_test_results \
  --test_env=HOME --test_output=all \
  --test_env=BAZELISK_USER_AGENT --test_env=COURSIER_OPTS --test_env=JAVA_TOOL_OPTIONS \
  --test_env=SSL_CERT_FILE \
  //internal/ibazel:ibazel_test

yes, I'm referencing env variables that are themselves undefined... maybe the env skew is there?

No difference with this: success on amd64 and arm64.

bazel test --announce_rc --curses=no --color=no --test_tag_filters=-nomacos --nocache_test_results \
  --test_env=HOME --test_output=all \
  //internal/ibazel:ibazel_test

I do see that BuildKite is using Xcode-16.2; I've got Xcode-16.3 (16E140) on both systems. I'll see if I can downgrade this afternoon (now+4 hours), even though that shouldn't affect (since the Go toolchain is self-contained, right?)

janwinkler1 · 2025-04-10T18:08:10Z

@chickenandpork thank you for your analysis!

so on the one hand thats good. because that race condition only occurs in CI, and we know that its not the case when you run the tests on your machines.. on the other hand thats also bad because CI still fails.

i will also take another look if i can find something. however, tbh i do not think that xcode version would make a difference but maybe lets try anyways?

i noticed that the test in CI runs with

--@io_bazel_rules_go//go/config:race, is there some flag to do that locally too? (bazel-contrib/rules_go#635)

janwinkler1 · 2025-04-11T08:46:00Z

update @achew22 and @chickenandpork:

i removed all the skipped tests as a result of collaborating with @chickenandpork and finding out that the increasing the timeouts (conservatively) was an easy fix
@chickenandpork i hope this is ok for you bc i just copied your idea into this PR in case @achew22 feels uneasy merging one PR where we disable tests
regarding the ibazel_test: i found out that the data race in CI was caused by a conflict between the test sending events to the real source watcher's channel and test cleanup closing that same channel. therefore, I chose to fix it by mocking both file watchers in this test, which prevents the race and properly isolates the test logic
regarding output runner test fix: as master progressed with renovate updates, especially for rules_go, i had to make some changes in output runner test. specifically, the jump from rules_go to 0.53 (from what i originally worked with 0.43) with significant changes involved. namely stderr outputs from genrules are no longer being captured or displayed in the same way as before. more precisely I couldn't find the "runacommand" strings at all - they simply weren't in the output being processed by the output runner. therefore I've taken the approach of updating the regex patterns to match reliable parts of the output that we know will be present - such as "INFO: Build completed successfully" or "INFO: Found 1 target".

PTAL

chickenandpork · 2025-04-14T18:26:08Z

FWIW, I think I fixed the race condition without mocking, but with specific updates that RenovateBot would ultimately be making.

It'd be awesome if my authorship on those few changes involving timeouts and other macroscopic changes was retained: I like to think it's like getting credit.

Go ahead and use my work if it helps :-/

I could pre-merge the changes to dependencies that (I think) eliminates the race condition. My goal is: what would allow this PR to merge so we can move forward.

I think in future we need to focus on what makes commits easier and faster to review -- likely easier past these major bumps.

janwinkler1 · 2025-04-15T13:50:37Z

@chickenandpork agreed.

i am open to split this PR apart into multiples to make the changes more atomic / simpler to review where ever possible.

TBH i do not feel comfortable to just use your work. no prob for me to leave this PR as it was before you opened #723, i was just focusing on keeping that thing running and wasnt sure whether @achew22 was comfortable merging something that actively disables test (see #712 (comment))

maybe we can come up with a simple upgrade plan with the single changes needed and then go from there?

i'd propose something like this:

keep chore: upgrade bazel to 7.4.1 #709: upgrading bazel version but not yet enabling bzlmod (however, I think this will only pass if we increase timeouts, e.g your work of increasing timeouts would be included there)
your new PR that fixes the race condition
stack simplified changes from this PR (chore: switch to bzlmod #712) on top of that with enabling bzlmod.

WDYT?

and sorry for getting ahead of myself.

chickenandpork · 2025-04-17T16:30:04Z

@chickenandpork agreed.

i am open to split this PR apart into multiples to make the changes more atomic / simpler to review where ever possible.

TBH i do not feel comfortable to just use your work. no prob for me to leave this PR as it was before you opened #723, i was just focusing on keeping that thing running and wasnt sure whether @achew22 was comfortable merging something that actively disables test (see #712 (comment))

maybe we can come up with a simple upgrade plan with the single changes needed and then go from there?

i'd propose something like this:

keep chore: upgrade bazel to 7.4.1 #709: upgrading bazel version but not yet enabling bzlmod (however, I think this will only pass if we increase timeouts, e.g your work of increasing timeouts would be included there)

your new PR that fixes the race condition

stack simplified changes from this PR (chore: switch to bzlmod #712) on top of that with enabling bzlmod.

WDYT?

I think we're near agreement, but it depends on @achew22 as well.

I think bazel-7.4.1 (709) is needed for bzlmod (712); I'm willing to take the obligation of unbreaking post-712 if we can those gains merged if @achew22 was OK with that since I think we've shown that it can pass.

In other orgs, I recall just pushing commits onto each other's PR branches (ie FB works that way: suggestion? just push a commit!). Outside that capability, a feature branch can collect PRs that are inter-dependent. A feature-branch with 709, 712, and 723 would likely pass. I recall a failure pushing commits to your PR branch on bazelbuild., which is why I was PR-ing your PR branch in your repo.

I could pre-push 723 and you could rebase on that if it lands, but the benefit of 723 by itself isn't obvious.

I can also pre-push the dependency changes needed to avoid the race-condition in macOS (macOS/arm64, I think) -- those should build. This chips-away at the things blocking you...

So: counter-plans:

if @achew22 will accept the risk, and merge 709, 712, at around the ba3235b / 0e51edb level, I will fast-follow 723 to fix macOS breakages (we've shown that it'll likely unbreak)
if that's too risky, it's the protective decision, we could merge a simplified 723, and that'll make 709/712 CI green; if not, pre-push the dependency fix vs race-condition, or your awesome mocking solution, or whatever else to unbreak.

option 2 is safer, and keeps the green status, gets some feel of progress, but it's really little bits laying out a runway for 709/712 to land.

and sorry for getting ahead of myself.

It's well-intended, it's all good. We seem to work in opposite timezones. I wish we could accept each other's PRs and move more quickly :) . I'm impatient :)

janwinkler1 · 2025-05-28T05:30:57Z

any update here @chickenandpork @achew22 ?

chickenandpork · 2025-05-30T08:35:55Z

This is greatly needed but delayed.

I would prefer this be rebased, the skip in #712 (comment) line 188 restored (and removal considered as a later parallel action), and let's reap the benefit of this.

FWIW, in trying to rework the various attempts, I'm getting a circular-dependency, so if this works as-is (or rebased) then it gets us fairly far along and may let us work in parallel.

chickenandpork · 2025-06-12T16:22:20Z

@janwinkler1 @achew22 I see that when these PRs linger for a while, they'll be closed as stale by a bot/automation.

janwinkler1 · 2025-06-12T19:06:22Z

@chickenandpork i will look into it beginning of next week. i am currently travelling.

achew22 · 2025-06-14T20:49:03Z

internal/e2e/output_runner/output_runner_test.go

-	}]`, sentinalFileName))
+      e2e.MustWriteFile(t, ".bazel_fix_commands.json", fmt.Sprintf(`
+      [{
+              "regex": "INFO: Build completed successfully",


I think since we're not matching on runacommand in any of these tests there is nothing that is validating that the output from a command (either in a .bzl print statement or from the executable itself) is actually parsed.

I think this lingers as my last concern before we can merge this.

@achew22 @janwinkler1 that missing “runacommand” was something I found myself fixing over and over. I may be able to look tonight (now+ 6-8 hrs from now) and get that permanently fixed.

@janwinkler1 if we're losing a unittest, I'll resurrect it (it's often been a timing issue so far -- that and a closed stdout/stderr blocking the output we harvest).

Please accept whatever change to merge this benefit and we'll build on that.

@achew22 & @chickenandpork
i tried matching on a runcommand for a bit today but without success, i hope i can make some more time this week.

janwinkler1 force-pushed the switch-to-bzlmod branch 7 times, most recently from cd84cfa to 47ebf70 Compare March 28, 2025 09:12

This was referenced Mar 28, 2025

chore: upgrade bazel to 7.4.1 #709

Open

Fix macos CI #715

Closed

janwinkler1 marked this pull request as ready for review March 28, 2025 10:16

janwinkler1 requested a review from achew22 as a code owner March 28, 2025 10:16

achew22 reviewed Mar 28, 2025

View reviewed changes

janwinkler1 requested a review from achew22 March 29, 2025 14:24

chickenandpork mentioned this pull request Apr 4, 2025

chore(deps): update dependency bazel to v8 #687

Open

janwinkler1 force-pushed the switch-to-bzlmod branch 3 times, most recently from 1a8a17f to 453569b Compare April 4, 2025 14:55

janwinkler1 changed the title ~~Switch to bzlmod and adapt tests~~ chore: switch to bzlmod Apr 4, 2025

janwinkler1 force-pushed the switch-to-bzlmod branch from 453569b to 8d024f5 Compare April 4, 2025 15:33

achew22 requested changes Apr 4, 2025

View reviewed changes

chickenandpork mentioned this pull request Apr 9, 2025

chore: Re-enable macOS tests #723

Closed

chickenandpork mentioned this pull request Apr 10, 2025

ibazel was not invoked from within a workspace #682

Closed

janwinkler1 requested a review from achew22 April 11, 2025 08:46

janwinkler1 added 10 commits April 11, 2025 10:56

chore: upgrade bazel to 7.4.1

ba3235b

chore: switch to bzlmod

0e51edb

Remove skip for ibazel test

a88af79

Try to mock source watcher in TestIBazelLoop to fix data race

67b8819

Increase timing exactly the same as proposed in bazelbuild#723

054b34d

Increase timing for live reload test

9e0f878

Increase timing for stream test

e592862

Remove spurios formatting diff

9dbafed

Update module files due to rebase on updated package commits

d9f2a37

Fix output runner test due to renovate updates

69899b9

janwinkler1 force-pushed the switch-to-bzlmod branch from 4799018 to 69899b9 Compare April 11, 2025 12:49

This was referenced Apr 11, 2025

Plans to forward all flags not recognized by ibazel #724

Open

Transfer bazel-watcher to bazel-contrib #669

Closed

achew22 requested changes Jun 14, 2025

View reviewed changes

Merge branch 'bazelbuild:master' into switch-to-bzlmod

8324fce

		// ModuleFileSuffix: "# Comment to force us to use MODULE files by default",
		// WorkspaceSuffix: "# Comment to force us to use MODULE files by default",

chore: switch to bzlmod #712

Are you sure you want to change the base?

chore: switch to bzlmod #712

Uh oh!

Conversation

janwinkler1 commented Mar 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

janwinkler1 Mar 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

achew22 commented Apr 3, 2025

Uh oh!

janwinkler1 commented Apr 4, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

janwinkler1 Apr 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chickenandpork commented Apr 8, 2025

Uh oh!

janwinkler1 commented Apr 9, 2025

Uh oh!

chickenandpork commented Apr 9, 2025

Uh oh!

janwinkler1 commented Apr 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chickenandpork commented Apr 9, 2025

Uh oh!

chickenandpork commented Apr 9, 2025

Uh oh!

chickenandpork commented Apr 9, 2025

Uh oh!

chickenandpork commented Apr 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

janwinkler1 commented Apr 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

janwinkler1 commented Apr 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chickenandpork commented Apr 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

janwinkler1 commented Apr 15, 2025

Uh oh!

chickenandpork commented Apr 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

janwinkler1 commented May 28, 2025

Uh oh!

chickenandpork commented May 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chickenandpork commented Jun 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

janwinkler1 commented Mar 23, 2025 •

edited

Loading

janwinkler1 Mar 29, 2025 •

edited

Loading

janwinkler1 Apr 10, 2025 •

edited

Loading

janwinkler1 commented Apr 9, 2025 •

edited

Loading

chickenandpork commented Apr 10, 2025 •

edited

Loading

janwinkler1 commented Apr 10, 2025 •

edited

Loading

janwinkler1 commented Apr 11, 2025 •

edited

Loading

chickenandpork commented Apr 14, 2025 •

edited

Loading

chickenandpork commented Apr 17, 2025 •

edited

Loading

chickenandpork commented May 30, 2025 •

edited

Loading

chickenandpork commented Jun 12, 2025 •

edited

Loading