Heads-up: Potential Delay in macOS Infrastructure for Node.js v23

[UlisesGascon]

TL;DR:

We need to confirm it first, but so far, it seems like the macOS infrastructure won't be ready for the Node.js v23 release planned on 2024-10-15 (see Release Issue #1034). This is mainly due to a delay in the Orka migration (see Build Issue #3686).

Original discussion in Slack: Direct Link

Context

How did we arrive at this?

For a long time, the macOS infrastructure has been a challenge for us. Many collaborators and releasers have suffered from it (lack of space, downtimes, etc.). Due to recent events, we ended up having a single provider for the macOS machines, which offered us the opportunity to rethink the infrastructure and mitigate much accumulated technical debt.

After extensive planning for a suitable alternative, we decided to re-engineer our test and release machines using Orka (see Build Issue #3686), thanks to a revamped agreement and a fantastic sponsorship opportunity from MacStadium (read more).

We decided to move from static VMs to ephemeral VMs. This offered us the opportunity to solve all the historical issues we had and allowed us to rebalance the workload more easily. As a result, we are using our resources more efficiently.

What is missing?

In general, we have already finalized the most critical parts of the new infrastructure (new cluster, VPN connections, Orka deployments using Jenkins, image generation using Packer, etc.), but we need more time to finalize the actual testing and migration of the workloads in Jenkins to the new Orka cluster (including the setup of new nodes).

More details: Issue Comment #2278119351

The remaining tasks might take us a bit longer, as we need to wait for more resources to be deployed, and we need to test and patch all the images (macOS version and architecture combinations) in all the pipelines. This is due to the limited availability of the team. I was able to dedicate several weekends in a row to prepare the infrastructure, but currently, I am dealing with some personal things in the following weeks that will limit my availability.

More details: Issue Comment #2352851401

Why does the current infrastructure not support Node.js v23?

Well, Node.js v23 requires a newer version of XCode. This means that we need at least macOS (13.x) in order to support GCC 12 (see Node Pull Request #53668). Our current infrastructure only supports macOS 10.15 and 11.x, which are already obsolete (see). The new infrastructure includes support for macOS 13.x.

What is the impact in terms of the releases?

In practical terms:

• We can't test macOS as part of the CI process, so we might introduce regressions.
• We won't be able to generate the signed binaries for macOS. As an example, check Node.js Latest Releases.
  • The files: node-v*-darwin-arm64.tar.gz, node-v*-darwin-arm64.tar.xz, node-v*-darwin-x64.tar.gz, node-v*-darwin-x64.tar.xz, and node-v*.pkg won't be generated. So macOS users won't be able to install it unless they compile it from source.

Next Steps

• Confirm with @ryanaslett and @UlisesGascon how confident we feel about getting the infrastructure ready on time, so we don't impact the release preparation.
• @nodejs/build and @nodejs/tsc to coordinate a mitigation plan in case the infrastructure won't be ready.

Potential Options/Mitigations

TBD/TBC

[mhdawson]

Well, Node.js v23 requires a newer version of macOS (13.x) in order to support GCC 12 (see #53668). Our current infrastructure only supports macOS 10.15 and 11.x, which are already obsolete. The new infrastructure includes support for macOS 13.x.

Is GCC12 already required to build v23? Want to understand if it is an option to build on the existing GCC level initially until we get the new machines. This does add the risk that we'll break people later when we move to the new gcc level, but we can weigh that risk versus not shipping macOS in the initial v23 release as an option.

[richardlau]

Well, Node.js v23 requires a newer version of macOS (13.x) in order to support GCC 12 (see #53668).

This isn't quite correct as on macOS we use xcode which is clang based (nothing to do with GCC). However there did seem to be general agreement from those more familiar with macOS that we should be on a later version of xcode.

[mhdawson]

We won't be able to generate the signed binaries for macOS. As an example, check Node.js Latest Releases.

I think @ryanaslett was fixed the release machine, ignoring the version of the compiler used would there be any other blockers for generating the signed binaries?

[richardlau]

Is GCC12 already required to build v23?

On platforms that use gcc, 12 is the new minimum supported version for Node.js 23. When last checked main still built with gcc 10 but there was at least one PR that was failing to build with gcc 10.

That doesn't apply to macOS as xcode is clang based (see above).

[mhdawson]

I think the options that we want to confirm are possible and then choose from include:

1. Ship 23 on time with macOS based on limited testing in GitHub Actions
2. Ship 23 on time with macOS based on testing in GitHub actions and a single run on the test CI (not sure if this is possible or not)
3. Ship 23 on time without macOS
4. Delay 23 for ??? long

Please add any other options, and identify concerns if some of the options are not possible (for example 2 may not be in the current state).

I don't believe we should consider 4) as I don't think we can guarrantee how long might be ??? and its important to ship on our consistent schedule.

[UlisesGascon]

Please add any other options, and identify concerns if some of the options are not possible (for example 2 may not be in the current state).

• Point 1: Worth having a discussion.
• Point 2: Might be possible as we have already managed to pass some tests in the cloned pipelines. See: https://ci.nodejs.org/job/node-test-commit-osx-ulises-macstadium-test/nodes=osx13-x64/lastBuild/. It will require some manual work from the Build team or from the releasers, but it seems achievable.
• In my opinion, Point 3 seems more realistic than Point 4.

The biggest issue will be signing and uploading the binaries. This is something we always handle using Jenkins and agents, not on our local machines. Not having signed releases would be worse than shipping version 23 without macOS support, as this would break many users and systems. macOS will flag the binaries when you try to install them if they are not properly signed.

[richardlau]

@UlisesGascon Would you (and/or @ryanaslett) be able to attend the TSC meeting tomorrow (2 Oct) in case there are questions when it is discussed?

[joyeecheung]

One thing to note is that many developers use macOS anyway, so we should already get good coverage from local testing (or I think if macOS gets broken people would notice very soon). It's unknown what versions of macOS people are using, though.

[mhdawson]

The biggest issue will be signing and uploading the binaries. This is something we always handle using Jenkins and agents, not on our local machines. Not having signed releases would be worse than shipping version 23 without macOS support, as this would break many users and systems. macOS will flag the binaries when you try to install them if they are not properly signed.

I'm not quite understanding the issue here. I think @ryanaslett fixed the existing release machine and I don't think we've stopped doing existing releases so I'm not sure how it will be different for 23 in terms of signing and uploading the binaries?

[UlisesGascon]

@UlisesGascon Would you (and/or @ryanaslett) be able to attend the TSC meeting tomorrow (2 Oct) in case there are questions when it is discussed?

I can make it!

[UlisesGascon]

I'm not quite understanding the issue here. I think @ryanaslett fixed the existing release machine and I don't think we've stopped doing existing releases so I'm not sure how it will be different for 23 in terms of signing and uploading the binaries?

The problem is the MacOS version that we use in the existing release Machine(s) is old (11.x), we will need at least 13.x in order to use a newer version of xcode that can support the release process for Node@23.

AFAIK we need to build the binary in that machine in order to sign it, but maybe this process is different in the Pipelines and we grab the binaries from a different place in order to sign them 🤔

[mhdawson]

@UlisesGascon can you remind me why the release process is different for node 23, assume apple changed something and we have only incorporated the changes into the main/23 line so far?

[targos]

The main question is: would we be fine with releasing v23.0.0 with the current toolchain, and changing it (potentially breaking users on older macOS in a non-major release) later?

[aduh95]

2. Ship 23 on time with macOS based on testing in GitHub actions and a single run on the test CI (not sure if this is possible or not)

Would it be possible to disable the Jenkins macOS jobs only on main? In that case, I think we could disable caching (assuming that would solve the disk space issue), meaning macOS builds on Jenkins would take a long time, but that would only affect release lines – and the relatively low frequency at which we need Jenkins builds on release lines, the jobs would be much less likely to build up.

[ryanaslett]

@UlisesGascon Would you (and/or @ryanaslett) be able to attend the TSC meeting tomorrow (2 Oct) in case there are questions when it is discussed?

I can attend as well

[mhdawson]

Would it be possible to disable the Jenkins macOS jobs only on main? In that case, I think we could disable caching (assuming that would solve the disk space issue), meaning macOS builds on Jenkins would take a long time, but that would only affect release lines – and the relatively low frequency at which we need Jenkins builds on release lines, the jobs would be much less likely to build up.

I think we could do that through the version selector - https://github.com/nodejs/build/blame/main/jenkins/scripts/VersionSelectorScript.groovy although I'm not 100% sure that would catch all CI jobs but it should catch the main PR test jobs.

[richardlau]

Looks like the V8 13.0 update will require the new infra and won't compile on the existing machines.

Refs: #55014 (comment)

This PR requires newer macOS versions in CI.

[mhdawson]

I think we can close this out as the release went out.