feat: add yarn berry support

[favna]

Thanks for contributing to the buildpack. To speed up the process of reviewing your pull request please provide us with:

• A short explanation of the proposed change:

I am making a first attempt at adding Yarn berry support to the CF NodeJS buildpack. This isn't yet finished because unit tests still need to be updated/added. I'm creating a Draft PR because I'm not all too familiar with Golang (admittedly big chunks of this PR were written with Copilot assistance) and I would like some early feedback. Some points I am particularly unsure about:

1. How to generate code with mockgen, exactly. I am developing on a MacBook and I have installed Go through Brew but I cannot figure out how to now install and use Mockgen. When I click the context menu buttons in VSCode it tells me mockgen is not on my PATH.. I managed to figure this out by using JetBrains GoLand IDE which had commands to run mockgen. Bliss JetBrains 🙏!
2. For the functions getYarnVersion and isYarnLocalCacheEnabled how to process the stdout of the commands and assign them to variables so they can be used for the return of the function. The current code was CoPilot generated but I have my strong suspicions it is incorrect because I am not using y.Command.Run like the Build function (now doBuildClassic) is. I Managed to figure this out through writing unit tests and some println's. Again bliss JetBrains GoLand for making it easy to run specific suits repeatedly.

This PR is sort of related to #229 but also in general it would be great if the CF NodeJS buildpack supported Yarn Berry because Yarn v1 is well and thoroughly deprecated by now and Yarn has recently released Yarn v4. There are many, many advantages to upgrading and there is practically no reason to still stick with Yarn v1.

Lastly I also completely separated checking s.IsVendored from the switch case below it because previously when a project was vendored it would always run npm, regardless of package manager used. That completely broke projects that were using Yarn, at least Yarn Berry. I had temporarily monkey-patched the Go code in a fork that we were using to alleviate this.

• An explanation of the use cases your change solves

Support for Yarn Berry (v2/v3/v4/future). This is something that the buildpack straight-up didn't support before and it should.

• I have viewed signed and have submitted the Contributor License Agreement

• I have made this pull request to the develop branch

• I have added an integration test

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: favna / name: Jeroen Claassens (d38507c, e03ae46, 03e5a2b, 27d1e44, 0955a63, fe3feca, 18add18, 966f9e8, f02197c, 47ef5a1, 08dfec6, 18cba95, 7a72e84, c47b013, ce7bdf2, 66765ba, 5f7191b, 2d4c4f4, 7056ecc, 8a9f29a, 93e0975, 138cecc, 2c0f1b4)

[brayanhenao]

@thitch97 Could you take a look on this PR?

Also, @favna thanks for your contribution, check the Unit/integration tests that are failing.

[favna]

@brayanhenao I would love to, but I'm not sure how I can without mockgen working and as mentioned I'm not sure how to set it up on my system. Any guidance would be much appreciated. I tried joining your Slack with the link in the readme, but it just gives me an error saying I don't have an account on the slack.

[favna]

@brayanhenao I have fixed the existing and added new unit tests. I will now start looking at integration tests.

Edit 2: Integration tests are running locally through Docker but I also seem to be hitting more platform setup issues but I see that GH has started now as well so hopefully they pass in the action runner.

[favna]

Very weird that unit tests failed just now in actions, they worked on my machine. I changed the way global cache is disabled now for Yarn and they still run fine locally:

[brayanhenao]

Hi @favna! Running both the unit and integration suites should not give you problems. All you need is Docker or a Cloud Foundry setup and Golang on the machine where you'll run the tests.

If you're hitting some errors in the configuration, paste the logs here so I can figure out what is happening.

For the integration tests, no worries if you don't have a cf environment. You can go the Docker route instead. Use the command:

./scripts/integration.sh --platform docker --github-token <your-github-token>

[favna]

@brayanhenao Added some logging for unit tests in an attempt to find out why they are failing on GitHub Actions but not locally. As for integration tests, I ran them with docker platform and used tee to dump both STDOUT and STDERR to a file, this is the file: integration.log

[TisVictress]

@favna This looks like a great start. However, I'm not sure why the decision was made to add support for berry in the paketo buildpack only and not in CF. Before you continue working on this, let me ask around and get back to you.

[favna]

I think the commit above finally fixes unit tests in GitHub Actions. Thanks to the run before that I was able to find out there were ANSI color escape codes so a quick prompt to Copilot later I added TERM=dumb to the env. If unit tests finally pass after that one I have 1 more commit pending to clean up all the logging and the manual writing to .yarnrc.yml in tests that I temporarily added.

[favna]

Any updates available @TisVictress ?

[favna]

I temporarily created a branch on my fork that I could use to slam GitHub Actions and find a solution for unit tests. Bunch of builds later I have finally resolved the dastardly issue so I pushed the final clean changes to this branch. You can see that unit tests and integration tests both passed here: https://github.com/favna/nodejs-buildpack/actions/runs/7179209025?pr=1 so once approved for this PR I assume they will also pass here.

[favna]

New year, new changes. Best wishes for 2024. @brayanhenao, @TisVictress can you give me any updates on the review process of this PR?

[brayanhenao]

Hi @favna,

We appreciate your effort in contributing to the project. Currently, supporting Yarn Berry isn't a top priority in our roadmap, and we don't have the capacity to thoroughly review the pull request at the moment.

Given the significant impact such changes can have on users of the Buildpack, we want to make sure any adjustments are carefully considered before implementation.

When we have the time or this change is prioritized we will be updating this PR with the respective comments.

Thank you for your understanding.

[favna]

I revisited this PR the past 2 days because we had some network issues in our CI/CD pipeline and I figured using PnP with Zero Installs (the Yarn berry equivalent of offline mirroring) would be a good solution because then our CI/CD would never have to download dependencies again. A lot of commits later and I can now say it works perfectly on our instance of Cloud Foundry. To give a bit of an idea of how it runs now here is a dump of the logging from the moment that this build pack starts to when it finishes:

Uploading app package for app with guid RANDOM-GUID-1
Creating build for app with guid RANDOM-GUID-1
Cell RANDOM-GUID-2 creating container for instance RANDOM-GUID-2
Cell RANDOM-GUID-2 successfully created container for instance RANDOM-GUID-2
Downloading app package...
Downloaded app package (127.1M)
-----> Download go 1.19
-----> Running go build supply
/tmp/buildpackdownloads/IDHERE ~
~
-----> Nodejs Buildpack version 1.8.22
-----> Bootstrapping python
-----> Installing python 3.11.8
Download [https://buildpacks.cloudfoundry.org/dependencies/python/python_3.11.8_linux_x64_cflinuxfs3_32c1b8e4.tgz]
-----> Installing binaries
engines.node (package.json): ^18
engines.npm (package.json): ^6
-----> Installing node 18.19.1
Download [https://buildpacks.cloudfoundry.org/dependencies/node/node_18.19.1_linux_x64_cflinuxfs3_8e86a72d.tgz]
Downloading and installing npm ^6 (replacing version 10.2.4)...
removed 105 packages, and changed 110 packages in 9s
4 packages are looking for funding
run `npm fund` for details
-----> Installing yarn 1.22.19
Download [https://buildpacks.cloudfoundry.org/dependencies/yarn/yarn_1.22.19_linux_noarch_any-stack_32d0e82e.tgz]
Installed yarn 4.1.0
-----> Creating runtime environment
Detected Yarn, checking your configured nodeLinker...
Yarn Berry is using Plug'n'Play (PnP) mode, detecting if Zero Installs is enabled by checking if the Yarn cache folder exists. For more info visit https://yarnpkg.com/features/caching#zero-installs
NODE_ENV=production
NODE_HOME=/tmp/contentsIDHERE/deps/0/node
NODE_MODULES_CACHE=true
NODE_VERBOSE=false
NPM_CONFIG_LOGLEVEL=error
NPM_CONFIG_PRODUCTION=true
-----> Building dependencies
Prebuild detected (vendored dependencies exist)
Rebuilding native dependencies
➤ YN0000: · Yarn 4.1.0
➤ YN0000: ┌ Resolution step
➤ YN0000: └ Completed in 0s 621ms
➤ YN0000: ┌ Fetch step
➤ YN0000: └ Completed in 0s 892ms
➤ YN0000: ┌ Link step
➤ YN0000: │ ESM support for PnP uses the experimental loader API and is therefore experimental
➤ YN0007: │ es5-ext@npm:0.10.64 must be built because it never has been before or the last one failed
➤ YN0007: │ esbuild@npm:0.19.12 must be built because it never has been before or the last one failed
➤ YN0000: └ Completed in 1s 863ms
➤ YN0000: · Done with warnings in 3s 609ms

Simply what this comes down to in terms of project setup is:

• Yarn v4.1.0
• yarnrc.yml with:

compressionLevel: mixed

enableGlobalCache: false

enableTelemetry: false

nodeLinker: pnp

yarnPath: .yarn/releases/yarn-4.1.0.cjs

supportedArchitectures:
  os:
    - current
    - darwin
    - win32
    - linux
  libc:
    - current
    - musl
    - glibc

(multiple architectures added because we develop on MacOS and Windows, CI/CD runs Alpine (musl) Linux docker images, CF runs Linux with glibc)

• .gitignore with content taken from the Yarn docs:

# Yarn with zero installs
.yarn/*
!.yarn/cache
!.yarn/patches
!.yarn/plugins
!.yarn/releases
!.yarn/sdks
!.yarn/versions

• This means that .yarn/cache is checked into Git which means that there is an offline mirror of all dependencies and CF doesn't need to run yarn install at all, yarn rebuild is enough to run applications.

I am using my fork as a buildpack through configuring my manifest.yml with it:

applications:
  - buildpacks:
      - https://github.com/favna/nodejs-buildpack#v1.8.22-yarn-berry

If someone wants a small repro repo lmk. The actual repo I am using this with is work private so I cannot share it.

[bvincent]

Thanks @favna, do you have any plan to release this feature @TisVictress & @brayanhenao ?