feat: Introduce ecosystem tests for popular plugins #127
Conversation
|
That none of the At least one of eg I also note that the selection criteria is similar to those outlined in the suggested |
| 2. On the `main` branch only | ||
| 3. On all PRs targeting the `main` branch, alongside existing CI jobs | ||
|
|
||
| Starting with a job separately from `main` ensures that unexpectedly high frequencies of breakages are caught early, without blocking `main` branch builds. |
There was a problem hiding this comment.
I'm a bit confused by this process. Is the intent that we end up with all three at the end? Or just number 3?
And I'm unclear on the value of number 2. Presumably, this is meant to run only after a PR is merged, but how will we be notified if the job fails?
There was a problem hiding this comment.
intent
My intent was that each step would replace the previous. As in, at the end, we end up with just 3, which is a superset of 2.
how will be notified
I was thinking the ❌ failing commit on the main branch.
I clarified the intent in c767f0f. Does that make sense?
There was a problem hiding this comment.
I was thinking the ❌ failing commit on the
mainbranch.
The problem with this is that it's passive -- it requires someone looking at the GitHub repo Code tab. (And then it requires that someone care enough to click on it to see what's going on.) I feel like this will be easy to miss. Is there some way to have a notification of some sort?
There was a problem hiding this comment.
GitHub user that merged the PR gets a notification if CI checks failed on the main branch, so perhaps that user could take further steps and/or notify the rest of the team in the team channel.
There was a problem hiding this comment.
I don't get GitHub emails so I'd never know. 😄 I think we need an automated solution.
There was a problem hiding this comment.
How about sending a Discord message to the #contributors channel? Something like https://github.com/Ilshidur/action-discord could automate it.
My only concern there would be spamming the channel if multiple successive builds fail. Folks might be merging PRs without keeping them up-to-date with main. We can always add logic to only run the message if the current commit build failed and the previous commit build succeeded.
Co-authored-by: Nicholas C. Zakas <[email protected]>
Co-authored-by: Milos Djermanovic <[email protected]>
| The new CI job will, for each plugin: | ||
|
|
||
| 1. Clone the plugin into a directory named `test/ecosystem/${plugin}` | ||
| 2. Run the plugin's package installation and build commands with [ni](https://github.com/antfu-collective/ni) |
There was a problem hiding this comment.
How will ni be installed? And will ni automatically install npm (or another package manager) if it's not found locally, and does this also work in a CI environment?
There was a problem hiding this comment.
installed
npm i -g ni,
automatically install
The nci command will.
I'll make those two points explicit.
Aside: I filed antfu-collective/ni#263 as a docs issue on ni to support this.
| 2. Run the plugin's package installation and build commands with [ni](https://github.com/antfu-collective/ni) | ||
| 3. Link the plugin to the current eslint installation | ||
| - This will have to be done manually, as ni does not support linking ([ni#85](https://github.com/antfu-collective/ni/issues/85 "ni issue 85: Maybe xxx link can join ni project")) | ||
| 4. Run the plugin's `test:eslint-compat` script with [ni](https://github.com/antfu-collective/ni) |
There was a problem hiding this comment.
Does ni also install Node.js in the correct version for the package?
There was a problem hiding this comment.
No, but last I checked, these packages are all fine with running on approximately the same Node.js versions as ESLint's CI. As in, they might specify preferred packageManager / .nvmrc versions, but they should pass CI.
|
|
||
| See [Plugin Selection](#plugin-selection) below for specifics on which plugins will be included. | ||
|
|
||
| > ⚠️ Plugins are currently being asked for feedback on the `test:eslint-compat` script. |
There was a problem hiding this comment.
That's good. If there are any questions I think we can also discuss them in this RFC.
|
@JoshuaKGoldberg just a reminder there are a few questions left. |
|
👍 I'd been hoping for an answer on the Discord messaging, #127 (comment). I just went ahead now and pushed it as part of the RFC for review. |
Summary
Adding an CI job to the
eslint/eslintrepo that checks changes against a small selection of third-party plugins.Related Issues
eslint/eslint#19139