chromedriver version does not match chrome version

[zburke]

Error Report

Following the instructions at https://frontside.com/bigtest/docs/platform:

$ yarn add bigtest --dev
$ yarn bigtest init
$ yarn bigtest server --launch=chrome

fails with

Error: session not created: This version of ChromeDriver only supports Chrome version 89
Current browser version is 88.0.4324.192 with binary path /Applications/Google Chrome.app/Contents/MacOS/Google 
...

Chrome 89 was released on 2021-03-02, six days ago, but it takes several days/weeks for Chrome releases to actually roll out. My Chrome install does not report that it is outdated, for example, and thus there is no way to update to v89.

This was all working on Thursday when I left the office, but was broken when I returned today and, for various reasons, nuked by lock file and ran a fresh build. I figured maybe I could pin the bigtest or ChromeDriver deps to whatever I was using on Thursday, but a resolutions entry for bigtest 0.13.2 didn't change anything and ChromeDriver is not actually a dependency:

$ yarn why ChromeDriver
yarn why v1.22.10
[1/4] 🤔  Why do we have the module "ChromeDriver"...?
[2/4] 🚚  Initialising dependency graph...
[3/4] 🔍  Finding dependency...
error We couldn't find a match!
✨  Done in 0.94s.

diagnostics

Argv

server,--launch=chrome

Stack

Error: session not created: This version of ChromeDriver only supports Chrome version 89
Current browser version is 88.0.4324.192 with binary path /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
at request (/Users/zburke/temp/e2e-yarn/node_modules/@bigtest/webdriver/dist/web-driver.js:65:19)
at request.next ()
at /Users/zburke/temp/e2e-yarn/node_modules/effection/dist/effection.js:1:8145
at e.r.advance (/Users/zburke/temp/e2e-yarn/node_modules/effection/dist/effection.js:1:7781)
at e.r.trapExit (/Users/zburke/temp/e2e-yarn/node_modules/effection/dist/effection.js:1:8107)
at t.n.finalize (/Users/zburke/temp/e2e-yarn/node_modules/effection/dist/effection.js:1:5618)
at t.n.resume (/Users/zburke/temp/e2e-yarn/node_modules/effection/dist/effection.js:1:4972)
at /Users/zburke/temp/e2e-yarn/node_modules/effection/dist/effection.js:1:7289
at processTicksAndRejections (internal/process/task_queues.js:93:5)

[zburke]

Full story: I have a working package-lock.json from last week's npm-based installation. Today's project was to get this running in a yarn workspace (because I am only on node v14/npm v6, because reasons). LMK if you are interested in the lock files.

[minkimcello]

@zburke Could you try updating to 0.13.3? I haven't tested it yet but I believe it was addressed.

https://github.com/thefrontside/bigtest/blob/v0/packages/bigtest/CHANGELOG.md#0133

GitHub

thefrontside/bigtest

Tests that speed up development, not slow it down. Contribute to thefrontside/bigtest development by creating an account on GitHub.

[zburke]

@minkimcello no luck:

$ yarn init
$ yarn add bigtest --dev
$ yarn add typescript --dev
$ yarn bigtest init
$ yarn bigtest server --launch=chrome
💥 

Additional details:

$ yarn --version
1.22.10
$ node --version
v14.15.4
$ npm --version
6.14.10
$ uname -a
Darwin XXX-hostname-redacted 19.6.0 Darwin Kernel Version 19.6.0: Tue Jan 12 22:13:05 PST 2021; root:xnu-6153.141.16~1/RELEASE_X86_64 x86_64

[cowboyd]

@zburke While we try to work through this, are you able to update Chrome?

[zburke]

@cowboyd, no, because Chrome doesn't believe itself to be outdated. I would love to force an update, but the update instructions indicate "If you can't find [the "Update Google Chrome"] button, you're on the latest version." I can't find that button. IOW, my Chrome thinks it's on the latest version.

If I understand Google's statement that releases "will roll out over the coming days/weeks", the problem is precisely that the release hasn't "rolled out" to me.

My npm-based install from last week does work, and I can muddle through with that. I just have to manually keep track of the interactors I patch inside node_modules/@folio/stripes-testing (hence the desire to configure a workspace). Not super convenient but NBD. Good luck!

[cowboyd]

There are no quick fixes here, but one which we're considering is to remove the hard dependency on chromedriver completely and let users just add chromedriver in their dev dependencies themselves. If we find chromedriver in your node modules path, we use that, then if we can't find it there we look in your system path, then, if we don't find it there we issue an error detailing how to either download or install a suitable chromedriver. But even then, we'll need a way to override it since its possible that people end up in a situation where their chrome version is different locally than the one in CI

[zburke]

There are no quick fixes here

Yes and no. If the problem is identifying which chromedriver version to depend so it matches the version of Chrome a user has installed, and that can't necessarily be done, well, crap, you can't solve the problem automatically. But you can easily provide the instructions for solving it manually: "Type chrome://version/ into the address bar to get your Chrome version, then run

yarn add chromedriver@thatversion

That's not really not a big deal (in fact, not really that different from yarn add bigtest --dev). You just need to know you need to do it.

[jnicklas]

The basic problem here is that if we add a dependency on chromedriver in bigtest, then that version might be outdated relatively to chrome, which should normally be evergreen and updated to the latest version. What is really frustrating is that even running yarn upgrade chromedriver does not update that dependency, since it is a transitive dependency. The user has to remove their yarn.lock if using yarn (I think the situation is better with npm), and reinstall. This is obviously not tenable.

The alternative is for the user to add a dependency on chromedriver directly, so instead of doing:

yarn add -D bigtest

They would have to do:

yarn add -D bigtest chromedriver

And then they can use whichever version of chromedriver they want.

Also frustrating is that basically all CI systems actually ship a version of chromedriver on their base images, which obviously matches their installed version of chromedriver. So it'd be better to just use that one! But just having the chromedriver package in your dependency graph, even as a transitive dependency, makes it insert itself in the$PATH, higher up than the system installed chromedriver. So unless we manually scan $PATH and do some guesswork which one is the system one, then we have no way of picking the system installed chromedriver.

The solution to that problem would be not installing via npm at all. Personally I have a bit of a distaste for installing this type of binary package via npm, this is what homebrew/apt are for. But it would leave us in a spot where the user would have to install chromedriver via a native package manager, which in the case of homebrew they might not even have installed.

I think there is no perfect solution here, it's all tradeoffs. We would like to get to the sweet spot of having a user experience that just works with no fiddling, but also perfect integration with CI systems where updates to the platform don't break your tests, but I think this is just not possible.

@jenweber I think this would be a good issue for you to take a look at as well, since it's very much relevant to making a good onboarding process. I'm happy to explain this in more detail if the above isn't too clear.

[zburke]

Lots of good info there about why this is such a thorny problem, @jnicklas! For me, the most important point is "no perfect solution here, it's all tradeoffs" but also my corollary: it's really not a big deal as long as you know you need to do it. i.e. as long as you say, "Now go install a compatible version of chromedriver with npm or homebrew or apt (sorry we couldn't do this for you; it's complicated; here's a link to an article that explains why)", that's fine.