Cross-browser testing strategy

[Nolski]

Overview

An issue was raised in a recent meeting when the torque edit feature didn't seem to work on an older version of Firefox. It was brought up that we may need a way to test our MediaWiki add-ons.

Drawbacks

The complexity of user interfaces means additional complexity when it comes to testing them. There's a variety of ways to do this:

• Unit testing javascript code
• Functional testing javascript code
• Visual diffing tests such as percy or storybook
• End 2 end testing using selenium

Beyond this, due to the browers ecosystem, it's difficult to control for things such as:

• browser version, js engine, etc.
• addons modifying the web page

Possible Solutions

Browser compatibility testing

Generally, testing browser compatibility uses selenium under the hood. In most cases, browser compat tests will test the latest version of different browsers. It is possible to test a variety of different browser versions but this requires multiple machines/VMs to be configured.

There's a number of pay-for services which do browser testing for you.

• Browser Stack
• Sauce Labs

They're both quite good, I've used browser stack before. Pricing varies and due to the nature of Selenium being a fairly resource intensive application, I'd expect to pay a fair bit more than your usual CI bill.

If we're looking at doing some less intensive selenium testing (perhaps just testing the latest version of each browser), this is more feasible and MediaWiki does seem to have some native support of Selenium tests. This would require further discussion on what sort of instance this would run on, what it would test, etc.

Pre-deploy javascript & CSS transpilation

We can use an application such as babel to ensure the javascript being deployed on our mediawiki servers is compliant with older javascript engines and lessens the likelihood of being mangled by MediaWiki's minifier. This would likely require we commit compiled javascript code & have some sort of javascript build process.