Port Mozilla tests to Higgs

[maximecb]

Mozilla have a large base of tests cases for their SunSpider JS engine, see:
http://hg.mozilla.org/mozilla-central/file/d812f80a0f1d/js/src/tests

We would like to port these tests to work with our own test framework (test-runner.js, which is executed with the make test command). This will require cherry-picking tests which are currently applicable to Higgs as some of these will use features which we do not yet support. It will also require changing parts of the tests to use our own test library (lib/test.js) instead of their own idiosyncratic assertion functions.

Each ported test should ideally be a standalone JS file which can be run by Higgs directly from the console (e.g.: ./higgs tests/mozilla/mytest.js). The ported tests should go under source/tests/mozilla. New issues should be opened for the tests that currently fail under Higgs.

Note that it's important to include the license files associated with the tests that get ported.

[morrisonlevi]

As an example test: tests/test262/ch12/12.1/12.1-1.js
The naming scheme inside the 06-sunspider directory doesn't match up with SunSpider's tests. What would you want to be done here?

Also, do you have any more guidance on which tests to import, or do you want basically all tests?

[maximecb]

I was hoping we could create a tests/test262 or tests/mozilla directory and place the tests in there. The number, 06, is because our test runner runs tests in alphabetical order. I tried to sort them from simplest or most fundamental to most complex.

The test lib is automatically included by the test runner. So we could implement extra testing functions needed for mozilla compatibility in there.

We'll probably need to do some hand-picking of tests. There are JS features Higgs doesn't support. What I'm hoping is that by adding these tests, we can discover fixable gaps in our JS support.

[iankronquist]

I'm working on this (I'm muricula on irc).

[iankronquist]

Hello again! The test suite you are interested in is called test262 (named after some committee or something). It can be found on Github: https://github.com/tc39/test262. Running it is as easy as:
./tools/packaging/test262.py --command /path/to/Higgs/source/higgs. It would be handy to set up a Travis CI post commit hook to run all the tests.

[maximecb]

Can you try running it and tell us what results you get? Does it work out of the box at all?

[iankronquist]

Some of the tests pass and some do not, I haven't let it run for the full time yet because I'm impatient. I'm trying to get them to run on Travis CI, but the build may time out, we'll see.
There are enough tests that it might be nice to see a fancy dashboard with which suites are passing and failing. I know Mozilla has such a dashboard for B2G, but I don't know about Spidermonkey.

[iankronquist]

The travis tests are running on my fork, if you're curious:
https://travis-ci.org/iankronquist/Higgs

[maximecb]

Seems a lot of them are failing due to missing libraries. See "undefined base" errors. Would be worth establishing what causes that.

[iankronquist]

So the build has stalled, perhaps while running test 15.4.4.15-3-28.
Fun fact, Travis CI builds will be killed after 50 minutes.
http://docs.travis-ci.com/user/build-configuration/#Job-Timeouts

[maximecb]

I don't think we necessarily need to run this on Travis. The point would be to get the tests to run once and maybe spot some more or less trivial bugs. Then we can open issues for those, create tests of our own and add them to our own test suite.

Right now, something is clearly preventing the tests from even running properly. Likely a missing dependency.