Mozilla Tips

Traces

Try Linux64 debug

Pernosco only processes failures in linux64-debug tests on try. If your try push doesn't run any linux64-debug tests, Pernosco will ignore it.

Hiding pushes from Pernosco

To force Pernosco to completely ignore a push, add nopernosco to your topmost commit.

Reproducing arbitrary Taskcluster job failures

Mozilla users can use the "self-serve" interface to request the reproduction of any supported Taskcluster test job. Use URLs of the form https://pernos.co/self-service-api/mozilla/<task-ID>/self-service.html

The pernosco automation will attempt to identify all distinct test failures (as identified by TEST-UNEXPECTED-FAIL or similar) associated with the test job so that the specific test that failed can be run under automation. Failures that do not identify a specific test as failing will not be detected and cannot be reproduced. (This is intentional, as attempting to run the entire test corpus from a test job would take a long time, result in an unwieldy trace, and usually result in a test failure unrelated to the failure you are trying to reproduce.)

These test failures will be presented as a list of test names with radio buttons next to them. If you do not see one or more tests listed, then one or more of the following is true:

1. No test failures were identified.
2. The test job in question isn't supported by pernosco's automation. This could be for many reasons:

• The test job was not on a supported platform. (Pernosco only works on linux64 for Mozilla automation.)
• The job type is not supported by pernosco's automation (yet). Consider asking in "Pernosco (Mozilla)" on https://chat.mozilla.org/.

When you click on the "Reproduce" button, you will eventually see an alert that the server received your request. This could take a few minutes and make you wonder if you pressed the button if a server is being spun-up to deal with your request. This is tracked by https://github.com/Pernosco/pernosco/issues/21

No-opt builds

Debugging works better with optimizations disabled. If you select a build-linux64-noopt/debug build in your try push (example) Pernosco will try using a noopt build to reproduce the failure.

Submitting local recordings to Pernosco

Use pernosco-submit. If you don't have the necessary credentials, email [email protected]. You must use a @mozilla.com email address in PERNOSCO_USER.

Accessing traces

There is no way to guess or list the https://pernos.co/debug/.../index.html session URLs (unless you are Pernosco staff). If Pernosco sends you a session URL, it is only accessible to whomever you share that URL with.

Also, Mozilla sessions by default are only accessible to Github users who have a verified @mozilla.com email address attached to their Github account. If this is impossible for you for some reason (e.g. you're a Mozilla volunteer), send email to [email protected] to get added.

We have made some Mozilla sessions public for demo purposes, but those are Mozilla recordings that we produced in Pernosco infrastructure using public information. We will not make user-uploaded recordings public unless specifically requested.

Analyzing Firefox

JS Tricks

DumpJSStack() in print expressions

Pernosco doesn't understand SpiderMonkey's internals, but it's possible to use DumpJSStack() inside pernosco's "print" expressions. Because its output consists of multiple lines in a potentially quite long string, it can help to use pernosco's =~ operator which builds on top of the rust regex crate to provide matches via expr =~ m/PATTTERN/ and replacements via expr =~ s/MATCH/REPLACEMENT/. For those of you who are most used to JS regexps, the mode flags go at the beginning of the pattern like (?m) for multi-line.

For example, a print expression of DumpJSStack() =~ s/(?m)^(.+dom\/serviceworkers.+)?$/${1}/ will result in the print expression evaluating to the first line of the DumpJSStack output that contains the substring dom/serviceworkers (even if multiple lines match). If no lines match then an empty string will be returned.

In the event the execution line is very busy with lots of arguments and you are having trouble when there are matches, you may also want to use Firefox's control-f find-in-page functionality with "Highlight All" enabled to also have Firefox highlight whatever fixed string you used.

Note that function calls are only available in "print" expressions and not in "condition" expressions because of the potentially unbounded execution costs.

Possible uses of this:

• Find uses of SpecialPowers.spawn by looking for executions of JSActor::SendQuery and a print expression of DumpJSStack() =~ s/(?m)^(.+spawn.+)?$/${1}/ produces matches like "0 spawn(target = \"[object XULFrameElement]\", args = \"\", task = [function]) [\"resource://specialpowers/SpecialPowersChild.jsm\":1543:16]"