Some URLs hang and PhantomJS refuses to die

[anjackson]

The crawler ran out of memory due to PhantomJS processes hanging about. We need to try to improve the rendering behaviour, but ensure they get killed either way. Not sure what the problem URIs are. The following process got killed so this may be useful test case:

/opt/phantomjs/bin/phantomjs --ssl-protocol=any --ignore-ssl-errors=true --web-security=false --proxy=warcprox:8000 phantomjs/phantomjs-render.js http://www.telegraph.co.uk/education/2017/04/19/british-children-spend-time-internet-almost-every-developed/ /tmp/webrender/tmpkh2sqnhe :root

IIRC I've also seen a lot of hanging processed from rendering http://www.thejc.com/

There's a lot of instances of http://www.theatrffynnon.co.uk/ and http://www.ysgolllanfynydd.co.uk/ at the moment, so that might be slow/difficult

[anjackson]

So, due to some apparent bugs/oddities in PhantomJS, the page.open can fail, after which the process hangs rather than exiting neatly.

Some clean-up of the code (see 18f894d and preceding commits) and testing indicated that the simplest option is simply to wait and then force the exit whether or not the page.open reported success. This has been included in v. 2.0.8.