Error when running node test.js

[kennedyrose]

I believe this is the issue with the amazon-crawler-app. Fixing this module should fix the problem there?

[clsedlacek]

I think this is the reason the crawler is stopping and won't move past old reviews- I turned on Horseman and Bluebird logging and we're getting the error described in these issues
johntitus/node-horseman/issues/188
ariya/phantomjs/issues/12750

horseman-crawl-err-resourceError.txt

It's been triggering intermittently and crashes the node.js process due to an unhandled error. There doesn't appear to be a resolution that fixes the failed GETs in Horseman yet (I've tried some of the things people said worked for them, but nothing helped), unfortunately. I'm not sure there's a reliable way to fix it in PhantomJS either, or whether it's possible to really apply that fix for Horseman. If it would be a good temporary solution, I could try to get the crawler to ignore dropped requests and continue to process the data.

[kennedyrose]

Well damn. But also good catch!

I think you're spot on. Getting it to ignore the request is probably best for now so it's not throwing us errors. In the meantime, hopefully the rest of the app is built so next it'll catch the missing ones on the next cron and we're fine long term. Maybe just subscribe to that issue so and when/if it gets updated we'll just update the module and call it a day.

You rock! Thanks for looking into this!