Unhandled reject Error: Failed to load url
#188
Comments
|
It's interesting that the One with |
|
Test case above submitted by @minhchu does not reliably reproduce the issue. 😕 |
|
Another case where the |
|
wow, haha 😁 |
|
This is an odd one. When I run the script it works fine and onLoadFInished is only triggered once after the open. What node version and operating system are you using? |
|
My Node.js version is v4.4.4 and my OS is OS X El Capitan v10.11.4. |
|
node -v 5.3.0, npm -v 3.3.12, Windows 7 64 bit. |
|
For what it's worth: I'm having this issue as well but it is very indeterminate for me. I've been unable to reproduce it reliably. My code runs on a loop and I see this error pop up around 25% of the time. I've used this to help reproduce it: function main() {
// ...horseman do stuff...
horseman.finally(() => {
console.log(`Waiting ${process.env.DELAY} minute(s)...`);
setTimeout(main, process.env.DELAY*60*1000);
});
}Hope this helps, and for the record: $ node --version
v6.2.2
$ npm --version
3.9.5
$ npm list node-horseman
...
└── [email protected] |
|
I also have this issue. For me it happens more times than it works. After a click() and waitForNextPage(), it throws the failed url error. |
|
I'm having this issue too: .value --> .click --> .waitForNextPage() --> "Failed to load url". Sometimes it happens after the click before it can even reach the wait. I can't identify any pattern in when it works and when it doesn't. |
|
Same issue here. Works intermittently, but mostly it doesn't. Makes horseman unusable for /me. 😢 |
|
Why can't you just catch the rejection @sayanriju? I have still been unable to reproduce this. |
|
Me too have this problem. It happens randomly. var horseman = new Horseman(Global.horsemanOptions);
usersList.forEach(function(user){
horseman
// Open the page
.open(WEB_PAGE)
.value('#loginform > input[name=username]', user.username)
.value('#loginform > input[name=password]', user.password)
.click('#loginform > button[type=submit]')
.waitForNextPage()
.screenshot(user.username + '.png')
.then(function(){
console.log(user.username + ' LOGGED IN')
})
.....
.....
});Error seems to be raised for the sequence: .click('#loginform > button[type=submit]')
.waitForNextPage()because commenting those statements, error is not raised (but others yes). The strange thing is that screenshots are taken correctly, they are images of a successfully login, but next statements are not correctly executed. I'm running it in: Does anybody know a solution for that? |
|
@OrtoNormale I don't think this is related to the bug you're seeing, but - you're trying to use horseman, which is asynchronous, in a synchronous loop. Each .action() takes time, but by the time the first page is open, your loop has probably completed. |
|
There was an error in my code snippet. At every iteration a new horseman object is initialized. The statement with new Horseman() has to be moved inside the loop. So every user has its own instance. But is'nt .open() waiting for page load before go on? |
|
Yes, but by the time the first user's page is open, the loop has likely completed, and the function doStuff(user){
var horseman = new Horseman(Global.horsemanOptions);
horseman
// Open the page
.open(WEB_PAGE)
.value('#loginform > input[name=username]', user.username)
.value('#loginform > input[name=password]', user.password)
.click('#loginform > button[type=submit]')
.waitForNextPage()
.screenshot(user.username + '.png')
.then(function(){
console.log(user.username + ' LOGGED IN')
})
}
usersList.forEach(doStuff); |
|
Thanks for the answer. As soon as possible I'll try and report. |
|
Unfortunately problem persits. Here the code I tried: var performLoginTask = function(usersList){
var task = function(user){
var horseman = new Horseman(Global.horsemanOptions);
horseman
.on('consoleMessage', function( msg ){
console.log(msg);
})
// Open page
.open(WEB_PAGE)
.value('#loginform > input[name=username]', user.username)
.value('#loginform > input[name=password]', user.password)
.click('#loginform > button[type=submit]')
.waitForNextPage()
.....
.....
.....
};
usersList.forEach(task);
};
performLoginTask(usersList);anyway it seems very similar to my version. Error is the following:
and it's thrown randomly (sometimes yes and sometimes not) and from not a specific user everytime. It's a really powerful tool and It is a shame not use this tool due to this problem. |
|
at the bottom of your horseman chain, can you add a .catch()? On Thu, Aug 4, 2016 at 1:06 PM, OrtoNormale [email protected]
|
|
Added catch, but errors still throwed: var performLoginTask = function(usersList){
var task = function(user){
var horseman = new Horseman(Global.horsemanOptions);
horseman
.on('consoleMessage', function( msg ){
console.log(msg);
})
// Open page
.open(WEB_PAGE)
.value('#loginform > input[name=username]', user.username)
.value('#loginform > input[name=password]', user.password)
.click('#loginform > button[type=submit]')
.waitForNextPage()
.....
.....
.....
.catch(function(error){console.log(error)})
};
usersList.forEach(task);
};
performLoginTask(usersList);and console.log() in block catch print the same text as the one in throwed error. |
|
Could the problem be related to a redirect (HTTP code 302) that is done after login? |
|
If it's being caught in the On Thu, Aug 4, 2016 at 1:45 PM, OrtoNormale [email protected]
|
|
The error is thrown only for one or two and randomly, not always the same users. Ok, I'll try to handle that error in catch(). |
|
How many users are you trying to do at the same time? I wonder if it's On Thu, Aug 4, 2016 at 2:29 PM, OrtoNormale [email protected]
|
|
At the moment max 10 users, in future maybe more. |
|
Echoing what's been said in the earlier comments. Horseman appears to hang after this happens. For me, it's reliably breaking my application. Note that I receive three My output, from the |
|
Worth noting: I, like @OrtoNormale, am This is just a guess... but Phantom following the |
|
As @shockey says, also in my case horseman jump directly in catch(){...} after that error is thrown without executing the rest of the code |
|
Can everyone running into this issue try setting their horseman timeout to something big, like 30 seconds, and retrying? |
|
The |
|
We are getting this too. What's the ETA on the fix? |
|
There is no ETA @design2dev, we still have no way to reproduce and be sure there even is a bug. Sometimes PhantomJS can legitimately fail to load a URL for some reason. |
|
I get it too. But instead of loop, it throws up randomly (re-run application). Also, why onLoadFinished is being triggered twice? |
|
Failed to GET url error is happening to me when I do |
|
You have to include the protocol @jharrowmortelliti. If you don't give one, I think PhantomJS tries |
|
I try to run the example at node-horseman/examples/links.js and it gives me the 'Failed to get url' error. |
|
Strange @jharrowmortelliti. When I run that example I get an error because the click selector no longer works, but once I change the selector to |
|
Any workaround for this issue? |
|
I encountered the same issue, in my case adding a |
|
Adding |
|
I also get this issue. |
|
What if there is no clicking in the function? All I'm doing is opening a page, and scraping content, but pretty regularly getting |
|
The error means what it says @webdev, PhantomJS failed to load a URL for some reason. This can be caused by a variety of things. |
|
I've been experiencing this issue recently as well, whilst making an acceptance testing framework for my workplace. I did some digging into the issue using phantomjs directly, and it looks like phantomjs, during redirects from the page itself, cancels requests, and this is what causes the fail, even if the page loads successfully. For context, in the application, when loading, the application will refresh. This is a legacy issue with it that currently won't change for a while. Doesn't cause any issues elsewhere though. Just with phantomjs test case; DEBUG=horseman output (url obscured); Adding output for the network requests received, we can see the following; From phantom and horseman, we can access the ttf file above; it is just the network requests being cancelled because of the redirect. Looking at the phantomjs issues, I found ariya/phantomjs#12750 which seems to be the same as the above here. It seems to specify that the issue is fixed in the 2.5 beta (which I haven't been able to test with unfortunately) but the issue doesn't feel to be in horseman for this. |
|
The following fixed my issue: QT_QPA_PLATFORM=offscreen Set that environment variable and the PhantomJS starts working again as does Horseman. As found here: ariya/phantomjs#14376 |
|
I have the issue similar to @Skyerin - one of the pages that I am scraping does HTTP 301 redirects after I fill out and submit a form causing the "Failed to load url" error. With @dhilditch env For me its consistent to reproduce in both MacOS and Dockered Ubuntu. Phantom 2.1.1. Out of ideas 🤔 |
|
@dhilditch that unfortunately didn't work for me. I had the same issue as @robertpallas did :( @robertpallas - on your environments, could you test it with the phantom 2.5 beta? you can provide the path to node-horseman; just see if that solves the issue at all for you (as it is stated in ariya/phantomjs#12750) ? Worth a shot at the very least... I didn't manage to get the beta working on both osx and centos, but you might have better luck than me. |
IssueMy issue is most probably related to one of the redirects being 0B HTTP 307 Internal Redirect from http to https. "Operation canceled" resourceError also comes as mentioned earlier in this thread. Recreation with HTTP only traffic with same HTTP redirects failed - Horseman works as expected. Phantom 2.5I was able to run Phantom 2.5 within Docker with Ubuntu 16.04 under it and also set ENV variable NextMaybe I can somehow force the redirect to go directly to https and skip that Internal Redirect step? Any other workaround ideas are welcome. |
|
@robertpallas I can't think of anything else you could try apart from modifying how your redirects are happening as they're happening from the page itself; unless we could add a delay somewhere somehow? One of the ideas I was going to look at (because I deal with a SPA) was to modify the reload code so that when it comes from phantom (based off a flag/specific user agent) so that it would either add a delay to the reload (allowing network requests to finish first), or stop the reload altogether and I would do that manually in my tests. The only other thing I could think of was just changing away from phantom and experimenting with chrome headless - although that is in a very barebones state and I do absolutely love how horseman makes things simple and lets itself be easily extensible. |
|
I have also this issue.
my code is: Any ideas for workarounds? |
|
I found that these steps fixed this issue for me:
I had previously had this issue repeatedly for any url that was complex (e.g. I'm not sure if it was something to do with the order in which I installed things? the fact that I'd installed node-horseman before phantom previously? Or that I'd previously installed packages that I was no longer using (e.g. Anyway, that's how I fixed my world. UPDATE: that's how I partially fixed my world - I am still seeing this issue intermittently :( The thing about the complexity of the page / speed of the page loading / assets in the page etc seems to hold true though - a simple page doesn't throw this error |
|
@geek-caroline |
|
Hi, I also experienced this. And I fixed this by changing my DNS to 8.8.8.8. It seems very similar with issue at docker for mac docker/for-mac#1317 |
|
@felipemullen you may well be right, I've not seen the problem again and I've used the fully qualified |
|
DNS set to 8.8.8.8, full https:// url, ignoreSSL: true, wait calls left and right ... getting this error fairly frequently. |
and others
@minhchu has reported a bug they are facing where an
Unhandled rejection Error: Failed to load urlis thrown. This is similar to #180 however the case is not due to SlimerJS in which #180 is specifically for. This new issue will track the new case @minhchu is facing.The test case which can reproduce the issue is below:
The output is below:
The text was updated successfully, but these errors were encountered: