Is it possible to skip a request?

[windbridges]

I would like to implement some filter that could determine whether a request from the queue will be requested by the browser, or whether it should be skipped. Obviously, this is when the next request has already been received from the queue, but has not yet been passed to the browser.

I thought preNavigationHooks was the way to go, but there's no access to the request there, nor any way to return any result.

Can you tell me if it's possible to implement this? Maybe there is such a hook in BrowserCrawler or even in RequestQueueue?

[B4nan]

Why are you adding requests you don't want to be executed into the queue in the first place? I'd say you should filter them right ahead.

How are you adding the requests to the queue? If it's via enqueueLinks helper, there is transformRequestFunction option you can use, if that method returns false, the request won't be queued.

[windbridges]

My situation is that the crawler has collected some database containing a huge number of queries for further processing. And at some point, we see that many of the collected URLs have a distinctive feature that clearly says that this URL should not be processed because it has no informational value.

So in order to save resources and time, we have to exclude these URLs from the database by a certain criterion. But to clear the queue and run the crawler again is not a good option. Therefore, I would like to write a filter function, which can be applied both before writing to the database, and before executing the request. That way I could change the url filter criteria at any time without having to clear the request queue.

[mnmkng]

I thought preNavigationHooks was the way to go, but there's no access to the request there

You do have access to the request. It's a property of the crawlingContext. There isn't any direct API to do this, but you can set the request to not retry and then throw.

preNavigationHooks: [
    async (crawlingContext, gotoOptions) => {
        const { request } = crawlingContext;
        request.noRetry = true;
        throw new Error('Request cancelled.');
    },
]

[windbridges]

Yes, that's what I need, thank you. Missed that crawlingContext has a 'request' property. And didn't even think that throw would behave the right way in this context.