You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In search.py, results_handler function is not properly pulling links from the Google search results (haven't tested Bing yet). I've been able to somewhat fix the issue by changing it to:
defresults_handler(self, link):
url=str(link.get('href'))
parsed_url=urllib.parse.urlparse(url)
query_params=urllib.parse.parse_qs(parsed_url.query)
actual_url=query_params.get('url', [None])[0] # Extract 'url' from query parametersifactual_urlandself.regex.match(actual_url):
self.results.append(actual_url)
logging.debug('Added URL: {}'.format(actual_url))
elifself.regex.match(url): # Fallback in case the URL is not in query paramsself.results.append(url)
logging.debug('Added URL: {}'.format(url))
This is more accurately finding results (testing by manual performing the dorks). There are other issues I've encountered like .pdf's not being downloaded due to being flagged as a bot, but that's a different issue. Would you prefer a pull request with the changes?
The text was updated successfully, but these errors were encountered:
In search.py, results_handler function is not properly pulling links from the Google search results (haven't tested Bing yet). I've been able to somewhat fix the issue by changing it to:
This is more accurately finding results (testing by manual performing the dorks). There are other issues I've encountered like .pdf's not being downloaded due to being flagged as a bot, but that's a different issue. Would you prefer a pull request with the changes?
The text was updated successfully, but these errors were encountered: