Add support for downloading Chromedriver versions 115+ #1478

jdholtz · 2023-08-16T13:14:16Z

This PR adds support for downloading Chromedriver versions 115+. This is necessary due to the Chromium team's change to Chromedriver's release process (see here).

If the version_main is 114 or older, the Chromedriver will still be downloaded using LATEST_RELEASE_{version}. If the version_main is specified and is 115+, the /latest-versions-per-milestone-with-downloads.json from the new JSON endpoint is used and the version is selected from the corresponding milestone. Last, if the version_main is not specified, the /last-known-good-versions-with-downloads.json endpoint is used to fetch the latest stable version.

In contrast with #1427, this PR uses the new JSON endpoints instead of reverting back to old versions if the LATEST_RELEASE endpoint isn't found (causing version discrepancy errors).

I also added compatibility for installing x86 and arm64 for Mac separately since the platform names changed for the new endpoints. However, I have only tested on Linux and Windows so it would be great if someone could test on Mac (x86 and ARM) It has been tested on Linux, Windows, and Mac with success. The Chromedriver doesn't work on ARM devices when downloading the ARM chromedriver, but it seems to work fine with the x64 version (possibly with Rosetta installed).

This also allows for users to download the Dev and Beta versions (currently 117 and 118) if they specify it using version_main.

Extracting just one file that was in a directory caused an error on Windows. Extracting all fixes this issue.

jdholtz · 2023-08-16T17:12:49Z

If anyone wants to test this change out, you can install this fork through pip:

pip install -e git+https://github.com/jdholtz/undetected-chromedriver.git@29551bd27954dacaf09864cf77935524db642c1b#egg=undetected_chromedriver

Note: If you get errors while installing the above, try to add single quotes around the URL (specifically for Zsh shells). Also, make sure you have Git installed

Also, make sure you have uninstalled the original undetected_chromedriver library before installing this fork.

azorkai · 2023-08-16T23:27:12Z

How to use this correctly? any example code?

File "C:\Python311\Lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 229, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: net::ERR_NAME_NOT_RESOLVED
  (Session info: chrome=116.0.5845.97)

jdholtz · 2023-08-16T23:50:54Z

@azorkai, just install it with the pip command I provided above and use this library as you normally would

azorkai · 2023-08-16T23:51:44Z

Already tried that but got this error;

File "C:\Python311\Lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 229, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: net::ERR_NAME_NOT_RESOLVED
  (Session info: chrome=116.0.5845.97)

jdholtz · 2023-08-16T23:53:03Z

Can you provide the exact code you are using? Also, do you get the same error when you use Chrome v114 or lower with the original v3.5.2 of undetected_chromedriver (not this fork)?

I'm fairly certain the ERR_NAME_NOT_RESOLVED error has to do with how you are using the webdriver after you initialize it.

Kinuseka · 2023-08-17T00:52:16Z

Already tried that but got this error;

File "C:\Python311\Lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 229, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: net::ERR_NAME_NOT_RESOLVED
  (Session info: chrome=116.0.5845.97)

Works fine for me. The error sounds like a DNS resolution error.

neekrom · 2023-08-17T01:18:21Z

On Mac M1 (ARM), getting this error

File ~/src/undetected-chromedriver/undetected_chromedriver/patcher.py:115, in Patcher._set_platform_name(self)
...
--> 123 is_arm_arch = any(["aarch64", "arm"] in platform.machine())
    125 if self.is_old_chromedriver:
    126     if is_arm_arch:

TypeError: 'in ' requires string as left operand, not list

On my machine, platform.machine() returns arm64, so maybe you should be checking for that instead though?

jdholtz · 2023-08-17T01:44:56Z

@neekrom, that line was actually totally wrong. The new commit should fix it. Did this ever work on your ARM machine before? There didn't seem to be code that would download the Chromedriver for ARM before this PR.

I edited the pip install command above to point to the latest fix

neekrom · 2023-08-17T02:07:11Z

@jdholtz I've actually never used this package before. I just tried it out today but found the ChromeDriver and browser version mismatch error, so tried this branch.

Now I'm getting a different error:

File ~/src/undetected-chromedriver/undetected_chromedriver/patcher.py:115, in Patcher._set_platform_name(self)
...
--> 126 if self.is_old_chromedriver:
    127     if is_arm_arch:
    128         platform_name += "_arm64"

AttributeError: 'Patcher' object has no attribute 'is_old_chromedriver'

It looks like you're referencing is_old_chromedriver before the line defining it in line 94 is executed.

jdholtz · 2023-08-17T02:19:46Z

Ha, I guess that's what I get for not testing. It should be good now.

matthewngc · 2023-08-17T02:28:36Z

Also on Mac M1 (ARM), getting this error after the latest fix:

File "~/src/undetected-chromedriver/undetected_chromedriver/__init__.py", line 466, in __init__
    super(Chrome, self).__init__(
  File "~/lib/python3.9/site-packages/selenium/webdriver/chrome/webdriver.py", line 45, in __init__
    super().__init__(
  File "~/lib/python3.9/site-packages/selenium/webdriver/chromium/webdriver.py", line 53, in __init__
    self.service.start()
  File "~/lib/python3.9/site-packages/selenium/webdriver/common/service.py", line 109, in start
    self.assert_process_still_running()
  File "~/lib/python3.9/site-packages/selenium/webdriver/common/service.py", line 122, in assert_process_still_running
    raise WebDriverException(f"Service {self._path} unexpectedly exited. Status code was: {return_code}")
selenium.common.exceptions.WebDriverException: Message: Service ~/Library/Application Support/undetected_chromedriver/undetected_chromedriver unexpectedly exited. Status code was: -9

jdholtz · 2023-08-17T02:44:53Z

@matthewngc what does python -c 'import platform; print(platform.machine())' output? It is possible that this package has never worked with ARM before, which might be the reason it was never supported with the code I added.

If you download and extract the chromedriver from this URL (should be Binary: chromedriver, Platform: mac-arm64), and then set that chromedriver as your driver_executable_path, do you still get the same error? In that case, there is more to this issue than just downloading the correct driver and I will remove that logic.

neekrom · 2023-08-17T02:58:08Z

I pointed driver_executable_path to the downloaded chromedriver, and it gives me the same error. It's looking like ARM just isn't supported in the first place. :(

WebDriverException: Message: Service ./chromedriver unexpectedly exited. Status code was: -9

matthewngc · 2023-08-17T03:04:02Z

python -c 'import platform; print(platform.machine())' outputs 'arm64'.

Also tried setting the downloaded chromedriver as the driver_executable_path and got the same error.
Had no issues running UC prior to the Chromedriver version 115+ issue either, so don't think it's an issue with ARM not being supported.

jdholtz · 2023-08-17T03:09:01Z

Strange. Does the x64 version work for you guys?

Apparently the x64 version works on ARM devices with Mac, so the platform now just chooses the correct platform name based on the Chromedriver version requested

Kinuseka · 2023-08-17T03:20:59Z

yes x64 works. This issue seems to be unrelated to this PR, but unique to mac arm64 only which leads to a still open issue months ago #947

jdholtz · 2023-08-17T03:37:52Z

I just removed support for downloading the ARM chromedriver since it doesn't seem to work. It appears that Rosetta can be installed on ARM Macs (if you are still running into issues) which will bring compatibility for the x64 chromedriver

m3az88 · 2023-08-17T08:28:04Z

any updates on this guys ? I dont want to downgrade my chrome

selenium.common.exceptions.WebDriverException: Message: unknown error: cannot connect to chrome at 127.0.0.1:53658
from session not created: This version of ChromeDriver only supports Chrome version 114
Current browser version is 116.0.5845.96

benfield97 · 2023-08-17T09:00:21Z

getting this error with your branch @jdholtz

import undetected_chromedriver as uc

ModuleNotFoundError: No module named 'undetected_chromedriver'

regular pip install undetected-chromedriver does not result in the same error

dexedrine-01 · 2023-08-17T09:25:02Z

selenium.common.exceptions.WebDriverException: Message: unknown error: cannot connect to chrome at 127.0.0.1:53658
from session not created: This version of ChromeDriver only supports Chrome version 114
Current browser version is 116.0.5845.96

If anyone wants to test this change out, you can install this fork through pip:
pip install -e git+https://github.com/jdholtz/undetected-chromedriver.git@29551bd27954dacaf09864cf77935524db642c1b#egg=undetected_chromedriver

Windows 10, x64. I installed it with your command and see this error. Is it needed to edit something else? I think after that I have two installed versions of driver and script use old, not your fork.

m3az88 · 2023-08-17T09:55:04Z

Can we try this PR with Mac m2 arm?
Shall we waiting for official update?

charlbury · 2023-08-24T09:45:55Z

If I start chrome manually, I can load https pages. If I start chrome with UC (driver = Chrome(options=options, driver_executable_path="C:\Users\data\Desktop\chromedriver.exe")) I cannot browse to any https sites.

jdholtz · 2023-08-24T11:33:53Z

@charlbury if you can reproduce this issue when you download the driver manually, that means this PR does not have to do with the issue you face. Are you using any other libraries with undetected_chromedriver such as selenium-wire?

charlbury · 2023-08-24T15:45:41Z

Yes, we are using selenium-wire but thought that

Selenium Wire will integrate with undetected-chromedriver if it finds it in your environment. This library will transparently modify ChromeDriver to prevent it from triggering anti-bot measures on websites.

Is there another step we need to take at the moment to have bot work?

jdholtz · 2023-08-24T16:10:04Z

@charlbury Selenium wire is not compatible with later versions of Selenium and UC. I have a PR to fix this issue in this repository and Selenium wire. For now, you can add this line options.set_capability("acceptInsecureCerts", True) before initializing Chrome which should fix the insecure connection (this capability is in Selenium wire but isn’t actually added due to changes in Selenium)

CB108 · 2023-08-24T18:41:01Z

Thank you for the update / fix. Much appreciated. uninstalled the old UC and ran the command provided above - it worked today 8/24/23

charlbury · 2023-08-24T20:05:15Z

For now, you can add this line options.set_capability("acceptInsecureCerts", True) before initializing Chrome

That fixed the problem for me. Cheers!

keanureano · 2023-08-25T02:59:32Z

pip install -e git+https://github.com/jdholtz/undetected-chromedriver.git@29551bd27954dacaf09864cf77935524db642c1b#egg=undetected_chromedriver

This is currently working on my Chrome v116 setup. Thank you!

orbar1 · 2023-08-25T08:43:07Z

@jdholtz would u be able to tell me what's the solution for it so far?
i legit got confused from all the answers.
appreciate it!

ultrafunkamsterdam · 2023-08-25T09:34:40Z

@jdholtz would be happy to merge, however we could get rid of the old way all together as they have endpoints for all versions as well. Is this working for both Intel and arm macs too?

ultrafunkamsterdam · 2023-08-25T10:01:49Z

I merged it anyway! new version = 3.5.3 also available from pypi!

Thanks @jdholtz

irfanlogisly · 2023-08-25T11:00:44Z

Hi Anyone using selenium java and facing issue on chrome 116.0.5845.111?

Kayquedias · 2023-08-25T12:28:38Z

I am still facing the same issue with chrome driver version. I built an executable with pyinstaller, in my local machine it runs normally, but when I run the build and run the executable, it returns to me the same error

selenium.common.exceptions.SessionNotCreatedException: Message: session not created: This version of ChromeDriver only supports Chrome version 114
Current browser version is 116.0.5845.111 with binary path C:\Program Files\Google\Chrome\Application\chrome.exe``

Error above

jdholtz · 2023-08-25T12:38:48Z

however we could get rid of the old way all together as they have endpoints for all versions as well

I looked at their website here as well as their JSON endpoints and it says Chromedriver is "supported since v115.0.5763.0" so I don't believe the endpoint has older versions as well (older versions also don't show up in the known-good-versions.json link).

Is this working for both Intel and arm macs too?

People have had mixed results with this. I believe it works fine on Intel macs. This comment says that it works on his ARM mac. However, people have had issues installing the driver manually (not using this PR) and attempting to get it to work so I don't exactly know what is going on (maybe they need Rosetta?).

Edit: I also got confirmation from someone on ARM Mac using my own project that includes v3.5.3 that it works for them

jdholtz · 2023-08-25T12:40:06Z

@jdholtz would u be able to tell me what's the solution for it so far?

@orbar1 The solution was to simply use the new endpoints for downloading the Chromedriver since the Chromium team changed the download process

jdholtz · 2023-08-25T12:40:38Z

@Kayquedias did you try with undetected_chromedriver v3.5.3?

chinesedfan · 2023-08-28T12:54:44Z

undetected_chromedriver/patcher.py

@@ -72,6 +62,10 @@ def __init__(
        prefix = "undetected"
        self.user_multi_procs = user_multi_procs

+        self.is_old_chromedriver = version_main and version_main <= 114


@ultrafunkamsterdam Did you get this error?

File "C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\site-packages\undetected_chromedriver\__init__.py", line 255, in __init__ user_multi_procs=user_multi_procs, File "C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\site-packages\undetected_chromedriver\patcher.py", line 65, in __init__ self.is_old_chromedriver = version_main and version_main <= 114 TypeError: '<=' not supported between instances of 'str' and 'int'

Are you passing version_main in as a string? It is supposed to be None or an int

@jdholtz Thanks for your quick reply. I didn't notice the type at the beginning.

spacb · 2023-08-28T21:04:59Z

Sorry to bother and thanks for your amazing effort and help.
I read all the comments but I can't understand how to use it once I ran this command pip install -e git+https://github.com/jdholtz/undetected-chromedriver.git@29551bd27954dacaf09864cf77935524db642c1b#egg=undetected_chromedriver

It create an src folder within my project and then how do i use it?

jdholtz · 2023-08-28T21:13:21Z

@spacb this PR is now included in the v3.5.3 of this library, so just do pip install undetected_chromedriver==3.5.3

spacb · 2023-08-28T21:14:29Z

Ok thanks I will try that and see how to get it to work.

tylertwentyeight · 2023-08-29T02:04:32Z

@spacb this PR is now included in the v3.5.3 of this library, so just do pip install undetected_chromedriver==3.5.3

I’ve been experiencing the same issues. This fixed it for me.

irfanlogisly · 2023-08-29T05:51:36Z

Hi Anyone using selenium java and facing issue on chrome 116.0.5845.111?

Hi, does anyone have solution for this issue ?

Kinuseka · 2023-08-29T07:03:23Z

Hi Anyone using selenium java and facing issue on chrome 116.0.5845.111?

Hi, does anyone have solution for this issue ?

Hi we cannot help you with that here, this is a discussion box for a PR about fixing an issue on undetected chromedriver,
I suggest opening an issue here as this is a python related project. Without further details we cannot help you with that.

zakcali · 2023-09-17T08:51:20Z

I now get this error with version 3.5.3:
This version of ChromeDriver only supports Chrome version 117
Current browser version is 116.0.5845.188

zakcali · 2023-09-17T09:04:25Z

I've ugraded my chrome version (manually from browser menu) to 117 and problem is solved

markthepixel · 2023-09-17T16:50:42Z

I now get this error with version 3.5.3: This version of ChromeDriver only supports Chrome version 117 Current browser version is 116.0.5845.188

If we updated uc to use the Selenium Manager is Selenium 4.12.0 it'll resolve this issue
#1571

jdholtz added 2 commits August 16, 2023 07:42

Add support for Chromedriver versions 115+

25877fd

Set exe name before using it

9102a51

ugorsahin mentioned this pull request Aug 16, 2023

urllib.error.HTTPError: HTTP Error 404: Not Found ugorsahin/TalkingHeads#25

Closed

Extract all contents from zip file when downloading Chromedriver

f91b7d8

Extracting just one file that was in a directory caused an error on Windows. Extracting all fixes this issue.

mshot mentioned this pull request Aug 16, 2023

chrome 116. Will UC work? #1480

Closed

Fix architecture checks for Mac devices

0ae871f

Fix reference to is_old_chromedriver when setting platform name

1ef8e8c

Remove ARM specification for Macs

29551bd

Apparently the x64 version works on ARM devices with Mac, so the platform now just chooses the correct platform name based on the Chromedriver version requested

ultrafunkamsterdam merged commit 01417d0 into ultrafunkamsterdam:master Aug 25, 2023
4 checks passed

EastArctica mentioned this pull request Aug 25, 2023

[BUG] Title "Chrome driver not compatible with version 116" charlesbel/Microsoft-Rewards-Farmer#316

Closed

2 tasks

chinesedfan reviewed Aug 28, 2023

View reviewed changes

This was referenced Aug 29, 2023

Error loading cookies into chromedriver v116 #1521

Closed

Error loading cookies into chromedriver v116, Message: invalid cookie domain (Session info: chrome=116.0.5845.111) #1522

Open

jdholtz deleted the 115-compatibility branch August 31, 2023 14:03

Add support for downloading Chromedriver versions 115+ #1478

Add support for downloading Chromedriver versions 115+ #1478

Conversation

jdholtz commented Aug 16, 2023 • edited Loading

jdholtz commented Aug 16, 2023 • edited Loading

azorkai commented Aug 16, 2023 • edited Loading

jdholtz commented Aug 16, 2023

azorkai commented Aug 16, 2023

jdholtz commented Aug 16, 2023 • edited Loading

Kinuseka commented Aug 17, 2023

neekrom commented Aug 17, 2023

jdholtz commented Aug 17, 2023 • edited Loading

neekrom commented Aug 17, 2023

jdholtz commented Aug 17, 2023

matthewngc commented Aug 17, 2023

jdholtz commented Aug 17, 2023

neekrom commented Aug 17, 2023

matthewngc commented Aug 17, 2023

jdholtz commented Aug 17, 2023

Kinuseka commented Aug 17, 2023

jdholtz commented Aug 17, 2023

m3az88 commented Aug 17, 2023

benfield97 commented Aug 17, 2023

dexedrine-01 commented Aug 17, 2023 • edited Loading

m3az88 commented Aug 17, 2023

charlbury commented Aug 24, 2023

jdholtz commented Aug 24, 2023

charlbury commented Aug 24, 2023

jdholtz commented Aug 24, 2023

CB108 commented Aug 24, 2023

charlbury commented Aug 24, 2023

keanureano commented Aug 25, 2023

orbar1 commented Aug 25, 2023

ultrafunkamsterdam commented Aug 25, 2023

ultrafunkamsterdam commented Aug 25, 2023 • edited Loading

irfanlogisly commented Aug 25, 2023

Kayquedias commented Aug 25, 2023

jdholtz commented Aug 25, 2023 • edited Loading

jdholtz commented Aug 25, 2023

jdholtz commented Aug 25, 2023

chinesedfan Aug 28, 2023

Choose a reason for hiding this comment

jdholtz Aug 28, 2023

Choose a reason for hiding this comment

chinesedfan Aug 28, 2023

Choose a reason for hiding this comment

spacb commented Aug 28, 2023

jdholtz commented Aug 28, 2023

spacb commented Aug 28, 2023

tylertwentyeight commented Aug 29, 2023

irfanlogisly commented Aug 29, 2023

Kinuseka commented Aug 29, 2023

zakcali commented Sep 17, 2023

zakcali commented Sep 17, 2023

markthepixel commented Sep 17, 2023

jdholtz commented Aug 16, 2023 •

edited

Loading

jdholtz commented Aug 16, 2023 •

edited

Loading

azorkai commented Aug 16, 2023 •

edited

Loading

jdholtz commented Aug 16, 2023 •

edited

Loading

jdholtz commented Aug 17, 2023 •

edited

Loading

dexedrine-01 commented Aug 17, 2023 •

edited

Loading

ultrafunkamsterdam commented Aug 25, 2023 •

edited

Loading

jdholtz commented Aug 25, 2023 •

edited

Loading