Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Authentication Failures again #28

Open
philosowaffle opened this issue May 18, 2021 · 83 comments · Fixed by #29
Open

Authentication Failures again #28

philosowaffle opened this issue May 18, 2021 · 83 comments · Fixed by #29
Assignees

Comments

@philosowaffle
Copy link

Howdy,

Seems Garmin has yet again tweaked some things and I have several users (including myself) running into auth failures.
philosowaffle/peloton-to-garmin#109

assert self.user.authenticate(), "Failed to authenticate garmin user."
AssertionError: Failed to authenticate garmin user.

Thank you in advance for any help!

@La0
Copy link
Owner

La0 commented May 18, 2021

Hey @philosowaffle ,

I reproduce the issue: the project unit tests are failing on the authentication.

I'll try to quickly find a fix, I'll keep you posted here

@La0 La0 self-assigned this May 18, 2021
@La0
Copy link
Owner

La0 commented May 18, 2021

This time it's Cloudflare rejecting the requests.

I get a nice error page, with the infamous 1020 Cloudflare error. This may take some time to fix...

Screenshot 2021-05-18 at 17-53-28 Garmin Maintenance

@philosowaffle
Copy link
Author

Oh no :(

Cloudflare is awesome for keeping our favorite sites online, but as an indy dev who just wants to automate things, I hate it lol

@La0
Copy link
Owner

La0 commented May 18, 2021

You can try this rough patch : https://github.com/La0/garmin-uploader/pull/29/files

I got some auth requests to work with this code, but now i'm getting HTTP 429 Too many requests 😭

@philosowaffle
Copy link
Author

Thank you for the quick patch attempt! I'm at work so won't be able to verify till tonight.

On the 429 error, I've had good luck with switching onto a VPN. The 429 seems tied to IP address, so if you hop on a vpn then you get another 10-20 requests before it throws 429 again. And I think the cool off period is about 1hr.

@kenlpeters
Copy link

How can we pull this fix. I would like to test it.

@Abasz
Copy link

Abasz commented May 18, 2021

I had similar problem with my code. Actually itis the cloudflare protection that kicks in. I got it working using cloudscraper library to make the request. Here is a similar py libarary where this is duscussed.

cyberjunky/python-garminconnect#46

@BebeMischa
Copy link

i can confirm Cyberjunky fix looks like its working, I'm testing it live now

@La0
Copy link
Owner

La0 commented May 18, 2021

@kenlpeters You can try my patch with the following pip command:

pip install https://github.com/La0/garmin-uploader/archive/fix-cloudfare-errors.zip

@kenlpeters
Copy link

Still get authentication failed. Not even too many requests

@philosowaffle
Copy link
Author

I checked out fix-cloudfare-errors, switched conftest.py to use my dev credentials, then ran the unit tests. Unfortunately, most are still failing with Authentication Failed. Same error if I use the default guploader creds as well.

@BebeMischa
Copy link

Double integration (two accounts on one HA instance) running OK for more than 3 hours now, see

home-assistant/core#50822

@La0
Copy link
Owner

La0 commented May 18, 2021

I'll try clouscrapper tommorow

@La0
Copy link
Owner

La0 commented May 19, 2021

The CI is ok on the current branch, except on Python 3.7 🤷‍♂️

@kenlpeters
Copy link

@La0 - let me know when 3.7 is working. Unfortunately that is the version I am running... go figure.

@La0
Copy link
Owner

La0 commented May 19, 2021

@kenlpeters I just re-ran the CI, and it worked. I get the same results locally on multiple Python versions, and on multiple accounts / servers.

Please test this version !

@kenlpeters
Copy link

I must not be downloading the proper version of the api.py. Can you link the right version.

@kenlpeters
Copy link

Any update on this. I am still running into the authentication error. I just think it is a fact that I am not pulling the proper version of the software.

@superm1
Copy link

superm1 commented May 20, 2021

@kenlpeters I just re-ran the CI, and it worked. I get the same results locally on multiple Python versions, and on multiple accounts / servers.

Please test this version !

It's working for me now thanks!

@kenlpeters
Copy link

Great how to get the newest version.

@superm1
Copy link

superm1 commented May 20, 2021

Any update on this. I am still running into the authentication error. I just think it is a fact that I am not pulling the proper version of the software.

For me I had the old version installed in ~/.local and nothing else there. So I removed all of that directory, then cloned this repo, switched branches and ran

python3 setup.py install --user

@kenlpeters
Copy link

kenlpeters commented May 20, 2021 via email

@kenlpeters
Copy link

I think I downloaded the proper branch (cloud fare fixes) and unzipped and run the setup.py. Still I am getting Authenication error.

@kenlpeters
Copy link

I tried the CI branch as well and it did not work. So obviously I am downloading the right branch.

@superm1
Copy link

superm1 commented May 20, 2021

I think I downloaded the proper branch (cloud fare fixes) and unzipped and run the setup.py. Still I am getting Authenication error.

Assuming you're not hitting a different failure and are running the wrong code still - if I was to guess, you're still getting the wrong copy from a local install somewhere else that has higher precedence (like site-packages or dist-packages). You should try doing pip uninstall garmin-uploader.

@kenlpeters
Copy link

Did you download the cloud fare fixes branch?

@kenlpeters
Copy link

Maybe i have not signed up so I am not sure what questions they ask. If one is what is the IP address your application will making calls from, then it definitely can be a white listing component.

@philosowaffle
Copy link
Author

philosowaffle commented May 26, 2021

Two things I tested just now that seems to be working:

  1. I updated Python to 3.9.5 (this may be irrelevant but want to mention just in case)
  2. I checked out and ran the cloudscraper branch.

For 1 - I chose to do this after reading through this linked issue

For 2 - I want to note that I may have run the wrong branch when I previously attempted to test the cloudscraper fix. But for the life of me I cannot remember. Sorry for the confusion, but it does seem to be working now.

Oddly enough, I tried 5 times, it worked on 4 attempts, failed on 1.

G:\Code\Git\garmin-uploader>git status
On branch cloudscraper
Your branch is up-to-date with 'origin/cloudscraper'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)

        gupload.ini

nothing added to commit but untracked files present (use "git add" to track)

G:\Code\Git\garmin-uploader>pip install .    
Processing g:\code\git\garmin-uploader
  DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default.
   pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555.
Collecting cloudscraper==1.2.58
  Using cached cloudscraper-1.2.58-py2.py3-none-any.whl (96 kB)
Requirement already satisfied: six>=1.10.0 in c:\users\bailey\appdata\local\programs\python\python39\lib\site-packages (from garmin-uploader==1.0.10) (1.16.0)
Requirement already satisfied: requests>=2.9.2 in c:\users\bailey\appdata\local\programs\python\python39\lib\site-packages 
(from cloudscraper==1.2.58->garmin-uploader==1.0.10) (2.25.1)
Collecting pyparsing>=2.4.7
  Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting requests-toolbelt>=0.9.1
  Using cached requests_toolbelt-0.9.1-py2.py3-none-any.whl (54 kB)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in c:\users\bailey\appdata\local\programs\python\python39\lib\site-packages (from requests>=2.9.2->cloudscraper==1.2.58->garmin-uploader==1.0.10) (1.26.5)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\bailey\appdata\local\programs\python\python39\lib\site-packages (from requests>=2.9.2->cloudscraper==1.2.58->garmin-uploader==1.0.10) (2020.12.5)
Requirement already satisfied: chardet<5,>=3.0.2 in c:\users\bailey\appdata\local\programs\python\python39\lib\site-packages (from requests>=2.9.2->cloudscraper==1.2.58->garmin-uploader==1.0.10) (4.0.0)
Requirement already satisfied: idna<3,>=2.5 in c:\users\bailey\appdata\local\programs\python\python39\lib\site-packages (from requests>=2.9.2->cloudscraper==1.2.58->garmin-uploader==1.0.10) (2.10)
Using legacy 'setup.py install' for garmin-uploader, since package 'wheel' is not installed.
Installing collected packages: requests-toolbelt, pyparsing, cloudscraper, garmin-uploader
  Attempting uninstall: garmin-uploader
    Found existing installation: garmin-uploader 1.0.9
    Uninstalling garmin-uploader-1.0.9:
      Successfully uninstalled garmin-uploader-1.0.9
    Running setup.py install for garmin-uploader ... done
Successfully installed cloudscraper-1.2.58 garmin-uploader-1.0.10 pyparsing-2.4.7 requests-toolbelt-0.9.1

G:\Code\Git\garmin-uploader>gupload tests/sample_file.tcx
2021-05-26 14:56:49,964 [INFO] Try to login on GarminConnect...
2021-05-26 14:56:51,867 [INFO] Logged in as [email protected]
2021-05-26 14:56:53,224 [INFO] Uploaded activity 6847673340 : sample_file.tcx
2021-05-26 14:56:53,224 [INFO] All done.

G:\Code\Git\garmin-uploader>python --version
Python 3.9.5

G:\Code\Git\garmin-uploader>gupload tests/sample_file.tcx
2021-05-26 14:59:30,359 [INFO] Try to login on GarminConnect...
2021-05-26 14:59:32,173 [CRITICAL] Login Failure: Login check failed.
Error: Invalid credentials

G:\Code\Git\garmin-uploader>gupload tests/sample_file.tcx
2021-05-26 14:59:49,262 [INFO] Try to login on GarminConnect...
2021-05-26 14:59:51,266 [INFO] Logged in as [email protected]
2021-05-26 14:59:51,776 [INFO] Activity already uploaded 6847673340 : sample_file.tcx
2021-05-26 14:59:51,776 [INFO] All done.

G:\Code\Git\garmin-uploader>gupload tests/sample_file.tcx
2021-05-26 15:00:11,777 [INFO] Try to login on GarminConnect...
2021-05-26 15:00:14,012 [INFO] Logged in as [email protected]
2021-05-26 15:00:15,255 [INFO] Activity already uploaded 6847673340 : sample_file.tcx
2021-05-26 15:00:15,255 [INFO] All done.

G:\Code\Git\garmin-uploader>gupload tests/sample_file.tcx
2021-05-26 15:00:34,871 [INFO] Try to login on GarminConnect...
2021-05-26 15:00:36,780 [INFO] Logged in as [email protected]
2021-05-26 15:00:37,272 [INFO] Activity already uploaded 6847673340 : sample_file.tcx
2021-05-26 15:00:37,272 [INFO] All done.

G:\Code\Git\garmin-uploader>

Other things I tested during this session (on the fix-cloudfare-errors branch):

  1. Using a VPN tested in Houston, TX, US West Coast, France -> auth failures in all cases
  2. I tried changing the UserAgent used in gupload -> no impact
  3. maybe authenticating and interacting in a browser as a user from your IP will whitelist you - my dev box is my personal pc, so I am regularly logged into Garmin Connect in the browser. I also test and work on a headless RPi4, up until just now, script authentication has failed on both.

I will continue to tentatively test the cloudscraper branch.... am a little paranoid of drawing attention myself and getting shut out again lol

@La0
Copy link
Owner

La0 commented May 27, 2021

What a mess... I'm using the last release myself and have no issues to upload activities.

I think they have some anti-cheat heuristics on a few factors:

  • headers & parameters may not have the valid country code ( a few url have hard-coded fr-FR in the lib)
  • sending requests too fast (the lib requests the login page and submit it a few milliseconds later)
  • same user-agent ?

I can't find how to get the geolocation argument from garmin (basically to tell if you are en-us or fr-fr, etc...)

Another attempt would be to try something like selenium, but that would be really heavy and hard to maintain too 😞

I don't have a silver bullet for now.

@kenlpeters
Copy link

Just an update from my end. I am using the cloud fix CI with V1 peloton-to-Garmin and it continues to work on my MS Azure Linux VM. I am using python 3.7 and my URLs are fr-FR. I apologized I have been saying my VM is an AWS one when it is an Azure in Central US.

@kenlpeters
Copy link

Let me know if there is anything I can test for you.

@dkmcgowan
Copy link

If you update v2 with cloudscrapper branch I'll pull it down and test

@dkmcgowan
Copy link

I'm the other comments can we just add a param for country code we manually put in and then add a delay for the login request and give that a shot

@philosowaffle
Copy link
Author

@dkmcgowan I have updated the V2 docker image to use the cloudscraper fix. You can pull latest and test it out. Seems to still be working for me.

@dkmcgowan
Copy link

This one is working for me, atleast the first run did to upload a few

@philosowaffle
Copy link
Author

I've integrated the CloudScraper fix into my latest build and am taking a poll (see last comment) to see who its working for.

@philosowaffle
Copy link
Author

So far the cloudscraper fix seems to be working for all of my users who have reported back (~10 so far). No one has reported that it is not working for them.

@zensonic
Copy link

I have just downloaded, compiled and installed python3.9 on raspbian.. that made things work again using gupload

2021-05-29 06:03:34,133 [INFO] Force uploading
2021-05-29 06:03:34,136 [INFO] - device 3895796269
2021-05-29 06:03:34,151 [INFO] Going to upload 368 activities to garmin connect
2021-05-29 06:07:03,077 [INFO] Uploaded files to garmin connect

@zed4
Copy link

zed4 commented May 30, 2021

cloudscraper fix seems to work for me as well. (i'm using this from within peloton-to-garmin)

@mahboc
Copy link

mahboc commented May 31, 2021

FWIW- Standard Installation on Mac Big Sur (11.13.1) Python 3.8 fails authentication.

@mahboc
Copy link

mahboc commented May 31, 2021

Following directions (cloudscraper) above seems to fix the issue for me.

@jmodena
Copy link

jmodena commented Jun 2, 2021

garmin-uploader 1.0.10 in Virtualbox machine with Ubuntu 20.04 (python 3.8.5) works perfectly.

@philosowaffle
Copy link
Author

philosowaffle commented Jun 16, 2021

@La0 - Question when you have a chance. My project has switched to using the cloudscraper fix with moderate success for most users. One issue we are seeing is when we attempt upload multiple activities at once. If we attempt to only upload 1 activity, then the upload succeeds. However, attempting to upload more than 1 at once results in an auth error with Garmin.

Below is a sample command that is executed. Email and Password are scrubbed intentionally:

C:\Users\mikel\OneDrive\Desktop\p2g_v2\python\upload\upload.exe -ge **email** -gp **password** -f "C:\Users\mikel\OneDrive\Desktop\p2g_v2\working\upload\0667494098494e6e891f97d3afa1d9a6_20_min_Low_Impact_Ride_with_Jenn_Sherman.fit" "C:\Users\mikel\OneDrive\Desktop\p2g_v2\working\upload\c92726553fb7431db53375943b0edbd5_20_min_Rock_Ride_with_Denis_Morton.fit" "C:\Users\mikel\OneDrive\Desktop\p2g_v2\working\upload\ccddfdc691e74e44b7256660e6b37796_30_min_2000s_Ride_with_Hannah_Frankson.fit"

Here is a comment with more details on repro steps.

I'm wondering if something changed in the cloudscraper fix where it attempts to authenticate more than once when uploading multiple files? Or if there is something I should be doing differently in these cases?

@PeregrineFalcon
Copy link

Want to report that using 1.0.10 is working on a Heroku install as a library dependency within https://github.com/flackdl/ifit-garmin-sync

@plandes
Copy link

plandes commented Jun 21, 2021

Hello. I am trying to incorporate this change as well, and resorted to adding api.py to my own project outright. However, I'm getting a 403 using it verbatim (with a small change of using module logging for the logger). Does anyone have any idea why I'd be getting a 403 (forbidden)?

Thanks in advance.

@hugoideler
Copy link

Hi! My experience is that the cloudscraper branch is working well, while the master branch has consistently failed (always resulting in the 'authentication failed' misnomer errors). In this thread I'm also reading a lot of positive feedback on this branch.

Would it be possible to merge the cloudscraper branch into master and bump the library version?

@kenlpeters
Copy link

Checking in here. Although the PelotonToGarminConsole is working wonderfully, I have started a new process of syncing iFit (for Norditrack) to Garmin. I am using the uploader again and it continues to get the too many requests error.

@app4g
Copy link

app4g commented Apr 17, 2023

I am getting auth errors with this and many other projects. It fails at getting the CSRF token/ticket.

I tried using https://github.com/felipeam86/garpy/tree/4ced33f4dcf018278e948b6eaea1ab6414d2ef5a and it worked.

After some looking at it, it seems that one of the difference may be where the response token is being looked for?

https://connect.garmin.com/modern?ticket=ST-0123456-aBCDefgh1iJkLmN5opQ9R-cas

coming from this method/function here
https://github.com/felipeam86/garpy/blob/4ced33f4dcf018278e948b6eaea1ab6414d2ef5a/garpy/client.py#L35

@Stoobiedoo
Copy link

After a year of working with the crowdscraper code, it is now resulting in 403 errors. Is anyone else running into connection troubles starting this week?

@app4g
Copy link

app4g commented Sep 28, 2023

yeah.. its down for most projects...
#lswiderski/mi-scale-exporter#20 (comment)

@oldnapalm
Copy link

garth is working

oldnapalm@7234be7

@pantomnesia
Copy link

pantomnesia commented Oct 5, 2023

After a year of working with the crowdscraper code, it is now resulting in 403 errors. Is anyone else running into connection troubles starting this week?

Same here. Simple test from cli

gupload -u myusername -p mypassword myfile.tcx

results in.....

2023-10-05 11:56:57,277 [INFO] Try to login on GarminConnect...
2023-10-05 11:57:00,300 [CRITICAL] Login Failure: Expecting value: line 1 column 1 (char 0)

`

@oldnapalm
Copy link

@pantomnesia see oldnapalm@7234be7 (notice that it will only fix authentication but you will still get upload error and updating activity name and type won't work because detailedImportResult no longer contains internalId oldnapalm@8344c0e).

Or just migrate to https://github.com/matin/garth

Example:

import os
import sys
import garth
from getpass import getpass

if len(sys.argv) < 2:
    print("Use %s <file name>" % sys.argv[0])
    sys.exit(1)

if not os.path.isfile(sys.argv[1]):
    print("File %s not found" % sys.argv[1])
    sys.exit(1)

tokens_dir = '~/.garth'

try:
    garth.resume(tokens_dir)
    garth.client.username
except:
    email = input("Enter email address: ")
    password = getpass("Enter password: ")
    try:
        garth.login(email, password)
        garth.save(tokens_dir)
    except Exception as exc:
        print(exc)
        sys.exit(1)

try:
    with open(sys.argv[1], "rb") as f:
        resp = garth.client.upload(f)
        print(resp)
except Exception as exc:
    print(exc)
    sys.exit(1)

sys.exit(0)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.