2FA element targets specified in src/var.py aren't on current login flow page

[rossry]

Describe the bug
The element targets specified in src/var.py don't seem to appear on the login flow page for users with a SMS security code. (Perhaps the page has been updated?) This causes the login attempt to timeout instead of proceeding to step_two_fa_notification().

src/var.py has:

TWO_FA_INPUT_EL_ID = os.environ.get('IBEAM_TWO_FA_INPUT_EL_ID', 'ID@@chlginput')

but my manual login attempt gives me a page with this element for the input box (see also screenshot):

<input type="text" class="form-control form-control-lg xyz-silver-response" name="silver-response" id="xyz-field-silver-response" placeholder="Security Code" aria-label="Security Code" autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" required="" aria-required="true">

To Reproduce
Steps to reproduce the behavior:

1. Set env.list to IBEAM_TWO_FA_HANDLER=GOOGLE_MSG and IBEAM_TWO_FA_SELECT_TARGET=One Time Passcode.
2. Observe no "Web messages is not authenticated..." message as claimed on the wiki -- instead, the log goes straight from I|Submitting the form to E|Timeout reached when waiting for authentication.

Expected behavior
Observe a "Web messages is not authenticated..." message (when running with GOOGLE_MSG for the first time) as claimed on the wiki. (Afterwards, use that handler to get the 2FA code from the message and use the code to login.)

Environment
IBeam version: 0.5.5 (from docker pull voyz/ibeam)
Docker image or standalone: Docker image
Python version (standalone users only): []
OS: Ubuntu 22.04.1 LTS

Additional context
I am hoping to get the GOOGLE_MSG 2FA logic set up in a docker instance, using the code from an SMS message to login. I understand that it's not possible(?) to set an login back to SMS auth after setting it to app-based auth, so I'm using a fresh secondary login (created with effectively read-only permissions on my trading account) that has defaulted to SMS auth.

Suggest a Fix
I suspect that ibeam/src/var.py should be changed at lines including (but perhaps not limited to?):

• TWO_FA_EL_ID on line 155
• TWO_FA_NOTIFICATION_EL on line 158
• TWO_FA_INPUT_EL_ID on line 161
  but I don't feel familiar enough with the codebase to suggest what good target specifications should be.

Additionally, it appears to me that the login flow page has elements for all 2FA status possibilities in the DOM, just with display:none. I'm not sure whether this is a problem for the search strategy?

[Voyz]

Hey @rossry thanks for the detailed bug description and for suggesting the fix. You're correct, it seems that the website has changed - that happens sometimes and we need to adjust the variables to match it.

Setting env var:

IBEAM_TWO_FA_EL_ID=ID@@xyz-field-silver-response
IBEAM_TWO_FA_INPUT_EL_ID=ID@@xyz-field-bronze-response

Should fix the issue.

Annoyingly, IBKR sometimes rolls out different versions of their website to different users (I guess based on geography), hence I'm currently reluctant to modify these vars straight away. However, this seems to be the problem also described in other issues previously: #198 #217 Seeing that this is a repeated problem, I'm going to modify these variables permanently, hoping that this is a consistent change for us all. I'll update this issue when this change is rolled out, while for now you can fix it by setting the env vars above.

---

I understand that it's not possible(?) to set an login back to SMS auth after setting it to app-based auth,

Sadly yes, it is not possible.

Additionally, it appears to me that the login flow page has elements for all 2FA status possibilities in the DOM, just with display:none. I'm not sure whether this is a problem for the search strategy?

If they haven't changed too much I think it should be fine. That has always been the case.

[rossry]

Appreciate the snappy response!

Setting those env vars got me further, but uncovered a second apparent issue -- it looks like ‎GoogleMessagesTwoFaHandler is getting stuck at a later point. I'll describe the situation below, but let me know if you'd prefer I move to a new ticket.

---

Logs (example at bottom of message) now suggest that the login logic is recognizing that 2FA is required and is proceeding to calling GoogleMessagesTwoFaHandler.get_two_fa_code(). There we end up hanging in the WebDriverWait on line 60 -- see log at bottom.

Hypothesizing further, I'd hazard a guess that the page at https://messages.google.com/web has changed? It looks like it expects the logged-out page to have a .bigger-qr-code that it can then get attribute data-qr-code out of. Instead, the page I get requires you to first click on a link, which we can select with .use-without-account-link.

If that's right, then I'd guess that line 56 and 62 should change to instead use a new _GOOG_AUTH_REQUIRED_CLASS (set from env var in the top section, but defaulting to use-without-account-link), line 65 should click the new sms_auth_el and then wait until the _GOOG_QR_CODE_CLASS and _GOOG_AUTH_REMEMBER_CLASS have both appeared before clicking the _GOOG_AUTH_REMEMBER_CLASS, and line 67 should do its own driver_2fa.find_elements(By.CLASS_NAME, _GOOG_QR_CODE_CLASS) instead of using sms_auth_el.

I'd write the PR, but I've only been running the containerized version for now so it's not trivial for me to play around and check whether it works.

---

2024-12-13 02:26:09,597|I| No active sessions, logging in...
2024-12-13 02:26:09,597|I| Loading auth webpage at https://localhost:5000/sso/Login?forwardTo=22&RL=1&ip2loc=on
2024-12-13 02:26:18,331|D| Targets: {'PASSWORD': Target(NAME@@password), 'SUBMIT': Target(CSS_SELECTOR@@.btn.btn-lg.btn-primary), 'SUCCESS': Target(TAG_NAME@@Client login succeeds), 'IBKEY_PROMO': Target(CLASS_NAME@@ibkey-promo-skip), 'TWO_FA': Target(ID@@xyz-field-silver-response), 'TWO_FA_NOTIFICATION': Target(CLASS_NAME@@login-step-notification), 'TWO_FA_INPUT': Target(ID@@xyz-field-bronze-response), 'TWO_FA_SELECT': Target(ID@@sf_select), 'USER_NAME': Target(NAME@@username), 'ERROR': Target(CSS_SELECTOR@@.xyz-errormessage)}
2024-12-13 02:26:18,387|I| Gateway auth webpage loaded
2024-12-13 02:26:18,387|I| Login attempt number 1
2024-12-13 02:26:18,702|D| target: Target(NAME@@username)
2024-12-13 02:26:24,126|I| Submitting the form
2024-12-13 02:26:25,234|D| target: Target(ID@@xyz-field-silver-response)
2024-12-13 02:26:25,234|I| Credentials correct, but Gateway requires two-factor authentication.
2024-12-13 02:26:25,235|I| Attempting to acquire 2FA code from: GoogleMessagesTwoFaHandler(driver_path=/usr/bin/chromedriver)
2024-12-13 02:30:26,912|I| Cleaning up the resources. Google MSG Driver: <selenium.webdriver.chrome.webdriver.WebDriver (session="ca70ef70c11ac852dec50d858f6d778a")>
2024-12-13 02:30:26,923|E| Error encountered while acquiring 2FA code. 
Exception:
  File "/srv/ibeam/ibeam_starter.py", line 172, in <module>
    success, shutdown, status = client.start_and_authenticate()
  File "/srv/ibeam/src/gateway_client.py", line 62, in start_and_authenticate
    success, shutdown, status = self.strategy_handler.try_authenticating(request_retries=request_retries)
  File "/srv/ibeam/src/handlers/strategy_handler.py", line 85, in try_authenticating
    return self._authentication_strategy_B(status, request_retries)
  File "/srv/ibeam/src/handlers/strategy_handler.py", line 140, in _authentication_strategy_B
    return self._log_in(status)
  File "/srv/ibeam/src/handlers/strategy_handler.py", line 151, in _log_in
    success, shutdown = self.login_handler.login()
  File "/srv/ibeam/src/handlers/login_handler.py", line 447, in login
    self.attempt(targets, wait_and_identify_trigger, driver)
  File "/srv/ibeam/src/handlers/login_handler.py", line 380, in attempt
    trigger, target = self.step_two_fa(targets, wait_and_identify_trigger, driver, self.two_fa_handler, self.strict_two_fa_code)
  File "/srv/ibeam/src/handlers/login_handler.py", line 258, in step_two_fa
    two_fa_code = handle_two_fa(two_fa_handler, driver, strict_two_fa_code)
  File "/srv/ibeam/src/handlers/login_handler.py", line 81, in handle_two_fa
    two_fa_code = two_fa_handler.get_two_fa_code(driver)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/ibeam/src/two_fa_handlers/google_msg_handler.py", line 60, in get_two_fa_code
    WebDriverWait(driver_2fa, 240).until(any_of(sms_auth_present, sms_code_present))
  File "/opt/venv/lib/python3.11/site-packages/selenium/webdriver/support/wait.py", line 95, in until
    raise TimeoutException(message, screen, stacktrace)

  <class 'selenium.common.exceptions.TimeoutException'> Message: 

2024-12-13 02:30:26,925|W| No 2FA code returned. Aborting authentication.
2024-12-13 02:30:27,925|I| Cleaning up the resources. Display: <pyvirtualdisplay.display.Display object at 0x7f9370a8d290> | Driver: <selenium.webdriver.chrome.webdriver.WebDriver (session="0c3ae8e8de8f12aa240918066e4082f5")>
2024-12-13 02:30:27,982|I| Logging in failed

[rossry]

More generally (and feel free to move this to another ticket), I do wonder how much it would be possible to have records of what a webdrivered page (IB's, or in this case Google's) is supposed to look like, and raise a more pointed message to the user about what is going on if it's clear that the page has been changed. Not sure exactly what the logical check would look like, but surely something is possible here.

(Even better would be testing to autonomously raise an alert about key page changes as an issue in github.)

[Voyz]

hey @rossry thanks for pointing all of these things out. Let me address them.

To start with:

I'd write the PR, but I've only been running the containerized version for now so it's not trivial for me to play around and check whether it works.

If you'd like to play around and check whether it works on your own within a containerised deployment, you can provide a modified version of this Google Msg Handler as a Custom Handler. Just copy paste the code to a new file, modify it, provide it through Inputs Directory and tell IBeam to use it by setting IBEAM_CUSTOM_TWO_FA_HANDLER.

My IBKR account doesn't use SMS authentication so I won't be able to do any tests on these fixes.

Hypothesizing further, I'd hazard a guess that the page at https://messages.google.com/web has changed?

Possibly, yes. It's not the first time, and it's expected this would happen every now and then 🤷‍♂️ We just gotta step in and fix it.

If that's right, then I'd guess that line 56 and 62 should change to instead use a new _GOOG_AUTH_REQUIRED_CLASS (set from env var in the top section, but defaulting to use-without-account-link), line 65 should click the new sms_auth_el and then wait until the _GOOG_QR_CODE_CLASS and _GOOG_AUTH_REMEMBER_CLASS have both appeared before clicking the _GOOG_AUTH_REMEMBER_CLASS, and line 67 should do its own driver_2fa.find_elements(By.CLASS_NAME, _GOOG_QR_CODE_CLASS) instead of using sms_auth_el.

I appreciate you going into this length to explain what needs to be changed. I've attempted implementing the first step by adding:

            pair_button_present = EC.presence_of_element_located((By.CLASS_NAME, 'use-without-account-link'))
            WebDriverWait(driver_2fa, 240).until(pair_button_present)
            pair_button_el = driver_2fa.find_element(By.CLASS_NAME, 'use-without-account-link')
            pair_button_el.click()

After we open the webpage. You can test it in 0.5.7-rc1.

The next steps aren't very clear to me, sorry. Would you be able to bulletpoint each change you require to make? Or feel free to post the modified code and I can publish a release candidate with it for you to test.

I do wonder how much it would be possible to have records of what a webdrivered page (IB's, or in this case Google's) is supposed to look like, and raise a more pointed message to the user about what is going on if it's clear that the page has been changed.

Yes, this is not a bad idea. For Google Messages this would work better, since IBKR changes its webpage depending on the type of 2FA you have enabled, and there's at least 3 different ways to do this. Furthermore, for this to be an CI/CD-type action there would need to be someone's credentials made available to carry out the authentication flow automatically. Nevertheless, doing just the basic login page wouldn't hurt. I don't have the resources to implement it at the moment, would you be interested in contributing it?