IBKey login fails, "An error occurred while communicating with the server."

[rossry]

Describe the bug
As recently as Dec 12, I was able to run ibeam and authenticate by responding to the IBKey popup. At some point in the past week, however, this stopped working, and now the IBKey popup flow breaks; this only occurs for login attempts happening through ibeam (or its proxy server), not when I log in to the same account directly by web (for which the IBKey popup flow works just fine). I've reproduced the behavior on two IB logins and three machines with distinct IP addresses.

To Reproduce
Steps to reproduce the behavior:

docker pull voyz/ibeam
docker run --env-file env.list -v ./inputs:/srv/inputs -v ./outputs:/srv/outputs -p 5000:5000 voyz/ibeam

Expected behavior

• IBKey notification appears
• open the notification -> select "approve" -> authenticate
• login flow continues in ibeam

Observed behavior

• IBKey notification appears
• open the notification -> select "approve"
• IBKey shows the following:
• ibind never receives a 2FA/IBKey auth

Reproduced across two logins, three machines with distinct IP addresses, and multiple days. Same behavior whether the login attempt is from ibeam_starter itself or navigating to https://localhost:5000/sso/Login?RL=1&locale=en_US once ibeam is running on port 5000.

Environment
IBeam version: 0.5.6
Docker image or standalone: docker image
Python version (standalone users only): []
OS: Ubuntu 22.04 (reproduced on 24.04)

Additional context
This might be a red herring, but it's my current best guess. Navigating to https://localhost:5000/sso/Login?RL=1&locale=en_US using Chrome reveals that a POST to /portal.proxy/v1/gsss/bulletins is getting 404. This 404 does not occur when hitting https://ndcdyn.interactivebrokers.com/sso/Login?RL=1&locale=en_US directly in Chrome incognito. Logs from a proxy that I introduced for better logging:

...
2024-12-19 11:41:10 [INFO] Forwarding GET request to: https://localhost:5000/images/common/logos/ibkr/interactive-brokers.svg
2024-12-19 11:41:11 [INFO] 24.130.62.43 - - [19/Dec/2024 11:41:11] "GET /images/common/logos/ibkr/interactive-brokers.svg HTTP/1.1" 200 -
2024-12-19 11:41:11 [INFO] 24.130.62.43 - - [19/Dec/2024 11:41:11] "GET /images/common/logos/ibkr/interactive-brokers.svg HTTP/1.1" 200 -
2024-12-19 11:41:11 [INFO] Forwarding GET request to: https://localhost:5000/fonts/proxima-nova/Proxima-Nova-Semibold.woff2
2024-12-19 11:41:11 [INFO] 24.130.62.43 - - [19/Dec/2024 11:41:11] "GET /fonts/proxima-nova/Proxima-Nova-Semibold.woff2 HTTP/1.1" 200 -
2024-12-19 11:41:11 [INFO] 24.130.62.43 - - [19/Dec/2024 11:41:11] "GET /fonts/proxima-nova/Proxima-Nova-Semibold.woff2 HTTP/1.1" 200 -
2024-12-19 11:41:11 [INFO] Forwarding POST request to: https://localhost:5000/portal.proxy/v1/gsss/bulletins
2024-12-19 11:41:11 [INFO] 24.130.62.43 - - [19/Dec/2024 11:41:11] code 404, message Not Found
2024-12-19 11:41:11 [INFO] 24.130.62.43 - - [19/Dec/2024 11:41:11] "POST /portal.proxy/v1/gsss/bulletins HTTP/1.1" 404 -
2024-12-19 11:41:11 [INFO] Forwarding GET request to: https://localhost:5000/en/includes/general/gdpr-am.php
2024-12-19 11:41:11 [INFO] 24.130.62.43 - - [19/Dec/2024 11:41:11] "GET /en/includes/general/gdpr-am.php HTTP/1.1" 200 -
2024-12-19 11:41:11 [INFO] 24.130.62.43 - - [19/Dec/2024 11:41:11] "GET /en/includes/general/gdpr-am.php HTTP/1.1" 200 -
...

This request is created in the following clientside javascript on /sso/Login:

<script type="text/javascript">
            function getMaintenaceMsg() {
                $('.gstat').hide();
                var params = {
                    p: 'login',
                    type: 'maintenance',
                    format: 'webapp'
                };
                $.ajax({
                    type: 'POST',
                    url: '/portal.proxy/v1/gsss/bulletins',
                    data: JSON.stringify(params),
                    contentType: 'application/json; charset=utf8',
                    success: function(data, txtStatus) {
                        if (data) {
                            $('.gstat').html(data).show();
                        }
                    }
                });
            }

            if (window != top) {
                top.location.href = location.href;
            }

            $(document).ready(function() {
                window.IBSSO.XYZ.inject({
                  WRAPPER: '.loginformWrapper',
                  INJECTED_CALLBACK: function(err) {
                      if (err) return;
                      $('.xyz-append-hook').append('<div class="gstat pt-3"></div>');
                      getMaintenaceMsg();
                  },
                  LOCALE: 'en_US',
                  SERVICE: 'AM.LOGIN',
                  SHOW_LOGIN_TYPE: true,
                  LOGIN_TYPE: '1',
                  SHOW_TITLE: true,
                  PASSWORD_RESET_URL: '/credential.recovery/login-help',
                  SHOW_PASSWORD_RESET: true,
                  SHOW_CRED_RECOVERY: true,
                  SHOW_REGISTER_LINK: true,
                  SHOW_RO_OPTION: false,
                  PAGE_TITLE: 'Login'
                })

                showCookieConsent();
            });

            function showCookieConsent() {
                if (document.location.hostname === 'localhost') {
                    return;
                }
                try {
                    if ((new URLSearchParams(document.location.search)).has('client_app')) {
                        // embedded context
                        return;
                    }
                } catch (e) {}
                $.ajax({
                    type: 'GET',
                    url: '/en/includes/general/gdpr-am.php',
                    success: function(data) {
                        if (data) {
                            $('body').prepend(data);
                            if ($('#cookie-modal').length) {
                                $('#cookie-modal').modal('show');
                            }
                        }
                    }
                });
            }
        </script>

inputs/conf.yaml:

ip2loc: "US"
proxyRemoteSsl: true
proxyRemoteHost: "https://3.api.ibkr.com"
listenPort: 5000
listenSsl: true
svcEnvironment: "v1"
sslCert: "cacert.jks"
sslPwd: "mywebapi"
authDelay: 3000
portalBaseURL: ""
serverOptions:
    blockedThreadCheckInterval: 1000000
    eventLoopPoolSize: 20
    workerPoolSize: 20
    maxWorkerExecuteTime: 100
    internalBlockingPoolSize: 20
cors:
    origin.allowed: "*"
    allowCredentials: false
webApps:
    - name: "demo"
      index: "index.html"
ips:
  allow:
    - 192.*
    - 131.216.*
    - 127.0.0.1
    - 0.0.0.0
    - 172.17.0.*
  deny:
    - 212.90.324.10

inputs/san.cnf taken from template. I am using self-signed cacert.jks / cacert.pem certificates produce according to instructions; these same certs had previously worked fine for ibeam login.

env.list:

IBEAM_AUTHENTICATION_STRATEGY=B
IBEAM_LOG_LEVEL=DEBUG

IBEAM_INPUTS_DIR=/srv/inputs
IBEAM_OUTPUTS_DIR=/srv/outputs
IBEAM_ERROR_SCREENSHOTS=True
IBEAM_PAGE_LOAD_TIMEOUT=30
IBEAM_OAUTH_TIMEOUT=60

IBEAM_TWO_FA_EL_ID=ID@@xyz-field-silver-response
IBEAM_TWO_FA_INPUT_EL_ID=ID@@xyz-field-bronze-response

I spent a supremely unproductive 32 minutes on the phone with IB's Technical Assistance team; when I confirmed that I was able to log in correctly from other machines than the affected one, the rep suggested that I "talk with my company's IT team", "talk with my company's networking team", and that they "do not troubleshoot device-specific issues" (which this was as I was able to login from other devices).

Suggest a Fix
Honestly, I'm baffled.

[Voyz]

hey @rossry thanks for describing your issue in so much detail and I'm sorry you're struggling with it.

To be honest, I'm baffled as well. If it works on some machines but not on the others than it could be some network configuration related to that bad machine. But as for specifics, I wouldn't know what else to suggest. Maybe try a different cloud provider? Can you see anything else that could be done here?