Confluence cloud interop broken

[Bdegraaf1234]

When upgrading from release v0.8.5 to v0.8.6, the catalog-server:kafka-init container is unable to connect to confluent cloud.

The error occurs during authentication:

[2024-03-18 11:28:22,586] ERROR - Schema registry ${confluence_cloud_url} not ready, responded with HTTP 401 Unauthorized: REST call to <${confluence_cloud_url}/subjects> failed (HTTP code 401 Unauthorized): {"error_code":401,"message":"Unauthorized"} (Emitters.kt:231)

The problem seems to originate here. I believe the issue was introduced in this commit aefea95

Debugging so far:

• Verified that the credentials are available to the pod
• Verified that manually providing these credentials to the url specified (${confluence_cloud_url}/subjects) works

[pvannierop]

I will share my analysis of this issue here. Confluent Cloud requires Basic Auth where the username is the api key and the password is the api secret. The error appears to be that a 401 http response is returned when making a request to the /subjects endpoint on Confluent Cloud (part of initializing the schemas client):

[2024-03-22 13:31:20,276] ERROR - Schema registry https://<hidden-hostname> not ready, responded with HTTP 401 Unauthorized: REST call to <https://<hidden-hostname>/subjects> failed (HTTP code 401 Unauthorized): {"error_code":401,"message":"Unauthorized"} (Emitters.kt:231)

The ktor client is instructed to send Basic Auth header when an apiKey and apiSecret are provided (See class SchemaRegistry.kt):

private val schemaClient: SchemaRetriever = schemaRetriever(baseUrl) {
        httpClient {
            timeout(10.seconds)
            if (apiKey != null && apiSecret != null) {
                install(Auth) {
                    basic {
                        credentials {
                            BasicAuthCredentials(username = apiKey, password = apiSecret)
                        }
                        realm = "Access to the '/' path"
                    }
                }
            }
        }
    }

401 http responses are common for Basic Auth because it is the first response of the server; after this the client can send the Basi Auth header (see here). It may be that we are experiencing 401 errors caused by this first server response. According to the docs, we can instruct ktor with the sendWithoutRequest parameter to send the Basic Auth header immediately to prevent the 401 http response, like so:

private val schemaClient: SchemaRetriever = schemaRetriever(baseUrl) {
        httpClient {
            timeout(10.seconds)
            if (apiKey != null && apiSecret != null) {
                install(Auth) {
                    basic {
                        sendWithoutRequest { true }
                        credentials {
                            BasicAuthCredentials(username = apiKey, password = apiSecret)
                        }
                        realm = "Access to the '/' path"
                    }
                }
            }
        }
    }

This does not solve the issue with the registration of schemas to Confluent Cloud, however.

I am certain failure to include Basic Auth headers is the cause of the error, because the error on the /subjects endpoint no longer occurs when I manually add the header in this call:

                        httpClient.request<List<String>> {
                            url("subjects")
                             // Force add Basic Auth
                            if (apiKey != null && apiSecret != null) {
                                logger.info("Basic auth applied.")
                                basicAuth(apiKey, apiSecret)
                            }
                       }

As far as I can see, the root cause of this problem appears to be that ktor does not correctly include the Basic Auth headers, although instructed to do so in the config.

[Bdegraaf1234]

Solved by #369