-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement Kafka REST from scratch #74
Conversation
* Add openapi generation and implement basic endpoint * Implement remaining endpoints * add api spec * Add cluster api and integration tests * fix null authorizedOperations * Add metadata and related urls to response * Introduce AdminClients to cache AdminClient by connection id and cluster id * refactor * broken wide * wip: needs refactor * works again * use constant var * Add tests for cluster API * Support confluent local using internal kafka rest * revert file changes * make util cass final * Add docstrings * fix * Reduce AdminClient request timeout to 10 seconds and handle exception * Use Caffeine to do time-based eviction on AdminClient instances * remove unused import * remove TODO * Put unrelated openapi spec changes back in place * Add exception mapper for UnsupportedOperationException
…inc/ide-sidecar into rohitsanj/implement-kafka-rest
Thanks, @rohitsanj. Please correct me if I'm wrong, but after this change we'll have two Kafka REST endpoints:
(The endpoints for both APIs, like nearly all the sidecar APIs) both require the session token be passed as the authz header.) Can you confirm that this approach was chosen because it makes implementing the |
I made some slight tweaks/fixes identified when testing this branch's sidecar executable against a debug instance of confluentinc/vscode:
I also ended up reverting changes made to |
@rohitsanj Did you manually create this file or did you use the GraalVM tracing agent? |
@flippingbits, I manually created the file based on this Quarkus guide. I considered using the GraalVM reachability metadata repository since |
src/main/java/io/confluent/idesidecar/restapi/cache/ClusterCache.java
Outdated
Show resolved
Hide resolved
src/main/java/io/confluent/idesidecar/restapi/cache/ClusterCache.java
Outdated
Show resolved
Hide resolved
src/main/java/io/confluent/idesidecar/restapi/cache/ClusterCache.java
Outdated
Show resolved
Hide resolved
src/main/java/io/confluent/idesidecar/restapi/cache/AdminClients.java
Outdated
Show resolved
Hide resolved
src/main/java/io/confluent/idesidecar/restapi/cache/AdminClients.java
Outdated
Show resolved
Hide resolved
src/main/java/io/confluent/idesidecar/restapi/cache/Clients.java
Outdated
Show resolved
Hide resolved
@rhauch May I get you to please take another look? Thanks 🙇 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice work, @rohitsanj. Let's get @flippingbits approval, too.
Can you also clarify what the impact will be on the UX with just this merged, but before #77 is merged? I'd rather us not get stuck with some functionality broken for more than a day or so.
Thank you!
The partitions dropdown in the message viewer tabs for confluent local topics will have "All partitions" as the only option with no ability to filter down on topic partitions. The UX/functionality remains intact for CCloud topic message viewer tabs. (Notice the expected 404 error log in bottom window) |
TLDR
This PR implements the Topic V3, Cluster V3 Kafka REST APIs from scratch using the Kafka
AdminClient
and swaps the underlying Kafka REST proxy target for confluent-local connections to point to ide-sidecar's own Kafka REST implementation, rather than to confluent-local's kafka-rest server. This will put the ide-sidecar Kafka REST impl in use from day 1 and give us early feedback to iterate.Background/Motivation
The ide-sidecar Quarkus application today exposes a wildcard route (what we call the “Kafka REST proxy”) at http://localhost:26636/kafka/v3/clusters/* and uses connection/cluster metadata passed as request headers to determine which Kafka REST server the request must be forwarded to. We want to extend this functionality to allow clients of ide-sidecar to be able to connect to any Kafka cluster, given appropriate configuration context. Based on this (internal) decision log, we decided to implement from scratch a minimal set of Kafka REST endpoints in ide-sidecar, though the major reasons for the current approach are as follows:
Description of code changes
kafka-rest.openapi.yaml
: Kafka REST OpenAPI spec taken from confluentinc/kafka-rest that is used to generate API models and resource interfaces. Seepom.xml
for theopenapi-generator-maven-plugin
configuration./internal/kafka/v3
. Hence, the path/clusters/{cluster_id}
in the original spec is instead/internal/kafka/v3/clusters/{cluster_id}
in our modified spec.target/generated-sources/openapi/src/gen/java
directory.Clients
abstract class was extracted out of theSchemaRegistryClients
class and made to be its parent. TheClients
class can be parameterized based on the client instance type that needs to be cached by connection id and client id.Clients
class caches client instances by cluster id in a Caffeine cache.AdminClients extends Clients<org.apache.kafka.clients.admin.AdminClient>
is used to cacheAdminClient
instances.expiresAfterAccess=5m
, this evicts and closes theAdminClient
client instances after 5 minutes of inactivity.io.confluent.idesidecar.restapi.kafkarest.model.Error
model, as opposed to using ide-sidecar's ownio.confluent.idesidecar.restapi.exceptions.Failure.Error
model. This is intentional in order to adhere to the kafka-rest OpenAPI spec as much as possible.LOCAL
(confluent local) connections was changed from the kafka-rest server (served athttp://localhost:8082/v3
in the confluent-local docker container), to ide-sidecar's own kafka-rest implementation served athttp://localhost:26636/internal/kafka/v3
.Design considerations
http://localhost:26636/kafka/v3
Kafka REST proxy routes. Thehttp://localhost:26636/internal/kafka/v3
endpoints serve as an implementation detail for talking to clusters using the Kafka-native protocol.Followup items
client.id
to something likeQuarkus for VS Code sidecar ${quarkus.application.version}