Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CLI-3355] Fix some confluent kafka topic consume errors #2970

Open
wants to merge 7 commits into
base: main
Choose a base branch
from

Conversation

sgagniere
Copy link
Member

@sgagniere sgagniere commented Dec 11, 2024

Release Notes

Breaking Changes

  • PLACEHOLDER

New Features

  • PLACEHOLDER

Bug Fixes

  • Fix a bug preventing users from consuming audit logs with confluent kafka topic consume
  • Fix bugs preventing users from producing or consuming from a Cloud cluster while logged out using confluent kafka topic [produce | consume] --bootstrap

Checklist

  • Leave this box unchecked if features are not yet available in production

What

  • The assumption that a SR cluster always exists is false in the audit log environment, so this PR changes the call to get SR ID and endpoint to only happen if the key or value type is avro, protobuf, or jsonschema (for consume)
  • For produce, only make a call to get the endpoint and ID if the user has not already provided the endpoint while logged out, to avoid a panic
  • The GetDataplaneToken call will fail for users consuming from a cloud cluster using --bootstrap while logged out, so this PR adds a check that the user is logged in before attempting to get the token

References

Test & Review

Tested with the following.

  • while logged in: confluent kafka topic consume -b confluent-audit-log-events
  • while logged out:
    • confluent kafka topic [produce | consume] -b confluent-audit-log-events --bootstrap <endpoint> --api-key <api_key> --api-secret <api_secret>
    • confluent kafka topic produce <topic> --bootstrap <cluster_endpoint> --api-key <api_key> --api-secret <api_secret> --value-format protobuf --schema <schema_id> --schema-registry-endpoint <sr_endpoint> --schema-registry-api-key <sr_key> --schema-registry-api-secret <sr_secret>

@sgagniere sgagniere requested a review from a team as a code owner December 11, 2024 00:55
@confluent-cla-assistant
Copy link

🎉 All Contributor License Agreements have been signed. Ready to merge.
Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.

@sgagniere sgagniere marked this pull request as draft December 11, 2024 02:11
@sgagniere sgagniere marked this pull request as ready for review December 12, 2024 01:11
internal/kafka/command_topic_consume.go Show resolved Hide resolved
Comment on lines +252 to +258
// Fetch the current SR cluster id and endpoint
if srEndpoint == "" {
srClusterId, srEndpoint, err = c.GetCurrentSchemaRegistryClusterIdAndEndpoint()
if err != nil {
return err
}
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if srEndpoint is not empty, so the srClusterId will not set, during normal consume without srClusterId the deserializer couldn't be initialized correctly.

Not sure if that's the case and being tested?

Comment on lines +643 to +650
var srClusterId string
if (schemaId.IsSet() || schema != "") && srEndpoint == "" {
srClusterId, srEndpoint, err = c.GetCurrentSchemaRegistryClusterIdAndEndpoint()
if err != nil {
return nil, nil, err
}
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here, will the basic produce have issues with the empty srClusterId if srEndpoint is passed from customer flag?

Copy link
Contributor

@channingdong channingdong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we also handle the auth.GetDataplaneToken(c.Context) in consumeOnPrem() and produceOnPrem()?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants