Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ruleSet and KMS setting for CLI CSFLE during init #2942

Open
wants to merge 25 commits into
base: main
Choose a base branch
from

Conversation

channingdong
Copy link
Contributor

@channingdong channingdong commented Nov 15, 2024

Release Notes

This PR serves as a following up PR to finalize the CSFLE feature on CLI, including the following changes:

Breaking Changes

  • PLACEHOLDER

New Features

  • Add ruleSet and Key Management Service (KMS) driver support to finalize the Client Side Field Level Encryption (CSFLE) feature for confluent kafka topic [produce | consume]

Bug Fixes

  • PLACEHOLDER

Checklist

  • Leave this box unchecked if features are not yet available in production

What

  • Add the KMS driver and field-level encryption executor for each schema type during serializer/deserializer init.
  • Add all the built-in schemas required as reference/extension schemas for the CSFLE features.
  • Add encryption related test cases for all 3 types of schemas.
  • For protobuf, use Golang file embedding feature to add the built-in schemas during compile time, and copy then to a local temporary folder in running time during Kafka topic produce/consume (same folder as the main schema).
  • Update the AVRO schema serializer such that ruleSet can be extracted correctly from the native Go object.
  • Update the PROTOBUF schema deserializer process to parse the ruleSet correctly.
  • Update the confluent-kafka-go library to latest version.
  • Update the serializer/deserializer unit tests such that the temporary schema directory creation and deletion only happens once before/after all tests start/finish, instead of doing the temp directory creation/deletion inside each individual unit test.

References

https://docs.confluent.io/cloud/current/security/encrypt/csfle/overview.html

Test & Review

Thorough manual verification have been conducted to test different types (string, JSON, Prototype, Avro) of schemas with simple, reference, nested and ruleSet (with KMS) features, which all show the passing result.

The details verification result under different category can be found here:
https://docs.google.com/document/d/1GwXz9hNOkub_Br-2nssoYWCf6elZBvwo7TMhCNYinwE/edit?tab=t.0#heading=h.1xcbdvagwov4

@channingdong channingdong requested a review from a team as a code owner November 15, 2024 21:08
@confluent-cla-assistant
Copy link

🎉 All Contributor License Agreements have been signed. Ready to merge.
Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.

Copy link
Member

@rayokota rayokota left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @channingdong , left some comments

pkg/serdes/avro_deserialization_provider.go Outdated Show resolved Hide resolved
pkg/serdes/avro_serialization_provider.go Outdated Show resolved Hide resolved
pkg/serdes/json_deserialization_provider.go Outdated Show resolved Hide resolved
pkg/serdes/json_serialization_provider.go Outdated Show resolved Hide resolved
pkg/serdes/protobuf_deserialization_provider.go Outdated Show resolved Hide resolved
pkg/serdes/protobuf_serialization_provider.go Outdated Show resolved Hide resolved
@channingdong channingdong changed the title RuleSet and KMS setting for CLI CSFLE Add ruleSet and KMS setting for CLI CSFLE during init Nov 26, 2024
rayokota
rayokota previously approved these changes Nov 26, 2024
Copy link
Member

@rayokota rayokota left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @channingdong , LGTM

sgagniere
sgagniere previously approved these changes Dec 5, 2024
Copy link
Member

@sgagniere sgagniere left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After syncing w/ Channing: we need to make the proto files available even when users don't have any copy on their machine

sgagniere
sgagniere previously approved these changes Dec 12, 2024
Copy link
Member

@sgagniere sgagniere left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants