You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, each topic has exactly one latest version which can only have one schema.
We already run into limitations with this design as per #80, because we want to have some sort of type definition for some usecase unrelated to the needs of the consumer that the schema was designed for. As a result we end up adding restrictions to the schema that go far beyond the consumer's own needs.
This might become a problem because in a world where validation errors become more than warnings, it makes the consumer more brittle and reject/DLQ messages that it would've been able to process fine.
Proposal is to add a concept of consumer groups to sentry-kafka-schemas, and then each consumer group is individually versioned (instead of the topic). As a result we can have an arbitrary amount of "latest" schemas for a topic, and even run validation for all of them in the snuba consumer, but only one schema's validation errors cause DLQ/rejection of messages. The rest can be warnings.
For documentation purposes, a combined view of all schemas can be rendered, or all schemas can even be combined into one using allOf if that is somehow useful.
The text was updated successfully, but these errors were encountered:
Currently, each topic has exactly one latest version which can only have one schema.
We already run into limitations with this design as per #80, because we want to have some sort of type definition for some usecase unrelated to the needs of the consumer that the schema was designed for. As a result we end up adding restrictions to the schema that go far beyond the consumer's own needs.
This might become a problem because in a world where validation errors become more than warnings, it makes the consumer more brittle and reject/DLQ messages that it would've been able to process fine.
Proposal is to add a concept of consumer groups to sentry-kafka-schemas, and then each consumer group is individually versioned (instead of the topic). As a result we can have an arbitrary amount of "latest" schemas for a topic, and even run validation for all of them in the snuba consumer, but only one schema's validation errors cause DLQ/rejection of messages. The rest can be warnings.
For documentation purposes, a combined view of all schemas can be rendered, or all schemas can even be combined into one using
allOf
if that is somehow useful.The text was updated successfully, but these errors were encountered: