Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mm2 updates #26

Open
wants to merge 8 commits into
base: master
Choose a base branch
from
Open

Mm2 updates #26

wants to merge 8 commits into from

Conversation

iturcino
Copy link

@iturcino iturcino commented Jun 4, 2020

Fixes #25

@OneCricketeer
Copy link
Owner

Thank you!

Overall makes sense. Few comments.

Otherwise, I'm not sure I understand why it was ultimately needed... MM2 should work on its own since this SMT doesn't package anything but Avro and the Registry client, which the HTTP client hasn't changed that much, and it's shaded here too

pom.xml Outdated Show resolved Hide resolved
pom.xml Show resolved Hide resolved
pom.xml Outdated Show resolved Hide resolved
pom.xml Show resolved Hide resolved
pom.xml Show resolved Hide resolved
… ZK lib in provided scope. Added property for jersey.bean.validation version
@iturcino
Copy link
Author

iturcino commented Jun 7, 2020

Updated PR according to comments here

Copy link
Owner

@OneCricketeer OneCricketeer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please bump the pom version as well?

@iturcino
Copy link
Author

iturcino commented Jun 9, 2020

Version bumped to v.0.2.1

@kokachev
Copy link

kokachev commented Jan 20, 2021

Given that this PR has been approved, can @OneCricketeer please merge it and build new 0.2.1 version?

@jipipayo
Copy link

are you going to merge that?

@OneCricketeer
Copy link
Owner

Hi all, I no longer use Kafka for my role, nor this SMT, so am not actively looking at maintaining it.

That being said, Confluent has documentation about migrating schemas, and I am curious if anyone has tried those methods and has feedback on how they differ from this repo?

@msillence
Copy link

Hi all, I no longer use Kafka for my role, nor this SMT, so am not actively looking at maintaining it.

That being said, Confluent has documentation about migrating schemas, and I am curious if anyone has tried those methods and has feedback on how they differ from this repo?

The approach listed here involves setting the schema registry into a special mode where you can specify the id of the schema when you load it. This does work and you can use mirror maker without this SMT and then some other program to simply push the existing schema and id's in

Limitations

  • only works if the target schema registry does not contain any of the id's in your source schema registry
  • other schema registries don't support this import mode such as apicurio though you presumably could use their backup and restore process
  • still requires something to clone the schema registry - may even need a smt if your using mm with an evolving schema

I like your approach and think it's odd that it's not part of mirror maker

Limitations of this approach

  • the magic byte isn't always a guarentee that a binary topic is avro such as the internal ksql or __consumer_offsets

my quick fix is to add an ignore list: iturcino#2

I'm tempted to think it would be better for my use case if it simply ignored the not found or bad magic number

@vl-kp
Copy link

vl-kp commented Apr 8, 2024

anyone can review and merge it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Mirrormaker 2 updates
6 participants