Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds mirrormaker example for GKE clusters #175

Closed
wants to merge 5 commits into from
Closed

Conversation

solsson
Copy link
Contributor

@solsson solsson commented May 1, 2018

Tested with the default #78 in GKE.

A couple of manifest values need customization:

  • WHITELIST
  • GROUP_ID
  • ip:s in source-endpoints.yml

Keeping resource limits low in order to reduce performance implications.

solsson added 5 commits May 1, 2018 12:43
to add a topic, but is quite dangerous for existing topics in case
you get an offset reset for some reason, like expiry.
the new-consumer-based mirror maker will change from 'range' to
'roundrobin' in an upcoming release
comma separated exact topic names
@solsson
Copy link
Contributor Author

solsson commented May 1, 2018

Does anyone know if there's a DNS name for a GKE cluster's nodes? That would solve bootstrap without dependence on ephemeral IP addresses.

@solsson
Copy link
Contributor Author

solsson commented May 1, 2018

Instead of having offset reset earliest which could lead to duplication, I think it makes sense to explicitly grant mirroring as triggered by the following error

[2018-05-01 17:16:47,351] ERROR [mirrormaker-thread-0] Mirror maker thread failure due to  (kafka.tools.MirrorMaker$MirrorMakerThread)
org.apache.kafka.clients.consumer.NoOffsetForPartitionException: Undefined offset with no reset policy for partitions: [ite-leanconstruction.yolean-live3.operations.001-0]
	at org.apache.kafka.clients.consumer.internals.SubscriptionState.resetMissingPositions(SubscriptionState.java:374)
	at org.apache.kafka.clients.consumer.KafkaConsumer.updateFetchPositions(KafkaConsumer.java:1794)
	at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1152)
	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1115)
	at kafka.tools.MirrorMaker$MirrorMakerNewConsumer.receive(MirrorMaker.scala:641)
	at kafka.tools.MirrorMaker$MirrorMakerThread.run(MirrorMaker.scala:428)
[2018-05-01 17:16:48,049] ERROR [mirrormaker-thread-0] Mirror maker thread exited abnormally, stopping the whole mirror maker. (kafka.tools.MirrorMaker$MirrorMakerThread)

The problem is that a new topic matching the whitelist would halt all other mirroring. I guess a group offset 0 could be set using kafkacat etc, but I have no strategy for not interrupting the mirroring of existing topics.

@solsson
Copy link
Contributor Author

solsson commented Apr 21, 2021

Use MM2

@solsson solsson closed this Apr 21, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant