Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement #728 Flush doc managers before oplog progress write #729

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

travisp
Copy link

@travisp travisp commented Aug 16, 2017

This is my proposal for issue #728. Basically it gives us the ability to use an async or buffering doc manager (such as kafka) and ensure that writes are sent prior to writing oplog progress.

In our Kafka Doc Manager, the implementation essentially looks like:

def flush(self):
  self.producer.flush()

It is a different purpose than commit(), because with an async or buffering doc manager data can be lost without the flush()

@makhdumi
Copy link
Contributor

We also implemented something like this in our own branch. It's critical that the oplog progress is only updated after all docmanagers have committed. I'm not sure why it already isn't. It caused us significant data loss.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants