Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

auto ignore table starts with _ #249

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

dec1985
Copy link

@dec1985 dec1985 commented Feb 7, 2018

Use pt-online-schema-change will auto generate a new table which name starts with '_', just ignore it will incr performance a lot!

@baloo
Copy link
Collaborator

baloo commented Feb 7, 2018

I don't think this is a good idea to do this. Some people may want those table.

I would prefer the consumer to just ignore those tables if he wants to.

@dec1985
Copy link
Author

dec1985 commented Feb 8, 2018

We have many big tables which have more than dozens millions rows, so we can only use pt-online-schema-change to alter table(this tool will make a new table which name starts with _ and ends with _new, then insert all rows into that new table and the rename new table to old table name). And we may alter table many times, such as alter a table everyday.

Use ignored_tables has to change the config files every time and restart that sync process, it's very painful.

Normal table's name always don't starts with "_".

Or maybe I add a config to control whether skip table's name with "_"?

@baloo
Copy link
Collaborator

baloo commented Feb 8, 2018

I know pt-online-schema-change and I also run it. But as of it's current state this is a breaking change for most users, I'm not merging this.

Maybe a better option would be a PR to have ignore_table optionaly include a user function?
Something like:

def ignore_pt_online_schema_change(tablename):
    return tablename.startswith('_')

ignore_tables = [
    'user_passwords',
    ignore_pt_online_schema_change
]
   
BinLogStreamReader(ignore_tables=ignore_tables)

Would that work for you?

@julien-duponchelle
Copy link
Owner

👍 for @baloo solution user can have use of table starting with _

But you are right with pt-online-schema-change you need to filter.

Alternative solution:

def ignore_pt_online_schema_change(tablename):
    return tablename.startswith('_')
   
BinLogStreamReader(ignore_tables_callback=ignore_pt_online_schema_change)

@dec1985
Copy link
Author

dec1985 commented Feb 9, 2018

noplay's solution would work for me, or I can add a controller which not filter table starts with "_" as its default value, same like noplay's solution in another way:

BinLogStreamReader(ignore_tables_startswith='_')

and ignore_tables_startswith default value is None.

@abrarsheikh
Copy link
Contributor

@baloo idea of callback functions sounds like a good idea. we are also seeing bunch of unnecessary events getting created by PT-OSC. it would be nice to get this feature in :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants