Implemented fetch and pull modes for the main application #45
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
MySQL connection configuration structure ('easymysql::connection_config')
extended with three additional parameters:
* 'connect_timeout' ('MYSQL_OPT_CONNECT_TIMEOUT')
* 'read_timeout' ('MYSQL_OPT_READ_TIMEOUT')
* 'write_timeout' ('MYSQL_OPT_WRITE_TIMEOUT')
that are used to set corresponding MySQL client options via 'mysql_options()'.
'main_config.json' sample configuration file and config file generation in
'binlog_streaming.binsrv' MTR test case updated correspondingly.
Main application extended with one more command line argument (the very first
one) called ''<operation_mode>' with two allowed values ('fetch' and 'pull').
In 'fetch' mode the application now connects to the remote MySQL server and
reads all events from all binary logs till the very last one
(till 'mysql_binlog_fetch()' returns an empty buffer) and exits. The replication
channel is opened in non-blocking mode.
In 'pull' mode the application also reads all available events till
the very last one but instead of exiting immediately continues to wait
for other events to come later. The replication channel is opened in blocking
mode. In order to be able to gracefully terminate this process, we use
'read_timeout' configuration parameter ('MYSQL_OPT_READ_TIMEOUT' MySQL client
option) after which blocking 'mysql_binlog_fetch()' call will terminate with
a timeout error. After this, the replication connection is closed and program
enters the 'idle' state doing nothing. Currently the idle time is hardcoded to
5 seconds. When this 'idle' time passes, the program attempts to re-establish
MySQL server connection and continue replication. Not being able to do so, will
not terminate the program. Instead, after another 'idle' period the next attempt
to re-establish connection will be made. This model will allow us to react to
termination requests (most probably to posix 'SIGINT' / 'SIGTERM' signals) not
only in the binlog event processing loop but during the 'idle' time as well.
Introduced 'binsrv::operation_mode_type' smart enum with two values ('fetch'
and 'pull') that is used to specify main application operation mode.
'easymysql::connection' now includes all the functionality available before in
the 'easymysql::binlog' class. Established connection can now be switched to
'replication' mode. 'easymysql::binlog' class removed completely.
The method that switches connection to replication mode
('easymysql::connection::switch_to_replication()') accepts an extra parameter
that specifies blocking / non-blocking replication mode.
Added explicit 'flush()' calls at the end of 'do_write_data_to_stream()' for
both 'binsrv::filesystem_storage_backend' and 'binsrv::s3_storage_backend' to
make sure that buffered data is immediately written to the disk.
'mysql-server' project GitHub links in the comments updated to version '8.0.37'.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
MySQL connection configuration structure ('easymysql::connection_config') extended with three additional parameters:
Main application extended with one more command line argument (the very first one) called ''<operation_mode>' with two allowed values ('fetch' and 'pull').
In 'fetch' mode the application now connects to the remote MySQL server and reads all events from all binary logs till the very last one (till 'mysql_binlog_fetch()' returns an empty buffer) and exits. The replication channel is opened in non-blocking mode.
In 'pull' mode the application also reads all available events till the very last one but instead of exiting immediately continues to wait for other events to come later. The replication channel is opened in blocking mode. In order to be able to gracefully terminate this process, we use 'read_timeout' configuration parameter ('MYSQL_OPT_READ_TIMEOUT' MySQL client option) after which blocking 'mysql_binlog_fetch()' call will terminate with a timeout error. After this, the replication connection is closed and program enters the 'idle' state doing nothing. Currently the idle time is hardcoded to 5 seconds. When this 'idle' time passes, the program attempts to re-establish MySQL server connection and continue replication. Not being able to do so, will not terminate the program. Instead, after another 'idle' period the next attempt to re-establish connection will be made. This model will allow us to react to termination requests (most probably to posix 'SIGINT' / 'SIGTERM' signals) not only in the binlog event processing loop but during the 'idle' time as well.
Introduced 'binsrv::operation_mode_type' smart enum with two values ('fetch' and 'pull') that is used to specify main application operation mode.
'easymysql::connection' now includes all the functionality available before in the 'easymysql::binlog' class. Established connection can now be switched to 'replication' mode. 'easymysql::binlog' class removed completely.
The method that switches connection to replication mode ('easymysql::connection::switch_to_replication()') accepts an extra parameter that specifies blocking / non-blocking replication mode.
Added explicit 'flush()' calls at the end of 'do_write_data_to_stream()' for both 'binsrv::filesystem_storage_backend' and 'binsrv::s3_storage_backend' to make sure that buffered data is immediately written to the disk.
'mysql-server' project GitHub links in the comments updated to version '8.0.37'.