Clone Client Implementation

The current version of this page is at https://github.com/facebook/mysql-5.6/wiki/Clone-Client-Implementation

The MyRocks clone client is a singleton. Its main data structure is a map from file IDs to in-progress file objects consisting of open file descriptors, target sizes, and sizes applied so far. New IDs are registered and fds opened upon receiving FILE_NAME_MAP_V1 from the donor, and applied sizes are bumped for each received FILE_CHUNK_V1. Once the applied size reaches the target size, the file is closed and its metadata moved to the fully-applied file list. This list is write-only in normal operation, but in the case of clone restart its contents are used to create the restart locator.

Same as in the Clone Donor Implementation, the allocated task ID set is maintained for serializing the main thread clone finish after all the worker threads completing. It also has a secondary purpose of verifying passed task IDs to the API entry points.

The clone also keeps a copy of the locator for its session, for validation and for creating the restart locator if needed.

API Implementations

• rocksdb_clone_apply_begin(HA_CLONE_MODE_START): save the received locator for later validation and for the case a restart locator will need creating. Assign the task ID of 0. In the case of in-place clone, create the temporary directories for data .rocksdb.cloned and for WALs .rocksdb.wal.cloned. Inside the temporary directory for data, create the in-progress clone marker .clone_in_progress file.
• rocksdb_clone_apply_begin(HA_CLONE_MODE_ADD_TASK): verify the received locator, assign a new task ID.
• rocksdb_clone_apply_begin(HA_CLONE_MODE_RESTART): serialize the current applied and completed file states into a restart locator.
• rocksdb_clone_apply(FILE_NAME_MAP_V1): creates a new file with the given name, and stores its ID in the ID-to-fd map. The file is created in a temporary directory named either .rocksdb.cloned or .rocksdb.wal.cloned depending on the file type.
• rocksdb_clone_apply(FILE_CHUNK_V1): calls apply_file_cbk with the provided data, advances current file offset, if it reaches the target size – closes the file and moves it from from the in-progress ID-to-fd map to the completed file ID set.
• rocksdb_clone_apply(ADD_ESTIMATE_V1): passes through the MyRocks clone size to the clone plugin through the callback.
• rocksdb_clone_end: unregister the task ID. If the main thread, finish the clone session. Delete .clone_in_progress marker file.