Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

zdbd: implement zdb-to-zdb queries #134

Open
maxux opened this issue Apr 20, 2022 · 1 comment
Open

zdbd: implement zdb-to-zdb queries #134

maxux opened this issue Apr 20, 2022 · 1 comment
Assignees
Milestone

Comments

@maxux
Copy link
Collaborator

maxux commented Apr 20, 2022

In order to solve different issue and old blocked behavior, implementing a server-to-server communication can solve many issues. First one would be the possibility to sync db without external tools.

@maxux maxux added this to the next milestone Apr 20, 2022
@maxux maxux self-assigned this Apr 20, 2022
@maxux
Copy link
Collaborator Author

maxux commented May 5, 2022

Workflow to sync a database in an efficient way (commands will needs to be executed as admin):

  1. First lock a namespace (in read-only) mode, with a special command (LOCKSYNC) to prepare to sync
    1.1. This command will reply current data id and current data offset
    1.2. Because the namespace is in read-only, no changes will occurs during the lock
  2. Note: we need to keep the namespace locked the shortest time possible
  3. Let's transfert the whole index, file by file
    3.1. Because we know data id from first command, we know how many index files to request
    3.2. Request index file, file-by-file INDEXFILE [id] [offset]
    3.3. Return value will be an array with: the payload, optional offset for chunk
    3.4. Because payload are limited to 8 MB, it's possible that payload needs to be chunked, let's repeat until we have the full file
  4. Now that we have the index fully sync locally, we can unlock the namespace (UNLOCKSYNC)
    4.1. Because data are immuable and we have the offset of the current last datafile, we can use local index to query the current database at this time snapshot, now we have two options:
    4.2. Or we use external trigger to request datafiles on runtime when needed
    4.3. Or we start downloading all datafiles "slowly" with a new command DATAFILE [id] [offset] which works the same way as the index
    4.4. Because the namespace is unlocked, we don't block anyone else and can take time to fetch the whole data

That way, we can sync a namespace without any external tools because we will talk zdb-to-zdb directly.

@maxux maxux modified the milestones: next, 2.2.0 Jul 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant