-
-
Notifications
You must be signed in to change notification settings - Fork 21
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
enhancements #15
base: master
Are you sure you want to change the base?
enhancements #15
Conversation
progress bar is in bytes now but I had to change the indentation so git thinks a lot of lines changes
Thank you @thiswillbeyourgithub for this PR. I have been quite busy the last months, so I don't have time to have a look now, but I will definitely look at it when I have time. |
…uthout might have more luck implementing this
Hi! There is really no rush, I am happy that you don't take it the wrong way as you said PR were not welcomed! But I felt that some features were easy to add and was necessary for me to use it anyway so it was too bad not to share it. Especially a way to exclude some files, have a progress bar and ability to make backups on local drive (especially for the first very large backup). I think the most important missing element is parallelization. The fastest speed I get using nfreezer is about 1MB/s which is way under what my laptop and internet connection are able to handle. But sadly I don't have the level yet to know how to manage parallel processes while coordinating the write to And after relooking at the original code, isn't there a way to get the full list of files needing to be uploaded using the main loop then creating a new parallelized loop that encrypts and uploads the file 4 by 4 or something ? It would then seem easier to make sure that flist is edited in a coordinated manner using lockfiles etc. But I have never done this so I don't feel comfortable trying this... |
Hello again, I had to use the threading module enough in the summer to feel comfortable enough to implement multithreaded file uploading in my last commits :) the logic is just that if the file is above a certain size, then instead of using pysftp to upload it, a new thread is created in which a new pysftp session is created to encrypt and send the file. This is definitely a gain in speed but launching a new pysftp session is a big overhead. A better idea would be to, when creating a backup: launch directly NUM_THREAD pysftp sessions, keeping them open and dispatching files to each session regardless of their size. But I realized that halfway and it would be a bit more complex so I'll leave it at that for now. edit: just to be clear, my change are mostly about the backup section, I have barely touched the restore section. I might do it in the future but just know that my changes have been tested only one way (I have not yet done restoration test) |
enhanced progress bar when reconstructing distant file list enhanced progress bar when restoring added inclusion and exclusion regex
* threaded functions are all rewritten, fixed, and moved at the top * restoring is now also multi threaded and seems to work fine * numerous other improvement I am really sorry to put this in such a messy commit but I figured that with the number of things that want to keep adding it doesn't make sense anymore to try to follow each chommit and the code change will have changed quite a lot from the original code
I hope you don't mind me creating a PR!
Note that I have no training in cryptography or security and intends solely to slightly improve your project