storage: Adding recursive feature to the storage command. #653

elkezza · 2024-11-26T12:11:00Z

Description

The user can now use a recursive flag to download files with tree directories, different use cases were considered as well:

The user can specify a directory with the -r flag and the directory with its sub-directory will be downloaded and its content.
if the sub-directory contains duplicated files it will skip the duplicated files and download the rest.
if the user specifies destination folder not ending "/", it will check if it's a file, otherwise it throws an error, if not it will create the file
if the user tries to download a file that already exists it will throw an error.

Checklist

Changelog updated (under Unreleased block)
Testing

Testing

Before the change

Show how the bucket on exoscale side looks like

PS C:\Users\Administrator\Desktop\Apps\Exoscale\CLI\exo-cli show anti-affinity-group with instances> exo storage list sos://test-99/      
  DIR  sub-1/
  DIR  sub-2/
PS C:\Users\Administrator\Desktop\Apps\Exoscale\CLI\exo-cli show anti-affinity-group with instances> exo storage list sos://test-99/sub-1/
2024-11-25 14:22:33 UTC    0 B  sub-1/
2024-11-25 15:54:59 UTC  431 B  sub-1/testDownload-2 copy.rtf
2024-11-25 15:57:04 UTC  431 B  sub-1/testDownload-2-copy.rtf
2024-11-25 15:32:41 UTC  431 B  sub-1/testDownload-2.rtf

Show how the local directory looks like

I have folder bucket-test-677 which contains only emptyfile.txt

PS C:\Users\Administrator\Desktop\Apps\Exoscale\CLI\exo-cli show anti-affinity-group with instances> dir "C:\Users\Administrator\Desktop\Apps\bucket-test-677"


    Directory: C:\Users\Administrator\Desktop\Apps\bucket-test-677


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a----        11/26/2024  11:59 AM              0 emptyfile.txt

testing to download directory to local directory `without` specifying `/` at the end

PS C:\Users\Administrator\Desktop\Apps\Exoscale\CLI\exo-cli show anti-affinity-group with instances> exo storage download sos://test-99/sub-1/ "C:\Users\Administrator\Desktop\Apps\bucket-test-677" -r                       
error: multiple files to download, destination must end with "/"

testing to download directory to local directory `with` specifying `/` at the end, it trigger an error `bucket-test-677 exists`

PS C:\Users\Administrator\Desktop\Apps\Exoscale\CLI\exo-cli show anti-affinity-group with instances> exo storage download sos://test-99/sub-1/ "C:\Users\Administrator\Desktop\Apps\bucket-test-677/" -r
error: file "C:\\Users\\Administrator\\Desktop\\Apps\\bucket-test-677" already exists, use flag `-f` to overwrite

testing to download file to local directory `with` `-r flag` it download it but without the root directories.


PS C:\Users\Administrator\Desktop\Apps\Exoscale\CLI\exo-cli show anti-affinity-group with instances> exo storage download sos://test-99/sub-1/testDownload-2.rtf "C:\Users\Administrator\Desktop\Apps\bucket-test-677/" -r
sub-1/testDownl… [==============================================================================] 431.00 b / 431.00 b | 0s

PS C:\Users\Administrator\Desktop\Apps\Exoscale\CLI\exo-cli show anti-affinity-group with instances> dir "C:\Users\Administrator\Desktop\Apps\bucket-test-677"                                                            


    Directory: C:\Users\Administrator\Desktop\Apps\bucket-test-677


Mode                 LastWriteTime         Length Name                                                                                                                                                                             
----                 -------------         ------ ----
-a----        11/26/2024  11:59 AM              0 emptyfile.txt
-a----        11/26/2024  12:03 PM            431 testDownload-2.rtf

After the change

testing to download directory to local directory `without` specifying `/` at the end, it download it with all content.

PS C:\Users\Administrator\Desktop\Apps\Exoscale\CLI\exo-cli show anti-affinity-group with instances> go run . storage download sos://test-99/sub-1/ "C:\Users\Administrator\Desktop\Apps\bucket-test-677" -r
sub-1/testDownl… [==============================================================================] 431.00 b / 431.00 b | 0s
sub-1/testDownl… [==============================================================================] 431.00 b / 431.00 b | 0s
sub-1/testDownl… [==============================================================================] 431.00 b / 431.00 b | 0s

checking the local directory after the download

PS C:\Users\Administrator\Desktop\Apps\Exoscale\CLI\exo-cli show anti-affinity-group with instances> dir "C:\Users\Administrator\Desktop\Apps\bucket-test-677"                                              


    Directory: C:\Users\Administrator\Desktop\Apps\bucket-test-677


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d-----        11/26/2024  12:06 PM                sub-1                                                                                                                                                                            
-a----        11/26/2024  11:59 AM              0 emptyfile.txt
-a----        11/26/2024  12:03 PM            431 testDownload-2.rtf                                                                                                                                                               


PS C:\Users\Administrator\Desktop\Apps\Exoscale\CLI\exo-cli show anti-affinity-group with instances> dir "C:\Users\Administrator\Desktop\Apps\bucket-test-677/sub-1"


    Directory: C:\Users\Administrator\Desktop\Apps\bucket-test-677\sub-1


Mode                 LastWriteTime         Length Name                                                                                                                                                                             
----                 -------------         ------ ----
-a----        11/26/2024  12:06 PM            431 testDownload-2 copy.rtf                                                                                                                                                          
-a----        11/26/2024  12:06 PM            431 testDownload-2-copy.rtf
-a----        11/26/2024  12:06 PM            431 testDownload-2.rtf

deleting the `testDownload-2.rtf` file from the local directory `sub-1`, to test what happens if try to download the sub-1 again where we are missing one file

PS C:\Users\Administrator\Desktop\Apps\Exoscale\CLI\exo-cli show anti-affinity-group with instances> del "C:\Users\Administrator\Desktop\Apps\bucket-test-677\sub-1\testDownload-2.rtf"
PS C:\Users\Administrator\Desktop\Apps\Exoscale\CLI\exo-cli show anti-affinity-group with instances> dir "C:\Users\Administrator\Desktop\Apps\bucket-test-677/sub-1"                   


    Directory: C:\Users\Administrator\Desktop\Apps\bucket-test-677\sub-1


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----                                                                                                                                                                             
-a----        11/26/2024  12:06 PM            431 testDownload-2 copy.rtf
-a----        11/26/2024  12:06 PM            431 testDownload-2-copy.rtf

downloading the `sub-1`, and the logs show that the duplicated files are ignored if the flag `-f` not used. and downloading the rest.


PS C:\Users\Administrator\Desktop\Apps\Exoscale\CLI\exo-cli show anti-affinity-group with instances> go run . storage download sos://test-99/sub-1/ "C:\Users\Administrator\Desktop\Apps\bucket-test-677" -r
error: file "C:\\Users\\Administrator\\Desktop\\Apps\\bucket-test-677\\sub-1\\testDownload-2 copy.rtf" already exists, use flag `-f` to overwrite
error: file "C:\\Users\\Administrator\\Desktop\\Apps\\bucket-test-677\\sub-1\\testDownload-2-copy.rtf" already exists, use flag `-f` to overwrite
sub-1/testDownl… [==============================================================================] 431.00 b / 431.00 b | 0s

… rest.

sauterp · 2024-11-26T13:49:30Z

I don't understand your "before"/"after" example because in the "before" example you are downloading only one file with the recursive flag... exo storage download sos://test-99/sub-1/testDownload-2.rtf "C:\Users\Administrator\Desktop\Apps\bucket-test-677/" -r

Also your directory names look like exo cli commands, which makes it hard to read.

sauterp

It looks OK, but can you explain your tests a bit more. I'm not sure I understand what they prove. thanks

sauterp · 2024-11-26T12:53:10Z

pkg/storage/sos/object.go

-			if strings.HasSuffix(config.Destination, "/") {
-				return path.Join(config.Destination, path.Base(aws.ToString(object.Key)))
+			if strings.HasSuffix(config.Destination, string(os.PathSeparator)) || dstInfo.IsDir() {
+				return filepath.Join(config.Destination, path.Base(aws.ToString(object.Key)))


Was the change to filepath needed to make it work on Windows?

Was the change to filepath needed to make it work on Windows?

yes, both string(os.PathSeparator and filepath for Windows os.

elkezza · 2024-11-26T14:52:44Z

I don't understand your "before"/"after" example because in the "before" example you are downloading only one file with the recursive flag... exo storage download sos://test-99/sub-1/testDownload-2.rtf "C:\Users\Administrator\Desktop\Apps\bucket-test-677/" -r

Also your directory names look like exo cli commands, which makes it hard to read.

thanks @sauterp for the feedback, now I updated the Description with more details explain what each command do, please let me know if you need more details.

elkezza added 2 commits November 25, 2024 23:43

Adding recursive feature to the storage command.

efae865

Skipping downloading the duplicate files and continue downloading the…

8742a7a

… rest.

elkezza requested a review from a team November 26, 2024 12:11

Update CHANGELOG.md

a5683f4

sauterp reviewed Nov 26, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

storage: Adding recursive feature to the storage command. #653

storage: Adding recursive feature to the storage command. #653

elkezza commented Nov 26, 2024 •

edited

Loading

sauterp commented Nov 26, 2024

sauterp left a comment

sauterp Nov 26, 2024

elkezza Nov 26, 2024

elkezza commented Nov 26, 2024

storage: Adding recursive feature to the storage command. #653

Are you sure you want to change the base?

storage: Adding recursive feature to the storage command. #653

Conversation

elkezza commented Nov 26, 2024 • edited Loading

Description

Checklist

Testing

Before the change

Show how the bucket on exoscale side looks like

Show how the local directory looks like

testing to download directory to local directory without specifying / at the end

testing to download directory to local directory with specifying / at the end, it trigger an error bucket-test-677 exists

testing to download file to local directory with -r flag it download it but without the root directories.

After the change

testing to download directory to local directory without specifying / at the end, it download it with all content.

checking the local directory after the download

deleting the testDownload-2.rtf file from the local directory sub-1, to test what happens if try to download the sub-1 again where we are missing one file

downloading the sub-1, and the logs show that the duplicated files are ignored if the flag -f not used. and downloading the rest.

sauterp commented Nov 26, 2024

sauterp left a comment

Choose a reason for hiding this comment

sauterp Nov 26, 2024

Choose a reason for hiding this comment

elkezza Nov 26, 2024

Choose a reason for hiding this comment

elkezza commented Nov 26, 2024

elkezza commented Nov 26, 2024 •

edited

Loading

testing to download directory to local directory `without` specifying `/` at the end

testing to download directory to local directory `with` specifying `/` at the end, it trigger an error `bucket-test-677 exists`

testing to download file to local directory `with` `-r flag` it download it but without the root directories.

testing to download directory to local directory `without` specifying `/` at the end, it download it with all content.

deleting the `testDownload-2.rtf` file from the local directory `sub-1`, to test what happens if try to download the sub-1 again where we are missing one file

downloading the `sub-1`, and the logs show that the duplicated files are ignored if the flag `-f` not used. and downloading the rest.