Replace AWS CLI by S3CMD to fix an issue with MultiPart on GCS

[sadortun]

Fix #110

[deitch]

Not 100% sure how I feel about replacing the official AWS S3 utilities with a 3rd-party tool. I have used s3cmd (quite a bit), but that is different than in something that is fairly widely used and will require support, not to mention concerns about vulnerabilities, where I worry far less about AWS-maintained tools than 3rd party.

Doesn't mean we cannot do it; just want to consider for a bit.

In any case, CI is failing. It looks like it isn't quite a drop-in replacement. We would need to fix that first?

[sadortun]

@deitch Can you have a look at the tests, i am really not sure where to look at.

The only obvious error i can see is :

+ AWS_ENDPOINT_OPT=' --host=http://s3:443/'
+ s3cmd --host=http://s3:443/ --access_key=abcdefg --secret_key=1234567 put /tmp/backups/db_backup_20190705032209.tgz s3://mybucket/6-0/data/db_backup_20190705032209.tgz
WARNING: Module python-magic is not available. Guessing MIME types based on file extensions.
ERROR: S3 error: 403 (InvalidAccessKeyId): The AWS Access Key Id you provided does not exist in our records.
+ '[' 77 -ne 0 ']'
+ return 1
+ '[' 1 -ne 0 ']'
+ exit_code=1

But i am not sure that the cause of the failed test

[deitch]

From the CI:

S3 LOGS:
+ docker logs b2b518bec4be20d2c9b894bf18bdfd6f9b089b13b2f8a863573f400b0593774b
[2019-07-05 03:21:38] INFO  WEBrick 1.3.1
[2019-07-05 03:21:38] INFO  ruby 2.1.5 (2014-11-13) [x86_64-linux-gnu]
[2019-07-05 03:21:38] INFO  WEBrick::HTTPServer#start: pid=1 port=443
[2019-07-05 03:22:09] ERROR bad Request-Line `\x16\x03\x01\x02\x00\x01\x00\x01�\x03\x03@R�=���t'�;H\x1A>�����_��,"��� ��L� �����5����=�CtKf�Mi&RȨ�b�"'�w�V\x00�\x13\x02\x13\x03\x13\x01�,�0�+�/̨̩\x00�\x00�\x00�\x00�̪�����$�(�'.
mysqlbackup-6.mysqltest - - [05/Jul/2019:03:22:09 UTC] "\x16\x03\x01\x02\x00\x01\x00\x01�\x03\x03@R�=���t'�;H\x1A>�����_��,"��� ��L� �����5����=�CtKf�Mi&RȨ�b�"'�w�V\x00�\x13\x02\x13\x03\x13\x01�,�0�+�/̨̩\x00�\x00�\x00�\x00�̪�����$�(�" 400 495
- -> 

s3cmd is doing something odd...

[deitch]

There is a number of issues here:

• The --host command does not take a URL, as in http://s3:443/ but just a hostname:port, as in s3:443.
• If you want to tell it not to use SSL, since you cannot pass the URL, you need to tell it explicitly --no-ssl.

Those two at least will get you a valid request. The rest actually isn't an invalid AWS key, but confusing error messages.

If we do this, you need to add parsing logic to AWS_ENPOINT_URL, as you now need to pass it to the s3cmd in the multiple arguments it wants - --host, --host-bucket, --no-ssl / --ssl - so that we can be consistent.