Add support for S3-compatible storage in Result Uploader

[zjmletang]

To upload files using Alibaba Cloud OSS, you must first have an Alibaba Cloud OSS account, which includes your Access Key (AK) and Secret Key (SK). Once you have these, please follow the steps below for configuration:

1. Create an OSS Bucket: Start by setting up a new OSS bucket in your Alibaba Cloud account.
2. Configure the alicloudoss.json File: Update this file with your bucket name and endpoint.
3. Set Environment Variables: Configure your AK and SK as environment variables by executing:
   export AUTOHCK_ALICLOUDOSS_ACCESS_KEY_ID=<AK>
   export AUTOHCK_ALICLOUDOSS_CLIENT_SECRET=<SK>
   After completing these steps, you'll be ready to upload files to Alibaba Cloud OSS.

for the problem description
please visit #471

Signed-off-by: Zhang JianMing [email protected]

[kostyanf14]

Please review and fix Rubocop code scanning error too.

[akihikodaki]

First, thank you for your contribution.

Please consider using AWS SDK instead; it should give chances to support many S3-compatible storages, including Alibaba Cloud OSS. If you definitely need a feature specific to Alibaba Cloud OSS, please explain it.

[zjmletang]

First, thank you for your contribution.

Please consider using AWS SDK instead; it should give chances to support many S3-compatible storages, including Alibaba Cloud OSS. If you definitely need a feature specific to Alibaba Cloud OSS, please explain it.

@akihikodaki , Thank you for your valuable suggestion! I chose the Alibaba Cloud SDK primarily because I work with Alibaba Cloud, which makes maintenance and troubleshooting more convenient for me. I hadn't fully considered the potential benefits of using the AWS SDK, as I usually select the SDK that corresponds to the cloud service I'm utilizing.

I do have a question regarding the definition of "result uploader." Currently, we are using Dropbox, and my initial understanding was that each storage service would have its own dedicated "result uploader," such as Dropbox, Alibaba Cloud OSS, AWS Storage, etc. Is this understanding correct? In other words, even though they are all S3-compatible storages, will there still be individual result uploaders for different products from various companies? I would appreciate your insight on whether you envision S3-compatible storages as being treated as a single result uploader.

[kostyanf14]

@zjmletang

I would appreciate your insight on whether you envision S3-compatible storages as being treated as a single result uploader.

We agree with the idea of having one S3-compatible result uploader with the ability to configure it for AWS, Alibaba Cloud, or even open-source storage (e.g., minio).

[zjmletang]

We agree with the idea of having one S3-compatible result uploader with the ability to configure it for AWS, Alibaba Cloud, or even open-source storage (e.g., minio).

@kostyanf14 Got it. I have turned the patch into a draft as I need some time to modify and test it

[zjmletang]

@kostyanf14 Could you please reopen my patch? I accidentally closed it while resetting the branch, and I didn’t see the option to reopen it afterward. I’m not sure if this is due to permission issues.

[YanVugenfirer]

@zjmletang I reopened

[kostyanf14]

@zjmletang Can I make PR ready for review?

[zjmletang]

@kostyanf14. Not ready yet; I haven't tested it thoroughly myself. I'll notify you once I finish the testing

[zjmletang]

@kostyanf14 , I am ready on my side. Please see the configuration document below

To upload files using S3-Compatible storage (supports AWS S3, Alibaba Cloud OSS, etc.), please follow the steps below for configuration:

1. Create an Account
   For AWS S3: Sign up for an AWS account and obtain your Access Key (AK) and Secret Key (SK).
   For Alibaba Cloud OSS: Similarly, create an Alibaba Cloud account and get your Access Key (AK) and Secret Key (SK).
2. Create a Storage Bucket
   Log in to your cloud provider's console (AWS or Alibaba Cloud) and create a new storage bucket.
3. Set Bucket Permissions
   Ensure that your bucket is configured to have public read permissions.
4. Configure s3compatible.json File
   Update the s3compatible.json file in your project directory with the following content:
   {
   "bucket_name": "<your_bucket_name>",
   "region": "<your_region>",
   "endpoint": "<your_endpoint>"
   }
   Example Configurations:
   For AWS S3:
   {
   "bucket_name": "auto-hck-result",
   "region": "us-east-1",
   "endpoint": "s3.amazonaws.com"
   }
   For Alibaba Cloud OSS:
   {
   "bucket_name": "auto-hck-result",
   "region": "oss-cn-hangzhou",
   "endpoint": "oss-cn-hangzhou.aliyuncs.com"
   }
   For specific values of region and endpoint, please refer to the official documentation of your cloud provider.
5. Set Environment Variables
   Configure your AK and SK as environment variables by executing the following commands:
   export AUTOHCK_S3_ACCESS_KEY_ID=<AK>
   export AUTOHCK_S3_SECRET_ACCESS_KEY=<SK>

[akihikodaki]

I prefer it to be named S3 instead of S3compatible for conciseness. Anything compatible with S3 should work with an uploader for S3 anyway so "compatible" adds no semantic value.

[akihikodaki]

There are few nits remaining and this pull request will be ready once they are handled. Thank you for keeping working on this feature!

[akihikodaki]

Now all my feedback is addressed. Thank you for your contribution!

[zjmletang]

There are few nits remaining and this pull request will be ready once they are handled. Thank you for keeping working on this feature!

I have made the changes. Thank you very much for your thorough guidance; I have learned a lot @akihikodaki @kostyanf14 @YanVugenfirer