How to control s3 upload by time not by number of retry.

[ShimYama]

I'm working on live stream encoder and I want to control my uploads by time.

In my scenario, small video files are queued to upload in every 3 seconds but if upload speed gets very slow for some reason, several files are queued. If upload speed does not recover, I have to give up uploading some old files to catch up the stream. So I have to abort uploading files when upload is not completed in 30 seconds, e.g., and restart from a new file.

I use S3::S3Client::PutObject() method to upload video files now but it seems I can't handle the method by processing time but how many times it retries using RetryStrategy class, so it seems to be impossible to know how long it will take to process the maximum.
I also trying to use TransferManager and monitor upload state every seconds. When time is ran out, I want to cancel the upload but I don't know how to implement because we don't have any reference source to cancel uploads in aws-doc-sdk-examples branch...

What is the best strategy for this situation?

[jmklix]

You could try overloading RetryStrategy to make it fit your use case, but I'm not sure this is the best/easiest solution for you. How large/small are your video files? I would recommend that you try using s3-crt as that might be more optimized for your use case. You can see how to use s3-crt Putobject here.

[ShimYama]

Depending on the settings, up to about 10 video files, each a few MB in size, are uploaded simultaneously every a few seconds.
I've already tried RetryStrategy but it seems that I can only control an interval of each retry and I can't handle when to give up upload process. For example, it does not seem to work well with RetryStrategy if I want to cancel the upload immediately if the upload is not complete more than one second after I start the upload.
You also recommend using aws-sdk-s3-crt. I think it is a light version of standard aws-sdk-s3 methods and there are no difference what I can do. Am I wrong?

[jmklix]

aws-sdk-s3-crt is not a light version of aws-sdk-s3. It uses the highly optimized Common RunTime libraries (CRT) that are written in C code to achieve high throughput for large files. I would recommend you give it a try.

[ShimYama]

I understood the difference.
However, I have already tried aws-sdk-s3-crt but it is not what I want.
The ideal performance is not the matter because if network gets slow, aws-sdk can't perform excellently.
I have to care when network speed decreases during live stream.

When uploading a file with S3Client::PutObject, there appears to be no way to interrupt the upload from the client until the upload is complete, even if it takes longer when the network is not disconnected.
RetryStrategy can't handle the first transaction timeout.

[sbiscigl]

theres a client configuration called lowSpeedLimit if you are using a curl http client.

/**
 * Average transfer speed in bytes per second that the transfer should be 
 * below during the request timeout interval for it to be considered too slow and abort.
 * Default 1 byte/second. Only for CURL client currently.
 */
unsigned long lowSpeedLimit = 1;

curl documentation. this will abort a request and return a error if the overall transfer rate drops too low.

alternatively for windows there is requestTimeoutMs.

/**
 * Socket read timeouts for HTTP clients on Windows. Default 3000 ms. 
 * This should be more than adequate for most services. However, 
 * if you are transferring large amounts of data or are worried about higher latencies, 
 * you should set to something that makes more sense for your use case. For Curl, 
 * it's the low speed time, which contains the time in number milliseconds that transfer 
 * speed should be below "lowSpeedLimit" for the library to consider it too slow and abort.
 * Note that for Curl this config is converted to seconds by rounding down to the nearest 
 * whole second except when the value is greater than 0 and less than 1000. 
 * In this case it is set to one second. When it's 0, low speed limit check will be disabled.
 * Note that for Windows when this config is 0, the behavior is not specified by Windows.
 */
long requestTimeoutMs = 0;

A s3 client could be configured like

#include <aws/core/Aws.h>
#include <aws/core/utils/logging/LogLevel.h>
#include <aws/s3/S3Client.h>

using namespace Aws;
using namespace Aws::S3;
using namespace Aws::Utils::Logging;

int main() {
    SDKOptions options;
    options.loggingOptions.logLevel = LogLevel::Debug;
    InitAPI(options);
    {
        S3ClientConfiguration config;
        config.lowSpeedLimit = 1;
        config.requestTimeoutMs = 3000;
        S3Client client{config};
    }
    ShutdownAPI(options);
    return 0;
}

to set up these configurations