S3 PutObject default "Cache-Control" value is set to "no-cache"

[ShimYama]

Describe the bug

When I upload files to Amazon S3 with sdk, their "Cache-Control" in metadata section is always set to "no-cache" even if I didn't set it.
I also uploaded files with Amazon CLI and their metadata don't have any "Cache-Control" keys.

I also tested sample program in aws-doc-sdk-examples repository and the same problem occurs.

Expected Behavior

Don't set anything if cache control is not set.

Current Behavior

• S3::Model::PutObjectRequest::SetCacheControl() is not used
  • "Cache-Control" is set to "no-cache"
• S3::Model::PutObjectRequest::SetCacheControl() is used (for example, SetCacheControl("max-age=1000");)
  • "Cache-Control" is overwritten by the value you set.

Reproduction Steps

Just upload files using S3::S3Client::PutObject() method without using S3::Model::PutObjectRequest::SetCacheControl().
You can reproduce it with sample source.

Possible Solution

No response

Additional Information/Context

No response

AWS CPP SDK version used

1.11.127

Compiler and Version used

Visual Studio 2015 Community / Build Engine version 14.0.25420.1

Operating System and version

Windows 10 Pro 22H2

[yasminetalby]

Hello @ShimYama ,

Thank you very much for your submission.

I am not sure I have understood your submission properly.

If you mean that there is no Control Cache header when not using S3::Model::PutObjectRequest::SetCacheControl(), then this is expected behavior. As you can see from the S3 documentation for PutObject, the Cache Control header is not required (see.

1. Behavior for the AWS CPP SDK

There is no cache-control header by default for PutObjectRequest : m_cacheControlHasBeenSet(false). Using S3::Model::PutObjectRequest::SetCacheControl() you can set the cache-control value (as observed).

If you take a look at the PutObjectRequest implementation:

 if(m_cacheControlHasBeenSet)
  {
    ss << m_cacheControl;
    headers.emplace("cache-control",  ss.str());
    ss.str("");
  }

2. Behavior for the AWS CLI

I am not sure with version of AWS CLI you are using however the behavior is as follow:

When you use the AWS CLI version 1 version of commands in the aws s3 namespace to copy a file from one Amazon S3 bucket location to another Amazon S3 bucket location, and that operation uses multipart copy, no file properties from the source object are copied to the destination object.

By default, the AWS CLI version 2 commands in the s3 namespace that perform multipart copies transfers all tags and the following set of properties from the source to the destination copy: content-type, content-language, content-encoding, content-disposition, cache-control, expires, and metadata.

This can result in additional AWS API calls to the Amazon S3 endpoint that would not have been made if you used AWS CLI version 1. These can include: HeadObject, GetObjectTagging, and PutObjectTagging.

If you need to change this default behavior in AWS CLI version 2 commands, use the --copy-props parameter to specify one of the following options:

default – The default value. Specifies that the copy includes all tags attached to the source object and the properties encompassed by the --metadata-directive parameter used for non-multipart copies: content-type, content-language, content-encoding, content-disposition, cache-control, expires, and metadata.

See AWS CLI V2 documentation.

3. Behavior for the AWS Console

To add a Cache-Control or Expires header field to Amazon S3 objects using the Amazon S3 console:

1. Sign in to the AWS Management Console and open the Amazon S3 console at : https://console.aws.amazon.com/s3/ .
2. In the list of buckets, choose the name of the bucket that contains the files that you are adding headers to.
3. Select the check box next to the name of the file or folder that you are adding headers to. When you add headers to a folder, it affects all the files inside that folder.
4. Choose Actions, then choose Edit metadata.
5. In the Add metadata panel, do the following:

• Choose Add metadata.
• For Type, choose System defined.
• For Key, choose the name of the header that you are adding (Cache-Control or Expires).
• For Value, enter a header value. For example, for a Cache-Control header, you could enter max-age=86400. For Expires, you could enter an expiration date and time such as Wed, 30 Jun 2021 09:28:00 GMT.

6. At the bottom of the page, choose Edit metadata.

See documentation.

For reference, you can read more about the cache control header here : https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html

Now if you mean that there is a control-cache header present with value set as "no-cache" when not using S3::Model::PutObjectRequest::SetCacheControl(), this is unexcepected behavior. However, I have attempted to reproduce this behavior and wasn't able to (which makes sense based on the PutObjectRequest implementation shared above). When I make a PutObjectRequest, without using the SetCacheControl(), no header is present.

If this is your experience, I would recommend to check through the console if you have set any metadata set for Cache-Control in the affected folder. (You can refer to the Behavior for the AWS Console above).

Let me know if you have any questions or need further informations.

Best regards,

Yasmine

[ShimYama]

Now if you mean that there is a control-cache header present with value set as "no-cache" when not using S3::Model::PutObjectRequest::SetCacheControl(), this is unexcepected behavior.

This is what I have experienced.

If this is your experience, I would recommend to check through the console if you have set any metadata set for Cache-Control in the affected folder. (You can refer to the Behavior for the AWS Console above).

I created a new bucket with default settings and uploaded files from sdk but this issue happened, so I don't think I've added a cache-control behavior settings to my bucket or folder...

I also check PutObjectRequest value to confirm that it doesn't have any cache control value.
PutObjectRequest::GetCacheControl() returns empty string and PutObjectRequest::CacheControlHasBeenSet() returns false so it seems ok...

My AWS CLI version
aws-cli/2.13.0 Python/3.11.4 Windows/10 exe/AMD64 prompt/off

[yasminetalby]

Hello @ShimYama ,

This is a very odd behavior. Especially if you have that PutObjectRequest::GetCacheControl() returns empty string and PutObjectRequest::CacheControlHasBeenSet() returns false. The only way I am able to have the cache-control:no-cache header is by specifically setting it :

Aws::S3::Model::PutObjectRequest my_request;
my_request.SetCacheControl("no-cache");

Have you checked through the console if you have set any metadata set for Cache-Control as mentioned above?
If nothing is set, could you please share your code sample (for bucket creation and putobject) as well as the log associated with the behavior (don't forget to remove any sensitive data from the log)?

Thank you very much for your time and collaboration.

Sincerely,

Yasmine

[ShimYama]

I created bucket via AWS console (i.e. on web browser) with default settings; I just set a name and a region of a bucket.

I can reproduce this issue with the sdk sample code below.
https://github.com/awsdocs/aws-doc-sdk-examples/blob/main/cpp/example_code/s3/put_object.cpp

I added some lines for logging but the core process is not modded.
This is the difference from the original sample code.

#ifndef TESTING_BUILD
+#include "aws/core/utils/logging/LogLevel.h"

int main() {
+   _putenv_s("AWS_EC2_METADATA_DISABLED", "true");
+
    Aws::SDKOptions options;
+   options.loggingOptions.logLevel = Aws::Utils::Logging::LogLevel::Trace;
    Aws::InitAPI(options);
    {
        //TODO(user): Change bucket_name to the name of a bucket in your account.
-       const Aws::String bucket_name = "<Enter bucket name>";
+       const Aws::String bucket_name = "<--MyBucketName-->";
        //TODO(user): Create a file called "my-file.txt" in the local folder where your executables are built to.
-       const Aws::String object_name = "<Enter file>";
+       const Aws::String object_name = "sample.txt";

        Aws::Client::ClientConfiguration clientConfig;
        // Optional: Set to the AWS Region in which the bucket was created (overrides config file).
-       // clientConfig.region = "us-east-1";
+       clientConfig.region = "ap-northeast-1";

        AwsDoc::S3::PutObject(bucket_name, object_name, clientConfig);
    }

    Aws::ShutdownAPI(options);

    return 0;
}

#endif  // TESTING_BUILD

And this is a log file.
aws_sdk_2023-08-02-22_masked.log

I hope this helps you to find what is wrong.

[yasminetalby]

Hello @ShimYama ,

Apologies but the log you have provided the header contains no cache-control header. It seems that the SDK is behaving as expected here and is not setting any "Cache-Control" in the metadata section.
Are you seeing this in the metadata section of the object in the console? In anyway, based on the log provided, this doesn't seem to be a CPP SDK issue.
I have also tested the sample provided and can't reproduce the issue (no Cache-Control is set in the object metadata if not using PutObjectRequest::GetCacheControl()).

The behavior with the sample is as follow:

1. With

Aws::S3::Model::PutObjectRequest my_request;
my_request.SetCacheControl("no-cache");

The request contains the following header : cache-control:no-cache. You can see the following metadata associated with the object in the AWS Console in the object property tab under the metadata:

Type : System defined | Key : Cache-Control | Value : no-cache

PutObjectRequest::GetCacheControl() returns no-cacha and PutObjectRequest::CacheControlHasBeenSet() returns true

2. With

Aws::S3::Model::PutObjectRequest my_request;
//my_request.SetCacheControl("no-cache");

(That is not: using SetCacheControl)

The request does not contain the cache-control header at all. There is no metadata of Type : System defined | Key : Cache-Control in the AWS Console in the object property tab under the metadata.

PutObjectRequest::GetCacheControl() returns empty string and PutObjectRequest::CacheControlHasBeenSet() returns false

I am not sure why you are seeing metadata associated with your object under the properties metadata tab of the object if you are creating a bucket. Are you doing any other modification/steps besides the one listed above?

I apologies for the redundancy but the question asked in my previous two comments is still unanswered:
If you check under the object properties tab under the metadata section, are you seeing Type : System defined | Key : Cache-Control | Value : no-cache else where are you seeing this as it is not present in the request header of the log you have provided above?

Best regards,

Yasmine

[ShimYama]

Thank you for your answer.

I didn't add any SetCacheControl settings, so PutObjectRequest::GetCacheControl() returns empty string and PutObjectRequest::CacheControlHasBeenSet() returns false.
However, if I go to the object properties tab in aws console, I found Type : System defined | Key : Cache-Control | Value : no-cache in the metadata section.

I also tested this source so that I didn't add any special setting to my bucket or something.

#include <aws/core/Aws.h>
#include <aws/core/utils/logging/LogLevel.h>
#include <aws/s3/S3Client.h>
#include <aws/s3/model/BucketLocationConstraint.h>
#include <aws/s3/model/CreateBucketRequest.h>
#include <aws/s3/model/GetObjectRequest.h>
#include <aws/s3/model/PutObjectRequest.h>
#include <iostream>
#include <fstream>

using namespace Aws;

static String bucketName = "<new bucket name>";
static String fileName   = "sample.txt";

int main()
{
	_putenv_s("AWS_EC2_METADATA_DISABLED", "true");

	SDKOptions options;
	options.loggingOptions.logLevel = Utils::Logging::LogLevel::Trace;

	InitAPI(options);
	{
		S3::S3ClientConfiguration clientConfig;
		clientConfig.scheme = Http::Scheme::HTTPS;
		clientConfig.region = Region::AP_NORTHEAST_1;

		S3::S3Client client(clientConfig);

		{
			S3::Model::CreateBucketRequest request;
			request.SetBucket(bucketName);

			S3::Model::CreateBucketConfiguration createBucketConfig;
			createBucketConfig.SetLocationConstraint(S3::Model::BucketLocationConstraintMapper::GetBucketLocationConstraintForName(clientConfig.region));
			request.SetCreateBucketConfiguration(createBucketConfig);

			auto outcome = client.CreateBucket(request);
			if (outcome.IsSuccess()) {
				std::cout << "Created bucket " << bucketName << " in the specified AWS Region." << std::endl;
			}
		}
		{
			S3::Model::PutObjectRequest request;
			request.SetBucket(bucketName);
			request.SetKey(fileName);

			std::shared_ptr<IOStream> inputData = std::make_shared<FStream>(fileName.c_str(), std::ios_base::in | std::ios_base::binary);
			request.SetBody(inputData);

			std::cout << "Cache Control has been set: " << (request.CacheControlHasBeenSet() ? "Yes" : "No") << std::endl;
			std::cout << "Cache Control value: " << request.GetCacheControl() << std::endl;
			auto outcome = client.PutObject(request);
			if (outcome.IsSuccess()) {
				std::cout << "Added object '" << fileName << "' to bucket '" << bucketName << "'." << std::endl;
			}
		}
		{
			S3::Model::GetObjectRequest request;
			request.SetBucket(bucketName);
			request.SetKey(fileName);

			auto outcome = client.GetObject(request);
			if (outcome.IsSuccess()) {
				std::cout << "Cache Control: " << outcome.GetResult().GetCacheControl() << std::endl;
			}
		}
	}

	ShutdownAPI(options);
	return 0;
}

And I got this result and a log.
aws_sdk_2023-08-03-22.log

E:\Develop\aws-test\build\Release>app.exe
Created bucket <new bucket name> in the specified AWS Region.
Cache Control has been set: No
Cache Control value:
Added object 'sample.txt' to bucket '<new bucket name>'.
Cache Control: no-cache

It seems my file was uploaded without cache control settings but for some reason s3 adds cache-control settings automatically.

[yasminetalby]

Hello @ShimYama ,

I have attempted this exact code sample and the file is still uploaded without cache control settings. I will reach out to s3 to see if they have any guidance to provide on why this behavior is happening, however it seems like this is not an SDK issue.
I will update this issue once I receive a response from the S3 team.

Thank you very much for providing all the requested information.

Best regards,

Yasmine

[gbrownewell]

Hi Yasmine,

Did the s3 team have any insight? This bug is causing us a great deal of trouble as well.

Regards,

[yasminetalby]

Hello @gbrownewell ,

I unfortunately haven't gotten a response from the S3 team yet. I will raise this again and provide an update here as soon as I hear back from them.

Best regards,

Yasmine

[jmklix]

Fixed with this: #2998

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.