X-Amz-Content-SHA256 should be present when querying Amazon OpenSearch Serverless

[JCKortlang]

Describe the bug

Only reproducible when querying an Amazon OpenSearchServerless collection with a private network policy (accessibly only via VPCE). I am unable to reproduce on a collection with a public network policy.

Method: GET, RequestUri: 'https://host-id.us-west-2.aoss.amazonaws.com/_cat/indices', Version: 1.1, Content: System.Net.Http.StringContent, Headers:

{

  Accept: */*
  accountid: 855676708012
  User-Agent: curl/8.4.0
  x-forwarded-for: 15.248.7.84
  x-forwarded-port: 443
  x-forwarded-proto: https
  X-Amz-Date: 20240422T182608Z
  x-amz-security-token:  <redacted>
  Host:host-id.us-west-2.aoss.amazonaws.com
  Authorization: AWS4-HMAC-SHA256 Credential=ASIA4OOSOMSWA3GBWWEE/20240422/us-west-2/aoss/aws4_request, SignedHeaders=accept;accountid;host;user-agent;x-amz-date;x-amz-security-token;x-forwarded-for;x-forwarded-port;x-forwarded-proto, Signature=ec37d19eaf6227efe60c26a8690242c2126ef5449ed29a3755105e939d93004b
  Content-Length: 0
  Content-Type: application/json; charset=utf-8

} with  made by arn:aws:sts::855676708012:assumed-role/InsightsStack-EtlStorageOpenSearchServerlessApiProx-UMqk35Yx0VO4/InsightsStack-EtlStorageOpenSearchServerlessApiPro-p6lr3QP2XtH5 failed with StatusCode: 403, ReasonPhrase: 'Forbidden', Version: 1.1, Content: System.Net.Http.HttpConnectionResponseContent, Headers:

{

  X-Request-ID: d8d04efc-0f9e-97d6-bf07-bc655fd42594
  Date: Mon, 22 Apr 2024 18:26:08 GMT
  x-aoss-response-hint: X01:gw-helper-deny
  Server: aoss-amazon
  Content-Type: application/json
  Content-Length: 121
} Forbidden is not successful with '{"status":403,"request-id":"d8d04efc-0f9e-97d6-bf07-bc655fd42594","error":{"reason":"403 Forbidden","type":"Forbidden"}}

--

Based on the documentation,

The following requirements apply when signing requests to OpenSearch Serverless collections when you construct HTTP requests with another clients.
You must specify the service name as aoss.
The x-amz-content-sha256 header is required for all AWS Signature Version 4 requests. It provides a hash of the request payload. If there's a request payload, set the value to its Secure Hash Algorithm (SHA) cryptographic hash (SHA256). If there's no request payload, set the value to e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855, which is the hash of an empty string.

https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-clients.html#serverless-signing

Expected Behavior

X-Amz-Content-SHA256 should be present when service identifier is 'aoss'

Current Behavior

X-Amz-Content-SHA256 should be present when querying Amazon OpenSearch Serverless

Reproduction Steps

Infra
Client -> APIG -> Lambda -> VPCE -> AOSS (private)

Via the APIGateway Service Console:

1. Create an API Gateway with a Lambda proxy integration.

Via the Lambda Service Console:

1. Create a Lambda function

Via the OpenSearch Service Console:

1. Create an Amazon OpenSearchServerless (AOSS) collection. When creating...
2. Configure the collection with a "private" network policy.
3. Create VPCE to access the private instance. Ensure the VPCE and Lambda are in the same subnet.
   1. Add an Inbound / Outbound rule for the Lambda SG
   2. Add an Inbound / Outbound rule for the AOSS SG
4. Configure the collection with a data access policy. Add the Lambda execution role to the policy.

---

Sample Lambda Code

using System.Net;
using System.Net.Security;
using System.Security.Authentication;
using System.Text;
using System.Text.Json;

using Amazon.Lambda.APIGatewayEvents;
using Amazon.Lambda.Core;
using Amazon.Lambda.Serialization.SystemTextJson;
using Amazon.Runtime;

using AWS.Lambda.Powertools.Logging;

using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.Logging;

using EnvironmentVariables = Insights.Shared.Variables.EnvironmentVariables;
using HttpMethod = System.Net.Http.HttpMethod;

// Assembly attribute to enable the Lambda function's JSON input to be converted into a .NET class.
[assembly: LambdaSerializer(typeof(DefaultLambdaJsonSerializer))]

namespace Functions;

//You can simplify to just query aoss directly in the lambda. This function is a generic proxy.
public class Function
{
    private const string ServiceKey = "awsService";
    private const string ServiceUriKey = "awsUri";
    private static readonly string Region = "us-west-2";
    /// <summary>
    /// Filter out query parameters used to configure the proxy request as these may cause the receiving service to fail. e.g. OpenSearch returns BadRequest when there are excess parameters
    /// </summary>
    private static readonly IReadOnlySet<string> ReservedQueryParameterKeys = new HashSet<string>([ServiceKey, ServiceUriKey]);
    /// <summary>
    /// Headers which should not be forwarded.
    /// See https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Connection
    /// See https://www.rfc-editor.org/rfc/rfc7230#section-6.1
    /// </summary>
    private static readonly IReadOnlySet<string> HopByHopHeaders = new HashSet<string>([
        //sigv4
        "authorization",
        //standard
        "host",
        "connection",
        "keep-alive",
        "proxy-authenticate",
        "proxy-authorization",
        "te",
        "trailer",
        "transfer-encoding",
        "upgrade"
    ]);

    private static readonly TimeSpan Timeout = TimeSpan.FromSeconds(30);
    private static HttpClient CreateHttpClient()
    {
        var handler = new SocketsHttpHandler
        {
            PooledConnectionLifetime = Timeout * 2,
            ConnectTimeout = Timeout,
            SslOptions = new SslClientAuthenticationOptions
            {
                EnabledSslProtocols = SslProtocols.None
            }
        };
        return new HttpClient(handler)
        {
            Timeout = Timeout
        };
    }

    public HttpClient HttpClient { get; init; } = CreateHttpClient();
    public ILogger Logger { get; init; } = AWS.Lambda.Powertools.Logging.Logger.Create<Function>();
    public Func<AWSCredentials> CredentialProvider { get; init; } = FallbackCredentialsFactory.GetCredentials;
    //Absurd levels of indirection to mock things out
    public Func<HttpClient, HttpRequestMessage, string, string, AWSCredentials, Task<HttpResponseMessage>> SignAndSendAsyncFunc { get; init; } = SignAndSendAsync;

    [Logging(LogEvent = true)]
    public async Task<APIGatewayHttpApiV2ProxyResponse> HandleRequestAsync(APIGatewayHttpApiV2ProxyRequest request, ILambdaContext context)
    {
        try
        {
            using HttpResponseMessage response = await this.SendRequestAsync(request);
            string content = await response.Content.ReadAsStringAsync();
            if (response.IsSuccessStatusCode == false)
            {
                string requestContent = await (response.RequestMessage?.Content?.ReadAsStringAsync() ?? Task.FromResult(string.Empty));
                this.Logger.LogError("'{Request}' with '{RequestContent}' failed with '{Response}' '{Content}'", response.RequestMessage, requestContent, response, content);
            }
            return new APIGatewayHttpApiV2ProxyResponse
            {
                StatusCode = (int) response.StatusCode,
                Headers = response.Headers.ToDictionary(kvp => kvp.Key, kvp => string.Join(",", kvp.Value)),
                Body = content,
                IsBase64Encoded = false
            };
        }
        catch (HttpRequestException e)
        {
            this.Logger.LogError(e, "Failed to proxy request to AWS service");
            return new APIGatewayHttpApiV2ProxyResponse
            {
                StatusCode = (int) (e.StatusCode ?? HttpStatusCode.InternalServerError),
                Headers = new Dictionary<string, string>
                {
                    { "Content-Type", "application/json; charset=utf-8" }
                },
                Body = JsonSerializer.Serialize(new ErrorResponse(e.StatusCode ?? HttpStatusCode.InternalServerError, e.Message, request.RequestContext.RequestId), new JsonSerializerOptions().ConfigureDefaults()),
                IsBase64Encoded = false
            };
        }
        catch (Exception e)
        {
            this.Logger.LogError(e, "Failed to proxy request to AWS service");
            return new APIGatewayHttpApiV2ProxyResponse
            {
                StatusCode = StatusCodes.Status400BadRequest,
                Headers = new Dictionary<string, string>
                {
                    { "Content-Type", "application/json; charset=utf-8" }
                },
                Body = JsonSerializer.Serialize(new ErrorResponse(HttpStatusCode.BadRequest, e.Message, request.RequestContext.RequestId), new JsonSerializerOptions().ConfigureDefaults()),
                IsBase64Encoded = false
            };
        }
    }

    private async Task<HttpResponseMessage> SendRequestAsync(APIGatewayHttpApiV2ProxyRequest apiRequest)
    {
        var headers = apiRequest.Headers
            .Select(kvp => new KeyValuePair<string, string>(kvp.Key.ToLowerInvariant(), kvp.Value))
            //Remove 'x-amz' and 'authorization' SigV4 headers as these will cause the signing to fail
            .Where(kvp => kvp.Key.StartsWith("x-amz") == false)
            .Where(kvp => HopByHopHeaders.Contains(kvp.Key) == false)
            .GroupBy(kvp => kvp.Key)
            .ToDictionary(group => group.Key, group => string.Join(',', group.Select(kvp => kvp.Value)));

        if (string.IsNullOrWhiteSpace(apiRequest.RequestContext.Http.Method))
        {
            throw new InvalidOperationException("APIGatewayHttpApiV2ProxyRequest.RequestContext.Http.Method is empty / not set");
        }
        var method = HttpMethod.Parse(apiRequest.RequestContext.Http.Method);

        bool isContentTypeRequired = HttpMethod.Put.Equals(method) || HttpMethod.Post.Equals(method) || HttpMethod.Patch.Equals(method);
        if (headers.TryGetValue("content-type", out string? contentType) == false || string.IsNullOrWhiteSpace(contentType))
        {
            if (isContentTypeRequired)
            {
                throw new InvalidOperationException($"APIGatewayProxyRequest.Header 'content-type' is empty / not set and required for method '{method}'");
            }
        }
        if (apiRequest.PathParameters.TryGetValue("proxy", out string? path) == false || string.IsNullOrWhiteSpace(path))
        {
            throw new InvalidOperationException("APIGatewayHttpApiV2ProxyRequest.PathParameters 'proxy' is empty / not set. Cannot forward request");
        }

        if (apiRequest.QueryStringParameters.TryGetValue(ServiceKey, out string? service) == false || string.IsNullOrWhiteSpace(service))
        {
            throw new InvalidOperationException(
                $"APIGatewayHttpApiV2ProxyRequest.QueryStringParameters '{ServiceKey}' is empty / not set. Cannot forward request. Expect service authorization token. " +
                $"e.g. 'aoss'. See https://docs.aws.amazon.com/service-authorization/latest/reference/reference_policies_actions-resources-contextkeys.html");
        }

        if (apiRequest.QueryStringParameters.TryGetValue(ServiceUriKey, out string? baseUri) == false || string.IsNullOrWhiteSpace(baseUri))
        {
            throw new InvalidOperationException(
                $"APIGatewayHttpApiV2ProxyRequest.QueryStringParameters '{ServiceUriKey}' is empty / not set. Cannot forward request. Expect absolute Uri. e.g. 'https://domain-id.us-west-2.aoss.amazonaws.com'");
        }

        var parameterList = apiRequest.QueryStringParameters
            .Where(kvp => ReservedQueryParameterKeys.Contains(kvp.Key) == false)
            .Select(kvp => $"{kvp.Key}={kvp.Value}")
            .ToList();
        string parameters = parameterList.Count == 0 ? string.Empty : "?" + string.Join("&", parameterList);
        //We need to add '/' because the proxy path parameter does not include it
        var requestUri = new Uri($"{baseUri}/{path}{parameters}", UriKind.Absolute);
        string body = (apiRequest.IsBase64Encoded ? Encoding.UTF8.GetString(Convert.FromBase64String(apiRequest.Body)) : apiRequest.Body) ?? string.Empty;
        var request = new HttpRequestMessage(method, requestUri)
        {
            Content = new StringContent(body)
        };

        foreach (KeyValuePair<string, string> header in headers)
        {
            //Content headers must go into the content headers else it throws
            if (header.Key.StartsWith("content"))
            {
                _ = request.Content.Headers.Remove(header.Key);
                _ = request.Content.Headers.TryAddWithoutValidation(header.Key, header.Value);
            }
            else
            {
                //Remove request defaults because .NET hates humanity
                _ = request.Headers.Remove(header.Key);
                _ = request.Headers.TryAddWithoutValidation(header.Key, header.Value);
            }
        }
        this.Logger.LogInformation("Sending {HttpRequestMessage} to {Service} at {Region}", request, service, Region);
        return await this.SignAndSendAsyncFunc.Invoke(this.HttpClient, request, Region, service, this.CredentialProvider.Invoke());
    }

    private static async Task<HttpResponseMessage> SignAndSendAsync(HttpClient client, HttpRequestMessage request, string region, string service, AWSCredentials credentials) =>
        await client.SendAsync(request, region, service, credentials);

    private sealed record ErrorResponse(HttpStatusCode StatusCode, string Message, string RequestId);
}

Possible Solution

Update https://github.com/FantasticFiasco/aws-signature-version-4/blob/master/src/Private/Signer.cs#L138

Additional Information/Context

Context

I am attempting to proxy requests through API Gateway to our private AOSS collection via VPCE.

Infra
Client -> APIG -> Lambda -> VPCE -> AOSS

Security

1. APIG
   1. IAM Authorizer
2. Lambda
   1. Execution Role
      1. Allows all aoss operations
   2. Verified it is in the same VPC / Subnets as VPCE
3. VPCE
   1. Security Group
      1. Allows Task Ingress / Egress
      2. Allows Lambda Ingress / Egress
      3. Allows AOSS Ingress / Egress
4. AOSS
   1. Network Policy - insightsstack-network-3cd87f61b1
      1. Allows VPCE
   2. DataAccess Policy - insightsstack-data-access-0c0bf1
      1. Allows ECS Task Role
      2. Allows Lambda Task Role

AWS .NET SDK and/or Package version used

3.7.303.38

Targeted .NET Platform

.NET 8

Operating System and version

MacOS 13.6.1 (22G313), Amazon Linux

[github-actions]

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[JCKortlang]

Wrong repo.

:shame:

See FantasticFiasco/aws-signature-version-4#1067