Inconsistency in AWS .NET SDK: Handling Empty Address List in WAF V2 UpdateIPSet Method

[DarthHaider]

Describe the bug

I'm encountering an issue in the AWS SDK for .NET, specifically when using the UpdateIPSet method within the WAF V2 wafv2Client class within a Lambda function. The problem arises when attempting to update an existing IP set to an empty list, a function that seems inconsistent with the corresponding AWS CLI call.

Expected Behavior

Using the AWS CLI, the following command can be executed:

aws wafv2 update-ip-set \
  --name "restapiname" \
  --scope REGIONAL \
  --id "e107584d-1112-212121-21218b21ad-9a0211212c2121" \
  --addresses "[]" \
  --lock-token "$LOCK_TOKEN"

Current Behavior

When using the .NET SDK's AmazonWAFV2Request, the following code snippet is used, but it returns an error when attempting to set the IP set to a null or empty array:

var updateIPSetRequest = new Amazon.WAFV2.Model.UpdateIPSetRequest
{
    Id = ipSetId,
    Scope = SCOPE_REGIONAL,
    Addresses = addresses ?? new System.Collections.Generic.List<System.String>(),
    LockToken = getIPSetResponse.LockToken,
    Name = ipSetName
};
var response = await wafClient.UpdateIPSetAsync(updateIPSetRequest);

The error message is:

An error occurred while processing the record: 1 validation error detected: Value null at 'addresses' failed to satisfy constraint: Member must not be null

Reproduction Steps

1. Create a new Amazon.WAFV2.Model.UpdateIPSetRequest object.
2. Set the Addresses property to an empty list within the Lambda function.
3. Call the UpdateIPSetAsync method on a wafClient instance.
4. Observe the error message.

Possible Solution

I believe this might be an SDK issue, as the current package listed on GitHub contains the following code:

[AWSProperty(Required=true)]
public List<string> Addresses
{
    get { return this._addresses; }
    set { this._addresses = value; }
}

// Check to see if Addresses property is set
internal bool IsSetAddresses()
{
    return this._addresses != null && this._addresses.Count > 0; 
}

Here, the method explicitly checks that the array is not empty, effectively preventing the setting of an empty address list.

Additional Information/Context

Lambda Function Code

using Amazon.Lambda.Core;
using Amazon.Lambda.SQSEvents;
using System.Text;
using System.Text.Json;
using Amazon.WAFV2;
using Amazon.WAFV2.Model;

// Assembly attribute to enable the Lambda function's JSON input to be converted into a .NET class.
[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]

namespace DeregisterIpInWafIpv4IpSet;
public class Function
{
    private const string SCOPE_REGIONAL = "REGIONAL"; // Scope used in WAF requests
    private string TargetGroupArn { get; set; } // Target group ARN
    private string IpSetArn { get; set; } // IP set ARN
    private AmazonWAFV2Client wafClient; // Client to interact with WAF

    public Function()
    {
        // Initialize properties from environment variables
        TargetGroupArn = Environment.GetEnvironmentVariable("TARGET_GROUP_ARN");
        IpSetArn = Environment.GetEnvironmentVariable("IP_SET_ARN");
        wafClient = new AmazonWAFV2Client();
    }

    // Main Lambda function handler
    public async Task<string> FunctionHandler(SQSEvent sqsEvent, ILambdaContext context)
    {
        UpdateIPSetResponse lastUpdateIPSetResponse = null;

        context.Logger.LogLine($"Update IP Set Response: {JsonSerializer.Serialize(sqsEvent)}");


        // Log and process each record in the SQS event
        foreach (var record in sqsEvent.Records)
        {
            try
            {
                lastUpdateIPSetResponse = await HandleRecord(record, context);
            }
            catch (Exception ex)
            {
                // Handle exceptions during record processing
                context.Logger.LogLine($"An error occurred while processing the record: {ex.Message}");
            }
        }

        return lastUpdateIPSetResponse != null ? JsonSerializer.Serialize(lastUpdateIPSetResponse) : "No updates were made";
    }

    // Handle individual record from SQS
    private async Task<UpdateIPSetResponse> HandleRecord(SQSEvent.SQSMessage record, ILambdaContext context)
    {
        // Extract IP address and log information
        string ipAddress = ExtractIpAddress(DeserializeBodyContent(record.Body)) + "/32";
        context.Logger.LogLine($"IP Address: {ipAddress}");

        // Extract IP set information
        string[] ipSetComponents = IpSetArn.Split('/');
        string ipSetName = ipSetComponents[^2];
        string ipSetId = ipSetComponents[^1];

        // Get current IP set details
        var getIPSetResponse = await wafClient.GetIPSetAsync(new GetIPSetRequest { Id = ipSetId, Scope = SCOPE_REGIONAL, Name = ipSetName });
        List<string> addresses = getIPSetResponse.IPSet.Addresses;
        context.Logger.LogLine($"Existing Addresses: {string.Join(", ", addresses)}");

        if (!addresses.Contains(ipAddress))
        {
            context.Logger.LogLine($"IP Address {ipAddress} already exists in the IP Set. No updates were made.");
            return null; // Return early if IP already exists
        }

        context.Logger.LogLine($"Before the removal: {JsonSerializer.Serialize(addresses)}");

        addresses.Remove(ipAddress);
        context.Logger.LogLine($"After the removal: {JsonSerializer.Serialize(addresses)}");

        // Create update request with new IP
        var updateIPSetRequest = new Amazon.WAFV2.Model.UpdateIPSetRequest
        {
            Id = ipSetId,
            Scope = SCOPE_REGIONAL,
            Addresses = addresses ?? new System.Collections.Generic.List<System.String>(),
            LockToken = getIPSetResponse.LockToken,
            Name = ipSetName
        };

        context.Logger.LogLine($"Update IP Request: {JsonSerializer.Serialize(updateIPSetRequest)}");

        context.Logger.LogLine($"Type of updateIPSetRequest.Addresses: {updateIPSetRequest.Addresses.GetType()}");

        // Update IP set and log response
        var response = await wafClient.UpdateIPSetAsync(updateIPSetRequest);
        context.Logger.LogLine($"Update IP Set Response: {JsonSerializer.Serialize(response)}");
        return response;
    }

    // Deserialize the body content into a dictionary
    private Dictionary<string, object> DeserializeBodyContent(string bodyContent)
    {
        return JsonSerializer.Deserialize<Dictionary<string, object>>(bodyContent);
    }

    // Extract IP address from JSON body
    private string ExtractIpAddress(Dictionary<string, object> bodyJson)
    {
        var detail = JsonSerializer.Deserialize<Dictionary<string, object>>(bodyJson["detail"].ToString());
        var requestParameters = JsonSerializer.Deserialize<Dictionary<string, object>>(detail["requestParameters"].ToString());
        var targets = JsonSerializer.Deserialize<List<Dictionary<string, object>>>(requestParameters["targets"].ToString());
        return targets[0]["id"].ToString();
    }
}

AWS .NET SDK and/or Package version used

<PackageReference Include="Amazon.Lambda.Core" Version="2.1.0" />
<PackageReference Include="Amazon.Lambda.Serialization.SystemTextJson" Version="2.3.1" />
<PackageReference Include="Amazon.Lambda.SQSEvents" Version="2.1.0" />
<PackageReference Include="AWSSDK.SQS" Version="3.7.200.22" />

Targeted .NET Platform

.NET 6

Operating System and version

Mac OS

[ashishdhingra]

Looks like the issue is due to the check publicRequest.IsSetAddresses() in UpdateIPSetRequestMarshaller.Marshall() method. The UpdateIPSetRequest.IsSetAddresses() returns true only if Addresses is not empty. Hence the UpdateIPSetRequestMarshaller doesn't marhall empty addresses list causing the exception from the service API.

This is a known issue in current SDK version where collections in model classes are initialized to empty collection, hence marshaller has no way to distinguish the intent of user for using empty collection, which is supported by service API in some cases. This is a likely candidate for not initializing collections to empty list by default in new major version of SDK.

The workaround could be to write wafv2.customizations.json under https://github.com/aws/aws-sdk-net/tree/master/generator/ServiceModels/wafv2, similar to what was done in 088a0c5 to expose public IsSetAddresses() method, which could be initialized explicitly by user to true to get around the issue.

[ashishdhingra]

@DarthHaider The workaround has been released in AWSSDK.WAFV2 version 3.7.202.5. To use the workaround, you would need to explicitly set IsAddressesSet to true as shown in below code:

using Amazon.WAFV2.Model;

using (AmazonWAFV2Client client = new AmazonWAFV2Client())
{
    var getResponse = await client.GetIPSetAsync(new GetIPSetRequest()
    {
        Id = "<<REDACTED-IPSet-Id>>",
        Name = "TestIPSet",
        Scope = Scope.REGIONAL
    });
    var response = await client.UpdateIPSetAsync(new UpdateIPSetRequest()
    {
        Id = "<<REDACTED-IPSet-Id>>",
        Scope = Scope.REGIONAL,
        Name= "TestIPSet",
        Addresses =new List<string>(),
        LockToken = getResponse.LockToken,
        IsAddressesSet = true
    });

    getResponse = await client.GetIPSetAsync(new GetIPSetRequest()
    {
        Id = "<<REDACTED-IPSet-Id>>",
        Name = "TestIPSet",
        Scope = Scope.REGIONAL
    });
}

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.