S3Control CreateJobAsync with LambdaInvoke throws 'Request invalid' exception due to empty UserArguments

[scott-brady]

Describe the bug

I am attempting to create an S3 Batch Operations job that invokes a Lambda function from an S3 inventory manifest. I am using the CreateJobAsync() method of the AmazonS3ControlClient class. This throws a Amazon.S3Control.AmazonS3ControlException with the message Request invalid.

Expected Behavior

No exception is thrown and the job is created successfully.

Current Behavior

Exception Amazon.S3Control.AmazonS3ControlException is thrown.

Reproduction Steps

I was able to reproduce with a simple .NET 8 console app as follows:

S3ControlClient.csproj:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net8.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="AWSSDK.S3Control" Version="3.7.400" />
  </ItemGroup>

</Project>

Program.cs:

using Amazon;
using Amazon.Runtime;
using Amazon.S3Control;
using Amazon.S3Control.Model;

var credentials = new SessionAWSCredentials("accessKey", "secretKey", "sessionToken");
var region = RegionEndpoint.APSoutheast2;

var config = new AmazonS3ControlConfig()
{
    RegionEndpoint = region,
};

using var s3ControlClient = new AmazonS3ControlClient(credentials, config);

var createJobRequest = new CreateJobRequest
{
    AccountId = "999999999999",
    Description = "My Job",
    ConfirmationRequired = true,
    Priority = 200,
    RoleArn = "arn:aws:iam::999999999999:role/myRole",
    Operation = new JobOperation
    {
        LambdaInvoke = new LambdaInvokeOperation
        {
            FunctionArn = "arn:aws:lambda:ap-southeast-2:999999999999:function:myFunction",
        }
    },
    Manifest = new JobManifest
    {
        Spec = new JobManifestSpec
        {
            Format = "S3InventoryReport_CSV_20161130",
        },
        Location = new JobManifestLocation
        {
            ObjectArn = "arn:aws:s3:::my-inventory-bucket/inventory/2024-07-21T01-00Z/manifest.json",
            ETag = "678081b240e5777a810379f7ba97917d"
        }
    },
    Report = new JobReport
    {
        Enabled = true,
        ReportScope = "FailedTasksOnly",
        Bucket = "arn:aws:s3:::my-report-bucket",
        Prefix = "job-report",
        Format = "Report_CSV_20180820"
    }
};

var createJobResponse = await s3ControlClient.CreateJobAsync(createJobRequest);

Console.WriteLine(createJobResponse.JobId);

Possible Solution

I was able to capture and replay the request to get a successful response.

The captured request was as follows:

<CreateJobRequest
	xmlns="http://awss3control.amazonaws.com/doc/2018-08-20/">
	<ClientRequestToken>756238c1-d32b-418d-bfd4-161cbac78182</ClientRequestToken>
	<ConfirmationRequired>true</ConfirmationRequired>
	<Description>My Job</Description>
	<Manifest>
		<Location>
			<ETag>678081b240e5777a810379f7ba97917d</ETag>
			<ObjectArn>arn:aws:s3:::my-inventory-bucket/inventory/2024-07-21T01-00Z/manifest.json</ObjectArn>
		</Location>
		<Spec>
			<Format>S3InventoryReport_CSV_20161130</Format>
		</Spec>
	</Manifest>
	<Operation>
		<LambdaInvoke>
			<FunctionArn>arn:aws:lambda:ap-southeast-2:999999999999:function:myFunction</FunctionArn>
			<InvocationSchemaVersion>1.0</InvocationSchemaVersion>
			<UserArguments />
		</LambdaInvoke>
	</Operation>
	<Priority>200</Priority>
	<Report>
		<Bucket>arn:aws:s3:::my-report-bucket</Bucket>
		<Enabled>true</Enabled>
		<Format>Report_CSV_20180820</Format>
		<Prefix>job-report</Prefix>
		<ReportScope>FailedTasksOnly</ReportScope>
	</Report>
	<RoleArn>arn:aws:iam::999999999999:role/myRole</RoleArn>
</CreateJobRequest>

This failed with a 400 Bad Request with the following response:

<ErrorResponse>
	<Error>
		<Code>InvalidRequest</Code>
		<Message>Request invalid</Message>
	</Error>
	<RequestId>Q5KH9XAF65EMXVYH</RequestId>
	<HostId>nyIXBZzz2fW2wx0XLxXsLHia3fC1JLFIROBRYic0uUl+n0ppTmIu2Xitke8RuOOCvIQIwpivRGQ=</HostId>
</ErrorResponse>

I noticed the UserArguments empty element which seemed unusual since I am using 1.0 for the InvocationSchemaVersion. So I removed this empty element and retried and the request then succeeded with a 200 OK with the following response:

<CreateJobResult>
  <JobId>9445cb5f-5324-4e3b-819b-cd619f788c01</JobId>
</CreateJobResult>

So it appears the issue is caused by the empty UserArguments element. This should not be there since I have not set UserArguments in my CreateJobRequest.

Additional Information/Context

No response

AWS .NET SDK and/or Package version used

AWSSDK.S3Control 3.7.400

Targeted .NET Platform

.NET 8.0

Operating System and version

macOS Sonoma 14.5

[dscpinheiro]

Apologies for the delay in the response, but this should be fixed in the latest version of the AWSSDK.S3Control package we just released (https://www.nuget.org/packages/AWSSDK.S3Control/3.7.400.12). I tried running your example and was able to create the job successfully.

The issue was a bug where we were writing the UserArguments property into the request body regardless if it was populated or not (this only impacted the CreateJob operation which is why we hadn't caught it before).

[github-actions]

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.