DisableFetchingTableMetadata leads to InvalidOperationException if PropertyConverter is used

[smusenok]

Describe the bug

Saving and deserializing documents throw exceptions when PropertyConverter is specified in DynamoDBRangeKeyAttribute and DisableFetchingTableMetadata = true.

Expected Behavior

Should not throw any exception.

Current Behavior

DynamoDBContext.SaveAsync throws exception:

   Unhandled exception. System.InvalidOperationException: Failed to determine the DynamoDB primitive type for property DateTime of type System.DateTime.
 ---> System.InvalidCastException: Unable to cast Amazon.DynamoDBv2.DocumentModel.UnconvertedDynamoDBEntry as Amazon.DynamoDBv2.DocumentModel.Primitive
   at Amazon.DynamoDBv2.DocumentModel.DynamoDBEntry.Validate[T](T item)
   at Amazon.DynamoDBv2.DocumentModel.DynamoDBEntry.ToPrimitive()
   at Amazon.DynamoDBv2.DocumentModel.Table.GetPrimitiveEntryTypeForProperty(PropertyStorage property, DynamoDBFlatConfig flatConfig)
   --- End of inner exception stack trace ---
   at Amazon.DynamoDBv2.DocumentModel.Table.GetPrimitiveEntryTypeForProperty(PropertyStorage property, DynamoDBFlatConfig flatConfig)
   at Amazon.DynamoDBv2.DocumentModel.Table.CreateTableFromItemStorageConfig(IAmazonDynamoDB client, TableConfig config, ItemStorageConfig itemStorageConfig, DynamoDBFlatConfig flatConfig)
   at Amazon.DynamoDBv2.DataModel.ItemStorageConfigCache.CreateStorageConfig(Type baseType, String actualTableName, DynamoDBFlatConfig flatConfig)
   at Amazon.DynamoDBv2.DataModel.ItemStorageConfigCache.GetConfig(Type type, DynamoDBFlatConfig flatConfig, Boolean conversionOnly)
   at Amazon.DynamoDBv2.DataModel.DynamoDBContext.ObjectToItemStorage(Object toStore, Type objectType, Boolean keysOnly, DynamoDBFlatConfig flatConfig)
   at Amazon.DynamoDBv2.DataModel.DynamoDBContext.SaveHelperAsync(Type valueType, Object value, DynamoDBOperationConfig operationConfig, CancellationToken cancellationToken)
   at Amazon.DynamoDBv2.DataModel.DynamoDBContext.SaveHelperAsync[T](T value, DynamoDBOperationConfig operationConfig, CancellationToken cancellationToken)

DynamoDBContext.FromDocument throws exception:

Unhandled exception. System.InvalidOperationException: Failed to determine the DynamoDB primitive type for property DateTime of type System.DateTime.
 ---> System.InvalidCastException: Unable to cast Amazon.DynamoDBv2.DocumentModel.UnconvertedDynamoDBEntry as Amazon.DynamoDBv2.DocumentModel.Primitive
   at Amazon.DynamoDBv2.DocumentModel.DynamoDBEntry.Validate[T](T item)
   at Amazon.DynamoDBv2.DocumentModel.DynamoDBEntry.ToPrimitive()
   at Amazon.DynamoDBv2.DocumentModel.Table.GetPrimitiveEntryTypeForProperty(PropertyStorage property, DynamoDBFlatConfig flatConfig)
   --- End of inner exception stack trace ---
   at Amazon.DynamoDBv2.DocumentModel.Table.GetPrimitiveEntryTypeForProperty(PropertyStorage property, DynamoDBFlatConfig flatConfig)
   at Amazon.DynamoDBv2.DocumentModel.Table.CreateTableFromItemStorageConfig(IAmazonDynamoDB client, TableConfig config, ItemStorageConfig itemStorageConfig, DynamoDBFlatConfig flatConfig)
   at Amazon.DynamoDBv2.DataModel.ItemStorageConfigCache.CreateStorageConfig(Type baseType, String actualTableName, DynamoDBFlatConfig flatConfig)
   at Amazon.DynamoDBv2.DataModel.ItemStorageConfigCache.GetConfig(Type type, DynamoDBFlatConfig flatConfig, Boolean conversionOnly)
   at Amazon.DynamoDBv2.DataModel.ItemStorageConfigCache.GetConfig[T](DynamoDBFlatConfig flatConfig, Boolean conversionOnly)
   at Amazon.DynamoDBv2.DataModel.DynamoDBContext.FromDocumentHelper[T](Document document, DynamoDBFlatConfig flatConfig)
   at Amazon.DynamoDBv2.DataModel.DynamoDBContext.FromDocument[T](Document document, DynamoDBOperationConfig operationConfig)
   at Amazon.DynamoDBv2.DataModel.DynamoDBContext.FromDocument[T](Document document)

Reproduction Steps

using Amazon.DynamoDBv2;
using Amazon.DynamoDBv2.DataModel;
using Amazon.DynamoDBv2.DocumentModel;
using Amazon.DynamoDBv2.Model;
using Amazon.Runtime;
using System.Globalization;

var config = new AmazonDynamoDBConfig { ServiceURL = "http://localhost:4566" };
var credentials = new BasicAWSCredentials("fake", "fake");
var dynamoDbClient = new AmazonDynamoDBClient(credentials, config);
var dynamoDbContext = new DynamoDBContext(dynamoDbClient, new DynamoDBContextConfig
{
    DisableFetchingTableMetadata = true
});

var tables = await dynamoDbClient.ListTablesAsync();
if (!tables.TableNames.Contains("TestTable"))
{
    await dynamoDbClient.CreateTableAsync(new CreateTableRequest
    {
        TableName = "TestTable",
        AttributeDefinitions =
        [
            new AttributeDefinition
            {
                AttributeName = "Id",
                AttributeType = ScalarAttributeType.S
            },
            new AttributeDefinition
            {
                AttributeName = "DateTime",
                AttributeType = ScalarAttributeType.S
            }
        ],
        KeySchema =
        [
            new KeySchemaElement
            {
                AttributeName = "Id",
                KeyType = KeyType.HASH
            },
            new KeySchemaElement
            {
                AttributeName = "DateTime",
                KeyType = KeyType.RANGE
            }
        ],
        ProvisionedThroughput = new ProvisionedThroughput
        {
            ReadCapacityUnits = 1,
            WriteCapacityUnits = 1
        }
    });

    Console.WriteLine("Table created");
}

await dynamoDbContext.SaveAsync(new TestType
{
    Id = "1",
    DateTime = DateTime.UtcNow
});
Console.WriteLine("Item saved.");

Dictionary<string, AttributeValue> itemAttributes = (await dynamoDbClient.QueryAsync(
    new QueryRequest
    {
        TableName = "TestTable",
        KeyConditionExpression = "Id = :Id",
        ExpressionAttributeValues = new Dictionary<string, AttributeValue>
        {
            [":Id"] = new("1")
        },
        Limit = 1
    },
    CancellationToken.None)).Items.FirstOrDefault()!;

var item = dynamoDbContext.FromDocument<TestType>(Document.FromAttributeMap(itemAttributes));

Console.WriteLine($"Item loaded: {item}");

[DynamoDBTable("TestTable")]
sealed record TestType
{
    [DynamoDBHashKey]
    public string Id { get; set; }

    [DynamoDBRangeKey(typeof(DateTimeConverter))]
    public DateTime DateTime { get; set; }
}

sealed class DateTimeConverter : IPropertyConverter
{
    public DynamoDBEntry ToEntry(object value)
    {
        return ((DateTime)value).ToString(@"yyyy-MM-dd\THH:mm:ss.fff\Z", CultureInfo.InvariantCulture);
    }

    public object FromEntry(DynamoDBEntry entry)
    {
        return DateTime.Parse(entry.AsString(), CultureInfo.InvariantCulture, DateTimeStyles.AdjustToUniversal);
    }
}

Possible Solution

No response

Additional Information/Context

Changing DisableFetchingTableMetadata to false makes this code work using AWSSDK.DynamoDBv2 3.7.303.9.
The last known version where this example works properly with DisableFetchingTableMetadata = true is 3.7.302.23.

AWS .NET SDK and/or Package version used

AWSSDK.DynamoDBv2 3.7.303.9

Targeted .NET Platform

.NET 8

Operating System and version

Windows 10, Ubuntu 22.04.4 LTS

[ashovlin]

Recent PR to this area: #3304

[bhoradc]

Hi @smusenok,

Thank you for reporting the issue. I am able to reproduce the InvalidOperationException with your provided code sample using AWSSDK.DynamoDBv2 3.7.302.26 and above.

Team will further investigate this issue and provide a fix, if applicable.

Regards,
Chaitanya

[muhammad-othman]

Hi @smusenok,
I was able to reproduce the same issue after the merge of this pull request #3304. was merged, it now occurs for a different reason which we will address and work on fixing.

In the meantime, if you need to use the DateTimeConverter, you can update the ToEntry method as follows. This modification should allow successful conversion of DateTime values:

    public DynamoDBEntry ToEntry(object value)
    {
        Primitive primitiveValue = ((DateTime)value).ToString(@"yyyy-MM-dd\THH:mm:ss.fff\Z", CultureInfo.InvariantCulture);
        return primitiveValue ;
    }

[smusenok]

Hi @muhammad-othman,
Thank you! Will use your implementation of DateTimeConverter.

[ashovlin]

We shipped AWSSDK.DynamoDBv2 version 3.7.303.16 yesterday, which contained #3327 which will attempt another cast to a valid type for a table or index key in the case where an IPropertyConverter doesn't return a Primitive directly.

Let us know if you're still seeing issues (or new issues) on the new version.

[github-actions]

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.