InvalidParameterType error restoring binary attribute

[nathaniel-holder]

Similar to this issue in another repo, I am trying to restore a table that has items with a binary attribute (attribute name text).

Here is the gulp command, and stack trace error:
C:\Development>call gulp --cwd D:\Development\DynamoDbBackUp restore --s3region us-west-2 --dbregion us-west-2 --s3bucket <hidden> --s3prefix DynamoDBFullBackups/2018-01-21T100521-0800/EventTagReply/ --dbtable EventTagReply [17:30:39] Working directory changed to D:\Development\DynamoDbBackUp [17:30:43] Using gulpfile D:\Development\DynamoDbBackUp\gulpfile.js [17:30:43] Starting 'restore'... BuildVersionListFromS3: 974.582ms RFS3 DynamoDBFullBackups/2018-01-21T100521-0800/EventTagReply/01a6e4bc978353e8538310b7721894e6: 1045.912ms P2D DynamoDBFullBackups/2018-01-21T100521-0800/EventTagReply/01a6e4bc978353e8538310b7721894e6: 13.106ms [17:30:46] 'restore' errored after 3.03 s [17:30:46] InvalidParameterType: Expected params.RequestItems['EventTagReply'][0].PutRequest.Item['text'].B to be a string, Buffer, Stream, Blob, or typed array object at ParamValidator.fail (D:\Development\DynamoDbBackUp\node_modules\aws-sdk\lib\param_validator.js:50:37) at ParamValidator.validatePayload (D:\Development\DynamoDbBackUp\node_modules\aws-sdk\lib\param_validator.js:255:10) at ParamValidator.validateScalar (D:\Development\DynamoDbBackUp\node_modules\aws-sdk\lib\param_validator.js:133:21) at ParamValidator.validateMember (D:\Development\DynamoDbBackUp\node_modules\aws-sdk\lib\param_validator.js:94:21) at ParamValidator.validateStructure (D:\Development\DynamoDbBackUp\node_modules\aws-sdk\lib\param_validator.js:75:14) at ParamValidator.validateMember (D:\Development\DynamoDbBackUp\node_modules\aws-sdk\lib\param_validator.js:88:21) at ParamValidator.validateMap (D:\Development\DynamoDbBackUp\node_modules\aws-sdk\lib\param_validator.js:117:14) at ParamValidator.validateMember (D:\Development\DynamoDbBackUp\node_modules\aws-sdk\lib\param_validator.js:92:21) at ParamValidator.validateStructure (D:\Development\DynamoDbBackUp\node_modules\aws-sdk\lib\param_validator.js:75:14) at ParamValidator.validateMember (D:\Development\DynamoDbBackUp\node_modules\aws-sdk\lib\param_validator.js:88:21)

In dynamodb console, the item row looks like this:
{"dateCreated":{"N":"1515430551666"},"dateUpdated":{"N":"1515430580606"},"eventTagID":{"S":"02f5838e-1f06-4677-9351-9923ebffe607"},"eventTagReplyID":{"S":"ff1e0367-b0d0-4239-b5ed-b68576d945a9"},"itemAnalysisID":{"S":"46b3dc88-2f63-4b97-a233-e6f264d04c84"},"ownerID":{"S":"32a3d164-b63f-4680-8eec-ec7c627fd02f"},"text":{"B":"AQICAHhrjsbKx1ato+u+EtU/k7EA9J9ZZCb7083pFVoa5tUI6QEmSOWnb5LIZ2wXGDywa75QAAAAijCBhwYJKoZIhvcNAQcGoHoweAIBADBzBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDKepNDsCyCeqDyBx+wIBEIBGP9EkRei32N/vaM23lWwf2SDDNshaqrK4a2qNIZgLRLV/BekrI2/yLzuOV16C4R5K0ebEQ+GmE22dCBQG2KUY5HY8lQOS5w=="}}

In the S3 backup file of the same item row, it looks like this:
{"eventTagID":{"S":"02f5838e-1f06-4677-9351-9923ebffe607"},"dateCreated":{"N":"1515430551666"},"itemAnalysisID":{"S":"46b3dc88-2f63-4b97-a233-e6f264d04c84"},"text":{"B":{"type":"Buffer","data":[1,2,2,0,120,107,142,198,202,199,86,173,163,235,190,18,213,63,147,177,0,244,159,89,100,38,251,211,205,233,21,90,26,230,213,8,233,1,38,72,229,167,111,146,200,103,108,23,24,60,176,107,190,80,0,0,0,138,48,129,135,6,9,42,134,72,134,247,13,1,7,6,160,122,48,120,2,1,0,48,115,6,9,42,134,72,134,247,13,1,7,1,48,30,6,9,96,134,72,1,101,3,4,1,46,48,17,4,12,167,169,52,59,2,200,39,170,15,32,113,251,2,1,16,128,70,63,209,36,69,232,183,216,223,239,104,205,183,149,108,31,217,32,195,54,200,90,170,178,184,107,106,141,33,152,11,68,181,127,5,233,43,35,111,242,47,59,142,87,94,130,225,30,74,209,230,196,67,225,166,19,109,157,8,20,6,216,165,24,228,118,60,149,3,146,231]}},"ownerID":{"S":"32a3d164-b63f-4680-8eec-ec7c627fd02f"},"dateUpdated":{"N":"1515430580606"},"eventTagReplyID":{"S":"ff1e0367-b0d0-4239-b5ed-b68576d945a9"}}

Am I doing something wrong in the backup or restore method? Has anyone else seen this error when restoring Binary data?

Thanks very much for your help!

~Nathaniel

[shevchenkos]

Hi, Nathaniel.

The main problem is the wrapper around Binary data.
It looks like the result of JSON stringify and parse methods.
The result of JSON.stringify(new Buffer(someString)) is {"type":"Buffer","data":[....]}
The JSON specifies that the type of object being transformed is a Buffer, and its data.

To get the string again, you need to perform 2 operations
let buffer = Buffer.from(JSON.parse(json).data); <-- currently does not exist in restoring
someString = buffer.toString('utf8'));

I see 2 ways to solve the problem

1. check if there is 'data' property in parsed JSON and create Buffer from it
2. save Binary data to S3 in some another way

[nathaniel-holder]

Thanks shevchenkos,

I fixed this by changing the restore code instead of the backup code in order to be able to restore from already backed up data.
Here's my fix:

diff --git a/lib/bin/restoreDynamoDb.js b/lib/bin/restoreDynamoDb.js
index dda0a5e..5bd6362 100644
--- a/lib/bin/restoreDynamoDb.js
+++ b/lib/bin/restoreDynamoDb.js
@@ -74,0 +75,7 @@ class RestoreDynamoDb {
+                Object.keys(fileContents).forEach(attributeName => {
+                    // Fix JSON.stringified Binary data
+                    let attr = fileContents[attributeName];
+                    if (attr.B && attr.B.type && (attr.B.type === 'Buffer') && attr.B.data) {
+                        attr.B = Buffer.from(attr.B.data);
+                    }
+                });

Would this fix be useful to others who restore binary encrypted data? If so, can it be included in a release?

Thanks.
~Nathaniel

[shevchenkos]

Hi, Nathaniel.

It would be nice if you did PR with the changes.
I'll wait for other answers a few days, but in any case I'll make a release with your changes.

[shevchenkos]

Merged and published. @nathaniel-holder, thanks for contribution.