Add resiliance for source buckets that have deletions while synchronizing

[siebrand]

I'm using S3P sync on an active source bucket, which in my case means that objects are being added and removed from it. Adding objects doesn't really have any impact, as far as I can tell; new objects are simply not listed, and will not be synchronized, and that's fine. However, when an object has been listed, and is deleted between being put in the queue for copying and being actually copied, this leads to a fatal problem, and output like this:

eachRecursive:
  startAfter:      :6C8i/6C8igoZYa988VoGYVR_KfL/2_renditions/thumbnail.jpg
  stopAt:          :6D
  usePrefixBisect: false
  error:           Error:
    class: class Error
    stack:
      NoSuchKey: The specified key does not exist.
          at Request.extractError (/root/node_modules/aws-sdk/lib/services/s3.js:711:35)
          at Request.callListeners (/root/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
          at Request.emit (/root/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
          at Request.emit (/root/node_modules/aws-sdk/lib/request.js:686:14)
          at Request.transition (/root/node_modules/aws-sdk/lib/request.js:22:10)
          at AcceptorStateMachine.runTo (/root/node_modules/aws-sdk/lib/state_machine.js:14:12)
          at /root/node_modules/aws-sdk/lib/state_machine.js:26:10
          at Request.<anonymous> (/root/node_modules/aws-sdk/lib/request.js:38:9)
          at Request.<anonymous> (/root/node_modules/aws-sdk/lib/request.js:688:12)
          at Request.callListeners (/root/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
          at Request.emit (/root/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
          at Request.emit (/root/node_modules/aws-sdk/lib/request.js:686:14)
          at Request.transition (/root/node_modules/aws-sdk/lib/request.js:22:10)
          at AcceptorStateMachine.runTo (/root/node_modules/aws-sdk/lib/state_machine.js:14:12)
          at /root/node_modules/aws-sdk/lib/state_machine.js:26:10
          at Request.<anonymous> (/root/node_modules/aws-sdk/lib/request.js:38:9)
          at Request.<anonymous> (/root/node_modules/aws-sdk/lib/request.js:688:12)
          at Request.callListeners (/root/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
          at callNextListener (/root/node_modules/aws-sdk/lib/sequential_executor.js:96:12)
          at IncomingMessage.onEnd (/root/node_modules/aws-sdk/lib/event_listeners.js:417:13)
          at IncomingMessage.emit (node:events:538:35)
          at IncomingMessage.emit (node:domain:475:12) 

Would it be possible to not make this a fatal error?

[shanebdavis]

Sounds like a reasonable request.

[shanebdavis]

I don't have an easy way to test this, but [email protected] should meet your needs. Sorry it took so long!

[siebrand]

This should be a game changer for my use case, and provide way more reliable and complete copies. Thanks so much for working on this. I’ll look into the logs for cases that are known to fail often next week Wednesday and will provide feedback here.

[shanebdavis]

Have you verified if this solved the problem?

This commit should solve this sort of problem:

019c843 5 months ago patch/improvement: now "NoSuchKey: The specified key does not exist" are logged but are not fatal

[siebrand]

Sorry, not tested because I haven't been able to get any further with using 3.5.x widely because of #90.