Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Syncing between two regions fails #72

Open
nicon89 opened this issue Nov 24, 2022 · 9 comments
Open

Syncing between two regions fails #72

nicon89 opened this issue Nov 24, 2022 · 9 comments
Labels
question Further information is requested

Comments

@nicon89
Copy link

nicon89 commented Nov 24, 2022

I'm trying to sync buckets between eu-central-1 and me-central-1 regions, but I'm getting error:

Error:
  class: class Error
  stack:
    IllegalLocationConstraintException: The me-central-1 location constraint is incompatible for the region specific endpoint this request was sent to.
        at Request.extractError (/home/ubuntu/.npm/_npx/301b345c5affbb3b/node_modules/aws-sdk/lib/services/s3.js:711:35)
        at Request.callListeners (/home/ubuntu/.npm/_npx/301b345c5affbb3b/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
        at Request.emit (/home/ubuntu/.npm/_npx/301b345c5affbb3b/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
        at Request.emit (/home/ubuntu/.npm/_npx/301b345c5affbb3b/node_modules/aws-sdk/lib/request.js:686:14)
        at Request.transition (/home/ubuntu/.npm/_npx/301b345c5affbb3b/node_modules/aws-sdk/lib/request.js:22:10)
        at AcceptorStateMachine.runTo (/home/ubuntu/.npm/_npx/301b345c5affbb3b/node_modules/aws-sdk/lib/state_machine.js:14:12)
        at /home/ubuntu/.npm/_npx/301b345c5affbb3b/node_modules/aws-sdk/lib/state_machine.js:26:10
        at Request.<anonymous> (/home/ubuntu/.npm/_npx/301b345c5affbb3b/node_modules/aws-sdk/lib/request.js:38:9)
        at Request.<anonymous> (/home/ubuntu/.npm/_npx/301b345c5affbb3b/node_modules/aws-sdk/lib/request.js:688:12)
        at Request.callListeners (/home/ubuntu/.npm/_npx/301b345c5affbb3b/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
        at Request.emit (/home/ubuntu/.npm/_npx/301b345c5affbb3b/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
        at Request.emit (/home/ubuntu/.npm/_npx/301b345c5affbb3b/node_modules/aws-sdk/lib/request.js:686:14)
        at Request.transition (/home/ubuntu/.npm/_npx/301b345c5affbb3b/node_modules/aws-sdk/lib/request.js:22:10)
        at AcceptorStateMachine.runTo (/home/ubuntu/.npm/_npx/301b345c5affbb3b/node_modules/aws-sdk/lib/state_machine.js:14:12)
        at /home/ubuntu/.npm/_npx/301b345c5affbb3b/node_modules/aws-sdk/lib/state_machine.js:26:10
        at Request.<anonymous> (/home/ubuntu/.npm/_npx/301b345c5affbb3b/node_modules/aws-sdk/lib/request.js:38:9)
        at Request.<anonymous> (/home/ubuntu/.npm/_npx/301b345c5affbb3b/node_modules/aws-sdk/lib/request.js:688:12)
        at Request.callListeners (/home/ubuntu/.npm/_npx/301b345c5affbb3b/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
        at callNextListener (/home/ubuntu/.npm/_npx/301b345c5affbb3b/node_modules/aws-sdk/lib/sequential_executor.js:96:12)
        at IncomingMessage.onEnd (/home/ubuntu/.npm/_npx/301b345c5affbb3b/node_modules/aws-sdk/lib/event_listeners.js:383:13)
        at IncomingMessage.emit (events.js:326:22)
        at IncomingMessage.EventEmitter.emit (domain.js:483:12)

I checked the documentation and I can't find any option for setting the region.

Is it possible to sync buckets in different regions using s3p?
@siebrand
Copy link

siebrand commented Nov 24, 2022 •
edited
Loading

Do you have an S3 endpoint in the VPC you have the instance in that you're trying to sync from? If yes, sync in a VPC without an S3 endpoint.

@shanebdavis
Copy link
Member

s3p was originally written with cross-region syncing in mind. However, it may not work in all configurations. Can you share your s3p command to provide more information?

@shanebdavis shanebdavis added the question Further information is requested label Dec 19, 2022
@nicon89
Copy link
Author

nicon89 commented Jan 14, 2023

Unfortunately no, I don't have it it my bash history anymore. I used different tool eventually... Thank you.

@nicon89 nicon89 closed this as completed Jan 14, 2023
@nicon89 nicon89 reopened this Jan 24, 2023
@nicon89
Copy link
Author

nicon89 commented Jan 24, 2023 •
edited
Loading

Right now I'm trying just to run summarize, but it's failing with same error:

ubuntu@xyz:~ $ npx s3p summarize --bucket XYZ
S3.list-error:
  bucket:     :XYZ
  prefix:     undefined
  startAfter: ""
  limit:      1000
  error:      Error:
    class: class Error
    stack:
      IllegalLocationConstraintException: The me-central-1 location constraint is incompatible for the region specific endpoint this request was sent to.
          at Request.extractError (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/services/s3.js:711:35)
          at Request.callListeners (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
          at Request.emit (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
          at Request.emit (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:686:14)
          at Request.transition (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:22:10)
          at AcceptorStateMachine.runTo (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/state_machine.js:14:12)
          at /usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/state_machine.js:26:10
          at Request.<anonymous> (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:38:9)
          at Request.<anonymous> (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:688:12)
          at Request.callListeners (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
          at Request.emit (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
          at Request.emit (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:686:14)
          at Request.transition (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:22:10)
          at AcceptorStateMachine.runTo (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/state_machine.js:14:12)
          at /usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/state_machine.js:26:10
          at Request.<anonymous> (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:38:9)
          at Request.<anonymous> (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:688:12)
          at Request.callListeners (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
          at callNextListener (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/sequential_executor.js:96:12)
          at IncomingMessage.onEnd (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/event_listeners.js:417:13)
          at IncomingMessage.emit (events.js:326:22)
          at IncomingMessage.EventEmitter.emit (domain.js:483:12)

eachRecursive:
  startAfter: ""
  stopAt:
    :~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  usePrefixBisect: false
  error:           Error:
    class: class Error
    stack:
      IllegalLocationConstraintException: The me-central-1 location constraint is incompatible for the region specific endpoint this request was sent to.
          at Request.extractError (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/services/s3.js:711:35)
          at Request.callListeners (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
          at Request.emit (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
          at Request.emit (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:686:14)
          at Request.transition (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:22:10)
          at AcceptorStateMachine.runTo (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/state_machine.js:14:12)
          at /usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/state_machine.js:26:10
          at Request.<anonymous> (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:38:9)
          at Request.<anonymous> (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:688:12)
          at Request.callListeners (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
          at Request.emit (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
          at Request.emit (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:686:14)
          at Request.transition (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:22:10)
          at AcceptorStateMachine.runTo (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/state_machine.js:14:12)
          at /usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/state_machine.js:26:10
          at Request.<anonymous> (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:38:9)
          at Request.<anonymous> (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:688:12)
          at Request.callListeners (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
          at callNextListener (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/sequential_executor.js:96:12)
          at IncomingMessage.onEnd (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/event_listeners.js:417:13)
          at IncomingMessage.emit (events.js:326:22)
          at IncomingMessage.EventEmitter.emit (domain.js:483:12)

Error:
  class: class Error
  stack:
    IllegalLocationConstraintException: The me-central-1 location constraint is incompatible for the region specific endpoint this request was sent to.
        at Request.extractError (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/services/s3.js:711:35)
        at Request.callListeners (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
        at Request.emit (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
        at Request.emit (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:686:14)
        at Request.transition (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:22:10)
        at AcceptorStateMachine.runTo (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/state_machine.js:14:12)
        at /usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/state_machine.js:26:10
        at Request.<anonymous> (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:38:9)
        at Request.<anonymous> (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:688:12)
        at Request.callListeners (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
        at Request.emit (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
        at Request.emit (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:686:14)
        at Request.transition (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:22:10)
        at AcceptorStateMachine.runTo (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/state_machine.js:14:12)
        at /usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/state_machine.js:26:10
        at Request.<anonymous> (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:38:9)
        at Request.<anonymous> (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/request.js:688:12)
        at Request.callListeners (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
        at callNextListener (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/sequential_executor.js:96:12)
        at IncomingMessage.onEnd (/usr/local/lib/node_modules/s3p/node_modules/aws-sdk/lib/event_listeners.js:417:13)
        at IncomingMessage.emit (events.js:326:22)
        at IncomingMessage.EventEmitter.emit (domain.js:483:12)

Bucket is in me-central-1, machine from which I run s3p is also from me-central-1.

@shanebdavis
Copy link
Member

That's a weird one. All the info I'm seeing on IllegalLocationConstraintException has to do with bucket creation... LocationConstraint. Are you able to do a "s3p summarize" on both buckets?

@guilhermegazzinelli
Copy link

guilhermegazzinelli commented Nov 20, 2023 •
edited
Loading

I'm heaving a weir behavior tying out S3P. My scenario is:
source_bucket: source ( sa-east-1 )
dest_bucket: dest ( us-east-1)

Command:
AWS_PROFILE=app_profile npx s3p sync --bucket source --to-bucket dest

S3.list-error:
 bucket:     :source
 prefix:     undefined
 startAfter: ""
 limit:      1000
 error:      Error:
   class: class Error
   stack:
     Error: Region is missing
         at default (/home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/config-resolver/dist-cjs/regionConfig/config.js:10:15)
         at /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/node-config-provider/dist-cjs/fromStatic.js:6:83
         at /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/property-provider/dist-cjs/chain.js:12:39

S3.list-error:
 bucket:     :source
 prefix:     undefined
 startAfter: :O
 limit:      1000
 error:      Error:
   class: class Error
   stack:
     Error: Region is missing
         at default (/home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/config-resolver/dist-cjs/regionConfig/config.js:10:15)
         at /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/node-config-provider/dist-cjs/fromStatic.js:6:83
         at /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/property-provider/dist-cjs/chain.js:12:39

eachRecursive:
 startAfter: ""
 stopAt:
   :~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 usePrefixBisect: false
 error:           Error:
   class: class Error
   stack:
     Error: Region is missing
         at default (/home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/config-resolver/dist-cjs/regionConfig/config.js:10:15)
         at /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/node-config-provider/dist-cjs/fromStatic.js:6:83
         at /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/property-provider/dist-cjs/chain.js:12:39

Error:
 class: class Error
 stack:
   Error: Region is missing
       at default (/home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/config-resolver/dist-cjs/regionConfig/config.js:10:15)
       at /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/node-config-provider/dist-cjs/fromStatic.js:6:83
       at /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/property-provider/dist-cjs/chain.js:12:39

If I pass AWS_REGION=us_east_1 the error is show in source s3 URL as follows:

S3.list-error:
  bucket:     :source
  prefix:     undefined
  startAfter: ""
  limit:      1000
  error:      Error:
    class: class S3ServiceException
    stack:
      PermanentRedirect: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.
          at throwDefaultError (/home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/smithy-client/dist-cjs/default-error-handler.js:8:22)
          at /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/smithy-client/dist-cjs/default-error-handler.js:18:39
          at de_ListObjectsV2CommandError (/home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:5105:20)
          at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
          at async /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
          at async /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/middleware-signing/dist-cjs/awsAuthMiddleware.js:14:20
          at async /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/middleware-retry/dist-cjs/retryMiddleware.js:27:46
          at async /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/region-redirect-endpoint-middleware.js:14:24
          at async /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/region-redirect-middleware.js:9:20
          at async /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:7:26

S3.list-error:
  bucket:     :source
  prefix:     undefined
  startAfter: :O
  limit:      1000
  error:      Error:
    class: class S3ServiceException
    stack:
      PermanentRedirect: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.
          at throwDefaultError (/home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/smithy-client/dist-cjs/default-error-handler.js:8:22)
          at /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/smithy-client/dist-cjs/default-error-handler.js:18:39
          at de_ListObjectsV2CommandError (/home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:5105:20)
          at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
          at async /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
          at async /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/middleware-signing/dist-cjs/awsAuthMiddleware.js:14:20
          at async /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/middleware-retry/dist-cjs/retryMiddleware.js:27:46
          at async /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/region-redirect-endpoint-middleware.js:14:24
          at async /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/region-redirect-middleware.js:9:20
          at async /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:7:26

Error:
  class: class S3ServiceException
  stack:
    PermanentRedirect: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.
        at throwDefaultError (/home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/smithy-client/dist-cjs/default-error-handler.js:8:22)
        at /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/smithy-client/dist-cjs/default-error-handler.js:18:39
        at de_ListObjectsV2CommandError (/home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:5105:20)
        at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
        at async /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
        at async /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/middleware-signing/dist-cjs/awsAuthMiddleware.js:14:20
        at async /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@smithy/middleware-retry/dist-cjs/retryMiddleware.js:27:46
        at async /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/region-redirect-endpoint-middleware.js:14:24
        at async /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/region-redirect-middleware.js:9:20
        at async /home/ggr/.npm/_npx/301b345c5affbb3b/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:7:26

Tested:

  • Summarize with correct region works, but not without AWS_REGION
  • Command sync works in aws ( aws s3 --profile app_profile sync s3://sources s3://dest

INFO:
S3P version = 3.5.2

@guilhermegazzinelli guilhermegazzinelli mentioned this issue Nov 20, 2023
Open
@shanebdavis
Copy link
Member

Since s3p is mostly just a smart wrapper around the AWS SDK and cli, maybe you can show me an example of the command you'd use in the aws-cli to copy between the two buckets. Do you specify an endpoint explicitly on the aws-cli? If so, that should give me a hint how to configure the AWS-SDK to do the same thing...

@leogargu
Copy link

leogargu commented Apr 9, 2024

Since s3p is mostly just a smart wrapper around the AWS SDK and cli, maybe you can show me an example of the command you'd use in the aws-cli to copy between the two buckets. Do you specify an endpoint explicitly on the aws-cli? If so, that should give me a hint how to configure the AWS-SDK to do the same thing...

With the aws cli there's the --source-region option (docs):

--source-region (string) When transferring objects from an s3 bucket to an s3 bucket, this specifies the region of the source bucket. Note the region specified by --region or through configuration of the CLI refers to the region of the destination bucket. If --source-region is not specified the region of the source will be the same as the region of the destination bucket.

@bit-herder
Copy link

i don't understand how this was developed specifically for this use case and there is no documentation on how to do a sync with buckets in different regions. There seems to be no way to specify the source region. Am i missing something fundamental here?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
question Further information is requested
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@shanebdavis @siebrand @bit-herder @leogargu @guilhermegazzinelli @nicon89