source-amazon-seller-partner: wait_to_avoid_fatal_errors does not work as advertised

[keiththompson]

Connector Name

source-amazon-seller-partner

Connector Version

4.4.7

What step the error happened?

During the sync

Relevant information

wait_to_avoid_fatal_errors does not work as you would expect it to, or at least I don't think it matches the intention of the original PR. The @threshold_period_decorator is added to the read_records function. From my local usage, each individual RECORD has an emitted_at has a gap as defined in STREAM_THRESHOLD_PERIOD.

Meaning in this example

"GET_AFN_INVENTORY_DATA": ReportRateLimits(threshold=2, period_in_minutes=25),

Rather than the report only being requested every 12.5 minutes, each item in the stream is read every 12.5 minutes which means the overall rate limiting here is massive. A Page of ~20 items takes about 4 hours.

Relevant log output

Contribute

• Yes, I want to contribute