[source-OpenExchangeRate] Airbyte connector failed with status code '400' and error message 'invalid_date'

[mouradbka]

Connector Name

source-open-exchange-rates

Connector Version

0.3.3

What step the error happened?

During the sync

Relevant information

Environment Details:

Source Connector: source-open-exchange-rates (Version: 0.3.3) (Free plan)
Destination Connector: destination-snowflake (Version: 3.15.2)
Airbyte Version: 1.2.0
Deployment Mode: OSS
Destination: Snowflake (AIRBYTE_DB, AIRBYTE_SCHEMA, AIRBYTE_WH)

Observed Behavior:

Sync attempt for the source open_exchange_rates fails with a Bad Request (400) error.
API endpoint:

https://openexchangerates.org/api/historical/{{ format_datetime(config['start_date'] if not stream_state else stream_state['timestamp'], '%Y-%m-%d') }}.json?app_id=****&base=USD

Error: invalid_date response from the open_exchange_rates API.

Error Stack Trace: The following traceback occurred during the sync:

airbyte_cdk.sources.streams.http.http_client.MessageRepresentationAirbyteTracedErrors: Bad request. Please check your request parameters.
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/airbyte_cdk/sources/abstract_source.py", line 133, in read
...

Metrics and Configuration Notes:

MetricClient shows a message: MetricClient has not been initialized. Must call MetricClientFactory.CreateMetricClient before using MetricClient.
Snowflake query logs indicate the tables and stages are being created successfully, but the sync fails due to source-related errors.

Relevant log output

Retry State and Backoff:

2024-11-27 03:53:56 platform > Retry State: ... successiveCompleteFailures=4, totalCompleteFailures=4
2024-11-27 03:53:56 platform > Backing off for: 4 minutes 30 seconds.

Workload Creation and Execution:

2024-11-27 03:58:26 platform > Workload ID: *******-*******_8_4_check launched and pipeline completed.

Connector Check Success:

2024-11-27 03:58:34 platform > Check succeeded for workload *******_8_4_check.

Sync Start:

2024-11-27 03:58:46 platform > Creating workload *******_8_4_sync.
2024-11-27 03:58:51 platform > Workload launched for synchronization.

Error During Source Sync:

2024-11-27 03:58:59 source > Error: Bad request. Please check your request parameters.
Stream: open_exchange_rates; Status Code: 400; Error: 'invalid_date'.

Replication Orchestration Failure:

2024-11-27 03:58:59 replication-orchestrator > Source process exited with non-zero exit code 1.
2024-11-27 03:59:06 replication-orchestrator > Sync summary: {
  "status": "failed",
  "bytesEmitted": 0,
  "recordsEmitted": 0,
  "startTime": 1732679936060,
  "endTime": 1732679946649
}

Destination Errors:

2024-11-27 03:59:00 destination > SQL Error: Table 'AIRBYTE_DB.OER_SCHEMA._AIRBYTE_RAW_OPEN_EXCHANGE_RATES' does not exist or not authorized.

Final Error Messages:

2024-11-27 03:59:06 replication-orchestrator > failures: [
  {
    "failureOrigin": "source",
    "failureType": "system_error",
    "internalMessage": "GET request failed with status code 400",
    "externalMessage": "Bad request. Check request parameters."
  },
  {
    "failureOrigin": "source",
    "failureType": "config_error",
    "externalMessage": "During sync, streams did not sync successfully due to invalid_date."
  }
]

Contribute

• Yes, I want to contribute

[btkcodedev]

Hi @mouradbka,
I've looked at this and verified the connector is working as intended.
Have you configured the date correctly
Could you please try again and get back.
CC: @marcosmarxm

[btkcodedev]

Have you reset connection and tried using this date format?

[btkcodedev]

However there is some redundancy in the manifest, fixed 'em all and tested the stream and hashed
PR: #48752

[mouradbka]

@btkcodedev Hi and thanks for your help !

The date format is as indicated, but the version of the connector is 0.3.3. Is that normal?
I have updated it but no improvement.

That's the error message I get:

'GET' request to 'https://openexchangerates.org/api/historical/%7B%7B%20format_datetime(%20config%5B'start_date'%5D%20if%20not%20stream_state%20else%20stream_state%5B'timestamp'%5D,%20'%25Y-%25m-%25d'%20)%20%7D%7D.json?app_id=****&base=USD' failed with status code '400' and error message 'invalid_date'

[btkcodedev]

Oh! the format datetime function is messed up within url, could you try the cloud builder using the manifest updated from this PR. #48752

Exact file: https://github.com/airbytehq/airbyte/blob/c4d32c19a6f37cfbaed10da2deff2b5fd5a995d7/airbyte-integrations/connectors/source-open-exchange-rates/manifest.yaml