[source-greenhouse] Application Stream Incremental on wrong cursor and missing custom fields

[jimmc2]

Connector Name

source-greenhouse

Connector Version

0.5.1

What step the error happened?

Other

Relevant information

The "Application" stream for Greenhouse is set to be incremental based on the cursor field "created_after" and table field "applied_at" based on the manifest here. This is an issue because any updates to an application are not caught by this connector set to "incremental" mode because the connector only fetches new applications that are since the last polled "applied_at" date. This requires anyone wanting updated data that would include all new applications and all updated applications to use a full refresh sync which greatly increases the number of requests sent to Greenhouse.

Additionally, the Application stream does not include the custom_fields and keyed_custom_fields included in the api response despite them being available in the raw data stored by Airbyte. I will submit a fix for both of these.

Relevant log output

No response

Contribute

• Yes, I want to contribute

[jimmc2]

Will base my PR on the changes in this PR where custom_fields were added to the Candidates stream

[marcosmarxm]

Thanks @jimmc2 I'll review during the week your contribution!

[V-Legendre]

Hey there 👋 +1 on this, we would like the cursor field to be "last_activity_at" instead of "applied_at" .
Since greenhouse api offers a way to filter on activity, it should be easy to make this change.