Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix connector harness performance ci run #31304

Merged
merged 28 commits into from
Oct 13, 2023
Merged

Fix connector harness performance ci run #31304

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
7e185e1
empty line to create a PR
xiaohansong Oct 11, 2023
9023148
Automated Commit - Formatting Changes
xiaohansong Oct 11, 2023
f167679
attempt a fix
xiaohansong Oct 11, 2023
dfc9c67
typo
xiaohansong Oct 11, 2023
8542cac
add logs
xiaohansong Oct 11, 2023
21cabaa
try adding path directly
xiaohansong Oct 11, 2023
86b6ca9
whoami?
xiaohansong Oct 11, 2023
406da8b
export path in write part
xiaohansong Oct 11, 2023
9dd5c7c
add secrets
xiaohansong Oct 12, 2023
3d98ddd
try different way to create cred
xiaohansong Oct 12, 2023
0fb5fef
Merge remote-tracking branch 'origin/master' into xiaohan/harness-fix
xiaohansong Oct 12, 2023
ba4f4f3
remove unnecessary printout
xiaohansong Oct 12, 2023
6a97539
make comment optional
xiaohansong Oct 12, 2023
4d42500
Merge branch 'master' into xiaohan/harness-fix
xiaohansong Oct 12, 2023
0adb090
update postgres catalog
xiaohansong Oct 12, 2023
c4a2bd3
Automated Commit - Formatting Changes
xiaohansong Oct 12, 2023
1ed2779
Merge branch 'master' into xiaohan/harness-fix
xiaohansong Oct 12, 2023
86f5f05
add more logs, suspecting it's not reading from the latest catalog
xiaohansong Oct 12, 2023
bd758f3
more logs
xiaohansong Oct 12, 2023
637a9df
remove gitref
xiaohansong Oct 12, 2023
2ab9f44
need to check local branch and master branch to do git diff
xiaohansong Oct 12, 2023
59eb0be
remove origin
xiaohansong Oct 12, 2023
a08ad54
need to fetch all in perf-test step
xiaohansong Oct 12, 2023
daa81f2
add back ref
xiaohansong Oct 12, 2023
4d07a76
remove logs
xiaohansong Oct 12, 2023
ceade6d
Merge branch 'master' into xiaohan/harness-fix
xiaohansong Oct 13, 2023
beee3ee
formatting
xiaohansong Oct 13, 2023
4b1a73d
Merge branch 'master' into xiaohan/harness-fix
xiaohansong Oct 13, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .github/workflows/connector-performance-command.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,13 @@ on:
inputs:
connector:
description: "Airbyte Connector"
type: choice
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!
Can this replace the code down the file that filters for supported connectors - the big ugly if

Copy link
Contributor Author

@xiaohansong xiaohansong Oct 13, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we still need that because we still want to trigger it from comment command

required: true
options:
- connectors/source-postgres
- connectors/source-mysql
- connectors/destination-snowflake
default: "connectors/source-postgres"
repo:
description: "Repo to check out code from. Defaults to the main airbyte repo. Set this when building connectors from forked repos."
required: false
Expand Down Expand Up @@ -102,6 +108,7 @@ jobs:
with:
repository: ${{ github.event.inputs.repo }}
ref: ${{ github.event.inputs.gitref }}
fetch-depth: 0 # This is to fetch the main branch in case we are running on a different branch.
- name: Install Java
uses: actions/setup-java@v3
with:
Expand All @@ -124,6 +131,7 @@ jobs:
echo "harness_type=$the_harness" >> "$GITHUB_OUTPUT"
- name: Write harness credentials
run: |
export PATH="$PATH:/root/.local/bin"
ci_credentials connectors-performance/$HARNESS_TYPE write-to-storage
env:
GCP_GSM_CREDENTIALS: ${{ secrets.GCP_GSM_CREDENTIALS }}
Expand All @@ -144,6 +152,8 @@ jobs:
connector_name=$(echo ${{ github.event.inputs.connector }} | cut -d / -f 2)
echo "Running ./gradlew :airbyte-integrations:connectors:$connector_name:build -x check"
./gradlew :airbyte-integrations:connectors:$connector_name:build -x check
env:
GCP_GSM_CREDENTIALS: ${{ secrets.GCP_GSM_CREDENTIALS }}
- name: KIND Kubernetes Cluster Setup
uses: helm/[email protected]
with:
Expand Down Expand Up @@ -180,6 +190,7 @@ jobs:
kubectl logs --tail=1 $POD | while read line ; do line=${line#"$PREFIX"}; line=${line%"$SUFFIX"}; echo $line >> $GITHUB_OUTPUT ; done
echo "$EOF" >> $GITHUB_OUTPUT
- name: Link comment to workflow run
if: github.event.inputs.comment-id
uses: peter-evans/create-or-update-comment@v2
with:
reactions: "+1"
Expand Down
91 changes: 4 additions & 87 deletions ...s-performance/source-harness/src/main/resources/catalogs/source-postgres/10m_catalog.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,54 +1,9 @@
{
"streams": [
{
"stream": {
"name": "purchases",
"namespace": "10m_users",
"json_schema": {
"type": "object",
"properties": {
"id": {
"type": "number",
"airbyte_type": "integer"
},
"user_id": {
"type": "number",
"airbyte_type": "integer"
},
"product_id": {
"type": "number",
"airbyte_type": "integer"
},
"returned_at": {
"type": "string",
"format": "date-time",
"airbyte_type": "timestamp_with_timezone"
},
"purchased_at": {
"type": "string",
"format": "date-time",
"airbyte_type": "timestamp_with_timezone"
},
"added_to_cart_at": {
"type": "string",
"format": "date-time",
"airbyte_type": "timestamp_with_timezone"
}
}
},
"default_cursor_field": [],
"supported_sync_modes": ["full_refresh", "incremental"],
"source_defined_primary_key": [["id"]]
},
"sync_mode": "full_refresh",
"primary_key": [["id"]],
"cursor_field": ["id"],
"destination_sync_mode": "append"
},
{
"stream": {
"name": "users",
"namespace": "10m_users",
"namespace": "public",
"json_schema": {
"type": "object",
"properties": {
Expand Down Expand Up @@ -111,51 +66,13 @@
},
"default_cursor_field": [],
"supported_sync_modes": ["full_refresh", "incremental"],
"source_defined_cursor": true,
"source_defined_primary_key": [["id"]]
},
"sync_mode": "full_refresh",
"primary_key": [["id"]],
"cursor_field": ["updated_at"],
"destination_sync_mode": "append"
},
{
"stream": {
"name": "products",
"namespace": "10m_users",
"json_schema": {
"type": "object",
"properties": {
"id": {
"type": "number",
"airbyte_type": "integer"
},
"make": {
"type": "string"
},
"year": {
"type": "string"
},
"model": {
"type": "string"
},
"price": {
"type": "number"
},
"created_at": {
"type": "string",
"format": "date-time",
"airbyte_type": "timestamp_with_timezone"
}
}
},
"default_cursor_field": [],
"supported_sync_modes": ["full_refresh", "incremental"],
"source_defined_primary_key": [["id"]]
},
"sync_mode": "full_refresh",
"primary_key": [["id"]],
"cursor_field": ["created_at"],
"destination_sync_mode": "append"
"cursor_field": [],
"destination_sync_mode": "overwrite"
}
]
}
91 changes: 4 additions & 87 deletions ...rs-performance/source-harness/src/main/resources/catalogs/source-postgres/1m_catalog.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,54 +1,9 @@
{
"streams": [
{
"stream": {
"name": "purchases",
"namespace": "1m_users",
"json_schema": {
"type": "object",
"properties": {
"id": {
"type": "number",
"airbyte_type": "integer"
},
"user_id": {
"type": "number",
"airbyte_type": "integer"
},
"product_id": {
"type": "number",
"airbyte_type": "integer"
},
"returned_at": {
"type": "string",
"format": "date-time",
"airbyte_type": "timestamp_with_timezone"
},
"purchased_at": {
"type": "string",
"format": "date-time",
"airbyte_type": "timestamp_with_timezone"
},
"added_to_cart_at": {
"type": "string",
"format": "date-time",
"airbyte_type": "timestamp_with_timezone"
}
}
},
"default_cursor_field": [],
"supported_sync_modes": ["full_refresh", "incremental"],
"source_defined_primary_key": [["id"]]
},
"sync_mode": "full_refresh",
"primary_key": [["id"]],
"cursor_field": ["id"],
"destination_sync_mode": "append"
},
{
"stream": {
"name": "users",
"namespace": "1m_users",
"namespace": "public",
"json_schema": {
"type": "object",
"properties": {
Expand Down Expand Up @@ -111,51 +66,13 @@
},
"default_cursor_field": [],
"supported_sync_modes": ["full_refresh", "incremental"],
"source_defined_cursor": true,
"source_defined_primary_key": [["id"]]
},
"sync_mode": "full_refresh",
"primary_key": [["id"]],
"cursor_field": ["updated_at"],
"destination_sync_mode": "append"
},
{
"stream": {
"name": "products",
"namespace": "1m_users",
"json_schema": {
"type": "object",
"properties": {
"id": {
"type": "number",
"airbyte_type": "integer"
},
"make": {
"type": "string"
},
"year": {
"type": "string"
},
"model": {
"type": "string"
},
"price": {
"type": "number"
},
"created_at": {
"type": "string",
"format": "date-time",
"airbyte_type": "timestamp_with_timezone"
}
}
},
"default_cursor_field": [],
"supported_sync_modes": ["full_refresh", "incremental"],
"source_defined_primary_key": [["id"]]
},
"sync_mode": "full_refresh",
"primary_key": [["id"]],
"cursor_field": ["created_at"],
"destination_sync_mode": "append"
"cursor_field": [],
"destination_sync_mode": "overwrite"
}
]
}
91 changes: 4 additions & 87 deletions ...s-performance/source-harness/src/main/resources/catalogs/source-postgres/20m_catalog.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,54 +1,9 @@
{
"streams": [
{
"stream": {
"name": "purchases",
"namespace": "20m_users",
"json_schema": {
"type": "object",
"properties": {
"id": {
"type": "number",
"airbyte_type": "integer"
},
"user_id": {
"type": "number",
"airbyte_type": "integer"
},
"product_id": {
"type": "number",
"airbyte_type": "integer"
},
"returned_at": {
"type": "string",
"format": "date-time",
"airbyte_type": "timestamp_with_timezone"
},
"purchased_at": {
"type": "string",
"format": "date-time",
"airbyte_type": "timestamp_with_timezone"
},
"added_to_cart_at": {
"type": "string",
"format": "date-time",
"airbyte_type": "timestamp_with_timezone"
}
}
},
"default_cursor_field": [],
"supported_sync_modes": ["full_refresh", "incremental"],
"source_defined_primary_key": [["id"]]
},
"sync_mode": "full_refresh",
"primary_key": [["id"]],
"cursor_field": ["id"],
"destination_sync_mode": "append"
},
{
"stream": {
"name": "users",
"namespace": "20m_users",
"namespace": "public",
"json_schema": {
"type": "object",
"properties": {
Expand Down Expand Up @@ -111,51 +66,13 @@
},
"default_cursor_field": [],
"supported_sync_modes": ["full_refresh", "incremental"],
"source_defined_cursor": true,
"source_defined_primary_key": [["id"]]
},
"sync_mode": "full_refresh",
"primary_key": [["id"]],
"cursor_field": ["updated_at"],
"destination_sync_mode": "append"
},
{
"stream": {
"name": "products",
"namespace": "20m_users",
"json_schema": {
"type": "object",
"properties": {
"id": {
"type": "number",
"airbyte_type": "integer"
},
"make": {
"type": "string"
},
"year": {
"type": "string"
},
"model": {
"type": "string"
},
"price": {
"type": "number"
},
"created_at": {
"type": "string",
"format": "date-time",
"airbyte_type": "timestamp_with_timezone"
}
}
},
"default_cursor_field": [],
"supported_sync_modes": ["full_refresh", "incremental"],
"source_defined_primary_key": [["id"]]
},
"sync_mode": "full_refresh",
"primary_key": [["id"]],
"cursor_field": ["created_at"],
"destination_sync_mode": "append"
"cursor_field": [],
"destination_sync_mode": "overwrite"
}
]
}
Loading