Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

structural and content changes to the sql_database doc #1623

Merged
merged 14 commits into from
Sep 12, 2024
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
Prev Previous commit
Next Next commit
Merge branch 'devel' into docs/modify-sql-verified-source-doc
akelad committed Sep 12, 2024
commit affea89dcdb0a8d061ef71ba304d1a925077ca2f
2 changes: 0 additions & 2 deletions .github/weaviate-compose.yml
Original file line number Diff line number Diff line change
@@ -11,8 +11,6 @@ services:
image: semitechnologies/weaviate:1.21.1
ports:
- 8080:8080
volumes:
- weaviate_data
restart: on-failure:0
environment:
QUERY_DEFAULTS_LIMIT: 25
30 changes: 24 additions & 6 deletions .github/workflows/test_common.yml
Original file line number Diff line number Diff line change
@@ -15,6 +15,10 @@ env:
RUNTIME__LOG_LEVEL: ERROR
RUNTIME__DLTHUB_TELEMETRY_ENDPOINT: ${{ secrets.RUNTIME__DLTHUB_TELEMETRY_ENDPOINT }}

# we need the secrets only for the rest_api_pipeline tests which are in tests/sources
# so we inject them only at the end
SOURCES__GITHUB__ACCESS_TOKEN: ${{ secrets.GITHUB_TOKEN }}

jobs:
get_docs_changes:
name: docs changes
@@ -87,11 +91,11 @@ jobs:
run: poetry install --no-interaction --with sentry-sdk

- run: |
poetry run pytest tests/common tests/normalize tests/reflection tests/sources tests/load/test_dummy_client.py tests/extract/test_extract.py tests/extract/test_sources.py tests/pipeline/test_pipeline_state.py
poetry run pytest tests/common tests/normalize tests/reflection tests/load/test_dummy_client.py tests/extract/test_extract.py tests/extract/test_sources.py tests/pipeline/test_pipeline_state.py
if: runner.os != 'Windows'
name: Run common tests with minimum dependencies Linux/MAC
- run: |
poetry run pytest tests/common tests/normalize tests/reflection tests/sources tests/load/test_dummy_client.py tests/extract/test_extract.py tests/extract/test_sources.py tests/pipeline/test_pipeline_state.py -m "not forked"
poetry run pytest tests/common tests/normalize tests/reflection tests/load/test_dummy_client.py tests/extract/test_extract.py tests/extract/test_sources.py tests/pipeline/test_pipeline_state.py -m "not forked"
if: runner.os == 'Windows'
name: Run common tests with minimum dependencies Windows
shell: cmd
@@ -122,15 +126,29 @@ jobs:
name: Run pipeline tests with pyarrow but no pandas installed Windows
shell: cmd
- name: Install pipeline dependencies
run: poetry install --no-interaction -E duckdb -E cli -E parquet --with sentry-sdk --with pipeline -E deltalake
- name: Install pipeline and sources dependencies
run: poetry install --no-interaction -E duckdb -E cli -E parquet -E deltalake -E sql_database --with sentry-sdk,pipeline,sources

- run: |
poetry run pytest tests/extract tests/pipeline tests/libs tests/cli/common tests/destinations tests/sources
if: runner.os != 'Windows'
name: Run extract and pipeline tests Linux/MAC
- run: |
poetry run pytest tests/extract tests/pipeline tests/libs tests/cli/common tests/destinations tests/sources -m "not forked"
if: runner.os == 'Windows'
name: Run extract tests Windows
shell: cmd
# here we upgrade sql alchemy to 2 an run the sql_database tests again
- name: Upgrade sql alchemy
run: poetry run pip install sqlalchemy==2.0.32

- run: |
poetry run pytest tests/extract tests/pipeline tests/libs tests/cli/common tests/destinations
poetry run pytest tests/sources/sql_database
if: runner.os != 'Windows'
name: Run extract and pipeline tests Linux/MAC
- run: |
poetry run pytest tests/extract tests/pipeline tests/libs tests/cli/common tests/destinations -m "not forked"
poetry run pytest tests/sources/sql_database
if: runner.os == 'Windows'
name: Run extract tests Windows
shell: cmd
6 changes: 3 additions & 3 deletions .github/workflows/test_destination_athena.yml
Original file line number Diff line number Diff line change
@@ -22,7 +22,7 @@ env:
RUNTIME__DLTHUB_TELEMETRY_ENDPOINT: ${{ secrets.RUNTIME__DLTHUB_TELEMETRY_ENDPOINT }}
ACTIVE_DESTINATIONS: "[\"athena\"]"
ALL_FILESYSTEM_DRIVERS: "[\"memory\"]"
EXCLUDED_DESTINATION_CONFIGURATIONS: "[\"athena-parquet-staging-iceberg\", \"athena-parquet-no-staging-iceberg\"]"
EXCLUDED_DESTINATION_CONFIGURATIONS: "[\"athena-parquet-iceberg-no-staging-iceberg\", \"athena-parquet-iceberg-staging-iceberg\"]"

jobs:
get_docs_changes:
@@ -73,11 +73,11 @@ jobs:
run: pwd && echo "$DLT_SECRETS_TOML" > tests/.dlt/secrets.toml

- run: |
poetry run pytest tests/load -m "essential"
poetry run pytest tests/load --ignore tests/load/sources -m "essential"
name: Run essential tests Linux
if: ${{ ! (contains(github.event.pull_request.labels.*.name, 'ci full') || !github.event_name == 'schedule')}}
- run: |
poetry run pytest tests/load
poetry run pytest tests/load --ignore tests/load/sources
name: Run all tests Linux
if: ${{ contains(github.event.pull_request.labels.*.name, 'ci full') || github.event_name == 'schedule'}}
4 changes: 2 additions & 2 deletions .github/workflows/test_destination_athena_iceberg.yml
Original file line number Diff line number Diff line change
@@ -73,11 +73,11 @@ jobs:
run: pwd && echo "$DLT_SECRETS_TOML" > tests/.dlt/secrets.toml

- run: |
poetry run pytest tests/load -m "essential"
poetry run pytest tests/load --ignore tests/load/sources -m "essential"
name: Run essential tests Linux
if: ${{ ! (contains(github.event.pull_request.labels.*.name, 'ci full') || github.event_name == 'schedule')}}
- run: |
poetry run pytest tests/load
poetry run pytest tests/load --ignore tests/load/sources
name: Run all tests Linux
if: ${{ contains(github.event.pull_request.labels.*.name, 'ci full') || github.event_name == 'schedule'}}
2 changes: 1 addition & 1 deletion .github/workflows/test_destination_bigquery.yml
Original file line number Diff line number Diff line change
@@ -72,5 +72,5 @@ jobs:
run: pwd && echo "$DLT_SECRETS_TOML" > tests/.dlt/secrets.toml

- run: |
poetry run pytest tests/load
poetry run pytest tests/load --ignore tests/load/sources
name: Run all tests Linux
14 changes: 7 additions & 7 deletions .github/workflows/test_destination_clickhouse.yml
Original file line number Diff line number Diff line change
@@ -68,14 +68,14 @@ jobs:

# OSS ClickHouse
- run: |
docker-compose -f "tests/load/clickhouse/clickhouse-compose.yml" up -d
docker compose -f "tests/load/clickhouse/clickhouse-compose.yml" up -d
echo "Waiting for ClickHouse to be healthy..."
timeout 30s bash -c 'until docker-compose -f "tests/load/clickhouse/clickhouse-compose.yml" ps | grep -q "healthy"; do sleep 1; done'
timeout 30s bash -c 'until docker compose -f "tests/load/clickhouse/clickhouse-compose.yml" ps | grep -q "healthy"; do sleep 1; done'
echo "ClickHouse is up and running"
name: Start ClickHouse OSS
- run: poetry run pytest tests/load -m "essential"
- run: poetry run pytest tests/load --ignore tests/load/sources -m "essential"
name: Run essential tests Linux (ClickHouse OSS)
if: ${{ ! (contains(github.event.pull_request.labels.*.name, 'ci full') || github.event_name == 'schedule')}}
env:
@@ -87,7 +87,7 @@ jobs:
DESTINATION__CLICKHOUSE__CREDENTIALS__HTTP_PORT: 8123
DESTINATION__CLICKHOUSE__CREDENTIALS__SECURE: 0

- run: poetry run pytest tests/load
- run: poetry run pytest tests/load --ignore tests/load/sources
name: Run all tests Linux (ClickHouse OSS)
if: ${{ contains(github.event.pull_request.labels.*.name, 'ci full') || github.event_name == 'schedule'}}
env:
@@ -101,16 +101,16 @@ jobs:

- name: Stop ClickHouse OSS
if: always()
run: docker-compose -f "tests/load/clickhouse/clickhouse-compose.yml" down -v
run: docker compose -f "tests/load/clickhouse/clickhouse-compose.yml" down -v

# ClickHouse Cloud
- run: |
poetry run pytest tests/load -m "essential"
poetry run pytest tests/load --ignore tests/load/sources -m "essential"
name: Run essential tests Linux (ClickHouse Cloud)
if: ${{ ! (contains(github.event.pull_request.labels.*.name, 'ci full') || github.event_name == 'schedule')}}
- run: |
poetry run pytest tests/load
poetry run pytest tests/load --ignore tests/load/sources
name: Run all tests Linux (ClickHouse Cloud)
if: ${{ contains(github.event.pull_request.labels.*.name, 'ci full') || github.event_name == 'schedule'}}
4 changes: 2 additions & 2 deletions .github/workflows/test_destination_databricks.yml
Original file line number Diff line number Diff line change
@@ -70,11 +70,11 @@ jobs:
run: pwd && echo "$DLT_SECRETS_TOML" > tests/.dlt/secrets.toml

- run: |
poetry run pytest tests/load -m "essential"
poetry run pytest tests/load --ignore tests/load/sources -m "essential"
name: Run essential tests Linux
if: ${{ ! (contains(github.event.pull_request.labels.*.name, 'ci full') || github.event_name == 'schedule')}}
- run: |
poetry run pytest tests/load
poetry run pytest tests/load --ignore tests/load/sources
name: Run all tests Linux
if: ${{ contains(github.event.pull_request.labels.*.name, 'ci full') || github.event_name == 'schedule'}}
8 changes: 4 additions & 4 deletions .github/workflows/test_destination_dremio.yml
Original file line number Diff line number Diff line change
@@ -43,7 +43,7 @@ jobs:
uses: actions/checkout@master

- name: Start dremio
run: docker-compose -f "tests/load/dremio/docker-compose.yml" up -d
run: docker compose -f "tests/load/dremio/docker-compose.yml" up -d

- name: Setup Python
uses: actions/setup-python@v4
@@ -68,7 +68,7 @@ jobs:
run: poetry install --no-interaction -E s3 -E gs -E az -E parquet --with sentry-sdk --with pipeline

- run: |
poetry run pytest tests/load
poetry run pytest tests/load --ignore tests/load/sources
if: runner.os != 'Windows'
name: Run tests Linux/MAC
env:
@@ -80,11 +80,11 @@ jobs:
DESTINATION__MINIO__CREDENTIALS__ENDPOINT_URL: http://127.0.0.1:9010
- run: |
poetry run pytest tests/load
poetry run pytest tests/load --ignore tests/load/sources
if: runner.os == 'Windows'
name: Run tests Windows
shell: cmd
- name: Stop dremio
if: always()
run: docker-compose -f "tests/load/dremio/docker-compose.yml" down -v
run: docker compose -f "tests/load/dremio/docker-compose.yml" down -v
4 changes: 2 additions & 2 deletions .github/workflows/test_destination_lancedb.yml
Original file line number Diff line number Diff line change
@@ -71,11 +71,11 @@ jobs:
run: poetry run pip install openai

- run: |
poetry run pytest tests/load -m "essential"
poetry run pytest tests/load --ignore tests/load/sources -m "essential"
name: Run essential tests Linux
if: ${{ ! (contains(github.event.pull_request.labels.*.name, 'ci full') || github.event_name == 'schedule')}}
- run: |
poetry run pytest tests/load
poetry run pytest tests/load --ignore tests/load/sources
name: Run all tests Linux
if: ${{ contains(github.event.pull_request.labels.*.name, 'ci full') || github.event_name == 'schedule'}}
80 changes: 80 additions & 0 deletions .github/workflows/test_destination_motherduck.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@

name: dest | motherduck

on:
pull_request:
branches:
- master
- devel
workflow_dispatch:
schedule:
- cron: '0 2 * * *'

concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true

env:
DLT_SECRETS_TOML: ${{ secrets.DLT_SECRETS_TOML }}

RUNTIME__SENTRY_DSN: https://[email protected]/4504819859914752
RUNTIME__LOG_LEVEL: ERROR
RUNTIME__DLTHUB_TELEMETRY_ENDPOINT: ${{ secrets.RUNTIME__DLTHUB_TELEMETRY_ENDPOINT }}

ACTIVE_DESTINATIONS: "[\"motherduck\"]"
ALL_FILESYSTEM_DRIVERS: "[\"memory\"]"

jobs:
get_docs_changes:
name: docs changes
uses: ./.github/workflows/get_docs_changes.yml
if: ${{ !github.event.pull_request.head.repo.fork || contains(github.event.pull_request.labels.*.name, 'ci from fork')}}

run_loader:
name: dest | motherduck tests
needs: get_docs_changes
if: needs.get_docs_changes.outputs.changes_outside_docs == 'true'
defaults:
run:
shell: bash
runs-on: "ubuntu-latest"

steps:

- name: Check out
uses: actions/checkout@master

- name: Setup Python
uses: actions/setup-python@v4
with:
python-version: "3.10.x"

- name: Install Poetry
uses: snok/[email protected]
with:
virtualenvs-create: true
virtualenvs-in-project: true
installer-parallel: true

- name: Load cached venv
id: cached-poetry-dependencies
uses: actions/cache@v3
with:
path: .venv
key: venv-${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('**/poetry.lock') }}-motherduck

- name: Install dependencies
run: poetry install --no-interaction -E motherduck -E s3 -E gs -E az -E parquet --with sentry-sdk --with pipeline

- name: create secrets.toml
run: pwd && echo "$DLT_SECRETS_TOML" > tests/.dlt/secrets.toml

- run: |
poetry run pytest tests/load --ignore tests/load/sources -m "essential"
name: Run essential tests Linux
if: ${{ ! (contains(github.event.pull_request.labels.*.name, 'ci full') || github.event_name == 'schedule')}}
- run: |
poetry run pytest tests/load --ignore tests/load/sources
name: Run all tests Linux
if: ${{ contains(github.event.pull_request.labels.*.name, 'ci full') || github.event_name == 'schedule'}}
2 changes: 1 addition & 1 deletion .github/workflows/test_destination_mssql.yml
Original file line number Diff line number Diff line change
@@ -75,5 +75,5 @@ jobs:
run: pwd && echo "$DLT_SECRETS_TOML" > tests/.dlt/secrets.toml

# always run full suite, also on branches
- run: poetry run pytest tests/load
- run: poetry run pytest tests/load --ignore tests/load/sources
name: Run tests Linux
4 changes: 2 additions & 2 deletions .github/workflows/test_destination_qdrant.yml
Original file line number Diff line number Diff line change
@@ -69,11 +69,11 @@ jobs:
run: poetry install --no-interaction -E qdrant -E parquet --with sentry-sdk --with pipeline

- run: |
poetry run pytest tests/load -m "essential"
poetry run pytest tests/load --ignore tests/load/sources -m "essential"
name: Run essential tests Linux
if: ${{ ! (contains(github.event.pull_request.labels.*.name, 'ci full') || github.event_name == 'schedule')}}
- run: |
poetry run pytest tests/load
poetry run pytest tests/load --ignore tests/load/sources
name: Run all tests Linux
if: ${{ contains(github.event.pull_request.labels.*.name, 'ci full') || github.event_name == 'schedule'}}
4 changes: 2 additions & 2 deletions .github/workflows/test_destination_snowflake.yml
Original file line number Diff line number Diff line change
@@ -70,11 +70,11 @@ jobs:
run: pwd && echo "$DLT_SECRETS_TOML" > tests/.dlt/secrets.toml

- run: |
poetry run pytest tests/load -m "essential"
poetry run pytest tests/load --ignore tests/load/sources -m "essential"
name: Run essential tests Linux
if: ${{ ! (contains(github.event.pull_request.labels.*.name, 'ci full') || github.event_name == 'schedule')}}
- run: |
poetry run pytest tests/load
poetry run pytest tests/load --ignore tests/load/sources
name: Run all tests Linux
if: ${{ contains(github.event.pull_request.labels.*.name, 'ci full') || github.event_name == 'schedule'}}
4 changes: 2 additions & 2 deletions .github/workflows/test_destination_synapse.yml
Original file line number Diff line number Diff line change
@@ -73,11 +73,11 @@ jobs:
run: pwd && echo "$DLT_SECRETS_TOML" > tests/.dlt/secrets.toml

- run: |
poetry run pytest tests/load -m "essential"
poetry run pytest tests/load --ignore tests/load/sources -m "essential"
name: Run essential tests Linux
if: ${{ ! (contains(github.event.pull_request.labels.*.name, 'ci full') || github.event_name == 'schedule')}}
- run: |
poetry run pytest tests/load
poetry run pytest tests/load --ignore tests/load/sources
name: Run all tests Linux
if: ${{ contains(github.event.pull_request.labels.*.name, 'ci full') || github.event_name == 'schedule'}}
7 changes: 4 additions & 3 deletions .github/workflows/test_destinations.yml
Original file line number Diff line number Diff line change
@@ -28,7 +28,8 @@ env:
RUNTIME__DLTHUB_TELEMETRY_ENDPOINT: ${{ secrets.RUNTIME__DLTHUB_TELEMETRY_ENDPOINT }}
# Test redshift and filesystem with all buckets
# postgres runs again here so we can test on mac/windows
ACTIVE_DESTINATIONS: "[\"redshift\", \"postgres\", \"duckdb\", \"filesystem\", \"dummy\", \"motherduck\"]"
ACTIVE_DESTINATIONS: "[\"redshift\", \"postgres\", \"duckdb\", \"filesystem\", \"dummy\"]"
# note that all buckets are enabled for testing

jobs:
get_docs_changes:
@@ -81,11 +82,11 @@ jobs:
run: pwd && echo "$DLT_SECRETS_TOML" > tests/.dlt/secrets.toml

- run: |
poetry run pytest tests/load -m "essential"
poetry run pytest tests/load --ignore tests/load/sources -m "essential"
name: Run essential tests Linux
if: ${{ ! (contains(github.event.pull_request.labels.*.name, 'ci full') || github.event_name == 'schedule')}}
- run: |
poetry run pytest tests/load
poetry run pytest tests/load --ignore tests/load/sources
name: Run all tests Linux
if: ${{ contains(github.event.pull_request.labels.*.name, 'ci full') || github.event_name == 'schedule'}}
2 changes: 1 addition & 1 deletion .github/workflows/test_doc_snippets.yml
Original file line number Diff line number Diff line change
@@ -60,7 +60,7 @@ jobs:
uses: actions/checkout@master

- name: Start weaviate
run: docker-compose -f ".github/weaviate-compose.yml" up -d
run: docker compose -f ".github/weaviate-compose.yml" up -d

- name: Setup Python
uses: actions/setup-python@v4
Loading
You are viewing a condensed version of this merge commit. You can view the full changes here.