-
Notifications
You must be signed in to change notification settings - Fork 0
/
infra_etl.qmd
21 lines (11 loc) · 958 Bytes
/
infra_etl.qmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# ELT
Consider the following examples to establish a data pipeline.
1. A zip file lands in data lake (`s3`/`minio`) daily.
2. Execute scripts in the server (`ec2`) to download/unzip/select/upload files based on `mtime`. It produces a file (`csv`) to track work done at the agreed cut-off time (`cron`). AWS `lambda` is another option.
3. `snowflake` external stage (`s3`) is triggered by a file (`txt`) to kicks off `snowpipe` and ingest data to DB as `variant`. Similar storage are `databrick`, `dremio`, `clickhouse`. The preferred formats are `parquet`, `iceberg`, `ADBC`.
4. Move data between platforms via `airbyte`.
5. Validate and transform DB raw to DB mart through `dbt`.
6. Automatize the process by a task scheduler `prefect`, `airflow`, `dagster`.
7. Create a dashboard for DB mart via `metabase`, `superset`.
![DuckDB cloud](pic/duckdb_01.png)
![DuckDB terminal](pic/duckdb_02.png)