Skip to content

Commit

Permalink
removes content that is not a tutorial, how to, reference or dlt rela…
Browse files Browse the repository at this point in the history
…te knowledge from docs
  • Loading branch information
rudolfix committed Sep 15, 2023
1 parent b7d6b43 commit 24d1f15
Show file tree
Hide file tree
Showing 22 changed files with 99 additions and 1,003 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -108,5 +108,5 @@ To try out schema evolution with `dlt`, check out our [colab demo.](https://cola
### Want more?

- Join our [Slack](https://join.slack.com/t/dlthub-community/shared_invite/zt-1slox199h-HAE7EQoXmstkP_bTqal65g)
- Read our [docs on implementing schema evolution](https://dlthub.com/docs/reference/explainers/schema-evolution)
- Read our [schema evolution blog post](https://dlthub.com/docs/blog/schema-evolution)
- Stay tuned for the next article in the series: *How to do schema evolution with* `dlt` *in the most effective way*
Original file line number Diff line number Diff line change
@@ -1,7 +1,12 @@
---
title: Schema evolution
description: Schema evolution with dlt
keywords: [schema evolution, schema versioning, data contracts]
slug: schema-evolution
title: "Schema Evolution"
authors:
name: Adrian Brudaru
title: Schema Evolution
url: https://github.com/adrianbr
image_url: https://avatars.githubusercontent.com/u/5762770?v=4
tags: [data engineer shortage, structured data, schema evolution]
---

# Schema evolution
Expand Down Expand Up @@ -131,10 +136,10 @@ business-logic tests, you would still need to implement them in a custom way.
## The implementation recipe

1. Use `dlt`. It will automatically infer and version schemas, so you can simply check if there are
changes. You can just use the [normaliser + loader](../../general-usage/pipeline.md) or
[build extraction with dlt](../../general-usage/resource.md). If you want to define additional
constraints, you can do so in the [schema](../../general-usage/schema.md).
1. [Define your slack hook](../../running-in-production/running.md#using-slack-to-send-messages) or
changes. You can just use the [normaliser + loader](https://dlthub.com/docs/general-usage/pipeline.md) or
[build extraction with dlt](https://dlthub.com/docs/general-usage/resource.md). If you want to define additional
constraints, you can do so in the [schema](https://dlthub.com/docs/general-usage/schema.md).
1. [Define your slack hook](https://dlthub.com/docs/running-in-production/running.md#using-slack-to-send-messages) or
create your own notification function. Make sure the slack channel contains the data producer and
any stakeholders.
1. [Capture the load job info and send it to the hook](../../running-in-production/running#inspect-save-and-alert-on-schema-changes).
1. [Capture the load job info and send it to the hook](https://dlthub.com/docs/running-in-production/running#inspect-save-and-alert-on-schema-changes).
2 changes: 0 additions & 2 deletions docs/website/docs/build-a-pipeline-tutorial.md
Original file line number Diff line number Diff line change
Expand Up @@ -413,8 +413,6 @@ These governance features in `dlt` pipelines contribute to better data managemen
compliance adherence, and overall data governance, promoting data consistency, traceability, and
control throughout the data processing lifecycle.

Read more about [schema evolution.](reference/explainers/schema-evolution.md)

### Scaling and finetuning

`dlt` offers several mechanism and configuration options to scale up and finetune pipelines:
Expand Down
17 changes: 17 additions & 0 deletions docs/website/docs/dlt-ecosystem/deployments/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
---
title: Deployments
description: dlt can run on almost any python environment and hardware
keywords: [dlt, running, environment]
---
import DocCardList from '@theme/DocCardList';

# Deployments
`dlt` runs wherever Python runs. Below you see walkthroughs for Airflow, GCP Cloud Functions, AWS Lambda and GitHub Actions but our users run us on
* in any notebook environment including Colab
* any orchestrator including Kestra, Prefect or Dagster
* local laptops with `duckdb` or `weaviate` as destinations
* Github codespaces and other devcontainers
* regular VMs from all major providers: AWS, GCP or Azure
* in containers via Docker, docker-compose and Kubernetes

<DocCardList />

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

56 changes: 0 additions & 56 deletions docs/website/docs/dlt-ecosystem/deployments/where-can-dlt-run.md

This file was deleted.

23 changes: 0 additions & 23 deletions docs/website/docs/general-usage/import-dlt.md

This file was deleted.

Loading

0 comments on commit 24d1f15

Please sign in to comment.