examples for docs #616

sh-rp · 2023-09-10T15:49:55Z

Description

This preview provides a scaffold for our examples. Notes:

This is based on the remove snip sync PR: remove snipsync (#613) #615
The full code example of every example is in the /code directory of example in the website/docs folder.
update_snippets is updated to sync the examples from the website into /docs/examples. It will copy over every file and take the "example" snippet from each file.
There is a unique component that all examples share, which is the header with some introductory info and stuff that is shared between all examples. This is an approach we can take in other parts of the docs too to reduce duplication.
The code example itself is just for illustration purposes for now.

* add custom snippets element

netlify · 2023-09-10T15:49:59Z

✅ Deploy Preview for dlt-hub-docs ready!

Name	Link
🔨 Latest commit	`0fcada4`
🔍 Latest deploy log	https://app.netlify.com/sites/dlt-hub-docs/deploys/651d280853ce9d0007066aea
😎 Deploy Preview	https://deploy-preview-616--dlt-hub-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

rudolfix

on top of the comments:
should we have docosaurus directly in /docs? o docs/docs? not in docs/website/docs?
we can just delete archives or skip it from being crawled by docosaursu

rudolfix · 2023-09-13T16:57:23Z

Makefile

@@ -52,7 +52,7 @@ lint:
 	# $(MAKE) lint-security

 test-and-lint-snippets:
-	poetry run mypy --config-file mypy.ini docs/website docs/examples
+	poetry run mypy --config-file mypy.ini --namespace-packages --explicit-package-bases docs/website docs/examples 


there's a script that creates missing init files, check make lint. or you do not want that to happen?

I have removed this again

docs/website/docs/examples/_examples-header.md

rudolfix · 2023-09-13T17:01:12Z

docs/website/docs/examples/_examples-header.md

+<div>{props.intro}</div>
+
+## Setup: Running this example on your machine
+<CodeBlock language="sh">


any reason to use this and not

```sh

Yes, the variable will not work in the markdown block unfortunately..

docs/website/docs/examples/incremental_loading/code/run-snippets.py

sh-rp · 2023-09-14T08:26:04Z

on top of the comments:
should we have docosaurus directly in /docs? o docs/docs? not in docs/website/docs?
we can just delete archives or skip it from being crawled by docosaursu

Yes, I think that would be nice. But we should do this when there are no open documentation branches.

rudolfix

Looks like we are good with it so let's write two examples and merge it. I'll take a look at pokemon and maybe add google sheets example?

docs/website/docs/examples/incremental_loading/code/run-snippets.py

rudolfix · 2023-09-15T14:58:39Z

dlt/common/schema/schema.py

@@ -392,7 +392,7 @@ def _coerce_non_null_value(self, table_columns: TTableSchemaColumns, table_name:
                raise CannotCoerceColumnException(table_name, col_name, py_type, table_columns[col_name]["data_type"], v)
            # otherwise we must create variant extension to the table
            # pass final=True so no more auto-variants can be created recursively
-            # TODO: generate callback so DLT user can decide what to do
+            # TODO: generate callback sodltuser can decide what to do


somehow surrounding spaces got eaten. and that happened in many places. I hope you can revert the commit and do that again?

rudolfix

instead of adding zendesk credentials, use the same method we have in verified sources (via google secrets)
also because we rely on credentials to run tests we prevent external contributors from using our CI. could we have a pytest skip that will skip the test if it detects we run on github CI and from the fork (easy - all in env variables)
python file names are always "run.py". those module names are used to create sections for config and secrets. let's use a meaningful name ie. zendesk, pokemon
title the example so it is clear what it does and what is the main highlight:

"Get Pokemon details in parallel using transformers"
"Load zendesk tickets incrementally"

Structure is OK, what I'd change
TLDR -> Summary - One sentence what example does
Highlights
3-5 bullet points with the most interesting stuff or building blocks used. ideal bullet point

one sentence explains what we do in the example + links the docs page (if applicable) + shows relevant code snippet (if the example is long)

IMO we do not need to include the whole snippet (maybe if it is short)

example:

we configure incremental loading for the resource that takes start and end date passed from the source (link to incremental docs) and allows it to use Airflow schedule (link to the docs)

code snippet

docs/website/docs/examples/_examples-header.md

+# go to example directory
+cd ./dlt/docs/examples/${props.slug}
+# install dlt
+pip3 install dlt


rudolfix · 2023-09-28T09:49:17Z

docs/website/docs/examples/incremental_loading/code/.dlt/config.toml

@@ -0,0 +1,3 @@
+# @@@DLT_SNIPPET_START example
+some_key="some_value"


why do we need this?

rudolfix · 2023-09-28T09:53:45Z

docs/website/docs/examples/incremental_loading/code/run-snippets.py

@@ -0,0 +1,130 @@
+
+
+def incremental_snippet() -> None:


@sh-rp running this as a test requires access to credentials. this will prevent people from contributing to the docs from forks.

we need to be able to skip this test if it is run from a fork on github-ci. alternatively we can disable this test and just lint it.

verified sources detect forks so you can start by looking and github workflows there

docs/examples/transformers/.dlt/config.toml

docs/website/docs/examples/incremental_loading/index.md

rudolfix · 2023-09-28T13:18:16Z

docs/website/docs/examples/transformers/code/run-snippets.py

+    # @@@DLT_SNIPPET_END example
+
+    # test assertions
+    row_counts = pipeline.last_trace.last_normalize_info.row_counts


wouldn't it be easier if we define examples directly in examples and execute it there with exec? then the code does not need to be copied. you have access to all locals anyway.

I prefer it this way, because we can add tests inbetween etc. But I can change it if you like.

rudolfix · 2023-09-28T13:20:10Z

docs/website/docs/examples/transformers/code/run-snippets.py

+    POKEMON_URL = "https://pokeapi.co/api/v2/pokemon"
+
+    # retrieve pokemon list
+    @dlt.resource(write_disposition="replace")


make it selected=False (so we do not store this in database)

rudolfix · 2023-09-28T13:20:57Z

docs/website/docs/examples/transformers/code/run-snippets.py

+    )
+
+    # the pokemon_list resource does not need to be loaded
+    load_info = pipeline.run([pokemon(), species()])


ok now it is clear why you have double evaluation. please create a dlt.source that returns 3 resources

the list resource

pokemons

species
use pipes to connect resources to transformers

Ok, I have done that, but you should check that it is done the right way. The tests pass.

rudolfix · 2023-09-28T13:22:45Z

docs/website/docs/examples/transformers/index.md

+
+## Using transformers with the pokemon api
+
+For this example we will be loading data from the [Poke Api](https://pokeapi.co/). We will load a list of pokemon from the


Look what @AstrakhantsevaAA did. 3-5 bullet points with highlights

we create two transformers that are connected to a single list resource with |

we load in parallel using requests

we load in parallel using async

we configure parallelism in config.toml

we deselect pokemon list by default to not load this in database
provide relevant docs links

rudolfix · 2023-09-28T13:22:58Z

docs/website/docs/examples/transformers/index.md

+```
+<!--@@@DLT_SNIPPET_END ./code/run-snippets.py::example-->
+
+### Example pokemon list data


I'd not include that

i like this, but I have removed it for now.

sh-rp · 2023-10-04T05:30:22Z

docs/examples/.dlt/secrets.toml

@@ -0,0 +1,26 @@
+# here is a file with the secrets for all the example pipelines in `examples` folder


@rudolfix should this be here?

rudolfix

LGTM! @sh-rp @AstrakhantsevaAA thanks for your hard work!

remove snipsync (#613)

34dcc91

* add custom snippets element

sh-rp force-pushed the d#/docs_examples branch 7 times, most recently from d9dce90 to b088cec Compare September 11, 2023 10:25

sh-rp changed the title ~~docs examples~~ examples for docs Sep 11, 2023

sh-rp force-pushed the d#/docs_examples branch 4 times, most recently from bebd527 to 249ebe7 Compare September 11, 2023 13:52

sh-rp added 2 commits September 11, 2023 15:56

remove snipsync

31d966f

Merge branch 'devel' into rfix/remove-snipsync

9ac7ee9

sh-rp force-pushed the d#/docs_examples branch 2 times, most recently from 78fadeb to 2323a3b Compare September 11, 2023 14:03

AstrakhantsevaAA changed the base branch from devel to rfix/remove-snipsync September 11, 2023 15:02

sh-rp force-pushed the rfix/remove-snipsync branch from 20614f2 to 15c5d0e Compare September 11, 2023 15:16

migrate performance snippets

180de6e

sh-rp force-pushed the rfix/remove-snipsync branch from 15c5d0e to 180de6e Compare September 11, 2023 15:27

sh-rp added 2 commits September 12, 2023 13:43

add missing init files

35a3c7d

small changes

4201946

rudolfix requested changes Sep 13, 2023

View reviewed changes

sh-rp marked this pull request as ready for review September 14, 2023 08:29

pr fixes

88d6912

rudolfix requested changes Sep 15, 2023

View reviewed changes

refine snippet watcher

5259d9d

sh-rp added 2 commits September 27, 2023 13:48

some work on the transformer example

4301dc0

more work on transformers example

78577ea

sh-rp force-pushed the d#/docs_examples branch from bef2ec9 to 78577ea Compare September 27, 2023 12:17

sh-rp and others added 7 commits September 27, 2023 14:21

make header smaller

f56dc6d

example for zendesk incremental loading

1243b46

move incremental loading example to right location

a99fa4c

added text and output example to incremental zendesk

ed7ede0

force build

eeafd62

allow secrets files in examples

84a83c1

clean up examples

eb7b780

sh-rp force-pushed the d#/docs_examples branch from 86b209f to eb7b780 Compare September 28, 2023 08:32

sh-rp added 2 commits September 28, 2023 10:54

add zendesk credentials

f1b7c93

fix zendesk example secrets pt 2

f6642c7

rudolfix requested changes Sep 28, 2023

View reviewed changes

AstrakhantsevaAA and others added 10 commits September 28, 2023 16:39

correct text and code snippets for zendesk example

98c3ea7

Merge branch 'devel' into d#/docs_examples

caea641

some PR fixes

4bb868a

Merge branch 'devel' into d#/docs_examples

ceb3fa1

add main clause

c6a503c

add config example

32ce681

Merge branch 'devel' into d#/docs_examples

07e8466

pytest marker to skip tests running in PRs from forks on github

5a8fe67

improvements in pokemon example

47f885a

fixes mypy

47d0465

sh-rp commented Oct 4, 2023

View reviewed changes

rudolfix added 2 commits October 4, 2023 10:50

removes more typings and adds comments to zendesk example

5a92beb

shortens example titles

0fcada4

rudolfix approved these changes Oct 4, 2023

View reviewed changes

rudolfix merged commit 58f8ad1 into devel Oct 4, 2023
30 of 31 checks passed

rudolfix deleted the d#/docs_examples branch October 4, 2023 09:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

examples for docs #616

examples for docs #616

sh-rp commented Sep 10, 2023 •

edited

Loading

netlify bot commented Sep 10, 2023 •

edited

Loading

rudolfix left a comment

rudolfix Sep 13, 2023

sh-rp Sep 14, 2023

rudolfix Sep 13, 2023

sh-rp Sep 14, 2023

sh-rp commented Sep 14, 2023

rudolfix left a comment

rudolfix Sep 15, 2023

rudolfix left a comment

This comment was marked as resolved.

rudolfix Sep 28, 2023

rudolfix Sep 28, 2023

rudolfix Sep 28, 2023

sh-rp Oct 2, 2023

rudolfix Sep 28, 2023

rudolfix Sep 28, 2023

sh-rp Oct 2, 2023

rudolfix Sep 28, 2023

rudolfix Sep 28, 2023

sh-rp Oct 2, 2023

sh-rp Oct 4, 2023

rudolfix left a comment

		@@ -0,0 +1,3 @@
		# @@@DLT_SNIPPET_START example
		some_key="some_value"


		## Using transformers with the pokemon api

		For this example we will be loading data from the [Poke Api](https://pokeapi.co/). We will load a list of pokemon from the

		@@ -0,0 +1,26 @@
		# here is a file with the secrets for all the example pipelines in `examples` folder

examples for docs #616

examples for docs #616

Conversation

sh-rp commented Sep 10, 2023 • edited Loading

Description

netlify bot commented Sep 10, 2023 • edited Loading

✅ Deploy Preview for dlt-hub-docs ready!

rudolfix left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sh-rp commented Sep 14, 2023

rudolfix left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rudolfix left a comment

Choose a reason for hiding this comment

This comment was marked as resolved.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rudolfix left a comment

Choose a reason for hiding this comment

sh-rp commented Sep 10, 2023 •

edited

Loading

netlify bot commented Sep 10, 2023 •

edited

Loading