-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
chore: correct structure for package
Signed-off-by: Bruno Campos <[email protected]>
- Loading branch information
Showing
15 changed files
with
71 additions
and
91 deletions.
There are no files selected for viewing
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,56 +1,15 @@ | ||
# dbt-source_db | ||
Welcome to your new dbt project! | ||
|
||
The `dbt-source_db` package allows you to specify the source database that dbt should read from. This enables reading from one database and writing to another. | ||
### Using the starter project | ||
|
||
## Getting Started | ||
Try running the following commands: | ||
- dbt run | ||
- dbt test | ||
|
||
Install the package: | ||
|
||
```bash | ||
dbt hub install dbt-labs/source_db | ||
``` | ||
|
||
In your `dbt_project.yml` file, add the package: | ||
|
||
```yml | ||
packages: | ||
- package: dbt-labs/source_db | ||
``` | ||
## Usage | ||
Set the `SRC_DB` environment variable to the source database you want dbt to read from: | ||
|
||
```bash | ||
export SRC_DB=dev_db | ||
``` | ||
|
||
Then run dbt as usual. The `ref()` and `source()` macros will read from `SRC_DB` instead of the default target database. | ||
|
||
Or you can set the variable within the same command. | ||
|
||
For example: | ||
|
||
```bash | ||
SRC_DB=dev_db dbt run | ||
``` | ||
|
||
This will read all sources and refs from `dev_db`, but write to the database in your profile/target. | ||
|
||
## Example | ||
|
||
You have a _sandbox_ and _production_ database. You want to test a new model `my_model` in _sandbox_, but reading data from _production_. | ||
|
||
Run the model: | ||
|
||
```bash | ||
SRC_DB=prod_db dbt run --models my_model | ||
``` | ||
|
||
This will read from `prod_db` but write `my_model` to _sandbox_. | ||
|
||
## Macro reference | ||
|
||
- `ref(model_name)`: Reads `model_name` from `SRC_DB` instead of target database. | ||
|
||
- `source(source_name, table_name)`: Reads `table_name` from `SRC_DB` instead of target database. | ||
### Resources: | ||
- Learn more about dbt [in the docs](https://docs.getdbt.com/docs/introduction) | ||
- Check out [Discourse](https://discourse.getdbt.com/) for commonly asked questions and answers | ||
- Join the [chat](https://community.getdbt.com/) on Slack for live discussions and support | ||
- Find [dbt events](https://events.getdbt.com) near you | ||
- Check out [the blog](https://blog.getdbt.com/) for the latest news on dbt's development and best practices |
File renamed without changes.
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,28 +1,64 @@ | ||
[0m12:10:52.513963 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'start', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x102d29ca0>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x104a76ca0>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x104a76e80>]} | ||
[0m12:17:17.686026 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'start', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x103ae0b80>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x130042d30>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x130042f10>]} | ||
|
||
|
||
============================== 12:10:52.520845 | b45290f9-b598-4557-b893-9968e7239325 ============================== | ||
[0m12:10:52.520845 [info ] [MainThread]: Running with dbt=1.6.1 | ||
[0m12:10:52.521671 [debug] [MainThread]: running dbt with arguments {'printer_width': '80', 'indirect_selection': 'eager', 'write_json': 'True', 'log_cache_events': 'False', 'partial_parse': 'True', 'cache_selected_only': 'False', 'profiles_dir': '/Users/bruno.campos/.dbt', 'debug': 'False', 'fail_fast': 'False', 'log_path': 'logs', 'version_check': 'True', 'warn_error': 'None', 'use_colors': 'True', 'use_experimental_parser': 'False', 'no_print': 'None', 'quiet': 'False', 'warn_error_options': 'WarnErrorOptions(include=[], exclude=[])', 'invocation_command': 'dbt init', 'introspect': 'True', 'log_format': 'default', 'target_path': 'None', 'static_parser': 'True', 'send_anonymous_usage_stats': 'True'} | ||
[0m12:13:22.147670 [debug] [MainThread]: Starter project path: /Users/bruno.campos/.virtualenv/[email protected]/lib/python3.9/site-packages/dbt/include/starter_project | ||
[0m12:13:22.160196 [info ] [MainThread]: | ||
Your new dbt project "source_db" was created! | ||
============================== 12:17:17.691038 | 8333030e-0484-4412-b02c-a04872e41557 ============================== | ||
[0m12:17:17.691038 [info ] [MainThread]: Running with dbt=1.6.1 | ||
[0m12:17:17.691496 [debug] [MainThread]: running dbt with arguments {'printer_width': '80', 'indirect_selection': 'eager', 'log_cache_events': 'False', 'write_json': 'True', 'partial_parse': 'True', 'cache_selected_only': 'False', 'warn_error': 'None', 'version_check': 'True', 'debug': 'False', 'log_path': '/Users/bruno.campos/GitRepos/source_db/source_db/logs', 'profiles_dir': '/Users/bruno.campos/.dbt', 'fail_fast': 'False', 'use_colors': 'True', 'use_experimental_parser': 'False', 'no_print': 'None', 'quiet': 'False', 'warn_error_options': 'WarnErrorOptions(include=[], exclude=[])', 'invocation_command': 'dbt debug', 'static_parser': 'True', 'introspect': 'True', 'target_path': 'None', 'log_format': 'default', 'send_anonymous_usage_stats': 'True'} | ||
[0m12:17:17.692178 [info ] [MainThread]: dbt version: 1.6.1 | ||
[0m12:17:17.692453 [info ] [MainThread]: python version: 3.9.17 | ||
[0m12:17:17.692691 [info ] [MainThread]: python path: /Users/bruno.campos/.virtualenv/[email protected]/bin/python | ||
[0m12:17:17.692929 [info ] [MainThread]: os info: macOS-13.4.1-arm64-arm-64bit | ||
[0m12:17:17.886181 [info ] [MainThread]: Using profiles dir at /Users/bruno.campos/.dbt | ||
[0m12:17:17.886505 [info ] [MainThread]: Using profiles.yml file at /Users/bruno.campos/.dbt/profiles.yml | ||
[0m12:17:17.886709 [info ] [MainThread]: Using dbt_project.yml file at /Users/bruno.campos/GitRepos/source_db/source_db/dbt_project.yml | ||
[0m12:17:17.887086 [info ] [MainThread]: adapter type: snowflake | ||
[0m12:17:17.887324 [info ] [MainThread]: adapter version: 1.6.1 | ||
[0m12:17:17.896095 [info ] [MainThread]: Configuration: | ||
[0m12:17:17.896345 [info ] [MainThread]: profiles.yml file [[32mOK found and valid[0m] | ||
[0m12:17:17.896537 [info ] [MainThread]: dbt_project.yml file [[32mOK found and valid[0m] | ||
[0m12:17:17.896725 [info ] [MainThread]: Required dependencies: | ||
[0m12:17:17.896967 [debug] [MainThread]: Executing "git --help" | ||
[0m12:17:17.915196 [debug] [MainThread]: STDOUT: "b"usage: git [-v | --version] [-h | --help] [-C <path>] [-c <name>=<value>]\n [--exec-path[=<path>]] [--html-path] [--man-path] [--info-path]\n [-p | --paginate | -P | --no-pager] [--no-replace-objects] [--bare]\n [--git-dir=<path>] [--work-tree=<path>] [--namespace=<name>]\n [--config-env=<name>=<envvar>] <command> [<args>]\n\nThese are common Git commands used in various situations:\n\nstart a working area (see also: git help tutorial)\n clone Clone a repository into a new directory\n init Create an empty Git repository or reinitialize an existing one\n\nwork on the current change (see also: git help everyday)\n add Add file contents to the index\n mv Move or rename a file, a directory, or a symlink\n restore Restore working tree files\n rm Remove files from the working tree and from the index\n\nexamine the history and state (see also: git help revisions)\n bisect Use binary search to find the commit that introduced a bug\n diff Show changes between commits, commit and working tree, etc\n grep Print lines matching a pattern\n log Show commit logs\n show Show various types of objects\n status Show the working tree status\n\ngrow, mark and tweak your common history\n branch List, create, or delete branches\n commit Record changes to the repository\n merge Join two or more development histories together\n rebase Reapply commits on top of another base tip\n reset Reset current HEAD to the specified state\n switch Switch branches\n tag Create, list, delete or verify a tag object signed with GPG\n\ncollaborate (see also: git help workflows)\n fetch Download objects and refs from another repository\n pull Fetch from and integrate with another repository or a local branch\n push Update remote refs along with associated objects\n\n'git help -a' and 'git help -g' list available subcommands and some\nconcept guides. See 'git help <command>' or 'git help <concept>'\nto read about a specific subcommand or concept.\nSee 'git help git' for an overview of the system.\n"" | ||
[0m12:17:17.915920 [debug] [MainThread]: STDERR: "b''" | ||
[0m12:17:17.916182 [info ] [MainThread]: - git [[32mOK found[0m] | ||
|
||
For more information on how to configure the profiles.yml file, | ||
please consult the dbt documentation here: | ||
[0m12:17:17.916410 [info ] [MainThread]: Connection: | ||
[0m12:17:17.916673 [info ] [MainThread]: account: zrgzuap-fa15311 | ||
[0m12:17:17.916856 [info ] [MainThread]: user: [email protected] | ||
[0m12:17:17.917032 [info ] [MainThread]: database: BRUNOC | ||
[0m12:17:17.917200 [info ] [MainThread]: warehouse: DATA_ANALYSTS | ||
[0m12:17:17.917365 [info ] [MainThread]: role: BRUNOC | ||
[0m12:17:17.917533 [info ] [MainThread]: schema: MARTS | ||
[0m12:17:17.917696 [info ] [MainThread]: authenticator: externalbrowser | ||
[0m12:17:17.917862 [info ] [MainThread]: private_key: None | ||
[0m12:17:17.918024 [info ] [MainThread]: private_key_path: None | ||
[0m12:17:17.918185 [info ] [MainThread]: private_key_passphrase: None | ||
[0m12:17:17.918347 [info ] [MainThread]: token: None | ||
[0m12:17:17.918508 [info ] [MainThread]: oauth_client_id: None | ||
[0m12:17:17.918670 [info ] [MainThread]: query_tag: None | ||
[0m12:17:17.918834 [info ] [MainThread]: client_session_keep_alive: False | ||
[0m12:17:17.918997 [info ] [MainThread]: host: None | ||
[0m12:17:17.919157 [info ] [MainThread]: port: None | ||
[0m12:17:17.919315 [info ] [MainThread]: proxy_host: None | ||
[0m12:17:17.919477 [info ] [MainThread]: proxy_port: None | ||
[0m12:17:17.919638 [info ] [MainThread]: protocol: None | ||
[0m12:17:17.919799 [info ] [MainThread]: connect_retries: 1 | ||
[0m12:17:17.919958 [info ] [MainThread]: connect_timeout: None | ||
[0m12:17:17.920121 [info ] [MainThread]: retry_on_database_errors: False | ||
[0m12:17:17.920283 [info ] [MainThread]: retry_all: False | ||
[0m12:17:17.920443 [info ] [MainThread]: insecure_mode: False | ||
[0m12:17:17.920602 [info ] [MainThread]: reuse_connections: None | ||
[0m12:17:17.920925 [info ] [MainThread]: Registered adapter: snowflake=1.6.1 | ||
[0m12:17:17.970800 [debug] [MainThread]: Acquiring new snowflake connection 'debug' | ||
[0m12:17:17.971857 [debug] [MainThread]: Using snowflake connection "debug" | ||
[0m12:17:17.972103 [debug] [MainThread]: On debug: select 1 as id | ||
[0m12:17:17.972331 [debug] [MainThread]: Opening a new connection, currently in state init | ||
[0m12:17:27.522832 [debug] [MainThread]: SQL status: SUCCESS 1 in 10.0 seconds | ||
[0m12:17:27.527870 [debug] [MainThread]: On debug: Close | ||
[0m12:17:27.593599 [info ] [MainThread]: Connection test: [[32mOK connection ok[0m] | ||
|
||
https://docs.getdbt.com/docs/configure-your-profile | ||
|
||
One more thing: | ||
|
||
Need help? Don't hesitate to reach out to us via GitHub issues or on Slack: | ||
|
||
https://community.getdbt.com/ | ||
|
||
Happy modeling! | ||
|
||
[0m12:13:22.160706 [info ] [MainThread]: Setting up your profile. | ||
[0m12:16:17.536157 [info ] [MainThread]: Profile source_db written to /Users/bruno.campos/.dbt/profiles.yml using target's profile_template.yml and your supplied values. Run 'dbt debug' to validate the connection. | ||
[0m12:16:17.539205 [debug] [MainThread]: Command `dbt init` succeeded at 12:16:17.538902 after 325.38 seconds | ||
[0m12:16:17.539890 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'end', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x102d29ca0>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x104a69a90>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x10550f550>]} | ||
[0m12:16:17.540609 [debug] [MainThread]: Flushing usage events | ||
[0m12:17:27.594695 [info ] [MainThread]: [32mAll checks passed![0m | ||
[0m12:17:27.596617 [debug] [MainThread]: Command `dbt debug` succeeded at 12:17:27.596293 after 10.21 seconds | ||
[0m12:17:27.599973 [debug] [MainThread]: Connection 'debug' was properly closed. | ||
[0m12:17:27.601037 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'end', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x103ae0b80>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x13192aca0>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x131a14280>]} | ||
[0m12:17:27.604086 [debug] [MainThread]: Flushing usage events |
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
This file was deleted.
Oops, something went wrong.
File renamed without changes.