Skip to content

Commit

Permalink
chore: correct structure for package
Browse files Browse the repository at this point in the history
Signed-off-by: Bruno Campos <[email protected]>
  • Loading branch information
BfdCampos committed Aug 31, 2023
1 parent dfedbc3 commit ec4ea39
Show file tree
Hide file tree
Showing 15 changed files with 71 additions and 91 deletions.
File renamed without changes.
63 changes: 11 additions & 52 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,56 +1,15 @@
# dbt-source_db
Welcome to your new dbt project!

The `dbt-source_db` package allows you to specify the source database that dbt should read from. This enables reading from one database and writing to another.
### Using the starter project

## Getting Started
Try running the following commands:
- dbt run
- dbt test

Install the package:

```bash
dbt hub install dbt-labs/source_db
```

In your `dbt_project.yml` file, add the package:

```yml
packages:
- package: dbt-labs/source_db
```
## Usage
Set the `SRC_DB` environment variable to the source database you want dbt to read from:

```bash
export SRC_DB=dev_db
```

Then run dbt as usual. The `ref()` and `source()` macros will read from `SRC_DB` instead of the default target database.

Or you can set the variable within the same command.

For example:

```bash
SRC_DB=dev_db dbt run
```

This will read all sources and refs from `dev_db`, but write to the database in your profile/target.

## Example

You have a _sandbox_ and _production_ database. You want to test a new model `my_model` in _sandbox_, but reading data from _production_.

Run the model:

```bash
SRC_DB=prod_db dbt run --models my_model
```

This will read from `prod_db` but write `my_model` to _sandbox_.

## Macro reference

- `ref(model_name)`: Reads `model_name` from `SRC_DB` instead of target database.

- `source(source_name, table_name)`: Reads `table_name` from `SRC_DB` instead of target database.
### Resources:
- Learn more about dbt [in the docs](https://docs.getdbt.com/docs/introduction)
- Check out [Discourse](https://discourse.getdbt.com/) for commonly asked questions and answers
- Join the [chat](https://community.getdbt.com/) on Slack for live discussions and support
- Find [dbt events](https://events.getdbt.com) near you
- Check out [the blog](https://blog.getdbt.com/) for the latest news on dbt's development and best practices
File renamed without changes.
File renamed without changes.
84 changes: 60 additions & 24 deletions logs/dbt.log
Original file line number Diff line number Diff line change
@@ -1,28 +1,64 @@
12:10:52.513963 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'start', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x102d29ca0>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x104a76ca0>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x104a76e80>]}
12:17:17.686026 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'start', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x103ae0b80>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x130042d30>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x130042f10>]}


============================== 12:10:52.520845 | b45290f9-b598-4557-b893-9968e7239325 ==============================
12:10:52.520845 [info ] [MainThread]: Running with dbt=1.6.1
12:10:52.521671 [debug] [MainThread]: running dbt with arguments {'printer_width': '80', 'indirect_selection': 'eager', 'write_json': 'True', 'log_cache_events': 'False', 'partial_parse': 'True', 'cache_selected_only': 'False', 'profiles_dir': '/Users/bruno.campos/.dbt', 'debug': 'False', 'fail_fast': 'False', 'log_path': 'logs', 'version_check': 'True', 'warn_error': 'None', 'use_colors': 'True', 'use_experimental_parser': 'False', 'no_print': 'None', 'quiet': 'False', 'warn_error_options': 'WarnErrorOptions(include=[], exclude=[])', 'invocation_command': 'dbt init', 'introspect': 'True', 'log_format': 'default', 'target_path': 'None', 'static_parser': 'True', 'send_anonymous_usage_stats': 'True'}
12:13:22.147670 [debug] [MainThread]: Starter project path: /Users/bruno.campos/.virtualenv/[email protected]/lib/python3.9/site-packages/dbt/include/starter_project
12:13:22.160196 [info ] [MainThread]:
Your new dbt project "source_db" was created!
============================== 12:17:17.691038 | 8333030e-0484-4412-b02c-a04872e41557 ==============================
12:17:17.691038 [info ] [MainThread]: Running with dbt=1.6.1
12:17:17.691496 [debug] [MainThread]: running dbt with arguments {'printer_width': '80', 'indirect_selection': 'eager', 'log_cache_events': 'False', 'write_json': 'True', 'partial_parse': 'True', 'cache_selected_only': 'False', 'warn_error': 'None', 'version_check': 'True', 'debug': 'False', 'log_path': '/Users/bruno.campos/GitRepos/source_db/source_db/logs', 'profiles_dir': '/Users/bruno.campos/.dbt', 'fail_fast': 'False', 'use_colors': 'True', 'use_experimental_parser': 'False', 'no_print': 'None', 'quiet': 'False', 'warn_error_options': 'WarnErrorOptions(include=[], exclude=[])', 'invocation_command': 'dbt debug', 'static_parser': 'True', 'introspect': 'True', 'target_path': 'None', 'log_format': 'default', 'send_anonymous_usage_stats': 'True'}
12:17:17.692178 [info ] [MainThread]: dbt version: 1.6.1
12:17:17.692453 [info ] [MainThread]: python version: 3.9.17
12:17:17.692691 [info ] [MainThread]: python path: /Users/bruno.campos/.virtualenv/[email protected]/bin/python
12:17:17.692929 [info ] [MainThread]: os info: macOS-13.4.1-arm64-arm-64bit
12:17:17.886181 [info ] [MainThread]: Using profiles dir at /Users/bruno.campos/.dbt
12:17:17.886505 [info ] [MainThread]: Using profiles.yml file at /Users/bruno.campos/.dbt/profiles.yml
12:17:17.886709 [info ] [MainThread]: Using dbt_project.yml file at /Users/bruno.campos/GitRepos/source_db/source_db/dbt_project.yml
12:17:17.887086 [info ] [MainThread]: adapter type: snowflake
12:17:17.887324 [info ] [MainThread]: adapter version: 1.6.1
12:17:17.896095 [info ] [MainThread]: Configuration:
12:17:17.896345 [info ] [MainThread]: profiles.yml file [OK found and valid]
12:17:17.896537 [info ] [MainThread]: dbt_project.yml file [OK found and valid]
12:17:17.896725 [info ] [MainThread]: Required dependencies:
12:17:17.896967 [debug] [MainThread]: Executing "git --help"
12:17:17.915196 [debug] [MainThread]: STDOUT: "b"usage: git [-v | --version] [-h | --help] [-C <path>] [-c <name>=<value>]\n [--exec-path[=<path>]] [--html-path] [--man-path] [--info-path]\n [-p | --paginate | -P | --no-pager] [--no-replace-objects] [--bare]\n [--git-dir=<path>] [--work-tree=<path>] [--namespace=<name>]\n [--config-env=<name>=<envvar>] <command> [<args>]\n\nThese are common Git commands used in various situations:\n\nstart a working area (see also: git help tutorial)\n clone Clone a repository into a new directory\n init Create an empty Git repository or reinitialize an existing one\n\nwork on the current change (see also: git help everyday)\n add Add file contents to the index\n mv Move or rename a file, a directory, or a symlink\n restore Restore working tree files\n rm Remove files from the working tree and from the index\n\nexamine the history and state (see also: git help revisions)\n bisect Use binary search to find the commit that introduced a bug\n diff Show changes between commits, commit and working tree, etc\n grep Print lines matching a pattern\n log Show commit logs\n show Show various types of objects\n status Show the working tree status\n\ngrow, mark and tweak your common history\n branch List, create, or delete branches\n commit Record changes to the repository\n merge Join two or more development histories together\n rebase Reapply commits on top of another base tip\n reset Reset current HEAD to the specified state\n switch Switch branches\n tag Create, list, delete or verify a tag object signed with GPG\n\ncollaborate (see also: git help workflows)\n fetch Download objects and refs from another repository\n pull Fetch from and integrate with another repository or a local branch\n push Update remote refs along with associated objects\n\n'git help -a' and 'git help -g' list available subcommands and some\nconcept guides. See 'git help <command>' or 'git help <concept>'\nto read about a specific subcommand or concept.\nSee 'git help git' for an overview of the system.\n""
12:17:17.915920 [debug] [MainThread]: STDERR: "b''"
12:17:17.916182 [info ] [MainThread]: - git [OK found]

For more information on how to configure the profiles.yml file,
please consult the dbt documentation here:
12:17:17.916410 [info ] [MainThread]: Connection:
12:17:17.916673 [info ] [MainThread]: account: zrgzuap-fa15311
12:17:17.916856 [info ] [MainThread]: user: [email protected]
12:17:17.917032 [info ] [MainThread]: database: BRUNOC
12:17:17.917200 [info ] [MainThread]: warehouse: DATA_ANALYSTS
12:17:17.917365 [info ] [MainThread]: role: BRUNOC
12:17:17.917533 [info ] [MainThread]: schema: MARTS
12:17:17.917696 [info ] [MainThread]: authenticator: externalbrowser
12:17:17.917862 [info ] [MainThread]: private_key: None
12:17:17.918024 [info ] [MainThread]: private_key_path: None
12:17:17.918185 [info ] [MainThread]: private_key_passphrase: None
12:17:17.918347 [info ] [MainThread]: token: None
12:17:17.918508 [info ] [MainThread]: oauth_client_id: None
12:17:17.918670 [info ] [MainThread]: query_tag: None
12:17:17.918834 [info ] [MainThread]: client_session_keep_alive: False
12:17:17.918997 [info ] [MainThread]: host: None
12:17:17.919157 [info ] [MainThread]: port: None
12:17:17.919315 [info ] [MainThread]: proxy_host: None
12:17:17.919477 [info ] [MainThread]: proxy_port: None
12:17:17.919638 [info ] [MainThread]: protocol: None
12:17:17.919799 [info ] [MainThread]: connect_retries: 1
12:17:17.919958 [info ] [MainThread]: connect_timeout: None
12:17:17.920121 [info ] [MainThread]: retry_on_database_errors: False
12:17:17.920283 [info ] [MainThread]: retry_all: False
12:17:17.920443 [info ] [MainThread]: insecure_mode: False
12:17:17.920602 [info ] [MainThread]: reuse_connections: None
12:17:17.920925 [info ] [MainThread]: Registered adapter: snowflake=1.6.1
12:17:17.970800 [debug] [MainThread]: Acquiring new snowflake connection 'debug'
12:17:17.971857 [debug] [MainThread]: Using snowflake connection "debug"
12:17:17.972103 [debug] [MainThread]: On debug: select 1 as id
12:17:17.972331 [debug] [MainThread]: Opening a new connection, currently in state init
12:17:27.522832 [debug] [MainThread]: SQL status: SUCCESS 1 in 10.0 seconds
12:17:27.527870 [debug] [MainThread]: On debug: Close
12:17:27.593599 [info ] [MainThread]: Connection test: [OK connection ok]

https://docs.getdbt.com/docs/configure-your-profile

One more thing:

Need help? Don't hesitate to reach out to us via GitHub issues or on Slack:

https://community.getdbt.com/

Happy modeling!

12:13:22.160706 [info ] [MainThread]: Setting up your profile.
12:16:17.536157 [info ] [MainThread]: Profile source_db written to /Users/bruno.campos/.dbt/profiles.yml using target's profile_template.yml and your supplied values. Run 'dbt debug' to validate the connection.
12:16:17.539205 [debug] [MainThread]: Command `dbt init` succeeded at 12:16:17.538902 after 325.38 seconds
12:16:17.539890 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'end', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x102d29ca0>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x104a69a90>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x10550f550>]}
12:16:17.540609 [debug] [MainThread]: Flushing usage events
12:17:27.594695 [info ] [MainThread]: All checks passed!
12:17:27.596617 [debug] [MainThread]: Command `dbt debug` succeeded at 12:17:27.596293 after 10.21 seconds
12:17:27.599973 [debug] [MainThread]: Connection 'debug' was properly closed.
12:17:27.601037 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'end', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x103ae0b80>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x13192aca0>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x131a14280>]}
12:17:27.604086 [debug] [MainThread]: Flushing usage events
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
15 changes: 0 additions & 15 deletions source_db/README.md

This file was deleted.

File renamed without changes.

0 comments on commit ec4ea39

Please sign in to comment.