feat: add dbt adapter unit testing command(s)

[bdewilde]

Hi! I don't think this is a feature request, so much as a "request for guidance" — and maybe docs, if they don't already exist. :)

I have a dbt project embedded within and run via meltano, in the usual transform sub-directory. I'm using dbt's built-in, pytest-based unit testing functionality to test the logic of my SQL transformations. I've been able to make this work by manually installing dbt-core and a db adapter (plus pytest) into the env where meltano is installed, then running pytest transform/tests as normal. This isn't ideal, however, because dbt is specified in the meltano project file and handled via the plugin framework, while this is a manual process that isn't guaranteed to use the same versions, etc. of dbt.

Is there a more convenient, safer way to access/activate the dbt (or whichever) plugin and run tests that require plugins' availability? I didn't see anything about this in the meltano environments or plugins docs; maybe I'm just missing something. Your guidance is appreciated!

[tayloramurphy]

@pnadolny13 or @meltano/engineering if you have a moment can you reply here?

@bdewilde are you on our slack community? There's lots of folks there who may be able to help answer your question there as well.

[bdewilde]

Hi @tayloramurphy , I'm in the dbt slack community, where there is a (relatively inactive) #tools-meltano channel. I searched there re: dbt unit testing via meltano, but nothing came up. Happy to join the meltano equivalent — I see the link on your website, under "Get Help" 🙂

Just for future reference: Are these sorts of questions preferred in that forum, or do GitHub issues also work for this?

[edgarrmondragon]

I have not tested this but I think a custom command might work. A new pytest command set to the appropriate executable similar to

hub/_data/meltano/utilities/dbt-duckdb/jwills.yml

Lines 33 to 37 in 65ce1ae

	initialize:
	args: initialize
	description: |
	Initialize a new dbt project. This will create a dbt_project.yml file, a profiles.yml file, and models directory.
	executable: dbt_extension

.

[bdewilde]

hi @edgarrmondragon , thanks for the suggestion! (and always delighted to learn from josh wills' example -- we were co-workers until last december, still miss having him around.) here's a snippet of my updated meltano project config:

plugins:
  utilities:
    - name: dbt-postgres
      variant: dbt-labs
      pip_url: dbt-core~=1.5.0 dbt-postgres~=1.5.0 git+https://github.com/meltano/dbt-ext.git@main
      config:
        dbname: ${DBT_POSTGRES_DBNAME}
      commands:
        data_test: test
        unit_test:
          executable: python -m pytest
          args: "-v transform/tests/functional/"

the "data test" command runs as expected:

$ docker compose exec meltano meltano run dbt-postgres:data_test
2023-07-25T14:21:50.828175Z [info     ] Environment 'dev' is active
2023-07-25T14:21:52.015793Z [info     ] Extension executing `dbt clean`... cmd_type=command name=dbt-postgres stdio=stderr
2023-07-25T14:21:55.356974Z [info     ] 14:21:55  Running with dbt=1.5.2 cmd_type=command name=dbt-postgres stdio=stderr
...
2023-07-25T14:22:06.598594Z [info     ] 14:22:06  Found 40 models, 196 tests, 9 snapshots, 0 analyses, 667 macros, 0 operations, 0 seed files, 21 sources, 0 exposures, 0 metrics, 0 groups cmd_type=command name=dbt-postgres stdio=stderr
...

unfortunately, the "unit test" command does not run:

$ docker compose exec meltano meltano run dbt-postgres:unit_test
2023-07-25T14:18:58.059153Z [info     ] Environment 'dev' is active
2023-07-25T14:18:58.860901Z [error    ] Block run completed.           block_type=InvokerCommand err=RunnerError("Cannot start plugin: Executable 'dbt_invoker' could not be found. Utility 'dbt-postgres' may not have been installed yet using `meltano install utility dbt-postgres`, or the executable name may be incorrect.") exit_codes={} set_number=0 success=False
Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to
join our friendly Slack community.

Run invocation could not be completed as block failed: Cannot start plugin: Executable 'dbt_invoker' could not be found. Utility 'dbt-postgres' may not have been installed yet using `meltano install utility dbt-postgres`, or the executable name may be incorrect.

that error was unchanged after i manually (re-)installed the plugin:

$ docker compose exec meltano meltano install utility dbt-postgres
Installing 1 plugins...
Installing utility 'dbt-postgres'...
Installed utility 'dbt-postgres'`

for good measure, i tried shelling into the container and running manually, without success -- though this error was expected:

$ docker compose exec meltano /bin/bash
root@aeaf009f35b3:/app# python -m pytest
ImportError while loading conftest '/app/transform/tests/conftest.py'.
transform/tests/conftest.py:3: in <module>
    import dbt.tests.util
E   ModuleNotFoundError: No module named 'dbt'

so, it seems like there's something else going on here. i'm hoping this is just user error -- i've been using meltano for a few months, but not yet confident in its correct usage. would welcome further suggestions, or i can take this to the meltano slack community per @tayloramurphy 's initial reply. thanks for your help!

[edgarrmondragon]

@bdewilde You probably want the executable to just be pytest. And ensure pytest is installed:

plugins:
  utilities:
    - name: dbt-postgres
      variant: dbt-labs
      # Add `pytest` to the pip install args, in case it's not included with dbt-core
      pip_url: dbt-core~=1.5.0 dbt-postgres~=1.5.0 git+https://github.com/meltano/dbt-ext.git@main pytest
      config:
        dbname: ${DBT_POSTGRES_DBNAME}
      commands:
        data_test: test
        unit_test:
          # Simplify the executable
          executable: pytest
          args: "-v transform/tests/functional/"

And run meltano install utility dbt-postgres --clean for good measure.

[bdewilde]

Hi @edgarrmondragon , thanks again for your help. I tried several variations of meltano project config, env build, and test commands, and finally got things to work as you described -- without the extra meltano install. It seems that I was running afoul of Docker caching (my meltano app is containerized, close to what's described in the official docs), such that changes to the utility plugin configuration weren't being pulled into the container. Don't really understand why, but happy to 🤷‍♂️ and move on!

Should I close this ticket out, or would you prefer to make any changes / get more info before closing it yourself?

[edgarrmondragon]

Glad you got it working @bdewilde!

I wonder if we should add this command to the plugin definition in the Hub cc @tayloramurphy @pnadolny13

[tayloramurphy]

@edgarrmondragon I'm supportive! We can indicate that it's mainly for testing an adapter, but it certainly wouldn't hurt to have it there. Can you either make a new issue or transfer this one?