Pytest discovery in VS code

[sjraaijmakers]

I am using VS code with databricks-connect v2.4.8, and I wrote some pytests. In conftest.py I created a fixture:

@pytest.fixture(scope="session")
def spark():
    spark = DatabricksSession.builder.getOrCreate()
    yield spark
    spark.stop()

Running the pytest command from within the terminal in the venv works just fine. However, when using the VS code test plugin to discover the tests I get this error:

.venv\lib\site-packages\databricks\sdk\config.py:130: in __init__
    raise ValueError(message) from e
E   ValueError: default auth: cannot configure default credentials, please check https://docs.databricks.com/en/dev-tools/auth.html#databricks-client-unified-authentication to configure credentials for your preferred authentication method. Config: host=MY_HOST, auth_type=databricks-cli, cluster_id=MY_CLUSTER_ID

Anyone any idea on how to solve this?

[ilia-db]

Do you have DATABRICKS_PROFILE env var set in the terminal? This error can happen when we can't identify what profile to use for the auth. By default we look into ~/.databrickscfg file for the DEFAULT profile, and if we can't find it, then auth fails. You don't need to have DEFAULT profile, and instead specify the name directly in the code like this:

spark = DatabricksSession.builder.profile("<profile-name>").getOrCreate()

[sjraaijmakers]

I do have a ~/.databrickscfg with a profile named DEFAULT:

[DEFAULT]
host=MY_HOST
auth_type=databricks-cli
cluster_id=MY_CLUSTER_ID

In the earlier provided error it does seem like Databricks can find this profile (showing MY_HOST and MY_CLUSTER_ID in the provided error). I also tried it with explicitly setting the profile to DEFAULT:

spark = DatabricksSession.builder.profile("DEFAULT").getOrCreate()

without success unfortunately.

I have the same error - terminal pytest works, while vscode plugin doesn't - when running the main_test.py file that comes with the stub when creating a new databricks project using the databricks-connect plugin

[ilia-db]

Hey, sorry, this issue got lost

If you are still experiencing the same problems, please checkout comments to this issue #1540

I've provided some guidance on how to run tests in the vscode there.

Closing this issue for now, please re-open if there are still any problems