Add watsonx spark config and setup files #6895

bynist · 2025-02-10T17:44:14Z

What are you changing in this pull request and why?

Adding IBM Watsonx Spark config and setup files

Checklist

I have reviewed the Content style guide so my content adheres to these guidelines.
The topic I'm writing about is for specific dbt version(s) and I have versioned it according to the version a whole page and/or version a block of content guidelines.
I have added checklist item(s) to this list for anything anything that needs to happen before this PR is merged, such as "needs technical review" or "change base branch."
The content in this PR requires a dbt release note, so I added one to the release notes page.

welcome · 2025-02-10T17:44:17Z

Hello!👋 Thanks for contributing to the dbt product documentation and opening this pull request! ✨
We use Markdown and some HTML to write the dbt product documentation. When writing content, you can use our style guide and content types to understand our writing standards and how we organize information in the dbt product docs.
We'll review your contribution and respond as soon as we can. 😄

vercel · 2025-02-10T17:44:19Z

@ReemaAlzaid is attempting to deploy a commit to the dbt-labs Team on Vercel.

A member of the Team first needs to authorize it.

website/docs/docs/core/connect-data-platform/watsonx-spark-setup.md

…up.md

website/docs/docs/core/connect-data-platform/watsonx-spark-setup.md

…up.md

website/docs/reference/resource-configs/watsonx-spark-config.md

DivyaLokesh · 2025-02-12T06:30:45Z

website/docs/reference/resource-configs/watsonx-spark-config.md

+- [watsonx.data SaaS Catalog](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-reg_database)
+- [watsonx.data Software Catalog](https://www.ibm.com/docs/en/watsonx/watsonxdata/2.1.x?topic=components-adding-data-source)
+
+### Extra configuration


Change Extra > Additional

DivyaLokesh · 2025-02-12T06:36:34Z

website/docs/docs/core/connect-data-platform/watsonx-spark-setup.md

+
+## Installing dbt-watsonx-spark
+
+Since dbt v1.8, installing an adapter no longer installs `dbt-core` automatically.


Combine line 33 and 34 as follows:
Use the following command to install the adapter:
Note: From dbt v1.8, installing an adapter no longer installs 'dbt-core' automatically. This is because adapters and dbt Core versions are decoupled to avoid overwriting dbt-core installations.

DivyaLokesh · 2025-02-12T06:40:36Z

website/docs/docs/core/connect-data-platform/watsonx-spark-setup.md

+```
+
+## Configuring `dbt-watsonx-spark`
+For IBM watsonx.data-specific configuration, please refer to [IBM watsonx.data configs.](https://www.ibm.com/docs/en/watsonx/watsonxdata/2.1.x?topic=spark-configuration-setting-up-your-profile)


remove "please"

DivyaLokesh · 2025-02-12T06:41:52Z

website/docs/docs/core/connect-data-platform/watsonx-spark-setup.md

+
+## Connecting to IBM watsonx.data Spark
+
+To connect dbt with watsonx.data Spark, you need to configure a profile in your `profiles.yml` file located in the `.dbt/` directory of your home folder. The following is an example configuration for connecting to IBM watsonx.data SaaS and Software instances:


Change, "you need to configure" to "configure a profile in your..."

DivyaLokesh · 2025-02-12T06:44:34Z

website/docs/docs/core/connect-data-platform/watsonx-spark-setup.md

+
+## Host parameters
+
+The following profile fields are required to configure watsonx.data Spark connections. For IBM watsonx.data SaaS or Software instances, you can get the `profile` details by clicking **View connect details** after `the query server` is in RUNNING stat, The Connection details page opens with the profile configuration.


To get the 'profile' details, click 'View connect details' when the 'query server' is in RUNNING status in watsonx.data (In watsonx.data (both SaaS or Software). The Connection details page opens...

Remove , after stat.

DivyaLokesh · 2025-02-12T06:45:58Z

website/docs/docs/core/connect-data-platform/watsonx-spark-setup.md

+## Host parameters
+
+The following profile fields are required to configure watsonx.data Spark connections. For IBM watsonx.data SaaS or Software instances, you can get the `profile` details by clicking **View connect details** after `the query server` is in RUNNING stat, The Connection details page opens with the profile configuration.
+Copy the connection details. Then Paste the connection details in the profiles.yml file that is located in .dbt of your home directory


Copy and paste the connection details in the profiles.yml...

DivyaLokesh · 2025-02-12T06:50:07Z

website/docs/docs/core/connect-data-platform/watsonx-spark-setup.md

+| `catalog`  | Required                      | The catalog that is associated with the Spark engine.                     | `my_catalog`      |
+| `use_ssl`  | Optional (default: **false**) | Specifies whether to use SSL.                                             | `true` or `false` |
+| `instance` | Required                      | For **SaaS** set it as CRN of watsonx.data. As for **Software**, set it as instance ID of watsonx.data| `1726574045872688`|
+| `user`     | Required                      | Your watsonx.data username                                                | `[email protected]`|


The authentication (username and passwords) are different for SaaS and Software and also there are some variations. Ensure to give all those details.

Also, change "Your watsonx.data username" > "Username for the watsonx.data instance"

DivyaLokesh · 2025-02-12T06:51:47Z

website/docs/docs/core/connect-data-platform/watsonx-spark-setup.md

+
+### Schemas and catalogs
+
+When selecting the catalog, ensure the user has read and write access. This selection does not limit your ability to query into the schema spcified/created but also serves as the default location for materialized `tables`, `views`, and `incremental`.


Did you mean to say "This selection does not limit your ability to query into the schema specified/created by other users?"

DivyaLokesh · 2025-02-12T06:52:48Z

website/docs/docs/core/connect-data-platform/watsonx-spark-setup.md

+| ---------- | ----------------------------- | ------------------------------------------------------------------------- | ----------------- |
+| `method`   | Required |    Specifies the connection method to the spark query server. Use `http`.    | `http`            |
+| `schema`   | Required|    To choose an existing schema within spark engine or create a new schema.  | `spark_schema`    |
+| `host`     | Required |    Hostname of your watsonx.data console. For more information, see [Getting connection information](https://www.ibm.com/docs/en/watsonx/watsonxdata/2.1.x?topic=references-getting-connection-information#connection_info__conn_info_).| `https://dataplatform.cloud.ibm.com`       |


your watsonx.data > the watsonx.data

DivyaLokesh · 2025-02-12T06:54:39Z

website/docs/docs/core/connect-data-platform/watsonx-spark-setup.md

+### SSL verification
+
+- If the Spark instance uses an unsecured HTTP connection, set `use_ssl` to `false`.
+- If the instance uses `HTTPS`, this parameter should be set to `true`.


Change "this parameter should be set to 'true' > set it 'true'

DivyaLokesh · 2025-02-12T06:55:42Z

website/docs/docs/core/connect-data-platform/watsonx-spark-setup.md

+
+## Additional parameters
+
+The following profile fields are optional to set up. They let you configure your instance session and dbt for your connection.


The following profile fields are optional. You can configure the instance session and dbt for the connection.

@DivyaLokesh I’ve addressed all the requested changes, Please let me know if there’s anything else

vercel · 2025-02-18T21:29:05Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated (UTC)
docs-getdbt-com	✅ Ready (Inspect)	Visit Preview	Feb 18, 2025 9:31pm

bynist and others added 3 commits February 10, 2025 10:48

adding Watsonx-spark config and setup files

522fcd3

Fixing alignment table issue

11183b3

Modifying on watsonx spark setup and config docs

c44afe4

bynist requested a review from a team as a code owner February 10, 2025 17:44

runleonarun added the new contributor Label for first-time contributors label Feb 10, 2025

github-actions bot added content Improvements or additions to content size: large This change will more than a week to address and might require more than one person labels Feb 10, 2025

mirnawong1 reviewed Feb 11, 2025

View reviewed changes

website/docs/docs/core/connect-data-platform/watsonx-spark-setup.md Outdated Show resolved Hide resolved

Update website/docs/docs/core/connect-data-platform/watsonx-spark-set…

fb5bfdd

…up.md

mirnawong1 reviewed Feb 11, 2025

View reviewed changes

website/docs/docs/core/connect-data-platform/watsonx-spark-setup.md Outdated Show resolved Hide resolved

Update website/docs/docs/core/connect-data-platform/watsonx-spark-set…

bd59429

…up.md

mirnawong1 reviewed Feb 11, 2025

View reviewed changes

website/docs/reference/resource-configs/watsonx-spark-config.md Outdated Show resolved Hide resolved

Update website/docs/reference/resource-configs/watsonx-spark-config.md

4411403

mirnawong1 reviewed Feb 11, 2025

View reviewed changes

website/docs/reference/resource-configs/watsonx-spark-config.md Outdated Show resolved Hide resolved

mirnawong1 reviewed Feb 11, 2025

View reviewed changes

website/docs/reference/resource-configs/watsonx-spark-config.md Outdated Show resolved Hide resolved

mirnawong1 reviewed Feb 11, 2025

View reviewed changes

website/docs/reference/resource-configs/watsonx-spark-config.md Outdated Show resolved Hide resolved

mirnawong1 added 3 commits February 11, 2025 10:09

Update website/docs/reference/resource-configs/watsonx-spark-config.md

1986925

Update website/docs/reference/resource-configs/watsonx-spark-config.md

35de721

Update website/docs/reference/resource-configs/watsonx-spark-config.md

149b231

mirnawong1 requested a review from amychen1776 February 11, 2025 10:10

mirnawong1 mentioned this pull request Feb 11, 2025

Adding dbt-watsonx-spark configuration and setup #6889

Closed

4 tasks

DivyaLokesh reviewed Feb 12, 2025

View reviewed changes

request changes updated

1cd139d

github-actions bot added size: medium This change will take up to a week to address and removed size: large This change will more than a week to address and might require more than one person labels Feb 12, 2025

amychen1776 approved these changes Feb 12, 2025

View reviewed changes

bynist requested a review from DivyaLokesh February 13, 2025 08:41

mirnawong1 and others added 2 commits February 14, 2025 11:18

Merge branch 'current' into add-Watsonx-spark-config-and-setup-files

ef3b100

Merge branch 'current' into add-Watsonx-spark-config-and-setup-files

27a31ff

amychen1776 enabled auto-merge February 18, 2025 21:26

mirnawong1 added 5 commits February 20, 2025 18:22

Merge branch 'current' into add-Watsonx-spark-config-and-setup-files

1dc0af8

Update watsonx-spark-config.md

e87b6f5

Update watsonx-spark-setup.md

05d793c

Update watsonx-spark-setup.md

c6e9fdb

Update watsonx-spark-setup.md

6e2db29

mirnawong1 approved these changes Feb 20, 2025

View reviewed changes

mirnawong1 disabled auto-merge February 20, 2025 18:47

mirnawong1 merged commit f68119e into dbt-labs:current Feb 20, 2025
3 of 5 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add watsonx spark config and setup files #6895

Add watsonx spark config and setup files #6895

bynist commented Feb 10, 2025

welcome bot commented Feb 10, 2025

vercel bot commented Feb 10, 2025

DivyaLokesh Feb 12, 2025

DivyaLokesh Feb 12, 2025 •

edited

Loading

DivyaLokesh Feb 12, 2025

DivyaLokesh Feb 12, 2025

DivyaLokesh Feb 12, 2025 •

edited

Loading

DivyaLokesh Feb 12, 2025

DivyaLokesh Feb 12, 2025 •

edited

Loading

DivyaLokesh Feb 12, 2025

bynist Feb 12, 2025

DivyaLokesh Feb 12, 2025

DivyaLokesh Feb 12, 2025

DivyaLokesh Feb 12, 2025

bynist Feb 16, 2025

vercel bot commented Feb 18, 2025 •

edited

Loading


		## Installing dbt-watsonx-spark

		Since dbt v1.8, installing an adapter no longer installs `dbt-core` automatically.


		## Connecting to IBM watsonx.data Spark

		To connect dbt with watsonx.data Spark, you need to configure a profile in your `profiles.yml` file located in the `.dbt/` directory of your home folder. The following is an example configuration for connecting to IBM watsonx.data SaaS and Software instances:


		## Host parameters

		The following profile fields are required to configure watsonx.data Spark connections. For IBM watsonx.data SaaS or Software instances, you can get the `profile` details by clicking View connect details after `the query server` is in RUNNING stat, The Connection details page opens with the profile configuration.


		### Schemas and catalogs

		When selecting the catalog, ensure the user has read and write access. This selection does not limit your ability to query into the schema spcified/created but also serves as the default location for materialized `tables`, `views`, and `incremental`.


		## Additional parameters

		The following profile fields are optional to set up. They let you configure your instance session and dbt for your connection.

Add watsonx spark config and setup files #6895

Add watsonx spark config and setup files #6895

Conversation

bynist commented Feb 10, 2025

What are you changing in this pull request and why?

Checklist

welcome bot commented Feb 10, 2025

vercel bot commented Feb 10, 2025

Choose a reason for hiding this comment

DivyaLokesh Feb 12, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DivyaLokesh Feb 12, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DivyaLokesh Feb 12, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vercel bot commented Feb 18, 2025 • edited Loading

DivyaLokesh Feb 12, 2025 •

edited

Loading

DivyaLokesh Feb 12, 2025 •

edited

Loading

DivyaLokesh Feb 12, 2025 •

edited

Loading

vercel bot commented Feb 18, 2025 •

edited

Loading