Standardizing server_side_parameters across connection methods

[Fleid]

This is a trendy topic, for sure :)

What we have right now

dbt-spark supports 4 connection methods, only one has server_side_parameters : ODBC.

      server_side_parameters:
        # cluster configuration parameters, otherwise applied via `SET` statements
        # for example:
        # "spark.databricks.delta.schema.autoMerge.enabled": True

For Thrift, Http and Session, there is no native way to provide these parameters. The usual workaround is to do so via hooks.

There are 2 common asks from here:

• Extend support as-is to Thrift and Session (also here and there). Nobody said anything about Http, but since we're here?
• Offer an alternative method to provide these parameters, via a string instead of a dictionary, so it can be loaded from environment variables at run time

I just want to get everybody in the same room to make sure I'm understanding the landscape right.

Ideal state

Across all 4 connection methods, dbt-spark supports 2 ways to provide Spark configuration options:

• server_side_parameters, as a dictionary
• spark_config_string, as a string (not sure about that name though)

Does that make sense? Everybody agrees here?

[hanna-liashchuk]

Having an option to pass parameters via one string is a good idea, I can see use cases for this, so LGTM :)
One thing to think about - if both specified, what behavior we expect?

[vinhnemo]

I think this update will be worth it, using spark_config_string is really working in my case. Hope we can go with it

[akashrn5]

We have to basically look at both spark properties with support dynamically setting and some has to be set before application even starts. So maintaining multiple configs doesn't make sense. But we have to even consider the situation where there can be multiple project refer to same profile, so keeping at project is always preferrable, and my solution works more in that way. Please have a look once before directly deciding.

[JCZuurmond]

@akashrn5, you raise a good point. I rarely set Spark configuration during job execution; exceptions might be configurations required for reading and writing data.

@Fleid : Do we know from dbt-spark users which configuration settings they would like to set?

[Fokko]

From the Iceberg perspective, this would be mostly catalog configuration, and packages and extensions to add Iceberg support to Spark.

[JCZuurmond]

I prefer the server_side_parameters, as a dictionary over the string

[Fokko]

I agree. I'd prefer to leave out the string option to avoid confusing users.

[akashrn5]

i agree with you @JCZuurmond for going with server_side_parameters as a dictionary, but this will not satisfy all use case. Also asi said multiple projects can refer to same profile so we cant keep at profile level assuming the multiple projects need same configs. So it has be at the project level and also it doesnt help in the case where lets say i want to configure some memory parameters like driver and executor memory which i can't and also this cannot be same for multiple projects. So i guess we have to support server_side_parameters at global level and then we have to introduce a config lets say user_configs which could be dictionary and then this can be used for any custom properties by user and can be used in all places irrespective of engines.

So we can have a priority like profile level server_side_prarameters > user_configs at project level > model_config as model level. This makes more sense and this could satisfy all the use cases. What do you guys this about this? Please let me know.

[petlevyi]

I also prefer the server_side_parameters, as a dictionary over the string.
I see this is ready to merge.
When this functionality will be added to the release?

[petlevyi]

I also prefer the server_side_parameters, as a dictionary over the string.
I see this is ready to merge.
When this functionality will be added to the release?

[JCZuurmond]

Hi all, @Fleid,

@Fokko and I sent today in implementing the server_side_parameters. An issue that remains is to define the server_side_parameters on the project level. My recommendation is to close this discussion and all related issues and PRs except the before mentioned one, and continue the conversation on defining the sever_side_parameters in the dbt_project.yml there.

The following issues and PRs are ready to be resolved and merged:

Description	Issue	Pull request
server_side_parameters support for Session	#690	#823
server_side_parameters support for HTTP	#824	#825
server_side_parameters support for Thrift	#387	#577
Force server_side_parameters keys and values to be strings	#826	#827
Add server_side_parameters to dbt docs		here

The following issue and PR can be closed:

Description	Issue	Pull request	Remark
spark_conf_string support for Thrift	#590	#591	Preference is the server_side_parameters above

[akashrn5]

@Fleid @JCZuurmond before closing this lets also put conclusion on bringing the server_side_parameters to project level based on above discussion and use cases and setting priority of configs at model_level>project_level>profile_level. Also i want to work on this part, so if we conclude on this, i can work on this and raise PR for the same. Thank you.

[JCZuurmond]

Hi @akashrn5, I recommend to continue that conversation in #529, but if @Fleid would like to keep it here, then we do that.