Support additional auth options for pins on Databricks

[juliasilge]

Originally posted by @jmbarbone in #839 (comment):

I've ran into a small issue within my workflow. We track multiple Databricks profiles with separate hosts and tokens configured in a .databrickscfg file. The envvars are then only temporarily set, so they aren't persistent in the session. This does lead to a less than intuitive error when trying to write:

some_custom_board_function <- function(profile, ...) {
  # simplified
  cfg <- ini::read.ini("~/.databrickscfg")
  profile <- mark::match_param(profile, names(cfg))
  config <- cfg[[profile]]
  withr::local_envvar(c(
    DATABRICKS_HOST = config$host, 
    DATABRICKS_TOKEN = config$token
  ))
  board_databricks(...)
  board
}

board <- some_custom_board_function()
pin_write(board, mtcars)
#> Using `name = 'mtcars'`
#> Guessing `type = 'rds'`
#> Error in `purrr::keep()`:
#> ℹ In index: 1.
#> ℹ With name: message.
#> Caused by error in `.x$is_directory`:
#> ! $ operator is invalid for atomic vectors

We have custom board creating and pin writing wrappers; everything still works with a few extra steps.

It would be nice to look for these settings in .databrickscfg neither explicitly passed or when those envvars are not set. This is currently how Databricks handles authentication across tools: Default methods for client unified authentication.

Still very excited for this update and looking forward to using it more.

[JavOrraca]

This is an ongoing issue for me as well that my company has multiple Databricks hosts requiring management of multiple tokens. A token argument in board_databricks() would be much appreciated.

And FWIW, I recommend against using a .databrickscfg if it's an end user interacting with Databricks UC Volumes (not a machine). The Databricks CLI required the use of .databrickscfg but the Databricks REST API is more flexible and has much more functionality compared to the super limited CLI.