Update from https://github.com/treeverse/patura/commit/e68fabdbe04875…

…1cc1af54c0f5aa66c2ba9254f6
treeverse · Nov 20, 2024 · fc6f3a6 · fc6f3a6
1 parent 4ebda01
commit fc6f3a6
Show file tree

Hide file tree

Showing 2 changed files with 23 additions and 1 deletion.
diff --git a/assets/js/search-data.json b/assets/js/search-data.json
@@ -134,7 +134,7 @@
   },"19": {
     "doc": "CLI Reference",
     "title": "CLI Configuration",
-    "content": "lakefs-databricks can be configured using either a YAML file or environment variables. Configuration file . By default, lakefs-databricks will look for a file named .lakefs-databricks.yaml in your home directory, but the configuration file can be specified by passing the -c (or --config) argument to lakefs-databricks (see below). | AWS | Azure | . unity: url: &lt;unity_url&gt; # Your Databricks workspace URL. e.g. https://dbc-&lt;workspace-id&gt;.cloud.databricks.com token: &lt;unity_token&gt; # The Personal access token you created during the Databricks Workspace preparation lakefs: secret_access_key: &lt;lakefs_secret_access_key&gt; # The secret_access_key you created during lakeFS Cloud setup access_key_id: &lt;lakefs_access_key_id&gt; # The access_key_id you created during lakeFS Cloud setup url: &lt;lakefs_endpoint_url&gt;/api/v1 # The Url of your lakeFS Cloud instance e.g. https://example-org.us-east-1.lakefscloud.io databricks_storage: type: s3 s3: access_key_id: &lt;aws_access_key_id&gt; # the access_key you created on the previous step secret_access_key: &lt;aws_secret_access_key&gt; # the secret_access_key you created on the previous step region: &lt;aws_region&gt; . unity: url: &lt;unity_url&gt; # Your Databricks workspace URL. e.g. https://dbc-&lt;workspace-id&gt;.cloud.databricks.com token: &lt;unity_token&gt; # The Personal access token you created during the Databricks Workspace preparation lakefs: secret_access_key: &lt;lakefs_secret_access_key&gt; # The secret_access_key you created during lakeFS Cloud setup access_key_id: &lt;lakefs_access_key_id&gt; # The access_key_id you created during lakeFS Cloud setup url: &lt;lakefs_endpoint_url&gt;/api/v1 # The Url of your lakeFS Cloud instance e.g. https://example-org.us-east-1.lakefscloud.io databricks_storage: type: azure azure: client_id: &lt;client_id&gt; # The client_id from the lakeFS Cloud Terraform output client_secret: &lt;client_secret&gt; # The client_secret from the lakeFS Cloud Terraform output tenant_id: &lt;tenant_id&gt; # The tenant_id from the lakeFS Cloud Terraform output . Environment variables . | AWS | Azure | . LAKEFS_DATABRICKS_UNITY_URL=&lt;unity_url&gt; LAKEFS_DATABRICKS_UNITY_TOKEN=&lt;unity_token&gt; LAKEFS_DATABRICKS_LAKEFS_URL=&lt;lakefs_installation_url&gt;/api/v1 LAKEFS_DATABRICKS_LAKEFS_ACCESS_KEY_ID=&lt;lakefs_access_key_id&gt; LAKEFS_DATABRICKS_LAKEFS_SECRET_ACCESS_KEY=&lt;lakefs_secret_access_key&gt; LAKEFS_DATABRICKS_DATABRICKS_STORAGE_TYPE=s3 LAKEFS_DATABRICKS_DATABRICKS_STORAGE_S3_ACCESS_KEY_ID=&lt;aws_access_key_id&gt; LAKEFS_DATABRICKS_DATABRICKS_STORAGE_S3_SECRET_ACCESS_KEY=&lt;aws_secret_access_key&gt; LAKEFS_DATABRICKS_DATABRICKS_STORAGE_S3_REGION=&lt;aws_region&gt; . LAKEFS_DATABRICKS_UNITY_URL=&lt;unity_url&gt; LAKEFS_DATABRICKS_UNITY_TOKEN=&lt;unity_token&gt; LAKEFS_DATABRICKS_LAKEFS_URL=&lt;lakefs_installation_url&gt;/api/v1 LAKEFS_DATABRICKS_LAKEFS_ACCESS_KEY_ID=&lt;lakefs_access_key_id&gt; LAKEFS_DATABRICKS_LAKEFS_SECRET_ACCESS_KEY=&lt;lakefs_secret_access_key&gt; LAKEFS_DATABRICKS_DATABRICKS_STORAGE_TYPE=azure LAKEFS_DATABRICKS_DATABRICKS_STORAGE_AZURE_CLIENT_ID=&lt;client_id&gt; LAKEFS_DATABRICKS_DATABRICKS_STORAGE_AZURE_CLIENT_SECRET=&lt;client_secret&gt; LAKEFS_DATABRICKS_DATABRICKS_STORAGE_AZURE_TENANT_ID=&lt;tenant_id&gt; . ",
+    "content": "lakefs-databricks can be configured using either a YAML file or environment variables. Configuration file . By default, lakefs-databricks will look for a file named .lakefs-databricks.yaml in your home directory, but the configuration file can be specified by passing the -c (or --config) argument to lakefs-databricks (see below). | AWS | Azure | . unity: url: &lt;unity_url&gt; # Your Databricks workspace URL. e.g. https://dbc-&lt;workspace-id&gt;.cloud.databricks.com token: &lt;unity_token&gt; # The Personal access token you created during the Databricks Workspace preparation lakefs: secret_access_key: &lt;lakefs_secret_access_key&gt; # The secret_access_key you created during lakeFS Cloud setup access_key_id: &lt;lakefs_access_key_id&gt; # The access_key_id you created during lakeFS Cloud setup url: &lt;lakefs_endpoint_url&gt;/api/v1 # The Url of your lakeFS Cloud instance e.g. https://example-org.us-east-1.lakefscloud.io databricks_storage: type: s3 s3: access_key_id: &lt;aws_access_key_id&gt; # the access_key you created on the previous step secret_access_key: &lt;aws_secret_access_key&gt; # the secret_access_key you created on the previous step region: &lt;aws_region&gt; . unity: url: &lt;unity_url&gt; # Your Databricks workspace URL. e.g. https://dbc-&lt;workspace-id&gt;.cloud.databricks.com token: &lt;unity_token&gt; # The Personal access token you created during the Databricks Workspace preparation lakefs: secret_access_key: &lt;lakefs_secret_access_key&gt; # The secret_access_key you created during lakeFS Cloud setup access_key_id: &lt;lakefs_access_key_id&gt; # The access_key_id you created during lakeFS Cloud setup url: &lt;lakefs_endpoint_url&gt;/api/v1 # The Url of your lakeFS Cloud instance e.g. https://example-org.us-east-1.lakefscloud.io databricks_storage: type: azure azure: client_id: &lt;client_id&gt; # The client_id from the lakeFS Cloud Terraform output client_secret: &lt;client_secret&gt; # The client_secret from the lakeFS Cloud Terraform output tenant_id: &lt;tenant_id&gt; # The tenant_id from the lakeFS Cloud Terraform output . Environment variables . | AWS | Azure | . LAKEFS_DATABRICKS_UNITY_URL=&lt;unity_url&gt; LAKEFS_DATABRICKS_UNITY_TOKEN=&lt;unity_token&gt; LAKEFS_DATABRICKS_LAKEFS_URL=&lt;lakefs_installation_url&gt;/api/v1 LAKEFS_DATABRICKS_LAKEFS_ACCESS_KEY_ID=&lt;lakefs_access_key_id&gt; LAKEFS_DATABRICKS_LAKEFS_SECRET_ACCESS_KEY=&lt;lakefs_secret_access_key&gt; LAKEFS_DATABRICKS_DATABRICKS_STORAGE_TYPE=s3 LAKEFS_DATABRICKS_DATABRICKS_STORAGE_S3_ACCESS_KEY_ID=&lt;aws_access_key_id&gt; LAKEFS_DATABRICKS_DATABRICKS_STORAGE_S3_SECRET_ACCESS_KEY=&lt;aws_secret_access_key&gt; LAKEFS_DATABRICKS_DATABRICKS_STORAGE_S3_REGION=&lt;aws_region&gt; . LAKEFS_DATABRICKS_UNITY_URL=&lt;unity_url&gt; LAKEFS_DATABRICKS_UNITY_TOKEN=&lt;unity_token&gt; LAKEFS_DATABRICKS_LAKEFS_URL=&lt;lakefs_installation_url&gt;/api/v1 LAKEFS_DATABRICKS_LAKEFS_ACCESS_KEY_ID=&lt;lakefs_access_key_id&gt; LAKEFS_DATABRICKS_LAKEFS_SECRET_ACCESS_KEY=&lt;lakefs_secret_access_key&gt; LAKEFS_DATABRICKS_DATABRICKS_STORAGE_TYPE=azure LAKEFS_DATABRICKS_DATABRICKS_STORAGE_AZURE_CLIENT_ID=&lt;client_id&gt; LAKEFS_DATABRICKS_DATABRICKS_STORAGE_AZURE_CLIENT_SECRET=&lt;client_secret&gt; LAKEFS_DATABRICKS_DATABRICKS_STORAGE_AZURE_TENANT_ID=&lt;tenant_id&gt; . Databricks Environment variables . lakefs-databricks supports standard Databricks environment variables to simplify setup by reusing existing configurations. If these variables are already set for Databricks, lakefs-databricks will automatically use them, reducing redundant configuration. Supported Variables . | DATABRICKS_HOST: Your Databricks workspace URL (e.g., https://dbc-.cloud.databricks.com). Can be overridden by: LAKEFS_DATABRICKS_UNITY_URL or unity.url in the config file. | DATABRICKS_TOKEN: Your Databricks Personal Access Token (PAT). Can be overridden by: LAKEFS_DATABRICKS_UNITY_TOKEN or unity.token in the config file. | . ",
     "url": "/cli-reference.html#cli-configuration",
 
     "relUrl": "/cli-reference.html#cli-configuration"

diff --git a/cli-reference.html b/cli-reference.html
@@ -547,6 +547,28 @@ <h4 id="environment-variables">
 </code></pre></div>    </div>
   </div>
 </div>
+<h4 id="databricks-environment-variables">
+
+
+    <a href="#databricks-environment-variables" class="anchor-heading"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Databricks Environment variables
+
+
+</h4>
+
+
+<p>lakefs-databricks supports standard Databricks environment variables to simplify setup by reusing existing configurations. If these variables are already set for Databricks, lakefs-databricks will automatically use them, reducing redundant configuration.</p>
+<h5 id="supported-variables">
+
+
+    <a href="#supported-variables" class="anchor-heading"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Supported Variables
+
+
+</h5>
+
+<ul>
+  <li>DATABRICKS_HOST: Your Databricks workspace URL (e.g., https://dbc-<workspace-id>.cloud.databricks.com). Can be overridden by: LAKEFS_DATABRICKS_UNITY_URL or unity.url in the config file.</workspace-id></li>
+  <li>DATABRICKS_TOKEN: Your Databricks Personal Access Token (PAT). Can be overridden by: LAKEFS_DATABRICKS_UNITY_TOKEN or unity.token in the config file.</li>
+</ul>
 <h2 id="command-reference">