Cannot use relative files

[sareini]

In databricks, I use git syncing when developing. Inside a directory, I have a config file my job can read and use to load some runtime variables.

Ex:

db_job = configparser.ConfigParser()
db_job.read('databricks.ini')

When using the extension, this loads nothing and it cannot find the file at the same directory level

[gbrueckl]

where exactly would that databricks.ini file reside?
inside your workspace (using Files in Repos) or on your DBFS?

if it is on DBFS, this will not work as your notebook is in the workspace and is not aware of the current path on DBFS what might be the case if you execute it as a Python job

if it is on the workspace, it currently also will not work but I could check to see if this is possible

just let me know what your exact scenario is

[sareini]

Its currently under a repo workspace.

So I have the git repo sync'd w/ databricks. And the structure of the folder is

my_directory/
my_directory/databricks.ini
my_directory/job1
...

where job1 is trying to read from databricks.ini. Something else I noticed is that I cant open that databricks.ini file via the plugin, but can the job1 file. Not sure if that is helpful!

[gbrueckl]

Non-code files like your .ini uses a different API to up/download then code files, but this feature will be added soon

Regarding your problem: does that same code work if you execute it via the Databricks Web UI?

[sareini]

Yep, works in the databricks UI! We have that files in repo feature enabled

[gbrueckl]

What library is that ConfigParser using to read the file?
Some regular python code like os?

[sareini]

Yep! And some others, fwiw it should come w/ any python and just uses some regular default python packages https://docs.python.org/3/library/configparser.html#module-configparser

Grabbed these from source code

from collections.abc import MutableMapping
from collections import ChainMap as _ChainMap
import functools
import io
import itertools
import os
import re
import sys
import warnings

[gbrueckl]

I think I found a fix which will be implemented with the next release!

[gbrueckl]

the issue should be fixed in this release, can you please try it and report back - thanks!
https://github.com/paiqo/Databricks-VSCode/releases/tag/v1.2.0

Note: it is not yet published to the VSCode gallery

[sareini]

It still isnt working for me. I hover over databricks.ini and it shows [Online Only].

I added

  "databricks.connection.default.exportFormats": {
     ...
    "Ini": ".ini"
  }

but it didnt help

[gbrueckl]

so the original issue should be fixed - which was executing your code from the first post

to modify the Files in Repo - this is currently only working if you open it via dbws:/ from your VSCode solution explorer
you can simply click the Orange icon in your Workspace Browser to add the dbws:/ to your VSCode solution explorer

browsing to your file from there should allow you to open and edit it

[sareini]

Yes that does work! Thank you so much! The notebook can run and load those files.

I did have to click Add Databricks to VSCode workspace for the code to execute. Without it, it still couldnt find the file.

[gbrueckl]

it should also work if you run it via the Workspace browser (basically running the local file)
of course, in this case the reference file also has to exist online in the databricks workspace as the code is running there and looking for the file there

can you test this again please as it worked in my tests

[sareini]

I opened the notebook via the workspace tab. I can see the ini file in the same directory, but it still cant read it. Unless I add it to the workspace.

If it could read it, the output would be

['test.ini']

[gbrueckl]

Are you using the Databricks kernel?!? Or you r local one? Get Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Muhammad Sareini ***@***.***> Sent: Friday, October 21, 2022 10:49:52 PM To: paiqo/Databricks-VSCode ***@***.***> Cc: Gerhard Brueckl ***@***.***>; Comment ***@***.***> Subject: Re: [paiqo/Databricks-VSCode] Cannot use relative files (Issue #101) [image]<https://user-images.githubusercontent.com/26151060/197286160-48d9df9a-9ebe-43e1-9892-f74fa2c0401e.png> I opened the notebook via the workspace tab. I can see the ini file in the same directory, but it still cant read it. Unless I add it to the workspace — Reply to this email directly, view it on GitHub<#101 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AC2VO4UHOHFCEJ75VMER4O3WEL6XBANCNFSM6AAAAAARHFLWZI>. You are receiving this because you commented.Message ID: ***@***.***>

[sareini]

I started a databricks cluster and specified that as the kernel

[gbrueckl]

ok, I think I found the issue
it further seems that DBR 11+ is required for this to work in the end

[gbrueckl]

just published v1.2.1 to the gallery!
please provide some final feedback

[sareini]

Downloaded the new version, double checked I have DBR 11+, and everything is working as expected! Thank you so much for the quick turnaround!