After running some notebooks cells gives an error "TypeError: Cannot read properties of undefined (reading 'status')"

[ktakanopy]

Describe the bug
After running some notebooks cells gives an error "TypeError: Cannot read properties of undefined (reading 'status')"

To Reproduce
Steps to reproduce the behavior:

1. Download Databricks extension and databricks power tools
2. Connect to a cluster
3. Create a notebook.ipynb
4. Run some cells in the notebook connected to a cluster.
5. You will receive an error "TypeError: Cannot read properties of undefined (reading 'status')"

Screenshots

System information:

1. MacOs Sonoma 14.5
2. Databricks Extension Version: 1.3.1

Databricks Extension Logs

• Which log file do I need to send?

Additional context

[kupalinka-lis]

I have the same issue with the same system info

[gbrueckl]

Please check the logs under Outputs -> databricks.vscode and post the here (remove sensitive information if necessary) Get Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Nika ***@***.***> Sent: Thursday, July 18, 2024 11:05:33 AM To: paiqo/Databricks-VSCode ***@***.***> Cc: Subscribed ***@***.***> Subject: Re: [paiqo/Databricks-VSCode] After running some notebooks cells gives an error "TypeError: Cannot read properties of undefined (reading 'status')" (Issue #196) I have the same issue with the same system info — Reply to this email directly, view it on GitHub<#196 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AC2VO4S4FBREVK5W3YB4TUDZM6AN3AVCNFSM6AAAAABLBPN4LWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZWGAYDGMZSHE>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[ktakanopy]

Hi @gbrueckl here the logs (I think it's the correct, please tell me if not)

	at parseErrorFromResponse (/path/to/.vscode/extensions/databricks.databricks-1.3.1-darwin-x64/out/extension.js:1855:14)
	at fn (/path/to/.vscode/extensions/databricks.databricks-1.3.1-darwin-x64/out/extension.js:2386:63)
	at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
	at async retry3 (/path/to/.vscode/extensions/databricks.databricks-1.3.1-darwin-x64/out/extension.js:1735:20)
	at async ApiClient8.request (/path/to/.vscode/extensions/databricks.databricks-1.3.1-darwin-x64/out/extension.js:2362:26)
	at async ClustersService._get (/path/to/.vscode/extensions/databricks.databricks-1.3.1-darwin-x64/out/extension.js:27967:16)
	at async ClustersService.get (/path/to/.vscode/extensions/databricks.databricks-1.3.1-darwin-x64/out/extension.js:27978:32)
	at async _Cluster.refresh (/path/to/.vscode/extensions/databricks.databricks-1.3.1-darwin-x64/out/extension.js:116965:20)
	at async Timeout._onTimeout (/path/to/.vscode/extensions/databricks.databricks-1.3.1-darwin-x64/out/extension.js:119248:7)
	```

[gbrueckl]

so the error message you posted comes from the original Databricks extension [databricks.databricks](https://marketplace.visualstudio.com/items?itemName=databricks.databricks) and is not related to this extension.
Can you check which Kernel you are using for your Notebook?

the logs related to this extension can be found here:

[rgalljamov]

I have the same issue on Windows and get the following logs (@gbrueckl, thank you for explaining where to find the right logs):

POST 1.2/commands/execute --> {"clusterId":"0326-150543-8brhnzrc","language":"python","contextId":"2674181048795868090","command":"import numpy as np\n\nimport matplotlib.pyplot as plt\n\n# %matplotlib inline\n\n# Generate random values\nx = np.random.rand(1000)\ny = np.random.rand(1000)\n\n# Create scatter plot\nplt.scatter(x, y, alpha=0.0125)\n\n# Add labels and title\nplt.xlabel('X')\nplt.ylabel('Y')\nplt.title('Random Scatter Plot')\n\n# Display the plot\nplt.show()"}
Response: {"error_code":403,"message":"JWT has expired"} ...
GET 1.2/commands/status --> {"clusterId":"0326-150543-8brhnzrc","contextId":"2674181048795868090"}
GET 1.2/commands/status failed! ERROR: " (403) Forbidden
ERROR: Error - Forbidden

It always happens when you pause programming for some time and come back later. Restarting VS Code solves the issue, but it would be great if the restart wouldn't be required.

Thank you for the great extension and in advance for the support.

[gbrueckl]

@rgalljamov so this error is related to this extension and i can have a look.
its basically complaining about an expired JWT token which I guess comes from the OAuth that used behind the scenes. You are using the Azure Connection Manager, right?

the token probably expires after 1 hour and we dont have any refresh in place yet as this only happens for the Azure Connection Manager
but we should be able to fix this :)

[rgalljamov]

Thank you very much for the swift reply and support @gbrueckl!

I'm using the official Databricks Extension in the newest version and OAuth U2M authentification. Therefore, your extension worked right away after installation, which was a great positive surprise. The expiration duration might be below an hour but I have not tested it yet.

[kupalinka-lis]

Just checked the error logs - I have the same JWT issue

Response: {"error_code":403,"message":"JWT has expired"} ...
GET 1.2/commands/status --> {"clusterId":"...","contextId":"..."}
GET 1.2/commands/status failed! ERROR: " (403) Forbidden
ERROR: Error - Forbidden

Thanks for looking on it!

[sreekanth9999]

were anyone able to fix this issue?

[gbrueckl]

I just published v2.3.5 to the marketplace which should hopefully fix this issue
we are now updating the JWT token every 30 minutes

[jaakko-paavola]

I get a slightly different error from trying to execute cells occurring every 20 mins or so:
"TypeError: Cannot read properties of undefined (reading 'error')"
and the output log for paiqo.databricks-vscode has:
"""
OST 1.2/commands/execute --> {"clusterId":"1111-222222-3333333","language":"python","contextId":"4444444444444444444","command":"print(df)"}
ERROR: p - Invalid response body while trying to fetch https://xxxxxxxx.cloud.databricks.com/api/1.2/commands/execute: aborted
"""

It may be giving me this error for several minutes for all the cells, but eventually it will recover, maybe following the appearance of this in the output log:
"""
Refreshing Databricks API Headers ...
"""

I remember seeing a text entry somewhere in the output log saying the headers are refreshed every 1800 seconds. Is there some way to make this more frequent?

When the error happens, I will also get:
"""
Response: {"clusters":[{"cluster_id":"...
Response: {"jobs":[{"job_id":"...
GET 2.1/jobs/runs/list --> {"job_id":null,"active_only":false,"completed_only":false,"offset":0,"limit":25,"expand_tasks":true}
GET 2.0/clusters/list --> null
GET 2.1/jobs/runs/list failed! ERROR: " (400) Bad Request
ERROR: Error - Bad Request
"""
when it tries to do the "remote state refresh" (I believe).

[gbrueckl]

may I ask if other features of the original Databricks extension still work at the time when you face those issues?
e.g. choosing a different cluster, starting/stopping a cluster, etc.

just want to make sure its not related to the token that we get from the original Databricks extension

[gbrueckl]

ok, did some more investigation:
the token we get from the original Databricks extension is valid for 1h and is refreshed automatically before it expires
as of now, I simply get the current token every 30 minutes
now it can happen that I get the token right before it times out - meaning for the next 30 minutes I will use an invalid token and all API calls fail
I think this is what happens in your case

I will see what I can do

[jaakko-paavola]

may I ask if other features of the original Databricks extension still work at the time when you face those issues? e.g. choosing a different cluster, starting/stopping a cluster, etc.

just want to make sure its not related to the token that we get from the original Databricks extension

With the original Databricks extension I never get to create a Databricks project folder successfully, because at the end of the setup process I bump into an error due to group policy (company computer) when the extension tries to execute some .exe file. So I don't get to have any functionality available through the original Databricks extension.

[gbrueckl]

is there actually a specific reason why you cannot/do not want to use notebooks?

[jaakko-paavola]

ok, did some more investigation: the token we get from the original Databricks extension is valid for 1h and is refreshed automatically before it expires as of now, I simply get the current token every 30 minutes now it can happen that I get the token right before it times out - meaning for the next 30 minutes I will use an invalid token and all API calls fail I think this is what happens in your case

I will see what I can do

After a couple more days of experience with this problem, I can say this:

• The error happens in an instant; I may have just executed a cell successfully a second ago but then the next one throws the error.
• The error persists for what I would estimate is 3-5 mins. I can try to time the exact duration next time I encounter this, if that would be useful information. Perhaps it's the same duration every time, but not sure.
• So most likely the cause is not the 30-minute token then.

[gbrueckl]

in the new solution I would decode the token to see when it expires
then wait until it expiry time and request a new token from the original extension which seems to update the token a couple of seconds before it expires
this should hopefully solve the problem but I still need some more testing

[jaakko-paavola]

is there actually a specific reason why you cannot/do not want to use notebooks?

I think you meant to post this here #202. I have at least three reasons to prefer the interactive window over notebook:

1. VsCode is much snappier/more responsive to use when dealing with a .py file instead of a .ipynb.
2. Debugging interactively is more convenient when you can send the current line or anything you paint with the cursor to be executed. Much faster than adding a cell above or below and then copying those things there and then executing the cell, and then removing the redundant cell.
3. I started coding data stuff with RStudio, so I got used to the interactive window.

[gbrueckl]

can you try this build: https://gbpublic001.blob.core.windows.net/files/databricks-vscode-2.3.6.vsix
it should fix the expiring token issue

[jaakko-paavola]

I can report that I no longer get this error. But the fix to the issue was not the build you made available above, because I still got the error after installing it. Instead, I think I managed to fix the error with proxy configurations in VsCode settings that made things work with my company's proxy server. Anyway, thanks for the help for troubleshooting and making the Databricks kernel available in the interactive window, which has a huge benefit for me!

[gbrueckl]

closing as it seems to work now