Add log viewing tools

[jacobtomlinson]

Viewing init logs when debugging a failed cluster start can be a little unpleasant.

To make viewing these logs a little easier I've added a couple of CLI utilities in dask-databricks to help you navigate them.

Listing clusters

You can get a full list of available logs with the dask databricks ls <path> command where the path is the DBFS location you configured your logs to write to.

$ dask databricks logs ls dbfs:/cluster_init_logs

  Cluster                Start time          Node Count   Node IPs
 ──────────────────────────────────────────────────────────────────────────────────────
  1234-987654-a1b2c3d4   Nov 16 2023 10:36   2            10.0.0.1, 10.0.0.2

Viewing logs

Once you have your cluster ID you can view the logs from the latest launch of that cluster with dask databricks cat <path> <cluster>.

$ dask databricks logs cat dbfs:/cluster_init_logs 1234-987654-a1b2c3d4
Cluster: 1234-987654-a1b2c3d4
Start time: Nov 16 2023 10:36
10.0.0.1: Start Python bootstrap
10.0.0.1: PYSPARK_PYTHON is /databricks/python3/bin/python
...

[skirui-source]

This doesn't seem to work for me--

So while launching the cluster, I configured the cluster logs to dbfs:/dbfs/databricks/skirui/1031-230718-l2ubf858 (destination)

On my local workstation, steps:

• I have cloned this repo, conda activate dask env and pip install -e . then i run:

$dask databricks logs ls dbfs:/dbfs/databricks/skirui/1031-230718-l2ubf858/init_scripts
Traceback (most recent call last):
  File "/home/skirui/miniforge3/envs/dask/bin/dask", line 8, in <module>
    sys.exit(main())
  File "/home/skirui/miniforge3/envs/dask/lib/python3.9/site-packages/dask/__main__.py", line 7, in main
    run_cli()
  File "/home/skirui/miniforge3/envs/dask/lib/python3.9/site-packages/dask/cli.py", line 129, in run_cli
    cli()
  File "/home/skirui/miniforge3/envs/dask/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/skirui/miniforge3/envs/dask/lib/python3.9/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/skirui/miniforge3/envs/dask/lib/python3.9/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/skirui/miniforge3/envs/dask/lib/python3.9/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/skirui/miniforge3/envs/dask/lib/python3.9/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/skirui/miniforge3/envs/dask/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/skirui/miniforge3/envs/dask/lib/python3.9/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/skirui/home/skirui/dask-databricks/dask_databricks/cli.py", line 172, in ls
    clusters = _get_logs_at_path(path)
  File "/home/skirui/home/skirui/dask-databricks/dask_databricks/cli.py", line 132, in _get_logs_at_path
    for node in dbutils.fs.ls(cluster.path + "/init_scripts"):
  File "/home/skirui/miniforge3/envs/dask/lib/python3.9/site-packages/databricks/sdk/dbutils.py", line 55, in ls
    for f in self._dbfs.list(dir):
  File "/home/skirui/miniforge3/envs/dask/lib/python3.9/site-packages/databricks/sdk/mixins/files.py", line 342, in list
    for file_info in super().list(path):
  File "/home/skirui/miniforge3/envs/dask/lib/python3.9/site-packages/databricks/sdk/service/files.py", line 340, in list
    json = self._api.do('GET', '/api/2.0/dbfs/list', query=query, headers=headers)
  File "/home/skirui/miniforge3/envs/dask/lib/python3.9/site-packages/databricks/sdk/core.py", line 1143, in do
    return retryable(self._perform)(method,
  File "/home/skirui/miniforge3/envs/dask/lib/python3.9/site-packages/databricks/sdk/retries.py", line 47, in wrapper
    raise err
  File "/home/skirui/miniforge3/envs/dask/lib/python3.9/site-packages/databricks/sdk/retries.py", line 29, in wrapper
    return func(*args, **kwargs)
  File "/home/skirui/miniforge3/envs/dask/lib/python3.9/site-packages/databricks/sdk/core.py", line 1235, in _perform
    raise self._make_nicer_error(response=response, **payload) from None
databricks.sdk.errors.mapping.NotFound: No file or directory exists on path dbfs:/dbfs/databricks/skirui/1031-230718-l2ubf858/init_scripts/1031-230718-l2ubf858_10_59_129_214/init_scripts.

Whereas running similar command with Databricks CLI yields:

databricks fs ls dbfs:/dbfs/databricks/skirui/1031-230718-l2ubf858/init_scripts
1031-230718-l2ubf858_10_59_137_53
1031-230718-l2ubf858_10_59_143_100
1031-230718-l2ubf858_10_59_158_103

Also. does this tool depend on an actively running cluster to work? jw

[jacobtomlinson]

I think you need to just run dask databricks logs ls dbfs:/dbfs/databricks/skirui which is the same value I expect you configured when creating the cluster. I should probably update this to allow specifying the full path too.