Unable to install and init hail in databricks on spark 3.5.0/scala 2.12.15 (15.4 LTS) when run in notebook

[GeorgeNikitinNV]

What happened?

Hail is installed:

import hail as hl
hl.version()

results in '0.2.134-952ae203dbbe'

However this throws:

import hail as hl
hl.init()

What interesting is is that I am able to run it from databricks web terminal in python or ipython without any problems.

I have tried building it using init scripts and docker images, switching JAVA versions etc... Nothing helps, could someone please help me out with this?

Version

0.2.134

Relevant log output

TypeError: 'JavaPackage' object is not callable
File <command-4227511648738071>, line 2
      1 import hail as hl
----> 2 hl.init()
File /databricks/python/lib/python3.11/site-packages/hail/backend/spark_backend.py:130, in SparkBackend.__init__(self, idempotent, sc, spark_conf, app_name, master, local, log, quiet, append, min_block_size, branching_factor, tmpdir, local_tmpdir, skip_logging_configuration, optimizer_iterations, gcs_requester_pays_project, gcs_requester_pays_buckets, copy_log_on_error)
    128     jhc = hail_package.HailContext.getOrCreate(jbackend, branching_factor, optimizer_iterations)
    129 else:
--> 130     jbackend = hail_package.backend.spark.SparkBackend.apply(
    131         jsc,
    132         app_name,
    133         master,
    134         local,
    135         log,
    136         True,
    137         append,
    138         skip_logging_configuration,
    139         min_block_size,
    140         tmpdir,
    141         local_tmpdir,
    142         gcs_requester_pays_project,
    143         gcs_requester_pays_buckets,
    144     )
    145     jhc = hail_package.HailContext.apply(jbackend, branching_factor, optimizer_iterations)
    147 self._jsc = jbackend.sc()