Fully embracing Jupyter Widgets / Jupyter comm

[manzt]

There are a lot of issues caused by trying to configure the servir component of higlass-python, in particular for remote environments. This stems from diverse requirements for the remote web-server, leading to litany of issues for end users (bad). We are running into the case where special config needs to happen to support this kind of dynamic data fetching, and it's really challenging to support these environments. #142 #140

But, Jupyter Widgets exist.

In theory, we could bypass all of this garbage by implementing a client-side data fetcher based on Jupyter comms (i.e., anywidget) in _widget.js. This would be able to re-use the Jupyter connection for sending tile data, rather configuring and running a background thread to host data.

Comms will be consistent across all notebooks environments, and require no extra configuration by end users. As long as they are able to load the widget, data loading should just work. This also comes at the benefit of being able to drop servir as required dependency (and jupyter-server-proxy). We can also get rid of all the custom server stuff.

[manzt]

@pkerpedjiev what would it take to tell higlass fronted to use a different DataFetcher for certain tracks? (from a viewConfig perspective).

[manzt]

In short, for "local" tilesets, rather than starting a background thread and temporary web-server, we reuse the Jupyter comm for displaying the widget to also send/receive requests for tiles. No more HTTP requests, just custom messages.

[manzt]

Ideally we could just extend DataFetcher from HiGlass, but its logic is so coupled to fetch. Here's the general idea.

import hglib from "https://esm.sh/[email protected]?deps=react@17,react-dom@17,pixi.js@6";
window.higlassDataFetchersByType = window.higlassDataFetchersByType || {};

/**
 * Detects server: 'jupyter', and creates a custom data entry for it.
 * @example { server: "jupyter", tilesetUid: "aa" } -> { tilesetUid: "aa", data: { type: "jupyter-<id>", tilesetUid: "aa" } }
 */
function resolveJupyterServer(viewConfig, dataFetcherId) {
  let copy = JSON.parse(JSON.stringify(viewConfig));
  for (let view of copy.views) {
    for (let track of Object.values(view.tracks).flat()) {
      if (track?.server === "jupyter") {
        delete track.server;
        track.data = track.data || {};
        track.data.type = dataFetcherId;
        track.data.tilesetUid = track.tilesetUid;
      }
    }
  }
  return copy;
}

class JupyterDataFetcher {
  #model;
  constructor(model, dataConfig) {
    this.#model = model;
    this.dataConfig = dataConfig;
  }
  async tilesetInfo(cb) {
    // get tilesetInfo with this.#model
  }
  async fetchTilesDebounced(cb, tileIds) {
    // get tileData with this.#model
  }
}

export default () => {
  let id = globalThis.crypto.randomUUID().split("-")[0];
  let dataFetcherId = `jupyter-${id}`;
  return {
    async initialize({ model }) {
      window.higlassDataFetchersByType[dataFetcherId] = {
        name: dataFetcherId,
        dataFetcher: class extends JupyterDataFetcher {
          constructor(dataConfig) {
            super(model, dataConfig);
          }
        },
      };
    },
    async render({ model, el }) {
      let viewconf = resolveJupyterServer(model.get("_viewconf"), dataFetcherId);
      let options = model.get("_options") ?? {};
      let api = await hglib.viewer(el, viewconf, options);
    },
  };
};

then to use on the Python side:

hg.track(type="heatmap", server="jupyter", tilesetUid="aaa")

We can can just have a global weakmap of created tilesets that the HiGlassWidget can call out to to respond to the font end.

[etowahadams]

This seems like it would be quite useful! I like the idea not having to do all of the servir stuff.

what would it take to tell higlass fronted to use a different DataFetcher for certain tracks? (from a viewConfig perspective).

If I understand what you're asking, I think you can register a new plugin datafetcher and use type: 'custom-datafetcher-name' in the data config? There's a function which switches datafetchers based on the type property in the data config.

https://github.com/higlass/higlass/blob/335f4eea6445d07baa5eda2e66077dca461b7777/app/scripts/plugins/get-data-fetcher.js#L27-L35

[manzt]

There's a function which switches datafetchers based on the type property in the data config.

Yup, exactly!