Browser overhead for overlays should be reduced

[tokee]

The overlays-branch loads the full amount of metadata into memory, which means high memory overhead for large (100K+ nodes) structures and long startup time. A more scalable approach would be to store expanded metadata in chunks of 1000 nodes or so and fetch them only when needed.

The current full metadata (linked.js) for a sample node in the branch overlay is

{d:"bbc.co.uk", fs:12, x:260.00223, y:-341.51917, r:5.0, in:"7(167.972244,-325.335693~230.078674,-352.999908#c0c0c0);143(178.626465,-311.485352~230.768097,-347.046112#c0c0c0)", out:""}

where both in and out can be quite large. The size of the average entry for a node in a 200K node sample graph was 500 bytes.

The minimum needed for supporting clicking and searching is

{d:"bbc.co.uk", x:260.00223, y:-341.51917, r:5.0}

which is closer to 50 bytes. Roughly speaking, switching to on demand loading of extra metadata for overlay generation would mean a 10× increase in maximum scale, measured in node count.

[tokee]

Unfortunately this collides with shortest path discovery, which needs the links readily available. With this, the minimum JSON blob would be

{d:"bbc.co.uk", x:260.00223, y:-341.51917, r:5.0, in:[7,143], out:[]}

The auxilary data would thus be

{fs:12, inx:"167.972244,-325.335693~230.078674,-352.999908#c0c0c0;178.626465,-311.485352~230.768097,-347.046112#c0c0c0", outx:""}