Weird non-performant scaling of add_edge!

[agchesebro]

The new add_edge! functionality has some puzzling performance issue with large graphs. For example, constructing a system of 300 Izhikevich neurons with the old syntax:

using Neuroblox, Random, Distributions, OrdinaryDiffEq

η_dist = Cauchy(0.12, 0.02)
N = 300
w = 1

blox = [IzhikevichNeuron(name=Symbol("Izh$i"), η=rand(η_dist), sⱼ=1.2308/N) for i in 1:N]

g = MetaDiGraph()
add_blox!.(Ref(g), blox)

@time for i ∈ 1:N
    for j ∈ 1:N
        add_edge!(g, i, j, Dict(:weight => w))
    end
end

takes ~0.2 seconds for the loop my computer, while with the new syntax:

g = MetaDiGraph()
add_blox!.(Ref(g), blox)

@time for i ∈ blox
    for j ∈ blox
        add_edge!(g, i => j; weight=w)
    end
end

takes ~16s. This scales with graph size (e.g., 200 is about the same for the first loop, but ~5s for the second).

[harisorgn]

I am not very surprised by this. The new syntax does more work internally; it's the find_blox step that probably causes the time increase as it parses the graph for each blox in each pair. All in all, it sacrifices performance for convenience.

We might be able to make it faster, although it's mainly using functions from Graphs. A better solution which we discussed with @MasonProtter some time in the past is to have our own custom graph type and make searching over it more efficient.

[agchesebro]

Thank you! Makes sense overall - for larger systems of neurons (~1k) I'll stick to the old syntax for performance then. One additional wrinkle: if you remove add_blox!.(Ref(g), blox) from the new syntax version, it throws a ExtraEquationsSystemException. So for example:

using Neuroblox, Random, Distributions, OrdinaryDiffEq

η_dist = Cauchy(0.12, 0.02)
N = 5
w = 1

blox = [IzhikevichNeuron(name=Symbol("Izh$i"), η=rand(η_dist), sⱼ=1.2308/N) for i in 1:N]

g = MetaDiGraph()
for i ∈ blox
    for j ∈ blox
        add_edge!(g, i => j; weight=w)
    end
end
@named sys = system_from_graph(g)

will say the system is unbalanced, but if you include the add_blox! before the edge connections it compiles just fine.

[harisorgn]

Good catch @agchesebro , there was a duplication of a blox if the first connection is a self-connection. Fixed in #469