Show progress in assembly

[cortner]

Does the showprogress line

    @showprogress pmap(packets) do p

correctly represent progress? I.e. does it account for the fact that some structures are much bigger than others and more expensive to assemble?

Note in #54 I do this manually.

My evidence for this is that my distributed assembly starts with > 1.5h, then drops to ca 1h for a while and then completes in around 25 minutes total.

[wcwitt]

I've been claiming this works for months, but it turns out I was mistaken, sorry.

I interpreted this, from the docs, to mean @showprogress uses length as a cost function:

The @showprogress macro wraps a for loop, comprehension, @distributed for loop, or map/pmap/reduce as long as the object being iterated over implements the length method

Which is why I set Base.length(d::DataPacket) = count_observations(d.data). But it doesn't work that way after all.

My latest experiments use your approach anyway - we can close this once they're merged.

[cortner]

related to #2

[cortner]

can this be closed now?

[wcwitt]

no, unfortunately.