Does there exist an efficient approach to convert between 64bit roaring and 32bit roaring?

[yingfeng]

We've met a situation that the conversion between 64bit roaring and 32bit roaring needs to be computed very fast. Currently, we can only use a naive solution that during iterating one bitmap while inserting into the other at the same time, which is pretty time consuming given a high bitmap cardinality. Is it possible to provide an efficient approach such that the conversion could be performed internally without decompression and compression one by one?

[lemire]

I do not understand this issue. Can you provide a sample Go code that you are using?

[yingfeng]

//m is an instance of 64bit roaring bitmap
//mp is an instance of 32bit roaring bitmap

is := m.Slice()
{
  vs := make([]uint32, len(is), len(is))
   for j, v := range is {
      vs[j] = uint32(v)
   }
   mp.AddMany(vs)
  }

This code snippet is used to extract ids from source roaring bitmap(eg 64bit roaring), and then insert to target roaring bitmap after the type conversion (eg 32bit roaring).

The overhead is pretty high, any suggestions to make this precedure accelerated?

[lemire]

What is a 64-bit roaring bitmap? There is no such thing in the library right now.

[yingfeng]

Oh, sorry,
I use the pilosa fork of the go roaring which provides the 64bit bitmap ...

[lemire]

Right. So the question is how to go from Pilosa to the roaring implementation in this repo, here, is that correct?

[yingfeng]

Using the WIP 64bit solution of this repo, could the conversion between 32bit and 64bit roaring have a faster approach?

[lemire]

Yeah... but the WIP is a WIP. We need help with it to make sure it meets people’s needs. Could you help?