Why is pickled array so large?

[nbecker]

I was exploring if awkward ragged arrays could save space when pickled compared to python lists of lists.

I did a little testing and found that awkward was using more space than python lists of lists format. I expected that awkward would be smaller since it was storing in a uniform int format, while python stores each element as object.

 len(dumps(ak.Array([[1,2,3]*100000,[4,5]*100000]).to_list()))
Out[58]: 1001155

In [59]: len(dumps(ak.Array([[1,2,3]*100000,[4,5]*100000])))
Out[59]: 4000463

This shows the python list format is 1/4 the size of the dump produced by the same awkward array. Any ideas on why, or other possibilities to dump ragged arrays of uniform data elements (int, float) more compactly?

[agoose77]

The nuance here is that the default integer type in Awkward Array is int64. So, when pickling the Awkward Array, each integer consumes 8 bytes. Meanwhile, Python's pickler knows how to densely pack integers (I think this routine: https://github.com/python/cpython/blob/2d3d9b4461d0e2cb475014868af3c2f241cb6495/Modules/_pickle.c#L2066). As such, particularly for small values, the difference between the two is stark.

If you care about space, then you can just use compression, e.g.

import io
import pickle
import numpy as np
import awkward as ak

def save_compressed(array, file_):
    form, length, container = ak.to_buffers(array)

    # Compress the arrays into a bytestream
    data = io.BytesIO()
    np.savez_compressed(data, **container)
    data.seek(0)

    # Use pickle for general serialisation
    pickle.dump([form, length, data], file_)

def load_compressed(file_):
    form, length, data = pickle.load(file_)
    container = np.load(data)
    return ak.from_buffers(form, length, container)

This is just using DEFLATE compression; a different compressor may produce better results.

[nbecker]

I see I was confused because I used a very small and non-random array for testing. If I use an array of random 64bit integers, then the size of the dump is not much larger than 8*#elements. Assuming the data is not actually compressible, this is as good as I could hope.
Thanks again!

[agoose77]

Fab! I'm marking my response as the answer given your approval, and I hope it proves useful! :)

[jpivarski]

Nice answer! It's also worth knowing that Python interns numbers from 0 to 100 because they're used so often. Consider these two Python runs, in which the total memory used (USS) is measured externally (adding up the sys.getsizeof of all the objects wouldn't account for the savings from multiple references to the same object, but USS does).

>>> import psutil, gc
>>> p = psutil.Process()
>>> gc.collect()
28
>>> p.memory_full_info().uss
49270784
>>> tmp = [i % 10 for i in range(100000000)]
>>> gc.collect()
0
>>> p.memory_full_info().uss
876728320
>>> (876728320 - 49270784) / 1000000
827.457536

and

>>> import psutil, gc
>>> p = psutil.Process()
>>> gc.collect()
28
>>> p.memory_full_info().uss
49299456
>>> tmp = [i for i in range(100000000)]
>>> gc.collect()
0
>>> p.memory_full_info().uss
4089290752
>>> (4089290752 - 49299456) / 1000000
4039.991296

Making a list of 100 million numbers from 0 up to 10 uses 827 MB.

Making a list of 100 million numbers from 0 up to 100 million (still less than an int32's MAXINT) uses 4040 MB.

It's because each of the numbers greater than 100 has to be stored as a full Python object, but the small numbers are just references to Python's cache. It's this kind of independence from the object model that storing data in arrays (NumPy, Awkward, etc.) provides.

Ack! I just realized that you're not talking about space in memory, but on disk with pickle. Well, since pickle keeps track of object identity, the same is probably true. Yes, it is:

>>> import pickle
>>> tmp = [i % 10 for i in range(100000000)]
>>> pickle.dump(tmp, open("/tmp/small-numbers.pkl", "wb"))
>>> tmp = [i for i in range(100000000)]
>>> pickle.dump(tmp, open("/tmp/big-numbers.pkl", "wb"))

% du -ks /tmp/*-numbers.pkl
488420	/tmp/big-numbers.pkl
195540	/tmp/small-numbers.pkl

It's a factor of 0.2 for data in memory and a factor of 0.4 for pickled data on disk. There's probably a reason for those round numbers, but I'm not going to go further into this rabbit hole.

Anyway, I find it amusing that memory/pickle size doesn't just depend on how many integers you store, but what the values of those integers are, well below the int32 MAXINT threshold.