lazy or not lazy

[rich-HJ]

Hi,

I am wondering if there is a way to know if a field is lazy or not? Something like print( cf.is_lazy( field_name). I had a look but could not see such functionality. The HJ framework reads a lot of files with cf.read which I presume will be lazy. I am try to trace where the field becomes non-lazy and causes memory issues. Thanks in advance.

[davidhassell]

Hi @rich-HJ

Good question! There is the Data.in_memory attribute:

>>> f.data.in_memory
False

but that only returns True if all of the data is in memory. Maybe that is what you need (?), but if any chunk of it is still lazy then it returns False.

If you want to a more granular approach, it's easy to write a little function that will do something a bit finer, following on from the approach of https://github.com/NCAS-CMS/cf-python/blob/v3.12.0/cf/data/data.py#L12180-L12196. E.g.

def percentage_of_data_in_memory(f):
    size_in_memory = 0
    for partition in f.data.partitions.matrix.flat:
        if partition.in_memory:
            size_in_memory += partition.size
 
    return 100 * size_in_memory / f.size

f = cf.example_field(0)
print(percentage_of_data_in_memory(f))
100.0

As an aside, we are close to completion of rewrite of the internals of cf-python to use dask instead of LAMA, for which the question of laziness will need to be redefined and re-evaluated ... mentioning #182 so that this thought gets considered further as part of that ...

Cheers,
David

[rich-HJ]

Great thanks for that! The granular approach is definitely the answer. I need to get my head around dask as everyone seems to be switching to it.