How to use filter/mask with calculate_all() ?

[Martin-Rey]

Hi all,

I am in situation where I would like to calculate a property for all halos in a timestep that satisfy a condition on a different property (all halos in a mass range for example).

It feels like this should be feasible since the web interface allows you to filter halos on a given condition very easily but I can't find the right syntax to make it work within python with calculate_all().

An example of this use case is actually provided here

tangos/tangos/examples/mergers.py

Lines 31 to 42 in 7fb6052

	def most_major_mergers_since(ts, Mvir_min=0.8e12, Mvir_max=2e12, z_merger_max=4.0, no_merger_value=None):
	"""Given a timestep, calculate the most major merger ratio since a given redshift for all halos in a specified mass range
	:parameter ts - The (end) timestep to take halos from
	:parameter Mvir_min, Mvir_max - the minimum and maximum halo masses to consider
	:parameter z_merger_max - the maximum redshift to search for mergers
	:parameter no_merger_value - the value to insert for a halo if no merger was found (default None)
	:type ts tangos.core.TimeStep
	"""
	dbid, Mvir = ts.calculate_all("dbid()","Mvir")
	mask = (Mvir>Mvir_min)*(Mvir<Mvir_max)

but in a wasteful way (e.g. calculate for all halos and then mask, rather than calculate just for the masked halos)

Is this something that is easily achievable ? Where is the definition of the underlying filtering command used by the web interface ?

[apontzen]

Unless the calculation is very expensive, you don't lose much by filtering afterwards. The web interface only filters in the front-end interface, behind the scenes it still calculates every row. But if you have a case where this is causing problems, it wouldn't be too hard to implement pre-filtering. You would want to create a temporary halo list (see https://github.com/pynbody/tangos/blob/master/tangos/temporary_halolist.py) and query against that (see https://github.com/pynbody/tangos/blob/master/tangos/core/halo.py#L300 for an example)

[Martin-Rey]

This is for an expensive live-calculation that I wanted to calculate across multiple halos.

The easier is probably to turn it into a written calculation for which filter are implemented through --include-only.

[apontzen]

Yes, that seems like a sensible solution although maybe one day it would be worth implementing a way to seamlessly filter before calculating.