Using get to obtain a dictionary with parameters from GIDCollections

[hakonsbm]

With the GIDCollection object one should be able to get parameters of all the nodes, or if it's a layer, get parameters of the layer. We will do this with the function get. This will replace the current GetStatus function, which will be marked as deprecated.

Input

Input to get can be one of the following:

• A single string (Python) or literal (SLI).
• An array of strings/literals.
• Several arrays containing several strings/literals, so one can get parameters several levels down into a nested parameter-dictionary.
  • But how does one know what belongs to which branch of the parameter tree? Maybe rather input naked strings/literals for the intermediate levels?

Output

The result should be a dictionary containing an entry for the GIDs and an entry for the corresponding parameter values, or more entries if one inputs multiple parameters. The entries contain arrays of the same length, so that each parameter value is mapped to the GID of the same index in the GID-array.

Examples

Python

>> nrns.get('V_m')
{'GID': [1, 2, ...], 'V_m': [-70, -71, ...]}

>> nrns[::10].get('V_m')
{'GID': [1, 11, ...], 'V_m': [-70, -80, ...]}

>> nrns[3].get('V_m')
{'GID': [3], 'V_m': [-73]}

>> nrns.get(['V_m', 'I_e'])
{'GID': [1, 2, ...], 'V_m': [-70, -71 ...],  'I_e': [5, 6, ...]}

>> sd.get('events', 'senders')
{'senders': [5, 24, 12, ...]}

>> layer.get(['center', 'positions'])
{'center': (0.0, 0.0), 'positions': ((0.6, 0.1), (0.4, 0.3), ...)}

SLI

SLI ] gidcollection /V_m get ==
<< /GID [1 2 ...] /V_m [-70 -71 ...] >>

SLI ] gidcollection [ /V_m /I_e ] get ==
<< /GID [1 2 ...] /V_m [-70 -71 ...] /I_e [5 6 ...] >>

SLI ] sd /events /senders get ==
<< /senders [5 24 12 ...] >>

SLI ] layer [ /center /positions ] get ==
<< /center [0.0 0.0] /positions [[0.6 0.1], [0.4 0.3]] >>

Current challenges

• Getting specific parameters several levels down into a nested status dictionary.
• One issue with layers. How to know when to get the layer parameters and not the node parameters?

[stinebuu]

I am wondering if get really should return the GID tuple if we only send in one string parameter. I have added PyNEST tests for the get function, and when I wrote the tests I found it a little unnecessary to write something like

nodes = nest.Create('iaf_psc_alpha', 10)

V_m = nodes.get('V_m')['V_m']
C_a = nodes.get('C_a')['C_a']

To me, I expect nodes.get('V_m') to return the membrane potential, and not a dictionary with GIDs as well as the membrane potential.

As a reference, nest.GetStatus(nodes, 'V_m') returns a tuple with the membrane potential, so I guess that is why I expect the same with get.

I think it makes perfect sense when you send in a list to get, as

g = nodes.get(['local', 'thread', 'vp'])
local = g['local']
thread = g['thread']
vp = g['vp']

makes sense to me.

I understand that if V_m is different for all the GIDs it can be good to have the actual GIDs there, or if the GIDCollection is sliced. But, at least on the PyNEST level, it is very easy to access your GIDs, and if you have a lot of GIDs, I don't think the returned GID-list helps that much.

I think it is nice that the return value of get is on the same form no matter if you send in a string or a list, and by dropping the GID tuple with a single string this will no longer be the case of course. But I still think it is something to consider.

[jougs]

@stinebuu: I fully agree with your observation and upon thinking about it again I am not even sure that the dictionary is a good idea. Usually user's should know what they asked for and in which order and the dictionary just adds overhead and indirections when accessing the data.

I also think we can in general leave out the GID. If user's want it GID, they can simply include the key global_id in their query.

We should in any case not try to always return the same type if this goes against user expectation, which it probably does in some cases.

What about the following semantics (using the examples by @hakonsbm)?

>>> nrns.get('V_m')
[-70.0, -71.0, ...]
# Rationale: the user asked for the membrane potentials of a set of neurons and thus
# will get them all in a list (or maybe better as a NumPy array).

>>> nrns[::10].get('V_m')
[-70.0, -80.0, ...]
# Same as above

>>> nrns[3].get('V_m')
-73.0
# Rationale: If the GIDCollection only contains a single element, users probably
# know this and thus would probably want to get the value directly instead of a
# single-element list.

>>> nrns.get(['V_m', 'I_e'])
[[-70.0, -71.0, ...],  [5.0, 6.0, ...]]
# Rationale: The user provides a list of keys and gets back a list of results. If the
# GC only contains a single element, the returned list could again contain the results
# just as values instead of lists. Maybe using tuples as outer containers is better.

>> sd.get('events', 'senders')
[5, 24, 12, ...]
# Rationale: This is basically the same as  getting V_m. Users ask for all event
# senders and will get them as a list. If the second argument is a list, the result
# would again be a list of lists. I did not yet think through how this would carry over
# to the case where sd contains multiple elements. Probably it just does ;-)

>> layer.get(['center', 'positions'])
[(0.0, 0.0), ((0.6, 0.1), (0.4, 0.3), ...)]
# Rationale: Users again know what they asked for and in which order  

Regarding this question from above:

• Several arrays containing several strings/literals, so one can get parameters several levels down into a nested parameter-dictionary.
  * But how does one know what belongs to which branch of the parameter tree? Maybe rather input naked strings/literals for the intermediate levels?

I think only the last argument to get should be allowed to be a list. This way the branching becomes unique up to that point and it is easy to retrieve multiple values from the final dictionary. If multiple nested dictionaries have to be queried, multiple calls to get should be used.

@heplesser, what do you think?

[stinebuu]

@jougs I think this looks sensible.

I also think that we should be able to call get() without any parameters, just as is possible with GetStatus. I for one do not know all the default values of a model, and it is useful to get all the information GetStatus have sometimes. I don't really know how this should look though. We can't just send lists then, because we wouldn't know what the lists refer to. Right now, with GetStatus you receive a list of dictionaries. I guess we could return a dictionary with lists instead. So, instead of

>>> nrns.get()
({'C_m': 250.0,
  'Ca': 0.0,
  ...
 },
 ...,
{'C_m': 250.0,
 'Ca': 0.0,
  ...,
 'vp': 0})

we could have

>>> nrns.get()
{'C_m': [250.0, 250.0, ..., 250.0],
 'Ca': [0.0, 0.0, ..., 0.0],
  ...
 'vp': [0, 0, ... , 0]
 }

[heplesser]

@stinebuu @jougs Thank you for all the good suggestions! But I still have several issues.

Single-node access

When a user explicitly writes

nrns[3].get('V_m')

it is indeed clear that a value of a single neuron is requested. But if I have a script like

e = nest.Create('iaf_psc_alpha', 2)
i =  nest.Create('iaf_psc_alpha', 1)
# ... lots of other code ...
for p in [e, i]:
    res = p.get('V_m')
    # ... work with res ...

I will have a problem, in particular, because at the point where I use� e and i, I do not explicitly see that i is a single-neuron collection.

I can think of three alternatives:

1. Always return arrays, also for single neurons. This ensures full consistency.
2. Slicing a GIDCollection by providing just a single index sets a single_node flag on a GC. If that flag is set, get() returns a single value for a single neuron.
3. Provide a separate method, get_single().

Dictionary vs list of arrays as return type

I strongly favour dictionaries instead of lists of lists or arrays, except when requesting a single field. I.e.,

>>> nrns.get(['V_m', 'V_th'])
{'V_m': [-70., ...], 'V_th': [-55., ...]}

>>> nrns.get('V_m')
[-70., ...]

>>> nrns[0].get('V_m')    # with alt 2 from above
-70

>>> nrns[0].get(['V_m', 'V_th'])    # with alt 2 from above
{'V_m': -70, 'V_th': -55}

• This is consistent with @stinebuu's proposal for nrns.get()
• When returning lists of arrays, all subsequent code must handle column numbering correctly. This demands user attention and can introduce hard to debug bugs if one changes the list of fields requested in the get() call. Dictionaries are self-documenting.
• Dictionaries can be turned straight into Pandas dataframes with proper field names, significantly simplifying further work with the data.

Hierarchical field addressing

I am not quite sure about how to solve this one in a consistent way. The main issue is how to select multiple fields at a deeper hierarchy level. The most relevant use case in that respect is events:

• I have a multimeter, recording multiple values from multiple neurons
• I want to read out senders, time, V_m, and V_th from events
  How do I express this?

We could set up the following rules:

• Let the call be get(arg0, arg1, arg2, ...)
• If argN is a string, it is a name at the top level of the field hierarchy
• If argN is a list or tuple [argN_0, argN_1, argN_2, ...]
  • argN_0 is a name at the top level of the field hierarchy; it must be a string
  • If argN_M is a string, it is looked up as a name in field argN_0
  • Otherwise, argN_M must be a list or tuple and this rule is applied recursively
• When using hierarchical addressing, field labels in the returned dictionary will be the lowest-level field names.

Examples:

>>> nrns.get('V_m', 'V_th') 
{'V_m': [-70, ...], 'V_th': [-55, ...]}

>>> sds.get('global_id', 'n_events')
{'global_id': [2, 3], 'n_events': [10, 11]}

>>> sds.get('global_id', 'events')
{'global_id': [2, 3], 'events': [{...}, {...}]}

>>> sds.get('global_id', ['events', 'senders', 'times', 'V_m', 'V_th'])
{'global_id': [2, 3], 'times': [[...], [...]], 'senders': [[...], [...]], 
 'V_m': [[...], [...]], 'V_th': [[...], [...]], }

[heplesser]

Summary after VC 1 December

Return values

1. If get() is called with a single field name as argument, a list or array of values is returned.
2. Values are ordered in the order of nodes in the GIDCollection on which get() is called.
3. If get() is called without arguments or with more than argument, a dictionary of lists or arrays is returned.
4. If get() is called without arguments, the returned dictionary will contain the full status information provided by the nodes enquired.
5. It is an implementation detail whether data is returned as list or array.
6. GIDCollection or the kernel may provide a flag that ensures that get() returns Pandas dataframes.

>>> nrns.get(['V_m', 'V_th'])
{'V_m': [-70., ...], 'V_th': [-55., ...]}

>>> nrns.get('V_m')
[-70., ...]

Hierarchical addressing

1. get() can be called with multiple arguments, get(arg0, arg1, ..., argN).
2. All arguments except argN must be strings (field names). argN may be a list of strings (field names).
3. Arguments arg0, arg1, ..., arg(N-1) specify a path into a hierarchical status dictionary.
4. get() returns the values specified by argN after descending the path.

>>> mm[0].get('events', ['times', 'V_m'])
{'times': [0.1, 0.2, ...], 'V_m': [-70.0, -69.5, ...]}

Open issues

Inhomogenous collections

nrns may contain different neuron models.

1. If they are all just copies of the same model, this is no problem.
2. If the user explicitly requests fields that all models support, this is also fine.
3. Problems occur if calling get() without arguments on a collection containing models with different fields.
4. There are three possible solutions
   1. Raise an exception
   2. Return a dictionary containing only the fields common to all models.
   3. Return the union of all fields, filling with NaN (numeric values) or None (non-numeric values) for nodes that do not support a given field.

The last option seems to be the most promising one.

Composite values

Some models, in particular recorders, provide data in form of arrays, e.g., times, senders, etc.

1. Calling get() on a single such node is unproblematic, see hierarchical indexing example above.
2. If get() is called on a GIDCollection containing multiple, e.g., multimeters, each multimeter provides a value, which just happens to be an array.

>>> mms.get('events', ['times', 'V_m'])
{'times': [[0.1, 0.2, ...], [0.1, 0.2, ...]] , 
 'V_m': [[-70.0, -69.5, ...], [-65.0, -71.0, ...]]}

1. This is systematic, but is it safe w.r.t. user errors?
2. If we consider it unsafe, we need to raise an exception in this case.

Option 1 is most likely sufficiently safe and would in almost all cases lead to errors that are easily noticed if users should mistake dimension.

[stinebuu]

Just as a reminder for me and @hakonsbm: Everything with get() discussed here (should) work, however, if you call get() on a composite, you get a <SLILiteral: None> in the places where the corresponding model does not have the parameter. This should be converted to a plain None at some point.