Different size in trial row for single column

[ahabdallah27]

I am adding trials and one of the columns can have a different length at each row. This seems to be causing a problem when I try to write the file because if I replace each row with the same data for that specific column, then I do not get the error when writing. Any thoughts on how to address this?

I am also curious about creating a timestamp. My data start time is 2 and end time is 1742172. I looked at tutorials that showed creating a list from the start time to the end time was needed. I don't understand the difference between data and timestamp so I am populating them with the same information. When writing the file, using the commented out portion my computer runs out of memory and I have to restart it. The uncommented portion works, but I don't believe it's correct. Am I setting up my time series wrong? I have 1341 trials of varying lengths of time. My understanding is that I am supposed to put all of the times of the trials in a single timestamp based on the feedback I got here: #69.

[CodyCBakerPhD]

I am adding trials and one of the columns can have a different length at each row.

This sounds like you want that column to be what's called a ragged array; this is accomplished using the index keyword argument when defining the trial column using add_trial_column (or add_column if directly constructing a DynamicTable) - you would set index=True for vector valued entries on each row, index=2 for matrix valued entries on each row (the dimensions do not need to be consistent over rows either)

I am also curious about creating a timestamp. My data start time is 2 and end time is 1742172. I looked at tutorials that showed creating a list from the start time to the end time was needed. I don't understand the difference between data and timestamp so I am populating them with the same information.

For trials, you are better off using the trials table rather than a TimeSeries - this table would have two required columns start_time and stop_time, whose values should be in units seconds relative to the session_start_time of the file. From the value of 1742172 it sounds like your time is currently an integer and might be in milli or microseconds (or even unix time)?

More generally, a TimeSeries is meant to represent some scalar, array, or matrix-valued change over time $\textbf{f} (t)$ and as such the first dimensions of data should be equal to the number of 'frames' acquired, which should either be regularly sampled (and using a scalar rate field to define the time basis) or irregular with the time points set as the vector of timestamps. Common examples of this include an ElectricalSeries which is voltage-valued and two-dimensional, with the second dimension being the number of channels or electrodes used to record neural activity.

[ahabdallah27]

Thanks, adding index=2 worked! Shouldn't the error occur on the trial creation instead of during saving?

With regards to the trials, the time would be scaled by .001 so yes it is in milliseconds. I am confused about the TimeSeries though. How can it span a vast timeframe when the trials are between different timeframes? In the example you provided, the trials go are like this: [0, 2] U [3, 5] U [6,8] U [9, 11] U [12, 14] U [15, 17] and the TimeSeries goes from 0.5 to 6.01. How does the TimeSeries have a smaller timeframe than the trials? Shouldn't it have the exact same times as the trials? In my case, I'd have trials like [2, 8] U [12, 560] U [650, 2090] U [3005, 3250] , ... so how would my TimeSeries look? Would it have timestamps similar to my times in the trials (therefore not continuous) with the voltage data column corresponding to the timestamp?

Additionally, what is rate_ts vs test_ts in the example website? I see a voltage unit in the rate_ts, but there isn't one in the time_ts, yet it is voltage?

Finally, let's say I do put actual data in my TimeSeries. It being so massive appears to be causing a memory issue. It's possible that if I break it up into the actual time ranges it wouldn't, but going from 2 to 1742172 in steps of 1 certainly did. I have yet to try it because I'd like clarification on the above first.

[CodyCBakerPhD]

Shouldn't the error occur on the trial creation instead of during saving?

Hah, io.write errors from HDMF are unfortunately quite common since it doesn't effectively know how to size things on disk until that step (whereas prior to that everything is just linked to the original in-memory objects). I'll ping @rly to see if we can have something try to catch this in-memory during the add_column call

I am confused about the TimeSeries though. How can it span a vast timeframe when the trials are between different timeframes? In the example you provided, the trials go are like this: [0, 2] U [3, 5] U [6,8] U [9, 11] U [12, 14] U [15, 17] and the TimeSeries goes from 0.5 to 6.01. How does the TimeSeries have a smaller timeframe than the trials?

I can't answer that without seeing your data; are you saying the first trial starts at the same time as the session start time, lasts for two seconds, then the next trial starts three seconds after the session and ends at the give second mark?

In general it is typical to have a TimeSeries that would span most of the trials; but this of course depends entirely on the experiment setup

Shouldn't it have the exact same times as the trials?

The point is they are two separate data streams; if you have a continuous signal being tracked periodically over time, then you have a way of partitioning global time according to some intervals being 'trials', then you have a way to compare the two relative to a common clock

Additionally, what is rate_ts vs test_ts in the example website?

They are both defined in the cell above; https://pynwb.readthedocs.io/en/stable/tutorials/general/plot_timeintervals.html#setup-creating-an-example-nwb-file-for-the-tutorial

test_ts is a dummy TimeSeries representing some distance (unit is in meters) being tracked over time; possibly representing something akin to the centroid of the subject in a navigation task

Finally, let's say I do put actual data in my TimeSeries. It being so massive appears to be causing a memory issue.

For this, check out the tutorial on iterative write to prevent loading everything into memory all at once

The best performing modern method, assuming your source data is a sliceable object such as a numpy.array, is the SliceableDataChunkIterator from neuroconv which you can import (after pip install neuroconv) using

from neuroconv.tools.hdmf import SliceableDataChunkIterator

[ahabdallah27]

I'll ping @rly to see if we can have something try to catch this in-memory during the add_column call

I believe it should be done during add_trial, not add_column, but thanks!

I can't answer that without seeing your data; are you saying the first trial starts at the same time as the session start time, lasts for two seconds, then the next trial starts three seconds after the session and ends at the give second mark?

The numbers I provided are from the example in the link so yes I assuming the same as you, but my point was that the TimeSeries has different times from the trial start and end times. https://pynwb.readthedocs.io/en/stable/tutorials/general/plot_timeintervals.html#setup-creating-an-example-nwb-file-for-the-tutorial

The point is they are two separate data streams; if you have a continuous signal being tracked periodically over time, then you have a way of partitioning global time according to some intervals being 'trials', then you have a way to compare the two relative to a common clock

Understood, I don't believe we have that, we collect all the data during the trials and don't collect additional data after a trial ends so the TimeSeries doesn't seem relevant to me. Therefore, I shouldn't have it in the trial table, right? I should instead add a column to the trial table for each data field that is being collected?

For this, check out the tutorial on iterative write to prevent loading everything into memory all at once
The best performing modern method, assuming your source data is a sliceable object such as a numpy.array, is the SliceableDataChunkIterator from neuroconv which you can import (after pip install neuroconv) using

Thanks, I'll look into this a bit more and report back if I have any issue.

[CodyCBakerPhD]

I should instead add a column to the trial table for each data field that is being collected?

This sounds like the best option!

That part of the tutorial is just there to show that it 'can' be done - I don't often see it used in practice. Or if I do it's usually a TimeSeriesReference object that also includes pre-calculated frame ranges of the series that fall within the trial

[ahabdallah27]

Thank you very much!

[ahabdallah27]

I am trying the DataChunkIterator and it's taking quite a while to save the data. I stopped the write at 10 minutes because I don't think this is working as expected, but please do let me know if that's expected. My ExperimentTime and data fields each have almost 2 million elements. One is 2 million x 96 so there is a lot of data. This is how I'm doing it:

RawInputValsTimeSeries = TimeSeries( #preferably SpatialSeries, but needs reference frame
    name="Raw Finger Values",
    data=DataChunkIterator(data=np.array(RawInputVals)),
    unit="V",
    timestamps=DataChunkIterator(data=np.array(ExperimentTime)),
    conversion=.001
)

FingerAnglesTIMRLTimeSeries = TimeSeries( #preferably SpatialSeries, but needs reference frame
    name="Finger Angles",
    data=DataChunkIterator(data=np.array(FingerAnglesTIMRL)),
    unit="Decimal Percent",
    timestamps=DataChunkIterator(data=np.array(ExperimentTime)),
    conversion=.001
)

NeuralFeatureTimeSeries = TimeSeries(
    name="Spiking-Band Power",
    data=DataChunkIterator(data=np.array(NeuralFeature)),
    unit="Hz",
    timestamps=DataChunkIterator(data=np.array(ExperimentTime)),
    conversion=.001
)

SampleWidthTimeSeries = TimeSeries(
    name="Width of Spiking-Band Power",
    data=DataChunkIterator(data=np.array(SampleWidth)),
    unit="Count",
    timestamps=DataChunkIterator(data=np.array(ExperimentTime)),
    conversion=.001
)

DecodeTimeSeries = TimeSeries( #preferably SpatialSeries, but needs reference frame
    name="Decoder",
    data=DataChunkIterator(data=np.array(Decode)),
    unit="Decimal Percent",
    timestamps=DataChunkIterator(data=np.array(ExperimentTime)),
    conversion=.001
)

behavior_module = nwbfile.create_processing_module(
    name="behavior", description="processed behavioral data"
)
behavior_module.add(FingerAnglesTIMRLTimeSeries)
behavior_module.add(NeuralFeatureTimeSeries)

nwbfile.add_acquisition(RawInputValsTimeSeries)
nwbfile.add_acquisition(SampleWidthTimeSeries)
nwbfile.add_acquisition(DecodeTimeSeries)

with NWBHDF5IO("myfile.nwb", "w") as nwbfileio:
    nwbfileio.write(nwbfile)

[CodyCBakerPhD]

The DataChunkIterator performance is sensitive to two keyword arguments - iter_axis (which should be the longer of the two, and defaults to 0, so you should be OK there) and buffer_size (in units of 'number of elements to load along the iter_axis; all other axes fill to the maximum). If you're set on using the DataChunkIterator, try increasing buffer_size to 100_000 or so and see if that helps any

Since all your data seem to be numpy.array types (and hence sliceable), I'd strongly recommend giving the SliceableDataChunkIterator a try

from neuroconv.tools.hdmf import SliceableDataChunkIterator

which should be perfectly interchangeable with what you have currently written, except that it handles the buffering pattern with more grace and intuition, and its keyword arguments are all easier to understand (since they are written in terms of total RAM usage, etc)

[ahabdallah27]

I used it and was able to write the file, but it's taking up almost 2GB. Is there any compression with these methods? I can't find documentation for SliceableDataChunkIterator. I searched for it on https://neuroconv.readthedocs.io/en/main/ and couldn't find anything.