Proposal: measurementLists Group as alternative to many measurementList1...measurementList2 indexed groups
#115
Conversation
…ed-Group measurementList `nirs(i)/data(j)/measurementList(k)`.
|
I am not in love with "measurementLists" btw... |
|
@sstucker thank you for tackling this, the proposed changes capture the proposed changes. Is some more general opening explanatory text required to explain the two options? |
snirf_specification.md
Outdated
| * **Type**: group | ||
| * **Location**: `/nirs(i)/data(j)/measurementLists` | ||
|
|
||
| The group for measurement list variables which map data array onto the to map the data array onto the probe geometry (sources and detectors), data type, and wavelength. This group's datasets are arrays with size `<number of channels>`, which each position describing the corresponding column in the data matrix. (i.e. the values at `measurementLists/sourceIndex(3)` and `measurementLists/detectorIndex(3)` correspond to `dataTimeSeries(:,3)`). |
|
Given that it's > 1 year for this, should we assume it's not relevant anymore? |
|
Issue #103 is still a problem and this solution is a sane way to handle it. I will return to this before the end of the year. |
|
This is an old issue but it remains pertinent and it would be nice to get this merged. Other than resolving the review comments, what needs to be done (beyond merging) to make sure that this is properly supported in, e.g, the validator? |
snirf_specification.md
Outdated
| The arrays of `measurementLists` are: | ||
|
|
||
| #### /nirs(i)/data(j)/measurementLists/sourceIndex | ||
| * **Presence**: required |
There was a problem hiding this comment.
for all the places in this chunk of code where we say "required", do we need to clarify that this is "required if measurementLists is utilized?
There was a problem hiding this comment.
It seems like this could be merged... but @sstucker is not working on this anymore. Can someone else make the fixes in the review comments?
Also, we need a to do list item to remind us that the validator needs to incorporate this. Do we just create a new Issue for getting that done? But the validator is not in this GitHub... hmm... I forget who is managing the validator. Actually, my group is managing pysnirf2 at https://github.com/BUNPC/pysnirf2
There was a problem hiding this comment.
I can update the validator logic in pysnirf2 as I recall working with files like this and have some other pending work there I want to release. I have the bandwidth for this, no worries
I do not feel comfortable finishing this feature for the specification itself.
There are some issues with this feature as I drafted it here, for instance, how do we deal with the missing values in the table? Should we delineate a column order for all possible channel datasets? Are NAN values supported by HDF5 and portable to all commonly used interfaces?
A more minor note: the measurementLists name only differs by a letter and may be confusing... I still don't like it
There was a problem hiding this comment.
@sstucker I am fine with measurementLists and am not confused by this being one letter different than measurementList.
I do wonder why we just didn't utilize the existing measurementList(1) for this as I described way back in #103 (comment), but I can live with measurementLists as I appreciate that it makes the intent of using arrays for describing the measurementList explicit.
|
@sstucker thank you for any help, much appreciated! Could you clarify where the NaN value situation may arise? What required fields do you envisaged being omitted in practice? |
|
@samuelpowell I do hope we can make good progress when we meet in 10 days. It would be great to resolve this soon. I am also now hitting the same issue you described way back in #103 (comment). |
There is no rule enforcing that any of the |
|
Following discussion in May 24 meeting, it was decided that this feature should be implemented as described. The issue regarding missing fields, which can arise when e.g. both processed and raw data are stored together was also discussed. The conclusion here is that a permissive approach is reasonable - when a data field is not required by a data type (e.g. a wavelength index for a chromphore) the value, if present, is ignored. This will be discussed further in #119. @sstucker will this work as implemented, e.g., will the validator be able to parse |
|
@samuelpowell It will require some work from me on the validator ahead of the next release, but I can do it. Let's get the ball rolling towards the release of the feature by merging this. |
|
Okay, I've addressed one of David's comments, merged master and updated the spell check. Before proceeding, what do you think of @dboas question "for all the places in this chunk of code where we say "required", do we need to clarify that this is "required if measurementLists is utilized?" |
|
Further to meeting of 10/07 it was determined that it would be good to include the text "required if measurementLists is present" in order to ensure the spec is literal, however the implementation overhead should be considered. @sreekanthkura7 to check the validator to determine
|
Added to required fields of measurementLists 'required if measurementLists is present'
|
I modified the spec to indicate that required fields of measurementLists are Sreekanth and I discussed the validator changes needed. He will work with Stephen on the validator once the pull request is merged. |
|
I'd like to get #151 merged first, then we can remove the same fields in this PR too. If anyone able to review the same we can make progress. |
|
@sreekanthkura7 , #151 has been merged. Can you remove the same fields from this PR as Sam requested in the comment above? |
|
@dboas , It looks like I don't have permission to edit this. Can someone please do this or give me the access to do? |
|
@sreekanthkura7 you should now be able to push commits to this PR |
Remove following fields that are removed in PR fNIRS#151. data.measurementList.moduleIndex data.measurementList.sourceModuleIndex data.measurementList.detectorModuleIndex probe.useLocalIndex
|
@samuelpowell The fields that were removed in #151 have now been removed in this PR as well |
|
Thanks @sreekanthkura7 I will re-review by Monday |
snirf_specification.md
Outdated
| @@ -537,6 +537,119 @@ As described below, optional variables `probe.sourceLabels` and | |||
| `probe.detectorLabels` are provided for indicating the instrument specific | |||
| label for sources and detectors. | |||
|
|
|||
| #### /nirs(i)/data(j)/measurementLists | |||
| * **Presence**: optional | |||
There was a problem hiding this comment.
Should this be 'required if measurementList is not present' for parity with @dboas changes to measurementList(k)?
snirf_specification.md
Outdated
|
|
||
| Must be 1-D array with length equal to the size of the second dimension of `/nirs(i)/data(j)/dataTimeSeries`. Units are optionally defined in `metaDataTags`. | ||
|
|
||
| #### /nirs(i)/data(j)/measurementLists/moduleIndex |
|
I think there was some confusion here @sreekanthkura7, the request from myself / @dboas was not to remove the exact same fields as in #151, as these would be removed by merging master into this branch. It was instead to remove the associated fields under |
|
@sreekanthkura7 I just noticed that the table of contents needs to be updated to link to measurementLists and all the sub-fields |
Implement changes to the measurementLists similar to those made in change request #157 on the measurementList. Update the presence of measurementLists. Reorder the descriptions of measurementLists for consistency. Revise the table of contents to include measurementLists. Update the format summary to include measurementLists.
|
@samuelpowell @dboas The changes discussed above have been implemented. Additionally, I updated the measurementLists in the summary table. However, I am uncertain if the type column for the measurementLists was updated correctly. Please review and let me know if any further adjustments are needed. |
|
@sstucker, now that this PR is merged, do you still have time to assist with the validator? I'm available to work with you on this as well. |
|
@sreekanthkura7 Yes, I can start a draft release of the validator. I will want to align the validator to the entire draft SNIRF release, so it would be good to get a changelog and draft release notes for the next SNIRF online soon. |
It has been highlighted by #103 that the Indexed Group, described as
is a wildly inefficient way to structure an HDF5 file.
This draft adds an alternative encoding of the measurementList.
There are a few known issues at this point, such as defining a character to be NaN at the index of channels which lack a particular value.