Improve identification of BIDS data files and sidecars #38
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In #36 @birajstha found two cases where the `is_associated_sidecar` function produces incorrect results. 1. When there is more than one non-sidecar json file with the same suffix in a directory. 2. When a file has a compound extension like '.surf.json'. Here add (failing) test cases to cover these errors.
- Use `parse_bids_entities` to get full compound extension rather than the final part from `Path.suffix`. - To determine if a file is a sidecar json, look for paths with same suffix that *don't* end in '.json'.
Test includes failing case for the 'ome.zarr' directories which are in fact data "files". Nb, I don't really like this pattern. IMO these directories should be e.g. tar archives. It doesn't seem worth breaking the rule that data files are files.
It turns out not all bids data "files" are actually files. Some can be directories, e.g. the '.ome.zarr' directories (see [here](https://bids-specification.readthedocs.io/en/stable/modality-specific-files/microscopy.html#file-formats)). However, it seems it may be safe to assume all BIDS files have a file extension. This is still a pretty poor test for whether a file is a valid BIDS file. But I don't want to do a test that is overly restictive or very involved.
Rather than match against fixed datatypes, just match any lowercase word in the correct path position.
- Make datatype and suffix required. This excludes metadata files like `*_scans.tsv` files. - Exclude files contained in directories that are treated as bids files themselves. Also remove the enforced `allowed_values` on the datatype entity.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes issues of mis-identifying BIDS data files and JSON sidecars brought up in #36.
'.surf.json'incorrectly treated as sidecars.'.ome.zarr'directories and MEG'.ds'directories.'*_scans.tsv'.