Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incorporating json into bids.File (issues #596 #371) #597

Merged
merged 17 commits into from
Aug 8, 2023
Merged
Show file tree
Hide file tree
Changes from 11 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
91 changes: 91 additions & 0 deletions +bids/+util/update_struct.m
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
function js = update_struct(js, varargin)
%
% Updates structure with new values.
% Can add new fields, replace field values, remove fields,
% and append new values to a cellarray.
%
% Designed for manipulating json structures and will not work
% on structarrays.
%
% USAGE::
%
% js = update_struct(key1, value1, key2, value2);
% js = update_struct(struct(key1, value1, ...
% key2, value2));
%
% Examples:
% ---------
% Adding and replacing existing fields:
% update_struct(struct('a', 'val_a'),...
% 'a', 'new_val', 'b', 'val_b')
% struct with fields:
% a: 'new_val'
% b: 'val_b'
% Removing field from structure:
% update_struct(struct('a', 'val_a', 'b', 'val_b'),
% 'b', [])
% struct with fields:
% a: 'val_a'
% Appending values to existing field:
% update_struct(struct('a', 'val_a', 'b', 'val_b'),
% 'b-add', 'val_b2')
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
% 'b-add', 'val_b2')
% 'b-add', 'val_b2')

Random thought.
I know this is an edge case in the world of bids because we would expect the field to be camel camse but what if one of the field to update is named "b-add", do you have then do something like: "b-add-add" ?

Copy link
Collaborator

@Remi-Gau Remi-Gau Aug 1, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe it would be better to have simpler function:

  • one for adding
  • one for removing
  • one for appending an element to a an array or a cell
  • one for popping an element from a array or a cell

or have the first argument specify what "action" to perform.

Copy link
Collaborator Author

@nbeliy nbeliy Aug 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, no matter how BIDS will decide to make life needlessly complicated, but if json is transformed into matlab structure, the fieldnames like b-add will be forbidden.

Splitting into several subcommand: metadata_add, metadata_append, metadata_rm can be implemented, and maybe is a good idea. But then it would be difficult to work with struct instead of parameters.

For poping (remove and return) is more complicated, as matlab do not modify current object, and multiple returns looks ugly:

 [file, val] = file.metadata_pop('val_a');

What is your take on not forcing to read metadata when creating File instance, and making it on request only?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the fieldnames like b-add will be forbidden

Oh dear... Matlab 101. Silly me for forgetting this... I guess that's what happens when you start switching back and forth between languages

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is your take on not forcing to read metadata when creating File instance, and making it on request only?

I think that the metadata reading will try to implement the inheritance principle (by reading potentially matching metadatafiles in the parent directories) and I don't remember if I have put the fail safe in place to avoid "weird" behaviors.

For example it may be while doing some file wrangling that I do not want the metadata from temp/task-foo_bold.json to b added to those of tmp/WIP/sub-01_task-foo_bold.json.

temp
├── task-foo_bold.json
└── WIP
    ├── sub-01_task-foo_bold.json
    └── sub-01_task-foo_bold.nii

I think this was the kind of concern I had. But improving the handling of the inheritance principle could help with things like this.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that inheritance should be part of bids (even if I don't like it due to all the complications). My question should the metadata be loaded implicitly or explicitly (f.set_metadata();).

Implicit reading of metadata imposes that the logic of retrieving json files, and reading them happens every time when a new instance is created, even if it's not needed.

The way how inheritance principle is implemented is an another issue -- it requires the knowledge of the layout of dataset, which is stored in layout structure, but not in File. For me (very personal opinion), inheritance principle should be applied during creation of layout, by setting File.metadata_files, but not in File.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just realized that set_metadata relies on bids.internal.get_meta_list that tries to implement the inheritance principle (same one use for bids.layout).

    function obj = set_metadata(obj)
      if isempty(obj.metadata_files)
        pattern = '^.*%s\\.json$';
        obj.metadata_files = bids.internal.get_meta_list(obj.path, pattern);
      end
      obj.metadata =  bids.internal.get_metadata(obj.metadata_files);
    end

I would say that changing this is beyond the scope of this PR which is on updating metadata. Just thought it was good to mention it.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For poping (remove and return) is more complicated, as matlab do not modify current object, and multiple returns looks ugly:

 [file, val] = file.metadata_pop('val_a');

I think I would just return the file object.

file = file.metadata_pop('FieldToPopFrom', 'value_to_pop');

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWI: will open a small PR to add a parameter to bids.internal.get_meta_list to decide to use the inheritance principle or not to list metadata files.

It could then be reused here to let user decide what they want to do.

% struct with fields:
% a: 'val_a'
% b: {'val_b'; 'val_b2'}
%

% (C) Copyright 2023 BIDS-MATLAB developers

if numel(varargin) == 0
% Nothing to do
return
end

if numel(varargin) > 1
key_list = varargin(1:2:end);
val_list = varargin(2:2:end);
elseif isstruct(varargin{1})
key_list = fieldnames(varargin{1});
val_list = cell(size(key_list));
for i = 1:numel(key_list)
val_list{i} = varargin{1}.(key_list{i});
end
else
id = bids.internal.camel_case('invalidInput');
msg = 'Not list of parameters or structure';
bids.internal.error_handling(mfilename(), id, msg, false, true);
end

for ii = 1:numel(key_list)
par_key = key_list{ii};
try
par_value = val_list{ii};

% Removing field from json structure
% Should use only empty double ([]) or any empth object?
if isempty(par_value) && isnumeric(par_value)
if isfield(js, par_key)
js = rmfield(js, par_key);
end
continue
end

if bids.internal.ends_with(par_key, '-add')
par_key = par_key(1:end - 4);
if isfield(js, par_key)
if ischar(js.(par_key))
par_value = {js.(par_key); par_value}; %#ok<AGROW>
else
par_value = [js.(par_key); par_value]; %#ok<AGROW>
end
end
end
js(1).(par_key) = par_value;

catch ME
id = bids.internal.camel_case('structError');
msg = sprintf('''%s'' (%d) -- %s', par_key, ii, ME.message);
bids.internal.error_handling(mfilename(), id, msg, false, true);
end
end
end
77 changes: 75 additions & 2 deletions +bids/File.m
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,38 @@
%
% file = bids.File(name_spec, 'use_schema', true);
%
% Load metadata (supporting inheritance).
%
% .. code-block:: matlab
%
% f = bids.File('tests/data/synthetic/sub-01/anat/sub-01_T1w.nii.gz');
% f = f.set_metadata();
%
% Acess metadata
%
% .. code-block:: matlab
%
% f.metadata()
% struct with fields:
% Manufacturer: 'Siemens'
% FlipAngle: 10
%
% Modify metadata
%
% .. code-block:: matlab
%
% f = f.metadata_update('Description', 'Source file');
% f.metadata()
% struct with fields:
% Manufacturer: 'Siemens'
% FlipAngle: 10
% Description: 'Source file'
%
% Export metadata as json:
%
% .. code-block:: matlab
%
% f.metadata_export()

% (C) Copyright 2021 BIDS-MATLAB developers

Expand All @@ -90,7 +122,7 @@

metadata_files = {} % list of metadata files related

metadata % list of metadata for this file
metadata = [] % list of metadata for this file

entity_required = {} % Required entities

Expand Down Expand Up @@ -165,7 +197,7 @@
obj = obj.use_schema();
end

obj = obj.set_metadata();
% obj = obj.set_metadata();

obj = obj.update();
end
Expand Down Expand Up @@ -707,6 +739,47 @@ function check_required_entities(obj)

end

% Functions related to metadata manipulation

function obj = metadata_update(obj, varargin)
% Update stored metadata with new values passed in varargin,
% which can be either a structure, or pairs of key-values.
%
% See also
% bids.util.update_struct
%
% USAGE::
%
% f = f.metadata_update(key1, value1, key2, value2);
% f = f.metadata_update(struct(key1, value1, ...
% key2, value2));

obj.metadata = bids.util.update_struct(obj.metadata, varargin{:});
end

function metadata_export(obj, varargin)
% Export current content of metadata to sidecar json with
% same name as current file. Metadata fields can be modified
% with new values passed in varargin, which can be either a structure,
% or pairs of key-values. These modifications do not affect
% current File object, and only exported into file. Use
% bids.File.metadata_update to update currect metadata.
%
% See also
% bids.util.update_struct
%
% USAGE::
%
% f.metadata_export(key1, value1, key2, value2);
% f.metadata_export(struct(key1, value1, ...
% key2, value2));
[path, ~, ~] = fileparts(obj.path);
out_file = fullfile(path, obj.json_filename);

der_json = bids.util.update_struct(obj.metadata, varargin{:});
bids.util.jsonencode(out_file, der_json, 'indent', ' ');
end

%% Things that might go private

function bids_file_error(obj, id, msg)
Expand Down
49 changes: 49 additions & 0 deletions tests/tests_utils/test_update_struct.m
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
function test_suite = test_update_struct %#ok<*STOUT>
try % assignment of 'localfunctions' is necessary in Matlab >= 2016
test_functions = localfunctions(); %#ok<*NASGU>
catch % no problem; early Matlab versions can use initTestSuite fine
end
initTestSuite;

end

function test_main_func()

% testing with parameters
js = struct([]);
js = bids.util.update_struct(js, 'key_a', 'val_a', 'key_b', 'val_b');
assertTrue(isfield(js, 'key_a'));
assertTrue(isfield(js, 'key_b'));
assertEqual(js.key_a, 'val_a');
assertEqual(js.key_b, 'val_b');

% testing with struct
test_struct.key_c = 'val_c';

js = bids.util.update_struct(js, test_struct);
assertTrue(isfield(js, 'key_c'));
assertEqual(js.key_c, 'val_c');

% testing update and removal of field
js = bids.util.update_struct(js, 'key_c', [], 'key_a', 'val_a2');
assertFalse(isfield(js, 'key_c'));
assertEqual(js.key_a, 'val_a2');

% testing concatenating as string cell
js = bids.util.update_struct(js, 'key_b-add', 'val_b2');
assertEqual(js.key_b, {'val_b'; 'val_b2'});

% testing concatenating numericals
js = bids.util.update_struct(js, 'key_b-add', 3);
assertEqual(js.key_b, {'val_b'; 'val_b2'; 3});
end

function test_exceptions()
% Invalid input
assertExceptionThrown(@() bids.util.update_struct(struct([]), 'key_b-add'), ...
'update_struct:invalidInput');
assertExceptionThrown(@() bids.util.update_struct(struct([]), ...
'key_b-add', [], ...
'key_c'), ...
'update_struct:structError');
end