Download the Opal view corresponding to a variable list

[ymarcon]

Given a variable list (a variable set/cart), a user with appropriate privileges (editor) should be able to download the corresponding Opal view where Mica datasets have been replaced by the original Opal project table. The purpose of this feature is to allow an easy switch from a variable selection in Mica to a data export in Opal.

Note that:

• as Mica is able to be connected to several Opal servers, there will be one view per Opal in which case the download is a zip file; the download web service should allow to discriminate by opal instance,
• the format of the view file will be in JSON and can be an input of the restore-view python command,
• Opal UI should be able to create a view from the JSON file (obiba/opal#3464) or directly by connecting to the Mica download link (to make obiba/opal#3441 effective, the user should be able to copy the URL of the variable set).

(from @sam-story)

[ymarcon]

@sam-story How to handle the situation described in obiba/opal#3437 ? Should Mica do best-effort at guessing how to join tables (based on the detected variables with ref entity type) ? The result might not be the one expected. Mica is not the most appropriate environment for making complex join views. Or the joins should be resolved when importing in Opal?

[ymarcon]

Thinking about that, any variable search query (not just the ones to extract a cart or named variable set) could be used to make Opal view(s).

[ymarcon]

The use case reported by @sam-story is that the researcher navigating in the Mica's data catalog are not identified (anonymous) and therefore their variable selections cannot be pinned (as a named variable list). They are also using an external Data access system that does not communicate with Mica. As a consequence, the data catalog transfer between the researcher and the data manager that will make a view in Opal, will be the JSON representation of the view downloaded from Mica.

This (configurable) service will not be activated by default. It will be available as button from the Cart page.

[sam-story]

We would like a Mica config to allow Mica to create a view per Opal project. E.g. the current design is to make one view file for each Opal instance, if this option is checked then Mica will create a view per Opal project per Opal instance.
The result of the download will be a zip file and the data manager will upload the views individually..

[meek0]

Decided to create a view per table in order to avoid variable name and value type clashes.

Some tables could have the same variable name and creating a view with two variables of the same name wouldn't make sense.

[ymarcon]

@meek0 that would make sense with longitudinal data, ie derived variables would be repeatable.

[sam-story]

We currently have have 142 tables in our study and often get requests for variables spread across 50+ tables. If Mica is creating 50+ views, we would need a way in Opal to combine these back into one view per project.

Ideally we would like a configuration option so views are just create per Opal project? Our import process ensures all variables are unique across the study, so we would be happy to turn this configuration option on.

Even if some users had multiple variables with the same name, an alternative solution could be worked to use the fully qualified table and variable in the script e.g. $('project.table:variable'). Note sure how this works with repeatable variables though.
Moving over to using the fully qualified path as standard could assist with issue obiba/opal#3440 in the future (although there a still many other considerations for this improvement).

[ymarcon]

@meek0 in the case of longitudinal data, the derived variables would be repeatable (same participant, different values), but we can also have the case where different populations are merged in a single view and then derived variables are not repeatable. Mica has no way to distinguish these two cases. We can consider that Mica should do best effort (no repeatable) and would apply a predefined/configurable strategy regarding the merge or not of the views (as @sam-story mentioned) (default is no merge as it is safer).

[meek0]

i could add a parameter to choose how to group the views:

• group views by project and table (default)
• group views by project and entity type

That way the decision could be made on a on download basis.

[ymarcon]

@meek0 what do you mean by "group views"? why is it always "by project"?

[meek0]

I was seeing this from the point of view of a set of variables.

Current implementation creates a view per table and to achieve this variables are grouped by their project.table.
I figured that for group by entity type we would need to group by project first but thinking about it more I can see that grouping by project would be unnecessary.

This way the choices can be :

• group by project.table
• group by entity type

[sam-story]

We still need the option to group by project, so the options would be

• group by project.table
• group by entity type
• group by project and entity type

Also we would want to be able to configure this at the system level. We will want everything downloaded by "project and entity type" and do not want (anonymous) users to be able to change it.

[ymarcon]

From mica server UI:

• select some variables,
• push them in the cart,
• click Opal Views:

Uncaught TypeError: Cannot read property 'split' of undefined
    at HTMLAnchorElement.onClick (ng-obiba.js:2913)
    at HTMLAnchorElement.dispatch (jquery.js:4737)
    at HTMLAnchorElement.elemData.handle (jquery.js:4549)

[meek0]

should be fixed by obiba/ng-obiba#115

[ymarcon]

• have variables from several different datasets in the cart
• select some variables from a single datasets from the cart
• click opal views: the variable selection is ignored (views from all)

[meek0]

for now variable view is created from a setID (cart or list) not from a selection of variables else one could forgo using cart or list.