Dataset details configurable

[nitrosx]

Issue Name

Make Dataset Details page configurable through site configuration

Summary

Currently the dataset details page is hard coded as is. There is no option to customize labels, fields order, and which field is in which tile.
We would like to be able to do so by the page structure in configuration.
This feature request arise from the need to restructure such page according to the facility requirements. In our case, the librarian is providing valuable feedback about how to present the dataset information in the most user friendly way. Unfortunately, we would nee to change the FE code to follow up, which will impact other facilities instances.

[nitrosx]

Configuration for the current dataset details view could be the following:
Some fields might be missing.

[
 {
  "name" : "general_information",
  "label" : "General Information",
  "color" : "#ffffff",
  "order" : 10,
  "fields" : [
   {
    "name" : "datasetName",
    "label" : "Dataset Name",
    "order" : 10,
   },
   {
    "name" : "description",
    "label" : "Description",
    "order" : 20,
   },
   {
    "name" : "pid",
    "label" : "Dataset Id (PID)",
    "order" : 20,
   },
   {
    "name" : "type",
    "label" : "Dataset Type",
    "order" : 30,
   },
   {
    "name" : "creationTime",
    "label" : "Creation Type",
    "order" : 40,
   },
   {
    "name" : "keywords",
    "label" : "Keywords",
    "order" : 80,
   }
  ]
 },

 {
  "name" : "creator_information",
  "label" : "Creator Information",
  "classes" : "",
  "order" : 20,
  "fields" : [
   {
    "name" : "owner",
    "label" : "Owner",
    "order" : 10,
   },
   {
    "name" : "principalInvestigator",
    "label" : "Principal Investigator",
    "order" : 20,
   },
   {
    "name" : "orcid",
    "label" : "Orcid",
    "order" : 30,
   },
   {
    "name" : "contactEmail",
    "label" : "Contact Email",
    "order" : 40,
   },
   {
    "name" : "ownerGroup",
    "label" : "Owner Group",
    "order" : 50,
   },
   {
    "name" : "access_groups",
    "label" : "Access Groups",
    "order" : 60,
   },
  ]
 },
 {
  "name" : "file_information",
  "label" : "File Information",
  "classes" : "",
  "order" : 30,
  "fields" : [
   {
    "name" : "sourceFolder",
    "label" : "Source Folder",
    "order" : 10,
   },
   {
    "name" : "sourceFolderHost",
    "label" : "Data Host",
    "order" : 20,
   },
   {
    "name" : "size",
    "label" : "Size",
    "order" : 30,
   },
   {
    "name" : "dataFormat",
    "label" : "Data Format",
    "order" : 40,
   },
  ]
 },
 {
  "name" : "related_documents",
  "label" : "Related Documents",
  "color" : "#ffffff",
  "order" : 50,
  "fields" : [
   {
    "name" : "proposal.Name",
    "label" : "Proposal",
    "order" : 10,
   },
   {
    "name" : "instrument.name",
    "label" : "Instrument",
    "order" : 20,
   },
   {
    "name" : "sample.name",
    "label" : "Sample",
    "order" : 30,
   },
   {
    "name" : "creationLocation",
    "label" : "Creation Location",
    "order" : 40,
   },
   {
    "name" : "inputDatasets",
    "label" : "Input Datasets",
    "order" : 50,
   },
  ]
 },
 {
  "name" : "scientific_metadata",
  "label" : "Scientific Metadata",
  "color" : "#ffffff",
  "order" : 60,
  "widget" : scientific_metadata
 }
}

[nitrosx]

The configuration that would meet our needs at ESS would be the following:

[
 {
  "name" : "general_information",
  "label" : "General Information",
  "color" : "#ffffff",
  "order" : 10,
  "fields" : [
   {
    "name" : "datasetName",
    "label" : "Dataset Name",
    "order" : 10,
   },
   {
    "name" : "description",
    "label" : "Description",
    "order" : 20,
   },
   {
    "name" : "startTime",
    "label" : "Start Date and Time",
    "order" : 30,
   },
   {
    "name" : "endTime",
    "label" : "End Date and Time",
    "order" : 40,
   },
   {
    "name" : "principalInvestigator",
    "label" : "Principal Investigator",
    "order" : 50,
   },
   {
    "name" : "proposalId",
    "label" : "Proposal Id",
    "order" : 60,
   },
   {
    "name" : "runNumber",
    "label" : "Run Number",
    "order" : 70,
   },
   {
    "name" : "keywords",
    "label" : "Keywords",
    "order" : 80,
   }
  ]
 },
 {
  "name" : "dataset_information",
  "label" : "Dataset Information",
  "color" : "#ffffff",
  "order" : 20,
  "fields" : [
   {
    "name" : "datasetName",
    "label" : "Dataset Name",
    "order" : 10,
   },
   {
    "name" : "pid",
    "label" : "Dataset Id (PID)",
    "order" : 20,
   },
   {
    "name" : "type",
    "label" : "Dataset Type",
    "order" : 30,
   },
   {
    "name" : "creationTime",
    "label" : "Creation Type",
    "order" : 40,
   },
  ]
 },
 {
  "name" : "contact_information",
  "label" : "Contact Information",
  "color" : "#ffffff",
  "order" : 30,
  "fields" : [
   {
    "name" : "principalInvestigator",
    "label" : "Principal Investigator",
    "order" : 10,
   },
   {
    "name" : "contactEmail",
    "label" : "Contact Email",
    "order" : 20,
   },
  ]
 },
 {
  "name" : "files_information",
  "label" : "Files Information",
  "color" : "#ffffff",
  "order" : 40,
  "fields" : [
   {
    "name" : "sourceFolder",
    "label" : "Source Folder",
    "order" : 10,
   },
   {
    "name" : "sourceFolderHost",
    "label" : "Data Host",
    "order" : 20,
   },
   {
    "name" : "numberOfFiles",
    "label" : "Number of Files",
    "order" : 30,
   },
   {
    "name" : "size",
    "label" : "Total Size",
    "order" : 40,
   },
   {
    "name" : "numberOfFilesArchived",
    "label" : "Number of Files Archived",
    "order" : 50,
   },
   {
    "name" : "packedSize",
    "label" : "Total Size Archived",
    "order" : 60,
   },
  ]
 },
 {
  "name" : "related_documents",
  "label" : "Related Documents",
  "color" : "#ffffff",
  "order" : 50,
  "fields" : [
   {
    "name" : "proposalId",
    "label" : "Proposal Id",
    "order" : 10,
   },
   {
    "name" : "proposal.Name",
    "label" : "Proposal Name",
    "order" : 20,
   },
   {
    "name" : "instrumentId",
    "label" : "Instrument Id",
    "order" : 30,
   },
   {
    "name" : "instrument.name",
    "label" : "Proposal Name",
    "order" : 40,
   },
   {
    "name" : "creationLocation",
    "label" : "Creation Location",
    "order" : 50,
   },
   {
    "name" : "inputDatasets",
    "label" : "Input Datasets",
    "order" : 60,
   },
  ]
 },
 {
  "name" : "scientific_metadata",
  "label" : "Scientific Metadata",
  "color" : "#ffffff",
  "order" : 60,
  "widget" : scientific_metadata
 },
 {
  "name" : "authorization",
  "label" : "Authorization",
  "color" : "#ffffff",
  "access" : ["Admin", "SIMS"],
  "order" : 70,
  "fields" : [
   {
    "name" : "ownerGroup",
    "label" : "Owner Group",
    "order" : 10,
   },
   {
    "name" : "access_groups",
    "label" : "Access Groups",
    "order" : 20,
   },
  ]
 }
}

[nitrosx]

which will create a dataset detail page with the following structure and labels:

- General Information
  - Dataset Name
  - Description
  - Start Date and Time
  - End Date and Time
  - Principal Investigator
  - Proposal Id
  - Run Number
  - Keywords

- Dataset Information
  - Dataset Name
  - Dataset Id (PID)
  - Dataset Type
  - Creation Type

- Contact Information
  - Principal Investigator
  - Contact Email


- Files Information
  - Source Folders
  - Source Host
  - Number of Files
  - Total Size
  - Number of Files archived
  - Total size archived

- Related Documents
  - Proposal Id
  - Proposal Name
  - Instrument Id
  - Instrument Name
  - Creation Location
  - Input Datasets

- Scientific Metadata

- Authorization (only admin)
  - Owner Group
  - Access Groups

[nitrosx]

After some brainstorming here at ESS, we might think that separate Dataset details components might be the way to go.
I will add more details soon.

Here the link to the relevant documentation to load component dynamically:
https://angular.dev/guide/components/programmatic-rendering

[nitrosx]

Additional references on how to dynamically load modules:
https://medium.com/lacolaco-blog/angular-dynamic-importing-large-libraries-8ec079603d0

[nitrosx]

After internal brainstorming, we thing that it might be more useful create a dataset details interface, which allows anybody to contribute customized details pages.
The only issue is that the all the custom components will need to be included in core and the image of the releases.
We would also need a rule based mechanism, that allows to select the correct component for each dataset.

[minottic]

is ScientificMetadata display also intended to be customisable?

why not checking iframes? I like the idea anyway

[minottic]

I think something to look into is custom scripts loading/execution for custom visualisation in the dataset details page. The datasetDetail page could send data to these scripts that will render it accordingly, inside the dataset details page. This can either be done with iframes or with direct-controlled script loading (I think). We can then create a visualisation registry where we deposit visualisers and enable script loading only from this registry. E.g., with iframes:

<iframe src="https://example.com/page?data=someValue"></iframe>

This snippet would live in the scicat FE details page

https://stackoverflow.com/questions/536538/pass-value-to-iframe-from-a-window

[dylanmcreynolds]

@minottic I have in the past floated the idea of a rules-based link configuration (#960). If we had a mechanism to configure links to display based on a set of rules that match metadata in the dataset, we could also make this produce the links in iframes. I have proposed iframes before but I feel like it was rejected for some reason.

Anyway, I have someone here who will soon start looking into the base rules layer of that proposal.

[Ingvord]

I have proposed iframes before but I feel like it was rejected for some reason.

Maybe because of this: https://stackoverflow.com/a/23178537

Also, why not just provide API for a build time plugin, and then in the runtime create a corresponding visualization object, based on the whatever parameters you have?

Thanks

[dylanmcreynolds]

Also, why not just provide API for a build time plugin, and then in the runtime create a corresponding visualization object, based on the whatever parameters you have?

I still think that such a runtime framework would benefit from some sort of selector api (similar to k8s) that can come up with a list of templated URLs. Each individual facility might have visualizations that are not directly tied to SciCat, that a user accesses from other frameworks.

[minottic]

@minottic I have in the past floated the idea of a rules-based link configuration (#960). If we had a mechanism to configure links to display based on a set of rules that match metadata in the dataset, we could also make this produce the links in iframes. I have proposed iframes before but I feel like it was rejected for some reason.

yes, for me the idea would mostly be the capability of loading external (maybe even local) scripts in the HTML. Being an iframe or not, I think it's a later decision. For example this might do without an iframe:

loadScript(src: string): Promise<void> {
  return new Promise((resolve, reject) => {
    const script = document.createElement('script');
    script.src = src;
    script.async = true;
    script.onload = resolve;
    script.onerror = () => reject(new Error(`Failed to load script: ${src}`));
    document.body.appendChild(script);
  });
}

ngOnInit() {
  this.loadScript('https://example.com/external-script.js').then(() => {
    (window as any).externalFunction({ key: 'value' }); // No sanitization involved
  });
}

I guess one advantage of scripts is their independence from the main FE and relative ease of development, which could make us progress rather fast with our current available efforts.

Also, why not just provide API for a build time plugin, and then in the runtime create a corresponding visualization object, based on the whatever parameters you have?

I like this idea, but I always struggle to understand how it would work in practice. Would we need to develop multiple angular components and install them all together in the FE container and then import them dynamically conditionally based on the need?

[bpedersen2]

Loading external scripts, especially from arbitrary sources seem like a very bad idea, as it opens up for very easy cross-site scripting. And iframe access to parent ressources (like session cookies etc.) gets more and more restricted in modern browsers, so this calls for more trouble in the future.

[minottic]

but what if these scripts can only come from a maintained registry (that we manage inside the scicatproject org on github) or from the /assets folder only?

[nitrosx]

Great discussion, everybody!!!
First step to is to implement a configurable view for the dataset (which ESS needs).
It will be selectable from configuration, to start with.

The work will allow us as core developers to gain the knowledge necessary to formalize how to allow additional core views that can be rule-based selected, but also how to allow third-party views.
The ultimate goal that I would like to achieve is to allow facilities to deploy the official image and inject in the container their custom views meeting their specific needs

[sbliven]

I think the discussion of how to serve facility-specific frontend components should be primarily discussed in #1519 (and SciCatProject/scicat-backend-next#862 on the backend). For now I expect we will continue to implement most features in core and then disable/enable them in the config file.

I do think that being able to customize the frontend is an important feature and can have a big impact on usability. I'm slightly concerned about the ballooning of config files. This can make it more difficult to set up and maintain scicat. For instance, config changes are often a source of merge conflicts. That shouldn't block configuration where necessary, but we should keep in mind the need to have good defaults and not require a long config file to start scicat.

I'll note that PSI also has a requirement to add some dataset info (namely integration with EMDB for some datasets), so we are quite interested in this.