Visualize 3D models locally without relying on sketchfab iframe

[luistoptal]

Pull request for issue: #2054

This is a pull request for the following functionalities:

• Add three.js widget to visualize dataset 3D models without relying on sketchfab. This module is selfcontained, can be reused anywhere on the site and takes a list of external links to 3D models as data input

How to test?

The following steps describe how I test this widget locally (if you know of a better way, please be my guest and I would love to know 🙂):

• download the models from the following urls to your computer (or feel free to use any files you want as long as they are LAS, STL, PLY or OBJ)
  • https://sketchfab.com/3d-models/leaf-01-point-cloud-5acb196dd274419c8e3c614e55d2cf8b (LAS file)
  • https://sketchfab.com/3d-models/beet-03-c63d100fb76042ecab2ec87f73da323e (STL file)
  • https://sketchfab.com/3d-models/alimentary-system-of-xenopus-laevis-nf-66-30b2ebcc9b364568a7a04ed87e81e6fd (PLY file)
  • https://sketchfab.com/3d-models/globigerina-bulloides-geo-b8502-865cm-shell-4-b7b47f84c7f84623b7036f585d89770a (OBJ file)
• Manually add the files to the assets folder, e.g: /assets/test-models/YOUR_FILES
• http://gigadb.gigasciencejournal.com/adminExternalLink/create add the files one by one as external links to any one of the published datasets (e.g. 100006). In the url field you should add something like this: /assets/test-models/filename, in External Link Type select "3d Models"
• Navigate to the dataset page, e.g.: http://gigadb.gigasciencejournal.com/dataset/100006
• You should see the "3D models" tab. When you open it, you should see the models you added in the dropdown
• The initial state of the viewer should be no loaded modal, and a visible play button
• when you click play, a model should be loaded and you should be able to interact with it
• the "help" and "fullscreen" buttons should work as intended, either with mouse or keyboard
• if you toggle fullscreen and / or resize the window, the viewer should be fully responsive and the model should remain visible
• if you select another model in the selector, the model should be immediately loaded

You can test error states if you repeat the above process by adding a file that is not a valid 3d model or not a LAS, STL, PLY or OBJ file (or you can just add a throw new Error('Ooops') in the line before await loadModel(file);)

Example of error message:

• it would be critical to test in a production-like environment with files / external links mimicking the production environment

How have functionalities been implemented?

3D models were previously visualized within an iframe taking a sketchfab url as input. This PR gets rid of the sketchfab iframe and replaces it with a JS widget that makes use of the three.js library to load and display an interactive 3D model

The main requirement is to load STL and OBJ file formats, and also (less common) PLY and LAS

Different options to locally load the model were considered (refer to the original ticket for some). I decided to use three.js because:

• It is widely popular and well known, essentially being the standard way to render 3D files in JS world
• It supports many formats via built-in loaders (LAS being an exception)
• It supports / will support WebXR via plugins. Note that WebXR is an experimental API and was not considered in this PR
• For a simple use case such as rendering a model with no animations and with basic interactions it is relatively simple to implement

Other options I checked had very low or null adoption and / or did not fully cover the use case. I opted for the widely adopted solution.

The changes can be grouped as follows:

• Mainly, everything related to the model viewer is within protected/js/model-viewer
• The partial protected/views/shared/_model_viewer.php renders the html for the widget, and loads the necessary scripts;
  • three.js imports are added to the head as a import map. That way, the three.js library can be imported as a module in the widget scripts
  • Yii::app()->assetManager->forceCopy = YII_DEBUG; makes sure the assets are rebuilt and not cached in development (NOTE: I am unclear on whether this targets specifically local dev, is this the correct solution? If this is a blocker, this line can be removed, but I found it necessary to add this line during local dev)
  • widget scripts are added to the body
  • modelviewer is initialized
• The partial is rendered in the dataset page protected/views/dataset/view.php. This page contains a switch statement that renders elements such an iframe depending on the external link types. In the switch statement, the links are handled one by one. In the case of 3D models, I deleted that case from the switch statement, and passed them to the partial as data property. Note that I had to do this because the partial, rather than handling individual links one by one, takes them all to build a select input from which to select the model to load
• less/modules/model-viewer.less takes care of all the styles related to the viewer

That covers all the changes outside of the protected/js/model-viewer folder, now related to the widget itself

• first, the idea is to abstract implementation details away from the consuming view. I suggest looking at the jsdocs decorating the modelViewer function and dig deeper only if curious about implementation details
• the widget is split into two core modules
  • ui: takes care of updating the state of the UI depending on the state of the viewer (i.e.: loading states, model loaded, no model loaded, error state)
    • the UI has 4 states:
      • idle: no model is loaded, a play button to load the currently selected model (by default, the first in the select input) should be visible
      • loading: a loading spinner should be displayed while loading a model
      • error: a danger alert should be displayed at the bottom with a relevant message. Use can select another model or click "play" again
      • success: model is loaded and should be interactive
  • viewer: it uses the three.js library to render a 3D model, and takes care of all related things (setting up scene, lights, cameras, controls, loading models...)
    • It is expected that all models are in one of LAS, STL, PLY or OBJ file formats. If another format is needed, the corresponding loader for such file format should be added (protected/js/model-viewer/viewer/components/models/loader.js)
    • among other things, this module takes care of making the viewer fully responsive
    • LAS files (data point files) needed a custom implementation, with a visualization looking like a relatively crude set of points. I see sketchfab doesn't do much better, and I understand LAS files are a minority (?) so I decided not to dig more on that field
    • the "look" of the 3D models is fully customizable. I opted for a metallic effect somewhat similar to sketchfab. The visuals can be fine-tuned / changed if needed as three.js is fully customizable
• the entry point modelViewer connects the two modules and initializes them

Any issues with implementation?

• I am unable to test on production or in a production-like environment
• it assumes the incoming files are defined as ExternalLinks and follow the interface protected/interfaces/DatasetExternalLinksInterface.php, if the requirement is different, for example, a different model such as protected/models/File.php, it should be relatively easy to adapt
• it assumes the site can fetch and load the incoming external links without cors issues, in short, cors is not handled in any way
• In Yii, what is the right way to access the current environment from within a .js file? For example to determine the isProduction value in protected/js/model-viewer/helpers/logger.js

Any changes to automated tests?

All tests pass

[kencho51]

Hi @luistoptal,

prasie: Was able to associate this https://sketchfab.com/3d-models/beet-03-c63d100fb76042ecab2ec87f73da323e stl file to dataset 100006, and display it on Chrome browser.

question: The 3D image cannot be shown in Safari nor Firefox browsers, can you help further look into it?

Safari

Firefox

[luistoptal]

@kencho51 looking at the images, it looks as if the CSS is not loaded. Because not only the 3D model is not shown but the buttons look as if the custom styles applied to them were not working. Can you try:

• re-run the ./up.sh script (but I assume you already did that)
• rebuild the .less styles using docker-compose run --rm less (if you run ./up.sh, it should already be done though)
• Disable Firefox and Safari browser cache and hard refresh the page

[kencho51]

Hi @luistoptal,

• Disable Firefox and Safari browser cache and hard refresh the page

After hard refresh/empty the cache in safari and firefox, the 3D image can be displayed now.

[kencho51]

Hi @luistoptal,

question: Would it be better to have acceptance tests/accessibility tests as there is an extra 3D Models tab and a 3D graph is displayed?
suggestion: For the 3D images source, we could put all the images in a s3 bucket and make them public, and then input the bucket url when trying to create external link, as below:

I have created a test bucket with 3 3D images:

1. https://s3.ap-northeast-1.wasabisys.com/test-gigadb-datasets/3d-models/100006/GeoB8502_865cm_Shell-4.obj
2. https://s3.ap-northeast-1.wasabisys.com/test-gigadb-datasets/3d-models/100006/beet_03.stl
3. https://s3.ap-northeast-1.wasabisys.com/test-gigadb-datasets/3d-models/100006/leaf_01.las

@rija, @pli888 , what do you think?

praise: I was able to deploy this PR to my staging, and display the 3D images from the above files stored in the s3 bucket.

[luistoptal]

@kencho51 that would work since the viewer simply loads and displays files accessible via public links, thanks for adding these files, now they can be used to update the tests

there is one test for the 3D models tab already present, it hadn't been updated

[luistoptal]

I extended the test that checks for the 3D Model tab

[pli888]

I moved the js widget to gigadb/app/client/js. Also, I checked the local deployment in Safari via a cloud crossbrowser testing tool and could see the models fine in Safari 18, the issue you report is from local dev or from staging?

The 3D model viewer functionality works on my dev, but there is a problem in my staging. When I go to a dataset page with a dataset that has a 3D model external link, I see a The asset "/gigadb/app/client/js/model-viewer" to be published does not exist error message. See https://gigadb-staging-mumbai.gigatools.net/dataset/100295.

[kencho51]

Hi @rija

@kencho51: have you tried with Safari, did it work for you?

It works on my Safari with version 18.3 (20620.2.4.11.5).

[rija]

Hi @luistoptal

praise: I saw that the 3D models tab is now in the style guide
praise: I've retested locally this PR, And all three models load correctly on Safari on my local dev.

(My pipeline is still building, I'll let you know the results from my AWS deployment later)

issue: However the contextual help and full screen callbacks still appear inline on Safari (including the example one from the style guide)

[luistoptal]

HI @rija

contextual help and full screen callbacks still appear inline on Safari

Can you plase add screenshots so I understand better what you see?

[rija]

HI @rija

contextual help and full screen callbacks still appear inline on Safari

Can you plase add screenshots so I understand better what you see?

Hi @luistoptal
don't worry, ignore my remark as after hard reset of the cache, it behaves correctly locally on Safari on the dataset page and on the style guide.

I'll let you know when I get your PR deployed on AWS

[rija]

Hi @luistoptal,

issue: Having deployed your PR to AWS, I'm now seeing the same problem as @pli888:

CException:

The asset "/gigadb/app/client/js/model-viewer" to be published does not exist.

Neither the web server nor the application server knows about that path, and they probably shouldn't anyway.

suggestion: can you pre-compile the model-viewer code into the top-level js/ directory (it is effectly our dist directory), so that is can be copied into both the web and application containers?
You would need to load the javascript in the page from that directory.
That's the approach we took for the File Upload Wizard. (that's why you can see fuw-1.0.0.js in the js directory, while the javascript source is under gigadb/app/client/web)

[luistoptal]

Hi @rija

So if I understand correctly, the folder /gigadb/app/client/js/model-viewer is not available in the dockerized production app?

will the following work?

$jsDir = Yii::getAlias('js/model-viewer'); // using the js folder instead of the gigadb folder
$jsUrl = Yii::app()->assetManager->publish($jsDir); // this will create a dir under the assets folder

Yii::app()->clientScript->registerScriptFile($jsUrl . '/index.js', CClientScript::POS_END, ['type' => 'module']);
?>

<script type="module">
  import { modelViewer } from "<?php echo $jsUrl; ?>/index.js";

  // code
</script>

or must I call the script directly from the js folder without using the php helpers? in which case, I do not understand how the assets folder works

[rija]

Hi @luistoptal,

I understand your confusion, sorry about that.
I forgot we were trying to use Yii assets mechanism here.

First, do not use the js/ directory for the javascript source code (although, to answer your question, it would fix the error) as it is not meant for that, it is used for the same purpose as the Yii assets directory (because I wasn't familiar with Yii assets when I was doing FUW).

You are right in realising that the problem is that the path /gigadb/app/client/js/model-viewer (which is the right place for the javascript source code to be) is not available in the in the dockerized production app.

So, given that you are using Yii assets, the correct fix is to make that path available to the dockerized production app.

We do so by copying the path into the Dockerfile for the production Web and application container by adding the following line:

COPY ./gigadb/app/client/js /gigadb/app/client/js

into ops/packaging/Production-Web-Dockerfile and ops/packaging/Production-Dockerfile

in the same section were the other COPY instructions are.

[luistoptal]

Hi @rija I added the copy command to the docker files as per your suggestion, thanks

[pli888]

@luistoptal Seems like there are conflicts with CHANGELOG.md and datatables.less. Can you rebase with latest develop branch code then I can test this PR again?

[rija]

Hi @luistoptal,

praise: Your last change to the dockerfiles fixed the asset publishing error, thanks.

issue: when loading the '.las' model (on any browser) I get the following CSP error the js console:

Content-Security-Policy: The page's settings blocked a worker script (worker-src) at blob:https://rija-staging.gigadb.host/ff12fa1f-6144-4fa4-a046-e14c8902478c from being executed because it violates the following directive: "child-src <https://www.protocols.io>](https://www.protocols.io) <https://hypothes.is>](https://hypothes.is) <https://www.rosaceae.org>](https://www.rosaceae.org) <https://sketchfab.com>](https://sketchfab.com) <https://codeocean.com>](https://codeocean.com) <https://tumormap.ucsc.edu>](https://tumormap.ucsc.edu)"

and the model-viewer shows that error:

Error: Failed to load worker las (#1 of 3) from https://unpkg.com/@loaders.gl/[email protected]/dist/las-worker.js.

suggestion: It sounds like there is a missing directive, maybe you could add a

; worker-src 'self' blob:

at the end (but within the double-quote) of the two lines starting with add_header Content-Security-Policy and add_header Content-Security-Policy-Report-Only respectively.

and I think we may also need to add https://unpkg.com to the script-src directive on both lines as it is not listed there.

[luistoptal]

Hi @rija I updated the nginx server config as you suggested

[rija]

Hi @luistoptal,

praise: the functionality now works for me in all web browsers
praise: I can see how ops/scripts/set_env_vars.sh can be useful and it's great a mention of it has been added in the docs
nitpick: I feel like the .gitlab-env-vars.example example file should go in ops/configuration/variables with the other variables related example/sample files

[luistoptal]

Hi @rija I relocated the example file as you suggested

[rija]

Hi @luistoptal,

based on the latest change, I'm happy to approve this PR

[kencho51]

Hi @luis,

Also, I have pushed a commit with a script to update env variables in gitlab. It might not be the best PR for this but for me it all stems from wanting to deploy this branch to production-like to test :) the script does not require any kind of changes to env vars anywhere (.env etc) if it is not used

Regarding this script which aims to set up variables for staging and live deployment:
question:(non-blocking) In order to execute this script, I have to manually update the GITLAB_PROJECT_ID and GITLAB_PRIVATE_TOKEN in the .secrets and also copy .gitlab-env-vars.example to .gitlab-env-vars?
suggestion: (non-blocking) Maybe create a block in SETUP_CI_CD_PIPELINE.md, for example, as below:

% cd gigadb-website
% ./up.sh 
% cp ops/configuration/variables/gitlab-env-vars.example .gitlab-env-vars
% ./ops/scripts/set_env_vars.sh

suggestion:(non-blocking) For var GITLAB_PRIVATE_TOKEN, you can get it from .env, while for the var GITLAB_PROJECT_ID, can it be replaced by PROJECT_VARIABLES_URL which is also in .env.

[luistoptal]

@kencho51 I applied your suggestions, thanks

[pli888]

praise: Functionality is now working on my staging and dev environments.
praise: New code looks good including new set_env_vars.sh script.

Happy to approve.

[rija]

when deploying, will need to run the following query:

insert into external_link (dataset_id, url, external_link_type_id) select dataset_id, location as url, 5 as external_link_type_id from file where location like '%.stl' or location like '%.obj' or location like '%.las' or location like '%.ply';