Add frontend for zarr import #6383

Dagobert42 · 2022-08-08T10:08:31Z

URL of deployed dev instance (used for testing):

https://addfrontendforzarrimport.webknossos.xyz/

Steps to test:

add and import zarr data:
- without credentials
  https://data-humerus.webknossos.org/data/zarr/scalable_minds/l4dense_motta_et_al_demo/segmentation
  https://data-humerus.webknossos.org/data/zarr/scalable_minds/l4dense_motta_et_al_demo/color
- https:// with credentials (ask @fm3)
- s3:// with credentials (ask @fm3)
merge different data layers together
labels in the credentials fields should change according to the protocol

Open:

add optionally displayable log
check that both datasources have same voxel size else warning when merging
check if dataset already exists
make layers unique without overriding old layers of same name

Issues:

fixes Allow import of remote Zarr datasets #6272
builds on Import Remote Zarr Datasets #6335

(Please delete unneeded items, merge only when none are left open)

Updated (unreleased) changelog
Ready for review

…some Toast UX idea

philippotto

cool stuff :) I left some feedback, but nothing major.

frontend/javascripts/admin/admin_rest_api.ts

frontend/javascripts/admin/dataset/dataset_add_zarr_view.tsx

fm3 · 2022-08-11T10:22:07Z

As discussed I pushed two changes to the base branch zarr-import

send the report as single string
catch exceptions for invalid uris

Feel free to merge that base branch into yours and adapt the front-end to the single-string report :)

frontend/javascripts/admin/dataset/dataset_add_zarr_view.tsx

normanrz · 2022-08-15T11:31:57Z

Some notes, can be a follow-up:

Nobody should see the datasource json. We should show a slimmed-down nice representation of it:
- Dataset name (editable)
- Dataset scale (editable)
- List of Layers
  - Name (editable)
  - Category (editable)
  - Bounding box (read-only)
  - List of mags (read-only)
  - Optional largest segment id (editable)
I would introduce radio buttons: No authentication, Basic auth. And only show user/pass when selecting basic auth. Same for S3 (Anonymous, With Credentials). Otherwise, it is too much information that needs to be put in.
Not a fan of the buttons stretching all the way from left to right. I feel they are easy to miss. Make them regular-sized and right-aligned.

Co-authored-by: Norman Rzepka <[email protected]>

Dagobert42 · 2022-08-17T09:05:22Z

I opened follow-up #6409 for making the datasource representation and editing prettier. The other suggestions I've adapted like so:

fm3 · 2022-08-17T09:07:55Z

Note that the backend currently does not support anonymous s3 access (compare #6401 ) – so maybe that should be disabled in the front-end too until that follow-up is solved

Dagobert42 · 2022-08-17T09:23:12Z

Ok the frontend will reflect this as such for now when s3:// is being used:

philippotto

nice 👍 code looks good and it works for me well.

* [WIP] import remote zarr datasets * read zarr header * make (rudimentary) datasource * WIP omeNgff import * detect axis order, voxel size and mag from omengff * add assertions, return Foxes * allow multiple layers in one ngff header, return all mags * pass name + voxel size, respect space units * pass credentials to mags, make layer names unique, add report * fix bbox detection, debug chunk reading errors * relax axis order assertion * allow inputting uris directly pointing to .zattrs,.zarray,.zgroup. experiment with array order * ZYX * error messages for everything that can go wrong * f2 * lint * Store AxisOrder with Mag, fix NonLocalReturn, fix fortran-order flip * fix no-partial-copying-needed case, fix unit axes, read element class independent of endianness * clean up * match error * add put route for datasource * add provider for http (not https) * increase https file system provider timeouts * pretty * add explaining comment for nested list * makedirs for new datasource * send correct error * clean up chunk loading, catch exceptions * send report as single string, catch exception for invalid uris * Detect Segmentation Layers Heuristic, Assert valid element class * include channel in AxisOrder for forward-compatibility * allow dots in dataset names (if they are not at first or last position) * Add frontend for zarr import (#6383) * basic zarr import * merge data layers of multiple sources * use credentials in exploreRemote improve frontend * [suggestion] minor wording improvement while totally not stealing awesome Toast UX idea * add rudimentary error for wrong credentials * add referrer modal onAdded * remove multiline URL exploration until UX concept is discussed * return and display exploration report * updated (unreleased) changelog * check voxel sizes and potentially warn when merging * give unique layer names to duplicates * ensure largest segment id exists * Edit s3 credentials label Co-authored-by: Norman Rzepka <[email protected]> * add radio buttons for authentication * make s3 always require credentials Co-authored-by: Florian M <[email protected]> Co-authored-by: Norman Rzepka <[email protected]> * update changelog pr number Co-authored-by: Jonathan Striebel <[email protected]> Co-authored-by: Arthur Hilbert <[email protected]> Co-authored-by: Norman Rzepka <[email protected]> Co-authored-by: Jonathan Striebel <[email protected]>

basic zarr import [WIP]

57d4f40

Dagobert42 force-pushed the add-frontend-for-zarr-import branch from 782609b to 57d4f40 Compare August 8, 2022 12:02

fm3 changed the base branch from master to import-zarr August 8, 2022 12:44

Dagobert42 added 8 commits August 8, 2022 17:04

merge data layers of multiple sources

80ccde8

use credentials in exploreRemote improve frontend

1506bd0

[suggestion] minor wording improvement while totally not stealing awe…

bb8f952

…some Toast UX idea

improve datasource storing, add Toast to view on import

52011da

add rudimentary error for wrong credentials

3821552

add referrer modal onAdded

634db00

Clean up + look and feel

5605896

remove multiline URL exploration until UX concept is discussed

49a7d6e

Dagobert42 requested a review from philippotto August 10, 2022 15:07

Dagobert42 added 3 commits August 11, 2022 10:48

fix logic error

b5fdd27

return and display exploration report

9dc78b4

updated (unreleased) changelog

641692d

philippotto requested changes Aug 11, 2022

View reviewed changes

check voxel sizes and potentially warn when merging

c35d0b5

philippotto reviewed Aug 11, 2022

View reviewed changes

frontend/javascripts/admin/dataset/dataset_add_zarr_view.tsx Outdated Show resolved Hide resolved

Dagobert42 added 6 commits August 11, 2022 16:05

minor code review changes

49db30e

Merge branch 'import-zarr' into add-frontend-for-zarr-import

e48eebf

fix logic in voxel size check

cd66733

code refactoring, adapt to log being returned as single string + pretty

0ebc007

handle possible storing errors better

8e23f04

fix explore log display

0592d02

Dagobert42 changed the title ~~[WIP] Add frontend for zarr import~~ Add frontend for zarr import Aug 12, 2022

philippotto reviewed Aug 12, 2022

View reviewed changes

frontend/javascripts/admin/dataset/dataset_add_zarr_view.tsx Outdated Show resolved Hide resolved

Dagobert42 added 3 commits August 12, 2022 10:33

more small review changes

0f6f7f3

give unique layer names to duplicates

93c6b52

ensure largest segment id exists

df52765

Merge branch 'import-zarr' into add-frontend-for-zarr-import

e26ae15

normanrz reviewed Aug 15, 2022

View reviewed changes

frontend/javascripts/admin/dataset/dataset_add_zarr_view.tsx Outdated Show resolved Hide resolved

frontend/javascripts/admin/dataset/dataset_add_zarr_view.tsx Outdated Show resolved Hide resolved

Merge branch 'import-zarr' into add-frontend-for-zarr-import

ceae0b7

fm3 and others added 6 commits August 16, 2022 08:45

Merge branch 'import-zarr' into add-frontend-for-zarr-import

4fed0d1

Merge branch 'import-zarr' into add-frontend-for-zarr-import

57a9904

Edit s3 credentials label

70fe515

Co-authored-by: Norman Rzepka <[email protected]>

Edit s3 credentials label

ae473eb

Co-authored-by: Norman Rzepka <[email protected]>

add radio buttons for authentication

c7b4e2c

make buttons smaller and right aligned

e1f3371

make s3 always require credentials

eceb743

Dagobert42 requested a review from philippotto August 17, 2022 13:06

Dagobert42 added 3 commits August 17, 2022 15:28

small wording changes

9d0f087

show json error when storing

ed6f999

prettier

1189592

philippotto approved these changes Aug 22, 2022

View reviewed changes

Dagobert42 merged commit 957e1b3 into import-zarr Aug 22, 2022

Dagobert42 deleted the add-frontend-for-zarr-import branch August 22, 2022 12:07

fm3 mentioned this pull request Aug 23, 2022

Import Remote Zarr Datasets #6335

Merged

33 tasks

philippotto mentioned this pull request Aug 30, 2022

Add JSON schema for zarr datasources #6434

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add frontend for zarr import #6383

Add frontend for zarr import #6383

Dagobert42 commented Aug 8, 2022 •

edited

Loading

philippotto left a comment

fm3 commented Aug 11, 2022

normanrz commented Aug 15, 2022

Dagobert42 commented Aug 17, 2022 •

edited

Loading

fm3 commented Aug 17, 2022

Dagobert42 commented Aug 17, 2022

philippotto left a comment

Add frontend for zarr import #6383

Add frontend for zarr import #6383

Conversation

Dagobert42 commented Aug 8, 2022 • edited Loading

URL of deployed dev instance (used for testing):

Steps to test:

Issues:

philippotto left a comment

Choose a reason for hiding this comment

fm3 commented Aug 11, 2022

normanrz commented Aug 15, 2022

Dagobert42 commented Aug 17, 2022 • edited Loading

fm3 commented Aug 17, 2022

Dagobert42 commented Aug 17, 2022

philippotto left a comment

Choose a reason for hiding this comment

Dagobert42 commented Aug 8, 2022 •

edited

Loading

Dagobert42 commented Aug 17, 2022 •

edited

Loading