RFC: Multi-image Subjects + Flipbook, MultiFrame, and SubjectGroup Viewers

[goplayoutside3]

Overview

I'd like to open a discussion about direction of development for the new FEM Flipbook Viewer. Some features that are handled as part of PFE's flipbook already exist in other viewers built for FEM. An ADR will follow this RFC.

Background Info

Multiframe Viewer

• Complete
• Invision Design - Ideal for portrait oriented subjects
• FEM Project using it: Corresponding with Quakers
• Storybook: MultiFrameViewer
• Features of this viewer
  1. Navigation between frames via thumbnails
  2. Navigation with next and previous buttons
  3. Drawing tool marks or transcription annotations are not saved/consistent between frames
  4. Image Toolbar features (pan, zoom, rotate, invert, etc) are reset between frames

Subject Group Viewer

• Complete?
• Invision Design - Ideal for subjects that have square dimensions?
• Live project using it: ?
• Storybook: SubjectGroupViewer
• Features of this viewer
  1. Group of images per subject are displayed in a grid
  2. Zoom/pan applies to all images simultaneously

Flipbook Viewer

• In Development
• Invision Design - Ideal for landscape oriented subjects
• PFE project using it: Wildwatch Burrowing Owl
• Features planned for this viewer
  1. Navigation between frames via thumbnails
  2. Navigation with next and previous buttons
  3. Drawing tool marks ARE saved/consistent between frames
  4. Image Toolbar features (pan, zoom, rotate, invert, etc) are NOT reset between frames
  5. A play/pause button loops through all frames of a subject
  6. A speed control changes the playback speed of the loop

Considerations

1. PFE's flipbook enables switching between flipbook and a "separate frames" view.

• Example project: Wildcam Gorongosa
• PFE Lab option "allow users to choose between flipbook and separate frames" is checked
• Button in the lower left corner of flipbook viewer switches the view to a column display

2. Drawing tools + flipbook viewer.

• Example project: Power to the People
• PFE Lab option "clone markers in all frames" is checked
• Drawing a box on any frame means the drawn mark will always display in the same spot when looping through frames

PFE lab options mentioned above:

Proposition for FEM architecture

1. In the case of "allow users to choose between flipbook and separate frames" being checked/enabled, the classifier UI will include a button that when clicked switches between FEM's Flipbook Viewer and FEM's Subject Group Viewer. This would require updating the Subject Group Viewer code because it was built a couple years ago. Alternatively, a completely new viewer could be built in FEM for displaying a multi-image subject as "separate frames" (i.e SeparateFramesViewer component).

2. When "clone markers in all frames" is checked/enabled FEM will use the Flipbook Viewer. When unchecked/disabled, FEM will use the Multiframe Viewer. My logic in this proposition is that if a project team would like marks/annotations unique to each frame for a multi-image subject, there's not a use case for the looping features of the flipbook. If a project is wanting annotations unique to each frame AND the looping feature, the proposed Video Annotator for Solar Jets is a better fit.

3. Addressing the difference in design per portrait or landscape oriented subjects, I think for cases when a project uses the Multiframe Viewer with landscape oriented multi-image subjects, the viewer's thumbnail navigation could switch to display below the subject and it would be a minor dev lift.

[eatyourgreens]

The subject group viewer is actually a group of subjects: each image is from a different subject in a subject set, with a different subject ID. It's used by Galaxy Zoo: Weird and Wonderful:
https://www.zooniverse.org/projects/zookeeper/galaxy-zoo-weird-and-wonderful/classify/workflow/18510

In Talk, each individual subject links back to its own Talk page.
https://www.zooniverse.org/projects/zookeeper/galaxy-zoo-weird-and-wonderful/talk/4510/2538533

Here are some other projects that have used subjects with multiple images in different ways:

• Etch-A-Cell ER (defaults to the third frame of five for each subject. You're only supposed to draw on the third frame.)
• Solar Stormwatch (Draw around the storm in all three frames, to get an idea of how it's evolving over time.)
• Space Warps DES (View images either side-by-side or as a slideshow. Drawing on one frame draws on all three frames.)

Hopefully those are useful as examples.

[eatyourgreens]

One more example: Chimp & See subjects are each a video plus an array of still frames from the video.

[lcjohnso]

In agreement with Jim's point: unless @shaunanoordin or others speak otherwise, I think we should view the Subject Group Viewer as entirely stand-alone for its special use case.

[mcbouslog]

Linking to Backyard Worlds: Planet 9 - multiple frames per subject, drawing tool marks are not cloned across frames

[eatyourgreens]

Slightly related: I’ve also reorganised the classifier to make it easier to see what types of subjects are supported.
https://github.com/zooniverse/front-end-monorepo/tree/master/packages/lib-classifier/src/store/subjects

[eatyourgreens]

Do we want to place a limit on how many images can be included in a subject here?

front-end-monorepo/packages/lib-classifier/src/store/subjects/ImageSubject/ImageSubject.js

Lines 5 to 9 in f2a16fd

	const ImageSubject = types
	.refinement('ImageSubject', Subject, subject => {
	const counts = createLocationCounts(subject)
	return subject.locations.length > 1 && counts.images === subject.locations.length
	})

[lcjohnso]

Thanks Jim for the above examples -- we should try to group these examples (and fill out with DB query data) into categories of shared needs from the viewer. Also: lets keep them coming! Feel free to add more in a thread below (especially those that reveal new feature needs).

[eatyourgreens]

I think we also shouldn’t feel like one viewer needs to offer all these different functions. That became difficult to manage in PFE.

I think it would be fine to have a viewer specifically for camera trap workflows, just like there’s a viewer for transcription workflows.

[seanmiller26]

Addressing the difference in design per portrait or landscape oriented subjects, I think for cases when a project uses the Multiframe Viewer with landscape oriented multi-image subjects, the viewer's thumbnail navigation could switch to display below the subject and it would be a minor dev lift.

A big reason to have the Flipbook controls below the subject is to differentiate it from the Multiframe viewer. The Multiframe viewer was made with Transcription projects in mind, so the majority of those subjects are pages. Not sure if any future transcription projects would have a landscape oriented subject, but who knows.

[eatyourgreens]

The operational reports for Operation War Diary were landscape, but it seems rare.

[mcbouslog]

I think a question to consider is how the separate frames view should be defined in FEM - as it's own (separate frames) subject viewer or a functionality within the flipbook viewer.

I'm not sure if an actual live/launched workflow exists that is utilizing the following, but in theory a workflow could exclusively be separate frames - like this staging workflow - and if we want to support that in FEM then I think we'd have to create a standalone separate frames viewer? But if no workflow is or likely will utilize separate frames exclusively, then I suppose it wouldn't make sense to create a standalone separate frame viewer.

How many flipbook viewer workflows allow the toggle to separate frames, and of those how do they breakdown between showing one column of separate frames vs 2 columns vs 3 columns? Those answers might help clarify what functionality we do or don't want to port from PFE to FEM.

[goplayoutside3]

Thanks all for the example projects and input! Here's the remaining questions/discussion I see so far:

1. Since the Subject Group Viewer is not adaptable to a multi-image subject, we'll need to build a new Separate Frames Viewer in FEM. The "separate frames" view in PFE means each image has its own toolbar for zoom, etc (see Mark's staging workflow). Do we have a design for this in FEM? Is this a feature we want to bring to FEM?

2. The initial features I listed for the planned Flipbook seem to cover most projects that use it in PFE, with a few exceptions. How do we want to address the exceptions? For instance, if we plan for FEM's Flipbook Viewer to be the catch-all for multi-image projects, the full list of features would include:

   1. Navigation between frames via thumbnails
   2. Navigation with next and previous buttons
   3. Drawing tool marks are saved/consistent between frames when "clone markers in all frames" is checked
   4. Drawing tool marks are NOT saved/consistent between frames when "clone markers in all frames" is unchecked
   5. Image Toolbar features (pan, zoom, rotate, invert, etc) are NOT reset between frames when "clone markers in all frames" is checked
   6. Image Toolbar features (pan, zoom, rotate, invert, etc) are reset between frames when "clone markers in all frames" is unchecked
   7. A play/pause button loops through all frames of a subject
   8. A speed control changes the playback speed of the loop
   9. There's a button to change to "separate frames" view

   (Features of the Multiframe Viewer are all included in that list, so is the Multiframe Viewer a special case for portrait oriented transcription projects only?)

[eatyourgreens]

The "separate frames" view in PFE means each image has its own toolbar for zoom, etc (see Mark's staging workflow).

Space Warps would prefer one zoom control for all three frames, but PFE’s subject viewer didn’t allow us to implement that. I’d be cautious about copying PFE. It doesn’t always reflect what a project team wanted.

[seanmiller26]

each image has its own toolbar for zoom, etc (see Mark's staging workflow). Do we have a design for this in FEM?

Designs for both side-by-side image options - one with individual toolbars and one with a shared toolbar

[lfortson]

The staging workflow above that Mark notes could be useful for what we had in mind for a "slices" viewer for the biomed projects.

[goplayoutside3]

A summary from today's call:

Features of Flipbook

Initial Effort

1. Tabbing between frames via thumbnails
2. Changing current frame with next and previous buttons
3. A play/pause button loops through all frames
4. A speed control changes the playback speed of the loop
5. Image toolbar (pan, zoom, invert, etc) are not reset between frames (this is consistent with PFE)

Next Effort (either an enhancement, or a completely different viewer component)

1. Drawing tools on top of flipbook viewer
2. Handing the "clone markers in all frames" lab option
3. A button to change between flipbook view and separate frames view

Flipbook's Separate Frames View

• This is a component specific to the flipbook viewer
  • Decision is a result of not wanting to reload a totally new viewer when a volunteer is halfway through annotation and decides to change to separate frames view or flipbook view
• Features and design will be the same as PFE
  • When "show flipbook view" is selected in "Choose how to display multiple images" lab dropdown, separate frames view is a 1-column layout and each image has its own toolbar

Considerations for the future

When "show separate frames" is selected in "Choose how to display multiple images" dropdown, additional options of 1 row or multi-column grid needs to be handled. This config scenario communicates that a project team wants a "Side by Side" viewer for multi-image subjects. @lcjohnso I think this is a good case for querying how many projects exist with the following config:

• "Show separate frames" is selected in "Choose how to display multiple images" dropdown AND "Allow users to choose flipbook or separate frames" is checked

If a project team intentionally displays multi-image subjects as separate frames first, do they really need a button to switch to the flipbook? If the answer is no, then the case where "show separate frames" is selected in "Choose how to display multiple images" dropdown could tell FEM to display a totally different viewer than flipbook, and we can separate out concerns for grid layouts and shared toolbars. I'll even argue that the "Allow users to choose flipbook or separate frames" checkbox should be disabled when a team wants separate frames as the initial view.

A follow-up thought: The above description highlights how the PFE lab design corresponds to PFE's architecture of "one subject viewer + many many config options". FEM's architecture is "many subject viewers + each have minimal config". Should we be making more of an effort to align the FEM lab used by teams once their project is migrated with FEM's subject viewer architecture?