RFC: FEM packages structure

[goplayoutside3]

Overview

This is a discussion stemming from a Slack convo about PFE's structure as a single page app (SPA) and avoiding a similar fate in FEM. Some of the disadvantages to building a SPA are it being an large, unwieldy codebase for developers to maintain, and being forced to use client-side routing. A note from @eatyourgreens : Client-side routing immediately makes the site harder to use with assistive technology, so it would be raising, rather than lowering, barriers to participation.

There's a lack of documentation about how to move forward when adding more pages to FEM, for instance by adding more app packages to the monorepo or consolidating pages into one Next.js app. I think it's pertinent to document the discussion because the /user stats pages are being developed in FEM this year.

Research

Prior to opening this discussion I searched for existing documentation on the topic, but couldn't find any conclusions about the planned structure of app and lib packages in FEM. I looked in Github Issues, Discussions, PRs, ADRs, and the #fem Slack channel (which began after FEM's repo was created).

Some relevant PRs, but they don't document an overarching plan:

• Init of app-project package: Initial setup for project app #9
• Init of app-about package Start app-about #731 (and subsequent renaming to app-content-pages Finish up Publications #768)
• Init of lib-classifer Start classifier #29
• Init of lib-async-states Add asyncStates lib #140
• Init of lib-react-components Add react components package #53

Discussions:

• What to do about the lab app? What to do about the lab app? #2430 (Has no conclusion)
• Closest topic to this discussion I could find: Alternative project setup - routing and refreshes #2448
  • Mentions that "we're planning to abstract out the mobx User store"
  • Says that crossing an app boundary means requesting the user details again
  • No conclusion about overall structure

Context

For context, here’s a helpful diagram created by @mcbouslog that illustrates zooniverse.org’s pages. The current pages in FEM are:

• /about (currently app-content-pages)
• /projects (currently app-project)

Pages that will eventually be “moved over from PFE” by being built into FEM are:

• / (homepage)
• /admin
• /users
• /talk
• /lab
• /collections & /favorites
• /organizations
• /settings
• /inbox & /notifications
• A few more one-off pages for things like privacy notice, unsubscribe, etc

Proposed Option 1

Pages are their own separate apps in FEM

Continuing with the pattern of /about being in app-content-pages while /projects is contained in app-project, new pages should be built in their own app package. For instance, /lab will be built as app-lab. This way, a volunteer visiting a route like /lab will not be required to download all of the Javascript bundles needed for unrelated pages such as /about.

This strategy would not require major refactoring of the current structure. However, we are currently on outdated versions of React and Next.js, and I think upgrading FEM to the latest of each should be required before adding a new app package.

We use mobx for state management so, each app package will have their own store.

Each app will have their own Express server and middleware as needed.

Each app will need to be run locally separately when developing new features.

Proposed Option 2

Eventual combination of all pages into one Next.js app package in FEM:

The existing app-project and app-content-pages packages would be combined into one big app package, and all other pages added later on would go into that app.

This option would put FEM in the same position as PFE - one large SPA.

It would require refactoring of the current structure:

• Combination of some Mobx store items
• Reorganization of storybook
• Translation dictionaries that are currently in app-project app-content-pages
• Next.js config consolidated to one app
• Deployment config
• Sentry config

Action Items

The goal here is to document FEM’s overarching plan - specifically how new pages should be added. The upcoming user stats pages can be set as a precedent for future additions. Documenting this goal will educate future Zooniverse developers and managers.

• Please comment to discuss your opinion on FEM structure and add details about the implications of an option
• If there’s another structure that should be added as Structure # 3, please describe it

I plan to follow-up on this topic with more documentation after ZTM meetings about PFE/FEM routing and the zooniverse.org homepage!

[eatyourgreens]

zooniverse.org is a web site, not an app. Why not build it as a web site, rather than a JavaScript app?

[eatyourgreens]

The two options here aren't really options. Both describe a client-side React app. The /pages router uses client-side routing, building zooniverse.org as a JS React app that runs in the browser. Client-side routing immediately makes the site harder to use with assistive technology, so we're raising, rather than lowering, barriers to participation.

I'm asking why there isn't an option here to build a web site, with server-side routing between pages? Why is a client-side JS app presented as the only option?

[goplayoutside3]

Please bear with me while I’m doing my best to understand and consolidate FEM's architecture plan going forward. I’m looking for current team help as we can't ask the original FEM authors. I want this Github Discussion to lead to an ADR that communicates our overarching plan.

[goplayoutside3]

Client-side routing immediately makes the site harder to use with assistive technology, so we're raising, rather than lowering, barriers to participation.

Thank you I'll add this to the intro overview comment.

[eatyourgreens]

I think what was useful to me was realising that NextJS runs _app.js both on the server and in the browser. That sets up a React app that runs client-side in the browser.

You can see this on the Publications and Team pages, which download and execute several hundred k of JS, even though they are both completely static pages.

[eatyourgreens]

Worth adding that NextJS with the /pages router starts at 250k for the client JS bundle, before you even write any code.

The current recommendation for a performant web page is 350k of JS or less.

Would it be worth noting which page routes can be generated completely on the server, with purely static content?

[mcbouslog]

Would separate apps for different services be a third structure or would that be structure 2? I think I'm getting confused by the use of "packages" when I equate that with libraries, but importing all the major services as libraries into a root app (structure 2?) I'm not sure is much different than structure 1? I thought an option would be individual apps (like we have app-projects and app-content now) for other services like talk, lab (could include project admin), users (could include user admin) and a (as small as possible as it'd be the homepage) root app.

[eatyourgreens]

No, you're right. There's no real difference between the options here. Both proposals build zooniverse.org as a single client-side JS app, which we've already got with PFE.

If I'm reading this correctly, there's also a proposal to have a single global JS state in the browser, which we also have in PFE.

What's missing here is any background of what we learnt from PFE eg. the difficulties we had debugging global JS state across projects.

There's also no justification here for building zooniverse.org as a JS app. It's simply assumed that that's the only option available to us.

[goplayoutside3]

Thanks both, I'll add new language to the overview comment that makes a better distinction between a single global JS state versus building a web site, with server-side routing between pages.

There's also no justification here for building zooniverse.org as a JS app. It's simply assumed that that's the only option available to us.

Please bear with me while I’m doing my best to understand FEM's architecture plan going forward. For instance, I don't have an opinion on the best architecture option yet, and I'm asking for team help.

[goplayoutside3]

What's missing here is any background of what we learnt from PFE eg. the difficulties we had debugging global JS state across projects.

Is background information documented for reference already somewhere?

[eatyourgreens]

I don't think it is unfortunately, except for a sentence in the very first ADR saying that we'll break PFE's monolithic JS app into smaller, more manageable chunks.

[eatyourgreens]

I think the problem that @mcbouslog is describing, and the problem that this discussion is trying to solve, is the problem of common layout components across Zooniverse pages eg. the repeated user and localisation logic between the two current NextJS apps.

That problem is solved by layouts in the new NextJS app router.

[eatyourgreens]

I'd also be wary of building any new code on top of the pages router in Next 12. It's still supported in Next 13, but the new router seems to be the direction that React and NextJS are going in.

[goplayoutside3]

Thank you - I'll update the overview comment to include Next 13 and how it's different than the version of Next 12 we're relying on at the moment.

[eatyourgreens]

Regarding client-side routing with the /pages router, how do we envisage that working for individual projects? This still isn't clear to me.

We've discussed how URLs are routed on our servers, but what's the plan for routing when /projects/owner/project is a client-side route, handled by the NextJS router in the browser?

Is the plan to ignore next/link for links to /projects? If so, how will links within existing FEM projects continue to use client-side routing in order to maintain project app state while a volunteer is classifying on a project?

In PFE we maintain a list of projects that should client-side redirect to our CDN:

https://github.com/zooniverse/Panoptes-Front-End/blob/8bfa3990a85ea2f4064c1332e888479e84149bea/app/monorepoUtils.js#L12-L38

When a client-side route loads one of those projects, we force the browser to reload the page, requesting that URL from our CDN.

https://github.com/zooniverse/Panoptes-Front-End/blob/8bfa3990a85ea2f4064c1332e888479e84149bea/app/MonorepoRoute.jsx#L7-L13

How will this work in the browser when client-side routing for all projects is handled by the NextJS /pages router, rather than React router in PFE? Will /pages/projects/[owner]/[project] maintain a list of projects that should redirect? That would invert our existing setup, but seems impractical.

[eatyourgreens]

next/link, next/image and next/router can be used inside libraries. Declare next as a peer dependency, then import them as static imports.

That's fine as long as your library is intended only to be used inside a NextJS app (so you couldn't import them inside the Zooniverse React Components lib, or the classifier, both of which are designed to work outside of NextJS.)

[goplayoutside3]

Thanks I'll remove this as a pro/con from the overview comment.

[eatyourgreens]

Can keep various packages at different versions of tech like React, etc

This isn't true. All the libraries must use the same version of React and MobX as the root NextJS app. That's why we declare React, MobX etc. as peer dependencies in our libraries. package.json for each NextJS app determines the actual versions that are bundled into the browser JS.

[goplayoutside3]

Thanks I'll remove this as a pro/con from the overview comment.

[eatyourgreens]

👍 mobx store consolidated to one app.

This is a disadvantage. Your mental model, and the store state, has to expand from a single project and workflow to all available projects plus Talk, user stats and anything else that's kept in the global store.

I'd recommend keeping contexts small and scoped to where they're needed. Since React 16, React is designed to support multiple context providers. mobx-react recommends using React context providers to provide stores.

[eatyourgreens]

One example of this going wrong in PFE: there's a long-running bug where going from Project A to Project B can show you Project B Talk with the Project A header and navigation. This happens because PFE has a single global state, and the stored project isn't always updated properly as you navigate around the app.

Bugs like this could be avoided by having individual project contexts for each project, which is what we built with the NextJS project app.

[goplayoutside3]

Thanks I'll update the description of how we use mobx in app-project in the overview comment.

[eatyourgreens]

We have individual Express servers, with Express middleware, for each app at the moment. In addition, the project app has NextJS middleware to rewrite project URLs.

Will there be any problems consolidating those into a single server? They aren’t mentioned in either option 1 or option 2.

[goplayoutside3]

Thanks I'll update the overview comment to include mention of Express and middleware.

[goplayoutside3]

@eatyourgreens and @mcbouslog I've updated the Discussion overview to describe two distinct options.

As mentioned in the Action Items sections, the goal is to document FEM's overarching plan and set a precedent for how new pages should be added to the monorepo. Thanks for you patience since we can't ask the original FEM authors. I'm doing my best to use context clues and develop an understanding of the big picture implications for how we develop FEM going forward.

[goplayoutside3]

Option 3 Discussed at ZTM 2023

At ZTM this week we discussed a third option for FEM's app and lib structure, and will work toward this third option as follows (also see project board: https://github.com/orgs/zooniverse/projects/41/).

app-root

• Create app-root with Next.js 13's App Router. This root structure is contingent on Next 13's ability to modularize the Zooniverse website. For instance, when a volunteer visits an /about page, the Javascript bundle downloaded by their browser should not contain /projects or any other unrelated route's code (Some dev research on details still needed).
• This package contains the homepage. Question: Does it contain the basic layout of ZooHeader + ZooFooter, too??
• It handles routing and pulls in code from the appropriate libraries (such as /users or /projects).

app-project --> lib-project

• Once app-root exists, the current app-project can be "converted" to lib-project.
• lib-project handles the UI/UX for /projects route.
• The suggested process for completing the conversion of app-project to lib-project is to create a lib-project package in FEM, build it without Next.js, then once it's ready switch the /projects route to go through app-root + lib-project, and finally delete the old app-project package.

app-content-pages --> lib-content-pages

• Once app-root exists, the current app-content-pages can be "converted" to lib-content-pages.
• lib-content-pages handles the UI/UX for the /about route.
• The suggested process for completing the conversion of app-content-pages to lib-content-pages is to create a lib-content-pages package in FEM, build it without Next.js, then once it's ready switch the /about route to go through app-root + lib-content-pages, and finally delete the old app-content-pages package.

lib-users

• This library will be built to handle the UI/UX for /users/stats and /groups/stats (current funded work).
• Eventually can handle other /users routes.

Existing packages that do not need refactoring:

• lib-classifier
• lib-react-components
• lib-panoptes-js
• lib-async-states
• lib-grommet-theme
• tools-standard

Bonus idea:
We chatted about a library idea that would handle the ZooHeader UI/UX components for registration, signin, notifications, and messages. The motivation behind this type of library is to avoid repetition of the react components involved in those features. However, should the ZooHeader simply be a part of the app-root, and therefore no repetition?

[goplayoutside3]

Connecting some documentation:

The initial creation of app-root has been merged: #5423
There is a project board with next steps in app-root development: https://github.com/orgs/zooniverse/projects/43

Important takeaways from PR 5423:

• In app-root we’re going to build a logged out homepage on the server, and then do client-side auth with existing lib-panoptes-js to show user-specific content.
• Libraries like Grommet and styled-components don’t have support for React Server Components (RSC) yet and they might never. For app-root we’re going ahead with Next.js 13's opinionated default to RSC, but going to try to work around those incompatible libraries via 'use client'. For now that means we're forcing almost all FEM code to run primarily on the client until there's more support for RSC.
• app-root is configured for ESM-only, however, simliar to RSC if we get really stuck using libraries that don't support ESM, then we’ll have to revert app-root config back to Common JS.
• Ideally, we'd like to do a deployment of app-root as a test to a url such as fe-static.preview (old test url that’s not used anymore ) in a very similar way as fe-project-branch.preview. This way can test K8s config with App Router.

[goplayoutside3]

"About Zooniverse" pages (content pages)

The About Zooniverse pages currently exist in app-content-pages, a Next.js 13 app using the Pages Router. There are new designs for the About Zooniverse pages (Figma) that need to be implemented in FEM. In order to move FEM toward the app-root structure, we plan to simultaneously develop app-root, but also split app-content-pages into two packages:

1. lib-content-pages - a React app that contains all page components except the ZooHeader and ZooFooter. It uses Contentful for Publications and Team data (and maybe more pages?). It will be imported into:
2. app-content-pages - an almost replica of app-root with Next.js 13 App Router and ZooHeader/ZooFooter layout. It will still be deployed to production at fe-content-pages.zooniverse.org. This app will exist until the "full routing switchover" happens next year when we intend to use app-root as the root domain instead. This will give us a chance to try out App Router in production prior to the full switchover.

Note that PFE's router.jsx contains redirect functions for zooniverse.org/about/team and zooniverse.org/about/publications to FEM because those two pages are in FEM's app-content-pages. There will likely be an interim period when PFE's router is used to redirect more pages (/about/resources, /about/faq) to FEM, and then eventually all /about pages.

Relevant project board: https://github.com/orgs/zooniverse/projects/40