Upgrade VZE to NextJS #1603

johnclary · 2024-11-15T00:25:14Z

Associated issues

Vision Zero Editor Tech Refresh atd-data-tech#19576

Here is the working prototype of the VZE in NextJS! I am envisioning that folks spend time with the code and add their questions/comments via review, and from there we convene a meeting to discuss next steps and scope issues.

Testing

Star your local stack and apply migrations and metadata
Start the nextjs app: see the README for instructions.
If you've tested this app previously, you may need to remove the old version of crashesListViewQueryConfig from your localStorage
What to test: you should be able to use the app without breaking it! Moreover, the current list of working features is in the readme, here.
Code review: all feedback is welcome, and especially related to the libraries i've chosen and overall project structure.
I have quite a few doubts and questions about the designing of the types, and a very important todo (i think) is to add in Zod schema validation to generate our core types and wrap all our API requests.

Ship list

Check migrations for any conflicts with latest migrations in main branch
Confirm Hasura role permissions for necessary access
Code reviewed
Product manager approved

johnclary · 2024-11-15T01:28:12Z

app/app/layout.tsx

+  children: React.ReactNode;
+}) {
+  return (
+    // <Html lang="en" style={{fontSize: "14px"}} data-bs-theme="dark">


i left this here bc the dark theme is glorious to dev on. would be fun to spin off an issue to make a UI toggle for it 🌚

after switching back to light mode a while back, maybe it is time for me to find my way back to the dark side. 😈

johnclary · 2024-11-15T01:30:17Z

app/app/layout.tsx

+}) {
+  return (
+    // <Html lang="en" style={{fontSize: "14px"}} data-bs-theme="dark">
+    <html lang="en" style={{ fontSize: "14px" }}>


my chrome browser's font size is 16, and the ACC designs go down to 12, maybe even smaller. it's an accessibility issue that needs more consideration 👀

fwiw this is what MUI does - https://mui.com/material-ui/customization/typography/#font-size which also seems to be the default of Bootstrap too - 1rem. Idk why the font looks so large with this override removed. But, i agree that it is something that needs more thought and can be adjusted globally later.

Ah, thanks pointing to MUI's approach—interesting that they default to 14px.

My understanding of the way it works is that 1rem = whatever the browser default is, unless specified in the html tag, in which case 1rem becomes the specified value.

johnclary · 2024-11-15T01:35:06Z

app/components/ChangeLog.tsx

+ * Each object in the array takes the format
+ * { field: <property name>, old: <old value>, new: <new value> }
+ */
+const getDiffArray = <T extends Record<string, unknown>>(


getting this typed correctl was a lot of trial and error, even though the result looks simple. these kinds of data-munging tasks are what i've found to be the most onerous to convert to typescript.

oof yeah, glad you figured this out! i need to learn more about this Record type.

It did make me think about whether we would want to add a linter rule to enforce no any type. https://typescript-eslint.io/rules/no-explicit-any/ I think that code reviews would keep us on track, but it could be worth thinking about now.

yes, i'm 100% on board for no any types allowed—and it appears to be enabled by default, at least in local dev.

const myVar: any = 2;

# npm run build 10:13 Error: Unexpected any. Specify a different type. @typescript-eslint/no-explicit-any

johnclary · 2024-11-15T01:43:59Z

app/components/ChangeLog.tsx

+ * Prettify the user name if it is `cris`
+ */
+const formatUserName = (userName: string): string =>
+  userName === "cris" ? "TxDOT CRIS" : userName;


i just want to point out that this function has a return type defined (string). typescript will infer the return type based on what your code returns, but as a best practice i think we want to strive for explicitly naming the return type.

there are some spots where i failed to do this and i'm keeping an eye out for them.

i like it! this is something that i could learn about for sure.

johnclary · 2024-11-15T02:18:26Z

app/contexts/Auth.tsx

Auth is implemented fully client-side, like the existing VZE.

basically, the flow is that a user logs into Auth0, the JWT is stored in localstorage, and the app uses that JWT to make calls to the Hasura API. Hasura validates the JWT and permissions. So our app is just a passthrough for the authentication. It manages the task of getting a token and refreshing it.

Auth0 does have a server-side lib, nextjs-auth0, which is specific to the app router. The big difference with that approach is that our own nextjs server (Netlify or Vercel) actually validates the JWT that is exchanged. To make that possible, we need to store our JWT secret in the app environment. (whereas the client-side setup gets by on the publicly-shareable env vars).

If we want to authenticate a server-side function, we'll need to adopt the nextjs-auth0 package. For example, if we wanted to replace the CR3 API. But for our current needs there is no benefit to using the nextjs Auth0 package, and in fact it adds the overhead of needing to manage a very sensitive secret in our local setup.

That's my digest of the tradeoffs as I understand them.

johnclary · 2024-11-15T02:25:25Z

app/contexts/Auth.tsx

@@ -0,0 +1,29 @@
+"use client";
+import { Auth0Provider } from "@auth0/auth0-react";


React context is a client-only feature, which is where the change in mental model really hit home for me.

If you're using a 3rd-party library that relies on context (like Auth0), you need to wrap it in "use client", unless the author does it for you. Here are the nextjs docs that discuss that.

johnclary · 2024-11-15T02:28:58Z

app/next.config.mjs

+        permanent: true,
+      },
+    ];
+  },


this file was generated by create next app and i added this redirect.

johnclary · 2024-11-15T02:32:08Z

app/package.json

+  },
+  "dependencies": {
+    "@auth0/auth0-react": "^2.2.4",
+    "@types/lodash": "^4.17.13",


the only lodash function we're using at the moment is isEqual, and as far as i can tell the bundler is doing a great job tree-shaking it.

i don't know what state the lodash project is in. i think v5 has been in the works for a very long time...

johnclary · 2024-11-15T02:33:35Z

app/package.json

+    "next": "14.2.13",
+    "react": "^18",
+    "react-bootstrap": "^2.10.4",
+    "react-datepicker": "^7.5.0",


this datepicker gets the job done, but it looks really dated (ha ha). i have not found a framework-agnostic alternative but didn't look very hard.

ah yes, here we go: https://github.com/gpbl/react-day-picker. this one is used by shadcn/ui.

as long as we don't go with the one that I picked for VZV! 🙈

johnclary · 2024-11-15T02:38:10Z

app/utils/formatters.ts

@@ -0,0 +1,24 @@
+import { format, parseISO } from "date-fns";
+
+// const formatCostToDollars = (cost: number | null) =>


i left this here because we'll need it to render comp costs on the location details pages

johnclary · 2024-11-15T19:27:50Z

app/components/ChangeDetailHeader.tsx

Most of the change log was copied verbatim from VZE, with the exception that it now renders even if there are no change records at all (fixing a weird quirk of VZE local dev where the change log is hidden)

johnclary · 2024-11-15T19:28:29Z

app/env_template

our env files are no longer committed to the repo 🎉

johnclary · 2024-11-15T19:29:12Z

app/styles/global.scss

we should be able to accomplish most of our theming here. bootstrap docs are pretty good about how to accomplish this, but getting all of the sass variable overrides correct will take some work.

johnclary · 2024-11-15T19:29:29Z

app/tsconfig.json

i did not create this file 🤖

johnclary · 2024-11-15T19:30:14Z

app/components/DataCard.tsx

+/**
+ * Generic component which renders editable fields in a Card
+ */
+export default function DataCard<T extends Record<string, unknown>>({


This component replaces the DataTable component in VZE. The only significant difference is that it fetches lookup table values when a lookup field is toggled into its edit state, as opposed to fetching all lookup values when the crash details page loads.

The lookup queries are cached, and in general are quick to load, but we need to test this with our power users, who may notice a slight performance degradation.

johnclary · 2024-11-15T19:34:20Z

app/types/types.ts

+ * The base definition of a database column referenced by our app — where
+ * <T> should be one of our core record types (Crash, Person, etc)
+ */
+export type ColBaseDef<T extends Record<string, unknown>> = {


ColBaseDef is the smallest building block of our types coming out of the database, and we can use them to build up any time of record: crashes, units, etc.

Also, you can see that I have extended from Record<string, unknown>> type, and i do that in many places. I think that's fine and good, but because it is used so often across the codebase we should probably give it a name and export it. something like <RecordWithStringKeys>. that's all it is: an object with keys that are always strings.

this will make more sense after reading about the Record utility type: https://www.typescriptlang.org/docs/handbook/utility-types.html#recordkeys-type :)

johnclary · 2024-11-15T19:37:11Z

app/types/types.ts

i started collecting most types here, but have also left component-specific types and interfaces near to their components. i don't feel strongly about how to organize these, but i do think we'll want to split out entity-specific types like crash, unit, etc. into their own files.

I also think that bringing zod into the mix might significantly change how we write out these types, so i didn't want to overthink it. See my note in zodExample.ts.

johnclary · 2024-11-15T19:46:13Z

app/utils/graphql.ts

@@ -0,0 +1,194 @@
+import { useState, useCallback, useEffect, useMemo } from "react";
+import useSWR, { SWRConfiguration } from "swr";


all of our data fetching is managed with SWR and graphql-request.

SWR is a Vercel-managed library designed specifically for client-side data fetching, and I really like working with it.

The Apollo setup for nextjs app router looks fairly involved, and i haven't messed with it. I'm open to it, but I stuck with SWR since it's familiar to me and is part of the nextjs ecosystem.

i think swr is great, and I don't see anything it doesn't cover that we got out of Apollo. imo Apollo client probably gives more than we need, and we also ran into some weird problems with fetching data for csv exports. Actually, lazy querying is something that we'll probably need, but maybe it is covered?

yep, as far as i have been able to test it, we can do the lazy querying with the conditional data fetching functionality and alternatively the useSWRInfinite hook — which i was recently looking at for paginating through the users API.

johnclary · 2024-11-15T19:51:11Z

app/utils/graphql.ts

+   * Dont refetch on network recon
+   */
+  revalidateOnReconnect: false,
+};


these are just my opinions to cut back on unnecessary API calls—SWR really loves to make API calls. i encourage y'all to check out the various settings available 🧪

johnclary · 2024-11-15T19:52:15Z

app/utils/graphql.ts

+     * dicussion here: https://community.auth0.com/t/getting-the-jwt-id-token-from-auth0-spa-js/28281/3
+     */
+    const idToken = await getIdTokenClaims();
+    const token = idToken?.__raw || "";


this hacky-feeling code is in the old VZE as well. more research needed here.

johnclary · 2024-11-15T19:57:23Z

app/utils/graphql.ts

+    query ? [query, variables] : null,
+    fetchWithAuth,
+    { ...DEFAULT_SWR_OPTIONS, ...(options || {}) }
+  );


here's where we call the actual useSWR hook—and a few important things to note:

the first argument is the value that SWR uses as the cache key. so each combination of query + variables gets cached.

passing null to the first argument is like an off switch, and enables conditional data fetching. this comes in super handy when you don't want to start a request until you have, e.g, a user's email address from the auth hook.

I think this may have answered my lazy query question. 👀

johnclary · 2024-11-15T20:03:18Z

app/utils/queryBuilder.ts

the queryBuilder replaces the graphqlAbstract in the VZE. basically, it takes a giant config object and spits out a graphql query. it puts the work of modifying the query configuration in the hands of UI components.

i said it in the last meeting, but i like this strategy to build what we need now and extend later if needed and NOT try to cover all of the functionality coming out of the Hasura API. 🙌 It is a nice iteration that builds of our knowledge gained through the first go-around and Moped refactor. Very cool stuff.

johnclary · 2024-11-15T20:04:33Z

app/utils/queryBuilder.ts

+    | "_is_null"
+    | "_ilike"
+    | "_in"
+    | "_nin";


hasura supports more operators, but i've only added those that we're actually using. we can add more as needed 👍

johnclary · 2024-11-15T20:48:34Z

app/configs/zodExample.ts

+ * At runtime, we would use the zod schema to validate api calls,
+ * and during development we have typechecking in sync with
+ * what zod is validating.
+ */


i don't know if the below code makes much sense, but these are my thoughts on how we might approach a tighter integration with schema validation☝️

i will find some links to other open source projects where i've seen this pattern.

Here's an example from the Dub project, where they define Zod schema and then declare types using z.infer

another approach that i think is fairly common is to use a tool like graphql-code-generator to create your type declarations from your graphql API.

Here's a demo project that does this:

Here are the code-generated types

And here is a hook that uses the type

The idea is that you would re-run the code-gen scripts whenever you API changes.

You can see in that generated.ts file that it produces a lot of code, because it builds up the entire graphql API starting with the scalars. It's a pretty cool learning tool, and I like the example utility types like Maybe and Exact, but I don't love having all these typings in the code base when you're only using a subset of them. Not to mention that you really need to hope that code generator is maintained throughout the life of your own project 😵

☝️ this second example does not do runtime type checking at all. it just hopes that the type declarations match what the API returns.

Once we adopt typescript across our project, one of the most likely sources for errors is going to be situations where our API does not return what we're expecting it to (for example, because we forgot to add a field to a query or we changed a field's type from number to string).

The reason we would reach for Zod (or Yup, etc) would be to centralize the error handling in those cases. We can check types before data passes from our API to UI components, and throw helpful error messages when something goes wrong.

i created cityofaustin/atd-data-tech#20024 for this and am going to do a little spike setting up a proof concept branch for how i think we want to structure this.

here is a branch/PR with a working setup for zod-driven typings:

Proof of concept for zod-driven types and validation #1612

sorting this out correctly and figuring out exactly what kind of tooling we want feels like the biggest open question for this app at the moment. I will update that PR with some more specific questions i have.

Once folks have a chance to catch up on this it will probably be worth scheduling a meeting to discuss :/

next up, i plan to look at the zod PoC PR early next week. I know you mentioned in Slack that it may not be your favorite at the moment. I think a separate spike into the gql codegen could be cool like you suggested. Especially if we could find a way to be selective about the generated types and what ends up being stored in this codebase.

I guess we would also need to think about what updating types would look like too if we added another CRIS value in the crash table for example. 🤔

Crash details: configure lookup tables for light condition, roadway part, and roadway system

…based on title

Export records from tables

…-counts Add total record count to tables

Crash details: Ability to swap addresses

johnclary commented Nov 15, 2024

View reviewed changes

app/env_template Outdated

Copy link

Member Author

johnclary Nov 15, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

our env files are no longer committed to the repo 🎉

johnclary commented Nov 15, 2024

View reviewed changes

app/tsconfig.json Outdated

Copy link

Member Author

johnclary Nov 15, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i did not create this file 🤖

johnclary commented Nov 15, 2024

View reviewed changes

johnclary requested review from roseeichelmann, frankhereford, mateoclarke, chiaberry, mddilley and tillyw November 15, 2024 20:39

johnclary commented Nov 15, 2024

View reviewed changes

chiaberry and others added 30 commits January 6, 2025 15:47

Merge branch 'main' into 20430-crash-details-lookup

2b784d9

add lighting conditions lookup

5092a6b

add roadpart to metadata and type and query

b732c64

Add roadway system

d6a6314

Add the secondary address lookup tables config

9b5aa2f

remove what i commented out

1434926

fix typo

ab65525

formatting

aacee8d

create swap address button component

88577f3

add swap address button to primary address data card

91b26c0

add docstring

24ea67e

add icon to button

49a7017

rework pagination styles and cleanup the dom

f6952c3

Merge pull request #1652 from cityofaustin/20430-crash-details-lookup

fae9da1

Crash details: configure lookup tables for light condition, roadway part, and roadway system

typos

83567ff

close modal after clicking the download button

7ff577d

use a file timestamp

136e4e3

resolve conf and merge upstream

54c162e

use border-0 class instead of inline style

97ed824

fix typo

706e226

pass HeaderActionButton as props

db29ede

update component name

9b736ec

update name

2625576

use abstracted HeaderActionButton instead of conditionally rendering …

daeec62

…based on title

unused import

e5a78a2

Merge pull request #1646 from cityofaustin/john/20456-record-export

e4c4c8c

Export records from tables

Merge pull request #1645 from cityofaustin/john/20202-page-and-record…

f62c10f

…-counts Add total record count to tables

import HeaderActionButtonProps

d5727c0

export interface HeaderActionButtonProps

9aaf89a

Merge pull request #1653 from cityofaustin/rose/swap-address

a72a20d

Crash details: Ability to swap addresses

		@@ -0,0 +1,29 @@
		"use client";
		import { Auth0Provider } from "@auth0/auth0-react";

		@@ -0,0 +1,24 @@
		import { format, parseISO } from "date-fns";

		// const formatCostToDollars = (cost: number \| null) =>

		@@ -0,0 +1,194 @@
		import { useState, useCallback, useEffect, useMemo } from "react";
		import useSWR, { SWRConfiguration } from "swr";

Upgrade VZE to NextJS #1603

Are you sure you want to change the base?

Upgrade VZE to NextJS #1603

Conversation

johnclary commented Nov 15, 2024 • edited Loading

Associated issues

Testing

Ship list

johnclary Nov 15, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

johnclary Nov 15, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

johnclary Nov 15, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

johnclary Dec 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

johnclary Nov 15, 2024 • edited Loading

Choose a reason for hiding this comment

johnclary Nov 15, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

johnclary Nov 15, 2024 • edited Loading

Choose a reason for hiding this comment

johnclary Nov 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

johnclary Nov 15, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

johnclary Nov 15, 2024 • edited Loading

Choose a reason for hiding this comment

johnclary Nov 15, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

johnclary Dec 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mddilley Nov 27, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

johnclary Nov 15, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

johnclary Nov 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mddilley Nov 27, 2024 • edited Loading

Choose a reason for hiding this comment

johnclary commented Nov 15, 2024 •

edited

Loading

johnclary Nov 15, 2024 •

edited

Loading

johnclary Nov 15, 2024 •

edited

Loading

johnclary Nov 15, 2024 •

edited

Loading

johnclary Dec 2, 2024 •

edited

Loading

johnclary Nov 15, 2024 •

edited

Loading

johnclary Nov 15, 2024 •

edited

Loading

johnclary Nov 15, 2024 •

edited

Loading

johnclary Nov 19, 2024 •

edited

Loading

johnclary Nov 15, 2024 •

edited

Loading

johnclary Nov 15, 2024 •

edited

Loading

johnclary Nov 15, 2024 •

edited

Loading

johnclary Dec 2, 2024 •

edited

Loading

mddilley Nov 27, 2024 •

edited

Loading

johnclary Nov 15, 2024 •

edited

Loading

johnclary Nov 19, 2024 •

edited

Loading

mddilley Nov 27, 2024 •

edited

Loading