feat(mrf-admin-view): hawkeye v1.0 #7965

kevin9foong · 2024-12-02T09:35:28Z

Problem

Currently, admins want a way to track the status of their multi-respondent submissions/workflows to see:

is the workflow completed / pending?
what is the current step of the workflow?

Closes FRM-1919

Solution

Add the above-mentioned information to the responses dashboard, csv download and individual responses page so admins can track the status.

Breaking Changes

No - this PR is backwards compatible

Before & After Screenshots

BEFORE:

AFTER:

New columns added and new name for timestamp column.

Tests

TC1: Hawk eye view dashboard shows the new status, current step and submission time of first step

Create 2 step MRF form without approval step
Submit step 1
Check responses dashboard, individual submission and download CSV. It should show pending status and correct step which is 1 of 2.
Submit step 2.
Check responses dashboard, individual submission and download CSV. It should show completed status and correct step which is 2 of 2.

TC2: Reject and approve show correctly.

Create a 3 step form with step 2 being approval step. Also, add a compulsory attachment field at step 1. When submitting the form in subsequent steps of the test case, always upload an attachment.
Submit step 1 and 2 of form, rejecting at step 2. (ie click No)
Check responses dashboard, individual submission and download CSV. It should show completed status and correct step which is 2 of 3.
Try download CSV with attachment. Ensure the csv file is correctly downloaded and the data in the csv is as expected and attachment files are downloaded.
Submit step 1, 2 and 3 of form, ensuring that step 2 is approve (ie click Yes)
Check responses dashboard, individual submission and download CSV. It should show completed status and correct step which is 3 of 3.
Try download CSV with attachment. Ensure the csv file is correctly downloaded and the data in the csv is as expected and attachment files are downloaded.

TC3: Storage mode and storage mode with with payments unaffected.

Create a storage mode form.
Submit and check the dashboard. Ensure status, current step and submission time of first step do not exist in the dashboard and the other expected columns exist (ie, same as before the changes introduced in this PR / refer to prod env)
Download the CSV, ensure that status, current step and submission time of first step do not exist in the csv.
Add payments to the storage mode form.
Submit and check the dashboard. Ensure status, current step and submission time of first step do not exist and the dashboard is same as previous (ie, same as before the changes introduced in this PR / refer to prod env).

datadog-opengovsg · 2024-12-09T15:57:02Z

Datadog Report

Branch report: feat/hawkeye-v1.0
Commit report: 66b411c
Test service: formsg

✅ 0 Failed, 2150 Passed, 1 Skipped, 5m 6.71s Total duration (1m 39.78s time saved)

kevin9foong · 2024-12-10T06:24:37Z

To reduce duplicate for the Step X of Y string into a common util function
To explore reusing types mrfMetadata

kevin9foong · 2024-12-13T03:37:51Z

shared/types/submission.ts

+
+const SubmittedNonApprovalStep = z.object({
+  isApproval: z.literal(false),
+  submittedAt: z.string(),


does zod have a timestamp type for validation? if so, maybe we can use it

yes - fixed referring to zod's datetime documentation with submittedAt: z.string().datetime({ precision: 3 })

kevin9foong · 2024-12-13T12:59:19Z

frontend/src/features/admin-form/responses/common/utils/mrfSubmissionView.ts

@@ -0,0 +1,26 @@
+import { WorkflowStatus } from '~shared/types'
+
+/** Gets the business friendly string for current step of workflow. */


these are helper functions to generate the view for mrf submissions responses

kevin9foong · 2024-12-13T13:00:13Z

frontend/vite.config.ts

+    define: {
+      // On local dev, global is undefined, causing decryption workers to fail.
+      // This is a workaround based on https://github.com/vitejs/vite/discussions/5912.
+      global: 'globalThis',


used the bypass an error faced as described by the comment.

I experimented with removing the whole global access chunk in decryption.worker.ts and it seemed to have resolved the csv generation issue on local.

Might want to look into that solution since I would prefer deleting code than to have more config.

removed the global reference + vite config, tested and it works well!

kevin9foong · 2024-12-13T13:03:06Z

src/app/modules/submission/submission.utils.ts

+/**
+ * Builds the metadata for a multirespondent form submission.
+ */
+export const buildMrfMetadata = ({


mrfMetadata refers to data which is not stored but derived from stored data when buildMrfMetadata is executed. This metadata is used in the submission and submissionMetadata requests to display metadata info about the submission in the responses table, downloaded csv file and individual submission response detail page.

tested to generate the correct metadata given the submission data

kevin9foong · 2024-12-13T13:41:22Z

...s/ResponsesPage/storage/utils/EncryptedResponseCsvGenerator/EncryptedResponseCsvGenerator.ts

@@ -112,7 +112,7 @@ export class EncryptedResponseCsvGenerator extends CsvGenerator {
    if (this.hasBeenProcessed) return

    // Create a header row in CSV using the fieldIdToQuestion map.
-    const headers = this.isMrf ? MRF_CSV_HEADERS : NON_MRF_CSV_HEADERS
+    const headers = this.isMrf ? [...MRF_CSV_HEADERS] : [...NON_MRF_CSV_HEADERS]


Note: The de-structuring is necessary as we are mutating the headers array below. De-structuring creates a new array which prevents mutations from affecting the referenced const array (ie, MRF_CSV_HEADERS or NON_MRF_CSV_HEADERS).

Otherwise, headers points to the MRF_CSV_HEADERS or NON_MRF_CSV_HEADERS const array and mutates the same memory address which is shared across multiple runs of process(). This causes issues in cases where the memory is shared/reused,specifically test cases, which causes the headers to contain the headers from the previous runs of process() and for the test cases to fail.

Good explanation to highlight that we need to create new array as we are mutating it at a later stage. Would recommend to put a short note in the code.

left a NOTE and TODO

…sible

kevin9foong · 2024-12-13T14:36:50Z

...min-form/responses/ResponsesPage/storage/UnlockedResponses/ResponsesTable/ResponsesTable.tsx

@@ -193,7 +194,8 @@ const MRF_RESPONSE_TABLE_COLUMNS: Column<ResponseColumnData>[] = [
    Header: MRF_FIRST_STEP_TIMESTAMP_LABEL,
    accessor: 'submissionTime',
    width: 250,
-    disableResizing: true
+    minWidth: 250,


Note: re-using the same fix as existing storage mode responses table columns to include minWidth which allows the bar to expand for the rightmost column when the screen is small and horizontal scroll is enabled.

linear · 2024-12-13T17:23:22Z

FRM-1919 MRF | Hawkeye v1.0

KenLSM · 2024-12-20T05:23:25Z

...min-form/responses/ResponsesPage/storage/UnlockedResponses/ResponsesTable/ResponsesTable.tsx

+  </Badge>
+)
+
+const NON_MRF_RESPONSE_TABLE_COLUMNS: Column<ResponseColumnData>[] = [


Would prefer to use "base" or "default" rather than "non-MRF". As non-MRF is not an exception. i.e., Payment is special, MRF is special, but non-MRF is not special.

Suggested change

const NON_MRF_RESPONSE_TABLE_COLUMNS: Column<ResponseColumnData>[] = [

const BASE_RESPONSE_TABLE_COLUMNS: Column<ResponseColumnData>[] = [

KenLSM · 2024-12-20T05:26:40Z

...min-form/responses/ResponsesPage/storage/UnlockedResponses/ResponsesTable/ResponsesTable.tsx

@@ -138,6 +238,16 @@ export const ResponsesTable = () => {
    }
  }, [filteredMetadata, metadata, submissionId])

+  const columns = useMemo(() => {


Good to see that this is extracted out to a function rather than choosing to add one more layer of ternary. 👍

KenLSM · 2024-12-20T05:47:02Z

frontend/src/features/admin-form/responses/common/utils/mrfSubmissionView.ts

+      return MRF_STATUS.COMPLETED
+    case WorkflowStatus.PENDING:
+      return MRF_STATUS.PENDING
+    default:


/nit since it is likely that all the WorkflowStatus is fully defined, we would want the typesystem to let us know that we need to update the cases here rather than it falling through to default.

made it more explicit for undefined to become '' and explicitly specified return literals of the function.

KenLSM · 2024-12-20T06:03:42Z

...s/ResponsesPage/storage/utils/EncryptedResponseCsvGenerator/EncryptedResponseCsvGenerator.ts

General notes on this generator. It has been awhile since I've looked at this file. I feel that using in-place mutation feels iffy. For instance, it can be difficult to follow the number of row.push(value) especially when it gets a little more complex as it is implicit rather than explicit.

Note that this comment is not regarding the changes that were introduced on this file, but I'm simply taking this opportunity to open up a discussion on potential refactors that we might want to consider in the future. The style of coding is likely an artifact of the past almost half a decade ago where FP is not as familiar.

Yes, ideally we would avoid mutation of the arrays as it can be hard to trace and debug. Will not introduce such a refactor in this PR to keep the scope defined to hawkeye.

Left a todo to document this desired change in the future

KenLSM · 2024-12-20T06:22:21Z

src/app/models/submission.server.model.ts

@@ -759,6 +791,7 @@ const buildSubmissionMetadata = (
  result: MetadataAggregateResult,
  currentNumber: number,
  paymentMeta?: PaymentAggregates,
+  mrfMeta?: MultiRespondentAggregates,


Would recommend to update the function definition to accept optional parameters as an object since I'm spotting a few uses of foo(paramX, paramY, null, paramZ).

const buildSubmissionMetadata = ( result: MetadataAggregateResult, currentNumber: number, options?: { paymentMeta?: PaymentAggregates, mrfMeta?: MultiRespondentAggregates, } )

refactored to utilise object argument

KenLSM · 2024-12-20T06:24:55Z

src/app/modules/submission/multirespondent-submission/multirespondent-submission.service.ts

@@ -469,7 +478,6 @@ export const updateMultiRespondentFormSubmission = ({
  encryptedPayload,
  logMeta,
 }: {
-  formId: string


Good spot 😆 👍

kevin9foong requested a review from KenLSM December 2, 2024 09:35

kevin9foong self-assigned this Dec 2, 2024

kevin9foong marked this pull request as draft December 2, 2024 09:35

kevin9foong requested a review from scottheng96 December 2, 2024 13:52

kevin9foong force-pushed the feat/hawkeye-v1.0 branch from 342a9e8 to 73eaece Compare December 11, 2024 09:36

kevin9foong commented Dec 13, 2024

View reviewed changes

kevin9foong temporarily deployed to staging-alt2 December 13, 2024 03:43 — with GitHub Actions Inactive

kevin9foong force-pushed the feat/hawkeye-v1.0 branch from db66d56 to b492151 Compare December 13, 2024 12:52

kevin9foong commented Dec 13, 2024

View reviewed changes

kevin9foong added 16 commits December 13, 2024 22:32

feat: add columns for mrf responses page

1c304c4

feat: fix default paddings for table

9f89d28

chore: update comment

42ae99a

feat: pipe step number to FE response table

c5c5cdc

feat: log step meta and pipe to fe response table

b817a59

fix: checking is form rejected error

f213752

feat: add status badges

6983771

feat: add csv download col

3614690

fix: add define global

936c973

feat: support cols in csv

009f840

fix: workflowStep being 0 is defined as falsy

c276fa4

feat: add approved rejected status in individualresponse

d5f044e

feat: map workflow status to business friendly status

6a28c16

fix: change imports to relative

8a8ddac

fix: failing unit tests

a8bdd88

feat: add datetime validation for submittedAt

609ccec

kevin9foong added 2 commits December 13, 2024 22:33

feat: resize table cols for mrf to follow storage mode as much as pos…

86c46c2

…sible

feat: update copy for first step

177f2fd

kevin9foong force-pushed the feat/hawkeye-v1.0 branch from fae67dc to c60e95a Compare December 13, 2024 14:34

fix: bug where last col does not expand in scrollable

75ac070

kevin9foong force-pushed the feat/hawkeye-v1.0 branch from c60e95a to 75ac070 Compare December 13, 2024 14:35

kevin9foong commented Dec 13, 2024

View reviewed changes

kevin9foong added 2 commits December 13, 2024 23:08

feat: add tc for testing the metadata builder

90eb8e5

feat: add tc for appending mrf meta to mrf submission stream

2384693

kevin9foong force-pushed the feat/hawkeye-v1.0 branch from 448f2d0 to 2384693 Compare December 13, 2024 15:08

kevin9foong added 7 commits December 13, 2024 23:32

feat: add tc for submission.server.model to fetch mrf meta

cd734b9

feat: add tc for csv generator for mrf

3d80ac3

feat: fix tc missing required fields

2139b9a

fix: bug where approved is marked as completed instead

c6eb27e

fix: submission.server.model tc

90d375b

fix: import ordering for lint

15f6d2f

feat: add chromatic story for mrf dashboard

2b9ca5d

kevin9foong marked this pull request as ready for review December 13, 2024 17:23

chore: remove redundant comment

0ef1e49

KenLSM reviewed Dec 20, 2024

View reviewed changes

kevin9foong added 5 commits December 23, 2024 10:51

chore: add docs on rationale of destructuring array

d3004bf

chore: rename from non-mrf to base

9ecac23

chore: refactor code to remove usage of undefined in argument

1ac3620

feat: make typing more explicit for case in getStatusFromWorkflowStatus

66b411c

fix: remove global access in decrypt and config in vite

2b8d32d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(mrf-admin-view): hawkeye v1.0 #7965

feat(mrf-admin-view): hawkeye v1.0 #7965

kevin9foong commented Dec 2, 2024 •

edited

Loading

datadog-opengovsg bot commented Dec 9, 2024 •

edited

Loading

kevin9foong commented Dec 10, 2024 •

edited

Loading

kevin9foong Dec 13, 2024

kevin9foong Dec 13, 2024 •

edited

Loading

kevin9foong Dec 13, 2024

kevin9foong Dec 13, 2024

KenLSM Dec 20, 2024 •

edited

Loading

kevin9foong Dec 23, 2024

kevin9foong Dec 13, 2024 •

edited

Loading

kevin9foong Dec 13, 2024

kevin9foong Dec 13, 2024 •

edited

Loading

KenLSM Dec 20, 2024

kevin9foong Dec 23, 2024

kevin9foong Dec 13, 2024 •

edited

Loading

linear bot commented Dec 13, 2024

KenLSM Dec 20, 2024 •

edited

Loading

kevin9foong Dec 23, 2024

KenLSM Dec 20, 2024

KenLSM Dec 20, 2024

kevin9foong Dec 23, 2024 •

edited

Loading

KenLSM Dec 20, 2024

kevin9foong Dec 23, 2024

kevin9foong Dec 23, 2024

KenLSM Dec 20, 2024

kevin9foong Dec 23, 2024

KenLSM Dec 20, 2024

		@@ -0,0 +1,26 @@
		import { WorkflowStatus } from '~shared/types'

		/** Gets the business friendly string for current step of workflow. */

	const NON_MRF_RESPONSE_TABLE_COLUMNS: Column<ResponseColumnData>[] = [
	const BASE_RESPONSE_TABLE_COLUMNS: Column<ResponseColumnData>[] = [

feat(mrf-admin-view): hawkeye v1.0 #7965

Are you sure you want to change the base?

feat(mrf-admin-view): hawkeye v1.0 #7965

Conversation

kevin9foong commented Dec 2, 2024 • edited Loading

Problem

Solution

Before & After Screenshots

Tests

datadog-opengovsg bot commented Dec 9, 2024 • edited Loading

Datadog Report

kevin9foong commented Dec 10, 2024 • edited Loading

Choose a reason for hiding this comment

kevin9foong Dec 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KenLSM Dec 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kevin9foong Dec 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kevin9foong Dec 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kevin9foong Dec 13, 2024 • edited Loading

Choose a reason for hiding this comment

linear bot commented Dec 13, 2024

KenLSM Dec 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kevin9foong Dec 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kevin9foong commented Dec 2, 2024 •

edited

Loading

datadog-opengovsg bot commented Dec 9, 2024 •

edited

Loading

kevin9foong commented Dec 10, 2024 •

edited

Loading

kevin9foong Dec 13, 2024 •

edited

Loading

KenLSM Dec 20, 2024 •

edited

Loading

kevin9foong Dec 13, 2024 •

edited

Loading

kevin9foong Dec 13, 2024 •

edited

Loading

kevin9foong Dec 13, 2024 •

edited

Loading

KenLSM Dec 20, 2024 •

edited

Loading

kevin9foong Dec 23, 2024 •

edited

Loading