Firebreak - Add reports to download live form data as CSV files #1681
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
stephencdaly
force-pushed
the
add-csv-download-reports
branch
3 times, most recently
from
5729c92 to
54dd2ee
Compare
stephencdaly
force-pushed
the
add-csv-download-reports
branch
9 times, most recently
from
689c5de to
d15dc1d
Compare
stephencdaly
force-pushed
the
add-csv-download-reports
branch
2 times, most recently
from
b5bb6c9 to
1b3e8ae
Compare
This report downloads a CSV file containing data about all live forms. This calls the forms-api "/api/v2/form-documents" endpoint to get the form document objects for all live forms. We get the form documents rather than the form model as the form document represents the state of the live form that is served to form fillers, whereas the form model might have changes that have been made to a draft version of the form. The forms-api endpoint supports pagination. Use this to avoid sending all form documents in a single large API response - default to 100 results per request page. To make requests to forms-api, use Net:HTTP rather than ActiveResource. I attempted to use ActiveResource at first but this required an additional gem and some additional hacking at to be able to consume the headers for pagination. When we consume the data from forms-admin, add in details about the organisation and group for the form and manipulate some of the data to produce a useful CSV report. For the tests, use a JSON fixutre file that was generated by making a real API request to forms-api with the data from the database seeds. Once we have transitioned to using the V2 API in forms-admin, it might make more sense to use factories to generate the response.
This report downloads a CSV file containing data about all questions in live forms. This works the same way as the report for getting data about forms themselves, but includes a row in the CSV per question in a live form.
Move the `UserReportService` in to a `reports` directory and under a `Reports` module to group the different reports services together.
stephencdaly
force-pushed
the
add-csv-download-reports
branch
from
1b3e8ae to
a6d2190
Compare
|
|
Looks good from a content perspective. This will be so useful - thank you! |
DavidBiddle
approved these changes
Jan 3, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What problem does this pull request solve?
Trello card: https://trello.com/c/mPtJhtWq/
Add 2 reports to get all data about live forms as a CSV, and to get all questions in all live forms as a CSV.
The intention of this is that it should make it easier to analyse data about forms and questions that exist on the platform without having to spend time to make dedicated reports. It should be quick and easy to add additional columns into these CSV reports as we add new features.
These reports call the forms-api "/api/v2/form-documents" endpoint added in alphagov/forms-api#663 to get the form document objects for all live forms. We get the form documents rather than the form model as the form document represents the state of the live form that is served to form fillers, whereas the form model might have changes that have been made to a draft version of the form.
The forms-api endpoint supports pagination. Use this to avoid sending all form documents in a single large API response - default to 100 results per request page.
To make requests to forms-api, use Net:HTTP rather than ActiveResource. I attempted to use ActiveResource at first but this required an additional gem and some additional hacking at to be able to consume the headers for pagination.
Example CSVs:
live_forms_report-2025-01-03 12_59_53 UTC.csv
live_questions_report-2025-01-03 13_03_47 UTC.csv
Relies on forms-api pull request: alphagov/forms-api#663
Things to consider when reviewing