Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use the normal hitobito API instead of the group_health endpoints #28

Draft
wants to merge 12 commits into
base: develop
Choose a base branch
from

Conversation

carlobeltrame
Copy link

@carlobeltrame carlobeltrame commented Aug 23, 2021

See #22 for context.
Currently, the healthcheck fetches absolutely all the data every night. This leads to a lot of large API calls with very few changed data. Also, hitobito needs to offer a specialized API, which is only used by this one application, next to it's existing, general-purpose API. This PR is a first step towards changing the architecture of healthcheck to fetch only the necessary data only when it is needed, and only through the normal generic hitobito JSON:API.

The idea is to use hitobito's normal API via an OAuth access token. An Abteilungsleiterin is offered a "sync" button in the healthcheck application, which triggers API calls to fetch only the necessary data for that specific Abteilung on demand. This alleviates the need for an opt-in flag in hitobito, and the users don't have to wait for a whole night when using healthcheck for the first time. Also, healthcheck would become more of a true microservice, and hitobito wouldn't need to know anything about healthcheck internals anymore.

This new approach requires some changes to hitobito, and some fundamental changes to the importing system of healthcheck.
I have begun implementing all of these changes here and in hitobito/hitobito#1378, hitobito/hitobito_pbs#204 and digio-ch/pbs-healthcheck-web#11. It also requires the MiData admins to grant the api scope to the healthcheck OAuth application, and add another callback URI with ?action=sync appended to it.

Any feedback is welcome.

  • Add a button for Abteilungsleitungen to sync manually
    • Styling of this button
    • Only show the button when the user has the permission to sync the group data
  • Also add buttons for syncing Abteilungen that are not in the HC database yet
    • Expose the group type and role type of all roles in the hitobito OAuth profile, so we can correctly select which groups to offer for syncing
  • Make the button clean everything we already know (and aggregated) about the Abteilung, to somewhat mitigate some of the issues with data deletion requests / data storing outside of MiData
    • Separate / mark people by their corresponding imported Abteilung, such that deleting the people that are in one Abteilung doesn't affect the data in another Abteilung at all
  • Make the button import the groups from hitobito
    • Expose deleted subgroups in the hitobito API as well, and import them
  • Make the button import the people and personRoles from hitobito
    • Add the capability to fetch deleted roles to the hitobito API and hitobito web interface
    • Give Abteilungsleitungen the permission to use this capability
    • Add a new filter "with_deleted" to the hitobito API that allows to fetch deleted and active people in a single API call
  • Make the button import the group's camps and courses from hitobito
    • Fix the start_date and end_date filters in the hitobito events API, so we can fetch events from past years as well
  • Make the button import the participations in the group's camps and courses from hitobito
    • Filter out participations of people that are not in that group? Import participations of unknown people as anonymous participations with null in the foreign key.
  • Make the button import the people's qualifications from hitobito
    • Add the qualifications to the regular hitobito API
  • After importing, run the aggregations, but only for the imported Abteilung
    • Don't run aggregations during nightly import
  • Remove any sensitive data syncing in the nightly import
    • Refactor the login flow such that it doesn't rely on any previously synced sensitive data
  • Fetch the group data in healthcheck-web via the healthcheck-core API, instead of via the response when logging in
  • Disable sync, invite and opt-out buttons depending on whether the group has been synced before
  • Disable inviting other people if the user doesn't have the permission to sync the group
  • Adapt README and other documentation in healthcheck
  • Adapt REST API documentation in hitobito
  • Conform to the Datenschutz-Nutzungsbedingungen of HealthCheck
    • "Der HealthCheck visualisiert nur Daten, die du auch in der MiData sehen könntest." - this is the actual goal of this PR, and thus is solved automatically by the architectural change. The HealthCheck can only access data in MiData which the AL has access to and to which the AL has explicitly granted access.
    • "Wird [der Opt-in] deaktiviert, wird die Übertragung gestoppt und alle Daten im HealthCheck werden vollständig gelöscht." - add a "delete all data of my Abteilung" button
    • "Wird eine Person in der MiData vollständig gelöscht, so werden alle persönlichen Daten dieser Person im HealthCheck ebenfalls gelöscht (Name, Adresse, Geburtsdatum, Geschlecht)" - delete or anonymize the contents of all tables containing sensitive data (not the aggregated ones) after syncing
  • Future task: Add regular API endpoints for all other non-sensitive data and use these when importing (qualification_kinds, event_kinds, role_types, group_types, participation_types, j_s_kinds, camp_states)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant