#710: Restore Entities, Purge Entities, ability to see deleted Entities #1349
Conversation
sadiqkhoja
force-pushed
the
features/710-entity-restore-purge
branch
from
9125d0e to
0713461
Compare
| const _getPurgedUuids = (all, uuids) => all(sql` | ||
| SELECT jsonb_array_elements_text(details -> 'entityUuids') AS uuid FROM audits a WHERE action = 'entities.purge' | ||
| INTERSECT | ||
| SELECT jsonb_array_elements_text(${JSON.stringify(uuids)})`) | ||
| .then(all.map(r => r.uuid)); | ||
|
|
||
| const _isPurgedUuid = (maybeOneFirst, uuid) => maybeOneFirst(sql` | ||
| SELECT TRUE FROM audits WHERE action = 'entities.purge' AND details -> 'entityUuids' @> ${JSON.stringify([uuid])} | ||
| `); |
There was a problem hiding this comment.
I can pass a single uuid in _getPurgedUuuids but my gut feeling is that _isPurgedUuid would be faster. 🤷♂️
There was a problem hiding this comment.
verified that _isPurgedUuid is using the index and is much quicker.
| log( | ||
| 'entity.delete', | ||
| entity.with({ acteeId: dataset.acteeId }), | ||
| { entity: { uuid: entity.uuid } } | ||
| ); |
There was a problem hiding this comment.
Made it consistent with other entity logs where uuid is under the entity object
| SELECT audits.* FROM audits | ||
| JOIN entity_def_sources ON (audits.details->>'sourceId')::INTEGER = entity_def_sources.id | ||
| JOIN entity_defs ON entity_def_sources.id = entity_defs."sourceId" | ||
| WHERE entity_defs."entityId" = ${entityId} AND audits.action = 'entity.bulk.create' |
There was a problem hiding this comment.
This fetches only entity.bulk.create event
| SELECT audits.* FROM audits | ||
| JOIN entities ON (audits.details->'entity'->>'uuid') = entities.uuid | ||
| WHERE entities.id = ${entityId} |
There was a problem hiding this comment.
The fetches all logs of the entity (create, update, delete, restore)
| @@ -9314,6 +9276,100 @@ paths: | |||
| schema: | |||
| $ref: '#/components/schemas/Error403' | |||
| x-codegen-request-body-name: body | |||
| delete: | |||
There was a problem hiding this comment.
moved delete after patch so that delete and undelete are together
| @@ -150,11 +150,19 @@ const getBySubmissionId = (submissionId, options) => ({ all }) => _getBySubmissi | |||
| // There is a separate query below to assemble full submission details for non-deleted | |||
| // submissions, but it was far too slow to have be part of this query. | |||
| const _getByEntityId = (fields, options, entityId) => sql` | |||
There was a problem hiding this comment.
Objective of the change in this file is to get entity.delete and entity.restore logs as well
sadiqkhoja
changed the title
| const PURGE_DAY_RANGE = config.has('default.taskSchedule.purge') | ||
| ? config.get('default.taskSchedule.purge') | ||
| : 30; // Default is 30 days |
There was a problem hiding this comment.
Maybe we should move this to central place, this is present in three file at the moment.
There was a problem hiding this comment.
await Entities.createMany(dataset, partials, sourceId, userAgent);
Should catch UUID conflict error if it is not provided in the request body (means backend has generated it and there is a collision) and generate a new one.
Part of getodk/central#710
What has been done to verify that this works as intended?
Written bunch of tests, ran it with update frontend as well.
Why is this the best possible solution? Were any other approaches considered?
I have saved UUIDs of the purged entities in the audits table with
entities.purgerow and created GIN index on the property - I have verified its usage by insert 100K+ records in audits table.Other option is to create a separate table to store all the UUID which we can do if we sense performance is not acceptable with the JSONB in audits table.
I was also thinking to just check for UUID in
entity.createaudits; if given UUID was ever created - but forentity.bulk.createwe don't log UUIDs.How does this change affect users? Describe intentional changes to behavior and behavior that could have accidentally been affected by code changes. In other words, what are the regression risks?
Now entity creation checks for the duplicate UUID in the purged history, we should verify that there is no regression.
Does this change require updates to the API documentation? If so, please update docs/api.yaml as part of this PR.
Yes
Before submitting this PR, please make sure you have:
make testand confirmed all checks still pass OR confirm CircleCI build passes