This repository contains the NVA data report API.
The steps below can be outlined briefly as:
- Pre-run
- Stop incoming live-update events
- Delete data from previous runs
- Delete all data in database
- Bulk upload
- Generate batches of document keys for upload
- Transform the data to a format compatible with the bulk-upload action
- Initiate bulk upload
- Verify data integrity
- Post-run
- Start incoming live-update events
- Remove all objects from S3 bucket
loader-input-files-{accountName}
- Turn off S3 event notifications for bucket
persisted-resources-{accountName}
In aws console, go to
S3 -> persisted-resources-{accountName} -> Properties -> Amazon EventBridge -> Edit -> Off - Press
ResetDatabaseButton
(TriggerDatabaseResetHandler
). This might take around a minute to complete. - Verify that database is empty. You can use SageMaker notebook to query the database*. Example
sparql queries:
or
SELECT (COUNT(DISTINCT ?g) as ?gCount) WHERE {GRAPH ?g {?s ?p ?o}}
SELECT ?g ?s ?p ?o WHERE {GRAPH ?g {?s ?p ?o}} LIMIT 100
- Generate key batches for both locations:
resources
andnvi-candidates
. Manually triggerGenerateKeyBatchesHandler
with the following input:{ "detail": { "location": "resources|nvi-candidates" } }
- Verify that
GenerateKeyBatchesHandler
is done processing (i.e. check logs) and that key batches have been generated S3 bucketdata-report-key-batches-{accountName}
- Trigger
BulkTransformerHandler
- Verify that
BulkTransformerHandler
is done processing (i.e. check logs) and that nquads have been generated S3 bucketloader-input-files-{accountName}
- Trigger
BulkDataLoader
- To check progress for bulk upload to Neptune. Trigger
BulkDataLoader
with the following input:{ "loadId": "{copy loadId UUID from test log}" }
- Verify that expected count is in database. Query for counting distinct named graphs:
SELECT (COUNT(DISTINCT ?g) as ?gCount) WHERE {GRAPH ?g {?s ?p ?o}}
- Turn on S3 event notifications for bucket
persisted-resources-{accountName}
. In aws console, go to
S3 -> persisted-resources-{accountName} -> Properties -> Amazon EventBridge -> Edit -> On
*Note: You can use SageMaker notebook to query the database. Notebook can be opened from the AWS console through SageMaker -> Notebooks -> Notebook instances -> Open JupyterLab