Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build a tool to import data from JSON files into zerobase using zerobase API #73

Open
brianok77 opened this issue Apr 29, 2020 · 3 comments
Assignees

Comments

@brianok77
Copy link

We need a tool that will read the JSON files extracted in #72 and make the proper zerobase smart-tracing-api calls to populate the staging database with the data from the experiment. Important: This tool will need to maintain its own series of maps from "source id" to "zerobase id" for sites and devices, so when loading the scans and test result data it can be associated with the correct sites/devices.

@brianok77
Copy link
Author

After speaking with @toadzky, we should instead load the orgs, sites, and qr codes directly from the simulation data by creating flat files and using the Neptune Bulk Load API (https://docs.aws.amazon.com/neptune/latest/userguide/bulk-load.html), and then use the api to create users and scans from the files. This is because using the API for orgs and sites will generate QR codes sent via email, etc. instead of providing the necessary data in the API payload. I'll update the story details when I have time to dig into exactly what this means for things like the graph structure and how we can create the GUIDs.

@brianok77
Copy link
Author

@toadzky is currently rearchitecting the backend to store data in DynamoDb and then push data into Neptune from DynamoDb using Streams. Therefore we will load data directly into DynamoDb instead of Neptune as mentioned in the last comment. @toadzsky will provide the DynamoDb schema ASAP so we can start programming this.

@brianok77 brianok77 self-assigned this May 12, 2020
@brianok77
Copy link
Author

The parsing of the JSON file, data transformation, and mock upload is done in a local branch. Starting discovery of the best way to upload this data with notation that it is sourced from simulation data (so we can easily clear the data if we want to run a new sim with "better" parameters).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant