Skip to content
This repository was archived by the owner on Feb 1, 2023. It is now read-only.

Allow users to run bulk ingest to load large volumes of data into a graph #51

Open
d47853 opened this issue Sep 2, 2020 · 1 comment
Labels
enhancement New feature or request

Comments

@d47853
Copy link
Member

d47853 commented Sep 2, 2020

The ingest should be carried out by lambdas which can run spark-submit jobs to the Kubernetes cluster. These lambdas should initially be developed outside of Kai and referenced via their ARN. The admins of Kai needs some way of adding ingest lambdas to the deployment. The easiest way I can think to do this is with configuration. You could do it via REST but that would require a new user pool etc.

The ingest objects should be stored in DynamoDB and should have the rough structure:

{
    "name": "My Ingest Job",
    "arn": "lambda arn",
    "arguments": {
        "inputFile": "text",
        "generatorJson": "json"
    }
} 

A Kai user should be able to retrieve these objects (minus the arn) and a UI should be able to use the arguments and their types to render a form that the user can fill in to trigger a bulk ingest.

@d47853 d47853 added the enhancement New feature or request label Sep 2, 2020
@d47853 d47853 changed the title Allow users to run bulk ingest to load large volumes of data into a graph. Allow users to run bulk ingest to load large volumes of data into a graph Sep 2, 2020
@d47853 d47853 assigned d47853 and unassigned d47853 Sep 2, 2020
@d47853
Copy link
Member Author

d47853 commented Sep 3, 2020

Happy for someone else to work on this. If no one want's it, I'll pick it up again later

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant