POC for unstructured data

[PriyaBasker23]

Things to consider to accomodate unstructured data

We've designed a structured data product architecture, but we're now looking to incorporate unstructured data (e.g., PDFs, images, XLS files) into our system. While unstructured data currently forms a smaller part of our data landscape, we aim to develop a mechanism for ingesting such data. It's important to note that this proof of concept (POC) focuses solely on the ingestion and gathering of information from unstructured data, with no processing involved.

Following our discussion, we can conclude the approach for the MVP to incorporate unstructured data.

Comment below the suggestions .

[PriyaBasker23]

Comments from @murdo-moj

• create an endpoint which allows you to upload unstructured data. this data skips the athena load process and is simply copied into s3 without being validated in any way.
• This might require naming changes in the API if the name "upload_data" survives the redesign

[PriyaBasker23]

User Journey

• Registration of Data product including metadata with no changes between structured and unstructured
• Metadata about the file ( is that necessary ?)
• Create an endpoint which allows you to upload unstructured data his data skips the athena load process and is simply copied into s3 ( Presigned url approach with limitation of size upto 5gb)
• Unstructured data will include ( image, xls, pdf , which dont need any processing )

[PriyaBasker23]

Comments from @MatMoore

if database/table params are changing to data-product / model then I think we could still accommodate unstructured data with that.
I'm thinking something like this:

• a data product always has metadata
• a data product has many models
• a model can be structured or unstructured
• if structured, it must have a schema