Implement super-rag webhook for setting a data status to the workflow.

[homanp]

⚠️ Please check that this feature request hasn't been suggested before.

• I searched previous Ideas in Discussions didn't find any similar feature requests.
• I searched previous Issues didn't find any similar feature requests.

🔖 Feature description

We need to notify the db of when data for a workflow has finished processing through the SuperRag pipeline. For this we need to utilize the Super-Rag webhook functionaltiy.

TODO:

• Create necessary database tables for tracking status of ingest.
• Update ingest status with webhook.
• Block incoming requests until data has finished loading.

The point is to give the workflow a status so we can visualize this in the api and sdk to allow a while loop similar to openai assisntants.

Acknowledgements

• My issue title is concise and descriptive.
• I have searched the existing issues to make sure this feature has not been requested yet.
• I have provided enough information for the maintainers to understand and evaluate this request.