Improve ontology processing

[syphax-bouazzouni]

Context

One of the most important, use cases in a data repository is the submission and uploading of resources, in 2023 we updated the UI/UX of the submission process #295, making the process more straightforward to add a new ontology or create a new submission/version.

The issue is now in the steps after, which are the uploading of the file, processing, and error handling, users lack information about what is happening after clicking on that save button. What is the status and progress of the file upload, the processing, and the parsing errors?

Issue

Our main problems currently are the following:

• We don't have any UI visual feedback, telling us the processing state of an ontology (email notifications are the only actual way that has the users to tell if their resource had parsed well or not)
• We can't monitor the ontologies processing issues (the only way that we have currently is to go to see the logs)
• Don't work (or hardly) for large files
• The process steps can be more isolated into small steps (e.g process_rdf=generate_rdf + append_to_triple_store + extract metadata + labels + mappings + ...)
• Our queue system is too simple (a Redis first in the first out queue), and can be (maybe) enhanced by using background processing tools like Sideqik or multithreading

Requirements

• update the big files upload to be in chunks and not have timeouts  ontoportal-lirmm/bioportal_web_ui#893
• Feature: Stream ontology processing  ontoportal-lirmm/ontologies_api#75
• Add logs section in the admin page to read the logs of the server  #639
• show parsing errors to users #320
• Merge ncbo cron in the ontologies_linked_data project and use https://resque.github.io/