Proposal for Real-Time IoT Perception Systems Based on Edge-Cloud Collaboration with Large Foundation Models #80
Conversation
… Collaboration with Large Foundation Models Signed-off-by: hsj576 <[email protected]>
|
|
||
| **Cloud-edge collaborative inference**, as an emerging technology, combines the processing capabilities of the cloud and edge devices to handle complex tasks more efficiently. The core idea of cloud-edge collaborative inference is to deploy large-scale models in the cloud, leveraging the powerful computing resources to handle complex tasks and challenging samples. On the other hand, deploying small-scale models on edge devices can meet the requirements for processing simple samples. In this architecture, edge devices only need to transmit sample data and communicate with the cloud, without bearing a large amount of computation tasks, greatly saving energy and computational resources. | ||
|
|
||
| <img src="images/云边协同改进.png" alt="云边协同改进" style="zoom: 25%;" /> |
There was a problem hiding this comment.
Good to learn the joint inference with SAM and the ''edge model'' on the cloud. It fixed the problem that SAM does not output the inference class but only the segmentation result. But it is weird to see an "edge model" on the cloud. You might want to change the "edge model" to a "small model"?
There was a problem hiding this comment.
Good to learn the joint inference with SAM and the ''edge model'' on the cloud. It fixed the problem that SAM does not output the inference class but only the segmentation result. But it is weird to see an "edge model" on the cloud. You might want to change the "edge model" to a "small model"?
I have changed the "edge model" to "RFNet model".
|
|
||
| **Cloud-edge collaborative inference**, as an emerging technology, combines the processing capabilities of the cloud and edge devices to handle complex tasks more efficiently. The core idea of cloud-edge collaborative inference is to deploy large-scale models in the cloud, leveraging the powerful computing resources to handle complex tasks and challenging samples. On the other hand, deploying small-scale models on edge devices can meet the requirements for processing simple samples. In this architecture, edge devices only need to transmit sample data and communicate with the cloud, without bearing a large amount of computation tasks, greatly saving energy and computational resources. | ||
|
|
||
| <img src="images/云边协同改进.png" alt="云边协同改进" style="zoom: 25%;" /> |
There was a problem hiding this comment.
How is this solution related to ianvs lifelong learning libs? It seems to me that it runs inference first then the unseen task recognition, but it should be clearly indicated in the architecture.
There was a problem hiding this comment.
How is this solution related to ianvs lifelong learning libs? It seems to me that it runs inference first then the unseen task recognition, but it should be clearly indicated in the architecture.
I have added the Ianvs-plugin workflow for the proposed solution.
Signed-off-by: hsj576 <[email protected]>
Signed-off-by: hsj576 <[email protected]>
|
|
||
| The overall architecture of the Cloud-edge Collaborative IoT Real-time Perception System is shown in the figure below. This system architecture considers applications in typical lifelong learning scenarios. Edge devices continuously collect images of the environment they are located and efficiently process these images using the Cloud-edge Collaborative Real-time Perception System. At the same time, the edge model is continuously trained through lifelong learning, based on the exploration of hard sample patterns, to ensure that the model can continuously adapt to the dynamic changes in the edge environment. | ||
|
|
||
|  |
There was a problem hiding this comment.
In the routine meeting, we see that training modules shall be included in the proposal. The suggestion is to separate the training and inference procedures and consider their modules respectively.
There was a problem hiding this comment.
In the routine meeting, we see that training modules shall be included in the proposal. The suggestion is to separate the training and inference procedures and consider their modules respectively.
I have separated the procedures of training and inference in the architecture figure.
| 9. Edge nodes output the cloud inference results, while the cloud node performs lifelong learning training on the edge model based on the samples in the cloud node's replay buffer. | ||
| 10. The cloud node synchronizes the updated Edge Model to the edge nodes. | ||
|
|
||
|  |
There was a problem hiding this comment.
As mentioned above, the training modules are also absent in the workflow.
There was a problem hiding this comment.
As mentioned above, the training modules are also absent in the workflow.
I have added training modules in the workflow.
|
/lgtm |
1 similar comment
|
/lgtm |
|
/approve |
Proposal for Real-Time IoT Perception Systems Based on Edge-Cloud Collaboration with Large Foundation Models
What type of PR is this?
/kind proposal
What this PR does / why we need it:
This PR provide a proposal for Real-Time IoT Perception Systems Based on Edge-Cloud Collaboration with Large Foundation Models.
Which issue(s) this PR fixes:
Fixes issue #76