Enabling OpenVINO #93
Conversation
Yaml config files under "cfg/examples/replication" were updated. Config "openvino" added to specify the target device for an OpenVINO model to execute. So far, it is supported to execute GQCNN OpenVINO models on CPU, GPU, or MYRIAD (Movidius NCS2). This config is set to OFF by default, for GQCNN with Tensorflow backend. Signed-off-by: Sharron LIU <[email protected]>
Tutorials added introducing how to - install the OpenVINO™ toolkit - freeze a GQCNN model - convert a GQCNN model - evaluate the GQCNN OpenVINO model Signed-off-by: Sharron LIU <[email protected]>
sharronliu
force-pushed
the
master
branch
2 times, most recently
from
d99e318 to
4c150a1
Compare
GQCNNTF.initialize_network() is updated to freeze the GQCNN graph, right after the initialization of the network. This code block is inactive unless explicitly switching the `if` condition into `True` when freezing is needed. A frozen graph is expected by the Model Optimizer of OpenVINO™. Signed-off-by: Sharron LIU <[email protected]>
GQCNNOpenVINO, derived from GQCNNTF, implements GQCNN inference with OpenVINO™ backend. GQCnnQualityFunction.__init__() is also updated, to load the GQCNNOpenVINO model. Signed-off-by: Sharron LIU <[email protected]>
There was a problem hiding this comment.
Hi @sharronliu,
Glad to hear that you are using the GQ-CNN, and thanks so much for porting it over to the Intel OpenVINO framework!
I left a few comments on the PR, and it would be great if you could take a look at them. In my comments I reference https://github.com/BerkeleyAutomation/gqcnn/tree/ncs, which was used in the past to deploy the FC-GQ-CNN on the Intel Movidius Myriad X with the NCSDK.
I'm also curious as to whether or not this works for the FC-GQ-CNN or only for the GQ-CNN. One thing we encountered earlier was that certain operations in the pair-wise softmax used by the FC-GQ-CNN were not yet supported by the NCSDK. It was either the split or concat or both. I'm not sure if OpenVINO supports these. The workaround was to implement them in NumPy post-inference.
Thanks,
Vishal
| @@ -552,6 +552,19 @@ def initialize_network(self, | |||
| if add_sigmoid: | |||
| self.add_sigmoid_to_output() | |||
|
|
|||
| # Freeze graph. Make it 'True' to freeze this graph | |||
There was a problem hiding this comment.
Would it be possible to automate this as part of gqcnn/model/openvino/network_openvino.py? I just don't like the user having to edit the source. See https://github.com/BerkeleyAutomation/gqcnn/blob/ncs/gqcnn/model/tf/ncs_fc_network_tf.py#L251 for an example. There when the user invokes compile(), which executes the compilation shell commands, the first step is to generate the protobuf.
There was a problem hiding this comment.
Yes I think the freezing codes could be put into a separate function (e.g. freeze()), like what you did for ncs_fc_network_tf.py.
An open question is where to invoke the freeze(). I noticed compile() is invoked each time when the FCGQCNN model is loaded.
While freeze() is necessary only for OpenVINO model conversion, which is usually done once the pre-trained model are updated. Would you mind if freeze() is invoked each time loading the GQCNN model?
Another open question is, network_openvino.py may not be the right place to do freezing. Then OpenVINO model is an intermediate representation of the original model. It's converted, no more depends on Tensorflow, nor needs to load the original TF model. While freezing is expected from the original TF model.
| ========================= | ||
| The toolkit is available from open source project `Intel® OpenVINO™ Toolkit`_. | ||
|
|
||
| .. note:: GQCNN uses two layers, RandomUniform and Floor, which are not supported by the 2019_R3 release of OpenVINO™. `PR #338 <https://github.com/opencv/dldt/pull/338>`_ adds the support for these two layers. |
There was a problem hiding this comment.
I didn't have these issues here: https://github.com/BerkeleyAutomation/gqcnn/blob/ncs/gqcnn/model/tf/ncs_fc_network_tf.py#L242, so I'm curious as to what the difference is. One is that I use self._graph.as_graph_def() instead of self._sess.graph_def, however I don't think that's it. Another is that the method in which the protobufs are written is a bit different. Finally I guess you are using OpenVINO whereas I was using the NCSDK, but I would think that they would support similar operations, right?
I also think it's a bit weird that the frozen graph would require RandomUniform because that should be strictly for training. If I recall correctly, only operations that the output directly depends on (i.e. things for inference) should be kept by: https://github.com/BerkeleyAutomation/gqcnn/blob/ncs/gqcnn/model/tf/ncs_fc_network_tf.py#L245. I don't exactly remember when/where Floor is used.
This might be worth looking into until your PR for the RandomUniform and Floor operations is approved. Even if you could just copy my code and try it out, that would be super helpful. Maybe something changed and it will still break.
There was a problem hiding this comment.
A little bit surprised hearing from you that RandomUniform or Floor is not used...
The OpenVINO Model Optimizer tool detects the exist of these two layers. Besides, the tensorboard visualization tool also illustrates these two layers, for pre-trained GQCNN 2.0, 2.1, 3.0, 4.0-pj, 4.0-suc downloaded by download_models.sh. I'm attaching the screen snapshot from tensorboard client, which might help you to recall something... :)
The usage of RandomUniform or Floor should not be introduced by OpenVINO or NCSDK. Both tools are inference stacks. While these two layers are observed in the original GQCNN graphs.
I also noticed these two layers were not introduced in the pre-trained models in Aug. 2018, when I started some works on GQCNN acceleration. My initial guessing is they were added later.
I won't think it related to self._graph.as_graph_def() or self._sess.graph_def, these two are equal if check == while freezing.
As you mentioned RandomUniform should be strictly for training. Would you suggest a right place to freeze the model? I'm thinking whether network_tf.py is the proper place, since it loads the GQCNNTF model for training. If the inference model is loaded in somewhere else, I can invoke the freezing there.
|
RandomUniform and Floor shown in the tensorboard visualization. |
|
@sharronliu Okay, I think I know what's going on and you're right. So the RandomUniform and Floor ops are coming from the dropout layers. Our earlier experimentation with the NCS was using the FC-GQ-CNN, in which all of the layers are convolutions (the fully-connected ones are converted at inference time), which don't have a dropout applied afterwards. We never actually tried to port the original GQ-CNN and thus never experienced this issue. Like you mentioned, the correct solution to this seems to be to remove the dropout layers in the inference graph. One option is to never include them entirely. The other is to post-process the graph definition and bypass those layers. I'm leaning towards the latter, however I'm a bit busy at the moment and will probably only get to it later this weekend. I'll let you know how it goes! Thanks, |
|
Vishal, thanks for your time and quick reply. I just added feedbacks to your previous comment on having an automated Then some explanation on NCSDK, now also part of OpenVINO. Which implies, the installation of OpenVINO will support deployment on all devices (CPU, GPU, MYRIAD, FPGA). This #93 was tested with MYRIADX also (with batch size forced to 1). BTW, I didn't mention FPGA in the Then regarding your question on FCGQCNN. I originally planned to support FCGQCNN also. While I met some issue when freezing the graph. Before getting a frozen FCGQCNN, this PR is tested for GQCNN so far. |
|
@visatish , I'm doing some minor changes for this PR based on the latest OpenVINO release version 2020.2. In this version of OpenVINO, the |
This PR enable OpenVINO™ technology for GQCNN. The original GQCNN models are converted to OpenVINO™ models, and deployed across various Intel devices for inference.
Evaluated OpenVINO™ models are: GQCNN-4.0-SUCTION, GQCNN-4.0-PJ, GQ-Suction, GQ-Bin-Picking-Eps90, GQ-Image-Wise. The evaluation is done in the same way as mentioned in the replication scripts under
scripts/policies/.I tested with Intel NUC6i7KYK at CPU and GPU. A list of supported device types are available here.
To replicate my results, the GQCNN OpenVINO™ models are available here for test purpose. These models were converted from the original pre-trained GQCNN models. Untar these models from gqcnn root:
tar -zxvf gqcnn.models.OpenVINO.20191119.tgz. Or you may convert the GQCNN models by yourself, following the tutorials in this PR.