Invalid filter channel number - conv on AMD

[theRTLmaker]

Hello,

I'm trying to use the DNNMark benchmark to characterize my AMD GPU.
My setup is;

• CentOS 7.5
• ROCm 3.0 (with all the included libraries rocmblas and miopen)
• AMD Vega 10 Frontier Edition

I'm able to successfully run all the layer tests, besides the convolution and dropout, with the provided files in config_examples.

When running the convolution (both forward and backward), I'm getting an error related to "Invalid filter channel number"

The convolution terminal output is the following:
I0428 13:03:25.984831 23359 test_bwd_conv.cc:11] DNNMark suites: Start... I0428 13:03:26.016577 23359 dnnmark.cc:226] Search and parse general DNNMark configuration I0428 13:03:26.016685 23359 dnnmark.cc:286] Search and parse layer configuration I0428 13:03:26.016721 23359 dnnmark.cc:300] Add [Convolution] layer I0428 13:03:26.016850 23359 dnn_layer.h:84] Layer name: conv1 I0428 13:03:26.016899 23359 dnn_layer.h:89] Previous layer: null I0428 13:03:26.016937 23359 dnnmark.cc:355] DNNMark: Initialize... I0428 13:03:26.016944 23359 dnnmark.cc:356] Running mode: 1 I0428 13:03:26.016950 23359 dnnmark.cc:357] Number of Layers: 1 I0428 13:03:26.016963 23359 dnnmark.cc:359] Layer type: 1 I0428 13:03:26.016968 23359 dnnmark.cc:361] DNNMark: Setup parameters of Convolution layer I0428 13:03:26.016974 23359 dnn_layer.h:110] Bottom dimension: N: 126 C: 3 H: 256 W: 256 I0428 13:03:26.017006 23359 data_manager.h:44] Create Data chunk of size 24772608 I0428 13:03:26.017105 23359 data_manager.h:101] Create data with ID: 0 I0428 13:03:26.017154 23359 data_manager.h:44] Create Data chunk of size 24772608 I0428 13:03:26.017248 23359 data_manager.h:101] Create data with ID: 1 I0428 13:03:26.017269 23359 data_manager.h:44] Create Data chunk of size 264241152 I0428 13:03:26.017419 23359 data_manager.h:101] Create data with ID: 2 I0428 13:03:26.017431 23359 data_manager.h:44] Create Data chunk of size 264241152 I0428 13:03:26.017577 23359 data_manager.h:101] Create data with ID: 3 I0428 13:03:26.017590 23359 data_manager.h:44] Create Data chunk of size 2400 I0428 13:03:26.017650 23359 data_manager.h:101] Create data with ID: 4 I0428 13:03:26.017660 23359 data_manager.h:44] Create Data chunk of size 2400 I0428 13:03:26.017670 23359 data_manager.h:101] Create data with ID: 5 I0428 13:03:26.046165 23359 data_manager.h:44] Create Data chunk of size 19660800 I0428 13:03:26.046329 23359 data_manager.h:101] Create data with ID: 6 I0428 13:03:26.046348 23359 conv_layer.h:220] Setting Bwd Filter Algo to I0428 13:03:26.048985 23359 data_manager.h:44] Create Data chunk of size 19660800 I0428 13:03:26.049124 23359 data_manager.h:101] Create data with ID: 7 I0428 13:03:26.050288 23359 data_manager.h:44] Create Data chunk of size 19660800 I0428 13:03:26.050415 23359 data_manager.h:101] Create data with ID: 8 I0428 13:03:26.050449 23359 dnnmark.cc:526] DNNMark: Running convolution backward: STARTED MIOpen Error: /root/driver/MLOpen/src/ocl/convolutionocl.cpp:122: Invalid filter channel number I0428 13:03:36.680456 23359 data_manager.h:53] Free Data chunk of size 19660800 I0428 13:03:36.680564 23359 data_manager.h:53] Free Data chunk of size 19660800 I0428 13:03:36.680598 23359 data_manager.h:53] Free Data chunk of size 19660800 I0428 13:03:36.680630 23359 data_manager.h:53] Free Data chunk of size 2400 I0428 13:03:36.680649 23359 data_manager.h:53] Free Data chunk of size 2400 I0428 13:03:36.680656 23359 data_manager.h:53] Free Data chunk of size 264241152 I0428 13:03:36.680737 23359 data_manager.h:53] Free Data chunk of size 264241152 I0428 13:03:36.680815 23359 data_manager.h:53] Free Data chunk of size 24772608 I0428 13:03:36.680847 23359 data_manager.h:53] Free Data chunk of size 24772608 MIOpen Error: 3 at /homelocal/fmendeslocal/DNNMark/core/include/dnn_utility.h983

When running the dropout (both forward and backward), the program is also trying to allocate an abnormal quantity of memory, that results in an error.

Dropout terminal output:
I0428 13:03:37.052639 23368 test_bwd_dropout.cc:14] DNNMark suites: Start... I0428 13:03:37.084635 23368 dnnmark.cc:226] Search and parse general DNNMark configuration I0428 13:03:37.084750 23368 dnnmark.cc:286] Search and parse layer configuration I0428 13:03:37.084781 23368 dnnmark.cc:300] Add [Dropout] layer I0428 13:03:37.084897 23368 dnn_layer.h:84] Layer name: dropout I0428 13:03:37.084964 23368 dnnmark.cc:355] DNNMark: Initialize... I0428 13:03:37.084971 23368 dnnmark.cc:356] Running mode: 1 I0428 13:03:37.084977 23368 dnnmark.cc:357] Number of Layers: 1 I0428 13:03:37.084990 23368 dnnmark.cc:359] Layer type: 8 I0428 13:03:37.084995 23368 dnnmark.cc:389] DNNMark: Setup parameters of Dropout layer I0428 13:03:37.085001 23368 dnn_layer.h:110] Bottom dimension: N: 100 C: 1000 H: 1 W: 1 I0428 13:03:37.085032 23368 data_manager.h:44] Create Data chunk of size 100000 I0428 13:03:37.085119 23368 data_manager.h:101] Create data with ID: 0 I0428 13:03:37.085168 23368 data_manager.h:44] Create Data chunk of size 100000 I0428 13:03:37.085182 23368 data_manager.h:101] Create data with ID: 1 I0428 13:03:37.085192 23368 data_manager.h:44] Create Data chunk of size 6065577211415516756 HIP Error at /homelocal/fmendeslocal/DNNMark/core/include/data_manager.h49 hipErrorMemoryAllocation I0428 13:03:37.085266 23368 data_manager.h:53] Free Data chunk of size 100000 I0428 13:03:37.085304 23368 data_manager.h:53] Free Data chunk of size 100000

Can somebody help me to understand what is wrong and how to fix it?

Thank you so much!