I have reduced the model size in half, but the RTF has doubled, what happend?

[Vuducbao913]

I'm trying to reduce the model size to reduce the inference time since the RTF is pretty high right now. I am trying to reduce the model size by reducing the number of ResnetBlock, num resolutions in the default model are 7, so I have reduced it to 3. The current model is about 27M parameters
Then I trained from scratch with the WSJ data, amazingly it gave better results than the pre-trained model on some of our real datasets.
You can listen to the audio here to see some of the differences
https://drive.google.com/drive/folders/1aU_3btAczyzLeecAQwzNFrwiZWzotgaG?usp=share_link

More importantly, however, the RTF has doubled. The current model is much smaller in size than the pre-trained model (27M compared to 65M), I took a lot of time to this problem, what is the problem here, have you ever encountered a similar case?

[ksasso1028]

The size of the feature maps are larger (less blocks to down sample) so the attention block is running on larger inputs. The spatial resolution is a tradeoff